About
What is MySpector?
MySpector is an automated web Inspection Tool. It permits to perform Web Monitoring Fully Automated in Real-Time. It can be used also as an Automated Web Testing Tool, and is a Codeless Web Automation framework. MySpector produces a comprehensive machine vision system made up of four components that together deliver an automatic solution to achieve the ultimate inspection that ensures maximum profit, virtually eliminates any manual repetitive tasks and provides consistent data for automated checking. The workflow of MySpector includes four configurable modules, executed in sequence, that performs the folowing specific tasks:
- Collector: Obtains data from a target (eg http, ftp, databases...)
- Extractor: Isolates a supbart of the data (eg XmlPath, HtmlPath, JsonPath, Text-Before-After-Between)
- Checker: Compares the extracted data to against a reference value (eg Text is[not] contained, number matching)
- Notifier: If the result of the Checker is true a list of notifications is executed (eg mail, http request, tweet...)
MySpector is a single source of truth to check, troubleshoot, debug, any virtual system.
With one unified product that covers an unlimited number of protocols and targets, it is easy to create automation of checking of your applications, with an unrivaled time to market.
With MySpector, you get a centralized context without added tools or tab switching.
MySpector is 100% SaaS based, so you don’t need to purchase, support, or maintain monitoring infrastructure—or pay administrative overhead.
Just configure a Trox and its own set of Triggers to start collecting data and wait for one of its auto-generated the notifications when remote ressource reach a given state.
The big benefits are...
- Provide standardised modules with high flexibility ( Http Protocol, HtmlXPath)
- Reduction in development time for supervision
- Eliminate hosting and maintenance costs
- 100% SAAS with no Deskop App to install
- Inspection/Notification focus solution, providing all tools to configure in a couple of minutes any new supervision
Price monitoring
During the BlackFriday edition of 2020 I was upset to by some prices which were not low enough for me. I had the idea to program a tool to be able to inspect regularly the price of the items to get notifications when price would go sufficiently down. I tried to think about a system to be flexible and usable in different context, and for different use cases. This is what lead me to create MySpector: a toolkit to automate inspection of numeric data.
I rapidly added the idea to check for some text to see when an object is not anymore 'Unavailable'.
With my technical background I also tried to design a multi protocol system, so from the beginning I choose an architecture which will allow MySpector to handle HTTP but also FTP, database connection, direct file access, or even local/remote process execution.
I applied the same principle for the notifications. When an event is trigerred it will be possible not only to send a mail, but also any kind of other notification we can think of (Tweet, slack/discord alert, ...).
In the realisation I like the intelectual challenge to find some ideas and to link them with my technical expertise to create a robust software.
I want MySpector to be your solution to perform automatic http inSpection on the web.
Nominal use cases
Three pictures are worth a thousand words. The first image below is the Dashboard showing 2 Troxes. A Trox is one entity grouping the Target, the Extraction rules, the Checker and the Notifications. The Dashboard shows the Name of the Troxes, if it is enabled for execution and the last result extracted. Even if the data belong to a Third Party website, MySpector isolates the core information:
Automated Web App Testing
The primary goal of MySpector is to monitor a price, but in its final version MySpector can also be used as a monitoring tool to ensure that a target system is behaving correctly. By configuring the extraction rules and the corresponding notifications the user can be alerted when a web server is not behaving correctly. MySpector can also be configured to automate web app testing at low cost and in a super fast fashion. Just configure a target Api in the URL, select the appropriate HTTP verb ( GET, POST, DELETE ...) put a check on the http Response ... and done! Even better you can use the JsonPath extractor to isolate with precision a given field if the target api sends some Json object. Isn't it incredible ? In the coming months, when FTP will be supported, the user will be able to create scenario like: "If on this target FTP server at 4PM, every day, this file is not there, then I want to call this HTTP URL". Yes with MySpector you can automate testing of your target web application without writing a single line of code (no need of Python neither PowerShell). MySpector can proactively monitor your endpoints with API Tests.
Multi-Protocol Gateway system
The idea behind MySpector is to create a Multi-Protocol Gateway system. MySpector is divided into four modules which are executed in sequence.
A multi protocol architecture has been designed for the Input and for the Output of MySpector:
The Input is the Collector module.
The Output is the Notify module.
For the Collector: because MySpector has been designed since the beginning to be multi protocol compliant, new Input protocols will be added during the life of MySpecor.
The very first version of MySpector propose a single protocol to collect data: the Http Protocol.
Imagine you request to monitor a host just to check that a machine is alive and that DNS still
works. MySpector will then add the Ping Protocol, and create a data contract that the Xtrax and Checker modules will control to check that ping is succesfull,
latency is valid. MySpector will then turn in a automated real time ping monitoring tool!
For the Notify: because MySpector has been designed since the beginning to be multi protocol. The same principle is applied for the Collector and for the Notifier with one specific difference: The notification is handled as a list. Hence MySpector will not only be able to notify with other protocols than Mail, but will also be able to send hybrid notifications for one Trox execution. You can think of trigering a WebCall back to automate an action and at the same time add another notification node with a Mail entry for you own personal interest, and why not adding a private tweet to your collegue who prefer to not store his mail on the database of MySpector.
For the technical audience, MySpector can be seen as an implementation of the dot net well known interface
INotifyPropertyChanged
on the web page level.
MySpector is a 'no code' framework: The user can configure it without writing a single line of code.
The user can configure the modules with the graphical web inteface to design simple worflows (Download/Extract/Check/Notify).
MySpector can be powerfull, it is designed to be highly configurable. This flexibility has a cost: the user must take time to enter the correct parameters.
For example the user can find a WebPage which display a $price, fails to find a reliable HtmlXpath because the DOM is updated by Javascript
-MySpector does NOT evaluates JavaScript-. However the technical user will use the 'Developer Tools' which is embedded in most web browsers, like Firefox,
via the Developer tools the user finds a http request to an Api. User copy the input of the Api (http headers and content) and can configure MySpector to call directly the Rest Api
to obtain Json data, and extract the price with the JsonPath Xtrax module.
Unlike other platform which try to guess what the user want to monitor and do not do what the user expect,
MySpector is a toolkit with technical module so you can reach whatever idea you have in mind.
I plan to have in the future the possibility to create dashboards with data extracted by the Troxes. I am sure some unexpected behavior could be found by
creating some loopback between the last stage of a trox (the notify part) to trigger other troxes with conditionnal or unconditionnal checkings.
Later, I also want to design Apis so the result extracted by the powerfull modules of MySpector will be easily accessed with simple http get requests,
thus pushing the interconnection of systems, and pushing the publication of data over the internet.
MySpector can also be used as a system monitoring tool. His feature with the programmable timeout can notify you if your target takes too much time to acknowledge.
The extraction field can be used to check if a status on a target dashboard is at the correct value, hence one scenario is that you check a remote dashboard with MySpector and get
notification if an unexpected status is found by MySpector.
Failover with Auto-Retry
To garantee less noise, and reliable results, MySpector implements a failover mechanism to collect the target. Whatever the protocol to access the target data, and whatever the target, if an error occurs when collecting the data such as a network issue, MySpector automatically retry 3 times after waiting an increasing period of: 10 seconds, then 20 seconds, then 30 seconds. Only after 4 attempts, if the data is still unavailable it will be considered as this. This provides you an embedded failover module to avoid fake signals, and to provide high quality data. If there is a network issue for 20 sec during the Trox execution, the resulting data will not be impacted, as it will be fetched again some seconds later.
Multi-Timeframe Trigering system
Because each customer has its own requirement in terms of targeted system checking period, the Trigger module is designed to handle user defined period with the user defined time frame. From the range of the hour to the week the user can configure with precision when the Troxes must be executed. For more advanced scenarios, the user can define a set of different triggers on the same Trox.
Underlying technology
MySpector is fully developped in C#. Software programming is done with Visual Studio 2019. The software is written with cutting edge Test Driven Development (TDD) methodology (with 220 unit tests running in less than 2 minutes, at the date of October 2022).
MySpector uses Blazor technology, which relies on Asp dot net Core 3 of Microsoft. Blazor is configured on server side to garantee an incredible fast loading time for customers, and benefit from the full dot net framework. Under the hood it relies on WebAssembly and web-Sockets that provide high performance, low latency, low bandwidth. The updates on Blazor pages are done in realtime, with a minimum data transfer on the network resulting from a sophisticated mechanism consisting in finding the differences from a tree of objects contained in the current page. The data is stored on a MySql database with an automated daily backup. The current version of MySpector runs on Windows, and should be installable on Linux because it relies on 'Linux compatible dependencies'. The binaries of MySpector are hosted on a IIS web server v10 on Windows Server 2022. Source code is stored on a (private) repository on GitHub. Deployment is still done manually, but should become automated soon. Analytics are stored on the local database and on Google Analytics. There is no other connection to any other website (besides the content of the Troxes).
SkyNet
MySpector is designed to become part of the SkyNet program. Soon, Terminators will be connected to MySpector. Terminators will setup the call to Google Api and Apple Api with GoogleId and AppleId of their human targets. Terminators will setup the Triggers of MySpector and will receive notifications at a lightning speed using the distributed SkyNet neural network to their dedicated TerminId. The use of MySpector will contribute to save the battery of Terminators and expand their autonomy. Terminators will get notified as soon as a human Geo-Tagged-Target will get geographically next to them.
About the Creator
I am Thierry Brémard, french, living in Germany. I have been graduated in 2008 of the engineering school ESIEE in Paris, where I obtained background in software programming, electronics and telecom. I am currently a professional software Engineer with more than 10 years in C# development. I've mostly worked in consulting firm as C# developper, and I like to use Visual Studio 2019.
I like this project because not only it is full stack (from web interface till database) but also requires skills of Sysadmin, Devops, Database architect.
Developing a full project like this one is an unprecedented source of knowledge growth and satisfaction.
I need to plan in advance recover scenarios and also do some basic marketing and communications to push this outstanding project
that will revolutionize the monitoring industry.
I am also happy to avoid discussion against other experts before action. Thus I avoid 'company like experts disagreement' which 90% of the time is an ego issue.
I am, however, always happy to hear feedback on the Myspector product.
contact: t.bremard[at]gmail.com
Some more keywords for SEO:
MySpector Automate inspection of remote web pages. Configure workflows to check if target reaches a given state. Automated checking and notifications.
Automated Web Testing Tool. Web Application automated testing. Codeless Web Automation. Automate URL Inspection Tool. Web Monitoring Fully Automated Real-Time. Crucial Web Inspection tool
with scheduler and triggers. Automated Web App Testing. Proactively Monitor Your Endpoints with API Tests.