Subscribe to GEN
Login to GEN
Add a Comment
Oversight 2 is now generally available. It has been running on four live sites ahead of release, and from today it is open to new clients. Existing Oversight 1.x clients will be moved across over the coming months.
The first Oversight was released in 2019. A few years ago it was rewritten and moved to a new platform, gaining a dedicated Proxmox sensor along the way, but underneath it was still the same product. It did a job, but we were never entirely happy with it. It was not comprehensive enough, and it shared too many of the habits that make people stop trusting their monitoring.
Oversight 2 is not another round of patching. It is a complete rewrite, started from a blank page with one question: why do people stop reading the alerts?
Anyone who has carried an on-call phone knows the answer. The problem is rarely that the monitoring missed something. It is that it said so much, so often, about so little that nobody could pick out what mattered from the noise. The usual culprits:
Oversight 2 was designed to deal with each of those, one by one.
A monitor that cries wolf is worse than none at all, because it teaches people to ignore the one alert that matters.
In a traditional monitor, each host belongs to one poller. That single vantage point decides whether the host is up, and when the poller has a bad day, so do you.
Oversight turns that round. Each check is attached to groups of probes in different places (London, Manchester and the cloud, say), and each group forms its own view. The state is then decided by how many of those views agree. One failing view is the internet having a moment. Several is a real problem. All of them is an outage. What each level of agreement means is set per object, so a payment gateway can go critical when only some locations fail, while a mirror merely raises a warning.
Two details are more important than they sound. A probe group that fails to report is left out of the count; it is never counted as healthy. And when a probe dies, its peers notice within about two minutes and take over its work, so the monitoring heals itself rather than going quiet.
When a Proxmox node drops out of quorum, its group and its site can follow it into alarm. Most monitoring either tells you about each one separately or sends "Sensor X is down, and 45 others". Oversight folds the whole incident into one message per rule, drawn as a tree with the worst first: the site, the group, the device that went critical, and the check beneath it that took it there. You read the cause and its consequences together.
The message can also open with a short overview in plain English, saying what has happened, where, what it probably means and how urgent it is. The overview is written by GEN's own AI models, running in GEN's own datacentres. Nothing is sent to a third-party AI provider, and no prompt is logged. That matters more than it might seem, because monitoring data is a map for an attacker: it names every system and shows which ones are failing right now. If the overview takes more than thirty seconds, the alarm is sent without it, so the AI can never delay an alert.
Each channel gets the message in the form it needs: HTML for email, Markdown for Matrix and Rocket.Chat, JSON for webhooks and a short summary for SMS. Recovery notices go only to the people who were told about the failure. Any rule can also be sent as a test to its real recipients, so you can check it works before an incident at three in the morning does it for you.
This is the part we are proudest of. A threshold tells you that something has broken. It cannot tell you that something is about to.
Picture a router whose memory has crept up steadily for three weeks. Every reading on the way up was within limits and every check passed, yet the router is going to fall over. No threshold will catch that, however cleverly it is set. Nor will the usual "anomaly band", because the band quietly rises with the drift.
Statistics in monitoring are nothing new, and most of the established systems offer some form of them: a baseline for each hour of the week with a band drawn round it, or an alert when a value strays more than so many standard deviations from its average. The weakness is in two choices somebody has to make. How wide should the band be, and over what period is "normal" measured? Three standard deviations is too tight for a naturally jumpy sensor and too loose for a steady one. Measure normal over a week, and a slow drift simply becomes the new normal.
Oversight makes neither choice. How far is too far is worked out from each sensor's own history: how restless it usually is, from each probe, on each time scale. Rather than one window, it watches several at once (the day, the week, the month, the same day of the week and the same week of the year), and compares each with its own past. A finding needs those windows to agree on the direction, and needs several different statistical measures to agree that the movement is real. A spike that comes and goes convinces none of them. A slow climb that shows on the day, then the week, then the month convinces them all.
A band says "this reading is high". Oversight says "this sensor has been heading one way for weeks, and that is not how it usually behaves".
When a sensor is moving like that, it is marked ABNORMAL. A reading that is simply odd for the time of week, such as a mail queue at twice its usual size for a Tuesday afternoon, is marked UNUSUAL. Neither is an alarm. Both count as working, both have their own quieter colour, and neither wakes anyone unless you ask it to.
This analysis is switched on sensor by sensor, not across everything, and that is deliberate. Run it over a whole estate and every sensor soon has an anomaly of its own, and people learn to ignore them all.
It has already earned its keep. It has caught a Proxmox Backup Server stuck in a verify job, the kind of fault that raises no error because, as far as the server is concerned, it is still working. It has also caught a failing SSD that was simply getting slower. There was nothing to alarm on, no errors and no failed checks, only a gradual slowdown, which is exactly what a threshold cannot see.
Even in its first fortnight of history, it spotted three checks on three different devices whose connect time rose from 1ms to 6ms over nine days, all measured from one probe. From a second probe, the same checks went the other way. The devices were fine; it was the probe's own host that was failing, and Oversight found it before anything had broken.
The usual way to prove mail is arriving is to have the monitoring system sign in to a mailbox and look. That means leaving IMAP or POP3 switched on, keeping a service account's password in the monitoring system and, on Microsoft 365, registering an app or carving an exception out of your sign-in policy. For all that, it proves only one direction.
Oversight sends a real, signed message from outside, which comes in through your MX like anyone else's mail. Your mailbox then forwards it straight back out through your own outbound route. One message proves that mail is both reaching you and leaving you, with no login to anything. If it does not come back, the result tells you which half to look at. It works with anything that can forward a message: Exchange, Microsoft 365, Postfix, a hosted mailbox or an appliance.
Every established monitor documents a way to run your own commands through its agent: a UserParameter, an NRPE command, a local check, a script sensor. That is how those agents are extended, and it is also how an agent meant only to read a host ends up able to act on it.
Oversight has two agents, one for Linux and one for Windows, and neither of them can run anything. There is no exec, no plug-ins, no custom checks and no write endpoint, and none will be added. Each runs under its own unprivileged account, never as root or administrator. Both also do their judging on the host, where the context is, and report a verdict in plain words: red means it is not working, amber means someone should look soon.
On Linux, it will tell you about a mirror that has lost a disk, a filesystem that has gone read-only, a service stuck in a crash loop, or a disk that will fill within twelve hours. A single command installs or upgrades it, and it runs on everything from servers and Synology units to 32-bit ARM boxes and NVIDIA GPU hosts.
On Windows, it watches for the failures administrators know all too well, and often catches them before the server falls over:
The events it watches for come from a fixed list that GEN maintains and signs, so a new Windows release needs only an updated list, not a new agent on every server. It runs on Windows 10 and Server 2016 onwards, on x64 or ARM, and installs from a single downloaded file, checked against its published hash, with no script to run.
Oversight 2 reads what modern platforms report about themselves, and turns a single fetch into as many alarms as the answer warrants:
Everything Oversight knows is available through its API: the estate tree, the current state of every object, readings, every change of state, availability worked out exactly as the dashboard works it out, and the charts themselves as images. It answers in JSON or CSV, in flat rows that Grafana reads as they come, so a wallboard, a customer portal or an archive can draw on it directly. Each integration gets its own key, which can be limited to one branch of the estate and to named addresses.
It works the other way too. A dedicated sensor type accepts readings sent to it, so anything that can make an HTTPS request can feed Oversight. A few lines of bash and curl in a cron job, a figure from an application's own database or a count from a script of your own is ingested exactly like a reading from a probe. The values are extracted, judged against conditions you set, rolled up through the tree, charted, reported on and alarmed on, like any other sensor.
At heart, most of the established monitors are software you run yourself: a central server, a database, pollers or proxies, and then the upgrades, backups and security patching of the one system that can see everything you own. Some do offer hosted editions, but underneath, the design is still one server and one poller deciding what is true.
Oversight is a managed service. GEN runs it end to end, so there is no monitoring server of your own to look after. We also run the probes that sit in your racks and offices, and they are built so that they can never become a way in:
Monitoring touches everything, so it must never be the weakest point on the network.
There is no per-sensor licence, no tiers and no prepayment. Each read (one result, from one probe group, for one check) costs a fixed number of credits depending on what it involves, and each credit costs £0.00002. A ping every minute from one location comes to about 86p a month. Rules, schedules, templates, escalation, email, Matrix, Rocket.Chat and webhooks are all included, and SMS and voice are metered separately. A probe group that is down produces no reads, so it costs nothing, and every check shows its monthly cost while you configure it. Usage is invoiced monthly in arrears.
We have compared Oversight with Zabbix, SolarWinds, PRTG, Nagios and Checkmk, feature by feature, in the Oversight comparison. Every statement there about another product comes from that vendor's own documentation, with the sources listed. Where a vendor documents nothing either way, the table says so rather than claiming a "no". They are all capable products, and some have been around for a very long time. We simply think the problem deserved a fresh start.
To see Oversight for yourself, visit oversight.im, read the Oversight service page, or raise a ticket and we will walk you through your own estate. Monitoring you can believe at three in the morning is not too much to ask.
--- This content is not legal or financial advice & Solely the opinions of the author ---
