When your uptime monitor goes down with your server

Monitoring works by comparing what you expected against what was observed. Every alert you have ever received exists because something was still running to notice the difference.

If the thing that notices is destroyed by the same event as the thing being watched, no difference is ever recorded. You do not get a late alert or a wrong alert. You get silence, and silence is exactly what a healthy system looks like.

The events that take both halves

None of these are exotic. Each one is a normal Tuesday.

• The OOM killer runs. Linux picks a victim by memory footprint. A monitoring stack with a database and a browser-based dashboard is frequently the largest thing on a small box that is not the app itself. Sometimes it takes the app; sometimes it takes the monitor; on a box that is genuinely out of memory it eventually takes both.
• The disk fills. Logs, an unrotated journal, a runaway upload directory. Postgres stops accepting writes, and so does the monitor’s own database. Neither can record what happened.
• The kernel panics, or the provider reboots the host. Both processes stop in the same instant and come back in the same instant. The gap between them is invisible, because nothing was awake to measure it.
• A network rule goes wrong. A misordered iptables change or a bad security group locks out the box. From outside, the app is gone. From the monitor, which is inside, everything is answering perfectly.
• The monitor’s own certificate expires. You cannot load the dashboard to find out what is wrong, which you discover at the moment you most need it.
• An upgrade goes sideways. apt upgrade pulls a new libc, something fails to restart, and you are debugging over a serial console while your alerting sits on the same disk.

In every case the honest summary is the same: the outage happened, and your monitoring recorded nothing, because it was part of the outage.

Working out how far apart you actually are

Independence is not binary. It is a ladder, and each rung survives strictly more than the one below it.

The monitor runs	Survives
In another container on the same host	A crash of the app process only
On the same host, outside containers	A container runtime failure
On another VM on the same hypervisor	A guest kernel panic
On another host in the same rack or availability zone	A single machine’s hardware failure
In another region at the same provider	A data centre or zone incident
At a different provider entirely	The provider’s control plane, billing, network and routing
On another continent	Submarine cable cuts, national routing incidents, regional BGP mistakes

Most self-hosted setups sit on the first or second rung and are described, in conversation, as though they were on the sixth. The test is one question: name an event that takes down the app but demonstrably leaves the monitor running. If you cannot answer without hedging, you are on a lower rung than you think.

There is a second question worth asking, because it catches the case people miss: who pays the bill for each machine? Two boxes at the same provider on the same card share a failure mode that has nothing to do with hardware. An expired card takes both.

The cheapest arrangement that works

You do not need much distance. You need some.

A second small box at a different provider. The smallest instance almost anywhere costs three to five euros a month and is more than enough to run a monitor for a handful of services. Different provider, different network, different invoice. This is the single highest-value change available to most self-hosted setups, and it is an afternoon of work.

A hosted external check. Someone else’s machines, someone else’s on-call, no second box to maintain. You give up interval granularity and notification breadth in exchange for never patching it. This is what we sell, and it is worth being plain that it is not the only answer.

Heartbeat inversion, for anything without a URL. Instead of something reaching in to poll your job, the job reaches out to say it finished. If the call does not arrive inside the window you set, that absence is the alert. This is the only technique on this page that works for a cron job on a machine with no inbound access at all — we wrote it up separately.

Something has to watch the watcher

If you self-host, the monitor becomes a production service with the unusual property that its failure is silent. It needs the same treatment as anything else you run: it can fill its disk, it can fail to restart after a reboot, its own TLS can lapse.

The recursion has to stop somewhere, and it stops when one link in the chain is somebody else’s problem. In practice that means either an external check pointed at your monitor’s own health endpoint, or a heartbeat your monitor sends outward on a schedule, so that its silence is noticed by something that is not it.

This is the arrangement we would actually recommend to someone who enjoys running their own infrastructure: keep the self-hosted monitor, because it is better at the deep work, and put one cheap external check on the monitor itself. You get the interval granularity and the notification channels where they matter, and you close the silent gap for the price of a coffee.

A checklist you can run in ten minutes

1. List every host involved in serving your application. Include the load balancer and the database.
2. For each monitor you run, write down which host it lives on.
3. Circle every monitor that shares a host, hypervisor, region, provider or payment method with the thing it watches.
4. For each circle, write the event that takes both. If you can write one, that monitor does not cover that event.
5. Move at least one check somewhere with no shared fate at all.
6. Point something at your monitor’s own health endpoint, or have it emit a heartbeat outward.
7. Break it on purpose. Stop the app and confirm the alert arrives. Then stop the monitor and confirm you find out.

Step seven is the one that gets skipped, and it is the only one that produces evidence. An untested alerting path is a belief, not a control.