How to monitor a cron job with a heartbeat URL

A scheduled job has a property that makes it uniquely dangerous: when it works, it is silent, and when it stops running entirely, it is also silent. Those two states are indistinguishable from outside, which is why backups are so often discovered to have been failing for four months.

You cannot poll a cron job. There is no URL to request and no port to open. The check has to run the other way round.

Why the built-in mechanisms do not save you

Cron does have a reporting mechanism. MAILTO sends you any output the job produces. In practice this fails for reasons that have nothing to do with cron:

• Most servers have no working local MTA, so the mail is written to a spool nobody reads.
• If it does send, it sends on every run, so you filter it, and then you have filtered the alert too.
• It only fires when the job runs and produces output. If cron itself is not running, if the crontab was wiped by a reimage, if the user was deleted, or if the machine is off, there is nothing to mail.

That last category is the one that matters. The failure you need to catch is not “the job errored”, it is “the job did not happen”.

The inversion

Instead of something reaching in to check the job, the job reaches out to say it finished. You register the URL with a monitor and tell it how often to expect a call. If the call does not arrive inside that window, the absence itself is the alert.

This is a dead man’s switch, and it has the property you want: it fails loudly. The machine being off, the crontab being empty, the disk being full, the script crashing on line one — all of them produce the same visible outcome, which is that no call arrived.

The minimal version

At the end of your script:

curl -fsS -m 10 --retry 3 https://ismyappup.com/beat/6f2a1c9e

The flags matter more than they look:

• -f makes curl exit non-zero on an HTTP error instead of cheerfully printing the error page.
• -sS silences the progress meter but keeps real errors, so cron’s output stays quiet unless something is wrong.
• -m 10 caps the whole request. Without it, a hung connection can leave your job running for hours and block the next one.
• --retry 3 survives a brief network blip, which would otherwise page you about a job that ran perfectly.

If you put this in a crontab directly, chain it with &&, never ;:

17 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 https://ismyappup.com/beat/6f2a1c9e

With ; the ping fires whether the backup succeeded or failed, and you have built a monitor that reports success unconditionally. This is the single most common mistake, and it is silent.

Doing it properly

For anything you actually care about, signal the failure too, and measure the duration:

#!/usr/bin/env bash
set -euo pipefail

BEAT="https://ismyappup.com/beat/6f2a1c9e"
ping() { curl -fsS -m 10 --retry 3 "$BEAT$1" >/dev/null || true; }

trap 'ping /fail' ERR

ping /start

pg_dump --format=custom "$DATABASE_URL" > /var/backups/db.dump
rclone copy /var/backups/db.dump remote:backups/

ping ""

Four things are happening here.

set -euo pipefail makes the script stop on the first error, on an undefined variable, and on a failure anywhere in a pipeline. Without pipefail, pg_dump | gzip > out.gz reports success when pg_dump fails and gzip happily compresses nothing.

The /start ping lets the monitor measure how long the job takes, which is how you find the backup that has been getting slower for six weeks and is about to overrun its window.

The ERR trap sends /fail if any command fails, so you hear about the error immediately rather than waiting for the grace period to lapse.

The || true on the ping itself is deliberate. A monitoring call that fails must never fail the job it is monitoring.

Grace periods

The monitor needs to know how long to wait before it decides the job is missing. Set the grace period to the job’s expected interval plus a realistic allowance for it running long — normally somewhere between 20% and 50%.

Too tight and you will be paged every time the nightly backup takes an extra four minutes. Too loose and a daily job that died on Monday reports healthy until Wednesday. If you have /start pings, look at the actual distribution of durations over a month and set the window from that rather than from a guess.

systemd timers

If you have moved off cron, the same idea attaches to the unit rather than the script:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStopPost=/bin/sh -c 'test "$EXIT_STATUS" = "0" && curl -fsS -m 10 --retry 3 https://ismyappup.com/beat/6f2a1c9e'

ExecStopPost runs whether the job succeeded or failed, and $EXIT_STATUS lets you ping only on success. This is better than putting the curl inside the script, because it also fires when the script is killed by the OOM killer or by a timeout.

What goes wrong

• Pinging at the start instead of the end. Now you are monitoring that cron fired, not that the work completed. Ping at the end, or ping at both and use /start only for timing.
• Pinging inside a loop. One successful item out of two hundred keeps the check green forever.
• Using ; instead of &&. Covered above. Worth checking your existing crontabs for this today.
• Cron’s PATH. Cron runs with a minimal environment. curl is usually at /usr/bin/curl, but rclone, aws and anything installed to /usr/local/bin frequently are not on the path. Use absolute paths, or set PATH at the top of the crontab.
• No timeout. A hung ping keeps the job alive and can stack overlapping runs until the machine falls over.
• Running under flock and never checking it. If a previous run is still holding the lock, flock exits without running your job and without pinging, which is correct — but only if your grace period is long enough to tell a genuinely stuck job from a slow one.

Test it by breaking it

Comment out the job, or stop the timer, and wait for the grace period to lapse. Confirm the alert arrives, in the channel you expect, on the device you will actually be holding.

Then put it back. An untested dead man’s switch is indistinguishable from a working one right up until the day it matters.