SSL certificate expiry: how to stop being surprised by it

Let’s Encrypt has been issuing free certificates with automatic renewal for a decade. Certificates still expire in production, regularly, at companies with competent engineers.

The reason is that “renewed” and “being served” are two different states, and almost nothing checks the second one.

Renewal is not deployment

certbot renew succeeding means a new file exists on disk. Your users are served whatever your web server loaded into memory, which may be the old one. The list of ways these diverge is longer than people expect:

• The reload hook did not run, or ran and failed. Nginx keeps serving the certificate it read at startup. The file on disk is current; the certificate on the wire is not. Everything looks fine to anyone who checks the file.
• The renewal ran on one node. Behind a load balancer with three application servers, a renewal that only fires on the machine you set up by hand covers a third of your traffic. The failure is intermittent by source IP, which makes it maddening to diagnose.
• The HTTP-01 challenge stopped working. Someone added a redirect rule, a WAF, or an authentication layer in front of /.well-known/acme-challenge/. Renewal has been failing for two months, into a log nobody reads.
• The certificate is not the one you are thinking of. Your CDN terminates TLS at the edge with its own certificate, so the origin certificate expires without any visible symptom until someone connects directly, or until the CDN starts validating the origin.
• It is not a web server. SMTP on 465 and 587, IMAP on 993, a Postgres server with ssl=on, an MQTT broker, an internal gRPC service. These frequently use certificates issued once by hand, and nothing renews them at all.
• The intermediate expired, not the leaf. Your certificate is valid, the chain you serve is not. Modern browsers often paper over this; older clients and non-browser HTTP libraries do not, so it presents as “the website is fine but the mobile app is broken”.
• A client certificate expired. Mutual TLS to a payment processor or a bank. Nothing about your public surface changes; a single integration simply starts refusing you.

Every one of these is invisible to a check that reads the file on disk, and visible to a check that opens a connection and looks at what comes back.

Checking what is actually served

The only reliable check is the one that connects the way a client would.

echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

-servername sets SNI. Leave it off on a host serving multiple sites and you will get the default certificate, which is often not the one you meant to check.

For a yes-or-no answer suitable for a script, -checkend takes seconds and exits non-zero if the certificate expires within that window:

openssl s_client -servername example.com -connect example.com:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -checkend $((14 * 86400)) \
  || echo "example.com expires within 14 days"

The same works for anything speaking TLS. Add -starttls smtp for port 587, -starttls imap for 143, -starttls postgres for 5432.

To see the whole chain as presented, rather than only the leaf:

openssl s_client -servername example.com -connect example.com:443 -showcerts </dev/null 2>/dev/null \
  | grep -E 'depth|verify|s:|i:'

If verify return code is anything other than 0 (ok), you have a chain problem regardless of what the expiry dates say.

What to monitor, and when to be told

Monitor every hostname and port that terminates TLS, not every domain you own. These are different lists, and the second one is usually shorter than the first.

Walk the actual surface: apex and www, every subdomain that resolves, your API, your status page, the admin interface nobody remembers, mail, and any internal service with a certificate. If a client opens a TLS connection to it, it belongs on the list.

Three alerts, not one:

• Thirty days out. Informational. If auto-renewal is healthy this should never fire, so when it does, it is telling you renewal is already broken and you have a month to fix it calmly.
• Fourteen days. Actionable. Something is wrong and it is now your week’s problem.
• Three days. Urgent, and it should reach someone who is awake.

The thirty-day alert is the valuable one, precisely because it fires long before anything breaks. If your only threshold is seven days, you have arranged to find out during the week you can least afford it.

Lifetimes are getting shorter

The CA/Browser Forum has agreed a schedule that steps maximum certificate lifetimes down over several years, reaching 47 days by 2029. Domain validation reuse shrinks alongside it.

The practical consequence is that manual renewal stops being viable everywhere, not just where it is already inconvenient. Anything renewed by hand once a year becomes something renewed by hand eight times a year, which nobody sustains. If you have services with hand-issued certificates — internal tools, mail, database TLS — the work of automating them is worth starting before the deadline makes it urgent.

Shorter lifetimes also make the deployment gap worse rather than better. More renewals means more opportunities for the reload hook to fail, and less slack between “renewal broke” and “the certificate is dead”.

A checklist

1. List every hostname and port that terminates TLS. Include mail, databases and internal services.
2. For each, check what is actually served, using s_client with the correct SNI, not the file on disk.
3. Confirm verify return code: 0 (ok) on each. A valid leaf with a broken chain still breaks clients.
4. Find anything renewed by hand and automate it, or write down who renews it and when.
5. Confirm your renewal hook reloads the service, and that it runs on every node, not just the one you built first.
6. Set alerts at thirty, fourteen and three days.
7. Once, deliberately, let a staging certificate get close to expiry and confirm the alert arrives.

Step seven is the difference between having monitoring and believing you have monitoring.