I found out my certificate had expired the way everybody finds out: a browser full of red on a service I use every day. That was annoying. The next thing I learned was worse. certbot.timer on that host had been failing on every run for months, and not one thing had told me about it.
The host is a Ubuntu 24.04 box on my home connection running nginx as a reverse proxy in front of a stack of Docker services. Residential WAN, which means inbound port 80 is blocked or in use by another tech minded person. That single fact is the root of the whole story, because port 80 is exactly what Let’s Encrypt’s HTTP-01 challenge needs. The ACME server has to reach http://yourdomain/.well-known/acme-challenge/<token> on port 80. It will follow a redirect once it gets there, but the first request is always to port 80 and there is no way to move it. If your ISP swallows that port, HTTP-01 is not something you tune. It is something you cannot use.
Nothing was going to shout at me about it either. Let’s Encrypt ended its expiration notification emails on June 4, 2025, so the safety net that used to catch exactly this failure is gone. A silent renewal failure now stays silent right up until the certificate dies and your browser tells you.
I needed a challenge type that never touches port 80, and an alerting path loud enough that the next failure reaches me at all. This post walks through both, in the order you would do them on your own host. Everything is in the companion repo at certbot-discord.
The half-solution that was already there
Before the rewrite, the host used certbot --manual --preferred-challenges dns, which meant pasting TXT records into Cloudflare by hand every ninety days. That works exactly once, in a terminal, with a human present. Under the systemd timer it produced this on every run:
An authentication script must be provided with --manual-auth-hook
when using the manual plugin non-interactively.
certbot was doing the right thing. The manual plugin cannot run unattended without a hook. A timer that fails identically twice a day, though, looks a lot like a timer doing nothing at all.
Why DNS-01 fixes it
DNS-01 proves control of a domain by publishing a TXT record at _acme-challenge.yourdomain, which Let’s Encrypt then looks up. Validation happens entirely over your DNS provider’s API. Port 80 is never involved, your firewall is never involved, and your ISP’s opinions about inbound traffic stop mattering.
DNS-01 also issues wildcards. HTTP-01 cannot issue *.example.com at all, and Let’s Encrypt only offers wildcards over DNS-01. Once you have one, adding a new service subdomain never involves the certificate again, which on a homelab separates “spin up a container” from “spin up a container and then remember the cert dance.”
It works for hosts the internet cannot reach, too. Because nothing has to connect to your server, you can get a publicly trusted certificate for a service that only listens on your LAN, or sits behind a VPN, or is reachable only through a tunnel. The one requirement is that you control the DNS zone. How your users reach the service is a separate question from how you get the certificate.
What you need before you start
- A domain whose nameservers point at Cloudflare. It does not matter whether individual records are proxied (orange cloud) or not; DNS-01 only needs the API.
- A Linux host with nginx and certbot installed. This was written on Ubuntu 24.04, and
sudo apt install certbotis enough. The setup script installs the Cloudflare plugin itself but assumes certbot is already there. - Root on that host.
- Outbound HTTPS. The host talks to
api.cloudflare.comand to Let’s Encrypt. Nothing inbound is required for issuance. - A Discord server where you can create webhooks, meaning the Manage Webhooks permission on it.
Clone the repo somewhere on the host, then deal with the placeholders before running anything. Every script ships with example.com, admin@example.com and /home/youruser in it. The README maps each occurrence by file and line, and grep confirms it:
| |
Fix every hit before you go further. A missed one usually surfaces later as a monitor watching a certificate path that does not exist.
Step 1: notifications first
It is tempting to do the certificate first, since that is the problem you came here with. Do the notifications first anyway. Every step after this one can fail, and with alerting already in place those failures reach you the day they happen.
One URL and no new services
There are perfectly good tools for this. Notifiarr, Pushbullet, ntfy, Gotify, healthchecks.io: all of them do notifications more thoroughly than a shell script does, and if you are already running one, use it. But each is another service to stand up, another account to hold, another key to rotate, and in several cases another container sitting right next to the containers it is meant to be watching. I already had a Discord server for the homelab with a channel nobody posts in, and a Discord webhook avoids all of that. It is a URL that accepts an HTTP POST. No SDK, no daemon, no auth flow, no OAuth refresh to break at three in the morning. You send it with curl.
Simplicity is the requirement here. An alerting path has to be more reliable than the machinery it watches, or a failure in the alerting hides the failure it exists to report. A webhook has almost no moving parts, and it lives in a script short enough to read in one sitting.
Creating the webhook
In Discord: Server Settings, then Integrations, then Webhooks, then New Webhook. Pick the channel you want alerts in, give it a name, and hit Copy Webhook URL. You get something shaped like https://discord.com/api/webhooks/<numeric id>/<token>.
Treat that URL as a credential. It grants nothing beyond posting to that one channel, but anyone holding it can post there, so it goes to /etc/discord-webhook with mode 0600 and is in the repo’s .gitignore next to the Cloudflare token. If it ever leaks, regenerate the webhook in Discord and the old URL dies immediately.
Now install the notifier:
| |
It puts discord-notify in /usr/local/bin, then runs discord-notify --setup, which prompts for the URL, stores it and posts a test message. If a webhook is already configured it keeps the existing one, so re-running the installer is safe.
The store itself is unremarkable:
Between the paste and that write there is a step I did not expect to need. Most terminals wrap pasted text in bracketed-paste escape sequences, and some editors and password managers add stray control characters or wrapping quotes. None of it is visible, all of it lands in the file, and the resulting POST fails in a way that looks exactly like Discord rejecting a valid URL:
Then it validates the cleaned result before saving anything, because the most common mistake by a wide margin is copying the channel link or a server invite:
The canary. and ptb. alternatives cover Discord’s beta clients, which hand out webhook URLs on those hostnames, and discordapp.com is the legacy domain that still shows up in older documentation. If validation fails, the script tells you how many characters it ended up with and shows the first thirty-four, which is usually enough to see what went wrong without printing the token.
Posting to it
Sending a message is a JSON POST. The body gets wrapped in an embed so alerts carry a coloured stripe. A red renewal failure reads differently from a grey routine notice when you are scrolling past on a phone:
color is a decimal integer, which is easy to get wrong the first time. The script accepts names and maps them:
Escape the body with python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))' and skip hand-rolled quoting entirely. Journal output contains quotes, backslashes and newlines as a matter of routine, and every one of them will eventually break a JSON string you assembled with printf.
The result is a command any script on the box can use with one pipe:
Wiring it to systemd
With a notifier that is just a command, the installer can hook it to anything systemd runs, using OnFailure and a template unit:
%N expands to the unit name without its suffix, so the drop-in on certbot.service instantiates discord-failure@certbot.service. That template runs a small helper which collects the unit’s state, result and exit code, then the last sixty journal lines, and pipes the lot to discord-notify. The same drop-in works for any unit, so the installer takes extra unit names as arguments:
| |
Sixty journal lines is usually more than Discord will take inline. Message content caps at 2000 characters and an embed description at 4096, and the script uses a conservative 1900 to leave room for the code fence it wraps around the body. Anything longer is uploaded as a .txt attachment with a twelve-line preview in the embed, so nothing gets cut off at whatever turns out to be the interesting part.
That attachment path cost me a real outage’s worth of confidence. The payload has to go through --form-string, not -F:
With -F, curl reads a ; inside the value as the start of a modifier such as ;type=, truncates the JSON there, and Discord rejects the whole post with a complaint that payload_json is not valid JSON. Journal output is full of semicolons, so this fails on every real alert while passing every short test message. The file part still uses -F, because it needs the ;type= modifier.
A notification system is proven when you have watched a real alert land. The setup test and a real alert take different code paths here, and only one of them worked. Fire a real one on purpose:
| |
That posts a failure-shaped alert for certbot.service, journal tail and all. If it does not appear in your channel, fix that before moving on.
Step 2: the Cloudflare API token
In the Cloudflare dashboard, go to My Profile, then API Tokens, then Create Token, and choose the “Edit zone DNS” template.
The template’s permission set is Zone, DNS, Edit, and the plugin documentation confirms that is all certbot needs.
Change Zone Resources from all zones to Include, Specific zone, and pick the one zone. That separates a credential which can edit one domain’s DNS from one which can edit everything in your account.
Set a TTL. An API token with no expiry is a credential that lives on a home server forever. Give it a year, or at least a cadence different than the certs you are renewing, and let the monitor remind you.
Use a scoped token. The Global API Key authenticates as your entire account and cannot be scoped to anything.
Cloudflare shows the token exactly once. The setup script will prompt for it, or you can pass it in the environment for a non-interactive run:
It lands where the plugin expects it, at mode 0600:
certbot checks that mode. If the file is readable by anyone else it emits an “Unsafe permissions on credentials configuration file” warning on every run, renewals included, and fixing the mode is the only way to stop it.
Step 3: issue the certificate
The same script continues into issuance. First it installs the plugin, but only if it is missing:
Then it builds the certbot invocation once and uses it twice:
| |
What each argument does:
--dry-run runs the whole flow against Let’s Encrypt’s staging environment, so a typo in your domain or a wrongly scoped token costs you nothing. Production rate limits are not generous enough to debug against.
--dns-cloudflare-propagation-seconds 30 is the wait between writing the TXT record and telling Let’s Encrypt to look for it. The default is shorter and you can lose that race, producing a validation failure that then succeeds on retry. Thirty seconds has been reliable for me. If you see intermittent failures, raise it.
-d "$DOMAIN" -d "*.${DOMAIN}" is the payoff. One lineage, apex plus wildcard.
--key-type ecdsa changes nothing on its own: certbot has defaulted to ECDSA P-256 for new certificates since version 2.0. I keep it in the script because a lineage’s key type is recorded at issuance and inherited on every renewal after that, so four words now pin it for the life of the certificate.
--cert-name pins the lineage directory name. Everything downstream, nginx and the monitor both, hardcodes /etc/letsencrypt/live/<cert-name>/, and without it you can end up with a -0001 suffix and a lot of confusion.
When it finishes it prints the expiry and the SANs:
Where certbot looks for hooks
Reloading nginx after renewal belongs in a deploy hook, which certbot runs only when a certificate was replaced:
Note the path carefully. certbot executes hooks from the pre/, post/ and deploy/ subdirectories only. A script sitting loose in renewal-hooks/ never runs, and you can watch it not run several times before the path occurs to you.
The three also differ in when they fire. Post-hooks run on every attempt, failures included. Deploy hooks run only when a certificate was replaced. Reloading nginx from a post-hook means reloading twice a day forever for no reason; reloading from a deploy hook means reloading roughly six times a year, when the files on disk changed.
One ordering gotcha lives in this repo. Both setup-cloudflare-certbot.sh and install-discord-notify.sh write that same deploy hook path, with different contents. The setup script installs a plain reload; the Discord installer installs a version that reloads and then posts a green “certificate renewed” notice, or a red one if the reload failed. Installing notifications first means the setup script overwrites the notifying version with the plain one. Re-run the notification installer after issuance to put it back:
| |
It is idempotent. Your existing webhook survives, so re-running costs nothing, and six renewal posts a year is a heartbeat telling you the chain still works.
Step 4: point nginx at the certificate
The certificate exists but nothing is serving it yet. In each TLS vhost:
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
Because the lineage is a wildcard, every vhost points at the same two files regardless of subdomain. Then the usual:
| |
Verify from the host rather than from your laptop, so DNS and proxying do not confuse the result:
You want to see the wildcard in the SANs and an expiry roughly ninety days out.
If you had older per-subdomain certificates before this, delete them now. They will fail on every renewal run otherwise:
Do that only after nginx is pointed at the new lineage, since deleting a certificate nginx still references will break the config test.
Step 5: confirm unattended renewal works
Renewal is what was broken for months here, so verify it properly.
| |
That exercises the real renewal path, plugin, credentials and hooks included, against staging. If it passes, the failure you started with is gone.
Then look at the timer, because the packaged schedule surprises people:
The packaged unit is OnCalendar=*-*-* 00,12:00:00 with RandomizedDelaySec=43200 and Persistent=true, which runs it twice a day at a randomised offset. certbot then renews a certificate only inside the last thirty days of its life, so most of those runs do nothing. The frequency exists so a transient failure has many chances to correct itself before anything expires.
Twice a day, every day, that unit was failing on my host. Now it posts to Discord.
Step 6: watching the Cloudflare token
Switching to DNS-01 traded a blocked port for an API token, and I gave that token an expiry date on purpose in step 2. If it lapses or gets revoked, renewal stops and I am back at the start of this post.
| |
That installs the check as a systemd timer (Monday 08:00, Persistent=true, fifteen minutes of jitter) rather than a cron job, specifically so the check itself gets an OnFailure handler. If the monitor breaks, you hear about the monitor.
The check asks Cloudflare directly:
The response carries status and expires_on, so rotating the token updates the monitor for free with nothing to edit. A FALLBACK_EXPIRY constant is used only when the API cannot be reached at all. If Cloudflare reports the token as invalid or expired, that is an immediate red alert, because renewal is already broken.
One threshold in there is deliberately odd:
The check runs weekly, which is why the threshold is thirty-five. A 30-day threshold can first fire on a run landing at 24 days remaining, since the previous week’s run sat at 31 and stayed quiet. Thirty-five guarantees the first warning arrives between 29 and 35 days out. Whenever a check interval and a warning threshold sit close together, the threshold has to absorb a full interval or you lose a cycle of notice.
Exercise the alert path:
| |
One caveat if you adapt this. /user/tokens/verify only accepts user API tokens. Cloudflare’s newer account-owned tokens verify at /accounts/{account_id}/tokens/verify and return code 1000, “Invalid API Token”, at the user endpoint even when they are perfectly valid and correctly scoped. If your token came from Manage Account, you need the other endpoint, and the monitor as written would tell you your working token is dead.
There is also scripts/cert-monitor.sh, which watches the certificate itself and alerts if it is inside thirty days and renewal fails. Nothing in the repo installs it: nginx-config-manager.sh cert-status expects it at ~/scripts/cert-monitor.sh, so copy it there yourself and add your own timer if you want it on a schedule. Its 30-day threshold deliberately overlaps certbot’s own renewal window, which makes it a second opinion on the same window.
Step 7: the emergency config
Everything above aims at not having an outage. The emergency config is for the outage you have anyway.
When a certificate problem is actively blocking access, either an expired cert that browsers refuse or missing cert files that make nginx -t fail so nginx will not reload at all, there is a TLS-free copy of the vhosts you can swap in. Services come back over plain HTTP while you fix the real problem.
| |
This one has no installer on purpose. Copy it to ~/scripts/ and run it from there, since you may be reaching for it on a box where systemd is already unhappy.
It had a bug with a common shape. If the emergency config file was missing, the copy failed, the live config stayed in place, nginx -t then validated that untouched live config, and the script printed success. During an outage, a false success is the worst answer available. Now it refuses:
And if the emergency config is present but fails validation, the backup is restored and nginx reloaded before it exits non-zero.
Generating the fallback from the live config
The bigger problem was that the emergency config used to be a hand-maintained snapshot. That is the wrong shape for a file only ever read during an outage: add, rename or remove a service and it goes stale, and you find that out at the exact moment you need it correct.
So generate-emergency-config.sh builds it from the live config instead. It pulls server_name and proxy_pass out of each server block and emits an HTTP-only equivalent:
Those two regexes carry constraints you need before pointing this at your own config. The block pattern expects top-level server { blocks that close with a } at the start of a line, which is conventional nginx formatting but not guaranteed. And the proxy_pass pattern only matches numeric http://IP:port upstreams, so if you proxy to Docker service names or named upstream blocks, those vhosts get skipped. Widen the pattern for your setup, then check the vhost count in the log against what you expect.
Each generated vhost keeps the ACME challenge location, so renewal still works while you are in emergency mode:
server {
listen 80;
server_name sonarr.example.com;
location /.well-known/acme-challenge/ {
root /var/www/html;
}
location / {
proxy_pass http://10.0.0.5:8989;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
}
}
The generator is mostly refusals. Each exists so it cannot make an outage worse:
| Situation | What it does |
|---|---|
| Live config has no TLS | Emergency mode is already applied, so the live config is the emergency config. Skips, so it cannot regenerate from itself and overwrite a good copy with a truncated one. |
| Output fails sanity checks | Empty file, unbalanced braces, no server blocks, or any ssl_certificate or listen 443 that leaked through. Candidate discarded, existing file untouched, red alert sent. |
| Redirect-only vhosts | Omitted deliberately. No proxy_pass means nothing to serve over HTTP. |
Install it behind both triggers:
| |
That gives you a .path unit that regenerates the moment the live config changes, and a weekly .timer at Monday 07:30 as a backstop for anything inotify does not see, such as the path unit being masked or the file being replaced by a method that raises no change event. Both report their own failures through the same Discord path.
Useful afterwards:
| |
The emergency file is now a generated artifact. Edit the live config and it follows; edit the emergency copy by hand and your changes vanish on the next run.
Rehearse it once on a quiet evening: run emergency, confirm the services answer over HTTP, run normal, confirm HTTPS is back. A fallback you have never exercised is a hypothesis.
What ends up running
| Unit | Schedule | Purpose |
|---|---|---|
certbot.timer | Twice daily, randomised | Renews inside the last 30 days. Alerts red on failure. |
| certbot deploy hook | On a real renewal | Reloads nginx, posts green. Roughly six a year. |
cloudflare-token-monitor.timer | Monday 08:00 | Warns at 35 days, escalates at 7, red if revoked. |
emergency-config-sync.path | On config change | Regenerates the HTTP-only fallback. |
emergency-config-sync.timer | Monday 07:30 | Backstop for the above. |
discord-failure@.service | On any wired failure | Posts the failed unit’s journal tail. |
When something goes wrong
| Symptom | Likely cause |
|---|---|
| Renewal fails with an authorization error from Cloudflare | Token scope. It needs Zone, DNS, Edit, and the zone has to be in Zone Resources. A token from the wrong Cloudflare account looks identical and fails the same way. |
| Intermittent validation failures that pass on retry | TXT propagation. Raise --dns-cloudflare-propagation-seconds. |
| “Unsafe permissions on credentials configuration file” on every run | chmod 600 the cloudflare.ini. This cannot be silenced any other way. |
| Token monitor reports a valid token as invalid | Account-owned token hitting the user verify endpoint. Use /accounts/{account_id}/tokens/verify. |
| Hook never runs | It sits in renewal-hooks/; certbot only reads the deploy/ subdirectory. |
| Discord returns “Invalid Form Body” | Usually an embed field over its limit, or -F where --form-string belongs. |
| Setup test posts fine but real alerts never arrive | The attachment path. Long bodies take different code than short ones. Fire discord-failure@certbot.service to test the real path. |
| No renewal success posts, failures work fine | The deploy hook got overwritten by the setup script. Re-run the notification installer. |
Adapting it
Almost none of this is Cloudflare-specific. certbot ships DNS plugins for Route 53, DigitalOcean, Google Cloud DNS, Linode, RFC 2136 and plenty more, and the only parts that change are the plugin name, the credentials file format, and the token monitor, which is Cloudflare-only. Everything else, the deploy hook, the OnFailure wiring, the generated fallback config, cares about none of it.
The notification side is even more portable. discord-notify takes a title, a colour and a body on stdin. Point it at a Slack or Teams incoming webhook by changing the payload shape, or swap it for an ntfy publish, and every caller keeps working, because none of them know what is on the other end.
What it bought
DNS-01 solved the renewal problem in about an hour. Everything after that was the harder half: making sure that when this breaks again, and it will, something tells me before a browser does. Renewal alerts on failure and posts on success. The token warns a month before it lapses. The emergency fallback keeps itself current and refuses to lie to me about whether it worked.
If you are running anything real on a residential connection, DNS-01 is very likely your only sane path to a certificate, and Cloudflare’s API makes issuance a fifteen-minute job. Do not stop there. An unattended system you never hear from is worse than a manual chore you remember, because at least you know the chore exists.
The alerting half is smaller than it looks. A webhook URL in a file with mode 0600, a curl wrapper, and OnFailure= on the units you care about is maybe an hour of work with nothing new running afterwards. Discord, Slack, Teams or ntfy, the destination is interchangeable. A failed timer that reaches a phone is the part I did not have.