I have already written about renewing a cert from behind a residential ISP, and that post is the reason this one exists.

The short version of it: my home connection blocks inbound port 80, which rules out Let’s Encrypt’s HTTP-01 challenge outright, so certbot on my nginx host does DNS-01 against Cloudflare instead. Getting there took a credentials file, the Cloudflare plugin, a systemd timer, and a deploy hook to reload nginx afterwards. It also took a Discord webhook whose only job is to shout at me when a renewal fails, because the thing that started that whole project was discovering that certbot had been failing twice a day for months and nothing had said a word.

All of that works. I still run it. But look at what it actually is: five moving parts bolted to the side of a web server that has no opinion about certificates whatsoever. nginx does not know what ACME is. It reads a certificate off disk. Something else is responsible for putting the right bytes in that file, something else again is responsible for reloading nginx once they are there, and a third thing is responsible for telling you when any of that stopped happening. Every one of those is a component I own, and the failure mode of the whole arrangement is maybe a discord notification if I did the logic right.

Caddy is the other design. It has an ACME client built in, and it treats a certificate as part of serving a site rather than as a prerequisite you arrange beforehand. You name a site in the configuration file. Caddy obtains the certificate, renews it in the background well ahead of expiry, and reloads itself. There is no timer to fail silently, because there is no timer. There is no deploy hook to reload the server, because the thing renewing the certificate is the server.

That is why I put Caddy in front of the identity stack I have been building in this series, and it is the whole of the decision. This post is the certificate half: the challenge, the build, the token, and why the certificate authentik already ships with cannot do this job. The next post is what I wired through it.

One piece of vocabulary

The previous post in this series defined several authentik terms and this one needs exactly one of them.

An outpost is the component that sits at the edge of a request and answers the question “is this person logged in”. Mine is the embedded one, which runs inside the authentik server container rather than as a separate deployment. It matters here only because it is the thing Caddy will eventually be talking to, and because it is why authentik has a certificate of its own at all.

Same challenge type, different owner

If you read the certbot post, the next part will look familiar, because the challenge type is identical. What changed is who runs it and what happens when it breaks.

Every internal name I care about lives under one domain and resolves through Pi-hole to a private address. There are no public DNS records for any of them.

Certificates from Let’s Encrypt are issued through a protocol called ACME, which is an automated way of proving to a certificate authority that you control a name before it hands you a certificate for it. The proving step is a challenge, and two of them matter here.

HTTP-01 has you place a file at a well-known path on your web server, which Let’s Encrypt then fetches over the public internet on port 80. It needs the name to resolve publicly, to an address they can reach, on a port your ISP has not swallowed. On my home line that last condition fails, and for these internal names the first one fails too. There is no configuration that rescues either. Let’s Encrypt cannot look up a name that has no public record, and it cannot reach a port your ISP is holding.

DNS-01 proves control differently. Instead of serving a file, you create a TXT record in the domain’s DNS zone and Let’s Encrypt looks it up. Nothing connects to your server. Nothing needs to reach it. The zone is at Cloudflare, so a scoped API token lets Caddy create the record, satisfy the challenge, and delete it again.

Here is the distinction: The zone has to be real, public, and hosted somewhere your software can reach by API. The records inside it do not have to exist, and the addresses they point at can be entirely private. A zone that lives only in a local DNS server or a hosts file cannot satisfy DNS-01, and the error you get talks about finding zero zones rather than about anything you would recognise as the actual problem.

So far, identical to the certbot arrangement. The difference is everything downstream of the challenge. With certbot I own the schedule, the reload, and the alerting, and a failure in any of the three is quiet. With Caddy the renewal is a background job inside the process that is already serving the site, and a failure surfaces in the same logs I would look at for any other problem with that server. I did not replace a challenge type. I deleted a supply chain.

The Caddyfile says as much at the top, in the global options block:

1
2
3
4
5
6

{
	email {$ACME_EMAIL}
	# DNS-01 via Cloudflare: these names resolve only on the LAN (Pi-hole),
	# so there is no public path for an HTTP-01 challenge.
	acme_dns cloudflare {$CF_API_TOKEN}
}

The braces on their own at the start of a Caddyfile are the global options block, applying to every site in the file. Putting acme_dns there makes DNS-01 the default for everything, so no individual site needs a tls directive of its own. {$ACME_EMAIL} and {$CF_API_TOKEN} are Caddy’s environment variable syntax, and both come from the same .env the rest of the stack uses. That is Caddy expanding them when it loads the file, not Compose expanding them before the container starts, which matters because the file is mounted rather than templated.

One genuine difference in the outcome, worth knowing before you copy this. On nginx I issue a wildcard, so a new subdomain never involves the certificate at all. Caddy issues a separate certificate per name on demand, the first time a request arrives for a site in its config. Both get you to “add a service without a certificate dance”, by opposite routes. The wildcard means one certificate covers names that do not exist yet. Caddy means a name that appears in the config gets its own certificate without being asked. The Caddy version leaks slightly more information, since every name ends up in a public certificate transparency log individually rather than hiding behind an asterisk, and that is a real if small consideration for internal hostnames.

But authentik already has a certificate

It does, and this is worth addressing, because the first time you deploy authentik you get working HTTPS without doing anything and it is reasonable to ask what Caddy is adding.

On first start, authentik generates a self-signed certificate and serves TLS on port 9443 with it. You can look at it from another machine:

1
2

echo | openssl s_client -connect 192.168.66.6:9443 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

1
2
3
4

subject=O=authentik, CN=authentik default certificate
issuer=O=authentik, CN=authentik default certificate
notBefore=Sep  2 04:04:06 2026 GMT
notAfter=Sep  2 04:04:06 2027 GMT

Subject and issuer are the same string, which is what makes it self-signed: nothing vouches for it except itself. It sits in authentik’s certificate store alongside a second entry called the Internal JWT Certificate, which is a different thing for a different job. That one signs tokens and is managed by authentik automatically. Do not confuse them, and do not go looking for the JWT certificate in a browser.

The self-signed certificate cannot be the one your browser sees, for a reason that is about trust distribution rather than cryptography. The certificate is fine. Nothing trusts it. To make a browser accept it you would have to install it as trusted on every device that reaches any service, and “every device” on a home network means laptops, phones, a tablet somebody else owns, a television, and whatever appliance you forgot about. Some of those have no mechanism for installing a certificate at all.

The failure mode if you skip that is worse than an inconvenience. You get an interstitial warning on every service, every device, and you train yourself to click through it. That habit does not stay in your homelab. The entire value of a browser certificate warning is that it is rare enough to be alarming, and a setup that produces one daily has quietly disabled a security control everywhere else you browse.

There is also a category of client that cannot click through. Anything machine to machine, a mobile app, a dashboard polling an API, will simply refuse the connection and report something unhelpful.

So Caddy owns every certificate a browser or an API client will see, and it gets them from a certificate authority the world already trusts, for names that resolve only on my LAN, without anything having to reach my server. That is the whole job.

What the self-signed certificate is genuinely good for is the hop nothing human looks at. Where Caddy and authentik share a Compose network, Caddy talks to http://server:9000 in plain HTTP over the Docker bridge, that traffic never touches a network interface, and there is nothing to encrypt it against.

That stops being true the moment a proxy is on a different host, and one of mine is. That problem, and the thing I got wrong solving it the first time, is the next post. Two smaller notes belong here though. The proxy provider has an optional certificate field, which stays empty in my configuration, because it applies only to proxy mode where the outpost terminates TLS itself. And authentik has a certificate discovery directory, mounted in the first post as ./certs:/certs, which imports any certificate you drop into it. If you ever do need authentik holding a publicly trusted certificate, having Caddy write its issued pair into that directory is the supported route. Mine is empty, because nothing in this design needs it (yet).

Building a Caddy that can do it

Stock Caddy cannot talk to Cloudflare, and this surprised me. I had to refer to documentation.

Caddy is built as a set of modules compiled into a single binary. DNS provider integrations are modules, and the official Docker image ships with none of them. So acme_dns cloudflare in a stock container is not a feature you switch on, it is a configuration error at startup. You have to build a binary that contains the module.

The tool for that is xcaddy, which compiles Caddy from source with whatever additional modules you name. The official image publishes a -builder variant containing Go and xcaddy specifically so you can do this in a Dockerfile, which makes the whole thing six lines:

1
2
3
4
5
6
7

# Caddy with the Cloudflare DNS plugin, needed for DNS-01 issuance.
# These hostnames have no public A records, so HTTP-01 cannot work.
FROM caddy:2.11.4-builder AS builder
RUN xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:2.11.4
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

That is a multi-stage build. The first stage compiles, the second stage is the ordinary runtime image with the custom binary copied over the stock one. What ships is a normal Caddy container that happens to know about Cloudflare, without a Go toolchain riding along inside it.

Pin both stages to the same explicit version. A builder on one version and a runtime on another gives you a binary compiled against one Caddy sitting in an image built for a different one, and the resulting failure will be strange rather than informative. Confirm what you actually built:

1

docker exec authentik-caddy-1 caddy version

1

v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=

The Cloudflare token

Create it under My Profile -> API Tokens, using the Edit zone DNS template. Two things matter.

The permissions are Zone.Zone:Read and Zone.DNS:Edit. The module’s documentation is explicit that a single token carrying both is the current recommendation, and that the older arrangement of separate zone and DNS tokens is deprecated and may be removed. If you are copying an older guide that has you create two, you are following instructions that were correct once.

The zone resources should be scoped to the one zone, not all zones. This is the difference between a credential that can edit DNS for one homelab domain and one that can edit DNS for everything in your Cloudflare account, and Cloudflare’s token editor makes the narrow one no harder to create than the broad one. The token lives in .env on the host, mode 0600, alongside the other secrets from the first post.

The Compose service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

  caddy:
    build:
      context: ./caddy
    restart: unless-stopped
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    security_opt:
      - no-new-privileges:true
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      CF_API_TOKEN: ${CF_API_TOKEN:?cloudflare DNS token required for cert issuance}
      ACME_EMAIL: ${ACME_EMAIL:-admin@homelabdomain.xyz}
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data      # ACME certs live here - must persist
      - caddy_config:/config
    depends_on:
      server:
        condition: service_healthy

Several things there are load bearing.

The capability block. Linux capabilities are the pieces of root’s power, split up so a process can hold one without holding all of them. cap_drop: ALL removes every one, and cap_add: NET_BIND_SERVICE adds back exactly one: the ability to bind a port below 1024. That is the only privileged thing Caddy needs, because ports 80 and 443 are in the reserved range. no-new-privileges:true prevents any process inside the container from acquiring more privilege than it started with, which closes off the setuid escalation route.

This is the same reasoning as the worker in the first post, applied in the other direction. There I removed a privilege the upstream file granted by default. Here the privilege genuinely is needed, so instead of granting root wholesale, I granted the one capability and took the rest away. A container that needs to bind port 80 does not need to load kernel modules or change file ownership arbitrarily.

443:443/udp. That is HTTP/3, which runs over QUIC, which is UDP. Publishing 443:443 gives you TCP only. Omit the UDP line and everything still works, just without HTTP/3, and nothing warns you. It is the sort of thing you discover eighteen months later.

The read-only Caddyfile mount. The :ro suffix. There is no reason for the container to be able to rewrite its own configuration, so it cannot.

caddy_data. This is the one that will bite somebody. That volume holds the ACME account key and every certificate Caddy has been issued. Destroy it and Caddy registers a new account, re-issues everything from scratch, and starts spending against Let’s Encrypt’s rate limits, which are not generous enough to absorb repeated accidents. The comment is in my file specifically so that a future me running docker compose down -v while debugging something unrelated has one chance to notice.

The ports, compared to the first post. There I bound authentik’s 9000 and 9443 to a single LAN address because they are an internal interface that only my own proxies call. Caddy’s 80 and 443 are published on every interface deliberately, because Caddy is the front door and answering is its job.

Watching it work

The first docker compose up -d after adding Caddy produces logs that narrate the whole ACME exchange, which is unusually pleasant:

1
2
3
4
5
6
7
8

{"logger":"tls.obtain","msg":"acquiring lock","identifier":"omv.homelabdomain.xyz"}
{"logger":"tls.obtain","msg":"obtaining certificate","identifier":"omv.homelabdomain.xyz"}
{"logger":"http","msg":"creating new account because no account for configured email is known to us"}
{"logger":"http","msg":"new ACME account registered","status":"valid"}
{"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"omv.homelabdomain.xyz","challenge_type":"dns-01"}
{"logger":"http.acme_client","msg":"authorization finalized","authz_status":"valid"}
{"logger":"http.acme_client","msg":"validations succeeded; finalizing order"}
{"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"omv.homelabdomain.xyz"}

challenge_type is the line to look for. If it says http-01, your global acme_dns is not being applied, and you are about to watch a challenge fail against a name the internet cannot resolve.

The whole exchange took about ten seconds per name, which I did not expect. DNS-01 has a reputation for being slow because of propagation waits, and against Cloudflare’s API it simply is not.

What this actually bought

A front door that obtains and renews its own certificates, for names the public internet cannot resolve, with no timer, no deploy hook, and no alerting channel whose purpose is to notice when the other two stopped working.

The certbot arrangement on my other host still runs and I am not tearing it out. It works, and the alerting I built for it means I would now hear about a failure. But it is five components solving a problem that one component can solve by itself, and I would not build it that way again if I were starting today.

What is missing is anything in front of those certificates. Right now Caddy is a very well dressed reverse proxy that will hand any request straight through to the service behind it. Putting an authentication check in that path, in two lines per service rather than thirty, is the next post.

Get it

All three files are in my TechbyJeff repo, under Docker/authentik/:

  • docker-compose.yml — the whole stack, including the Caddy service block above. ACME_EMAIL is a required variable there rather than carrying my address as a default.
  • caddy/Dockerfile — the two-stage build, reproduced in full above.
  • caddy/Caddyfile — my working file with the domain changed. It runs ahead of this post: the global options block above is the top of it, and the rest is forward auth, which is the next post.

The .env holding the Cloudflare token is not in that repo and should not be in yours.

Sources

This is part two of a series on the identity stack in my homelab. Part one covered deploying authentik itself. Part three wires forward auth through this proxy, and fixes a cross-host hop I had wrong. Part four covers Cloudflare, and what changes when the identity provider becomes the one internet-facing thing you own. Part five covers Tautulli, which has its own login it does not want to give up. Part six covers OpenMediaVault, which has no delegated authentication mode at all.