The last post gave me a reverse proxy that manages its own certificates and hands every request straight through to the service behind it. This post puts an authentication check in that path.
The mechanism is forward auth, defined at length in the first post: the proxy stops mid-request, asks a separate service whether this one is allowed, and either forwards it with identity headers attached or sends the browser off to log in. The service answering is authentik’s outpost, running inside the authentik server container in my case, for the reasons that post gives.
nginx does forward auth with a module called auth_request, and the pattern authentik documents for it is correct, well tested, and thirty lines long. One service cost this much in the vhost I had in front of my storage box:
| |
Nothing there is wrong. It’s thirty lines before a single byte gets proxied, and it appears once per service, in full, with the upstream address as the only thing that differs between copies. Nine services means nine copies.
That’s fine while you’re pasting, and the obvious objection is that it stays fine afterwards, because an editor changes nine identical blocks about as easily as one. That’s true, and the fix at the end of this post is that kind of change: nine identical blocks, one uniform edit, no cleverness required.
The cost isn’t the typing. It’s that a bulk edit is a change you have to verify, where a single source of truth is a change you can read. When the behavior lives in one place, a one-line diff tells you what is now in effect everywhere. When it lives in nine, correctness is a claim about nine files, and the only way to hold that claim is to go and count. That’s why the last section of this post ends with a command that prints the running configuration.
Substitution also only covers changes that are uniform. It matches nothing in the copy that somebody tweaked for one service eight months ago and that no longer looks quite like its siblings, so it silently fixes eight of nine and reports success. And it doesn’t help at all when the change is structural, when a block has to move relative to another block, or be wrapped in something, or stop existing.
Then the copies outlive the edit. When I went looking on my own host afterwards, the old cleartext endpoint still appeared fifteen times: in backup files, and in two vhosts nginx no longer loads. None of it was in effect, all of it was in the search space, and every one of those hits is something you have to look at and rule out before you can say you are done.
That doesn’t make nine copies unworkable. It makes them a standing tax you pay on every change, in verification, forever. Two lines per service does not.
Look at the first line of that block again. It is http://, it crosses a physical network, and the request it makes carries the user’s session cookie. I had been running it that way for a while without noticing, and what made me notice was writing it down for you. More on that at the end.
The fix is the second half of this post. I left the wrong version in place above, alongside the corrected one, because the wrong version is what the documentation gives you and what you will most likely have.
One snippet, every service
The Caddy equivalent:
| |
And a complete site that uses it:
Two lines.
A snippet in Caddy is a named block written in parentheses that does nothing on its own and gets pulled into a site with import. Snippets take positional arguments, referenced as {args[0]}, {args[1]} and so on. The upstream address is the only thing that differs between my services, so it’s the only argument. Adding a service is one site block. Changing how forward auth behaves everywhere is one edit in one place. The nginx arrangement couldn’t give me that.
Restricting by source address
The @denied line defines a named matcher: Caddy’s way of naming a condition you can reuse. This one matches any request arriving from outside loopback, my LAN, the Tailscale range and Docker’s private space. 100.64.0.0/10 is the carrier-grade NAT block Tailscale allocates from, so this covers me when I’m away from home over the VPN. Anything matching gets a 403 and never reaches the rest.
This is defense in depth. authentik is already deciding who gets in. But an allowlist means that if I misconfigure a flow, or authentik has a bad afternoon, the failure mode is still “unreachable from outside”.
Where Caddy reorders your directives
Those two lines sit inside the route block, and for a long time they sat outside it. That was wrong. The reasoning that put them there sounds airtight.
Outside the block, Caddy’s own directive ordering governs when they run, not the order I wrote them in. I knew that, and I checked the part I thought mattered: respond does sort ahead of reverse_proxy, so I concluded the deny would be evaluated first. The premise is true. The conclusion doesn’t follow from it, because outside the block the deny isn’t competing with reverse_proxy at all. It’s competing with route, and route sorts ahead of respond.
The adapted configuration came out as the route block first and the deny second. Since reverse_proxy {args[0]} inside that route answers the request, the deny was never reached. The allowlist was dead. Every service importing the snippet carried a 403 in its configuration that couldn’t fire, and nothing said so: the config parsed, the services worked, and the only thing missing was the layer I had put there in case the other one failed.
Moving the two lines inside route fixes it. Written order is preserved, the deny is evaluated first, and a request from outside the allowlist gets its 403.
I hit that same edge again later in this file, in a worse form. A bare respond 404 written after a reverse_proxy, intended as a catch-all for unmatched paths, doesn’t run last. It sorts ahead of the proxy and returns 404 for everything, including the paths you meant to proxy. The file reads correctly top to bottom and the behavior is the opposite of what it says.
The fix is to wrap the block in route { }. That suspends the reordering and executes in written order. That’s also why route is there in the snippet above, and for the same reason: the outpost path has to be proxied straight through before anything tries to authenticate it. Without route, the endpoints that issue your session can end up behind the authentication check that depends on them, and the symptom is a redirect loop that looks like an authentik fault and is not.
Never reason about a Caddyfile’s control flow from the order of lines in the file. Run caddy adapt --pretty against it to print the JSON configuration Caddy executes, and read the order out of that instead.
Passing the outpost path through
| |
Everything under that path goes to authentik unauthenticated. This is where the sign-in redirect lands, where the callback returns, and where the browser collects its session cookie. It has to be reachable without a session, because it creates the session.
Look at the upstream: http://server:9000. The Compose service name. Caddy and authentik are in the same stack, so Docker’s embedded DNS resolves it and the traffic never leaves the bridge network. This is also why binding authentik’s published ports to a single LAN address in the first post didn’t affect Caddy at all. Caddy isn’t using the published port. It’s talking to the container.
forward_auth
Caddy’s forward_auth makes a subrequest to the given address at the given path and decides based on the response. A 2xx means allow, and the original request continues to the real upstream. A 401 or a redirect goes back to the client, and that’s how the browser ends up at a login page.
It’s the same idea as the nginx block at the top of this post: subrequest, decide, redirect on failure, carry a header through on success. But there, auth_request, the error_page 401, the internal @signin location and the hand-managed Set-Cookie juggling are four separate pieces you assemble yourself, once per service. Caddy folds the pattern into one directive, small enough to put in a snippet.
The uri ends in /auth/caddy and that suffix carries meaning. authentik exposes a different endpoint per proxy flavor: auth/nginx, auth/caddy, auth/traefik and others, because each proxy wants the answer shaped differently. Point Caddy at the nginx endpoint and you get a response it doesn’t know how to act on.
Trimming the header list
| |
By default forward_auth discards the auth response’s headers. copy_headers names the ones to carry onto the upstream request, and this list is the actual identity handoff: username, group membership, email, display name, and the stable user ID. An application that supports delegated authentication reads one of these and logs the user in without asking anything. An application that doesn’t will ignore them entirely, which is what the last two posts in this series are about.
authentik’s documented example copies a considerably longer list, adding entitlements, a JWT, and a set of X-Authentik-Meta-* headers describing the outpost, provider and application. I trimmed it to five because those five are the only ones anything downstream of me reads. Every header in that list is an assertion you are making to the upstream on the user’s behalf, and a header the upstream trusts is a header you have to be certain a client can’t set for themselves. Copying fewer is asserting less.
One warning from the upstream documentation that is easy to skip past: those header names are case-sensitive. Get it wrong and they arrive empty, with nothing in the logs to say so. Copy the names exactly.
trusted_proxies
| |
This tells Caddy whose X-Forwarded-For header to believe. private_ranges is a built-in shorthand for the RFC 1918 space plus loopback.
Without it, a client could assert its own forwarded address, and authentik would log and evaluate policy against a value the client chose. This is a short line with a large blast radius, and if I were reviewing somebody else’s forward auth config it is the first thing I would look for. The upstream example notes that scoping it to the outpost’s specific address is stricter still, and if your outpost has a stable address, do that.
Nine providers, one per application
The Caddy half only works because of how the providers are configured, and this is the part I got wrong first and had to redo.
There are nine applications in authentik, each with its own proxy provider, each in forward_single mode. Read out of the database:
| |
Proxy providers have three modes. Proxy mode makes authentik itself the reverse proxy, forwarding to an internal host. Forward single does forward auth for one application on its own domain. Forward domain does forward auth for every application under a parent domain, from a single provider, with a single cookie.
I built this with domain mode first, because it’s the arrangement that makes site blocks shortest: one provider, one cookie scoped to the parent domain, nothing to configure per service. Then I tried to write down what you would do if you wanted one person to reach one service and not another, and could not.
authentik’s documentation is blunt about it. Of domain mode, it says “you cannot restrict individual applications to different users with separate application-level policies”, and recommends single-application mode when each application needs separate access rules. The reason is structural rather than a missing feature. Policies bind to application objects, and in domain mode there is no application object per service to bind them to. There’s one provider covering a whole domain. The escape hatch I assumed existed does not.
There are nine now. The visible proof that they are separate authorization objects is that each yields a distinct client_id: nine providers, nine client IDs, no sharing. Cookie domain is empty on all of them, because in single mode each provider scopes its own.
What I expected to lose and didn’t is single sign-on. The authentik session is held at the identity provider, so reaching a second service redirects through authentik and comes straight back without a prompt. You get one login and nine independently governable applications: the combination I assumed I had to choose between.
Note where this currently stands. internal_host is empty on all nine, which is correct, because that field applies only to proxy mode where authentik does the proxying itself. And the applications currently have zero policy bindings, so any authentik user still reaches all nine. The structure is in place and the policy is not. That’s a real distinction and I would rather state it than imply this setup is doing access control it is not yet doing. The difference from an hour earlier is that filling it in is now possible.
One structural detail I found interesting while reading the schema: the proxy provider table’s primary key is a foreign key to the OAuth2 provider table. Proxy providers are built on top of authentik’s OAuth2 implementation. That’s where those client IDs come from, and why the setup asks you to choose an authorization flow even though nobody sees a consent screen.
A session cookie on a wire
Back to that http://192.168.66.6:9000 in the nginx block.
The services that live next to my application host’s nginx are still proxied by it, and there’s no reason to route them across hosts to reach a different proxy. But their forward auth subrequests have to reach the outpost on the other machine. Those subrequests carry the user’s live session cookie, so they’re as sensitive as the session itself, and they were crossing a physical network in cleartext.
No documentation states it: the transport question for forward auth is decided by where your proxy sits. Same host, over loopback or a container bridge, plaintext is fine and there’s nothing to encrypt against. Different host, it crosses a wire, and a cookie on a wire needs both encryption and a verified peer.
My first instinct for fixing it was also wrong, in the obvious way, so I will name it. You point the cross-host proxy at authentik’s TLS port, and because that certificate is self-signed and will never chain to a public root, you turn verification off. Don’t do that. Verification off means you encrypt the connection without ever establishing who you encrypted it to, which stops a passive observer and does nothing at all about an active one. It’s worse than plaintext in one specific way: plaintext looks insecure and gets treated accordingly, while unverified TLS looks fine in a config review.
The better answer uses the machinery from the previous post. Caddy is already obtaining publicly trusted certificates for names that resolve nowhere on the public internet, so let it hold certificates for the application hostnames too and serve only the outpost path on them:
| |
The route wrapper is the bug from earlier in this post, recorded where I hit it. Without it the trailing respond 404 sorts ahead of reverse_proxy and the block returns 404 for everything.
How one address serves seven names
The rest of this depends on a piece of TLS vocabulary that is easy to skip past.
SNI stands for Server Name Indication. It exists because of an ordering problem: TLS happens before HTTP. The server has to choose a certificate and send it during the handshake, before it has seen a single HTTP header, Host included. Without help, a server could only ever present one certificate per address and port, which is why hosting several HTTPS sites on one IP address used to mean one certificate covering all of them, or one address per site.
SNI is the fix. The client states, in the clear, at the very start of the handshake, which hostname it is trying to reach. The server uses that to pick the matching certificate, and the connection proceeds normally from there.
So Caddy can hold seven differently named certificates on one address and port and hand out the right one, because every client announces which one it wants before anything else happens.
The property that makes this design work, though, is that the hostname a client sends in SNI has nothing to do with how it found the address. nginx connects to a bare IP address, announces sonarr.homelabdomain.xyz in the handshake, receives the certificate for that name, and verifies it against that same name. At no point does anything look the name up. It’s a label carried inside the connection.
That’s the same trick as the previous post’s certificates, seen from the other end. There, a name got a real certificate without needing a public address record. Here, a client uses that name without needing any address record at all.
Each mode reads a different field
That comment about Host cost me an outage to learn.
My first version of this endpoint used one shared hostname for all seven services and rewrote Host to it. That worked, and I had checked that it worked, and I was wrong about why. Under forward_domain, the outpost resolves which application is being requested from X-Original-URL, so Host is ignored and you can point it anywhere. Under forward_single, the outpost resolves the application from the Host header, matched against the provider’s external_host.
The moment I moved to per-application providers, every service returned 404. Not a subtle degradation, an immediate and total one, from a configuration change three sections away in a different file on a different machine.
The fix is what you see above: Caddy matches the seven real hostnames, and nginx preserves Host. Nothing needs a DNS record pointing at the auth host, because nginx connects by IP and uses the name only for SNI and verification.
A proxy provider’s mode changes which request field is authoritative. Verifying that a header override is safe under one mode tells you nothing about the other.
Rewriting the nginx block
| |
proxy_ssl_name $host is what makes one location block serve seven hostnames: SNI and certificate verification both follow the request’s own host.
proxy_ssl_session_reuse has to be off when proxy_ssl_name is a variable. TLS session cache entries are keyed per upstream, not per SNI name. With reuse on, the session negotiated for the first hostname gets offered for the next one, and the handshake fails.
The diagnostic signature is what told me where to look: which services failed changed between reloads. Nothing in the configuration was per-host, so per-host symptoms that move aren’t a per-host problem. Shifting failures across identical configuration mean shared state, and shared state in a TLS path means a cache.
Check that your error log exists before you debug anything. Mine had error_log /dev/null; in it, inherited from some past cleanup. The proxy sitting in front of my entire authentication path was discarding every error it produced. My first two diagnostic passes came back with nothing and I was reduced to guessing, which is the worst possible state to be in while changing an auth configuration.
It now logs at warn to a real file. That’s what let me stop guessing and start reading.
What this endpoint exposes
I checked each of these.
No DNS record is needed. As above, nginx reaches the auth host by IP and uses the hostname only for SNI and certificate verification. That keeps the authentication hot path free of any dependency on my DNS server, a good property for the component everything else depends on.
The names expose one path and nothing else. The @outpost matcher proxies /outpost.goauthentik.io/* and the trailing respond 404 catches everything else, so the admin interface and every other authentik endpoint are unreachable through these names on this listener. The source address allowlist applies on top of that.
Verification failures are loud. Because proxy_ssl_verify is on, a certificate that failed to validate surfaces as a 502. Services returning 302 to the login page is therefore positive evidence that verification is working.
Cleaning up after a migration
For the storage box there was a period where both proxies were configured. nginx on the application host had a vhost for it, Caddy on the authentik host had a site block for it, and only whichever one DNS pointed at was serving. That was an artifact of migrating. The nginx symlinks are now removed from sites-enabled, with the files left in sites-available in case I want to read them later, and Caddy is unambiguously the path.
Resolve that state quickly. Two proxies configured for one hostname works perfectly until somebody changes a DNS record, at which point the service starts answering with a different set of headers, allowlists and timeouts than it did yesterday. It’s the same shape as the firewall problem in the first post: two things are both true, only one of them is in effect, and no part of the system mentions it.
A related habit. When you think you have removed something, check what the software loads; the directory contents can say otherwise. grep across /etc/nginx still finds the old cleartext endpoint in my backup files and in the two dormant sites-available copies. nginx -T, which prints the fully assembled running configuration, finds zero. The second number describes reality, and they disagree by fifteen.
What is still missing
Every request now arrives over a certificate the client trusts, gets checked against a real identity provider before it reaches an application, and carries a verified identity in headers when it gets there. The subrequest that makes that decision no longer crosses a wire in the clear.
That last clause is only true because I wrote this down. The configuration had been working perfectly throughout, which is the problem: nothing about it was going to fail in a way that got my attention.
This is rubber duck debugging with the patience taken out. Explaining a thing forces you to serialize it, and serializing it means you are no longer allowed to skim. Reading proxy_pass http://192.168.66.6:9000/outpost.goauthentik.io for the hundredth time, I saw an endpoint, because the endpoint is the part I care about when something is broken. Writing the sentence that explains it to you, I had to say what that request carried, and the scheme at the front stopped being punctuation. A duck will sit through any amount of hand-waving. A reader will not, and you know that while you’re typing.
Two things are conspicuously absent. Nothing here says how the login page itself is reachable, which has consequences I didn’t anticipate and is the whole of the next post. And nothing here shows that the applications behind all this care about the headers they are being handed.
They vary enormously. A few read them and log the user straight in. Others have their own network-based bypass and ignore them entirely. The awkward ones have no delegated authentication mode of any kind, so the user authenticates to authentik, arrives at the application, and is asked to log in again by software that never learned the first login happened.
Get it
The Caddy side is in my TechbyJeff repo under Docker/authentik/:
caddy/Caddyfilecontains everything above: the(authentik)snippet, a site block that imports it, and the per-application TLS endpoint. It’s my working file with the domain changed and the header comment lists what you need to edit.docker-compose.ymlis the stack the snippet’shttp://server:9000upstream resolves against.
The nginx blocks are reproduced in full above, because they’re excerpts from a vhost file specific to my services. Both the Caddy matchers and the nginx proxy_pass need your own addresses before any of it will do anything for you.
The .env holding the Cloudflare token isn’t in that repo and shouldn’t be in yours.
Sources
- Caddy: the
forward_authdirective - Caddy: Caddyfile concepts, snippets and imports
- Caddy: the
routedirective and directive order - authentik: Caddy configuration for proxy providers
- authentik: nginx configuration for proxy providers
- authentik: proxy providers and forward auth modes
- nginx:
ngx_http_auth_request_module - nginx:
proxy_ssl_verifyand related directives
Part three of a series on the identity stack in my homelab. Before this: deploying authentik and certificates. After it: putting the identity provider on the internet, then Tautulli, OpenMediaVault and Proxmox over OIDC.