The last post gave me a reverse proxy that manages its own certificates and hands every request straight through to the service behind it. This post puts an authentication check in that path.
The mechanism is called forward auth, and the short definition is this: before passing a request to your application, the proxy pauses and asks a separate service, out of band, whether this request is allowed. If the answer is yes it forwards the request, usually adding headers describing who the user is. If the answer is no it sends the browser off to log in. The application is never modified and usually never learns any of it happened. The service answering the question is authentik’s outpost, which in my case runs inside the authentik server container rather than as a separate deployment. The first post covers what an outpost is at more length, and why the embedded one is the right choice for a single host.
nginx does forward auth with a module called auth_request, and the pattern authentik documents for it is correct, well tested, and thirty lines long. Here is what one service cost in the vhost I had in front of my storage box:
| |
Nothing there is wrong. It is thirty lines before a single byte gets proxied, and it appears once per service, in full, with the upstream address as the only thing that differs between copies. Nine services means nine copies.
That is fine while you are pasting, and the obvious objection is that it stays fine afterwards, because an editor changes nine identical blocks about as easily as one. That is true and worth conceding up front, because the fix at the end of this post is exactly that kind of change: nine identical blocks, one uniform edit, no cleverness required.
The cost is not the typing. It is that a bulk edit is a change you have to verify, where a single source of truth is a change you can read. When the behaviour lives in one place, a one-line diff tells you what is now in effect everywhere. When it lives in nine, correctness is a claim about nine files, and the only honest way to hold that claim is to go and count. That is why the last section of this post ends with a command that prints the running configuration rather than with the word “done”.
Substitution also only covers changes that are uniform. It matches nothing in the copy that somebody tweaked for one service eight months ago and that no longer looks quite like its siblings, so it silently fixes eight of nine and reports success. And it does not help at all when the change is structural rather than textual, when a block has to move relative to another block, or be wrapped in something, or stop existing.
Then the copies outlive the edit. When I went looking on my own host afterwards, the old cleartext endpoint still appeared fifteen times: in backup files, and in two vhosts nginx no longer loads. None of it was in effect. All of it was in the search space, and every one of those hits is a thing you have to look at and rule out before you can say you are done.
None of that makes nine copies unworkable. It makes them a standing tax you pay on every change, forever, in verification rather than in typing. Two lines per service does not.
That is not a hypothetical, and it is the reason this post has the shape it does. Look at the first line of that block again. It is http://. It crosses a physical network. And the request it makes carries the user’s session cookie.
I had been running it that way for a while without noticing. What made me notice was writing it down for you.
Which is worth saying out loud, because it is the best argument I have for publishing this kind of thing rather than keeping it in a private runbook. Configuration you wrote yourself is close to invisible to you. You know what you meant, so you read what you meant, and a line goes on meaning that forever.
If that sounds familiar, it is rubber duck debugging wearing a different hat. The idea has been in the folklore since The Pragmatic Programmer put it there in 1999: you are stuck, so you explain the problem out loud, line by line, to a rubber duck sitting on your desk, and somewhere in the explanation you hear the mistake come out of your own mouth. The duck contributes nothing. It cannot. The entire mechanism is that explaining something forces you to serialise it, and serialising it means you are no longer allowed to skim.
That is precisely what happened with this line. Reading proxy_pass http://192.168.66.6:9000/outpost.goauthentik.io for the hundredth time, I saw an endpoint, because the endpoint is the part I care about when something is broken. Writing the sentence that explains it to you, I had to say what that request was and what it carried, and at that point the scheme at the front stopped being punctuation.
Writing for publication is rubber ducking with the patience taken out. A duck will sit through any amount of hand-waving. A reader will not, and you know that while you are typing, so you pre-empt the objection instead of skating past it. A line you cannot justify stands out in a way it never does in your own editor, and I have caught more of my own mistakes writing posts than reviewing configs.
There is a second half to that, and it is the part that actually raises the bar. Everything I publish here, somebody copies. Accepting a shortcut on a network I control is one thing. Writing it up as a pattern, with a session cookie crossing a wire in the clear, hands it to everyone who follows along and trusts that I thought about it. That is a different standard, and it is a good one to be held to.
So the fix is the second half of this post, and I have left the wrong version in place above rather than quietly publishing the corrected one, because the wrong version is what the documentation gives you and what you will most likely have.
One snippet, every service
Here is the Caddy equivalent:
| |
And a complete site that uses it:
Two lines. That is the point of the exercise.
A snippet in Caddy is a named block written in parentheses that does nothing on its own and gets pulled into a site with import. Snippets take positional arguments, referenced as {args[0]}, {args[1]} and so on. The upstream address is the only thing that differs between my services, so it is the only argument. Adding a service is one site block. Changing how forward auth behaves everywhere is one edit in one place, which is exactly what the nginx arrangement could not give me.
The network allowlist
The @denied line defines a named matcher, which is Caddy’s way of naming a condition you can reuse. This one matches any request whose source address is not loopback, not my LAN, not the Tailscale range, and not Docker’s private space. 100.64.0.0/10 is the carrier-grade NAT block Tailscale allocates from, so this covers me when I am away from home over the VPN. Anything matching gets a 403 and never reaches the rest.
This is defence in depth rather than the primary control. authentik is already deciding who gets in. But an allowlist means that if I misconfigure a flow, or authentik has a bad afternoon, the failure mode is still “unreachable from outside” rather than “open to whoever finds it”.
Directive order is not file order
Those two lines sit inside the route block, and for a long time they sat outside it. That was wrong, and it is worth showing why, because the reasoning that put them there is the kind that sounds airtight.
Outside the block, Caddy’s own directive ordering governs when they run rather than the order I wrote them in. I knew that, and I checked the part I thought mattered: respond does sort ahead of reverse_proxy, so I concluded the deny would be evaluated first. The premise is true. The conclusion does not follow from it, because outside the block the deny is not competing with reverse_proxy at all. It is competing with route — and route sorts ahead of respond.
So the adapted configuration came out as the route block first and the deny second. Since reverse_proxy {args[0]} inside that route answers the request, the deny was never reached. The allowlist was dead. Every service importing the snippet carried a 403 in its configuration that could not fire, and nothing said so: the config parsed, the services worked, and the only thing missing was the layer I had put there in case the other one failed.
Moving the two lines inside route fixes it. Written order is preserved, the deny is evaluated first, and a request from outside the allowlist gets its 403 instead of a page.
It is worth knowing exactly how sharp that edge is, because I found out the hard way later in this same file. A bare respond 404 written after a reverse_proxy, intended as a catch-all for unmatched paths, does not run last. It sorts ahead of the proxy and returns 404 for everything, including the paths you meant to proxy. The file reads correctly top to bottom and the behaviour is the opposite of what it says.
The fix is to wrap the block in route { }, which suspends the reordering and executes in written order. That is also why route is there in the snippet above, and the reason matters: the outpost path has to be proxied straight through before anything tries to authenticate it. Without route, the endpoints that issue your session can end up behind the authentication check that depends on them, and the symptom is a redirect loop that looks like an authentik fault and is not.
So never reason about a Caddyfile’s control flow from the order of lines in the file. Run caddy adapt --pretty against it, which prints the JSON configuration Caddy actually executes, and read the order out of that. It takes a second and it is the only source of truth.
The outpost passthrough
| |
Everything under that path goes to authentik unauthenticated. This is where the sign-in redirect lands, where the callback returns, and where the browser collects its session cookie. It has to be reachable without a session, because it is the thing that creates the session.
Look at the upstream: http://server:9000. Not an IP address, not a hostname, the Compose service name. Caddy and authentik are in the same stack, so Docker’s embedded DNS resolves it and the traffic never leaves the bridge network. This is also why binding authentik’s published ports to a single LAN address in the first post did not affect Caddy at all. Caddy is not using the published port. It is talking to the container.
forward_auth
Caddy’s forward_auth makes a subrequest to the given address at the given path and decides based on the response. A 2xx means allow, and the original request continues to the real upstream. A 401 or a redirect is returned to the client, which is how the browser ends up at a login page.
It is the same idea as the nginx block at the top of this post: subrequest, decide, redirect on failure, carry a header through on success. But there, auth_request, the error_page 401, the internal @signin location and the hand-managed Set-Cookie juggling are four separate pieces you assemble yourself, once per service. Caddy folds the entire pattern into one directive, which is what makes it small enough to be worth putting in a snippet at all.
The uri ends in /auth/caddy and that suffix is not decoration. authentik exposes a different endpoint per proxy flavour: auth/nginx, auth/caddy, auth/traefik and others, because each proxy wants the answer shaped differently. Point Caddy at the nginx endpoint and you get a response it does not know how to act on.
The headers, and why mine is a short list
| |
By default forward_auth discards the auth response’s headers. copy_headers names the ones to carry onto the upstream request, and this list is the actual identity handoff: username, group membership, email, display name, and the stable user ID. An application that supports delegated authentication reads one of these and logs the user in without asking anything. An application that does not will ignore them entirely, which is what the last two posts in this series are about.
authentik’s documented example copies a considerably longer list, adding entitlements, a JWT, and a set of X-Authentik-Meta-* headers describing the outpost, provider and application. I trimmed it to five because those five are the only ones anything downstream of me reads. Every header in that list is an assertion you are making to the upstream on the user’s behalf, and a header the upstream trusts is a header you have to be certain a client cannot set for themselves. Copying fewer is not cargo-culting less carefully, it is asserting less.
One warning from the upstream documentation that is easy to skip past: the capitalisation of those header names matters. Get it wrong and they arrive empty rather than failing loudly, which is a debugging session you can skip by copying the names exactly.
trusted_proxies
| |
This tells Caddy whose X-Forwarded-For header to believe. private_ranges is a built-in shorthand for the RFC 1918 space plus loopback.
Without it, a client could assert its own forwarded address, and authentik would log and evaluate policy against a value the client chose. This is a short line with a large blast radius, and if I were reviewing somebody else’s forward auth config it is the first thing I would look for. The upstream example notes that scoping it to the outpost’s specific address is stricter still, which is worth doing if your outpost has a stable address.
The authentik side: one provider per application
The Caddy half only works because of how the providers are configured, and this is the part I got wrong first and had to redo.
There are nine applications in authentik, each with its own proxy provider, each in forward_single mode. Read out of the database:
| |
Proxy providers have three modes. Proxy mode makes authentik itself the reverse proxy, forwarding to an internal host. Forward single does forward auth for one application on its own domain. Forward domain does forward auth for every application under a parent domain, from a single provider, with a single cookie.
I built this with domain mode first, because it is the arrangement that makes site blocks shortest: one provider, one cookie scoped to the parent domain, nothing to configure per service. Then I tried to write down what you would do if you wanted one person to reach one service and not another, and could not.
authentik’s documentation is blunt about it. Of domain mode, it says “you cannot restrict individual applications to different users with separate application-level policies”, and recommends single-application mode when each application needs separate access rules. The reason is structural rather than a missing feature. Policies bind to application objects, and in domain mode there is no application object per service to bind them to. There is one provider covering a whole domain. The escape hatch I assumed existed does not.
So there are nine now. The visible proof that they are genuinely separate authorization objects is that each yields a distinct client_id: nine providers, nine client IDs, no sharing. Cookie domain is empty on all of them, because in single mode each provider scopes its own.
What I expected to lose and did not is single sign-on. The authentik session is held at the identity provider, not by the individual providers, so reaching a second service redirects through authentik and comes straight back without a prompt. You get one login and nine independently governable applications, which is the combination I assumed I had to choose between.
Two honest notes on the current state. internal_host is empty on all nine, which is correct rather than an oversight, because that field applies only to proxy mode where authentik does the proxying itself. And the applications currently have zero policy bindings, so any authentik user still reaches all nine. The structure is in place and the policy is not. That is a real distinction and I would rather state it than imply this setup is doing access control it is not yet doing. The difference from an hour earlier is that filling it in is now possible.
One structural detail I found interesting while reading the schema: the proxy provider table’s primary key is a foreign key to the OAuth2 provider table. Proxy providers are built on top of authentik’s OAuth2 implementation rather than being a parallel thing, which is where those client IDs come from and why the setup asks you to choose an authorisation flow even though nobody sees a consent screen.
The hop I had wrong
Back to that http://192.168.66.6:9000 in the nginx block.
The services that live next to my application host’s nginx are still proxied by it, and there is no reason to route them across hosts to reach a different proxy. But their forward auth subrequests have to reach the outpost, which is on the other machine. Those subrequests carry the user’s live session cookie, which makes them as sensitive as the session itself, and they were crossing a physical network in cleartext.
That is worth stating as a general rule, because it is not obvious from any documentation: the transport question for forward auth is decided by where your proxy sits. Same host, over loopback or a container bridge, plaintext is fine and there is nothing to encrypt against. Different host, it crosses a wire, and a cookie on a wire needs both encryption and a verified peer.
My first instinct for fixing it was wrong too, and it is the obvious wrong answer, so it is worth naming. You point the cross-host proxy at authentik’s TLS port, and because that certificate is self-signed and will never chain to a public root, you turn verification off. Do not do that. Verification off means you encrypt the connection without ever establishing who you encrypted it to, which stops a passive observer and does nothing at all about an active one. It is worse than plaintext in one specific way: plaintext looks insecure and gets treated accordingly, while unverified TLS looks fine in a config review.
The better answer uses the machinery from the previous post. Caddy is already obtaining publicly trusted certificates for names that resolve nowhere on the public internet, so let it hold certificates for the application hostnames too and serve only the outpost path on them:
| |
The route wrapper is the bug from earlier in this post, recorded where I hit it. Without it the trailing respond 404 sorts ahead of reverse_proxy and the block returns 404 for everything.
What SNI is, and why one address serves seven names
The rest of this depends on a piece of TLS vocabulary that is easy to skip past, so here it is properly.
SNI stands for Server Name Indication. It exists because of an ordering problem: TLS happens before HTTP. The server has to choose a certificate and send it during the handshake, before it has seen a single HTTP header, which means before it has seen Host. Without help, a server could only ever present one certificate per address and port, which is why hosting several HTTPS sites on one IP address used to mean one certificate covering all of them, or one address per site.
SNI is the fix. The client states, in the clear, at the very start of the handshake, which hostname it is trying to reach. The server uses that to pick the matching certificate, and the connection proceeds normally from there.
Two consequences matter for this endpoint.
The first is that Caddy can hold seven differently named certificates on one address and port, and hand out the right one, because every client announces which one it wants before anything else happens.
The second is the useful one, and it is why this design works at all: the hostname a client sends in SNI has nothing to do with how it found the address. nginx connects to a bare IP address, announces sonarr.homelabdomain.xyz in the handshake, receives the certificate for that name, and verifies it against that same name. At no point does anything look the name up. It is a label carried inside the connection, not an instruction about where to connect.
That is the same trick as the previous post’s certificates, seen from the other end. There, a name got a real certificate without needing a public address record. Here, a client uses that name without needing any address record at all.
The two modes read different fields
That comment about Host is the most useful thing in this post and it cost me an outage to learn.
My first version of this endpoint used one shared hostname for all seven services and rewrote Host to it. That worked, and I had checked that it worked, and I was wrong about why. Under forward_domain, the outpost resolves which application is being requested from X-Original-URL, so Host genuinely does not matter and you can point it anywhere. Under forward_single, the outpost resolves the application from the Host header, matched against the provider’s external_host.
So the moment I moved to per-application providers, every service returned 404. Not a subtle degradation, an immediate and total one, from a configuration change three sections away in a different file on a different machine.
The fix is what you see above: Caddy matches the seven real hostnames rather than one neutral one, and nginx preserves Host instead of overriding it. Nothing needs a DNS record pointing at the auth host, because nginx connects by IP and uses the name only for SNI and verification.
If you take one operational lesson from this: a proxy provider’s mode changes which request field is authoritative. Verifying that a header override is safe under one mode tells you nothing about the other.
The nginx side, and two things that bit me
| |
proxy_ssl_name $host is what makes one location block serve seven hostnames: SNI and certificate verification both follow the request’s own host.
proxy_ssl_session_reuse has to be off when proxy_ssl_name is a variable. TLS session cache entries are keyed per upstream, not per SNI name. With reuse on, the session negotiated for the first hostname gets offered for the next one, and the handshake fails.
The diagnostic signature is worth memorising, because it is what told me where to look: which services failed changed between reloads. Nothing in the configuration was per-host, so per-host symptoms that move are not a per-host problem. Shifting failures across identical configuration mean shared state, and shared state in a TLS path means a cache.
Check that your error log exists before you debug anything. Mine had error_log /dev/null; in it, inherited from some past cleanup. The proxy sitting in front of my entire authentication path was discarding every error it produced. My first two diagnostic passes came back with nothing and I was reduced to guessing, which is the worst possible state to be in while changing an auth configuration.
It now logs at warn to a real file. Of the three fixes in this section that is arguably the most valuable, because it is the one that let me stop guessing and start reading.
What this endpoint does and does not expose
Three properties, each checked rather than assumed.
No DNS record is needed. As above, nginx reaches the auth host by IP and uses the hostname only for SNI and certificate verification. That keeps the authentication hot path free of any dependency on my DNS server, which is a genuinely good property for the component everything else depends on.
The names expose one path and nothing else. The @outpost matcher proxies /outpost.goauthentik.io/* and the trailing respond 404 catches everything else, so the admin interface and every other authentik endpoint are unreachable through these names on this listener. The source address allowlist applies on top of that.
Verification failures are loud. Because proxy_ssl_verify is on, a certificate that failed to validate surfaces as a 502 rather than quietly falling back. Services returning 302 to the login page is therefore positive evidence that verification is working, not just that the config parsed.
Cleaning up after a migration
For the storage box there was a period where both proxies were configured. nginx on the application host had a vhost for it, Caddy on the authentik host had a site block for it, and only whichever one DNS pointed at was actually serving. That was an artifact of migrating rather than a decision. The nginx symlinks are now removed from sites-enabled, with the files left in sites-available in case I want to read them later, and Caddy is unambiguously the path.
Resolve that state quickly rather than living with it. Two proxies configured for one hostname works perfectly until somebody changes a DNS record, at which point the service starts answering with a different set of headers, allowlists and timeouts than it did yesterday, with no error anywhere to explain why. Nothing is broken, so nothing tells you. It is the same shape as the firewall problem in the first post: two things are both true, only one of them is in effect, and no part of the system considers that worth mentioning.
A related habit worth stealing. When you think you have removed something, check what the software loads rather than what the directory contains. grep across /etc/nginx still finds the old cleartext endpoint in my backup files and in the two dormant sites-available copies. nginx -T, which prints the fully assembled running configuration, finds zero. The second number is the one that describes reality, and they disagree by fifteen.
What is still missing
Every request now arrives over a certificate the client trusts, gets checked against a real identity provider before it reaches an application, and carries a verified identity in headers when it gets there. The subrequest that makes that decision no longer crosses a wire in the clear.
That last clause is only true because I wrote this down. The configuration had been working perfectly the entire time, which is exactly the problem: nothing about it was going to fail in a way that got my attention. If you take one habit from this post rather than one configuration, take that one. Explain your setup to something that cannot nod along. A duck works. A reader works better, because a reader might copy it.
Two things are conspicuously absent. The first is any account of how the login page itself is reachable, which turns out to have more consequences than it looks like and is the whole of the next post. The second is any evidence that the applications behind all this actually care about the headers they are being handed.
They vary enormously. Some read them and log the user straight in. Some have their own network-based bypass and ignore them entirely. And some have no delegated authentication mode of any kind, so the user authenticates to authentik, arrives at the application, and is asked to log in again by software that never learned the first login happened.
That last category is where this series gets interesting.
Get it
The Caddy side is in my TechbyJeff repo under Docker/authentik/:
caddy/Caddyfilecontains everything above: the(authentik)snippet, a site block that imports it, and the per-application TLS endpoint. It is my working file with the domain changed and the header comment lists what you need to edit.docker-compose.ymlis the stack the snippet’shttp://server:9000upstream resolves against.
The nginx blocks are reproduced in full above rather than published, because they are excerpts from a vhost file that is specific to my services. Both the Caddy matchers and the nginx proxy_pass need your own addresses before any of it will do anything for you.
The .env holding the Cloudflare token is not in that repo and should not be in yours.
Sources
- Caddy: the
forward_authdirective - Caddy: Caddyfile concepts, snippets and imports
- Caddy: the
routedirective and directive order - authentik: Caddy configuration for proxy providers
- authentik: nginx configuration for proxy providers
- authentik: proxy providers and forward auth modes
- nginx:
ngx_http_auth_request_module - nginx:
proxy_ssl_verifyand related directives
This is part three of a series on the identity stack in my homelab. Part one covered deploying authentik itself. Part two covered certificates. Part four covers Cloudflare, and what changes when the identity provider becomes the one internet-facing thing you own. Part five covers Tautulli, which has its own login it does not want to give up. Part six covers OpenMediaVault, which has no delegated authentication mode at all.