Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain

- hardening/: ufw after.rules block for DOCKER-USER (ctorigdstport, not dport),
  fail2ban jail.local with sshd (systemd backend) and caddy-probes (chain DOCKER-USER),
  filter for the JSON access log, sshd hardening.conf, README with the order and
  the two traps measured in the episode
- caddy/Dockerfile: xcaddy build with caddy-ratelimit, compose builds the image
- Caddyfile: common snippet (JSON access log, Server header stripped), vault /admin 404,
  rate_limit zones on the vault token endpoint and the git login form
- HARDENING.md: firewall and fail2ban sections rewritten around what the port scan saw

Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA

hardening/README.md

@@ -0,0 +1,65 @@
+# hardening/ (episode 10)
+
+The four files the episode 1 checklist promised and never delivered: keys-only
+SSH, a firewall that actually closes Docker's published ports, and fail2ban with
+an SSH jail and a Caddy jail. Everything here was measured on the demo box, not
+assumed.
+
+## Order matters
+
+1. **SSH first, with a lifeline.** Copy `sshd/hardening.conf` to
+   `/etc/ssh/sshd_config.d/hardening.conf`, run `sshd -t`, then
+   `systemctl reload ssh`. Open a second SSH session and confirm it works
+   **before** you close the first one. Key auth is untouched by this change, but
+   a typo in the file is how people lock themselves out.
+
+2. **ufw, then the Docker fix.** Install ufw and allow only what the host itself
+   listens on:
+
+   ```
+   ufw default deny incoming
+   ufw default allow outgoing
+   ufw allow 22/tcp
+   ufw allow 51820/udp      # WireGuard of the NetBird client
+   ufw allow in on wt0      # the mesh
+   ufw enable
+   ```
+
+   Do **not** allow 80/443/2222 here. Those are Docker published ports, and ufw
+   cannot see them anyway (next point). Set a fallback before `ufw enable` in
+   case you fat-finger a rule and there is no serial console:
+   `nohup bash -c 'sleep 300; ufw --force disable' &` - then kill it once a
+   fresh SSH session confirms you are still in.
+
+3. **DOCKER-USER.** Paste the block from `ufw-after.rules` at the end of
+   `/etc/ufw/after.rules` (and `after6.rules`), then `ufw reload`. Without it,
+   `ufw deny incoming` is a lie: a colleague who runs
+   `docker run -p 9999:80 ...` to debug something publishes port 9999 straight
+   to the internet, ufw and all. Confirm with `iptables -S DOCKER-USER`.
+
+   Why `--ctorigdstport` and not `--dport`: DNAT rewrites the destination port
+   to the container's port before this chain runs, so `--dport` matches the
+   wrong port. Measured on the box, the naive `--dport 80,443,2222` version let
+   a `-p 9999:80` container through (its DNAT'd port is 80) and blocked git-ssh
+   (2222 DNAT'd to 22). `--ctorigdstport` matches the published port.
+
+4. **fail2ban last.** Copy `fail2ban/jail.local` and
+   `fail2ban/filter.d/caddy-probes.conf` into `/etc/fail2ban/`, then
+   `systemctl restart fail2ban`. Two jails:
+   - `sshd`: `backend = systemd`, because Debian 13 has no `/var/log/auth.log`
+     without rsyslog.
+   - `caddy-probes`: reads the JSON access log and bans on repeated
+     401/403/404/429. Its ban must land in `DOCKER-USER`, not INPUT - a ban in
+     INPUT never touches Caddy's forwarded traffic and the bot keeps getting
+     404s. Set `chain = DOCKER-USER` as a plain jail parameter, not inside
+     `banaction[...]`, or the `action_` template's `chain="%(chain)s"`
+     overrides it back to INPUT.
+
+## The one operational trap
+
+`ufw reload` rewrites DOCKER-USER from `after.rules`, which flushes the chain and
+throws away fail2ban's ban jumps with it. After any `ufw reload`, run
+`systemctl restart fail2ban` to put the jumps (and the active bans, which
+fail2ban keeps in its own database) back. A plain `systemctl restart docker`
+does not have this problem: DOCKER-USER keeps both our rules and the fail2ban
+jumps across a Docker restart (measured).

hardening/fail2ban/filter.d/caddy-probes.conf

@@ -0,0 +1,8 @@
+# /etc/fail2ban/filter.d/caddy-probes.conf   (episode 10)
+# Matches the JSON access log: bots that harvest 401/403/404/429 (probing for
+# /.env, /wp-login.php, /xmlrpc.php, or hammering a login past the rate limit).
+# The whole log entry is one line; client_ip sits in the request object and
+# status at the top level, so client_ip always comes before status.
+[Definition]
+failregex = ^.*"client_ip":"<HOST>".*"status":(401|403|404|429)\b.*$
+ignoreregex =

hardening/fail2ban/jail.local

@@ -0,0 +1,33 @@
+# /etc/fail2ban/jail.local   (episode 10)
+[DEFAULT]
+bantime  = 1h
+findtime = 10m
+maxretry = 5
+# localhost and the NetBird mesh range are never banned. Do NOT put your own
+# home/office IP here - if you never guess passwords you never get banned, and
+# an ignore entry is one more secret to keep out of a public repo.
+ignoreip = 127.0.0.1/8 ::1 100.125.0.0/16
+banaction = iptables-multiport
+
+[sshd]
+enabled  = true
+# Debian 13 ships no /var/log/auth.log without rsyslog; read the journal instead.
+backend  = systemd
+maxretry = 5
+findtime = 10m
+bantime  = 1h
+
+[caddy-probes]
+enabled  = true
+backend  = auto
+logpath  = /opt/selfhostedops-stack/logs/caddy/access.log
+filter   = caddy-probes
+maxretry = 10
+findtime = 1m
+bantime  = 1h
+port     = http,https
+# The ban has to land in DOCKER-USER, not INPUT - Caddy's traffic is forwarded,
+# a ban in INPUT never touches it and the "banned" bot keeps getting answers.
+# chain must be a plain parameter here: the default action_ template passes
+# chain="%(chain)s", so putting [chain=...] inside banaction gets overridden.
+chain = DOCKER-USER

hardening/sshd/hardening.conf

@@ -0,0 +1,10 @@
+# /etc/ssh/sshd_config.d/hardening.conf   (episode 10)
+# Keys only. Password guessing is what the 130k login attempts in eight days were.
+PasswordAuthentication no
+KbdInteractiveAuthentication no
+# Root stays reachable by key on this single-admin box. On a box with a real
+# ops user, set this to "no" and log in as that user instead.
+PermitRootLogin prohibit-password
+#
+# After editing: sshd -t && systemctl reload ssh
+# TEST A SECOND SSH SESSION BEFORE YOU CLOSE THIS ONE.

hardening/ufw-after.rules

@@ -0,0 +1,29 @@
+# episode 10 - paste this block at the END of /etc/ufw/after.rules
+# (and the same block in /etc/ufw/after6.rules for IPv6).
+#
+# Why: Docker publishes ports by DNAT in the FORWARD path, not INPUT. ufw only
+# filters INPUT, so `ufw deny incoming` does NOT close a published container
+# port - it stays wide open. DOCKER-USER is the one chain Docker runs first in
+# FORWARD and never flushes on its own, so the fix goes here.
+#
+# The catch inside the catch: DNAT rewrites the destination port to the
+# container's port BEFORE this chain sees the packet. Matching --dport would
+# match the wrong (translated) port. So match the ORIGINAL published port with
+# conntrack --ctorigdstport instead.
+#
+# Only eth0 (the public NIC) is filtered. wt0 (the NetBird mesh) and the docker
+# bridges are left alone on purpose - a rule without `-i eth0` here breaks the
+# mesh route to n8n (episode 8) and traffic between containers.
+#
+# Replace eth0 with your public interface, and the port list with what you
+# actually publish.
+*filter
+:DOCKER-USER - [0:0]
+-A DOCKER-USER -i eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
+-A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 80 -j RETURN
+-A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 443 -j RETURN
+-A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 2222 -j RETURN
+-A DOCKER-USER -i eth0 -p udp -m conntrack --ctorigdstport 443 -j RETURN
+-A DOCKER-USER -i eth0 -p udp -m conntrack --ctorigdstport 3478 -j RETURN
+-A DOCKER-USER -i eth0 -j DROP
+COMMIT