History

Maximilian Grimm bed41cfaf3 Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain - hardening/: ufw after.rules block for DOCKER-USER (ctorigdstport, not dport), fail2ban jail.local with sshd (systemd backend) and caddy-probes (chain DOCKER-USER), filter for the JSON access log, sshd hardening.conf, README with the order and the two traps measured in the episode - caddy/Dockerfile: xcaddy build with caddy-ratelimit, compose builds the image - Caddyfile: common snippet (JSON access log, Server header stripped), vault /admin 404, rate_limit zones on the vault token endpoint and the git login form - HARDENING.md: firewall and fail2ban sections rewritten around what the port scan saw Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA		2026-09-11 10:20:56 +02:00
..
fail2ban	Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain	2026-09-11 10:20:56 +02:00
sshd	Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain	2026-09-11 10:20:56 +02:00
README.md	Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain	2026-09-11 10:20:56 +02:00
ufw-after.rules	Episode 10: hardening - firewall rules Docker cannot bypass, Caddy access log and rate limiting, fail2ban in the right chain	2026-09-11 10:20:56 +02:00

README.md

hardening/ (episode 10)

The four files the episode 1 checklist promised and never delivered: keys-only SSH, a firewall that actually closes Docker's published ports, and fail2ban with an SSH jail and a Caddy jail. Everything here was measured on the demo box, not assumed.

Order matters

SSH first, with a lifeline. Copy sshd/hardening.conf to /etc/ssh/sshd_config.d/hardening.conf, run sshd -t, then systemctl reload ssh. Open a second SSH session and confirm it works before you close the first one. Key auth is untouched by this change, but a typo in the file is how people lock themselves out.
ufw, then the Docker fix. Install ufw and allow only what the host itself listens on:
```
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 51820/udp      # WireGuard of the NetBird client
ufw allow in on wt0      # the mesh
ufw enable
```
Do not allow 80/443/2222 here. Those are Docker published ports, and ufw cannot see them anyway (next point). Set a fallback before ufw enable in case you fat-finger a rule and there is no serial console: nohup bash -c 'sleep 300; ufw --force disable' & - then kill it once a fresh SSH session confirms you are still in.
DOCKER-USER. Paste the block from ufw-after.rules at the end of /etc/ufw/after.rules (and after6.rules), then ufw reload. Without it, ufw deny incoming is a lie: a colleague who runs docker run -p 9999:80 ... to debug something publishes port 9999 straight to the internet, ufw and all. Confirm with iptables -S DOCKER-USER.

Why --ctorigdstport and not --dport: DNAT rewrites the destination port to the container's port before this chain runs, so --dport matches the wrong port. Measured on the box, the naive --dport 80,443,2222 version let a -p 9999:80 container through (its DNAT'd port is 80) and blocked git-ssh (2222 DNAT'd to 22). --ctorigdstport matches the published port.
fail2ban last. Copy fail2ban/jail.local and fail2ban/filter.d/caddy-probes.conf into /etc/fail2ban/, then systemctl restart fail2ban. Two jails:
- sshd: backend = systemd, because Debian 13 has no /var/log/auth.log without rsyslog.
- caddy-probes: reads the JSON access log and bans on repeated 401/403/404/429. Its ban must land in DOCKER-USER, not INPUT - a ban in INPUT never touches Caddy's forwarded traffic and the bot keeps getting 404s. Set chain = DOCKER-USER as a plain jail parameter, not inside banaction[...], or the action_ template's chain="%(chain)s" overrides it back to INPUT.

The one operational trap

ufw reload rewrites DOCKER-USER from after.rules, which flushes the chain and throws away fail2ban's ban jumps with it. After any ufw reload, run systemctl restart fail2ban to put the jumps (and the active bans, which fail2ban keeps in its own database) back. A plain systemctl restart docker does not have this problem: DOCKER-USER keeps both our rules and the fail2ban jumps across a Docker restart (measured).