Episode 12/13: CrowdSec next to fail2ban, and an AppSec WAF in Caddy

Measured on the demo box against the same 24h of real traffic.

Episode 12 (detection, blocklist, firewall):
- crowdsec/ with install notes and the traps: port 8080 clash, empty first
  blocklist pull, replay forensic-mode dropping events (FIFO workaround),
  ssh-time-based-bf shipping remediation:false, the AppSec collections that
  refuse to start without extra rules.
- firewall bouncer in nftables mode (input + forward), which blocks
  Docker-published ports and survives ufw reload; iptables/DOCKER-USER does not.
- HARDENING.md episode 12 section with the numbers: fail2ban 4 banned IPs vs
  CrowdSec 23 detected / 1 blocked, blocklist overlap 50% of IPs but 19% of
  attempts, and the mode=normal blind spot for key-only root attempts.

Episode 13 (AppSec WAF):
- caddy/Dockerfile now builds the CrowdSec bouncer and AppSec handler alongside
  rate-limit, pinned to v0.14.1.
- Caddyfile imports a crowdsec snippet (bouncer + appsec) on every site except
  vpn; compose passes CROWDSEC_BOUNCER_KEY and host.docker.internal.
- Measured: virtual patching blocks known CVEs only (7 of 41 probes, near-zero
  false positives); OWASP CRS raises that to 27 of 41 but breaks git push and
  wiki editing at default paranoia. Default stays vpatch+generic.

fail2ban config unchanged. No secrets or real domains/IPs in tracked files.

caddy/Dockerfile

@@ -1,9 +1,13 @@
 # episode 10: the stock caddy:2-alpine image cannot rate limit. The directive
 # lives in a plugin (github.com/mholt/caddy-ratelimit) that has to be compiled
-# in with xcaddy - there is no runtime "enable" for it. Two stages so the final
-# image stays the same small alpine, just with our own caddy binary.
+# in with xcaddy - there is no runtime "enable" for it.
+# episode 12: same trick for CrowdSec. The bouncer (http) and the AppSec
+# handler (appsec) are plugins too, pinned to a release tag instead of @main.
 FROM caddy:2-builder-alpine AS builder
-RUN xcaddy build --with github.com/mholt/caddy-ratelimit
+RUN xcaddy build \
+    --with github.com/mholt/caddy-ratelimit \
+    --with github.com/hslatman/caddy-crowdsec-bouncer/http@v0.14.1 \
+    --with github.com/hslatman/caddy-crowdsec-bouncer/appsec@v0.14.1
 
 FROM caddy:2-alpine
 COPY --from=builder /usr/bin/caddy /usr/bin/caddy

caddy/README.md

@@ -1,16 +1,23 @@
-# caddy/ (episode 10)
+# caddy/ (episode 10, extended in episode 12)
 
-Custom Caddy image with the rate limiting plugin compiled in.
+Custom Caddy image with the rate limiting plugin, the CrowdSec bouncer and the
+CrowdSec AppSec handler compiled in (see `crowdsec/README.md`).
 
 The stock `caddy:2-alpine` has no `rate_limit` directive; it is not a runtime
 flag, the module must be built into the binary with `xcaddy`. `compose.yml`
 carries both `image:` and `build: ./caddy`, so `docker compose build caddy`
-produces `selfhostedops/caddy:ratelimit` locally and `docker compose up -d`
+produces `selfhostedops/caddy:crowdsec` locally and `docker compose up -d`
 uses it.
 
 Do not build on a small box while the stack is running - a Go build can push a
 2 GB machine into OOM and take the live services down with it. Build it
 elsewhere (a bigger machine, or a throwaway host) and move it over:
 
-    docker build -t selfhostedops/caddy:ratelimit ./caddy
-    docker save selfhostedops/caddy:ratelimit | ssh root@box docker load
+    docker build -t selfhostedops/caddy:crowdsec ./caddy
+    docker save selfhostedops/caddy:crowdsec | ssh root@box docker load
+
+Measured in episode 12: the build with all three modules took 60 s on a 12 GB
+build host, the image is 158 MB (the rate-limit-only image was 154 MB).
+
+The modules are pinned to the release tag (`@v0.14.1`), not `@main` as the
+upstream README suggests, so a rebuild next month gives the same binary.