Episode 12/13: CrowdSec next to fail2ban, and an AppSec WAF in Caddy

Measured on the demo box against the same 24h of real traffic.

Episode 12 (detection, blocklist, firewall):
- crowdsec/ with install notes and the traps: port 8080 clash, empty first
  blocklist pull, replay forensic-mode dropping events (FIFO workaround),
  ssh-time-based-bf shipping remediation:false, the AppSec collections that
  refuse to start without extra rules.
- firewall bouncer in nftables mode (input + forward), which blocks
  Docker-published ports and survives ufw reload; iptables/DOCKER-USER does not.
- HARDENING.md episode 12 section with the numbers: fail2ban 4 banned IPs vs
  CrowdSec 23 detected / 1 blocked, blocklist overlap 50% of IPs but 19% of
  attempts, and the mode=normal blind spot for key-only root attempts.

Episode 13 (AppSec WAF):
- caddy/Dockerfile now builds the CrowdSec bouncer and AppSec handler alongside
  rate-limit, pinned to v0.14.1.
- Caddyfile imports a crowdsec snippet (bouncer + appsec) on every site except
  vpn; compose passes CROWDSEC_BOUNCER_KEY and host.docker.internal.
- Measured: virtual patching blocks known CVEs only (7 of 41 probes, near-zero
  false positives); OWASP CRS raises that to 27 of 41 but breaks git push and
  wiki editing at default paranoia. Default stays vpatch+generic.

fail2ban config unchanged. No secrets or real domains/IPs in tracked files.

HARDENING.md

@@ -86,6 +86,12 @@ advice from episode 1 got wrong:
   the ban in INPUT; Caddy's traffic is forwarded, so a bot "banned" in INPUT
   keeps getting 404s (measured). Set `chain = DOCKER-USER` as a plain jail
   parameter and the same bot runs into a refused connection instead.
+- **Once passwords are off, `mode = normal` goes half blind** (episode 12).
+  Key-only root attempts log as `Connection closed by authenticating user
+  root [preauth]`, which the normal sshd filter ignores. Measured: 192 attempts
+  from one IP in 24 h, zero seen. `mode = aggressive` in the jail counts them;
+  it also counts the second line of each invalid-user attempt, so maxretry 5
+  bans after roughly 3 tries.
 
 Operational note: `ufw reload` flushes `DOCKER-USER` and takes fail2ban's ban
 jumps with it, run `systemctl restart fail2ban` afterwards to restore them.
@@ -374,3 +380,70 @@ agent doing the whole job against seven tools gets the counts right often but
 not always; splitting the work into three narrow roles, each with one skill
 file (`agents/skills/`), trades more tokens for a more reliable report. The
 numbers are in the lab notes.
+
+## Episode 12: CrowdSec next to fail2ban, same box, same logs
+
+A viewer asked for it: does CrowdSec replace fail2ban? We ran both on the demo
+box and gave them the same 24 hours of real traffic (fail2ban live, CrowdSec by
+replaying the identical sshd log). Setup and traps: `crowdsec/README.md`.
+
+**What fail2ban caught in 24 h** (password auth already off since episode 10):
+1,458 SSH login attempts from 218 IPs. fail2ban banned **4 IPs** (6 bans).
+Two blind spots, both measured:
+
+- *Slow bots.* One IP tried 273 times, one attempt every ~4.5 minutes. Never 5
+  in 10 minutes, so never banned.
+- *Root without a password path.* `Connection closed by authenticating user
+  root [preauth]` is not a failure in the default `mode = normal` of the sshd
+  filter. One IP made 192 of those attempts and fail2ban did not see a single
+  one. `mode = aggressive` matches 2,192 of 2,345 lines instead of 696.
+
+**What CrowdSec detected on the same log:** 23 IPs, including the slow one.
+**What it would have banned: 1 IP.** 22 of the 23 came from
+`ssh-time-based-bf`, and that scenario ships with `remediation: false`: it
+raises an alert (and a community signal) but no decision. Out of the box,
+CrowdSec *sees* more and *blocks* less than fail2ban on this traffic. If you
+want the slow ones banned, add a profile for that scenario instead of editing
+it: a modified hub scenario is "tainted", its signals stop counting, and a
+non-contributing engine drops to the 3,000-IP Lite blocklist.
+
+**The community blocklist:** 15,000 IPs on day one (after a restart, see the
+README). Of the 218 IPs that tried to log in during the previous 24 h, **109
+were already on it**, but those 109 made only **19 %** of the attempts. The list
+knows the one-shot scanners; the persistent bots that hammer a single box were
+mostly not on it (1 of the top 10). None of the 4 IPs fail2ban banned were
+listed. The two tools overlap less than you would think.
+
+**The firewall bouncer and Docker, again.** In its default iptables mode the
+bouncer only hooks `INPUT`: a banned IP gets a timeout on SSH and still 200 on
+every Docker-published port. `DOCKER-USER` fixes it until `ufw reload` flushes
+the chain (it takes fail2ban's jump with it; nothing restores either). The
+bouncer's **nftables mode** hooks `input` and `forward` in its own table, blocks
+the Docker ports, and survived `ufw reload` untouched.
+
+**Controlled attack, both watching:**
+
+| | fail2ban | CrowdSec |
+|---|---|---|
+| invalid user, 1 try/s | banned after 5 tries (~5 s) | detected after 3 tries, firewall drop ~3 s later |
+| `root`, key-only, 1 try/s | never | detected after 6 tries, drop ~3 s later |
+| 12 web probes on vault (`/.env` answers 422) | no ban (filter counts 401/403/404/429 only) | detected after 3 probes, drop ~3 s later |
+
+fail2ban acts the moment it matches (REJECT, "connection refused"). CrowdSec
+decides fast but the bouncer polls every 10 s, so a few more requests slip
+through (DROP, timeout).
+
+**AppSec (WAF) in Caddy:** every exploit request (`/.env`, `/.git/config`, the
+PHPUnit RCE) got a **403 on the first try**, while normal requests in between
+stayed 200. Two blocks and the IP is banned at the firewall. Cost: +1 to 2 ms
+median time to first byte, one more round-trip to the host per request.
+
+**What it costs on a 4 GB box:** CrowdSec ~200 MB RSS, firewall bouncer
+~30 MB, fail2ban ~50 MB. The engine, the bouncers and the hub are MIT or Apache
+licensed and free; the 15k community list is free as long as you share
+signals. Bigger lists and the console's premium tier are paid.
+
+**Verdict for this stack:** run both. fail2ban reacts instantly to the loud
+ones on the lines you tell it to watch; CrowdSec adds the slow bots (as
+alerts), the blocklist and the WAF, at 4x the memory. Neither replaces the
+other on these numbers.