ops-readonly gets two read-only questions, waf_blocked and waf_rule_info. The
rule ids per blocked request are in CrowdSec's alerts, not in its log, and
alerts need machine credentials that can also write, so waf-export.py runs on
the host and the container gets one more read-only mount.
Workflow G lets a local 3B model draft CRS route exclusions as a Forgejo issue.
The model never writes rule syntax, a code node checks every field against the
tool output, the only write permission is an issue in one repo, trusted sources
live in the workflow and the trigger needs a token.
waf-propose.py does the same without a model: 5 of 5 false positives, no
exclusion for an attack, 0.03 s. The model proposed an exclusion for nearly
every attack it saw. Numbers and the traps we hit are in HARDENING.md.
Measured on the demo box against the same 24h of real traffic.
Episode 12 (detection, blocklist, firewall):
- crowdsec/ with install notes and the traps: port 8080 clash, empty first
blocklist pull, replay forensic-mode dropping events (FIFO workaround),
ssh-time-based-bf shipping remediation:false, the AppSec collections that
refuse to start without extra rules.
- firewall bouncer in nftables mode (input + forward), which blocks
Docker-published ports and survives ufw reload; iptables/DOCKER-USER does not.
- HARDENING.md episode 12 section with the numbers: fail2ban 4 banned IPs vs
CrowdSec 23 detected / 1 blocked, blocklist overlap 50% of IPs but 19% of
attempts, and the mode=normal blind spot for key-only root attempts.
Episode 13 (AppSec WAF):
- caddy/Dockerfile now builds the CrowdSec bouncer and AppSec handler alongside
rate-limit, pinned to v0.14.1.
- Caddyfile imports a crowdsec snippet (bouncer + appsec) on every site except
vpn; compose passes CROWDSEC_BOUNCER_KEY and host.docker.internal.
- Measured: virtual patching blocks known CVEs only (7 of 41 probes, near-zero
false positives); OWASP CRS raises that to 27 of 41 but breaks git push and
wiki editing at default paranoia. Default stays vpatch+generic.
fail2ban config unchanged. No secrets or real domains/IPs in tracked files.
- hardening/: ufw after.rules block for DOCKER-USER (ctorigdstport, not dport),
fail2ban jail.local with sshd (systemd backend) and caddy-probes (chain DOCKER-USER),
filter for the JSON access log, sshd hardening.conf, README with the order and
the two traps measured in the episode
- caddy/Dockerfile: xcaddy build with caddy-ratelimit, compose builds the image
- Caddyfile: common snippet (JSON access log, Server header stripped), vault /admin 404,
rate_limit zones on the vault token endpoint and the git login form
- HARDENING.md: firewall and fail2ban sections rewritten around what the port scan saw
Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA
Three workflows that connect the services from episodes 1 to 7: a Forgejo push
becomes a phone notification, an Uptime Kuma outage writes and closes its own
incident page in Outline, and a nightly schedule writes one ops digest.
The point of the episode is the split between webhooks and the editor. Caddy
publishes only the webhook path prefixes under the n8n host and answers an
empty 404 for everything else, so the editor stays reachable over the mesh from
episode 5 and never gets a login page on the open internet.
Also in here:
- n8n pinned to a version instead of :latest, after episode 7 showed that every
restore is otherwise an upgrade
- the 2.x environment that actually matters, including the file access
restriction pointed at one directory rather than the root
- the Forgejo allowed host list, without which a webhook to a private address
is silently dropped
- fixed subnets for both docker networks, after a restore handed them out in
the other order and quietly broke the mesh route
- a note on the encryption key: it decrypts every stored credential, it lives in
the n8n volume, and a backup without it restores a hub that cannot log in
Claude-Session: https://claude.ai/code/session_01Md8yhBKp5e7xMmCyxoEgdm
- compose: SSO_ENABLED, SSO_AUTHORITY and the two client credentials, with the
undocumented redirect path written down where people will look for it
- .env.example: was referenced by the README since episode 1 but never actually
committed, so a fresh clone could not follow the quickstart
- README: the four steps, plus the debugging notes that cost the most time
(no SSO line at startup, 404 on routes without query parameters, HEAD on the
discovery endpoint, and the three-step member join)
- HARDENING: never copy db.sqlite3 alone while the container runs. WAL mode
keeps recent writes in a side file, so that backup silently misses data
