Pocket ID replaces Forgejo as the OIDC provider for Outline, Vaultwarden and
NetBird; Forgejo logs in through it. Uptime Kuma sits behind tinyauth via
Caddy forward_auth (group ops), the public status page stays public.
offboarding/offboard.py switches one person off in every service with an API
and prints what a human still has to do: the shared passwords they could read.
Measured: SSO lock alone left 10 of 13 ways open for 45 minutes; the script
leaves 1 of 14, the vault copy on their laptop.
HARDENING.md: the ufw hairpin that broke Vaultwarden SSO and Kuma, the mesh
route into the docker bridges, and a fail2ban exception for Pocket ID.
Measured on the demo box against the same 24h of real traffic.
Episode 12 (detection, blocklist, firewall):
- crowdsec/ with install notes and the traps: port 8080 clash, empty first
blocklist pull, replay forensic-mode dropping events (FIFO workaround),
ssh-time-based-bf shipping remediation:false, the AppSec collections that
refuse to start without extra rules.
- firewall bouncer in nftables mode (input + forward), which blocks
Docker-published ports and survives ufw reload; iptables/DOCKER-USER does not.
- HARDENING.md episode 12 section with the numbers: fail2ban 4 banned IPs vs
CrowdSec 23 detected / 1 blocked, blocklist overlap 50% of IPs but 19% of
attempts, and the mode=normal blind spot for key-only root attempts.
Episode 13 (AppSec WAF):
- caddy/Dockerfile now builds the CrowdSec bouncer and AppSec handler alongside
rate-limit, pinned to v0.14.1.
- Caddyfile imports a crowdsec snippet (bouncer + appsec) on every site except
vpn; compose passes CROWDSEC_BOUNCER_KEY and host.docker.internal.
- Measured: virtual patching blocks known CVEs only (7 of 41 probes, near-zero
false positives); OWASP CRS raises that to 27 of 41 but breaks git push and
wiki editing at default paranoia. Default stays vpatch+generic.
fail2ban config unchanged. No secrets or real domains/IPs in tracked files.
The agent runs in n8n on the box and reaches its model over the mesh from
episode five, so no prompt leaves the network. It gets three read-only tools
(recent commits, monitor status, wiki search) and returns one suggested
action as a field. A switch compares that field against three allowed values,
and only one of them reaches ops-actions, a small runner holding its own
hard-coded allowlists for actions and targets.
Measured while building this: the same prompt injection was harmless in every
place the agent reached through a tool (twenty runs, no hits) and worked every
time it arrived in the alert itself, through a poisoned monitor name (thirteen
of thirteen). The allowlist rejected all thirteen. It guards the action field
only, so the attacker-written summary still reached the wiki unchanged; the
triage note now truncates and strips every field it did not write itself.
Three workflows that connect the services from episodes 1 to 7: a Forgejo push
becomes a phone notification, an Uptime Kuma outage writes and closes its own
incident page in Outline, and a nightly schedule writes one ops digest.
The point of the episode is the split between webhooks and the editor. Caddy
publishes only the webhook path prefixes under the n8n host and answers an
empty 404 for everything else, so the editor stays reachable over the mesh from
episode 5 and never gets a login page on the open internet.
Also in here:
- n8n pinned to a version instead of :latest, after episode 7 showed that every
restore is otherwise an upgrade
- the 2.x environment that actually matters, including the file access
restriction pointed at one directory rather than the root
- the Forgejo allowed host list, without which a webhook to a private address
is silently dropped
- fixed subnets for both docker networks, after a restore handed them out in
the other order and quietly broke the mesh route
- a note on the encryption key: it decrypts every stored credential, it lives in
the n8n volume, and a backup without it restores a hub that cannot log in
Claude-Session: https://claude.ai/code/session_01Md8yhBKp5e7xMmCyxoEgdm
- compose: SSO_ENABLED, SSO_AUTHORITY and the two client credentials, with the
undocumented redirect path written down where people will look for it
- .env.example: was referenced by the README since episode 1 but never actually
committed, so a fresh clone could not follow the quickstart
- README: the four steps, plus the debugging notes that cost the most time
(no SSO line at startup, 404 on routes without query parameters, HEAD on the
discovery endpoint, and the three-step member join)
- HARDENING: never copy db.sqlite3 alone while the container runs. WAL mode
keeps recent writes in a side file, so that backup silently misses data
