ops-readonly gets kuma_alerts (host export kuma-export.py, one more read-only
mount). kuma-triage.py sorts the same data without a model. Flow H runs the
agent with skill alert-triage.md. Flow B: one open incident per monitor, an
UP closes all of them, titles name start and end and stay under Outline's
limit. B and D accept Kuma only with a header token. Monitoring notes: ntfy
rejects Kuma messages for monitors without a URL, resend for long outages.
Pocket ID replaces Forgejo as the OIDC provider for Outline, Vaultwarden and
NetBird; Forgejo logs in through it. Uptime Kuma sits behind tinyauth via
Caddy forward_auth (group ops), the public status page stays public.
offboarding/offboard.py switches one person off in every service with an API
and prints what a human still has to do: the shared passwords they could read.
Measured: SSO lock alone left 10 of 13 ways open for 45 minutes; the script
leaves 1 of 14, the vault copy on their laptop.
HARDENING.md: the ufw hairpin that broke Vaultwarden SSO and Kuma, the mesh
route into the docker bridges, and a fail2ban exception for Pocket ID.
ops-readonly gets two read-only questions, waf_blocked and waf_rule_info. The
rule ids per blocked request are in CrowdSec's alerts, not in its log, and
alerts need machine credentials that can also write, so waf-export.py runs on
the host and the container gets one more read-only mount.
Workflow G lets a local 3B model draft CRS route exclusions as a Forgejo issue.
The model never writes rule syntax, a code node checks every field against the
tool output, the only write permission is an issue in one repo, trusted sources
live in the workflow and the trigger needs a token.
waf-propose.py does the same without a model: 5 of 5 false positives, no
exclusion for an attack, 0.03 s. The model proposed an exclusion for nearly
every attack it saw. Numbers and the traps we hit are in HARDENING.md.
Measured on the demo box against the same 24h of real traffic.
Episode 12 (detection, blocklist, firewall):
- crowdsec/ with install notes and the traps: port 8080 clash, empty first
blocklist pull, replay forensic-mode dropping events (FIFO workaround),
ssh-time-based-bf shipping remediation:false, the AppSec collections that
refuse to start without extra rules.
- firewall bouncer in nftables mode (input + forward), which blocks
Docker-published ports and survives ufw reload; iptables/DOCKER-USER does not.
- HARDENING.md episode 12 section with the numbers: fail2ban 4 banned IPs vs
CrowdSec 23 detected / 1 blocked, blocklist overlap 50% of IPs but 19% of
attempts, and the mode=normal blind spot for key-only root attempts.
Episode 13 (AppSec WAF):
- caddy/Dockerfile now builds the CrowdSec bouncer and AppSec handler alongside
rate-limit, pinned to v0.14.1.
- Caddyfile imports a crowdsec snippet (bouncer + appsec) on every site except
vpn; compose passes CROWDSEC_BOUNCER_KEY and host.docker.internal.
- Measured: virtual patching blocks known CVEs only (7 of 41 probes, near-zero
false positives); OWASP CRS raises that to 27 of 41 but breaks git push and
wiki editing at default paranoia. Default stays vpatch+generic.
fail2ban config unchanged. No secrets or real domains/IPs in tracked files.
New: ops-readonly, the read-only twin of the episode 9 ops-actions runner
(fixed question list, three read-only mounts, no docker socket, read_only,
cap_drop ALL). Four role skills in agents/skills/, loaded at run time from
this repo's Forgejo. Two workflows: E (one agent, seven tools, one skill) and
F (three agents, one skill each, the workflow is the orchestrator).
Measured over 20 runs: no run of either variant produced all numbers and
lists correct. The solo agent confuses similar counts and skips a tool; the
crew calls every tool but drops fields at each role boundary. Sharpening the
skill files did not help the crew, it made it worse. Logs are attacker text:
an instruction in a request path is carried into the report as data, not
obeyed, but carried.
Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA
- hardening/: ufw after.rules block for DOCKER-USER (ctorigdstport, not dport),
fail2ban jail.local with sshd (systemd backend) and caddy-probes (chain DOCKER-USER),
filter for the JSON access log, sshd hardening.conf, README with the order and
the two traps measured in the episode
- caddy/Dockerfile: xcaddy build with caddy-ratelimit, compose builds the image
- Caddyfile: common snippet (JSON access log, Server header stripped), vault /admin 404,
rate_limit zones on the vault token endpoint and the git login form
- HARDENING.md: firewall and fail2ban sections rewritten around what the port scan saw
Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA
The agent runs in n8n on the box and reaches its model over the mesh from
episode five, so no prompt leaves the network. It gets three read-only tools
(recent commits, monitor status, wiki search) and returns one suggested
action as a field. A switch compares that field against three allowed values,
and only one of them reaches ops-actions, a small runner holding its own
hard-coded allowlists for actions and targets.
Measured while building this: the same prompt injection was harmless in every
place the agent reached through a tool (twenty runs, no hits) and worked every
time it arrived in the alert itself, through a poisoned monitor name (thirteen
of thirteen). The allowlist rejected all thirteen. It guards the action field
only, so the attacker-written summary still reached the wiki unchanged; the
triage note now truncates and strips every field it did not write itself.
Three workflows that connect the services from episodes 1 to 7: a Forgejo push
becomes a phone notification, an Uptime Kuma outage writes and closes its own
incident page in Outline, and a nightly schedule writes one ops digest.
The point of the episode is the split between webhooks and the editor. Caddy
publishes only the webhook path prefixes under the n8n host and answers an
empty 404 for everything else, so the editor stays reachable over the mesh from
episode 5 and never gets a login page on the open internet.
Also in here:
- n8n pinned to a version instead of :latest, after episode 7 showed that every
restore is otherwise an upgrade
- the 2.x environment that actually matters, including the file access
restriction pointed at one directory rather than the root
- the Forgejo allowed host list, without which a webhook to a private address
is silently dropped
- fixed subnets for both docker networks, after a restore handed them out in
the other order and quietly broke the mesh route
- a note on the encryption key: it decrypts every stored credential, it lives in
the n8n volume, and a backup without it restores a hub that cannot log in
Claude-Session: https://claude.ai/code/session_01Md8yhBKp5e7xMmCyxoEgdm
- compose: SSO_ENABLED, SSO_AUTHORITY and the two client credentials, with the
undocumented redirect path written down where people will look for it
- .env.example: was referenced by the README since episode 1 but never actually
committed, so a fresh clone could not follow the quickstart
- README: the four steps, plus the debugging notes that cost the most time
(no SSO line at startup, 404 on routes without query parameters, HEAD on the
discovery endpoint, and the three-step member join)
- HARDENING: never copy db.sqlite3 alone while the container runs. WAL mode
keeps recent writes in a side file, so that backup silently misses data
