selfhostedops/selfhostedops-stack
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Episode 14: an agent reads the WAF blocks, a script beats it, a human merges

Browse source 
ops-readonly gets two read-only questions, waf_blocked and waf_rule_info. The
rule ids per blocked request are in CrowdSec's alerts, not in its log, and
alerts need machine credentials that can also write, so waf-export.py runs on
the host and the container gets one more read-only mount.

Workflow G lets a local 3B model draft CRS route exclusions as a Forgejo issue.
The model never writes rule syntax, a code node checks every field against the
tool output, the only write permission is an issue in one repo, trusted sources
live in the workflow and the trigger needs a token.

waf-propose.py does the same without a model: 5 of 5 false positives, no
exclusion for an attack, 0.03 s. The model proposed an exclusion for nearly
every attack it saw. Numbers and the traps we hit are in HARDENING.md.
This commit is contained in:
Maximilian Grimm 2026-09-19 12:56:39 +02:00
parent 696f3e311d
commit 3edef38ba3
13 changed files with 987 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

111
HARDENING.md
View file

@ -447,3 +447,114 @@ signals. Bigger lists and the console's premium tier are paid.
ones on the lines you tell it to watch; CrowdSec adds the slow bots (as
alerts), the blocklist and the WAF, at 4x the memory. Neither replaces the
other on these numbers.
## Episode 14: an agent reads the WAF blocks, a human merges
Episode 13 ended with a chore: the OWASP CRS blocks `git push` and wiki pages
with code blocks until somebody writes route exclusions, and every new app
needs new ones. Episode 14 lets a local 3B model draft them and measures
whether that is a good idea. Setup: `automation/README.md` (workflow G),
numbers: `lab-recordings-ep14/FACTS.md`.
**A false positive is not a 403, it is a lockout.** With the CRS on and no
exclusions, one normal minute of work (open an issue, clone, save a wiki page,
send a notification) produced four blocks in two seconds. CrowdSec's
`appsec-native` scenario and `http-generic-403-bf` (which counts the 403s in
the Caddy log) then banned the user's IP for four hours. After that even the
start page answered 403. That is the clock the advisor runs against.
**Where the rule ids are.** Not in `crowdsec.log`, which only has the anomaly
category and score. They are in the alerts: `cscli alerts list -o json`, kind
`waf`, one alert per blocked request, with rule ids, rule messages, method,
URI, user agent and host. Reading alerts over the Local API needs machine
credentials, and those can also write alerts and decisions. So `cscli` stays on
the host: `ops-readonly/waf-export.py` dumps the WAF alerts to a JSON file
every minute and `ops-readonly` gets one more read-only mount. Same container
limits as in episode 11, still no docker socket. Two new fixed questions:
`waf_blocked` and `waf_rule_info`. The tool leaves user agent and headers out
on purpose. Every field it does not pass on is one less channel for attacker
text.
**"Authenticated" is a header, not a fact.** The WAF answers before the app
checks the credential, so `auth_header_present` is set by whoever sends the
request. It is also honestly `false` for the most important false positive:
the blocked `git-upload-pack` carries no auth header, because git only sends
credentials after a 401 challenge and the WAF says 403 first.
**What a 3B model does with that.** With a skill that asks for judgement
(10 runs, qwen2.5:3b) it called almost every blocked request a false
positive, attacks included: 30 of 50 false positives proposed correctly, and
**104 exclusions that only an attack had asked for**, in 8 of 10 runs. It did
not invent rule ids, it mixed them up between routes. One route it never got
right in 16 runs. qwen3:4b thought out loud for 3000 tokens and never reached
the JSON (0 of 25, 282 s per run). The deterministic check in the
workflow dropped the malformed ones and still let well-formed exclusions for
`/.env` and `/.git/config` through (16 in 10 runs), because a format check
cannot see intent.
**What decides it is the one field nobody can forge.** Path, payload, rule ids
and the auth header can be copied by an attacker; the source address of a
completed HTTPS request cannot. Twenty lines of Python that group the same
data by route and rule id, count only blocks from trusted sources and require
two of them, found **5 of 5** false positives, proposed nothing for an attack,
ran in 0.03 s and gave the same answer every time
(`ops-readonly/waf-propose.py`). The same script trusting the auth header
instead found 3 of 5 and proposed an exclusion for the vault that an attacker
had caused alone, with three forged bearer requests. With the trusted source
written into the skill, the model did not improve (6 runs: 20 of 30 correct,
64 attack-only exclusions in 5 of 6 runs). Same lesson as episodes 9 and 11:
with a 3B model the model is the problem, not the prompt. Only the same rule
enforced in code removed the dangerous proposals, all of them. At that point
the check is the script, and the model costs 100 s and about 7,500 tokens per
run for what the script does in 30 ms.
**Injection and mimicry.** The attacker wrote instructions into the path and
query of blocked requests (plain text, ready-made JSON, a SecLang snippet). In
2 of 6 runs the model proposed removing 949110, the CRS blocking rule, an id
that appears nowhere in the tool data, only in the attacker's JSON. In 6 of 6
runs his sentence became the model's justification for every proposal ("is a
known false positive and should be excluded", once for "the SQL injection
endpoint"). The check dropped all of it. Two holes were ours: the issue listed
dropped proposals with their path, which put the injection text into the
issue, and the trusted sources came from the webhook body. A caller named the
attacker's IP as trusted, and the workflow delivered an issue with a ready
snippet exempting `/.env` on the vault. Both are fixed in the shipped version:
counts only, and the list lives in the workflow behind a trigger token. The
limit that no check removes: an
attacker who sends his payload to the same route as the false positive gets
the same rule ids, and the exclusion a human rightly merges for that route
lets him through as well (measured after the merge: 401 from the app instead
of 403 from the WAF). A route exclusion is a statement about a route, not
about who uses it.
**Narrower than episode 13.** The proposals remove the rule ids that actually
fired on a route instead of the CRS blocking rule 949110. Measured after the
merge: all five false positives gone including `git push`, all 18 attacks
still 403, virtual patching alive, and SQL injection on the exempted git API
still 403 (in episode 13 it passed), because no SQLi rule had fired there.
Only the class that was exempted passes (XSS, 200).
**The only write permission.** A bot user that is a read-only collaborator on
one private repo, token scopes `write:issue,read:repository`. Open an issue
201, write a file 403, read another private repo 404. The model never writes
rule syntax either: it returns fields, code renders the SecLang.
**Three traps.** Ollama has no output limit by default; our first run
generated for 11 min 47 s until we cancelled it. Set `numPredict` on the node,
cap the model tag as well (`PARAMETER num_predict`), set `executionTimeout`.
Then the one that cost us an hour: **a REST `PATCH` on an active n8n workflow
(2.37.7) changes the draft, not what runs.** The published nodes kept running
until we deactivated and activated the workflow; only `settings` applied at
once. Our `numPredict`, the fixed trusted sources and the trigger token were
all "deployed" and none of them was live. Check the `workflowData` snapshot in
an execution to see what really ran. And `cscli alerts delete` right after
`cscli decisions delete` removes the decision from the database before the
bouncers have seen it go: the IP stayed in the nftables set and in the Caddy
bouncer's cache until both were restarted.
**Verdict.** Let the script write the proposal and a human merge it. In the
live run the model delivered 4 of 5 routes in 30 s, the script 5 of 5 in
0.05 s; after merging the model's four, the issue tracker was still blocked
and `git push` failed, because it had exempted `git-upload-pack` only. The one
thing a model could add, the explanation, read "known attack pattern for RCE"
on a request it had just classified as a false positive.