Episode 13: correct the CRS exclusion mechanism (ruleRemoveById, not ruleEngine)

Follow-up measurements showed the first exclusion approach was too broad.
ctl:ruleEngine=Off disables the whole Coraza engine for the route, which also
turns off the native virtual-patching rules: a .env probe on such a route
returned 404 (no WAF) instead of 403.

The exclusion now uses ctl:ruleRemoveById=949110,959100 (the CRS inbound and
outbound anomaly-score blockers) so the engine stays on and virtual patching
keeps firing on the excluded routes. Measured: .env / PHPUnit / .git/config
still 403 on those routes, generic SQLi/XSS pass, non-excluded hosts keep full
CRS (27 of 41). Rule order relative to the CRS rule no longer matters.

Also measured and documented: RemoveInBandRuleByName in a pre_eval hook does
nothing to CRS (SQLi stayed 403 globally and per route); a full authenticated
Vaultwarden flow (register, login, sync, cipher create) passes under full CRS
on the vault host. README traps and numbers updated accordingly.

crowdsec/README.md

@@ -141,14 +141,26 @@ SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook
 bodies. Six normal workflows, all 403. For a stack where people push git and
 write wiki pages, unusable as-is.
 
-The fix is per-route exclusion, not turning CRS off globally.
-`appsec-rules/crs-route-exclusions.yaml` is a small SecLang rule that runs in
-phase 1 and sets `ctl:ruleEngine=Off` for exactly the legitimate application
-routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy publish, n8n
-webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`. `ctl:ruleEngine`
-only affects the Coraza/SecLang engine, so the native virtual-patching and
-generic rules still run on those routes; only the generic CRS body/URL
-inspection is skipped there.
+The fix is a per-route exclusion. `appsec-rules/crs-route-exclusions.yaml` is a
+small SecLang rule that runs in phase 1 and, for exactly the legitimate
+application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy
+publish, n8n webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`,
+removes the CRS anomaly blocking rules with
+`ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100` (949110 is the CRS
+inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the
+request but no longer blocks it on those routes.
+
+Do **not** use `ctl:ruleEngine=Off` for this. It turns the whole Coraza engine
+off for the request, and measured on the box that also disables the native
+CrowdSec virtual-patching rules: on a route excluded with `ruleEngine=Off`, a
+`.env` probe returned 404 (no WAF at all) instead of 403. With
+`ctl:ruleRemoveById` the engine stays on, so virtual patching keeps firing:
+measured, `.env`, the PHPUnit RCE and `.git/config` still return 403 on the
+excluded routes, while generic SQLi/XSS now pass (that is the point).
+
+Rule order does not matter for `ctl:ruleRemoveById` (a phase-1 ctl removes the
+phase-2 blocker regardless of file position); the exclusion sits first only for
+readability. It would matter for `ruleEngine=Off`, one more reason to avoid it.
 
 `appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule
 (must come **before** crs so the ctl takes effect in phase 1), crs, then
@@ -161,26 +173,26 @@ cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/
 crowdsec -t && systemctl restart crowdsec
 ```
 
-Measured with the tuned config: **0 false positives** across all the workflows
-above, still **27 of 41** probes blocked on the non-excluded routes. The cost is
-precise and worth stating out loud: on an excluded route (say the Forgejo API),
-generic SQLi/XSS now passes to the app (virtual patching still covers known
-CVEs); on a non-excluded host (vault, status) the full CRS applies. Latency
-added over vpatch: about +2 ms time-to-first-byte per request. RAM: CRS adds
-roughly 15 to 40 MB over vpatch (RSS settles near 185 MB, peaks higher right
-after loading the rule data).
+Measured with the tuned config: **0 false positives** across every workflow
+above (git clone/push, Forgejo issues and file contents, Outline documents with
+an attack-looking code block, ntfy publish, n8n webhooks, and a full
+authenticated Vaultwarden register/login/sync/cipher-create), still **27 of 41**
+probes blocked on the non-excluded routes. The cost is precise: on an excluded
+route (say the Forgejo API) generic SQLi/XSS now passes to the app, while
+virtual patching still catches known CVEs there, and on a non-excluded host
+(vault, status) the full CRS applies. Latency added over vpatch: about +2 ms
+time-to-first-byte per request. RAM: see the episode notes; CRS settles around
+the same order as vpatch once loaded, roughly +15 to 40 MB.
 
 Traps found while tuning:
 - `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not**
-  disable CRS: the bouncer name does not match the compiled SecLang rules. Use
-  `ctl:ruleEngine` inside SecLang instead.
-- The exclusion rule must be listed before `crowdsecurity/crs` in the config, or
-  the phase-1 `ctl` runs too late.
+  disable CRS: the bouncer name does not match the compiled SecLang rules
+  (measured, globally and per route, SQLi stayed 403). Use `ctl:` inside SecLang.
+  Docs: https://docs.crowdsec.net/docs/appsec/hooks
+- `ctl:ruleEngine=Off` is too broad: it disables virtual patching on the route
+  too. Use `ctl:ruleRemoveById` for the CRS blocking rules and keep the engine on.
 - Every new app or new public path needs its own exclusion. A missed one blocks
   a real workflow, and it fails only after login, so it is easy to miss.
-- Authenticated Vaultwarden and Outline sync with real encrypted payloads were
-  not exercised against CRS here; test your own before trusting CRS on those
-  hosts.
 
 **What this repo runs by default: vpatch + generic** (the `appsec-default`
 config, zero false positives, no per-app tuning). The CRS files above are the

crowdsec/appsec-configs/appsec-tuned.yaml

@@ -1,9 +1,10 @@
 name: selfhostedops/appsec-tuned
 default_remediation: ban
-# ep13: vpatch + generic (nativ) plus OWASP CRS (SecLang). Eine lokale
-# SecLang-Regel schaltet die CRS-Engine je legitimer Route ab (ctl:ruleEngine),
-# die native vpatch-Abdeckung bleibt dort erhalten. Die Exclusion-Regel muss
-# VOR crowdsecurity/crs stehen, damit ctl in Phase 1 greift.
+# ep13: vpatch + generic (nativ) plus OWASP CRS (SecLang). Die lokale Regel
+# crs-route-exclusions entfernt je legitimer Route nur die CRS-Anomalie-
+# Blockregeln (ctl:ruleRemoveById), die native vpatch-Abdeckung bleibt dort
+# erhalten. Reihenfolge zu crowdsecurity/crs ist egal (Phase 1 vor Phase 2),
+# der Uebersicht halber steht die Ausnahme oben.
 inband_rules:
   - crowdsecurity/base-config
   - selfhostedops/crs-route-exclusions

crowdsec/appsec-rules/crs-route-exclusions.yaml

@@ -1,26 +1,28 @@
 name: selfhostedops/crs-route-exclusions
-# ep13: schaltet die CRS-Coraza-Engine (SecLang) fuer genau die legitimen
-# Anwendungs-Routen ab, auf denen CRS sonst Fehlalarme wirft. ctl:ruleEngine=Off
-# betrifft nur die SecLang-Regeln (CRS + base-config), die nativen vpatch- und
-# generic-Regeln von CrowdSec laufen weiter, auch auf diesen Routen.
+# ep13: entfernt je legitimer Route NUR die CRS-Anomalie-Blockregeln (949110
+# inbound, 959100 outbound), sodass das generische CRS dort nicht mehr sperrt.
+# Die Coraza-Engine bleibt an (ruleEngine On), deshalb laufen die nativen
+# CrowdSec-vpatch-Regeln auf diesen Routen weiter (bekannte CVEs bleiben
+# geschuetzt). NICHT ctl:ruleEngine=Off benutzen, das schaltet auch vpatch ab.
+# Die Regel muss VOR crowdsecurity/crs stehen (Phase 1).
 seclang_rules:
   # Forgejo git smart-HTTP (clone/push), binaerer Pack-Body
   - |
-    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900101,phase:1,pass,nolog,chain,msg:'ep13 exclude git smart-http'"
-      SecRule REQUEST_URI "@rx (git-upload-pack|git-receive-pack|/info/refs\?service=)" "ctl:ruleEngine=Off"
-  # Forgejo REST-API: Issues, Datei-Inhalte (Nutzer schreiben Code/SQL/.env)
+    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900101,phase:1,pass,nolog,chain,msg:'ep13 crs-off git smart-http'"
+      SecRule REQUEST_URI "@rx (git-upload-pack|git-receive-pack|/info/refs\?service=)" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"
+  # Forgejo REST-API: Issues, Datei-Inhalte
   - |
-    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900102,phase:1,pass,nolog,chain,msg:'ep13 exclude forgejo api'"
-      SecRule REQUEST_URI "@beginsWith /api/v1/" "ctl:ruleEngine=Off"
-  # Outline Dokument-API (Codebloecke)
+    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900102,phase:1,pass,nolog,chain,msg:'ep13 crs-off forgejo api'"
+      SecRule REQUEST_URI "@beginsWith /api/v1/" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"
+  # Outline Dokument-API
   - |
-    SecRule REQUEST_HEADERS:Host "@beginsWith wiki." "id:900103,phase:1,pass,nolog,chain,msg:'ep13 exclude outline api'"
-      SecRule REQUEST_URI "@beginsWith /api/" "ctl:ruleEngine=Off"
-  # ntfy Publish (POST /<topic>, freier Text)
+    SecRule REQUEST_HEADERS:Host "@beginsWith wiki." "id:900103,phase:1,pass,nolog,chain,msg:'ep13 crs-off outline api'"
+      SecRule REQUEST_URI "@beginsWith /api/" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"
+  # ntfy Publish (POST /<topic>)
   - |
-    SecRule REQUEST_HEADERS:Host "@beginsWith ntfy." "id:900104,phase:1,pass,nolog,chain,msg:'ep13 exclude ntfy publish'"
-      SecRule REQUEST_METHOD "@streq POST" "ctl:ruleEngine=Off"
-  # n8n Webhook-Pfade (beliebiges JSON)
+    SecRule REQUEST_HEADERS:Host "@beginsWith ntfy." "id:900104,phase:1,pass,nolog,chain,msg:'ep13 crs-off ntfy publish'"
+      SecRule REQUEST_METHOD "@streq POST" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"
+  # n8n Webhook-Pfade
   - |
-    SecRule REQUEST_HEADERS:Host "@beginsWith n8n." "id:900105,phase:1,pass,nolog,chain,msg:'ep13 exclude n8n webhook'"
-      SecRule REQUEST_URI "@beginsWith /webhook" "ctl:ruleEngine=Off"
+    SecRule REQUEST_HEADERS:Host "@beginsWith n8n." "id:900105,phase:1,pass,nolog,chain,msg:'ep13 crs-off n8n webhook'"
+      SecRule REQUEST_URI "@beginsWith /webhook" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"