From 22300d7a288a9aeda6975ebd35f4124b66ecf572 Mon Sep 17 00:00:00 2001 From: Maximilian Grimm Date: Wed, 16 Sep 2026 21:06:32 +0200 Subject: [PATCH] Episode 13: correct the CRS exclusion mechanism (ruleRemoveById, not ruleEngine) Follow-up measurements showed the first exclusion approach was too broad. ctl:ruleEngine=Off disables the whole Coraza engine for the route, which also turns off the native virtual-patching rules: a .env probe on such a route returned 404 (no WAF) instead of 403. The exclusion now uses ctl:ruleRemoveById=949110,959100 (the CRS inbound and outbound anomaly-score blockers) so the engine stays on and virtual patching keeps firing on the excluded routes. Measured: .env / PHPUnit / .git/config still 403 on those routes, generic SQLi/XSS pass, non-excluded hosts keep full CRS (27 of 41). Rule order relative to the CRS rule no longer matters. Also measured and documented: RemoveInBandRuleByName in a pre_eval hook does nothing to CRS (SQLi stayed 403 globally and per route); a full authenticated Vaultwarden flow (register, login, sync, cipher create) passes under full CRS on the vault host. README traps and numbers updated accordingly. --- crowdsec/README.md | 58 +++++++++++-------- crowdsec/appsec-configs/appsec-tuned.yaml | 9 +-- .../appsec-rules/crs-route-exclusions.yaml | 38 ++++++------ 3 files changed, 60 insertions(+), 45 deletions(-) diff --git a/crowdsec/README.md b/crowdsec/README.md index a52884e..c1fba29 100644 --- a/crowdsec/README.md +++ b/crowdsec/README.md @@ -141,14 +141,26 @@ SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook bodies. Six normal workflows, all 403. For a stack where people push git and write wiki pages, unusable as-is. -The fix is per-route exclusion, not turning CRS off globally. -`appsec-rules/crs-route-exclusions.yaml` is a small SecLang rule that runs in -phase 1 and sets `ctl:ruleEngine=Off` for exactly the legitimate application -routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy publish, n8n -webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`. `ctl:ruleEngine` -only affects the Coraza/SecLang engine, so the native virtual-patching and -generic rules still run on those routes; only the generic CRS body/URL -inspection is skipped there. +The fix is a per-route exclusion. `appsec-rules/crs-route-exclusions.yaml` is a +small SecLang rule that runs in phase 1 and, for exactly the legitimate +application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy +publish, n8n webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`, +removes the CRS anomaly blocking rules with +`ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100` (949110 is the CRS +inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the +request but no longer blocks it on those routes. + +Do **not** use `ctl:ruleEngine=Off` for this. It turns the whole Coraza engine +off for the request, and measured on the box that also disables the native +CrowdSec virtual-patching rules: on a route excluded with `ruleEngine=Off`, a +`.env` probe returned 404 (no WAF at all) instead of 403. With +`ctl:ruleRemoveById` the engine stays on, so virtual patching keeps firing: +measured, `.env`, the PHPUnit RCE and `.git/config` still return 403 on the +excluded routes, while generic SQLi/XSS now pass (that is the point). + +Rule order does not matter for `ctl:ruleRemoveById` (a phase-1 ctl removes the +phase-2 blocker regardless of file position); the exclusion sits first only for +readability. It would matter for `ruleEngine=Off`, one more reason to avoid it. `appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule (must come **before** crs so the ctl takes effect in phase 1), crs, then @@ -161,26 +173,26 @@ cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/ crowdsec -t && systemctl restart crowdsec ``` -Measured with the tuned config: **0 false positives** across all the workflows -above, still **27 of 41** probes blocked on the non-excluded routes. The cost is -precise and worth stating out loud: on an excluded route (say the Forgejo API), -generic SQLi/XSS now passes to the app (virtual patching still covers known -CVEs); on a non-excluded host (vault, status) the full CRS applies. Latency -added over vpatch: about +2 ms time-to-first-byte per request. RAM: CRS adds -roughly 15 to 40 MB over vpatch (RSS settles near 185 MB, peaks higher right -after loading the rule data). +Measured with the tuned config: **0 false positives** across every workflow +above (git clone/push, Forgejo issues and file contents, Outline documents with +an attack-looking code block, ntfy publish, n8n webhooks, and a full +authenticated Vaultwarden register/login/sync/cipher-create), still **27 of 41** +probes blocked on the non-excluded routes. The cost is precise: on an excluded +route (say the Forgejo API) generic SQLi/XSS now passes to the app, while +virtual patching still catches known CVEs there, and on a non-excluded host +(vault, status) the full CRS applies. Latency added over vpatch: about +2 ms +time-to-first-byte per request. RAM: see the episode notes; CRS settles around +the same order as vpatch once loaded, roughly +15 to 40 MB. Traps found while tuning: - `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not** - disable CRS: the bouncer name does not match the compiled SecLang rules. Use - `ctl:ruleEngine` inside SecLang instead. -- The exclusion rule must be listed before `crowdsecurity/crs` in the config, or - the phase-1 `ctl` runs too late. + disable CRS: the bouncer name does not match the compiled SecLang rules + (measured, globally and per route, SQLi stayed 403). Use `ctl:` inside SecLang. + Docs: https://docs.crowdsec.net/docs/appsec/hooks +- `ctl:ruleEngine=Off` is too broad: it disables virtual patching on the route + too. Use `ctl:ruleRemoveById` for the CRS blocking rules and keep the engine on. - Every new app or new public path needs its own exclusion. A missed one blocks a real workflow, and it fails only after login, so it is easy to miss. -- Authenticated Vaultwarden and Outline sync with real encrypted payloads were - not exercised against CRS here; test your own before trusting CRS on those - hosts. **What this repo runs by default: vpatch + generic** (the `appsec-default` config, zero false positives, no per-app tuning). The CRS files above are the diff --git a/crowdsec/appsec-configs/appsec-tuned.yaml b/crowdsec/appsec-configs/appsec-tuned.yaml index 0b1ceed..3c04a75 100644 --- a/crowdsec/appsec-configs/appsec-tuned.yaml +++ b/crowdsec/appsec-configs/appsec-tuned.yaml @@ -1,9 +1,10 @@ name: selfhostedops/appsec-tuned default_remediation: ban -# ep13: vpatch + generic (nativ) plus OWASP CRS (SecLang). Eine lokale -# SecLang-Regel schaltet die CRS-Engine je legitimer Route ab (ctl:ruleEngine), -# die native vpatch-Abdeckung bleibt dort erhalten. Die Exclusion-Regel muss -# VOR crowdsecurity/crs stehen, damit ctl in Phase 1 greift. +# ep13: vpatch + generic (nativ) plus OWASP CRS (SecLang). Die lokale Regel +# crs-route-exclusions entfernt je legitimer Route nur die CRS-Anomalie- +# Blockregeln (ctl:ruleRemoveById), die native vpatch-Abdeckung bleibt dort +# erhalten. Reihenfolge zu crowdsecurity/crs ist egal (Phase 1 vor Phase 2), +# der Uebersicht halber steht die Ausnahme oben. inband_rules: - crowdsecurity/base-config - selfhostedops/crs-route-exclusions diff --git a/crowdsec/appsec-rules/crs-route-exclusions.yaml b/crowdsec/appsec-rules/crs-route-exclusions.yaml index d20e006..64789f2 100644 --- a/crowdsec/appsec-rules/crs-route-exclusions.yaml +++ b/crowdsec/appsec-rules/crs-route-exclusions.yaml @@ -1,26 +1,28 @@ name: selfhostedops/crs-route-exclusions -# ep13: schaltet die CRS-Coraza-Engine (SecLang) fuer genau die legitimen -# Anwendungs-Routen ab, auf denen CRS sonst Fehlalarme wirft. ctl:ruleEngine=Off -# betrifft nur die SecLang-Regeln (CRS + base-config), die nativen vpatch- und -# generic-Regeln von CrowdSec laufen weiter, auch auf diesen Routen. +# ep13: entfernt je legitimer Route NUR die CRS-Anomalie-Blockregeln (949110 +# inbound, 959100 outbound), sodass das generische CRS dort nicht mehr sperrt. +# Die Coraza-Engine bleibt an (ruleEngine On), deshalb laufen die nativen +# CrowdSec-vpatch-Regeln auf diesen Routen weiter (bekannte CVEs bleiben +# geschuetzt). NICHT ctl:ruleEngine=Off benutzen, das schaltet auch vpatch ab. +# Die Regel muss VOR crowdsecurity/crs stehen (Phase 1). seclang_rules: # Forgejo git smart-HTTP (clone/push), binaerer Pack-Body - | - SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900101,phase:1,pass,nolog,chain,msg:'ep13 exclude git smart-http'" - SecRule REQUEST_URI "@rx (git-upload-pack|git-receive-pack|/info/refs\?service=)" "ctl:ruleEngine=Off" - # Forgejo REST-API: Issues, Datei-Inhalte (Nutzer schreiben Code/SQL/.env) + SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900101,phase:1,pass,nolog,chain,msg:'ep13 crs-off git smart-http'" + SecRule REQUEST_URI "@rx (git-upload-pack|git-receive-pack|/info/refs\?service=)" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100" + # Forgejo REST-API: Issues, Datei-Inhalte - | - SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900102,phase:1,pass,nolog,chain,msg:'ep13 exclude forgejo api'" - SecRule REQUEST_URI "@beginsWith /api/v1/" "ctl:ruleEngine=Off" - # Outline Dokument-API (Codebloecke) + SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900102,phase:1,pass,nolog,chain,msg:'ep13 crs-off forgejo api'" + SecRule REQUEST_URI "@beginsWith /api/v1/" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100" + # Outline Dokument-API - | - SecRule REQUEST_HEADERS:Host "@beginsWith wiki." "id:900103,phase:1,pass,nolog,chain,msg:'ep13 exclude outline api'" - SecRule REQUEST_URI "@beginsWith /api/" "ctl:ruleEngine=Off" - # ntfy Publish (POST /, freier Text) + SecRule REQUEST_HEADERS:Host "@beginsWith wiki." "id:900103,phase:1,pass,nolog,chain,msg:'ep13 crs-off outline api'" + SecRule REQUEST_URI "@beginsWith /api/" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100" + # ntfy Publish (POST /) - | - SecRule REQUEST_HEADERS:Host "@beginsWith ntfy." "id:900104,phase:1,pass,nolog,chain,msg:'ep13 exclude ntfy publish'" - SecRule REQUEST_METHOD "@streq POST" "ctl:ruleEngine=Off" - # n8n Webhook-Pfade (beliebiges JSON) + SecRule REQUEST_HEADERS:Host "@beginsWith ntfy." "id:900104,phase:1,pass,nolog,chain,msg:'ep13 crs-off ntfy publish'" + SecRule REQUEST_METHOD "@streq POST" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100" + # n8n Webhook-Pfade - | - SecRule REQUEST_HEADERS:Host "@beginsWith n8n." "id:900105,phase:1,pass,nolog,chain,msg:'ep13 exclude n8n webhook'" - SecRule REQUEST_URI "@beginsWith /webhook" "ctl:ruleEngine=Off" + SecRule REQUEST_HEADERS:Host "@beginsWith n8n." "id:900105,phase:1,pass,nolog,chain,msg:'ep13 crs-off n8n webhook'" + SecRule REQUEST_URI "@beginsWith /webhook" "ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100"