Episode 13: tune the OWASP CRS with per-route exclusions

Measured follow-up to the AppSec section. Virtual patching blocks 7 of 41
probes with near-zero false positives; the OWASP CRS raises that to 27 of 41
but, at default paranoia, blocks six legitimate workflows (git push, Forgejo
issues and file contents, Outline documents, ntfy publish, n8n webhooks).

crowdsec/appsec-rules/crs-route-exclusions.yaml disables the SecLang/Coraza
engine (ctl:ruleEngine=Off) for exactly those application routes, matched by
host + URI, so virtual patching still runs there. crowdsec/appsec-configs/
appsec-tuned.yaml wires it in before the CRS rule. Measured with tuning: zero
false positives, still 27 of 41 blocked on non-excluded routes, about +2 ms
per request, ~15-40 MB more RAM.

Trap: RemoveInBandRuleByName in a pre_eval hook does not disable CRS (it is
SecLang, not a native rule); ctl:ruleEngine in phase 1 is the working lever,
and the exclusion rule must be listed before the CRS rule.

The stack runs vpatch+generic by default (zero false positives, no per-app
tuning). The CRS recipe ships as a documented opt-in in crowdsec/README.md.

crowdsec/README.md

@@ -119,3 +119,70 @@ attacking IP**. Custom or modified ("tainted") scenarios and manual decisions
 are not pushed unless you enroll in the console. Your own penetration tests are
 real alerts too: switch `sharing: false` under `api.server.online_client`
 while you attack your own box, or you report your test machine.
+
+## Optional: OWASP CRS for generic coverage, and the tuning it needs (episode 13)
+
+Virtual patching (the default above) only blocks known-CVE signatures. Measured
+against 41 real probes: **7 blocked**. Generic SQLi, XSS, Log4Shell-via-header,
+Struts OGNL and path traversal all passed, because there is no rule for them,
+only per-CVE ones. `base-config` is just the Coraza body-processor setup, not
+the OWASP Core Rule Set.
+
+The CRS is in the hub as an appsec-rule. Adding it raises coverage to **27 of 41**:
+
+```
+cscli appsec-rules install crowdsecurity/crs
+```
+
+But at default paranoia the CRS blocks legitimate use. Measured false positives
+with CRS on and nothing else changed: git clone/push over HTTPS (the pack
+negotiation body), Forgejo issues and file contents whose text contains code or
+SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook
+bodies. Six normal workflows, all 403. For a stack where people push git and
+write wiki pages, unusable as-is.
+
+The fix is per-route exclusion, not turning CRS off globally.
+`appsec-rules/crs-route-exclusions.yaml` is a small SecLang rule that runs in
+phase 1 and sets `ctl:ruleEngine=Off` for exactly the legitimate application
+routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy publish, n8n
+webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`. `ctl:ruleEngine`
+only affects the Coraza/SecLang engine, so the native virtual-patching and
+generic rules still run on those routes; only the generic CRS body/URL
+inspection is skipped there.
+
+`appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule
+(must come **before** crs so the ctl takes effect in phase 1), crs, then
+vpatch and generic. Deploy it instead of the default:
+
+```
+cp appsec-rules/crs-route-exclusions.yaml /etc/crowdsec/appsec-rules/
+cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/
+# point acquis.d/appsec.yaml at selfhostedops/appsec-tuned instead of crowdsecurity/appsec-default
+crowdsec -t && systemctl restart crowdsec
+```
+
+Measured with the tuned config: **0 false positives** across all the workflows
+above, still **27 of 41** probes blocked on the non-excluded routes. The cost is
+precise and worth stating out loud: on an excluded route (say the Forgejo API),
+generic SQLi/XSS now passes to the app (virtual patching still covers known
+CVEs); on a non-excluded host (vault, status) the full CRS applies. Latency
+added over vpatch: about +2 ms time-to-first-byte per request. RAM: CRS adds
+roughly 15 to 40 MB over vpatch (RSS settles near 185 MB, peaks higher right
+after loading the rule data).
+
+Traps found while tuning:
+- `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not**
+  disable CRS: the bouncer name does not match the compiled SecLang rules. Use
+  `ctl:ruleEngine` inside SecLang instead.
+- The exclusion rule must be listed before `crowdsecurity/crs` in the config, or
+  the phase-1 `ctl` runs too late.
+- Every new app or new public path needs its own exclusion. A missed one blocks
+  a real workflow, and it fails only after login, so it is easy to miss.
+- Authenticated Vaultwarden and Outline sync with real encrypted payloads were
+  not exercised against CRS here; test your own before trusting CRS on those
+  hosts.
+
+**What this repo runs by default: vpatch + generic** (the `appsec-default`
+config, zero false positives, no per-app tuning). The CRS files above are the
+measured opt-in for anyone who wants generic coverage and will budget the
+per-route tuning.

crowdsec/appsec-configs/appsec-tuned.yaml

@@ -0,0 +1,15 @@
+name: selfhostedops/appsec-tuned
+default_remediation: ban
+# ep13: vpatch + generic (nativ) plus OWASP CRS (SecLang). Eine lokale
+# SecLang-Regel schaltet die CRS-Engine je legitimer Route ab (ctl:ruleEngine),
+# die native vpatch-Abdeckung bleibt dort erhalten. Die Exclusion-Regel muss
+# VOR crowdsecurity/crs stehen, damit ctl in Phase 1 greift.
+inband_rules:
+  - crowdsecurity/base-config
+  - selfhostedops/crs-route-exclusions
+  - crowdsecurity/crs
+  - crowdsecurity/vpatch-*
+  - crowdsecurity/generic-*
+outofband_rules:
+  - crowdsecurity/experimental-*
+  - crowdsecurity/appsec-generic-test

crowdsec/appsec-rules/crs-route-exclusions.yaml

@@ -0,0 +1,26 @@
+name: selfhostedops/crs-route-exclusions
+# ep13: schaltet die CRS-Coraza-Engine (SecLang) fuer genau die legitimen
+# Anwendungs-Routen ab, auf denen CRS sonst Fehlalarme wirft. ctl:ruleEngine=Off
+# betrifft nur die SecLang-Regeln (CRS + base-config), die nativen vpatch- und
+# generic-Regeln von CrowdSec laufen weiter, auch auf diesen Routen.
+seclang_rules:
+  # Forgejo git smart-HTTP (clone/push), binaerer Pack-Body
+  - |
+    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900101,phase:1,pass,nolog,chain,msg:'ep13 exclude git smart-http'"
+      SecRule REQUEST_URI "@rx (git-upload-pack|git-receive-pack|/info/refs\?service=)" "ctl:ruleEngine=Off"
+  # Forgejo REST-API: Issues, Datei-Inhalte (Nutzer schreiben Code/SQL/.env)
+  - |
+    SecRule REQUEST_HEADERS:Host "@beginsWith git." "id:900102,phase:1,pass,nolog,chain,msg:'ep13 exclude forgejo api'"
+      SecRule REQUEST_URI "@beginsWith /api/v1/" "ctl:ruleEngine=Off"
+  # Outline Dokument-API (Codebloecke)
+  - |
+    SecRule REQUEST_HEADERS:Host "@beginsWith wiki." "id:900103,phase:1,pass,nolog,chain,msg:'ep13 exclude outline api'"
+      SecRule REQUEST_URI "@beginsWith /api/" "ctl:ruleEngine=Off"
+  # ntfy Publish (POST /<topic>, freier Text)
+  - |
+    SecRule REQUEST_HEADERS:Host "@beginsWith ntfy." "id:900104,phase:1,pass,nolog,chain,msg:'ep13 exclude ntfy publish'"
+      SecRule REQUEST_METHOD "@streq POST" "ctl:ruleEngine=Off"
+  # n8n Webhook-Pfade (beliebiges JSON)
+  - |
+    SecRule REQUEST_HEADERS:Host "@beginsWith n8n." "id:900105,phase:1,pass,nolog,chain,msg:'ep13 exclude n8n webhook'"
+      SecRule REQUEST_URI "@beginsWith /webhook" "ctl:ruleEngine=Off"