selfhostedops-stack/crowdsec/README.md

crowdsec/ (episode 12)

CrowdSec next to fail2ban on the same box, both reading the same logs. This folder holds the files that differ from a stock install. Everything was measured on the demo box; the numbers are in HARDENING.md (episode 12).

CrowdSec runs on the host, not in the stack: it reads the sshd journal directly and drives the host firewall. Caddy talks to it over the Docker bridge.

Install (Debian 13)

curl -fsSL https://install.crowdsec.net | sh
apt install crowdsec

Port 8080 trap. The Local API defaults to 127.0.0.1:8080. On this stack the NetBird dashboard already publishes 127.0.0.1:8080, so the installer prints one line ("Not attempting to start crowdsec, port 8080 is already used") and leaves the service inactive. Move it:

sed -i 's#listen_uri: 127.0.0.1:8080#listen_uri: 0.0.0.0:8090#' /etc/crowdsec/config.yaml
sed -i 's#url: http://127.0.0.1:8080#url: http://127.0.0.1:8090#' /etc/crowdsec/local_api_credentials.yaml

(0.0.0.0 only because the Caddy container has to reach it, see below. If you skip the Caddy bouncer, keep 127.0.0.1:8090.)

First start pulls nothing. The very first community-blocklist pull ran before the scenario list was registered ("scenario list is empty, will not pull yet", 0 entries). A systemctl restart crowdsec pulled 15,000 IPs a minute later. Check with cscli decisions list --origin CAPI | wc -l.

Log sources and collections

cscli collections install crowdsecurity/caddy
cp acquis.d/caddy.yaml /etc/crowdsec/acquis.d/

The installer already adds crowdsecurity/sshd and a journald source for ssh.service.

Firewall bouncer

apt install crowdsec-firewall-bouncer-iptables

The package registers itself with the Local API. Then apply firewall-bouncer.yaml (mode nftables) and restart it. Why not the default iptables mode: it only inserts into INPUT. SSH is blocked, but every Docker-published port (80, 443, 2222) stays reachable for a banned IP, the same hole as fail2ban in episode 10. Adding DOCKER-USER fixes that until the next ufw reload, which flushes the chain and nobody puts the rule back.

Caddy bouncer and AppSec (WAF)

1. Build the Caddy image with the modules (caddy/Dockerfile), elsewhere, and load it on the box.

2. AppSec collections, plus the rules the default config references but the collections do not install (without them crowdsec does not start: "no appsec-rules found for pattern crowdsecurity/experimental-*", then the same for generic-*):

   cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
   cscli appsec-rules install crowdsecurity/experimental-no-user-agent \
     crowdsecurity/generic-freemarker-ssti crowdsecurity/generic-wordpress-uploads-listing \
     crowdsecurity/generic-wordpress-uploads-php
   cp acquis.d/appsec.yaml /etc/crowdsec/acquis.d/
   crowdsec -t && systemctl restart crowdsec
   

   Run crowdsec -t first. A failed restart means no Local API; the firewall keeps its current sets, but no new decisions arrive.

3. Let the Docker network in, and only that network:

   ufw allow proto tcp from 172.18.0.0/16 to 172.17.0.1 port 8090,7422
   

   The container reaches the host as host.docker.internal (extra_hosts: host-gateway in compose.yml, which is 172.17.0.1). From the internet both ports stay filtered. The mesh interface wt0 is allowed wholesale by episode 5, so mesh peers can reach both; each still needs an API key.

4. cscli bouncers add caddy-bouncer -o raw and put the key into .env as CROWDSEC_BOUNCER_KEY, then docker compose up -d caddy.

The vpn. site does not import the crowdsec snippet: it carries the NetBird gRPC streams, and a WAF round-trip on every stream message is not what you want between your peers.

Replaying old logs (and the forensic-mode trap)

crowdsec -dsn file:///path/to/log -type syslog is documented as replay mode. With CrowdSec 1.8.1 it shut down one second after reading the file and dropped every event still in the pipeline: 47 or 49 alerts on two identical runs, none after 02:07 of a 24 h log. Feeding the same file through a FIFO that stays open 60 s after the last line gives the full, repeatable result (133 alerts, twice):

mkfifo /tmp/replay.fifo
(cat sshd.log; sleep 60) > /tmp/replay.fifo &
crowdsec -c replay.yaml -order-event -dsn file:///tmp/replay.fifo -type syslog

Use a separate config (replay.yaml) with its own database, its own listen_uri and no online_client, or the replayed alerts land in your live database and go out as signals.

What goes out to CrowdSec

With the default config, every alert from an unmodified hub scenario is pushed as a signal: scenario name and hash, timestamp, your machine id, and the attacking IP. Custom or modified ("tainted") scenarios and manual decisions are not pushed unless you enroll in the console. Your own penetration tests are real alerts too: switch sharing: false under api.server.online_client while you attack your own box, or you report your test machine.

Optional: OWASP CRS for generic coverage, and the tuning it needs (episode 13)

Virtual patching (the default above) only blocks known-CVE signatures. Measured against 41 real probes: 7 blocked. Generic SQLi, XSS, Log4Shell-via-header, Struts OGNL and path traversal all passed, because there is no rule for them, only per-CVE ones. base-config is just the Coraza body-processor setup, not the OWASP Core Rule Set.

The CRS is in the hub as an appsec-rule. Adding it raises coverage to 27 of 41:

cscli appsec-rules install crowdsecurity/crs

But at default paranoia the CRS blocks legitimate use. Measured false positives with CRS on and nothing else changed: git clone/push over HTTPS (the pack negotiation body), Forgejo issues and file contents whose text contains code or SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook bodies. Six normal workflows, all 403. For a stack where people push git and write wiki pages, unusable as-is.

The fix is a per-route exclusion. appsec-rules/crs-route-exclusions.yaml is a small SecLang rule that runs in phase 1 and, for exactly the legitimate application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy publish, n8n webhooks), matched by REQUEST_HEADERS:Host + REQUEST_URI, removes the CRS anomaly blocking rules with ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100 (949110 is the CRS inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the request but no longer blocks it on those routes.

Do not use ctl:ruleEngine=Off for this. It turns the whole Coraza engine off for the request, and measured on the box that also disables the native CrowdSec virtual-patching rules: on a route excluded with ruleEngine=Off, a .env probe returned 404 (no WAF at all) instead of 403. With ctl:ruleRemoveById the engine stays on, so virtual patching keeps firing: measured, .env, the PHPUnit RCE and .git/config still return 403 on the excluded routes, while generic SQLi/XSS now pass (that is the point).

Rule order does not matter for ctl:ruleRemoveById (a phase-1 ctl removes the phase-2 blocker regardless of file position); the exclusion sits first only for readability. It would matter for ruleEngine=Off, one more reason to avoid it.

appsec-configs/appsec-tuned.yaml wires it up: base-config, the exclusion rule (must come before crs so the ctl takes effect in phase 1), crs, then vpatch and generic. Deploy it instead of the default:

cp appsec-rules/crs-route-exclusions.yaml /etc/crowdsec/appsec-rules/
cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/
# point acquis.d/appsec.yaml at selfhostedops/appsec-tuned instead of crowdsecurity/appsec-default
crowdsec -t && systemctl restart crowdsec

Measured with the tuned config: 0 false positives across every workflow above (git clone/push, Forgejo issues and file contents, Outline documents with an attack-looking code block, ntfy publish, n8n webhooks, and a full authenticated Vaultwarden register/login/sync/cipher-create), still 27 of 41 probes blocked on the non-excluded routes. The cost is precise: on an excluded route (say the Forgejo API) generic SQLi/XSS now passes to the app, while virtual patching still catches known CVEs there, and on a non-excluded host (vault, status) the full CRS applies. Latency added over vpatch: about +2 ms time-to-first-byte per request. RAM measured settled: +9 MB over vpatch after five minutes (vpatch ~184 MB, CRS+exclusions ~193 MB). The ~25-45 MB people cite is a load-time peak, not the steady cost.

Traps found while tuning:

• RemoveInBandRuleByName("crowdsecurity/crs") in a pre_eval hook does not disable CRS: the bouncer name does not match the compiled SecLang rules (measured, globally and per route, SQLi stayed 403). Use ctl: inside SecLang. Docs: https://docs.crowdsec.net/docs/appsec/hooks
• ctl:ruleEngine=Off is too broad: it disables virtual patching on the route too. Use ctl:ruleRemoveById for the CRS blocking rules and keep the engine on.
• Every new app or new public path needs its own exclusion. A missed one blocks a real workflow, and it fails only after login, so it is easy to miss.

Episode 14: narrower exclusions, and who drafts them

appsec-rules/crs-route-exclusions-by-rule.yaml is the finer variant: per route it removes only the rule ids that fired on legitimate requests there, instead of the blocking rule 949110. Measured on the same box: all false positives gone (including git push, which hides behind the failing clone and never shows up in the log on its own), 18 of 18 attacks still 403, virtual patching alive, and SQL injection on the exempted git API still 403, because no SQLi rule had fired on that route. To use it, list selfhostedops/crs-route-exclusions-by-rule in appsec-tuned.yaml instead of selfhostedops/crs-route-exclusions.

The rule ids per blocked request are not in crowdsec.log. They are in the alerts (cscli alerts list -o json, kind waf). ops-readonly/waf-export.py dumps them for the read-only tool, ops-readonly/waf-propose.py turns them into a proposal, and workflow G (automation/README.md) lets a local model try the same. Read HARDENING.md (episode 14) before you let the model do it.

Two things to know before you test this on your own box:

• With the CRS on and no exclusions, four false positives within two seconds banned the legitimate user for four hours (crowdsecurity/appsec-native plus LePresidente/http-generic-403-bf). Whitelist your own test source first.
• Do not run cscli alerts delete right after cscli decisions delete. The decision is gone from the database before the bouncers have seen it go, and the IP stays blocked until you restart them.

What this repo runs by default: vpatch + generic (the appsec-default config, zero false positives, no per-app tuning). The CRS files above are the measured opt-in for anyone who wants generic coverage and will budget the per-route tuning.