# crowdsec/ (episode 12) CrowdSec next to fail2ban on the same box, both reading the same logs. This folder holds the files that differ from a stock install. Everything was measured on the demo box; the numbers are in `HARDENING.md` (episode 12). CrowdSec runs on the **host**, not in the stack: it reads the sshd journal directly and drives the host firewall. Caddy talks to it over the Docker bridge. ## Install (Debian 13) ``` curl -fsSL https://install.crowdsec.net | sh apt install crowdsec ``` **Port 8080 trap.** The Local API defaults to `127.0.0.1:8080`. On this stack the NetBird dashboard already publishes `127.0.0.1:8080`, so the installer prints one line ("Not attempting to start crowdsec, port 8080 is already used") and leaves the service **inactive**. Move it: ``` sed -i 's#listen_uri: 127.0.0.1:8080#listen_uri: 0.0.0.0:8090#' /etc/crowdsec/config.yaml sed -i 's#url: http://127.0.0.1:8080#url: http://127.0.0.1:8090#' /etc/crowdsec/local_api_credentials.yaml ``` (`0.0.0.0` only because the Caddy container has to reach it, see below. If you skip the Caddy bouncer, keep `127.0.0.1:8090`.) **First start pulls nothing.** The very first community-blocklist pull ran before the scenario list was registered ("scenario list is empty, will not pull yet", 0 entries). A `systemctl restart crowdsec` pulled 15,000 IPs a minute later. Check with `cscli decisions list --origin CAPI | wc -l`. ## Log sources and collections ``` cscli collections install crowdsecurity/caddy cp acquis.d/caddy.yaml /etc/crowdsec/acquis.d/ ``` The installer already adds `crowdsecurity/sshd` and a journald source for `ssh.service`. ## Firewall bouncer ``` apt install crowdsec-firewall-bouncer-iptables ``` The package registers itself with the Local API. Then apply `firewall-bouncer.yaml` (mode `nftables`) and restart it. Why not the default iptables mode: it only inserts into `INPUT`. SSH is blocked, but every Docker-published port (80, 443, 2222) stays reachable for a banned IP, the same hole as fail2ban in episode 10. Adding `DOCKER-USER` fixes that until the next `ufw reload`, which flushes the chain and nobody puts the rule back. ## Caddy bouncer and AppSec (WAF) 1. Build the Caddy image with the modules (`caddy/Dockerfile`), elsewhere, and load it on the box. 2. AppSec collections, plus the rules the default config references but the collections do not install (without them crowdsec **does not start**: "no appsec-rules found for pattern crowdsecurity/experimental-*", then the same for `generic-*`): ``` cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules cscli appsec-rules install crowdsecurity/experimental-no-user-agent \ crowdsecurity/generic-freemarker-ssti crowdsecurity/generic-wordpress-uploads-listing \ crowdsecurity/generic-wordpress-uploads-php cp acquis.d/appsec.yaml /etc/crowdsec/acquis.d/ crowdsec -t && systemctl restart crowdsec ``` Run `crowdsec -t` first. A failed restart means no Local API; the firewall keeps its current sets, but no new decisions arrive. 3. Let the Docker network in, and only that network: ``` ufw allow proto tcp from 172.18.0.0/16 to 172.17.0.1 port 8090,7422 ``` The container reaches the host as `host.docker.internal` (`extra_hosts: host-gateway` in `compose.yml`, which is `172.17.0.1`). From the internet both ports stay filtered. The mesh interface `wt0` is allowed wholesale by episode 5, so mesh peers can reach both; each still needs an API key. 4. `cscli bouncers add caddy-bouncer -o raw` and put the key into `.env` as `CROWDSEC_BOUNCER_KEY`, then `docker compose up -d caddy`. The `vpn.` site does not import the `crowdsec` snippet: it carries the NetBird gRPC streams, and a WAF round-trip on every stream message is not what you want between your peers. ## Replaying old logs (and the forensic-mode trap) `crowdsec -dsn file:///path/to/log -type syslog` is documented as replay mode. With CrowdSec 1.8.1 it shut down one second after reading the file and dropped every event still in the pipeline: 47 or 49 alerts on two identical runs, none after 02:07 of a 24 h log. Feeding the same file through a FIFO that stays open 60 s after the last line gives the full, repeatable result (133 alerts, twice): ``` mkfifo /tmp/replay.fifo (cat sshd.log; sleep 60) > /tmp/replay.fifo & crowdsec -c replay.yaml -order-event -dsn file:///tmp/replay.fifo -type syslog ``` Use a separate config (`replay.yaml`) with its own database, its own `listen_uri` and **no `online_client`**, or the replayed alerts land in your live database and go out as signals. ## What goes out to CrowdSec With the default config, every alert from an unmodified hub scenario is pushed as a signal: scenario name and hash, timestamp, your machine id, and **the attacking IP**. Custom or modified ("tainted") scenarios and manual decisions are not pushed unless you enroll in the console. Your own penetration tests are real alerts too: switch `sharing: false` under `api.server.online_client` while you attack your own box, or you report your test machine. ## Optional: OWASP CRS for generic coverage, and the tuning it needs (episode 13) Virtual patching (the default above) only blocks known-CVE signatures. Measured against 41 real probes: **7 blocked**. Generic SQLi, XSS, Log4Shell-via-header, Struts OGNL and path traversal all passed, because there is no rule for them, only per-CVE ones. `base-config` is just the Coraza body-processor setup, not the OWASP Core Rule Set. The CRS is in the hub as an appsec-rule. Adding it raises coverage to **27 of 41**: ``` cscli appsec-rules install crowdsecurity/crs ``` But at default paranoia the CRS blocks legitimate use. Measured false positives with CRS on and nothing else changed: git clone/push over HTTPS (the pack negotiation body), Forgejo issues and file contents whose text contains code or SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook bodies. Six normal workflows, all 403. For a stack where people push git and write wiki pages, unusable as-is. The fix is a per-route exclusion. `appsec-rules/crs-route-exclusions.yaml` is a small SecLang rule that runs in phase 1 and, for exactly the legitimate application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy publish, n8n webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`, removes the CRS anomaly blocking rules with `ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100` (949110 is the CRS inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the request but no longer blocks it on those routes. Do **not** use `ctl:ruleEngine=Off` for this. It turns the whole Coraza engine off for the request, and measured on the box that also disables the native CrowdSec virtual-patching rules: on a route excluded with `ruleEngine=Off`, a `.env` probe returned 404 (no WAF at all) instead of 403. With `ctl:ruleRemoveById` the engine stays on, so virtual patching keeps firing: measured, `.env`, the PHPUnit RCE and `.git/config` still return 403 on the excluded routes, while generic SQLi/XSS now pass (that is the point). Rule order does not matter for `ctl:ruleRemoveById` (a phase-1 ctl removes the phase-2 blocker regardless of file position); the exclusion sits first only for readability. It would matter for `ruleEngine=Off`, one more reason to avoid it. `appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule (must come **before** crs so the ctl takes effect in phase 1), crs, then vpatch and generic. Deploy it instead of the default: ``` cp appsec-rules/crs-route-exclusions.yaml /etc/crowdsec/appsec-rules/ cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/ # point acquis.d/appsec.yaml at selfhostedops/appsec-tuned instead of crowdsecurity/appsec-default crowdsec -t && systemctl restart crowdsec ``` Measured with the tuned config: **0 false positives** across every workflow above (git clone/push, Forgejo issues and file contents, Outline documents with an attack-looking code block, ntfy publish, n8n webhooks, and a full authenticated Vaultwarden register/login/sync/cipher-create), still **27 of 41** probes blocked on the non-excluded routes. The cost is precise: on an excluded route (say the Forgejo API) generic SQLi/XSS now passes to the app, while virtual patching still catches known CVEs there, and on a non-excluded host (vault, status) the full CRS applies. Latency added over vpatch: about +2 ms time-to-first-byte per request. RAM measured settled: +9 MB over vpatch after five minutes (vpatch ~184 MB, CRS+exclusions ~193 MB). The ~25-45 MB people cite is a load-time peak, not the steady cost. Traps found while tuning: - `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not** disable CRS: the bouncer name does not match the compiled SecLang rules (measured, globally and per route, SQLi stayed 403). Use `ctl:` inside SecLang. Docs: https://docs.crowdsec.net/docs/appsec/hooks - `ctl:ruleEngine=Off` is too broad: it disables virtual patching on the route too. Use `ctl:ruleRemoveById` for the CRS blocking rules and keep the engine on. - Every new app or new public path needs its own exclusion. A missed one blocks a real workflow, and it fails only after login, so it is easy to miss. **What this repo runs by default: vpatch + generic** (the `appsec-default` config, zero false positives, no per-app tuning). The CRS files above are the measured opt-in for anyone who wants generic coverage and will budget the per-route tuning.