22300d7a28
Follow-up measurements showed the first exclusion approach was too broad. ctl:ruleEngine=Off disables the whole Coraza engine for the route, which also turns off the native virtual-patching rules: a .env probe on such a route returned 404 (no WAF) instead of 403. The exclusion now uses ctl:ruleRemoveById=949110,959100 (the CRS inbound and outbound anomaly-score blockers) so the engine stays on and virtual patching keeps firing on the excluded routes. Measured: .env / PHPUnit / .git/config still 403 on those routes, generic SQLi/XSS pass, non-excluded hosts keep full CRS (27 of 41). Rule order relative to the CRS rule no longer matters. Also measured and documented: RemoveInBandRuleByName in a pre_eval hook does nothing to CRS (SQLi stayed 403 globally and per route); a full authenticated Vaultwarden flow (register, login, sync, cipher create) passes under full CRS on the vault host. README traps and numbers updated accordingly.
200 lines
9.3 KiB
Markdown
200 lines
9.3 KiB
Markdown
# crowdsec/ (episode 12)
|
|
|
|
CrowdSec next to fail2ban on the same box, both reading the same logs. This
|
|
folder holds the files that differ from a stock install. Everything was
|
|
measured on the demo box; the numbers are in `HARDENING.md` (episode 12).
|
|
|
|
CrowdSec runs on the **host**, not in the stack: it reads the sshd journal
|
|
directly and drives the host firewall. Caddy talks to it over the Docker
|
|
bridge.
|
|
|
|
## Install (Debian 13)
|
|
|
|
```
|
|
curl -fsSL https://install.crowdsec.net | sh
|
|
apt install crowdsec
|
|
```
|
|
|
|
**Port 8080 trap.** The Local API defaults to `127.0.0.1:8080`. On this stack
|
|
the NetBird dashboard already publishes `127.0.0.1:8080`, so the installer
|
|
prints one line ("Not attempting to start crowdsec, port 8080 is already
|
|
used") and leaves the service **inactive**. Move it:
|
|
|
|
```
|
|
sed -i 's#listen_uri: 127.0.0.1:8080#listen_uri: 0.0.0.0:8090#' /etc/crowdsec/config.yaml
|
|
sed -i 's#url: http://127.0.0.1:8080#url: http://127.0.0.1:8090#' /etc/crowdsec/local_api_credentials.yaml
|
|
```
|
|
|
|
(`0.0.0.0` only because the Caddy container has to reach it, see below. If you
|
|
skip the Caddy bouncer, keep `127.0.0.1:8090`.)
|
|
|
|
**First start pulls nothing.** The very first community-blocklist pull ran
|
|
before the scenario list was registered ("scenario list is empty, will not
|
|
pull yet", 0 entries). A `systemctl restart crowdsec` pulled 15,000 IPs a
|
|
minute later. Check with `cscli decisions list --origin CAPI | wc -l`.
|
|
|
|
## Log sources and collections
|
|
|
|
```
|
|
cscli collections install crowdsecurity/caddy
|
|
cp acquis.d/caddy.yaml /etc/crowdsec/acquis.d/
|
|
```
|
|
|
|
The installer already adds `crowdsecurity/sshd` and a journald source for
|
|
`ssh.service`.
|
|
|
|
## Firewall bouncer
|
|
|
|
```
|
|
apt install crowdsec-firewall-bouncer-iptables
|
|
```
|
|
|
|
The package registers itself with the Local API. Then apply
|
|
`firewall-bouncer.yaml` (mode `nftables`) and restart it. Why not the
|
|
default iptables mode: it only inserts into `INPUT`. SSH is blocked, but every
|
|
Docker-published port (80, 443, 2222) stays reachable for a banned IP, the same
|
|
hole as fail2ban in episode 10. Adding `DOCKER-USER` fixes that until the next
|
|
`ufw reload`, which flushes the chain and nobody puts the rule back.
|
|
|
|
## Caddy bouncer and AppSec (WAF)
|
|
|
|
1. Build the Caddy image with the modules (`caddy/Dockerfile`), elsewhere, and
|
|
load it on the box.
|
|
2. AppSec collections, plus the rules the default config references but the
|
|
collections do not install (without them crowdsec **does not start**:
|
|
"no appsec-rules found for pattern crowdsecurity/experimental-*", then the
|
|
same for `generic-*`):
|
|
|
|
```
|
|
cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
|
|
cscli appsec-rules install crowdsecurity/experimental-no-user-agent \
|
|
crowdsecurity/generic-freemarker-ssti crowdsecurity/generic-wordpress-uploads-listing \
|
|
crowdsecurity/generic-wordpress-uploads-php
|
|
cp acquis.d/appsec.yaml /etc/crowdsec/acquis.d/
|
|
crowdsec -t && systemctl restart crowdsec
|
|
```
|
|
|
|
Run `crowdsec -t` first. A failed restart means no Local API; the firewall
|
|
keeps its current sets, but no new decisions arrive.
|
|
3. Let the Docker network in, and only that network:
|
|
|
|
```
|
|
ufw allow proto tcp from 172.18.0.0/16 to 172.17.0.1 port 8090,7422
|
|
```
|
|
|
|
The container reaches the host as `host.docker.internal` (`extra_hosts:
|
|
host-gateway` in `compose.yml`, which is `172.17.0.1`). From the internet
|
|
both ports stay filtered. The mesh interface `wt0` is allowed wholesale by
|
|
episode 5, so mesh peers can reach both; each still needs an API key.
|
|
4. `cscli bouncers add caddy-bouncer -o raw` and put the key into `.env` as
|
|
`CROWDSEC_BOUNCER_KEY`, then `docker compose up -d caddy`.
|
|
|
|
The `vpn.` site does not import the `crowdsec` snippet: it carries the NetBird
|
|
gRPC streams, and a WAF round-trip on every stream message is not what you want
|
|
between your peers.
|
|
|
|
## Replaying old logs (and the forensic-mode trap)
|
|
|
|
`crowdsec -dsn file:///path/to/log -type syslog` is documented as replay mode.
|
|
With CrowdSec 1.8.1 it shut down one second after reading the file and dropped
|
|
every event still in the pipeline: 47 or 49 alerts on two identical runs, none
|
|
after 02:07 of a 24 h log. Feeding the same file through a FIFO that stays open
|
|
60 s after the last line gives the full, repeatable result (133 alerts, twice):
|
|
|
|
```
|
|
mkfifo /tmp/replay.fifo
|
|
(cat sshd.log; sleep 60) > /tmp/replay.fifo &
|
|
crowdsec -c replay.yaml -order-event -dsn file:///tmp/replay.fifo -type syslog
|
|
```
|
|
|
|
Use a separate config (`replay.yaml`) with its own database, its own
|
|
`listen_uri` and **no `online_client`**, or the replayed alerts land in your
|
|
live database and go out as signals.
|
|
|
|
## What goes out to CrowdSec
|
|
|
|
With the default config, every alert from an unmodified hub scenario is pushed
|
|
as a signal: scenario name and hash, timestamp, your machine id, and **the
|
|
attacking IP**. Custom or modified ("tainted") scenarios and manual decisions
|
|
are not pushed unless you enroll in the console. Your own penetration tests are
|
|
real alerts too: switch `sharing: false` under `api.server.online_client`
|
|
while you attack your own box, or you report your test machine.
|
|
|
|
## Optional: OWASP CRS for generic coverage, and the tuning it needs (episode 13)
|
|
|
|
Virtual patching (the default above) only blocks known-CVE signatures. Measured
|
|
against 41 real probes: **7 blocked**. Generic SQLi, XSS, Log4Shell-via-header,
|
|
Struts OGNL and path traversal all passed, because there is no rule for them,
|
|
only per-CVE ones. `base-config` is just the Coraza body-processor setup, not
|
|
the OWASP Core Rule Set.
|
|
|
|
The CRS is in the hub as an appsec-rule. Adding it raises coverage to **27 of 41**:
|
|
|
|
```
|
|
cscli appsec-rules install crowdsecurity/crs
|
|
```
|
|
|
|
But at default paranoia the CRS blocks legitimate use. Measured false positives
|
|
with CRS on and nothing else changed: git clone/push over HTTPS (the pack
|
|
negotiation body), Forgejo issues and file contents whose text contains code or
|
|
SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook
|
|
bodies. Six normal workflows, all 403. For a stack where people push git and
|
|
write wiki pages, unusable as-is.
|
|
|
|
The fix is a per-route exclusion. `appsec-rules/crs-route-exclusions.yaml` is a
|
|
small SecLang rule that runs in phase 1 and, for exactly the legitimate
|
|
application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy
|
|
publish, n8n webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`,
|
|
removes the CRS anomaly blocking rules with
|
|
`ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100` (949110 is the CRS
|
|
inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the
|
|
request but no longer blocks it on those routes.
|
|
|
|
Do **not** use `ctl:ruleEngine=Off` for this. It turns the whole Coraza engine
|
|
off for the request, and measured on the box that also disables the native
|
|
CrowdSec virtual-patching rules: on a route excluded with `ruleEngine=Off`, a
|
|
`.env` probe returned 404 (no WAF at all) instead of 403. With
|
|
`ctl:ruleRemoveById` the engine stays on, so virtual patching keeps firing:
|
|
measured, `.env`, the PHPUnit RCE and `.git/config` still return 403 on the
|
|
excluded routes, while generic SQLi/XSS now pass (that is the point).
|
|
|
|
Rule order does not matter for `ctl:ruleRemoveById` (a phase-1 ctl removes the
|
|
phase-2 blocker regardless of file position); the exclusion sits first only for
|
|
readability. It would matter for `ruleEngine=Off`, one more reason to avoid it.
|
|
|
|
`appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule
|
|
(must come **before** crs so the ctl takes effect in phase 1), crs, then
|
|
vpatch and generic. Deploy it instead of the default:
|
|
|
|
```
|
|
cp appsec-rules/crs-route-exclusions.yaml /etc/crowdsec/appsec-rules/
|
|
cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/
|
|
# point acquis.d/appsec.yaml at selfhostedops/appsec-tuned instead of crowdsecurity/appsec-default
|
|
crowdsec -t && systemctl restart crowdsec
|
|
```
|
|
|
|
Measured with the tuned config: **0 false positives** across every workflow
|
|
above (git clone/push, Forgejo issues and file contents, Outline documents with
|
|
an attack-looking code block, ntfy publish, n8n webhooks, and a full
|
|
authenticated Vaultwarden register/login/sync/cipher-create), still **27 of 41**
|
|
probes blocked on the non-excluded routes. The cost is precise: on an excluded
|
|
route (say the Forgejo API) generic SQLi/XSS now passes to the app, while
|
|
virtual patching still catches known CVEs there, and on a non-excluded host
|
|
(vault, status) the full CRS applies. Latency added over vpatch: about +2 ms
|
|
time-to-first-byte per request. RAM: see the episode notes; CRS settles around
|
|
the same order as vpatch once loaded, roughly +15 to 40 MB.
|
|
|
|
Traps found while tuning:
|
|
- `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not**
|
|
disable CRS: the bouncer name does not match the compiled SecLang rules
|
|
(measured, globally and per route, SQLi stayed 403). Use `ctl:` inside SecLang.
|
|
Docs: https://docs.crowdsec.net/docs/appsec/hooks
|
|
- `ctl:ruleEngine=Off` is too broad: it disables virtual patching on the route
|
|
too. Use `ctl:ruleRemoveById` for the CRS blocking rules and keep the engine on.
|
|
- Every new app or new public path needs its own exclusion. A missed one blocks
|
|
a real workflow, and it fails only after login, so it is easy to miss.
|
|
|
|
**What this repo runs by default: vpatch + generic** (the `appsec-default`
|
|
config, zero false positives, no per-app tuning). The CRS files above are the
|
|
measured opt-in for anyone who wants generic coverage and will budget the
|
|
per-route tuning.
|