199 lines
9.4 KiB
Markdown
199 lines
9.4 KiB
Markdown
# crowdsec/ (episode 12)
|
|
|
|
CrowdSec next to fail2ban on the same box, both reading the same logs. This
|
|
folder holds the files that differ from a stock install. Everything was
|
|
measured on the demo box; the numbers are in `HARDENING.md` (episode 12).
|
|
|
|
CrowdSec runs on the **host**, not in the stack: it reads the sshd journal
|
|
directly and drives the host firewall. Caddy talks to it over the Docker
|
|
bridge.
|
|
|
|
## Install (Debian 13)
|
|
|
|
```
|
|
curl -fsSL https://install.crowdsec.net | sh
|
|
apt install crowdsec
|
|
```
|
|
|
|
**Port 8080 trap.** The Local API defaults to `127.0.0.1:8080`. On this stack
|
|
the NetBird dashboard already publishes `127.0.0.1:8080`, so the installer
|
|
prints one line ("Not attempting to start crowdsec, port 8080 is already
|
|
used") and leaves the service **inactive**. Move it:
|
|
|
|
```
|
|
sed -i 's#listen_uri: 127.0.0.1:8080#listen_uri: 0.0.0.0:8090#' /etc/crowdsec/config.yaml
|
|
sed -i 's#url: http://127.0.0.1:8080#url: http://127.0.0.1:8090#' /etc/crowdsec/local_api_credentials.yaml
|
|
```
|
|
|
|
(`0.0.0.0` only because the Caddy container has to reach it, see below. If you
|
|
skip the Caddy bouncer, keep `127.0.0.1:8090`.)
|
|
|
|
**First start pulls nothing.** The very first community-blocklist pull ran
|
|
before the scenario list was registered ("scenario list is empty, will not
|
|
pull yet", 0 entries). A `systemctl restart crowdsec` pulled 15,000 IPs a
|
|
minute later. Check with `cscli decisions list --origin CAPI | wc -l`.
|
|
|
|
## Log sources and collections
|
|
|
|
```
|
|
cscli collections install crowdsecurity/caddy
|
|
cp acquis.d/caddy.yaml /etc/crowdsec/acquis.d/
|
|
```
|
|
|
|
The installer already adds `crowdsecurity/sshd` and a journald source for
|
|
`ssh.service`.
|
|
|
|
## Firewall bouncer
|
|
|
|
```
|
|
apt install crowdsec-firewall-bouncer-iptables
|
|
```
|
|
|
|
The package registers itself with the Local API. Then apply
|
|
`firewall-bouncer.yaml` (mode `nftables`) and restart it. Why not the
|
|
default iptables mode: it only inserts into `INPUT`. SSH is blocked, but every
|
|
Docker-published port (80, 443, 2222) stays reachable for a banned IP, the same
|
|
hole as fail2ban in episode 10. Adding `DOCKER-USER` fixes that until the next
|
|
`ufw reload`, which flushes the chain and nobody puts the rule back.
|
|
|
|
## Caddy bouncer and AppSec (WAF)
|
|
|
|
1. Build the Caddy image with the modules (`caddy/Dockerfile`), elsewhere, and
|
|
load it on the box.
|
|
2. AppSec collections, plus the rules the default config references but the
|
|
collections do not install (without them crowdsec **does not start**:
|
|
"no appsec-rules found for pattern crowdsecurity/experimental-*", then the
|
|
same for `generic-*`):
|
|
|
|
```
|
|
cscli collections install crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
|
|
cscli appsec-rules install crowdsecurity/experimental-no-user-agent \
|
|
crowdsecurity/generic-freemarker-ssti crowdsecurity/generic-wordpress-uploads-listing \
|
|
crowdsecurity/generic-wordpress-uploads-php
|
|
cp acquis.d/appsec.yaml /etc/crowdsec/acquis.d/
|
|
crowdsec -t && systemctl restart crowdsec
|
|
```
|
|
|
|
Run `crowdsec -t` first. A failed restart means no Local API; the firewall
|
|
keeps its current sets, but no new decisions arrive.
|
|
3. Let the Docker network in, and only that network:
|
|
|
|
```
|
|
ufw allow proto tcp from 172.18.0.0/16 to 172.17.0.1 port 8090,7422
|
|
```
|
|
|
|
The container reaches the host as `host.docker.internal` (`extra_hosts:
|
|
host-gateway` in `compose.yml`, which is `172.17.0.1`). From the internet
|
|
both ports stay filtered. The mesh interface `wt0` is allowed wholesale by
|
|
episode 5, so mesh peers can reach both; each still needs an API key.
|
|
4. `cscli bouncers add caddy-bouncer -o raw` and put the key into `.env` as
|
|
`CROWDSEC_BOUNCER_KEY`, then `docker compose up -d caddy`.
|
|
|
|
The `vpn.` site does not import the `crowdsec` snippet: it carries the NetBird
|
|
gRPC streams, and a WAF round-trip on every stream message is not what you want
|
|
between your peers.
|
|
|
|
## Replaying old logs (and the forensic-mode trap)
|
|
|
|
`crowdsec -dsn file:///path/to/log -type syslog` is documented as replay mode.
|
|
With CrowdSec 1.8.1 it shut down one second after reading the file and dropped
|
|
every event still in the pipeline: 47 or 49 alerts on two identical runs, none
|
|
after 02:07 of a 24 h log. Feeding the same file through a FIFO that stays open
|
|
60 s after the last line gives the full, repeatable result (133 alerts, twice):
|
|
|
|
```
|
|
mkfifo /tmp/replay.fifo
|
|
(cat sshd.log; sleep 60) > /tmp/replay.fifo &
|
|
crowdsec -c replay.yaml -order-event -dsn file:///tmp/replay.fifo -type syslog
|
|
```
|
|
|
|
Use a separate config (`replay.yaml`) with its own database, its own
|
|
`listen_uri` and **no `online_client`**, or the replayed alerts land in your
|
|
live database and go out as signals.
|
|
|
|
## What goes out to CrowdSec
|
|
|
|
With the default config, every alert from an unmodified hub scenario is pushed
|
|
as a signal: scenario name and hash, timestamp, your machine id, and **the
|
|
attacking IP**. Custom or modified ("tainted") scenarios and manual decisions
|
|
are not pushed unless you enroll in the console. Your own penetration tests are
|
|
real alerts too: switch `sharing: false` under `api.server.online_client`
|
|
while you attack your own box, or you report your test machine.
|
|
|
|
## Optional: OWASP CRS for generic coverage, and the tuning it needs (episode 13)
|
|
|
|
Virtual patching (the default above) only blocks known-CVE signatures. Measured
|
|
against 41 real probes: **7 blocked**. Generic SQLi, XSS, Log4Shell-via-header,
|
|
Struts OGNL and path traversal all passed, because there is no rule for them,
|
|
only per-CVE ones. `base-config` is just the Coraza body-processor setup, not
|
|
the OWASP Core Rule Set.
|
|
|
|
The CRS is in the hub as an appsec-rule. Adding it raises coverage to **27 of 41**:
|
|
|
|
```
|
|
cscli appsec-rules install crowdsecurity/crs
|
|
```
|
|
|
|
But at default paranoia the CRS blocks legitimate use. Measured false positives
|
|
with CRS on and nothing else changed: git clone/push over HTTPS (the pack
|
|
negotiation body), Forgejo issues and file contents whose text contains code or
|
|
SQL, Outline documents with a code block, ntfy publish bodies, n8n webhook
|
|
bodies. Six normal workflows, all 403. For a stack where people push git and
|
|
write wiki pages, unusable as-is.
|
|
|
|
The fix is a per-route exclusion. `appsec-rules/crs-route-exclusions.yaml` is a
|
|
small SecLang rule that runs in phase 1 and, for exactly the legitimate
|
|
application routes (Forgejo git smart-HTTP and REST API, Outline API, ntfy
|
|
publish, n8n webhooks), matched by `REQUEST_HEADERS:Host` + `REQUEST_URI`,
|
|
removes the CRS anomaly blocking rules with
|
|
`ctl:ruleRemoveById=949110,ctl:ruleRemoveById=959100` (949110 is the CRS
|
|
inbound anomaly-score blocker, 959100 the outbound one). CRS still scores the
|
|
request but no longer blocks it on those routes.
|
|
|
|
Do **not** use `ctl:ruleEngine=Off` for this. It turns the whole Coraza engine
|
|
off for the request, and measured on the box that also disables the native
|
|
CrowdSec virtual-patching rules: on a route excluded with `ruleEngine=Off`, a
|
|
`.env` probe returned 404 (no WAF at all) instead of 403. With
|
|
`ctl:ruleRemoveById` the engine stays on, so virtual patching keeps firing:
|
|
measured, `.env`, the PHPUnit RCE and `.git/config` still return 403 on the
|
|
excluded routes, while generic SQLi/XSS now pass (that is the point).
|
|
|
|
Rule order does not matter for `ctl:ruleRemoveById` (a phase-1 ctl removes the
|
|
phase-2 blocker regardless of file position); the exclusion sits first only for
|
|
readability. It would matter for `ruleEngine=Off`, one more reason to avoid it.
|
|
|
|
`appsec-configs/appsec-tuned.yaml` wires it up: base-config, the exclusion rule
|
|
(must come **before** crs so the ctl takes effect in phase 1), crs, then
|
|
vpatch and generic. Deploy it instead of the default:
|
|
|
|
```
|
|
cp appsec-rules/crs-route-exclusions.yaml /etc/crowdsec/appsec-rules/
|
|
cp appsec-configs/appsec-tuned.yaml /etc/crowdsec/appsec-configs/
|
|
# point acquis.d/appsec.yaml at selfhostedops/appsec-tuned instead of crowdsecurity/appsec-default
|
|
crowdsec -t && systemctl restart crowdsec
|
|
```
|
|
|
|
Measured with the tuned config: **0 false positives** across every workflow
|
|
above (git clone/push, Forgejo issues and file contents, Outline documents with
|
|
an attack-looking code block, ntfy publish, n8n webhooks, and a full
|
|
authenticated Vaultwarden register/login/sync/cipher-create), still **27 of 41**
|
|
probes blocked on the non-excluded routes. The cost is precise: on an excluded
|
|
route (say the Forgejo API) generic SQLi/XSS now passes to the app, while
|
|
virtual patching still catches known CVEs there, and on a non-excluded host
|
|
(vault, status) the full CRS applies. Latency added over vpatch: about +2 ms
|
|
time-to-first-byte per request. RAM measured settled: +9 MB over vpatch after five minutes (vpatch ~184 MB, CRS+exclusions ~193 MB). The ~25-45 MB people cite is a load-time peak, not the steady cost.
|
|
|
|
Traps found while tuning:
|
|
- `RemoveInBandRuleByName("crowdsecurity/crs")` in a `pre_eval` hook does **not**
|
|
disable CRS: the bouncer name does not match the compiled SecLang rules
|
|
(measured, globally and per route, SQLi stayed 403). Use `ctl:` inside SecLang.
|
|
Docs: https://docs.crowdsec.net/docs/appsec/hooks
|
|
- `ctl:ruleEngine=Off` is too broad: it disables virtual patching on the route
|
|
too. Use `ctl:ruleRemoveById` for the CRS blocking rules and keep the engine on.
|
|
- Every new app or new public path needs its own exclusion. A missed one blocks
|
|
a real workflow, and it fails only after login, so it is easy to miss.
|
|
|
|
**What this repo runs by default: vpatch + generic** (the `appsec-default`
|
|
config, zero false positives, no per-app tuning). The CRS files above are the
|
|
measured opt-in for anyone who wants generic coverage and will budget the
|
|
per-route tuning.
|