bb2b5b5e22
Three workflows that connect the services from episodes 1 to 7: a Forgejo push becomes a phone notification, an Uptime Kuma outage writes and closes its own incident page in Outline, and a nightly schedule writes one ops digest. The point of the episode is the split between webhooks and the editor. Caddy publishes only the webhook path prefixes under the n8n host and answers an empty 404 for everything else, so the editor stays reachable over the mesh from episode 5 and never gets a login page on the open internet. Also in here: - n8n pinned to a version instead of :latest, after episode 7 showed that every restore is otherwise an upgrade - the 2.x environment that actually matters, including the file access restriction pointed at one directory rather than the root - the Forgejo allowed host list, without which a webhook to a private address is silently dropped - fixed subnets for both docker networks, after a restore handed them out in the other order and quietly broke the mesh route - a note on the encryption key: it decrypts every stored credential, it lives in the n8n volume, and a backup without it restores a hub that cannot log in Claude-Session: https://claude.ai/code/session_01Md8yhBKp5e7xMmCyxoEgdm
195 lines
7.3 KiB
Markdown
195 lines
7.3 KiB
Markdown
# ~$ Hardening checklist
|
|
|
|
This is the non-negotiable baseline before real company data touches the box.
|
|
Every item is copy-paste ready for Debian/Ubuntu.
|
|
|
|
## 1. SSH: keys only, no root
|
|
|
|
```bash
|
|
adduser ops && usermod -aG sudo,docker ops
|
|
mkdir -p /home/ops/.ssh && cp ~/.ssh/authorized_keys /home/ops/.ssh/ \
|
|
&& chown -R ops:ops /home/ops/.ssh && chmod 700 /home/ops/.ssh
|
|
```
|
|
|
|
`/etc/ssh/sshd_config.d/hardening.conf`:
|
|
|
|
```
|
|
PermitRootLogin no
|
|
PasswordAuthentication no
|
|
KbdInteractiveAuthentication no
|
|
```
|
|
|
|
```bash
|
|
systemctl restart ssh
|
|
# TEST A NEW SSH SESSION BEFORE CLOSING THIS ONE.
|
|
```
|
|
|
|
## 2. Firewall — four ports, nothing else
|
|
|
|
```bash
|
|
apt install -y ufw
|
|
ufw default deny incoming
|
|
ufw allow 80/tcp 443/tcp 443/udp # caddy (http/https/h3)
|
|
ufw allow 2222/tcp # git ssh
|
|
ufw allow 51820/udp # wireguard (only if using --profile vpn)
|
|
ufw allow 22/tcp # your ssh (move it if you want)
|
|
ufw enable
|
|
```
|
|
|
|
Note: Docker's published ports bypass ufw INPUT rules — that's exactly why this
|
|
compose file publishes only caddy, git-ssh and wireguard, and keeps postgres/redis
|
|
on the `internal` network with no ports at all. Don't add `ports:` to internal
|
|
services "for debugging" and forget them.
|
|
|
|
## 3. Automatic security updates
|
|
|
|
```bash
|
|
apt install -y unattended-upgrades
|
|
dpkg-reconfigure -plow unattended-upgrades
|
|
```
|
|
|
|
## 4. fail2ban (ssh brute force)
|
|
|
|
```bash
|
|
apt install -y fail2ban # default ssh jail is enough to start
|
|
```
|
|
|
|
## 5. Vaultwarden specifics
|
|
|
|
- `VAULTWARDEN_SIGNUPS_ALLOWED=false` the moment your team is on board.
|
|
- Use an argon2 `ADMIN_TOKEN` (`vaultwarden hash` / see wiki), not a plain string.
|
|
A plain one makes Vaultwarden print a NOTICE on every start. It is telling
|
|
the truth.
|
|
- **Never back up `db.sqlite3` on its own while the container runs.** The
|
|
database is in WAL mode: recent writes live in `db.sqlite3-wal` and are not
|
|
in the main file yet. Copy it alone and you get a backup that silently
|
|
misses your newest users and entries, and you find out on restore day.
|
|
Either stop the container first (see section 7), or take a consistent copy:
|
|
|
|
```bash
|
|
docker compose exec vaultwarden sh -c "sqlite3 /data/db.sqlite3 \".backup '/data/backup.sqlite3'\""
|
|
```
|
|
|
|
- Restore the backup somewhere else, once, before you trust it. A backup you
|
|
have never restored is a belief, not a backup.
|
|
- The admin panel (`/admin`) is for setup — consider blocking it in Caddy afterwards:
|
|
|
|
```
|
|
vault.{$DOMAIN} {
|
|
@admin path /admin*
|
|
respond @admin 404
|
|
reverse_proxy vaultwarden:80
|
|
}
|
|
```
|
|
|
|
## 6. Pin your images
|
|
|
|
`latest` is fine on day one, dangerous on day 300. Pin majors at minimum
|
|
(`caddy:2-alpine`, `forgejo:11`, `uptime-kuma:1` already are). Watchtower blindly
|
|
auto-updating a password manager at 3 a.m. is how you get incidents — update
|
|
deliberately: `docker compose pull && docker compose up -d` after reading release notes.
|
|
n8n and Outline move fast; Vaultwarden occasionally has breaking web-vault pairs.
|
|
|
|
## 7. Backups (minimum viable, today)
|
|
|
|
```bash
|
|
# stop, snapshot volumes, start — cron it nightly until Episode 10 (borgmatic offsite)
|
|
docker compose stop
|
|
tar czf /backup/stack-$(date +%F).tgz -C /var/lib/docker/volumes .
|
|
docker compose start
|
|
```
|
|
|
|
Plus: enable your provider's server snapshots. Two clicks, saves your company.
|
|
|
|
## 8. What this list is NOT
|
|
|
|
Not covered here, on purpose (own episodes): 2FA enforcement
|
|
per service, Netbird zero-trust mesh, central logging, CrowdSec, offsite borg.
|
|
Baseline first. Perfect later.
|
|
|
|
## CI: the two ways to hand your server away
|
|
|
|
Both of these look like shortcuts and both give anyone who can push a branch full
|
|
control of the machine.
|
|
|
|
**1. Mounting the host Docker socket into the runner.** Every forum answer suggests
|
|
it, because it works instantly:
|
|
|
|
```yaml
|
|
volumes:
|
|
- /var/run/docker.sock:/var/run/docker.sock # do not do this
|
|
```
|
|
|
|
A job can then start a privileged container on the host, mount `/` and read every
|
|
volume on the box, including the password manager sitting in the next container.
|
|
Use the docker-in-docker sidecar in `compose.yml` instead. It costs six lines and
|
|
the jobs talk to a daemon that has nothing of yours in it.
|
|
|
|
**2. An unrestricted SSH key for the deploy step.** If the key in your CI secrets
|
|
can run any command, the pipeline is a root shell with extra steps. Lock it to one
|
|
command in `authorized_keys`:
|
|
|
|
```
|
|
command="/usr/local/bin/deploy-myapp",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-ed25519 AAAA...
|
|
```
|
|
|
|
The script receives the argument from the job in `$SSH_ORIGINAL_COMMAND`. Sanitise
|
|
it — the example in `examples/deploy-hello-ops.sh` strips everything that is not a
|
|
hex character, because it only ever expects an image tag. The deploy user needs
|
|
Docker access, so the forced command is the thing containing it; without the forced
|
|
command, membership in the `docker` group is equivalent to root.
|
|
|
|
## Automation: the small door and the one file (episode 8)
|
|
|
|
**Publish paths, not hosts.** n8n needs a public address for webhooks that come
|
|
from outside, and for nothing else. The site block in `Caddyfile` serves
|
|
`/webhook/*`, `/webhook-test/*` and `/webhook-waiting/*` and answers an empty
|
|
404 for every other path, so the editor, the login page and the REST API are not
|
|
part of the public surface. The check is one line:
|
|
|
|
```bash
|
|
curl -o /dev/null -w '%{http_code}\n' https://n8n.<DOMAIN>/rest/settings # 404
|
|
curl -o /dev/null -w '%{http_code}\n' -X POST https://n8n.<DOMAIN>/webhook/<path> # 200
|
|
```
|
|
|
|
`/webhook-test/*` only does anything while somebody has "Test workflow" open in
|
|
the editor. If you never test against an outside sender, drop it from the
|
|
matcher and the surface gets smaller again.
|
|
|
|
**Senders inside the box should stay inside the box.** Forgejo and Uptime Kuma
|
|
reach `http://n8n:5678/webhook/...` over the docker network. No certificate, no
|
|
published port, no trip through the public IP. Forgejo blocks private targets by
|
|
default; `FORGEJO__webhook__ALLOWED_HOST_LIST: "external,n8n"` allows exactly
|
|
that one name and leaves the rest of the block in place.
|
|
|
|
**`/home/node/.n8n/config` is the whole credential store.** It contains
|
|
`encryptionKey`, and every credential in `database.sqlite` is encrypted with it.
|
|
Lose the file and the database is landfill: n8n starts, writes a fresh key, the
|
|
workflows still look fine, and the first run fails with
|
|
|
|
```
|
|
Credentials could not be decrypted. The likely reason is that a different
|
|
"encryptionKey" was used to encrypt the data.
|
|
```
|
|
|
|
The episode 7 backup covers the whole volume, so the key is in the snapshot.
|
|
Confirm it instead of assuming:
|
|
|
|
```bash
|
|
restic ls latest | grep n8n_data/_data/config
|
|
```
|
|
|
|
If you set `N8N_ENCRYPTION_KEY` in the environment, that value wins, the file is
|
|
never written, and the key then has to live in your password manager and survive
|
|
the box.
|
|
|
|
**Keep the file nodes on a leash.** `N8N_RESTRICT_FILE_ACCESS_TO: /backup` lets
|
|
the digest workflow read the backup log and nothing else. Left unset or widened
|
|
to `/`, a workflow, or anyone who can edit one, can read the encryption key from
|
|
inside n8n itself.
|
|
|
|
**Pin the image, pin the subnets.** `docker.n8n.io/n8nio/n8n:latest` turns every
|
|
restore into an unplanned upgrade. The `ipam` blocks under `networks:` do the
|
|
same job for addresses: Compose hands subnets out in creation order, so a
|
|
restore onto a fresh box can renumber them, and every mesh route that points at
|
|
the old range stops working without an error anywhere.
|