Episode 7: nightly restic backup to S3 with consistent dumps, heartbeat, and a restore script for a fresh box
Claude-Session: https://claude.ai/code/session_01Y1BTehJjJ4Chr6YscVRXmn
This commit is contained in:
Maximilian Grimm
parent
80d5b00b67
commit
cf374770c4
7 changed files with 444 additions and 0 deletions
180
backup/README.md
Normal file
180
backup/README.md
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
# Backup and restore - lose the whole stack once and get it back (episode 7)
|
||||
|
||||
One script, one systemd timer, one S3 bucket. Every night the stack dumps
|
||||
its databases, restic ships everything to the bucket, old snapshots are
|
||||
pruned, and Uptime Kuma gets a heartbeat. The restore is a second script
|
||||
that turns an empty Debian box into the running stack in a few minutes.
|
||||
|
||||
Measured on the demo box (2 vCPU, 4 GB, ~590 MB of data): first backup
|
||||
11 s, every night after that about 5 s and a few MB. Restore on a fresh
|
||||
box: 10 s install, 7 s restore, 68 s `docker compose up` (image pulls),
|
||||
services answering with their old certificates after about two minutes.
|
||||
|
||||
## What goes into the backup
|
||||
|
||||
| Path | Why |
|
||||
| --- | --- |
|
||||
| `/opt/selfhostedops-stack` | `compose.yml`, `Caddyfile`, `.env`, this folder incl. `dumps/` |
|
||||
| `/opt/netbird` | NetBird compose, `config.yaml`, `dashboard.env` |
|
||||
| `/var/lib/docker/volumes` | every named volume: Forgejo, Outline (Postgres + uploads), Vaultwarden, Uptime Kuma, ntfy, n8n, NetBird, **and Caddy's `caddy_data`** (the Let's Encrypt certificates and account, so the restored box serves valid HTTPS before DNS has moved) |
|
||||
|
||||
Excluded, because a restore rebuilds them for free:
|
||||
|
||||
| Volume | Why it is out |
|
||||
| --- | --- |
|
||||
| `dind_data` | the CI runner's private docker image cache, 1.8 GB of pulled layers |
|
||||
| `runner_data` | the runner registration, one `forgejo-runner register` recreates it |
|
||||
| `docker_certs` | TLS between runner and dind, generated on first start |
|
||||
| `metadata.db` | docker's own volume registry; belongs to the docker on that box |
|
||||
|
||||
The whole volumes directory is the target, so a volume a later episode adds
|
||||
is backed up without anyone editing the script. Forgetting to add a new
|
||||
volume to a hand-maintained list is the classic way backups rot.
|
||||
|
||||
## Application-consistent without downtime
|
||||
|
||||
A raw copy of a running database file is a coin flip. So before restic runs:
|
||||
|
||||
- Outline: `pg_dump` through `docker compose exec outline-postgres`
|
||||
-> `dumps/outline.sql.gz`
|
||||
- Forgejo, Vaultwarden, Uptime Kuma, ntfy, NetBird (all SQLite):
|
||||
`sqlite3 <db> ".backup <copy>"` in a throwaway `keinos/sqlite3` container
|
||||
-> `dumps/*.db`. That is SQLite's online backup API, safe while the app
|
||||
keeps writing (a `cp` of a WAL-mode database is not).
|
||||
|
||||
The raw volume files are still in the snapshot. In the normal case they
|
||||
come back fine (the restore rehearsal below used exactly those). The dumps
|
||||
are the fallback for the day they do not.
|
||||
|
||||
## Install (on the box)
|
||||
|
||||
```bash
|
||||
apt-get install -y restic # Debian 13 ships 0.18.0
|
||||
cd /opt/selfhostedops-stack
|
||||
mkdir -p backup && cp backup/.restic-env.example backup/.restic-env
|
||||
chmod 600 backup/.restic-env && nano backup/.restic-env # bucket, key, password
|
||||
docker pull keinos/sqlite3:latest
|
||||
|
||||
set -a; . backup/.restic-env; set +a
|
||||
restic init
|
||||
```
|
||||
|
||||
Any S3-compatible target works: Hetzner Object Storage, Backblaze B2,
|
||||
Wasabi, MinIO or Garage on a second box. Give the key access to **this one
|
||||
bucket only**: the key sits on the box it protects, and a box that is
|
||||
compromised must not be able to reach the other buckets.
|
||||
|
||||
Put the restic password in Vaultwarden **and** somewhere outside the stack.
|
||||
Vaultwarden is inside the backup; if the box is gone, so is the vault that
|
||||
holds the password to the backup of the vault.
|
||||
|
||||
Then the timer:
|
||||
|
||||
```bash
|
||||
ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.service /etc/systemd/system/
|
||||
ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now selfhostedops-backup.timer
|
||||
systemctl list-timers selfhostedops-backup.timer
|
||||
```
|
||||
|
||||
`OnCalendar=*-*-* 03:00:00 Europe/Berlin` with `Persistent=true`: a box that
|
||||
was down at 03:00 runs the backup as soon as it is back.
|
||||
|
||||
Run one by hand first and read the summary:
|
||||
|
||||
```bash
|
||||
./backup.sh
|
||||
restic snapshots
|
||||
restic stats --mode raw-data
|
||||
restic check
|
||||
```
|
||||
|
||||
Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`, applied
|
||||
with `--prune` after every run. Two runs on the same day count as one day:
|
||||
the policy keeps the last one of the day (and the oldest snapshot overall),
|
||||
the one in between is pruned.
|
||||
|
||||
### Heartbeat
|
||||
|
||||
The last line of the script hits the Uptime Kuma push monitor from episode
|
||||
6 with `?status=up&ping=<seconds>`. If the script fails it sends
|
||||
`status=down`; if it never runs, Kuma raises the alarm itself after the
|
||||
monitor's grace period. Set `KUMA_PUSH_URL` in `.restic-env`.
|
||||
|
||||
## Restore runbook
|
||||
|
||||
Rehearsed on a fresh Hetzner cpx22 with Debian 13. Everything below is
|
||||
`backup/restore.sh`; the phases are listed so you can do them by hand.
|
||||
|
||||
```bash
|
||||
# 0. fresh box, as root. Bring the one file that is not in the backup:
|
||||
mkdir -p /opt/selfhostedops-stack/backup
|
||||
scp .restic-env root@NEWBOX:/opt/selfhostedops-stack/backup/ # from your vault
|
||||
curl -fsSLO https://git.example.com/you/selfhostedops-stack/raw/branch/main/backup/restore.sh
|
||||
|
||||
# 1. install docker, compose plugin, restic (measured: 10 s)
|
||||
bash restore.sh install
|
||||
|
||||
# 2. named volumes first, then the files (measured: 7 s)
|
||||
bash restore.sh restore
|
||||
# = restic ls latest /var/lib/docker/volumes | ... | docker volume create
|
||||
# restic restore latest --target /
|
||||
|
||||
# 3. start (measured: 68 s)
|
||||
bash restore.sh up
|
||||
# = cd /opt/selfhostedops-stack && docker compose up -d
|
||||
# cd /opt/netbird && docker compose up -d
|
||||
```
|
||||
|
||||
Verify before touching DNS. The old names still point at the old box, so
|
||||
pin them to the new IP for curl:
|
||||
|
||||
```bash
|
||||
NEW=203.0.113.10
|
||||
for h in git wiki vault status vpn ntfy; do
|
||||
curl -sS -o /dev/null -w "%{http_code} tls=%{ssl_verify_result} $h\n" \
|
||||
--resolve $h.ops.example.com:443:$NEW https://$h.ops.example.com/
|
||||
done
|
||||
curl -sS --resolve git.ops.example.com:443:$NEW \
|
||||
'https://git.ops.example.com/api/v1/repos/search' | jq '.data[].full_name'
|
||||
curl -sS --resolve wiki.ops.example.com:443:$NEW https://wiki.ops.example.com/_health
|
||||
curl -sS --resolve vault.ops.example.com:443:$NEW https://vault.ops.example.com/alive
|
||||
```
|
||||
|
||||
`tls=0` on every line means the certificates came out of `caddy_data` and
|
||||
are the originals. Then move the DNS record, delete the old box.
|
||||
|
||||
The restored box has `.restic-env` but no timer. Install the timer there
|
||||
the same way once it is the box; two hosts writing to one repository works
|
||||
but muddles the snapshot list.
|
||||
|
||||
## Gotchas we hit
|
||||
|
||||
- **A function named `install` in a bash script calls itself.** The first
|
||||
restore script had `install() { ... install -m 0755 -d /etc/apt/keyrings ... }`
|
||||
and looped for ten minutes before anyone looked. Prefix your phase
|
||||
functions or call `/usr/bin/install`.
|
||||
- **`keinos/sqlite3` runs as an unprivileged user.** It can neither read
|
||||
the application databases nor write into `dumps/`: `Error: cannot open
|
||||
"/dumps/forgejo.db"`. `docker run --user 0` fixes it.
|
||||
- **`trap ... ERR` does not fire inside functions** unless the script sets
|
||||
`-E` (`set -Eeuo pipefail`). Without it the failure heartbeat never goes out.
|
||||
- **`docker compose up` warns about every restored volume:** `volume
|
||||
"selfhostedops-stack_forgejo_data" already exists but was not created by
|
||||
Docker Compose`. It is a warning, compose uses the volume. Do not `down -v`
|
||||
to make it go away.
|
||||
- **`:latest` images mean a restore is also an upgrade.** Outline ran two
|
||||
schema migrations on the restored database because the fresh pull was
|
||||
newer than the image on the old box. It worked here; pin tags if you want
|
||||
a restore that changes nothing but the box.
|
||||
- **Ownership comes back with the files** (restic restores uid/gid), but
|
||||
only if you restore as root with `--target /`. Restoring into a scratch
|
||||
directory and copying by hand loses the `git` user on Forgejo's data.
|
||||
- **The restored Uptime Kuma starts monitoring and alerting immediately**,
|
||||
in parallel with the old box until DNS moves. Expect duplicate
|
||||
notifications for that window.
|
||||
- **Excluded targets still show up in the snapshot's path list** if you
|
||||
pass them as backup arguments and exclude them at the same time. Pass the
|
||||
parent directory and let `--exclude` do the work; the path list stays
|
||||
honest.
|
||||
Loading…
Add table
Add a link
Reference in a new issue