selfhostedops-stack/backup/README.md

Backup and restore - lose the whole stack once and get it back (episode 7)

One script, one systemd timer, one S3 bucket. Every night the stack dumps its databases, restic ships everything to the bucket, old snapshots are pruned, and Uptime Kuma gets a heartbeat. The restore is a second script that turns an empty Debian box into the running stack in a few minutes.

Measured on the demo box (2 vCPU, 4 GB, ~590 MB of data): first backup 11 s, every night after that about 5 s and a few MB. Restore on a fresh box: 10 s install, 7 s restore, 68 s docker compose up (image pulls), services answering with their old certificates after about two minutes.

What goes into the backup

Path	Why
/opt/selfhostedops-stack	compose.yml, Caddyfile, .env, this folder incl. dumps/
/opt/netbird	NetBird compose, config.yaml, dashboard.env
/var/lib/docker/volumes	every named volume: Forgejo, Outline (Postgres + uploads), Vaultwarden, Uptime Kuma, ntfy, n8n, NetBird, and Caddy's caddy_data (the Let's Encrypt certificates and account, so the restored box serves valid HTTPS before DNS has moved)

Excluded, because a restore rebuilds them for free:

Volume	Why it is out
dind_data	the CI runner's private docker image cache, 1.8 GB of pulled layers
runner_data	the runner registration, one forgejo-runner register recreates it
docker_certs	TLS between runner and dind, generated on first start
metadata.db	docker's own volume registry; belongs to the docker on that box

The whole volumes directory is the target, so a volume a later episode adds is backed up without anyone editing the script. Forgetting to add a new volume to a hand-maintained list is the classic way backups rot.

Application-consistent without downtime

A raw copy of a running database file is a coin flip. So before restic runs:

• Outline: pg_dump through docker compose exec outline-postgres -> dumps/outline.sql.gz
• Forgejo, Vaultwarden, Uptime Kuma, ntfy, NetBird (all SQLite): sqlite3 <db> ".backup <copy>" in a throwaway keinos/sqlite3 container -> dumps/*.db. That is SQLite's online backup API, safe while the app keeps writing (a cp of a WAL-mode database is not).

The raw volume files are still in the snapshot. In the normal case they come back fine (the restore rehearsal below used exactly those). The dumps are the fallback for the day they do not.

Install (on the box)

apt-get install -y restic                 # Debian 13 ships 0.18.0
cd /opt/selfhostedops-stack
mkdir -p backup && cp backup/.restic-env.example backup/.restic-env
chmod 600 backup/.restic-env && nano backup/.restic-env   # bucket, key, password
docker pull keinos/sqlite3:latest

set -a; . backup/.restic-env; set +a
restic init

Any S3-compatible target works: Hetzner Object Storage, Backblaze B2, Wasabi, MinIO or Garage on a second box. Give the key access to this one bucket only: the key sits on the box it protects, and a box that is compromised must not be able to reach the other buckets.

Put the restic password in Vaultwarden and somewhere outside the stack. Vaultwarden is inside the backup; if the box is gone, so is the vault that holds the password to the backup of the vault.

Then the timer:

ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.service /etc/systemd/system/
ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now selfhostedops-backup.timer
systemctl list-timers selfhostedops-backup.timer

OnCalendar=*-*-* 03:00:00 Europe/Berlin with Persistent=true: a box that was down at 03:00 runs the backup as soon as it is back.

Run one by hand first and read the summary:

./backup.sh
restic snapshots
restic stats --mode raw-data
restic check

Retention is --keep-daily 7 --keep-weekly 4 --keep-monthly 6, applied with --prune after every run. Two runs on the same day count as one day: the policy keeps the last one of the day (and the oldest snapshot overall), the one in between is pruned.

Heartbeat

The last line of the script hits the Uptime Kuma push monitor from episode 6 with ?status=up&ping=<seconds>. If the script fails it sends status=down; if it never runs, Kuma raises the alarm itself after the monitor's grace period. Set KUMA_PUSH_URL in .restic-env.

Restore runbook

Rehearsed on a fresh Hetzner cpx22 with Debian 13. Everything below is backup/restore.sh; the phases are listed so you can do them by hand.

# 0. fresh box, as root. Bring the one file that is not in the backup:
mkdir -p /opt/selfhostedops-stack/backup
scp .restic-env root@NEWBOX:/opt/selfhostedops-stack/backup/    # from your vault
curl -fsSLO https://git.example.com/you/selfhostedops-stack/raw/branch/main/backup/restore.sh

# 1. install docker, compose plugin, restic            (measured: 10 s)
bash restore.sh install

# 2. named volumes first, then the files                (measured: 7 s)
bash restore.sh restore
#    = restic ls latest /var/lib/docker/volumes | ... | docker volume create
#      restic restore latest --target /

# 3. start                                              (measured: 68 s)
bash restore.sh up
#    = cd /opt/selfhostedops-stack && docker compose up -d
#      cd /opt/netbird && docker compose up -d

Verify before touching DNS. The old names still point at the old box, so pin them to the new IP for curl:

NEW=203.0.113.10
for h in git wiki vault status vpn ntfy; do
  curl -sS -o /dev/null -w "%{http_code} tls=%{ssl_verify_result} $h\n" \
    --resolve $h.ops.example.com:443:$NEW https://$h.ops.example.com/
done
curl -sS --resolve git.ops.example.com:443:$NEW \
  'https://git.ops.example.com/api/v1/repos/search' | jq '.data[].full_name'
curl -sS --resolve wiki.ops.example.com:443:$NEW https://wiki.ops.example.com/_health
curl -sS --resolve vault.ops.example.com:443:$NEW https://vault.ops.example.com/alive

tls=0 on every line means the certificates came out of caddy_data and are the originals. Then move the DNS record, delete the old box.

The restored box has .restic-env but no timer. Install the timer there the same way once it is the box; two hosts writing to one repository works but muddles the snapshot list.

Gotchas we hit

• A function named install in a bash script calls itself. The first restore script had install() { ... install -m 0755 -d /etc/apt/keyrings ... } and looped for ten minutes before anyone looked. Prefix your phase functions or call /usr/bin/install.
• keinos/sqlite3 runs as an unprivileged user. It can neither read the application databases nor write into dumps/: Error: cannot open "/dumps/forgejo.db". docker run --user 0 fixes it.
• trap ... ERR does not fire inside functions unless the script sets -E (set -Eeuo pipefail). Without it the failure heartbeat never goes out.
• docker compose up warns about every restored volume: volume "selfhostedops-stack_forgejo_data" already exists but was not created by Docker Compose. It is a warning, compose uses the volume. Do not down -v to make it go away.
• :latest images mean a restore is also an upgrade. Outline ran two schema migrations on the restored database because the fresh pull was newer than the image on the old box. It worked here; pin tags if you want a restore that changes nothing but the box.
• Ownership comes back with the files (restic restores uid/gid), but only if you restore as root with --target /. Restoring into a scratch directory and copying by hand loses the git user on Forgejo's data.
• The restored Uptime Kuma starts monitoring and alerting immediately, in parallel with the old box until DNS moves. Expect duplicate notifications for that window.
• Excluded targets still show up in the snapshot's path list if you pass them as backup arguments and exclude them at the same time. Pass the parent directory and let --exclude do the work; the path list stays honest.