selfhostedops-stack/backup/README.md

# Backup and restore - lose the whole stack once and get it back (episode 7)

One script, one systemd timer, one S3 bucket. Every night the stack dumps
its databases, restic ships everything to the bucket, old snapshots are
pruned, and Uptime Kuma gets a heartbeat. The restore is a second script
that turns an empty Debian box into the running stack in a few minutes.

Measured on the demo box (2 vCPU, 4 GB, ~590 MB of data): first backup
11 s, every night after that about 5 s and a few MB. Restore on a fresh
box: 10 s install, 7 s restore, 68 s `docker compose up` (image pulls),
services answering with their old certificates after about two minutes.

## What goes into the backup

| Path | Why |
| --- | --- |
| `/opt/selfhostedops-stack` | `compose.yml`, `Caddyfile`, `.env`, this folder incl. `dumps/` |
| `/opt/netbird` | NetBird compose, `config.yaml`, `dashboard.env` |
| `/var/lib/docker/volumes` | every named volume: Forgejo, Outline (Postgres + uploads), Vaultwarden, Uptime Kuma, ntfy, n8n, NetBird, **and Caddy's `caddy_data`** (the Let's Encrypt certificates and account, so the restored box serves valid HTTPS before DNS has moved) |

Excluded, because a restore rebuilds them for free:

| Volume | Why it is out |
| --- | --- |
| `dind_data` | the CI runner's private docker image cache, 1.8 GB of pulled layers |
| `runner_data` | the runner registration, one `forgejo-runner register` recreates it |
| `docker_certs` | TLS between runner and dind, generated on first start |
| `metadata.db` | docker's own volume registry; belongs to the docker on that box |

The whole volumes directory is the target, so a volume a later episode adds
is backed up without anyone editing the script. Forgetting to add a new
volume to a hand-maintained list is the classic way backups rot.

## Application-consistent without downtime

A raw copy of a running database file is a coin flip. So before restic runs:

- Outline: `pg_dump` through `docker compose exec outline-postgres`
  -> `dumps/outline.sql.gz`
- Forgejo, Vaultwarden, Uptime Kuma, ntfy, NetBird (all SQLite):
  `sqlite3 <db> ".backup <copy>"` in a throwaway `keinos/sqlite3` container
  -> `dumps/*.db`. That is SQLite's online backup API, safe while the app
  keeps writing (a `cp` of a WAL-mode database is not).

The raw volume files are still in the snapshot. In the normal case they
come back fine (the restore rehearsal below used exactly those). The dumps
are the fallback for the day they do not.

## Install (on the box)

```bash
apt-get install -y restic                 # Debian 13 ships 0.18.0
cd /opt/selfhostedops-stack
mkdir -p backup && cp backup/.restic-env.example backup/.restic-env
chmod 600 backup/.restic-env && nano backup/.restic-env   # bucket, key, password
docker pull keinos/sqlite3:latest

set -a; . backup/.restic-env; set +a
restic init
```

Any S3-compatible target works: Hetzner Object Storage, Backblaze B2,
Wasabi, MinIO or Garage on a second box. Give the key access to **this one
bucket only**: the key sits on the box it protects, and a box that is
compromised must not be able to reach the other buckets.

Put the restic password in Vaultwarden **and** somewhere outside the stack.
Vaultwarden is inside the backup; if the box is gone, so is the vault that
holds the password to the backup of the vault.

Then the timer:

```bash
ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.service /etc/systemd/system/
ln -s /opt/selfhostedops-stack/backup/selfhostedops-backup.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now selfhostedops-backup.timer
systemctl list-timers selfhostedops-backup.timer
```

`OnCalendar=*-*-* 03:00:00 Europe/Berlin` with `Persistent=true`: a box that
was down at 03:00 runs the backup as soon as it is back.

Run one by hand first and read the summary:

```bash
./backup.sh
restic snapshots
restic stats --mode raw-data
restic check
```

Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`, applied
with `--prune` after every run. Two runs on the same day count as one day:
the policy keeps the last one of the day (and the oldest snapshot overall),
the one in between is pruned.

### Heartbeat

The last line of the script hits the Uptime Kuma push monitor from episode
6 with `?status=up&ping=<seconds>`. If the script fails it sends
`status=down`; if it never runs, Kuma raises the alarm itself after the
monitor's grace period. Set `KUMA_PUSH_URL` in `.restic-env`.

## Restore runbook

Rehearsed on a fresh Hetzner cpx22 with Debian 13. Everything below is
`backup/restore.sh`; the phases are listed so you can do them by hand.

```bash
# 0. fresh box, as root. Bring the one file that is not in the backup:
mkdir -p /opt/selfhostedops-stack/backup
scp .restic-env root@NEWBOX:/opt/selfhostedops-stack/backup/    # from your vault
curl -fsSLO https://git.example.com/you/selfhostedops-stack/raw/branch/main/backup/restore.sh

# 1. install docker, compose plugin, restic            (measured: 10 s)
bash restore.sh install

# 2. named volumes first, then the files                (measured: 7 s)
bash restore.sh restore
#    = restic ls latest /var/lib/docker/volumes | ... | docker volume create
#      restic restore latest --target /

# 3. start                                              (measured: 68 s)
bash restore.sh up
#    = cd /opt/selfhostedops-stack && docker compose up -d
#      cd /opt/netbird && docker compose up -d
```

Verify before touching DNS. The old names still point at the old box, so
pin them to the new IP for curl:

```bash
NEW=203.0.113.10
for h in git wiki vault status vpn ntfy; do
  curl -sS -o /dev/null -w "%{http_code} tls=%{ssl_verify_result} $h\n" \
    --resolve $h.ops.example.com:443:$NEW https://$h.ops.example.com/
done
curl -sS --resolve git.ops.example.com:443:$NEW \
  'https://git.ops.example.com/api/v1/repos/search' | jq '.data[].full_name'
curl -sS --resolve wiki.ops.example.com:443:$NEW https://wiki.ops.example.com/_health
curl -sS --resolve vault.ops.example.com:443:$NEW https://vault.ops.example.com/alive
```

`tls=0` on every line means the certificates came out of `caddy_data` and
are the originals. Then move the DNS record, delete the old box.

The restored box has `.restic-env` but no timer. Install the timer there
the same way once it is the box; two hosts writing to one repository works
but muddles the snapshot list.

## Gotchas we hit

- **A function named `install` in a bash script calls itself.** The first
  restore script had `install() { ... install -m 0755 -d /etc/apt/keyrings ... }`
  and looped for ten minutes before anyone looked. Prefix your phase
  functions or call `/usr/bin/install`.
- **`keinos/sqlite3` runs as an unprivileged user.** It can neither read
  the application databases nor write into `dumps/`: `Error: cannot open
  "/dumps/forgejo.db"`. `docker run --user 0` fixes it.
- **`trap ... ERR` does not fire inside functions** unless the script sets
  `-E` (`set -Eeuo pipefail`). Without it the failure heartbeat never goes out.
- **`docker compose up` warns about every restored volume:** `volume
  "selfhostedops-stack_forgejo_data" already exists but was not created by
  Docker Compose`. It is a warning, compose uses the volume. Do not `down -v`
  to make it go away.
- **`:latest` images mean a restore is also an upgrade.** Outline ran two
  schema migrations on the restored database because the fresh pull was
  newer than the image on the old box. It worked here; pin tags if you want
  a restore that changes nothing but the box.
- **Ownership comes back with the files** (restic restores uid/gid), but
  only if you restore as root with `--target /`. Restoring into a scratch
  directory and copying by hand loses the `git` user on Forgejo's data.
- **The restored Uptime Kuma starts monitoring and alerting immediately**,
  in parallel with the old box until DNS moves. Expect duplicate
  notifications for that window.
- **Excluded targets still show up in the snapshot's path list** if you
  pass them as backup arguments and exclude them at the same time. Pass the
  parent directory and let `--exclude` do the work; the path list stays
  honest.