selfhostedops/selfhostedops-stack
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Episode 17: updates one service at a time, a health check that asks real questions, every tag pinned

Browse source 
updates/health.sh: 43 checks across all services (API answers, git refs, a push
message round trip, SSO redirects and token endpoint, mesh state, backup age).
updates/update.sh: per service pull or build, snapshot, recreate, wait for the
health check, put image and data back on red and stop the run.
backup/backup.sh: online dumps for n8n, Pocket ID and tinyauth; BACKUP_TAG so
pre-update snapshots are not removed by the nightly keep-daily rule.
compose.yml and caddy/Dockerfile: exact versions instead of latest, 16, 7-alpine
and caddy:2-alpine.
This commit is contained in:
Maximilian Grimm 2026-10-08 22:04:29 +02:00
parent 1655d04d7d
commit 55b2fe7646
7 changed files with 380 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

24
backup/backup.sh
View file

@ -3,7 +3,8 @@
#
# Application-consistent, without stopping anything:
# 1. pg_dump of Outline's Postgres -> backup/dumps/outline.sql.gz
# 2. online SQLite backups (Forgejo, Vaultwarden, Uptime Kuma, ntfy, NetBird)
# 2. online SQLite backups (Forgejo, Vaultwarden, Uptime Kuma, ntfy, n8n,
# Pocket ID, tinyauth, NetBird)
# -> backup/dumps/*.db
# 3. restic backup of the stack directory, /opt/netbird and every stateful
# docker volume (the raw files, plus the dumps from 1 and 2)
@ -22,6 +23,12 @@ LOG_FILE="$BACKUP_DIR/backup.log"
PROJECT=${PROJECT:-selfhostedops-stack}
VOLUMES=/var/lib/docker/volumes
# Tag of the snapshot. The nightly timer uses "nightly". updates/update.sh
# runs this script with BACKUP_TAG=pre-update: those snapshots are left alone
# by forget below. Without that, "keep 7 daily" would drop the snapshot from
# before the update as soon as a second backup runs on the same day (episode 17).
BACKUP_TAG=${BACKUP_TAG:-nightly}
# Small alpine image with a current sqlite3, used for the online SQLite copies.
SQLITE_IMAGE=${SQLITE_IMAGE:-keinos/sqlite3:latest}
@ -77,6 +84,9 @@ sqlite_backup "${PROJECT}_vaultwarden_data" db.sqlite3 vaultwarden.sqlite3
sqlite_backup "${PROJECT}_kuma_data" kuma.db kuma.db
sqlite_backup "${PROJECT}_ntfy_data" user.db ntfy-user.db
sqlite_backup "${PROJECT}_ntfy_cache" cache.db ntfy-cache.db
sqlite_backup "${PROJECT}_n8n_data" database.sqlite n8n.sqlite
sqlite_backup "${PROJECT}_pocket_id_data" pocket-id.db pocket-id.db
sqlite_backup "${PROJECT}_tinyauth_data" tinyauth.db tinyauth.db
sqlite_backup "netbird_netbird_data" store.db netbird-store.db
sqlite_backup "netbird_netbird_data" idp.db netbird-idp.db
sqlite_backup "netbird_netbird_data" events.db netbird-events.db
@ -91,12 +101,16 @@ EXCLUDES=(--exclude "$VOLUMES/metadata.db" --exclude "$VOLUMES/backingFsBlockDev
for v in "${EXCLUDE_VOLUMES[@]}"; do
EXCLUDES+=(--exclude "$VOLUMES/${PROJECT}_$v")
done
restic backup --tag nightly "${EXCLUDES[@]}" \
restic backup --tag "$BACKUP_TAG" "${EXCLUDES[@]}" \
"$STACK_DIR" "$NETBIRD_DIR" "$VOLUMES"
log "[4/5] restic forget + prune"
restic forget --tag nightly --group-by host,tags \
--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
if [ "$BACKUP_TAG" = nightly ]; then
log "[4/5] restic forget + prune"
restic forget --tag nightly --group-by host,tags \
--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
else
log "[4/5] skipped: snapshots tagged $BACKUP_TAG are kept until you remove them"
fi
END=$(date +%s)
DURATION=$((END - START))