Episode 6: Uptime Kuma monitors that mean something, ntfy push server, status page, second eye

Claude-Session: https://claude.ai/code/session_01Y1BTehJjJ4Chr6YscVRXmn

monitoring/README.md

@@ -0,0 +1,161 @@
+# Monitoring and alerting - notice before your customer does (episode 6)
+
+Uptime Kuma has been in the stack since episode 1, but a monitor nobody hears
+is a dashboard, not an alarm. This episode adds three things: a push channel
+you own (ntfy), a public status page on its own hostname, and a second Kuma
+outside the server, because a box cannot report its own death.
+
+## ntfy: the push channel
+
+One container, one topic. Kuma publishes to `alerts`, your phone subscribes to
+it with the ntfy app (Android, iOS, or the web UI at `https://ntfy.<DOMAIN>`).
+
+```yaml
+  ntfy:
+    image: binwiederhier/ntfy:v2.28.0
+    restart: unless-stopped
+    command: serve
+    environment:
+      NTFY_BASE_URL: https://ntfy.${DOMAIN}
+      NTFY_BEHIND_PROXY: "true"
+      NTFY_AUTH_DEFAULT_ACCESS: deny-all   # nobody reads or writes without a user
+      NTFY_AUTH_FILE: /var/lib/ntfy/user.db
+      NTFY_CACHE_FILE: /var/cache/ntfy/cache.db
+      NTFY_ATTACHMENT_CACHE_DIR: /var/cache/ntfy/attachments
+      NTFY_ENABLE_LOGIN: "true"
+    volumes:
+      - ntfy_data:/var/lib/ntfy
+      - ntfy_cache:/var/cache/ntfy
+    networks: [proxy]
+```
+
+Caddy block (websockets and long polling need nothing special):
+
+```
+ntfy.{$DOMAIN} {
+	reverse_proxy ntfy:80
+}
+```
+
+Start it, reload Caddy, then create two users: one that may only write to the
+topic (that is Kuma) and one that may read it (that is you).
+
+```bash
+docker compose up -d ntfy
+docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
+
+docker compose exec -e NTFY_PASSWORD='<kuma password>' ntfy ntfy user add --role=user kuma
+docker compose exec -e NTFY_PASSWORD='<your password>' ntfy ntfy user add --role=user max
+docker compose exec ntfy ntfy access kuma alerts write-only
+docker compose exec ntfy ntfy access max alerts read-write
+docker compose exec ntfy ntfy access          # prints the whole table
+```
+
+Prove the lock works before you trust it:
+
+```bash
+# anonymous publish: 403
+curl -d "anyone can post here?" https://ntfy.<DOMAIN>/alerts
+# authenticated publish: lands
+curl -u kuma:<pw> -H "Title: Hello from the box" -d "First message" https://ntfy.<DOMAIN>/alerts
+# read it back as a reader
+curl -u max:<pw> "https://ntfy.<DOMAIN>/alerts/json?poll=1"
+```
+
+## Uptime Kuma: notification, monitors, status page
+
+All of this is done in the Kuma web UI at `https://status.<DOMAIN>/dashboard`.
+
+1. Settings -> Notifications -> Setup Notification. Type **ntfy**, server
+   `https://ntfy.<DOMAIN>`, topic `alerts`, authentication *username and
+   password* with the `kuma` user. Tick *Default enabled* and *Apply on all
+   existing monitors*, press *Test*, watch it arrive on your phone.
+2. Monitors. Public services get an **HTTP(s) - Keyword** monitor, because a
+   200 from a reverse proxy says nothing about the app behind it. Keywords
+   that work with the stack: `Forgejo`, `Outline`, `Vaultwarden`, `NetBird`.
+   Interval 60 s, retries 1, retry interval 20 s, *Certificate Expiry
+   Notification* on. Settings -> Notifications -> TLS expiry: 7, 14, 21 days.
+3. Services that left the public internet in episode 5 are monitored from
+   inside the Docker network. n8n: `http://n8n:5678/healthz`. Kuma sits on
+   the same `proxy` network, so the container name resolves.
+4. Groups `Public` and `Internal` keep the dashboard readable. A **Push**
+   monitor `Nightly backup` with a 90000 s interval turns a backup job into
+   something that alerts when it *does not* run: the job calls the push URL,
+   Kuma complains when the call is missing for 25 hours.
+5. Status page: slug `status`, title *Company Status*, add only the public
+   monitors. Under *Domain Names* enter `status.<DOMAIN>`: Kuma then serves
+   the page at the root of that hostname and keeps the dashboard at
+   `/dashboard`. No extra Caddy rule needed.
+
+Verify:
+
+```bash
+curl -s https://status.<DOMAIN>/ | grep -o "<title>[^<]*</title>"     # Company Status
+curl -s https://status.<DOMAIN>/api/status-page/status | python3 -m json.tool
+```
+
+## The second eye
+
+Stop the server and both Kuma and ntfy stop with it: no alert. So a second
+Uptime Kuma runs on a machine that is not that server (a laptop at home, a
+Raspberry Pi, a colleague's NAS). LAN only, no public route.
+
+```yaml
+name: kuma-home
+services:
+  uptime-kuma:
+    image: louislam/uptime-kuma:1
+    restart: unless-stopped
+    ports:
+      - "3001:3001"
+    volumes:
+      - kuma_data:/app/data
+volumes:
+  kuma_data:
+```
+
+Its monitors: keyword on `https://status.<DOMAIN>/` (keyword *Company Status*),
+keyword on `https://git.<DOMAIN>/` (keyword *Forgejo*), ping on the server IP.
+Its notification must not depend on the server either, so it uses the public
+`https://ntfy.sh` with a random, unguessable topic (`sho-` plus twelve random
+characters). Subscribe to that topic in the same ntfy app, next to your own.
+
+## Rehearse the outage
+
+Do not wait for the real one. `docker stop selfhostedops-stack-outline-1`,
+start a stopwatch, `docker start` it when the phone buzzes. With the settings
+above the DOWN alert takes between 20 and 80 seconds (interval plus one
+retry), the UP alert follows about a minute after the container is healthy.
+
+## Gotchas
+
+- `NTFY_AUTH_DEFAULT_ACCESS: deny-all` also blocks the *reader*. A user with
+  `write-only` cannot poll the topic (403). Give yourself `read-write`, give
+  Kuma `write-only`, never the other way round.
+- The ntfy CLI takes the password from the `NTFY_PASSWORD` environment
+  variable. Pass it with `docker compose exec -e NTFY_PASSWORD=...`;
+  `ntfy user add` without it wants a terminal and hangs in scripts.
+- A plain HTTP monitor is happy with any 2xx. Caddy answers 2xx for a dead
+  backend in some setups (a login page, a maintenance page). Use keyword
+  monitors for anything a customer sees.
+- Switching a monitor from a dead public URL to the internal one fires an
+  *Up* notification the moment it works again. Expected, but it surprises
+  people the first time.
+- Kuma 1.x refuses monitor intervals above 24 days and the push monitor's
+  interval is also its grace period: 90000 s means "alert after 25 hours
+  without a ping", not "check every 25 hours".
+- The status page is rendered server side for the domain root, the
+  dashboard stays a single page app. A keyword monitor on the status page
+  therefore works (the title is in the HTML), a keyword monitor on
+  `/dashboard` does not.
+- The *Domain Names* field only works when the request really arrives with
+  that `Host` header. Behind Caddy that is the default; behind a proxy that
+  rewrites the host header, set `trustProxy` in Kuma's settings and forward
+  the header.
+- The second Kuma uses the public ntfy.sh without authentication. Anyone who
+  guesses the topic can read your alerts and post fake ones. Twelve random
+  characters make guessing impractical; still, do not put secrets in monitor
+  names.
+- `louislam/uptime-kuma:1` is the 1.x line (1.23.17 at recording time).
+  2.x exists (2.5.3) and migrates the database on first start. Pin what you
+  test, upgrade on purpose, not by `docker compose pull`.