selfhostedops-stack/monitoring

Monitoring and alerting - notice before your customer does (episode 6)

Uptime Kuma has been in the stack since episode 1, but a monitor nobody hears is a dashboard, not an alarm. This episode adds three things: a push channel you own (ntfy), a public status page on its own hostname, and a second Kuma outside the server, because a box cannot report its own death.

ntfy: the push channel

One container, one topic. Kuma publishes to alerts, your phone subscribes to it with the ntfy app (Android, iOS, or the web UI at https://ntfy.<DOMAIN>).

  ntfy:
    image: binwiederhier/ntfy:v2.28.0
    restart: unless-stopped
    command: serve
    environment:
      NTFY_BASE_URL: https://ntfy.${DOMAIN}
      NTFY_BEHIND_PROXY: "true"
      NTFY_AUTH_DEFAULT_ACCESS: deny-all   # nobody reads or writes without a user
      NTFY_AUTH_FILE: /var/lib/ntfy/user.db
      NTFY_CACHE_FILE: /var/cache/ntfy/cache.db
      NTFY_ATTACHMENT_CACHE_DIR: /var/cache/ntfy/attachments
      NTFY_ENABLE_LOGIN: "true"
    volumes:
      - ntfy_data:/var/lib/ntfy
      - ntfy_cache:/var/cache/ntfy
    networks: [proxy]

Caddy block (websockets and long polling need nothing special):

ntfy.{$DOMAIN} {
	reverse_proxy ntfy:80
}

Start it, reload Caddy, then create two users: one that may only write to the topic (that is Kuma) and one that may read it (that is you).

docker compose up -d ntfy
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

docker compose exec -e NTFY_PASSWORD='<kuma password>' ntfy ntfy user add --role=user kuma
docker compose exec -e NTFY_PASSWORD='<your password>' ntfy ntfy user add --role=user max
docker compose exec ntfy ntfy access kuma alerts write-only
docker compose exec ntfy ntfy access max alerts read-write
docker compose exec ntfy ntfy access          # prints the whole table

Prove the lock works before you trust it:

# anonymous publish: 403
curl -d "anyone can post here?" https://ntfy.<DOMAIN>/alerts
# authenticated publish: lands
curl -u kuma:<pw> -H "Title: Hello from the box" -d "First message" https://ntfy.<DOMAIN>/alerts
# read it back as a reader
curl -u max:<pw> "https://ntfy.<DOMAIN>/alerts/json?poll=1"

Uptime Kuma: notification, monitors, status page

All of this is done in the Kuma web UI at https://status.<DOMAIN>/dashboard.

1. Settings -> Notifications -> Setup Notification. Type ntfy, server https://ntfy.<DOMAIN>, topic alerts, authentication username and password with the kuma user. Tick Default enabled and Apply on all existing monitors, press Test, watch it arrive on your phone.
2. Monitors. Public services get an HTTP(s) - Keyword monitor, because a 200 from a reverse proxy says nothing about the app behind it. Keywords that work with the stack: Forgejo, Outline, Vaultwarden, NetBird. Interval 60 s, retries 1, retry interval 20 s, Certificate Expiry Notification on. Settings -> Notifications -> TLS expiry: 7, 14, 21 days.
3. Services that left the public internet in episode 5 are monitored from inside the Docker network. n8n: http://n8n:5678/healthz. Kuma sits on the same proxy network, so the container name resolves.
4. Groups Public and Internal keep the dashboard readable. A Push monitor Nightly backup with a 90000 s interval turns a backup job into something that alerts when it does not run: the job calls the push URL, Kuma complains when the call is missing for 25 hours.
5. Status page: slug status, title Company Status, add only the public monitors. Under Domain Names enter status.<DOMAIN>: Kuma then serves the page at the root of that hostname and keeps the dashboard at /dashboard. No extra Caddy rule needed.

Verify:

curl -s https://status.<DOMAIN>/ | grep -o "<title>[^<]*</title>"     # Company Status
curl -s https://status.<DOMAIN>/api/status-page/status | python3 -m json.tool

The second eye

Stop the server and both Kuma and ntfy stop with it: no alert. So a second Uptime Kuma runs on a machine that is not that server (a laptop at home, a Raspberry Pi, a colleague's NAS). LAN only, no public route.

name: kuma-home
services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    restart: unless-stopped
    ports:
      - "3001:3001"
    volumes:
      - kuma_data:/app/data
volumes:
  kuma_data:

Its monitors: keyword on https://status.<DOMAIN>/ (keyword Company Status), keyword on https://git.<DOMAIN>/ (keyword Forgejo), ping on the server IP. Its notification must not depend on the server either, so it uses the public https://ntfy.sh with a random, unguessable topic (sho- plus twelve random characters). Subscribe to that topic in the same ntfy app, next to your own.

Rehearse the outage

Do not wait for the real one. docker stop selfhostedops-stack-outline-1, start a stopwatch, docker start it when the phone buzzes. With the settings above the DOWN alert takes between 20 and 80 seconds (interval plus one retry), the UP alert follows about a minute after the container is healthy.

Gotchas

• NTFY_AUTH_DEFAULT_ACCESS: deny-all also blocks the reader. A user with write-only cannot poll the topic (403). Give yourself read-write, give Kuma write-only, never the other way round.
• The ntfy CLI takes the password from the NTFY_PASSWORD environment variable. Pass it with docker compose exec -e NTFY_PASSWORD=...; ntfy user add without it wants a terminal and hangs in scripts.
• A plain HTTP monitor is happy with any 2xx. Caddy answers 2xx for a dead backend in some setups (a login page, a maintenance page). Use keyword monitors for anything a customer sees.
• Switching a monitor from a dead public URL to the internal one fires an Up notification the moment it works again. Expected, but it surprises people the first time.
• Kuma 1.x refuses monitor intervals above 24 days and the push monitor's interval is also its grace period: 90000 s means "alert after 25 hours without a ping", not "check every 25 hours".
• The status page is rendered server side for the domain root, the dashboard stays a single page app. A keyword monitor on the status page therefore works (the title is in the HTML), a keyword monitor on /dashboard does not.
• The Domain Names field only works when the request really arrives with that Host header. Behind Caddy that is the default; behind a proxy that rewrites the host header, set trustProxy in Kuma's settings and forward the header.
• The second Kuma uses the public ntfy.sh without authentication. Anyone who guesses the topic can read your alerts and post fake ones. Twelve random characters make guessing impractical; still, do not put secrets in monitor names.
• louislam/uptime-kuma:1 is the 1.x line (1.23.17 at recording time). 2.x exists (2.5.3) and migrates the database on first start. Pin what you test, upgrade on purpose, not by docker compose pull.

What episode 16 found in this setup

• ntfy rejects Kuma's message for monitors without a URL. Ping, port, group and push monitors have no URL; Kuma still adds a "view" button and ntfy answers 400 parameter 'url' is required for action 'view'. The alert never arrives, only the Kuma log shows it. Give those monitors a URL (any page, for example the status page); in the Kuma database that is the url column, the edit form hides it for these types.
• Kuma only alerts on a change. A monitor that stays down is announced once. Set "Resend Notification if Down X times consecutively" (for a 60 s interval, 60 means once an hour), otherwise a twelve day outage looks exactly like a one minute one.
• Every Kuma restart announces every down monitor again. After a restart the first beat is PENDING, the next DOWN counts as a change. Anything that opens tickets from Kuma must skip a DOWN for a monitor that is already open.