selfhostedops-stack/monitoring/README.md

# Monitoring and alerting - notice before your customer does (episode 6)

Uptime Kuma has been in the stack since episode 1, but a monitor nobody hears
is a dashboard, not an alarm. This episode adds three things: a push channel
you own (ntfy), a public status page on its own hostname, and a second Kuma
outside the server, because a box cannot report its own death.

## ntfy: the push channel

One container, one topic. Kuma publishes to `alerts`, your phone subscribes to
it with the ntfy app (Android, iOS, or the web UI at `https://ntfy.<DOMAIN>`).

```yaml
  ntfy:
    image: binwiederhier/ntfy:v2.28.0
    restart: unless-stopped
    command: serve
    environment:
      NTFY_BASE_URL: https://ntfy.${DOMAIN}
      NTFY_BEHIND_PROXY: "true"
      NTFY_AUTH_DEFAULT_ACCESS: deny-all   # nobody reads or writes without a user
      NTFY_AUTH_FILE: /var/lib/ntfy/user.db
      NTFY_CACHE_FILE: /var/cache/ntfy/cache.db
      NTFY_ATTACHMENT_CACHE_DIR: /var/cache/ntfy/attachments
      NTFY_ENABLE_LOGIN: "true"
    volumes:
      - ntfy_data:/var/lib/ntfy
      - ntfy_cache:/var/cache/ntfy
    networks: [proxy]
```

Caddy block (websockets and long polling need nothing special):

```
ntfy.{$DOMAIN} {
	reverse_proxy ntfy:80
}
```

Start it, reload Caddy, then create two users: one that may only write to the
topic (that is Kuma) and one that may read it (that is you).

```bash
docker compose up -d ntfy
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

docker compose exec -e NTFY_PASSWORD='<kuma password>' ntfy ntfy user add --role=user kuma
docker compose exec -e NTFY_PASSWORD='<your password>' ntfy ntfy user add --role=user max
docker compose exec ntfy ntfy access kuma alerts write-only
docker compose exec ntfy ntfy access max alerts read-write
docker compose exec ntfy ntfy access          # prints the whole table
```

Prove the lock works before you trust it:

```bash
# anonymous publish: 403
curl -d "anyone can post here?" https://ntfy.<DOMAIN>/alerts
# authenticated publish: lands
curl -u kuma:<pw> -H "Title: Hello from the box" -d "First message" https://ntfy.<DOMAIN>/alerts
# read it back as a reader
curl -u max:<pw> "https://ntfy.<DOMAIN>/alerts/json?poll=1"
```

## Uptime Kuma: notification, monitors, status page

All of this is done in the Kuma web UI at `https://status.<DOMAIN>/dashboard`.

1. Settings -> Notifications -> Setup Notification. Type **ntfy**, server
   `https://ntfy.<DOMAIN>`, topic `alerts`, authentication *username and
   password* with the `kuma` user. Tick *Default enabled* and *Apply on all
   existing monitors*, press *Test*, watch it arrive on your phone.
2. Monitors. Public services get an **HTTP(s) - Keyword** monitor, because a
   200 from a reverse proxy says nothing about the app behind it. Keywords
   that work with the stack: `Forgejo`, `Outline`, `Vaultwarden`, `NetBird`.
   Interval 60 s, retries 1, retry interval 20 s, *Certificate Expiry
   Notification* on. Settings -> Notifications -> TLS expiry: 7, 14, 21 days.
3. Services that left the public internet in episode 5 are monitored from
   inside the Docker network. n8n: `http://n8n:5678/healthz`. Kuma sits on
   the same `proxy` network, so the container name resolves.
4. Groups `Public` and `Internal` keep the dashboard readable. A **Push**
   monitor `Nightly backup` with a 90000 s interval turns a backup job into
   something that alerts when it *does not* run: the job calls the push URL,
   Kuma complains when the call is missing for 25 hours.
5. Status page: slug `status`, title *Company Status*, add only the public
   monitors. Under *Domain Names* enter `status.<DOMAIN>`: Kuma then serves
   the page at the root of that hostname and keeps the dashboard at
   `/dashboard`. No extra Caddy rule needed.

Verify:

```bash
curl -s https://status.<DOMAIN>/ | grep -o "<title>[^<]*</title>"     # Company Status
curl -s https://status.<DOMAIN>/api/status-page/status | python3 -m json.tool
```

## The second eye

Stop the server and both Kuma and ntfy stop with it: no alert. So a second
Uptime Kuma runs on a machine that is not that server (a laptop at home, a
Raspberry Pi, a colleague's NAS). LAN only, no public route.

```yaml
name: kuma-home
services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    restart: unless-stopped
    ports:
      - "3001:3001"
    volumes:
      - kuma_data:/app/data
volumes:
  kuma_data:
```

Its monitors: keyword on `https://status.<DOMAIN>/` (keyword *Company Status*),
keyword on `https://git.<DOMAIN>/` (keyword *Forgejo*), ping on the server IP.
Its notification must not depend on the server either, so it uses the public
`https://ntfy.sh` with a random, unguessable topic (`sho-` plus twelve random
characters). Subscribe to that topic in the same ntfy app, next to your own.

## Rehearse the outage

Do not wait for the real one. `docker stop selfhostedops-stack-outline-1`,
start a stopwatch, `docker start` it when the phone buzzes. With the settings
above the DOWN alert takes between 20 and 80 seconds (interval plus one
retry), the UP alert follows about a minute after the container is healthy.

## Gotchas

- `NTFY_AUTH_DEFAULT_ACCESS: deny-all` also blocks the *reader*. A user with
  `write-only` cannot poll the topic (403). Give yourself `read-write`, give
  Kuma `write-only`, never the other way round.
- The ntfy CLI takes the password from the `NTFY_PASSWORD` environment
  variable. Pass it with `docker compose exec -e NTFY_PASSWORD=...`;
  `ntfy user add` without it wants a terminal and hangs in scripts.
- A plain HTTP monitor is happy with any 2xx. Caddy answers 2xx for a dead
  backend in some setups (a login page, a maintenance page). Use keyword
  monitors for anything a customer sees.
- Switching a monitor from a dead public URL to the internal one fires an
  *Up* notification the moment it works again. Expected, but it surprises
  people the first time.
- Kuma 1.x refuses monitor intervals above 24 days and the push monitor's
  interval is also its grace period: 90000 s means "alert after 25 hours
  without a ping", not "check every 25 hours".
- The status page is rendered server side for the domain root, the
  dashboard stays a single page app. A keyword monitor on the status page
  therefore works (the title is in the HTML), a keyword monitor on
  `/dashboard` does not.
- The *Domain Names* field only works when the request really arrives with
  that `Host` header. Behind Caddy that is the default; behind a proxy that
  rewrites the host header, set `trustProxy` in Kuma's settings and forward
  the header.
- The second Kuma uses the public ntfy.sh without authentication. Anyone who
  guesses the topic can read your alerts and post fake ones. Twelve random
  characters make guessing impractical; still, do not put secrets in monitor
  names.
- `louislam/uptime-kuma:1` is the 1.x line (1.23.17 at recording time).
  2.x exists (2.5.3) and migrates the database on first start. Pin what you
  test, upgrade on purpose, not by `docker compose pull`.


## What episode 16 found in this setup

- **ntfy rejects Kuma's message for monitors without a URL.** Ping, port,
  group and push monitors have no URL; Kuma still adds a "view" button and
  ntfy answers `400 parameter 'url' is required for action 'view'`. The alert
  never arrives, only the Kuma log shows it. Give those monitors a URL (any
  page, for example the status page); in the Kuma database that is the `url`
  column, the edit form hides it for these types.
- **Kuma only alerts on a change.** A monitor that stays down is announced
  once. Set "Resend Notification if Down X times consecutively" (for a 60 s
  interval, 60 means once an hour), otherwise a twelve day outage looks
  exactly like a one minute one.
- **Every Kuma restart announces every down monitor again.** After a restart
  the first beat is PENDING, the next DOWN counts as a change. Anything that
  opens tickets from Kuma must skip a DOWN for a monitor that is already open.