Episode 16: Kuma alerts sorted by rules, a local agent and a script
ops-readonly gets kuma_alerts (host export kuma-export.py, one more read-only mount). kuma-triage.py sorts the same data without a model. Flow H runs the agent with skill alert-triage.md. Flow B: one open incident per monitor, an UP closes all of them, titles name start and end and stay under Outline's limit. B and D accept Kuma only with a header token. Monitoring notes: ntfy rejects Kuma messages for monitors without a URL, resend for long outages.
This commit is contained in:
parent
54ab43dac7
commit
1655d04d7d
10 changed files with 705 additions and 67 deletions
|
|
@ -7,7 +7,7 @@ here depends on a vendor connector that might change next quarter.
|
|||
| File | Trigger | What it does |
|
||||
|---|---|---|
|
||||
| `a-deploy-notifier.json` | Forgejo `push` webhook | Turns a push into one push notification: repo, branch, author, commit subject, link to the commit. |
|
||||
| `b-incident-responder.json` | Uptime Kuma webhook notification | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it. On UP: finds that document, appends the downtime, renames it to "Resolved", sends the all clear. |
|
||||
| `b-incident-responder.json` | Uptime Kuma webhook notification (header token since episode 16) | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it, unless one is already open for that monitor. On UP: closes every open document of that monitor, appends the downtime, renames it to "Resolved: <monitor> (down ..., back ...)", sends the all clear. |
|
||||
| `c-nightly-ops-digest.json` | Schedule, 07:00 daily | Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary. |
|
||||
|
||||
## The point of this episode
|
||||
|
|
@ -329,3 +329,36 @@ twenty lines, 0.03 s, the same output every time. On the frozen measurement
|
|||
window it found 5 of 5 false positives and proposed nothing for an attack. Run
|
||||
it first. Numbers for both, and for the injection and mimicry tests, are in
|
||||
`lab-recordings-ep14/FACTS.md`.
|
||||
|
||||
|
||||
## B fixed, H - Kuma alert triage (episode 16)
|
||||
|
||||
**What was wrong with B.** Measured on the demo box, not guessed:
|
||||
|
||||
- Uptime Kuma sends DOWN again after its own restart (PENDING to DOWN counts
|
||||
as a change). The old B opened a second incident each time and an UP closed
|
||||
only the first one it found. Two incidents stayed "open" for three weeks.
|
||||
- The resolved title kept the time the outage started, so
|
||||
"Resolved: Vault (2026-09-16 17:59:05 UTC)" read like "resolved at 17:59"
|
||||
while Vault was down. It was resolved a week later.
|
||||
- A long monitor name (the test used 111 characters, the title became 147) made Outline reject the title
|
||||
(`validation_error`); B failed silently, no document, no push.
|
||||
- `/webhook/kuma-incident` is published to the internet like every n8n
|
||||
webhook in this stack. Anyone could post a fake UP and close an incident.
|
||||
|
||||
**The fix.** One decision node (`Decide`) instead of the IF: DOWN with an open
|
||||
incident is skipped, UP closes all open incidents of the monitor, each with
|
||||
its own start time, titles are cut. B and D only accept calls that carry the
|
||||
header `x-kuma-token`; Kuma sends it (notification setting "Additional
|
||||
Headers", JSON `{"x-kuma-token": "..."}`). Without it: 403.
|
||||
|
||||
**H - Kuma alert triage.** A local model sorts one window of Kuma state into
|
||||
the alerts a human must see, through `ops-readonly` (`/read/kuma_alerts`,
|
||||
exported on the host by `ops-readonly/kuma-export.py`), skill
|
||||
`agents/skills/alert-triage.md` from Forgejo, a check node that drops alerts
|
||||
naming monitors that are not in the tool result, and one ntfy push built by
|
||||
code. Kept as an example, not as a recommendation: on the episode 16 test
|
||||
window the 28 line script `ops-readonly/kuma-triage.py` (no dependencies) found 7 of 7
|
||||
incidents with 7 messages, the 3B model named healthy monitors as down in
|
||||
every run and followed a prompt injection in a monitor name. Details in the
|
||||
episode notes.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue