selfhostedops/selfhostedops-stack
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Episode 16: Kuma alerts sorted by rules, a local agent and a script

Browse source 
ops-readonly gets kuma_alerts (host export kuma-export.py, one more read-only
mount). kuma-triage.py sorts the same data without a model. Flow H runs the
agent with skill alert-triage.md. Flow B: one open incident per monitor, an
UP closes all of them, titles name start and end and stay under Outline's
limit. B and D accept Kuma only with a header token. Monitoring notes: ntfy
rejects Kuma messages for monitors without a URL, resend for long outages.
This commit is contained in:
Maximilian Grimm 2026-09-29 22:14:50 +02:00
parent 54ab43dac7
commit 1655d04d7d
10 changed files with 705 additions and 67 deletions
Download patch file Download diff file Expand all files Collapse all files

28
ops-readonly/kuma-triage.py Normal file
View file

@ -0,0 +1,28 @@
#!/usr/bin/env python3
# kuma-triage.py < kuma_alerts.json (episode 16): the same answer as the agent, no model.
# Down now = alert (still_down if it started before the window). 3+ drops in the window = flapping.
# Recovered drops are reported once. Cert < 14 days. Same host + same kind + start within 90 s = one alert.
import json, sys, datetime as dt
d = json.load(sys.stdin); t0 = d["window"]["from"][:19]; out = []
ts = lambda s: dt.datetime.fromisoformat(s.replace("T", " ")[:19])
for m in d["monitors"]:
if m["type"] == "group" or not m["active"]:
continue
downs = [c["t"] for c in m["changes"] if c["status"] == "DOWN"]
host = m["target"].split("//")[-1].split("/")[0].split(":")[0]
if m["now"]["status"] == "DOWN":
out.append(dict(kind="still_down" if m["now"]["since"] < t0 else "outage", start=m["now"]["since"], host=host, monitors=[m["name"]], summary="down since %s: %s" % (m["now"]["since"], m["now"]["msg"][:80])))
elif len(downs) >= 3:
out.append(dict(kind="flapping", start=downs[0], host=host, monitors=[m["name"]], summary="%d drops in the window" % len(downs)))
elif downs:
out.append(dict(kind="recovered", start=downs[0], host=host, monitors=[m["name"]], summary="was down from %s, up again" % downs[0]))
if m["tls_days_left"] is not None and m["tls_days_left"] < 14:
out.append(dict(kind="cert_expiring", start="", host=host, monitors=[m["name"]], summary="certificate expires in %d days" % m["tls_days_left"]))
merged = []
for a in sorted(out, key=lambda a: (a["kind"], a["host"], a["start"])):
p = merged[-1] if merged else None
if p and a["kind"] == p["kind"] != "cert_expiring" and a["host"] == p["host"] and abs((ts(a["start"]) - ts(p["start"])).total_seconds()) <= 90:
p["monitors"] += a["monitors"]
else:
merged.append(a)
print(json.dumps({"alerts": [{k: a[k] for k in ("monitors", "kind", "summary")} for a in merged]}, indent=1))