Episode 16: Kuma alerts sorted by rules, a local agent and a script
ops-readonly gets kuma_alerts (host export kuma-export.py, one more read-only mount). kuma-triage.py sorts the same data without a model. Flow H runs the agent with skill alert-triage.md. Flow B: one open incident per monitor, an UP closes all of them, titles name start and end and stay under Outline's limit. B and D accept Kuma only with a header token. Monitoring notes: ntfy rejects Kuma messages for monitors without a URL, resend for long outages.
This commit is contained in:
Maximilian Grimm
parent
54ab43dac7
commit
1655d04d7d
10 changed files with 705 additions and 67 deletions
|
|
@ -7,7 +7,7 @@ here depends on a vendor connector that might change next quarter.
|
|||
| File | Trigger | What it does |
|
||||
|---|---|---|
|
||||
| `a-deploy-notifier.json` | Forgejo `push` webhook | Turns a push into one push notification: repo, branch, author, commit subject, link to the commit. |
|
||||
| `b-incident-responder.json` | Uptime Kuma webhook notification | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it. On UP: finds that document, appends the downtime, renames it to "Resolved", sends the all clear. |
|
||||
| `b-incident-responder.json` | Uptime Kuma webhook notification (header token since episode 16) | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it, unless one is already open for that monitor. On UP: closes every open document of that monitor, appends the downtime, renames it to "Resolved: <monitor> (down ..., back ...)", sends the all clear. |
|
||||
| `c-nightly-ops-digest.json` | Schedule, 07:00 daily | Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary. |
|
||||
|
||||
## The point of this episode
|
||||
|
|
@ -329,3 +329,36 @@ twenty lines, 0.03 s, the same output every time. On the frozen measurement
|
|||
window it found 5 of 5 false positives and proposed nothing for an attack. Run
|
||||
it first. Numbers for both, and for the injection and mimicry tests, are in
|
||||
`lab-recordings-ep14/FACTS.md`.
|
||||
|
||||
|
||||
## B fixed, H - Kuma alert triage (episode 16)
|
||||
|
||||
**What was wrong with B.** Measured on the demo box, not guessed:
|
||||
|
||||
- Uptime Kuma sends DOWN again after its own restart (PENDING to DOWN counts
|
||||
as a change). The old B opened a second incident each time and an UP closed
|
||||
only the first one it found. Two incidents stayed "open" for three weeks.
|
||||
- The resolved title kept the time the outage started, so
|
||||
"Resolved: Vault (2026-09-16 17:59:05 UTC)" read like "resolved at 17:59"
|
||||
while Vault was down. It was resolved a week later.
|
||||
- A long monitor name (the test used 111 characters, the title became 147) made Outline reject the title
|
||||
(`validation_error`); B failed silently, no document, no push.
|
||||
- `/webhook/kuma-incident` is published to the internet like every n8n
|
||||
webhook in this stack. Anyone could post a fake UP and close an incident.
|
||||
|
||||
**The fix.** One decision node (`Decide`) instead of the IF: DOWN with an open
|
||||
incident is skipped, UP closes all open incidents of the monitor, each with
|
||||
its own start time, titles are cut. B and D only accept calls that carry the
|
||||
header `x-kuma-token`; Kuma sends it (notification setting "Additional
|
||||
Headers", JSON `{"x-kuma-token": "..."}`). Without it: 403.
|
||||
|
||||
**H - Kuma alert triage.** A local model sorts one window of Kuma state into
|
||||
the alerts a human must see, through `ops-readonly` (`/read/kuma_alerts`,
|
||||
exported on the host by `ops-readonly/kuma-export.py`), skill
|
||||
`agents/skills/alert-triage.md` from Forgejo, a check node that drops alerts
|
||||
naming monitors that are not in the tool result, and one ntfy push built by
|
||||
code. Kept as an example, not as a recommendation: on the episode 16 test
|
||||
window the 28 line script `ops-readonly/kuma-triage.py` (no dependencies) found 7 of 7
|
||||
incidents with 7 messages, the 3B model named healthy monitors as down in
|
||||
every run and followed a prompt injection in a monitor name. Details in the
|
||||
episode notes.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,14 @@
|
|||
"httpMethod": "POST",
|
||||
"path": "kuma-incident",
|
||||
"responseMode": "onReceived",
|
||||
"options": {}
|
||||
"options": {},
|
||||
"authentication": "headerAuth"
|
||||
},
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "",
|
||||
"name": "Kuma webhook token"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -31,40 +38,6 @@
|
|||
"jsCode": "\n// Uptime Kuma posts one JSON body for every state change. Reduce it to the\n// four fields both branches need, so nothing below has to know Kuma's shape.\nconst b = $json.body ?? $json;\nconst hb = b.heartbeat ?? {};\nconst mon = b.monitor ?? {};\nconst down = Number(hb.status) === 0;\nreturn [{\n json: {\n monitor: mon.name ?? 'unknown monitor',\n monitorUrl: mon.url ?? '',\n down,\n at: hb.time ? new Date(hb.time.replace(' ', 'T') + 'Z').toISOString() : new Date().toISOString(),\n reason: hb.msg ?? b.msg ?? '',\n },\n}];\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "b3",
|
||||
"name": "Down?",
|
||||
"type": "n8n-nodes-base.if",
|
||||
"typeVersion": 2.2,
|
||||
"position": [
|
||||
220,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"conditions": {
|
||||
"options": {
|
||||
"caseSensitive": true,
|
||||
"leftValue": "",
|
||||
"typeValidation": "strict",
|
||||
"version": 2
|
||||
},
|
||||
"conditions": [
|
||||
{
|
||||
"id": "c1",
|
||||
"leftValue": "={{ $json.down }}",
|
||||
"rightValue": "",
|
||||
"operator": {
|
||||
"type": "boolean",
|
||||
"operation": "true",
|
||||
"singleValue": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"combinator": "and"
|
||||
},
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "b4",
|
||||
"name": "Build incident doc",
|
||||
|
|
@ -75,7 +48,7 @@
|
|||
-140
|
||||
],
|
||||
"parameters": {
|
||||
"jsCode": "\n// One incident document per outage. \"Status: open\" is the marker the UP\n// branch searches for later, so it stays machine readable on purpose.\nconst e = $json;\nconst stamp = e.at.replace('T', ' ').replace(/\\..*/, ' UTC');\nconst title = 'Incident: ' + e.monitor + ' (' + stamp + ')';\n// A markdown list, not plain lines: Outline collapses single newlines inside\n// a paragraph, which would turn the whole record into one unreadable blob.\nconst text = [\n '- Monitor: ' + e.monitor,\n '- Status: open',\n '- Down since: ' + e.at,\n '- Reason: ' + (e.reason || 'no detail from the monitor'),\n e.monitorUrl ? '- Checked URL: ' + e.monitorUrl : '',\n '',\n '---',\n '',\n 'Opened automatically by n8n. The resolution is appended when the monitor reports UP again.',\n].filter(Boolean).join('\\n');\nreturn [{ json: { title, payload: { title, text, collectionId: 'REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID', publish: true } } }];\n"
|
||||
"jsCode": "\n// One incident document per outage. \"Status: open\" is the marker the UP\n// branch searches for later, so it stays machine readable on purpose.\nconst e = $('Read event').first().json;\nconst stamp = e.at.replace('T', ' ').replace(/\\..*/, ' UTC');\nconst title = 'Incident: ' + e.monitor.slice(0, 60) + ' (' + stamp + ')';\n// A markdown list, not plain lines: Outline collapses single newlines inside\n// a paragraph, which would turn the whole record into one unreadable blob.\nconst text = [\n '- Monitor: ' + e.monitor,\n '- Status: open',\n '- Down since: ' + e.at,\n '- Reason: ' + (e.reason || 'no detail from the monitor'),\n e.monitorUrl ? '- Checked URL: ' + e.monitorUrl : '',\n '',\n '---',\n '',\n 'Opened automatically by n8n. The resolution is appended when the monitor reports UP again.',\n].filter(Boolean).join('\\n');\nreturn [{ json: { title, payload: { title, text, collectionId: 'REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID', publish: true } } }];\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -163,7 +136,7 @@
|
|||
"sendBody": true,
|
||||
"contentType": "json",
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ collectionId: \"REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID\", limit: 25 }) }}"
|
||||
"jsonBody": "={{ JSON.stringify({ collectionId: \"6352d2b9-48e9-41e0-9bdb-535db44b774a\", limit: 100 }) }}"
|
||||
},
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
|
|
@ -174,7 +147,7 @@
|
|||
},
|
||||
{
|
||||
"id": "b8",
|
||||
"name": "Find open incident",
|
||||
"name": "Decide",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
|
|
@ -182,7 +155,7 @@
|
|||
140
|
||||
],
|
||||
"parameters": {
|
||||
"jsCode": "\n// Find the still open incident for this monitor and close it out.\nconst e = $('Read event').first().json;\nconst docs = $input.first().json.data ?? [];\nconst open = docs.find(d =>\n String(d.text ?? '').includes('Monitor: ' + e.monitor) &&\n String(d.text ?? '').includes('Status: open'));\n\nif (!open) {\n return [{ json: { found: false, monitor: e.monitor, duration: 'unknown',\n note: 'no open incident document for this monitor' } }];\n}\n\nconst m = String(open.text).match(/Down since:\\s*(\\S+)/);\nconst since = m ? new Date(m[1]) : null;\nconst back = new Date(e.at);\nconst secs = since ? Math.max(0, Math.round((back - since) / 1000)) : null;\nconst human = secs === null ? 'unknown'\n : secs < 90 ? secs + ' s'\n : Math.floor(secs / 60) + ' min ' + (secs % 60) + ' s';\n\nconst text = String(open.text).replace('Status: open', 'Status: resolved')\n + '\\n- Back up: ' + e.at + '\\n- Downtime: ' + human + '\\n';\n\nreturn [{ json: {\n found: true, monitor: e.monitor, duration: human, seconds: secs,\n docUrl: open.url ?? '',\n payload: { id: open.id, title: String(open.title).replace('Incident:', 'Resolved:'), text, publish: true },\n}}];\n"
|
||||
"jsCode": "\n// Episode 16. Kuma repeats a DOWN after its own restart (PENDING -> DOWN is\n// \"important\"). Before this fix every DOWN opened a new incident and an UP\n// closed only one, so two incidents stayed \"open\" forever. Now: one open\n// incident per monitor, and an UP closes all of them.\nconst e = $('Read event').first().json;\nconst docs = $input.first().json.data ?? [];\nconst open = docs.filter(d =>\n String(d.text ?? '').includes('Monitor: ' + e.monitor) &&\n String(d.text ?? '').includes('Status: open'));\nconst fmt = s => String(s).replace('T', ' ').replace(/:\\d\\d(\\.\\d+)?Z$/, ' UTC');\nif (e.down) {\n if (open.length) return [{ json: { action: 'skip', monitor: e.monitor, note: 'already open: ' + open[0].title } }];\n return [{ json: { action: 'open', ...e } }];\n}\nif (!open.length) return [{ json: { action: 'skip', monitor: e.monitor, note: 'UP without an open incident' } }];\n// One title per document with its own start: an orphan from an older outage\n// must not borrow a start date from another one (lab16 first fix did, #385).\nconst fmt2 = d => fmt(d.toISOString());\nreturn open.map(d => {\n const m = String(d.text).match(/Down since:\\s*(\\S+)/); const since = m ? new Date(m[1]) : null;\n const secs = since ? Math.max(0, Math.round((new Date(e.at) - since) / 1000)) : null;\n const human = secs === null ? 'unknown' : secs < 90 ? secs + ' s' : Math.floor(secs / 60) + ' min ' + (secs % 60) + ' s';\n return { json: { action: 'close', monitor: e.monitor, duration: human, payload: { id: d.id, publish: true,\n title: 'Resolved: ' + e.monitor.slice(0, 40) + ' (down ' + (since ? fmt2(since) : '?') + ', back ' + fmt(e.at) + ')',\n text: String(d.text).replace('Status: open', 'Status: resolved') + '\\n- Back up: ' + e.at + '\\n- Downtime: ' + human + '\\n' } } };\n});\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -244,14 +217,113 @@
|
|||
"sendBody": true,
|
||||
"contentType": "raw",
|
||||
"rawContentType": "text/plain",
|
||||
"body": "=Downtime {{ $('Find open incident').first().json.duration }}. Incident closed."
|
||||
"body": "=Downtime {{ $('Decide').first().json.duration }}. Incident closed."
|
||||
},
|
||||
"credentials": {
|
||||
"httpBasicAuth": {
|
||||
"id": "REPLACE_WITH_YOUR_CREDENTIAL_ID",
|
||||
"name": "ntfy (n8n user)"
|
||||
}
|
||||
},
|
||||
"executeOnce": true
|
||||
},
|
||||
{
|
||||
"name": "Route",
|
||||
"type": "n8n-nodes-base.switch",
|
||||
"typeVersion": 3.2,
|
||||
"position": [
|
||||
700,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"rules": {
|
||||
"values": [
|
||||
{
|
||||
"conditions": {
|
||||
"options": {
|
||||
"caseSensitive": true,
|
||||
"leftValue": "",
|
||||
"typeValidation": "strict",
|
||||
"version": 2
|
||||
},
|
||||
"combinator": "and",
|
||||
"conditions": [
|
||||
{
|
||||
"id": "r0",
|
||||
"leftValue": "={{ $json.action }}",
|
||||
"rightValue": "open",
|
||||
"operator": {
|
||||
"type": "string",
|
||||
"operation": "equals"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"renameOutput": true,
|
||||
"outputKey": "open"
|
||||
},
|
||||
{
|
||||
"conditions": {
|
||||
"options": {
|
||||
"caseSensitive": true,
|
||||
"leftValue": "",
|
||||
"typeValidation": "strict",
|
||||
"version": 2
|
||||
},
|
||||
"combinator": "and",
|
||||
"conditions": [
|
||||
{
|
||||
"id": "r1",
|
||||
"leftValue": "={{ $json.action }}",
|
||||
"rightValue": "close",
|
||||
"operator": {
|
||||
"type": "string",
|
||||
"operation": "equals"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"renameOutput": true,
|
||||
"outputKey": "close"
|
||||
},
|
||||
{
|
||||
"conditions": {
|
||||
"options": {
|
||||
"caseSensitive": true,
|
||||
"leftValue": "",
|
||||
"typeValidation": "strict",
|
||||
"version": 2
|
||||
},
|
||||
"combinator": "and",
|
||||
"conditions": [
|
||||
{
|
||||
"id": "r2",
|
||||
"leftValue": "={{ $json.action }}",
|
||||
"rightValue": "skip",
|
||||
"operator": {
|
||||
"type": "string",
|
||||
"operation": "equals"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"renameOutput": true,
|
||||
"outputKey": "skip"
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Duplicate or nothing open",
|
||||
"type": "n8n-nodes-base.noOp",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
950,
|
||||
250
|
||||
],
|
||||
"parameters": {}
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
|
|
@ -270,14 +342,36 @@
|
|||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Down?",
|
||||
"node": "List Ops Log",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Down?": {
|
||||
"List Ops Log": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Decide",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Decide": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Route",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Route": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
|
|
@ -288,7 +382,14 @@
|
|||
],
|
||||
[
|
||||
{
|
||||
"node": "List Ops Log",
|
||||
"node": "Close incident doc",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"node": "Duplicate or nothing open",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
|
|
@ -317,28 +418,6 @@
|
|||
]
|
||||
]
|
||||
},
|
||||
"List Ops Log": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Find open incident",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Find open incident": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Close incident doc",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Close incident doc": {
|
||||
"main": [
|
||||
[
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@
|
|||
"parameters": {
|
||||
"httpMethod": "POST",
|
||||
"path": "ops-triage",
|
||||
"authentication": "headerAuth",
|
||||
"responseMode": "onReceived",
|
||||
"options": {}
|
||||
},
|
||||
|
|
@ -16,7 +17,13 @@
|
|||
-360,
|
||||
0
|
||||
],
|
||||
"webhookId": "e1633df9-d6dd-4c29-9c67-a9940d3018d6"
|
||||
"webhookId": "e1633df9-d6dd-4c29-9c67-a9940d3018d6",
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "",
|
||||
"name": "Kuma webhook token"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
|
|
|
|||
354
automation/h-kuma-alert-triage.json
Normal file
354
automation/h-kuma-alert-triage.json
Normal file
|
|
@ -0,0 +1,354 @@
|
|||
{
|
||||
"name": "H - Kuma alert triage",
|
||||
"nodes": [
|
||||
{
|
||||
"name": "Start",
|
||||
"type": "n8n-nodes-base.webhook",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"webhookId": "REPLACE_ON_IMPORT",
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "",
|
||||
"name": "Kuma triage trigger"
|
||||
}
|
||||
},
|
||||
"parameters": {
|
||||
"httpMethod": "POST",
|
||||
"path": "kuma-alert-triage",
|
||||
"authentication": "headerAuth",
|
||||
"responseMode": "onReceived",
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Load skill",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [
|
||||
220,
|
||||
0
|
||||
],
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "",
|
||||
"name": "Forgejo API"
|
||||
}
|
||||
},
|
||||
"parameters": {
|
||||
"method": "GET",
|
||||
"url": "http://forgejo:3000/api/v1/repos/opsadmin/selfhostedops-stack/raw/agents/skills/alert-triage.md",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "httpHeaderAuth",
|
||||
"options": {
|
||||
"response": {
|
||||
"response": {
|
||||
"responseFormat": "text"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Build prompts",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
440,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"jsCode": "\n// The skill file is markdown with a small frontmatter block for humans; the\n// model gets everything below it. Model tags come from a fixed list, never\n// from the caller.\nfunction body(md) { return String(md ?? '').replace(/^---[\\s\\S]*?\\n---\\s*/, '').trim(); }\nconst b = $('Start').first().json.body ?? {};\nconst MODELS = ['qwen2.5-advisor:3b', 'qwen3-advisor:4b'];\nconst model = MODELS.includes(b.model) ? b.model : MODELS[0];\n// qwen3 thinks before it answers: at 3000 tokens it never reached the JSON (lab16 runs #380, #402, #410)\nconst numPredict = model.startsWith('qwen3') ? 8000 : 3000;\nreturn [{ json: {\n system: body($json.data),\n task: 'Call kuma_alerts once, then answer with the JSON object described in your instructions.',\n model, numPredict, label: String(b.label ?? '').slice(0, 40), deliver: b.deliver === true,\n} }];\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Alert triage agent",
|
||||
"type": "@n8n/n8n-nodes-langchain.agent",
|
||||
"typeVersion": 2.2,
|
||||
"position": [
|
||||
680,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"promptType": "define",
|
||||
"text": "={{ $json.task }}",
|
||||
"hasOutputParser": false,
|
||||
"options": {
|
||||
"systemMessage": "={{ $json.system }}",
|
||||
"maxIterations": 4,
|
||||
"returnIntermediateSteps": true,
|
||||
"enableStreaming": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Ollama over the mesh",
|
||||
"type": "@n8n/n8n-nodes-langchain.lmChatOllama",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
600,
|
||||
240
|
||||
],
|
||||
"credentials": {
|
||||
"ollamaApi": {
|
||||
"id": "",
|
||||
"name": "Ollama (home lab over mesh)"
|
||||
}
|
||||
},
|
||||
"parameters": {
|
||||
"model": "={{ $('Build prompts').first().json.model }}",
|
||||
"options": {
|
||||
"temperature": 0,
|
||||
"numCtx": 12288,
|
||||
"numPredict": "={{ $('Build prompts').first().json.numPredict }}",
|
||||
"keepAlive": "30m",
|
||||
"think": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "kuma_alerts",
|
||||
"type": "@n8n/n8n-nodes-langchain.toolHttpRequest",
|
||||
"typeVersion": 1.1,
|
||||
"position": [
|
||||
800,
|
||||
240
|
||||
],
|
||||
"credentials": {
|
||||
"httpHeaderAuth": {
|
||||
"id": "",
|
||||
"name": "ops-readonly token"
|
||||
}
|
||||
},
|
||||
"parameters": {
|
||||
"toolDescription": "Every Uptime Kuma monitor for the current window: state now and since when, each status change in the window, heartbeat counts, TLS days left. Read only, no arguments. Call it once, first.",
|
||||
"method": "GET",
|
||||
"url": "http://ops-readonly:8098/read/kuma_alerts",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "httpHeaderAuth"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Check alerts",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
1020,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"jsCode": "\nfunction readObject(s) {\n if (s && typeof s === 'object') return s;\n const t = String(s ?? '');\n const a = t.indexOf('{'); const b = t.lastIndexOf('}');\n if (a === -1 || b <= a) return {};\n for (let end = b; end > a; end = t.lastIndexOf('}', end - 1)) {\n try { return JSON.parse(t.slice(a, end + 1)); } catch (e) { /* keep trying */ }\n }\n return {};\n}\n// Deterministic. Every alert is checked against what kuma_alerts returned in\n// THIS run. It filters, it never repairs: a missing alert stays missing.\nconst meta = $('Build prompts').first().json;\nconst raw = String($json.output ?? '');\nlet tool = null;\nfor (const s of ($json.intermediateSteps ?? [])) {\n if (s?.action?.tool === 'kuma_alerts') { try { tool = JSON.parse(s.observation); } catch (e) { /* not json */ } }\n}\nconst mons = new Map((tool?.monitors ?? []).map(m => [m.name, m]));\nconst KINDS = ['still_down', 'outage', 'recovered', 'flapping', 'cert_expiring'];\nconst got = readObject(raw);\nconst list = Array.isArray(got.alerts) ? got.alerts : [];\nconst accepted = []; const rejected = {};\nconst drop = r => { rejected[r] = (rejected[r] ?? 0) + 1; };\nif (!tool) drop('no kuma_alerts call in this run');\nfor (const a of (tool ? list : [])) {\n const names = Array.isArray(a?.monitors) ? a.monitors.map(String) : [];\n if (!KINDS.includes(a?.kind)) { drop('unknown kind'); continue; }\n if (!names.length || names.some(n => !mons.has(n))) { drop('monitor not in the tool result'); continue; }\n if (a.kind === 'recovered' && names.some(n => mons.get(n).now.status === 'DOWN')) { drop('recovered, but DOWN now'); continue; }\n accepted.push({ monitors: names, kind: a.kind });\n}\nreturn [{ json: { model: meta.model, label: meta.label, deliver: meta.deliver, tool_called: !!tool, raw, accepted, rejected } }];\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Build message",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
1240,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"jsCode": "\n// Code writes the message, never the model: kind and monitor names only. The\n// model's summary text is not sent, it is a channel for whatever the monitored\n// pages said.\nconst j = $json;\nconst L = { still_down: 'STILL DOWN', outage: 'DOWN', recovered: 'was down, up again', flapping: 'FLAPPING', cert_expiring: 'cert expiring' };\nconst lines = j.accepted.map(a => L[a.kind] + ': ' + a.monitors.map(n => n.slice(0, 60)).join(', '));\nconst dropped = Object.entries(j.rejected).map(([k, v]) => v + 'x ' + k).join('; ');\nreturn [{ json: { ...j, title: j.accepted.length + ' alert(s) from the triage agent (' + j.model + ')',\n message: (lines.join('\\n') || 'nothing needs a human') + (dropped ? '\\n(dropped: ' + dropped + ')' : '') } }];\n"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Deliver?",
|
||||
"type": "n8n-nodes-base.if",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
1460,
|
||||
0
|
||||
],
|
||||
"parameters": {
|
||||
"conditions": {
|
||||
"options": {
|
||||
"caseSensitive": true,
|
||||
"leftValue": "",
|
||||
"typeValidation": "loose",
|
||||
"version": 2
|
||||
},
|
||||
"conditions": [
|
||||
{
|
||||
"id": "c1",
|
||||
"leftValue": "={{ $json.deliver }}",
|
||||
"rightValue": "",
|
||||
"operator": {
|
||||
"type": "boolean",
|
||||
"operation": "true",
|
||||
"singleValue": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"combinator": "and"
|
||||
},
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Push to lab16",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [
|
||||
1680,
|
||||
-100
|
||||
],
|
||||
"credentials": {
|
||||
"httpBasicAuth": {
|
||||
"id": "",
|
||||
"name": "ntfy (n8n user)"
|
||||
}
|
||||
},
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "http://ntfy/lab16",
|
||||
"authentication": "genericCredentialType",
|
||||
"genericAuthType": "httpBasicAuth",
|
||||
"sendHeaders": true,
|
||||
"headerParameters": {
|
||||
"parameters": [
|
||||
{
|
||||
"name": "X-Title",
|
||||
"value": "={{ $json.title }}"
|
||||
},
|
||||
{
|
||||
"name": "X-Priority",
|
||||
"value": "4"
|
||||
}
|
||||
]
|
||||
},
|
||||
"sendBody": true,
|
||||
"contentType": "raw",
|
||||
"rawContentType": "text/plain",
|
||||
"body": "={{ $json.message }}",
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Not delivered (measurement run)",
|
||||
"type": "n8n-nodes-base.noOp",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
1680,
|
||||
100
|
||||
],
|
||||
"parameters": {}
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
"Start": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Load skill",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Load skill": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Build prompts",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Build prompts": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Alert triage agent",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Alert triage agent": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Check alerts",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Check alerts": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Build message",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Build message": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Deliver?",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Deliver?": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Push to lab16",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
],
|
||||
[
|
||||
{
|
||||
"node": "Not delivered (measurement run)",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Ollama over the mesh": {
|
||||
"ai_languageModel": [
|
||||
[
|
||||
{
|
||||
"node": "Alert triage agent",
|
||||
"type": "ai_languageModel",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"kuma_alerts": {
|
||||
"ai_tool": [
|
||||
[
|
||||
{
|
||||
"node": "Alert triage agent",
|
||||
"type": "ai_tool",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
},
|
||||
"settings": {
|
||||
"executionOrder": "v1",
|
||||
"executionTimeout": 600
|
||||
},
|
||||
"active": false
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue