ccbddab46c
The agent runs in n8n on the box and reaches its model over the mesh from episode five, so no prompt leaves the network. It gets three read-only tools (recent commits, monitor status, wiki search) and returns one suggested action as a field. A switch compares that field against three allowed values, and only one of them reaches ops-actions, a small runner holding its own hard-coded allowlists for actions and targets. Measured while building this: the same prompt injection was harmless in every place the agent reached through a tool (twenty runs, no hits) and worked every time it arrived in the alert itself, through a poisoned monitor name (thirteen of thirteen). The allowlist rejected all thirteen. It guards the action field only, so the attacker-written summary still reached the wiki unchanged; the triage note now truncates and strips every field it did not write itself.
216 lines
10 KiB
Markdown
216 lines
10 KiB
Markdown
# Automation (episode 8)
|
|
|
|
Three workflows that wire the other services together. They are the reason n8n
|
|
sits in this compose file, and they are all built from HTTP requests, so nothing
|
|
here depends on a vendor connector that might change next quarter.
|
|
|
|
| File | Trigger | What it does |
|
|
|---|---|---|
|
|
| `a-deploy-notifier.json` | Forgejo `push` webhook | Turns a push into one push notification: repo, branch, author, commit subject, link to the commit. |
|
|
| `b-incident-responder.json` | Uptime Kuma webhook notification | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it. On UP: finds that document, appends the downtime, renames it to "Resolved", sends the all clear. |
|
|
| `c-nightly-ops-digest.json` | Schedule, 07:00 daily | Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary. |
|
|
|
|
## The point of this episode
|
|
|
|
Automation needs webhooks. Webhooks look like they need a public address. They
|
|
do not, at least not for everything:
|
|
|
|
- **Senders that live on the box** (Forgejo, Uptime Kuma) call
|
|
`http://n8n:5678/webhook/...` straight over the docker network. That traffic
|
|
never touches the internet, needs no certificate and no port.
|
|
- **Senders out on the internet** need one public path, and exactly one. The
|
|
Caddyfile publishes `/webhook/*`, `/webhook-test/*` and `/webhook-waiting/*`
|
|
under `n8n.<DOMAIN>` and answers an empty 404 for everything else. The editor,
|
|
the REST API and the login page are not reachable from outside at all.
|
|
- **You** reach the editor over the mesh from episode 5, on the container
|
|
address, on plain HTTP inside the tunnel.
|
|
|
|
Two different 404s tell you which layer answered: an empty body is Caddy
|
|
refusing the path, a JSON body is n8n saying the webhook is not registered.
|
|
|
|
## Importing them
|
|
|
|
1. Create the three credentials in n8n first. Their names in these files are
|
|
`ntfy (n8n user)`, `Outline API` and `Forgejo API`; keep the names and n8n
|
|
will attach them on import.
|
|
- `ntfy (n8n user)`: Basic Auth, an ntfy user with write access to the topics.
|
|
`docker compose exec -e NTFY_PASSWORD='...' ntfy ntfy user add --role=user n8n`
|
|
then `ntfy access n8n alerts write-only` and the same for `deploys`.
|
|
- `Outline API`: Header Auth, name `Authorization`, value `Bearer ol_api_...`
|
|
(Outline: Settings, API tokens).
|
|
- `Forgejo API`: Header Auth, name `Authorization`, value `token ...`
|
|
(a read-only token is enough: `read:repository`).
|
|
2. Import each JSON, then replace the placeholders:
|
|
- `REPLACE_WITH_YOUR_CREDENTIAL_ID` is filled in when you pick the credential
|
|
in the node.
|
|
- `REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID` is the collection the incident
|
|
and digest documents are written into. Create a collection, open it, the id
|
|
is in `collections.info` or in the URL of the collection settings.
|
|
- `REPLACE_ON_IMPORT` is the webhook id; n8n generates a fresh one.
|
|
- `ops.example.com` appears in the two links that go into push messages.
|
|
3. Activate all three.
|
|
|
|
## Wiring the senders
|
|
|
|
Forgejo, per repository: Settings, Webhooks, Forgejo, URL
|
|
`http://n8n:5678/webhook/forgejo-push`, content type `application/json`,
|
|
event `Push`.
|
|
|
|
Forgejo refuses to deliver to a private address unless you tell it not to. That
|
|
is the `FORGEJO__webhook__ALLOWED_HOST_LIST: "external,n8n"` line in
|
|
`compose.yml`. Without it the push is accepted, the workflow never runs, and the
|
|
only trace is one line in the Forgejo log:
|
|
|
|
```
|
|
webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny 'n8n(172.18.0.3:5678)'
|
|
```
|
|
|
|
Uptime Kuma: Settings, Notifications, add a **Webhook** notification, URL
|
|
`http://n8n:5678/webhook/kuma-incident`, content type `application/json`, then
|
|
attach it to the monitors you want incident documents for. Kuma sends one POST
|
|
per state change, with `heartbeat.status` 0 for down and 1 for up.
|
|
|
|
## The one file that matters for backups
|
|
|
|
`/home/node/.n8n/config` holds `encryptionKey`. Every credential in the database
|
|
is encrypted with it. The database alone is worth nothing:
|
|
|
|
```
|
|
Credentials could not be decrypted. The likely reason is that a different
|
|
"encryptionKey" was used to encrypt the data.
|
|
```
|
|
|
|
The workflows survive, the credentials do not, and n8n quietly writes a fresh
|
|
key on first start when the file is missing, so nothing looks broken until a
|
|
workflow runs. The episode 7 backup covers the whole `n8n_data` volume, so the
|
|
key is in the snapshot. Verify it rather than assume it:
|
|
|
|
```bash
|
|
restic ls latest | grep n8n_data/_data/config
|
|
```
|
|
|
|
If you set `N8N_ENCRYPTION_KEY` in the environment instead, that value wins and
|
|
the file is never written; then the key belongs in your password manager, and it
|
|
has to survive independently of the box.
|
|
|
|
## Notes
|
|
|
|
- `c-nightly-ops-digest.json` reads `/backup/backup.log`, which `compose.yml`
|
|
bind-mounts read-only from episode 7. Without that file the digest reports
|
|
"no backup log found" and still writes the document.
|
|
- `N8N_RESTRICT_FILE_ACCESS_TO: /backup` is what allows that read. Without the
|
|
variable, 2.x answers "Access to the file is not allowed." for every path.
|
|
Do not widen it to `/`: a workflow could then read the encryption key.
|
|
- The digest matches commit responses to repositories by position. That holds
|
|
because the HTTP node returns exactly one item per input item when the
|
|
response format is text and errors continue.
|
|
- These workflows use plain HTTP Request nodes on purpose. A Forgejo node, an
|
|
Outline node and an ntfy node either do not exist or would add a dependency
|
|
for three URL calls.
|
|
|
|
## D - Ops triage agent (episode 9)
|
|
|
|
An agent that reads, and an automation that decides.
|
|
|
|
`d-ops-triage-agent.json` plus the two tool sub-workflows
|
|
`tool-monitor-status.json` and `tool-ops-log-search.json`.
|
|
|
|
Uptime Kuma calls `POST /webhook/ops-triage` when a monitor changes state. A
|
|
Code node reduces the Kuma body to five fields and writes the prompt. The AI
|
|
Agent node then has three read-only tools:
|
|
|
|
| Tool | What it reads |
|
|
| --- | --- |
|
|
| `get_recent_commits` | the last five commits on `opsadmin/hello-ops` |
|
|
| `get_monitor_status` | up/down and 24 hour uptime of every monitor |
|
|
| `search_ops_log` | the three best hits in the Ops Log wiki collection |
|
|
|
|
The model runs on hardware you own. The Ollama credential points at a mesh
|
|
address, not at a public one, and the machine that answers has no port open to
|
|
the internet.
|
|
|
|
### The part that matters
|
|
|
|
The agent never executes anything. It returns a JSON object with a field
|
|
`proposed_action`. A Code node called **Check allowlist** holds a hand written
|
|
map:
|
|
|
|
const ALLOWED = {
|
|
restart_container: { target: 'your-container-name' },
|
|
open_incident: {},
|
|
notify_only: {},
|
|
};
|
|
|
|
Anything the model proposes that is not a key in that map becomes
|
|
`decision: 'rejected'`, and a Switch node routes it to a branch that writes the
|
|
proposal down and does nothing else. The Switch never routes on the model's own
|
|
string, only on `decision`.
|
|
|
|
For the one action that does change state, there is a second lock:
|
|
`ops-actions`, a 85 line service in the `ops-actions/` directory. It has the
|
|
docker socket, and it has two allowlists of its own: which actions exist at
|
|
all, and which containers may be touched. Start it with
|
|
|
|
docker compose --profile agent up -d ops-actions
|
|
|
|
Measured, on this stack: an action that is on both lists runs in about 0.4 s;
|
|
an action that is not gets a 403 and one line in `/var/log/ops-actions.log`.
|
|
|
|
### Two locks, because the first one has a known hole
|
|
|
|
A commit message, a monitor name and a wiki page are all things somebody else
|
|
can write. All three end up in the model's context. In this lab, an instruction
|
|
placed in a monitor name made a 3B model answer with
|
|
`"proposed_action": "stop_all_containers"` in 10 out of 10 runs. The allowlist
|
|
discarded all ten. Nothing was executed.
|
|
|
|
What the allowlist does *not* protect is the free text. `summary` goes into the
|
|
wiki and onto a phone, and the model will happily put attacker supplied words
|
|
there. Treat everything the agent writes as untrusted output, and cut it to
|
|
length before it becomes a document title: Outline rejects titles over 100
|
|
characters, and a monitor name carrying a paragraph of injected text is exactly
|
|
that.
|
|
|
|
### Choosing a model
|
|
|
|
Small models differ less in what they say than in whether they use the tools at
|
|
all and whether they keep the output contract. Measured on an integrated GPU,
|
|
same incident, eight runs each:
|
|
|
|
| Model | ran clean | used all 3 tools | action on the allowlist | run time |
|
|
| --- | --- | --- | --- | --- |
|
|
| qwen3:1.7b | 8/8 | 0/8 | 8/8 | 4.4 to 8.4 s |
|
|
| qwen2.5:3b | 8/8 | 8/8 | 8/8 | 10.1 to 15.6 s |
|
|
| llama3.2:3b | 8/8 | 7/8 | 4/8 | 7.0 to 15.3 s |
|
|
| qwen3:4b | 4/5 | 1/5 | 4/5 | 291 to 569 s |
|
|
|
|
`qwen3:1.7b` answers without ever calling a tool. `llama3.2:3b` copies the
|
|
schema line and returns `open_incident | notify_only` as the action. `qwen3:4b`
|
|
reasons for minutes. `qwen2.5:3b` is the smallest one that does the job here.
|
|
|
|
### Do not switch on "Require Specific Output Format"
|
|
|
|
The Agent node's output parser adds an invisible tool named
|
|
`format_final_json_response`. Small local models write that name into the
|
|
answer text instead of calling it, and the run dies with *"Model output doesn't
|
|
fit required format"* or *"Model output wrapper is an empty object"*. Ask for
|
|
the JSON in the system message instead and parse it yourself. The Check
|
|
allowlist node in this export does that: it takes the outermost `{ ... }` out
|
|
of the answer, so a model that wraps its JSON in prose still works.
|
|
|
|
### Keep the tool answers small
|
|
|
|
The first version called Uptime Kuma and Outline directly with an HTTP Request
|
|
Tool. Kuma answers with 24 hours of heartbeats (30 kB), Outline with whole
|
|
documents (15 kB for three hits). The context went from 347 to 27,758 tokens
|
|
and the model stopped triaging and started summarising ping times. The node's
|
|
"Optimize Response" did not help: `Truncate Response` only exists for `text`
|
|
and `html`, and neither `Field Containing Data` nor the field selector reduced
|
|
these two answers.
|
|
|
|
The fix is the two sub-workflows: HTTP node, then a Code node that keeps four
|
|
lines per monitor and three snippets per search. Same information, 1,260 tokens
|
|
instead of 27,758.
|
|
|
|
Sub-workflows used as tools must be **active**, otherwise every call comes back
|
|
as *"Workflow is not active and cannot be executed."*
|