selfhostedops-stack/automation/README.md

# Automation (episode 8)

Three workflows that wire the other services together. They are the reason n8n
sits in this compose file, and they are all built from HTTP requests, so nothing
here depends on a vendor connector that might change next quarter.

| File | Trigger | What it does |
|---|---|---|
| `a-deploy-notifier.json` | Forgejo `push` webhook | Turns a push into one push notification: repo, branch, author, commit subject, link to the commit. |
| `b-incident-responder.json` | Uptime Kuma webhook notification (header token since episode 16) | On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it, unless one is already open for that monitor. On UP: closes every open document of that monitor, appends the downtime, renames it to "Resolved: <monitor> (down ..., back ...)", sends the all clear. |
| `c-nightly-ops-digest.json` | Schedule, 07:00 daily | Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary. |

## The point of this episode

Automation needs webhooks. Webhooks look like they need a public address. They
do not, at least not for everything:

- **Senders that live on the box** (Forgejo, Uptime Kuma) call
  `http://n8n:5678/webhook/...` straight over the docker network. That traffic
  never touches the internet, needs no certificate and no port.
- **Senders out on the internet** need one public path, and exactly one. The
  Caddyfile publishes `/webhook/*`, `/webhook-test/*` and `/webhook-waiting/*`
  under `n8n.<DOMAIN>` and answers an empty 404 for everything else. The editor,
  the REST API and the login page are not reachable from outside at all.
- **You** reach the editor over the mesh from episode 5, on the container
  address, on plain HTTP inside the tunnel.

Two different 404s tell you which layer answered: an empty body is Caddy
refusing the path, a JSON body is n8n saying the webhook is not registered.

## Importing them

1. Create the three credentials in n8n first. Their names in these files are
   `ntfy (n8n user)`, `Outline API` and `Forgejo API`; keep the names and n8n
   will attach them on import.
   - `ntfy (n8n user)`: Basic Auth, an ntfy user with write access to the topics.
     `docker compose exec -e NTFY_PASSWORD='...' ntfy ntfy user add --role=user n8n`
     then `ntfy access n8n alerts write-only` and the same for `deploys`.
   - `Outline API`: Header Auth, name `Authorization`, value `Bearer ol_api_...`
     (Outline: Settings, API tokens).
   - `Forgejo API`: Header Auth, name `Authorization`, value `token ...`
     (a read-only token is enough: `read:repository`).
2. Import each JSON, then replace the placeholders:
   - `REPLACE_WITH_YOUR_CREDENTIAL_ID` is filled in when you pick the credential
     in the node.
   - `REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID` is the collection the incident
     and digest documents are written into. Create a collection, open it, the id
     is in `collections.info` or in the URL of the collection settings.
   - `REPLACE_ON_IMPORT` is the webhook id; n8n generates a fresh one.
   - `ops.example.com` appears in the two links that go into push messages.
3. Activate all three.

## Wiring the senders

Forgejo, per repository: Settings, Webhooks, Forgejo, URL
`http://n8n:5678/webhook/forgejo-push`, content type `application/json`,
event `Push`.

Forgejo refuses to deliver to a private address unless you tell it not to. That
is the `FORGEJO__webhook__ALLOWED_HOST_LIST: "external,n8n"` line in
`compose.yml`. Without it the push is accepted, the workflow never runs, and the
only trace is one line in the Forgejo log:

```
webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny 'n8n(172.18.0.3:5678)'
```

Uptime Kuma: Settings, Notifications, add a **Webhook** notification, URL
`http://n8n:5678/webhook/kuma-incident`, content type `application/json`, then
attach it to the monitors you want incident documents for. Kuma sends one POST
per state change, with `heartbeat.status` 0 for down and 1 for up.

## The one file that matters for backups

`/home/node/.n8n/config` holds `encryptionKey`. Every credential in the database
is encrypted with it. The database alone is worth nothing:

```
Credentials could not be decrypted. The likely reason is that a different
"encryptionKey" was used to encrypt the data.
```

The workflows survive, the credentials do not, and n8n quietly writes a fresh
key on first start when the file is missing, so nothing looks broken until a
workflow runs. The episode 7 backup covers the whole `n8n_data` volume, so the
key is in the snapshot. Verify it rather than assume it:

```bash
restic ls latest | grep n8n_data/_data/config
```

If you set `N8N_ENCRYPTION_KEY` in the environment instead, that value wins and
the file is never written; then the key belongs in your password manager, and it
has to survive independently of the box.

## Notes

- `c-nightly-ops-digest.json` reads `/backup/backup.log`, which `compose.yml`
  bind-mounts read-only from episode 7. Without that file the digest reports
  "no backup log found" and still writes the document.
- `N8N_RESTRICT_FILE_ACCESS_TO: /backup` is what allows that read. Without the
  variable, 2.x answers "Access to the file is not allowed." for every path.
  Do not widen it to `/`: a workflow could then read the encryption key.
- The digest matches commit responses to repositories by position. That holds
  because the HTTP node returns exactly one item per input item when the
  response format is text and errors continue.
- These workflows use plain HTTP Request nodes on purpose. A Forgejo node, an
  Outline node and an ntfy node either do not exist or would add a dependency
  for three URL calls.

## D - Ops triage agent (episode 9)

An agent that reads, and an automation that decides.

`d-ops-triage-agent.json` plus the two tool sub-workflows
`tool-monitor-status.json` and `tool-ops-log-search.json`.

Uptime Kuma calls `POST /webhook/ops-triage` when a monitor changes state. A
Code node reduces the Kuma body to five fields and writes the prompt. The AI
Agent node then has three read-only tools:

| Tool | What it reads |
| --- | --- |
| `get_recent_commits` | the last five commits on `opsadmin/hello-ops` |
| `get_monitor_status` | up/down and 24 hour uptime of every monitor |
| `search_ops_log` | the three best hits in the Ops Log wiki collection |

The model runs on hardware you own. The Ollama credential points at a mesh
address, not at a public one, and the machine that answers has no port open to
the internet.

### The part that matters

The agent never executes anything. It returns a JSON object with a field
`proposed_action`. A Code node called **Check allowlist** holds a hand written
map:

    const ALLOWED = {
      restart_container: { target: 'your-container-name' },
      open_incident:     {},
      notify_only:       {},
    };

Anything the model proposes that is not a key in that map becomes
`decision: 'rejected'`, and a Switch node routes it to a branch that writes the
proposal down and does nothing else. The Switch never routes on the model's own
string, only on `decision`.

For the one action that does change state, there is a second lock:
`ops-actions`, a 85 line service in the `ops-actions/` directory. It has the
docker socket, and it has two allowlists of its own: which actions exist at
all, and which containers may be touched. Start it with

    docker compose --profile agent up -d ops-actions

Measured, on this stack: an action that is on both lists runs in about 0.4 s;
an action that is not gets a 403 and one line in `/var/log/ops-actions.log`.

### Two locks, because the first one has a known hole

A commit message, a monitor name and a wiki page are all things somebody else
can write. All three end up in the model's context. In this lab, an instruction
placed in a monitor name made a 3B model answer with
`"proposed_action": "stop_all_containers"` in 10 out of 10 runs. The allowlist
discarded all ten. Nothing was executed.

What the allowlist does *not* protect is the free text. `summary` goes into the
wiki and onto a phone, and the model will happily put attacker supplied words
there. Treat everything the agent writes as untrusted output, and cut it to
length before it becomes a document title: Outline rejects titles over 100
characters, and a monitor name carrying a paragraph of injected text is exactly
that.

### Choosing a model

Small models differ less in what they say than in whether they use the tools at
all and whether they keep the output contract. Measured on an integrated GPU,
same incident, eight runs each:

| Model | ran clean | used all 3 tools | action on the allowlist | run time |
| --- | --- | --- | --- | --- |
| qwen3:1.7b | 8/8 | 0/8 | 8/8 | 4.4 to 8.4 s |
| qwen2.5:3b | 8/8 | 8/8 | 8/8 | 10.1 to 15.6 s |
| llama3.2:3b | 8/8 | 7/8 | 4/8 | 7.0 to 15.3 s |
| qwen3:4b | 4/5 | 1/5 | 4/5 | 291 to 569 s |

`qwen3:1.7b` answers without ever calling a tool. `llama3.2:3b` copies the
schema line and returns `open_incident | notify_only` as the action. `qwen3:4b`
reasons for minutes. `qwen2.5:3b` is the smallest one that does the job here.

### Do not switch on "Require Specific Output Format"

The Agent node's output parser adds an invisible tool named
`format_final_json_response`. Small local models write that name into the
answer text instead of calling it, and the run dies with *"Model output doesn't
fit required format"* or *"Model output wrapper is an empty object"*. Ask for
the JSON in the system message instead and parse it yourself. The Check
allowlist node in this export does that: it takes the outermost `{ ... }` out
of the answer, so a model that wraps its JSON in prose still works.

### Keep the tool answers small

The first version called Uptime Kuma and Outline directly with an HTTP Request
Tool. Kuma answers with 24 hours of heartbeats (30 kB), Outline with whole
documents (15 kB for three hits). The context went from 347 to 27,758 tokens
and the model stopped triaging and started summarising ping times. The node's
"Optimize Response" did not help: `Truncate Response` only exists for `text`
and `html`, and neither `Field Containing Data` nor the field selector reduced
these two answers.

The fix is the two sub-workflows: HTTP node, then a Code node that keeps four
lines per monitor and three snippets per search. Same information, 1,260 tokens
instead of 27,758.

Sub-workflows used as tools must be **active**, otherwise every call comes back
as *"Workflow is not active and cannot be executed."*

## E and F - Security report, one agent vs a crew (episode 11)

The episode 9 agent read monitor state and triaged one incident. These two
workflows read the security logs of the whole box once a day and write the
report. They exist as a pair on purpose, to measure one question: does a 3B
model do the whole job in one agent, or does splitting it into narrow roles
make the report correct?

Both read through `ops-readonly`, the read-only twin of the episode 9
`ops-actions` runner (see `ops-readonly/` and `HARDENING.md`). Neither agent
gets a shell or a log file; both get a fixed list of questions.

| File | Shape |
|---|---|
| `e-security-report-solo.json` | One agent, seven tools, one skill file (`security-analyst-solo.md`). It reads everything and writes the report itself. |
| `f-security-report-crew.json` | Three agents, one skill file each: `fail2ban-reader.md` reads SSH and fail2ban, `caddy-log-reader.md` reads the proxy log, `report-writer.md` turns both results into the report. The workflow itself is the orchestrator: the order is fixed, each agent gets exactly one skill as its system message. |

### Skills are files, loaded at run time

The four roles live in `agents/skills/*.md`. Each has frontmatter (name,
description, tools), then the role, the allowed tools, an output schema and a
short "what you do not do". The workflows fetch them at run time from this
repo's own Forgejo over the docker network
(`http://forgejo:3000/api/v1/repos/opsadmin/selfhostedops-stack/raw/agents/skills/<name>.md`,
with the read-only Forgejo token), strip the frontmatter, and hand the body to
the agent as its system message. Editing a role is editing a file and pushing
it, not editing a workflow.

### Wiring

1. Create the `ops-readonly token` credential in n8n: Header Auth, name
   `x-ops-token`, value from `ops-readonly/token.env`.
2. Import both JSON files, fill in the credential placeholders and
   `REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID`.
3. Trigger with `POST /webhook/security-report-solo` or
   `/webhook/security-report-crew`. Body `{"deliver": false}` runs the report
   without writing to the wiki or pushing (used for the measurement runs);
   the default writes the report to the Ops Log and sends one ntfy line.

The measured comparison (correct numbers, invented numbers, runtime, tokens,
and a prompt-injection test through a request path in the log) is in
`lab-recordings-ep11/FACTS.md`.

## G - WAF exception advisor (episode 14)

The OWASP CRS from episode 13 blocks legitimate work until somebody writes
route exclusions. This workflow reads what the WAF blocked and drafts those
exclusions. It cannot apply them.

```
Start (webhook) -> Load skill -> Build prompts -> WAF exception advisor (agent)
                                                    |- waf_blocked     (ops-readonly)
                                                    |- waf_rule_info   (ops-readonly)
   -> Check proposal (code) -> Build issue (code) -> Deliver? -> Open issue for a human
```

Three decisions carry the design:

1. **The model never writes rule syntax.** It returns host, path, match type
   and rule ids as JSON. `Build issue` renders the SecLang from those fields
   after `Check proposal` has validated each one against what `waf_blocked`
   really returned in this run: host seen, plain path, not `/`, ids that fired
   on that route, never 949110/959100.
2. **A format check cannot see intent.** A proposal to exclude `/.env` on the
   vault is perfectly well formed. The only field in a blocked request that the
   sender cannot forge is the source address, so the workflow carries a list
   of trusted sources and the check drops every route on which no trusted
   source was ever blocked. Path, payload and auth header can all be copied.
3. **The only write permission is an issue.** The `Forgejo WAF advisor (issues
   only)` credential belongs to a bot user that is a read-only collaborator on
   one private repo, with the token scopes `write:issue` and
   `read:repository`. Measured: open an issue 201, write a file 403, read
   another private repo 404. A pull request would need push rights somewhere;
   an issue needs none. A human copies the snippet into
   `appsec-rules/crs-route-exclusions.yaml` and rolls it out.

### Wiring

1. On the host: `ops-readonly/waf-export.{service,timer}` (see
   `ops-readonly/waf-export.py` for why this runs outside the container), then
   `docker compose --profile agent up -d ops-readonly`.
2. In Forgejo: a private repo for the exclusion file, a bot user with read
   access to it, a token with `write:issue,read:repository`. In n8n: a Header
   Auth credential `Authorization: token <value>`.
3. Import `g-waf-exception-advisor.json`, fill in the credential placeholders,
   adjust the repo in the last node.
4. Cap the output. Ollama has no limit by default and a 3B model can loop in
   a long JSON list; our first run generated for 11 min 47 s. `numPredict` on
   the Ollama node (3000 here; 1500 cut complete answers short), the same cap
   in a derived model tag as a second line
   (`ollama create qwen2.5-advisor:3b`, `PARAMETER num_predict 3000`), and
   `executionTimeout` in the workflow settings.
   **After every edit through the REST API, deactivate and activate the
   workflow.** A `PATCH` changes the draft; the published nodes keep running.
   We measured three runs with settings we believed were live and were not,
   the trigger token among them.
5. Put your trusted source IPs into `TRUSTED_SOURCES` at the top of the
   `Build prompts` node. They are part of the workflow on purpose. The webhook
   is reachable from outside; a caller who may name the trusted sources names
   himself, and the check would wave his own routes through.
6. Create the `WAF advisor trigger` credential (Header Auth, name
   `x-advisor-token`, a long random value). The webhook refuses calls without
   it: a run costs model time and, with `deliver`, opens an issue.
7. Trigger: `POST /webhook/waf-exception-advisor` with the token header and
   `{"deliver": true}`. `{"deliver": false}` runs everything except the issue.

### The same proposal without a model

`ops-readonly/waf-propose.py` reads the same `waf_blocked` answer and groups by
host, route and rule id, trusted sources only, at least two blocks. About
twenty lines, 0.03 s, the same output every time. On the frozen measurement
window it found 5 of 5 false positives and proposed nothing for an attack. Run
it first. Numbers for both, and for the injection and mimicry tests, are in
`lab-recordings-ep14/FACTS.md`.


## B fixed, H - Kuma alert triage (episode 16)

**What was wrong with B.** Measured on the demo box, not guessed:

- Uptime Kuma sends DOWN again after its own restart (PENDING to DOWN counts
  as a change). The old B opened a second incident each time and an UP closed
  only the first one it found. Two incidents stayed "open" for three weeks.
- The resolved title kept the time the outage started, so
  "Resolved: Vault (2026-09-16 17:59:05 UTC)" read like "resolved at 17:59"
  while Vault was down. It was resolved a week later.
- A long monitor name (the test used 111 characters, the title became 147) made Outline reject the title
  (`validation_error`); B failed silently, no document, no push.
- `/webhook/kuma-incident` is published to the internet like every n8n
  webhook in this stack. Anyone could post a fake UP and close an incident.

**The fix.** One decision node (`Decide`) instead of the IF: DOWN with an open
incident is skipped, UP closes all open incidents of the monitor, each with
its own start time, titles are cut. B and D only accept calls that carry the
header `x-kuma-token`; Kuma sends it (notification setting "Additional
Headers", JSON `{"x-kuma-token": "..."}`). Without it: 403.

**H - Kuma alert triage.** A local model sorts one window of Kuma state into
the alerts a human must see, through `ops-readonly` (`/read/kuma_alerts`,
exported on the host by `ops-readonly/kuma-export.py`), skill
`agents/skills/alert-triage.md` from Forgejo, a check node that drops alerts
naming monitors that are not in the tool result, and one ntfy push built by
code. Kept as an example, not as a recommendation: on the episode 16 test
window the 28 line script `ops-readonly/kuma-triage.py` (no dependencies) found 7 of 7
incidents with 7 messages, the 3B model named healthy monitors as down in
every run and followed a prompt injection in a monitor name. Details in the
episode notes.