selfhostedops-stack/automation/README.md

Automation (episode 8)

Three workflows that wire the other services together. They are the reason n8n sits in this compose file, and they are all built from HTTP requests, so nothing here depends on a vendor connector that might change next quarter.

File	Trigger	What it does
a-deploy-notifier.json	Forgejo push webhook	Turns a push into one push notification: repo, branch, author, commit subject, link to the commit.
b-incident-responder.json	Uptime Kuma webhook notification	On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it. On UP: finds that document, appends the downtime, renames it to "Resolved", sends the all clear.
c-nightly-ops-digest.json	Schedule, 07:00 daily	Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary.

The point of this episode

Automation needs webhooks. Webhooks look like they need a public address. They do not, at least not for everything:

• Senders that live on the box (Forgejo, Uptime Kuma) call http://n8n:5678/webhook/... straight over the docker network. That traffic never touches the internet, needs no certificate and no port.
• Senders out on the internet need one public path, and exactly one. The Caddyfile publishes /webhook/*, /webhook-test/* and /webhook-waiting/* under n8n.<DOMAIN> and answers an empty 404 for everything else. The editor, the REST API and the login page are not reachable from outside at all.
• You reach the editor over the mesh from episode 5, on the container address, on plain HTTP inside the tunnel.

Two different 404s tell you which layer answered: an empty body is Caddy refusing the path, a JSON body is n8n saying the webhook is not registered.

Importing them

1. Create the three credentials in n8n first. Their names in these files are ntfy (n8n user), Outline API and Forgejo API; keep the names and n8n will attach them on import.
   • ntfy (n8n user): Basic Auth, an ntfy user with write access to the topics. docker compose exec -e NTFY_PASSWORD='...' ntfy ntfy user add --role=user n8n then ntfy access n8n alerts write-only and the same for deploys.
   • Outline API: Header Auth, name Authorization, value Bearer ol_api_... (Outline: Settings, API tokens).
   • Forgejo API: Header Auth, name Authorization, value token ... (a read-only token is enough: read:repository).
2. Import each JSON, then replace the placeholders:
   • REPLACE_WITH_YOUR_CREDENTIAL_ID is filled in when you pick the credential in the node.
   • REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID is the collection the incident and digest documents are written into. Create a collection, open it, the id is in collections.info or in the URL of the collection settings.
   • REPLACE_ON_IMPORT is the webhook id; n8n generates a fresh one.
   • ops.example.com appears in the two links that go into push messages.
3. Activate all three.

Wiring the senders

Forgejo, per repository: Settings, Webhooks, Forgejo, URL http://n8n:5678/webhook/forgejo-push, content type application/json, event Push.

Forgejo refuses to deliver to a private address unless you tell it not to. That is the FORGEJO__webhook__ALLOWED_HOST_LIST: "external,n8n" line in compose.yml. Without it the push is accepted, the workflow never runs, and the only trace is one line in the Forgejo log:

webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny 'n8n(172.18.0.3:5678)'

Uptime Kuma: Settings, Notifications, add a Webhook notification, URL http://n8n:5678/webhook/kuma-incident, content type application/json, then attach it to the monitors you want incident documents for. Kuma sends one POST per state change, with heartbeat.status 0 for down and 1 for up.

The one file that matters for backups

/home/node/.n8n/config holds encryptionKey. Every credential in the database is encrypted with it. The database alone is worth nothing:

Credentials could not be decrypted. The likely reason is that a different
"encryptionKey" was used to encrypt the data.

The workflows survive, the credentials do not, and n8n quietly writes a fresh key on first start when the file is missing, so nothing looks broken until a workflow runs. The episode 7 backup covers the whole n8n_data volume, so the key is in the snapshot. Verify it rather than assume it:

restic ls latest | grep n8n_data/_data/config

If you set N8N_ENCRYPTION_KEY in the environment instead, that value wins and the file is never written; then the key belongs in your password manager, and it has to survive independently of the box.

Notes

• c-nightly-ops-digest.json reads /backup/backup.log, which compose.yml bind-mounts read-only from episode 7. Without that file the digest reports "no backup log found" and still writes the document.
• N8N_RESTRICT_FILE_ACCESS_TO: /backup is what allows that read. Without the variable, 2.x answers "Access to the file is not allowed." for every path. Do not widen it to /: a workflow could then read the encryption key.
• The digest matches commit responses to repositories by position. That holds because the HTTP node returns exactly one item per input item when the response format is text and errors continue.
• These workflows use plain HTTP Request nodes on purpose. A Forgejo node, an Outline node and an ntfy node either do not exist or would add a dependency for three URL calls.

D - Ops triage agent (episode 9)

An agent that reads, and an automation that decides.

d-ops-triage-agent.json plus the two tool sub-workflows tool-monitor-status.json and tool-ops-log-search.json.

Uptime Kuma calls POST /webhook/ops-triage when a monitor changes state. A Code node reduces the Kuma body to five fields and writes the prompt. The AI Agent node then has three read-only tools:

Tool	What it reads
get_recent_commits	the last five commits on opsadmin/hello-ops
get_monitor_status	up/down and 24 hour uptime of every monitor
search_ops_log	the three best hits in the Ops Log wiki collection

The model runs on hardware you own. The Ollama credential points at a mesh address, not at a public one, and the machine that answers has no port open to the internet.

The part that matters

The agent never executes anything. It returns a JSON object with a field proposed_action. A Code node called Check allowlist holds a hand written map:

const ALLOWED = {
  restart_container: { target: 'your-container-name' },
  open_incident:     {},
  notify_only:       {},
};

Anything the model proposes that is not a key in that map becomes decision: 'rejected', and a Switch node routes it to a branch that writes the proposal down and does nothing else. The Switch never routes on the model's own string, only on decision.

For the one action that does change state, there is a second lock: ops-actions, a 85 line service in the ops-actions/ directory. It has the docker socket, and it has two allowlists of its own: which actions exist at all, and which containers may be touched. Start it with

docker compose --profile agent up -d ops-actions

Measured, on this stack: an action that is on both lists runs in about 0.4 s; an action that is not gets a 403 and one line in /var/log/ops-actions.log.

Two locks, because the first one has a known hole

A commit message, a monitor name and a wiki page are all things somebody else can write. All three end up in the model's context. In this lab, an instruction placed in a monitor name made a 3B model answer with "proposed_action": "stop_all_containers" in 10 out of 10 runs. The allowlist discarded all ten. Nothing was executed.

What the allowlist does not protect is the free text. summary goes into the wiki and onto a phone, and the model will happily put attacker supplied words there. Treat everything the agent writes as untrusted output, and cut it to length before it becomes a document title: Outline rejects titles over 100 characters, and a monitor name carrying a paragraph of injected text is exactly that.

Choosing a model

Small models differ less in what they say than in whether they use the tools at all and whether they keep the output contract. Measured on an integrated GPU, same incident, eight runs each:

Model	ran clean	used all 3 tools	action on the allowlist	run time
qwen3:1.7b	8/8	0/8	8/8	4.4 to 8.4 s
qwen2.5:3b	8/8	8/8	8/8	10.1 to 15.6 s
llama3.2:3b	8/8	7/8	4/8	7.0 to 15.3 s
qwen3:4b	4/5	1/5	4/5	291 to 569 s

qwen3:1.7b answers without ever calling a tool. llama3.2:3b copies the schema line and returns open_incident | notify_only as the action. qwen3:4b reasons for minutes. qwen2.5:3b is the smallest one that does the job here.

Do not switch on "Require Specific Output Format"

The Agent node's output parser adds an invisible tool named format_final_json_response. Small local models write that name into the answer text instead of calling it, and the run dies with "Model output doesn't fit required format" or "Model output wrapper is an empty object". Ask for the JSON in the system message instead and parse it yourself. The Check allowlist node in this export does that: it takes the outermost { ... } out of the answer, so a model that wraps its JSON in prose still works.

Keep the tool answers small

The first version called Uptime Kuma and Outline directly with an HTTP Request Tool. Kuma answers with 24 hours of heartbeats (30 kB), Outline with whole documents (15 kB for three hits). The context went from 347 to 27,758 tokens and the model stopped triaging and started summarising ping times. The node's "Optimize Response" did not help: Truncate Response only exists for text and html, and neither Field Containing Data nor the field selector reduced these two answers.

The fix is the two sub-workflows: HTTP node, then a Code node that keeps four lines per monitor and three snippets per search. Same information, 1,260 tokens instead of 27,758.

Sub-workflows used as tools must be active, otherwise every call comes back as "Workflow is not active and cannot be executed."