selfhostedops-stack/automation/README.md

Automation (episode 8)

Three workflows that wire the other services together. They are the reason n8n sits in this compose file, and they are all built from HTTP requests, so nothing here depends on a vendor connector that might change next quarter.

File	Trigger	What it does
a-deploy-notifier.json	Forgejo push webhook	Turns a push into one push notification: repo, branch, author, commit subject, link to the commit.
b-incident-responder.json	Uptime Kuma webhook notification	On DOWN: opens an incident document in the wiki and sends a high priority alert with a link to it. On UP: finds that document, appends the downtime, renames it to "Resolved", sends the all clear.
c-nightly-ops-digest.json	Schedule, 07:00 daily	Reads commits of the last 24 hours from Forgejo, availability from the Uptime Kuma status page, and the last restic snapshot from the episode 7 backup log. Writes one wiki document and sends a one line summary.

The point of this episode

Automation needs webhooks. Webhooks look like they need a public address. They do not, at least not for everything:

• Senders that live on the box (Forgejo, Uptime Kuma) call http://n8n:5678/webhook/... straight over the docker network. That traffic never touches the internet, needs no certificate and no port.
• Senders out on the internet need one public path, and exactly one. The Caddyfile publishes /webhook/*, /webhook-test/* and /webhook-waiting/* under n8n.<DOMAIN> and answers an empty 404 for everything else. The editor, the REST API and the login page are not reachable from outside at all.
• You reach the editor over the mesh from episode 5, on the container address, on plain HTTP inside the tunnel.

Two different 404s tell you which layer answered: an empty body is Caddy refusing the path, a JSON body is n8n saying the webhook is not registered.

Importing them

1. Create the three credentials in n8n first. Their names in these files are ntfy (n8n user), Outline API and Forgejo API; keep the names and n8n will attach them on import.
   • ntfy (n8n user): Basic Auth, an ntfy user with write access to the topics. docker compose exec -e NTFY_PASSWORD='...' ntfy ntfy user add --role=user n8n then ntfy access n8n alerts write-only and the same for deploys.
   • Outline API: Header Auth, name Authorization, value Bearer ol_api_... (Outline: Settings, API tokens).
   • Forgejo API: Header Auth, name Authorization, value token ... (a read-only token is enough: read:repository).
2. Import each JSON, then replace the placeholders:
   • REPLACE_WITH_YOUR_CREDENTIAL_ID is filled in when you pick the credential in the node.
   • REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID is the collection the incident and digest documents are written into. Create a collection, open it, the id is in collections.info or in the URL of the collection settings.
   • REPLACE_ON_IMPORT is the webhook id; n8n generates a fresh one.
   • ops.example.com appears in the two links that go into push messages.
3. Activate all three.

Wiring the senders

Forgejo, per repository: Settings, Webhooks, Forgejo, URL http://n8n:5678/webhook/forgejo-push, content type application/json, event Push.

Forgejo refuses to deliver to a private address unless you tell it not to. That is the FORGEJO__webhook__ALLOWED_HOST_LIST: "external,n8n" line in compose.yml. Without it the push is accepted, the workflow never runs, and the only trace is one line in the Forgejo log:

webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny 'n8n(172.18.0.3:5678)'

Uptime Kuma: Settings, Notifications, add a Webhook notification, URL http://n8n:5678/webhook/kuma-incident, content type application/json, then attach it to the monitors you want incident documents for. Kuma sends one POST per state change, with heartbeat.status 0 for down and 1 for up.

The one file that matters for backups

/home/node/.n8n/config holds encryptionKey. Every credential in the database is encrypted with it. The database alone is worth nothing:

Credentials could not be decrypted. The likely reason is that a different
"encryptionKey" was used to encrypt the data.

The workflows survive, the credentials do not, and n8n quietly writes a fresh key on first start when the file is missing, so nothing looks broken until a workflow runs. The episode 7 backup covers the whole n8n_data volume, so the key is in the snapshot. Verify it rather than assume it:

restic ls latest | grep n8n_data/_data/config

If you set N8N_ENCRYPTION_KEY in the environment instead, that value wins and the file is never written; then the key belongs in your password manager, and it has to survive independently of the box.

Notes

• c-nightly-ops-digest.json reads /backup/backup.log, which compose.yml bind-mounts read-only from episode 7. Without that file the digest reports "no backup log found" and still writes the document.
• N8N_RESTRICT_FILE_ACCESS_TO: /backup is what allows that read. Without the variable, 2.x answers "Access to the file is not allowed." for every path. Do not widen it to /: a workflow could then read the encryption key.
• The digest matches commit responses to repositories by position. That holds because the HTTP node returns exactly one item per input item when the response format is text and errors continue.
• These workflows use plain HTTP Request nodes on purpose. A Forgejo node, an Outline node and an ntfy node either do not exist or would add a dependency for three URL calls.

D - Ops triage agent (episode 9)

An agent that reads, and an automation that decides.

d-ops-triage-agent.json plus the two tool sub-workflows tool-monitor-status.json and tool-ops-log-search.json.

Uptime Kuma calls POST /webhook/ops-triage when a monitor changes state. A Code node reduces the Kuma body to five fields and writes the prompt. The AI Agent node then has three read-only tools:

Tool	What it reads
get_recent_commits	the last five commits on opsadmin/hello-ops
get_monitor_status	up/down and 24 hour uptime of every monitor
search_ops_log	the three best hits in the Ops Log wiki collection

The model runs on hardware you own. The Ollama credential points at a mesh address, not at a public one, and the machine that answers has no port open to the internet.

The part that matters

The agent never executes anything. It returns a JSON object with a field proposed_action. A Code node called Check allowlist holds a hand written map:

const ALLOWED = {
  restart_container: { target: 'your-container-name' },
  open_incident:     {},
  notify_only:       {},
};

Anything the model proposes that is not a key in that map becomes decision: 'rejected', and a Switch node routes it to a branch that writes the proposal down and does nothing else. The Switch never routes on the model's own string, only on decision.

For the one action that does change state, there is a second lock: ops-actions, a 85 line service in the ops-actions/ directory. It has the docker socket, and it has two allowlists of its own: which actions exist at all, and which containers may be touched. Start it with

docker compose --profile agent up -d ops-actions

Measured, on this stack: an action that is on both lists runs in about 0.4 s; an action that is not gets a 403 and one line in /var/log/ops-actions.log.

Two locks, because the first one has a known hole

A commit message, a monitor name and a wiki page are all things somebody else can write. All three end up in the model's context. In this lab, an instruction placed in a monitor name made a 3B model answer with "proposed_action": "stop_all_containers" in 10 out of 10 runs. The allowlist discarded all ten. Nothing was executed.

What the allowlist does not protect is the free text. summary goes into the wiki and onto a phone, and the model will happily put attacker supplied words there. Treat everything the agent writes as untrusted output, and cut it to length before it becomes a document title: Outline rejects titles over 100 characters, and a monitor name carrying a paragraph of injected text is exactly that.

Choosing a model

Small models differ less in what they say than in whether they use the tools at all and whether they keep the output contract. Measured on an integrated GPU, same incident, eight runs each:

Model	ran clean	used all 3 tools	action on the allowlist	run time
qwen3:1.7b	8/8	0/8	8/8	4.4 to 8.4 s
qwen2.5:3b	8/8	8/8	8/8	10.1 to 15.6 s
llama3.2:3b	8/8	7/8	4/8	7.0 to 15.3 s
qwen3:4b	4/5	1/5	4/5	291 to 569 s

qwen3:1.7b answers without ever calling a tool. llama3.2:3b copies the schema line and returns open_incident | notify_only as the action. qwen3:4b reasons for minutes. qwen2.5:3b is the smallest one that does the job here.

Do not switch on "Require Specific Output Format"

The Agent node's output parser adds an invisible tool named format_final_json_response. Small local models write that name into the answer text instead of calling it, and the run dies with "Model output doesn't fit required format" or "Model output wrapper is an empty object". Ask for the JSON in the system message instead and parse it yourself. The Check allowlist node in this export does that: it takes the outermost { ... } out of the answer, so a model that wraps its JSON in prose still works.

Keep the tool answers small

The first version called Uptime Kuma and Outline directly with an HTTP Request Tool. Kuma answers with 24 hours of heartbeats (30 kB), Outline with whole documents (15 kB for three hits). The context went from 347 to 27,758 tokens and the model stopped triaging and started summarising ping times. The node's "Optimize Response" did not help: Truncate Response only exists for text and html, and neither Field Containing Data nor the field selector reduced these two answers.

The fix is the two sub-workflows: HTTP node, then a Code node that keeps four lines per monitor and three snippets per search. Same information, 1,260 tokens instead of 27,758.

Sub-workflows used as tools must be active, otherwise every call comes back as "Workflow is not active and cannot be executed."

E and F - Security report, one agent vs a crew (episode 11)

The episode 9 agent read monitor state and triaged one incident. These two workflows read the security logs of the whole box once a day and write the report. They exist as a pair on purpose, to measure one question: does a 3B model do the whole job in one agent, or does splitting it into narrow roles make the report correct?

Both read through ops-readonly, the read-only twin of the episode 9 ops-actions runner (see ops-readonly/ and HARDENING.md). Neither agent gets a shell or a log file; both get a fixed list of questions.

File	Shape
e-security-report-solo.json	One agent, seven tools, one skill file (security-analyst-solo.md). It reads everything and writes the report itself.
f-security-report-crew.json	Three agents, one skill file each: fail2ban-reader.md reads SSH and fail2ban, caddy-log-reader.md reads the proxy log, report-writer.md turns both results into the report. The workflow itself is the orchestrator: the order is fixed, each agent gets exactly one skill as its system message.

Skills are files, loaded at run time

The four roles live in agents/skills/*.md. Each has frontmatter (name, description, tools), then the role, the allowed tools, an output schema and a short "what you do not do". The workflows fetch them at run time from this repo's own Forgejo over the docker network (http://forgejo:3000/api/v1/repos/opsadmin/selfhostedops-stack/raw/agents/skills/<name>.md, with the read-only Forgejo token), strip the frontmatter, and hand the body to the agent as its system message. Editing a role is editing a file and pushing it, not editing a workflow.

Wiring

1. Create the ops-readonly token credential in n8n: Header Auth, name x-ops-token, value from ops-readonly/token.env.
2. Import both JSON files, fill in the credential placeholders and REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID.
3. Trigger with POST /webhook/security-report-solo or /webhook/security-report-crew. Body {"deliver": false} runs the report without writing to the wiki or pushing (used for the measurement runs); the default writes the report to the Ops Log and sends one ntfy line.

The measured comparison (correct numbers, invented numbers, runtime, tokens, and a prompt-injection test through a request path in the log) is in lab-recordings-ep11/FACTS.md.

G - WAF exception advisor (episode 14)

The OWASP CRS from episode 13 blocks legitimate work until somebody writes route exclusions. This workflow reads what the WAF blocked and drafts those exclusions. It cannot apply them.

Start (webhook) -> Load skill -> Build prompts -> WAF exception advisor (agent)
                                                    |- waf_blocked     (ops-readonly)
                                                    |- waf_rule_info   (ops-readonly)
   -> Check proposal (code) -> Build issue (code) -> Deliver? -> Open issue for a human

Three decisions carry the design:

1. The model never writes rule syntax. It returns host, path, match type and rule ids as JSON. Build issue renders the SecLang from those fields after Check proposal has validated each one against what waf_blocked really returned in this run: host seen, plain path, not /, ids that fired on that route, never 949110/959100.
2. A format check cannot see intent. A proposal to exclude /.env on the vault is perfectly well formed. The only field in a blocked request that the sender cannot forge is the source address, so the workflow carries a list of trusted sources and the check drops every route on which no trusted source was ever blocked. Path, payload and auth header can all be copied.
3. The only write permission is an issue. The Forgejo WAF advisor (issues only) credential belongs to a bot user that is a read-only collaborator on one private repo, with the token scopes write:issue and read:repository. Measured: open an issue 201, write a file 403, read another private repo 404. A pull request would need push rights somewhere; an issue needs none. A human copies the snippet into appsec-rules/crs-route-exclusions.yaml and rolls it out.

Wiring

1. On the host: ops-readonly/waf-export.{service,timer} (see ops-readonly/waf-export.py for why this runs outside the container), then docker compose --profile agent up -d ops-readonly.
2. In Forgejo: a private repo for the exclusion file, a bot user with read access to it, a token with write:issue,read:repository. In n8n: a Header Auth credential Authorization: token <value>.
3. Import g-waf-exception-advisor.json, fill in the credential placeholders, adjust the repo in the last node.
4. Cap the output. Ollama has no limit by default and a 3B model can loop in a long JSON list; our first run generated for 11 min 47 s. numPredict on the Ollama node (3000 here; 1500 cut complete answers short), the same cap in a derived model tag as a second line (ollama create qwen2.5-advisor:3b, PARAMETER num_predict 3000), and executionTimeout in the workflow settings. After every edit through the REST API, deactivate and activate the workflow. A PATCH changes the draft; the published nodes keep running. We measured three runs with settings we believed were live and were not, the trigger token among them.
5. Put your trusted source IPs into TRUSTED_SOURCES at the top of the Build prompts node. They are part of the workflow on purpose. The webhook is reachable from outside; a caller who may name the trusted sources names himself, and the check would wave his own routes through.
6. Create the WAF advisor trigger credential (Header Auth, name x-advisor-token, a long random value). The webhook refuses calls without it: a run costs model time and, with deliver, opens an issue.
7. Trigger: POST /webhook/waf-exception-advisor with the token header and {"deliver": true}. {"deliver": false} runs everything except the issue.

The same proposal without a model

ops-readonly/waf-propose.py reads the same waf_blocked answer and groups by host, route and rule id, trusted sources only, at least two blocks. About twenty lines, 0.03 s, the same output every time. On the frozen measurement window it found 5 of 5 false positives and proposed nothing for an attack. Run it first. Numbers for both, and for the injection and mimicry tests, are in lab-recordings-ep14/FACTS.md.