selfhostedops/selfhostedops-stack
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Episode 11: a read-only agent that reads the security logs, one agent vs a crew of three

Browse source 
New: ops-readonly, the read-only twin of the episode 9 ops-actions runner
(fixed question list, three read-only mounts, no docker socket, read_only,
cap_drop ALL). Four role skills in agents/skills/, loaded at run time from
this repo's Forgejo. Two workflows: E (one agent, seven tools, one skill) and
F (three agents, one skill each, the workflow is the orchestrator).

Measured over 20 runs: no run of either variant produced all numbers and
lists correct. The solo agent confuses similar counts and skips a tool; the
crew calls every tool but drops fields at each role boundary. Sharpening the
skill files did not help the crew, it made it worse. Logs are attacker text:
an instruction in a request path is carried into the report as data, not
obeyed, but carried.

Claude-Session: https://claude.ai/code/session_017iP581CYtPfxdZYPPAyWZA
This commit is contained in:
Maximilian Grimm 2026-09-13 10:01:13 +02:00
parent bed41cfaf3
commit d147f21490
13 changed files with 2088 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

43
automation/README.md
View file

@ -214,3 +214,46 @@ instead of 27,758.
Sub-workflows used as tools must be **active**, otherwise every call comes back
as *"Workflow is not active and cannot be executed."*
## E and F - Security report, one agent vs a crew (episode 11)
The episode 9 agent read monitor state and triaged one incident. These two
workflows read the security logs of the whole box once a day and write the
report. They exist as a pair on purpose, to measure one question: does a 3B
model do the whole job in one agent, or does splitting it into narrow roles
make the report correct?
Both read through `ops-readonly`, the read-only twin of the episode 9
`ops-actions` runner (see `ops-readonly/` and `HARDENING.md`). Neither agent
gets a shell or a log file; both get a fixed list of questions.
| File | Shape |
|---|---|
| `e-security-report-solo.json` | One agent, seven tools, one skill file (`security-analyst-solo.md`). It reads everything and writes the report itself. |
| `f-security-report-crew.json` | Three agents, one skill file each: `fail2ban-reader.md` reads SSH and fail2ban, `caddy-log-reader.md` reads the proxy log, `report-writer.md` turns both results into the report. The workflow itself is the orchestrator: the order is fixed, each agent gets exactly one skill as its system message. |
### Skills are files, loaded at run time
The four roles live in `agents/skills/*.md`. Each has frontmatter (name,
description, tools), then the role, the allowed tools, an output schema and a
short "what you do not do". The workflows fetch them at run time from this
repo's own Forgejo over the docker network
(`http://forgejo:3000/api/v1/repos/opsadmin/selfhostedops-stack/raw/agents/skills/<name>.md`,
with the read-only Forgejo token), strip the frontmatter, and hand the body to
the agent as its system message. Editing a role is editing a file and pushing
it, not editing a workflow.
### Wiring
1. Create the `ops-readonly token` credential in n8n: Header Auth, name
`x-ops-token`, value from `ops-readonly/token.env`.
2. Import both JSON files, fill in the credential placeholders and
`REPLACE_WITH_YOUR_OUTLINE_COLLECTION_ID`.
3. Trigger with `POST /webhook/security-report-solo` or
`/webhook/security-report-crew`. Body `{"deliver": false}` runs the report
without writing to the wiki or pushing (used for the measurement runs);
the default writes the report to the Ops Log and sends one ntfy line.
The measured comparison (correct numbers, invented numbers, runtime, tokens,
and a prompt-injection test through a request path in the log) is in
`lab-recordings-ep11/FACTS.md`.