3edef38ba3
ops-readonly gets two read-only questions, waf_blocked and waf_rule_info. The rule ids per blocked request are in CrowdSec's alerts, not in its log, and alerts need machine credentials that can also write, so waf-export.py runs on the host and the container gets one more read-only mount. Workflow G lets a local 3B model draft CRS route exclusions as a Forgejo issue. The model never writes rule syntax, a code node checks every field against the tool output, the only write permission is an issue in one repo, trusted sources live in the workflow and the trigger needs a token. waf-propose.py does the same without a model: 5 of 5 false positives, no exclusion for an attack, 0.03 s. The model proposed an exclusion for nearly every attack it saw. Numbers and the traps we hit are in HARDENING.md.
268 lines
12 KiB
Markdown
268 lines
12 KiB
Markdown
# ~$ selfhostedops-stack
|
|
|
|
**The €15/month self-hosted company stack.** Git hosting, wiki, password manager,
|
|
monitoring, automation and VPN — one server, one compose file, no per-seat pricing.
|
|
|
|
From the video: **[Self-Hosted Company Stack for €15/Month (Full Setup)](https://youtube.com/@selfhostedops)**
|
|
— Series *Self-Hosted Company OS*, Episode 1.
|
|
|
|
```
|
|
┌──────────────────────────────────────┐
|
|
│ caddy :80/:443 │
|
|
│ automatic HTTPS for *.DOMAIN │
|
|
└──┬─────┬───────┬───────┬──────┬──────┘
|
|
│ │ │ │ │
|
|
git.─────► │ wiki.──► │ status.──► │
|
|
│ │ │ │ │
|
|
┌────▼─┐ ┌─▼────┐ ┌▼─────┐ ┌▼───┐ ┌▼────┐
|
|
│forge │ │outline│ │vault │ │kuma│ │ n8n │
|
|
│ jo │ │+pg+redis│warden│ │ │ │ │
|
|
└──────┘ └──────┘ └──────┘ └────┘ └─────┘
|
|
also: wg-easy (optional, --profile vpn)
|
|
```
|
|
|
|
| Service | URL | Replaces | Typical SaaS price |
|
|
|---|---|---|---|
|
|
| [Forgejo](https://forgejo.org) | `git.…` | GitHub Team | $4/user/mo |
|
|
| [Outline](https://www.getoutline.com) | `wiki.…` | Notion | $10/user/mo |
|
|
| [Vaultwarden](https://github.com/dani-garcia/vaultwarden) | `vault.…` | 1Password | $8/user/mo |
|
|
| [Uptime Kuma](https://github.com/louislam/uptime-kuma) | `status.…` | Pingdom & co. | $20+/mo |
|
|
| [n8n](https://n8n.io) | `n8n.…` (webhooks only, episode 8) | Zapier | €26+/mo |
|
|
| [wg-easy](https://github.com/wg-easy/wg-easy) | `vpn.…` | Tailscale | $6/user/mo |
|
|
|
|
All images are multi-arch (amd64 + arm64) — a €14 ARM cloud box (e.g. Hetzner
|
|
CAX31) runs the whole thing with room to spare.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
# 0. A Debian/Ubuntu server with Docker:
|
|
curl -fsSL https://get.docker.com | sh
|
|
|
|
# 1. DNS: ONE wildcard record *.ops.example.com -> your server IP (A + AAAA)
|
|
|
|
# 2. Configure
|
|
git clone https://github.com/selfhostedops/selfhostedops-stack.git
|
|
cd selfhostedops-stack
|
|
cp .env.example .env
|
|
nano .env # domain, email, 3x openssl rand -hex 32
|
|
|
|
# 3. Launch
|
|
docker compose up -d
|
|
```
|
|
|
|
Ninety seconds later: `git.` `wiki.` `vault.` `status.` `n8n.` — all with valid HTTPS.
|
|
|
|
## First-run order (5 minutes)
|
|
|
|
1. **Forgejo** (`git.…`): create the admin account on the setup page.
|
|
Registration is already disabled via env — this is a company server.
|
|
2. **Forgejo → SSO for the wiki:** Site Administration → Applications →
|
|
Create OAuth2 app. Redirect URI: `https://wiki.<DOMAIN>/auth/oidc.callback`.
|
|
Put the client id/secret into `.env`, then `docker compose up -d outline`.
|
|
3. **Outline** (`wiki.…`): click "Continue with Forgejo". Your git server is
|
|
your SSO provider — one account, both tools.
|
|
4. **Vaultwarden** (`vault.…`): create your account, install the Bitwarden
|
|
apps against `https://vault.<DOMAIN>`. When the team is on board:
|
|
`VAULTWARDEN_SIGNUPS_ALLOWED=false` and re-up.
|
|
5. **Uptime Kuma** (`status.…`): create admin, add HTTPS monitors for the
|
|
other five subdomains. Meta, but correct.
|
|
6. **n8n** (`n8n.…`): create the owner account. This thing becomes the brain
|
|
of the stack in later episodes (AI agents incoming).
|
|
|
|
## FAQ
|
|
|
|
**Why no Netbird? You said mesh VPN.**
|
|
Full self-hosted Netbird (management plane, signal, TURN, IdP) deserves its own
|
|
video — that's Episode 5. `wg-easy` gets you a working WireGuard VPN today with
|
|
one env var. Skip both if you don't need remote access yet.
|
|
|
|
**Backups?**
|
|
`docker compose down`, snapshot/borg the named volumes, `up -d`. A proper
|
|
3-2-1 setup (borgmatic + offsite) is Episode 10. Until then: at minimum enable
|
|
your host provider's snapshot backups.
|
|
|
|
**Can I run only some services?**
|
|
Yes — comment out what you don't need in `compose.yml` and remove its Caddyfile
|
|
block. Everything is independent except Outline → Forgejo (SSO) and
|
|
Outline → its postgres/redis.
|
|
|
|
**Is `latest` okay for images?**
|
|
For a homelab: pragmatic. For production: pin versions and read HARDENING.md.
|
|
|
|
## CI: your own runners (Episode 2)
|
|
|
|
Forgejo ships Actions. Turn it on, give it a runner, and you have GitHub Actions
|
|
with no minute counter.
|
|
|
|
```bash
|
|
# 1. Enable it (already set in compose.yml)
|
|
# FORGEJO__actions__ENABLED and repo.actions in DEFAULT_REPO_UNITS.
|
|
# Without repo.actions in the default units, the Actions tab stays hidden
|
|
# and it looks like the feature is broken. Existing repos: flip it per repo.
|
|
|
|
# 2. One shared secret for declarative registration
|
|
SECRET=$(openssl rand -hex 20)
|
|
UUID=$(python3 -c 'import uuid,sys;print(uuid.UUID(bytes=sys.argv[1][:16].encode()))' "$SECRET")
|
|
printf 'RUNNER_SECRET=%s\nRUNNER_UUID=%s\n' "$SECRET" "$UUID" >> .env
|
|
|
|
# 3. Tell Forgejo about it
|
|
docker compose exec -u git forgejo forgejo forgejo-cli actions register \
|
|
--name box-runner --labels docker --secret "$SECRET"
|
|
|
|
# 4. Fill in the runner config and start the CI profile
|
|
cp runner-config.example.yml runner-config.yml
|
|
sed -i "s|https://git.example.com|https://git.$DOMAIN|; s|\${RUNNER_UUID}|$UUID|; s|\${RUNNER_SECRET}|$SECRET|" runner-config.yml
|
|
docker compose --profile ci up -d
|
|
```
|
|
|
|
Then check Site Administration → Runners: the runner should be online with the
|
|
label `docker`.
|
|
|
|
**Four things that cost people an evening:**
|
|
|
|
- A registration **token from the web UI is not a shared secret**. The declarative
|
|
path above only accepts the secret you generated. Feed it the web token and the
|
|
runner connects and never receives a job, with no error anywhere.
|
|
- The `connections:` block lives under a top-level **`server:`** key in runner
|
|
13.x. Older examples put it under `runner:`, which fails with
|
|
`0 server connections configured`.
|
|
- **An empty `labels:` list means every job waits forever.** No error, just a
|
|
queued job and a healthy-looking runner.
|
|
- The image behind a label **must contain node** — Forgejo actions are JavaScript.
|
|
Pointing `docker` at a docker-cli image gives you
|
|
`node: executable file not found`. Use a node image and let the build job fetch
|
|
a static docker client (see `examples/ci.yml`).
|
|
|
|
`examples/ci.yml` is a complete pipeline: test, build a container, push it to
|
|
Forgejo's built-in registry, and deploy it over an SSH key that can only run one
|
|
command (`examples/deploy-hello-ops.sh`). Read the CI section of
|
|
[HARDENING.md](HARDENING.md) before you wire the deploy step — there are two
|
|
comfortable ways to hand your whole server to anyone who can push a branch.
|
|
|
|
## One login for the vault (Episode 4)
|
|
|
|
Your Forgejo already knows your team. Let it log them into the password
|
|
manager too, so offboarding is one switch instead of a checklist.
|
|
|
|
```bash
|
|
# 1. Create the OAuth2 app in Forgejo
|
|
# Site Administration -> Applications -> OAuth2 Application
|
|
# Name: Vaultwarden
|
|
# Redirect URI: https://vault.<DOMAIN>/identity/connect/oidc-signin
|
|
# Confidential client: yes
|
|
#
|
|
# That redirect path is not documented anywhere in the UI, and a wrong one
|
|
# only fails AFTER the Forgejo login. Copy it character for character.
|
|
|
|
# 2. Put the client id and secret in .env
|
|
VAULTWARDEN_SSO_CLIENT_ID=...
|
|
VAULTWARDEN_SSO_CLIENT_SECRET=...
|
|
|
|
# 3. Restart just that container
|
|
docker compose up -d vaultwarden
|
|
|
|
# 4. Log in: https://vault.<DOMAIN>
|
|
# Type your work mail, hit "Use single sign-on", authorize once in Forgejo.
|
|
```
|
|
|
|
Vaultwarden will then ask you to set a master password. That is not a bug.
|
|
Forgejo proved *who you are*; nothing so far produced the key that *decrypts
|
|
your vault*. Your master password never leaves your browser, and the server
|
|
stores blobs it cannot read. Two different jobs, two different mechanisms.
|
|
|
|
Debugging notes, all of them learned the hard way:
|
|
|
|
- Vaultwarden logs nothing about SSO at startup. It builds the OIDC client on
|
|
the first login attempt, not before. Watch `docker compose logs -f
|
|
vaultwarden` *while* you log in instead: you will see a `prevalidate` call
|
|
and then the `oidc-signin` callback coming back with a code.
|
|
- `/identity/connect/authorize` and the callback both answer `404` when called
|
|
without query parameters. That is route matching, not a wrong path.
|
|
- `curl -I` on Forgejo's `/.well-known/openid-configuration` returns 404 because
|
|
there is no HEAD route. Use `curl -s` and it is a normal 200.
|
|
- Turn on `SSO_SIGNUPS_MATCH_EMAIL` **before** anyone logs in, or you get
|
|
duplicate accounts instead of linked ones.
|
|
|
|
Then make it a team vault: create an organization, add collections, invite
|
|
people. Note that joining is three steps, not two: you invite, they log in,
|
|
and then an owner has to **confirm** them. Until that confirmation, a member
|
|
sees no shared items at all.
|
|
|
|
## Alerts that reach you (Episode 6)
|
|
|
|
A monitor nobody hears is a dashboard. Episode 6 adds a push channel you own
|
|
(ntfy, `ntfy.<DOMAIN>`), keyword monitors for everything a customer sees,
|
|
internal monitors for the services that left the internet in episode 5, a
|
|
public status page at the root of `status.<DOMAIN>`, and a second Uptime Kuma
|
|
outside the server, because a box cannot report its own death.
|
|
|
|
```bash
|
|
docker compose up -d ntfy
|
|
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
|
|
docker compose exec -e NTFY_PASSWORD='<pw>' ntfy ntfy user add --role=user kuma
|
|
docker compose exec ntfy ntfy access kuma alerts write-only
|
|
```
|
|
|
|
The full walkthrough, the second-eye compose file and the gotchas are in
|
|
**[monitoring/README.md](monitoring/README.md)**.
|
|
|
|
## Automation that connects the rest (Episode 8)
|
|
|
|
n8n is the piece that makes the other six talk to each other. A push becomes a
|
|
notification, a monitor going red becomes an incident document, and every
|
|
morning one document says what the box did overnight.
|
|
|
|
The interesting part is the address. Automation needs webhooks, webhooks look
|
|
like they need a public host, and the editor still has no business being on the
|
|
internet. So the Caddyfile publishes **paths, not a host**:
|
|
|
|
```
|
|
n8n.{$DOMAIN} {
|
|
@webhooks path /webhook/* /webhook-test/* /webhook-waiting/*
|
|
handle @webhooks {
|
|
reverse_proxy n8n:5678
|
|
}
|
|
handle {
|
|
respond 404
|
|
}
|
|
}
|
|
```
|
|
|
|
Everything else under `n8n.<DOMAIN>` answers an empty 404: the editor, the login
|
|
page, the REST API. You open the editor over the mesh from episode 5. Senders
|
|
that live on the box (Forgejo, Uptime Kuma) skip the public address entirely and
|
|
call `http://n8n:5678/webhook/...` over the docker network.
|
|
|
|
The three workflows, the credentials they need, the two settings that make
|
|
Forgejo and n8n cooperate, and the one file whose loss makes every credential
|
|
worthless are in **[automation/README.md](automation/README.md)**.
|
|
|
|
## Hardening
|
|
|
|
Before you put real company data on this box, read **[HARDENING.md](HARDENING.md)**.
|
|
The short version: SSH keys only, a firewall that also closes Docker's forwarded
|
|
ports (`DOCKER-USER`, not just ufw), fail2ban on SSH and Caddy, unattended
|
|
security updates, pinned image tags, offsite backups. The ready-to-paste configs
|
|
from episode 10 (ufw `after.rules`, fail2ban jails and filter, sshd drop-in) live
|
|
in **[hardening/](hardening/)**.
|
|
|
|
Episode 12 adds CrowdSec next to fail2ban (not instead of it): community
|
|
blocklist, a firewall bouncer in nftables mode that Docker cannot bypass, and an
|
|
AppSec WAF inside Caddy. Install notes and the traps we measured are in
|
|
**[crowdsec/](crowdsec/)**. Note the Caddy image now needs the key
|
|
`CROWDSEC_BOUNCER_KEY` in `.env`.
|
|
|
|
Episode 14 asks who should write the WAF exclusions the CRS needs. A local 3B
|
|
model reads the blocked requests and drafts them as an issue; a human merges.
|
|
Measured against twenty lines of Python on the same data, the script wins:
|
|
5 of 5 false positives and no exclusion for an attack, where the model proposed
|
|
an exclusion for nearly every attack it saw. What decides it is the one field
|
|
in a blocked request that the sender cannot forge, the source address. The
|
|
tool, the script and workflow G are in **[ops-readonly/](ops-readonly/)** and
|
|
**[automation/](automation/)**, the reasoning in HARDENING.md.
|
|
|
|
---
|
|
|
|
MIT — do whatever. If it saved you money, a YouTube subscription is the price:
|
|
**[@selfhostedops](https://youtube.com/@selfhostedops)**
|