Julien Gautier
4934fc29da
`act_runner`'s default `container.force_pull: true` re-issues a `docker pull` at the start of every job, adding 10-30 s of registry round-trip per job even when every layer is already locally cached. The job images are already pinned to specific tags (catthehacker ubuntu :act-22.04 + :full-22.04), so the implicit pull is both pure overhead and contradictory with the deliberate-upgrade policy ADR-0015 spells out for the runner image itself. Mount a shared `infra/runner-config.yaml` into all three runners with `force_pull: false`, point each runner at it via the `CONFIG_FILE` env var, and document the pre-pull procedure (one-shot `docker pull` on the runner host, plus the upgrade playbook) in infra/README.md "Job image pinning and pre-pull". The pre-pull is also folded into the "First-time registration" walkthrough so a fresh setup is correct end-to-end.
134 lines
11 KiB
Markdown
134 lines
11 KiB
Markdown
# `infra/`
|
||
|
||
Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).
|
||
|
||
| Subject | File / Folder | ADR / Reference |
|
||
| -------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------- |
|
||
| Self-hosted CI runners (Gitea Actions) | [`ci-runners.compose.yml`](ci-runners.compose.yml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
|
||
| Shared `act_runner` configuration | [`runner-config.yaml`](runner-config.yaml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
|
||
| Runtime state of the runners | `data/` (git-ignored after `.gitignore`) | — |
|
||
| Env-vars template for the runners | `.env.example` (`.env` is git-ignored) | — |
|
||
|
||
Future folders / files that will land here as the corresponding ADRs ship:
|
||
|
||
- **`local/`** — Docker Compose for the developer's machine (Postgres + Redis + OTel collector). Triggered by the first feature that needs Redis (sessions, ADR-0010).
|
||
- **`prod/`** — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).
|
||
|
||
---
|
||
|
||
## CI runners — `ci-runners.compose.yml`
|
||
|
||
Three self-hosted [`act_runner`](https://gitea.com/gitea/act_runner) instances, registered with the project's Gitea organisation, labelled `self-hosted` + `on-prem` (the labels referenced by every job in `.gitea/workflows/*`). Three matches the floor recommended by [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.
|
||
|
||
### First-time registration
|
||
|
||
```bash
|
||
cd infra/
|
||
|
||
# 1. Generate a registration token in Gitea.
|
||
# Site Administration → Actions → Runners → "Create new Runner"
|
||
# (or, for org-scoped runners: Organisation Settings → Actions → Runners).
|
||
# The token is one-time and short-lived; don't lose it.
|
||
|
||
# 2. Configure .env (which is git-ignored).
|
||
cp .env.example .env
|
||
$EDITOR .env
|
||
# Set GITEA_INSTANCE_URL (https, no trailing slash) and
|
||
# GITEA_RUNNER_REGISTRATION_TOKEN.
|
||
|
||
# 3. Pre-pull the job images so the runner doesn't have to (see
|
||
# "Job image pinning and pre-pull" below for the rationale).
|
||
docker pull catthehacker/ubuntu:act-22.04
|
||
docker pull catthehacker/ubuntu:full-22.04
|
||
|
||
# 4. Bring the runners up.
|
||
docker compose -f ci-runners.compose.yml up -d
|
||
|
||
# 5. Verify in Gitea: the three runners appear as online with the
|
||
# self-hosted, on-prem labels. If a runner doesn't come online,
|
||
# inspect its logs:
|
||
docker compose -f ci-runners.compose.yml logs runner-1
|
||
```
|
||
|
||
After the first successful boot, each runner stores its credentials under `data/runner-N/.runner`. The registration token is no longer needed and **should be removed** from `.env`. Subsequent restarts (`docker compose restart`) authenticate from the persisted credential.
|
||
|
||
### Operational tips
|
||
|
||
- **Rotation of one runner at a time** — to upgrade the image or change config, restart runners one by one (`docker compose restart runner-1`, wait, runner-2, …) so the CI pipeline is never starved.
|
||
- **Logs** — `docker compose logs -f --tail=100 runner-N` for a single runner; jobs being executed appear here.
|
||
- **Disk pressure** — the runner caches each job's container image in `/var/lib/docker` on the host. On a small host, prune periodically (`docker system prune -af` while no job is running).
|
||
- **Adding a fourth runner** — copy any `runner-N` block in the compose file, increment the suffix in `container_name`, `GITEA_RUNNER_NAME`, and the `data/` mount path. Then `docker compose up -d`. The runner registers using the same `GITEA_RUNNER_REGISTRATION_TOKEN` (which must be regenerated if it has expired).
|
||
|
||
### Security — Docker socket exposure
|
||
|
||
The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn containers. **This grants the runner root-equivalent access to the host's Docker daemon.** A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:
|
||
|
||
- **Trust boundary:** only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
|
||
- **Dedicated host:** run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
|
||
- **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes.
|
||
- **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.
|
||
|
||
### Cache server (deferred)
|
||
|
||
`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v4` (`cache: 'pnpm'`), `actions/cache`, and similar. In the current setup it does **not** work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned via the mounted `/var/run/docker.sock` come up on Docker's default `bridge` network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching.
|
||
|
||
For now `cache: 'pnpm'` is left **disabled** in `.gitea/workflows/ci.yml`. `pnpm install --frozen-lockfile` is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything.
|
||
|
||
When this is reactivated, the right fix is one of:
|
||
|
||
- attach jobs to the same Docker network as the runners (`runner.container.network` in act_runner's `config.yaml`, then advertise the cache `host` on the bridge IP); or
|
||
- bind act_runner's cache server on a fixed `host_port` reachable from any container (host gateway IP), and set `ACTIONS_CACHE_URL` accordingly.
|
||
|
||
Either path needs a single-runner spike before being rolled out to the three.
|
||
|
||
### `act_runner` image pinning
|
||
|
||
The compose pins `gitea/act_runner:0.2.13`. Update the pin deliberately, not via `:latest`:
|
||
|
||
1. Read the act_runner [release notes](https://gitea.com/gitea/act_runner/releases) for breaking changes.
|
||
2. Edit the three image references (`runner-1`, `runner-2`, `runner-3`).
|
||
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject.
|
||
4. Roll one runner at a time (rotation tip above).
|
||
|
||
The matching CI workflows refer to runner _labels_ (not images), so a runner-image upgrade does not affect `.gitea/workflows/*`.
|
||
|
||
### Job image pinning and pre-pull
|
||
|
||
`act_runner` runs each job inside a container whose image is selected by the runner's _labels_. Two images are in use:
|
||
|
||
| Label | Image | Used by |
|
||
| ---------------------- | -------------------------------- | ---------------------------------- |
|
||
| `self-hosted` | `catthehacker/ubuntu:act-22.04` | `check`, `scan`, `commits`, `a11y` |
|
||
| `on-prem` | `catthehacker/ubuntu:act-22.04` | (alias of `self-hosted`) |
|
||
| (per-job `container:`) | `catthehacker/ubuntu:full-22.04` | `perf` (Lighthouse needs Chrome) |
|
||
|
||
[`runner-config.yaml`](runner-config.yaml) sets `container.force_pull: false`. Without that, act_runner re-issues a `docker pull` at the start of every single job (~10–30 s of registry round-trip even when every layer is already cached), which both wastes wall-clock and contradicts our policy of upgrading job images deliberately rather than implicitly via `:latest`.
|
||
|
||
The trade-off: the host Docker daemon must already hold the images locally. Pre-pull them once after a fresh runner host install:
|
||
|
||
```bash
|
||
docker pull catthehacker/ubuntu:act-22.04
|
||
docker pull catthehacker/ubuntu:full-22.04
|
||
```
|
||
|
||
Upgrading to a newer tag is a deliberate three-step process:
|
||
|
||
1. Edit `GITEA_RUNNER_LABELS` (in [`ci-runners.compose.yml`](ci-runners.compose.yml)) and / or the per-job `container.image:` (in `.gitea/workflows/*`) to the new tag.
|
||
2. On the runner host, `docker pull <new-tag>` so the image is locally available before the next CI job starts.
|
||
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject; one of `chore(deps): upgrade CI job image to ...`.
|
||
|
||
Old, no-longer-referenced images can be reaped during the periodic `docker system prune -af` (see "Disk pressure" above).
|
||
|
||
---
|
||
|
||
## Future infra concerns — placeholders
|
||
|
||
These are listed here so a contributor knows where to expect related files; they don't exist yet.
|
||
|
||
| File | Purpose | Triggered by |
|
||
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
||
| `local/dev.compose.yml` | Postgres 17 + Redis (Sentinel-flavoured single-node) + OTel collector for local dev | First feature that needs Redis or end-to-end OTel traces |
|
||
| `local/init/postgres/*.sql` | Bootstrap SQL to create the Postgres roles `audit_owner` / `audit_writer` / `audit_reader` / `audit_archiver` (per ADR-0013) and provision the dev DB | Same as above |
|
||
| `prod/*` | On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on) | The on-prem infrastructure ADR (phase 3b) |
|
||
| `runbooks/*.md` | Operational runbooks (incident response, secret rotation, runner upgrade procedure, …) | First incident, or when ops cadence justifies them |
|