Files
apf_portal/infra/README.md
T
Julien Gautier 9a57f21c6d
CI / scan (pull_request) Failing after 1m27s
CI / commits (pull_request) Successful in 1m28s
CI / check (pull_request) Successful in 1m34s
CI / a11y (pull_request) Successful in 1m3s
CI / perf (pull_request) Successful in 1m52s
fix(ci): disable broken pnpm cache on actions/setup-node
`act_runner`'s built-in GitHub-Actions-cache server binds inside the
runner container on the compose-defined `apf-portal-act-runners`
bridge. Jobs spawned via the mounted Docker socket come up on
Docker's default `bridge` network and cannot reach it. Every job
opting into `cache: 'pnpm'` therefore eats ~2 min ETIMEDOUT on
restore at the start and another ~2 min on save at the end — for
zero cache hits. Across the five jobs that's ~20 min wasted per CI
run.

Drop `cache: 'pnpm'` from all five jobs. `pnpm install
--frozen-lockfile` is fast enough on the warm store inside the job
container that the cache layer is currently no value-add anyway.

Document the proper fix (cross-container networking / fixed-port
cache binding) in infra/README.md so a future infra spike can
re-enable caching cleanly.
2026-05-04 15:01:59 +02:00

8.6 KiB
Raw Blame History

infra/

Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).

Subject File / Folder ADR / Reference
Self-hosted CI runners (Gitea Actions) ci-runners.compose.yml ADR-0015 §"Runners"
Runtime state of the runners data/ (git-ignored after .gitignore)
Env-vars template for the runners .env.example (.env is git-ignored)

Future folders / files that will land here as the corresponding ADRs ship:

  • local/ — Docker Compose for the developer's machine (Postgres + Redis + OTel collector). Triggered by the first feature that needs Redis (sessions, ADR-0010).
  • prod/ — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).

CI runners — ci-runners.compose.yml

Three self-hosted act_runner instances, registered with the project's Gitea organisation, labelled self-hosted + on-prem (the labels referenced by every job in .gitea/workflows/*). Three matches the floor recommended by ADR-0015 §"Runners" — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.

First-time registration

cd infra/

# 1. Generate a registration token in Gitea.
#    Site Administration → Actions → Runners → "Create new Runner"
#    (or, for org-scoped runners: Organisation Settings → Actions → Runners).
#    The token is one-time and short-lived; don't lose it.

# 2. Configure .env (which is git-ignored).
cp .env.example .env
$EDITOR .env
#    Set GITEA_INSTANCE_URL (https, no trailing slash) and
#    GITEA_RUNNER_REGISTRATION_TOKEN.

# 3. Bring the runners up.
docker compose -f ci-runners.compose.yml up -d

# 4. Verify in Gitea: the three runners appear as online with the
#    self-hosted, on-prem labels. If a runner doesn't come online,
#    inspect its logs:
docker compose -f ci-runners.compose.yml logs runner-1

After the first successful boot, each runner stores its credentials under data/runner-N/.runner. The registration token is no longer needed and should be removed from .env. Subsequent restarts (docker compose restart) authenticate from the persisted credential.

Operational tips

  • Rotation of one runner at a time — to upgrade the image or change config, restart runners one by one (docker compose restart runner-1, wait, runner-2, …) so the CI pipeline is never starved.
  • Logsdocker compose logs -f --tail=100 runner-N for a single runner; jobs being executed appear here.
  • Disk pressure — the runner caches each job's container image in /var/lib/docker on the host. On a small host, prune periodically (docker system prune -af while no job is running).
  • Adding a fourth runner — copy any runner-N block in the compose file, increment the suffix in container_name, GITEA_RUNNER_NAME, and the data/ mount path. Then docker compose up -d. The runner registers using the same GITEA_RUNNER_REGISTRATION_TOKEN (which must be regenerated if it has expired).

Security — Docker socket exposure

The compose mounts /var/run/docker.sock into each runner so jobs can spawn containers. This grants the runner root-equivalent access to the host's Docker daemon. A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:

  • Trust boundary: only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
  • Dedicated host: run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
  • No host filesystem mounts beyond the docker socket: the compose intentionally does not mount /, /etc, or any project source. Workflows that need data on the host must do so via Docker volumes.
  • Future hardening (out of scope of v1): migrate to rootless Docker on the runner host, or to a DinD (Docker-in-Docker) sidecar so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.

Cache server (deferred)

act_runner ships a built-in GitHub-Actions-cache-compatible server, used by actions/setup-node@v4 (cache: 'pnpm'), actions/cache, and similar. In the current setup it does not work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined apf-portal-act-runners bridge, while jobs spawned via the mounted /var/run/docker.sock come up on Docker's default bridge network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min ETIMEDOUT at the start (restore) and end (save) of every job that opts into caching.

For now cache: 'pnpm' is left disabled in .gitea/workflows/ci.yml. pnpm install --frozen-lockfile is fast enough on a warm pnpm store inside the job image (~3060 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything.

When this is reactivated, the right fix is one of:

  • attach jobs to the same Docker network as the runners (runner.container.network in act_runner's config.yaml, then advertise the cache host on the bridge IP); or
  • bind act_runner's cache server on a fixed host_port reachable from any container (host gateway IP), and set ACTIONS_CACHE_URL accordingly.

Either path needs a single-runner spike before being rolled out to the three.

Image pinning

The compose pins gitea/act_runner:0.2.13. Update the pin deliberately, not via :latest:

  1. Read the act_runner release notes for breaking changes.
  2. Edit the three image references (runner-1, runner-2, runner-3).
  3. Commit on a feature branch with a chore(deps): Conventional Commits subject.
  4. Roll one runner at a time (rotation tip above).

The matching CI workflows refer to runner labels (not images), so a runner-image upgrade does not affect .gitea/workflows/*.


Future infra concerns — placeholders

These are listed here so a contributor knows where to expect related files; they don't exist yet.

File Purpose Triggered by
local/dev.compose.yml Postgres 17 + Redis (Sentinel-flavoured single-node) + OTel collector for local dev First feature that needs Redis or end-to-end OTel traces
local/init/postgres/*.sql Bootstrap SQL to create the Postgres roles audit_owner / audit_writer / audit_reader / audit_archiver (per ADR-0013) and provision the dev DB Same as above
prod/* On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on) The on-prem infrastructure ADR (phase 3b)
runbooks/*.md Operational runbooks (incident response, secret rotation, runner upgrade procedure, …) First incident, or when ops cadence justifies them