apf_portal/infra/README.md

infra/

Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).

Subject	File / Folder	ADR / Reference
Self-hosted CI runners (Gitea Actions)	ci-runners.compose.yml	ADR-0015 §"Runners"
Shared act_runner configuration	runner-config.yaml	ADR-0015 §"Runners"
CI runners convenience script	ci-runners.sh	See "Convenience script" below
Runtime state of the runners	data/ (git-ignored after .gitignore)	—
Env-vars template for the runners	.env.example (.env is git-ignored)	—
Local-dev runtime stack	local/	ADR-0006, ADR-0010, ADR-0012, ADR-0013

Future folders / files that will land here as the corresponding ADRs ship:

• prod/ — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).

---

CI runners — ci-runners.compose.yml

Three self-hosted act_runner instances, registered with the project's Gitea organisation, labelled self-hosted + on-prem (the labels referenced by every job in .gitea/workflows/*). Three matches the floor recommended by ADR-0015 §"Runners" — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.

First-time registration

cd infra/

# 1. Generate a registration token in Gitea.
#    Site Administration → Actions → Runners → "Create new Runner"
#    (or, for org-scoped runners: Organisation Settings → Actions → Runners).
#    The token is one-time and short-lived; don't lose it.

# 2. Configure .env (which is git-ignored).
cp .env.example .env
$EDITOR .env
#    Set GITEA_INSTANCE_URL (https, no trailing slash) and
#    GITEA_RUNNER_REGISTRATION_TOKEN.

# 3. Pre-pull the job images and bring the runners up. The script
#    chains the two — see "Job image pinning and pre-pull" below
#    for the rationale.
./ci-runners.sh up --prepull

# 4. Verify in Gitea: the three runners appear as online with the
#    self-hosted, on-prem labels. If a runner doesn't come online,
#    inspect its logs:
./ci-runners.sh logs runner-1

After the first successful boot, each runner stores its credentials under data/runner-N/.runner. The registration token is no longer needed and should be removed from .env. Subsequent restarts (./ci-runners.sh restart … or direct docker compose restart …) authenticate from the persisted credential.

Convenience script — ci-runners.sh

ci-runners.sh is a thin wrapper around docker compose -f ci-runners.compose.yml ... for the everyday verbs. Two reasons to use it:

1. Hides the compose-file path on every command. ./ci-runners.sh up instead of docker compose -f ci-runners.compose.yml up -d.
2. rotate automates the rolling restart the "Operational tips" below recommend: runner-1 → wait → runner-2 → wait → runner-3, so the CI pipeline always has at least N-1 runners online while you push a config change.

Command	Effect
./ci-runners.sh up	Bring the three runner containers up
./ci-runners.sh up --prepull	Pre-pull the job images (act-22.04 + :full-22.04) on the host first
./ci-runners.sh down	Stop and remove the containers (preserves data/runner-N/.runner credentials)
./ci-runners.sh restart <runner>	Restart one runner
./ci-runners.sh rotate	Rolling restart of every runner with a 15 s pause between each
./ci-runners.sh status	docker compose ps for the runner services
./ci-runners.sh logs [runner]	Follow logs (one runner or all of them)
./ci-runners.sh pull-images	Pre-pull / refresh the job images (idempotent)

Anything not matching one of the named verbs is passed through to docker compose -f ci-runners.compose.yml .... Run ./ci-runners.sh help for the full reference.

For the destructive down -v (wipes data/, forces re-registration with a fresh Gitea token), the script intentionally doesn't offer a verb — invoke docker compose -f ci-runners.compose.yml down -v directly so the path is explicit at the typing level.

Operational tips

• Rotation of one runner at a time — to upgrade the image or change config, run ./ci-runners.sh rotate (or restart manually one by one — ./ci-runners.sh restart runner-1, wait, …) so the CI pipeline is never starved.
• Logs — ./ci-runners.sh logs runner-N (or docker compose logs -f --tail=100 runner-N) for a single runner; jobs being executed appear here.
• Disk pressure — the runner caches each job's container image in /var/lib/docker on the host. On a small host, prune periodically (docker system prune -af while no job is running).
• Adding a fourth runner — copy any runner-N block in the compose file, increment the suffix in container_name, GITEA_RUNNER_NAME, and the data/ mount path. Add the new name to the RUNNERS=(…) array at the top of ci-runners.sh so rotate and restart learn about it. Then ./ci-runners.sh up (or docker compose up -d). The runner registers using the same GITEA_RUNNER_REGISTRATION_TOKEN (which must be regenerated if it has expired).

Security — Docker socket exposure

The compose mounts /var/run/docker.sock into each runner so jobs can spawn containers. This grants the runner root-equivalent access to the host's Docker daemon. A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:

• Trust boundary: only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
• Dedicated host: run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
• No host filesystem mounts beyond the docker socket: the compose intentionally does not mount /, /etc, or any project source. Workflows that need data on the host must do so via Docker volumes.
• Future hardening (out of scope of v1): migrate to rootless Docker on the runner host, or to a DinD (Docker-in-Docker) sidecar so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.

Cache server (deferred)

act_runner ships a built-in GitHub-Actions-cache-compatible server, used by actions/setup-node@v4 (cache: 'pnpm'), actions/cache, and similar. In the current setup it does not work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined apf-portal-act-runners bridge, while jobs spawned via the mounted /var/run/docker.sock come up on Docker's default bridge network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min ETIMEDOUT at the start (restore) and end (save) of every job that opts into caching.

For now cache: 'pnpm' is left disabled in .gitea/workflows/ci.yml. pnpm install --frozen-lockfile is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything.

When this is reactivated, the right fix is one of:

• attach jobs to the same Docker network as the runners (runner.container.network in act_runner's config.yaml, then advertise the cache host on the bridge IP); or
• bind act_runner's cache server on a fixed host_port reachable from any container (host gateway IP), and set ACTIONS_CACHE_URL accordingly.

Either path needs a single-runner spike before being rolled out to the three.

act_runner image pinning

The compose pins gitea/act_runner:0.2.13. Update the pin deliberately, not via :latest:

1. Read the act_runner release notes for breaking changes.
2. Edit the three image references (runner-1, runner-2, runner-3).
3. Commit on a feature branch with a chore(deps): Conventional Commits subject.
4. Roll one runner at a time (rotation tip above).

The matching CI workflows refer to runner labels (not images), so a runner-image upgrade does not affect .gitea/workflows/*.

Job image pinning and pre-pull

act_runner runs each job inside a container whose image is selected by the runner's labels. Two images are in use:

Label	Image	Used by
self-hosted	catthehacker/ubuntu:act-22.04	check, scan, commits, a11y
on-prem	catthehacker/ubuntu:act-22.04	(alias of self-hosted)
(per-job container:)	catthehacker/ubuntu:full-22.04	perf (Lighthouse needs Chrome)

runner-config.yaml sets container.force_pull: false. Without that, act_runner re-issues a docker pull at the start of every single job (~10–30 s of registry round-trip even when every layer is already cached), which both wastes wall-clock and contradicts our policy of upgrading job images deliberately rather than implicitly via :latest.

The trade-off: the host Docker daemon must already hold the images locally. Pre-pull them once after a fresh runner host install:

docker pull catthehacker/ubuntu:act-22.04
docker pull catthehacker/ubuntu:full-22.04

Upgrading to a newer tag is a deliberate three-step process:

1. Edit GITEA_RUNNER_LABELS (in ci-runners.compose.yml) and / or the per-job container.image: (in .gitea/workflows/*) to the new tag.
2. On the runner host, docker pull <new-tag> so the image is locally available before the next CI job starts.
3. Commit on a feature branch with a chore(deps): Conventional Commits subject; one of chore(deps): upgrade CI job image to ....

Old, no-longer-referenced images can be reaped during the periodic docker system prune -af (see "Disk pressure" above).

---

Local-dev stack — local/

A Docker Compose recipe spinning up the runtime services the BFF and ADRs assume — Postgres, Redis, OpenTelemetry Collector — plus optional viewers (pgweb, Jaeger UI) gated behind Compose profiles. Designed to start in a single command on a contributor's WSL2 / Linux / macOS host.

File	Role
local/dev.sh	Convenience wrapper around docker compose — see "Convenience script" below
local/dev.compose.yml	Service definitions: postgres, redis, otel-collector, plus pgweb and jaeger behind profiles
local/.env.example	Credentials + ports template (copy to .env, which is git-ignored)
local/init/postgres/01-init.sql	Bootstrap SQL for ADR-0013: audit roles + schema, applied on first boot only
local/otel-collector.yaml	Collector pipeline: OTLP receivers → batch → debug exporter (always) + forward to Jaeger when active

First-time setup

# 1. Configure local secrets (copy template, edit, do not commit).
cp infra/local/.env.example infra/local/.env
$EDITOR infra/local/.env
#    Set strong dev values for POSTGRES_PASSWORD and REDIS_PASSWORD
#    (defaults in the template are placeholders that the compose
#    rejects with `must be set in infra/local/.env` if left as-is).

# 2. Bring up the core stack (postgres + redis + otel-collector).
./infra/local/dev.sh up

# 3. (Optional) Activate viewers when needed:
./infra/local/dev.sh up dbtools         # adds pgweb
./infra/local/dev.sh up observability   # adds Jaeger UI
./infra/local/dev.sh up all             # core + every profile

# 4. Verify health.
./infra/local/dev.sh status

Convenience script — dev.sh

local/dev.sh is a thin wrapper around docker compose -f dev.compose.yml ... with two reasons to exist:

1. Hides the Compose-profile gotcha. docker compose down only operates on services whose profile is currently active — anything started under --profile X keeps running unless the same flag is on down. The script always passes every profile in scope on teardown / status / log commands, so profile-gated services (pgweb, Jaeger) are never accidentally orphaned.
2. Ergonomic verbs for the common workflows. ./dev.sh up all, ./dev.sh stop pgweb, ./dev.sh logs otel-collector, etc.

Run ./infra/local/dev.sh help for the full reference. Cheat-sheet:

Command	Effect
./infra/local/dev.sh up	Core only (postgres + redis + otel-collector)
./infra/local/dev.sh up all	Core + every profile
./infra/local/dev.sh up dbtools	Core + pgweb
./infra/local/dev.sh up observability	Core + Jaeger
./infra/local/dev.sh down	Tear down the whole stack (every profile in scope)
./infra/local/dev.sh down -v	Tear down + wipe named volumes (incl. audit-roles bootstrap)
./infra/local/dev.sh stop pgweb	Stop one service (containers stay around)
./infra/local/dev.sh status	docker compose ps, with every profile visible
./infra/local/dev.sh logs otel-collector	Follow logs
./infra/local/dev.sh exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"	Run a command inside a service

Anything not matching one of the named verbs is passed through to docker compose -f dev.compose.yml ... (with every profile flagged in), so you keep the full Compose surface available — ./dev.sh config, ./dev.sh top, ./dev.sh inspect …, etc.

If you prefer to call docker compose directly, every example below shows the raw command alongside the script form.

Service endpoints (defaults)

Service	Host port	Purpose
Postgres	5432	DB connection — postgres://portal:<pwd>@localhost:5432/portal_dev
Redis	6379	Sessions, OBO cache (per ADR-0010 / ADR-0014)
OTel Collector gRPC	4317	OTEL_EXPORTER_OTLP_ENDPOINT for the BFF and the SPA
OTel Collector HTTP	4318	OTLP/HTTP variant
pgweb (profile)	8081	http://localhost:8081 — Postgres GUI
Jaeger UI (profile)	16686	http://localhost:16686 — trace explorer

All ports are overridable via .env if the host machine has conflicts.

Operational tips

• Persistence — state lives in named Docker volumes (apf-portal-postgres-data, apf-portal-redis-data). Survives docker compose down. Use docker compose -f dev.compose.yml down -v to wipe (also wipes the audit-roles bootstrap, which re-runs on the next fresh boot).

• Profile symmetry — dev.sh down (and status, logs, …) always include every profile in scope, so profile-gated services are caught. If you bypass the script and call docker compose down directly, you must pass the same --profile flags as on up, otherwise pgweb and Jaeger keep running silently. Either pass them again, or export COMPOSE_PROFILES=dbtools,observability in your shell or infra/local/.env.

• Bootstrap re-run — the SQL in local/init/postgres/ only runs on a fresh Postgres data volume. To replay after editing the file, down -v (loses all dev data) or run the SQL manually with docker compose exec postgres psql -U portal -d portal_dev -f /docker-entrypoint-initdb.d/01-init.sql.

• Logs — docker compose -f dev.compose.yml logs -f <service> to follow a single service. otel-collector is the loudest — its debug exporter prints every span / metric / log it receives.

• Image upgrades — same policy as the runner image (deliberate, not via :latest). Renovate's docker-compose manager will surface bumps automatically once the dashboard rule allows them.

Production parity

This stack is dev-only. The corresponding production layout (HA Postgres, Redis Sentinel cluster, OTel Collector with a real backend, secret manager) lives in the future on-prem-infrastructure ADR — see prod/ placeholder below.

---

Future infra concerns — placeholders

These are listed here so a contributor knows where to expect related files; they don't exist yet.

File	Purpose	Triggered by
prod/*	On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on)	The on-prem infrastructure ADR (phase 3b)
runbooks/*.md	Operational runbooks (incident response, secret rotation, runner upgrade procedure, …)	First incident, or when ops cadence justifies them