apf_portal/infra/README.md

# `infra/`

Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).

| Subject                                | File / Folder                                      | ADR / Reference                                                                                                                                                                                                                                                                               |
| -------------------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Self-hosted CI runners (Gitea Actions) | [`ci-runners.compose.yml`](ci-runners.compose.yml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md)                                                                                                                                                                                                                           |
| Shared `act_runner` configuration      | [`runner-config.yaml`](runner-config.yaml)         | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md)                                                                                                                                                                                                                           |
| CI runners convenience script          | [`ci-runners.sh`](ci-runners.sh)                   | See "Convenience script" below                                                                                                                                                                                                                                                                |
| Runtime state of the runners           | `data/` (git-ignored after `.gitignore`)           | —                                                                                                                                                                                                                                                                                             |
| Env-vars template for the runners      | `.env.example` (`.env` is git-ignored)             | —                                                                                                                                                                                                                                                                                             |
| Local-dev runtime stack                | [`local/`](local/)                                 | [ADR-0006](../docs/decisions/0006-persistence-postgresql-prisma.md), [ADR-0010](../docs/decisions/0010-session-management-redis.md), [ADR-0012](../docs/decisions/0012-observability-pino-opentelemetry.md), [ADR-0013](../docs/decisions/0013-audit-trail-separated-postgres-append-only.md) |

Future folders / files that will land here as the corresponding ADRs ship:

- **`prod/`** — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).

---

## CI runners — `ci-runners.compose.yml`

Three self-hosted [`act_runner`](https://gitea.com/gitea/act_runner) instances, registered with the project's Gitea organisation, labelled `self-hosted` + `on-prem` (the labels referenced by every job in `.gitea/workflows/*`). Three matches the floor recommended by [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.

### First-time registration

```bash
cd infra/

# 1. Generate a registration token in Gitea.
#    Site Administration → Actions → Runners → "Create new Runner"
#    (or, for org-scoped runners: Organisation Settings → Actions → Runners).
#    The token is one-time and short-lived; don't lose it.

# 2. Configure .env (which is git-ignored).
cp .env.example .env
$EDITOR .env
#    Set GITEA_INSTANCE_URL (https, no trailing slash) and
#    GITEA_RUNNER_REGISTRATION_TOKEN.

# 3. Pre-pull the job images and bring the runners up. The script
#    chains the two — see "Job image pinning and pre-pull" below
#    for the rationale.
./ci-runners.sh up --prepull

# 4. Verify in Gitea: the three runners appear as online with the
#    self-hosted, on-prem labels. If a runner doesn't come online,
#    inspect its logs:
./ci-runners.sh logs runner-1
```

After the first successful boot, each runner stores its credentials under `data/runner-N/.runner`. The registration token is no longer needed and **should be removed** from `.env`. Subsequent restarts (`./ci-runners.sh restart …` or direct `docker compose restart …`) authenticate from the persisted credential.

### Convenience script — `ci-runners.sh`

[`ci-runners.sh`](ci-runners.sh) is a thin wrapper around `docker compose -f ci-runners.compose.yml ...` for the everyday verbs. Two reasons to use it:

1. **Hides the compose-file path** on every command. `./ci-runners.sh up` instead of `docker compose -f ci-runners.compose.yml up -d`.
2. **`rotate` automates the rolling restart** the "Operational tips" below recommend: runner-1 → wait → runner-2 → wait → runner-3, so the CI pipeline always has at least N-1 runners online while you push a config change.

| Command                            | Effect                                                                         |
| ---------------------------------- | ------------------------------------------------------------------------------ |
| `./ci-runners.sh up`               | Bring the three runner containers up                                           |
| `./ci-runners.sh up --prepull`     | Pre-pull the job images (`act-22.04` + `:full-22.04`) on the host first        |
| `./ci-runners.sh down`             | Stop and remove the containers (preserves `data/runner-N/.runner` credentials) |
| `./ci-runners.sh restart <runner>` | Restart one runner                                                             |
| `./ci-runners.sh rotate`           | Rolling restart of every runner with a 15 s pause between each                 |
| `./ci-runners.sh status`           | `docker compose ps` for the runner services                                    |
| `./ci-runners.sh logs [runner]`    | Follow logs (one runner or all of them)                                        |
| `./ci-runners.sh pull-images`      | Pre-pull / refresh the job images (idempotent)                                 |

Anything not matching one of the named verbs is passed through to `docker compose -f ci-runners.compose.yml ...`. Run `./ci-runners.sh help` for the full reference.

For the destructive `down -v` (wipes `data/`, forces re-registration with a fresh Gitea token), the script intentionally **doesn't** offer a verb — invoke `docker compose -f ci-runners.compose.yml down -v` directly so the path is explicit at the typing level.

### Operational tips

- **Rotation of one runner at a time** — to upgrade the image or change config, run `./ci-runners.sh rotate` (or restart manually one by one — `./ci-runners.sh restart runner-1`, wait, …) so the CI pipeline is never starved.
- **Logs** — `./ci-runners.sh logs runner-N` (or `docker compose logs -f --tail=100 runner-N`) for a single runner; jobs being executed appear here.
- **Disk pressure** — the runner caches each job's container image in `/var/lib/docker` on the host. On a small host, prune periodically (`docker system prune -af` while no job is running).
- **Adding a fourth runner** — copy any `runner-N` block in the compose file, increment the suffix in `container_name`, `GITEA_RUNNER_NAME`, and the `data/` mount path. Add the new name to the `RUNNERS=(…)` array at the top of `ci-runners.sh` so `rotate` and `restart` learn about it. Then `./ci-runners.sh up` (or `docker compose up -d`). The runner registers using the same `GITEA_RUNNER_REGISTRATION_TOKEN` (which must be regenerated if it has expired).

### Security — Docker socket exposure

The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn containers. **This grants the runner root-equivalent access to the host's Docker daemon.** A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:

- **Trust boundary:** only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
- **Dedicated host:** run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
- **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes.
- **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.

### Cache server (deferred)

`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v4` (`cache: 'pnpm'`), `actions/cache`, and similar. In the current setup it does **not** work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned via the mounted `/var/run/docker.sock` come up on Docker's default `bridge` network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching.

For now `cache: 'pnpm'` is left **disabled** in `.gitea/workflows/ci.yml`. `pnpm install --frozen-lockfile` is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything.

When this is reactivated, the right fix is one of:

- attach jobs to the same Docker network as the runners (`runner.container.network` in act_runner's `config.yaml`, then advertise the cache `host` on the bridge IP); or
- bind act_runner's cache server on a fixed `host_port` reachable from any container (host gateway IP), and set `ACTIONS_CACHE_URL` accordingly.

Either path needs a single-runner spike before being rolled out to the three.

### `act_runner` image pinning

The compose pins `gitea/act_runner:0.2.13`. Update the pin deliberately, not via `:latest`:

1. Read the act_runner [release notes](https://gitea.com/gitea/act_runner/releases) for breaking changes.
2. Edit the three image references (`runner-1`, `runner-2`, `runner-3`).
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject.
4. Roll one runner at a time (rotation tip above).

The matching CI workflows refer to runner _labels_ (not images), so a runner-image upgrade does not affect `.gitea/workflows/*`.

### Job image pinning and pre-pull

`act_runner` runs each job inside a container whose image is selected by the runner's _labels_. Two images are in use:

| Label                  | Image                            | Used by                            |
| ---------------------- | -------------------------------- | ---------------------------------- |
| `self-hosted`          | `catthehacker/ubuntu:act-22.04`  | `check`, `scan`, `commits`, `a11y` |
| `on-prem`              | `catthehacker/ubuntu:act-22.04`  | (alias of `self-hosted`)           |
| (per-job `container:`) | `catthehacker/ubuntu:full-22.04` | `perf` (Lighthouse needs Chrome)   |

[`runner-config.yaml`](runner-config.yaml) sets `container.force_pull: false`. Without that, act_runner re-issues a `docker pull` at the start of every single job (~10–30 s of registry round-trip even when every layer is already cached), which both wastes wall-clock and contradicts our policy of upgrading job images deliberately rather than implicitly via `:latest`.

The trade-off: the host Docker daemon must already hold the images locally. Pre-pull them once after a fresh runner host install:

```bash
docker pull catthehacker/ubuntu:act-22.04
docker pull catthehacker/ubuntu:full-22.04
```

Upgrading to a newer tag is a deliberate three-step process:

1. Edit `GITEA_RUNNER_LABELS` (in [`ci-runners.compose.yml`](ci-runners.compose.yml)) and / or the per-job `container.image:` (in `.gitea/workflows/*`) to the new tag.
2. On the runner host, `docker pull <new-tag>` so the image is locally available before the next CI job starts.
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject; one of `chore(deps): upgrade CI job image to ...`.

Old, no-longer-referenced images can be reaped during the periodic `docker system prune -af` (see "Disk pressure" above).

---

## Local-dev stack — `local/`

A Docker Compose recipe spinning up the runtime services the BFF and ADRs assume — Postgres, Redis, OpenTelemetry Collector — plus optional viewers (pgweb, Jaeger UI) gated behind Compose profiles. Designed to start in a single command on a contributor's WSL2 / Linux / macOS host.

| File                                                                 | Role                                                                                                 |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`local/dev.sh`](local/dev.sh)                                       | Convenience wrapper around `docker compose` — see "Convenience script" below                         |
| [`local/dev.compose.yml`](local/dev.compose.yml)                     | Service definitions: postgres, redis, otel-collector, plus pgweb and jaeger behind profiles          |
| [`local/.env.example`](local/.env.example)                           | Credentials + ports template (copy to `.env`, which is git-ignored)                                  |
| [`local/init/postgres/01-init.sql`](local/init/postgres/01-init.sql) | Bootstrap SQL for ADR-0013: audit roles + schema, applied on first boot only                         |
| [`local/otel-collector.yaml`](local/otel-collector.yaml)             | Collector pipeline: OTLP receivers → batch → debug exporter (always) + forward to Jaeger when active |

### First-time setup

```bash
# 1. Configure local secrets (copy template, edit, do not commit).
cp infra/local/.env.example infra/local/.env
$EDITOR infra/local/.env
#    Set strong dev values for POSTGRES_PASSWORD and REDIS_PASSWORD
#    (defaults in the template are placeholders that the compose
#    rejects with `must be set in infra/local/.env` if left as-is).

# 2. Bring up the core stack (postgres + redis + otel-collector).
./infra/local/dev.sh up

# 3. (Optional) Activate viewers when needed:
./infra/local/dev.sh up dbtools         # adds pgweb
./infra/local/dev.sh up observability   # adds Jaeger UI
./infra/local/dev.sh up all             # core + every profile

# 4. Verify health.
./infra/local/dev.sh status
```

### Convenience script — `dev.sh`

[`local/dev.sh`](local/dev.sh) is a thin wrapper around `docker compose -f dev.compose.yml ...` with two reasons to exist:

1. **Hides the Compose-profile gotcha.** `docker compose down` only operates on services whose profile is currently active — anything started under `--profile X` keeps running unless the same flag is on `down`. The script always passes every profile in scope on teardown / status / log commands, so profile-gated services (pgweb, Jaeger) are never accidentally orphaned.
2. **Ergonomic verbs** for the common workflows. `./dev.sh up all`, `./dev.sh stop pgweb`, `./dev.sh logs otel-collector`, etc.

Run `./infra/local/dev.sh help` for the full reference. Cheat-sheet:

| Command                                                                         | Effect                                                       |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `./infra/local/dev.sh up`                                                       | Core only (postgres + redis + otel-collector)                |
| `./infra/local/dev.sh up all`                                                   | Core + every profile                                         |
| `./infra/local/dev.sh up dbtools`                                               | Core + pgweb                                                 |
| `./infra/local/dev.sh up observability`                                         | Core + Jaeger                                                |
| `./infra/local/dev.sh down`                                                     | Tear down the whole stack (every profile in scope)           |
| `./infra/local/dev.sh down -v`                                                  | Tear down + wipe named volumes (incl. audit-roles bootstrap) |
| `./infra/local/dev.sh stop pgweb`                                               | Stop one service (containers stay around)                    |
| `./infra/local/dev.sh status`                                                   | `docker compose ps`, with every profile visible              |
| `./infra/local/dev.sh logs otel-collector`                                      | Follow logs                                                  |
| `./infra/local/dev.sh exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"` | Run a command inside a service                               |

Anything not matching one of the named verbs is passed through to `docker compose -f dev.compose.yml ...` (with every profile flagged in), so you keep the full Compose surface available — `./dev.sh config`, `./dev.sh top`, `./dev.sh inspect …`, etc.

If you prefer to call `docker compose` directly, every example below shows the raw command alongside the script form.

### Service endpoints (defaults)

| Service             | Host port | Purpose                                                             |
| ------------------- | --------- | ------------------------------------------------------------------- |
| Postgres            | 5432      | DB connection — `postgres://portal:<pwd>@localhost:5432/portal_dev` |
| Redis               | 6379      | Sessions, OBO cache (per ADR-0010 / ADR-0014)                       |
| OTel Collector gRPC | 4317      | `OTEL_EXPORTER_OTLP_ENDPOINT` for the BFF and the SPA               |
| OTel Collector HTTP | 4318      | OTLP/HTTP variant                                                   |
| pgweb (profile)     | 8081      | http://localhost:8081 — Postgres GUI                                |
| Jaeger UI (profile) | 16686     | http://localhost:16686 — trace explorer                             |

All ports are overridable via `.env` if the host machine has conflicts.

### Operational tips

- **Persistence** — state lives in named Docker volumes (`apf-portal-postgres-data`, `apf-portal-redis-data`). Survives `docker compose down`. Use `docker compose -f dev.compose.yml down -v` to wipe (also wipes the audit-roles bootstrap, which re-runs on the next fresh boot).
- **Profile symmetry** — `dev.sh down` (and `status`, `logs`, …) always include every profile in scope, so profile-gated services are caught. If you bypass the script and call `docker compose down` directly, you must pass the same `--profile` flags as on `up`, otherwise pgweb and Jaeger keep running silently. Either pass them again, or `export COMPOSE_PROFILES=dbtools,observability` in your shell or `infra/local/.env`.

- **Bootstrap re-run** — the SQL in `local/init/postgres/` only runs on a **fresh** Postgres data volume. To replay after editing the file, `down -v` (loses all dev data) or run the SQL manually with `docker compose exec postgres psql -U portal -d portal_dev -f /docker-entrypoint-initdb.d/01-init.sql`.
- **Logs** — `docker compose -f dev.compose.yml logs -f <service>` to follow a single service. `otel-collector` is the loudest — its `debug` exporter prints every span / metric / log it receives.
- **Image upgrades** — same policy as the runner image (deliberate, not via `:latest`). Renovate's docker-compose manager will surface bumps automatically once the dashboard rule allows them.

### Production parity

This stack is **dev-only**. The corresponding production layout (HA Postgres, Redis Sentinel cluster, OTel Collector with a real backend, secret manager) lives in the future on-prem-infrastructure ADR — see `prod/` placeholder below.

---

## Future infra concerns — placeholders

These are listed here so a contributor knows where to expect related files; they don't exist yet.

| File            | Purpose                                                                                   | Triggered by                                       |
| --------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `prod/*`        | On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on) | The on-prem infrastructure ADR (phase 3b)          |
| `runbooks/*.md` | Operational runbooks (incident response, secret rotation, runner upgrade procedure, …)    | First incident, or when ops cadence justifies them |