julien/apf_portal
1
0
Fork 0
Code Issues 1 Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
161e2ecb1c1839e8d03de22a740aa5cfb971b542
apf_portal/infra/README.md
T
julien a463199728
CI / commits (push) Has been skipped
Details
CI / check (push) Successful in 2m3s
Details
CI / scan (push) Successful in 2m18s
Details
CI / a11y (push) Successful in 1m0s
Details
CI / perf (push) Successful in 3m10s
Details
feat(infra): reactivate act_runner cache by sharing the runners network (#82) 
## Summary

Closes the deferred-since-day-one cache-server gap (documented as "Cache server (deferred)" in `infra/README.md` and mentioned every time we hit a slow CI install).

**Root cause.** `act_runner`'s built-in cache server binds inside the runner container and advertises an IP on the compose-defined `apf-portal-act-runners` bridge — but jobs are spawned via the mounted `/var/run/docker.sock`, which puts them on Docker's anonymous default `bridge`. The advertised URL is unreachable from the job, every cache request burns a ~2 min `ETIMEDOUT` (restore + save), the hit rate is zero.

**Fix.** Tell `act_runner` to attach jobs to the same compose-defined bridge as the runners, via `container.network` in the shared `runner-config.yaml`. The advertised cache URL becomes a normal internal-network DNS hop, jobs reach the cache server, `cache: 'pnpm'` works end-to-end.

**Blast-radius trade-off** (bounded). Every container on `apf-portal-act-runners` is one of our runner containers, plus the jobs they spawn — all of which already have full docker-socket access. Sharing a network doesn't widen what a malicious workflow can already do; it just lets jobs reach the cache server.

## What lands

- `infra/runner-config.yaml` — add `container.network: apf-portal-act-runners`. Surface the `cache.enabled: true` default explicitly so the toggle is discoverable.
- `.gitea/workflows/ci.yml` — re-enable `cache: 'pnpm'` on every `actions/setup-node` step (5 jobs). Drop the now-stale block comment that explained the disablement.
- `.gitea/workflows/security-scheduled.yml` — same on the two setup-node steps.
- `infra/README.md` "Cache server" section rewritten — was `"(deferred)"`, now describes the working setup, rationale, and the disable toggle.
- `ci.yml`'s Trivy comment trimmed to drop the cross-reference to the deferred-cache-server section that no longer exists.

## Roll-out (manual, post-merge, on the runner host)

```bash
cd <repo>/infra
git pull
./ci-runners.sh rotate
```

`rotate` recreates the containers with the new `runner-config.yaml` mount intact (rolling restart, ~15 s pause between each runner so the CI pipeline stays online).

## Test plan

- [ ] CI green on this PR (the gates run on the runners as configured **before** rollout, so this PR's run is one last "uncached" cycle).
- [ ] After rollout, the next CI run's `Set up Node.js` step shows the cache restore attempt **succeed quickly** (no ETIMEDOUT). The `Run pnpm install --frozen-lockfile` step on the first post-rollout run still reports `Progress: resolved N, reused 0, downloaded N` (cold seed).
- [ ] The **second** post-rollout run reports `reused N, downloaded 0` (or a small downloaded delta if Renovate moved a dep meanwhile) — the cache hit is real.
- [ ] `Complete job` step at the end no longer shows `reserveCache failed: connect ETIMEDOUT` warnings.
- [ ] Wall-clock for a typical PR's CI drops by ~5-10 min (5 jobs × ~30-90 s saved on `pnpm install` + the 2× ~2 min ETIMEDOUTs we used to eat).

---------

Co-authored-by: Julien Gautier <julien.gautier@apf.asso.fr>
Reviewed-on: #82
2026-05-10 19:03:58 +02:00

237 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# `infra/`
Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).
| Subject | File / Folder | ADR / Reference |
| -------------------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Self-hosted CI runners (Gitea Actions) | [`ci-runners.compose.yml`](ci-runners.compose.yml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
| Shared `act_runner` configuration | [`runner-config.yaml`](runner-config.yaml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
| CI runners convenience script | [`ci-runners.sh`](ci-runners.sh) | See "Convenience script" below |
| Runtime state of the runners | `data/` (git-ignored after `.gitignore`) | — |
| Env-vars template for the runners | `.env.example` (`.env` is git-ignored) | — |
| Local-dev runtime stack | [`local/`](local/) | [ADR-0006](../docs/decisions/0006-persistence-postgresql-prisma.md), [ADR-0010](../docs/decisions/0010-session-management-redis.md), [ADR-0012](../docs/decisions/0012-observability-pino-opentelemetry.md), [ADR-0013](../docs/decisions/0013-audit-trail-separated-postgres-append-only.md) |
Future folders / files that will land here as the corresponding ADRs ship:
- **`prod/`** — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).
---
## CI runners — `ci-runners.compose.yml`
Three self-hosted [`act_runner`](https://gitea.com/gitea/act_runner) instances, registered with the project's Gitea organisation, labelled `self-hosted` + `on-prem` (the labels referenced by every job in `.gitea/workflows/*`). Three matches the floor recommended by [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.
### First-time registration
```bash
cd infra/
# 1. Generate a registration token in Gitea.
# Site Administration → Actions → Runners → "Create new Runner"
# (or, for org-scoped runners: Organisation Settings → Actions → Runners).
# The token is one-time and short-lived; don't lose it.
# 2. Configure .env (which is git-ignored).
cp .env.example .env
$EDITOR .env
# Set GITEA_INSTANCE_URL (https, no trailing slash) and
# GITEA_RUNNER_REGISTRATION_TOKEN.
# 3. Pre-pull the job images and bring the runners up. The script
# chains the two — see "Job image pinning and pre-pull" below
# for the rationale.
./ci-runners.sh up --prepull
# 4. Verify in Gitea: the three runners appear as online with the
# self-hosted, on-prem labels. If a runner doesn't come online,
# inspect its logs:
./ci-runners.sh logs runner-1
```
After the first successful boot, each runner stores its credentials under `data/runner-N/.runner`. The registration token is no longer needed and **should be removed** from `.env`. Subsequent restarts (`./ci-runners.sh restart …` or direct `docker compose restart …`) authenticate from the persisted credential.
### Convenience script — `ci-runners.sh`
[`ci-runners.sh`](ci-runners.sh) is a thin wrapper around `docker compose -f ci-runners.compose.yml ...` for the everyday verbs. Two reasons to use it:
1. **Hides the compose-file path** on every command. `./ci-runners.sh up` instead of `docker compose -f ci-runners.compose.yml up -d`.
2. **`rotate` automates the rolling restart** the "Operational tips" below recommend: runner-1 → wait → runner-2 → wait → runner-3, so the CI pipeline always has at least N-1 runners online while you push a config change.
| Command | Effect |
| ---------------------------------- | ------------------------------------------------------------------------------ |
| `./ci-runners.sh up` | Bring the three runner containers up |
| `./ci-runners.sh up --prepull` | Pre-pull the job images (`act-22.04` + `:full-22.04`) on the host first |
| `./ci-runners.sh down` | Stop and remove the containers (preserves `data/runner-N/.runner` credentials) |
| `./ci-runners.sh restart <runner>` | Restart one runner |
| `./ci-runners.sh rotate` | Rolling restart of every runner with a 15 s pause between each |
| `./ci-runners.sh status` | `docker compose ps` for the runner services |
| `./ci-runners.sh logs [runner]` | Follow logs (one runner or all of them) |
| `./ci-runners.sh pull-images` | Pre-pull / refresh the job images (idempotent) |
Anything not matching one of the named verbs is passed through to `docker compose -f ci-runners.compose.yml ...`. Run `./ci-runners.sh help` for the full reference.
For the destructive `down -v` (wipes `data/`, forces re-registration with a fresh Gitea token), the script intentionally **doesn't** offer a verb — invoke `docker compose -f ci-runners.compose.yml down -v` directly so the path is explicit at the typing level.
### Operational tips
- **Rotation of one runner at a time** — to upgrade the image or change config, run `./ci-runners.sh rotate` (or restart manually one by one — `./ci-runners.sh restart runner-1`, wait, …) so the CI pipeline is never starved.
- **Logs** — `./ci-runners.sh logs runner-N` (or `docker compose logs -f --tail=100 runner-N`) for a single runner; jobs being executed appear here.
- **Disk pressure** — the runner caches each job's container image in `/var/lib/docker` on the host. On a small host, prune periodically (`docker system prune -af` while no job is running).
- **Adding a fourth runner** — copy any `runner-N` block in the compose file, increment the suffix in `container_name`, `GITEA_RUNNER_NAME`, and the `data/` mount path. Add the new name to the `RUNNERS=(…)` array at the top of `ci-runners.sh` so `rotate` and `restart` learn about it. Then `./ci-runners.sh up` (or `docker compose up -d`). The runner registers using the same `GITEA_RUNNER_REGISTRATION_TOKEN` (which must be regenerated if it has expired).
### Security — Docker socket exposure
The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn containers. **This grants the runner root-equivalent access to the host's Docker daemon.** A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:
- **Trust boundary:** only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
- **Dedicated host:** run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
- **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes.
- **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.
### Cache server
`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v6` (`cache: 'pnpm'`), `actions/cache`, and similar. The default behaviour does **not** work in our compose-based setup: the runner container is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned through the mounted `/var/run/docker.sock` come up on Docker's anonymous `bridge` network — the cache server binds inside the runner on a random port, advertises an IP on the runners' bridge, and the job can't reach it. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching.
The fix is in [`runner-config.yaml`](runner-config.yaml): `container.network: apf-portal-act-runners` instructs `act_runner` to attach every job container to the same compose-defined bridge as the runners. Job → runner is now an internal-network DNS hop, the advertised cache URL is reachable, and `cache: 'pnpm'` works end-to-end. The `cache: 'pnpm'` flag is enabled on every `actions/setup-node` step in `.gitea/workflows/ci.yml` and `.gitea/workflows/security-scheduled.yml`.
The blast-radius trade-off is bounded: every container on `apf-portal-act-runners` is one of our runner containers (plus the jobs they spawn), all of which already have full docker-socket access. Sharing a network does not widen what a malicious workflow can already do; it just lets jobs reach the cache server.
If the cache ever needs to be disabled (debugging cache-hit issues, etc.), set `cache.enabled: false` in `runner-config.yaml` and `./ci-runners.sh rotate`.
### `act_runner` image pinning
The compose pins `gitea/act_runner:0.2.13`. Update the pin deliberately, not via `:latest`:
1. Read the act_runner [release notes](https://gitea.com/gitea/act_runner/releases) for breaking changes.
2. Edit the three image references (`runner-1`, `runner-2`, `runner-3`).
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject.
4. Roll one runner at a time (rotation tip above).
The matching CI workflows refer to runner _labels_ (not images), so a runner-image upgrade does not affect `.gitea/workflows/*`.
### Job image pinning and pre-pull
`act_runner` runs each job inside a container whose image is selected by the runner's _labels_. Two images are in use:
| Label | Image | Used by |
| ---------------------- | -------------------------------- | ---------------------------------- |
| `self-hosted` | `catthehacker/ubuntu:act-22.04` | `check`, `scan`, `commits`, `a11y` |
| `on-prem` | `catthehacker/ubuntu:act-22.04` | (alias of `self-hosted`) |
| (per-job `container:`) | `catthehacker/ubuntu:full-22.04` | `perf` (Lighthouse needs Chrome) |
[`runner-config.yaml`](runner-config.yaml) sets `container.force_pull: false`. Without that, act_runner re-issues a `docker pull` at the start of every single job (~10–30 s of registry round-trip even when every layer is already cached), which both wastes wall-clock and contradicts our policy of upgrading job images deliberately rather than implicitly via `:latest`.
The trade-off: the host Docker daemon must already hold the images locally. Pre-pull them once after a fresh runner host install:
```bash
docker pull catthehacker/ubuntu:act-22.04
docker pull catthehacker/ubuntu:full-22.04
```
Upgrading to a newer tag is a deliberate three-step process:
1. Edit `GITEA_RUNNER_LABELS` (in [`ci-runners.compose.yml`](ci-runners.compose.yml)) and / or the per-job `container.image:` (in `.gitea/workflows/*`) to the new tag.
2. On the runner host, `docker pull <new-tag>` so the image is locally available before the next CI job starts.
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject; one of `chore(deps): upgrade CI job image to ...`.
Old, no-longer-referenced images can be reaped during the periodic `docker system prune -af` (see "Disk pressure" above).
---
## Local-dev stack — `local/`
A Docker Compose recipe spinning up the runtime services the BFF and ADRs assume — Postgres, Redis, OpenTelemetry Collector — plus optional viewers (pgweb, Jaeger UI) gated behind Compose profiles. Designed to start in a single command on a contributor's WSL2 / Linux / macOS host.
| File | Role |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`local/dev.sh`](local/dev.sh) | Convenience wrapper around `docker compose` — see "Convenience script" below |
| [`local/dev.compose.yml`](local/dev.compose.yml) | Service definitions: postgres, redis, otel-collector, plus pgweb and jaeger behind profiles |
| [`local/.env.example`](local/.env.example) | Credentials + ports template (copy to `.env`, which is git-ignored) |
| [`local/init/postgres/01-init.sql`](local/init/postgres/01-init.sql) | Bootstrap SQL for ADR-0013: audit roles + schema, applied on first boot only |
| [`local/otel-collector.yaml`](local/otel-collector.yaml) | Collector pipeline: OTLP receivers → batch → debug exporter (always) + forward to Jaeger when active |
### First-time setup
```bash
# 1. Configure local secrets (copy template, edit, do not commit).
cp infra/local/.env.example infra/local/.env
$EDITOR infra/local/.env
# Set strong dev values for POSTGRES_PASSWORD and REDIS_PASSWORD
# (defaults in the template are placeholders that the compose
# rejects with `must be set in infra/local/.env` if left as-is).
# 2. Bring up the core stack (postgres + redis + otel-collector).
./infra/local/dev.sh up
# 3. (Optional) Activate viewers when needed:
./infra/local/dev.sh up dbtools # adds pgweb
./infra/local/dev.sh up observability # adds Jaeger UI
./infra/local/dev.sh up all # core + every profile
# 4. Verify health.
./infra/local/dev.sh status
```
### Convenience script — `dev.sh`
[`local/dev.sh`](local/dev.sh) is a thin wrapper around `docker compose -f dev.compose.yml ...` with two reasons to exist:
1. **Hides the Compose-profile gotcha.** `docker compose down` only operates on services whose profile is currently active — anything started under `--profile X` keeps running unless the same flag is on `down`. The script always passes every profile in scope on teardown / status / log commands, so profile-gated services (pgweb, Jaeger) are never accidentally orphaned.
2. **Ergonomic verbs** for the common workflows. `./dev.sh up all`, `./dev.sh stop pgweb`, `./dev.sh logs otel-collector`, etc.
Run `./infra/local/dev.sh help` for the full reference. Cheat-sheet:
| Command | Effect |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `./infra/local/dev.sh up` | Core only (postgres + redis + otel-collector) |
| `./infra/local/dev.sh up all` | Core + every profile |
| `./infra/local/dev.sh up dbtools` | Core + pgweb |
| `./infra/local/dev.sh up observability` | Core + Jaeger |
| `./infra/local/dev.sh down` | Tear down the whole stack (every profile in scope) |
| `./infra/local/dev.sh down -v` | Tear down + wipe named volumes (incl. audit-roles bootstrap) |
| `./infra/local/dev.sh stop pgweb` | Stop one service (containers stay around) |
| `./infra/local/dev.sh status` | `docker compose ps`, with every profile visible |
| `./infra/local/dev.sh logs otel-collector` | Follow logs |
| `./infra/local/dev.sh exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"` | Run a command inside a service |
Anything not matching one of the named verbs is passed through to `docker compose -f dev.compose.yml ...` (with every profile flagged in), so you keep the full Compose surface available — `./dev.sh config`, `./dev.sh top`, `./dev.sh inspect …`, etc.
If you prefer to call `docker compose` directly, every example below shows the raw command alongside the script form.
### Service endpoints (defaults)
| Service | Host port | Purpose |
| ------------------- | --------- | ------------------------------------------------------------------- |
| Postgres | 5432 | DB connection — `postgres://portal:<pwd>@localhost:5432/portal_dev` |
| Redis | 6379 | Sessions, OBO cache (per ADR-0010 / ADR-0014) |
| OTel Collector gRPC | 4317 | `OTEL_EXPORTER_OTLP_ENDPOINT` for the BFF and the SPA |
| OTel Collector HTTP | 4318 | OTLP/HTTP variant |
| pgweb (profile) | 8081 | http://localhost:8081 — Postgres GUI |
| Jaeger UI (profile) | 16686 | http://localhost:16686 — trace explorer |
All ports are overridable via `.env` if the host machine has conflicts.
### Operational tips
- **Persistence** — state lives in named Docker volumes (`apf-portal-postgres-data`, `apf-portal-redis-data`). Survives `docker compose down`. Use `docker compose -f dev.compose.yml down -v` to wipe (also wipes the audit-roles bootstrap, which re-runs on the next fresh boot).
- **Profile symmetry** — `dev.sh down` (and `status`, `logs`, …) always include every profile in scope, so profile-gated services are caught. If you bypass the script and call `docker compose down` directly, you must pass the same `--profile` flags as on `up`, otherwise pgweb and Jaeger keep running silently. Either pass them again, or `export COMPOSE_PROFILES=dbtools,observability` in your shell or `infra/local/.env`.
- **Bootstrap re-run** — the SQL in `local/init/postgres/` only runs on a **fresh** Postgres data volume. To replay after editing the file, `down -v` (loses all dev data) or run the SQL manually with `docker compose exec postgres psql -U portal -d portal_dev -f /docker-entrypoint-initdb.d/01-init.sql`.
- **Logs** — `docker compose -f dev.compose.yml logs -f <service>` to follow a single service. `otel-collector` is the loudest — its `debug` exporter prints every span / metric / log it receives.
- **Image upgrades** — same policy as the runner image (deliberate, not via `:latest`). Renovate's docker-compose manager will surface bumps automatically once the dashboard rule allows them.
### Production parity
This stack is **dev-only**. The corresponding production layout (HA Postgres, Redis Sentinel cluster, OTel Collector with a real backend, secret manager) lives in the future on-prem-infrastructure ADR — see `prod/` placeholder below.
---
## Future infra concerns — placeholders
These are listed here so a contributor knows where to expect related files; they don't exist yet.
| File | Purpose | Triggered by |
| --------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `prod/*` | On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on) | The on-prem infrastructure ADR (phase 3b) |
| `runbooks/*.md` | Operational runbooks (incident response, secret rotation, runner upgrade procedure, …) | First incident, or when ops cadence justifies them |
Reference in New Issue View Git Blame Copy Permalink