From 2f411789356d38986f8fad34f62e22dd33fbbd15 Mon Sep 17 00:00:00 2001 From: Julien Gautier Date: Sun, 10 May 2026 18:54:04 +0200 Subject: [PATCH] feat(infra): reactivate act_runner cache by sharing the runners network MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deferred-since-day-one cache-server gap (documented as "Cache server (deferred)" in infra/README.md, mentioned every time we hit a slow CI install). Root cause: act_runner's built-in cache server binds inside the runner container and advertises an IP on the compose-defined `apf-portal-act-runners` bridge — but jobs are spawned via the mounted /var/run/docker. sock, which puts them on Docker's anonymous default `bridge` instead. The advertised URL is unreachable from the job, every cache request burns a ~2 min ETIMEDOUT (restore + save), the hit rate is zero. Fix: tell act_runner to attach jobs to the same compose-defined bridge as the runners, via `container.network` in the shared runner-config.yaml. The advertised cache URL becomes a normal internal-network DNS hop, jobs reach the cache server, and `cache: 'pnpm'` works end-to-end. The blast-radius trade-off is bounded: every container on the apf-portal-act-runners network is one of our runner containers (plus the jobs they spawn), all of which already have full docker-socket access. Sharing a network does not widen what a malicious workflow can already do; it just lets jobs reach the cache server. Changes: - infra/runner-config.yaml: add `container.network: apf-portal- act-runners`. Surface the `cache.enabled: true` default explicitly so a future contributor knows where the toggle is. - .gitea/workflows/ci.yml: re-enable `cache: 'pnpm'` on every actions/setup-node step (5 jobs). Drop the now-stale block comment that explained the disablement. - .gitea/workflows/security-scheduled.yml: same on the two setup-node steps in this workflow. - infra/README.md "Cache server" section rewritten — was "(deferred)", now describes the working setup, with the rationale and blast-radius note. The disable toggle is documented inline. - ci.yml's Trivy comment trimmed to drop the cross-reference to the deferred-cache-server section that no longer exists. Roll-out on the runner host (manual, post-merge): cd /infra git pull ./ci-runners.sh rotate `rotate` recreates the containers with the new runner-config.yaml mount intact. The first CI job after rollout seeds the cache from cold (~30-60 s install); subsequent jobs should report `reused N` instead of `downloaded N` in the `pnpm install --frozen-lockfile` line and finish a few minutes faster overall. --- .gitea/workflows/ci.yml | 18 ++++++------------ .gitea/workflows/security-scheduled.yml | 2 ++ infra/README.md | 13 +++++-------- infra/runner-config.yaml | 19 +++++++++++++++++++ 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index c4be67d..7123a19 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -2,15 +2,6 @@ # Thin YAML — orchestration lives in package.json scripts (ci:check, # ci:audit, ci:commits, ci:perf) and Nx targets. Any change to gate # behaviour belongs in those scripts, not in this file. -# -# `cache: 'pnpm'` is intentionally NOT enabled on actions/setup-node -# below: act_runner's built-in GitHub-Actions-cache server is bound on -# the runner container and is unreachable from job containers (which -# run on Docker's default network, not the runners' compose network). -# Each request burns a ~2 min ETIMEDOUT on restore + another on save, -# for zero hit rate. Once the runner network/cache config is fixed -# (tracked in infra/README.md "Cache server"), re-add `cache: 'pnpm'` -# here. name: CI @@ -50,6 +41,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm ci:check @@ -71,14 +63,13 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' # Dependency vulnerability scan. Trivy is a Go binary, not an npm # package, so it cannot live in package.json scripts as cleanly # as audit/lint do. # # We deliberately avoid `aquasecurity/trivy-action`. On cache - # miss (our act_runner cache server is unreachable from job - # containers — see infra/README.md "Cache server (deferred)"), - # the action falls back to `git clone github.com/aquasecurity/ + # miss the action falls back to `git clone github.com/aquasecurity/ # trivy` to fetch its install script, using `actions/checkout` # which defaults `with.token` to `${{ github.token }}` (Gitea's # auto-token, useless for github.com). The clone hits the @@ -171,6 +162,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: COMMIT_LINT_FROM=origin/main pnpm ci:commits @@ -195,6 +187,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm ci:perf - uses: actions/upload-artifact@v7 @@ -212,6 +205,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile # Placeholder until the e2e a11y suite (axe-core via Playwright, # per ADR-0016) is wired with the first real screens. The job diff --git a/.gitea/workflows/security-scheduled.yml b/.gitea/workflows/security-scheduled.yml index 9e890a5..15ad38a 100644 --- a/.gitea/workflows/security-scheduled.yml +++ b/.gitea/workflows/security-scheduled.yml @@ -33,6 +33,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' # Full-tree Trivy (no skip-dirs, no severity filter — the per-PR # gate filters by severity for speed; this run wants the full # surface for the security feed). Manual install + curl, same @@ -92,6 +93,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm exec lhci collect --url=${{ vars.LHCI_PROD_URL }} --numberOfRuns=3 - run: pnpm exec lhci assert --config=./lighthouserc.js diff --git a/infra/README.md b/infra/README.md index b1d1d1f..03e82fa 100644 --- a/infra/README.md +++ b/infra/README.md @@ -88,18 +88,15 @@ The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn con - **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes. - **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value. -### Cache server (deferred) +### Cache server -`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v4` (`cache: 'pnpm'`), `actions/cache`, and similar. In the current setup it does **not** work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned via the mounted `/var/run/docker.sock` come up on Docker's default `bridge` network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching. +`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v6` (`cache: 'pnpm'`), `actions/cache`, and similar. The default behaviour does **not** work in our compose-based setup: the runner container is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned through the mounted `/var/run/docker.sock` come up on Docker's anonymous `bridge` network — the cache server binds inside the runner on a random port, advertises an IP on the runners' bridge, and the job can't reach it. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching. -For now `cache: 'pnpm'` is left **disabled** in `.gitea/workflows/ci.yml`. `pnpm install --frozen-lockfile` is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything. +The fix is in [`runner-config.yaml`](runner-config.yaml): `container.network: apf-portal-act-runners` instructs `act_runner` to attach every job container to the same compose-defined bridge as the runners. Job → runner is now an internal-network DNS hop, the advertised cache URL is reachable, and `cache: 'pnpm'` works end-to-end. The `cache: 'pnpm'` flag is enabled on every `actions/setup-node` step in `.gitea/workflows/ci.yml` and `.gitea/workflows/security-scheduled.yml`. -When this is reactivated, the right fix is one of: +The blast-radius trade-off is bounded: every container on `apf-portal-act-runners` is one of our runner containers (plus the jobs they spawn), all of which already have full docker-socket access. Sharing a network does not widen what a malicious workflow can already do; it just lets jobs reach the cache server. -- attach jobs to the same Docker network as the runners (`runner.container.network` in act_runner's `config.yaml`, then advertise the cache `host` on the bridge IP); or -- bind act_runner's cache server on a fixed `host_port` reachable from any container (host gateway IP), and set `ACTIONS_CACHE_URL` accordingly. - -Either path needs a single-runner spike before being rolled out to the three. +If the cache ever needs to be disabled (debugging cache-hit issues, etc.), set `cache.enabled: false` in `runner-config.yaml` and `./ci-runners.sh rotate`. ### `act_runner` image pinning diff --git a/infra/runner-config.yaml b/infra/runner-config.yaml index 887cd72..06066ba 100644 --- a/infra/runner-config.yaml +++ b/infra/runner-config.yaml @@ -14,3 +14,22 @@ container: # host (see infra/README.md "Updating the runner job images") — not # implicitly via :latest. force_pull: false + + # Attach every job to the same Docker bridge network as the runner + # containers themselves. Default behaviour (no value) is for jobs to + # join Docker's anonymous "bridge" network — different from the + # compose-defined `apf-portal-act-runners` bridge — which leaves + # jobs unable to reach the runner-hosted cache server (the IP it + # binds to and advertises is on the runners' bridge, not the jobs'). + # Sharing the network closes the gap and lets `actions/setup-node`'s + # `cache: 'pnpm'` work end-to-end. The blast radius is bounded: + # every container on this network is one of our runner containers, + # all of which already have full docker-socket access. + network: apf-portal-act-runners + +# Built-in cache server — defaults are good (enabled, listens on a +# pseudo-random port, stores cache under the runner's writable home). +# Documented here only so a future contributor knows where the toggle +# lives if it has to be disabled. +cache: + enabled: true