From a46319972878fbcfa8f53bbbed1497760f723d0c Mon Sep 17 00:00:00 2001 From: Julien Gautier Date: Sun, 10 May 2026 19:03:58 +0200 Subject: [PATCH] feat(infra): reactivate act_runner cache by sharing the runners network (#82) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Closes the deferred-since-day-one cache-server gap (documented as "Cache server (deferred)" in `infra/README.md` and mentioned every time we hit a slow CI install). **Root cause.** `act_runner`'s built-in cache server binds inside the runner container and advertises an IP on the compose-defined `apf-portal-act-runners` bridge — but jobs are spawned via the mounted `/var/run/docker.sock`, which puts them on Docker's anonymous default `bridge`. The advertised URL is unreachable from the job, every cache request burns a ~2 min `ETIMEDOUT` (restore + save), the hit rate is zero. **Fix.** Tell `act_runner` to attach jobs to the same compose-defined bridge as the runners, via `container.network` in the shared `runner-config.yaml`. The advertised cache URL becomes a normal internal-network DNS hop, jobs reach the cache server, `cache: 'pnpm'` works end-to-end. **Blast-radius trade-off** (bounded). Every container on `apf-portal-act-runners` is one of our runner containers, plus the jobs they spawn — all of which already have full docker-socket access. Sharing a network doesn't widen what a malicious workflow can already do; it just lets jobs reach the cache server. ## What lands - `infra/runner-config.yaml` — add `container.network: apf-portal-act-runners`. Surface the `cache.enabled: true` default explicitly so the toggle is discoverable. - `.gitea/workflows/ci.yml` — re-enable `cache: 'pnpm'` on every `actions/setup-node` step (5 jobs). Drop the now-stale block comment that explained the disablement. - `.gitea/workflows/security-scheduled.yml` — same on the two setup-node steps. - `infra/README.md` "Cache server" section rewritten — was `"(deferred)"`, now describes the working setup, rationale, and the disable toggle. - `ci.yml`'s Trivy comment trimmed to drop the cross-reference to the deferred-cache-server section that no longer exists. ## Roll-out (manual, post-merge, on the runner host) ```bash cd /infra git pull ./ci-runners.sh rotate ``` `rotate` recreates the containers with the new `runner-config.yaml` mount intact (rolling restart, ~15 s pause between each runner so the CI pipeline stays online). ## Test plan - [ ] CI green on this PR (the gates run on the runners as configured **before** rollout, so this PR's run is one last "uncached" cycle). - [ ] After rollout, the next CI run's `Set up Node.js` step shows the cache restore attempt **succeed quickly** (no ETIMEDOUT). The `Run pnpm install --frozen-lockfile` step on the first post-rollout run still reports `Progress: resolved N, reused 0, downloaded N` (cold seed). - [ ] The **second** post-rollout run reports `reused N, downloaded 0` (or a small downloaded delta if Renovate moved a dep meanwhile) — the cache hit is real. - [ ] `Complete job` step at the end no longer shows `reserveCache failed: connect ETIMEDOUT` warnings. - [ ] Wall-clock for a typical PR's CI drops by ~5-10 min (5 jobs × ~30-90 s saved on `pnpm install` + the 2× ~2 min ETIMEDOUTs we used to eat). --------- Co-authored-by: Julien Gautier Reviewed-on: https://git.unespace.com/julien/apf_portal/pulls/82 --- .gitea/workflows/ci.yml | 18 ++++++------------ .gitea/workflows/security-scheduled.yml | 2 ++ infra/README.md | 13 +++++-------- infra/runner-config.yaml | 19 +++++++++++++++++++ 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index c4be67d..7123a19 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -2,15 +2,6 @@ # Thin YAML — orchestration lives in package.json scripts (ci:check, # ci:audit, ci:commits, ci:perf) and Nx targets. Any change to gate # behaviour belongs in those scripts, not in this file. -# -# `cache: 'pnpm'` is intentionally NOT enabled on actions/setup-node -# below: act_runner's built-in GitHub-Actions-cache server is bound on -# the runner container and is unreachable from job containers (which -# run on Docker's default network, not the runners' compose network). -# Each request burns a ~2 min ETIMEDOUT on restore + another on save, -# for zero hit rate. Once the runner network/cache config is fixed -# (tracked in infra/README.md "Cache server"), re-add `cache: 'pnpm'` -# here. name: CI @@ -50,6 +41,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm ci:check @@ -71,14 +63,13 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' # Dependency vulnerability scan. Trivy is a Go binary, not an npm # package, so it cannot live in package.json scripts as cleanly # as audit/lint do. # # We deliberately avoid `aquasecurity/trivy-action`. On cache - # miss (our act_runner cache server is unreachable from job - # containers — see infra/README.md "Cache server (deferred)"), - # the action falls back to `git clone github.com/aquasecurity/ + # miss the action falls back to `git clone github.com/aquasecurity/ # trivy` to fetch its install script, using `actions/checkout` # which defaults `with.token` to `${{ github.token }}` (Gitea's # auto-token, useless for github.com). The clone hits the @@ -171,6 +162,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: COMMIT_LINT_FROM=origin/main pnpm ci:commits @@ -195,6 +187,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm ci:perf - uses: actions/upload-artifact@v7 @@ -212,6 +205,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile # Placeholder until the e2e a11y suite (axe-core via Playwright, # per ADR-0016) is wired with the first real screens. The job diff --git a/.gitea/workflows/security-scheduled.yml b/.gitea/workflows/security-scheduled.yml index 9e890a5..15ad38a 100644 --- a/.gitea/workflows/security-scheduled.yml +++ b/.gitea/workflows/security-scheduled.yml @@ -33,6 +33,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' # Full-tree Trivy (no skip-dirs, no severity filter — the per-PR # gate filters by severity for speed; this run wants the full # surface for the security feed). Manual install + curl, same @@ -92,6 +93,7 @@ jobs: - uses: actions/setup-node@v6 with: node-version-file: '.nvmrc' + cache: 'pnpm' - run: pnpm install --frozen-lockfile - run: pnpm exec lhci collect --url=${{ vars.LHCI_PROD_URL }} --numberOfRuns=3 - run: pnpm exec lhci assert --config=./lighthouserc.js diff --git a/infra/README.md b/infra/README.md index b1d1d1f..03e82fa 100644 --- a/infra/README.md +++ b/infra/README.md @@ -88,18 +88,15 @@ The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn con - **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes. - **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value. -### Cache server (deferred) +### Cache server -`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v4` (`cache: 'pnpm'`), `actions/cache`, and similar. In the current setup it does **not** work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned via the mounted `/var/run/docker.sock` come up on Docker's default `bridge` network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching. +`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v6` (`cache: 'pnpm'`), `actions/cache`, and similar. The default behaviour does **not** work in our compose-based setup: the runner container is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned through the mounted `/var/run/docker.sock` come up on Docker's anonymous `bridge` network — the cache server binds inside the runner on a random port, advertises an IP on the runners' bridge, and the job can't reach it. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching. -For now `cache: 'pnpm'` is left **disabled** in `.gitea/workflows/ci.yml`. `pnpm install --frozen-lockfile` is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything. +The fix is in [`runner-config.yaml`](runner-config.yaml): `container.network: apf-portal-act-runners` instructs `act_runner` to attach every job container to the same compose-defined bridge as the runners. Job → runner is now an internal-network DNS hop, the advertised cache URL is reachable, and `cache: 'pnpm'` works end-to-end. The `cache: 'pnpm'` flag is enabled on every `actions/setup-node` step in `.gitea/workflows/ci.yml` and `.gitea/workflows/security-scheduled.yml`. -When this is reactivated, the right fix is one of: +The blast-radius trade-off is bounded: every container on `apf-portal-act-runners` is one of our runner containers (plus the jobs they spawn), all of which already have full docker-socket access. Sharing a network does not widen what a malicious workflow can already do; it just lets jobs reach the cache server. -- attach jobs to the same Docker network as the runners (`runner.container.network` in act_runner's `config.yaml`, then advertise the cache `host` on the bridge IP); or -- bind act_runner's cache server on a fixed `host_port` reachable from any container (host gateway IP), and set `ACTIONS_CACHE_URL` accordingly. - -Either path needs a single-runner spike before being rolled out to the three. +If the cache ever needs to be disabled (debugging cache-hit issues, etc.), set `cache.enabled: false` in `runner-config.yaml` and `./ci-runners.sh rotate`. ### `act_runner` image pinning diff --git a/infra/runner-config.yaml b/infra/runner-config.yaml index 887cd72..06066ba 100644 --- a/infra/runner-config.yaml +++ b/infra/runner-config.yaml @@ -14,3 +14,22 @@ container: # host (see infra/README.md "Updating the runner job images") — not # implicitly via :latest. force_pull: false + + # Attach every job to the same Docker bridge network as the runner + # containers themselves. Default behaviour (no value) is for jobs to + # join Docker's anonymous "bridge" network — different from the + # compose-defined `apf-portal-act-runners` bridge — which leaves + # jobs unable to reach the runner-hosted cache server (the IP it + # binds to and advertises is on the runners' bridge, not the jobs'). + # Sharing the network closes the gap and lets `actions/setup-node`'s + # `cache: 'pnpm'` work end-to-end. The blast radius is bounded: + # every container on this network is one of our runner containers, + # all of which already have full docker-socket access. + network: apf-portal-act-runners + +# Built-in cache server — defaults are good (enabled, listens on a +# pseudo-random port, stores cache under the runner's writable home). +# Documented here only so a future contributor knows where the toggle +# lives if it has to be disabled. +cache: + enabled: true