julien/apf_portal
1
0
Fork 0
Code Issues 1 Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
288610b9ba9b3bbe7df81859949f48055cacb7bb
apf_portal/infra/README.md
T
julien fc9b63f24a
CI / check (push) Successful in 1m36s
Details
CI / commits (push) Has been skipped
Details
CI / scan (push) Successful in 2m9s
Details
CI / a11y (push) Successful in 1m9s
Details
CI / perf (push) Successful in 3m1s
Details
feat(infra): add dev.sh wrapper for the local-dev compose stack (#68) 
## Summary

Two recurring frictions on the local-dev stack:

1. **Compose-profile asymmetry** — `docker compose down` only operates on services whose profile is currently active. Anything brought up with `--profile X` keeps running unless the same flag is passed on `down`. pgweb and Jaeger silently survived several `down -v` invocations before we noticed (#67 documented the gotcha; this PR makes it impossible to hit if you use the wrapper).
2. **Verbose invocations** — typing `docker compose -f infra/local/dev.compose.yml --profile … <verb>` for routine ops gets old fast.

Add [`infra/local/dev.sh`](infra/local/dev.sh) as a thin wrapper. Always passes every profile in scope on teardown / status / log commands, exposes ergonomic verbs:

| Command                                       | Effect                                                          |
| --------------------------------------------- | --------------------------------------------------------------- |
| `./infra/local/dev.sh up`                     | Core only (postgres + redis + otel-collector)                   |
| `./infra/local/dev.sh up all`                 | Core + every profile                                            |
| `./infra/local/dev.sh up dbtools`             | Core + pgweb                                                    |
| `./infra/local/dev.sh up observability`       | Core + Jaeger                                                   |
| `./infra/local/dev.sh down [-v]`              | Tear down (every profile in scope, no orphaned services)        |
| `./infra/local/dev.sh stop <service>`         | Stop one service (containers stay around)                       |
| `./infra/local/dev.sh restart <service>`      | Restart one service                                             |
| `./infra/local/dev.sh status`                 | `ps` with every profile visible                                 |
| `./infra/local/dev.sh logs [service]`         | Follow logs                                                     |
| `./infra/local/dev.sh exec <service> <cmd>`   | Run a command inside a container                                |

Anything not matching one of the named verbs is passed through to `docker compose -f dev.compose.yml ...` (with every profile flagged in), so the full Compose surface remains available.

## Doc updates

- **`infra/README.md`** — new "Convenience script" subsection with the cheat-sheet table; "First-time setup" rewritten to use the script; the standalone "Profile symmetry" tip from #67 is collapsed into a one-liner since the script now handles it (the note remains as a fallback for direct `docker compose` users).
- **`docs/development.md` §3** — points at the script for the typical setup flow.

The compose file itself is unchanged.

## Test plan

- [ ] `./infra/local/dev.sh help` prints the usage block.
- [ ] `./infra/local/dev.sh up` brings up the 3 core services (no pgweb / no jaeger).
- [ ] `./infra/local/dev.sh up all` adds pgweb and Jaeger.
- [ ] `./infra/local/dev.sh down -v` stops and removes all 5 containers (incl. pgweb and Jaeger), wipes the postgres-data and redis-data volumes.
- [ ] `./infra/local/dev.sh stop pgweb` stops just pgweb.
- [ ] `./infra/local/dev.sh logs otel-collector` follows that service's logs.
- [ ] `./infra/local/dev.sh exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "\du"` lists the audit roles.
- [ ] `./infra/local/dev.sh stop` (no arg) errors with a clear message.

---------

Co-authored-by: Julien Gautier <julien.gautier@apf.asso.fr>
Reviewed-on: #68
2026-05-09 21:50:37 +02:00

220 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# `infra/`
Infrastructure-as-code artefacts for the project. Separate from application code and from documentation: this folder contains the recipes and configs that the team and ops use to stand up running infrastructure (CI runners, future local-dev databases, future on-prem deploy assets).
| Subject | File / Folder | ADR / Reference |
| -------------------------------------- | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Self-hosted CI runners (Gitea Actions) | [`ci-runners.compose.yml`](ci-runners.compose.yml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
| Shared `act_runner` configuration | [`runner-config.yaml`](runner-config.yaml) | [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) |
| Runtime state of the runners | `data/` (git-ignored after `.gitignore`) | — |
| Env-vars template for the runners | `.env.example` (`.env` is git-ignored) | — |
| Local-dev runtime stack | [`local/`](local/) | [ADR-0006](../docs/decisions/0006-persistence-postgresql-prisma.md), [ADR-0010](../docs/decisions/0010-session-management-redis.md), [ADR-0012](../docs/decisions/0012-observability-pino-opentelemetry.md), [ADR-0013](../docs/decisions/0013-audit-trail-separated-postgres-append-only.md) |
Future folders / files that will land here as the corresponding ADRs ship:
- **`prod/`** — On-prem deploy manifests (HA Postgres, Redis Sentinel, OTel collector + backend, secret manager). Triggered by the on-prem infrastructure ADR (phase 3b).
---
## CI runners — `ci-runners.compose.yml`
Three self-hosted [`act_runner`](https://gitea.com/gitea/act_runner) instances, registered with the project's Gitea organisation, labelled `self-hosted` + `on-prem` (the labels referenced by every job in `.gitea/workflows/*`). Three matches the floor recommended by [ADR-0015 §"Runners"](../docs/decisions/0015-cicd-gitea-actions.md) — one runner is enough to validate the pipeline; two leave no slack; three keep CI flowing if one runner is down for upgrade or maintenance.
### First-time registration
```bash
cd infra/
# 1. Generate a registration token in Gitea.
# Site Administration → Actions → Runners → "Create new Runner"
# (or, for org-scoped runners: Organisation Settings → Actions → Runners).
# The token is one-time and short-lived; don't lose it.
# 2. Configure .env (which is git-ignored).
cp .env.example .env
$EDITOR .env
# Set GITEA_INSTANCE_URL (https, no trailing slash) and
# GITEA_RUNNER_REGISTRATION_TOKEN.
# 3. Pre-pull the job images so the runner doesn't have to (see
# "Job image pinning and pre-pull" below for the rationale).
docker pull catthehacker/ubuntu:act-22.04
docker pull catthehacker/ubuntu:full-22.04
# 4. Bring the runners up.
docker compose -f ci-runners.compose.yml up -d
# 5. Verify in Gitea: the three runners appear as online with the
# self-hosted, on-prem labels. If a runner doesn't come online,
# inspect its logs:
docker compose -f ci-runners.compose.yml logs runner-1
```
After the first successful boot, each runner stores its credentials under `data/runner-N/.runner`. The registration token is no longer needed and **should be removed** from `.env`. Subsequent restarts (`docker compose restart`) authenticate from the persisted credential.
### Operational tips
- **Rotation of one runner at a time** — to upgrade the image or change config, restart runners one by one (`docker compose restart runner-1`, wait, runner-2, …) so the CI pipeline is never starved.
- **Logs** — `docker compose logs -f --tail=100 runner-N` for a single runner; jobs being executed appear here.
- **Disk pressure** — the runner caches each job's container image in `/var/lib/docker` on the host. On a small host, prune periodically (`docker system prune -af` while no job is running).
- **Adding a fourth runner** — copy any `runner-N` block in the compose file, increment the suffix in `container_name`, `GITEA_RUNNER_NAME`, and the `data/` mount path. Then `docker compose up -d`. The runner registers using the same `GITEA_RUNNER_REGISTRATION_TOKEN` (which must be regenerated if it has expired).
### Security — Docker socket exposure
The compose mounts `/var/run/docker.sock` into each runner so jobs can spawn containers. **This grants the runner root-equivalent access to the host's Docker daemon.** A malicious workflow could spawn arbitrary containers, mount host paths, escalate privileges. Mitigations:
- **Trust boundary:** only register the runners against repositories controlled by the org. Gitea's runner-registration UI lets you scope a runner to an organisation, a single repository, or instance-wide. Prefer the narrowest scope.
- **Dedicated host:** run these containers on a host that does not also run production services or hold sensitive data. The runner host is in the trust boundary of any developer who can push to a repo it serves.
- **No host filesystem mounts beyond the docker socket:** the compose intentionally does not mount `/`, `/etc`, or any project source. Workflows that need data on the host must do so via Docker volumes.
- **Future hardening (out of scope of v1):** migrate to **rootless Docker** on the runner host, or to a **DinD (Docker-in-Docker) sidecar** so the runner cannot escape into the host daemon. Decided when the org's RSSI confirms the security posture, or when the runner host is shared with anything else of value.
### Cache server (deferred)
`act_runner` ships a built-in GitHub-Actions-cache-compatible server, used by `actions/setup-node@v4` (`cache: 'pnpm'`), `actions/cache`, and similar. In the current setup it does **not** work because the runner containers and the job containers live on different Docker networks: the runner is on the compose-defined `apf-portal-act-runners` bridge, while jobs spawned via the mounted `/var/run/docker.sock` come up on Docker's default `bridge` network. The cache server binds inside the runner container on a random port — unreachable from the job. The symptom is a ~2 min `ETIMEDOUT` at the start (restore) and end (save) of every job that opts into caching.
For now `cache: 'pnpm'` is left **disabled** in `.gitea/workflows/ci.yml`. `pnpm install --frozen-lockfile` is fast enough on a warm pnpm store inside the job image (~30–60 s cold) that the cache layer adds no value as long as it doesn't actually transfer anything.
When this is reactivated, the right fix is one of:
- attach jobs to the same Docker network as the runners (`runner.container.network` in act_runner's `config.yaml`, then advertise the cache `host` on the bridge IP); or
- bind act_runner's cache server on a fixed `host_port` reachable from any container (host gateway IP), and set `ACTIONS_CACHE_URL` accordingly.
Either path needs a single-runner spike before being rolled out to the three.
### `act_runner` image pinning
The compose pins `gitea/act_runner:0.2.13`. Update the pin deliberately, not via `:latest`:
1. Read the act_runner [release notes](https://gitea.com/gitea/act_runner/releases) for breaking changes.
2. Edit the three image references (`runner-1`, `runner-2`, `runner-3`).
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject.
4. Roll one runner at a time (rotation tip above).
The matching CI workflows refer to runner _labels_ (not images), so a runner-image upgrade does not affect `.gitea/workflows/*`.
### Job image pinning and pre-pull
`act_runner` runs each job inside a container whose image is selected by the runner's _labels_. Two images are in use:
| Label | Image | Used by |
| ---------------------- | -------------------------------- | ---------------------------------- |
| `self-hosted` | `catthehacker/ubuntu:act-22.04` | `check`, `scan`, `commits`, `a11y` |
| `on-prem` | `catthehacker/ubuntu:act-22.04` | (alias of `self-hosted`) |
| (per-job `container:`) | `catthehacker/ubuntu:full-22.04` | `perf` (Lighthouse needs Chrome) |
[`runner-config.yaml`](runner-config.yaml) sets `container.force_pull: false`. Without that, act_runner re-issues a `docker pull` at the start of every single job (~10–30 s of registry round-trip even when every layer is already cached), which both wastes wall-clock and contradicts our policy of upgrading job images deliberately rather than implicitly via `:latest`.
The trade-off: the host Docker daemon must already hold the images locally. Pre-pull them once after a fresh runner host install:
```bash
docker pull catthehacker/ubuntu:act-22.04
docker pull catthehacker/ubuntu:full-22.04
```
Upgrading to a newer tag is a deliberate three-step process:
1. Edit `GITEA_RUNNER_LABELS` (in [`ci-runners.compose.yml`](ci-runners.compose.yml)) and / or the per-job `container.image:` (in `.gitea/workflows/*`) to the new tag.
2. On the runner host, `docker pull <new-tag>` so the image is locally available before the next CI job starts.
3. Commit on a feature branch with a `chore(deps):` Conventional Commits subject; one of `chore(deps): upgrade CI job image to ...`.
Old, no-longer-referenced images can be reaped during the periodic `docker system prune -af` (see "Disk pressure" above).
---
## Local-dev stack — `local/`
A Docker Compose recipe spinning up the runtime services the BFF and ADRs assume — Postgres, Redis, OpenTelemetry Collector — plus optional viewers (pgweb, Jaeger UI) gated behind Compose profiles. Designed to start in a single command on a contributor's WSL2 / Linux / macOS host.
| File | Role |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`local/dev.sh`](local/dev.sh) | Convenience wrapper around `docker compose` — see "Convenience script" below |
| [`local/dev.compose.yml`](local/dev.compose.yml) | Service definitions: postgres, redis, otel-collector, plus pgweb and jaeger behind profiles |
| [`local/.env.example`](local/.env.example) | Credentials + ports template (copy to `.env`, which is git-ignored) |
| [`local/init/postgres/01-init.sql`](local/init/postgres/01-init.sql) | Bootstrap SQL for ADR-0013: audit roles + schema, applied on first boot only |
| [`local/otel-collector.yaml`](local/otel-collector.yaml) | Collector pipeline: OTLP receivers → batch → debug exporter (always) + forward to Jaeger when active |
### First-time setup
```bash
# 1. Configure local secrets (copy template, edit, do not commit).
cp infra/local/.env.example infra/local/.env
$EDITOR infra/local/.env
# Set strong dev values for POSTGRES_PASSWORD and REDIS_PASSWORD
# (defaults in the template are placeholders that the compose
# rejects with `must be set in infra/local/.env` if left as-is).
# 2. Bring up the core stack (postgres + redis + otel-collector).
./infra/local/dev.sh up
# 3. (Optional) Activate viewers when needed:
./infra/local/dev.sh up dbtools # adds pgweb
./infra/local/dev.sh up observability # adds Jaeger UI
./infra/local/dev.sh up all # core + every profile
# 4. Verify health.
./infra/local/dev.sh status
```
### Convenience script — `dev.sh`
[`local/dev.sh`](local/dev.sh) is a thin wrapper around `docker compose -f dev.compose.yml ...` with two reasons to exist:
1. **Hides the Compose-profile gotcha.** `docker compose down` only operates on services whose profile is currently active — anything started under `--profile X` keeps running unless the same flag is on `down`. The script always passes every profile in scope on teardown / status / log commands, so profile-gated services (pgweb, Jaeger) are never accidentally orphaned.
2. **Ergonomic verbs** for the common workflows. `./dev.sh up all`, `./dev.sh stop pgweb`, `./dev.sh logs otel-collector`, etc.
Run `./infra/local/dev.sh help` for the full reference. Cheat-sheet:
| Command | Effect |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `./infra/local/dev.sh up` | Core only (postgres + redis + otel-collector) |
| `./infra/local/dev.sh up all` | Core + every profile |
| `./infra/local/dev.sh up dbtools` | Core + pgweb |
| `./infra/local/dev.sh up observability` | Core + Jaeger |
| `./infra/local/dev.sh down` | Tear down the whole stack (every profile in scope) |
| `./infra/local/dev.sh down -v` | Tear down + wipe named volumes (incl. audit-roles bootstrap) |
| `./infra/local/dev.sh stop pgweb` | Stop one service (containers stay around) |
| `./infra/local/dev.sh status` | `docker compose ps`, with every profile visible |
| `./infra/local/dev.sh logs otel-collector` | Follow logs |
| `./infra/local/dev.sh exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"` | Run a command inside a service |
Anything not matching one of the named verbs is passed through to `docker compose -f dev.compose.yml ...` (with every profile flagged in), so you keep the full Compose surface available — `./dev.sh config`, `./dev.sh top`, `./dev.sh inspect …`, etc.
If you prefer to call `docker compose` directly, every example below shows the raw command alongside the script form.
### Service endpoints (defaults)
| Service | Host port | Purpose |
| ------------------- | --------- | ------------------------------------------------------------------- |
| Postgres | 5432 | DB connection — `postgres://portal:<pwd>@localhost:5432/portal_dev` |
| Redis | 6379 | Sessions, OBO cache (per ADR-0010 / ADR-0014) |
| OTel Collector gRPC | 4317 | `OTEL_EXPORTER_OTLP_ENDPOINT` for the BFF and the SPA |
| OTel Collector HTTP | 4318 | OTLP/HTTP variant |
| pgweb (profile) | 8081 | http://localhost:8081 — Postgres GUI |
| Jaeger UI (profile) | 16686 | http://localhost:16686 — trace explorer |
All ports are overridable via `.env` if the host machine has conflicts.
### Operational tips
- **Persistence** — state lives in named Docker volumes (`apf-portal-postgres-data`, `apf-portal-redis-data`). Survives `docker compose down`. Use `docker compose -f dev.compose.yml down -v` to wipe (also wipes the audit-roles bootstrap, which re-runs on the next fresh boot).
- **Profile symmetry** — `dev.sh down` (and `status`, `logs`, …) always include every profile in scope, so profile-gated services are caught. If you bypass the script and call `docker compose down` directly, you must pass the same `--profile` flags as on `up`, otherwise pgweb and Jaeger keep running silently. Either pass them again, or `export COMPOSE_PROFILES=dbtools,observability` in your shell or `infra/local/.env`.
- **Bootstrap re-run** — the SQL in `local/init/postgres/` only runs on a **fresh** Postgres data volume. To replay after editing the file, `down -v` (loses all dev data) or run the SQL manually with `docker compose exec postgres psql -U portal -d portal_dev -f /docker-entrypoint-initdb.d/01-init.sql`.
- **Logs** — `docker compose -f dev.compose.yml logs -f <service>` to follow a single service. `otel-collector` is the loudest — its `debug` exporter prints every span / metric / log it receives.
- **Image upgrades** — same policy as the runner image (deliberate, not via `:latest`). Renovate's docker-compose manager will surface bumps automatically once the dashboard rule allows them.
### Production parity
This stack is **dev-only**. The corresponding production layout (HA Postgres, Redis Sentinel cluster, OTel Collector with a real backend, secret manager) lives in the future on-prem-infrastructure ADR — see `prod/` placeholder below.
---
## Future infra concerns — placeholders
These are listed here so a contributor knows where to expect related files; they don't exist yet.
| File | Purpose | Triggered by |
| --------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `prod/*` | On-prem deployment manifests (k8s, Compose, or whatever the on-prem infra ADR settles on) | The on-prem infrastructure ADR (phase 3b) |
| `runbooks/*.md` | Operational runbooks (incident response, secret rotation, runner upgrade procedure, …) | First incident, or when ops cadence justifies them |
Reference in New Issue View Git Blame Copy Permalink