From 60659225209079d654a128e6cb1b351530302b58 Mon Sep 17 00:00:00 2001 From: Julien Gautier Date: Sun, 10 May 2026 02:10:53 +0200 Subject: [PATCH] docs(development): add observability dev-loop section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-0012 phase 1 + phase 2 are wired (BFF + SPA), so the "Observability dev-loop" placeholder in the roadmap table now has real content to point at. Promote it from §9 (was §8) future-work list to a full §5 walkthrough sitting between "Daily commands" and "Dependency updates". What §5 covers: - Bringing up the observability stack (`./infra/local/dev.sh up observability`) with the endpoint table. - Reading a trace in Jaeger — service-dropdown filter, span attribute keys to look at, the trace-id-as-pivot pattern. - Correlating a trace with the BFF Pino logs via `trace_id` and `request_id` (the latter the per-request UUID nestjs-cls provides). - Reading SPA spans from the browser (DevTools Network + Jaeger UI cross-check). - Common gotchas — observability profile not active, CORS pre-flight on `/v1/traces`, `propagateTraceHeaderCorsUrls` mismatches, NODE_ENV mis-set, EADDRINUSE. - Pointer at the "wired as features land" deltas (CLS keys for session/user/audience, redact list, custom domain spans, prod backend) so a contributor knows what's intentionally not yet there. Renumber the rest of the doc accordingly (5→6 / 6→7 / 7→8 / 8→9) and drop the now-implemented "Observability dev-loop" line from the §9 roadmap table. `docs/README.md` cross-link updated to mention the new section. --- docs/README.md | 2 +- docs/development.md | 119 +++++++++++++++++++++++++++++++++++++------- 2 files changed, 101 insertions(+), 20 deletions(-) diff --git a/docs/README.md b/docs/README.md index b5f0a14..5120c81 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,7 +13,7 @@ This is the entry point to all project documentation. It is maintained automatic ### Daily development -- [development.md](development.md) — repo layout, prerequisites, initial setup, daily commands, dependency updates (Renovate), conventional commit cycle. Day-to-day reference for working on the project. +- [development.md](development.md) — repo layout, prerequisites, initial setup, daily commands, observability dev-loop (Jaeger UI, log ↔ trace correlation), dependency updates (Renovate), conventional commit cycle. Day-to-day reference for working on the project. ### Architecture diff --git a/docs/development.md b/docs/development.md index 0de249b..1a0fda1 100644 --- a/docs/development.md +++ b/docs/development.md @@ -229,7 +229,89 @@ pnpm ci:perf # production build + Lighthouse CI against the static-served --- -## 5. Dependency updates (Renovate) +## 5. Observability dev-loop + +The BFF and the SPA are both wired with OpenTelemetry tracing and structured Pino logging (per [ADR-0012](decisions/0012-observability-pino-opentelemetry.md)). This section is the practical guide to using them while debugging — finding a trace, reading the logs that go with it, untangling a slow request. + +### Bring up the observability stack + +Traces land in the Jaeger UI bundled in `infra/local/dev.compose.yml` under the `observability` profile. The OpenTelemetry Collector runs in the core profile; without Jaeger active, traces still go to the Collector but nothing visualises them — they appear as warnings in `dev.sh logs otel-collector` and are dropped after the buffer fills. + +```bash +./infra/local/dev.sh up observability # core stack + Jaeger +pnpm nx serve portal-bff # http://localhost:3000 +pnpm nx serve portal-shell # http://localhost:4200 +``` + +| Endpoint | Purpose | +| ---------------------- | --------------------------------------------------------------------------------------- | +| http://localhost:16686 | Jaeger UI — search and explore traces | +| http://localhost:4318 | OTLP/HTTP receiver on the Collector — both the BFF and the SPA push spans here directly | +| (BFF stdout) | Pino logs — JSON in prod, `pino-pretty` colourised one-liners in dev | +| (Browser DevTools) | SPA spans — visible in Network as POSTs to `/v1/traces` | + +### Reading a trace in Jaeger + +1. Open http://localhost:16686. +2. Pick the **Service** dropdown — `portal-bff` for backend traces, `portal-shell` for SPA traces. After a fetch SPA → BFF, the trace shows up under whichever service initiated the root span (`portal-shell` if the user clicked, `portal-bff` if it was a direct curl). +3. **Find Traces** lists recent traces; click one to drill in. Each row in the timeline view is a span. Spans are nested by parent-child: `document_load` → `fetch` → `HTTP GET /api/...` → `prisma.findMany` → `pg.SELECT`. +4. Click a span to see its attributes, events, and resource. Useful keys to look at: + - `http.method`, `http.status_code`, `http.url` — request shape + - `db.statement`, `db.system` — actual SQL run by Prisma + - `service.name` / `service.version` — which app emitted the span + - `trace_id` (top of the trace) — what to grep in the logs + +### Correlating a trace with the BFF logs + +Every Pino log line emitted by the BFF during a traced request carries a `trace_id` and `span_id` field, injected automatically by `@opentelemetry/instrumentation-pino`. Concretely: + +```bash +# 1. Pick a trace_id in the Jaeger URL bar (the long hex string after /trace/). +# 2. Grep the BFF stdout — capture from the serve-portal-bff terminal, +# or pipe the dev server output to a file. +pnpm nx serve portal-bff 2>&1 | tee /tmp/portal-bff.log +grep '"trace_id":""' /tmp/portal-bff.log +``` + +In production the same field exists on every JSON log line; the log aggregator's UI can filter by it. Same trace, same `trace_id`, all the lines. + +The `request_id` field (per-request UUID stored in `nestjs-cls`) is also on every line — useful for filtering when you don't have a trace handy yet. + +### Reading SPA spans from the browser + +The SPA pushes its spans straight to the Collector via OTLP/HTTP. Two ways to inspect what gets sent: + +- **DevTools → Network tab**: filter on `localhost:4318/v1/traces`. Each entry is a batch of spans encoded as JSON. Useful for diagnosing "why doesn't my span show up" — if the POST never happens, the span never left. +- **Jaeger UI → service `portal-shell`**: same data, after the Collector forwards it. End-to-end view. + +The auto-instrumentations in v1 cover: + +- `document_load` — one span per page load, with sub-spans for DNS, TCP, request/response, DOM +- `fetch` — one span per outgoing fetch, with W3C `traceparent` propagated to the BFF (so the BFF picks up the same trace id and emits child spans) +- `user_interaction` — clicks, keypresses, submits + +### Common gotchas + +- **No spans appear in Jaeger but the Collector logs receive them**. The `observability` profile probably wasn't active when you brought the stack up. `./infra/local/dev.sh status` should list `apf-portal-jaeger`. If not, `./infra/local/dev.sh up observability`. +- **CORS errors in DevTools when the SPA POSTs to `/v1/traces`**. The Collector's `cors.allowed_origins` in `infra/local/otel-collector.yaml` must include the SPA dev origin (`http://localhost:4200`). It does by default; if you're serving on a non-default port, add it there and re-up the Collector. +- **`document_load` span appears but `fetch` spans do not propagate to the BFF as children**. Two failure modes: + - The BFF's CORS allowlist for `traceparent` / `tracestate` is missing — see `apps/portal-bff/src/main.ts` `enableCors`. + - The fetch URL doesn't match `propagateTraceHeaderCorsUrls` in `apps/portal-shell/src/observability/tracing.ts`. The default regex matches `localhost:3000`; non-default BFF ports need to be added. +- **Logs come out raw JSON in dev instead of `pino-pretty`**. `NODE_ENV` is set to `production` somewhere upstream (Nx's serve target normally exports `development`). Check `process.env.NODE_ENV` from inside the BFF. +- **`nx serve portal-bff` errors with `EADDRINUSE :::3000`**. A previous BFF process didn't shut down cleanly. `pkill -f portal-bff` or `ss -ltnp | grep :3000` + `kill `. + +### What's not in v1 + +A few things ADR-0012 sketches that aren't wired yet, in case you go looking: + +- CLS keys `session_id`, `user_id_hash`, `audience` — placeholders only; populated by guards/interceptors when ADR-0009 (auth) and ADR-0010 (sessions) land. +- Pino `redact` list for PII — wired with the first DTO carrying redactable data. +- Custom domain spans (`tracer.startActiveSpan('user.create', …)`) — added per service method as the corresponding feature ships. On the SPA side, custom spans across `await` need explicit `context.with(...)` since the workspace is zoneless. +- Production OTLP backend (Tempo / Loki / similar) — phase 3b on-prem ADR. The current setup is dev-only; the Collector's `debug` exporter dumps everything to stdout, useful for local but not what gets pointed at a real backend. + +--- + +## 6. Dependency updates (Renovate) [Renovate](https://docs.renovatebot.com/) runs as a scheduled workflow ([`.gitea/workflows/renovate.yml`](../.gitea/workflows/renovate.yml)) and opens PRs against `main` for dependency updates. Daily at 03:00 UTC, plus on-demand via `workflow_dispatch`. @@ -265,7 +347,7 @@ Repo → Actions → "Renovate" workflow → Run workflow. Useful when you've ju --- -## 6. Conventional commit cycle +## 7. Conventional commit cycle 1. Branch from `main` with a short slug: @@ -344,7 +426,7 @@ The template guides without enforcing — sections can be left blank when irrele --- -## 7. Where to look +## 8. Where to look | Question | Doc | | ------------------------------------------- | ---------------------------------------------------------------------------- | @@ -356,24 +438,23 @@ The template guides without enforcing — sections can be left blank when irrele --- -## 8. Sections to come — roadmap by phase +## 9. Sections to come — roadmap by phase This doc starts as a phase-1 + cross-cutting reference. As features for later phases land, the corresponding sections below are filled in directly. Each entry is mapped to the ADR / implementation work that unlocks it, so a contributor can see when each section becomes real and what triggers it. When a section grows beyond a short subsection, it is extracted to its own file under `docs/development/`. Per the documentation convention (see [README.md](README.md)), we group into a folder once we have at least three related files; this doc is then re-organised into an index pointing at the extracted files. Until then, all sections live here. -| Future section | Phase | Triggered by | -| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Auth dev-loop** — Microsoft 365 Developer tenant configuration, MSAL Node connection, OIDC code-flow walkthrough, switching between dev and prod-like tenants. | 2 | Auth flow code lands ([ADR-0009](decisions/0009-auth-flow-oidc-pkce-msal-node.md)) once the dev tenant is provisioned by IT. | -| **Session inspection** — reading the Redis session store in dev, decrypting the AES-GCM `tokens` blob with the dev key, force-logout patterns. | 2 | Sessions module lands ([ADR-0010](decisions/0010-session-management-redis.md)). | -| **MFA step-up debugging** — triggering claims-challenge flows, verifying `mfaVerifiedAt` freshness, testing the SPA HTTP interceptor that handles 401 + claims challenge. | 2 | First `@RequireMfa()` route lands ([ADR-0011](decisions/0011-mfa-enforcement-entra-conditional-access.md)). | -| **Observability dev-loop** — viewing traces in the Jaeger UI provisioned by `infra/local/dev.compose.yml --profile observability`, reading Pino logs with `pino-pretty`, correlating front spans to BFF spans via `traceparent`. | 2 | OTel SDK setup lands ([ADR-0012](decisions/0012-observability-pino-opentelemetry.md)). | -| **Audit-log inspection workflow** — querying `audit.events` as `audit_reader`, joining with app logs by `trace_id`, validating the append-only role grants in dev. | 2 | Audit module lands ([ADR-0013](decisions/0013-audit-trail-separated-postgres-append-only.md)). | -| **Downstream API integration recipe** — adding a new `DownstreamApiConfig`, choosing the auth strategy (OBO vs service+assertion), wiring resilience policies, testing with a mocked downstream. | 2 | First downstream client lands ([ADR-0014](decisions/0014-downstream-api-access-obo-pattern.md)). | -| **Component patterns library** — the in-house, spartan-style components (Angular CDK + Tailwind) as they ship, with a11y notes per component (keyboard model, ARIA, screen-reader expectations). | 5b suite | First non-placeholder component in `libs/shared/ui/`. | -| **a11y testing workflow** — running axe-core via Playwright locally, screen-reader testing notes (NVDA / VoiceOver / TalkBack), the APF user-panel cadence and how to triage findings. | 3a | First Playwright e2e suite touching real screens ([ADR-0016](decisions/0016-accessibility-baseline-wcag-aa-targeted-aaa.md)). | -| **Performance debugging** — running Lighthouse CI locally with full config, reading the HTML reports, using `source-map-explorer` to investigate bundle bloat, interpreting BFF p95/p99 from OTel. | 3a | Lighthouse already wired in CI ([ADR-0017](decisions/0017-performance-budgets-lighthouse-ci.md)); section grows when first real route is added to the critical-routes list. | -| **Debugging tips** — Angular DevTools, NestJS inspector, Prisma query log, OTel trace navigation, common gotchas. | cross | Accumulates organically as the team encounters them. | -| **Release workflow** — tag-driven release, what `release.yml` does, version bumping, changelog generation from Conventional Commits. | 3b | On-prem infrastructure ADR + populated `release.yml`. | -| **GitLab migration runbook** — when the org migrates Gitea → GitLab, how the workflows are ported, which level-2 sections of [ADR-0015](decisions/0015-cicd-gitea-actions.md) get superseded. | future | GitLab migration ADR (6–18 months horizon). | -| **Architecture overview diagrams** — high-level component diagrams, data-flow diagrams, trust boundaries (for security review). | cross | First major architecture review or onboarding cohort ≥ 3 contributors. | +| Future section | Phase | Triggered by | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Auth dev-loop** — Microsoft 365 Developer tenant configuration, MSAL Node connection, OIDC code-flow walkthrough, switching between dev and prod-like tenants. | 2 | Auth flow code lands ([ADR-0009](decisions/0009-auth-flow-oidc-pkce-msal-node.md)) once the dev tenant is provisioned by IT. | +| **Session inspection** — reading the Redis session store in dev, decrypting the AES-GCM `tokens` blob with the dev key, force-logout patterns. | 2 | Sessions module lands ([ADR-0010](decisions/0010-session-management-redis.md)). | +| **MFA step-up debugging** — triggering claims-challenge flows, verifying `mfaVerifiedAt` freshness, testing the SPA HTTP interceptor that handles 401 + claims challenge. | 2 | First `@RequireMfa()` route lands ([ADR-0011](decisions/0011-mfa-enforcement-entra-conditional-access.md)). | +| **Audit-log inspection workflow** — querying `audit.events` as `audit_reader`, joining with app logs by `trace_id`, validating the append-only role grants in dev. | 2 | Audit module lands ([ADR-0013](decisions/0013-audit-trail-separated-postgres-append-only.md)). | +| **Downstream API integration recipe** — adding a new `DownstreamApiConfig`, choosing the auth strategy (OBO vs service+assertion), wiring resilience policies, testing with a mocked downstream. | 2 | First downstream client lands ([ADR-0014](decisions/0014-downstream-api-access-obo-pattern.md)). | +| **Component patterns library** — the in-house, spartan-style components (Angular CDK + Tailwind) as they ship, with a11y notes per component (keyboard model, ARIA, screen-reader expectations). | 5b suite | First non-placeholder component in `libs/shared/ui/`. | +| **a11y testing workflow** — running axe-core via Playwright locally, screen-reader testing notes (NVDA / VoiceOver / TalkBack), the APF user-panel cadence and how to triage findings. | 3a | First Playwright e2e suite touching real screens ([ADR-0016](decisions/0016-accessibility-baseline-wcag-aa-targeted-aaa.md)). | +| **Performance debugging** — running Lighthouse CI locally with full config, reading the HTML reports, using `source-map-explorer` to investigate bundle bloat, interpreting BFF p95/p99 from OTel. | 3a | Lighthouse already wired in CI ([ADR-0017](decisions/0017-performance-budgets-lighthouse-ci.md)); section grows when first real route is added to the critical-routes list. | +| **Debugging tips** — Angular DevTools, NestJS inspector, Prisma query log, OTel trace navigation, common gotchas. | cross | Accumulates organically as the team encounters them. | +| **Release workflow** — tag-driven release, what `release.yml` does, version bumping, changelog generation from Conventional Commits. | 3b | On-prem infrastructure ADR + populated `release.yml`. | +| **GitLab migration runbook** — when the org migrates Gitea → GitLab, how the workflows are ported, which level-2 sections of [ADR-0015](decisions/0015-cicd-gitea-actions.md) get superseded. | future | GitLab migration ADR (6–18 months horizon). | +| **Architecture overview diagrams** — high-level component diagrams, data-flow diagrams, trust boundaries (for security review). | cross | First major architecture review or onboarding cohort ≥ 3 contributors. |