docs(development): add observability dev-loop section
ADR-0012 phase 1 + phase 2 are wired (BFF + SPA), so the "Observability dev-loop" placeholder in the roadmap table now has real content to point at. Promote it from §9 (was §8) future-work list to a full §5 walkthrough sitting between "Daily commands" and "Dependency updates". What §5 covers: - Bringing up the observability stack (`./infra/local/dev.sh up observability`) with the endpoint table. - Reading a trace in Jaeger — service-dropdown filter, span attribute keys to look at, the trace-id-as-pivot pattern. - Correlating a trace with the BFF Pino logs via `trace_id` and `request_id` (the latter the per-request UUID nestjs-cls provides). - Reading SPA spans from the browser (DevTools Network + Jaeger UI cross-check). - Common gotchas — observability profile not active, CORS pre-flight on `/v1/traces`, `propagateTraceHeaderCorsUrls` mismatches, NODE_ENV mis-set, EADDRINUSE. - Pointer at the "wired as features land" deltas (CLS keys for session/user/audience, redact list, custom domain spans, prod backend) so a contributor knows what's intentionally not yet there. Renumber the rest of the doc accordingly (5→6 / 6→7 / 7→8 / 8→9) and drop the now-implemented "Observability dev-loop" line from the §9 roadmap table. `docs/README.md` cross-link updated to mention the new section.
This commit is contained in:
Julien Gautier
+100
-19
@@ -229,7 +229,89 @@ pnpm ci:perf # production build + Lighthouse CI against the static-served
|
||||
|
||||
---
|
||||
|
||||
## 5. Dependency updates (Renovate)
|
||||
## 5. Observability dev-loop
|
||||
|
||||
The BFF and the SPA are both wired with OpenTelemetry tracing and structured Pino logging (per [ADR-0012](decisions/0012-observability-pino-opentelemetry.md)). This section is the practical guide to using them while debugging — finding a trace, reading the logs that go with it, untangling a slow request.
|
||||
|
||||
### Bring up the observability stack
|
||||
|
||||
Traces land in the Jaeger UI bundled in `infra/local/dev.compose.yml` under the `observability` profile. The OpenTelemetry Collector runs in the core profile; without Jaeger active, traces still go to the Collector but nothing visualises them — they appear as warnings in `dev.sh logs otel-collector` and are dropped after the buffer fills.
|
||||
|
||||
```bash
|
||||
./infra/local/dev.sh up observability # core stack + Jaeger
|
||||
pnpm nx serve portal-bff # http://localhost:3000
|
||||
pnpm nx serve portal-shell # http://localhost:4200
|
||||
```
|
||||
|
||||
| Endpoint | Purpose |
|
||||
| ---------------------- | --------------------------------------------------------------------------------------- |
|
||||
| http://localhost:16686 | Jaeger UI — search and explore traces |
|
||||
| http://localhost:4318 | OTLP/HTTP receiver on the Collector — both the BFF and the SPA push spans here directly |
|
||||
| (BFF stdout) | Pino logs — JSON in prod, `pino-pretty` colourised one-liners in dev |
|
||||
| (Browser DevTools) | SPA spans — visible in Network as POSTs to `/v1/traces` |
|
||||
|
||||
### Reading a trace in Jaeger
|
||||
|
||||
1. Open http://localhost:16686.
|
||||
2. Pick the **Service** dropdown — `portal-bff` for backend traces, `portal-shell` for SPA traces. After a fetch SPA → BFF, the trace shows up under whichever service initiated the root span (`portal-shell` if the user clicked, `portal-bff` if it was a direct curl).
|
||||
3. **Find Traces** lists recent traces; click one to drill in. Each row in the timeline view is a span. Spans are nested by parent-child: `document_load` → `fetch` → `HTTP GET /api/...` → `prisma.findMany` → `pg.SELECT`.
|
||||
4. Click a span to see its attributes, events, and resource. Useful keys to look at:
|
||||
- `http.method`, `http.status_code`, `http.url` — request shape
|
||||
- `db.statement`, `db.system` — actual SQL run by Prisma
|
||||
- `service.name` / `service.version` — which app emitted the span
|
||||
- `trace_id` (top of the trace) — what to grep in the logs
|
||||
|
||||
### Correlating a trace with the BFF logs
|
||||
|
||||
Every Pino log line emitted by the BFF during a traced request carries a `trace_id` and `span_id` field, injected automatically by `@opentelemetry/instrumentation-pino`. Concretely:
|
||||
|
||||
```bash
|
||||
# 1. Pick a trace_id in the Jaeger URL bar (the long hex string after /trace/).
|
||||
# 2. Grep the BFF stdout — capture from the serve-portal-bff terminal,
|
||||
# or pipe the dev server output to a file.
|
||||
pnpm nx serve portal-bff 2>&1 | tee /tmp/portal-bff.log
|
||||
grep '"trace_id":"<the hex>"' /tmp/portal-bff.log
|
||||
```
|
||||
|
||||
In production the same field exists on every JSON log line; the log aggregator's UI can filter by it. Same trace, same `trace_id`, all the lines.
|
||||
|
||||
The `request_id` field (per-request UUID stored in `nestjs-cls`) is also on every line — useful for filtering when you don't have a trace handy yet.
|
||||
|
||||
### Reading SPA spans from the browser
|
||||
|
||||
The SPA pushes its spans straight to the Collector via OTLP/HTTP. Two ways to inspect what gets sent:
|
||||
|
||||
- **DevTools → Network tab**: filter on `localhost:4318/v1/traces`. Each entry is a batch of spans encoded as JSON. Useful for diagnosing "why doesn't my span show up" — if the POST never happens, the span never left.
|
||||
- **Jaeger UI → service `portal-shell`**: same data, after the Collector forwards it. End-to-end view.
|
||||
|
||||
The auto-instrumentations in v1 cover:
|
||||
|
||||
- `document_load` — one span per page load, with sub-spans for DNS, TCP, request/response, DOM
|
||||
- `fetch` — one span per outgoing fetch, with W3C `traceparent` propagated to the BFF (so the BFF picks up the same trace id and emits child spans)
|
||||
- `user_interaction` — clicks, keypresses, submits
|
||||
|
||||
### Common gotchas
|
||||
|
||||
- **No spans appear in Jaeger but the Collector logs receive them**. The `observability` profile probably wasn't active when you brought the stack up. `./infra/local/dev.sh status` should list `apf-portal-jaeger`. If not, `./infra/local/dev.sh up observability`.
|
||||
- **CORS errors in DevTools when the SPA POSTs to `/v1/traces`**. The Collector's `cors.allowed_origins` in `infra/local/otel-collector.yaml` must include the SPA dev origin (`http://localhost:4200`). It does by default; if you're serving on a non-default port, add it there and re-up the Collector.
|
||||
- **`document_load` span appears but `fetch` spans do not propagate to the BFF as children**. Two failure modes:
|
||||
- The BFF's CORS allowlist for `traceparent` / `tracestate` is missing — see `apps/portal-bff/src/main.ts` `enableCors`.
|
||||
- The fetch URL doesn't match `propagateTraceHeaderCorsUrls` in `apps/portal-shell/src/observability/tracing.ts`. The default regex matches `localhost:3000`; non-default BFF ports need to be added.
|
||||
- **Logs come out raw JSON in dev instead of `pino-pretty`**. `NODE_ENV` is set to `production` somewhere upstream (Nx's serve target normally exports `development`). Check `process.env.NODE_ENV` from inside the BFF.
|
||||
- **`nx serve portal-bff` errors with `EADDRINUSE :::3000`**. A previous BFF process didn't shut down cleanly. `pkill -f portal-bff` or `ss -ltnp | grep :3000` + `kill <PID>`.
|
||||
|
||||
### What's not in v1
|
||||
|
||||
A few things ADR-0012 sketches that aren't wired yet, in case you go looking:
|
||||
|
||||
- CLS keys `session_id`, `user_id_hash`, `audience` — placeholders only; populated by guards/interceptors when ADR-0009 (auth) and ADR-0010 (sessions) land.
|
||||
- Pino `redact` list for PII — wired with the first DTO carrying redactable data.
|
||||
- Custom domain spans (`tracer.startActiveSpan('user.create', …)`) — added per service method as the corresponding feature ships. On the SPA side, custom spans across `await` need explicit `context.with(...)` since the workspace is zoneless.
|
||||
- Production OTLP backend (Tempo / Loki / similar) — phase 3b on-prem ADR. The current setup is dev-only; the Collector's `debug` exporter dumps everything to stdout, useful for local but not what gets pointed at a real backend.
|
||||
|
||||
---
|
||||
|
||||
## 6. Dependency updates (Renovate)
|
||||
|
||||
[Renovate](https://docs.renovatebot.com/) runs as a scheduled workflow ([`.gitea/workflows/renovate.yml`](../.gitea/workflows/renovate.yml)) and opens PRs against `main` for dependency updates. Daily at 03:00 UTC, plus on-demand via `workflow_dispatch`.
|
||||
|
||||
@@ -265,7 +347,7 @@ Repo → Actions → "Renovate" workflow → Run workflow. Useful when you've ju
|
||||
|
||||
---
|
||||
|
||||
## 6. Conventional commit cycle
|
||||
## 7. Conventional commit cycle
|
||||
|
||||
1. Branch from `main` with a short slug:
|
||||
|
||||
@@ -344,7 +426,7 @@ The template guides without enforcing — sections can be left blank when irrele
|
||||
|
||||
---
|
||||
|
||||
## 7. Where to look
|
||||
## 8. Where to look
|
||||
|
||||
| Question | Doc |
|
||||
| ------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
@@ -356,24 +438,23 @@ The template guides without enforcing — sections can be left blank when irrele
|
||||
|
||||
---
|
||||
|
||||
## 8. Sections to come — roadmap by phase
|
||||
## 9. Sections to come — roadmap by phase
|
||||
|
||||
This doc starts as a phase-1 + cross-cutting reference. As features for later phases land, the corresponding sections below are filled in directly. Each entry is mapped to the ADR / implementation work that unlocks it, so a contributor can see when each section becomes real and what triggers it.
|
||||
|
||||
When a section grows beyond a short subsection, it is extracted to its own file under `docs/development/`. Per the documentation convention (see [README.md](README.md)), we group into a folder once we have at least three related files; this doc is then re-organised into an index pointing at the extracted files. Until then, all sections live here.
|
||||
|
||||
| Future section | Phase | Triggered by |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Auth dev-loop** — Microsoft 365 Developer tenant configuration, MSAL Node connection, OIDC code-flow walkthrough, switching between dev and prod-like tenants. | 2 | Auth flow code lands ([ADR-0009](decisions/0009-auth-flow-oidc-pkce-msal-node.md)) once the dev tenant is provisioned by IT. |
|
||||
| **Session inspection** — reading the Redis session store in dev, decrypting the AES-GCM `tokens` blob with the dev key, force-logout patterns. | 2 | Sessions module lands ([ADR-0010](decisions/0010-session-management-redis.md)). |
|
||||
| **MFA step-up debugging** — triggering claims-challenge flows, verifying `mfaVerifiedAt` freshness, testing the SPA HTTP interceptor that handles 401 + claims challenge. | 2 | First `@RequireMfa()` route lands ([ADR-0011](decisions/0011-mfa-enforcement-entra-conditional-access.md)). |
|
||||
| **Observability dev-loop** — viewing traces in the Jaeger UI provisioned by `infra/local/dev.compose.yml --profile observability`, reading Pino logs with `pino-pretty`, correlating front spans to BFF spans via `traceparent`. | 2 | OTel SDK setup lands ([ADR-0012](decisions/0012-observability-pino-opentelemetry.md)). |
|
||||
| **Audit-log inspection workflow** — querying `audit.events` as `audit_reader`, joining with app logs by `trace_id`, validating the append-only role grants in dev. | 2 | Audit module lands ([ADR-0013](decisions/0013-audit-trail-separated-postgres-append-only.md)). |
|
||||
| **Downstream API integration recipe** — adding a new `DownstreamApiConfig`, choosing the auth strategy (OBO vs service+assertion), wiring resilience policies, testing with a mocked downstream. | 2 | First downstream client lands ([ADR-0014](decisions/0014-downstream-api-access-obo-pattern.md)). |
|
||||
| **Component patterns library** — the in-house, spartan-style components (Angular CDK + Tailwind) as they ship, with a11y notes per component (keyboard model, ARIA, screen-reader expectations). | 5b suite | First non-placeholder component in `libs/shared/ui/`. |
|
||||
| **a11y testing workflow** — running axe-core via Playwright locally, screen-reader testing notes (NVDA / VoiceOver / TalkBack), the APF user-panel cadence and how to triage findings. | 3a | First Playwright e2e suite touching real screens ([ADR-0016](decisions/0016-accessibility-baseline-wcag-aa-targeted-aaa.md)). |
|
||||
| **Performance debugging** — running Lighthouse CI locally with full config, reading the HTML reports, using `source-map-explorer` to investigate bundle bloat, interpreting BFF p95/p99 from OTel. | 3a | Lighthouse already wired in CI ([ADR-0017](decisions/0017-performance-budgets-lighthouse-ci.md)); section grows when first real route is added to the critical-routes list. |
|
||||
| **Debugging tips** — Angular DevTools, NestJS inspector, Prisma query log, OTel trace navigation, common gotchas. | cross | Accumulates organically as the team encounters them. |
|
||||
| **Release workflow** — tag-driven release, what `release.yml` does, version bumping, changelog generation from Conventional Commits. | 3b | On-prem infrastructure ADR + populated `release.yml`. |
|
||||
| **GitLab migration runbook** — when the org migrates Gitea → GitLab, how the workflows are ported, which level-2 sections of [ADR-0015](decisions/0015-cicd-gitea-actions.md) get superseded. | future | GitLab migration ADR (6–18 months horizon). |
|
||||
| **Architecture overview diagrams** — high-level component diagrams, data-flow diagrams, trust boundaries (for security review). | cross | First major architecture review or onboarding cohort ≥ 3 contributors. |
|
||||
| Future section | Phase | Triggered by |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Auth dev-loop** — Microsoft 365 Developer tenant configuration, MSAL Node connection, OIDC code-flow walkthrough, switching between dev and prod-like tenants. | 2 | Auth flow code lands ([ADR-0009](decisions/0009-auth-flow-oidc-pkce-msal-node.md)) once the dev tenant is provisioned by IT. |
|
||||
| **Session inspection** — reading the Redis session store in dev, decrypting the AES-GCM `tokens` blob with the dev key, force-logout patterns. | 2 | Sessions module lands ([ADR-0010](decisions/0010-session-management-redis.md)). |
|
||||
| **MFA step-up debugging** — triggering claims-challenge flows, verifying `mfaVerifiedAt` freshness, testing the SPA HTTP interceptor that handles 401 + claims challenge. | 2 | First `@RequireMfa()` route lands ([ADR-0011](decisions/0011-mfa-enforcement-entra-conditional-access.md)). |
|
||||
| **Audit-log inspection workflow** — querying `audit.events` as `audit_reader`, joining with app logs by `trace_id`, validating the append-only role grants in dev. | 2 | Audit module lands ([ADR-0013](decisions/0013-audit-trail-separated-postgres-append-only.md)). |
|
||||
| **Downstream API integration recipe** — adding a new `DownstreamApiConfig`, choosing the auth strategy (OBO vs service+assertion), wiring resilience policies, testing with a mocked downstream. | 2 | First downstream client lands ([ADR-0014](decisions/0014-downstream-api-access-obo-pattern.md)). |
|
||||
| **Component patterns library** — the in-house, spartan-style components (Angular CDK + Tailwind) as they ship, with a11y notes per component (keyboard model, ARIA, screen-reader expectations). | 5b suite | First non-placeholder component in `libs/shared/ui/`. |
|
||||
| **a11y testing workflow** — running axe-core via Playwright locally, screen-reader testing notes (NVDA / VoiceOver / TalkBack), the APF user-panel cadence and how to triage findings. | 3a | First Playwright e2e suite touching real screens ([ADR-0016](decisions/0016-accessibility-baseline-wcag-aa-targeted-aaa.md)). |
|
||||
| **Performance debugging** — running Lighthouse CI locally with full config, reading the HTML reports, using `source-map-explorer` to investigate bundle bloat, interpreting BFF p95/p99 from OTel. | 3a | Lighthouse already wired in CI ([ADR-0017](decisions/0017-performance-budgets-lighthouse-ci.md)); section grows when first real route is added to the critical-routes list. |
|
||||
| **Debugging tips** — Angular DevTools, NestJS inspector, Prisma query log, OTel trace navigation, common gotchas. | cross | Accumulates organically as the team encounters them. |
|
||||
| **Release workflow** — tag-driven release, what `release.yml` does, version bumping, changelog generation from Conventional Commits. | 3b | On-prem infrastructure ADR + populated `release.yml`. |
|
||||
| **GitLab migration runbook** — when the org migrates Gitea → GitLab, how the workflows are ported, which level-2 sections of [ADR-0015](decisions/0015-cicd-gitea-actions.md) get superseded. | future | GitLab migration ADR (6–18 months horizon). |
|
||||
| **Architecture overview diagrams** — high-level component diagrams, data-flow diagrams, trust boundaries (for security review). | cross | First major architecture review or onboarding cohort ≥ 3 contributors. |
|
||||
|
||||
Reference in New Issue
Block a user