apf_portal/docs/decisions/0013-audit-trail-separated-postgres-append-only.md

---
status: accepted
date: 2026-04-29
last-updated: 2026-05-13
decision-makers: R&D Lead
tags: [security, observability, data]
---

# Audit trail — separated append-only Postgres schema, decoupled from app logs

## Context and Problem Statement

[ADR-0012](0012-observability-pino-opentelemetry.md) fixed the application observability stack: structured logs and traces, all carrying the same `trace_id`, optimised for debugging and operations. Application logs are noisy, broadly accessible, and have a relatively short retention window — appropriate for ops, inappropriate for security forensics or compliance evidence.

Audit logs serve a different purpose: they record _security-relevant_ events authoritatively, must survive years, must not be alterable by anyone (including operators), and must be queryable by auditors with _different_ access controls than developers. Mixing the two streams is the recurring failure mode that turns an audit log into noise — or worse, lets a developer accidentally `UPDATE` a row.

We need a fixed answer to: which events are recorded, where they are stored, who can read them, who can write them, who can never delete or modify them, how long they are kept, and how they relate to the application observability stream.

## Decision Drivers

- Compliance posture: an audit log that can be tampered with is no audit log.
- Strict separation of concern from application logs — different sink, different access, different retention, different volume profile.
- Forensic queryability: an investigator must be able to reconstruct the timeline of any user's interactions with the system.
- Cross-reference with the app observability stream by `trace_id` so an audit event in isolation can be enriched with the surrounding app log.
- Avoid premature complexity: full cryptographic chaining and WORM storage are not required to start, but the path to them must remain open.
- Performance: per-event INSERT must not be in the critical latency path of unrelated operations.

## Considered Options

### Sink

- **Dedicated `audit` schema in the same PostgreSQL instance as business data.** (Chosen.)
- Dedicated PostgreSQL instance for audit only.
- Same Pino logger as app logs, with a different category.
- Push to a SIEM (Splunk, Wazuh, Graylog, etc.).
- Append-only file with rotation.

### Writer

- **Direct Prisma INSERT, in the same transaction as the business action when applicable.** (Chosen.)
- Background queue (Redis Streams, RabbitMQ, etc.) decoupled from the request path.
- Pino transport extension.

### Tamper evidence

- **Append-only at the database role level (REVOKE UPDATE / DELETE / TRUNCATE on the writer and on every other role except a dedicated archiver).** (Chosen for v1.)
- Cryptographic chaining (each row carries a hash linking it to the previous row) — deferred unless compliance demands.
- WORM-capable storage at the infrastructure layer — deferred to future infrastructure ADR if needed.

### Failure semantics

- **Audit failures are blocking — if the audit INSERT fails for an in-scope event, the operation fails.** (Chosen.)
- Best-effort — audit failures are logged but the operation succeeds.

### Retention

- **365 days default, override via env per environment.** (Chosen.)
- Indefinite (until a cleanup is requested manually).
- Aligned with a specific regulation (HDS, PCI, ISO 27001 — to be revisited if and when an applicable framework is identified).

## Decision Outcome

**Sink.** A dedicated **`audit` PostgreSQL schema** in the same database instance as the business data ([ADR-0006](0006-persistence-postgresql-prisma.md)). This requires the Prisma `multiSchema` preview feature, which is enabled in the BFF's schema. The schema is owned by a dedicated `audit_owner` role; only `audit_writer` can `INSERT`, only `audit_reader` can `SELECT`, only `audit_archiver` can `DELETE` rows older than the retention threshold, and `UPDATE` / `TRUNCATE` is granted to no role at all.

**Schema (Prisma model):**

```prisma
model AuditEvent {
  id            String        @id @default(uuid()) @db.Uuid
  occurredAt    DateTime      @default(now()) @map("occurred_at") @db.Timestamptz
  eventType     String        @map("event_type")
  actorIdHash   String?       @map("actor_id_hash")          // matches user_id_hash from ADR-0012
  actorAudience Audience?     @map("actor_audience")          // ADR-0008
  sessionId     String?       @map("session_id")              // ADR-0010
  traceId       String?       @map("trace_id")                // ADR-0012 — cross-reference key
  sourceIp      String?       @map("source_ip") @db.Inet
  userAgent     String?       @map("user_agent")
  outcome       AuditOutcome
  details       Json          @default("{}")

  @@map("events")
  @@schema("audit")
  @@index([eventType])
  @@index([actorIdHash])
  @@index([occurredAt])
  @@index([traceId])
}

enum AuditOutcome {
  success
  failure
}
```

`actor_id_hash` uses the **same salt and hash function** as `user_id_hash` in [ADR-0012](0012-observability-pino-opentelemetry.md), so an investigator can join an audit row with app log lines without any per-investigation re-hashing.

**Writer.** A NestJS `AuditService` exposes a single typed method per event family. Inside a request, the service uses the same Prisma transaction as the business action where applicable; outside a request (background expirations, scheduled jobs), it uses a fresh transaction. Writer connection runs under the `audit_writer` role and has only `INSERT` on `audit.events` — any attempt to `UPDATE` or `DELETE` is rejected by Postgres regardless of code intent.

> **Implementation trap — Prisma ORM cannot be used for the write.** Prisma's `tx.auditEvent.create(...)` issues `INSERT … RETURNING *` to hydrate the entity it returns. Postgres requires the `SELECT` privilege on every column listed in `RETURNING`, and `audit_writer` has `INSERT` only by design — there is no `SELECT` grant on the writer role. The ORM path therefore fails at runtime with `PostgresError 42501 / "permission denied for table events"`, an error whose message mentions neither `SELECT` nor `RETURNING`. The write path uses **parameterised `$executeRawUnsafe`** with no `RETURNING` clause; the schema-level `id UUID @default(uuid())` from Prisma is replaced server-side with `gen_random_uuid()` in the SQL. This is a deliberate consequence of the role-separation contract and is pinned by a spec test. The alternative — granting `SELECT` on `audit.events` to `audit_writer` — would collapse the writer / reader role separation that the rest of this ADR rests on, so we go the other way.

**Events emitted in v1.**

| `event_type`                   | When                                                                                                                       | `outcome` |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | --------- |
| `auth.sign_in`                 | OIDC callback creates a session                                                                                            | `success` |
| `auth.sign_in.failed`          | OIDC callback rejects (state mismatch, signature, `iss` not allowlisted, MFA assertion missing, audience misclassified, …) | `failure` |
| `auth.sign_out`                | `POST /auth/logout` invalidates a session                                                                                  | `success` |
| `auth.session.expired`         | idle or absolute timeout fires (ADR-0010)                                                                                  | `success` |
| `auth.session.revoked`         | session is force-deleted (admin or self)                                                                                   | `success` |
| `auth.token.validation.failed` | a token presented mid-session fails validation (signature, `iss`, audience, claim mismatch)                                | `failure` |
| `auth.mfa.assertion.failed`    | the BFF rejects a session for missing or weak `amr` (ADR-0011)                                                             | `failure` |
| `authz.deny`                   | a guard rejects an authenticated request because the user's `audience`/claims don't authorise the action                   | `failure` |
| `admin.access_denied`          | `AdminRoleGuard` rejects an authenticated non-admin on `/api/admin/*`                                                      | `denied`  |
| `admin.audit.query`            | admin called `GET /api/admin/audit` (paginated rows). Payload carries `filters` + `resultCount`                            | `success` |
| `admin.audit.stats.query`      | admin called `GET /api/admin/audit/stats` (aggregations). Payload carries `filters` + `total`                              | `success` |
| `admin.users.query`            | admin called `GET /api/admin/users` (user-directory listing). Payload carries `filters` + `resultCount`                    | `success` |

The `details` JSONB field carries event-specific information (e.g. expected vs received `iss`, denied route, claim names involved). Sensitive material (full tokens, claims that should never leave the BFF) is _never_ placed in `details` — the same redaction posture as app logs applies, enforced by typed event payloads at the writer's boundary.

Hooks for **admin actions** and **sensitive data access** are designed-in: the writer accepts those event families today, but no v1 caller emits them. They are exercised by unit tests so the path stays alive.

**Decoupling and cross-reference.** Audit events do not flow through the Pino app logger. They flow through `AuditService` directly to Postgres. Each event carries the `trace_id` from the request-scoped CLS (ADR-0012); an investigator joining audit and app log streams on `trace_id` can reconstruct the full timeline of a request. The two streams have:

| Aspect           | App logs                                                   | Audit log                                      |
| ---------------- | ---------------------------------------------------------- | ---------------------------------------------- |
| Sink             | stdout → collector → on-prem backend                       | PostgreSQL `audit.events`                      |
| Volume           | high (every request, every internal step)                  | low (security-relevant events only)            |
| Retention        | short (operational, weeks–months)                          | long (≥ 365 days, env-driven)                  |
| Access           | broad (devs, ops)                                          | restricted (`audit_reader` role; security/SOC) |
| Mutability       | already gone if rotated; otherwise immutable by convention | immutable by Postgres role grants              |
| Trace identifier | yes                                                        | yes (same value)                               |

**Failure semantics.** Audit writes are **blocking**. If the INSERT fails (DB unreachable, role misconfigured, schema drift), the in-flight operation fails: the user gets a 503, the BFF emits a critical app log line, and on-prem alerting fires. This is the security-positive default ("if you cannot audit, you cannot act"). Trade-off acknowledged: the audit DB becomes part of the trust path, and an audit DB outage degrades the application. The mitigation is HA Postgres in prod (covered by the future infrastructure ADR), not best-effort writes.

**Retention.** Default 365 days, overridable per environment via `AUDIT_RETENTION_DAYS`. A daily scheduled job (`pg_cron` extension if available, otherwise a Nest scheduled task running under `audit_archiver`) deletes rows whose `occurred_at` is older than the threshold. The job is idempotent and emits its own audit event (`audit.retention.purge`) with the count of rows removed.

> **Compliance note.** The 365-day default is engineering-prudent, not legally derived. Specific regulations applicable to the host organisation (RGPD/GDPR retention principles, sectoral rules — health, finance — if any) may demand longer or shorter windows, or full archival before deletion. The retention default is a starting point. The legal review is owed by the organisation; this ADR does not pretend to settle it.

**GDPR / right-of-erasure** interactions: audit events are typically retained under a "legal obligation" or "legitimate interest" basis even after a user's right-of-erasure request. The userId itself is already pseudonymised (`actor_id_hash`); the join key to user identity exists in the live DB only and disappears with account deletion, leaving the audit row scientifically pseudonymous. Whether this is legally sufficient is the legal team's call. The current implementation is compatible with both "keep" and "anonymise further" policies (we can null `actor_id_hash` for archived users without violating append-only because _zeroing_ counts as a sanctioned operation under a yet-to-define `audit_redactor` role — designed-in, not implemented in v1).

**Reader endpoints.** Two BFF routes consume `audit.events` via the `audit_reader` role, both under `/api/admin/audit` and guarded by `@RequireAdmin` per [ADR-0020](0020-portal-admin-app.md):

| Route                        | Purpose                                                                                                             | Read pattern                                                                           |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `GET /api/admin/audit`       | Paginated raw rows for the audit-log table viewer                                                                   | `SELECT … LIMIT/OFFSET` capped at 200 rows                                             |
| `GET /api/admin/audit/stats` | Three aggregations (events per day, outcome breakdown, events per day by event-type) for the audit-log "Charts" tab | Three `GROUP BY` queries over the filtered set, Redis-cached for 300 s per filter-hash |

The stats endpoint introduces a 5-minute Redis cache: audit rows are append-only so past aggregations are stable, but new events are continuously inserted — a 5-minute TTL is the chosen compromise. Admins see at most 5-minute-stale aggregations (acceptable for "approximate dashboard" use; not appropriate for "did the last event just land" debugging, which the paginated `GET /api/admin/audit` is for). Cache keys are the SHA-256 of the canonical (sorted-keys) JSON of the filter object, scoped under the `audit:stats:` Redis key prefix. Cache writes are best-effort — a Redis-write failure does not fail the response. Each stats query emits its own audit row (`admin.audit.stats.query`) with `total` (size of the aggregated set) as the deterrent signal, distinct from `admin.audit.query`'s `resultCount`.

**Configuration (env-driven).**

| Variable                      | Purpose                                                                                                 |
| ----------------------------- | ------------------------------------------------------------------------------------------------------- |
| `AUDIT_DATABASE_URL`          | Postgres connection for the BFF as `audit_writer` (separate creds, different from the business DB role) |
| `AUDIT_RETENTION_DAYS`        | default `365`; the BFF refuses to start with a value below `30`                                         |
| `AUDIT_ARCHIVER_DATABASE_URL` | connection for the archiver job as `audit_archiver` (used by the scheduled task only)                   |
| `LOG_USER_ID_SALT`            | shared with ADR-0012 — must be identical to keep the `actor_id_hash` join key consistent                |

### Consequences

- Good, because audit log integrity rests on Postgres role grants — operationally enforced by the database, not by application discipline.
- Good, because the `trace_id` makes audit events instantly cross-referenceable with the rich app log, without coupling the two streams.
- Good, because the dedicated schema and roles let a future SOC team be granted `audit_reader` without giving them anything else.
- Good, because keeping the audit DB on the same instance avoids the operational doubling of running two Postgres clusters; the schema/role separation gives most of the isolation benefits for a fraction of the cost.
- Good, because the retention default is conservative and overridable, so adapting to a stricter regulatory regime is a config change, not a refactor.
- Good, because hooks for admin actions and sensitive data access are in place but inert — adding such an event family in a future module is one writer call.
- Bad, because blocking on audit writes means the audit DB is part of the trust path; an audit DB incident degrades the application. Mitigated by HA in prod, made explicit here so it is owned, not surprising.
- Bad, because true tamper evidence in the cryptographic sense (chained hashes, signed entries) is not in v1. A privileged Postgres operator with both credentials and intent could in principle bypass the role grants. Mitigated by access discipline; full chaining deferred unless compliance requires it.
- Bad, because retention purges via `pg_cron` or a Nest task are themselves an operational item — they must be monitored, alerted on failure, and audited (via a meta-event).
- Bad, because GDPR right-of-erasure interactions are _acknowledged_ but not _resolved_ here — the legal review is owed; the architecture allows either policy.
- Neutral, because a dedicated audit Postgres instance can be substituted for the schema without code change (just `AUDIT_DATABASE_URL` pointed elsewhere). The trade-off can be revisited.

### Confirmation

**Wired in the foundation PR:**

- `apps/portal-bff/prisma/schema.prisma` enables the `multiSchema` preview, declares the `audit` schema alongside `public`, and carries the `AuditEvent` model with `AuditAudience` (`workforce | customer`) and `AuditOutcome` (`success | failure | denied`) enums.
- The migration `prisma/migrations/*_init_audit_schema/migration.sql` creates `audit.events`, `ALTER`s table + enum types to be owned by `audit_owner`, and re-applies the role grants explicitly: `INSERT` to `audit_writer`, `SELECT` to `audit_reader`, `SELECT, DELETE` to `audit_archiver` (SELECT is needed for archiver to evaluate the `created_at` predicate of "delete older than retention"). No grant of `UPDATE` or `TRUNCATE` to anyone — including the migrator's own login at runtime; only fresh schema migrations amend the table.
- The roles themselves and the schema with default privileges are provisioned earlier by `infra/local/init/postgres/01-init.sql` (dev) — production replicates the same SQL via the future on-prem infrastructure ADR.
- `apps/portal-bff/src/audit/audit.service.ts` exposes a single `AuditWriter.recordEvent(input)` method. Every write runs in a transaction whose first statement is `SET LOCAL ROLE audit_writer`, so the runtime contract holds even if the BFF connection is otherwise privileged. The INSERT itself is a **parameterised `$executeRawUnsafe`**, not `tx.auditEvent.create(...)` — see the "Implementation trap" callout in the Writer section above for the RETURNING-requires-SELECT explanation. `trace_id` is auto-resolved from the active OTel span; `actor_id_hash` is read from CLS or accepted as an explicit override (placeholder until ADR-0009 / ADR-0010 land their guards). Failures propagate — no catch-and-swallow, per "blocking writes: no audit ⇒ no action".
- BFF connects via the shared `DATABASE_URL` (the role switch is per-transaction). A separate `AUDIT_DATABASE_URL` connection pool is the production hardening, deferred — see "wired as features land" below.
- Smoke-tested end to end against the local-dev Postgres: `audit_writer` INSERTs successfully, fails on `UPDATE` and `DELETE`; `audit_archiver` SELECTs + DELETEs successfully.

**Wired as the corresponding features land:**

- One typed method per event family on `AuditWriter` — `signIn`, `signInFailed`, `signOut`, `sessionExpired`, `sessionRevoked`, `tokenValidationFailed`, `mfaAssertionFailed`, `authzDeny`, `adminAction`, `sensitiveDataAccess` — added as the matching feature ships (ADR-0009 / ADR-0010 / ADR-0011 / future authz). v1 keeps the surface narrow with the single `recordEvent` so events can be emitted today (e.g. by the auth flow once it lands) while the typed catalogue accretes on a real basis.
- `AUDIT_DATABASE_URL` separate connection (a distinct pool with `audit_writer`-only login credentials) — defense-in-depth that locks down what an at-runtime SQL injection can do further. v1 mitigates with `SET LOCAL ROLE` at the cost of sharing the pool with public-schema reads/writes; v2 splits.
- Startup self-test probe — a deliberate failing `UPDATE` against `audit.events` during boot, asserting it is rejected; if it succeeds, the BFF refuses to start. Lands with the connection split above.
- Retention purge job invoking the `audit_archiver` role daily via cron, emitting `audit.retention.purge` with `outcome = failure` on error. Operational concern — phase-3b infra ADR.
- Auth controller, MFA guard, sessions module, and authorization guards each call `AuditWriter` on the relevant outcomes. Tests assert one audit row per event.
- Integration test verifying that the same user produces the same `actor_id_hash` in both Pino logs and audit rows (same `LOG_USER_ID_SALT`, same hashing path) — wired with the auth + LOG_USER_ID_SALT enforcement.
- `AUDIT_RETENTION_DAYS < 30` and missing required env vars prevent BFF startup — wired with the retention job.
- Live-DB integration tests asserting the role contract (UPDATE/TRUNCATE rejected at runtime) — Testcontainers harness, separate PR.

## Pros and Cons of the Options

### Sink

#### Dedicated `audit` schema in the same Postgres (chosen)

- Good, because shared infrastructure with the business DB (HA, backup, monitoring) yet logically isolated by schema and role.
- Good, because transactional INSERT alongside a business action is trivial.
- Bad, because not a hard physical isolation; a Postgres-level compromise reaches both schemas. Mitigated by role separation.

#### Dedicated Postgres instance for audit only

- Good, because hard physical isolation, separate backups, separate access keys.
- Bad, because doubled operational surface (HA cluster, backup, monitoring, restoration drills) for a v1 yield that does not justify it.

#### Same Pino logger with a different category

- Bad, because the audit stream inherits the access controls of the app log stream — no separation. Bricolage.

#### Push to a SIEM

- Good, because aligned with how mature SOC operations consume audit data.
- Bad, because we do not have a SIEM yet; a future SIEM can ingest from the audit schema (via CDC or scheduled export) without touching application code.

#### Append-only file with rotation

- Bad, because no concurrent-safe writes, no queryability, fragile ops, hard to enforce immutability across rotations.

### Writer

#### Direct Prisma INSERT, transactional with the business action (chosen)

- Good, because either both succeed or both fail — no orphan business rows missing their audit trail.
- Good, because no extra moving part to operate.
- Bad, because the audit DB is in the critical path of the request. Already discussed under failure semantics.

#### Background queue

- Good, because the request path is decoupled from the audit DB.
- Bad, because an event sitting in a queue when the queue dies is a missing audit event. Loud failure modes are harder to design and audit.
- Bad, because adds a queue (Redis Streams, RabbitMQ) to the v1 surface — out of proportion.

#### Pino transport extension

- Bad, because piggybacks audit on the app-log path — exactly the coupling we are trying to prevent.

### Tamper evidence

#### Append-only at the role level (chosen)

- Good, because enforced by Postgres, not by application discipline.
- Bad, because a privileged DB operator can in principle override grants. The credentials and the intent must coincide — this is the standard enterprise trust model.

#### Cryptographic chaining

- Good, because tamper-evident even against the DB operator.
- Bad, because adds complexity to inserts (hashing the previous row), to schema migrations, and to range queries. Re-evaluated when a compliance regime mandates it.

#### WORM storage

- Good, because true immutability at the storage layer.
- Bad, because infrastructure-level decision deferred to the on-prem ADR.

### Failure semantics

#### Blocking (chosen)

- Good, because security-positive: no audit means no action.
- Bad, because audit DB is part of the request trust path.

#### Best-effort

- Good, because the application keeps running through audit-DB incidents.
- Bad, because a sufficiently long incident creates audit gaps that compliance frameworks treat as evidence of negligence.

### Retention

#### 365 days, env-overridable (chosen)

- Good, because conservative starting point; legally adjustable later.
- Bad, because the engineering default is not the legal default; the legal review remains owed.

#### Indefinite

- Bad, because GDPR principles favour proportionality; unbounded retention is hard to defend without a specific basis.

#### Tied to a specific regulation

- Bad, because we don't know yet which regulation applies. To be revisited when the org clarifies.

## More Information

- OWASP Logging Cheat Sheet — Audit logging section: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- PostgreSQL roles, GRANT, REVOKE: https://www.postgresql.org/docs/current/sql-grant.html
- PostgreSQL multiSchema with Prisma: https://www.prisma.io/docs/orm/prisma-schema/data-model/multi-schema
- `pg_cron`: https://github.com/citusdata/pg_cron
- GDPR/RGPD on retention and pseudonymisation: https://gdpr-info.eu/art-5-gdpr/
- Related ADRs: [ADR-0006](0006-persistence-postgresql-prisma.md) (Postgres + Prisma), [ADR-0008](0008-identity-model-entra-workforce-dual-audience.md) (audience), [ADR-0009](0009-auth-flow-oidc-pkce-msal-node.md) (auth flow events), [ADR-0010](0010-session-management-redis.md) (session lifecycle events), [ADR-0011](0011-mfa-enforcement-entra-conditional-access.md) (MFA events), [ADR-0012](0012-observability-pino-opentelemetry.md) (`trace_id`, `LOG_USER_ID_SALT` shared); future: infrastructure ADR (HA Postgres, backup, possibly WORM), legal review of retention, optional SIEM integration ADR, optional cryptographic-chaining ADR if compliance demands it.