apf_portal/docs/decisions/0013-audit-trail-separated-postgres-append-only.md

status, date, last-updated, decision-makers, tags

status	date	last-updated	decision-makers	tags
accepted	2026-04-29	2026-05-13	R&D Lead	security observability data

Audit trail — separated append-only Postgres schema, decoupled from app logs

Context and Problem Statement

ADR-0012 fixed the application observability stack: structured logs and traces, all carrying the same trace_id, optimised for debugging and operations. Application logs are noisy, broadly accessible, and have a relatively short retention window — appropriate for ops, inappropriate for security forensics or compliance evidence.

Audit logs serve a different purpose: they record security-relevant events authoritatively, must survive years, must not be alterable by anyone (including operators), and must be queryable by auditors with different access controls than developers. Mixing the two streams is the recurring failure mode that turns an audit log into noise — or worse, lets a developer accidentally UPDATE a row.

We need a fixed answer to: which events are recorded, where they are stored, who can read them, who can write them, who can never delete or modify them, how long they are kept, and how they relate to the application observability stream.

Decision Drivers

• Compliance posture: an audit log that can be tampered with is no audit log.
• Strict separation of concern from application logs — different sink, different access, different retention, different volume profile.
• Forensic queryability: an investigator must be able to reconstruct the timeline of any user's interactions with the system.
• Cross-reference with the app observability stream by trace_id so an audit event in isolation can be enriched with the surrounding app log.
• Avoid premature complexity: full cryptographic chaining and WORM storage are not required to start, but the path to them must remain open.
• Performance: per-event INSERT must not be in the critical latency path of unrelated operations.

Considered Options

Sink

• Dedicated audit schema in the same PostgreSQL instance as business data. (Chosen.)
• Dedicated PostgreSQL instance for audit only.
• Same Pino logger as app logs, with a different category.
• Push to a SIEM (Splunk, Wazuh, Graylog, etc.).
• Append-only file with rotation.

Writer

• Direct Prisma INSERT, in the same transaction as the business action when applicable. (Chosen.)
• Background queue (Redis Streams, RabbitMQ, etc.) decoupled from the request path.
• Pino transport extension.

Tamper evidence

• Append-only at the database role level (REVOKE UPDATE / DELETE / TRUNCATE on the writer and on every other role except a dedicated archiver). (Chosen for v1.)
• Cryptographic chaining (each row carries a hash linking it to the previous row) — deferred unless compliance demands.
• WORM-capable storage at the infrastructure layer — deferred to future infrastructure ADR if needed.

Failure semantics

• Audit failures are blocking — if the audit INSERT fails for an in-scope event, the operation fails. (Chosen.)
• Best-effort — audit failures are logged but the operation succeeds.

Retention

• 365 days default, override via env per environment. (Chosen.)
• Indefinite (until a cleanup is requested manually).
• Aligned with a specific regulation (HDS, PCI, ISO 27001 — to be revisited if and when an applicable framework is identified).

Decision Outcome

Sink. A dedicated audit PostgreSQL schema in the same database instance as the business data (ADR-0006). This requires the Prisma multiSchema preview feature, which is enabled in the BFF's schema. The schema is owned by a dedicated audit_owner role; only audit_writer can INSERT, only audit_reader can SELECT, only audit_archiver can DELETE rows older than the retention threshold, and UPDATE / TRUNCATE is granted to no role at all.

Schema (Prisma model):

model AuditEvent {
  id            String        @id @default(uuid()) @db.Uuid
  occurredAt    DateTime      @default(now()) @map("occurred_at") @db.Timestamptz
  eventType     String        @map("event_type")
  actorIdHash   String?       @map("actor_id_hash")          // matches user_id_hash from ADR-0012
  actorAudience Audience?     @map("actor_audience")          // ADR-0008
  sessionId     String?       @map("session_id")              // ADR-0010
  traceId       String?       @map("trace_id")                // ADR-0012 — cross-reference key
  sourceIp      String?       @map("source_ip") @db.Inet
  userAgent     String?       @map("user_agent")
  outcome       AuditOutcome
  details       Json          @default("{}")

  @@map("events")
  @@schema("audit")
  @@index([eventType])
  @@index([actorIdHash])
  @@index([occurredAt])
  @@index([traceId])
}

enum AuditOutcome {
  success
  failure
}

actor_id_hash uses the same salt and hash function as user_id_hash in ADR-0012, so an investigator can join an audit row with app log lines without any per-investigation re-hashing.

Writer. A NestJS AuditService exposes a single typed method per event family. Inside a request, the service uses the same Prisma transaction as the business action where applicable; outside a request (background expirations, scheduled jobs), it uses a fresh transaction. Writer connection runs under the audit_writer role and has only INSERT on audit.events — any attempt to UPDATE or DELETE is rejected by Postgres regardless of code intent.

Implementation trap — Prisma ORM cannot be used for the write. Prisma's tx.auditEvent.create(...) issues INSERT … RETURNING * to hydrate the entity it returns. Postgres requires the SELECT privilege on every column listed in RETURNING, and audit_writer has INSERT only by design — there is no SELECT grant on the writer role. The ORM path therefore fails at runtime with PostgresError 42501 / "permission denied for table events", an error whose message mentions neither SELECT nor RETURNING. The write path uses parameterised $executeRawUnsafe with no RETURNING clause; the schema-level id UUID @default(uuid()) from Prisma is replaced server-side with gen_random_uuid() in the SQL. This is a deliberate consequence of the role-separation contract and is pinned by a spec test. The alternative — granting SELECT on audit.events to audit_writer — would collapse the writer / reader role separation that the rest of this ADR rests on, so we go the other way.

Events emitted in v1.

event_type	When	outcome
auth.sign_in	OIDC callback creates a session	success
auth.sign_in.failed	OIDC callback rejects (state mismatch, signature, iss not allowlisted, MFA assertion missing, audience misclassified, …)	failure
auth.sign_out	POST /auth/logout invalidates a session	success
auth.session.expired	idle or absolute timeout fires (ADR-0010)	success
auth.session.revoked	session is force-deleted (admin or self)	success
auth.token.validation.failed	a token presented mid-session fails validation (signature, iss, audience, claim mismatch)	failure
auth.mfa.assertion.failed	the BFF rejects a session for missing or weak amr (ADR-0011)	failure
authz.deny	a guard rejects an authenticated request because the user's audience/claims don't authorise the action	failure

The details JSONB field carries event-specific information (e.g. expected vs received iss, denied route, claim names involved). Sensitive material (full tokens, claims that should never leave the BFF) is never placed in details — the same redaction posture as app logs applies, enforced by typed event payloads at the writer's boundary.

Hooks for admin actions and sensitive data access are designed-in: the writer accepts those event families today, but no v1 caller emits them. They are exercised by unit tests so the path stays alive.

Decoupling and cross-reference. Audit events do not flow through the Pino app logger. They flow through AuditService directly to Postgres. Each event carries the trace_id from the request-scoped CLS (ADR-0012); an investigator joining audit and app log streams on trace_id can reconstruct the full timeline of a request. The two streams have:

Aspect	App logs	Audit log
Sink	stdout → collector → on-prem backend	PostgreSQL audit.events
Volume	high (every request, every internal step)	low (security-relevant events only)
Retention	short (operational, weeks–months)	long (≥ 365 days, env-driven)
Access	broad (devs, ops)	restricted (audit_reader role; security/SOC)
Mutability	already gone if rotated; otherwise immutable by convention	immutable by Postgres role grants
Trace identifier	yes	yes (same value)

Failure semantics. Audit writes are blocking. If the INSERT fails (DB unreachable, role misconfigured, schema drift), the in-flight operation fails: the user gets a 503, the BFF emits a critical app log line, and on-prem alerting fires. This is the security-positive default ("if you cannot audit, you cannot act"). Trade-off acknowledged: the audit DB becomes part of the trust path, and an audit DB outage degrades the application. The mitigation is HA Postgres in prod (covered by the future infrastructure ADR), not best-effort writes.

Retention. Default 365 days, overridable per environment via AUDIT_RETENTION_DAYS. A daily scheduled job (pg_cron extension if available, otherwise a Nest scheduled task running under audit_archiver) deletes rows whose occurred_at is older than the threshold. The job is idempotent and emits its own audit event (audit.retention.purge) with the count of rows removed.

Compliance note. The 365-day default is engineering-prudent, not legally derived. Specific regulations applicable to the host organisation (RGPD/GDPR retention principles, sectoral rules — health, finance — if any) may demand longer or shorter windows, or full archival before deletion. The retention default is a starting point. The legal review is owed by the organisation; this ADR does not pretend to settle it.

GDPR / right-of-erasure interactions: audit events are typically retained under a "legal obligation" or "legitimate interest" basis even after a user's right-of-erasure request. The userId itself is already pseudonymised (actor_id_hash); the join key to user identity exists in the live DB only and disappears with account deletion, leaving the audit row scientifically pseudonymous. Whether this is legally sufficient is the legal team's call. The current implementation is compatible with both "keep" and "anonymise further" policies (we can null actor_id_hash for archived users without violating append-only because zeroing counts as a sanctioned operation under a yet-to-define audit_redactor role — designed-in, not implemented in v1).

Configuration (env-driven).

Variable	Purpose
AUDIT_DATABASE_URL	Postgres connection for the BFF as audit_writer (separate creds, different from the business DB role)
AUDIT_RETENTION_DAYS	default 365; the BFF refuses to start with a value below 30
AUDIT_ARCHIVER_DATABASE_URL	connection for the archiver job as audit_archiver (used by the scheduled task only)
LOG_USER_ID_SALT	shared with ADR-0012 — must be identical to keep the actor_id_hash join key consistent

Consequences

• Good, because audit log integrity rests on Postgres role grants — operationally enforced by the database, not by application discipline.
• Good, because the trace_id makes audit events instantly cross-referenceable with the rich app log, without coupling the two streams.
• Good, because the dedicated schema and roles let a future SOC team be granted audit_reader without giving them anything else.
• Good, because keeping the audit DB on the same instance avoids the operational doubling of running two Postgres clusters; the schema/role separation gives most of the isolation benefits for a fraction of the cost.
• Good, because the retention default is conservative and overridable, so adapting to a stricter regulatory regime is a config change, not a refactor.
• Good, because hooks for admin actions and sensitive data access are in place but inert — adding such an event family in a future module is one writer call.
• Bad, because blocking on audit writes means the audit DB is part of the trust path; an audit DB incident degrades the application. Mitigated by HA in prod, made explicit here so it is owned, not surprising.
• Bad, because true tamper evidence in the cryptographic sense (chained hashes, signed entries) is not in v1. A privileged Postgres operator with both credentials and intent could in principle bypass the role grants. Mitigated by access discipline; full chaining deferred unless compliance requires it.
• Bad, because retention purges via pg_cron or a Nest task are themselves an operational item — they must be monitored, alerted on failure, and audited (via a meta-event).
• Bad, because GDPR right-of-erasure interactions are acknowledged but not resolved here — the legal review is owed; the architecture allows either policy.
• Neutral, because a dedicated audit Postgres instance can be substituted for the schema without code change (just AUDIT_DATABASE_URL pointed elsewhere). The trade-off can be revisited.

Confirmation

Wired in the foundation PR:

• apps/portal-bff/prisma/schema.prisma enables the multiSchema preview, declares the audit schema alongside public, and carries the AuditEvent model with AuditAudience (workforce | customer) and AuditOutcome (success | failure | denied) enums.
• The migration prisma/migrations/*_init_audit_schema/migration.sql creates audit.events, ALTERs table + enum types to be owned by audit_owner, and re-applies the role grants explicitly: INSERT to audit_writer, SELECT to audit_reader, SELECT, DELETE to audit_archiver (SELECT is needed for archiver to evaluate the created_at predicate of "delete older than retention"). No grant of UPDATE or TRUNCATE to anyone — including the migrator's own login at runtime; only fresh schema migrations amend the table.
• The roles themselves and the schema with default privileges are provisioned earlier by infra/local/init/postgres/01-init.sql (dev) — production replicates the same SQL via the future on-prem infrastructure ADR.
• apps/portal-bff/src/audit/audit.service.ts exposes a single AuditWriter.recordEvent(input) method. Every write runs in a transaction whose first statement is SET LOCAL ROLE audit_writer, so the runtime contract holds even if the BFF connection is otherwise privileged. The INSERT itself is a parameterised $executeRawUnsafe, not tx.auditEvent.create(...) — see the "Implementation trap" callout in the Writer section above for the RETURNING-requires-SELECT explanation. trace_id is auto-resolved from the active OTel span; actor_id_hash is read from CLS or accepted as an explicit override (placeholder until ADR-0009 / ADR-0010 land their guards). Failures propagate — no catch-and-swallow, per "blocking writes: no audit ⇒ no action".
• BFF connects via the shared DATABASE_URL (the role switch is per-transaction). A separate AUDIT_DATABASE_URL connection pool is the production hardening, deferred — see "wired as features land" below.
• Smoke-tested end to end against the local-dev Postgres: audit_writer INSERTs successfully, fails on UPDATE and DELETE; audit_archiver SELECTs + DELETEs successfully.

Wired as the corresponding features land:

• One typed method per event family on AuditWriter — signIn, signInFailed, signOut, sessionExpired, sessionRevoked, tokenValidationFailed, mfaAssertionFailed, authzDeny, adminAction, sensitiveDataAccess — added as the matching feature ships (ADR-0009 / ADR-0010 / ADR-0011 / future authz). v1 keeps the surface narrow with the single recordEvent so events can be emitted today (e.g. by the auth flow once it lands) while the typed catalogue accretes on a real basis.
• AUDIT_DATABASE_URL separate connection (a distinct pool with audit_writer-only login credentials) — defense-in-depth that locks down what an at-runtime SQL injection can do further. v1 mitigates with SET LOCAL ROLE at the cost of sharing the pool with public-schema reads/writes; v2 splits.
• Startup self-test probe — a deliberate failing UPDATE against audit.events during boot, asserting it is rejected; if it succeeds, the BFF refuses to start. Lands with the connection split above.
• Retention purge job invoking the audit_archiver role daily via cron, emitting audit.retention.purge with outcome = failure on error. Operational concern — phase-3b infra ADR.
• Auth controller, MFA guard, sessions module, and authorization guards each call AuditWriter on the relevant outcomes. Tests assert one audit row per event.
• Integration test verifying that the same user produces the same actor_id_hash in both Pino logs and audit rows (same LOG_USER_ID_SALT, same hashing path) — wired with the auth + LOG_USER_ID_SALT enforcement.
• AUDIT_RETENTION_DAYS < 30 and missing required env vars prevent BFF startup — wired with the retention job.
• Live-DB integration tests asserting the role contract (UPDATE/TRUNCATE rejected at runtime) — Testcontainers harness, separate PR.

Pros and Cons of the Options

Sink

Dedicated audit schema in the same Postgres (chosen)

• Good, because shared infrastructure with the business DB (HA, backup, monitoring) yet logically isolated by schema and role.
• Good, because transactional INSERT alongside a business action is trivial.
• Bad, because not a hard physical isolation; a Postgres-level compromise reaches both schemas. Mitigated by role separation.

Dedicated Postgres instance for audit only

• Good, because hard physical isolation, separate backups, separate access keys.
• Bad, because doubled operational surface (HA cluster, backup, monitoring, restoration drills) for a v1 yield that does not justify it.

Same Pino logger with a different category

• Bad, because the audit stream inherits the access controls of the app log stream — no separation. Bricolage.

Push to a SIEM

• Good, because aligned with how mature SOC operations consume audit data.
• Bad, because we do not have a SIEM yet; a future SIEM can ingest from the audit schema (via CDC or scheduled export) without touching application code.

Append-only file with rotation

• Bad, because no concurrent-safe writes, no queryability, fragile ops, hard to enforce immutability across rotations.

Writer

Direct Prisma INSERT, transactional with the business action (chosen)

• Good, because either both succeed or both fail — no orphan business rows missing their audit trail.
• Good, because no extra moving part to operate.
• Bad, because the audit DB is in the critical path of the request. Already discussed under failure semantics.

Background queue

• Good, because the request path is decoupled from the audit DB.
• Bad, because an event sitting in a queue when the queue dies is a missing audit event. Loud failure modes are harder to design and audit.
• Bad, because adds a queue (Redis Streams, RabbitMQ) to the v1 surface — out of proportion.

Pino transport extension

• Bad, because piggybacks audit on the app-log path — exactly the coupling we are trying to prevent.

Tamper evidence

Append-only at the role level (chosen)

• Good, because enforced by Postgres, not by application discipline.
• Bad, because a privileged DB operator can in principle override grants. The credentials and the intent must coincide — this is the standard enterprise trust model.

Cryptographic chaining

• Good, because tamper-evident even against the DB operator.
• Bad, because adds complexity to inserts (hashing the previous row), to schema migrations, and to range queries. Re-evaluated when a compliance regime mandates it.

WORM storage

• Good, because true immutability at the storage layer.
• Bad, because infrastructure-level decision deferred to the on-prem ADR.

Failure semantics

Blocking (chosen)

• Good, because security-positive: no audit means no action.
• Bad, because audit DB is part of the request trust path.

Best-effort

• Good, because the application keeps running through audit-DB incidents.
• Bad, because a sufficiently long incident creates audit gaps that compliance frameworks treat as evidence of negligence.

Retention

365 days, env-overridable (chosen)

• Good, because conservative starting point; legally adjustable later.
• Bad, because the engineering default is not the legal default; the legal review remains owed.

Indefinite

• Bad, because GDPR principles favour proportionality; unbounded retention is hard to defend without a specific basis.

Tied to a specific regulation

• Bad, because we don't know yet which regulation applies. To be revisited when the org clarifies.

More Information

• OWASP Logging Cheat Sheet — Audit logging section: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
• PostgreSQL roles, GRANT, REVOKE: https://www.postgresql.org/docs/current/sql-grant.html
• PostgreSQL multiSchema with Prisma: https://www.prisma.io/docs/orm/prisma-schema/data-model/multi-schema
• pg_cron: https://github.com/citusdata/pg_cron
• GDPR/RGPD on retention and pseudonymisation: https://gdpr-info.eu/art-5-gdpr/
• Related ADRs: ADR-0006 (Postgres + Prisma), ADR-0008 (audience), ADR-0009 (auth flow events), ADR-0010 (session lifecycle events), ADR-0011 (MFA events), ADR-0012 (trace_id, LOG_USER_ID_SALT shared); future: infrastructure ADR (HA Postgres, backup, possibly WORM), legal review of retention, optional SIEM integration ADR, optional cryptographic-chaining ADR if compliance demands it.