feat(mrf): MRF v4 minimal e2e — plumber native-envelope snapshot + reconstruction + gates (S4)

[kevin9foong]

Problem

MRF webhooks today only ship to plumber, and as a V3-shaped payload read straight off the live submission row. Two things block the v4 rollout: (1) an MRF row is mutated in place every step, so a delayed webhook retry would re-read the row and deliver the latest step's bytes, not the step that actually fired; and (2) there is no machinery to snapshot a step, reconstruct its payload, or decide per-consumer what shape/keys to ship. This slice lays the minimal end-to-end v4 path — the tracer bullet the rest of the epic builds on — for the privileged plumber consumer only, on initial send.

Closes #9744. Part of PRD #9740 (MRF v4 Webhooks — Option A). Stacked on #9743 (step-token write-guard).

Solution

On initial send to a plumber v4 form, we now freeze an immutable native-envelope snapshot to a dedicated S3 bucket before the step commits, then reconstruct the webhook payload from live row + snapshot. The v4 payload is the MRF envelope verbatim (content under the submission public key + the wrapped submission secret key as a read credential) — no re-encryption to the form key. Everything is dormant behind the V4 gate — answer-object-encryption (master) plus the consumer-class flags below (mrf-step-write-token, enable-mrf-webhooks); with V4 off, production is byte-identical to today.

The correctness-critical logic lands as small, isolated, unit-tested modules (reviewable one commit at a time):

• M6 snapshot store — one zod-validated object per write attempt at {formId}/{submissionId}/{submissionIndex}/{token}.json, through a single key-builder and a single fail-loud parser (missing / malformed / unknown _v all raise the same alarmable MRF_WEBHOOK_SNAPSHOT_DATA_INTEGRITY). Create-if-absent (IfNoneMatch:'*') with fresh-UUID collision retry.
• M7 producer — freezes the native envelope (content + that step's wrapped read key + native verifiedContent), because the row overwrites the keypair every step.
• M4 policy — the single place any decision branches on the consumer URL: content shape, read-key inclusion, step-token inclusion.
• M3 reconstruction — (liveRow, snapshot?) → WebhookData; with a submissionIndex the snapshot must exist (fail loud, never a live-row fallback), version is derived (v4→3), read key comes from the frozen snapshot; without one it returns today's live-row payload unchanged. This one function is what makes initial-send and retry byte-identical.

S3-first ordering (atomicity by ordering, not a shared txn)

sequenceDiagram
    participant R as Respondent
    participant BE as Backend
    participant S3 as V4 snapshot bucket
    participant DB as Mongo row
    participant WH as Plumber webhook
    R->>BE: PUT next-step submission
    BE->>S3: PUT snapshot, create-if-absent
    Note over BE,S3: fresh-UUID retry on collision. A failed PUT aborts the save.
    BE->>DB: save, recording the winning snapshotToken on the step
    Note over DB: lost version race becomes 409. Loser object is a benign orphan, no wipe.
    BE->>WH: reconstruct from row and snapshot, then send v4 payload
    Note over BE,WH: missing or malformed snapshot fails loud, never a live-row fallback

Loading

V4 gate — delivery and content shape

One diagram for both decisions a webhook goes through: whether it is delivered (the send gate) and what shape it carries (the encryption gate).

flowchart TD
    A[MRF submission] --> M{"answer-object-encryption ON? (master)"}

    %% AOE off — no V4 anywhere; shape is always V3
    M -->|no| B0{"Webhook type?"}
    B0 -->|plumber| SV3["Delivered · V3 (mrfVersion 1)"]
    B0 -->|generic or zapier| G0{"enable-mrf-webhooks ON?"}
    G0 -->|yes| SV3
    G0 -->|no| NOSEND["Not delivered"]

    %% AOE on — V4 possible, per consumer class
    M -->|yes| B{"Webhook type?"}
    B -->|none| NW["V4 stored · no webhook (mrfVersion 2)"]
    B -->|plumber| P{"mrf-step-write-token ON?"}
    P -->|yes| SV4["Delivered · V4 native envelope (mrfVersion 2)"]
    P -->|no| SV3
    B -->|generic or zapier| G{"enable-mrf-webhooks AND webhookFormat = v4 AND mrf-step-write-token?"}
    G -->|yes| SV4
    G -->|no| G0

Loading

answer-object-encryption is the master switch (PRD #9740 §D13): no form ships V4 while it is off, and post the V4-BE revert (#9775) stored responses are V3-shaped, so a V4 form upgrades via adaptV3ToV4 (mrfVersion 2) while every other path stays V3 (mrfVersion 1). Plumber is never gated by enable-mrf-webhooks in either dimension — it is always delivered to, and its shape depends only on AOE + mrf-step-write-token. enable-mrf-webhooks gates the generic/zapier branch alone: both whether it is delivered and (with webhookFormat = v4) whether it is V4.

Alternatives considered

• Commit the step first, then PUT the snapshot. Rejected — a failed PUT after commit would leave a committed step with no snapshot, breaking the guaranteed retry path and tripping the fail-loud reconstruction error. S3-first trades that for a benign orphan on an aborted txn, which is the safe direction; the orphan is never referenced (the token is recorded only on commit) and never read.
• A per-step deterministic {submissionIndex} key. Rejected — two concurrent writers to the same step would clobber each other, so reconstruction could return a rejected writer's bytes. A per-attempt UUID recorded on the winning submittedSteps entry means the loser's object is a harmless orphan and reconstruction always reads the winner.
• Re-encrypt the v4 payload to the form key (the old Option-D shape). Rejected per PRD PRD: MRF v4 Webhooks — Option A (step-token write-guard + native-envelope v4, v1 backward-compat) #9740 — the v4 payload stays in the native envelope and ships the wrapped submission secret key as a read credential; this is safe only because the same mrf-step-write-token flip also turns on the next-step write-guard (S3, S3 — MRF step-token write-guard (primitive, lifecycle, magic-link token, PUT gate, grace migration) #9743).

Breaking Changes

No — backwards compatible. With V4 off, plumber still receives V3 and no-webhook forms still get V4 — byte-identical to today, no snapshot written, no backfill. Generic/zapier MRF forms receive no MRF webhook today, so gating generic delivery behind enable-mrf-webhooks (default off) silences nothing that exists.

Tests

Start from a 3-step workflow MRF form whose webhook URL is a plumber URL (https://plumber.gov.sg/webhooks/...). Requires the V4 bucket provisioned/localstack-created (see #9753 / init-localstack.sh).

TC1: Plumber v4 initial send writes a snapshot and delivers the native envelope

• Turn answer-object-encryption and mrf-step-write-token ON. Submit step 1.
• Confirm a snapshot object exists in SUBMISSION_HISTORY_V4_S3_BUCKET at {formId}/{submissionId}/0/{token}.json and the row's submittedSteps[0].snapshotToken matches that token.
• Confirm the delivered webhook payload is the native envelope: encryptedContent equals the row's, carries encryptedSubmissionSecretKey, version === 3, and has no contentFormat / no step-token fields.
• Confirm the content decrypts with the submission secret key unwrapped from encryptedSubmissionSecretKey.

TC2: Flag-off parity (byte-identical to today)

• With mrf-step-write-token OFF, run a full plumber MRF fill/advance.
• Confirm the payload is the existing V3 shape (mrfVersion 1), NO snapshot is written, and the body matches today's delivery.

TC3: Fail-loud on a missing/corrupt snapshot

• With the flags on, delete (or corrupt) the snapshot object for a just-submitted step, then trigger the send.
• Confirm delivery does NOT fall back to the live row: no webhook is sent, an error is logged, and the mrf.snapshot.data_integrity_error metric is emitted (the string a Datadog monitor keys on).

TC4: Concurrent same-step submit → single winner

• Fire two concurrent submissions for the same step.
• Confirm exactly one submittedSteps entry is appended; the loser gets HTTP 409 (not a 5xx); the two attempts wrote to distinct S3 keys; and the delivered payload is the winner's bytes.

TC5: Payload-size guard

• Construct a multi-step submission whose reconstructed v4 body exceeds WEBHOOK_MAX_CONTENT_LENGTH (1 MB).
• Confirm no webhook is sent (no silent truncation) and the mrf.webhook.payload_too_large metric is emitted. Also confirm a realistic multi-step MRF stays within the limit.

TC6: Send gate for generic

• Point the form's webhook at a generic URL. With enable-mrf-webhooks OFF, submit a step → confirm NO webhook fires.
• Turn enable-mrf-webhooks ON → confirm the webhook fires.

TC7: Config is compulsory

• Unset SUBMISSION_HISTORY_V4_S3_BUCKET and boot the backend → confirm it throws a clear startup validation error.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request