feat(mrf): freeze a v4 snapshot before commit and send the reconstructed payload (S4) - #9822
Conversation
|
Tick the box to add this pull request to the merge queue (same as
|
701fed7 to
3352448
Compare
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Activates the multirespondent (MRF) v4 initial webhook path by writing a v4 snapshot before committing the step, reconstructing the webhook payload from live row + snapshot, and sending that reconstructed payload (with route parity guarantees).
Changes:
- Allow initial webhook sending to accept a pre-built
WebhookView(to support snapshot-reconstructed payloads). - Write v4 snapshots S3-first pre-commit and thread the in-memory snapshot into post-submit actions for initial send reconstruction.
- Add CI coverage to assert two-route parity (snapshot-backed vs live-row) for v4 payloads (modulo presigned URL values).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| apps/backend/src/app/modules/webhook/webhook.service.ts | Accept optional pre-built webhook view for initial sends (enables snapshot-reconstructed payloads). |
| apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts | Implements S3-first snapshot write + v4 reconstruction for initial webhook send eligibility. |
| apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.controller.ts | Threads snapshot from save result into post-submission actions. |
| apps/backend/src/app/modules/submission/multirespondent-submission/webhook/tests/initial-send-route-parity.spec.ts | Adds parity tests ensuring snapshot-backed and live-row routes produce identical v4 payloads (modulo URL values). |
| apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts | Adds integration-style unit tests for snapshot write ordering, eligibility gating, and reconstruction wiring. |
| apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.controller.spec.ts | Adds controller-level coverage for snapshot write failure mapping and aborted post-actions. |
| const policy = getWebhookPayloadPolicy({ | ||
| webhookType: webhookType === 'plumber' ? 'plumber' : 'generic', | ||
| isStepWriteTokenEnabled, | ||
| submissionIndex, | ||
| submittedStepsLength: submission.submittedSteps?.length ?? 0, | ||
| }) |
There was a problem hiding this comment.
resolved: this is intentional, since zaper is 'generic' consumer class.
| const growthbookWith = (enableMrfWebhooks: boolean) => | ||
| ({ | ||
| isOn: jest.fn().mockReturnValue(enableMrfWebhooks), | ||
| getFeatureValue: jest.fn((_flag: string, def: unknown) => def), | ||
| }) as any |
| attachmentDownloadUrls: Object.fromEntries( | ||
| Object.entries(data.attachmentDownloadUrls ?? {}).map(([key, url]) => [ | ||
| key, | ||
| new URL(url).pathname, | ||
| ]), | ||
| ), |
There was a problem hiding this comment.
resolved: fixed.
3352448 to
8972d77
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (2)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1191
- This
ResultAsyncchain is created inside avoid-returning helper and is neither returned nor explicitly marked as fire-and-forget (e.g.void <promise>). If the repo enforcesno-floating-promises(common in TS), this will fail lint; more importantly, it’s easy for future refactors to accidentally stop executing or observing this async work. Recommendation (mandatory if lint exists): either (1) return aResultAsyncfromsendMrfInitialWebhookIfEligibleand compose/await it in the caller, or (2) explicitly prefix this chain withvoidat the call site / in this helper to document intentional non-awaiting and satisfy floating-promise checks.
ResultAsync.fromPromise(
submission.getWebhookView(),
() => new DatabaseError(),
)
.andThen((liveView) => {
const policy = getWebhookPayloadPolicy({
webhookType: webhookType === 'plumber' ? 'plumber' : 'generic',
isStepWriteTokenEnabled,
submissionIndex,
submittedStepsLength: submission.submittedSteps?.length ?? 0,
})
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// Passed only when a snapshot exists; reconstruction fails loud
// otherwise.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
}).asyncAndThen((data) => {
const webhookView: WebhookView = { data }
return WebhookFactory.sendInitialWebhook(
submission,
webhookUrl,
isRetryEnabled,
webhookView,
).map(() => undefined)
})
})
.mapErr((error) => {
logger.error({
message: 'Multirespondent submission webhook error',
meta: logMeta,
error,
})
})
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1172
- The comment is misleading: this code path also calls
reconstructMrfWebhookDatawhensnapshotis undefined (and passessubmissionIndex: undefined), so it’s not accurate to say reconstruction 'fails loud otherwise'. Suggest updating the comment to the precise contract (e.g., 'When snapshot is provided, submissionIndex is required; otherwise reconstruction uses live data only').
snapshot,
// Passed only when a snapshot exists; reconstruction fails loud
// otherwise.
submissionIndex: snapshot ? submissionIndex : undefined,
8972d77 to
747ca71
Compare
747ca71 to
140b6cd
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1174
- The inline comment is inconsistent with the behavior in this function: the code explicitly supports the no-snapshot route (it passes
snapshotas undefined and omitssubmissionIndex). Update the comment to reflect the intended contract (e.g., reconstruction should fall back to live-row whensnapshotis absent, and only requiressubmissionIndexwhen a snapshot is present), or adjust the call if the function truly must fail without a snapshot.
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// Passed only when a snapshot exists; reconstruction fails loud
// otherwise.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
}).asyncAndThen((data) => {
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1135
- This helper is used from both create and update post-submission actions, but it always logs "Sending initial webhook…". This makes production logs ambiguous and can break log-based dashboards/alerts that previously distinguished create vs update. Consider renaming the helper and/or parametrizing the log message (e.g., "create" vs "update") so observability remains accurate.
logger.info({
message: 'Sending initial webhook for multirespondent submission',
meta: { ...logMeta, webhookType },
})
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts:67
- This mock replaces
writeV4Snapshot/readV4Snapshotwith barejest.fn()across the entire test file, but default return values are only configured inside the new nested describe block. If any other existing tests in this file hit a snapshot-enabled path, they may fail at runtime becausewriteV4Snapshot()returnsundefined(not aResultAsync). To make the suite robust, set safe defaults in a top-levelbeforeEach(or in the mock factory) so these mocks always return a validResultAsyncunless a test overrides them.
jest.mock('../webhook/submission-snapshot.store', () => {
const actual = jest.requireActual('../webhook/submission-snapshot.store')
return {
...actual,
writeV4Snapshot: jest.fn(),
readV4Snapshot: jest.fn(),
}
})
140b6cd to
7a42870
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1135
- This log line is emitted from both create and update post-submission flows (since the same helper is called in both places), but the message always says 'initial webhook'. That will make production debugging ambiguous. Consider changing the message to a neutral 'Sending webhook for multirespondent submission', or pass an explicit context (e.g. 'create' | 'update') into
sendMrfInitialWebhookIfEligibleand include it in the log meta/message.
logger.info({
message: 'Sending initial webhook for multirespondent submission',
meta: { ...logMeta, webhookType },
})
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts:3771
- The
growthbookWithhelper returns the sameisOnvalue for every flag, which can accidentally enable/disable unrelated flags (e.g.mrfStepWriteToken) and make future test changes brittle as more flags are consulted. Prefer using the more explicitgrowthbookWithFlags({ enableMrfWebhooks, mrfStepWriteToken })everywhere, or updategrowthbookWithto only affect the intended flag(s) and default others tofalse.
const growthbookWith = (enableMrfWebhooks: boolean) =>
({
isOn: jest.fn().mockReturnValue(enableMrfWebhooks),
getFeatureValue: jest.fn((_flag: string, def: unknown) => def),
}) as any
apps/backend/src/app/modules/webhook/webhook.service.ts:255
- Adding an optional positional parameter here makes call sites easier to misread and harder to extend safely (especially if more optional args are added). Consider switching this function (and its callers) to accept a single options object (e.g.
{ submission, webhookUrl, isRetryEnabled, webhookView }) to make the API self-documenting and reduce the risk of argument-order mistakes.
submission: IEncryptedSubmissionSchema | IMultirespondentSubmissionSchema,
webhookUrl: string,
isRetryEnabled: boolean,
webhookView?: WebhookView,
d898a67 to
54eb639
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1169
- Typo in comment: 'RATONALE' should be 'RATIONALE'.
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// RATONALE: if snapshot does not exist, we use the live row.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1170
- Overloading
submissionIndexto mean both 'which step to reconstruct' and 'whether to reconstruct at all' (by passingundefined) makes the call-site harder to reason about and couples behavior toreconstructMrfWebhookData’s handling ofundefined. A clearer approach is to always pass the actualsubmissionIndex, and let reconstruction choose snapshot-vs-live-row based onsnapshotpresence (or an explicit boolean likeuseSnapshot).
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// RATONALE: if snapshot does not exist, we use the live row.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
}).asyncAndThen((data) => {
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts:3865
- This mocked
WebhookViewshape looks inconsistent with the PR’s stated wire contract (e.g., v4 sends assertingversion === 4andcreatedtypically being serialized). Using a mock that more closely matchesgetWebhookView()output (or deriving the view from a real model instance, as the parity test does) would make these reconstruction/send-path tests more representative and reduce the risk of false confidence.
const buildLiveWebhookView = (): WebhookView =>
({
data: {
formId: mockFormId,
submissionId: 'live-submission-id',
encryptedContent: 'live-content',
encryptedSubmissionSecretKey: 'live-read-key',
verifiedContent: 'live-verified',
version: 2,
created: new Date(),
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Suppressed comments (2)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1169
- Typo in comment: 'RATONALE' should be 'RATIONALE'.
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// RATONALE: if snapshot does not exist, we use the live row.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:892
- Snapshot-build + write + token-recording logic is now duplicated across create and update flows. This is easy to accidentally diverge (e.g. attachmentMetadata conversion, createdAt source, token recording timing). Consider extracting a shared helper that (a) builds the v4 snapshot, (b) writes it, and (c) returns
{ snapshot, token }(or directly mutates the step meta), so both code paths stay consistent and future changes only need to be made once.
const snapshot = shouldWriteSnapshot
? buildV4Snapshot({
formId: String(form._id),
submissionId: String(submissionObjectId),
submissionIndex: 0,
workflowStep: 0,
encryptedContent,
encryptedSubmissionSecretKey,
verifiedContent,
attachmentMetadata: Object.fromEntries(
attachmentMetadata ?? new Map(),
),
createdAt: submittedStepMeta.submittedAt,
})
: undefined
const writeSnapshotIfNeeded: ResultAsync<undefined, SnapshotWriteError> =
snapshot
? writeV4Snapshot(snapshot).map(({ token }) => {
submittedStepMeta.snapshotTokens = { v4: token }
return undefined
})
: okAsync(undefined)
| // RATIONALE: Generate the submissionId up front so the S3 snapshot key can be built | ||
| // BEFORE the row is persisted (Snapshot saves to S3 first ordering). | ||
| const submissionObjectId = new mongoose.Types.ObjectId() | ||
|
|
||
| const submissionContent: MultirespondentSubmissionContent & { | ||
| _id: mongoose.Types.ObjectId | ||
| } = { | ||
| _id: submissionObjectId, |
| ? buildV4Snapshot({ | ||
| formId: String(form._id), | ||
| submissionId: String(submissionObjectId), | ||
| submissionIndex: 0, |
54eb639 to
00b13b9
Compare
|
TC1: webhook payload |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
apps/backend/src/app/modules/webhook/webhook.service.ts:275
- The
const webhookViewToUse = ...statement is missing a terminating semicolon; in codebases that enforce semicolons via linting/formatting this can fail CI. Add a semicolon after the conditional expression assignment.
const webhookViewToUse = webhookView
? okAsync(webhookView)
: ResultAsync.fromPromise(
submission.getWebhookView(),
() => new DatabaseError(),
)
return webhookViewToUse.andThen((webhookView) =>
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1170
- Typo in comment: 'RATONALE' should be 'RATIONALE'.
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// RATONALE: if snapshot does not exist, we use the live row.
submissionIndex: snapshot ? submissionIndex : undefined,
policy,
}).asyncAndThen((data) => {
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:892
- The snapshot build +
writeV4Snapshot+snapshotTokensassignment pattern is duplicated in both create and update flows. Consider extracting a small helper that (a) builds the v4 snapshot object and (b) returns aResultAsyncthat writes it and applies the token to the step metadata. This reduces duplication and helps ensure future changes (e.g., snapshot schema evolution or logging) remain consistent between create/update.
const snapshot = shouldWriteSnapshot
? buildV4Snapshot({
formId: String(form._id),
submissionId: String(submissionObjectId),
submissionIndex: 0,
workflowStep: 0,
encryptedContent,
encryptedSubmissionSecretKey,
verifiedContent,
attachmentMetadata: Object.fromEntries(
attachmentMetadata ?? new Map(),
),
createdAt: submittedStepMeta.submittedAt,
})
: undefined
const writeSnapshotIfNeeded: ResultAsync<undefined, SnapshotWriteError> =
snapshot
? writeV4Snapshot(snapshot).map(({ token }) => {
submittedStepMeta.snapshotTokens = { v4: token }
return undefined
})
: okAsync(undefined)
00b13b9 to
e065744
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (4)
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1168
- Typo in comment: 'RATONALE' should be 'RATIONALE'.
return reconstructMrfWebhookData({
liveData: liveView.data,
snapshot,
// RATONALE: if snapshot does not exist, we use the live row.
submissionIndex: snapshot ? submissionIndex : undefined,
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1108
- This helper is invoked from both post-create and post-update flows, but its name implies create-only semantics ('Initial'). Consider renaming to something lifecycle-agnostic (e.g., sendMrfWebhookIfEligible / sendMrfStepWebhookIfEligible) to avoid misleading call sites and future misuse.
const sendMrfInitialWebhookIfEligible = ({
submission,
snapshot,
webhookUrl,
isRetryEnabled,
growthbook,
logMeta,
}: {
apps/backend/src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts:1133
- This log message will be emitted for update sends too (since the same helper is used in both post-create and post-update actions). Please make the message accurately reflect the operation (e.g., include whether it's create vs update or just say 'Sending webhook...') to keep observability and incident triage unambiguous.
logger.info({
message: 'Sending initial webhook for multirespondent submission',
meta: { ...logMeta, webhookType },
})
apps/backend/src/app/modules/submission/multirespondent-submission/tests/multirespondent-submission.service.spec.ts:4056
- PR description states that with
mrf-step-write-tokenoff, plumber should keep receiving V3 and no snapshot should be written. This test (and the impliedshouldWriteV4Snapshot/send gating behavior) asserts the opposite for plumber. Please align either (a) the implementation/tests to the stated flag behavior, or (b) update the PR description to reflect the actual plumber gating rules.
it('still writes a snapshot for a plumber V4 row with no flags on', async () => {
const result = await createMultiRespondentFormSubmission({
form: buildV4Form(),
encryptedPayload: buildV4Payload(),
logMeta: { action: 'test' },
growthbook: growthbookWithFlags({}),
})
expect(result.isOk()).toBe(true)
expect(MockSnapshotStore.writeV4Snapshot).toHaveBeenCalledTimes(1)
})
e065744 to
76f84b9
Compare
eliotlim
left a comment
There was a problem hiding this comment.
LGTM! 👍 to be merged as a stack.
The gate was an inline `hasWebhook ? 1 : 2` inside encryptSubmission —
untestable on its own and keyed on mere URL *presence*. Extracts it as a
pure `getMrfVersion({ webhookType, isStepWriteTokenEnabled })` so its
decision table is unit-enumerable, and keys it on consumer *class* instead:
no webhook -> V4 (unchanged from develop)
plumber, token ON -> V4 (the new arm)
plumber, token OFF -> V3 (unchanged: today's downgrade)
generic / zapier -> V4
This is a *storage* decision — what the row holds — and deliberately not an
input to whether a webhook is delivered. `enable-mrf-webhooks` is therefore
not a parameter here at all; delivery is gated separately by the
send-eligibility function.
Note the direction: the pipeline is V4-native in-process, and V3 is produced
by adapting *down* for consumers that cannot parse V4. This commit only
widens which consumer classes keep the native shape; with
`mrf-step-write-token` off, plumber reduces to develop's behaviour exactly.
`webhookFormat` lands on the shared FormWebhook type as the field a later
slice keys generic's opt-in on. Nothing in production reads it — the specs
here pin that getMrfVersion *ignores* it, so a form carrying the field
cannot change shape until that slice arrives with the fallback that makes it
safe.
sendInitialWebhook always derived its payload from the live row via getWebhookView. Adds an optional pre-built view so a caller that has already reconstructed the payload from a frozen snapshot can pass it straight through; omitting it keeps the existing live-row behaviour exactly. Additive and inert — no caller passes it yet.
…ted payload
Activates the pieces added by the preceding commits. Nothing before this
commit is reachable from a request.
Write path — S3 first, then commit:
PUT snapshot (create-if-absent, fresh-UUID retry on collision)
-> save the row, recording snapshotTokens.v4 on the winning step
-> respond
-> send, reconstructing from row + the in-memory snapshot
The ordering is the atomicity mechanism; there is no shared transaction. A
failed PUT aborts the save, so the respondent gets a 500 with nothing
committed and a safe retry. The rejected alternative — commit first, then PUT
— would leave a committed step with no snapshot, which breaks the guaranteed
retry path and trips the fail-loud reconstruction error. S3-first trades that
for an orphaned object on an aborted save, which is the safe direction: the
token is recorded only on commit, so the orphan is never referenced and never
read. Deliberately not deleted — a delete on the error path is another
failure surface for no benefit.
On create the submission `_id` is generated up front so the snapshot key can
be built before the row exists. On update the step is appended only *after*
the token is recorded, so the value survives mongoose's subdocument cast; a
lost version race therefore surfaces as a 409, with the loser's object left
as a harmless orphan.
Send path:
- The snapshot object is threaded through the HTTP response in memory rather
than re-read from S3. A read-back would mostly re-verify what a 200 from a
conditional PUT already guarantees, while adding a failure surface on the
happy path: a transient GET would collapse into SnapshotDataIntegrityError
and drop the webhook for a submission whose snapshot was perfectly healthy.
It is also deliberately not re-derived from the row — buildV4Snapshot is
pure over row fields today, but a later slice's form-key copy exists
nowhere on the row, so re-deriving would reintroduce a live-row fallback
through a side door.
- A V3 row still ships verbatim via the legacy getWebhookView branch. Only a
V4 row goes through the policy and reconstruction.
- Whether a step got a snapshot is a *storage* decision, so the two V4 routes
(retries on/off) must ship the same bytes. initial-send-route-parity.spec
pins that, modulo attachment presigned URL values, which are minted per
send — the URL keys and the object each points at must still match.
There is no pre-send payload-size guard. An earlier revision had one; it
reused a *response* cap as a request cap, measured the body before
presigning, applied to only one of the two routes, and on trip returned
success *without sending* — no webhook record, no retry, nothing
admin-visible. An oversized POST already yields a recorded, retried, alarmable
413. A real outbound limit, if ever wanted, belongs in sendWebhook after
presigning and applied to every route.
With `mrf-step-write-token` off, plumber keeps receiving today's V3 payload
and no snapshot is written.
76f84b9 to
7b5c667
Compare

Problem
Everything below this PR in the stack is dormant — the snapshot store, the payload policy, reconstruction and the gates exist but nothing reaches them from a request. This PR is the activation: the minimal end-to-end v4 path (the tracer bullet the rest of PRD #9740 builds on), for the privileged plumber consumer only, on initial send.
Closes #9744.
Solution
On initial send to a plumber V4 form, freeze an immutable native-envelope snapshot before the step commits, then reconstruct the webhook payload from
live row + snapshot. The v4 payload is the MRF envelope verbatim — content under the submission public key, plus the wrapped submission secret key as a read credential. No re-encryption to the form key (per PRD #9740); that is safe only because the samemrf-step-write-tokenflip also turns on the next-step write-guard (#9758).S3-first ordering (atomicity by ordering, not a shared txn)
sequenceDiagram participant R as Respondent participant BE as Backend participant S3 as V4 snapshot bucket participant DB as Mongo row participant WH as Plumber webhook R->>BE: PUT next-step submission BE->>S3: PUT snapshot, create-if-absent Note over BE,S3: fresh-UUID retry on collision. A failed PUT aborts the save (500, nothing committed). BE->>DB: save, recording snapshotTokens.v4 on the winning step Note over DB: lost version race becomes 409. Loser object is a benign orphan, no wipe. BE-->>R: response (the snapshot object is threaded past it in memory) BE->>WH: reconstruct from row + in-memory snapshot, then send v4 payload Note over BE,WH: no S3 read-back on initial send. Retry (S5) reads by recorded token.The ordering is the atomicity mechanism. A failed PUT aborts the save, so the respondent gets a 500 with nothing committed and a safe retry.
On create, the submission
_idis generated up front so the snapshot key can be built before the row exists. On update, the step is appended only after the token is recorded, so the value survives mongoose's subdocument cast; a lost version race therefore surfaces as a 409, with the loser's object left as a harmless orphan.Two-route parity
Whether a step got a snapshot is a storage decision, so the two plumber-V4 routes must ship the same bytes:
versioninitial-send-route-parity.spec.tspins that in CI, modulo attachment presigned URL values (minted per send) — the URL keys and the S3 object each points at must match.A V3 row still ships verbatim via the legacy
getWebhookViewbranch; only a V4 row goes through the policy and reconstruction.Alternatives considered
SnapshotDataIntegrityErrorand drop the webhook for a submission whose snapshot was perfectly healthy. The object is threaded in memory instead, and deliberately not re-derived from the row —buildV4Snapshotis pure over row fields for v4, but the v1 form-key copy a later slice adds exists nowhere on the row, so re-deriving would reintroduce a live-row fallback through a side door.sendWebhookafter presigning, applied to every route, with its own justification.Breaking Changes
None in this PR. With
mrf-step-write-tokenoff, plumber keeps receiving today's V3 payload and no snapshot is written.The one deliberate wire change in this stack —
versiondescribing the content format — is isolated in #9820 and needs plumber sign-off there, not here.Tests
Requires the V4 bucket provisioned/localstack-created (#9753). Start from a 3-step workflow MRF form with a plumber webhook URL.
TC1: Plumber v4 initial send writes a snapshot and delivers the native envelope
mrf-step-write-tokenON, webhook retries ON. Submit step 1.{formId}/{submissionId}/0/{token}.jsonandsubmittedSteps[0].snapshotTokens.v4matches that token.encryptedContentequals the row's, carriesencryptedSubmissionSecretKey,version === 4, no step-token fields.encryptedSubmissionSecretKey.TC2: Flag-off parity
mrf-step-write-tokenOFF, full plumber fill/advance → V3 shape, rowmrfVersion 1, NO snapshot written.TC3: Two-route parity for a V4 row
version(4both), modulo presigned URL values. (Gated byinitial-send-route-parity.spec.ts.)TC4: Attachments are frozen per step
attachmentDownloadUrlspresigns step 1's keys.TC5: Send gate for generic
enable-mrf-webhooksOFF → NO webhook fires. Turn it ON (withmrf-step-write-tokenalso ON, which delivery requires) → fires with the Case A v4 payload.TC6: Snapshot tokens never leave the server
snapshotTokens;nextStepRecipientEmailsstill present for the MRF metadata column.Webhook payload content format, per case
Every case below is with the whole stack merged.
X-FormSG-Signatureis unchanged throughout;snapshotTokensnever appears in any of them (#9817).mrf-step-write-tokenenable-mrf-webhooksversiongetWebhookViewCase A — plumber V4 (TC1/TC3): the native envelope
Identical field-for-field on both the snapshot and no-snapshot routes, modulo presigned URL values:
{ "data": { "formId": "66c0f0…", "submissionId": "66c0f1…", "created": "2026-08-03T04:15:22.180Z", "version": 4, "encryptedContent": "<V4 responses, encrypted to the submission public key>", "encryptedSubmissionSecretKey": "<submission secret key, wrapped to the form public key>", "verifiedContent": "<native verifiedContent, key omitted when absent>", "attachmentDownloadUrls": { "<fieldId>": "<presigned GET url, 1h expiry>" }, "paymentContent": {}, "workflowContent": { "workflow": [ /* FormWorkflowDto */ ], "workflowStep": 0, "submittedSteps": [ { "isApproval": false, "submittedAt": "2026-08-03T04:15:22.180Z", "nextStepRecipientEmails": ["next@agency.gov.sg"], "submitterId": "S1234567D" } ] } } }No re-encryption to the form key: the consumer unwraps
encryptedSubmissionSecretKeywith the form secret key, then decryptsencryptedContent. No step-token field —includeEncryptedStepTokenis computed by the policy but nothing in this stack reads it.On the snapshot route
encryptedContent,verifiedContent,encryptedSubmissionSecretKey,attachmentDownloadUrlsandworkflowContent.workflowStepall come from the frozen object, andsubmittedStepsis truncated tosubmissionIndex + 1. On the no-snapshot route they come from the live row andsubmittedStepsis untruncated — which for an initial send is the same content, since the sending step is the latest one. That equality is whatinitial-send-route-parity.spec.tspins.Case B — plumber, flag off (TC2): today's V3 payload
Same envelope, delivered by the legacy
getWebhookViewbranch without consulting the policy:{ "data": { "version": 3, "encryptedContent": "<V3-shaped responses (adaptV4ToV3), same crypto>", "encryptedSubmissionSecretKey": "<unchanged from develop>", "verifiedContent": "<unchanged>", "attachmentDownloadUrls": { "<fieldId>": "<presigned GET url>" }, "paymentContent": {}, "workflowContent": { "workflow": [ /* … */ ], "workflowStep": 0, "submittedSteps": [ /* … */ ] } } }Identical to develop except
version, which #9820 changes from4to3— the only wire change in the stack, and it is live independently of the flag.Case C — generic / zapier (TC8)
Delivery needs both
enable-mrf-webhooksandmrf-step-write-token. Since the payload policy keyscontentFormatonmrf-step-write-tokenand not on consumer class, a send-eligible generic consumer receives Case A's payload — native envelope,version: 4, includingencryptedSubmissionSecretKey. There is no combination in this stack that delivers a V3 payload to a generic consumer: with the flag off it is not delivered at all.Unchanged
Non-MRF encrypt submissions are untouched: no
workflowContent, noencryptedSubmissionSecretKey,versionstill the row's own field. Retry sends (S5) reconstruct from the recorded token and therefore reproduce Case A byte-for-byte apart from freshly minted presigned URLs.Part of PRD #9740 (MRF v4 Webhooks — Option A). Top of the stack; the prerequisites are #9817 → #9818 → #9819 → #9820 → #9821. Previously a single 24-commit PR — split so each layer can be reviewed and merged on its own.