feat: reland v4 BE handling (reverts #9775)

[scottheng96]

Problem

PR #9775 reverted the V4 BE handling (PR 2/5 of the MRF V3→V4 response migration, #9637, together with its bug fixes #9769 and #9770) so the Dependabot security fixes could be released independently. With that release out, the V4 BE work needs to land again — PR 3/5 (FE sends V4) is blocked until the backend accepts V4-native submissions.

Solution

Revert the revert: this PR reverts cdc25de (#9775), restoring the V4 BE pipeline exactly as it was before the revert. The conflict resolution was verified byte-identical to the pre-revert tree (cdc25de38^) across all 17 files, sitting cleanly on top of the SDK/response-v4 utils that #9776 cherry-picked back onto develop (those are untouched by this PR).

The restored state includes all four bug fixes that were part of the original V4 BE line:

1. Changed-field re-validation (fix(mrf): re-validate only changed fields on MRF submissions #9769) — previousResponses plumbed into checkIsResponseChangedV4 so only changed fields are re-validated, fixing rejection of verified email/mobile fields with expired OTP signatures on later workflow steps.
2. Wildcard email domain matching (fix(validation): apply wildcard email domain matching to V4 validator #9770) — V4 email domain validator uses emailDomainMatchesAllowed.
3. Attachment filename fix (part of feat(mrf): V4 response migration — PR 2/5 BE V4-native + V3 wire shim #9637) — triggerGuardDutyScanThenDownloadCleanFileChainV4 promotes the real filename into answer.value, so attachments no longer surface as a bare quarantine UUID in email/PDF Q&A, response JSON, webhooks, and admin CSV.
4. Webhook V3 encryption path (part of feat(mrf): V4 response migration — PR 2/5 BE V4-native + V3 wire shim #9637) — forms with a webhook URL still encrypt as V3 (mrfVersion: 1) via adaptV4ToV3, so existing webhook consumers keep working unchanged.

Breaking Changes

• No - this PR is backwards compatible. Stale V3 FE clients are adapted to V4 at the receiver (adaptV3ToV4 shim with WARN logging for rollout monitoring), and webhook consumers continue to receive V3-shaped encrypted payloads.

Tests

Unit suites pass locally: multirespondent-submission, submission.service, and email-validation (10 suites, 291 tests) plus response-v4 (7 tests).

TC1: MRF submission end-to-end (V4 path)

• Create a multirespondent form with text, email (verified), and attachment fields plus a second workflow step
• Submit step 1 with all fields filled and OTP-verify the email field
• Verify that the submission succeeds and responses render correctly in the admin view

TC2: Verified field carried across workflow steps (#9769 regression)

• Complete step 1 with a verified email field
• Wait past OTP signature expiry (or submit step 2 much later), editing only a non-verified field
• Verify that the step 2 submission succeeds without "answer does not have valid signature" errors

TC3: Wildcard email domain restriction (#9770 regression)

• Configure an email field with an allowed wildcard domain (e.g. @*.gov.sg)
• Submit with a matching subdomain address (e.g. user@agency.gov.sg)
• Verify that the submission is accepted, and a non-matching domain is rejected

TC4: Attachment filename in downstream sinks

• Submit an MRF form with an attachment on a form with email notifications
• Verify that the email Q&A, response JSON, and admin CSV show the uploaded filename with extension (not a UUID)

TC5: Webhook compatibility

• Configure a webhook on an MRF form and submit
• Verify that the webhook consumer can decrypt the payload as V3-shaped (mrfVersion: 1)

TC6: Stale V3 client shim

• POST an MRF submission with a V3-shaped body (version >= 3 and < 4)
• Verify that the submission is adapted to V4, succeeds, and a WARN log is emitted

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request