fix(identity,ci): rc.92's failed live smoke was a stale-edge-propagation race, not a code bug — deploy-staging.sh now verifies - #175
Merged
Conversation
…ion race, not a code bug — deploy-staging.sh now verifies The rc.92 auth-Wave-1 deploy's smoke ran ~18s after `wrangler deploy` reported success and hit an edge PoP still serving the OLD worker version: a brand-new route (/auth/code) 404'd, a brand-new response header was simply absent. Byte-identical to a real code bug until the deploy log timestamps are read against the failure timestamps — confirmed live: a manual probe against staging minutes later showed the code mint working correctly, and wrangler's own "Current Version ID" output didn't match what /health was reporting at smoke time. `wrangler deploy` returning success means the upload was ACCEPTED, not that every edge PoP is serving it yet. Fixed at the root: GET /health now reports env.CF_VERSION_METADATA.id (Workers' native version-metadata binding, wired in both wrangler.toml scopes); deploy-staging.sh polls it against a pre-deploy baseline (up to 90s) before ever invoking smoke, failing loudly instead of racing a stale edge. deploy-prod.sh is untouched (manual-approval gated, no equivalent race). Also fixes a real robustness bug the incident exposed: two resume-by-code/ resume-by-link smoke steps called fetch() with a blank URL when an earlier step had already failed, throwing an UNCAUGHT exception that killed the entire remaining suite (TOTP, billing, GFS snapshots, voice, semantic search — everything downstream of the failure). Both are now guarded: a failed step stays one failed assertion, never a process crash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What actually happened (evidence, not the original hypothesis)
The triage that assigned this fix hypothesized a D1/miniflare-vs-real-D1 divergence in the code-mint path. That hypothesis is wrong, and I want to say so plainly before the fix, because the correct fix depends on the correct diagnosis:
createMagicLink's 6-digit code is generated byrandomNumericCode()in memory, before the D1 INSERT even runs — its value cannot depend on any D1 return-shape/RETURNING/.metabehavior, real or simulated. There is no code path inauth-handlers.tswhere the link header is set and the code header isn't, for a given deployed version — both are written from the sameextraobject in the sameif (deps.exposeDevLinks)block.What I found instead, from
gh run view 29575426905 --log(the failing deploy-staging.yml run) and a live probe against staging:wrangler deploy --env stagingfor the identity worker reported success at 11:02:07.54 ("Deployed parachute-identity-staging triggers", Current Version ID5dfc52f1-…).x-parachute-dev-magic-code— FAILED at 11:02:25.68,code null. The very next step,POST /auth/code, got 404 — the exact symptom of a route that doesn't exist yet on an OLD worker version, since/auth/codeis brand new in this PR.POST /auth/magicagainst live staging myself (minutes later, well past propagation):x-parachute-dev-magic-code: 076496came back correctly, alongside the link header, on the first try. I then walked the full code-verify flow live — verify-by-code, session mint, the link dying once the code consumed the shared row, a wrong-code neutral failure — all correct.This is the exact signature of the smoke request landing on an edge PoP that hadn't yet picked up the new Worker version (
wrangler deployreporting success only means the upload was accepted, not that every PoP is serving it — the smoke ran ~18s later and still lost the race). Nothing about the auth-code logic itself needed to change; #174's code was correct as shipped.A second, real bug this incident exposed (unrelated to the propagation race, but a genuine defect): the new resume-by-code smoke block called
fetch(rcLoc, …)wherercLoccould be""after an earlier step failed — Bun'sfetch()throwsTypeError: fetch() URL must not be a blank stringon that, uncaught, which killed the entire remaining ~900-line smoke suite (TOTP, billing, GFS snapshots, voice transcription, semantic search — everything downstream never ran). The pre-existing resume-by-link block has the identical latent pattern (never triggered before because nothing it touches was ever a brand-new route).What live smoke proves that vitest cannot (per the ask)
vitest-pool-workers' 796 green tests prove the logic is correct against a Worker that is unambiguously the code under test, invoked in-process — there is no "which version answered this request" question inside a single vitest run. What only a live, freshly-deployed staging smoke can prove, and what this fix specifically targets:
wrangler deploy's own request happened to hit.3** That the whole ~900-line smoke suite survives a real failure without cascading — the blank-URL crash only manifests when an earlier live HTTP round-trip genuinely comes back unexpected; vitest's mocked/in-process requests never hit this failure mode because nothing in that environment 404s a route that exists in the same test run.
The fix
GET /healthnow reportsenv.CF_VERSION_METADATA.id— Workers' native version-metadata binding ([version_metadata]in wrangler.toml, both scopes; confirmed via the Cloudflare docs before wiring it in, not assumed).Env.CF_VERSION_METADATAis optional so bare/test configs still type-check;/health'sversionfield degrades tonullwhen unbound.deploy-staging.shpolls/healthagainst a pre-deploy baseline (up to 90s, 3s interval) after the identity worker'swrangler deploy --env staging, before ever handing off tosmoke-staging.ts. Times out loudly (exit 1, clear message) rather than silently racing a possibly-stale edge — the CI failure mode flips from "cryptic cascade of unrelated-looking FAILs" to "deploy script explicitly says propagation didn't complete in time."deploy-prod.shis completely untouched — it's manual-approval-gated (GitHub Environment required-reviewer rule) so there's no equivalent time pressure, and the ask was explicitly "do not touch prod anything." The[version_metadata]binding IS added to the wrangler.toml top-level scope too (since the identity worker's code — including/health— is shared between environments; there's no way to make staging's/healthversion-aware without production's config also carrying the binding), but that's an inert additive config/response-shape change, not an active deploy-time behavior change, and it only takes effect through the normal gated prod-deploy process.fail()-ing that one assertion instead of throwing and aborting the suite.Gates (2 runs each, stable — after clearing ~161 stray
workerdprocesses left over from repeated earlier local runs, which caused one transient false-negative I want to be upfront about rather than paper over)workers/identity:bun run typecheckclean (both runs) ·bun x vitest run— 33 files, 796 tests passed (both runs, unchanged from feat(identity): auth Wave 1 — sign-in code (magic-link short form), ceremony code entry, post-consent return path #174 — this PR doesn't touch auth logic).bun run typecheckclean (both runs) ·bun run test— 153 pass, 0 fail, 484 expect() calls (both runs).bunx wrangler deploy --dry-run(both the top-level/production and--env stagingconfigs) — confirmsenv.CF_VERSION_METADATAbinds cleanly as "Worker Version Metadata" in both scopes, no new config errors (the pre-existing "vars not inherited" warnings forCONSOLE_REDIRECT_HOST/BOUND_ORIGINS/VAULT_PUBLIC_ORIGIN/send_emailare unrelated and predate this PR — staging intentionally leaves those unset).bash -n scripts/deploy-staging.shandbun build scripts/smoke-staging.ts(syntax checks) — this PR can't run a real deploy-then-smoke cycle from here without actually deploying, which is exactly the step this fix protects; the next real staging deploy is the live proof.Files
workers/identity/wrangler.toml—[version_metadata]binding, top-level +[env.staging]workers/identity/src/env.ts—CF_VERSION_METADATAtypeworkers/identity/src/index.ts—/healthgainsversionworkers/identity/test/ops.test.ts— the/healthtest asserts the shape (not a hardcoded value — miniflare simulates its own fake id)scripts/deploy-staging.sh— the propagation pollscripts/smoke-staging.ts— blank-URL crash guards (both resume blocks)0.0.8-rc.93Not merging — same strict reviewer delta-verifies, then the ladder re-runs per the original ask.
https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB