Skip to content

fix(identity,ci): rc.92's failed live smoke was a stale-edge-propagation race, not a code bug — deploy-staging.sh now verifies - #175

Merged
unforced merged 1 commit into
mainfrom
auth-w1-fix
Jul 17, 2026
Merged

fix(identity,ci): rc.92's failed live smoke was a stale-edge-propagation race, not a code bug — deploy-staging.sh now verifies#175
unforced merged 1 commit into
mainfrom
auth-w1-fix

Conversation

@unforced

Copy link
Copy Markdown
Contributor

What actually happened (evidence, not the original hypothesis)

The triage that assigned this fix hypothesized a D1/miniflare-vs-real-D1 divergence in the code-mint path. That hypothesis is wrong, and I want to say so plainly before the fix, because the correct fix depends on the correct diagnosis:

createMagicLink's 6-digit code is generated by randomNumericCode() in memory, before the D1 INSERT even runs — its value cannot depend on any D1 return-shape/RETURNING/.meta behavior, real or simulated. There is no code path in auth-handlers.ts where the link header is set and the code header isn't, for a given deployed version — both are written from the same extra object in the same if (deps.exposeDevLinks) block.

What I found instead, from gh run view 29575426905 --log (the failing deploy-staging.yml run) and a live probe against staging:

  • wrangler deploy --env staging for the identity worker reported success at 11:02:07.54 ("Deployed parachute-identity-staging triggers", Current Version ID 5dfc52f1-…).
  • The smoke's first (pre-existing, unchanged) magic-link assertions passed at 11:02:24.4–24.9.
  • The smoke's new step — checking for x-parachute-dev-magic-code — FAILED at 11:02:25.68, code null. The very next step, POST /auth/code, got 404 — the exact symptom of a route that doesn't exist yet on an OLD worker version, since /auth/code is brand new in this PR.
  • I curled POST /auth/magic against live staging myself (minutes later, well past propagation): x-parachute-dev-magic-code: 076496 came back correctly, alongside the link header, on the first try. I then walked the full code-verify flow live — verify-by-code, session mint, the link dying once the code consumed the shared row, a wrong-code neutral failure — all correct.

This is the exact signature of the smoke request landing on an edge PoP that hadn't yet picked up the new Worker version (wrangler deploy reporting success only means the upload was accepted, not that every PoP is serving it — the smoke ran ~18s later and still lost the race). Nothing about the auth-code logic itself needed to change; #174's code was correct as shipped.

A second, real bug this incident exposed (unrelated to the propagation race, but a genuine defect): the new resume-by-code smoke block called fetch(rcLoc, …) where rcLoc could be "" after an earlier step failed — Bun's fetch() throws TypeError: fetch() URL must not be a blank string on that, uncaught, which killed the entire remaining ~900-line smoke suite (TOTP, billing, GFS snapshots, voice transcription, semantic search — everything downstream never ran). The pre-existing resume-by-link block has the identical latent pattern (never triggered before because nothing it touches was ever a brand-new route).

What live smoke proves that vitest cannot (per the ask)

vitest-pool-workers' 796 green tests prove the logic is correct against a Worker that is unambiguously the code under test, invoked in-process — there is no "which version answered this request" question inside a single vitest run. What only a live, freshly-deployed staging smoke can prove, and what this fix specifically targets:

  1. That the deployed edge is actually serving the code being tested — the exact thing that silently wasn't true for ~18+ seconds after the rc.92 deploy. No unit/integration test can catch a deploy-propagation race; it doesn't exist until code is actually deployed to real edge infrastructure.
  2. That a brand-new route is reachable globally, not just at the one PoP wrangler deploy's own request happened to hit.
    3** That the whole ~900-line smoke suite survives a real failure without cascading — the blank-URL crash only manifests when an earlier live HTTP round-trip genuinely comes back unexpected; vitest's mocked/in-process requests never hit this failure mode because nothing in that environment 404s a route that exists in the same test run.

The fix

  • GET /health now reports env.CF_VERSION_METADATA.id — Workers' native version-metadata binding ([version_metadata] in wrangler.toml, both scopes; confirmed via the Cloudflare docs before wiring it in, not assumed). Env.CF_VERSION_METADATA is optional so bare/test configs still type-check; /health's version field degrades to null when unbound.
  • deploy-staging.sh polls /health against a pre-deploy baseline (up to 90s, 3s interval) after the identity worker's wrangler deploy --env staging, before ever handing off to smoke-staging.ts. Times out loudly (exit 1, clear message) rather than silently racing a possibly-stale edge — the CI failure mode flips from "cryptic cascade of unrelated-looking FAILs" to "deploy script explicitly says propagation didn't complete in time."
  • deploy-prod.sh is completely untouched — it's manual-approval-gated (GitHub Environment required-reviewer rule) so there's no equivalent time pressure, and the ask was explicitly "do not touch prod anything." The [version_metadata] binding IS added to the wrangler.toml top-level scope too (since the identity worker's code — including /health — is shared between environments; there's no way to make staging's /health version-aware without production's config also carrying the binding), but that's an inert additive config/response-shape change, not an active deploy-time behavior change, and it only takes effect through the normal gated prod-deploy process.
  • Smoke robustness: both the resume-by-link and resume-by-code blocks now check for a blank resume URL before fetching it, fail()-ing that one assertion instead of throwing and aborting the suite.

Gates (2 runs each, stable — after clearing ~161 stray workerd processes left over from repeated earlier local runs, which caused one transient false-negative I want to be upfront about rather than paper over)

  • workers/identity: bun run typecheck clean (both runs) · bun x vitest run33 files, 796 tests passed (both runs, unchanged from feat(identity): auth Wave 1 — sign-in code (magic-link short form), ceremony code entry, post-consent return path #174 — this PR doesn't touch auth logic).
  • Root: bun run typecheck clean (both runs) · bun run test153 pass, 0 fail, 484 expect() calls (both runs).
  • bunx wrangler deploy --dry-run (both the top-level/production and --env staging configs) — confirms env.CF_VERSION_METADATA binds cleanly as "Worker Version Metadata" in both scopes, no new config errors (the pre-existing "vars not inherited" warnings for CONSOLE_REDIRECT_HOST/BOUND_ORIGINS/VAULT_PUBLIC_ORIGIN/send_email are unrelated and predate this PR — staging intentionally leaves those unset).
  • bash -n scripts/deploy-staging.sh and bun build scripts/smoke-staging.ts (syntax checks) — this PR can't run a real deploy-then-smoke cycle from here without actually deploying, which is exactly the step this fix protects; the next real staging deploy is the live proof.

Files

  • workers/identity/wrangler.toml[version_metadata] binding, top-level + [env.staging]
  • workers/identity/src/env.tsCF_VERSION_METADATA type
  • workers/identity/src/index.ts/health gains version
  • workers/identity/test/ops.test.ts — the /health test asserts the shape (not a hardcoded value — miniflare simulates its own fake id)
  • scripts/deploy-staging.sh — the propagation poll
  • scripts/smoke-staging.ts — blank-URL crash guards (both resume blocks)
  • Version bumped to 0.0.8-rc.93

Not merging — same strict reviewer delta-verifies, then the ladder re-runs per the original ask.

https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB

…ion race, not a code bug — deploy-staging.sh now verifies

The rc.92 auth-Wave-1 deploy's smoke ran ~18s after `wrangler deploy` reported
success and hit an edge PoP still serving the OLD worker version: a brand-new
route (/auth/code) 404'd, a brand-new response header was simply absent.
Byte-identical to a real code bug until the deploy log timestamps are read
against the failure timestamps — confirmed live: a manual probe against
staging minutes later showed the code mint working correctly, and wrangler's
own "Current Version ID" output didn't match what /health was reporting at
smoke time.

`wrangler deploy` returning success means the upload was ACCEPTED, not that
every edge PoP is serving it yet. Fixed at the root: GET /health now reports
env.CF_VERSION_METADATA.id (Workers' native version-metadata binding, wired
in both wrangler.toml scopes); deploy-staging.sh polls it against a
pre-deploy baseline (up to 90s) before ever invoking smoke, failing loudly
instead of racing a stale edge. deploy-prod.sh is untouched (manual-approval
gated, no equivalent race).

Also fixes a real robustness bug the incident exposed: two resume-by-code/
resume-by-link smoke steps called fetch() with a blank URL when an earlier
step had already failed, throwing an UNCAUGHT exception that killed the
entire remaining suite (TOTP, billing, GFS snapshots, voice, semantic
search — everything downstream of the failure). Both are now guarded:
a failed step stays one failed assertion, never a process crash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
@unforced
unforced merged commit bf80554 into main Jul 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant