smoke-staging: snapshot live restore round-trip times out (2 consecutive runs, 2026-07-17)

[unforced]

Runs 29549592082 (c6da9da) and 29552948228 (531670e) both failed ONLY on:

FAIL snapshots: live restore round-trip threw (non-fatal — sections continue) — TimeoutError: The operation timed out.

Last green: run 29545169744 (2026-07-17T00:34Z). The two failing runs bracket commits (#163 my. routing config, #165 SPA pin bump) that touch nothing in the snapshot/restore path, and #163's changes are prod-only routes (staging routes = []), so this looks environmental, not a code regression.

Suspects, in order:

1. Debris accumulation — staging is at ~140 smoke-created vaults (usage: vaults=140); the restore round-trip (snapshot trigger → R2 → new-vault DO import) may have crossed the smoke's client-side fetch timeout as the fleet grew.
2. Staging DO/R2 latency at this hour.

Worth doing: bump the smoke's timeout budget for the restore section (it's a genuinely heavy round-trip), and/or add a staging debris-sweep (delete arrive-*/tierbox-* vaults older than N days) so the environment stays representative.

Non-blocking for releases whose diffs don't touch snapshots — but a red staging gate masks real failures, so it should not stay red.

[unforced]

Opened PR #173 with the fix — bundled with the C2 semantic live-smoke coverage since both live in scripts/smoke-staging.ts territory.

Fix applied:

1. scripts/smoke-staging.ts §17 (the restore round trip) — every fetch in the section now carries an explicit AbortSignal.timeout(120_000) (2 min), replacing whatever implicit default the client was hitting. This is a generous, self-documenting budget for what really is a heavy round trip (a full DO snapshot sweep across every staging vault, then an R2→new-DO tar import).
2. New scripts/staging-sweep.ts — an operator-run debris sweep. Deletes the D1 vaults ownership row for vaults matching the smoke's own throwaway prefixes (box-, imp-, arrive-, tierbox-, mockbox-, enforcebox-) older than a configurable cutoff (default 1 day). Dry-run by default; pass --yes to apply. Never wired into CI/deploy — pure manual lever. Structurally staging-only: unlike backfill-plans.ts/set-operator-role.ts, there is no TARGET=prod escape hatch at all — the D1 database name is hardcoded to staging's, plus a defense-in-depth runtime assertion. Never touches the literal demo vault regardless of age.

Known limitation, documented in the script's header: there is no vault-teardown verb anywhere in this codebase (DELETE /account/vaults/:name is still a 501 not_implemented stub, and the vault DO's internal-verb table has no delete/wipe entry) — so the sweep only frees the D1 ownership row. The underlying Durable Object storage + any R2 attachments are left orphaned, not reclaimed. That's an acceptable trade for what this addresses (the debris count, which is what the nightly snapshot sweep and the smoke's own restore round trip both iterate over) — a hibernated, unowned, orphaned DO bills near-zero storage. A real teardown verb is separate, larger work; flagging it here in case it's worth its own follow-up issue once the orphaned storage itself becomes a concern.

Once this merges and deploys, running bun scripts/staging-sweep.ts --yes should clear the current ~140-vault debris back down.

[unforced]

Sweep executed 2026-07-17 ~10:40 UTC: 98 aged debris vaults deleted (prefix-matched, >1 day old, demo protected). Staging fleet ~140 → ~40. Combined with rc.91's 120s timeout budget, the next few staging runs tell us whether debris was the cause — if the restore round-trip stays green for 3 consecutive runs, close this.

[unforced]

Third data point (run 29577056445, post-sweep, post-120s-budget): the restore round-trip STILL times out with the fleet at ~40 vaults and a 120s AbortSignal. The debris hypothesis is dead — the operation genuinely exceeds 2 minutes on staging. This is now a product-latency signal, not test noise: a paid user's restore would feel this too. Next step: instrument the restore path (snapshot fetch → target-DO import replay) with timing marks to find which leg eats the time; suspect the import replay under workerd CPU limits. De-prioritize timeout-raising — measure first.