The root design finding from the nine-day staging outage. #218 fixes the immediate cause; this closes the door on it recurring.
What actually happened, verified against git
438ba40^ is 9508dfa, the last green staging run (2026-07-17T06:33).- The restore round-trip and the fleet sweep both existed at
9508dfa and that run was green — marker counts and 'snapshots:' assertion counts are identical across 438ba40^ -> 438ba40 (8/8), and runSweep appears in both.
438ba40 added RESTORE_ROUNDTRIP_TIMEOUT_MS = 120_000 + AbortSignal.timeout(...) to a section that was already slow but still completing. The staging fleet had grown (167 -> 260), the sweep exceeded 120s, and it aborted at the first awaited call in the section.
The defect
runSweep() and the seven restore-round-trip assertions live in one shared try block (smoke-staging.ts ~1415-1493), with the sweep first. So an abort in the sweep skips every assertion behind it:
- sweep answers 200 with a summary · 2. sweep took >=1 snapshot (not capped) · 3. console History lists the restore point + restore door · 4. restore POST -> 302 creates a new vault · 5. success notice renders with the attachments caveat · 6. owner mints a token for the RESTORED vault · 7. restored vault round-trips 7 notes verbatim
Those seven were passing on 2026-07-17 and have not executed since. A timeout added to a shared block silently disabled seven green assertions, and the gate reported a single failure line either way.
Why #218 alone isn't sufficient
#218 makes the sweep O(1) so it shouldn't time out — the fix relies on the sweep never throwing. If it ever does again (R2 hiccup, cold DO, future growth), the restore round-trip goes dark exactly as before, except now the run is green with an advisory rather than red. The blinding mechanism is untouched.
Fix
Split into two independent try blocks: the sweep in its own advisory-eligible block, the restore round-trip in a separate one. A sweep hiccup then cannot take the restore contract down with it. Same treatment is worth auditing for the other six wrapped sections — any place a setup call and a contract assertion share a block has the same shape.
The root design finding from the nine-day staging outage. #218 fixes the immediate cause; this closes the door on it recurring.
What actually happened, verified against git
438ba40^is9508dfa, the last green staging run (2026-07-17T06:33).9508dfaand that run was green — marker counts and'snapshots:'assertion counts are identical across438ba40^->438ba40(8/8), andrunSweepappears in both.438ba40addedRESTORE_ROUNDTRIP_TIMEOUT_MS = 120_000+AbortSignal.timeout(...)to a section that was already slow but still completing. The staging fleet had grown (167 -> 260), the sweep exceeded 120s, and it aborted at the first awaited call in the section.The defect
runSweep()and the seven restore-round-trip assertions live in one sharedtryblock (smoke-staging.ts~1415-1493), with the sweep first. So an abort in the sweep skips every assertion behind it:Those seven were passing on 2026-07-17 and have not executed since. A timeout added to a shared block silently disabled seven green assertions, and the gate reported a single failure line either way.
Why #218 alone isn't sufficient
#218 makes the sweep O(1) so it shouldn't time out — the fix relies on the sweep never throwing. If it ever does again (R2 hiccup, cold DO, future growth), the restore round-trip goes dark exactly as before, except now the run is green with an advisory rather than red. The blinding mechanism is untouched.
Fix
Split into two independent
tryblocks: the sweep in its own advisory-eligible block, the restore round-trip in a separate one. A sweep hiccup then cannot take the restore contract down with it. Same treatment is worth auditing for the other six wrapped sections — any place a setup call and a contract assertion share a block has the same shape.