fix(recovery): _recover rides out the wipe's 503 with bounded retry (v2.9.2)

[mabry1985]

Summary

• At a universe wipe, SpaceTraders 503s on /register for the first minutes while it rebuilds. _recover (the supervisor's on_crash) re-registered once, so that 503 made it return False and the supervisor stopped — stranding the fleet on a dead token.
• The supervisor gives on_crash exactly one shot per crash (its recovered latch won't re-invoke it until the runner runs again, and the runner can't run without a fresh token), so a single failed re-register is terminal.
• Observed live 2026-07-05: the engine stopped at the wipe and did not restart until a manual start ~24 min later. Registration only recovered via the agent's own out-of-band st_register retry — which does not restart the ops engine.

Fix

• Retry the re-register in place across the rebuild window (bounded backoff (0, 15, 30, 45, 60, 90, 90, 90, 90) ≈ 9 min; first try immediate so a server that's already back recovers with no wait).
• Give up (return False → supervisor stops the storm) only on a terminal failure — call sign claimed / no account token / no call sign — or after the budget.
• Cancellation-safe: an operator stop cancels at the asyncio.sleep checkpoint (CancelledError is BaseException, not caught by the except Exception around the re-register).

Test plan

• test_wipe_recovery.py (5 cases): transient-503-then-recover, claimed→terminal-no-retry, persistent-503→give-up-after-budget, fast-path-first-try, non-reset-crash-rekicks
• full suite: 331 passed; ruff clean

Follow-up (filed separately)

If recovery lands out-of-band (agent st_register) after the supervisor already stopped, nothing restarts the engine; and the one-shot on_crash is a host-level (protoAgent) limitation. Tracked for a separate change.

🤖 Generated with Claude Code

[protoquinn]

👀 Quinn is reviewing — verdict (PASS / WARN / FAIL) + findings to follow.

[protoquinn]

QA Audit — PR #52 | fix(recovery): _recover rides out the wipe's 503 with bounded retry (v2.9.2)

VERDICT: WARN (CI still settling — will auto-promote to APPROVED on terminal-green)

---

CI Status

• test: ⏳ in_progress

Diff Review

• fleet.py: _recover() gains a bounded retry loop (_RESET_REREGISTER_BACKOFF ≈ 9 min) around recover_from_reset(), riding out the server's post-wipe rebuild 503s. Terminal failures (claimed call sign, no account token, no call sign) abort immediately — no wasted retries.
• _reset_recovery_terminal(): clean substring matcher for the unrecoverable statuses.
• test_wipe_recovery.py: 5 focused cases covering transient-recovery, terminal-no-retry, budget-exhaustion, fast-path, and non-reset pass-through. 331 suite passed, ruff clean.
• Version bump 2.9.1 → 2.9.2 in protoagent.plugin.yaml.

Observations

• LOW: clawpatch structural pass (gateway) returned 0 findings across 7 mapped features — no cross-file hazards detected.
• LOW: except Exception around recover_from_reset() catches all non-BaseException types (including potential code bugs). Bounded retry (9 attempts) limits blast radius; _record logs each attempt for debugging. Acceptable for the acknowledged one-shot-per-crash constraint.
• No unresolved CodeRabbit threads. Clean diff, well-tested, addresses an observed live incident (2026-07-05 wipe).

Approve-on-green policy: once test completes terminal-green, this COMMENT review auto-promotes to APPROVED and the PR merges. No re-review needed unless the head moves.

— Quinn, QA Engineer

[protoquinn]

Submitted COMMENT review on protoLabsAI/spacetraders-plugin#52.