Supervisor: on_crash is one-shot per crash — plugins must re-implement bounded recovery retry

[mabry1985]

Observation

In graph/supervisor.py _watch, on_crash is invoked at most once per crash: the recovered latch is set before the call and only reset when the runner successfully runs again (recovered = False on the running branch). So if on_crash returns True but the re-kicked runner crashes again for the same reason, on_crash is not called a second time — the supervisor just re-kicks the runner blindly.

if self._on_crash is not None and not recovered:
    recovered = True
    handled = bool(await _maybe_await(self._on_crash(self._result)))
    if not handled:
        self._want = False   # stop
    ...
rekicks += 1
self._spawn_runner()         # re-kick WITHOUT calling on_crash again

Why it matters

For recovery that needs several attempts over minutes — e.g. a SpaceTraders universe wipe: the API 503s on re-register until the universe finishes rebuilding — a plugin's on_crash gets a single shot. If that one attempt fails, returning True just re-kicks a runner that can't succeed (dead token), and returning False stops the engine. Neither is right.

The plugin worked around this by doing its own bounded in-place retry loop inside on_crash (blocking it for up to ~9 min): protoLabsAI/spacetraders-plugin#52. That works but blocks the watchdog and forces every plugin to re-implement backoff.

Proposal

Offer a first-class retry policy so on_crash can be re-invoked with backoff instead of being latched off:

• e.g. on_crash may return a sentinel/RETRY (or a (handled, retry_after) tuple), and the supervisor re-calls it after retry_after up to a bound; OR
• an on_crash_retries / on_crash_backoff kwarg on supervise() that re-invokes on_crash on repeated same-cause crashes rather than re-kicking the runner blindly.

Either keeps the retry cadence in the supervisor (cancellable, observable) instead of a multi-minute blocking sleep inside each plugin's on_crash.

Repro / reference

protoLabsAI/spacetraders-plugin#52 (live 2026-07-05 wipe: engine stopped, did not auto-restart).