Skip to content

fix: doctor answers whether a drain would work - #55

Merged
aicayzer merged 1 commit into
mainfrom
fix/doctor-reports-drain-readiness
Sep 1, 2026
Merged

aicayzer merged 1 commit into
mainfrom
fix/doctor-reports-drain-readiness

Conversation

@aicayzer

@aicayzer aicayzer commented Sep 1, 2026

Copy link
Copy Markdown
Owner

"Would a drain refuse here?" is the question an operator has after every upgrade, and until now the only way to answer it was to attempt the drain — on a busy pool, with real jobs at stake. Checking the thing by doing the thing is exactly wrong when the thing is destructive if it misjudges.

doctor now reports, per pool, which loaded runners predate the graceful-shutdown setting:

INFO  marfa: runner(s) 1, 2, 3 started before graceful shutdown was available, so
      '--drain' will refuse on them. They run and take work normally; only
      draining is affected.
INFO  marfa: fix: 'runpool rewrite-agents', then let the pool cycle — the idle
      sweep does this on its own, or 'runpool down marfa' once it is idle.

A note, not a failure. Such a pool is running and taking work perfectly normally. Nothing is wrong until someone tries to drain it, and the drain refuses safely by itself. This exists so nobody has to find that out the hard way.

It reads the loaded agent's environment, not the plist on disk — the same distinction _rp_drain_pool relies on, and the one that mattered in practice at the 0.10.0 upgrade when both machines had correct plists on disk while every loaded runner still had the old behaviour.

Found immediately after releasing 0.10.0, when neither machine could be cycled and there was no read-only way to ask what state its runners were in.

Verified live against a real pool. Five new cases in tests/drain-guards.sh (28 total); bash -n and shellcheck --severity=warning clean, all four suites pass.

"Would a drain refuse here?" is the question an operator has after
every upgrade, and until now the only way to answer it was to attempt
the drain — on a busy pool, with real jobs at stake. Checking the thing
by doing the thing is exactly wrong when the thing is destructive if it
misjudges.

doctor now reports, per pool, which loaded runners predate the
graceful-shutdown setting. A note rather than a failure: such a pool
runs and takes work perfectly normally, and only --drain is affected,
which refuses safely on its own. Nothing is wrong; this exists so
nobody has to find that out the hard way.

Found immediately after the 0.10.0 release, when neither machine could
be cycled and there was no read-only way to ask what state its runners
were actually in.
@aicayzer
aicayzer merged commit 2d548c3 into main Sep 1, 2026
2 checks passed
@aicayzer
aicayzer deleted the fix/doctor-reports-drain-readiness branch September 1, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant