Skip to content

feat: stop autoscale waking a pool for a run that will never start - #71

Merged
aicayzer merged 1 commit into
mainfrom
feat/stuck-queue-guard
Sep 5, 2026
Merged

feat: stop autoscale waking a pool for a run that will never start#71
aicayzer merged 1 commit into
mainfrom
feat/stuck-queue-guard

Conversation

@aicayzer

@aicayzer aicayzer commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Closes #65.

The bug

A GitHub run can enter queued and stay there permanently with zero jobs attached. _rp_autoscale counted queued runs, so it woke the pool, found nothing to do, idled out after RUNPOOL_IDLE_SECS and woke again. Observed in the wild: 29 wakes in a day, spaced 21 minutes apart through the night, against a normal five to eight, for two days.

Nothing reported it, because a pool that wakes and stands down is behaving exactly as designed.

The rule

Known-fruitless runs are subtracted from the queued count, never suppressed as a pool. One dead run in one repository must not blind an org pool to the other fourteen, and every other queued run still wakes it normally.

A strike is earned only when state/pools/<name>.started has changed since that run was last judged, which is the only available evidence that waking for it achieved nothing:

  • a tick count would give three strikes in three minutes to a pool that cannot start at all, then refuse it real work once its agents were repaired;
  • elapsed time would punish a laptop that slept overnight with work genuinely queued.

A whole-queue fingerprint was the first design and does not survive review: on a busy org the set changes whenever any other run arrives or completes, so a stuck run never accrues consecutive evidence.

Suppression clears itself when the run leaves the queued set, and there is a daily re-arm worth exactly one wake so a run held by something transient, an upstream concurrency group being the realistic case, cannot be held for ever.

Surface

  • doctor warns, naming the run, its age and its job count, with the cancel command.
  • status says how many runs a pool is ignoring, in both the table and --json, and in --local too since the state is local.
  • The notifier fires once per hold at warning, keyed on the run rather than the pool so a receiver deduping per pool cannot swallow the next one.
  • RUNPOOL_STUCK_WAKES (default 3, 0 disables), through all four sites of the precedence block.

Also here

_rp_queued_runs discards the response body on failure. gh prints GitHub's error JSON to stdout, so a watch list naming a repository that has been renamed or made private used to hand the caller a line beginning {"message":"Not Found" — which reached an arithmetic expansion and aborted it. Covered by a test case.

The API call count per pool is unchanged; only the page size and the --jq differ.

Verification

tests/stuck-queue-guard.sh is new, offline, and covers 25 cases: the pure rule (including the unchanged-started-stamp regression and the org held-plus-new case) and one autoscale loop end to end with gh stubbed, asserting three wakes then silence, that a held pool still costs exactly one call, and that the notification fires once carrying the run id.

bash -n, shellcheck --severity=warning and all eight tests pass.

A run can enter queued and stay there permanently with no jobs attached.
Autoscale counted it, woke the pool, found nothing, idled out and woke
again, every twenty-one minutes for as long as the run existed. Nothing
reported it, because a pool that wakes and stands down is behaving as
designed.

Known-fruitless runs are now subtracted from the queued count instead,
so one dead run in one repository cannot blind an org pool to the other
fourteen. A strike is earned only when the pool's started stamp has
changed since that run was last judged, which is the only evidence that
waking for it achieved nothing: a tick count would punish a pool that
cannot start at all, and elapsed time would punish a laptop that slept.

Reported by doctor and status, and once through the notifier.

Closes #65
@aicayzer
aicayzer merged commit 08918c7 into main Sep 5, 2026
2 checks passed
@aicayzer
aicayzer deleted the feat/stuck-queue-guard branch September 5, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autoscale wakes forever for a run that will never start

1 participant