feat: stop autoscale waking a pool for a run that will never start - #71
Merged
Conversation
A run can enter queued and stay there permanently with no jobs attached. Autoscale counted it, woke the pool, found nothing, idled out and woke again, every twenty-one minutes for as long as the run existed. Nothing reported it, because a pool that wakes and stands down is behaving as designed. Known-fruitless runs are now subtracted from the queued count instead, so one dead run in one repository cannot blind an org pool to the other fourteen. A strike is earned only when the pool's started stamp has changed since that run was last judged, which is the only evidence that waking for it achieved nothing: a tick count would punish a pool that cannot start at all, and elapsed time would punish a laptop that slept. Reported by doctor and status, and once through the notifier. Closes #65
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #65.
The bug
A GitHub run can enter
queuedand stay there permanently with zero jobs attached._rp_autoscalecounted queued runs, so it woke the pool, found nothing to do, idled out afterRUNPOOL_IDLE_SECSand woke again. Observed in the wild: 29 wakes in a day, spaced 21 minutes apart through the night, against a normal five to eight, for two days.Nothing reported it, because a pool that wakes and stands down is behaving exactly as designed.
The rule
Known-fruitless runs are subtracted from the queued count, never suppressed as a pool. One dead run in one repository must not blind an org pool to the other fourteen, and every other queued run still wakes it normally.
A strike is earned only when
state/pools/<name>.startedhas changed since that run was last judged, which is the only available evidence that waking for it achieved nothing:A whole-queue fingerprint was the first design and does not survive review: on a busy org the set changes whenever any other run arrives or completes, so a stuck run never accrues consecutive evidence.
Suppression clears itself when the run leaves the queued set, and there is a daily re-arm worth exactly one wake so a run held by something transient, an upstream concurrency group being the realistic case, cannot be held for ever.
Surface
doctorwarns, naming the run, its age and its job count, with the cancel command.statussays how many runs a pool is ignoring, in both the table and--json, and in--localtoo since the state is local.warning, keyed on the run rather than the pool so a receiver deduping per pool cannot swallow the next one.RUNPOOL_STUCK_WAKES(default 3,0disables), through all four sites of the precedence block.Also here
_rp_queued_runsdiscards the response body on failure.ghprints GitHub's error JSON to stdout, so a watch list naming a repository that has been renamed or made private used to hand the caller a line beginning{"message":"Not Found"— which reached an arithmetic expansion and aborted it. Covered by a test case.The API call count per pool is unchanged; only the page size and the
--jqdiffer.Verification
tests/stuck-queue-guard.shis new, offline, and covers 25 cases: the pure rule (including the unchanged-started-stamp regression and the org held-plus-new case) and one autoscale loop end to end withghstubbed, asserting three wakes then silence, that a held pool still costs exactly one call, and that the notification fires once carrying the run id.bash -n,shellcheck --severity=warningand all eight tests pass.