feat: runpool doctor, and queue time in stats - #35
Merged
Conversation
Answering that today means running status, reading the log, and already knowing where to look for the parts neither of those covers. One of the checks is covered by nothing at all: `schedule install` writes the tick and clean agents and nothing ever looks at them again, so an unloaded tick agent means no pool autoscales, every job waits for a manual `up`, and `status` reports every pool as perfectly healthy — because locally they are. `doctor` runs the lot in one pass and prints a remedy against each finding: gh and its authentication, the scheduler agents, per-pool registrations and launch agents, org pools with no watch list, disk headroom against RUNPOOL_BASE, the config's permissions, and the organisation's runner-group setting. Non-zero exit when something is actually wrong. Strictly read-only, and that is a boundary rather than a preference: a diagnostic that repairs is one nobody can run safely while confused, and every repair already exists as its own command. It lives beside _rp_status in lib/scheduler.sh for the same reason. Two things are shared rather than restated. The registration judgement moves into _rp_gh_state, which status now renders as its note and doctor as a check, so the two cannot drift from each other or from how GitHub behaves. The organisation runner-group read moves into lib/common.sh as _rp_org_allows_public, returning true, false or unknown — register consults it once when a pool is created, and the setting can be switched on the day after. Reported and never re-derived: doctor does not enumerate an organisation's public repositories, for the reasons in SECURITY.md. Closes #33
Queue time is the figure that answers "do I need more runners", because more runners help if and only if work is waiting. Duration says what a job costs and load says how contended the machine was; neither moves when capacity changes. The job hook fires when a runner picks a job up, so the whole wait before that moment is invisible locally and only GitHub knows it — which left the one number worth acting on reachable solely as a TSV out of contrib/telemetry-join.sh. `runpool stats --queue` prints its median and p90 per workflow / job, in the shape the duration table already uses. Behind a flag on purpose. This file reads only local files and the join makes one gh api call per unique run; a network fan-out behind a command people run casually is the failure `status --json --local` exists to prevent. contrib/telemetry-join.sh is invoked rather than absorbed. It carries two traps that took working out — run id plus runner name matches several API jobs and needs the nearest-start tie-break, and created_at is run-level rather than job-level — and two copies is two places to get them wrong. The telemetry path is passed as $1, so the script does not fall back to re-entering whatever runpool happens to be on PATH. The qualifier ships with the number, every time. queue_s conflates a cold pool waking with an unfinished dependency with a genuine shortage of runners, and only the last is fixed by capacity. This file exists in its present form because a canned analysis already shipped a confidently wrong answer on exactly that, and a bare "median queue 47s" is the same blind spot with a new number on it. Closes #34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two reporting commands, in one branch because they share
bin/runpool's dispatch and help, the README command table andskills/runpool/SKILL.md. Neither changes anything on a machine.Closes #33, closes #34.
runpool doctor(#33)One read-only command for "why is nothing picking this up". Lives beside
_rp_statusinlib/scheduler.sh, because it reports and changes nothing and that is the file's concern.The check that earns it: nothing today looks at the tick and clean agents after
schedule installwrites them. If the tick agent is not loaded, no pool autoscales, every job waits for a manualrunpool up, andstatusreports every pool as perfectly healthy — because locally they are. The rest are each visible somewhere and never together:ghauthentication (_rp_requirecovers presence only), registrations, per-pool launch agents, org pools with an empty watch list, disk headroom againstRUNPOOL_BASErather than/, the config's permissions, and the organisation's runner-group setting.Every failure names its remedy. Exit is non-zero when something is actually wrong, zero otherwise; warnings do not fail it.
Two things are shared rather than restated.
_rp_gh_statenow holds the one judgement about a pool's registration.statusrenders it as its note anddoctoras a check with a fix, so a second copy cannot drift from the first or from how GitHub actually behaves.status's output is byte-identical before and after._rp_org_allows_publicmoves the runner-group read intolib/common.sh, returningtrue,falseorunknown—unknowndeliberately not folded intofalse, since reading it needsadmin:org.registerconsults it once at create; the setting can be switched on the day after. Reported and never re-derived: no enumeration of an organisation's public repositories, perSECURITY.md.Strictly read-only, and a boundary rather than a preference: a diagnostic that repairs is one nobody can run safely while confused, and every repair already exists as its own command. The config's mode is checked and the pools file's is not, because
install.shomits thatchmodon purpose.runpool stats --queue(#34)Median and p90 of the wait before a runner picked each job up, per
workflow / job, in the shape the duration table already uses.stats.lib/stats.shreads only local files; the join makes onegh apicall per unique run. A network fan-out behind a casually-run command is the failurestatus --json --localexists to prevent.contrib/telemetry-join.shis invoked, not absorbed. It carries two traps that took working out — the run-id fan-out with its nearest-start tie-break, andcreated_atbeing run-level rather than job-level — and two copies is two places to get them wrong. The telemetry path is passed as$1, so the script does not fall back to re-entering whateverrunpoolis onPATH.queue_sconflates a cold pool waking with an unfinished dependency with a genuine shortage of runners, and only the last is fixed by capacity. This file exists in its present form because a canned analysis already shipped a confidently wrong answer on exactly that.Verification
/bin/bash -nunder/bin/bash3.2.57, and shellcheck--severity=warningvia the Docker image inCONTRIBUTING.md. Both clean.doctorwatched failing, not only passing. Against a scratchRUNPOOL_BASE,RUNPOOL_LOG_DIRandRUNPOOL_POOLS_FILE, with hand-written pool configs and no registration, resize or removal anywhere: paused, no pools, an incomplete pool config, missing launch agent plists, an org pool with no watch list, a pool GitHub has no runners for, an unloaded tick agent,ghabsent,ghpresent but unauthenticated, every API call failing, a config at mode 644, and — on two small attached volumes — the sub-5GB and sub-20GB disk branches. Both exit codes confirmed. Theofflineclassification was exercised with a cannedghreturning registered-but-none-online.--queuecross-checked againstcontrib/telemetry-join.shon the same 400-record subset: 198 joined rows, 15 job types, and every median and p90 agrees exactly with the quantiles computed independently from the raw TSV. Two runs produced identical tables.statusand defaultstatsdiffed againstmainon the same fixtures: identical apart from the intended pointer to--queue.