Skip to content

feat: runpool doctor, and queue time in stats - #35

Merged
aicayzer merged 2 commits into
mainfrom
feat/doctor-and-queue-time
Aug 21, 2026
Merged

feat: runpool doctor, and queue time in stats#35
aicayzer merged 2 commits into
mainfrom
feat/doctor-and-queue-time

Conversation

@aicayzer

@aicayzer aicayzer commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Two reporting commands, in one branch because they share bin/runpool's dispatch and help, the README command table and skills/runpool/SKILL.md. Neither changes anything on a machine.

Closes #33, closes #34.

runpool doctor (#33)

One read-only command for "why is nothing picking this up". Lives beside _rp_status in lib/scheduler.sh, because it reports and changes nothing and that is the file's concern.

The check that earns it: nothing today looks at the tick and clean agents after schedule install writes them. If the tick agent is not loaded, no pool autoscales, every job waits for a manual runpool up, and status reports every pool as perfectly healthy — because locally they are. The rest are each visible somewhere and never together: gh authentication (_rp_require covers presence only), registrations, per-pool launch agents, org pools with an empty watch list, disk headroom against RUNPOOL_BASE rather than /, the config's permissions, and the organisation's runner-group setting.

Every failure names its remedy. Exit is non-zero when something is actually wrong, zero otherwise; warnings do not fail it.

Two things are shared rather than restated.

  • _rp_gh_state now holds the one judgement about a pool's registration. status renders it as its note and doctor as a check with a fix, so a second copy cannot drift from the first or from how GitHub actually behaves. status's output is byte-identical before and after.
  • _rp_org_allows_public moves the runner-group read into lib/common.sh, returning true, false or unknownunknown deliberately not folded into false, since reading it needs admin:org. register consults it once at create; the setting can be switched on the day after. Reported and never re-derived: no enumeration of an organisation's public repositories, per SECURITY.md.

Strictly read-only, and a boundary rather than a preference: a diagnostic that repairs is one nobody can run safely while confused, and every repair already exists as its own command. The config's mode is checked and the pools file's is not, because install.sh omits that chmod on purpose.

runpool stats --queue (#34)

Median and p90 of the wait before a runner picked each job up, per workflow / job, in the shape the duration table already uses.

  • Behind a flag, not in default stats. lib/stats.sh reads only local files; the join makes one gh api call per unique run. A network fan-out behind a casually-run command is the failure status --json --local exists to prevent.
  • contrib/telemetry-join.sh is invoked, not absorbed. It carries two traps that took working out — the run-id fan-out with its nearest-start tie-break, and created_at being run-level rather than job-level — and two copies is two places to get them wrong. The telemetry path is passed as $1, so the script does not fall back to re-entering whatever runpool is on PATH.
  • The qualifier ships with the number, every time. queue_s conflates a cold pool waking with an unfinished dependency with a genuine shortage of runners, and only the last is fixed by capacity. This file exists in its present form because a canned analysis already shipped a confidently wrong answer on exactly that.

Verification

  • /bin/bash -n under /bin/bash 3.2.57, and shellcheck --severity=warning via the Docker image in CONTRIBUTING.md. Both clean.
  • doctor watched failing, not only passing. Against a scratch RUNPOOL_BASE, RUNPOOL_LOG_DIR and RUNPOOL_POOLS_FILE, with hand-written pool configs and no registration, resize or removal anywhere: paused, no pools, an incomplete pool config, missing launch agent plists, an org pool with no watch list, a pool GitHub has no runners for, an unloaded tick agent, gh absent, gh present but unauthenticated, every API call failing, a config at mode 644, and — on two small attached volumes — the sub-5GB and sub-20GB disk branches. Both exit codes confirmed. The offline classification was exercised with a canned gh returning registered-but-none-online.
  • --queue cross-checked against contrib/telemetry-join.sh on the same 400-record subset: 198 joined rows, 15 job types, and every median and p90 agrees exactly with the quantiles computed independently from the raw TSV. Two runs produced identical tables.
  • status and default stats diffed against main on the same fixtures: identical apart from the intended pointer to --queue.

Answering that today means running status, reading the log, and already
knowing where to look for the parts neither of those covers. One of the
checks is covered by nothing at all: `schedule install` writes the tick
and clean agents and nothing ever looks at them again, so an unloaded
tick agent means no pool autoscales, every job waits for a manual `up`,
and `status` reports every pool as perfectly healthy — because locally
they are.

`doctor` runs the lot in one pass and prints a remedy against each
finding: gh and its authentication, the scheduler agents, per-pool
registrations and launch agents, org pools with no watch list, disk
headroom against RUNPOOL_BASE, the config's permissions, and the
organisation's runner-group setting. Non-zero exit when something is
actually wrong.

Strictly read-only, and that is a boundary rather than a preference: a
diagnostic that repairs is one nobody can run safely while confused, and
every repair already exists as its own command. It lives beside
_rp_status in lib/scheduler.sh for the same reason.

Two things are shared rather than restated. The registration judgement
moves into _rp_gh_state, which status now renders as its note and doctor
as a check, so the two cannot drift from each other or from how GitHub
behaves. The organisation runner-group read moves into lib/common.sh as
_rp_org_allows_public, returning true, false or unknown — register
consults it once when a pool is created, and the setting can be switched
on the day after.

Reported and never re-derived: doctor does not enumerate an
organisation's public repositories, for the reasons in SECURITY.md.

Closes #33
Queue time is the figure that answers "do I need more runners", because
more runners help if and only if work is waiting. Duration says what a
job costs and load says how contended the machine was; neither moves
when capacity changes. The job hook fires when a runner picks a job up,
so the whole wait before that moment is invisible locally and only
GitHub knows it — which left the one number worth acting on reachable
solely as a TSV out of contrib/telemetry-join.sh.

`runpool stats --queue` prints its median and p90 per workflow / job, in
the shape the duration table already uses.

Behind a flag on purpose. This file reads only local files and the join
makes one gh api call per unique run; a network fan-out behind a command
people run casually is the failure `status --json --local` exists to
prevent.

contrib/telemetry-join.sh is invoked rather than absorbed. It carries two
traps that took working out — run id plus runner name matches several API
jobs and needs the nearest-start tie-break, and created_at is run-level
rather than job-level — and two copies is two places to get them wrong.
The telemetry path is passed as $1, so the script does not fall back to
re-entering whatever runpool happens to be on PATH.

The qualifier ships with the number, every time. queue_s conflates a cold
pool waking with an unfinished dependency with a genuine shortage of
runners, and only the last is fixed by capacity. This file exists in its
present form because a canned analysis already shipped a confidently
wrong answer on exactly that, and a bare "median queue 47s" is the same
blind spot with a new number on it.

Closes #34
@aicayzer
aicayzer merged commit 648e289 into main Aug 21, 2026
2 checks passed
@aicayzer
aicayzer deleted the feat/doctor-and-queue-time branch August 21, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant