diff --git a/AGENTS.md b/AGENTS.md index 0a86970..592c874 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,9 +27,10 @@ bin/runpool the executable and its dispatcher lib/common.sh config, logging, pool loading, launch agents, deregistration lib/lifecycle.sh register, set-count, up, down, reregister, remove lib/apply.sh the pools file, and reconciling the machine to it -lib/scheduler.sh status, autoscale, sweep, clean, schedule +lib/scheduler.sh status, doctor, autoscale, sweep, clean, schedule lib/notify.sh the optional notifier hook and what triggers it -lib/stats.sh job durations from recorded telemetry +lib/stats.sh job durations from recorded telemetry, and queue times + via contrib/telemetry-join.sh contrib/ optional pieces the user opts into: job hook, webhook notifier, demo status fixture skills/runpool/ agent skill for *using* runpool, shipped with the tool diff --git a/README.md b/README.md index dc06397..ae4347f 100644 --- a/README.md +++ b/README.md @@ -58,16 +58,19 @@ The first job after a quiet spell waits about a minute for its pool to come up. | `apply [--dry-run] [--file PATH]` | Reconcile the machine to a file describing its pools | | `up` / `down ` | Bring a pool online, or stand it down | | `status [--json]` | Local state alongside what GitHub actually sees | +| `doctor` | Why is nothing picking this up. Reports; changes nothing | | `pools` | List registered pools | | `reregister ` | Recreate GitHub registrations, keeping the local install | | `remove ` | Deregister and delete a pool | | `clean [pool]` | Prune work directories, temp, diagnostics, old binaries, caches | -| `stats` | What jobs actually cost, from recorded telemetry | +| `stats [--queue]` | What jobs actually cost, from recorded telemetry. `--queue` adds the wait before each job started | | `pause` / `resume` | Global kill switch | | `schedule install\|remove` | The background agents that drive everything above | **`status --json --local` skips the GitHub query**, reporting those fields as `null`. Anything refreshing on a timer should use it: one API call per pool per minute is thousands a day, and it makes a passive readout fail whenever the network does. +**`doctor` answers "why is nothing picking this up" in one command.** It checks `gh` and its authentication, that GitHub still has the registrations, that the launch agents exist — including the tick agent, which nothing else looks at and without which no pool autoscales at all — and then disk headroom, config permissions and the organisation's runner-group setting. Each failure comes with what to do about it, and it exits non-zero when something is actually wrong. It reports and repairs nothing, so it is safe to run at any moment, including mid-job. + `skills/runpool/` is an agent skill for *using* RunPool: wiring a repository to local CI, choosing a scope, and diagnosing a job that queues and never starts. ## Describing a machine's pools @@ -133,7 +136,7 @@ Unset, it reports nothing and works as well. `contrib/notify-webhook.sh` is a re - **For an organisation, that control is GitHub's, not RunPool's.** A runner group carries `allows_public_repositories`, it is `false` by default, and runners land in the default group, so public repos in the org do not get them. RunPool reads that setting when you register and warns only if it has been turned on. [SECURITY.md](SECURITY.md) covers the whole picture, including what RunPool deliberately does not do. - **A runner can look healthy while GitHub has dropped it.** GitHub prunes registrations that have not connected for a long time. The local install still starts and connects and then picks up nothing, so jobs queue forever against a pool reporting as running. That is what the `github` column in `status` is for, and `reregister` fixes it. - **`services:` and `container:` do not force a hosted runner.** Those two workflow keys are Linux-only, but an ordinary `docker run` inside a step works anywhere Docker does, including here. -- **More runners is not obviously more throughput**, and the contention warning scales with pool size: it defaults to six times core count, while a busy pool of N runners reaches roughly N times core count on its own. `runpool stats` and `contrib/telemetry-join.sh` settle both questions on your machine, using queue time rather than argument. +- **More runners is not obviously more throughput**, and the contention warning scales with pool size: it defaults to six times core count, while a busy pool of N runners reaches roughly N times core count on its own. `runpool stats` describes what jobs cost and `runpool stats --queue` adds the wait before each one started, which is the figure that moves when capacity changes. Read it with the qualifier it prints: a wait can be a cold pool waking or a dependency that has not finished, and neither is fixed by more runners. `contrib/telemetry-join.sh` gives you the raw rows to separate them. ## Not on a Mac? diff --git a/bin/runpool b/bin/runpool index 79e5e11..2c9cbc4 100755 --- a/bin/runpool +++ b/bin/runpool @@ -66,8 +66,14 @@ runpool — on-demand self-hosted GitHub Actions runner pools for macOS status [--json] [--local] local state alongside what GitHub actually sees; --local skips the GitHub query entirely + doctor why is nothing picking this up: gh, registrations, + launch agents, disk, permissions. Reports and + changes nothing; exits non-zero if something is wrong pools list registered pools - stats job durations by concurrency, to size the pool with data + stats [--queue] what recorded jobs cost, to size the pool with data + --queue adds the wait before a runner picked each + job up, joining the records to GitHub: one API call + per run, so it is not part of plain stats reregister recreate GitHub registrations, keep the local install remove deregister and delete a pool @@ -98,8 +104,9 @@ case "${cmd}" in down-all) _rp_down_all "$@" ;; remove) _rp_remove "$@" ;; status) _rp_status "$@" ;; + doctor) _rp_doctor "$@" ;; pools) _rp_pools ;; - stats) _rp_stats ;; + stats) _rp_stats "$@" ;; tick) _rp_tick ;; autoscale) _rp_autoscale ;; sweep) _rp_sweep ;; diff --git a/lib/common.sh b/lib/common.sh index 1ba6503..bc1a6fe 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -261,6 +261,34 @@ _rp_scope_path() { if [ "$1" = "org" ]; then echo "/orgs/$2"; else echo "/repos/$2"; fi } +# Whether organisation $1's DEFAULT runner group lets public repositories use +# its runners. Echoes 'true', 'false' or 'unknown'. +# +# 'unknown' is a third answer and not a synonym for 'false': reading runner +# groups needs admin:org, and a caller that folded the two together would +# report a permission problem as an all-clear. +# +# GitHub owns this control at organisation scope. The setting defaults to false +# and runners land in the default group because config.sh is never passed +# --runnergroup, so RunPool reads it and reports it and does nothing else. In +# particular it does NOT enumerate the organisation's public repositories to +# re-derive the same answer; SECURITY.md and AGENTS.md both state why the +# repository and organisation cases are deliberately asymmetric. +# +# Shared rather than inline because it now has two callers that must agree: +# `register` reads it once when a pool is created, and `doctor` reads it on +# every run — the setting can be switched on long after the pool exists. +_rp_org_allows_public() { + local pub + pub=$(gh api "/orgs/$1/actions/runner-groups" \ + --jq '[.runner_groups[] | select(.default == true) | .allows_public_repositories][0]' 2>/dev/null) + case "${pub}" in + true) echo true ;; + false) echo false ;; + *) echo unknown ;; + esac +} + # --------------------------------------------------------------------------- # Runner binary # --------------------------------------------------------------------------- diff --git a/lib/lifecycle.sh b/lib/lifecycle.sh index 0fa3e9f..49f34e2 100644 --- a/lib/lifecycle.sh +++ b/lib/lifecycle.sh @@ -22,7 +22,7 @@ _rp_register() { return 1 } - local scope="" target="" count="2" watch="" clean="" tok allow_public=0 vis="" pub="" + local scope="" target="" count="2" watch="" clean="" tok allow_public=0 vis="" while [ $# -gt 0 ]; do case "$1" in --repo|--org|--count|--watch) @@ -121,9 +121,11 @@ _rp_register() { # # A warning rather than a refusal, and no failing closed, precisely because # this is not RunPool's control to enforce. - pub=$(gh api "/orgs/${target}/actions/runner-groups" \ - --jq '[.runner_groups[] | select(.default == true) | .allows_public_repositories][0]' 2>/dev/null) - case "${pub}" in + # + # The read itself lives in lib/common.sh, because `doctor` reports the same + # setting and this one is consulted only at create: it can be switched on + # the day after and nothing here would ever mention it again. + case "$(_rp_org_allows_public "${target}")" in true) _rp_log "WARNING: the default runner group on ${target} has allows_public_repositories=true, so public repositories in that organisation can use these runners. Turn it off in the organisation's Actions runner-group settings unless that is deliberate." ;; diff --git a/lib/scheduler.sh b/lib/scheduler.sh index 7e3a25b..720a5e6 100644 --- a/lib/scheduler.sh +++ b/lib/scheduler.sh @@ -18,6 +18,32 @@ _rp_gh_runners() { [ -n "${out}" ] && echo "${out}" || echo "? ?" } +# The one judgement about a pool's registration, as a single token: +# +# unreachable GitHub could not be asked, so nothing here is a pool fault +# unregistered GitHub has no runners at all — jobs queue forever +# offline runners are up locally and none has reached GitHub +# miscount GitHub's count differs from what the pool expects +# ok +# +# `status` renders this as a note in a table and `doctor` as a check with a +# remedy attached. They are two presentations of one decision, and a second +# copy of the decision is a second thing to keep in step with how GitHub +# actually behaves — which is the part that took three weeks to learn once. +# +# The order is load-bearing: '?' has to be tested before anything treats these +# as numbers, and 'unregistered' before 'miscount', which would otherwise +# swallow it. +# $1 registered $2 online $3 running locally $4 expected count +_rp_gh_state() { + if [ "$1" = "?" ]; then echo unreachable + elif [ "$1" = "0" ]; then echo unregistered + elif [ "$3" -gt 0 ] && [ "$2" = "0" ]; then echo offline + elif [ "$1" != "$4" ]; then echo miscount + else echo ok + fi +} + # How many of a pool's agents are loaded. _rp_running_in() { local pool="$1" count="$2" i=1 running=0 @@ -53,15 +79,12 @@ _rp_status() { total_running=$(( total_running + running )); total_busy=$(( total_busy + busy )) note="" - if [ "${reg}" = "?" ]; then - note=" (github unreachable)" - elif [ "${reg}" = "0" ]; then - note=" ** NOT REGISTERED — jobs will queue forever **"; warn=1 - elif [ "${running}" -gt 0 ] && [ "${online}" = "0" ]; then - note=" ** started but not connecting to github **"; warn=1 - elif [ "${reg}" != "${POOL_COUNT}" ]; then - note=" (github has ${reg}, pool expects ${POOL_COUNT})" - fi + case "$(_rp_gh_state "${reg}" "${online}" "${running}" "${POOL_COUNT}")" in + unreachable) note=" (github unreachable)" ;; + unregistered) note=" ** NOT REGISTERED — jobs will queue forever **"; warn=1 ;; + offline) note=" ** started but not connecting to github **"; warn=1 ;; + miscount) note=" (github has ${reg}, pool expects ${POOL_COUNT})" ;; + esac printf " %-10s %-4s %-20s running %s/%s busy %s github %s/%s%s\n" \ "${p}" "${POOL_SCOPE}" "${POOL_TARGET}" "${running}" "${POOL_COUNT}" \ @@ -139,6 +162,284 @@ _rp_pools() { return 0 } +# --------------------------------------------------------------------------- +# doctor — "why is nothing picking this up", in one command +# --------------------------------------------------------------------------- +# Every check here is answerable today from `status`, the log, and a launchctl +# invocation nobody thinks to run. The value is having them in one place with a +# remedy attached, and one of them is answerable from nothing at all: if the +# tick agent is not loaded, no pool autoscales, every job waits for a manual +# `runpool up`, and `status` reports every pool as perfectly healthy — because +# locally they are. +# +# STRICTLY READ-ONLY, and that is a boundary rather than a preference. A +# diagnostic that repairs is one nobody can run safely while confused, and each +# repair already exists as its own command. This file reports; lib/lifecycle.sh +# changes things. The moment this prunes a disk or re-registers a pool it +# belongs over there instead. +# +# Findings accumulate in two file-scope counters rather than being returned, +# because a function has one integer exit status and these have to survive a +# loop over the pools. +_rp_doctor_fails=0 +_rp_doctor_warns=0 + +# Four markers, all four characters wide so the messages line up: +# ok nothing to do +# note context, or a check that could not be run — not a judgement +# warn worth knowing, does not fail the command +# FAIL something is actually wrong, and the exit status says so +# $1 headline, $2 optional remedy on a continuation line. +_rp_doctor_ok() { printf ' ok %s\n' "$1"; } +_rp_doctor_note() { printf ' note %s\n' "$1"; } +_rp_doctor_warn() { + _rp_doctor_warns=$(( _rp_doctor_warns + 1 )) + printf ' warn %s\n' "$1" + [ -n "${2:-}" ] && printf ' %s\n' "$2" + return 0 # the test above is the last command, and an absent remedy is fine +} +_rp_doctor_fail() { + _rp_doctor_fails=$(( _rp_doctor_fails + 1 )) + printf ' FAIL %s\n' "$1" + [ -n "${2:-}" ] && printf ' %s\n' "$2" + return 0 +} + +_rp_doctor() { + [ $# -eq 0 ] || { _rp_err "doctor takes no arguments (usage: runpool doctor)"; return 1; } + + local gh_ok=1 pools=0 seen_orgs="" p running gh reg online + local tick clean i missing avail_kb free mode other + + _rp_doctor_fails=0 + _rp_doctor_warns=0 + + echo "runpool doctor" + + # --- the global kill switch -------------------------------------------- + echo "" + echo "global" + if _rp_paused; then + _rp_doctor_fail "runpool is paused: every pool is down and autoscale is off" \ + "fix: runpool resume" + else + _rp_doctor_ok "not paused" + fi + + # --- gh ---------------------------------------------------------------- + # _rp_require tests for the binary and stops there, which is the whole gap: + # an expired token fails every API call and each caller then degrades to its + # own quiet fallback. _rp_gh_runners echoes '? ?' and reports as unreachable; + # _rp_autoscale reads a queued count of zero and never brings anything up. + # + # command -v rather than _rp_require, because that writes to stderr and this + # report is a structured thing on stdout. + echo "" + echo "github cli" + if ! command -v gh >/dev/null 2>&1; then + gh_ok=0 + _rp_doctor_fail "gh is not installed, and every GitHub call runpool makes goes through it" \ + "fix: brew install gh, then gh auth login" + elif ! gh auth status >/dev/null 2>&1; then + gh_ok=0 + _rp_doctor_fail "gh is installed but not authenticated, so nothing can register, poll or report" \ + "fix: gh auth login (checks below that ask GitHub are skipped)" + else + _rp_doctor_ok "gh is installed and authenticated" + fi + + # --- the scheduler agents ---------------------------------------------- + # The highest-value check here, and the only one nothing else performs. + # `schedule install` writes these two once and nothing ever looks at them + # again. + echo "" + echo "scheduler" + tick="${RUNPOOL_LABEL_NS}.tick" + clean="${RUNPOOL_LABEL_NS}.clean" + if _rp_agent_loaded "${tick}"; then + _rp_doctor_ok "the tick agent is loaded: autoscale, idle standdown, contention and health" + elif [ -f "${HOME}/Library/LaunchAgents/${tick}.plist" ]; then + _rp_doctor_fail "the tick agent is installed but not loaded, so nothing autoscales: a queued job waits for a manual 'runpool up' and every pool still reports as healthy" \ + "fix: runpool schedule install (rewrites and reloads it)" + else + _rp_doctor_fail "the tick agent is not installed, so nothing autoscales: a queued job waits for a manual 'runpool up' and every pool still reports as healthy" \ + "fix: runpool schedule install" + fi + if _rp_agent_loaded "${clean}"; then + _rp_doctor_ok "the clean agent is loaded: daily prune at 04:00" + else + _rp_doctor_warn "the clean agent is not loaded, so work directories, diagnostics and superseded runner binaries accrue with nothing collecting them" \ + "fix: runpool schedule install, or 'runpool clean' by hand" + fi + + # --- pools -------------------------------------------------------------- + echo "" + echo "pools" + for p in $(_rp_pool_names); do + # Counted before the load, and a failed load reported rather than skipped. + # Everything else in the tool passes over an unreadable config with one + # line on stderr, so a pool that exists and cannot be read looks, from + # every table, exactly like a pool that was never registered. + pools=$(( pools + 1 )) + if ! _rp_load_pool "${p}" 2>/dev/null; then + _rp_doctor_fail "${p}: $(_rp_pool_conf "${p}") cannot be read, or is missing one of POOL_SCOPE, POOL_TARGET, POOL_COUNT and POOL_DIR — every other command skips this pool silently" \ + "fix: repair the file, or delete it and register the pool again" + continue + fi + running="$(_rp_running_in "${p}" "${POOL_COUNT}")" + + # A pool is meant to sit down, so 'not loaded' says nothing and is not + # reported. A MISSING plist is different: _rp_up refuses on the first one + # it cannot find, and that refusal is the first anybody hears of it. + missing=0 + i=1 + while [ "${i}" -le "${POOL_COUNT}" ]; do + [ -f "${RUNPOOL_AGENT_DIR}/$(_rp_label "${p}" "${i}").plist" ] || missing=$(( missing + 1 )) + i=$(( i + 1 )) + done + [ "${missing}" -gt 0 ] && _rp_doctor_fail \ + "${p}: ${missing} of ${POOL_COUNT} launch agent(s) missing, so 'runpool up ${p}' will refuse" \ + "fix: runpool rewrite-agents" + + # An org pool with an empty watch list never autoscales. GitHub reports + # queued runs per repository and not per organisation, so _rp_autoscale has + # nothing to poll and the pool waits for a manual 'runpool up' forever — + # while looking entirely healthy everywhere else, which is exactly the + # failure this command exists for. Local, free, and no API call. + [ "${POOL_SCOPE}" = "org" ] && [ -z "${POOL_WATCH:-}" ] && _rp_doctor_fail \ + "${p}: an org pool with no watched repositories never autoscales, because github reports queued runs per repository rather than per organisation" \ + "fix: give it --watch OWNER/REPO,... in ~/.config/runpool/pools and 'runpool apply'" + + if [ "${gh_ok}" = "0" ]; then + _rp_doctor_note "${p}: ${POOL_SCOPE} ${POOL_TARGET}, ${running}/${POOL_COUNT} running locally (github not checked)" + continue + fi + gh="$(_rp_gh_runners)"; reg="${gh% *}"; online="${gh#* }" + case "$(_rp_gh_state "${reg}" "${online}" "${running}" "${POOL_COUNT}")" in + unreachable) + _rp_doctor_note "${p}: github could not be reached, so only local state is known: ${running}/${POOL_COUNT} running" ;; + unregistered) + _rp_doctor_fail "${p}: github has no runners registered for ${POOL_TARGET}, so every job routed here queues forever" \ + "fix: runpool reregister ${p} (github prunes registrations after a long idle spell)" ;; + offline) + _rp_doctor_fail "${p}: ${running} runner(s) started locally and none has reached github" \ + "fix: runpool reregister ${p}" ;; + miscount) + # A note and not a warning. _rp_gh_runners counts every runner at the + # scope, and an organisation's scope covers other pools and other + # machines, so a count above POOL_COUNT is entirely normal there. + _rp_doctor_note "${p}: github has ${reg} runner(s) at ${POOL_SCOPE} ${POOL_TARGET}, the pool expects ${POOL_COUNT} (an org scope also counts other pools and other machines)" ;; + *) + _rp_doctor_ok "${p}: ${POOL_SCOPE} ${POOL_TARGET}, github has ${reg}, ${running}/${POOL_COUNT} running locally" ;; + esac + done + [ "${pools}" = "0" ] && _rp_doctor_fail \ + "no pools are registered, so there is nothing to pick a job up" \ + "fix: runpool register --repo OWNER/REPO (or --org ORG)" + + # --- disk --------------------------------------------------------------- + # Against RUNPOOL_BASE and not '/'. The base can sit on a secondary or + # external volume, and the only free space that matters is the one the + # runners actually write into. + echo "" + echo "disk" + avail_kb=$(df -k "${RUNPOOL_BASE}" 2>/dev/null | awk 'NR == 2 {print $4}') + case "${avail_kb}" in + ''|*[!0-9]*) + _rp_doctor_note "could not read the free space on ${RUNPOOL_BASE}" ;; + *) + # Reported in MB below a gigabyte. Whole-GB division renders the most + # alarming case in the range, a nearly full volume, as a flat '0GB'. + if [ "${avail_kb}" -lt 1048576 ]; then + free="$(( avail_kb / 1024 ))MB" + else + free="$(( avail_kb / 1048576 ))GB" + fi + # Persistent runners never clean up after themselves and the numbers are + # large: a self-update strands roughly 580MB of superseded binaries per + # runner and diagnostics reach roughly 150MB per runner, so a pool of + # four turns over several GB between cleans. Pointed at `clean`, never + # pruned here: see the read-only note above. + if [ "${avail_kb}" -lt 5242880 ]; then + _rp_doctor_fail "${free} free on ${RUNPOOL_BASE}: a job that fills the disk fails in ways that look like a defect in the code" \ + "fix: runpool clean" + elif [ "${avail_kb}" -lt 20971520 ]; then + _rp_doctor_warn "${free} free on ${RUNPOOL_BASE}" \ + "fix: runpool clean prunes work dirs, temp, diagnostics, superseded binaries and package stores" + else + _rp_doctor_ok "${free} free on ${RUNPOOL_BASE}" + fi ;; + esac + + # --- security ----------------------------------------------------------- + echo "" + echo "security" + # The config, and deliberately NOT the pools file. install.sh chmods the + # config 600 because that is where a notifier's endpoint and token go, and + # omits the chmod on the pools file on purpose: it holds no credentials, + # which is the whole reason it is safe to copy between machines. + # + # Only a regular file is checked. RUNPOOL_CONFIG=/dev/null is the documented + # way to isolate an invocation and is mode 666 by definition, so testing it + # would fail every isolated run for no reason. + if [ ! -e "${RUNPOOL_CONFIG}" ]; then + _rp_doctor_note "no config at ${RUNPOOL_CONFIG}, so every setting is at its default" + elif [ ! -f "${RUNPOOL_CONFIG}" ]; then + _rp_doctor_note "${RUNPOOL_CONFIG} is not a regular file, so its permissions are not checked" + else + mode=$(stat -f '%Lp' "${RUNPOOL_CONFIG}" 2>/dev/null) + case "${mode}" in + ''|*[!0-7]*) + _rp_doctor_note "could not read the mode of ${RUNPOOL_CONFIG}" ;; + *) + other=$(( 8#${mode} & 8#077 )) + if [ "${other}" -ne 0 ]; then + _rp_doctor_fail "${RUNPOOL_CONFIG} is mode ${mode}, so other users on this machine can read it, and it is where a notifier's endpoint and token live" \ + "fix: chmod 600 ${RUNPOOL_CONFIG}" + else + _rp_doctor_ok "${RUNPOOL_CONFIG} is mode ${mode}, owner only" + fi ;; + esac + fi + + # The organisation runner-group setting. `register` consults it once, when a + # pool is created; it can be switched on the day after and nothing would ever + # mention it again. Reported and never re-derived — enumerating an + # organisation's public repositories to work out the same answer is + # explicitly not RunPool's job. See SECURITY.md. + if [ "${gh_ok}" = "1" ]; then + for p in $(_rp_pool_names); do + _rp_load_pool "${p}" || continue + [ "${POOL_SCOPE}" = "org" ] || continue + # Several pools can share one organisation; ask about each one once. + case " ${seen_orgs} " in *" ${POOL_TARGET} "*) continue ;; esac + seen_orgs="${seen_orgs} ${POOL_TARGET}" + case "$(_rp_org_allows_public "${POOL_TARGET}")" in + true) + _rp_doctor_warn "${POOL_TARGET}: the default runner group has allows_public_repositories=true, so a public repository in that organisation can run on these runners" \ + "fix: turn it off in the organisation's Actions runner-group settings, unless it is deliberate" ;; + false) + _rp_doctor_ok "${POOL_TARGET}: allows_public_repositories=false on the default runner group" ;; + *) + _rp_doctor_note "${POOL_TARGET}: could not read the runner groups (needs admin:org) — check allows_public_repositories in the organisation's Actions settings" ;; + esac + done + fi + + echo "" + if [ "${_rp_doctor_fails}" -gt 0 ]; then + printf '%s problem(s) and %s warning(s). Nothing above was changed.\n' \ + "${_rp_doctor_fails}" "${_rp_doctor_warns}" + return 1 + fi + if [ "${_rp_doctor_warns}" -gt 0 ]; then + printf 'No problems, %s warning(s). Nothing above was changed.\n' "${_rp_doctor_warns}" + return 0 + fi + echo "Everything checks out. Nothing above was changed." + return 0 +} + # --------------------------------------------------------------------------- # autoscale — bring a pool up when it has queued work # --------------------------------------------------------------------------- diff --git a/lib/stats.sh b/lib/stats.sh index 5cc94bd..af3761d 100644 --- a/lib/stats.sh +++ b/lib/stats.sh @@ -13,7 +13,10 @@ # So this prints what a person wants at a glance and points at the data, and # at the tool that joins it. See contrib/telemetry-join.sh. # -# Requires RUNPOOL_TELEMETRY=1 and the job hook. Reads only local files. +# Requires RUNPOOL_TELEMETRY=1 and the job hook. Reads only local files, with +# one deliberate exception: `--queue` runs contrib/telemetry-join.sh, which asks +# GitHub for the wait the hook cannot see. That is precisely why it is a flag +# and not part of the default readout. _rp_stats_file() { echo "${RUNPOOL_BASE}/telemetry/jobs.jsonl"; } @@ -55,11 +58,10 @@ _rp_stats_quantiles() { ' } -_rp_stats() { - local f rows n_done first last cores peak_load total key median p90 runs - - f="$(_rp_stats_file)" - +# Both readouts need the same two things to exist. Explains itself and returns +# non-zero when they do not, so either caller can simply stop. +_rp_stats_have_data() { + local f="$1" if [ ! -s "${f}" ]; then echo "No telemetry recorded yet." echo "" @@ -69,16 +71,35 @@ _rp_stats() { echo "" echo "Then: runpool rewrite-agents && runpool down-all" echo "Data accrues as jobs run. It records timings and machine state only." - return 0 + return 1 fi - - n_done=$(grep -c '"phase":"completed"' "${f}" 2>/dev/null || echo 0) - if [ "${n_done}" -eq 0 ]; then + if [ "$(grep -c '"phase":"completed"' "${f}" 2>/dev/null || echo 0)" -eq 0 ]; then echo "Telemetry is recording, but no job has completed yet." echo " starts recorded: $(grep -c '"phase":"started"' "${f}" 2>/dev/null || echo 0)" - return 0 + return 1 fi + return 0 +} + +_rp_stats() { + local arg queue=0 f + for arg in "$@"; do + case "${arg}" in + --queue) queue=1 ;; + *) _rp_err "unknown flag: ${arg} (usage: runpool stats [--queue])"; return 1 ;; + esac + done + f="$(_rp_stats_file)" + # Missing data is explained, not an error: nothing is wrong with a machine + # that has not switched telemetry on. + _rp_stats_have_data "${f}" || return 0 + if [ "${queue}" = "1" ]; then _rp_stats_queue "${f}"; else _rp_stats_local "${f}"; fi +} +_rp_stats_local() { + local f="$1" rows n_done first last cores peak_load total key median p90 runs + + n_done=$(grep -c '"phase":"completed"' "${f}" 2>/dev/null || echo 0) rows="$(_rp_stats_rows "${f}")" first=$(head -1 "${f}" | _rp_stats_field ts) last=$(tail -1 "${f}" | _rp_stats_field ts) @@ -118,11 +139,103 @@ EOF echo "GitHub knows about the same runs, which adds the queue times the job" echo "hook cannot see:" echo "" - echo " contrib/telemetry-join.sh > joined.tsv" + echo " runpool stats --queue median and p90 queue time, per job" + echo " contrib/telemetry-join.sh one row per job, everything joined" echo "" echo " records: ${f}" } +# --------------------------------------------------------------------------- +# --queue — the wait before a runner picked the job up +# --------------------------------------------------------------------------- +# Queue time is the figure that answers "do I need more runners", because more +# runners help if and only if work is waiting. The job hook fires when a runner +# *picks a job up*, so the whole wait before that moment is invisible locally +# and only GitHub knows it. +# +# BEHIND A FLAG, and not part of plain `runpool stats`. The header of this file +# says it reads only local files; the join makes one gh api call per unique run. +# A network fan-out behind a command people run casually is exactly the failure +# `status --json --local` exists to prevent, so the two readouts stay separate. +# +# contrib/telemetry-join.sh is INVOKED, not reimplemented here. It carries two +# correctness traps that took working out — run id plus runner name matches +# several API jobs and needs the nearest-start tie-break, and created_at is +# run-level rather than job-level — and a second copy is a second place to get +# them wrong. +_rp_stats_queue() { + local f="$1" join joined rows n key median p90 runs + _rp_require gh || return 1 + + # RUNPOOL_ROOT is resolved by bin/runpool, following symlinks, and is visible + # to everything it sources. The Homebrew formula installs contrib/ beside + # bin/ and lib/ under libexec, so this resolves for a tap install too. + join="${RUNPOOL_ROOT:-}/contrib/telemetry-join.sh" + [ -x "${join}" ] || { + _rp_err "cannot run ${join} — contrib/ ships with runpool; check the install, or chmod +x it" + return 1 + } + + echo "Joining the local records to GitHub: one API call per run, so give it a moment." + echo "" + # The telemetry path is passed explicitly. Left to work it out, the script + # falls back to `runpool status --json --local` and picks up whatever runpool + # is on PATH, which is not necessarily the one being run. + joined="$("${join}" "${f}" 2>/dev/null)" || { + _rp_err "the join failed — run '${join} ${f}' directly to see why" + return 1 + } + + # Reshaped to the same "valueworkflow / job" that _rp_stats_rows emits, + # so the grouping below is the grouping the duration table already uses. + # Column 6 is queue_s, 2 is workflow, 3 is job; NR > 1 drops the header. + rows="$(printf '%s\n' "${joined}" \ + | awk -F'\t' 'NR > 1 && NF >= 6 && $6 ~ /^-?[0-9]+$/ { printf "%s\t%s / %s\n", $6, $2, $3 }')" + n=$(printf '%s\n' "${rows}" | grep -c . ) + + if [ "${n}" -eq 0 ]; then + echo "The join produced no rows." + echo "" + echo "Every record has to match a job GitHub still has, and run history is" + echo "kept for a limited time, so records older than that no longer join." + echo "Run it directly to see what came back:" + echo "" + echo " ${join} ${f}" + return 0 + fi + + printf 'Queue time %s job record(s) joined to GitHub\n' "${n}" + echo "" + printf ' %-34s %6s %10s %10s\n' "job" "runs" "median" "p90" + printf '%s\n' "${rows}" | awk -F'\t' '{print $2}' | sort -u | while IFS= read -r key; do + [ -n "${key}" ] || continue + # Split on the tab the quantile helper emits, as the duration table does. + IFS="$(printf '\t')" read -r median p90 runs < joined.tsv" +} + # Seconds to something readable. _rp_stats_dur() { local s="${1:-0}" diff --git a/skills/runpool/SKILL.md b/skills/runpool/SKILL.md index f70951a..af8b37d 100644 --- a/skills/runpool/SKILL.md +++ b/skills/runpool/SKILL.md @@ -19,6 +19,7 @@ On-demand self-hosted GitHub Actions runner pools for macOS. Pools wake when job ```bash command -v runpool || echo "not installed" runpool status +runpool doctor # if anything looks wrong, or a job is queued and waiting ``` Not installed: @@ -118,20 +119,32 @@ Keep publish, deploy and OIDC jobs on hosted runners too: npm provenance require ## Diagnosing "the job is queued and nothing happens" -Work down this list. +**Start here, before `status` and before the log.** ```bash -runpool status +runpool doctor ``` -- **`** NOT REGISTERED **`** — GitHub has pruned the registrations after a long idle spell. The local install is untouched and looks perfectly healthy, which is what makes this confusing. Fix: `runpool reregister `. -- **`running N/N` but `github 0/N`** — the runners started but are not reaching GitHub. Same fix. -- **`GLOBAL: paused`** — someone hit the kill switch. `runpool resume`. -- **`running 0/N` and the job is genuinely queued** — the tick brings a pool up within about a minute. Wait one minute before intervening. `runpool up ` forces it. -- **`running 0/N` on an *org* pool that never comes up on its own** — check the pool's `watch` array in `status --json`. If it is empty, autoscale has nothing to poll, because GitHub reports queued runs per repository rather than per organisation. Add the repository to the pool's line in `~/.config/runpool/pools` and `runpool apply`, or `runpool register` it with `--watch` in the first place. -- **Everything looks right but the job still waits** — check routing rather than capacity. The workflow's `runs-on` may not resolve to `self-hosted`, or its labels may not match the pool's. +It works down the whole list below in one pass, prints a remedy against each failure, and **exits non-zero when something is actually wrong**. It reports and repairs nothing, so it is safe at any moment including mid-job — and so nothing it finds is fixed until you run the command it names. + +What it finds, and what each finding means: + +- **the tick agent is not loaded** — nothing autoscales, so every job waits for a manual `runpool up` while every pool still reports as perfectly healthy. **This is the one failure no other command surfaces.** Fix: `runpool schedule install`. +- **`gh` is not authenticated** — every API call fails and each caller degrades quietly rather than complaining: `status` reports GitHub as unreachable, and autoscale reads a queued count of zero and never wakes anything. Fix: `gh auth login`. +- **github has no runners registered** — GitHub prunes registrations after a long idle spell. The local install is untouched and looks entirely healthy, which is what makes this confusing. Fix: `runpool reregister `. +- **started locally and none has reached github** — the runners are up and not connecting. Same fix. +- **runpool is paused** — someone hit the kill switch. Fix: `runpool resume`. +- **launch agents missing** — `runpool up` refuses on the first plist it cannot find. Fix: `runpool rewrite-agents`. +- **an org pool with no watched repositories** — it never autoscales, because GitHub reports queued runs per repository rather than per organisation. Fix: give it `--watch` and `runpool apply`. See *`--watch` is org-only and matters* above. +- **disk, config permissions, the organisation's runner-group setting** — each with its own remedy. None of these stops a job being picked up, but they are the things nothing else ever looks at. + +**Two situations `doctor` deliberately reports as healthy, because they are.** + +- **`running 0/N` with a job genuinely queued** — the tick brings a pool up within about a minute. Wait a minute before intervening; `runpool up ` forces it. +- **A clean report and the job still waits** — the problem is routing, not capacity. The workflow's `runs-on` may not resolve to `self-hosted`, or its labels may not match the pool's. RunPool controls only whether the runners are up and cannot see either. ```bash +runpool status # the same picture as a table, one row per pool runpool status --json # machine-readable, for scripting runpool status --json --local # same shape, no GitHub call; use this on a timer tail -50 ~/Library/Logs/runpool/runpool.log @@ -155,11 +168,23 @@ Resizing refuses while a job is running. Wait rather than forcing. Telemetry plus GitHub can answer this properly. `runpool stats` deliberately will not: it describes what jobs cost and stops there, because every analysis baked into the tool is a blind spot with a version number. +**Start with queue time, because it is the only figure that moves when capacity changes.** Duration says what a job costs and load says how contended the machine was; neither responds to another runner. + +```bash +runpool stats --queue +``` + +Median and p90 of the wait before each job started, per `workflow / job`. It is behind a flag and not part of plain `stats` because it joins the local records to GitHub — one API call per run — and `stats` otherwise reads nothing but local files. + +**Never quote the number without the qualifier it prints.** A queue time conflates three different situations and only one of them is a shortage of runners; the trap is spelled out below and the command repeats it every time for that reason. On an on-demand pool the first job after a quiet spell always shows about a minute of queue while the pool wakes, and no amount of capacity removes it. + +**Then go to the raw rows to separate them:** + ```bash contrib/telemetry-join.sh > joined.tsv ``` -One row per job: duration, queue time, load at start, concurrency, and the raw created and started timestamps. Analyse that, do not trust a canned summary. +One row per job: duration, queue time, load at start, concurrency, and the raw created and started timestamps. Analyse that, do not trust a canned summary — `--queue` included. **Four traps, each of which has produced a confidently wrong answer here.**