From 4ae20daf204c73e5792ad1dcd9bd5d826cc95350 Mon Sep 17 00:00:00 2001 From: Mac Anderson Date: Wed, 9 Sep 2026 13:53:31 -0700 Subject: [PATCH] fix(scripts): stop an empty ci run list reading as ten unverified commits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `check-main-verified.sh` reads the run history with one `gh run list` call and defaults every commit to `missing` when no run in that list matches it. An empty list therefore matches nothing, and each of the ten commits it checks becomes a separate finding. `gh` returns `[]` with exit 0 for a transient API fault as readily as for a genuinely empty history. On 2026-09-09 one such read failed the canary's `main carries no unverified commit` job and filed a tracking issue saying nothing had verified ten commits — while `ci` on the tip, cd7d53c, had already concluded success. An auth or permission failure exits non-zero and was already caught as an unknown; this shape was not. The two cases are separable: the read asks for the last 60 `ci` runs on `main` whatever their age, and an outage creates no runs and destroys none, so it cannot empty that list. Empty means the run could not see the history, which is an unknown and exits 0 — the discipline this script's own header already states. A commit absent from a populated list is still reported, which is the gap the script watches, and a case pins that so the fix cannot buy its safety with the coverage. Witness: the three new cases in scripts/test-main-verified.sh fail on the previous script (exit 1, "FAILED — main carries commits nothing verified") and pass on this one. Suite: 38 passed, 0 failed. Closes #6475 --- scripts/check-main-verified.sh | 28 +++++++++++++++++++++---- scripts/test-main-verified.sh | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 4 deletions(-) diff --git a/scripts/check-main-verified.sh b/scripts/check-main-verified.sh index 5e60d485b..99be38e6a 100755 --- a/scripts/check-main-verified.sh +++ b/scripts/check-main-verified.sh @@ -61,10 +61,11 @@ # # ── It fails OPEN, at every unknown ────────────────────────────────────────── # -# No `gh`, an unreachable API, an unparseable answer: report and exit 0. This -# is a monitor, and a monitor that can itself block a merge is worse than the -# gap it watches — the same argument `main-red-claim.sh`'s header makes about -# a claim check. Every unknown is loud, never silent. +# No `gh`, an unreachable API, an unparseable answer, a run list that came back +# empty so no commit could be matched: report and exit 0. This is a monitor, +# and a monitor that can itself block a merge is worse than the gap it watches +# — the same argument `main-red-claim.sh`'s header makes about a claim check. +# Every unknown is loud, never silent. # # ── `--announce` reaches a person, on codeql-canary.sh's shape ─────────────── # @@ -233,6 +234,25 @@ elif ! runs="$(gh run list --workflow ci.yml --branch main --limit 60 \ unknown "could not reach the Actions API" fi +# An empty list is a blind read, not a finding about every commit at once. +# +# `gh` answers a transient API fault with `[]` and exit 0 as readily as it +# answers a genuinely empty history, and the loop below defaults each commit +# to `missing`. So on 2026-09-09 one bad read reported all ten commits as +# unverified and filed an issue saying nothing had checked a tree whose `ci` +# run on cd7d53c had already concluded success. An auth or permission failure +# exits non-zero and is already caught above; this is the shape that does not. +# +# The two cases are separable because the read asks for the last 60 `ci` runs +# on `main` whatever their age. An outage creates no runs, and destroys none, +# so it cannot empty that list — older runs stay in it, and a commit absent +# from a populated list is still reported, which is the gap this script +# watches. Empty means this run could not see the history at all, and that is +# an unknown. +if [ -z "$runs" ]; then + unknown "the ci run list came back empty, so no commit could be matched" +fi + now_epoch="$(date -u +%s 2>/dev/null || echo 0)" # Seconds since an RFC-3339 timestamp, or 0 when this platform's `date` will diff --git a/scripts/test-main-verified.sh b/scripts/test-main-verified.sh index 1764712e0..91688b427 100755 --- a/scripts/test-main-verified.sh +++ b/scripts/test-main-verified.sh @@ -199,6 +199,44 @@ case "$out" in *) fail=$((fail + 1)); echo "FAIL an unknown exited 0 without saying so:"; echo "$out" ;; esac +# ── E: an empty run list is a blind read ───────────────────────────────────── +# +# `gh run list` returns `[]` with exit 0 for a transient fault as readily as +# for an empty history, and the verdict loop defaults every commit to +# `missing`. On 2026-09-09 that turned one bad read into ten findings and a +# filed issue claiming nothing had verified a tree whose `ci` run had gone +# green seventeen seconds earlier. An auth failure exits non-zero and was +# already an unknown; this shape was not. +empty_list_commits="$(commit $A 'a merge the API could not answer for') +$(commit $B 'the merge before it')" + +want "an empty run list is an unknown, not a finding" expect-pass \ + "$empty_list_commits" "" \ + "UNKNOWN" + +want "...and says the list is what came back empty" expect-pass \ + "$empty_list_commits" "" \ + "came back empty" + +# It also files no issue, which is the half of a misreport somebody has to +# undo by hand. +out="$("$SCRIPT" --fixture-commits "$empty_list_commits" --fixture-runs "" \ + --announce --dry-run 2>&1)" +case "$out" in + *"issue create"*) + fail=$((fail + 1)); echo "FAIL an empty run list filed an issue:"; echo "$out" ;; + *) pass=$((pass + 1)); echo "ok ...and files no issue over a read it could not make" ;; +esac + +# The coverage this must not buy its safety with. A populated list that simply +# has no run for the commit is the real gap (f1f36660's shape above), and it +# stays a failure — the guard turns on the list being empty, never on a commit +# being absent from it. +want "a commit missing from a POPULATED list is still reported" expect-fail \ + "$(commit $A 'a merge nothing ran for')" \ + "$B completed success $(now_iso)" \ + "missing" + # ── N: several commits, one bad ────────────────────────────────────────────── # The reported window is a run of merges, not one; a guard that stopped at the # first verified commit would have missed three of the four that day.