TESTING: validation program for /audit:issues — run it before generalising the method

[mmcky]

A validation program for /audit:issues, and the reason for one: the skill has been executed exactly once, by hand, as a runbook — never as a skill. Its abstractions (the doctrine, the deliverable shape, the five phases) are generalised from that single execution, and three more skills are planned on top of them. This is the audit counterpart to what #10 and #14 did for benchmark, and it settles the open question parked in #12 — whether the shared method deserves more structure, or less.

What prompted it

A read of the whole plugin (2026-07-27) started from the suspicion that it was over-complicated. That is not what it found. Line count is not the problem; abstraction ahead of evidence is. Compare benchmark:review-acceleration, which is comparably long but earns it with reproduced verdicts, dated validation runs, calibration anchors, and sentences like "never present warm-only speedups as the headline — ge_arrow measured 1.4–4.8× faster warm and ~45× slower as-used." One sentence like that prevents a class of wrong report. /audit:issues has good categories — fixed-but-open, scheduled-but-escaped, agreed-but-never-filed — and not one worked instance of any of them.

Three defects were real enough to fix without a run, and are done (audit 0.1.1):

Fixed	Was
Phase checkpoints are named, and phase 2 appends per item	Resumability was promised in three files, but phases 2 and 3 named no artifact — so a lost session resumed only if the next one guessed the same filename. Phase 2 is also the phase a long run dies inside, and it was specified to write on completion.
The bundle scales to the tracker	Four documents were unconditional; under ~30 open issues that is three files of padding, and padding makes a report less checkable.
meta.fetched_by replaces meta.authenticated	The old field could only ever be true — preflight exits on every unauthenticated path. A provenance field that cannot vary is not evidence.

Everything else needs a run to settle, which is what this issue is for.

The claims under test

Each of these is asserted somewhere in the plugin and is currently unfalsified. A run that does not try to break them is not a test.

Claim	Asserted in	How a run falsifies it
Resumes across a lost session	README, SKILL.md, doctrine §4	Kill the run mid-phase-2 and restart. Does it resume at the right item — not re-verifying the whole set, not silently skipping the item it died on?
Thread-complete on the closed side	doctrine rule 3	Spot-check closed issues whose decisive content is only in the thread. Does the report reflect it, or only the title and stateReason?
Every claim carries an evidence class	doctrine §2	Sample findings. Any untagged claim is a defect by the doctrine's own rule; any [verified] whose citation does not support it is worse.
The self-audit closes gaps rather than disclosing them	doctrine §5	Are unaccounted numbers explained one by one, or waved at collectively?
Read-only	doctrine §3	Tracker and working-tree diff either side of the run. Must be empty.
Sibling checks catch wave escapes	quantecon-context.md	Does it find at least one real cross-repo miss? If a repo has none, seed a known one and check it is caught.
Tiering slots into the repo's existing plan	SKILL.md phase 4	Does the report name the plan anchor it discovered, and would a maintainer agree with T0?

Run matrix

Sizes measured 2026-07-27. Sequenced deliberately — these are multi-hour runs and the first one has to be the anchor.

#	Repo	Family	Open issues	All items	What it tests
1	QuantEcon/action-translation	action-*	55	228	Regression anchor. The only tracker the runbook has been through, so the skill run can be diffed against the hand run — the benchmark calibration pattern. Also: consumer-visible tiering, and verify-against-the-tag rather than main.
2	QuantEcon/lecture-jax	lecture-*	29	341	Small open set, large PR history. Exercises the new bundle scale rule right at its boundary, and campaign/sibling detection where SYNC: PRs are the norm.
3	QuantEcon/meta	org-wide	161	319	The hardest case for the doctrine: issues with no code to verify against. What does [verified] mean when the artifact is a decision? If the answer is "nothing", the doctrine needs a fourth evidence class or an explicit exclusion.
4	QuantEcon/lecture-python.myst	lecture-*	73	1003	Endurance, and a stretch goal. Does phase 2 judgement hold quality at item 70 the way it did at item 5? Scope to issues, using PRs as evidence only.

One private project-* repo is worth adding if a run is cheap, since it is the only way to exercise the auth path that quantecon-context.md § Access describes.

What each run records

A run document in reviews/, following the convention set by reviews/validation-run-ge_arrow-2026-07-22.md: reviews/audit-run-<repo>-<date>.md. Not the bundle itself — the bundle belongs in the audited repo per deliverables.md.

• Snapshot provenance: repo, timestamp, fetched_by, item counts, unaccounted numbers.
• Cost and duration: wall clock and rough token spend per phase, so "expect hours, not minutes" stops being a guess.
• Every claim in the table above, marked held or broken, with the evidence.
• Interruption log: where it was killed, what resuming actually did.
• What the doctrine did and did not transfer. This is the payload. A rule that was cited and load-bearing, a rule that never came up, and a rule the run had to work around are three different verdicts, and only the third is a bug.
• A maintainer's read of the tiering — the one thing no self-audit can check.

Exit criteria

The program is done when all of these hold, not when the runs finish:

• Runs 1–3 complete, each with a run document and a delivered bundle a maintainer would act on
• Every claim in the table is either confirmed with evidence or amended in the doctrine — no claim left in the "asserted, untested" state that prompted this issue
• One worked example captured into audit/references/examples/ — a real issue through verification to catalog entry, with its evidence tag. This is the single highest-value artifact the program produces, and it is what benchmark has and audit does not.
• Unit tests for reconcile and require_thread_objects (carried from #12; coverage.json is load-bearing evidence and currently untested)
• The packaging question below, answered from evidence

The packaging question, deliberately deferred

The plugin ships three plugin-level reference files for a family of one, and docs/developing-skills.md says the opposite in its own words: "Only SKILL.md is required … Adding either before you need it just makes the skill harder to read." The observable cost today is small — about six passages exist purely to disclaim the generality they just asserted (deliverables.md opens by calling itself "a worked example, not a contract"), and the split changes nothing about what the skill produces.

That is a reason to leave it alone until there is evidence, not a reason to refactor now. Run 3 is the sharpest test: QuantEcon/meta is far enough from action-translation that whatever survives it is probably genuinely general, and whatever does not was a description of the first execution wearing a doctrine's clothes. Decide after run 3 — merge, split, or leave — and record the reasoning here.

Tracking: #12 (the audit plugin plan).

[mmcky]

The three fixes described above are open as #17 — named phase checkpoints with per-item appends, a bundle that scales to the tracker, and meta.fetched_by replacing the always-true authenticated flag. That PR also records the five consolidation proposals that were withdrawn on merit, so the packaging question arrives here undecided rather than half-answered.

[mmcky]

Phase 1 executed against QuantEcon/action-translation on merged audit 0.1.1, as the first step of run 1. The snapshot is deterministic, so this half of the run is valid regardless of who drives the rest.

Items	116 issues + 112 PRs = 228, numbers 1..228 fully accounted, 0 unaccounted
Threads	open issues 49 comments / 30 items · closed issues 80 / 32 · open PRs 2 comments + 5 reviews · closed PRs 28 comments + 377 reviews / 105
Wall clock	11.3 s for the whole tracker, two API round trips
Provenance	fetched_by populated; gh 2.96.0

Two things that already update the plan. The tracker has moved since #11 measured it (111 issues / 110 PRs → 116 / 112), which is a small argument for the run document recording its own snapshot rather than citing an earlier count. And phase 1 is not the expensive part by any margin — 11 seconds against an estimated multi-hour phase 2 — so any cost concern about re-snapshotting on resume is unfounded.

Finding 0, found before the run started: the read-only boundary and the working directory disagree

Three statements in the plugin, each defensible alone:

• doctrine §3 — read-only means "no closing, labelling, milestoning, commenting, or editing, and no branch or file changes in the audited repo".
• deliverables.md § Where it goes — the bundle's first-preference destination is "the audited repo's own notes system — .dev/audits/<date>-<subject>/".
• SKILL.md 0.1.1 — --out defaults to .audit/<repo>-<YYYY-MM-DD>/, and the natural working directory for a run is a checkout of the audited repo, since phase 2 verifies claims against its default branch.

Together they instruct the audit to write its snapshot and its bundle into the repo it promised to make no file changes in. Nothing is committed or pushed, and the files are untracked — but §3's wording does not draw that distinction, and the 0.1.1 default made the collision concrete rather than theoretical. This predates that change: deliverables.md and doctrine §3 have disagreed since #11.

It needs a decision rather than a quiet convention, and the two candidate readings differ in what they actually guarantee:

1. Narrow §3 to mutation of content and history — no commits, no pushes, no tracker changes, no edits to tracked files — and state explicitly that a run may write its own working directory. Keeps the guarantee that matters (an audit never changes the thing it describes) and admits that an audit produces files.
2. Keep §3 literal and require the working directory to live outside the audited checkout, with the bundle delivered into the notes system by a human as a separate step.

Reading 1 looks right — an audit that produces a report necessarily writes something, and the property worth protecting is that the report still describes the repo it was run against. But it is a doctrine change, so it belongs to whoever owns the doctrine, and the run should not silently pick one. Recorded here as the program's first finding.

[mmcky]

Finding 0 resolved, and the run is now documented. #19 (audit 0.1.2) narrows doctrine §3 to what it was always protecting — content and history: no commits, no pushes, no branches, no edits to tracked files, no tracker mutation — and states that a run may write its own working directory. An audit that produces a report necessarily produces files; the boundary is about mutation, not about writing. deliverables.md gains the matching half: writing the bundle is the audit's job, committing it is a human step.

Preparing that also produced a better rule than the one 0.1.1 shipped, and it only came from looking at a real repo: action-translation already gitignores .dev/scratch/*, so a run there leaves git status completely clean and needs no .gitignore edit — which would itself be a change to a tracked file. The working directory is now discovery-ordered: an already-ignored scratch location, else an untracked .audit/, else outside the checkout.

docs/tutorial-run-an-audit.md is the runnable form of this program: install, working directory, invocation, the phase 1 numbers to expect, and — at step 4 — the instruction to deliberately interrupt phase 2 and restart, which is the only test of the checkpoint fix in #17. Step 6 is a ten-point review checklist ending with the two checks no self-audit can make (is the tiering right; would you act on this), and step 7 is the run-document skeleton.

Run 1 is now blocked only on someone installing the plugin and driving it. The evidence this program collects is the reviewer's judgement, so the run should be done by a maintainer rather than by the session that wrote the fix.