TESTING: audit run 1 on action-translation — seven plugin defects, and a cost claim off by an order of magnitude

[mmcky]

Seven defects from run 1 of the validation program (#16) — the first execution of /audit:issues as a skill, against QuantEcon/action-translation on 2026-07-28. Full record: reviews/audit-run-action-translation-2026-07-28.md.

The bundle itself is good — four documents, 56 open issues verified against main rather than their threads, a wave escape found, coverage reconciling exactly against coverage.json. Eight of the ten review checks held. What follows is what running it exposed that reading it could not, including one defect in the checkpoint fix shipped the day before in #17.

Severity 1 — an evidence rule that lets a wrong citation pass

1. [verified] citations are not checked for reachability. The audit's headline finding cites commit b99b431 as where a file's history ends. That commit is real and does touch the file — but it is not on main; it lives only on an unmerged branch, while the audit's own header says "verified against main @ 2c3d624". The conclusion survives independent checking; the citation does not. A reviewer who checks it finds nothing, which is worse than an untagged claim, because the tag invites trust.

doctrine §2 requires file:line, a merged PR, or a tag — and never says confirm the commit is an ancestor of the ref you named. Fix: state that requirement in §2, and have the skill run git merge-base --is-ancestor <sha> <ref> before tagging a commit citation [verified].

2. The closed side is never checkpointed. findings.md held one section — 56 open issues. The 62 closed issues were verified and written straight into the catalog. Half of phase 2 sat outside the checkpoint, and it breaks the resume rule #17 introduced: "resume at the lowest number in issues.json with no entry" would have re-verified all 62 closed issues from scratch. Fix: phase 2 appends both state blocks, and the resume rule reads state from issues.json rather than assuming one block.

Severity 2 — contradictions and a false premise

3. The bundle lands where it cannot be delivered. deliverables.md names .dev/audits/<date>-<subject>/; SKILL.md's working-directory table (added in #17) lists the bundle under --out. The run followed SKILL.md, so the bundle sits in gitignored scratch and cannot be committed without moving it. The two docs now contradict each other. The convention half is a suggestion at QuantEcon/QuantEcon.manual#140; the plugin half is deciding whether --out is a working directory, a delivery destination, or both.

4. The cost claim is wrong by an order of magnitude, and doctrine §4 rests on it. 230 items took ~23 minutes end to end (snapshot 11 s, verify 9 min, relate 3 min, write 10 min). audit/README.md promises "hours, not minutes, on a repo with a hundred items" and SKILL.md "a multi-hour run".

This one cuts against machinery this repo argued for. doctrine §4 justifies phase checkpointing on "bulk audits outlive sessions" — at 23 minutes they do not. The honest correction is not to delete checkpointing (a 1000-item tracker scales up, interruption remains possible, and the incremental log proved independently useful as an audit trail) but to stop justifying it with a premise the first measurement contradicts, and to replace the estimate with a measured one.

Severity 3 — cheap fixes

5. Provenance links point at a machine-local plugin cache. The bundle cites ../../../.claude/plugins/cache/quantecon/audit/0.1.2/references/doctrine.md — dead for every reader on another machine, in a document meant to be committed into the audited repo. deliverables.md should say to cite the version-pinned GitHub URL.

6. Preflight does not check token scopes. The run's one conclusion-changing gap — Projects v2 membership — came from a token lacking read:project. The audit handled it well: stated in the residue, marked conclusion-changing rather than additive, with gh auth refresh -s read:project named as the remedy. But fetch_tracker.py preflight already fails fast on missing gh and missing auth, and could check scopes in the same first minute.

7. A location over-claimed in the summary. The concluding summary reported selectForwardCandidates "citing the issue number in its docstring"; the #106 references are in a test file. Substance right, location wrong — the same class of error as defect 1, one notch less consequential.

The gap this run did not close

Resumability is still untested. Tutorial step 4 asks the operator to interrupt phase 2 and restart; the run went straight through. findings.md shows a clean single pass, which proves the checkpoint materialises — it did not exist at all before #17 — but not that resuming from it works. Defect 2 means it would have misbehaved had the run died in the closed-set pass.

Run 2 should be interrupted deliberately, and should be a different repo type per #16's matrix.

Suggested sequencing

Defects 1 and 2 are one doctrine-level PR; both concern whether an audit's own record can be trusted. Defect 3 waits on QuantEcon/QuantEcon.manual#140. Defect 4 is a docs correction plus a decision about what §4 should claim instead. Defects 5–7 are small and independent.

[mmcky]

Defect 4 — option A done in #24 (audit 0.1.3): duration claims corrected in four places, and doctrine §4 re-derived from three grounds that hold at any duration, strongest being that the checkpoint is evidence — run 1's 56-of-56 coverage reconciliation was done against findings.md rather than the report.

The replacement justification is better than the one it replaces because it can be checked, which is what §1 rule 8 asks of every other claim in that document. §4 is renamed Checkpointing accordingly.

Option B — whether the phase apparatus should shrink — is #23, deferred to run 2 on the grounds that one measurement cannot distinguish "over-built" from "easy repo". It carries a concrete instance: phase 3's links.md was superseded eight minutes later, in-session, without ever being read.

Defects 1, 2, 3, 5, 6 and 7 remain open.