audit: resolve the read-only/working-directory contradiction, and a tutorial for running one

.claude-plugin/marketplace.json

@@ -21,7 +21,7 @@
     {
       "name": "audit",
       "source": "./audit",
-      "version": "0.1.1",
+      "version": "0.1.2",
       "description": "Bulk, read-only audits of a QuantEcon repository — issue triage, PR review, technical debt, translation parity — each producing an evidence-cited report bundle"
     }
   ]

.gitignore

@@ -11,3 +11,9 @@ benchmark/references/examples/*/results/env.json
 # MyST docs site build output
 _build/
 node_modules/
+
+# Deliberately uncommitted working notes (AGENTS.md, "Working in this repo").
+# Ignored rather than merely discouraged: it sits in the repo root, so a
+# `git add -A` sweeps it into a commit, and it routinely holds unsent drafts,
+# correspondence and other people's names.
+NEXT-SESSION.md

AGENTS.md

@@ -35,6 +35,7 @@ Before adding a paragraph, check whether it already exists. If it does, link to
 | Using the skills: setup, invocation, what to expect | [docs/using-skills.md](docs/using-skills.md) |
 | Contributing: layout, conventions, dev loop, local testing, versioning, PR flow | [docs/developing-skills.md](docs/developing-skills.md) |
 | Running an evaluation by hand, end to end | [docs/tutorial-run-an-evaluation.md](docs/tutorial-run-an-evaluation.md) |
+| Running a whole-tracker audit, and reviewing what it produces | [docs/tutorial-run-an-audit.md](docs/tutorial-run-an-audit.md) |
 | The benchmark skill: modes, report format, manual pipeline | [benchmark/README.md](benchmark/README.md) |
 | Rubric: dimensions, weights, anchors, verdict bands | [benchmark/references/EVALUATION_FRAMEWORK.md](benchmark/references/EVALUATION_FRAMEWORK.md) |
 | Style rule text and schema | `QuantEcon/style-guide` (upstream — never authored in this repo) |

audit/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "audit",
   "description": "Bulk, read-only audits of a QuantEcon repository — issue triage, PR review, technical debt, translation parity — each producing an evidence-cited report bundle",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "author": { "name": "QuantEcon" }
 }

audit/references/deliverables.md

@@ -12,7 +12,7 @@ What every audit owes its reader is much smaller, and is in the last section.
 2. The paired `project-*` repo, when the audit serves a program rather than a repo (`project-translation/reports/` for a translation audit).
 3. A dedicated `audit-*` repo, per [QEP-3](https://github.com/QuantEcon/qeps/pull/7), once an audit is recurring and its findings are worth publishing.
 
-Confirm the destination before writing, and name it in the report. Private inputs stay in private destinations: a `project-*` repo's contents must not be summarised into a public one.
+Confirm the destination before writing, and name it in the report. Writing the bundle there is the audit's job; **committing or publishing it is not** — that is a human step, taken after reading it ([doctrine §3](doctrine.md#3-read-only-boundary)). Private inputs stay in private destinations: a `project-*` repo's contents must not be summarised into a public one.
 
 ## The `/audit:issues` bundle
 

audit/references/doctrine.md

@@ -27,10 +27,12 @@ An unqualified claim is a defect. `[inferred]` is legitimate and often the most
 
 ## 3. Read-only boundary
 
-**These skills observe and report. They never mutate the tracker** — no closing, labelling, milestoning, commenting, or editing, and no branch or file changes in the audited repo.
+**These skills observe and report. They never mutate what they audit** — no closing, labelling, milestoning, commenting or editing on the tracker, and in the audited repo no commits, no pushes, no branches, and no changes to tracked files.
 
 That boundary is what makes the family safe to run headlessly and safe to point at any repo in the org. Every action an audit recommends lands as a proposal in the report, executed later by a human or by a separate, deliberately-invoked step. An audit that half-applied its own findings would also be an audit whose report no longer describes the repo it was run against.
 
+**A run may write its own working directory**, including inside the audited checkout. An audit that produces a report necessarily produces files, and what this boundary protects is the repo's *content and history*, not its untracked filesystem. So: the working directory stays untracked, nothing is added to `.gitignore` — that would itself be an edit to a tracked file — and committing or publishing the bundle is a human step taken after reading it. Prefer a location the repo already ignores where one exists; a skill's own working-directory rule says where. This resolves a contradiction with [deliverables.md](deliverables.md), which delivers the bundle *into* the audited repo's notes system: the rule is about mutation, not about writing.
+
 Corollary: an audit never writes into the plugin directory, and its outputs never live in `QuantEcon/skills`. See [deliverables.md](deliverables.md) for where they go.
 
 ## 4. Surviving a long run

audit/skills/issues/SKILL.md

@@ -36,7 +36,13 @@ Then, in the audited repo: the notes system, `CHANGELOG.md`, the latest release
 
 ## Working directory
 
-Everything the run produces goes under `--out`, defaulting to `.audit/<repo>-<YYYY-MM-DD>/`:
+Runs happen in a checkout of the audited repo, because phase 2 verifies claims against its default branch. **Put the working directory somewhere the repo already ignores**, so a run leaves `git status` clean and no `.gitignore` edit is needed ([doctrine §3](../../references/doctrine.md#3-read-only-boundary)) — in that order:
+
+1. An ignored scratch location in the notes system — `.dev/scratch/audit-<YYYY-MM-DD>/` in QuantEcon repos that have one, where `.dev/scratch/*` is already gitignored.
+2. `.audit/<repo>-<YYYY-MM-DD>/` at the checkout root, left untracked and uncommitted.
+3. Outside the checkout entirely, if the repo would show either as dirty.
+
+State which was used in the report's method section. Everything the run produces goes under it:
 
 | Path | Written by | Holds |
 |---|---|---|

docs/tutorial-run-an-audit.md

@@ -0,0 +1,110 @@
+# Tutorial: run a whole-tracker audit
+
+This walks `/audit:issues` end to end against **[QuantEcon/action-translation](https://github.com/QuantEcon/action-translation)** — 228 items, the repo the runbook was first executed against by hand.
+
+It differs from the [evaluation tutorial](tutorial-run-an-evaluation.md) in one important way. That one reproduces a committed reference, so every number you produce can be checked. Here there is no reference: `/audit:issues` has **never been run as a skill**, only by hand as a runbook, so its method is generalised from a single execution. This run *is* the experiment ([skills#16](https://github.com/QuantEcon/skills/issues/16)), and the part no automation can supply is your judgement of the output. Step 6 is therefore not optional garnish — it is the result.
+
+Canonical references (this tutorial points, never restates): the procedure in [SKILL.md](../audit/skills/issues/SKILL.md), the method in [doctrine.md](../audit/references/doctrine.md), the org conventions in [quantecon-context.md](../audit/references/quantecon-context.md), the output contract in [deliverables.md](../audit/references/deliverables.md).
+
+## What you need
+
+- The `audit` plugin installed (Step 0). It is deliberately not in the lecture-repo auto-install block — it is maintainer tooling.
+- `gh` authenticated. Preflight refuses to start without it, because the anonymous API returns nothing for the org's private repos and is capped at 60 requests/hour.
+- A checkout of the audited repo. Phase 2 verifies claims against its default branch, so a tracker-only run cannot do the job.
+- **Hours, not minutes**, and a session you can afford to interrupt — interrupting it is one of the tests.
+
+## Step 0 — install the plugin
+
+```
+/plugin marketplace add QuantEcon/skills
+/plugin install audit@quantecon
+```
+
+Confirm with `/plugin list`; the version should read **0.1.2**. If `/audit:issues` does not appear in the slash menu, see [using-skills § troubleshooting](using-skills.md#updating-and-troubleshooting) — natural-language invocation ("audit every issue in this repo") works on any version.
+
+## Step 1 — put the working directory where the repo already ignores it
+
+```bash
+cd ~/work/quantecon/action-translation
+git status --short          # must be clean before you start — it is the read-only baseline
+git check-ignore -v .dev/scratch/x    # → .gitignore:… .dev/scratch/*
+```
+
+`action-translation` has a `.dev/` notes system whose `.dev/scratch/*` is already gitignored, which makes it the first-choice working directory: the run leaves `git status` completely clean, and no `.gitignore` edit is needed — that would itself be a change to a tracked file. Repos without one fall back to an untracked `.audit/` at the root ([SKILL.md § Working directory](../audit/skills/issues/SKILL.md)).
+
+## Step 2 — invoke
+
+```
+/audit:issues QuantEcon/action-translation --out .dev/scratch/audit-2026-07-28
+```
+
+Before phase 1 the skill *discovers* its inputs rather than asking for them: the notes system (here `.dev/` — `STATE.md`, `PLAN.md`, `FUTURE.md`, `decisions/`), the label policy, the work-plan anchor to tier against, and any prior audits. It should ask you only where discovery is genuinely ambiguous — two plausible plan anchors, say — and never merely because something came up empty. **Every resolved input must appear in the report's method section**; that is the first thing to check in Step 6.
+
+## Step 3 — phase 1, the snapshot (~11 seconds)
+
+Deterministic, and the only phase driven by a script rather than judgement. Measured 2026-07-27:
+
+```
+fetching issues from QuantEcon/action-translation …
+  116 issues
+fetching pull requests from QuantEcon/action-translation …
+  112 pull requests
+
+  numbers 1..228: 228 accounted, 0 unaccounted
+  open issues     55    49 comments across 30
+  closed issues   61    80 comments across 32
+  open PRs         2    2 comments across 1, 5 reviews across 2
+  closed PRs     110    28 comments across 18, 377 reviews across 105
+```
+
+Three things to read off it. **`0 unaccounted`** means every number in `1..228` was captured — a non-zero count is not automatically wrong (deleted or transferred items, PR numbers burned by branches that never opened) but each one now owes an explanation in Step 5. **No truncation warning** — a stream returning exactly at `--limit` is indistinguishable from a truncated one. And **`meta.json`'s `fetched_by` should be you**, since visibility is per-account.
+
+Your counts will differ from the ones above — the tracker moves (#11 measured 111/110 in July). That is expected and is why the report quotes its own snapshot timestamp rather than an earlier count.
+
+## Step 4 — phase 2, verify — and interrupt it
+
+The long phase: 116 items checked against the default branch rather than against what their threads claim. Findings are appended to `findings.md` **one entry per item, as each is verified**.
+
+**This is the test.** Once 20–30 entries exist, interrupt the session — close it, or press Esc twice. Then open a new session in the same directory and re-invoke the same command. What should happen: it reads `findings.md`, resumes at the lowest issue number with no entry, and re-verifies only the last entry (which may have been half-written). What would be a failure: restarting from item 1, skipping the item it died on, or duplicating entries.
+
+Resumability is asserted in three separate files and has never been tested. Until this run, phases 2 and 3 named no artifacts at all, so a resumed session could only work if it happened to invent the same filename — the fix is [#17](https://github.com/QuantEcon/skills/pull/17), and this is what checks it.
+
+While it runs, `tail findings.md` occasionally. Every status claim should carry `[verified]`, `[stated]` or `[inferred]`, and a `[verified]` should cite `file:line`, a merged PR, or a tag — never a comment.
+
+## Step 5 — phases 3 to 5
+
+Phase 3 writes the cross-link graph to `links.md`. Phase 4 tiers into the plan it discovered in Step 2 and writes the bundle. Phase 5 reconciles against `coverage.json` and folds any correction *back into* the documents rather than appending an erratum.
+
+With 55 open issues this run should produce the **full four-document bundle**; a repo under about 30 open issues should instead fold the catalog and links into the report. That threshold is new and untested, so note whether four documents felt right at 55 or merely dutiful. The bundle's destination is `.dev/audits/<date>-issues/` — the audit writes it there, but **committing it is your call, not the run's**.
+
+## Step 6 — review the output
+
+The run cannot check any of this about itself. Ten checks, the last two of which only you can make:
+
+| # | Check | Where to look | What failure looks like |
+|---|---|---|---|
+| 1 | Method section is complete | report §1 | The plan anchor, label policy and prior-audit search are not all named |
+| 2 | Every claim is tagged | catalog entries | An untagged status claim — a defect by the doctrine's own rule |
+| 3 | `[verified]` means verified | sample 5 "fixed" calls | Citation is a thread comment rather than code, a merged PR, or a tag |
+| 4 | Code beat the thread | sample 5 more | The report repeats "fixed in #204" without saying it checked the branch |
+| 5 | The closed side was read | closed-set verification | Closed issues summarised from title and `stateReason` only; no deferred remainders surfaced |
+| 6 | Coverage is honest | `README.md` index | Unaccounted numbers waved at collectively; residue (inline review comments, GraphQL-only data) not stated |
+| 7 | Siblings were checked | external cross-link registry | No sibling considered, in an org where the same fix lands in several repos as `SYNC:` PRs |
+| 8 | Drafted comments are safe | GC tier | A closing keyword immediately before an `owner/repo#N` reference — that closes the *upstream* item when it lands |
+| 9 | **Is the tiering right?** | report tiering section | T0 that isn't this week's work, or a tier list not actually tied to `.dev/PLAN.md`. You are the authority; the run is guessing |
+| 10 | **Would you act on it?** | the whole bundle | The only check that matters, and the only one no self-audit can make |
+
+Then confirm the boundary held: `git status --short` shows nothing but your ignored working directory, and the tracker is unchanged — no comments, no labels, no closures.
+
+## Step 7 — record the run
+
+Findings belong in this repo; the bundle does not. Write `reviews/audit-run-action-translation-2026-07-28.md`, following the [ge_arrow validation run](../reviews/validation-run-ge_arrow-2026-07-22.md):
+
+- **Setup** — repo, snapshot timestamp, `fetched_by`, item counts, unaccounted numbers, plugin version.
+- **Cost** — wall clock and rough token spend per phase, so "expect hours" stops being a guess.
+- **The ten checks** from Step 6, each held or broken, with the evidence.
+- **Interruption log** — where you killed it, what resuming actually did.
+- **What the doctrine did and did not transfer** — the payload. A rule that was cited and load-bearing, a rule that never came up, and a rule the run had to work around are three different verdicts, and only the third is a bug.
+- **Your read of the bundle** — checks 9 and 10 in prose.
+
+Post the summary to [skills#16](https://github.com/QuantEcon/skills/issues/16). Anything that broke becomes an issue against the plugin, not a note in the margin.

docs/using-skills.md

@@ -12,6 +12,7 @@ For lecture authors, reviewers, and maintainers who want to *use* the skills. (B
 /plugin marketplace add QuantEcon/skills
 /plugin install qe@quantecon           # author-facing base skills
 /plugin install benchmark@quantecon    # lecture-evaluation tooling
+/plugin install audit@quantecon        # maintainer-facing bulk audits
 ```
 
 **In CI** — `anthropics/claude-code-action@v1` accepts `plugin_marketplaces` and `plugins` inputs directly; see the repo README for the workflow snippet.
@@ -36,6 +37,7 @@ Three ways, all equivalent:
 |---|---|---|---|
 | `qe` | `/qe:check-style` + `check-{writing,math,code,figures,jax,refs}` | Style-guide compliance for lecture source, by rule ID | scaffolding — [skills#3](https://github.com/QuantEcon/skills/issues/3) |
 | `benchmark` | `/benchmark:review-acceleration` | Score a NumPy→JAX/Numba conversion (review mode) or assess whether a lecture is worth converting (triage mode) | system landed — [guide](../benchmark/README.md), [skills#4](https://github.com/QuantEcon/skills/issues/4) |
+| `audit` | `/audit:issues` | Sweep a whole tracker: verify each issue's status against the code rather than the thread, tier the open set into the repo's plan, deliver a report bundle. Read-only — it recommends, never applies | runbook landed, method not yet validated — [tutorial](tutorial-run-an-audit.md), [guide](../audit/README.md), [skills#16](https://github.com/QuantEcon/skills/issues/16) |
 
 ## Updating and troubleshooting
 

myst.yml

@@ -22,6 +22,7 @@ project:
         - file: docs/using-skills.md
         - file: docs/developing-skills.md
         - file: docs/tutorial-run-an-evaluation.md
+        - file: docs/tutorial-run-an-audit.md
     - title: Plugins
       children:
         - file: benchmark/README.md