From 133c8fd4e9fe93b4ac06ec78ab3563e39b92a5f5 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:10:30 +1000 Subject: [PATCH 01/10] Audit plugin: the bulk-audit family and its shared doctrine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A third plugin for maintainer-facing work: portfolio-wide, read-only sweeps of one repository that deliver a report bundle. Membership is gated on three tests together — bulk, read-only, bundle output — which keeps single-item review outside the family and stops the plugin becoming a general runbook dump. The method is authored once at plugin level so the four planned skills share it rather than restating it: doctrine.md (trust rules, evidence classes, the read-only boundary, phased checkpointing, the coverage self-audit), quantecon-context.md (repo types, label ownership, the cross-repo graph, notes-system discovery, access), and deliverables.md (the bundle contract and where bundles may land). Read-only is structural rather than cautious. It is what makes the family safe to point at any repo and safe to run headlessly, and it keeps a report honest: an audit that half-applied its own findings would describe a repo that no longer exists. Co-Authored-By: Claude Opus 5 --- audit/.claude-plugin/plugin.json | 6 +++ audit/README.md | 57 +++++++++++++++++++++++++ audit/references/deliverables.md | 37 +++++++++++++++++ audit/references/doctrine.md | 60 +++++++++++++++++++++++++++ audit/references/quantecon-context.md | 53 +++++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100644 audit/.claude-plugin/plugin.json create mode 100644 audit/README.md create mode 100644 audit/references/deliverables.md create mode 100644 audit/references/doctrine.md create mode 100644 audit/references/quantecon-context.md diff --git a/audit/.claude-plugin/plugin.json b/audit/.claude-plugin/plugin.json new file mode 100644 index 0000000..bd3a268 --- /dev/null +++ b/audit/.claude-plugin/plugin.json @@ -0,0 +1,6 @@ +{ + "name": "audit", + "description": "Bulk, read-only audits of a QuantEcon repository — issue triage, PR review, technical debt, translation parity — each producing an evidence-cited report bundle", + "version": "0.1.0", + "author": { "name": "QuantEcon" } +} diff --git a/audit/README.md b/audit/README.md new file mode 100644 index 0000000..1cbd7ec --- /dev/null +++ b/audit/README.md @@ -0,0 +1,57 @@ +# audit + +Bulk audits of a QuantEcon repository. Where `qe` serves an author working on one lecture and `benchmark` evaluates one conversion, these skills sweep a whole portfolio — every issue, every PR, a whole codebase, a whole translated series — and deliver a report. + +## Skills + +| Skill | Audits | Status | +|---|---|---| +| [`/audit:issues`](skills/issues/SKILL.md) | Every issue, open and closed: status verified against the code, tiered into the repo's plan | runbook landed | +| `/audit:prs` | Every open PR: does it solve a real issue, is it mergeable, what should the review say | planned | +| `/audit:tech-debt` | A codebase's accumulated debt, with a filing-ready issue catalog | planned | +| `/audit:translations` | Parity between a source series and a translation (`lecture-python.myst` ↔ `lecture-python.zh-cn`) | planned | + +Planned skills are tracked in [CATALOG.md](https://github.com/QuantEcon/skills/blob/main/CATALOG.md); this plugin ships with the first one so the shared doctrine is proven against a real procedure before three more are written on top of it. + +## What belongs here + +Three tests, all of which must pass: + +1. **Bulk** — it sweeps a portfolio, not an item. Reviewing one PR's technical quality is not an audit; reviewing all of them is. +2. **Read-only** — it observes and reports. No skill here mutates a tracker, a branch, or a file in the audited repo ([doctrine §3](references/doctrine.md#3-read-only-boundary)). +3. **Report bundle** — the output is the four-document bundle in [deliverables.md](references/deliverables.md), not a chat answer. + +The read/write line is deliberate and mirrors the org's own automation split, where the family boundary *is* the permission boundary. Anything that acts on findings — filing the catalog as issues, posting the drafted comments, applying labels — is a separate human-invoked step, which is what makes this family safe to point at any repo and safe to run headlessly. + +## Shared references + +Skills are thin; the method lives once at plugin level. + +| Document | Owns | +|---|---| +| [references/doctrine.md](references/doctrine.md) | Trust rules, evidence classes, read-only boundary, phases, coverage self-audit | +| [references/quantecon-context.md](references/quantecon-context.md) | Repo types, label ownership, the cross-repo graph, notes-system discovery, access | +| [references/deliverables.md](references/deliverables.md) | The report bundle contract and where bundles are allowed to land | +| [scripts/](scripts/) | Deterministic phase-1 machinery | + +## Running one + +``` +/audit:issues QuantEcon/action-translation +``` + +Audits are long. They work from a frozen snapshot and checkpoint each phase to disk, so a run that loses its session resumes at the last completed phase rather than restarting — and every number in the report refers to one point in time. Expect hours, not minutes, on a repo with a hundred items. + +Headless runs work the same way: + +```yaml +- uses: anthropics/claude-code-action@v1 + with: + plugin_marketplaces: "https://github.com/QuantEcon/skills.git" + plugins: "audit@quantecon" + prompt: "/audit:issues QuantEcon/action-translation" +``` + +## A note on naming + +The repo's convention is verb-first skill names, and these are nouns. The verb is the plugin: `/audit:issues` and `/audit:translations` read as commands at the point of use, which is where the convention's purpose lies, and `audit-issues` inside an `audit` plugin would stutter at every invocation. The subject-noun names also keep the four skills parallel, which matters more here than in `qe` — the family is defined by a shared method applied to different subjects. diff --git a/audit/references/deliverables.md b/audit/references/deliverables.md new file mode 100644 index 0000000..a478ccf --- /dev/null +++ b/audit/references/deliverables.md @@ -0,0 +1,37 @@ +# The report bundle + +Every audit in this plugin delivers the same four-document bundle. A shared shape means a reader who has read one audit can read any of them, and a later audit can diff against an earlier one mechanically (doctrine rule 7). + +## Where it goes + +**Never into `QuantEcon/skills`.** This repo holds the procedure; the findings belong with the thing audited. In order of preference: + +1. The audited repo's own notes system — `.dev/audits/-/` or equivalent. +2. The paired `project-*` repo, when the audit serves a program rather than a repo (`project-translation/reports/` for a translation audit). +3. A dedicated `audit-*` repo, per [QEP-3](https://github.com/QuantEcon/qeps/pull/7), once an audit is recurring and its findings are worth publishing. + +Confirm the destination before phase 4 and name it in the report. Private inputs stay in private destinations: a `project-*` repo's contents must not be summarised into a public one. + +## The four documents + +**`01--report.md`** — the argument. Method and evidence base; the snapshot timestamp; portfolio statistics; the findings that change a status, each with its evidence tag; the tiering, tied to the repo's existing plan; policy alignment; and a time-boxed execution order for whoever acts on it. + +**`02--catalog.md`** — the enumeration. A legend, then a full summary table (item · tier · action · type · priority), then one entry per open item: + +> **Bold header line** — the item and its one-line characterisation +> *Status:* what is actually true, with evidence and its tag +> *Recommend:* the proposed action, and what it depends on +> *Links:* related items, in and out of this repo + +Close with the verification of the closed set (explicitly thread-complete), and write up anything agreed-but-never-filed as a proposed new item, in a form that could be filed as-is. + +**`03--links.md`** — the graph. Clusters and their anchor items; a concrete table of missing links worth adding; true orphans and over-dense hubs; and the external cross-link registry (sibling repos, program and meta issues, evidence PRs). + +**`README.md`** — the index. Headline numbers, what each document is for, and **the coverage statement**: exactly what was and was not inspected, per [doctrine §5](doctrine.md#5-coverage-self-audit). This is the first thing a reader should be able to check and the last thing the audit should write. + +## Rules that apply to all four + +- **Tag every claim** with its evidence class ([doctrine §2](doctrine.md#2-evidence-classes)). An untagged status claim is a defect. +- **Recommendations are proposals.** Write them in the imperative for whoever executes ("close, citing PR #204"), but the audit does not execute them. +- **Suggested comment text is a draft, in a fenced block, clearly marked as unsent** — and subject to the closing-keyword hazard in [quantecon-context.md](quantecon-context.md#the-cross-repo-graph-is-the-point). +- **Date the bundle and name the snapshot.** Two audits of the same repo must be distinguishable at a glance, and every number in a bundle refers to one point in time. diff --git a/audit/references/doctrine.md b/audit/references/doctrine.md new file mode 100644 index 0000000..8391bb8 --- /dev/null +++ b/audit/references/doctrine.md @@ -0,0 +1,60 @@ +# Audit doctrine + +The method every skill in this plugin follows. Skills own their subject matter — what to verify, how to tier, what the link graph means — and inherit everything here. Authored once; a skill that needs a rule cites this file rather than restating it. + +## 1. What makes a bulk audit trustworthy + +1. **Never trust an artifact's claim about its own state.** Every "fixed in X", "scheduled into Y", "superseded by Z" is a hypothesis until checked against the current default branch, `CHANGELOG.md`, and merged PRs. Cite `file:line`, a PR, or a tag for each verified claim. +2. **Check both directions.** Open items that are actually resolved, *and* closed items whose fix never really landed. An audit that only prunes is half an audit. +3. **Read every thread, closed ones included.** Closed threads carry decisions, deferred remainders, and agreed-but-never-filed work. Skipping the closed side was the one gap in this runbook's first execution, and it is the default failure mode — the snapshot captures closed threads precisely so there is no cost excuse. +4. **Distinguish guarded from fixed.** "Now fails loudly" is not "root cause resolved". Record which kind of remainder each item carries. +5. **Respect explicit maintainer intent.** A "keeping this open because…" comment beats tidiness. Note it and move on; never recommend against a stated decision without new evidence. +6. **Draft policies are conditional.** Where the governing policy is itself a draft or an open PR, recommend only what is already canonical, mark the rest *post-acceptance*, and never pre-empt the policy's own tooling. +7. **Diff against prior audits.** If an earlier audit exists, reconcile with it: convergence is corroboration, divergence needs an explanation. +8. **Match confidence to inspection.** Every claim's evidence class must be recoverable from the writeup. + +## 2. Evidence classes + +Findings are tagged so a reader can tell what was actually inspected. This is the machine-checkable form of rule 8, and it is what lets a reviewer trust a long report without re-running it. + +| Tag | Means | Minimum citation | +|---|---|---| +| `[verified]` | Checked against code, a diff, or a build artifact | `file:line`, a merged PR, or a tag | +| `[stated]` | Asserted by a human in a thread or a notes file | Comment URL or `file:line` | +| `[inferred]` | The audit's own reasoning across sources | The sources it reasons from | + +An unqualified claim is a defect. `[inferred]` is legitimate and often the most valuable class — but it must never be dressed as `[verified]`, and a status change recommended on `[inferred]` alone should say so in the recommendation itself. + +## 3. Read-only boundary + +**These skills observe and report. They never mutate the tracker** — no closing, labelling, milestoning, commenting, or editing, and no branch or file changes in the audited repo. + +That boundary is what makes the family safe to run headlessly and safe to point at any repo in the org. Every action an audit recommends lands as a proposal in the report, executed later by a human or by a separate, deliberately-invoked step. An audit that half-applied its own findings would also be an audit whose report no longer describes the repo it was run against. + +Corollary: an audit never writes into the plugin directory, and its outputs never live in `QuantEcon/skills`. See [deliverables.md](deliverables.md) for where they go. + +## 4. Phases, and surviving a long run + +Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep. Each skill runs the same five phases, and **each phase writes its output to the working directory before the next begins**, so a lost session resumes at the last completed phase instead of restarting. + +| Phase | Produces | Resumable from | +|---|---|---| +| 1. Snapshot | `meta.json`, `issues.json`, `prs.json`, `coverage.json` | — | +| 2. Verify | per-item findings with evidence tags | the snapshot | +| 3. Relate | the cross-link / parity graph | phase 2 | +| 4. Write | the report bundle | phases 2–3 | +| 5. Self-audit | the coverage statement, folded back in | all of the above | + +Phase 1 is deterministic and belongs in [`../scripts/`](../scripts/), not in model judgement. Fetching once also **freezes the audit's point in time**: every later claim refers to the snapshot, and "events after the snapshot" becomes a stated property of the report rather than an unnoticed gap. Record the snapshot timestamp in the report; never silently mix fresh API reads into a later phase. + +## 5. Coverage self-audit + +Before delivering, reconcile — and **close any gap found rather than merely disclosing it**. `coverage.json` from phase 1 does the mechanical half: items captured against the number sequence `1..max`, threads captured against items with comments, and a truncation flag when a stream returns exactly at the fetch limit. + +The judgement half is the skill's: + +- Explain every unaccounted number. Gaps are usually deleted or transferred items, or numbers burned by branches that never opened a PR — but an unexplained gap looks identical to a silent truncation. +- State the residue explicitly: what was *not* inspected. PR review threads? Native sub-issue and Projects membership, both invisible to REST? Anything after the snapshot time? +- If the self-audit changes a finding, fold the change back into the documents. Never append a correction section — a report whose conclusions and body disagree is worse than one that is simply late. + +**No silent caps.** If the audit bounded its own coverage anywhere — sampled, capped at top-N, skipped a class of item — say so in the coverage statement. Silent truncation reads as complete coverage, which is the one failure an auditor cannot recover from. diff --git a/audit/references/quantecon-context.md b/audit/references/quantecon-context.md new file mode 100644 index 0000000..1059d1d --- /dev/null +++ b/audit/references/quantecon-context.md @@ -0,0 +1,53 @@ +# Auditing a QuantEcon repository + +What a generic tracker audit gets wrong about this org, and the conventions every skill in this plugin resolves against. Policies are cited, never restated — several are live drafts, and a copy here would be a copy that drifts. + +## Repo type sets the expectations + +[QEP-3 (Draft, qeps#7)](https://github.com/QuantEcon/qeps/pull/7) names the repo families. Type is the strongest prior an audit has about what its issues *mean*, and tiering that ignores it will mis-rank almost everything: + +| Family | What the tracker mostly holds | Audit consequence | +|---|---|---| +| `lecture-*` | Content corrections, build breakage from environment upgrades, style campaigns | Build-breaking issues are release-gating; content issues rarely are. Many are one instance of a campaign — check siblings before tiering. | +| `action-*`, `workflow-*` | Consumer-visible behaviour, release gates | Anything a downstream repo consumes via `uses:` is higher-stakes than its thread suggests. Verify against the tag, not `main`. | +| `status-*` | Machine-updated reporters | The data is generated; issues are about the collector or the contract. Never recommend edits to generated data. | +| `project-*` | Narrative, decisions, roadmaps (private) | Issues are decision records. "Stale" often means *decided and not written up*, which is a re-homing action, not a close. | +| `QuantEcon/meta` | Org-wide coordination | An issue in a repo that is really org-wide belongs here; an issue here that is really repo-local belongs there. Flag the mismatch, don't move it. | + +If the repo's own type is ambiguous, say so once and audit it as the closest family rather than inventing a sixth. + +## Labels: recommend, never apply + +The standard set is [QEP-2 (Draft, qeps#2)](https://github.com/QuantEcon/qeps/pull/2) — 19 core labels plus 2 lecture-extension labels — re-recording the settled decision in [meta#324](https://github.com/QuantEcon/meta/issues/324). Because QEP-2 is still a draft, doctrine rule 6 applies: recommend only already-canonical labels, mark anything else *post-acceptance*. + +Application is owned by the `qe` CLI, not by an audit: + +``` +qe gh labels check # read-only drift report, non-zero exit on drift +``` + +An audit may report label drift and cite that command. It must never hand-apply or hand-prune labels — profiles (`lectures`, `software`) decide which repos carry which subset, and hand-editing desynchronises a repo from its profile in a way the next `sync` will silently revert. + +## The cross-repo graph is the point + +QuantEcon's lecture repos are near-siblings, and the same fix lands in several of them as `SYNC:` PRs. A single-repo audit that ignores this produces confident, wrong conclusions — most often "this is stale" about an issue whose fix landed in a sibling, or "unique" about the fourth instance of a known campaign. + +For every open item, ask: does a sibling repo carry the same issue, and has it been resolved there? Is this one step of a rollout whose other steps are elsewhere? Is there a program or meta issue that owns the whole family? Record these in the report's external cross-link registry, and treat a sibling's merged fix as `[verified]` evidence only after checking that *this* repo's default branch actually contains it — a wave escape is exactly a fix that was claimed for a repo and never landed in it. + +**Hazard when writing cross-repo references.** Any suggested comment or issue body the audit drafts must avoid a GitHub closing keyword (`close(s|d)`, `fix(es|ed)`, `resolve(s|d)`) immediately before an `owner/repo#N` reference. GitHub's auto-linker treats that as a closing reference and will close the *referenced* upstream item when the text lands on a default branch. Use neutral wording — "Mirrors the change in `owner/repo#N`", "See `owner/repo#N`", "Ports the fix from `owner/repo#N`". Plain same-repo `#N` links are unaffected. + +## Finding the plan to slot into + +Doctrine says slot findings into the repo's existing plan, never invent a parallel one. QuantEcon repos keep that plan in one of several places; look in this order and state which one was used: + +1. `.dev/` — `STATE.md`, `PLAN.md`, `FUTURE.md`, `decisions/`, `log/` +2. Root-level `PLAN.md`, `ROADMAP.md`, `NEXT-STEPS.md`, `CATALOG.md` +3. `AGENTS.md` / `CLAUDE.md` — often the only durable convention record in a small repo +4. The paired `project-*` repo, for work that belongs to a program rather than a repo +5. A live work-plan or tracking issue in the tracker itself + +If none exists, say so and tier against the repo's milestones instead. Proposing a notes system is out of scope — that is a change to the repo, and audits do not change repos. + +## Access + +Authenticated `gh` is assumed. Several QuantEcon repos are private (`project-*`, `style-guide`, `cli`, `jupyteach`, `atlas.quantecon.org`), so the unauthenticated fallbacks that work for public repos — the anonymous REST endpoint, `codeload` tarballs, `raw.githubusercontent.com`, scraping embedded JSON out of issue HTML — return nothing there. The snapshot script fails on missing auth for exactly this reason. On a public repo without `gh`, the fallbacks are viable but share a 60 requests/hour per-IP budget: fetch metadata first, comment threads second, and expect to lose the tail. From 6c2cb5d6852b2759ee128ff7e6cfbb88c023cd6e Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:10:30 +1000 Subject: [PATCH 02/10] Audit: deterministic tracker snapshot with coverage reconciliation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 of every audit is mechanical, so it runs as a script rather than as model judgement. fetch_tracker.py captures issues and PRs in any state with full comment threads in two round trips, which is what makes reading closed threads free — the one gap the first execution of the issue runbook found. The snapshot also freezes the audit's point in time, so every later phase reads one instant and "events after the snapshot" becomes a stated property of the report rather than an unnoticed gap. coverage.json does the mechanical half of the self-audit: items against the number sequence 1..max, thread counts split open/closed, and a truncation flag for a stream that returns exactly at the fetch limit — indistinguishable from truncation, so it is surfaced, not swallowed. Preflight refuses to start without gh and auth, because the anonymous API is 60 req/h per IP and returns nothing for the org's private repos. Validated against QuantEcon/action-translation: 111 issues + 110 PRs, numbers 1..221 fully accounted, 123 comments across 60 threads. Co-Authored-By: Claude Opus 5 --- audit/scripts/README.md | 28 +++++ audit/scripts/fetch_tracker.py | 199 +++++++++++++++++++++++++++++++++ 2 files changed, 227 insertions(+) create mode 100644 audit/scripts/README.md create mode 100644 audit/scripts/fetch_tracker.py diff --git a/audit/scripts/README.md b/audit/scripts/README.md new file mode 100644 index 0000000..2937381 --- /dev/null +++ b/audit/scripts/README.md @@ -0,0 +1,28 @@ +# Deterministic audit machinery + +Phase 1 of every audit is mechanical, so it belongs here rather than in model judgement ([doctrine §4](../references/doctrine.md#4-phases-and-surviving-a-long-run)). Stdlib only, driving `gh`; no install step. + +## `fetch_tracker.py` + +Snapshots a repository's whole tracker — issues and PRs, any state, with full comment threads — and reconciles the capture against the number sequence it should fill. + +```bash +python ${CLAUDE_PLUGIN_ROOT}/scripts/fetch_tracker.py OWNER/REPO --out /snapshot +``` + +| Output | Contents | +|---|---| +| `meta.json` | Repo, snapshot time (UTC), `gh` version, auth state, the fields requested | +| `issues.json` | Every issue, any state, each with its comment thread | +| `prs.json` | Every PR, with comments, reviews, and `closingIssuesReferences` | +| `coverage.json` | Items against `1..max`, thread counts by open/closed, truncation flag | + +Three properties the audit depends on: + +- **Thread-complete on the closed side.** Comment threads come down inside the list call, so reading closed threads — doctrine rule 3, and the gap the first execution of this runbook found — costs nothing extra. +- **Frozen in time.** Every later phase reads this snapshot, so the report describes one instant and "events after the snapshot" is a stated property rather than a silent gap. +- **Honest about its own limits.** `unaccounted_numbers` lists gaps in the number sequence for the audit to explain, and `truncation_risk` flags a stream that returned exactly at `--limit`, which is indistinguishable from truncation. Neither is swallowed. + +Preflight refuses to start without `gh` and authentication, because the anonymous API is 60 requests/hour per IP and returns nothing at all for the org's private repos. + +Not captured, because REST does not return them: native sub-issue links and Projects membership. Fetch those with `gh api graphql` when a tracker issue depends on them, and disclose them in the coverage statement regardless. diff --git a/audit/scripts/fetch_tracker.py b/audit/scripts/fetch_tracker.py new file mode 100644 index 0000000..9e1e2fb --- /dev/null +++ b/audit/scripts/fetch_tracker.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python3 +"""Snapshot a repository's whole issue and PR tracker to disk, then reconcile +what was captured against what should exist. + + python fetch_tracker.py QuantEcon/action-translation --out .audit/action-translation + +Why a script instead of letting the skill run `gh` ad hoc: a portfolio audit is +long, and every phase after the first re-reads the same data. Fetching once into +a frozen snapshot makes the run cheap, makes it resumable after a lost session, +and pins every later claim to a single point in time — so "events after scrape +time" is a stated property of the report rather than an unnoticed gap. + +Comment threads come down inside the same list call, so the snapshot is +thread-complete for both open *and* closed items in two API round trips. + +Writes into `--out`: + + meta.json repo, snapshot time (UTC), gh version, auth state, counts + issues.json every issue, any state, with its full comment thread + prs.json every PR, any state, with comments, reviews, and closing refs + coverage.json the mechanical half of the coverage self-audit + +Stdlib only, so it runs anywhere `gh` does. +""" + +import argparse +import json +import shutil +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +ISSUE_FIELDS = [ + "number", "title", "state", "stateReason", "labels", "milestone", + "assignees", "author", "body", "comments", "createdAt", "updatedAt", + "closedAt", "url", +] + +PR_FIELDS = [ + "number", "title", "state", "labels", "milestone", "author", "body", + "comments", "reviews", "createdAt", "updatedAt", "closedAt", "mergedAt", + "mergeCommit", "headRefName", "baseRefName", "url", + "closingIssuesReferences", +] + + +def die(msg): + print(f"fetch_tracker: {msg}", file=sys.stderr) + sys.exit(1) + + +def run_gh(args): + """Run a gh subcommand, returning parsed JSON.""" + proc = subprocess.run( + ["gh", *args], capture_output=True, text=True, + ) + if proc.returncode != 0: + die(f"`gh {' '.join(args)}` failed:\n{proc.stderr.strip()}") + try: + return json.loads(proc.stdout) + except json.JSONDecodeError as exc: + die(f"`gh {' '.join(args)}` returned invalid JSON — {exc}") + + +def preflight(repo): + """Fail in the first minute, not the third hour.""" + if shutil.which("gh") is None: + die("the GitHub CLI (`gh`) is not on PATH — install it, or fall back to " + "the unauthenticated route documented in the skill") + auth = subprocess.run( + ["gh", "auth", "status"], capture_output=True, text=True, + ) + if auth.returncode != 0: + die("`gh auth status` reports no authentication. Private QuantEcon repos " + "(project-*, style-guide, cli) are unreachable unauthenticated, and " + "the anonymous API allows only 60 requests/hour. Run `gh auth login`.") + if repo.count("/") != 1 or not all(repo.split("/")): + die(f"expected OWNER/REPO, got {repo!r}") + # Confirms the repo exists and is visible to these credentials. + run_gh(["repo", "view", repo, "--json", "name"]) + return auth.stdout + auth.stderr + + +def reconcile(issues, prs, limit): + """Reconcile captured items against the number sequence they should fill. + + GitHub draws issue and PR numbers from one per-repo sequence, so a complete + capture accounts for every number from 1 to the highest one seen. Gaps are + real (deleted or transferred items, and PR numbers burned by never-opened + branches) but they are exactly what a silent truncation also looks like — + so they are reported as unaccounted numbers for the audit to explain, never + swallowed here. + """ + issue_numbers = {i["number"] for i in issues} + pr_numbers = {p["number"] for p in prs} + seen = issue_numbers | pr_numbers + highest = max(seen) if seen else 0 + unaccounted = [n for n in range(1, highest + 1) if n not in seen] + + def thread_stats(items): + with_comments = [i for i in items if i.get("comments")] + return { + "total": len(items), + "with_comments": len(with_comments), + "comments_captured": sum(len(i.get("comments") or []) for i in with_comments), + } + + open_issues = [i for i in issues if i["state"].upper() == "OPEN"] + closed_issues = [i for i in issues if i["state"].upper() != "OPEN"] + + return { + "highest_number": highest, + "items_captured": len(seen), + "issues": len(issues), + "prs": len(prs), + "unaccounted_numbers": unaccounted, + "unaccounted_count": len(unaccounted), + "threads_open_issues": thread_stats(open_issues), + "threads_closed_issues": thread_stats(closed_issues), + "threads_prs": thread_stats(prs), + "truncation_risk": { + "limit": limit, + "issues_at_limit": len(issues) >= limit, + "prs_at_limit": len(prs) >= limit, + }, + } + + +def main(): + parser = argparse.ArgumentParser( + description="Snapshot a repo's full issue/PR tracker for a bulk audit.", + ) + parser.add_argument("repo", help="OWNER/REPO, e.g. QuantEcon/action-translation") + parser.add_argument("--out", required=True, help="snapshot directory to write") + parser.add_argument( + "--limit", type=int, default=1000, + help="max items per stream (default 1000); coverage.json flags a stream " + "that comes back at the limit, because that is indistinguishable " + "from truncation", + ) + args = parser.parse_args() + + auth_note = preflight(args.repo) + out = Path(args.out) + out.mkdir(parents=True, exist_ok=True) + + print(f"fetching issues from {args.repo} …") + issues = run_gh([ + "issue", "list", "-R", args.repo, "--state", "all", + "--limit", str(args.limit), "--json", ",".join(ISSUE_FIELDS), + ]) + print(f" {len(issues)} issues") + + print(f"fetching pull requests from {args.repo} …") + prs = run_gh([ + "pr", "list", "-R", args.repo, "--state", "all", + "--limit", str(args.limit), "--json", ",".join(PR_FIELDS), + ]) + print(f" {len(prs)} pull requests") + + gh_version = subprocess.run( + ["gh", "--version"], capture_output=True, text=True, + ).stdout.splitlines()[0] + + coverage = reconcile(issues, prs, args.limit) + meta = { + "repo": args.repo, + "snapshot_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "gh_version": gh_version, + "authenticated": "Logged in" in auth_note, + "limit": args.limit, + "issue_fields": ISSUE_FIELDS, + "pr_fields": PR_FIELDS, + } + + (out / "issues.json").write_text(json.dumps(issues, indent=2) + "\n") + (out / "prs.json").write_text(json.dumps(prs, indent=2) + "\n") + (out / "coverage.json").write_text(json.dumps(coverage, indent=2) + "\n") + (out / "meta.json").write_text(json.dumps(meta, indent=2) + "\n") + + print(f"\nsnapshot written to {out}/ ({meta['snapshot_utc']})") + print(f" numbers 1..{coverage['highest_number']}: " + f"{coverage['items_captured']} accounted, " + f"{coverage['unaccounted_count']} unaccounted") + oi = coverage["threads_open_issues"] + ci = coverage["threads_closed_issues"] + print(f" open issues {oi['total']:>4} ({oi['with_comments']} with threads, " + f"{oi['comments_captured']} comments)") + print(f" closed issues {ci['total']:>4} ({ci['with_comments']} with threads, " + f"{ci['comments_captured']} comments)") + if coverage["truncation_risk"]["issues_at_limit"] or coverage["truncation_risk"]["prs_at_limit"]: + print("\n WARNING: a stream came back exactly at --limit; re-run with a " + "higher --limit before trusting the counts.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From c296d62d9676131f6858d4e6faafac22e7037106 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:10:30 +1000 Subject: [PATCH 03/10] Audit: the issue-triage runbook as /audit:issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runbook's four paste-in fields become optional arguments with documented discovery and stated fallbacks, so the common invocation is just the repo. Its numbered sections split along the doctrine boundary: the trust rules and coverage self-audit move to plugin level, and the skill keeps what is specific to issues — per-item verification, the finding categories to hunt, GC/T0-T3 tiering, and the link graph. QuantEcon adaptations over the generic runbook: tier by repo type, since a build break in a lecture repo and a consumer-visible change in an action repo outrank their thread activity; check siblings before concluding, because "resolved in a sibling" and "one step of a rollout" are the two wrong conclusions a single-repo audit reaches here; leave label application to qe gh labels; and keep GitHub closing keywords out of drafted cross-repo references, which would otherwise close the upstream item when the text lands. Co-Authored-By: Claude Opus 5 --- audit/skills/issues/SKILL.md | 87 ++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 audit/skills/issues/SKILL.md diff --git a/audit/skills/issues/SKILL.md b/audit/skills/issues/SKILL.md new file mode 100644 index 0000000..2ef3cee --- /dev/null +++ b/audit/skills/issues/SKILL.md @@ -0,0 +1,87 @@ +--- +name: issues +description: Audit every issue in a GitHub repository, open and closed — verify each status against the code rather than the thread, hunt fixed-but-open and never-landed-fix candidates, tier the open set into the repo's existing plan, and deliver a report bundle with a cross-link map. Read-only: it recommends tracker changes but never makes them. Use for a whole-tracker review, not a single issue. +--- + +# audit:issues + +Whole-tracker review of one repository. Long-running by design — a hundred-issue repo is a multi-hour run — so it works from a frozen snapshot and checkpoints every phase to disk. + +> **Status: first runbook of the `audit` family.** The procedure below is the QuantEcon-adapted form of a runbook that has been executed once end to end; the gap that execution found (closed threads unread) is now doctrine rule 3 and is closed by the snapshot. Plan and family design: [CATALOG.md §3](https://github.com/QuantEcon/skills/blob/main/CATALOG.md). + +## Invocation + +``` +/audit:issues [OWNER/REPO] [--out DIR] +``` + +Both arguments are optional. With no repo, audit the current checkout's `origin`. Everything else is discovered: + +| Input | Discovery | On failure | +|---|---|---| +| Project-notes system | The search order in [quantecon-context.md](../../references/quantecon-context.md#finding-the-plan-to-slot-into) | Say none was found; tier against milestones | +| Label policy | [QEP-2](https://github.com/QuantEcon/qeps/pull/2), status read at run time | Treat as draft — recommend canonical labels only | +| Work-plan anchor | The live plan or tracking issue in the tracker | Tier against the notes system alone | +| Prior audits | Earlier audit bundles in the notes system; closed "priority order" issues | Note that this is the first audit | + +Ask only when discovery is ambiguous — two plausible plan anchors, say — not when it simply comes up empty. Report every resolved input in the report's method section. + +## Read this first + +- [doctrine.md](../../references/doctrine.md) — trust rules, evidence classes, the read-only boundary, phases, coverage self-audit +- [quantecon-context.md](../../references/quantecon-context.md) — repo types, label ownership, the cross-repo graph, the closing-keyword hazard +- [deliverables.md](../../references/deliverables.md) — the bundle contract + +Then, in the audited repo: the notes system, `CHANGELOG.md`, the latest release notes, and `AGENTS.md`/`CLAUDE.md`. + +## Phase 1 — snapshot + +```bash +python ${CLAUDE_PLUGIN_ROOT}/scripts/fetch_tracker.py OWNER/REPO --out /snapshot +``` + +Preflight fails in the first minute on missing `gh`, missing auth, or an invisible repo — read the error rather than working around it. The snapshot captures issues and PRs in any state with full comment threads, so closed threads cost nothing extra to read and there is no excuse for skipping them. + +Two things REST does not return, so the snapshot cannot hold them: **native sub-issue links** and **Projects membership**. Fetch those with `gh api graphql` where a parent/tracker issue depends on them, and list them in the coverage statement either way. + +Read `coverage.json` before proceeding. A stream at the fetch limit, or unaccounted numbers, is a phase 1 problem — fix it here, not in the writeup. + +## Phase 2 — verify + +Per [doctrine §1](../../references/doctrine.md#1-what-makes-a-bulk-audit-trustworthy), the thread is a hypothesis and the default branch is the evidence. + +**Each open issue.** Does the complaint still reproduce on the default branch — check the named file or function, not the claim about it? Did a merged PR or a campaign claim it, and does the diff actually contain the change (this is where wave escapes hide)? Is it a duplicate of another open issue — same fix, no mutual link? Is it superseded, where a newer issue is the better carrier — then port the lessons across and close the old one as superseded, never silently. Is it blocked on an external decision — annotate the hold and split live from gated scope, so the issue does not read as wholly frozen when part of it is actionable. For parents and trackers: are the children attached natively, and is it still the plan of record? + +**Each closed issue.** Is `stateReason` sensible? Does it map to a shipped PR or a recorded decision? Were the thread's remainders re-homed somewhere still open, or did they die with the thread? + +**Hunt these categories actively** — they are what the audit exists to find: fixed-but-open · scheduled-but-escaped · unlinked duplicates · delivered-elsewhere (close when the dependency lands) · validated-purpose-served · agreed-but-never-filed, where a thread proposal got a nod and no issue · stale notes-system lines, which are reported as nits rather than tracker actions. + +Sibling-repo checks belong here too: for QuantEcon, "resolved in a sibling" and "one step of a rollout" are the two most common wrong conclusions a single-repo audit reaches. + +## Phase 3 — relate + +Parse every `#N` in all issue **and** PR bodies, both directions. Two parsing caveats: a range reference (`#169–#176`) matches its endpoints only, so check milestone membership before declaring the middle numbers orphaned; and a comment thread reconstructed from HTML can start mid-thread, so attribute cautiously. + +Produce the cluster map, the table of missing links worth adding (duplicate pairs, origin↔carrier, complementary checks, family orphans), true orphans and over-dense hubs, and the external cross-link registry. + +## Phase 4 — tier and write + +Slot into the repo's existing plan; never invent a parallel one. Tier by repo type — a build break in a `lecture-*` repo and a consumer-visible change in an `action-*` repo outrank their thread activity. + +| Tier | Contents | +|---|---| +| **GC** | Verified done, superseded, or duplicate. Each gets a one-line closing comment, drafted and unsent. | +| **T0** | This week: live breakage, security exposure, release-gate reds, scheduled plan steps, small verified fixes. | +| **T1** | Aligned with the current phase or milestone of the repo's plan. | +| **T2** | Production quality, grouped by family — issues sharing a fix-shape are one unit of work. | +| **T3** | Decision-gated, external-dependency, deliberately deferred, parking lots. | + +Priority labels only for genuine outliers, a handful either way, and only if the policy provides them. Then write the bundle per [deliverables.md](../../references/deliverables.md), as `01-issue-triage-report.md`, `02-issue-catalog.md`, `03-issue-links.md`, `README.md`. + +## Phase 5 — self-audit + +Run [doctrine §5](../../references/doctrine.md#5-coverage-self-audit) against `coverage.json`: reconcile the counts, explain every unaccounted number, confirm threads were read on both the open and closed sides, and state the residue — PR review threads, GraphQL-only data, anything after the snapshot timestamp. Fold any change back into the documents rather than appending a correction. + +## What this skill does not do + +It does not close, label, comment, or edit — see [doctrine §3](../../references/doctrine.md#3-read-only-boundary). The tracker pass is a separate, human-driven step working from the bundle, and label application belongs to `qe gh labels`. Judging one PR's technical quality is likewise out of scope; that is a per-item review, not a portfolio audit. From 7628643cf31665b7696fa2df46584454a2deb8d2 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:10:30 +1000 Subject: [PATCH 04/10] Docs: register the audit plugin in the marketplace, README, and catalog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Marketplace entry at 0.1.0, catalogue minor-bumped for the new plugin. CATALOG.md §3 records the family design and the evidence behind it, which is breadth rather than per-repo frequency: a tracker audit is a once-a-year event for one repo, but the org has ~245 of them and three of the four procedures have already been run by hand. audit is deliberately left out of the lecture-repo auto-install block. The plugin is the enable unit, so including it would put maintainer tooling in every author's command list — the same audience separation that keeps benchmark out of the qe prefix. Co-Authored-By: Claude Opus 5 --- .claude-plugin/marketplace.json | 12 +++++++++++- CATALOG.md | 12 ++++++++++++ README.md | 11 ++++++++++- 3 files changed, 33 insertions(+), 2 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 5ca793e..eeeb42f 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -1,7 +1,7 @@ { "name": "quantecon", "description": "QuantEcon's shared Claude Code skills and tools", - "version": "0.1.0", + "version": "0.2.0", "plugins": [ { "name": "qe", @@ -22,6 +22,16 @@ }, "version": "0.1.0", "description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code" + }, + { + "name": "audit", + "source": { + "source": "github", + "repo": "QuantEcon/skills", + "path": "audit" + }, + "version": "0.1.0", + "description": "Bulk, read-only audits of a QuantEcon repository — issue triage, PR review, technical debt, translation parity — each producing an evidence-cited report bundle" } ] } diff --git a/CATALOG.md b/CATALOG.md index 20aced4..ff6caf1 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -33,3 +33,15 @@ This delivers the planned `qestyle-linter` + `qestyle` pair as Claude Code skill ## 2. `/benchmark:review-acceleration` — finish Blocked on the eight evaluation scripts (@xuanguang-li agreed 2026-07-07 to package and send). **Unblocked now:** the full rubric (7 weighted dimensions, verdict bands, HIGH/LOW calibration cases) is specified in the lecture-python.myst#717 thread and can move into `references/rubric.md` ahead of the scripts. Validation cases: lecture-python.myst#717 and #654. + +## 3. `audit` — the bulk-audit family + +Portfolio-wide, read-only sweeps of one repository, each delivering a report bundle. Four skills: **`/audit:issues`** (every issue, open and closed — landed), then **`/audit:prs`** (every open PR: does it solve a real issue, is it mergeable, what should the review say), **`/audit:tech-debt`** (a codebase's debt plus a filing-ready issue catalog), **`/audit:translations`** (parity between a source series and its translation). + +**Evidence:** unlike the style family, the case here is not per-repo frequency — a tracker audit is a once-or-twice-a-year event for any one repo. It is *breadth*: the org has ~245 non-archived repos, and three of the four procedures have already been executed by hand — the issue triage this plugin ships, the `quantecon-py` technical-debt report, and the zh-cn translation work. Each hand run cost hours of re-derivation because the method lived in a pasted prompt. + +**Structure (proposed 2026-07-26):** four sibling skills, no umbrella — unlike `qe`'s categories, these are distinct procedures over distinct inputs, so the `qe` pattern to reuse is plugin-level sharing, not the umbrella. The method is authored once in `audit/references/` (`doctrine.md`, `quantecon-context.md`, `deliverables.md`) and each `SKILL.md` carries only its own subject matter. Membership test: bulk **and** read-only **and** report-bundle output — which keeps single-item review (`/benchmark:review-acceleration`) outside the family and stops the plugin becoming a general runbook dump. + +**Why read-only is structural, not cautious:** the boundary mirrors the org's own automation split, where the family line *is* the permission line. It makes the family safe to point at any repo and safe to run headlessly, and it keeps a report honest — an audit that half-applied its findings would describe a repo that no longer exists. Acting on a bundle (filing the catalog, posting drafted comments, `qe gh labels sync`) is a separate human-invoked step. + +**Long-run machinery:** every skill runs the same five phases (snapshot → verify → relate → write → self-audit), each checkpointed to disk so a lost session resumes rather than restarts, and all reading one frozen snapshot so the report describes a single point in time. Phase 1 is deterministic (`audit/scripts/fetch_tracker.py`), which also makes the coverage self-audit mechanical rather than narrated. diff --git a/README.md b/README.md index 3ee864a..5153ce4 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,9 @@ Each plugin bundles one area of work — a skill (the instructions Claude follow |---|---|---|---| | `qe` | `/qe:check-style` (+ `check-writing`, `check-math`, `check-code`, `check-figures`, `check-jax`, `check-refs`) | scaffolding | [CATALOG.md](CATALOG.md), work plan in `project-style-guide` | | `benchmark` | `/benchmark:review-acceleration` | under construction | [meta#335](https://github.com/QuantEcon/meta/issues/335) | +| `audit` | `/audit:issues` (+ planned `prs`, `tech-debt`, `translations`) | first runbook landed | [CATALOG.md](CATALOG.md) §3 | -The `qe` plugin is the author-facing surface: one memorable prefix for the skills authors use while editing lectures and preparing PRs. `check-style` is the umbrella (whole lecture, optional category filter, e.g. `/qe:check-style lectures/aiyagari.md figures math`); the per-category sub-skills run the same shared rules individually. `benchmark` is a specialist family for maintainers evaluating accelerated implementations. See [CATALOG.md](CATALOG.md) for the plan and [FUTURE-IDEAS.md](FUTURE-IDEAS.md) for parked candidates. +The `qe` plugin is the author-facing surface: one memorable prefix for the skills authors use while editing lectures and preparing PRs. `check-style` is the umbrella (whole lecture, optional category filter, e.g. `/qe:check-style lectures/aiyagari.md figures math`); the per-category sub-skills run the same shared rules individually. `benchmark` is a specialist family for maintainers evaluating accelerated implementations. `audit` is the maintainer-facing family for bulk, read-only sweeps of a whole repository — every issue, every PR, a whole codebase, a whole translated series — each delivering a report bundle rather than a chat answer. See [CATALOG.md](CATALOG.md) for the plan and [FUTURE-IDEAS.md](FUTURE-IDEAS.md) for parked candidates. ## Installation @@ -33,12 +34,15 @@ Lecture repositories opt in by checking the following into their `.claude/settin } ``` +`audit` is deliberately absent from the lecture-repo block: it is maintainer tooling, and the plugin is the enable unit, so auto-installing it would put org-wide audit skills in every author's command list. Maintainers install it themselves. + ### Manual (any project) ``` /plugin marketplace add QuantEcon/skills /plugin install qe@quantecon /plugin install benchmark@quantecon +/plugin install audit@quantecon ``` ### CI (GitHub Actions) @@ -68,6 +72,11 @@ benchmark/ # specialist plugin .claude-plugin/plugin.json skills/review-acceleration/SKILL.md scripts/ # supporting Python scripts the skill drives +audit/ # maintainer-facing bulk-audit plugin + .claude-plugin/plugin.json + skills/issues/SKILL.md # one skill per audit subject + references/ # shared doctrine, org context, bundle contract + scripts/ # deterministic snapshot + coverage machinery ``` ## Contributing From 0e86b2cca0a78dc3fa872a31d329241daa500b29 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:21:03 +1000 Subject: [PATCH 05/10] Audit: state the naming rule, not a local exception MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Skill names here are objects because the plugin is the verb. That is the general form of the repo's convention — an invocation reads as a command — rather than a deviation from it: where the plugin is a namespace or domain (qe, benchmark) the verb has nowhere to live but the skill; where the plugin is the verb, the skill is the object. Also records why audit beat review. Review is already the per-item word here, so a /review:prs sweeping every open PR would sit beside /review assessing one — erasing the bulk-vs-single-item line the family is built on. The convention amendment itself belongs in docs/developing-skills.md, which arrives with #5; this is the rationale in the meantime. Co-Authored-By: Claude Opus 5 --- audit/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/audit/README.md b/audit/README.md index 1cbd7ec..9a39f5e 100644 --- a/audit/README.md +++ b/audit/README.md @@ -54,4 +54,8 @@ Headless runs work the same way: ## A note on naming -The repo's convention is verb-first skill names, and these are nouns. The verb is the plugin: `/audit:issues` and `/audit:translations` read as commands at the point of use, which is where the convention's purpose lies, and `audit-issues` inside an `audit` plugin would stutter at every invocation. The subject-noun names also keep the four skills parallel, which matters more here than in `qe` — the family is defined by a shared method applied to different subjects. +Skill names here are objects, because the plugin is the verb: `/audit:issues`, `/audit:translations`. The repo's rule is that an invocation reads as a command, and where the plugin name is a namespace or a domain (`qe`, `benchmark`) the verb has nowhere to live but the skill — hence `check-style`, `review-acceleration`. Where the plugin name is itself the verb, the skill is the object; `/audit:audit-issues` would stutter at every invocation. + +Keep the form consistent across the family. The failure mode is not the noun names but mixing them — an `/audit:compare-translations` beside `/audit:issues` breaks the parallelism that makes the four memorable. + +`audit` was chosen over `review` for the same reason the family excludes single-item work: `review` is already the per-item word here (`/benchmark:review-acceleration`, and PR review generally), so a `/review:prs` that sweeps every open PR would sit one keystroke from reviewing one. `audit` also matches QEP-3's `audit-` repo prefix and already connotes observe-and-report, which is the boundary this plugin enforces. From 5fba814e792eee59ee649b13f703460155d5d0d9 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:30:03 +1000 Subject: [PATCH 06/10] Audit: harden the snapshot's provenance and capture claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot review on #11. The `authenticated` field was derived by substring-matching `gh auth status` output, which preflight has already gated on — so it could only ever be true, but would silently record false if gh reworded or localized. A provenance field that can lie undermines the thing the snapshot exists for. preflight now returns True on its only non-exiting path and meta records that. Thread fields are shape-asserted at capture. Everything downstream assumes `comments`/`reviews` hold lists of objects; a gh build returning counts would have crashed later with a bare TypeError, and a differently-shaped payload would have under-reported threads while the report still claimed thread-completeness. Both now fail by name at the point of capture. The missing-gh error pointed at "the unauthenticated route documented in the skill", which read as an offer of a fallback this script does not implement. It now names the file, and states the two constraints that make it a real choice rather than a shrug. Negative-tested: counts and lists-of-non-objects rejected, empty and absent threads pass through. Co-Authored-By: Claude Opus 5 --- audit/scripts/README.md | 2 +- audit/scripts/fetch_tracker.py | 53 +++++++++++++++++++++++++++------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/audit/scripts/README.md b/audit/scripts/README.md index 2937381..0580e95 100644 --- a/audit/scripts/README.md +++ b/audit/scripts/README.md @@ -21,7 +21,7 @@ Three properties the audit depends on: - **Thread-complete on the closed side.** Comment threads come down inside the list call, so reading closed threads — doctrine rule 3, and the gap the first execution of this runbook found — costs nothing extra. - **Frozen in time.** Every later phase reads this snapshot, so the report describes one instant and "events after the snapshot" is a stated property rather than a silent gap. -- **Honest about its own limits.** `unaccounted_numbers` lists gaps in the number sequence for the audit to explain, and `truncation_risk` flags a stream that returned exactly at `--limit`, which is indistinguishable from truncation. Neither is swallowed. +- **Honest about its own limits.** `unaccounted_numbers` lists gaps in the number sequence for the audit to explain, and `truncation_risk` flags a stream that returned exactly at `--limit`, which is indistinguishable from truncation. Neither is swallowed. The thread fields are shape-asserted at capture, so a `gh` build returning comment *counts* rather than objects fails immediately and by name, instead of crashing three phases later or quietly under-reporting the threads the doctrine claims were read. Preflight refuses to start without `gh` and authentication, because the anonymous API is 60 requests/hour per IP and returns nothing at all for the org's private repos. diff --git a/audit/scripts/fetch_tracker.py b/audit/scripts/fetch_tracker.py index 9e1e2fb..c39e10a 100644 --- a/audit/scripts/fetch_tracker.py +++ b/audit/scripts/fetch_tracker.py @@ -64,10 +64,18 @@ def run_gh(args): def preflight(repo): - """Fail in the first minute, not the third hour.""" + """Fail in the first minute, not the third hour. + + Returns True. It only ever returns — every failure path exits — so the + caller can record authentication in the snapshot's provenance without + re-deriving it from `gh`'s human-readable output, which is free to change + wording or be localized. + """ if shutil.which("gh") is None: - die("the GitHub CLI (`gh`) is not on PATH — install it, or fall back to " - "the unauthenticated route documented in the skill") + die("the GitHub CLI (`gh`) is not on PATH. Install it, or take the " + "unauthenticated route in references/quantecon-context.md — which " + "this script does not implement, works only for public repos, and " + "shares a 60 requests/hour per-IP budget.") auth = subprocess.run( ["gh", "auth", "status"], capture_output=True, text=True, ) @@ -79,7 +87,29 @@ def preflight(repo): die(f"expected OWNER/REPO, got {repo!r}") # Confirms the repo exists and is visible to these credentials. run_gh(["repo", "view", repo, "--json", "name"]) - return auth.stdout + auth.stderr + return True + + +def require_thread_objects(items, kind, field, gh_version): + """Assert that `gh` returned full thread objects, not counts. + + Everything downstream — the thread-complete claim the doctrine rests on, + and the comment tallies in coverage.json — assumes these fields hold lists + of objects. A `gh` build that returned a count would crash later with a + bare TypeError, and a differently-shaped payload would silently under-report + threads, leaving the snapshot claiming coverage it does not have. Checking + here turns both into one explicit failure at the point of capture. + """ + for item in items: + value = item.get(field) + if not value: # absent, null, or legitimately empty + continue + if not isinstance(value, list) or not isinstance(value[0], dict): + die(f"{gh_version} returned {kind} `{field}` as " + f"{type(value).__name__}, not a list of objects (first seen on " + f"#{item.get('number')}). This snapshot would not be " + f"thread-complete — check the field against your gh version " + f"before trusting any audit built on it.") def reconcile(issues, prs, limit): @@ -141,15 +171,20 @@ def main(): ) args = parser.parse_args() - auth_note = preflight(args.repo) + authenticated = preflight(args.repo) out = Path(args.out) out.mkdir(parents=True, exist_ok=True) + gh_version = subprocess.run( + ["gh", "--version"], capture_output=True, text=True, + ).stdout.splitlines()[0] + print(f"fetching issues from {args.repo} …") issues = run_gh([ "issue", "list", "-R", args.repo, "--state", "all", "--limit", str(args.limit), "--json", ",".join(ISSUE_FIELDS), ]) + require_thread_objects(issues, "issue", "comments", gh_version) print(f" {len(issues)} issues") print(f"fetching pull requests from {args.repo} …") @@ -157,18 +192,16 @@ def main(): "pr", "list", "-R", args.repo, "--state", "all", "--limit", str(args.limit), "--json", ",".join(PR_FIELDS), ]) + require_thread_objects(prs, "pull request", "comments", gh_version) + require_thread_objects(prs, "pull request", "reviews", gh_version) print(f" {len(prs)} pull requests") - gh_version = subprocess.run( - ["gh", "--version"], capture_output=True, text=True, - ).stdout.splitlines()[0] - coverage = reconcile(issues, prs, args.limit) meta = { "repo": args.repo, "snapshot_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"), "gh_version": gh_version, - "authenticated": "Logged in" in auth_note, + "authenticated": authenticated, "limit": args.limit, "issue_fields": ISSUE_FIELDS, "pr_fields": PR_FIELDS, From 71727850f201eaa370883f0b8192df6245e93756 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Sun, 26 Jul 2026 21:30:03 +1000 Subject: [PATCH 07/10] Audit: re-home the HTML-thread caveat to the doc that owns it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot review on #11. Phase 3 of the issues runbook warned that a thread reconstructed from HTML can start mid-conversation — a caveat inherited from the source runbook's unauthenticated route, which phase 1 never takes now that the snapshot comes from gh. Left there it muddied the trust model: a reader could not tell which capture path the warning applied to. It moves to quantecon-context.md beside the fallbacks it describes, gaining the two things it was missing — that such claims are [stated] at best, and that an audit taking that route must say so in its coverage statement. Phase 3 keeps the range-reference caveat, which is about link parsing and applies to every path. Co-Authored-By: Claude Opus 5 --- audit/references/quantecon-context.md | 2 ++ audit/skills/issues/SKILL.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/audit/references/quantecon-context.md b/audit/references/quantecon-context.md index 1059d1d..34bfebc 100644 --- a/audit/references/quantecon-context.md +++ b/audit/references/quantecon-context.md @@ -51,3 +51,5 @@ If none exists, say so and tier against the repo's milestones instead. Proposing ## Access Authenticated `gh` is assumed. Several QuantEcon repos are private (`project-*`, `style-guide`, `cli`, `jupyteach`, `atlas.quantecon.org`), so the unauthenticated fallbacks that work for public repos — the anonymous REST endpoint, `codeload` tarballs, `raw.githubusercontent.com`, scraping embedded JSON out of issue HTML — return nothing there. The snapshot script fails on missing auth for exactly this reason. On a public repo without `gh`, the fallbacks are viable but share a 60 requests/hour per-IP budget: fetch metadata first, comment threads second, and expect to lose the tail. + +A thread reconstructed that way carries a caveat the `gh` path does not: it can begin mid-conversation, so attribute comments cautiously and treat "the thread says" claims as `[stated]` at best. An audit that took this route must say so in its coverage statement — the snapshot script's guarantees do not apply to it. diff --git a/audit/skills/issues/SKILL.md b/audit/skills/issues/SKILL.md index 2ef3cee..b76ae0d 100644 --- a/audit/skills/issues/SKILL.md +++ b/audit/skills/issues/SKILL.md @@ -60,7 +60,7 @@ Sibling-repo checks belong here too: for QuantEcon, "resolved in a sibling" and ## Phase 3 — relate -Parse every `#N` in all issue **and** PR bodies, both directions. Two parsing caveats: a range reference (`#169–#176`) matches its endpoints only, so check milestone membership before declaring the middle numbers orphaned; and a comment thread reconstructed from HTML can start mid-thread, so attribute cautiously. +Parse every `#N` in all issue **and** PR bodies, both directions. One parsing caveat: a range reference (`#169–#176`) matches its endpoints only, so check milestone membership before declaring the middle numbers orphaned. Produce the cluster map, the table of missing links worth adding (duplicate pairs, origin↔carrier, complementary checks, family orphans), true orphans and over-dense hubs, and the external cross-link registry. From 3194e387a1b557744c53d5a478e9a540c29e9ab4 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 27 Jul 2026 09:06:18 +1000 Subject: [PATCH 08/10] Audit: count PR reviews in coverage, not just comments (review 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The snapshot fetched `reviews` and shape-asserted them, then dropped them from every tally: `thread_stats` counted `comments` alone, so a PR whose whole conversation happened in review bodies reported as unread. On the runbook's own example repo that is not a corner case — closed PRs carry 374 reviews against 28 comments, so phase 5 was measuring 7% of the PR discussion from a number that read like all of it. Counters are now per-field, PRs split open/closed the way issues already were, and the terminal summary prints the PR rows it previously computed and discarded. Paying this before `/audit:prs` is written keeps one definition of captured discussion rather than two. The residue wording tightens with it. Review *bodies* are captured, so the gap left to disclose is inline line comments, which the list call does not return — named in scripts/README.md with the call that fetches them. --- audit/references/doctrine.md | 4 +-- audit/scripts/README.md | 6 ++-- audit/scripts/fetch_tracker.py | 59 +++++++++++++++++++++++----------- audit/skills/issues/SKILL.md | 2 +- 4 files changed, 48 insertions(+), 23 deletions(-) diff --git a/audit/references/doctrine.md b/audit/references/doctrine.md index 8391bb8..ffbc283 100644 --- a/audit/references/doctrine.md +++ b/audit/references/doctrine.md @@ -49,12 +49,12 @@ Phase 1 is deterministic and belongs in [`../scripts/`](../scripts/), not in mod ## 5. Coverage self-audit -Before delivering, reconcile — and **close any gap found rather than merely disclosing it**. `coverage.json` from phase 1 does the mechanical half: items captured against the number sequence `1..max`, threads captured against items with comments, and a truncation flag when a stream returns exactly at the fetch limit. +Before delivering, reconcile — and **close any gap found rather than merely disclosing it**. `coverage.json` from phase 1 does the mechanical half: items captured against the number sequence `1..max`, discussion captured per field it can live in — comments for issues, comments *and* reviews for PRs, split open and closed on both sides — and a truncation flag when a stream returns exactly at the fetch limit. The judgement half is the skill's: - Explain every unaccounted number. Gaps are usually deleted or transferred items, or numbers burned by branches that never opened a PR — but an unexplained gap looks identical to a silent truncation. -- State the residue explicitly: what was *not* inspected. PR review threads? Native sub-issue and Projects membership, both invisible to REST? Anything after the snapshot time? +- State the residue explicitly: what was *not* inspected. Inline review comments, which the snapshot's review bodies do not include? Native sub-issue and Projects membership, both invisible to REST? Anything after the snapshot time? - If the self-audit changes a finding, fold the change back into the documents. Never append a correction section — a report whose conclusions and body disagree is worse than one that is simply late. **No silent caps.** If the audit bounded its own coverage anywhere — sampled, capped at top-N, skipped a class of item — say so in the coverage statement. Silent truncation reads as complete coverage, which is the one failure an auditor cannot recover from. diff --git a/audit/scripts/README.md b/audit/scripts/README.md index 0580e95..b011b80 100644 --- a/audit/scripts/README.md +++ b/audit/scripts/README.md @@ -15,14 +15,16 @@ python ${CLAUDE_PLUGIN_ROOT}/scripts/fetch_tracker.py OWNER/REPO --out /sna | `meta.json` | Repo, snapshot time (UTC), `gh` version, auth state, the fields requested | | `issues.json` | Every issue, any state, each with its comment thread | | `prs.json` | Every PR, with comments, reviews, and `closingIssuesReferences` | -| `coverage.json` | Items against `1..max`, thread counts by open/closed, truncation flag | +| `coverage.json` | Items against `1..max`, discussion counts by open/closed — comments for issues, comments and reviews for PRs — truncation flag | Three properties the audit depends on: - **Thread-complete on the closed side.** Comment threads come down inside the list call, so reading closed threads — doctrine rule 3, and the gap the first execution of this runbook found — costs nothing extra. -- **Frozen in time.** Every later phase reads this snapshot, so the report describes one instant and "events after the snapshot" is a stated property rather than a silent gap. +- **Frozen in time.** Every later phase reads this snapshot, so the report describes one instant and "events after the snapshot" is a stated property rather than a silent gap. Items are written in number order rather than `gh`'s undocumented default sort, so two snapshots of the same repo differ only where the tracker did. - **Honest about its own limits.** `unaccounted_numbers` lists gaps in the number sequence for the audit to explain, and `truncation_risk` flags a stream that returned exactly at `--limit`, which is indistinguishable from truncation. Neither is swallowed. The thread fields are shape-asserted at capture, so a `gh` build returning comment *counts* rather than objects fails immediately and by name, instead of crashing three phases later or quietly under-reporting the threads the doctrine claims were read. Preflight refuses to start without `gh` and authentication, because the anonymous API is 60 requests/hour per IP and returns nothing at all for the org's private repos. Not captured, because REST does not return them: native sub-issue links and Projects membership. Fetch those with `gh api graphql` when a tracker issue depends on them, and disclose them in the coverage statement regardless. + +One more gap worth naming, because it is easy to mistake for coverage: `reviews` holds review bodies and verdicts, not the inline comments left on individual lines. A PR can be counted as reviewed while its line-level discussion goes unread. Fetch those per PR with `gh api repos/OWNER/REPO/pulls/N/comments` where a finding turns on them — and list them as residue when it does not. diff --git a/audit/scripts/fetch_tracker.py b/audit/scripts/fetch_tracker.py index c39e10a..e073eab 100644 --- a/audit/scripts/fetch_tracker.py +++ b/audit/scripts/fetch_tracker.py @@ -94,7 +94,7 @@ def require_thread_objects(items, kind, field, gh_version): """Assert that `gh` returned full thread objects, not counts. Everything downstream — the thread-complete claim the doctrine rests on, - and the comment tallies in coverage.json — assumes these fields hold lists + and the thread tallies in coverage.json — assumes these fields hold lists of objects. A `gh` build that returned a count would crash later with a bare TypeError, and a differently-shaped payload would silently under-report threads, leaving the snapshot claiming coverage it does not have. Checking @@ -128,16 +128,31 @@ def reconcile(issues, prs, limit): highest = max(seen) if seen else 0 unaccounted = [n for n in range(1, highest + 1) if n not in seen] - def thread_stats(items): - with_comments = [i for i in items if i.get("comments")] - return { - "total": len(items), - "with_comments": len(with_comments), - "comments_captured": sum(len(i.get("comments") or []) for i in with_comments), - } - - open_issues = [i for i in issues if i["state"].upper() == "OPEN"] - closed_issues = [i for i in issues if i["state"].upper() != "OPEN"] + def thread_stats(items, fields=("comments",)): + """Count captured discussion per field it can live in. + + An issue's discussion is all in `comments`; a PR's is split across + `comments` and `reviews`, and a PR reviewed but never commented on has + no comments at all. A single tally over `comments` would report such a + PR as unread — so the counters stay per-field, and phase 5 measures the + discussion that was actually captured rather than a subset of it. + """ + stats = {"total": len(items)} + for field in fields: + present = [i for i in items if i.get(field)] + stats[f"items_with_{field}"] = len(present) + stats[f"{field}_captured"] = sum(len(i.get(field) or []) for i in present) + return stats + + def split_by_state(items): + return ( + [i for i in items if i["state"].upper() == "OPEN"], + [i for i in items if i["state"].upper() != "OPEN"], + ) + + open_issues, closed_issues = split_by_state(issues) + open_prs, closed_prs = split_by_state(prs) + pr_fields = ("comments", "reviews") return { "highest_number": highest, @@ -148,7 +163,8 @@ def thread_stats(items): "unaccounted_count": len(unaccounted), "threads_open_issues": thread_stats(open_issues), "threads_closed_issues": thread_stats(closed_issues), - "threads_prs": thread_stats(prs), + "threads_open_prs": thread_stats(open_prs, pr_fields), + "threads_closed_prs": thread_stats(closed_prs, pr_fields), "truncation_risk": { "limit": limit, "issues_at_limit": len(issues) >= limit, @@ -216,12 +232,19 @@ def main(): print(f" numbers 1..{coverage['highest_number']}: " f"{coverage['items_captured']} accounted, " f"{coverage['unaccounted_count']} unaccounted") - oi = coverage["threads_open_issues"] - ci = coverage["threads_closed_issues"] - print(f" open issues {oi['total']:>4} ({oi['with_comments']} with threads, " - f"{oi['comments_captured']} comments)") - print(f" closed issues {ci['total']:>4} ({ci['with_comments']} with threads, " - f"{ci['comments_captured']} comments)") + + def summarise(label, key, fields): + stats = coverage[key] + detail = ", ".join( + f"{stats[f'{f}_captured']} {f} across {stats[f'items_with_{f}']}" + for f in fields + ) + print(f" {label:<14}{stats['total']:>4} {detail}") + + summarise("open issues", "threads_open_issues", ["comments"]) + summarise("closed issues", "threads_closed_issues", ["comments"]) + summarise("open PRs", "threads_open_prs", ["comments", "reviews"]) + summarise("closed PRs", "threads_closed_prs", ["comments", "reviews"]) if coverage["truncation_risk"]["issues_at_limit"] or coverage["truncation_risk"]["prs_at_limit"]: print("\n WARNING: a stream came back exactly at --limit; re-run with a " "higher --limit before trusting the counts.") diff --git a/audit/skills/issues/SKILL.md b/audit/skills/issues/SKILL.md index b76ae0d..e628de3 100644 --- a/audit/skills/issues/SKILL.md +++ b/audit/skills/issues/SKILL.md @@ -80,7 +80,7 @@ Priority labels only for genuine outliers, a handful either way, and only if the ## Phase 5 — self-audit -Run [doctrine §5](../../references/doctrine.md#5-coverage-self-audit) against `coverage.json`: reconcile the counts, explain every unaccounted number, confirm threads were read on both the open and closed sides, and state the residue — PR review threads, GraphQL-only data, anything after the snapshot timestamp. Fold any change back into the documents rather than appending a correction. +Run [doctrine §5](../../references/doctrine.md#5-coverage-self-audit) against `coverage.json`: reconcile the counts, explain every unaccounted number, confirm threads were read on both the open and closed sides, and state the residue — inline review comments, GraphQL-only data, anything after the snapshot timestamp. Fold any change back into the documents rather than appending a correction. ## What this skill does not do From a389bb86da39459d09977073045c1eb767a18370 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 27 Jul 2026 09:06:29 +1000 Subject: [PATCH 09/10] Audit: write the snapshot in number order (review 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Items were written in whatever order `gh` returned them. That order is stable enough in practice, but it is undocumented and not ours — and the snapshot is the evidence every later phase and every cited claim rests on, so its layout should not be a property of the CLI version that produced it. Sorting by number makes two runs over an unchanged tracker byte-identical in issues.json, prs.json, and coverage.json, so a re-fetch diffs down to what actually changed on the tracker. Verified against the runbook's example repo. Not sorting JSON keys: `gh` already emits `--json` fields in a fixed order, so it would add a diff without adding a guarantee. --- audit/scripts/fetch_tracker.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/audit/scripts/fetch_tracker.py b/audit/scripts/fetch_tracker.py index e073eab..1da8a86 100644 --- a/audit/scripts/fetch_tracker.py +++ b/audit/scripts/fetch_tracker.py @@ -201,6 +201,10 @@ def main(): "--limit", str(args.limit), "--json", ",".join(ISSUE_FIELDS), ]) require_thread_objects(issues, "issue", "comments", gh_version) + # Order by number rather than inheriting `gh`'s default sort, which is + # undocumented and free to change. The snapshot is the audit's evidence, so + # two runs over an unchanged tracker should differ only where it did. + issues.sort(key=lambda i: i["number"]) print(f" {len(issues)} issues") print(f"fetching pull requests from {args.repo} …") @@ -210,6 +214,7 @@ def main(): ]) require_thread_objects(prs, "pull request", "comments", gh_version) require_thread_objects(prs, "pull request", "reviews", gh_version) + prs.sort(key=lambda p: p["number"]) print(f" {len(prs)} pull requests") coverage = reconcile(issues, prs, args.limit) From 3656f7253f59a066eb8b7a46bb9edca0d6d3a432 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 27 Jul 2026 10:45:15 +1000 Subject: [PATCH 10/10] Audit: state the method as advice, not as contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pieces of scaffolding were written as requirements after a single execution of a single audit: the four-document bundle, the five phases, and "produces a bundle" as a membership test. One worked example is not enough to generalise from, and each would have forced the next skill to fit a shape derived from issue triage — a debt audit has no link graph to put in `03-links.md`, and a small audit should be free to be one document. So the shape becomes a worked example and the obligations stay. What an audit owes its reader is now its own short section — a coverage statement, an evidence tag per claim, recommendations marked as proposals, drafted comments marked unsent, a date and a named snapshot — and none of it presumes a file count. Doctrine §4 keeps the rule that earns its place (checkpoint each phase before starting the next, because long runs lose sessions) and demotes the five-phase division to what `/audit:issues` happened to need. Membership drops to two tests, bulk and read-only. Read-only is the one doing real work: it is a permission boundary, not a shape preference. The general half of the naming note goes too, now that the repo's rule is deliberately permissive; what stays is the choice specific to this plugin, `audit` over `review`. Candidate skills move from CATALOG.md to #12, and are marked candidate rather than planned — none has the evidence a skill here normally carries before it is written. --- audit/README.md | 25 ++++++++++++------------- audit/references/deliverables.md | 31 ++++++++++++++++++++----------- audit/references/doctrine.md | 12 ++++++++---- audit/scripts/README.md | 2 +- audit/skills/issues/SKILL.md | 4 ++-- 5 files changed, 43 insertions(+), 31 deletions(-) diff --git a/audit/README.md b/audit/README.md index 9a39f5e..71b0012 100644 --- a/audit/README.md +++ b/audit/README.md @@ -7,21 +7,22 @@ Bulk audits of a QuantEcon repository. Where `qe` serves an author working on on | Skill | Audits | Status | |---|---|---| | [`/audit:issues`](skills/issues/SKILL.md) | Every issue, open and closed: status verified against the code, tiered into the repo's plan | runbook landed | -| `/audit:prs` | Every open PR: does it solve a real issue, is it mergeable, what should the review say | planned | -| `/audit:tech-debt` | A codebase's accumulated debt, with a filing-ready issue catalog | planned | -| `/audit:translations` | Parity between a source series and a translation (`lecture-python.myst` ↔ `lecture-python.zh-cn`) | planned | +| `/audit:prs` | Every open PR: does it solve a real issue, is it mergeable, what should the review say | candidate | +| `/audit:tech-debt` | A codebase's accumulated debt, with a filing-ready issue catalog | candidate | +| `/audit:translations` | Parity between a source series and a translation (`lecture-python.myst` ↔ `lecture-python.zh-cn`) | candidate | -Planned skills are tracked in [CATALOG.md](https://github.com/QuantEcon/skills/blob/main/CATALOG.md); this plugin ships with the first one so the shared doctrine is proven against a real procedure before three more are written on top of it. +Only the first is written. The rest are candidates, tracked in [issue #12](https://github.com/QuantEcon/skills/issues/12) — each still needs the evidence a skill here normally carries before anyone writes it. Shipping one first is the point: the shared method gets proven against a real procedure before more are built on top of it. ## What belongs here -Three tests, all of which must pass: +Two tests: 1. **Bulk** — it sweeps a portfolio, not an item. Reviewing one PR's technical quality is not an audit; reviewing all of them is. 2. **Read-only** — it observes and reports. No skill here mutates a tracker, a branch, or a file in the audited repo ([doctrine §3](references/doctrine.md#3-read-only-boundary)). -3. **Report bundle** — the output is the four-document bundle in [deliverables.md](references/deliverables.md), not a chat answer. -The read/write line is deliberate and mirrors the org's own automation split, where the family boundary *is* the permission boundary. Anything that acts on findings — filing the catalog as issues, posting the drafted comments, applying labels — is a separate human-invoked step, which is what makes this family safe to point at any repo and safe to run headlessly. +The read/write line is the one that matters, and it is deliberate: it mirrors the org's own automation split, where the family boundary *is* the permission boundary. Anything that acts on findings — filing the catalog as issues, posting the drafted comments, applying labels — is a separate human-invoked step, which is what makes this family safe to point at any repo and safe to run headlessly. + +An audit also produces a written report rather than a chat answer, since the point is something a reader can check later. That says nothing about how long it is or how many files it takes — see [deliverables.md](references/deliverables.md), which describes what `/audit:issues` produces without requiring the next skill to match it. ## Shared references @@ -29,10 +30,10 @@ Skills are thin; the method lives once at plugin level. | Document | Owns | |---|---| -| [references/doctrine.md](references/doctrine.md) | Trust rules, evidence classes, read-only boundary, phases, coverage self-audit | +| [references/doctrine.md](references/doctrine.md) | Trust rules, evidence classes, read-only boundary, checkpointing, coverage self-audit | | [references/quantecon-context.md](references/quantecon-context.md) | Repo types, label ownership, the cross-repo graph, notes-system discovery, access | -| [references/deliverables.md](references/deliverables.md) | The report bundle contract and where bundles are allowed to land | -| [scripts/](scripts/) | Deterministic phase-1 machinery | +| [references/deliverables.md](references/deliverables.md) | What an audit owes its reader, where reports may land, and the `/audit:issues` bundle as a worked example | +| [scripts/](scripts/) | Deterministic fetch machinery | ## Running one @@ -54,8 +55,6 @@ Headless runs work the same way: ## A note on naming -Skill names here are objects, because the plugin is the verb: `/audit:issues`, `/audit:translations`. The repo's rule is that an invocation reads as a command, and where the plugin name is a namespace or a domain (`qe`, `benchmark`) the verb has nowhere to live but the skill — hence `check-style`, `review-acceleration`. Where the plugin name is itself the verb, the skill is the object; `/audit:audit-issues` would stutter at every invocation. - -Keep the form consistent across the family. The failure mode is not the noun names but mixing them — an `/audit:compare-translations` beside `/audit:issues` breaks the parallelism that makes the four memorable. +Skill names here are objects because the plugin is the verb: `/audit:issues`, `/audit:translations`. Both read as commands, which is the part that matters, and `/audit:audit-issues` would stutter at every invocation. `audit` was chosen over `review` for the same reason the family excludes single-item work: `review` is already the per-item word here (`/benchmark:review-acceleration`, and PR review generally), so a `/review:prs` that sweeps every open PR would sit one keystroke from reviewing one. `audit` also matches QEP-3's `audit-` repo prefix and already connotes observe-and-report, which is the boundary this plugin enforces. diff --git a/audit/references/deliverables.md b/audit/references/deliverables.md index a478ccf..83ec12a 100644 --- a/audit/references/deliverables.md +++ b/audit/references/deliverables.md @@ -1,6 +1,8 @@ -# The report bundle +# Reporting -Every audit in this plugin delivers the same four-document bundle. A shared shape means a reader who has read one audit can read any of them, and a later audit can diff against an earlier one mechanically (doctrine rule 7). +What an audit hands back. The four-document bundle below is what `/audit:issues` produces; it is a worked example, not a contract every skill in this plugin has to satisfy. A skill should produce what its subject actually needs — a debt audit's output is a filing-ready catalog, a parity audit's is a divergence list, and a small audit may reasonably be a single document. Reach for the bundle where it fits, and don't manufacture a section to fill a slot. + +What every audit owes its reader is much smaller, and is in the last section. ## Where it goes @@ -10,9 +12,11 @@ Every audit in this plugin delivers the same four-document bundle. A shared shap 2. The paired `project-*` repo, when the audit serves a program rather than a repo (`project-translation/reports/` for a translation audit). 3. A dedicated `audit-*` repo, per [QEP-3](https://github.com/QuantEcon/qeps/pull/7), once an audit is recurring and its findings are worth publishing. -Confirm the destination before phase 4 and name it in the report. Private inputs stay in private destinations: a `project-*` repo's contents must not be summarised into a public one. +Confirm the destination before writing, and name it in the report. Private inputs stay in private destinations: a `project-*` repo's contents must not be summarised into a public one. + +## The `/audit:issues` bundle -## The four documents +Four documents, which suit a whole-tracker review because it has an argument to make, a long enumeration to carry, and a graph worth drawing separately. Another audit may need two of these, or none. **`01--report.md`** — the argument. Method and evidence base; the snapshot timestamp; portfolio statistics; the findings that change a status, each with its evidence tag; the tiering, tied to the repo's existing plan; policy alignment; and a time-boxed execution order for whoever acts on it. @@ -25,13 +29,18 @@ Confirm the destination before phase 4 and name it in the report. Private inputs Close with the verification of the closed set (explicitly thread-complete), and write up anything agreed-but-never-filed as a proposed new item, in a form that could be filed as-is. -**`03--links.md`** — the graph. Clusters and their anchor items; a concrete table of missing links worth adding; true orphans and over-dense hubs; and the external cross-link registry (sibling repos, program and meta issues, evidence PRs). +**`03--links.md`** — the graph. Clusters and their anchor items; a concrete table of missing links worth adding; true orphans and over-dense hubs; and the external cross-link registry (sibling repos, program and meta issues, evidence PRs). This one is the most tracker-specific of the four — an audit whose subject has no interesting link structure should simply not produce it. + +**`README.md`** — the index. Headline numbers, what each document is for, and the coverage statement. + +Where a later audit does reuse this shape, keeping the names and the entry format identical is worth something: a reader who has read one bundle can read the next, and two audits of the same subject diff mechanically ([doctrine rule 7](doctrine.md#1-what-makes-a-bulk-audit-trustworthy)). That is a reason to converge where the shape fits, not a reason to force it. -**`README.md`** — the index. Headline numbers, what each document is for, and **the coverage statement**: exactly what was and was not inspected, per [doctrine §5](doctrine.md#5-coverage-self-audit). This is the first thing a reader should be able to check and the last thing the audit should write. +## What any audit owes its reader -## Rules that apply to all four +Short list, and this part is not advisory — each item is what keeps a long report checkable by someone who will not re-run it. -- **Tag every claim** with its evidence class ([doctrine §2](doctrine.md#2-evidence-classes)). An untagged status claim is a defect. -- **Recommendations are proposals.** Write them in the imperative for whoever executes ("close, citing PR #204"), but the audit does not execute them. -- **Suggested comment text is a draft, in a fenced block, clearly marked as unsent** — and subject to the closing-keyword hazard in [quantecon-context.md](quantecon-context.md#the-cross-repo-graph-is-the-point). -- **Date the bundle and name the snapshot.** Two audits of the same repo must be distinguishable at a glance, and every number in a bundle refers to one point in time. +- **A coverage statement.** Exactly what was and was not inspected, per [doctrine §5](doctrine.md#5-coverage-self-audit). In a bundle it belongs in the index; in a single document it is a section. It is the first thing a reader should be able to check and the last thing the audit should write. +- **An evidence tag on every claim** ([doctrine §2](doctrine.md#2-evidence-classes)). An untagged status claim is a defect. +- **Recommendations marked as proposals.** Write them in the imperative for whoever executes ("close, citing PR #204"), but the audit does not execute them. +- **Drafted comment text marked as unsent**, in a fenced block — and subject to the closing-keyword hazard in [quantecon-context.md](quantecon-context.md#the-cross-repo-graph-is-the-point). +- **A date and a named snapshot.** Two audits of the same subject must be distinguishable at a glance, and every number in a report refers to one point in time. diff --git a/audit/references/doctrine.md b/audit/references/doctrine.md index ffbc283..85cdca3 100644 --- a/audit/references/doctrine.md +++ b/audit/references/doctrine.md @@ -33,19 +33,23 @@ That boundary is what makes the family safe to run headlessly and safe to point Corollary: an audit never writes into the plugin directory, and its outputs never live in `QuantEcon/skills`. See [deliverables.md](deliverables.md) for where they go. -## 4. Phases, and surviving a long run +## 4. Surviving a long run -Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep. Each skill runs the same five phases, and **each phase writes its output to the working directory before the next begins**, so a lost session resumes at the last completed phase instead of restarting. +Bulk audits outlive sessions. Context runs out, rate limits bite, machines sleep. One rule follows from that, and it is about checkpointing rather than about structure: **write each phase's output to the working directory before starting the next**, so a lost session resumes where it stopped instead of restarting. How a skill divides itself into phases is its own business — the division below is one that worked, not a template to fill. + +Two things are worth doing whatever the division. **Fetch once, and fetch first**, deterministically, in [`../scripts/`](../scripts/) rather than in model judgement. That also **freezes the audit's point in time**: every later claim refers to the snapshot, so "events after the snapshot" becomes a stated property of the report rather than an unnoticed gap. Record the snapshot timestamp; never silently mix fresh API reads into a later phase. + +`/audit:issues` uses five phases, which suit an audit that must capture a whole tracker, check it item by item, and then write at length: | Phase | Produces | Resumable from | |---|---|---| | 1. Snapshot | `meta.json`, `issues.json`, `prs.json`, `coverage.json` | — | | 2. Verify | per-item findings with evidence tags | the snapshot | | 3. Relate | the cross-link / parity graph | phase 2 | -| 4. Write | the report bundle | phases 2–3 | +| 4. Write | the report | phases 2–3 | | 5. Self-audit | the coverage statement, folded back in | all of the above | -Phase 1 is deterministic and belongs in [`../scripts/`](../scripts/), not in model judgement. Fetching once also **freezes the audit's point in time**: every later claim refers to the snapshot, and "events after the snapshot" becomes a stated property of the report rather than an unnoticed gap. Record the snapshot timestamp in the report; never silently mix fresh API reads into a later phase. +A shorter audit may collapse verify and write, and one whose subject has no interesting link structure has no phase 3 at all. What it cannot skip is the last one, for the reason in §5. ## 5. Coverage self-audit diff --git a/audit/scripts/README.md b/audit/scripts/README.md index b011b80..1c33a3f 100644 --- a/audit/scripts/README.md +++ b/audit/scripts/README.md @@ -1,6 +1,6 @@ # Deterministic audit machinery -Phase 1 of every audit is mechanical, so it belongs here rather than in model judgement ([doctrine §4](../references/doctrine.md#4-phases-and-surviving-a-long-run)). Stdlib only, driving `gh`; no install step. +Fetching is mechanical, so it belongs here rather than in model judgement ([doctrine §4](../references/doctrine.md#4-surviving-a-long-run)). Stdlib only, driving `gh`; no install step. ## `fetch_tracker.py` diff --git a/audit/skills/issues/SKILL.md b/audit/skills/issues/SKILL.md index e628de3..a3b0b21 100644 --- a/audit/skills/issues/SKILL.md +++ b/audit/skills/issues/SKILL.md @@ -7,7 +7,7 @@ description: Audit every issue in a GitHub repository, open and closed — verif Whole-tracker review of one repository. Long-running by design — a hundred-issue repo is a multi-hour run — so it works from a frozen snapshot and checkpoints every phase to disk. -> **Status: first runbook of the `audit` family.** The procedure below is the QuantEcon-adapted form of a runbook that has been executed once end to end; the gap that execution found (closed threads unread) is now doctrine rule 3 and is closed by the snapshot. Plan and family design: [CATALOG.md §3](https://github.com/QuantEcon/skills/blob/main/CATALOG.md). +> **Status: first runbook of the `audit` family.** The procedure below is the QuantEcon-adapted form of a runbook that has been executed once end to end; the gap that execution found (closed threads unread) is now doctrine rule 3 and is closed by the snapshot. Plan and open questions: [issue #12](https://github.com/QuantEcon/skills/issues/12). ## Invocation @@ -30,7 +30,7 @@ Ask only when discovery is ambiguous — two plausible plan anchors, say — not - [doctrine.md](../../references/doctrine.md) — trust rules, evidence classes, the read-only boundary, phases, coverage self-audit - [quantecon-context.md](../../references/quantecon-context.md) — repo types, label ownership, the cross-repo graph, the closing-keyword hazard -- [deliverables.md](../../references/deliverables.md) — the bundle contract +- [deliverables.md](../../references/deliverables.md) — what a report owes its reader, where it may land, and this skill's four-document shape Then, in the audited repo: the notes system, `CHANGELOG.md`, the latest release notes, and `AGENTS.md`/`CLAUDE.md`.