Propose the audit plugin: bulk repository audits, starting with /audit:issues - #11
Merged
Conversation
A third plugin for maintainer-facing work: portfolio-wide, read-only sweeps of one repository that deliver a report bundle. Membership is gated on three tests together — bulk, read-only, bundle output — which keeps single-item review outside the family and stops the plugin becoming a general runbook dump. The method is authored once at plugin level so the four planned skills share it rather than restating it: doctrine.md (trust rules, evidence classes, the read-only boundary, phased checkpointing, the coverage self-audit), quantecon-context.md (repo types, label ownership, the cross-repo graph, notes-system discovery, access), and deliverables.md (the bundle contract and where bundles may land). Read-only is structural rather than cautious. It is what makes the family safe to point at any repo and safe to run headlessly, and it keeps a report honest: an audit that half-applied its own findings would describe a repo that no longer exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 of every audit is mechanical, so it runs as a script rather than as model judgement. fetch_tracker.py captures issues and PRs in any state with full comment threads in two round trips, which is what makes reading closed threads free — the one gap the first execution of the issue runbook found. The snapshot also freezes the audit's point in time, so every later phase reads one instant and "events after the snapshot" becomes a stated property of the report rather than an unnoticed gap. coverage.json does the mechanical half of the self-audit: items against the number sequence 1..max, thread counts split open/closed, and a truncation flag for a stream that returns exactly at the fetch limit — indistinguishable from truncation, so it is surfaced, not swallowed. Preflight refuses to start without gh and auth, because the anonymous API is 60 req/h per IP and returns nothing for the org's private repos. Validated against QuantEcon/action-translation: 111 issues + 110 PRs, numbers 1..221 fully accounted, 123 comments across 60 threads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runbook's four paste-in fields become optional arguments with documented discovery and stated fallbacks, so the common invocation is just the repo. Its numbered sections split along the doctrine boundary: the trust rules and coverage self-audit move to plugin level, and the skill keeps what is specific to issues — per-item verification, the finding categories to hunt, GC/T0-T3 tiering, and the link graph. QuantEcon adaptations over the generic runbook: tier by repo type, since a build break in a lecture repo and a consumer-visible change in an action repo outrank their thread activity; check siblings before concluding, because "resolved in a sibling" and "one step of a rollout" are the two wrong conclusions a single-repo audit reaches here; leave label application to qe gh labels; and keep GitHub closing keywords out of drafted cross-repo references, which would otherwise close the upstream item when the text lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Marketplace entry at 0.1.0, catalogue minor-bumped for the new plugin. CATALOG.md §3 records the family design and the evidence behind it, which is breadth rather than per-repo frequency: a tracker audit is a once-a-year event for one repo, but the org has ~245 of them and three of the four procedures have already been run by hand. audit is deliberately left out of the lecture-repo auto-install block. The plugin is the enable unit, so including it would put maintainer tooling in every author's command list — the same audience separation that keeps benchmark out of the qe prefix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Skill names here are objects because the plugin is the verb. That is the general form of the repo's convention — an invocation reads as a command — rather than a deviation from it: where the plugin is a namespace or domain (qe, benchmark) the verb has nowhere to live but the skill; where the plugin is the verb, the skill is the object. Also records why audit beat review. Review is already the per-item word here, so a /review:prs sweeping every open PR would sit beside /review assessing one — erasing the bulk-vs-single-item line the family is built on. The convention amendment itself belongs in docs/developing-skills.md, which arrives with #5; this is the rationale in the meantime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new maintainer-facing audit plugin to the QuantEcon skills marketplace, introducing a structured, read-only “bulk audit” family and landing the first runbook (/audit:issues) plus deterministic snapshot machinery to support long-running, resumable audits.
Changes:
- Registers a new
auditplugin in the marketplace and documents installation/positioning alongsideqeandbenchmark. - Adds shared audit doctrine + QuantEcon-specific context + deliverables contract as plugin-level references.
- Introduces phase-1 deterministic tooling (
audit/scripts/fetch_tracker.py) and the first skill runbook (audit/skills/issues/SKILL.md).
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents the new audit plugin, its audience boundary, and install instructions. |
| CATALOG.md | Adds the audit family plan and rationale as catalog entry §3. |
| .claude-plugin/marketplace.json | Registers the audit plugin and bumps marketplace version to 0.2.0. |
| audit/.claude-plugin/plugin.json | Defines the audit plugin manifest (name/description/version). |
| audit/README.md | Plugin-level overview, scope boundary, and naming rationale. |
| audit/skills/issues/SKILL.md | Lands the /audit:issues runbook (phased, snapshot-driven, bundle output). |
| audit/scripts/README.md | Documents deterministic phase-1 snapshot tooling and outputs. |
| audit/scripts/fetch_tracker.py | Implements tracker snapshot + coverage reconciliation for issues/PRs. |
| audit/references/doctrine.md | Codifies audit-wide trust rules, evidence tags, read-only boundary, phases, self-audit. |
| audit/references/quantecon-context.md | QuantEcon-specific repo-type expectations, label policy, cross-repo hazards, discovery order. |
| audit/references/deliverables.md | Defines the 4-document report bundle contract and allowed destinations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot review on #11. The `authenticated` field was derived by substring-matching `gh auth status` output, which preflight has already gated on — so it could only ever be true, but would silently record false if gh reworded or localized. A provenance field that can lie undermines the thing the snapshot exists for. preflight now returns True on its only non-exiting path and meta records that. Thread fields are shape-asserted at capture. Everything downstream assumes `comments`/`reviews` hold lists of objects; a gh build returning counts would have crashed later with a bare TypeError, and a differently-shaped payload would have under-reported threads while the report still claimed thread-completeness. Both now fail by name at the point of capture. The missing-gh error pointed at "the unauthenticated route documented in the skill", which read as an offer of a fallback this script does not implement. It now names the file, and states the two constraints that make it a real choice rather than a shrug. Negative-tested: counts and lists-of-non-objects rejected, empty and absent threads pass through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot review on #11. Phase 3 of the issues runbook warned that a thread reconstructed from HTML can start mid-conversation — a caveat inherited from the source runbook's unauthenticated route, which phase 1 never takes now that the snapshot comes from gh. Left there it muddied the trust model: a reader could not tell which capture path the warning applied to. It moves to quantecon-context.md beside the fallbacks it describes, gaining the two things it was missing — that such claims are [stated] at best, and that an audit taking that route must say so in its coverage statement. Phase 3 keeps the range-reference caveat, which is about link parsing and applies to every path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mmcky
marked this pull request as ready for review
July 26, 2026 11:44
Contributor
Author
|
Full audit review summary for this PR:\n\nFindings (ordered by severity):\n\n1. Medium: PR thread coverage metrics undercount discussion volume in coverage.json.\nEvidence: audit/scripts/fetch_tracker.py lines 131-137 and 149-151; audit/scripts/fetch_tracker.py line 196 validates reviews are present, but coverage counters only include comments.\nRisk: Phase-5 self-audit confidence can be overstated for PR discussion completeness.\nSuggestion: Add separate review counters for PRs (for example reviews_with_content and reviews_captured) in coverage.json and terminal summary output.\n\n2. Low: Snapshot output is not normalized for stable mechanical diffs across runs.\nEvidence: deliverables expect mechanical diffs, but fetch output ordering can drift run-to-run.\nSuggestion: Sort issues and PRs by number before writing JSON artifacts (and optionally sort JSON keys).\n\n3. Low: No automated tests for core reconciliation and schema-guard logic.\nEvidence: require_thread_objects and reconcile are central safeguards and currently untested.\nSuggestion: Add focused unit tests with small synthetic fixtures for shape checks, unaccounted number reconciliation, and truncation flags.\n\nValidation performed:\n- CI status on this PR is passing (validate job).\n- scripts/validate.py passes on this branch (all 3 plugins valid).\n- audit/scripts/fetch_tracker.py compiles and runs in smoke testing, producing meta.json, issues.json, prs.json, and coverage.json as documented.\n\nBottom line: no merge-blocking issues found; strongest follow-up is improving PR review-thread coverage accounting so metrics match coverage claims exactly. |
The snapshot fetched `reviews` and shape-asserted them, then dropped them from every tally: `thread_stats` counted `comments` alone, so a PR whose whole conversation happened in review bodies reported as unread. On the runbook's own example repo that is not a corner case — closed PRs carry 374 reviews against 28 comments, so phase 5 was measuring 7% of the PR discussion from a number that read like all of it. Counters are now per-field, PRs split open/closed the way issues already were, and the terminal summary prints the PR rows it previously computed and discarded. Paying this before `/audit:prs` is written keeps one definition of captured discussion rather than two. The residue wording tightens with it. Review *bodies* are captured, so the gap left to disclose is inline line comments, which the list call does not return — named in scripts/README.md with the call that fetches them.
Items were written in whatever order `gh` returned them. That order is stable enough in practice, but it is undocumented and not ours — and the snapshot is the evidence every later phase and every cited claim rests on, so its layout should not be a property of the CLI version that produced it. Sorting by number makes two runs over an unchanged tracker byte-identical in issues.json, prs.json, and coverage.json, so a re-fetch diffs down to what actually changed on the tracker. Verified against the runbook's example repo. Not sorting JSON keys: `gh` already emits `--json` fields in a fixed order, so it would add a diff without adding a guarantee.
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
"Verb-first skill names" was a proxy for the rule that actually matters — the invocation reads as a command — and it only held because both live plugins name a domain, leaving the skill as the only place a verb could go. A plugin whose name *is* the verb inverts that: `/audit:issues` is imperative already, and `/audit:audit-issues` would stutter to satisfy the letter of a rule while breaking its intent. So the convention now names both forms rather than treating one as the default and the other as an exception. Namespace plugins (`qe`, `benchmark`) carry the verb in the skill; verb plugins (`audit`) carry it in the plugin and leave the skill as the object. A family of sibling procedures over different inputs is the case the second form exists for — it states the shared half of the name once. The constraint that replaces verb-first is consistency within a plugin. Mixing the forms is what actually costs something: one `/audit:compare-translations` beside `/audit:issues` breaks the parallelism that makes a family predictable, which is worth more than any single sharper verb. Depends on #11 for its examples: `audit` must land before this does.
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
"Verb-first skill names" reads as a rule about skills, but the thing it was protecting is that the invocation scans as an imperative — and the whole `/plugin:skill` string is what a user types. `/audit:issues` satisfies that with the verb in the plugin, yet fails the rule as written, so the rule had to move. It moves by getting weaker, not by growing a taxonomy. Both shapes are shown, neither is preferred, and the absence of a preference is stated so a later contributor reads it as undecided rather than unspecified. Ranking them now would mean generalising from three plugins and no usage; the question is parked in FUTURE-IDEAS.md with the specific thing worth watching — whether mixing shapes inside one plugin actually causes trouble or merely looks untidy. Depends on #11 for its example: `audit` must land before this does.
4 tasks
Three pieces of scaffolding were written as requirements after a single execution of a single audit: the four-document bundle, the five phases, and "produces a bundle" as a membership test. One worked example is not enough to generalise from, and each would have forced the next skill to fit a shape derived from issue triage — a debt audit has no link graph to put in `03-links.md`, and a small audit should be free to be one document. So the shape becomes a worked example and the obligations stay. What an audit owes its reader is now its own short section — a coverage statement, an evidence tag per claim, recommendations marked as proposals, drafted comments marked unsent, a date and a named snapshot — and none of it presumes a file count. Doctrine §4 keeps the rule that earns its place (checkpoint each phase before starting the next, because long runs lose sessions) and demotes the five-phase division to what `/audit:issues` happened to need. Membership drops to two tests, bulk and read-only. Read-only is the one doing real work: it is a permission boundary, not a shape preference. The general half of the naming note goes too, now that the repo's rule is deliberately permissive; what stays is the choice specific to this plugin, `audit` over `review`. Candidate skills move from CATALOG.md to #12, and are marked candidate rather than planned — none has the evidence a skill here normally carries before it is written.
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
"Verb-first skill names" reads as a rule about skills, but the thing it was protecting is that the invocation scans as an imperative — and the whole `/plugin:skill` string is what a user types. `/audit:issues` satisfies that with the verb in the plugin, yet fails the rule as written, so the rule had to move. It moves by getting weaker, not by growing a taxonomy. Both shapes are shown, neither is preferred, and the absence of a preference is stated so a later contributor reads it as undecided rather than unspecified. Ranking them now would mean generalising from three plugins and no usage; the question is parked in FUTURE-IDEAS.md with the specific thing worth watching — whether mixing shapes inside one plugin actually causes trouble or merely looks untidy. Depends on #11 for its example: `audit` must land before this does.
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
Post-#11 reconciliation. README gains audit in the plugin table and the documentation index, and its layout note drops "bundle contract", which the plugin no longer has. The marketplace entry's move to the co-located `./audit` form is not here — it belongs in "Install fix: co-located plugin sources", which is the commit that established that form, and the rebase carried it there.
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
…, skill wired) (#5) * Add the lecture evaluation system: rubric engine, calibration, worked cases The quantitative evaluation system for lecture code rewrites developed and validated on QuantEcon/lecture-python.myst#717 and #654: - references/EVALUATION_FRAMEWORK.md — the standard in prose: 7 weighted dimensions, numeric scoring anchors, structural checklists, verdict bands, worked HIGH/LOW examples - scripts/scoring/ — the standard as code: rubric.py (deterministic evidence -> score), score.py (engine/CLI), EVIDENCE_TEMPLATE.json (the judgement contract: measured numbers + cited yes/no answers) - scripts/calibration/ — the shared aiyagari Bellman benchmark pinning the "25x as-used = score 5" efficiency anchor - references/examples/{ge_arrow,markov_asset}/ — two complete worked evaluations (measurement scripts, results, evidence, reports): ge_arrow 2.85/5 mixed/wash; markov_asset 2.25/5 net regression (build-breaking bug) Content as delivered 2026-07-21; placed at plugin-convention paths. Path/link integration follows in a separate commit. * Integrate the evaluation system into the plugin layout Integration on top of the landed package (content authored by @xuanguang-li; this commit is path/plumbing only plus docs): - score.py takes a lecture directory path (works from any cwd) instead of a name resolved against the old package root - ge_arrow scripts use local imports (import model_old), matching the markov_asset idiom, so every script runs directly from its directory - run_all.py (both examples): scoring call updated to the new layout, lecture dir derived not hardcoded, and a provenance stamp written to results/env.json (python/platform/numpy/jax/quantecon versions) -- the seed of the QuantEcon/meta#335 shared result schema - All relative links in EVALUATION_FRAMEWORK.md and the two reports rewritten for the new layout (verified: no dangling references) - scripts/README.md rewritten: engine layout, the three-step scoring contract, the evaluate-a-new-lecture recipe - SKILL.md updated: system landed, operational procedure now points at the real engine/templates, worked cases become regression anchors - benchmark plugin 0.1.0 -> 0.2.0 (marketplace kept in sync) Verified: both example scorecards regenerate byte-identically from the new layout; scripts/validate.py green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * scoring: verdict from the rounded total; fix stale paths in docstring/template Addresses Copilot review on #5: - rubric.py computes the verdict band from the rounded total, so the band always agrees with the number displayed (raw FP sums can land at 2.4999999999999996 for combinations that are exactly 2.50 in exact arithmetic; 797/78125 score combinations were affected) - rubric.py docstring points at ../../references/EVALUATION_FRAMEWORK.md - EVIDENCE_TEMPLATE.json _how cites the actual CLI form (scripts/scoring/score.py <lecture-dir> from the plugin root) Both committed scorecards regenerate unchanged (neither sits at a band edge). The x64-divergence guard in score_correctness is deliberately left as authored — rubric semantics stay with the standard's author. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Persist the headline metrics; fix two static-metrics inconsistencies From the detailed logic review of the evaluation scripts: - run_all.py (both examples) now captures the JSON lines printed by the fresh-process scripts (as_used_total, cold_start) into results/as_used.json / results/cold_start.json, with the derived as_used_speedup - the headline metric previously lived only on the console, though the docstrings already claimed aggregation - ge_arrow static_metrics: remove the duplicated "@" pattern that double-counted concept token hits (informational metric only; regenerated results: old.concept_token_hits 110 -> 105) - markov_asset static_metrics: rename statements_for_one_asset -> statements_for_one_result, matching EVIDENCE_TEMPLATE.json and the ge_arrow template vocabulary (values unchanged; results regenerated) Both scorecards regenerate byte-identically - no scored value changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Document the reference examples: logic check and provenance benchmark/references/examples/README.md explains both canonical cases in detail - the economics, the two implementations, where every evidence number comes from, and why each verdict is what it is - and records the 2026-07-21 line-by-line verification (scorecard byte reproduction, evidence-results cross-checks, rubric edge audit, fairness audit) plus the known caveats (M1 hand-curated readability inputs, m3 x64 stamping, n6 sweep asymmetry). These examples are the regression baseline for the skill; their accuracy is now auditable rather than asserted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Final-review fixes: harden run_all, share the env stamp, doc consistency From the adversarially-verified final review (31-agent pass over the full PR): Robustness (both run_all.py files — all four confirmed by execution): - guard against JSON-scalar stdout lines (json.loads succeeds on `42`/ `null`; .get then raised AttributeError and aborted the pipeline) - track per-step returncodes; failed step titles now go into the provenance stamp so a partial run cannot claim full provenance for stale results - symmetric total_s guard in the as_used_speedup derivation (bare numpy-side index could KeyError) - warn on duplicate mode keys instead of silently overwriting Simplification (integrator-authored code only): - the byte-identical 18-line write_env block duplicated in both run_all.py files becomes shared scripts/scoring/env_stamp.py, invoked like score.py (-26 LOC net; the shared meta#335 schema now has one definition) Consistency: - gitignore the per-run generated results (as_used.json, cold_start.json, env.json) and annotate their doc citations as generated-not-committed (committing them faithfully is impossible here: the local env is jax 0.10.1 vs the reports' 0.4.35) - examples/README: fix 'logic 5' -> 4 (scorecard and its own arithmetic say 4; with 5 the listed scores sum to 3.00, not 2.85) - REPORT link labels updated to match their (already-correct) targets; markov REPORT's stale statements_for_one_asset key renamed - evidence.json _how strings and score.py's scorecard _note now cite scripts/scoring/... (scorecards regenerated; only the _note changed) - SKILL.md no longer restates the weight vector and verdict bands -- it points at EVALUATION_FRAMEWORK.md sections 1-2 and rubric.py, so recalibration cannot drift the copies - scripts/README: commands documented as running from the plugin root Both scorecards regenerate byte-identically under the updated engine; validate.py green; env_stamp smoke-tested including steps_failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: skill usage, repo setup, and validated triage mode (#6) * Docs: plugin guide, skill usage, repo setup — with triage mode validated - benchmark/README.md: the plugin's user guide — review mode (session walkthrough, report format), triage mode ("should this lecture be converted?"), manual pipeline quickstart, plugin map - SKILL.md: triage-mode section (the prospective subset: baseline as-used total, pattern match against the calibrated poles, crossover check, readability-cost forecast, and the weight-algebra decision rule) - docs/using-skills.md: consumer guide — setup paths, invocation forms, report-first expectations, troubleshooting - docs/developing-skills.md: contributor guide — layout, conventions, dev loop, versioning, squash-merge/stacking and external-author attribution patterns - README.md: documentation index Triage mode is empirically validated before being documented: blind triage using only baseline-side data (fresh runs: ge_arrow 0.028s, markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s) reproduces all three known verdicts (don't convert / don't convert / convert), and correctly cannot predict conversion-quality defects (markov_asset's build bug) — that scope limit is documented with it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: fix command count and map-table link text Addresses Copilot review on #6: the manual-install snippet is three commands, not two; the benchmark map row's link text now matches its target (scripts/README.md) with the engine path named in the description instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: scale the evidence discipline by output tier, not blanket The evidence-file + engine pattern applies to skills that aggregate judgements into scored verdicts; findings-list skills need only cited claims. developing-skills.md gets the three-tier discipline; CATALOG gains the principle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Design review: corrections of record + merged three-way synthesis Two independent design critiques of the evaluation system (a fresh unframed session; a 36-agent adversarial workflow with steelman defense) were merged in reviews/. Three of our own claims were falsified by execution and are corrected here: - markov_asset's lecture DOES build in notebook order: a stale global err masks the stray err.throw(), silently disabling the checkify stability validation (worse than a crash, but not a build failure). Erratum prepended to the REPORT; wording corrected in examples README, SKILL.md, plugin README; correction posted on lecture-python.myst#654 - "mirrors the lecture exactly": both reference replays deviate from the lectures' construction patterns; certification corrected - "medians over repeats": false for the as-used totals (single pass per side); fairness-audit wording corrected; triage validation noted as in-sample reviews/ holds the independent report and the merged synthesis: which critiques survived the steelman defense (convention-only safety couplings; readability instrument inverting its own exemplars; the review/triage mode contradiction; band-label semantics; noise vs resolution) and which were defended (ratio form, weighted total, min() shape, per-consequence billing), plus the ranked v2 plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update references after withdrawal of the #654 comments The evaluation findings briefly posted to lecture-python.myst#654 were withdrawn; the PR will receive one authoritative evaluation after the rubric-v2 revision and a full skill run, rather than a comment-and-correction trail. Erratum and merged review updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rubric v2: enforced couplings, no-conversion verdict, sensitivity stamp, K-repeat Implements items 1-4 of #7 (the changes that survived the three-way design review), validated against both committed evidence files: - score_all derives the logic&design bug-cap from the correctness evidence (builds / x64-divergence) instead of trusting a hand-set boolean, and gates the verdict: correctness 1 caps at net regression, correctness 2 at mixed/wash. The review's honest-evidence 4.2 hole (float32 catastrophe, no logic bug -> "merge") now gates to net regression. - no-conversion verdict: baseline as-used under the 1 s materiality floor (a labeled policy choice) + slower as-used candidate -> the scorecard says don't convert instead of scoring the polish. Reconciles review with triage. - sensitivity stamp in score.py: every scored input perturbed one at a time (bools flipped, counts +/-1, floats +/-10%); scorecard stamped robust/fragile with deciding flips listed. - K-repeat as-used: run_all.py repeats each as-used side 3x in fresh processes; the headline speedup is a median, per-run speedups feed a contested-band annotation in the engine. Re-validation: ge_arrow re-scores 2.85 (unchanged), verdict now no-conversion (candidate band mixed/wash), stamped fragile with exactly the review's demonstrated flips. markov_asset re-scores 2.25 (unchanged), no-conversion + gated net regression, stamped robust across all 29 perturbations - the gate absorbs the one-concept band flip the review demonstrated. Both band movements are deliberate v2 changes, noted in the reports. Benchmark plugin bumped to 0.3.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Skill wiring: plugin-root anchoring and workspace evaluation directory Operationalizes /benchmark:review-acceleration for installed-plugin runs (#4 item 3): evaluations are built under <workspace>/benchmark-eval/<lecture>/ with the plugin read-only at CLAUDE_PLUGIN_ROOT (run_all.py already resolves the shared engine from that env var); preconditions stated up front; extraction/replay diff check added to scaffold; the v2 verdict outputs (gates, no-conversion, sensitivity stamp) carried through the procedure, triage decision rule, and calibration anchors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: local testing tiers and the switch back to the production marketplace Adds a Testing locally section to the contributor guide: --plugin-dir for skill iteration, a local-path marketplace for full install simulation before merging (test from a consuming project; the checkout's branch is what gets served; the marketplace name collides with production), and the remove/add/install sequence to return to the GitHub source afterwards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validation run: ge_arrow re-evaluated from a fresh checkout, verdict reproduced The skills#8 dry run, targeting the motivating PR the system was first developed on (lecture-python.myst#717) rather than the reserved #654 acceptance case. Fresh partial clone at base 8cfba4c / head 8c2d0d7, wired workspace procedure (benchmark-eval/<lecture>/ with CLAUDE_PLUGIN_ROOT), jax 0.10.1 vs the reference 0.4.35: reproduces 2.85 / no-conversion / fragile with the same three deciding flips; every measured quantity moved only within its band. Full cross-comparison in reviews/validation-run-ge_arrow-2026-07-22.md. Fixes surfaced by the run: - ge_arrow check_equivalence.py now writes equivalence_x64.json under JAX_ENABLE_X64 instead of clobbering the as-shipped results (the markov_asset template already did this per-regime) - model_old.py fidelity note discloses the cosmetic whitespace normalisation found by diffing against a fresh extraction - both evidence files now record source_pr + base/head SHAs (provenance was previously PR-number-and-branch only) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: hands-on evaluation tutorial built on the ge_arrow validation run Walks the review-acceleration procedure end-to-end by hand - checkout at the recorded SHAs, workspace scaffold, K-repeat measurement, the two precision regimes and why each exists, evidence, scoring, band-based cross-comparison - with every command and number taken from the recorded validation run so readers can check their results against a committed reference. Linked from the README docs table and the benchmark guide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: AGENTS.md as the canonical agent guide, led by single source of truth Adds the repo-level instructions file for AI agents and contributors, following the QuantEcon.manual convention: AGENTS.md is canonical and CLAUDE.md imports it. Its governing principle is @jstac's — skills point to existing documentation in the manual wherever possible instead of repeating what the manual says — worked out concretely for this repo: rule text stays upstream in style-guide, numbers live once, every topic has an owning doc, and cross-boundary references are links rather than copies. The rest of the file is a doc map plus the conventions that aren't written down anywhere else (commit subjects, scratch notes, writing for the GitHub renderer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Install fix: co-located plugin sources, required owner, validator guards Three install-surface bugs surfaced by @xuanguang-li while testing the benchmark plugin (#10): - marketplace.json omitted the required top-level `owner` object, so `/plugin marketplace add` failed schema validation for every user. - Both plugins declared a remote github source pointing back at this repo, forcing an install-time re-clone over SSH (ED25519 failure) to reach a subdirectory already present in the added marketplace copy. Switched to the documented co-located pattern — relative-path sources (`./qe`, `./benchmark`) — so install uses the local copy: no SSH, no auth prerequisite. - validate.py never checked `owner` and assumed an object source, so it passed a manifest the installer rejects. It now requires `owner.name`, resolves the relative-path source form, and hard-fails any plugin whose source points back at this repo. Also documents the version-gated `/plugin:skill` slash form (v2.1.216+) and the natural-language fallback in docs/using-skills.md. Marketplace version 0.1.0 -> 0.1.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Validator: report manifest errors instead of crashing on them The resolve_source refactor removed the `path` local from check_plugin, but three error strings still referenced it — so the name-mismatch, no-skills-dir, and empty-skills branches raised NameError instead of printing the diagnostic the validator exists to print. CI stayed green only because a healthy tree never enters those branches (review A1). Hoist the repo-relative path once after resolve_source returns and reuse it everywhere. Both reachable branches verified by hand: deleting benchmark/skills/ and emptying it now yield the intended one-line diagnostics with exit 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Line endings: normalize the two CRLF files, add .gitattributes EVALUATION_FRAMEWORK.md and ge_arrow_REPORT.md were CRLF in an otherwise-LF repo, so any future edit of either would render as a whole-file diff burying the real change (review E3). Pure normalization — zero content change, verified with `git diff --ignore-cr-at-eol`. The .gitattributes makes the normalization structural instead of a contributor convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Scoring: validate evidence before scoring; honest sensitivity stamp Three guards from the PR #5 review, all closing the same failure class — a contract that was documented but not enforced: - validate_evidence() (review B3, B4): score.py now refuses evidence that omits a scored input the verdict gates read, or marks a structural criterion met without a citation. Both new v2 fields failed open: a missing baseline_as_used_seconds silently disarmed the no-conversion verdict, and stripping every citation left the score unchanged. as_used_runs must now be present; [] remains legal as an explicit single-run declaration. Both evidence files gain the key (score-neutral: same single-run path). The check is a separate pass over authored evidence, never inside a scorer, so score_all stays a pure function of evidence and the perturbation search never silently drops mutants that trip authoring checks. - Honest perturbation count (review E5): tested increments only after a successful scoring call, and perturbations that raise are recorded in perturbations_skipped with the exception instead of being silently counted in the denominator the stamp is judged on. - robust-at-floor (review C1, floor half): a verdict already in the bottom band cannot be perturbed downward, so zero deciding flips there is partly the band's geometry, not evidence strength. markov_asset now stamps robust-at-floor with the reason attached; SKILL.md carries the stamp verbatim into reports, so plain "robust" was asserting support the run never demonstrated. Scorecards regenerated: totals, verdicts, and gates unchanged; the diff is the new fields plus markov_asset's stamp wording. The rubric's floor comment also stops restating the measured baselines (review E2) and points at evidence.json as the value the gate reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Rubric: matches_under_x64 caps correctness on its own (review C3) The x64-divergence cap required max_delta_shipped > 1e-8 as well, in both score_correctness and the derived logic&design bug cap. That conjunct made the guard structurally unable to fire when the shipped float32 delta is small — which is exactly the "wrong economics masked by low precision" case the framework's correctness section names as the thing this dimension guards. A candidate with divergent logic and a lucky 1e-12 shipped delta scored correctness 5 and total 3.25; it now scores correctness 1, logic_design capped at 3, total 2.30 gated to net regression. The flag's semantics come from the repo's own usage: the worked examples record TRUE for x64-noise residuals (~1e-14 to ~1e-11), so FALSE asserts the economics genuinely differ — not a failure of bitwise identity. Under that reading, agreement as shipped is luck, not correctness, and the cap needs no second condition. No committed scorecard changes: both worked examples have max_delta_shipped > 1e-8, so they were already caught by the old conjunct — verified byte-identical regeneration under both engines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fixtures: synthetic evidence pinning the rubric v2 paths (review B1) Neither worked example exercises the v2 headline features: both take the single-run efficiency fallback (no as_used_runs), and markov_asset hand-sets the correctness-bug flag, so the derived-cap path never runs on committed data. The newest scoring behaviour was tested only by hand-probing. references/fixtures/rubric_v2 is a synthetic evidence file — not an evaluation — whose one job is to make five untested paths execute on every scoring run: the unconditional x64 cap, the derived bug cap firing against a hand-set FALSE, the as_used_runs median, the contested-band annotation (runs straddle the 1.3x edge), and a verdict gate reporting the ungated total. The baseline sits above the 1 s floor on purpose so no-conversion does not mask the paths under test; every source string starts SYNTHETIC: so nobody cites the numbers as evidence about a lecture. Kept out of references/examples/ because an example records what was measured about a real PR and a fixture is a test input — conflating them invites citing synthetic data. Its own sensitivity line documents the stake: flipping matches_under_x64 alone swings the outcome from gated net regression to 4.70 clear improvement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * CI: scorecards must reproduce from evidence (review B2) The system's central claim — no score is ever written by hand — was verified only in reviewers' terminals. CI now regenerates all three committed scorecards (both worked examples and the v2 fixture) and fails on any diff, which catches a hand-edited scorecard, an unintended scoring change, and — the important case — an intended scoring change whose baselines were not regenerated. There the failure is desirable: the fix is to re-run and commit, which forces the verdict-moving diff into the PR where a reviewer sees it. Stdlib only, so no install step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Docs: single-source the baselines; state the new rubric behaviour Review E2/E4 plus the doc side of the engine changes: - The 0.028 s / 0.087 s vs 0.035 s / 0.18 s discrepancy was two honest measurements of the same quantity presented as one. The README table now labels its column triage-time (2026-07-21) and says the gate reads each lecture's own baseline_as_used_seconds; the framework and SKILL.md stop restating the numbers and point at evidence.json (review E2). - README quotes markov_asset's verdict as the scorecard emits it — no-conversion with the banded quality alongside — instead of the stale "2.25 net regression" (review E4). - Framework Sec. 1 states the unconditional x64 cap with its rationale, the three stamp values including robust-at-floor, and records the measured-vs-adjudicated conflation in the perturbation walk as a known limit for v3: read the deciding-flip list, not the stamp alone. - SKILL.md step 5 forbids reporting robust-at-floor as robust; sanity anchors and the tutorial's quoted output line updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Reviews: the PR #5 review of record, behind the filed item numbers The 2026-07-25 external review of this PR, committed verbatim alongside the repo's other review records so the item numbers cited on #7 and #4 (C2, C4, C5, D, E1, E6, E7) resolve without the PR comment. A labeled disposition note maps items to what was applied (4fffbc9..8388e86) and where the remainder is tracked; the text is otherwise as received. Every checkable claim in it was reproduced against the branch before the fixes landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Docs: loosen the skill-naming rule rather than replace it "Verb-first skill names" reads as a rule about skills, but the thing it was protecting is that the invocation scans as an imperative — and the whole `/plugin:skill` string is what a user types. `/audit:issues` satisfies that with the verb in the plugin, yet fails the rule as written, so the rule had to move. It moves by getting weaker, not by growing a taxonomy. Both shapes are shown, neither is preferred, and the absence of a preference is stated so a later contributor reads it as undecided rather than unspecified. Ranking them now would mean generalising from three plugins and no usage; the question is parked in FUTURE-IDEAS.md with the specific thing worth watching — whether mixing shapes inside one plugin actually causes trouble or merely looks untidy. Depends on #11 for its example: `audit` must land before this does. * Docs: catalog what shipped, and hold the conventions loosely Two changes, one idea: this repo is three plugins old, and its docs were describing more certainty than it has. CATALOG.md becomes a list of what is merged and runnable, with each plugin's state stated honestly (qe is scaffolding and says so) and a tracking issue beside it. It was previously the active plan, which meant it described skills nobody could run and drifted from the repo every time work moved. Plans now live in the issues — #3, #4, and #12 for the audit plugin — where they can change without anyone mistaking them for a description of what exists. README.md drops its duplicate plugin table and points here; the qe sub-skills point at #3 rather than at a catalogue entry that no longer carries their plan. The conventions are reframed as guidance. developing-skills.md now says so at the top, says only SKILL.md is required — a skill with nothing mechanical to run should be one file, not a directory tree — and marks the three conventions that are genuinely load-bearing, each with its reason. The rest is what one or two examples happened to need, and a contributor with a reason to depart should depart and say so in the PR, because a second example is how any of this becomes a real convention. The test that separates the two: a rule earns firmness when it keeps a skill's output checkable by someone who will not re-run it. Report-first, cited claims, and the plugin-root constraint pass it. Report shapes, phase divisions and naming forms do not. * Docs: fold the audit plugin into the rebased docs Post-#11 reconciliation. README gains audit in the plugin table and the documentation index, and its layout note drops "bundle contract", which the plugin no longer has. The marketplace entry's move to the co-located `./audit` form is not here — it belongs in "Install fix: co-located plugin sources", which is the commit that established that form, and the rebase carried it there. --------- Co-authored-by: Xuanguang Li <xuanguang-li@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jul 28, 2026
…utorial for running one Preparing the first real run of /audit:issues (#16) surfaced a contradiction the plugin has carried since #11, and answering it needed a decision rather than a convention. **Finding 0.** doctrine §3 said "no branch or file changes in the audited repo"; deliverables.md says the bundle's first-choice destination is that repo's own `.dev/audits/`; and 0.1.1's `--out` default writes there too. Together they told the audit to write into the repo it promised not to touch. §3 is now narrowed to what it was always protecting — content and history: no commits, no pushes, no branches, no edits to tracked files — and states that a run may write its own working directory. Mutation, not writing, is the boundary. **Where that directory goes** is now discovery-ordered, from contact with a real repo: action-translation already gitignores `.dev/scratch/*`, so an audit run there leaves `git status` completely clean and needs no `.gitignore` edit — which would itself be a change to a tracked file. Prefer an already-ignored scratch location, fall back to an untracked `.audit/`, then to outside the checkout. **docs/tutorial-run-an-audit.md** walks the procedure end to end against action-translation (228 items), with the phase 1 numbers measured 2026-07-27. Two things distinguish it from the evaluation tutorial: there is no committed reference to reproduce, because the skill has never been run as a skill — so the run is the experiment — and step 4 asks the reader to deliberately interrupt phase 2 and restart, which is the only test of the checkpoint fix in #17. Step 6 is a ten-point review checklist whose last two items (is the tiering right, would you act on this) are the ones no self-audit can make. Also wires audit into docs/using-skills.md, which has had no audit row since the plugin shipped, plus the AGENTS.md doc map and the site TOC. NEXT-SESSION.md is added to .gitignore. AGENTS.md already says it must not be committed, but it sits in the repo root where `git add -A` sweeps it up, and it routinely holds unsent drafts and correspondence — a convention is the wrong mechanism for that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jul 28, 2026
…utorial for running one (#19) Preparing the first real run of /audit:issues (#16) surfaced a contradiction the plugin has carried since #11, and answering it needed a decision rather than a convention. **Finding 0.** doctrine §3 said "no branch or file changes in the audited repo"; deliverables.md says the bundle's first-choice destination is that repo's own `.dev/audits/`; and 0.1.1's `--out` default writes there too. Together they told the audit to write into the repo it promised not to touch. §3 is now narrowed to what it was always protecting — content and history: no commits, no pushes, no branches, no edits to tracked files — and states that a run may write its own working directory. Mutation, not writing, is the boundary. **Where that directory goes** is now discovery-ordered, from contact with a real repo: action-translation already gitignores `.dev/scratch/*`, so an audit run there leaves `git status` completely clean and needs no `.gitignore` edit — which would itself be a change to a tracked file. Prefer an already-ignored scratch location, fall back to an untracked `.audit/`, then to outside the checkout. **docs/tutorial-run-an-audit.md** walks the procedure end to end against action-translation (228 items), with the phase 1 numbers measured 2026-07-27. Two things distinguish it from the evaluation tutorial: there is no committed reference to reproduce, because the skill has never been run as a skill — so the run is the experiment — and step 4 asks the reader to deliberately interrupt phase 2 and restart, which is the only test of the checkpoint fix in #17. Step 6 is a ten-point review checklist whose last two items (is the tiering right, would you act on this) are the ones no self-audit can make. Also wires audit into docs/using-skills.md, which has had no audit row since the plugin shipped, plus the AGENTS.md doc map and the site TOC. NEXT-SESSION.md is added to .gitignore. AGENTS.md already says it must not be committed, but it sits in the repo root where `git add -A` sweeps it up, and it routinely holds unsent drafts and correspondence — a convention is the wrong mechanism for that. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposes a third plugin,
audit, for maintainer-facing bulk audits, and lands the first of its four runbooks. Draft while the family design is up for discussion — the shape matters more than the one skill, because three more get written on top of it.The family
Four procedures that are all the same kind of thing: portfolio-wide, read-only, evidence-cited, report-producing.
/audit:issues/audit:prs/audit:tech-debt/audit:translationslecture-python.myst↔lecture-python.zh-cn)Membership is gated on three tests together — bulk, read-only, report-bundle output. That keeps single-item review outside the family (
/benchmark:review-accelerationjudges one conversion; a future single-PR merge review would also live outside) and stops the plugin becoming a general runbook dump.Why a separate plugin rather than skills in
qeThe plugin is the install and enable unit, so it is also the audience boundary.
qewas defined as the author-facing surface; putting org-wide audit skills in it would hand every author and RA maintainer tooling they will never run. This is the same call that keepsbenchmarkout of theqeprefix.auditis correspondingly left out of the lecture-repo auto-install block in the README — maintainers install it themselves.Read-only is structural, not caution. It mirrors the org's own automation split, where the family line is the permission line; it makes the family safe to point at any repo and safe to run headlessly; and it keeps a report honest, since an audit that half-applied its findings would describe a repo that no longer exists. Acting on a bundle — filing the catalog, posting drafted comments,
qe gh labels sync— stays a separate human-invoked step.Structure: four siblings, no umbrella
Unlike
qe's categories, these are distinct procedures over distinct inputs, so theqepattern worth reusing is plugin-level sharing rather than the umbrella. The method is authored once and eachSKILL.mdcarries only its own subject matter:audit/references/doctrine.mdaudit/references/quantecon-context.mdaudit/references/deliverables.mdaudit/scripts/fetch_tracker.pyLong-running by design
These are hour-scale runs, which drives two design choices. Every skill runs five phases — snapshot, verify, relate, write, self-audit — and checkpoints each to disk, so a session lost to a context limit or a sleeping machine resumes at the last completed phase instead of restarting. All phases read one frozen snapshot, so the report describes a single instant and "events after the snapshot" is a stated property rather than an unnoticed gap.
fetch_tracker.pycaptures issues and PRs in any state with full comment threads in two round trips — which is what makes reading closed threads free, the one gap the first execution of this runbook found.coverage.jsondoes the mechanical half of the self-audit: items against the number sequence1..max, thread counts split open/closed, and a truncation flag for a stream returning exactly at the fetch limit.Validated against the runbook's own example repo:
What changed from the source runbook
The four paste-in fields become optional arguments with documented discovery and stated fallbacks, so the common invocation is
/audit:issues QuantEcon/action-translation. The generic method moved todoctrine.md; the skill kept what is specific to issues. Beyond that, the QuantEcon adaptations:lecture-*repo and a consumer-visible change in anaction-*repo outrank their thread activity.status-*issues are about the collector, never the generated data;project-*"stale" usually means decided-and-unwritten, which is a re-homing action rather than a close.SYNC:PRs propagate one fix across lecture repos. A sibling's merged fix counts as verified only after checking that this repo's default branch contains it.qe gh labels check/sync, and hand-pruning desynchronises a repo from its profile.owner/repo#Nreference — that auto-closes the upstream item when the text lands on a default branch.project-*,style-guide,cli, so preflight fails loudly on missing auth rather than degrading silently.[verified]/[stated]/[inferred], with a minimum citation each — making the runbook's "match confidence to inspection" rule checkable by a reader instead of a matter of tone.project-*repo, or eventually anaudit-*repo per QEP-3.Review questions
auditas the plugin name —/audit:issuesreads as a command, lines up with QEP-3'saudit-repo prefix, and already connotes observe-and-report, which is the family's defining property. Rejected:reporter(names the output, not the activity),qe-audit(would make/qe:stop meaning "the author toolkit", andbenchmarksets the precedent against it), andreview— the closest call, rejected because it collides with single-item review, which is the distinction the family is built on./review:prs(sweep every open PR) would sit one keystroke from/review(assess this one PR), andbenchmark:review-accelerationplus CATALOG.md's "focal point is PR management" already use "review" in the per-item sense.Skill names are objects, not verbs — proposed as a convention amendment, not an exception.
docs/developing-skills.mdsays verb-first skill names, which is a proxy for the rule that actually matters: the invocation reads as a command. Where the plugin is a namespace or domain (qe,benchmark) the verb has nowhere to live but the skill; where the plugin is the verb (audit), the skill is the object. Proposed replacement for that bullet:The real risk is not the noun names but mixing them — an
/audit:compare-translationslanding beside/audit:issuesbreaks the parallelism that makes the family memorable, which is the argument for writing the rule down rather than reverting.docs/arrives with Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5, so this amendment cannot land here; it is a small follow-up once Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5 merges, and the rationale sits inaudit/README.mdmeanwhile.Evidence bar.
CATALOG.md's principle is few, high-frequency skills validated against PR history. This family is justified on breadth instead — ~245 non-archived repos, and three of the four procedures already run by hand (this triage, thequantecon-pydebt report, the zh-cn work). Worth stating explicitly, since it is a different argument from the style family's.A tracking issue in the
#3/#4style should be opened for this plugin on merge; the docs currently point atCATALOG.md §3rather than a fabricated issue number.Note for the merge order
This branch is cut from
main, so its marketplace entry uses the same remote-source form as the two existing entries. #5 converts all entries to the co-located./<plugin>form that its validator requires; whichever lands second needs a one-line follow-up to match. No behavioural overlap otherwise.python scripts/validate.pypasses with all three plugins.🤖 Generated with Claude Code