Docs: skill usage, repo setup, and validated triage mode

CATALOG.md

@@ -7,6 +7,7 @@ Current focus, validated against ~630 merged PRs across the four main lecture re
 - **Few, high-frequency skills** over many niche ones; every skill is validated against actual PR history.
 - **Focal point is PR management**: consistent review results, plus the same checks run by authors on a working copy before opening a PR.
 - **Report first, fix on request.** Skills produce a structured report and offer fixes; they never silently edit. Safe in CI, authors stay in control.
+- **Cited claims; computed scores.** Every finding carries a citation (rule ID + `file:line`, or a number + its source). Skills whose output is a findings list need nothing more. Skills that aggregate judgements into a scored verdict use the evidence-file pattern from the benchmark plugin: judgement recorded as cited answers, every score computed by a deterministic engine, never typed by hand (see [docs/developing-skills.md](docs/developing-skills.md)).
 
 ## 1. Style skill family — flagship
 

README.md

@@ -13,6 +13,14 @@ Each plugin bundles one area of work — a skill (the instructions Claude follow
 
 The `qe` plugin is the author-facing surface: one memorable prefix for the skills authors use while editing lectures and preparing PRs. `check-style` is the umbrella (whole lecture, optional category filter, e.g. `/qe:check-style lectures/aiyagari.md figures math`); the per-category sub-skills run the same shared rules individually. `benchmark` is a specialist family for maintainers evaluating accelerated implementations. See [CATALOG.md](CATALOG.md) for the plan and [FUTURE-IDEAS.md](FUTURE-IDEAS.md) for parked candidates.
 
+## Documentation
+
+| Guide | For |
+|---|---|
+| [docs/using-skills.md](docs/using-skills.md) | Authors/reviewers: setup, invoking skills, what to expect |
+| [docs/developing-skills.md](docs/developing-skills.md) | Contributors: layout, conventions, dev loop, versioning, PR flow |
+| [benchmark/README.md](benchmark/README.md) | The evaluation skill: review mode, triage mode, report format, manual pipeline |
+
 ## Installation
 
 ### Automatic (lecture repos)

benchmark/README.md

@@ -0,0 +1,77 @@
+# benchmark plugin
+
+Evaluation tooling for QuantEcon lecture code rewrites — the question it answers is never "is JAX faster?" but **"does this implementation earn its place in this lecture?"** Lectures are teaching materials first and programs second; the plugin's rubric weights readability (0.25) above efficiency (0.15) on purpose.
+
+One skill, two modes:
+
+| Mode | Question | Needs | Produces |
+|---|---|---|---|
+| **Review** | Did this conversion PR improve the lecture? | baseline + candidate implementations | A scored report with a merge recommendation |
+| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A predicted verdict band with the binding constraint named |
+
+Status: evaluation system landed (v0.2.0); skill wiring tracked in [skills#4](https://github.com/QuantEcon/skills/issues/4). The system was developed and validated by [@xuanguang-li](https://github.com/xuanguang-li) on [lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654).
+
+## Using the skill
+
+```
+/benchmark:review-acceleration <PR number or baseline..candidate refs>   # review mode
+/benchmark:review-acceleration should we convert <lecture>?              # triage mode
+```
+
+### Review mode — what you get
+
+The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format:
+
+1. **TL;DR** — weighted score, verdict band, the decisive facts in one paragraph
+2. **Dimension table** — weight / score / weighted contribution / one-line driver each
+3. **What changed** — before/after implementation shape
+4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit
+5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes
+
+See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, net regression) for complete real reports. Verdict bands, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2.
+
+**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score.
+
+### Triage mode — before any code is written
+
+Four checks, using only the existing lecture:
+
+1. **Baseline as-used total** — replay the lecture's real call sequence (the NumPy half of an `as_used_total.py` template). This bounds the entire possible win: a lecture whose compute totals 30ms has nothing to give.
+2. **Workload-pattern match** — against the two calibrated poles: **aiyagari-shaped** (large fixed-shape arrays, many re-solves, stable static args → measured ~24× as-used win) vs **ge_arrow-shaped** (tiny models, fresh static args per call → measured ~45× as-used loss).
+3. **Crossover comparison** — the lecture's problem sizes vs the warm crossover-n from the scaling data.
+4. **Readability-cost forecast** — which concepts the conversion would force on readers (static args, `lax` carries, checkify, the float32/x64 distinction), against the prerequisite-concept bands.
+
+Then the decision rule that falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted (band 3→5), while readability (0.25) losing two bands costs −0.50 — **a conversion that costs meaningful readability cannot break even on speed alone**; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy.
+
+**Validation (2026-07-21):** triage applied blind (baseline-side data only) to the three known cases reproduces every known verdict:
+
+| Case | Baseline total | Pattern | Triage says | Full evaluation said |
+|---|---|---|---|---|
+| ge_arrow | 0.028 s | n=2/3, fresh static args | don't convert | 2.85 wash; 45× slower as-used |
+| markov_asset | 0.087 s | n=5/25, LAPACK-bound | don't convert | 2.25 net regression |
+| aiyagari pattern | 54.3 s | 200×7 fixed, 20 re-solves | convert | 23.8× as-used win |
+
+Scope limit, confirmed by the same test: triage predicts whether the prize is worth pursuing — it cannot predict conversion-quality outcomes (markov_asset's masked `err.throw()` defect was a property of the PR, invisible to triage). Note also that this validation is **in-sample** — the three cases are the ones the thresholds were calibrated on; out-of-sample validation accumulates as fresh lectures are triaged.
+
+## Manual usage (no skill)
+
+The full recipe is in [scripts/README.md](scripts/README.md) ("Evaluating a new lecture"); quickstart from this directory:
+
+```bash
+conda activate quantecon
+python references/examples/<lecture>/scripts/run_all.py      # measure + provenance stamp
+# fill references/examples/<lecture>/evidence.json (numbers + cited yes/no answers)
+python scripts/scoring/score.py references/examples/<lecture>
+```
+
+Sanity anchors: re-running either worked example must reproduce **2.85** / **2.25**.
+
+## Map
+
+| Path | What |
+|---|---|
+| [skills/review-acceleration/](skills/review-acceleration/SKILL.md) | The skill (procedure, both modes) |
+| [scripts/README.md](scripts/README.md) | Deterministic engine (`scripts/scoring/`): rubric, scorer, evidence template, provenance stamp |
+| [scripts/calibration/](scripts/calibration/bellman_bench.py) | The shared HIGH-efficiency anchor (~24× ⇒ score 5) |
+| [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) | The standard in prose — weights, anchors, checklists, verdict bands |
+| [references/examples/](references/examples/README.md) | Two complete worked evaluations + the logic-check/provenance audit; the regression baseline |

benchmark/references/examples/README.md

@@ -66,7 +66,7 @@ Lucas-tree, consol, and call-option pricing over a Markov chain (default: 25-sta
 
 ### The bug and the near-critical precision finding
 
-- `smoke_test.py` demonstrates the crash exactly as the lecture calls it: `call_option_jit(...)` → `NameError: name 'err' is not defined`. Two lecture cells (consol/call-option cell, Exercise 1) depend on it → **the lecture does not build as shipped** → correctness 1 by the does-not-build override.
+- `smoke_test.py` demonstrates the failure in a clean namespace: `call_option_jit(...)` → `NameError: name 'err' is not defined` → correctness 1 by the does-not-build override, under the system's fresh-process measurement regime. **Corrected 2026-07-21:** the notebook itself executes in cell order — earlier cells bind a global `err` that the stray `err.throw()` silently resolves to, which means the checkify stability validation is *never actually performed* in the shipped lecture (a masked failure rather than a crash); a reader copying the function into a clean namespace hits the `NameError`. See the REPORT's erratum.
 - `check_equivalence.py` additionally compares a **bug-patched copy** (verified line-identical to shipped logic minus the stray `err.throw()`) to establish that the *intended* logic is right: under x64 every asset matches NumPy to ≈1e-11. As shipped (float32) drift reaches **1.02e-2** on the exercise model — and that model's spectral radius (1.0618) sits **0.002 below** the stability bound 1/β = 1.0638, so float32 is not merely imprecise but close to flipping the stability check itself. Regimes are stored separately (`equivalence_x64_{True,False}.json`) with the x64 flag stamped in `_meta` — the pattern ge_arrow's template should adopt (caveat m3).
 
 ### The measurements
@@ -102,7 +102,7 @@ correctness 1 (does not build) + readability 2 + efficiency 2 + logic 3 (capped)
 2. **Scorecard reproduction:** `score.py` regenerates both committed scorecards **byte-identically** from `evidence.json` alone.
 3. **Evidence↔results cross-check (scripted):** every quantitative evidence slot matches its results-file source (Δprereq, docstring coverage, max\|Δ\| shipped, crash record, statements).
 4. **Rubric edge audit:** brute force over all 5⁷ score combinations found FP band-edge misclassifications (797 cases), fixed by computing the verdict from the rounded total; neither reference case was affected.
-5. **Fairness audit:** `block_until_ready` on every JAX timing; fresh processes for as-used; medians over repeats; identical call sequences per side; disclosed patches only where timing is otherwise impossible.
+5. **Fairness audit:** `block_until_ready` on every JAX timing; fresh processes for as-used; medians over repeats in the warm/scaling benchmarks (**correction:** the as-used totals themselves are single passes per side — a known limitation, tracked for v2); identical call sequences per side at the level the replay scripts encode (**correction:** later review found both replays deviate from the lecture in construction patterns — see the design-review documents in `reviews/`); disclosed patches only where timing is otherwise impossible.
 
 ## Known caveats (recorded, deliberate, or pending upstream)
 
@@ -111,5 +111,5 @@ correctness 1 (does not build) + readability 2 + efficiency 2 + logic 3 (capped)
 | **M1** | `n_prerequisite_concepts` (readability driver, weight 0.25) and `statements_for_one_result` (ergonomics) are **hand-curated judgements encoded in the measurement scripts**, disclosed as such — not AST measurements. When the skill adapts templates to a new lecture it authors these lists, so they need the same citation discipline as the structural checklists. Proposed: move them into `evidence.json` as cited judgement slots. | Raised with the system's author (@xuanguang-li) on skills PR #5 |
 | m3 | ge_arrow's `equivalence.json` does not stamp the x64 regime and is overwritten between regimes; markov_asset's split-file pattern (`equivalence_x64_{bool}.json` + `_meta`) is the better template | Adopt on next template revision |
 | n6 | `sweep_bench` asymmetry: the old sweep computes only α (constructor + `wealth_distribution`), while the new one-call API forcibly computes everything — faithful to each API as used, but part of the measured sweep disadvantage is API-induced | Documented; by design (the API's cost is real) |
-| — | markov_asset's as-used timing requires the bug-patched `call_option`; the shipped code cannot complete at all | Disclosed in script, output record, and report |
+| — | markov_asset's as-used timing requires the bug-patched `call_option`; the shipped code cannot complete in a clean namespace (in notebook order it runs with the validation silently masked — see the REPORT erratum) | Disclosed in script, output record, and report |
 | — | Benchmarks are CPU-only; timings vary ±~15% run-to-run — the rubric keys on orders of magnitude | Framework limitation note |

benchmark/references/examples/markov_asset/markov_asset_REPORT.md

@@ -1,5 +1,7 @@
 # Evaluation Report — `markov_asset.md`: NumPy (`main`) → JAX (`update_markov_asset`)
 
+> **Erratum (2026-07-21).** This report states that "the lecture does not build as shipped." Design review later established this is false *as worded*: executing the notebook's cells **in order** completes cleanly (and the PR's preview CI passes), because earlier cells bind a global `err` immediately before `call_option_jit` is first traced, so the stray `err.throw()` inside `call_option` resolves to that stale, already-checked object and silently does nothing. The defect is real and arguably worse than a crash: the stale-global masking means the checkify stability validation inside `call_option` is **never actually performed** in the shipped lecture — on the model whose spectral radius sits 0.002 below the stability bound — and any reader copying the function into a clean namespace hits the `NameError` this report describes. The evaluation's `builds: false` refers to fresh-process/clean-namespace execution of the extracted implementation (the system's declared measurement regime); the verdict and must-fix list stand. Nothing has been posted to [lecture-python.myst#654](https://github.com/QuantEcon/lecture-python.myst/pull/654) — the PR will receive one authoritative evaluation once the rubric-v2 revision lands and the skill has run the full protocol. See the design-review documents in `reviews/`.
+
 Applies the system in [`../../EVALUATION_FRAMEWORK.md`](../../EVALUATION_FRAMEWORK.md) to the NumPy→JAX conversion of `markov_asset.md`. All numbers are reproduced by `scripts/run_all.py` (CPU, jax 0.4.35, numpy 2.1.3, quantecon) into `results/`. Every dimension score is **computed from [`evidence.json`](evidence.json) by the shared rubric** (`../../../scripts/scoring/rubric.py`) — see `results/scorecard.json` for the derivation.
 
 ## TL;DR — weighted score **2.25 / 5** → *net regression as shipped: do not merge until fixed*

benchmark/skills/review-acceleration/SKILL.md

@@ -23,12 +23,23 @@ Given a baseline implementation (usually `main`) and a candidate (usually a PR b
 
 Never present warm-only speedups as the headline — the ge_arrow case measured 1.4–4.8× faster warm and ~45× slower as-used.
 
+## Triage mode (no candidate yet)
+
+When the question is "should this lecture be converted at all," run the prospective subset — only the existing lecture is needed:
+
+1. **Baseline as-used total**: adapt just the baseline half of an `as_used_total.py` template and replay the lecture's real call sequence — this bounds the maximum possible win (a 30 ms lecture has nothing to give).
+2. **Pattern-match** against the calibrated poles: aiyagari-shaped (large fixed shapes, many re-solves, stable static args → ~24× win) vs ge_arrow-shaped (tiny models, fresh static args per call → ~45× loss).
+3. **Crossover check**: the lecture's problem sizes vs warm crossover-n.
+4. **Readability-cost forecast**: which prerequisite concepts the conversion would force.
+
+Decision rule from the weights: efficiency (0.15) gains at most +0.30 weighted; readability (0.25) losing two bands costs −0.50 — a conversion that costs meaningful readability cannot break even on speed alone, and structural wins are usually achievable in the baseline library. Report a predicted verdict band with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow (0.028 s → don't convert), markov_asset (0.087 s → don't convert), and the aiyagari pattern (54.3 s → convert) reproduced all three known verdicts; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so.
+
 ## Calibration baseline (regression anchors)
 
 The two worked evaluations in `references/examples/` are the validation baseline — re-running their pipelines must reproduce these verdicts:
 
 - **`ge_arrow`** ([#717](https://github.com/QuantEcon/lecture-python.myst/pull/717)): **2.85/5 — mixed/wash.** Tiny 2×2/3×3 economies, fresh static args per call → ~45× slower as-used despite warm wins.
-- **`markov_asset`** ([#654](https://github.com/QuantEcon/lecture-python.myst/pull/654)): **2.25/5 — net regression.** Build-breaking `NameError` (stray `err.throw()`), float32 drift near a critical stability margin.
+- **`markov_asset`** ([#654](https://github.com/QuantEcon/lecture-python.myst/pull/654)): **2.25/5 — net regression.** A stray `err.throw()` that crashes in any clean namespace and, in notebook order, silently disables the checkify stability validation (a masked failure — see the REPORT erratum); float32 drift near a critical stability margin.
 - **HIGH anchor:** the aiyagari Bellman pattern (`scripts/calibration/bellman_bench.py`) — large fixed-shape arrays, many re-solves; ~25× faster as-used → the "score 5" calibration.
 
 The rubric will also be distilled into the QuantEcon manual as the companion to the JAX style page ([QuantEcon.manual#104](https://github.com/QuantEcon/QuantEcon.manual/issues/104)).