QuantEcon · mmcky · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026
diff --git a/README.md b/README.md
@@ -13,6 +13,14 @@ Each plugin bundles one area of work — a skill (the instructions Claude follow
 
 The `qe` plugin is the author-facing surface: one memorable prefix for the skills authors use while editing lectures and preparing PRs. `check-style` is the umbrella (whole lecture, optional category filter, e.g. `/qe:check-style lectures/aiyagari.md figures math`); the per-category sub-skills run the same shared rules individually. `benchmark` is a specialist family for maintainers evaluating accelerated implementations. See [CATALOG.md](CATALOG.md) for the plan and [FUTURE-IDEAS.md](FUTURE-IDEAS.md) for parked candidates.
 
+## Documentation
+
+| Guide | For |
+|---|---|
+| [docs/using-skills.md](docs/using-skills.md) | Authors/reviewers: setup, invoking skills, what to expect |
+| [docs/developing-skills.md](docs/developing-skills.md) | Contributors: layout, conventions, dev loop, versioning, PR flow |
+| [benchmark/README.md](benchmark/README.md) | The evaluation skill: review mode, triage mode, report format, manual pipeline |
+
 ## Installation
 
 ### Automatic (lecture repos)

diff --git a/benchmark/README.md b/benchmark/README.md
@@ -0,0 +1,77 @@
+# benchmark plugin
+
+Evaluation tooling for QuantEcon lecture code rewrites — the question it answers is never "is JAX faster?" but **"does this implementation earn its place in this lecture?"** Lectures are teaching materials first and programs second; the plugin's rubric weights readability (0.25) above efficiency (0.15) on purpose.
+
+One skill, two modes:
+
+| Mode | Question | Needs | Produces |
+|---|---|---|---|
+| **Review** | Did this conversion PR improve the lecture? | baseline + candidate implementations | A scored report with a merge recommendation |
+| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A predicted verdict band with the binding constraint named |
+
+Status: evaluation system landed (v0.2.0); skill wiring tracked in [skills#4](https://github.com/QuantEcon/skills/issues/4). The system was developed and validated by [@xuanguang-li](https://github.com/xuanguang-li) on [lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654).
+
+## Using the skill
+
+```
+/benchmark:review-acceleration <PR number or baseline..candidate refs>   # review mode
+/benchmark:review-acceleration should we convert <lecture>?              # triage mode
+```
+
+### Review mode — what you get
+
+The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format:
+
+1. **TL;DR** — weighted score, verdict band, the decisive facts in one paragraph
+2. **Dimension table** — weight / score / weighted contribution / one-line driver each
+3. **What changed** — before/after implementation shape
+4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit
+5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes
+
+See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, net regression) for complete real reports. Verdict bands, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2.
+
+**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score.
+
+### Triage mode — before any code is written
+
+Four checks, using only the existing lecture:
+
+1. **Baseline as-used total** — replay the lecture's real call sequence (the NumPy half of an `as_used_total.py` template). This bounds the entire possible win: a lecture whose compute totals 30ms has nothing to give.
+2. **Workload-pattern match** — against the two calibrated poles: **aiyagari-shaped** (large fixed-shape arrays, many re-solves, stable static args → measured ~24× as-used win) vs **ge_arrow-shaped** (tiny models, fresh static args per call → measured ~45× as-used loss).
+3. **Crossover comparison** — the lecture's problem sizes vs the warm crossover-n from the scaling data.
+4. **Readability-cost forecast** — which concepts the conversion would force on readers (static args, `lax` carries, checkify, the float32/x64 distinction), against the prerequisite-concept bands.
+
+Then the decision rule that falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted (band 3→5), while readability (0.25) losing two bands costs −0.50 — **a conversion that costs meaningful readability cannot break even on speed alone**; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy.
+
+**Validation (2026-07-21):** triage applied blind (baseline-side data only) to the three known cases reproduces every known verdict:
+
+| Case | Baseline total | Pattern | Triage says | Full evaluation said |
+|---|---|---|---|---|
+| ge_arrow | 0.028 s | n=2/3, fresh static args | don't convert | 2.85 wash; 45× slower as-used |
+| markov_asset | 0.087 s | n=5/25, LAPACK-bound | don't convert | 2.25 net regression |
+| aiyagari pattern | 54.3 s | 200×7 fixed, 20 re-solves | convert | 23.8× as-used win |
+
+Scope limit, confirmed by the same test: triage predicts whether the prize is worth pursuing — it cannot predict conversion-quality outcomes (markov_asset's build-breaking bug was a property of the PR, invisible to triage).
+
+## Manual usage (no skill)
+
+The full recipe is in [scripts/README.md](scripts/README.md) ("Evaluating a new lecture"); quickstart from this directory:
+
+```bash
+conda activate quantecon
+python references/examples/<lecture>/scripts/run_all.py      # measure + provenance stamp
+# fill references/examples/<lecture>/evidence.json (numbers + cited yes/no answers)
+python scripts/scoring/score.py references/examples/<lecture>
+```
+
+Sanity anchors: re-running either worked example must reproduce **2.85** / **2.25**.
+
+## Map
+
+| Path | What |
+|---|---|
+| [skills/review-acceleration/](skills/review-acceleration/SKILL.md) | The skill (procedure, both modes) |
+| [scripts/scoring/](scripts/README.md) | Deterministic engine: rubric, scorer, evidence template, provenance stamp |
+| [scripts/calibration/](scripts/calibration/bellman_bench.py) | The shared HIGH-efficiency anchor (~24× ⇒ score 5) |
+| [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) | The standard in prose — weights, anchors, checklists, verdict bands |
+| [references/examples/](references/examples/README.md) | Two complete worked evaluations + the logic-check/provenance audit; the regression baseline |
diff --git a/benchmark/skills/review-acceleration/SKILL.md b/benchmark/skills/review-acceleration/SKILL.md
@@ -23,6 +23,17 @@ Given a baseline implementation (usually `main`) and a candidate (usually a PR b
 
 Never present warm-only speedups as the headline — the ge_arrow case measured 1.4–4.8× faster warm and ~45× slower as-used.
 
+## Triage mode (no candidate yet)
+
+When the question is "should this lecture be converted at all," run the prospective subset — only the existing lecture is needed:
+
+1. **Baseline as-used total**: adapt just the baseline half of an `as_used_total.py` template and replay the lecture's real call sequence — this bounds the maximum possible win (a 30 ms lecture has nothing to give).
+2. **Pattern-match** against the calibrated poles: aiyagari-shaped (large fixed shapes, many re-solves, stable static args → ~24× win) vs ge_arrow-shaped (tiny models, fresh static args per call → ~45× loss).
+3. **Crossover check**: the lecture's problem sizes vs warm crossover-n.
+4. **Readability-cost forecast**: which prerequisite concepts the conversion would force.
+
+Decision rule from the weights: efficiency (0.15) gains at most +0.30 weighted; readability (0.25) losing two bands costs −0.50 — a conversion that costs meaningful readability cannot break even on speed alone, and structural wins are usually achievable in the baseline library. Report a predicted verdict band with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow (0.028 s → don't convert), markov_asset (0.087 s → don't convert), and the aiyagari pattern (54.3 s → convert) reproduced all three known verdicts; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so.
+
 ## Calibration baseline (regression anchors)
 
 The two worked evaluations in `references/examples/` are the validation baseline — re-running their pipelines must reproduce these verdicts:

diff --git a/docs/developing-skills.md b/docs/developing-skills.md
@@ -0,0 +1,50 @@
+# Developing skills
+
+For contributors adding or modifying plugins in this repo. (Using them: [using-skills.md](using-skills.md).)
+
+## Repo layout
+
+```
+.claude-plugin/marketplace.json   # the catalogue — every plugin registers here
+scripts/validate.py               # manifest + frontmatter validation (CI runs this)
+docs/                             # these guides
+<plugin>/                         # one directory per plugin
+  .claude-plugin/plugin.json      # name, description, version
+  README.md                       # the plugin's user guide
+  skills/<skill-name>/SKILL.md    # one directory per skill
+  scripts/                        # deterministic scripts the skills drive
+  references/                     # rule/rubric content the skills read
+```
+
+The two live plugins show the two shapes: `qe` (umbrella skill + thin per-category sub-skills sharing plugin-level rules and scripts) and `benchmark` (one skill driving a deterministic engine with worked examples as its regression baseline).
+
+## Conventions
+
+- **Verb-first skill names** (`check-style`, `review-acceleration`) — they read as commands.
+- **Description quality matters**: the SKILL.md frontmatter `description` is what natural-language invocation matches against. State what the skill does, what it measures, and when to use it. `validate.py` rejects descriptions too short to trigger reliably.
+- **Report first, fix on request** — skills never silently edit; anything `build_risk` or output-changing (RNG streams) is presented, never auto-applied.
+- **Deterministic before LLM**: put everything mechanical in `scripts/` (checkable, testable, zero-false-positive bar); reserve the skill's judgement for what genuinely needs it, and require citations when it judges.
+- **Don't duplicate content across docs** — one canonical location, pointers elsewhere. Rule text, weights, and thresholds especially: restated copies drift.
+- **Self-contained plugins**: an installed plugin ships only its own directory. No relative links or paths that escape the plugin root; use absolute GitHub URLs for repo-level files, and anchor runtime paths for the installed context (issue #4 tracks the `${CLAUDE_PLUGIN_ROOT}` pattern).
+
+## Development loop
+
+```bash
+# try a plugin against a real project without installing it
+claude --plugin-dir ./benchmark
+
+# validate everything the marketplace serves
+python scripts/validate.py
+```
+
+`validate.py` checks: every catalogue entry resolves to a real directory; `plugin.json` agrees with `marketplace.json` on name/version/description; every SKILL.md has frontmatter whose `name` matches its directory. Negative-test your changes (break something on purpose; the validator must fail loudly) — a malformed manifest breaks installation silently in every consuming repo.
+
+## Versioning and releases
+
+Bump the version in **both** `plugin.json` and the plugin's `marketplace.json` entry — the validator enforces they match. Scaffolding → first usable content is a minor bump (the benchmark plugin's evaluation-system landing was 0.1.0 → 0.2.0).
+
+## PR flow
+
+- Branch, PR, CI must be green. This repo **squash-merges** — stacked branches need `git rebase --onto origin/main <old-base>` after the base PR merges (already-upstream commits drop automatically).
+- External contributions land with the contributor as git author (`--author`, GitHub noreply address unless they prefer otherwise) and integration fixes as separate commits — see PR #5 for the pattern.
+- Plans live in [CATALOG.md](../CATALOG.md) (active) and [FUTURE-IDEAS.md](../FUTURE-IDEAS.md) (parked); per-plugin work items live in issues ([#3](https://github.com/QuantEcon/skills/issues/3) style, [#4](https://github.com/QuantEcon/skills/issues/4) benchmark). The style-guide rule content is authored in `QuantEcon/style-guide`, never here — this repo's `qe` plugin consumes a rendered snapshot ([project-style-guide#6](https://github.com/QuantEcon/project-style-guide/issues/6)).
diff --git a/docs/using-skills.md b/docs/using-skills.md
@@ -0,0 +1,45 @@
+# Using QuantEcon skills
+
+For lecture authors, reviewers, and maintainers who want to *use* the skills. (Building new ones: [developing-skills.md](developing-skills.md).)
+
+## Setup
+
+**In a lecture repo that has opted in** — nothing to do. Repos that check the marketplace into `.claude/settings.json` (see the [repo README](../README.md)) install everything automatically when you open the repo in Claude Code and trust it.
+
+**Anywhere else** — two commands in a Claude Code session:
+
+```
+/plugin marketplace add QuantEcon/skills
+/plugin install qe@quantecon           # author-facing base skills
+/plugin install benchmark@quantecon    # lecture-evaluation tooling
+```
+
+**In CI** — `anthropics/claude-code-action@v1` accepts `plugin_marketplaces` and `plugins` inputs directly; see the repo README for the workflow snippet.
+
+## Invoking a skill
+
+Three ways, all equivalent:
+
+1. **Slash command** — type `/` and pick from the menu, e.g. `/benchmark:review-acceleration 717`. Trailing words are passed to the skill as arguments.
+2. **Natural language** — describe what you want ("check this lecture's figures against the style guide"; "is this JAX conversion actually an improvement?") and the matching skill triggers on its description.
+3. **Category entry points** — some plugins expose thin sub-skills (`/qe:check-figures`, `/qe:check-math`, …) so a narrow check is one keystroke and shows up in autocomplete.
+
+## What to expect
+
+- **Report first, fix on request.** Skills produce a structured report and *offer* changes; they never silently edit your files. Risky fixes (anything that breaks builds or changes published figures, e.g. RNG-stream changes) are presented but never auto-applied.
+- **Evidence, not vibes.** Reports cite rule IDs, `file:line` locations, and measured numbers. The benchmark plugin goes further: its scores are computed by a deterministic engine from recorded evidence — the session shows the full derivation.
+- **The same skill works pre-PR and in review.** Run it on your working copy before opening a PR (catch issues early), or point it at an open PR (consistent review).
+
+## The plugins
+
+| Plugin | Skills | What they do | Status |
+|---|---|---|---|
+| `qe` | `/qe:check-style` + `check-{writing,math,code,figures,jax,refs}` | Style-guide compliance for lecture source, by rule ID | scaffolding — [skills#3](https://github.com/QuantEcon/skills/issues/3) |
+| `benchmark` | `/benchmark:review-acceleration` | Score a NumPy→JAX/Numba conversion (review mode) or assess whether a lecture is worth converting (triage mode) | system landed — [guide](../benchmark/README.md), [skills#4](https://github.com/QuantEcon/skills/issues/4) |
+
+## Updating and troubleshooting
+
+- **Update**: `/plugin` → marketplace → update, or reinstall; repos with the settings.json opt-in track the marketplace automatically.
+- **Skill not in the menu?** Check the plugin is installed and enabled (`/plugin`), and that you trusted the repo. In settings-managed repos, `enabledPlugins` must list it.
+- **A skill reports "not yet operational"** — it's scaffolding; its issue link says what's pending.
+- **Version pinning**: plugin versions live in the marketplace catalogue; CI validates that every manifest is consistent, so a broken install is a bug — please open an issue.