PLAN: benchmark — from v0 outline to operational /benchmark:review-acceleration

[mmcky]

Development plan for taking /benchmark:review-acceleration from its v0 outline to an operational skill. This is workstream B of the benchmarking programme (QuantEcon/meta#335); the evaluation system it packages was designed and validated by @xuanguang-li on lecture-python.myst#717.

Where we are

In place	Pending
Skill scaffold with the v0 procedure (equivalence → static metrics → as-used benchmark → 7-dimension weighted scoring → verdict-banded report)	Operational skill body
benchmark/scripts/README.md documenting the 8 evaluation scripts	The scripts themselves — @xuanguang-li agreed (2026-07-07) to package and send them
Full rubric specified in the lecture-python.myst#717 thread (dimensions, weights, numeric anchors, verdict bands, worked HIGH/LOW examples)	Rubric transcribed into references/rubric.md
Named calibration endpoints: HIGH = aiyagari Bellman pattern (~25× faster as-used), LOW = ge_arrow (~45× slower as-used)	Validation runs against the two live PRs

Work items

1. references/rubric.md (unblocked — the natural next PR)

• Transcribe the rubric from the #717 thread: 7 dimensions with weights (correctness & numerical fidelity 0.20, readability & pedagogical clarity 0.25, computational efficiency as-used 0.15, logic & design 0.15, coding style & idiom 0.10, API ergonomics 0.10, maintainability 0.05), the 1–5 numeric scoring anchors per dimension, and the verdict bands (≥4.0 merge; 3.0–3.9 merge after fixes; 2.5–2.9 revisit; <2.5 do not merge)
• Include the worked HIGH/LOW calibration cases (aiyagari pattern, ge_arrow) with their measured numbers, as scoring anchors
• Encode the measurement standard: as-used, fresh-process totals (JIT compile time included) are the headline; warm timings reported alongside, never alone; crossover-n curve; recompile audit (one recompile per distinct static-arg value or shape)

2. Land the evaluation scripts (blocked on the script handoff)

• Receive and commit the 8 scripts: check_equivalence.py, static_metrics.py, benchmark.py, cold_start.py, sweep_bench.py, as_used_total.py, bellman_bench.py, run_all.py
• Generalise from ge_arrow-specific code to take a lecture/implementation pair as input (baseline ref + candidate ref + lecture path)
• Add an environment descriptor to every emitted result (hardware, library versions, runner class) — aligning with the shared result-schema deliverable in PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335, which names this the first blocking foundation

3. Operational skill body (blocked on 1–2)

• Replace the v0 outline with the driving procedure: given baseline/candidate refs for one lecture, run the scripts, score each dimension against the rubric anchors with cited evidence, compute the weighted total, emit the verdict + a fix list mapping each recommendation to the dimension it lifts
• Report format suitable for posting as a PR review comment
• Guardrail from the calibration data: warm-only speedup claims are flagged, never accepted as the headline (the ge_arrow lesson — warm 1.4–4.8× faster, as-used ~45× slower)

4. Validation (blocked on 3)

• Run against lecture-python.myst#717 (ge_arrow) — expect agreement with the published evaluation (weighted ≈2.95, do-not-merge-as-is); note the branch is far behind main and needs updating first
• Run against lecture-python.myst#654 (markov_asset) — the second named validation case
• Sanity-check the HIGH anchor on the aiyagari Bellman pattern

5. Documentation + rollout (blocked on 4)

• Distill the rubric into the QuantEcon manual as the companion to the JAX style page — QuantEcon.manual#104 tracks this ("skill first, then documentation")
• Tick off the corresponding workstream-B items on PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335; version bump; README status flip from "under construction"

Boundary

/qe:check-jax (skills#3) checks how JAX is written against the style guide; this skill judges whether an accelerated implementation earns its place in a lecture — the boundary QuantEcon.manual#104 draws. The "should this lecture use JAX at all" question belongs here.

🤖 Generated with Claude Code

[mmcky]

Status update: @xuanguang-li delivered the package (2026-07-21) and it exceeds what this plan expected — not 8 raw scripts but a complete system with the rubric machine-encoded (scoring/rubric.py: scores are deterministic functions of a cited evidence.json, never hand-typed). Landing in #5. Consequences for the work items:

• Item 1 (rubric transcription) — superseded. EVALUATION_FRAMEWORK.md + scripts/scoring/ land as the canonical rubric; it post-dates the #717 thread version (structural 4-item checklists, override caps, a does-not-build gate; ge_arrow re-scored 2.85 vs the thread's 2.95). QuantEcon.manual#104 should distill from these files, not the thread.
• Item 2 (scripts) — landed, design revised. Per-lecture measurement scripts are adapted templates, not a generalised harness — template adaptation is precisely the step the skill automates, and lecture diversity (ge_arrow's λ-sweep vs markov_asset's checkify) genuinely resists a rigid CLI. The provenance stamp (results/env.json) is added in Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5, seeding the PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335 shared schema.
• Item 3 (skill body) — unblocked. SKILL.md's procedure now points at the real engine; remaining work is making the skill drive it end-to-end on a fresh PR.
• Item 4 (validation) — largely delivered. Both named validation cases are complete worked evaluations and now serve as the regression baseline (both scorecards regenerate byte-identically in Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5). markov_asset's findings (2.25/5, build-breaking bug) were reported to [markov_asset] Update to JAX and runtime comparison lecture-python.myst#654.
• Item 5 (manual docs + rollout) — unchanged, now unblocked once Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5 merges.

[mmcky]

Housekeeping remainders from the PR #5 review (response comment has the applied set; methodology items went to #7):

• E1 — results/env.json is gitignored for the reference examples, but evidence records source PRs and SHAs, not library versions, so there is no committed record of what produced the reference measurements. Suggest committing an env.reference.json for the two worked cases.
• E6 — in score_readability, the docstring sub-score bottoms out at 2 while the prereq sub-score reaches 1, so inside the worse-of-two a docstring catastrophe can never be the sole cause of a 1. Minor asymmetry, worth a deliberate yes/no.
• E7 — scripts/scoring/ has no __init__.py; the sys.path.insert works for the CLI but python -m does not. Matters only if the engine gets imported from elsewhere.

[mmcky]

Status after #5 merged — the "Where we are" table above is now stale in the plugin's favour.

Landed: the plugin is at 0.3.0 with the rubric in benchmark/references/EVALUATION_FRAMEWORK.md, the deterministic scoring engine in benchmark/scripts/scoring/ (rubric.py, score.py, env_stamp.py), rubric v2 in full (#7), two complete worked evaluations as regression baselines, synthetic fixtures pinning the v2 paths, and CI that reproduces every scorecard from its evidence file. The skill body is operational for workspace runs with ${CLAUDE_PLUGIN_ROOT} anchoring. The plugin guide and the evaluation framework are now published at https://quantecon.github.io/skills/.

Work item 2 resolved differently from the plan, which is worth recording. This issue anticipated landing eight generalised evaluation scripts as a fixed harness. What shipped instead is measurement templates inside the worked examples, adapted per lecture by the skill — SKILL.md states that adapting them is the skill's job and that there is deliberately no rigid harness, because the call sequence that matters is the lecture's own. So the checkbox for "receive and commit the 8 scripts / generalise from ge_arrow-specific code" will not be ticked as written; the need behind it is met. The environment descriptor from that item did land, as env_stamp.py.

Genuinely still open: the full v2 evaluation of lecture-python.myst#654 and its adjudication under the expert-parallel protocol (#10), which together are the skill's acceptance test. That PR thread still has its clean slate.