Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired)

[mmcky]

Lands @xuanguang-li's complete evaluation system for lecture code rewrites — the deliverable #4 was waiting on, developed and validated on QuantEcon/lecture-python.myst#717 and QuantEcon/lecture-python.myst#654. Two commits, deliberately split: commit 1 is the package as delivered, authored by @xuanguang-li; commit 2 is path/plumbing integration plus docs.

What lands

Where	What
references/EVALUATION_FRAMEWORK.md	The standard in prose: 7 weighted dimensions, numeric anchors, structural checklists, verdict bands, worked HIGH/LOW examples
scripts/scoring/	The standard as code: rubric.py (evidence → score, deterministically — never hand-typed), score.py (CLI), EVIDENCE_TEMPLATE.json (the judgement contract: measured numbers + cited true/false answers)
scripts/calibration/	The shared aiyagari Bellman benchmark pinning the "25× as-used = score 5" anchor
references/examples/ge_arrow/	Complete worked evaluation: 2.85/5 mixed/wash (#717)
references/examples/markov_asset/	Complete worked evaluation: 2.25/5 net regression — build-breaking bug (#654; reported there)

The two worked cases double as the regression baseline: the skill must reproduce their verdicts. Verified in this PR — both scorecards regenerate byte-identically from the new layout via python scripts/scoring/score.py references/examples/<lecture>.

Integration (commit 2)

• score.py takes a lecture directory path (runs from any cwd); ge_arrow scripts converted to local imports matching the markov_asset idiom
• run_all.py writes a provenance stamp (results/env.json: python/platform/library versions) — the seed of the PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335 shared result + environment-descriptor schema
• All relative links in the framework and reports rewritten for the new layout (no dangling references)
• scripts/README.md rewritten around the three-step contract (measure → record evidence → score); SKILL.md procedure now points at the real engine and templates, with the worked cases as calibration anchors
• benchmark plugin 0.1.0 → 0.2.0, marketplace kept in sync; validate.py green

Design note

Per-lecture measurement scripts are adapted templates, not a fixed harness — adapting them to each lecture's examples and call sequence is exactly the step the skill automates, while scoring stays deterministic. This supersedes the "generalised CLI" idea in #4 item 2.

🤖 Generated with Claude Code

Update 2026-07-22 — docs merged in, rubric v2, skill wired (plugin 0.3.0)

This PR now also carries three follow-ups so the whole system can be tested from this branch before landing on main:

1. PR Docs: skill usage, repo setup, and validated triage mode #6 squash-merged in (2aea122): user/developer docs, the benchmark user guide, validated triage mode, the design-review record (reviews/), and the markov_asset corrections of record.
2. Rubric v2 (719e825) — the four engine quick-wins from PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7 that survived the three-way design review, each validated against the committed evidence files: safety couplings enforced in score_all (derived logic&design bug-cap; correctness 1/2 gates the verdict band — the review's honest-evidence 4.2 hole now gates to net regression), the no-conversion verdict (reconciles review mode with triage), a one-flip sensitivity stamp (robust/fragile with deciding flips), and K-repeat as-used (median of 3 fresh-process runs with contested-band annotation).
3. Skill wiring (a36a744): evaluations run in the user's workspace (benchmark-eval/<lecture>/) with the plugin read-only at CLAUDE_PLUGIN_ROOT; preconditions and the extraction/replay diff check added to the procedure.

Re-validation: both totals are unchanged (ge_arrow 2.85, markov_asset 2.25); both verdicts now lead with no-conversion, markov_asset's additionally gated at net regression. ge_arrow stamps fragile (the review's demonstrated flips, listed in the scorecard); markov_asset stamps robust — the v2 gate absorbs the one-concept band flip the review demonstrated. Band movements are documented as deliberate v2 changes in both reports. Plugin 0.2.0 → 0.3.0, marketplace in sync, validate.py green.

Update 2026-07-25 — review hardening (7 commits)

An item-by-item external review of this PR was applied in 4fffbc9..8388e86; the full mapping from review items to changes is in the response comment. Headlines: the validator reports manifest errors instead of raising NameError; CI now regenerates all committed scorecards from evidence and fails on any diff; score.py validates evidence before scoring (a missing gate input or an uncited met-criterion is an error, closing the fail-open paths); matches_under_x64 caps correctness at 1 unconditionally (the Copilot x64 thread, now resolved there); a synthetic fixture (references/fixtures/rubric_v2) pins the five previously-untested v2 paths on every CI run.

One correction to the 2026-07-22 update above: markov_asset's stamp is now robust-at-floor, not robust — zero deciding flips at the bottom band is partly the band's geometry (nothing can push a floored verdict lower), and the scorecard now says so rather than overclaiming. Totals and verdicts are otherwise unchanged: ge_arrow 2.85, markov_asset 2.25, both leading with no-conversion. Deferred methodology items are filed on #7, remaining housekeeping on #4.

[Copilot]

Pull request overview

This PR lands the benchmark plugin’s lecture-acceleration evaluation framework, including a prose rubric plus a deterministic scoring engine and two worked evaluation baselines (ge_arrow, markov_asset) that can be regenerated from evidence files.

Changes:

• Adds a deterministic scoring engine (scripts/scoring/) that computes dimension scores + weighted totals from per-lecture evidence.json.
• Adds a shared efficiency calibration benchmark (scripts/calibration/bellman_bench.*) and updates plugin/marketplace versions to 0.2.0.
• Adds the evaluation framework document plus two fully-worked evaluation example directories (scripts, results, evidence, reports) used as regression anchors.

Reviewed changes

Copilot reviewed 42 out of 43 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
README.md	Updates plugin status messaging for benchmark.
benchmark/skills/review-acceleration/SKILL.md	Replaces placeholder procedure with the new 3-step evidence→score workflow and calibration anchors.
benchmark/scripts/scoring/score.py	Adds the scoring CLI that reads <lecture>/evidence.json and writes <lecture>/results/scorecard.json.
benchmark/scripts/scoring/rubric.py	Implements the rubric as code (weights, thresholds, checklists, verdict bands).
benchmark/scripts/scoring/EVIDENCE_TEMPLATE.json	Introduces the evidence schema template for new evaluations.
benchmark/scripts/README.md	Documents the scripts layout and end-to-end evaluation workflow.
benchmark/scripts/calibration/bellman_bench.py	Adds the shared high-end efficiency calibration benchmark.
benchmark/scripts/calibration/bellman_bench.json	Commits the benchmark output pinning the “score 5” efficiency anchor.
benchmark/references/examples/markov_asset/scripts/static_metrics.py	Adds markov_asset static metrics extraction used in scoring evidence.
benchmark/references/examples/markov_asset/scripts/smoke_test.py	Adds a runnable smoke test for old/new markov_asset implementations.
benchmark/references/examples/markov_asset/scripts/run_all.py	Adds markov_asset pipeline orchestration + provenance stamp + shared scoring invocation.
benchmark/references/examples/markov_asset/scripts/model_old.py	Adds extracted baseline NumPy implementation under evaluation.
benchmark/references/examples/markov_asset/scripts/model_new.py	Adds extracted candidate JAX implementation under evaluation (verbatim).
benchmark/references/examples/markov_asset/scripts/check_equivalence.py	Adds markov_asset equivalence checking (float32 vs x64) and patched-call-option comparison.
benchmark/references/examples/markov_asset/scripts/benchmark.py	Adds markov_asset warm scaling benchmark.
benchmark/references/examples/markov_asset/scripts/as_used_total.py	Adds markov_asset as-used (cold-inclusive) end-to-end timing script.
benchmark/references/examples/markov_asset/results/static_metrics.json	Adds captured metrics output used by the evidence/scorecard.
benchmark/references/examples/markov_asset/results/scorecard.json	Adds computed scorecard output for markov_asset baseline.
benchmark/references/examples/markov_asset/results/scaling.json	Adds captured scaling results for markov_asset baseline.
benchmark/references/examples/markov_asset/results/equivalence_x64_True.json	Adds captured x64 equivalence results for markov_asset baseline.
benchmark/references/examples/markov_asset/results/equivalence_x64_False.json	Adds captured float32-as-shipped equivalence results for markov_asset baseline.
benchmark/references/examples/markov_asset/markov_asset_REPORT.md	Adds the worked markov_asset evaluation report.
benchmark/references/examples/markov_asset/evidence.json	Adds the worked markov_asset evidence file consumed by the scorer.
benchmark/references/examples/ge_arrow/scripts/sweep_bench.py	Adds ge_arrow sweep benchmark used by the evaluation pipeline.
benchmark/references/examples/ge_arrow/scripts/static_metrics.py	Adds ge_arrow static metrics extraction used in scoring evidence.
benchmark/references/examples/ge_arrow/scripts/run_all.py	Adds ge_arrow pipeline orchestration + provenance stamp + shared scoring invocation.
benchmark/references/examples/ge_arrow/scripts/model_old.py	Adds extracted baseline NumPy implementation under evaluation.
benchmark/references/examples/ge_arrow/scripts/model_new.py	Adds extracted candidate JAX implementation under evaluation (verbatim).
benchmark/references/examples/ge_arrow/scripts/cold_start.py	Adds cold-start latency measurement for ge_arrow evaluation.
benchmark/references/examples/ge_arrow/scripts/check_equivalence.py	Adds ge_arrow equivalence checking across all lecture examples.
benchmark/references/examples/ge_arrow/scripts/benchmark.py	Adds ge_arrow benchmark (as-used, warm, scaling) used in scoring evidence.
benchmark/references/examples/ge_arrow/scripts/as_used_total.py	Adds ge_arrow as-used (cold-inclusive) end-to-end timing script.
benchmark/references/examples/ge_arrow/results/sweep.json	Adds captured sweep benchmark results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/static_metrics.json	Adds captured metrics output used by the evidence/scorecard.
benchmark/references/examples/ge_arrow/results/scorecard.json	Adds computed scorecard output for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/equivalence.json	Adds captured equivalence results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/benchmark.json	Adds captured benchmark results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/ge_arrow_REPORT.md	Adds the worked ge_arrow evaluation report.
benchmark/references/examples/ge_arrow/evidence.json	Adds the worked ge_arrow evidence file consumed by the scorer.
benchmark/references/EVALUATION_FRAMEWORK.md	Adds the evaluation framework specification, anchors, and workflow.
benchmark/.claude-plugin/plugin.json	Bumps benchmark plugin version to 0.2.0.
.gitignore	Adds standard Python/macOS ignore patterns.
.claude-plugin/marketplace.json	Updates marketplace version to 0.2.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mmcky]

Detailed logic review completed — all 17 scripts line-by-line, plus scripted cross-checks of every evidence value against its results-file source. Full write-up now lives in the PR as benchmark/references/examples/README.md (the logic-check and provenance document for the canonical examples).

Verdict: logic sound, methodology fair, every traceable number consistent. Fairness properties verified: block_until_ready on all JAX timings, _clear_cache() for cold trials, medians over repeats, identical call sequences per side in the as-used replays, the markov_asset bug patched only where timing is otherwise impossible (disclosed in docstring and output record), and a non-strawman NumPy baseline in the calibration (same vectorised algorithm, agreement to 1e-14).

Fixed in 433462b (mechanical, no scored value changes; both scorecards still regenerate byte-identically):

	Fix
M2	The headline as-used metric lived only on the console — run_all.py now persists the fresh-process scripts' JSON into results/as_used.json (+ cold_start.json for ge_arrow) with the derived as_used_speedup, as the docstrings already claimed
m4	ge_arrow static-metrics had a duplicated @ pattern double-counting concept token hits (informational metric; regenerated 110 → 105)
m5	markov_asset used statements_for_one_asset where the template/engine vocabulary is statements_for_one_result (values unchanged)

For @xuanguang-li's consideration (rubric/design territory — deliberately not changed here):

1. M1: n_prerequisite_concepts (the readability driver, weight 0.25 — the heaviest input in the system) and statements_for_one_result are hand-curated judgements encoded in the measurement scripts, disclosed as such. When the skill adapts templates to a new lecture it will author these lists, so they arguably belong in evidence.json as cited judgement slots — one citation per claimed concept, the same discipline the structural checklists enforce — rather than script constants.
2. m3: ge_arrow's equivalence.json doesn't stamp the x64 regime and is overwritten between regimes; markov_asset's equivalence_x64_{bool}.json + _meta pattern is the better template — worth back-porting on the next revision.
3. n6: sweep_bench's old side computes only α while the new one-call API forcibly computes everything — faithful to each API as used, and arguably the point (the API's cost is real), but worth a sentence in the script docstring.

Also still open from the earlier Copilot round: the matches_under_x64 guard semantics (see the reply on that thread) — same category, the standard's author's call.

🤖 Generated with Claude Code

[mmcky]

Final review completed (31-agent adversarially-verified pass: integration correctness traced path-by-path, measured duplication analysis, doc-overlap mapping, installed-plugin packaging, fresh-eyes diff review, and an execution agent that ran every documented command). Fixes landed in eacd5fe; both scorecards still regenerate byte-identically and every documented command works as written.

Two additional items for @xuanguang-li (found by the fresh-eyes pass; your files, your call):

1. The framework's HIGH-anchor table doesn't match the committed calibration data. EVALUATION_FRAMEWORK.md §Dim-3 quotes 1664/69 ms (24×) and 29.3/1.16 s (25×), but the committed scripts/calibration/bellman_bench.json records 2955/112 ms (26.4×) and 54.29/2.28 s (23.8×) — absolute timings differ ~1.8×, well beyond the ±15% run-to-run band, i.e. the prose table and the committed JSON come from different runs/machines. The speedups agree (~24–26×, so the anchor itself is sound); options are updating the table to the committed values or noting the table is from the original validation machine.

2. The jax 0.4.x pins in the docstrings are drifting — four places pin it while environments move on (the machine here now has jax 0.10.1). The new per-run env.json stamp (with steps_failed for partial runs) is the durable answer; the prose pins could soften to "the env recorded in results/env.json".

Deferred simplification bundle (measured, verified outcome-preserving, but touching your scripts' structure — proposed for whenever a third example lands or the next re-measurement, whichever comes first):

Consolidation	Measured	Est.
Shared run_steps.py runner (run_all is now ~89% identical across examples and fully lecture-independent)	78/88 LCS-identical lines	−50 LOC now, −70/lecture after
markov_asset's bug-patched call_option exists as a renamed clone in two scripts — the equivalence-verified copy is not the timed copy	AST-identical after alpha-renaming	one patched_call.py sibling, −16 LOC
ge_arrow's four lecture economies hand-built at four sites	~53 LOC	one economies.py sibling, −30 LOC
static_metrics AST core shared — best bundled with the M1 (prerequisite-concepts into evidence.json) decision	49–50 LCS lines, already diverging	conditional on M1

Deliberately not proposed: sharing the median-timer/results-path/allclose helpers — measured as a net negative (couples the templates the adaptation model wants independent).

🤖 Generated with Claude Code

[mmcky]

Applied the review (items referenced by its numbering) in seven commits, 4fffbc9..8388e86. Every checkable claim in the review reproduced locally before anything was changed.

Invariant held throughout: neither published evaluation moved — ge_arrow is still 2.85 / no-conversion / fragile, markov_asset still 2.25 / no-conversion / gated. All scorecards regenerate byte-identically (md5-checked across repeated runs); the only scorecard diffs are additive fields plus markov_asset's stamp wording (see C1 below).

Item	Change
A1	validate.py reports the diagnostic instead of raising NameError on the three malformed-plugin paths; both reachable branches tested by hand
B2	CI regenerates all three committed scorecards and fails on any diff — a rubric change that moves a verdict now has to land its regenerated baselines in the same PR
B3	score.py refuses evidence that omits a scored input the verdict gates read. Deviation from the suggested fix: as_used_runs must be present but [] stays legal as an explicit single-run declaration — requiring it non-empty would have invalidated both committed evidence files and made single-run unexpressible
B4	Structural criteria marked met must carry citations; stripping every citation now errors with 9 named violations instead of scoring an identical 2.25. Implemented as a validation pass over authored evidence, not inside score_structural, so score_all stays pure and the sensitivity search never silently drops mutants
B1	Synthetic fixture references/fixtures/rubric_v2 executes the five untested v2 paths (unconditional x64 cap, derived bug cap, runs median, contested band, gate with ungated total) on every CI run; kept out of examples/ and marked SYNTHETIC throughout so nobody cites it as measurement
C3	matches_under_x64: false caps correctness at 1 unconditionally, in both the scorer and the derived bug cap. The repo's own usage settles the semantics the review left open: the examples record TRUE for x64-noise residuals, so FALSE asserts the economics differ. The masked-divergence case (divergent logic, 1e-12 shipped delta) went from correctness 5 / total 3.25 to correctness 1 / 2.30 gated. Isolated in its own commit since it changes verdict-moving semantics
C1	Floor half only: markov_asset now stamps robust-at-floor — zero deciding flips at the bottom band is partly geometry (nothing can push it lower), and reporting that as plain robust asserted support the run never demonstrated. The measured-vs-adjudicated partition is recorded in the framework as a known limit for v3
E2	Diagnosed as two honest measurements of the same quantity, not staleness: the README table is now labelled triage-time (2026-07-21), and the framework, SKILL.md, and rubric.py stop restating numbers and point at evidence.json as what the gate reads
E3	.gitattributes (* text=auto eol=lf) plus pure normalization of the two CRLF files, committed separately so content diffs stay readable
E4	README quotes markov_asset's verdict as the scorecard emits it
E5	Perturbation counter increments only on successful scoring; raising perturbations are recorded in perturbations_skipped with the exception rather than silently absorbed into the denominator

Deferred, deliberately: C2 (floor keying on the baseline alone — a policy decision worth making explicitly, though note triage already decides from the baseline alone, so the asymmetry is a live review/triage disagreement), C5 (pin the run-pairing definition before any as_used_runs data is committed — zip truncates silently and median-of-ratios ≠ ratio-of-medians), C4 (fill_evidence.py; B3's guard now catches the consequence of the transcription gap it addresses), D (comparative-vs-absolute — v3, since making docstring_cov a delta moves ge_arrow's readability score), and E1/E6/E7. These can be tracked in #4.

[mmcky]

@xuanguang-li I am going to merge this PR to keep working on this in the context of the repo in general. The only change may be repointing to main rather than the branch but let me know if you have any trouble.