TESTING: review + expert-parallel protocol for the benchmark skill (@xuanguang-li)

[mmcky]

Testing and review protocol for @xuanguang-li — everything we'd like your eyes on, in order, with the machinery validation already done so your time goes to judgement, not reproduction.

Context. Your evaluation system is packaged as the benchmark plugin on PR #5: the deterministic engine (benchmark/scripts/scoring/), the framework (EVALUATION_FRAMEWORK.md), your ge_arrow (#717 → 2.85) and markov_asset (#654 → 2.25) evaluations as regression anchors, and a rubric-v2 revision from the 2026-07-21 adversarial design review (your measurement foundation was endorsed by both review tracks and is unchanged). We have already re-validated end-to-end from a fresh checkout of lecture-python.myst#717 on jax 0.10.1: exact reproduction of 2.85 / no-conversion / fragile with the same deciding flips — see reviews/validation-run-ge_arrow-2026-07-22.md. You do not need to repeat that exercise.

1. Review the rubric v2 changes (on PR #5)

The v2 changes touch only the verdict layer of your rubric — dimension scoring is untouched. Please push back on anything; rubric semantics stay yours.

• Verdict gates — correctness 1 caps the verdict at net regression, correctness 2 at mixed/wash, enforced in score_all (rubric.py); the logic&design bug-cap is now derived from the correctness evidence (builds / x64 divergence) rather than a hand-set boolean
• No-conversion verdict — fires when the baseline as-used total is under a 1 s materiality floor and the candidate is slower as-used; reconciles review mode with your triage bound (the polished-ge_arrow case). The 1 s floor is a labeled policy choice — is it the right one?
• Sensitivity stamp — score.py perturbs every scored input one at a time (booleans flipped, counts ±1, floats ±10%) and stamps the scorecard robust/fragile with deciding flips listed
• K-repeat as-used — median of 3 fresh-process runs per side, with a contested-band annotation when runs span a band edge
• The flagged Copilot thread: x64 divergence currently forces correctness 1 only when shipped drift also exceeds 1e-8 (your original semantics). Copilot suggested unconditional; we deliberately left it as authored — your call, in the PR #5 review thread

2. Three design calls (comment on #7)

• Readability instrument v2 — docstring coverage inverts your own exemplars (odu.py, your LOW example, measures 0.86; flagship-aiyagari style ~0.41–0.55, capped at 2). Proposal: equation-traceability — per numbered equation, cite the single implementing expression; fraction traceable old vs new, same citation discipline as your checklists
• Extraction/replay verification — a mechanical diff of extracted code and replayed call sequence against the lecture's cells (our validation run caught an undisclosed whitespace normalisation this way); longer-term, whether executing the lecture itself at both refs should become the as-used source
• Confirm or amend the x64 guard decision from item 1 as a rubric-level statement

3. Optional — see the skill in action

Not required (the validation already covers the machinery), but if you want to feel out what reviewers will receive, from a lecture-python.myst checkout:

git clone https://github.com/QuantEcon/skills ~/skills
cd ~/skills && git checkout land-evaluation-system

then in a Claude Code session:

/plugin marketplace add ~/skills
/plugin install benchmark@quantecon
/benchmark:review-acceleration 717

Evaluation artifacts land in benchmark-eval/<lecture>/ in your workspace; #717 should land on exactly 2.85 / no-conversion / fragile (your numbers are the answer key). Any conversion PR works. The by-hand walkthrough of the same steps is docs/tutorial-run-an-evaluation.md. What we most want from this: critique the report — is it something you would sign?

4. The acceptance test — expert-parallel on lecture-python.myst#654

The part only you can do (#8 milestone). Protocol:

1. The skill runs a complete v2 evaluation of #654 from scratch (fresh extraction, measurement, evidence, score, report) — we drive this and post the report here, not on #654
2. You evaluate #654 independently (or, if you prefer, adjudicate the skill's report line by line — your choice; independent-first is the stronger test)
3. We reconcile: every discrepancy resolved by evidence, with the rubric amended where your judgement and the instrument disagree
4. The reconciled report becomes the single authoritative comment on #654 — which is being kept deliberately clean until then

• Skill evaluation of #654 posted on this issue
• Independent expert evaluation (or line-by-line adjudication) by @xuanguang-li
• Reconciliation recorded; rubric amendments (if any) filed on PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7
• Authoritative comment posted to lecture-python.myst#654

Comment here, on PR #5, or on #7 — whatever is easiest. Relates to #4 (skill wiring), #7 (rubric v2), #8 (work plan).

[xuanguang-li]

Hi @mmcky,

Thanks for this helpful summary!

I followed the Optional — see the skill in action to replicate the test, but there are some problems when I ran the commands in the terminal.

1. marketplace.json schema error
   After git clone https://github.com/QuantEcon/skills ~/skills, running /plugin marketplace add ~/skills failed with: Invalid schema: ... owner: Invalid input: expected object, received undefined.

The manifest was missing the required owner object (name + optional email). Adding it resolved the error.

2. Plugin clone goes over SSH, not HTTPS
   /plugin install benchmark@quantecon tried to clone over SSH (git@github.com) and failed with No ED25519 host key is known for github.com.

The cause is that the marketplace only indexes the plugin — the plugin source resolves to a separate repo that gets cloned fresh. That's reasonable, but it means a first-time user needs working GitHub SSH auth before the install can succeed. Getting through it required trusting GitHub's host key and registering an SSH key on my account. A note in the README about the auth prerequisite (and whether the plugin's source repo is public or access-restricted) would save a lot of time.

3. Skill invocation isn't discoverable
   After a successful install, /benchmark:review-acceleration 717 returned Unknown command: /benchmark:review-acceleration Args from unknown skill: 717, even though /plugin reported Status: Enabled, and /plugin enable benchmark@quantecon responded: "already enabled".

The plugin ships skills only — no commands/ directory — so it wasn't obvious from the outside how the documented invocation was meant to resolve.

Environment: Windows, PowerShell, Claude Code CLI, benchmark@quantecon v0.3.0.

The skill functions well after resolving these problems. The friction was all upstream of actually using it.

[mmcky]

Thanks @xuanguang-li — this is exactly the friction we needed surfaced, and all three were real. Each is now fixed at the root on the branch (PR #5). Here's what each one was and what changed.

1. Missing owner. You were right: owner is a required top-level object in the marketplace schema and ours omitted it, so /plugin marketplace add failed schema validation for everyone. Added "owner": { "name": "QuantEcon" }. The real gap was that our own scripts/validate.py never checked for owner, so it happily passed a manifest the installer rejects — it now requires it, so this can't slip through CI again.

2. SSH clone. Root cause found. Each plugin entry declared its source as the remote GitHub repo (source: { source: "github", repo: "QuantEcon/skills", … }). Because the plugins live inside the marketplace repo you'd just added, that told Claude Code to discard the local copy and re-clone QuantEcon/skills from GitHub — over SSH — purely to reach a subdirectory already on disk. That's your ED25519 failure. The fix is the documented pattern for a plugin co-located with its marketplace: a relative-path source ("source": "./benchmark"). Install now uses the copy already present, so there is no SSH step and no auth prerequisite at all — rather than document the prerequisite, we removed it. The validator now rejects any plugin whose source points back at this repo.

3. Unknown command. Two things were in play, and your read that it "wasn't obvious how the invocation resolves" was on target. First, the SSH-fought install may have left the plugin registered-but-not-populated — your error said unknown skill, not unknown plugin — which a clean install after fix #2 should settle. Second, and the more likely direct cause: the plugin-prefixed form /benchmark:review-acceleration is version-gated. Before Claude Code v2.1.216 the skill registered under the bare /review-acceleration and the prefixed form didn't resolve. What's your claude --version? If it's below 2.1.216 that alone explains it. Two fallbacks that work on any version: the bare /review-acceleration, and natural language — describe the task ("evaluate whether the JAX conversion in lecture-python.myst#654 is worth merging") and the skill triggers on its description. We deliberately did not add a commands/ shim for the prefixed form: skills already are slash commands on current Claude Code, and a second entry point would only be one more thing to drift.

To retry: pull the branch again and re-run /plugin marketplace add + /plugin install benchmark@quantecon — the install should be clean now, with no SSH. If /benchmark:review-acceleration 654 still doesn't resolve, send me your version and use /review-acceleration 654 or the natural-language form in the meantime.

None of this touched the evaluation machinery — it was all install surface, exactly as you called it, upstream of the skill actually running. The rubric-v2 review and the lecture-python.myst#654 acceptance test remain the parts we most want your eyes on.

[xuanguang-li]

Thanks @mmcky. This solves all the problems.

My Claude Code version is 2.1.218. I think it's the latest version.

After this update, the skill setup becomes very straightforward. I can use the /benchmark command after cloning the repo without setting /plugin marketplace add ~/skills and /plugin install benchmark@quantecon.

[mmcky]

Interesting @xuanguang-li -- I wonder if this is a new skill provided by Claude or if this is our custom benchmarking tool?