Docs: skill usage, repo setup, and validated triage mode - #6
Merged
Conversation
- benchmark/README.md: the plugin's user guide — review mode (session
walkthrough, report format), triage mode ("should this lecture be
converted?"), manual pipeline quickstart, plugin map
- SKILL.md: triage-mode section (the prospective subset: baseline
as-used total, pattern match against the calibrated poles, crossover
check, readability-cost forecast, and the weight-algebra decision
rule)
- docs/using-skills.md: consumer guide — setup paths, invocation forms,
report-first expectations, troubleshooting
- docs/developing-skills.md: contributor guide — layout, conventions,
dev loop, versioning, squash-merge/stacking and external-author
attribution patterns
- README.md: documentation index
Triage mode is empirically validated before being documented: blind
triage using only baseline-side data (fresh runs: ge_arrow 0.028s,
markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s)
reproduces all three known verdicts (don't convert / don't convert /
convert), and correctly cannot predict conversion-quality defects
(markov_asset's build bug) — that scope limit is documented with it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a set of repo + plugin documentation pages covering (1) how to install/invoke QuantEcon skills, (2) how to develop/validate plugins in this repository, and (3) how to use the benchmark evaluation skill in both review and validated triage mode.
Changes:
- Adds top-level documentation entrypoints in
README.mdand introduces two new guides:docs/using-skills.mdanddocs/developing-skills.md. - Documents
benchmarkplugin usage (review + triage modes) inbenchmark/README.md. - Extends the
review-accelerationskill procedure to include a triage mode section.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds a Documentation section linking to the new guides and the benchmark plugin user guide. |
| docs/using-skills.md | New end-user guide for setup, invocation, expectations, and troubleshooting. |
| docs/developing-skills.md | New contributor guide for repo layout, conventions, validation workflow, versioning, and PR flow. |
| benchmark/skills/review-acceleration/SKILL.md | Adds a “Triage mode” procedure and decision rule rationale to the skill doc. |
| benchmark/README.md | New benchmark plugin user guide describing review vs triage, manual pipeline quickstart, and repo map. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Addresses Copilot review on #6: the manual-install snippet is three commands, not two; the benchmark map row's link text now matches its target (scripts/README.md) with the engine path named in the description instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The evidence-file + engine pattern applies to skills that aggregate judgements into scored verdicts; findings-list skills need only cited claims. developing-skills.md gets the three-tier discipline; CATALOG gains the principle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent design critiques of the evaluation system (a fresh unframed session; a 36-agent adversarial workflow with steelman defense) were merged in reviews/. Three of our own claims were falsified by execution and are corrected here: - markov_asset's lecture DOES build in notebook order: a stale global err masks the stray err.throw(), silently disabling the checkify stability validation (worse than a crash, but not a build failure). Erratum prepended to the REPORT; wording corrected in examples README, SKILL.md, plugin README; correction posted on lecture-python.myst#654 - "mirrors the lecture exactly": both reference replays deviate from the lectures' construction patterns; certification corrected - "medians over repeats": false for the as-used totals (single pass per side); fairness-audit wording corrected; triage validation noted as in-sample reviews/ holds the independent report and the merged synthesis: which critiques survived the steelman defense (convention-only safety couplings; readability instrument inverting its own exemplars; the review/triage mode contradiction; band-label semantics; noise vs resolution) and which were defended (ratio form, weighted total, min() shape, per-consequence billing), plus the ranked v2 plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9 tasks
The evaluation findings briefly posted to lecture-python.myst#654 were withdrawn; the PR will receive one authoritative evaluation after the rubric-v2 revision and a full skill run, rather than a comment-and-correction trail. Erratum and merged review updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
12 tasks
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
* Docs: plugin guide, skill usage, repo setup — with triage mode validated
- benchmark/README.md: the plugin's user guide — review mode (session
walkthrough, report format), triage mode ("should this lecture be
converted?"), manual pipeline quickstart, plugin map
- SKILL.md: triage-mode section (the prospective subset: baseline
as-used total, pattern match against the calibrated poles, crossover
check, readability-cost forecast, and the weight-algebra decision
rule)
- docs/using-skills.md: consumer guide — setup paths, invocation forms,
report-first expectations, troubleshooting
- docs/developing-skills.md: contributor guide — layout, conventions,
dev loop, versioning, squash-merge/stacking and external-author
attribution patterns
- README.md: documentation index
Triage mode is empirically validated before being documented: blind
triage using only baseline-side data (fresh runs: ge_arrow 0.028s,
markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s)
reproduces all three known verdicts (don't convert / don't convert /
convert), and correctly cannot predict conversion-quality defects
(markov_asset's build bug) — that scope limit is documented with it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Docs: fix command count and map-table link text
Addresses Copilot review on #6: the manual-install snippet is three
commands, not two; the benchmark map row's link text now matches its
target (scripts/README.md) with the engine path named in the
description instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Docs: scale the evidence discipline by output tier, not blanket
The evidence-file + engine pattern applies to skills that aggregate
judgements into scored verdicts; findings-list skills need only cited
claims. developing-skills.md gets the three-tier discipline; CATALOG
gains the principle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Design review: corrections of record + merged three-way synthesis
Two independent design critiques of the evaluation system (a fresh
unframed session; a 36-agent adversarial workflow with steelman
defense) were merged in reviews/. Three of our own claims were
falsified by execution and are corrected here:
- markov_asset's lecture DOES build in notebook order: a stale global
err masks the stray err.throw(), silently disabling the checkify
stability validation (worse than a crash, but not a build failure).
Erratum prepended to the REPORT; wording corrected in examples
README, SKILL.md, plugin README; correction posted on
lecture-python.myst#654
- "mirrors the lecture exactly": both reference replays deviate from
the lectures' construction patterns; certification corrected
- "medians over repeats": false for the as-used totals (single pass
per side); fairness-audit wording corrected; triage validation
noted as in-sample
reviews/ holds the independent report and the merged synthesis:
which critiques survived the steelman defense (convention-only safety
couplings; readability instrument inverting its own exemplars; the
review/triage mode contradiction; band-label semantics; noise vs
resolution) and which were defended (ratio form, weighted total,
min() shape, per-consequence billing), plus the ranked v2 plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update references after withdrawal of the #654 comments
The evaluation findings briefly posted to lecture-python.myst#654 were
withdrawn; the PR will receive one authoritative evaluation after the
rubric-v2 revision and a full skill run, rather than a
comment-and-correction trail. Erratum and merged review updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jul 27, 2026
…, skill wired) (#5) * Add the lecture evaluation system: rubric engine, calibration, worked cases The quantitative evaluation system for lecture code rewrites developed and validated on QuantEcon/lecture-python.myst#717 and #654: - references/EVALUATION_FRAMEWORK.md — the standard in prose: 7 weighted dimensions, numeric scoring anchors, structural checklists, verdict bands, worked HIGH/LOW examples - scripts/scoring/ — the standard as code: rubric.py (deterministic evidence -> score), score.py (engine/CLI), EVIDENCE_TEMPLATE.json (the judgement contract: measured numbers + cited yes/no answers) - scripts/calibration/ — the shared aiyagari Bellman benchmark pinning the "25x as-used = score 5" efficiency anchor - references/examples/{ge_arrow,markov_asset}/ — two complete worked evaluations (measurement scripts, results, evidence, reports): ge_arrow 2.85/5 mixed/wash; markov_asset 2.25/5 net regression (build-breaking bug) Content as delivered 2026-07-21; placed at plugin-convention paths. Path/link integration follows in a separate commit. * Integrate the evaluation system into the plugin layout Integration on top of the landed package (content authored by @xuanguang-li; this commit is path/plumbing only plus docs): - score.py takes a lecture directory path (works from any cwd) instead of a name resolved against the old package root - ge_arrow scripts use local imports (import model_old), matching the markov_asset idiom, so every script runs directly from its directory - run_all.py (both examples): scoring call updated to the new layout, lecture dir derived not hardcoded, and a provenance stamp written to results/env.json (python/platform/numpy/jax/quantecon versions) -- the seed of the QuantEcon/meta#335 shared result schema - All relative links in EVALUATION_FRAMEWORK.md and the two reports rewritten for the new layout (verified: no dangling references) - scripts/README.md rewritten: engine layout, the three-step scoring contract, the evaluate-a-new-lecture recipe - SKILL.md updated: system landed, operational procedure now points at the real engine/templates, worked cases become regression anchors - benchmark plugin 0.1.0 -> 0.2.0 (marketplace kept in sync) Verified: both example scorecards regenerate byte-identically from the new layout; scripts/validate.py green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * scoring: verdict from the rounded total; fix stale paths in docstring/template Addresses Copilot review on #5: - rubric.py computes the verdict band from the rounded total, so the band always agrees with the number displayed (raw FP sums can land at 2.4999999999999996 for combinations that are exactly 2.50 in exact arithmetic; 797/78125 score combinations were affected) - rubric.py docstring points at ../../references/EVALUATION_FRAMEWORK.md - EVIDENCE_TEMPLATE.json _how cites the actual CLI form (scripts/scoring/score.py <lecture-dir> from the plugin root) Both committed scorecards regenerate unchanged (neither sits at a band edge). The x64-divergence guard in score_correctness is deliberately left as authored — rubric semantics stay with the standard's author. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Persist the headline metrics; fix two static-metrics inconsistencies From the detailed logic review of the evaluation scripts: - run_all.py (both examples) now captures the JSON lines printed by the fresh-process scripts (as_used_total, cold_start) into results/as_used.json / results/cold_start.json, with the derived as_used_speedup - the headline metric previously lived only on the console, though the docstrings already claimed aggregation - ge_arrow static_metrics: remove the duplicated "@" pattern that double-counted concept token hits (informational metric only; regenerated results: old.concept_token_hits 110 -> 105) - markov_asset static_metrics: rename statements_for_one_asset -> statements_for_one_result, matching EVIDENCE_TEMPLATE.json and the ge_arrow template vocabulary (values unchanged; results regenerated) Both scorecards regenerate byte-identically - no scored value changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Document the reference examples: logic check and provenance benchmark/references/examples/README.md explains both canonical cases in detail - the economics, the two implementations, where every evidence number comes from, and why each verdict is what it is - and records the 2026-07-21 line-by-line verification (scorecard byte reproduction, evidence-results cross-checks, rubric edge audit, fairness audit) plus the known caveats (M1 hand-curated readability inputs, m3 x64 stamping, n6 sweep asymmetry). These examples are the regression baseline for the skill; their accuracy is now auditable rather than asserted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Final-review fixes: harden run_all, share the env stamp, doc consistency From the adversarially-verified final review (31-agent pass over the full PR): Robustness (both run_all.py files — all four confirmed by execution): - guard against JSON-scalar stdout lines (json.loads succeeds on `42`/ `null`; .get then raised AttributeError and aborted the pipeline) - track per-step returncodes; failed step titles now go into the provenance stamp so a partial run cannot claim full provenance for stale results - symmetric total_s guard in the as_used_speedup derivation (bare numpy-side index could KeyError) - warn on duplicate mode keys instead of silently overwriting Simplification (integrator-authored code only): - the byte-identical 18-line write_env block duplicated in both run_all.py files becomes shared scripts/scoring/env_stamp.py, invoked like score.py (-26 LOC net; the shared meta#335 schema now has one definition) Consistency: - gitignore the per-run generated results (as_used.json, cold_start.json, env.json) and annotate their doc citations as generated-not-committed (committing them faithfully is impossible here: the local env is jax 0.10.1 vs the reports' 0.4.35) - examples/README: fix 'logic 5' -> 4 (scorecard and its own arithmetic say 4; with 5 the listed scores sum to 3.00, not 2.85) - REPORT link labels updated to match their (already-correct) targets; markov REPORT's stale statements_for_one_asset key renamed - evidence.json _how strings and score.py's scorecard _note now cite scripts/scoring/... (scorecards regenerated; only the _note changed) - SKILL.md no longer restates the weight vector and verdict bands -- it points at EVALUATION_FRAMEWORK.md sections 1-2 and rubric.py, so recalibration cannot drift the copies - scripts/README: commands documented as running from the plugin root Both scorecards regenerate byte-identically under the updated engine; validate.py green; env_stamp smoke-tested including steps_failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: skill usage, repo setup, and validated triage mode (#6) * Docs: plugin guide, skill usage, repo setup — with triage mode validated - benchmark/README.md: the plugin's user guide — review mode (session walkthrough, report format), triage mode ("should this lecture be converted?"), manual pipeline quickstart, plugin map - SKILL.md: triage-mode section (the prospective subset: baseline as-used total, pattern match against the calibrated poles, crossover check, readability-cost forecast, and the weight-algebra decision rule) - docs/using-skills.md: consumer guide — setup paths, invocation forms, report-first expectations, troubleshooting - docs/developing-skills.md: contributor guide — layout, conventions, dev loop, versioning, squash-merge/stacking and external-author attribution patterns - README.md: documentation index Triage mode is empirically validated before being documented: blind triage using only baseline-side data (fresh runs: ge_arrow 0.028s, markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s) reproduces all three known verdicts (don't convert / don't convert / convert), and correctly cannot predict conversion-quality defects (markov_asset's build bug) — that scope limit is documented with it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: fix command count and map-table link text Addresses Copilot review on #6: the manual-install snippet is three commands, not two; the benchmark map row's link text now matches its target (scripts/README.md) with the engine path named in the description instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Docs: scale the evidence discipline by output tier, not blanket The evidence-file + engine pattern applies to skills that aggregate judgements into scored verdicts; findings-list skills need only cited claims. developing-skills.md gets the three-tier discipline; CATALOG gains the principle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Design review: corrections of record + merged three-way synthesis Two independent design critiques of the evaluation system (a fresh unframed session; a 36-agent adversarial workflow with steelman defense) were merged in reviews/. Three of our own claims were falsified by execution and are corrected here: - markov_asset's lecture DOES build in notebook order: a stale global err masks the stray err.throw(), silently disabling the checkify stability validation (worse than a crash, but not a build failure). Erratum prepended to the REPORT; wording corrected in examples README, SKILL.md, plugin README; correction posted on lecture-python.myst#654 - "mirrors the lecture exactly": both reference replays deviate from the lectures' construction patterns; certification corrected - "medians over repeats": false for the as-used totals (single pass per side); fairness-audit wording corrected; triage validation noted as in-sample reviews/ holds the independent report and the merged synthesis: which critiques survived the steelman defense (convention-only safety couplings; readability instrument inverting its own exemplars; the review/triage mode contradiction; band-label semantics; noise vs resolution) and which were defended (ratio form, weighted total, min() shape, per-consequence billing), plus the ranked v2 plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update references after withdrawal of the #654 comments The evaluation findings briefly posted to lecture-python.myst#654 were withdrawn; the PR will receive one authoritative evaluation after the rubric-v2 revision and a full skill run, rather than a comment-and-correction trail. Erratum and merged review updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rubric v2: enforced couplings, no-conversion verdict, sensitivity stamp, K-repeat Implements items 1-4 of #7 (the changes that survived the three-way design review), validated against both committed evidence files: - score_all derives the logic&design bug-cap from the correctness evidence (builds / x64-divergence) instead of trusting a hand-set boolean, and gates the verdict: correctness 1 caps at net regression, correctness 2 at mixed/wash. The review's honest-evidence 4.2 hole (float32 catastrophe, no logic bug -> "merge") now gates to net regression. - no-conversion verdict: baseline as-used under the 1 s materiality floor (a labeled policy choice) + slower as-used candidate -> the scorecard says don't convert instead of scoring the polish. Reconciles review with triage. - sensitivity stamp in score.py: every scored input perturbed one at a time (bools flipped, counts +/-1, floats +/-10%); scorecard stamped robust/fragile with deciding flips listed. - K-repeat as-used: run_all.py repeats each as-used side 3x in fresh processes; the headline speedup is a median, per-run speedups feed a contested-band annotation in the engine. Re-validation: ge_arrow re-scores 2.85 (unchanged), verdict now no-conversion (candidate band mixed/wash), stamped fragile with exactly the review's demonstrated flips. markov_asset re-scores 2.25 (unchanged), no-conversion + gated net regression, stamped robust across all 29 perturbations - the gate absorbs the one-concept band flip the review demonstrated. Both band movements are deliberate v2 changes, noted in the reports. Benchmark plugin bumped to 0.3.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Skill wiring: plugin-root anchoring and workspace evaluation directory Operationalizes /benchmark:review-acceleration for installed-plugin runs (#4 item 3): evaluations are built under <workspace>/benchmark-eval/<lecture>/ with the plugin read-only at CLAUDE_PLUGIN_ROOT (run_all.py already resolves the shared engine from that env var); preconditions stated up front; extraction/replay diff check added to scaffold; the v2 verdict outputs (gates, no-conversion, sensitivity stamp) carried through the procedure, triage decision rule, and calibration anchors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: local testing tiers and the switch back to the production marketplace Adds a Testing locally section to the contributor guide: --plugin-dir for skill iteration, a local-path marketplace for full install simulation before merging (test from a consuming project; the checkout's branch is what gets served; the marketplace name collides with production), and the remove/add/install sequence to return to the GitHub source afterwards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validation run: ge_arrow re-evaluated from a fresh checkout, verdict reproduced The skills#8 dry run, targeting the motivating PR the system was first developed on (lecture-python.myst#717) rather than the reserved #654 acceptance case. Fresh partial clone at base 8cfba4c / head 8c2d0d7, wired workspace procedure (benchmark-eval/<lecture>/ with CLAUDE_PLUGIN_ROOT), jax 0.10.1 vs the reference 0.4.35: reproduces 2.85 / no-conversion / fragile with the same three deciding flips; every measured quantity moved only within its band. Full cross-comparison in reviews/validation-run-ge_arrow-2026-07-22.md. Fixes surfaced by the run: - ge_arrow check_equivalence.py now writes equivalence_x64.json under JAX_ENABLE_X64 instead of clobbering the as-shipped results (the markov_asset template already did this per-regime) - model_old.py fidelity note discloses the cosmetic whitespace normalisation found by diffing against a fresh extraction - both evidence files now record source_pr + base/head SHAs (provenance was previously PR-number-and-branch only) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: hands-on evaluation tutorial built on the ge_arrow validation run Walks the review-acceleration procedure end-to-end by hand - checkout at the recorded SHAs, workspace scaffold, K-repeat measurement, the two precision regimes and why each exists, evidence, scoring, band-based cross-comparison - with every command and number taken from the recorded validation run so readers can check their results against a committed reference. Linked from the README docs table and the benchmark guide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docs: AGENTS.md as the canonical agent guide, led by single source of truth Adds the repo-level instructions file for AI agents and contributors, following the QuantEcon.manual convention: AGENTS.md is canonical and CLAUDE.md imports it. Its governing principle is @jstac's — skills point to existing documentation in the manual wherever possible instead of repeating what the manual says — worked out concretely for this repo: rule text stays upstream in style-guide, numbers live once, every topic has an owning doc, and cross-boundary references are links rather than copies. The rest of the file is a doc map plus the conventions that aren't written down anywhere else (commit subjects, scratch notes, writing for the GitHub renderer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Install fix: co-located plugin sources, required owner, validator guards Three install-surface bugs surfaced by @xuanguang-li while testing the benchmark plugin (#10): - marketplace.json omitted the required top-level `owner` object, so `/plugin marketplace add` failed schema validation for every user. - Both plugins declared a remote github source pointing back at this repo, forcing an install-time re-clone over SSH (ED25519 failure) to reach a subdirectory already present in the added marketplace copy. Switched to the documented co-located pattern — relative-path sources (`./qe`, `./benchmark`) — so install uses the local copy: no SSH, no auth prerequisite. - validate.py never checked `owner` and assumed an object source, so it passed a manifest the installer rejects. It now requires `owner.name`, resolves the relative-path source form, and hard-fails any plugin whose source points back at this repo. Also documents the version-gated `/plugin:skill` slash form (v2.1.216+) and the natural-language fallback in docs/using-skills.md. Marketplace version 0.1.0 -> 0.1.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Validator: report manifest errors instead of crashing on them The resolve_source refactor removed the `path` local from check_plugin, but three error strings still referenced it — so the name-mismatch, no-skills-dir, and empty-skills branches raised NameError instead of printing the diagnostic the validator exists to print. CI stayed green only because a healthy tree never enters those branches (review A1). Hoist the repo-relative path once after resolve_source returns and reuse it everywhere. Both reachable branches verified by hand: deleting benchmark/skills/ and emptying it now yield the intended one-line diagnostics with exit 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Line endings: normalize the two CRLF files, add .gitattributes EVALUATION_FRAMEWORK.md and ge_arrow_REPORT.md were CRLF in an otherwise-LF repo, so any future edit of either would render as a whole-file diff burying the real change (review E3). Pure normalization — zero content change, verified with `git diff --ignore-cr-at-eol`. The .gitattributes makes the normalization structural instead of a contributor convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Scoring: validate evidence before scoring; honest sensitivity stamp Three guards from the PR #5 review, all closing the same failure class — a contract that was documented but not enforced: - validate_evidence() (review B3, B4): score.py now refuses evidence that omits a scored input the verdict gates read, or marks a structural criterion met without a citation. Both new v2 fields failed open: a missing baseline_as_used_seconds silently disarmed the no-conversion verdict, and stripping every citation left the score unchanged. as_used_runs must now be present; [] remains legal as an explicit single-run declaration. Both evidence files gain the key (score-neutral: same single-run path). The check is a separate pass over authored evidence, never inside a scorer, so score_all stays a pure function of evidence and the perturbation search never silently drops mutants that trip authoring checks. - Honest perturbation count (review E5): tested increments only after a successful scoring call, and perturbations that raise are recorded in perturbations_skipped with the exception instead of being silently counted in the denominator the stamp is judged on. - robust-at-floor (review C1, floor half): a verdict already in the bottom band cannot be perturbed downward, so zero deciding flips there is partly the band's geometry, not evidence strength. markov_asset now stamps robust-at-floor with the reason attached; SKILL.md carries the stamp verbatim into reports, so plain "robust" was asserting support the run never demonstrated. Scorecards regenerated: totals, verdicts, and gates unchanged; the diff is the new fields plus markov_asset's stamp wording. The rubric's floor comment also stops restating the measured baselines (review E2) and points at evidence.json as the value the gate reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Rubric: matches_under_x64 caps correctness on its own (review C3) The x64-divergence cap required max_delta_shipped > 1e-8 as well, in both score_correctness and the derived logic&design bug cap. That conjunct made the guard structurally unable to fire when the shipped float32 delta is small — which is exactly the "wrong economics masked by low precision" case the framework's correctness section names as the thing this dimension guards. A candidate with divergent logic and a lucky 1e-12 shipped delta scored correctness 5 and total 3.25; it now scores correctness 1, logic_design capped at 3, total 2.30 gated to net regression. The flag's semantics come from the repo's own usage: the worked examples record TRUE for x64-noise residuals (~1e-14 to ~1e-11), so FALSE asserts the economics genuinely differ — not a failure of bitwise identity. Under that reading, agreement as shipped is luck, not correctness, and the cap needs no second condition. No committed scorecard changes: both worked examples have max_delta_shipped > 1e-8, so they were already caught by the old conjunct — verified byte-identical regeneration under both engines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fixtures: synthetic evidence pinning the rubric v2 paths (review B1) Neither worked example exercises the v2 headline features: both take the single-run efficiency fallback (no as_used_runs), and markov_asset hand-sets the correctness-bug flag, so the derived-cap path never runs on committed data. The newest scoring behaviour was tested only by hand-probing. references/fixtures/rubric_v2 is a synthetic evidence file — not an evaluation — whose one job is to make five untested paths execute on every scoring run: the unconditional x64 cap, the derived bug cap firing against a hand-set FALSE, the as_used_runs median, the contested-band annotation (runs straddle the 1.3x edge), and a verdict gate reporting the ungated total. The baseline sits above the 1 s floor on purpose so no-conversion does not mask the paths under test; every source string starts SYNTHETIC: so nobody cites the numbers as evidence about a lecture. Kept out of references/examples/ because an example records what was measured about a real PR and a fixture is a test input — conflating them invites citing synthetic data. Its own sensitivity line documents the stake: flipping matches_under_x64 alone swings the outcome from gated net regression to 4.70 clear improvement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * CI: scorecards must reproduce from evidence (review B2) The system's central claim — no score is ever written by hand — was verified only in reviewers' terminals. CI now regenerates all three committed scorecards (both worked examples and the v2 fixture) and fails on any diff, which catches a hand-edited scorecard, an unintended scoring change, and — the important case — an intended scoring change whose baselines were not regenerated. There the failure is desirable: the fix is to re-run and commit, which forces the verdict-moving diff into the PR where a reviewer sees it. Stdlib only, so no install step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Docs: single-source the baselines; state the new rubric behaviour Review E2/E4 plus the doc side of the engine changes: - The 0.028 s / 0.087 s vs 0.035 s / 0.18 s discrepancy was two honest measurements of the same quantity presented as one. The README table now labels its column triage-time (2026-07-21) and says the gate reads each lecture's own baseline_as_used_seconds; the framework and SKILL.md stop restating the numbers and point at evidence.json (review E2). - README quotes markov_asset's verdict as the scorecard emits it — no-conversion with the banded quality alongside — instead of the stale "2.25 net regression" (review E4). - Framework Sec. 1 states the unconditional x64 cap with its rationale, the three stamp values including robust-at-floor, and records the measured-vs-adjudicated conflation in the perturbation walk as a known limit for v3: read the deciding-flip list, not the stamp alone. - SKILL.md step 5 forbids reporting robust-at-floor as robust; sanity anchors and the tutorial's quoted output line updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Reviews: the PR #5 review of record, behind the filed item numbers The 2026-07-25 external review of this PR, committed verbatim alongside the repo's other review records so the item numbers cited on #7 and #4 (C2, C4, C5, D, E1, E6, E7) resolve without the PR comment. A labeled disposition note maps items to what was applied (4fffbc9..8388e86) and where the remainder is tracked; the text is otherwise as received. Every checkable claim in it was reproduced against the branch before the fixes landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Docs: loosen the skill-naming rule rather than replace it "Verb-first skill names" reads as a rule about skills, but the thing it was protecting is that the invocation scans as an imperative — and the whole `/plugin:skill` string is what a user types. `/audit:issues` satisfies that with the verb in the plugin, yet fails the rule as written, so the rule had to move. It moves by getting weaker, not by growing a taxonomy. Both shapes are shown, neither is preferred, and the absence of a preference is stated so a later contributor reads it as undecided rather than unspecified. Ranking them now would mean generalising from three plugins and no usage; the question is parked in FUTURE-IDEAS.md with the specific thing worth watching — whether mixing shapes inside one plugin actually causes trouble or merely looks untidy. Depends on #11 for its example: `audit` must land before this does. * Docs: catalog what shipped, and hold the conventions loosely Two changes, one idea: this repo is three plugins old, and its docs were describing more certainty than it has. CATALOG.md becomes a list of what is merged and runnable, with each plugin's state stated honestly (qe is scaffolding and says so) and a tracking issue beside it. It was previously the active plan, which meant it described skills nobody could run and drifted from the repo every time work moved. Plans now live in the issues — #3, #4, and #12 for the audit plugin — where they can change without anyone mistaking them for a description of what exists. README.md drops its duplicate plugin table and points here; the qe sub-skills point at #3 rather than at a catalogue entry that no longer carries their plan. The conventions are reframed as guidance. developing-skills.md now says so at the top, says only SKILL.md is required — a skill with nothing mechanical to run should be one file, not a directory tree — and marks the three conventions that are genuinely load-bearing, each with its reason. The rest is what one or two examples happened to need, and a contributor with a reason to depart should depart and say so in the PR, because a second example is how any of this becomes a real convention. The test that separates the two: a rule earns firmness when it keeps a skill's output checkable by someone who will not re-run it. Report-first, cited claims, and the plugin-root constraint pass it. Report shapes, phase divisions and naming forms do not. * Docs: fold the audit plugin into the rebased docs Post-#11 reconciliation. README gains audit in the plugin table and the documentation index, and its layout note drops "bundle contract", which the plugin no longer has. The marketplace entry's move to the co-located `./audit` form is not here — it belongs in "Install fix: co-located plugin sources", which is the commit that established that form, and the rebase carried it there. --------- Co-authored-by: Xuanguang Li <xuanguang-li@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5 (retargets to main when it merges) — documentation for the evaluation skill and the repo in general, written for @xuanguang-li to review and test alongside the system itself.
What's here
benchmark/README.md — the plugin's user guide: the two modes (review a conversion PR / triage whether a lecture is worth converting), what a session and its report look like, the manual pipeline quickstart, and a map of the plugin. Deliberately pointer-based: weights/anchors/bands stay canonical in EVALUATION_FRAMEWORK.md.
Triage mode — documented only after being tested. Blind triage using baseline-side data alone (fresh measurements this machine: ge_arrow total 0.028s, markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s) reproduces all three known verdicts, and the scope limit is part of the doc: triage cannot predict conversion-quality defects like markov_asset's build bug. The decision rule falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted while readability (0.25) losing two bands costs −0.50, so a conversion that costs meaningful readability cannot break even on speed alone. Also added as a section in SKILL.md.
docs/using-skills.md — for authors/reviewers: the three setup paths (settings.json opt-in, manual install, CI), invocation forms, the report-first/never-silently-edit contract, troubleshooting.
docs/developing-skills.md — for contributors: layout, conventions (verb-first names, deterministic-before-LLM, no cross-doc duplication, self-contained plugins), the dev loop (
claude --plugin-dir,validate.pywith negative testing), versioning, and the squash-merge/stacked-branch + external-author attribution patterns from this PR series.All relative links verified resolving;
validate.pygreen.@xuanguang-li — the useful test here: follow benchmark/README.md cold (both the manual pipeline and the triage recipe) and note anywhere the docs and your intent diverge.
🤖 Generated with Claude Code