TESTING: benchmark triage run on wald_friedman — five plugin gaps, and a floor that measures the wrong quantity

[mmcky]

First end-to-end exercise of /benchmark:review-acceleration in triage mode — the prospective, no-candidate path — against a lecture nobody had evaluated before. It produced a clear verdict, and it surfaced five gaps in the plugin. Opening this to discuss the gaps; the lecture-side defect it turned up is filed separately. See QuantEcon/lecture-python.myst#1008.

What was run

wald_friedman, one of the four lectures tagged Benchmark in the current rng-triage/JAX-candidacy review (Table C: navy_captain, wald_friedman, wald_friedman_2, likelihood_ratio_process_2). None of the four has a PR open, so there is no candidate implementation and the full baseline-vs-candidate review path does not apply — this is the triage subset described in SKILL.md § "Triage mode (no candidate yet)".

Run as an installed plugin would run: CLAUDE_PLUGIN_ROOT pointing at a read-only benchmark/, workspace under benchmark-eval/wald_friedman/ inside the lecture checkout, quantecon conda env (Python 3.13.9, numpy 2.3.5, jax 0.10.1, jaxlib 0.10.1, numba 0.62.1, macOS arm64, 10 cores, CPU-only).

Result: no-conversion. Baseline as-used total 5.586 s (median of 3 fresh processes, 5.28–5.75), but only ~4% of that is addressable by a vmap-style rewrite — 47% is scipy.integrate.quad inside js_dist, 25% is numba compilation, 21% is pure-Python driver loops. Amdahl ceiling on the whole conversion is 1.05×. Separately, vmap over lax.while_loop steps every lane until the last one stops, and the measured max/mean stopping-time ratio is 3.7×–7.4×, so the conversion would make the only part it can touch do several times more work. Full report and results JSON are in the workspace; happy to attach or push them wherever is useful.

Finding 1 — no timeout on measurement subprocesses

references/examples/*/scripts/run_all.py calls subprocess.run(argv, cwd=HERE, check=False) with no timeout=. Lecture code with data-dependent loops can fail to terminate, and when it does the pipeline hangs with no output and no error — indistinguishable from slow progress. This happened twice here, ~10 minutes each, saturating all ten cores, before the cause was found. A timeout= plus a steps_failed entry would convert a hang into recorded provenance, which is what the stamp already exists to do. This is the highest-value fix of the five: it is a few lines, and the failure mode it prevents is total.

Finding 2 — triage mode has no scaffold

SKILL.md documents triage mode in prose, including the instruction to "adapt just the baseline half of an as_used_total.py template". The plugin ships nothing for it. grep -rn triage over scripts/ and references/ matches only three comment lines in rubric.py; both worked examples are review-mode (baseline plus candidate); EVIDENCE_TEMPLATE.json and score.py assume a candidate exists, so score.py cannot be run on a triage evaluation at all.

Everything this run needed had to be invented: a cell extractor, a build_model_old.py that slices definition-bearing regions out of the lecture, a verbatim-fidelity gate, and a triage driver. That is maybe 250 lines of scaffolding standing between "I want to know whether to convert this lecture" and any answer.

Worth weighing against demand: Table C alone lists eight JAX candidates with no PR open, versus seven open JAX conversion PRs. Triage is the more common entry point and it is the unsupported one. A minimal triage bundle — baseline-only as_used_total.py, a triage results schema, and a short report template — would likely be the single highest-leverage addition to the plugin.

Finding 3 — env_stamp.py does not record numba

scripts/scoring/env_stamp.py stamps numpy, jax, jaxlib, quantecon. Every baseline in this plugin's remit is NumPy+Numba, and numba's version and threading layer materially move the measured numbers — a run on a different numba, or with NUMBA_NUM_THREADS set differently, is not comparable, and results/env.json would not show it. One-line fix; adding numba and scipy to the package tuple covers it.

Finding 4 — the 1 s materiality floor measures the wrong quantity

rubric.py sets NO_CONVERSION_BASELINE_S = 1.0 and the no-conversion gate reads baseline_as_used_seconds against it. wald_friedman measures 5.586 s and clears the floor by 5.6× — yet it is a textbook no-conversion, because 96% of that time is structurally unreachable by the conversion being contemplated.

The floor answers "is there time here?" when the decision needs "is there addressable time here?". Both blind-validated don't-convert anchors (ge_arrow at 0.035 s, and markov_asset) sit two orders of magnitude below the floor, so a lecture that clears the floor while having nothing to sell is a shape the calibration has not exercised. The comment in rubric.py is candid that the exact placement inside the gap is not load-bearing — this run suggests the quantity may not be either.

Two directions worth discussing. Either evaluate the floor against the addressable share rather than the total, which needs the decomposition to become a measured artifact; or leave the gate alone and have triage report the addressable fraction as a first-class number, on the grounds that the gate only fires in review mode where a candidate exists to be measured. I lean to the second as the cheaper change, but the first is what would have caught this lecture automatically.

Finding 5 — subprocess output is captured, not streamed

run_all.py uses capture_output=True, so a long-running probe emits nothing until it exits. Combined with finding 1, a hang looks exactly like slow progress. Tee-ing progress to a file under results/ is a cheap workaround and is what the probes in this run ended up doing.

What held up well

The prescribed procedure did real work and should not get lost among the gaps.

The instruction to diff the extracted code and the replayed call sequence against the lecture's cells before measuring caught drift on its very first run. Making that a hard gate — every code line in the extracted module and every replayed call must appear verbatim in the lecture source — cost little and is now the thing I would trust least without.

The insistence on as-used over warm timings is what decided this lecture. The headline "N=20,000 simulation" that made wald_friedman look like a candidate runs in 5.0 ms warm; its 0.789 s stage is ~0.76 s of numba compilation. A warm-only or a naive whole-cell measurement would have read that as 0.8 s of real simulation work and pointed the other way.

The fresh-process K-repeat with a median headline behaved well: 5.755 / 5.586 / 5.276 s, tight enough that none of the conclusions rest on run-to-run noise.

Suggested triage for these

Finding	Cost	Priority
1 — subprocess timeout	a few lines	high; prevents total failure
3 — numba in env_stamp	one line	high; cheap correctness
2 — triage scaffold	a small bundle	high value, medium cost
5 — stream progress	small	medium
4 — floor semantics	design discussion	needs a decision, not a patch

Related: #4 (benchmark plan), #7 (rubric v2), #10 (testing protocol).

[mmcky]

@xuanguang-li — would value your read on this, since it comes out of the first triage-mode run of the system you built, and two of the five findings are design calls rather than patches.

The two I'd most like your view on:

Finding 4, the materiality floor. You set NO_CONVERSION_BASELINE_S = 1.0 and the comment in rubric.py is explicit that the exact placement inside the ge_arrow-to-aiyagari gap is not load-bearing. This run suggests the quantity may be the load-bearing part rather than the threshold: wald_friedman measures 5.59 s and clears the floor by 5.6×, yet 96% of that time is scipy.quad, numba compilation, and pure-Python driver loops — none of it reachable by the conversion in question. The prange kernels the conversion would actually replace are ~0.24 s, so the Amdahl ceiling is 1.05×. Both your validated don't-convert anchors sit far below the floor, so a lecture that clears it while having nothing to sell is a shape the calibration hasn't seen. Do you want the gate to key on addressable time, which means the decomposition has to become a measured artifact? Or leave the gate alone on the grounds that it only fires in review mode where a candidate exists, and have triage report the addressable fraction as a first-class number? I lean to the second as much the cheaper change, but the first is what would have caught this lecture automatically.

Finding 2, the triage scaffold. Triage mode is documented in SKILL.md but ships nothing — no baseline-only template, no results schema, no report shape. Table C of the current review lists eight JAX candidates with no PR open against seven open conversion PRs, so triage looks like the more common entry point and it's the unsupported one. Is a minimal triage bundle something you'd want in the plugin, and if so does the shape this run improvised look right to you, or is there a different decomposition you had in mind when you wrote the triage section?

Findings 1, 3 and 5 are small and I'm happy to just do them unless you object — subprocess timeout= feeding steps_failed, adding numba and scipy to the env_stamp package tuple, and tee-ing probe progress to results/.

One thing worth saying plainly: the parts of the procedure that did the real work here are yours. The verbatim extraction-diff instruction caught drift on its first run, and the as-used-over-warm discipline is the whole ballgame on this lecture — the "N=20,000 simulation" that made it look like a candidate is 5.0 ms warm behind 760 ms of numba compilation. A warm-only or whole-cell measurement would have pointed the other way.

The lecture defect this run tripped over is separate. See QuantEcon/lecture-python.myst#1008.