[MaxGhenis]

Motivation

The two raw regeneration tests asserted regenerated == committed bytes on optimizer output. Optimizer floats move in the last ulp across BLAS implementations (the PR #57 lesson) and across reruns, so byte-equality failures were unreadable — indistinguishable from noise. A 2026-08-14 local failure of the Oregon test was attributed to exactly that noise class. It turned out to be masking a real regression (below).

What changed

• tests/conftest.py (new): shared assert_artifact_values_match fixture — a strict structural value lock. Dicts must have identical key lists (same keys, same order), lists identical lengths, non-float leaves must match exactly with exact types (int vs float vs bool enforced); floats tolerate max(1e-9 · |expected|, 1e-12). The explicit abs=1e-12 floor is load-bearing at the 91+294 exact-zero optimizer-clipped donor weights; every committed nonzero float exceeds 1e-3, so rel governs everywhere else.
• Both raw regeneration tests now parse regenerated and committed JSON and assert the value lock instead of byte equality. skipif gates unchanged (still local-cache-only).
• Serializer round-trip tests (both files, run in CI): serialize_results(json.loads(raw)) == raw — value-locking the raw tests dropped their incidental coverage of serializer formatting; the round-trip restores it cache-free, which the old local-only byte tests never gave CI.
• Planted-drift meta-tests (run in CI): a relative 1e-6 float mutation, a non-float change, a dropped key, an int retyped to float, a key reorder, and a 2e-12 lift of an exact-zero donor weight must each fail the lock; a one-ulp nudge of a nonzero leaf must pass. test_value_lock_tolerance_contract executably pins the tolerance function.
• Docstring rule extended: raw regeneration is value-locked, not byte-locked, with the verified rationale.
• Committed artifacts keep exact-byte pins: the Oregon SHA-256 test is untouched; the RI/KY contract test gains the SHA-256 pin it was missing (6b8b9270…).
• The regression the lock caught, fixed: build_results serialized the entire module-level CPI_U table into outcome_definitions.cpi_u_annual_average; Audit extension to FY2012: RI/KY pre-periods verified, threshold series complete #59 widened that table to 2012–2016 for RI/KY, silently changing Oregon's regeneration output after its artifact was committed. The emission is now scoped to the panel years, mirroring the RIKY emitter. Verified: the scoped dict equals the committed artifact's dict exactly; nothing else consumes the field.

Verification

• First full local tier (CI-exact command, both raw regenerations included): 1 failed, 276 passed in 33:27 — the one failure was the Oregon raw test flagging outcome_definitions.cpi_u_annual_average with the exact divergent path: the value lock catching the stale-CPI regression on its first run. The RI/KY raw regeneration (~30 min) passed under the value lock, demonstrating the noise-absorption side.
• A leaf-walk diagnostic confirmed the CPI key widening was the only divergence (every other leaf within tolerance), and a regenerated draw passes the lock against the committed artifact once the emission is scoped.
• Second full local tier at the fixed commit: 280 passed in 36:34 — fully green, both raw regeneration tests included.
• Adversarial review (4-agent workflow + cross-family sol review): planted-mutation verification confirmed live on artifact copies (1e-6 fails at the exact path on both artifacts, one-ulp passes, 1e-8 boundary fails); a comparator attack sweep (type confusions, orderings, ±ulp perturbation of every float in both artifacts) found no false-pass/false-fail beyond the abs-floor documentation gap, which is now explicit, documented, and executably pinned.
• CI green (meta-tests, round-trips, and SHA pins run on ubuntu; raw tests skip there by design).

🤖 Generated with Claude Code

[MaxGhenis]

Final merge-gate review (sol, gpt-5.6-sol via codex) — verdict on 972fa3d

APPROVE

• analysis/event_study.py:489 — severity note — build_results now emits cpi_u_annual_average from YEARS (2017–2024), not the module-wide CPI_U (2012–2024). A direct runtime comparison against analysis/event_study_results.json returned exact dict equality and identical key order: ['2017', ..., '2024']; no RI/KY CPI coverage was narrowed because build_riky_results still iterates RIKY_YEARS at line 651.
• tests/conftest.py:10 — severity note — The shared comparator implements the stated value lock: dicts require the same ordered key list, lists require the same type and length and recurse positionally, non-float leaves require identical concrete types and values, and float leaves alone use pytest.approx(expected, rel=1e-9, abs=1e-12), i.e. the documented max(1e-9 * |expected|, 1e-12) acceptance bound.
• tests/test_event_study.py:109 — severity note — Serializer-format coverage is restored independently of optimizer output: both Oregon’s round-trip here and RI/KY’s at tests/test_riky_event_study.py:164 assert serialize_results(json.loads(raw)) == raw. Direct execution returned true for both committed artifacts, whose SHA-256 digests also exactly match the byte pins at tests/test_event_study.py:103 and tests/test_riky_event_study.py:95.
• tests/test_event_study.py:122 — severity note — The absolute-floor rationale is both documented and executable: the contract test accepts 9e-13 and rejects 2e-12 against exact zero, while the planted mutation at line 164 locates a committed exact-zero donor weight and proves a 2e-12 lift fails. Artifact traversal found 91 Oregon and 294 RI/KY exact-zero float leaves; the smallest nonzero magnitudes were 0.009067733935758882 and 0.0011906325516472527, confirming the documented claim that every nonzero float exceeds 1e-3 and therefore uses the relative term.
• tests/test_riky_event_study.py:175 — severity note — The changed-file fast tier passed locally (13 passed, 2 raw tests deselected in 3.54s), including both serializer round trips and adversarial comparator tests; Ruff also reported All checks passed! for all four changed files.
• .github/workflows/ci.yml:11 — severity note — Live PR metadata identifies base 3048f6d and head 972fa3d, matching the local refs; GitHub reports mergeable=MERGEABLE, mergeStateStatus=CLEAN, and the required test check completed successfully. The reviewed diff is exactly three commits and four files (analysis/event_study.py, tests/conftest.py, tests/test_event_study.py, tests/test_riky_event_study.py), and git diff --check origin/main...HEAD is clean.
• tests/conftest.py:24 — severity note — An independent mutation matrix confirmed the comparator rejects reordered dict keys, shortened and reordered lists, bool/int and int/float substitutions, 2e-12 zero drift, and 2e-9 relative drift, while accepting 9e-13 zero noise and 5e-10 relative noise.
• analysis/event_study.py:489 — severity note — Repository-wide usage search found cpi_u_annual_average only in the two result emitters and their two committed JSON artifacts; no downstream code consumes the Oregon dict, so limiting it to Oregon’s panel years has no hidden call-site impact.