You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Regime-robustness block for the distributional backtest #88
Diane Schanzenbach's 8/27 feedback (email thread "sampling error"): the QC system made several discrete shifts across our FY2012–24 panel, which is why Bauer–Schanzenbach stayed post-pandemic. Answer with numbers, not argument.
Extend analysis/persistence_backtest.py (merged in #87) with a regime_robustness block:
Regime-crossing test: fit variance components on FY2012–19 only; score FY2022–24 targets with the same CRPS/pinball/coverage battery. If the persistence widths transfer across the pandemic break, the "shifts did not break prediction" claim gets its strongest form.
Post-pandemic-targets summary: the existing per-cell rows already cover targets 2022–24 — add a filtered summary table alongside the full-panel one.
Memo + locks per the existing conventions (generated-memo equality, live input hashes, exact regeneration; see tests/test_persistence_backtest.py).
Context: the answers went to Diane in the held 8/27 reply ("models fit only on earlier years still score well on 2022-2024, and the persistence model wins 2024 outright") — this issue makes that claim a committed artifact. Existing per-year CRPS in persistence_backtest_results.json already shows 2022/2023/2024 results from expanding windows; the new piece is the frozen-2012–19 fit.
Gate: sol review before merge (both prior rounds' reports under ~/.cache/axiom-oracles/snap-fy27/backtest-review/).
Activity