Spec §12 makes this a precondition, not a postmortem: "算力账之外必须算统计账 … 用 bootstrap CI 宽度随 n 的收敛关系倒推所需 case 数；预算内达不到功效的对比在设计期就降级为探索性结论". This file is that calculation, run against the real v0.1.0 data rather than assumed.
Reproduce: python scripts/power_check.py
Paired bootstrap over the case set (5,000 resamples, the same estimator report.damage_records uses), qwen3-1.7b, content_correct:
| chain | comparison | baseline | quantized | damage | 95% CI | verdict |
|---|---|---|---|---|---|---|
| gguf | f16 → Q4_K_M | 0.833 | 0.750 | +0.083 | [+0.000, +0.208] |
not credible — touches 0 |
| mlx | bf16 → 4bit | 0.833 | 0.833 | +0.000 | [+0.000, +0.000] |
no damage detected |
(lenient tier. The strict tier gives the identical CI: it is the same two cases that flip. Strict scores every function_call case 0 by construction — that gap is the parser-gap signal, not a scoring bug — so 8 of the 24 cases are constant there and carry no information about damage either way.)
Only 2 of 24 cases discriminate f16 from Q4_K_M. The width of the CI is driven by that discordant count, not by the accuracy level — which is why a headline like "Q4 costs you 8 points" was not sayable at n=24, and was correctly reported as inconclusive.
Simulated: paired differences drawn at the stated discordance rates, 5,000-resample bootstrap per trial, 400 trials per cell, requiring the 95% CI lower bound to clear 0 in ≥90% of trials.
| true (net) damage | reverse flips (quant beats baseline) | cases needed |
|---|---|---|
| 0.05 | 0% | 140 |
| 0.05 | 10% | >400 |
| 0.10 | 0% | 68 |
| 0.10 | 10% | 304 |
| 0.15 | 0% | 48 |
| 0.15 | 10% | 152 |
Reverse flips do not cancel out. They leave the mean damage where it was and widen the CI, so a 10% reverse-flip rate costs 4–5× the sample size — which is why the column exists, and why the required n must be recomputed against the measured flip rate rather than read off the optimistic column.
Spec §12 anticipates effect sizes of 0.05–0.10. So the pack needs to be on the order of 140 cases, and the measured f16→Q4_K_M damage (+0.083) sits exactly in the band that 24 cases cannot resolve.
Reverse flips are the dominant unknown. At Q4 there were none (2 discordant, both baseline-right). At Q2_K/Q3_K_M — the tiers experiments 2 and 3 are actually about — noise cuts both ways and the required n climbs steeply. Recompute this table against the measured flip rate once the full ladder has run; do not assume the 0% column.
- The pack grows to 150 single-turn cases (v0.6.0, 18 → 150). Not "more is better" — 150 is what a 0.05–0.10 effect costs.
- Damage at Q5_K_M/Q8_0 will likely stay inconclusive, and that is a result, not a failure: the honest statement is a bound ("no damage detectable above ±0.07"), which is what a deployer asking "is Q8 safe?" actually needs.
- dev/test halves have ~75 cases each, enough for ~0.10 but not 0.05. So the split is applied where spec §6 requires it —
prescribeselects on dev,confirmreports on test — and not to the Section A damage tables, which select nothing and therefore have nothing to overfit. A damage number is a measurement; a prescription is a choice. Only the choice needs holding out. - Every run that predates the v0.6.0 pack is not comparable to one after it: the pack's sha256 is in the report header for exactly this reason, and
scripts/rescore_report.pyrefuses to rescore across pack versions.