The data (community wave 1)
Strict (--lossless) streaming on real 16GB machines measures 1.5–3.4 tok/s (#41 #55 #57 #61), vs the dev-machine approximation's 23–26. The one 18GB machine (#59) measures 9.8 — 4x better with just +2GB RAM. bolt on the same machines is fine (20–59 tok/s). The gap is systematic, not noise.
What we reproduced on the dev machine (M1 Max 64GB, forced C=128)
| condition |
strict code-256 |
| no pressure (the old approximation regime) |
29.3 tok/s |
| + 36GiB incompressible RAM ballast (≈12GB effective free, sweeping to stay resident) |
14.7 |
| same ballast, C=64 |
13.7 (≈ parity) |
So real RAM pressure explains ~2x — and is now locally reproducible — but not the remaining ~5x.
Hypothesis for the rest: Metal working-set overflow
The C=128 tier's wired footprint is ~11.4GB (arena ~9GB + resident dense weights; RSS-verified). A 16GB Mac's Metal recommendedMaxWorkingSetSize is ≈ 10.9GB — the working set exceeds the GPU budget, so the driver re-pages buffers per command buffer. strict touches a changing expert set every token (worst case for residency); bolt's frozen residency is stable (why it survives). The 18GB machine (limit ≈ 12.3GB) fits, hence 9.8 tok/s. A 64GB dev machine (limit ≈ 48GB) cannot reproduce this mechanism, which is exactly why the approximation missed it.
The probe — 16GB owners, this is a 5-minute contribution
No new binary needed. On v0.3.3+:
QWISP_DEVICE_RAM=8 qwisp benchtest
This forces C=64 (wired ≈ 6.9GB — inside the GPU budget). If the strict row jumps from ~2 to ≳10 tok/s, the hypothesis is confirmed and the fix (right-sizing the 16GB tier) ships. If it stays ~2, we look at raw NAND random-read latency next. Post the report via the one-click URL as usual (the env table records the forced tier).
Caveat: at C=64 the bolt rows may be less stable (tighter residency; a LOOPY long-600 is more likely) — expected, and useful data for #47. The strict row is the datum here.
Estimation-accuracy follow-ups (separate PRs)
scripts/ballast.py (the incompressible-RAM simulator) checked in as the official small-RAM approximation recipe.qwisp benchtest env table gains startup free-memory + Metal working-set limit, so every future report is pressure-labeled.
The data (community wave 1)
Strict (
--lossless) streaming on real 16GB machines measures 1.5–3.4 tok/s (#41 #55 #57 #61), vs the dev-machine approximation's 23–26. The one 18GB machine (#59) measures 9.8 — 4x better with just +2GB RAM. bolt on the same machines is fine (20–59 tok/s). The gap is systematic, not noise.What we reproduced on the dev machine (M1 Max 64GB, forced C=128)
So real RAM pressure explains ~2x — and is now locally reproducible — but not the remaining ~5x.
Hypothesis for the rest: Metal working-set overflow
The C=128 tier's wired footprint is ~11.4GB (arena ~9GB + resident dense weights; RSS-verified). A 16GB Mac's Metal
recommendedMaxWorkingSetSizeis ≈ 10.9GB — the working set exceeds the GPU budget, so the driver re-pages buffers per command buffer. strict touches a changing expert set every token (worst case for residency); bolt's frozen residency is stable (why it survives). The 18GB machine (limit ≈ 12.3GB) fits, hence 9.8 tok/s. A 64GB dev machine (limit ≈ 48GB) cannot reproduce this mechanism, which is exactly why the approximation missed it.The probe — 16GB owners, this is a 5-minute contribution
No new binary needed. On v0.3.3+:
This forces C=64 (wired ≈ 6.9GB — inside the GPU budget). If the strict row jumps from ~2 to ≳10 tok/s, the hypothesis is confirmed and the fix (right-sizing the 16GB tier) ships. If it stays ~2, we look at raw NAND random-read latency next. Post the report via the one-click URL as usual (the env table records the forced tier).
Caveat: at C=64 the bolt rows may be less stable (tighter residency; a LOOPY long-600 is more likely) — expected, and useful data for #47. The strict row is the datum here.
Estimation-accuracy follow-ups (separate PRs)
scripts/ballast.py(the incompressible-RAM simulator) checked in as the official small-RAM approximation recipe.qwisp benchtestenv table gains startup free-memory + Metal working-set limit, so every future report is pressure-labeled.