kv: 8-bit KV cache as a bolt-tier long-context decode lever (Step 0 — measure the ceiling before any kernel)

[penta2himajin]

Observation

KV lives on the 10 full-attention layers only — the 30 linear_attention (GDN) layers hold a fixed conv + recurrent state that does not grow with context (SeedlessEngine.freshCaches(), ~62MB total). Per full-attn layer the cache is [2, ctx, 256] float16 for each of K and V, so:

2 (K,V) × 2 kv-heads × 256 head_dim × 2 B × 10 layers = 20 KB / context token

ctx	KV (f16)	8-bit saves
8K	160 MB	80 MB
32K	640 MB	320 MB
64K	1.25 GB	0.6 GB
262,144 (model max)	5.0 GB	2.5 GB

This is not a memory issue, and filing it as one would be a mistake. At the context lengths a 16GB machine actually runs, 8-bit KV saves 80–320MB, while #165 describes a 4.5GB expert arena that is never released on the same machine. Two orders of magnitude apart.

It is a long-context decode issue. Measured 2026-08-01 (resident tier, AC, jacquard measure, same binary): decode 96.4 tok/s at 52-token context -> 36.1 ± 1.2 tok/s at 35,890. At that context each decoded token reads 718 MB of KV against ~566 MB of expert weights — KV is the majority of per-token traffic. Halving it is the largest single lever on the curve that #122's speculation gate exists to manage (48K: 18.6 -> 32 tok/s post-gate).

Constraints that shape this before any design

• It can never be in strict. Lossy KV is L3 by this project's own taxonomy (near-lossless: relaxed acceptance, margin gate, lossy KV, expert skip). Strict L1 is bit-exact reproduction of the quantised greedy stream; a requantised KV changes attention outputs. This is a bolt-tier / opt-in feature or it is nothing.
• It lands on the frozen forward path. The KV read is sdpa_rows, named explicitly in the do-not-touch set. Any implementation must be an additive, flag-off kernel variant beside it, never an edit to it.
• It touches the prefix cache format. PrefixPersist snapshots attention KV; a quantised cache changes what is written and restored, including artifacts already on disk.
• SuffixSpec verify reads the same KV, so feat(spec): mechanical speculation gate — fix long-context decode collapse (#119) #122's gate economics move with this — the win must be measured with the gate in its shipped state, not with spec disabled.

Step 0 — measure the ceiling, write no kernel

1. Decode tok/s curve at ctx in {4K, 16K, 32K, 64K}, resident tier, jacquard measure, arms paired.
2. For each point compute the KV share of per-step bytes from the shapes above and the measured step time.
3. The 8-bit ceiling is then 1 / (1 - kv_share/2) — an upper bound that assumes dequant is free, which it is not.

Kill criteria, fixed before looking: if the projected ceiling at 32K is under 10%, close not-planned — 32K is already a context where KV is the majority of traffic, so a lever that cannot clear 10% there will not clear it anywhere users are. If it clears, the next step is a dequant-cost estimate in the attention kernel, not an implementation.

Quality, if it ever ships

Measured on bolt's existing teacher-forced per-token fidelity harness. Free-run divergence is greedy chaos and is not a fidelity number — do not quote it.

Refs

#122 (long-context spec gate — the curve this attacks) · #165 (the multi-GB memory lever on the same machines; do not confuse the two) · SeedlessEngine.freshCaches() · sdpa_rows (frozen) · PrefixPersist

Origin: community request (Reddit) asking for KV quantisation to fit an M2 MacBook Pro — the sizing above is why that is the wrong lever for that machine, and the right one for long context.