Measure the int8 multi-row headroom, and keep the four-bit guard that was protecting us

[AndreSlavescu]

The int4 multi-row win reproduces and is stronger than published — 1.45x, 1.73x, 1.54x at rows 8,
16, 32 through the prepared executor. The int8 half of this task does not exist, and finding that
out was more instructive than the number.

Why int4 wins is bandwidth, not batching

MLX re-reads weights per row tile. benchmarks/quantized_row_headroom.py measures the consequence per
width, against a measured 120.6 GB/s streaming ceiling:

rows	int4 GB/s	int8 GB/s	int4 gap	int8 gap
1	62.0	86.0	1.94x	1.40x
8	41.8	65.7	2.88x	1.84x
16	35.6	62.3	3.39x	1.94x
32	37.5	65.3	3.21x	1.85x

The gap columns bound what any kernel could win. int4 leaves 3.4x at sixteen rows and our kernel
captures about half. Effective bandwidth rather than time, because eight-bit weights are twice the
bytes and a ratio of times can't tell a slower kernel from a worse use of the memory system.

I tried the int8 kernel, and the speedup was the bug

Half of the int8 gap would be ~1.4x, worth wanting. The repacking generalises to eight bits in a few
lines, the weights load, and the kernel measured 1.6x to 2.0x faster than MLX.

It was also wrong, at relative error 2.5 to 2.9 — and the speed and the wrongness are the same
fact. lower_affine_matmul emits NAX affine fragments with block_size=4 and takes no bit width, so
it decoded eight nibbles per word where the data held four bytes, read half the values, and finished
early. A large speedup arriving together with a wrong answer is one bug, not one win and one bug.

What I got wrong, and what it cost

The four-bit check in MLXAffineWeight.from_mlx is load-bearing, not a formatting preference, and
I loosened it without checking that anything downstream honoured the parameter — the bit width was
threaded into cache keys but never into the lowering. It's restored, its message now says why, and a
test pins it, because relaxing it looks harmless and the failure is silent.

Only the tuner's agreement gate stopped the fast wrong kernel being selected. That's the gate doing
its job, not a reason to rely on it.

The generalised repacking is reverted too. It was correct, but a correct-looking eight-bit path sitting
beside a four-bit-only lowering is precisely the trap that just caught me.

Supporting eight bits means teaching the lowering a bit width — real compiler work against a 1.4x
prize, not a check to relax.

680 pass, 7 skipped.

🤖 Generated with Claude Code