Label every published result with the precision it was measured at

[AndreSlavescu]

Three artifacts reported end-to-end model speedups with no statement of the precision behind them,
and they were the most quotable numbers in the repository. m5-mlx-lm-bf16-models.json shows decode
speedups of 1.37x to 1.75x — and those runs had meTile quantizing the down projection to affine8
while MLX ran bf16. A representation change, not a kernel win, with nothing in the file saying so.

I quoted from that file earlier in exactly the way it invites.

The writer was never the problem

It refuses to emit an unlabelled result at schema 19, _validate_suite even rejects a label that
disagrees with the recorded plan, and the chart renderer has been printing "mixed precision; not
BF16-vs-BF16" all along. What none of them could do is police artifacts written at schemas 2, 3 and 5,
before labels existed — which is precisely where the unlabelled ones were. Validation at write time
cannot reach a file already on disk.

So the audit now covers the directory rather than the writer, and age is no longer an exemption.

Two requirements, one borrowed from MLPerf

Every published result must carry a precision class and say whether both sides ran the same weight
representation. A mixed-precision result must additionally carry an accuracy metric, following
MLPerf's rule for quantized submissions: arbitrary reproducible quantization is allowed, but it must be
described and must meet an accuracy target. A speedup at a different precision with no accuracy
evidence beside it is not a result anyone can act on. The calibration fidelity justifying the affine8
decode was already recorded — it simply wasn't connected to the claim.

Labels are derived, not re-measured

The label is a function of which features ran, and every file records that in selected_plan. The
backfill calls the same _precision_comparison the writer uses rather than reimplementing its rules —
a second implementation would be free to drift, and the audit checking labels against plans would then
be checking a copy. Each is marked derived_from: selected_plan so nobody mistakes it for something the
original run emitted. Baseline dtype stays as the function reports it: those schemas recorded no model
dtype, and inferring one from the filename would be a guess dressed as data.

What the labels say — the point of the exercise

artifact	class
m5-mlx-lm-bf16-models.json	mixed_precision_affine_int8_decode ← the 1.37x–1.75x figures
m5-mlx-lm-models.json	same_precision
m5-mlx-lm-bf16-dense-qwen15.json	same_precision

So the 4-bit end-to-end numbers are like-for-like, and decode there is parity. The bf16 figures are
not, and should not be quoted as a speedup.

4 audit tests over every published file. 679 pass with 7 correctly skipping the mixed-precision accuracy
check as same-representation comparisons.

Sources: MLPerf Inference Rules

🤖 Generated with Claude Code