Skip to content

Patch the newest models too, so their equivalence tests stop skipping - #26

Merged
AndreSlavescu merged 1 commit into
mainfrom
qwen3-model-coverage
Jul 30, 2026
Merged

Patch the newest models too, so their equivalence tests stop skipping#26
AndreSlavescu merged 1 commit into
mainfrom
qwen3-model-coverage

Conversation

@AndreSlavescu

Copy link
Copy Markdown
Owner

Sixteen cases in the model matrix were skipping, not passing — and a skip reads like a pass in a
summary. Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-27B and Qwen3-VL-4B reported "patches nothing", so their
logits agreed with MLX for the uninteresting reason that meTile was not running.

No kernel was missing — a name check was

The allowlist was keyed on a set of modules plus a hardcoded class name of "MLP". Every
architecture involved computes the same gated MLP — down_proj(swiglu(gate_proj(x), up_proj(x))),
where swiglu(gate, x) is nn.silu(gate) * x, which is exactly what _execute_quantized_mlp
computes — but they don't all call the class MLP:

model MLP class block class
Qwen3.5 / 3.6 qwen3_next.Qwen3NextMLP qwen3_5.DecoderLayer
Qwen3-VL qwen3.MLP qwen3.TransformerBlock

Membership is now explicit (module, class) pairs in two registries, because a claim about an
implementation shouldn't be spelled as a claim about a name.

Graph fusion keeps a stricter registry: its replacement reproduces a specific residual structure.
Qwen3.5's DecoderLayer is included only because _attention_module now resolves the attention by
attribute — that layer is hybrid, with a GatedDeltaNet bound to linear_attn on every layer
that isn't a multiple of full_attention_interval and no self_attn at all, so a replacement
calling self.self_attn would have crashed. Resolving per call rather than per class is the whole
adaptation; a block binding neither name falls back instead of guessing.

Skips: 16 → 0. All ten newly active cases pass bit-exact on the first run — both token equality
and decode logits.

Lighting them up surfaced a flake the skips had been hiding

In a test this change doesn't touch: Qwen3-VL attention failed bit-exactness in one of five full
matrix runs, while passing in isolation and passing with its own model's cases run alone. Needing
the whole matrix is the signature of memory pressure — and this file never released a model.
Eighty-two tests each load a checkpoint, up to 15 GB, and letting the Python reference go does not
return device memory, because MLX caches freed buffers. A fixture now drops both after every test.

Six full matrix runs since are clean, and consistently faster — 227–235s against 294–304s
which is independent support for the diagnosis rather than for the fix merely hiding it.

Not proof. Five clean runs against a one-in-five rate is likely but not decisive. The mechanism,
the timing change, and a recorded precedent in this project all point the same way.

82 pass with no skips; 655 in the fast suite.

🤖 Generated with Claude Code

Sixteen cases in the model matrix were skipping, not passing, and a skip reads like a pass in a
summary. Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-27B and Qwen3-VL-4B reported "patches nothing", so their
logits agreed with MLX for the uninteresting reason that meTile was not running.

The cause was an allowlist keyed on a set of modules plus a hardcoded class name of "MLP". Every
architecture involved computes the same gated MLP -- `down_proj(swiglu(gate_proj(x), up_proj(x)))`
where `swiglu(gate, x)` is `nn.silu(gate) * x`, which is what `_execute_quantized_mlp` computes --
but they do not all call the class MLP. Qwen3.5 and Qwen3.6 reach it as Qwen3NextMLP from
mlx_lm.models.qwen3_next, and Qwen3-VL's is in mlx_lm.models.qwen3, which was simply not listed. No
kernel was missing; a name check was excluding three of the newest models and a VLM.

Membership is now explicit (module, class) pairs in two registries, because a claim about an
implementation should not be spelled as a claim about a name. Graph fusion keeps a stricter one: its
replacement reproduces a specific residual structure, and Qwen3.5's DecoderLayer is included only
because `_attention_module` now resolves the attention by attribute. That layer is a hybrid -- on
every layer that is not a multiple of full_attention_interval the attention is a GatedDeltaNet bound
to `linear_attn` and `self_attn` does not exist at all -- so a replacement calling `self.self_attn`
would have crashed. Resolving per call rather than per class is the whole adaptation; a block
binding neither name falls back instead of guessing.

Skips went from 16 to 0, and all ten newly active cases pass bit-exact on the first run, both token
equality and decode logits.

Lighting them up also surfaced a flake that the skips had been hiding, in a test my change does not
touch: Qwen3-VL attention failed bit-exactness in one of five full matrix runs, while passing in
isolation and passing with its own model's cases run alone. It needed the whole matrix, which is the
signature of memory pressure, and this file never released a model -- eighty-two tests each load a
checkpoint, up to fifteen gigabytes, and letting the Python reference go does not return device
memory because MLX caches freed buffers. A fixture now drops both after every test.

Six full matrix runs since are clean, and they are also consistently faster, 227 to 235 seconds
against 294 to 304 before, which is independent support for the diagnosis rather than for the fix
merely hiding it. Not proof: five clean runs against a one-in-five rate is likely but not decisive.
The mechanism, the timing change and the precedent in this project all point the same way.

82 pass with no skips; 655 in the fast suite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AndreSlavescu
AndreSlavescu merged commit db6cb62 into main Jul 30, 2026
4 checks passed
@AndreSlavescu
AndreSlavescu deleted the qwen3-model-coverage branch July 30, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant