Patch the newest models too, so their equivalence tests stop skipping - #26
Merged
Conversation
Sixteen cases in the model matrix were skipping, not passing, and a skip reads like a pass in a summary. Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-27B and Qwen3-VL-4B reported "patches nothing", so their logits agreed with MLX for the uninteresting reason that meTile was not running. The cause was an allowlist keyed on a set of modules plus a hardcoded class name of "MLP". Every architecture involved computes the same gated MLP -- `down_proj(swiglu(gate_proj(x), up_proj(x)))` where `swiglu(gate, x)` is `nn.silu(gate) * x`, which is what `_execute_quantized_mlp` computes -- but they do not all call the class MLP. Qwen3.5 and Qwen3.6 reach it as Qwen3NextMLP from mlx_lm.models.qwen3_next, and Qwen3-VL's is in mlx_lm.models.qwen3, which was simply not listed. No kernel was missing; a name check was excluding three of the newest models and a VLM. Membership is now explicit (module, class) pairs in two registries, because a claim about an implementation should not be spelled as a claim about a name. Graph fusion keeps a stricter one: its replacement reproduces a specific residual structure, and Qwen3.5's DecoderLayer is included only because `_attention_module` now resolves the attention by attribute. That layer is a hybrid -- on every layer that is not a multiple of full_attention_interval the attention is a GatedDeltaNet bound to `linear_attn` and `self_attn` does not exist at all -- so a replacement calling `self.self_attn` would have crashed. Resolving per call rather than per class is the whole adaptation; a block binding neither name falls back instead of guessing. Skips went from 16 to 0, and all ten newly active cases pass bit-exact on the first run, both token equality and decode logits. Lighting them up also surfaced a flake that the skips had been hiding, in a test my change does not touch: Qwen3-VL attention failed bit-exactness in one of five full matrix runs, while passing in isolation and passing with its own model's cases run alone. It needed the whole matrix, which is the signature of memory pressure, and this file never released a model -- eighty-two tests each load a checkpoint, up to fifteen gigabytes, and letting the Python reference go does not return device memory because MLX caches freed buffers. A fixture now drops both after every test. Six full matrix runs since are clean, and they are also consistently faster, 227 to 235 seconds against 294 to 304 before, which is independent support for the diagnosis rather than for the fix merely hiding it. Not proof: five clean runs against a one-in-five rate is likely but not decisive. The mechanism, the timing change and the precedent in this project all point the same way. 82 pass with no skips; 655 in the fast suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixteen cases in the model matrix were skipping, not passing — and a skip reads like a pass in a
summary. Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-27B and Qwen3-VL-4B reported "patches nothing", so their
logits agreed with MLX for the uninteresting reason that meTile was not running.
No kernel was missing — a name check was
The allowlist was keyed on a set of modules plus a hardcoded class name of
"MLP". Everyarchitecture involved computes the same gated MLP —
down_proj(swiglu(gate_proj(x), up_proj(x))),where
swiglu(gate, x)isnn.silu(gate) * x, which is exactly what_execute_quantized_mlpcomputes — but they don't all call the class
MLP:qwen3_next.Qwen3NextMLPqwen3_5.DecoderLayerqwen3.MLPqwen3.TransformerBlockMembership is now explicit
(module, class)pairs in two registries, because a claim about animplementation shouldn't be spelled as a claim about a name.
Graph fusion keeps a stricter registry: its replacement reproduces a specific residual structure.
Qwen3.5's
DecoderLayeris included only because_attention_modulenow resolves the attention byattribute — that layer is hybrid, with a GatedDeltaNet bound to
linear_attnon every layerthat isn't a multiple of
full_attention_intervaland noself_attnat all, so a replacementcalling
self.self_attnwould have crashed. Resolving per call rather than per class is the wholeadaptation; a block binding neither name falls back instead of guessing.
Skips: 16 → 0. All ten newly active cases pass bit-exact on the first run — both token equality
and decode logits.
Lighting them up surfaced a flake the skips had been hiding
In a test this change doesn't touch: Qwen3-VL attention failed bit-exactness in one of five full
matrix runs, while passing in isolation and passing with its own model's cases run alone. Needing
the whole matrix is the signature of memory pressure — and this file never released a model.
Eighty-two tests each load a checkpoint, up to 15 GB, and letting the Python reference go does not
return device memory, because MLX caches freed buffers. A fixture now drops both after every test.
Six full matrix runs since are clean, and consistently faster — 227–235s against 294–304s —
which is independent support for the diagnosis rather than for the fix merely hiding it.
Not proof. Five clean runs against a one-in-five rate is likely but not decisive. The mechanism,
the timing change, and a recorded precedent in this project all point the same way.
82 pass with no skips; 655 in the fast suite.
🤖 Generated with Claude Code