Observation
encodeMoERouteRows emits two dispatches per layer:
qmm8_rows: postNorm[H] → gateLogits[E=256], dispatched as E/8 threadgroupsroute_top8_rows: gateLogits[E] → inds[Ktop] + scores[Ktop], dispatched as one threadgroup of 256 threads
Step 2 is the shape that hurts most under a serial encoder: it holds the GPU's ordered slot while using a single threadgroup, and it forces a full barrier around a 512-byte intermediate (sc.gl) that no one else reads. At M=1 there are 40 of these per token; the same structure repeats at every verify width.
Change
One kernel producing inds/scores directly from postNorm, with the E=256 logits held in threadgroup memory instead of round-tripping through sc.gl.
At E=256 the logits fit trivially in threadgroup memory, so the fused kernel can compute the gate matvec across its threadgroups and then run the existing top-k reduction. The awkward part is that step 1 is currently multi-threadgroup (E/8) and step 2 is single-threadgroup — a naive fusion either serializes the matvec into one threadgroup (losing the parallelism of step 1) or needs a threadgroup-level reduction. Which of those is faster is not obvious and should be measured on the isolated kernel before wiring — the same discipline #137 arrived at the hard way.
Keep sc.gl written when diagRouteBufs != nil — diag_copy_route reads raw gl for the notes/11 Stage-0 / notes/13 recalib telemetry, and that path must stay byte-identical.
Effect
If #143's P3 shows the bubble share is large, this issue's real value is the barrier removal rather than the dispatch count — the fused kernel is one of the few places where a single-threadgroup dispatch can be deleted outright rather than merely overlapped.
Bit-exactness
Not constructive here, unlike #144 — this is a new kernel. The accumulation order of the gate matvec and the tie-breaking order of the top-8 selection must both be preserved exactly. Routing is a near-tie-sensitive path: notes/01 §1-2 records that route decode was validated lossless specifically because near-ties were checked, and notes/00 documents what happens when a near-tie flips. A top-k whose comparison order differs from route_top8_rows is a silent behaviour change even when every float is bit-identical.
Treat the tie-break order as part of the contract, not an implementation detail.
Gates
- G-A (locked, model-free): fused vs unfused produce bit-identical
inds and scores on synthetic logits, including adversarial near-tie and exact-tie inputs at the top-8 boundary, M ∈ {1, 2, 8}. Bumps RAWTESTS total = N.
- G-B (lossless, model): spec-vs-greedy 512/512 × 4 regimes; flag-off byte-unchanged (
QWISP_FUSE_ROUTE=0). Bolt path checked separately (slot_remap consumes inds immediately after).
- G-C (wiring): reviewer confirms the
diagRouteBufs telemetry path still sees raw gl, and that the bias variant (route_top8_rows_bias, notes/12 Stage 1 案B) either fuses identically or falls back to the unfused pair when routeBiasEps > 0.
- G-D (speed, report-only): isolated kernel A/B first (fused vs
qmm8 + route_top8 back to back), then end-to-end tok/s. If the isolated A/B is not a clear win, stop — the dispatch-count saving alone does not justify a new kernel on the routing path.
Refs
#143 (dispatch inventory) · #144 (the other, constructive-bit-exactness half of the same dispatch-count reduction) · notes/00 (near-tie failure modes) · notes/01 §1-2 (route decode lossless validation) · notes/11, notes/12, notes/13 (telemetry + bias consumers of gl/inds) · SeedlessFusedVerify.encodeMoERouteRows
Observation
encodeMoERouteRowsemits two dispatches per layer:qmm8_rows: postNorm[H] → gateLogits[E=256], dispatched as E/8 threadgroupsroute_top8_rows: gateLogits[E] → inds[Ktop] + scores[Ktop], dispatched as one threadgroup of 256 threadsStep 2 is the shape that hurts most under a serial encoder: it holds the GPU's ordered slot while using a single threadgroup, and it forces a full barrier around a 512-byte intermediate (
sc.gl) that no one else reads. At M=1 there are 40 of these per token; the same structure repeats at every verify width.Change
One kernel producing
inds/scoresdirectly frompostNorm, with the E=256 logits held in threadgroup memory instead of round-tripping throughsc.gl.At E=256 the logits fit trivially in threadgroup memory, so the fused kernel can compute the gate matvec across its threadgroups and then run the existing top-k reduction. The awkward part is that step 1 is currently multi-threadgroup (E/8) and step 2 is single-threadgroup — a naive fusion either serializes the matvec into one threadgroup (losing the parallelism of step 1) or needs a threadgroup-level reduction. Which of those is faster is not obvious and should be measured on the isolated kernel before wiring — the same discipline #137 arrived at the hard way.
Keep
sc.glwritten whendiagRouteBufs != nil—diag_copy_routereads rawglfor the notes/11 Stage-0 / notes/13 recalib telemetry, and that path must stay byte-identical.Effect
sc.glwrite→read round trip (512B/layer at M=1; M×E at verify widths).If #143's P3 shows the bubble share is large, this issue's real value is the barrier removal rather than the dispatch count — the fused kernel is one of the few places where a single-threadgroup dispatch can be deleted outright rather than merely overlapped.
Bit-exactness
Not constructive here, unlike #144 — this is a new kernel. The accumulation order of the gate matvec and the tie-breaking order of the top-8 selection must both be preserved exactly. Routing is a near-tie-sensitive path: notes/01 §1-2 records that route decode was validated lossless specifically because near-ties were checked, and notes/00 documents what happens when a near-tie flips. A top-k whose comparison order differs from
route_top8_rowsis a silent behaviour change even when every float is bit-identical.Treat the tie-break order as part of the contract, not an implementation detail.
Gates
indsandscoreson synthetic logits, including adversarial near-tie and exact-tie inputs at the top-8 boundary, M ∈ {1, 2, 8}. Bumps RAWTESTStotal = N.QWISP_FUSE_ROUTE=0). Bolt path checked separately (slot_remapconsumesindsimmediately after).diagRouteBufstelemetry path still sees rawgl, and that the bias variant (route_top8_rows_bias, notes/12 Stage 1 案B) either fuses identically or falls back to the unfused pair whenrouteBiasEps > 0.qmm8 + route_top8back to back), then end-to-end tok/s. If the isolated A/B is not a clear win, stop — the dispatch-count saving alone does not justify a new kernel on the routing path.Refs
#143 (dispatch inventory) · #144 (the other, constructive-bit-exactness half of the same dispatch-count reduction) · notes/00 (near-tie failure modes) · notes/01 §1-2 (route decode lossless validation) · notes/11, notes/12, notes/13 (telemetry + bias consumers of
gl/inds) ·SeedlessFusedVerify.encodeMoERouteRows