diff --git a/DECODE_OPTIMIZATION_RESEARCH.md b/DECODE_OPTIMIZATION_RESEARCH.md
new file mode 100644
index 00000000..dc33686b
--- /dev/null
+++ b/DECODE_OPTIMIZATION_RESEARCH.md
@@ -0,0 +1,140 @@
+# Nemotron-Nano-30B Decode Optimization — Research & Synthesis (GB10 / sm_121)
+
+**Date:** 2026-06-05
+**Author:** research synthesis (web + reasoning), no code touched, spark untouched.
+**Question:** vLLM gets **55.8 tok/s @ 20.9 GB**; we get **54.2 tok/s @ 18.4 GB** at byte-matched NVFP4 recipe. vLLM reads MORE bytes yet runs faster → its kernels are more byte-efficient. What is it doing that we aren't, and what else can push decode higher?
+
+---
+
+## TL;DR — the one finding that matters
+
+The gap is **not** in the dense GEMV path and **not** in precision. At batch=1 the dense layers (lm_head, attn projections, Mamba in/out proj, shared expert) are **memory-bound and tensor cores do NOT help** — this is now firmly established (see §A). Our scalar-dequant GEMV is the structurally correct choice there, and our `lm_head` already runs at 99% of roofline.
+
+The gap is concentrated in **two places vLLM does structurally differently**:
+
+1. **The MoE expert GEMM.** With top-6 experts × intermediate rows, the MoE GEMM has *enough M* to use Blackwell **tensor cores via TMA-warp-specialized grouped GEMM** (FlashInfer/CUTLASS `b12x`). This is the ONE matmul in the decode where M>1, so it's the one place tensor cores legitimately win — and it's also our single biggest token share (45.6%) and our worst-efficiency kernels (shared-expert 48–53%, gather 71–74%). The public CUTLASS data shows the *exact same FP4 MoE grouped-GEMM* going **14.6 → 39 tok/s purely by switching `compute_120a → compute_120f` to enable TMA warp-specialized tactics** — a 2.7× kernel-efficiency swing with zero precision change [cutlass#3096].
+2. **Kernel fusion that collapses dispatch count.** vLLM fuses (a) the whole MoE — *token dispatch + W1 GEMM + SwiGLU + W2 GEMM into a single kernel* (FlashInfer `b12x_fused_moe`), and (b) RoPE+Quant+Q-write on the attention side, "eliminating two intermediate memory round-trips" [vllm-blackwell-blog, flashinfer-docs]. We run ~390 kernels/token with our gather-up / silu / gather-down as *separate* dispatches, each round-tripping the intermediate activation through DRAM.
+
+Everything else (FP8-vs-INT8, vectorized loads, cp.async, persistent kernels, megakernel) is either already-proven-flat for us or structurally a non-win at batch=1. Speculative decode (EAGLE3/MTP) is the one *additional* lever that is **net-positive at batch=1** and that we appear to have prematurely dismissed (see §3) — but it's a quality/infra cost, not a pure-speed freebie.
+
+---
+
+## §A — The settled fact: tensor cores do NOT help batch-1 dense GEMV
+
+This kills a whole class of "are we leaving the FP8/FP4 MMA path on the table?" ideas for the *dense* layers, so it's worth nailing down before the ranked list.
+
+- A dedicated NVFP4 batch-1 GEMV study (M=7168, K=16384, L=1) found the best hand-written CUDA-core kernel hit **26.7 µs vs 8.6 µs speed-of-light (~32% of SoL)**, and the author's explicit conclusion: *"The kernel is fundamentally memory-bound. FP4 data is tiny (4 bits per element)… FMA latency hiding is irrelevant when every cycle is waiting on memory."* Even the top competitors only reached ~2× SoL, and a plain PyTorch call scored 22.4 µs — i.e. custom tensor-core tricks bought almost nothing [amandeepsp-nvfp4-gemv].
+- Blackwell's `tcgen05.mma` accumulates in **TMEM** and *always needs ≥4 epilogue warps to drain a 128-wide tile*; the practical minimum tile is **BLOCK_M=128**. At M=1 you pad 1→128 and waste ~99% of the tensor-core ALU [gau-nernst-tcgen05, vllm batch-1 GEMV note]. vLLM itself ships a runtime-dispatched custom op specifically because *"Triton's GEMV pads M=1 up to the tensor-core tile and wastes most of the ALU at batch size 1."*
+
+**Implication for us:** our `lm_head` at 99% and the dense projections near roofline are *already at the ceiling*. Do not chase an FP8/FP4 tensor-core GEMV for lm_head / attn-proj / shared-up/down / Mamba-proj. That door is closed by physics, and we'd be re-proving what vLLM and the GEMV-hackathon already proved.
+
+The MoE expert GEMM is the **exception**: top-6 routing gives it real M, so tensor cores are legitimately on the table there (§1).
+
+---
+
+## LIKELY-REAL WINS (ranked)
+
+### 1. Fuse the MoE into a single grouped-GEMM kernel (dispatch + W1 + SwiGLU + W2) — **HIGHEST PAYOFF**
+**What:** Replace our separate `moe_gather_up` → `silu` → `moe_gather_down` dispatches with one fused kernel that does token-dispatch, the up-GEMM, SwiGLU, and the down-GEMM in a single launch, keeping the intermediate activation in shared memory / registers instead of round-tripping it to DRAM. This is exactly FlashInfer `b12x_fused_moe` / `cutlass_fused_moe` [flashinfer-docs, vllm-pr#40082].
+
+**Why it helps given our profiling:** MoE is **45.6% of our token** and contains our two worst-efficiency kernels (shared 48–53%, gather 71–74%). The 71–74% gather efficiency is precisely the "scatter penalty + intermediate round-trip" that fusion removes — the up-output never hits DRAM, so the down-GEMM doesn't re-read it, and the dispatch/scatter is amortized inside one kernel. The vLLM fused path also uses a `block_id → (expert_id, offset)` indirection table so variable-sized expert batches run *in one launch without padding waste* — directly attacking our dynamic-scatter cost.
+
+**Expected impact:** This is where vLLM's "reads more bytes but goes faster" advantage lives. Removing the intermediate activation round-trip on up→down (0.69 GB of gather traffic + the silu pass) plus dispatch fusion is plausibly **5–10%** end-to-end. The profiling map's own "path-to-75" math already credits ~1.0 ms (≈7%) to fixing gather efficiency alone; fusion is the mechanism that gets there.
+
+**Effort:** HIGH. This is the big structural rewrite — a grouped-GEMM with in-kernel SwiGLU in the `#[kernel]` DSL. Note the playbook already shows `moe_down_swiglu_accum_int4_chain8` exists (PR #201), so a fused down+swiglu is partially precedented; the harder part is the up+swiglu+down single-launch grouped form.
+**Risk:** MEDIUM. Correctness of grouped-GEMM offset math + SwiGLU numerics (playbook: silu wants 2e-4 / f16 5e-2 tols). CUDA-graph-safety must be preserved.
+**Sources:** [vllm-pr#40082], [flashinfer-docs cutlass_fused_moe], [vllm-fused-moe-design].
+
+### 2. Multi-warp the shared-expert (already #1 on the in-tree list) + apply the *same* coalescing to gather
+**What:** The profiling map's #1 ranked item — `shared_up_q4` / `shared_down_acc` are single-warp-per-row at 48–53%; mirror the multi-warp `rows_per_tg` coalesced kernel to target 80%+. Blocked earlier by an f32-vs-f16 scale-type mismatch in the accum fusion — fix the dtype (playbook §"f32-vs-f16 scale type" is a recurring trap).
+
+**Why it helps:** Pure kernel-efficiency, no precision change, clearest measured headroom (the *weakest* kernels in the whole token). This is the lowest-risk real win and is *independent* of the big MoE fusion in §1 — do it first as a down-payment.
+**Expected impact:** −0.4–0.5 ms/tok (≈3%) per the in-tree estimate.
+**Effort:** LOW–MEDIUM. **Risk:** LOW (the dtype fix is understood).
+**Source:** in-tree `PROFILING_32K_DECODE.md` ranked item #1.
+
+### 3. Use TMA-warp-specialized tactics for the MoE grouped-GEMM (the `compute_120f` lesson)
+**What:** This is the *mechanism* behind why vLLM's MoE is byte-efficient, and it's CUDA-toolchain-level, not algorithm-level. The public CUTLASS thread shows the identical FP4 MoE grouped GEMM on desktop Blackwell going **14.6 tok/s (`compute_120a`, CUDA 12.8) → 39 tok/s (`compute_120f`, CUDA 13.0)** — a **2.7×** swing — purely because `compute_120f` lets the autotuner select **TMA warp-specialized grouped-GEMM kernels** instead of falling back to non-TMA variants [cutlass#3096]. For us this means: if/when our MoE GEMM uses tensor cores at all (per §1, M is large enough there), the build must target the `f`-suffix arch and use TMA (async bulk copy) to feed the MMA. Note: **TMA/cp.async is only worth it for the tensor-core MoE tile** — the playbook already proved `simdgroup_async_copy` / cp.async-style prefetch is *dead* for batch-1 GEMV (amortizes only over kilo-FLOP tiles), so do NOT apply it to the dense GEMVs.
+
+**Why it helps:** This is the single largest *documented* kernel-efficiency multiplier on this exact hardware+model class, and it's specific to the one matmul where tensor cores apply.
+**Expected impact:** Large *if* we adopt a tensor-core MoE GEMM (it's the enabling toolchain piece for §1's grouped GEMM, not a standalone lever). On its own, on our current scalar-MoE, it does nothing.
+**Effort:** Coupled to §1 (need a tensor-core grouped GEMM first). **Risk:** sm_121 toolchain fragility — note vLLM hit `nvidia-cutlass-dsl==4.5.0 generates faulty PTX for SM121`, must pin 4.4.2; and `compute_120f` needs CUDA 13.0 [vllm-pr#40082, cutlass#3096].
+**Sources:** [cutlass#3096], [vllm-pr#40082].
+
+### 4. Fuse the attention pre-amble: RoPE + Quant + (K/Q write) into one kernel
+**What:** vLLM fuses "RoPE + Quant + Q Write (Decode)" into a single kernel, "eliminating two intermediate memory round-trips" [vllm-blackwell-blog]. Our equivalent rope/quant/kv-write are separate tiny dispatches.
+
+**Why it helps:** Each is "≈0 ms real work" per our profiler, BUT we run ~390 kernels/token and the *dispatch + DRAM round-trip* of the intermediate is the real cost, not the FLOPs. Collapsing 3 dispatches→1 removes two activation round-trips and two launch latencies per layer. With CUDA graphs the launch cost is already low, so the win here is the round-trip elimination, modest but cheap.
+**Expected impact:** 1–3% (small per-op, but it's per-layer × 23+ layers).
+**Effort:** LOW–MEDIUM. **Risk:** LOW. **Source:** [vllm-blackwell-blog].
+
+---
+
+## NET-POSITIVE BUT QUALITY/INFRA-COSTED (re-examine — we may have been wrong)
+
+### 5. Speculative decode (EAGLE3 / MTP) — **contradicts our "net-loss at batch=1" prior**
+**What:** A small draft proposes k tokens; the target verifies all k in one forward pass. At B=1 the GPU is bandwidth-bound with **idle compute**, so the verification pass costs ~the same as one normal token while yielding >1 accepted token.
+
+**Why our prior may be wrong:** The evidence says spec-decode is **net-positive precisely at B=1** and *loses* benefit as batch grows (opposite of the usual intuition): *"At B=1, even a modest acceptance rate yields meaningful latency gains because the GPU has substantial idle capacity"* [hf-eagle3]. Concrete measured single-stream numbers on a comparable hybrid/MoE-scale model: **120 → 168 tok/s (1.39×) at 40% acceptance, 2.4 accepted/verify, k=6**, with a 277 MB draft against a 30B-class target [hf-eagle3]. The "7× weight reads" overhead (6 draft + 1 target) is dwarfed by the tiny draft size and absorbed by spare bandwidth.
+
+**The catch (why it's costed, not free):**
+- Needs a **trained draft head / MTP module** for *this* model (Nemotron-3-Nano ships MTP-style heads in some configs — worth checking the checkpoint). Without one, you train/distill it = real infra cost.
+- **Acceptance rate is the whole ballgame.** At 40% accept you get 1.39×; below ~25% it can go net-negative. Reasoning models with long deterministic spans tend to accept *well*, which is favorable here.
+- Verification pass on a **hybrid Mamba+MoE** model is more complex than on a dense transformer: the **Mamba2 state must be advanced/rolled-back per draft token** (the SSM is sequential), which is the non-trivial engineering piece and a place our hybrid architecture is harder than vLLM's dense-transformer spec path.
+- At our 273 GB/s (vs the 3.35 TB/s box in the cited result) the *bandwidth slack* is tighter, so the realized speedup will be lower than 1.39× — but still plausibly net-positive at decent acceptance.
+
+**Expected impact:** **+15–35%** single-stream tok/s *if* acceptance ≥40% and the Mamba-state rollback is solved — by far the largest single lever, but the only one that changes the token-count math instead of the per-token cost.
+**Effort:** HIGH (draft head + Mamba2 state checkpoint/rollback in the decode loop). **Risk:** MEDIUM–HIGH (acceptance-rate dependent; Mamba rollback correctness).
+**Recommendation:** **Re-open this.** Verify whether the NVFP4 checkpoint ships an MTP head, and prototype acceptance-rate measurement before committing — that single number decides go/no-go.
+**Sources:** [hf-eagle3], [modular-specdec], [sglang-specdec].
+
+### 6. Sub-4-bit experts (Q3 / NVFP4-with-tighter-grouping) — the path past 75, quality-costed
+**What:** Drop experts to ~3-bit. The experts dominate MoE bytes; trimming 4→3 bit cuts the largest byte share.
+**Why/impact:** Mechanically ~+15–20% (matches the in-tree note that 75 needs Q3). **Quality cost is real** and no engine ships it at 32K for this model. Mark as a quality-traded option, not a clean win.
+**Effort:** MEDIUM (quant recipe + QAD-style recovery). **Risk:** HIGH on quality. **Source:** in-tree map; [nvidia-nemotron-nvfp4] (NVIDIA themselves stop at 4-bit experts + 8-bit elsewhere via QAD, i.e. they judged sub-4-bit not worth the quality hit).
+
+---
+
+## PROBABLY FLAT / DEAD-ENDS (do NOT re-litigate — with the *why*)
+
+| Idea | Why it's flat for us |
+|---|---|
+| **FP8/FP4 tensor-core GEMV for dense layers** (lm_head, attn-proj, shared, Mamba-proj) | Batch-1 dense GEMV is memory-bound; tensor cores need M≥128, you pad 1→128 and waste the ALU. Proven by the NVFP4-GEMV study (~32% SoL even hand-tuned) and tcgen05's 4-epilogue-warp/BLOCK_M=128 floor. Our lm_head is already 99%. §A. |
+| **cp.async / TMA prefetch on dense GEMVs** | Amortizes only over kilo-FLOP tensor-core tiles. Playbook already measured `simdgroup_async_copy` dead at 2–4 KB matvec X-loads. Only valid inside the §1/§3 tensor-core MoE tile. |
+| **Persistent / megakernel for the whole decode** | Already tried: cooperative-launch megakernel can't fit GB10's 48 SMs. Persistent-kernel for *all* of decode is the same class. (Fusing *within* MoE per §1 is different and fine.) |
+| **Vectorized uint4 loads, MT_GEMV_2ROW/VEC, rpt sweeps, multiwarp-shared(prev attempt), gather cache-streaming hints** | All measured flat or worse in-tree (uint4 −34%, gather rpt flat because bottleneck is scatter not warp count). |
+| **TG-memory cache for X-broadcast in matvec** | Playbook: −33pp on M-series; L1 already absorbs the redundancy. Same logic on GB10's L1/L2. |
+| **INT8 → FP8(E4M3) for the Q8 layers** | Both are 8-bit = same byte budget = same bandwidth at batch-1. NVIDIA's E4M3 vs our INT8/f32-scale is a numerical/quality nuance, **not** a speed lever (the baseline doc already says "minor diff"). Don't expect tok/s from it. |
+| **`concat_mla_k` / MLA KV tricks** | vLLM's MLA-specific KV kernel — Nemotron-Nano is **not** MLA (it's hybrid Mamba2 + standard GQA attention on a minority of layers). Not applicable. |
+
+---
+
+## On the Mamba2 decode step specifically (§4 of the brief)
+Mamba is only **15% of our token** and our in/out-proj already run 64–76%. The public Mamba2 fusion wins (PyTorch blog, vLLM SSD) are **all prefill chunked-scan** — there is **no published faster single-step selective-scan kernel** for decode; the decode path is already a cheap conv1d + per-channel state update [pytorch-mamba2, vllm-mamba2]. The only decode-relevant note: vLLM split conv1d into separate prefill/decode kernels to avoid mixed-batch slowdown — not relevant to our pure-decode bench. **Verdict: low priority.** The Mamba in/out-proj headroom (64–76%) is ordinary GEMV multi-warp tuning, same bucket as §2, not a Mamba-algorithm change. (Caveat in §5: if we do spec-decode, the Mamba *state rollback* becomes the hard part — that's the one place Mamba decode work pays off.)
+
+---
+
+## Recommended sequence (most-confident first)
+1. **§2 multi-warp shared-expert** (fix the f16 scale dtype) — low risk, clear headroom, ~3%, down-payment.
+2. **§4 fuse rope+quant+kv-write** — low risk, ~1–3%, cheap.
+3. **§1 fused MoE grouped-GEMM (dispatch+up+swiglu+down)** + **§3 TMA/`compute_120f` toolchain** — the big structural win that mirrors vLLM; ~5–10%. This is the one that closes the "vLLM reads more, goes faster" gap.
+4. **§5 spec-decode acceptance-rate spike** — measure acceptance on the real checkpoint *before* building; if ≥40% and Mamba-rollback tractable, it's the largest lever (+15–35%) and the realistic path *past* parity to a genuine win.
+5. **§6 Q3 experts** — only if a quality budget exists; the documented route to ~75 but quality-costed.
+
+§1+§3 together should plausibly take us from 54.2 to **parity-plus** at matched precision (closing the 3% and adding a few). §5 is what turns "at parity" into "beats vLLM" on single-stream. The dense-GEMV tensor-core fantasy (§A) is dead — stop looking there.
+
+---
+
+## Sources
+- vLLM Blackwell WideEP blog (RoPE+Quant fusion, NVFP4 MoE dispatch, FlashInfer TRTLLM-Gen GEMM): https://vllm.ai/blog/2026-02-03-dsr1-gb200-part1
+- vLLM PR #40082 — FlashInfer b12x fused MoE + FP4 dense GEMM for SM120/121 (fuses dispatch+W1+SwiGLU+W2; CuTe-DSL warp-MMA adaptive tiling for small-M decode; +1.8% 1P / +6.0% 8P on DGX Spark Qwen3-30B-A3B-NVFP4; pin cutlass-dsl 4.4.2): https://github.com/vllm-project/vllm/pull/40082
+- CUTLASS issue #3096 — FP4 MoE grouped GEMM on desktop Blackwell, 14.6→39 tok/s via compute_120f TMA warp-specialized tactics; Marlin W4A16 baseline 46–49 tok/s: https://github.com/NVIDIA/cutlass/issues/3096
+- "Twelve Attempts at an FP4 Kernel" (batch-1 NVFP4 GEMV is memory-bound, ~32% SoL, tensor cores don't help): https://amandeepsp.github.io/blog/nvfp4-blackwell-gemv/
+- tcgen05 for dummies (TMEM, ≥4 epilogue warps / BLOCK_M=128 floor → unsuitable for M=1): https://gau-nernst.github.io/tcgen05/
+- FlashInfer cutlass_fused_moe docs (single-call fused MoE, block_id→expert mapping, variable expert batches no padding): https://docs.flashinfer.ai/generated/flashinfer.fused_moe.cutlass_fused_moe.html
+- vLLM fused-MoE design / moe_align_block_size (persistent grouped GEMM, indirect token addressing, padding-to-block): https://docs.vllm.ai/en/latest/design/moe_kernel_features/
+- EAGLE3 single-stream B=1 (1.39× @ 40% accept, 120→168 tok/s; net-positive at B=1, shrinks with batch): https://huggingface.co/blog/lujangusface/tw-eagle3-gpu
+- Modular spec-decode / SGLang spec-decode (B=1 bandwidth-bound idle-compute rationale): https://docs.modular.com/max/serve/speculative-decoding/ , https://sgl-project.github.io/advanced_features/speculative_decoding.html
+- PyTorch "Accelerating Mamba2 with Kernel Fusion" (prefill SSD only, 1.5–2.5×; no decode single-step win): https://pytorch.org/blog/accelerating-mamba2-with-kernel-fusion/
+- vLLM Mamba2 conv1d prefill/decode split (PR #17146), SM121 Triton crash note (#37431): https://github.com/vllm-project/vllm/pull/17146
+- Nemotron-3-Nano-30B-A3B-NVFP4 model card / paper (128 experts, top-6 + 1 shared, 3.5B active, QAD 4-bit experts): https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 , https://arxiv.org/pdf/2512.20848
diff --git a/PROFILING_32K_DECODE.md b/PROFILING_32K_DECODE.md
new file mode 100644
index 00000000..d108d256
--- /dev/null
+++ b/PROFILING_32K_DECODE.md
@@ -0,0 +1,66 @@
+# Nemotron-Nano-30B-A3B — 32K Decode Profiling Map (GB10 / ASUS GX10)
+
+**Goal:** 75+ tok/s decode @ 32,768 ctx. **Current:** 68.2 tok/s graph-batched (14.63 ms/token), full-quality Q4, argmax 1234.
+
+**Config:** `NEMOTRON_FAKECTX=32768 NEMOTRON_GRAPH=1 NEMOTRON_DEVROUTER=1 NEMOTRON_Q4CACHE=1 NEMOTRON_F16KV=1`, GB10 sm_121, LPDDR5X 273 GB/s peak.
+
+**Method:** per-op CUDA-event profiler (`NEMOTRON_PROFILE=1`, eager, sync-around-each-op). ⚠️ The profiler synchronizes per op, so **ms/tok is sync-inflated for tiny elementwise ops** (silu/rope/rms_norm/conv read ~0 bytes → their "8 ms" is sync overhead, real ≈0). **Trust the `GB/s` column and the ablation, not the per-op ms.**
+
+## The reference: achievable bandwidth = ~189 GB/s
+`lm_head` is a big contiguous Q4 GEMV and runs at **187 GB/s ≈ 99% of the achievable roofline**. So **189 GB/s is reachable on this hardware** — any kernel below it has headroom that is *kernel efficiency*, not a hardware wall.
+
+## Per-kernel map (real, bandwidth-bound work)
+
+| kernel | GB read/tok | eff GB/s | % of 189 achievable | calls/tok | verdict |
+|---|---|---|---|---|---|
+| **lm_head** | 0.176 | **187** | **99%** | 1 | ✅ saturated — no headroom |
+| m_in_proj (Mamba) | 0.319 | 144 | 76% | 23 | 🟡 mild headroom |
+| moe_gather_up | 0.345 | 140 | 74% | 23 | 🟡 **scatter penalty** |
+| moe_gather_down | 0.346 | 134 | 71% | 23 | 🟡 **scatter penalty** |
+| m_out_proj (Mamba) | 0.127 | 120 | 64% | 23 | 🟡 headroom |
+| shared_up_q4 | 0.115 | 101 | 53% | 23 | 🔴 **single-warp, big headroom** |
+| shared_down_acc | 0.115 | 90 | 48% | 23 | 🔴 **single-warp, big headroom** |
+| sdpa_2pass | 0.202 | 78 | 41% | 6 | 🟡 KV-read, latency-bound |
+| silu/rope/rms_norm/conv/ssm/router | ~0 | — | — | 12–52 | sync-artifact, real ≈0 ms |
+
+**host overhead:** 0.75 ms/tok (eager; graphs remove most of it).
+
+**Confirmation run (warmer box) — ranking reproduced, efficiencies thermal-sensitive:**
+`lm_head` **186.8 GB/s = 99%** (stable), moe_gather_up/down **119/115 = 63%/61%**, m_in_proj **121 = 64%**, m_out_proj **118 = 63%**, shared_up_q4 **77 = 41%**, shared_down_acc **68 = 36%**, sdpa_2pass **58 = 31%**. Absolute % drifts with temperature but `lm_head`≈99% and the under-performer ranking are invariant. **Headroom is real and likely *larger* than the first table suggests.**
+
+## Category ablation (skip-based ground truth)
+- **MoE total: 7.3 ms (45.6%)** — gather up/down + gate + shared experts
+- **Attn + lm_head + norms: 6.3 ms (39.4%)** — q/k/v/o proj, sdpa, lm_head
+- **Mamba: 2.4 ms (15.0%)** — in/out proj, conv1d, ssm_step
+
+## KEY INSIGHT — this is NOT a hardware wall
+`lm_head` proves 189 GB/s is achievable. The dominant kernels run well under it:
+- **shared-expert (up 53% / down 48%)**: weakest. Cause = single-warp-per-row (no multi-warp `rows_per_tg`). Fix = mirror the multi-warp coalesced kernel → target 80%+.
+- **moe_gather up/down (74%/71%)**: the top-6-of-128 **dynamic scatter** (6 experts at different offsets) costs ~25% vs lm_head's contiguous read. Levers: cache-streaming hints (`ld.global.cs`), better expert-block locality, or compaction.
+- **m_out_proj (64%) / m_in_proj (76%)**: mild headroom.
+
+## Path-to-75 math (clean, no precision loss)
+Token = 14.63 ms. Pulling the underperformers toward the proven 189 GB/s:
+- shared-expert 0.23 GB @ ~95 → @ 170 GB/s: saves ~1.0 ms
+- moe_gather 0.69 GB @ ~137 → @ 175 GB/s: saves ~1.0 ms
+- → ~12.6 ms = **~79 tok/s**. **75 is reachable on efficiency alone.**
+
+## Ruled out (measured, no gain)
+- SDPA 2-pass BC4 / TILED variants: **much worse** (56 ms vs 14 ms)
+- SDPA split-K block sweep (64–512): flat
+- `MT_MOE_RPT` 1–4 on gather: flat (gather bottleneck is scatter, not warp count)
+- `--use_fast_math`: no change
+- `MT_GEMV_2ROW`, `MT_GEMV_VEC`: crash
+- uint4 vectorized loads: −34% (starves the pipe)
+- f16-KV: +2 tok/s (banked)
+
+## Ranked opportunities (next work)
+1. **shared_up_q4 + shared_down_acc → multi-warp (`rows_per_tg`)** — 48–53% → 80%+, est. **−0.4–0.5 ms/tok**. Lowest risk, clearest headroom. *(blocked earlier by f32-vs-f16 scale type mismatch in the accum fusion — fix the dtype.)*
+2. **moe_gather scatter efficiency** — 71–74% → 85%+. Try `ld.global.cs` cache-streaming hints (inline PTX) + expert-block locality. Biggest share of the token (45%), so highest absolute payoff if the scatter penalty is partly cache-pollution.
+3. **m_out_proj** (64%) — multi-warp / config tune.
+
+## Banked optimizations (in the 68.2)
+CUDA graphs (+6.5%), Q4 disk cache (setup 120 s→20 s), MoE rpt2 default, parallel dequant, f16-KV, FMAD-on, `__ldg`/`__restrict__`/`__expf` codegen.
+
+---
+*Generated from the in-tree `NEMOTRON_PROFILE=1` per-op profiler (wh-butter-modeltests/src/lib.rs). Re-run: `NEMOTRON_PROFILE=1 NEMOTRON_FAKECTX=32768 NEMOTRON_DECODE=24 ... nemotron_decode_bench`.*
diff --git a/PROFILING_PREFILL.md b/PROFILING_PREFILL.md
new file mode 100644
index 00000000..c2a52000
--- /dev/null
+++ b/PROFILING_PREFILL.md
@@ -0,0 +1,26 @@
+# Nemotron-Nano-30B BATCHED PREFILL — per-op profiling map
+
+- Device: GB10 sm_121 (GB10 Blackwell)
+- S (prompt tokens): 2048
+- Clean batched throughput: **74.3 tok/s** (13.46 ms/tok)
+- Profiled pass wall (sync-bracketed, inflated): 28.465s; summed op time: 13.536s
+- vLLM-on-GB10 reference: pp2048@d0=6395, @d8192=4993, @d32768=2734 tok/s
+- Tensor-core peak assumed: 1000 TFLOP/s (bf16 dense)
+
+| op | ms | % | calls | TFLOP/s | %peak |
+|---|---:|---:|---:|---:|---:|
+| moe_experts | 7137.65 | 52.7% | 69 | 0.790 | 0.08% |
+| proj_gemm | 3764.71 | 27.8% | 70 | 1.121 | 0.11% |
+| moe_shared | 1680.60 | 12.4% | 46 | 1.119 | 0.11% |
+| ssm_scan | 673.69 | 5.0% | 23 | 0.147 | 0.01% |
+| sdpa_prefill | 202.75 | 1.5% | 6 | 1.017 | 0.10% |
+| moe_router | 32.37 | 0.2% | 23 | 1.001 | 0.10% |
+| slice/cast | 31.36 | 0.2% | 140 | — | — |
+| rms_norm | 12.01 | 0.1% | 52 | — | — |
+| lm_head | 1.01 | 0.0% | 1 | 0.699 | 0.07% |
+
+## Gap analysis
+- `proj_gemm`/`lm_head` running far below %peak → projection GEMMs not at tensor-core roofline (f32 matmul, not bf16-MMA; dequant overhead separate).
+- `ssm_scan` high % → the sequential-in-T `ssm_step_record` is the Mamba bottleneck → Milestone B: chunked/parallel SSD scan.
+- `moe_experts`/`moe_shared` high % with many calls → per-token MoE gather loop → Milestone B: Q4 batched-expert GEMM over S.
+- `host_conv` time is CPU (host-bridged) → move causal conv on-device for S-batched.
diff --git a/README.md b/README.md
index 4ef38c75..e116bb03 100644
--- a/README.md
+++ b/README.md
@@ -6,6 +6,16 @@ A minimal, dependency-light LLM inference library for Apple Silicon, built on pr
 
 **Just really smooth, fast AI on your Mac!** 🚀
 
+## Architecture
+
+Butter is a Rust + Swift inference engine spanning 35 model families, with resident decode across four GPU backends (Apple Metal, NVIDIA CUDA, AMD HIP, and Vulkan) via [iron](https://github.com/thewafflehaus/iron)'s `#[kernel]` DSL. The diagram below traces the engine stack from model loading through the per-token dispatch loop down to the shared kernel layer.
+
+![Butter architecture](docs/architecture.png)
+
+## Scope & naming
+
+Butter began under its original, Apple/Metal-focused name. It now runs across NVIDIA (GB10), AMD, and Vulkan-class GPUs via iron, so the Apple-specific original name no longer reflected the multi-backend scope — hence the rename to Butter (kernel DSL renamed to Iron).
+
 ## Status
 
 Early bootstrap — the dense-text, hybrid, vision-language, and audio model waves have all landed; end-to-end inference runs real HuggingFace checkpoints across every shipped family.
diff --git a/docs/BACKENDS.md b/docs/BACKENDS.md
new file mode 100644
index 00000000..62b4a965
--- /dev/null
+++ b/docs/BACKENDS.md
@@ -0,0 +1,68 @@
+# Adding a backend (ROCm, Vulkan, …)
+
+The engine is built so a new GPU backend needs **only two things**; everything
+above the `Device` trait — ops, models, loader, runtime, the whole verified
+model suite — comes for free.
+
+```
+              ┌──────────────────────────────────────────────┐
+  SHARED      │  wh-butter-models / wh-butter-modeltests  (model graphs) │
+  (no per-    │  wh-butter-ops        (the op seam, → Kernel IR)    │
+   backend    │  wh-butter-loader     (GGUF / SafeTensors + dequant)│
+   code)      │  wh-butter-core       (Device trait · Tensor)       │
+              └───────────────────────┬──────────────────────┘
+                                      │  the ONLY boundary
+              ┌───────────────────────▼──────────────────────┐
+  PER-BACKEND │  1. impl Device  (alloc/upload/download/        │
+  (all you    │       dispatch/synchronize)  — ~150 lines      │
+   write)     │  2. iron codegen target  (emit the        │
+              │       backend's kernel language)               │
+              └────────────────────────────────────────────────┘
+```
+
+## The two pieces
+
+**1. A `Device` impl** (`crates/backends/wh-butter-<name>/src/lib.rs`). Mirror
+`wh-butter-cuda` / `wh-butter-metal`: wrap the backend's buffers + module/PSO cache, and
+implement the five `wh_butter_core::Device` methods. `alloc`/`upload`/`download` move
+bytes; `dispatch(kernel, bindings, grid)` binds the operands and launches.
+`wh-butter-ops` only ever calls these — it never names a concrete backend.
+
+**2. A iron codegen target.** `wh-iron-codegen` already emits **MSL**
+(`src/msl/`) and **CUDA** (`src/cuda/`) from one `#[kernel]` DSL.
+- **ROCm/HIP** is the cheapest: HIP is ~source-compatible with CUDA C, so the
+  `cuda` backend is the starting point — most kernels compile under `hipcc`
+  unchanged; differences are wavefront size (64 vs 32) and a few intrinsics.
+- **Vulkan** needs an SPIR-V/GLSL compute emit (a new `src/vulkan/`).
+
+## Wiring the tests (one file, not N)
+
+The model forwards live **once** in `wh-butter-modeltests` as `verify_*(&dyn Device)`,
+collected by `run_all(dev, plat)`. Each backend has a **single** test file:
+
+```rust
+// crates/backends/wh-butter-<name>/tests/all_models.rs
+use wh_butter_<name>::<Name>Device;
+#[test]
+fn all_models() {
+    let Some(dev) = <Name>Device::create().expect("<name>") else { return };
+    wh_butter_modeltests::run_all(dev.as_ref(), "<Name> GPU");
+}
+```
+
+That's the whole verification surface. The 10-model suite (GPT-2, Pythia,
+GPT-Neo, OLMo-2, Gemma-2, Phi-1.5, StableLM-2, OLMoE, Mamba2, Falcon-H1 —
+LayerNorm/parallel-residual/post-norm/geGLU/MoE/SSM/hybrid families) runs on
+the new backend with **zero model code**, checked against the same HF oracles.
+
+## Checklist for a new backend
+
+1. `crates/backends/butter-rocm/` — `Device` impl + `unsafe impl Send/Sync` as needed.
+2. iron codegen target (HIP: fork `src/cuda/`; tune wavefront 64).
+3. `tests/all_models.rs` (copy the template above).
+4. `cargo test -p butter-rocm --test all_models` → the full suite, vs HF.
+5. Add the crate to `[workspace].members`.
+
+No changes to `wh-butter-ops`, `wh-butter-models`, `wh-butter-loader`, or any model — those are
+shared. If a kernel diverges per backend, it diverges in iron's codegen,
+not in the engine.
diff --git a/docs/DSV4_PORT.md b/docs/DSV4_PORT.md
new file mode 100644
index 00000000..b06f7889
--- /dev/null
+++ b/docs/DSV4_PORT.md
@@ -0,0 +1,100 @@
+# DeepSeek-V4-Flash → shared Rust engine — implementation spec
+
+Status: **groundwork / roadmap.** DSv4-Flash is a research-grade architecture
+with several novel subsystems; this maps the exact forward, weights, kernels,
+and config so the Rust port is a build problem, not a discovery problem.
+
+> ⚠️ The Swift reference is itself **partial**: full-attention layers work,
+> but CSA/HCA compression + the Lightning Indexer are WIP (greedy decode NaNs
+> on those layers). A fully-correct end-to-end DSv4 reference does not yet
+> exist on either side — port the full-attention path first and treat
+> CSA/HCA as a parallel research track.
+
+## Config (DSv4-Flash)
+
+hidden 4096 · layers 43 · vocab 129280 · n_heads 64 · n_kv_heads 1 ·
+head_dim 512 · qk_rope_head_dim 64 (nope 448 / rope 64) · q_lora_rank 1024 ·
+o_lora_rank 1024 · o_groups 8 · rope_theta 10000 (compress 160000) ·
+sliding_window 128 · n_routed_experts 256 · experts_per_tok 6 ·
+n_shared_experts 1 · moe_intermediate 2048 · scoring `sqrtsoftplus` ·
+routed_scaling 1.5 · swiglu_limit 10.0 · rms_norm_eps 1e-6 ·
+mHC: hc_mult 4, sinkhorn_iters 20 · **activations f32** (f16 drifts ~0.08
+logits over 43 layers).
+
+## Per-layer forward (decode)
+
+mHC sinkhorn-split → mHC collapse (4ch→1) → attn_norm → **MLA**:
+q_a → q_a_norm → q_b → per-head unit-RMS q-norm → partial-RoPE(q tail) ;
+kv → kv_a_norm → partial-RoPE(kv tail) → sliding-window cache append →
+SDPA d512 + per-head sink → inverse partial-RoPE → grouped O-LoRA (8×) →
+mHC expand. Then mHC split → collapse → ffn_norm → **DSv4-MoE**
+(sqrtsoftplus router + bias → top-6 → per-expert clamped-SwiGLU + 1 shared)
+→ mHC expand.
+
+## iron kernels (exist, pass the CUDA corpus)
+
+`iron_dsv4_partial_rope` (qk,out,head_dim,n_nope,half_rot,position,theta,
+inverse,freq_scale,ext,corr_low,corr_high) · `iron_dsv4_indexer_score` ·
+`iron_dsv4_indexer_topk_block` · `iron_dsv4_mhc_sinkhorn_split` ·
+`iron_dsv4_mhc_collapse` · `iron_dsv4_mhc_expand` · `iron_dsv4_compressor_pool` ·
+`iron_sdpa_decode_d512_sink` (reused MLX) · `iron_moe_router_sqrtsoftplus` ·
+`iron_dsv4_swiglu_limit` · dequant: `iron_dsv4_mxfp4_dequant`,
+`iron_dsv4_fp8_block_dequant`.
+
+## GGUF weights (blk.N.*) + quant
+
+q_a/q_b/kv/output_a/output_b/shexp: **q8_0** · gate_exps/up_exps: **iq2_xxs**
+· down_exps: **q2_K** · ffn_gate_inp/mHC/compressor/indexer: **f16** ·
+norms/sinks/scales/biases: **f32**. token_embd f16, output q8_0.
+(See Swift `Models/Text/DeepSeekV4Text.swift` loader + `Loader/GGUF/`.)
+
+## Build order (each its own focused session)
+
+1. **GGUF v3 loader + dequant** (q8_0, q2_K, iq2_xxs, f16, f32) — gate for everything.
+2. **MLA attention builder** — partial-RoPE + sink-SDPA ops wired into wh-butter-ops; validate the op-composition vs CPU on both platforms (kernels already pass the corpus).
+3. **DSv4-MoE** — sqrtsoftplus router + clamped-SwiGLU + shared expert (extends the verified `wh_butter_models::moe`).
+4. **mHC** — sinkhorn-split / collapse / expand (the 4-channel residual).
+5. **Full-attention layers end-to-end** on CUDA, diff vs Swift full-attn output.
+6. **CSA/HCA + indexer** — research track; reference is WIP.
+
+## Real checkpoint — reverse-engineered layout (2026-06, IQ2XXS 86 GB, CUDA box)
+
+GGUF dequant verified vs gguf-py on the real checkpoint **on the GB10 box**
+(43 blocks, 1328 tensors): Q8_0 Δ4.0e-7, Q2_K Δ4.9e-7, IQ2_XXS Δ4.7e-7.
+Composites (MLA/MoE/mHC/swiglu_limit/router) all pass on CUDA vs CPU.
+
+**Config (`deepseek4.*` metadata):** hidden 4096 · 64 heads · head_dim
+(key/value_length) 512 · q_lora 1024 · output_lora 1024 · output_groups 8 ·
+rope_dim 64 (⇒ n_nope 448, half_rot 32) · eps 1e-6 · experts 256 (used 6,
+shared 1, ffn 2048, weights_scale 1.5, norm true) · swiglu_clamp 10 · mHC
+count 4, sinkhorn_iters 20.
+
+**blk.N tensor → struct map** (ggml `[in,out]` dims = row-major `[out,in]` =
+struct layout, no transpose):
+
+| GGUF tensor | dims | → struct field |
+|---|---|---|
+| `attn_norm.weight` | [4096] | `MlaWeights.attn_norm` |
+| `attn_q_a.weight` | [4096,1024] | `q_a` [1024,4096] |
+| `attn_q_a_norm.weight` | [1024] | `q_a_norm` |
+| `attn_q_b.weight` | [1024,32768] | `q_b` [32768,1024] |
+| `attn_kv.weight` | [4096,512] | `kv` [512,4096] |
+| `attn_kv_a_norm.weight` | [512] | `kv_a_norm` |
+| `attn_sinks.weight` | [64] | `sink` (per-head) |
+| `attn_output_a.weight` | [4096,8192] | `output_a` = 8 groups × [1024,4096] |
+| `attn_output_b.weight` | [8192,4096] | `output_b` [4096,8192] |
+| `hc_attn_fn.weight` | [16384,24] | `MhcWeights.hc_fn` [24,16384] |
+| `hc_attn_base.weight` | [24] | `hc_base` |
+| `hc_attn_scale.weight` | [3] | `hc_scale` (pre/post/comb — a tensor, not a const) |
+| `ffn_gate_inp.weight` | [4096,256] | MoE router |
+| `ffn_{gate,up}_exps.weight` | [4096,2048,256] | 256 experts (IQ2_XXS) |
+| `ffn_down_exps.weight` | [2048,4096,256] | 256 experts (Q2_K) |
+| `ffn_{gate,up,down}_shexp.weight` | … | shared expert |
+| `ffn_gate_tid2eid.weight` | [6,129280] | hash-routing table |
+
+**Full-forward blocker (now named precisely):** the `attention.indexer`
+(head_count 64, key_length 128, top_k 512) + `sliding_window 128` — the CSA
+sparse-attention indexer, WIP in the reference itself ⇒ no correct oracle for
+the full 43-layer forward. Everything *else* (dense MLA + mHC + MoE per layer)
+is op-verified; a real-weight single-layer forward (GPU-vs-CPU) is turnkey from
+the map above.
diff --git a/docs/PERF.md b/docs/PERF.md
new file mode 100644
index 00000000..0892bf02
--- /dev/null
+++ b/docs/PERF.md
@@ -0,0 +1,86 @@
+# BUTTER performance — status, numbers, and the #1 bottleneck
+
+## Current throughput (GPT-2-124M, incremental KV cache)
+
+| platform | prefill | decode | one-time | output |
+|---|---|---|---|---|
+| CUDA (GB10 sm_121) | 52.5 tok/s | 24.6 tok/s (41 ms/tok) | 16s upload + 42s JIT | exact HF match |
+| Metal — resident bufs | 24 tok/s | **17.6 tok/s (57 ms/tok)** | 11s upload + 32s JIT | exact HF match |
+| Metal — old shim | 11 tok/s | 9.4 tok/s (107 ms/tok) | (re-uploaded weights/dispatch) | exact HF match |
+
+## Per-dispatch micro-bench — Rust vs Swift, same Apple GPU (settles the FFI-overhead question)
+
+gemv (2048×2048), per-commit = one command buffer per op + wait:
+
+| path | per-dispatch | batched (N ops / 1 cmd buffer) |
+|---|---|---|
+| Rust → Metal (resident weights) | **178 µs** | (via `dispatch_chain`) |
+| Rust → Metal (old host-shadow shim) | 1868 µs | — |
+| Swift → Metal (native `Ops.gemv`) | **170 µs** | **22 µs/op** |
+
+**Rust and Swift per-dispatch are within ~4%** — both pay the same Metal
+`commit()`+wait, so the host language is not the variable (there's no Swift→Rust
+FFI in the inference path anyway). The earlier 10× Rust gap was purely the
+host-shadow shim re-uploading weights every dispatch, now fixed.
+
+The real lever is **batching**: Swift's 22 µs/op (7.6× faster) comes from
+encoding many ops into one command buffer (its `Ops.gemv(on: cmd)` API). Rust
+gets the same via `dispatch_chain` — the Device trait currently commits per op;
+batching a whole layer into one command buffer is the next win, identical in
+kind to what Swift does. tiny add: 188 µs (shim) → 161 µs (resident).
+
+## Fixed: resident-buffer fast path (was a host-shadow copy-in/out shim)
+
+`wh-butter-metal` now caches a GPU-resident `ResidentBuffer` on each `MetalBuffer`
+(via the runtime's `upload_resident`), so pure inputs (weights) upload **once**
+and bind resident across every dispatch instead of being re-staged from a host
+shadow each call. Outputs still flow through host bytes (preserves in-place
+reads + readback). Verified correct across elementwise / reduction / multi-output
+/ in-place / MLA-composite kernels and all real-model forwards. Result: gemv 10×,
+decode 1.9× (107→57 ms/tok).
+
+### Original root cause (for the record)
+
+`crates/backends/wh-butter-metal/src/lib.rs::dispatch` (and `synchronize`, which is a
+no-op "dispatch is synchronous") does, **per op**:
+
+1. read **every input operand's bytes from a host shadow** (`shadow(buf).data.read().clone()`),
+2. hand them to `Context::dispatch_with_grid` as a `BTreeMap<String, Vec<u8>>` (re-uploaded),
+3. run + wait,
+4. copy each output back into the host shadow.
+
+There are **no persistent device buffers**. Every dispatch re-stages all
+operands through host memory and synchronizes. A token = ~120 ops, so weights
+and activations cross the bus ~120×/token, each a commit+wait. That is the
+107 ms, not compute and not host round-trips in the *test* (those barely
+mattered — cutting them 3× changed nothing).
+
+This also explains why "device-resident weights" gave 26× earlier: that win was
+from **not recomputing host-side weight transposes** (`conv_t`) every step, not
+from any actual device residency — the backend re-uploads regardless.
+
+The CUDA backend already wraps real device buffers (`CudaDevice` alloc/htod/dtoh),
+which is why GB10 is ~2.6× faster and has more headroom.
+
+## The fix (highest-leverage perf item)
+
+Make `MetalDevice` hold **persistent `MTLBuffer`s** keyed by the `DeviceBuffer`
+handle, and a `dispatch` that **binds device buffers by handle** instead of
+re-uploading bytes — upload once on `alloc`/`upload`, never copy back unless
+`download` is called. Requires the `wh_iron_runtime::Context` Metal face to
+accept pre-resident buffer handles (a small Context API addition) rather than a
+bytes map. Then:
+
+- weights upload once, stay resident; activations stay resident across layers;
+- `synchronize` becomes a real fence; dispatches enqueue into one command buffer
+  per token and sync once (kills the per-op commit+wait).
+
+Expected: the dispatch-bound 42× overhead largely collapses; F16 weights then
+become worthwhile (halves the resident footprint + the now-dominant bandwidth).
+
+## Secondary perf items (after the above)
+
+- Cache the compiled metallib / cubin to kill the 32–42 s one-time JIT.
+- Fuse the decode path (fewer dispatches) — esp. QKV proj + the SwiGLU/GELU MLP.
+- Device-side KV cache (append + attend without host reorg/re-upload).
+- Batched decode (batch > 1).
diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md
new file mode 100644
index 00000000..03d4875a
--- /dev/null
+++ b/docs/VERIFICATION.md
@@ -0,0 +1,95 @@
+# BUTTER shared-engine verification matrix
+
+One Rust codebase (`wh-butter-models` + `wh-butter-ops`) over the `wh-butter-core::Device`
+trait, run on each backend, **argmax checked against HF transformers** for a
+single-token forward (`forward_single` — token attends to itself at pos 0,
+which is exactly HF's 1-token forward). The multi-token KV-cache decode loop
+is the runtime's job; this validates the model graph + every op.
+
+## Dense transformer-LLM family — one builder, auto-config (`load_hf`)
+
+`load_hf` reads HF `config.json` for geometry and detects arch flags from the
+tensor names (qk-norm by `q_norm.weight`, QKV bias by `q_proj.bias`). No
+per-model code.
+
+| model | arch detail | CUDA (GB10 / sm_121) | Metal (Apple GPU) | HF argmax (tok 9707) |
+|---|---|:---:|:---:|:---:|
+| Qwen3-0.6B    | qk-norm, tied, hd128       | ✅ 21806 | ✅ 21806 | 21806 |
+| Qwen2.5-0.5B  | QKV bias, tied, hd64       | ✅ 11    | ✅ 11    | 11 |
+| SmolLM2-135M  | Llama-arch, tied, hidden 576 | ✅ 28  | ✅ 28    | 28 |
+
+Three distinct architectures (qk-norm / QKV-bias / plain-Llama) and a
+non-128-multiple hidden (576, handled by the strided `mt_rms_norm_wide`
+fallback) — all auto-loaded by `load_hf`, identical argmax on CUDA, Metal,
+and HF. Same code path covers the rest of the family as weights are staged:
+**Llama 3.x, Mistral, Yi, Phi, Starcoder2, OLMo, InternLM2, Granite3, …** —
+dense Llama-style transformers differing only by config + the qk-norm/bias
+flags `load_hf` already detects.
+
+## Architecture-path coverage — distinct non-Llama mechanisms (host-orchestrated, vs HF argmax/top-k)
+
+Each model below was chosen to exercise a *mechanism* the dense `load_hf` path
+doesn't, on the shared op layer. All single-token forwards, argmax/top-k vs HF.
+
+| model | mechanism exercised | CUDA | Metal | HF |
+|---|---|:---:|:---:|:---:|
+| GPT-2 124M | LayerNorm-LLM, learned-pos, Conv1D (transposed) weights, gelu_new, tied | ✅ 198 | ✅ 198 | 198 |
+| Pythia-160m | GPT-NeoX: parallel residual, interleaved per-head QKV, partial rotary | ✅ 285 | ✅ 285 | 285 |
+| Gemma-2-2b | √hidden embed-scale, RMSNorm(1+w), 4 norms/layer, geGLU, GQA hd256, softcaps | ✅ top3 | ✅ top3 | [9707,235265,110] |
+| Phi-1.5 | single shared norm → parallel attn+MLP, separate q/k/v+bias, partial rotary | ✅ top3 | ✅ top3 | [11,13,546] |
+| OLMo-2-1B | **post-norm** (norm on sublayer output) + qk-norm over full proj, SwiGLU | ✅ top3 | ✅ top3 | [198,8,13] |
+| StableLM-2-1.6B | LayerNorm(+bias), q/k/v bias, partial rotary, SwiGLU | ✅ top3 | ✅ top3 | [341,11,280] |
+| GPT-Neo-125M | learned-pos + LayerNorm, separate q/k/v (no bias), no attn scaling, tied | ✅ top3 | ✅ top3 | [28,59,91] |
+
+(`load_hf` already covers qk-norm / QKV-bias / plain-Llama / GQA via Qwen3·Qwen2.5·SmolLM2.)
+
+## Prefill primitive — multi-token causal forward (vs HF argmax/top-k)
+
+The model tests above are single-token (pos 0). These verify the **multi-token
+causal prefill** path — each position attends over [0, i] via `sdpa_decode`
+with `n_kv = i+1` against the per-head K/V cache (`kv_stride = seq_len`).
+
+| test | what it adds | CUDA | Metal | HF |
+|---|---|:---:|:---:|:---:|
+| GPT-2 prefill (8 tok) | causal masking + learned positions | ✅ | ✅ | top3 [11,21831,7586] |
+| Llama prefill (7 tok) | **RoPE-at-position** + GQA + SwiGLU (SmolVLM text model) | ⏳ | ✅ | top3 [12642,4052,216] |
+
+### Throughput (correctness-first, not yet optimized) — GPT-2-124M, incremental KV cache + device-resident weights
+| platform | prefill | decode | one-time (weight upload / kernel JIT) | output |
+|---|---|---|---|---|
+| CUDA (GB10) | 52.5 tok/s | **24.6 tok/s** (41 ms/tok) | 16.1s / 42.4s | exact HF match |
+| Metal (Apple GPU) | 11 tok/s | **9.4 tok/s** (107 ms/tok) | 10.7s / 31.5s | exact HF match |
+
+Resident weights gave a 26× decode speedup over re-uploading per step. Remaining
+overhead is per-layer host round-trips + KV re-upload (device-resident activations
++ device KV cache is the next win) and F32-only compute. Not production tok/s yet.
+
+With these, every primitive for end-to-end generation + multimodal prefill is
+verified. A full VLM forward = [SigLIP tower] → [SmolVLM connector] → splice
+image embeds into the text sequence → [this Llama causal prefill] — each piece
+independently confirmed vs HF on the shared op layer.
+
+## Exotic families — dedicated builders / ops (in progress)
+
+| family | needs | status |
+|---|---|---|
+| MoE — OLMoE / Qwen2-MoE / GPT-OSS / Granite4 | router → top-k → per-expert SwiGLU (+ optional shared expert) | ✅ **full real OLMoE-1B-7B verified vs HF both platforms** (argmax 310 on Metal + CUDA) — 16-layer, 64-expert, top-8, no-renorm (`norm_topk_prob=false`), no shared expert, qk-norm over the *full* 2048 projection, MHA hd128, sharded BF16 checkpoint via the mmap sharded loader. Plus the real Qwen2-MoE block (sigmoid-gated shared expert, max\|Δ\|4.6e-7 both platforms). |
+| DeepSeek-V4 (MLA + DSv4-MoE + mHC) | full novel arch | ✅ **entire compute path + GGUF loader verified** (MLA/MoE/mHC composites both platforms vs CPU; F32/F16/Q8_0/Q2_K/IQ2_XXS dequant vs gguf-py on the 81GB checkpoint). Full 43-layer forward blocked on CSA/HCA sparse-attention (WIP in the reference itself — no correct oracle). |
+| SSM — Mamba2, Jamba, LFM2 | conv1d + SSD selective-scan + gated RMSNorm | ✅ **full real Mamba2-130m verified vs HF both platforms** (argmax 310 on CUDA + Metal) — 24-layer forward: in_proj → conv1d+silu → SSD scan → gated RMSNorm → out_proj, on the shared op layer. |
+| Hybrid SSM+attention — **Falcon-H1**, Jamba, Zamba | per-layer Mamba2 mixer ∥ attention, µP multipliers | ✅ **full real Falcon-H1-0.5B verified vs HF both platforms** (top-3 [593,531,587] on CUDA + Metal) — each of 36 layers runs the Mamba2 mixer AND GQA attention in parallel off one shared norm, summed into the residual, then SwiGLU FFN; all µP multipliers handled exactly. Reuses `conv1d_step`+`ssm_step` and `sdpa_decode` together in one model. |
+| VLM — SigLIP/CLIP towers · Pixtral, SmolVLM2, FastVLM, Idefics3, MiniCPMV | vision tower + projector + the LLM builder | ✅ **full real SigLIP-base vision tower verified vs HF both platforms** (last_hidden_state max\|Δ\| 9e-5 / sum −792.79 vs HF −792.795, Metal + CUDA) — 12-layer bidirectional ViT: patch-embed (conv-as-matmul) + position-embed + LayerNorm + full self-attention + GELU(tanh)-MLP, via the new `matmul`/`layer_norm`/`gelu` ops (all unit-checked vs CPU both platforms). The LLM half is the dense-Llama family above; a VLM = this tower → connector → that LLM. **SmolVLM (Idefics3) connector now verified vs HF both platforms** (pixel-shuffle gathering a 4×4 patch block into each token's channels + modality projection; out max\|Δ\| 5e-4 / sum 79.85 vs HF 79.852, Metal+CUDA). All three pieces of a VLM (SigLIP tower, connector, dense-Llama LLM) are independently verified; the remaining work is only the end-to-end token-splice plumbing. |
+| Audio — Whisper (enc **+ dec**) · Parakeet, Voxtral, StyleTTS2, … | encoder/decoder + audio front-end + cross-attention | ✅ **full real Whisper encoder AND encoder→decoder STT verified vs HF both platforms.** Encoder (whisper-tiny): last_hidden_state max\|Δ\| 2e-5 — conv front-end (Conv1d as im2col+`matmul`) + sinusoidal pos + bidirectional transformer. **Full STT (whisper-base): argmax 50362 = HF on Metal+CUDA** — adds the **cross-attention** path (decoder Q over the encoder's 1500 K/V states, per layer) + causal self-attn + tied lm_head. Cross-attention is the mechanism every encoder-decoder + many VLMs share. |
+
+## How to verify a model
+
+```sh
+# Metal (Mac)
+MODEL_DIR=/path/to/hf_model TOK=9707 EXPECT=<hf_argmax> \
+  cargo test -p wh-butter-metal --test hf_model -- --nocapture
+# CUDA (GB10)
+MODEL_DIR=/path/to/hf_model TOK=9707 EXPECT=<hf_argmax> \
+  cargo test -p wh-butter-cuda --features cuda --test hf_model -- --nocapture
+```
+
+`EXPECT` is HF transformers' argmax for the same single input token — the
+oracle. Swift BUTTER is itself HF-validated, so matching HF ≡ matching Swift.
diff --git a/docs/architecture.png b/docs/architecture.png
new file mode 100644
index 00000000..151f1e0c
Binary files /dev/null and b/docs/architecture.png differ
diff --git a/docs/butter-before-after.png b/docs/butter-before-after.png
new file mode 100644
index 00000000..fd21cca9
Binary files /dev/null and b/docs/butter-before-after.png differ
diff --git a/docs/butter-before-after.svg b/docs/butter-before-after.svg
new file mode 100644
index 00000000..b6d7326c
--- /dev/null
+++ b/docs/butter-before-after.svg
@@ -0,0 +1,156 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1680" height="1120" viewBox="0 0 1680 1120" font-family="-apple-system, Helvetica, Arial, sans-serif">
+  <defs>
+    <style>
+      .title  { font-size: 34px; font-weight: 700; fill: #0f172a; }
+      .sub    { font-size: 16px; fill: #64748b; }
+      .panel  { font-size: 24px; font-weight: 700; fill: #0f172a; }
+      .lbl    { font-size: 16px; font-weight: 600; fill: #0f172a; }
+      .small  { font-size: 13.5px; fill: #334155; }
+      .tiny   { font-size: 12px; fill: #64748b; }
+      .badge  { font-size: 12px; font-weight: 700; }
+      .arr    { stroke: #94a3b8; stroke-width: 2.2; fill: none; marker-end: url(#ah); }
+    </style>
+    <marker id="ah" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto">
+      <path d="M0,0 L7,3 L0,6 Z" fill="#94a3b8"/>
+    </marker>
+  </defs>
+
+  <rect width="1680" height="1120" fill="#f8fafc"/>
+  <text x="60" y="58" class="title">BUTTER — F*cking Fast AI</text>
+  <text x="60" y="84" class="sub">before: Swift, Apple-only  ·  after: one product, two engines, every GPU — shared kernels</text>
+
+  <!-- divider -->
+  <line x1="840" y1="110" x2="840" y2="1080" stroke="#cbd5e1" stroke-width="2" stroke-dasharray="6 6"/>
+
+  <!-- ============================ BEFORE ============================ -->
+  <text x="60" y="150" class="panel">BEFORE</text>
+  <text x="60" y="174" class="sub">Swift inference library, native Metal</text>
+
+  <!-- swift engine box -->
+  <rect x="60" y="196" width="700" height="470" rx="14" fill="#eff6ff" stroke="#60a5fa" stroke-width="2"/>
+  <text x="80" y="226" class="lbl" fill="#1d4ed8">BUTTER (Swift)</text>
+
+  <rect x="84" y="244" width="652" height="78" rx="9" fill="#ffffff" stroke="#bfdbfe"/>
+  <text x="100" y="270" class="lbl">~35 model families  ·  Vision  ·  Audio</text>
+  <text x="100" y="292" class="small">Llama · Qwen · Gemma · Nemotron · DeepSeek-V4 · Mamba · MoE …</text>
+  <text x="100" y="311" class="tiny">forward passes, written in Swift</text>
+
+  <rect x="84" y="334" width="652" height="96" rx="9" fill="#ffffff" stroke="#bfdbfe"/>
+  <text x="100" y="360" class="lbl">Swift runtime</text>
+  <text x="100" y="383" class="small">Device.swift · BufferPool · Ops · KVCache · Generation · Sampler</text>
+  <text x="100" y="405" class="small">Loaders: GGUF / SafeTensors / HF</text>
+  <text x="100" y="424" class="tiny">all Swift, all Apple</text>
+
+  <rect x="84" y="442" width="652" height="64" rx="9" fill="#ffffff" stroke="#bfdbfe"/>
+  <text x="100" y="468" class="lbl">IronSwift</text>
+  <text x="100" y="490" class="small">pre-compiled kernels.metallib (AOT)  ·  PSOCache</text>
+
+  <!-- metal -->
+  <rect x="84" y="528" width="652" height="56" rx="9" fill="#f1f5f9" stroke="#cbd5e1"/>
+  <text x="100" y="562" class="lbl">Metal → Apple GPU</text>
+
+  <rect x="84" y="600" width="652" height="48" rx="9" fill="#f1f5f9" stroke="#cbd5e1"/>
+  <text x="100" y="630" class="small" font-weight="600">iPhone · iPad · Mac  —  and nothing else</text>
+
+  <!-- iron feeds it -->
+  <rect x="60" y="690" width="700" height="64" rx="12" fill="#ecfdf5" stroke="#34d399" stroke-width="2"/>
+  <text x="80" y="716" class="lbl" fill="#047857">iron (Rust kernel toolchain)</text>
+  <text x="80" y="738" class="small">one IR → MSL → kernels.metallib</text>
+  <path d="M410,690 L410,650" class="arr"/>
+  <text x="424" y="676" class="tiny">tile build --emit</text>
+
+  <rect x="60" y="800" width="700" height="120" rx="12" fill="#fff7ed" stroke="#fb923c" stroke-width="1.6"/>
+  <text x="80" y="828" class="lbl" fill="#c2410c">The ceiling</text>
+  <text x="80" y="854" class="small">• "Metal" + "Apple" baked into the name and the code.</text>
+  <text x="80" y="877" class="small">• No path to NVIDIA / AMD / portable GPUs.</text>
+  <text x="80" y="900" class="small">• Want CUDA? Rewrite the whole model surface from scratch.</text>
+
+  <!-- ============================ AFTER ============================ -->
+  <text x="880" y="150" class="panel">AFTER</text>
+  <text x="880" y="174" class="sub">two engines, one product, shared kernels</text>
+
+  <!-- SWIFT engine (kept) -->
+  <rect x="880" y="196" width="368" height="300" rx="14" fill="#eff6ff" stroke="#60a5fa" stroke-width="2"/>
+  <text x="900" y="226" class="lbl" fill="#1d4ed8">BUTTER (Swift) — unchanged</text>
+  <rect x="755" y="200" width="118" height="22" rx="11" fill="#1d4ed8"/>
+  <text x="763" y="216" class="badge" fill="#ffffff">PRIMARY · APPLE</text>
+  <text x="900" y="252" class="small">~35 models · Vision · Audio · DSv4</text>
+  <text x="900" y="274" class="small">Swift runtime + loaders</text>
+  <text x="900" y="296" class="small">IronSwift (AOT metallib)</text>
+  <rect x="900" y="312" width="328" height="40" rx="8" fill="#f1f5f9" stroke="#cbd5e1"/>
+  <text x="916" y="338" class="small" font-weight="600">Metal → iPhone · iPad · Mac</text>
+  <text x="900" y="384" class="lbl">still first-class · ships today</text>
+  <text x="900" y="408" class="tiny">tests pass · DeepSeek-V4 runs (same as before)</text>
+  <text x="900" y="452" class="tiny" fill="#1d4ed8">can ALSO call the shared Rust layer ↓ (wh-butter-ffi)</text>
+  <text x="900" y="472" class="tiny">via C-ABI — proven: Swift links libwh_butter_ffi</text>
+
+  <!-- RUST engine (new) -->
+  <rect x="1268" y="196" width="352" height="300" rx="14" fill="#fff7ed" stroke="#fb923c" stroke-width="2"/>
+  <text x="1288" y="226" class="lbl" fill="#c2410c">BUTTER (Rust) — new</text>
+  <rect x="1500" y="200" width="112" height="22" rx="11" fill="#c2410c"/>
+  <text x="1508" y="216" class="badge" fill="#ffffff">CROSS-PLATFORM</text>
+  <text x="1288" y="252" class="small">wh-butter-models / loader / runtime</text>
+  <text x="1288" y="272" class="tiny">backend-neutral (porting in progress)</text>
+  <rect x="1288" y="286" width="312" height="34" rx="8" fill="#ffffff" stroke="#fed7aa"/>
+  <text x="1304" y="308" class="small" font-weight="600">wh-butter-ops — the op seam (real)</text>
+  <rect x="1288" y="328" width="312" height="34" rx="8" fill="#fde68a" stroke="#f59e0b"/>
+  <text x="1304" y="350" class="lbl">wh-butter-core: one Device trait</text>
+  <text x="1288" y="384" class="tiny">add/mul proven on GB10 through the trait</text>
+  <text x="1288" y="406" class="tiny">wh-butter-ffi (C-ABI) → Swift &amp; any C caller</text>
+  <text x="1288" y="428" class="tiny">wh-butter-cli enumerates live devices</text>
+
+  <!-- shared seam line -->
+  <path d="M1064,496 L1064,540" class="arr"/>
+  <path d="M1444,496 L1444,540" class="arr"/>
+
+  <!-- backends row -->
+  <rect x="880" y="542" width="740" height="120" rx="12" fill="#fffbeb" stroke="#f59e0b" stroke-width="1.6"/>
+  <text x="900" y="568" class="lbl" fill="#92400e">Backends — one crate each, behind the Device trait</text>
+
+  <rect x="900" y="582" width="168" height="64" rx="9" fill="#dcfce7" stroke="#22c55e" stroke-width="2"/>
+  <text x="916" y="606" class="lbl" fill="#15803d">wh-butter-cuda</text>
+  <text x="916" y="626" class="small">REAL · GB10 sm_121</text>
+  <text x="916" y="642" class="tiny">NVRTC→PTX→driver</text>
+
+  <rect x="1086" y="582" width="168" height="64" rx="9" fill="#ffffff" stroke="#cbd5e1"/>
+  <text x="1102" y="606" class="lbl">wh-butter-metal</text>
+  <text x="1102" y="626" class="tiny">stub (metal-rs)</text>
+  <text x="1102" y="642" class="tiny">Swift is primary on Apple</text>
+
+  <rect x="1272" y="582" width="158" height="64" rx="9" fill="#ffffff" stroke="#cbd5e1"/>
+  <text x="1288" y="606" class="lbl">wh-butter-vulkan</text>
+  <text x="1288" y="626" class="tiny">stub (SPIR-V)</text>
+  <text x="1288" y="642" class="tiny">portable GPUs</text>
+
+  <rect x="1448" y="582" width="152" height="64" rx="9" fill="#ffffff" stroke="#e2e8f0" stroke-dasharray="4 3"/>
+  <text x="1464" y="606" class="lbl" fill="#94a3b8">+ ROCm …</text>
+  <text x="1464" y="626" class="tiny">one crate = one backend</text>
+
+  <!-- hardware -->
+  <rect x="900" y="582" width="0" height="0"/>
+  <text x="900" y="690" class="small" font-weight="600">NVIDIA</text>
+  <text x="1102" y="690" class="small" font-weight="600">Apple</text>
+  <text x="1288" y="690" class="small" font-weight="600">AMD / Intel / Android</text>
+  <path d="M984,646 L984,676" class="arr"/>
+  <path d="M1170,646 L1170,676" class="arr"/>
+  <path d="M1351,646 L1351,676" class="arr"/>
+
+  <!-- shared kernels feed BOTH engines -->
+  <rect x="880" y="720" width="740" height="92" rx="12" fill="#ecfdf5" stroke="#34d399" stroke-width="2.4"/>
+  <text x="900" y="748" class="lbl" fill="#047857">iron — shared kernel toolchain (one IR)</text>
+  <text x="900" y="772" class="small">→ MSL (.metallib) for Swift     →  CUDA C++ / PTX for Rust     →  SPIR-V (Vulkan, next)</text>
+  <text x="900" y="794" class="tiny">4164/4164 kernels bit-accurate on GB10 · same kernels both engines run</text>
+  <path d="M1064,720 L1064,668" class="arr"/>
+  <path d="M1240,720 L1240,496" class="arr" stroke-dasharray="5 4"/>
+  <text x="1250" y="610" class="tiny" fill="#047857" transform="rotate(0)">same IR</text>
+
+  <!-- payoff -->
+  <rect x="880" y="836" width="740" height="84" rx="12" fill="#f0fdf4" stroke="#86efac" stroke-width="1.6"/>
+  <text x="900" y="864" class="lbl" fill="#166534">The unlock</text>
+  <text x="900" y="889" class="small">• Swift keeps shipping to iPhone — untouched, primary.</text>
+  <text x="900" y="910" class="small">• Same engine now runs NVIDIA today, AMD/Vulkan next — add a backend, not a rewrite.</text>
+
+  <!-- footer -->
+  <text x="60" y="1040" class="tiny">Proven: Swift suite passes (parity) · DeepSeek-V4 runs on Swift · CUDA + Swift both consume the shared wh-butter-core/wh-butter-ops layer · live GB10 through the Device trait.</text>
+  <text x="60" y="1062" class="tiny">github.com/TheTom/butter (private) — Swift at repo root, Rust engine under rust/, iron shared.</text>
+</svg>
diff --git a/docs/butter-shared-models.png b/docs/butter-shared-models.png
new file mode 100644
index 00000000..8d9990bd
Binary files /dev/null and b/docs/butter-shared-models.png differ
diff --git a/docs/butter-shared-models.svg b/docs/butter-shared-models.svg
new file mode 100644
index 00000000..93ae00b0
--- /dev/null
+++ b/docs/butter-shared-models.svg
@@ -0,0 +1,127 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1500" height="1180" viewBox="0 0 1500 1180" font-family="-apple-system, Helvetica, Arial, sans-serif">
+  <defs>
+    <style>
+      .title { font-size: 32px; font-weight: 700; fill: #0f172a; }
+      .sub   { font-size: 16px; fill: #64748b; }
+      .h     { font-size: 21px; font-weight: 700; fill: #0f172a; }
+      .lbl   { font-size: 16px; font-weight: 600; fill: #0f172a; }
+      .small { font-size: 13.5px; fill: #334155; }
+      .tiny  { font-size: 12px; fill: #64748b; }
+      .badge { font-size: 12px; font-weight: 700; fill:#fff; }
+      .arr   { stroke: #94a3b8; stroke-width: 2.4; fill: none; marker-end: url(#ah); }
+      .arrG  { stroke: #34d399; stroke-width: 2.6; fill: none; marker-end: url(#ahg); }
+    </style>
+    <marker id="ah" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#94a3b8"/></marker>
+    <marker id="ahg" markerWidth="10" markerHeight="10" refX="7" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#34d399"/></marker>
+  </defs>
+
+  <rect width="1500" height="1180" fill="#f8fafc"/>
+  <text x="56" y="56" class="title">BUTTER — models as a shared layer (target)</text>
+  <text x="56" y="82" class="sub">write each model ONCE (Rust) · run on every backend · Swift app calls in · zero per-platform duplication</text>
+
+  <!-- ===== Swift app shell (top-left) ===== -->
+  <rect x="56" y="118" width="430" height="150" rx="14" fill="#eff6ff" stroke="#60a5fa" stroke-width="2"/>
+  <text x="76" y="148" class="h" fill="#1d4ed8">Swift app (Apple)</text>
+  <text x="76" y="176" class="small">SwiftUI · AVFoundation capture · CoreML</text>
+  <text x="76" y="198" class="small">iPhone · iPad · Mac packaging</text>
+  <text x="76" y="226" class="small" font-weight="600">calls the shared engine via wh-butter-ffi (C-ABI)</text>
+  <text x="76" y="248" class="tiny">native Apple bits stay Swift; inference goes to Rust</text>
+
+  <!-- oracle note -->
+  <rect x="516" y="118" width="430" height="150" rx="14" fill="#f8fafc" stroke="#cbd5e1" stroke-width="1.6" stroke-dasharray="5 4"/>
+  <text x="536" y="148" class="h" fill="#475569">Swift models = the oracle</text>
+  <text x="536" y="176" class="small">the existing ~35 Swift forward passes</text>
+  <text x="536" y="198" class="small">validate each Rust port bit-for-bit,</text>
+  <text x="536" y="220" class="small">then retire for inference.</text>
+  <text x="536" y="248" class="tiny">no throwaway work — they're the ground truth</text>
+
+  <!-- arrow swift -> models -->
+  <path d="M271,268 L271,300" class="arr"/>
+  <path d="M731,268 C731,290 530,290 510,300" class="arr" stroke-dasharray="5 4"/>
+  <text x="690" y="292" class="tiny" fill="#94a3b8">diff vs</text>
+
+  <!-- ===== SHARED: models ===== -->
+  <rect x="56" y="300" width="1388" height="150" rx="14" fill="#ecfdf5" stroke="#10b981" stroke-width="2.6"/>
+  <rect x="1296" y="306" width="140" height="22" rx="11" fill="#059669"/>
+  <text x="1312" y="322" class="badge">WRITTEN ONCE</text>
+  <text x="76" y="332" class="h" fill="#047857">wh-butter-models (Rust) — one codebase</text>
+  <text x="76" y="362" class="small">35 families collapse to ~6–10 block-builders + per-model config (like ggml):</text>
+  <text x="76" y="386" class="small" font-weight="600">transformer-LLM builder</text>
+  <text x="76" y="406" class="tiny">Llama · Qwen · Gemma · Mistral · Yi · Phi · SmolLM · Starcoder · OLMo …</text>
+  <rect x="520" y="372" width="270" height="46" rx="8" fill="#ffffff" stroke="#a7f3d0"/>
+  <text x="534" y="392" class="small" font-weight="600">MoE / MLA builder</text>
+  <text x="534" y="410" class="tiny">DeepSeek-V4 · GPT-OSS · Granite4</text>
+  <rect x="800" y="372" width="250" height="46" rx="8" fill="#ffffff" stroke="#a7f3d0"/>
+  <text x="814" y="392" class="small" font-weight="600">SSM builder</text>
+  <text x="814" y="410" class="tiny">Mamba2 · Jamba · FalconH1 · LFM2</text>
+  <rect x="1060" y="372" width="300" height="46" rx="8" fill="#ffffff" stroke="#a7f3d0"/>
+  <text x="1074" y="392" class="small" font-weight="600">VLM / Audio builders</text>
+  <text x="1074" y="410" class="tiny">Pixtral · SmolVLM2 · FastVLM · TTS/STT</text>
+
+  <path d="M750,450 L750,486" class="arrG"/>
+
+  <!-- ===== SHARED: ops seam ===== -->
+  <rect x="56" y="486" width="1388" height="66" rx="12" fill="#d1fae5" stroke="#10b981" stroke-width="2"/>
+  <text x="76" y="514" class="h" fill="#047857">wh-butter-ops — the op seam</text>
+  <text x="76" y="538" class="small">rms_norm · matmul · attention · rope · softmax · sampling  →  builds iron Kernel IR</text>
+  <text x="1180" y="525" class="tiny" fill="#047857">add/mul real today;</text>
+  <text x="1180" y="542" class="tiny" fill="#047857">heavy ops = next step</text>
+
+  <path d="M750,552 L750,588" class="arrG"/>
+
+  <!-- ===== SHARED: device trait ===== -->
+  <rect x="56" y="588" width="1388" height="58" rx="12" fill="#fde68a" stroke="#f59e0b" stroke-width="2.2"/>
+  <text x="76" y="624" class="h" fill="#92400e">wh-butter-core — one Device trait  (alloc · upload · dispatch · sync)</text>
+
+  <!-- arrows to backends -->
+  <path d="M270,646 L270,690" class="arr"/>
+  <path d="M580,646 L580,690" class="arr"/>
+  <path d="M890,646 L890,690" class="arr"/>
+  <path d="M1200,646 L1200,690" class="arr"/>
+
+  <!-- ===== backends ===== -->
+  <rect x="120" y="690" width="300" height="86" rx="10" fill="#dcfce7" stroke="#22c55e" stroke-width="2"/>
+  <text x="140" y="718" class="lbl" fill="#15803d">wh-butter-cuda</text>
+  <text x="140" y="740" class="small">REAL · GB10 sm_121</text>
+  <text x="140" y="760" class="tiny">NVRTC → PTX → driver</text>
+
+  <rect x="440" y="690" width="300" height="86" rx="10" fill="#fef9c3" stroke="#eab308" stroke-width="2"/>
+  <text x="460" y="718" class="lbl" fill="#854d0e">wh-butter-metal (metal-rs)</text>
+  <text x="460" y="740" class="small" font-weight="700" fill="#b45309">THE KEY PIECE TO BUILD</text>
+  <text x="460" y="760" class="tiny">makes Apple run the shared Rust models</text>
+
+  <rect x="760" y="690" width="280" height="86" rx="10" fill="#ffffff" stroke="#cbd5e1"/>
+  <text x="780" y="718" class="lbl">wh-butter-vulkan</text>
+  <text x="780" y="740" class="small">portable GPUs</text>
+  <text x="780" y="760" class="tiny">SPIR-V (iron target next)</text>
+
+  <rect x="1060" y="690" width="280" height="86" rx="10" fill="#ffffff" stroke="#e2e8f0" stroke-dasharray="4 3"/>
+  <text x="1080" y="718" class="lbl" fill="#94a3b8">butter-rocm …</text>
+  <text x="1080" y="740" class="small" fill="#94a3b8">AMD</text>
+  <text x="1080" y="760" class="tiny">one crate = one backend</text>
+
+  <!-- hardware -->
+  <path d="M270,776 L270,806" class="arr"/>
+  <path d="M590,776 L590,806" class="arr"/>
+  <path d="M900,776 L900,806" class="arr"/>
+  <path d="M1200,776 L1200,806" class="arr"/>
+  <text x="232" y="828" class="lbl">NVIDIA</text>
+  <text x="540" y="828" class="lbl">Apple GPU</text>
+  <text x="838" y="828" class="lbl">AMD / Intel / Android</text>
+  <text x="1150" y="828" class="lbl">…</text>
+
+  <!-- iron kernels feed backends -->
+  <rect x="56" y="868" width="1388" height="78" rx="12" fill="#ecfdf5" stroke="#34d399" stroke-width="2.4"/>
+  <text x="76" y="896" class="h" fill="#047857">iron — shared kernels (one IR → MSL · CUDA/PTX · SPIR-V)</text>
+  <text x="76" y="922" class="small">the kernels every backend runs · 4164/4164 bit-accurate on GB10 · already shared</text>
+  <path d="M300,868 C300,840 280,800 270,778" class="arrG" stroke-dasharray="5 4"/>
+  <text x="305" y="850" class="tiny" fill="#047857">feeds the Device backends</text>
+
+  <!-- payoff -->
+  <rect x="56" y="978" width="1388" height="150" rx="14" fill="#f0fdf4" stroke="#86efac" stroke-width="1.8"/>
+  <text x="76" y="1008" class="h" fill="#166534">No duplication</text>
+  <text x="76" y="1038" class="small">• Each model lives in ONE file (Rust). CUDA, Metal, Vulkan, ROCm all run the same code — the backend swaps underneath.</text>
+  <text x="76" y="1062" class="small">• Adding a GPU vendor = one backend crate, not 35 model rewrites. Adding a model = one builder, every GPU gets it free.</text>
+  <text x="76" y="1086" class="small">• Swift stays native for the iPhone app + is the bit-exact oracle the ports are checked against.</text>
+  <text x="76" y="1110" class="small" font-weight="600" fill="#166534">Cost: port ~6–10 builders (not 35) once + build the metal-rs backend. Then models are genuinely shared.</text>
+</svg>
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index f871a5b6..667cb00f 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -532,6 +532,7 @@ dependencies = [
 name = "wh-butter-models"
 version = "0.1.0"
 dependencies = [
+ "rayon",
  "serde_json",
  "wh-butter-core",
  "wh-butter-loader",
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 6433fad5..96b4b3d0 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -49,6 +49,7 @@ serde_json = "1"
 memmap2 = "0.9"
 parking_lot = "0.12"   # faster, unpoisoned locks on the dispatch hot path
 rustc-hash = "2"       # FxHash for already-hashed / densely-numeric keys
+rayon = "1"            # Laguna's host-side layer-byte cache build (wh-butter-models)
 
 
 # Perf build for benchmarks (fat-LTO inlines parking_lot locks, Arc/Rc clones,
diff --git a/rust/crates/backends/wh-butter-cuda/Cargo.toml b/rust/crates/backends/wh-butter-cuda/Cargo.toml
index 3705b6e9..329d4b20 100644
--- a/rust/crates/backends/wh-butter-cuda/Cargo.toml
+++ b/rust/crates/backends/wh-butter-cuda/Cargo.toml
@@ -112,3 +112,15 @@ required-features = ["cuda"]
 [[test]]
 name = "all_models"
 required-features = ["cuda"]
+
+[[test]]
+name = "proj_fp8_bench"
+required-features = ["cuda"]
+
+[[test]]
+name = "laguna"
+required-features = ["cuda"]
+
+[[test]]
+name = "colcopy"
+required-features = ["cuda"]
diff --git a/rust/crates/backends/wh-butter-cuda/src/imp.rs b/rust/crates/backends/wh-butter-cuda/src/imp.rs
index d480d335..07ff4a0e 100644
--- a/rust/crates/backends/wh-butter-cuda/src/imp.rs
+++ b/rust/crates/backends/wh-butter-cuda/src/imp.rs
@@ -74,8 +74,26 @@ impl CudaDevice {
         }
     }
 
+    /// Cache key for a compiled module. The bare kernel name is NOT unique:
+    /// registry kernels share one name across their dtype variants (the ops
+    /// layer keys its IR cache by (name, dtype), but the same name reaches
+    /// this backend for every variant). Keying compiled modules by name alone
+    /// silently reused the first-compiled dtype's binary for every other
+    /// dtype, which reads with the wrong element stride and faults (or worse,
+    /// corrupts silently). Fold each param's dtype into the key.
+    fn module_key(kernel: &Kernel) -> String {
+        use std::fmt::Write as _;
+        let mut key = kernel.name.clone();
+        key.push('|');
+        for p in &kernel.params {
+            let _ = write!(key, "{:?},", p.dtype);
+        }
+        key
+    }
+
     fn module_for(&self, kernel: &Kernel) -> Result<Arc<CachedModule>> {
-        if let Some(m) = self.modules.read().unwrap().get(&kernel.name) {
+        let mkey = Self::module_key(kernel);
+        if let Some(m) = self.modules.read().unwrap().get(&mkey) {
             return Ok(m.clone());
         }
         let cg = CudaGenerator::new();
@@ -95,7 +113,7 @@ impl CudaDevice {
         self.modules
             .write()
             .unwrap()
-            .insert(kernel.name.clone(), cached.clone());
+            .insert(mkey, cached.clone());
         Ok(cached)
     }
 }
@@ -181,7 +199,7 @@ impl Device for CudaDevice {
     fn dispatch(&self, kernel: &Kernel, bindings: &[Binding], grid: Grid) -> Result<()> {
         let module = self.module_for(kernel)?;
         let func = module.module.function(&kernel.name).map_err(dispatch_err)?;
-        let skey = (kernel.name.clone(), grid.block[0]);
+        let skey = (Self::module_key(kernel), grid.block[0]);
         let shared = if let Some(&s) = self.shared.read().unwrap().get(&skey) {
             s
         } else {
@@ -413,6 +431,20 @@ impl Device for CudaDevice {
         self.dev.moe_grouped_cutlass(ab.ptr, wb.ptr, ob.ptr, group_rows, expert_ids, n, k).map_err(dispatch_err)
     }
 
+    // NOTE(rebrand-merge): the backends branch also implemented
+    // moe_marlin_gemm / marlin_repack / marlin_build_routing here, calling
+    // through to `self.dev.{moe_marlin_gemm,marlin_repack,marlin_build_routing}`
+    // on the raw `wh_iron_runtime::CudaDevice`. Those methods don't exist on
+    // `thewafflehaus/iron@dev`'s CudaDevice (confirmed: no `marlin` symbol
+    // anywhere in that repo) — the marlin/NVFP4 CUDA runtime support was
+    // apparently only ever on the pre-rebrand kernel-toolchain fork this
+    // branch was originally pinned to. Left unimplemented here (falls through to the
+    // wh-butter-core default "unsupported on this backend" stub) until the
+    // iron runtime crate grows real marlin support; re-add the three
+    // pass-through impls verbatim from this crate's pre-rebrand-merge git
+    // history (`tom/feat/cuda-hip-vulkan-backends`, `src/imp.rs`, the commit
+    // just before this rebrand merge) once it does.
+
     fn moe_grouped_cutlass_fp4(
         &self,
         a: &dyn DeviceBuffer,
diff --git a/rust/crates/backends/wh-butter-cuda/tests/all_models.rs b/rust/crates/backends/wh-butter-cuda/tests/all_models.rs
index bc1c5155..45810c41 100644
--- a/rust/crates/backends/wh-butter-cuda/tests/all_models.rs
+++ b/rust/crates/backends/wh-butter-cuda/tests/all_models.rs
@@ -8,11 +8,10 @@ fn all_models_on_cuda() {
     wh_butter_modeltests::run_all(dev.as_ref(), "GB10 sm_121");
 }
 
-/// FAST FUSE_SLICECAST lever A/B on a small same-family model (loads in seconds).
-/// See `wh_butter_modeltests::bench_smallmodel_fuse_slicecast`. Default mamba2-130m;
-/// FUSE_SC_MODEL=falcon for Falcon-H1-0.5B. Off-vs-on bit-exact + timing.
-#[test]
-fn smallmodel_fuse_slicecast_ab() {
-    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA — skip"); return; };
-    wh_butter_modeltests::bench_smallmodel_fuse_slicecast(dev.as_ref(), "GB10 sm_121");
-}
+// NOTE(rebrand-merge): this file used to also have a `smallmodel_fuse_slicecast_ab`
+// test calling `wh_butter_modeltests::bench_smallmodel_fuse_slicecast`, but that
+// function doesn't exist anywhere in the modeltests crate on the original
+// tom/feat/cuda-hip-vulkan-backends branch either (pre-existing dead reference,
+// not something this merge introduced — the FUSE_SLICECAST bench was
+// apparently never actually landed in wh-butter-modeltests). Dropped rather
+// than stubbed since there's no reference implementation to port.
diff --git a/rust/crates/backends/wh-butter-cuda/tests/colcopy.rs b/rust/crates/backends/wh-butter-cuda/tests/colcopy.rs
new file mode 100644
index 00000000..ceeef2f2
--- /dev/null
+++ b/rust/crates/backends/wh-butter-cuda/tests/colcopy.rs
@@ -0,0 +1,99 @@
+#![cfg(feature = "cuda")]
+//! strided_col_copy launch-geometry regression: the op dispatches with a
+//! fixed 64-thread block. Before the fix, the grid was `n / 64` (integer
+//! division, no ceiling), so whenever `s * width` wasn't a multiple of 64
+//! the tail elements were silently dropped (never written into dst, left
+//! as zeroed garbage). Covers a deliberately non-64-aligned shape (the
+//! Laguna-style head-count case, e.g. width=72) plus a 64-aligned control
+//! shape to confirm the fix didn't regress the common case.
+use wh_butter_core::{DType, Device, Tensor};
+use wh_butter_cuda::CudaDevice;
+use wh_butter_ops::strided_col_copy;
+
+fn tb(v: &[f32]) -> Vec<u8> { v.iter().flat_map(|x| x.to_le_bytes()).collect() }
+fn fb(b: &[u8]) -> Vec<f32> { b.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect() }
+fn tn(d: &dyn Device, v: &[f32]) -> Tensor { Tensor::new(d.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32) }
+
+// Host-computed expectation: dst[ti*width + ci] = src[ti*stride + col_off + ci].
+fn host_expected(src: &[f32], s: usize, stride: usize, col_off: usize, width: usize) -> Vec<f32> {
+    (0..s)
+        .flat_map(|ti| (0..width).map(move |ci| src[ti * stride + col_off + ci]))
+        .collect()
+}
+
+fn run_case(name: &str, s: usize, stride: usize, col_off: usize, width: usize) {
+    let Some(dev) = CudaDevice::create().expect("cuda") else {
+        eprintln!("no CUDA, skipping {name}");
+        return;
+    };
+    let d = dev.as_ref();
+    let src: Vec<f32> = (0..s * stride).map(|i| ((i as f32) * 0.019).sin()).collect();
+    let src_t = tn(d, &src);
+
+    let out = strided_col_copy(d, &src_t, s, stride, col_off, width)
+        .unwrap_or_else(|e| panic!("{name}: strided_col_copy dispatch failed: {e:?}"));
+    d.synchronize().unwrap();
+
+    let mut bytes = vec![0u8; s * width * 4];
+    d.download(out.buffer.as_ref(), &mut bytes).unwrap();
+    let got = fb(&bytes);
+    let want = host_expected(&src, s, stride, col_off, width);
+
+    assert_eq!(got.len(), want.len(), "{name}: output length mismatch");
+    let mut max_diff = 0f32;
+    let mut first_bad: Option<usize> = None;
+    for (i, (g, w)) in got.iter().zip(want.iter()).enumerate() {
+        let diff = (g - w).abs();
+        if diff > max_diff { max_diff = diff; }
+        if diff > 1e-6 && first_bad.is_none() { first_bad = Some(i); }
+    }
+    eprintln!("[{name}] s={s} stride={stride} col_off={col_off} width={width} n={} max|GPU-CPU|={max_diff:e}", s * width);
+    if let Some(i) = first_bad {
+        panic!(
+            "{name}: mismatch at flat index {i} (ti={}, ci={}): got={} want={}",
+            i / width, i % width, got[i], want[i]
+        );
+    }
+}
+
+/// Deliberately non-64-aligned shape: s*width = 3*72 = 216, not a multiple of
+/// 64. This is the Laguna-style case (head counts 48/72) that exposed the
+/// launch-geometry bug: with the old `n / 64` grid, only 192 of 216 elements
+/// were ever dispatched, so the last 24 (rows 2's tail) came back as
+/// zero-initialized garbage instead of the real column values.
+#[test]
+fn strided_col_copy_non_64_aligned_shape() {
+    run_case("non_64_aligned", 3, 100, 4, 72);
+}
+
+/// Control: a 64-aligned shape (s*width divisible by 64) exercising the
+/// previously-working path, to confirm the div_ceil + in-kernel guard fix
+/// didn't regress the common (aligned) case.
+#[test]
+fn strided_col_copy_64_aligned_control() {
+    run_case("64_aligned_control", 4, 200, 8, 64);
+}
+
+/// Micro-repro for the prefill layer-1 fault: f16 table gather at the exact
+/// failing shapes (table [2, 3072] f16 from a cast, indices [20] u32 of 0/1).
+#[test]
+fn gather_f16_prefill_shapes() {
+    use wh_butter_core::{DType, Tensor};
+    use wh_butter_cuda::CudaDevice;
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let t = 2usize;
+    let hidden = 3072usize;
+    let x: Vec<f32> = (0..t * hidden).map(|i| (i as f32) * 0.001 - 3.0).collect();
+    let xb: Vec<u8> = x.iter().flat_map(|v| v.to_le_bytes()).collect();
+    let xf32 = Tensor::new(d.upload(&xb).unwrap(), vec![t, hidden], DType::F32);
+    let xf16 = wh_butter_ops::cast_f32_f16(d, &xf32).unwrap();
+    let idx: Vec<u32> = vec![0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0];
+    let ib: Vec<u8> = idx.iter().flat_map(|v| v.to_le_bytes()).collect();
+    let ti = Tensor::new(d.upload(&ib).unwrap(), vec![idx.len()], DType::U32);
+    let out = wh_butter_ops::gather(d, &xf16, &ti).expect("gather f16 dispatch");
+    d.synchronize().expect("post-gather sync");
+    let mut ob = vec![0u8; idx.len() * hidden * 2];
+    d.download(out.buffer.as_ref(), &mut ob).expect("download");
+    eprintln!("gather f16 prefill shapes: OK (first half-word {:02x}{:02x})", ob[0], ob[1]);
+}
diff --git a/rust/crates/backends/wh-butter-cuda/tests/f16norm_f32in.rs b/rust/crates/backends/wh-butter-cuda/tests/f16norm_f32in.rs
deleted file mode 100644
index 1a92798d..00000000
--- a/rust/crates/backends/wh-butter-cuda/tests/f16norm_f32in.rs
+++ /dev/null
@@ -1,70 +0,0 @@
-#![cfg(feature = "cuda")]
-//! Isolate add_rms_norm_f16norm with REAL in-model f32 inputs (a/b/w all f32).
-//! Compares residual + normed against a CPU oracle AND the working add_rms_norm.
-use wh_butter_core::{DType, Device, Tensor};
-use wh_butter_cuda::CudaDevice;
-use wh_butter_ops::{add_rms_norm, add_rms_norm_f16norm};
-
-fn tb(v:&[f32])->Vec<u8>{v.iter().flat_map(|x|x.to_le_bytes()).collect()}
-fn fb32(b:&[u8])->Vec<f32>{b.chunks_exact(4).map(|c|f32::from_le_bytes(c.try_into().unwrap())).collect()}
-fn f16_to_f32(bits:u16)->f32{
-    let sign=((bits>>15)&1) as u32; let exp=((bits>>10)&0x1f) as u32; let mant=(bits&0x3ff) as u32;
-    let f=if exp==0 { if mant==0 { sign<<31 } else { // subnormal
-        let mut e=-1i32; let mut m=mant; while m & 0x400 ==0 { m<<=1; e-=1; } m&=0x3ff;
-        ((sign<<31)|(((e+127+(-14))as u32)<<23)|(m<<13)) } }
-      else if exp==0x1f { (sign<<31)|0x7f800000|(mant<<13) }
-      else { (sign<<31)|((exp+(127-15))<<23)|(mant<<13) };
-    f32::from_bits(f)
-}
-fn fb16(b:&[u8])->Vec<f32>{b.chunks_exact(2).map(|c|f16_to_f32(u16::from_le_bytes([c[0],c[1]]))).collect()}
-fn tn(d:&dyn Device,v:&[f32],sh:Vec<usize>)->Tensor{Tensor::new(d.upload(&tb(v)).unwrap(),sh,DType::F32)}
-
-#[test]
-fn f16norm_f32_inputs_matches_oracle(){
-    let Some(dev)=CudaDevice::create().expect("cuda") else { eprintln!("no CUDA — skip"); return; };
-    let d = dev.as_ref();
-    let (rows, n) = (8usize, 4096usize); // real hidden size
-    let eps = 1e-5f32;
-    let a: Vec<f32> = (0..rows*n).map(|i| ((i % 13) as f32 * 0.07 - 0.4)).collect();
-    let b: Vec<f32> = (0..rows*n).map(|i| ((i % 7) as f32 * 0.05 - 0.15)).collect();
-    let w: Vec<f32> = (0..n).map(|i| 0.5 + (i % 5) as f32 * 0.1).collect();
-    // CPU oracle
-    let mut res_ref = vec![0f32; rows*n]; let mut nrm_ref = vec![0f32; rows*n];
-    for r in 0..rows { let mut ssq=0f32; for c in 0..n { let s=a[r*n+c]+b[r*n+c]; res_ref[r*n+c]=s; ssq+=s*s; }
-        let rms=1f32/(ssq/n as f32+eps).sqrt(); for c in 0..n { nrm_ref[r*n+c]=res_ref[r*n+c]*rms*w[c]; } }
-
-    let at=tn(d,&a,vec![rows,n]); let bt=tn(d,&b,vec![rows,n]); let wt=tn(d,&w,vec![n]);
-    // f32 control
-    let (r32, n32) = add_rms_norm(d, &at, &bt, &wt, eps).unwrap();
-    // fused f16norm
-    let (rf, nf) = add_rms_norm_f16norm(d, &at, &bt, &wt, eps).unwrap();
-    d.synchronize().unwrap();
-    assert_eq!(rf.dtype, DType::F32, "residual must be f32");
-    assert_eq!(nf.dtype, DType::F16, "normed must be f16");
-
-    let mut g=vec![0u8;rows*n*4]; d.download(rf.buffer.as_ref(),&mut g).unwrap(); let res_got=fb32(&g);
-    let mut h=vec![0u8;rows*n*2]; d.download(nf.buffer.as_ref(),&mut h).unwrap(); let nrm_got=fb16(&h);
-    let mut c32=vec![0u8;rows*n*4]; d.download(r32.buffer.as_ref(),&mut c32).unwrap(); let res_c=fb32(&c32);
-    let mut cn=vec![0u8;rows*n*4]; d.download(n32.buffer.as_ref(),&mut cn).unwrap(); let nrm_c=fb32(&cn);
-
-    // EXACT-FUSION CHECK: fused f16 normed must equal cast_f32_f16(f32wrap normed) bit-for-bit.
-    let nrm_c_f16 = wh_butter_ops::cast_f32_f16(d, &n32).unwrap();
-    d.synchronize().unwrap();
-    let mut hc=vec![0u8;rows*n*2]; d.download(nrm_c_f16.buffer.as_ref(),&mut hc).unwrap();
-    let mut hf=vec![0u8;rows*n*2]; d.download(nf.buffer.as_ref(),&mut hf).unwrap();
-    let bitdiff = hc.iter().zip(hf.iter()).filter(|(a,b)| a!=b).count();
-    eprintln!("FUSED vs cast_f32_f16(f32wrap): differing f16 BYTES = {bitdiff} / {}", rows*n*2);
-    let res_nan=res_got.iter().any(|x|!x.is_finite());
-    let nrm_nan=nrm_got.iter().any(|x|!x.is_finite());
-    let mut res_err=0f32; for i in 0..rows*n { res_err=res_err.max((res_got[i]-res_ref[i]).abs()); }
-    let mut nrm_err=0f32; for i in 0..rows*n { let d0=(nrm_got[i]-nrm_ref[i]).abs(); let den=nrm_ref[i].abs().max(1e-3); nrm_err=nrm_err.max(d0/den); }
-    let mut res_vs32=0f32; for i in 0..rows*n { res_vs32=res_vs32.max((res_got[i]-res_c[i]).abs()); }
-    eprintln!("f16norm f32-in: res_nan={res_nan} nrm_nan={nrm_nan} res|Δ|oracle={res_err:.3e} nrm_rel|Δ|oracle={nrm_err:.3e} res|Δ|vs_f32wrap={res_vs32:.3e}");
-    eprintln!("  res_got[0..4]={:?} ref={:?}", &res_got[0..4], &res_ref[0..4]);
-    eprintln!("  nrm_got[0..4]={:?} ref={:?}", &nrm_got[0..4], &nrm_ref[0..4]);
-    eprintln!("  f32wrap nrm[0..4]={:?}", &nrm_c[0..4]);
-    assert!(!res_nan && !nrm_nan, "NaN in fused outputs");
-    assert!(res_err < 1e-3, "residual wrong vs oracle: {res_err}");
-    assert!(nrm_err < 2e-2, "normed wrong vs oracle (f16 tol): {nrm_err}");
-    assert!(res_vs32 < 1e-3, "residual diverges from f32 wrapper: {res_vs32}");
-}
diff --git a/rust/crates/backends/wh-butter-cuda/tests/laguna.rs b/rust/crates/backends/wh-butter-cuda/tests/laguna.rs
new file mode 100644
index 00000000..23c5cd52
--- /dev/null
+++ b/rust/crates/backends/wh-butter-cuda/tests/laguna.rs
@@ -0,0 +1,507 @@
+// Copyright 2026 Eric Kryski (@ekryski) and Tom Turney (@TheTom)
+// SPDX-License-Identifier: Apache-2.0
+//! Laguna-S-2.1 decode smoke on CUDA. Env-gated: set BUTTER_LAGUNA_GGUF to the
+//! model's GGUF path (the 71G checkpoint), otherwise the test skips.
+use wh_butter_cuda::CudaDevice;
+#[test]
+fn laguna_host_dequant() {
+    wh_butter_modeltests::verify_laguna_host_dequant();
+}
+
+#[test]
+fn laguna_host_layer0() {
+    wh_butter_modeltests::verify_laguna_host_layer0();
+}
+
+/// Isolated rope_yarn_partial check with synthetic data. At position 0 the
+/// rotation angle is zero for every dim, so rotated dims must equal
+/// input * mscale (mscale = attn_factor exactly; the wrapper applies no
+/// internal re-multiply) and passthrough dims must be bit-identical.
+#[test]
+fn laguna_rope_yarn_unit() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let n_heads = 3usize;
+    let hd = 8usize;
+    let n_rot = 4u32;
+    let n: usize = n_heads * hd;
+    let x: Vec<f32> = (0..n).map(|i| (i as f32) * 0.25 - 2.0).collect();
+    let xb: Vec<u8> = x.iter().flat_map(|v| v.to_le_bytes()).collect();
+    let t = Tensor::new(d.upload(&xb).unwrap(), vec![n_heads, hd], DType::F32);
+
+    let attn_factor = 1.4852030263919618f32;
+    let freq_scale = 1.0f32 / 128.0;
+    let out = wh_butter_ops::rope_yarn_partial(d, &t, 0, n_rot, 500000.0, freq_scale, 1.0, attn_factor, 32.0, 1.0, 8192).unwrap();
+    let mut ob = vec![0u8; n * 4];
+    d.synchronize().unwrap();
+    d.download(out.buffer.as_ref(), &mut ob).unwrap();
+    let o: Vec<f32> = ob.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+
+    let mscale = attn_factor; // attn_factor is the final mscale, no internal re-multiply
+    let mut bad = 0;
+    for h in 0..n_heads {
+        for i in 0..hd {
+            let idx = h * hd + i;
+            let expect = if (i as u32) < n_rot { x[idx] * mscale } else { x[idx] };
+            if (o[idx] - expect).abs() > 1e-4 * expect.abs().max(1.0) {
+                eprintln!("pos0 MISMATCH h={h} i={i}: got {} expect {expect}", o[idx]);
+                bad += 1;
+            }
+        }
+    }
+    assert_eq!(bad, 0, "rope_yarn_partial wrong at position 0 ({bad} bad elems)");
+    eprintln!("rope pos-0 identity*mscale: OK (mscale={mscale})");
+
+    // Exact model geometry: 48 heads x 128 dims, n_rot 64, position 0. The
+    // same identity*mscale invariant must hold; this catches launch-geometry
+    // bugs the tiny shape above cannot (block y = 64, grid x = 48).
+    let n_heads2 = 48usize;
+    let hd2 = 128usize;
+    let n2 = n_heads2 * hd2;
+    let x2: Vec<f32> = (0..n2).map(|i| ((i * 2654435761usize) % 1000) as f32 * 0.002 - 1.0).collect();
+    let xb2: Vec<u8> = x2.iter().flat_map(|v| v.to_le_bytes()).collect();
+    let t2 = Tensor::new(d.upload(&xb2).unwrap(), vec![n_heads2, hd2], DType::F32);
+    let out2 = wh_butter_ops::rope_yarn_partial(d, &t2, 0, 64, 500000.0, freq_scale, 1.0, attn_factor, 32.0, 1.0, 8192).unwrap();
+    let mut ob2 = vec![0u8; n2 * 4];
+    d.synchronize().unwrap();
+    d.download(out2.buffer.as_ref(), &mut ob2).unwrap();
+    let o2: Vec<f32> = ob2.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+    let mut bad2 = 0;
+    for h in 0..n_heads2 {
+        for i in 0..hd2 {
+            let idx = h * hd2 + i;
+            let expect = if i < 64 { x2[idx] * mscale } else { x2[idx] };
+            if (o2[idx] - expect).abs() > 1e-4 * expect.abs().max(1.0) {
+                if bad2 < 6 { eprintln!("model-geom pos0 MISMATCH h={h} i={i}: got {} expect {expect}", o2[idx]); }
+                bad2 += 1;
+            }
+        }
+    }
+    assert_eq!(bad2, 0, "rope_yarn_partial wrong at model geometry pos 0 ({bad2} bad elems)");
+    eprintln!("rope model-geometry pos-0: OK");
+}
+
+/// Isolated per-head softplus gate check with synthetic data.
+#[test]
+fn laguna_gate_unit() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let n_heads = 4usize;
+    let hd = 8usize;
+    let attn: Vec<f32> = (0..n_heads * hd).map(|i| (i as f32) * 0.1 - 1.0).collect();
+    let g: Vec<f32> = vec![-2.0, 0.0, 1.5, 30.0];
+    let up = |v: &[f32]| -> Tensor {
+        let b: Vec<u8> = v.iter().flat_map(|x| x.to_le_bytes()).collect();
+        Tensor::new(d.upload(&b).unwrap(), vec![v.len()], DType::F32)
+    };
+    let ta = up(&attn).reshaped(vec![n_heads, hd]);
+    let tg = up(&g);
+    let out = wh_butter_ops::gate_softplus_mul_perhead(d, &ta, &tg).unwrap();
+    let mut ob = vec![0u8; attn.len() * 4];
+    d.synchronize().unwrap();
+    d.download(out.buffer.as_ref(), &mut ob).unwrap();
+    let o: Vec<f32> = ob.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+    let mut bad = 0;
+    for h in 0..n_heads {
+        let sp = if g[h] > 20.0 { g[h] } else { (1.0 + g[h].exp()).ln() };
+        for i in 0..hd {
+            let idx = h * hd + i;
+            let expect = attn[idx] * sp;
+            if (o[idx] - expect).abs() > 1e-5 * expect.abs().max(1.0) {
+                eprintln!("gate MISMATCH h={h} i={i}: got {} expect {expect}", o[idx]);
+                bad += 1;
+            }
+        }
+    }
+    assert_eq!(bad, 0, "gate_softplus_mul_perhead wrong ({bad} bad elems)");
+    eprintln!("gate softplus per-head: OK");
+}
+
+/// Batched-prefill `rope_yarn_partial_many` must match a T-loop of the
+/// scalar-position `rope_yarn_partial` (the proven decode kernel), one call
+/// per token row at that row's absolute position. Exercises both a YaRN
+/// full-attention-layer shape (n_rot < head_dim, ext_factor != 0) and a
+/// plain sliding-layer shape (n_rot == head_dim, ext_factor == 0), matching
+/// Laguna's two RoPE configs.
+#[test]
+fn laguna_rope_yarn_many_matches_looped() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let up = |v: &[f32]| -> Tensor {
+        let b: Vec<u8> = v.iter().flat_map(|x| x.to_le_bytes()).collect();
+        Tensor::new(d.upload(&b).unwrap(), vec![v.len()], DType::F32)
+    };
+    let dl = |t: &Tensor, n: usize| -> Vec<f32> {
+        let mut b = vec![0u8; n * 4];
+        d.synchronize().unwrap();
+        d.download(t.buffer.as_ref(), &mut b).unwrap();
+        b.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect()
+    };
+
+    // (n_rot, theta, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow, n_orig_ctx, label)
+    let configs: [(u32, f32, f32, f32, f32, f32, f32, u32, &str); 2] = [
+        (64, 500000.0, 1.0 / 128.0, 1.0, 1.4852030263919618, 32.0, 1.0, 8192, "full/yarn"),
+        (128, 10000.0, 1.0, 0.0, 1.0, 32.0, 1.0, 8192, "sliding/plain"),
+    ];
+
+    for (n_rot, theta, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow, n_orig_ctx, label) in configs {
+        let t = 5usize;
+        let n_heads = 6usize;
+        let hd = 128usize;
+        let n = t * n_heads * hd;
+        let x: Vec<f32> = (0..n).map(|i| ((i * 2654435761usize) % 2000) as f32 * 0.001 - 1.0).collect();
+        let positions: Vec<u32> = (0..t).map(|i| (7 + i * 13) as u32).collect(); // non-contiguous, non-zero-based
+
+        // Batched: one dispatch over [T, n_heads, hd].
+        let xt = up(&x).reshaped(vec![t, n_heads, hd]);
+        let posb = Tensor::new(
+            d.upload(&positions.iter().flat_map(|p| p.to_le_bytes()).collect::<Vec<u8>>()).unwrap(),
+            vec![t], DType::U32,
+        );
+        let got = wh_butter_ops::rope_yarn_partial_many(
+            d, &xt, &posb, n_heads, n_rot, theta, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow, n_orig_ctx,
+        ).unwrap();
+        let gv = dl(&got, n);
+
+        // Looped reference: one rope_yarn_partial call per token row at its
+        // own absolute position, over that row's [n_heads, hd] slab.
+        let mut ev = vec![0f32; n];
+        for (ti, &pos) in positions.iter().enumerate() {
+            let row = &x[ti * n_heads * hd..(ti + 1) * n_heads * hd];
+            let rt = up(row).reshaped(vec![n_heads, hd]);
+            let ro = wh_butter_ops::rope_yarn_partial(
+                d, &rt, pos, n_rot, theta, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow, n_orig_ctx,
+            ).unwrap();
+            let rov = dl(&ro, n_heads * hd);
+            ev[ti * n_heads * hd..(ti + 1) * n_heads * hd].copy_from_slice(&rov);
+        }
+
+        let mut bad = 0;
+        for i in 0..n {
+            if (gv[i] - ev[i]).abs() > 1e-4 * ev[i].abs().max(1.0) {
+                if bad < 6 {
+                    eprintln!("[{label}] MISMATCH i={i}: many={} looped={}", gv[i], ev[i]);
+                }
+                bad += 1;
+            }
+        }
+        assert_eq!(bad, 0, "rope_yarn_partial_many mismatch vs looped rope_yarn_partial ({label}, {bad} bad elems)");
+        eprintln!("rope_yarn_partial_many vs looped [{label}]: OK");
+    }
+}
+
+/// Batched-prefill `gate_softplus_mul_perhead_many` must match a T-loop of
+/// the scalar-row `gate_softplus_mul_perhead` (the proven decode kernel),
+/// one call per token row.
+#[test]
+fn laguna_gate_many_matches_looped() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let up = |v: &[f32]| -> Tensor {
+        let b: Vec<u8> = v.iter().flat_map(|x| x.to_le_bytes()).collect();
+        Tensor::new(d.upload(&b).unwrap(), vec![v.len()], DType::F32)
+    };
+    let dl = |t: &Tensor, n: usize| -> Vec<f32> {
+        let mut b = vec![0u8; n * 4];
+        d.synchronize().unwrap();
+        d.download(t.buffer.as_ref(), &mut b).unwrap();
+        b.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect()
+    };
+
+    let t = 4usize;
+    let n_heads = 5usize; // deliberately not 64-aligned (Laguna's real head counts: 48/72)
+    let hd = 8usize;
+    let n = t * n_heads * hd;
+    let attn: Vec<f32> = (0..n).map(|i| (i as f32) * 0.05 - 2.0).collect();
+    let g: Vec<f32> = (0..t * n_heads).map(|i| (i as f32) * 3.7 - 6.0).collect(); // spans <0, ~0, >20 branches
+
+    let at = up(&attn).reshaped(vec![t, n_heads, hd]);
+    let gt = up(&g);
+    let got = wh_butter_ops::gate_softplus_mul_perhead_many(d, &at, &gt, n_heads).unwrap();
+    let gv = dl(&got, n);
+
+    let mut ev = vec![0f32; n];
+    for ti in 0..t {
+        let arow = &attn[ti * n_heads * hd..(ti + 1) * n_heads * hd];
+        let grow = &g[ti * n_heads..(ti + 1) * n_heads];
+        let art = up(arow).reshaped(vec![n_heads, hd]);
+        let grt = up(grow);
+        let ro = wh_butter_ops::gate_softplus_mul_perhead(d, &art, &grt).unwrap();
+        let rov = dl(&ro, n_heads * hd);
+        ev[ti * n_heads * hd..(ti + 1) * n_heads * hd].copy_from_slice(&rov);
+    }
+
+    let mut bad = 0;
+    for i in 0..n {
+        if (gv[i] - ev[i]).abs() > 1e-5 * ev[i].abs().max(1.0) {
+            if bad < 6 {
+                eprintln!("MISMATCH i={i}: many={} looped={}", gv[i], ev[i]);
+            }
+            bad += 1;
+        }
+    }
+    assert_eq!(bad, 0, "gate_softplus_mul_perhead_many mismatch vs looped gate_softplus_mul_perhead ({bad} bad elems)");
+    eprintln!("gate_softplus_mul_perhead_many vs looped: OK");
+}
+
+/// `sdpa_decode_nbuf` (the CUDA-graph-capturable raw-CUDA fallback used by
+/// Laguna's decode path under `BUTTER_LAGUNA_GRAPH=1`) must match the registry
+/// `ops::sdpa_decode` kernel's output, same dense-causal single-pass
+/// online-softmax math, just reading `n_kv` from a device buffer at kernel
+/// runtime instead of a launch-time baked-in scalar. Checked at `n_kv` in
+/// {1, 7, 512} (kv_stride fixed at 512) x `heads_per_group` in {6, 9}:
+/// Laguna's actual GQA ratios (48 full-attention / 72 sliding-window Q heads
+/// over 8 KV heads).
+#[test]
+fn laguna_sdpa_decode_nbuf_matches_registry() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+    let hd = 128usize;
+    let n_kv_heads = 4usize;
+    let kv_stride = 512u32;
+    let scale = 1.0 / (hd as f32).sqrt();
+
+    let ramp = |n: usize, step: f32, start: f32| -> Vec<f32> {
+        (0..n).map(|i| ((start + i as f32 * step) % 2.0) - 1.0).collect()
+    };
+    let up = |v: &[f32]| -> Tensor {
+        let b: Vec<u8> = v.iter().flat_map(|x| x.to_le_bytes()).collect();
+        Tensor::new(d.upload(&b).unwrap(), vec![v.len()], DType::F32)
+    };
+
+    for &heads_per_group in &[6usize, 9usize] {
+        let n_q_heads = n_kv_heads * heads_per_group;
+        let q = ramp(n_q_heads * hd, 0.013, -0.4);
+        let k = ramp(n_kv_heads * kv_stride as usize * hd, 0.011, -0.5);
+        let v = ramp(n_kv_heads * kv_stride as usize * hd, 0.007, -0.3);
+        let tq = up(&q).reshaped(vec![n_q_heads, hd]);
+        let tk = up(&k).reshaped(vec![n_kv_heads, kv_stride as usize, hd]);
+        let tv = up(&v).reshaped(vec![n_kv_heads, kv_stride as usize, hd]);
+
+        for &n_kv in &[1u32, 7, 512] {
+            let expect = wh_butter_ops::sdpa_decode(d, &tq, &tk, &tv, hd, n_kv, kv_stride, heads_per_group as u32, scale).unwrap();
+            let mut eb = vec![0u8; n_q_heads * hd * 4];
+            d.synchronize().unwrap();
+            d.download(expect.buffer.as_ref(), &mut eb).unwrap();
+            let ev: Vec<f32> = eb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+
+            let n_kv_buf = Tensor::new(d.upload(&n_kv.to_le_bytes()).unwrap(), vec![1], DType::U32);
+            let got = wh_butter_ops::sdpa_decode_nbuf(d, &tq, &tk, &tv, hd, &n_kv_buf, kv_stride, heads_per_group as u32, scale).unwrap();
+            let mut gb = vec![0u8; n_q_heads * hd * 4];
+            d.synchronize().unwrap();
+            d.download(got.buffer.as_ref(), &mut gb).unwrap();
+            let gv: Vec<f32> = gb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+
+            let mut bad = 0;
+            for i in 0..ev.len() {
+                if (ev[i] - gv[i]).abs() > 1e-4 * ev[i].abs().max(1.0) {
+                    if bad < 6 {
+                        eprintln!("MISMATCH hpg={heads_per_group} n_kv={n_kv} i={i}: nbuf={} registry={}", gv[i], ev[i]);
+                    }
+                    bad += 1;
+                }
+            }
+            assert_eq!(bad, 0, "sdpa_decode_nbuf mismatch vs registry sdpa_decode (heads_per_group={heads_per_group}, n_kv={n_kv}, {bad} bad elems)");
+        }
+    }
+    eprintln!("sdpa_decode_nbuf matches registry sdpa_decode: OK");
+}
+
+/// `iron_moe_gather_q4_swiglu` (the fused gate+up+SwiGLU MoE gather kernel
+/// used by Laguna's decode path under `BUTTER_LAGUNA_FUSE=1`) must match the
+/// unfused compose: `moe_gather_q4` run twice (gate stack, up stack) then a
+/// host-side SwiGLU on the downloaded pair. Synthetic 4-expert stack, top_k
+/// 3, inter 8, hid 32 (small but exercises the same Q4 block-32 / f16
+/// per-block-scale layout the real Laguna MoE stacks use). Modeled on
+/// `laguna_gate_unit` above: synthetic buffers, upload, compare with a
+/// relative tolerance.
+///
+/// TODO(rebrand-merge): `wh_butter_ops::moe_gather_q4` / `moe_gather_q4_swiglu`
+/// dispatch `iron_moe_gather_q4` / `iron_moe_gather_q4_swiglu` via
+/// `wh_iron_std`'s kernel IR, but the pinned `thewafflehaus/iron@dev` kernel
+/// library only ships `iron_moe_gather_q4_relu2/_down/_down_accum` — the
+/// plain (non-relu2) and `_swiglu` variants this test needs are not present
+/// upstream yet. Ignored until those kernels land in the iron repo; do not
+/// silently unignore without confirming they exist.
+#[ignore = "blocked: iron_moe_gather_q4 / iron_moe_gather_q4_swiglu kernels not yet in thewafflehaus/iron@dev"]
+#[test]
+fn laguna_moe_swiglu_fused_unit() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+
+    let n_exp = 4usize;
+    let inter = 8usize;
+    let hid = 32usize;
+    let top_k = 3usize;
+
+    // f32 -> f16 bit conversion, exactly as the Laguna GGUF loader converts
+    // `quantize_q4`'s host f32 per-block scales before upload (the gather
+    // kernels read scales as f16; uploading raw f32 bits would silently
+    // reinterpret them as garbage f16s).
+    fn half_bits(f: f32) -> u16 {
+        let x = f.to_bits();
+        let sign = ((x >> 16) & 0x8000) as u16;
+        let e = ((x >> 23) & 0xff) as i32 - 112;
+        if e <= 0 { return sign; }
+        if e >= 0x1f { return sign | 0x7c00; }
+        let m = (x >> 13) & 0x3ff;
+        let round = (x >> 12) & 1;
+        let v = ((e as u32) << 10) | m;
+        sign | ((v + round) as u16)
+    }
+    let f16_bytes = |v: &[f32]| -> Vec<u8> { v.iter().flat_map(|&f| half_bits(f).to_le_bytes()).collect() };
+    let f32_bytes = |v: &[f32]| -> Vec<u8> { v.iter().flat_map(|x| x.to_le_bytes()).collect() };
+    let u32_bytes = |v: &[u32]| -> Vec<u8> { v.iter().flat_map(|x| x.to_le_bytes()).collect() };
+
+    let mut s = 0x9e3779b9u32;
+    let mut rng = || { s ^= s << 13; s ^= s >> 17; s ^= s << 5; (s as f32 / u32::MAX as f32) - 0.5 };
+
+    let gate_w: Vec<f32> = (0..n_exp * inter * hid).map(|_| rng()).collect();
+    let up_w: Vec<f32> = (0..n_exp * inter * hid).map(|_| rng()).collect();
+    let x: Vec<f32> = (0..hid).map(|_| rng()).collect();
+
+    let (gate_qs, gate_sc) = wh_butter_ops::quantize_q4(&gate_w, n_exp * inter, hid);
+    let (up_qs, up_sc) = wh_butter_ops::quantize_q4(&up_w, n_exp * inter, hid);
+
+    let gate_qs_t = Tensor::new(d.upload(&u32_bytes(&gate_qs)).unwrap(), vec![gate_qs.len()], DType::U32);
+    let gate_sc_t = Tensor::new(d.upload(&f16_bytes(&gate_sc)).unwrap(), vec![gate_sc.len()], DType::F16);
+    let up_qs_t = Tensor::new(d.upload(&u32_bytes(&up_qs)).unwrap(), vec![up_qs.len()], DType::U32);
+    let up_sc_t = Tensor::new(d.upload(&f16_bytes(&up_sc)).unwrap(), vec![up_sc.len()], DType::F16);
+    let x_t = Tensor::new(d.upload(&f32_bytes(&x)).unwrap(), vec![hid], DType::F32);
+
+    let idx: Vec<u32> = vec![2, 0, 3];
+    assert_eq!(idx.len(), top_k);
+    let idx_t = Tensor::new(d.upload(&u32_bytes(&idx)).unwrap(), vec![top_k], DType::U32);
+
+    // Fused: one dispatch, gate+up+silu(gate)*up in the kernel.
+    let fused = wh_butter_ops::moe_gather_q4_swiglu(d, &gate_qs_t, &gate_sc_t, &up_qs_t, &up_sc_t, &x_t, &idx_t, top_k, inter, hid).unwrap();
+    let mut fb = vec![0u8; top_k * inter * 4];
+    d.synchronize().unwrap();
+    d.download(fused.buffer.as_ref(), &mut fb).unwrap();
+    let fv: Vec<f32> = fb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+
+    // Unfused compose: two plain gathers + host-side SwiGLU on the download.
+    let gates = wh_butter_ops::moe_gather_q4(d, &gate_qs_t, &gate_sc_t, &x_t, &idx_t, top_k, inter, hid).unwrap();
+    let ups = wh_butter_ops::moe_gather_q4(d, &up_qs_t, &up_sc_t, &x_t, &idx_t, top_k, inter, hid).unwrap();
+    let mut gb = vec![0u8; top_k * inter * 4];
+    let mut ub = vec![0u8; top_k * inter * 4];
+    d.synchronize().unwrap();
+    d.download(gates.buffer.as_ref(), &mut gb).unwrap();
+    d.download(ups.buffer.as_ref(), &mut ub).unwrap();
+    let gv: Vec<f32> = gb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+    let uv: Vec<f32> = ub.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+
+    let mut bad = 0;
+    for i in 0..top_k * inter {
+        let silu_g = gv[i] / (1.0 + (-gv[i]).exp());
+        let expect = silu_g * uv[i];
+        if (fv[i] - expect).abs() > 1e-3 * expect.abs().max(1.0) {
+            eprintln!("swiglu-fused MISMATCH i={i}: fused={} expect={expect} (gate={} up={})", fv[i], gv[i], uv[i]);
+            bad += 1;
+        }
+    }
+    assert_eq!(bad, 0, "iron_moe_gather_q4_swiglu wrong vs unfused compose ({bad} bad elems)");
+    eprintln!("moe gather q4 fused swiglu vs unfused compose: OK");
+}
+
+/// `sdpa_multi_tc_varlen`'s new sliding-window block skip (`win != 0`) must
+/// match the `win = 0` full-range/mask-only path exactly, the skip only
+/// shrinks which query rows each KV block's GEMMs cover, the per-element
+/// `seg_lo` mask inside `sdpa_tc_softmax_varlen` is still the correctness
+/// source of truth either way, so both should reach the same result.
+///
+/// `T = 16` query rows, sliding window 4, but `base_kv = 4080` so this
+/// chunk continues a much longer prefix (`kv_stride = n_kv = 4096`). That
+/// makes `n_kv` exceed the kernel's fixed 2048-wide KV block (`bk =
+/// 2048.min(n_kv)`), so the KV range splits into two blocks: block 0
+/// `[0, 2048)` and block 1 `[2048, 4096)`. With window 4 every query's
+/// `seg_lo` sits at `4077..4092`, entirely inside block 1, so under `win=4`
+/// block 0 should be skipped ENTIRELY (q_cnt=0) while block 1 is computed
+/// for every row, genuinely exercising the row-range skip, not just
+/// threading the new parameter through as a no-op. Modeled on
+/// `laguna_sdpa_decode_nbuf_matches_registry`'s synthetic-ramp-data +
+/// download + relative-tolerance-compare harness.
+#[test]
+fn laguna_sdpa_varlen_window_skip_matches_full_range() {
+    use wh_butter_core::{DType, Tensor};
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    let d = dev.as_ref();
+
+    let hd = 8usize;
+    let n_kv_heads = 1usize;
+    let heads_per_group = 2usize;
+    let n_q_heads = n_kv_heads * heads_per_group;
+    let t = 16usize;              // n_query (this chunk's rows)
+    let base_kv = 4080u32;        // this chunk continues a long prefix
+    let kv_stride = 4096u32;      // n_kv = base_kv + t = 4096 > bk(2048) => 2 blocks
+    let window = 4u32;
+    let scale = 1.0 / (hd as f32).sqrt();
+
+    let ramp = |n: usize, step: f32, start: f32| -> Vec<f32> {
+        (0..n).map(|i| ((start + i as f32 * step) % 2.0) - 1.0).collect()
+    };
+    let up = |v: &[f32]| -> Tensor {
+        let b: Vec<u8> = v.iter().flat_map(|x| x.to_le_bytes()).collect();
+        Tensor::new(d.upload(&b).unwrap(), vec![v.len()], DType::F32)
+    };
+
+    let q = ramp(t * n_q_heads * hd, 0.013, -0.4);
+    let k = ramp(n_kv_heads * kv_stride as usize * hd, 0.011, -0.5);
+    let v = ramp(n_kv_heads * kv_stride as usize * hd, 0.007, -0.3);
+    let tq = up(&q).reshaped(vec![t, n_q_heads, hd]);
+    let tk = up(&k).reshaped(vec![n_kv_heads, kv_stride as usize, hd]);
+    let tv = up(&v).reshaped(vec![n_kv_heads, kv_stride as usize, hd]);
+
+    // Exactly Laguna's sliding-layer prefill convention:
+    // seg_lo[i] = max(0, (base_kv+i) - (window-1)).
+    let seg_lo_h: Vec<u32> = (0..t)
+        .map(|i| ((base_kv as i64 + i as i64) - (window as i64 - 1)).max(0) as u32)
+        .collect();
+    let seg_lo = Tensor::new(
+        d.upload(&seg_lo_h.iter().flat_map(|x| x.to_le_bytes()).collect::<Vec<u8>>()).unwrap(),
+        vec![t], DType::U32,
+    );
+
+    let out_full = wh_butter_ops::sdpa_multi_tc_varlen(
+        d, &tq, &tk, &tv, &seg_lo, 0, hd, n_q_heads as u32,
+        base_kv, t as u32, kv_stride, heads_per_group as u32, true, scale, 0,
+    ).unwrap();
+    let out_win = wh_butter_ops::sdpa_multi_tc_varlen(
+        d, &tq, &tk, &tv, &seg_lo, 0, hd, n_q_heads as u32,
+        base_kv, t as u32, kv_stride, heads_per_group as u32, true, scale, window,
+    ).unwrap();
+
+    let n = t * n_q_heads * hd;
+    let dl = |ten: &Tensor| -> Vec<f32> {
+        let mut b = vec![0u8; n * 4];
+        d.synchronize().unwrap();
+        d.download(ten.buffer.as_ref(), &mut b).unwrap();
+        b.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect()
+    };
+    let fv = dl(&out_full);
+    let wv = dl(&out_win);
+
+    let mut bad = 0;
+    for i in 0..n {
+        if (fv[i] - wv[i]).abs() > 1e-4 * fv[i].abs().max(1.0) {
+            if bad < 6 {
+                eprintln!("MISMATCH i={i}: win=0 {} vs win={window} {}", fv[i], wv[i]);
+            }
+            bad += 1;
+        }
+    }
+    assert_eq!(bad, 0, "sdpa_multi_tc_varlen window skip (win={window}) mismatch vs win=0 full-range ({bad} bad elems)");
+    eprintln!("sdpa_multi_tc_varlen window skip vs full-range: OK");
+}
+
+#[test]
+fn laguna_on_cuda() {
+    let Some(dev) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA, skip"); return; };
+    wh_butter_modeltests::verify_laguna(dev.as_ref(), "GB10 sm_121");
+}
diff --git a/rust/crates/backends/wh-butter-cuda/tests/moe_grouped_mma_test.rs b/rust/crates/backends/wh-butter-cuda/tests/moe_grouped_mma_test.rs
index 38340185..e81d9ccc 100644
--- a/rust/crates/backends/wh-butter-cuda/tests/moe_grouped_mma_test.rs
+++ b/rust/crates/backends/wh-butter-cuda/tests/moe_grouped_mma_test.rs
@@ -6,10 +6,10 @@
 
 use wh_butter_cuda::CudaDevice;
 use wh_butter_core::{DType, Device, Tensor};
-use wh_butter_ops::{moe_grouped_gemm_mma, moe_grouped_gemm_cutlass, cast_f32_f16, cast_f16_f32, lt_fp4_quant_g, lt_fp4_quant_g_46, dec_ue4m3,
+use wh_butter_ops::{gemm_cublas_f32out, fwht128, gemm_fp8, fp8_quant, moe_q4_grouped_mma, moe_q4_grouped_mma_dev, quantize_q4, moe_grouped_gemm_mma, moe_grouped_gemm_cutlass, cast_f32_f16, cast_f16_f32, lt_fp4_quant_g, lt_fp4_quant_g_46, dec_ue4m3,
     moe_grouped_gemm_w4a8, w4a8_actquant_grouped, w4a8_packw, w4a8_sfa_offsets,
     moe_grouped_gemm_w8a8, w8a8_actquant_grouped, w8a8_packw,
-    lt_fp4_quant_slab, lt_fp4_quant_grouped, moe_grouped_gemm_cutlass_fp4};
+    lt_fp4_quant_slab, lt_fp4_quant_grouped, moe_grouped_gemm_cutlass_fp4, rms_norm, gemm_cublas, cast_f32_bf16, cast_bf16_f32};
 
 fn rng(n: usize, seed: u64) -> Vec<f32> {
     let mut v = vec![0f32; n];
@@ -79,7 +79,7 @@ fn run_cutlass(d: &dyn Device, n_exp: usize, k: usize, n: usize, groups: &[(usiz
     let w_dev = cast_f32_f16(d, &up(&w_f, vec![n_exp, n, k])).unwrap().reshaped(vec![n_exp, n, k]);
     let out = match moe_grouped_gemm_cutlass(d, &a_dev, &w_dev, &g_starts, &expert_ids, n, k) {
         Ok(o) => o,
-        Err(e) => { eprintln!("cutlass unavailable ({e:?}) — skip"); return; }
+        Err(e) => { eprintln!("cutlass unavailable ({e:?}), skip"); return; }
     };
     let got = dl(d, &cast_f16_f32(d, &out).unwrap());
     let mut exp = vec![0f32; mt*n];
@@ -112,11 +112,11 @@ fn run_w4a8(d: &dyn Device, n_exp: usize, k: usize, n: usize, groups: &[(usize,
     // W4A8: fp8 e4m3 acts (group-aware per-32 SFA) x mxfp4 weights (per-32 SFB).
     let (sfa_off, sfa_bytes) = w4a8_sfa_offsets(&g_starts, k);
     let (a_q, a_sf) = match w4a8_actquant_grouped(d, &a_dev, &g_starts.iter().zip(g_starts.iter().skip(1)).map(|(s,e)|(e-s) as i32).collect::<Vec<_>>(), &sfa_off, sfa_bytes, mt, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w4a8 actquant unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w4a8 actquant unavailable ({e:?}): skip"); return; } };
     let (w_pack, w_sf, sfb_eb) = match w4a8_packw(d, &w_dev, n_exp, n, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w4a8 packw unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w4a8 packw unavailable ({e:?}): skip"); return; } };
     let out = match moe_grouped_gemm_w4a8(d, &a_q, &a_sf, &sfa_off, &w_pack, &w_sf, sfb_eb, &g_starts, &expert_ids, n, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w4a8 gemm unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w4a8 gemm unavailable ({e:?}): skip"); return; } };
     let got = dl(d, &cast_f16_f32(d, &out).unwrap());
 
     // host f32 reference + cosine
@@ -156,11 +156,11 @@ fn run_w8a8(d: &dyn Device, n_exp: usize, k: usize, n: usize, groups: &[(usize,
     // W4A8: fp8 e4m3 acts (group-aware per-32 SFA) x mxfp4 weights (per-32 SFB).
     let (sfa_off, sfa_bytes) = w4a8_sfa_offsets(&g_starts, k);
     let (a_q, a_sf) = match w8a8_actquant_grouped(d, &a_dev, &g_starts.iter().zip(g_starts.iter().skip(1)).map(|(s,e)|(e-s) as i32).collect::<Vec<_>>(), &sfa_off, sfa_bytes, mt, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w8a8 actquant unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w8a8 actquant unavailable ({e:?}): skip"); return; } };
     let (w_pack, w_sf, sfb_eb) = match w8a8_packw(d, &w_dev, n_exp, n, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w8a8 packw unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w8a8 packw unavailable ({e:?}): skip"); return; } };
     let out = match moe_grouped_gemm_w8a8(d, &a_q, &a_sf, &sfa_off, &w_pack, &w_sf, sfb_eb, &g_starts, &expert_ids, n, k) {
-        Ok(o) => o, Err(e) => { eprintln!("w8a8 gemm unavailable ({e:?}) — skip"); return; } };
+        Ok(o) => o, Err(e) => { eprintln!("w8a8 gemm unavailable ({e:?}): skip"); return; } };
     let got = dl(d, &cast_f16_f32(d, &out).unwrap());
 
     // host f32 reference + cosine
@@ -261,7 +261,7 @@ fn moe_grouped_w4a8_down() {
 
 #[test]
 fn moe_grouped_w4a8_down_pad() {
-    // DOWN-proj K padded to 1920 (=15*128) — verifies K%128 is the constraint.
+    // DOWN-proj K padded to 1920 (=15*128), verifies K%128 is the constraint.
     let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
     run_w4a8(d.as_ref(), 8, 1920, 2688, &[(0,96),(3,40),(5,128),(7,17)]);
 }
@@ -284,7 +284,7 @@ fn moe_grouped_mma_bench() {
     let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
     let dev = d.as_ref();
     // Full S=2048 MoE up-proj: 128 experts × 96 rows, N=inter=1856, K=hid=2688.
-    let (n_exp, n, k, rows) = (128usize, 1856usize, 2688usize, 96usize);
+    let n_exp: usize = std::env::var("NEXP").ok().and_then(|v|v.parse().ok()).unwrap_or(128); let (n, k, rows) = (1856usize, 2688usize, 96usize);
     let mut g_starts = vec![0usize]; let mut eids = vec![];
     for e in 0..n_exp { eids.push(e); g_starts.push(g_starts.last().unwrap() + rows); }
     let mt = *g_starts.last().unwrap();
@@ -344,3 +344,277 @@ fn lt_fp4_46_unit() {
     let rms=|v:&[f32]|(v.iter().map(|x|(*x as f64)*(*x as f64)).sum::<f64>()/v.len() as f64).sqrt();
     eprintln!("  rms: input={:.4} base={:.4} _46={:.4}", rms(&xf), rms(&rec0), rms(&rec6));
 }
+
+#[test]
+fn moe_grouped_cutlass_bench() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let n_exp: usize = std::env::var("NEXP").ok().and_then(|v|v.parse().ok()).unwrap_or(128); let (n, k, rows) = (1856usize, 2688usize, 96usize);
+    let mut g_starts = vec![0usize]; let mut eids = vec![];
+    for e in 0..n_exp { eids.push(e); g_starts.push(g_starts.last().unwrap() + rows); }
+    let mt = *g_starts.last().unwrap();
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let a_dev = cast_f32_f16(dev, &up(&rng(mt*k, 1), vec![mt, k])).unwrap().reshaped(vec![mt, k]);
+    let w_dev = cast_f32_f16(dev, &up(&rng(n_exp*n*k, 2), vec![n_exp, n, k])).unwrap().reshaped(vec![n_exp, n, k]);
+    let (wp4, wsf4, gw4) = lt_fp4_quant_slab(dev, &w_dev, n_exp, n, k).unwrap();
+    let (ap4, asf4, aoff4, ags4) = lt_fp4_quant_grouped(dev, &a_dev, mt, k, &g_starts).unwrap();
+    let _ = moe_grouped_gemm_cutlass_fp4(dev, &ap4, &asf4, &aoff4, &ags4, &wp4, &wsf4, &gw4, &g_starts, &eids, n, k).unwrap();
+    dev.synchronize().ok();
+    let iters = 50;
+    let t = std::time::Instant::now();
+    for _ in 0..iters { let _ = moe_grouped_gemm_cutlass_fp4(dev, &ap4, &asf4, &aoff4, &ags4, &wp4, &wsf4, &gw4, &g_starts, &eids, n, k).unwrap(); }
+    dev.synchronize().ok();
+    let ms = t.elapsed().as_secs_f64() * 1000.0 / iters as f64;
+    let tflops = 2.0 * mt as f64 * n as f64 * k as f64 / (ms * 1e-3) / 1e12;
+    eprintln!("moe_grouped_CUTLASS_bench: mt={mt} N={n} K={k} -> {ms:.3} ms/call, {tflops:.1} TFLOP/s (dense-fp4 ref ~240-306)");
+}
+
+#[test]
+fn f16_gemm_ceiling_bench() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    for (m,n,k) in [(2048usize,2048usize,2048usize),(4096,4096,4096),(2048,10304,2688),(8192,8192,8192)] {
+        let x = cast_f32_f16(dev, &up(&rng(m*k, 1), vec![m,k])).unwrap().reshaped(vec![m,k]);
+        let w = cast_f32_f16(dev, &up(&rng(n*k, 2), vec![n,k])).unwrap().reshaped(vec![n,k]);
+        let _ = gemm_cublas_f32out(dev, &x, &w, m, n, k).unwrap();
+        dev.synchronize().ok();
+        let iters=30; let t=std::time::Instant::now();
+        for _ in 0..iters { let _=gemm_cublas_f32out(dev,&x,&w,m,n,k).unwrap(); }
+        dev.synchronize().ok();
+        let ms=t.elapsed().as_secs_f64()*1000.0/iters as f64;
+        let tf=2.0*m as f64*n as f64*k as f64/(ms*1e-3)/1e12;
+        eprintln!("f16_ceiling m={m} n={n} k={k} -> {ms:.3} ms, {tf:.1} TFLOP/s");
+    }
+}
+
+#[test]
+fn fwht_rotate_invariance() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let (m,n,k) = (64usize, 64usize, 256usize); // k multiple of 128
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let x = cast_f32_f16(dev, &up(&rng(m*k,1), vec![m,k])).unwrap().reshaped(vec![m,k]);
+    let w = cast_f32_f16(dev, &up(&rng(n*k,2), vec![n,k])).unwrap().reshaped(vec![n,k]);
+    let y1 = dl(dev, &gemm_cublas_f32out(dev, &x, &w, m, n, k).unwrap());
+    let xh = fwht128(dev, &x, m, k).unwrap();
+    let wh = fwht128(dev, &w, n, k).unwrap();
+    let y2 = dl(dev, &gemm_cublas_f32out(dev, &xh, &wh, m, n, k).unwrap());
+    let c = cos(&y1, &y2);
+    eprintln!("fwht_rotate_invariance: cos(y1,y2)={c:.6} (want >0.999)");
+    assert!(c > 0.999, "rotate-invariance broken cos={c}");
+}
+
+#[test]
+fn moe_q4_grouped_bench() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let n_exp: usize = std::env::var("NEXP").ok().and_then(|v|v.parse().ok()).unwrap_or(128); let (n, k, rows) = (1856usize, 2688usize, 96usize);
+    let mt = n_exp * rows;
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let a = cast_f32_f16(dev, &up(&rng(mt*k, 1), vec![mt,k])).unwrap().reshaped(vec![mt,k]);
+    let (qs_v, sc_v) = quantize_q4(&rng(n_exp*n*k, 2), n_exp*n, k);
+    let qs_b: Vec<u8> = qs_v.iter().flat_map(|x| x.to_le_bytes()).collect();
+    let qs = Tensor::new(dev.upload(&qs_b).unwrap(), vec![qs_v.len()], DType::U32);
+    let sc = cast_f32_f16(dev, &up(&sc_v, vec![sc_v.len()])).unwrap();
+    let offs: Vec<u8> = (0..=n_exp).flat_map(|e| ((e*rows) as u32).to_le_bytes()).collect();
+    let off = Tensor::new(dev.upload(&offs).unwrap(), vec![n_exp+1], DType::U32);
+    let _ = moe_q4_grouped_mma_dev(dev, &a, &qs, &sc, &off, n_exp, mt, n, k).unwrap();
+    dev.synchronize().ok();
+    let iters = 50;
+    let t = std::time::Instant::now();
+    for _ in 0..iters { let _ = moe_q4_grouped_mma_dev(dev, &a, &qs, &sc, &off, n_exp, mt, n, k).unwrap(); }
+    dev.synchronize().ok();
+    let ms = t.elapsed().as_secs_f64() * 1000.0 / iters as f64;
+    let tf = 2.0 * mt as f64 * n as f64 * k as f64 / (ms * 1e-3) / 1e12;
+    eprintln!("moe_q4_grouped_bench: mt={mt} N={n} K={k} -> {ms:.3} ms, {tf:.1} TFLOP/s (f16 ceiling ~85, cutlass-fp4 ref 60)");
+}
+
+// BUTTER_LAGUNA_SCHED=1 A/B: descending-group-order tile emission + banded
+// N-block/tile grid transpose (see moe_q4_grouped_mma / _dev in wh-butter-ops).
+// Deliberately skewed groups (one big 200-row group, several tiny 3-row
+// groups, an empty group) so the reorder + banding actually rearranges tiles
+// relative to the default expert-order/tile-major launch.
+//
+// The real correctness bar here is BUTTER_LAGUNA_SCHED=1 vs the default path,
+// NOT vs the host f32 reference: both device paths run the identical Q4
+// dequant + mma math per tile (same tile inputs, same K-reduction order
+// inside a tile), the only thing the flag changes is which CTA runs which
+// (tile, N-block) pair and in what order. Since tile-to-output-element
+// mapping is unchanged (disjoint writes per tile) and per-tile accumulation
+// order is unchanged (task constraint: "float accumulation order within a
+// tile is unchanged"), the two orderings should be near bit-exact: 1e-3
+// relative is generous headroom, not a quantization-noise budget. The
+// host-f32 reference is used only as a loose sanity check (cosine) that the
+// Q4 pipeline itself is producing a real signal, not to gate the A/B.
+#[test]
+fn moe_q4_grouped_laguna_sched_ab() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let (n_exp, k, n) = (6usize, 128usize, 128usize);
+    // (expert_id, rows): order is deliberately skewed and includes a
+    // genuinely empty group to exercise the padding-tile path.
+    let groups: [(usize, usize); 6] = [(0, 200), (1, 3), (2, 0), (3, 3), (4, 3), (5, 37)];
+    let mut g_starts = vec![0usize];
+    let mut expert_ids = vec![];
+    for &(e, r) in &groups { expert_ids.push(e); g_starts.push(g_starts.last().unwrap() + r); }
+    let mt = *g_starts.last().unwrap();
+
+    let a_f = rng(mt * k, 41);
+    let w_f = rng(n_exp * n * k, 43);
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let a_dev = cast_f32_f16(dev, &up(&a_f, vec![mt, k])).unwrap().reshaped(vec![mt, k]);
+    let (qs_v, sc_v) = quantize_q4(&w_f, n_exp * n, k);
+    let qs_b: Vec<u8> = qs_v.iter().flat_map(|x| x.to_le_bytes()).collect();
+    let qs = Tensor::new(dev.upload(&qs_b).unwrap(), vec![qs_v.len()], DType::U32);
+    let sc = cast_f32_f16(dev, &up(&sc_v, vec![sc_v.len()])).unwrap();
+
+    // host f32 reference (loose sanity check only, see comment above)
+    let mut exp = vec![0f32; mt * n];
+    for g in 0..groups.len() {
+        let (eid, _) = groups[g];
+        for t in g_starts[g]..g_starts[g + 1] {
+            for nn in 0..n {
+                let mut acc = 0f32;
+                for kk in 0..k { acc += a_f[t * k + kk] * w_f[(eid * n + nn) * k + kk]; }
+                exp[t * n + nn] = acc;
+            }
+        }
+    }
+    let sanity_cos = cos(&exp, &dl(dev, &cast_f16_f32(dev, &moe_q4_grouped_mma(
+        dev, &a_dev, &qs, &sc, &g_starts, &expert_ids, n, k).unwrap()).unwrap()));
+    eprintln!("moe_q4_grouped_laguna_sched_ab: sanity cosine (Q4 default vs f32 host ref) = {sanity_cos:.6}");
+    assert!(sanity_cos > 0.99, "moe_q4_grouped_mma default path looks broken, cosine {sanity_cos:.6} vs host ref");
+
+    // the actual A/B: BUTTER_LAGUNA_SCHED=1 vs default, max relative diff.
+    let rel_diff = |a: &[f32], b: &[f32]| -> f32 {
+        let mut maxr = 0f32;
+        for (x, y) in a.iter().zip(b.iter()) {
+            let rel = (x - y).abs() / x.abs().max(1e-2);
+            if rel > maxr { maxr = rel; }
+        }
+        maxr
+    };
+
+    // --- host-descriptor path (moe_q4_grouped_mma) ---
+    let out_default = dl(dev, &cast_f16_f32(dev, &moe_q4_grouped_mma(
+        dev, &a_dev, &qs, &sc, &g_starts, &expert_ids, n, k).unwrap()).unwrap());
+    unsafe { std::env::set_var("BUTTER_LAGUNA_SCHED", "1"); }
+    let out_sched = dl(dev, &cast_f16_f32(dev, &moe_q4_grouped_mma(
+        dev, &a_dev, &qs, &sc, &g_starts, &expert_ids, n, k).unwrap()).unwrap());
+    unsafe { std::env::remove_var("BUTTER_LAGUNA_SCHED"); }
+    let r_host = rel_diff(&out_default, &out_sched);
+    eprintln!("moe_q4_grouped_laguna_sched_ab (host path): sched-vs-default max_rel={r_host:.2e}");
+    assert!(r_host < 1e-3, "host path: BUTTER_LAGUNA_SCHED=1 vs default max_rel {r_host:.2e} > 1e-3");
+
+    // --- fully-on-device descriptor path (moe_q4_grouped_mma_dev) ---
+    let offs: Vec<u8> = g_starts.iter().flat_map(|&s| (s as u32).to_le_bytes()).collect();
+    let off = Tensor::new(dev.upload(&offs).unwrap(), vec![g_starts.len()], DType::U32);
+    let out_dev_default = dl(dev, &cast_f16_f32(dev, &moe_q4_grouped_mma_dev(
+        dev, &a_dev, &qs, &sc, &off, n_exp, mt, n, k).unwrap()).unwrap());
+    unsafe { std::env::set_var("BUTTER_LAGUNA_SCHED", "1"); }
+    let out_dev_sched = dl(dev, &cast_f16_f32(dev, &moe_q4_grouped_mma_dev(
+        dev, &a_dev, &qs, &sc, &off, n_exp, mt, n, k).unwrap()).unwrap());
+    unsafe { std::env::remove_var("BUTTER_LAGUNA_SCHED"); }
+    let r_dev = rel_diff(&out_dev_default, &out_dev_sched);
+    eprintln!("moe_q4_grouped_laguna_sched_ab (dev path): sched-vs-default max_rel={r_dev:.2e}");
+    assert!(r_dev < 1e-3, "dev path: BUTTER_LAGUNA_SCHED=1 vs default max_rel {r_dev:.2e} > 1e-3");
+
+    // the dev-path descriptor builder (on-device, offsets-only) must also
+    // agree with the host-descriptor builder for both orderings.
+    let r_cross_default = rel_diff(&out_default, &out_dev_default);
+    let r_cross_sched = rel_diff(&out_sched, &out_dev_sched);
+    eprintln!("moe_q4_grouped_laguna_sched_ab: host-vs-dev max_rel default={r_cross_default:.2e} sched={r_cross_sched:.2e}");
+    assert!(r_cross_default < 1e-3 && r_cross_sched < 1e-3, "host vs dev descriptor builders disagree");
+
+    eprintln!("moe_q4_grouped_laguna_sched_ab: PASS (default vs BUTTER_LAGUNA_SCHED=1 agree within 1e-3 on skewed groups)");
+}
+
+#[test]
+fn moe_cutlass_f16_bench() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let n_exp: usize = std::env::var("NEXP").ok().and_then(|v|v.parse().ok()).unwrap_or(128); let (n, k, rows) = (1856usize, 2688usize, 96usize);
+    let mut g_starts = vec![0usize]; let mut eids = vec![];
+    for e in 0..n_exp { eids.push(e); g_starts.push(g_starts.last().unwrap() + rows); }
+    let mt = *g_starts.last().unwrap();
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let a = cast_f32_f16(dev, &up(&rng(mt*k, 1), vec![mt,k])).unwrap().reshaped(vec![mt,k]);
+    let w = cast_f32_f16(dev, &up(&rng(n_exp*n*k, 2), vec![n_exp*n, k])).unwrap().reshaped(vec![n_exp*n, k]);
+    let _ = moe_grouped_gemm_cutlass(dev, &a, &w, &g_starts, &eids, n, k).unwrap();
+    dev.synchronize().ok();
+    let iters = 50;
+    let t = std::time::Instant::now();
+    for _ in 0..iters { let _ = moe_grouped_gemm_cutlass(dev, &a, &w, &g_starts, &eids, n, k).unwrap(); }
+    dev.synchronize().ok();
+    let ms = t.elapsed().as_secs_f64() * 1000.0 / iters as f64;
+    let tf = 2.0 * mt as f64 * n as f64 * k as f64 / (ms * 1e-3) / 1e12;
+    eprintln!("moe_cutlass_f16_bench: mt={mt} N={n} K={k} -> {ms:.3} ms, {tf:.1} TFLOP/s (q4-grouped ref 13.7, cutlass-fp4 60)");
+}
+
+// Isolates the bf16-stream NaN: run rms_norm + gemm_tc in f32 vs bf16, report cosine + NaN.
+#[test]
+fn bf16_kernel_isolate() {
+    let Some(d) = CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev = d.as_ref();
+    let (rows, hid) = (2048usize, 2688usize);
+    let up = |v: &[f32], sh: Vec<usize>| -> Tensor {
+        Tensor::new(dev.upload(&tb(v)).unwrap(), vec![v.len()], DType::F32).reshaped(sh) };
+    let x = up(&rng(rows*hid, 7), vec![rows, hid]);
+    let w = up(&rng(hid, 9), vec![hid]);
+    let cos = |a: &[f32], b: &[f32]| -> f32 {
+        let (mut dot, mut na, mut nb) = (0f64,0f64,0f64);
+        for i in 0..a.len() { dot+=a[i] as f64*b[i] as f64; na+=(a[i] as f64).powi(2); nb+=(b[i] as f64).powi(2); }
+        (dot/(na.sqrt()*nb.sqrt())) as f32 };
+    let has_nan = |v: &[f32]| v.iter().any(|x| x.is_nan());
+
+    // --- rms_norm f32 vs bf16(wide) ---
+    let n_f32 = dl(dev, &rms_norm(dev, &x, &w, 1e-5).unwrap());
+    let xb = cast_f32_bf16(dev, &x).unwrap();
+    let wb = cast_f32_bf16(dev, &w).unwrap();
+    let n_bf = rms_norm(dev, &xb, &wb, 1e-5).unwrap();
+    let n_bf_f = dl(dev, &cast_bf16_f32(dev, &n_bf).unwrap());
+    eprintln!("RMS_NORM bf16-wide: NaN={}  cosine_vs_f32={:.6}", has_nan(&n_bf_f), cos(&n_f32, &n_bf_f));
+
+    // --- gemm_tc f32-ref vs bf16 ---  out[rows,hid] = x[rows,hid] . W[hid,hid]^T
+    let wsq = up(&rng(hid*hid, 11), vec![hid, hid]);
+    let xh = cast_f32_f16(dev, &x).unwrap();
+    let wh = cast_f32_f16(dev, &wsq).unwrap();
+    let g_f16 = dl(dev, &cast_f16_f32(dev, &gemm_cublas(dev, &xh, &wh, rows, hid, hid).unwrap()).unwrap());
+    let xbb = cast_f32_bf16(dev, &x).unwrap();
+    let wbb = cast_f32_bf16(dev, &wsq).unwrap();
+    let g_bf = dl(dev, &cast_bf16_f32(dev, &gemm_cublas(dev, &xbb, &wbb, rows, hid, hid).unwrap()).unwrap());
+    eprintln!("GEMM_TC bf16: NaN={}  cosine_vs_f16={:.6}", has_nan(&g_bf), cos(&g_f16, &g_bf));
+}
+
+#[test]
+fn fp8_rate_bench() {
+    let Some(d)=CudaDevice::create().expect("cuda") else { eprintln!("no CUDA"); return; };
+    let dev=d.as_ref();
+    let up=|v:&[f32],sh:Vec<usize>|->Tensor{ Tensor::new(dev.upload(&tb(v)).unwrap(),vec![v.len()],DType::F32).reshaped(sh) };
+    for (m,n,k) in [(2048usize,2688usize,2688usize),(2048,8192,2688),(4096,4096,4096)] {
+        let x=up(&rng(m*k,1),vec![m,k]); let w=up(&rng(n*k,2),vec![n,k]);
+        // fp8 path
+        let (xq,xsc)=fp8_quant(dev,&cast_f32_f16(dev,&x).unwrap(),m*k).unwrap();
+        let (wq,wsc)=fp8_quant(dev,&cast_f32_f16(dev,&w).unwrap(),n*k).unwrap();
+        let _=gemm_fp8(dev,&xq,&xsc,&wq,&wsc,m,n,k,true).unwrap(); dev.synchronize().ok();
+        let it=50; let t=std::time::Instant::now();
+        for _ in 0..it { let _=gemm_fp8(dev,&xq,&xsc,&wq,&wsc,m,n,k,true).unwrap(); }
+        dev.synchronize().ok(); let ms8=t.elapsed().as_secs_f64()*1000.0/it as f64;
+        let tf8=2.0*m as f64*n as f64*k as f64/(ms8*1e-3)/1e12;
+        // f16 path
+        let xh=cast_f32_f16(dev,&x).unwrap(); let wh=cast_f32_f16(dev,&w).unwrap();
+        let _=gemm_cublas(dev,&xh,&wh,m,n,k).unwrap(); dev.synchronize().ok();
+        let t=std::time::Instant::now();
+        for _ in 0..it { let _=gemm_cublas(dev,&xh,&wh,m,n,k).unwrap(); }
+        dev.synchronize().ok(); let ms16=t.elapsed().as_secs_f64()*1000.0/it as f64;
+        let tf16=2.0*m as f64*n as f64*k as f64/(ms16*1e-3)/1e12;
+        eprintln!("fp8_rate_bench m={m} n={n} k={k}: fp8 {ms8:.3}ms {tf8:.1} TFLOP/s | f16 {ms16:.3}ms {tf16:.1} TFLOP/s | fp8/f16 speedup {:.2}x", tf8/tf16);
+    }
+}
diff --git a/rust/crates/wh-butter-core/src/lib.rs b/rust/crates/wh-butter-core/src/lib.rs
index b906af0c..a70042f1 100644
--- a/rust/crates/wh-butter-core/src/lib.rs
+++ b/rust/crates/wh-butter-core/src/lib.rs
@@ -269,6 +269,34 @@ pub trait Device: Send + Sync {
     ) -> Result<()> {
         Err(Error::Msg("moe_grouped_cutlass unsupported on this backend".into()))
     }
+    /// Validated NVFP4 grouped-MoE GEMM (marlin). Device buffers; routing arrays
+    /// (`sorted_token_ids`/`expert_ids`/`num_tokens_past_padded`) are DEVICE i32.
+    #[allow(clippy::too_many_arguments)]
+    fn moe_marlin_gemm(
+        &self,
+        _a: &dyn DeviceBuffer, _b: &dyn DeviceBuffer, _c: &dyn DeviceBuffer, _c_tmp: &dyn DeviceBuffer,
+        _b_scales: &dyn DeviceBuffer, _global_scale: &dyn DeviceBuffer, _topk_weights: &dyn DeviceBuffer,
+        _sorted_token_ids: &dyn DeviceBuffer, _expert_ids: &dyn DeviceBuffer, _num_tokens_past_padded: &dyn DeviceBuffer,
+        _workspace: &dyn DeviceBuffer,
+        _moe_block: usize, _num_experts: usize, _top_k: usize, _mul_topk_weights: bool,
+        _prob_m: usize, _prob_n: usize, _prob_k: usize, _num_groups: usize, _group_size: usize,
+        _sms: i32, _use_fp32_reduce: bool,
+    ) -> Result<()> {
+        Err(Error::Msg("moe_marlin_gemm unsupported on this backend".into()))
+    }
+
+    /// Repack one expert's GPTQ-packed `[K/8,N]` u32 weights into marlin tile layout.
+    #[allow(clippy::too_many_arguments)]
+    fn marlin_repack(
+        &self, _b_q: &dyn DeviceBuffer, _out: &dyn DeviceBuffer,
+        _size_k: usize, _size_n: usize, _sms: i32, _max_shared_mem: i32,
+    ) -> Result<()> {
+        Err(Error::Msg("marlin_repack unsupported on this backend".into()))
+    }
+    #[allow(clippy::too_many_arguments)]
+    fn marlin_build_routing(&self, _off:&dyn DeviceBuffer,_n_exp:usize,_blk:usize,_mt:usize,_stid:&dyn DeviceBuffer,_eid:&dyn DeviceBuffer,_ntpp:&dyn DeviceBuffer)->Result<()>{
+        Err(Error::Msg("marlin_build_routing unsupported on this backend".into()))
+    }
 
     /// CUTLASS grouped block-scaled NVFP4 MoE GEMM: `out[t,n] = Σ_k a[t,k]·
     /// w[eid(t)][n,k]` over sorted-token groups, both operands packed e2m1 +
diff --git a/rust/crates/wh-butter-loader/src/gguf.rs b/rust/crates/wh-butter-loader/src/gguf.rs
index a9d6d7cf..da58b160 100644
--- a/rust/crates/wh-butter-loader/src/gguf.rs
+++ b/rust/crates/wh-butter-loader/src/gguf.rs
@@ -67,6 +67,11 @@ pub struct Gguf {
     pub metadata_str: BTreeMap<String, String>,
     /// Scalar f32 metadata (e.g. `qwen2.rope.freq_base`, `*.rms_norm_eps`).
     pub metadata_f32: BTreeMap<String, f32>,
+    /// The path passed to `open` (part 0 for a split GGUF), kept so callers
+    /// that need to identify the source file on disk (e.g. an on-disk cache
+    /// keyed off the file's size/mtime) don't have to thread the original
+    /// path alongside every `&Gguf` themselves.
+    source_path: String,
     /// String-array metadata (e.g. `tokenizer.ggml.tokens`, `.merges`).
     pub metadata_arr_str: BTreeMap<String, Vec<String>>,
     /// Int-array metadata (e.g. `tokenizer.ggml.token_type`).
@@ -355,6 +360,7 @@ impl Gguf {
             parts,
             data_starts,
             tensors,
+            source_path: path.to_string(),
             metadata_u32: m.metadata_u32,
             metadata_str: m.metadata_str,
             metadata_f32: m.metadata_f32,
@@ -364,6 +370,12 @@ impl Gguf {
         })
     }
 
+    /// The path this `Gguf` was opened from (part 0's path for a split GGUF,
+    /// see [`Self::open`]).
+    pub fn path(&self) -> &str {
+        &self.source_path
+    }
+
     /// Read a `u32`/int scalar from metadata under `key`.
     pub fn meta_u32(&self, key: &str) -> Option<u32> {
         self.metadata_u32.get(key).copied()
diff --git a/rust/crates/wh-butter-models/Cargo.toml b/rust/crates/wh-butter-models/Cargo.toml
index 05a1ccc9..b414b84e 100644
--- a/rust/crates/wh-butter-models/Cargo.toml
+++ b/rust/crates/wh-butter-models/Cargo.toml
@@ -11,3 +11,4 @@ wh-butter-core.workspace = true
 wh-butter-ops.workspace = true
 wh-butter-loader.workspace = true
 serde_json.workspace = true
+rayon.workspace = true
diff --git a/rust/crates/wh-butter-models/src/laguna.rs b/rust/crates/wh-butter-models/src/laguna.rs
new file mode 100644
index 00000000..340bf737
--- /dev/null
+++ b/rust/crates/wh-butter-models/src/laguna.rs
@@ -0,0 +1,2979 @@
+// Copyright 2026 Eric Kryski (@ekryski) and Tom Turney (@TheTom)
+// SPDX-License-Identifier: Apache-2.0
+
+//! Laguna-S-2.1 (GGUF arch "laguna"): a 117.55B / 8.14B-active sigmoid-routed
+//! MoE transformer with a per-head softplus attention-output gate, per-head
+//! QK-RMSNorm, and hybrid full/sliding-window attention (period-4, dense-
+//! first). Layer 0 is a dense SwiGLU MLP (`mlp_only_layers = [0]`); layers
+//! 1..47 are MoE (256 experts, top-10, sigmoid router with a score-
+//! correction bias, one always-on shared expert). Full-attention layers use
+//! YaRN RoPE over the first half of each head's dims; sliding-window layers
+//! (window 512) use plain RoPE over the full head dim.
+//!
+//! Decode: single-token forward with a persistent KV cache (growing for
+//! full-attention layers, a fixed-512 ring buffer for sliding layers).
+//! Prefill: chunked batched forward (see [`prefill`]) with a linear sliding
+//! window scratch compacted into the decode ring afterward.
+//!
+//! Weights load straight from GGUF (see [`wh_butter_loader::gguf::Gguf`]):
+//! dequantize each tensor to f32 once, then requantize into this engine's Q4
+//! format ([`wh_butter_ops::quantize_q4`]) and drop the f32, the GGUF k-quant
+//! (Q4_K/Q6_K/...) never needs to match our block layout. Embeddings, the LM
+//! head, and all norms stay dense f32 (small relative to the expert stacks
+//! and precision-sensitive for routing/final logits).
+
+use wh_butter_core::{DType, Device, Error, Result, Tensor};
+use wh_butter_loader::gguf::Gguf;
+use wh_butter_ops as ops;
+use rayon::prelude::*;
+use std::sync::atomic::{AtomicU64, Ordering};
+
+fn u32s_to_bytes(v: &[u32]) -> Vec<u8> {
+    v.iter().flat_map(|x| x.to_le_bytes()).collect()
+}
+fn f32s_to_bytes(v: &[f32]) -> Vec<u8> {
+    v.iter().flat_map(|x| x.to_le_bytes()).collect()
+}
+/// The `iron_gemv_q4_coalesced*` kernels read per-block scales as f16 (the
+/// `d_f32: Tensor<f16>` param, name notwithstanding); uploading raw f32 bits
+/// there silently reinterprets them as garbage f16s. Convert on the host at
+/// load, exactly like the validated `qload(.., f16=true)` path in modeltests.
+fn f32s_to_f16_bytes(v: &[f32]) -> Vec<u8> {
+    fn half_bits(f: f32) -> u16 {
+        let x = f.to_bits();
+        let sign = ((x >> 16) & 0x8000) as u16;
+        let e = ((x >> 23) & 0xff) as i32 - 112;
+        if e <= 0 { return sign; }
+        if e >= 0x1f { return sign | 0x7c00; }
+        let m = (x >> 13) & 0x3ff;
+        let round = (x >> 12) & 1;
+        let v = ((e as u32) << 10) | m;
+        sign | ((v + round) as u16)
+    }
+    v.iter().flat_map(|&f| half_bits(f).to_le_bytes()).collect()
+}
+
+// ─────────────────────────────────────────────────────────────────────────
+// On-disk cache of converted engine-format weight blobs.
+//
+// `load_gguf` is the ~7.5min-cold-load path this exists to shortcut: per
+// tensor it dequantizes GGUF (Q4_K/Q6_K/F16) to f32 on one CPU thread,
+// requantizes to this engine's Q4 format (`ops::quantize_q4`), optionally
+// permutes/interleaves/packs for BUTTER_LAGUNA_MARLIN / _MOEFUSE / _W4A8, then
+// uploads. Every one of those UPLOAD-READY byte blobs is cached here, one
+// file per blob, so a warm reload skips straight to `read file -> upload`.
+//
+// LOUD REMINDER: bump `FORMAT_VERSION` whenever ANY of that byte-shaping
+// logic changes (quantize_q4's block layout, permute_q4_to_marlin, the
+// moefuse interleave order, w4a8_packw's packed layout, the f16 scale
+// conversion, ...). Bumping it changes every blob's cache key, so stale
+// pre-bump blobs are simply never read back (treated as a miss and rebuilt)
+// instead of silently reinterpreted.
+const FORMAT_VERSION: u32 = 1;
+
+const CACHE_MAGIC: [u8; 4] = *b"FLC1"; // "Butter Laguna Cache v1"
+
+// `dtype_tag` is purely descriptive (a debugging aid if someone inspects a
+// cache dir by hand), the cache never interprets a blob's bytes itself, it
+// only ever hands them back verbatim to `dev.upload`.
+const DTYPE_U32: u8 = 0;
+const DTYPE_F32: u8 = 1;
+const DTYPE_F16: u8 = 2;
+/// Opaque packed bytes (Marlin-tile-permuted qs, W4A8 mxfp4 packed weights /
+/// scale factors, ...): anything that isn't a flat array of one of the
+/// above elementary types.
+const DTYPE_RAW: u8 = 3;
+
+fn fnv1a64(bytes: &[u8]) -> u64 {
+    let mut h: u64 = 0xcbf29ce484222325;
+    for &b in bytes {
+        h ^= b as u64;
+        h = h.wrapping_mul(0x100000001b3);
+    }
+    h
+}
+
+/// Header: magic (4B) + FORMAT_VERSION (u32 LE) + dtype tag (1B) + payload
+/// len (u64 LE), then the payload. Written atomically (tmp file + rename) so
+/// a crash mid-write never leaves a corrupt file at the final path, readers
+/// only ever see a complete blob or no file at all.
+fn write_cache_blob(path: &std::path::Path, dtype_tag: u8, payload: &[u8]) -> std::io::Result<()> {
+    use std::io::Write;
+    let tmp = path.with_extension(format!("tmp-{}-{:?}", std::process::id(), std::thread::current().id()));
+    {
+        let mut f = std::fs::File::create(&tmp)?;
+        f.write_all(&CACHE_MAGIC)?;
+        f.write_all(&FORMAT_VERSION.to_le_bytes())?;
+        f.write_all(&[dtype_tag])?;
+        f.write_all(&(payload.len() as u64).to_le_bytes())?;
+        f.write_all(payload)?;
+    }
+    std::fs::rename(&tmp, path)
+}
+
+/// Reads back a blob written by [`write_cache_blob`]. Returns `None` for any
+/// validation failure (missing file, bad magic, mismatched FORMAT_VERSION,
+/// truncated payload), every failure mode is treated as a plain cache miss,
+/// never an error, so a corrupt or foreign-version cache dir just gets
+/// silently rebuilt.
+fn read_cache_blob(path: &std::path::Path) -> Option<Vec<u8>> {
+    let bytes = std::fs::read(path).ok()?;
+    if bytes.len() < 17 || bytes[0..4] != CACHE_MAGIC {
+        return None;
+    }
+    let fv = u32::from_le_bytes(bytes[4..8].try_into().unwrap());
+    if fv != FORMAT_VERSION {
+        return None;
+    }
+    let len = u64::from_le_bytes(bytes[9..17].try_into().unwrap()) as usize;
+    let payload = &bytes[17..];
+    if payload.len() != len {
+        return None;
+    }
+    Some(payload.to_vec())
+}
+
+/// On-disk cache of converted engine-format weight blobs, keyed by a
+/// caller-chosen artifact key (conventionally the GGUF tensor name, possibly
+/// suffixed for composite/variant artifacts) folded together with
+/// [`FORMAT_VERSION`], the source GGUF's identity (size + mtime), and every
+/// `BUTTER_LAGUNA_*` load-shaping env flag, so a changed checkpoint, a changed
+/// requant flag combo, or a bumped `FORMAT_VERSION` each land in their own
+/// namespace within the same cache directory rather than colliding.
+///
+/// `Send + Sync` (every field is): built once per [`load_gguf`] call and
+/// shared (via `&CacheCtx`) across the rayon worker threads that build the
+/// per-layer byte blobs in parallel on a cache miss.
+pub struct CacheCtx {
+    /// `None` when the cache directory couldn't be created, every lookup
+    /// then unconditionally misses (rebuild every artifact, every load), so
+    /// a read-only or unwritable filesystem degrades to "cache disabled"
+    /// instead of failing the load.
+    dir: Option<std::path::PathBuf>,
+    version_tag: u64,
+    hits: AtomicU64,
+    misses: AtomicU64,
+}
+
+impl CacheCtx {
+    /// `BUTTER_LAGUNA_CACHE` overrides the cache directory; otherwise it lives
+    /// alongside the GGUF at `<gguf_path>.butter-cache/`.
+    pub fn new(g: &Gguf) -> Self {
+        let path = g.path();
+        let dir = match std::env::var("BUTTER_LAGUNA_CACHE") {
+            Ok(d) if !d.is_empty() => std::path::PathBuf::from(d),
+            _ => std::path::PathBuf::from(format!("{path}.butter-cache")),
+        };
+        let dir = match std::fs::create_dir_all(&dir) {
+            Ok(()) => Some(dir),
+            Err(e) => {
+                eprintln!(
+                    "Laguna cache: could not create cache dir {} ({e}), caching disabled, every load will fully rebuild",
+                    dir.display()
+                );
+                None
+            }
+        };
+        // Source-GGUF identity: (size, mtime) rather than a content hash,
+        // hashing an 80GB+ checkpoint on every load would itself dominate
+        // load time. A changed file (new export, re-quant, ...) almost
+        // always changes at least one of these; a false-negative (same
+        // size+mtime, different content) is the same class of risk `make`
+        // and every mtime-based build cache accepts.
+        let (size, mtime) = std::fs::metadata(path)
+            .map(|m| {
+                let mtime = m
+                    .modified()
+                    .ok()
+                    .and_then(|t| t.duration_since(std::time::UNIX_EPOCH).ok())
+                    .map(|d| d.as_secs())
+                    .unwrap_or(0);
+                (m.len(), mtime)
+            })
+            .unwrap_or((0, 0));
+        let flags = ["BUTTER_LAGUNA_MICRO", "BUTTER_LAGUNA_MOEFUSE", "BUTTER_LAGUNA_MARLIN", "BUTTER_LAGUNA_W4A8", "BUTTER_LAGUNA_W4A8_ONLY"]
+            .iter()
+            .map(|k| format!("{k}={}", std::env::var(k).unwrap_or_default()))
+            .collect::<Vec<_>>()
+            .join("|");
+        let version_tag = fnv1a64(format!("v{FORMAT_VERSION}|{size}|{mtime}|{flags}").as_bytes());
+        CacheCtx { dir, version_tag, hits: AtomicU64::new(0), misses: AtomicU64::new(0) }
+    }
+
+    fn blob_path(&self, key: &str) -> Option<std::path::PathBuf> {
+        let dir = self.dir.as_ref()?;
+        let mut sanitized = String::with_capacity(key.len());
+        for c in key.chars() {
+            if c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-') {
+                sanitized.push(c);
+            } else {
+                sanitized.push('_');
+            }
+        }
+        let h = fnv1a64(format!("{key}\u{0}{}", self.version_tag).as_bytes());
+        Some(dir.join(format!("{sanitized}-{h:016x}.bin")))
+    }
+
+    /// One cached byte blob. `build` runs (and its result gets written to
+    /// disk) only on a miss.
+    pub fn get_or_build(&self, key: &str, dtype_tag: u8, build: impl FnOnce() -> Result<Vec<u8>>) -> Result<Vec<u8>> {
+        if let Some(path) = self.blob_path(key) {
+            if let Some(bytes) = read_cache_blob(&path) {
+                self.hits.fetch_add(1, Ordering::Relaxed);
+                return Ok(bytes);
+            }
+        }
+        let bytes = build()?;
+        self.misses.fetch_add(1, Ordering::Relaxed);
+        if let Some(path) = self.blob_path(key) {
+            if let Err(e) = write_cache_blob(&path, dtype_tag, &bytes) {
+                eprintln!("Laguna cache: failed writing '{key}': {e}");
+            }
+        }
+        Ok(bytes)
+    }
+
+    /// Same contract as [`Self::get_or_build`] but for an artifact whose
+    /// bytes are naturally co-produced by ONE expensive build step (e.g. a
+    /// `Q4Dense`'s `qs` + `scales`, both products of a single
+    /// `quantize_q4` call, or a `Q4Dense`+`MarlinDense` pair sharing one
+    /// `quantize_q4` pass). Checks every key first; `build` runs at most
+    /// once, only if at least one key is missing: so a full hit never
+    /// redoes the shared work, and a full miss never redoes it either.
+    pub fn get_or_build_multi(&self, keys: &[(&str, u8)], build: impl FnOnce() -> Result<Vec<Vec<u8>>>) -> Result<Vec<Vec<u8>>> {
+        let paths: Vec<Option<std::path::PathBuf>> = keys.iter().map(|(k, _)| self.blob_path(k)).collect();
+        let cached: Option<Vec<Vec<u8>>> = paths.iter().map(|p| p.as_ref().and_then(|p| read_cache_blob(p))).collect();
+        if let Some(all) = cached {
+            self.hits.fetch_add(keys.len() as u64, Ordering::Relaxed);
+            return Ok(all);
+        }
+        let blobs = build()?;
+        if blobs.len() != keys.len() {
+            return Err(Error::Msg(format!("Laguna cache: build produced {} blobs, expected {} for keys {:?}", blobs.len(), keys.len(), keys.iter().map(|(k, _)| *k).collect::<Vec<_>>())));
+        }
+        self.misses.fetch_add(keys.len() as u64, Ordering::Relaxed);
+        for ((key, dtype), (path, bytes)) in keys.iter().zip(paths.iter().zip(blobs.iter())) {
+            if let Some(path) = path {
+                if let Err(e) = write_cache_blob(path, *dtype, bytes) {
+                    eprintln!("Laguna cache: failed writing '{key}': {e}");
+                }
+            }
+        }
+        Ok(blobs)
+    }
+
+    pub fn stats(&self) -> (u64, u64) {
+        (self.hits.load(Ordering::Relaxed), self.misses.load(Ordering::Relaxed))
+    }
+
+    pub fn dir_display(&self) -> String {
+        match &self.dir {
+            Some(d) => d.display().to_string(),
+            None => "<disabled>".to_string(),
+        }
+    }
+}
+
+/// One `[m, k]` row-major f32 weight requantized to this engine's Q4 format
+/// and uploaded. `m`/`k` match the `gemv_q4` convention (`k` must be a
+/// multiple of 32).
+struct Q4Dense {
+    qs: Tensor,
+    scales: Tensor,
+    m: usize,
+    k: usize,
+}
+impl Q4Dense {
+    // Construction (dequant -> quantize_q4 -> upload, or the cache-hit
+    // shortcut straight from disk) now lives in the free functions around
+    // `q4dense_bytes_from_flat_cached` / `upload_q4dense_bytes` below, so
+    // every construction site (plain, Marlin-shared, cached, parallel-phase)
+    // goes through the same on-disk-cache-aware path instead of each having
+    // its own uncached `dev.upload` call.
+    fn gemv(&self, dev: &dyn Device, x: &Tensor) -> Result<Tensor> {
+        ops::gemv_q4(dev, &self.qs, &self.scales, x, self.m, self.k, self.m)
+    }
+    fn gemv_accum(&self, dev: &dyn Device, x: &Tensor, acc: &Tensor, scale: &Tensor) -> Result<()> {
+        ops::gemv_q4_accum(dev, &self.qs, &self.scales, x, acc, scale, self.m, self.k, self.m)
+    }
+}
+
+/// One `[n_out, k_in]` row-major f32 weight, requantized to Q4 and permuted
+/// into the tensor-core Marlin tile-major layout ([`ops::permute_q4_to_marlin`])
+/// for [`ops::moe_w4a16_marlin`]. Built ONLY under `BUTTER_LAGUNA_MARLIN=1`, as
+/// an ADDITIONAL resident copy alongside the plain [`Q4Dense`] decode uses,
+/// this struct is prefill-only (see `prefill_layer`'s marlin branches).
+///
+/// `n_out` must be a multiple of 64 (`permute_q4_to_marlin`'s tile height);
+/// `k_in` a multiple of 32 (Q4 block size). A dense `[n_out, k_in]` matrix is
+/// fed to the (expert-batched) Marlin GEMM as a single "expert" (`n_exp=1`):
+/// every row of `x` carries expert-id 0 via a persistent all-zero `indices`
+/// buffer (see `prefill`'s `marlin_zero_idx`).
+struct MarlinDense {
+    qs: Tensor,     // Marlin tile-major Q4: [(n_out/64)*(k_in/32)*256] u32
+    scales: Tensor, // f16, STANDARD (non-permuted) per-block layout: [n_out*(k_in/32)].
+    // `moe_w4a16_marlin`'s kernel doc is explicit that scales are read in the
+    // SAME layout `quantize_q4` produces; only `qs` gets tile-permuted.
+    n_out: usize,
+    k_in: usize,
+}
+impl MarlinDense {
+    // Construction lives in `marlindense_bytes_from_flat_cached` /
+    // `upload_marlindense_bytes` below (same rationale as `Q4Dense`'s impl
+    // block doc above).
+    /// `x` is `[t, k_in]` f16; `zero_idx` is a persistent all-zero `u32`
+    /// buffer with at least `t` elements (the 1-expert "every row is expert
+    /// 0" trick). Returns `[t, n_out]` f16, cast back with `ops::cast_f16_f32`
+    /// wherever the caller needs f32 (see `prefill_layer`).
+    fn call(&self, dev: &dyn Device, x_f16: &Tensor, zero_idx: &Tensor, t: usize) -> Result<Tensor> {
+        ops::moe_w4a16_marlin(dev, x_f16, &self.qs, &self.scales, zero_idx, t, self.n_out, self.k_in)
+    }
+}
+
+/// BUTTER_LAGUNA_MARLIN=1: route dense prefill projections (QKV concat, o_proj,
+/// layer-0 dense FFN, shared-expert FFN) through the tensor-core
+/// [`MarlinDense`] GEMM instead of `gemm_q4_mpp`. Read once at load (to build
+/// the extra Marlin-layout weight copies below) and once per `prefill_layer`
+/// call (mirrors `laguna_micro_enabled`'s load/call split). Decode is
+/// entirely unaffected, it never reads this env.
+fn laguna_marlin_enabled() -> bool {
+    std::env::var("BUTTER_LAGUNA_MARLIN").map(|v| v == "1").unwrap_or(false)
+}
+
+/// Marlin-packed `[wq;wk;wv]` concat (gate excluded, its row count, 48 or 72,
+/// is never 64-aligned, so it stays on `gemm_q4_mpp` via `AttnWeights::g_proj`).
+/// `n_q_rows + 2*n_kv_rows` is 64-aligned for both Laguna-S-2.1 layer types
+/// (full: 6144+2*1024=8192, sliding: 9216+2*1024=11264), verified at build
+/// time rather than assumed, see `load_gguf`.
+struct QkvMarlin {
+    mat: MarlinDense, // [n_q_rows + 2*n_kv_rows, hidden]
+    n_q_rows: usize,
+    n_kv_rows: usize,
+}
+
+/// Marlin-packed `[gate;up]` concat (both halves share `n_ff` rows and
+/// `hidden` columns), used for the dense layer-0 FFN and the always-on
+/// shared expert, both of whose `2*n_ff` is 64-aligned for Laguna-S-2.1
+/// (24576 and 2048 respectively).
+struct GateUpMarlin {
+    mat: MarlinDense, // [2*n_ff, hidden]
+    n_ff: usize,
+}
+
+/// The always-on shared expert's Marlin copies (gate/up concat + down),
+/// built regardless of whether decode's [`SharedExpert`] resolved to
+/// `Split` or `Fused`, this struct reads the shexp GGUF tensors directly.
+struct SharedMarlin {
+    gateup: GateUpMarlin,
+    down: MarlinDense, // [hidden, n_ff]
+}
+
+/// `n_expert` stacked `[m, k]` Q4 weights, requantized expert-by-expert and
+/// concatenated into one device buffer per projection. A per-expert view is
+/// a byte-offset slice, no host round-trip to select an expert's weights,
+/// only the (tiny) expert-id lookup needs to be on the host.
+struct Q4ExpertStack {
+    qs: Tensor,
+    scales: Tensor,
+    m: usize,
+    k: usize,
+    qs_words_per_expert: usize,
+    scale_elems_per_expert: usize,
+}
+impl Q4ExpertStack {
+    // Construction lives in `q4_expert_stack_bytes_cached` /
+    // `upload_expert_stack_bytes` below (same rationale as `Q4Dense`'s impl
+    // block doc above); the per-expert `quantize_q4` loop that used to live
+    // here is now inside that cached builder's closure.
+    /// Device-side view of expert `e`'s `(qs, scales)`, a byte-offset slice
+    /// into the shared stack, no copy.
+    fn expert_view(&self, e: usize) -> (Tensor, Tensor) {
+        let qw = self.qs_words_per_expert;
+        let sw = self.scale_elems_per_expert;
+        let qs = Tensor {
+            buffer: self.qs.buffer.clone(),
+            offset: self.qs.offset + e * qw * 4,
+            shape: vec![qw],
+            dtype: DType::U32,
+        };
+        let sc = Tensor {
+            buffer: self.scales.buffer.clone(),
+            offset: self.scales.offset + e * sw * 2, // f16 scales: 2 bytes each
+            shape: vec![sw],
+            dtype: DType::F16,
+        };
+        (qs, sc)
+    }
+    fn gemv(&self, dev: &dyn Device, e: usize, x: &Tensor) -> Result<Tensor> {
+        let (qs, sc) = self.expert_view(e);
+        ops::gemv_q4(dev, &qs, &sc, x, self.m, self.k, self.m)
+    }
+    fn gemv_accum(&self, dev: &dyn Device, e: usize, x: &Tensor, acc: &Tensor, scale: &Tensor) -> Result<()> {
+        let (qs, sc) = self.expert_view(e);
+        ops::gemv_q4_accum(dev, &qs, &sc, x, acc, scale, self.m, self.k, self.m)
+    }
+}
+
+/// BUTTER_LAGUNA_W4A8=1: one `[n_expert, n, k]` dequantized f32 expert-major
+/// slab packed into the mxfp4 (e2m1 nibbles + per-32-block ue8m0 scale
+/// factor) layout `ops::moe_grouped_gemm_w4a8` consumes as its weight
+/// operand. Mirrors the NEMOTRON_W4A8_MOE loader's `buildw` closure
+/// (`wh-butter-modeltests/src/lib.rs`, around `NEMOTRON_W4A8_MOE`) exactly: f32 →
+/// f16 upload, zero-pad `k` up to the mxf8f6f4 SF atom width (128) via
+/// `ops::pad_rows_f16` (a no-op clone when `k` is already 128-aligned, true
+/// for both Laguna-S-2.1 K dims, `hidden=3072` and `n_ff_exp=1024`, so this
+/// is a documented no-op on the shipped checkpoint, not dead code for a
+/// hypothetical one), then `ops::w4a8_packw`. Neither this struct nor the
+/// nemotron recipe it mirrors applies any outlier clipping or per-channel
+/// rescale before packing, the packer's own per-32 ue8m0 block scale is the
+/// only quantization-error mitigation on the weight side (mxfp4 is a
+/// coarser format than the engine's own Q4: e2m1 has only 16 representable
+/// magnitudes per block vs Q4's calibrated int4 codebook, so this is
+/// expected to be lossier, the box's greedy-continuation gate decides
+/// acceptance, per this file's module doc).
+struct W4a8ExpertStack {
+    packed: Tensor, // e2m1 nibble-packed, `ops::w4a8_packw`'s `packed` output
+    sf: Tensor,     // per-32-block ue8m0 scale factors, atom-padded SFB layout
+    /// Per-expert SFB stride in bytes (`ops::w4a8_packw`'s `sfb_exp_elems`
+    /// return, fed straight through as `moe_grouped_gemm_w4a8`'s
+    /// `sfb_exp_bytes`, see that fn's doc, the name is elem-count but the
+    /// GEMM wants it as a byte stride, matching the nemotron call site).
+    sfb_exp_bytes: i64,
+    /// Original (unpadded) input width, `hidden` for gate/up (incl. the
+    /// fused `2*n_ff_exp`-row stack), `n_ff_exp` for down. Not read at
+    /// prefill time (only `kpad` drives the padding/quant calls below),
+    /// kept for debugging/asserts, mirroring `Q4Dense`/`Q4ExpertStack`'s own
+    /// `k` fields.
+    #[allow(dead_code)]
+    k: usize,
+    /// `k` rounded up to the mxf8f6f4 SF atom width (128), the width
+    /// actually packed and what every `w4a8_sfa_offsets`/
+    /// `w4a8_actquant_grouped` call at prefill time must pad activations to
+    /// before quantizing against this stack.
+    kpad: usize,
+}
+
+/// BUTTER_LAGUNA_W4A8=1: pack a dequantized `[n_expert, n, k]` f32 expert-major
+/// slab (same `flat` a caller would otherwise only feed to `Q4ExpertStack`'s
+/// builder, see the call sites in `build_moe_w4a8_sequential`, which
+/// dequantize ONCE and hand the same slab to both) into a [`W4a8ExpertStack`],
+/// threading the packed output through `cache`. See that struct's doc for the
+/// exact recipe (f32→f16, pad-to-128, `ops::w4a8_packw`) and why it applies
+/// no extra outlier handling.
+///
+/// UNLIKE every other cached artifact in this file, the pack itself
+/// (`ops::pad_rows_f16` + `ops::w4a8_packw`) runs ON THE DEVICE, not on the
+/// host, it needs `dev`, so it can never be moved into the rayon-parallel
+/// per-layer CPU pass (`build_layer_bytes`); it always runs sequentially, on
+/// whichever thread calls it (see `build_moe_w4a8_sequential`'s doc). On a
+/// cache MISS this does exactly what the pre-cache `pack_w4a8_expert_stack`
+/// did (upload raw f16, pad, pack) plus a download-back-to-host of the
+/// result to write the cache blob. On a HIT, all of that: including the
+/// f16 upload and both GPU kernels, is skipped: the packed bytes are
+/// re-uploaded straight from disk.
+fn pack_w4a8_expert_stack_cached(dev: &dyn Device, cache: &CacheCtx, key: &str, flat: &[f32], n_expert: usize, n: usize, k: usize) -> Result<W4a8ExpertStack> {
+    let kpad = k.div_ceil(128) * 128;
+    let packed_key = format!("{key}.w4a8.packed");
+    let sf_key = format!("{key}.w4a8.sf");
+    let meta_key = format!("{key}.w4a8.meta");
+    let blobs = cache.get_or_build_multi(
+        &[(packed_key.as_str(), DTYPE_RAW), (sf_key.as_str(), DTYPE_RAW), (meta_key.as_str(), DTYPE_RAW)],
+        || -> Result<Vec<Vec<u8>>> {
+            let wt = Tensor::new(dev.upload(&f32s_to_f16_bytes(flat))?, vec![n_expert * n, k], DType::F16);
+            let wt_pad = ops::pad_rows_f16(dev, &wt, n_expert * n, k, kpad)?;
+            let (packed, sf, sfb_exp_bytes) = ops::w4a8_packw(dev, &wt_pad, n_expert, n, kpad)?;
+            dev.synchronize()?;
+            let mut packed_bytes = vec![0u8; packed.elem_count() * 4];
+            dev.download(packed.buffer.as_ref(), &mut packed_bytes)?;
+            let mut sf_bytes = vec![0u8; sf.elem_count() * 4];
+            dev.download(sf.buffer.as_ref(), &mut sf_bytes)?;
+            Ok(vec![packed_bytes, sf_bytes, sfb_exp_bytes.to_le_bytes().to_vec()])
+        },
+    )?;
+    let mut it = blobs.into_iter();
+    let packed_bytes = it.next().unwrap();
+    let sf_bytes = it.next().unwrap();
+    let meta_bytes = it.next().unwrap();
+    let packed = Tensor::new(dev.upload(&packed_bytes)?, vec![packed_bytes.len() / 4], DType::U32);
+    let sf = Tensor::new(dev.upload(&sf_bytes)?, vec![sf_bytes.len() / 4], DType::U32);
+    let sfb_exp_bytes = i64::from_le_bytes(meta_bytes[..8].try_into().map_err(|_| Error::Msg("Laguna cache: truncated w4a8 meta blob".into()))?);
+    Ok(W4a8ExpertStack { packed, sf, sfb_exp_bytes, k, kpad })
+}
+
+/// YaRN RoPE parameters for one layer-type (full-attention or sliding).
+#[derive(Debug, Clone, Copy)]
+pub struct RopeParams {
+    pub n_rot: u32,
+    pub theta: f32,
+    pub freq_scale: f32,
+    pub ext_factor: f32,
+    pub attn_factor: f32,
+    pub beta_fast: f32,
+    pub beta_slow: f32,
+    pub n_orig_ctx: u32,
+}
+
+/// Architecture config for Laguna-S-2.1, read from GGUF `laguna.*` metadata
+/// (arch spec fields fall back to the model's known-fixed values when a key
+/// is absent, e.g. weightless test fixtures or a schema this loader hasn't
+/// seen yet, always cross-check against a real GGUF's `laguna.*` keys when
+/// wiring a new checkpoint).
+#[derive(Debug, Clone)]
+pub struct LagunaConfig {
+    pub hidden: usize,
+    pub n_layers: usize,
+    pub head_dim: usize,
+    pub n_kv_heads: usize,
+    /// Q head count on a full-attention layer (`i % swa_period == 0`).
+    pub n_head_full: usize,
+    /// Q head count on a sliding-window layer.
+    pub n_head_sliding: usize,
+    /// Hybrid attention period: layer `i` is full-attention iff `i % swa_period == 0`.
+    pub swa_period: usize,
+    pub sliding_window: usize,
+    pub eps: f32,
+    pub vocab: usize,
+    /// Leading dense (non-MoE) block count (Laguna-S-2.1: 1, i.e. layer 0 only).
+    pub n_layer_dense_lead: usize,
+    pub n_ff_dense: usize, // layer-0 dense MLP intermediate size
+    pub n_expert: usize,
+    pub n_expert_used: usize, // top_k
+    pub n_ff_exp: usize,
+    pub n_ff_shexp: usize,
+    pub routed_scaling: f32,
+    pub rope_full: RopeParams,
+    pub rope_sliding: RopeParams,
+}
+
+impl LagunaConfig {
+    pub fn is_full_layer(&self, i: usize) -> bool {
+        i % self.swa_period == 0
+    }
+    pub fn is_dense_layer(&self, i: usize) -> bool {
+        i < self.n_layer_dense_lead
+    }
+    pub fn n_head(&self, i: usize) -> usize {
+        if self.is_full_layer(i) { self.n_head_full } else { self.n_head_sliding }
+    }
+    pub fn heads_per_group(&self, i: usize) -> u32 {
+        (self.n_head(i) / self.n_kv_heads) as u32
+    }
+    pub fn attn_scale(&self) -> f32 {
+        1.0 / (self.head_dim as f32).sqrt()
+    }
+    pub fn rope_for(&self, i: usize) -> RopeParams {
+        if self.is_full_layer(i) { self.rope_full } else { self.rope_sliding }
+    }
+
+    /// The literal Laguna-S-2.1 spec values (48 layers), used as fallbacks
+    /// when a GGUF key is missing.
+    fn defaults() -> Self {
+        LagunaConfig {
+            hidden: 3072,
+            n_layers: 48,
+            head_dim: 128,
+            n_kv_heads: 8,
+            n_head_full: 48,
+            n_head_sliding: 72,
+            swa_period: 4,
+            sliding_window: 512,
+            eps: 1e-6,
+            vocab: 100352,
+            n_layer_dense_lead: 1,
+            n_ff_dense: 12288,
+            n_expert: 256,
+            n_expert_used: 10,
+            n_ff_exp: 1024,
+            n_ff_shexp: 1024,
+            routed_scaling: 2.5,
+            rope_full: RopeParams {
+                n_rot: 64,
+                theta: 500000.0,
+                freq_scale: 1.0 / 128.0,
+                ext_factor: 1.0,
+                attn_factor: 1.4852030263919618,
+                beta_fast: 32.0,
+                beta_slow: 1.0,
+                n_orig_ctx: 8192,
+            },
+            rope_sliding: RopeParams {
+                n_rot: 128,
+                theta: 10000.0,
+                freq_scale: 1.0,
+                ext_factor: 0.0,
+                attn_factor: 1.0,
+                beta_fast: 32.0,
+                beta_slow: 1.0,
+                n_orig_ctx: 8192,
+            },
+        }
+    }
+
+    /// Read `laguna.*` GGUF metadata, falling back to [`Self::defaults`] for
+    /// any key this loader doesn't find (see the module doc, verify key
+    /// names against a real GGUF before trusting a mismatch silently).
+    pub fn from_gguf(g: &Gguf) -> Self {
+        let d = Self::defaults();
+        let u = |k: &str, def: usize| g.meta_u32(k).map(|v| v as usize).unwrap_or(def);
+        let f = |k: &str, def: f32| g.meta_f32(k).unwrap_or(def);
+
+        let n_layers = u("laguna.block_count", d.n_layers);
+        let sliding_window = u("laguna.attention.sliding_window", d.sliding_window);
+        let n_rot_full = u("laguna.rope.dimension_count", d.rope_full.n_rot as usize) as u32;
+        let n_rot_sliding = u("laguna.rope.dimension_count_swa", d.rope_sliding.n_rot as usize) as u32;
+
+        LagunaConfig {
+            hidden: u("laguna.embedding_length", d.hidden),
+            n_layers,
+            head_dim: u("laguna.attention.key_length", d.head_dim),
+            n_kv_heads: u("laguna.attention.head_count_kv", d.n_kv_heads),
+            // The per-layer-type Q head split isn't a single scalar GGUF key
+            // in this loader (see the struct doc); keep the spec constants.
+            n_head_full: d.n_head_full,
+            n_head_sliding: d.n_head_sliding,
+            swa_period: d.swa_period,
+            sliding_window,
+            eps: f("laguna.attention.layer_norm_rms_epsilon", d.eps),
+            vocab: u("laguna.vocab_size", d.vocab),
+            n_layer_dense_lead: u("laguna.leading_dense_block_count", d.n_layer_dense_lead),
+            n_ff_dense: u("laguna.feed_forward_length", d.n_ff_dense),
+            n_expert: u("laguna.expert_count", d.n_expert),
+            n_expert_used: u("laguna.expert_used_count", d.n_expert_used),
+            n_ff_exp: u("laguna.expert_feed_forward_length", d.n_ff_exp),
+            n_ff_shexp: u("laguna.expert_shared_feed_forward_length", d.n_ff_shexp),
+            routed_scaling: f("laguna.expert_weights_scale", d.routed_scaling),
+            rope_full: {
+                // The GGUF is authoritative over the HF config: this export is
+                // 256K (factor 32, yarn_attn_factor 1 + 0.1 ln 32 = 1.34657),
+                // NOT the HF card's 1M / factor 128. Read the stored scaling
+                // keys; the spec constants are only a last-resort fallback.
+                let factor = f("laguna.rope.scaling.factor", 1.0 / d.rope_full.freq_scale);
+                let attn_factor = g.meta_f32("laguna.rope.scaling.yarn_attn_factor")
+                    .unwrap_or_else(|| 1.0 + 0.1 * factor.ln());
+                RopeParams {
+                    n_rot: n_rot_full,
+                    theta: f("laguna.rope.freq_base", d.rope_full.theta),
+                    freq_scale: 1.0 / factor,
+                    ext_factor: d.rope_full.ext_factor,
+                    attn_factor,
+                    beta_fast: f("laguna.rope.scaling.yarn_beta_fast", d.rope_full.beta_fast),
+                    beta_slow: f("laguna.rope.scaling.yarn_beta_slow", d.rope_full.beta_slow),
+                    n_orig_ctx: u("laguna.rope.scaling.original_context_length", d.rope_full.n_orig_ctx as usize) as u32,
+                }
+            },
+            rope_sliding: RopeParams {
+                n_rot: n_rot_sliding,
+                theta: f("laguna.rope.freq_base_swa", d.rope_sliding.theta),
+                freq_scale: d.rope_sliding.freq_scale,
+                ext_factor: d.rope_sliding.ext_factor,
+                attn_factor: d.rope_sliding.attn_factor,
+                beta_fast: d.rope_sliding.beta_fast,
+                beta_slow: d.rope_sliding.beta_slow,
+                n_orig_ctx: d.rope_sliding.n_orig_ctx,
+            },
+        }
+    }
+}
+
+/// BUTTER_LAGUNA_MICRO=1 enables three decode-path micro-fusions (QKVG concat
+/// gemv, shared-expert fused swiglu gather, o_proj residual accum). The env
+/// is read once at load time (to decide which weight layout to build below,
+/// [`AttnProj`] / [`SharedExpert`]) and once per [`decode_step`] (to decide
+/// the runtime code path). Both reads see the same value: the env is fixed
+/// for the process lifetime (required for CUDA-graph capture anyway, see
+/// [`GraphBufs`]), so load-time and decode-time never disagree.
+fn laguna_micro_enabled() -> bool {
+    std::env::var("BUTTER_LAGUNA_MICRO").map(|v| v == "1").unwrap_or(false)
+}
+
+/// BUTTER_LAGUNA_MOEFUSE=1 fuses the routed MoE gate and up expert stacks
+/// (`ffn_gate_exps`, `ffn_up_exps`) into ONE per-layer Q4 stack at load time,
+/// so `prefill_layer`'s two `ops::moe_q4_grouped_mma_dev` calls over the gate
+/// stack and the up stack (same sorted activation `a`, same `offsets`)
+/// collapse into a single call with `n_out = 2*n_ff_exp`. See
+/// [`RoutedGateUp::Fused`] for the exact per-expert row layout and the
+/// decode-side tradeoff this implies. Read once at load time (to decide which
+/// weight layout to build, same one-layout-resident rule as
+/// [`laguna_micro_enabled`]); `decode_layer`/`prefill_layer` never re-read the
+/// env, they just match on the [`RoutedGateUp`] variant `load_gguf` already
+/// picked.
+fn laguna_moefuse_enabled() -> bool {
+    std::env::var("BUTTER_LAGUNA_MOEFUSE").map(|v| v == "1").unwrap_or(false)
+}
+
+/// BUTTER_LAGUNA_W4A8=1: route prefill's routed-MoE grouped GEMMs (gate/up,
+/// down, the `moe_gateup_gemm`/`moe_down_gemm` stages that dominate the pp
+/// profile, ~73% combined) through the W4A8 grouped-MoE GEMM (fp8 e4m3
+/// activations x mxfp4 weights, cutlass mxf8f6f4) instead of the W4A16
+/// `ops::moe_q4_grouped_mma_dev` path, mirroring the NEMOTRON_W4A8_MOE
+/// recipe (`wh-butter-modeltests`'s `NEMOTRON_W4A8_MOE` build closure, and its
+/// `w4a8_moe` prefill branch) exactly. Read once at load time (to build the
+/// extra [`W4a8ExpertStack`]-packed weight copies below, ADDITIONAL to the
+/// resident Q4 stacks decode still needs, see [`MoeFfnW4a8`]'s doc for the
+/// resident-memory cost) and once per `prefill_layer` call is NOT needed:
+/// the load-time-built `Option<MoeFfnW4a8>` on `MoeFfn` already encodes
+/// whether this layer has a W4A8 path, so `prefill_layer` just matches on
+/// it (same convention as `shared_marlin`/`qkv_marlin`). Decode is entirely
+/// unaffected, it never reads this env or the `w4a8` field.
+fn laguna_w4a8_enabled() -> bool {
+    std::env::var("BUTTER_LAGUNA_W4A8").map(|v| v == "1").unwrap_or(false)
+}
+
+/// QKV(+gate) projection weights. `Fused` concatenates all four projections'
+/// rows into one `Q4Dense` at load time (built ONLY under
+/// `BUTTER_LAGUNA_MICRO=1`, so a layer never carries both layouts resident at
+/// once, that would double the ~2.4GB of attention-projection weights).
+/// Row order in `wqkvg` is `[wq rows; wk rows; wv rows; wgate rows]`, all
+/// sharing `k = hidden`; concatenation happens on the dequantized f32 slabs
+/// BEFORE `quantize_q4`, so each row's Q4 block boundaries are unaffected
+/// (`quantize_q4` blocks are per-row, block size 32 along `k`).
+enum AttnProj {
+    Split { wq: Q4Dense, wk: Q4Dense, wv: Q4Dense, wgate: Q4Dense },
+    Fused { wqkvg: Q4Dense, n_q_rows: usize, n_kv_rows: usize, n_g_rows: usize },
+}
+
+/// Per-layer attention weights (Q4, except the two `[head_dim]` per-head
+/// norms which stay f32).
+struct AttnWeights {
+    /// This layer's Q head count, derived from the loaded Q projection's
+    /// actual row count (`n_q_rows / head_dim`) rather than `LagunaConfig`'s
+    /// hardcoded 48/72 split, so a real GGUF's per-layer head count always
+    /// wins over the spec constants if the two ever disagree.
+    n_head: usize,
+    attn_norm: Tensor, // [hidden] f32
+    proj: AttnProj,
+    wo: Q4Dense,    // [hidden, n_head*head_dim]
+    q_norm: Tensor, // [head_dim] f32
+    k_norm: Tensor, // [head_dim] f32
+    /// BUTTER_LAGUNA_MARLIN=1 prefill-only extras (see [`QkvMarlin`]/
+    /// [`MarlinDense`]'s docs); `None` when the env is unset OR (defensively)
+    /// when a dimension isn't 64-row-aligned. `g_proj` is a plain standalone
+    /// `Q4Dense` (NOT Marlin, gate's 48/72 rows are never 64-aligned) built
+    /// only under the same env, so the marlin prefill branch has gate access
+    /// independent of whether `proj` above resolved to `Split` or `Fused`.
+    qkv_marlin: Option<QkvMarlin>,
+    g_proj: Option<Q4Dense>,
+    wo_marlin: Option<MarlinDense>,
+}
+
+/// Layer-0's dense SwiGLU MLP.
+struct DenseFfn {
+    ffn_norm: Tensor, // [hidden] f32
+    gate: Q4Dense,    // [n_ff_dense, hidden]
+    up: Q4Dense,      // [n_ff_dense, hidden]
+    down: Q4Dense,    // [hidden, n_ff_dense]
+    /// BUTTER_LAGUNA_MARLIN=1 prefill-only extras, see [`GateUpMarlin`]/
+    /// [`MarlinDense`]'s docs. `None` when the env is unset.
+    gateup_marlin: Option<GateUpMarlin>,
+    down_marlin: Option<MarlinDense>,
+}
+
+/// The always-on shared expert's gate+up projections. `Fused` repacks both
+/// as single-expert (`n_expert=1`) [`Q4ExpertStack`]s at load time (built
+/// ONLY under `BUTTER_LAGUNA_MICRO=1`, same one-layout-resident rule as
+/// [`AttnProj::Fused`]) plus a persistent one-element `idx=[0]` device
+/// buffer, so decode can drive both through the same
+/// `ops::moe_gather_q4_swiglu` the routed experts use (`top_k=1`) instead of
+/// two plain gemvs + a host-orchestrated swiglu. `down` stays a plain
+/// `Q4Dense` either way, only gate+up fuse into the gather.
+struct SharedExpertFused {
+    gate: Q4ExpertStack, // n_expert=1, [n_ff_shexp, hidden]
+    up: Q4ExpertStack,   // n_expert=1, [n_ff_shexp, hidden]
+    down: Q4Dense,       // [hidden, n_ff_shexp]
+    /// Persistent device buffer holding the constant `u32` `0`, the single
+    /// "expert index" `moe_gather_q4_swiglu` needs for a 1-expert stack.
+    zero_idx: Tensor,
+}
+enum SharedExpert {
+    Split { gate: Q4Dense, up: Q4Dense, down: Q4Dense },
+    Fused(SharedExpertFused),
+}
+
+/// Routed-expert gate+up weight layout. `Split` is the plain per-projection
+/// layout ([`Q4ExpertStack`] each). `Fused` is built ONLY under
+/// `BUTTER_LAGUNA_MOEFUSE=1` and REPLACES `Split` (see
+/// [`laguna_moefuse_enabled`]'s doc, same one-layout-resident rule as
+/// [`AttnProj`]/[`SharedExpert`]): expert `e`'s slab is `[2*n_ff_exp, hidden]`
+/// with gate rows at local offset `[0, n_ff_exp)` and up rows at local offset
+/// `[n_ff_exp, 2*n_ff_exp)`, i.e. per-expert interleaved (gate_0, up_0,
+/// gate_1, up_1, ...), NOT all gate experts followed by all up experts. This
+/// is the SAME single stack that `prefill_layer` and `decode_layer` both
+/// read, so fusing gate+up does not add a second resident copy of the expert
+/// weights.
+///
+/// Decode tradeoff: the unfused layout lets decode drive gate+up through one
+/// `ops::moe_gather_q4_swiglu` call under `BUTTER_LAGUNA_FUSE=1` (fused inline
+/// SwiGLU, reads the activation once). The fused stack's per-expert row
+/// layout does not fit that kernel's `row = e*inter + local` indexing for two
+/// SEPARATE logical projections sharing one physical stack, so decode falls
+/// back to one `ops::moe_gather_q4` (inter=2*n_ff_exp) plus a host-orchestrated
+/// column split and `ops::swiglu` instead (see `decode_layer`'s MoE branch).
+/// That is roughly the pre-BUTTER_LAGUNA_FUSE decode path (about 1 tok/s slower
+/// than the fused gather), traded for prefill's halved grouped-GEMM call
+/// count. Default OFF.
+enum RoutedGateUp {
+    Split { gate: Q4ExpertStack, up: Q4ExpertStack },
+    Fused(Q4ExpertStack), // per-expert [2*n_ff_exp, hidden]
+}
+
+/// BUTTER_LAGUNA_W4A8=1 mirror of [`RoutedGateUp`], packed as
+/// [`W4a8ExpertStack`]s instead of [`Q4ExpertStack`]s. Which variant is
+/// built follows the SAME `BUTTER_LAGUNA_MOEFUSE` choice `RoutedGateUp` made
+/// for this layer (`load_gguf` builds both from the identical dequantized
+/// f32 slab, so `Fused` here always pairs with `RoutedGateUp::Fused` and
+/// vice versa), `prefill_layer` never needs to check the two enums agree,
+/// they're constructed together.
+enum RoutedGateUpW4a8 {
+    Split { gate: W4a8ExpertStack, up: W4a8ExpertStack },
+    Fused(W4a8ExpertStack), // per-expert [2*n_ff_exp, hidden]
+}
+
+/// BUTTER_LAGUNA_W4A8=1 prefill-only routed-expert extras: mxfp4-packed
+/// [`RoutedGateUpW4a8`] + down [`W4a8ExpertStack`], built ADDITIONALLY to
+/// (never replacing) `MoeFfn`'s existing `gateup`/`down` [`Q4ExpertStack`]s
+///, decode always drives the Q4 stacks, this is prefill-only, mirroring
+/// every other `BUTTER_LAGUNA_*` weight-layout env in this file. Unlike those,
+/// this ADDS roughly 30GB of resident memory across all MoE layers (the
+/// routed expert stacks are the bulk of the model, Q4 stays resident for
+/// decode, so this genuinely doubles the routed-expert footprint rather
+/// than trading one layout for another); acceptable for a prefill-throughput
+/// win on hardware with the headroom, but unlike `BUTTER_LAGUNA_MARLIN`
+/// (which is roughly memory-neutral, permuting the SAME quantized weight)
+/// this is NOT a "just flip it on" env on a memory-constrained box.
+struct MoeFfnW4a8 {
+    gateup: RoutedGateUpW4a8,
+    down: W4a8ExpertStack, // per-expert [hidden, n_ff_exp]
+}
+
+/// A routed MoE FFN block (layers `n_layer_dense_lead..`).
+struct MoeFfn {
+    ffn_norm: Tensor,  // [hidden] f32
+    router: Tensor,    // [n_expert, hidden] f32: kept dense for routing precision
+    bias: Tensor,      // [n_expert] f32: e_score_correction_bias (selection only)
+    gateup: RoutedGateUp,
+    down: Q4ExpertStack, // per-expert [hidden, n_ff_exp]
+    shared_expert: SharedExpert,
+    /// BUTTER_LAGUNA_MARLIN=1 prefill-only shared-expert extras, see
+    /// [`SharedMarlin`]'s doc. `None` when the env is unset. The routed
+    /// per-expert `gate`/`up`/`down` stacks above stay on the existing
+    /// grouped-MMA prefill path (`ops::moe_q4_grouped_mma_dev`), that's
+    /// already a tensor-core grouped GEMM, not the GEMV-class kernel this
+    /// Marlin work targets.
+    shared_marlin: Option<SharedMarlin>,
+    /// BUTTER_LAGUNA_W4A8=1 prefill-only routed-expert extras, see
+    /// [`MoeFfnW4a8`]'s doc. `None` when the env is unset; when `Some`,
+    /// `prefill_layer` routes the routed gate/up/down grouped GEMMs through
+    /// `ops::moe_grouped_gemm_w4a8` instead of `ops::moe_q4_grouped_mma_dev`
+    /// (the shared expert and everything else in the layer is untouched).
+    w4a8: Option<MoeFfnW4a8>,
+}
+
+enum LayerFfn {
+    Dense(DenseFfn),
+    Moe(MoeFfn),
+}
+
+struct LagunaLayer {
+    attn: AttnWeights,
+    ffn: LayerFfn,
+}
+
+/// LM head resident as Q8 (the f32 head was a 1.2GB read per token, ~10% of
+/// the whole decode bandwidth budget; Q8 quarters it at negligible logit
+/// error). Same layout `wh_butter_ops::gemv_q8` consumes: 4 codes/u32, f32 scale
+/// per 32-block.
+struct Q8Head {
+    qs: Tensor,
+    scales: Tensor,
+    m: usize,
+    k: usize,
+}
+
+/// Full Laguna-S-2.1 weights, resident on `dev`.
+pub struct LagunaModel {
+    pub cfg: LagunaConfig,
+    tok_embd: Tensor, // [vocab, hidden] f32
+    layers: Vec<LagunaLayer>,
+    out_norm: Tensor, // [hidden] f32
+    lm_head: Q8Head,  // [vocab, hidden] q8
+    /// Cached device-resident `1.0f32` scalar, reused as the shared expert's
+    /// `gemv_q4_accum` weight (unconditional, unweighted accumulate).
+    one: Tensor,
+}
+
+/// Persistent decode-time KV cache: one entry per layer, growing (`cap =
+/// max_seq`) for full-attention layers, a fixed 512-slot ring for sliding
+/// layers. RoPE is applied (with the token's absolute position) before the
+/// K row is cached, so ring order doesn't matter for attention: softmax
+/// over the cached K/V set is order-invariant once every slot is filled.
+struct KvSlot {
+    k: Tensor, // [n_kv_heads, cap, head_dim]
+    v: Tensor,
+    cap: usize,
+}
+pub struct LagunaKvCache {
+    slots: Vec<KvSlot>,
+}
+impl LagunaKvCache {
+    pub fn new(dev: &dyn Device, cfg: &LagunaConfig, max_seq: usize) -> Result<Self> {
+        let mut slots = Vec::with_capacity(cfg.n_layers);
+        for i in 0..cfg.n_layers {
+            let cap = if cfg.is_full_layer(i) { max_seq } else { cfg.sliding_window };
+            let n = cfg.n_kv_heads * cap * cfg.head_dim;
+            let k = Tensor::new(dev.alloc_zeroed(n * 4)?, vec![cfg.n_kv_heads, cap, cfg.head_dim], DType::F32);
+            let v = Tensor::new(dev.alloc_zeroed(n * 4)?, vec![cfg.n_kv_heads, cap, cfg.head_dim], DType::F32);
+            slots.push(KvSlot { k, v, cap });
+        }
+        Ok(LagunaKvCache { slots })
+    }
+}
+
+/// Per-sliding-layer LINEAR K/V scratch used ONLY during [`prefill`]. The
+/// existing sliding-layer decode cache (a `KvSlot` inside [`LagunaKvCache`])
+/// is a fixed `sliding_window`-slot RING, safe for single-token decode
+/// (write one slot, evict the oldest), but NOT safe to batch-write during
+/// prefill: a `kv_append_many` writing a whole T-token chunk into the ring
+/// would evict entries that EARLIER queries in that SAME chunk still need to
+/// attend to (their window hasn't slid that far yet). Prefill instead writes
+/// every sliding-layer token into a LINEAR buffer sized to the whole prompt
+/// (`[n_kv_heads, prompt_cap, head_dim]`, absolute positions, no
+/// wraparound), attends with `seg_lo[i] = max(0, pos_i - (sliding_window -
+/// 1))` via `ops::sdpa_multi_tc_varlen`, and once the whole prompt has been
+/// processed, [`prefill`] compacts the last `sliding_window` positions into
+/// the decode ring (`ops::kv_compact_ring`) so `decode_step` /
+/// `LagunaDecodeCtx` can continue from `pos = tokens.len()` exactly as if
+/// every prompt token had gone through the per-token decode path.
+///
+/// Full-attention layers need no scratch, they reuse `LagunaKvCache`'s
+/// already-growing per-layer buffer directly (a batched write there is safe:
+/// nothing is ever evicted).
+///
+/// Memory: `n_kv_heads * prompt_cap * head_dim * 4 bytes * 2 (K+V)` per
+/// sliding layer. At `prompt_cap = 32768` (this v1's documented supported
+/// prompt-length ceiling) that is `8 * 32768 * 128 * 4 * 2 ≈ 256MiB` per
+/// sliding layer; Laguna-S-2.1 has 36 sliding layers (48 total, minus 12
+/// full-attention layers every 4th), so a full-size scratch is `≈9.7GB`.
+/// Allocate ONE scratch sized to the largest prompt the caller expects to
+/// prefill, not per call.
+pub struct LagunaPrefillScratch {
+    /// Indexed by layer; `None` for full-attention layers (`i % swa_period == 0`).
+    slots: Vec<Option<KvSlot>>,
+    prompt_cap: usize,
+}
+
+impl LagunaPrefillScratch {
+    pub fn new(dev: &dyn Device, cfg: &LagunaConfig, prompt_cap: usize) -> Result<Self> {
+        let mut slots = Vec::with_capacity(cfg.n_layers);
+        for i in 0..cfg.n_layers {
+            if cfg.is_full_layer(i) {
+                slots.push(None);
+                continue;
+            }
+            let n = cfg.n_kv_heads * prompt_cap * cfg.head_dim;
+            let k = Tensor::new(dev.alloc_zeroed(n * 4)?, vec![cfg.n_kv_heads, prompt_cap, cfg.head_dim], DType::F32);
+            let v = Tensor::new(dev.alloc_zeroed(n * 4)?, vec![cfg.n_kv_heads, prompt_cap, cfg.head_dim], DType::F32);
+            slots.push(Some(KvSlot { k, v, cap: prompt_cap }));
+        }
+        Ok(LagunaPrefillScratch { slots, prompt_cap })
+    }
+}
+
+fn dims_out_in(t: &wh_butter_loader::gguf::GgufTensor, name: &str) -> Result<(usize, usize)> {
+    // GGUF dims are fastest-first: a 2-D `[out, in]` weight is stored as
+    // `[in, out]` (the reference C++ implementation's `create_tensor({k, m}, ...)` convention), which
+    // is exactly the `(m, k)` `gemv_q4` wants with zero transposition.
+    if t.dims.len() != 2 {
+        return Err(Error::Msg(format!("{name}: expected 2-D tensor, got dims {:?}", t.dims)));
+    }
+    Ok((t.dims[1] as usize, t.dims[0] as usize))
+}
+
+/// Read `name`'s `(m, k)` from its GGUF header dims alone, cheap (metadata
+/// only, no mmap data touched), so this is always safe to call eagerly, even
+/// when the tensor's actual bytes end up not being dequantized at all (a
+/// full cache hit for that tensor's requantized artifact).
+fn tensor_dims(g: &Gguf, name: &str) -> Result<(usize, usize)> {
+    let t = g.tensor(name).ok_or_else(|| Error::Msg(format!("gguf: tensor '{name}' missing")))?;
+    dims_out_in(t, name)
+}
+
+// ── Bytes-mirror types ──────────────────────────────────────────────────
+//
+// Host-side (no `Tensor`, no `dev`) mirrors of the device-resident weight
+// structs above, holding UPLOAD-READY byte blobs instead of `Tensor`s. This
+// is the payload of `build_layer_bytes` (the rayon-parallel, cache-checked,
+// pure-CPU half of loading one layer), `upload_layer` below turns one of
+// these into the real `LagunaLayer` by doing nothing but `dev.upload` +
+// `Tensor::new` calls, sequentially.
+
+struct Q4DenseBytes { qs: Vec<u8>, scales: Vec<u8>, m: usize, k: usize }
+struct MarlinDenseBytes { qs: Vec<u8>, scales: Vec<u8>, n_out: usize, k_in: usize }
+struct Q4ExpertStackBytes { qs: Vec<u8>, scales: Vec<u8>, m: usize, k: usize }
+
+enum AttnProjBytes {
+    Fused { wqkvg: Q4DenseBytes, n_q_rows: usize, n_kv_rows: usize, n_g_rows: usize },
+    Split { wq: Q4DenseBytes, wk: Q4DenseBytes, wv: Q4DenseBytes, wgate: Q4DenseBytes },
+}
+struct QkvMarlinBytes { mat: MarlinDenseBytes, n_q_rows: usize, n_kv_rows: usize }
+struct GateUpMarlinBytes { mat: MarlinDenseBytes, n_ff: usize }
+struct SharedMarlinBytes { gateup: GateUpMarlinBytes, down: MarlinDenseBytes }
+
+struct AttnWeightsBytes {
+    n_head: usize,
+    attn_norm: Vec<u8>,
+    proj: AttnProjBytes,
+    wo: Q4DenseBytes,
+    q_norm: Vec<u8>,
+    k_norm: Vec<u8>,
+    qkv_marlin: Option<QkvMarlinBytes>,
+    g_proj: Option<Q4DenseBytes>,
+    wo_marlin: Option<MarlinDenseBytes>,
+}
+
+struct DenseFfnBytes {
+    ffn_norm: Vec<u8>,
+    gate: Q4DenseBytes,
+    up: Q4DenseBytes,
+    down: Q4DenseBytes,
+    gateup_marlin: Option<GateUpMarlinBytes>,
+    down_marlin: Option<MarlinDenseBytes>,
+}
+
+struct SharedExpertFusedBytes { gate: Q4ExpertStackBytes, up: Q4ExpertStackBytes, down: Q4DenseBytes }
+enum SharedExpertBytes {
+    Split { gate: Q4DenseBytes, up: Q4DenseBytes, down: Q4DenseBytes },
+    Fused(SharedExpertFusedBytes),
+}
+enum RoutedGateUpBytes {
+    Split { gate: Q4ExpertStackBytes, up: Q4ExpertStackBytes },
+    Fused(Q4ExpertStackBytes),
+}
+
+struct MoeFfnBytes {
+    ffn_norm: Vec<u8>,
+    router: Vec<u8>,
+    bias: Vec<u8>,
+    /// `None` under BUTTER_LAGUNA_W4A8=1: the routed gate/up/down stacks are
+    /// built afterward, sequentially, in `upload_layer` /
+    /// `build_moe_w4a8_sequential` instead of here, W4A8 packing
+    /// (`ops::pad_rows_f16` / `ops::w4a8_packw`) is a GPU op and can't run on
+    /// a rayon worker thread, and it needs the SAME dequantized slab the Q4
+    /// stack does, so keeping the two together (both sequential) avoids
+    /// dequantizing the routed tensor twice.
+    gateup: Option<RoutedGateUpBytes>,
+    down: Option<Q4ExpertStackBytes>,
+    shared_expert: SharedExpertBytes,
+    shared_marlin: Option<SharedMarlinBytes>,
+}
+
+enum LayerFfnBytes { Dense(DenseFfnBytes), Moe(MoeFfnBytes) }
+struct LayerBytes { attn: AttnWeightsBytes, ffn: LayerFfnBytes }
+
+// ── Cache-aware builders (host bytes in, cache-checked, no `dev`) ──────────
+
+/// The core cached-requant primitive: `build_flat` (a DEFERRED dequant, it
+/// only runs on a miss, so a cache hit never touches the GGUF mmap) produces
+/// the `[m, k]` f32 slab, which gets `quantize_q4`'d and cached as its
+/// `qs`/`scales` byte pair under `key`.
+fn q4dense_bytes_from_flat_cached(cache: &CacheCtx, key: &str, m: usize, k: usize, build_flat: impl FnOnce() -> Result<Vec<f32>>) -> Result<Q4DenseBytes> {
+    let qs_key = format!("{key}.q4.qs");
+    let sc_key = format!("{key}.q4.scales");
+    let blobs = cache.get_or_build_multi(&[(qs_key.as_str(), DTYPE_U32), (sc_key.as_str(), DTYPE_F16)], || {
+        let flat = build_flat()?;
+        let (qs, scales) = ops::quantize_q4(&flat, m, k);
+        Ok(vec![u32s_to_bytes(&qs), f32s_to_f16_bytes(&scales)])
+    })?;
+    let mut it = blobs.into_iter();
+    Ok(Q4DenseBytes { qs: it.next().unwrap(), scales: it.next().unwrap(), m, k })
+}
+
+/// Marlin-only counterpart of [`q4dense_bytes_from_flat_cached`]: quantizes
+/// AND Marlin-permutes inside the (deferred, miss-only) build closure.
+fn marlindense_bytes_from_flat_cached(cache: &CacheCtx, key: &str, n_out: usize, k_in: usize, build_flat: impl FnOnce() -> Result<Vec<f32>>) -> Result<MarlinDenseBytes> {
+    let qs_key = format!("{key}.marlin.qs");
+    let sc_key = format!("{key}.marlin.scales");
+    let blobs = cache.get_or_build_multi(&[(qs_key.as_str(), DTYPE_U32), (sc_key.as_str(), DTYPE_F16)], || {
+        let flat = build_flat()?;
+        let (qs_std, scales) = ops::quantize_q4(&flat, n_out, k_in);
+        let qs_mar = ops::permute_q4_to_marlin(&qs_std, 1, n_out, k_in);
+        Ok(vec![u32s_to_bytes(&qs_mar), f32s_to_f16_bytes(&scales)])
+    })?;
+    let mut it = blobs.into_iter();
+    Ok(MarlinDenseBytes { qs: it.next().unwrap(), scales: it.next().unwrap(), n_out, k_in })
+}
+
+/// Cached counterpart of the old `load_dense_and_marlin`: one deferred
+/// `quantize_q4` pass feeds BOTH the plain `Q4DenseBytes` (`.q4.*` keys,
+/// always built) and, when `want_marlin && m % 64 == 0`, an additional
+/// `MarlinDenseBytes` (`.marlin.*` keys) permuted from the SAME `qs_std`. A
+/// full cache hit (all keys present) skips `build_flat`, and therefore the
+/// GGUF dequant, entirely.
+fn q4dense_and_marlin_bytes_cached(cache: &CacheCtx, key: &str, m: usize, k: usize, want_marlin: bool, build_flat: impl FnOnce() -> Result<Vec<f32>>) -> Result<(Q4DenseBytes, Option<MarlinDenseBytes>)> {
+    let do_marlin = want_marlin && m % 64 == 0;
+    let qs_key = format!("{key}.q4.qs");
+    let sc_key = format!("{key}.q4.scales");
+    let mqs_key = format!("{key}.marlin.qs");
+    let msc_key = format!("{key}.marlin.scales");
+    let mut keys: Vec<(&str, u8)> = vec![(qs_key.as_str(), DTYPE_U32), (sc_key.as_str(), DTYPE_F16)];
+    if do_marlin {
+        keys.push((mqs_key.as_str(), DTYPE_U32));
+        keys.push((msc_key.as_str(), DTYPE_F16));
+    }
+    let blobs = cache.get_or_build_multi(&keys, || {
+        let flat = build_flat()?;
+        let (qs_std, scales) = ops::quantize_q4(&flat, m, k);
+        let mut out = vec![u32s_to_bytes(&qs_std), f32s_to_f16_bytes(&scales)];
+        if do_marlin {
+            let qs_mar = ops::permute_q4_to_marlin(&qs_std, 1, m, k);
+            out.push(u32s_to_bytes(&qs_mar));
+            out.push(f32s_to_f16_bytes(&scales));
+        }
+        Ok(out)
+    })?;
+    let mut it = blobs.into_iter();
+    let dense = Q4DenseBytes { qs: it.next().unwrap(), scales: it.next().unwrap(), m, k };
+    let marlin = if do_marlin {
+        Some(MarlinDenseBytes { qs: it.next().unwrap(), scales: it.next().unwrap(), n_out: m, k_in: k })
+    } else {
+        None
+    };
+    Ok((dense, marlin))
+}
+
+/// Cached dense f32 vector (norms, router, bias, embeddings, ...), the
+/// upload bytes are literally `f32s_to_bytes(flat)`, so caching mostly buys
+/// skipping `build_flat`'s dequant on a hit (which matters a lot for the
+/// two big ones, `token_embd.weight` / `output.weight`, and is free for the
+/// tiny per-layer norms).
+fn f32vec_bytes_cached(cache: &CacheCtx, key: &str, build_flat: impl FnOnce() -> Result<Vec<f32>>) -> Result<Vec<u8>> {
+    cache.get_or_build(&format!("{key}.f32vec"), DTYPE_F32, || Ok(f32s_to_bytes(&build_flat()?)))
+}
+
+/// Cached [`Q4ExpertStack`] bytes: `build_flat` (deferred) produces the
+/// `[n_expert, m, k]` expert-major f32 slab; the per-expert `quantize_q4`
+/// loop (the heavy part) only runs on a miss.
+fn q4_expert_stack_bytes_cached(cache: &CacheCtx, key: &str, n_expert: usize, m: usize, k: usize, build_flat: impl FnOnce() -> Result<Vec<f32>>) -> Result<Q4ExpertStackBytes> {
+    let qs_key = format!("{key}.q4x.qs");
+    let sc_key = format!("{key}.q4x.scales");
+    let blobs = cache.get_or_build_multi(&[(qs_key.as_str(), DTYPE_U32), (sc_key.as_str(), DTYPE_F16)], || {
+        let flat = build_flat()?;
+        let per_expert = m * k;
+        if flat.len() != per_expert * n_expert {
+            return Err(Error::Msg(format!(
+                "Q4ExpertStack '{key}': flat len {} != n_expert*m*k ({} * {} * {})",
+                flat.len(), n_expert, m, k
+            )));
+        }
+        let qw = per_expert / 32 * 4; // quantize_q4: m*k/32 blocks, 4 u32/block
+        let sw = per_expert / 32; // 1 f32 scale/block
+        let mut qs_all = Vec::with_capacity(qw * n_expert);
+        let mut sc_all = Vec::with_capacity(sw * n_expert);
+        for e in 0..n_expert {
+            let slab = &flat[e * per_expert..(e + 1) * per_expert];
+            let (qs, sc) = ops::quantize_q4(slab, m, k);
+            qs_all.extend_from_slice(&qs);
+            sc_all.extend_from_slice(&sc);
+        }
+        Ok(vec![u32s_to_bytes(&qs_all), f32s_to_f16_bytes(&sc_all)])
+    })?;
+    let mut it = blobs.into_iter();
+    Ok(Q4ExpertStackBytes { qs: it.next().unwrap(), scales: it.next().unwrap(), m, k })
+}
+
+/// Cached [`Q8Head`] bytes via [`Gguf::q8_repack`] (deferred, only runs on a
+/// miss). `(m, k)` come from the tensor's own GGUF dims (matches
+/// `q8_repack`'s internal derivation exactly, both read `dims[1]`/`dims[0]`
+/// the same way `dims_out_in` does), so they're known without calling
+/// `q8_repack` and therefore available even on a full cache hit.
+fn q8head_bytes_cached(cache: &CacheCtx, g: &Gguf, name: &str, key: &str) -> Result<(Vec<u8>, Vec<u8>, usize, usize)> {
+    let (m, k) = tensor_dims(g, name)?;
+    let qs_key = format!("{key}.q8.qs");
+    let sc_key = format!("{key}.q8.scales");
+    let blobs = cache.get_or_build_multi(&[(qs_key.as_str(), DTYPE_U32), (sc_key.as_str(), DTYPE_F32)], || {
+        let (qs, sc, _m, _k) = g.q8_repack(name)?;
+        Ok(vec![u32s_to_bytes(&qs), f32s_to_bytes(&sc)])
+    })?;
+    let mut it = blobs.into_iter();
+    Ok((it.next().unwrap(), it.next().unwrap(), m, k))
+}
+
+// ── Upload-phase helpers (sequential, `dev`-only, no CPU-heavy work) ───────
+
+fn upload_q4dense_bytes(dev: &dyn Device, b: Q4DenseBytes) -> Result<Q4Dense> {
+    Ok(Q4Dense {
+        qs: Tensor::new(dev.upload(&b.qs)?, vec![b.qs.len() / 4], DType::U32),
+        scales: Tensor::new(dev.upload(&b.scales)?, vec![b.scales.len() / 2], DType::F16),
+        m: b.m,
+        k: b.k,
+    })
+}
+fn upload_marlindense_bytes(dev: &dyn Device, b: MarlinDenseBytes) -> Result<MarlinDense> {
+    Ok(MarlinDense {
+        qs: Tensor::new(dev.upload(&b.qs)?, vec![b.qs.len() / 4], DType::U32),
+        scales: Tensor::new(dev.upload(&b.scales)?, vec![b.scales.len() / 2], DType::F16),
+        n_out: b.n_out,
+        k_in: b.k_in,
+    })
+}
+fn upload_expert_stack_bytes(dev: &dyn Device, b: Q4ExpertStackBytes) -> Result<Q4ExpertStack> {
+    let per_expert = b.m * b.k;
+    Ok(Q4ExpertStack {
+        qs: Tensor::new(dev.upload(&b.qs)?, vec![b.qs.len() / 4], DType::U32),
+        scales: Tensor::new(dev.upload(&b.scales)?, vec![b.scales.len() / 2], DType::F16),
+        m: b.m,
+        k: b.k,
+        qs_words_per_expert: per_expert / 32 * 4,
+        scale_elems_per_expert: per_expert / 32,
+    })
+}
+fn upload_f32_bytes(dev: &dyn Device, bytes: Vec<u8>) -> Result<Tensor> {
+    let n = bytes.len() / 4;
+    Ok(Tensor::new(dev.upload(&bytes)?, vec![n], DType::F32))
+}
+
+/// Pure-CPU half of loading one layer: dequantize (deferred inside every
+/// cache closure, a cache HIT never touches the GGUF mmap) + requantize /
+/// permute / interleave every weight this layer needs, each artifact
+/// individually cache-checked. Touches neither `dev` nor any device buffer,
+/// so this is exactly the per-layer unit `load_gguf` runs across a rayon
+/// thread pool on a cache miss (`into_par_iter` over layer indices, see
+/// that fn's doc). Under BUTTER_LAGUNA_W4A8=1 the routed gate/up/down stacks
+/// are left `None` here and built afterward, sequentially, by
+/// `build_moe_w4a8_sequential` (see [`MoeFfnBytes::gateup`]'s doc for why).
+#[allow(clippy::too_many_arguments)]
+fn build_layer_bytes(g: &Gguf, cache: &CacheCtx, cfg: &LagunaConfig, i: usize, micro: bool, marlin: bool, moefuse: bool, w4a8: bool) -> Result<LayerBytes> {
+    let p = format!("blk.{i}");
+    let q_name = format!("{p}.attn_q.weight");
+    let k_name = format!("{p}.attn_k.weight");
+    let v_name = format!("{p}.attn_v.weight");
+    let gate_name = format!("{p}.attn_gate.weight");
+
+    // n_head is derived from wq's own GGUF dims (48 full-attention / 72
+    // sliding for Laguna-S-2.1), not the LagunaConfig formula: see the
+    // AttnWeights doc comment. Dims-only reads (no dequant) so this is cheap
+    // regardless of cache state.
+    let (n_q_rows, hidden_k) = tensor_dims(g, &q_name)?;
+    let (n_kv_rows, _) = tensor_dims(g, &k_name)?;
+    let (n_kv_rows_v, _) = tensor_dims(g, &v_name)?;
+    let (n_g_rows, _) = tensor_dims(g, &gate_name)?;
+    debug_assert_eq!(n_kv_rows, n_kv_rows_v, "wk/wv row count mismatch");
+    let n_head = n_q_rows / cfg.head_dim;
+
+    // BUTTER_LAGUNA_MICRO=1: concatenate the four dequantized f32 slabs BEFORE
+    // quantize_q4 (row order q,k,v,gate; all share k=hidden), so decode can
+    // do one gemv_q4 instead of four. Only ONE of the two layouts is ever
+    // built per layer, building both would double the resident
+    // attention-projection weights (~2.4GB total).
+    let proj = if micro {
+        let n_rows = n_q_rows + n_kv_rows + n_kv_rows_v + n_g_rows;
+        let wqkvg = q4dense_bytes_from_flat_cached(cache, &format!("{p}.attn_qkvg_fused"), n_rows, hidden_k, || {
+            let mut concat = Vec::with_capacity(n_rows * hidden_k);
+            concat.extend_from_slice(&g.dequant_f32(&q_name)?);
+            concat.extend_from_slice(&g.dequant_f32(&k_name)?);
+            concat.extend_from_slice(&g.dequant_f32(&v_name)?);
+            concat.extend_from_slice(&g.dequant_f32(&gate_name)?);
+            Ok(concat)
+        })?;
+        AttnProjBytes::Fused { wqkvg, n_q_rows, n_kv_rows, n_g_rows }
+    } else {
+        AttnProjBytes::Split {
+            wq: q4dense_bytes_from_flat_cached(cache, &q_name, n_q_rows, hidden_k, || g.dequant_f32(&q_name))?,
+            wk: q4dense_bytes_from_flat_cached(cache, &k_name, n_kv_rows, hidden_k, || g.dequant_f32(&k_name))?,
+            wv: q4dense_bytes_from_flat_cached(cache, &v_name, n_kv_rows_v, hidden_k, || g.dequant_f32(&v_name))?,
+            wgate: q4dense_bytes_from_flat_cached(cache, &gate_name, n_g_rows, hidden_k, || g.dequant_f32(&gate_name))?,
+        }
+    };
+
+    // BUTTER_LAGUNA_MARLIN=1: pack [wq;wk;wv] (gate excluded, see QkvMarlin's
+    // doc) into a Marlin-tile-major MarlinDense for the tensor-core
+    // batched-prefill GEMM, built ALONGSIDE (never instead of) the AttnProj
+    // decode copy above. `g_proj` is a standalone plain Q4Dense so the
+    // marlin prefill branch has gate access independent of whether `proj`
+    // above resolved to Split or Fused.
+    let qkv_marlin = if marlin {
+        let n_rows_qkv = n_q_rows + n_kv_rows + n_kv_rows_v;
+        if n_rows_qkv % 64 == 0 {
+            let mat = marlindense_bytes_from_flat_cached(cache, &format!("{p}.attn_qkv"), n_rows_qkv, hidden_k, || {
+                let mut concat = Vec::with_capacity(n_rows_qkv * hidden_k);
+                concat.extend_from_slice(&g.dequant_f32(&q_name)?);
+                concat.extend_from_slice(&g.dequant_f32(&k_name)?);
+                concat.extend_from_slice(&g.dequant_f32(&v_name)?);
+                Ok(concat)
+            })?;
+            Some(QkvMarlinBytes { mat, n_q_rows, n_kv_rows })
+        } else {
+            None
+        }
+    } else {
+        None
+    };
+    let g_proj = if marlin {
+        Some(q4dense_bytes_from_flat_cached(cache, &format!("{gate_name}.standalone"), n_g_rows, hidden_k, || g.dequant_f32(&gate_name))?)
+    } else {
+        None
+    };
+
+    let wo_name = format!("{p}.attn_output.weight");
+    let (wo_m, wo_k) = tensor_dims(g, &wo_name)?;
+    let (wo, wo_marlin) = q4dense_and_marlin_bytes_cached(cache, &wo_name, wo_m, wo_k, marlin, || g.dequant_f32(&wo_name))?;
+
+    let attn_norm_name = format!("{p}.attn_norm.weight");
+    let q_norm_name = format!("{p}.attn_q_norm.weight");
+    let k_norm_name = format!("{p}.attn_k_norm.weight");
+    let attn = AttnWeightsBytes {
+        n_head,
+        attn_norm: f32vec_bytes_cached(cache, &attn_norm_name, || g.dequant_f32(&attn_norm_name))?,
+        proj,
+        wo,
+        q_norm: f32vec_bytes_cached(cache, &q_norm_name, || g.dequant_f32(&q_norm_name))?,
+        k_norm: f32vec_bytes_cached(cache, &k_norm_name, || g.dequant_f32(&k_norm_name))?,
+        qkv_marlin,
+        g_proj,
+        wo_marlin,
+    };
+
+    let ffn = if cfg.is_dense_layer(i) {
+        // BUTTER_LAGUNA_MARLIN=1: [gate;up] concat (2*n_ff_dense=24576,
+        // 64-aligned) plus down, both Marlin-packed alongside the plain
+        // per-projection Q4Dense copies decode/gemm_q4_mpp use.
+        let gate_name2 = format!("{p}.ffn_gate.weight");
+        let up_name2 = format!("{p}.ffn_up.weight");
+        let down_name2 = format!("{p}.ffn_down.weight");
+        let (ff_m, ff_k) = tensor_dims(g, &gate_name2)?;
+        let gate = q4dense_bytes_from_flat_cached(cache, &gate_name2, ff_m, ff_k, || g.dequant_f32(&gate_name2))?;
+        let up = q4dense_bytes_from_flat_cached(cache, &up_name2, ff_m, ff_k, || g.dequant_f32(&up_name2))?;
+        let (down_m, down_k) = tensor_dims(g, &down_name2)?;
+        let (down, down_marlin) = q4dense_and_marlin_bytes_cached(cache, &down_name2, down_m, down_k, marlin, || g.dequant_f32(&down_name2))?;
+        let gateup_marlin = if marlin && (2 * ff_m) % 64 == 0 {
+            let mat = marlindense_bytes_from_flat_cached(cache, &format!("{p}.ffn_gateup"), 2 * ff_m, ff_k, || {
+                let mut concat = Vec::with_capacity(2 * ff_m * ff_k);
+                concat.extend_from_slice(&g.dequant_f32(&gate_name2)?);
+                concat.extend_from_slice(&g.dequant_f32(&up_name2)?);
+                Ok(concat)
+            })?;
+            Some(GateUpMarlinBytes { mat, n_ff: ff_m })
+        } else {
+            None
+        };
+        let ffn_norm_name = format!("{p}.ffn_norm.weight");
+        LayerFfnBytes::Dense(DenseFfnBytes {
+            ffn_norm: f32vec_bytes_cached(cache, &ffn_norm_name, || g.dequant_f32(&ffn_norm_name))?,
+            gate,
+            up,
+            down,
+            gateup_marlin,
+            down_marlin,
+        })
+    } else {
+        let router_name = format!("{p}.ffn_gate_inp.weight");
+        let router = f32vec_bytes_cached(cache, &router_name, || g.dequant_f32(&router_name))?;
+
+        let gate_shexp = format!("{p}.ffn_gate_shexp.weight");
+        let up_shexp = format!("{p}.ffn_up_shexp.weight");
+        let down_shexp = format!("{p}.ffn_down_shexp.weight");
+
+        // BUTTER_LAGUNA_MICRO=1: repack the shared expert's gate+up as
+        // single-expert (n_expert=1) stacks so decode can drive them through
+        // the same moe_gather_q4_swiglu the routed experts use (top_k=1)
+        // instead of two plain gemvs + a host-side swiglu. `down` stays a
+        // plain Q4Dense either way. Only one layout is ever built per layer
+        // (same memory-doubling rationale as AttnProj above).
+        let shared_expert = if micro {
+            let gate = q4_expert_stack_bytes_cached(cache, &format!("{gate_shexp}.se1"), 1, cfg.n_ff_shexp, cfg.hidden, || g.dequant_f32(&gate_shexp))?;
+            let up = q4_expert_stack_bytes_cached(cache, &format!("{up_shexp}.se1"), 1, cfg.n_ff_shexp, cfg.hidden, || g.dequant_f32(&up_shexp))?;
+            let (dm, dk) = tensor_dims(g, &down_shexp)?;
+            let down = q4dense_bytes_from_flat_cached(cache, &down_shexp, dm, dk, || g.dequant_f32(&down_shexp))?;
+            SharedExpertBytes::Fused(SharedExpertFusedBytes { gate, up, down })
+        } else {
+            let (gm, gk) = tensor_dims(g, &gate_shexp)?;
+            let (um, uk) = tensor_dims(g, &up_shexp)?;
+            let (dm, dk) = tensor_dims(g, &down_shexp)?;
+            SharedExpertBytes::Split {
+                gate: q4dense_bytes_from_flat_cached(cache, &gate_shexp, gm, gk, || g.dequant_f32(&gate_shexp))?,
+                up: q4dense_bytes_from_flat_cached(cache, &up_shexp, um, uk, || g.dequant_f32(&up_shexp))?,
+                down: q4dense_bytes_from_flat_cached(cache, &down_shexp, dm, dk, || g.dequant_f32(&down_shexp))?,
+            }
+        };
+
+        // BUTTER_LAGUNA_MARLIN=1: the shared expert's [gate;up] concat
+        // (2*n_ff_shexp=2048, 64-aligned) plus down, Marlin-packed straight
+        // from the shexp GGUF tensors, independent of whether
+        // `shared_expert` above resolved to Split or Fused. The routed
+        // per-expert stacks (gate/up/down just below) are NOT Marlin-packed:
+        // prefill already drives them through the grouped-MMA GEMM
+        // (`ops::moe_q4_grouped_mma_dev`), a different tensor-core kernel
+        // this Marlin work doesn't target.
+        let shared_marlin = if marlin {
+            let (sm, sk) = tensor_dims(g, &gate_shexp)?;
+            let (sdm, sdk) = tensor_dims(g, &down_shexp)?;
+            if (2 * sm) % 64 == 0 && sdm % 64 == 0 {
+                let mat = marlindense_bytes_from_flat_cached(cache, &format!("{p}.shexp_gateup"), 2 * sm, sk, || {
+                    let mut concat = Vec::with_capacity(2 * sm * sk);
+                    concat.extend_from_slice(&g.dequant_f32(&gate_shexp)?);
+                    concat.extend_from_slice(&g.dequant_f32(&up_shexp)?);
+                    Ok(concat)
+                })?;
+                let down = marlindense_bytes_from_flat_cached(cache, &format!("{down_shexp}.standalone"), sdm, sdk, || g.dequant_f32(&down_shexp))?;
+                Some(SharedMarlinBytes { gateup: GateUpMarlinBytes { mat, n_ff: sm }, down })
+            } else {
+                None
+            }
+        } else {
+            None
+        };
+
+        let gate_exps_name = format!("{p}.ffn_gate_exps.weight");
+        let up_exps_name = format!("{p}.ffn_up_exps.weight");
+        let down_exps_name = format!("{p}.ffn_down_exps.weight");
+
+        // BUTTER_LAGUNA_W4A8=1: the routed gate/up/down stacks are built
+        // sequentially afterward (`build_moe_w4a8_sequential`), see
+        // `MoeFfnBytes::gateup`'s doc.
+        let (gateup, down) = if w4a8 {
+            (None, None)
+        } else if moefuse {
+            // BUTTER_LAGUNA_MOEFUSE=1: build ONLY the fused [gate;up] stack
+            // (see RoutedGateUp::Fused's doc for the row layout), never both
+            // layouts at once, the routed expert stacks are the bulk of
+            // the model's resident memory and a second copy isn't
+            // affordable.
+            let per_expert = cfg.n_ff_exp * cfg.hidden;
+            let n_expert = cfg.n_expert;
+            let (n_ff_exp, hidden) = (cfg.n_ff_exp, cfg.hidden);
+            let fused = q4_expert_stack_bytes_cached(cache, &format!("{p}.ffn_gateup_exps_fused"), n_expert, 2 * n_ff_exp, hidden, || {
+                let gate_flat = g.dequant_f32(&gate_exps_name)?;
+                let up_flat = g.dequant_f32(&up_exps_name)?;
+                if gate_flat.len() != per_expert * n_expert || up_flat.len() != per_expert * n_expert {
+                    return Err(Error::Msg(format!(
+                        "ffn_gate/up_exps len mismatch gate={} up={} expected {} each (n_expert={n_expert} n_ff_exp={n_ff_exp} hidden={hidden})",
+                        gate_flat.len(), up_flat.len(), per_expert * n_expert,
+                    )));
+                }
+                let mut fused = Vec::with_capacity(2 * per_expert * n_expert);
+                for e in 0..n_expert {
+                    fused.extend_from_slice(&gate_flat[e * per_expert..(e + 1) * per_expert]);
+                    fused.extend_from_slice(&up_flat[e * per_expert..(e + 1) * per_expert]);
+                }
+                Ok(fused)
+            })?;
+            let down = q4_expert_stack_bytes_cached(cache, &down_exps_name, cfg.n_expert, cfg.hidden, cfg.n_ff_exp, || g.dequant_f32(&down_exps_name))?;
+            (Some(RoutedGateUpBytes::Fused(fused)), Some(down))
+        } else {
+            let gate = q4_expert_stack_bytes_cached(cache, &gate_exps_name, cfg.n_expert, cfg.n_ff_exp, cfg.hidden, || g.dequant_f32(&gate_exps_name))?;
+            let up = q4_expert_stack_bytes_cached(cache, &up_exps_name, cfg.n_expert, cfg.n_ff_exp, cfg.hidden, || g.dequant_f32(&up_exps_name))?;
+            let down = q4_expert_stack_bytes_cached(cache, &down_exps_name, cfg.n_expert, cfg.hidden, cfg.n_ff_exp, || g.dequant_f32(&down_exps_name))?;
+            (Some(RoutedGateUpBytes::Split { gate, up }), Some(down))
+        };
+
+        let ffn_norm_name = format!("{p}.ffn_norm.weight");
+        let bias_name = format!("{p}.exp_probs_b.bias");
+        LayerFfnBytes::Moe(MoeFfnBytes {
+            ffn_norm: f32vec_bytes_cached(cache, &ffn_norm_name, || g.dequant_f32(&ffn_norm_name))?,
+            router,
+            bias: f32vec_bytes_cached(cache, &bias_name, || g.dequant_f32(&bias_name))?,
+            gateup,
+            down,
+            shared_expert,
+            shared_marlin,
+        })
+    };
+
+    Ok(LayerBytes { attn, ffn })
+}
+
+/// BUTTER_LAGUNA_W4A8=1 sequential (non-rayon-parallel) build of one MoE
+/// layer's routed gate/up/down. Mirrors the original inline `load_gguf`
+/// logic exactly: dequantize each routed GGUF tensor ONCE and feed BOTH the
+/// resident Q4 stack decode needs and the ADDITIONAL W4A8-packed stack
+/// prefill needs (see [`MoeFfnW4a8`]'s doc), the Q4 side is routed through
+/// [`CacheCtx`] (pure CPU; kept here rather than in the parallel
+/// `build_layer_bytes` pass purely so it shares that one dequant with the
+/// W4A8 pack below instead of paying for it twice) and the W4A8 pack itself
+/// through [`pack_w4a8_expert_stack_cached`] (a GPU op, must run on this
+/// thread, see that fn's doc). Not parallelized across layers: acceptable
+/// since BUTTER_LAGUNA_W4A8 is an opt-in, default-off, prefill-throughput
+/// experiment, not the path the "fast load" goal targets.
+fn build_moe_w4a8_sequential(dev: &dyn Device, g: &Gguf, cache: &CacheCtx, cfg: &LagunaConfig, i: usize) -> Result<(RoutedGateUp, Q4ExpertStack, Option<MoeFfnW4a8>)> {
+    let p = format!("blk.{i}");
+    let gate_exps_name = format!("{p}.ffn_gate_exps.weight");
+    let up_exps_name = format!("{p}.ffn_up_exps.weight");
+    let down_exps_name = format!("{p}.ffn_down_exps.weight");
+    let moefuse = laguna_moefuse_enabled();
+    // BUTTER_LAGUNA_W4A8_ONLY=1: PREFILL BENCH MODE. Dual expert residency (Q4
+    // for decode + mxfp4 for W4A8 prefill) is ~112GB and cannot fit; this
+    // mode uploads a 1-expert DUMMY Q4 stack (decode would be wrong, never
+    // run it in this mode) so the W4A8 prefill speed can be measured. The
+    // real fix is a single shared expert format both paths read.
+    let w4a8_only = std::env::var("BUTTER_LAGUNA_W4A8_ONLY").map(|v| v == "1").unwrap_or(false);
+
+    let gate_flat = g.dequant_f32(&gate_exps_name)?;
+    let up_flat = g.dequant_f32(&up_exps_name)?;
+
+    let (gateup, gateup_w4a8) = if moefuse {
+        let per_expert = cfg.n_ff_exp * cfg.hidden;
+        if gate_flat.len() != per_expert * cfg.n_expert || up_flat.len() != per_expert * cfg.n_expert {
+            return Err(Error::Msg(format!(
+                "ffn_gate/up_exps len mismatch gate={} up={} expected {} each (n_expert={} n_ff_exp={} hidden={})",
+                gate_flat.len(), up_flat.len(), per_expert * cfg.n_expert, cfg.n_expert, cfg.n_ff_exp, cfg.hidden
+            )));
+        }
+        let mut fused = Vec::with_capacity(2 * per_expert * cfg.n_expert);
+        for e in 0..cfg.n_expert {
+            fused.extend_from_slice(&gate_flat[e * per_expert..(e + 1) * per_expert]);
+            fused.extend_from_slice(&up_flat[e * per_expert..(e + 1) * per_expert]);
+        }
+        let w4a8s = pack_w4a8_expert_stack_cached(dev, cache, &format!("{p}.ffn_gateup_exps_fused"), &fused, cfg.n_expert, 2 * cfg.n_ff_exp, cfg.hidden)?;
+        let q4 = if w4a8_only {
+            let b = q4_expert_stack_bytes_cached(cache, &format!("{p}.ffn_gateup_exps_fused.w4a8only1"), 1, 2 * cfg.n_ff_exp, cfg.hidden, || Ok(fused[..2 * per_expert].to_vec()))?;
+            upload_expert_stack_bytes(dev, b)?
+        } else {
+            let b = q4_expert_stack_bytes_cached(cache, &format!("{p}.ffn_gateup_exps_fused"), cfg.n_expert, 2 * cfg.n_ff_exp, cfg.hidden, || Ok(fused))?;
+            upload_expert_stack_bytes(dev, b)?
+        };
+        (RoutedGateUp::Fused(q4), Some(RoutedGateUpW4a8::Fused(w4a8s)))
+    } else {
+        let gate_b = q4_expert_stack_bytes_cached(cache, &gate_exps_name, cfg.n_expert, cfg.n_ff_exp, cfg.hidden, || Ok(gate_flat.clone()))?;
+        let up_b = q4_expert_stack_bytes_cached(cache, &up_exps_name, cfg.n_expert, cfg.n_ff_exp, cfg.hidden, || Ok(up_flat.clone()))?;
+        let gate_q4 = upload_expert_stack_bytes(dev, gate_b)?;
+        let up_q4 = upload_expert_stack_bytes(dev, up_b)?;
+        let gate_w4a8 = pack_w4a8_expert_stack_cached(dev, cache, &gate_exps_name, &gate_flat, cfg.n_expert, cfg.n_ff_exp, cfg.hidden)?;
+        let up_w4a8 = pack_w4a8_expert_stack_cached(dev, cache, &up_exps_name, &up_flat, cfg.n_expert, cfg.n_ff_exp, cfg.hidden)?;
+        (RoutedGateUp::Split { gate: gate_q4, up: up_q4 }, Some(RoutedGateUpW4a8::Split { gate: gate_w4a8, up: up_w4a8 }))
+    };
+
+    let down_flat = g.dequant_f32(&down_exps_name)?;
+    let down_w4a8 = pack_w4a8_expert_stack_cached(dev, cache, &down_exps_name, &down_flat, cfg.n_expert, cfg.hidden, cfg.n_ff_exp)?;
+    let down = if w4a8_only {
+        let b = q4_expert_stack_bytes_cached(cache, &format!("{down_exps_name}.w4a8only1"), 1, cfg.hidden, cfg.n_ff_exp, || Ok(down_flat[..cfg.hidden * cfg.n_ff_exp].to_vec()))?;
+        upload_expert_stack_bytes(dev, b)?
+    } else {
+        let b = q4_expert_stack_bytes_cached(cache, &down_exps_name, cfg.n_expert, cfg.hidden, cfg.n_ff_exp, || Ok(down_flat))?;
+        upload_expert_stack_bytes(dev, b)?
+    };
+
+    // Both halves are always built together in this fn (it only ever runs
+    // under BUTTER_LAGUNA_W4A8=1), so `gateup_w4a8` is always `Some` here:
+    // `.expect` just makes that invariant explicit instead of a silent
+    // `.unwrap()`.
+    let w4a8_moe = Some(MoeFfnW4a8 { gateup: gateup_w4a8.expect("w4a8 branch always builds a gateup pack"), down: down_w4a8 });
+    Ok((gateup, down, w4a8_moe))
+}
+
+/// Sequential half of loading one layer: upload every byte blob
+/// `build_layer_bytes` produced (or, under BUTTER_LAGUNA_W4A8=1, that this fn
+/// itself builds via `build_moe_w4a8_sequential`, see [`MoeFfnBytes::gateup`]'s
+/// doc) and assemble the `Tensor`-based [`LagunaLayer`]. Never parallelized
+/// across layers, `dev.upload`/device dispatch calls only ever happen on
+/// the caller's thread (the loop in `load_gguf`), since backends aren't
+/// guaranteed safe to drive concurrently from multiple threads.
+fn upload_layer(dev: &dyn Device, g: &Gguf, cache: &CacheCtx, cfg: &LagunaConfig, i: usize, w4a8: bool, lb: LayerBytes) -> Result<LagunaLayer> {
+    let attn = {
+        let a = lb.attn;
+        let proj = match a.proj {
+            AttnProjBytes::Fused { wqkvg, n_q_rows, n_kv_rows, n_g_rows } => {
+                AttnProj::Fused { wqkvg: upload_q4dense_bytes(dev, wqkvg)?, n_q_rows, n_kv_rows, n_g_rows }
+            }
+            AttnProjBytes::Split { wq, wk, wv, wgate } => AttnProj::Split {
+                wq: upload_q4dense_bytes(dev, wq)?,
+                wk: upload_q4dense_bytes(dev, wk)?,
+                wv: upload_q4dense_bytes(dev, wv)?,
+                wgate: upload_q4dense_bytes(dev, wgate)?,
+            },
+        };
+        let wo = upload_q4dense_bytes(dev, a.wo)?;
+        let qkv_marlin = a.qkv_marlin.map(|qm| -> Result<QkvMarlin> {
+            Ok(QkvMarlin { mat: upload_marlindense_bytes(dev, qm.mat)?, n_q_rows: qm.n_q_rows, n_kv_rows: qm.n_kv_rows })
+        }).transpose()?;
+        let g_proj = a.g_proj.map(|b| upload_q4dense_bytes(dev, b)).transpose()?;
+        let wo_marlin = a.wo_marlin.map(|b| upload_marlindense_bytes(dev, b)).transpose()?;
+        AttnWeights {
+            n_head: a.n_head,
+            attn_norm: upload_f32_bytes(dev, a.attn_norm)?,
+            proj,
+            wo,
+            q_norm: upload_f32_bytes(dev, a.q_norm)?,
+            k_norm: upload_f32_bytes(dev, a.k_norm)?,
+            qkv_marlin,
+            g_proj,
+            wo_marlin,
+        }
+    };
+
+    let ffn = match lb.ffn {
+        LayerFfnBytes::Dense(d) => {
+            let gateup_marlin = d.gateup_marlin.map(|gm| -> Result<GateUpMarlin> {
+                Ok(GateUpMarlin { mat: upload_marlindense_bytes(dev, gm.mat)?, n_ff: gm.n_ff })
+            }).transpose()?;
+            LayerFfn::Dense(DenseFfn {
+                ffn_norm: upload_f32_bytes(dev, d.ffn_norm)?,
+                gate: upload_q4dense_bytes(dev, d.gate)?,
+                up: upload_q4dense_bytes(dev, d.up)?,
+                down: upload_q4dense_bytes(dev, d.down)?,
+                gateup_marlin,
+                down_marlin: d.down_marlin.map(|b| upload_marlindense_bytes(dev, b)).transpose()?,
+            })
+        }
+        LayerFfnBytes::Moe(m) => {
+            let shared_expert = match m.shared_expert {
+                SharedExpertBytes::Split { gate, up, down } => SharedExpert::Split {
+                    gate: upload_q4dense_bytes(dev, gate)?,
+                    up: upload_q4dense_bytes(dev, up)?,
+                    down: upload_q4dense_bytes(dev, down)?,
+                },
+                SharedExpertBytes::Fused(f) => {
+                    let zero_idx = Tensor::new(dev.upload(&0u32.to_le_bytes())?, vec![1], DType::U32);
+                    SharedExpert::Fused(SharedExpertFused {
+                        gate: upload_expert_stack_bytes(dev, f.gate)?,
+                        up: upload_expert_stack_bytes(dev, f.up)?,
+                        down: upload_q4dense_bytes(dev, f.down)?,
+                        zero_idx,
+                    })
+                }
+            };
+            let shared_marlin = m.shared_marlin.map(|sm| -> Result<SharedMarlin> {
+                Ok(SharedMarlin {
+                    gateup: GateUpMarlin { mat: upload_marlindense_bytes(dev, sm.gateup.mat)?, n_ff: sm.gateup.n_ff },
+                    down: upload_marlindense_bytes(dev, sm.down)?,
+                })
+            }).transpose()?;
+
+            let (gateup, down, w4a8_moe) = if w4a8 {
+                build_moe_w4a8_sequential(dev, g, cache, cfg, i)?
+            } else {
+                let gateup = match m.gateup.expect("non-W4A8 build_layer_bytes always sets gateup") {
+                    RoutedGateUpBytes::Fused(q) => RoutedGateUp::Fused(upload_expert_stack_bytes(dev, q)?),
+                    RoutedGateUpBytes::Split { gate, up } => {
+                        RoutedGateUp::Split { gate: upload_expert_stack_bytes(dev, gate)?, up: upload_expert_stack_bytes(dev, up)? }
+                    }
+                };
+                let down = upload_expert_stack_bytes(dev, m.down.expect("non-W4A8 build_layer_bytes always sets down"))?;
+                (gateup, down, None)
+            };
+
+            LayerFfn::Moe(MoeFfn {
+                ffn_norm: upload_f32_bytes(dev, m.ffn_norm)?,
+                router: Tensor::new(dev.upload(&m.router)?, vec![cfg.n_expert, cfg.hidden], DType::F32),
+                bias: upload_f32_bytes(dev, m.bias)?,
+                gateup,
+                down,
+                shared_expert,
+                shared_marlin,
+                w4a8: w4a8_moe,
+            })
+        }
+    };
+
+    Ok(LagunaLayer { attn, ffn })
+}
+
+/// Load Laguna-S-2.1 straight from a GGUF file: parse `laguna.*` metadata for
+/// the config, then stream every tensor through `dequant_f32` → requantize
+/// (Q4 for projections/experts, dense f32 for embeddings/norms/router) →
+/// upload, dropping the f32 immediately after each tensor.
+///
+/// Every uploaded byte blob goes through an on-disk [`CacheCtx`] first (see
+/// that struct's doc): a warm reload of an unchanged GGUF under the same
+/// `BUTTER_LAGUNA_*` flags skips straight to `read file -> upload` for every
+/// weight, instead of redoing the dequant+requant work. On a cache MISS, the
+/// pure-CPU half of that work (`build_layer_bytes`) runs across a rayon
+/// thread pool, one task per layer: layers are fully independent (no shared
+/// mutable state besides the cache's own atomic hit/miss counters and the
+/// read-only `g`/`cfg`), so this is an embarrassingly parallel fan-out.
+/// Every `dev.upload` call happens afterward, sequentially, in layer order
+/// (`upload_layer`), backends aren't guaranteed safe to drive from multiple
+/// threads at once, so nothing in the parallel phase ever touches `dev`.
+pub fn load_gguf(dev: &dyn Device, g: &Gguf) -> Result<LagunaModel> {
+    let load_t0 = std::time::Instant::now();
+    let cfg = LagunaConfig::from_gguf(g);
+    let cache = CacheCtx::new(g);
+
+    // token_embd / output are [vocab, hidden] (GGUF fastest-first dims
+    // already give the flat data in row-major [vocab, hidden] order, see
+    // `dims_out_in`).
+    let tok_embd = {
+        let bytes = f32vec_bytes_cached(&cache, "token_embd.weight", || g.dequant_f32("token_embd.weight"))?;
+        Tensor::new(dev.upload(&bytes)?, vec![cfg.vocab, cfg.hidden], DType::F32)
+    };
+
+    let out_norm = {
+        let bytes = f32vec_bytes_cached(&cache, "output_norm.weight", || g.dequant_f32("output_norm.weight"))?;
+        upload_f32_bytes(dev, bytes)?
+    };
+    let head_name = if g.tensor("output.weight").is_some() { "output.weight" } else { "token_embd.weight" };
+    let lm_head = {
+        let (qs_bytes, sc_bytes, m, k) = q8head_bytes_cached(&cache, g, head_name, head_name)?;
+        Q8Head {
+            qs: Tensor::new(dev.upload(&qs_bytes)?, vec![qs_bytes.len() / 4], DType::U32),
+            // gemv_q8 reads f32 scales (unlike the q4 family's f16), matching
+            // the proven resident-Q8 loader path.
+            scales: Tensor::new(dev.upload(&sc_bytes)?, vec![sc_bytes.len() / 4], DType::F32),
+            m,
+            k,
+        }
+    };
+    // Tiny process-lifetime constant, not worth routing through the cache.
+    let one = Tensor::new(dev.upload(&1.0f32.to_le_bytes())?, vec![1], DType::F32);
+
+    let micro = laguna_micro_enabled();
+    let marlin = laguna_marlin_enabled();
+    let moefuse = laguna_moefuse_enabled();
+    let w4a8 = laguna_w4a8_enabled();
+
+    // Pure-CPU half, rayon-parallel WITHIN a bounded window of layers.
+    // Building all 48 layers before uploading holds tens of GB of blob bytes
+    // plus each worker's multi-GB f32 dequant transients at once, which
+    // OOM-kills the process on the shared 128GB. Window of 4: at most 4
+    // layers' transients + blobs in flight, uploaded and dropped before the
+    // next window starts. BUTTER_LAGUNA_LOAD_WINDOW overrides.
+    let window: usize = std::env::var("BUTTER_LAGUNA_LOAD_WINDOW")
+        .ok().and_then(|v| v.parse().ok()).filter(|&w| w >= 1).unwrap_or(4);
+    let mut layers = Vec::with_capacity(cfg.n_layers);
+    let mut start_l = 0usize;
+    while start_l < cfg.n_layers {
+        let end_l = (start_l + window).min(cfg.n_layers);
+        let batch: Vec<LayerBytes> = (start_l..end_l)
+            .into_par_iter()
+            .map(|i| build_layer_bytes(g, &cache, &cfg, i, micro, marlin, moefuse, w4a8))
+            .collect::<Result<Vec<_>>>()?;
+        // Sequential half: every dev.upload / device dispatch call, in order.
+        for (off, lb) in batch.into_iter().enumerate() {
+            layers.push(upload_layer(dev, g, &cache, &cfg, start_l + off, w4a8, lb)?);
+        }
+        start_l = end_l;
+    }
+
+    let (hits, misses) = cache.stats();
+    eprintln!(
+        "Laguna: load done in {:.2}s ({hits} cache hits, {misses} misses, cache dir {})",
+        load_t0.elapsed().as_secs_f64(),
+        cache.dir_display(),
+    );
+
+    Ok(LagunaModel { cfg, tok_embd, layers, out_norm, lm_head, one })
+}
+
+/// One decode layer: RMSNorm → QKV proj (+ per-head gate proj on the same
+/// normed input) → per-head QK-RMSNorm → RoPE → KV-cache append → SDPA →
+/// per-head softplus gate → o_proj → residual → RMSNorm → FFN (dense or
+/// MoE) → residual. `pos` is the token's absolute sequence position.
+#[allow(clippy::too_many_arguments)]
+/// BUTTER_LAGUNA_DEBUG=2: dump a tensor's stats (first-bad-op bisection inside
+/// a layer). No-op cost is one env read per call site when disabled upstream.
+fn dbg_stats(dev: &dyn Device, tag: &str, t: &Tensor) {
+    let n = t.elem_count();
+    let mut b = vec![0u8; n * 4];
+    if dev.synchronize().is_err() || dev.download(t.buffer.as_ref(), &mut b).is_err() {
+        eprintln!("    {tag}: <download failed>");
+        return;
+    }
+    let f: Vec<f32> = b.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+    let n_nan = f.iter().filter(|v| v.is_nan()).count();
+    let l2 = f.iter().map(|v| (*v as f64) * (*v as f64)).sum::<f64>().sqrt();
+    let amax = f.iter().filter(|v| v.is_finite()).fold(0f32, |a, &v| a.max(v.abs()));
+    eprintln!("    {tag}: n={n} |x|2={l2:.3e} amax={amax:.3e} nan={n_nan} head={:?}", &f[..4.min(n)]);
+}
+
+/// Per-step device buffers that make `decode_layer` CUDA-graph-capturable:
+/// a captured graph replays the exact same recorded kernel launches every
+/// token, so any value that changes per token (RoPE position, KV length)
+/// must be read from a buffer whose CONTENTS the caller refreshes between
+/// replays, rather than passed as a scalar baked in at capture time. The
+/// buffer POINTERS are fixed for the lifetime of a `LagunaDecodeCtx`; only
+/// the u32 values inside them change. `n_kv_full`/`n_kv_sliding` hold
+/// `pos+1` / `min(pos+1, sliding_window)` respectively, which one a given
+/// layer reads is chosen at call time by `cfg.is_full_layer(layer_idx)`,
+/// which never changes across replays, so the choice of POINTER is itself
+/// capture-stable even though the VALUE behind it isn't.
+struct GraphBufs<'a> {
+    n_kv_full: &'a Tensor,
+    n_kv_sliding: &'a Tensor,
+}
+
+#[allow(clippy::too_many_arguments)]
+fn decode_layer(
+    dev: &dyn Device,
+    cfg: &LagunaConfig,
+    layer_idx: usize,
+    w: &LagunaLayer,
+    kv: &KvSlot,
+    x: &Tensor,
+    pos: u32,
+    full_posbuf: &Tensor,
+    sliding_posbuf: &Tensor,
+    one: &Tensor,
+    graph_bufs: Option<&GraphBufs>,
+    micro: bool,
+) -> Result<Tensor> {
+    let deep_dbg = layer_idx == 0
+        && std::env::var("BUTTER_LAGUNA_DEBUG").map(|v| v == "2").unwrap_or(false);
+    let hd = cfg.head_dim;
+    let n_head = w.attn.n_head;
+    let n_kv_heads = cfg.n_kv_heads;
+    let is_full = cfg.is_full_layer(layer_idx);
+    let rope = cfg.rope_for(layer_idx);
+    let heads_per_group = (n_head / n_kv_heads) as u32;
+
+    // ── attention ────────────────────────────────────────────────────
+    if deep_dbg { dbg_stats(dev, "x_in", x); }
+    let h = ops::rms_norm(dev, x, &w.attn.attn_norm, cfg.eps)?;
+    if deep_dbg { dbg_stats(dev, "attn_norm", &h); }
+    // g_proj consumes the SAME pre-attention normed input as q/k/v (not the
+    // attention output), matches the reference engine's graph exactly.
+    //
+    // BUTTER_LAGUNA_MICRO=1 (fusion 1): one gemv_q4 on the concatenated
+    // [wq;wk;wv;wgate] weight instead of four separate gemvs, then four
+    // device-side slices (cheap 36KB/4KB/4KB/0.2KB copies) split the result
+    // back into q/k/v/gate_raw. `ops::slice` is used rather than zero-copy
+    // Tensor offset views because the gemv/gather kernels bind buffer BASE
+    // pointers and ignore Tensor.offset.
+    let (q, k, v, gate_raw) = match &w.attn.proj {
+        AttnProj::Fused { wqkvg, n_q_rows, n_kv_rows, n_g_rows } => {
+            let qkvg = wqkvg.gemv(dev, &h)?;
+            let q = ops::slice(dev, &qkvg, 0, *n_q_rows)?;
+            let k = ops::slice(dev, &qkvg, *n_q_rows, *n_kv_rows)?;
+            let v = ops::slice(dev, &qkvg, n_q_rows + n_kv_rows, *n_kv_rows)?;
+            let gate_raw = ops::slice(dev, &qkvg, n_q_rows + 2 * n_kv_rows, *n_g_rows)?;
+            (q, k, v, gate_raw)
+        }
+        AttnProj::Split { wq, wk, wv, wgate } => {
+            let q = wq.gemv(dev, &h)?;
+            let k = wk.gemv(dev, &h)?;
+            let v = wv.gemv(dev, &h)?;
+            let gate_raw = wgate.gemv(dev, &h)?; // [n_head]
+            (q, k, v, gate_raw)
+        }
+    };
+    if deep_dbg {
+        dbg_stats(dev, "q_proj", &q);
+        dbg_stats(dev, "k_proj", &k);
+        dbg_stats(dev, "v_proj", &v);
+        dbg_stats(dev, "g_proj", &gate_raw);
+    }
+
+    let q = q.reshaped(vec![n_head, hd]);
+    let k = k.reshaped(vec![n_kv_heads, hd]);
+    let v = v.reshaped(vec![n_kv_heads, hd]);
+
+    // Per-head QK RMSNorm (head_dim-level), before RoPE.
+    let q = ops::rms_norm(dev, &q, &w.attn.q_norm, cfg.eps)?;
+    let k = ops::rms_norm(dev, &k, &w.attn.k_norm, cfg.eps)?;
+    if deep_dbg {
+        dbg_stats(dev, "q_norm", &q);
+        dbg_stats(dev, "k_norm", &k);
+    }
+
+    // YaRN (full layers) / plain (sliding layers) partial-rotary RoPE.
+    // Graph mode reads `position` from the (capture-stable-pointer) posbuf
+    // instead of taking it as a scalar baked in at capture time, same math,
+    // see `ops::rope_yarn_partial_posbuf`'s doc.
+    let (q, k) = if graph_bufs.is_some() {
+        let posbuf = if is_full { full_posbuf } else { sliding_posbuf };
+        let q = ops::rope_yarn_partial_posbuf(
+            dev, &q, posbuf, rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor,
+            rope.attn_factor, rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+        )?;
+        let k = ops::rope_yarn_partial_posbuf(
+            dev, &k, posbuf, rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor,
+            rope.attn_factor, rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+        )?;
+        (q, k)
+    } else {
+        let q = ops::rope_yarn_partial(
+            dev, &q, pos, rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor,
+            rope.attn_factor, rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+        )?;
+        let k = ops::rope_yarn_partial(
+            dev, &k, pos, rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor,
+            rope.attn_factor, rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+        )?;
+        (q, k)
+    };
+    if deep_dbg {
+        dbg_stats(dev, "q_rope", &q);
+        dbg_stats(dev, "k_rope", &k);
+    }
+
+    // KV cache append: full layers grow (posbuf = pos), sliding layers write
+    // a ring slot (posbuf = pos % sliding_window). RoPE already used the
+    // ABSOLUTE position above, so ring order doesn't affect the attention
+    // math, only which slot gets overwritten.
+    let posbuf = if is_full { full_posbuf } else { sliding_posbuf };
+    ops::kv_append(dev, &k, &kv.k, posbuf, hd, kv.cap, n_kv_heads * hd)?;
+    ops::kv_append(dev, &v, &kv.v, posbuf, hd, kv.cap, n_kv_heads * hd)?;
+
+    // Graph mode reads the filled KV length from a (capture-stable-pointer)
+    // device buffer instead of a baked-in scalar, same dense-causal math,
+    // see `ops::sdpa_decode_nbuf`'s doc.
+    let attn = if let Some(gb) = graph_bufs {
+        let n_kv_buf = if is_full { gb.n_kv_full } else { gb.n_kv_sliding };
+        ops::sdpa_decode_nbuf(
+            dev, &q, &kv.k, &kv.v, hd, n_kv_buf, kv.cap as u32, heads_per_group, cfg.attn_scale(),
+        )?
+    } else {
+        let n_kv_used = if is_full { pos + 1 } else { (pos + 1).min(cfg.sliding_window as u32) };
+        ops::sdpa_decode(
+            dev, &q, &kv.k, &kv.v, hd, n_kv_used, kv.cap as u32, heads_per_group, cfg.attn_scale(),
+        )?
+    }; // [n_head, hd]
+    if deep_dbg { dbg_stats(dev, "sdpa", &attn); }
+
+    // Per-head softplus gate, before o_proj.
+    let gated = ops::gate_softplus_mul_perhead(dev, &attn, &gate_raw)?;
+    let gated_flat = gated.reshaped(vec![n_head * hd]);
+    if deep_dbg { dbg_stats(dev, "gated", &gated); }
+    // BUTTER_LAGUNA_MICRO=1 (fusion 3): copy x into a fresh accumulator (there
+    // is no tensor-clone op; `ops::slice(dev, x, 0, hidden)` is a device copy)
+    // then `wo.gemv_accum` computes x1 = x + wo·gated in one kernel, instead
+    // of materializing `o = wo.gemv(...)` and adding it separately. Same op
+    // count (copy+accum vs gemv+add), this fusion trades materializing `o`
+    // for materializing the copy, so it wins only if the accum kernel avoids
+    // enough bandwidth to offset that; benchmarking decides, not this diff.
+    let x1 = if micro {
+        let acc = ops::slice(dev, x, 0, cfg.hidden)?;
+        w.attn.wo.gemv_accum(dev, &gated_flat, &acc, one)?;
+        if deep_dbg { dbg_stats(dev, "x_attn_res", &acc); }
+        acc
+    } else {
+        let o = w.attn.wo.gemv(dev, &gated_flat)?;
+        let x1 = ops::add(dev, x, &o)?;
+        if deep_dbg {
+            dbg_stats(dev, "o_proj", &o);
+            dbg_stats(dev, "x_attn_res", &x1);
+        }
+        x1
+    };
+
+    // ── FFN ──────────────────────────────────────────────────────────
+    let x2 = match &w.ffn {
+        LayerFfn::Dense(f) => {
+            let h2 = ops::rms_norm(dev, &x1, &f.ffn_norm, cfg.eps)?;
+            let gate = f.gate.gemv(dev, &h2)?;
+            let up = f.up.gemv(dev, &h2)?;
+            let act = ops::swiglu(dev, &gate, &up)?;
+            let down = f.down.gemv(dev, &act)?;
+            if deep_dbg {
+                dbg_stats(dev, "ffn_norm", &h2);
+                dbg_stats(dev, "ffn_act", &act);
+                dbg_stats(dev, "ffn_down", &down);
+            }
+            ops::add(dev, &x1, &down)?
+        }
+        LayerFfn::Moe(f) => {
+            let moe_dbg = (1..=2).contains(&layer_idx)
+                && std::env::var("BUTTER_LAGUNA_DEBUG").map(|v| v == "2").unwrap_or(false);
+            let h2 = ops::rms_norm(dev, &x1, &f.ffn_norm, cfg.eps)?;
+            let logits = ops::gemv(dev, &f.router, &h2)?; // [n_expert]
+            if moe_dbg {
+                dbg_stats(dev, &format!("moe{layer_idx}_x_in"), x);
+                dbg_stats(dev, &format!("moe{layer_idx}_x_attn_res"), &x1);
+                dbg_stats(dev, &format!("moe{layer_idx}_ffn_norm"), &h2);
+                dbg_stats(dev, &format!("moe{layer_idx}_router_logits"), &logits);
+            }
+            let (idx, wts) = ops::moe_router_device(
+                dev, &logits, &f.bias, cfg.n_expert, cfg.n_expert_used, cfg.routed_scaling,
+            )?;
+            if moe_dbg {
+                let mut wb = vec![0u8; cfg.n_expert_used * 4];
+                let _ = dev.synchronize();
+                let _ = dev.download(wts.buffer.as_ref(), &mut wb);
+                let w: Vec<f32> = wb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+                let mut ib = vec![0u8; cfg.n_expert_used * 4];
+                let _ = dev.download(idx.buffer.as_ref(), &mut ib);
+                let iv: Vec<u32> = ib.chunks_exact(4).map(|c| u32::from_le_bytes(c.try_into().unwrap())).collect();
+                eprintln!("    moe{layer_idx}_topk idx={iv:?} wts={w:?}");
+            }
+
+            // Fully device-side expert path: gather-GEMV the top_k gate and up
+            // stacks (the gather kernels index the expert slab by `idx` inside
+            // the kernel; a byte-offset Tensor view does NOT work here because
+            // dispatch binds buffer BASE pointers and drops Tensor.offset),
+            // SwiGLU elementwise on the [top_k*inter] pair, gather the down
+            // stack, then router-weighted sum into the accumulator.
+            //
+            // BUTTER_LAGUNA_FUSE=1 replaces the two gathers + host swiglu with
+            // one fused gate+up+SwiGLU gather kernel: same `h2` read once
+            // instead of twice, one launch instead of two gathers plus the
+            // elementwise pass. Default (unset) keeps the unfused path. Only
+            // available for RoutedGateUp::Split: `moe_gather_q4_swiglu` reads
+            // gate and up as two SEPARATE `[n_exp*inter, hid]` stacks with
+            // `row = e*inter + local` indexing, which the fused stack's
+            // per-expert-interleaved rows do not satisfy.
+            //
+            // RoutedGateUp::Fused (BUTTER_LAGUNA_MOEFUSE=1) instead reads the
+            // ONE fused stack with a single moe_gather_q4 call at
+            // inter=2*n_ff_exp (gathers both gate and up rows for the
+            // selected experts in one launch), then splits the result's
+            // columns into gate/up with extract_cols and runs the same
+            // host-orchestrated swiglu as the pre-BUTTER_LAGUNA_FUSE path
+            // above, see RoutedGateUp::Fused's doc for the tradeoff this
+            // implies (loses the fused-gather micro-win, about 1 tok/s).
+            let act = match &f.gateup {
+                RoutedGateUp::Fused(gu) => {
+                    let inter = 2 * cfg.n_ff_exp;
+                    let both = ops::moe_gather_q4(dev, &gu.qs, &gu.scales, &h2, &idx, cfg.n_expert_used, inter, cfg.hidden)?;
+                    let gates = ops::extract_cols(dev, &both, cfg.n_expert_used, inter, 0, cfg.n_ff_exp)?;
+                    let ups = ops::extract_cols(dev, &both, cfg.n_expert_used, inter, cfg.n_ff_exp, cfg.n_ff_exp)?;
+                    ops::swiglu(dev, &gates, &ups)?
+                }
+                RoutedGateUp::Split { gate, up } => {
+                    if std::env::var("BUTTER_LAGUNA_FUSE").map(|v| v == "1").unwrap_or(false) {
+                        ops::moe_gather_q4_swiglu(
+                            dev, &gate.qs, &gate.scales, &up.qs, &up.scales, &h2, &idx,
+                            cfg.n_expert_used, cfg.n_ff_exp, cfg.hidden,
+                        )?
+                    } else {
+                        let gates = ops::moe_gather_q4(dev, &gate.qs, &gate.scales, &h2, &idx, cfg.n_expert_used, cfg.n_ff_exp, cfg.hidden)?;
+                        let ups = ops::moe_gather_q4(dev, &up.qs, &up.scales, &h2, &idx, cfg.n_expert_used, cfg.n_ff_exp, cfg.hidden)?;
+                        ops::swiglu(dev, &gates, &ups)?
+                    }
+                }
+            };
+            let downs = ops::moe_gather_down(dev, &f.down.qs, &f.down.scales, &act, &idx, cfg.n_expert_used, cfg.n_ff_exp, cfg.hidden)?;
+            let acc = Tensor::new(dev.alloc_zeroed(cfg.hidden * 4)?, vec![cfg.hidden], DType::F32);
+            ops::moe_weighted_sum(dev, &downs, &wts, &acc, cfg.hidden, cfg.n_expert_used)?;
+
+            if moe_dbg { dbg_stats(dev, &format!("moe{layer_idx}_acc_routed"), &acc); }
+
+            // Always-on shared expert: same accumulator, unweighted (scale = 1,
+            // the model-wide cached `one` scalar, no gating/router weight).
+            //
+            // BUTTER_LAGUNA_MICRO=1 (fusion 2): the gate+up projections were
+            // repacked at load time as single-expert (n_expert=1) stacks, so
+            // one moe_gather_q4_swiglu call (top_k=1, the persistent
+            // zero_idx buffer selecting "expert 0") replaces the separate
+            // gate gemv + up gemv + host-orchestrated swiglu. `down` is
+            // unchanged either way (plain gemv_accum into the routed acc).
+            match &f.shared_expert {
+                SharedExpert::Fused(se) => {
+                    let sact = ops::moe_gather_q4_swiglu(
+                        dev, &se.gate.qs, &se.gate.scales, &se.up.qs, &se.up.scales,
+                        &h2, &se.zero_idx, 1, cfg.n_ff_shexp, cfg.hidden,
+                    )?;
+                    se.down.gemv_accum(dev, &sact, &acc, one)?;
+                }
+                SharedExpert::Split { gate, up, down } => {
+                    let sgate = gate.gemv(dev, &h2)?;
+                    let sup = up.gemv(dev, &h2)?;
+                    let sact = ops::swiglu(dev, &sgate, &sup)?;
+                    down.gemv_accum(dev, &sact, &acc, one)?;
+                }
+            }
+
+            if moe_dbg { dbg_stats(dev, &format!("moe{layer_idx}_acc_final"), &acc); }
+            ops::add(dev, &x1, &acc)?
+        }
+    };
+
+    Ok(x2)
+}
+
+/// One BATCHED prefill layer: `T` prompt-token rows processed in one forward
+/// instead of `T` sequential [`decode_layer`] calls. Same op sequence as
+/// `decode_layer` (RMSNorm → QKV(+gate) proj → per-head QK-RMSNorm → RoPE →
+/// KV-cache write → SDPA → per-head softplus gate → o_proj → residual →
+/// RMSNorm → FFN (dense or MoE) → residual), just every op runs on `[T, ...]`
+/// batched tensors through the batched-prefill ops (`gemm_q4_mpp`,
+/// `rope_yarn_partial_many`, `kv_append_many`, `sdpa_multi_tc_varlen`,
+/// `gate_softplus_mul_perhead_many`, the on-device MoE
+/// route+sort+grouped-GEMM+scatter pipeline) instead of the per-token gemv
+/// family. `x` is `[T, hidden]`; `start` is the ABSOLUTE sequence position of
+/// row 0 in this chunk (`positions[i] == start + i`, uploaded once by the
+/// caller and shared across every layer of this chunk).
+///
+/// Full-attention layers write straight into `kv`'s persistent growing cache
+/// (batched writes there are safe, nothing is evicted) and attend the full
+/// causal prefix (`seg_lo` all-zero). Sliding-window layers write into
+/// `scratch`'s LINEAR per-layer buffer instead (see [`LagunaPrefillScratch`]'s
+/// doc for why the ring can't be batch-written) and attend a per-row window
+/// `seg_lo[i] = max(0, (start+i) - (sliding_window-1))`.
+#[allow(clippy::too_many_arguments)]
+
+/// BUTTER_LAGUNA_PP_PROF=1: coarse stage profile of the prefill path. psync
+/// points synchronize, so the elapsed time of each sync attributes the queued
+/// work since the previous point to the CURRENT tag. Coarse but names the wall.
+static PROF: std::sync::Mutex<Option<std::collections::BTreeMap<String, std::time::Duration>>> =
+    std::sync::Mutex::new(None);
+fn prof_add(tag: &str, d: std::time::Duration) {
+    let mut g = PROF.lock().unwrap();
+    let m = g.get_or_insert_with(Default::default);
+    *m.entry(tag.to_string()).or_default() += d;
+}
+/// Print and reset the accumulated stage profile.
+pub fn prof_report() {
+    if let Some(m) = PROF.lock().unwrap().take() {
+        let total: f64 = m.values().map(|d| d.as_secs_f64()).sum();
+        eprintln!("  prefill stage profile (sync-attributed, total {total:.3}s):");
+        let mut rows: Vec<_> = m.into_iter().collect();
+        rows.sort_by(|a, b| b.1.cmp(&a.1));
+        for (tag, d) in rows {
+            eprintln!("    {tag:<20} {:>8.3}s  {:>5.1}%", d.as_secs_f64(), 100.0 * d.as_secs_f64() / total.max(1e-9));
+        }
+    }
+}
+
+fn prefill_layer(
+    dev: &dyn Device,
+    cfg: &LagunaConfig,
+    layer_idx: usize,
+    w: &LagunaLayer,
+    kv: &LagunaKvCache,
+    scratch: &LagunaPrefillScratch,
+    x: &Tensor, // [T, hidden] f32
+    t: usize,
+    start: usize,
+    positions: &Tensor, // [T] u32, positions[i] = start + i
+    // BUTTER_LAGUNA_MARLIN=1: a persistent all-zero `u32` buffer with at least
+    // `t` elements, shared by every Marlin call in this layer (and every
+    // other layer/chunk of the same `prefill()` call, see that fn's doc).
+    // `None` when the env is unset; `Some` implies "check each per-weight
+    // `Option<MarlinDense>` before using it" since an individual weight may
+    // still be `None` (dimension not 64-aligned).
+    marlin_zero_idx: Option<&Tensor>,
+) -> Result<Tensor> {
+    let hd = cfg.head_dim;
+    let hidden = cfg.hidden;
+    let n_head = w.attn.n_head;
+    let n_kv_heads = cfg.n_kv_heads;
+    let is_full = cfg.is_full_layer(layer_idx);
+    let rope = cfg.rope_for(layer_idx);
+    let heads_per_group = (n_head / n_kv_heads) as u32;
+    // BUTTER_LAGUNA_PREFILL_SYNC=1: synchronize after each stage so an async
+    // device fault surfaces AT the guilty op instead of a later module load.
+    let prof = std::env::var("BUTTER_LAGUNA_PP_PROF").is_ok();
+    let psync = |tag: &str| -> Result<()> {
+        if prof || std::env::var("BUTTER_LAGUNA_PREFILL_SYNC").is_ok() {
+            let t0 = std::time::Instant::now();
+            dev.synchronize().map_err(|e| {
+                Error::Msg(format!("prefill fault layer {layer_idx} stage {tag}: {e:?}"))
+            })?;
+            if prof {
+                prof_add(tag, t0.elapsed());
+            }
+        }
+        Ok(())
+    };
+
+    // ── attention ────────────────────────────────────────────────────
+    let h = ops::rms_norm(dev, x, &w.attn.attn_norm, cfg.eps)
+        .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} attn_norm: {e:?}")))?; // [T, hidden]
+    psync("attn_norm")?;
+
+    // g_proj consumes the SAME pre-attention normed input as q/k/v, exactly
+    // like decode_layer. BUTTER_LAGUNA_MICRO=1 (Fused): one gemm_q4_mpp on the
+    // concatenated [wq;wk;wv;wgate] weight, then COLUMN-range extraction
+    // (`ops::extract_cols`, NOT `ops::slice`, which only slices a flat 1-D
+    // buffer; the fused output here is `[T, n_rows_total]` row-major, so each
+    // sub-projection is a strided column range, not a contiguous byte range).
+    let (q, k, v, gate_raw) = if let (Some(zero_idx), Some(qkvm)) = (marlin_zero_idx, w.attn.qkv_marlin.as_ref()) {
+        // BUTTER_LAGUNA_MARLIN=1: tensor-core Marlin GEMM on the [wq;wk;wv]
+        // concat instead of gemm_q4_mpp's scattered-nibble-read path. Marlin
+        // consumes/emits f16. `extract_cols` is now f16-native, so extract
+        // straight out of `qkv_f16` instead of casting the whole [T,
+        // n_rows_qkv] result to f32 first, the extraction itself then moves
+        // half the bytes (f16 in/out vs f32 in/out). Every one of q/k/v
+        // still ends up cast to f32 individually right after, though:
+        // q/k RMSNorm reads `w.attn.q_norm`/`k_norm`, which are loaded
+        // f32-only (no f16 weight copy exists, see `AttnWeights`),
+        // `rope_yarn_partial_many`'s raw-CUDA kernel is hardcoded `float*`
+        // (f32 only, see its doc), and the persistent KV cache
+        // (`LagunaKvCache`/`LagunaPrefillScratch`) is allocated f32-only,
+        // so kv_append_many's destination forces v to f32 too. Net effect:
+        // same total cast bytes as before (one f32-only chain either way),
+        // but the extract itself is now cheaper, and it's one whole-buffer
+        // cast fewer to schedule.
+        // g_proj stays on gemm_q4_mpp, its 48/72 rows are never 64-aligned
+        //, reading the standalone `g_proj` Q4Dense built for exactly this
+        // (independent of whether `proj` above resolved to Split or Fused).
+        let h_f16 = ops::cast_f32_f16(dev, &h)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin qkv cast_f16: {e:?}")))?;
+        let qkv_f16 = qkvm.mat.call(dev, &h_f16, zero_idx, t)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin qkv gemm: {e:?}")))?;
+        psync("qkv_proj_marlin")?;
+        let n_rows_qkv = qkvm.n_q_rows + 2 * qkvm.n_kv_rows;
+        let q_f16 = ops::extract_cols(dev, &qkv_f16, t, n_rows_qkv, 0, qkvm.n_q_rows)?;
+        let k_f16 = ops::extract_cols(dev, &qkv_f16, t, n_rows_qkv, qkvm.n_q_rows, qkvm.n_kv_rows)?;
+        let v_f16 = ops::extract_cols(dev, &qkv_f16, t, n_rows_qkv, qkvm.n_q_rows + qkvm.n_kv_rows, qkvm.n_kv_rows)?;
+        let q = ops::cast_f16_f32(dev, &q_f16)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin q cast_f32: {e:?}")))?;
+        let k = ops::cast_f16_f32(dev, &k_f16)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin k cast_f32: {e:?}")))?;
+        let v = ops::cast_f16_f32(dev, &v_f16)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin v cast_f32: {e:?}")))?;
+        let g_proj = w.attn.g_proj.as_ref().expect(
+            "AttnWeights::qkv_marlin and ::g_proj are both built together, gated on the same \
+             BUTTER_LAGUNA_MARLIN load-time read, see load_gguf",
+        );
+        let gate_raw = ops::gemm_q4_mpp(dev, &h, &g_proj.qs, &g_proj.scales, t, g_proj.m, g_proj.k)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} g_proj gemm: {e:?}")))?;
+        (q, k, v, gate_raw)
+    } else {
+        match &w.attn.proj {
+            AttnProj::Fused { wqkvg, n_q_rows, n_kv_rows, n_g_rows } => {
+                let n_rows_total = n_q_rows + 2 * n_kv_rows + n_g_rows;
+                let qkvg = ops::gemm_q4_mpp(dev, &h, &wqkvg.qs, &wqkvg.scales, t, n_rows_total, hidden)?; // [T, n_rows_total]
+                let q = ops::extract_cols(dev, &qkvg, t, n_rows_total, 0, *n_q_rows)?;
+                let k = ops::extract_cols(dev, &qkvg, t, n_rows_total, *n_q_rows, *n_kv_rows)?;
+                let v = ops::extract_cols(dev, &qkvg, t, n_rows_total, n_q_rows + n_kv_rows, *n_kv_rows)?;
+                let gate_raw = ops::extract_cols(dev, &qkvg, t, n_rows_total, n_q_rows + 2 * n_kv_rows, *n_g_rows)?;
+                (q, k, v, gate_raw)
+            }
+            AttnProj::Split { wq, wk, wv, wgate } => {
+                let q = ops::gemm_q4_mpp(dev, &h, &wq.qs, &wq.scales, t, wq.m, wq.k)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} q_proj gemm: {e:?}")))?;
+                psync("q_proj")?;
+                let k = ops::gemm_q4_mpp(dev, &h, &wk.qs, &wk.scales, t, wk.m, wk.k)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} k_proj gemm: {e:?}")))?;
+                let v = ops::gemm_q4_mpp(dev, &h, &wv.qs, &wv.scales, t, wv.m, wv.k)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} v_proj gemm: {e:?}")))?;
+                let gate_raw = ops::gemm_q4_mpp(dev, &h, &wgate.qs, &wgate.scales, t, wgate.m, wgate.k)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} g_proj gemm: {e:?}")))?; // [T, n_head]
+                (q, k, v, gate_raw)
+            }
+        }
+    };
+
+    // Per-head QK RMSNorm (rms_norm normalizes the last dim, so a
+    // [T*heads, head_dim] view is correct per (token,head)), then batched
+    // RoPE, YaRN on full layers, plain on sliding (`cfg.rope_for` already
+    // picked the right params for each, same as decode_layer).
+    psync("qkvg_proj")?;
+    let q2 = ops::rms_norm(dev, &q.reshaped(vec![t * n_head, hd]), &w.attn.q_norm, cfg.eps)
+        .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} q_norm: {e:?}")))?;
+    let k2 = ops::rms_norm(dev, &k.reshaped(vec![t * n_kv_heads, hd]), &w.attn.k_norm, cfg.eps)
+        .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} k_norm: {e:?}")))?;
+    psync("qk_norm")?;
+    let q3 = ops::rope_yarn_partial_many(
+        dev, &q2.reshaped(vec![t, n_head, hd]), positions, n_head,
+        rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor, rope.attn_factor,
+        rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+    ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} q_rope: {e:?}")))?;
+    psync("q_rope")?;
+    let k3 = ops::rope_yarn_partial_many(
+        dev, &k2.reshaped(vec![t, n_kv_heads, hd]), positions, n_kv_heads,
+        rope.n_rot, rope.theta, rope.freq_scale, rope.ext_factor, rope.attn_factor,
+        rope.beta_fast, rope.beta_slow, rope.n_orig_ctx,
+    ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} k_rope: {e:?}")))?;
+    psync("k_rope")?;
+    let v3 = v.reshaped(vec![t, n_kv_heads, hd]);
+
+    // KV cache write + attention. Full layers grow the persistent cache
+    // directly (safe: nothing is ever evicted). Sliding layers write the
+    // LINEAR prefill scratch instead, see `LagunaPrefillScratch`'s doc for
+    // why the decode ring cannot be batch-written mid-prompt.
+    let attn = if is_full {
+        let slot = &kv.slots[layer_idx];
+        ops::kv_append_many(dev, &k3, positions, &slot.k, n_kv_heads, hd, slot.cap)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} kv_append k: {e:?}")))?;
+        ops::kv_append_many(dev, &v3, positions, &slot.v, n_kv_heads, hd, slot.cap)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} kv_append v: {e:?}")))?;
+        psync("kv_write")?;
+        // Every query attends the full causal prefix: seg_lo = 0 for all rows.
+        let seg_lo = Tensor::new(dev.alloc_zeroed(t * 4)?, vec![t], DType::U32);
+        // Full-attention layer: no sliding window, so the block skip stays
+        // off (win=0), every KV block is reachable from every query row.
+        ops::sdpa_multi_tc_varlen(
+            dev, &q3, &slot.k, &slot.v, &seg_lo, 0, hd, n_head as u32,
+            start as u32, t as u32, slot.cap as u32, heads_per_group, true, cfg.attn_scale(), 0,
+        ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} sdpa_varlen_1: {e:?}")))?
+    } else {
+        let slot = scratch.slots[layer_idx].as_ref().ok_or_else(|| {
+            Error::Msg(format!("prefill_layer: layer {layer_idx} is sliding-window but scratch has no slot for it"))
+        })?;
+        ops::kv_append_many(dev, &k3, positions, &slot.k, n_kv_heads, hd, slot.cap)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} kv_append k: {e:?}")))?;
+        ops::kv_append_many(dev, &v3, positions, &slot.v, n_kv_heads, hd, slot.cap)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} kv_append v: {e:?}")))?;
+        psync("kv_write")?;
+        // Per-row sliding window: query at absolute position p attends
+        // [max(0, p - (window-1)), p]. All values are >= 0, so a u32 tensor
+        // (the convention every other seg_lo caller in this codebase uses)
+        // carries them losslessly even though the raw-CUDA kernel reads them
+        // as `int`.
+        let window = cfg.sliding_window as i64;
+        let seg_lo_h: Vec<u32> = (0..t)
+            .map(|i| ((start + i) as i64 - (window - 1)).max(0) as u32)
+            .collect();
+        let seg_lo = Tensor::new(dev.upload(&u32s_to_bytes(&seg_lo_h))?, vec![t], DType::U32);
+        // Sliding-window layer: `seg_lo` above was built with exactly the
+        // `seg_lo[i] = max(0, (start+i) - (window-1))` convention the
+        // window-mode block skip expects, so pass the same window width as
+        // `win`, it prunes the KV blocks each chunk of query rows can't
+        // reach, on top of the (already-applied) per-element mask.
+        ops::sdpa_multi_tc_varlen(
+            dev, &q3, &slot.k, &slot.v, &seg_lo, 0, hd, n_head as u32,
+            start as u32, t as u32, slot.cap as u32, heads_per_group, true, cfg.attn_scale(),
+            cfg.sliding_window as u32,
+        ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} sdpa_varlen_0: {e:?}")))?
+    }; // [T, n_head, hd]
+
+    // Per-head softplus gate, o_proj, residual.
+    psync("sdpa")?;
+    let gated = ops::gate_softplus_mul_perhead_many(dev, &attn, &gate_raw, n_head)
+        .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} gate_many: {e:?}")))?; // [T, n_head, hd]
+    psync("gate")?;
+    let gated_flat = gated.reshaped(vec![t, n_head * hd]);
+    let o = if let (Some(zero_idx), Some(wo_m)) = (marlin_zero_idx, w.attn.wo_marlin.as_ref()) {
+        let gated_f16 = ops::cast_f32_f16(dev, &gated_flat)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin o_proj cast_f16: {e:?}")))?;
+        let o_f16 = wo_m.call(dev, &gated_f16, zero_idx, t)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin o_proj gemm: {e:?}")))?;
+        ops::cast_f16_f32(dev, &o_f16)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} marlin o_proj cast_f32: {e:?}")))?
+    } else {
+        ops::gemm_q4_mpp(dev, &gated_flat, &w.attn.wo.qs, &w.attn.wo.scales, t, w.attn.wo.m, w.attn.wo.k)
+            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} o_gemm: {e:?}")))? // [T, hidden]
+    };
+    let x1 = ops::add(dev, &x.reshaped(vec![t * hidden]), &o.reshaped(vec![t * hidden]))?.reshaped(vec![t, hidden]);
+    psync("o_proj_res")?;
+
+    // ── FFN ──────────────────────────────────────────────────────────
+    let x2 = match &w.ffn {
+        LayerFfn::Dense(f) => {
+            let h2 = ops::rms_norm(dev, &x1, &f.ffn_norm, cfg.eps)?; // [T, hidden]
+            let down = if let (Some(zero_idx), Some(gum), Some(down_m)) =
+                (marlin_zero_idx, f.gateup_marlin.as_ref(), f.down_marlin.as_ref())
+            {
+                // BUTTER_LAGUNA_MARLIN=1: [gate;up] concat as one Marlin GEMM,
+                // split + SwiGLU straight in f16 (extract_cols is f16-native
+                // and mt_swiglu is registered for f16 in wh-iron-std), then
+                // a second Marlin GEMM for down, `down_m.call` wants f16
+                // input directly. Unlike the attention QKV split above, this
+                // chain has no f32-only consumer in the middle (no RMSNorm
+                // weight, no rope, no f32 KV cache write), so the old
+                // cast_f16_f32(gateup)/cast_f32_f16(act) pair is pure
+                // overhead and is removed entirely; only `h2`'s initial
+                // f32->f16 cast (Marlin's input requirement) and the final
+                // f16->f32 cast (the f32 residual stream) remain.
+                let h2_f16 = ops::cast_f32_f16(dev, &h2)?;
+                let gateup_f16 = gum.mat.call(dev, &h2_f16, zero_idx, t)?; // [T, 2*n_ff] f16
+                let gate = ops::extract_cols(dev, &gateup_f16, t, 2 * gum.n_ff, 0, gum.n_ff)?;
+                let up = ops::extract_cols(dev, &gateup_f16, t, 2 * gum.n_ff, gum.n_ff, gum.n_ff)?;
+                let act_f16 = ops::swiglu(dev, &gate, &up)?; // [T, n_ff] f16
+                let down_f16 = down_m.call(dev, &act_f16, zero_idx, t)?; // [T, hidden] f16
+                ops::cast_f16_f32(dev, &down_f16)?
+            } else {
+                let gate = ops::gemm_q4_mpp(dev, &h2, &f.gate.qs, &f.gate.scales, t, f.gate.m, f.gate.k)?;
+                let up = ops::gemm_q4_mpp(dev, &h2, &f.up.qs, &f.up.scales, t, f.up.m, f.up.k)?;
+                let act = ops::swiglu(dev, &gate, &up)?;
+                ops::gemm_q4_mpp(dev, &act, &f.down.qs, &f.down.scales, t, f.down.m, f.down.k)?
+            };
+            ops::add(dev, &x1.reshaped(vec![t * hidden]), &down.reshaped(vec![t * hidden]))?.reshaped(vec![t, hidden])
+        }
+        LayerFfn::Moe(f) => {
+            let h2 = ops::rms_norm(dev, &x1, &f.ffn_norm, cfg.eps)?; // [T, hidden]
+
+            // Batched sigmoid router + on-device counting-sort: the SAME
+            // sigmoid+bias+top_k math as decode's `moe_router_device`, over
+            // all T rows in one pass. `logits` is the small dense f32 router
+            // GEMM (router stays f32, precision-sensitive); route+sort is
+            // fully on device (no host sync, mirrors the NemotronH
+            // NEMOTRON_DEVSORT/devdesc prefill pattern).
+            let logits = ops::matmul(dev, &f.router, &h2)
+                .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} router_matmul: {e:?}")))?; // [T, n_expert]
+            psync("moe_router_logits")?;
+            let (sorted_tok, sorted_wt, offsets) = ops::moe_route_sort_device(
+                dev, &logits, &f.bias, cfg.n_expert, cfg.n_expert_used, cfg.routed_scaling,
+            )?;
+            let mt = t * cfg.n_expert_used;
+            if std::env::var("BUTTER_LAGUNA_PREFILL_SYNC").is_ok() {
+                let _ = dev.synchronize();
+                let mut stb = vec![0u8; mt * 4];
+                let _ = dev.download(sorted_tok.buffer.as_ref(), &mut stb);
+                let st: Vec<u32> = stb.chunks_exact(4).map(|c| u32::from_le_bytes(c.try_into().unwrap())).collect();
+                let mut ofb = vec![0u8; (cfg.n_expert + 1) * 4];
+                let _ = dev.download(offsets.buffer.as_ref(), &mut ofb);
+                let of: Vec<u32> = ofb.chunks_exact(4).map(|c| u32::from_le_bytes(c.try_into().unwrap())).collect();
+                eprintln!("    prefill moe layer {layer_idx}: sorted_tok={st:?} offsets_last={} (expect {mt})", of[cfg.n_expert]);
+            }
+
+            // Gather the sorted-order activation (f16, the grouped-MMA
+            // input dtype) once, then TWO grouped Q4 GEMMs (gate stack, up
+            // stack) over the sorted rows, elementwise SwiGLU, one grouped
+            // Q4 GEMM down. `moe_q4_grouped_mma_dev` builds its tile
+            // descriptors ON DEVICE from `offsets`, no host g_starts/
+            // expert_ids vectors, no host sync.
+            //
+            // NOTE ON DTYPE (RoutedGateUp::Fused branch below): the down
+            // grouped-GEMM's raw CUDA kernel reads its activation input as
+            // `const __half*` unconditionally (see `MOE_Q4_GROUPED_MMA_SRC`
+            // in wh-butter-ops), f16 in, no f32 path exists. Combined with
+            // `moe_q4_grouped_mma_dev` always emitting f16 and `extract_cols`
+            // now being f16-native, the gate/up split + SwiGLU chain has NO
+            // f32-only consumer anywhere in it, so it runs start to finish in
+            // f16 with zero casts (the old cast_f16_f32/cast_f32_f16 pair
+            // existed only because `extract_cols`/`ops::swiglu` used to
+            // require f32 input; both now take f16 directly).
+            psync("moe_route_sort")?;
+            let h2_f16 = ops::cast_f32_f16(dev, &h2)
+                .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} cast_f16: {e:?}")))?; // [T, hidden]
+            let a = ops::gather(dev, &h2_f16, &sorted_tok)
+                .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} moe_gather_rows: {e:?}")))?; // [mt, hidden] f16, sorted-token order
+
+            // BUTTER_LAGUNA_W4A8=1: route the routed gate/up/down grouped
+            // GEMMs through the W4A8 grouped-MoE GEMM (fp8 e4m3 activations
+            // x mxfp4 weights, cutlass mxf8f6f4) instead of
+            // `moe_q4_grouped_mma_dev`'s W4A16 path, see `MoeFfnW4a8`'s doc
+            // for the weight side. `f.w4a8` was decided once at load time
+            // (`laguna_w4a8_enabled`'s doc), so this just matches the
+            // `Option` rather than re-reading the env.
+            //
+            // Unlike `moe_q4_grouped_mma_dev` (device-built tile
+            // descriptors, zero host sync: see the NOTE ON DTYPE below),
+            // `ops::w4a8_sfa_offsets`/`w4a8_actquant_grouped`/
+            // `moe_grouped_gemm_w4a8` all take HOST `g_starts`/`expert_ids`
+            // (mirroring their wh-butter-modeltests NEMOTRON_W4A8_MOE call site
+            // exactly, no device-side descriptor-building variant of this
+            // GEMM exists in wh-butter-ops today). So this branch pays ONE small
+            // device->host download per layer (`offsets` is `n_expert+1`
+            // u32s, 1028 bytes for Laguna-S-2.1's 256 experts) to turn the
+            // on-device routing offsets into host group-starts, the one
+            // host sync this batched-prefill path otherwise avoids
+            // entirely (decode-era work eliminated exactly this kind of
+            // per-layer host round-trip for the W4A16 grouped path; W4A8
+            // reintroduces a tiny one because `ops::w4a8_*` has no
+            // on-device-descriptor sibling yet). `Device::download` blocks
+            // until prior queued work completes, so no separate
+            // `dev.synchronize()` call is needed first (same convention as
+            // the NEMOTRON_W4A8_MOE call site).
+            let down_out = if let Some(w4a8) = f.w4a8.as_ref() {
+                let mut ofb = vec![0u8; (cfg.n_expert + 1) * 4];
+                dev.download(offsets.buffer.as_ref(), &mut ofb)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 offsets download: {e:?}")))?;
+                let offs: Vec<usize> = ofb.chunks_exact(4)
+                    .map(|c| u32::from_le_bytes(c.try_into().unwrap()) as usize).collect();
+                let mut g_starts: Vec<usize> = vec![0];
+                let mut expert_ids: Vec<usize> = Vec::new();
+                for e in 0..cfg.n_expert {
+                    if offs[e + 1] > offs[e] {
+                        expert_ids.push(e);
+                        g_starts.push(offs[e + 1]);
+                    }
+                }
+                let group_rows: Vec<i32> = g_starts.windows(2).map(|w| (w[1] - w[0]) as i32).collect();
+                psync("w4a8_offsets_dl")?;
+
+                // gate/up both read the same pre-FFN-norm activation, so
+                // they share `k = hidden` and therefore the same `kpad`,
+                // grab it off whichever stack this layer built
+                // (`RoutedGateUpW4a8::Fused`'s single stack, or `::Split`'s
+                // `gate` stack; both give the identical padded width).
+                let hid_pad = match &w4a8.gateup {
+                    RoutedGateUpW4a8::Fused(s) => s.kpad,
+                    RoutedGateUpW4a8::Split { gate, .. } => gate.kpad,
+                };
+                let a_pad = ops::pad_rows_f16(dev, &a, mt, cfg.hidden, hid_pad)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gateup pad: {e:?}")))?;
+                let (uoff, ubytes) = ops::w4a8_sfa_offsets(&g_starts, hid_pad);
+                let (a_q, a_sf) = ops::w4a8_actquant_grouped(dev, &a_pad, &group_rows, &uoff, ubytes, mt, hid_pad)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gateup actquant: {e:?}")))?;
+                psync("w4a8_gateup_actquant")?;
+
+                // Same fused-vs-split shape as the W4A16 branch below: ONE
+                // grouped GEMM to `2*n_ff_exp` for `Fused`, TWO separate
+                // grouped GEMMs (sharing the same quantized activation) for
+                // `Split`, then the identical SwiGLU either way.
+                let act = match &w4a8.gateup {
+                    RoutedGateUpW4a8::Fused(gu) => {
+                        let n_ff = cfg.n_ff_exp;
+                        let gateup_out = ops::moe_grouped_gemm_w4a8(
+                            dev, &a_q, &a_sf, &uoff, &gu.packed, &gu.sf, gu.sfb_exp_bytes,
+                            &g_starts, &expert_ids, 2 * n_ff, hid_pad,
+                        ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gateup gemm: {e:?}")))?; // [mt, 2*n_ff_exp] f16
+                        psync("w4a8_gateup_gemm")?;
+                        let gate_out = ops::extract_cols(dev, &gateup_out, mt, 2 * n_ff, 0, n_ff)?;
+                        let up_out = ops::extract_cols(dev, &gateup_out, mt, 2 * n_ff, n_ff, n_ff)?;
+                        ops::swiglu(dev, &gate_out, &up_out)
+                            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gateup swiglu: {e:?}")))? // [mt, n_ff_exp] f16
+                    }
+                    RoutedGateUpW4a8::Split { gate, up } => {
+                        let gate_out = ops::moe_grouped_gemm_w4a8(
+                            dev, &a_q, &a_sf, &uoff, &gate.packed, &gate.sf, gate.sfb_exp_bytes,
+                            &g_starts, &expert_ids, cfg.n_ff_exp, hid_pad,
+                        ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gate gemm: {e:?}")))?; // [mt, n_ff_exp] f16
+                        let up_out = ops::moe_grouped_gemm_w4a8(
+                            dev, &a_q, &a_sf, &uoff, &up.packed, &up.sf, up.sfb_exp_bytes,
+                            &g_starts, &expert_ids, cfg.n_ff_exp, hid_pad,
+                        ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 up gemm: {e:?}")))?; // [mt, n_ff_exp] f16
+                        psync("w4a8_gateup_gemm")?;
+                        ops::swiglu(dev, &gate_out, &up_out)
+                            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 gateup swiglu: {e:?}")))? // [mt, n_ff_exp] f16
+                    }
+                };
+
+                let inter_pad = w4a8.down.kpad;
+                let act_pad = ops::pad_rows_f16(dev, &act, mt, cfg.n_ff_exp, inter_pad)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 down pad: {e:?}")))?;
+                let (doff, dbytes) = ops::w4a8_sfa_offsets(&g_starts, inter_pad);
+                let (d_q, d_sf) = ops::w4a8_actquant_grouped(dev, &act_pad, &group_rows, &doff, dbytes, mt, inter_pad)
+                    .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 down actquant: {e:?}")))?;
+                psync("w4a8_down_actquant")?;
+                let down_out = ops::moe_grouped_gemm_w4a8(
+                    dev, &d_q, &d_sf, &doff, &w4a8.down.packed, &w4a8.down.sf, w4a8.down.sfb_exp_bytes,
+                    &g_starts, &expert_ids, cfg.hidden, inter_pad,
+                ).map_err(|e| Error::Msg(format!("prefill layer {layer_idx} w4a8 down gemm: {e:?}")))?; // [mt, hidden] f16
+                down_out
+            } else {
+                // BUTTER_LAGUNA_MOEFUSE=1 (RoutedGateUp::Fused): ONE grouped Q4 GEMM
+                // with n_out = 2*n_ff_exp over the fused [gate;up] stack instead
+                // of two separate grouped GEMMs, halving the grouped-GEMM call
+                // count (each call is itself two kernel launches, an on-device
+                // tile-descriptor build plus the GEMM, so this also halves the
+                // launch count). See the NOTE ON DTYPE below: the whole
+                // split+SwiGLU chain runs in f16 with no cast, mirroring the
+                // shared-expert Marlin [gate;up] split above.
+                let act = match &f.gateup {
+                    RoutedGateUp::Fused(gu) => {
+                        let n_ff = cfg.n_ff_exp;
+                        let gateup_out = ops::moe_q4_grouped_mma_dev(
+                            dev, &a, &gu.qs, &gu.scales, &offsets, cfg.n_expert, mt, 2 * n_ff, cfg.hidden,
+                        )
+                        .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} moe_gateup_gemm_fused: {e:?}")))?; // [mt, 2*n_ff_exp] f16
+                        psync("moe_gateup_gemm")?;
+                        // f16 straight through, see the NOTE ON DTYPE below.
+                        let gate_out = ops::extract_cols(dev, &gateup_out, mt, 2 * n_ff, 0, n_ff)?;
+                        let up_out = ops::extract_cols(dev, &gateup_out, mt, 2 * n_ff, n_ff, n_ff)?;
+                        ops::swiglu(dev, &gate_out, &up_out)
+                            .map_err(|e| Error::Msg(format!("prefill layer {layer_idx} moe_act_swiglu: {e:?}")))? // [mt, n_ff_exp] f16
+                    }
+                    RoutedGateUp::Split { gate, up } => {
+                        let gate_out = ops::moe_q4_grouped_mma_dev(
+                            dev, &a, &gate.qs, &gate.scales, &offsets, cfg.n_expert, mt, cfg.n_ff_exp, cfg.hidden,
+                        )?; // [mt, n_ff_exp] f16
+                        let up_out = ops::moe_q4_grouped_mma_dev(
+                            dev, &a, &up.qs, &up.scales, &offsets, cfg.n_expert, mt, cfg.n_ff_exp, cfg.hidden,
+                        )?; // [mt, n_ff_exp] f16
+                        psync("moe_gateup_gemm")?;
+                        ops::swiglu(dev, &gate_out, &up_out)? // [mt, n_ff_exp] f16
+                    }
+                };
+                let down_out = ops::moe_q4_grouped_mma_dev(
+                    dev, &act, &f.down.qs, &f.down.scales, &offsets, cfg.n_expert, mt, cfg.hidden, cfg.n_ff_exp,
+                )?; // [mt, hidden] f16
+                down_out
+            };
+
+            // Weighted scatter-add back to token order: acc[t,h] = sum over
+            // this token's top_k sorted rows of down_out[r,h] * sorted_wt[r]
+            // (router weight already includes routed_scaling, no extra
+            // `unscale` factor needed here, unlike Nemotron's ReLU²/256 trick).
+            let acc = Tensor::new(dev.alloc_zeroed(t * cfg.hidden * 4)?, vec![t, cfg.hidden], DType::F32);
+            psync("moe_down_gemm")?;
+            ops::moe_scatter_add_det_dev(
+                dev, &down_out, &sorted_tok, &sorted_wt, &acc, t, mt, cfg.hidden, 1.0, true,
+            )?;
+
+            // Always-on shared expert: same batched gate→up→SwiGLU→down as a
+            // routed expert, just over ALL T rows unweighted (decode's
+            // unweighted `gemv_accum(scale=1)` equivalent). Handles both
+            // weight layouts (BUTTER_LAGUNA_MICRO=1's single-expert
+            // Q4ExpertStack fusion, or the plain Q4Dense split) uniformly by
+            // pulling a (qs, scales, m, k) view for gate/up either way,
+            // `expert_view(0)`'s byte offset is always zero for a 1-expert
+            // stack, so it's safe to feed into `gemm_q4_mpp` even though that
+            // op's `Binding::Buffer` ignores a non-zero `Tensor.offset`.
+            psync("moe_scatter")?;
+            let sdown_out = if let (Some(zero_idx), Some(sm)) = (marlin_zero_idx, f.shared_marlin.as_ref()) {
+                // BUTTER_LAGUNA_MARLIN=1: same [gate;up]-concat-then-split
+                // Marlin pattern as the dense-layer FFN above. `h2_f16` was
+                // already computed for the routed-expert gather just above,
+                // reuse it instead of re-casting. Split + SwiGLU stay in f16
+                // end to end (extract_cols f16-native, mt_swiglu f16-native,
+                // `sm.down.call` wants f16 in), the old cast_f16_f32(gateup)
+                // / cast_f32_f16(sact) pair is removed; only the final
+                // f16->f32 cast remains, needed because `sdown_out` is added
+                // into the f32 MoE accumulator (`acc2`) below.
+                let gateup_f16 = sm.gateup.mat.call(dev, &h2_f16, zero_idx, t)?; // [T, 2*n_ff_shexp] f16
+                let sgate = ops::extract_cols(dev, &gateup_f16, t, 2 * sm.gateup.n_ff, 0, sm.gateup.n_ff)?;
+                let sup = ops::extract_cols(dev, &gateup_f16, t, 2 * sm.gateup.n_ff, sm.gateup.n_ff, sm.gateup.n_ff)?;
+                let sact_f16 = ops::swiglu(dev, &sgate, &sup)?; // [T, n_ff_shexp] f16
+                let sdown_f16 = sm.down.call(dev, &sact_f16, zero_idx, t)?; // [T, hidden] f16
+                ops::cast_f16_f32(dev, &sdown_f16)? // [T, hidden]
+            } else {
+                let (sg_qs, sg_sc, sg_m, sg_k, su_qs, su_sc, su_m, su_k, sdown) = match &f.shared_expert {
+                    SharedExpert::Split { gate, up, down } => (
+                        gate.qs.clone(), gate.scales.clone(), gate.m, gate.k,
+                        up.qs.clone(), up.scales.clone(), up.m, up.k,
+                        down,
+                    ),
+                    SharedExpert::Fused(se) => {
+                        let (gqs, gsc) = se.gate.expert_view(0);
+                        let (uqs, usc) = se.up.expert_view(0);
+                        (gqs, gsc, se.gate.m, se.gate.k, uqs, usc, se.up.m, se.up.k, &se.down)
+                    }
+                };
+                let sgate = ops::gemm_q4_mpp(dev, &h2, &sg_qs, &sg_sc, t, sg_m, sg_k)?;
+                let sup = ops::gemm_q4_mpp(dev, &h2, &su_qs, &su_sc, t, su_m, su_k)?;
+                let sact = ops::swiglu(dev, &sgate, &sup)?;
+                ops::gemm_q4_mpp(dev, &sact, &sdown.qs, &sdown.scales, t, sdown.m, sdown.k)? // [T, hidden]
+            };
+            let acc2 = ops::add(dev, &acc.reshaped(vec![t * cfg.hidden]), &sdown_out.reshaped(vec![t * cfg.hidden]))?;
+
+            ops::add(dev, &x1.reshaped(vec![t * hidden]), &acc2)?.reshaped(vec![t, hidden])
+        }
+    };
+
+    Ok(x2)
+}
+
+/// Batched (multi-token) **prefill**: process the whole `tokens` prompt
+/// `chunk` tokens at a time (default recommendation: 1024), writing every
+/// token's K/V into `kv`/`scratch`, and return the LAST token's next-token
+/// logits, the prefill counterpart of [`decode_step`]'s per-token forward.
+/// Each chunk's `T` rows run through every layer as batched GEMMs
+/// ([`ops::gemm_q4_mpp`], the grouped-MMA MoE pipeline, `sdpa_multi_tc_varlen`)
+/// instead of `T` sequential [`decode_step`] calls.
+///
+/// `scratch` must have been sized (via [`LagunaPrefillScratch::new`]) for AT
+/// LEAST `tokens.len()`. `kv` must have been sized (via
+/// [`LagunaKvCache::new`]) for at least `tokens.len()` plus however many
+/// decode tokens the caller intends to generate afterward, same contract as
+/// feeding the same prompt through `decode_step` one token at a time.
+///
+/// After the LAST chunk, this compacts each sliding-window layer's linear
+/// prefill scratch (the last `min(sliding_window, tokens.len())` positions)
+/// into `kv`'s decode ring, so `decode_step` / `LagunaDecodeCtx` can continue
+/// from `pos = tokens.len()` and produce the IDENTICAL greedy continuation a
+/// pure per-token `decode_step` loop over the same prompt would.
+///
+/// `chunk` only bounds how many prompt tokens are embedded/projected/attended
+/// in one forward pass (memory for the batched intermediates scales with
+/// it); correctness is independent of the choice.
+pub fn prefill(
+    dev: &dyn Device,
+    model: &LagunaModel,
+    kv: &LagunaKvCache,
+    scratch: &LagunaPrefillScratch,
+    tokens: &[u32],
+    chunk: usize,
+) -> Result<Tensor> {
+    let cfg = &model.cfg;
+    let total = tokens.len();
+    if total == 0 {
+        return Err(Error::Msg("prefill: empty prompt".into()));
+    }
+    if total > scratch.prompt_cap {
+        return Err(Error::Msg(format!(
+            "prefill: prompt len {total} exceeds scratch prompt_cap {}", scratch.prompt_cap
+        )));
+    }
+    let chunk = chunk.max(1);
+
+    // BUTTER_LAGUNA_MARLIN=1: ONE persistent all-zero `u32` indices buffer,
+    // sized to `chunk` (every chunk's row count `t = chunk.min(total-start)`
+    // is `<= chunk`), reused by every `MarlinDense::call` in every layer of
+    // every chunk this `prefill()` invocation processes. The Marlin GEMM's
+    // grouped-MoE ABI wants a per-row expert id; a dense `[n_out, k_in]`
+    // matrix is fed as a single "expert 0", so every row's id is 0: see
+    // `MarlinDense`'s doc. Built even if a given layer's per-weight
+    // `Option<MarlinDense>` ends up `None` (dimension not 64-aligned); the
+    // cost is one small zeroed alloc, and `prefill_layer` only reads through
+    // this when a `Some(MarlinDense)` is also present.
+    let marlin_zero_idx = if laguna_marlin_enabled() {
+        Some(Tensor::new(dev.alloc_zeroed(chunk * 4)?, vec![chunk], DType::U32))
+    } else {
+        None
+    };
+
+    let mut x_last: Option<Tensor> = None;
+    let mut t_last = 0usize;
+    let mut start = 0usize;
+    while start < total {
+        let t = chunk.min(total - start);
+        let chunk_tokens = &tokens[start..start + t];
+
+        let ids = Tensor::new(dev.upload(&u32s_to_bytes(chunk_tokens))?, vec![t], DType::U32);
+        let mut x = ops::gather(dev, &model.tok_embd, &ids)
+            .map_err(|e| Error::Msg(format!("prefill embed gather: {e:?}")))?
+            .reshaped(vec![t, cfg.hidden]);
+
+        let pos_vec: Vec<u32> = (0..t).map(|i| (start + i) as u32).collect();
+        let positions = Tensor::new(dev.upload(&u32s_to_bytes(&pos_vec))?, vec![t], DType::U32);
+
+        if std::env::var("BUTTER_LAGUNA_PREFILL_SYNC").is_ok() {
+            dev.synchronize().map_err(|e| Error::Msg(format!("prefill fault pre-layers (embed/positions): {e:?}")))?;
+        }
+        for (i, layer) in model.layers.iter().enumerate() {
+            x = prefill_layer(dev, cfg, i, layer, kv, scratch, &x, t, start, &positions, marlin_zero_idx.as_ref())?
+                .into();
+            if std::env::var("BUTTER_LAGUNA_PREFILL_SYNC").is_ok() {
+                dev.synchronize().map_err(|e| Error::Msg(format!("prefill fault after layer {i}: {e:?}")))?;
+            }
+        }
+
+        x_last = Some(x);
+        t_last = t;
+        start += t;
+    }
+    let x = x_last.expect("prefill: total > 0 was checked above, so at least one chunk always runs");
+
+    // Final norm + q8 lm_head on the LAST token of the LAST chunk only (the
+    // next-token logits), cheaper than decode's reference-engine-style "norm all rows
+    // then slice" since RMSNorm is per-row-independent anyway.
+    let last_row = ops::slice(dev, &x.reshaped(vec![t_last * cfg.hidden]), (t_last - 1) * cfg.hidden, cfg.hidden)?;
+    let xn = ops::rms_norm(dev, &last_row, &model.out_norm, cfg.eps)?;
+    let logits = ops::gemv_q8(
+        dev, &model.lm_head.qs, &model.lm_head.scales, &xn, model.lm_head.m, model.lm_head.k, model.lm_head.m,
+    )?;
+
+    // Compact each sliding-window layer's linear prefill scratch into the
+    // persistent decode ring, one-time cost, so decode can continue from
+    // `pos = total` exactly as it does today.
+    let copy_len = total.min(cfg.sliding_window);
+    let copy_start = total - copy_len;
+    for i in 0..cfg.n_layers {
+        if cfg.is_full_layer(i) {
+            continue;
+        }
+        let sslot = scratch.slots[i].as_ref().expect("sliding layer always has a scratch slot");
+        let dslot = &kv.slots[i];
+        ops::kv_compact_ring(dev, &sslot.k, &dslot.k, cfg.n_kv_heads, sslot.cap, cfg.head_dim, dslot.cap, copy_start, copy_len)?;
+        ops::kv_compact_ring(dev, &sslot.v, &dslot.v, cfg.n_kv_heads, sslot.cap, cfg.head_dim, dslot.cap, copy_start, copy_len)?;
+    }
+
+    Ok(logits)
+}
+
+/// Persistent per-step device scalars decode reads instead of uploading a
+/// fresh 1-element buffer every call. Allocated ONCE (in [`DecodeWorkspace::new`])
+/// and refreshed in place every step via [`DecodeWorkspace::update`], the
+/// refresh always happens BEFORE the (eager or captured-graph) decode work
+/// that reads them, and never inside a graph-capture region, so it's safe
+/// regardless of whether the caller is warming up, capturing, or replaying.
+///
+/// `full_pos`/`sliding_pos` are decode_layer's existing posbuf inputs (KV
+/// cache write index, full layers grow, sliding layers ring); `n_kv_full`/
+/// `n_kv_sliding` are the NEW graph-mode buffers `sdpa_decode_nbuf` reads
+/// (`pos+1` / `min(pos+1, sliding_window)`). Eager (non-graph) decode only
+/// ever reads `token_id`/`full_pos`/`sliding_pos`; `n_kv_full`/`n_kv_sliding`
+/// are only consulted by `decode_layer` when `BUTTER_LAGUNA_GRAPH=1`.
+pub struct DecodeWorkspace {
+    token_id: Tensor,
+    full_pos: Tensor,
+    sliding_pos: Tensor,
+    n_kv_full: Tensor,
+    n_kv_sliding: Tensor,
+}
+
+impl DecodeWorkspace {
+    pub fn new(dev: &dyn Device) -> Result<Self> {
+        let zero = || -> Result<Tensor> {
+            Ok(Tensor::new(dev.upload(&0u32.to_le_bytes())?, vec![1], DType::U32))
+        };
+        Ok(DecodeWorkspace {
+            token_id: zero()?,
+            full_pos: zero()?,
+            sliding_pos: zero()?,
+            n_kv_full: zero()?,
+            n_kv_sliding: zero()?,
+        })
+    }
+
+    /// Refresh every workspace buffer for the token at `pos`. Five tiny
+    /// (single-`u32`) raw-CUDA writes, in place: no reallocation, so the
+    /// buffers' device pointers (what a captured graph binds) never move.
+    /// MUST be called outside any graph-capture region.
+    pub fn update(&self, dev: &dyn Device, cfg: &LagunaConfig, token_id: u32, pos: u32) -> Result<()> {
+        // TODO(perf): 5 tiny kernel launches per step (one per scalar). Still
+        // far cheaper than 5 cuMemAlloc+H2D uploads, but a single fused
+        // "write 5 u32s" raw kernel (one launch, 5 scalar args) would shave a
+        // bit more host-side launch overhead off the outside-the-graph cost
+        // once this becomes the next bottleneck.
+        ops::write_u32(dev, &self.token_id, token_id)?;
+        ops::write_u32(dev, &self.full_pos, pos)?;
+        let sliding_pos = pos % cfg.sliding_window as u32;
+        ops::write_u32(dev, &self.sliding_pos, sliding_pos)?;
+        ops::write_u32(dev, &self.n_kv_full, pos + 1)?;
+        let n_kv_sliding = (pos + 1).min(cfg.sliding_window as u32);
+        ops::write_u32(dev, &self.n_kv_sliding, n_kv_sliding)?;
+        Ok(())
+    }
+}
+
+/// Single-token decode: embed → every layer (self-attention + cache append +
+/// FFN) → final RMSNorm → lm_head → next-token logits `[vocab]`. `pos` is
+/// this token's absolute sequence position (0 for the first token); the
+/// caller advances it and re-calls per generated token, reusing `kv` across
+/// calls so the cache persists.
+///
+/// Reads `token_id`/RoPE-position/KV-length from `ws` rather than uploading
+/// fresh 1-element buffers every call (the former per-step host upload this
+/// function used to do), the caller must have already called
+/// `ws.update(dev, &model.cfg, token_id, pos)` for this step. This is both a
+/// standalone eager-mode win (no per-step `cuMemAlloc`+H2D for those
+/// scalars) and what makes the whole step CUDA-graph-capturable: a graph
+/// replays the exact recorded kernel launches, so every per-token input has
+/// to sit behind a buffer POINTER that stays fixed across replays.
+///
+/// This is the path used both by plain eager decode (no env set) and by
+/// [`LagunaDecodeCtx`]'s warmup steps and one-time capture step; graph
+/// REPLAY (after capture) does not call this function at all, it re-runs
+/// the already-recorded launches via `Device::graph_launch`.
+///
+/// `BUTTER_LAGUNA_GRAPH=1` switches `decode_layer`'s RoPE/SDPA to the
+/// buffer-driven (capture-safe) kernels; without it, behavior is byte-for-byte
+/// identical to before this refactor (same kernels, same scalar arguments,
+/// just reading `token_id`/`pos` off a persistent buffer instead of a fresh
+/// one, the values are the same either way).
+pub fn decode_step(
+    dev: &dyn Device,
+    model: &LagunaModel,
+    kv: &LagunaKvCache,
+    ws: &DecodeWorkspace,
+    pos: u32,
+) -> Result<Tensor> {
+    let cfg = &model.cfg;
+    let mut x = ops::gather(dev, &model.tok_embd, &ws.token_id)?.reshaped(vec![cfg.hidden]);
+
+    let graph_mode = std::env::var("BUTTER_LAGUNA_GRAPH").is_ok();
+    let graph_bufs = if graph_mode {
+        Some(GraphBufs { n_kv_full: &ws.n_kv_full, n_kv_sliding: &ws.n_kv_sliding })
+    } else {
+        None
+    };
+    // Read once per decode_step, passed down as a bool: all three micro-
+    // fusions (QKVG concat gemv, shared-expert fused swiglu gather, o_proj
+    // residual accum) are gated on this single env var. Fixed for the
+    // process lifetime (same requirement CUDA-graph capture already has,
+    // see GraphBufs's doc), so this per-step read always agrees with the
+    // load-time read that picked AttnProj/SharedExpert's Fused vs Split.
+    let micro = laguna_micro_enabled();
+
+    // BUTTER_LAGUNA_DEBUG=1: per-layer residual stats (L2 norm, NaN count) to
+    // bisect a divergence to the first bad layer. Hot path unaffected when
+    // the env is unset. Downloads mid-step, so it is INCOMPATIBLE with graph
+    // capture/replay, `LagunaDecodeCtx::step` refuses to run at all when
+    // this is set (see its doc); a bare `decode_step` call with both env
+    // vars set would otherwise try to `dev.download` from inside a capture
+    // region, which is forbidden by the CUDA driver.
+    let debug = std::env::var("BUTTER_LAGUNA_DEBUG").map(|v| v == "1").unwrap_or(false);
+    for (i, layer) in model.layers.iter().enumerate() {
+        x = decode_layer(dev, cfg, i, layer, &kv.slots[i], &x, pos, &ws.full_pos, &ws.sliding_pos, &model.one, graph_bufs.as_ref(), micro)?;
+        if debug {
+            let mut xb = vec![0u8; cfg.hidden * 4];
+            dev.synchronize()?;
+            dev.download(x.buffer.as_ref(), &mut xb)?;
+            let xf: Vec<f32> = xb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+            let n_nan = xf.iter().filter(|v| v.is_nan()).count();
+            let l2 = xf.iter().map(|v| (*v as f64) * (*v as f64)).sum::<f64>().sqrt();
+            let amax = xf.iter().fold(0f32, |a, &v| if v.abs() > a { v.abs() } else { a });
+            eprintln!("  layer {i:2} ({}{}): |x|2={l2:.3e} amax={amax:.3e} nan={n_nan} head={:?}",
+                if cfg.is_full_layer(i) { "full" } else { "swa " },
+                if cfg.is_dense_layer(i) { ", dense" } else { "" },
+                &xf[..4]);
+        }
+    }
+
+    let xn = ops::rms_norm(dev, &x, &model.out_norm, cfg.eps)?;
+    ops::gemv_q8(dev, &model.lm_head.qs, &model.lm_head.scales, &xn, model.lm_head.m, model.lm_head.k, model.lm_head.m)
+}
+
+/// CUDA-graph-capturing decode driver: collapses Laguna's ~1000+ per-token
+/// kernel launches (48 layers x ~22 ops) into ONE `cuGraphLaunch` replay per
+/// token, removing host-launch-gap overhead. Enabled by running with
+/// `BUTTER_LAGUNA_GRAPH=1` (also read by `decode_step`/`decode_layer` to pick
+/// the buffer-driven RoPE/SDPA kernels, see their docs).
+///
+/// Protocol (mirrors the NemotronH `NEMOTRON_GRAPH` precedent in
+/// `wh-butter-modeltests`): the first two `step()` calls run [`decode_step`]
+/// eagerly ("warm up N=2 full steps normally", the pool's bucket free-list
+/// reaches a deterministic steady state after the first call, since every
+/// step walks the identical fixed 48-layer op sequence regardless of
+/// `token_id`/`pos`, so the second call already allocates nothing new). The
+/// THIRD call captures: `Device::begin_capture` + one more `decode_step` +
+/// `Device::end_capture`. Capture only RECORDS the stream's kernel launches
+/// (it does not execute them), so that third call's `decode_step` result is
+/// not yet real data; `step()` immediately follows the capture with one
+/// `Device::graph_launch` to actually compute that token's logits. Every
+/// call after that just refreshes the workspace and replays the graph.
+///
+/// The logits `Tensor` returned from the capture step is kept alive for the
+/// lifetime of this `LagunaDecodeCtx` (never dropped) so its device buffer's
+/// pointer can never be recycled by the allocator's pool, the same buffer
+/// is what every subsequent `graph_launch` writes into, and what `step()`
+/// returns (a cheap `Arc` clone) every time thereafter.
+///
+/// `BUTTER_LAGUNA_DEBUG` downloads mid-step and is incompatible with capture;
+/// `step()` returns an error immediately if it's set, rather than silently
+/// producing a broken capture.
+pub struct LagunaDecodeCtx {
+    ws: DecodeWorkspace,
+    /// Executable-graph handle from `Device::end_capture` (an opaque `u64`;
+    /// see the `Device` trait doc, CUDA's `CUgraphExec` cast through
+    /// `usize`). `None` until the capture step has run.
+    graph: Option<u64>,
+    /// The fixed output buffer graph replays write into (see the struct doc).
+    logits: Option<Tensor>,
+    /// Steps run so far via this ctx (warmup + capture; stops incrementing
+    /// once capture has happened, replay doesn't touch it).
+    step_count: usize,
+}
+
+impl LagunaDecodeCtx {
+    /// Eager warmup steps before the capture step (brief: "warm up N=2 full
+    /// steps normally"). Capture happens on call number `WARMUP_STEPS + 1`.
+    const WARMUP_STEPS: usize = 2;
+
+    /// `model`/`kv`/`max_seq` are accepted (matching the `LagunaKvCache::new`
+    /// shape) for parity with the decode driver's usual construction
+    /// arguments and to leave room for future workspace sizing; today's
+    /// workspace is five fixed 1-element buffers, independent of `max_seq`.
+    pub fn new(dev: &dyn Device, model: &LagunaModel, kv: &LagunaKvCache, max_seq: usize) -> Result<Self> {
+        let _ = (model, kv, max_seq);
+        Ok(LagunaDecodeCtx { ws: DecodeWorkspace::new(dev)?, graph: None, logits: None, step_count: 0 })
+    }
+
+    /// Decode one token. Identical call shape to [`decode_step`] plus `self`;
+    /// internally dispatches to eager warmup, one-time capture, or graph
+    /// replay depending on how many steps have run. See the struct doc for
+    /// the full protocol.
+    pub fn step(
+        &mut self,
+        dev: &dyn Device,
+        model: &LagunaModel,
+        kv: &LagunaKvCache,
+        token_id: u32,
+        pos: u32,
+    ) -> Result<Tensor> {
+        if std::env::var("BUTTER_LAGUNA_DEBUG").is_ok() {
+            return Err(Error::Msg(
+                "LagunaDecodeCtx::step: BUTTER_LAGUNA_DEBUG is incompatible with CUDA-graph \
+                 capture/replay (the debug path downloads mid-step, which the CUDA driver \
+                 forbids during capture). Unset BUTTER_LAGUNA_DEBUG or use plain decode_step."
+                    .into(),
+            ));
+        }
+
+        // Refresh the workspace BEFORE any capture/replay, always outside
+        // the captured region, whether we're warming up, about to capture,
+        // or replaying an already-captured graph.
+        self.ws.update(dev, &model.cfg, token_id, pos)?;
+
+        if let Some(exec) = self.graph {
+            dev.graph_launch(exec)?;
+            return Ok(self
+                .logits
+                .clone()
+                .expect("LagunaDecodeCtx: graph captured but no logits tensor saved"));
+        }
+
+        self.step_count += 1;
+        if self.step_count <= Self::WARMUP_STEPS {
+            // Eager warmup: reaches pool steady state (deterministic alloc
+            // order, every step walks the same fixed layer sequence).
+            return decode_step(dev, model, kv, &self.ws, pos);
+        }
+
+        // step_count == WARMUP_STEPS + 1: capture this step's op sequence.
+        // Capture only RECORDS launches; it does not execute them, so
+        // `logits` is not yet valid data, one `graph_launch` right after
+        // `end_capture` actually computes it (against the workspace values
+        // written above, so it's the correct result for THIS token).
+        dev.begin_capture()?;
+        let logits = decode_step(dev, model, kv, &self.ws, pos)?;
+        let exec = dev.end_capture()?;
+        self.graph = Some(exec);
+        self.logits = Some(logits.clone());
+        dev.graph_launch(exec)?;
+        Ok(logits)
+    }
+}
diff --git a/rust/crates/wh-butter-models/src/lib.rs b/rust/crates/wh-butter-models/src/lib.rs
index 9a7640e4..a8e6ea3b 100644
--- a/rust/crates/wh-butter-models/src/lib.rs
+++ b/rust/crates/wh-butter-models/src/lib.rs
@@ -22,5 +22,6 @@ pub trait Model: Send + Sync {
 
 pub mod dsv4;
 pub mod gguf_tokenizer;
+pub mod laguna;
 pub mod llama;
 pub mod moe;
diff --git a/rust/crates/wh-butter-modeltests/src/lib.rs b/rust/crates/wh-butter-modeltests/src/lib.rs
index b29acc24..542e7599 100644
--- a/rust/crates/wh-butter-modeltests/src/lib.rs
+++ b/rust/crates/wh-butter-modeltests/src/lib.rs
@@ -160,6 +160,733 @@ pub fn run_all(d: &dyn Device, plat: &str) {
     verify_olmoe(d, plat);
     verify_mamba2(d, plat);
     verify_falcon_h1(d, plat);
+    verify_laguna(d, plat);
+}
+
+
+/// Laguna-S-2.1 decode smoke test: env-gated on `BUTTER_LAGUNA_GGUF` (path to
+/// the model's GGUF file). Loads straight from GGUF (no HF safetensors dir,
+/// this model doesn't have a small reference checkpoint to compare against),
+/// runs a handful of decode steps from BOS + a short token sequence with a
+/// persistent KV cache, and prints the logits argmax per step. No HF oracle
+/// assertion (unlike the other `verify_*`, there's no cheap reference to
+/// compare against for a 117B model), just a load-bearing "it runs and
+/// produces finite, plausible-looking output" smoke check.
+/// Host-only Laguna GGUF dequant sanity: dump amax/rms per tensor for a few
+/// layer-0 weights. A healthy checkpoint has projection weights of O(0.01..1);
+/// amax in the hundreds+ means the k-quant dequant path is mis-unpacking
+/// (scale/min layout). Env-gated on BUTTER_LAGUNA_GGUF, no device needed.
+pub fn verify_laguna_host_dequant() {
+    let Ok(path) = std::env::var("BUTTER_LAGUNA_GGUF") else {
+        eprintln!("BUTTER_LAGUNA_GGUF not set, skipping host dequant check");
+        return;
+    };
+    let g = match wh_butter_loader::gguf::Gguf::open(&path) {
+        Ok(g) => g,
+        Err(e) => { eprintln!("gguf open failed: {e:?}"); return; }
+    };
+    // BUTTER_LAGUNA_DECODE_IDS="69377,13829": detokenize ids without a model
+    // load (fast oracle-comparison helper).
+    if let Ok(ids_s) = std::env::var("BUTTER_LAGUNA_DECODE_IDS") {
+        if let Ok(tok) = wh_butter_models::gguf_tokenizer::GgufTokenizer::from_gguf(&g) {
+            let ids: Vec<u32> = ids_s.split(',').filter_map(|s| s.trim().parse().ok()).collect();
+            for &id in &ids {
+                eprintln!("token {id} -> {:?}", tok.decode(&[id]));
+            }
+            eprintln!("joined: {:?}", tok.decode(&ids));
+        }
+        return;
+    }
+    for name in [
+        "token_embd.weight",
+        "blk.0.attn_norm.weight",
+        "blk.0.attn_q.weight",
+        "blk.0.attn_k.weight",
+        "blk.0.attn_gate.weight",
+        "blk.0.ffn_gate.weight",
+        "blk.0.attn_q_norm.weight",
+    ] {
+        let Some(t) = g.tensor(name) else { eprintln!("{name}: MISSING"); continue; };
+        let ty = t.ggml_type;
+        match g.dequant_f32(name) {
+            Ok(f) => {
+                let n = f.len();
+                let amax = f.iter().fold(0f32, |a, &v| a.max(v.abs()));
+                let rms = (f.iter().map(|v| (*v as f64) * (*v as f64)).sum::<f64>() / n as f64).sqrt();
+                let n_nan = f.iter().filter(|v| !v.is_finite()).count();
+                eprintln!("{name}: dtype={ty:?} n={n} amax={amax:.4e} rms={rms:.4e} nonfinite={n_nan} head={:?}", &f[..4.min(n)]);
+            }
+            Err(e) => eprintln!("{name}: dequant failed: {e:?}"),
+        }
+    }
+}
+
+/// Laguna-S-2.1 EXACT host f32 reference for layer 0, single decode step,
+/// token id 2 (BOS) at position 0. Companion to the device instrumented run
+/// (`BUTTER_LAGUNA_DEBUG=2` in `wh_butter_models::laguna::decode_layer`): every
+/// sub-step below prints the same head/|x|2/amax stats so the two logs can be
+/// diffed by eye to find the first place the device path goes wrong.
+///
+/// Every projection is computed TWO ways: `exact` (plain f32 matvec on the
+/// GGUF-dequantized weight) and `roundtrip` (that same weight pushed through
+/// [`wh_butter_ops::quantize_q4`] and dequantized back on host, with the block
+/// scale pushed through an f32->f16->f32 roundtrip first, matching how the
+/// device stores Q4 scales). The device output should be compared against
+/// `roundtrip`, not `exact`: decode always runs on Q4 weights regardless of
+/// the GGUF's original per-tensor dtype (see `wh_butter_models::laguna::load_dense`).
+///
+/// Env-gated on BUTTER_LAGUNA_GGUF, no device/CUDA needed.
+pub fn verify_laguna_host_layer0() {
+    let Ok(path) = std::env::var("BUTTER_LAGUNA_GGUF") else {
+        eprintln!("BUTTER_LAGUNA_GGUF not set, skipping host layer-0 reference");
+        return;
+    };
+    let g = match wh_butter_loader::gguf::Gguf::open(&path) {
+        Ok(g) => g,
+        Err(e) => { eprintln!("gguf open failed: {e:?}"); return; }
+    };
+
+    // Laguna-S-2.1 layer-0 spec constants (dense SwiGLU MLP, full-attention
+    // layer, see wh_butter_models::laguna::LagunaConfig::defaults).
+    let hidden = 3072usize;
+    let head_dim = 128usize;
+    let n_head = 48usize;
+    let n_kv = 8usize;
+    let heads_per_group = n_head / n_kv; // 6
+    let n_ff = 12288usize;
+    let eps = 1e-6f32;
+    let token = 2usize;
+
+    // Position-0 YaRN detail: the full-attention layer's rope has ext_factor
+    // != 0, so the kernel re-multiplies attn_factor by (1 + 0.1*ln(1/freq_scale))
+    // (freq_scale = 1/128, so 1/freq_scale = 128). At position 0 every
+    // rotation angle is zero (theta_extrap = position * ... = 0), so this
+    // collapses to a scalar multiply by mscale over the first n_rot dims of
+    // every head; dims [n_rot, head_dim) pass through untouched.
+    let attn_factor = 1.4852030263919618f32;
+    // mscale = attn_factor exactly: the reference stack pre-divides on the
+    // host to cancel its kernel's internal re-multiply, so the net magnitude
+    // scale equals the stored attention_factor. Our kernel applies it directly.
+    let mscale = attn_factor;
+    let n_rot = 64usize; // full-layer rope.dimension_count
+
+    let dq = |name: &str| -> Vec<f32> {
+        g.dequant_f32(name).unwrap_or_else(|e| panic!("dequant {name} failed: {e:?}"))
+    };
+
+    // f32 <-> f16 (round-to-nearest), matching the device's scale-upload path
+    // (`wh_butter_models::laguna::f32s_to_f16_bytes`), the gemv_q4 kernels read
+    // per-block Q4 scales as f16.
+    fn f32_to_f16(f: f32) -> u16 {
+        let x = f.to_bits();
+        let sign = ((x >> 16) & 0x8000) as u16;
+        let e = ((x >> 23) & 0xff) as i32 - 112;
+        if e <= 0 { return sign; }
+        if e >= 0x1f { return sign | 0x7c00; }
+        let m = (x >> 13) & 0x3ff;
+        let round = (x >> 12) & 1;
+        let v = ((e as u32) << 10) | m;
+        sign | ((v + round) as u16)
+    }
+    fn f16_to_f32(bits: u16) -> f32 {
+        let sign = ((bits >> 15) & 1) as u32;
+        let exp = ((bits >> 10) & 0x1f) as u32;
+        let mant = (bits & 0x3ff) as u32;
+        let out = if exp == 0 {
+            if mant == 0 { sign << 31 } else {
+                let mut e = -1i32;
+                let mut m = mant;
+                while m & 0x400 == 0 { m <<= 1; e -= 1; }
+                (sign << 31) | (((e + 127 - 15) as u32) << 23) | ((m & 0x3ff) << 13)
+            }
+        } else if exp == 0x1f {
+            (sign << 31) | (0xff << 23) | (mant << 13)
+        } else {
+            (sign << 31) | ((exp + 127 - 15) << 23) | (mant << 13)
+        };
+        f32::from_bits(out)
+    }
+
+    // Q4 round-trip: quantize_q4 (block 32, symmetric int4 in [-7,7], scale =
+    // amax/7), then push the scale through f16 before dequantizing back:
+    // mirrors exactly what the CUDA gemv_q4 kernel reads for its scales.
+    let q4_roundtrip = |w: &[f32], m: usize, k: usize| -> Vec<f32> {
+        let (qs, scales) = wh_butter_ops::quantize_q4(w, m, k);
+        let bpr = k / 32;
+        let mut out = vec![0f32; m * k];
+        for r in 0..m {
+            for b in 0..bpr {
+                let d = f16_to_f32(f32_to_f16(scales[r * bpr + b]));
+                for word in 0..4 {
+                    let packed = qs[r * bpr * 4 + b * 4 + word];
+                    for i in 0..8 {
+                        let nib = ((packed >> (i * 4)) & 0xf) as i32;
+                        let q = if nib >= 8 { nib - 16 } else { nib };
+                        out[r * k + b * 32 + word * 8 + i] = q as f32 * d;
+                    }
+                }
+            }
+        }
+        out
+    };
+
+    // Row-major [m,k] @ [k] matvec, f64 accumulation (reference precision).
+    let matvec = |w: &[f32], m: usize, k: usize, x: &[f32]| -> Vec<f32> {
+        let mut out = vec![0f32; m];
+        for r in 0..m {
+            let row = &w[r * k..(r + 1) * k];
+            let mut acc = 0f64;
+            for i in 0..k { acc += row[i] as f64 * x[i] as f64; }
+            out[r] = acc as f32;
+        }
+        out
+    };
+
+    let rms_norm = |x: &[f32], w: &[f32], eps: f32| -> Vec<f32> {
+        let n = x.len();
+        let ms = x.iter().map(|&v| (v as f64) * (v as f64)).sum::<f64>() / n as f64;
+        let inv = 1.0 / (ms + eps as f64).sqrt();
+        (0..n).map(|i| (x[i] as f64 * inv) as f32 * w[i]).collect()
+    };
+
+    // Per-head RMSNorm: same formula, applied independently over each
+    // head_dim-length row (q_norm/k_norm weight is shared across heads).
+    let rms_norm_perhead = |x: &[f32], w: &[f32], n_heads: usize, hd: usize, eps: f32| -> Vec<f32> {
+        let mut out = vec![0f32; n_heads * hd];
+        for hh in 0..n_heads {
+            let row = rms_norm(&x[hh * hd..(hh + 1) * hd], w, eps);
+            out[hh * hd..(hh + 1) * hd].copy_from_slice(&row);
+        }
+        out
+    };
+
+    let softplus = |x: f32| -> f32 { if x > 20.0 { x } else { (1.0 + x.exp()).ln() } };
+    let silu = |x: f32| -> f32 { x / (1.0 + (-x).exp()) };
+
+    let stats = |label: &str, v: &[f32]| {
+        let l2 = v.iter().map(|&x| (x as f64) * (x as f64)).sum::<f64>().sqrt();
+        let amax = v.iter().fold(0f32, |a, &x| a.max(x.abs()));
+        eprintln!("    {label}: head={:?} |x|2={l2:.3e} amax={amax:.3e}", &v[..4.min(v.len())]);
+    };
+
+    // Flags a sub-step whose first 4 values differ from the device dbg line
+    // (pasted from the instrumented CUDA run) by more than 5% relative.
+    let check = |label: &str, v: &[f32], device_head: &[f32]| {
+        let mut worst = 0f32;
+        for i in 0..device_head.len().min(v.len()) {
+            let dv = device_head[i];
+            let rel = if dv.abs() > 1e-12 { (v[i] - dv).abs() / dv.abs() } else { (v[i] - dv).abs() };
+            worst = worst.max(rel);
+        }
+        if worst > 0.05 {
+            eprintln!("    {label}: MISMATCH? max rel diff over head vs device = {worst:.3}");
+        }
+    };
+
+    eprintln!("── Laguna-S-2.1 host f32 reference, layer 0, token {token}, pos 0 ──");
+
+    // (a) embed: dequant_f32 gives row-major [vocab, hidden] (GGUF fastest-
+    // dim-first == hidden fastest, matching load_gguf's load_embed_like).
+    let embd = dq("token_embd.weight");
+    let x: Vec<f32> = embd[token * hidden..(token + 1) * hidden].to_vec();
+    stats("x_in", &x);
+    check("x_in", &x, &[0.28873444, -0.12119293, 0.08377075, -0.12119293]);
+
+    // (b) attn_norm
+    let attn_norm_w = dq("blk.0.attn_norm.weight");
+    let h = rms_norm(&x, &attn_norm_w, eps);
+    stats("attn_norm", &h);
+
+    // weights
+    let wq = dq("blk.0.attn_q.weight");
+    let wk = dq("blk.0.attn_k.weight");
+    let wv = dq("blk.0.attn_v.weight");
+    let wo = dq("blk.0.attn_output.weight");
+    let wg = dq("blk.0.attn_gate.weight");
+    let q_norm_w = dq("blk.0.attn_q_norm.weight");
+    let k_norm_w = dq("blk.0.attn_k_norm.weight");
+    let ffn_norm_w = dq("blk.0.ffn_norm.weight");
+    let w_gate_ffn = dq("blk.0.ffn_gate.weight");
+    let w_up_ffn = dq("blk.0.ffn_up.weight");
+    let w_down_ffn = dq("blk.0.ffn_down.weight");
+
+    let qdim = n_head * head_dim; // 6144
+    let kvdim = n_kv * head_dim; // 1024
+
+    let wq_rt = q4_roundtrip(&wq, qdim, hidden);
+    let wk_rt = q4_roundtrip(&wk, kvdim, hidden);
+    let wv_rt = q4_roundtrip(&wv, kvdim, hidden);
+    let wg_rt = q4_roundtrip(&wg, n_head, hidden);
+
+    // (d) q/k/v/g projections
+    let q_exact = matvec(&wq, qdim, hidden, &h);
+    let q_rt = matvec(&wq_rt, qdim, hidden, &h);
+    stats("q_proj exact", &q_exact);
+    stats("q_proj roundtrip", &q_rt);
+    check("q_proj", &q_rt, &[-0.0017298572, 0.041349545, -0.016936049, -0.0050829584]);
+
+    let k_exact = matvec(&wk, kvdim, hidden, &h);
+    let k_rt = matvec(&wk_rt, kvdim, hidden, &h);
+    stats("k_proj exact", &k_exact);
+    stats("k_proj roundtrip", &k_rt);
+
+    let v_exact = matvec(&wv, kvdim, hidden, &h);
+    let v_rt = matvec(&wv_rt, kvdim, hidden, &h);
+    stats("v_proj exact", &v_exact);
+    stats("v_proj roundtrip", &v_rt);
+
+    let g_exact = matvec(&wg, n_head, hidden, &h);
+    let g_rt = matvec(&wg_rt, n_head, hidden, &h);
+    stats("g_proj exact", &g_exact);
+    stats("g_proj roundtrip", &g_rt);
+
+    // (e) per-head QK RMSNorm
+    let q_norm_exact = rms_norm_perhead(&q_exact, &q_norm_w, n_head, head_dim, eps);
+    let q_norm_rt = rms_norm_perhead(&q_rt, &q_norm_w, n_head, head_dim, eps);
+    let k_norm_exact = rms_norm_perhead(&k_exact, &k_norm_w, n_kv, head_dim, eps);
+    let k_norm_rt = rms_norm_perhead(&k_rt, &k_norm_w, n_kv, head_dim, eps);
+    stats("q_norm exact", &q_norm_exact);
+    stats("q_norm roundtrip", &q_norm_rt);
+    check("q_norm", &q_norm_rt, &[-0.04173411, 1.265691, -0.5209586, -0.06744661]);
+    stats("k_norm exact", &k_norm_exact);
+    stats("k_norm roundtrip", &k_norm_rt);
+
+    // (f) RoPE at position 0 (see mscale comment above).
+    let apply_rope0 = |v: &[f32], n_heads: usize| -> Vec<f32> {
+        let mut out = v.to_vec();
+        for hh in 0..n_heads {
+            for i in 0..n_rot { out[hh * head_dim + i] *= mscale; }
+        }
+        out
+    };
+    let q_rope_exact = apply_rope0(&q_norm_exact, n_head);
+    let q_rope_rt = apply_rope0(&q_norm_rt, n_head);
+    let k_rope_exact = apply_rope0(&k_norm_exact, n_kv);
+    let k_rope_rt = apply_rope0(&k_norm_rt, n_kv);
+    stats("q_rope exact", &q_rope_exact);
+    stats("q_rope roundtrip", &q_rope_rt);
+    stats("k_rope exact", &k_rope_exact);
+    stats("k_rope roundtrip", &k_rope_rt);
+
+    // (g) attention, n_kv_used = 1: softmax over a single cached position is
+    // always 1, so every q-head's output is just its group's cached v.
+    let sdpa = |v_h: &[f32]| -> Vec<f32> {
+        let mut out = vec![0f32; n_head * head_dim];
+        for qh in 0..n_head {
+            let kvh = qh / heads_per_group;
+            out[qh * head_dim..(qh + 1) * head_dim]
+                .copy_from_slice(&v_h[kvh * head_dim..(kvh + 1) * head_dim]);
+        }
+        out
+    };
+    let sdpa_exact = sdpa(&v_exact);
+    let sdpa_rt = sdpa(&v_rt);
+    stats("sdpa exact", &sdpa_exact);
+    stats("sdpa roundtrip", &sdpa_rt);
+    check("sdpa", &sdpa_rt, &[-0.0042726835, -0.119657904, 0.03584555, -0.02707379]);
+
+    // (h) per-head softplus gate, applied before o_proj.
+    let gate_mul = |attn: &[f32], gate: &[f32]| -> Vec<f32> {
+        let mut out = vec![0f32; attn.len()];
+        for hh in 0..n_head {
+            let sp = softplus(gate[hh]);
+            for i in 0..head_dim { out[hh * head_dim + i] = attn[hh * head_dim + i] * sp; }
+        }
+        out
+    };
+    let gated_exact = gate_mul(&sdpa_exact, &g_exact);
+    let gated_rt = gate_mul(&sdpa_rt, &g_rt);
+    stats("gated exact", &gated_exact);
+    stats("gated roundtrip", &gated_rt);
+    check("gated", &gated_rt, &[-0.0016357758, -0.04581044, 0.013723293, -0.010365067]);
+
+    // (i) o_proj
+    let wo_rt = q4_roundtrip(&wo, hidden, qdim);
+    let o_exact = matvec(&wo, hidden, qdim, &gated_exact);
+    let o_rt = matvec(&wo_rt, hidden, qdim, &gated_rt);
+    stats("o_proj exact", &o_exact);
+    stats("o_proj roundtrip", &o_rt);
+    check("o_proj", &o_rt, &[-0.037135, 0.013525429, -0.012531518, -0.044919115]);
+
+    // (j) residual
+    let x1_exact: Vec<f32> = (0..hidden).map(|i| x[i] + o_exact[i]).collect();
+    let x1_rt: Vec<f32> = (0..hidden).map(|i| x[i] + o_rt[i]).collect();
+    stats("x_attn_res exact", &x1_exact);
+    stats("x_attn_res roundtrip", &x1_rt);
+    check("x_attn_res", &x1_rt, &[0.25159943, -0.107667506, 0.07123923, -0.16611205]);
+
+    // (k) dense SwiGLU FFN
+    let h2_exact = rms_norm(&x1_exact, &ffn_norm_w, eps);
+    let h2_rt = rms_norm(&x1_rt, &ffn_norm_w, eps);
+    stats("ffn_norm exact", &h2_exact);
+    stats("ffn_norm roundtrip", &h2_rt);
+    check("ffn_norm", &h2_rt, &[0.0009642678, 0.00045515582, 0.00072807475, 0.0006906503]);
+
+    let w_gate_rt = q4_roundtrip(&w_gate_ffn, n_ff, hidden);
+    let w_up_rt = q4_roundtrip(&w_up_ffn, n_ff, hidden);
+    let w_down_rt = q4_roundtrip(&w_down_ffn, hidden, n_ff);
+
+    let gate_exact = matvec(&w_gate_ffn, n_ff, hidden, &h2_exact);
+    let gate_rt = matvec(&w_gate_rt, n_ff, hidden, &h2_rt);
+    let up_exact = matvec(&w_up_ffn, n_ff, hidden, &h2_exact);
+    let up_rt = matvec(&w_up_rt, n_ff, hidden, &h2_rt);
+
+    let act_exact: Vec<f32> = (0..n_ff).map(|i| silu(gate_exact[i]) * up_exact[i]).collect();
+    let act_rt: Vec<f32> = (0..n_ff).map(|i| silu(gate_rt[i]) * up_rt[i]).collect();
+    stats("ffn_act exact", &act_exact);
+    stats("ffn_act roundtrip", &act_rt);
+    check("ffn_act", &act_rt, &[5.9853085e-7, -0.00017139639, 5.882993e-9, -4.645081e-5]);
+
+    let down_exact = matvec(&w_down_ffn, hidden, n_ff, &act_exact);
+    let down_rt = matvec(&w_down_rt, hidden, n_ff, &act_rt);
+    stats("ffn_down exact", &down_exact);
+    stats("ffn_down roundtrip", &down_rt);
+    check("ffn_down", &down_rt, &[-0.0008143601, 0.00016136278, -0.00094114855, -0.00012908175]);
+
+    let x2_exact: Vec<f32> = (0..hidden).map(|i| x1_exact[i] + down_exact[i]).collect();
+    let x2_rt: Vec<f32> = (0..hidden).map(|i| x1_rt[i] + down_rt[i]).collect();
+    stats("x2 (layer0 out) exact", &x2_exact);
+    stats("x2 (layer0 out) roundtrip", &x2_rt);
+
+    eprintln!("── Laguna-S-2.1 host layer-0 reference complete ──");
+}
+
+pub fn verify_laguna(d: &dyn Device, plat: &str) {
+    use wh_butter_models::gguf_tokenizer::GgufTokenizer;
+    use wh_butter_models::laguna;
+
+    let Ok(path) = std::env::var("BUTTER_LAGUNA_GGUF") else {
+        eprintln!("BUTTER_LAGUNA_GGUF not set, skipping Laguna-S-2.1 decode smoke test");
+        return;
+    };
+    let Ok(g) = wh_butter_loader::gguf::Gguf::open(&path) else {
+        eprintln!("Laguna: failed to open GGUF at {path}, skipping");
+        return;
+    };
+
+    let model = match laguna::load_gguf(d, &g) {
+        Ok(m) => m,
+        Err(e) => {
+            eprintln!("Laguna: load_gguf failed: {e:?}, skipping");
+            return;
+        }
+    };
+
+    // BUTTER_LAGUNA_PP=<n>: standalone batched-prefill throughput bench over a
+    // synthetic n-token prompt (token id sequence i % 50000, any valid ids
+    // exercise the same code path; this is a speed bench, not a semantic
+    // eval). BUTTER_LAGUNA_PREFILL_CHUNK overrides the chunk size (default
+    // 1024). Prints "pp tok/s" and returns, no generation, no decode.
+    if let Ok(pp_s) = std::env::var("BUTTER_LAGUNA_PP") {
+        // Comma-separated sizes bench the whole ladder on ONE model load
+        // (the 71G requant load dominates a cycle).
+        let sizes: Vec<usize> = pp_s.split(',').filter_map(|v| v.trim().parse().ok()).collect();
+        // BUTTER_LAGUNA_PREFILL_CHUNK also accepts a comma list: the harness
+        // benches every (size, chunk) combo on the single model load.
+        let chunks: Vec<usize> = std::env::var("BUTTER_LAGUNA_PREFILL_CHUNK")
+            .ok().map(|v| v.split(',').filter_map(|c| c.trim().parse().ok()).collect())
+            .filter(|v: &Vec<usize>| !v.is_empty())
+            .unwrap_or_else(|| vec![1024]);
+        for &(mut chunk) in chunks.iter() {
+        chunk = chunk.max(1);
+        for &n in sizes.iter().filter(|&&n| n > 0) {
+        let synth_tokens: Vec<u32> = (0..n).map(|i| (i % 50000) as u32).collect();
+        let Ok(kv_pp) = laguna::LagunaKvCache::new(d, &model.cfg, n + 1) else {
+            eprintln!("Laguna PP bench: KV cache alloc failed, skipping");
+            continue;
+        };
+        let Ok(scratch_pp) = laguna::LagunaPrefillScratch::new(d, &model.cfg, n) else {
+            eprintln!("Laguna PP bench: prefill scratch alloc failed, skipping");
+            continue;
+        };
+        // One warmup pass (first-touch device allocs / kernel resolution),
+        // then the timed pass, matches the bench-honest convention used
+        // elsewhere in this harness (e.g. the BUTTER_LAGUNA_SWEEP loop above).
+        if laguna::prefill(d, &model, &kv_pp, &scratch_pp, &synth_tokens, chunk).is_err() {
+            eprintln!("Laguna PP bench: warmup prefill failed, skipping timed run");
+            continue;
+        }
+        let _ = d.synchronize();
+        let kv_pp2 = match laguna::LagunaKvCache::new(d, &model.cfg, n + 1) {
+            Ok(k) => k,
+            Err(e) => { eprintln!("Laguna PP bench: KV cache realloc failed: {e:?}"); continue; }
+        };
+        let scratch_pp2 = match laguna::LagunaPrefillScratch::new(d, &model.cfg, n) {
+            Ok(s) => s,
+            Err(e) => { eprintln!("Laguna PP bench: scratch realloc failed: {e:?}"); continue; }
+        };
+        let t0 = std::time::Instant::now();
+        match laguna::prefill(d, &model, &kv_pp2, &scratch_pp2, &synth_tokens, chunk) {
+            Ok(logits) => {
+                let _ = d.synchronize();
+                let dt = t0.elapsed().as_secs_f64();
+                let tps = n as f64 / dt;
+                let mut lb = vec![0u8; model.cfg.vocab * 4];
+                let _ = d.download(logits.buffer.as_ref(), &mut lb); // liveness check on the returned tensor
+                eprintln!("Laguna prefill bench (n={n}, chunk={chunk}, {plat}): {tps:.2} pp tok/s");
+                laguna::prof_report();
+            }
+            Err(e) => eprintln!("Laguna PP bench: timed prefill failed: {e:?}"),
+        }
+        }
+        }
+        return;
+    }
+
+    let bos = g.meta_u32("tokenizer.ggml.bos_token_id").unwrap_or(0);
+    // A short prompt: BOS + whatever the tokenizer produces for a trivial
+    // string, falling back to a couple of low-valued ids if the tokenizer
+    // metadata is unavailable (this is a decode-plumbing smoke test, not a
+    // semantic eval, any valid token ids exercise the same code path).
+    let mut tokens = vec![bos];
+    if let Ok(tok) = GgufTokenizer::from_gguf(&g) {
+        tokens.extend(tok.encode("Hello"));
+    } else {
+        tokens.extend_from_slice(&[1, 2]);
+    }
+
+    let max_seq = tokens.len()
+        + std::env::var("BUTTER_LAGUNA_GEN").ok().and_then(|v| v.parse::<usize>().ok()).unwrap_or(16)
+        + 1;
+    let Ok(kv) = laguna::LagunaKvCache::new(d, &model.cfg, max_seq) else {
+        eprintln!("Laguna: KV cache alloc failed, skipping");
+        return;
+    };
+
+    // BUTTER_LAGUNA_SWEEP="moe_rpt,gemv_rpt;moe_rpt,gemv_rpt;...": bench each
+    // warp-count combo in ONE process (the 71G load dominates a bench cycle,
+    // so re-capturing a fresh graph per combo amortizes it). Each combo gets
+    // a fresh KV cache + decode ctx; graph capture bakes the env values read
+    // at capture time, so set_var before ctx creation takes effect per combo.
+    if let Ok(sweep) = std::env::var("BUTTER_LAGUNA_SWEEP") {
+        let n_gen: usize = std::env::var("BUTTER_LAGUNA_GEN").ok().and_then(|v| v.parse().ok()).unwrap_or(64);
+        for combo in sweep.split(';').filter(|s| !s.is_empty()) {
+            let parts: Vec<&str> = combo.split(',').collect();
+            if parts.len() != 2 { eprintln!("sweep: bad combo {combo:?}"); continue; }
+            // set_var is unsafe in this edition; single-threaded bench context.
+            unsafe {
+                std::env::set_var("MT_MOE_RPT", parts[0].trim());
+                std::env::set_var("MT_GEMV_RPT", parts[1].trim());
+            }
+            let Ok(kv2) = laguna::LagunaKvCache::new(d, &model.cfg, tokens.len() + n_gen + 1) else {
+                eprintln!("sweep: kv alloc failed"); return;
+            };
+            let Ok(mut ctx2) = laguna::LagunaDecodeCtx::new(d, &model, &kv2, tokens.len() + n_gen + 1) else {
+                eprintln!("sweep: ctx failed"); return;
+            };
+            let mut cur2: Option<u32> = None;
+            let mut t0 = std::time::Instant::now();
+            let total2 = tokens.len() + n_gen;
+            let mut first_gen: Vec<u32> = Vec::new();
+            let mut died = false;
+            for pos in 0..total2 {
+                let tok = if pos < tokens.len() { tokens[pos] } else { cur2.unwrap() };
+                match ctx2.step(d, &model, &kv2, tok, pos as u32) {
+                    Ok(logits) => {
+                        let am = match wh_butter_ops::argmax_f32_device(d, &logits) {
+                            Ok(a) => a,
+                            Err(e) => { eprintln!("sweep argmax failed: {e:?}"); died = true; break; }
+                        };
+                        let mut ib = [0u8; 4];
+                        let _ = d.synchronize();
+                        if d.download(am.buffer.as_ref(), &mut ib).is_err() { died = true; break; }
+                        let a = u32::from_le_bytes(ib);
+                        if pos >= tokens.len() && first_gen.len() < 6 { first_gen.push(a); }
+                        cur2 = Some(a);
+                        if pos + 1 == tokens.len() { t0 = std::time::Instant::now(); }
+                    }
+                    Err(e) => { eprintln!("sweep combo {combo}: step failed: {e:?}"); died = true; break; }
+                }
+            }
+            if !died {
+                let tps = n_gen as f64 / t0.elapsed().as_secs_f64();
+                eprintln!("SWEEP moe_rpt={} gemv_rpt={}: {tps:.2} tok/s first_gen={first_gen:?}", parts[0].trim(), parts[1].trim());
+            }
+        }
+        eprintln!("✅ Laguna sweep complete ({plat}).");
+        return;
+    }
+
+    // BUTTER_LAGUNA_GRAPH=1: capture one full decode step as a CUDA graph and
+    // replay it per token instead of relaunching ~1000+ kernels every step
+    // (see `wh_butter_models::laguna::LagunaDecodeCtx`). Default (unset): plain
+    // eager `decode_step`, now reading a persistent per-step workspace
+    // instead of uploading fresh token-id/position buffers every call,
+    // same kernels, same scalars, byte-identical output to before.
+    let graph_mode = std::env::var("BUTTER_LAGUNA_GRAPH").is_ok();
+    let mode_label = if graph_mode { "graph" } else { "eager" };
+    let ws = if graph_mode {
+        None
+    } else {
+        match laguna::DecodeWorkspace::new(d) {
+            Ok(w) => Some(w),
+            Err(e) => {
+                eprintln!("Laguna: decode workspace alloc failed: {e:?}, skipping");
+                return;
+            }
+        }
+    };
+    let mut ctx = if graph_mode {
+        match laguna::LagunaDecodeCtx::new(d, &model, &kv, max_seq) {
+            Ok(c) => Some(c),
+            Err(e) => {
+                eprintln!("Laguna: LagunaDecodeCtx::new failed: {e:?}, skipping");
+                return;
+            }
+        }
+    } else {
+        None
+    };
+
+    // Greedy generation: feed the prompt, then feed each argmax back in.
+    // BUTTER_LAGUNA_GEN overrides the generated-token count (default 16).
+    // Prints the detokenized continuation for direct comparison against the
+    // reference engine's temp-0 output on the same prompt, plus decode t/s
+    // over the generation phase (each step is one full 48-layer forward).
+    let n_gen: usize = std::env::var("BUTTER_LAGUNA_GEN").ok().and_then(|v| v.parse().ok()).unwrap_or(16);
+    let tok_dec = GgufTokenizer::from_gguf(&g).ok();
+    let mut generated: Vec<u32> = Vec::new();
+    let mut cur: Option<u32> = None;
+    let mut gen_t0 = std::time::Instant::now();
+    let total = tokens.len() + n_gen;
+    // BUTTER_LAGUNA_FAST=1: device-side argmax, download 4 bytes/step instead
+    // of the full 400KB logit row (the bench-honest greedy path). Default
+    // keeps the full download for logit stats/top-5 debugging.
+    let fast = std::env::var("BUTTER_LAGUNA_FAST").map(|v| v == "1").unwrap_or(false);
+
+    // BUTTER_LAGUNA_PREFILL=1: run the whole prompt through the batched
+    // `laguna::prefill` path in ONE (chunked) forward instead of feeding it
+    // token-by-token through decode_step/ctx.step below, then resume the
+    // IDENTICAL per-token decode loop from `pos = tokens.len()`. This is the
+    // correctness gate for the batched-prefill work: the greedy continuation
+    // must come out byte-identical to the decode-only path (same prompt,
+    // same seed token handed to decode at `pos = tokens.len()`).
+    // BUTTER_LAGUNA_PREFILL_CHUNK overrides the chunk size (default 1024).
+    let prefill_mode = std::env::var("BUTTER_LAGUNA_PREFILL").map(|v| v == "1").unwrap_or(false);
+    let prefill_chunk: usize = std::env::var("BUTTER_LAGUNA_PREFILL_CHUNK")
+        .ok().and_then(|v| v.parse().ok()).unwrap_or(2048);
+    let start_pos = if prefill_mode {
+        let Ok(scratch) = laguna::LagunaPrefillScratch::new(d, &model.cfg, tokens.len()) else {
+            eprintln!("Laguna: prefill scratch alloc failed, skipping");
+            return;
+        };
+        let logits = match laguna::prefill(d, &model, &kv, &scratch, &tokens, prefill_chunk) {
+            Ok(l) => l,
+            Err(e) => {
+                eprintln!("Laguna: prefill failed: {e:?}");
+                return;
+            }
+        };
+        let mut lb = vec![0u8; model.cfg.vocab * 4];
+        let _ = d.synchronize();
+        if d.download(logits.buffer.as_ref(), &mut lb).is_err() {
+            eprintln!("Laguna: prefill logits download failed");
+            return;
+        }
+        let lf: Vec<f32> = lb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+        let argmax = wh_butter_runtime::argmax(&lf);
+        eprintln!(
+            "Laguna prefill (prompt_len={}, chunk={prefill_chunk}, {plat}): last-token argmax = {argmax}",
+            tokens.len()
+        );
+        cur = Some(argmax as u32);
+        gen_t0 = std::time::Instant::now(); // generation phase starts here, same convention as decode-only mode
+        tokens.len()
+    } else {
+        0
+    };
+
+    for pos in start_pos..total {
+        let tok = if pos < tokens.len() { tokens[pos] } else { cur.unwrap() };
+        // Graph mode: LagunaDecodeCtx internally warms up (eager), captures
+        // once, then replays the captured graph. Eager mode: refresh the
+        // persistent workspace by hand, then call decode_step directly:
+        // the argmax/download below still happen outside either path,
+        // exactly as before.
+        let step_result = if let Some(ctx) = ctx.as_mut() {
+            ctx.step(d, &model, &kv, tok, pos as u32)
+        } else {
+            let w = ws.as_ref().expect("eager path always has a workspace");
+            if let Err(e) = w.update(d, &model.cfg, tok, pos as u32) {
+                eprintln!("Laguna step {pos}: workspace update failed: {e:?}");
+                return;
+            }
+            laguna::decode_step(d, &model, &kv, w, pos as u32)
+        };
+        match step_result {
+            Ok(logits) => {
+                if fast {
+                    let Ok(am) = wh_butter_ops::argmax_f32_device(d, &logits) else {
+                        eprintln!("Laguna step {pos}: device argmax failed");
+                        return;
+                    };
+                    let mut ib = [0u8; 4];
+                    let _ = d.synchronize();
+                    if d.download(am.buffer.as_ref(), &mut ib).is_err() {
+                        eprintln!("Laguna step {pos}: argmax download failed");
+                        return;
+                    }
+                    let argmax = u32::from_le_bytes(ib);
+                    if pos >= tokens.len() {
+                        generated.push(argmax);
+                    }
+                    cur = Some(argmax);
+                    if pos + 1 == tokens.len() {
+                        gen_t0 = std::time::Instant::now();
+                    }
+                    if pos + 1 == total {
+                        let dt = gen_t0.elapsed().as_secs_f64();
+                        let tps = n_gen as f64 / dt;
+                        let text = tok_dec.as_ref().map(|t| t.decode(&generated)).unwrap_or_default();
+                        eprintln!("Laguna greedy continuation ({n_gen} tokens, {tps:.2} tok/s device-argmax, {mode_label}): {text:?}");
+                    }
+                    continue;
+                }
+                let mut lb = vec![0u8; model.cfg.vocab * 4];
+                let _ = d.synchronize();
+                if d.download(logits.buffer.as_ref(), &mut lb).is_err() {
+                    eprintln!("Laguna step {pos} (token {tok}): logits download failed");
+                    return;
+                }
+                let lf: Vec<f32> = lb.chunks_exact(4).map(|c| f32::from_le_bytes(c.try_into().unwrap())).collect();
+                let argmax = wh_butter_runtime::argmax(&lf);
+                // Logit sanity: NaN/inf counts, range, and top-5 ids. An argmax
+                // parked at the last vocab id is the classic all-NaN signature.
+                let n_nan = lf.iter().filter(|v| v.is_nan()).count();
+                let n_inf = lf.iter().filter(|v| v.is_infinite()).count();
+                let (mut mn, mut mx) = (f32::INFINITY, f32::NEG_INFINITY);
+                for &v in &lf {
+                    if v.is_finite() {
+                        if v < mn { mn = v; }
+                        if v > mx { mx = v; }
+                    }
+                }
+                let mut order: Vec<usize> = (0..lf.len()).collect();
+                order.sort_by(|&a, &b| lf[b].total_cmp(&lf[a]));
+                let top5: Vec<(usize, f32)> = order.iter().take(5).map(|&i| (i, lf[i])).collect();
+                eprintln!("Laguna-S-2.1 decode step {pos} (token {tok}) on {plat}: argmax = {argmax}");
+                eprintln!("  logits: nan={n_nan} inf={n_inf} finite range [{mn:.4}, {mx:.4}] top5={top5:?}");
+                if pos >= tokens.len() {
+                    generated.push(argmax as u32);
+                }
+                cur = Some(argmax as u32);
+                if pos + 1 == tokens.len() {
+                    gen_t0 = std::time::Instant::now(); // time the pure-generation phase
+                }
+                if pos + 1 == total {
+                    let dt = gen_t0.elapsed().as_secs_f64();
+                    let tps = n_gen as f64 / dt;
+                    let text = tok_dec.as_ref().map(|t| t.decode(&generated)).unwrap_or_default();
+                    eprintln!("Laguna greedy continuation ({n_gen} tokens, {tps:.2} tok/s incl logit download, {mode_label}): {text:?}");
+                }
+            }
+            Err(e) => {
+                eprintln!("Laguna step {pos} (token {tok}) failed: {e:?}");
+                return;
+            }
+        }
+    }
+    eprintln!("✅ Laguna-S-2.1 decode smoke test completed ({plat}).");
 }
 
 // exact-erf GELU (Abramowitz-Stegun) — shared by GPT-NeoX / Whisper-style nets
@@ -3538,7 +4265,7 @@ pub fn bench_nemotron(d: &dyn Device, plat: &str) {
                                         pf!(7, attn_flops, wh_butter_ops::sdpa_flash_wmma(d, &qr.reshaped(vec![s, nq, hd]), &kcache.reshaped(vec![nkv, cap, hd]), &vcache.reshaped(vec![nkv, cap, hd]), hd, nq as u32, base as u32, s as u32, cap as u32, (nq/nkv) as u32, true, ascale).unwrap()) }
                                     _ => {
                                 // Packed: block-diagonal varlen flash-attn (each query stays in its segment).
-                                pf!(7, attn_flops, sdpa_multi_tc_varlen(d, &qr.reshaped(vec![s, nq, hd]), &kcache.reshaped(vec![nkv, cap, hd]), &vcache.reshaped(vec![nkv, cap, hd]), seg_lo, pk_seg_len, hd, nq as u32, base as u32, s as u32, cap as u32, (nq/nkv) as u32, true, ascale).unwrap())
+                                pf!(7, attn_flops, sdpa_multi_tc_varlen(d, &qr.reshaped(vec![s, nq, hd]), &kcache.reshaped(vec![nkv, cap, hd]), &vcache.reshaped(vec![nkv, cap, hd]), seg_lo, pk_seg_len, hd, nq as u32, base as u32, s as u32, cap as u32, (nq/nkv) as u32, true, ascale, 0).unwrap())
                                     }
                                 }
                             } else if hd == 128 && std::env::var("NEMOTRON_FLASH_MMA_OFF").is_err()
diff --git a/rust/crates/wh-butter-ops/src/lib.rs b/rust/crates/wh-butter-ops/src/lib.rs
index 5677d1f8..0de49c50 100644
--- a/rust/crates/wh-butter-ops/src/lib.rs
+++ b/rust/crates/wh-butter-ops/src/lib.rs
@@ -3714,6 +3714,16 @@ pub fn sdpa_multi_tc_varlen(
     heads_per_group: u32,
     causal: bool,
     scale: f32,
+    // Sliding-window block skip. 0 = off (no effect on any existing caller).
+    // When `win > 0`, `seg_lo` is expected to encode a sliding window of
+    // width `win` (`seg_lo[r] = max(0, (base_kv+r) - (win-1))`, exactly what
+    // Laguna's sliding-layer prefill already uploads) and the per-KV-block
+    // query-row range fed to the QKᵀ/PV GEMMs is restricted to the rows
+    // whose window can possibly reach that block, see the derivation at
+    // the `q_lo`/`q_cnt` computation below. Independent of `seg_len`/the
+    // packed-segment skip: a caller should pass exactly one of the two
+    // (Laguna always passes `seg_len = 0` and this `win`).
+    win: u32,
 ) -> Result<Tensor> {
     let hd = head_dim;
     let nq = n_q_heads as usize;
@@ -3766,21 +3776,58 @@ pub fn sdpa_multi_tc_varlen(
     let mut kb0 = 0usize;
     while kb0 < n_kv {
         let blk = bk.min(n_kv - kb0);
-        // Segment-skip: when each KV-block is exactly one packed segment
-        // (seg_len == bk), only that segment's query rows [kb0, kb0+blk) attend
-        // to this block — every other row masks to p=0. Restrict the expensive
-        // QKᵀ/PV tensor-core GEMMs to that range (O((NL)²)→O(NL²)); the cheap
-        // full-range softmax/merge correctly no-op the off-segment rows.
-        let (q_lo, q_cnt) = if seg_len != 0 && seg_len as usize == bk { (kb0, blk) } else { (0, sq) };
-        for g in 0..nkv {
-            let q_off = g * hpg * sq * hd * 2 + q_lo * hd * 2;
-            let k_off = (g * n_kv + kb0) * hd * 2;
-            let s_off = g * hpg * sq * blk * 2 + q_lo * blk * 2;
-            dev.gemm_strided_batched_off(
-                qh.buffer.as_ref(),     q_off, (sq*hd) as i64 * el,
-                kh.buffer.as_ref(),     k_off, 0,
-                scores.buffer.as_ref(), s_off, (sq*blk) as i64 * el,
-                q_cnt, blk, hd, hpg, DType::F16)?;
+        // Query-range skip: restrict the expensive QKᵀ/PV tensor-core GEMMs
+        // to the rows that can possibly attend anywhere in KV block [kb0,
+        // kb0+blk), the cheap full-range softmax/merge below still apply
+        // the real per-element mask, so a range that's a SUPERSET of the
+        // true set is always safe (the extra rows just mask to p=0); a
+        // range narrower than the true set would silently drop attention
+        // mass. Only ever shrink, never tighten past that guarantee.
+        //
+        // Packed-segment mode (seg_len == bk): each KV-block is exactly one
+        // packed segment, so only that segment's own rows [kb0, kb0+blk)
+        // attend to it, row index and absolute KV position coincide in
+        // this mode (packed sequences are laid out from base_kv=0).
+        //
+        // Sliding-window mode (win != 0): causal means query absolute
+        // position i_abs can reach KV position kb0 (the block's first
+        // index) only if i_abs >= kb0. The window means i_abs can reach KV
+        // position kb0+blk-1 (the block's last index) only if that index is
+        // still inside the window, i.e. seg_lo[i] = i_abs - win + 1 <=
+        // kb0+blk-1, i.e. i_abs < kb0 + blk + win. So the absolute-position
+        // range that can touch this block is [kb0, kb0+blk+win). Row `r` of
+        // this call sits at absolute position `base + r` (base_kv is this
+        // chunk's starting position), so convert that absolute range to a
+        // row-index range and clamp into [0, sq).
+        let (q_lo, q_cnt) = if win != 0 {
+            let lo_abs = kb0 as i64;
+            let hi_abs_excl = kb0 as i64 + blk as i64 + win as i64;
+            let lo_r = (lo_abs - base as i64).max(0);
+            let hi_r = (hi_abs_excl - base as i64).min(sq as i64);
+            if hi_r > lo_r { (lo_r as usize, (hi_r - lo_r) as usize) } else { (0, 0) }
+        } else if seg_len != 0 && seg_len as usize == bk {
+            (kb0, blk)
+        } else {
+            (0, sq)
+        };
+        // q_cnt == 0: no row in this chunk can reach this block at all (the
+        // derivation above is a superset of the true reachable set, so an
+        // empty computed range proves the true set is also empty). Skip the
+        // QKᵀ/PV GEMMs and vprep entirely; softmax/merge stay unconditional
+        // below (softmax re-derives bm/bl fresh every iteration straight
+        // from the mask, and merge's `l_b <= 0` early-return means it never
+        // reads the skipped o_blk/vt buffers, so this is correctness-safe).
+        if q_cnt > 0 {
+            for g in 0..nkv {
+                let q_off = g * hpg * sq * hd * 2 + q_lo * hd * 2;
+                let k_off = (g * n_kv + kb0) * hd * 2;
+                let s_off = g * hpg * sq * blk * 2 + q_lo * blk * 2;
+                dev.gemm_strided_batched_off(
+                    qh.buffer.as_ref(),     q_off, (sq*hd) as i64 * el,
+                    kh.buffer.as_ref(),     k_off, 0,
+                    scores.buffer.as_ref(), s_off, (sq*blk) as i64 * el,
+                    q_cnt, blk, hd, hpg, DType::F16)?;
+            }
         }
         // varlen softmax: causal upper (base+r) AND segment lower (seg_lo[r]).
         dev.dispatch_raw_cuda(SDPA_TC_SOFTMAX_VARLEN_SRC, "sdpa_tc_softmax_varlen.cu", "sdpa_tc_softmax_varlen",
@@ -3789,19 +3836,21 @@ pub fn sdpa_multi_tc_varlen(
               (seg_lo.buffer.as_ref(), seg_lo.offset)],
             &[i(nq as i32), i(sq as i32), i(blk as i32), i(kb0 as i32), i(base as i32)],
             [(nq*sq) as u32, 1, 1], [256,1,1], 0, false)?;
-        dev.dispatch_raw_cuda(SDPA_TC_VPREP_SRC, "sdpa_tc_vprep.cu", "sdpa_tc_vprep",
-            &[(v.buffer.as_ref(), v.offset), (vt.buffer.as_ref(), 0)],
-            &[i(nkv as i32), i(kv_stride as i32), i(hd as i32), i(kb0 as i32), i(blk as i32), i(in_f32)],
-            blk256(nkv*blk*hd), [256,1,1], 0, false)?;
-        for g in 0..nkv {
-            let p_off = g * hpg * sq * blk * 2 + q_lo * blk * 2;
-            let v_off = g * hd * blk * 2;
-            let o_off = g * hpg * sq * hd * 2 + q_lo * hd * 2;
-            dev.gemm_strided_batched_off(
-                p_blk.buffer.as_ref(), p_off, (sq*blk) as i64 * el,
-                vt.buffer.as_ref(),    v_off, 0,
-                o_blk.buffer.as_ref(), o_off, (sq*hd) as i64 * el,
-                q_cnt, hd, blk, hpg, DType::F16)?;
+        if q_cnt > 0 {
+            dev.dispatch_raw_cuda(SDPA_TC_VPREP_SRC, "sdpa_tc_vprep.cu", "sdpa_tc_vprep",
+                &[(v.buffer.as_ref(), v.offset), (vt.buffer.as_ref(), 0)],
+                &[i(nkv as i32), i(kv_stride as i32), i(hd as i32), i(kb0 as i32), i(blk as i32), i(in_f32)],
+                blk256(nkv*blk*hd), [256,1,1], 0, false)?;
+            for g in 0..nkv {
+                let p_off = g * hpg * sq * blk * 2 + q_lo * blk * 2;
+                let v_off = g * hd * blk * 2;
+                let o_off = g * hpg * sq * hd * 2 + q_lo * hd * 2;
+                dev.gemm_strided_batched_off(
+                    p_blk.buffer.as_ref(), p_off, (sq*blk) as i64 * el,
+                    vt.buffer.as_ref(),    v_off, 0,
+                    o_blk.buffer.as_ref(), o_off, (sq*hd) as i64 * el,
+                    q_cnt, hd, blk, hpg, DType::F16)?;
+            }
         }
         dev.dispatch_raw_cuda(SDPA_TC_MERGE_SRC, "sdpa_tc_merge.cu", "sdpa_tc_merge",
             &[(o_blk.buffer.as_ref(), 0), (bm.buffer.as_ref(), 0), (bl.buffer.as_ref(), 0),
@@ -8111,3 +8160,1452 @@ pub fn add_rms_norm(
     )?;
     Ok((residual, normed))
 }
+
+// ============================================================
+// Backend-specific ops ported from the long-lived CUDA/HIP/Vulkan
+// backends branch (feat/cuda-hip-vulkan-backends), rebranded onto
+// Butter/Iron naming. These dispatch either self-contained embedded
+// CUDA source (dev.dispatch_raw_cuda) or per-backend Device trait
+// methods (dev.moe_marlin_gemm / dev.marlin_repack / ...) — they do
+// NOT depend on the wh-iron kernel-IR registry, so they carry over
+// unchanged by the kernel-dependency gap noted below for
+// moe_gather_q4 / moe_gather_q4_swiglu.
+// ============================================================
+
+const ARGMAX_F32_SRC: &str = r#"
+extern "C" __global__ void butter_argmax_f32(
+    const float* __restrict__ x, unsigned* __restrict__ out, int n)
+{
+    __shared__ float sv[1024];
+    __shared__ unsigned si[1024];
+    int tid = threadIdx.x;
+    float best = -3.4028235e38f;
+    unsigned bi = 0u;
+    for (int i = tid; i < n; i += blockDim.x) {
+        float v = x[i];
+        if (v > best || (v == best && (unsigned)i < bi)) { best = v; bi = (unsigned)i; }
+    }
+    sv[tid] = best; si[tid] = bi;
+    __syncthreads();
+    for (int s = blockDim.x / 2; s > 0; s >>= 1) {
+        if (tid < s) {
+            if (sv[tid + s] > sv[tid] || (sv[tid + s] == sv[tid] && si[tid + s] < si[tid])) {
+                sv[tid] = sv[tid + s]; si[tid] = si[tid + s];
+            }
+        }
+        __syncthreads();
+    }
+    if (tid == 0) out[0] = si[0];
+}
+"#;
+
+const CONV1D_PREFILL_V4_SRC: &str = r#"
+#include <cuda_runtime.h>
+
+__device__ __forceinline__ float silu_f32(float x) {
+    return x / (1.0f + expf(-x));
+}
+
+extern "C" __global__ void conv1d_causal_prefill_v4(
+    const float* __restrict__ xbc,
+    const float* __restrict__ w,
+    const float* __restrict__ bias,
+    float* __restrict__ y,
+    int conv_dim4, int kc, int total4)
+{
+    int idx = (int)(blockIdx.x * blockDim.x + threadIdx.x);
+    if (idx >= total4) return;
+    int ti = idx / conv_dim4;
+    int ci4 = idx - ti * conv_dim4;
+    const float4* __restrict__ x4 = reinterpret_cast<const float4*>(xbc);
+    const float4* __restrict__ w4 = reinterpret_cast<const float4*>(w);
+    const float4* __restrict__ b4 = reinterpret_cast<const float4*>(bias);
+    float4 acc = b4[ci4];
+    #pragma unroll
+    for (int k = 0; k < 8; ++k) {
+        if (k >= kc) break;
+        int src_t = ti - (kc - 1 - k);
+        if (src_t >= 0) {
+            float4 xv = x4[(size_t)src_t * conv_dim4 + ci4];
+            float4 wv = w4[(size_t)k * conv_dim4 + ci4];
+            acc.x += xv.x * wv.x;
+            acc.y += xv.y * wv.y;
+            acc.z += xv.z * wv.z;
+            acc.w += xv.w * wv.w;
+        }
+    }
+    acc.x = silu_f32(acc.x);
+    acc.y = silu_f32(acc.y);
+    acc.z = silu_f32(acc.z);
+    acc.w = silu_f32(acc.w);
+    reinterpret_cast<float4*>(y)[idx] = acc;
+}
+
+extern "C" __global__ void conv1d_causal_prefill_split_v4(
+    const float* __restrict__ xbc,
+    const float* __restrict__ w,
+    const float* __restrict__ bias,
+    float* __restrict__ x_out,
+    float* __restrict__ b_out,
+    float* __restrict__ c_out,
+    int conv_dim4, int di4, int ng_ds4, int kc, int total4)
+{
+    int idx = (int)(blockIdx.x * blockDim.x + threadIdx.x);
+    if (idx >= total4) return;
+    int ti = idx / conv_dim4;
+    int ci4 = idx - ti * conv_dim4;
+    const float4* __restrict__ x4 = reinterpret_cast<const float4*>(xbc);
+    const float4* __restrict__ w4 = reinterpret_cast<const float4*>(w);
+    const float4* __restrict__ b4 = reinterpret_cast<const float4*>(bias);
+    float4 acc = b4[ci4];
+    #pragma unroll
+    for (int k = 0; k < 8; ++k) {
+        if (k >= kc) break;
+        int src_t = ti - (kc - 1 - k);
+        if (src_t >= 0) {
+            float4 xv = x4[(size_t)src_t * conv_dim4 + ci4];
+            float4 wv = w4[(size_t)k * conv_dim4 + ci4];
+            acc.x += xv.x * wv.x;
+            acc.y += xv.y * wv.y;
+            acc.z += xv.z * wv.z;
+            acc.w += xv.w * wv.w;
+        }
+    }
+    acc.x = silu_f32(acc.x);
+    acc.y = silu_f32(acc.y);
+    acc.z = silu_f32(acc.z);
+    acc.w = silu_f32(acc.w);
+    if (ci4 < di4) {
+        reinterpret_cast<float4*>(x_out)[(size_t)ti * di4 + ci4] = acc;
+    } else if (ci4 < di4 + ng_ds4) {
+        int j = ci4 - di4;
+        reinterpret_cast<float4*>(b_out)[(size_t)ti * ng_ds4 + j] = acc;
+    } else {
+        int j = ci4 - di4 - ng_ds4;
+        reinterpret_cast<float4*>(c_out)[(size_t)ti * ng_ds4 + j] = acc;
+    }
+}
+"#;
+
+const EXTRACT_COLS_F16_SRC: &str = r#"
+#include <cuda_fp16.h>
+extern "C" __global__ void butter_extract_cols_f16(
+    const __half* __restrict__ src, __half* __restrict__ dst,
+    int rows, int stride, int col_off, int width)
+{
+    long i = (long)blockIdx.x * blockDim.x + threadIdx.x;
+    long total = (long)rows * width;
+    if (i >= total) return;
+    int c = (int)(i % width);
+    long r = i / width;
+    dst[i] = src[r * stride + col_off + c];
+}
+"#;
+
+const EXTRACT_COLS_SRC: &str = r#"
+extern "C" __global__ void butter_extract_cols(
+    const float* __restrict__ src, float* __restrict__ dst,
+    int rows, int stride, int col_off, int width)
+{
+    long i = (long)blockIdx.x * blockDim.x + threadIdx.x;
+    long total = (long)rows * width;
+    if (i >= total) return;
+    int c = (int)(i % width);
+    long r = i / width;
+    dst[i] = src[r * stride + col_off + c];
+}
+"#;
+
+const GATE_SOFTPLUS_MUL_PERHEAD_MANY_SRC: &str = r#"
+extern "C" __global__ void butter_gate_softplus_mul_perhead_many(
+    const float* __restrict__ attn, const float* __restrict__ g,
+    float* __restrict__ out, int head_dim)
+{
+    long row = (long)blockIdx.x;   // flat (t * n_heads + h)
+    int i = threadIdx.x;
+    if (i >= head_dim) return;
+    float gv = g[row];
+    float sp = gv > 20.0f ? gv : log1pf(expf(gv));
+    out[row * head_dim + i] = attn[row * head_dim + i] * sp;
+}
+"#;
+
+const GATE_SOFTPLUS_MUL_PERHEAD_SRC: &str = r#"
+extern "C" __global__ void butter_gate_softplus_mul_perhead(
+    const float* __restrict__ attn, const float* __restrict__ g,
+    float* __restrict__ out, int head_dim, int n_heads)
+{
+    int h = blockIdx.x;
+    int i = threadIdx.x;
+    if (h >= n_heads || i >= head_dim) return;
+    float gv = g[h];
+    // softplus(x) = log1p(exp(x)); linear for large x where exp(x) overflows.
+    float sp = gv > 20.0f ? gv : log1pf(expf(gv));
+    out[h * head_dim + i] = attn[h * head_dim + i] * sp;
+}
+"#;
+
+const KV_COMPACT_RING_SRC: &str = r#"
+extern "C" __global__ void butter_kv_compact_ring(
+    const float* __restrict__ src, float* __restrict__ dst,
+    int n_kv_heads, int src_cap, int hd, int ring_cap, int copy_start, int copy_len)
+{
+    long i = (long)blockIdx.x * blockDim.x + threadIdx.x;
+    long total = (long)n_kv_heads * copy_len * hd;
+    if (i >= total) return;
+    int d = (int)(i % hd);
+    long t = i / hd;
+    int c = (int)(t % copy_len);   // 0..copy_len
+    int h = (int)(t / copy_len);   // kv head
+    int src_pos = copy_start + c;  // absolute position within the scratch
+    int dst_slot = src_pos % ring_cap;
+    dst[((long)h * ring_cap + dst_slot) * hd + d] = src[((long)h * src_cap + src_pos) * hd + d];
+}
+"#;
+
+const MAMBA_SPLIT_V4_SRC: &str = r#"
+#include <cuda_runtime.h>
+
+extern "C" __global__ void mamba_split_proj_v4(
+    const float* __restrict__ proj,
+    float* __restrict__ z_out,
+    float* __restrict__ xbc_out,
+    float* __restrict__ dt_out,
+    int in_proj_out4, int di4, int conv_dim4, int m_nh4, int total4)
+{
+    int idx = (int)(blockIdx.x * blockDim.x + threadIdx.x);
+    if (idx >= total4) return;
+    int row4 = di4 + conv_dim4 + m_nh4;
+    int ti = idx / row4;
+    int ci4 = idx - ti * row4;
+    const float4* __restrict__ src4 = reinterpret_cast<const float4*>(proj);
+    float4 v = src4[(size_t)ti * in_proj_out4 + ci4];
+    if (ci4 < di4) {
+        reinterpret_cast<float4*>(z_out)[(size_t)ti * di4 + ci4] = v;
+    } else if (ci4 < di4 + conv_dim4) {
+        int j = ci4 - di4;
+        reinterpret_cast<float4*>(xbc_out)[(size_t)ti * conv_dim4 + j] = v;
+    } else {
+        int j = ci4 - di4 - conv_dim4;
+        reinterpret_cast<float4*>(dt_out)[(size_t)ti * m_nh4 + j] = v;
+    }
+}
+
+__device__ __forceinline__ float mamba_softplus(float x) {
+    return x > 20.0f ? x : logf(1.0f + expf(x));
+}
+
+extern "C" __global__ void mamba_split_proj_softplus_v4(
+    const float* __restrict__ proj,
+    const float* __restrict__ bias,
+    float* __restrict__ z_out,
+    float* __restrict__ xbc_out,
+    float* __restrict__ dt_out,
+    int in_proj_out4, int di4, int conv_dim4, int m_nh4, int total4)
+{
+    int idx = (int)(blockIdx.x * blockDim.x + threadIdx.x);
+    if (idx >= total4) return;
+    int row4 = di4 + conv_dim4 + m_nh4;
+    int ti = idx / row4;
+    int ci4 = idx - ti * row4;
+    const float4* __restrict__ src4 = reinterpret_cast<const float4*>(proj);
+    float4 v = src4[(size_t)ti * in_proj_out4 + ci4];
+    if (ci4 < di4) {
+        reinterpret_cast<float4*>(z_out)[(size_t)ti * di4 + ci4] = v;
+    } else if (ci4 < di4 + conv_dim4) {
+        int j = ci4 - di4;
+        reinterpret_cast<float4*>(xbc_out)[(size_t)ti * conv_dim4 + j] = v;
+    } else {
+        int j = ci4 - di4 - conv_dim4;
+        int b = j << 2;
+        v.x = mamba_softplus(v.x + bias[b + 0]);
+        v.y = mamba_softplus(v.y + bias[b + 1]);
+        v.z = mamba_softplus(v.z + bias[b + 2]);
+        v.w = mamba_softplus(v.w + bias[b + 3]);
+        reinterpret_cast<float4*>(dt_out)[(size_t)ti * m_nh4 + j] = v;
+    }
+}
+"#;
+
+
+
+
+
+
+
+const MULSCALAR16_SRC: &str = r#"
+#include <cuda_fp16.h>
+extern "C" __global__ void butter_mul_scalar_f16(const __half* __restrict__ a, __half* __restrict__ out, float sc, long n){
+    long i=(long)blockIdx.x*blockDim.x+threadIdx.x; long stride=(long)gridDim.x*blockDim.x;
+    for(;i<n;i+=stride) out[i]=__float2half(__half2float(a[i])*sc);
+}
+"#;
+
+const RELU2_SCALE_BF16_SRC: &str = r#"
+#include <cuda_bf16.h>
+extern "C" __global__ void relu2_scale_bf16(
+    const __nv_bfloat16* __restrict__ inp,
+    __nv_bfloat16*       __restrict__ out,
+    int n, float scale)
+{
+    int i = (int)(blockIdx.x * blockDim.x + threadIdx.x);
+    if (i < n) {
+        float v = __bfloat162float(inp[i]);
+        if (v < 0.f) v = 0.f;
+        // bf16 range (~3e38) holds relu^2 for this model; no f16-style clamp needed.
+        out[i] = __float2bfloat16(v * v * scale);
+    }
+}
+"#;
+
+const ROPE_YARN_PARTIAL_MANY_SRC: &str = r#"
+extern "C" __global__ void butter_rope_yarn_partial_many(
+    const float* __restrict__ qk, float* __restrict__ out,
+    const unsigned int* __restrict__ positions,
+    int head_dim, int n_rot, int n_heads,
+    float theta_base, float freq_scale, float ext_factor, float attn_factor,
+    float corr_low, float corr_high)
+{
+    int t = blockIdx.x;
+    int h = blockIdx.y;
+    int tid = threadIdx.z;
+    int half_rot = n_rot / 2;
+    int position = (int)positions[t];
+    long row_off = ((long)t * n_heads + h) * head_dim;
+    const float* row_in  = qk  + row_off;
+    float*       row_out = out + row_off;
+
+    if (tid < half_rot) {
+        int j = tid;
+        float theta_scale  = powf(theta_base, -2.0f / (float)n_rot);
+        float theta_extrap = (float)position * powf(theta_scale, (float)j);
+        float theta_interp = freq_scale * theta_extrap;
+        float theta        = theta_interp;
+        float mscale       = attn_factor;
+        if (ext_factor != 0.0f) {
+            float y        = ((float)j - corr_low) / fmaxf(0.001f, corr_high - corr_low);
+            float ramp     = 1.0f - fminf(1.0f, fmaxf(0.0f, y));
+            float ramp_mix = ramp * ext_factor;
+            theta  = theta_interp * (1.0f - ramp_mix) + theta_extrap * ramp_mix;
+        }
+        float cos_t = cosf(theta) * mscale;
+        float sin_t = sinf(theta) * mscale;
+        float x0 = row_in[j];
+        float x1 = row_in[j + half_rot];
+        row_out[j]            = x0 * cos_t - x1 * sin_t;
+        row_out[j + half_rot] = x0 * sin_t + x1 * cos_t;
+    } else {
+        int p   = tid - half_rot;
+        int idx = n_rot + 2 * p;
+        if (idx < head_dim) {
+            row_out[idx]     = row_in[idx];
+            row_out[idx + 1] = row_in[idx + 1];
+        }
+    }
+}
+"#;
+
+const ROPE_YARN_PARTIAL_POSBUF_SRC: &str = r#"
+extern "C" __global__ void butter_rope_yarn_partial_posbuf(
+    const float* __restrict__ qk, float* __restrict__ out,
+    const unsigned int* __restrict__ pos_buf,
+    int head_dim, int n_rot,
+    float theta_base, float freq_scale, float ext_factor, float attn_factor,
+    float corr_low, float corr_high)
+{
+    int position = (int)pos_buf[0];
+    int h = blockIdx.x;
+    int tid = threadIdx.y;
+    int half_rot = n_rot / 2;
+    const float* row_in  = qk  + (long)h * head_dim;
+    float*       row_out = out + (long)h * head_dim;
+
+    if (tid < half_rot) {
+        int j = tid;
+        float theta_scale  = powf(theta_base, -2.0f / (float)n_rot);
+        float theta_extrap = (float)position * powf(theta_scale, (float)j);
+        float theta_interp = freq_scale * theta_extrap;
+        float theta        = theta_interp;
+        float mscale       = attn_factor;
+        if (ext_factor != 0.0f) {
+            float y        = ((float)j - corr_low) / fmaxf(0.001f, corr_high - corr_low);
+            float ramp     = 1.0f - fminf(1.0f, fmaxf(0.0f, y));
+            float ramp_mix = ramp * ext_factor;
+            theta  = theta_interp * (1.0f - ramp_mix) + theta_extrap * ramp_mix;
+        }
+        float cos_t = cosf(theta) * mscale;
+        float sin_t = sinf(theta) * mscale;
+        float x0 = row_in[j];
+        float x1 = row_in[j + half_rot];
+        row_out[j]            = x0 * cos_t - x1 * sin_t;
+        row_out[j + half_rot] = x0 * sin_t + x1 * cos_t;
+    } else {
+        int p   = tid - half_rot;
+        int idx = n_rot + 2 * p;
+        if (idx < head_dim) {
+            row_out[idx]     = row_in[idx];
+            row_out[idx + 1] = row_in[idx + 1];
+        }
+    }
+}
+"#;
+
+const ROPE_YARN_PARTIAL_SRC: &str = r#"
+extern "C" __global__ void butter_rope_yarn_partial(
+    const float* __restrict__ qk, float* __restrict__ out,
+    int head_dim, int n_rot, int position,
+    float theta_base, float freq_scale, float ext_factor, float attn_factor,
+    float corr_low, float corr_high)
+{
+    int h = blockIdx.x;
+    int tid = threadIdx.y;
+    int half_rot = n_rot / 2;
+    const float* row_in  = qk  + (long)h * head_dim;
+    float*       row_out = out + (long)h * head_dim;
+
+    if (tid < half_rot) {
+        int j = tid;
+        float theta_scale  = powf(theta_base, -2.0f / (float)n_rot);
+        float theta_extrap = (float)position * powf(theta_scale, (float)j);
+        float theta_interp = freq_scale * theta_extrap;
+        float theta        = theta_interp;
+        // attn_factor IS the final magnitude scale (mscale). The reference
+        // stack cancels its kernel's internal 1 + 0.1*ln(1/freq_scale)
+        // re-multiply by pre-dividing on the host, so the net scale equals the
+        // model's stored attention_factor; we just apply it directly.
+        float mscale       = attn_factor;
+        if (ext_factor != 0.0f) {
+            float y        = ((float)j - corr_low) / fmaxf(0.001f, corr_high - corr_low);
+            float ramp     = 1.0f - fminf(1.0f, fmaxf(0.0f, y));
+            float ramp_mix = ramp * ext_factor;
+            theta  = theta_interp * (1.0f - ramp_mix) + theta_extrap * ramp_mix;
+        }
+        float cos_t = cosf(theta) * mscale;
+        float sin_t = sinf(theta) * mscale;
+        float x0 = row_in[j];
+        float x1 = row_in[j + half_rot];
+        row_out[j]            = x0 * cos_t - x1 * sin_t;
+        row_out[j + half_rot] = x0 * sin_t + x1 * cos_t;
+    } else {
+        // Passthrough tail: copy the two contiguous elements at [n_rot + 2p,
+        // n_rot + 2p + 1) unrotated (matches the reference NEOX kernel's
+        // untouched-tail convention).
+        int p   = tid - half_rot;
+        int idx = n_rot + 2 * p;
+        if (idx < head_dim) {
+            row_out[idx]     = row_in[idx];
+            row_out[idx + 1] = row_in[idx + 1];
+        }
+    }
+}
+"#;
+
+const SDPA_DECODE_NBUF_SRC: &str = r#"
+extern "C" __global__ void butter_sdpa_decode_nbuf(
+    const float* __restrict__ q, const float* __restrict__ k, const float* __restrict__ v,
+    float* __restrict__ out, const unsigned int* __restrict__ n_kv_buf,
+    int head_dim, int kv_stride, int heads_per_group, float scale)
+{
+    int q_head = blockIdx.x;
+    int kv_head = q_head / heads_per_group;
+    int tid = threadIdx.x;
+    int lane = tid & 31;
+    int sg = tid >> 5;
+    int ns = blockDim.x >> 5;
+    unsigned int n_kv = n_kv_buf[0];
+
+    __shared__ float tg_max[32];
+    __shared__ float tg_sum[32];
+    __shared__ float tg_out0[1056];
+    __shared__ float tg_out1[1056];
+    __shared__ float tg_out2[1056];
+    __shared__ float tg_out3[1056];
+
+    int q_off = q_head * head_dim;
+    long kv_head_base = (long)kv_head * kv_stride * head_dim;
+    int d0 = lane * 4;
+
+    float qs[4];
+    #pragma unroll
+    for (int i = 0; i < 4; i++) qs[i] = q[q_off + d0 + i] * scale;
+
+    float run_max = -3.4028235e38f;
+    float run_sum = 0.0f;
+    float os[4] = {0.0f, 0.0f, 0.0f, 0.0f};
+
+    for (unsigned int t = sg; t < n_kv; t += ns) {
+        long base = kv_head_base + (long)t * head_dim;
+        long kv0 = base + d0;
+        float partial = 0.0f;
+        #pragma unroll
+        for (int i = 0; i < 4; i++) partial += qs[i] * k[kv0 + i];
+        for (int mask = 16; mask > 0; mask >>= 1) partial += __shfl_xor_sync(0xffffffffu, partial, mask);
+        float score = partial;
+        float new_max = fmaxf(score, run_max);
+        float factor = expf(run_max - new_max);
+        float weight = expf(score - new_max);
+        run_sum = run_sum * factor + weight;
+        run_max = new_max;
+        #pragma unroll
+        for (int i = 0; i < 4; i++) os[i] = os[i] * factor + weight * v[kv0 + i];
+    }
+
+    if (lane == 0) { tg_max[sg] = run_max; tg_sum[sg] = run_sum; }
+    __syncthreads();
+    if (sg == 0) {
+        float g_max_raw = (lane < ns) ? tg_max[lane] : -3.4028235e38f;
+        float g_max = g_max_raw;
+        for (int mask = 16; mask > 0; mask >>= 1) g_max = fmaxf(g_max, __shfl_xor_sync(0xffffffffu, g_max, mask));
+        float g_sum_in = (lane < ns) ? tg_sum[lane] * expf(g_max_raw - g_max) : 0.0f;
+        float g_sum = g_sum_in;
+        for (int mask = 16; mask > 0; mask >>= 1) g_sum += __shfl_xor_sync(0xffffffffu, g_sum, mask);
+        if (lane == 0) { tg_max[0] = g_max; tg_sum[0] = g_sum; }
+    }
+    __syncthreads();
+    float g_max = tg_max[0];
+    float g_sum = tg_sum[0];
+    float rescale = (g_sum > 0.0f) ? expf(run_max - g_max) / g_sum : 0.0f;
+    int stride = ns + 1;
+    int idx = lane * stride + sg;
+    tg_out0[idx] = os[0] * rescale;
+    tg_out1[idx] = os[1] * rescale;
+    tg_out2[idx] = os[2] * rescale;
+    tg_out3[idx] = os[3] * rescale;
+    __syncthreads();
+    if (sg == 0) {
+        float so0 = 0.0f, so1 = 0.0f, so2 = 0.0f, so3 = 0.0f;
+        for (int g = 0; g < ns; g++) {
+            int ri = lane * stride + g;
+            so0 += tg_out0[ri];
+            so1 += tg_out1[ri];
+            so2 += tg_out2[ri];
+            so3 += tg_out3[ri];
+        }
+        int out_off = q_off + d0;
+        out[out_off]     = so0;
+        out[out_off + 1] = so1;
+        out[out_off + 2] = so2;
+        out[out_off + 3] = so3;
+    }
+}
+"#;
+
+const WRITE_U32_SRC: &str = r#"
+extern "C" __global__ void butter_write_u32(unsigned int* __restrict__ buf, unsigned int val)
+{
+    buf[0] = val;
+}
+"#;
+
+fn marlin_f32_to_f16(f: f32) -> u16 {
+    let x = f.to_bits();
+    let sign = ((x >> 16) & 0x8000) as u16;
+    let e = ((x >> 23) & 0xff) as i32 - 112; // 127 - 15
+    if e <= 0 { return sign; }
+    if e >= 0x1f { return sign | 0x7c00; }
+    let m = (x >> 13) & 0x3ff;
+    let round = (x >> 12) & 1;
+    let v = ((e as u32) << 10) | m;
+    sign | ((v + round) as u16)
+}
+
+/// Elementwise `out = a * scale` for an f16 tensor (raw CUDA). Used to undo a
+/// /256 pre-scale that keeps an f16 intermediate inside the f16 range.
+pub fn mul_scalar_f16(dev: &dyn Device, a: &Tensor, scale: f32) -> Result<Tensor> {
+    let out = Tensor::empty(dev, a.shape.clone(), DType::F16)?;
+    let n = a.elem_count();
+    dev.dispatch_raw_cuda(MULSCALAR16_SRC, "mul_scalar_f16.cu", "butter_mul_scalar_f16",
+        &[(a.buffer.as_ref(), a.offset), (out.buffer.as_ref(), 0)],
+        &[scale.to_le_bytes().to_vec(), (n as i64).to_le_bytes().to_vec()],
+        [(n as u32).div_ceil(256).max(1), 1, 1], [256, 1, 1], 0, false)?;
+    Ok(out)
+}
+
+
+/// Fused raw-float4 causal conv1d+SiLU that writes the post-conv x/B/C split
+/// surfaces directly, deleting the temp [s,conv_dim] conv output + the split
+/// launch. Equivalent to conv1d_causal_prefill + mamba_split_conv. Escape with
+/// NEMOTRON_CONV_PREFILL_SPLIT_OFF=1 (falls back to that pair).
+pub fn conv1d_causal_prefill_split(
+    dev: &dyn Device,
+    xbc_in: &Tensor,
+    w: &Tensor,
+    bias: &Tensor,
+    s: usize,
+    conv_dim: usize,
+    di: usize,
+    ng_ds: usize,
+    kc: usize,
+) -> Result<(Tensor, Tensor, Tensor)> {
+    let raw_ok = std::env::var("NEMOTRON_CONV_PREFILL_RAW_OFF").is_err()
+        && std::env::var("NEMOTRON_CONV_PREFILL_SPLIT_OFF").is_err()
+        && xbc_in.dtype == DType::F32 && w.dtype == DType::F32 && bias.dtype == DType::F32
+        && xbc_in.offset % 16 == 0 && w.offset % 16 == 0 && bias.offset % 16 == 0
+        && di + 2 * ng_ds == conv_dim
+        && di % 4 == 0 && ng_ds % 4 == 0 && conv_dim % 4 == 0;
+    if raw_ok {
+        let x_out = Tensor::empty(dev, vec![s * di], DType::F32)?;
+        let b_out = Tensor::empty(dev, vec![s * ng_ds], DType::F32)?;
+        let c_out = Tensor::empty(dev, vec![s * ng_ds], DType::F32)?;
+        let i = |v: i32| v.to_le_bytes().to_vec();
+        let total4 = (s * conv_dim / 4) as u32;
+        if dev.dispatch_raw_cuda(
+            CONV1D_PREFILL_V4_SRC,
+            "conv1d_prefill_v4.cu",
+            "conv1d_causal_prefill_split_v4",
+            &[
+                (xbc_in.buffer.as_ref(), xbc_in.offset),
+                (w.buffer.as_ref(), w.offset),
+                (bias.buffer.as_ref(), bias.offset),
+                (x_out.buffer.as_ref(), 0),
+                (b_out.buffer.as_ref(), 0),
+                (c_out.buffer.as_ref(), 0),
+            ],
+            &[
+                i((conv_dim / 4) as i32),
+                i((di / 4) as i32),
+                i((ng_ds / 4) as i32),
+                i(kc as i32),
+                i(total4 as i32),
+            ],
+            [total4.div_ceil(256), 1, 1],
+            [256, 1, 1],
+            0,
+            false,
+        ).is_ok() {
+            return Ok((x_out, b_out, c_out));
+        }
+    }
+    let y = conv1d_causal_prefill(dev, xbc_in, w, bias, s, conv_dim, kc)?;
+    mamba_split_conv(dev, &y, s, conv_dim, di, ng_ds)
+}
+
+/// Extract `[s, width]` columns from row-major `src [s, stride]` at column offset `col_off`.
+/// Returns `dst [s * width]`, a contiguous slab. Used to carve z, xbc, dt_raw out of
+/// the [s, in_proj_out] projection matrix on device (NEMOTRON_CONV_DEVICE path).
+
+/// Extract a `[rows, width]` column range starting at `col_off` from every
+/// row of a `[rows, stride]` row-major buffer (`stride >= col_off + width`).
+/// Used to split Laguna's fused `[T, n_q_rows+2*n_kv_rows+n_g_rows]`
+/// QKVG-concat batched-prefill projection back into its Q/K/V/gate column
+/// ranges (and, in f16, to split a Marlin/grouped-MoE-GEMM's fused
+/// `[gate;up]` output without a cast first). Originally added as a
+/// workaround for a `strided_col_copy` launch-geometry bug (fixed grid
+/// dropped the tail whenever `rows*width` wasn't a multiple of 64, true for
+/// Laguna's gate column range, whose width is the per-layer head count of 48
+/// or 72, neither 64-aligned, times an arbitrary chunk length).
+/// `strided_col_copy` now div_ceils its grid and guards in-kernel, so this
+/// raw CUDA path is kept as-is only to avoid touching a call site mid-flight
+/// elsewhere; this kernel's grid is sized to cover every element regardless
+/// of `rows*width`'s divisibility. f32 and f16 (dispatched on `src.dtype`);
+/// any other dtype errors.
+pub fn extract_cols(dev: &dyn Device, src: &Tensor, rows: usize, stride: usize, col_off: usize, width: usize) -> Result<Tensor> {
+    let (elem_bytes, cuda_src, prog_name, fn_name) = match src.dtype {
+        DType::F32 => (DType::F32.size_bytes(), EXTRACT_COLS_SRC, "extract_cols.cu", "butter_extract_cols"),
+        DType::F16 => (DType::F16.size_bytes(), EXTRACT_COLS_F16_SRC, "extract_cols_f16.cu", "butter_extract_cols_f16"),
+        other => return Err(Error::Msg(format!("extract_cols: unsupported dtype {other:?} (f32/f16 only)"))),
+    };
+    if rows != 0 && width != 0 {
+        let need = (rows - 1) * stride + col_off + width; // max read is src[need - 1]
+        let have = src.buffer.len() / elem_bytes;
+        if need > have {
+            return Err(Error::Msg(format!(
+                "extract_cols OOB: rows={rows} stride={stride} col_off={col_off} width={width} \
+                 needs {need} {:?} elems but bound buffer holds {have}", src.dtype
+            )));
+        }
+    }
+    let dst = Tensor::empty(dev, vec![rows * width], src.dtype)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let total = (rows * width) as u32;
+    dev.dispatch_raw_cuda(
+        cuda_src, prog_name, fn_name,
+        &[(src.buffer.as_ref(), src.offset), (dst.buffer.as_ref(), 0)],
+        &[i(rows as i32), i(stride as i32), i(col_off as i32), i(width as i32)],
+        [total.div_ceil(256).max(1), 1, 1], [256, 1, 1], 0, false,
+    )?;
+    Ok(dst)
+}
+
+/// Batched softplus + tiled bias: `dst[ti*n + hi] = softplus(src[ti*n + hi] + bias[hi])`.
+/// Converts the raw `[s, m_nh]` dt_raw tensor + dt_bias to `dt_all [s, m_nh]` on device.
+/// (NEMOTRON_CONV_DEVICE path, replaces the CPU softplus loop.)
+
+/// Device argmax of an f32 vector: returns a 1-element u32 tensor holding the
+/// index of the maximum (ties: lowest index). Greedy decode downloads these 4
+/// bytes instead of the whole logit row.
+pub fn argmax_f32_device(dev: &dyn Device, x: &Tensor) -> Result<Tensor> {
+    let n = x.elem_count();
+    let out = Tensor::empty(dev, vec![1], DType::U32)?;
+    let i = |v: i32| v.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(
+        ARGMAX_F32_SRC, "argmax_f32.cu", "butter_argmax_f32",
+        &[(x.buffer.as_ref(), x.offset), (out.buffer.as_ref(), 0)],
+        &[i(n as i32)],
+        [1, 1, 1], [1024, 1, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+/// Batched MoE expert gather GEMV, plain (no fused activation): one
+/// `[top_k*inter]` output from the contiguous `[n_exp*inter, hid]` Q4 stack.
+/// The SwiGLU MoE path calls this for the gate and up stacks, then applies
+/// the activation elementwise before the down gather. F16 per-block scales.
+
+#[allow(clippy::type_complexity)]
+pub fn moe_route_sort_device_rowpos(
+    dev: &dyn Device, gate_logits: &Tensor, bias: &Tensor, n_exp: usize, top_k: usize, scale: f32,
+) -> Result<(Tensor, Tensor, Tensor, Tensor, Tensor)> {
+    let s = gate_logits.elem_count() / n_exp;
+    let mt = s * top_k;
+    let idx = Tensor::empty(dev, vec![mt], DType::U32)?;
+    let wt = Tensor::empty(dev, vec![mt], DType::F32)?;
+    let counts = Tensor::new(dev.alloc_zeroed(n_exp * 4).map_err(|e| Error::Msg(format!("{e:?}")))?, vec![n_exp], DType::U32);
+    let offsets = Tensor::empty(dev, vec![n_exp + 1], DType::U32)?;
+    let cursor = Tensor::empty(dev, vec![n_exp], DType::U32)?;
+    let sorted_tok = Tensor::empty(dev, vec![mt], DType::U32)?;
+    let sorted_wt = Tensor::empty(dev, vec![mt], DType::F32)?;
+    let rowpos = Tensor::empty(dev, vec![mt], DType::U32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    let blk = if n_exp >= 256 { 256 } else { n_exp.max(32) } as u32;
+    dev.dispatch_raw_cuda(MOE_BATCHED_ROUTER_SRC, "moe_route_sort.cu", "moe_batched_router",
+        &[(gate_logits.buffer.as_ref(), gate_logits.offset), (bias.buffer.as_ref(), bias.offset),
+          (idx.buffer.as_ref(), 0), (wt.buffer.as_ref(), 0)],
+        &[i(s as i32), i(n_exp as i32), i(top_k as i32), f(scale)],
+        [s as u32, 1, 1], [blk, 1, 1], 0, false)?;
+    dev.dispatch_raw_cuda(MOE_HIST_SRC, "moe_hist.cu", "moe_hist",
+        &[(idx.buffer.as_ref(), 0), (counts.buffer.as_ref(), 0)],
+        &[i(mt as i32)], [(mt as u32).div_ceil(256), 1, 1], [256, 1, 1], 0, false)?;
+    dev.dispatch_raw_cuda(MOE_PREFIX_SRC, "moe_prefix.cu", "moe_prefix",
+        &[(counts.buffer.as_ref(), 0), (offsets.buffer.as_ref(), 0), (cursor.buffer.as_ref(), 0)],
+        &[i(n_exp as i32)], [1, 1, 1], [32, 1, 1], 0, false)?;
+    let _ = &cursor;
+    dev.dispatch_raw_cuda(MOE_SCATTER_SORT_SRC, "moe_scatter_sort.cu", "moe_scatter_sort_rowpos",
+        &[(idx.buffer.as_ref(), 0), (wt.buffer.as_ref(), 0), (offsets.buffer.as_ref(), 0),
+          (sorted_tok.buffer.as_ref(), 0), (sorted_wt.buffer.as_ref(), 0), (rowpos.buffer.as_ref(), 0)],
+        &[i(mt as i32), i(top_k as i32), i(n_exp as i32)], [((n_exp * 32) as u32).div_ceil(128), 1, 1], [128, 1, 1], 0, false)?;
+    Ok((sorted_tok, sorted_wt, offsets, rowpos, wt))
+}
+
+
+/// Convert per-16 NVFP4 block-scale VALUES `[ngroup, n]` (row-major; value =
+/// dec_ue4m3(byte)*global) into the marlin S0E5M3 fp8 byte layout the GEMM reads.
+/// Mirrors marlin_permute_scales(group<K) + nvfp4_marlin_process_scales (f16,
+/// scale_factor=1): scale_perm 64-permute, [0,2,1,3] 4-interleave, x2^7, zero-if<2,
+/// reinterpret f16 bits `<<1`, take the high byte (= S0E5M3). Returns `[ngroup*n]` u8.
+pub fn marlin_s0e5m3_scales(vals: &[f32], ngroup: usize, n: usize) -> Vec<u8> {
+    let total = ngroup * n;
+    assert_eq!(vals.len(), total);
+    assert_eq!(total % 64, 0, "marlin_s0e5m3_scales: ngroup*n must be a multiple of 64");
+    // scale_perm (64): [i + 8*j for i in 0..8 for j in 0..8]
+    let mut sp = [0usize; 64];
+    let mut idx = 0;
+    for i in 0..8 { for j in 0..8 { sp[idx] = i + 8 * j; idx += 1; } }
+    // A: 64-chunk column permute (marlin_permute_scales)
+    let mut s1 = vec![0f32; total];
+    for r in 0..(total / 64) {
+        for j in 0..64 { s1[r * 64 + j] = vals[r * 64 + sp[j]]; }
+    }
+    // B: 4-chunk interleave [0,2,1,3]
+    let il = [0usize, 2, 1, 3];
+    let mut s2 = vec![0f32; total];
+    for r in 0..(total / 4) {
+        for j in 0..4 { s2[r * 4 + j] = s1[r * 4 + il[j]]; }
+    }
+    // C: x2^7, zero-if<2, f16 bits <<1, high byte = S0E5M3
+    let mut out = vec![0u8; total];
+    for p in 0..total {
+        let mut v = s2[p] * 128.0;
+        if v < 2.0 { v = 0.0; }
+        let shifted = marlin_f32_to_f16(v) << 1;
+        out[p] = (shifted >> 8) as u8;
+    }
+    out
+}
+
+
+/// CUDA-graph-capturable fallback of [`sdpa_decode`] for `head_dim == 128`
+/// (Laguna's only head_dim): reads the filled KV length `n_kv` from a
+/// 1-element `u32` device buffer instead of a baked-in launch scalar. Dense
+/// causal only (no sink / sliding-window mask), Laguna's sliding layers
+/// bound the KV walk via the buffer's VALUE (the ring's current filled
+/// length), so a separate window mask isn't needed. f32 only. See the
+/// kernel-source doc above for the algorithm.
+#[allow(clippy::too_many_arguments)]
+pub fn sdpa_decode_nbuf(
+    dev: &dyn Device,
+    q: &Tensor,
+    k: &Tensor,
+    v: &Tensor,
+    head_dim: usize,
+    n_kv_buf: &Tensor,
+    kv_stride: u32,
+    heads_per_group: u32,
+    scale: f32,
+) -> Result<Tensor> {
+    if head_dim != 128 {
+        return Err(Error::Msg(format!("sdpa_decode_nbuf: only head_dim 128 is implemented, got {head_dim}")));
+    }
+    let n_q_heads = q.elem_count() / head_dim;
+    let out = Tensor::empty(dev, vec![n_q_heads, head_dim], DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(
+        SDPA_DECODE_NBUF_SRC, "sdpa_decode_nbuf.cu", "butter_sdpa_decode_nbuf",
+        &[
+            (q.buffer.as_ref(), q.offset),
+            (k.buffer.as_ref(), k.offset),
+            (v.buffer.as_ref(), v.offset),
+            (out.buffer.as_ref(), 0),
+            (n_kv_buf.buffer.as_ref(), n_kv_buf.offset),
+        ],
+        &[i(head_dim as i32), i(kv_stride as i32), i(heads_per_group as i32), f(scale)],
+        [n_q_heads as u32, 1, 1], [1024, 1, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+/// GQA-aware split-K flash-decode SDPA (ported from a reference ML framework's `sdpa_vector_2pass`, head_dim
+/// 128). Pass 1: grid `(n_kv_heads, blocks)`, block `(32, gqa_factor)`: one
+/// simdgroup per Q-head of the group, so the `gqa_factor` heads SHARE each K/V
+/// load (the single-pass kernel re-reads the shared KV head `gqa_factor`×, at
+/// 32K that's the dominant cost). Each block-row strides a `1/blocks` slice of
+/// the KV positions, emitting per-block (max, sum, partial_o). Pass 2: grid
+/// `(n_q_heads)`, block 1024: online-softmax merge of the `blocks` partials.
+/// `blocks` MUST be a multiple of 32 (pass-2 reducer constraint). Kernel math
+/// is registry-validated (`test_butter_sdpa_decode_2pass_combined`, f32/f16/bf16).
+
+/// DETERMINISTIC fully-on-device scatter-add (no host sync): builds the per-token
+/// CSR (toff, tok_rows) ON DEVICE from `sorted_tok` (hist→prefix→stable scan) then
+/// runs the deterministic moe_scatter_add_det kernel. Same result as the host
+/// `moe_scatter_add_det` but host-sync-free → graph-capturable.
+#[allow(clippy::too_many_arguments)]
+/// x2 variant of moe_scatter_add_topk_det_f16: 2 hidden-dims/thread via __half2
+/// vectorized loads. Exact; falls back to the x1 kernel on odd hid.
+pub fn moe_scatter_add_topk_det_f16_x2(
+    dev: &dyn Device,
+    dn: &Tensor,
+    rowpos: &Tensor,
+    wts_tok: &Tensor,
+    acc: &Tensor,
+    s: usize,
+    top_k: usize,
+    hid: usize,
+    unscale: f32,
+) -> Result<()> {
+    if hid % 2 != 0 {
+        return moe_scatter_add_topk_det_f16(dev, dn, rowpos, wts_tok, acc, s, top_k, hid, unscale);
+    }
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let pairs = (hid as u32).div_ceil(2);
+    dev.dispatch_raw_cuda(
+        MOE_SCATTER_ADD_DET_SRC,
+        "moe_scatter_add_det.cu",
+        "moe_scatter_add_topk_det_f16_x2",
+        &[(dn.buffer.as_ref(), 0), (rowpos.buffer.as_ref(), rowpos.offset),
+          (wts_tok.buffer.as_ref(), wts_tok.offset), (acc.buffer.as_ref(), 0)],
+        &[i(s as i32), i(top_k as i32), i(hid as i32), unscale.to_le_bytes().to_vec()],
+        [s as u32, pairs.div_ceil(128), 1], [128, 1, 1], 0, false)
+}
+
+
+#[allow(clippy::too_many_arguments)]
+pub fn moe_scatter_add_topk_det_f16(
+    dev: &dyn Device,
+    dn: &Tensor,
+    rowpos: &Tensor,
+    wt_token_major: &Tensor,
+    acc: &Tensor,
+    s: usize,
+    top_k: usize,
+    hid: usize,
+    unscale: f32,
+) -> Result<()> {
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(
+        MOE_SCATTER_ADD_DET_SRC,
+        "moe_scatter_add_det.cu",
+        "moe_scatter_add_topk_det_f16",
+        &[
+            (dn.buffer.as_ref(), 0),
+            (rowpos.buffer.as_ref(), rowpos.offset),
+            (wt_token_major.buffer.as_ref(), wt_token_major.offset),
+            (acc.buffer.as_ref(), 0),
+        ],
+        &[i(s as i32), i(top_k as i32), i(hid as i32), f(unscale)],
+        [s as u32, (hid as u32).div_ceil(128), 1],
+        [128, 1, 1],
+        0,
+        false,
+    )?;
+    Ok(())
+}
+
+/// Expand per-expert CSR offsets `[n_exp+1]` into a per-row expert-id vector
+/// `[mt]` on device: `idx[r] = e` for every `r` in `[off[e], off[e+1])`.
+/// Graph-safe replacement for the host `triples.map(|(e,_,_)| e)` build -- lets
+/// the on-device Marlin W4A16 MoE path get per-row expert ids without a host dl.
+
+/// relu(x)^2 * scale for a BF16 tensor: reads BF16, squares in FP32, writes BF16.
+/// Unlike the f16 variant there is no square-safe clamp (bf16 exponent range holds
+/// the squared activations), so the FFN intermediate is not range-distorted.
+pub fn relu2_scale_bf16(dev: &dyn Device, x: &Tensor, scale: f32) -> Result<Tensor> {
+    let n = x.elem_count();
+    let out = Tensor::empty(dev, x.shape.clone(), DType::BF16)?;
+    let block = 256u32;
+    let grid = (n as u32).div_ceil(block);
+    dev.dispatch_raw_cuda(RELU2_SCALE_BF16_SRC, "relu2_scale_bf16.cu", "relu2_scale_bf16",
+        &[(x.buffer.as_ref(), x.offset), (out.buffer.as_ref(), 0)],
+        &[(n as i32).to_le_bytes().to_vec(), scale.to_le_bytes().to_vec()],
+        [grid, 1, 1], [block, 1, 1], 0, false)?;
+    Ok(out)
+}
+
+
+/// YaRN-scaled partial-rotary RoPE (NEOX pairing) for a `[n_heads, head_dim]`
+/// Q or K tensor at sequence `position`. `n_rot` is the number of rotated
+/// dims (`<= head_dim`; the rest pass through unrotated). Pass `n_rot ==
+/// head_dim`, `freq_scale = 1.0`, `ext_factor = 0.0`, `attn_factor = 1.0` for
+/// plain (non-YaRN) RoPE, `beta_fast`/`beta_slow`/`n_orig_ctx` are then
+/// unused (the ramp contributes zero once `ext_factor` is zero). f32 only.
+#[allow(clippy::too_many_arguments)]
+pub fn rope_yarn_partial(
+    dev: &dyn Device,
+    qk: &Tensor,
+    position: u32,
+    n_rot: u32,
+    theta_base: f32,
+    freq_scale: f32,
+    ext_factor: f32,
+    attn_factor: f32,
+    beta_fast: f32,
+    beta_slow: f32,
+    n_orig_ctx: u32,
+) -> Result<Tensor> {
+    let head_dim = *qk.shape.last().ok_or_else(|| Error::Msg("rope_yarn_partial: scalar input".into()))?;
+    let n_heads = qk.elem_count() / head_dim;
+    if n_rot == 0 || n_rot as usize > head_dim || n_rot % 2 != 0 {
+        return Err(Error::Msg(format!(
+            "rope_yarn_partial: n_rot {n_rot} invalid for head_dim {head_dim}"
+        )));
+    }
+    // ggml_rope_yarn_corr_dims: the [low, high] band (in rotated-pair-index
+    // units) where the rotary frequency ramps from interpolated (YaRN-scaled)
+    // to extrapolated (raw). Degenerate but harmless when ext_factor == 0.
+    let corr_dim = |n_rotations: f32| -> f32 {
+        n_rot as f32 * (n_orig_ctx as f32 / (n_rotations * 2.0 * std::f32::consts::PI)).ln()
+            / (2.0 * theta_base.ln())
+    };
+    let corr_low = corr_dim(beta_fast).floor().max(0.0);
+    let corr_high = corr_dim(beta_slow).ceil().min(n_rot as f32 - 1.0);
+
+    let out = Tensor::empty(dev, qk.shape.clone(), DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    let half = (head_dim / 2) as u32;
+    dev.dispatch_raw_cuda(
+        ROPE_YARN_PARTIAL_SRC, "rope_yarn_partial.cu", "butter_rope_yarn_partial",
+        &[(qk.buffer.as_ref(), qk.offset), (out.buffer.as_ref(), 0)],
+        &[
+            i(head_dim as i32), i(n_rot as i32), i(position as i32),
+            f(theta_base), f(freq_scale), f(ext_factor), f(attn_factor),
+            f(corr_low), f(corr_high),
+        ],
+        [n_heads as u32, 1, 1], [1, half, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+
+/// CUDA-graph-capturable variant of [`rope_yarn_partial`]: identical math,
+/// but `position` is read from a 1-element `u32` device buffer at kernel
+/// runtime instead of being baked in as a launch-time scalar. A captured
+/// graph replays the exact same recorded launch every token, so any value
+/// that changes per token (the sequence position) has to live behind a
+/// stable pointer whose CONTENTS the caller updates before each replay,
+/// rather than as an argument frozen at capture time. `corr_low`/`corr_high`
+/// depend only on the (per-layer-type, capture-time-constant) RoPE constants,
+/// so those stay ordinary scalars. Used by Laguna's decode path under
+/// `BUTTER_LAGUNA_GRAPH=1`; the plain scalar-position `rope_yarn_partial` above
+/// remains the default (eager, non-graph) path.
+#[allow(clippy::too_many_arguments)]
+pub fn rope_yarn_partial_posbuf(
+    dev: &dyn Device,
+    qk: &Tensor,
+    pos_buf: &Tensor,
+    n_rot: u32,
+    theta_base: f32,
+    freq_scale: f32,
+    ext_factor: f32,
+    attn_factor: f32,
+    beta_fast: f32,
+    beta_slow: f32,
+    n_orig_ctx: u32,
+) -> Result<Tensor> {
+    let head_dim = *qk.shape.last().ok_or_else(|| Error::Msg("rope_yarn_partial_posbuf: scalar input".into()))?;
+    let n_heads = qk.elem_count() / head_dim;
+    if n_rot == 0 || n_rot as usize > head_dim || n_rot % 2 != 0 {
+        return Err(Error::Msg(format!(
+            "rope_yarn_partial_posbuf: n_rot {n_rot} invalid for head_dim {head_dim}"
+        )));
+    }
+    // Same YaRN corr-dims derivation as `rope_yarn_partial`, a function of
+    // the RoPE constants only, never the runtime position.
+    let corr_dim = |n_rotations: f32| -> f32 {
+        n_rot as f32 * (n_orig_ctx as f32 / (n_rotations * 2.0 * std::f32::consts::PI)).ln()
+            / (2.0 * theta_base.ln())
+    };
+    let corr_low = corr_dim(beta_fast).floor().max(0.0);
+    let corr_high = corr_dim(beta_slow).ceil().min(n_rot as f32 - 1.0);
+
+    let out = Tensor::empty(dev, qk.shape.clone(), DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    let half = (head_dim / 2) as u32;
+    dev.dispatch_raw_cuda(
+        ROPE_YARN_PARTIAL_POSBUF_SRC, "rope_yarn_partial_posbuf.cu", "butter_rope_yarn_partial_posbuf",
+        &[
+            (qk.buffer.as_ref(), qk.offset),
+            (out.buffer.as_ref(), 0),
+            (pos_buf.buffer.as_ref(), pos_buf.offset),
+        ],
+        &[
+            i(head_dim as i32), i(n_rot as i32),
+            f(theta_base), f(freq_scale), f(ext_factor), f(attn_factor),
+            f(corr_low), f(corr_high),
+        ],
+        [n_heads as u32, 1, 1], [1, half, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+
+/// Batched [`rope_yarn_partial`]: applies YaRN-scaled partial-rotary RoPE to
+/// ALL `T` rows of a `[T, n_heads, head_dim]` Q or K tensor in ONE dispatch,
+/// each row `t` rotated at `positions[t]` (one absolute sequence position per
+/// token row) instead of a single scalar `position` shared by every row.
+/// Replaces the T-loop of per-token `rope_yarn_partial` calls in the Laguna
+/// prefill attention path. Same math/params as `rope_yarn_partial` (pass
+/// `n_rot == head_dim`, `freq_scale = 1.0`, `ext_factor = 0.0`,
+/// `attn_factor = 1.0` for plain, non-YaRN RoPE: Laguna's sliding-window
+/// layers use exactly this). Grid `[T, n_heads, 1]`, block `[1, 1, head_dim/2]`
+/// (covers both the rotated-pair range and the passthrough-tail range, same
+/// as the scalar-position kernel). f32 only.
+#[allow(clippy::too_many_arguments)]
+pub fn rope_yarn_partial_many(
+    dev: &dyn Device,
+    qk: &Tensor,
+    positions: &Tensor,
+    n_heads: usize,
+    n_rot: u32,
+    theta_base: f32,
+    freq_scale: f32,
+    ext_factor: f32,
+    attn_factor: f32,
+    beta_fast: f32,
+    beta_slow: f32,
+    n_orig_ctx: u32,
+) -> Result<Tensor> {
+    let head_dim = *qk.shape.last().ok_or_else(|| Error::Msg("rope_yarn_partial_many: scalar input".into()))?;
+    let t = qk.elem_count() / (n_heads * head_dim);
+    if positions.elem_count() != t {
+        return Err(Error::Msg(format!(
+            "rope_yarn_partial_many: positions len {} != T {t}", positions.elem_count()
+        )));
+    }
+    if n_rot == 0 || n_rot as usize > head_dim || n_rot % 2 != 0 {
+        return Err(Error::Msg(format!(
+            "rope_yarn_partial_many: n_rot {n_rot} invalid for head_dim {head_dim}"
+        )));
+    }
+    // Same YaRN corr-dims derivation as `rope_yarn_partial`, depends only on
+    // the (per-layer-type, chunk-invariant) RoPE constants, never on position.
+    let corr_dim = |n_rotations: f32| -> f32 {
+        n_rot as f32 * (n_orig_ctx as f32 / (n_rotations * 2.0 * std::f32::consts::PI)).ln()
+            / (2.0 * theta_base.ln())
+    };
+    let corr_low = corr_dim(beta_fast).floor().max(0.0);
+    let corr_high = corr_dim(beta_slow).ceil().min(n_rot as f32 - 1.0);
+
+    let out = Tensor::empty(dev, qk.shape.clone(), DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let f = |x: f32| x.to_le_bytes().to_vec();
+    let half = (head_dim / 2) as u32;
+    dev.dispatch_raw_cuda(
+        ROPE_YARN_PARTIAL_MANY_SRC, "rope_yarn_partial_many.cu", "butter_rope_yarn_partial_many",
+        &[
+            (qk.buffer.as_ref(), qk.offset),
+            (out.buffer.as_ref(), 0),
+            (positions.buffer.as_ref(), positions.offset),
+        ],
+        &[
+            i(head_dim as i32), i(n_rot as i32), i(n_heads as i32),
+            f(theta_base), f(freq_scale), f(ext_factor), f(attn_factor),
+            f(corr_low), f(corr_high),
+        ],
+        [t as u32, n_heads as u32, 1], [1, 1, half], 0, false,
+    )?;
+    Ok(out)
+}
+
+/// Write a single `u32` scalar into an existing 1-element device buffer, in
+/// place, without reallocating. Laguna's CUDA-graph decode path
+/// (`BUTTER_LAGUNA_GRAPH=1`) keeps a small persistent "workspace" of such
+/// buffers (token id, RoPE position, KV length) whose device POINTERS get
+/// captured into the graph and must never move; only their contents change
+/// from token to token. Call this OUTSIDE any graph-capture region, it must
+/// run eagerly every step, right before the (captured or replayed) decode
+/// work that reads the buffer.
+
+/// Write a single `u32` scalar into an existing 1-element device buffer, in
+/// place, without reallocating. Laguna's CUDA-graph decode path
+/// (`BUTTER_LAGUNA_GRAPH=1`) keeps a small persistent "workspace" of such
+/// buffers (token id, RoPE position, KV length) whose device POINTERS get
+/// captured into the graph and must never move; only their contents change
+/// from token to token. Call this OUTSIDE any graph-capture region, it must
+/// run eagerly every step, right before the (captured or replayed) decode
+/// work that reads the buffer.
+pub fn write_u32(dev: &dyn Device, buf: &Tensor, val: u32) -> Result<()> {
+    dev.dispatch_raw_cuda(
+        WRITE_U32_SRC, "write_u32.cu", "butter_write_u32",
+        &[(buf.buffer.as_ref(), buf.offset)],
+        &[val.to_le_bytes().to_vec()],
+        [1, 1, 1], [1, 1, 1], 0, false,
+    )
+}
+
+
+/// Per-head softplus attention gate: `out[h, i] = attn[h, i] * softplus(g[h])`
+/// for `attn` `[n_heads, head_dim]` and `g` `[n_heads]` (one raw gate scalar
+/// per head, broadcast over `head_dim`). Laguna's XS.2/S.2 attention-output
+/// gate, applied before `o_proj`. f32 only.
+pub fn gate_softplus_mul_perhead(dev: &dyn Device, attn: &Tensor, g: &Tensor) -> Result<Tensor> {
+    let head_dim = *attn.shape.last().ok_or_else(|| Error::Msg("gate_softplus_mul_perhead: scalar attn".into()))?;
+    let n_heads = attn.elem_count() / head_dim;
+    if g.elem_count() != n_heads {
+        return Err(Error::Msg(format!(
+            "gate_softplus_mul_perhead: g has {} elems, expected {n_heads} (one per head)",
+            g.elem_count()
+        )));
+    }
+    let out = Tensor::empty(dev, attn.shape.clone(), DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(
+        GATE_SOFTPLUS_MUL_PERHEAD_SRC, "gate_softplus_mul_perhead.cu", "butter_gate_softplus_mul_perhead",
+        &[(attn.buffer.as_ref(), attn.offset), (g.buffer.as_ref(), g.offset), (out.buffer.as_ref(), 0)],
+        &[i(head_dim as i32), i(n_heads as i32)],
+        [n_heads as u32, 1, 1], [head_dim as u32, 1, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+
+/// Batched [`gate_softplus_mul_perhead`]: `out[t, h, i] = attn[t, h, i] *
+/// softplus(g[t, h])` for `attn` `[T, n_heads, head_dim]` and `g` `[T,
+/// n_heads]` (one raw gate scalar per token-head, broadcast over
+/// `head_dim`). Replaces the T-loop of per-token `gate_softplus_mul_perhead`
+/// calls in the Laguna prefill attention path, applied before `o_proj`,
+/// same as the decode path. f32 only.
+pub fn gate_softplus_mul_perhead_many(dev: &dyn Device, attn: &Tensor, g: &Tensor, n_heads: usize) -> Result<Tensor> {
+    let head_dim = *attn.shape.last().ok_or_else(|| Error::Msg("gate_softplus_mul_perhead_many: scalar attn".into()))?;
+    let rows = attn.elem_count() / head_dim;
+    if g.elem_count() != rows {
+        return Err(Error::Msg(format!(
+            "gate_softplus_mul_perhead_many: g has {} elems, expected {rows} (T*n_heads)",
+            g.elem_count()
+        )));
+    }
+    let _ = n_heads; // rows already folds T*n_heads; kept for call-site clarity/shape docs
+    let out = Tensor::empty(dev, attn.shape.clone(), DType::F32)?;
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(
+        GATE_SOFTPLUS_MUL_PERHEAD_MANY_SRC, "gate_softplus_mul_perhead_many.cu", "butter_gate_softplus_mul_perhead_many",
+        &[(attn.buffer.as_ref(), attn.offset), (g.buffer.as_ref(), g.offset), (out.buffer.as_ref(), 0)],
+        &[i(head_dim as i32)],
+        [rows as u32, 1, 1], [head_dim as u32, 1, 1], 0, false,
+    )?;
+    Ok(out)
+}
+
+/// One-time compaction of a linear prefill K/V scratch buffer into a
+/// fixed-size decode ring: copies the LAST `copy_len` absolute positions
+/// `[copy_start, copy_start+copy_len)` from `src` `[n_kv_heads, src_cap,
+/// head_dim]` into `dst` `[n_kv_heads, ring_cap, head_dim]` at ring slot
+/// `abs_pos % ring_cap`. Used once per sliding-window layer after a Laguna
+/// prefill finishes the whole prompt, to hand off from the prefill-only
+/// linear scratch (`LagunaPrefillScratch`, sized to the prompt) to the
+/// persistent decode ring (`LagunaKvCache`'s fixed `sliding_window`-slot
+/// buffer) so decode can continue exactly where prefill left off. f32 only.
+
+/// One-time compaction of a linear prefill K/V scratch buffer into a
+/// fixed-size decode ring: copies the LAST `copy_len` absolute positions
+/// `[copy_start, copy_start+copy_len)` from `src` `[n_kv_heads, src_cap,
+/// head_dim]` into `dst` `[n_kv_heads, ring_cap, head_dim]` at ring slot
+/// `abs_pos % ring_cap`. Used once per sliding-window layer after a Laguna
+/// prefill finishes the whole prompt, to hand off from the prefill-only
+/// linear scratch (`LagunaPrefillScratch`, sized to the prompt) to the
+/// persistent decode ring (`LagunaKvCache`'s fixed `sliding_window`-slot
+/// buffer) so decode can continue exactly where prefill left off. f32 only.
+#[allow(clippy::too_many_arguments)]
+pub fn kv_compact_ring(
+    dev: &dyn Device,
+    src: &Tensor,
+    dst: &Tensor,
+    n_kv_heads: usize,
+    src_cap: usize,
+    head_dim: usize,
+    ring_cap: usize,
+    copy_start: usize,
+    copy_len: usize,
+) -> Result<()> {
+    if copy_len == 0 {
+        return Ok(());
+    }
+    let i = |x: i32| x.to_le_bytes().to_vec();
+    let total = (n_kv_heads * copy_len * head_dim) as u32;
+    dev.dispatch_raw_cuda(
+        KV_COMPACT_RING_SRC, "kv_compact_ring.cu", "butter_kv_compact_ring",
+        &[(src.buffer.as_ref(), src.offset), (dst.buffer.as_ref(), dst.offset)],
+        &[
+            i(n_kv_heads as i32), i(src_cap as i32), i(head_dim as i32),
+            i(ring_cap as i32), i(copy_start as i32), i(copy_len as i32),
+        ],
+        [total.div_ceil(256).max(1), 1, 1], [256, 1, 1], 0, false,
+    )
+}
+
+
+/// Fused raw-float4 in_proj split (z/xbc/dt) WITH dt-bias+softplus folded into
+/// the dt band. Replaces mamba_split_proj + softplus_add_rows. Escape with
+/// NEMOTRON_MAMBA_SPLIT_RAW_OFF=1 or NEMOTRON_MAMBA_SPLIT_SOFTPLUS_OFF=1.
+pub fn mamba_split_proj_softplus(
+    dev: &dyn Device,
+    proj: &Tensor,
+    bias: &Tensor,
+    s: usize,
+    in_proj_out: usize,
+    di: usize,
+    conv_dim: usize,
+    m_nh: usize,
+) -> Result<(Tensor, Tensor, Tensor)> {
+    let raw_ok = std::env::var("NEMOTRON_MAMBA_SPLIT_RAW_OFF").is_err()
+        && std::env::var("NEMOTRON_MAMBA_SPLIT_SOFTPLUS_OFF").is_err()
+        && proj.dtype == DType::F32 && bias.dtype == DType::F32
+        && proj.offset % 16 == 0 && bias.offset % 4 == 0
+        && di % 4 == 0 && conv_dim % 4 == 0 && m_nh % 4 == 0 && in_proj_out % 4 == 0;
+    if raw_ok {
+        let z_out  = Tensor::empty(dev, vec![s * di], DType::F32)?;
+        let xbc_out = Tensor::empty(dev, vec![s * conv_dim], DType::F32)?;
+        let dt_out  = Tensor::empty(dev, vec![s * m_nh], DType::F32)?;
+        let i = |v: i32| v.to_le_bytes().to_vec();
+        let total4 = (s * (di + conv_dim + m_nh) / 4) as u32;
+        if dev.dispatch_raw_cuda(
+            MAMBA_SPLIT_V4_SRC,
+            "mamba_split_v4.cu",
+            "mamba_split_proj_softplus_v4",
+            &[
+                (proj.buffer.as_ref(), proj.offset),
+                (bias.buffer.as_ref(), bias.offset),
+                (z_out.buffer.as_ref(), 0),
+                (xbc_out.buffer.as_ref(), 0),
+                (dt_out.buffer.as_ref(), 0),
+            ],
+            &[i((in_proj_out / 4) as i32), i((di / 4) as i32), i((conv_dim / 4) as i32), i((m_nh / 4) as i32), i(total4 as i32)],
+            [total4.div_ceil(256), 1, 1],
+            [256, 1, 1],
+            0,
+            false,
+        ).is_ok() {
+            return Ok((z_out, xbc_out, dt_out));
+        }
+    }
+    let (z, xbc, dt_raw) = mamba_split_proj(dev, proj, s, in_proj_out, di, conv_dim, m_nh)?;
+    let dt = softplus_add_rows(dev, &dt_raw, bias, s, m_nh)?;
+    Ok((z, xbc, dt))
+}
+
+
+// ── Fused Mamba conv-output split (Part-B: 3 strided_col_copy → 1 dispatch) ─
+//
+// Splits yc_silu [s, conv_dim] into x [s, di], b [s, ng*ds], c [s, ng*ds]
+// in ONE device dispatch.
+
+/// `sfb_exp_bytes` = per-expert SFB stride in bytes (from `w8a8_packw`).
+#[allow(clippy::too_many_arguments)]
+/// Validated NVFP4 (f16-act) grouped-MoE GEMM (marlin). `a` f16 `[m,K]`, `b` fp4
+/// marlin-tile u32 `[n_exp,K/16,N*2]`, `b_scales` S0E5M3 fp8 (u32-viewed),
+/// `global_scale` f32 `[n_exp]`, `topk_weights` f32 `[m*top_k]`; routing
+/// (`sorted_token_ids`/`expert_ids`/`num_tokens_past_padded`) device i32.
+/// Returns f16 `[m*top_k, N]`. mul_topk_weights + fp32 reduce on.
+#[allow(clippy::too_many_arguments)]
+pub fn moe_marlin_gemm(
+    dev: &dyn Device,
+    a: &Tensor, b: &Tensor, b_scales: &Tensor, global_scale: &Tensor, topk_weights: &Tensor,
+    sorted_token_ids: &Tensor, expert_ids: &Tensor, num_tokens_past_padded: &Tensor,
+    prob_m: usize, prob_n: usize, prob_k: usize, num_experts: usize, top_k: usize,
+    group_size: usize, moe_block: usize, sms: i32,
+) -> Result<Tensor> {
+    let mt = (prob_m * top_k).max(1);
+    let out = Tensor::empty(dev, vec![mt, prob_n.max(1)], DType::F16)?;
+    let num_groups = prob_k / group_size;
+    let ctmp = ((sms as usize) * 4 * moe_block * 256).max(1);
+    let c_tmp = Tensor::new(dev.alloc_zeroed(ctmp * 4)?, vec![ctmp], DType::F32);
+    let wsn = ((sms as usize) * 4).max(1);
+    let ws = Tensor::new(dev.alloc_zeroed(wsn * 4)?, vec![wsn], DType::U32);
+    dev.moe_marlin_gemm(
+        a.buffer.as_ref(), b.buffer.as_ref(), out.buffer.as_ref(), c_tmp.buffer.as_ref(),
+        b_scales.buffer.as_ref(), global_scale.buffer.as_ref(), topk_weights.buffer.as_ref(),
+        sorted_token_ids.buffer.as_ref(), expert_ids.buffer.as_ref(), num_tokens_past_padded.buffer.as_ref(),
+        ws.buffer.as_ref(),
+        moe_block, num_experts, top_k, true, prob_m, prob_n, prob_k, num_groups, group_size, sms, true,
+    )?;
+    Ok(out)
+}
+
+/// Repack one expert's GPTQ-packed `[K/8,N]` u32 weights -> marlin tile `[K/16,N*2]` u32 (device).
+
+/// Repack one expert's GPTQ-packed `[K/8,N]` u32 weights -> marlin tile `[K/16,N*2]` u32 (device).
+pub fn marlin_repack(dev: &dyn Device, b_q: &Tensor, size_k: usize, size_n: usize, sms: i32, max_shared_mem: i32) -> Result<Tensor> {
+    let out_elems = ((size_k / 16) * (size_n * 2)).max(1);
+    let out = Tensor::new(dev.alloc(out_elems * 4)?, vec![out_elems], DType::U32);
+    dev.marlin_repack(b_q.buffer.as_ref(), out.buffer.as_ref(), size_k, size_n, sms, max_shared_mem)?;
+    Ok(out)
+}
+
+/// On-device marlin routing build from per-expert offsets `off` [n_exp+1] (device i32).
+
+/// On-device marlin routing build from per-expert offsets `off` [n_exp+1] (device i32).
+pub fn marlin_build_routing(dev:&dyn Device, off:&Tensor, n_exp:usize, blk:usize, mt:usize)->Result<(Tensor,Tensor,Tensor)>{
+    let max_blocks = mt/blk + n_exp + 1;
+    let stid=Tensor::new(dev.alloc((max_blocks*blk).max(1)*4)?, vec![(max_blocks*blk).max(1)], DType::U32);
+    let eid=Tensor::new(dev.alloc(max_blocks.max(1)*4)?, vec![max_blocks.max(1)], DType::U32);
+    let ntpp=Tensor::new(dev.alloc_zeroed(4)?, vec![1], DType::U32);
+    dev.marlin_build_routing(off.buffer.as_ref(), n_exp, blk, mt, stid.buffer.as_ref(), eid.buffer.as_ref(), ntpp.buffer.as_ref())?;
+    Ok((stid,eid,ntpp))
+}
+
+
+/// Fused block-Hadamard(128) along K + global amax. Returns `(rotated [rows,k]
+/// f16, amax [1] u32 bits)` for a subsequent fp8 quant that skips its own amax.
+pub fn fwht128_amax(dev: &dyn Device, x: &Tensor, rows: usize, k: usize) -> Result<(Tensor, Tensor)> {
+    if k % 128 != 0 { return Err(Error::Msg(format!("fwht128_amax: k {k} %128 != 0"))); }
+    let y = Tensor::new(dev.alloc(rows * k * 2)?, vec![rows * k], DType::F16);
+    let amax = Tensor::new(dev.alloc_zeroed(4)?, vec![1], DType::U32);
+    let nb = (rows * (k / 128)) as u32;
+    let i = |v: i32| v.to_le_bytes().to_vec();
+    let l = |v: i64| v.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fwht128_amax",
+        &[(x.buffer.as_ref(), x.offset), (y.buffer.as_ref(), 0), (amax.buffer.as_ref(), 0)],
+        &[l(rows as i64), i(k as i32)],
+        [nb, 1, 1], [128, 1, 1], 0, false)?;
+    Ok((y.reshaped(vec![rows, k]), amax))
+}
+
+/// Per-tensor fp8 quant using a PRECOMPUTED amax (from `fwht128_amax`), with
+/// optional clip. Returns `(q bytes, sc [2] f32)`. Skips the amax pass.
+
+/// Per-tensor fp8 quant using a PRECOMPUTED amax (from `fwht128_amax`), with
+/// optional clip. Returns `(q bytes, sc [2] f32)`. Skips the amax pass.
+pub fn fp8_quant_pre(dev: &dyn Device, x: &Tensor, n: usize, amax: &Tensor, clip: f32) -> Result<(Tensor, Tensor)> {
+    let i = |v: i32| v.to_le_bytes().to_vec();
+    let l = |v: i64| v.to_le_bytes().to_vec();
+    let sc = Tensor::new(dev.alloc(8)?, vec![2], DType::F32);
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fp8_scale",
+        &[(amax.buffer.as_ref(), 0), (sc.buffer.as_ref(), 0)],
+        &[clip.to_le_bytes().to_vec()], [1, 1, 1], [1, 1, 1], 0, false)?;
+    let q = Tensor::new(dev.alloc(n)?, vec![n], DType::U32);
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fp8_quant",
+        &[(x.buffer.as_ref(), x.offset), (q.buffer.as_ref(), 0), (sc.buffer.as_ref(), 0)],
+        &[l(n as i64)],
+        [(n as u32).div_ceil(256 * 4).max(1), 1, 1], [256, 1, 1], 0, false)?;
+    Ok((q, sc))
+}
+
+/// Fused-path pass 1: block-Hadamard(128) along K + global amax, NO rotated
+/// output (saves the [rows,k] f16 write+read vs `fwht128_amax`). Returns amax bits.
+
+/// Fused-path pass 1: block-Hadamard(128) along K + global amax, NO rotated
+/// output (saves the [rows,k] f16 write+read vs `fwht128_amax`). Returns amax bits.
+pub fn fwht128_amaxonly(dev: &dyn Device, x: &Tensor, rows: usize, k: usize) -> Result<Tensor> {
+    if k % 128 != 0 { return Err(Error::Msg(format!("fwht128_amaxonly: k {k} %128 != 0"))); }
+    let amax = Tensor::new(dev.alloc_zeroed(4)?, vec![1], DType::U32);
+    let nb = (rows * (k / 128)) as u32;
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fwht128_amaxonly",
+        &[(x.buffer.as_ref(), x.offset), (amax.buffer.as_ref(), 0)],
+        &[(rows as i64).to_le_bytes().to_vec(), (k as i32).to_le_bytes().to_vec()],
+        [nb, 1, 1], [128, 1, 1], 0, false)?;
+    Ok(amax)
+}
+
+/// Fused-path pass 2: re-rotate `x` (Hadamard 128 along K) AND quant to e4m3 in
+/// one kernel reading `x` directly (no rotated-tensor round-trip), using a
+/// precomputed amax. Returns `(q bytes, sc [2] f32)`.
+
+/// Fused-path pass 2: re-rotate `x` (Hadamard 128 along K) AND quant to e4m3 in
+/// one kernel reading `x` directly (no rotated-tensor round-trip), using a
+/// precomputed amax. Returns `(q bytes, sc [2] f32)`.
+pub fn fwht128_quant_pre(dev: &dyn Device, x: &Tensor, rows: usize, k: usize, amax: &Tensor, clip: f32) -> Result<(Tensor, Tensor)> {
+    if k % 128 != 0 { return Err(Error::Msg(format!("fwht128_quant_pre: k {k} %128 != 0"))); }
+    let sc = Tensor::new(dev.alloc(8)?, vec![2], DType::F32);
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fp8_scale",
+        &[(amax.buffer.as_ref(), 0), (sc.buffer.as_ref(), 0)],
+        &[clip.to_le_bytes().to_vec()], [1, 1, 1], [1, 1, 1], 0, false)?;
+    let n = rows * k;
+    let q = Tensor::new(dev.alloc(n)?, vec![n], DType::U32);
+    let nb = (rows * (k / 128)) as u32;
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fwht128_quant",
+        &[(x.buffer.as_ref(), x.offset), (q.buffer.as_ref(), 0), (sc.buffer.as_ref(), 0)],
+        &[(rows as i64).to_le_bytes().to_vec(), (k as i32).to_le_bytes().to_vec()],
+        [nb, 1, 1], [128, 1, 1], 0, false)?;
+    Ok((q, sc))
+}
+
+/// Block Walsh-Hadamard transform (128) along K of an f16 `[rows, k]` tensor.
+/// k must be a multiple of 128. Orthonormal; used to spread activation outliers
+/// before a per-tensor fp8 quant (and applied identically to the weight offline).
+
+/// Block Walsh-Hadamard transform (128) along K of an f16 `[rows, k]` tensor.
+/// k must be a multiple of 128. Orthonormal; used to spread activation outliers
+/// before a per-tensor fp8 quant (and applied identically to the weight offline).
+pub fn fwht128(dev: &dyn Device, x: &Tensor, rows: usize, k: usize) -> Result<Tensor> {
+    if k % 128 != 0 { return Err(Error::Msg(format!("fwht128: k {k} %128 != 0"))); }
+    let y = Tensor::new(dev.alloc(rows * k * 2)?, vec![rows * k], DType::F16);
+    let nb = (rows * (k / 128)) as u32;
+    let i = |v: i32| v.to_le_bytes().to_vec();
+    let l = |v: i64| v.to_le_bytes().to_vec();
+    dev.dispatch_raw_cuda(MOE_FP4_SRC, "moe_fp4.cu", "fwht128",
+        &[(x.buffer.as_ref(), x.offset), (y.buffer.as_ref(), 0)],
+        &[l(rows as i64), i(k as i32)],
+        [nb, 1, 1], [128, 1, 1], 0, false)?;
+    Ok(y.reshaped(vec![rows, k]))
+}
+
+/// FP8 e4m3 dense GEMM with per-tensor scales (vendor-recipe precision for
+/// the Mamba in/out projections + shared expert on this model family).
+
+
+// ── Blocked pending upstream iron kernels ─────────────────────────────
+//
+// The backends branch's `moe_gather_q4` / `moe_gather_q4_swiglu` dispatch
+// `iron_moe_gather_q4` / `iron_moe_gather_q4_swiglu` via
+// `wh_iron_std::kernels::moe::moe_gather_q4::kernel_ir_for`. As of this
+// rebrand merge, `thewafflehaus/iron@dev` ships only the
+// `iron_moe_gather_q4_relu2` / `_down` / `_down_accum` variants of that
+// kernel family — the plain (non-relu2) and `_swiglu` variants these two
+// functions need do not exist upstream yet. Kept as always-erroring stubs
+// (matching signatures) so callers/tests compiled against this crate still
+// link — see `wh-butter-cuda/tests/laguna.rs`'s
+// `#[ignore]`d `laguna_moe_swiglu_fused_unit`. TODO(rebrand-merge): land
+// `iron_moe_gather_q4` + `iron_moe_gather_q4_swiglu` in the iron kernels
+// repo, then replace these stubs with real `cached_ir` dispatches (see the
+// sibling `_relu2`/`_down`/`_down_accum` call sites above for the pattern).
+pub fn moe_gather_q4(_dev: &dyn Device, _qs: &Tensor, _scales: &Tensor, _x: &Tensor, _idx: &Tensor, _top_k: usize, _inter: usize, _hid: usize) -> Result<Tensor> {
+    Err(Error::Msg("moe_gather_q4: blocked — iron_moe_gather_q4 kernel not yet in thewafflehaus/iron@dev (only _relu2/_down/_down_accum variants exist)".into()))
+}
+
+#[allow(clippy::too_many_arguments)]
+pub fn moe_gather_q4_swiglu(_dev: &dyn Device, _gate_qs: &Tensor, _gate_scales: &Tensor, _up_qs: &Tensor, _up_scales: &Tensor, _x: &Tensor, _idx: &Tensor, _top_k: usize, _inter: usize, _hid: usize) -> Result<Tensor> {
+    Err(Error::Msg("moe_gather_q4_swiglu: blocked — iron_moe_gather_q4_swiglu kernel not yet in thewafflehaus/iron@dev".into()))
+}
diff --git a/rust/nvfp4_reference/README.md b/rust/nvfp4_reference/README.md
new file mode 100644
index 00000000..06927997
--- /dev/null
+++ b/rust/nvfp4_reference/README.md
@@ -0,0 +1,36 @@
+# NVFP4 (mxf4nvf4) reference harnesses — GB10 / sm_121a
+
+Standalone CUDA probes that reverse-engineered + validated the native Blackwell
+**NVFP4 block-scaled tensor-core mma** for the Nemotron-H prefill MoE GEMM
+(closing the gap to vLLM's 6395 tok/s @S2048; the Q4 grouped path is ~10 TFLOP/s,
+NVFP4 mma peaks ~386 TFLOP/s throttled on GB10).
+
+The instruction (validated computing bit-exact):
+```
+mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3
+```
+Build (accelerated-arch-only — base sm_121 PTX is rejected by ptxas):
+```
+nvcc -gencode arch=compute_121a,code=sm_121a <file>.cu -o <out>
+```
+
+## Files (build order = discovery order)
+| file | what it establishes |
+|------|---------------------|
+| `fp4_probe.cu` / `nvfp4_fix.cu` | the mxf4nvf4 mma **assembles** on sm_121a; the scale token is `ue4m3` (unsigned), not `e4m3` |
+| `fp4_compute.cu` | mma **computes correctly** (all-ones → D=64); C/D accumulator layout (c0,c1→row gid,col tig*2+{0,1}; c2,c3→row gid+8); e2m1 1.0=0x2, ue4m3 1.0=0x38 |
+| `fp4_derive.cu` / `fp4_deriveB.cu` | empirical A/B fragment **row-map** (one-hot sweep): A reg0,2→gid / reg1,3→gid+8; B reg0,1→gid |
+| `fp4_validate.cu` | full random-tile **data layout** bit-exact vs f32 (scales=1) |
+| `fp4_scale.cu` / `fp4_scaleB.cu` / `fp4_byteblk.cu` / `fp4_perm.cu` | derive the **scale operand** layout (scale_vec::4X, byte→block, lane mapping) |
+| **`fp4_aligned.cu`** | **THE reference loader** — full layout + random ue4m3 scales bit-exact vs f32 (bad=0/128). Mirror this in the kernel. |
+| `qnvfp4_test.cpp` | weight quant accuracy: NVFP4 ≈ Q4 rel-rmse (host) |
+| `nvfp4_gemm.cu` | multi-K-tile accumulation + per-tensor global-scale epilogue |
+| `nvfp4_peak.cu` | raw mma throughput microbench (386 TFLOP/s throttled @539MHz) |
+| `nvfp4_tiled.cu` | tiled GEMM at real expert dims [96×1856×2688] = 58 TFLOP/s (~6× Q4) |
+| **`nvfp4_packed.cu`** | **in-kernel gather** from the `quantize_nvfp4` packed `[N, codes(K/8 u32)+scales(K/16 u8)]` layout — the core of the production kernel |
+| `nvfp4_actq.cu` | device activation quantizer: f16 `[rows,K]` → NVFP4 codes+scales+global |
+
+These are dev/reference, NOT part of the cargo build. The production path is
+`quantize_nvfp4` (wh-butter-ops, committed) + the forthcoming `MOE_FP4_GROUPED_MMA_SRC`.
+Full state + the exact fragment/scale layout recipe: memory note
+`nemotron-grouped-moe-gemm.md` (2026-06-08 sections).
diff --git a/rust/nvfp4_reference/fp4_aligned.cu b/rust/nvfp4_reference/fp4_aligned.cu
new file mode 100644
index 00000000..3ab5b752
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_aligned.cu
@@ -0,0 +1,39 @@
+#include <cstdio>
+#include <cstdlib>
+__device__ __host__ float dec4(int nib){int s=(nib>>3)&1,e=(nib>>1)&3,m=nib&1;float v;if(e==0)v=0.5f*m;else v=(1.0f+0.5f*m)*(1<<(e-1));return s?-v:v;}
+__device__ __host__ float decS(unsigned char b){int e=(b>>3)&0xF,m=b&7;if(e==0)return m/8.0f*0.0078125f;return (1.0f+m/8.0f)*exp2f((float)(e-7));}
+__constant__ unsigned char Ac[16*64],Bc[8*64],SAc[16*4],SBc[8*4];
+__device__ int byteForBlock(int blk){int t[4]={0,2,1,3};return t[blk];}
+__global__ void kr(float* D){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3;
+  unsigned a[4]={0,0,0,0},b[2]={0,0},sA=0x38383838,sB=0x38383838;
+  // ALIGNED load: for each weight k, compute its fragment slot from (block,w)
+  // B row = gid (this lane). place B[gid][K]
+  for(int K=0;K<64;K++){int blk=K>>4,w=K&15; int ttig=(blk>=2?2:0)+(w>=8?1:0); if(ttig!=tig)continue; int reg=blk&1; int nib=w&7; b[reg]|=(unsigned)Bc[gid*64+K]<<(nib*4);}
+  // A rows gid (regs0/2) and gid+8 (regs1/3)
+  for(int half=0;half<2;half++){int row=gid+half*8;
+    for(int K=0;K<64;K++){int blk=K>>4,w=K&15;int ttig=(blk>=2?2:0)+(w>=8?1:0);if(ttig!=tig)continue;int nib=w&7;
+      int reg = half==0 ? (blk&1?2:0) : (blk&1?3:1);
+      a[reg]|=(unsigned)Ac[row*64+K]<<(nib*4);}}
+  // scales: A scale lane tig0->row gid, tig1->row gid+8; byte=byteForBlock(blk)
+  if(tig==0||tig==1){int row=gid+(tig==1?8:0);sA=0;for(int blk=0;blk<4;blk++)sA|=(unsigned)SAc[row*4+blk]<<(byteForBlock(blk)*8);}
+  if(tig==0){sB=0;for(int blk=0;blk<4;blk++)sB|=(unsigned)SBc[gid*4+blk]<<(byteForBlock(blk)*8);}
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  D[gid*8+tig*2]=c0;D[gid*8+tig*2+1]=c1;D[(gid+8)*8+tig*2]=c2;D[(gid+8)*8+tig*2+1]=c3;
+}
+int main(){
+  unsigned char A[16*64],B[8*64],SA[16*4],SB[8*4];srand(11);
+  for(int i=0;i<16*64;i++)A[i]=rand()&15;for(int i=0;i<8*64;i++)B[i]=rand()&15;
+  for(int i=0;i<16*4;i++)SA[i]=((6+(rand()%3))<<3)|(rand()&7);for(int i=0;i<8*4;i++)SB[i]=((6+(rand()%3))<<3)|(rand()&7);
+  cudaMemcpyToSymbol(Ac,A,sizeof A);cudaMemcpyToSymbol(Bc,B,sizeof B);cudaMemcpyToSymbol(SAc,SA,sizeof SA);cudaMemcpyToSymbol(SBc,SB,sizeof SB);
+  float* d;cudaMalloc(&d,128*4);cudaMemset(d,0,128*4);kr<<<1,32>>>(d);
+  if(cudaDeviceSynchronize()){printf("err\n");return 1;}
+  float h[128];cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+  int bad=0;float mre=0;
+  for(int m=0;m<16;m++)for(int n=0;n<8;n++){float ref=0;for(int blk=0;blk<4;blk++){float s=0;for(int kk=0;kk<16;kk++){int K=blk*16+kk;s+=dec4(A[m*64+K])*dec4(B[n*64+K]);}ref+=s*decS(SA[m*4+blk])*decS(SB[n*4+blk]);}float re=ref!=0?(ref-h[m*8+n])/ref:ref-h[m*8+n];if(re<0)re=-re;if(re>mre)mre=re;if(re>1e-2f){bad++;if(bad<=3)printf("MISS[%d][%d] ref=%.3f got=%.3f\n",m,n,ref,h[m*8+n]);}}
+  printf("bad=%d/128 mre=%.5f %s\n",bad,mre,bad?"WRONG":"<<<< NVFP4 mma FULLY SOLVED (data+scale)");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_byteblk.cu b/rust/nvfp4_reference/fp4_byteblk.cu
new file mode 100644
index 00000000..92a51f51
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_byteblk.cu
@@ -0,0 +1,25 @@
+#include <cstdio>
+// A=1.0 everywhere; B=1.0 ONLY in data k-block bd (k in [bd*16,bd*16+16)), else 0.
+// sB=1; sA: lane(gid0,tig0) byte TB=2.0, rest 1.0. D[0][0]=16*sA[block of bd]. If byte TB scales block bd -> 32 else 16.
+__global__ void k(float* D,int bd,int TB){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3;
+  unsigned a0=0x22222222,a1=a0,a2=a0,a3=a0;       // A all 1.0
+  // B = 1.0 only where data k-block==bd. lane tig holds k=tig*16..+15 = block tig. regs b0(k tig*16+0..7),b1(+8..15)
+  unsigned b0=(tig==bd)?0x22222222:0, b1=(tig==bd)?0x22222222:0;
+  unsigned sA=0x38383838,sB=0x38383838;
+  if(tig==0 && gid==0) sA=(sA&~(0xFFu<<(TB*8)))|(0x40u<<(TB*8));
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  D[gid*8+tig*2]=c0;D[gid*8+tig*2+1]=c1;D[(gid+8)*8+tig*2]=c2;D[(gid+8)*8+tig*2+1]=c3;
+}
+int main(){
+  float* d;cudaMalloc(&d,128*4);float h[128];
+  printf("      bd0   bd1   bd2   bd3   (D[0][0]; 32=byte scales that block)\n");
+  for(int TB=0;TB<4;TB++){ printf("byte%d ",TB);
+    for(int bd=0;bd<4;bd++){cudaMemset(d,0,128*4);k<<<1,32>>>(d,bd,TB);cudaDeviceSynchronize();cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);printf("%5.0f ",h[0]);}
+    printf("\n");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_compute.cu b/rust/nvfp4_reference/fp4_compute.cu
new file mode 100644
index 00000000..ae530a11
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_compute.cu
@@ -0,0 +1,33 @@
+#include <cstdio>
+// Validate the NVFP4 mma COMPUTES (not just assembles) on GB10/sm_121a.
+// A[16x64]=B[8x64]=e2m1(1.0)=0x2 nibble → 0x22222222/reg; scales=ue4m3(1.0)=0x38 → 0x38383838.
+// D[i][j] = sum_{k0..63} 1*1 * sA*sB = 64. Layout-independent (all elems equal).
+__global__ void k(float* D){
+  unsigned a0=0x22222222,a1=0x22222222,a2=0x22222222,a3=0x22222222;
+  unsigned b0=0x22222222,b1=0x22222222;
+  unsigned sA=0x38383838, sB=0x38383838;
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile(
+   "mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%0,%1,%2,%3}, {%10}, {0,0}, {%11}, {0,0};"
+   : "+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3)
+   : "r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  int lane=threadIdx.x;
+  // C/D layout: c0,c1 -> row=gid, col=tig*2+{0,1}; c2,c3 -> row=gid+8
+  int gid=lane>>2, tig=lane&3;
+  D[(gid)*8 + tig*2+0]=c0; D[(gid)*8 + tig*2+1]=c1;
+  D[(gid+8)*8 + tig*2+0]=c2; D[(gid+8)*8 + tig*2+1]=c3;
+}
+int main(){
+  float* d; cudaMalloc(&d,16*8*4); cudaMemset(d,0,16*8*4);
+  k<<<1,32>>>(d);
+  cudaError_t e=cudaDeviceSynchronize();
+  if(e){printf("LAUNCH ERR: %s\n",cudaGetErrorString(e));return 1;}
+  float h[128]; cudaMemcpy(h,d,16*8*4,cudaMemcpyDeviceToHost);
+  int ok=0,bad=0; float mn=1e9,mx=-1e9;
+  for(int i=0;i<128;i++){ if(h[i]==64.0f)ok++; else bad++; if(h[i]<mn)mn=h[i]; if(h[i]>mx)mx=h[i]; }
+  printf("D[0..7]: %.1f %.1f %.1f %.1f %.1f %.1f %.1f %.1f\n",h[0],h[1],h[2],h[3],h[4],h[5],h[6],h[7]);
+  printf("==64: %d/128   range[%.3f,%.3f]\n",ok,mn,mx);
+  printf(ok==128?"COMPUTE OK (all 64)\n":"MISMATCH (expected all 64)\n");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_derive.cu b/rust/nvfp4_reference/fp4_derive.cu
new file mode 100644
index 00000000..04cd8438
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_derive.cu
@@ -0,0 +1,26 @@
+#include <cstdio>
+// Derive A fragment row-map: one-hot A at (target_lane tl, target_reg tr, nibble tn)=e2m1 1.0,
+// B=all-ones, scales=1 → D[m][n]=sum_k A[m][k] = 1.0 only on A's row. Read which D row lights.
+__global__ void k(float* D,int tl,int tr,int tn){
+  unsigned a[4]={0,0,0,0}, b0=0x22222222,b1=0x22222222, sA=0x38383838,sB=0x38383838;
+  if((int)threadIdx.x==tl) a[tr] |= (0x2u<<(tn*4));
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  int g=threadIdx.x>>2, t=threadIdx.x&3;
+  D[g*8+t*2]=c0; D[g*8+t*2+1]=c1; D[(g+8)*8+t*2]=c2; D[(g+8)*8+t*2+1]=c3;
+}
+int main(){
+  float* d; cudaMalloc(&d,128*4); float h[128];
+  printf("lane reg nib -> rows_with_nonzero_D (and value)\n");
+  for(int tl=0;tl<32;tl++) for(int tr=0;tr<4;tr++){ int tn=0;
+    cudaMemset(d,0,128*4); k<<<1,32>>>(d,tl,tr,tn); cudaDeviceSynchronize();
+    cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+    // find rows that are nonzero
+    char buf[128]=""; float val=0;
+    for(int r=0;r<16;r++){ int nz=0; for(int c=0;c<8;c++) if(h[r*8+c]!=0){nz++; val=h[r*8+c];} if(nz){ char t[16]; sprintf(t,"r%d(%.1f) ",r,val); strcat(buf,t);} }
+    if(tl<8 || tl==31) printf("L%-2d reg%d -> %s\n",tl,tr,buf[0]?buf:"(none)");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_deriveB.cu b/rust/nvfp4_reference/fp4_deriveB.cu
new file mode 100644
index 00000000..a46fd588
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_deriveB.cu
@@ -0,0 +1,25 @@
+#include <cstdio>
+#include <cstring>
+// Derive B fragment row-map: one-hot B at (tl,tr,tn)=e2m1 1.0, A=all-ones, scales=1.
+// D[m][n]=sum_k A[m][k]*B[n][k]=sum_k B[n][k] = 1.0 only at B's row n (all m). Read which D col lights.
+__global__ void k(float* D,int tl,int tr,int tn){
+  unsigned a0=0x22222222,a1=0x22222222,a2=0x22222222,a3=0x22222222, b[2]={0,0}, sA=0x38383838,sB=0x38383838;
+  if((int)threadIdx.x==tl) b[tr] |= (0x2u<<(tn*4));
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  int g=threadIdx.x>>2,t=threadIdx.x&3;
+  D[g*8+t*2]=c0; D[g*8+t*2+1]=c1; D[(g+8)*8+t*2]=c2; D[(g+8)*8+t*2+1]=c3;
+}
+int main(){
+  float* d; cudaMalloc(&d,128*4); float h[128];
+  printf("B: lane reg -> cols_with_nonzero_D\n");
+  for(int tl=0;tl<32;tl++) for(int tr=0;tr<2;tr++){
+    cudaMemset(d,0,128*4); k<<<1,32>>>(d,tl,tr,0); cudaDeviceSynchronize();
+    cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+    char buf[64]=""; for(int c=0;c<8;c++){int nz=0; for(int r=0;r<16;r++) if(h[r*8+c]!=0)nz++; if(nz){char t[8];sprintf(t,"c%d ",c);strcat(buf,t);}}
+    if(tl<8||tl==31) printf("L%-2d reg%d -> %s\n",tl,tr,buf[0]?buf:"(none)");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_full.cu b/rust/nvfp4_reference/fp4_full.cu
new file mode 100644
index 00000000..2f671bd9
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_full.cu
@@ -0,0 +1,45 @@
+#include <cstdio>
+#include <cstdlib>
+__device__ __host__ float dec4(int nib){int s=(nib>>3)&1,e=(nib>>1)&3,m=nib&1;float v;if(e==0)v=0.5f*m;else v=(1.0f+0.5f*m)*(1<<(e-1));return s?-v:v;}
+// ue4m3 decode (unsigned e4m3: 4 exp bias7, 3 mant)
+__device__ __host__ float decS(unsigned char b){int e=(b>>3)&0xF,m=b&7;if(e==0)return m/8.0f*0.0078125f;return (1.0f+m/8.0f)*exp2f((float)(e-7));}
+__constant__ unsigned char Ac[16*64],Bc[8*64],SAc[16*4],SBc[8*4];
+__global__ void k(float* D){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3;
+  unsigned a[4]={0,0,0,0},b[2]={0,0};
+  for(int j=0;j<8;j++){
+    a[0]|=(unsigned)Ac[gid*64+tig*16+0+j]<<(j*4);  a[2]|=(unsigned)Ac[gid*64+tig*16+8+j]<<(j*4);
+    a[1]|=(unsigned)Ac[(gid+8)*64+tig*16+0+j]<<(j*4); a[3]|=(unsigned)Ac[(gid+8)*64+tig*16+8+j]<<(j*4);
+    b[0]|=(unsigned)Bc[gid*64+tig*16+0+j]<<(j*4); b[1]|=(unsigned)Bc[gid*64+tig*16+8+j]<<(j*4);
+  }
+  // scales: tig0 -> row gid (A) / row gid (B); tig1 -> row gid+8 (A); else 1.0(0x38)
+  unsigned sA=0x38383838,sB=0x38383838;
+  if(tig==0){ sA=0; for(int x=0;x<4;x++) sA|=(unsigned)SAc[gid*4+x]<<(x*8); }
+  else if(tig==1){ sA=0; for(int x=0;x<4;x++) sA|=(unsigned)SAc[(gid+8)*4+x]<<(x*8); }
+  if(tig==0){ sB=0; for(int x=0;x<4;x++) sB|=(unsigned)SBc[gid*4+x]<<(x*8); }
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  D[gid*8+tig*2]=c0;D[gid*8+tig*2+1]=c1;D[(gid+8)*8+tig*2]=c2;D[(gid+8)*8+tig*2+1]=c3;
+}
+int main(){
+  unsigned char A[16*64],B[8*64],SA[16*4],SB[8*4];srand(11);
+  for(int i=0;i<16*64;i++)A[i]=rand()&15; for(int i=0;i<8*64;i++)B[i]=rand()&15;
+  // scales: exp in [6,8], mant random -> decode 0.5..~3.75
+  for(int i=0;i<16*4;i++)SA[i]=((6+(rand()%3))<<3)|(rand()&7);
+  for(int i=0;i<8*4;i++)SB[i]=((6+(rand()%3))<<3)|(rand()&7);
+  cudaMemcpyToSymbol(Ac,A,sizeof A);cudaMemcpyToSymbol(Bc,B,sizeof B);
+  cudaMemcpyToSymbol(SAc,SA,sizeof SA);cudaMemcpyToSymbol(SBc,SB,sizeof SB);
+  float* d;cudaMalloc(&d,128*4);cudaMemset(d,0,128*4);k<<<1,32>>>(d);
+  if(cudaDeviceSynchronize()){printf("launch err\n");return 1;}
+  float h[128];cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+  int bad=0;float maxre=0;
+  for(int m=0;m<16;m++)for(int n=0;n<8;n++){
+    float ref=0; for(int blk=0;blk<4;blk++){float sub=0;for(int kk=0;kk<16;kk++){int K=blk*16+kk;sub+=dec4(A[m*64+K])*dec4(B[n*64+K]);} ref+=sub*decS(SA[m*4+blk])*decS(SB[n*4+blk]);}
+    float got=h[m*8+n];float re=ref!=0?(ref-got)/ref:(ref-got);if(re<0)re=-re;if(re>maxre)maxre=re;
+    if(re>1e-2f){bad++;if(bad<=4)printf("MISS D[%d][%d] ref=%.4f got=%.4f\n",m,n,ref,got);}
+  }
+  printf("bad=%d/128  max_rel_err=%.5f  %s\n",bad,maxre,bad?"SCALE/LAYOUT WRONG":"FULL LAYOUT+SCALE CONFIRMED");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_perm.cu b/rust/nvfp4_reference/fp4_perm.cu
new file mode 100644
index 00000000..782f43e5
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_perm.cu
@@ -0,0 +1,42 @@
+#include <cstdio>
+#include <cstdlib>
+__device__ __host__ float dec4(int nib){int s=(nib>>3)&1,e=(nib>>1)&3,m=nib&1;float v;if(e==0)v=0.5f*m;else v=(1.0f+0.5f*m)*(1<<(e-1));return s?-v:v;}
+__device__ __host__ float decS(unsigned char b){int e=(b>>3)&0xF,m=b&7;if(e==0)return m/8.0f*0.0078125f;return (1.0f+m/8.0f)*exp2f((float)(e-7));}
+__constant__ unsigned char Ac[16*64],Bc[8*64],SAc[16*4],SBc[8*4];
+__constant__ int PERM[4];
+__global__ void k(float* D){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3;
+  unsigned a[4]={0,0,0,0},b[2]={0,0};
+  for(int j=0;j<8;j++){
+    a[0]|=(unsigned)Ac[gid*64+tig*16+j]<<(j*4); a[2]|=(unsigned)Ac[gid*64+tig*16+8+j]<<(j*4);
+    a[1]|=(unsigned)Ac[(gid+8)*64+tig*16+j]<<(j*4); a[3]|=(unsigned)Ac[(gid+8)*64+tig*16+8+j]<<(j*4);
+    b[0]|=(unsigned)Bc[gid*64+tig*16+j]<<(j*4); b[1]|=(unsigned)Bc[gid*64+tig*16+8+j]<<(j*4);
+  }
+  // pack scale: byte PERM[blk] gets block blk's scale
+  unsigned sA=0x38383838,sB=0x38383838; int row=-1;
+  if(tig==0)row=gid; else if(tig==1)row=gid+8;
+  if(row>=0){sA=0;for(int blk=0;blk<4;blk++)sA|=(unsigned)SAc[row*4+blk]<<(PERM[blk]*8);}
+  if(tig==0){sB=0;for(int blk=0;blk<4;blk++)sB|=(unsigned)SBc[gid*4+blk]<<(PERM[blk]*8);}
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  D[gid*8+tig*2]=c0;D[gid*8+tig*2+1]=c1;D[(gid+8)*8+tig*2]=c2;D[(gid+8)*8+tig*2+1]=c3;
+}
+int main(){
+  unsigned char A[16*64],B[8*64],SA[16*4],SB[8*4];srand(11);
+  for(int i=0;i<16*64;i++)A[i]=rand()&15;for(int i=0;i<8*64;i++)B[i]=rand()&15;
+  for(int i=0;i<16*4;i++)SA[i]=((6+(rand()%3))<<3)|(rand()&7);for(int i=0;i<8*4;i++)SB[i]=((6+(rand()%3))<<3)|(rand()&7);
+  cudaMemcpyToSymbol(Ac,A,sizeof A);cudaMemcpyToSymbol(Bc,B,sizeof B);cudaMemcpyToSymbol(SAc,SA,sizeof SA);cudaMemcpyToSymbol(SBc,SB,sizeof SB);
+  float ref[128];
+  for(int m=0;m<16;m++)for(int n=0;n<8;n++){float r=0;for(int blk=0;blk<4;blk++){float s=0;for(int kk=0;kk<16;kk++){int K=blk*16+kk;s+=dec4(A[m*64+K])*dec4(B[n*64+K]);}r+=s*decS(SA[m*4+blk])*decS(SB[n*4+blk]);}ref[m*8+n]=r;}
+  int perms[24][4],np=0;int p[4]={0,1,2,3};
+  // generate all 24 permutations
+  for(int a=0;a<4;a++)for(int bb=0;bb<4;bb++)for(int c=0;c<4;c++)for(int dd=0;dd<4;dd++){if(a!=bb&&a!=c&&a!=dd&&bb!=c&&bb!=dd&&c!=dd){perms[np][0]=a;perms[np][1]=bb;perms[np][2]=c;perms[np][3]=dd;np++;}}
+  float* d;cudaMalloc(&d,128*4);float h[128];
+  for(int pi=0;pi<np;pi++){cudaMemcpyToSymbol(PERM,perms[pi],sizeof(int)*4);cudaMemset(d,0,128*4);k<<<1,32>>>(d);cudaDeviceSynchronize();cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+    int bad=0;float mre=0;for(int i=0;i<128;i++){float re=ref[i]!=0?(ref[i]-h[i])/ref[i]:ref[i]-h[i];if(re<0)re=-re;if(re>mre)mre=re;if(re>1e-2f)bad++;}
+    if(bad<8)printf("PERM[%d %d %d %d] bad=%d mre=%.4f %s\n",perms[pi][0],perms[pi][1],perms[pi][2],perms[pi][3],bad,mre,bad==0?"<<< MATCH":"");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_probe.cu b/rust/nvfp4_reference/fp4_probe.cu
new file mode 100644
index 00000000..c44463a2
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_probe.cu
@@ -0,0 +1,16 @@
+#include <cuda_fp4.h>
+#include <mma.h>
+// Probe: does sm_121a assemble a block-scaled FP4 (e2m1 x mxfp4) mma.sync?
+__global__ void probe(const unsigned* a,const unsigned* b,float* d,const unsigned* sfa,const unsigned* sfb){
+  unsigned ra[4]={a[0],a[1],a[2],a[3]};
+  unsigned rb[2]={b[0],b[1]};
+  float rc[4]={0,0,0,0};
+  // Blackwell block-scaled MXF4 mma (sm_120a/121a): m16n8k64, e2m1 x e2m1, ue8m0 scales
+  asm volatile(
+   "mma.sync.aligned.m16n8k64.row.col.kind::mxf4.block_scale.f32.e2m1.e2m1.f32.ue8m0 "
+   "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%0,%1,%2,%3}, {%10}, {0,0}, {%11}, {0,0};"
+   : "+f"(rc[0]),"+f"(rc[1]),"+f"(rc[2]),"+f"(rc[3])
+   : "r"(ra[0]),"r"(ra[1]),"r"(ra[2]),"r"(ra[3]),"r"(rb[0]),"r"(rb[1]),"r"(sfa[0]),"r"(sfb[0]));
+  d[threadIdx.x]=rc[0];
+}
+int main(){return 0;}
diff --git a/rust/nvfp4_reference/fp4_scale.cu b/rust/nvfp4_reference/fp4_scale.cu
new file mode 100644
index 00000000..61082284
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_scale.cu
@@ -0,0 +1,26 @@
+#include <cstdio>
+#include <cstring>
+// Derive scale-block map. A=B=e2m1 1.0. sB=all 1.0. sA: base all 1.0(0x38), except thread TL byte TB=2.0(0x40).
+// D[m][n]=sum_{b=0..3} 16 * sA_eff[m][b]  (A=B=1,sB=1). Baseline all = 64. The (m,b) that (TL,TB) drives -> D[m]+=16.
+__global__ void k(float* D,int TL,int TB){
+  unsigned a0=0x22222222,a1=a0,a2=a0,a3=a0,b0=a0,b1=a0;
+  unsigned sA=0x38383838, sB=0x38383838;
+  if((int)threadIdx.x==TL){ sA = (sA & ~(0xFFu<<(TB*8))) | (0x40u<<(TB*8)); }
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  int g=threadIdx.x>>2,t=threadIdx.x&3;
+  D[g*8+t*2]=c0; D[g*8+t*2+1]=c1; D[(g+8)*8+t*2]=c2; D[(g+8)*8+t*2+1]=c3;
+}
+int main(){
+  float* d; cudaMalloc(&d,128*4); float h[128];
+  printf("scaleA: (thread TL, byte TB) -> rows where D jumped 64->80 (delta from 2.0x scale)\n");
+  for(int TL=0;TL<32;TL++) for(int TB=0;TB<4;TB++){
+    cudaMemset(d,0,128*4); k<<<1,32>>>(d,TL,TB); cudaDeviceSynchronize();
+    cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+    char buf[128]=""; for(int r=0;r<16;r++){ if(h[r*8]>64.5f){char t[24];sprintf(t,"r%d(%.0f) ",r,h[r*8]);strcat(buf,t);} }
+    if((TL<8)||TL==31) printf("TL%-2d B%d -> %s\n",TL,TB,buf[0]?buf:"(no change)");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_scaleB.cu b/rust/nvfp4_reference/fp4_scaleB.cu
new file mode 100644
index 00000000..037d036f
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_scaleB.cu
@@ -0,0 +1,23 @@
+#include <cstdio>
+#include <cstring>
+// Derive sB map: A=B=1, sA=1. one-hot sB byte TB of thread TL =2.0. D[m][n]=sum_blk 16*sB[n][blk]. find col n that jumps.
+__global__ void k(float* D,int TL,int TB){
+  unsigned a0=0x22222222,a1=a0,a2=a0,a3=a0,b0=a0,b1=a0,sA=0x38383838,sB=0x38383838;
+  if((int)threadIdx.x==TL) sB=(sB&~(0xFFu<<(TB*8)))|(0x40u<<(TB*8));
+  float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  int g=threadIdx.x>>2,t=threadIdx.x&3;
+  D[g*8+t*2]=c0;D[g*8+t*2+1]=c1;D[(g+8)*8+t*2]=c2;D[(g+8)*8+t*2+1]=c3;
+}
+int main(){
+  float* d;cudaMalloc(&d,128*4);float h[128];
+  printf("sB: (TL,TB) -> cols that jumped\n");
+  for(int TL=0;TL<32;TL++)for(int TB=0;TB<4;TB++){
+    cudaMemset(d,0,128*4);k<<<1,32>>>(d,TL,TB);cudaDeviceSynchronize();cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+    char buf[64]="";for(int c=0;c<8;c++)if(h[c]>64.5f){char t[8];sprintf(t,"c%d ",c);strcat(buf,t);}
+    if(TL<12||TL==31)printf("TL%-2d B%d -> %s\n",TL,TB,buf[0]?buf:"-");
+  }
+  return 0;
+}
diff --git a/rust/nvfp4_reference/fp4_validate.cu b/rust/nvfp4_reference/fp4_validate.cu
new file mode 100644
index 00000000..1f749455
--- /dev/null
+++ b/rust/nvfp4_reference/fp4_validate.cu
@@ -0,0 +1,45 @@
+#include <cstdio>
+#include <cstdlib>
+// Full random m16n8k64 tile validation. A[16][64],B[8][64] random e2m1 nibbles, scales=1.
+// Hypothesis map: gid=lane>>2,tig=lane&3.
+//  A reg0:row gid,k=tig*16+0..7 ; reg2:row gid,k=tig*16+8..15 ; reg1:row gid+8,k=tig*16+0..7 ; reg3:row gid+8,k=tig*16+8..15
+//  B reg0:row gid,k=tig*16+0..7 ; reg1:row gid,k=tig*16+8..15
+__device__ __host__ float dec(int nib){
+  int s=(nib>>3)&1,e=(nib>>1)&3,m=nib&1; float v;
+  if(e==0) v=0.5f*m; else v=(1.0f+0.5f*m)*(1<<(e-1));
+  return s? -v:v;
+}
+__constant__ unsigned char Ac[16*64], Bc[8*64];
+__global__ void k(float* D){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3;
+  unsigned a[4]={0,0,0,0}, b[2]={0,0};
+  for(int j=0;j<8;j++){
+    a[0]|=(unsigned)Ac[(gid)*64    + tig*16+0+j]<<(j*4);
+    a[2]|=(unsigned)Ac[(gid)*64    + tig*16+8+j]<<(j*4);
+    a[1]|=(unsigned)Ac[(gid+8)*64  + tig*16+0+j]<<(j*4);
+    a[3]|=(unsigned)Ac[(gid+8)*64  + tig*16+8+j]<<(j*4);
+    b[0]|=(unsigned)Bc[(gid)*64    + tig*16+0+j]<<(j*4);
+    b[1]|=(unsigned)Bc[(gid)*64    + tig*16+8+j]<<(j*4);
+  }
+  unsigned sA=0x38383838,sB=0x38383838; float c0=0,c1=0,c2=0,c3=0;
+  asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+   :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  D[gid*8+tig*2]=c0; D[gid*8+tig*2+1]=c1; D[(gid+8)*8+tig*2]=c2; D[(gid+8)*8+tig*2+1]=c3;
+}
+int main(){
+  unsigned char A[16*64],B[8*64]; srand(7);
+  for(int i=0;i<16*64;i++)A[i]=rand()&15; for(int i=0;i<8*64;i++)B[i]=rand()&15;
+  cudaMemcpyToSymbol(Ac,A,sizeof(A)); cudaMemcpyToSymbol(Bc,B,sizeof(B));
+  float* d; cudaMalloc(&d,128*4); cudaMemset(d,0,128*4); k<<<1,32>>>(d);
+  if(cudaDeviceSynchronize()){printf("launch err\n");return 1;}
+  float h[128]; cudaMemcpy(h,d,128*4,cudaMemcpyDeviceToHost);
+  int bad=0; float maxerr=0;
+  for(int m=0;m<16;m++)for(int n=0;n<8;n++){
+    float ref=0; for(int kk=0;kk<64;kk++) ref+=dec(A[m*64+kk])*dec(B[n*64+kk]);
+    float got=h[m*8+n]; float e=ref-got; if(e<0)e=-e; if(e>maxerr)maxerr=e;
+    if(e>1e-2f){bad++; if(bad<=4)printf("MISS D[%d][%d] ref=%.3f got=%.3f\n",m,n,ref,got);}
+  }
+  printf("bad=%d/128  maxerr=%.5f  %s\n",bad,maxerr,bad?"LAYOUT WRONG":"LAYOUT CONFIRMED");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/nvfp4_actq.cu b/rust/nvfp4_reference/nvfp4_actq.cu
new file mode 100644
index 00000000..76842495
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_actq.cu
@@ -0,0 +1,36 @@
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <cuda_fp16.h>
+// Device activation quantizer: f16 act[rows,K] -> NVFP4 (e2m1 codes u32 8/word, ue4m3 scales[rows,K/16], global).
+// global passed in (real model: a reduce kernel; here host-computed). One thread per block-16.
+__device__ unsigned char encS_d(float s){if(s<=0)return 0;int be=0,bm=0;float bd=1e30f;
+  for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;be=e;bm=m;}}return (be<<3)|bm;}
+__device__ float decS_d(unsigned char b){int e=(b>>3)&0xF,m=b&7;return (e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));}
+__device__ int encE_d(float x,float s){float v=x/s;int sg=v<0;v=fabsf(v);
+  const float L[8]={0,.5f,1,1.5f,2,3,4,6};int b=0;float bd=1e30f;for(int i=0;i<8;i++){float d=fabsf(v-L[i]);if(d<bd){bd=d;b=i;}}return (sg<<3)|b;}
+__global__ void actq(const __half* a,unsigned* codes,unsigned char* scales,int rows,int K,float global){
+  int blk=blockIdx.x*blockDim.x+threadIdx.x; int nblk=rows*(K/16); if(blk>=nblk)return;
+  int r=blk/(K/16), b=blk%(K/16); const __half* ab=a+(size_t)r*K+b*16;
+  float v[16],amax=0; for(int i=0;i<16;i++){v[i]=__half2float(ab[i]);amax=fmaxf(amax,fabsf(v[i]));}
+  unsigned char sb=encS_d((amax/6.f)/global); scales[blk]=sb; float bs=decS_d(sb)*global; if(bs<=0)bs=1e-9f;
+  for(int i=0;i<16;i++){int elem=b*16+i,word=elem>>3,nib=elem&7;int code=encE_d(v[i],bs);atomicOr(&codes[(size_t)r*(K/8)+word],(unsigned)code<<(nib*4));}
+}
+int main(){
+  int rows=512,K=2688; std::vector<float> A((size_t)rows*K);srand(7);
+  for(auto&x:A){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  std::vector<__half> Ah(A.size()); for(size_t i=0;i<A.size();i++)Ah[i]=__float2half(A[i]);
+  float gmax=0;for(float x:A)gmax=fmaxf(gmax,fabsf(x)); float global=gmax/6.f/448.f;
+  __half* dA;unsigned* dC;unsigned char* dS; cudaMalloc(&dA,Ah.size()*2);cudaMalloc(&dC,(size_t)rows*(K/8)*4);cudaMalloc(&dS,(size_t)rows*(K/16));
+  cudaMemcpy(dA,Ah.data(),Ah.size()*2,cudaMemcpyHostToDevice);cudaMemset(dC,0,(size_t)rows*(K/8)*4);
+  int nblk=rows*(K/16); actq<<<(nblk+255)/256,256>>>(dA,dC,dS,rows,K,global);
+  if(cudaDeviceSynchronize()){printf("err\n");return 1;}
+  std::vector<unsigned> C((size_t)rows*(K/8));std::vector<unsigned char> S((size_t)rows*(K/16));
+  cudaMemcpy(C.data(),dC,C.size()*4,cudaMemcpyDeviceToHost);cudaMemcpy(S.data(),dS,S.size(),cudaMemcpyDeviceToHost);
+  // dequant + compare to original (f16-rounded)
+  const float L[8]={0,.5f,1,1.5f,2,3,4,6}; double sse=0,ss=0;
+  for(int r=0;r<rows;r++)for(int b=0;b<K/16;b++){int e=(S[r*(K/16)+b]>>3)&0xF,m=S[r*(K/16)+b]&7;float bs=((e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7)))*global;
+    for(int i=0;i<16;i++){int elem=b*16+i,word=elem>>3,nib=elem&7;unsigned cd=(C[(size_t)r*(K/8)+word]>>(nib*4))&0xf;float sign=(cd&8)?-1:1;float rec=sign*L[cd&7]*bs;float orig=__half2float(Ah[(size_t)r*K+elem]);double d=rec-orig;sse+=d*d;ss+=(double)orig*orig;}}
+  printf("device act-quant [%dx%d]: rel_rmse=%.3f%%  %s\n",rows,K,100*sqrt(sse/ss),sqrt(sse/ss)<0.15?"<<< OK (matches quantize_nvfp4)":"CHECK");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/nvfp4_fix.cu b/rust/nvfp4_reference/nvfp4_fix.cu
new file mode 100644
index 00000000..eeefebf3
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_fix.cu
@@ -0,0 +1,12 @@
+#include <cuda_fp4.h>
+__global__ void probe(const unsigned* a,const unsigned* b,float* d,const unsigned* sa,const unsigned* sb){
+  unsigned ra[4]={a[0],a[1],a[2],a[3]}; unsigned rb[2]={b[0],b[1]};
+  float rc[4]={0,0,0,0};
+  asm volatile(
+   "mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+   "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%0,%1,%2,%3}, {%10}, {0,0}, {%11}, {0,0};"
+   : "+f"(rc[0]),"+f"(rc[1]),"+f"(rc[2]),"+f"(rc[3])
+   : "r"(ra[0]),"r"(ra[1]),"r"(ra[2]),"r"(ra[3]),"r"(rb[0]),"r"(rb[1]),"r"(sa[0]),"r"(sb[0]));
+  d[threadIdx.x]=rc[0];
+}
+int main(){return 0;}
diff --git a/rust/nvfp4_reference/nvfp4_gemm.cu b/rust/nvfp4_reference/nvfp4_gemm.cu
new file mode 100644
index 00000000..60e7c035
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_gemm.cu
@@ -0,0 +1,57 @@
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <algorithm>
+static const float E2M1[8]={0,.5f,1,1.5f,2,3,4,6};
+__host__ int enc(float x,float s){float v=x/s;int sg=v<0;v=fabsf(v);int b=0;float bd=1e30f;for(int i=0;i<8;i++){float d=fabsf(v-E2M1[i]);if(d<bd){bd=d;b=i;}}return (sg<<3)|b;}
+__host__ float qS(float s){if(s<=0)return 1e-9f;float bv=1e-9f,bd=1e30f;for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;bv=v;}}return bv;}
+__host__ unsigned char encS(float s){if(s<=0)return 0;int be=0,bm=0;float bd=1e30f;for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;be=e;bm=m;}}return (be<<3)|bm;}
+// device: nibble codes + ue4m3 scale bytes, in ALIGNED layout per /tmp/fp4_aligned.cu
+// Buffers: per ktile, A nibbles[16*64], W nibbles[8*64]; scales packed per row blk.
+#define NT 4                  // K = NT*64
+__constant__ unsigned char Ac[16*64*NT],Wc[8*64*NT],SAc[16*4*NT],SWc[8*4*NT];
+__device__ int b4b(int blk){int t[4]={0,2,1,3};return t[blk];}
+__global__ void gemm(float* C,float gA,float gW){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3; float c0=0,c1=0,c2=0,c3=0;
+  for(int t=0;t<NT;t++){
+    const unsigned char* A=Ac+t*16*64;const unsigned char* W=Wc+t*8*64;
+    const unsigned char* SA=SAc+t*16*4;const unsigned char* SW=SWc+t*8*4;
+    unsigned a[4]={0,0,0,0},b[2]={0,0},sA=0,sB=0;
+    for(int K=0;K<64;K++){int blk=K>>4,w=K&15,ttig=(blk>=2?2:0)+(w>=8?1:0);if(ttig!=tig)continue;int nib=w&7,reg=blk&1;
+      b[reg]|=(unsigned)W[gid*64+K]<<(nib*4);
+      a[(blk&1?2:0)]|=(unsigned)A[gid*64+K]<<(nib*4);      // row gid -> regs0/2
+      a[(blk&1?3:1)]|=(unsigned)A[(gid+8)*64+K]<<(nib*4);  // row gid+8 -> regs1/3
+    }
+    if(tig==0||tig==1){int row=gid+(tig==1?8:0);for(int blk=0;blk<4;blk++)sA|=(unsigned)SA[row*4+blk]<<(b4b(blk)*8);}
+    if(tig==0){for(int blk=0;blk<4;blk++)sB|=(unsigned)SW[gid*4+blk]<<(b4b(blk)*8);}
+    asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+     "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+     :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  }
+  float g=gA*gW;
+  C[gid*8+tig*2]=c0*g;C[gid*8+tig*2+1]=c1*g;C[(gid+8)*8+tig*2]=c2*g;C[(gid+8)*8+tig*2+1]=c3*g;
+}
+int main(){
+  int M=16,N=8,K=NT*64; std::vector<float> A(M*K),W(N*K);
+  srand(3);for(auto&x:A){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  for(auto&x:W){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  // quantize A,W -> nibbles + ue4m3 block16 scales + global, ALIGNED per ktile
+  float gAmax=0,gWmax=0;for(float x:A)gAmax=std::max(gAmax,fabsf(x));for(float x:W)gWmax=std::max(gWmax,fabsf(x));
+  float gA=gAmax/6.f/448.f, gW=gWmax/6.f/448.f;
+  std::vector<unsigned char> An(M*64*NT),Wn(N*64*NT),SAb(M*4*NT),SWb(N*4*NT);
+  auto Q=[&](std::vector<float>&Wm,int rows,std::vector<unsigned char>&nib,std::vector<unsigned char>&sb,float glob){
+    for(int r=0;r<rows;r++)for(int t=0;t<NT;t++)for(int blk=0;blk<4;blk++){int k0=t*64+blk*16;float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(Wm[r*K+k0+j]));
+      float bs=qS((amax/6.f)/glob)*glob; if(bs<=0)bs=1e-9f; sb[t*rows*4 + r*4+blk]=encS((amax/6.f)/glob);
+      for(int j=0;j<16;j++)nib[t*rows*64 + r*64+blk*16+j]=enc(Wm[r*K+k0+j],bs);}
+  };
+  Q(A,16,An,SAb,gA); Q(W,8,Wn,SWb,gW);
+  cudaMemcpyToSymbol(Ac,An.data(),An.size());cudaMemcpyToSymbol(Wc,Wn.data(),Wn.size());
+  cudaMemcpyToSymbol(SAc,SAb.data(),SAb.size());cudaMemcpyToSymbol(SWc,SWb.data(),SWb.size());
+  float* d;cudaMalloc(&d,M*N*4);gemm<<<1,32>>>(d,gA,gW);
+  if(cudaDeviceSynchronize()){printf("err\n");return 1;}
+  std::vector<float> h(M*N);cudaMemcpy(h.data(),d,M*N*4,cudaMemcpyDeviceToHost);
+  double sse=0,ss=0;for(int m=0;m<M;m++)for(int n=0;n<N;n++){double ref=0;for(int k=0;k<K;k++)ref+=(double)A[m*K+k]*W[n*K+k];double e=ref-h[m*N+n];sse+=e*e;ss+=ref*ref;}
+  printf("NVFP4 GEMM [%dx%dx%d]  rel_rmse vs f32 = %.3f%%  (quant-noise expected ~10%%)\n",M,N,K,100*sqrt(sse/ss));
+  printf("C[0][0..3]=%.4f %.4f %.4f %.4f\n",h[0],h[1],h[2],h[3]);
+  return 0;
+}
diff --git a/rust/nvfp4_reference/nvfp4_packed.cu b/rust/nvfp4_reference/nvfp4_packed.cu
new file mode 100644
index 00000000..d3cef47d
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_packed.cu
@@ -0,0 +1,58 @@
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <vector>
+#include <algorithm>
+static const float E2M1[8]={0,.5f,1,1.5f,2,3,4,6};
+int enc(float x,float s){float v=x/s;int sg=v<0;v=fabsf(v);int b=0;float bd=1e30f;for(int i=0;i<8;i++){float d=fabsf(v-E2M1[i]);if(d<bd){bd=d;b=i;}}return (sg<<3)|b;}
+unsigned char encS(float s){if(s<=0)return 0;int be=0,bm=0;float bd=1e30f;for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;be=e;bm=m;}}return (be<<3)|bm;}
+// quantize_nvfp4 layout: codes[row*(K/8)+elem/8] nibble elem%8; scales[row*(K/16)+blk] ue4m3; global
+void qnv(const std::vector<float>&W,int rows,int K,std::vector<unsigned>&codes,std::vector<unsigned char>&scal,float glob){
+  codes.assign((size_t)rows*(K/8),0); scal.assign((size_t)rows*(K/16),0);
+  for(int r=0;r<rows;r++)for(int b=0;b<K/16;b++){int base=r*K+b*16;float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(W[base+j]));
+    unsigned char sb=encS((amax/6.f)/glob); scal[r*(K/16)+b]=sb;
+    int e=(sb>>3)&0xF,m=sb&7; float bs=((e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7)))*glob; if(bs<=0)bs=1e-9f;
+    for(int j=0;j<16;j++){int elem=b*16+j;int code=enc(W[base+j],bs);codes[(size_t)r*(K/8)+elem/8]|=(unsigned)code<<((elem%8)*4);}}
+}
+__device__ int b4b(int blk){int t[4]={0,2,1,3};return t[blk];}
+// In-kernel gather from packed quantize_nvfp4 layout. One warp -> C[16x8] over K.
+__global__ void gemm(const unsigned* Acode,const unsigned char* Asc,const unsigned* Wcode,const unsigned char* Wsc,
+                     float* C,int K,float g){
+  int lane=threadIdx.x,gid=lane>>2,tig=lane&3; int KT=K/64; float c0=0,c1=0,c2=0,c3=0;
+  for(int t=0;t<KT;t++){
+    unsigned a[4]={0,0,0,0},b[2]={0,0},sA=0,sB=0;
+    for(int kk=0;kk<64;kk++){int blk=kk>>4,w=kk&15,ttig=(blk>=2?2:0)+(w>=8?1:0);if(ttig!=tig)continue;int nib=w&7;int gK=t*64+kk;
+      // W row gid
+      {int e=gK,word=e>>3,sh=(e&7)*4;unsigned code=(Wcode[(size_t)gid*(K/8)+word]>>sh)&0xf; b[blk&1]|=code<<(nib*4);}
+      // A rows gid (regs0/2) and gid+8 (regs1/3)
+      {int e=gK,word=e>>3,sh=(e&7)*4;
+        unsigned ca=(Acode[(size_t)gid*(K/8)+word]>>sh)&0xf; a[(blk&1?2:0)]|=ca<<(nib*4);
+        unsigned cb=(Acode[(size_t)(gid+8)*(K/8)+word]>>sh)&0xf; a[(blk&1?3:1)]|=cb<<(nib*4);}
+    }
+    // scales: 4 blocks of this tile = global blocks t*4..t*4+3
+    if(tig==0||tig==1){int row=gid+(tig==1?8:0);for(int blk=0;blk<4;blk++)sA|=(unsigned)Asc[(size_t)row*(K/16)+t*4+blk]<<(b4b(blk)*8);}
+    if(tig==0){for(int blk=0;blk<4;blk++)sB|=(unsigned)Wsc[(size_t)gid*(K/16)+t*4+blk]<<(b4b(blk)*8);}
+    asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+     "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+     :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a[0]),"r"(a[1]),"r"(a[2]),"r"(a[3]),"r"(b[0]),"r"(b[1]),"r"(sA),"r"(sB));
+  }
+  C[gid*8+tig*2]=c0*g;C[gid*8+tig*2+1]=c1*g;C[(gid+8)*8+tig*2]=c2*g;C[(gid+8)*8+tig*2+1]=c3*g;
+}
+int main(){
+  int K=2688; std::vector<float> A(16*K),W(8*K);srand(5);
+  for(auto&x:A){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  for(auto&x:W){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  float gam=0,gwm=0;for(float x:A)gam=std::max(gam,fabsf(x));for(float x:W)gwm=std::max(gwm,fabsf(x));
+  float gA=gam/6.f/448.f,gW=gwm/6.f/448.f;
+  std::vector<unsigned> Ac,Wc;std::vector<unsigned char> As,Ws; qnv(A,16,K,Ac,As,gA); qnv(W,8,K,Wc,Ws,gW);
+  unsigned *dAc,*dWc;unsigned char *dAs,*dWs;float* dC;
+  cudaMalloc(&dAc,Ac.size()*4);cudaMalloc(&dWc,Wc.size()*4);cudaMalloc(&dAs,As.size());cudaMalloc(&dWs,Ws.size());cudaMalloc(&dC,16*8*4);
+  cudaMemcpy(dAc,Ac.data(),Ac.size()*4,cudaMemcpyHostToDevice);cudaMemcpy(dWc,Wc.data(),Wc.size()*4,cudaMemcpyHostToDevice);
+  cudaMemcpy(dAs,As.data(),As.size(),cudaMemcpyHostToDevice);cudaMemcpy(dWs,Ws.data(),Ws.size(),cudaMemcpyHostToDevice);
+  gemm<<<1,32>>>(dAc,dAs,dWc,dWs,dC,K,gA*gW);
+  if(cudaDeviceSynchronize()){printf("err\n");return 1;}
+  float h[128];cudaMemcpy(h,dC,16*8*4,cudaMemcpyDeviceToHost);
+  double sse=0,ss=0;for(int m=0;m<16;m++)for(int n=0;n<8;n++){double ref=0;for(int k=0;k<K;k++)ref+=(double)A[m*K+k]*W[n*K+k];double e=ref-h[m*8+n];sse+=e*e;ss+=ref*ref;}
+  printf("in-kernel packed gather GEMM [16x8x%d]: rel_rmse=%.2f%%  %s\n",K,100*sqrt(sse/ss),sqrt(sse/ss)<0.2?"<<< CORRECT (matches quantize_nvfp4 layout)":"WRONG");
+  return 0;
+}
diff --git a/rust/nvfp4_reference/nvfp4_peak.cu b/rust/nvfp4_reference/nvfp4_peak.cu
new file mode 100644
index 00000000..31ddb9f6
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_peak.cu
@@ -0,0 +1,25 @@
+#include <cstdio>
+__global__ void peak(float* out,int iters){
+  unsigned a0=0x24242424,a1=a0,a2=a0,a3=a0,b0=a0,b1=a0,sA=0x38383838,sB=0x38383838;
+  float c0=0,c1=0,c2=0,c3=0;
+  for(int i=0;i<iters;i++){
+    asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+     "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+     :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+    a0+=c0>1e30f?1:0; // fake dep to prevent hoist, never taken
+  }
+  if(c0==1234.5f)out[threadIdx.x]=c0+c1+c2+c3;
+}
+int main(){
+  int blocks=264, tpb=256, iters=20000; // 264*8 warps
+  float* d;cudaMalloc(&d,4096);
+  peak<<<blocks,tpb>>>(d,5); cudaDeviceSynchronize(); // warmup
+  cudaEvent_t s,e;cudaEventCreate(&s);cudaEventCreate(&e);
+  cudaEventRecord(s); peak<<<blocks,tpb>>>(d,iters); cudaEventRecord(e);cudaEventSynchronize(e);
+  float ms=0;cudaEventElapsedTime(&ms,s,e);
+  double warps=(double)blocks*(tpb/32);
+  double flop=warps*(double)iters*16*8*64*2;
+  printf("mxf4nvf4 m16n8k64 peak: %.1f TFLOP/s  (%.2f ms, %.0f warps x %d iters)\n",flop/(ms*1e-3)/1e12,ms,warps,iters);
+  // also int4 m16n8k64 for comparison (s32 acc)
+  return 0;
+}
diff --git a/rust/nvfp4_reference/nvfp4_tiled.cu b/rust/nvfp4_reference/nvfp4_tiled.cu
new file mode 100644
index 00000000..90954faf
--- /dev/null
+++ b/rust/nvfp4_reference/nvfp4_tiled.cu
@@ -0,0 +1,65 @@
+#include <cstdio>
+#include <cstdlib>
+#include <cmath>
+#include <vector>
+#include <algorithm>
+static const float E2M1[8]={0,.5f,1,1.5f,2,3,4,6};
+int enc(float x,float s){float v=x/s;int sg=v<0;v=fabsf(v);int b=0;float bd=1e30f;for(int i=0;i<8;i++){float d=fabsf(v-E2M1[i]);if(d<bd){bd=d;b=i;}}return (sg<<3)|b;}
+float qS(float s){if(s<=0)return 1e-9f;float bv=1e-9f,bd=1e30f;for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;bv=v;}}return bv;}
+unsigned char encS(float s){if(s<=0)return 0;int be=0,bm=0;float bd=1e30f;for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.f)*0.0078125f:(1.f+m/8.f)*exp2f((float)(e-7));float d=fabsf(s-v);if(d<bd){bd=d;be=e;bm=m;}}return (be<<3)|bm;}
+__device__ int b4b(int blk){int t[4]={0,2,1,3};return t[blk];}
+// MT=M/16 tiles, NT=N/8, KT=K/64. Apack[mt][kt][lane][4], Wpack[nt][kt][lane][2], Asc[mt][kt][lane], Wsc[nt][kt][gid]
+__global__ void gemm(const unsigned* Ap,const unsigned* Wp,const unsigned* Asc,const unsigned* Wsc,
+                     float* C,int MT,int NT,int KT,int N,float g){
+  int warp=(blockIdx.x*blockDim.x+threadIdx.x)>>5; int lane=threadIdx.x&31;
+  if(warp>=MT*NT)return; int mt=warp/NT, nt=warp%NT; int gid=lane>>2,tig=lane&3;
+  float c0=0,c1=0,c2=0,c3=0;
+  for(int kt=0;kt<KT;kt++){
+    const unsigned* A=Ap+((size_t)(mt*KT+kt)*32+lane)*4;
+    const unsigned* W=Wp+((size_t)(nt*KT+kt)*32+lane)*2;
+    unsigned a0=A[0],a1=A[1],a2=A[2],a3=A[3],b0=W[0],b1=W[1];
+    unsigned sA=Asc[(size_t)(mt*KT+kt)*32+lane], sB=Wsc[(size_t)(nt*KT+kt)*8+gid];
+    asm volatile("mma.sync.aligned.m16n8k64.row.col.kind::mxf4nvf4.block_scale.scale_vec::4X.f32.e2m1.e2m1.f32.ue4m3 "
+     "{%0,%1,%2,%3},{%4,%5,%6,%7},{%8,%9},{%0,%1,%2,%3},{%10},{0,0},{%11},{0,0};"
+     :"+f"(c0),"+f"(c1),"+f"(c2),"+f"(c3):"r"(a0),"r"(a1),"r"(a2),"r"(a3),"r"(b0),"r"(b1),"r"(sA),"r"(sB));
+  }
+  int r0=mt*16,col=nt*8+tig*2;
+  C[(r0+gid)*N+col]=c0*g;C[(r0+gid)*N+col+1]=c1*g;C[(r0+gid+8)*N+col]=c2*g;C[(r0+gid+8)*N+col+1]=c3*g;
+}
+int main(){
+  int M=96,N=1856,K=2688,MT=M/16,NT=N/8,KT=K/64;
+  std::vector<float> A((size_t)M*K),W((size_t)N*K);srand(3);
+  for(auto&x:A){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  for(auto&x:W){float u1=(rand()+1.)/(RAND_MAX+2.),u2=(rand()+1.)/(RAND_MAX+2.);x=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.05f;}
+  float gam=0,gwm=0;for(float x:A)gam=std::max(gam,fabsf(x));for(float x:W)gwm=std::max(gwm,fabsf(x));
+  float gA=gam/6.f/448.f,gW=gwm/6.f/448.f;
+  std::vector<unsigned> Ap((size_t)MT*KT*32*4,0),Wp((size_t)NT*KT*32*2,0),Asc((size_t)MT*KT*32,0),Wsc((size_t)NT*KT*8,0);
+  // pack A
+  for(int mt=0;mt<MT;mt++)for(int kt=0;kt<KT;kt++)for(int lane=0;lane<32;lane++){int gid=lane>>2,tig=lane&3;unsigned a[4]={0,0,0,0},sA=0;
+    for(int half=0;half<2;half++){int row=mt*16+gid+half*8;for(int Kk=0;Kk<64;Kk++){int blk=Kk>>4,w=Kk&15,tt=(blk>=2?2:0)+(w>=8?1:0);if(tt!=tig)continue;int nib=w&7;int gK=kt*64+Kk;
+      // block scale (block16) for this element
+      int k0=gK-(w);/*block start*/ int bstart=(gK/16)*16; float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(A[(size_t)row*K+bstart+j]));
+      float bs=qS((amax/6.f)/gA)*gA;if(bs<=0)bs=1e-9f; a[(blk&1?(half?3:2):(half?1:0))]|=(unsigned)enc(A[(size_t)row*K+gK],bs)<<(nib*4);}}
+    if(tig==0||tig==1){int row=mt*16+gid+(tig==1?8:0);for(int blk=0;blk<4;blk++){int bstart=(kt*64+blk*16);float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(A[(size_t)row*K+bstart+j]));sA|=(unsigned)encS((amax/6.f)/gA)<<(b4b(blk)*8);}}
+    Ap[((size_t)(mt*KT+kt)*32+lane)*4+0]=a[0];Ap[((size_t)(mt*KT+kt)*32+lane)*4+1]=a[1];Ap[((size_t)(mt*KT+kt)*32+lane)*4+2]=a[2];Ap[((size_t)(mt*KT+kt)*32+lane)*4+3]=a[3];Asc[(size_t)(mt*KT+kt)*32+lane]=sA;}
+  // pack W
+  for(int nt=0;nt<NT;nt++)for(int kt=0;kt<KT;kt++)for(int lane=0;lane<32;lane++){int gid=lane>>2,tig=lane&3;unsigned b[2]={0,0};int row=nt*8+gid;
+    for(int Kk=0;Kk<64;Kk++){int blk=Kk>>4,w=Kk&15,tt=(blk>=2?2:0)+(w>=8?1:0);if(tt!=tig)continue;int nib=w&7,reg=blk&1;int gK=kt*64+Kk;int bstart=(gK/16)*16;float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(W[(size_t)row*K+bstart+j]));float bs=qS((amax/6.f)/gW)*gW;if(bs<=0)bs=1e-9f;b[reg]|=(unsigned)enc(W[(size_t)row*K+gK],bs)<<(nib*4);}
+    Wp[((size_t)(nt*KT+kt)*32+lane)*2+0]=b[0];Wp[((size_t)(nt*KT+kt)*32+lane)*2+1]=b[1];
+    if(tig==0){unsigned sW=0;for(int blk=0;blk<4;blk++){int bstart=(kt*64+blk*16);float amax=0;for(int j=0;j<16;j++)amax=std::max(amax,fabsf(W[(size_t)row*K+bstart+j]));sW|=(unsigned)encS((amax/6.f)/gW)<<(b4b(blk)*8);}Wsc[(size_t)(nt*KT+kt)*8+gid]=sW;}}
+  unsigned *dAp,*dWp,*dAsc,*dWsc;float* dC;
+  cudaMalloc(&dAp,Ap.size()*4);cudaMalloc(&dWp,Wp.size()*4);cudaMalloc(&dAsc,Asc.size()*4);cudaMalloc(&dWsc,Wsc.size()*4);cudaMalloc(&dC,(size_t)M*N*4);
+  cudaMemcpy(dAp,Ap.data(),Ap.size()*4,cudaMemcpyHostToDevice);cudaMemcpy(dWp,Wp.data(),Wp.size()*4,cudaMemcpyHostToDevice);
+  cudaMemcpy(dAsc,Asc.data(),Asc.size()*4,cudaMemcpyHostToDevice);cudaMemcpy(dWsc,Wsc.data(),Wsc.size()*4,cudaMemcpyHostToDevice);
+  int warps=MT*NT,tpb=256,blocks=(warps*32+tpb-1)/tpb;
+  float g=gA*gW;
+  gemm<<<blocks,tpb>>>(dAp,dWp,dAsc,dWsc,dC,MT,NT,KT,N,g);cudaDeviceSynchronize();
+  cudaEvent_t s,e;cudaEventCreate(&s);cudaEventCreate(&e);int rep=50;
+  cudaEventRecord(s);for(int i=0;i<rep;i++)gemm<<<blocks,tpb>>>(dAp,dWp,dAsc,dWsc,dC,MT,NT,KT,N,g);cudaEventRecord(e);cudaEventSynchronize(e);
+  float ms=0;cudaEventElapsedTime(&ms,s,e);ms/=rep;
+  std::vector<float> h((size_t)M*N);cudaMemcpy(h.data(),dC,(size_t)M*N*4,cudaMemcpyDeviceToHost);
+  double sse=0,ss=0;for(int m=0;m<M;m++)for(int n=0;n<8;n++){double ref=0;for(int k=0;k<K;k++)ref+=(double)A[(size_t)m*K+k]*W[(size_t)n*K+k];double er=ref-h[(size_t)m*N+n];sse+=er*er;ss+=ref*ref;}
+  double flop=2.0*M*N*K;
+  printf("NVFP4 tiled GEMM [%dx%dx%d]: %.3f ms = %.1f TFLOP/s  rel_rmse(first8col)=%.2f%%\n",M,N,K,ms,flop/(ms*1e-3)/1e12,100*sqrt(sse/ss));
+  return 0;
+}
diff --git a/rust/nvfp4_reference/qnvfp4_test.cpp b/rust/nvfp4_reference/qnvfp4_test.cpp
new file mode 100644
index 00000000..c4f987ba
--- /dev/null
+++ b/rust/nvfp4_reference/qnvfp4_test.cpp
@@ -0,0 +1,40 @@
+#include <cstdio>
+#include <cmath>
+#include <cstdlib>
+#include <vector>
+#include <algorithm>
+// e2m1 codebook (magnitudes), index 0..7
+static const float E2M1[8]={0,0.5f,1,1.5f,2,3,4,6};
+float enc_e2m1(float x,float s,int&code){ // quantize x with scale s -> nearest e2m1*s
+  float v=x/s; int sign=v<0; v=fabsf(v); int best=0; float bd=1e30f;
+  for(int i=0;i<8;i++){float d=fabsf(v-E2M1[i]); if(d<bd){bd=d;best=i;}}
+  code=(sign<<3)|best; return (sign?-1:1)*E2M1[best]*s;
+}
+// ue4m3 quantize a positive scale value -> nearest representable (4exp bias7,3mant)
+float quant_ue4m3(float s){ if(s<=0)return 1e-9f; float best=1e30f,bd=1e30f;
+  for(int e=0;e<16;e++)for(int m=0;m<8;m++){float v=(e==0)?(m/8.0f)*0.0078125f:(1.0f+m/8.0f)*exp2f((float)(e-7)); float d=fabsf(s-v); if(d<bd){bd=d;best=v;}} return best; }
+int main(){
+  srand(1); int K=2688, N=64; // a representative weight block [N rows x K]
+  std::vector<float> W(N*K); double e=0;
+  // realistic-ish weights: gaussian * small
+  for(auto&w:W){ float u1=(rand()+1.0)/(RAND_MAX+2.0),u2=(rand()+1.0)/(RAND_MAX+2.0); w=sqrtf(-2*logf(u1))*cosf(6.2831853f*u2)*0.02f; }
+  // --- NVFP4: per-row block-16 ue4m3 scale, e2m1 codes, + per-tensor global ---
+  double sse_nv=0,ss=0; float gmax=0; for(float w:W)gmax=std::max(gmax,fabsf(w));
+  float global=gmax/6.0f/448.0f; // global so per-block amax/6 fits ue4m3 range (~448)
+  for(int n=0;n<N;n++)for(int b=0;b<K/16;b++){
+    float amax=0; for(int j=0;j<16;j++)amax=std::max(amax,fabsf(W[n*K+b*16+j]));
+    float bs=quant_ue4m3((amax/6.0f)/global)*global; if(bs<=0)bs=1e-9f;
+    for(int j=0;j<16;j++){int c; float r=enc_e2m1(W[n*K+b*16+j],bs,c); double d=r-W[n*K+b*16+j]; sse_nv+=d*d; ss+=(double)W[n*K+b*16+j]*W[n*K+b*16+j]; }
+  }
+  // --- Q4: block-32 signed int4 amax/7, f16 scale (approx as f32) ---
+  double sse_q4=0; for(int n=0;n<N;n++)for(int b=0;b<K/32;b++){
+    float amax=0; for(int j=0;j<32;j++)amax=std::max(amax,fabsf(W[n*K+b*32+j]));
+    float s=amax/7.0f; if(s<=0)s=1e-9f;
+    for(int j=0;j<32;j++){int q=(int)lroundf(W[n*K+b*32+j]/s); q=std::max(-7,std::min(7,q)); float r=q*s; double d=r-W[n*K+b*32+j]; sse_q4+=d*d; }
+  }
+  printf("weight [%dx%d]  ||W||^2=%.4g\n",N,K,ss);
+  printf("NVFP4 (e2m1 blk16 ue4m3+global) : rel_rmse = %.4f%%\n",100*sqrt(sse_nv/ss));
+  printf("Q4    (int4 blk32 amax/7 f16)   : rel_rmse = %.4f%%\n",100*sqrt(sse_q4/ss));
+  printf(sse_nv<=sse_q4*1.5?"NVFP4 accuracy ~comparable-or-better to Q4 -> argmax should hold\n":"NVFP4 noticeably worse -> argmax RISK, validate carefully\n");
+  return 0;
+}
diff --git a/swift/ffi-demo/main.swift b/swift/ffi-demo/main.swift
new file mode 100644
index 00000000..2b728238
--- /dev/null
+++ b/swift/ffi-demo/main.swift
@@ -0,0 +1,26 @@
+// Copyright 2026 Eric Kryski (@ekryski) and Tom Turney (@TheTom)
+// SPDX-License-Identifier: Apache-2.0
+//
+// Proof that Swift consumes the shared Butter Rust engine: this links the
+// libwh_butter_ffi static library and calls the C-ABI bridge in
+// rust/crates/wh-butter-ffi. Build + run with swift/ffi-demo/run.sh.
+import Foundation
+
+func take(_ p: UnsafeMutablePointer<CChar>?) -> String {
+    guard let p else { return "<null>" }
+    defer { butter_string_free(p) }
+    return String(cString: p)
+}
+
+print("Butter Rust engine, called from Swift")
+print("  version:           \(take(butter_version()))")
+print("  compiled backends: \(take(butter_compiled_backends()))")
+
+if let dev = butter_open_device() {
+    print("  live device:       [\(take(butter_device_backend(dev)))] \(take(butter_device_name(dev)))")
+    butter_close_device(dev)
+} else {
+    print("  live device:       none on this host")
+    print("  (expected on Apple — Swift's native Metal engine is primary here;")
+    print("   the same shared layer runs natively on CUDA via wh-butter-cuda.)")
+}
diff --git a/swift/ffi-demo/run.sh b/swift/ffi-demo/run.sh
new file mode 100755
index 00000000..e2203a6d
--- /dev/null
+++ b/swift/ffi-demo/run.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Build the Rust FFI staticlib + compile/link/run the Swift driver.
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+RUST="$HERE/../../rust"
+HDR="$RUST/crates/wh-butter-ffi/include"
+LIB="$RUST/target/debug"
+
+( cd "$RUST" && cargo build -p wh-butter-ffi )
+
+swiftc "$HERE/main.swift" \
+  -import-objc-header "$HDR/butter.h" \
+  -L "$LIB" -lwh_butter_ffi \
+  -o "$HERE/butter-ffi-demo"
+
+exec "$HERE/butter-ffi-demo"