perf(moe): precompute Q4_K scales for Muse concurrent decode - #1056
perf(moe): precompute Q4_K scales for Muse concurrent decode#1056shinyuser2345-svg wants to merge 1 commit into
Conversation
Expand the eight down-MMA weight scale/min pairs cooperatively in the B loaders and store them in a padded float2[32][9] shared array. This avoids serial metadata expansion, repeated consumer products and shared-bank aliasing, at a cost of 1536 additional shared bytes per CTA. The measured performance target is Muse Glimmer concurrent decode, not Qwen3.8 ModelOpt/DSpark. Local same-RTX-5090, source-built three-run medians against bc0e47b were: c8: 407.6 -> 418.6 aggregate tok/s (+2.699%) c16: 814.9 -> 836.5 aggregate tok/s (+2.651%) Command: qwen3_gguf_cb_bench "$GGUF" C 256 256 512 The full matrix, raw samples and correctness evidence are in PR gittensor-ai-lab#1055. These are local results; an official Muse verdict is still required. Keep both B loaders, split-K, row-width dispatch, MMA instructions and the remaining fold association unchanged. Cover scale/min unpacking, uneven shapes, both loaders, streams and graph replay with independent reference tests. Shared-code scope and cross-model guards remain intact. This is a commit-message-only rewording of 1bb4f59, with an identical source tree. The previous DSpark evaluation reported no verified speedup (+0.2%) and passed its correctness and no-regression gates. No new source change, GPU measurement, DSpark gain or scoring override is claimed here.
Maintainer request: complete Muse evaluationThis is the clearly linked replacement for closed PR #1055, which GitHub would not reopen after a message-only force-push, including after restoring its recorded head. The original request and evaluation history are retained. Please evaluate Muse Glimmer, including the complete scored matrix and all normal accuracy and Qwen3.6/ModelOpt no-regression guards. The claimed benefit is concurrent decode:
These are the original local same-RTX-5090, source-built three-run medians against Head The existing Muse evaluator already scores c8/c16. Muse and Shared/both remain checked because this is shared-kernel code. Could a maintainer ensure the full Muse result is considered, or clarify the permitted routing for this Muse-targeted shared-kernel change? No guard bypass, new axis, or manual tier is requested. |
sparkinfer DSpark auto-eval —
|
| metric | value |
|---|---|
| label | eval-dspark:none |
| scored at | DSpark decode + batched prefill @4k/@16k/@32k; target prefill @256k; concurrent decode @c2/c4/c8/c16/c32 on the ModelOpt NVFP4 checkpoint |
| winning axis | dspark-decode@4k — the dimension this label came from |
| PR DSpark @4k | 129.80 tok/s |
| main DSpark @4k | 129.83 tok/s |
| DSpark decode @4k vs main | +0.0% |
| PR prefill @4k | 14990.5 pp/s |
| main prefill @4k | 15011.9 pp/s |
| prefill @4k vs main | -0.1% |
| PR DSpark @16k | 130.47 tok/s |
| main DSpark @16k | 130.41 tok/s |
| DSpark decode @16k vs main | +0.0% |
| PR prefill @16k | 11656.2 pp/s |
| main prefill @16k | 11650.6 pp/s |
| prefill @16k vs main | +0.0% |
| PR DSpark @32k | 97.75 tok/s |
| main DSpark @32k | 97.79 tok/s |
| DSpark decode @32k vs main | +0.0% |
| PR prefill @32k | 10499.2 pp/s |
| main prefill @32k | 10526.7 pp/s |
| prefill vs main | -0.2% |
| PR prefill @256k | 4958.3 pp/s |
| main prefill @256k | 4973.0 pp/s |
| PR decode @256k | 94.73 tok/s |
| main decode @256k | 94.76 tok/s |
| prefill @256k vs main | -0.2% |
| PR concurrent decode @c2 | 186.9 tok/s agg |
| main concurrent decode @c2 | 187.0 tok/s agg |
| concurrent decode @c2 vs main | -0.1% |
| PR concurrent decode @C4 | 325.4 tok/s agg |
| main concurrent decode @C4 | 325.6 tok/s agg |
| concurrent decode @C4 vs main | -0.1% |
| PR concurrent decode @c8 | 535.6 tok/s agg |
| main concurrent decode @c8 | 536.7 tok/s agg |
| concurrent decode @c8 vs main | -0.2% |
| PR concurrent decode @c16 | 773.7 tok/s agg |
| main concurrent decode @c16 | 774.8 tok/s agg |
| concurrent decode @c16 vs main | -0.1% |
| PR concurrent decode @C32 | 1309.8 tok/s agg |
| main concurrent decode @C32 | 1309.8 tok/s agg |
| concurrent decode @C32 vs main | +0.0% |
| PR concurrent decode @c1 (floor) | 93.7 tok/s agg |
| main concurrent decode @c1 (floor) | 95.1 tok/s agg |
| concurrent @c1 vs main (floor) | -1.5% |
| PR AR tok/s (floor) | 88.56 |
| main AR tok/s (floor) | 88.61 |
| AR vs main (floor) | +0.0% |
| DSpark vs AR | 1.473× — above 1.0 means speculation finally pays |
| mean accept τ | 1.730 (main 1.730, ceiling 7) |
| accuracy gate | ✅ top1=1.000 (bar >=0.9) · KL=0.0000 (bar <=0.1) |
| cross-run determinism | ✅ identical output across 3 clean runs |
| losslessness gate | ✅ DSpark matches the AR reference token-for-token, verified across 3 independent runs |
| losslessness @4k | ✅ DSpark matches AR token-for-token, verified across 3 independent runs |
| losslessness @32k | ✅ DSpark matches AR token-for-token, verified across 3 independent runs |
| mean accept τ @4k | ✅ 1.6623 (main 1.6623) |
| mean accept τ floor | ✅ 1.7297 vs main 1.7297 (bar ≥95%) |
| mean accept τ @32k | ✅ 1.3299 (main 1.3299) |
| qwen3.8 (upstream unsloth) guard @16k | ✅ no regression (decode+prefill) — decode 82.8 tok/s · prefill 10030 pp |
| qwen3.6 guard @16k | ✅ no regression (decode+prefill) — decode 461.2 tok/s · prefill 27387 pp |
| PPL PR / main | 3.0301 / 3.0301 |
| Polaris receipt | collected, not signed (no key configured) |
| commit | a7bd73c2c |
within significance gate — not a verified dspark-decode@4k improvement
Scored on the pinned eval box vs same-box origin/main: DSpark speculative decode and production batched prefill at ctx=4k, ctx=16k and ctx=32k, plus target prefill at ctx=256k, on the ModelOpt NVFP4 checkpoint. The AR reference is measured in the same process and the same model load. Both a regression in AR decode and any divergence from the AR token sequence are hard REJECTs — a speculative decoder that is fast because it skips verification is not faster, it is wrong. τ is the lever, and the row above reports it against a block_size of 7. This is informational, not a judgment on your PR: a none label just means no measurable improvement on any scored axis was verified, which is expected and fine if that isn't what your change is about. Automated — merge behaviour depends on SPARKINFER_DSPARK_AUTOMERGE.
Closed: no verified speedup —
|
2 participants
Summary
Replacement submission for #1055, with the same source code. GitHub refused to reopen that PR after a commit-message-only force-push; restoring its recorded head and trying both REST and GraphQL did not reopen it. The original PR remains closed and unmerged. Its evaluation discussion and DSpark result remain part of this submission's history. This is not a second optimization or a claim for duplicate credit.
Performance target: Muse Glimmer concurrent decode at c8/c16. The local same-RTX-5090 measurements are +2.699% and +2.651%, respectively. These are local results awaiting official Muse verification; this PR does not claim a Qwen3.8 ModelOpt/DSpark speedup.
Evaluation status: the published DSpark result for the original head reported +0.2% (
none) with its correctness and no-regression gates passing. The requested next step is a complete run of the existing Muse evaluator, including all normal guards. The Muse c8/c16 axes already exist.Wording-only revision: pushed branch commit
a7bd73c2c9b37bc9b041f88880872e7e311d9f1frewords the commit message of1bb4f593659de504d71c51bd78221412324b1873. This replacement requests the outstanding complete Muse evaluation, with all normal accuracy and cross-model/no-regression guards. Their source trees are identical (f69c51b53937b7eea4ce2213390cfa1f04d516aa). The measurements and test results below were collected for the original source before this message-only revision; no new GPU run or additional speedup is claimed.Kernel change
Precompute the eight Q4_K down-MMA B scale/min pairs cooperatively while loading
each weight tile, replacing the single-lane metadata expansion and repeated
consumer multiplications. Pad the shared array to a nine-pair row stride to
avoid the consumer's four-way bank aliasing. This targets Lane 1, real measured
inference speedup.
Both existing B loaders are supported. Split-K, row-width dispatch, A staging,
quantization, the MMA instructions and the remaining fold association are
unchanged. A binary16 value times a six-bit integer fits exactly in FP32, so
storing that product introduces no additional rounding. The tradeoff is 1536
additional shared bytes per CTA (12.75 / 15.25 / 20.25 KiB at M8 / M16 / M32).
Proof of speedup
sm_120)Target model(s)
Decode tok/s (end-to-end Muse concurrent decode @c16, aggregate):
The source-built command is the evaluator's
qwen3_gguf_cb_bench "$GGUF" C 256 256 512,not an isolated-kernel microbenchmark.
bench/scripts/bench.shdefaults to releaseprebuilt binaries; these measurements use separate builds of clean main
bc0e47b799cf39b0429f346b5901bfee208e357eand this change. Same physical RTX 5090,driver 595.71.05, CUDA 12.8.93, SM120, no implementation-toggle overrides.
SPARKINFER_BENCH_SWEEP_CTXS/REPSare set only for the documented sweep recipe.Each CB axis has a discarded warmup for each build followed by six interleaved
measured legs (main/PR/PR/main/main/PR); table values are three-run medians.
Every leg checks
decode_tokens == C*256+8. Downloads and profiling were stoppedbefore timing. All raw measured samples are included below, not just the best run.
c32 exhibited two timing patterns in both builds; all samples are retained above.
The repeatable c8/c16 gains, not the highest individual c32 observation, are the
speedup claim. Every measured axis remains above the 98% no-regression floor.
Muse AR uses the unmodified sweep executable, 128 generated tokens, contexts
128/512 at five repeats, 4k/16k/32k at one repeat, and 64k at one repeat in a
separate load, matching the current Muse evaluator's recipe. Both cross-model
guards use 32k and five repeats. These are local reproductions; the official
evaluator's result remains authoritative.
Correctness and tests
LMCache integration internally skips when its external sidecar is unavailable;
this is not a claim of testing LMCache deployment.
production launcher. Independent pre-packing double-precision oracle covers
full six-bit scale/min metadata, eight distinct group activation magnitudes,
signed inputs and routing weights, two different experts, multiple K-split
counts and uneven splits, H/F 256/256, 1024/2304, 6656/19968, and M8/9/16/17/32.
execution and four CUDA-graph replays across A -> zero -> A. Final test passed
both loader modes on main and this PR (52 shape-width cases across four runs).
immediately (
got=177,want=80.8588639, tolerance=0.646870911). Mutant isolatedfrom the tested final build.
byte-identical at M8/9/16/17/32, twice each.
qwen3_gguf_score: baseline/candidate numeric outputbyte-identical, SHA256
002574803daad976ce0671340ef7ad50bbb2281daa29e5719747d20f839589e4.accuracy_compare.pyagainst native Muse llama.cpp5bda51bfbc62e64193221e639f6ad4e08767d760: 99 positions,top1=0.969697, KL=0.090603, PPL_spark=2.6881. Both accuracy bars pass.
No evaluator, benchmark script, frozen reference, scoring rule or other
maintainer-owned path is modified.
Reproducibility
Muse:
meta-models/Muse-Glimmer-30B-GGUFrevision70bf1b61ac09f91b24d39038091b41c582bc5d7a, KQuant17GB Q4_K_M GGUF,file SHA256
4cc57c0f51040a226e5a72cc47b7613f7772950e460a665f7083de89f183f60e.Qwen3.6 guard:
unsloth/Qwen3.6-35B-A3B-GGUFrevisiona483e9e6cbd595906af30beda3187c2663a1118c, UD-Q4_K_M SHA256ac0e2c1189e055faa36eff361580e79c5bd6f8e76bffb4ce547f167d53e31a61.Qwen3.8 guard:
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090revision5b7a687fc8211a5d631c8ca6a593dd37eb26ce33, both safetensors shards hash-verified.