perf(moe): precompute Q4_K scales for Muse concurrent decode - #1055
shinyuser2345-svg wants to merge 1 commit into
Conversation
sparkinfer DSpark auto-eval —
|
| metric | value |
|---|---|
| label | eval-dspark:none |
| scored at | DSpark decode + batched prefill @4k/@16k/@32k; target prefill @256k; concurrent decode @c2/c4/c8/c16/c32 on the ModelOpt NVFP4 checkpoint |
| winning axis | cb-decode@c4 — the dimension this label came from |
| PR DSpark @4k | 129.89 tok/s |
| main DSpark @4k | 129.98 tok/s |
| DSpark decode @4k vs main | +0.0% |
| PR prefill @4k | 15044.6 pp/s |
| main prefill @4k | 15075.8 pp/s |
| prefill @4k vs main | -0.2% |
| PR DSpark @16k | 130.62 tok/s |
| main DSpark @16k | 130.45 tok/s |
| DSpark decode @16k vs main | +0.1% |
| PR prefill @16k | 11708.8 pp/s |
| main prefill @16k | 11717.2 pp/s |
| prefill @16k vs main | +0.0% |
| PR DSpark @32k | 97.82 tok/s |
| main DSpark @32k | 97.82 tok/s |
| DSpark decode @32k vs main | +0.0% |
| PR prefill @32k | 10530.3 pp/s |
| main prefill @32k | 10551.0 pp/s |
| prefill vs main | -0.1% |
| PR prefill @256k | 4989.9 pp/s |
| main prefill @256k | 5000.4 pp/s |
| PR decode @256k | 94.79 tok/s |
| main decode @256k | 94.79 tok/s |
| prefill @256k vs main | -0.2% |
| PR concurrent decode @c2 | 187.0 tok/s agg |
| main concurrent decode @c2 | 187.3 tok/s agg |
| concurrent decode @c2 vs main | -0.2% |
| PR concurrent decode @C4 | 326.2 tok/s agg |
| main concurrent decode @C4 | 325.4 tok/s agg |
| concurrent decode @C4 vs main | +0.2% |
| PR concurrent decode @c8 | 537.2 tok/s agg |
| main concurrent decode @c8 | 537.8 tok/s agg |
| concurrent decode @c8 vs main | -0.1% |
| PR concurrent decode @c16 | 774.9 tok/s agg |
| main concurrent decode @c16 | 775.4 tok/s agg |
| concurrent decode @c16 vs main | -0.1% |
| PR concurrent decode @C32 | 1312.1 tok/s agg |
| main concurrent decode @C32 | 1312.1 tok/s agg |
| concurrent decode @C32 vs main | +0.0% |
| PR concurrent decode @c1 (floor) | 95.2 tok/s agg |
| main concurrent decode @c1 (floor) | 95.1 tok/s agg |
| concurrent @c1 vs main (floor) | +0.1% |
| PR AR tok/s (floor) | 88.63 |
| main AR tok/s (floor) | 88.64 |
| AR vs main (floor) | +0.0% |
| DSpark vs AR | 1.474× — above 1.0 means speculation finally pays |
| mean accept τ | 1.730 (main 1.730, ceiling 7) |
| accuracy gate | ✅ top1=1.000 (bar >=0.9) · KL=0.0000 (bar <=0.1) |
| cross-run determinism | ✅ identical output across 3 clean runs |
| losslessness gate | ✅ DSpark matches the AR reference token-for-token, verified across 3 independent runs |
| losslessness @4k | ✅ DSpark matches AR token-for-token, verified across 3 independent runs |
| losslessness @32k | ✅ DSpark matches AR token-for-token, verified across 3 independent runs |
| mean accept τ @4k | ✅ 1.6623 (main 1.6623) |
| mean accept τ floor | ✅ 1.7297 vs main 1.7297 (bar ≥95%) |
| mean accept τ @32k | ✅ 1.3299 (main 1.3299) |
| qwen3.8 (upstream unsloth) guard @16k | ✅ no regression (decode+prefill) — decode 82.8 tok/s · prefill 10087 pp |
| qwen3.6 guard @16k | ✅ no regression (decode+prefill) — decode 460.5 tok/s · prefill 27518 pp |
| PPL PR / main | 3.0301 / 3.0301 |
| Polaris receipt | collected, not signed (no key configured) |
| commit | 1bb4f5936 |
within significance gate — not a verified cb-decode@c4 improvement
Scored on the pinned eval box vs same-box origin/main: DSpark speculative decode and production batched prefill at ctx=4k, ctx=16k and ctx=32k, plus target prefill at ctx=256k, on the ModelOpt NVFP4 checkpoint. The AR reference is measured in the same process and the same model load. Both a regression in AR decode and any divergence from the AR token sequence are hard REJECTs — a speculative decoder that is fast because it skips verification is not faster, it is wrong. τ is the lever, and the row above reports it against a block_size of 7. This is informational, not a judgment on your PR: a none label just means no measurable improvement on any scored axis was verified, which is expected and fine if that isn't what your change is about. Automated — merge behaviour depends on SPARKINFER_DSPARK_AUTOMERGE.
Closed: no verified speedup —
|
|
Update: the replacement is open as PR #1056. GitHub refused to reopen #1055 after the commit-message-only force-push, including after restoring the recorded head and trying REST and GraphQL. Please continue the requested complete Muse evaluation on #1056. The source is unchanged; no new GPU result, extra optimization, or duplicate credit is claimed. This original PR remains closed and unmerged. Original Muse evaluation request, retained for historyMuse evaluation request — wording clarified, source unchangedPlease evaluate this PR with the existing Muse Glimmer evaluator, including its complete scored matrix, accuracy checks, and Qwen3.6/ModelOpt no-regression guards. The commit message, PR title, and opening summary now explicitly identify Muse concurrent decode at c8/c16 as the performance target. Pushed branch commit The original local, source-built RTX 5090 measurements against
Command: The original head's DSpark verdict remains valid historical evidence: no verified speedup (+0.2%), with its correctness and no-regression gates passing. I am not disputing it or requesting a DSpark reward. Both claimed axes are already in the Muse evaluator. No new axis is needed. Muse and Shared/both remain checked because this touches a shared kernel, per the contribution guidance. Could a maintainer coordinate reopening for that complete Muse run, or clarify the permitted model declaration for a Muse-targeted shared-kernel change? I have not reopened it merely to retry the same DSpark measurement. No guard bypass, manual tier, or scoring override is requested. |
Expand the eight down-MMA weight scale/min pairs cooperatively in the B loaders and store them in a padded float2[32][9] shared array. This avoids serial metadata expansion, repeated consumer products and shared-bank aliasing, at a cost of 1536 additional shared bytes per CTA. The measured performance target is Muse Glimmer concurrent decode, not Qwen3.8 ModelOpt/DSpark. Local same-RTX-5090, source-built three-run medians against bc0e47b were: c8: 407.6 -> 418.6 aggregate tok/s (+2.699%) c16: 814.9 -> 836.5 aggregate tok/s (+2.651%) Command: qwen3_gguf_cb_bench "$GGUF" C 256 256 512 The full matrix, raw samples and correctness evidence are in PR gittensor-ai-lab#1055. These are local results; an official Muse verdict is still required. Keep both B loaders, split-K, row-width dispatch, MMA instructions and the remaining fold association unchanged. Cover scale/min unpacking, uneven shapes, both loaders, streams and graph replay with independent reference tests. Shared-code scope and cross-model guards remain intact. This is a commit-message-only rewording of 1bb4f59, with an identical source tree. The previous DSpark evaluation reported no verified speedup (+0.2%) and passed its correctness and no-regression gates. No new source change, GPU measurement, DSpark gain or scoring override is claimed here.
shinyuser2345-svg
changed the title
2 participants
Summary
Continued in replacement PR #1056. GitHub refused to reopen this PR after the message-only force-push, including after restoring its recorded head. This PR remains closed and unmerged; its evidence and evaluation history are preserved here.
Performance target: Muse Glimmer concurrent decode at c8/c16. The local same-RTX-5090 measurements are +2.699% and +2.651%, respectively. These are local results awaiting official Muse verification; this PR does not claim a Qwen3.8 ModelOpt/DSpark speedup.
Evaluation status: the published DSpark result for the original head reported +0.2% (
none) with its correctness and no-regression gates passing. The requested next step is a complete run of the existing Muse evaluator, including all normal guards. The Muse c8/c16 axes already exist.Wording-only revision: pushed branch commit
a7bd73c2c9b37bc9b041f88880872e7e311d9f1frewords the commit message of1bb4f593659de504d71c51bd78221412324b1873. The same source is now submitted in replacement PR #1056; this closed PR retains the original head and its evaluation history. Their source trees are identical (f69c51b53937b7eea4ce2213390cfa1f04d516aa). The measurements and test results below were collected for the original source before this message-only revision; no new GPU run or additional speedup is claimed.Kernel change
Precompute the eight Q4_K down-MMA B scale/min pairs cooperatively while loading
each weight tile, replacing the single-lane metadata expansion and repeated
consumer multiplications. Pad the shared array to a nine-pair row stride to
avoid the consumer's four-way bank aliasing. This targets Lane 1, real measured
inference speedup.
Both existing B loaders are supported. Split-K, row-width dispatch, A staging,
quantization, the MMA instructions and the remaining fold association are
unchanged. A binary16 value times a six-bit integer fits exactly in FP32, so
storing that product introduces no additional rounding. The tradeoff is 1536
additional shared bytes per CTA (12.75 / 15.25 / 20.25 KiB at M8 / M16 / M32).
Proof of speedup
sm_120)Target model(s)
Decode tok/s (end-to-end Muse concurrent decode @c16, aggregate):
The source-built command is the evaluator's
qwen3_gguf_cb_bench "$GGUF" C 256 256 512,not an isolated-kernel microbenchmark.
bench/scripts/bench.shdefaults to releaseprebuilt binaries; these measurements use separate builds of clean main
bc0e47b799cf39b0429f346b5901bfee208e357eand this change. Same physical RTX 5090,driver 595.71.05, CUDA 12.8.93, SM120, no implementation-toggle overrides.
SPARKINFER_BENCH_SWEEP_CTXS/REPSare set only for the documented sweep recipe.Each CB axis has a discarded warmup for each build followed by six interleaved
measured legs (main/PR/PR/main/main/PR); table values are three-run medians.
Every leg checks
decode_tokens == C*256+8. Downloads and profiling were stoppedbefore timing. All raw measured samples are included below, not just the best run.
c32 exhibited two timing patterns in both builds; all samples are retained above.
The repeatable c8/c16 gains, not the highest individual c32 observation, are the
speedup claim. Every measured axis remains above the 98% no-regression floor.
Muse AR uses the unmodified sweep executable, 128 generated tokens, contexts
128/512 at five repeats, 4k/16k/32k at one repeat, and 64k at one repeat in a
separate load, matching the current Muse evaluator's recipe. Both cross-model
guards use 32k and five repeats. These are local reproductions; the official
evaluator's result remains authoritative.
Correctness and tests
LMCache integration internally skips when its external sidecar is unavailable;
this is not a claim of testing LMCache deployment.
production launcher. Independent pre-packing double-precision oracle covers
full six-bit scale/min metadata, eight distinct group activation magnitudes,
signed inputs and routing weights, two different experts, multiple K-split
counts and uneven splits, H/F 256/256, 1024/2304, 6656/19968, and M8/9/16/17/32.
execution and four CUDA-graph replays across A -> zero -> A. Final test passed
both loader modes on main and this PR (52 shape-width cases across four runs).
immediately (
got=177,want=80.8588639, tolerance=0.646870911). Mutant isolatedfrom the tested final build.
byte-identical at M8/9/16/17/32, twice each.
qwen3_gguf_score: baseline/candidate numeric outputbyte-identical, SHA256
002574803daad976ce0671340ef7ad50bbb2281daa29e5719747d20f839589e4.accuracy_compare.pyagainst native Muse llama.cpp5bda51bfbc62e64193221e639f6ad4e08767d760: 99 positions,top1=0.969697, KL=0.090603, PPL_spark=2.6881. Both accuracy bars pass.
No evaluator, benchmark script, frozen reference, scoring rule or other
maintainer-owned path is modified.
Reproducibility
Muse:
meta-models/Muse-Glimmer-30B-GGUFrevision70bf1b61ac09f91b24d39038091b41c582bc5d7a, KQuant17GB Q4_K_M GGUF,file SHA256
4cc57c0f51040a226e5a72cc47b7613f7772950e460a665f7083de89f183f60e.Qwen3.6 guard:
unsloth/Qwen3.6-35B-A3B-GGUFrevisiona483e9e6cbd595906af30beda3187c2663a1118c, UD-Q4_K_M SHA256ac0e2c1189e055faa36eff361580e79c5bd6f8e76bffb4ce547f167d53e31a61.Qwen3.8 guard:
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090revision5b7a687fc8211a5d631c8ca6a593dd37eb26ce33, both safetensors shards hash-verified.