perf(k3): batched spec-round propose + e2e A/B and round anatomy - #936
Open
xiaguan wants to merge 3 commits into
Open
perf(k3): batched spec-round propose + e2e A/B and round anatomy#936xiaguan wants to merge 3 commits into
xiaguan wants to merge 3 commits into
Conversation
decode_spec proposed per slot — one dspark forward, one Markov readback, and one full pipeline drain per active slot per round. propose() always took the batch; the caller just never handed it one (disjoint &mut slot states). A sorted split_at_mut walk over the slot array hands out the states, and the drafts un-permute back to batch order for the verify split loop. Per round this is one draft forward over active*7 rows and a single device round-trip instead of N; the dense draft prolog/epilog rows now batch across slots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xiaguan <751080330@qq.com>
…wall SPEED-Bench EP16 A/B (spec wins 1.74x on low-entropy c1, loses 1.8x on high-entropy c1 and -27% throughput at c4 mixed) plus the nsys round anatomy: verify 61.4ms vs propose ~3ms, and the verify step is small-batch math, not launch overhead. Reorders the perf follow-up accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: xiaguan <751080330@qq.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 939cc69479
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (chunkwise FlashKDA vs the fused core — near-tie argmaxes flip); the | ||
| `spec_verify` gates certify what is exact instead. | ||
| `spec_verify` gates certify what is exact instead. E2e on the EP16 fleet | ||
| (SPEED-Bench, 2026-08-19): **1.74× on low-entropy c1, 1.8× SLOWER on |
There was a problem hiding this comment.
The reviewed commit message has no Signed-off-by: trailer, so the repository's DCO check will reject this commit regardless of the code changes. Recreate or rebase the commit with --signoff before merging.
AGENTS.md reference: AGENTS.md:L182-L182
Useful? React with 👍 / 👎.
Collaborator
Author
…gaMoE cross-tray stalls Signed-off-by: xiaguan <751080330@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two things from the agreed spec-decode perf follow-up (#931):
decode_specnow makes one drafter call per roundover every active slot instead of one per slot. A sorted
split_at_mutwalk hands
propose()the disjoint&mutslot states it alwayssupported; drafts un-permute back to batch order for the verify split
loop. One Markov readback / pipeline drain per round instead of N, and
the dense draft prolog/epilog rows batch across slots.
to come first — they did, and they reorder the rest of the plan
(documented in
docs/models/k3/mtp-dspark.md).Bench headlines (full 896-expert EP16, SPEED-Bench throughput_1k)
nsys on pruned EP4: round p50 64.6 ms = verify 61.4 + propose ~3. The
verify step is not launch-bound (5.5 ms of gaps): it's small-batch
math — small cuBLAS GEMMs 22.3 ms, MLA 15 ms, per-row
b1elementwise7.5 ms; MegaMoE only 9.6 ms. So the old "verify CUDA graph" idea buys
≤5 ms; packing the verify step's per-row/per-group math across rows is
the next real lever, and propose graphing only matters after that fixes
the geometry it would capture.
Validation
same GEMM shapes). c8 concurrent forks in the known near-tie noise
class (geometry change retiles GEMMs; feat(k3): speculative decoding with the RadixArk DSpark drafter #931 certified this class), with
per-slot acceptance in-band (dev 3.59 vs main 3.45 tokens/round mean,
no collapse → no anchor/state cross-wiring).
dspark_reference_cross_checkgreen (tray09, both propose rounds).spec_verify6/6 green on the branch build.pegainfer-k3lib tests,cargo fmt --checkclean.Bench artifacts:
~/bench-results/k3-spec-e2e-20260819/, trace~/k3-prof/specround.nsys-rep.🤖 Generated with Claude Code