feat(speechlm2): add native MTP speculative decoding in vLLM - #16196
Conversation
a3aa648 to
47bafa8
Compare
5c5df5f to
fec1163
Compare
|
/ok to test fec1163 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
47bafa8 to
ad7ee0f
Compare
fec1163 to
9ddd6e9
Compare
|
/ok to test 9ddd6e9 |
@pzelasko, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 975069e |
df50825 to
f7bd44b
Compare
4cfaabc to
99997a9
Compare
|
/ok to test 99997a9 |
|
/ok to test ab12e19 |
|
/ok to test cc0f526 |
|
/ok to test 7ad6873 |
|
Stack propagation update: no PR-specific production-patch changes were added here. The existing five commits were rebased onto the repaired PR #16195 and are patch-identical under |
|
/ok to test d6ea026 |
|
/ok to test 05e0971 |
KunalDhawan
left a comment
KunalDhawan
left a comment
There was a problem hiding this comment.
Amazing work, LGTM!
One non-blocking note for future maintenance: this still depends on vLLM speculative decoding internals (MTPModelTypes / SpeculativeConfig.hf_config_override), so we should revalidate this path carefully on future vLLM bumps. Not a blocker for this PR.
|
/ok to test c8db68e |
|
/ok to test 55f8060 |
|
/ok to test 7b8a6f9 |
|
/ok to test cf2f432 |
|
/ok to test aac175d |
|
/ok to test 68054df |
|
[🤖]: Hi @pzelasko 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully. So it might be time to merge this PR or get some approvals. |
Serialize the built physical MTP pattern and depth, preserve repeated-head semantics, suppress stale legacy enablement, and expose the compatibility fields vLLM needs without persisting transient image-token state. Original-Commit: ff04e76 Original-Commit: 4a4cda4 Original-Commit: 0873181 Original-Commit: b8ecf5e Co-authored-by: slyne deng <slyned@nvidia.com> Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Register a spawn-safe speculative config hook and SpeechLM draft model, fuse target audio embeddings in the draft, split target/draft weights, pad vocabulary heads, support repeated physical layers, and remap packed expert tensors. Original-Commit: ff04e76 Original-Commit: fc2eb55 Original-Commit: 9b1d625 Original-Commit: 25c2ae6 Original-Commit: abef79f Original-Commit: 4a4cda4 Original-Commit: 0873181 Co-authored-by: slyne deng <slyned@nvidia.com> Co-authored-by: Codex <codex@openai.com> Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Keep the reconstructed MTP test file byte-identical to the immutable stack-top reference; the current formatter would otherwise rewrite this shared import. Original-Commit: 0873181 Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
3 participants
feat(speechlm2): add native MTP speculative decoding in vLLM
Base:
mainHead:
codex/pr16158-rewrite/05-mtp-servingTip:
2863841158fa513dbd9980a12543e02c020632fcSummary
Reviewer guide
The review question is: does MTP add a correct speculative draft model on top of the already reviewed target-only export and serving path?
Baseline checkpoint publication, portable encoder reconstruction, ordinary initialization, and ordinary multimodal inference are established by PR 4 and absent from this layer.
Attribution
The replay preserves original authors, dates, source-commit references, and author-matching DCO sign-offs.
Validation
git diff --check: passed.uv lock --check --offline: passed.