Add DFlash speculative decoding support to NeMo SpeechLM vLLM plugin

[Slyne]

Important

Stack 6/7. PRs #16192 through #16196 are merged. This PR supersedes #16129 and is now rooted directly at main; the dependency-only tip is #16241.

What does this PR do?

Add checkpoint-backed DFlash and DFlash2 speculative-decoding inference support for NeMo SpeechLM models served through vLLM.

Collection: SpeechLM (nemo.collections.speechlm2)

Changelog

• Register the NeMo SpeechLM target with the vLLM SupportsEagle3 hidden-state contract used by DFlash.
• Preserve the SALM audio-conditioned target path while exposing decoder hidden states required by the draft model.
• Route Automodel DFlash and DFlash2 checkpoint architectures through vLLM speculative decoding.
• Normalize DFlash2 architecture names only when required by the installed vLLM registry.
• Harden model-alias registration for lazy and class-backed vLLM registry entries.
• Fail clearly when a selected target backbone does not implement the required Eagle3 methods.
• Document DFlash2 inference, its V2 model-runner requirement, and the pinned vLLM runtime revision.
• Add focused coverage for registration, routing, multimodal hidden-state delegation, and supported target backbones.

Scope

This is inference-only support. It does not train, fine-tune, convert, or bundle DFlash or DFlash2 checkpoints. Draft checkpoints must be trained or fine-tuned separately for the target Lightning backbone.

Usage

vllm serve /path/to/vllm-ready-speechlm-checkpoint \
  --trust-remote-code \
  --speculative-config '{
    "method": "dflash",
    "model": "/path/to/vllm-ready-dflash-checkpoint",
    "num_speculative_tokens": 6
  }'

For DFlash2, use the pinned runtime documented in docs/source/speechlm2/vllm_dflash.rst and point model at a separately trained checkpoint declaring a supported DFlash2 architecture.

Validation

• Final Claude Code Opus max review: NO COMPLAINTS.
• Original complete SpeechLM vLLM plugin run in the pinned runtime: 135 passed.
• Current focused stack suite after rebasing: 403 passed, 20 skipped.
• Complete-stack pre-commit, compilation, git diff --check, and offline lock validation: passed.
• All seven branch commits replay patch-identically and retain their original authors and author-matching DCO sign-offs.
• End-to-end LibriSpeech evaluation: 5,559/5,559 unique outputs for each draft, with zero empty generations, zero non-stop completions, and zero reasoning output.

Model	LibriSpeech clean WER	LibriSpeech other WER	Acceptance rate	Mean acceptance length
Hero7 target-only reference	1.30%	3.34%	N/A	N/A
Original DFlash run	1.31%	3.31%	N/A	N/A
This branch, DFlash	1.29%	3.25%	45.27%	3.716
Original DFlash2 run	1.30%	3.38%	N/A	N/A
This branch, DFlash2	1.30%	3.25%	62.57%	4.754

WER values use the exact PR142 OpenASR leaderboard normalizer with thinking disabled.

GitHub Actions CI

Trusted PRs run automatically through copy-pr-bot. For an untrusted PR, a maintainer can trigger CI by commenting:

/ok to test 2f5c05d0238d8b76492dacdae38ceda9f82974bb

Before your PR is ready for review

• Followed the contributor guidelines.
• Added necessary tests.
• Added necessary documentation.
• Kept optional vLLM imports plugin-scoped and guarded.
• Merge feat(speechlm2): add native MTP speculative decoding in vLLM #16196 first, or retarget this PR appropriately.

Additional information

• Depends on feat(speechlm2): add native MTP speculative decoding in vLLM #16196.
• Supersedes Add DFlash speculative decoding support to NeMo SpeechLM vLLM plugin #16129.
• Followed by dependency-only build(speechlm2): add cut-cross-entropy to compiled extras #16241.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pzelasko]

/ok to test 0319f2f

[pzelasko]

/ok to test 2286574

[pzelasko]

/ok to test d0d99aa

The base branch was changed.

[pzelasko]

/ok to test 2f5c05d

[github-actions]

[🤖]: Hi @Slyne 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.