[Roadmap] Speculative decoding (qwen3 first, shared primitives)

[xiaguan]

Consolidating the speculative-decoding threads into one place so we coordinate instead of colliding. Grew out of #325. cc @wjinxu @scatyf3 @kitty-eu-org @CAICAIIs — please holler on / claim the lanes you want.

Where we are

DFlash speculative decoding shipped on main (#436 + #442): the full loop — draft model loading → batched draft forward → target "verify" forward (piecewise CUDA graph) → acceptance → KV rollback → OpenAI serving — behind --dflash-draft-model-path. Greedy-lossless (lm-eval gsm8k strict-match identical spec on/off), validated on 5070 Ti / 5090, matches or beats vLLM dflash across c1/c8/c16. Design notes: docs/models/qwen3/dflash-speculative-decoding.md.

So speculation is a real subsystem now, not an experiment. This roadmap is about growing it into a first-class, multi-method, multi-model one — without each method/model reimplementing the controller.

Workstreams

Brackets = current lead / interested. Shout if you want to take or share one.

Methods

• DFlash [@xiaguan] — shipped. Next: draft-side piecewise CUDA graph (the remaining ~16% of the decode launch gap).
• EAGLE / EAGLE3 [@scatyf3, feat(qwen3): explore speculative decoding #325] — orthogonal draft method. We deliberately deferred the proposer-trait abstraction to land EAGLE cleanly, so this is the natural driver for the shared proposer interface below.
• n-gram / prompt-lookup [@wjinxu, PR: feat(qwen3): n-gram speculative decoding #349] — no draft weights; great for surfacing exactly what the scheduler / paged KV / CUDA graph / sampling actually need from a speculator.
• Qwen3.5 DFlash [@CAICAIIs, qwen35: explore DFlash speculative decoding #434] — extend DFlash to the qwen3.5 line.

Performance

• Draft-side piecewise CUDA graph. @kitty-eu-org's feat(qwen3-4b-dflash): add Qwen3-4B-DFlash draft model crate #439 has draft-forward kernels (non-causal dense/ragged-batch prefill attention, K-only norm+RoPE) that look directly reusable here — if they beat what's on main, worth landing as standalone kernel PRs.

Capability & correctness

• Speculative sampling. Today speculation is greedy-only: a sampling (temperature / top_k / top_p) request correctly falls back to plain decode — no wrong distribution, but no speedup, and one sampling request currently disables speculation for its whole batch. Lossless rejection-sampling acceptance (accept with min(1, p_target/q_draft), resample from (p_target − q_draft)₊ on reject) would unlock the speedup for non-greedy requests. Lives in the verify controller.

Usability & metrics

• Acceptance-rate / draft-hit / speculative-token-throughput metrics, serving semantics, config ergonomics.

Shared primitives

• Factor the reusable pieces — proposer trait, verify + accept, draft cache, KV rollback — into a shared module (the way sampling lives in a shared sampler), so EAGLE3 / n-gram / DFlash / per-model engines don't each reimplement the controller.

How to plug in

Comment with the lane you want and we'll coordinate ownership here. The #439 timing overlap — a parallel draft crate that landed after the integration shipped — is exactly what this thread is meant to prevent: let's claim lanes before writing thousands of lines. 🙂

Related: #325 (origin), #349 (n-gram), #439 (draft crate), #434 (qwen3.5 DFlash).

[kitty-eu-org]

I'd love to claim the lane for Qwen3-4B-DFlash integration. I can handle the refactoring to properly port the draft model crate into the main branch and adapt it to the shared primitives we're setting up. Let's coordinate here! 🙌

[xiaguan]

Thanks @kitty-eu-org, glad you want in!

The DFlash code already on main (#436 + #442) could definitely use more tidying up — that'd be a great place to help, and it'll set things up nicely for the n-gram (#349) and EAGLE-3 (#454) PRs coming next. Want to take that?

[n-WN]

Tree-draft notes for whoever takes the tree lane (analysis only — I know this is being worked on; #707 is sensibly chain-first). I surveyed how engines reconcile tree drafting with CUDA graphs; conclusions:

Ship: fixed-envelope masked tree (the SGLang pattern).

1. Fix num_steps (depth), topk (branching), num_draft_tokens (total verify budget) as server config. Tensor shapes never vary at runtime.
2. Build the tree on GPU with fixed-shape outputs: tree mask, positions, retrieve index/next-token/next-sibling arrays (SGLang's build_tree_kernel_efficient in sgl-kernel is the reference). EAGLE-2-style score reranking lives inside this kernel — the "dynamic tree" survives only as tensor contents, never shapes. That is what SGLang actually ships; EAGLE-2's threshold-based expansion pruning with variable node counts is not shipped by any engine.
3. Verify with custom-mask tree attention at fixed draft_tokens per request; acceptance walk on GPU (verify_tree_greedy-style); commit/rewind KV pages for the accepted path only.
4. Even more graph-friendly alternative: Sequoia-style offline-optimized static topology — the mask is fully precomputed at startup (their paper explicitly uses PyTorch CUDA graphs for this reason), so no tree-build kernel at all.
5. Offer 2–3 static presets ("latency" vs "economy" tree sizes) selected at startup — bucketed graphs, not dynamic shapes.

Avoid:

• Input-dependent per-step topologies (OPT-Tree, DySpec, DDD) unless degenerated to fixed-budget + mask as above.
• ProPD-style mid-forward branch pruning with mask regeneration inside the target forward — worst possible graph fit.
• Any CPU-side tree construction or per-step host sync in the hot path — this is exactly why SGLang's overlapped SpecV2 path is topk=1-only today.
• Per-request variable tree shapes. (Per-request variable verify lengths are different — that's a scheduler problem and tractable; see the adaptive-verify-window RFC.)

Sobering datapoints before sizing the effort:

• vLLM benchmarked tree decoding and deliberately ships none: past low request rates, verifying ~64 tokens to gain 1–2 accepted ones loses (Red Hat vLLM/EAGLE-3 blog). Their V1 verifier is chain-only.
• EAGLE-3 paper's own engine number: 6.5× at bs=1 → 1.38× at bs=64 (SGLang). Tree/chain choice is a latency feature at low concurrency; at serving load the verify budget matters more than the tree shape (which is what the confidence-scheduled variable-window line attacks).
• Kernel options when the time comes: Triton custom-mask path (SGLang), FlashInfer custom mask, or DeFT (arXiv 2404.00242) if a dedicated flash tree-attention kernel is wanted. Watch graph memory (buckets × tree size) and mamba/SSM state rewind on the accepted fork if hybrid models ever get speculation.

Related: opened #717 (DSpark Phase 2 confidence truncation, near-term) and #718 (Phase 3 adaptive-verify-window RFC).