Load speculative draft weights via the model's public load_weights - #10
Load speculative draft weights via the model's public load_weights#10aoshen02 wants to merge 1 commit into
Conversation
Route draft weight updates through the draft model's public load_weights instead of a bespoke per-parameter copy path. - Remove _speco_update_draft_weights and the _draft_param_name_candidates / _draft_fused_param_candidates / _load_draft_param helpers (the eagle3-only "graph-safe" path). - update_draft_weights_from_ipc now loads both eagle3 and dflash through draft_model.load_weights(...). Each draft wrapper (Eagle3LlamaForCausalLM / DFlashLagunaForCausalLM / DFlashQwen3ForCausalLM) already maps checkpoint-style names into the nested draft module and, for dflash, rebuilds the fused-KV buffers, so the manual inner_model._build_fused_kv_buffers() is dropped too. - Prefer vLLM's stable Worker.get_draft_model() accessor when resolving the draft. Requires vLLM's draft load_weights to prepend "model." idempotently (only when the name is not already prefixed); pairs with a vLLM-side change. Not yet GPU-verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: aoshen <aoshen@inferact.ai>
There was a problem hiding this comment.
Code Review
This pull request simplifies the draft model weight loading logic in verl_speco/integration/vllm_runtime.py by removing custom parameter matching and loading helpers. It now leverages vLLM's stable get_draft_model accessor when available and delegates weight loading directly to the draft model's public load_weights method. Feedback suggests checking if the proposer is None before attempting to resolve the draft model via get_draft_model to prevent potential runtime exceptions when speculative decoding is inactive.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| proposer = self._speco_resolve_draft_proposer() | ||
| # Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is | ||
| # resolved exactly the way vLLM does (V1 drafter / V2 speculator, and | ||
| # get_model() vs .model) instead of duplicating that logic here. | ||
| get_draft_model = getattr(self, "get_draft_model", None) | ||
| if callable(get_draft_model): | ||
| return get_draft_model(), proposer | ||
| # Fallback for vLLM versions without Worker.get_draft_model(). | ||
| if proposer is None: | ||
| return None, None |
There was a problem hiding this comment.
If proposer is None, it indicates that speculative decoding is not enabled or initialized. Checking proposer is None first avoids calling get_draft_model() when speculative decoding is inactive, which prevents potential AttributeError or other runtime exceptions depending on vLLM's internal state.
| proposer = self._speco_resolve_draft_proposer() | |
| # Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is | |
| # resolved exactly the way vLLM does (V1 drafter / V2 speculator, and | |
| # get_model() vs .model) instead of duplicating that logic here. | |
| get_draft_model = getattr(self, "get_draft_model", None) | |
| if callable(get_draft_model): | |
| return get_draft_model(), proposer | |
| # Fallback for vLLM versions without Worker.get_draft_model(). | |
| if proposer is None: | |
| return None, None | |
| proposer = self._speco_resolve_draft_proposer() | |
| if proposer is None: | |
| return None, None | |
| # Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is | |
| # resolved exactly the way vLLM does (V1 drafter / V2 speculator, and | |
| # get_model() vs .model) instead of duplicating that logic here. | |
| get_draft_model = getattr(self, "get_draft_model", None) | |
| if callable(get_draft_model): | |
| return get_draft_model(), proposer | |
| # Fallback for vLLM versions without Worker.get_draft_model(). |
What
Route speculative draft weight updates through the draft model's public
load_weights, instead of a bespoke per-parameter copy path._speco_update_draft_weightsand the_draft_param_name_candidates/_draft_fused_param_candidates/_load_draft_paramhelpers (the eagle3-only"graph-safe" path).
update_draft_weights_from_ipcnow loads both eagle3 and dflash viadraft_model.load_weights(...). Each draft wrapper(
Eagle3LlamaForCausalLM/DFlashLagunaForCausalLM/DFlashQwen3ForCausalLM)already maps checkpoint-style names into the nested draft module and, for
dflash, rebuilds the fused-KV buffers at the end — so the manual
inner_model._build_fused_kv_buffers()is dropped too.Worker.get_draft_model()accessor when resolving the draft.Net: +14 / −136 lines.
Why
The eagle3 path previously bypassed
load_weightsbecause vLLM's draftload_weightsunconditionally prepended"model."to non-lm_headnames,double-prefixing verl's internal names. Loading through the public
load_weightsmakes the RL-loop draft update identical to how the draft isloaded from checkpoint at startup, and drops ~120 lines of parallel
name-translation / fusion logic.
Dependency (please read)
This relies on vLLM's draft
load_weightsprepending"model."idempotently(only when the name is not already
model.-prefixed) — a paired vLLM-side change.Against a vLLM without that guard, eagle3 names would double-prefix. This PR
should land together with / after that vLLM change.
Testing
python -m py_compile verl_speco/integration/vllm_runtime.pypasses.draft weight update in an RL loop to confirm every translated name resolves
through the public loader and that dflash fused-KV rebuilds correctly. Marked
draft until verified.
AI assistance
Drafted with AI assistance (Claude). Must be reviewed line-by-line and
E2E-tested by a human maintainer before merge.