Skip to content

Load speculative draft weights via the model's public load_weights - #10

Draft
aoshen02 wants to merge 1 commit into
verl-project:mainfrom
aoshen02:draft-update-via-public-load-weights
Draft

Load speculative draft weights via the model's public load_weights#10
aoshen02 wants to merge 1 commit into
verl-project:mainfrom
aoshen02:draft-update-via-public-load-weights

Conversation

@aoshen02

@aoshen02 aoshen02 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What

Route speculative draft weight updates through the draft model's public
load_weights, instead of a bespoke per-parameter copy path.

  • Remove _speco_update_draft_weights and the _draft_param_name_candidates /
    _draft_fused_param_candidates / _load_draft_param helpers (the eagle3-only
    "graph-safe" path).
  • update_draft_weights_from_ipc now loads both eagle3 and dflash via
    draft_model.load_weights(...). Each draft wrapper
    (Eagle3LlamaForCausalLM / DFlashLagunaForCausalLM / DFlashQwen3ForCausalLM)
    already maps checkpoint-style names into the nested draft module and, for
    dflash, rebuilds the fused-KV buffers at the end — so the manual
    inner_model._build_fused_kv_buffers() is dropped too.
  • Prefer vLLM's stable Worker.get_draft_model() accessor when resolving the draft.

Net: +14 / −136 lines.

Why

The eagle3 path previously bypassed load_weights because vLLM's draft
load_weights unconditionally prepended "model." to non-lm_head names,
double-prefixing verl's internal names. Loading through the public
load_weights makes the RL-loop draft update identical to how the draft is
loaded from checkpoint at startup, and drops ~120 lines of parallel
name-translation / fusion logic.

Dependency (please read)

This relies on vLLM's draft load_weights prepending "model." idempotently
(only when the name is not already model.-prefixed) — a paired vLLM-side change.
Against a vLLM without that guard, eagle3 names would double-prefix. This PR
should land together with / after that vLLM change.

Testing

  • python -m py_compile verl_speco/integration/vllm_runtime.py passes.
  • ⚠️ Not yet GPU-verified. Needs an end-to-end run of eagle3 and dflash
    draft weight update in an RL loop to confirm every translated name resolves
    through the public loader and that dflash fused-KV rebuilds correctly. Marked
    draft until verified.

AI assistance

Drafted with AI assistance (Claude). Must be reviewed line-by-line and
E2E-tested by a human maintainer before merge.

Route draft weight updates through the draft model's public load_weights
instead of a bespoke per-parameter copy path.

- Remove _speco_update_draft_weights and the _draft_param_name_candidates /
  _draft_fused_param_candidates / _load_draft_param helpers (the eagle3-only
  "graph-safe" path).
- update_draft_weights_from_ipc now loads both eagle3 and dflash through
  draft_model.load_weights(...). Each draft wrapper (Eagle3LlamaForCausalLM /
  DFlashLagunaForCausalLM / DFlashQwen3ForCausalLM) already maps checkpoint-style
  names into the nested draft module and, for dflash, rebuilds the fused-KV
  buffers, so the manual inner_model._build_fused_kv_buffers() is dropped too.
- Prefer vLLM's stable Worker.get_draft_model() accessor when resolving the draft.

Requires vLLM's draft load_weights to prepend "model." idempotently (only when
the name is not already prefixed); pairs with a vLLM-side change. Not yet
GPU-verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: aoshen <aoshen@inferact.ai>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request simplifies the draft model weight loading logic in verl_speco/integration/vllm_runtime.py by removing custom parameter matching and loading helpers. It now leverages vLLM's stable get_draft_model accessor when available and delegates weight loading directly to the draft model's public load_weights method. Feedback suggests checking if the proposer is None before attempting to resolve the draft model via get_draft_model to prevent potential runtime exceptions when speculative decoding is inactive.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 1102 to 1111
proposer = self._speco_resolve_draft_proposer()
# Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is
# resolved exactly the way vLLM does (V1 drafter / V2 speculator, and
# get_model() vs .model) instead of duplicating that logic here.
get_draft_model = getattr(self, "get_draft_model", None)
if callable(get_draft_model):
return get_draft_model(), proposer
# Fallback for vLLM versions without Worker.get_draft_model().
if proposer is None:
return None, None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If proposer is None, it indicates that speculative decoding is not enabled or initialized. Checking proposer is None first avoids calling get_draft_model() when speculative decoding is inactive, which prevents potential AttributeError or other runtime exceptions depending on vLLM's internal state.

Suggested change
proposer = self._speco_resolve_draft_proposer()
# Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is
# resolved exactly the way vLLM does (V1 drafter / V2 speculator, and
# get_model() vs .model) instead of duplicating that logic here.
get_draft_model = getattr(self, "get_draft_model", None)
if callable(get_draft_model):
return get_draft_model(), proposer
# Fallback for vLLM versions without Worker.get_draft_model().
if proposer is None:
return None, None
proposer = self._speco_resolve_draft_proposer()
if proposer is None:
return None, None
# Prefer vLLM's stable accessor (Worker.get_draft_model) so the draft is
# resolved exactly the way vLLM does (V1 drafter / V2 speculator, and
# get_model() vs .model) instead of duplicating that logic here.
get_draft_model = getattr(self, "get_draft_model", None)
if callable(get_draft_model):
return get_draft_model(), proposer
# Fallback for vLLM versions without Worker.get_draft_model().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant