Skip to content

test(ci): wire P-EAGLE and Domino into the example test runner - #74

Merged
tpx818 merged 1 commit into
verl-project:mainfrom
khazic:khazic/test/peagle-domino-example-ci
Aug 31, 2026
Merged

test(ci): wire P-EAGLE and Domino into the example test runner#74
tpx818 merged 1 commit into
verl-project:mainfrom
khazic:khazic/test/peagle-domino-example-ci

Conversation

@khazic

@khazic khazic commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

What

ci/run_example_test.sh covers eagle3, megatron-eagle3, dflash and dspark. P-EAGLE and Domino have unit and integration coverage plus a standalone GPU smoke script, but no example lane, so the workflow they actually ship with is never exercised end to end.

Neither is an engine-level speculative algorithm, so neither can be trained inside the rollout loop. Both go through the two-stage separate-training workflow that examples/run_qwen3-8b_drafter_domino_peagle_separate_training.sh already implements:

stage entrypoint algorithm
1, collect verl_speco.main, mode=collect_only P-EAGLE reads EAGLE3 features, Domino reads DFlash features
2, train verl_speco.draft_train_launcher, mode=offline PEAGLE / DOMINO

This PR adds gpu/vllm/peagle and gpu/vllm/domino to the runner. They invoke that example once per stage, with RUN_STAGE=collect then RUN_STAGE=train, since the two stages need different Hydra overrides and the example applies "$@" to both.

The runner overrides only what CI has to control: where the two stages meet on disk, and the sizes that keep a smoke run cheap. The two-stage shape, the collect-to-train algorithm pairing and the per-algorithm hyperparameters stay owned by the example, so the lane tests the example as written instead of a private copy of it.

Stage 1 collects with a drafter that is already in the CI model cache (the EAGLE3 and DFlash drafters the existing lanes use), and stage 2 cold-starts its drafter from the target config, so no new model needs to be provisioned.

Two supporting fixes

The stage-2 device count could not be overridden. draft_train_launcher._find_override returns the first matching override rather than the last, the opposite of Hydra's last-wins rule that every other override in this lane relies on. The example passes speco.draft_training.num_gpus_per_node before "$@", so an appended value was silently ignored and stage 2 would have launched with the example's device count on a runner that does not have that many. The example now reads the count from DRAFT_TRAIN_GPUS_PER_NODE, which the runner sets from SPECO_ACCELERATOR_COUNT. (Changing the launcher to last-wins would be the deeper fix, but it changes behavior for existing callers, so it is left out of this PR.)

The dry-run tests could deadlock. They fed the runner into bash -s over stdin. In dry-run mode the runner exits early, which races the writer still filling the stdin pipe, and past roughly 14 KB of script the test hangs instead of failing. They now write the script to a temporary file and execute that. This reproduced on the runner as it stands in this PR, so it is fixed here rather than left for whoever next grows the file.

Scope

  • GPU only. P-EAGLE attends through torch.nn.attention.flex_attention, which the NPU runtime does not provide.
  • No workflow matrix includes the new lanes yet, so this PR costs no runner time. Enabling them is a one-line matrix addition in gpu_vllm_unit_tests.yml once you are happy with the wiring, and I am glad to do it here if you prefer.

Test

  • pytest tests/examples tests/integration tests/compat tests/config: 170 passed, 49 skipped.
  • bash -n ci/run_example_test.sh examples/*.sh.
  • Dry runs of both new lanes inspected by hand, asserting each stage gets the overrides it should and none of the other lane's.

Neither algorithm is servable by the rollout engines, so both are trained
through the two-stage separate-training workflow: stage 1 rolls out under
the engine algorithm whose hidden-state layout the drafter consumes and
writes a feature store, stage 2 trains the drafter offline from that store.
The example script for that workflow existed but no CI lane invoked it, so
`ci/run_example_test.sh` accepted only the single-invocation drafters.

Add `peagle` and `domino` cases that run the example twice, once per stage,
and cover both lanes with dry-run assertions. The runner overrides only what
CI must control (where the stages meet on disk, and the sizes that keep a
smoke run cheap); the two-stage shape and the per-algorithm hyperparameters
stay owned by the example.

Two supporting fixes:

* The standalone launcher resolves its device count from the first matching
  override, not the last, so an appended `num_gpus_per_node` could never
  lower the example's value and stage 2 would launch with the example's
  device count regardless of the runner. The example now reads that count
  from `DRAFT_TRAIN_GPUS_PER_NODE`.
* The dry-run tests fed the runner to `bash -s` over stdin, which races the
  script's early exit once the script grows past a few kilobytes. They now
  run it from a file.

Signed-off-by: khazic <khazzz1c@gmail.com>
@tpx818
tpx818 merged commit bc50a42 into verl-project:main Aug 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants