[cfg, recipe, tests] feat: enable DAPO dynamic sampling on V1 omni loop (#446 Phase 3)

[OnePunchMonk]

Summary

Implements Phase 3 of #446 (Qwen3-Omni Thinker DAPO): enables dynamic
sampling (group filtering) on the V1 omni trainer.

Stacked on #502 (Phase 2), which is itself stacked on #456 (Phase 1) —
neither merged yet. This branch is built on top of dapo-p2-overlong-reward,
so the diff against main includes both #456's and #502's commits. Please
review only the commit on top
(feat: enable DAPO dynamic sampling on V1 omni loop (#446 Phase 3)) until
#456 and #502 land; I'll rebase and shrink the diff as each merges.

What's actually wired

Phase 0 (documented in #456) already found that upstream verl's V1
PPOTrainer builds a refillable replay buffer that natively implements group
filtering (algorithm.filter_groups) — OmniPPOTrainerSync subclasses
PPOTrainerSync without overriding the generation loop, so this consumes it
unchanged. Phase 3 is therefore config-only, no new trainer code:

• Adds examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh,
  a copy of the Phase 1 AVQA launcher with algorithm.filter_groups.enable=true,
  algorithm.filter_groups.metric=acc, and algorithm.filter_groups.max_inflight_gen_batches=2.
  metric=acc is the reliable key: naive/dapo reward managers always
  populate reward_extra_info["acc"] regardless of the custom reward
  function's own return keys (verl.experimental.reward_loop.reward_manager.naive),
  and this key is already exercised in tests/workers/config/test_omni_config_on_cpu.py's
  OmniAlgoConfig Hydra-conversion contract from [cfg, recipe, tests] feat: add Qwen3-Omni Thinker DAPO phase one #456.
• Switches the reward manager to dapo (source=register), reusing [cfg, recipe, tests] feat: wire DAPO overlong reward buffer on V1 omni loop #502's
  overlong-buffer wiring — group filtering requires the streaming reward path
  (trainer_base.py asserts not reward.reward_model.enable or reward.reward_model.enable_resource_pool), which the colocated default
  reward model does not satisfy.
• Adds test_dapo_dynamic_sampling_example_launcher_has_phase_three_contract
  to tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py, asserting the
  filter_groups/reward-manager contract and that the shared token-level DAPO
  policy settings (clip-higher, token-mean, GRPO, no KL) still hold.
• Documents the design in examples/dapo_trainer/README.md.

Not in scope: MMK12/AVQA-specific reward-manager tuning beyond what
Phase 1/2 already validate, and any new trainer-side dynamic-sampling code —
Phase 0 confirmed none is needed.

Test plan

CPU (all green locally):

python3 -m pytest tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py -q

Not yet run: a GPU e2e smoke exercising algorithm.filter_groups.enable=true
end-to-end (synthetic all-correct/all-wrong groups getting dropped and
re-generated, per #446's Phase 3 exit criterion). I don't have further
Modal/GPU access in this pass — flagging this explicitly rather than claiming
GPU coverage I don't have. This PR stays in draft until that smoke is run and
until #456/#502 land underneath it.

AI usage disclosure

Claude Code was used to trace the config contract (confirming filter_groups
is handled entirely in upstream verl's V1 replay buffer, not by
OmniPPOTrainerSync) and to write the launcher and tests above. A human
submitter (onepunchmonk) will review every changed line and run the GPU
smoke before taking this out of draft.

[Copilot]

🟡 Changes recommended

A new CPU unit test currently depends on downloading a HuggingFace tokenizer at runtime, which risks CI failures in offline/locked-down environments.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR (Phase 3 of #446, stacked on prior phases) extends the Qwen3-Omni Thinker DAPO V1 omni training recipes/config to enable dynamic sampling via algorithm.filter_groups, while also carrying the supporting config plumbing (reward.reward_kwargs) and CPU/e2e contract tests plus documentation updates.

Changes:

• Enable DAPO dynamic sampling (group filtering) via a new AVQA LoRA launcher and corresponding CPU contract assertions.
• Plumb reward.reward_kwargs through the omni reward config (and regenerate _generated_*.yaml) to support registered reward managers like dapo.
• Expand CPU tests for OmniAlgoConfig inheritance/Hydra conversion and add an overlong-reward manager contract test.

File summaries

File	Description
verl_omni/trainer/config/reward/reward.yaml	Adds reward.reward_kwargs to pass manager-specific kwargs (e.g., overlong buffer) through Hydra.
verl_omni/trainer/config/algorithm.py	Makes OmniAlgoConfig inherit upstream AlgoConfig so online-policy-gradient fields survive conversion.
verl_omni/trainer/config/_generated_omni_trainer.yaml	Regenerated to include reward_kwargs defaults.
verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml	Regenerated to include reward_kwargs defaults.
verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml	Regenerated to include reward_kwargs defaults.
verl_omni/trainer/config/_generated_diffusion_trainer.yaml	Regenerated to include reward_kwargs defaults.
tests/workers/config/test_omni_config_on_cpu.py	Adds assertions that OmniAlgoConfig is an AlgoConfig and that inherited online fields survive Hydra conversion.
tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py	Adds CPU launcher-contract tests for Phase 1/2/3 DAPO scripts (including dynamic sampling contract).
tests/utils/test_dapo_overlong_reward_on_cpu.py	Adds a direct DAPORewardManager contract test to ensure overlong penalty changes reward.
tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh	Adds a tiny-random V1 DAPO+LoRA e2e smoke script (without dynamic sampling) that exercises overlong buffer plumbing.
examples/dapo_trainer/README.md	Documents Phase 1/2/3 design and the dynamic-sampling/overlong-buffer config contract.
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh	Adds Phase 1 AVQA LoRA launcher (DAPO policy settings, no dynamic sampling).
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh	Adds Phase 3 AVQA LoRA launcher enabling algorithm.filter_groups dynamic sampling.
docs/index.md	Adds the DAPO trainer docs page to the Sphinx Examples toctree.

Review details

Suppressed comments (1)

tests/utils/test_dapo_overlong_reward_on_cpu.py:50

• This CPU unit test downloads the HuggingFace "gpt2" tokenizer at runtime, which can make CI flaky/fail in offline environments. Prefer a tiny local dummy tokenizer for this reward-manager contract test so it stays fully hermetic.

    tokenizer = AutoTokenizer.from_pretrained("gpt2")
    return DAPORewardManager(config, tokenizer, _compute_score)

• Files reviewed: 15/15 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.