[doc] feat: document Qwen3-Omni Thinker DAPO support matrix (#446 Phase 5) by OnePunchMonk · Pull Request #542 · verl-project/verl-omni

OnePunchMonk · 2026-09-04T16:36:53Z

Summary

Implements Phase 5 of #446 (Qwen3-Omni Thinker DAPO): docs and support
matrix, the final phase of the RFC.

Stacked on #541 (Phase 4), which is stacked on #540 (Phase 3), #502
(Phase 2), and #456 (Phase 1) — none merged yet. This branch is built on
top of dapo-p4-mmk12-recipe, so the diff against main includes all four
PRs' commits. Please review only the commit on top
(feat: document Qwen3-Omni Thinker DAPO support matrix (#446 Phase 5))
until the earlier phases land; I'll rebase and shrink the diff as each
merges.

What's actually wired

Adds a Qwen3-Omni-Thinker × DAPO row (marked WIP, since Phases 1-4
above are not yet GPU-validated by a maintainer) to the top-level
README.md algorithm support matrix.
Adds a recipe-comparison table to examples/dapo_trainer/README.md listing
all three launchers (Phase 1 baseline, Phase 3 dynamic sampling, Phase 4
MMK12) against dynamic-sampling/overlong-buffer support.
Adds the RFC's explicit warning, called out as this phase's exit
criterion: reward.reward_manager.name=dapo alone selects only the DAPO
reward manager, not the algorithm — the algorithm is selected by
actor_rollout_ref.actor.policy_loss.loss_mode and
algorithm.adv_estimator.

Test plan

Docs-only change. No code paths touched; nothing to run beyond confirming
the markdown tables render (checked locally).

AI usage disclosure

Claude Code was used to draft this documentation from the RFC's stated
Phase 5 exit criterion and the launchers added in Phases 1-4. A human
submitter (onepunchmonk) will review every changed line before taking this
out of draft.

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

Assisted-by: AI tooling Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

…i loop (verl-project#446 Phase 2) Phase 1 (verl-project#456) left the overlong reward buffer disabled everywhere; verl's registered DAPORewardManager already reads reward.reward_kwargs.overlong_buffer_cfg and reward.reward_kwargs.max_resp_len off the config, but verl_omni's reward.yaml never declared reward_kwargs, so any override there was silently dropped for the autogenerated trainer configs. - Add reward.reward_kwargs (overlong_buffer_cfg + max_resp_len) to reward.yaml and regenerate the four _generated_*.yaml trainer configs. - Enable overlong shaping in the DAPO tiny-random smoke test (reward.reward_manager.name=dapo path only; the naive-manager AVQA example is unaffected and documented as such). - Add a CPU test that drives DAPORewardManager directly and asserts the overlong penalty changes the reward on a truncated dummy response — the Phase 2 exit criterion from verl-project#446. - Split the shared launcher-contract test helper so the "no overlong" check only applies to the naive-manager example, and add overlong-specific assertions for the dapo-manager smoke test. This does not touch dynamic sampling (Phase 3) or the reward-loop metrics plumbing beyond what DAPORewardManager already logs in reward_extra_info. Draft: opened for early feedback while GPU LoRA validation on real Qwen3-Omni-30B-A3B-Instruct + gsm8k is still pending (see PR description). Co-authored-by: Claude Code

Co-authored-by: Claude Code

…op (verl-project#446 Phase 3) Confirms the upstream V1 PPOTrainer replay buffer already implements group filtering, so Phase 3 is config-only: adds a dynamic-sampling AVQA launcher with algorithm.filter_groups.enable=true/metric=acc on the streaming DAPO reward path, plus a CPU launcher-contract test.

…l-project#446 Phase 4) Applies the Phase 1-3 DAPO contract (token-level clip-higher, dynamic sampling, overlong shaping) to a second modality, MMK12 math reasoning, reusing the existing mmk12_reward.py scorer and data pipeline. Adds a CPU launcher-contract test mirroring the AVQA one.

…ject#446 Phase 5) Adds a DAPO row (WIP) to the top-level README support matrix, a recipe comparison table, and an explicit warning that reward.reward_manager.name=dapo alone does not select the DAPO algorithm.

Copilot

🟡 Changes recommended

A newly added CPU test uses AutoTokenizer.from_pretrained("gpt2"), which is likely to break or slow CI due to network/model download requirements.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Qwen3-Omni Thinker × DAPO documentation (support matrix + recipes) and the underlying config/test scaffolding needed to make the DAPO reward/algorithm knobs survive config conversion and be exercised in CPU/e2e checks.

Changes:

Document DAPO support status for Qwen3-Omni Thinker in the top-level README support matrix and add a DAPO trainer example write-up + recipe comparison.
Extend reward config schema with reward.reward_kwargs (overlong buffer/max response length) and regenerate trainer configs.
Make OmniAlgoConfig inherit upstream AlgoConfig and add CPU/e2e tests + launchers to validate the Phase 1–4 DAPO contract.

File summaries

File	Description
verl_omni/trainer/config/reward/reward.yaml	Adds `reward_kwargs` pass-through for reward-manager-specific knobs (e.g., DAPO overlong buffer).
verl_omni/trainer/config/algorithm.py	Switches `OmniAlgoConfig` to inherit from upstream `AlgoConfig`.
verl_omni/trainer/config/_generated_omni_trainer.yaml	Regenerated config to include `reward_kwargs`.
verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml	Regenerated config to include `reward_kwargs`.
verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml	Regenerated config to include `reward_kwargs`.
verl_omni/trainer/config/_generated_diffusion_trainer.yaml	Regenerated config to include `reward_kwargs`.
tests/workers/config/test_omni_config_on_cpu.py	Adds assertions that inherited online-policy-gradient fields survive Hydra → dataclass conversion.
tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py	Adds CPU contract tests ensuring launch scripts encode the expected DAPO settings across phases.
tests/utils/test_dapo_overlong_reward_on_cpu.py	Adds a CPU test asserting overlong shaping changes reward output.
tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh	Adds a tiny-random multi-GPU smoke script for DAPO + overlong buffer.
README.md	Adds a WIP DAPO row for Qwen3-Omni-Thinker in the algorithm support matrix.
examples/dapo_trainer/README.md	Adds DAPO recipe documentation and phase-by-phase explanation/warnings.
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh	Adds Phase 1 baseline AVQA launcher (no dynamic sampling / no overlong).
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh	Adds Phase 3 AVQA launcher with group filtering + streaming reward path.
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh	Adds Phase 4 MMK12 launcher with dynamic sampling + overlong shaping.
docs/index.md	Adds DAPO trainer docs page to the Sphinx toctree.

Review details

Files reviewed: 17/17 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    tokenizer = AutoTokenizer.from_pretrained("gpt2")
+    return DAPORewardManager(config, tokenizer, _compute_score)


+    <td rowspan="3">Qwen3-Omni-Thinker</td>
+    <td rowspan="3">Omni-modality</td>
+    <td rowspan="3">Text / Image / Video / Audio</td>
    <td>DPO</td>
    <td>✅</td>
  </tr>
    <tr>
    <td>GSPO</td>
    <td>✅</td>
+  </tr>
+    <tr>
+    <td>DAPO</td>
+    <td>WIP</td>


WenzheWang and others added 12 commits August 28, 2026 15:38

[cfg, recipe, tests] feat: add Thinker DAPO phase one

c3f0b3f

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[doc] fix: expose DAPO example in docs

bef712e

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[cfg, recipe, tests] fix: address AVQA validation feedback

8323ead

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[cfg, recipe, tests] fix: harden DAPO contracts

6969a3c

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[tests] fix: match rollout correction mapping contract

c467198

Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[doc, tests] fix: address DAPO review feedback

4a8eaf7

Assisted-by: AI tooling Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[cfg, doc, tests] fix: simplify DAPO review scope

bc0bee6

Assisted-by: AI tooling Signed-off-by: Wenzhe Wang <w_wenzhe@163.com>

[cfg, tests] chore: trim redundant comments

7dc3bd9

Co-authored-by: Claude Code

[doc] feat: document Qwen3-Omni Thinker DAPO support matrix (verl-pro…

d4c4043

…ject#446 Phase 5) Adds a DAPO row (WIP) to the top-level README support matrix, a recipe comparison table, and an explicit warning that reward.reward_manager.name=dapo alone does not select the DAPO algorithm.

Copilot AI lite review requested due to automatic review settings September 4, 2026 16:36

Copilot started reviewing on behalf of OnePunchMonk September 4, 2026 16:37 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

[doc] feat: document Qwen3-Omni Thinker DAPO support matrix (#446 Phase 5) - #542

[doc] feat: document Qwen3-Omni Thinker DAPO support matrix (#446 Phase 5)#542
OnePunchMonk wants to merge 12 commits into
verl-project:mainfrom
OnePunchMonk:dapo-p5-docs

OnePunchMonk commented Sep 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		tokenizer = AutoTokenizer.from_pretrained("gpt2")
		return DAPORewardManager(config, tokenizer, _compute_score)

Uh oh!

Conversation

OnePunchMonk commented Sep 4, 2026

Summary

What's actually wired

Test plan

AI usage disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants