[cfg, recipe, tests] feat: add MMK12 DAPO recipe on V1 omni loop (#446 Phase 4)

[OnePunchMonk]

Summary

Implements Phase 4 of #446 (Qwen3-Omni Thinker DAPO): applies the Phase 1-3
DAPO contract to a second modality/dataset, MMK12 math reasoning.

Stacked on #540 (Phase 3), which is stacked on #502 (Phase 2), which is
stacked on #456 (Phase 1) — none merged yet. This branch is built on top of
dapo-p3-dynamic-sampling, so the diff against main includes all three
PRs' commits. Please review only the commit on top
(feat: add MMK12 DAPO recipe on V1 omni loop (#446 Phase 4)) until the
earlier phases land; I'll rebase and shrink the diff as each merges.

What's actually wired

Per #446's non-goals, this is explicitly not a new adapter or reward path —
it reuses the existing mmk12_reward.py scorer and
examples/gspo_trainer/data_process/mmk12.py data pipeline (already used by
the GSPO MMK12 recipe) and combines them with the DAPO policy/reward
contract from Phases 1-3:

• Adds examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh:
  same vanilla clip-higher / token-mean / GRPO / no-KL policy settings as the
  AVQA dynamic-sampling launcher, algorithm.filter_groups.enable=true, and
  the overlong buffer sized for MMK12's longer 12288-token max response
  (overlong_buffer_cfg.len=1024 vs AVQA's 128, scaled to the same ~8% of
  max_response_length).
• Adds test_dapo_mmk12_example_launcher_has_phase_four_contract mirroring
  the existing AVQA contract tests.
• Documents the recipe in examples/dapo_trainer/README.md.

Test plan

CPU (all green locally):

python3 -m pytest tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py -q

Not yet run: any GPU e2e on real MMK12 data or a tiny-random smoke. I
don't have further Modal/GPU access or the MMK12 dataset in this pass, so the
overlong_buffer_cfg.len=1024 sizing is a scaled estimate from the AVQA
value, not empirically tuned — flagging this explicitly. This PR stays in
draft until a GPU run validates it and until the earlier phases land
underneath it.

AI usage disclosure

Claude Code was used to derive this recipe from the existing AVQA DAPO
launchers and the GSPO MMK12 recipe, and to write the test above. A human
submitter (onepunchmonk) will review every changed line and run a GPU
validation before taking this out of draft.

[Copilot]

🟡 Changes recommended

The docs toctree references a missing examples/dapo_trainer.md page and a new CPU test uses AutoTokenizer.from_pretrained("gpt2"), which can break CI by requiring network access.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the Qwen3-Omni Thinker DAPO “config + recipe” stack to a second dataset/modality (MMK12), while also wiring the shared DAPO config contract (overlong reward kwargs + dynamic sampling knobs) into the omni trainer config and adding CPU contract tests.

Changes:

• Add DAPO launchers for AVQA (phase-1 baseline), AVQA + dynamic sampling (phase-3), and MMK12 + dynamic sampling + overlong shaping (phase-4).
• Plumb reward.reward_kwargs (overlong buffer config + max_resp_len) into the omni reward config and regenerate trainer YAMLs.
• Add CPU tests to validate the launcher/config “DAPO contract” and (directly) the DAPO overlong penalty behavior.

File summaries

File	Description
verl_omni/trainer/config/reward/reward.yaml	Adds reward.reward_kwargs so DAPO reward manager can receive overlong-buffer settings.
verl_omni/trainer/config/algorithm.py	Updates OmniAlgoConfig to inherit upstream AlgoConfig so online/DAPO fields survive structured config conversion.
verl_omni/trainer/config/_generated_omni_trainer.yaml	Regenerated to include reward.reward_kwargs defaults.
verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml	Regenerated to include reward.reward_kwargs defaults.
verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml	Regenerated to include reward.reward_kwargs defaults.
verl_omni/trainer/config/_generated_diffusion_trainer.yaml	Regenerated to include reward.reward_kwargs defaults.
tests/workers/config/test_omni_config_on_cpu.py	Adds assertions that OmniAlgoConfig is an AlgoConfig and that inherited online fields can be set via Hydra.
tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py	Adds “launcher contract” tests for phase 1/3/4 DAPO scripts and the tiny-random smoke script.
tests/utils/test_dapo_overlong_reward_on_cpu.py	Adds a CPU unit test asserting overlong penalty changes reward with DAPORewardManager.
tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh	Adds a tiny-random GPU smoke script for DAPO + overlong shaping (no dynamic sampling).
examples/dapo_trainer/README.md	Documents the phased DAPO recipes and how they map to AVQA/MMK12.
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh	Adds phase-1 AVQA DAPO launcher (no dynamic sampling, naive reward manager).
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh	Adds phase-3 AVQA launcher with filter_groups enabled and streaming reward path via DAPO manager.
examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh	Adds phase-4 MMK12 launcher reusing mmk12 reward + overlong shaping + dynamic sampling.
docs/index.md	Adds a Sphinx toctree entry for a DAPO example docs page.

Review details

Suppressed comments (1)

tests/utils/test_dapo_overlong_reward_on_cpu.py:51

• Avoid from_pretrained("gpt2") here (Hub download). Build a minimal local tokenizer and pass that into DAPORewardManager so the test is fully offline/reproducible.

        }
    )
    tokenizer = AutoTokenizer.from_pretrained("gpt2")
    return DAPORewardManager(config, tokenizer, _compute_score)

• Files reviewed: 16/16 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.