[doc] feat: document Qwen3-Omni Thinker DAPO support matrix (#446 Phase 5)

README.md

@@ -180,15 +180,19 @@ Visit our [documentation](https://verl-omni.readthedocs.io/en/latest/index.html)
     <td>Planned</td>
   </tr>
   <tr>
-    <td rowspan="2">Qwen3-Omni-Thinker</td>
-    <td rowspan="2">Omni-modality</td>
-    <td rowspan="2">Text / Image / Video / Audio</td>
+    <td rowspan="3">Qwen3-Omni-Thinker</td>
+    <td rowspan="3">Omni-modality</td>
+    <td rowspan="3">Text / Image / Video / Audio</td>
     <td>DPO</td>
     <td>✅</td>
   </tr>
     <tr>
     <td>GSPO</td>
     <td>✅</td>
+  </tr>
+    <tr>
+    <td>DAPO</td>
+    <td>WIP</td>
   </tr>
     <tr>
     <td rowspan="2">Qwen3-TTS</td>

docs/examples/dapo_trainer.md

@@ -0,0 +1 @@
+../../examples/dapo_trainer/README.md

docs/index.md

@@ -72,6 +72,7 @@ algo/performance.md
 examples/flowgrpo_trainer.md
 examples/flowdppo_trainer.md
 examples/dpo_trainer.md
+examples/dapo_trainer.md
 examples/dancegrpo_trainer.md
 examples/diffusionnft_trainer.md
 examples/grpoguard_trainer.md

examples/dapo_trainer/README.md

@@ -0,0 +1,117 @@
+# Qwen3-Omni Thinker DAPO Trainer
+
+Last updated: 09/04/2026
+
+This example provides the first Qwen3-Omni Thinker DAPO milestone on the V1
+omni trainer: GPU LoRA training on multimodal AVQA with clip-higher,
+token-level policy gradient, GRPO advantages, and the AVQA choice reward.
+
+The Phase 1 launcher intentionally disables dynamic sampling:
+
+```text
+algorithm.filter_groups.enable=false
+```
+
+It also does not enable the overlong reward buffer. Those components are kept
+out of this baseline so the token-level DAPO policy path can be validated
+independently. In this Phase 1 recipe, DAPO refers to vanilla token-level
+policy loss with asymmetric clipping, GRPO advantages, and no KL penalty. The
+registered naive reward manager calls the AVQA `choice_reward`; the reward
+manager name alone does not select the optimization algorithm.
+
+**Phase 2 (#446): overlong reward buffer.** Overlong shaping is wired through
+`reward.reward_kwargs` and only applies with `reward.reward_manager.name=dapo`
+(`source=register`) — it is a no-op under the `naive` manager this example
+uses. See `tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh` for
+a working `name=dapo` recipe with overlong shaping enabled, and
+`tests/utils/test_dapo_overlong_reward_on_cpu.py` for the reward-shape
+contract: `reward.reward_kwargs.overlong_buffer_cfg.{enable,len,penalty_factor,log}`
+and `reward.reward_kwargs.max_resp_len`.
+
+**Phase 3 (#446): dynamic sampling.** The upstream V1 `PPOTrainer` replay
+buffer already implements group filtering — this is config-only, no new
+trainer code. `run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh` sets
+`algorithm.filter_groups.enable=true` with `metric=acc` (the naive/DAPO
+reward managers always populate `reward_extra_info["acc"]`, see
+`verl.experimental.reward_loop.reward_manager.naive`), so the trainer drops
+uniform-reward groups (all-correct or all-wrong) and keeps generating until
+`data.train_batch_size` qualified prompts are collected, bounded by
+`algorithm.filter_groups.max_inflight_gen_batches`. Group filtering requires a
+streaming reward path (`reward.reward_model.enable=false`, the default), so
+this recipe uses the `dapo` reward manager rather than Phase 1's `naive` one.
+
+**Phase 4 (#446): a second modality.**
+`run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh` applies the same Phase 1-3 DAPO
+contract (token-level clip-higher, dynamic sampling, overlong shaping) to
+MMK12 math reasoning, reusing the existing `mmk12_reward.py` scorer and data
+pipeline from `examples/gspo_trainer/data_process/mmk12.py`. It is not a new
+adapter — only the dataset, reward scorer, and `reward_kwargs.overlong_buffer_cfg.len`
+(sized for MMK12's longer 12288-token responses) differ from the AVQA
+recipe.
+
+**Phase 5 (#446): support matrix.** The top-level [README](../../README.md)
+now lists Qwen3-Omni-Thinker × DAPO as WIP (Phase 1-4 recipes above are not
+yet GPU-validated by a maintainer). **`reward.reward_manager.name=dapo` alone
+does not select the DAPO algorithm** — it only selects the DAPO reward
+manager (overlong buffer, math parsing). The optimization algorithm is
+selected by `actor_rollout_ref.actor.policy_loss.loss_mode` and
+`algorithm.adv_estimator`; running `loss_mode=gspo` with
+`reward_manager.name=dapo` still trains GSPO. See the
+[RFC](https://github.com/verl-project/verl-omni/issues/446) for the full
+GSPO-vs-DAPO knob table.
+
+## Recipes
+
+| Launcher | Dataset | Dynamic sampling | Overlong buffer |
+| --- | --- | --- | --- |
+| `run_qwen3_omni_thinker_dapo_lora_v1.sh` | AVQA | ✗ (Phase 1 baseline) | ✗ |
+| `run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh` | AVQA | ✅ | ✅ |
+| `run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh` | MMK12 | ✅ | ✅ |
+
+## Run
+
+Download and extract the AVQA-R1-6K data, then convert it from the repository
+root:
+
+```bash
+python examples/gspo_trainer/data_process/avqa.py \
+    --input_dir /path/to/raw/AVQA_R1 \
+    --output_dir ~/data/avqa_r1_6k
+```
+
+The converted parquet stores absolute image and audio paths. Every Ray worker
+must mount the converted dataset and its media files at the same absolute path
+used during conversion.
+
+Install the audio and multimodal processing dependencies on every Ray worker,
+then launch:
+
+```bash
+pip install -e ".[audio]"
+pip install qwen-vl-utils
+```
+
+```bash
+bash examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh
+```
+
+The default model is `~/models/Qwen/Qwen3-Omni-30B-A3B-Instruct`. Override the
+model, data, or any Hydra setting without editing the script:
+
+```bash
+MODEL_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \
+TRAIN_FILE=/path/to/train.parquet \
+VAL_FILE=/path/to/validation.parquet \
+bash examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh \
+    trainer.total_training_steps=2
+```
+
+Validation runs once before training and every 10 steps by default. It uses
+greedy decoding (`n=1`, `do_sample=false`, `temperature=0`, `top_p=1`,
+`top_k=-1`) over the full validation split. Plot
+`val-core/avqa_r1_6k/reward/mean@1` against the trainer step for the directly
+comparable in-trainer validation curve.
+
+Only the Thinker LoRA adapters are trained. Talker, code2wav, code predictor,
+visual projection, and audio-tower modules are excluded, and the vision tower
+is frozen, matching the existing GSPO V1 baseline.

examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA training on AVQA with dynamic sampling.
+#
+# Phase 3 (#446): enables algorithm.filter_groups so the V1 replay buffer
+# drops uniform-reward groups (all-correct / all-wrong) and keeps generating
+# until data.train_batch_size qualified prompts are collected, up to
+# algorithm.filter_groups.max_inflight_gen_batches concurrent generation
+# batches. This requires a streaming (non-colocated) reward path, so the
+# reward manager below is `dapo` via the registered reward loop, matching the
+# Phase 2 overlong-buffer smoke rather than Phase 1's `naive` manager.
+#
+# Data preparation (run once):
+#   python examples/gspo_trainer/data_process/avqa.py \
+#       --input_dir <path_to_raw_AVQA_R1> \
+#       --output_dir ~/data/avqa_r1_6k
+#
+# Runtime dependencies (all Ray worker nodes):
+#   pip install -e ".[audio]"
+#   pip install qwen-vl-utils
+
+set -xeuo pipefail
+
+# Make verl_omni available to Ray workers.
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+MODEL_PATH=${MODEL_PATH:-"$HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/avqa_r1_6k/train.parquet"}
+VAL_FILE=${VAL_FILE:-"$HOME/data/avqa_r1_6k/validation.parquet"}
+
+python3 -m verl_omni.trainer.main_omni \
+    data.train_files="${TRAIN_FILE}" \
+    data.val_files="${VAL_FILE}" \
+    data.train_batch_size=128 \
+    data.max_prompt_length=4096 \
+    data.max_response_length=12288 \
+    data.shuffle=true \
+    data.seed=42 \
+    data.val_max_samples=-1 \
+    data.validation_shuffle=false \
+    data.truncation='error' \
+    data.filter_overlong_prompts=true \
+    data.custom_cls.path=pkg://verl_omni.utils.dataset.omni_rl_datasets \
+    data.custom_cls.name=QwenOmniRLHFDataset \
+    +data.mm_processor_kwargs.sampling_rate=16000 \
+    actor_rollout_ref.model.path="${MODEL_PATH}" \
+    actor_rollout_ref.model.lora_rank=32 \
+    actor_rollout_ref.model.lora_alpha=64 \
+    actor_rollout_ref.model.lora_dtype=float32 \
+    actor_rollout_ref.model.lora.merge=true \
+    actor_rollout_ref.model.enable_gradient_checkpointing=true \
+    actor_rollout_ref.model.use_remove_padding=true \
+    actor_rollout_ref.model.exclude_modules=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" \
+    actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+    actor_rollout_ref.actor.freeze_vision_tower=true \
+    actor_rollout_ref.actor.strategy=fsdp2 \
+    actor_rollout_ref.actor.optim.lr=3e-6 \
+    actor_rollout_ref.actor.optim.weight_decay=0.01 \
+    actor_rollout_ref.actor.optim.clip_grad=1.0 \
+    actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+    actor_rollout_ref.actor.use_dynamic_bsz=true \
+    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=20480 \
+    actor_rollout_ref.actor.use_kl_loss=false \
+    actor_rollout_ref.actor.entropy_coeff=0 \
+    actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+    actor_rollout_ref.actor.clip_ratio_low=0.2 \
+    actor_rollout_ref.actor.clip_ratio_high=0.28 \
+    actor_rollout_ref.actor.clip_ratio_c=10.0 \
+    actor_rollout_ref.actor.loss_agg_mode=token-mean \
+    actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+    actor_rollout_ref.actor.fsdp_config.param_offload=true \
+    actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+    actor_rollout_ref.rollout.name=vllm_omni \
+    actor_rollout_ref.rollout.n=16 \
+    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
+    actor_rollout_ref.rollout.gpu_memory_utilization=0.7 \
+    actor_rollout_ref.rollout.load_format=safetensors \
+    actor_rollout_ref.rollout.prompt_length=4160 \
+    actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+    actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=20480 \
+    actor_rollout_ref.rollout.enable_prefix_caching=false \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=256 \
+    actor_rollout_ref.rollout.cudagraph_capture_sizes=[1,2,4,8,16,32,64,128,256] \
+    actor_rollout_ref.rollout.val_kwargs.n=1 \
+    actor_rollout_ref.rollout.val_kwargs.do_sample=false \
+    actor_rollout_ref.rollout.val_kwargs.temperature=0 \
+    actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \
+    actor_rollout_ref.rollout.val_kwargs.top_k=-1 \
+    actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+    actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=20480 \
+    actor_rollout_ref.ref.fsdp_config.param_offload=true \
+    actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+    algorithm.trainer_type=policy_gradient \
+    algorithm.sample_source=online \
+    algorithm.adv_estimator=grpo \
+    algorithm.use_kl_in_reward=false \
+    algorithm.filter_groups.enable=true \
+    algorithm.filter_groups.metric=acc \
+    algorithm.filter_groups.max_inflight_gen_batches=2 \
+    reward.reward_model.enable=false \
+    reward.reward_manager.source=register \
+    reward.reward_manager.name=dapo \
+    reward.custom_reward_function.path=verl_omni/utils/reward_score/choice_reward.py \
+    reward.custom_reward_function.name=compute_score \
+    trainer.val_before_train=true \
+    trainer.balance_batch=true \
+    trainer.critic_warmup=0 \
+    trainer.logger='["console","wandb"]' \
+    trainer.project_name=dapo \
+    trainer.experiment_name=qwen3_omni_thinker_lora_avqa_dynamic_sampling \
+    trainer.n_gpus_per_node=4 \
+    trainer.nnodes=1 \
+    trainer.save_freq=50 \
+    trainer.test_freq=10 \
+    trainer.total_epochs=10 \
+    "$@"

examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh

@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA training on MMK12 with dynamic sampling.
+#
+# Phase 4 (#446): the AVQA DAPO recipes (Phase 1/2/3) applied to a second
+# modality/dataset, MMK12, reusing the existing GSPO MMK12 data pipeline and
+# reward scorer. Same token-level DAPO policy contract as the AVQA dynamic
+# sampling launcher: vanilla clip-higher, token-mean, GRPO advantages, no KL,
+# algorithm.filter_groups enabled on the streaming `dapo` reward manager, and
+# the overlong response-length buffer from Phase 2.
+#
+# Data preparation (run once):
+#   pip install math-verify
+#   python examples/gspo_trainer/data_process/mmk12.py \
+#       --local_dataset_path <path_to_raw_mmk12> \
+#       --local_save_dir ~/data/mmk12
+#
+# Runtime dependencies (all Ray worker nodes):
+#   pip install math-verify    # required by mmk12_reward.py
+#   pip install qwen-vl-utils  # required for multimodal data processing
+
+set -xeuo pipefail
+
+# Make verl_omni available to Ray workers.
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+MODEL_PATH=${MODEL_PATH:-"$HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/mmk12/train.parquet"}
+VAL_FILE=${VAL_FILE:-"$HOME/data/mmk12/test.parquet"}
+
+python3 -m verl_omni.trainer.main_omni \
+    data.train_files="${TRAIN_FILE}" \
+    data.val_files="${VAL_FILE}" \
+    data.train_batch_size=128 \
+    data.max_prompt_length=4096 \
+    data.max_response_length=12288 \
+    data.shuffle=true \
+    data.seed=42 \
+    data.val_max_samples=-1 \
+    data.validation_shuffle=false \
+    data.truncation='error' \
+    data.filter_overlong_prompts=true \
+    actor_rollout_ref.model.path="${MODEL_PATH}" \
+    actor_rollout_ref.model.lora_rank=32 \
+    actor_rollout_ref.model.lora_alpha=64 \
+    actor_rollout_ref.model.lora_dtype=float32 \
+    actor_rollout_ref.model.lora.merge=true \
+    actor_rollout_ref.model.enable_gradient_checkpointing=true \
+    actor_rollout_ref.model.use_remove_padding=true \
+    actor_rollout_ref.model.exclude_modules=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" \
+    actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+    actor_rollout_ref.actor.freeze_vision_tower=true \
+    actor_rollout_ref.actor.strategy=fsdp2 \
+    actor_rollout_ref.actor.optim.lr=3e-6 \
+    actor_rollout_ref.actor.optim.weight_decay=0.01 \
+    actor_rollout_ref.actor.optim.clip_grad=1.0 \
+    actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+    actor_rollout_ref.actor.use_dynamic_bsz=true \
+    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=30720 \
+    actor_rollout_ref.actor.use_kl_loss=false \
+    actor_rollout_ref.actor.entropy_coeff=0 \
+    actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+    actor_rollout_ref.actor.clip_ratio_low=0.2 \
+    actor_rollout_ref.actor.clip_ratio_high=0.28 \
+    actor_rollout_ref.actor.clip_ratio_c=10.0 \
+    actor_rollout_ref.actor.loss_agg_mode=token-mean \
+    actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+    actor_rollout_ref.actor.fsdp_config.param_offload=true \
+    actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+    actor_rollout_ref.rollout.name=vllm_omni \
+    actor_rollout_ref.rollout.n=16 \
+    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
+    actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
+    actor_rollout_ref.rollout.load_format=safetensors \
+    actor_rollout_ref.rollout.prompt_length=4160 \
+    actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+    actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=30720 \
+    actor_rollout_ref.rollout.enable_prefix_caching=false \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+    +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=256 \
+    actor_rollout_ref.rollout.val_kwargs.n=1 \
+    actor_rollout_ref.rollout.val_kwargs.do_sample=false \
+    actor_rollout_ref.rollout.val_kwargs.temperature=0 \
+    actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \
+    actor_rollout_ref.rollout.val_kwargs.top_k=-1 \
+    actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+    actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=30720 \
+    actor_rollout_ref.ref.fsdp_config.param_offload=true \
+    actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+    algorithm.trainer_type=policy_gradient \
+    algorithm.sample_source=online \
+    algorithm.adv_estimator=grpo \
+    algorithm.use_kl_in_reward=false \
+    algorithm.filter_groups.enable=true \
+    algorithm.filter_groups.metric=acc \
+    algorithm.filter_groups.max_inflight_gen_batches=2 \
+    reward.reward_model.enable=false \
+    reward.reward_manager.source=register \
+    reward.reward_manager.name=dapo \
+    reward.custom_reward_function.path=verl_omni/utils/reward_score/mmk12_reward.py \
+    reward.custom_reward_function.name=compute_score \
+    reward.reward_kwargs.max_resp_len=12288 \
+    reward.reward_kwargs.overlong_buffer_cfg.enable=true \
+    reward.reward_kwargs.overlong_buffer_cfg.len=1024 \
+    reward.reward_kwargs.overlong_buffer_cfg.penalty_factor=1.0 \
+    reward.reward_kwargs.overlong_buffer_cfg.log=true \
+    trainer.val_before_train=true \
+    trainer.balance_batch=true \
+    trainer.critic_warmup=0 \
+    trainer.logger='["console","wandb"]' \
+    trainer.project_name=dapo \
+    trainer.experiment_name=qwen3_omni_thinker_lora_mmk12_dynamic_sampling \
+    trainer.n_gpus_per_node=4 \
+    trainer.nnodes=1 \
+    trainer.save_freq=50 \
+    trainer.test_freq=10 \
+    trainer.total_epochs=10 \
+    "$@"