diff --git a/README.md b/README.md
index 19c614da5..075fa9ca4 100644
--- a/README.md
+++ b/README.md
@@ -180,15 +180,19 @@ Visit our [documentation](https://verl-omni.readthedocs.io/en/latest/index.html)
| Planned |
- | Qwen3-Omni-Thinker |
- Omni-modality |
- Text / Image / Video / Audio |
+ Qwen3-Omni-Thinker |
+ Omni-modality |
+ Text / Image / Video / Audio |
DPO |
✅ |
| GSPO |
✅ |
+
+
+ | DAPO |
+ WIP |
| Qwen3-TTS |
diff --git a/docs/examples/dapo_trainer.md b/docs/examples/dapo_trainer.md
new file mode 120000
index 000000000..7e662ccd6
--- /dev/null
+++ b/docs/examples/dapo_trainer.md
@@ -0,0 +1 @@
+../../examples/dapo_trainer/README.md
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
index 7541bbc27..ddbf995d7 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -72,6 +72,7 @@ algo/performance.md
examples/flowgrpo_trainer.md
examples/flowdppo_trainer.md
examples/dpo_trainer.md
+examples/dapo_trainer.md
examples/dancegrpo_trainer.md
examples/diffusionnft_trainer.md
examples/grpoguard_trainer.md
diff --git a/examples/dapo_trainer/README.md b/examples/dapo_trainer/README.md
new file mode 100644
index 000000000..08184ceee
--- /dev/null
+++ b/examples/dapo_trainer/README.md
@@ -0,0 +1,117 @@
+# Qwen3-Omni Thinker DAPO Trainer
+
+Last updated: 09/04/2026
+
+This example provides the first Qwen3-Omni Thinker DAPO milestone on the V1
+omni trainer: GPU LoRA training on multimodal AVQA with clip-higher,
+token-level policy gradient, GRPO advantages, and the AVQA choice reward.
+
+The Phase 1 launcher intentionally disables dynamic sampling:
+
+```text
+algorithm.filter_groups.enable=false
+```
+
+It also does not enable the overlong reward buffer. Those components are kept
+out of this baseline so the token-level DAPO policy path can be validated
+independently. In this Phase 1 recipe, DAPO refers to vanilla token-level
+policy loss with asymmetric clipping, GRPO advantages, and no KL penalty. The
+registered naive reward manager calls the AVQA `choice_reward`; the reward
+manager name alone does not select the optimization algorithm.
+
+**Phase 2 (#446): overlong reward buffer.** Overlong shaping is wired through
+`reward.reward_kwargs` and only applies with `reward.reward_manager.name=dapo`
+(`source=register`) — it is a no-op under the `naive` manager this example
+uses. See `tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh` for
+a working `name=dapo` recipe with overlong shaping enabled, and
+`tests/utils/test_dapo_overlong_reward_on_cpu.py` for the reward-shape
+contract: `reward.reward_kwargs.overlong_buffer_cfg.{enable,len,penalty_factor,log}`
+and `reward.reward_kwargs.max_resp_len`.
+
+**Phase 3 (#446): dynamic sampling.** The upstream V1 `PPOTrainer` replay
+buffer already implements group filtering — this is config-only, no new
+trainer code. `run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh` sets
+`algorithm.filter_groups.enable=true` with `metric=acc` (the naive/DAPO
+reward managers always populate `reward_extra_info["acc"]`, see
+`verl.experimental.reward_loop.reward_manager.naive`), so the trainer drops
+uniform-reward groups (all-correct or all-wrong) and keeps generating until
+`data.train_batch_size` qualified prompts are collected, bounded by
+`algorithm.filter_groups.max_inflight_gen_batches`. Group filtering requires a
+streaming reward path (`reward.reward_model.enable=false`, the default), so
+this recipe uses the `dapo` reward manager rather than Phase 1's `naive` one.
+
+**Phase 4 (#446): a second modality.**
+`run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh` applies the same Phase 1-3 DAPO
+contract (token-level clip-higher, dynamic sampling, overlong shaping) to
+MMK12 math reasoning, reusing the existing `mmk12_reward.py` scorer and data
+pipeline from `examples/gspo_trainer/data_process/mmk12.py`. It is not a new
+adapter — only the dataset, reward scorer, and `reward_kwargs.overlong_buffer_cfg.len`
+(sized for MMK12's longer 12288-token responses) differ from the AVQA
+recipe.
+
+**Phase 5 (#446): support matrix.** The top-level [README](../../README.md)
+now lists Qwen3-Omni-Thinker × DAPO as WIP (Phase 1-4 recipes above are not
+yet GPU-validated by a maintainer). **`reward.reward_manager.name=dapo` alone
+does not select the DAPO algorithm** — it only selects the DAPO reward
+manager (overlong buffer, math parsing). The optimization algorithm is
+selected by `actor_rollout_ref.actor.policy_loss.loss_mode` and
+`algorithm.adv_estimator`; running `loss_mode=gspo` with
+`reward_manager.name=dapo` still trains GSPO. See the
+[RFC](https://github.com/verl-project/verl-omni/issues/446) for the full
+GSPO-vs-DAPO knob table.
+
+## Recipes
+
+| Launcher | Dataset | Dynamic sampling | Overlong buffer |
+| --- | --- | --- | --- |
+| `run_qwen3_omni_thinker_dapo_lora_v1.sh` | AVQA | ✗ (Phase 1 baseline) | ✗ |
+| `run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh` | AVQA | ✅ | ✅ |
+| `run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh` | MMK12 | ✅ | ✅ |
+
+## Run
+
+Download and extract the AVQA-R1-6K data, then convert it from the repository
+root:
+
+```bash
+python examples/gspo_trainer/data_process/avqa.py \
+ --input_dir /path/to/raw/AVQA_R1 \
+ --output_dir ~/data/avqa_r1_6k
+```
+
+The converted parquet stores absolute image and audio paths. Every Ray worker
+must mount the converted dataset and its media files at the same absolute path
+used during conversion.
+
+Install the audio and multimodal processing dependencies on every Ray worker,
+then launch:
+
+```bash
+pip install -e ".[audio]"
+pip install qwen-vl-utils
+```
+
+```bash
+bash examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh
+```
+
+The default model is `~/models/Qwen/Qwen3-Omni-30B-A3B-Instruct`. Override the
+model, data, or any Hydra setting without editing the script:
+
+```bash
+MODEL_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \
+TRAIN_FILE=/path/to/train.parquet \
+VAL_FILE=/path/to/validation.parquet \
+bash examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh \
+ trainer.total_training_steps=2
+```
+
+Validation runs once before training and every 10 steps by default. It uses
+greedy decoding (`n=1`, `do_sample=false`, `temperature=0`, `top_p=1`,
+`top_k=-1`) over the full validation split. Plot
+`val-core/avqa_r1_6k/reward/mean@1` against the trainer step for the directly
+comparable in-trainer validation curve.
+
+Only the Thinker LoRA adapters are trained. Talker, code2wav, code predictor,
+visual projection, and audio-tower modules are excluded, and the vision tower
+is frozen, matching the existing GSPO V1 baseline.
diff --git a/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh
new file mode 100755
index 000000000..5001d0bf0
--- /dev/null
+++ b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA training on AVQA with dynamic sampling.
+#
+# Phase 3 (#446): enables algorithm.filter_groups so the V1 replay buffer
+# drops uniform-reward groups (all-correct / all-wrong) and keeps generating
+# until data.train_batch_size qualified prompts are collected, up to
+# algorithm.filter_groups.max_inflight_gen_batches concurrent generation
+# batches. This requires a streaming (non-colocated) reward path, so the
+# reward manager below is `dapo` via the registered reward loop, matching the
+# Phase 2 overlong-buffer smoke rather than Phase 1's `naive` manager.
+#
+# Data preparation (run once):
+# python examples/gspo_trainer/data_process/avqa.py \
+# --input_dir \
+# --output_dir ~/data/avqa_r1_6k
+#
+# Runtime dependencies (all Ray worker nodes):
+# pip install -e ".[audio]"
+# pip install qwen-vl-utils
+
+set -xeuo pipefail
+
+# Make verl_omni available to Ray workers.
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+MODEL_PATH=${MODEL_PATH:-"$HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/avqa_r1_6k/train.parquet"}
+VAL_FILE=${VAL_FILE:-"$HOME/data/avqa_r1_6k/validation.parquet"}
+
+python3 -m verl_omni.trainer.main_omni \
+ data.train_files="${TRAIN_FILE}" \
+ data.val_files="${VAL_FILE}" \
+ data.train_batch_size=128 \
+ data.max_prompt_length=4096 \
+ data.max_response_length=12288 \
+ data.shuffle=true \
+ data.seed=42 \
+ data.val_max_samples=-1 \
+ data.validation_shuffle=false \
+ data.truncation='error' \
+ data.filter_overlong_prompts=true \
+ data.custom_cls.path=pkg://verl_omni.utils.dataset.omni_rl_datasets \
+ data.custom_cls.name=QwenOmniRLHFDataset \
+ +data.mm_processor_kwargs.sampling_rate=16000 \
+ actor_rollout_ref.model.path="${MODEL_PATH}" \
+ actor_rollout_ref.model.lora_rank=32 \
+ actor_rollout_ref.model.lora_alpha=64 \
+ actor_rollout_ref.model.lora_dtype=float32 \
+ actor_rollout_ref.model.lora.merge=true \
+ actor_rollout_ref.model.enable_gradient_checkpointing=true \
+ actor_rollout_ref.model.use_remove_padding=true \
+ actor_rollout_ref.model.exclude_modules=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" \
+ actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+ actor_rollout_ref.actor.freeze_vision_tower=true \
+ actor_rollout_ref.actor.strategy=fsdp2 \
+ actor_rollout_ref.actor.optim.lr=3e-6 \
+ actor_rollout_ref.actor.optim.weight_decay=0.01 \
+ actor_rollout_ref.actor.optim.clip_grad=1.0 \
+ actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+ actor_rollout_ref.actor.use_dynamic_bsz=true \
+ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.actor.use_kl_loss=false \
+ actor_rollout_ref.actor.entropy_coeff=0 \
+ actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+ actor_rollout_ref.actor.clip_ratio_low=0.2 \
+ actor_rollout_ref.actor.clip_ratio_high=0.28 \
+ actor_rollout_ref.actor.clip_ratio_c=10.0 \
+ actor_rollout_ref.actor.loss_agg_mode=token-mean \
+ actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+ actor_rollout_ref.actor.fsdp_config.param_offload=true \
+ actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+ actor_rollout_ref.rollout.name=vllm_omni \
+ actor_rollout_ref.rollout.n=16 \
+ actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.7 \
+ actor_rollout_ref.rollout.load_format=safetensors \
+ actor_rollout_ref.rollout.prompt_length=4160 \
+ actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.rollout.enable_prefix_caching=false \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=256 \
+ actor_rollout_ref.rollout.cudagraph_capture_sizes=[1,2,4,8,16,32,64,128,256] \
+ actor_rollout_ref.rollout.val_kwargs.n=1 \
+ actor_rollout_ref.rollout.val_kwargs.do_sample=false \
+ actor_rollout_ref.rollout.val_kwargs.temperature=0 \
+ actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \
+ actor_rollout_ref.rollout.val_kwargs.top_k=-1 \
+ actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.ref.fsdp_config.param_offload=true \
+ actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+ algorithm.trainer_type=policy_gradient \
+ algorithm.sample_source=online \
+ algorithm.adv_estimator=grpo \
+ algorithm.use_kl_in_reward=false \
+ algorithm.filter_groups.enable=true \
+ algorithm.filter_groups.metric=acc \
+ algorithm.filter_groups.max_inflight_gen_batches=2 \
+ reward.reward_model.enable=false \
+ reward.reward_manager.source=register \
+ reward.reward_manager.name=dapo \
+ reward.custom_reward_function.path=verl_omni/utils/reward_score/choice_reward.py \
+ reward.custom_reward_function.name=compute_score \
+ trainer.val_before_train=true \
+ trainer.balance_batch=true \
+ trainer.critic_warmup=0 \
+ trainer.logger='["console","wandb"]' \
+ trainer.project_name=dapo \
+ trainer.experiment_name=qwen3_omni_thinker_lora_avqa_dynamic_sampling \
+ trainer.n_gpus_per_node=4 \
+ trainer.nnodes=1 \
+ trainer.save_freq=50 \
+ trainer.test_freq=10 \
+ trainer.total_epochs=10 \
+ "$@"
diff --git a/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh
new file mode 100755
index 000000000..484a8e4ef
--- /dev/null
+++ b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA training on MMK12 with dynamic sampling.
+#
+# Phase 4 (#446): the AVQA DAPO recipes (Phase 1/2/3) applied to a second
+# modality/dataset, MMK12, reusing the existing GSPO MMK12 data pipeline and
+# reward scorer. Same token-level DAPO policy contract as the AVQA dynamic
+# sampling launcher: vanilla clip-higher, token-mean, GRPO advantages, no KL,
+# algorithm.filter_groups enabled on the streaming `dapo` reward manager, and
+# the overlong response-length buffer from Phase 2.
+#
+# Data preparation (run once):
+# pip install math-verify
+# python examples/gspo_trainer/data_process/mmk12.py \
+# --local_dataset_path \
+# --local_save_dir ~/data/mmk12
+#
+# Runtime dependencies (all Ray worker nodes):
+# pip install math-verify # required by mmk12_reward.py
+# pip install qwen-vl-utils # required for multimodal data processing
+
+set -xeuo pipefail
+
+# Make verl_omni available to Ray workers.
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+MODEL_PATH=${MODEL_PATH:-"$HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/mmk12/train.parquet"}
+VAL_FILE=${VAL_FILE:-"$HOME/data/mmk12/test.parquet"}
+
+python3 -m verl_omni.trainer.main_omni \
+ data.train_files="${TRAIN_FILE}" \
+ data.val_files="${VAL_FILE}" \
+ data.train_batch_size=128 \
+ data.max_prompt_length=4096 \
+ data.max_response_length=12288 \
+ data.shuffle=true \
+ data.seed=42 \
+ data.val_max_samples=-1 \
+ data.validation_shuffle=false \
+ data.truncation='error' \
+ data.filter_overlong_prompts=true \
+ actor_rollout_ref.model.path="${MODEL_PATH}" \
+ actor_rollout_ref.model.lora_rank=32 \
+ actor_rollout_ref.model.lora_alpha=64 \
+ actor_rollout_ref.model.lora_dtype=float32 \
+ actor_rollout_ref.model.lora.merge=true \
+ actor_rollout_ref.model.enable_gradient_checkpointing=true \
+ actor_rollout_ref.model.use_remove_padding=true \
+ actor_rollout_ref.model.exclude_modules=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" \
+ actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+ actor_rollout_ref.actor.freeze_vision_tower=true \
+ actor_rollout_ref.actor.strategy=fsdp2 \
+ actor_rollout_ref.actor.optim.lr=3e-6 \
+ actor_rollout_ref.actor.optim.weight_decay=0.01 \
+ actor_rollout_ref.actor.optim.clip_grad=1.0 \
+ actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+ actor_rollout_ref.actor.use_dynamic_bsz=true \
+ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=30720 \
+ actor_rollout_ref.actor.use_kl_loss=false \
+ actor_rollout_ref.actor.entropy_coeff=0 \
+ actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+ actor_rollout_ref.actor.clip_ratio_low=0.2 \
+ actor_rollout_ref.actor.clip_ratio_high=0.28 \
+ actor_rollout_ref.actor.clip_ratio_c=10.0 \
+ actor_rollout_ref.actor.loss_agg_mode=token-mean \
+ actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+ actor_rollout_ref.actor.fsdp_config.param_offload=true \
+ actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+ actor_rollout_ref.rollout.name=vllm_omni \
+ actor_rollout_ref.rollout.n=16 \
+ actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
+ actor_rollout_ref.rollout.load_format=safetensors \
+ actor_rollout_ref.rollout.prompt_length=4160 \
+ actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=30720 \
+ actor_rollout_ref.rollout.enable_prefix_caching=false \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=256 \
+ actor_rollout_ref.rollout.val_kwargs.n=1 \
+ actor_rollout_ref.rollout.val_kwargs.do_sample=false \
+ actor_rollout_ref.rollout.val_kwargs.temperature=0 \
+ actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \
+ actor_rollout_ref.rollout.val_kwargs.top_k=-1 \
+ actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=30720 \
+ actor_rollout_ref.ref.fsdp_config.param_offload=true \
+ actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+ algorithm.trainer_type=policy_gradient \
+ algorithm.sample_source=online \
+ algorithm.adv_estimator=grpo \
+ algorithm.use_kl_in_reward=false \
+ algorithm.filter_groups.enable=true \
+ algorithm.filter_groups.metric=acc \
+ algorithm.filter_groups.max_inflight_gen_batches=2 \
+ reward.reward_model.enable=false \
+ reward.reward_manager.source=register \
+ reward.reward_manager.name=dapo \
+ reward.custom_reward_function.path=verl_omni/utils/reward_score/mmk12_reward.py \
+ reward.custom_reward_function.name=compute_score \
+ reward.reward_kwargs.max_resp_len=12288 \
+ reward.reward_kwargs.overlong_buffer_cfg.enable=true \
+ reward.reward_kwargs.overlong_buffer_cfg.len=1024 \
+ reward.reward_kwargs.overlong_buffer_cfg.penalty_factor=1.0 \
+ reward.reward_kwargs.overlong_buffer_cfg.log=true \
+ trainer.val_before_train=true \
+ trainer.balance_batch=true \
+ trainer.critic_warmup=0 \
+ trainer.logger='["console","wandb"]' \
+ trainer.project_name=dapo \
+ trainer.experiment_name=qwen3_omni_thinker_lora_mmk12_dynamic_sampling \
+ trainer.n_gpus_per_node=4 \
+ trainer.nnodes=1 \
+ trainer.save_freq=50 \
+ trainer.test_freq=10 \
+ trainer.total_epochs=10 \
+ "$@"
diff --git a/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh
new file mode 100755
index 000000000..57cbd73db
--- /dev/null
+++ b/examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA training on AVQA without dynamic sampling.
+# Data preparation (run once):
+# python examples/gspo_trainer/data_process/avqa.py \
+# --input_dir \
+# --output_dir ~/data/avqa_r1_6k
+#
+# Runtime dependencies (all Ray worker nodes):
+# pip install -e ".[audio]"
+# pip install qwen-vl-utils
+
+set -xeuo pipefail
+
+# Make verl_omni available to Ray workers.
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+MODEL_PATH=${MODEL_PATH:-"$HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct"}
+TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/avqa_r1_6k/train.parquet"}
+VAL_FILE=${VAL_FILE:-"$HOME/data/avqa_r1_6k/validation.parquet"}
+
+python3 -m verl_omni.trainer.main_omni \
+ data.train_files="${TRAIN_FILE}" \
+ data.val_files="${VAL_FILE}" \
+ data.train_batch_size=128 \
+ data.max_prompt_length=4096 \
+ data.max_response_length=12288 \
+ data.shuffle=true \
+ data.seed=42 \
+ data.val_max_samples=-1 \
+ data.validation_shuffle=false \
+ data.truncation='error' \
+ data.filter_overlong_prompts=true \
+ data.custom_cls.path=pkg://verl_omni.utils.dataset.omni_rl_datasets \
+ data.custom_cls.name=QwenOmniRLHFDataset \
+ +data.mm_processor_kwargs.sampling_rate=16000 \
+ actor_rollout_ref.model.path="${MODEL_PATH}" \
+ actor_rollout_ref.model.lora_rank=32 \
+ actor_rollout_ref.model.lora_alpha=64 \
+ actor_rollout_ref.model.lora_dtype=float32 \
+ actor_rollout_ref.model.lora.merge=true \
+ actor_rollout_ref.model.enable_gradient_checkpointing=true \
+ actor_rollout_ref.model.use_remove_padding=true \
+ actor_rollout_ref.model.exclude_modules=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" \
+ actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+ actor_rollout_ref.actor.freeze_vision_tower=true \
+ actor_rollout_ref.actor.strategy=fsdp2 \
+ actor_rollout_ref.actor.optim.lr=3e-6 \
+ actor_rollout_ref.actor.optim.weight_decay=0.01 \
+ actor_rollout_ref.actor.optim.clip_grad=1.0 \
+ actor_rollout_ref.actor.ppo_mini_batch_size=16 \
+ actor_rollout_ref.actor.use_dynamic_bsz=true \
+ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.actor.use_kl_loss=false \
+ actor_rollout_ref.actor.entropy_coeff=0 \
+ actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+ actor_rollout_ref.actor.clip_ratio_low=0.2 \
+ actor_rollout_ref.actor.clip_ratio_high=0.28 \
+ actor_rollout_ref.actor.clip_ratio_c=10.0 \
+ actor_rollout_ref.actor.loss_agg_mode=token-mean \
+ actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+ actor_rollout_ref.actor.fsdp_config.param_offload=true \
+ actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+ actor_rollout_ref.rollout.name=vllm_omni \
+ actor_rollout_ref.rollout.n=16 \
+ actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.7 \
+ actor_rollout_ref.rollout.load_format=safetensors \
+ actor_rollout_ref.rollout.prompt_length=4160 \
+ actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.rollout.enable_prefix_caching=false \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.max_num_seqs=256 \
+ actor_rollout_ref.rollout.cudagraph_capture_sizes=[1,2,4,8,16,32,64,128,256] \
+ actor_rollout_ref.rollout.val_kwargs.n=1 \
+ actor_rollout_ref.rollout.val_kwargs.do_sample=false \
+ actor_rollout_ref.rollout.val_kwargs.temperature=0 \
+ actor_rollout_ref.rollout.val_kwargs.top_p=1.0 \
+ actor_rollout_ref.rollout.val_kwargs.top_k=-1 \
+ actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.ref.fsdp_config.param_offload=true \
+ actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+ algorithm.trainer_type=policy_gradient \
+ algorithm.sample_source=online \
+ algorithm.adv_estimator=grpo \
+ algorithm.use_kl_in_reward=false \
+ algorithm.filter_groups.enable=false \
+ reward.reward_manager.source=register \
+ reward.reward_manager.name=naive \
+ reward.custom_reward_function.path=verl_omni/utils/reward_score/choice_reward.py \
+ reward.custom_reward_function.name=compute_score \
+ trainer.val_before_train=true \
+ trainer.balance_batch=true \
+ trainer.critic_warmup=0 \
+ trainer.logger='["console","wandb"]' \
+ trainer.project_name=dapo \
+ trainer.experiment_name=qwen3_omni_thinker_lora_avqa_wo_dynamic_sampling \
+ trainer.n_gpus_per_node=4 \
+ trainer.nnodes=1 \
+ trainer.save_freq=50 \
+ trainer.test_freq=10 \
+ trainer.total_epochs=10 \
+ "$@"
diff --git a/tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh b/tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh
new file mode 100755
index 000000000..cdab2bd31
--- /dev/null
+++ b/tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh
@@ -0,0 +1,118 @@
+#!/usr/bin/env bash
+# Qwen3-Omni Thinker DAPO + LoRA V1 smoke without dynamic sampling.
+
+set -xeuo pipefail
+
+if [[ "${SKIP_COMPAT_DEPS_INSTALL:-0}" != "1" ]]; then
+ uv pip install --system --break-system-packages transformers==5.12.1 accelerate==1.14.0 peft==0.19.1
+fi
+
+export NCCL_IB_DISABLE=1
+export CPATH=/usr/include${CPATH:+:$CPATH}
+export RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
+export VERL_USE_EXTERNAL_MODULES=verl_omni
+
+NUM_GPUS=${NUM_GPUS:-2}
+MODEL_PATH=${MODEL_PATH:-}
+DATA_DIR=${DATA_DIR:-${HOME}/data/gsm8k}
+TOTAL_TRAIN_STEPS=${TOTAL_TRAIN_STEPS:-2}
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+EXCLUDE_MODULES=".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*"
+
+MODEL_PATH="${MODEL_PATH:-${HOME}/models/tiny-random/Qwen3-Omni}"
+python3 "${REPO_ROOT}/tests/special_e2e/build_qwen3_omni_tiny_random.py" \
+ --output-dir "${MODEL_PATH}" --force
+
+if [ ! -f "${DATA_DIR}/train.parquet" ]; then
+ python3 "${REPO_ROOT}/tests/special_e2e/create_dummy_math_data.py" \
+ --local_save_dir "${DATA_DIR}"
+fi
+
+python3 -m verl_omni.trainer.main_omni \
+ data.train_files="${DATA_DIR}/train.parquet" \
+ data.val_files="${DATA_DIR}/test.parquet" \
+ data.train_batch_size=4 \
+ data.max_prompt_length=256 \
+ data.max_response_length=512 \
+ data.val_max_samples=4 \
+ data.truncation='error' \
+ data.filter_overlong_prompts=true \
+ actor_rollout_ref.model.path="${MODEL_PATH}" \
+ +actor_rollout_ref.model.override_config.attn_implementation=sdpa \
+ actor_rollout_ref.model.lora_rank=8 \
+ actor_rollout_ref.model.lora_alpha=16 \
+ actor_rollout_ref.model.lora_dtype=float32 \
+ actor_rollout_ref.model.lora.merge=true \
+ actor_rollout_ref.model.enable_gradient_checkpointing=true \
+ actor_rollout_ref.model.use_remove_padding=true \
+ actor_rollout_ref.model.exclude_modules="${EXCLUDE_MODULES}" \
+ actor_rollout_ref.model.target_modules="['q_proj','k_proj','v_proj','o_proj']" \
+ actor_rollout_ref.actor.freeze_vision_tower=true \
+ actor_rollout_ref.actor.strategy=fsdp2 \
+ actor_rollout_ref.actor.optim.lr=3e-6 \
+ actor_rollout_ref.actor.optim.weight_decay=0.01 \
+ actor_rollout_ref.actor.optim.clip_grad=1.0 \
+ actor_rollout_ref.actor.ppo_mini_batch_size=4 \
+ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
+ actor_rollout_ref.actor.use_dynamic_bsz=true \
+ actor_rollout_ref.actor.ppo_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.actor.use_kl_loss=false \
+ actor_rollout_ref.actor.entropy_coeff=0 \
+ actor_rollout_ref.actor.policy_loss.loss_mode=vanilla \
+ actor_rollout_ref.actor.clip_ratio_low=0.2 \
+ actor_rollout_ref.actor.clip_ratio_high=0.28 \
+ actor_rollout_ref.actor.clip_ratio_c=10.0 \
+ actor_rollout_ref.actor.loss_agg_mode=token-mean \
+ actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16 \
+ actor_rollout_ref.actor.fsdp_config.param_offload=true \
+ actor_rollout_ref.actor.fsdp_config.optimizer_offload=true \
+ actor_rollout_ref.rollout.name=vllm_omni \
+ actor_rollout_ref.rollout.n=2 \
+ actor_rollout_ref.rollout.temperature=0.8 \
+ actor_rollout_ref.rollout.tensor_model_parallel_size="${NUM_GPUS}" \
+ actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
+ actor_rollout_ref.rollout.max_num_seqs=16 \
+ actor_rollout_ref.rollout.load_format=safetensors \
+ actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2 \
+ actor_rollout_ref.rollout.enable_prefix_caching=false \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.output_mode="ar" \
+ +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe" \
+ actor_rollout_ref.rollout.val_kwargs.n=1 \
+ actor_rollout_ref.rollout.val_kwargs.temperature=1.0 \
+ actor_rollout_ref.rollout.val_kwargs.top_p=0.7 \
+ actor_rollout_ref.ref.strategy=fsdp2 \
+ actor_rollout_ref.ref.log_prob_use_dynamic_bsz=true \
+ actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=20480 \
+ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2 \
+ actor_rollout_ref.ref.fsdp_config.param_offload=true \
+ actor_rollout_ref.ref.fsdp_config.model_dtype=bfloat16 \
+ algorithm.trainer_type=policy_gradient \
+ algorithm.sample_source=online \
+ algorithm.adv_estimator=grpo \
+ algorithm.use_kl_in_reward=false \
+ algorithm.filter_groups.enable=false \
+ reward.reward_manager.source=register \
+ reward.reward_manager.name=dapo \
+ reward.reward_kwargs.max_resp_len=512 \
+ reward.reward_kwargs.overlong_buffer_cfg.enable=true \
+ reward.reward_kwargs.overlong_buffer_cfg.len=128 \
+ reward.reward_kwargs.overlong_buffer_cfg.penalty_factor=1.0 \
+ reward.reward_kwargs.overlong_buffer_cfg.log=true \
+ trainer.val_before_train=false \
+ trainer.balance_batch=true \
+ trainer.critic_warmup=0 \
+ trainer.logger=console \
+ trainer.project_name=verl-test \
+ trainer.experiment_name=dapo-qwen3-omni-thinker-lora-e2e-v1-wo-dynamic-sampling \
+ trainer.n_gpus_per_node="${NUM_GPUS}" \
+ trainer.nnodes=1 \
+ trainer.test_freq=1 \
+ trainer.save_freq=-1 \
+ trainer.resume_mode=disable \
+ trainer.total_training_steps="${TOTAL_TRAIN_STEPS}" \
+ "$@"
+
+echo "Qwen3-Omni Thinker DAPO+LoRA e2e V1 smoke without dynamic sampling passed."
diff --git a/tests/utils/test_dapo_overlong_reward_on_cpu.py b/tests/utils/test_dapo_overlong_reward_on_cpu.py
new file mode 100644
index 000000000..87d4a3574
--- /dev/null
+++ b/tests/utils/test_dapo_overlong_reward_on_cpu.py
@@ -0,0 +1,82 @@
+# Copyright 2026 Bytedance Ltd. and/or its affiliates
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Overlong penalty must change the reward on a truncated response (#446 Phase 2)."""
+
+import numpy as np
+import torch
+from omegaconf import OmegaConf
+from transformers import AutoTokenizer
+from verl import DataProto
+from verl.experimental.reward_loop.reward_manager.dapo import DAPORewardManager
+
+MAX_RESP_LEN = 16
+OVERLONG_BUFFER_LEN = 4
+OVERLONG_PENALTY_FACTOR = 1.0
+
+
+def _compute_score(data_source, solution_str, ground_truth, extra_info=None):
+ return 1.0
+
+
+def _build_manager(overlong_enable: bool) -> DAPORewardManager:
+ config = OmegaConf.create(
+ {
+ "reward": {
+ "reward_kwargs": {
+ "overlong_buffer_cfg": {
+ "enable": overlong_enable,
+ "len": OVERLONG_BUFFER_LEN,
+ "penalty_factor": OVERLONG_PENALTY_FACTOR,
+ "log": True,
+ },
+ "max_resp_len": MAX_RESP_LEN,
+ }
+ }
+ }
+ )
+ tokenizer = AutoTokenizer.from_pretrained("gpt2")
+ return DAPORewardManager(config, tokenizer, _compute_score)
+
+
+def _make_truncated_response() -> DataProto:
+ # valid_len == MAX_RESP_LEN, i.e. exceed_len == OVERLONG_BUFFER_LEN -> full penalty.
+ response_ids = torch.randint(0, 100, (1, MAX_RESP_LEN))
+ attention_mask = torch.ones(1, MAX_RESP_LEN, dtype=torch.long)
+ non_tensors = {
+ "data_source": np.array(["dummy"], dtype=object),
+ "reward_model": np.array([{"ground_truth": "x"}], dtype=object),
+ "extra_info": np.array([{}], dtype=object),
+ }
+ return DataProto.from_dict(
+ tensors={"responses": response_ids, "attention_mask": attention_mask},
+ non_tensors=non_tensors,
+ )
+
+
+def test_overlong_penalty_changes_reward_on_truncated_response():
+ batch = _make_truncated_response()
+
+ disabled = _build_manager(overlong_enable=False)
+ enabled = _build_manager(overlong_enable=True)
+
+ result_disabled = disabled.loop.run_until_complete(disabled.run_single(batch))
+ result_enabled = enabled.loop.run_until_complete(enabled.run_single(batch))
+
+ assert result_disabled["reward_score"] == 1.0
+ assert "overlong" not in result_disabled["reward_extra_info"]
+
+ assert result_enabled["reward_score"] < result_disabled["reward_score"]
+ assert result_enabled["reward_extra_info"]["overlong"]
+ assert result_enabled["reward_extra_info"]["overlong_reward"] < 0
diff --git a/tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py b/tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py
new file mode 100644
index 000000000..aa4f9f426
--- /dev/null
+++ b/tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py
@@ -0,0 +1,154 @@
+# Copyright 2026 Bytedance Ltd. and/or its affiliates
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+
+DAPO_WITHOUT_DYNAMIC_SAMPLING_SETTINGS = (
+ "python3 -m verl_omni.trainer.main_omni",
+ "actor_rollout_ref.actor.policy_loss.loss_mode=vanilla",
+ "actor_rollout_ref.actor.clip_ratio_low=0.2",
+ "actor_rollout_ref.actor.clip_ratio_high=0.28",
+ "actor_rollout_ref.actor.clip_ratio_c=10.0",
+ "actor_rollout_ref.actor.loss_agg_mode=token-mean",
+ "actor_rollout_ref.actor.use_kl_loss=false",
+ "actor_rollout_ref.actor.entropy_coeff=0",
+ "algorithm.trainer_type=policy_gradient",
+ "algorithm.sample_source=online",
+ "algorithm.adv_estimator=grpo",
+ "algorithm.use_kl_in_reward=false",
+ "algorithm.filter_groups.enable=false",
+ "reward.reward_manager.source=register",
+ '+actor_rollout_ref.rollout.engine_kwargs.vllm_omni.pipeline_name="qwen3_omni_moe"',
+)
+
+
+def _script_settings(script: str) -> set[str]:
+ return {line.strip().removesuffix("\\").rstrip() for line in script.splitlines()}
+
+
+def _assert_dapo_without_dynamic_sampling_contract(script: str) -> None:
+ settings = _script_settings(script)
+ assert set(DAPO_WITHOUT_DYNAMIC_SAMPLING_SETTINGS) <= settings
+ assert "actor_rollout_ref.actor.policy_loss.loss_mode=gspo" not in settings
+ assert "algorithm.filter_groups.enable=true" not in settings
+
+
+def test_dapo_example_launcher_has_phase_one_contract():
+ repo_root = Path(__file__).parents[2]
+ launcher = (repo_root / "examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh").read_text(
+ encoding="utf-8"
+ )
+
+ _assert_dapo_without_dynamic_sampling_contract(launcher)
+ settings = _script_settings(launcher)
+ assert ".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" in launcher
+ assert "actor_rollout_ref.actor.freeze_vision_tower=true" in settings
+ assert 'TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/avqa_r1_6k/train.parquet"}' in launcher
+ assert 'VAL_FILE=${VAL_FILE:-"$HOME/data/avqa_r1_6k/validation.parquet"}' in launcher
+ assert {
+ "data.custom_cls.name=QwenOmniRLHFDataset",
+ "data.seed=42",
+ "data.val_max_samples=-1",
+ "data.validation_shuffle=false",
+ "reward.reward_manager.name=naive",
+ "reward.custom_reward_function.path=verl_omni/utils/reward_score/choice_reward.py",
+ "reward.custom_reward_function.name=compute_score",
+ "trainer.val_before_train=true",
+ "trainer.test_freq=10",
+ "actor_rollout_ref.rollout.val_kwargs.n=1",
+ "actor_rollout_ref.rollout.val_kwargs.do_sample=false",
+ "actor_rollout_ref.rollout.val_kwargs.temperature=0",
+ "actor_rollout_ref.rollout.val_kwargs.top_p=1.0",
+ "actor_rollout_ref.rollout.val_kwargs.top_k=-1",
+ } <= settings
+ assert "actor_rollout_ref.rollout.val_kwargs.temperature=1.0" not in settings
+ assert "actor_rollout_ref.rollout.val_kwargs.top_p=0.7" not in settings
+ assert "data.val_max_samples=4" not in settings
+ assert "data.validation_shuffle=true" not in settings
+ assert "trainer.val_before_train=false" not in settings
+ assert "overlong_buffer_cfg" not in launcher
+
+
+def test_dapo_tiny_random_smoke_matches_example_contract():
+ repo_root = Path(__file__).parents[2]
+ smoke = (repo_root / "tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh").read_text(encoding="utf-8")
+
+ _assert_dapo_without_dynamic_sampling_contract(smoke)
+ settings = _script_settings(smoke)
+ assert "reward.reward_manager.name=dapo" in settings
+ assert "build_qwen3_omni_tiny_random.py" in smoke
+ assert "SKIP_COMPAT_DEPS_INSTALL:-0" in smoke
+ assert 'trainer.total_training_steps="${TOTAL_TRAIN_STEPS}"' in smoke
+
+ assert "data.max_response_length=512" in settings
+ assert {
+ "reward.reward_kwargs.max_resp_len=512",
+ "reward.reward_kwargs.overlong_buffer_cfg.enable=true",
+ "reward.reward_kwargs.overlong_buffer_cfg.len=128",
+ "reward.reward_kwargs.overlong_buffer_cfg.penalty_factor=1.0",
+ "reward.reward_kwargs.overlong_buffer_cfg.log=true",
+ } <= settings
+
+
+def test_dapo_dynamic_sampling_example_launcher_has_phase_three_contract():
+ repo_root = Path(__file__).parents[2]
+ launcher = (
+ repo_root
+ / "examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_dynamic_sampling_lora_v1.sh"
+ ).read_text(encoding="utf-8")
+
+ settings = _script_settings(launcher)
+ # Same token-level DAPO policy contract as Phase 1, minus filter_groups.enable=false.
+ assert set(DAPO_WITHOUT_DYNAMIC_SAMPLING_SETTINGS) - {"algorithm.filter_groups.enable=false"} <= settings
+ assert "actor_rollout_ref.actor.policy_loss.loss_mode=gspo" not in settings
+ assert ".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" in launcher
+ assert "actor_rollout_ref.actor.freeze_vision_tower=true" in settings
+ assert {
+ "algorithm.filter_groups.enable=true",
+ "algorithm.filter_groups.metric=acc",
+ "reward.reward_model.enable=false",
+ "reward.reward_manager.name=dapo",
+ "reward.custom_reward_function.path=verl_omni/utils/reward_score/choice_reward.py",
+ "reward.custom_reward_function.name=compute_score",
+ } <= settings
+ # filter_groups requires the streaming (non-colocated) reward path.
+ assert "reward.reward_model.enable=true" not in settings
+
+
+def test_dapo_mmk12_example_launcher_has_phase_four_contract():
+ repo_root = Path(__file__).parents[2]
+ launcher = (
+ repo_root / "examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_mmk12_v1.sh"
+ ).read_text(encoding="utf-8")
+
+ settings = _script_settings(launcher)
+ # Same token-level DAPO policy contract as the AVQA dynamic sampling launcher.
+ assert set(DAPO_WITHOUT_DYNAMIC_SAMPLING_SETTINGS) - {"algorithm.filter_groups.enable=false"} <= settings
+ assert "actor_rollout_ref.actor.policy_loss.loss_mode=gspo" not in settings
+ assert ".*talker.*|.*code2wav.*|.*code_predictor.*|.*visual.*|.*audio_tower.*" in launcher
+ assert "actor_rollout_ref.actor.freeze_vision_tower=true" in settings
+ assert {
+ "algorithm.filter_groups.enable=true",
+ "algorithm.filter_groups.metric=acc",
+ "reward.reward_model.enable=false",
+ "reward.reward_manager.name=dapo",
+ "reward.custom_reward_function.path=verl_omni/utils/reward_score/mmk12_reward.py",
+ "reward.custom_reward_function.name=compute_score",
+ "reward.reward_kwargs.overlong_buffer_cfg.enable=true",
+ "reward.reward_kwargs.overlong_buffer_cfg.len=1024",
+ "reward.reward_kwargs.overlong_buffer_cfg.penalty_factor=1.0",
+ "reward.reward_kwargs.max_resp_len=12288",
+ } <= settings
+ assert 'TRAIN_FILE=${TRAIN_FILE:-"$HOME/data/mmk12/train.parquet"}' in launcher
+ assert 'VAL_FILE=${VAL_FILE:-"$HOME/data/mmk12/test.parquet"}' in launcher
diff --git a/tests/workers/config/test_omni_config_on_cpu.py b/tests/workers/config/test_omni_config_on_cpu.py
index a73e34a4e..ffbd7c077 100644
--- a/tests/workers/config/test_omni_config_on_cpu.py
+++ b/tests/workers/config/test_omni_config_on_cpu.py
@@ -17,6 +17,7 @@
import os
import pytest
+from verl.trainer.config.algorithm import AlgoConfig, FilterGroupsConfig
from verl.workers.config.model import MtpConfig
from verl_omni.trainer.config.algorithm import OmniAlgoConfig
@@ -33,6 +34,10 @@ def test_defaults(self):
assert cfg.adv_estimator == "dpo"
assert cfg.norm_adv_by_std_in_grpo is True
assert cfg.global_std is True
+ assert isinstance(cfg, AlgoConfig)
+ assert cfg.use_kl_in_reward is False
+ assert cfg.filter_groups is None
+ assert cfg.kl_ctrl.type == "fixed"
@pytest.mark.parametrize(
"field_name, value",
@@ -45,6 +50,47 @@ def test_invalid_values_raise(self, field_name, value):
with pytest.raises(ValueError):
OmniAlgoConfig(**{field_name: value})
+ def test_inherited_online_fields_survive_hydra_conversion(self):
+ from hydra import compose, initialize_config_dir
+ from verl.utils.config import omega_conf_to_dataclass
+
+ import verl_omni
+
+ config_dir = os.path.join(os.path.dirname(verl_omni.__file__), "trainer/config")
+ with initialize_config_dir(config_dir=config_dir, version_base=None):
+ cfg = compose(
+ config_name="omni_trainer",
+ overrides=[
+ "algorithm.filter_groups.enable=true",
+ "algorithm.filter_groups.metric=acc",
+ "algorithm.filter_groups.max_num_gen_batches=7",
+ "algorithm.use_kl_in_reward=true",
+ "algorithm.kl_penalty=low_var_kl",
+ "algorithm.kl_ctrl.type=adaptive",
+ "algorithm.kl_ctrl.kl_coef=0.02",
+ "algorithm.kl_ctrl.target_kl=0.03",
+ "algorithm.rollout_correction.rollout_is=token",
+ "algorithm.rollout_correction.rollout_is_threshold=3.0",
+ ],
+ )
+
+ algorithm_cfg: OmniAlgoConfig = omega_conf_to_dataclass(cfg.algorithm)
+
+ assert isinstance(algorithm_cfg, OmniAlgoConfig)
+ assert isinstance(algorithm_cfg, AlgoConfig)
+ assert isinstance(algorithm_cfg.filter_groups, FilterGroupsConfig)
+ assert algorithm_cfg.filter_groups.enable is True
+ assert algorithm_cfg.filter_groups.metric == "acc"
+ assert algorithm_cfg.filter_groups.max_num_gen_batches == 7
+ assert algorithm_cfg.use_kl_in_reward is True
+ assert algorithm_cfg.kl_penalty == "low_var_kl"
+ assert algorithm_cfg.kl_ctrl.type == "adaptive"
+ assert algorithm_cfg.kl_ctrl.kl_coef == pytest.approx(0.02)
+ assert algorithm_cfg.kl_ctrl.target_kl == pytest.approx(0.03)
+ assert algorithm_cfg.rollout_correction is not None
+ assert algorithm_cfg.rollout_correction.get("rollout_is") == "token"
+ assert algorithm_cfg.rollout_correction.get("rollout_is_threshold") == pytest.approx(3.0)
+
class TestOmniLossConfig:
def test_defaults(self):
diff --git a/verl_omni/trainer/config/_generated_diffusion_trainer.yaml b/verl_omni/trainer/config/_generated_diffusion_trainer.yaml
index 5037b7875..235383b16 100644
--- a/verl_omni/trainer/config/_generated_diffusion_trainer.yaml
+++ b/verl_omni/trainer/config/_generated_diffusion_trainer.yaml
@@ -468,6 +468,13 @@ reward:
name: compute_score
reward_functions: {}
aggregation: weighted_sum
+ reward_kwargs:
+ overlong_buffer_cfg:
+ enable: false
+ len: 0
+ penalty_factor: 0.0
+ log: true
+ max_resp_len: null
reward_manager:
_target_: verl.workers.config.reward_model.RewardManagerConfig
source: importlib
diff --git a/verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml b/verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml
index ca59c25e2..cbad00ed0 100644
--- a/verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml
+++ b/verl_omni/trainer/config/_generated_diffusion_veomni_trainer.yaml
@@ -509,6 +509,13 @@ reward:
name: compute_score
reward_functions: {}
aggregation: weighted_sum
+ reward_kwargs:
+ overlong_buffer_cfg:
+ enable: false
+ len: 0
+ penalty_factor: 0.0
+ log: true
+ max_resp_len: null
reward_manager:
_target_: verl.workers.config.reward_model.RewardManagerConfig
source: importlib
diff --git a/verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml b/verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml
index 4784af554..e0f782bb0 100644
--- a/verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml
+++ b/verl_omni/trainer/config/_generated_omni_megatron_trainer.yaml
@@ -912,6 +912,13 @@ reward:
name: compute_score
reward_functions: {}
aggregation: weighted_sum
+ reward_kwargs:
+ overlong_buffer_cfg:
+ enable: false
+ len: 0
+ penalty_factor: 0.0
+ log: true
+ max_resp_len: null
reward_manager:
_target_: verl.workers.config.reward_model.RewardManagerConfig
source: register
diff --git a/verl_omni/trainer/config/_generated_omni_trainer.yaml b/verl_omni/trainer/config/_generated_omni_trainer.yaml
index d25b41e55..a3721152c 100644
--- a/verl_omni/trainer/config/_generated_omni_trainer.yaml
+++ b/verl_omni/trainer/config/_generated_omni_trainer.yaml
@@ -872,6 +872,13 @@ reward:
name: compute_score
reward_functions: {}
aggregation: weighted_sum
+ reward_kwargs:
+ overlong_buffer_cfg:
+ enable: false
+ len: 0
+ penalty_factor: 0.0
+ log: true
+ max_resp_len: null
reward_manager:
_target_: verl.workers.config.reward_model.RewardManagerConfig
source: importlib
diff --git a/verl_omni/trainer/config/algorithm.py b/verl_omni/trainer/config/algorithm.py
index 5eaf4f137..3b97f9a31 100644
--- a/verl_omni/trainer/config/algorithm.py
+++ b/verl_omni/trainer/config/algorithm.py
@@ -18,7 +18,7 @@
from typing import Optional
from verl.base_config import BaseConfig
-from verl.trainer.config.algorithm import RolloutCorrectionConfig
+from verl.trainer.config.algorithm import AlgoConfig, RolloutCorrectionConfig
from verl_omni.trainer.diffusion.diffusion_trainer_utils import OLD_POLICY_DECAY_SCHEDULES
@@ -60,7 +60,7 @@ def __post_init__(self):
@dataclass
-class OmniAlgoConfig(BaseConfig):
+class OmniAlgoConfig(AlgoConfig):
"""Omni-specific algorithm config."""
trainer_type: str = "direct_preference"
diff --git a/verl_omni/trainer/config/reward/reward.yaml b/verl_omni/trainer/config/reward/reward.yaml
index c53ec73fb..03414a5e7 100644
--- a/verl_omni/trainer/config/reward/reward.yaml
+++ b/verl_omni/trainer/config/reward/reward.yaml
@@ -29,6 +29,16 @@ reward_functions: {}
# Aggregation method for multi-reward. Only "weighted_sum" is supported.
aggregation: weighted_sum
+# Extra kwargs read directly by the reward manager (e.g. verl's "dapo" manager).
+reward_kwargs:
+ # max_resp_len must exceed overlong_buffer_cfg.len when enable is true.
+ overlong_buffer_cfg:
+ enable: false
+ len: 0
+ penalty_factor: 0.0
+ log: true
+ max_resp_len: null
+
# reward manager configuration
reward_manager:
_target_: verl.workers.config.reward_model.RewardManagerConfig