diff --git a/README.md b/README.md index 19c614da5..5a89a4cae 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ It originated from the multi-modal generation RL effort in `verl`, and now has a ## News 🔥 +- **[2026-09]** [Wan2.2 DanceGRPO](examples/dancegrpo_trainer/README.md) now defaults to the V1 sync trainer (TransferQueue). The v0 auto-detect launcher is deprecated for CUDA. - **[2026-08]** **[MiniMax-H3](examples/diffusionnft_trainer/minimax_h3/README.md)** text-to-audio-video is now supported with DiffusionNFT. See the [T2VA LoRA example](examples/diffusionnft_trainer/minimax_h3/README.md). - **[2026-08]** We have released [v0.2.0](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0) for faster diffusion rl and more stable Qwen3-Omni multimodal training. - **[2026-08]** [LTX2.3](examples/flowgrpo_trainer/ltx2/README.md) text-to-video+audio model is now supported with FlowGRPO. diff --git a/docs/examples/config.md b/docs/examples/config.md index 8243c15b2..6ab27410a 100644 --- a/docs/examples/config.md +++ b/docs/examples/config.md @@ -1,6 +1,6 @@ # Config Explanation -Last updated: 08/23/2026 +Last updated: 09/01/2026 VeRL-Omni builds on [verl](https://github.com/verl-project/verl) and reuses the same Hydra config surface for shared RL trainer fields (`data`, FSDP actor / @@ -110,8 +110,13 @@ standalone rollout GPUs from the Ray cluster. `actor_rollout_ref.rollout.agent.num_workers` controls CPU request concurrency; it does not allocate rollout GPUs and does not need to match `rollout.n_gpus_per_node`. -On a CUDA Ray cluster, the Wan2.2 auto-device recipe forwards trailing Hydra -overrides, so the same topology can be launched with the NCCL checkpoint backend: +This topology is v0-only (`trainer.use_v1=false`). The default CUDA DanceGRPO +recipe is now the V1 sync launcher (`run_wan22_5b_t2v_hpsv3_v1.sh`); use the +**deprecated** v0 auto-detect script below when you need +`actor_rollout_ref.separate`. + +On a CUDA Ray cluster, that v0 recipe forwards trailing Hydra overrides, so the +same topology can be launched with the NCCL checkpoint backend: ```bash bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh \ @@ -315,7 +320,10 @@ These sit on the diffusion trainer YAML (in addition to shared verl trainer fiel - `trainer.video_fps`: FPS for videos written to `rollout_data_dir` / `validation_data_dir` and logged to W&B (image runs ignore this). - `trainer.rollout_data_save_freq`: Dump train rollout every N steps (`1` = every step, `<= 0` = never). - `trainer.rollout_data_max_samples` / `validation_data_max_samples`: Cap samples dumped per train / val run (`null` = all). -- `trainer.use_v1`: Use the V1 trainer (TransferQueue + ReplayBuffer). When `false`, legacy v0 diffusion trainer. +- `trainer.use_v1`: Use the V1 trainer (TransferQueue + ReplayBuffer). When `false`, + the legacy v0 diffusion trainer. Wan2.2 DanceGRPO on CUDA now defaults to V1 + via `run_wan22_5b_t2v_hpsv3_v1.sh`; the v0 auto-detect launcher is deprecated + for CUDA. - `trainer.v1.*`: V1 mode / sampler / async placeholders (`trainer_mode`, `max_off_policy_threshold`, …). See {doc}`../start/diffusion_v1`. ### `reward` — visual reward manager diff --git a/docs/start/diffusion_v1.md b/docs/start/diffusion_v1.md index 3bc4dccb2..cd178e359 100644 --- a/docs/start/diffusion_v1.md +++ b/docs/start/diffusion_v1.md @@ -1,11 +1,12 @@ # Diffusion V1 training -Last updated: 08/25/2026 +Last updated: 09/01/2026 This guide runs the diffusion V1 trainer in synchronous or separate-asynchronous mode using the provided Stable Diffusion 3.5 Medium FlowGRPO OCR recipes. The V1 trainer uses TransferQueue and ReplayBuffer to move rollout trajectories into -the training loop. +the training loop. Wan2.2 DanceGRPO on CUDA also defaults to the V1 sync +recipe; see {doc}`../examples/dancegrpo_trainer`. The examples support a single-node NVIDIA GPU setup. Sync mode uses two GPUs for the colocated actor and rollout plus one reward GPU. Separate-async mode also @@ -89,6 +90,18 @@ Checkpoints are written by default to: checkpoints/flow_grpo/sd35_medium_ocr_lora_v1 ``` +### Wan2.2 DanceGRPO (default CUDA recipe) + +Wan2.2 DanceGRPO on CUDA now defaults to the same V1 sync trainer: + +```bash +bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh +``` + +See {doc}`../examples/dancegrpo_trainer` for dataset and HPSv3 setup. The +legacy v0 auto-detect script (`run_wan22_5b_t2v_hpsv3_auto.sh`) is +**deprecated** for CUDA and remains for NPU. + ## Run V1 separate-async mode Launch the separate-async recipe: diff --git a/docs/start/models.md b/docs/start/models.md index d249000c1..d33d7ae31 100644 --- a/docs/start/models.md +++ b/docs/start/models.md @@ -1,6 +1,6 @@ # Supported Models -Last updated: 08/19/2026. +Last updated: 09/01/2026. VeRL-Omni supports RL post-training for generative models across image, video, audio, and omni modalities. This page catalogues every model with a ready-to-run @@ -157,7 +157,12 @@ guidance scale 4.0, SDE window 2 over `[0,5]` at noise level 1.2, validation at | Trainer | Example script | GPU config | |---------|---------------|------------| -| DanceGRPO (HPSv3) | `examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_npu.sh` | 8×NPU (Ascend 800T A2) | +| DanceGRPO (HPSv3, V1 sync) | `examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh` | 8×GPU | +| DanceGRPO (HPSv3, v0, NPU) | `examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh` | 16×NPU (Ascend 800T A2) | + +The CUDA default is the V1 sync recipe (`main_diffusion_v1`, TransferQueue). +The v0 auto-detect launcher is **deprecated** for CUDA and remains for NPU +until a V1 NPU recipe lands. **Reward model:** HPSv3 (Human Preference Score v3) — local safetensors checkpoint placed at `$WORKSPACE/CKPT/HPSv3/HPSv3.safetensors`. diff --git a/examples/dancegrpo_trainer/README.md b/examples/dancegrpo_trainer/README.md index bce52dabf..d6311f89c 100644 --- a/examples/dancegrpo_trainer/README.md +++ b/examples/dancegrpo_trainer/README.md @@ -1,16 +1,24 @@ # DanceGRPO Trainer -Last updated: 08/07/2026 +Last updated: 09/01/2026 This example shows how to post-train `Wan2.2-TI2V-5B` with DanceGRPO on text-to-video generation tasks. DanceGRPO extends FlowGRPO with a score-based SDE step formulation for improved numerical stability during rollout sampling. -For the base Flow-GRPO setup, see [Examples - FlowGRPO Trainer](https://verl-omni.readthedocs.io/en/latest/examples/flowgrpo_trainer.html). For algorithm details, see [Algorithms - Flow-GRPO](../../docs/algo/flowgrpo.md). +For the base Flow-GRPO setup, see [Examples - FlowGRPO Trainer](https://verl-omni.readthedocs.io/en/latest/examples/flowgrpo_trainer.html). For algorithm details, see [Algorithms - Flow-GRPO](../../docs/algo/flowgrpo.md). For the V1 trainer (TransferQueue + ReplayBuffer), see [Diffusion V1 training](../../docs/start/diffusion_v1.md). + +The **default CUDA recipe** is the V1 sync launcher: + +- [`run_wan22_5b_t2v_hpsv3_v1.sh`](wan22/run_wan22_5b_t2v_hpsv3_v1.sh) — **GPU**, **V1 sync** (`verl_omni.trainer.main_diffusion_v1`) + +> **Deprecated:** `run_wan22_5b_t2v_hpsv3_auto.sh` is the legacy v0 launcher +> (`verl_omni.trainer.main_diffusion`). It remains for NPU auto-detect and +> backward compatibility. New CUDA runs should use the V1 script. ## Installation Follow the [installation guide](../../docs/start/install.md) to set up the base environment. -The provided script auto-detects whether NPUs or GPUs are available and configures the run accordingly (16 NPUs or 8 GPUs on a single node). +The V1 CUDA script targets 8 GPUs on a single node. The deprecated v0 auto-detect script still configures 16 NPUs or 8 GPUs. ## Prepare the dataset @@ -45,17 +53,26 @@ This produces: ## Run training -### HPSv3 reward +### HPSv3 reward (default: V1 sync) Launch the HPSv3 example from the repository root: ```bash -bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh +bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh ``` -For CUDA V1 sync (TransferQueue + ReplayBuffer), use `examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh`. +The V1 script is CUDA-only and selects `PolicyGradientDiffusionTrainerV1Sync` via `trainer.v1.trainer_mode=sync`. TransferQueue is force-enabled inside the runner. -The script auto-detects the device (`npu` via `npu-smi info`, or `gpu` via `nvidia-smi`) and exits with an error if neither is found. +For Ascend NPU, or to keep the legacy v0 trainer, use the auto-detect script: + +```bash +bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh +``` + +> **Deprecated:** the auto-detect script uses the v0 `main_diffusion` trainer. +> Prefer the V1 launcher on CUDA. The v0 script auto-detects the device +> (`npu` via `npu-smi info`, or `gpu` via `nvidia-smi`) and exits with an +> error if neither is found. #### Configurable environment variables @@ -78,10 +95,10 @@ TRAIN_FILES_PATH=/data/my_train.parquet \ VAL_FILES_PATH=/data/my_val.parquet \ MODEL_NAME=/path/to/local/model \ CUSTOM_REWARD_MODEL_PATH=/path/to/HPSv3.safetensors \ -bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh +bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh ``` -The script runs `python3 -m verl_omni.trainer.main_diffusion` with: +The V1 script runs `python3 -m verl_omni.trainer.main_diffusion_v1` with: - `algorithm.adv_estimator=dance_grpo` - `actor_rollout_ref.model.path=Wan-AI/Wan2.2-TI2V-5B-Diffusers` @@ -91,9 +108,13 @@ The script runs `python3 -m verl_omni.trainer.main_diffusion` with: - `actor_rollout_ref.rollout.algo.noise_level=1.2` - `actor_rollout_ref.rollout.algo.sde_window_size=2` - `reward.custom_reward_function.name=compute_score_hpsv3` -- `trainer.n_gpus_per_node=16` (NPU) or `8` (GPU) +- `trainer.use_v1=true` +- `trainer.v1.trainer_mode=sync` +- `trainer.n_gpus_per_node=8` - `trainer.total_training_steps=120` +The deprecated v0 auto-detect script still uses `python3 -m verl_omni.trainer.main_diffusion` and sets `trainer.n_gpus_per_node=16` (NPU) or `8` (GPU). + ## SDE variants DanceGRPO supports three SDE step variants via `actor_rollout_ref.rollout.algo.sde_type`: @@ -121,10 +142,10 @@ The script sets: ```bash trainer.project_name=dance_grpo -trainer.experiment_name=wan22_5b_t2v_hpsv3_npu # or wan22_5b_t2v_hpsv3_gpu +trainer.experiment_name=wan22_5b_t2v_hpsv3_gpu_v1 ``` -These values are set automatically based on the detected device. Override them by editing the `PROJECT_NAME` and `EXPERIMENT_NAME` variables in the script. +The V1 script hard-codes that experiment name. The deprecated v0 auto-detect script still sets `wan22_5b_t2v_hpsv3_npu` or `wan22_5b_t2v_hpsv3_gpu` from the detected device. Override them by editing the `PROJECT_NAME` and `EXPERIMENT_NAME` variables in the script. ### Diffusion-specific metrics diff --git a/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh b/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh index 460e0963d..1949f96d3 100644 --- a/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh +++ b/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh @@ -1,4 +1,9 @@ #!/bin/bash +# DEPRECATED (CUDA): This v0 launcher uses verl_omni.trainer.main_diffusion. +# New CUDA runs should use the V1 sync recipe: +# bash examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh +# This script remains for NPU auto-detect and backward compatibility. +# # Wan2.2 LoRA RL with DanceGRPO # # Model: Wan-AI/Wan2.2-TI2V-5B-Diffusers (text+image-to-video, used in T2V mode) @@ -9,7 +14,6 @@ # # Reference: https://github.com/XueZeyue/DanceGRPO and https://github.com/verl-project/verl-recipe/blob/main/dance_grpo/dance_grpo_mindspeed_mm/ # -set -x if npu-smi info &>/dev/null; then DEVICE="npu" @@ -20,6 +24,12 @@ else exit 1 fi echo "Detected device: $DEVICE" +if [ "$DEVICE" = "gpu" ]; then + echo "WARNING: run_wan22_5b_t2v_hpsv3_auto.sh is the deprecated v0 trainer." >&2 + echo "For CUDA, use examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh instead." >&2 +fi + +set -x if [ "$DEVICE" = "npu" ]; then export VERL_DATAPROTO_SERIALIZATION_METHOD=numpy diff --git a/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh b/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh index 1df24e35b..75ad610e8 100644 --- a/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh +++ b/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_v1.sh @@ -5,13 +5,13 @@ # Algorithm: DanceGRPO (reuses FlowGRPO's advantage estimator and loss) # Reward: HPSv3 (Human Preference Score v3) - custom reward model # -# This is the CUDA v1 counterpart of run_wan22_5b_t2v_hpsv3_auto.sh. It uses +# This is the default CUDA recipe for Wan2.2 DanceGRPO. It uses # `verl_omni.trainer.main_diffusion_v1`, which selects # `PolicyGradientDiffusionTrainerV1Sync` via `trainer.v1.trainer_mode=sync`. # TransferQueue is force-enabled inside the runner, so it does not need to be -# set on the CLI. NPU auto-detect remains on the v0 script. +# set on the CLI. NPU auto-detect remains on the deprecated v0 script. # -# Reference (legacy v0 script): +# Reference (deprecated v0 script): # verl-omni/examples/dancegrpo_trainer/wan22/run_wan22_5b_t2v_hpsv3_auto.sh # Reference: https://github.com/XueZeyue/DanceGRPO and https://github.com/verl-project/verl-recipe/blob/main/dance_grpo/dance_grpo_mindspeed_mm/ set -x