[3/N][model, rollout, reward] feat: add Qwen3-TTS GRPO support

.github/actions/gpu-smoke-prepare/action.yml

@@ -20,7 +20,7 @@ runs:
         export UV_CACHE_DIR="${UV_CACHE_DIR:-${HOME}/.cache/uv}"
         git config --global http.postBuffer 524288000 || true
         # Base image PyPI mirror may lag pypi.org (e.g. kernels, fa3-fwd).
-        uv pip install --system --break-system-packages ".[gpu,dev,audio]"
+        uv pip install --system --break-system-packages ".[gpu,dev,audio,omni]"
         uv pip install --system --break-system-packages "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@$(cat .github/vllm_omni_pin.txt)"
         uv pip install --system --break-system-packages --no-deps --reinstall "verl @ git+https://github.com/verl-project/verl.git@$(cat .github/verl_pin.txt)"
         uv pip install --system --break-system-packages TransferQueue==0.1.8
@@ -30,7 +30,10 @@ runs:
         # TODO: rm --no-deps when VeOmni supports the vLLM torch pin (torch 2.13 as of vLLM 0.28)
         uv pip install --system --break-system-packages veomni==0.1.11 --no-deps
         uv pip install --system --break-system-packages torchcodec librosa soundfile av audioread
-        uv pip install --system --break-system-packages "transformers[mistral-common]==5.14.1"
+        # The released qwen-tts package targets Transformers 4.57.3. Install the
+        # tested upstream Transformers 5 source without changing this repo's stack.
+        uv pip install --system --break-system-packages --no-deps --reinstall \
+          "qwen-tts @ git+https://github.com/QwenLM/Qwen3-TTS.git@$(cat .github/qwen_tts_pin.txt)"
         # NCCL checkpoint engine (diffusion v1 separate_async weight sync).
         uv pip install --system --break-system-packages pyzmq
         uv pip install --system --break-system-packages cupy-cuda12x || uv pip install --system --break-system-packages cupy-cuda13x

.github/qwen_tts_pin.txt

@@ -0,0 +1 @@
+00969daa8064e23adc9e5f52cdf20cf247f94159

.github/workflows/cpu_unit_tests.yml

@@ -10,6 +10,7 @@ on:
       - "tests/**/*_on_cpu.py"
       - "pyproject.toml"
       - .github/workflows/cpu_unit_tests.yml
+      - .github/qwen_tts_pin.txt
       - .github/vllm_omni_pin.txt
       - .github/verl_pin.txt
   pull_request:
@@ -22,6 +23,7 @@ on:
       - "tests/**/*_on_cpu.py"
       - "pyproject.toml"
       - .github/workflows/cpu_unit_tests.yml
+      - .github/qwen_tts_pin.txt
       - .github/vllm_omni_pin.txt
       - .github/verl_pin.txt
 
@@ -56,6 +58,7 @@ jobs:
           cache: pip
           cache-dependency-path: |
             pyproject.toml
+            .github/qwen_tts_pin.txt
             .github/vllm_omni_pin.txt
             .github/verl_pin.txt
       - name: Install dependencies
@@ -64,7 +67,9 @@ jobs:
           pip install "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@$(cat .github/vllm_omni_pin.txt)"
           pip install TransferQueue==0.1.8
           pip install --no-deps "verl @ git+https://github.com/verl-project/verl.git@$(cat .github/verl_pin.txt)"
-          pip install ".[dev]"
+          pip install ".[omni,dev]"
+          pip install --no-deps \
+            "qwen-tts @ git+https://github.com/QwenLM/Qwen3-TTS.git@$(cat .github/qwen_tts_pin.txt)"
           pip install --no-deps -e .
       - name: Verify the documented train install resolves
         run: |

.github/workflows/gpu_smoke.yml

@@ -14,6 +14,7 @@ on:
       - "tests/special_e2e/**"
       - "pyproject.toml"
       - .github/workflows/gpu_smoke.yml
+      - .github/qwen_tts_pin.txt
       - .github/vllm_omni_pin.txt
       - .github/verl_pin.txt
       - .github/actions/gpu-smoke-prepare/**
@@ -32,6 +33,7 @@ on:
       - "tests/special_e2e/**"
       - "pyproject.toml"
       - .github/workflows/gpu_smoke.yml
+      - .github/qwen_tts_pin.txt
       - .github/vllm_omni_pin.txt
       - .github/verl_pin.txt
       - .github/actions/gpu-smoke-prepare/**

README.md

@@ -199,16 +199,20 @@ Visit our [documentation](https://verl-omni.readthedocs.io/en/latest/index.html)
     <td>✅</td>
   </tr>
     <tr>
-    <td rowspan="2">Qwen3-TTS</td>
-    <td rowspan="2">Audio-modality</td>
-    <td rowspan="2">Text → Audio</td>
+    <td rowspan="3">Qwen3-TTS</td>
+    <td rowspan="3">Audio-modality</td>
+    <td rowspan="3">Text → Audio</td>
     <td>DPO</td>
     <td>WIP</td>
   </tr>
-    <tr>
+  <tr>
     <td>GSPO</td>
     <td>WIP</td>
   </tr>
+  <tr>
+    <td>GRPO</td>
+    <td>✅</td>
+  </tr>
 </table>
 
 

docs/examples/qwen3_tts/grpo_trainer_qwen3_tts.md

@@ -0,0 +1 @@
+../../../examples/grpo_trainer/qwen3_tts/README.md

docs/index.md

@@ -84,6 +84,7 @@ examples/mixgrpo_trainer.md
 examples/diffusionopd_trainer.md
 examples/flowgrpo_trainer_sd35_drm.md
 examples/bagel/flowgrpo_trainer_bagel.md
+examples/qwen3_tts/grpo_trainer_qwen3_tts.md
 examples/qwen_image_edit/flowgrpo_trainer_qwen_image_edit.md
 examples/ltx2/flowgrpo_trainer_ltx2.md
 examples/minimax_h3/diffusionnft_trainer_minimax_h3.md

docs/start/models.md

@@ -235,6 +235,21 @@ parquet pairs and does not start rollout or reward workers.
 
 ---
 
+### Qwen3-TTS-12Hz-0.6B Base
+
+| Property | Detail |
+|----------|--------|
+| **Hugging Face ID** | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` |
+| **Trainable component** | Talker codec-0 policy, full-parameter example |
+| **Rollout** | Two-stage vLLM-Omni Talker + code2wav pipeline |
+| **Algorithm** | Stock GRPO, vanilla PPO loss, optional direct KL |
+| **Reward** | Generic decoded-audio reward; SpeechJudge-BTRM external scorer example |
+
+The example uses two training GPUs and an independently deployed audio scorer.
+See [Qwen3-TTS GRPO with an audio reward](../../examples/grpo_trainer/qwen3_tts/README.md).
+
+---
+
 ## Model Architecture Summary
 
 | Model | Architecture | Text encoder |
@@ -248,6 +263,7 @@ parquet pairs and does not start rollout or reward workers.
 | MiniMax-H3 | MiniMax H3 transformer | H3 text encoder |
 | BAGEL | Unified MM | — |
 | Qwen3-Omni-30B | Omni MoE | Qwen3 |
+| Qwen3-TTS-12Hz-0.6B | Talker + code2wav | Qwen3 |
 
 ---
 
@@ -262,7 +278,8 @@ parquet pairs and does not start rollout or reward workers.
 | CLAP | `laion/larger_clap_general` | Audio | LTX-2.3 (Flow-GRPO), MiniMax-H3 (DiffusionNFT) | Local transformers load |
 | ImageBind | Local `.pth` | Audio + Video | LTX-2.3 (Flow-GRPO), MiniMax-H3 (DiffusionNFT) | Local ImageBind package (CC-BY-NC-SA 4.0) |
 | DiNa-LRM | HTTP latent scorer | Diffusion latents | SD3.5 (Flow-GRPO DRM) | Separate `diffusion-rm` process, safetensors HTTP |
-| HTTP scorer | External HTTP service | Any | Any model | Gunicorn/Flask, pickle protocol |
+| HTTP scorer | External HTTP service | Image/audio | Any model | Pickle image or JSON audio protocol |
+| SpeechJudge-BTRM | `RMSnow/SpeechJudge-BTRM` | Audio quality | Qwen3-TTS example | External service; CC-BY-NC-4.0 |
 | JPEG incompressibility | Rule-based | Image stats | Any diffusion model | No model process needed |
 
 For end-to-end instructions on setting up each reward, see the respective
@@ -272,18 +289,20 @@ trainer's README in `examples/`.
 
 ## Which Trainer for Which Model?
 
-| Algorithm | Qwen-Image | Qwen-Image-Edit | SD3.5 | Wan2.2 | LTX-2.3 | MiniMax-H3 | BAGEL | Qwen3-Omni |
-|-----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
-| Flow-GRPO | ✅ | ✅ | ✅ | — | ✅ | WIP | ✅ | — |
-| Flow-DPPO | ✅ | — | — | — | — | — | — | — |
-| GRPO-Guard | ✅ | — | — | — | — | — | — | — |
-| Mix-GRPO | ✅ | — | — | — | — | — | — | — |
-| DanceGRPO | — | — | — | ✅ | — | — | — | — |
-| DPO | ✅ | — | ✅ | — | — | — | — | ✅ |
-| DiffusionNFT | ✅ | — | — | — | — | ✅ | — | — |
-| [DiffusionOPD](../algo/diffusion_opd.md) (incl. MOPD) | — | — | ✅ | — | — | — | — | — |
-| GSPO (incl. OPD) | — | — | — | — | — | — | — | ✅ |
-
-HunyuanImage-3.0 (MixGRPO / SRPO) and Qwen3-TTS (DPO / GSPO) appear on the
-project README as Planned or WIP and do not yet have a ready-to-run recipe, so
-they are omitted from the catalogue above.
+| Algorithm | Qwen-Image | Qwen-Image-Edit | SD3.5 | Wan2.2 | LTX-2.3 | MiniMax-H3 | BAGEL | Qwen3-Omni | Qwen3-TTS |
+|-----------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| GRPO | — | — | — | — | — | — | — | — | ✅ |
+| Flow-GRPO | ✅ | ✅ | ✅ | — | ✅ | WIP | ✅ | — | — |
+| Flow-DPPO | ✅ | — | — | — | — | — | — | — | — |
+| GRPO-Guard | ✅ | — | — | — | — | — | — | — | — |
+| Mix-GRPO | ✅ | — | — | — | — | — | — | — | — |
+| DanceGRPO | — | — | — | ✅ | — | — | — | — | — |
+| DPO | ✅ | — | ✅ | — | — | — | — | ✅ | WIP |
+| DiffusionNFT | ✅ | — | — | — | — | ✅ | — | — | — |
+| [DiffusionOPD](../algo/diffusion_opd.md) (incl. MOPD) | — | — | ✅ | — | — | — | — | — | — |
+| GSPO (incl. OPD) | — | — | — | — | — | — | — | ✅ | WIP |
+
+HunyuanImage-3.0 (MixGRPO / SRPO) appears on the project README as Planned or
+WIP and does not yet have a ready-to-run recipe, so it is omitted from the
+catalogue above. Qwen3-TTS DPO and GSPO remain WIP; its ready-to-run GRPO recipe
+is listed above.

examples/grpo_trainer/qwen3_tts/README.md

@@ -0,0 +1,160 @@
+# Qwen3-TTS GRPO with an audio reward
+
+Last updated: 09/04/2026.
+
+This example full-parameter tunes the codec-0 policy of
+`Qwen/Qwen3-TTS-12Hz-0.6B-Base`. It uses verl's stock GRPO advantage,
+vanilla PPO policy loss, and optional direct reference-model KL. The other
+15 codec codebooks and code2wav stage remain frozen but are retained so every
+candidate can be decoded and scored as audio.
+
+The launcher follows the V1 omni-model integration guide: it calls
+`verl_omni.trainer.main_omni` and expresses the recipe as CLI overrides on the
+standard `omni_trainer` config, without a model-specific Trainer or config tree.
+
+SpeechJudge-BTRM is one possible pointwise scorer. SpeechJudge's published vLLM
+entry point targets the pairwise generative GRM, while BTRM uses a scalar reward
+head with Transformers. This example therefore keeps reward inference behind the
+generic audio HTTP protocol instead of adding a SpeechJudge-specific Trainer
+path. The scorer may run in a separate environment from the Transformers 5.x
+vLLM training stack.
+
+## Algorithm background
+
+This recipe applies the paper's TTS GRPO flow to Qwen3-TTS: grouped codec-token
+rollouts are decoded, scored, converted to group-relative advantages, and
+replayed with optional reference KL. It optimizes codec-0, the autoregressive
+policy sequence described by the Qwen3-TTS architecture, while retaining all 16
+codebooks for replay and waveform decoding. The HTTP scorer is configurable, so
+this is not an exact reproduction of the paper's CER-and-NLL reward. See the
+references below for the algorithm and multi-codebook design details.
+
+## Install
+
+Install the engine before the training stack:
+
+```bash
+uv pip install -e ".[gpu]" --torch-backend=auto
+uv pip install "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@$(cat .github/vllm_omni_pin.txt)"
+uv pip install -e ".[omni,train,dev]"
+uv pip install --no-deps --reinstall \
+  "qwen-tts @ git+https://github.com/QwenLM/Qwen3-TTS.git@$(cat .github/qwen_tts_pin.txt)"
+```
+
+The pinned Qwen3-TTS revision is the upstream Transformers 5 support change
+from Qwen3-TTS PR #360. Its package metadata requires Transformers 5.15.1 or
+newer, while this repository intentionally caps Transformers at 5.14.1. The
+`--no-deps` flag preserves that repository-wide cap; the `omni` extra owns the
+runtime dependencies, including `torchaudio==2.11.0` to match vLLM's Torch pin,
+and CI tests the exact Qwen3-TTS revision from `.github/qwen_tts_pin.txt` on this
+stack. The released `qwen-tts==0.1.1` source targets Transformers 4.57.3 and
+cannot be imported unchanged here. The adapter registers the upstream config
+and model with `AutoConfig` and `AutoModelForTextToWaveform`; it does not carry
+a local Transformers compatibility layer. The system `sox` executable is also
+required by qwen-tts.
+
+## Data
+
+Training and validation parquet rows use the normal verl format:
+
+```python
+{
+    "data_source": "tts",
+    "prompt": [{"role": "user", "content": "Text to synthesize"}],
+    "reward_model": {"style": "model", "ground_truth": "Text to synthesize"},
+    "extra_info": {"id": "stable-id", "split": "train"},
+}
+```
+
+Use disjoint prompts. The default recipe evaluates the same complete 100-row
+validation parquet at step 0 and every 20 updates. It uses the rollout engine's
+global seed; model-specific per-request seed derivation is intentionally outside
+this integration.
+
+The concatenated replay layout also requires one fixed speaker embedding JSON.
+Generate it once with the official Qwen3-TTS Base model's
+`extract_speaker_embedding` API from a 24 kHz reference recording, then reuse
+the same file for the entire run.
+
+## Audio scorer protocol
+
+The configured endpoint receives one JSON request per candidate:
+
+```json
+{
+  "protocol_version": "1",
+  "waveform_f32_base64": "...",
+  "num_samples": 24000,
+  "sample_rate": 24000,
+  "prompt": "Text to synthesize",
+  "metadata": {"id": "stable-id"}
+}
+```
+
+It must return `{"score": 1.25}` and may include additional scalar metrics.
+The client retries only transient network, timeout, HTTP 408/429, and 5xx
+failures. Missing, malformed, or non-finite results stop the run instead of
+being converted to a valid zero reward.
+
+For SpeechJudge-BTRM, deploy the official
+[`AmphionTeam/SpeechJudge`](https://github.com/AmphionTeam/SpeechJudge) code and
+[`RMSnow/SpeechJudge-BTRM`](https://huggingface.co/RMSnow/SpeechJudge-BTRM)
+checkpoint in a separate environment, then expose its pointwise score through
+this protocol. The official [`main_grm_vllm.py`](https://github.com/AmphionTeam/SpeechJudge/blob/master/infer/main_grm_vllm.py)
+runs a different, pairwise generative GRM path; the BTRM entry point is
+[`main_btrm.py`](https://github.com/AmphionTeam/SpeechJudge/blob/master/infer/main_btrm.py).
+Pin the SpeechJudge source revision and runtime versions in the service
+deployment. SpeechJudge-BTRM is licensed CC-BY-NC-4.0.
+
+## Train
+
+```bash
+MODEL_PATH=/path/to/Qwen3-TTS-12Hz-0.6B-Base \
+TRAIN_FILE=/path/to/train.parquet \
+VAL_FILE=/path/to/fixed-validation-100.parquet \
+SPK_EMBED_PATH=/path/to/speaker.json \
+SCORER_URL=http://scorer-host:18080/score \
+OUTPUT_DIR=/path/to/output \
+bash examples/grpo_trainer/qwen3_tts/run_qwen3_tts_grpo.sh
+```
+
+The example defaults are `B=4`, `G=8`, `lr=1e-6` with 10 warmup steps and a
+constant schedule, direct `low_var_kl` with coefficient `0.12`, two GPUs, and
+500 updates. These are recipe values, not algorithm requirements.
+`norm_adv_by_std_in_grpo` remains at the upstream default. The actor and
+reference keep persistent parameters in FP32, while FSDP uses BF16 parameters
+for forward and backward computation with FP32 gradient reduction and buffers.
+The actor's AdamW state therefore remains FP32, and rollout inference remains
+BF16.
+
+For a two-update implementation smoke test:
+
+```bash
+TOTAL_TRAINING_STEPS=2 TEST_FREQ=-1 SAVE_FREQ=-1 RESUME_MODE=disable \
+OUTPUT_DIR=outputs/qwen3_tts_grpo_smoke \
+bash examples/grpo_trainer/qwen3_tts/run_qwen3_tts_grpo.sh \
+  trainer.val_before_train=false trainer.log_val_generations=0
+```
+
+This smoke proves rollout, finite audio reward, optimizer update, and
+post-update weight sync only. It is not evidence that GRPO improves held-out
+speech quality; that requires the complete fixed-validation curve and paired
+human listening evaluation.
+
+The CI-oriented wrapper at
+[`tests/special_e2e/run_qwen3_tts_grpo_smoke.sh`](../../../tests/special_e2e/run_qwen3_tts_grpo_smoke.sh)
+creates deterministic fixtures, uses an in-process CPU duration reward, and runs
+two updates with a pinned tiny-random checkpoint rebuilt for the 16-codebook
+actor contract.
+
+## References
+
+- Chang Liu, Ya-Jun Hu, Ying-Ying Gao, Shi-Lei Zhang, and Zhen-Hua Ling.
+  [Group Relative Policy Optimization for Text-to-Speech with Large Language
+  Models](https://arxiv.org/abs/2509.18798), 2025.
+- Hangrui Hu et al. [Qwen3-TTS Technical
+  Report](https://arxiv.org/abs/2601.15621), 2026.
+- QwenLM. [Qwen3-TTS PR #360: Support Transformers
+  5](https://github.com/QwenLM/Qwen3-TTS/pull/360), 2026.
+- Dong Zhang et al. [SpeechAlign: Aligning Speech Generation to Human
+  Preferences](https://arxiv.org/abs/2404.05600), 2024.