[3/N][model, rollout, reward] feat: add Qwen3-TTS GRPO support

README.md

@@ -191,16 +191,20 @@ Visit our [documentation](https://verl-omni.readthedocs.io/en/latest/index.html)
     <td>✅</td>
   </tr>
     <tr>
-    <td rowspan="2">Qwen3-TTS</td>
-    <td rowspan="2">Audio-modality</td>
-    <td rowspan="2">Text → Audio</td>
+    <td rowspan="3">Qwen3-TTS</td>
+    <td rowspan="3">Audio-modality</td>
+    <td rowspan="3">Text → Audio</td>
     <td>DPO</td>
     <td>WIP</td>
   </tr>
-    <tr>
+  <tr>
     <td>GSPO</td>
     <td>WIP</td>
   </tr>
+  <tr>
+    <td>GRPO</td>
+    <td>✅</td>
+  </tr>
 </table>
 
 

docs/api/pipelines.rst

@@ -4,25 +4,32 @@ Pipelines Interface
 Last updated: |today| (API docstrings are auto-generated).
 
 A *pipeline* in VeRL-Omni packages everything needed to plug a particular
-diffusion model architecture into the training loop:
+model architecture into the training loop. Two adapter families are available:
 
-- a **training-side adapter** subclassing
+- autoregressive omni models use a training-side
+  :class:`~verl_omni.pipelines.model_base.OmniModelBase` and an optional
+  rollout-side :class:`~verl_omni.pipelines.model_base.OmniRolloutPipelineBase`;
+- diffusion models use a training-side adapter subclassing
   :class:`~verl_omni.pipelines.model_base.DiffusionModelBase` that handles
   scheduler setup, model-input construction, and the per-step forward /
   reverse-sampling logic used by RL algorithms (e.g. FlowGRPO);
-- an optional **rollout-side adapter** registered via
+- their optional rollout-side adapter is registered via
   :class:`~verl_omni.pipelines.model_base.VllmOmniPipelineBase` that hooks
   into vLLM-Omni's diffusion serving stack to expose log-probabilities.
 
-Adapters are auto-selected by matching the pair
-``(DiffusionModelConfig.architecture, DiffusionModelConfig.algorithm)`` against the
-registered ``(architecture, algorithm)`` key. The architecture is read from the
-model's ``model_index.json``; the algorithm string is taken from the model config's
-``actor_rollout_ref.model.algorithm`` value.
+Autoregressive training adapters are selected by ``(architecture, model_stage)``;
+their rollout adapters are selected by the vLLM-Omni ``pipeline_name``. Diffusion
+adapters are selected by matching
+``(DiffusionModelConfig.architecture, DiffusionModelConfig.algorithm)`` against a
+registered ``(architecture, algorithm)`` key. Diffusion architecture is read from
+``model_index.json`` and the algorithm from
+``actor_rollout_ref.model.algorithm``.
 
 .. autosummary::
    :nosignatures:
 
+   verl_omni.pipelines.model_base.OmniModelBase
+   verl_omni.pipelines.model_base.OmniRolloutPipelineBase
    verl_omni.pipelines.model_base.DiffusionModelBase
    verl_omni.pipelines.model_base.VllmOmniPipelineBase
    verl_omni.pipelines.qwen_image_flow_grpo.QwenImage
@@ -33,6 +40,20 @@ model's ``model_index.json``; the algorithm string is taken from the model confi
 Model Base
 ~~~~~~~~~~~~~~~~~
 
+.. autoclass:: verl_omni.pipelines.model_base.OmniModelBase
+   :members: register, get_class, get_class_by_name,
+             register_auto_classes,
+             get_strip_modules, configure_processor, configure_tokenizer,
+             configure_model, prepare_model_inputs
+
+.. autoclass:: verl_omni.pipelines.model_base.OmniRolloutPipelineBase
+   :members: register, get_class,
+             build_stage_configs, rollout_flags, weight_sync_stage_ids,
+             get_pipeline_id, ensure_pipeline_registered, get_engine_hf_overrides,
+             get_stage_engine_extras, prepare_engine_prompt,
+             postprocess_agent_loop_output,
+             combine_engine_outputs
+
 .. autoclass:: verl_omni.pipelines.model_base.DiffusionModelBase
    :members: register, get_class,
              build_scheduler, set_timesteps,

docs/api/reward.rst

@@ -6,8 +6,9 @@ Last updated: |today| (API docstrings are auto-generated).
 VeRL-Omni reward pipelines support both rule-based scoring (e.g. JPEG
 compressibility) and model-based generative reward models (e.g. OCR via a
 vision-language model served behind an OpenAI-compatible router). Reward
-computation is dispatched per sample by the
-:class:`~verl_omni.reward_loop.reward_manager.VisualRewardManager`, which
+computation is dispatched per sample by modality-specific reward managers,
+including :class:`~verl_omni.reward_loop.reward_manager.VisualRewardManager`
+and :class:`~verl_omni.reward_loop.reward_manager.AudioRewardManager`, which
 plugs into :class:`~verl_omni.reward_loop.reward_loop.OmniRewardLoopManager` —
 verl's :class:`~verl.experimental.reward_loop.RewardLoopManager` extended with
 profiler control over the reward-model rollout servers.
@@ -17,8 +18,10 @@ profiler control over the reward-model rollout servers.
 
    verl_omni.reward_loop.reward_loop.OmniRewardLoopManager
    verl_omni.reward_loop.reward_manager.VisualRewardManager
+   verl_omni.reward_loop.reward_manager.AudioRewardManager
    verl_omni.utils.reward_score.default_compute_score_image
    verl_omni.utils.reward_score.http_scorer_client.compute_score
+   verl_omni.utils.reward_score.audio_http_scorer_client.compute_score
    verl_omni.utils.reward_score.unified_reward.compute_score_unified_reward
 
 Reward Loop Manager
@@ -33,6 +36,13 @@ Reward Manager
 .. autoclass:: verl_omni.reward_loop.reward_manager.VisualRewardManager
    :members: __init__, run_single
 
+.. autoclass:: verl_omni.reward_loop.reward_manager.AudioRewardManager
+   :members: __init__, run_single
+
+``AudioRewardManager`` reads ``audio`` and ``audio_sample_rate`` from rollout
+``extra_info``, validates a finite CPU float waveform, and calls a synchronous
+or asynchronous custom scorer with ``solution_audio=(waveform, sample_rate)``.
+
 Default Score Dispatcher
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -60,6 +70,12 @@ HTTP Scorer Client
 .. automodule:: verl_omni.utils.reward_score.http_scorer_client
    :members: compute_score
 
+Audio HTTP Scorer Client
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: verl_omni.utils.reward_score.audio_http_scorer_client
+   :members: compute_score
+
 UnifiedReward Scorer
 ^^^^^^^^^^^^^^^^^^^^^
 

docs/contributing/integrating_an_omni_model.md

@@ -13,10 +13,10 @@ under [`verl_omni/pipelines/`](https://github.com/verl-project/verl-omni/tree/ma
 Decide which **training stage** you want to train and how the model decomposes:
 
 - **Stage-split**: Multi-component omni models (thinker → talker → code2wav)
-  train only the text-understanding head during RL post-training. Other
-  components are stripped before FSDP wrapping to save memory. This is the
-  Qwen3-Omni pattern — the thinker is the autoregressive language model; talker
-  and codec are inference-only.
+  train one selected autoregressive stage during RL post-training. Other
+  components are stripped before FSDP wrapping to save memory. Qwen3-Omni
+  trains the thinker; Qwen3-TTS trains the talker's codec-0 policy while its
+  decoder remains rollout-only.
 - **Encoder-frozen**: Vision/audio encoders are typically frozen during RL
   training (`freeze_vision_tower=True`). The training adapter's
   `get_strip_modules` excludes them from the trainable set if they are separate
@@ -60,6 +60,11 @@ adapt each implementation to your model's architecture:
   `module._no_split_modules` to the correct decoder layer class for FSDP.
   This method runs before FSDP wrapping and LoRA injection.
 
+- **`register_auto_classes()`** (optional): Register classes supplied by an
+  optional model package with the appropriate Transformers Auto APIs. Qwen3-TTS
+  registers the official `qwen-tts` config and model with `AutoConfig` and
+  `AutoModelForTextToWaveform`; the FSDP engine still owns `from_pretrained`.
+
 - **`prepare_model_inputs(model_inputs, micro_batch, model_config)`**
   (optional): Validate model-native trajectory or conditioning data retained by
   rollout and add it to the actor forward inputs. Per-sample rollout data starts
@@ -70,6 +75,8 @@ adapt each implementation to your model's architecture:
   sequence alone cannot reconstruct the exact sampled trajectory. Missing
   required fields or inconsistent shapes should raise an actionable error; the
   adapter must not silently reconstruct a different trajectory.
+  Qwen3-TTS uses this hook to consume text tokens and all 16 codec codebooks
+  from its model-owned replay payload while optimizing codec-0 log-probabilities.
 
 Reference:
 [`verl_omni/pipelines/qwen3_omni/thinker_training_adapter.py`](../../verl_omni/pipelines/qwen3_omni/thinker_training_adapter.py)
@@ -93,10 +100,20 @@ and implement:
 - **`get_pipeline_id(pipeline_mode)`**: Return the vLLM-Omni pipeline
   `model_type` string, used when auto-generating the deploy config YAML.
 
-Optional overrides: `ensure_pipeline_registered` (register non-standard
-pipeline variants with vLLM-Omni), `get_engine_hf_overrides` (HF config
-overrides like `enable_audio_output: false`), `get_stage_engine_extras`
-(per-stage overrides like `model_arch`).
+Optional overrides fall into four groups:
+
+- Pipeline setup: `ensure_pipeline_registered`, `get_engine_hf_overrides`, and
+  `get_stage_engine_extras`.
+- Resource behavior: `weight_sync_stage_ids`.
+- Request construction: `prepare_engine_prompt`.
+- Multi-stage output assembly: `combine_engine_outputs`. The AR generation
+  strategy derives retained output modalities from stages marked
+  `final_output` in the pipeline topology.
+
+Their defaults preserve the existing single-output AR behavior. Override only
+the hooks required by the model. For example, Qwen3-TTS synchronizes actor
+weights only to its talker stage and retains both codec and waveform outputs;
+the decoder stage never receives actor weights.
 
 When training an omni model's autoregressive Talker stage, also override
 `postprocess_agent_loop_output`. Put the sampled policy sequence in
@@ -207,6 +224,9 @@ KV is cheap relative to starving decode.
 Reference:
 [`examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh`](../../examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh)
 
+For a talker-stage full-parameter example, see
+[`examples/grpo_trainer/qwen3_tts/run_qwen3_tts_grpo.sh`](../../examples/grpo_trainer/qwen3_tts/run_qwen3_tts_grpo.sh).
+
 ## 6. Common pitfalls
 
 These pitfalls are drawn from the Qwen3-Omni adapter. Some are
@@ -234,3 +254,10 @@ model-specific — verify each against your own model's architecture.
   in `configure_tokenizer` and assign it to `tokenizer.chat_template`.
   verl's dataset loader calls `tokenizer.apply_chat_template()` and will
   fail without a template.
+
+- **Actor/rollout probability consistency**: Autoregressive codec policies may
+  combine several codebook embeddings before predicting the selected token.
+  Match actor, reference, rollout, and weight-sync dtypes, then verify selected
+  token log-probabilities before training. The Qwen3-TTS recipe uses BF16 for
+  all four paths. Treat `diff_mean` and Pearson as consistency diagnostics, not
+  evidence of speech quality or bitwise agreement with an FP32 execution.

docs/examples/qwen3_tts/grpo_trainer_qwen3_tts.md

@@ -0,0 +1 @@
+../../../examples/grpo_trainer/qwen3_tts/README.md

docs/index.md

@@ -80,6 +80,7 @@ examples/mixgrpo_trainer.md
 examples/diffusionopd_trainer.md
 examples/flowgrpo_trainer_sd35_drm.md
 examples/bagel/flowgrpo_trainer_bagel.md
+examples/qwen3_tts/grpo_trainer_qwen3_tts.md
 examples/qwen_image_edit/flowgrpo_trainer_qwen_image_edit.md
 examples/ltx2/flowgrpo_trainer_ltx2.md
 examples/minimax_h3/diffusionnft_trainer_minimax_h3.md

docs/start/http_scorer.md

@@ -1,10 +1,17 @@
 (http_scorer)=
 # Using an External HTTP Scorer Service
 
-Last updated: 08/09/2026
+Last updated: 08/28/2026
 
 VeRL-Omni ships a generic HTTP reward client (`verl_omni.utils.reward_score.http_scorer_client`) that sends generated images to an external scorer service over HTTP and returns the score. This is useful when your reward model is too large to co-locate with training, needs a different runtime (e.g., a separate GPU pool), or is shared across multiple experiments.
 
+Audio rollouts use `verl_omni.utils.reward_score.audio_http_scorer_client`.
+That client sends JSON containing a base64-encoded float32 waveform,
+`sample_rate`, target `prompt`, and scalar metadata. The service returns a JSON
+object with a finite `score` and optional diagnostics. See the
+[Qwen3-TTS GRPO example](../../examples/grpo_trainer/qwen3_tts/README.md) for
+the complete audio protocol and configuration.
+
 ## How it works
 
 ```text

docs/start/models.md

@@ -223,6 +223,21 @@ rather than a separate per-stage YAML file.
 
 ---
 
+### Qwen3-TTS-12Hz-0.6B Base
+
+| Property | Detail |
+|----------|--------|
+| **Hugging Face ID** | `Qwen/Qwen3-TTS-12Hz-0.6B-Base` |
+| **Trainable component** | Talker codec-0 policy, full-parameter example |
+| **Rollout** | Two-stage vLLM-Omni Talker + code2wav pipeline |
+| **Algorithm** | Stock GRPO, vanilla PPO loss, optional direct KL |
+| **Reward** | Generic decoded-audio reward; SpeechJudge-BTRM external scorer example |
+
+The example uses two training GPUs and an independently deployed audio scorer.
+See [Qwen3-TTS GRPO with an audio reward](../../examples/grpo_trainer/qwen3_tts/README.md).
+
+---
+
 ## Model Architecture Summary
 
 | Model | Architecture | Text encoder |
@@ -245,7 +260,8 @@ rather than a separate per-stage YAML file.
 | Qwen2.5-VL-3B-Instruct | `Qwen/Qwen2.5-VL-3B-Instruct` | Vision-Language | SD3.5 (Flow-GRPO) | vLLM, TP=1, dedicated pool |
 | PickScore | `yuvalkirstain/PickScore_v1` | Vision (preference) | Qwen-Image-Edit (Flow-GRPO), BAGEL (PickScore recipe) | Local CLIP load, async workers |
 | HPSv3 | Local `.safetensors` | Vision (aesthetic) | Wan2.2 (DanceGRPO) | Local safetensors load |
-| HTTP scorer | External HTTP service | Any | Any model | Gunicorn/Flask, pickle protocol |
+| HTTP scorer | External HTTP service | Image/audio | Any model | Pickle image or JSON audio protocol |
+| SpeechJudge-BTRM | `RMSnow/SpeechJudge-BTRM` | Audio quality | Qwen3-TTS example | External service; CC-BY-NC-4.0 |
 | JPEG incompressibility | Rule-based | Image stats | Any diffusion model | No model process needed |
 
 For end-to-end instructions on setting up each reward, see the respective