[2/N][reward] feat: add pointwise audio HTTP reward support

[dongbo910220]

What does this PR do?

This PR adds generic pointwise audio reward support for rollouts that already contain a decoded waveform:

• AudioRewardManager validates and forwards one waveform and sample rate per sample to a custom reward function;
• audio_http_scorer_client sends a strict JSON request to an external scorer and returns one finite scalar score;
• focused CPU tests cover waveform validation, sync/async scorers, JSON protocol validation, connection reuse, and transient retry behavior.

This does not duplicate #282. That PR implements a Qwen3-TTS-specific training pipeline with reward-side codec decoding and multiple reward modes. This PR is model-agnostic: it accepts an already-decoded waveform and implements only the pointwise reward-manager and HTTP transport boundary.

Checklist Before Starting

• Search for similar PRs. Queries: audio reward, HTTP scorer, and issue 428 references.
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)

Test

python -m pytest --noconftest -q \
  tests/reward_loop/test_audio_reward_manager_on_cpu.py \
  tests/utils/reward_score/test_audio_http_scorer_client_on_cpu.py

all passed

API and Usage Example

python -m verl_omni.trainer.main_omni \
  reward.custom_reward_function.path=pkg://verl_omni.utils.reward_score.audio_http_scorer_client \
  reward.custom_reward_function.name=compute_score \
  +reward.custom_reward_function.reward_kwargs.server_url=http://scorer:8000/score \
  reward.reward_manager.source=importlib \
  reward.reward_manager.name=AudioRewardManager \
  reward.reward_manager.module.path=pkg://verl_omni.reward_loop.reward_manager

The scorer receives JSON with a base64-encoded float32 waveform, sample rate, prompt, and scalar metadata. It must return {"score": <finite number>}.

Design & Code Changes

• Require audio and audio_sample_rate in rollout extra_info and reject malformed, empty, multi-sample, or non-finite waveforms.
• Support synchronous and asynchronous custom score functions without changing the reward-loop contract.
• Forward optional reward-router context to custom audio scorers, matching the visual reward-manager contract.
• Use a strict versioned JSON protocol instead of pickle for external audio.
• Reuse one aiohttp.ClientSession per event loop and retry only transient network, timeout, HTTP 408/429, and 5xx failures.
• Fail immediately on semantic 4xx errors, malformed responses, and non-finite scores.
• Document the manager and protocol in the public reward API and HTTP scorer guide.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code.

[Copilot]

🟡 Changes recommended

There are confirmed functional bugs in async scorer detection (partial-wrapped async functions) and waveform serialization (silently flattening non-1D waveforms) that can break real configurations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds a model-agnostic audio reward path to verl-omni by introducing an AudioRewardManager that validates decoded waveforms from rollout extra_info, plus an async JSON-over-HTTP client for sending waveforms to an external scorer and retrieving a single finite scalar score. It also updates public docs and adds CPU tests covering key validation and retry behaviors.

Changes:

• Added AudioRewardManager to validate/route (waveform, sample_rate) to custom reward functions (sync or async).
• Added audio_http_scorer_client implementing a strict JSON protocol, connection reuse, and bounded retry logic for transient failures.
• Added CPU tests and updated reward/HTTP scorer documentation to cover the new audio scoring pathway.

File summaries

File	Description
verl_omni/utils/reward_score/audio_http_scorer_client.py	New async JSON HTTP client for pointwise audio reward scoring, with strict request/response validation and retries.
verl_omni/reward_loop/reward_manager/audio.py	New reward manager that extracts/validates waveforms from rollout data and calls a custom scorer per sample.
verl_omni/reward_loop/reward_manager/init.py	Exports AudioRewardManager from the reward_manager package.
tests/utils/reward_score/test_audio_http_scorer_client_on_cpu.py	CPU tests for audio HTTP client protocol validation, retry behavior, session reuse, and thread offloading.
tests/reward_loop/test_audio_reward_manager_on_cpu.py	CPU tests for waveform validation/extraction, sync+async scorers, and failure modes.
docs/start/http_scorer.md	Documents the audio HTTP scorer client JSON protocol and retry/fail-closed behavior.
docs/api/reward.rst	Adds AudioRewardManager and audio_http_scorer_client to public reward API documentation.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ruihanglix]

look good to me

[zhtmike]

approvable after the evidence line is fixed; nothing unnecessary or fallback-shaped found — dual-layout audio extraction matches what the agent loop and ray_diffusion_trainer actually produce, fail-closed everywhere, async detection is correct through verl's reward_kwargs partial wrapping.

1. PR body: test claim is stale — says "34 passed" but the head collects 42 (ran locally: 42 passed). rerun and update the body.

2. verl_omni/utils/reward_score/audio_http_scorer_client.py: session-per-loop cache + bounded retry + fail-closed validation is now the third near-verbatim copy of this skeleton (http_scorer_client, latent_http_scorer_client, this). fine to land as-is given precedent, but let's extract a shared transport next time one of the three changes.

3. verl_omni/reward_loop/reward_manager/audio.py:41-42: reward_router_address / reward_model_tokenizer are stored but never used — visual.py forwards both into compute_score. drop the attributes or forward them for parity.

4. The two Copilot findings look stale at this head: partial-wrapped async fns are detected fine (inspect.iscoroutinefunction unwraps partials on py≥3.11; same pattern as visual.py), and _serialize_request raises on non-1D rather than silently flattening. no action needed.

AI assistance (ZCode) was used for this review.

[dongbo910220]

@zhtmike Thanks for the review.

1. I reran the focused tests and updated the PR body result to all passed.
2. Agreed. I kept the transport implementation scoped to this PR; the next change to any of the three clients should extract their shared transport.
3. AudioRewardManager now forwards the optional reward router address, reward model tokenizer, and model name to custom scorers, matching VisualRewardManager. I also added a regression test for this path.
4. Confirmed. No additional changes were made for the two stale Copilot findings.

[zhtmike]

All CI green, thanks for the work