Skip to content

[VoiceChat] Add accuracy validation and thinker parity test - #2

Open
deepbuilder wants to merge 2 commits into
rystewart-nvidia:voicechat-supportfrom
deepbuilder:voicechat-parity-suite
Open

[VoiceChat] Add accuracy validation and thinker parity test#2
deepbuilder wants to merge 2 commits into
rystewart-nvidia:voicechat-supportfrom
deepbuilder:voicechat-parity-suite

Conversation

@deepbuilder

@deepbuilder deepbuilder commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Stacked on sgl-project#34873base is rystewart-nvidia:voicechat-support, not main. Merging here first lets the VoiceChat work reach SGLang reviewers as one PR.

Two independent commits, docs and one test. No changes to model or config code.

1. Accuracy validation (874fc3b) — docs only

examples/voicechat/accuracy.md, plus one row in the Contents table.

Records measured agreement against the vLLM-Omni implementation of the same checkpoint (added in vllm-omni#5842), on a 15.61 s reference sample — 196 acoustic frames on the 12.5 Hz timeline. SGLang runs the thinker in bfloat16; the vLLM-Omni reference runs float32.

stage comparison result
Thinker (NemotronDuplexHForCausalLM) frame-locked text timeline, token for token 196/196 (100%)
Audio to text, end to end sidecar perception into the thinker, same reference timeline 196/196 (100%)

The second row is the deployed path: a WAV in, a text timeline out, through the sidecar's streaming perception encoder and the thinker together.

Three things the doc is explicit about, because each is easy to misread:

  • Perception and codec are NeMo modules on both sides. nemo_audio_sidecar.py imports PerceptionCacheManager and RVQVAEModel from nemo.collections.speechlm2, and so does the reference. Comparing them measures streaming behaviour, not model correctness — the sidecar encodes frame by frame with a cache, the reference encodes the whole utterance at once. That difference is small and deterministic (cosine 0.99943, max abs 2.8e-02, identical with the perception CUDA graph on or off) and changes no tokens on this sample.
  • The talker is not compared, by construction. It samples, so two implementations consume independent RNG streams and agree on only ~50% of codes even when both are correct. Measured 49.95%. No seed closes that. A real comparison would need pre-sampling distributions, teacher-forced, rather than emitted codes.
  • Single sample. Exact agreement on the reference input. Not a claim that streaming perception is numerically equivalent to full-utterance encoding in general.

2. Thinker parity test (8af39fc)

test/manual/voicechat_thinker_parity.py. One new file, nothing else touched.

The thinker decodes greedily, so this is a true cross-implementation gate: the timeline can be compared against another implementation of the same model and must agree exactly. It does, 196/196.

Acoustic frames are supplied as a saved tensor rather than computed in the test, so it depends only on the thinker — no encoder, no audio stack, no sidecar. A failure points at one component.

No reference artifacts are checked in. Passing --emit without a reference runs the stage and writes its output.

Lives under test/manual/ because it needs the full checkpoint, a converted stage, and a GPU.

Still to come

Reproducibility details for the parity check are not in this PR yet: a checksum for the input WAV, the published vLLM-Omni artifacts (reference_text_tokens.json and the acoustic frames consumed by --reference-tokens and --acoustic-frames), and the vLLM-Omni commit and commands used to generate them. The input WAV and the 3.5 MB frames tensor need a download location first — see the discussion below.

Testing

  • Thinker gate — 196/196 on the reference sample
  • Lint clean under the repo's pinned black / isort / ruff / codespell

python/sglang/srt/models/eartts.py, python/sglang/srt/configs/eartts.py and test/registered/unit/models/test_voicechat_models.py are byte-identical to this branch's base — verified as zero diff lines.


CI States

Latest PR Test (Base): ❌ Run #32328257111
Latest PR Test (Extra): ❌ Run #32328257140

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 15, 2026
raghavendran ramakrishnan added 2 commits August 19, 2026 23:20
…tation

Records the measured agreement between these stages and the vLLM-Omni
implementation of the same checkpoint, on a 15.61 s reference sample (196
acoustic frames at 12.5 Hz).

The thinker's frame-locked text timeline matches token for token, 196/196, in
bfloat16 against a float32 reference. Driving the same comparison end to end --
a WAV through the sidecar's streaming perception encoder into the thinker --
also gives 196/196, so the streaming encoder's small deterministic difference
from full-utterance encoding (cosine 0.99943) changes no tokens on this sample.

The talker is deliberately excluded rather than reported as a low number. It
samples, so two implementations draw from independent RNG streams and agree on
only ~50% of codes even when both are correct; no seed closes that. The doc says
so explicitly, along with what a real talker comparison would require.

Perception and codec are NeMo's own modules on both sides, so those comparisons
measure streaming behaviour rather than model correctness. Documented as such.

Single-sample results, scoped as such in the doc.
Drives NemotronDuplexHForCausalLM over a streaming session and compares the
emitted text timeline against a reference, token for token. The thinker decodes
greedily, so this is a true parity check: the timeline can be compared against
another implementation of the same model and must agree exactly. It does,
196/196 on the reference sample.

Acoustic frames are supplied as a saved tensor rather than computed in the test,
so it depends only on the thinker -- no encoder, no audio stack, no sidecar. A
failure therefore points at one component.

No reference artifacts are checked in. Passing --emit without a reference runs
the stage and writes its output, so references can be generated once from a
known-good source and used to gate later changes.

The talker is not covered. It samples, so two implementations draw from
independent RNG streams in their own order and disagree on roughly half the
codes even when both are correct; an exact gate is not meaningful there.

Lives under test/manual/ because it needs the full checkpoint, a converted
stage, and a GPU.
@deepbuilder
deepbuilder force-pushed the voicechat-parity-suite branch from 2698b5a to 8af39fc Compare August 20, 2026 03:26
@deepbuilder deepbuilder changed the title [VoiceChat] Add accuracy validation and parity gates [VoiceChat] Add accuracy validation and thinker parity test Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant