[VoiceChat] Add accuracy validation and thinker parity test - #2
Open
deepbuilder wants to merge 2 commits into
Open
[VoiceChat] Add accuracy validation and thinker parity test#2deepbuilder wants to merge 2 commits into
deepbuilder wants to merge 2 commits into
Conversation
added 2 commits
August 19, 2026 23:20
…tation Records the measured agreement between these stages and the vLLM-Omni implementation of the same checkpoint, on a 15.61 s reference sample (196 acoustic frames at 12.5 Hz). The thinker's frame-locked text timeline matches token for token, 196/196, in bfloat16 against a float32 reference. Driving the same comparison end to end -- a WAV through the sidecar's streaming perception encoder into the thinker -- also gives 196/196, so the streaming encoder's small deterministic difference from full-utterance encoding (cosine 0.99943) changes no tokens on this sample. The talker is deliberately excluded rather than reported as a low number. It samples, so two implementations draw from independent RNG streams and agree on only ~50% of codes even when both are correct; no seed closes that. The doc says so explicitly, along with what a real talker comparison would require. Perception and codec are NeMo's own modules on both sides, so those comparisons measure streaming behaviour rather than model correctness. Documented as such. Single-sample results, scoped as such in the doc.
Drives NemotronDuplexHForCausalLM over a streaming session and compares the emitted text timeline against a reference, token for token. The thinker decodes greedily, so this is a true parity check: the timeline can be compared against another implementation of the same model and must agree exactly. It does, 196/196 on the reference sample. Acoustic frames are supplied as a saved tensor rather than computed in the test, so it depends only on the thinker -- no encoder, no audio stack, no sidecar. A failure therefore points at one component. No reference artifacts are checked in. Passing --emit without a reference runs the stage and writes its output, so references can be generated once from a known-good source and used to gate later changes. The talker is not covered. It samples, so two implementations draw from independent RNG streams in their own order and disagree on roughly half the codes even when both are correct; an exact gate is not meaningful there. Lives under test/manual/ because it needs the full checkpoint, a converted stage, and a GPU.
deepbuilder
force-pushed
the
voicechat-parity-suite
branch
from
August 20, 2026 03:26
2698b5a to
8af39fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on sgl-project#34873 — base is
rystewart-nvidia:voicechat-support, notmain. Merging here first lets the VoiceChat work reach SGLang reviewers as one PR.Two independent commits, docs and one test. No changes to model or config code.
1. Accuracy validation (
874fc3b) — docs onlyexamples/voicechat/accuracy.md, plus one row in the Contents table.Records measured agreement against the vLLM-Omni implementation of the same checkpoint (added in vllm-omni#5842), on a 15.61 s reference sample — 196 acoustic frames on the 12.5 Hz timeline. SGLang runs the thinker in bfloat16; the vLLM-Omni reference runs float32.
NemotronDuplexHForCausalLM)The second row is the deployed path: a WAV in, a text timeline out, through the sidecar's streaming perception encoder and the thinker together.
Three things the doc is explicit about, because each is easy to misread:
nemo_audio_sidecar.pyimportsPerceptionCacheManagerandRVQVAEModelfromnemo.collections.speechlm2, and so does the reference. Comparing them measures streaming behaviour, not model correctness — the sidecar encodes frame by frame with a cache, the reference encodes the whole utterance at once. That difference is small and deterministic (cosine 0.99943, max abs 2.8e-02, identical with the perception CUDA graph on or off) and changes no tokens on this sample.2. Thinker parity test (
8af39fc)test/manual/voicechat_thinker_parity.py. One new file, nothing else touched.The thinker decodes greedily, so this is a true cross-implementation gate: the timeline can be compared against another implementation of the same model and must agree exactly. It does, 196/196.
Acoustic frames are supplied as a saved tensor rather than computed in the test, so it depends only on the thinker — no encoder, no audio stack, no sidecar. A failure points at one component.
No reference artifacts are checked in. Passing
--emitwithout a reference runs the stage and writes its output.Lives under
test/manual/because it needs the full checkpoint, a converted stage, and a GPU.Still to come
Reproducibility details for the parity check are not in this PR yet: a checksum for the input WAV, the published vLLM-Omni artifacts (
reference_text_tokens.jsonand the acoustic frames consumed by--reference-tokensand--acoustic-frames), and the vLLM-Omni commit and commands used to generate them. The input WAV and the 3.5 MB frames tensor need a download location first — see the discussion below.Testing
python/sglang/srt/models/eartts.py,python/sglang/srt/configs/eartts.pyandtest/registered/unit/models/test_voicechat_models.pyare byte-identical to this branch's base — verified as zero diff lines.CI States
Latest PR Test (Base): ❌ Run #32328257111
Latest PR Test (Extra): ❌ Run #32328257140