Feat/voice robustness - #99
Conversation
Add a "Noise: browser / RNNoise / off" selector to voice.html. RNNoise (@jitsi/rnnoise-wasm from jsDelivr) runs inside the capture worklet as a mic preprocessor: 48 kHz context, 480-sample frames denoised in place, then 3:1 decimation back to 16 kHz so the wire format is unchanged. Transport-agnostic by design — the same preprocessing carries over to a future WebRTC channel. Browser noiseSuppression is disabled while RNNoise is active (never two suppressors); echoCancellation stays on in every mode since the barge-in echo gates rely on it. Asset fetch failure, wasm init failure, and a browser refusing a 48 kHz context all degrade gracefully to the previous behavior. The hello reports the worklet's output rate (ctx/3 when denoising) so STT negotiation stays correct. Footer layout: mode selects (Turns, Noise) move to their own row so labels stay readable.
A wrong "incomplete" from Smart Turn (observed on short closers like "Thank you." p=0.036 and interjections like "No, no." p=0.378) costs the full hold before the turn runs — and no mid-hold re-score can rescue it, since the EOU backend trims trailing silence and would reproduce the same window and score. 3.0s matches both references: LiveKit's current endpointing max_delay default (6.0s was the old default) and Pipecat's SmartTurnParams stop_secs fallback. With our ~1.2s STT VAD silence before the hold arms, worst-case total is ~4.2s.
Mix transcript confidence into endpointing: audio-incomplete + finished text shortens the HOLD; audio-complete + hedge text (e.g. "I don't know.") delays via a short HOLD and a VAD delay bump. Floor VAD min_delay at 0.45s so confident scores never commit in ~0ms. Also fix hold expiry stretching on the commit's own partials, rescue stuck STT partials that never commit, and stop the playground from splitting one agent reply across two bubbles on stray mic partials.
…e stretch Wire VideoSDK Namo as the local text EOU with a terminal-punct lexical rescue so finished questions stop eating the incomplete HOLD. Cut the audio-incomplete/text-complete hold to 0.35s, and only extend HOLD on post-commit Silero speech so late STT refinements cannot floor short holds at the 2s grace window. Playground auto-starts; demo defaults to Groq Llama 3.1 8B Instant.
Mirror timbal.voice module layout (session, turn_detection, namo, smart_turn, endpointing, eou, vad, playback, realtime) out of tests/core.
…tials Refuse CONTINUE-merge of ultra-early short trailers onto punctuated turns so the LLM isn't cancelled/restarted; keep question VAD-splits and late barge-ins. Preserve LocalAudio NEW_TURN for those barge-ins (don't HOLD during TTS). Skip TranscriptPartial for VAD-vetoed barge-in hallucinations.
…ply barge-in Add Deepgram Flux/Nova realtime STT alongside ElevenLabs Scribe, wire provider resolution + Flux→provider turn-detector pairing, and expand the /voice playground (STT picker, LLM catalog, Foundry-style layout, tool status, cold-start warmup). Drop queued AudioOutput on interrupt so long TTS replies cut immediately instead of draining the WS backlog. Also coerce nullish tool args in serialization and add a slow get_datetime demo tool for tool-call dead-air testing.
PR SummaryHigh Risk Overview Introduces Namo ONNX text end-of-turn scoring ( VoiceSession fixes include stale-partial force-commit, HOLD that defers during TTS instead of interrupting, safer barge-in / queued-audio drop, The voice playground is reworked (STT/turn/LLM/noise-suppression controls, RNNoise worklet, explicit Start/Stop sessions). Tests and docs cover Deepgram, Namo, endpointing, and session edge cases; the turn-modes demo adds a slow Reviewed by Cursor Bugbot for commit 2b672ce. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Warm the session model override (not only agent.model), and on unknown stt_provider clear Flux/Nova model ids from the ElevenLabs fallback. Skip late_duplicate while a user turn or HOLD is active so prefix continuations reach CONTINUE_TURN instead of being ignored.
…LD defer test session_started was advertising the STT class name (DeepgramFluxSTT) instead of elevenlabs/deepgram-flux/deepgram-nova. Also wait for a post-commit partial before racing the hold timer so Windows CI doesn't expire mid-utterance.
push_audio and the flush loop both drained PCM then awaited _ws.send without a shared lock, so frames could overlap or reorder on one socket.
…all-clock The Windows flake compared turn start to the driver inject loop; inject only enqueues, while grace keys off _last_partial_at after drain. Wait for each partial to process and assert the hold stays armed past its timeout.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ffe35f4. Configure here.
2 participants
No description provided.