Feat/voice replay harness - #103
Merged
Merged
Conversation
Every voice test in python/tests/voice/ injects TranscriptEvents through
a mock STT, which covers the session state machine and nothing below it:
no audio, no real STT connection, no VAD, no endpointing. The bugs that
reach production live in that gap.
This replays ElevenLabs-synthesized speech into a real VoiceSession at
real-time pace, acting as a fake browser across the three seams the
session already exposes (paced audio in, event stream out, playback
acks). No session changes were needed. 20 scenarios across support,
food ordering, medical intake, banking and coding help.
Expectations are declarative and compare normalized similarity, never
verbatim transcripts, because STT wobbles between runs. Gating is
delta-based against a committed baseline.json rather than absolute
thresholds, which rot as providers drift. Scenarios that cannot pass
today are marked known_failure with a written reason: they report but
do not gate, and their desired expectations stay in the file instead of
being rewritten to match broken behavior.
Findings so far, all reproducible:
- Trailing-modifier continuations are invisible to every detector. The
fragment Flux commits mid-pause ("The pain started.") is a complete
sentence in isolation and Namo scores it 1.00; only prosody marks the
continuation, and SmartTurn v3 misses it too.
- Flux's end_of_turn_confidence cannot fix that. Across 28 commits,
fragments scored 0.690-0.920 and true turn ends 0.701-0.904. The best
threshold catches 4 of 5 fragments while wrongly holding 35% of real
turn ends. It measures whether speech stopped, not whether the thought
finished.
- lexical beats provider on Flux: 233ms vs 280ms median eou-to-audio and
one fewer known failure, since support_pause merges correctly there
instead of splitting and self-barging-in.
- Reproduced a genuine ghost turn (a spurious single-token 'I' commit)
and an intermittent split of short confirmations.
Needs ELEVENLABS_API_KEY and DEEPGRAM_API_KEY, makes real network calls
and runs in real time, so it is deliberately outside the default test
path and unlike the rest of benchmarks/, which measures pure framework
overhead.
The replay harness synthesized every Say() on its own, so ElevenLabs rendered each mid-sentence fragment as a complete sentence and gave it a falling, finished contour. A speaker who pauses mid-sentence does not. Measured with Smart Turn v3 on identical words ending at the same boundary, "I want to return an item I bought" scores 0.986 (finished) standalone and 0.019 (mid-thought) cut from the fluent render — a swing of 0.967 from intonation alone. The pause scenarios were testing TTS phrasing rather than turn-taking. fluent() now renders such sentences in one TTS call and slices them at the character timestamps ElevenLabs returns with the audio. Three known failures disappeared outright on every detector — food_list_pause, support_pause, banking_digits_pause — and their markers are deleted rather than relaxed. Also adds support_closer, the only scenario covering what the text-complete hold tier protects, after nearly removing that tier on evidence that turned out to be blind: disabling it merges medical_hesitant_pause 5/5 with no suite regression, but costs 2.7s of dead air on an under-scored closer (speech end to reply 0.91s -> 3.59s). Behavior left alone. No product code changes. Two candidate fixes were measured and rejected: restricting the effective_text_eou punctuation rescue to interrogatives is a one-for-one wash, and gating the hold-shrink on a confidently-wrong audio score was fitted to the old fixture — on faithful audio a real fragment scores 0.054, below the closer it was meant to separate from. All three cells re-baselined at 100%: local 19/19, lexical 19/19, provider 16/16. Known failures 6 -> 3.
--stt and --detector now cross into cells, each with its own scorecard, baseline entry and gate, plus a scenario x cell grid and within-cell flaky detection. --jobs overlaps the whole queue. Parallelism turned out to be nearly free, contradicting the assumption --jobs was built under. Across four full-suite runs, two serial and two at --jobs 4, p50 landed in 214-235ms serial and 213-222ms parallel: two serial runs of one cell differ as much as serial differs from parallel. Elapsed went 426s -> 114s. Real-time pacing is why - a replayed session is asleep between 20ms frames, so sessions interleave in the gaps instead of competing for inference. The guard stays regardless: --update-baseline is refused above --jobs 1 and latency is never compared across differing concurrency, neither of which the gate can verify at compare time. Widening to Nova and ElevenLabs reversed the detector verdict. Flux does turn detection itself, so all four detectors land within 10 points there and the conclusion drawn from that column - that the audio EOU buys nothing Namo does not already provide - was an artifact of the backend. On the two backends that return transcripts without deciding turns, local leads (89/79/87% against lexical's 92/68/89%) and provider collapses to 61/64%, scoring 0% on mid-sentence pauses because it defers to an STT that is not endpointing. Smart Turn earns its keep exactly where the STT does not endpoint for you. Two expectations were quietly wrong once a second provider existed. banking_digits failed in seven of twelve cells having committed exactly one turn in all twelve, because Flux writes "four four seven" where Nova writes "447" - and the suite had already been bitten by this and fixed it by rewriting the script to match Flux, the one fix that cannot survive another provider. normalize() now spells digit runs out. food_long_pause demanded a merge because that had been seen on Flux, so Nova failed it for splitting a 3.0s pause; it now asserts ContentPreserved rather than a turn count. Marker keys resolve cell-first (deepgram-nova/lexical, then deepgram-nova/*, then lexical, then *) so a Flux observation cannot excuse another backend again. The scenario library goes 21 -> 38, aimed at the edges: barge-in at 150ms, a one-word "Stop.", two interruptions in one session, an interruption made of the assistant's own words, a backchannel that must not interrupt, three-part sentences, self-corrections, a 12s run-on, two finished sentences 0.9s apart, and pure silence. Barge-in survived all of it - 36/36 in every backend, including the one-word case that MIN_BARGE_IN_PARTIAL_WORDS should have dropped, because the commit still lands even when the partial is ignored. Five scenarios had been confirming a solved problem; every new failure is a stop mid-thought. The most useful find is that the metric was hiding failures. coding_double_pause passed on ElevenLabs having dropped "inside a test" entirely: whole-string similarity scored the remainder 0.84 against the full sentence, over the 0.8 bar. The same bug passed banking_digits_pause with "291" missing from an account number. Merged and ContentPreserved now match each spoken fragment against its best transcript window, where a dropped part scores 0.40 instead of 0.84. Two green cells were lying. Three new failures belong to the STT, landing identically under all four detectors: Nova transcribes and commits "mm-hmm" so an acknowledgement silences the assistant, Nova endpoints at clause boundaries inside a 12s run-on, and ElevenLabs merges two finished sentences 0.9s apart. Timbal has no notion of a backchannel at all. Two older findings sharpened: the ghost single-token 'I' turn under Flux + provider was blamed on digit strings until medical_filler_midway reproduced it verbatim on ordinary words, and --repeat 3 caught coding_pause splitting 1-in-3 under provider after it had sat in the baseline as a clean pass, having only ever been run once per cell. No product code changes. All three Flux cells re-baselined serially at 100% - lexical 34/34, local 32/32, provider 27/27, p50 226-228ms, zero ghost turns, zero errors. Nova and ElevenLabs are measured every run but not baselined; their genuine failures need per-cell markers first. Known failures 3 -> 10 of 38, eight scoped to specific cells and eight intermittent, which is worth watching: a suite that marks everything interesting gates nothing.
eou->audio starts counting at the accepted commit, so every second a hold spends deciding costs nothing measurable, and pass/fail only ever saw whether the turn eventually merged. Between them the two metrics could not see hold policy at all. That blindness had teeth. Disabling the text-complete hold tier reads as 11 scenario-cells fixed and zero regressions, taking every local cell to 95% - Flux 89->95, Nova 79->95, ElevenLabs 87->95 - and on that evidence it looks like an obvious win. Timing speech end to commit says otherwise: it fires on 21% of scenario-cells, and the cost lands on cases that were already correct. Six barge-in cells take +2.6s before the assistant answers an ordinary question, three closers take +2.6s, and medical_long_utterance is counted as a fix while taking +8.2s. The tier stays. So: record speech end -> turn accepted per commit, add MaxDeadAir, and gate dead-air p50 per cell alongside latency with the same ratio, sample floor and --jobs-mismatch suppression. The two gates share _compare_timing because the old latency block returned early on a jobs mismatch and would have skipped dead air entirely - the same shape of bug as the one this commit is about. support_closer swaps its "deliberately carries no MaxLatency" note for MaxDeadAir(3500). It sits at 1.9s median and 2.8s worst today and lands near 4.5s with the tier removed, so the ceiling catches that specific regression. It is a tripwire on six samples, not a tight bound; the per-cell gate is the real instrument. Measured across the three gated cells: p50 560-699ms, p95 1.6-2.2s, n~50 each, all still 100%. Not gating yet - baseline.json has no dead_air_p50 entries until it is regenerated serially, so _compare_timing returns early for now. One thing this immediately surfaced and did not explain: dead air reaches 7.5s on banking_digits and banking_digits_pause, a plain account number with no hold involved, which is longer than anything the tier does. No product code changes. The tier experiment above was run by temporarily setting TEXT_COMPLETE_HOLD_TIMEOUT_SECS to 3.0 and reverted.
Formatter pass over the voice module, no behaviour change. Verified by comparing the parsed AST of each file against HEAD: all five are identical, so the diff is line joining and one list exploding onto separate lines. Split out from the endpointing work in the same tree precisely because it is large and empty — 111 changed lines in eou.py alone, which read at a glance like someone had rewritten the dangling-token table.
The VAD endpointing fast path only armed when the turn detector exposed an audio EOU model. But its job is telling the STT to finalize, which has nothing to do with how the turn end was decided, so the coupling locked it away from the configuration that needs it most: a text-only detector on an STT that does not endpoint aggressively just waits. Measured on deepgram-nova/lexical, a spoken account number sat 7.5s from speech end to commit, against 750ms for the same audio under local. Flux hid it completely by endpointing itself. The audio score was also the sole driver of the delay, not just an arming precondition, so there was no "arm without a model" path to switch on. endpointing_delay only needs a P(complete), so the endpointer now sizes the delay from the text score when no audio score exists, and skips the incomplete-text bump in that mode to avoid pricing the same signal twice. Arming Silero also switches on three behaviours that read its speech history as evidence about the user — barge-in vetoing, hallucination vetoing, hold extension — all of which branch on `_endpointer is None` and take conservative defaults today. Those stay off unless an audio EOU exists (`_vad_evidence`). Silero being loaded is not a reason to start vetoing barge-ins on a detector that never has, and a latency fix should not change barge-in policy as a side effect. Measured: banking_digits on nova/lexical 7.5s -> 1.07s, 3/3. The three gated Flux cells hold at 100% with dead air slightly better in all three (725->663, 819->638, 550->526ms). Barge-in under lexical is 9/9 across all three backends. banking_digits_pause on nova/lexical now splits, which is the honest outcome: the old merge was the 7.5s stall swallowing a 1.4s gap, and that cell splits pauses 80% of the time anyway. Two harness corrections fell out. support_barge_in capped heard text at 80 chars, inside the harness's own 250ms ack resolution — runs landed at 82 and 85 with the interrupt woperfectly, reading as a barge-in regression on a change that had none; now 100. And food_backchannel was marked only for deepgram-flux/local until a re-baseline caught the same "Mhmm." commit under lexical, so it is scoped to deepgram-flux/* with the measured rate. The refuse-to-save-on-failure guard is what caught it, rather than a green baseline being written over a real failure. Baselines regenerated serially so dead air gates: p50 725/819/550ms at n~50 per cell, above the 20-sample floor. Adds sweep.py and HarnessConfig.detector_params so a parameter can be varied without editing product constants and re-running by hand — which is how the hold tier came to be measured at 0.35 and 3.0 and nowhere between. First result, 390 runs over the pause-merge family: merges go 58% -> 90% across that range while median dead air stays flat at 793-889ms, because the tier fires on a minority of turns and its cost is a tail effect (p95 2270 -> 3617ms). 1.2 scores 81% at p95 1944ms. Not applied. On the full 12-cell id 1.2 is +7 on the three local cells against +-2/cell run-to-run noise, and no value of the timeout moves medical_self_correction (50%) or banking_correction (33%) — the two trailing-modifier cases with real consequences. The knob is not the lever for those. Default stays 0.35.
The tier shortens a HOLD when the audio model says incomplete but the transcript reads finished. 0.35 was chosen on the reasoning that a second tax on top of the endpointer's ~0.5-3s would feel like dead air, and was never revisited, because nothing could price it: eou->audio starts counting at the accepted commit, so hold duration is invisible to it by construction, and pass/fail only ever saw whether a turn merged. Swept once dead air became measurable. Over 0.35/0.8/1.2/2.0/3.0 on the pause-merge family, merges go 58% -> 90% while median dead air stays flat at 793-889ms. That flatness is the finding: the tier fires only when both signals disagree, a minority of turns, so the median cannot move and the cost is entirely in the tail. 0.35 looked free because the median is the wrong statistic for it. Confirmed on the full suite at --repeat 3, 684 runs across all three STT backends under `local`: 84% -> 92% correct, with every scenario this tier exists to protect unchanged at 100% — support_closer, all four barge-ins, banking_short_reject, coding_followup_after_reply. coding_double_pause 33 -> 100%, medical_long_utterance 67 -> 100%, medical_hesitant_pause 0 -> 67%, banking_confirmation 78 -> 100%. One scenario slips, food_trailing_preposition 100 -> 89%, one run in nine. The cost is p95 dead air 1969 -> 2683ms. 3.0 scores higher on merges still (90% vs 81%) and is not worth taking: p95 goes to 3617ms and support_pause_short starts failing, which is a short-pause floor case that should never be affected by a hold at all. Gated Flux cells stay at 100% and their dead-air p50 improved (725->680, 819->615, 550->486ms), so no re-baseline is needed. The sweep also bounds what this knob can do. medical_self_correction tops out at 33% and banking_correction at 22% at every value tried — the two trailing-modifier cases with real consequences, where the agent acts on the retracted half of a self-correction. Their problem is not how long thld runs but that nothing arms one when both signals read finished, so no timeout fixes them. Adds --all to sweep.py: pricing a candidate needs the whole suite, since the cost of holding longer lands on closers and barge-ins that the pause-merge family excludes by construction.
_likely_stt_echo exists so speaker bleed does not make the assistant interrupt itself. Every run of this harness so far — roughly 3,000 of them — fed clean user-only audio, so it had never been exercised once. coding_barge_in_echo proved only that it does not fire on genuine speech, which is the easy half. ScriptFeeder can now mix the assistant's own output back into the mic at a configurable gain (--aec-leak), capped at 1s of buffered audio so TTS running ahead of realtime does not leak the assistant's voice into silence long after it stopped talking. support_echo_silence scripts the case that isolates it: the user says nothing for the entire reply, so anything committed came out of the assistant's own mouth. On deepgram-flux/local it is clean at 0.15, fails 1 in 3 at 0.20, and 4 in 6 at 0.30 — the assistant cutting itself off mid-reply and committing its own words as a user turn. Real barge-in still passes at 0.30, so thisho getting through rather than the suppressor over-firing. The committed ghosts say why: 'Our retailers.', 'Arise retailers.', 'has aroused retailers.' — three manglings of the reply's own tail, '...authorized retailers.' The suppressor is a text-similarity check against what the assistant just said, so echo the STT transcribes badly stops resembling its source and passes the filter built to catch it. Clean echo is suppressed; garbled echo is not, which is the opposite of the intuition that louder bleed is the dangerous case, and it is why the boundary is probabilistic rather than a threshold — it turns on how the STT mis-hears a given phrase, not on gain. No fix here, and text similarity alone cannot carry this. The endpointer's Silero speech history is the obvious corroborating signal and _vad_vetoes_barge_in already uses it, but only when an audio EOU exists. Leak is opt-in and defaults to 0.0. Default suite unchanged: 39 scenarios, three gated Flux cells at 100%.feat(bench): simulate AEC leak, and find the echo suppressor failing _likely_stt_echo exists so speaker bleed does not make the assistant interrupt itself. Every run of this harness so far — roughly 3,000 of them — fed clean user-only audio, so it had never been exercised once. coding_barge_in_echo proved only that it does not fire on genuine speech, which is the easy half. ScriptFeeder can now mix the assistant's own output back into the mic at a configurable gain (--aec-leak), capped at 1s of buffered audio so TTS running ahead of realtime does not leak the assistant's voice into silence long after it stopped talking. support_echo_silence scripts the case that isolates it: the user says nothing for the entire reply, so anything committed came out of the assistant's own mouth. On deepgram-flux/local it is clean at 0.15, fails 1 in 3 at 0.20, and 4 in 6 at 0.30 — the assistant cutting itself off mid-reply and committing its own words as a user turn. Real barge-in still passes at 0.30, so thisho getting through rather than the suppressor over-firing. The committed ghosts say why: 'Our retailers.', 'Arise retailers.', 'has aroused retailers.' — three manglings of the reply's own tail, '...authorized retailers.' The suppressor is a text-similarity check against what the assistant just said, so echo the STT transcribes badly stops resembling its source and passes the filter built to catch it. Clean echo is suppressed; garbled echo is not, which is the opposite of the intuition that louder bleed is the dangerous case, and it is why the boundary is probabilistic rather than a threshold — it turns on how the STT mis-hears a given phrase, not on gain. No fix here, and text similarity alone cannot carry this. The endpointer's Silero speech history is the obvious corroborating signal and _vad_vetoes_barge_in already uses it, but only when an audio EOU exists. Leak is opt-in and defaults to 0.0. Default suite unchanged: 39 scenarios, three gated Flux cells at 100%.
Two findings from the AEC leak work, neither of which changes product code. _likely_stt_echo's fuzzy branch is unreachable for any reply longer than about 100 characters. SequenceMatcher.ratio() is 2M/(len(c)+len(tail)) with M <= len(c), and the tail is sized at max(3*len(c), 100), so the score is capped at 0.50 against a 0.68 gate — a perfect match scores well under the threshold. It can only fire while the assistant's whole reply is still shorter than the window, which is why every observed ghost came from the tail of a long reply and none was an exact substring. A best-window comparison was tried and reverted. It separates the recorded ghosts (0.615-0.963) from clean user phrases (0.340-0.500) cleanly, and takes support_echo_silence at 0.30 leak from 4-in-6 failing to 6/6 passing — while breaking barge-in outright. support_barge_in passes at 0.30 leak today and failed every run with the change, leaving the assistant uninterruptible, which is the worse of the two failures. The threshold was set against the wrong sample. Under leak the STT transcribes a blend of user speech and echo, not one or the other, and the blend lands between the distributions that looked so well separated on clean phrases. There are not two populations here to threshold apart. That leaves the audio reference — correlating the mic against recently played output, the way real AEC does, with both signals already available to the session. Silero is not the alternative despite being to hand: _vad_vetoes_barge_in already notes that speaker echo carries energy, so VAD cannot distinguish it from speech.
Nova and ElevenLabs were measured on every run and gated nothing, so nine of twelve cells could regress silently — including the four where the detector does real work, which is where the tier retune and the endpointer fix both moved behaviour. Censused those nine at --repeat 3 (1,041 runs): 77 failing scenario-cells, but not 77 findings. 29 belong to `heuristic` and 24 to `provider`, and neither can hold a mid-sentence pause off Flux at all — one has no EOU model, the other delegates to an STT that is not endpointing. Marking those 53 would record a single architectural fact 53 times and turn known_failure into wallpaper. So deepgram-nova and elevenlabs x local and lexical are baselined at 100%, taking the gate from 3 cells to 7, at a cost of 24 cell-scoped markers each citing a measured rate. deepgram-flux/heuristic and heuristic/provider on the other two backends stay measured and reported but ungated: their pause failures are a property of picka detector with nothing to hold with. Markers cite three shared reasons rather than 24 bespoke ones — _TEXT_ONLY_PAUSE_SHORTFALL, _AUDIO_PAUSE_SHORTFALL, _EL_OVERMERGE — so a fix to any one shows up as a block of unexpected passes rather than a scatter to chase individually. Known failures are 18 of 39 scenarios now, which is high. It is also concentrated: nova/lexical carries 13 of them and scores 20% on pause merges by measurement, so the ratio reflects a detector limit rather than a suite that has given up. support_trailing_conjunction picked up an elevenlabs/lexical marker after passing 3/3 in the census and failing the confirmation run — 3/4, marked intermittent. All seven cells were baselined serially in this pass, including the three Flux cells, which had been holding pre-retune dead-air numbers (725/819/550ms) the gate only had slack against. They now read 668/656/489ms against the 1.2s tier the code actually ships.
A HOLD only extended on a new STT partial, which loses a race the
provider creates. When it commits on a short silence, the next
fragment's audio is already in flight — measured on ElevenLabs at a 0.3s
VAD threshold, ~1.3s before any partial existed to extend on — so a 1.2s
hold fired mid-sentence and split the utterance. Holds now also extend on
Silero hearing speech, capped at 3s per hold because echo surviving an
imperfect canceller carries energy and an uncapped mic-driven hold could
be held open for as long as the assistant speaks.
This is why the ElevenLabs 1.2s vad_silence_threshold_secs looked
load-bearing: at 0.3s the pause-merge family scored 47% against 86% at
1.2s, and that 39-point gap was the provider's VAD doing the holding our
own merge path was failing at. It is 6 points now, so the threshold is a
tunable rather than a crutch — and the last support_pause marker,
deepgram-nova/lexical, is dropped: a text-only detecs no prosody to
hold on, so it was expiring mid-pause, and it passes 3/3 now. It was
never marked intermittent, so it had been failing reliably.
Confirming that on the full matrix caught the gate reporting the fix as a
defect. count_ghost_turns asked whether a committed turn resembled *one*
script entry at >=0.6, so a correct three-fragment merge — about 2.5x
longer than any fragment it was compared against — resembled none of
them. `ghost turns 1 -> 3` in exactly the four cells where merging
improved, three repeats of coding_double_pause, the only scenario of 38
with three fragments and so the only one that could expose it. It
mis-scored the inverse too: medical_long_utterance split three ways
counted two ghosts, because a piece of a run-on does not resemble the
whole. Turns are matched against a window of the whole script now
(best_window, already here for HeardPrefix), since turn boundaries are
the thing under test and are not expected to line up with script
boundaries. Under three words keeps the pntry rule, or a spurious
lone "I" — which Flux really does emit — stops being countable.
Chasing two apparent flakes then found the harness mis-measuring
ElevenLabs across the whole pause family. 69 failures read `fragments
never heard`, the worst class here; 68 were ElevenLabs, over 13
scenarios, concentrated in `provider` (31) and `heuristic` (29) — the two
detectors with no Timbal-side endpointing, so the two that wait on the
provider's own commit. AwaitCommit sampled its watermark when the last
Say *started*, and ElevenLabs commits ~1.6s after speech ends, so the
*previous* fragment's commit satisfied the wait meant for the fragment
under test. What was left was a 0.8s tail plus a 0.5s drain against a
commit needing ~1.0s more, and the turn died in teardown. Nova commits in
~600ms, before the next fragment is spoken, so its watermark stayed
honest — the bias tracked commit latency and penalised the slower
provider twice.
Waits are quiescence-based now: satisfied by an event arriving after the
's audio finished feeding, or by the session going quiet for 1.0s
with something already in hand. Requiring only the first was tried and is
wrong in the other direction — the provider can hold all the audio it
needs before the harness finishes handing the clip over, and food_simple
commits '...a large coffee and a croissant' without the final '?' for
exactly that reason, so nothing ever arrives "after". That cost 4 runs in
a 111-run cell, medical_barge_in_twice taking 46s to fail on three turns
it had already committed correctly. The comment arguing for the early
watermark was right that the case exists and wrong about which utterance
it belonged to.
food_long_pause's elevenlabs/lexical marker is dropped with it. It blamed
the provider for dropping audio across the gap; the provider splits,
4/4, content intact. Its note also claimed ElevenLabs merged here, which
is what one observed turn instead of two looks like if you count turns.
Baselines may now be taken in parallel. --update-baseline refused
--job> 1 on the grounds that contention would bake the machine's load
into the latency the gate compares against; measured on
deepgram-nova/local, --quick, 3 repeats, it does not — p50 278ms and p95
407ms at --jobs 6 against 280ms and 453ms serial, the only outlier being
a 962ms first-run model load in the *serial* run. eou->audio is mostly
spent waiting on STT and TTS sockets rather than competing for CPU. The
refusal bought nothing a busy machine at --jobs 1 would not also spoil
and priced a baseline at six times what it needs to cost. What keeps it
sound is unchanged: a baseline records its own concurrency and latency
comparison is declined across a mismatch.
Re-baselined: 1386 runs, 12 cells, zero session errors, ghost turns 0 in
eleven of twelve. Five Deepgram cells at 100% and --jobs 6, including
deepgram-flux/provider at 81/81. The ElevenLabs cells are refused and
keep stale --jobs 1 entries, because they are dropping barge-in commits
outright: medical_barge_in, support_barge_in, support_barge_in_insta
and support_barge_in_late under elevenlabs/lexical went 12/12 to 5/12 in
four hours. Not this change — forcing the waits back to resolving
immediately reproduces it identically at 0/3, the same scenario passes
3/3 on deepgram-nova/lexical, and ElevenLabs passes 4/4 on non-barge-in
scenarios. The trace shows the interrupting turn interrupting and then
never committing while invented partials ("I don't know.", "Yeah.")
arrive on the trailing silence, each one restarting the 1.2s timer that
would have committed it.
Also fixes four red tests in python/tests/voice/ that predate this: one
asserting the pre-retune 0.35s, and three setting session._endpointer
without _vad_evidence, so every test covering VAD veto behaviour was
exercising the no-evidence path. sweep.py can vary STT params (stt.*) and
no longer buries its own result table under DEBUG logs.
The stale-partial watchdog exists to force a commit when a provider transcribes speech and then never commits it — the live failure where the words hang as a "…" caption forever. It measured staleness from the last partial arrival, which a provider that hallucinates on trailing silence refreshes indefinitely: ElevenLabs emits invented partials 1.0-1.2s apart against a 2.5s threshold, so the same churn that stops it committing kept our rescue permanently disarmed. The user's interruption was lost outright in 8 of 12 barge-in runs. Mic silence now counts as well: if Silero heard essentially nothing in the window, the user demonstrably stopped speaking and the provider clearly will not commit, whatever it is still emitting. Measured 4/12 -> 15/24 on the ElevenLabs barge-ins with no session errors, and clean across 156 runs on the four baselined Deepgram cells. Not gated on _vad_evidence: that flag guards inferences which can suppress shing the user did, and this only rescues speech that would otherwise be lost. Two alternatives measured worse and are documented so they are not retried: commit_strategy="manual" (0/12, every run erroring on assistant audio) and dropping the synthesis fallback's text-stability requirement (4/12, and it synthesized a hallucinated "Yeah." as a turn). Tests cover both directions — churning partials over a quiet mic must rescue, a hot mic must still defer — the first verified to fail on the old anchor. Also drops the README's claim that --update-baseline refuses --jobs > 1, which is no longer true.
…s speech _likely_stt_echo compared a commit against a tail sized at 3x its length. ratio() is 2M/(len(c)+len(tail)), so that capped the score at 0.50 against a 0.68 threshold: the fuzzy branch was unreachable and a perfect echo scored below it. Only exact substrings were ever suppressed, which is why echo the STT garbled slightly walked through the filter built to catch it. Score against a slice of the tail the commit's own length instead, aligned on their longest shared run. Threshold calibrated on transcripts from --aec-leak runs: garbled echo scores 0.68-1.00, genuine barge-ins 0.24-0.58, and 0.65 sits in the gap nearer the echo side, since over-suppression makes the assistant uninterruptible. At leak 0.15 the gated suite goes 92% -> 100% with ghost turns halved; the four baselined clean cells stay at 100% over 126 runs. Also makes --aec-leak a real axis. Cells were matched to records by rebuilding "stt/detector", so any run with a leak or a swept param matched no cell at all: no scorecard, no gating, and --update-baseline silently did nothing. Records now carry the full label and the leak gain.
Per-scenario pass rates are matched by name, so they stay comparable under -s / --quick. Ghost turns, latency and dead air are not: each is computed over whatever subset ran, so comparing them against a full-suite baseline compares two different populations. This is not theoretical. Running the barge-in subset to check the new echo threshold reported "ghost turns 1 -> 3" while the cell was clean, because those eight scenarios carry most of the suite's ghost turns. The other direction is worse: a subset that excludes them reports an improvement just as falsely, and an improvement is not something anyone goes looking behind. compare() now takes `partial` and reports the aggregates as not comparable, which is the reasoning that already makes --update-baseline refuse a filtered run. Also records the cross-cell validation of the echo threshold from 403d4fd. All seven baselined cells were run. deepgram-flux/provider is the decisive one, since _likely_stt_echo is the only semantic filter between a partial and cancelling the reply there: 53/54, with all ten barge-in scenarios passing every repeat and support_echo_silence still uninterrupted. ElevenLabs' 17/24 and 21/24 are its documented hallucination on trailing silence ('Yes.', 'Yeah.'), not over-suppression -- the genuine barge-in is heard in every one.
Echo that commits just after playback was never checked: both call sites
gated on assistant_active, which drops when playback drains while the
transcript for that audio is still in flight. TurnState now carries
seconds_since_assistant_active and the check runs 2s past playback. Bounded
because assistant_text outlives its turn — with no deadline, a user utterance
resembling the last reply would be dropped, and dropping it means no new turn
starts to clear that text, so the suppression would never lift.
That alone fixed nothing. Tracing coding_followup_after_reply showed the
suppressor refusing 'memory access.' as echo, then the stale-partial watchdog
synthesizing it into a turn three seconds later: an IGNOREd commit does not
bump _last_commit_at, so suppressed echo still looks stranded. The watchdog
exists to rescue user speech an over-eager AEC ducked below the provider's
commit threshold, which is the one thing echo is not, sit now runs the same
check before rescuing.
coding_followup_after_reply 0/3 -> 3/3; ghost turns across the 78-run leak
suite 12 -> 2. Clean cells unmoved: flux/local, flux/lexical and nova/local
at 100%, flux/provider and nova/lexical each losing only the pre-existing
food_long_pause flake.
Retires eleven of the thirteen leak markers. support_barge_in_one_word keeps
one for a different reason: the STT merges the echo tail and the user's word
into a single commit ('retailers. Stop.'), so suppression and admission are
one decision over two utterances.
Two changes, both aimed at ElevenLabs' habit of inventing stock phrases on trailing silence, which cost the barge-in scenarios real turns. A hallucinated "Yeah." overwrote _latest_partial_text and refreshed the staleness anchor, so the user's actual words were both gone and un-rescuable — the same churn that stops the provider committing also disarms the watchdog built to catch that. A partial is now ignored for those two purposes when a stranded partial is already waiting, the new text is not that one refined, and Silero heard nothing recent. All three are required: silence alone would break the watchdog's founding case, where the user speaks quietly under playback and neither the provider VAD nor Silero registers it. The detector still sees the text; deciding whether to interrupt on it belongs to _vad_vetoes_barge_in. The harness had the larger share. settle_secs was a flat 1.0s against a shortest hold tier of 1.2s, so quiesnce could not distinguish a finished session from one deliberately debouncing, and runs were torn down mid-hold: 'Sorry, one more thing.' commits as a 1.5s lexical_hold and the run ends before it fires, scoring as the product losing a turn. Now derived from the detector's own hold timeouts, because getting it wrong is invisible and looks like someone else's bug — as it did here, where these failures were previously recorded as provider variance. Four EL barge-in scenarios 15/24 -> 24/24 with no ghost turns. Full suite on cells that could not previously be baselined: elevenlabs/local 61/62, elevenlabs/lexical 63/66, zero session errors. Deepgram unmoved, and the pause family is 16/16 under the longer settle.
…mits The echo suppressor was dropping genuine user speech. `food_long_pause` on elevenlabs/lexical went 4/4 -> 0/4: "Pepperoni pizza, please." after a reply of "Got it — a large pepperoni pizza." scores 0.67 against the resemblance check and gets suppressed, on a clean run where no echo exists. The threshold was calibrated only on barge-ins, where the user changes the subject; confirmations, disambiguation and digit readback put the user's next words into the reply by design, and there the two populations interleave (0.67/0.78 genuine against 0.76/0.79 echo). No threshold separates them. Split the check accordingly. A verbatim substring of what is playing is evidence and always counts. Garbled resemblance is a guess, and only runs once the same session has produced verbatim echo. That trades leak suppression back (leak 0.15: ~100% with 13 markers -> 85% with 2) because the latch rarely arms: leak severity determines transcription quality, so a leak quiet enough to be garbled is garbled consistently, while verbatim echo comes from a louder leak. The regimes are near-disjoint rather than sequential. Kept anyway because it changes which error you get — unconditional, a working AEC drops real speech; latched, a leaking AEC produces ghost turns. Timing is the actual fix. Also here: - suppress short commits with no Silero speech since the last commit, which is how ElevenLabs' "Yes."/"Yeah." on trailing silence became turns (6 of 9 EL failures) - baseline refusal now triggers only when a scenario fails every repeat, so a probabilistic axis can gate at all; at --repeat 2 five leak scenarios looked broken and three were intermittent at 4 - first baseline for deepgram-flux/local[leak=0.15]
The echo latch armed itself on clean runs and cost real turns. `food_long_pause` went 100% -> 0% on deepgram-flux/provider and deepgram-nova/lexical, with 'pepperoni pizza, please.' suppressed as echo while no leak was injected — which only the resemblance branch can do, so proof had arrived from somewhere. It came from the user's own partials: the STT emits `Pepperoni`, then `Pepperoni pizza`, and that is a verbatim substring of "Got it — a large pepperoni pizza." So the assumption in the previous commit was wrong. A verbatim substring *can* be produced by a user saying the same thing, and confirmation flows are where it happens, because the reply restates the request. Length does not separate them either: the echo this catches is 'memory access' at 13 chars, the false proof is 'pepperoni pizza' at 15. What separates them is the kind of event. A partial is text the STT is still revising; a commit is a claim it has settled Only commits arm the latch now. Suppressing an echo-shaped partial is unchanged and was never the problem. food_long_pause returns to 100% on both cells, and banking_correction and support_barge_in clear with it. Then the full matrix, 12 cells x 39 scenarios x 3 repeats: - zero ghost turns in 1386 runs, where they were routine this morning - all 12 cells baselined, up from 7 — and 2 of those 7 held pass=1.0 entries predating today, so they would have passed a gate they should have failed - the hold path priced for the first time: holdless detectors score 65-69% on Nova and ElevenLabs against 98-100% with a hold, failing one family (pause merging), marked per cell with _NO_HOLD_TO_MERGE Per cell rather than per detector because deepgram-flux/heuristic scores 90% on that same holdless detector — Flux holds the fragment provider-side — so a blanket rule would mark nine scenarios it passes and hide regressions in them.
…hold honestly "Send it to account 447. Sorry." commits with holding=False — both EOU signals score it complete and both are right, so no hold is armed and the correction becomes a second turn. What marks it incomplete is discourse rather than prosody or grammar: the marker announces a retraction. local now holds on a finished sentence plus a trailing marker, on the short tier, since the correction follows within ~300ms and a false positive shouldn't cost 3s. Requiring the sentence-then-marker shape is the safety margin — a bare "Sorry." is a whole turn, "I'm sorry." is an apology. banking_correction 6/6 on flux/local and medical_self_correction 6/6 on nova/local, both previously marked; two markers retired, three downgraded. Deepgram local cells gate at 100%, EL holds its 96.8% baseline, and the scenarios that dipped there never trigger the rule. Found re-sweeping vad_silence_threshold_secs under the fixed harness. 1.2 stays: looked two points behind at --repeat 3, but at 6 the gap is 94% vs 83%, and 0.8 scoring below both neighbours marks the first result as underpowered. The value isn't buying merges generally — it absorbs these corrections before any detector sees them, which is why the defect shows up on Flux and Nova instead.
provider has no hold, so on Flux eot_threshold *is* the turn policy — the only setting that can merge a pause there, in the pairing Flux exists for. We shipped Deepgram's default and measured only eot_timeout_ms. Swept over the pause family it is a clean monotonic curve, unlike ElevenLabs': 46% / 68% / 82% / 91% merges at 0.6 / 0.7 / 0.8 / 0.9, for 448 / 635 / 859 / 1944ms of dead air. 0.8 buys fourteen points for 214ms; 0.9 buys nine more for a second and is slower than ElevenLabs. All four Flux cells now gate at 100% with zero ghosts, re-baselined. The patience introduced a ghost mode: a stray 'I' committing its own turn after a finished one, because Flux holds the turn open long enough to transcribe the trailing audio and a holdless detector takes it. It slipped past _is_garbage_commit (alphanumeric) and _is_unvoiced_hallucination (the user really had just spoken); a one-word commit needs neither test. _is_contentless_single_token drops function words that cannot be an utterance, and not the one-word turns that carry intent — banking_short_reject is the single word "No." Also found: eot_threshold=0.4 is rejected with a 400 and the harness scored 78 session errors as a 0% row rather than reporting invalid config; banking_correction is 0% at every threshold, so that marker is structural; and Flux hears "no wait" as "no weight", which explains medical_self_correction there without reference to holds.
--stt-param eot_threshold=0.4 is outside Flux's accepted range, so every session died on the handshake and the sweep scored 78 refusals as a 0% row — indistinguishable from a setting that merely merges nothing. SWEEPABLE_STT_KEYS validates key names; encoding each provider's ranges beside them would go stale the first time one changed theirs, and would only catch the values we thought to guess. Noticing the refusal catches all of them. config_rejection matches a 4xx on the handshake, and only 4xx: that is deterministic for a given config, while a timeout or 5xx is the transient the repeats average over. The sweep stops scheduling the value, prints the provider's message in place of the row and "-" instead of "0%" per scenario, exits 1. cli.py reports the cell as refused before scoring, so it cannot be baselined or gated against, exits 2. Partial refusals still score the runs that connected.
Every number in the README came from studio-clean 16kHz TTS, which is the standing reason not to trust the endpointing values for real callers: both thresholds tuned in that document decide when speech has stopped, and a noise floor is what makes that judgement hard. degrade.py adds a noise bed (pink or white, levelled per scenario against its own speech) and the telephone band (300-3400Hz + G.711 mu-law, hand-rolled since audioop is gone in 3.13), applied in ScriptFeeder.stream in the order a real path applies: speech and echo, then the room, then the line. Noise runs through the pauses on purpose — noise that stops with the speaker tells the endpointer where the turn ended. 16 offline checks via --verify, and the codec costs 0.16ms of the 20ms frame budget. First run on deepgram-flux/provider against a clean control at matched concurrency: correctness holds up (27/27 -> 25/27, zero ghost turns at both 15dB and 5dB) but the latency tail doesot — dead air p95 1442 -> 2282ms with p50 eou->audio flat, which is Flux's end-of-turn confidence taking longer on a noisy signal. That makes eot_threshold=0.8, chosen on clean audio, an open question rather than a settled one. No markers or baseline for the axis yet: the leak axis earned its table only after a first run showed which failures recurred.
Two ways a session could go quiet forever, closed: - A hung LLM, tool, or TTS drain now hits a per-turn wall-clock deadline (60s default, turn_timeout_secs in voice_config, <=0 disables). On expiry the session emits SessionError, speaks a short apology if nothing has played yet, and stays open for the next turn. The apology is bounded too — when the TTS is the hung component, an unbounded rescue would hang exactly like the turn it rescues. - A dead STT socket already ends the call, but only because the receive loop enqueues its end-of-stream sentinel on every exit path; drop it and the caller keeps talking to a session that can no longer hear. New tests pin the sentinel for normal and abnormal closes, that only abnormal ones surface an error, and that audio pushed at a dead socket is dropped rather than raised.
A provider-initiated close with a normal code ended the session silently (Deepgram), while our own requested teardown could surface a spurious error (ElevenLabs). Both receive loops now gate on the _stop flag: requested closes are silent, unrequested ones — 1000/1001 included — surface an error before the end-of-stream sentinel. ElevenLabs audio sends also survive a dead socket now, instead of killing the periodic flusher and raising out of commit() into the VAD endpointing path. Dead-but-open TCP stays the websockets library's job (default ping keepalive, kept by both adapters) and flows through the same paths. The connection-loss suite runs the full matrix (2 providers x 4 close codes), and the hung-agent timeout test no longer races the retiring turn's finally block (1-in-5 flake, caught by the full-suite run).
… position POST /voice/rtc (aiortc, WHIP-style offer/answer): mic and TTS ride Opus tracks, session events ride the client's data channel, and the downlink is paced server-side in 20ms frames — so barge-in truncation is exact from the transport's own clock, no playback acks, and the unspoken tail is dropped before it's ever sent. - voice/webrtc.py: PcmQueueTrack, PacedPlaybackTracker, track_to_pcm - server/voice.py: session build + event serialization extracted and shared with the WS handler (which now advertises transport: websocket) - server/rtc.py: signaling, pc lifecycle, 501 without aiortc (now part of timbal[voice]), TIMBAL_STUN_URL / TIMBAL_TURN_* for ICE - playground: transport dropdown; WebRTC forces browser noise suppression (RNNoise lives in the worklet and would be silently bypassed) - tests: pacing/played-axis units + a real aiortc loopback through the route
…anscript One MP3 per session (mixed mono by default, split stereo optional) on the real call timeline: the mic is the clock, TTS drains against it, and a barge-in drops the unheard tail exactly like interrupted.heard_text does. Plus a JSON manifest (transcript offsets + TurnMetrics — the latency chips) and an async on_saved hook for platform upload. Server-config only; a client hello can never switch recording on or off. session_started gains session_id and session_transcript gains started_at/offset_ms so clients can join recordings to conversations. Co-authored-by: Cursor <cursoragent@cursor.com>
…onfirmed ingest
With TIMBAL_VOICE_RECORDING_UPLOAD=platform, both recording files are PUT to
orgs/{org}/projects/{project}/sessions/{session_id} as multipart after the
manifest lands, through the shared platform client. 2xx deletes the local
copies; 4xx keeps them and stops; 5xx/network retries on a 1s-base ×5 backoff
capped at 5 min for ~1h before giving up. The upload is a fire-and-forget task,
so a live session and its teardown are never blocked, and a crash mid-upload
leaves both files intact for the sweeper. Verified e2e against the dev platform
(fresh push, idempotent re-push, 400 id mismatch, cross-project 409/403).
Platform asks folded in: recording env (dir/layout/bitrate/upload) is read per
session, not at boot, so CRIU-restored boxes pick up injected env; platform
identity (org/project/env/app/rev) is stamped into manifest meta; the manifest
write is atomic (tmp + rename) so "json exists ⇒ mp3 complete" is race-free.
tform.utils._request grows PUT support plus overridable backoff and timeout.
PR SummaryLow Risk Overview Core pieces: Reviewed by Cursor Bugbot for commit b1ac080. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…commits Three harness bugs that made correct runs look broken (or the reverse). format_grid and cross_cell_flaky keyed on bare stt/detector while scorecards use the full cell label, so leak/sweep runs rendered as · and flaky reporting could merge distinct axis cells. NoGhostTurns still compared per script line while the gated ghost count used a script window — one attribution rule now. _settled treated any post-Say commit as done and could quiet-fallback while newer speech was still only a partial; waits now stay open on in-flight partials, and settle_secs also clears the STT's own endpointing delay.
…on disk asyncio.wait_for forked the agent turn into a child Task on 3.11, dropping set_run_context so metrics and barge-in truncation broke. Empty MP3 closes also unlinked the file on Windows; pad silence and pin heuristic in WS tests.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 014f446. Configure here.
pedrolivaresanchez
approved these changes
Jul 28, 2026
session.py was the module everything imported from, so metrics.py could not import VoiceSessionEvent without a cycle — session.py and realtime.py carried three function-local `from .metrics import ...` imports to dodge it. Move the leaves out: events.py (VoiceSessionEvent + subclasses, TranscriptEntry) and providers.py (SpeechToText/TextToSpeech/TTSStream, AudioInputConfig/AudioOutputConfig). Both cycles are gone and the local imports are now top-level. realtime.py no longer imports session.py at all, so the turn-based and speech-to-speech engines are siblings rather than one depending on the other, and adding an STT provider no longer pulls in the 2300-line engine. Public API is unchanged: timbal.voice re-exports every moved name. First-party and test imports move off timbal.voice.session, which was only keeping those names alive by accident. Also stamp run_id on TurnMetrics from the turn's RunContext, so a slow turn in a metrics view can joined back to its trace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.