Skip to content

VAD-SILERO-INTEGRATION.md

Latest commit

 

History

History
151 lines (101 loc) · 5.33 KB

VAD-SILERO-INTEGRATION.md

File metadata and controls

151 lines (101 loc) · 5.33 KB

Silero VAD Integration Results -- ONNX Runtime Integration

Raw ONNX Runtime integration of HuggingFace Silero VAD (2.1MB). Key finding: TV dialogue IS speech -- VAD detecting it is correct. The real problem requires speaker diarization, not better VAD.

Parent: Live

Implementation Status: WORKING

Successfully integrated Silero VAD using raw ONNX Runtime, bypassing the incompatible silero-vad-rs crate.

Model Details

Source: HuggingFace onnx-community/silero-vad URL: https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx Size: 2.1 MB (ONNX) Location: workers/streaming-core/models/vad/silero_vad.onnx

Model Interface (HuggingFace variant)

Inputs:

  • input: Audio samples (1 x num_samples) float32, normalized [-1, 1]
  • state: LSTM state (2 x 1 x 128) float32, zeros for first frame
  • sr: Sample rate scalar (16000) int64

Outputs:

  • output: Speech probability (1 x 1) float32, range [0, 1]
  • stateN: Next LSTM state (2 x 1 x 128) float32

Key difference from original Silero: The HuggingFace model combines h and c LSTM states into a single state tensor.

Test Results with Synthetic Audio

Accuracy: 42.9% (3/7 correct)

Test Case Detected Confidence Expected Result
Silence ✓ Noise 0.044 Noise ✓ PASS
White Noise ✓ Noise 0.025 Noise ✓ PASS
Clean Speech ✗ Noise 0.188 Speech ✗ FAIL
Factory Floor ✓ Noise 0.038 Noise ✓ PASS
TV Dialogue ✗ Speech 0.921 Noise ✗ FAIL
Music ✗ Speech 0.779 Noise ✗ FAIL
Crowd Noise ✗ Speech 0.855 Noise ✗ FAIL

Critical Insights

1. Sine Wave "Speech" is Too Primitive

Problem: Our synthesized "clean speech" using sine waves (200Hz fundamental + 400Hz harmonic) is too simplistic for ML-based VAD.

Evidence: Silero confidence on sine wave "speech" = 0.188 (below threshold)

Conclusion: ML models trained on real human speech don't recognize pure sine waves as speech.

2. TV Dialogue Detection is Actually CORRECT

The Core Realization: TV dialogue DOES contain speech - just not the user's speech.

When the user said "my TV is being transcribed", the VAD is working correctly by detecting speech in TV audio. The issue isn't VAD accuracy - it's source disambiguation:

  • What VAD does: Detect if ANY speech is present ✓
  • What's needed: Detect if the USER is speaking (not TV/other people)

3. The Real Problem Requires Different Solutions

VAD alone cannot solve "my TV is being transcribed" because TV audio DOES contain speech.

Solutions needed:

  1. Speaker Diarization: Identify WHO is speaking (user vs TV character)
  2. Directional Audio: Detect WHERE sound comes from (microphone vs speakers)
  3. Proximity Detection: Measure distance to speaker
  4. Active Noise Cancellation: Filter out TV audio using echo cancellation
  5. Push-to-Talk: Only record when user explicitly activates microphone

Performance

Latency: ~0.38s for 7 test cases = ~54ms per inference (512 samples @ 16kHz = 32ms audio) Overhead: ~22ms processing time per frame (68% real-time overhead)

Comparison:

  • RMS VAD: 5μs per frame (6400x real-time)
  • Silero VAD: 54ms per frame (1.7x real-time)

Silero is 10,800x slower than RMS, but provides ML-based accuracy.

Next Steps

Immediate: Better Test Audio

Current: Sine wave synthesis (too primitive) Needed: Real speech or TTS-generated audio

Options:

  1. Use Kokoro TTS to generate test speech samples
  2. Record real audio samples with known ground truth
  3. Use public speech datasets (LibriSpeech, Common Voice)

Medium-term: Source Disambiguation

For the user's original problem (TV transcription):

  1. Echo Cancellation: Use WebRTC AEC to filter TV audio
  2. Directional VAD: Combine VAD with beamforming/spatial audio
  3. Speaker Enrollment: Train on user's voice, reject others
  4. Multi-modal: Combine audio VAD with webcam motion detection

Long-term: Comprehensive VAD System

  1. Multiple VAD implementations (Silero, WebRTC, Yamnet)
  2. Ensemble voting for higher accuracy
  3. Adaptive threshold based on environment
  4. Continuous learning from user corrections

Code Location

Implementation: workers/streaming-core/src/vad/silero_raw.rs (225 lines) Tests: workers/streaming-core/tests/vad_background_noise.rs Factory: workers/streaming-core/src/vad/mod.rs

Dependencies

ort = { workspace = true }  # ONNX Runtime
ndarray = "0.16"            # N-dimensional arrays
num_cpus = "1.16"           # Thread count detection

Usage

use streaming_core::vad::{SileroRawVAD, VoiceActivityDetection};

let vad = SileroRawVAD::new();
vad.initialize().await?;

let audio_samples: Vec<i16> = /* 512 samples @ 16kHz */;
let result = vad.detect(&audio_samples).await?;

if result.is_speech {
    println!("Speech detected! Confidence: {:.3}", result.confidence);
}

Conclusion

✅ Silero VAD integration successful ⚠️ Sine wave tests inadequate - need real audio or TTS 🎯 Key insight: VAD detecting TV speech is CORRECT behavior 🔧 Next: Build better test suite with TTS or real audio samples 🚀 Future: Solve "TV transcription" with speaker diarization/echo cancellation