Speaker labeling (recognition) in Rust

[thewh1teagle]

How can I use this library from Rust? I want to use the speaker recognition feature so I'll know who it talking and when.
As for the segmentaion, I'm thinking about using the word segments returning from whisper.cpp. is that good enough or it's bad to embed voice on each word?

[csukuangfj]

sorry, we don't support rust at present.

[csukuangfj]

At the end of the audio, could you pad some zeros, e.g., 0.5 seconds of zeros.

[thewh1teagle]

I added padding of 1 second in the load_audio but still not detected at all speech between 13s-20s
Maybe I didn't added them correctly, but also tried to add 1 seconds of silence with ffmpeg

Happens also with other files. after 10 seconds it doesn't detect speech

main.py

# wget https://github.com/snakers4/silero-vad/raw/master/files/silero_vad.onnx
# wget https://github.com/thewh1teagle/sherpa-rs/raw/main/samples/motivation.wav
# pip3 install soundfile numpy sherpa_onnx
# python3 main.py

from pathlib import Path
from typing import Tuple

import numpy as np
import sherpa_onnx
import soundfile as sf 


def load_audio(filename: str) -> Tuple[np.ndarray, int]:
    data, sample_rate = sf.read(
        filename,
        always_2d=True,
        dtype="float32",
    )
    data = data[:, 0]  # use only the first channel
    samples = np.ascontiguousarray(data)
    
    # Add 1 seconds of padding to the end of the samples
    padding_samples = int(sample_rate * 1)
    samples = np.concatenate((samples, np.zeros(padding_samples, dtype=samples.dtype)))
    
    return samples, sample_rate

def main():
    samples, sample_rate = load_audio("motivation.wav")
    config = sherpa_onnx.VadModelConfig()
    config.silero_vad.model = "silero_vad.onnx"
    config.sample_rate = sample_rate

    window_size = config.silero_vad.window_size

    vad = sherpa_onnx.VoiceActivityDetector(config, buffer_size_in_seconds=3)
    while len(samples) > window_size:
        vad.accept_waveform(samples[:window_size])
        samples = samples[window_size:]
        if vad.is_speech_detected():
            while not vad.empty():
                start_sec = vad.front.start / sample_rate
                duration_sec = len(vad.front.samples) / sample_rate
                print(f"start={start_sec}s duration={duration_sec}s")
                vad.pop()

main()

[csukuangfj]

Could you upload the test wave for reproducing?
Please also describe the expected behavior.

[thewh1teagle]

The test file available here samples/motivation.wav (16khz, mono)
When I run the attached script, it produce the following:

python3 test.py 
start=0.926s duration=1.678s
start=3.774s duration=2.414s
start=7.518s duration=1.998s

The audio length is 20s and there's loud and clear speech from 13s to 20s which is not detected.
expected behavior: it should find speech also in 13s to 20s

[thewh1teagle]

With original silero-vad it works correctly:

Result:

python3 main.py
[{'start': 0.738, 'end': 2.43}, {'start': 3.81, 'end': 6.174}, {'start': 7.554, 'end': 9.15}, {'start': 12.77, 'end': 20.0}]

As you can see it detects speech from 12s to 20s.

Code:

# wget https://github.com/thewh1teagle/sherpa-rs/raw/main/samples/motivation.wav
# pip install torch torchaudio
# python3 main.py

import torch 
torch.set_num_threads(1)

model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad')
(get_speech_timestamps, _, read_audio, _, _) = utils

wav = read_audio('samples/motivation.wav')
sample_rate = 16000

def convert_samples_to_seconds(timestamps, sample_rate):
    return [{'start': ts['start'] / sample_rate, 'end': ts['end'] / sample_rate} for ts in timestamps]

speech_timestamps = get_speech_timestamps(wav, model)
readable_timestamps = convert_samples_to_seconds(speech_timestamps, sample_rate)
print(readable_timestamps)

Should I open new issue in sherpa-onnx?

Also, may be related, debug output:

✗ cargo run --example diarize
   Compiling sherpa-rs v0.1.4-beta.1 (/Users/user/Documents/sherpa-rs)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.23s
     Running `target/debug/examples/diarize`
/Users/user/Documents/sherpa-rs/target/debug/build/sherpa-rs-sys-83bc881326eb63e6/out/sherpa-onnx/sherpa-onnx/c-api/c-api.cc:SherpaOnnxCreateVoiceActivityDetector:812 VadModelConfig(silero_vad=SilerVadModelConfig(model="silero_vad.onnx", threshold=0.5, min_silence_duration=0.4, min_speech_duration=0.4, window_size=512), sample_rate=16000, num_threads=1, provider="coreml", debug=True)
(speaker 0) start=0.936s duration=1.672s
(speaker 0) start=3.784s duration=2.408s
(speaker 0) start=7.528s duration=1.992s
/Users/user/Documents/sherpa-rs/target/debug/build/sherpa-rs-sys-83bc881326eb63e6/out/sherpa-onnx/sherpa-onnx/csrc/circular-buffer.cc:Push:96 Overflow! n: 512, size: 48000, n+size: 48512, capacity: 48000. Increase capacity to: 96000
/Users/user/Documents/sherpa-rs/target/debug/build/sherpa-rs-sys-83bc881326eb63e6/out/sherpa-onnx/sherpa-onnx/csrc/circular-buffer.cc:Push:96 Overflow! n: 512, size: 95616, n+size: 96128, capacity: 96000. Increase capacity to: 192000

Update:
I created new issue about it