Bug or a feature in streaming_utils.py BatchedFrameASRTDT? Is it possible to run parakeet-tdt-0.6b-v2 in streaming mode?

[SatenHarutyunyan]

In this huggingface discussion https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/discussions/3
It is said that its possible to use parakeet tdt in streaming mode and an inference bug was detected and corrected. I run
this script but get empty strings as prediction.

# Evaluate Parakeet in streaming mode python examples/asr/asr_chunked_inference/rnnt/speech_to_text_buffered_infer_rnnt.py \ model_path="parakeet_downlaoded/parakeet-tdt-0.6b-v2.nemo" \ audio_dir="audio_dir" \ output_filename="output.json" \ chunk_len_in_secs=0.2 \ total_buffer_in_secs=0.8 \ model_stride=4 \ batch_size=1

Pulled from main, commit=259d684e73c45091f0b6144342133e6ceb7e824c installed with pip install '.[all]'

[deepanshu-yadav]

It is difficult to do so. But possible. Someone has converted to lightweight integer models https://github.com/k2-fsa/sherpa-onnx/tree/master/scripts/nemo/parakeet-tdt-0.6b-v2 . I made an example https://github.com/deepanshu-yadav/voice-form-filler on how to wrap this around a server and send audio packets and return the final inference. It works okayish on my laptop without any GPU.

[SatenHarutyunyan]

There is a bug in the script
#13797
Thats why its not working. Will wait for the fix.

[csukuangfj]

Is it possible to run parakeet-tdt-0.6b-v2 in streaming mode?

Definitely possible, without any GPUs, on your phones.

You can find a pre-built Android APK for this model at
https://hf-mirror.com/csukuangfj/sherpa-onnx-apk/blob/main/vad-asr-simulated-streaming/1.12.6/sherpa-onnx-1.12.6-arm64-v8a-simulated_streaming_asr-en-parakeet_tdt_0.6b_v2.apk

More APKs are available at
https://k2-fsa.github.io/sherpa/onnx/android/apk-simulate-streaming-asr.html

Everything is open-sourced at
https://github.com/k2-fsa/sherpa-onnx

See also
https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/nemo-transducer-models.html#sherpa-onnx-nemo-parakeet-tdt-0-6b-v2-int8-english