Faster speech to text using JAX

[Mage-Enderman]

https://github.com/sanchit-gandhi/whisper-jax

Found out about it from this video
https://www.youtube.com/watch?v=E6-V86n61Qg

[I5UCC]

Hey!

At first, thanks for caring about this project enough, that you want to make it better, I appreciate it!

So, a few things about this. This project is currently running whisper on the CTranslate2 inference engine, aka faster-whisper.

I am aware that whisper-jax exists, yet for the edge transcription problem aka real time transcription, at least according to my testing and some others, there is no improvement in performance, while using more memory than faster-whisper does.

whisper-jax does not implement the same transcription logic as openai/whisper, so the transcription quality is probably reduced and the output less consistent. The batching strategy that whisper-jax is using can make it much faster for longer files, yet faster-whisper is faster and more efficient for shorter files, which is a much more relevant metric for the problem this project addresses.

Unless running whisper-jax on a TPU, there isn't anything beneficial to it for this project in particular. Not that anyone would run a dedicated TPU in their system in the first place.

[Mage-Enderman]

Good to know!
Thanks for the speedy reply and for making such a useful application