Not a Real-time model？

[jiayukk]

Based on my analysis of the Vocos model code, I believe your model is also non-causal.

[ysharma3501]

Yes it’s not casual right now, streaming is probably future work. It should already be able to do “fake” streaming by inputting let’s say 500ms chunks and it will probably take maybe 20ms or less on a cpu.

[jiayukk]

Do you want to train a causal version of the model? I've modified convolution padding, and now it has a causal version. Maybe we can try the effect.

[ASchneiderBR]

Great project, congratulations! This would be awesome as a real-time VST filter so we can use it in many applications, like OBS, EqualizerAPO and so on. Thanks.

[ysharma3501]

Thanks @ASchneiderBR and yes @jiayukk it would be interesting for sure. Do you have the code? Would be happy to train the model.

[jiayukk]

You can replace the models.py file in your installed vocos package with https://github.com/jiayukk/LavaSR/blob/main/LavaSR/enhancer/causal_vocos.py.

[jiayukk]

I think it would be better to design a single, unified model capable of both SE and BWE, rather than using separate SE and BWE models simultaneously. You might draw inspiration from Microsoft's SSI Challenge. Moving towards more unified models is definitely a major trend in current speech modeling.

[ysharma3501]

Thanks will check it out. I did test out training LavaSR with some denoising capability too, seemed to work decently actually.

In future work I will definitely try to make it work with much more variety of tasks in a single model(bwe, denoise, dereverb, seperation, etc.) with different conditioning only.