Release additional cuda memory between audio inputs

[Khimer]

Hello! Thank you very much for your work. I encountered an issue with releasing GPU memory when using https://github.com/k2-fsa/sherpa-onnx/blob/master/sherpa-onnx/python/sherpa_onnx/offline_recognizer.py. While performing speech recognition, the model requests additional GPU memory. However, after recognizing a new audio input, this additional memory is not released (as it would be with gc.collect() or torch.cuda.empty_cache()) even after deleting the stream object. Instead, it remains occupied by the model object. As a result, I can only free up the resource by deleting the model object, which is inconvenient because I have to reload the model for each new recognition. Could you suggest a way to free up the additional GPU memory without deleting the model object?

[csukuangfj]

Is the memory usage increased if you start a new recognition? If not, then it is expected.

[Khimer]

Yes, that's correct, thank you for your response. So, there is no way to explicitly free the additional memory? I was hoping to use the model on another framework after recognition, but I no longer have enough memory for it.

[csukuangfj]

you need to save the model to somewhere, right?

you can free the model if you want to free all gpu memory.

[Khimer]

I have enough GPU memory to hold both models and process audio with one of them. I was hoping to recognize speech in the audio, free the additional memory without deleting the model, apply the second model, and then free the additional memory again. The time it takes to delete and reload the model is comparable to the recognition time, so I would like to avoid such a scenario.