Method for Faster Inference

[fakerybakery]

Hi, might it be possible to merge this repository upstream? It's slightly faster than the original version. Thanks!

[jhowilbur]

@fakerybakery, can you help me understand the main point of making this other repo faster?
I tried to find it here in the commits, but I need help finding out why.

[fakerybakery]

Hi @jhowilbur
In the API repo, the authors mentioned that in their fork they made it 30% faster:

We have made some improvements to Tortoise to make the inference ~30% faster, and welcome contributions on our repo to improve it further!

Source

The commit messages say they made it faster by not moving the models between CPU and GPU.

Hope this helps!

[fakerybakery]

Hi @manmay-nakhashi, has this been resolved?

[manmay-nakhashi]

Yes we don't do offloading to cpu in api_fast.py

[fakerybakery]

The api_fast.py can't work on MPS (#658) since the fast API requires autocast and autocast isn't yet supported on MPS.

Yes we don't do offloading to cpu in api_fast.py

Does this mean support for the regular API is not planned @manmay-nakhashi?