We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
部署CodeFuse-CodeLlama-34B-4bits推理,推理速度太慢了,使用A100推理,显存占用大概21G,max_length为512,推理速度接近10min以上,这个框架下有相关加速推理方案吗?
The text was updated successfully, but these errors were encountered:
这个框架不做这个事情。在modelcache那的项目可以提一下。那边有计划做这些事。如果没有其它问题,这个问题将在6点之后进行关闭
Sorry, something went wrong.
No branches or pull requests
部署CodeFuse-CodeLlama-34B-4bits推理,推理速度太慢了,使用A100推理,显存占用大概21G,max_length为512,推理速度接近10min以上,这个框架下有相关加速推理方案吗?
The text was updated successfully, but these errors were encountered: