Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add LoRA Support for Chat Completion in SGLang #1936

Open
2 tasks done
mssongit opened this issue Nov 6, 2024 · 0 comments
Open
2 tasks done

[Feature] Add LoRA Support for Chat Completion in SGLang #1936

mssongit opened this issue Nov 6, 2024 · 0 comments

Comments

@mssongit
Copy link
Contributor

mssongit commented Nov 6, 2024

Checklist

Motivation

Currently, SGLang supports LoRA via the generate API, but it is not accessible when using the OpenAI Chat Completion API. Adding LoRA support to the OpenAI Chat Completion API would enable users to leverage LoRA functionality consistently across both APIs, providing a more seamless experience.
https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/openai_api/protocol.py#L257

Related resources

vLLM LoRA Support for v1 Chat Completion: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/serving_completion.py#L38

vLLM currently provides LoRA support for v1 Chat Completion, demonstrating how LoRA can be integrated to improve training efficiency in language models. Examining vLLM's implementation may provide insights or code references that could assist in adding this feature to SGLang.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant