Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Enhance Model Info Lookup for litellm_proxy Prefixed Models #6368

Open
openhands-agent opened this issue Jan 20, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@openhands-agent
Copy link
Contributor

What problem or use case are you trying to solve?
The current model info lookup using model_name gets confused with litellm_proxy prefixed models like litellm_proxy/bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0, which leads to incorrect model defaults and no prompt caching. There's a need to map this to the correct model name to load the proper 'Model info'.

Describe the UX of the solution the user wants
The user wants a straightforward way to look up model information without manually stripping prefixes or configuring model names.

User's thoughts on the technical implementation?
User noticed the lookup occurs in llm.py. A potential solution involves modifying to accommodate the litellm_proxy prefix, possibly by removing it before processing, as seen in user-specific code changes.

Describe the alternatives considered
Current workaround involves modifying llm.py to strip the prefix manually, but this isn't ideal for all scenarios.

Additional context

Issue Created By: Steve Cosenza on Slack

Link to Slack Thread: Slack Link

@mamoodi mamoodi added the enhancement New feature or request label Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants