-
Notifications
You must be signed in to change notification settings - Fork 2.1k
feat(mcp): add openai_generic provider for LiteLLM/Ollama compatibility #1120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lehcode
wants to merge
4
commits into
getzep:main
Choose a base branch
from
lehcode:feat/mcp-openai-generic-provider
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,970
−1,649
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
danielchalef
added a commit
that referenced
this pull request
Dec 22, 2025
Author
|
Added auto-fallback for providers without json_schema support: e680665 |
Add support for `openai_generic` LLM provider in the MCP server factory.
This provider uses `OpenAIGenericClient` which calls `/chat/completions`
with `response_format` for structured output instead of the `/responses`
endpoint. This enables compatibility with:
- LiteLLM proxy
- Ollama
- vLLM
- Any OpenAI-compatible API that doesn't support `/responses`
The `/responses` endpoint is only available on OpenAI's native API, so
this provider is essential for self-hosted LLM deployments.
Usage in config.yaml:
```yaml
llm:
provider: "openai_generic"
model: "your-model"
providers:
openai:
api_key: ${OPENAI_API_KEY}
api_url: ${OPENAI_BASE_URL}
```
Detect when providers (e.g., LiteLLM with Gemini) return schema definition instead of data and automatically switch to json_object mode with schema embedded in prompt. - Add _is_schema_returned_as_data() detection helper - Add instance-level _use_json_object_mode fallback state - Modify _generate_response() to support dual modes - Fallback persists for client lifetime after first trigger
- Add _extract_json() method to handle responses with trailing content - Simplify _is_json_schema() detection logic - Handle "Extra data" JSON parse errors gracefully
- Document openai_generic provider in README.md with LiteLLM and Ollama examples - Add provider configuration to .env.example - Add unit tests for _is_schema_returned_as_data() and _extract_json() methods
453e8c0 to
45bfe80
Compare
FlibbertyGibbitz
pushed a commit
to RuneLabs-ai/graphiti
that referenced
this pull request
Jan 5, 2026
Adds configuration files for running Graphiti with LiteLLM proxy: - docker-compose-local.yml: Isolated Neo4j (ports 7475/7688) + MCP server (8001) - config-local.yaml: Uses openai_generic provider with gpt-oss-120b model - .env.local.example: Template for LiteLLM credentials This setup routes LLM calls through LiteLLM for unified logging and local model support, while embeddings go to OpenAI via LiteLLM. Requires PR getzep#1120 openai_generic provider to be merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add support for
openai_genericLLM provider in the MCP server factory.This provider uses
OpenAIGenericClientwhich calls/chat/completionswithresponse_formatfor structured output instead of the/responsesendpoint.Problem
The default
openaiprovider usesOpenAIClientwhich calls OpenAI's/responsesendpoint. This endpoint is only available on OpenAI's native API and fails with:When using these backends, users get errors like:
Solution
Add
openai_genericcase toLLMClientFactorythat:OpenAIGenericClient(which already exists in graphiti-core)api_key,api_url)/chat/completionsendpoint withresponse_formatfor structured outputjson_schematojson_objectfor providers that don't support strict schemasUsage
Type of Change
Objective
Enable Graphiti to work with OpenAI-compatible APIs like LiteLLM proxy and Ollama. Some providers (e.g., Gemini via LiteLLM) don't support
json_schemaresponse format and return the schema definition instead of data. This PR adds automatic detection and fallback tojson_objectmode, plus robust JSON extraction for providers that append explanatory text after JSON output.Testing
_is_schema_returned_as_data,_extract_json)/chat/completionsendpoint is called (not/responses)add_memoryand entity extraction workingBreaking Changes
Checklist
make lintpasses)Related Issues
N/A