Open Server is my attempt to recreate an OpenAI Compatible Server for generating text, images, embeddings, and storing them in vector databases. It also includes a chat functionality.
The server's request and responses are very similar to OpenAI's API with additional fields needed for different providers. It uses Langchain for the LLM part (Robust and powerful with callbacks) and provider SDKs for image generation and more.
- 9 December 2023 : Added OpenRouter and NeuroAPI to providers as config based on OpenAI provider. Added cost calculation to config and routes in usage.
- Python SDK
- Docker Compose
- ( Soon ) Example Next.JS Front working with Open Server
- Monitoring for LLM generation (LLM Monitor & Prompt Layer)
- ( Soon ) Audio Translation & Generation & Transcription APIs (Local with Whisper, ElevenLabs)
To install the required packages:
pip install -r requirements.txt
To run the server:
python -m openserver.main
Note: For GPU support, you need to manually install pytorch and llama-cpp-python, based on your GPU type (CUDA & ROCM).
This section lists the key features implemented in Open Server:
-
- Streaming
- Function Calling
-
- Text-to-Image
This file is used to store API keys, URLs, and other similar information. It has a YAML structure and can be used to configure various aspects of the server.
Example config.yaml:
OPENAI_API_KEY: YOUR_OPEN_API_KEY
PALM_API_KEY: YOUR_PALM_API_KEY
DB_NAME: test
DB_HOST: localhost
DB_USERNAME: admin
DB_PASSWORD: admin
# Add more configuration options as needed...These configurations are stored in separate files for better organization and modularity. Each config file follows the YAML structure.
Example LLM Config (llm_config.yaml):
chat_providers:
palm:
name: palm
models:
- "models/text-bison-001"
available: true
# Add more LLM configs as needed...Example Image Config (image_config.yaml):
image_models:
novita:
name: novita
models:
- "dynavisionXLAllInOneStylized_release0534bakedvae_129001.safetensors"
available: true
api_key: true
api_key_name: NOVITA_API_KEY
# Add more image configs as needed...Example Prompt Config (prompts_config.yaml):
prompts:
function_call:
name: "function_call"
file: "/prompts/function_call.txt"
# Add more prompt configs as needed...Example Vector Database Config (vectordb_config.yaml):
embeddings:
cohere:
name: cohere
models:
- "embed-english-light-v2.0"
- "embed-english-v2.0"
available: true
api_key: true
vectordbs:
chromadb:
available: true
# Add more vector database configs as needed...Feel free to modify and extend these configurations according to your specific needs.
| Provider | Completion | Chat | Function Calling | Streaming | ||
|---|---|---|---|---|---|---|
| ------------- | ------------- | ------------- | ------------- | ------------- | ||
| openai | ✅ | ✅ | ✅ | ✅ | ||
| cohere | ✅ | ✅ | ✅ | |||
| huggingface | ✅ | ✅ | ||||
| together-ai | ✅ | ✅ | ✅* | ✅ | ||
| google-palm | ✅ | ✅ | ✅ | ✅ | ||
| ai21 | ✅ | ✅ | ✅ | |||
| fireworks | ✅ | ✅ | ✅* | ✅ | ||
| llama-cpp-python | ✅ | ✅ | ✅ |
- Some models of provider support Function Calling.
| Provider | Txt2Img | Img2Img | Upscale | ||
|---|---|---|---|---|---|
| ------------- | ------------- | ------------- | ------------- | ||
| openai | ✅ | ✅ | |||
| together-ai | ✅ | ||||
| novita | ✅ | ✅ | ✅ | ||
| segmind | ✅ | ✅ |
palm, huggingface, openai, gradient, cohere
chromadb, lancedb, milvus, pinecone, qdrant,redis, weaviate
To contribute: Clone the repo locally -> Make a change -> Submit a PR with the change.
Here's how to modify the repo locally: Step 1: Clone the repo
git clone https://github.com/mj23978/openserver.git
Step 2: Navigate into the project, and install dependencies:
cd openserver
pip install -r requirements.txt
Step 3: Submit a PR with your changes! 🚀
- push your fork to your GitHub repo
- submit a PR from there