🚀 Building an OpenAI-Compatible API with Open-Source LLM: Rate-Limiting, Custom API Keys 🔐, and Streamlit Authentication 🌐

Welcome to the OpenAI-Compatible API with Open-Source LLMs repository! This project provides a comprehensive guide and implementation for building your own API service using open-source Large Language Models (LLMs). The API is designed to be compatible with the OpenAI API, and includes features such as rate limiting, custom API keys for authentication, and a Streamlit-based authentication system using Azure email communication.

---

🚀 Key Features

• OpenAI-Compatible API: Fully aligned with OpenAI’s chat/completions route for seamless integration.
• Secure API Key Management: Use the Streamlit app to create, manage, and delete API keys.
• Rate Limiting: Prevents abuse and ensures fair usage through a custom rate-limiting system.
• Azure Email Authentication: Implements a simple email-based authentication using Azure Communication Services for sending OTPs.
• FastAPI for LLM Serving: Serve open-source LLMs using FastAPI, with model serving by vLLM for efficient, paged attention mechanism.

---

📂 File Structure

• .env: Stores environment variables.
• api_key_backend.py: Manages API key creation, deletion, and database integration.
• app.py: Streamlit app for user login, logout, and API key management.
• auths.py: Implements email-based authentication, OTP validation, and session management.
• configs.py: Configuration settings for the application, including database and API limits.
• requirements.txt: Python dependencies required to run the project.
• serving.py: FastAPI code to serve the LLM with support for paged attention mechanism.
• testing_api.py: To ensure the API functions as expected and compatible with OpenAI client.

---

🔧 Setup Instructions

Prerequisites

• Python 3.9+
• Azure Communication Service credentials (for email OTP system)
• Virtual environment

Installation Steps

1. Clone the Repository:

   git clone https://github.com/ritun16/openai-compatible-fastapi.git
   cd openai-compatible-fastapi

2. Create Virtual Environment:

   python -m venv venv
   source venv/bin/activate

3. Install Dependencies:

   pip install -r requirements.txt

4. Set Up Environment Variables: Create a .env file in the root directory with the following keys

   AZURE_EMAIL_COMM_CONN_STR=<connection_string>
   AZURE_EMAIL_SENDER_ADD=<sender_email_address>
   NGROK_AUTH_TOKEN=<ngrok_auth_token> # Optional

5. Run the Streamlit App: Start the authentication and API key management app

   streamlit run app.py

6. Run the FastAPI Server: Start the LLM-serving API.

   uvicorn serving:app --reload --port 8000