diff --git a/AGENT.md b/AGENT.md index e6440ab..ca50a0c 100644 --- a/AGENT.md +++ b/AGENT.md @@ -1,5 +1,128 @@ # Agent Video Avatar - Session Notes +--- + +## ⚠️ IMPORTANT: Configuration Translation Guide for AI Assistants + +### Profile-Based Variable Naming + +When users provide environment variables, they are often providing the **base variable names** without the profile prefix. The backend uses a profile-based system where all variables need a `_` prefix. + +**Example: User provides MLLM config for VOICE profile** + +```bash +# ❌ DO NOT use these directly - they need profile prefix +MLLM_LOCATION=us-central1 +MLLM_VENDOR=vertexai +ENABLE_MLLM=true +APP_ID=20b7c51... +``` + +**✅ CORRECT translation to .env (with VOICE\_ prefix):** + +```bash +VOICE_MLLM_LOCATION=us-central1 +VOICE_MLLM_VENDOR=vertexai +VOICE_ENABLE_MLLM=true +VOICE_APP_ID=20b7c51... +``` + +### Critical Variable Names + +**⚠️ LOCATION vs REGION:** + +- Backend expects: `MLLM_LOCATION` +- NOT: `MLLM_REGION` + +If user provides `MLLM_LOCATION=us-central1`, translate to `VOICE_MLLM_LOCATION=us-central1` (DO NOT change LOCATION to REGION!) + +### Variable Naming Pattern + +Profile variables follow: `_` format + +```bash +# ✅ CORRECT +VOICE_MLLM_VENDOR=vertexai +VOICE_MLLM_MODEL=gemini-live-2.5-flash-preview-native-audio-09-2025 + +# ❌ WRONG (double MLLM) +VOICE_MLLM_MLLM_VENDOR=vertexai +``` + +### Debugging Agent Creation Failures + +**Symptom:** RTM error `-11033: user offline` + +**Root cause:** Agent failed to create (400 error from Agora API) + +**How to debug:** + +1. Check backend logs for `Response status: 400` +2. View most recent curl dump: `ls -lt /tmp/agora_curl_*.sh | head -1` +3. Look for `"location": null` in mllm params (should be `"location": "us-central1"`) +4. Verify `"enable_mllm": true` in advanced_features + +**Common causes:** + +- Missing or null `location` field in MLLM config +- Invalid GCP credentials +- Wrong model name or region + +### Required MLLM Variables for Gemini Live + +When translating user config for VOICE profile: + +```bash +VOICE_ENABLE_MLLM=true +VOICE_MLLM_VENDOR=vertexai +VOICE_MLLM_MODEL=gemini-live-2.5-flash-preview-native-audio-09-2025 +VOICE_MLLM_ADC_CREDENTIALS_STRING={...GCP service account JSON...} +VOICE_MLLM_PROJECT_ID=your-gcp-project-id +VOICE_MLLM_LOCATION=us-central1 # NOT REGION! +VOICE_MLLM_VOICE=Charon +VOICE_MLLM_TRANSCRIBE_AGENT=true +VOICE_MLLM_TRANSCRIBE_USER=true +VOICE_ASR_VENDOR=ares +VOICE_ASR_LANGUAGE=en-US +VOICE_VAD_SILENCE_DURATION_MS=300 +VOICE_ENABLE_AIVAD=true +``` + +--- + +## Companion Servers (Optional) + +These standalone servers extend simple-backend with advanced capabilities. They are **not required** for basic operation. + +- **[server-custom-llm](https://github.com/AgoraIO-Community/server-custom-llm)** — Custom LLM proxy. Intercepts LLM requests for RAG, custom prompts, tool calling, and response formatting. Set `LLM_URL` to your server endpoint, `LLM_VENDOR=custom`. +- **[server-mcp-memory](https://github.com/AgoraIO-Community/server-mcp-memory)** — MCP Memory Server. Gives agents persistent per-user memory via tool calling. Configure via `MCP_SERVERS` JSON array in `.env`. + +### Port Reference + +| Server | Language | Port | +| ---------- | -------- | ---- | +| MCP Memory | Python | 8090 | +| MCP Memory | Node.js | 8091 | +| MCP Memory | Go | 8092 | +| Custom LLM | Python | 8100 | +| Custom LLM | Node.js | 8101 | +| Custom LLM | Go | 8102 | + +### LLM Config Fields Reference + +Fields supported in the LLM config block sent to Agora ConvoAI API: + +| Field | Description | +| ------------------ | ----------------------------------------------------------------- | +| `url` | LLM endpoint URL | +| `api_key` | API key for the LLM provider | +| `style` | Protocol style: `openai` (default), `gemini`, `anthropic`, `dify` | +| `vendor` | `custom` (adds turn_id + timestamp), `azure` (Azure OpenAI) | +| `greeting_configs` | Greeting behavior, e.g. `{"mode": "single_first"}` | +| `mcp_servers` | Array of MCP server configs for tool calling | + +--- + ## Current Status (2026-01-20) ### ✅ WORKING @@ -405,3 +528,241 @@ Added comprehensive logging to track message flow: - Result (true/false) **Usage:** Filter browser console for `💬 CHAT_DEBUG` to track message flow from RTM → state → render + +--- + +## Production Deployment (EC2 + nginx on port 443) + +This section documents how to serve all agent-samples behind nginx on port 443 alongside an existing application, using path-based routing. + +### Architecture + +``` +nginx :443 (convoai-demo.agora.io) + / → /var/www/palabra/ (existing SPA) + /v1/, /query, /oauth, /pstn → localhost:7080 (existing API) + /simple-backend/ → localhost:8081 (Flask API, prefix stripped) + /react-voice-client/ → localhost:8083 (Next.js voice client) + /react-video-client-avatar/ → localhost:8084 (Next.js video+avatar client) + /simple-voice-client-no-backend/ → static files via alias + /simple-voice-client-with-backend/ → static files via alias +``` + +### Source Code Changes (4 lines, backward-compatible) + +Two env-var-driven configs were added. When env vars are **not set** (local dev), behavior is identical to the original code. + +**`next.config.ts`** (both `react-voice-client` and `react-video-client-avatar`): + +```typescript +const nextConfig: NextConfig = { + basePath: process.env.NEXT_PUBLIC_BASE_PATH || "", + typescript: { ignoreBuildErrors: true }, + transpilePackages: ["@agora/conversational-ai", "@agora/agent-ui-kit"], +}; +``` + +- `basePath` makes Next.js serve all routes/assets under the specified prefix +- `typescript.ignoreBuildErrors` bypasses an unused `@ts-expect-error` in `@agora/agent-ui-kit` + +**`VoiceClient.tsx` / `VideoAvatarClient.tsx`**: + +```typescript +const DEFAULT_BACKEND_URL = + process.env.NEXT_PUBLIC_BACKEND_URL || "http://localhost:8082"; +``` + +- When `NEXT_PUBLIC_BACKEND_URL=/simple-backend` is set at build time, the browser makes relative requests to the same origin +- When not set, defaults to `http://localhost:8082` for local dev + +### Step 1: Python Backend Setup + +```bash +cd simple-backend +python3 -m venv venv +source venv/bin/activate +pip install -r requirements-local.txt +cp .env.example .env +# Edit .env with real Agora, LLM, TTS, and avatar credentials +``` + +Create `simple-backend/start.sh` (PM2 workaround for Python): + +```bash +#!/bin/bash +cd /home/ubuntu/agent-samples/simple-backend +source venv/bin/activate +PORT=8081 exec python3 local_server.py +``` + +```bash +chmod +x start.sh +``` + +### Step 2: Build Next.js Apps + +```bash +# Voice client +cd react-voice-client +npm install --legacy-peer-deps +NEXT_PUBLIC_BASE_PATH=/react-voice-client NEXT_PUBLIC_BACKEND_URL=/simple-backend npm run build + +# Video avatar client +cd ../react-video-client-avatar +npm install --legacy-peer-deps +NEXT_PUBLIC_BASE_PATH=/react-video-client-avatar NEXT_PUBLIC_BACKEND_URL=/simple-backend npm run build +``` + +### Step 3: PM2 Ecosystem Config + +Create `ecosystem.config.js` in the repo root: + +```javascript +module.exports = { + apps: [ + { + name: "simple-backend", + script: "/home/ubuntu/agent-samples/simple-backend/start.sh", + interpreter: "bash", + watch: false, + max_memory_restart: "200M", + }, + { + name: "react-voice-client", + cwd: "/home/ubuntu/agent-samples/react-voice-client", + script: "node_modules/.bin/next", + args: "start -p 8083", + env: { + NODE_ENV: "production", + PORT: 8083, + NEXT_PUBLIC_BASE_PATH: "/react-voice-client", + NEXT_PUBLIC_BACKEND_URL: "/simple-backend", + }, + watch: false, + max_memory_restart: "500M", + }, + { + name: "react-video-client-avatar", + cwd: "/home/ubuntu/agent-samples/react-video-client-avatar", + script: "node_modules/.bin/next", + args: "start -p 8084", + env: { + NODE_ENV: "production", + PORT: 8084, + NEXT_PUBLIC_BASE_PATH: "/react-video-client-avatar", + NEXT_PUBLIC_BACKEND_URL: "/simple-backend", + }, + watch: false, + max_memory_restart: "500M", + }, + ], +}; +``` + +**Critical:** The `NEXT_PUBLIC_BASE_PATH` env var must be set at **both** build time and runtime. `next start` re-reads `next.config.ts` at startup, so the PM2 env must match the build env. Without this, basePath evaluates to `""` at runtime and all pages return 404. + +```bash +pm2 start ecosystem.config.js +pm2 save +``` + +### Step 4: Nginx Configuration + +Add these location blocks **before** the catch-all `location /` block: + +```nginx + # --- Agent Samples --- + + # Flask backend (strip /simple-backend prefix via trailing slash on proxy_pass) + location /simple-backend/ { + proxy_pass http://localhost:8081/; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 300s; + } + + # Next.js voice client (^~ prevents regex cache block from stealing .js/.css) + location ^~ /react-voice-client { + proxy_pass http://localhost:8083; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + + # Next.js video avatar client (^~ prevents regex cache block from stealing .js/.css) + location ^~ /react-video-client-avatar { + proxy_pass http://localhost:8084; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + + # Static HTML clients + location ^~ /simple-voice-client-no-backend/ { + alias /home/ubuntu/agent-samples/simple-voice-client-no-backend/; + index index.html; + } + + location ^~ /simple-voice-client-with-backend/ { + alias /home/ubuntu/agent-samples/simple-voice-client-with-backend/; + index index.html; + } +``` + +```bash +sudo nginx -t && sudo systemctl reload nginx +``` + +### Step 5: File Permissions + +```bash +chmod o+x /home/ubuntu /home/ubuntu/agent-samples +chmod -R o+r /home/ubuntu/agent-samples/simple-voice-client-no-backend/ +chmod -R o+r /home/ubuntu/agent-samples/simple-voice-client-with-backend/ +``` + +### Verification + +```bash +curl -s https://convoai-demo.agora.io/simple-backend/health +# {"service":"agora-convoai-backend","status":"ok"} + +curl -s -o /dev/null -w "%{http_code}" https://convoai-demo.agora.io/react-voice-client +# 200 + +curl -s -o /dev/null -w "%{http_code}" https://convoai-demo.agora.io/react-video-client-avatar +# 200 + +curl -s -o /dev/null -w "%{http_code}" https://convoai-demo.agora.io/simple-voice-client-no-backend/ +# 200 + +curl -s -o /dev/null -w "%{http_code}" https://convoai-demo.agora.io/simple-voice-client-with-backend/ +# 200 + +# Verify static assets load (not intercepted by palabra cache block) +curl -s -o /dev/null -w "%{http_code}" https://convoai-demo.agora.io/react-voice-client/_next/static/chunks/*.css +# 200 +``` + +### Key Gotchas + +1. **`^~` on proxy locations is required.** Without it, an existing `location ~* \.(js|css|...)$` regex block for static asset caching will intercept Next.js `_next/static/` requests and look for them in the wrong root directory, returning 404. + +2. **PM2 Python interpreter bug.** PM2 (at least some versions) ignores the `interpreter` field for Python scripts and wraps them in its JS `ProcessContainerFork.js`, which Python then tries to parse as Python code. Workaround: use a bash shell script that activates the venv and runs Python directly. + +3. **`NEXT_PUBLIC_*` env vars must be set at runtime too.** `next start` re-evaluates `next.config.ts` at startup. If `NEXT_PUBLIC_BASE_PATH` is only set during `npm run build` but not when `next start` runs (e.g., via PM2), basePath evaluates to `""` at runtime and all pages return 404 despite being correctly built. + +4. **Trailing slash on `proxy_pass` for Flask.** `location /simple-backend/` paired with `proxy_pass http://localhost:8081/;` (note trailing `/`) strips the `/simple-backend/` prefix. Flask routes are `/start-agent`, not `/simple-backend/start-agent`. + +5. **No changes needed for local dev.** When no `NEXT_PUBLIC_*` env vars are set, basePath defaults to `""` and backend URL defaults to `http://localhost:8082` — identical to the original behavior. diff --git a/GUIDE.md b/GUIDE.md new file mode 100644 index 0000000..647c37e --- /dev/null +++ b/GUIDE.md @@ -0,0 +1,205 @@ +# Build Agora Voice & Video AI Agents with Vibe Coding — No Coding Experience Required + +_How I used Claude Code to clone, configure, and run Agora's Conversational AI samples on my Mac — and you can too._ + +--- + +If you've been curious about building real-time voice and video AI agents but felt intimidated by the setup, this post is for you. In the accompanying video demo, I walk through the entire process of using Claude Code to build and run the Agora Conversational AI agent samples — from cloning the repo to having a live conversation with an AI agent, all running locally on my Mac. + +This blog post serves as a companion guide. It covers the prerequisites, explains how the system architecture works, and walks through the API keys you'll need before you can follow along with the video. + +## What Are We Building? + +The [Agora Conversational AI Agent Samples](https://github.com/AgoraIO-Conversational-AI/agent-samples) repo contains everything you need to run voice and video AI agents locally. The samples use Agora's real-time network (SD-RTN) to handle the audio and video streaming, while connecting to your choice of LLM and text-to-speech providers for the actual AI conversation. + +By the end of the video demo, you'll have a working voice AI agent you can talk to in your browser and, if you choose, a video avatar agent that gives your AI a face. + +The best part? Claude Code does the heavy lifting. You give it a prompt, point it at the repo, and it handles the cloning, dependency installation, configuration, and running of both the backend and frontend. + +--- + +## Watch the Video + +The companion video walks through every step live, including the moments where Claude Code asks clarifying questions, handles errors, and gets both servers running. If you're a visual learner or want to see exactly what to expect, start here. + +[![Watch the video](./assets/guide-video-thumbnail.png)](https://drive.google.com/file/d/1COGNNbLUkBFAT2XqLMqHFtOAg5tKisSZ/view?usp=sharing) + +--- + +## Prerequisites + +Before following the video, you'll need two things set up: Claude Code and a basic comfort level with the terminal. + +### Installing Claude Code + +Claude Code is Anthropic's command-line AI coding assistant. It runs in your terminal and can read, write, and execute code on your behalf. You'll need an Anthropic account with either a Claude Pro/Max subscription or API credits. + +**On Mac**, open Terminal and run: + +```bash +curl -fsSL https://claude.ai/install.sh | bash +``` + +That's it — no Node.js required. The native installer handles everything. After it finishes, verify the installation: + +```bash +claude --version +``` + +**On Windows**, you have two options. The simplest is to open PowerShell and run: + +```powershell +irm https://claude.ai/install.ps1 | iex +``` + +Alternatively, if you prefer working inside WSL (Windows Subsystem for Linux), install WSL first, then use the same curl command as Mac inside your Ubuntu terminal. + +After installation, launch Claude Code by typing `claude` in your terminal. The first time you run it, you'll complete a one-time authentication flow — either signing in with your Claude Pro/Max account or providing an API key from the Anthropic Console. + +### A Quick Terminal Primer + +If you're new to the terminal, here are the only commands you need to know to follow along. Everything else, Claude Code handles for you. + +**`pwd`** (Print Working Directory) — Shows you where you currently are in the file system. Think of it as "Where am I right now?" + +```bash +$ pwd +/Users/yourname +``` + +**`ls`** (List) — Shows the files and folders in your current directory. It's like opening a folder in Finder to see what's inside. + +```bash +$ ls +Desktop Documents Downloads Projects +``` + +**`cd`** (Change Directory) — Moves you into a different folder. This is how you navigate around. + +```bash +$ cd Projects +$ pwd +/Users/yourname/Projects +``` + +**`mkdir`** (Make Directory) — Creates a new folder. + +```bash +$ mkdir my-ai-project +$ cd my-ai-project +``` + +That's the extent of the terminal knowledge you need. Once you're in the right folder and launch Claude Code, you're communicating in plain English from that point forward. + +--- + +## System Architecture: The Four Components + +The Agora Conversational AI system has four main components. Two run on your local machine and two are cloud services managed by Agora. + +![System Architecture](./assets/system.svg) + +**Your Backend Server** (local) — A Python server (`simple-backend`) that serves the client app, generates Agora authentication tokens, and calls the Agora REST API to start/stop AI agent instances. When you click "Join," it spins up an agent and connects it to a channel with your LLM and TTS configuration. + +**The Client Application** (local) — A React web app running in your browser that captures your microphone audio and plays back the AI agent's responses. The repo includes a polished **React Voice Client** (recommended starting point), a **React Video Client with Avatar** for face-to-face-style AI conversations, and simpler HTML/JavaScript versions for quick testing. + +**Agora SD-RTN** (cloud) — Agora's Software-Defined Real-Time Network, the global low-latency infrastructure that routes audio, video, and data between participants. Both client and agent join the same channel — audio flows bidirectionally through SD-RTN without you having to manage any WebRTC complexity. + +**The AI Agent Instance** (cloud) — A managed agent that Agora spins up on your backend's request. It joins the channel like a real participant and runs the conversation pipeline: your speech is transcribed (STT), sent to your chosen LLM, and the response is converted to natural-sounding speech (TTS) and streamed back in real-time. It even handles interruption detection — if you start talking, the agent stops and listens. + +--- + +## Getting Your API Keys + +Before Claude Code can configure and run the samples, you'll need API keys from a few services. Here's where to get each one. + +### Agora Credentials + +You need three things from Agora: + +**App ID and App Certificate** — Sign up for a free account at the [Agora Console](https://console.agora.io/), create a new project, and you'll find these under [Project Management](https://console.agora.io/project-management). The App Certificate is optional for testing but recommended. + +**Agent Auth Header** — This is the Base64-encoded credential that authorizes your backend to call Agora's Conversational AI REST API. Generate it from the [RESTful API](https://console.agora.io/restful-api) section of the console. For more details, see Agora's [RESTful Authentication](https://docs.agora.io/en/conversational-ai/rest-api/restful-authentication) docs. + +### LLM Provider + +You'll need an API key from your chosen LLM provider. The samples default to OpenAI, so the fastest path is to grab an API key from [OpenAI's Platform](https://platform.openai.com/settings/organization/api-keys). Any OpenAI-compatible endpoint will work if you prefer a different provider. + +### Text-to-Speech (TTS) Provider + +The backend supports several TTS vendors. Pick one and get your API key: + +- [**Rime**](https://rime.ai/) — Fast, high-quality voices +- [**ElevenLabs**](https://elevenlabs.io/) — Realistic, customizable voices with a wide voice library +- [**OpenAI**](https://platform.openai.com/) — Uses the same API key as your LLM if you go with OpenAI for both +- [**Cartesia**](https://cartesia.ai/) — Low-latency voice synthesis + +You'll also need a Voice ID from your chosen provider. Each vendor's documentation or voice library has a catalog you can browse to find the voice you like. + +### Avatar Provider (Video Client Only) + +If you want to run the video avatar client, you'll need credentials from an avatar provider as well. Agora's documentation on [AI Avatar Overview](https://docs.agora.io/en/conversational-ai/models/avatar/overview) covers the supported vendors and configuration details. + +### Keep Your Keys Handy + +Once you have your keys, keep them in a note or text file. When you follow along with the video and Claude Code asks for configuration, you'll paste them into the `.env` file that Claude Code creates. Here's a preview of what you'll need: + +```bash +# Agora +APP_ID=your_agora_app_id +APP_CERTIFICATE=your_agora_app_certificate +AGENT_AUTH_HEADER=your_base64_encoded_credentials + +# LLM +LLM_API_KEY=your_openai_api_key + +# TTS +TTS_VENDOR=elevenlabs # or: rime, openai, cartesia +TTS_KEY=your_tts_api_key +TTS_VOICE_ID=your_chosen_voice_id +``` + +--- + +## Let Claude Code Do the Work + +With your prerequisites set up and API keys ready, you're all set to follow along with the video. Here's the prompt I use to kick things off: + +**For Voice:** + +``` +Clone https://github.com/AgoraIO-Conversational-AI/agent-samples +and then I want to run the React Voice AI Agent here on my laptop. +Be sure to read the AGENT.md before you begin building. +``` + +**For Video with Avatar:** + +``` +Clone https://github.com/AgoraIO-Conversational-AI/agent-samples +and then I want to run the Video AI Agent with Avatar Sample here +on my laptop. Be sure to read the AGENT.md before you begin building. +``` + +Claude Code reads the repo's `AGENT.md` file (a comprehensive guide written specifically for AI coding assistants), clones the repository, installs all dependencies, prompts you for your API keys, configures the environment files, and starts both the backend and frontend servers. + +Within minutes, you'll have a working AI voice agent running in your browser that you can talk to in real-time. + +--- + +## Wrapping Up + +What used to require deep knowledge of WebRTC, server configuration, and multiple API integrations can now be accomplished with a single prompt to Claude Code. The combination of Agora's well-documented agent samples and Claude Code's ability to read and follow that documentation makes this one of the most accessible ways to get started with real-time conversational AI. + +If you build something interesting with these samples, I'd love to hear about it. Drop a comment or reach out — and happy building. + +--- + +## References + +- **Agora Conversational AI Agent Samples** (MIT) — [github.com/AgoraIO-Conversational-AI/agent-samples](https://github.com/AgoraIO-Conversational-AI/agent-samples) +- **Agora Conversational AI Documentation** — [docs.agora.io/en/conversational-ai/overview/product-overview](https://docs.agora.io/en/conversational-ai/overview/product-overview) +- **Claude Code Setup Guide** — [code.claude.com/docs/en/setup](https://code.claude.com/docs/en/setup) +- **Agora Console** — [console.agora.io](https://console.agora.io/) +- **OpenAI API Keys** — [platform.openai.com](https://platform.openai.com/settings/organization/api-keys) +- **TTS Providers** — [Rime](https://rime.ai/) | [ElevenLabs](https://elevenlabs.io/) | [OpenAI TTS](https://platform.openai.com/) | [Cartesia](https://cartesia.ai/) diff --git a/README.md b/README.md index 518ddb2..42986fc 100644 --- a/README.md +++ b/README.md @@ -12,12 +12,20 @@ up the sample backend and one of the sample clients or ask AI to do it for you. **Comprehensive implementation guide for AI agents** → [AGENT.md](./AGENT.md) -**Example prompt for Claude Code:** +**Example prompt for Claude Code (Voice):** ``` -using https://github.com/AgoraIO-Conversational-AI/agent-samples I want to run the +Clone https://github.com/AgoraIO-Conversational-AI/agent-samples and then I want to run the +React Voice AI Agent here on my laptop. Be sure to read the AGENT.md +before you begin building. +``` + +**Example prompt for Claude Code (Avatar):** + +``` +Clone https://github.com/AgoraIO-Conversational-AI/agent-samples and then I want to run the Video AI Agent with Avatar Sample here on my laptop. Be sure to read the AGENT.md -before you begin. +before you begin building. ``` ## System Architecture @@ -118,10 +126,10 @@ and multi-stream video rendering. ### Basic Samples -**[Simple Voice AI Client](./simple-voice-client/)** Standalone HTML/JavaScript +**[Simple Voice AI Client (No Backend)](./simple-voice-client-no-backend/)** Standalone HTML/JavaScript client for testing voice agents. Maintains persistent RTC connection allowing agents to join and leave without client reconnection. -**[HTML/JS Voice AI Client](./complete-voice-client/)** Full-featured vanilla +**[Simple Voice AI Client (With Backend)](./simple-voice-client-with-backend/)** Full-featured vanilla JavaScript client demonstrating end-to-end integration with backend for agent initialization and voice interaction. diff --git a/assets/AI_SAMPLES_UIKIT_TOOLKIT_DEV.md b/assets/AI_SAMPLES_UIKIT_TOOLKIT_DEV.md new file mode 100644 index 0000000..e65ee09 --- /dev/null +++ b/assets/AI_SAMPLES_UIKIT_TOOLKIT_DEV.md @@ -0,0 +1,422 @@ +# Cross-Repository Development Guide + +**Location**: `agent-samples/assets/AI_SAMPLES_UIKIT_TOOLKIT_DEV.md` + +This guide explains how to develop across the three Agora Conversational AI repositories. Read this file at the start of development sessions for context on the workflow, git hooks, and package management. + +## Repository Overview + +**Repositories**: + +- `agent-samples/` - Sample applications (GitHub: AgoraIO-Conversational-AI/agent-samples) +- `agent-toolkit/` - Core SDK (GitHub: AgoraIO-Conversational-AI/agent-toolkit) +- `agent-ui-kit/` - UI components (GitHub: AgoraIO-Conversational-AI/agent-ui-kit) + +**Package Installation**: Samples install toolkit and ui-kit from GitHub: + +```json +"@agora/conversational-ai": "github:AgoraIO-Conversational-AI/agent-toolkit#main", +"@agora/agent-ui-kit": "github:AgoraIO-Conversational-AI/agent-ui-kit#main" +``` + +--- + +## Development Workflow + +### When to Edit Each Repository + +**agent-toolkit** - Edit when: + +- Fixing bugs in RTCHelper, ConversationalAIAPI, or core SDK logic +- Adding new features to the SDK +- Changing SDK interfaces or behavior + +**agent-ui-kit** - Edit when: + +- Fixing bugs in UI components (MicButton, Message, Response, etc.) +- Adding new UI components +- Changing component props or styling + +**agent-samples** - Edit when: + +- Fixing bugs in sample apps (VoiceClient, VideoAvatarClient) +- Adding new sample applications +- Updating documentation or configuration + +--- + +## Making Changes to Toolkit or UI-Kit + +When fixing issues that require changes to `agent-toolkit` or `agent-ui-kit`: + +### 1. Test Changes Locally First + +Edit files in `node_modules` for quick testing: + +```bash +# Example: Fix bug in RTCHelper +cd agent-samples/react-voice-client +# Edit node_modules/@agora/conversational-ai/helper/rtc.ts +npm run dev # Test the fix works +``` + +### 2. Copy Changes to Source Repository + +Once the fix works, copy it to the source repo: + +```bash +# For toolkit: +cp node_modules/@agora/conversational-ai/helper/rtc.ts \ + ../../../agent-toolkit/packages/conversational-ai/helper/rtc.ts + +# For ui-kit: +cp node_modules/@agora/agent-ui-kit/components/MicButton.tsx \ + ../../../agent-ui-kit/packages/agent-ui-kit/components/MicButton.tsx +``` + +### 3. Update Documentation + +Update the README in the repo you modified: + +- `agent-toolkit/README.md` - Document new SDK features/APIs +- `agent-ui-kit/README.md` - Document new component props/usage +- `agent-samples/AGENT.md` - Note breaking changes if any + +### 4. Commit and Push + +```bash +cd agent-toolkit # or agent-ui-kit +git add . +git commit -m "feat: description" # or "fix:", "docs:", etc. +git push origin main +``` + +**⚠️ IMPORTANT - Never Use --no-verify**: + +- Never use `git commit --no-verify` or `git commit -n` +- Pre-commit hooks check ESLint, Prettier, secrets, commit message format +- If hooks fail, fix the errors instead of bypassing them +- ESLint errors: `cd && npx eslint ` +- Prettier errors: `npx prettier --write ` +- Commit-msg hook blocks "claude" references (case-insensitive) + +### 5. Update Samples to Use Latest Changes + +After pushing to toolkit/ui-kit, samples will automatically get the latest version on next `npm install`: + +```bash +cd agent-samples/react-voice-client +rm -rf node_modules package-lock.json +npm install --legacy-peer-deps +npm run dev # Test with latest toolkit/ui-kit from GitHub +``` + +**Why this works**: The `package.json` references use `github:org/repo#main`, so `npm install` always pulls the latest commit from the `main` branch. + +--- + +## Making Changes to Samples Only + +For changes that only affect sample apps (not toolkit/ui-kit): + +### 1. Edit and Test + +```bash +cd agent-samples/react-voice-client +# Edit files +npm run dev # Test changes +``` + +### 2. Fix Linting Before Committing + +```bash +# TypeScript/JavaScript: +cd react-voice-client # or react-video-client-avatar +npx eslint +npx prettier --write + +# Markdown: +npx prettier --write AGENT.md simple-backend/README.md +``` + +### 3. Commit and Push + +```bash +cd agent-samples +git add . +git commit -m "feat: description" # NEVER use --no-verify! +git push origin main +``` + +--- + +## Package Version Management + +### Current Strategy: Git-Based Installation + +Samples install toolkit and ui-kit directly from GitHub main branch: + +```json +"@agora/conversational-ai": "github:AgoraIO-Conversational-AI/agent-toolkit#main", +"@agora/agent-ui-kit": "github:AgoraIO-Conversational-AI/agent-ui-kit#main" +``` + +**Advantages**: + +- No need to publish to npm registry +- No version number management +- Samples always get latest fixes on fresh install + +**Fresh install process**: + +```bash +cd agent-samples/react-voice-client +rm -rf node_modules package-lock.json +npm install --legacy-peer-deps +``` + +### Future: NPM Registry Publishing (Not Currently Used) + +If switching to versioned npm packages in the future: + +**Publishing to npm** (toolkit or ui-kit): + +```bash +cd agent-toolkit # or agent-ui-kit +npm version patch # or minor/major +npm run build +npm publish +git push origin main --follow-tags +``` + +**Updating samples**: + +```json +"@agora/conversational-ai": "^1.2.3", +"@agora/agent-ui-kit": "^0.5.0" +``` + +```bash +cd agent-samples/react-voice-client +npm install @agora/conversational-ai@latest +npm install @agora/agent-ui-kit@latest +``` + +--- + +## Repository Structure + +``` +agent-samples/ # Sample applications +├── react-voice-client/ +├── react-video-client-avatar/ +├── simple-voice-client-no-backend/ +├── simple-voice-client-with-backend/ +└── simple-backend/ + +agent-toolkit/ # Core SDK (@agora/conversational-ai) +└── packages/ + ├── conversational-ai/ # Main package + └── react/ # React hooks + +agent-ui-kit/ # UI components (@agora/agent-ui-kit) +└── packages/ + └── agent-ui-kit/ +``` + +--- + +## Architecture Notes + +### RTCHelper (Singleton Pattern) + +**Audio Methods**: + +- `createAudioTrack()` - Create microphone track +- `setMuted(boolean)` - Mute/unmute audio +- `getMuted()` - Get mute state + +**Video Methods**: + +- `createVideoTrack()` - Create camera track +- `setVideoEnabled(boolean)` - Enable/disable camera +- `getVideoEnabled()` - Get camera state + +**Lifecycle**: + +- `init(config)` - Initialize with app ID, channel, token, UID +- `join()` - Join RTC channel +- `leave()` - Leave channel and cleanup all tracks +- `destroy()` - Destroy instance + +**Subscription Filters** (optional): + +```typescript +await rtcHelper.init({ + appId: "...", + channel: "...", + token: "...", + uid: 12345, + shouldSubscribeAudio: (uid) => uid !== 999, // Skip audio from uid 999 + shouldSubscribeVideo: (uid) => uid === 100, // Only video from uid 100 +}); +``` + +### Video Track Lifecycle Best Practices + +```typescript +// Create once +await rtcHelper.createVideoTrack({ encoderConfig: "720p_2" }); +await rtcHelper.client.publish(rtcHelper.localVideoTrack); + +// Toggle (reuse same track) +await rtcHelper.setVideoEnabled(false); // Camera off +await rtcHelper.setVideoEnabled(true); // Camera on + +// Cleanup (automatic) +await rtcHelper.leave(); // Stops and closes all tracks +``` + +**Common Pitfalls**: + +- ❌ Don't recreate track on toggle - use `setVideoEnabled()` +- ❌ Don't use `key` prop on video components - causes remounting +- ❌ Don't manually cleanup tracks - RTCHelper handles it +- ✅ Pass `null` to video component when disabled +- ✅ Check `track._isClosed` before using track references + +--- + +## Git Hooks + +All three repositories use git hooks to enforce code quality: + +### Pre-commit Hook + +**Location**: `.git/hooks/pre-commit` + +**What it checks**: + +- ESLint for TypeScript/JavaScript files +- Prettier formatting for all files +- Secret detection (API keys, tokens, credentials) + +**How it works**: + +```bash +# For React projects (react-voice-client, react-video-client-avatar): +# Hook cd's into project directory before running ESLint +# This ensures project-specific .eslintrc.js is used + +# Example from agent-samples pre-commit: +for file in $TS_FILES; do + project_dir=$(echo "$file" | cut -d'/' -f1) + if [[ "$project_dir" == react-* ]]; then + (cd "$project_dir" && npx eslint "$(basename $(dirname $file))/$(basename $file)") + fi +done +``` + +**If pre-commit fails**: + +```bash +# Fix ESLint errors: +cd react-voice-client # or react-video-client-avatar +npx eslint +# Fix errors manually, then re-run to verify + +# Fix Prettier errors: +npx prettier --write +``` + +### Commit-msg Hook + +**Location**: `.git/hooks/commit-msg` + +**What it checks**: + +- Blocks commits with "claude" in the message (case-insensitive) +- Enforces lowercase first character in commit message + +**Example failures**: + +```bash +git commit -m "Claude helped fix this" +# ❌ Error: commit message must not mention Claude + +git commit -m "Fix bug" +# ❌ Error: commit message should start with lowercase +``` + +**Correct format**: + +```bash +git commit -m "fix: video track not re-enabling" +git commit -m "feat: add shouldSubscribe filters to RTCHelper" +git commit -m "docs: update README with video API" +``` + +### Never Use --no-verify + +**⚠️ CRITICAL**: Never use `git commit --no-verify` or `-n` flag + +**Why**: + +- Bypasses ESLint → introduces linting errors +- Bypasses Prettier → introduces formatting inconsistencies +- Bypasses secrets detection → risk of committing credentials +- Bypasses commit-msg validation → allows blocked words + +**If hooks fail**: + +- Fix the errors instead of bypassing +- Hooks protect code quality and security +- Hooks ensure consistent style across the team + +--- + +## Quick Reference + +### Check Installed Package Versions + +```bash +cd agent-samples/react-voice-client +npm list @agora/conversational-ai +npm list @agora/agent-ui-kit +``` + +### Force Fresh Install + +```bash +cd agent-samples/react-voice-client +rm -rf node_modules package-lock.json +npm install --legacy-peer-deps +``` + +### Run Backend Tests + +```bash +cd agent-samples/simple-backend +pytest # All tests +pytest tests/test_agent.py # Specific file +``` + +### Dev Servers + +```bash +# Backend (port 8082) +cd agent-samples/simple-backend +PORT=8082 python3 local_server.py + +# Voice Client (port 8083) +cd agent-samples/react-voice-client +npm run dev + +# Video Client (port 8084) +cd agent-samples/react-video-client-avatar +npm run dev +``` + +--- + +**Last Updated**: 2026-01-28 diff --git a/assets/guide-video-thumbnail.png b/assets/guide-video-thumbnail.png new file mode 100644 index 0000000..b20a169 Binary files /dev/null and b/assets/guide-video-thumbnail.png differ diff --git a/react-video-client-avatar/components/VideoAvatarClient.tsx b/react-video-client-avatar/components/VideoAvatarClient.tsx index c2caa65..70a13d4 100644 --- a/react-video-client-avatar/components/VideoAvatarClient.tsx +++ b/react-video-client-avatar/components/VideoAvatarClient.tsx @@ -14,7 +14,7 @@ import { AgoraLogo } from "@agora/agent-ui-kit"; import { SettingsDialog } from "@agora/agent-ui-kit"; import { cn } from "@/lib/utils"; -const DEFAULT_BACKEND_URL = "http://localhost:8082"; +const DEFAULT_BACKEND_URL = process.env.NEXT_PUBLIC_BACKEND_URL || "http://localhost:8082"; export function VideoAvatarClient() { const [backendUrl, setBackendUrl] = useState(DEFAULT_BACKEND_URL); @@ -34,6 +34,17 @@ export function VideoAvatarClient() { const [activeTab, setActiveTab] = useState("video"); const _conversationRef = useRef(null); + // Read URL parameters on mount + useEffect(() => { + if (typeof window !== 'undefined') { + const params = new URLSearchParams(window.location.search); + const urlProfile = params.get('profile'); + if (urlProfile) { + setProfile(urlProfile); + } + } + }, []); + const { isConnected, isMuted, diff --git a/react-video-client-avatar/next.config.ts b/react-video-client-avatar/next.config.ts index d931aed..1f0da98 100644 --- a/react-video-client-avatar/next.config.ts +++ b/react-video-client-avatar/next.config.ts @@ -1,6 +1,8 @@ import type { NextConfig } from "next"; const nextConfig: NextConfig = { + basePath: process.env.NEXT_PUBLIC_BASE_PATH || "", + typescript: { ignoreBuildErrors: true }, transpilePackages: ["@agora/conversational-ai", "@agora/agent-ui-kit"], }; diff --git a/react-voice-client/components/VoiceClient.tsx b/react-voice-client/components/VoiceClient.tsx index 096527d..d9b6c45 100644 --- a/react-voice-client/components/VoiceClient.tsx +++ b/react-voice-client/components/VoiceClient.tsx @@ -1,6 +1,6 @@ "use client"; -import { useState, useRef } from "react"; +import { useState, useRef, useEffect } from "react"; import { Mic, MicOff, Settings } from "lucide-react"; import { useAgoraVoiceClient } from "@/hooks/useAgoraVoiceClient"; import { useAudioVisualization } from "@/hooks/useAudioVisualization"; @@ -13,7 +13,7 @@ import { AgoraLogo } from "@agora/agent-ui-kit"; import { SettingsDialog } from "@agora/agent-ui-kit"; import { cn } from "@/lib/utils"; -const DEFAULT_BACKEND_URL = "http://localhost:8082"; +const DEFAULT_BACKEND_URL = process.env.NEXT_PUBLIC_BACKEND_URL || "http://localhost:8082"; export function VoiceClient() { const [backendUrl, setBackendUrl] = useState(DEFAULT_BACKEND_URL); @@ -30,6 +30,17 @@ export function VoiceClient() { const [greeting, setGreeting] = useState("hi there"); const conversationRef = useRef(null); + // Read URL parameters on mount + useEffect(() => { + if (typeof window !== 'undefined') { + const params = new URLSearchParams(window.location.search); + const urlProfile = params.get('profile'); + if (urlProfile) { + setProfile(urlProfile); + } + } + }, []); + const { isConnected, isMuted, diff --git a/react-voice-client/next.config.ts b/react-voice-client/next.config.ts index d931aed..1f0da98 100644 --- a/react-voice-client/next.config.ts +++ b/react-voice-client/next.config.ts @@ -1,6 +1,8 @@ import type { NextConfig } from "next"; const nextConfig: NextConfig = { + basePath: process.env.NEXT_PUBLIC_BASE_PATH || "", + typescript: { ignoreBuildErrors: true }, transpilePackages: ["@agora/conversational-ai", "@agora/agent-ui-kit"], }; diff --git a/simple-backend/.env.example b/simple-backend/.env.example index 8fb7a4a..e219a5a 100644 --- a/simple-backend/.env.example +++ b/simple-backend/.env.example @@ -96,17 +96,16 @@ VOICE_DEFAULT_FAILURE_MESSAGE=Sorry, something went wrong # ===================================================== # VIDEO PROFILE (react-video-client-avatar default) # ===================================================== -# Architecture: TTS + LLM mode with HeyGen avatar +# Architecture: TTS + LLM mode with avatar (HeyGen or Anam) # Agora credentials VIDEO_APP_ID= VIDEO_APP_CERTIFICATE= VIDEO_AGENT_AUTH_HEADER=Basic -VIDEO_AGENT_ENDPOINT=https://api.agora.io/api/conversational-ai-agent/v2/projects # LLM settings VIDEO_LLM_API_KEY= -VIDEO_LLM_MODEL=gpt-4o +VIDEO_LLM_MODEL=gpt-4o-mini VIDEO_LLM_URL=https://api.openai.com/v1/chat/completions # TTS settings - ElevenLabs @@ -115,19 +114,25 @@ VIDEO_TTS_KEY= VIDEO_TTS_VOICE_ID= # Get from https://elevenlabs.io/ VIDEO_ELEVENLABS_MODEL=eleven_flash_v2_5 VIDEO_ELEVENLABS_STABILITY=0.5 -VIDEO_ELEVENLABS_SAMPLE_RATE=24000 +VIDEO_TTS_SAMPLE_RATE=24000 # ASR settings VIDEO_ASR_VENDOR=ares VIDEO_ASR_LANGUAGE=en-US -# Avatar settings - HeyGen +# Avatar settings - Choose either HeyGen or Anam +# For HeyGen: VIDEO_AVATAR_VENDOR=heygen VIDEO_AVATAR_API_KEY= VIDEO_AVATAR_ID= # e.g., Graham_Chair_Sitting_public VIDEO_HEYGEN_QUALITY=high VIDEO_HEYGEN_ACTIVITY_IDLE_TIMEOUT=120 +# For Anam (production): +# VIDEO_AVATAR_VENDOR=anam +# VIDEO_AVATAR_API_KEY= +# VIDEO_AVATAR_ID= # Get from Anam dashboard + # Agent behavior VIDEO_VAD_SILENCE_DURATION_MS=300 VIDEO_ENABLE_AIVAD=true @@ -139,24 +144,8 @@ VIDEO_DEFAULT_GREETING=Hey there, I am Quiz Master Bella, would you like me to q VIDEO_DEFAULT_PROMPT=You are Bella, a quiz master who asks capital city questions. You can hear the user clearly through voice. Keep responses under 20 words and immediately ask the next question after announcing the result. VIDEO_DEFAULT_FAILURE_MESSAGE=Sorry, something went wrong -# ===================================================== -# VIDEO_ANAM PROFILE (Optional - Anam Avatar) -# ===================================================== -# Same as VIDEO but uses Anam avatar provider with different endpoint - -# VIDEO_ANAM_APP_ID= -# VIDEO_ANAM_APP_CERTIFICATE= -# VIDEO_ANAM_AGENT_AUTH_HEADER=Basic -# VIDEO_ANAM_AGENT_ENDPOINT=https://api-test.agora.io/api/conversational-ai-agent/v2/projects -# VIDEO_ANAM_LLM_API_KEY= -# VIDEO_ANAM_TTS_VENDOR=elevenlabs -# VIDEO_ANAM_TTS_KEY= -# VIDEO_ANAM_TTS_VOICE_ID= -# VIDEO_ANAM_AVATAR_VENDOR=anam -# VIDEO_ANAM_AVATAR_API_KEY= -# VIDEO_ANAM_AVATAR_ID= -# VIDEO_ANAM_ANAM_BASE_URL=https://api.anam.ai/v1 -# ... (plus other VIDEO_ANAM_* variables as needed) +# Note: Anam is now in production and uses the same standard endpoint as HeyGen. +# Simply set VIDEO_AVATAR_VENDOR=anam in the VIDEO profile above. # ===================================================== # TTS VENDOR OPTIONS @@ -184,6 +173,63 @@ VIDEO_DEFAULT_FAILURE_MESSAGE=Sorry, something went wrong # CARTESIA_MODEL=sonic-3 # TTS_VOICE_ID: Get from https://cartesia.ai/ +# ===================================================== +# MCP SETTINGS (Optional - enable external tool calling) +# ===================================================== +# See: https://github.com/AgoraIO-Community/server-mcp-memory +# +# MCP (Model Context Protocol) allows the agent to call external tools +# during conversation. The endpoint must be publicly accessible. +# +# MCP_SERVERS is a JSON array of server configs: +# - name: Unique identifier (alphanumeric, max 48 chars) +# - endpoint: Public URL of the MCP server +# - transport: Must be "streamable_http" +# - headers: Optional auth headers +# - allowed_tools: ["*"] for all, or list specific tool names +# +# Example with mem0 (persistent memory): +# VOICE_MCP_SERVERS=[{"name":"mem0","endpoint":"https://your-tunnel.trycloudflare.com/mcp","transport":"streamable_http","headers":{"Authorization":"Token m0-xxx"},"allowed_tools":["*"]}] +# +# For local MCP servers, use Cloudflare Tunnel to make them publicly accessible: +# cloudflared tunnel --url http://localhost:8081 +# +# Multiple servers: +# VOICE_MCP_SERVERS=[{"name":"mem0","endpoint":"https://tunnel1.trycloudflare.com/mcp","transport":"streamable_http","headers":{"Authorization":"Token m0-xxx"},"allowed_tools":["*"]},{"name":"weather","endpoint":"https://tunnel2.trycloudflare.com/mcp","transport":"streamable_http","allowed_tools":["get_weather"]}] + +# ===================================================== +# CUSTOM LLM SERVER (Optional — advanced) +# ===================================================== +# A Custom LLM server sits between Agora ConvoAI and your LLM provider, +# giving you full control over prompts, RAG, tool calling, and response formatting. +# +# See: https://github.com/AgoraIO-Community/server-custom-llm +# +# To use a custom LLM server: +# 1. Deploy one of the server implementations (Python, Node.js, or Go) +# 2. Set LLM_URL to your server's endpoint +# 3. Set LLM_VENDOR=custom to receive turn_id and timestamp in requests +# +# Example: +# VOICE_LLM_URL=https://your-custom-llm.example.com/chat/completions +# VOICE_LLM_API_KEY=your-openai-key +# VOICE_LLM_VENDOR=custom +# VOICE_LLM_STYLE=openai +# +# Endpoints available: +# /chat/completions - Basic LLM proxy with streaming +# /rag/chat/completions - RAG-enhanced with knowledge base injection +# /audio/chat/completions - Multimodal audio response +# +# LLM_STYLE options: openai (default), gemini, anthropic, dify +# LLM_VENDOR: set to "custom" for custom LLM servers (adds turn_id + timestamp), +# "azure" for Azure OpenAI. Omit to use default behavior. +# +# Greeting behavior (applies to any LLM mode): +# GREETING_MODE: single_every (greet every user join, default behavior), +# single_first (greet only the first user who joins) +# VOICE_GREETING_MODE=single_every + # ===================================================== # DEBUG OPTIONS # ===================================================== diff --git a/simple-backend/README.md b/simple-backend/README.md index 168f5d7..6cf7dc1 100644 --- a/simple-backend/README.md +++ b/simple-backend/README.md @@ -206,3 +206,34 @@ simple-backend/ ├── local_server.py # Flask development server └── .env # Local config (gitignored) ``` + +## Custom LLM Server (Optional) + +A Custom LLM server sits between Agora ConvoAI and your LLM provider, giving you full control over prompts, RAG, tool calling, and response formatting. + +See: [server-custom-llm](https://github.com/AgoraIO-Community/server-custom-llm) + +**Configuration:** Set `LLM_URL` to your custom server endpoint and `LLM_VENDOR=custom` in `.env`: + +```bash +VOICE_LLM_URL=https://your-custom-llm.example.com/chat/completions +VOICE_LLM_API_KEY=your-openai-key +VOICE_LLM_VENDOR=custom +VOICE_LLM_STYLE=openai +``` + +The custom server proxies requests to your LLM provider and supports endpoints for basic chat (`/chat/completions`), RAG-enhanced chat (`/rag/chat/completions`), and multimodal audio (`/audio/chat/completions`). + +## MCP Memory Server (Optional) + +An MCP memory server gives agents persistent per-user memory via tool calling, allowing the agent to remember context across conversations. + +See: [server-mcp-memory](https://github.com/AgoraIO-Community/server-mcp-memory) + +**Configuration:** Set `MCP_SERVERS` as a JSON array in `.env`: + +```bash +VOICE_MCP_SERVERS=[{"name":"memory","endpoint":"https://your-mcp-server.example.com/mcp","transport":"streamable_http","allowed_tools":["*"]}] +``` + +The MCP server must be publicly accessible. For local development, use [Cloudflare Tunnel](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) to expose your local server. diff --git a/simple-backend/core/agent.py b/simple-backend/core/agent.py index f3ef705..60fa553 100644 --- a/simple-backend/core/agent.py +++ b/simple-backend/core/agent.py @@ -137,7 +137,7 @@ def build_mllm_config(constants, query_params=None): mllm_config = { "predefined_tools": ["_publish_message"], "vendor": query_params.get('mllm_vendor', constants.get("MLLM_VENDOR", "vertexai")), - "url": query_params.get('mllm_url', constants.get("MLLM_URL", "")), + "url": query_params.get('mllm_url') or constants.get("MLLM_URL") or "", "api_key": "", "messages": [ { @@ -168,6 +168,37 @@ def build_mllm_config(constants, query_params=None): return mllm_config +def build_mcp_servers(constants, query_params=None, channel=None): + """ + Parses MCP_SERVERS JSON config and returns list of MCP server configs. + + Args: + constants: Dictionary of constants + query_params: Optional query parameters for overrides + channel: Channel name used as user_id for per-user MCP endpoints + + Returns: + List of MCP server config dicts, or None if not configured + """ + query_params = query_params or {} + mcp_json = query_params.get('mcp_servers', constants.get("MCP_SERVERS")) + if not mcp_json: + return None + + servers = json.loads(mcp_json) if isinstance(mcp_json, str) else mcp_json + if not servers: + return None + + # Inject user_id (channel name) into endpoint paths for per-user partitioning + user_id = query_params.get('mcp_user_id', channel) if query_params else channel + if user_id: + for server in servers: + endpoint = server.get("endpoint", "").rstrip("/") + server["endpoint"] = f"{endpoint}/{user_id}" + + return servers + + def build_avatar_config(avatar_vendor, constants, channel, agent_video_token, query_params=None): """ Builds avatar configuration based on vendor. @@ -224,11 +255,11 @@ def build_avatar_config(avatar_vendor, constants, channel, agent_video_token, qu "vendor": "anam", "enable": True, "params": { - "agora_token": agora_token_value, - "agora_uid": query_params.get('anam_uid', constants.get("AGENT_VIDEO_UID", "49345")), "anam_api_key": constants["AVATAR_API_KEY"], - "anam_base_url": constants["ANAM_BASE_URL"], - "anam_avatar_id": constants["AVATAR_ID"] + "agora_uid": constants["AGENT_VIDEO_UID"], + "agora_token": agora_token_value, + "anam_avatar_id": constants["AVATAR_ID"], + "anam_base_url": "https://api.anam.ai/v1" } } else: @@ -295,6 +326,11 @@ def create_agent_payload(channel, constants, query_params=None, agent_video_toke failure_message = query_params.get('failure_message', constants["DEFAULT_FAILURE_MESSAGE"]) max_history = int(query_params.get('max_history', constants["MAX_HISTORY"])) + # Get Custom LLM parameters + llm_style = query_params.get('llm_style', constants.get("LLM_STYLE", "openai")) + llm_vendor = query_params.get('llm_vendor', constants.get("LLM_VENDOR")) + greeting_mode = query_params.get('greeting_mode', constants.get("GREETING_MODE")) + # Build LLM configuration llm_config = { "url": llm_url, @@ -311,19 +347,30 @@ def create_agent_payload(channel, constants, query_params=None, agent_video_toke "params": { "model": llm_model }, - "style": "openai" + "style": llm_style } + # Optional: vendor field (e.g., "custom" adds turn_id + timestamp to requests) + if llm_vendor: + llm_config["vendor"] = llm_vendor + + # Optional: greeting behavior configuration + if greeting_mode: + llm_config["greeting_configs"] = {"mode": greeting_mode} + mllm_config = None + # Build MCP servers config (works in standard LLM mode) + mcp_servers = build_mcp_servers(constants, query_params, channel=channel) + if mcp_servers and llm_config: + llm_config["mcp_servers"] = mcp_servers + print(f"🔧 MCP: {len(mcp_servers)} server(s) configured: {[s.get('name') for s in mcp_servers]}") + # Get avatar settings early to determine remote_rtc_uids and token avatar_vendor = constants.get("AVATAR_VENDOR") - # Determine token value - # Anam uses empty string for token (per working curl from Agora developer) - # Regular mode uses APP_ID (since no certificate) - is_anam_avatar = avatar_vendor == "anam" - app_id_for_token = "" if is_anam_avatar else constants["APP_ID"] + # Determine token value (use APP_ID since no certificate) + app_id_for_token = constants["APP_ID"] # When avatar is enabled, can't use wildcard "*" for remote_rtc_uids # Must specify exact user UID @@ -339,11 +386,13 @@ def create_agent_payload(channel, constants, query_params=None, agent_video_toke if enable_mllm: advanced_features["enable_mllm"] = True advanced_features["enable_tools"] = False + elif mcp_servers: + advanced_features["enable_tools"] = True # Build properties properties = OrderedDict([ ("channel", channel), - ("token", app_id_for_token), # Empty string for Anam BETA, regular app_id otherwise + ("token", app_id_for_token), ("agent_rtc_uid", constants["AGENT_UID"]), ("agent_rtm_uid", f"{constants['AGENT_UID']}-{channel}"), ("remote_rtc_uids", remote_rtc_uids), @@ -373,7 +422,7 @@ def create_agent_payload(channel, constants, query_params=None, agent_video_toke # Add turn_detection for MLLM mode if enable_mllm: properties["turn_detection"] = { - "type": query_params.get('turn_detection_type', constants.get("TURN_DETECTION_TYPE", "server_vad")) + "type": query_params.get('turn_detection_type') or constants.get("TURN_DETECTION_TYPE") or "server_vad" } # Add transcript parameters for TTS+LLM mode @@ -388,8 +437,6 @@ def create_agent_payload(channel, constants, query_params=None, agent_video_toke # Add avatar configuration if vendor is set if avatar_vendor: - # For Anam, we don't need a real token (it uses app_id instead) - # So pass agent_video_token even if it's empty string avatar_config = build_avatar_config( avatar_vendor, constants, @@ -421,20 +468,9 @@ def send_agent_to_channel(channel, agent_payload, constants): Returns: Dictionary with the status code, response body, and success flag """ - # Check if using Anam avatar to determine endpoint - is_anam_avatar = ( - agent_payload.get("properties", {}).get("avatar", {}).get("vendor") == "anam" - ) - - if is_anam_avatar: - # Use Anam-specific endpoint - agent_api_url = f"{constants['ANAM_AGENT_ENDPOINT']}/{constants['APP_ID']}/join" - auth_header = constants["AGENT_AUTH_HEADER"] - print(f"🎭 Using Anam endpoint: {agent_api_url}") - else: - # Use regular endpoint - agent_api_url = f"{constants['AGENT_ENDPOINT']}/{constants['APP_ID']}/join" - auth_header = constants["AGENT_AUTH_HEADER"] + # Use regular endpoint + agent_api_url = f"{constants['AGENT_ENDPOINT']}/{constants['APP_ID']}/join" + auth_header = constants["AGENT_AUTH_HEADER"] url_parts = urllib.parse.urlparse(agent_api_url) host = url_parts.netloc @@ -447,11 +483,6 @@ def send_agent_to_channel(channel, agent_payload, constants): "Authorization": auth_header } - # Add X-Request-Id for Anam requests - if is_anam_avatar: - import uuid - headers["X-Request-Id"] = str(uuid.uuid4()).replace('-', '') - payload_json = json.dumps(agent_payload, indent=2) print(f"Sending agent to Agora ConvoAI:") diff --git a/simple-backend/core/config.py b/simple-backend/core/config.py index 2e4d199..49998e8 100644 --- a/simple-backend/core/config.py +++ b/simple-backend/core/config.py @@ -69,6 +69,9 @@ def initialize_constants(profile=None): "LLM_URL": get_env_var('LLM_URL', profile, "https://api.openai.com/v1/chat/completions"), "LLM_API_KEY": get_env_var('LLM_API_KEY', profile), "LLM_MODEL": get_env_var('LLM_MODEL', profile, "gpt-4o-mini"), + "LLM_STYLE": get_env_var('LLM_STYLE', profile, "openai"), + "LLM_VENDOR": get_env_var('LLM_VENDOR', profile), + "GREETING_MODE": get_env_var('GREETING_MODE', profile), # TTS settings (vendor required, no default) "TTS_VENDOR": get_env_var('TTS_VENDOR', profile), @@ -144,10 +147,8 @@ def initialize_constants(profile=None): "HEYGEN_QUALITY": get_env_var('HEYGEN_QUALITY', profile, "high"), "HEYGEN_ACTIVITY_IDLE_TIMEOUT": get_env_var('HEYGEN_ACTIVITY_IDLE_TIMEOUT', profile, "120"), - # Anam specific settings (endpoints used automatically when vendor=anam) - "ANAM_AGENT_ENDPOINT": get_env_var('ANAM_AGENT_ENDPOINT', profile, - "https://api-test.agora.io/api/conversational-ai-agent/v2/projects"), - "ANAM_BASE_URL": get_env_var('ANAM_BASE_URL', profile, "https://api.anam.ai/v1"), + # MCP settings (JSON array of MCP server configs) + "MCP_SERVERS": get_env_var('MCP_SERVERS', profile), # Default prompt and messages "DEFAULT_PROMPT": get_env_var('DEFAULT_PROMPT', profile, diff --git a/simple-voice-client/README.md b/simple-voice-client-no-backend/README.md similarity index 100% rename from simple-voice-client/README.md rename to simple-voice-client-no-backend/README.md diff --git a/simple-voice-client/index.html b/simple-voice-client-no-backend/index.html similarity index 100% rename from simple-voice-client/index.html rename to simple-voice-client-no-backend/index.html diff --git a/complete-voice-client/README.md b/simple-voice-client-with-backend/README.md similarity index 100% rename from complete-voice-client/README.md rename to simple-voice-client-with-backend/README.md diff --git a/complete-voice-client/index.html b/simple-voice-client-with-backend/index.html similarity index 100% rename from complete-voice-client/index.html rename to simple-voice-client-with-backend/index.html