AgoraIO-Conversational-AI · BenWeekes · Apr 16, 2026 · Apr 16, 2026 · Apr 16, 2026 · Apr 16, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,41 @@
+# AI Agent Instructions
+
+This repository uses progressive disclosure documentation. Docs live under
+`docs/ai/` in three levels.
+
+## How to Load
+
+1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo.
+2. Load ALL 8 files in `docs/ai/L1/`. They are small — load all upfront.
+3. Follow L2 deep-dive links only when L1 isn't detailed enough.
+
+## Git Conventions
+
+### Commit messages — conventional commits
+
+- **Format:** `type: description` or `type(scope): description`
+- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation)
+- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `feat(auth): add token refresh`
+- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature`
+- **Present tense** — "add feature", not "added feature"
+
+### Branch names
+
+- **Format:** `type/short-description` — lowercase, hyphen-separated
+- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/`
+- **Examples:** `feat/token-refresh`, `fix/null-pointer`, `docs/progressive-disclosure`
+
+### General rules
+
+- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4
+- **No Co-Authored-By trailers** — omit AI attribution lines
+- **No --no-verify** — let git hooks run normally
+- **No git config changes** — do not modify user.name or user.email
+
+## Doc Commands
+
+| Command       | When to use                                  |
+| ------------- | -------------------------------------------- |
+| generate docs | no `docs/ai/` directory exists yet           |
+| update docs   | code changed since last `last_reviewed` date |
+| test docs     | verify docs give agents the right context    |
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -1 +1,3 @@
+Read @AGENTS.md for AI agent instructions, git conventions, and progressive disclosure docs.
+
 Read [AGENT.md](AGENT.md) for project context, configuration guides, and implementation details. Read [README.md](README.md) for architecture overview and getting started.
diff --git a/docs/ai/L0_repo_card.md b/docs/ai/L0_repo_card.md
@@ -0,0 +1,27 @@
+# agent-samples — Repo Card
+
+> Multi-app sample stack demonstrating Agora Conversational AI with Python backend, React voice/video clients, and extensible recipes.
+
+## Identity
+
+| Field         | Value                                        |
+| ------------- | -------------------------------------------- |
+| Repo          | `AgoraIO-Conversational-AI/agent-samples`    |
+| Type          | `distributed-system`                         |
+| Language      | Python (backend), TypeScript/React (clients) |
+| Deploy Target | PM2 + nginx (EC2), AWS Lambda (backend)      |
+| Owner         | Agora Conversational AI                      |
+| Last Reviewed | 2026-04-16                                   |
+
+## L1 — Summaries
+
+| File                                     | Purpose                                              | Audience       |
+| ---------------------------------------- | ---------------------------------------------------- | -------------- |
+| [01_setup](L1/01_setup.md)               | Environment setup, quick commands, env vars          | Use & Maintain |
+| [02_architecture](L1/02_architecture.md) | System design, component diagram, data flow          | Maintain       |
+| [03_code_map](L1/03_code_map.md)         | Directory tree, module map, "where does X live?"     | Maintain       |
+| [04_conventions](L1/04_conventions.md)   | Naming, patterns, error handling, testing            | Maintain       |
+| [05_workflows](L1/05_workflows.md)       | Step-by-step: add feature, deploy, configure         | Use            |
+| [06_interfaces](L1/06_interfaces.md)     | API contracts, message schemas, SDK integration      | Use & Maintain |
+| [07_gotchas](L1/07_gotchas.md)           | Critical gotchas, tribal knowledge, incident lessons | Maintain       |
+| [08_security](L1/08_security.md)         | Security model, trust boundaries, auth, secrets      | Maintain       |
diff --git a/docs/ai/L1/01_setup.md b/docs/ai/L1/01_setup.md
@@ -0,0 +1,94 @@
+# 01 Setup
+
+> Environment setup, dependencies, ports, and quick commands for the multi-app sample stack.
+
+## Prerequisites
+
+| Requirement | Version | Notes                                        |
+| ----------- | ------- | -------------------------------------------- |
+| Node.js     | ≥20     | Required for React clients; use `nvm use 20` |
+| Python      | 3.9+    | Required for backend                         |
+| npm         | ≥9      | Package manager for React clients            |
+| pip         | latest  | Python dependency management                 |
+
+- React clients use **npm** (not pnpm/yarn); always use `--legacy-peer-deps`
+- `package-lock.json` is gitignored in both React clients
+- Pre-commit hook runs Prettier on markdown — run `npx prettier --write` before committing `.md` files
+- Commit-msg hook blocks "claude" (case-insensitive) — omit AI tool names
+
+## Ports
+
+| App                       | Port | Profile |
+| ------------------------- | ---- | ------- |
+| simple-backend            | 8082 | —       |
+| react-voice-client        | 8083 | VOICE   |
+| react-video-client-avatar | 8084 | VIDEO   |
+
+## Quick Start
+
+### Backend
+
+```bash
+cd simple-backend
+python3 -m venv venv && source venv/bin/activate
+pip install -r requirements-local.txt
+cp .env.example .env  # edit with your Agora credentials
+python3 -u local_server.py
+```
+
+### Voice Client
+
+```bash
+cd react-voice-client
+npm install --legacy-peer-deps
+npm run dev
+```
+
+### Video Avatar Client
+
+```bash
+cd react-video-client-avatar
+npm install --legacy-peer-deps
+npm run dev
+```
+
+## Required Environment Variables (Backend)
+
+| Variable          | Purpose                    | Required? |
+| ----------------- | -------------------------- | --------- |
+| `APP_ID`          | Agora App ID               | Yes       |
+| `APP_CERTIFICATE` | Agora App Certificate      | Yes       |
+| `CUSTOMER_ID`     | Agora REST API customer ID | Yes       |
+| `CUSTOMER_SECRET` | Agora REST API secret      | Yes       |
+
+Profile-specific variables use the pattern `<PROFILE>_<VARIABLE>` (e.g., `VOICE_TTS_VENDOR=rime`).
+
+## Quick Commands
+
+| Command                      | Where          | What It Does               |
+| ---------------------------- | -------------- | -------------------------- |
+| `python3 -u local_server.py` | simple-backend | Start backend (unbuffered) |
+| `npm run dev`                | react clients  | Start Next.js dev server   |
+| `npm run build`              | react clients  | Production build           |
+| `pytest`                     | simple-backend | Run backend tests          |
+| `npx prettier --write .`     | any directory  | Format markdown files      |
+
+## PM2 Production Mode
+
+```bash
+pm2 start ecosystem.config.js
+```
+
+Starts all apps (backend + both clients) with correct ports and environment variables.
+
+## Common Setup Issues
+
+- **Python output buffering** — always use `python3 -u` or `PYTHONUNBUFFERED=1`; without this, logs silently buffer
+- **npm vs pnpm** — this repo uses npm, not pnpm; `pnpm install` will not work
+- **Node version** — Next.js 16 requires Node ≥20.9.0; use `nvm use 20`
+- **Missing --legacy-peer-deps** — React 19 + some packages need `--legacy-peer-deps`
+- **MLLM_LOCATION not MLLM_REGION** — Agora expects `MLLM_LOCATION` (e.g., `us-central1`)
+
+## Related Deep Dives
+
+- [Profile Configuration](L2/profile_configuration.md) — Profile-based env var system, MLLM setup, vendor config
diff --git a/docs/ai/L1/02_architecture.md b/docs/ai/L1/02_architecture.md
@@ -0,0 +1,98 @@
+# 02 Architecture
+
+> System design overview: multi-app stack with Python backend, React clients, and Agora Conversational AI integration.
+
+## High-Level Architecture
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│                     User's Browser                            │
+│                                                               │
+│  ┌─────────────────────┐     ┌──────────────────────────┐   │
+│  │ react-voice-client   │     │ react-video-client-avatar │   │
+│  │ (Next.js, port 8083) │     │ (Next.js, port 8084)      │   │
+│  │                      │     │                           │   │
+│  │ - Audio capture      │     │ - Audio + video capture   │   │
+│  │ - Voice visualization│     │ - Avatar rendering        │   │
+│  │ - Transcript display │     │ - Biomarker panels        │   │
+│  └──────────┬───────────┘     └──────────┬────────────────┘   │
+│             │                            │                    │
+│             └────────────┬───────────────┘                    │
+│                          │ HTTP                               │
+└──────────────────────────┼────────────────────────────────────┘
+                           │
+              ┌────────────▼────────────┐
+              │   simple-backend         │
+              │   (Flask, port 8082)     │
+              │                          │
+              │   - Token generation     │
+              │   - Agent lifecycle      │
+              │   - Profile config       │
+              │   - Optional auth        │
+              └────────────┬────────────┘
+                           │ HTTP
+              ┌────────────▼────────────┐
+              │   Agora ConvoAI API      │
+              │                          │
+              │   - Agent creation       │
+              │   - STT + LLM + TTS     │
+              │   - RTC audio transport  │
+              │   - RTM messaging        │
+              └─────────────────────────┘
+```
+
+## App Responsibilities
+
+| App                         | Language   | Role                                  |
+| --------------------------- | ---------- | ------------------------------------- |
+| `simple-backend`            | Python     | Token generation, agent orchestration |
+| `react-voice-client`        | TypeScript | Audio-only conversational UI          |
+| `react-video-client-avatar` | TypeScript | Video avatar conversational UI        |
+
+## Request Lifecycle
+
+```
+1. Client → GET /start-agent?channel=X&profile=VOICE
+2. Backend → initialize_constants("voice")  # load profile env vars
+3. Backend → build_token_with_rtm()          # v007 token generation
+4. Backend → create_agent_payload()          # assemble Agora API payload
+5. Backend → send_agent_to_channel()         # POST to Agora ConvoAI API
+6. Backend → return {token, uid, channel, appid, agent_response}
+7. Client → RTC join(channel, uid, token)    # audio/video connection
+8. Client → RTM subscribe(channel)           # transcript messages
+9. Client → AgoraVoiceAI.init()              # toolkit integration
+10. [Conversation flows via RTC audio + RTM messages]
+11. Client → GET /hangup-agent?agent_id=X
+12. Backend → POST to Agora leave API
+```
+
+## SDK Layer
+
+React clients depend on three npm packages:
+
+| Package                      | Purpose                       |
+| ---------------------------- | ----------------------------- |
+| `agora-rtc-sdk-ng`           | RTC audio/video transport     |
+| `agora-rtm`                  | Real-time messaging           |
+| `agora-agent-client-toolkit` | Conversational AI integration |
+| `@agora/agent-ui-kit`        | Pre-built UI components       |
+
+## Key Design Decisions
+
+- **Profile-based configuration** — backend loads env vars dynamically per request using `<PROFILE>_<VARIABLE>` pattern; supports unlimited profiles without code changes
+- **Stateless backend** — no database, no session storage (except optional auth); all state lives in Agora's infrastructure
+- **Async custom LLM registration** — `/register-agent` called in background thread; agent starts even if custom LLM is down
+- **Memory-only auth tokens** — JWT tokens held in React refs, not localStorage; clears on page refresh for security
+- **Vendor abstraction** — TTS, ASR, avatar, and MLLM vendors configured via env vars; backend builds vendor-specific payloads
+
+## Optional Extensions
+
+- **Custom LLM** — proxy LLM calls through your own server for augmentation
+- **MCP servers** — tool calling via Model Context Protocol
+- **Biomarkers** — Shen (video vitals, client-side) and Thymia (voice biomarkers, server-side)
+- **Auth** — Google OAuth + SMS 2FA + encrypted session memory
+
+## Related Deep Dives
+
+- [Profile Configuration](L2/profile_configuration.md) — Profile system, vendor configs, MLLM setup
+- [Agent Lifecycle](L2/agent_lifecycle.md) — Payload building, API calls, custom LLM registration
diff --git a/docs/ai/L1/03_code_map.md b/docs/ai/L1/03_code_map.md
@@ -0,0 +1,91 @@
+# 03 Code Map
+
+> Directory structure, module responsibilities, and where to find things in the multi-app repo.
+
+## Top-Level Layout
+
+```
+agent-samples/
+├── simple-backend/               # Python Flask backend (port 8082)
+│   ├── local_server.py           # Flask routes and server entry
+│   ├── lambda_handler.py         # AWS Lambda wrapper
+│   ├── core/                     # Business logic modules
+│   │   ├── config.py             # Profile-based env var loading
+│   │   ├── agent.py              # Payload building + Agora API calls
+│   │   ├── tokens.py             # v007 token generation (RTC+RTM)
+│   │   ├── auth.py               # Optional OAuth + SMS 2FA
+│   │   ├── consultant_dashboard.py  # Optional dashboard integration
+│   │   ├── phone_numbers.py      # Phone validation
+│   │   └── utils.py              # Shared utilities
+│   ├── templates/                # HTML templates for auth pages
+│   ├── tests/                    # Pytest test suite
+│   └── start.sh                  # PM2 wrapper script
+│
+├── react-voice-client/           # React voice UI (port 8083)
+│   ├── app/                      # Next.js app directory
+│   │   ├── page.tsx              # Main page
+│   │   ├── layout.tsx            # Root layout
+│   │   └── globals.css           # Global styles
+│   ├── components/
+│   │   ├── VoiceClient.tsx       # Main voice component
+│   │   └── ThemeToggle.tsx       # Dark/light mode
+│   ├── hooks/
+│   │   ├── useAgoraVoiceClient.ts     # RTC+RTM wrapper
+│   │   ├── useAudioVisualization.ts   # Volume bars animation
+│   │   ├── use-audio-devices.ts       # Mic selection
+│   │   ├── use-is-mobile.ts           # Responsive detection
+│   │   └── useAutoScroll.ts           # Transcript auto-scroll
+│   └── lib/theme/                # Theme utilities
+│
+├── react-video-client-avatar/    # React video avatar UI (port 8084)
+│   ├── (same structure as voice client)
+│   ├── components/
+│   │   └── VideoAvatarClient.tsx # Main video avatar component
+│   ├── hooks/
+│   │   ├── useAgoraVideoClient.ts     # RTC+RTM+video wrapper
+│   │   └── useShenai.ts               # Video biomarkers hook
+│   └── public/shenai-sdk/        # Shen.AI WASM SDK (vendored)
+│
+├── recipes/                      # Configuration recipes
+│   ├── therapist.md              # Wellness/biomarker demo
+│   └── thymia.md                 # Voice biomarker integration
+│
+├── design/                       # Design rationale documents
+│   ├── AI_SAMPLES_DESIGN.md
+│   └── AI_SAMPLES_UIKIT_TOOLKIT_DEV.md
+│
+├── simple-voice-client-no-backend/    # Vanilla JS client (no backend)
+├── simple-voice-client-with-backend/  # Vanilla JS client (with backend)
+├── ecosystem.config.js           # PM2 production config
+└── session-timeline.sh           # Operational debugging script
+```
+
+## Core Files by Task
+
+| Task                         | Start Here                                                   |
+| ---------------------------- | ------------------------------------------------------------ |
+| Change agent creation logic  | `simple-backend/core/agent.py`                               |
+| Add a new profile variable   | `simple-backend/core/config.py`                              |
+| Modify token generation      | `simple-backend/core/tokens.py`                              |
+| Add a backend route          | `simple-backend/local_server.py`                             |
+| Change voice UI              | `react-voice-client/components/VoiceClient.tsx`              |
+| Change video avatar UI       | `react-video-client-avatar/components/VideoAvatarClient.tsx` |
+| Modify RTC/RTM connection    | `*/hooks/useAgoraVoiceClient.ts` or `useAgoraVideoClient.ts` |
+| Add a new recipe             | `recipes/`                                                   |
+| Change production deployment | `ecosystem.config.js`                                        |
+| Update backend tests         | `simple-backend/tests/`                                      |
+
+## Backend Module Responsibilities
+
+| Module              | Responsibility                                      |
+| ------------------- | --------------------------------------------------- |
+| `config.py`         | Profile-based env var loading, defaults, validation |
+| `agent.py`          | Build Agora API payloads, send HTTP requests        |
+| `tokens.py`         | v007 token generation with RTC+RTM services         |
+| `auth.py`           | Google OAuth, SMS 2FA, encrypted session memory     |
+| `local_server.py`   | Flask routes, CORS, request handling                |
+| `lambda_handler.py` | AWS Lambda entry point (wraps Flask app)            |
+
+## Related Deep Dives
+
+- [Agent Lifecycle](L2/agent_lifecycle.md) — Payload building details, vendor-specific configs
Original file line number	Diff line number	Diff line change
		@@ -1 +1,3 @@
		Read @AGENTS.md for AI agent instructions, git conventions, and progressive disclosure docs.

		Read [AGENT.md](AGENT.md) for project context, configuration guides, and implementation details. Read [README.md](README.md) for architecture overview and getting started.