diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..621dc6d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,138 @@ +# Agent Development Guide + +For coding agents working in `recipe-client-rn-quickstart`. This repository is the +**React Native client** recipe in the Agora Conversational AI recipes family — a bare +React Native (TypeScript) app wired to a keyless Python FastAPI token backend. + +## How to Load + +This repository uses progressive disclosure documentation. Docs live under +`docs/ai/` in three levels. + +1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo. +2. This repo declares `Recipe Role: base`; read [docs/ai/RECIPE.md](docs/ai/RECIPE.md) before changing reusable recipe contracts. +3. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront. +4. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md). + +The sections below remain the canonical contributor handbook for hands-on work; +the `docs/ai/` tree is the structured summary used by AI agents. + +## System shape + +- **`rn/`** — React Native 0.86 / TypeScript 5 app. Two Agora engine adapters + (`RtcEngineAdapter`, `RtmEngineAdapter`) implement the `agora-agent-client-toolkit` + interfaces over `react-native-agora` (RTC) and `agora-react-native-rtm` (RTM 2.x). + `AgoraSession` owns session lifecycle; `useCallStore` owns all React state. +- **`server/`** — Python FastAPI token service (:8000). Owns Agora token generation and + a keyless cascading agent pipeline: `DeepgramSTT(nova-3)` → `OpenAI` (Agora-managed) + → `MiniMaxTTS`. SDK: `agora-agents>=2.3.0`. No `OPENAI_API_KEY` required by default. +- Auth: Token007 from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` in the backend. +- Transcript delivery: `data_channel="rtm"` with `advanced_features={"enable_rtm": True}`. + +## Session ordering + +`RTM login → RTC join → AgoraVoiceAI.init → subscribeMessage → subscribeChannel → /startAgent` + +Both `ai.subscribeMessage(channel)` **and** `rtm.subscribeChannel(channel)` must run before +`/startAgent`. The TS toolkit's `subscribeMessage` only binds listeners; RTM 2.x requires +an explicit channel subscribe to deliver messages. + +## Routing / ownership + +- UI, RTC/RTM lifecycle, and session orchestration live in `rn/src/`. +- Token generation and agent lifecycle live in `server/src/`. +- The React Native app calls the backend directly over HTTP (no rewrite proxy — this is a + native app, not a browser). Backend URL is hardcoded in `rn/src/config.ts`. +- Android emulator reaches the host backend at `10.0.2.2:8000`; iOS Simulator uses `localhost:8000`. + +## Supported modes + +- **Local:** start `server/` manually (`python src/server.py`) then run the RN app + (`npx react-native run-android` or `run-ios`). +- **Deploy:** host the server behind HTTPS; update `AGENT_BACKEND_URL` in `rn/src/config.ts` + and use a real device. A backend image is published to + `ghcr.io/AgoraIO-Conversational-AI/recipe-client-rn-quickstart` on `v*` tags. + +## Env vars + +| Variable | Default | Notes | +|------------------------|------------------------|-------| +| `AGORA_APP_ID` | — | **required** | +| `AGORA_APP_CERTIFICATE`| — | **required** — server-only, never in the app | +| `OPENAI_MODEL` | `gpt-4o-mini` | Agora-managed vendor model | +| `OPENAI_API_KEY` | — (keyless) | Optional BYO; Agora manages by default | +| `AGENT_GREETING` | built-in line | Override the opening utterance | + +## Patterns + +- Keep `AGORA_APP_CERTIFICATE` in `server/.env.local`; never pass it to the RN app. +- Keep `data_channel="rtm"` and `advanced_features={"enable_rtm": True}` in the backend agent. +- Always call both `ai.subscribeMessage(channel)` and `rtm.subscribeChannel(channel)` before `/startAgent`. +- Use the `(turnId, type)` composite key for transcript upserts — `CallState.ts` owns this. + +## Anti-patterns + +- Do not embed `AGORA_APP_CERTIFICATE` or `OPENAI_API_KEY` in `rn/src/config.ts`, `app.json`, or native plists/manifests. +- Do not remove `data_channel="rtm"` or `enable_rtm: True` — transcript delivery will silently stop. +- Do not call `/startAgent` before `subscribeChannel` resolves — the agent greeting will be missed. +- Do not skip `pod install` after `npm install` changes native packages on iOS. + +## Commands + +```bash +# Backend +cd server && python src/server.py + +# RN app +cd rn && npx react-native run-android # Android +cd rn && npx react-native run-ios # iOS (requires pod install first) + +# Tests +cd rn && npx tsc --noEmit # TypeScript check (no toolchain needed) +cd rn && npm test # Jest unit tests (no server needed) +cd rn/android && ./gradlew assembleDebug # native Android build +cd server && pytest -q # backend unit tests (no cloud, no creds) +``` + +## Done criteria + +1. Run `npx tsc --noEmit` (always) and `npm test` for any RN source change. +2. For native dependency changes: run `pod install` (iOS) or `./gradlew assembleDebug` (Android). +3. For backend changes: run `cd server && pytest -q`. +4. If you change required env vars or setup steps, update the root README, `server/README.md`, and `server/.env.example` together. +5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md). + +## Git Conventions + +### Commit messages — conventional commits + +- **Format:** `type: description` or `type(scope): description` +- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation) +- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `fix(rn): correct rtm subscribe order` +- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature` +- **Present tense** — "add feature", not "added feature" + +### Branch names + +- **Format:** `type/short-description` — lowercase, hyphen-separated +- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/` +- **Examples:** `feat/add-settings-screen`, `fix/rtm-subscribe-order`, `docs/progressive-disclosure` + +### General rules + +- **Repo-local `AGENTS.md` is the authoritative source for repo conventions.** +- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions. +- **No Co-Authored-By trailers** — omit AI attribution lines. +- **No `--no-verify`** — let git hooks run normally. +- **No git config changes** — do not modify `user.name` or `user.email`. + +## Doc Commands + +| Command | When to use | +| ------------- | ---------------------------------------------------------------------------- | +| generate docs | No `docs/ai/` directory exists yet | +| update docs | Code changed since the `Last Reviewed` date in L0 | +| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) | +| fix docs | Close findings from a docs review or test run | + +See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/standard/progressive-disclosure-standard.md) and [workflows](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/workflows/progressive-disclosure-docs.md) for the full specification. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..eea2d6d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +This project uses AGENTS.md instead of a CLAUDE.md file. + +Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project. diff --git a/docs/ai/L0_repo_card.md b/docs/ai/L0_repo_card.md new file mode 100644 index 0000000..ee9e420 --- /dev/null +++ b/docs/ai/L0_repo_card.md @@ -0,0 +1,36 @@ +# recipe-client-rn-quickstart — Repo Card + +> Bare React Native (TypeScript) voice-agent quickstart: two Agora engine adapters wire `agora-agent-client-toolkit` to `react-native-agora` (RTC) and `agora-react-native-rtm` (RTM), backed by a keyless Python FastAPI token service. + +## Identity + +| Field | Value | +| -------------- | ------------------------------------------------------------------------------------- | +| Repo | `AgoraIO-Conversational-AI/recipe-client-rn-quickstart` | +| Type | `frontend-app` (React Native client + bundled token backend) | +| Language | React Native 0.86 / TypeScript 5 + Python 3.12 (FastAPI token service) | +| Deploy Target | iOS Simulator / Android emulator or device; server/ as standalone FastAPI service | +| Owner | Agora Conversational AI DevEx | +| Last Reviewed | 2026-06-25 | +| Recipe Role | `base` | +| Recipe Version | `1.0.0` | +| Recipe Status | `experimental` | + +## L1 — Summaries + +The Audience column helps agents prioritise: **Use** = consuming the recipe's behavior, **Maintain** = modifying internals. + +| File | Purpose | Audience | +| ---------------------------------------- | ------------------------------------------------------------------------------------- | -------------- | +| [01_setup](L1/01_setup.md) | Node + npm + iOS/Android toolchain, backend venv/uv, env vars, build/run commands | Use & Maintain | +| [02_architecture](L1/02_architecture.md) | App → token service → Agora ConvoAI topology, session lifecycle | Maintain | +| [03_code_map](L1/03_code_map.md) | `rn/src/`, `rn/ios/`, `rn/android/`, `server/src/` layout and key file duties | Maintain | +| [04_conventions](L1/04_conventions.md) | RN/TypeScript idioms, custom hook state, adapter pattern, Python backend conventions | Maintain | +| [05_workflows](L1/05_workflows.md) | Add a screen, change token endpoint, run on iOS/Android, toggle mic, run tests | Use | +| [06_interfaces](L1/06_interfaces.md) | Token-service API contract, `AgentConfig` shape, toolkit event surface | Use & Maintain | +| [07_gotchas](L1/07_gotchas.md) | Native autolinking, Pods, Android emulator host alias, RTM subscribe order | Maintain | +| [08_security](L1/08_security.md) | Token handling, App Certificate server-only, no secrets in the RN bundle | Maintain | + +## Recipe Profile + +This repo declares `Recipe Role: base`. See [RECIPE.md](RECIPE.md) for extension points, invariants, and stable contracts before changing reusable surfaces. diff --git a/docs/ai/L1/01_setup.md b/docs/ai/L1/01_setup.md new file mode 100644 index 0000000..b279fb7 --- /dev/null +++ b/docs/ai/L1/01_setup.md @@ -0,0 +1,87 @@ +# 01 · Setup + +> Install the React Native toolchain, the Python token backend, configure env vars, and run the app on Android or iOS. + +## Prerequisites + +| Tool | Minimum | Notes | +|------|---------|-------| +| Node.js | 22.11.0 | enforced in `rn/package.json` `engines` field | +| npm | bundled with Node | used in `rn/`; npm ci in CI | +| Java (JDK) | 21 (temurin) | Android build (`assembleDebug`) | +| Android SDK | API 33+ | via `android-actions/setup-android` in CI; locally via Android Studio | +| Xcode | latest | iOS Simulator; macOS only | +| CocoaPods | system | iOS native dependency install (`pod install`) | +| Python | 3.12 recommended (3.10+ supported) | backend venv | +| uv | any | recommended for venv/install; `pip` also works | + +Agora App ID and App Certificate are required (no third-party LLM key — the pipeline is Agora-managed/keyless). + +## Install — backend + +```bash +cd server +uv venv venv && . venv/bin/activate +uv pip install -r requirements.txt -r requirements-dev.txt +``` + +Copy env template and fill credentials: + +```bash +cp .env.example .env.local +# edit .env.local: set AGORA_APP_ID and AGORA_APP_CERTIFICATE +``` + +## Install — React Native app + +```bash +cd rn +npm install + +# iOS only — install native pods +cd ios && pod install && cd .. +``` + +## Configure env — backend + +Backend env file: `server/.env.local` (template: `server/.env.example`). + +| Variable | Required | Default | Notes | +| ---------------------- | :------: | ------------------------------------ | ---------------------------------------------- | +| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID | +| `AGORA_APP_CERTIFICATE`| ✅ | — | Agora Console → Project → App Certificate | +| `OPENAI_MODEL` | | `gpt-4o-mini` | OpenAI model used by the Agora-managed vendor | +| `OPENAI_API_KEY` | | — (keyless by default) | BYO only — Agora manages the key by default | +| `AGENT_GREETING` | | `"Hi there! How can I help you today?"` | Override the agent's opening line | + +## Configure — app backend URL + +`rn/src/config.ts` exports `AGENT_BACKEND_URL`. Default is `http://10.0.2.2:8000` (Android emulator host alias for `localhost`). For the iOS Simulator change to `http://localhost:8000`. + +## Run + +```bash +# Terminal 1 — backend +cd server && python src/server.py # binds 0.0.0.0:8000 + +# Terminal 2 — Android +cd rn && npx react-native run-android # starts Metro + installs on emulator/device + +# Terminal 2 — iOS +cd rn && npx react-native run-ios # requires macOS + Xcode + pod install done +``` + +Grant microphone permission when prompted, then tap **Connect**. + +## Quick test commands + +```bash +cd rn && npx tsc --noEmit # TypeScript compile check (no native toolchain needed) +cd rn && npm test # Jest unit tests (mocked fetch — no server needed) +cd rn/android && ./gradlew assembleDebug # native Android build smoke +cd server && pytest -q # backend unit tests (no cloud, no creds) +``` + +## Related Deep Dives + +- [native_dependency_setup.md](L2/native_dependency_setup.md) — CocoaPods, Android autolinking, and Pods troubleshooting. diff --git a/docs/ai/L1/02_architecture.md b/docs/ai/L1/02_architecture.md new file mode 100644 index 0000000..c7e7294 --- /dev/null +++ b/docs/ai/L1/02_architecture.md @@ -0,0 +1,62 @@ +# 02 · Architecture + +> React Native app calls the token backend directly over HTTP, then uses two Agora engine adapters to join both RTC (audio) and RTM (transcript) channels via `agora-agent-client-toolkit`. + +## Topology + +``` +React Native App (rn/) + │ HTTP fetch + ▼ +Token Backend (server/, :8000) + │ GET /get_config → returns app_id, token, uid, channelName, agentUid + │ POST /startAgent → starts the Agora ConvoAI agent in the channel + │ POST /stopAgent → stops the agent + ▼ +Agora ConvoAI Cloud + │ user audio (RTC) → DeepgramSTT → OpenAI (Agora-managed) → MiniMaxTTS + │ agent audio (RTC) → back to user's channel + │ transcript (RTM) → MessageEvent delivered to RtmEngineAdapter + ▼ +agora-agent-client-toolkit (AgoraVoiceAI) + │ TRANSCRIPT_UPDATED → CallState.useCallStore → CallScreen FlatList + │ AGENT_STATE_CHANGED → CallState.agentState badge +``` + +## Key components + +| Component | Location | Role | +|-----------|----------|------| +| `BackendApi` | `rn/src/BackendApi.ts` | HTTP fetch wrapper for the three token-service endpoints | +| `AgoraSession` | `rn/src/agora/AgoraSession.ts` | Orchestrates `RtcEngineAdapter` + `RtmEngineAdapter` + `AgoraVoiceAI`; owns session lifecycle | +| `RtcEngineAdapter` | `rn/src/agora/RtcEngineAdapter.ts` | Adapts `react-native-agora` `IRtcEngine` to the toolkit's `RTCEngine` interface | +| `RtmEngineAdapter` | `rn/src/agora/RtmEngineAdapter.ts` | Adapts `agora-react-native-rtm` `RTMClient` to the toolkit's `RTMEngine` interface; translates native `MessageEvent` shape | +| `useCallStore` | `rn/src/CallState.ts` | React custom hook owning `phase`, `turns`, `agentState`, `micMuted`; drives `AgoraSession` | +| `LandingScreen` | `rn/src/ui/LandingScreen.tsx` | Pre-call UI; requests Android microphone permission | +| `CallScreen` | `rn/src/ui/CallScreen.tsx` | In-call UI: transcript `FlatList` + mute/end controls | +| `server/src/agent.py` | `server/` | Agora-managed cascading pipeline (Deepgram STT → OpenAI → MiniMaxTTS) via `agora-agents>=2.3.0` | + +## Session lifecycle + +1. `useCallStore.connect()` → `BackendApi.getConfig()` → receives `{appId, token, uid, channelName, agentUid}`. +2. `AgoraSession.start(config)` → `RtmEngineAdapter.login(token)` → `RtcEngineAdapter.join(...)` (mic published, role Broadcaster). +3. `AgoraVoiceAI.init({ rtcEngine, rtmConfig, renderMode: TEXT })`. +4. `ai.subscribeMessage(channelName)` binds toolkit listeners; `rtm.subscribeChannel(channelName)` activates RTM 2.x delivery — **both required, in this order**. +5. `BackendApi.startAgent(channelName, agentUid, uid)` → Agora ConvoAI cloud starts the agent. +6. Agent greets user; RTM `MessageEvent` → `RtmEngineAdapter` → `AgoraVoiceAI` → `TRANSCRIPT_UPDATED` → `CallScreen` re-renders. +7. `end()` → `stopAgent(agentId)` → `AgoraSession.stop()` (RTM unsubscribe + logout, RTC leave). + +## Pipeline (server side) + +`DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless) → `MiniMaxTTS` — no `llm/` service; single-process cascade. Data channel is `"rtm"` with `enable_rtm: True`; transcript/state/metrics ride RTM. + +## Tech decisions + +- **Engine adapters** — `agora-agent-client-toolkit` is engine-agnostic; adapters translate native RN SDK event shapes to the toolkit's `RTCEngine`/`RTMEngine` interfaces. +- **RTM 2.x explicit subscribe** — the TS toolkit's `subscribeMessage` only binds listeners; `client.subscribe(channel)` must be called separately (unlike the native iOS/Android toolkits which subscribe internally). +- **Keyless pipeline** — no third-party LLM key required; Agora manages the OpenAI vendor. `OPENAI_API_KEY` is optional BYO. +- **Android emulator alias** — `10.0.2.2` is the emulator's route to the host `localhost:8000`. + +## Related Deep Dives + +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — RTM event translation, presence-state synthesis, subscribe ordering. diff --git a/docs/ai/L1/03_code_map.md b/docs/ai/L1/03_code_map.md new file mode 100644 index 0000000..78471f4 --- /dev/null +++ b/docs/ai/L1/03_code_map.md @@ -0,0 +1,79 @@ +# 03 · Code Map + +> Directory layout for `rn/` (React Native app) and `server/` (Python token service). + +## Top-level layout + +``` +recipe-client-rn-quickstart/ +├── rn/ # React Native app (TypeScript) +├── server/ # Python FastAPI token/agent backend +├── Dockerfile # builds the server/ backend image +└── .github/workflows/ + ├── ci.yml # server pytest + RN tsc/jest/assembleDebug + └── docker.yml # build + smoke-test the backend image +``` + +## `rn/` — React Native app + +``` +rn/ +├── App.tsx # root component; renders LandingScreen or CallScreen +├── index.js # RN entry point (registers AppRegistry) +├── src/ +│ ├── config.ts # AGENT_BACKEND_URL + AUTO_CONNECT flag +│ ├── BackendApi.ts # HTTP fetch wrapper (getConfig, startAgent, stopAgent) +│ ├── CallState.ts # useCallStore custom hook — phase, turns, agentState, micMuted +│ ├── agora/ +│ │ ├── AgoraSession.ts # session orchestrator (RTM login → RTC join → toolkit init → subscribe) +│ │ ├── RtcEngineAdapter.ts # react-native-agora → toolkit RTCEngine adapter +│ │ └── RtmEngineAdapter.ts # agora-react-native-rtm → toolkit RTMEngine adapter + event translation +│ └── ui/ +│ ├── LandingScreen.tsx # pre-call screen; requests Android RECORD_AUDIO permission +│ └── CallScreen.tsx # in-call screen: agent state badge, transcript FlatList, mute/end +├── __tests__/ +│ └── BackendApi.test.ts # Jest unit test (mocked fetch) for envelope decoding +├── android/ +│ └── app/src/main/ +│ └── AndroidManifest.xml # INTERNET + RECORD_AUDIO permissions +├── ios/ +│ ├── Podfile # CocoaPods: use_native_modules! + react_native_post_install +│ └── RnQuickstart.xcodeproj/ +├── package.json # deps: react-native-agora, agora-react-native-rtm, agora-agent-client-toolkit +├── tsconfig.json # extends @react-native/typescript-config +├── metro.config.js # default Metro config +└── babel.config.js +``` + +## `server/` — Python FastAPI token service + +``` +server/ +├── src/ +│ ├── server.py # FastAPI app; GET /get_config, POST /startAgent, POST /stopAgent +│ └── agent.py # Agent class; cascading Deepgram→OpenAI→MiniMax pipeline; sessions dict +├── tests/ +│ ├── conftest.py # fake env fixture (AGORA_APP_ID, AGORA_APP_CERTIFICATE) +│ ├── test_config.py # basic Agent construction smoke +│ └── test_agent_construction.py # AgoraAgent construction + FakeSession integration +├── scripts/ +│ └── run_fake_server.py # local dev helper (not CI gate) +├── requirements.txt # fastapi, uvicorn, agora-agents>=2.3.0, python-dotenv, socksio +├── requirements-dev.txt # pytest +└── .env.example # template: AGORA_APP_ID, AGORA_APP_CERTIFICATE, optional vars +``` + +## Key file responsibilities + +| File | What it owns | +|------|-------------| +| `rn/src/agora/RtmEngineAdapter.ts` | Native `MessageEvent` → toolkit `{publisher, messageType, message}` translation; presence `stateItems[]` → `stateChanged` synthesis | +| `rn/src/agora/AgoraSession.ts` | RTM-before-RTC ordering; `subscribeMessage` + `subscribeChannel` ordering pre-`startAgent` | +| `rn/src/CallState.ts` | Turn upsert by `(turnId, type)` composite key; all React state | +| `server/src/agent.py` | Agora-managed STT/LLM/TTS cascade; `_sessions` dict for stop-by-id; `data_channel="rtm"` | +| `server/src/server.py` | Token generation (`generate_convo_ai_token`), route dispatch, error mapping | + +## Related Deep Dives + +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — adapter internals in detail. +- [native_dependency_setup.md](L2/native_dependency_setup.md) — Pods and Android autolinking. diff --git a/docs/ai/L1/04_conventions.md b/docs/ai/L1/04_conventions.md new file mode 100644 index 0000000..ab8f6e2 --- /dev/null +++ b/docs/ai/L1/04_conventions.md @@ -0,0 +1,42 @@ +# 04 · Conventions + +> TypeScript/React Native idioms, the adapter pattern, React state conventions, and Python backend patterns. + +## React Native / TypeScript + +| Convention | Detail | +|------------|--------| +| Single entry custom hook | `useCallStore()` in `CallState.ts` owns all session state and returns a `CallStore` type; screens receive only what they display | +| Phase enum | `Phase = 'idle' \| 'connecting' \| 'inCall' \| 'error'` — `App.tsx` switches on `phase` to mount `LandingScreen` or `CallScreen` | +| Transcript composite key | Turns are upserted by `` `${turnId}-${type}` `` (Map keyed by this string); `type = 'user' \| 'agent'` discriminated from `item.metadata.object` | +| `useRef` for objects | `apiRef`, `sessionRef`, `agentIdRef` use `useRef` — stable across re-renders, not reactive | +| Android permission before connect | `LandingScreen` calls `PermissionsAndroid.request(RECORD_AUDIO)` on Connect press; iOS is handled by native Info.plist | +| `DIAG` log prefix | Diagnostic `console.log` lines use `'DIAG'` prefix throughout; safe to remove in production | +| `AUTO_CONNECT` flag | `src/config.ts` — `false` by default; set `true` for CI logcat gating without manual tap | + +## Adapter pattern + +Each native Agora SDK has a different event / method shape. The adapters (`RtcEngineAdapter`, `RtmEngineAdapter`) wrap the native SDK and implement the toolkit's `RTCEngine` / `RTMEngine` interfaces. This keeps `AgoraSession` and the toolkit free from native SDK specifics. + +| Interface | Adapter | Native SDK | +|-----------|---------|------------| +| `RTCEngine` | `RtcEngineAdapter` | `react-native-agora` (`IRtcEngine`) | +| `RTMEngine` | `RtmEngineAdapter` | `agora-react-native-rtm` (`RTMClient`) | + +- **Bridged listener map** — `RtmEngineAdapter` stores `toolkitListener → nativeListener` in `bridged` so `removeEventListener` can deregister the exact native closure. +- **No toolkit join** — the toolkit does not join channels; `AgoraSession` calls `rtc.join(...)` directly. + +## Python backend + +| Convention | Detail | +|------------|--------| +| `{ code, msg, data }` envelope | All success responses; `data` present only when there is a payload | +| Error mapping | `ValueError→400`, `RuntimeError→500`, else 500 via `_to_http_error` | +| `Agent()` raises on missing creds | `__init__` raises `ValueError` if `AGORA_APP_ID`/`AGORA_APP_CERTIFICATE` absent; server catches and sets `agent = None` | +| `_sessions` in-memory dict | Maps `agent_id → session`; `stop()` falls back to `client.stop_agent(agent_id)` if the session is missing (restart resilience) | +| camelCase request fields | `StartAgentRequest` uses `channelName`, `rtcUid`, `userUid` to match the RN client's JSON | +| `data_channel: "rtm"` | Always set; required for transcript delivery to the toolkit | + +## Related Deep Dives + +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — bridged listener map and event translation in depth. diff --git a/docs/ai/L1/05_workflows.md b/docs/ai/L1/05_workflows.md new file mode 100644 index 0000000..580b61e --- /dev/null +++ b/docs/ai/L1/05_workflows.md @@ -0,0 +1,98 @@ +# 05 · Workflows + +> Common modification workflows: add a screen, change the token endpoint, run on each platform, toggle mic, run tests. + +## Add a new screen + +1. Create `rn/src/ui/YourScreen.tsx` as a functional component, props-only (no local state). +2. Add a new `Phase` value to `Phase` in `CallState.ts` if the screen corresponds to a new call state. +3. Extend `useCallStore()` with any actions the screen needs. +4. In `App.tsx` `AppContent`, add a branch in the `inCall`/else switch to mount your screen. +5. Run `npx tsc --noEmit` to verify. + +## Change the token endpoint URL + +Edit `rn/src/config.ts`: +```ts +export const AGENT_BACKEND_URL = 'http://:8000'; +``` +- Android emulator → host machine: `http://10.0.2.2:8000` +- iOS Simulator → host machine: `http://localhost:8000` +- Physical device → use the host machine's LAN IP + +## Run on Android + +```bash +# Terminal 1 +cd server && python src/server.py + +# Terminal 2 — emulator or USB device +cd rn && npx react-native run-android +``` + +Ensure an emulator is running (`emulator -list-avds`) or a USB device is connected (`adb devices`). + +## Run on iOS (macOS only) + +```bash +# Install pods (first time or after npm install) +cd rn/ios && pod install + +# Terminal 1 +cd server && python src/server.py + +# Terminal 2 +cd rn && npx react-native run-ios +``` + +Change `AGENT_BACKEND_URL` to `http://localhost:8000` in `rn/src/config.ts` for the Simulator. + +## Toggle microphone + +`CallScreen` shows a **Mute / Unmute** button calling `onToggleMic`. `useCallStore.toggleMic()` calls `sessionRef.current.setMicMuted(next)` → `RtcEngineAdapter.setMicMuted(muted)` → `engine.muteLocalAudioStream(muted)`. The `micMuted` state drives the button label. + +## Run TypeScript check (no toolchain needed) + +```bash +cd rn && npx tsc --noEmit +``` + +Pins Agora SDK + toolkit signatures; runs in CI without Android/iOS toolchain. + +## Run Jest unit tests + +```bash +cd rn && npm test +``` + +Tests in `__tests__/BackendApi.test.ts` mock `fetch`; no server or device needed. + +## Run backend tests (no cloud, no creds) + +```bash +cd server && pytest -q +``` + +`conftest.py` provides `fake_env` fixture; `test_agent_construction.py` stubs the SDK session. + +## Run Android native build (CI gate) + +```bash +cd rn/android && ./gradlew assembleDebug +``` + +Requires Java 21 and Android SDK. Validates that native autolinking resolves (`react-native-agora`, `agora-react-native-rtm`). + +## Change agent greeting or pipeline + +Edit `server/.env.local` (or env vars): +- `AGENT_GREETING` — override the opening line +- `OPENAI_MODEL` — change the LLM model (default `gpt-4o-mini`) +- `OPENAI_API_KEY` — provide a BYO key (optional; Agora-managed by default) + +To change STT/LLM/TTS vendors, edit `server/src/agent.py` in the `Agent.start()` method where `stt`, `llm`, `tts` are constructed. + +## Related Deep Dives + +- [native_dependency_setup.md](L2/native_dependency_setup.md) — Pods and autolinking troubleshooting. +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — session start/stop ordering. diff --git a/docs/ai/L1/06_interfaces.md b/docs/ai/L1/06_interfaces.md new file mode 100644 index 0000000..1ce6ef9 --- /dev/null +++ b/docs/ai/L1/06_interfaces.md @@ -0,0 +1,88 @@ +# 06 · Interfaces + +> Token-service API contract, `AgentConfig` shape, `BackendApi` client, Agora toolkit event surface, and env vars. + +## Token-service API (server/, port 8000) + +The RN app calls these directly (no rewrite proxy — this is a native app, not a browser). + +### `GET /get_config` + +- Query (optional): `channel?: string`, `uid?: int` (≤ 0 or missing → server generates one). +- Returns `data`: `{ app_id, token, uid (string), channel_name, agent_uid (string) }`. +- Token is a Token007 RTC+RTM token, expiry 3600 s, for a concrete non-zero UID. + +### `POST /startAgent` + +- Body: `{ channelName: string, rtcUid: int, userUid: int, parameters?: object }`. + - `parameters.output_audio_codec?: string` is the only honored parameter field. +- Returns `data`: `{ agent_id, channel_name, status: "started" }`. +- Returns 500 if `AGORA_APP_ID`/`AGORA_APP_CERTIFICATE` are absent (`agent` is `None`). + +### `POST /stopAgent` + +- Body: `{ agentId: string }`. +- Returns `{ code: 0, msg: "success" }` (no `data`). + +## Response envelope + +```json +{ "code": 0, "msg": "success", "data": { ... } } +``` + +`data` is omitted when the route has no payload. Non-zero `code` = error. + +## `AgentConfig` (RN app) + +Returned by `BackendApi.getConfig()`: + +```ts +interface AgentConfig { + appId: string; + token: string; + uid: string; // user UID as string + channelName: string; + agentUid: string; // agent UID as string +} +``` + +## `BackendApi` client (`rn/src/BackendApi.ts`) + +| Method | Call | +|--------|------| +| `getConfig()` | `GET /get_config?uid=0` → `AgentConfig` | +| `startAgent(channelName, rtcUid, userUid)` | `POST /startAgent` → `agent_id: string` | +| `stopAgent(agentId)` | `POST /stopAgent` → `void` | + +## Toolkit events (`agora-agent-client-toolkit`) + +`AgoraVoiceAI` emits these events via `AgoraVoiceAIEvents`: + +| Event | Payload | Consumed by | +|-------|---------|-------------| +| `TRANSCRIPT_UPDATED` | `TranscriptHelperItem[]` | `AgoraSession` → `useCallStore.applyTranscript` | +| `AGENT_STATE_CHANGED` | `(agentUserId: string, event: StateChangeEvent)` | `useCallStore` → `agentState` | +| `AGENT_ERROR` | `(agentUserId: string, error: Error)` | `useCallStore` → `error` state | + +`AgoraVoiceAI.init()` parameters: + +| Field | Value | +|-------|-------| +| `rtcEngine` | `RtcEngineAdapter` instance | +| `rtmConfig.rtmEngine` | `RtmEngineAdapter` instance | +| `renderMode` | `TranscriptHelperMode.TEXT` | + +## Environment variables — backend + +| Variable | Required | Default | Notes | +| ---------------------- | :------: | ------------- | -------------------------------------------- | +| `AGORA_APP_ID` | ✅ | — | Agora project App ID | +| `AGORA_APP_CERTIFICATE`| ✅ | — | Agora project App Certificate | +| `OPENAI_MODEL` | | `gpt-4o-mini` | OpenAI model for Agora-managed vendor | +| `OPENAI_API_KEY` | | — (keyless) | Optional BYO; Agora manages key by default | +| `AGENT_GREETING` | | built-in | Override opening utterance | +| `PORT` | | `8000` | Server port (env only; do not add to .env.example) | + +## Related Deep Dives + +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — `RTMEngine` contract in depth. diff --git a/docs/ai/L1/07_gotchas.md b/docs/ai/L1/07_gotchas.md new file mode 100644 index 0000000..bdd4c30 --- /dev/null +++ b/docs/ai/L1/07_gotchas.md @@ -0,0 +1,61 @@ +# 07 · Gotchas + +> Non-obvious platform pitfalls: native linking, Pods, Android emulator networking, RTM subscribe ordering, and backend quirks. + +## RTM 2.x requires an explicit channel subscribe + +The TypeScript toolkit's `subscribeMessage(channelName)` only registers event listeners — it does **not** subscribe the RTM channel. Unlike the native iOS/Android toolkits (which subscribe internally), the TS version on React Native requires: + +```ts +ai.subscribeMessage(channelName); // bind listeners first +await rtm.subscribeChannel(channelName); // then activate delivery +``` + +Reversing the order or omitting `subscribeChannel` silently drops all transcript messages. + +## Subscribe BEFORE `/startAgent` + +`AgoraSession.start()` calls `subscribeMessage` + `subscribeChannel` before returning, so the caller must fire `startAgent` only after `session.start()` resolves. The agent's greeting arrives immediately on join; missing the subscribe means missing the first transcript turn. + +## Android emulator host alias + +The backend binds `0.0.0.0:8000` on the host machine. From inside an Android emulator, `localhost` and `127.0.0.1` refer to the emulator itself — not the host. Use `10.0.2.2` (the emulator's route to the host loopback). This is hardcoded in `rn/src/config.ts`. For an iOS Simulator, `localhost` works because the Simulator shares the host network stack. + +## iOS CocoaPods must be run after `npm install` + +`react-native-agora` and `agora-react-native-rtm` ship native code installed via CocoaPods. After any `npm install` that changes these packages, run: + +```bash +cd rn/ios && pod install +``` + +Failing to do so results in missing native modules at runtime (`RCTBridge: Module 'AgoraRtcNg' could not be found`). There is no `pod install` equivalent for Android — autolinking handles it at Gradle build time. + +## Android autolinking requires a fresh Gradle build + +Autolinking for `react-native-agora` and `agora-react-native-rtm` is applied at `./gradlew assembleDebug` time. If you install a new version of either package: + +```bash +cd rn/android && ./gradlew clean && ./gradlew assembleDebug +``` + +## `RECORD_AUDIO` permission on Android + +The `LandingScreen` calls `PermissionsAndroid.request(RECORD_AUDIO)` at connect time. If the user denies it, `RtcEngineAdapter.join()` still calls `engine.enableAudio()` and `joinChannel()`, but no audio will be captured. Guard on the return value of `ensureMicPermission()` (the screen already does this). + +## Presence `stateItems` shape difference + +The agent publishes its state (e.g. `state: "speaking"`) as RTM presence state. The RN RTM SDK delivers this as `PresenceEvent.stateItems: { key, value }[]`, not as a flat object. `RtmEngineAdapter.makeNativeListener('presence')` folds `stateItems` into a map and synthesizes `stateChanged: { state, turn_id }` — the shape the toolkit's `_handleRtmPresence` expects. Do not bypass this translation. + +## `agent` is `None` after server startup if env vars are missing + +`server.py` catches `ValueError` from `Agent()` and sets `agent = None`. All routes guard `if agent is None → 500`. The server still starts but every API call returns 500. Check `AGORA_APP_ID` / `AGORA_APP_CERTIFICATE` in `.env.local`. + +## Cleartext traffic on Android + +`AndroidManifest.xml` sets `android:usesCleartextTraffic="${usesCleartextTraffic}"`. In debug builds (`build.gradle` defaultConfig) this is `true`, allowing HTTP to `10.0.2.2:8000`. In release builds use HTTPS or configure a Network Security Config. + +## Related Deep Dives + +- [native_dependency_setup.md](L2/native_dependency_setup.md) — detailed CocoaPods and autolinking resolution. +- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — subscribe ordering and event translation. diff --git a/docs/ai/L1/08_security.md b/docs/ai/L1/08_security.md new file mode 100644 index 0000000..6675af5 --- /dev/null +++ b/docs/ai/L1/08_security.md @@ -0,0 +1,45 @@ +# 08 · Security + +> Token handling, App Certificate server-side enforcement, no secrets in the RN bundle, microphone permissions, and CORS. + +## App Certificate stays in the backend + +`AGORA_APP_CERTIFICATE` is used only in `server/src/server.py` to call `generate_convo_ai_token()`. It is never passed to the React Native app. The app receives only the derived, short-lived Token007 (expiry 3600 s). Never embed the App Certificate or App ID + Certificate together in the RN bundle. + +## Token007 is the only credential the app holds + +`BackendApi.getConfig()` returns `token` (a Token007 string) plus `appId`. The `appId` is considered public (needed by the SDK to initialize). The token grants access to a specific channel for a specific UID with a fixed expiry. It is: +- stored only in `AgoraSession` memory (not persisted to AsyncStorage or disk) +- discarded on `AgoraSession.stop()` + +## No secrets in the RN bundle + +| What | Location | Why it's safe | +|------|----------|---------------| +| `AGORA_APP_CERTIFICATE` | `server/.env.local` | backend-only; never sent to app | +| `OPENAI_API_KEY` (BYO) | `server/.env.local` | backend-only; never sent to app | +| Token007 | in-memory `AgoraSession` | short-lived, single-channel, single-UID | +| `AGENT_BACKEND_URL` | `rn/src/config.ts` | URL of the backend; not a credential | + +Do not add certificate, LLM keys, or any persistent secrets to `rn/src/config.ts`, `app.json`, or native Info.plist/AndroidManifest. + +## `OPENAI_API_KEY` is optional (keyless default) + +The pipeline uses the Agora-managed OpenAI vendor by default — no user-supplied LLM key is required. If `OPENAI_API_KEY` is set in `server/.env.local`, it is passed to the `OpenAI` vendor as a BYO key. It is **never** transmitted to the RN app. + +## Microphone permission + +- **Android**: `AndroidManifest.xml` declares `RECORD_AUDIO`; `LandingScreen` calls `PermissionsAndroid.request(RECORD_AUDIO)` at runtime before `connect()`. +- **iOS**: `Info.plist` must contain `NSMicrophoneUsageDescription` (the bare RN project scaffold includes this). The system prompts on first `joinChannel`. + +## CORS + +`server.py` adds `CORSMiddleware` with `allow_origins=["*"]`. Because this is a native app (not a browser), CORS does not gate access to the token service. For production deployments, restrict `allow_origins` to known origins if the backend is also exposed to web clients. + +## Cleartext HTTP in debug + +The Android app reaches `10.0.2.2:8000` over plain HTTP (`usesCleartextTraffic=true` in debug). For production builds, host the backend behind HTTPS and remove the cleartext allowance. + +## Related Deep Dives + +- None. Token issuance details are in [06_interfaces](06_interfaces.md). diff --git a/docs/ai/L1/L2/_index.md b/docs/ai/L1/L2/_index.md new file mode 100644 index 0000000..f08c489 --- /dev/null +++ b/docs/ai/L1/L2/_index.md @@ -0,0 +1,6 @@ +# Deep Dives Index + +| Document | Summary | Load When | +|----------|---------|-----------| +| [rtm_adapter_and_session.md](rtm_adapter_and_session.md) | `RtmEngineAdapter` event translation, bridged listener map, presence synthesis, session subscribe ordering | Touching transcript delivery, RTM subscribe order, `RtmEngineAdapter`, or presence-state handling | +| [native_dependency_setup.md](native_dependency_setup.md) | CocoaPods install flow, Android autolinking, common native linking errors and fixes | Setting up iOS, troubleshooting missing native modules, upgrading `react-native-agora` or `agora-react-native-rtm` | diff --git a/docs/ai/L1/L2/native_dependency_setup.md b/docs/ai/L1/L2/native_dependency_setup.md new file mode 100644 index 0000000..730bce3 --- /dev/null +++ b/docs/ai/L1/L2/native_dependency_setup.md @@ -0,0 +1,87 @@ +# Native Dependency Setup — Deep Dive + +**When to Read This:** You are setting up the iOS build for the first time, troubleshooting missing native modules at runtime, upgrading `react-native-agora` or `agora-react-native-rtm`, or diagnosing a failed `assembleDebug`. + +## How native linking works in this project + +React Native 0.86 uses **autolinking** — no manual `MainApplication.java` registration required. Both `react-native-agora` and `agora-react-native-rtm` ship native modules that are automatically linked. + +| Platform | Mechanism | When it runs | +|----------|-----------|-------------| +| iOS | CocoaPods (`use_native_modules!` in Podfile) | `pod install` | +| Android | Gradle autolinking (`react-native.gradle`) | `./gradlew assembleDebug` | + +## iOS — CocoaPods + +### Install pods + +```bash +cd rn/ios +pod install +``` + +Must be run: +- After initial `npm install` +- After upgrading `react-native-agora` or `agora-react-native-rtm` +- After any `npm install` that changes native packages + +### What the Podfile does + +`rn/ios/Podfile` calls `use_native_modules!` (resolves native packages from `node_modules`) and `use_react_native!` (installs the RN core pods). `react_native_post_install` patches minimum deployment targets. + +The Agora RTC and RTM SDKs include prebuilt XCFrameworks — CocoaPods downloads them from the Agora CDN. This can be slow on first install (~300 MB). + +### Common iOS errors + +| Error | Cause | Fix | +|-------|-------|-----| +| `Module 'AgoraRtcNg' could not be found` | `pod install` not run after `npm install` | `cd ios && pod install` | +| `Unable to find a specification for 'AgoraRtcEngine_iOS'` | CocoaPods cache stale | `pod repo update && pod install` | +| Build fails on arm64 (Apple Silicon) | Rosetta or arch mismatch | `arch -x86_64 pod install` or ensure CocoaPods is native arm64 | +| `min_ios_version_supported` conflict | Pods require a higher iOS deployment target than RN default | Update target in Podfile (`platform :ios, '14.0'`) if needed | + +## Android — Gradle autolinking + +### How autolinking resolves native packages + +At build time, Gradle's `react-native.gradle` plugin scans `node_modules` for packages with `android` native code and generates `PackageList.java`. Both `react-native-agora` and `agora-react-native-rtm` register their native modules here automatically. + +### Required permissions + +`rn/android/app/src/main/AndroidManifest.xml` declares: +- `INTERNET` — required for Agora SDK network connectivity +- `RECORD_AUDIO` — declared statically; requested at runtime via `PermissionsAndroid` + +### Cleartext traffic + +`android:usesCleartextTraffic="${usesCleartextTraffic}"` — in debug builds this is `true` (via `build.gradle` `manifestPlaceholders`), allowing HTTP to `10.0.2.2:8000`. + +### Common Android errors + +| Error | Cause | Fix | +|-------|-------|-----| +| `Could not find com.agora.example:...` | Agora Maven repo not in `build.gradle` | Check that `react-native-agora`'s `build.gradle` adds the Agora Maven repo; usually autolinking handles this | +| `Duplicate class` on `agora-react-native-rtm` + `react-native-agora` | Shared native transitive deps | Check for `exclude group:` rules in `rn/android/app/build.gradle` | +| `RECORD_AUDIO: Permission denied` at runtime | User denied permission | App handles in `LandingScreen.ensureMicPermission()` — check return value | +| `./gradlew: Permission denied` | gradle wrapper not executable | `chmod +x rn/android/gradlew` | + +### Clean build + +If autolinking seems stale after a package upgrade: + +```bash +cd rn/android && ./gradlew clean && ./gradlew assembleDebug +``` + +## Java version requirement + +CI uses JDK 21 (Temurin). Gradle 8.x bundled with RN 0.86 requires JDK 17+. JDK 21 is the verified combination. Check with: + +```bash +java -version # should report 21.x +``` + +## Related L1 Files + +- [01_setup.md](../01_setup.md) — prerequisite list and install commands. +- [07_gotchas.md](../07_gotchas.md) — Pods gotcha and cleartext traffic note. diff --git a/docs/ai/L1/L2/rtm_adapter_and_session.md b/docs/ai/L1/L2/rtm_adapter_and_session.md new file mode 100644 index 0000000..5f92854 --- /dev/null +++ b/docs/ai/L1/L2/rtm_adapter_and_session.md @@ -0,0 +1,109 @@ +# RTM Adapter and Session Lifecycle — Deep Dive + +**When to Read This:** You are modifying transcript delivery, RTM subscribe ordering, the `RtmEngineAdapter` event translation, presence-state synthesis, or the `AgoraSession` start/stop lifecycle. This document explains *why* the adapter is shaped the way it is and where the non-obvious constraints come from. + +## Why a custom `RtmEngineAdapter` is necessary + +The `agora-agent-client-toolkit` defines a `RTMEngine` interface: `addEventListener`, `removeEventListener`, `publish`. The `agora-react-native-rtm` SDK provides an `RTMClient` with a different event shape. The adapter bridges them. + +### `MessageEvent` translation + +The toolkit's `_handleRtmMessage` reads each `'message'` event as: + +```ts +{ publisher: string, messageType: string, message: string } +``` + +The RN SDK's `MessageEvent` has: + +```ts +{ + publisher: string, + message: string | Uint8Array, + messageType: number, // binary enum (0=binary, 1=string) — NOT a string discriminator + customType: string, // the string field the toolkit wants as `messageType` +} +``` + +`makeNativeListener('message')` translates this to the toolkit shape: + +```ts +return (event: MessageEvent) => { + listener({ + publisher: event.publisher, + messageType: event.customType, // carry customType, not the numeric enum + message: event.message, + }); +}; +``` + +### Presence `stateItems` synthesis + +The Agora ConvoAI agent publishes its state as RTM presence state changes. The RN SDK delivers `PresenceEvent.stateItems: { key: string, value: string }[]`. The toolkit's `_handleRtmPresence` expects: + +```ts +{ publisher, timestamp, stateChanged: { state: string, turn_id: string } } +``` + +`makeNativeListener('presence')` folds `stateItems` into a map and synthesizes `stateChanged`: + +```ts +const states: Record = {}; +for (const item of event.stateItems ?? []) { + if (item.key != null) states[item.key] = item.value ?? ''; +} +const stateChanged = typeof states.state === 'string' + ? { state: states.state, turn_id: states.turn_id ?? '0' } + : undefined; +``` + +This matches the native iOS/Android toolkit behavior (which reads `states["state"]` / `states["turn_id"]`). + +### Bridged listener map + +To support `removeEventListener`, the adapter stores a `Map>`. When the toolkit calls `removeEventListener(event, listener)`, the adapter looks up the native closure and deregisters exactly it. Without this, `removeEventListener` would do nothing (closures are unique per call). + +```ts +private bridged = new Map>(); +``` + +## Subscribe ordering in `AgoraSession.start()` + +Order matters. Violating it silently drops transcript or greeting messages. + +``` +1. rtm.login(token) // authenticate with RTM +2. rtc.join(appId, token, channel, uid) // join audio channel (mic published) +3. AgoraVoiceAI.init({ rtcEngine, rtmConfig, renderMode }) +4. ai.on(TRANSCRIPT_UPDATED, ...) // bind app callbacks +5. ai.on(AGENT_STATE_CHANGED, ...) +6. ai.subscribeMessage(channelName) // bind toolkit's internal RTM listeners +7. rtm.subscribeChannel(channelName) // activate RTM 2.x channel delivery ← DO NOT skip +8. [caller] BackendApi.startAgent(...) // start the agent AFTER subscribe +``` + +Step 6 alone is **not enough** for RTM 2.x. Unlike the native iOS/Android SDKs (where subscribe is internal), the TS toolkit only registers listeners in step 6. The channel must be subscribed explicitly (step 7) before any messages arrive. + +## `AgoraSession.stop()` teardown + +``` +1. ai.unsubscribe() // remove toolkit's RTM listeners +2. rtm.unsubscribeChannel(channel) // deactivate RTM channel delivery +3. rtc.leave() // leave RTC channel +4. rtm.logout() // logout RTM +5. ai.destroy() // release toolkit resources +6. rtc.destroy() // release RTC engine +7. rtm.release() // release RTM client +``` + +Errors are caught in a try/finally; all refs are nulled in the finally block so a failed stop doesn't leave a zombie session. + +## Backend `_sessions` and stop resilience + +`server/src/agent.py` stores `agent_id → session` in `_sessions`. `stop()` pops the session and calls `session.stop()`. If the session is missing (backend restarted mid-call), it falls back to `client.stop_agent(agent_id)`, which is the Agora REST stop. This prevents `stopAgent` from returning an error when called after a server restart. + +## Related L1 Files + +- [02_architecture.md](../02_architecture.md) — topology and session lifecycle overview. +- [04_conventions.md](../04_conventions.md) — bridged listener map and adapter pattern. +- [07_gotchas.md](../07_gotchas.md) — subscribe ordering gotchas. diff --git a/docs/ai/RECIPE.md b/docs/ai/RECIPE.md new file mode 100644 index 0000000..7eb77d9 --- /dev/null +++ b/docs/ai/RECIPE.md @@ -0,0 +1,95 @@ +--- +recipe_version: 1.0.0 +recipe_status: experimental +extension_points: + - id: app.screens + name: React Native screens and navigation + - id: app.backend-url + name: Token service URL in rn/src/config.ts + - id: agent.pipeline + name: STT/LLM/TTS vendors and pipeline parameters in server/src/agent.py + - id: agent.greeting + name: Agent opening utterance via AGENT_GREETING env var +invariants: + - id: secrets.server-only + summary: AGORA_APP_CERTIFICATE and OPENAI_API_KEY (BYO) stay in the Python backend; the app receives only the derived Token007. + - id: rtm.subscribe-before-start + summary: RTM channel subscription (subscribeMessage + subscribeChannel) must complete before /startAgent is called. + - id: rtm.explicit-subscribe + summary: agora-react-native-rtm 2.x requires explicit client.subscribe(channel); the TS toolkit's subscribeMessage alone does not activate message delivery. + - id: data-channel.rtm + summary: Backend always sets data_channel="rtm" and advanced_features.enable_rtm=True; the toolkit depends on this for transcript/state. +stable_contracts: + - id: env.required + summary: AGORA_APP_ID and AGORA_APP_CERTIFICATE are required; OPENAI_API_KEY is optional (Agora-managed keyless by default). + - id: api.core-routes + summary: GET /get_config, POST /startAgent, and POST /stopAgent are the stable token-service endpoints. + - id: response.envelope + summary: Successful responses use { code: 0, msg: "success", data: ... }; data is omitted on stopAgent. + - id: agent-config.shape + summary: getConfig returns { app_id, token, uid (string), channel_name, agent_uid (string) }. +--- + +# Recipe Contract + +This base recipe defines the reusable surface for a React Native (TypeScript) Agora Conversational AI quickstart using `agora-agent-client-toolkit` with two engine adapters. + +## Recipe Role + +- Role: `base` recipe (self-contained, clone-and-run; no `Extends` pin). +- Target audience: developers building a React Native voice-agent app backed by a keyless Agora Conversational AI pipeline. +- Reuse model: clone, set `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` in `server/.env.local`, run the server, run the app, then customize screens or pipeline. + +## Recipe Scope + +- Python FastAPI token generation and agent lifecycle (Deepgram STT → OpenAI → MiniMaxTTS, Agora-managed). +- Two React Native engine adapters (`RtcEngineAdapter`, `RtmEngineAdapter`) wrapping `react-native-agora` and `agora-react-native-rtm`. +- `AgoraSession` orchestrating RTM login → RTC join → toolkit init → subscribe → startAgent. +- `useCallStore` React hook owning all call state. +- `LandingScreen` + `CallScreen` minimal UI. + +## Baseline Implementation Guidance + +Use this repo's adapter implementations as the starting point for any RN client. The RTM event translation, presence synthesis, and subscribe ordering are non-obvious — do not recreate from memory. Copy verified patterns from `rn/src/agora/`. + +## Extension Points + +| ID | Surface | How to extend | Required follow-up | +|----|---------|---------------|--------------------| +| `app.screens` | `rn/src/ui/`, `rn/src/CallState.ts`, `rn/App.tsx` | Add screens, add Phase values, add actions to `useCallStore` | Run `npx tsc --noEmit`; verify Android with `assembleDebug` | +| `app.backend-url` | `rn/src/config.ts` | Change `AGENT_BACKEND_URL` for real device or remote backend | Update README with the correct host alias | +| `agent.pipeline` | `server/src/agent.py` `Agent.start()` | Swap `DeepgramSTT`, `OpenAI`, `MiniMaxTTS` for other vendors | Run `cd server && pytest -q`; verify `data_channel="rtm"` is preserved | +| `agent.greeting` | `AGENT_GREETING` env var or `server/.env.local` | Set a custom opening utterance | No code change needed | + +## Invariants + +- `AGORA_APP_CERTIFICATE` lives only in `server/.env.local`; never expose it to the RN app. +- RTM channel must be subscribed (both `ai.subscribeMessage` and `rtm.subscribeChannel`) before `startAgent` is called. +- `data_channel="rtm"` and `advanced_features={"enable_rtm": True}` must remain set on the backend agent; the toolkit depends on them. +- The `agora-react-native-rtm` 2.x SDK requires explicit channel subscribe — the TS toolkit does not subscribe internally. + +## Stable Contracts + +| Contract | Stable shape | +|----------|-------------| +| Required backend env | `AGORA_APP_ID`, `AGORA_APP_CERTIFICATE` | +| Optional backend env | `OPENAI_MODEL`, `OPENAI_API_KEY`, `AGENT_GREETING` | +| `GET /get_config` | Returns `data.app_id`, `data.token`, `data.uid` (string), `data.channel_name`, `data.agent_uid` (string) | +| `POST /startAgent` | Body `{ channelName, rtcUid, userUid, parameters? }`; returns `data.agent_id`, `data.channel_name`, `data.status` | +| `POST /stopAgent` | Body `{ agentId }`; returns `{ code: 0, msg: "success" }` | +| Success envelope | `{ "code": 0, "msg": "success", "data": ... }` | + +## Internal / Subject to Change + +- Visual layout, StyleSheet values, and button labels in `rn/src/ui/`. +- Exact STT model (`nova-3`), LLM model (`gpt-4o-mini`), TTS voice, as long as `data_channel="rtm"` is preserved. +- In-memory `_sessions` dict details; the stable behavior is start-by-channel and stop-by-returned-`agent_id`. +- `agora-agents` SDK minor-version behavior; this recipe lower-bounds `>=2.3.0`. + +## Related Progressive Disclosure Docs + +- `L1/01_setup.md` — install, env, and run commands. +- `L1/02_architecture.md` — topology and session lifecycle. +- `L1/06_interfaces.md` — route, AgentConfig, and toolkit event contracts. +- `L1/L2/rtm_adapter_and_session.md` — RTM adapter and subscribe ordering in depth. +- `L1/L2/native_dependency_setup.md` — CocoaPods and Android autolinking. diff --git a/docs/ai/test-results.md b/docs/ai/test-results.md new file mode 100644 index 0000000..33585e4 --- /dev/null +++ b/docs/ai/test-results.md @@ -0,0 +1,92 @@ +# Progressive Disclosure — Test Results + +> Test run for `recipe-client-rn-quickstart` progressive disclosure docs. +> Date: 2026-06-25 · Standard: AgoraIO-Community/ai-devkit progressive-disclosure. + +## Step 1 — Structural checks + +| Check | Result | +|-------|--------| +| `L0_repo_card.md` ≤ 50 lines | Pass (36) | +| All 8 L1 files present | Pass | +| Each L1 has purpose blockquote + Related Deep Dives | Pass | +| L1 line counts in 80–200 target | **Mixed** (42–98) — 04 and 08 below 80; see note | +| L2 `_index.md` present | Pass | +| Each L2 opens with "When to Read This" callout | Pass (2/2) | +| Relative links resolve (docs/ai/ + AGENTS.md, file targets) | Pass (31/31, 0 broken file targets; 2 directory links in AGENTS.md resolve as directories — valid on GitHub) | +| AGENTS.md has How to Load / Git Conventions / Doc Commands | Pass | + +**Note on L1 line counts:** `04_conventions.md` (42) and `08_security.md` (45) are below the 80-line soft target. Both are table-dense and information-complete. The standard warns against padding, so they were left concise. Accepted deviation. + +## Step 2/3 — Question runs + +Questions span the five standard categories. Each answer was verified against repo source before being marked Pass. "Level" is the lowest disclosure level that fully answers the question. + +### Setup & Build + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 1 | How do I run this locally? | Start `server/` with `python src/server.py`; run the RN app with `npx react-native run-android` or `run-ios` | `L1/01_setup.md` ↔ `README.md` | L1 | Pass | +| 2 | Which env vars are required? | `AGORA_APP_ID`, `AGORA_APP_CERTIFICATE` — no LLM key needed (keyless pipeline) | `L1/01_setup.md`, `06_interfaces.md` ↔ `server/.env.example`, `agent.py` | L1 | Pass | +| 3 | Is there a TypeScript check I can run without a device? | Yes: `cd rn && npx tsc --noEmit` — ran in CI without Android/iOS toolchain | `L1/01_setup.md` ↔ CI `ci.yml` | L1 | Pass (ran: 0 errors) | + +### Test & Run + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 4 | How do I run Jest tests without a server? | `cd rn && npm test` — mocks `fetch`; no server or device needed | `L1/01_setup.md` ↔ `__tests__/BackendApi.test.ts` | L1 | Pass (ran: 2 passed) | +| 5 | How do I run backend tests without cloud credentials? | `cd server && pytest -q` with `venv_test` (agora-agents 2.3.0); `conftest.py` provides fake env | `L1/01_setup.md` ↔ `server/tests/conftest.py` | L1 | Pass (ran with venv_test: 2 passed; note: `venv` has stale 2.2.0 — see note below) | +| 6 | What does the CI gate run for the RN job? | `npm ci` → `tsc --noEmit` → `npm test --ci` → `assembleDebug` (JDK 21 + Android SDK) | `L1/01_setup.md` ↔ `.github/workflows/ci.yml` | L1 | Pass | + +**Note on backend venv:** The repo ships two venvs. `venv/` has `agora-agents 2.2.0` (stale); `venv_test/` has `2.3.0` (correct). The `test_agent_construction.py` test fails with 2.2.0 due to the 2.3.x constructor change. This is a pre-existing local state issue, not a docs issue. Use `venv_test` or recreate `venv` with `uv pip install -r requirements.txt`. + +### Conventions + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 7 | What response shape does the backend use? | `{ code: 0, msg: "success", data: {...} }`; `data` omitted on stopAgent | `L1/04_conventions.md`, `06_interfaces.md` ↔ `server.py` | L1 | Pass | +| 8 | How does turn-type discrimination work for transcript? | `item.metadata.object === MessageType.AGENT_TRANSCRIPTION` → `'agent'`, else `'user'`; composite key `` `${turnId}-${type}` `` | `L1/04_conventions.md` ↔ `CallState.ts` | L1 | Pass | +| 9 | What are the commit/branch conventions? | Conventional commits `type: description`; branches `type/short-description`; no AI tool names | `AGENTS.md` Git Conventions | L0 | Pass | + +### Development + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 10 | How do I add a new screen? | Create `rn/src/ui/YourScreen.tsx`; add Phase value in `CallState.ts`; add action in `useCallStore`; branch in `App.tsx AppContent` | `L1/05_workflows.md` ↔ `App.tsx`, `CallState.ts` | L1 | Pass | +| 11 | How do I change the token endpoint URL? | Edit `AGENT_BACKEND_URL` in `rn/src/config.ts`; Android emulator = `10.0.2.2:8000`, iOS Simulator = `localhost:8000` | `L1/05_workflows.md`, `07_gotchas.md` ↔ `config.ts` | L1 | Pass | +| 12 | Where does token generation live and what must stay server-side? | `server/src/server.py` `generate_convo_ai_token`; `AGORA_APP_CERTIFICATE` must never reach the RN app | `L1/02_architecture.md`, `08_security.md` ↔ `server.py` | L1 | Pass | + +### Deep Dive + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 13 | Why must `subscribeChannel` be called separately from `subscribeMessage`? | TS toolkit's `subscribeMessage` only binds listeners; RTM 2.x requires `client.subscribe(channel)` to activate message delivery (unlike native iOS/Android toolkits) | `L2/rtm_adapter_and_session.md` ↔ `RtmEngineAdapter.ts`, `AgoraSession.ts` | L2 | Pass | +| 14 | How is the RN RTM `PresenceEvent` translated for the toolkit? | `stateItems: {key,value}[]` folded into a map; `stateChanged: {state, turn_id}` synthesized | `L2/rtm_adapter_and_session.md` ↔ `RtmEngineAdapter.ts` | L2 | Pass | +| 15 | Why must CocoaPods be re-run after `npm install` changes? | `react-native-agora` and `agora-react-native-rtm` ship native code installed via Pods; JS-side package changes are not reflected until `pod install` re-runs | `L2/native_dependency_setup.md` ↔ `ios/Podfile`, `07_gotchas.md` | L2 | Pass | + +## Step 4 — Analysis + +- All 15 questions answered at the expected disclosure level (12 at L1, 3 at L2). +- No missing-coverage findings; no broken file references. +- Two soft deviations: `04_conventions.md` and `08_security.md` line counts below 80 (accepted; concise/table-dense). +- Pre-existing local state issue: `venv/` contains stale `agora-agents 2.2.0`; `venv_test/` has the correct `2.3.0`. Backend tests pass with the correct venv. Documented but not a docs correctness issue. + +## Step 5 — Summary + +| Category | Questions | Pass | Notes | +|----------|:---------:|:----:|-------| +| Setup & Build | 3 | 3 | `tsc --noEmit` executed: 0 errors | +| Test & Run | 3 | 3 | Jest: 2 passed; backend pytest (venv_test): 2 passed | +| Conventions | 3 | 3 | — | +| Development | 3 | 3 | — | +| Deep Dive | 3 | 3 | resolved at L2 as designed | +| **Total** | **15** | **15** | — | + +## Step 6 — Fixes / Retest + +No failing questions; no fixes required. Evidence executed during this run: + +- `cd rn && npx tsc --noEmit` → 0 errors. +- `cd rn && npm test --ci` → 2 passed. +- `cd server && pytest -q` (venv_test, agora-agents 2.3.0) → 2 passed. +- Relative link check → 33 checked, 0 broken file targets (2 directory links in AGENTS.md are valid GitHub directory links).