Skip to content

docs: add progressive disclosure documentation #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
plutoless wants to merge 1 commit into
base: fix/agora-agents-2.3.x
Choose a base branch
from
Open

docs: add progressive disclosure documentation #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Agent Development Guide

For coding agents working in `recipe-client-rn-quickstart`. This repository is the
**React Native client** recipe in the Agora Conversational AI recipes family — a bare
React Native (TypeScript) app wired to a keyless Python FastAPI token backend.

## How to Load

This repository uses progressive disclosure documentation. Docs live under
`docs/ai/` in three levels.

1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo.
2. This repo declares `Recipe Role: base`; read [docs/ai/RECIPE.md](docs/ai/RECIPE.md) before changing reusable recipe contracts.
3. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront.
4. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md).

The sections below remain the canonical contributor handbook for hands-on work;
the `docs/ai/` tree is the structured summary used by AI agents.

## System shape

- **`rn/`** — React Native 0.86 / TypeScript 5 app. Two Agora engine adapters
(`RtcEngineAdapter`, `RtmEngineAdapter`) implement the `agora-agent-client-toolkit`
interfaces over `react-native-agora` (RTC) and `agora-react-native-rtm` (RTM 2.x).
`AgoraSession` owns session lifecycle; `useCallStore` owns all React state.
- **`server/`** — Python FastAPI token service (:8000). Owns Agora token generation and
a keyless cascading agent pipeline: `DeepgramSTT(nova-3)` → `OpenAI` (Agora-managed)
→ `MiniMaxTTS`. SDK: `agora-agents>=2.3.0`. No `OPENAI_API_KEY` required by default.
- Auth: Token007 from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` in the backend.
- Transcript delivery: `data_channel="rtm"` with `advanced_features={"enable_rtm": True}`.

## Session ordering

`RTM login → RTC join → AgoraVoiceAI.init → subscribeMessage → subscribeChannel → /startAgent`

Both `ai.subscribeMessage(channel)` **and** `rtm.subscribeChannel(channel)` must run before
`/startAgent`. The TS toolkit's `subscribeMessage` only binds listeners; RTM 2.x requires
an explicit channel subscribe to deliver messages.

## Routing / ownership

- UI, RTC/RTM lifecycle, and session orchestration live in `rn/src/`.
- Token generation and agent lifecycle live in `server/src/`.
- The React Native app calls the backend directly over HTTP (no rewrite proxy — this is a
native app, not a browser). Backend URL is hardcoded in `rn/src/config.ts`.
- Android emulator reaches the host backend at `10.0.2.2:8000`; iOS Simulator uses `localhost:8000`.

## Supported modes

- **Local:** start `server/` manually (`python src/server.py`) then run the RN app
(`npx react-native run-android` or `run-ios`).
- **Deploy:** host the server behind HTTPS; update `AGENT_BACKEND_URL` in `rn/src/config.ts`
and use a real device. A backend image is published to
`ghcr.io/AgoraIO-Conversational-AI/recipe-client-rn-quickstart` on `v*` tags.

## Env vars

| Variable | Default | Notes |
|------------------------|------------------------|-------|
| `AGORA_APP_ID` | — | **required** |
| `AGORA_APP_CERTIFICATE`| — | **required** — server-only, never in the app |
| `OPENAI_MODEL` | `gpt-4o-mini` | Agora-managed vendor model |
| `OPENAI_API_KEY` | — (keyless) | Optional BYO; Agora manages by default |
| `AGENT_GREETING` | built-in line | Override the opening utterance |

## Patterns

- Keep `AGORA_APP_CERTIFICATE` in `server/.env.local`; never pass it to the RN app.
- Keep `data_channel="rtm"` and `advanced_features={"enable_rtm": True}` in the backend agent.
- Always call both `ai.subscribeMessage(channel)` and `rtm.subscribeChannel(channel)` before `/startAgent`.
- Use the `(turnId, type)` composite key for transcript upserts — `CallState.ts` owns this.

## Anti-patterns

- Do not embed `AGORA_APP_CERTIFICATE` or `OPENAI_API_KEY` in `rn/src/config.ts`, `app.json`, or native plists/manifests.
- Do not remove `data_channel="rtm"` or `enable_rtm: True` — transcript delivery will silently stop.
- Do not call `/startAgent` before `subscribeChannel` resolves — the agent greeting will be missed.
- Do not skip `pod install` after `npm install` changes native packages on iOS.

## Commands

```bash
# Backend
cd server && python src/server.py

# RN app
cd rn && npx react-native run-android # Android
cd rn && npx react-native run-ios # iOS (requires pod install first)

# Tests
cd rn && npx tsc --noEmit # TypeScript check (no toolchain needed)
cd rn && npm test # Jest unit tests (no server needed)
cd rn/android && ./gradlew assembleDebug # native Android build
cd server && pytest -q # backend unit tests (no cloud, no creds)
```

## Done criteria

1. Run `npx tsc --noEmit` (always) and `npm test` for any RN source change.
2. For native dependency changes: run `pod install` (iOS) or `./gradlew assembleDebug` (Android).
3. For backend changes: run `cd server && pytest -q`.
4. If you change required env vars or setup steps, update the root README, `server/README.md`, and `server/.env.example` together.
5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md).

## Git Conventions

### Commit messages — conventional commits

- **Format:** `type: description` or `type(scope): description`
- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation)
- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `fix(rn): correct rtm subscribe order`
- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature`
- **Present tense** — "add feature", not "added feature"

### Branch names

- **Format:** `type/short-description` — lowercase, hyphen-separated
- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/`
- **Examples:** `feat/add-settings-screen`, `fix/rtm-subscribe-order`, `docs/progressive-disclosure`

### General rules

- **Repo-local `AGENTS.md` is the authoritative source for repo conventions.**
- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions.
- **No Co-Authored-By trailers** — omit AI attribution lines.
- **No `--no-verify`** — let git hooks run normally.
- **No git config changes** — do not modify `user.name` or `user.email`.

## Doc Commands

| Command | When to use |
| ------------- | ---------------------------------------------------------------------------- |
| generate docs | No `docs/ai/` directory exists yet |
| update docs | Code changed since the `Last Reviewed` date in L0 |
| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) |
| fix docs | Close findings from a docs review or test run |

See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/standard/progressive-disclosure-standard.md) and [workflows](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/workflows/progressive-disclosure-docs.md) for the full specification.
3 changes: 3 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
This project uses AGENTS.md instead of a CLAUDE.md file.

Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project.
36 changes: 36 additions & 0 deletions docs/ai/L0_repo_card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# recipe-client-rn-quickstart — Repo Card

> Bare React Native (TypeScript) voice-agent quickstart: two Agora engine adapters wire `agora-agent-client-toolkit` to `react-native-agora` (RTC) and `agora-react-native-rtm` (RTM), backed by a keyless Python FastAPI token service.

## Identity

| Field | Value |
| -------------- | ------------------------------------------------------------------------------------- |
| Repo | `AgoraIO-Conversational-AI/recipe-client-rn-quickstart` |
| Type | `frontend-app` (React Native client + bundled token backend) |
| Language | React Native 0.86 / TypeScript 5 + Python 3.12 (FastAPI token service) |
| Deploy Target | iOS Simulator / Android emulator or device; server/ as standalone FastAPI service |
| Owner | Agora Conversational AI DevEx |
| Last Reviewed | 2026-06-25 |
| Recipe Role | `base` |
| Recipe Version | `1.0.0` |
| Recipe Status | `experimental` |

## L1 — Summaries

The Audience column helps agents prioritise: **Use** = consuming the recipe's behavior, **Maintain** = modifying internals.

| File | Purpose | Audience |
| ---------------------------------------- | ------------------------------------------------------------------------------------- | -------------- |
| [01_setup](L1/01_setup.md) | Node + npm + iOS/Android toolchain, backend venv/uv, env vars, build/run commands | Use & Maintain |
| [02_architecture](L1/02_architecture.md) | App → token service → Agora ConvoAI topology, session lifecycle | Maintain |
| [03_code_map](L1/03_code_map.md) | `rn/src/`, `rn/ios/`, `rn/android/`, `server/src/` layout and key file duties | Maintain |
| [04_conventions](L1/04_conventions.md) | RN/TypeScript idioms, custom hook state, adapter pattern, Python backend conventions | Maintain |
| [05_workflows](L1/05_workflows.md) | Add a screen, change token endpoint, run on iOS/Android, toggle mic, run tests | Use |
| [06_interfaces](L1/06_interfaces.md) | Token-service API contract, `AgentConfig` shape, toolkit event surface | Use & Maintain |
| [07_gotchas](L1/07_gotchas.md) | Native autolinking, Pods, Android emulator host alias, RTM subscribe order | Maintain |
| [08_security](L1/08_security.md) | Token handling, App Certificate server-only, no secrets in the RN bundle | Maintain |

## Recipe Profile

This repo declares `Recipe Role: base`. See [RECIPE.md](RECIPE.md) for extension points, invariants, and stable contracts before changing reusable surfaces.
87 changes: 87 additions & 0 deletions docs/ai/L1/01_setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# 01 · Setup

> Install the React Native toolchain, the Python token backend, configure env vars, and run the app on Android or iOS.

## Prerequisites

| Tool | Minimum | Notes |
|------|---------|-------|
| Node.js | 22.11.0 | enforced in `rn/package.json` `engines` field |
| npm | bundled with Node | used in `rn/`; npm ci in CI |
| Java (JDK) | 21 (temurin) | Android build (`assembleDebug`) |
| Android SDK | API 33+ | via `android-actions/setup-android` in CI; locally via Android Studio |
| Xcode | latest | iOS Simulator; macOS only |
| CocoaPods | system | iOS native dependency install (`pod install`) |
| Python | 3.12 recommended (3.10+ supported) | backend venv |
| uv | any | recommended for venv/install; `pip` also works |

Agora App ID and App Certificate are required (no third-party LLM key — the pipeline is Agora-managed/keyless).

## Install — backend

```bash
cd server
uv venv venv && . venv/bin/activate
uv pip install -r requirements.txt -r requirements-dev.txt
```

Copy env template and fill credentials:

```bash
cp .env.example .env.local
# edit .env.local: set AGORA_APP_ID and AGORA_APP_CERTIFICATE
```

## Install — React Native app

```bash
cd rn
npm install

# iOS only — install native pods
cd ios && pod install && cd ..
```

## Configure env — backend

Backend env file: `server/.env.local` (template: `server/.env.example`).

| Variable | Required | Default | Notes |
| ---------------------- | :------: | ------------------------------------ | ---------------------------------------------- |
| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE`| ✅ | — | Agora Console → Project → App Certificate |
| `OPENAI_MODEL` | | `gpt-4o-mini` | OpenAI model used by the Agora-managed vendor |
| `OPENAI_API_KEY` | | — (keyless by default) | BYO only — Agora manages the key by default |
| `AGENT_GREETING` | | `"Hi there! How can I help you today?"` | Override the agent's opening line |

## Configure — app backend URL

`rn/src/config.ts` exports `AGENT_BACKEND_URL`. Default is `http://10.0.2.2:8000` (Android emulator host alias for `localhost`). For the iOS Simulator change to `http://localhost:8000`.

## Run

```bash
# Terminal 1 — backend
cd server && python src/server.py # binds 0.0.0.0:8000

# Terminal 2 — Android
cd rn && npx react-native run-android # starts Metro + installs on emulator/device

# Terminal 2 — iOS
cd rn && npx react-native run-ios # requires macOS + Xcode + pod install done
```

Grant microphone permission when prompted, then tap **Connect**.

## Quick test commands

```bash
cd rn && npx tsc --noEmit # TypeScript compile check (no native toolchain needed)
cd rn && npm test # Jest unit tests (mocked fetch — no server needed)
cd rn/android && ./gradlew assembleDebug # native Android build smoke
cd server && pytest -q # backend unit tests (no cloud, no creds)
```

## Related Deep Dives

- [native_dependency_setup.md](L2/native_dependency_setup.md) — CocoaPods, Android autolinking, and Pods troubleshooting.
62 changes: 62 additions & 0 deletions docs/ai/L1/02_architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# 02 · Architecture

> React Native app calls the token backend directly over HTTP, then uses two Agora engine adapters to join both RTC (audio) and RTM (transcript) channels via `agora-agent-client-toolkit`.

## Topology

```
React Native App (rn/)
│ HTTP fetch
▼
Token Backend (server/, :8000)
│ GET /get_config → returns app_id, token, uid, channelName, agentUid
│ POST /startAgent → starts the Agora ConvoAI agent in the channel
│ POST /stopAgent → stops the agent
▼
Agora ConvoAI Cloud
│ user audio (RTC) → DeepgramSTT → OpenAI (Agora-managed) → MiniMaxTTS
│ agent audio (RTC) → back to user's channel
│ transcript (RTM) → MessageEvent delivered to RtmEngineAdapter
▼
agora-agent-client-toolkit (AgoraVoiceAI)
│ TRANSCRIPT_UPDATED → CallState.useCallStore → CallScreen FlatList
│ AGENT_STATE_CHANGED → CallState.agentState badge
```

## Key components

| Component | Location | Role |
|-----------|----------|------|
| `BackendApi` | `rn/src/BackendApi.ts` | HTTP fetch wrapper for the three token-service endpoints |
| `AgoraSession` | `rn/src/agora/AgoraSession.ts` | Orchestrates `RtcEngineAdapter` + `RtmEngineAdapter` + `AgoraVoiceAI`; owns session lifecycle |
| `RtcEngineAdapter` | `rn/src/agora/RtcEngineAdapter.ts` | Adapts `react-native-agora` `IRtcEngine` to the toolkit's `RTCEngine` interface |
| `RtmEngineAdapter` | `rn/src/agora/RtmEngineAdapter.ts` | Adapts `agora-react-native-rtm` `RTMClient` to the toolkit's `RTMEngine` interface; translates native `MessageEvent` shape |
| `useCallStore` | `rn/src/CallState.ts` | React custom hook owning `phase`, `turns`, `agentState`, `micMuted`; drives `AgoraSession` |
| `LandingScreen` | `rn/src/ui/LandingScreen.tsx` | Pre-call UI; requests Android microphone permission |
| `CallScreen` | `rn/src/ui/CallScreen.tsx` | In-call UI: transcript `FlatList` + mute/end controls |
| `server/src/agent.py` | `server/` | Agora-managed cascading pipeline (Deepgram STT → OpenAI → MiniMaxTTS) via `agora-agents>=2.3.0` |

## Session lifecycle

1. `useCallStore.connect()` → `BackendApi.getConfig()` → receives `{appId, token, uid, channelName, agentUid}`.
2. `AgoraSession.start(config)` → `RtmEngineAdapter.login(token)` → `RtcEngineAdapter.join(...)` (mic published, role Broadcaster).
3. `AgoraVoiceAI.init({ rtcEngine, rtmConfig, renderMode: TEXT })`.
4. `ai.subscribeMessage(channelName)` binds toolkit listeners; `rtm.subscribeChannel(channelName)` activates RTM 2.x delivery — **both required, in this order**.
5. `BackendApi.startAgent(channelName, agentUid, uid)` → Agora ConvoAI cloud starts the agent.
6. Agent greets user; RTM `MessageEvent` → `RtmEngineAdapter` → `AgoraVoiceAI` → `TRANSCRIPT_UPDATED` → `CallScreen` re-renders.
7. `end()` → `stopAgent(agentId)` → `AgoraSession.stop()` (RTM unsubscribe + logout, RTC leave).

## Pipeline (server side)

`DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless) → `MiniMaxTTS` — no `llm/` service; single-process cascade. Data channel is `"rtm"` with `enable_rtm: True`; transcript/state/metrics ride RTM.

## Tech decisions

- **Engine adapters** — `agora-agent-client-toolkit` is engine-agnostic; adapters translate native RN SDK event shapes to the toolkit's `RTCEngine`/`RTMEngine` interfaces.
- **RTM 2.x explicit subscribe** — the TS toolkit's `subscribeMessage` only binds listeners; `client.subscribe(channel)` must be called separately (unlike the native iOS/Android toolkits which subscribe internally).
- **Keyless pipeline** — no third-party LLM key required; Agora manages the OpenAI vendor. `OPENAI_API_KEY` is optional BYO.
- **Android emulator alias** — `10.0.2.2` is the emulator's route to the host `localhost:8000`.

## Related Deep Dives

- [rtm_adapter_and_session.md](L2/rtm_adapter_and_session.md) — RTM event translation, presence-state synthesis, subscribe ordering.
Loading
Loading