Skip to content

docs: add progressive disclosure documentation #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
plutoless wants to merge 1 commit into
base: fix/agora-agents-2.3.x
Choose a base branch
from
Open

docs: add progressive disclosure documentation #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Agent Development Guide

For coding agents working in `recipe-client-android-quickstart`. This repository is the
**Android client** recipe in the Agora Conversational AI recipes family.

## How to Load

This repository uses progressive disclosure documentation. Docs live under
`docs/ai/` in three levels.

1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo.
2. This repo declares `Recipe Role: base`; read [docs/ai/RECIPE.md](docs/ai/RECIPE.md) before changing reusable recipe contracts.
3. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront.
4. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md).

The sections below remain the canonical contributor handbook for hands-on work;
the `docs/ai/` tree is the structured summary used by AI agents.

## System shape

- **`server/`** — Python FastAPI token service (:8000). Owns Agora token generation and agent session lifecycle. Pipeline: `DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless) → `MiniMaxTTS`. SDK: `agora-agents>=2.3.0`.
- **`android/`** — Kotlin / Jetpack Compose app. `CallViewModel` owns business state; `AgoraSession` owns the Agora SDK lifecycle (RTC + RTM + vendored ConversationalAIAPI toolkit).
- Auth: Token007 from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` (server-side only).
- Zero-key: no LLM API key required; OpenAI vendor is Agora-managed.

## Pipeline

`DeepgramSTT` → `OpenAI` (Agora-managed) → `MiniMaxTTS`. Cascading STT/LLM/TTS vendors, not a single MLLM. VAD is configured on `AgoraAgent` turn detection in `server/src/agent.py`.

## Session ordering (critical)

The RTC/RTM/toolkit initialization order is strict:

1. Create `RtcEngine` + `RtmClient` + `ConversationalAIAPIImpl`.
2. `rtm.login(token)` — inside success callback:
- `api.loadAudioSettings(AUDIO_SCENARIO_AI_CLIENT)` — BEFORE `joinChannel`.
- `rtc.joinChannel(...)` as `CLIENT_ROLE_BROADCASTER`, `publishMicrophoneTrack=true`, `autoSubscribeAudio=true`.
- `api.subscribeMessage(channelName)` — wait for success callback, THEN call `POST /startAgent`.

Violating this order causes lost transcript events or no audio. See [docs/ai/L1/07_gotchas.md](docs/ai/L1/07_gotchas.md) and [docs/ai/L1/L2/sdk_session_integration.md](docs/ai/L1/L2/sdk_session_integration.md).

## Emulator vs device

- **Emulator:** `AGENT_BACKEND_URL = http://10.0.2.2:8000` (default; `10.0.2.2` is the host loopback alias).
- **Physical device:** set `AGENT_BACKEND_URL` to the host's LAN IP in `android/app/build.gradle.kts`.

`AGENT_BACKEND_URL` is a `buildConfigField` — changing it requires a Gradle rebuild.

## Vendored toolkit

`android/app/src/main/java/io/agora/scene/convoai/convoaiApi/` is copied verbatim from
`AgoraIO-Community/Conversational-AI-Demo` (MIT). Do not modify it. A `CovLogger` shim routes
its logging to `android.util.Log`.

## Env vars

| Variable | Default | Notes |
|---|---|---|
| `AGORA_APP_ID` | — | required |
| `AGORA_APP_CERTIFICATE` | — | required; never sent to the app |
| `OPENAI_MODEL` | `gpt-4o-mini` | Agora-managed; no key needed |
| `OPENAI_API_KEY` | — | optional; BYO only |
| `AGENT_GREETING` | built-in | Optional opening line override |

## Patterns

- Keep `AGORA_APP_CERTIFICATE` and token generation in `server/`.
- Call `subscribeMessage` and wait for success before `POST /startAgent`.
- All `BackendApi` calls run on `Dispatchers.IO` inside `viewModelScope.launch`.
- Mic mute via `adjustRecordingSignalVolume(0/100)`; do not toggle `publishMicrophoneTrack` mid-call.

## Anti-patterns

- Do not call Agora SDK methods from Compose composables — route through `CallViewModel` → `AgoraSession`.
- Do not embed `AGORA_APP_CERTIFICATE` or any secret in the APK.
- Do not modify `convoaiApi/` — it is a vendored copy.
- Do not skip the `subscribeMessage` → `onReady` callback before calling `startAgent`.
- Do not call `joinChannel` before `loadAudioSettings`.

## Commands

```bash
# Backend
cd server
uv venv venv && . venv/bin/activate
uv pip install -r requirements.txt -r requirements-dev.txt
python src/server.py

# Backend tests (no cloud, no creds)
cd server && pytest -q

# Android (emulator or device connected)
cd android && ./gradlew installDebug

# Android unit tests (no emulator)
cd android && ./gradlew testDebugUnitTest

# Android build only
cd android && ./gradlew assembleDebug
```

## Done criteria

1. Run the narrowest relevant verification command.
2. Android-affecting changes: `./gradlew testDebugUnitTest` passes.
3. Backend-affecting changes: `cd server && pytest -q` passes.
4. If you change required env vars or setup steps, update the root README and the server README together.
5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md).

## Git Conventions

### Commit messages — conventional commits

- **Format:** `type: description` or `type(scope): description`
- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation)
- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `fix(server): validate uid before token`
- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature`
- **Present tense** — "add feature", not "added feature"

### Branch names

- **Format:** `type/short-description` — lowercase, hyphen-separated
- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/`
- **Examples:** `feat/add-transcript-screen`, `fix/emulator-url`, `docs/progressive-disclosure`

### General rules

- **Repo-local `AGENTS.md` is the authoritative source for repo conventions.**
- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions.
- **No Co-Authored-By trailers** — omit AI attribution lines.
- **No `--no-verify`** — let git hooks run normally.
- **No git config changes** — do not modify `user.name` or `user.email`.

## Doc Commands

| Command | When to use |
| ------------- | ---------------------------------------------------------------------------- |
| generate docs | No `docs/ai/` directory exists yet |
| update docs | Code changed since the `Last Reviewed` date in L0 |
| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) |
| fix docs | Close findings from a docs review or test run |

See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/standard/progressive-disclosure-standard.md) and [workflows](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/workflows/progressive-disclosure-docs.md) for the full specification.
3 changes: 3 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
This project uses AGENTS.md instead of a CLAUDE.md file.

Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project.
36 changes: 36 additions & 0 deletions docs/ai/L0_repo_card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# recipe-client-android-quickstart — Repo Card

> Android (Kotlin + Jetpack Compose) voice-agent quickstart. The app owns the Agora RTC + RTM lifecycle and renders live transcripts via the vendored ConversationalAIAPI toolkit. A key-less Python/FastAPI backend mints tokens and runs a managed STT→LLM→TTS cascade.

## Identity

| Field | Value |
| -------------- | ---------------------------------------------------------------------------------- |
| Repo | `AgoraIO-Conversational-AI/recipe-client-android-quickstart` |
| Type | `frontend-app` (Android client + bundled token service) |
| Language | Kotlin / Android (Jetpack Compose) + Python 3.12 (FastAPI token service) |
| Deploy Target | Android device / emulator; `server/` as a reachable FastAPI service |
| Owner | Agora Conversational AI DevEx |
| Last Reviewed | 2026-06-25 |
| Recipe Role | `base` |
| Recipe Version | `1.0.0` |
| Recipe Status | `experimental` |

## L1 — Summaries

The Audience column helps agents prioritise: **Use** = consuming the recipe's behavior, **Maintain** = modifying internals.

| File | Purpose | Audience |
| ---------------------------------------- | ------------------------------------------------------------------------------------- | -------------- |
| [01_setup](L1/01_setup.md) | Android Studio / Gradle / SDK toolchain, backend venv, env vars, build + run steps | Use & Maintain |
| [02_architecture](L1/02_architecture.md) | App ↔ token service ↔ Agora ConvoAI topology and session lifecycle | Maintain |
| [03_code_map](L1/03_code_map.md) | `android/` and `server/` trees with key file responsibilities | Maintain |
| [04_conventions](L1/04_conventions.md) | Kotlin/Compose patterns, coroutine/IO threading, Python FastAPI idioms | Maintain |
| [05_workflows](L1/05_workflows.md) | Common changes: add screen, change token endpoint, build/run on device/emulator | Use |
| [06_interfaces](L1/06_interfaces.md) | Token-service REST contract, Agora SDK surface, env vars, response envelope | Use & Maintain |
| [07_gotchas](L1/07_gotchas.md) | Emulator host alias, ordering constraints, cleartext traffic, vendored toolkit rules | Maintain |
| [08_security](L1/08_security.md) | Token handling, App Certificate stays server-side, Android permissions | Maintain |

## Recipe Profile

This repo declares `Recipe Role: base`. See [RECIPE.md](RECIPE.md) for extension points, invariants, and stable contracts before changing reusable surfaces.
70 changes: 70 additions & 0 deletions docs/ai/L1/01_setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# 01 · Setup

> Install the Android toolchain and the Python backend, configure env vars, and build and run the quickstart locally.

## Prerequisites

| Tool | Required version | Notes |
| ---- | ---------------- | ----- |
| Android Studio | Latest stable (Iguana / Jellyfish+) | Bundled JDK 17+ works |
| Java | 17 (temurin or bundled) | CI uses `temurin@21`; local JDK 17 is sufficient |
| Android SDK | compileSdk 35, minSdk 24 | SDK Manager → Android 14 (API 35) + build tools |
| Python | 3.10+ | Backend only; CI uses 3.12 |
| `uv` | any recent | Fastest way to create the venv; plain `python -m venv` also works |

## Configure env

Create `server/.env.local` (copy from `server/.env` if present):

| Variable | Required | Default | Notes |
| ----------------------- | :------: | ---------------- | ------------------------------------------------- |
| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE` | ✅ | — | Agora Console → Project → App Certificate |
| `OPENAI_MODEL` | | `gpt-4o-mini` | Agora-managed OpenAI vendor; keyless by default |
| `OPENAI_API_KEY` | | — | BYO only — set if your account requires it |
| `AGENT_GREETING` | | built-in line | Optional opening utterance override |

The `AGORA_APP_CERTIFICATE` stays on the server and is never sent to the app.

## Run the backend

```bash
cd server
uv venv venv && . venv/bin/activate
uv pip install -r requirements.txt -r requirements-dev.txt
python src/server.py # serves on 0.0.0.0:8000
```

Backend unit tests (no cloud, no real creds):

```bash
cd server && pytest -q
```

## Build and run the Android app

### Via Android Studio

Open `android/` in Android Studio → Run on a connected device or emulator.

### Via Gradle command line

```bash
cd android
./gradlew assembleDebug # build only
./gradlew installDebug # install to running emulator / device
./gradlew testDebugUnitTest # JVM unit tests (no emulator needed)
```

## Emulator vs physical device

| Scenario | `AGENT_BACKEND_URL` | How to set |
| -------- | ------------------- | ---------- |
| Android emulator | `http://10.0.2.2:8000` (default) | Default in `app/build.gradle.kts` |
| Physical device | `http://<host-LAN-IP>:8000` | Edit `buildConfigField` in `android/app/build.gradle.kts` |

`10.0.2.2` is the emulator's alias for the host machine's loopback. Physical devices cannot use it.

## Related Deep Dives

- [gradle_dependencies.md](L2/gradle_dependencies.md) — Agora SDK Maven coordinates, version catalog, and dependency resolution.
58 changes: 58 additions & 0 deletions docs/ai/L1/02_architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 02 · Architecture

> Three-layer topology: the Android app drives RTC + RTM, fetches config and manages the agent session via the Python token service, and receives live transcript events from the ConversationalAIAPI toolkit over RTM.

## Topology

```
Android App (Compose UI + CallViewModel)
│ GET /get_config ──────────────────────────────────────────┐
│ POST /startAgent ──────────────────────────────────────────┤
│ POST /stopAgent ──────────────────────────────────────────┤
│ ▼
│ Token Service (server/, :8000)
│ │ generate_convo_ai_token()
│ │ DeepgramSTT → OpenAI → MiniMaxTTS
│ ▼
│ Agora ConvoAI Cloud
│ RTC audio ◀──────────────────────────────────────────────▶ (agent audio channel)
│ RTM messages ◀───────────────────────────────────────────── transcript / state / metrics
▼
ConversationalAIAPIImpl (vendored toolkit)
→ onTranscriptUpdated / onAgentStateChanged → CallViewModel → Compose UI
```

- **`android/`** — Kotlin / Jetpack Compose. `CallViewModel` owns business logic; `AgoraSession` owns the Agora SDK lifecycle (RTC engine + RTM client + toolkit).
- **`server/`** — Python FastAPI (:8000). Owns token generation (`generate_convo_ai_token`) and agent session lifecycle (`Agent.start()` / `Agent.stop()`).
- **Vendored toolkit** — `convoaiApi/` (copied MIT source). Attaches to the live `RtcEngine` + `RtmClient` and parses RTM messages into structured transcript/state callbacks.

## Request lifecycle

1. App `GET /get_config` → server mints a Token007 RTC+RTM token, returns `{app_id, token, uid, channel_name, agent_uid}`.
2. App creates `RtcEngine` + `RtmClient`, logs in to RTM, calls `loadAudioSettings()`, joins the RTC channel, then calls `subscribeMessage(channelName)`.
3. App `POST /startAgent {channelName, rtcUid=agent_uid, userUid=uid}` → server starts managed cascade (DeepgramSTT → OpenAI → MiniMaxTTS) and returns `agent_id`.
4. Agent audio streams into the channel; RTM delivers transcript/state/metrics to the toolkit.
5. App `POST /stopAgent {agentId}` → `session.stop()` → RTC `leaveChannel` → RTM `logout`.

## Why subscribe before startAgent

The toolkit's `subscribeMessage(channelName)` must complete **before** `POST /startAgent` is called. If the agent sends its greeting before RTM is subscribed, the first transcript events are lost. `AgoraSession.start()` enforces this ordering; see [07_gotchas](07_gotchas.md).

## Pipeline (server-side)

`DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless by default) → `MiniMaxTTS`. The pipeline is a cascading STT→LLM→TTS set of vendors; no separate `llm/` service. VAD is configured on `AgoraAgent` turn detection.

## Key abstractions

| Class / File | Where | Responsibility |
| ------------ | ----- | -------------- |
| `CallViewModel` | `android/` | `StateFlow` state machine, coroutine orchestration, `BackendApi` calls |
| `AgoraSession` | `android/` | RTC engine + RTM client lifecycle, `IConversationalAIAPIEventHandler` |
| `BackendApi` | `android/` | OkHttp REST client; decodes `{code, data}` envelope |
| `ConversationalAIAPIImpl` | `android/convoaiApi/` | Vendored toolkit; parses RTM messages into callbacks |
| `Agent` | `server/src/` | `AgoraAgent` construction, session map, start/stop |
| `server.py` | `server/src/` | FastAPI routes, token issuance, error mapping |

## Related Deep Dives

- [sdk_session_integration.md](L2/sdk_session_integration.md) — RTC/RTM/toolkit wiring, ordering constraints, and teardown sequence.
Loading
Loading