diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d0c1d51 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,143 @@ +# Agent Development Guide + +For coding agents working in `recipe-client-android-quickstart`. This repository is the +**Android client** recipe in the Agora Conversational AI recipes family. + +## How to Load + +This repository uses progressive disclosure documentation. Docs live under +`docs/ai/` in three levels. + +1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo. +2. This repo declares `Recipe Role: base`; read [docs/ai/RECIPE.md](docs/ai/RECIPE.md) before changing reusable recipe contracts. +3. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront. +4. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md). + +The sections below remain the canonical contributor handbook for hands-on work; +the `docs/ai/` tree is the structured summary used by AI agents. + +## System shape + +- **`server/`** — Python FastAPI token service (:8000). Owns Agora token generation and agent session lifecycle. Pipeline: `DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless) → `MiniMaxTTS`. SDK: `agora-agents>=2.3.0`. +- **`android/`** — Kotlin / Jetpack Compose app. `CallViewModel` owns business state; `AgoraSession` owns the Agora SDK lifecycle (RTC + RTM + vendored ConversationalAIAPI toolkit). +- Auth: Token007 from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` (server-side only). +- Zero-key: no LLM API key required; OpenAI vendor is Agora-managed. + +## Pipeline + +`DeepgramSTT` → `OpenAI` (Agora-managed) → `MiniMaxTTS`. Cascading STT/LLM/TTS vendors, not a single MLLM. VAD is configured on `AgoraAgent` turn detection in `server/src/agent.py`. + +## Session ordering (critical) + +The RTC/RTM/toolkit initialization order is strict: + +1. Create `RtcEngine` + `RtmClient` + `ConversationalAIAPIImpl`. +2. `rtm.login(token)` — inside success callback: + - `api.loadAudioSettings(AUDIO_SCENARIO_AI_CLIENT)` — BEFORE `joinChannel`. + - `rtc.joinChannel(...)` as `CLIENT_ROLE_BROADCASTER`, `publishMicrophoneTrack=true`, `autoSubscribeAudio=true`. + - `api.subscribeMessage(channelName)` — wait for success callback, THEN call `POST /startAgent`. + +Violating this order causes lost transcript events or no audio. See [docs/ai/L1/07_gotchas.md](docs/ai/L1/07_gotchas.md) and [docs/ai/L1/L2/sdk_session_integration.md](docs/ai/L1/L2/sdk_session_integration.md). + +## Emulator vs device + +- **Emulator:** `AGENT_BACKEND_URL = http://10.0.2.2:8000` (default; `10.0.2.2` is the host loopback alias). +- **Physical device:** set `AGENT_BACKEND_URL` to the host's LAN IP in `android/app/build.gradle.kts`. + +`AGENT_BACKEND_URL` is a `buildConfigField` — changing it requires a Gradle rebuild. + +## Vendored toolkit + +`android/app/src/main/java/io/agora/scene/convoai/convoaiApi/` is copied verbatim from +`AgoraIO-Community/Conversational-AI-Demo` (MIT). Do not modify it. A `CovLogger` shim routes +its logging to `android.util.Log`. + +## Env vars + +| Variable | Default | Notes | +|---|---|---| +| `AGORA_APP_ID` | — | required | +| `AGORA_APP_CERTIFICATE` | — | required; never sent to the app | +| `OPENAI_MODEL` | `gpt-4o-mini` | Agora-managed; no key needed | +| `OPENAI_API_KEY` | — | optional; BYO only | +| `AGENT_GREETING` | built-in | Optional opening line override | + +## Patterns + +- Keep `AGORA_APP_CERTIFICATE` and token generation in `server/`. +- Call `subscribeMessage` and wait for success before `POST /startAgent`. +- All `BackendApi` calls run on `Dispatchers.IO` inside `viewModelScope.launch`. +- Mic mute via `adjustRecordingSignalVolume(0/100)`; do not toggle `publishMicrophoneTrack` mid-call. + +## Anti-patterns + +- Do not call Agora SDK methods from Compose composables — route through `CallViewModel` → `AgoraSession`. +- Do not embed `AGORA_APP_CERTIFICATE` or any secret in the APK. +- Do not modify `convoaiApi/` — it is a vendored copy. +- Do not skip the `subscribeMessage` → `onReady` callback before calling `startAgent`. +- Do not call `joinChannel` before `loadAudioSettings`. + +## Commands + +```bash +# Backend +cd server +uv venv venv && . venv/bin/activate +uv pip install -r requirements.txt -r requirements-dev.txt +python src/server.py + +# Backend tests (no cloud, no creds) +cd server && pytest -q + +# Android (emulator or device connected) +cd android && ./gradlew installDebug + +# Android unit tests (no emulator) +cd android && ./gradlew testDebugUnitTest + +# Android build only +cd android && ./gradlew assembleDebug +``` + +## Done criteria + +1. Run the narrowest relevant verification command. +2. Android-affecting changes: `./gradlew testDebugUnitTest` passes. +3. Backend-affecting changes: `cd server && pytest -q` passes. +4. If you change required env vars or setup steps, update the root README and the server README together. +5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md). + +## Git Conventions + +### Commit messages — conventional commits + +- **Format:** `type: description` or `type(scope): description` +- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation) +- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `fix(server): validate uid before token` +- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature` +- **Present tense** — "add feature", not "added feature" + +### Branch names + +- **Format:** `type/short-description` — lowercase, hyphen-separated +- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/` +- **Examples:** `feat/add-transcript-screen`, `fix/emulator-url`, `docs/progressive-disclosure` + +### General rules + +- **Repo-local `AGENTS.md` is the authoritative source for repo conventions.** +- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions. +- **No Co-Authored-By trailers** — omit AI attribution lines. +- **No `--no-verify`** — let git hooks run normally. +- **No git config changes** — do not modify `user.name` or `user.email`. + +## Doc Commands + +| Command | When to use | +| ------------- | ---------------------------------------------------------------------------- | +| generate docs | No `docs/ai/` directory exists yet | +| update docs | Code changed since the `Last Reviewed` date in L0 | +| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) | +| fix docs | Close findings from a docs review or test run | + +See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/standard/progressive-disclosure-standard.md) and [workflows](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/workflows/progressive-disclosure-docs.md) for the full specification. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..eea2d6d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,3 @@ +This project uses AGENTS.md instead of a CLAUDE.md file. + +Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project. diff --git a/docs/ai/L0_repo_card.md b/docs/ai/L0_repo_card.md new file mode 100644 index 0000000..c141ca5 --- /dev/null +++ b/docs/ai/L0_repo_card.md @@ -0,0 +1,36 @@ +# recipe-client-android-quickstart — Repo Card + +> Android (Kotlin + Jetpack Compose) voice-agent quickstart. The app owns the Agora RTC + RTM lifecycle and renders live transcripts via the vendored ConversationalAIAPI toolkit. A key-less Python/FastAPI backend mints tokens and runs a managed STT→LLM→TTS cascade. + +## Identity + +| Field | Value | +| -------------- | ---------------------------------------------------------------------------------- | +| Repo | `AgoraIO-Conversational-AI/recipe-client-android-quickstart` | +| Type | `frontend-app` (Android client + bundled token service) | +| Language | Kotlin / Android (Jetpack Compose) + Python 3.12 (FastAPI token service) | +| Deploy Target | Android device / emulator; `server/` as a reachable FastAPI service | +| Owner | Agora Conversational AI DevEx | +| Last Reviewed | 2026-06-25 | +| Recipe Role | `base` | +| Recipe Version | `1.0.0` | +| Recipe Status | `experimental` | + +## L1 — Summaries + +The Audience column helps agents prioritise: **Use** = consuming the recipe's behavior, **Maintain** = modifying internals. + +| File | Purpose | Audience | +| ---------------------------------------- | ------------------------------------------------------------------------------------- | -------------- | +| [01_setup](L1/01_setup.md) | Android Studio / Gradle / SDK toolchain, backend venv, env vars, build + run steps | Use & Maintain | +| [02_architecture](L1/02_architecture.md) | App ↔ token service ↔ Agora ConvoAI topology and session lifecycle | Maintain | +| [03_code_map](L1/03_code_map.md) | `android/` and `server/` trees with key file responsibilities | Maintain | +| [04_conventions](L1/04_conventions.md) | Kotlin/Compose patterns, coroutine/IO threading, Python FastAPI idioms | Maintain | +| [05_workflows](L1/05_workflows.md) | Common changes: add screen, change token endpoint, build/run on device/emulator | Use | +| [06_interfaces](L1/06_interfaces.md) | Token-service REST contract, Agora SDK surface, env vars, response envelope | Use & Maintain | +| [07_gotchas](L1/07_gotchas.md) | Emulator host alias, ordering constraints, cleartext traffic, vendored toolkit rules | Maintain | +| [08_security](L1/08_security.md) | Token handling, App Certificate stays server-side, Android permissions | Maintain | + +## Recipe Profile + +This repo declares `Recipe Role: base`. See [RECIPE.md](RECIPE.md) for extension points, invariants, and stable contracts before changing reusable surfaces. diff --git a/docs/ai/L1/01_setup.md b/docs/ai/L1/01_setup.md new file mode 100644 index 0000000..c885154 --- /dev/null +++ b/docs/ai/L1/01_setup.md @@ -0,0 +1,70 @@ +# 01 · Setup + +> Install the Android toolchain and the Python backend, configure env vars, and build and run the quickstart locally. + +## Prerequisites + +| Tool | Required version | Notes | +| ---- | ---------------- | ----- | +| Android Studio | Latest stable (Iguana / Jellyfish+) | Bundled JDK 17+ works | +| Java | 17 (temurin or bundled) | CI uses `temurin@21`; local JDK 17 is sufficient | +| Android SDK | compileSdk 35, minSdk 24 | SDK Manager → Android 14 (API 35) + build tools | +| Python | 3.10+ | Backend only; CI uses 3.12 | +| `uv` | any recent | Fastest way to create the venv; plain `python -m venv` also works | + +## Configure env + +Create `server/.env.local` (copy from `server/.env` if present): + +| Variable | Required | Default | Notes | +| ----------------------- | :------: | ---------------- | ------------------------------------------------- | +| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID | +| `AGORA_APP_CERTIFICATE` | ✅ | — | Agora Console → Project → App Certificate | +| `OPENAI_MODEL` | | `gpt-4o-mini` | Agora-managed OpenAI vendor; keyless by default | +| `OPENAI_API_KEY` | | — | BYO only — set if your account requires it | +| `AGENT_GREETING` | | built-in line | Optional opening utterance override | + +The `AGORA_APP_CERTIFICATE` stays on the server and is never sent to the app. + +## Run the backend + +```bash +cd server +uv venv venv && . venv/bin/activate +uv pip install -r requirements.txt -r requirements-dev.txt +python src/server.py # serves on 0.0.0.0:8000 +``` + +Backend unit tests (no cloud, no real creds): + +```bash +cd server && pytest -q +``` + +## Build and run the Android app + +### Via Android Studio + +Open `android/` in Android Studio → Run on a connected device or emulator. + +### Via Gradle command line + +```bash +cd android +./gradlew assembleDebug # build only +./gradlew installDebug # install to running emulator / device +./gradlew testDebugUnitTest # JVM unit tests (no emulator needed) +``` + +## Emulator vs physical device + +| Scenario | `AGENT_BACKEND_URL` | How to set | +| -------- | ------------------- | ---------- | +| Android emulator | `http://10.0.2.2:8000` (default) | Default in `app/build.gradle.kts` | +| Physical device | `http://:8000` | Edit `buildConfigField` in `android/app/build.gradle.kts` | + +`10.0.2.2` is the emulator's alias for the host machine's loopback. Physical devices cannot use it. + +## Related Deep Dives + +- [gradle_dependencies.md](L2/gradle_dependencies.md) — Agora SDK Maven coordinates, version catalog, and dependency resolution. diff --git a/docs/ai/L1/02_architecture.md b/docs/ai/L1/02_architecture.md new file mode 100644 index 0000000..86caf69 --- /dev/null +++ b/docs/ai/L1/02_architecture.md @@ -0,0 +1,58 @@ +# 02 · Architecture + +> Three-layer topology: the Android app drives RTC + RTM, fetches config and manages the agent session via the Python token service, and receives live transcript events from the ConversationalAIAPI toolkit over RTM. + +## Topology + +``` +Android App (Compose UI + CallViewModel) + │ GET /get_config ──────────────────────────────────────────┐ + │ POST /startAgent ──────────────────────────────────────────┤ + │ POST /stopAgent ──────────────────────────────────────────┤ + │ ▼ + │ Token Service (server/, :8000) + │ │ generate_convo_ai_token() + │ │ DeepgramSTT → OpenAI → MiniMaxTTS + │ ▼ + │ Agora ConvoAI Cloud + │ RTC audio ◀──────────────────────────────────────────────▶ (agent audio channel) + │ RTM messages ◀───────────────────────────────────────────── transcript / state / metrics + ▼ +ConversationalAIAPIImpl (vendored toolkit) + → onTranscriptUpdated / onAgentStateChanged → CallViewModel → Compose UI +``` + +- **`android/`** — Kotlin / Jetpack Compose. `CallViewModel` owns business logic; `AgoraSession` owns the Agora SDK lifecycle (RTC engine + RTM client + toolkit). +- **`server/`** — Python FastAPI (:8000). Owns token generation (`generate_convo_ai_token`) and agent session lifecycle (`Agent.start()` / `Agent.stop()`). +- **Vendored toolkit** — `convoaiApi/` (copied MIT source). Attaches to the live `RtcEngine` + `RtmClient` and parses RTM messages into structured transcript/state callbacks. + +## Request lifecycle + +1. App `GET /get_config` → server mints a Token007 RTC+RTM token, returns `{app_id, token, uid, channel_name, agent_uid}`. +2. App creates `RtcEngine` + `RtmClient`, logs in to RTM, calls `loadAudioSettings()`, joins the RTC channel, then calls `subscribeMessage(channelName)`. +3. App `POST /startAgent {channelName, rtcUid=agent_uid, userUid=uid}` → server starts managed cascade (DeepgramSTT → OpenAI → MiniMaxTTS) and returns `agent_id`. +4. Agent audio streams into the channel; RTM delivers transcript/state/metrics to the toolkit. +5. App `POST /stopAgent {agentId}` → `session.stop()` → RTC `leaveChannel` → RTM `logout`. + +## Why subscribe before startAgent + +The toolkit's `subscribeMessage(channelName)` must complete **before** `POST /startAgent` is called. If the agent sends its greeting before RTM is subscribed, the first transcript events are lost. `AgoraSession.start()` enforces this ordering; see [07_gotchas](07_gotchas.md). + +## Pipeline (server-side) + +`DeepgramSTT(nova-3, en)` → `OpenAI` (Agora-managed, keyless by default) → `MiniMaxTTS`. The pipeline is a cascading STT→LLM→TTS set of vendors; no separate `llm/` service. VAD is configured on `AgoraAgent` turn detection. + +## Key abstractions + +| Class / File | Where | Responsibility | +| ------------ | ----- | -------------- | +| `CallViewModel` | `android/` | `StateFlow` state machine, coroutine orchestration, `BackendApi` calls | +| `AgoraSession` | `android/` | RTC engine + RTM client lifecycle, `IConversationalAIAPIEventHandler` | +| `BackendApi` | `android/` | OkHttp REST client; decodes `{code, data}` envelope | +| `ConversationalAIAPIImpl` | `android/convoaiApi/` | Vendored toolkit; parses RTM messages into callbacks | +| `Agent` | `server/src/` | `AgoraAgent` construction, session map, start/stop | +| `server.py` | `server/src/` | FastAPI routes, token issuance, error mapping | + +## Related Deep Dives + +- [sdk_session_integration.md](L2/sdk_session_integration.md) — RTC/RTM/toolkit wiring, ordering constraints, and teardown sequence. diff --git a/docs/ai/L1/03_code_map.md b/docs/ai/L1/03_code_map.md new file mode 100644 index 0000000..bb02ff0 --- /dev/null +++ b/docs/ai/L1/03_code_map.md @@ -0,0 +1,70 @@ +# 03 · Code Map + +> Where things live. Two top-level modules: `android/` (Kotlin/Compose app) and `server/` (FastAPI token service). No shared orchestration script — each module runs independently. + +## Root + +| Path | Responsibility | +| --------------------- | -------------------------------------------------------------------------- | +| `README.md` | Layout, run steps, lifecycle walkthrough, SDK versions. | +| `AGENTS.md` | Coding-agent handbook + How to Load / Git Conventions / Doc Commands. | +| `Dockerfile` | Backend-only image (`python:3.12-slim`, port 8000). | +| `.github/workflows/` | `ci.yml` (server pytest + android assembleDebug + unit tests), `docker.yml`. | +| `NOTICE` | Third-party MIT attribution for the vendored toolkit. | + +## `server/` — FastAPI token service (:8000) + +| Path | Responsibility | +| --------------------------------- | ------------------------------------------------------------------------ | +| `src/server.py` | FastAPI app, CORS, route handlers (`/get_config`, `/startAgent`, `/stopAgent`), error mapping, uvicorn entrypoint. | +| `src/agent.py` | `Agent` class: `AsyncAgora` client, cascading vendors, `start()`/`stop()`, `_sessions` map. | +| `scripts/run_fake_server.py` | Boots `server.app` with `FakeAgent` for integration smoke tests. | +| `tests/test_agent_construction.py`| Builds the real `AgoraAgent`, fakes the SDK session, asserts response shape. | +| `tests/test_config.py` | `Agent` construction smoke with stub creds. | +| `tests/conftest.py` | `fake_env` fixture; no cloud, no real creds. | +| `requirements.txt` | Runtime deps (`fastapi`, `uvicorn`, `agora-agents>=2.3.0`). | +| `requirements-dev.txt` | Dev deps (`pytest`, `httpx`). | + +## `android/` — Kotlin / Compose app + +| Path | Responsibility | +| --------------------------------- | ------------------------------------------------------------------------ | +| `app/build.gradle.kts` | SDK dependencies, `AGENT_BACKEND_URL` `buildConfigField`, compileSdk 35. | +| `gradle/libs.versions.toml` | Version catalog: AGP 8.5.2, Kotlin 2.0.20, RTC 4.5.1, RTM 2.2.3. | +| `settings.gradle.kts` | Maven repos (Google, MavenCentral, jitpack), single `:app` module. | +| `app/src/main/AndroidManifest.xml`| `INTERNET` + `RECORD_AUDIO` permissions; `usesCleartextTraffic="true"`. | + +### `app/src/main/java/io/agora/recipe/androidquickstart/` + +| File | Responsibility | +| -------------------- | --------------------------------------------------------------------------- | +| `MainActivity.kt` | Single-activity entry point; phase-driven screen routing (Landing / Call). | +| `CallViewModel.kt` | `AndroidViewModel`; `StateFlow` for phase, turns, mic, agent state, error; coroutine orchestration. | +| `AgoraSession.kt` | Creates `RtcEngineEx` + `RtmClient`; wires toolkit; implements `IConversationalAIAPIEventHandler`. | +| `BackendApi.kt` | OkHttp synchronous client; `@Serializable` envelopes for `/get_config`, `/startAgent`, `/stopAgent`. | + +### `app/src/main/java/.../ui/` + +| File | Responsibility | +| ----------------- | ------------------------------------ | +| `LandingScreen.kt`| Permission request; Connect button. | +| `CallScreen.kt` | Transcript `LazyColumn`, Mute/End controls. | + +### `app/src/main/java/io/agora/scene/convoai/convoaiApi/` (vendored, do not modify) + +| Path | Responsibility | +| ------------------ | --------------------------------------------------------------------------- | +| `IConversationalAIAPI.kt` | Public interface + callback types (`Transcript`, `StateChangeEvent`, etc.). | +| `ConversationalAIAPIImpl.kt` | RTM subscription, message dispatch, handler management. | +| `subRender/v1/`, `subRender/v3/`| RTM message parsers (v1 word-level, v3 full-text transcript modes). | +| `ConversationalAIUtils.kt` | Shared utility helpers. | + +### `app/src/test/` + +| File | Responsibility | +| ------------------ | -------------------------------------------------------------- | +| `BackendApiTest.kt`| JVM unit tests for `BackendApi` using `MockWebServer`. | + +## Related Deep Dives + +- None. For runtime flow see [02_architecture](02_architecture.md); for contracts see [06_interfaces](06_interfaces.md). diff --git a/docs/ai/L1/04_conventions.md b/docs/ai/L1/04_conventions.md new file mode 100644 index 0000000..d77ada8 --- /dev/null +++ b/docs/ai/L1/04_conventions.md @@ -0,0 +1,49 @@ +# 04 · Conventions + +> Coding patterns for both the Android app and the Python server. Follow these to keep the app and backend aligned. + +## Android (Kotlin / Compose) + +- **ViewModel owns state** — `CallViewModel` holds all `StateFlow`s (`phase`, `turns`, `micMuted`, `agentState`, `errorMessage`). Compose screens are stateless; they only read flows and invoke ViewModel functions. +- **IO on `Dispatchers.IO`** — all `BackendApi` calls (`getConfig`, `startAgent`, `stopAgent`) are wrapped in `withContext(Dispatchers.IO)` inside `viewModelScope.launch`. +- **Callback → coroutine bridging** — `AgoraSession.start()` accepts a `onReady(error?)` callback; `CallViewModel` resumes its coroutine from that callback. +- **Upsert by `(turnId, type)`** — `CallViewModel.upsertTranscript` deduplicates by `(turnId, type)`. A turn has separate `USER` and `AGENT` rows; the composite key `"$turnId-${type}"` is used as the `LazyColumn` item key. +- **`adjustRecordingSignalVolume`** — mic mute is implemented by setting recording volume to 0 (mute) or 100 (restore), not by toggling `publishMicrophoneTrack`. +- **Vendored toolkit** — `convoaiApi/` is copied verbatim (MIT). Do not modify it. A `CovLogger` shim in `io.agora.scene.convoai` routes its internal logging to `android.util.Log`. + +## Android build + +- Kotlin 2.0.20 + AGP 8.5.2; Compose BOM 2024.09.02; `kotlinOptions.jvmTarget = "17"`. +- Dependencies declared in `gradle/libs.versions.toml` version catalog; reference via `libs.*` aliases. +- `AGENT_BACKEND_URL` is a `buildConfigField`; change it in `android/app/build.gradle.kts`, not at runtime. +- `usesCleartextTraffic="true"` is set in the manifest for local HTTP dev. Remove or scope with a `network_security_config` before production. + +## Backend (Python / FastAPI) + +- Async throughout: route handlers are `async def`; agent uses `AsyncAgora` and `create_async_session`. +- Pydantic models for request bodies (`StartAgentRequest`, `StopAgentRequest`). Field names are **camelCase** (`channelName`, `rtcUid`, `userUid`) to match the Android client. +- Error mapping is centralized: `_to_http_error()` maps `ValueError → 400`, `RuntimeError → 500`, else 500. Raise plain `ValueError`/`RuntimeError` in business logic; let the route convert. +- Env loaded from `server/.env.local` then `server/.env` with `override=True`. Use `os.getenv`; don't hard-code credentials. + +## Response envelope + +All backend JSON responses use: + +```json +{ "code": 0, "msg": "success", "data": { } } +``` + +`data` is present only when the route returns a payload. The Android `BackendApi` treats a missing or non-zero `code` as a decoding error. + +## Testing approach + +- **Android**: JVM unit tests in `app/src/test/` use `MockWebServer` (OkHttp); no emulator required. Run with `./gradlew testDebugUnitTest`. +- **Server**: `pytest` in `server/`; `conftest.py` provides `fake_env` fixture with stub credentials; no cloud, no real creds needed. `test_agent_construction.py` fakes the SDK session to exercise the real `AgoraAgent` build path. + +## Doc upkeep + +When you change request/response contracts, env vars, or workflow, update the Android client, server, README, **and** the matching `docs/ai/L1/` file together, then bump `Last Reviewed` in [L0](../L0_repo_card.md). + +## Related Deep Dives + +- None. diff --git a/docs/ai/L1/05_workflows.md b/docs/ai/L1/05_workflows.md new file mode 100644 index 0000000..92f8cb0 --- /dev/null +++ b/docs/ai/L1/05_workflows.md @@ -0,0 +1,75 @@ +# 05 · Workflows + +> Step-by-step guides for common changes in this recipe. Each ends with the narrowest verify command to run. + +## Add a new screen or UI control + +1. Create a `@Composable` in `android/app/src/main/java/.../ui/`. +2. Add state to `CallViewModel` as a new `MutableStateFlow`; expose it as `StateFlow`. +3. Update the `when (phase)` dispatch in `MainActivity.kt` if routing changes. +4. Verify: `./gradlew testDebugUnitTest` (for ViewModel logic); open in Android Studio and run on device/emulator for visual check. + +## Change the token service backend URL + +The backend URL is baked into the APK as a `buildConfigField`: + +1. Edit `android/app/build.gradle.kts`: + ```kotlin + buildConfigField("String", "AGENT_BACKEND_URL", "\"http://192.168.1.42:8000\"") + ``` +2. Rebuild: `./gradlew assembleDebug`. + +For emulator, use `http://10.0.2.2:8000` (default). For a physical device, use the host LAN IP. + +## Change the agent prompt / greeting / model + +1. Greeting: set `AGENT_GREETING` in `server/.env.local`, or edit the default in `server/src/agent.py`. +2. Model: set `OPENAI_MODEL` (default `gpt-4o-mini`). +3. STT/TTS vendors: edit `Agent.start()` in `server/src/agent.py` — `DeepgramSTT` and `MiniMaxTTS` config. +4. Verify: `cd server && pytest -q`. + +## Add a new server route + +1. Add the FastAPI handler in `server/src/server.py` (return `{code, msg, data}` envelope). +2. Add a corresponding method in `android/.../BackendApi.kt`. +3. Call from `CallViewModel` via `withContext(Dispatchers.IO)`. +4. Verify: add a `MockWebServer` test in `android/.../BackendApiTest.kt`; run `./gradlew testDebugUnitTest`. + +## Build and run locally (combined) + +```bash +# Terminal 1 — backend +cd server && uv venv venv && . venv/bin/activate +uv pip install -r requirements.txt -r requirements-dev.txt +python src/server.py + +# Terminal 2 — Android (emulator must be running or device connected) +cd android && ./gradlew installDebug +``` + +## Run server tests (no creds, no cloud) + +```bash +cd server && pytest -q +``` + +## Verify before finishing + +| Change touches… | Run | +| ---------------------------- | --------------------------------------------- | +| Android UI / ViewModel only | `./gradlew testDebugUnitTest` | +| Backend logic / vendors | `cd server && pytest -q` | +| Backend API contract | Add `MockWebServer` test + `./gradlew testDebugUnitTest` | +| End-to-end (manual) | Start server, install APK, tap Connect | + +## Deploy + +1. Deploy `server/` (or use the backend Docker image `ghcr.io/AgoraIO-Conversational-AI/recipe-client-android-quickstart` on `v*` tags) to a reachable host. +2. Set `AGORA_APP_ID` and `AGORA_APP_CERTIFICATE` in the backend environment. +3. Update `AGENT_BACKEND_URL` in `android/app/build.gradle.kts` to the deployed URL. +4. Build a release APK / AAB and distribute normally. + +## Related Deep Dives + +- [sdk_session_integration.md](L2/sdk_session_integration.md) — RTC/RTM/toolkit wiring detail. +- [gradle_dependencies.md](L2/gradle_dependencies.md) — SDK version management. diff --git a/docs/ai/L1/06_interfaces.md b/docs/ai/L1/06_interfaces.md new file mode 100644 index 0000000..6f2e292 --- /dev/null +++ b/docs/ai/L1/06_interfaces.md @@ -0,0 +1,74 @@ +# 06 · Interfaces + +> Boundary contracts: token-service REST routes, the Android `BackendApi` surface, env vars, the response envelope, and key Agora SDK entry points. + +## Token service routes (port 8000) + +The Android app calls these directly via `BackendApi`. + +### `GET /get_config` + +- Query (optional): `channel?: string`, `uid?: int` (≤ 0 or missing → server generates one). +- Returns `data`: `{ app_id, token, uid (string), channel_name, agent_uid (string) }`. +- Token is a Token007 RTC+RTM combined token, expiry 3600s, for a concrete non-zero UID. + +### `POST /startAgent` + +- Body: `{ channelName: string, rtcUid: int, userUid: int, parameters?: object }`. + - `parameters.output_audio_codec?: string` — optional codec override. +- Returns `data`: `{ agent_id, channel_name, status: "started" }`. +- 400 if `channelName`, `agent_uid`, or `user_uid` invalid (empty / non-positive). + +### `POST /stopAgent` + +- Body: `{ agentId: string }`. +- Returns `{ code: 0, msg: "success" }` (no `data`). + +## Response envelope + +```json +{ "code": 0, "msg": "success", "data": { } } +``` + +`data` is omitted when the route has no payload. The Android `BackendApi` deserializes via `@Serializable` data classes; decoding fails on non-zero `code` or missing fields. + +## Android `BackendApi` interface + +| Method | HTTP call | Returns | +| ------ | --------- | ------- | +| `getConfig(): AgentConfig` | `GET /get_config?uid=0` | `AgentConfig(app_id, token, uid, channel_name, agent_uid)` | +| `startAgent(channelName, rtcUid, userUid): String` | `POST /startAgent` | `agent_id` string | +| `stopAgent(agentId: String)` | `POST /stopAgent` | Unit (fires and completes) | + +`BackendApi` is synchronous (blocking OkHttp); always call from `Dispatchers.IO`. + +## Agora SDK surface (used in `AgoraSession`) + +| SDK class / call | Purpose | +| ---------------- | ------- | +| `RtcEngine.create(RtcEngineConfig)` | Create the RTC engine; must set `mAudioScenario = AUDIO_SCENARIO_AI_CLIENT`. | +| `rtc.joinChannel(token, channel, uid, ChannelMediaOptions)` | Join with `publishMicrophoneTrack=true`, `autoSubscribeAudio=true`, `clientRoleType=BROADCASTER`. | +| `RtmClient.create(RtmConfig)` | Create the RTM client; `uid` must be the same integer as RTC. | +| `rtm.login(token, callback)` | Authenticate RTM; RTC join and toolkit subscribe happen inside the success callback. | +| `ConversationalAIAPIImpl(ConversationalAIAPIConfig)` | Create the toolkit; `renderMode=TranscriptRenderMode.Text`. | +| `api.loadAudioSettings(audioScenario)` | Must be called **before** `rtc.joinChannel`. | +| `api.subscribeMessage(channelName, callback)` | Must complete **before** `POST /startAgent`. | +| `api.unsubscribeMessage(channelName, callback)` | Call first in teardown. | +| `rtc.adjustRecordingSignalVolume(0 or 100)` | Mute (0) / unmute (100) the mic without re-publishing. | +| `RtcEngine.destroy()` + `RtmClient.release()` | Final cleanup; called after leaveChannel + RTM logout. | + +## Environment variables + +| Variable | Scope | Required | Default | Notes | +| ----------------------- | ---------- | :------: | ---------------- | ----- | +| `AGORA_APP_ID` | server | ✅ | — | Agora Console App ID | +| `AGORA_APP_CERTIFICATE` | server | ✅ | — | Never sent to app | +| `OPENAI_MODEL` | server | | `gpt-4o-mini` | Agora-managed OpenAI vendor | +| `OPENAI_API_KEY` | server | | — | BYO only; optional | +| `AGENT_GREETING` | server | | built-in line | Opening utterance override | +| `AGENT_BACKEND_URL` | Android build config | | `http://10.0.2.2:8000` | Set in `android/app/build.gradle.kts` | +| `PORT` | server (env only) | | `8000` | Read from env; set at runtime, not in `.env.local` | + +## Related Deep Dives + +- [sdk_session_integration.md](L2/sdk_session_integration.md) — detailed ordering and teardown of the RTC/RTM/toolkit lifecycle. diff --git a/docs/ai/L1/07_gotchas.md b/docs/ai/L1/07_gotchas.md new file mode 100644 index 0000000..0f78094 --- /dev/null +++ b/docs/ai/L1/07_gotchas.md @@ -0,0 +1,43 @@ +# 07 · Gotchas + +> Non-obvious pitfalls specific to this Android + ConversationalAI quickstart. Read before changing the session setup, Gradle config, or server wiring. + +## `subscribeMessage` must complete before `POST /startAgent` + +If `subscribeMessage(channelName)` is not called (and its callback received) before the agent is started, the agent's greeting and early transcript events arrive on RTM before the client is subscribed — they are silently dropped. `AgoraSession.start()` calls `subscribeMessage` inside the RTM `login` success callback and only calls `onReady(null)` after subscribe succeeds. Do not reorder this. + +## `loadAudioSettings` must precede `joinChannel` + +`ConversationalAIAPIImpl.loadAudioSettings(audioScenario)` configures internal audio processing. Calling it after `joinChannel` may cause degraded audio or no-op behavior. In `AgoraSession.start()`, it is called immediately before `joinChannel` inside the RTM `login` success callback. + +## Emulator uses `10.0.2.2`, not `localhost` + +The Android emulator cannot reach `localhost` or `127.0.0.1` on the host machine. The host loopback is `10.0.2.2`. The default `AGENT_BACKEND_URL` is set to `http://10.0.2.2:8000` for this reason. Physical devices must use the host's actual LAN IP — `10.0.2.2` does not work on real hardware. + +## `AGENT_BACKEND_URL` is a build-time constant + +`AGENT_BACKEND_URL` is injected via `buildConfigField` in `android/app/build.gradle.kts`. Changing it requires a Gradle rebuild — there is no runtime override. For release builds pointing at a deployed backend, edit this field before building the release APK. + +## `usesCleartextTraffic="true"` is only for local dev + +The manifest sets `android:usesCleartextTraffic="true"` to allow HTTP to the local backend. This must be removed or replaced with a `network_security_config` scoped to debug builds before any production deployment. + +## Vendored toolkit must not be modified + +`convoaiApi/` is a vendored copy from `AgoraIO-Community/Conversational-AI-Demo` (MIT). Changes to it will be overwritten if the toolkit is re-vendored. If you need to customize behavior, either wrap the public interface or update the vendored copy and update `NOTICE`. + +## Toolkit `renderMode` is `Text` only + +`ConversationalAIAPIConfig` is constructed with `renderMode = TranscriptRenderMode.Text`. This activates the v3 transcript parser. Word-level (v1) mode is not wired in this recipe. Do not mix `renderMode.Word` with the existing `onTranscriptUpdated` handling. + +## `adjustRecordingSignalVolume` for mute, not `publishMicrophoneTrack` + +Mute is implemented via `rtcEngine?.adjustRecordingSignalVolume(0)` rather than toggling `publishMicrophoneTrack`. The app remains a BROADCASTER throughout the call; the signal is simply silenced. Toggling `publishMicrophoneTrack` mid-call causes unnecessary channel option re-negotiation. + +## Server is shared with the SwiftUI recipe + +`server/src/server.py` and `server/src/agent.py` were originally written for the iOS (SwiftUI) quickstart and re-used here. The FastAPI title and server README still mention "SwiftUI". This is cosmetic — the pipeline and routes are identical. Do not refactor the server title unless you intend to decouple the recipes. + +## Related Deep Dives + +- [sdk_session_integration.md](L2/sdk_session_integration.md) — full ordering and teardown sequence. diff --git a/docs/ai/L1/08_security.md b/docs/ai/L1/08_security.md new file mode 100644 index 0000000..2da1bac --- /dev/null +++ b/docs/ai/L1/08_security.md @@ -0,0 +1,52 @@ +# 08 · Security + +> Trust boundaries, secret handling, Android permissions, and notes for production hardening. + +## Trust boundaries + +| Hop | Auth | +| --- | ---- | +| Android app → token service | None in local dev (HTTP on the same LAN). | +| Token service → Agora Cloud | Token007, generated from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE`. | +| Agora Cloud → AI vendors | Agora-managed credentials (keyless). `OPENAI_API_KEY` optional for BYO. | +| App ↔ Agora RTC/RTM | Token007 issued per `get_config` call; expiry 3600s. | + +## Secret handling + +- **`AGORA_APP_CERTIFICATE` stays on the server.** It lives only in `server/.env.local` (gitignored) and is never returned to the app. The app receives only a short-lived Token007. +- **No secrets in the APK.** `AGENT_BACKEND_URL` is baked in at build time but is a URL, not a credential. +- `server/.env.local` is gitignored; `server/.env.example` (if present) ships placeholders only. +- Tokens (`generate_convo_ai_token`) expire after 3600s; a new token is fetched on each `GET /get_config`. + +## Android permissions + +| Permission | Declared in manifest | Purpose | +| ---------- | :------------------: | ------- | +| `INTERNET` | Yes | HTTP to token service + Agora RTC/RTM | +| `RECORD_AUDIO` | Yes | Microphone for voice input; requested at runtime in `LandingScreen` via `ActivityResultContracts.RequestPermission` | + +The app requests `RECORD_AUDIO` just before the user taps Connect. If denied, `onConnect` is not called. + +## Cleartext traffic + +`android:usesCleartextTraffic="true"` is set in the manifest for local HTTP development. Before production: +- Scope it to debug builds via a `network_security_config` XML. +- Or use HTTPS for the backend and remove the flag entirely. + +## CORS + +The backend sets `CORSMiddleware` with `allow_origins=["*"]`. This is open by design for a local/dev recipe. Lock down origins before any production deployment. + +## Validation + +- `Agent.start()` rejects empty `channel_name` and non-positive `agent_uid`/`user_uid` before issuing tokens or starting a session. +- Route errors are sanitized: `_log_route_error` logs only non-`None` context; SDK exceptions map to 400/500 without leaking internals beyond the message. + +## Deployment notes + +- The published Docker image is **backend-only** (`:8000`); it does not bundle secrets. Pass `AGORA_APP_ID` and `AGORA_APP_CERTIFICATE` as environment variables at container start. +- Build the Android app in release mode with ProGuard/R8; ensure `AGENT_BACKEND_URL` points at an HTTPS endpoint. + +## Related Deep Dives + +- None. diff --git a/docs/ai/L1/L2/_index.md b/docs/ai/L1/L2/_index.md new file mode 100644 index 0000000..7f4263e --- /dev/null +++ b/docs/ai/L1/L2/_index.md @@ -0,0 +1,6 @@ +# Deep Dives Index + +| Document | Summary | Load When | +| -------- | ------- | --------- | +| [sdk_session_integration.md](sdk_session_integration.md) | RTC/RTM/toolkit wiring, ordering constraints, and teardown sequence | Touching `AgoraSession`, changing join options, or debugging transcript loss | +| [gradle_dependencies.md](gradle_dependencies.md) | Agora SDK Maven coordinates, version catalog aliases, and dependency resolution | Upgrading SDK versions, adding new dependencies, or diagnosing build resolution errors | diff --git a/docs/ai/L1/L2/gradle_dependencies.md b/docs/ai/L1/L2/gradle_dependencies.md new file mode 100644 index 0000000..bde7969 --- /dev/null +++ b/docs/ai/L1/L2/gradle_dependencies.md @@ -0,0 +1,67 @@ +# Gradle Dependencies + +**When to Read This:** You are upgrading Agora SDK versions, adding a new library, diagnosing dependency resolution errors, or understanding the version catalog setup. + +## Version catalog (`android/gradle/libs.versions.toml`) + +All library versions are declared centrally; modules reference them via `libs.*` aliases: + +| Alias | Module | Version | +| ----- | ------ | ------- | +| `libs.agora.rtc` | `io.agora.rtc:full-sdk` | `4.5.1` | +| `libs.agora.rtm` | `io.agora:agora-rtm` | `2.2.3` | +| `libs.okhttp` | `com.squareup.okhttp3:okhttp` | `4.12.0` | +| `libs.okhttp.mockwebserver` | `com.squareup.okhttp3:mockwebserver` | `4.12.0` | +| `libs.kotlinx.serialization.json` | `org.jetbrains.kotlinx:kotlinx-serialization-json` | `1.7.1` | +| `libs.gson` | `com.google.code.gson:gson` | `2.11.0` | +| `libs.kotlinx.coroutines.android` | `org.jetbrains.kotlinx:kotlinx-coroutines-android` | `1.8.1` | +| `libs.compose.bom` | `androidx.compose:compose-bom` | `2024.09.02` | + +Plugins: + +| Alias | ID | Version | +| ----- | -- | ------- | +| `libs.plugins.android.application` | `com.android.application` | AGP `8.5.2` | +| `libs.plugins.kotlin.android` | `org.jetbrains.kotlin.android` | `2.0.20` | +| `libs.plugins.kotlin.serialization` | `org.jetbrains.kotlin.plugin.serialization` | `2.0.20` | +| `libs.plugins.kotlin.compose` | `org.jetbrains.kotlin.plugin.compose` | `2.0.20` | + +## Repository resolution order (`settings.gradle.kts`) + +``` +google() → mavenCentral() → jitpack (https://www.jitpack.io) +``` + +Agora RTC and RTM SDKs resolve from **Maven Central** — no custom Agora Maven repo is needed. The Agora RTM library resolves from the `io.agora` group on Maven Central. JitPack is present for any transitive dependencies that require it. + +## Adding a new library + +1. Add a `[versions]` entry in `libs.versions.toml` (if the version is new). +2. Add a `[libraries]` entry with `module` and `version.ref`. +3. Reference via `implementation(libs.)` in `android/app/build.gradle.kts`. +4. Sync in Android Studio or run `./gradlew :app:dependencies` to verify resolution. + +## Upgrading Agora SDKs + +1. Update `agoraRtc` and/or `agoraRtm` version strings in `[versions]`. +2. Check the [Agora Android SDK release notes](https://docs.agora.io/en/video-calling/overview/release-notes) for breaking API changes. +3. The ConversationalAIAPI toolkit in `convoaiApi/` is a vendored copy and does not automatically track SDK updates — re-vendor it from the upstream demo repo if the toolkit's RTC/RTM API usage diverges. +4. Run `./gradlew assembleDebug testDebugUnitTest` to catch compile errors. + +## `compileSdk` and `minSdk` + +| Field | Value | Notes | +| ----- | ----- | ----- | +| `compileSdk` | 35 | Android 14; required by latest Agora RTC SDK | +| `targetSdk` | 35 | Matches compileSdk | +| `minSdk` | 24 | Android 7.0; minimum supported by Agora RTC 4.x | +| `jvmTarget` | 17 | Kotlin compiler + `compileOptions` both set to 17 | + +## Compose BOM + +Compose dependencies (`material3`, `ui`) are resolved through the BOM (`libs.compose.bom`); individual Compose artifact versions are omitted from `build.gradle.kts`. To upgrade Compose, bump `composeBom` in the version catalog. + +## Related L1 + +- [01_setup](../01_setup.md) — build and run commands. +- [03_code_map](../03_code_map.md) — file locations. diff --git a/docs/ai/L1/L2/sdk_session_integration.md b/docs/ai/L1/L2/sdk_session_integration.md new file mode 100644 index 0000000..f754e45 --- /dev/null +++ b/docs/ai/L1/L2/sdk_session_integration.md @@ -0,0 +1,89 @@ +# SDK Session Integration + +**When to Read This:** You are modifying `AgoraSession.kt`, changing the RTC join options, debugging lost transcripts or the agent not hearing audio, or implementing a teardown variation. + +## Overview + +`AgoraSession` owns the full Agora SDK lifecycle: RTC engine + RTM client + ConversationalAIAPI toolkit. The ordering of SDK calls is strict and dictated by both Agora SDK requirements and the transcript delivery guarantee. + +## Initialization order (inside `AgoraSession.start()`) + +``` +1. RtcEngine.create(RtcEngineConfig) + - mAudioScenario = AUDIO_SCENARIO_AI_CLIENT + - mChannelProfile = CHANNEL_PROFILE_LIVE_BROADCASTING + +2. RtmClient.create(RtmConfig) + - uid = rtcUid.toString() + +3. ConversationalAIAPIImpl(ConversationalAIAPIConfig) + - rtcEngine = rtc, rtmClient = rtm + - renderMode = TranscriptRenderMode.Text + +4. api.addHandler(this) ← before RTM login + +5. rtm.login(token) { + // Inside success callback: + 6. api.loadAudioSettings(AUDIO_SCENARIO_AI_CLIENT) + 7. rtc.joinChannel(token, channelName, rtcUid, ChannelMediaOptions) + - publishMicrophoneTrack = true + - autoSubscribeAudio = true + - clientRoleType = CLIENT_ROLE_BROADCASTER + 8. api.subscribeMessage(channelName) { onReady(...) } + // onReady(null) only after subscribeMessage succeeds + } + +9. Caller: POST /startAgent ← only after onReady(null) +``` + +Skipping or reordering steps 6–8 causes audio or transcript issues. See [07_gotchas](../07_gotchas.md). + +## ChannelMediaOptions — why BROADCASTER + +The app must publish its microphone track for the agent to hear the user. `CLIENT_ROLE_BROADCASTER` + `publishMicrophoneTrack = true` + `autoSubscribeAudio = true` is the correct combination for a voice agent quickstart. An audience role would suppress microphone publishing. + +## RTM UID alignment + +The `RtmConfig` is built with the same `rtcUid.toString()` used for the RTC join. The ConversationalAIAPI toolkit uses this alignment to correlate RTM messages to RTC speakers. If the UIDs diverge, transcript speaker attribution breaks. + +## Mute implementation + +Mic mute uses `rtcEngine.adjustRecordingSignalVolume(0)` (silence the recording signal) rather than `updateChannelMediaOptions(publishMicrophoneTrack = false)`. This avoids mid-call channel option re-negotiation and is the pattern verified in the iOS quickstart. + +## Teardown order (inside `AgoraSession.stop()`) + +``` +1. api.unsubscribeMessage(channelName) +2. api.removeHandler(this) +3. api.destroy() +4. rtcEngine?.leaveChannel() +5. rtcEngine = null; RtcEngine.destroy() ← static destroy +6. rtmClient?.logout(...) +7. rtmClient = null; RtmClient.release() ← static release +``` + +`RtcEngine.destroy()` and `RtmClient.release()` are static calls that release native resources. They must come after the instance references are nulled to avoid double-free. The toolkit must be destroyed before the engines. + +## Transcript upsert keying + +The toolkit delivers `Transcript` events for the same `(turnId, type)` multiple times as the text streams in (the agent narrates word by word, then finalizes). `CallViewModel.upsertTranscript` deduplicates by `(turnId, type)`: + +- Key in the list: `indexOfFirst { it.turnId == t.turnId && it.type == t.type }`. +- `LazyColumn` item key: `"$turnId-${type}"` — stable across recompositions. +- Sort: ascending by `turnId`, then `USER` before `AGENT` within a turn. + +`TranscriptRenderMode.Text` (v3) delivers the final full text per update. Do not switch to `Word` mode without updating the upsert and display logic. + +## ConversationalAIAPIConfig fields + +| Field | Value used | Notes | +| ----- | ---------- | ----- | +| `rtcEngine` | the live `RtcEngineEx` instance | Must be the same instance used for `joinChannel` | +| `rtmClient` | the live `RtmClient` instance | Must be the same instance used for `login` | +| `renderMode` | `TranscriptRenderMode.Text` | Activates v3 full-text transcript parser | + +## Related L1 + +- [02_architecture](../02_architecture.md) — topology overview. +- [06_interfaces](../06_interfaces.md) — Agora SDK surface table. +- [07_gotchas](../07_gotchas.md) — ordering pitfalls. diff --git a/docs/ai/RECIPE.md b/docs/ai/RECIPE.md new file mode 100644 index 0000000..0e57f03 --- /dev/null +++ b/docs/ai/RECIPE.md @@ -0,0 +1,98 @@ +--- +recipe_version: 1.0.0 +recipe_status: experimental +extension_points: + - id: android.ui + name: Compose screens, ViewModel state, and navigation + - id: android.session + name: AgoraSession RTC/RTM options, audio settings, mute behavior + - id: server.agent-pipeline + name: STT/LLM/TTS vendor configuration, greeting, and VAD turn detection + - id: server.routes + name: Token service REST routes and response envelope +invariants: + - id: secrets.server-only + summary: AGORA_APP_CERTIFICATE stays in the server; the app receives only a short-lived Token007. + - id: subscribe-before-start + summary: subscribeMessage(channelName) must complete before POST /startAgent to avoid lost transcript events. + - id: load-audio-before-join + summary: loadAudioSettings() must be called before joinChannel. + - id: broadcaster-role + summary: The app joins as CLIENT_ROLE_BROADCASTER with publishMicrophoneTrack=true so the agent can hear the user. + - id: token.uid-concrete + summary: Backend resolves missing, zero, or negative UIDs before issuing a token. +stable_contracts: + - id: env.required + summary: AGORA_APP_ID and AGORA_APP_CERTIFICATE are required server-side. + - id: api.core-routes + summary: GET /get_config, POST /startAgent, and POST /stopAgent remain the client-facing contract. + - id: response.envelope + summary: Successful backend responses use { code, data } or { code } without data. + - id: android.build-config + summary: AGENT_BACKEND_URL is injected as a buildConfigField; changing it requires a Gradle rebuild. +--- + +# Recipe Contract + +This base recipe defines the reusable surface for an Android Kotlin/Compose client with a Python FastAPI token service for Agora Conversational AI. + +## Recipe Role + +- Role: `base` recipe (self-contained, clone-and-run; no `Extends` pin). +- Target audience: developers building an Android voice-agent app with Agora ConvoAI and a managed STT→LLM→TTS backend pipeline. +- Reuse model: clone, set `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE` in `server/.env.local`, run backend, build and install APK. + +## Recipe Scope + +- Python FastAPI token generation and managed agent lifecycle (DeepgramSTT → OpenAI → MiniMaxTTS, keyless). +- Android Kotlin/Compose UI: `LandingScreen` (permission + connect), `CallScreen` (transcript + mute/end), `CallViewModel` (state machine). +- `AgoraSession`: RTC engine + RTM client + vendored ConversationalAIAPI toolkit lifecycle. +- `BackendApi`: OkHttp REST client with `{ code, data }` envelope decoding. + +## Baseline Implementation Guidance + +Use this repo's source and progressive disclosure docs as the starting point. Do not recreate the Agora SDK lifecycle from memory — the ordering constraints (subscribe before startAgent, loadAudioSettings before join, BROADCASTER role) are non-obvious and verified in the iOS quickstart before being applied here. + +## Extension Points + +| ID | Surface | How to extend | Required follow-up | +| -- | ------- | ------------- | ------------------ | +| `android.ui` | `ui/LandingScreen.kt`, `ui/CallScreen.kt`, `CallViewModel.kt` | Add Compose screens, new `StateFlow`s, or ViewModel functions. | Preserve session lifecycle ownership in `AgoraSession`; do not call SDK methods from Compose directly. | +| `android.session` | `AgoraSession.kt` | Change `ChannelMediaOptions`, audio scenario, mute strategy, or add callbacks. | Keep ordering constraints: loadAudioSettings → joinChannel → subscribeMessage → startAgent. | +| `server.agent-pipeline` | `server/src/agent.py` | Change STT, LLM, TTS vendors, greeting, VAD config, or session parameters. | Run `cd server && pytest -q` after changes. | +| `server.routes` | `server/src/server.py` | Add FastAPI routes. | Add corresponding method in `BackendApi.kt` + `MockWebServer` unit test. | + +## Invariants + +- `AGORA_APP_CERTIFICATE` never reaches the Android app; only Token007 is returned. +- `subscribeMessage(channelName)` completes before `POST /startAgent` — first transcript events would be lost otherwise. +- `loadAudioSettings()` is called before `rtc.joinChannel()`. +- The app joins as `CLIENT_ROLE_BROADCASTER` with `publishMicrophoneTrack = true`. +- Backend resolves UID ≤ 0 to a random concrete UID before token issuance. + +## Stable Contracts + +| Contract | Stable shape | +| -------- | ------------ | +| Required server env | `AGORA_APP_ID`, `AGORA_APP_CERTIFICATE` | +| Optional server env | `OPENAI_MODEL`, `OPENAI_API_KEY`, `AGENT_GREETING` | +| Android build config | `AGENT_BACKEND_URL` (default `http://10.0.2.2:8000`) | +| `GET /get_config` | Query `channel?`, `uid?`; returns `data.app_id`, `data.token`, `data.uid`, `data.channel_name`, `data.agent_uid`. | +| `POST /startAgent` | Body `{ channelName, rtcUid, userUid, parameters? }`; returns `data.agent_id`, `data.channel_name`, `data.status`. | +| `POST /stopAgent` | Body `{ agentId }`; returns `{ code: 0, msg: "success" }`. | +| Success envelope | `{ "code": 0, "msg": "success", "data": ... }` where the route has data. | + +## Internal / Subject to Change + +- Compose layout, colors, typography, and screen composition. +- Exact STT/TTS vendor model names and voice IDs. +- In-memory `Agent._sessions`; the stable behavior is start by channel/user and stop by returned `agent_id`. +- Vendored toolkit internals in `convoaiApi/`; the stable surface is `IConversationalAIAPI` + `IConversationalAIAPIEventHandler`. + +## Related Progressive Disclosure Docs + +- `L1/01_setup.md` — setup, env, and build commands. +- `L1/02_architecture.md` — topology and session lifecycle. +- `L1/05_workflows.md` — common modification workflows. +- `L1/06_interfaces.md` — route, env, and SDK contracts. +- `L1/L2/sdk_session_integration.md` — full RTC/RTM/toolkit wiring detail. diff --git a/docs/ai/test-results.md b/docs/ai/test-results.md new file mode 100644 index 0000000..6faba90 --- /dev/null +++ b/docs/ai/test-results.md @@ -0,0 +1,94 @@ +# Progressive Disclosure — Test Results + +> Test run for `recipe-client-android-quickstart` progressive disclosure docs. +> Date: 2026-06-25 · Standard: AgoraIO-Community/ai-devkit progressive-disclosure. + +## Step 1 — Structural checks + +| Check | Result | +| --------------------------------------------------- | ------ | +| `L0_repo_card.md` ≤ 50 lines | Pass (36) | +| All 8 L1 files present | Pass | +| Each L1 has purpose blockquote + Related Deep Dives | Pass | +| L2 `_index.md` present | Pass | +| Each L2 opens with "When to Read This" callout | Pass (2/2) | +| Relative links resolve (`docs/ai/` + AGENTS.md) | Pass (35/35, 0 broken) | +| AGENTS.md has How to Load / Git Conventions / Doc Commands | Pass | + +**Note on L1 line counts:** files are table-dense and information-complete but +run 43–75 lines, under the 80–200 soft target. The standard favors tables over +prose and warns against bloat, so they were left concise rather than padded. +Accepted deviation; revisit if a section needs more depth. + +## Step 2/3 — Question runs + +Questions span the five standard categories. Each answer was checked against the +repo source before being marked Pass. "Level" is the lowest disclosure level +that fully answers the question. + +### Setup & Build + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 1 | How do I run the backend locally? | `cd server && uv venv venv && . venv/bin/activate && uv pip install -r requirements.txt -r requirements-dev.txt && python src/server.py` (serves on 0.0.0.0:8000). | `L1/01_setup.md` ↔ `README.md` | L1 | Pass | +| 2 | Which server env vars are required? | `AGORA_APP_ID` and `AGORA_APP_CERTIFICATE` only. No LLM key required (keyless pipeline). | `L1/01_setup.md`, `06_interfaces.md` ↔ `server/src/agent.py` | L1 | Pass | +| 3 | How do I run the Android app on an emulator? | `cd android && ./gradlew installDebug`; backend must be running; emulator uses `http://10.0.2.2:8000` by default. | `L1/01_setup.md` ↔ `android/app/build.gradle.kts` | L1 | Pass | + +### Test & Run + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 4 | How do I run server tests without cloud creds? | `cd server && pytest -q`; `conftest.py` fakes env + stubs `load_dotenv`. | `L1/04_conventions.md`, `01_setup.md` ↔ `tests/conftest.py` | L1 | Pass | +| 5 | How do I run Android tests without an emulator? | `cd android && ./gradlew testDebugUnitTest`; uses `MockWebServer` (OkHttp), no emulator needed. | `L1/04_conventions.md` ↔ `BackendApiTest.kt` | L1 | Pass | +| 6 | Are there server tests currently passing? | `test_config.py` passes; `test_agent_construction.py` fails due to pre-existing `AgoraAgent(client=...)` API mismatch from an incomplete 2.3.x migration. Noted below. | `tests/test_agent_construction.py` ↔ `server/src/agent.py` | L1 | Pass (accurately documented) | + +### Conventions + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 7 | What response shape do backend routes use? | `{ code, msg, data }`; `data` omitted when no payload. Android `BackendApi` treats non-zero `code` as an error. | `L1/04_conventions.md`, `06_interfaces.md` ↔ `server.py` | L1 | Pass | +| 8 | How are errors mapped to HTTP codes in the backend? | `ValueError→400`, `RuntimeError→500`, else 500 via `_to_http_error`. | `L1/04_conventions.md` ↔ `server/src/server.py` | L1 | Pass | +| 9 | What are the commit/branch conventions? | Conventional commits `type: description`; branches `type/short-description`; no AI tool names; no Co-Authored-By. | `AGENTS.md` Git Conventions | L1 | Pass | + +### Development + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 10 | How do I point the Android app at a physical device backend? | Edit `buildConfigField("String", "AGENT_BACKEND_URL", "\"http://:8000\"")` in `android/app/build.gradle.kts` and rebuild. | `L1/05_workflows.md`, `07_gotchas.md` ↔ `android/app/build.gradle.kts` | L1 | Pass | +| 11 | How do I change the agent prompt or greeting? | Set `AGENT_GREETING` in `server/.env.local`; or edit the default string in `server/src/agent.py`. | `L1/05_workflows.md` ↔ `agent.py` | L1 | Pass | +| 12 | Where does token generation live and what secret must not leave there? | `generate_convo_ai_token` in `server/src/server.py`; `AGORA_APP_CERTIFICATE` stays on the server; app receives only a Token007. | `L1/02_architecture.md`, `08_security.md` ↔ `server.py`, `agent.py` | L1 | Pass | + +### Deep Dive + +| # | Question | Expected answer | Source of truth | Level | Status | +|---|----------|-----------------|-----------------|-------|--------| +| 13 | What is the strict initialization order for the Android Agora session? | (1) RtcEngine → (2) RtmClient → (3) ConversationalAIAPIImpl → (4) addHandler → (5) rtm.login → inside callback: loadAudioSettings → joinChannel → subscribeMessage → then POST /startAgent. | `L2/sdk_session_integration.md` ↔ `AgoraSession.kt` | L2 | Pass | +| 14 | Why must `subscribeMessage` complete before `POST /startAgent`? | Agent sends its greeting immediately on start via RTM; events arrive before subscribe completes and are silently dropped. `AgoraSession.start()` serializes this via the `subscribeMessage` success callback. | `L2/sdk_session_integration.md`, `07_gotchas.md` ↔ `AgoraSession.kt` | L2 | Pass | +| 15 | How do I upgrade the Agora RTC or RTM SDK version? | Bump `agoraRtc`/`agoraRtm` in `android/gradle/libs.versions.toml`; re-vendor `convoaiApi/` if toolkit API has drifted; run `./gradlew assembleDebug testDebugUnitTest`. | `L2/gradle_dependencies.md` ↔ `libs.versions.toml` | L2 | Pass | + +## Step 4 — Analysis + +- All 15 questions answered at the expected disclosure level (12 at L1, 3 at L2). +- No missing-coverage findings; no broken references (35 links checked, 0 broken). +- One soft deviation: L1 line counts below the 80–200 target (accepted; concise/table-dense). +- Android build verification (`./gradlew assembleDebug`) was not run — Android toolchain not available in this environment. `./gradlew testDebugUnitTest` was not run for the same reason; `BackendApiTest.kt` uses `MockWebServer` and does not require an emulator. +- Pre-existing server test failure noted: `test_agent_construction.py` fails with `TypeError: Agent.__init__() got an unexpected keyword argument 'client'` due to an incomplete agora-agents 2.3.x migration in `server/src/agent.py`. This is a pre-existing code issue, not a documentation issue. `test_config.py` passes (1 passed). + +## Step 5 — Summary + +| Category | Questions | Pass | Notes | +| -------------- | :-------: | :--: | ----- | +| Setup & Build | 3 | 3 | — | +| Test & Run | 3 | 3 | server test pre-existing failure documented; Android tests not run (no toolchain) | +| Conventions | 3 | 3 | — | +| Development | 3 | 3 | — | +| Deep Dive | 3 | 3 | resolved at L2 as designed | +| **Total** | **15** | **15** | — | + +## Step 6 — Fixes / Retest + +No failing doc questions; no doc fixes required. Evidence executed during this run: + +- `pytest -q` (server venv) → `1 passed, 1 failed`; failure is pre-existing code issue in `agent.py`, not in docs. +- Relative link check → `35 checked, 0 broken`. +- Android build/unit tests not executed (Android toolchain unavailable in this environment).