diff --git a/AGENTS.md b/AGENTS.md index 42eb380..174fc67 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,41 +1,138 @@ -# Agent Guide +# Agent Development Guide -Use this file as the primary agent-facing guide for `agora-convoai-quickstart-nextjs`. +This guide is for coding agents making changes in `agent-quickstart-nextjs`. + +## How to Load + +This repository uses progressive disclosure documentation. Docs live under `docs/ai/` in three levels. + +1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo. +2. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront. +3. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md). + +The sections below (Start Here, Patterns, Anti-Patterns, etc.) remain the canonical contributor handbook for hands-on work; the `docs/ai/` tree is the structured summary used by AI agents. ## Start Here - Read [README.md](./README.md) for setup, commands, verification, and deployment. -- Use [DOCS/GUIDE.md](./DOCS/GUIDE.md) for the long-form build walkthrough. -- Use [DOCS/TEXT_STREAMING_GUIDE.md](./DOCS/TEXT_STREAMING_GUIDE.md) for transcript and RTM behavior. +- Use [docs/GUIDE.md](./docs/GUIDE.md) for the long-form build walkthrough. +- Use [docs/TEXT_STREAMING_GUIDE.md](./docs/TEXT_STREAMING_GUIDE.md) for transcript and RTM behavior. +- For layout and responsibilities inside `components/`, `app/api/`, and `lib/`, use [docs/ai/L1/03_code_map.md](docs/ai/L1/03_code_map.md) and [docs/ai/L1/02_architecture.md](docs/ai/L1/02_architecture.md). ## Current System Shape -- Next.js 16 App Router with React 19 and TypeScript -- Browser RTC via `agora-rtc-react` -- RTM transcripts via `agora-rtm` -- Transcript/runtime helpers via `agora-agent-client-toolkit` -- Shared UI primitives via `agora-agent-uikit` -- Token and agent lifecycle routes inside `app/api` +- App shell: Next.js 16 App Router, React 19, and TypeScript +- Client RTC: `agora-rtc-react` hooks over `agora-rtc-sdk-ng` +- Messaging: `agora-rtm` for transcripts, agent state, metrics, and error events +- Toolkit core: `agora-agent-client-toolkit` for `AgoraVoiceAI`, transcript helpers, and turn status +- UI components: `agora-agent-uikit` for visualizer, transcript, and mic controls +- Server SDK: `agora-agent-server-sdk` for managed agent session startup +- API routes: token generation, agent invite, chat, and stop routes live in `app/api` +- Default agent config: Agora-managed STT, LLM, and TTS; no third-party vendor keys are required for the base quickstart + +## Supported Modes + +### Local Development + +- Run from the repo root with `pnpm run dev`. +- Next.js serves the app and the route handlers at `http://localhost:3000`. +- Local credentials are read from `.env.local`, usually written by `agora project env write .env.local`. + +### Vercel Deployment + +- Deploy the repository as a single Next.js app. +- Set `NEXT_PUBLIC_AGORA_APP_ID` and `NEXT_AGORA_APP_CERTIFICATE` in the deployment target. +- Keep `NEXT_AGORA_APP_CERTIFICATE` server-side only. + +## Routing / Ownership + +- UI and RTC/RTM client lifecycle live in `components`. +- Browser-facing API routes live in `app/api`. +- Shared constants and transcript normalization live in `lib`. +- If a workflow, request contract, or ownership boundary changes, update `README.md`, `docs/GUIDE.md`, `docs/TEXT_STREAMING_GUIDE.md`, and this file in the same change. ## Key Files -- `app/api/generate-agora-token/route.ts`: RTC + RTM token generation -- `app/api/invite-agent/route.ts`: managed agent session startup -- `app/api/stop-conversation/route.ts`: agent shutdown -- `components/LandingPage.tsx`: session bootstrap, RTM setup, provider wiring -- `components/ConversationComponent.tsx`: RTC join, transcript flow, visualizer, renewals -- `lib/agora.ts`: shared agent UID defaults -- `env.local.example`: local environment template +- `app/api/generate-agora-token/route.ts`: issues RTC + RTM tokens for the browser user. +- `app/api/invite-agent/route.ts`: starts the managed agent session; edit here for system prompt, VAD, model, or voice changes. +- `app/api/stop-conversation/route.ts`: stops the agent session. +- `app/api/chat/completions/route.ts`: optional OpenAI-compatible SSE proxy for a custom LLM (not wired by default). +- `components/LandingPage.tsx`: session bootstrap, RTM setup, provider wiring, and conversation lifecycle. +- `components/ConversationComponent.tsx`: RTC join, mic publication, `AgoraVoiceAI` init, transcript state, and renewals. +- `components/QuickstartConversationLayout.tsx`: in-call header, transcript rail, and controls dock. +- `components/QuickstartPipelineMetrics.tsx`: per-stage latency chips from `AGENT_METRICS`. +- `components/QuickstartTranscriptPanel.tsx`: live transcript rail. +- `lib/agora.ts`: shared agent UID defaults. +- `lib/conversation.ts`: transcript normalization and visualizer state mapping. +- `env.local.example`: local environment template. +- `scripts/verify-api-contracts.ts`: route contract verification. + +## Patterns + +### StrictMode Guard (`isReady`) + +Both `useJoin` and `useLocalMicrophoneTrack` are gated by `isReady` to prevent double initialization in React StrictMode dev mode. The cleanup fires synchronously before any `setTimeout`, so only the real second mount's timer fires. + +```tsx +const [isReady, setIsReady] = useState(false); +useEffect(() => { + let cancelled = false; + const id = setTimeout(() => { + if (!cancelled) setIsReady(true); + }, 0); + return () => { + cancelled = true; + clearTimeout(id); + setIsReady(false); + }; +}, []); +const { isConnected: joinSuccess } = useJoin(config, isReady); +const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); +``` + +### Hook Ownership + +- `useJoin` owns `client.leave()`; never call it manually. +- `useLocalMicrophoneTrack` owns track lifecycle; do not manually call `.close()`. +- `usePublish` owns publish state; mute with `track.setEnabled()` and do not manually unpublish. + +### AgoraVoiceAI Init + +Initialize `AgoraVoiceAI` from `agora-agent-client-toolkit` inside `ConversationComponent`, gated on `isReady && joinSuccess`. + +```tsx +useEffect(() => { + if (!isReady || !joinSuccess) return; + // AgoraVoiceAI.init() is called here exactly once. +}, [isReady, joinSuccess]); +``` + +`isReady` becomes true only after the StrictMode fake-unmount cycle completes. Once `isReady` is true, React does not double invoke the effect for later dependency changes such as `joinSuccess` becoming true. + +### Transcript and UI Mapping + +- Manage `transcript` and `agentState` through `useState` plus `ai.on(TRANSCRIPT_UPDATED, ...)` and `ai.on(AGENT_STATE_CHANGED, ...)`. +- The toolkit uses `uid="0"` as a sentinel for the local user's speech. Remap that value to `client.uid` in the `TRANSCRIPT_UPDATED` callback or `ConvoTextStream` will render user speech on the wrong side. +- Include `INTERRUPTED` turns in `messageList`; filter only `IN_PROGRESS`. If the agent's first turn is interrupted and omitted, `messageList` stays empty and `ConvoTextStream` never auto-opens. + +### Tokens and Styling + +- RTM token access must come from `RtcTokenBuilder.buildTokenWithRtm`; a standard RTC-only token does not grant RTM access. +- Tailwind must scan uikit classes with `./node_modules/agora-agent-uikit/dist/**/*.{js,mjs}` in `tailwind.config.ts`. ## Working Rules -- Keep the RTC client creation StrictMode-safe with `useRef`, not `useMemo`. -- Keep the token route on `RtcTokenBuilder.buildTokenWithRtm`. +- Prefer the smallest change that keeps the quickstart copyable and production-style. +- Keep RTC client creation StrictMode-safe with `useRef`, not `useMemo`. +- Keep token generation on `RtcTokenBuilder.buildTokenWithRtm`. - Keep transcript UID remapping aligned with the toolkit sentinel behavior. -- Keep README, `DOCS/GUIDE.md`, and `DOCS/TEXT_STREAMING_GUIDE.md` aligned with implementation changes. +- Do not require third-party vendor API keys unless the code actually introduces a BYOK provider path. +- Keep README, `docs/GUIDE.md`, and `docs/TEXT_STREAMING_GUIDE.md` aligned with implementation changes. ## Commands +From the repo root: + ```bash pnpm install pnpm run doctor @@ -52,8 +149,69 @@ pnpm run verify:api pnpm run build ``` +## Verification Safety + +- Safe without live Agora credentials: + - `pnpm run lint` + - `pnpm run typecheck` + - `pnpm run verify:api` + - `pnpm run build` +- Requires local env setup but not a live Agora session: + - `pnpm run doctor` + - `pnpm run verify` +- Often blocked inside restricted sandboxes because of port binding or process spawning: + - `pnpm run dev` + +## Anti-Patterns / What NOT To Do + +- Do not call `client.leave()` manually; it breaks `useJoin` cleanup. +- Do not call `localMicrophoneTrack.close()` manually; it breaks hook ownership. +- Do not remove the `isReady` guard. +- Do not set `reactStrictMode: false` as a workaround. +- Do not use the deprecated `turnDetection.type: 'agora_vad'` flat API; use `turnDetection.config.start_of_speech` and `turnDetection.config.end_of_speech`. +- Do not replace `RtcTokenBuilder.buildTokenWithRtm` with an RTC-only token builder. +- Do not hide SDK requirements only in `CLAUDE.md`; all agent-facing guidance belongs in `AGENTS.md`. + ## Done Criteria -1. Run the narrowest relevant validation command. +Before finishing a change: + +1. Run the narrowest relevant verification command. 2. For shipped app/runtime changes, ensure `pnpm run verify` passes. -3. Update the root README and any affected docs when workflow or architecture guidance changes. +3. If you changed files in `components/` or `app/api/`, verify that `docs/GUIDE.md`, `docs/TEXT_STREAMING_GUIDE.md`, `README.md`, and this file still match the implementation. +4. Update root README and affected docs when workflow, request contracts, architecture, or environment guidance changes. +5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md). + +## Git Conventions + +### Commit messages — conventional commits + +- **Format:** `type: description` or `type(scope): description` +- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation) +- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `feat(api): add stop-conversation status flag` +- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature` +- **Present tense** — "add feature", not "added feature" +- **PR number appended** — `feat: add feature (#123)` + +### Branch names + +- **Format:** `type/short-description` — lowercase, hyphen-separated +- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/` +- **Examples:** `feat/agent-metrics`, `fix/transcript-uid`, `docs/progressive-disclosure` + +### General rules + +- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions. +- **No Co-Authored-By trailers** — omit AI attribution lines. +- **No `--no-verify`** — let git hooks run normally. +- **No git config changes** — do not modify `user.name` or `user.email`. + +## Doc Commands + +| Command | When to use | +| --------------- | ------------------------------------------------------------ | +| generate docs | No `docs/ai/` directory exists yet | +| update docs | Code changed since the `Last Reviewed` date in L0 | +| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) | + +The generator and tester live in the [AgoraIO-Community/ai-devkit](https://github.com/AgoraIO-Community/ai-devkit) skill set. See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/progressive-disclosure-standard.md) for the full specification. diff --git a/CLAUDE.md b/CLAUDE.md index a4effe4..eea2d6d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,104 +1,3 @@ -# CLAUDE.md +This project uses AGENTS.md instead of a CLAUDE.md file. -> Read `AGENTS.md` for the current project map, commands, and validation rules before making changes. - -## Project - -Next.js 16 (App Router) quickstart demonstrating Agora Conversational AI Engine. Developers copy this code — production quality and idiomatic patterns are required in every change. - -## Commands - -```bash -pnpm run doctor # local prerequisites and env checks -pnpm dev # start dev server (http://localhost:3000) -pnpm run verify # doctor + lint + typecheck + API contracts + production build -``` - -## Key Patterns - -### StrictMode Guard (`isReady`) - -Both `useJoin` and `useLocalMicrophoneTrack` are gated by `isReady` to prevent double-initialization in React StrictMode dev mode. The cleanup fires synchronously before any `setTimeout`, so only the real second mount's timer fires. - -```tsx -const [isReady, setIsReady] = useState(false); -useEffect(() => { - let cancelled = false; - const id = setTimeout(() => { if (!cancelled) setIsReady(true); }, 0); - return () => { cancelled = true; clearTimeout(id); setIsReady(false); }; -}, []); -const { isConnected: joinSuccess } = useJoin(config, isReady); -const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); -``` - -Do not remove this pattern. Do not set `reactStrictMode: false` as a workaround. - -### Hook Ownership - -- `useJoin` owns `client.leave()` — **never call it manually** -- `useLocalMicrophoneTrack` owns track lifecycle — **no manual `.close()`** -- `usePublish` owns publish state — mute via `track.setEnabled()` only, never unpublish manually - -### AgoraVoiceAI Init — Raw Toolkit with `isReady && joinSuccess` Gate - -`AgoraVoiceAI` (from `agora-agent-client-toolkit`) is initialized in a `useEffect` inside `ConversationComponent`, gated on `isReady && joinSuccess`: - -```tsx -useEffect(() => { - if (!isReady || !joinSuccess) return; - // AgoraVoiceAI.init() called here — exactly once, no StrictMode interference -}, [isReady, joinSuccess]); -``` - -**Why `isReady && joinSuccess` works:** -- `isReady` is `true` only after the StrictMode fake-unmount cycle completes (via `setTimeout(fn, 0)` pattern). -- Once `isReady` is `true`, React does NOT double-invoke the effect for subsequent dependency changes (`joinSuccess` becoming `true`). This means `AgoraVoiceAI.init()` is called exactly once. - -`transcript` and `agentState` are managed via `useState` + `ai.on(TRANSCRIPT_UPDATED, ...)` / `ai.on(AGENT_STATE_CHANGED, ...)` directly. - -### UID Remapping - -The toolkit uses `uid="0"` as a sentinel for the local user's speech. The uikit treats `uid===0` as an AI message. Remap in the `TRANSCRIPT_UPDATED` callback to `client.uid` or ConvoTextStream will show user speech on the wrong side. - -### `messageList` Filter - -Include `INTERRUPTED` turns in `messageList` (filter only `IN_PROGRESS`). If the agent's first turn is interrupted, omitting it means `messageList` stays empty and `ConvoTextStream` never auto-opens. - -## Architecture - -| Layer | Package | Role | -|---|---|---| -| Client UI | `agora-rtc-react` | RTC hooks (`useJoin`, `useLocalMicrophoneTrack`, `usePublish`, etc.) | -| Toolkit core | `agora-agent-client-toolkit` | `AgoraVoiceAI`, `TurnStatus` enum, `TranscriptHelperItem` types | -| UI components | `agora-agent-uikit` | `AudioVisualizer`, `ConvoTextStream`, `MicButtonWithVisualizer` | -| Server SDK | `agora-agent-server-sdk` | Builder pattern — `AgoraClient` → `Agent` → `session.start()` | -| Messaging | `agora-rtm` | RTM transport for transcripts | - -RTM token must be generated with `RtcTokenBuilder.buildTokenWithRtm` — a standard RTC-only token does not grant RTM access. - -Tailwind must scan uikit classes: `./node_modules/agora-agent-uikit/dist/**/*.{js,mjs}` in `tailwind.config.ts`. - -## Important Files - -| File | Purpose | -|---|---| -| `components/ConversationComponent.tsx` | Core real-time UI — all Agora hooks, `AgoraVoiceAI` init, transcript state | -| `components/LandingPage.tsx` | Entry point — session setup, RTM client lifecycle, parallel agent+RTM init | -| `app/api/invite-agent/route.ts` | Starts AI agent — edit for system prompt, VAD, model, voice | -| `app/api/generate-agora-token/route.ts` | Issues RTC+RTM token for the browser user | -| `app/api/stop-conversation/route.ts` | Stops the agent | -| `DOCS/GUIDE.md` | Step-by-step build guide — must stay in sync with implementation | -| `DOCS/TEXT_STREAMING_GUIDE.md` | Text streaming / transcription deep-dive | -| `AGENTS.md` | Primary agent-facing project map and validation guide | - -## After Changing Implementation Files - -After editing anything in `components/` or `app/api/`, manually verify that `DOCS/GUIDE.md`, `DOCS/TEXT_STREAMING_GUIDE.md`, `README.md`, and `AGENTS.md` still match the implementation, then run `pnpm run verify`. - -## What NOT To Do - -- Do not call `client.leave()` manually (breaks `useJoin` cleanup) -- Do not call `localMicrophoneTrack.close()` manually (breaks hook ownership) -- Do not remove the `isReady` guard -- Do not set `reactStrictMode: false` -- Do not use the deprecated `turnDetection.type: 'agora_vad'` flat API — use `turnDetection.config.start_of_speech` / `end_of_speech` +Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project. diff --git a/DOCS/GUIDE.md b/DOCS/GUIDE.md deleted file mode 100644 index 054bba3..0000000 --- a/DOCS/GUIDE.md +++ /dev/null @@ -1,1594 +0,0 @@ -# Build a Conversational AI App with Next.js and Agora - -Conversational Voice AI is transforming how people interact with AI. It allows you to have a real-time conversation with an AI agent, and actually get something done without wasting time typing out your thoughts and trying to format them into a clever prompt. It's a major shift in the way people interact with AI. - -But given the investment that developers and businesses have made in building their own text based agents that run through custom LLM workflows, there's reluctance to adopt this new paradigm. Especially if it means having to give up all that investment or even worse, hobble it by only connecting them as tools/function calls. - -This is why we built the Agora Conversational AI Engine. It allows you to connect your existing LLM workflows to an Agora channel, and have a real-time conversation with the AI agent. - -In this guide, we'll build a real-time audio conversation application that connects users with an AI agent powered by Agora's Conversational AI Engine. The app will be built with Next.js, React, and TypeScript. We'll take an incremental approach, starting with the core real-time communication components and then adding Agora's Convo AI Engine. - -By the end of this guide, you will have a real-time audio conversation application that connects users with an AI agent powered by Agora's Conversational AI Engine. - -## Prerequisites - -Before starting, for the guide you're going to need to have: - -- Node.js (v22 or higher; see `engines` in `package.json`) -- A basic understanding of React with TypeScript and Next.js. -- [An Agora account](https://console.agora.io/signup) - _first 10k minutes each month are free_ -- Conversational AI service [activated on your AppID](https://console.agora.io/) - -## Project Setup - -Let's start by creating a new Next.js project with TypeScript support. - -```bash -pnpm create next-app@latest ai-conversation-app -cd ai-conversation-app -``` - -When prompted, select these options: - -- TypeScript: Yes -- ESLint: Yes -- Tailwind CSS: Yes -- Use `src/` directory: No -- App Router: Yes -- Use Turbopack: No -- Customize import alias: Yes (use the default `@/*`) - -Next, install the required Agora dependencies: - -- [agora-rtc-react](https://www.npmjs.com/package/agora-rtc-react) — Agora's React SDK for real-time audio/video -- [agora-rtm](https://www.npmjs.com/package/agora-rtm) — Agora's Real-Time Messaging SDK (used for transcripts) -- [agora-token](https://www.npmjs.com/package/agora-token) — Agora's Token Builder (server-side) -- [agora-agent-server-sdk](https://www.npmjs.com/package/agora-agent-server-sdk) — invites and manages the AI agent (server-side) -- [agora-agent-client-toolkit](https://www.npmjs.com/package/agora-agent-client-toolkit) — handles transcript events from the AI agent -- [agora-agent-uikit](https://www.npmjs.com/package/agora-agent-uikit) — ready-made UI components for the conversation interface - -```bash -pnpm add agora-rtc-react agora-rtm agora-token agora-agent-server-sdk@^1.3.1 agora-agent-client-toolkit agora-agent-uikit -``` - -For UI components, we'll use shadcn/ui in this guide, but you can use any UI library of your choice or create custom components: - -```bash -pnpm dlx shadcn@latest init -``` - -As we go through this guide, you'll have to create new files in specific directories. So, before we start let's create these new directories. - -In your project root directory, create the `app/api/`, `components/`, and `types/` directories, and add the `.env.local` file: - -```bash -mkdir app/api components types -touch .env.local -``` - -Your project directory should now have a structure like this: - -``` -├── app/ -│ ├── api/ -│ ├── globals.css -│ ├── layout.tsx -│ └── page.tsx -├── components/ -├── types/ -├── .env.local -└── (... Existing files and directories) -``` - -### Tailwind Configuration - -The `agora-agent-uikit` package ships pre-built components with Tailwind class names. To ensure those classes are included in your production build, add the uikit's `dist/` folder to Tailwind's `content` array in `tailwind.config.ts`: - -```typescript -import type { Config } from 'tailwindcss'; - -const config: Config = { - content: [ - './pages/**/*.{js,ts,jsx,tsx,mdx}', - './components/**/*.{js,ts,jsx,tsx,mdx}', - './app/**/*.{js,ts,jsx,tsx,mdx}', - // Scan the uikit's compiled output so its Tailwind classes are included - './node_modules/agora-agent-uikit/dist/**/*.{js,mjs}', - ], - // ... rest of your config -}; - -export default config; -``` - -## Landing Page Component - -Let's begin by setting up our landing page that initializes the Agora client and sets up the `AgoraProvider`. - -Create the `LandingPage` component file at `components/LandingPage.tsx`: - -```bash -touch components/LandingPage.tsx -``` - -For now we'll keep this component simple, and fill it in with more functionality as we progress through the guide. I've included comments throughout the code to help you understand what's happening. At a high level, we're importing the Agora React SDK and creating the AgoraRTC client, and then passing it to the `AgoraProvider` so all child components use the same `client` instance. - -Add the following code to the `LandingPage.tsx` file: - -```typescript -'use client'; - -import { useRef, useState } from 'react'; -import dynamic from 'next/dynamic'; - -// Agora requires access to the browser's WebRTC API, -// - which throws an error if it's loaded via SSR -// Create a component that has SSR disabled, -// - and use it to load the AgoraRTC components on the client side -const AgoraProvider = dynamic( - async () => { - // Dynamically import Agora's components - const { AgoraRTCProvider, default: AgoraRTC } = await import( - 'agora-rtc-react' - ); - - return { - default: ({ children }: { children: React.ReactNode }) => { - // Persist the RTC client across StrictMode's simulated remounts. - const clientRef = useRef | null>(null); - if (!clientRef.current) { - clientRef.current = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' }); - } - - // The provider makes the client available to all child components - return {children}; - }, - }; - }, - { ssr: false } // Important: disable SSR for this component -); - -export default function LandingPage() { - // Basic setup, we'll add more functionality as we progress through the guide. - return ( -
-

- Agora AI Conversation -

- -
-

- When was the last time you had an intelligent conversation? -

- - {/* Placeholder for our start conversation button */} -
- -
- - -
- "PLACEHOLDER: We'll add the conversation component here" -
-
-
-
- ); -} -``` - -Now update your `app/page.tsx` file to use this landing page: - -```typescript -import LandingPage from '@/components/LandingPage'; - -export default function Home() { - return ; -} -``` - -## Basic Agora React JS Implementation - -With the landing page setup we can focus on implementing Agora's React JS SDK to handle the core RTC functionality, like joining a channel, publishing audio, receiving audio, and handling the Agora SDK events. - -Create a file at `components/ConversationComponent.tsx`, - -```bash -touch components/ConversationComponent.tsx -``` - -Add the following code: - -```typescript -'use client'; - -import { useState, useEffect } from 'react'; -import { - useRTCClient, - useLocalMicrophoneTrack, - useRemoteUsers, - useClientEvent, - useIsConnected, - useJoin, - usePublish, - RemoteUser, - UID, -} from 'agora-rtc-react'; - -export default function ConversationComponent() { - // Access the client from the provider context - const client = useRTCClient(); - - // Track connection status - const isConnected = useIsConnected(); - - // Manage microphone state - const [isEnabled, setIsEnabled] = useState(true); - - // Track remote users (like our AI agent) - const remoteUsers = useRemoteUsers(); - - // StrictMode guard: delay useJoin's ready flag until after the fake-unmount - // cycle completes. React StrictMode fires cleanup synchronously before any - // setTimeout callback, so the first (fake) mount's timeout is always cancelled. - // Only the real second mount's timeout fires, meaning useJoin joins exactly once. - const [isReady, setIsReady] = useState(false); - useEffect(() => { - let cancelled = false; - const id = setTimeout(() => { - if (!cancelled) setIsReady(true); - }, 0); - return () => { - cancelled = true; - clearTimeout(id); - setIsReady(false); - }; - }, []); - - // Join the channel once isReady (after StrictMode's fake-unmount cycle) - const { isConnected: joinSuccess } = useJoin( - { - appid: process.env.NEXT_PUBLIC_AGORA_APP_ID!, // Load APP_ID from env.local - channel: 'test-channel', - token: 'replace-with-token', - uid: 0, // Join with UID 0 and Agora will assign a unique ID when the user joins - }, - isReady - ); - - // Create mic track only after the StrictMode cycle completes. - // Do NOT pass isEnabled here — that ties track lifetime to mute state and - // breaks the Web Audio graph. Mute uses track.setEnabled() only. - const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); - - // Publish our microphone track to the channel - usePublish([localMicrophoneTrack]); - - // Set up event handlers for client events - useClientEvent(client, 'user-joined', (user) => { - console.log('Remote user joined:', user.uid); - }); - - useClientEvent(client, 'user-left', (user) => { - console.log('Remote user left:', user.uid); - }); - - // Toggle microphone via setEnabled — usePublish owns publish state - const toggleMicrophone = async () => { - if (localMicrophoneTrack) { - const next = !isEnabled; - await localMicrophoneTrack.setEnabled(next); - setIsEnabled(next); - } - }; - - // useJoin handles client.leave() on unmount — no manual cleanup needed here - - return ( -
-
-

- {/* Display the connection status */} - Connection Status: {isConnected ? 'Connected' : 'Disconnected'} -

-
- - {/* Display remote users */} -
- {remoteUsers.length > 0 ? ( - remoteUsers.map((user) => ( -
- -
- )) - ) : ( -

No remote users connected

- )} -
- - {/* Microphone control */} - -
- ); -} -``` - -This component is the foundation for our real-time audio communication, so let's recap the Agora React hooks that we're using: - -- `useRTCClient`: Gets access to the Agora RTC client from the provider we set up in the landing page -- `useLocalMicrophoneTrack`: Creates and manages the user's microphone input -- `useRemoteUsers`: Keeps track of other users in the channel (our AI agent will appear here) -- `useJoin`: Handles joining the channel with the specified parameters -- `usePublish`: Publishes our audio track to the channel so others can hear us -- `useClientEvent`: Sets up event handlers for important events like users joining or leaving - -> **Note on React StrictMode:** In development, React StrictMode mounts components twice to surface bugs. Keep the RTC client in a `useRef`, not a `useMemo`, so the provider does not create two client instances during that cycle. The `isReady` guard then prevents `useJoin` from calling `client.join()` during the fake first mount. - -> **Note:** We are loading the `APP_ID` from the environment variables using the non-null assertion operator, so make sure to set it in `.env.local` file. - -We need to add this component to our `LandingPage.tsx` file. Start by importing the component, and then add it to the AgoraProvider component. - -```typescript -// Previous imports remain the same as before... -// Dynamically import the ConversationComponent with ssr disabled -const ConversationComponent = dynamic(() => import('./ConversationComponent'), { - ssr: false, -}); -// Previous code remains the same as before... - - -; -``` - -Next, we'll implement token authentication, to add a layer of security to our application. - -## Token Generation and Management - -The Agora team strongly recommends using token-based authentication for all your apps, especially in production environments. In this step, we'll create a route to generate these tokens and update our `LandingPage` and `ConversationComponent` to use them. - -### Token Generation Route - -Let's break down what the token generation route needs to do: - -1. Generate a secure Agora token using our App ID and Certificate -2. Create a unique channel name for each conversation -3. Return token, along with the channel name, and UID we used to generate it, back to the client -4. Support token refresh, using existing channel name and UID - -Create a new file at `app/api/generate-agora-token/route.ts`: - -```bash -mkdir app/api/generate-agora-token -touch app/api/generate-agora-token/route.ts -``` - -Add the following code: - -```typescript -import { NextRequest, NextResponse } from 'next/server'; -import { RtcTokenBuilder, RtcRole } from 'agora-token'; - -// Access environment variables -const APP_ID = process.env.NEXT_PUBLIC_AGORA_APP_ID; -const APP_CERTIFICATE = process.env.NEXT_AGORA_APP_CERTIFICATE; -const EXPIRATION_TIME_IN_SECONDS = 3600; // Token valid for 1 hour - -// Helper function to generate unique channel names -function generateChannelName(): string { - // Combine timestamp and random string for uniqueness - const timestamp = Date.now(); - const random = Math.random().toString(36).substring(2, 8); - return `ai-conversation-${timestamp}-${random}`; -} - -export async function GET(request: NextRequest) { - console.log('Generating Agora token...'); - - // Verify required environment variables are set - if (!APP_ID || !APP_CERTIFICATE) { - console.error('Agora credentials are not set'); - return NextResponse.json( - { error: 'Agora credentials are not set' }, - { status: 500 }, - ); - } - - // Get query parameters (if any) - const { searchParams } = new URL(request.url); - const uidStr = searchParams.get('uid') || '0'; - const uid = parseInt(uidStr); - - // Use provided channel name or generate new one - const channelName = searchParams.get('channel') || generateChannelName(); - - // Calculate token expiration time - const expirationTime = - Math.floor(Date.now() / 1000) + EXPIRATION_TIME_IN_SECONDS; - - try { - // Generate the token using Agora's Token Builder SDK (RTC + RTM for text streaming) - console.log( - 'Building RTC+RTM token with UID:', - uid, - 'Channel:', - channelName, - ); - const token = RtcTokenBuilder.buildTokenWithRtm( - APP_ID, - APP_CERTIFICATE, - channelName, - uid, - RtcRole.PUBLISHER, // User can publish audio/video - expirationTime, - expirationTime, - ); - - console.log('Token generated successfully (RTC + RTM)'); - // Return the token and session information to the client - return NextResponse.json({ - token, - uid: uid.toString(), - channel: channelName, - }); - } catch (error) { - console.error('Error generating Agora token:', error); - return NextResponse.json( - { error: 'Failed to generate Agora token', details: error }, - { status: 500 }, - ); - } -} -``` - -This route handles token generation for our application, so let's recap the important features: - -- Generates unique channel names using timestamps and random strings to avoid collisions -- Generates a secure token using the App ID and Certificate -- Accepts url parameters for refreshing tokens using an existing channel name and user ID -- Uses `buildTokenWithRtm` to generate a combined RTC + RTM token — RTM is needed for the transcript feature we'll add later - -> **Note:** This route is loading the APP_ID and APP_CERTIFICATE from the environment variables, so make sure to set them in your `.env.local` file. - -### Updating the Landing Page to Request Tokens - -With the token route setup, let's update the landing page to handle all session setup logic. First, we'll need to create some type definitions. - -Create a file at `types/conversation.ts`: - -```bash -touch types/conversation.ts -``` - -Add the following code: - -```typescript -import type { RTMClient } from 'agora-rtm'; - -// Types for Agora token data -export interface AgoraTokenData { - token: string; - uid: string; - channel: string; - agentId?: string; -} - -export interface AgoraRenewalTokens { - rtcToken: string; - rtmToken: string; -} - -// Props for our conversation component -export interface ConversationComponentProps { - agoraData: AgoraTokenData; - rtmClient: RTMClient; // RTM client for transcript delivery - onTokenWillExpire: (uid: string) => Promise; - onEndConversation: () => void; -} - -// Types for the agent invitation API -export interface ClientStartRequest { - requester_id: string; - channel_name: string; -} - -export interface AgentResponse { - agent_id: string; - create_ts: number; - state: string; -} - -export interface StopConversationRequest { - agent_id: string; -} -``` - -Open the `components/LandingPage.tsx` file and update it with the full implementation. The key additions here are: - -- **Module preloading**: Kick off dynamic imports on mount so they're cached when the user clicks "Try it Now" — this eliminates the ~1–2 second import delay -- **Parallel setup**: Agent invite and RTM login run in `Promise.all` — both only need the token, so there's no reason to run them sequentially -- **RTM client**: Created here and passed down to `ConversationComponent` so the toolkit can subscribe to transcript messages - -```typescript -'use client'; - -import { useState, useRef, Suspense, useEffect } from 'react'; -import dynamic from 'next/dynamic'; -import type { RTMClient } from 'agora-rtm'; -import type { - AgoraTokenData, - ClientStartRequest, - AgentResponse, -} from '../types/conversation'; - -// Dynamically import the ConversationComponent with ssr disabled -const ConversationComponent = dynamic(() => import('./ConversationComponent'), { - ssr: false, -}); - -// Dynamically import AgoraRTC and AgoraRTCProvider -const AgoraProvider = dynamic( - async () => { - const { AgoraRTCProvider, default: AgoraRTC } = await import('agora-rtc-react'); - return { - default: ({ children }: { children: React.ReactNode }) => { - const clientRef = useRef | null>(null); - if (!clientRef.current) { - clientRef.current = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' }); - } - return {children}; - }, - }; - }, - { ssr: false } -); - -export default function LandingPage() { - const [showConversation, setShowConversation] = useState(false); - - // Preload heavy modules on mount so they're already cached when the user - // clicks "Try it Now" — eliminates the dynamic-import delay on first click. - useEffect(() => { - import('agora-rtc-react').catch(() => {}); - import('agora-rtm').catch(() => {}); - }, []); - - const [isLoading, setIsLoading] = useState(false); - const [error, setError] = useState(null); - const [agoraData, setAgoraData] = useState(null); - const [rtmClient, setRtmClient] = useState(null); - const [agentJoinError, setAgentJoinError] = useState(false); - - const handleStartConversation = async () => { - setIsLoading(true); - setError(null); - setAgentJoinError(false); - - try { - // Step 1: Fetch RTC token + channel - console.log('Fetching Agora token...'); - const agoraResponse = await fetch('/api/generate-agora-token'); - const responseData = await agoraResponse.json(); - - if (!agoraResponse.ok) { - throw new Error(`Failed to generate Agora token: ${JSON.stringify(responseData)}`); - } - - // Step 2: Run agent invite and RTM setup in parallel — both only need the token. - // RTM must be ready before ConversationComponent mounts (the toolkit needs - // a fully-connected RTM client). Agent invite is non-fatal. - const [agentData, rtm] = await Promise.all([ - // 2a. Start the AI agent - fetch('/api/invite-agent', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - requester_id: responseData.uid, - channel_name: responseData.channel, - } as ClientStartRequest), - }) - .then(async (res) => { - if (!res.ok) { setAgentJoinError(true); return null; } - return res.json() as Promise; - }) - .catch((err) => { - console.error('Failed to start conversation with agent:', err); - setAgentJoinError(true); - return null; - }), - - // 2b. Set up RTM for transcript delivery - (async () => { - const { default: AgoraRTM } = await import('agora-rtm'); - const rtm = new AgoraRTM.RTM( - process.env.NEXT_PUBLIC_AGORA_APP_ID!, - String(Date.now()) - ); - await rtm.login({ token: responseData.token }); - await rtm.subscribe(responseData.channel); - console.log('RTM ready, channel:', responseData.channel); - return rtm as RTMClient; - })(), - ]); - - // Step 3: All dependencies ready — store state and show conversation - setRtmClient(rtm); - setAgoraData({ ...responseData, agentId: agentData?.agent_id }); - setShowConversation(true); - } catch (err) { - setError('Failed to start conversation. Please try again.'); - console.error('Error starting conversation:', err); - } finally { - setIsLoading(false); - } - }; - - const handleTokenWillExpire = async (uid: string) => { - try { - const response = await fetch( - `/api/generate-agora-token?channel=${agoraData?.channel}&uid=${uid}` - ); - const data = await response.json(); - if (!response.ok) throw new Error('Failed to generate new token'); - return data.token; - } catch (error) { - console.error('Error renewing token:', error); - throw error; - } - }; - - const handleEndConversation = async () => { - // Stop the AI agent - if (agoraData?.agentId) { - try { - const response = await fetch('/api/stop-conversation', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ agent_id: agoraData.agentId }), - }); - if (!response.ok) { - console.error('Failed to stop agent:', await response.text()); - } else { - console.log('Agent stopped successfully'); - } - } catch (error) { - console.error('Error stopping agent:', error); - } - } - - // Tear down RTM — owned here since we created it here - rtmClient?.logout().catch((err) => console.error('RTM logout error:', err)); - setRtmClient(null); - setShowConversation(false); - }; - - return ( -
-
-
-

Speak with Agent

- {!showConversation && ( -

- Experience the power of Agora's Conversational AI Engine. -

- )} - {!showConversation ? ( - <> - - {error &&

{error}

} - - ) : agoraData && rtmClient ? ( - <> - {agentJoinError && ( -
- Failed to connect with AI agent. The conversation may not work - as expected. -
- )} - Loading conversation...
}> - - - - - - ) : ( -

Failed to load conversation data.

- )} -
-
- - ); -} -``` - -### Updating the Conversation Component to Use Tokens - -Now update `ConversationComponent` to accept the props from `LandingPage`. The component also gains proper token renewal and an end-conversation button. - -```typescript -// Previous imports remain the same as before... -import type { ConversationComponentProps } from '../types/conversation'; - -export default function ConversationComponent({ - agoraData, - rtmClient, - onTokenWillExpire, - onEndConversation, -}: ConversationComponentProps) { - // ... existing state declarations ... - const [joinedUID, setJoinedUID] = useState(0); - - // isReady guard (same pattern as above) - const [isReady, setIsReady] = useState(false); - useEffect(() => { - let cancelled = false; - const id = setTimeout(() => { - if (!cancelled) setIsReady(true); - }, 0); - return () => { - cancelled = true; - clearTimeout(id); - setIsReady(false); - }; - }, []); - - // Update useJoin to use token and channel from props - const { isConnected: joinSuccess } = useJoin( - { - appid: process.env.NEXT_PUBLIC_AGORA_APP_ID!, - channel: agoraData.channel, - token: agoraData.token, - uid: parseInt(agoraData.uid, 10) || 0, - }, - isReady, - ); - - const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); - - // Capture the RTC-assigned UID for token renewal - useEffect(() => { - if (joinSuccess && client) { - setJoinedUID(client.uid as UID); - console.log('Join successful, using UID:', client.uid); - } - }, [joinSuccess, client]); - - // Token renewal — RTC and RTM now renew with separate tokens - const handleTokenWillExpire = useCallback(async () => { - if (!onTokenWillExpire || !joinedUID) return; - try { - const { rtcToken, rtmToken } = await onTokenWillExpire( - joinedUID.toString(), - ); - await client?.renewToken(rtcToken); - await rtmClient.renewToken(rtmToken); - console.log('Successfully renewed Agora RTC and RTM tokens'); - } catch (error) { - console.error('Failed to renew Agora token:', error); - } - }, [client, onTokenWillExpire, joinedUID, rtmClient]); - - useClientEvent(client, 'token-privilege-will-expire', handleTokenWillExpire); - useClientEvent(client, 'connection-state-change', (curState, prevState) => { - console.log(`Connection state changed from ${prevState} to ${curState}`); - }); - - // ... rest of component -} -``` - -### Quick Test - -Now that we have our basic RTC functionality and token generation working, let's test the application. - -1. Run the application using `pnpm dev` -2. Open the application in your browser, using the url `http://localhost:3000` -3. Click on the "Try it Now" button -4. You should see the connection status change to "Connected" - -## Add Agora's Conversational AI Engine - -Now that we have the basic RTC functionality working, let's integrate Agora's Conversational AI service. In this next section we'll: - -1. Create an API route for inviting the AI agent to our channel -2. Configure Agora Start Request, including our choice of LLM endpoint and TTS provider -3. Create a route for stopping the conversation - -### Invite Agent Route - -The `agora-agent-server-sdk` simplifies agent creation by handling token generation and the Agora REST API internally. Create the route file at `app/api/invite-agent/route.ts`: - -```bash -mkdir app/api/invite-agent -touch app/api/invite-agent/route.ts -``` - -Add the following code: - -```typescript -import { NextRequest, NextResponse } from 'next/server'; -import { - AgoraClient, - Agent, - Area, - DeepgramSTT, - ExpiresIn, - MiniMaxTTS, - OpenAI, -} from 'agora-agent-server-sdk'; -import { ClientStartRequest, AgentResponse } from '@/types/conversation'; -import { DEFAULT_AGENT_UID } from '@/lib/agora'; - -// System prompt that defines the agent's personality and behavior. -// Swap this out to change what the agent talks about. -const ADA_PROMPT = `You are **Ada**, a developer advocate AI from **Agora**. You help developers understand and build with Agora's Conversational AI platform. - -# What Agora Actually Is -Agora is a real-time communications company. The product you represent is the **Agora Conversational AI Engine** — it lets developers add voice AI agents to any app by connecting ASR, LLM, and TTS into a real-time pipeline over Agora's SD-RTN (Software Defined Real-Time Network). Key facts: -- The product is called the **Conversational AI Engine** (not "Chorus", not "Harmony", or any other name you might invent) -- It runs a full ASR → LLM → TTS pipeline with sub-500ms latency -- It supports Deepgram, Microsoft, and others for ASR; OpenAI, Anthropic, and others for LLM; ElevenLabs, Microsoft, and others for TTS -- Agora's SD-RTN is its global real-time network infrastructure — not "SDRTN" -- MCP in this context means **Model Context Protocol** (Anthropic's open standard for connecting AI models to tools/data), not "multi-channel processing" -- Agora does not have a product called Chorus, Harmony, or any similar name — do not invent product names - -# Honesty Rule -If you don't know a specific fact about Agora, say so plainly and suggest checking docs.agora.io. Never invent product names, feature names, or capabilities. - -# Persona & Tone -- Friendly, technically credible, concise. You're a peer who builds things, not a support agent. -- Plain English. No marketing fluff. - -# Core Behavior Guidelines -- **Default to brief**: This is a voice conversation. Keep most replies to 1–2 sentences. Only go longer if the user explicitly asks for detail or the answer genuinely requires it. -- **Never list or enumerate**: No bullet points, no numbered steps. Say the single most important thing. -- **Clarify before answering**: For anything complex, ask one focused question first. -- **Ask at most one question per turn**: Never stack questions. -- **Guide, don't lecture**: Unlock the next step, not everything at once.`; - -// First thing the agent says when a user joins the channel. -// Set NEXT_AGENT_GREETING in .env.local to override. -const GREETING = - process.env.NEXT_AGENT_GREETING ?? - `Hi there! I'm Ada, your virtual assistant from Agora. How can I help?`; - -// agentUid identifies the AI in the RTC channel — must match NEXT_PUBLIC_AGENT_UID on the client -const agentUid = process.env.NEXT_PUBLIC_AGENT_UID ?? String(DEFAULT_AGENT_UID); - -function requireEnv(name: string): string { - const value = process.env[name]; - if (!value) throw new Error(`Missing required environment variable: ${name}`); - return value; -} - -export async function POST(request: NextRequest) { - try { - const body: ClientStartRequest = await request.json(); - const { requester_id, channel_name } = body; - - const appId = requireEnv('NEXT_PUBLIC_AGORA_APP_ID'); - const appCertificate = requireEnv('NEXT_AGORA_APP_CERTIFICATE'); - - if (!channel_name || !requester_id) { - return NextResponse.json( - { error: 'channel_name and requester_id are required' }, - { status: 400 }, - ); - } - - const client = new AgoraClient({ - area: Area.US, - appId, - appCertificate, - }); - - // Default agent configuration: Agora-managed STT, LLM, and TTS. - // Optional BYOK examples can be added if you want to swap providers. - const agent = new Agent({ - name: `conversation-${Date.now()}-${Math.random().toString(36).substring(2, 8)}`, - instructions: ADA_PROMPT, - greeting: GREETING, - failureMessage: 'Please wait a moment.', - maxHistory: 50, - turnDetection: { - config: { - speech_threshold: 0.5, - start_of_speech: { - mode: 'vad', - vad_config: { - interrupt_duration_ms: 160, - prefix_padding_ms: 300, - }, - }, - end_of_speech: { - mode: 'vad', - vad_config: { - silence_duration_ms: 480, - }, - }, - }, - }, - advancedFeatures: { enable_rtm: true, enable_tools: true }, - }) - .withStt( - new DeepgramSTT({ - model: 'nova-3', - language: 'en', - }), - ) - .withLlm( - new OpenAI({ - model: 'gpt-4o-mini', - greetingMessage: GREETING, - failureMessage: 'Please wait a moment.', - maxHistory: 15, - params: { - max_tokens: 1024, - temperature: 0.7, - top_p: 0.95, - }, - }), - ) - .withTts( - new MiniMaxTTS({ - model: 'speech_2_6_turbo', - voiceId: 'English_captivating_female1', - }), - ); - - const session = agent.createSession(client, { - channel: channel_name, - agentUid, - remoteUids: [requester_id], - idleTimeout: 30, - expiresIn: ExpiresIn.hours(1), - }); - - const agentId = await session.start(); - - return NextResponse.json({ - agent_id: agentId, - create_ts: Math.floor(Date.now() / 1000), - state: 'RUNNING', - } as AgentResponse); - } catch (error) { - console.error('Error starting conversation:', error); - return NextResponse.json( - { - error: - error instanceof Error - ? error.message - : 'Failed to start conversation', - }, - { status: 500 }, - ); - } -} -``` - -This quickstart uses **`agora-agent-server-sdk` ^1.3.1** with **DeepgramSTT**, **OpenAI** (`gpt-4o-mini`), and **MiniMaxTTS** through Agora-managed defaults, so the base quickstart only needs Agora credentials. The SDK still supports other STT/LLM/TTS providers if you want to enable the optional BYOK examples. - -> **Note:** Only Agora App ID and App Certificate are required for this default agent configuration. `NEXT_PUBLIC_AGENT_UID` is an optional override that defaults to `123456`. See the environment variables reference at the end of this guide. Optional `NEXT_LLM_URL` / `NEXT_LLM_API_KEY` apply only if you enable the optional BYOK block or use the optional `app/api/chat/completions` proxy. - -### Stop Conversation Route - -After the agent joins the conversation, we need a way to remove them. The `stop-conversation` route uses the `agora-agent-server-sdk` to stop the agent. - -Create a file at `app/api/stop-conversation/route.ts`: - -```bash -mkdir app/api/stop-conversation -touch app/api/stop-conversation/route.ts -``` - -Add the following code: - -```typescript -import { NextResponse } from 'next/server'; -import { AgoraClient, Area } from 'agora-agent-server-sdk'; -import { StopConversationRequest } from '@/types/conversation'; - -export async function POST(request: Request) { - try { - const body: StopConversationRequest = await request.json(); - const { agent_id } = body; - - if (!agent_id) { - return NextResponse.json( - { error: 'agent_id is required' }, - { status: 400 }, - ); - } - - const appId = process.env.NEXT_PUBLIC_AGORA_APP_ID; - const appCertificate = process.env.NEXT_AGORA_APP_CERTIFICATE; - if (!appId || !appCertificate) { - throw new Error( - 'Missing Agora configuration. Set NEXT_PUBLIC_AGORA_APP_ID and NEXT_AGORA_APP_CERTIFICATE.', - ); - } - - const client = new AgoraClient({ - area: Area.US, - appId, - appCertificate, - }); - await client.stopAgent(agent_id); - - return NextResponse.json({ success: true }); - } catch (error) { - console.error('Error stopping conversation:', error); - return NextResponse.json( - { - error: - error instanceof Error - ? error.message - : 'Failed to stop conversation', - }, - { status: 500 }, - ); - } -} -``` - -## Add the Agora UI Kit and Toolkit - -Rather than building custom microphone buttons and audio visualizers from scratch, Agora provides two packages that handle the UI and transcript logic for you: - -- **`agora-agent-uikit`** — pre-built components: `AgentVisualizer`, `MicButtonWithVisualizer`, `ConvoTextStream` -- **`agora-agent-client-toolkit`** — `AgoraVoiceAI` class that subscribes to RTM transcript events and emits transcript, state, and error events - -Both packages are already installed. Now we'll wire them into the `ConversationComponent`. - -### How the Transcript System Works - -When the AI agent speaks (or the user speaks), the Agora Conversational AI Engine sends transcript updates over RTM. The flow is: - -1. The agent is configured with `enable_rtm: true` (done in the invite-agent route above) -2. On the client, `AgoraVoiceAI.init()` connects to the RTM channel and listens for these events -3. It fires a `TRANSCRIPT_UPDATED` event with a normalized list of transcript items -4. We remap the local user UID, adapt the transcript items into `IMessageListItem`, and pass them to `ConvoTextStream` for display - -### Microphone Selector (Optional) - -When users have multiple microphones, a device selector improves the experience. Create `components/MicrophoneSelector.tsx`: - -```bash -touch components/MicrophoneSelector.tsx -``` - -This component uses `AgoraRTC.getMicrophones()` to list devices, shows a dropdown when multiple devices exist, and supports hot-swap when devices are plugged/unplugged. It only renders when there is more than one microphone. See the implementation in `components/MicrophoneSelector.tsx` in the companion repository for the full code using shadcn `DropdownMenu`. - -### Complete ConversationComponent - -Here is the full updated `ConversationComponent` using the toolkit and uikit: - -```typescript -'use client'; - -import { useState, useEffect, useCallback, useMemo } from 'react'; -import { X } from 'lucide-react'; -import { setParameter } from 'agora-rtc-sdk-ng/esm'; -import { - useRTCClient, - useLocalMicrophoneTrack, - useRemoteUsers, - useClientEvent, - useJoin, - usePublish, - RemoteUser, - UID, -} from 'agora-rtc-react'; -import { - AgoraVoiceAI, - AgoraVoiceAIEvents, - AgentState, - MessageSalStatus, - TranscriptHelperMode, - TurnStatus, - type TranscriptHelperItem, - type UserTranscription, - type AgentTranscription, -} from 'agora-agent-client-toolkit'; -import { - AgentVisualizer, - ConvoTextStream, - type AgentVisualizerState, - type IMessageListItem, -} from 'agora-agent-uikit'; -import { MicButtonWithVisualizer } from 'agora-agent-uikit/rtc'; -import { Button } from '@/components/ui/button'; -import { DEFAULT_AGENT_UID } from '@/lib/agora'; -import { MicrophoneSelector } from './MicrophoneSelector'; -import { ConnectionStatusPanel } from './ConnectionStatusPanel'; -import { - getConversationIssueSeverity, - type ConnectionIssue, -} from './ConversationErrorCard'; -import type { ConversationComponentProps } from '@/types/conversation'; - -type ToolkitMessage = TranscriptHelperItem>; - -function mapAgentVisualizerState( - agentState: AgentState | null, - isAgentConnected: boolean, - connectionState: string, -): AgentVisualizerState { - if (connectionState === 'DISCONNECTED' || connectionState === 'DISCONNECTING') { - return 'disconnected'; - } - if (connectionState === 'CONNECTING' || connectionState === 'RECONNECTING') { - return 'joining'; - } - if (!isAgentConnected) { - return 'not-joined'; - } - switch (agentState) { - case 'listening': - return 'listening'; - case 'thinking': - return 'analyzing'; - case 'speaking': - return 'talking'; - default: - return 'ambient'; - } -} - -function toMessageListItem(item: ToolkitMessage): IMessageListItem { - return { - turn_id: item.turn_id, - uid: Number(item.uid) || 0, - text: typeof item.text === 'string' ? item.text : '', - status: item.status as unknown as IMessageListItem['status'], - createdAt: typeof item._time === 'number' ? item._time * 1000 : undefined, - }; -} - -export default function ConversationComponent({ - agoraData, - rtmClient, - onTokenWillExpire, - onEndConversation, -}: ConversationComponentProps) { - const client = useRTCClient(); - const remoteUsers = useRemoteUsers(); - const [isEnabled, setIsEnabled] = useState(true); - const [isAgentConnected, setIsAgentConnected] = useState(false); - const [isConnectionDetailsOpen, setIsConnectionDetailsOpen] = useState(false); - const [connectionState, setConnectionState] = useState('CONNECTING'); - const agentUID = process.env.NEXT_PUBLIC_AGENT_UID ?? String(DEFAULT_AGENT_UID); - const [joinedUID, setJoinedUID] = useState(0); - const [agentState, setAgentState] = useState(null); - const [connectionIssues, setConnectionIssues] = useState([]); - - // StrictMode guard: delay useJoin's ready flag until after the fake-unmount - // cycle completes. React StrictMode fires cleanup synchronously before any - // setTimeout callback, so the first (fake) mount's timeout is always cancelled. - // Only the real second mount's timeout fires, meaning useJoin joins exactly once. - const [isReady, setIsReady] = useState(false); - useEffect(() => { - let cancelled = false; - const id = setTimeout(() => { - if (!cancelled) setIsReady(true); - }, 0); - return () => { - cancelled = true; - clearTimeout(id); - setIsReady(false); - }; - }, []); - - const { isConnected: joinSuccess } = useJoin( - { - appid: process.env.NEXT_PUBLIC_AGORA_APP_ID!, - channel: agoraData.channel, - token: agoraData.token, - uid: parseInt(agoraData.uid, 10) || 0, - }, - isReady - ); - - // Create mic track only after the StrictMode cycle completes (isReady). - // Passing true here creates two tracks in StrictMode — the first publishes, - // then StrictMode cleanup closes it and the second takes over, causing an - // audio gap. Do NOT pass isEnabled — that ties track lifetime to mute state - // and breaks the Web Audio graph. Mute uses track.setEnabled() only. - const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); - - // ENABLE_AUDIO_PTS is a module-level SDK parameter (not on the client instance). - // It must be set before publishing audio for transcript timing to be accurate. - useEffect(() => { - if (!client) return; - try { - setParameter('ENABLE_AUDIO_PTS', true); - } catch (error) { - console.warn('Could not set ENABLE_AUDIO_PTS:', error); - } - }, [client]); - - // Capture the RTC-assigned UID for token renewal - useEffect(() => { - if (joinSuccess && client) { - setJoinedUID(client.uid as UID); - } - }, [joinSuccess, client]); - - // --- Transcript via AgoraVoiceAI --- - const [rawTranscript, setRawTranscript] = useState([]); - - useEffect(() => { - if (!isReady || !joinSuccess) return; - - let cancelled = false; - - (async () => { - try { - const ai = await AgoraVoiceAI.init({ - rtcEngine: client, - rtmConfig: { rtmEngine: rtmClient }, - renderMode: TranscriptHelperMode.TEXT, - enableLog: true, - }); - - if (cancelled) return; - - ai.on(AgoraVoiceAIEvents.TRANSCRIPT_UPDATED, (messages) => { - setRawTranscript([...messages] as ToolkitMessage[]); - }); - ai.on(AgoraVoiceAIEvents.AGENT_STATE_CHANGED, (_, event) => { - setAgentState(event.state); - }); - ai.on(AgoraVoiceAIEvents.MESSAGE_ERROR, (agentUserId, error) => { - setConnectionIssues((prev) => [ - { - id: `${Date.now()}-${agentUserId}-${error.code}`, - source: 'rtm', - agentUserId, - code: error.code, - message: error.message, - timestamp: error.timestamp * 1000, - }, - ...prev, - ].slice(0, 6)); - }); - ai.on(AgoraVoiceAIEvents.MESSAGE_SAL_STATUS, (agentUserId, salStatus) => { - if ( - salStatus.status === MessageSalStatus.VP_REGISTER_FAIL || - salStatus.status === MessageSalStatus.VP_REGISTER_DUPLICATE - ) { - setConnectionIssues((prev) => [ - { - id: `${Date.now()}-${agentUserId}-${salStatus.status}`, - source: 'rtm', - agentUserId, - code: salStatus.status, - message: `SAL status: ${salStatus.status}`, - timestamp: salStatus.timestamp * 1000, - }, - ...prev, - ].slice(0, 6)); - } - }); - ai.subscribeMessage(agoraData.channel); - } catch (error) { - if (!cancelled) console.error('[ConversationalAI] init error:', error); - } - })(); - - return () => { - cancelled = true; - try { - const ai = AgoraVoiceAI.getInstance(); - if (ai) { - ai.unsubscribe(); - ai.destroy(); - } - } catch {} - setRawTranscript([]); - }; - }, [isReady, joinSuccess]); - - const transcript = useMemo(() => { - const localUID = String(client.uid); - return rawTranscript.map((m) => - m.uid === '0' ? { ...m, uid: localUID } : m - ); - }, [rawTranscript, client.uid]); - - // Completed (END + INTERRUPTED) messages shown as history. - // INTERRUPTED must be included — if the agent's first turn is cut off, - // messageList stays empty and ConvoTextStream never auto-opens. - const messageList = useMemo( - () => transcript.filter((m) => m.status !== TurnStatus.IN_PROGRESS).map(toMessageListItem), - [transcript] - ); - - const currentInProgressMessage = useMemo(() => { - const m = transcript.find((x) => x.status === TurnStatus.IN_PROGRESS); - return m ? toMessageListItem(m) : null; - }, [transcript]); - - // Publish local mic once the track exists; usePublish waits for RTC connection. - usePublish([localMicrophoneTrack]); - - useClientEvent(client, 'user-joined', (user) => { - if (user.uid.toString() === agentUID) setIsAgentConnected(true); - }); - - useClientEvent(client, 'user-left', (user) => { - if (user.uid.toString() === agentUID) setIsAgentConnected(false); - }); - - // Sync isAgentConnected with remoteUsers (covers cases where user-joined/left are missed) - useEffect(() => { - setIsAgentConnected( - remoteUsers.some((user) => user.uid.toString() === agentUID) - ); - }, [remoteUsers, agentUID]); - - useClientEvent(client, 'connection-state-change', (curState) => { - setConnectionState(curState); - }); - - const handleMicToggle = useCallback(async () => { - const next = !isEnabled; - const track = localMicrophoneTrack; - if (!track) { - setIsEnabled(next); - return; - } - try { - await track.setEnabled(next); - setIsEnabled(next); - } catch (error) { - console.error('Failed to toggle microphone:', error); - } - }, [isEnabled, localMicrophoneTrack]); - - const handleTokenWillExpire = useCallback(async () => { - if (!onTokenWillExpire || !joinedUID) return; - try { - const { rtcToken, rtmToken } = await onTokenWillExpire(joinedUID.toString()); - await client?.renewToken(rtcToken); - await rtmClient.renewToken(rtmToken); - } catch (error) { - console.error('Failed to renew Agora token:', error); - } - }, [client, onTokenWillExpire, joinedUID, rtmClient]); - - useClientEvent(client, 'token-privilege-will-expire', handleTokenWillExpire); - - const connectionSeverity = useMemo<'normal' | 'warning' | 'error'>(() => { - if (connectionState === 'DISCONNECTED' || connectionState === 'DISCONNECTING') { - return 'error'; - } - if (connectionState === 'CONNECTING' || connectionState === 'RECONNECTING') { - return 'warning'; - } - if (connectionIssues.length === 0) return 'normal'; - return connectionIssues.some((issue) => getConversationIssueSeverity(issue) === 'error') - ? 'error' - : 'warning'; - }, [connectionState, connectionIssues]); - - const visualizerState = useMemo( - () => mapAgentVisualizerState(agentState, isAgentConnected, connectionState), - [agentState, isAgentConnected, connectionState] - ); - - return ( -
-
- setIsConnectionDetailsOpen((open) => !open)} - /> -
- -
- -
- - {/* Remote users keep RTC audio subscribed. The visualizer itself is driven by RTM state. */} -
- - {remoteUsers.map((user) => ( -
- -
- ))} -
- -
- - -
- - {/* Scrollable transcript panel — auto-opens when the agent's first turn completes */} - -
- ); -} -``` - -### Key uikit Components - -| Component | Import | Description | -| ------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------- | -| `AgentVisualizer` | `agora-agent-uikit` | Agent-state-driven visualizer for not-joined, listening, analyzing, talking, and disconnected states | -| `ConvoTextStream` | `agora-agent-uikit` | Floating chat panel showing live and completed transcript turns | -| `MicButtonWithVisualizer` | `agora-agent-uikit/rtc` | Mic button with built-in Web Audio visualization | -| `ConnectionStatusPanel` | local component | Compact status dot plus expandable RTM/agent issue panel | - -### Key toolkit Classes - -| Export | Description | -| --------------------------------------- | ------------------------------------------------------------------------------------- | -| `AgoraVoiceAI` | Subscribes to RTM transcript events from the AI agent | -| `AgoraVoiceAIEvents.TRANSCRIPT_UPDATED` | Fires whenever a transcript turn is created or updated | -| `TurnStatus` | `IN_PROGRESS`, `END`, `INTERRUPTED` — filters which turns to show as history vs. live | -| `TranscriptHelperMode.TEXT` | Text-only rendering mode | - -## Testing - -Now that we have all the components in place, let's finish by testing the application. - -### Starting the Development Server - -To start the development server: - -```bash -pnpm dev -``` - -> **Note:** Make sure your `.env.local` file is properly configured with all the necessary credentials. There is a complete list of environment variables at the end of this guide. - -Open your browser to `http://localhost:3000` and test. - -### Common Issues and Solutions - -- **Agent not joining**: - - Verify your Agora Conversational AI credentials - - Check console for specific error messages - - Ensure your TTS configuration is valid - -- **Audio not working**: - - Check browser permissions for microphone access - - Verify the microphone is enabled in the app - - Check if audio tracks are properly published - -- **Transcripts not appearing**: - - Confirm `enable_rtm: true` is set in the agent config - - Check that the RTM client logged in successfully before `ConversationComponent` mounted - - Verify the RTC + RTM token was generated with `buildTokenWithRtm` - -- **Token errors**: - - Verify App ID and App Certificate are correct - - Ensure token renewal logic is working - - Check for proper error handling in token-related functions - -- **Channel connection issues**: - - Check network connectivity - - Verify Agora service status - -## Customizations - -Agora Conversational AI Engine supports a number of customizations. - -### Customizing the Agent - -In the invite-agent route, the `instructions` prop shapes how the AI agent responds. Modify the `ADA_PROMPT` constant to customize the agent's personality: - -```typescript -// In app/api/invite-agent/route.ts -const ADA_PROMPT = `You are a friendly and helpful assistant named Alex. Your personality is warm, patient, and slightly humorous...`; -``` - -Update the `greeting` to control the initial message the agent speaks when joining the channel (or set `NEXT_AGENT_GREETING` in `.env.local`): - -```typescript -const GREETING = - process.env.NEXT_AGENT_GREETING ?? `Hello! How can I assist you today?`; -``` - -### Customizing the Voice - -The default agent configuration uses **MiniMaxTTS** with `model: 'speech_2_6_turbo'` and `voiceId: 'English_captivating_female1'`. To use another vendor, replace `.withTts(...)` with the matching SDK type and, if needed, enable the optional BYOK environment variables for that provider. - -### Fine-tuning Voice Activity Detection - -Adjust `turnDetection` in the Agent config to optimize conversation flow: - -```typescript -// In app/api/invite-agent/route.ts -turnDetection: { - config: { - speech_threshold: 0.6, // Sensitivity to background noise (higher = less sensitive) - start_of_speech: { - mode: 'vad', - vad_config: { - interrupt_duration_ms: 200, // ms of speech before interruption triggers - prefix_padding_ms: 400, // audio captured before speech is detected - }, - }, - end_of_speech: { - mode: 'vad', - vad_config: { - silence_duration_ms: 600, // ms of silence before turn ends - }, - }, - }, -}, -``` - -## Complete Environment Variables Reference - -Here's a complete list of environment variables for your `.env.local` file: - -``` -# Agora Configuration -NEXT_PUBLIC_AGORA_APP_ID= -NEXT_AGORA_APP_CERTIFICATE= -NEXT_PUBLIC_AGENT_UID=123456 - -# Optional BYOK examples -# NEXT_LLM_URL=https://api.openai.com/v1/chat/completions -# NEXT_LLM_API_KEY= -# NEXT_DEEPGRAM_API_KEY= -# NEXT_ELEVENLABS_API_KEY= -# NEXT_ELEVENLABS_VOICE_ID= -``` - -## Next Steps - -Congratulations! You've built a full real-time conversational AI app with Next.js and Agora. The architecture you've built — with the `agora-agent-server-sdk` on the server and the `agora-agent-client-toolkit` + `agora-agent-uikit` on the client — gives you a solid foundation to customize the agent's personality, swap LLM/TTS providers, and scale to production. - -For more information about [Agora's Conversational AI Engine](https://www.agora.io/en/products/conversational-ai-engine/) check out the [official documentation](https://docs.agora.io/en/). - -Happy building! diff --git a/DOCS/TEXT_STREAMING_GUIDE.md b/DOCS/TEXT_STREAMING_GUIDE.md deleted file mode 100644 index 718b348..0000000 --- a/DOCS/TEXT_STREAMING_GUIDE.md +++ /dev/null @@ -1,363 +0,0 @@ -# Let's Talk Text: Streaming Transcriptions in Your Conversational AI App - -![Conversation AI Client - Text Streaming Demo](../.github/assets/Conversation-Ai-Client.gif) - -So, you've built an amazing conversational AI app using Agora, maybe following our main [Conversational AI Guide](./GUIDE.md). Your users can chat with an AI, just like talking to a person. But what about _seeing_ the conversation? That's where text streaming comes in. - -This guide focuses on adding real-time text transcriptions to your audio-based AI conversations. Think of it as subtitles for your AI chat. - -## Prerequisites - -Install the Agora client toolkit and UI kit: - -```bash -pnpm add agora-agent-client-toolkit agora-agent-uikit agora-rtm -``` - -- **`agora-agent-client-toolkit`** provides the `AgoraVoiceAI` class for receiving transcription data over RTM, managing message state, and emitting updates via events. -- **`agora-agent-uikit`** provides `ConvoTextStream` for transcript UI and `AgentVisualizer` for agent state UI. -- **`agora-rtm`** is the Agora Real-Time Messaging SDK — the transport layer that carries transcript data from the agent to the client. - -Why bother adding text when the primary interaction is voice? Good question! Here's why it's a game-changer: - -1. **Accessibility is Key**: Text opens up your app to users with hearing impairments. Inclusivity matters! -2. **Memory Aid**: A text transcript lets users quickly scan back through the conversation. -3. **Loud Places, No Problem**: Transcriptions ensure the message gets through, even if the audio is hard to hear. -4. **Did it Hear Me Right?**: Seeing the transcription confirms the AI understood the user correctly (or reveals when it didn't!). -5. **Beyond Voice**: Lists, code snippets, and URLs are easier to digest visually. Text streaming enables true multi-modal interaction. - -Ready to add this superpower to your app? Let's dive in. - -## The Blueprint: How Text Streaming Fits In - -Adding text streaming involves three main players in your codebase: - -1. **The Brains (`agora-agent-client-toolkit`)**: The `AgoraVoiceAI` class. It uses RTM to receive transcription data, manages message states (e.g., "is the AI still talking?"), and emits updates via events. -2. **The Face (`ConvoTextStream` and `AgentVisualizer` from `agora-agent-uikit`)**: Pre-built UI components for transcript history and agent state. -3. **The Conductor (`ConversationComponent.tsx`)**: Handles the Agora RTC/RTM connection, initializes `AgoraVoiceAI`, subscribes to transcript and state events, remaps UIDs, adapts the transcript shape, and passes data down to the UI kit. - -Here's how they communicate: - -```mermaid -flowchart LR - A[User/AI via Agora RTM] -- Raw Data --> B(AgoraVoiceAI) - B -- TRANSCRIPT_UPDATED --> C(ConversationComponent State) - C -- messageList + currentInProgressMessage --> D(ConvoTextStream UI) -``` - -Raw data arrives from the agent over RTM. `AgoraVoiceAI` processes it and emits transcript events. `ConversationComponent` remaps UIDs and updates state. `ConvoTextStream` renders the result. - -## Following the Data: The Message Lifecycle - -```mermaid -graph LR - A[Raw Data via RTM] --> B(AgoraVoiceAI Processes); - B --> C(TRANSCRIPT_UPDATED Event); - C --> D(ConversationComponent Maps & Updates State); - D --> E(ConvoTextStream Renders); - - subgraph RTM [Agora RTM] - A - end - subgraph Toolkit [agora-agent-client-toolkit] - B - C - end - subgraph ConvoComp [ConversationComponent] - D - end - subgraph UIKit [agora-agent-uikit] - E - end -``` - -1. **RTM Stream**: Transcription data arrives via Agora RTM — both user speech and AI agent output. -2. **Message Processing**: `AgoraVoiceAI` processes raw chunks, identifies user vs. agent, and tracks turn status. -3. **Event Emission**: Emits `TRANSCRIPT_UPDATED` with a `TranscriptHelperItem[]` array. -4. **State Updates**: `ConversationComponent` remaps UIDs, separates completed and in-progress turns, updates React state. -5. **UI Rendering**: `ConvoTextStream` re-renders with the new message list. - -This pipeline handles messages that are still streaming ("in-progress"), fully delivered ("completed"), or cut off ("interrupted"). - -## Decoding the Data: Message Types - -### User Transcriptions (What the User Said) - -```typescript -interface UserTranscription { - object: 'user.transcription'; - final: boolean; // Is this the final, complete transcription? -} -``` - -The `final` flag is important — intermediate results may change slightly before the turn ends. - -### Agent Transcriptions (What the AI is Saying) - -```typescript -interface AgentTranscription { - object: 'assistant.transcription'; - quiet: boolean; // Was this generated during a quiet period? - turn_seq_id: number; // Unique ID for this conversational turn - turn_status: TurnStatus; // IN_PROGRESS, END, or INTERRUPTED -} -``` - -Agent messages often arrive word-by-word or phrase-by-phrase. The `turn_status` tells you when the AI starts and finishes speaking. - -### Message Interruptions - -When a user starts talking while the AI is still speaking, an interrupt is sent. `AgoraVoiceAI` uses this to mark the in-progress AI message as `INTERRUPTED` in the transcript history. - -## Meet `AgoraVoiceAI`: The Heart of Text Streaming - -The `AgoraVoiceAI` class from `agora-agent-client-toolkit` handles: - -1. **Listening**: Subscribes to the RTM channel to receive transcription data. -2. **Processing**: Decodes messages, identifies user vs. agent, and tracks turn status. -3. **Managing State**: Tracks whether each message is streaming (`IN_PROGRESS`), finished (`END`), or cut off (`INTERRUPTED`). -4. **Ordering & Buffering**: Ensures messages are handled in the correct sequence. -5. **Notifying**: Emits `TRANSCRIPT_UPDATED` whenever the transcript history changes. - -### Message Status - -```typescript -// From agora-agent-client-toolkit -export enum TurnStatus { - IN_PROGRESS = 0, // Still being received/streamed (AI is talking) - END = 1, // Finished normally - INTERRUPTED = 2, // Cut off before completion -} -``` - -Your UI uses this to decide whether to show a streaming indicator or treat the message as final. - -### Render Modes - -```typescript -export enum TranscriptHelperMode { - TEXT = 'text', // Each agent message chunk is a complete block (recommended) - WORD = 'word', // Word-by-word streaming when timing info is available - CHUNK = 'chunk', // Chunk-based streaming -} -``` - -This quickstart uses `TranscriptHelperMode.TEXT`. - -## Wiring Up `AgoraVoiceAI` in `ConversationComponent` - -Initialize `AgoraVoiceAI` inside a `useEffect` that fires only after `joinSuccess` — this ensures the RTC client is connected and avoids React StrictMode double-initialization. - -```typescript -import { - AgoraVoiceAI, - AgoraVoiceAIEvents, - AgentState, - TranscriptHelperMode, - TurnStatus, - type TranscriptHelperItem, - type UserTranscription, - type AgentTranscription, -} from 'agora-agent-client-toolkit'; -import { - AgentVisualizer, - ConvoTextStream, - type AgentVisualizerState, - type IMessageListItem, -} from 'agora-agent-uikit'; - -type ToolkitMessage = TranscriptHelperItem>; - -// Inside ConversationComponent: -const [rawTranscript, setRawTranscript] = useState([]); -const [agentState, setAgentState] = useState(null); - -useEffect(() => { - if (!joinSuccess || !client) return; - - let cancelled = false; - - (async () => { - try { - const ai = await AgoraVoiceAI.init({ - rtcEngine: client, - rtmConfig: { rtmEngine: rtmClient }, - renderMode: TranscriptHelperMode.TEXT, - enableLog: true, - }); - - ai.on(AgoraVoiceAIEvents.TRANSCRIPT_UPDATED, (messages) => { - if (!cancelled) setRawTranscript(messages as ToolkitMessage[]); - }); - ai.on(AgoraVoiceAIEvents.AGENT_STATE_CHANGED, (_, event) => { - if (!cancelled) setAgentState(event.state); - }); - - ai.subscribeMessage(agoraData.channel); - console.log('[ConversationalAI] toolkit connected, listening for transcripts'); - } catch (error) { - if (!cancelled) console.error('[ConversationalAI] init error:', error); - } - })(); - - return () => { - cancelled = true; - try { - const ai = AgoraVoiceAI.getInstance(); - if (ai) { - ai.unsubscribe(); - ai.destroy(); - } - } catch {} - setRawTranscript([]); - }; - // client and rtmClient are stable for the component lifetime - // eslint-disable-next-line react-hooks/exhaustive-deps -}, [joinSuccess]); -``` - -### Why `joinSuccess` instead of mount? - -React StrictMode mounts components twice in development. The fake first mount's `joinSuccess` is always `false`, so the guard prevents any double-init. By the time `joinSuccess` becomes `true`, the double-mount cycle is already done. - -### Why remap `uid === '0'`? - -The toolkit assigns `uid = "0"` to the local user's speech (since the user's actual UID is unknown at init time). The uikit's `isAIMessage` check treats `uid === 0` as an AI message. Without remapping, your own speech appears on the wrong side of the chat. - -## Separating In-Progress from Completed Messages - -This project no longer uses `transcriptToMessageList()` from the uikit at runtime. Instead, it adapts the transcript items locally into the `IMessageListItem` shape expected by `ConvoTextStream`: - -```typescript -import { TurnStatus } from 'agora-agent-client-toolkit'; -import type { IMessageListItem } from 'agora-agent-uikit'; - -function toMessageListItem(item: ToolkitMessage): IMessageListItem { - return { - turn_id: item.turn_id, - uid: Number(item.uid) || 0, - text: typeof item.text === 'string' ? item.text : '', - status: item.status as unknown as IMessageListItem['status'], - createdAt: typeof item._time === 'number' ? item._time * 1000 : undefined, - }; -} - -const transcript = useMemo(() => { - const localUID = String(client.uid); - return rawTranscript.map((m) => - m.uid === '0' ? { ...m, uid: localUID } : m - ); -}, [rawTranscript, client.uid]); - -// Completed turns (END + INTERRUPTED) become the scrollable message history. -// INTERRUPTED must be included — if the agent's first turn is cut off, -// messageList stays empty and ConvoTextStream never auto-opens. -const messageList = useMemo( - () => transcript.filter((m) => m.status !== TurnStatus.IN_PROGRESS).map(toMessageListItem), - [transcript] -); - -// The single in-progress turn, if any, is shown as a live streaming bubble. -const currentInProgressMessage = useMemo(() => { - const m = transcript.find((x) => x.status === TurnStatus.IN_PROGRESS); - return m ? toMessageListItem(m) : null; -}, [transcript]); -``` - -The local adapter is intentional. The current repo hit a browser runtime issue with the uikit converter export, so the quickstart now converts transcript items itself. - -## Rendering with `ConvoTextStream` - -Drop `ConvoTextStream` into your JSX. It handles smart scrolling, auto-open on first message, mobile responsiveness, and streaming indicators. This quickstart also renders `AgentVisualizer` from RTM agent state: - -```typescript -import { AgentVisualizer, ConvoTextStream } from 'agora-agent-uikit'; - - - -``` - -`agentUID` tells the component which messages came from the AI vs. the user, so it can render them on the correct side of the chat. - -`visualizerState` is mapped from `AgentState` values: - -- `listening` → `listening` -- `thinking` → `analyzing` -- `speaking` → `talking` -- `idle` / `silent` → `ambient` - -### What `ConvoTextStream` does for you - -- **Smart auto-scroll**: Follows new messages when you're at the bottom; stays put when you scroll up to read history -- **Auto-open**: Panel opens automatically when the first message arrives (desktop only) -- **Streaming indicator**: Shows a live animation while the agent's current turn is in progress -- **Interrupted turns**: Displays completed text even for turns the user cut off -- **Mobile layout**: compact floating panel sized above the mic controls - -## Setting Up the RTM Client - -The RTM client must be created, logged in, and subscribed to the channel **before** `ConversationComponent` mounts — the toolkit needs a fully connected RTM client at init time. - -Create it in `LandingPage` as part of the session setup: - -```typescript -const { default: AgoraRTM } = await import('agora-rtm'); -const rtm = new AgoraRTM.RTM( - process.env.NEXT_PUBLIC_AGORA_APP_ID!, - String(Date.now()) -); -await rtm.login({ token: responseData.token }); -await rtm.subscribe(responseData.channel); -``` - -Then pass it as a prop to `ConversationComponent`. On end conversation, call `rtmClient.logout()` in `LandingPage` — RTM is owned by the same scope that created it. - -For token renewal, this project now uses separate renewal requests: - -- RTC renews with the post-join RTC UID -- RTM renews with the universal `uid=0` token - -## Token Requirements - -The token used for RTM login must be generated with `RtcTokenBuilder.buildTokenWithRtm()` — a standard RTC-only token does not grant RTM access. - -```typescript -// In your token generation route: -import { RtcTokenBuilder, RtcRole } from 'agora-token'; - -const token = RtcTokenBuilder.buildTokenWithRtm( - APP_ID, - APP_CERTIFICATE, - channelName, - uid, - RtcRole.PUBLISHER, - expirationTime, - expirationTime, -); -``` - -When the token nears expiry, renew both transports with their respective tokens: - -```typescript -const { rtcToken, rtmToken } = await onTokenWillExpire(joinedUID.toString()); -await client?.renewToken(rtcToken); -await rtmClient.renewToken(rtmToken); -``` - -## Agent-Side Configuration - -Text streaming requires the agent to be started with RTM enabled. In `app/api/invite-agent/route.ts`: - -```typescript -const agent = new Agent({ - // ... - advancedFeatures: { enable_rtm: true }, -}); -``` - -Without `enable_rtm: true`, the agent joins the channel but never sends transcript messages over RTM, so `TRANSCRIPT_UPDATED` will never fire. diff --git a/app/api/chat/completions/route.ts b/app/api/chat/completions/route.ts index e8886d5..28b31e6 100644 --- a/app/api/chat/completions/route.ts +++ b/app/api/chat/completions/route.ts @@ -3,6 +3,18 @@ import { streamText } from 'ai'; import { createOpenAI } from '@ai-sdk/openai'; import { randomUUID } from 'crypto'; +type ChatBody = { + messages?: Array<{ role: string; content: unknown }>; + model?: string; + stream?: boolean; + [key: string]: unknown; +}; + +type ChatCompletionsDeps = { + createOpenAIClient: typeof createOpenAI; + streamTextImpl: typeof streamText; +}; + /** * OpenAI-compatible Chat Completions endpoint backed by Vercel AI SDK. * @@ -12,87 +24,97 @@ import { randomUUID } from 'crypto'; * Extension point: add RAG retrieval, tool calls, guards, etc. before/after * the streamText call. */ -export async function POST(request: NextRequest) { - // ── Config ──────────────────────────────────────────────────────────────── - const apiKey = process.env.NEXT_LLM_API_KEY; - const llmUrl = process.env.NEXT_LLM_URL; - // Model is pinned here — change this to switch models without other config changes. - // Never use body.model; that would allow callers to route to arbitrary models. - const modelId = 'gpt-4o'; - - if (!apiKey || !llmUrl) { - return NextResponse.json( - { error: 'NEXT_LLM_API_KEY and NEXT_LLM_URL must be set' }, - { status: 500 } - ); - } - - // @ai-sdk/openai needs a base URL, not the full /chat/completions path - const baseURL = llmUrl.replace(/\/chat\/completions\/?$/, ''); - - let body: { - messages?: Array<{ role: string; content: unknown }>; - model?: string; - stream?: boolean; - [key: string]: unknown; - }; +export function createChatCompletionsHandler({ + createOpenAIClient, + streamTextImpl, +}: ChatCompletionsDeps) { + return async function POST(request: NextRequest) { + // ── Config ──────────────────────────────────────────────────────────────── + const apiKey = process.env.NEXT_LLM_API_KEY; + const llmUrl = process.env.NEXT_LLM_URL; + // Model is pinned here — change this to switch models without other config changes. + // Never use body.model; that would allow callers to route to arbitrary models. + const modelId = 'gpt-4o'; + + if (!apiKey || !llmUrl) { + return NextResponse.json( + { error: 'NEXT_LLM_API_KEY and NEXT_LLM_URL must be set' }, + { status: 500 }, + ); + } + + // @ai-sdk/openai needs a base URL, not the full /chat/completions path + const baseURL = llmUrl.replace(/\/chat\/completions\/?$/, ''); + + let body: ChatBody; + + try { + body = await request.json(); + } catch { + return NextResponse.json({ error: 'Invalid JSON body' }, { status: 400 }); + } + + const openai = createOpenAIClient({ apiKey, baseURL }); - try { - body = await request.json(); - } catch { - return NextResponse.json({ error: 'Invalid JSON body' }, { status: 400 }); - } - - const openai = createOpenAI({ apiKey, baseURL }); - - const result = streamText({ - // modelId is always sourced from the environment — body.model is ignored - model: openai(modelId), - messages: (body.messages ?? []) as NonNullable[0]['messages']>, - }); - - const encoder = new TextEncoder(); - const id = `chatcmpl-${randomUUID()}`; - const created = Math.floor(Date.now() / 1000); - const model = body.model ?? modelId; - - const sseChunk = (delta: Record, finishReason: string | null = null) => - encoder.encode( - `data: ${JSON.stringify({ - id, - object: 'chat.completion.chunk', - created, - model, - choices: [{ index: 0, delta, finish_reason: finishReason }], - })}\n\n` - ); - - const stream = new ReadableStream({ - async start(controller) { - try { - // Role-only first chunk (OpenAI convention) - controller.enqueue(sseChunk({ role: 'assistant', content: '' })); - - for await (const chunk of result.textStream) { - controller.enqueue(sseChunk({ content: chunk })); + const result = streamTextImpl({ + // modelId is always sourced from the environment — body.model is ignored + model: openai(modelId), + messages: (body.messages ?? []) as NonNullable< + Parameters[0]['messages'] + >, + }); + + const encoder = new TextEncoder(); + const id = `chatcmpl-${randomUUID()}`; + const created = Math.floor(Date.now() / 1000); + const model = body.model ?? modelId; + + const sseChunk = ( + delta: Record, + finishReason: string | null = null, + ) => + encoder.encode( + `data: ${JSON.stringify({ + id, + object: 'chat.completion.chunk', + created, + model, + choices: [{ index: 0, delta, finish_reason: finishReason }], + })}\n\n`, + ); + + const stream = new ReadableStream({ + async start(controller) { + try { + // Role-only first chunk (OpenAI convention) + controller.enqueue(sseChunk({ role: 'assistant', content: '' })); + + for await (const chunk of result.textStream) { + controller.enqueue(sseChunk({ content: chunk })); + } + + controller.enqueue(sseChunk({}, 'stop')); + controller.enqueue(encoder.encode('data: [DONE]\n\n')); + controller.close(); + } catch (err) { + console.error('[custom-llm] Stream error:', err); + controller.error(err); } + }, + }); - controller.enqueue(sseChunk({}, 'stop')); - controller.enqueue(encoder.encode('data: [DONE]\n\n')); - controller.close(); - } catch (err) { - console.error('[custom-llm] Stream error:', err); - controller.error(err); - } - }, - }); - - return new NextResponse(stream, { - status: 200, - headers: { - 'Content-Type': 'text/event-stream', - 'Cache-Control': 'no-cache', - Connection: 'keep-alive', - }, - }); + return new NextResponse(stream, { + status: 200, + headers: { + 'Content-Type': 'text/event-stream', + 'Cache-Control': 'no-cache', + Connection: 'keep-alive', + }, + }); + }; } + +export const POST = createChatCompletionsHandler({ + createOpenAIClient: createOpenAI, + streamTextImpl: streamText, +}); diff --git a/app/api/generate-agora-token/route.ts b/app/api/generate-agora-token/route.ts index 1e6b306..c2701cd 100644 --- a/app/api/generate-agora-token/route.ts +++ b/app/api/generate-agora-token/route.ts @@ -24,7 +24,7 @@ export async function GET(request: NextRequest) { const { searchParams } = new URL(request.url); const uidStr = searchParams.get('uid'); const parsedUid = uidStr ? parseInt(uidStr, 10) : Number.NaN; - const uid = Number.isNaN(parsedUid) + const uid = Number.isNaN(parsedUid) || parsedUid <= 0 ? Math.floor(Math.random() * 9_999_000) + 1000 : parsedUid; const channelName = searchParams.get('channel') || generateChannelName(); diff --git a/docs/ai/L0_repo_card.md b/docs/ai/L0_repo_card.md new file mode 100644 index 0000000..ab5a637 --- /dev/null +++ b/docs/ai/L0_repo_card.md @@ -0,0 +1,29 @@ +# Agora Conversational AI Next.js Quickstart — Repo Card + +> Official Next.js quickstart for building browser-based voice AI agents with Agora Conversational AI Engine. + +## Identity + +| Field | Value | +| --- | --- | +| Repo | `AgoraIO-Conversational-AI/agent-quickstart-nextjs` | +| Type | `frontend-app` | +| Language | TypeScript, Next.js 16 App Router, React 19 | +| Deploy Target | Local Node.js dev server, Vercel | +| Owner | Agora Conversational AI / DevEx | +| Last Reviewed | 2026-05-22 | + +## L1 — Summaries + +The Audience column helps agents prioritise: **Use** = consuming the repo behavior, **Maintain** = changing internals. + +| File | Purpose | Audience | +| --- | --- | --- | +| [01_setup](L1/01_setup.md) | Environment setup, quick commands, env vars, verification safety | Use & Maintain | +| [02_architecture](L1/02_architecture.md) | End-to-end runtime architecture and data flow | Maintain | +| [03_code_map](L1/03_code_map.md) | Directory/module map and ownership boundaries | Maintain | +| [04_conventions](L1/04_conventions.md) | Coding conventions, lifecycle patterns, and doc sync rules | Maintain | +| [05_workflows](L1/05_workflows.md) | Common task recipes (run, modify, validate, ship) | Use & Maintain | +| [06_interfaces](L1/06_interfaces.md) | API contracts, event payloads, env contracts | Use & Maintain | +| [07_gotchas](L1/07_gotchas.md) | High-impact pitfalls and known failure modes | Maintain | +| [08_security](L1/08_security.md) | Secret handling, trust boundaries, auth/token model | Maintain | diff --git a/docs/ai/L1/01_setup.md b/docs/ai/L1/01_setup.md new file mode 100644 index 0000000..25f37ea --- /dev/null +++ b/docs/ai/L1/01_setup.md @@ -0,0 +1,110 @@ +# 01 Setup + +> Environment setup, commands, and safe verification flow for this quickstart. + +## Runtime Requirements + +- Node.js `>=22` (`package.json` engines field). +- `pnpm` package manager. +- Agora CLI (`agora`) for project binding and environment bootstrap. +- Agora project with Conversational AI enabled. + +## Install and Bootstrap + +1. Install dependencies. +2. Bind an Agora project. +3. Write `.env.local`. +4. Verify setup before running. + +```bash +pnpm install +agora login +agora project use +agora project env write .env.local +agora project doctor --deep +``` + +## Required Environment Variables + +- `NEXT_PUBLIC_AGORA_APP_ID`: Agora project App ID. +- `NEXT_AGORA_APP_CERTIFICATE`: Agora App Certificate (server only). + +Optional: + +- `NEXT_PUBLIC_AGENT_UID` (defaults to `123456`). +- `NEXT_AGENT_GREETING`. +- BYOK keys (`NEXT_DEEPGRAM_API_KEY`, `NEXT_LLM_URL`, `NEXT_LLM_API_KEY`, `NEXT_ELEVENLABS_API_KEY`, `NEXT_ELEVENLABS_VOICE_ID`). + +## Primary Commands + +```bash +pnpm run dev +pnpm run lint +pnpm run typecheck +pnpm run verify:api +pnpm run build +pnpm run verify +``` + +## Verification Safety + +Safe without live session: + +- `pnpm run lint` +- `pnpm run typecheck` +- `pnpm run verify:api` +- `pnpm run build` + +Requires env/project binding: + +- `pnpm run doctor` +- `pnpm run verify` + +## Local Run Notes + +- App + API routes run at `http://localhost:3000`. +- Session starts from `QuickstartPreCallCard` (`Try it now`) and bootstraps token + RTM + invite flow. +- If transcript or agent join fails, first run `agora project doctor --deep`. + +## CI Expectations + +- Build workflow badge exists in root `README.md`. +- Pre-ship expectation: `pnpm run verify` passes. +- Route contract tests are executed by `scripts/verify-api-contracts.ts`. + +## Troubleshooting Matrix + +| Symptom | Probable Cause | First Check | Fix Path | +| --- | --- | --- | --- | +| Agent never joins | Invite route or env mismatch | `pnpm run doctor` and invite route logs | Verify `NEXT_PUBLIC_AGENT_UID` and invite payload | +| Transcript missing | RTM token capability missing | Token route implementation | Ensure `buildTokenWithRtm` remains unchanged | +| `verify` fails at doctor | Project not bound | `agora project use` output | Re-bind project and rewrite `.env.local` | +| Mic publishes but no agent response | Agent start failed | UI warning (`agentJoinError`) | Inspect `/api/invite-agent` response | + +## Local-Only vs Deploy-Specific + +Local: + +- Uses `.env.local` created by `agora project env write`. +- Uses `next dev --webpack`. +- Best for flow debugging and transcript behavior checks. + +Vercel: + +- Requires environment vars configured per environment scope. +- Keep `NEXT_AGORA_APP_CERTIFICATE` private server variable. +- Use `pnpm run build` locally before pushing deployment changes. + +## Setup Change Checklist + +When setup docs/config change: + +1. Update `README.md` environment/commands sections. +2. Update `env.local.example` if variable set changes. +3. Update `docs/ai/L1/01_setup.md` and `L0_repo_card.md` `Last Reviewed`. +4. Run at least `pnpm run typecheck` and `pnpm run verify:api`. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Full start/join/teardown sequence. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — RTM transcript/event pipeline internals. diff --git a/docs/ai/L1/02_architecture.md b/docs/ai/L1/02_architecture.md new file mode 100644 index 0000000..1b15dbc --- /dev/null +++ b/docs/ai/L1/02_architecture.md @@ -0,0 +1,103 @@ +# 02 Architecture + +> Runtime architecture for browser RTC/RTM, Next.js route handlers, and Agora managed agent session. + +## High-Level Shape + +- Next.js App Router frontend and API routes in one deployable app. +- Browser joins Agora RTC channel and uses RTM for transcript/state/metrics/errors. +- Server-side routes mint token and call Agora Agent Server SDK. +- Agent executes STT -> LLM -> TTS pipeline in Agora cloud. + +## Component Graph + +```text +Browser UI (LandingPage + ConversationComponent) + -> GET /api/generate-agora-token + -> POST /api/invite-agent + -> RTC join/publish mic + -> RTM subscribe + AgoraVoiceAI events + -> POST /api/stop-conversation + +Next.js API routes + -> agora-token (RtcTokenBuilder.buildTokenWithRtm) + -> agora-agent-server-sdk (start/stop managed agent) + +Agora Cloud + -> Agent session (Deepgram STT + OpenAI LLM + MiniMax TTS by default) + -> RTM payloads (transcript, state, metrics, error) +``` + +## Start Sequence + +1. UI fetches RTC+RTM token and channel. +2. UI invites agent and initializes RTM in parallel. +3. UI mounts conversation view. +4. `useJoin` connects RTC once `isReady` guard passes. +5. `AgoraVoiceAI.init()` subscribes transcript/state/metrics streams. + +## End Sequence + +1. UI calls `/api/stop-conversation` with `agent_id` if present. +2. UI logs out RTM client. +3. RTC hook ownership handles leave/unpublish cleanup. +4. Component state resets to pre-call shell. + +## Core State Domains + +- Session bootstrap: `LandingPage` (`agoraData`, `rtmClient`, loading/error flags). +- RTC transport and mic: `ConversationComponent` + `agora-rtc-react` hooks. +- Transcript + agent state: `AgoraVoiceAI` events mapped through `lib/conversation.ts`. +- Metrics and connection issues: `AGENT_METRICS`, `MESSAGE_ERROR`, `SAL_STATUS`, RTM fallback parsing. + +## External Dependencies + +- `agora-rtc-react` / `agora-rtc-sdk-ng` for media transport. +- `agora-rtm` for data channel. +- `agora-agent-client-toolkit` and `agora-agent-uikit` for conversation logic/UI. +- `agora-agent-server-sdk` for managed agent lifecycle. + +## Deployment Modes + +- Local development via `pnpm run dev`. +- Vercel deployment as single Next.js app with server env vars. + +## Data and Control Boundaries + +- Browser never sees the app certificate; only receives signed short-lived tokens. +- Agent lifecycle control (`start`, `stop`) is server-routed. +- Transcript/state/metrics are data-plane RTM events from agent to browser. +- UI control-plane actions (start/end, renew) originate in `LandingPage`. + +## Internal Interfaces Between Components + +`LandingPage` -> `ConversationComponent` props: + +- `agoraData` (`token`, `uid`, `channel`, optional `agentId`) +- `rtmClient` (already logged-in and subscribed) +- `onTokenWillExpire(uid)` callback for dual-token renewal +- `onEndConversation()` callback for teardown and route stop call + +`ConversationComponent` -> child UI components: + +- normalized transcript items and current in-progress turn +- agent visualizer state derived from transport + semantic state +- connection issue list and derived severity +- recent metric window for stage latency chips + +## Why the App Router Structure Matters + +- API handlers under `app/api` co-deploy with UI and share env management. +- Client components isolate browser-only SDK usage via dynamic import and `ssr: false`. +- This avoids SSR-side access to WebRTC-dependent modules. + +## Change Impact Hints + +- Changes to token or invite routes affect both startup and renewal paths. +- Changes to transcript mapping can break both transcript panel and visualizer semantics. +- Changes to RTM setup in `LandingPage` affect toolkit subscription readiness. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Detailed bootstrapping and teardown timeline. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — Event mapping, UID remap, in-progress/completed segmentation. diff --git a/docs/ai/L1/03_code_map.md b/docs/ai/L1/03_code_map.md new file mode 100644 index 0000000..90cf9b5 --- /dev/null +++ b/docs/ai/L1/03_code_map.md @@ -0,0 +1,88 @@ +# 03 Code Map + +> Directory-level ownership map and where to change behavior safely. + +## Top-Level Layout + +```text +app/ Next.js routes + API handlers +components/ Client UI and RTC/RTM lifecycle +lib/ Shared constants and transcript helpers +scripts/ Verification and doctor helpers +docs/ Human-oriented guides +docs/ai/ Progressive disclosure docs (this system) +public/ Static assets and branding +types/ Shared TypeScript route/component contracts +``` + +## API Route Ownership (`app/api`) + +- `generate-agora-token/route.ts`: builds RTC+RTM token via `buildTokenWithRtm`. +- `invite-agent/route.ts`: validates input/env, configures and starts agent session. +- `stop-conversation/route.ts`: stops agent and handles idempotent already-stopping cases. +- `chat/completions/route.ts`: optional OpenAI-compatible SSE proxy for custom LLM path. + +## Client Ownership (`components`) + +- `LandingPage.tsx`: pre-call shell, token/invite/RTM bootstrap, conversation mount/unmount. +- `ConversationComponent.tsx`: RTC join, mic publish, toolkit init, transcript/metrics/issues state. +- `QuickstartConversationLayout.tsx`: in-call framing and slots. +- `QuickstartTranscriptPanel.tsx`: live transcript panel. +- `QuickstartPipelineMetrics.tsx`: latency chips from metrics stream. +- `ConnectionStatusPanel.tsx` + `ConversationErrorCard.tsx`: issue rendering/severity. + +## Shared Logic (`lib`) + +- `agora.ts`: default constants (`DEFAULT_AGENT_UID`). +- `conversation.ts`: transcript normalization, spacing cleanup, timestamp normalization, visualizer state mapping. + +## Validation and Tooling + +- `scripts/verify-api-contracts.ts`: imports route handlers and validates contract behavior. +- `scripts/doctor.mjs`: local setup checks consumed by `pnpm run doctor`. +- `tailwind.config.ts`: includes `agora-agent-uikit` dist classes in content scan. + +## Fast File Lookup + +- Change agent prompt/model/VAD -> `app/api/invite-agent/route.ts`. +- Change token policy/channel naming -> `app/api/generate-agora-token/route.ts`. +- Change transcript mapping behavior -> `lib/conversation.ts` + `components/ConversationComponent.tsx`. +- Change session bootstrap UX -> `components/LandingPage.tsx`. + +## Additional Component Roles + +- `QuickstartPreCallCard.tsx`: start CTA and pre-call messaging. +- `QuickstartConversationLayout.tsx`: shared in-call composition shell. +- `MicrophoneSelector.tsx`: input-device selection UI. +- `ConnectionStatusPanel.tsx`: summary + detailed connection issue panel. +- `ErrorBoundary.tsx`: runtime guardrail for conversation subtree. + +## Type Contract Locations + +- `types/conversation.ts`: request/response payloads and component prop types. +- `types/env.d.ts`: typed environment variable expectations. +- `types/jsx.d.ts` and `react-jsx.d.ts`: JSX typing support details. + +## Static and Styling Assets + +- `public/*`: icons, logos, and heading SVG assets used in pre-call/in-call experience. +- `app/globals.css` and `styles/globals.css`: baseline theme/layout styles. +- `tailwind.config.ts`: utility class scan and theme extension. + +## Verification Path Mapping + +- API contract behavior test: `scripts/verify-api-contracts.ts`. +- Environment and prerequisites check: `scripts/doctor.mjs`. +- Aggregate check chain: `pnpm run verify` script in `package.json`. + +## Ownership Boundaries + +- `components/` owns client runtime lifecycle and UI state. +- `app/api/` owns privileged operations needing app certificate. +- `lib/` owns pure transforms reusable across client modules. +- `docs/` owns human-facing implementation narrative and runbooks. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Cross-file call path during start/stop. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — Mapping and rendering flow from toolkit events. diff --git a/docs/ai/L1/04_conventions.md b/docs/ai/L1/04_conventions.md new file mode 100644 index 0000000..aeeb779 --- /dev/null +++ b/docs/ai/L1/04_conventions.md @@ -0,0 +1,87 @@ +# 04 Conventions + +> Implementation conventions that protect lifecycle correctness, transcript accuracy, and docs/code alignment. + +## React and RTC Lifecycle + +- Keep RTC client creation StrictMode-safe with `useRef`, not `useMemo`. +- Use `isReady` guard (`setTimeout(..., 0)` pattern) before `useJoin` and `useLocalMicrophoneTrack`. +- Do not disable React StrictMode as a workaround. + +## Hook Ownership Rules + +- `useJoin` owns `client.leave()`; never leave manually. +- `useLocalMicrophoneTrack` owns track lifecycle; never call `.close()` manually. +- `usePublish` owns publish state; mute with `track.setEnabled()`. + +## Transcript Conventions + +- `uid === "0"` from toolkit is local-user sentinel and must be remapped to real `client.uid`. +- Include `INTERRUPTED` in history list; only filter out `IN_PROGRESS`. +- Normalize punctuation spacing for compacted provider output. +- Normalize timestamps to milliseconds before issue rendering. + +## Token and RTM Contract + +- Token route must use `RtcTokenBuilder.buildTokenWithRtm`. +- RTC-only token builders break RTM login/subscription. +- Renewal flow independently mints RTC and RTM tokens while keeping shared channel. + +## API and Error Handling Style + +- Route handlers validate required body/env early and return clear status codes. +- Stop route is idempotent for already-stopping/not-found agent states. +- Frontend logs detailed technical errors and shows concise user-facing messages. + +## Styling and UI Kit Rules + +- Tailwind config must scan `./node_modules/agora-agent-uikit/dist/**/*.{js,mjs}`. +- Prefer existing quickstart layout components over introducing parallel shells. + +## Documentation Synchronization + +When changing workflow/contracts/ownership in `components` or `app/api`, update: + +- `README.md` +- `docs/GUIDE.md` +- `docs/TEXT_STREAMING_GUIDE.md` +- `AGENTS.md` +- relevant `docs/ai/L1/*.md` and `docs/ai/L0_repo_card.md` `Last Reviewed` + +## Git Conventions + +- Conventional commits: `type: description` or `type(scope): description`. +- Branch names: `type/short-description` lowercase with hyphens. +- No AI tool names in commit/PR text. +- No `--no-verify`, no git identity config edits. + +## Route Handler Conventions + +- Parse and validate request body at the top of handler. +- Return `400` for client payload problems and `500` for internal failures. +- Keep response payloads explicit and JSON-serializable. +- Prefer helper functions for repeated env checks or error classification. + +## Commenting and Readability Style + +- Comments should capture lifecycle reasoning or non-obvious constraints. +- Avoid comments that merely restate code. +- Keep runtime-sensitive constraints close to the guarded code path. + +## Verification Conventions + +- For route or contract changes, run `pnpm run verify:api` first. +- For UI/client runtime changes, run at least lint + typecheck + build. +- Prefer narrow checks during iteration, full `pnpm run verify` before handoff. + +## Pull Request Hygiene + +- Keep scope small and copyable for quickstart consumers. +- Include doc updates in same change when workflow/contracts are touched. +- Use present-tense lowercase conventional commit descriptions. +- Avoid adding hidden requirements not represented in `env.local.example` and README. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Why StrictMode guard and hook ownership are required. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — UID remapping and turn-status handling rationale. diff --git a/docs/ai/L1/05_workflows.md b/docs/ai/L1/05_workflows.md new file mode 100644 index 0000000..40735c1 --- /dev/null +++ b/docs/ai/L1/05_workflows.md @@ -0,0 +1,99 @@ +# 05 Workflows + +> Repeatable task recipes for common quickstart changes and validation loops. + +## Run Locally + +1. `pnpm install` +2. `agora login` +3. `agora project use ` +4. `agora project env write .env.local` +5. `pnpm run doctor` +6. `pnpm run dev` + +If start fails, run `agora project doctor --deep`. + +## Change Agent Behavior + +Target file: `app/api/invite-agent/route.ts`. + +Typical edits: + +- System prompt (`ADA_PROMPT`). +- Greeting default (`NEXT_AGENT_GREETING`). +- VAD (`turnDetection.config.*`). +- STT/LLM/TTS model/provider blocks. + +Validation path: + +1. `pnpm run lint` +2. `pnpm run typecheck` +3. `pnpm run verify:api` +4. `pnpm run build` + +## Change Token or Session Bootstrap + +Token behavior: + +- Edit `app/api/generate-agora-token/route.ts`. +- Preserve RTM-capable token generation. + +Bootstrap behavior: + +- Edit `components/LandingPage.tsx`. +- Keep invite + RTM setup parallelized before conversation mount. + +## Change Transcript Rendering + +1. Update transforms in `lib/conversation.ts`. +2. Update wiring in `components/ConversationComponent.tsx`. +3. Ensure `IN_PROGRESS` is separated from history, `INTERRUPTED` retained in history. +4. Re-check `docs/TEXT_STREAMING_GUIDE.md` for consistency. + +## Ship-Readiness Workflow + +1. Run `pnpm run verify`. +2. Confirm docs alignment (`README`, guides, `AGENTS`, `docs/ai`). +3. Use conventional commit and branch naming. + +## Progressive Disclosure Doc Workflow + +- `generate docs`: create `docs/ai/` tree when absent. +- `update docs`: refresh after workflow/interface/security changes. +- `test docs`: execute question-based validation and write `docs/ai/test-results.md`. + +## Workflow: Add a New API Route + +1. Add route under `app/api//route.ts`. +2. Define payload types in `types/conversation.ts` if shared with client. +3. Add/update contract verification in `scripts/verify-api-contracts.ts`. +4. Run `pnpm run verify:api` and `pnpm run typecheck`. +5. Update `README.md` and `docs/ai/L1/06_interfaces.md`. + +## Workflow: Modify Transcript UX + +1. Update transforms in `lib/conversation.ts`. +2. Update render usage in transcript/layout components. +3. Validate edge states (`IN_PROGRESS`, `INTERRUPTED`, empty history). +4. Reconcile guidance in `docs/TEXT_STREAMING_GUIDE.md`. +5. Run `pnpm run lint` and `pnpm run build`. + +## Workflow: Enable BYOK Provider Path + +1. Uncomment relevant provider block in invite route. +2. Add env vars to `.env.local` and `env.local.example`. +3. Keep default no-key path intact for baseline quickstart behavior. +4. Document changes in README environment section. +5. Re-run `pnpm run verify` before shipping. + +## Workflow: Docs Refresh After Runtime Changes + +1. Update L1 files matching changed subsystem. +2. Update or add L2 deep dives if L1 explanation exceeds concise bounds. +3. Bump `Last Reviewed` in `L0_repo_card.md`. +4. Re-run docs test and append retest notes for any fixes. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Full runtime sequence for bootstrap and teardown tasks. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — Required checks when editing transcript flow. diff --git a/docs/ai/L1/06_interfaces.md b/docs/ai/L1/06_interfaces.md new file mode 100644 index 0000000..9e7d68f --- /dev/null +++ b/docs/ai/L1/06_interfaces.md @@ -0,0 +1,105 @@ +# 06 Interfaces + +> Contracts at repo boundaries: API routes, env vars, runtime events, and shared TypeScript payloads. + +## HTTP Route Contracts + +### `GET /api/generate-agora-token` + +Query params: + +- `uid` optional; invalid/zero resolves to random RTM-safe UID. +- `channel` optional; defaults to generated `ai-conversation--`. + +Success response: + +```json +{ "token": "...", "uid": "1234", "channel": "ai-conversation-..." } +``` + +Failure response: `{ "error": string, "details"?: string }` with `500`. + +### `POST /api/invite-agent` + +Body (`ClientStartRequest`): + +```json +{ "requester_id": "1234", "channel_name": "ai-conversation-..." } +``` + +Success (`AgentResponse`): + +```json +{ "agent_id": "...", "create_ts": 1710000000, "state": "RUNNING" } +``` + +Validation failures return `400`; server failures return `500`. + +### `POST /api/stop-conversation` + +Body (`StopConversationRequest`): `{ "agent_id": "..." }`. + +Responses: + +- `{ "success": true }` +- `{ "success": true, "state": "already-stopping" }` for idempotent stop state +- `{ "error": string }` on failure + +### `POST /api/chat/completions` + +Optional SSE proxy path (not default runtime path). Requires `NEXT_LLM_API_KEY` and `NEXT_LLM_URL` when used. + +## Event/Data Interfaces + +- RTM transcript/state/metrics/errors consumed through `AgoraVoiceAI` event emitter. +- Raw RTM `message` event parsed as fallback for `message.error` and `message.sal_status` payloads. +- `AGENT_METRICS` payloads displayed by `QuickstartPipelineMetrics`. + +## Environment Contract + +Required: + +- `NEXT_PUBLIC_AGORA_APP_ID` +- `NEXT_AGORA_APP_CERTIFICATE` + +Optional and behavior-affecting: + +- `NEXT_PUBLIC_AGENT_UID` +- `NEXT_AGENT_GREETING` +- BYOK provider variables + +## Test Coverage for Interfaces + +- `scripts/verify-api-contracts.ts` asserts token generation, input validation, env failures, and SSE framing cases. + +## Shared Client-Side Interfaces + +From `types/conversation.ts` (high-use): + +- `AgoraTokenData`: token bootstrap payload consumed by `LandingPage`. +- `AgoraRenewalTokens`: renewal callback result (`rtcToken`, `rtmToken`). +- `ConversationComponentProps`: runtime dependencies for in-call component. + +## Interface Invariants + +- Token payload must always include `token`, `uid`, `channel`. +- Invite route requires both `requester_id` and `channel_name`. +- Stop route requires `agent_id`; missing should never be tolerated silently. +- Token route should always return UID as string for downstream compatibility. + +## Event Interface Notes + +- Metrics stream entries are append-only in component state, capped to recent window. +- Connection issue records carry `source`, `agentUserId`, code/message, timestamp. +- SAL and signaling fallback payloads are parsed defensively because message schema can vary. + +## Backward Compatibility Guidance + +- If route response shape changes, update both client consumers and contract tests in same change. +- If adding fields, keep existing fields stable to avoid quickstart consumer breakage. +- Reflect interface changes in README and L1 docs to keep sample copyable. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — How route contracts are used in sequence. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — Event-level contract mapping. diff --git a/docs/ai/L1/07_gotchas.md b/docs/ai/L1/07_gotchas.md new file mode 100644 index 0000000..c6d59d2 --- /dev/null +++ b/docs/ai/L1/07_gotchas.md @@ -0,0 +1,81 @@ +# 07 Gotchas + +> High-impact pitfalls that regularly break session startup, transcript rendering, or lifecycle cleanup. + +## Critical Runtime Pitfalls + +- Using RTC-only token generation breaks RTM login and transcript/state events. +- Removing `isReady` guard can trigger StrictMode double-initialization and duplicate/missing tracks. +- Manual `client.leave()` conflicts with `useJoin` cleanup contract. +- Manual `localMicrophoneTrack.close()` conflicts with hook-owned lifecycle. + +## Transcript Pitfalls + +- Not remapping toolkit `uid="0"` causes user turns to render as agent turns. +- Dropping `INTERRUPTED` from message history can keep transcript panel from auto-opening on first interrupted turn. +- Skipping punctuation/timestamp normalization creates inconsistent transcript readability and issue time ordering. + +## Agent Startup Pitfalls + +- Missing `NEXT_PUBLIC_AGORA_APP_ID`/`NEXT_AGORA_APP_CERTIFICATE` yields hard 500s on token/invite/stop routes. +- Mismatch between `NEXT_PUBLIC_AGENT_UID` and invite route `agentUid` causes agent presence confusion. +- RTM subscription failures may only surface through SAL status or raw signaling fallback events. + +## Frontend Lifecycle Pitfalls + +- Initializing toolkit before `joinSuccess` often causes missing subscriptions. +- Tying mic track creation directly to mute state can break visualizer audio graph. +- Failing to teardown RTM client on session end leaks subscriptions and stale events. + +## Docs/Process Pitfalls + +- Changing `components/` or `app/api/` without syncing README/GUIDE/TEXT_STREAMING/AGENTS leads to stale operator guidance. +- Updating workflows/contracts without updating `docs/ai/L1` and L0 `Last Reviewed` breaks progressive disclosure trust. + +## Fast Triage Checklist + +1. Run `agora project doctor --deep`. +2. Verify token route still uses `buildTokenWithRtm`. +3. Check `uid="0"` remap path. +4. Check `isReady` guard and hook ownership constraints. +5. Inspect connection issues panel for RTM/SAL/agent error signals. + +## Frequent Regression Patterns + +- Refactoring token route and accidentally removing RTM capability. +- Simplifying transcript list logic and unintentionally dropping interrupted turns. +- Moving toolkit init into mount-only effect and reintroducing StrictMode double-init. +- Replacing `useRef` RTC client storage with recreated client object per render. + +## Symptom-to-File Debug Guide + +| Symptom | First Files to Inspect | +| --- | --- | +| RTM login fails | `app/api/generate-agora-token/route.ts`, `components/LandingPage.tsx` | +| Agent starts but no transcript | `components/ConversationComponent.tsx`, `lib/conversation.ts` | +| Conversation hangs on end | `components/LandingPage.tsx`, `app/api/stop-conversation/route.ts` | +| Metrics panel empty | `components/ConversationComponent.tsx`, `components/QuickstartPipelineMetrics.tsx` | + +## Sandbox and Local Dev Caveats + +- `pnpm run dev` can fail in restricted environments due to port/process limits. +- Route contract checks are better suited for restricted CI/sandbox contexts. +- Some failures are environment-binding issues, not code regressions. + +## Pre-merge Gotcha Checklist + +- Confirm no manual `leave()` or `close()` lifecycle calls were introduced. +- Confirm transcript mapping still remaps sentinel local UID. +- Confirm token renewal still returns both RTC and RTM tokens. +- Confirm docs were updated when workflow/interface behavior changed. + +## Incident Learning Notes + +- Connection issue deduplication intentionally uses a small time window to avoid noisy cascades. +- Invite failures are intentionally non-fatal to allow UI fallback state visibility. +- Raw RTM fallback parsing exists because higher-level hooks may miss some signaling payloads in edge conditions. + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Start/stop race and lifecycle ownership details. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — Transcript edge cases and failure surfaces. diff --git a/docs/ai/L1/08_security.md b/docs/ai/L1/08_security.md new file mode 100644 index 0000000..86bb863 --- /dev/null +++ b/docs/ai/L1/08_security.md @@ -0,0 +1,96 @@ +# 08 Security + +> Security model, trust boundaries, secret handling, and safety controls for this quickstart. + +## Trust Boundaries + +- Browser is untrusted and never receives `NEXT_AGORA_APP_CERTIFICATE`. +- Next.js server routes hold credentials and mint scoped, expiring tokens. +- Agora cloud executes managed agent pipeline using server-issued credentials. + +## Secret Handling Rules + +- Keep `NEXT_AGORA_APP_CERTIFICATE` server-side only. +- Do not expose BYOK provider API keys to client bundles. +- Store secrets in `.env.local` for dev and deployment secret store in Vercel. +- `env.local.example` documents expected keys without real values. + +## Token Security Model + +- Tokens expire (default 1 hour). +- Token endpoint allows caller-provided UID/channel but still uses server secret signing. +- Renewal flow requests new RTC/RTM tokens near expiry. +- RTM capability is required and intentionally embedded by `buildTokenWithRtm`. + +## Input Validation and Failure Handling + +- Route handlers validate required fields and env availability. +- Errors return structured JSON with bounded detail. +- Stop route treats already-stopping/not-found agent as idempotent success. + +## Agent Behavior Safety + +- Prompt includes explicit honesty and non-hallucination policy for product claims. +- Agent failure message is constrained (`Please wait a moment.`) for degraded-path behavior. + +## Operational Security Practices + +- Run `pnpm run verify` before release changes. +- Avoid logging secrets; current logs are operational and should remain non-secret. +- Use least-privilege project bindings when managing Agora environments. + +## Known Limits + +- This quickstart is a sample app; it does not implement user auth/tenant isolation. +- If productionizing, add authenticated route access and per-user authorization checks. + +## Security Review Checklist + +1. Confirm `NEXT_AGORA_APP_CERTIFICATE` is never referenced in client files. +2. Confirm token minting still uses server route only. +3. Confirm route error payloads do not leak secrets. +4. Confirm BYOK keys are optional and remain server-side. +5. Confirm docs do not instruct users to expose secrets in public config. + +## Threat Notes for This Sample + +- Token misuse risk is bounded by token expiry but still requires secure key custody. +- Public start/stop endpoints are unauthenticated in sample form; production needs auth. +- RTM message payloads are parsed defensively but should be treated as untrusted input. + +## Hardening Steps for Productionization + +- Add authenticated identity and authorization to all mutation routes. +- Scope agent start permissions per user/session ownership. +- Add rate limiting for token and invite endpoints. +- Add request tracing IDs for security incident investigation. +- Add environment-specific secret rotation policy and monitoring. + +## Deployment Secret Checklist (Vercel) + +- `NEXT_PUBLIC_AGORA_APP_ID` set for all required environments. +- `NEXT_AGORA_APP_CERTIFICATE` set as server-side secret only. +- Optional BYOK keys set only when related provider block is enabled. +- Preview environments use non-production credentials. + +## Security-Relevant Files + +- `app/api/generate-agora-token/route.ts` +- `app/api/invite-agent/route.ts` +- `app/api/stop-conversation/route.ts` +- `env.local.example` +- `README.md` environment section + +## Audit Trigger Events + +Re-audit security docs when any of these change: + +- token issuance logic +- agent start/stop route authorization assumptions +- environment variable set or naming +- provider integration path (default vs BYOK) + +## Related Deep Dives + +- [conversation_lifecycle.md](L2/conversation_lifecycle.md) — Token issuance and renewal sequence details. +- [transcript_pipeline.md](L2/transcript_pipeline.md) — RTM event surfaces and error propagation boundaries. diff --git a/docs/ai/L1/L2/_index.md b/docs/ai/L1/L2/_index.md new file mode 100644 index 0000000..b309edf --- /dev/null +++ b/docs/ai/L1/L2/_index.md @@ -0,0 +1,9 @@ +# Deep Dives Index + +| Document | Summary | Load When | +| --- | --- | --- | +| [conversation_lifecycle.md](conversation_lifecycle.md) | End-to-end start, join, streaming, renewal, and teardown sequence | Changing session bootstrap, token flow, agent invite, or teardown logic | +| [invite_agent_config.md](invite_agent_config.md) | Managed agent prompt, VAD, model/provider chain, and session options | Editing agent behavior, BYOK provider wiring, or voice config | +| [strict_mode_lifecycle.md](strict_mode_lifecycle.md) | React StrictMode-safe RTC/toolkit initialization patterns | Modifying `useJoin`, mic track lifecycle, or `AgoraVoiceAI.init` timing | +| [token_model.md](token_model.md) | RTC+RTM token build/renewal model and failure modes | Changing token minting, renewal, UID/channel semantics, or expiry | +| [transcript_pipeline.md](transcript_pipeline.md) | Transcript/event normalization from AgoraVoiceAI and RTM fallback parsing | Changing transcript rendering, metrics/error handling, or RTM integration | diff --git a/docs/ai/L1/L2/conversation_lifecycle.md b/docs/ai/L1/L2/conversation_lifecycle.md new file mode 100644 index 0000000..c2a6ec8 --- /dev/null +++ b/docs/ai/L1/L2/conversation_lifecycle.md @@ -0,0 +1,66 @@ +# Conversation Lifecycle + +> **When to Read This:** Load this when changing how a conversation session starts, renews, or stops, especially across `LandingPage`, `ConversationComponent`, and `app/api/*` routes. + +## Overview + +This quickstart orchestrates one user session across four coupled systems: + +- Next.js browser UI state. +- Agora RTC media transport. +- Agora RTM data transport. +- Agora managed agent backend lifecycle. + +The correctness target is single-init, predictable teardown, and no leaked RTM/RTC resources. + +## Detailed Start Sequence + +1. User clicks start in `QuickstartPreCallCard`. +2. `LandingPage.handleStartConversation` calls `GET /api/generate-agora-token`. +3. In parallel: +- `POST /api/invite-agent` starts managed agent session. +- RTM client is created, logs in with token, subscribes to channel. +4. On success, `agoraData` + `rtmClient` are stored and `ConversationComponent` mounts. +5. `useJoin` and `useLocalMicrophoneTrack` remain gated by `isReady` StrictMode guard. +6. After join success, `AgoraVoiceAI.init()` binds RTC+RTM engines and subscribes messages. +7. UI renders visualizer, transcript, and metrics panels. + +## StrictMode Safety Contract + +- `isReady` flips `true` only after the fake-unmount cycle finishes. +- First fake mount timer is synchronously cancelled; second real mount timer survives. +- This keeps join and track init single-run in development. + +Breaking this guard causes duplicate client/track/toolkit initialization patterns that are hard to recover from. + +## Token Renewal Flow + +- Renewal callback receives actual joined RTC UID. +- Two renewal calls are made in parallel: +- RTC renewal for current joined UID. +- RTM renewal for original login UID. +- Both target same channel. +- Result returns `rtcToken` + `rtmToken` to caller. + +## Teardown Sequence + +1. If `agentId` exists, call `POST /api/stop-conversation`. +2. Stop route calls `client.stopAgent(agent_id)` and treats already-stopping states as success. +3. Frontend logs out RTM client and clears RTM state. +4. `showConversation` reset unmounts conversation view. +5. `agora-rtc-react` hook ownership handles leave/unpublish/track cleanup. +6. Toolkit cleanup path unsubscribes and destroys `AgoraVoiceAI` singleton. + +## Failure Modes and Recovery + +- Invite failure is non-fatal to render path: UI still mounts and shows warning. +- RTM bootstrap failure is fatal to start path because transcript pipeline depends on it. +- Stop failures are logged, but UI teardown still continues. + +## Cross-File Dependencies + +- `components/LandingPage.tsx`: bootstrap orchestration and renewal callback. +- `components/ConversationComponent.tsx`: join/toolkit/mic runtime behavior. +- `app/api/generate-agora-token/route.ts`: RTM-capable token minting. +- `app/api/invite-agent/route.ts`: agent pipeline config. +- `app/api/stop-conversation/route.ts`: idempotent stop semantics. diff --git a/docs/ai/L1/L2/invite_agent_config.md b/docs/ai/L1/L2/invite_agent_config.md new file mode 100644 index 0000000..d5750b4 --- /dev/null +++ b/docs/ai/L1/L2/invite_agent_config.md @@ -0,0 +1,152 @@ +# Invite Agent Config + +> **When to Read This:** Load this document when you are changing the agent's prompt, voice, VAD behavior, model selection, or wiring a bring-your-own-key (BYOK) provider. + +## Where It Lives + +All of the managed agent configuration is built in `app/api/invite-agent/route.ts`. The route receives `{ requester_id, channel_name }` from `LandingPage`, constructs an `Agent` from `agora-agent-server-sdk`, and starts a session bound to the requester's RTC channel. + +## Top-Level Constants + +| Constant | Default | Purpose | +| ------------------- | ---------------------------------------------------- | --------------------------------------------------------- | +| `ADA_PROMPT` | Long-form instructions for "Ada", an Agora assistant | The system prompt for the LLM. | +| `GREETING` | Friendly first line | Spoken on session start unless `NEXT_AGENT_GREETING` set. | + +`NEXT_AGENT_GREETING` overrides `GREETING` at runtime. `ADA_PROMPT` has no env override — edit the constant. + +## The Agent Builder Chain + +`AgoraClient` is constructed first — it carries the region and credentials for all API calls. `area` belongs here, not on the session. + +```ts +const client = new AgoraClient({ area: Area.US, appId, appCertificate }); + +const agent = new Agent({ + name: `conversation-${Date.now()}-${randomHex}`, + instructions: ADA_PROMPT, + greeting: process.env.NEXT_AGENT_GREETING ?? GREETING, + failureMessage: 'Please wait a moment.', + maxHistory: 50, + turnDetection: { + config: { + speech_threshold: 0.5, + start_of_speech: { /* VAD on-start params */ }, + end_of_speech: { /* VAD on-end params */ }, + }, + }, + advancedFeatures: { enable_rtm: true, enable_tools: true }, + parameters: { + data_channel: 'rtm', + enable_error_message: true, + enable_metrics: true, + }, +}) + .withStt(new DeepgramSTT({ model: 'nova-3', language: 'en' })) + .withLlm(new OpenAI({ + model: 'gpt-4o-mini', + greetingMessage: GREETING, + failureMessage: 'Please wait a moment.', + maxHistory: 15, + params: { max_tokens: 1024, temperature: 0.7, top_p: 0.95 }, + })) + .withTts(new MiniMaxTTS({ + model: 'speech_2_6_turbo', + voiceId: 'English_captivating_female1', + })); +``` + +## Session Options + +`createSession` takes the `AgoraClient` as its first argument, then the session options object. `session.start()` is called separately and returns the `agentId`. + +```ts +const session = agent.createSession(client, { + channel: channel_name, + agentUid, + remoteUids: [requester_id], + idleTimeout: 30, + expiresIn: ExpiresIn.hours(1), + debug: false, +}); +const agentId = await session.start(); +``` + +| Option | Effect | +| ------------- | ------------------------------------------------------------------------------------ | +| `channel` | The RTC channel name the agent joins. | +| `agentUid` | The UID the agent occupies in the channel — must match `NEXT_PUBLIC_AGENT_UID`. | +| `remoteUids` | Restricts the agent to the requester's UID — protects against cross-channel sniping. | +| `idleTimeout` | Seconds of silence before the session ends. | +| `expiresIn` | Hard ceiling on session length, mirrors the 1-hour RTC token. | +| `debug` | Logs Agora REST API calls to the console when `true`. | + +## Editing Each Surface + +### Change the prompt + +Edit `ADA_PROMPT`. Keep it under the LLM's context window — `gpt-4o-mini` handles thousands of tokens but very long prompts amplify latency. + +### Change the greeting + +Either edit `GREETING` (changes everyone) or set `NEXT_AGENT_GREETING` in `.env.local` / Vercel (changes the deployment only). + +### Change VAD behavior + +Edit `turnDetection.config.start_of_speech` and `turnDetection.config.end_of_speech`. Both blocks accept the new VAD param shape — do **not** revert to the deprecated `turnDetection.type: 'agora_vad'`. + +### Swap the STT model + +Replace the `DeepgramSTT` constructor. To use Deepgram with a BYOK key, set `NEXT_DEEPGRAM_API_KEY` and pass `apiKey: process.env.NEXT_DEEPGRAM_API_KEY` to the constructor. + +### Swap the LLM + +Replace `OpenAI` with another LLM class from `agora-agent-server-sdk`. For a custom URL, point the constructor at `process.env.NEXT_LLM_URL` and pass `apiKey: process.env.NEXT_LLM_API_KEY`. Wiring through `app/api/chat/completions/route.ts` is documented in `docs/ai/L1/05_workflows.md`. + +### Swap the TTS + +Replace `MiniMaxTTS`. ElevenLabs is the common BYOK choice — use `NEXT_ELEVENLABS_API_KEY` and `NEXT_ELEVENLABS_VOICE_ID`. The commented BYOK example in the route shows the constructor shape. + +## Response Contract + +On success the route returns `AgentResponse`: + +```json +{ + "agent_id": "string", + "create_ts": 1700000000, + "state": "RUNNING" +} +``` + +`agent_id` is what `LandingPage` later passes to `/api/stop-conversation`. + +## Verification + +`scripts/verify-api-contracts.ts` mocks `Agent.prototype.createSession` and asserts: + +- Missing `channel_name` or `requester_id` → `400`. +- Mocked success → `200` with `agent_id`, `create_ts`, `state`. + +After editing this file, run: + +```bash +pnpm run verify:api +pnpm run typecheck +``` + +## Failure Modes + +| Symptom | Cause | +| ------------------------------------------------------ | -------------------------------------------------------------- | +| `400 channel_name and requester_id are required` | Browser sent an empty body or wrong field names. | +| `500 Agora credentials are not set` | `NEXT_AGORA_APP_CERTIFICATE` missing in env. | +| Agent joins but never speaks | TTS misconfigured (wrong `voiceId` or missing BYOK key). | +| Agent state stuck on `IDLE` | `enable_rtm: true` missing or RTM client not subscribed yet. | +| `verify:api` fails on the route | New required field added without updating the harness. | + +## See Also + +- [Back to Workflows](../05_workflows.md) +- [Back to Interfaces](../06_interfaces.md) +- [Token Model](token_model.md) diff --git a/docs/ai/L1/L2/strict_mode_lifecycle.md b/docs/ai/L1/L2/strict_mode_lifecycle.md new file mode 100644 index 0000000..7a4de49 --- /dev/null +++ b/docs/ai/L1/L2/strict_mode_lifecycle.md @@ -0,0 +1,106 @@ +# StrictMode Lifecycle + +> **When to Read This:** Load this document when you are touching `useJoin`, `useLocalMicrophoneTrack`, `usePublish`, the `AgoraRTCProvider` wiring, or `AgoraVoiceAI.init` — anything that could fire twice under React StrictMode. + +## Why It Matters + +React 19 StrictMode runs the dev lifecycle as mount → unmount → mount. RTC and `AgoraVoiceAI` hold real network and device resources. A naive integration: + +- Joins the RTC channel twice and gets one rejection plus one ghost connection. +- Acquires two microphone tracks and leaves one orphaned. +- Initializes `AgoraVoiceAI` twice, doubling RTM subscribers and transcript handlers. + +Setting `reactStrictMode: false` masks the issue and lets it surface later in production HMR-equivalent paths. + +## The `isReady` Pattern + +`components/ConversationComponent.tsx`: + +```tsx +const [isReady, setIsReady] = useState(false); +useEffect(() => { + let cancelled = false; + const id = setTimeout(() => { + if (!cancelled) setIsReady(true); + }, 0); + return () => { + cancelled = true; + clearTimeout(id); + setIsReady(false); + }; +}, []); + +const { isConnected: joinSuccess } = useJoin(config, isReady); +const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady); +``` + +The cleanup fires synchronously before any `setTimeout(..., 0)` task runs. In StrictMode's fake-unmount, that cancels the first scheduled `setIsReady(true)`. Only the real second mount actually flips `isReady` to true, and the join/mic hooks activate exactly once. + +## RTC Client in `useRef` + +The RTC client lives inside a dynamically imported `AgoraRTCProvider`: + +- `useRef` keeps the same instance across StrictMode mounts. +- `useMemo` would recreate the client on the second mount and break `useJoin`'s cleanup of the first one. +- The `AgoraRTCProvider` itself is dynamically imported because `agora-rtc-sdk-ng` touches `window` during initialization. + +## AgoraVoiceAI Initialization Order + +`AgoraVoiceAI.init` is **async** and must be called with `await`. Because `useEffect` callbacks cannot be async directly, wrap the call in an IIFE. Use a `cancelled` flag to discard the result if the effect was cleaned up before `init` resolved. + +```tsx +useEffect(() => { + if (!isReady || !joinSuccess) return; + let cancelled = false; + (async () => { + const ai = await AgoraVoiceAI.init({ + rtcEngine: client, + rtmConfig: { rtmEngine: rtmClient }, + renderMode: TranscriptHelperMode.TEXT, + enableLog: true, + }); + if (cancelled) return; + // attach listeners + ai.subscribeMessage(channel) + })(); + return () => { + cancelled = true; + try { + const ai = AgoraVoiceAI.getInstance(); + if (ai) { ai.unsubscribe(); ai.destroy(); } + } catch {} + }; +}, [isReady, joinSuccess]); +``` + +Critical points: + +- The effect depends on `isReady` AND `joinSuccess`. Both must be true. +- Once `isReady` is true, it does not flip back to false during the same real mount. React does not re-run this effect for later changes to `joinSuccess` going `false → true → false`. +- There is no `disconnect()` method — cleanup is `ai.unsubscribe()` followed by `ai.destroy()`. +- The cleanup tears down `AgoraVoiceAI` so the next real mount starts clean. + +## Hook Ownership Rules (do not break) + +| Hook | Owns | Anti-pattern | +| -------------------------- | ----------------------------- | -------------------------------------------------- | +| `useJoin` | `client.leave()` | Calling `client.leave()` manually in cleanup | +| `useLocalMicrophoneTrack` | Track creation + `.close()` | Calling `track.close()` after StrictMode unmount | +| `usePublish` | publish state | Manually `unpublish` to mute (use `setEnabled`) | + +If you need to mute, call `localMicrophoneTrack.setEnabled(false)`. The hooks will publish/unpublish correctly when the track flips state. + +## Failure Modes If You Break the Pattern + +| Symptom | Likely cause | +| ------------------------------------------------------ | -------------------------------------------------- | +| Two simultaneous RTC sessions, one rejected | `useJoin` activated before `isReady` settled | +| Microphone busy / device errors in dev | Track created twice; second one orphaned | +| Transcript events duplicated | `AgoraVoiceAI.init` ran twice | +| `client.leave is undefined` during cleanup | Client recreated via `useMemo` mid-flight | +| Agent state never advances past `IDLE` | RTM `subscribeMessage` ran before RTM was logged in | + +## See Also + +- [Back to Conventions](../04_conventions.md) +- [Back to Gotchas](../07_gotchas.md) +- [Transcript Pipeline](transcript_pipeline.md) diff --git a/docs/ai/L1/L2/token_model.md b/docs/ai/L1/L2/token_model.md new file mode 100644 index 0000000..4c6a913 --- /dev/null +++ b/docs/ai/L1/L2/token_model.md @@ -0,0 +1,100 @@ +# Token Model + +> **When to Read This:** Load this document when you are changing how tokens are built, renewed, or distributed between RTC and RTM clients. + +## The One Token Rule + +This quickstart issues **one** token string per request that carries both RTC and RTM privileges. The builder used is `RtcTokenBuilder.buildTokenWithRtm` from `agora-token`. An RTC-only token does not authorize RTM login — using `buildTokenWithUid` will silently break RTM. + +## Token Build + +`app/api/generate-agora-token/route.ts` (shape): + +```ts +const EXPIRATION_TIME_IN_SECONDS = 3600; +const uid = parsedUidFromQuery > 0 ? parsedUidFromQuery : generateUid(); +const channel = parsedChannelFromQuery ?? randomChannelName(); + +const token = RtcTokenBuilder.buildTokenWithRtm( + appId, + appCertificate, + channel, + String(uid), + role, + EXPIRATION_TIME_IN_SECONDS, + EXPIRATION_TIME_IN_SECONDS, +); + +return NextResponse.json({ token, uid: String(uid), channel }); +``` + +Notes: + +- `uid` is stringified in the response but the browser parses it numerically before calling `useJoin`. +- `uid=0`, negative UIDs, and missing UIDs all generate a non-zero UID. Agora RTC accepts `0` as auto-assign, but RTM login needs the token subject to match a concrete non-zero user ID. +- `channel` is generated server-side when the caller omits it. The browser uses whatever the route returns. +- `EXPIRATION_TIME_IN_SECONDS` is 1 hour — keep it aligned with the `expiresIn: ExpiresIn.hours(1)` value in `invite-agent/route.ts`. + +## Initial Distribution + +``` +Browser (LandingPage) + └─▶ GET /api/generate-agora-token + └─▶ { token, uid, channel } + ├─▶ useJoin(uid, token, channel) ← RTC + └─▶ rtmClient.login({ token }) ← RTM (same token string) +``` + +## Token Renewal Sequence + +RTC fires `token-privilege-will-expire` roughly 30s before expiry. The handler does NOT renew with a single token — it fetches two: + +```ts +async function handleTokenWillExpire(joinedUid: UID) { + if (!joinedUid) return; // skip if RTC never reported a uid + const [rtcRes, rtmRes] = await Promise.all([ + fetch(`/api/generate-agora-token?uid=${joinedUid}&channel=${agoraData.channel}`), + fetch(`/api/generate-agora-token?uid=${agoraData.uid}&channel=${agoraData.channel}`), + ]); + const rtcJson = await rtcRes.json(); + const rtmJson = await rtmRes.json(); + await client.renewToken(rtcJson.token); + await rtmClient.renewToken(rtmJson.token); +} +``` + +Why two fetches? + +- The browser may have joined RTC with a server-assigned UID different from the one used at RTM login (RTM logs in with the original `agoraData.uid`). +- Renewing RTC and RTM separately keeps each client's identity stable across renewal. + +## Verification Coverage + +`scripts/verify-api-contracts.ts` mocks `RtcTokenBuilder.buildTokenWithRtm` to return a sentinel string and asserts: + +- `200` status. +- Response includes `token`, `uid`, `channel`, and `uid=0` returns a generated non-zero UID. +- The mock was called with the expected arity (8 args). + +If you change the builder signature or expiry, update the harness. + +## Failure Modes + +| Symptom | Cause | +| ---------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| RTM login throws `INVALID_TOKEN` | Token built with RTC-only builder. | +| RTC disconnects ~1 hour into a call with no renewal | `handleTokenWillExpire` not wired or returned early because `joinedUid` was 0. | +| RTM keeps working but RTC drops | Only RTC `renewToken` failed; check `rtcRes` JSON for `error`. | +| `500 Agora credentials are not set` | `NEXT_AGORA_APP_CERTIFICATE` missing or empty in server env. | + +## Security Considerations + +- The certificate never leaves the server. The route returns the signed token only. +- The route does not authenticate the caller — see `08_security.md`. +- Token expiry is the only revocation mechanism; if you need session-level revocation, surface a "stop-conversation" call to clients and rely on `/api/stop-conversation` to terminate the agent. + +## See Also + +- [Back to Setup](../01_setup.md) +- [Back to Interfaces](../06_interfaces.md) +- [Back to Security](../08_security.md) diff --git a/docs/ai/L1/L2/transcript_pipeline.md b/docs/ai/L1/L2/transcript_pipeline.md new file mode 100644 index 0000000..d264714 --- /dev/null +++ b/docs/ai/L1/L2/transcript_pipeline.md @@ -0,0 +1,59 @@ +# Transcript Pipeline + +> **When to Read This:** Load this when modifying transcript rendering, event handling, UID mapping, metrics/error panels, or RTM fallback parsing. + +## Overview + +Transcript and call diagnostics are delivered through AgoraVoiceAI events over RTM, then normalized for UI kit components. + +Primary pipeline: + +1. RTM/RTC data arrives. +2. `AgoraVoiceAI` emits transcript/state/metrics/error events. +3. `ConversationComponent` stores raw transcript and issue streams. +4. `lib/conversation.ts` normalizes UID/text/timestamps. +5. UI panels render message history + in-progress turn separately. + +## Transcript Normalization Stages + +- UID remap: toolkit `uid="0"` -> actual local RTC UID string. +- Text cleanup: punctuation spacing normalization. +- Timestamp normalization: seconds vs milliseconds unified. +- Status segmentation: +- `IN_PROGRESS` displayed as live bubble. +- `END` + `INTERRUPTED` included in scrollable history. + +## Why `INTERRUPTED` Must Stay in History + +If the first agent turn is interrupted and omitted, `messageList` can remain empty and transcript auto-open logic may never engage. Keeping interrupted turns preserves user-visible causality. + +## Agent Visualizer Mapping + +`mapAgentVisualizerState` prioritizes RTC transport state before agent semantic state: + +- `DISCONNECTED`/`DISCONNECTING` -> `disconnected` +- `CONNECTING`/`RECONNECTING` -> `joining` +- no agent presence -> `not-joined` +- otherwise map agent states (`listening`, `thinking`, `speaking`, idle-like) + +This avoids showing optimistic speech/listen states while transport is degraded. + +## Metrics and Error Streams + +Handled signals: + +- `AGENT_METRICS` for per-stage latency chips. +- `MESSAGE_ERROR` and `AGENT_ERROR` for surfaced issues. +- `MESSAGE_SAL_STATUS` for registration failures (`VP_REGISTER_FAIL`, `VP_REGISTER_DUPLICATE`). + +Fallback parser: + +- Raw RTM `message` listener parses JSON payloads for `message.error` and `message.sal_status` to catch issues not raised via higher-level events. + +## Coupled Files + +- `components/ConversationComponent.tsx`: event subscriptions and issue collection. +- `lib/conversation.ts`: transcript + visualizer mapping transforms. +- `components/QuickstartTranscriptPanel.tsx`: transcript render sink. +- `components/QuickstartPipelineMetrics.tsx`: metrics render sink. +- `components/ConnectionStatusPanel.tsx`: surfaced connection diagnostics. diff --git a/docs/ai/RECIPE.md b/docs/ai/RECIPE.md new file mode 100644 index 0000000..c9c6cb3 --- /dev/null +++ b/docs/ai/RECIPE.md @@ -0,0 +1,41 @@ +# Quickstart Recipe Profile + +This repo is a reusable quickstart sample for building browser voice-agent experiences with Agora Conversational AI Engine. + +## Recipe Role + +- Role: `base` quickstart recipe. +- Target audience: developers bootstrapping a production-style Next.js voice agent app. +- Reuse model: clone, bind project, run, then customize prompt/pipeline/UI. + +## Stable Contracts Consumers Rely On + +- Route interfaces: +- `GET /api/generate-agora-token` +- `POST /api/invite-agent` +- `POST /api/stop-conversation` +- Session bootstrap ownership: +- `components/LandingPage.tsx` owns pre-call bootstrap and RTM client lifecycle. +- `components/ConversationComponent.tsx` owns joined-session RTC/toolkit lifecycle. +- Transcript normalization helper boundary in `lib/conversation.ts`. + +## Supported Customization Surface + +- Agent persona, greeting, VAD, STT/LLM/TTS models in `app/api/invite-agent/route.ts`. +- Pre-call/in-call UX in `components/QuickstartPreCallCard.tsx` and layout components. +- Optional BYOK path through commented provider blocks and env vars. + +## Invariants + +- Keep `RtcTokenBuilder.buildTokenWithRtm` for RTM-capable tokens. +- Preserve StrictMode `isReady` guard for join/mic initialization. +- Preserve UID remap (`uid="0"`) and `INTERRUPTED` message-list inclusion. +- Keep documentation synchronized when workflows/contracts change. + +## Consumer Onboarding Recipe + +1. Clone or scaffold from template. +2. Bind Agora project and write `.env.local`. +3. Run `pnpm run doctor` and `pnpm run dev`. +4. Validate with `pnpm run verify` before sharing modifications. +5. Customize agent behavior and UI using the supported surfaces above. diff --git a/docs/ai/test-results.md b/docs/ai/test-results.md new file mode 100644 index 0000000..f5e8840 --- /dev/null +++ b/docs/ai/test-results.md @@ -0,0 +1,77 @@ +# PD Documentation Test Results + +Tested: 2026-05-22 +Agent: Codex (GPT-5) +Repo: AgoraIO-Conversational-AI/agent-quickstart-nextjs + +## Summary + +- Total questions: 10 +- Passed: 10 (correct answer, right level) +- L1 gaps: 0 (needed L2 but L1 should have sufficed) +- L2 gaps: 0 (needed L2 that doesn't exist) +- Cross-ref issues: 0 (L2 exists but wasn't found) + +## Structural Check Findings (Initial Pass) + +- `07_gotchas.md` and `08_security.md` were below the 80-line L1 minimum. +- `CLAUDE.md` had an AGENTS reference, but lacked the explicit `@AGENTS.md` line required by the workflow. + +These findings were fixed before running the question pass. + +## Results + +### Setup & Build + +| # | Question | Answer Correct? | Files Read | Level Loaded | Result | +| - | -------- | --------------- | ---------- | ------------ | ------ | +| 1 | How do I install dependencies and bootstrap local env? | Yes | L0, 01_setup, README | L0+L1 | Pass | +| 2 | What env vars are required for the base quickstart? | Yes | 01_setup, 06_interfaces, README | L1 | Pass | +| 3 | What command should run before shipping? | Yes | 01_setup, 05_workflows, AGENTS | L1 | Pass | + +### Test & Run + +| # | Question | Answer Correct? | Files Read | Level Loaded | Result | +| - | -------- | --------------- | ---------- | ------------ | ------ | +| 4 | How do I run the route contract checks? | Yes | 01_setup, 03_code_map, package.json | L1 | Pass | +| 5 | How do I start the project locally and diagnose join failures? | Yes | 01_setup, 07_gotchas, README | L1 | Pass | + +### Conventions + +| # | Question | Answer Correct? | Files Read | Level Loaded | Result | +| - | -------- | --------------- | ---------- | ------------ | ------ | +| 6 | What lifecycle ownership rules must not be violated for RTC hooks? | Yes | 04_conventions, AGENTS | L1 | Pass | +| 7 | What git naming and commit format conventions apply? | Yes | AGENTS, 04_conventions | L1 | Pass | + +### Development + +| # | Question | Answer Correct? | Files Read | Level Loaded | Result | +| - | -------- | --------------- | ---------- | ------------ | ------ | +| 8 | How would I modify the agent prompt/model/VAD settings safely? | Yes | 05_workflows, 03_code_map, 02_architecture | L1 | Pass | +| 9 | Where should I edit transcript mapping and what invariants must stay intact? | Yes | 03_code_map, 04_conventions, 07_gotchas | L1 | Pass | + +### Deep Dive + +| # | Question | Answer Correct? | Files Read | Level Loaded | Result | +| - | -------- | --------------- | ---------- | ------------ | ------ | +| 10 | Why does the app separate in-progress vs completed transcript turns and keep INTERRUPTED turns? | Yes | 07_gotchas, transcript_pipeline.md | L1+L2 | Pass | + +## Recommended Fixes + +- [x] Expand `07_gotchas.md` and `08_security.md` to meet L1 line-budget requirements. +- [x] Add explicit `@AGENTS.md` reference in `CLAUDE.md` while preserving existing content. + +## Review Fix Retest + +Retested: 2026-05-22 + +| Finding | Source checked | Docs changed | Result | Notes | +| ------- | -------------- | ------------ | ------ | ----- | +| L1 line-count minimum not met in two files | `docs/ai/L1/07_gotchas.md`, `docs/ai/L1/08_security.md`; line counts via shell | `docs/ai/L1/07_gotchas.md`, `docs/ai/L1/08_security.md` | Pass | Both files now satisfy 80-200 line rule. | +| Missing explicit `@AGENTS.md` reference in CLAUDE file | `CLAUDE.md`, ai-devkit workflow requirement | `CLAUDE.md` | Pass | Added explicit reference without replacing existing content. | + +## Notes on Test Method + +- Fresh sub-agent sessions were not used in this environment. +- Question pass was executed manually against generated docs and source files listed above. +- Link integrity check for `docs/ai/` relative links passed. diff --git a/package.json b/package.json index 9a17a32..71a8bb9 100644 --- a/package.json +++ b/package.json @@ -32,7 +32,7 @@ "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.454.0", - "next": "^16.2.1", + "next": "^16.2.6", "react": "^19.0.0", "react-dom": "^19.0.0", "tailwind-merge": "^2.6.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 9eb3102..4cf8615 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -54,8 +54,8 @@ importers: specifier: ^0.454.0 version: 0.454.0(react@19.2.4) next: - specifier: ^16.2.1 - version: 16.2.1(@babel/core@7.29.0)(@opentelemetry/api@1.9.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4) + specifier: ^16.2.6 + version: 16.2.6(@babel/core@7.29.0)(@opentelemetry/api@1.9.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4) react: specifier: ^19.0.0 version: 19.2.4 @@ -608,56 +608,56 @@ packages: '@napi-rs/wasm-runtime@0.2.12': resolution: {integrity: sha512-ZVWUcfwY4E/yPitQJl481FjFo3K22D6qF0DuFH6Y/nbnE11GY5uguDxZMGXPQ8WQ0128MXQD7TnfHyK4oWoIJQ==} - '@next/env@16.2.1': - resolution: {integrity: sha512-n8P/HCkIWW+gVal2Z8XqXJ6aB3J0tuM29OcHpCsobWlChH/SITBs1DFBk/HajgrwDkqqBXPbuUuzgDvUekREPg==} + '@next/env@16.2.6': + resolution: {integrity: sha512-gd8HoHN4ufj73WmR3JmVolrpJR47ILK6LouP5xElPglaVxir6e1a7VzvTvDWkOoPXT9rkkTzyCxBu4yeZfZwcw==} '@next/eslint-plugin-next@16.2.2': resolution: {integrity: sha512-IOPbWzDQ+76AtjZioaCjpIY72xNSDMnarZ2GMQ4wjNLvnJEJHqxQwGFhgnIWLV9klb4g/+amg88Tk5OXVpyLTw==} - '@next/swc-darwin-arm64@16.2.1': - resolution: {integrity: sha512-BwZ8w8YTaSEr2HIuXLMLxIdElNMPvY9fLqb20LX9A9OMGtJilhHLbCL3ggyd0TwjmMcTxi0XXt+ur1vWUoxj2Q==} + '@next/swc-darwin-arm64@16.2.6': + resolution: {integrity: sha512-ZJGkkcNfYgrrMkqOdZ7zoLa1TOy0qpcMfk/z4Mh/FKUz40gVO+HNQWqmLxf67Z5WB64DRp0dhEbyHfel+6sJUg==} engines: {node: '>= 10'} cpu: [arm64] os: [darwin] - '@next/swc-darwin-x64@16.2.1': - resolution: {integrity: sha512-/vrcE6iQSJq3uL3VGVHiXeaKbn8Es10DGTGRJnRZlkNQQk3kaNtAJg8Y6xuAlrx/6INKVjkfi5rY0iEXorZ6uA==} + '@next/swc-darwin-x64@16.2.6': + resolution: {integrity: sha512-v/YLBHIY132Ced3puBJ7YJKw1lqsCrgcNo2aRJlCEyQrrCeRJlvGlnmxhPxNQI3KE3N1DN5r9TPNPvka3nq5RQ==} engines: {node: '>= 10'} cpu: [x64] os: [darwin] - '@next/swc-linux-arm64-gnu@16.2.1': - resolution: {integrity: sha512-uLn+0BK+C31LTVbQ/QU+UaVrV0rRSJQ8RfniQAHPghDdgE+SlroYqcmFnO5iNjNfVWCyKZHYrs3Nl0mUzWxbBw==} + '@next/swc-linux-arm64-gnu@16.2.6': + resolution: {integrity: sha512-RPOvqlYBbcQjkz9VQQDZ2T2bARIjXZV1KFlt+V2Mr6SW/e4I9fcKsaA0hdyf2FHoTlsV2xnBd5Y912rP/1Ce6w==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] - '@next/swc-linux-arm64-musl@16.2.1': - resolution: {integrity: sha512-ssKq6iMRnHdnycGp9hCuGnXJZ0YPr4/wNwrfE5DbmvEcgl9+yv97/Kq3TPVDfYome1SW5geciLB9aiEqKXQjlQ==} + '@next/swc-linux-arm64-musl@16.2.6': + resolution: {integrity: sha512-URUTu1+dMkxJsPFgm+OeEvq9wf5sujw0EvgYy80TDGHTSLTnIHeqb0Eu8A3sC95IRgjejQL+kC4mw+4yPxiAXA==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] - '@next/swc-linux-x64-gnu@16.2.1': - resolution: {integrity: sha512-HQm7SrHRELJ30T1TSmT706IWovFFSRGxfgUkyWJZF/RKBMdbdRWJuFrcpDdE5vy9UXjFOx6L3mRdqH04Mmx0hg==} + '@next/swc-linux-x64-gnu@16.2.6': + resolution: {integrity: sha512-DOj182mPV8G3UkrayLoREM5YEYI+Dk5wv7Ox9xl1fFibAELEsFD0lDPfHIeILlutMMfdyhlzYPELG3peuKaurw==} engines: {node: '>= 10'} cpu: [x64] os: [linux] - '@next/swc-linux-x64-musl@16.2.1': - resolution: {integrity: sha512-aV2iUaC/5HGEpbBkE+4B8aHIudoOy5DYekAKOMSHoIYQ66y/wIVeaRx8MS2ZMdxe/HIXlMho4ubdZs/J8441Tg==} + '@next/swc-linux-x64-musl@16.2.6': + resolution: {integrity: sha512-HKQ5SP/V/ub73UvF7n/zeJlxk2kLmtL7Wzrg4WfmkjmNos5onJ2tKu7yZOPdL18A6Svfn3max29ym+ry7NkK4g==} engines: {node: '>= 10'} cpu: [x64] os: [linux] - '@next/swc-win32-arm64-msvc@16.2.1': - resolution: {integrity: sha512-IXdNgiDHaSk0ZUJ+xp0OQTdTgnpx1RCfRTalhn3cjOP+IddTMINwA7DXZrwTmGDO8SUr5q2hdP/du4DcrB1GxA==} + '@next/swc-win32-arm64-msvc@16.2.6': + resolution: {integrity: sha512-LZXpTlPyS5v7HhSmnvsLGP3iIYgYOBnc8r8ArlT55sGHV89bR2HlDdBjWQ+PY6SJMmk8TuVGFuxalnP3k/0Dwg==} engines: {node: '>= 10'} cpu: [arm64] os: [win32] - '@next/swc-win32-x64-msvc@16.2.1': - resolution: {integrity: sha512-qvU+3a39Hay+ieIztkGSbF7+mccbbg1Tk25hc4JDylf8IHjYmY/Zm64Qq1602yPyQqvie+vf5T/uPwNxDNIoeg==} + '@next/swc-win32-x64-msvc@16.2.6': + resolution: {integrity: sha512-F0+4i0h9J6C4eE3EAPWsoCk7UW/dbzOjyzxY0qnDUOYFu6FFmdZ6l97/XdV3/Nz3VYyO7UWjyEJUXkGqcoXfMA==} engines: {node: '>= 10'} cpu: [x64] os: [win32] @@ -2251,8 +2251,8 @@ packages: natural-compare@1.4.0: resolution: {integrity: sha512-OWND8ei3VtNC9h7V60qff3SVobHr996CTwgxubgyQYEpg290h9J0buyECNNJexkFm5sOajh5G116RYA1c8ZMSw==} - next@16.2.1: - resolution: {integrity: sha512-VaChzNL7o9rbfdt60HUj8tev4m6d7iC1igAy157526+cJlXOQu5LzsBXNT+xaJnTP/k+utSX5vMv7m0G+zKH+Q==} + next@16.2.6: + resolution: {integrity: sha512-qOVgKJg1+At15NpeUP+eJgCHvTCgXsogweq87Ri/Ix7PkqQHg4sdaXmSFqKlgaIXE4kW0g25LE68W87UANlHtw==} engines: {node: '>=20.9.0'} hasBin: true peerDependencies: @@ -3324,34 +3324,34 @@ snapshots: '@tybys/wasm-util': 0.10.1 optional: true - '@next/env@16.2.1': {} + '@next/env@16.2.6': {} '@next/eslint-plugin-next@16.2.2': dependencies: fast-glob: 3.3.1 - '@next/swc-darwin-arm64@16.2.1': + '@next/swc-darwin-arm64@16.2.6': optional: true - '@next/swc-darwin-x64@16.2.1': + '@next/swc-darwin-x64@16.2.6': optional: true - '@next/swc-linux-arm64-gnu@16.2.1': + '@next/swc-linux-arm64-gnu@16.2.6': optional: true - '@next/swc-linux-arm64-musl@16.2.1': + '@next/swc-linux-arm64-musl@16.2.6': optional: true - '@next/swc-linux-x64-gnu@16.2.1': + '@next/swc-linux-x64-gnu@16.2.6': optional: true - '@next/swc-linux-x64-musl@16.2.1': + '@next/swc-linux-x64-musl@16.2.6': optional: true - '@next/swc-win32-arm64-msvc@16.2.1': + '@next/swc-win32-arm64-msvc@16.2.6': optional: true - '@next/swc-win32-x64-msvc@16.2.1': + '@next/swc-win32-x64-msvc@16.2.6': optional: true '@nodelib/fs.scandir@2.1.5': @@ -5131,9 +5131,9 @@ snapshots: natural-compare@1.4.0: {} - next@16.2.1(@babel/core@7.29.0)(@opentelemetry/api@1.9.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4): + next@16.2.6(@babel/core@7.29.0)(@opentelemetry/api@1.9.0)(react-dom@19.2.4(react@19.2.4))(react@19.2.4): dependencies: - '@next/env': 16.2.1 + '@next/env': 16.2.6 '@swc/helpers': 0.5.15 baseline-browser-mapping: 2.10.10 caniuse-lite: 1.0.30001781 @@ -5142,14 +5142,14 @@ snapshots: react-dom: 19.2.4(react@19.2.4) styled-jsx: 5.1.6(@babel/core@7.29.0)(react@19.2.4) optionalDependencies: - '@next/swc-darwin-arm64': 16.2.1 - '@next/swc-darwin-x64': 16.2.1 - '@next/swc-linux-arm64-gnu': 16.2.1 - '@next/swc-linux-arm64-musl': 16.2.1 - '@next/swc-linux-x64-gnu': 16.2.1 - '@next/swc-linux-x64-musl': 16.2.1 - '@next/swc-win32-arm64-msvc': 16.2.1 - '@next/swc-win32-x64-msvc': 16.2.1 + '@next/swc-darwin-arm64': 16.2.6 + '@next/swc-darwin-x64': 16.2.6 + '@next/swc-linux-arm64-gnu': 16.2.6 + '@next/swc-linux-arm64-musl': 16.2.6 + '@next/swc-linux-x64-gnu': 16.2.6 + '@next/swc-linux-x64-musl': 16.2.6 + '@next/swc-win32-arm64-msvc': 16.2.6 + '@next/swc-win32-x64-msvc': 16.2.6 '@opentelemetry/api': 1.9.0 sharp: 0.34.5 transitivePeerDependencies: diff --git a/scripts/verify-api-contracts.ts b/scripts/verify-api-contracts.ts index da2556b..036e1f4 100644 --- a/scripts/verify-api-contracts.ts +++ b/scripts/verify-api-contracts.ts @@ -68,6 +68,230 @@ async function verifyGenerateAgoraTokenRoute() { } } +async function verifyGenerateAgoraTokenReplacesZeroUid() { + const { GET: generateAgoraToken } = + await import('../app/api/generate-agora-token/route'); + const originalBuildTokenWithRtm = RtcTokenBuilder.buildTokenWithRtm; + let tokenBuilderArgs: unknown[] | null = null; + + RtcTokenBuilder.buildTokenWithRtm = ((...args: unknown[]) => { + tokenBuilderArgs = args; + return 'mock-rtc-rtm-token'; + }) as typeof RtcTokenBuilder.buildTokenWithRtm; + + try { + const request = new NextRequest( + 'http://localhost:3000/api/generate-agora-token?uid=0&channel=test-channel', + ); + const response = await generateAgoraToken(request); + const body = await getJson(response); + + assert( + response.status === 200, + 'GET /api/generate-agora-token?uid=0 should return 200', + ); + assert( + typeof body.uid === 'string' && body.uid !== '0', + 'GET /api/generate-agora-token?uid=0 should generate an RTM-safe uid', + ); + assert( + Array.isArray(tokenBuilderArgs) && tokenBuilderArgs[3] === body.uid, + 'buildTokenWithRtm should mint the token for the generated uid', + ); + } finally { + RtcTokenBuilder.buildTokenWithRtm = originalBuildTokenWithRtm; + } +} + +async function verifyChatCompletionsMissingEnv() { + const { createChatCompletionsHandler } = + await import('../app/api/chat/completions/route'); + const originalApiKey = process.env.NEXT_LLM_API_KEY; + const originalUrl = process.env.NEXT_LLM_URL; + + delete process.env.NEXT_LLM_API_KEY; + delete process.env.NEXT_LLM_URL; + + const handler = createChatCompletionsHandler({ + createOpenAIClient: (() => { + throw new Error('createOpenAI should not be called when env is missing'); + }) as never, + streamTextImpl: (() => { + throw new Error('streamText should not be called when env is missing'); + }) as never, + }); + + try { + const request = new NextRequest( + 'http://localhost:3000/api/chat/completions', + { + body: JSON.stringify({ messages: [] }), + method: 'POST', + }, + ); + const response = await handler(request); + const body = await getJson(response); + + assert( + response.status === 500, + 'POST /api/chat/completions should reject missing LLM env', + ); + assert( + body.error === 'NEXT_LLM_API_KEY and NEXT_LLM_URL must be set', + 'POST /api/chat/completions should explain missing LLM env', + ); + } finally { + if (originalApiKey === undefined) { + delete process.env.NEXT_LLM_API_KEY; + } else { + process.env.NEXT_LLM_API_KEY = originalApiKey; + } + if (originalUrl === undefined) { + delete process.env.NEXT_LLM_URL; + } else { + process.env.NEXT_LLM_URL = originalUrl; + } + } +} + +async function verifyChatCompletionsInvalidJson() { + const { createChatCompletionsHandler } = + await import('../app/api/chat/completions/route'); + const originalApiKey = process.env.NEXT_LLM_API_KEY; + const originalUrl = process.env.NEXT_LLM_URL; + process.env.NEXT_LLM_API_KEY = 'test-key'; + process.env.NEXT_LLM_URL = 'https://example.test/v1/chat/completions'; + + const handler = createChatCompletionsHandler({ + createOpenAIClient: (() => { + throw new Error('createOpenAI should not be called for invalid JSON'); + }) as never, + streamTextImpl: (() => { + throw new Error('streamText should not be called for invalid JSON'); + }) as never, + }); + + try { + const request = new NextRequest( + 'http://localhost:3000/api/chat/completions', + { + body: '{not json', + method: 'POST', + }, + ); + const response = await handler(request); + const body = await getJson(response); + + assert( + response.status === 400, + 'POST /api/chat/completions should reject invalid JSON', + ); + assert( + body.error === 'Invalid JSON body', + 'POST /api/chat/completions should explain invalid JSON', + ); + } finally { + if (originalApiKey === undefined) { + delete process.env.NEXT_LLM_API_KEY; + } else { + process.env.NEXT_LLM_API_KEY = originalApiKey; + } + if (originalUrl === undefined) { + delete process.env.NEXT_LLM_URL; + } else { + process.env.NEXT_LLM_URL = originalUrl; + } + } +} + +async function verifyChatCompletionsSseDone() { + const { createChatCompletionsHandler } = + await import('../app/api/chat/completions/route'); + const originalApiKey = process.env.NEXT_LLM_API_KEY; + const originalUrl = process.env.NEXT_LLM_URL; + process.env.NEXT_LLM_API_KEY = 'test-key'; + process.env.NEXT_LLM_URL = 'https://example.test/v1/chat/completions'; + + let capturedBaseUrl: string | undefined; + let capturedModelId: string | undefined; + let capturedMessages: unknown; + + const handler = createChatCompletionsHandler({ + createOpenAIClient: ((options: { baseURL?: string }) => { + capturedBaseUrl = options.baseURL; + return (modelId: string) => { + capturedModelId = modelId; + return { modelId }; + }; + }) as never, + streamTextImpl: ((options: { messages?: unknown }) => { + capturedMessages = options.messages; + return { + textStream: (async function* () { + yield 'hello'; + yield ' world'; + })(), + }; + }) as never, + }); + + try { + const request = new NextRequest( + 'http://localhost:3000/api/chat/completions', + { + body: JSON.stringify({ + model: 'caller-model-ignored-for-routing', + messages: [{ role: 'user', content: 'Hi' }], + }), + method: 'POST', + }, + ); + const response = await handler(request); + const text = await response.text(); + + assert( + response.status === 200, + 'POST /api/chat/completions should return 200 for a valid request', + ); + assert( + response.headers.get('content-type') === 'text/event-stream', + 'POST /api/chat/completions should return SSE content type', + ); + assert( + capturedBaseUrl === 'https://example.test/v1', + 'POST /api/chat/completions should pass base URL without /chat/completions', + ); + assert( + capturedModelId === 'gpt-4o', + 'POST /api/chat/completions should route to the pinned server model', + ); + assert( + JSON.stringify(capturedMessages) === + JSON.stringify([{ role: 'user', content: 'Hi' }]), + 'POST /api/chat/completions should pass request messages to streamText', + ); + assert( + text.includes('data: [DONE]'), + 'POST /api/chat/completions should terminate with [DONE]', + ); + assert( + text.includes('"content":"hello"') && text.includes('"content":" world"'), + 'POST /api/chat/completions should stream text chunks as OpenAI-compatible deltas', + ); + } finally { + if (originalApiKey === undefined) { + delete process.env.NEXT_LLM_API_KEY; + } else { + process.env.NEXT_LLM_API_KEY = originalApiKey; + } + if (originalUrl === undefined) { + delete process.env.NEXT_LLM_URL; + } else { + process.env.NEXT_LLM_URL = originalUrl; + } + } +} + async function verifyInviteAgentValidation() { const { POST: inviteAgent } = await import('../app/api/invite-agent/route'); const request = new NextRequest('http://localhost:3000/api/invite-agent', { @@ -224,6 +448,10 @@ async function verifyStopConversationSuccess() { async function main() { await verifyGenerateAgoraTokenRoute(); + await verifyGenerateAgoraTokenReplacesZeroUid(); + await verifyChatCompletionsMissingEnv(); + await verifyChatCompletionsInvalidJson(); + await verifyChatCompletionsSseDone(); await verifyInviteAgentValidation(); await verifyInviteAgentSuccess(); await verifyStopConversationValidation();