Skip to content

Agents #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
digitallysavvy merged 9 commits into from
May 26, 2026
Merged

Agents #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7d48fb1
docs: add progressive disclosure documentation
digitallysavvy May 15, 2026
2d207bd
docs rename
digitallysavvy May 15, 2026
052b43d
docs rename
digitallysavvy May 15, 2026
d760234
docs: fix ai doc path case, API signatures, and AgoraVoiceAI lifecycle
digitallysavvy May 15, 2026
b132f42
updated nextjs
digitallysavvy May 15, 2026
edf034b
nextjs token and chat completion contracts
digitallysavvy May 15, 2026
ecbf09c
docs: fix AGENTS.md module links and clarify area/chat-route in L1 docs
digitallysavvy May 16, 2026
6637e21
updated docs
digitallysavvy May 26, 2026
7c2eb50
updated Claude.md
digitallysavvy May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
202 changes: 180 additions & 22 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,138 @@
# Agent Guide
# Agent Development Guide

Use this file as the primary agent-facing guide for `agora-convoai-quickstart-nextjs`.
This guide is for coding agents making changes in `agent-quickstart-nextjs`.

## How to Load

This repository uses progressive disclosure documentation. Docs live under `docs/ai/` in three levels.

1. Read [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md) to identify the repo.
2. Load ALL 8 files in [docs/ai/L1/](docs/ai/L1/). They are small — load all upfront.
3. Follow L2 deep-dive links only when L1 isn't detailed enough. The index is at [docs/ai/L1/L2/_index.md](docs/ai/L1/L2/_index.md).

The sections below (Start Here, Patterns, Anti-Patterns, etc.) remain the canonical contributor handbook for hands-on work; the `docs/ai/` tree is the structured summary used by AI agents.

## Start Here

- Read [README.md](./README.md) for setup, commands, verification, and deployment.
- Use [DOCS/GUIDE.md](./DOCS/GUIDE.md) for the long-form build walkthrough.
- Use [DOCS/TEXT_STREAMING_GUIDE.md](./DOCS/TEXT_STREAMING_GUIDE.md) for transcript and RTM behavior.
- Use [docs/GUIDE.md](./docs/GUIDE.md) for the long-form build walkthrough.
- Use [docs/TEXT_STREAMING_GUIDE.md](./docs/TEXT_STREAMING_GUIDE.md) for transcript and RTM behavior.
- For layout and responsibilities inside `components/`, `app/api/`, and `lib/`, use [docs/ai/L1/03_code_map.md](docs/ai/L1/03_code_map.md) and [docs/ai/L1/02_architecture.md](docs/ai/L1/02_architecture.md).

## Current System Shape

- Next.js 16 App Router with React 19 and TypeScript
- Browser RTC via `agora-rtc-react`
- RTM transcripts via `agora-rtm`
- Transcript/runtime helpers via `agora-agent-client-toolkit`
- Shared UI primitives via `agora-agent-uikit`
- Token and agent lifecycle routes inside `app/api`
- App shell: Next.js 16 App Router, React 19, and TypeScript
- Client RTC: `agora-rtc-react` hooks over `agora-rtc-sdk-ng`
- Messaging: `agora-rtm` for transcripts, agent state, metrics, and error events
- Toolkit core: `agora-agent-client-toolkit` for `AgoraVoiceAI`, transcript helpers, and turn status
- UI components: `agora-agent-uikit` for visualizer, transcript, and mic controls
- Server SDK: `agora-agent-server-sdk` for managed agent session startup
- API routes: token generation, agent invite, chat, and stop routes live in `app/api`
- Default agent config: Agora-managed STT, LLM, and TTS; no third-party vendor keys are required for the base quickstart

## Supported Modes

### Local Development

- Run from the repo root with `pnpm run dev`.
- Next.js serves the app and the route handlers at `http://localhost:3000`.
- Local credentials are read from `.env.local`, usually written by `agora project env write .env.local`.

### Vercel Deployment

- Deploy the repository as a single Next.js app.
- Set `NEXT_PUBLIC_AGORA_APP_ID` and `NEXT_AGORA_APP_CERTIFICATE` in the deployment target.
- Keep `NEXT_AGORA_APP_CERTIFICATE` server-side only.

## Routing / Ownership

- UI and RTC/RTM client lifecycle live in `components`.
- Browser-facing API routes live in `app/api`.
- Shared constants and transcript normalization live in `lib`.
- If a workflow, request contract, or ownership boundary changes, update `README.md`, `docs/GUIDE.md`, `docs/TEXT_STREAMING_GUIDE.md`, and this file in the same change.

## Key Files

- `app/api/generate-agora-token/route.ts`: RTC + RTM token generation
- `app/api/invite-agent/route.ts`: managed agent session startup
- `app/api/stop-conversation/route.ts`: agent shutdown
- `components/LandingPage.tsx`: session bootstrap, RTM setup, provider wiring
- `components/ConversationComponent.tsx`: RTC join, transcript flow, visualizer, renewals
- `lib/agora.ts`: shared agent UID defaults
- `env.local.example`: local environment template
- `app/api/generate-agora-token/route.ts`: issues RTC + RTM tokens for the browser user.
- `app/api/invite-agent/route.ts`: starts the managed agent session; edit here for system prompt, VAD, model, or voice changes.
- `app/api/stop-conversation/route.ts`: stops the agent session.
- `app/api/chat/completions/route.ts`: optional OpenAI-compatible SSE proxy for a custom LLM (not wired by default).
- `components/LandingPage.tsx`: session bootstrap, RTM setup, provider wiring, and conversation lifecycle.
- `components/ConversationComponent.tsx`: RTC join, mic publication, `AgoraVoiceAI` init, transcript state, and renewals.
- `components/QuickstartConversationLayout.tsx`: in-call header, transcript rail, and controls dock.
- `components/QuickstartPipelineMetrics.tsx`: per-stage latency chips from `AGENT_METRICS`.
- `components/QuickstartTranscriptPanel.tsx`: live transcript rail.
- `lib/agora.ts`: shared agent UID defaults.
- `lib/conversation.ts`: transcript normalization and visualizer state mapping.
- `env.local.example`: local environment template.
- `scripts/verify-api-contracts.ts`: route contract verification.

## Patterns

### StrictMode Guard (`isReady`)

Both `useJoin` and `useLocalMicrophoneTrack` are gated by `isReady` to prevent double initialization in React StrictMode dev mode. The cleanup fires synchronously before any `setTimeout`, so only the real second mount's timer fires.

```tsx
const [isReady, setIsReady] = useState(false);
useEffect(() => {
let cancelled = false;
const id = setTimeout(() => {
if (!cancelled) setIsReady(true);
}, 0);
return () => {
cancelled = true;
clearTimeout(id);
setIsReady(false);
};
}, []);
const { isConnected: joinSuccess } = useJoin(config, isReady);
const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady);
```

### Hook Ownership

- `useJoin` owns `client.leave()`; never call it manually.
- `useLocalMicrophoneTrack` owns track lifecycle; do not manually call `.close()`.
- `usePublish` owns publish state; mute with `track.setEnabled()` and do not manually unpublish.

### AgoraVoiceAI Init

Initialize `AgoraVoiceAI` from `agora-agent-client-toolkit` inside `ConversationComponent`, gated on `isReady && joinSuccess`.

```tsx
useEffect(() => {
if (!isReady || !joinSuccess) return;
// AgoraVoiceAI.init() is called here exactly once.
}, [isReady, joinSuccess]);
```

`isReady` becomes true only after the StrictMode fake-unmount cycle completes. Once `isReady` is true, React does not double invoke the effect for later dependency changes such as `joinSuccess` becoming true.

### Transcript and UI Mapping

- Manage `transcript` and `agentState` through `useState` plus `ai.on(TRANSCRIPT_UPDATED, ...)` and `ai.on(AGENT_STATE_CHANGED, ...)`.
- The toolkit uses `uid="0"` as a sentinel for the local user's speech. Remap that value to `client.uid` in the `TRANSCRIPT_UPDATED` callback or `ConvoTextStream` will render user speech on the wrong side.
- Include `INTERRUPTED` turns in `messageList`; filter only `IN_PROGRESS`. If the agent's first turn is interrupted and omitted, `messageList` stays empty and `ConvoTextStream` never auto-opens.

### Tokens and Styling

- RTM token access must come from `RtcTokenBuilder.buildTokenWithRtm`; a standard RTC-only token does not grant RTM access.
- Tailwind must scan uikit classes with `./node_modules/agora-agent-uikit/dist/**/*.{js,mjs}` in `tailwind.config.ts`.

## Working Rules

- Keep the RTC client creation StrictMode-safe with `useRef`, not `useMemo`.
- Keep the token route on `RtcTokenBuilder.buildTokenWithRtm`.
- Prefer the smallest change that keeps the quickstart copyable and production-style.
- Keep RTC client creation StrictMode-safe with `useRef`, not `useMemo`.
- Keep token generation on `RtcTokenBuilder.buildTokenWithRtm`.
- Keep transcript UID remapping aligned with the toolkit sentinel behavior.
- Keep README, `DOCS/GUIDE.md`, and `DOCS/TEXT_STREAMING_GUIDE.md` aligned with implementation changes.
- Do not require third-party vendor API keys unless the code actually introduces a BYOK provider path.
- Keep README, `docs/GUIDE.md`, and `docs/TEXT_STREAMING_GUIDE.md` aligned with implementation changes.

## Commands

From the repo root:

```bash
pnpm install
pnpm run doctor
Expand All @@ -52,8 +149,69 @@ pnpm run verify:api
pnpm run build
```

## Verification Safety

- Safe without live Agora credentials:
- `pnpm run lint`
- `pnpm run typecheck`
- `pnpm run verify:api`
- `pnpm run build`
- Requires local env setup but not a live Agora session:
- `pnpm run doctor`
- `pnpm run verify`
- Often blocked inside restricted sandboxes because of port binding or process spawning:
- `pnpm run dev`

## Anti-Patterns / What NOT To Do

- Do not call `client.leave()` manually; it breaks `useJoin` cleanup.
- Do not call `localMicrophoneTrack.close()` manually; it breaks hook ownership.
- Do not remove the `isReady` guard.
- Do not set `reactStrictMode: false` as a workaround.
- Do not use the deprecated `turnDetection.type: 'agora_vad'` flat API; use `turnDetection.config.start_of_speech` and `turnDetection.config.end_of_speech`.
- Do not replace `RtcTokenBuilder.buildTokenWithRtm` with an RTC-only token builder.
- Do not hide SDK requirements only in `CLAUDE.md`; all agent-facing guidance belongs in `AGENTS.md`.

## Done Criteria

1. Run the narrowest relevant validation command.
Before finishing a change:

1. Run the narrowest relevant verification command.
2. For shipped app/runtime changes, ensure `pnpm run verify` passes.
3. Update the root README and any affected docs when workflow or architecture guidance changes.
3. If you changed files in `components/` or `app/api/`, verify that `docs/GUIDE.md`, `docs/TEXT_STREAMING_GUIDE.md`, `README.md`, and this file still match the implementation.
4. Update root README and affected docs when workflow, request contracts, architecture, or environment guidance changes.
5. If the change touches workflows, interfaces, gotchas, or security details, update the matching file under [docs/ai/L1/](docs/ai/L1/) and bump `Last Reviewed` in [docs/ai/L0_repo_card.md](docs/ai/L0_repo_card.md).

## Git Conventions

### Commit messages — conventional commits

- **Format:** `type: description` or `type(scope): description`
- **Types:** `feat:` (new feature), `fix:` (bug fix), `chore:` (maintenance, version bumps), `test:` (test additions/changes), `docs:` (documentation)
- **Scoped variant:** `feat(scope):`, `fix(scope):` — e.g. `feat(api): add stop-conversation status flag`
- **Lowercase after prefix** — `feat: add feature`, not `feat: Add feature`
- **Present tense** — "add feature", not "added feature"
- **PR number appended** — `feat: add feature (#123)`

### Branch names

- **Format:** `type/short-description` — lowercase, hyphen-separated
- **Types match commit types:** `feat/`, `fix/`, `chore/`, `test/`, `docs/`
- **Examples:** `feat/agent-metrics`, `fix/transcript-uid`, `docs/progressive-disclosure`

### General rules

- **No AI tool names** — never mention claude, cursor, copilot, cody, aider, gemini, codex, chatgpt, or gpt-3/4 in commit messages or PR descriptions.
- **No Co-Authored-By trailers** — omit AI attribution lines.
- **No `--no-verify`** — let git hooks run normally.
- **No git config changes** — do not modify `user.name` or `user.email`.

## Doc Commands

| Command | When to use |
| --------------- | ------------------------------------------------------------ |
| generate docs | No `docs/ai/` directory exists yet |
| update docs | Code changed since the `Last Reviewed` date in L0 |
| test docs | Verify docs give agents the right context (writes `docs/ai/test-results.md`) |

The generator and tester live in the [AgoraIO-Community/ai-devkit](https://github.com/AgoraIO-Community/ai-devkit) skill set. See the [progressive disclosure standard](https://github.com/AgoraIO-Community/ai-devkit/blob/main/docs/progressive-disclosure-standard.md) for the full specification.
105 changes: 2 additions & 103 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,104 +1,3 @@
# CLAUDE.md
This project uses AGENTS.md instead of a CLAUDE.md file.

> Read `AGENTS.md` for the current project map, commands, and validation rules before making changes.

## Project

Next.js 16 (App Router) quickstart demonstrating Agora Conversational AI Engine. Developers copy this code — production quality and idiomatic patterns are required in every change.

## Commands

```bash
pnpm run doctor # local prerequisites and env checks
pnpm dev # start dev server (http://localhost:3000)
pnpm run verify # doctor + lint + typecheck + API contracts + production build
```

## Key Patterns

### StrictMode Guard (`isReady`)

Both `useJoin` and `useLocalMicrophoneTrack` are gated by `isReady` to prevent double-initialization in React StrictMode dev mode. The cleanup fires synchronously before any `setTimeout`, so only the real second mount's timer fires.

```tsx
const [isReady, setIsReady] = useState(false);
useEffect(() => {
let cancelled = false;
const id = setTimeout(() => { if (!cancelled) setIsReady(true); }, 0);
return () => { cancelled = true; clearTimeout(id); setIsReady(false); };
}, []);
const { isConnected: joinSuccess } = useJoin(config, isReady);
const { localMicrophoneTrack } = useLocalMicrophoneTrack(isReady);
```

Do not remove this pattern. Do not set `reactStrictMode: false` as a workaround.

### Hook Ownership

- `useJoin` owns `client.leave()` — **never call it manually**
- `useLocalMicrophoneTrack` owns track lifecycle — **no manual `.close()`**
- `usePublish` owns publish state — mute via `track.setEnabled()` only, never unpublish manually

### AgoraVoiceAI Init — Raw Toolkit with `isReady && joinSuccess` Gate

`AgoraVoiceAI` (from `agora-agent-client-toolkit`) is initialized in a `useEffect` inside `ConversationComponent`, gated on `isReady && joinSuccess`:

```tsx
useEffect(() => {
if (!isReady || !joinSuccess) return;
// AgoraVoiceAI.init() called here — exactly once, no StrictMode interference
}, [isReady, joinSuccess]);
```

**Why `isReady && joinSuccess` works:**
- `isReady` is `true` only after the StrictMode fake-unmount cycle completes (via `setTimeout(fn, 0)` pattern).
- Once `isReady` is `true`, React does NOT double-invoke the effect for subsequent dependency changes (`joinSuccess` becoming `true`). This means `AgoraVoiceAI.init()` is called exactly once.

`transcript` and `agentState` are managed via `useState` + `ai.on(TRANSCRIPT_UPDATED, ...)` / `ai.on(AGENT_STATE_CHANGED, ...)` directly.

### UID Remapping

The toolkit uses `uid="0"` as a sentinel for the local user's speech. The uikit treats `uid===0` as an AI message. Remap in the `TRANSCRIPT_UPDATED` callback to `client.uid` or ConvoTextStream will show user speech on the wrong side.

### `messageList` Filter

Include `INTERRUPTED` turns in `messageList` (filter only `IN_PROGRESS`). If the agent's first turn is interrupted, omitting it means `messageList` stays empty and `ConvoTextStream` never auto-opens.

## Architecture

| Layer | Package | Role |
|---|---|---|
| Client UI | `agora-rtc-react` | RTC hooks (`useJoin`, `useLocalMicrophoneTrack`, `usePublish`, etc.) |
| Toolkit core | `agora-agent-client-toolkit` | `AgoraVoiceAI`, `TurnStatus` enum, `TranscriptHelperItem` types |
| UI components | `agora-agent-uikit` | `AudioVisualizer`, `ConvoTextStream`, `MicButtonWithVisualizer` |
| Server SDK | `agora-agent-server-sdk` | Builder pattern — `AgoraClient` → `Agent` → `session.start()` |
| Messaging | `agora-rtm` | RTM transport for transcripts |

RTM token must be generated with `RtcTokenBuilder.buildTokenWithRtm` — a standard RTC-only token does not grant RTM access.

Tailwind must scan uikit classes: `./node_modules/agora-agent-uikit/dist/**/*.{js,mjs}` in `tailwind.config.ts`.

## Important Files

| File | Purpose |
|---|---|
| `components/ConversationComponent.tsx` | Core real-time UI — all Agora hooks, `AgoraVoiceAI` init, transcript state |
| `components/LandingPage.tsx` | Entry point — session setup, RTM client lifecycle, parallel agent+RTM init |
| `app/api/invite-agent/route.ts` | Starts AI agent — edit for system prompt, VAD, model, voice |
| `app/api/generate-agora-token/route.ts` | Issues RTC+RTM token for the browser user |
| `app/api/stop-conversation/route.ts` | Stops the agent |
| `DOCS/GUIDE.md` | Step-by-step build guide — must stay in sync with implementation |
| `DOCS/TEXT_STREAMING_GUIDE.md` | Text streaming / transcription deep-dive |
| `AGENTS.md` | Primary agent-facing project map and validation guide |

## After Changing Implementation Files

After editing anything in `components/` or `app/api/`, manually verify that `DOCS/GUIDE.md`, `DOCS/TEXT_STREAMING_GUIDE.md`, `README.md`, and `AGENTS.md` still match the implementation, then run `pnpm run verify`.

## What NOT To Do

- Do not call `client.leave()` manually (breaks `useJoin` cleanup)
- Do not call `localMicrophoneTrack.close()` manually (breaks hook ownership)
- Do not remove the `isReady` guard
- Do not set `reactStrictMode: false`
- Do not use the deprecated `turnDetection.type: 'agora_vad'` flat API — use `turnDetection.config.start_of_speech` / `end_of_speech`
Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project.
Loading