Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/concepts/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Each method returns a new `Agent` instance with the updated configuration. Vendo
| `withLlm` | `withLlm(vendor: LlmVendor): Agent<TTSSampleRate, TArea>` | Set the LLM vendor (global or CN) |
| `withTts` | `withTts<SR>(vendor: TtsVendor<SR>): Agent<SR, TArea>` | Set the TTS vendor (tracks sample rate type) |
| `withStt` | `withStt(vendor: SttVendor): Agent<TTSSampleRate, TArea>` | Set the STT vendor (global or CN) |
| `withMllm` | `withMllm(vendor: BaseMLLM): Agent<TTSSampleRate, TArea>` | Set the MLLM vendor (for multimodal flow). Not compatible with `withAvatar()`. |
| `withMllm` | `withMllm(vendor: GlobalMllmVendor \| CNMllmVendor): Agent<TTSSampleRate, TArea>` | Set the global or Chinese mainland MLLM vendor. Not compatible with `withAvatar()`. |
| `withAvatar` | `withAvatar<SR>(this: Agent<SR, TArea>, vendor: AvatarVendor<SR>): Agent<SR, TArea>` | Set the avatar vendor (enforces TTS sample rate match). Requires the cascading pipeline; not supported with `withMllm()`. |
| `withTurnDetection` | `withTurnDetection(config: TurnDetectionConfig): Agent<TTSSampleRate, TArea>` | Configure `turn_detection.language` and cascading-flow SOS/EOS detection |
| `withInterruption` | `withInterruption(config: InterruptionConfig): Agent<TTSSampleRate, TArea>` | Configure unified interruption behavior |
Expand Down
12 changes: 8 additions & 4 deletions docs/concepts/vendors.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ const llm = new OpenAI({
| `DeepgramTTS` | Deepgram | Configurable |
| `GradiumTTS` | Gradium | Configurable |
| `MistralTTS` | Mistral | Not configurable |
| `TypecastTTS` | Typecast | Not configurable |
| `GenericTTS` | Generic OpenAI-compatible HTTP TTS | Configurable |
| `SarvamTTS` | Sarvam AI | Configurable |
| `XAiTTS` | xAI | Configurable |
Expand Down Expand Up @@ -87,7 +88,7 @@ The `sampleRate` is critical when using avatars. See [Avatar Integration](../gui
| `GoogleSTT` | Google Speech | `projectId`, `location`, `adcCredentialsString`, `language` |
| `AmazonSTT` | Amazon Transcribe | `accessKey`, `secretKey`, `region`, `language` |
| `AssemblyAISTT` | AssemblyAI | `apiKey`, `language`, `uri?` |
| `AresSTT` | Agora ARES | |
| `AresSTT` | Agora ARES | `keywords?`, `additionalParams?` |
| `SarvamSTT` | Sarvam AI | `apiKey`, `language` |
| `XAiSTT` | xAI | `apiKey`, `language?`, `baseUrl?`, `sampleRate?` |

Expand All @@ -110,9 +111,11 @@ MLLM (Multimodal LLM) vendors handle audio end-to-end — no separate STT or TTS
| Class | Provider | Key constructor params |
| ---------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OpenAIRealtime` | OpenAI Realtime API | `apiKey`, `model?`, `url?`, `greetingMessage?`, `failureMessage?`, `inputModalities?`, `outputModalities?`, `messages?`, `turnDetection?` |
| `AzureOpenAIRealtime` | Azure OpenAI Realtime (global) | `apiKey`, `url`, `turnDetection`, `model?`, `voice?`, `instructions?`, `params?`, `messages?`, `outputModalities?`, `maxHistory?`, `greetingMessage?` |
| `GeminiLive` | Google Gemini Live API | `apiKey`, `model`, `url?`, `voice?`, `greetingMessage?`, `failureMessage?`, `inputModalities?`, `outputModalities?`, `messages?`, `turnDetection?` |
| `VertexAI` | Vertex AI Gemini Live | `model`, `url?`, `projectId`, `location`, `adcCredentialsString`, `voice?`, `greetingMessage?`, `failureMessage?`, `inputModalities?`, `outputModalities?`, `messages?`, `turnDetection?` |
| `XaiGrok` | xAI Grok (`mllm.vendor`: `xai`) | `apiKey`, `url?`, `voice?`, `language?`, `sampleRate?`, `greetingMessage?`, `failureMessage?`, `inputModalities?`, `outputModalities?`, `messages?`, `turnDetection?` |
| `QwenOmni` | Alibaba Cloud Qwen Omni (CN) | `apiKey`, `model`, `turnDetection`, `url?`, `voice?`, `greetingMessage?`, `failureMessage?` |

`XAiSTT` and `XAiTTS` are the cascading-pipeline xAI wrappers. `XaiGrok` is the realtime MLLM wrapper.

Expand All @@ -132,14 +135,15 @@ See [MLLM Flow Guide](../guides/mllm-flow.md) for full examples.

## CN vendors (Chinese mainland)

CN vendor classes use the same builder methods (`.withLlm()`, `.withStt()`, `.withTts()`, `.withAvatar()`). `client.area` does not restrict which explicit vendor classes you can pass, and it also affects the default ASR vendor when `.withStt()` is omitted:
CN vendor classes use the same builder methods (`.withLlm()`, `.withStt()`, `.withTts()`, `.withMllm()`, `.withAvatar()`). `client.area` does not restrict which explicit vendor classes you can pass, and it also affects the default ASR vendor when `.withStt()` is omitted:

- `Area.CN` defaults to `FengmingSTT`
- all other areas default to `AresSTT`

| Category | Examples |
|---|---|
| LLM | `AliyunLLM`, `BytedanceLLM`, `DeepSeekLLM`, `TencentLLM`, `CustomLLM` |
| MLLM | `QwenOmni` |
| STT | `FengmingSTT`, `TencentSTT`, `MicrosoftCNSTT`, `XfyunSTT`, `XfyunBigModelSTT`, `XfyunDialectSTT` |
| TTS | `GenericTTS` (shared with Global), `MiniMaxCNTTS`, `MicrosoftCNTTS`, `TencentTTS`, `CosyVoiceTTS`, `BytedanceDuplexTTS`, `StepFunTTS` |
| Avatar | `SensetimeAvatar`, `SpatiusAvatar` |
Expand All @@ -150,7 +154,7 @@ See [Regional Routing](../guides/regional-routing.md) and [Vendor Reference](../

Avatars provide a visual representation for the agent. Several avatar vendors require a specific TTS sample rate — this is enforced at both compile time and runtime.

> Avatars currently require the cascading ASR + LLM + TTS pipeline. They are not supported with MLLM (`OpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok`); combining the two throws at `Agent.toProperties()` and `AgentSession.start()`.
> Avatars currently require the cascading ASR + LLM + TTS pipeline. They are not supported with MLLM (`OpenAIRealtime`, `AzureOpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok`, `QwenOmni`); combining the two throws at `Agent.toProperties()` and `AgentSession.start()`.

| Class | Provider | Required TTS sample rate |
| ------------------ | ------------------------------------------- | ------------------------ |
Expand All @@ -162,7 +166,7 @@ Avatars provide a visual representation for the agent. Several avatar vendors re
| `SensetimeAvatar` | SenseTime (CN) | Provider-defined |
| `SpatiusAvatar` | Spatius (CN) | Provider-defined |

CN LLM, STT, TTS, and avatar vendors (`AliyunLLM`, `FengmingSTT`, `MiniMaxCNTTS`, `SensetimeAvatar`, `SpatiusAvatar`, …) are listed in [Vendor Reference](../reference/vendors.md) and [Regional Routing](../guides/regional-routing.md).
CN MLLM, LLM, STT, TTS, and avatar vendors (`QwenOmni`, `AliyunLLM`, `FengmingSTT`, `MiniMaxCNTTS`, `SensetimeAvatar`, `SpatiusAvatar`, …) are listed in [Vendor Reference](../reference/vendors.md) and [Regional Routing](../guides/regional-routing.md).

See [Avatar Integration](../guides/avatars.md) for full examples and the sample-rate constraint details.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/agent-builder-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,5 +349,5 @@ const agentId = await session.start();

- [Agent Reference](../reference/agent.md) — full API signatures
- [Cascading Flow](./cascading-flow.md) — ASR → LLM → TTS setup
- [MLLM Flow](./mllm-flow.md) — end-to-end audio with OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok
- [MLLM Flow](./mllm-flow.md) — end-to-end audio with global providers or Chinese mainland Qwen Omni
- [Regional Routing](./regional-routing.md) — client area and geofence
2 changes: 1 addition & 1 deletion docs/guides/avatars.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Avatars attach a visual representation to the agent's audio output.

## Avatars require the cascading pipeline

Avatars currently only work with the cascading **ASR + LLM + TTS** pipeline. They are **not** supported with MLLM (`OpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok`). AgentKit rejects the combination at `Agent.toProperties()` and at `AgentSession.start()` so you see a clear error before the request reaches the backend:
Avatars currently only work with the cascading **ASR + LLM + TTS** pipeline. They are **not** supported with MLLM (`OpenAIRealtime`, `AzureOpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok`, `QwenOmni`). AgentKit rejects the combination at `Agent.toProperties()` and at `AgentSession.start()` so you see a clear error before the request reaches the backend:

```
Avatars are only supported with the cascading ASR + LLM + TTS pipeline.
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/cascading-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,5 +197,5 @@ await session.start();

## Next steps

- [MLLM Flow](./mllm-flow.md) — skip the ASR → LLM → TTS pipeline with OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok
- [MLLM Flow](./mllm-flow.md) — skip the ASR → LLM → TTS pipeline with a global MLLM provider or Chinese mainland Qwen Omni
- [Avatar Integration](./avatars.md) — add a visual avatar to the agent
53 changes: 50 additions & 3 deletions docs/guides/mllm-flow.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,25 @@
---
sidebar_position: 2
title: MLLM Flow (Multimodal)
description: Use OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok for end-to-end audio processing.
description: Use global or Chinese mainland MLLM providers for end-to-end audio processing.
---

# MLLM Flow (Multimodal)

In MLLM mode, a single multimodal model handles audio input and output end-to-end — no separate STT or TTS step. This reduces latency and is required for OpenAI Realtime, Gemini Live, Vertex AI, and xAI Grok.
In MLLM mode, a single multimodal model handles audio input and output end-to-end — no separate STT or TTS step. AgentKit supports global OpenAI Realtime, Azure OpenAI Realtime, Gemini Live, Vertex AI, and xAI Grok providers, plus Qwen Omni for Chinese mainland deployments.

## When to use MLLM

- You want the lowest-latency conversational experience
- You are using OpenAI Realtime API, Google Gemini Live, Vertex AI Gemini Live, or xAI Grok Realtime
- You are using OpenAI Realtime API, Azure OpenAI Realtime, Google Gemini Live, Vertex AI Gemini Live, xAI Grok Realtime, or Qwen Omni
- You don't need fine-grained control over STT/TTS vendor selection

## Requirements

Call `agent.withMllm(vendor)` — that's it. MLLM mode is enabled automatically through `mllm.enable`. The `withLlm()`, `withTts()`, and `withStt()` methods are not needed — the MLLM vendor handles everything.

`AzureOpenAIRealtime` is a global MLLM (`GlobalMllmVendor`). `QwenOmni` is a Chinese mainland MLLM (`CNMllmVendor`). As with the other explicit vendor helpers, `client.area` controls Agora REST routing but does not prevent you from selecting a provider explicitly.

## Limitations

Avatars are not supported with MLLM at this time. The avatar publisher requires the cascading ASR + LLM + TTS pipeline, so combining `withMllm()` with `withAvatar()` throws at `Agent.toProperties()` and `AgentSession.start()`:
Expand Down Expand Up @@ -63,6 +65,31 @@ console.log('Realtime agent running:', agentId);
await session.stop();
```

## Example: Azure OpenAI Realtime (global)

```typescript
import { AgoraClient, Area, Agent, AzureOpenAIRealtime } from 'agora-agents';

const client = new AgoraClient({
area: Area.US,
appId: 'your-app-id',
appCertificate: 'your-app-certificate',
});

const agent = new Agent({ client }).withMllm(new AzureOpenAIRealtime({
apiKey: 'your-azure-openai-key',
url: 'wss://example.openai.azure.com/openai/realtime',
params: {
instructions: 'You are a conversational AI agent developed by Agora.',
model: 'gpt-realtime-2',
voice: 'alloy',
},
outputModalities: ['audio'],
maxHistory: 32,
turnDetection: { mode: 'server_vad' },
}));
```

## Example: Gemini Live

```typescript
Expand Down Expand Up @@ -126,6 +153,26 @@ const agentId = await session.start();
console.log('Grok agent running:', agentId);
```

## Example: Qwen Omni (Chinese mainland)

```typescript
import { AgoraClient, Area, Agent, QwenOmni } from 'agora-agents';

const client = new AgoraClient({
area: Area.CN,
appId: 'your-app-id',
appCertificate: 'your-app-certificate',
});

const agent = new Agent({ client }).withMllm(new QwenOmni({
apiKey: 'your-dashscope-key',
model: 'qwen-omni-turbo-realtime',
voice: 'Cherry',
greetingMessage: 'Hello, Qwen Omni is ready.',
turnDetection: { mode: 'server_vad' },
}));
```

## Turn detection in MLLM mode

Configure MLLM turn detection on the MLLM vendor with `turnDetection`. When set, `mllm.turn_detection` overrides the top-level `turn_detection` object.
Expand Down
5 changes: 3 additions & 2 deletions docs/guides/regional-routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,18 @@ const client = new AgoraClient({

## Vendor classes

Import any supported vendor class and pass it to `.withStt()`, `.withLlm()`, and `.withTts()`. `client.area` does not restrict explicit provider choice, and it also determines the default ASR vendor when `.withStt()` is omitted:
Import any supported vendor class and pass it to `.withStt()`, `.withLlm()`, `.withTts()`, or `.withMllm()`. `client.area` does not restrict explicit provider choice, and it also determines the default ASR vendor when `.withStt()` is omitted:

- `Area.CN` defaults to `FengmingSTT`
- all other areas default to `AresSTT`

| Typical global providers | Typical CN providers |
|---|---|
| `DeepgramSTT`, `OpenAI`, `MiniMaxTTS`, `ElevenLabsTTS`, `GenericTTS`, … | `FengmingSTT`, `AliyunLLM`, `MiniMaxCNTTS`, `TencentTTS`, `GenericTTS`, … |
| `DeepgramSTT`, `OpenAI`, `MiniMaxTTS`, `AzureOpenAIRealtime`, `GenericTTS`, … | `FengmingSTT`, `AliyunLLM`, `MiniMaxCNTTS`, `QwenOmni`, `GenericTTS`, … |

See the tables below for the full catalog. You may combine any explicit vendor with any `client.area`.
`GenericTTS` is included in both the exported `GlobalTtsVendor` and `CNTtsVendor` types.
MLLM helpers retain their region in the exported `GlobalMllmVendor` and `CNMllmVendor` types: Azure is global and Qwen Omni is CN.

Global client example:

Expand Down
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ The `agora-agents` package lets you build real-time voice AI agents on the [Agor

**Cascading flow** uses ASR -> LLM -> TTS and supports the broadest set of vendor combinations.

**MLLM flow** uses a multimodal model such as OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok for end-to-end audio.
**MLLM flow** uses a multimodal model such as OpenAI Realtime, Azure OpenAI Realtime, Gemini Live, Vertex AI, xAI Grok, or Qwen Omni for end-to-end audio.

## Start here

- Start with [Quick Start](./getting-started/quick-start.md). It shows the baseline app-credentials setup and starts a cascading ASR -> LLM -> TTS agent.
- Use [MLLM Flow Guide](./guides/mllm-flow.md) when your agent uses one realtime multimodal model, such as OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok.
- Use [MLLM Flow Guide](./guides/mllm-flow.md) when your agent uses one realtime multimodal model, including global providers and Chinese mainland Qwen Omni.
- Use [Cascading Flow Guide](./guides/cascading-flow.md) for more examples of the default ASR -> LLM -> TTS flow, including provider-specific configuration.

## How the SDK is organized
Expand Down Expand Up @@ -46,7 +46,7 @@ npm install agora-agents
| [AgentSession](./concepts/session.md) | State machine, lifecycle methods, events |
| [Vendors](./concepts/vendors.md) | LLM, TTS, STT, MLLM, and Avatar provider catalog |
| [Cascading Flow Guide](./guides/cascading-flow.md) | Step-by-step ASR -> LLM -> TTS |
| [MLLM Flow Guide](./guides/mllm-flow.md) | OpenAI Realtime, Gemini Live, Vertex AI, and xAI Grok |
| [MLLM Flow Guide](./guides/mllm-flow.md) | Global MLLM providers and Chinese mainland Qwen Omni |
| [Avatar Integration](./guides/avatars.md) | LiveAvatar, Generic Avatar, Anam, Akool, SenseTime (CN), and HeyGen integration |
| [Agent Builder Features](./guides/agent-builder-features.md) | Turn detection, SAL, filler words, and advanced agent options |
| [Regional Routing](./guides/regional-routing.md) | Area enum, domain pool, failover |
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ Set the STT vendor. Accepts global classes (`DeepgramSTT`, `SpeechmaticsSTT`,

If you omit `withStt()`, AgentKit still sends an ASR config automatically based on `client.area`: `fengming` for `Area.CN`, otherwise `ares`.

### `withMllm(vendor: BaseMLLM): Agent<TTSSampleRate, TArea>`
### `withMllm(vendor: GlobalMllmVendor | CNMllmVendor): Agent<TTSSampleRate, TArea>`

Set the MLLM vendor for multimodal mode. Pass `OpenAIRealtime`, `GeminiLive`, `VertexAI`, or `XaiGrok`. Calling `withMllm()` automatically sets `mllm.enable = true`. MLLM mode does not require `withTts()` / `withLlm()` / `withStt()`.
Set the MLLM vendor for multimodal mode. Pass a global vendor (`OpenAIRealtime`, `AzureOpenAIRealtime`, `GeminiLive`, `VertexAI`, `XaiGrok`) or the Chinese mainland vendor `QwenOmni`. Calling `withMllm()` automatically sets `mllm.enable = true`. MLLM mode does not require `withTts()` / `withLlm()` / `withStt()`.

> Avatars are only supported with the cascading ASR + LLM + TTS pipeline. Combining `withMllm()` with `withAvatar()` throws at `toProperties()` and `session.start()`.

Expand Down
Loading
Loading