Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ below instead of assuming the old list.
| `messages.citations` | Final assistant `transcript.final` events may carry normalized citations collected from frontend retrieval in the same turn | `test/gateway-event-schema.test.mjs`, `server/test/realtime-presentation-runtime.test.mjs` |
| `realtime.conversation-client-v1` | `WS /api/realtime`, published event constants, and message schemas form the replaceable text/audio/multimodal Conversation Client boundary | `test/gateway-event-schema.test.mjs`, `test/custom-conversation-client.test.mjs` |
| `realtime.gateway-client-protocol-v6-handshake` | The same WebSocket accepts an opt-in 6.0 `session.hello`, returns correlated `session.ready`, negotiates implemented capabilities, and normalizes 6.0 input aliases into the existing business path | `test/gateway-client-protocol.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | Negotiated 6.0 Clients can publish registered semantic Client Events and use correlated Task, permission, and conversation-history commands over the same WebSocket; existing REST routes call the same command service as compatibility aliases | `test/gateway-client-protocol.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/client-command-runtime.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | Negotiated 6.0 Clients can publish registered semantic Client Events and use correlated Task, permission, conversation-history, and session output-voice commands over the same WebSocket; existing REST routes call the same command service as compatibility aliases | `test/gateway-client-protocol.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/client-command-runtime.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-agent-delivery` | Client Events, Task results and progress, and permission prompts cross one provider-neutral `AgentDelivery` boundary with `handle`, `context`, `respond`, and `interrupt` modes | `server/test/agent-delivery.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/realtime-provider.test.mjs`, `server/test/announcement-manager.test.mjs` |
| `realtime.gateway-client-protocol-v6-client-actions` | Correlated `client.action.request/result` messages execute Client-owned environment operations; `enter_sleep` is capability-gated and sleeping commits only after Client success | `test/gateway-client-protocol.test.mjs`, `server/test/client-action-port.test.mjs`, `server/test/gateway-client-handshake.test.mjs`, `desktop/test/enter-sleep-flow.test.mjs` |
| `realtime.gateway-client-protocol-v6-reference-client-replay` | The shared reference Client SDK owns handshake, command correlation, Client Actions, reconnect, and recovery; Task pushes use bounded `sequence` replay and WebUI, Desktop, and TUI share one conformance suite | `test/gateway-client-sdk.test.mjs`, `test/gateway-client-conformance.test.mjs`, `server/test/gateway-client-protocol-session.test.mjs`, `server/test/gateway-client-replay-buffer.test.mjs` |
| `realtime.gateway-client-protocol-v6-reference-client-replay` | The shared reference Client SDK owns handshake, command correlation, `updateOutputVoice()`, Client Actions, reconnect, and recovery; Task pushes use bounded `sequence` replay and WebUI, Desktop, and TUI share one conformance suite | `test/gateway-client-sdk.test.mjs`, `test/gateway-client-conformance.test.mjs`, `server/test/gateway-client-protocol-session.test.mjs`, `server/test/gateway-client-replay-buffer.test.mjs` |
| `desktop.orb-shell` | The orb form's main-process contract ships: `bindOrbShell` answers the channels the shipped preload sends | `desktop/test/orb-shell.test.mjs` |
| `desktop.orb-window-factory` | `createOrbWindow` owns the orb window recipe; its `destroy()` is the host's synchronous teardown path (renderer exit is what releases the microphone) | `desktop/test/orb-window.test.mjs` |
| `desktop.orb-placement` | `createOrbPlacement` covers the default anchor, display clamping and drop persistence | `desktop/test/orb-placement.test.mjs` |
Expand All @@ -101,7 +101,7 @@ is unsupported and breaks without notice.
| --- | --- |
| `qwen-audio-agent/electron` | **CJS**: `load()` (every contract in one namespace), `PRELOAD_PATH` |
| `qwen-audio-agent/gateway-protocol` | `GATEWAY_PROTOCOL_VERSION`, `GATEWAY_CAPABILITIES` |
| `qwen-audio-agent/gateway-client-protocol` | GCP 6.0 envelope and handshake schemas, parsers, capability constants, and reference Client helpers |
| `qwen-audio-agent/gateway-client-protocol` | GCP 6.0 envelope, handshake and runtime-command schemas, parsers, capability constants, and reference Client helpers |
| `qwen-audio-agent/gateway-client-sdk` | `GatewayClient`: WebSocket lifecycle, 6.0 handshake, request correlation, Client Actions, bounded replay, and reconnect recovery |
| `qwen-audio-agent/gateway-client-profiles` | Reference capability profiles for WebUI, Desktop, and TUI |
| `qwen-audio-agent/client-events` | Client Event definition registry, built-in definitions, routing policies, and `GatewayEventRouter` for Gateway extensions |
Expand Down
6 changes: 3 additions & 3 deletions docs/contract.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,10 @@ Task 事件提供与 A2A 对齐的 `submitted`、
| `messages.citations` | 最终助手 `transcript.final` 可以携带同一轮前台检索产生的规范化 Citation | `test/gateway-event-schema.test.mjs`、`server/test/realtime-presentation-runtime.test.mjs` |
| `realtime.conversation-client-v1` | `WS /api/realtime`、公开事件常量与消息 Schema 共同构成可替换的文本/音频/多模态对话客户端边界 | `test/gateway-event-schema.test.mjs`、`test/custom-conversation-client.test.mjs` |
| `realtime.gateway-client-protocol-v6-handshake` | 同一 WebSocket 可选择以 6.0 `session.hello` 接入,返回有关联关系的 `session.ready`,协商已实现能力,并把 6.0 输入别名归一化到现有业务路径 | `test/gateway-client-protocol.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | 协商后的 6.0 Client 可以通过同一 WebSocket 发布已注册的语义 Client Event,并使用有关联结果的 Task、权限和对话历史命令;现有 REST 路由调用同一命令服务作为兼容别名 | `test/gateway-client-protocol.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/client-command-runtime.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | 协商后的 6.0 Client 可以通过同一 WebSocket 发布已注册的语义 Client Event,并使用有关联结果的 Task、权限、对话历史和会话输出音色命令;现有 REST 路由调用同一命令服务作为兼容别名 | `test/gateway-client-protocol.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/client-command-runtime.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-agent-delivery` | Client Event、Task 结果与低频进展、权限请求统一跨越 Provider 无关 `AgentDelivery` 边界,并支持 `handle`、`context`、`respond`、`interrupt` 四种模式 | `server/test/agent-delivery.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/realtime-provider.test.mjs`、`server/test/announcement-manager.test.mjs` |
| `realtime.gateway-client-protocol-v6-client-actions` | 有关联关系的 `client.action.request/result` 执行 Client 自有环境操作;`enter_sleep` 按 capability 暴露,只有 Client 成功后才提交 sleeping | `test/gateway-client-protocol.test.mjs`、`server/test/client-action-port.test.mjs`、`server/test/gateway-client-handshake.test.mjs`、`desktop/test/enter-sleep-flow.test.mjs` |
| `realtime.gateway-client-protocol-v6-reference-client-replay` | 共享参考 Client SDK 统一处理握手、命令关联、Client Action、重连与状态恢复;Task 推送以 `sequence` 有限回放,WebUI、Desktop、TUI 共用一致性测试 | `test/gateway-client-sdk.test.mjs`、`test/gateway-client-conformance.test.mjs`、`server/test/gateway-client-protocol-session.test.mjs`、`server/test/gateway-client-replay-buffer.test.mjs` |
| `realtime.gateway-client-protocol-v6-reference-client-replay` | 共享参考 Client SDK 统一处理握手、命令关联、`updateOutputVoice()`、Client Action、重连与状态恢复;Task 推送以 `sequence` 有限回放,WebUI、Desktop、TUI 共用一致性测试 | `test/gateway-client-sdk.test.mjs`、`test/gateway-client-conformance.test.mjs`、`server/test/gateway-client-protocol-session.test.mjs`、`server/test/gateway-client-replay-buffer.test.mjs` |
| `desktop.orb-shell` | 悬浮球形态的主进程契约随包发布:`bindOrbShell` 应答随包 preload 发出的全部通道 | `desktop/test/orb-shell.test.mjs` |
| `desktop.orb-window-factory` | `createOrbWindow` 持有悬浮球窗口配方;其 `destroy()` 是宿主的同步销毁路径(渲染进程退出才能确定性释放麦克风) | `desktop/test/orb-window.test.mjs` |
| `desktop.orb-placement` | `createOrbPlacement` 覆盖默认锚点、显示器夹取与拖放持久化 | `desktop/test/orb-placement.test.mjs` |
Expand All @@ -86,7 +86,7 @@ Task 事件提供与 A2A 对齐的 `submitted`、
| --- | --- |
| `qwen-audio-agent/electron` | **CJS**:`load()`(一个命名空间拿到全部契约)、`PRELOAD_PATH` |
| `qwen-audio-agent/gateway-protocol` | `GATEWAY_PROTOCOL_VERSION`、`GATEWAY_CAPABILITIES` |
| `qwen-audio-agent/gateway-client-protocol` | GCP 6.0 信封与握手 Schema、解析器、能力常量和参考 Client Helper |
| `qwen-audio-agent/gateway-client-protocol` | GCP 6.0 信封、握手与运行时命令 Schema、解析器、能力常量和参考 Client Helper |
| `qwen-audio-agent/gateway-client-sdk` | `GatewayClient`:WebSocket 生命周期、6.0 握手、请求关联、Client Action、有限回放和重连恢复 |
| `qwen-audio-agent/gateway-client-profiles` | WebUI、Desktop、TUI 的参考 capability profile |
| `qwen-audio-agent/client-events` | 供 Gateway 扩展使用的 Client Event Definition Registry、内置定义、路由 Policy 与 `GatewayEventRouter` |
Expand Down
46 changes: 45 additions & 1 deletion docs/gateway-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ The Client connects to `ws://<gateway>/api/realtime`. The first message is `sess
"permissions.respond",
"conversation.history",
"client.events",
"session.output_voice",
"client.actions.desktop.presence.enter_sleep",
"session.replay"
],
Expand All @@ -70,11 +71,19 @@ The Client connects to `ws://<gateway>/api/realtime`. The first message is `sess
"voice_enabled": true,
"input_enabled": true,
"output_enabled": true,
"text_only": false
"text_only": false,
"output_voice": "longanlufeng"
}
}
```

`connection.output_voice` is an optional session-scoped output voice preference.
The Gateway leaves its interpretation to the active Realtime Provider; when it
is absent, the Provider keeps its deployment-level default. Providers that only
accept a voice in their initial session configuration require a fresh Realtime
Session when the voice changes. At runtime the Gateway performs that upstream
rebuild while preserving the Client's GCP connection and Gateway session.

Gateway returns the selected version and capability intersection:

```jsonc
Expand All @@ -93,6 +102,7 @@ Gateway returns the selected version and capability intersection:
"permissions.respond",
"conversation.history",
"client.events",
"session.output_voice",
"client.actions.desktop.presence.enter_sleep",
"session.replay"
]
Expand Down Expand Up @@ -135,6 +145,12 @@ trusted fields are accepted from event data. The first built-in definition is
handles the event without pretending it is user input; GCP3 now projects it
through the shared Agent Delivery boundary.

A deterministic handler may use narrow effects supplied by its Gateway host,
such as selecting one deployment-owned Assistant Profile for the current
Realtime Session. The Client still sends only schema-validated identifiers:
event data never becomes instructions, and arbitrary prompt text is not an
allowed effect input.

### 3.3 GCP3 delivery rollout

GCP3 implements the provider-neutral value and all four routing modes from
Expand Down Expand Up @@ -313,8 +329,36 @@ The active Client uses the same WebSocket for runtime commands and queries. Each
| `permission.respond` | C→G | Resolve the currently pending authorization request |
| `task.input.respond` | C→G | Continue the same Task with requested user input, or decline/cancel that interaction |
| `conversation.history` | C→G | Read the bounded, client-safe conversation projection |
| `session.output_voice.update` | C→G | Change this session's output voice; the result is `session.output_voice.updated` |
| `session.replay` | C→G | Replay eligible server pushes after a sequence cursor |

After negotiating `session.output_voice`, clients may call
`GatewayClient.updateOutputVoice(voice)`. Its wire request and result are:

```jsonc
{
"type": "session.output_voice.update",
"event_id": "evt_client_voice_1",
"voice": "longanlufeng"
}
```

```jsonc
{
"type": "session.output_voice.updated",
"event_id": "evt_gateway_voice_1",
"request_event_id": "evt_client_voice_1",
"voice": "longanlufeng",
"changed": true,
"reconnecting": true
}
```

`changed` reports whether the preference changed; `reconnecting` reports
whether the Gateway is rebuilding the upstream Realtime Session with the new
voice. A Provider without session-voice support returns the correlated
`output_voice_unsupported` error, so the Client never branches on Provider name.

`permission.respond.decision` accepts `once`, `always`, or `reject`: allow only
the current operation, always allow during the current frontend session, or
reject only the current operation.
Expand Down
41 changes: 40 additions & 1 deletion docs/gateway-protocol.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Client 连接 `ws://<gateway>/api/realtime`,第一条消息必须是 `session.
"permissions.respond",
"conversation.history",
"client.events",
"session.output_voice",
"client.actions.desktop.presence.enter_sleep",
"session.replay"
],
Expand All @@ -70,11 +71,17 @@ Client 连接 `ws://<gateway>/api/realtime`,第一条消息必须是 `session.
"voice_enabled": true,
"input_enabled": true,
"output_enabled": true,
"text_only": false
"text_only": false,
"output_voice": "longanlufeng"
}
}
```

`connection.output_voice` 是可选的会话级输出音色偏好。Gateway 将它交给当前
Realtime Provider 解释;不设置时继续使用 Provider 的部署级默认音色。对于仅允许在
首次会话配置音色的 Provider,Gateway 在运行时切换时只重建上游 Provider Session;
客户端 GCP 连接与 Gateway 会话保持不变。

Gateway 返回协商后的版本与能力交集:

```jsonc
Expand All @@ -93,6 +100,7 @@ Gateway 返回协商后的版本与能力交集:
"permissions.respond",
"conversation.history",
"client.events",
"session.output_voice",
"client.actions.desktop.presence.enter_sleep",
"session.replay"
]
Expand Down Expand Up @@ -130,6 +138,10 @@ Event data 提供。首个内置定义是 `desktop.presence.sleep_requested`。G
校验、保存和确定性处理,不会把它伪装成用户输入;GCP3 已经把它投影到统一的 Agent
Delivery 边界。

确定性 Handler 可以使用 Gateway 宿主提供的窄效果,例如从部署方拥有的
Assistant Profile 白名单中为当前 Realtime Session 选择一项。Client 仍然只能发送
Schema 校验过的标识:Event data 不会直接变成指令,效果也不接受任意 Prompt 文本。

### 3.3 GCP3 Delivery 落地

GCP3 实现第 6 节定义的 Provider 无关值与四种路由模式。Task 最终结果、有意义的低频
Expand Down Expand Up @@ -299,8 +311,35 @@ Client Action 不替代 MCP、OpenAPI、ACP 或 A2A。它只用于当前 Client
| `permission.respond` | C→G | 处理当前等待中的授权请求 |
| `task.input.respond` | C→G | 把用户补充输入交回同一 Task,或拒绝/取消这次交互 |
| `conversation.history` | C→G | 读取有界、对 Client 安全的对话投影 |
| `session.output_voice.update` | C→G | 更新当前会话的输出音色;结果为 `session.output_voice.updated` |
| `session.replay` | C→G | 从 sequence 游标回放符合条件的服务端推送 |

协商 `session.output_voice` 能力后,客户端可以直接调用 SDK 的
`GatewayClient.updateOutputVoice(voice)`。对应线协议为:

```jsonc
{
"type": "session.output_voice.update",
"event_id": "evt_client_voice_1",
"voice": "longanlufeng"
}
```

```jsonc
{
"type": "session.output_voice.updated",
"event_id": "evt_gateway_voice_1",
"request_event_id": "evt_client_voice_1",
"voice": "longanlufeng",
"changed": true,
"reconnecting": true
}
```

`changed` 表示偏好是否变化,`reconnecting` 表示 Gateway 是否正在用新音色重建上游
Realtime Session。Provider 不支持会话音色时返回关联错误
`output_voice_unsupported`,Client 不需要识别具体 Provider。

`permission.respond.decision` 支持 `once`、`always` 和 `reject`,分别表示
仅允许当前操作、当前前端会话内始终允许,以及仅拒绝当前操作。

Expand Down
12 changes: 10 additions & 2 deletions docs/scenarios/smart-cockpit.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,20 @@
| Component | Example implementation | Replaceable boundary |
|---|---|---|
| `client/` | React cockpit UI + Browser Audio | GCP 6.0 / Gateway Client SDK |
| `gateway.mjs` | qwen-audio-agent Gateway + foreground Realtime Agent | Framework reuse and scenario composition |
| `agent/` | Small A2A cockpit Agent | BackendPort / A2A / ACP / custom Adapter |
| `gateway/` | qwen-audio-agent Gateway + foreground Realtime Agent | Framework reuse and scenario composition |
| `agent/` | Qwen3.8-Flash A2A cockpit Agent | BackendPort / A2A / ACP / custom Adapter |
| `service/` | Cockpit environment, state, rules, tools, and external integrations | HTTP/SSE / MCP / customer protocols |

An independent cockpit service owns vehicle, route, media, and order state. The UI observes it over HTTP/SSE while the foreground and backend Agents use scoped MCP surfaces. The Gateway neither parses scenario objects nor acts as a business-state event bus.

The example also supports voice-created, cockpit-scoped custom workflows. The backend Agent loads a workflow and composes existing MCP tools; user workflows do not dynamically change the Gateway protocol, MCP tool set, or A2A Agent Card.

The foreground can also switch among the Healer, Action, and Sharp Assistant
Profiles inside the current Realtime Session. A GCP Client Event carries only
an allowlisted ID; the Gateway maps it to deployment-owned Markdown under
`gateway/assistant/` and applies it to the next turn with `session.update`. Arbitrary
Client-supplied prompt text is never accepted.

In a customer deployment, the conversation layer may be the only retained implementation. The cockpit UI and backend Agent can both be customer-owned. A custom UI does not inherit the framework WebUI; it implements GCP plus its own audio, page, and business-state channels.

## Run
Expand Down
Loading