Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ runtime-command plane, Agent Delivery, Client Actions, reference Client SDK,
and bounded replay all share the same WebSocket.
This contract index remains authoritative for implemented behavior.

The current health-contract version is `5.5.0`. The additive `5.5` line ships
The current health-contract version is `5.6.0`. The additive `5.6` line exposes
a provider-neutral frontend memory control plane for replaceable clients. The
additive `5.5` line ships
the shared reference Client SDK, bounded Task-event replay, and reconnect state
recovery. First-party WebUI, Desktop, and TUI clients now pass the same
conformance suite and no longer use internal REST routes for Task control,
Expand Down Expand Up @@ -76,6 +78,7 @@ below instead of assuming the old list.
| `tasks.structured-results-authorization` | Native Task events use A2A-aligned work states and expose factual `result`, typed `artifacts`, and `authorization`, without prescribing speech or UI | `test/gateway-event-schema.test.mjs`, `server/test/task-state.test.mjs` |
| `tasks.unified-id-updates` | A Task exposes one short `id`; `task.updated` carries adapter-normalized incremental messages and artifacts | `test/gateway-event-schema.test.mjs`, `server/test/task-manager.test.mjs` |
| `messages.citations` | Final assistant `transcript.final` events may carry normalized citations collected from frontend retrieval in the same turn | `test/gateway-event-schema.test.mjs`, `server/test/realtime-presentation-runtime.test.mjs` |
| `frontend.memory-control` | `GET/PATCH /api/memory` lets replaceable clients list and exactly edit the same provider-backed USER/MEMORY documents used by Realtime, without depending on a storage implementation | `server/test/gateway-application.test.mjs` |
| `realtime.conversation-client-v1` | `WS /api/realtime`, published event constants, and message schemas form the replaceable text/audio/multimodal Conversation Client boundary | `test/gateway-event-schema.test.mjs`, `test/custom-conversation-client.test.mjs` |
| `realtime.gateway-client-protocol-v6-handshake` | The same WebSocket accepts an opt-in 6.0 `session.hello`, returns correlated `session.ready`, negotiates implemented capabilities, and normalizes 6.0 input aliases into the existing business path | `test/gateway-client-protocol.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | Negotiated 6.0 Clients can publish registered semantic Client Events and use correlated Task, permission, conversation-history, and session output-voice commands over the same WebSocket; existing REST routes call the same command service as compatibility aliases | `test/gateway-client-protocol.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/client-command-runtime.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
Expand Down Expand Up @@ -174,6 +177,8 @@ await orb.load()
| Endpoint | Purpose |
| --- | --- |
| `GET /api/health` | Liveness, capability discovery and runtime status; includes `protocolVersion`, `capabilities`, `gatewayInstanceId`, `voiceConfigured`, `inputSuspension`, `voiceClients`, `backend` |
| `GET /api/memory` | List the current owner's bounded, provider-neutral frontend memory documents |
| `PATCH /api/memory` | Apply exact revision-checked edits to those documents; stale revisions return `409` |
| `POST /api/input/suspend` | Take the microphone: `{ owner, reason?, ttlMs? }`; default TTL 15 s, cap 300 s |
| `POST /api/input/resume` | Release it: `{ owner }` |
| `GET /api/input` | Current suspension status |
Expand Down
6 changes: 5 additions & 1 deletion docs/contract.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
Delivery、Client Action、参考 Client SDK 与有限回放均落在同一条 WebSocket 上。
已实现行为仍以本契约索引为准。

当前健康契约版本为 `5.5.0`。新增的 `5.5` 能力提供共享参考 Client SDK、有限 Task
当前健康契约版本为 `5.6.0`。新增的 `5.6` 能力为可替换客户端提供
Provider 无关的前台记忆控制面。`5.5` 能力提供共享参考 Client SDK、有限 Task
事件回放与断线状态恢复。第一方 WebUI、Desktop 和 TUI 已通过同一套一致性测试,
Task 控制、权限决策和对话历史不再依赖内部 REST 路由。`5.4` 能力提供有关联关系的 Client Action 与共享
Presence 状态机;`5.3` 增加 Provider 无关 Agent Delivery;`5.2` 在协商后的 6.0
Expand Down Expand Up @@ -62,6 +63,7 @@ Task 事件提供与 A2A 对齐的 `submitted`、
| `tasks.structured-results-authorization` | 原生 Task 事件使用与 A2A 对齐的工作状态,并暴露事实性 `result`、类型化 `artifacts` 与 `authorization`,不规定播报或 UI | `test/gateway-event-schema.test.mjs`、`server/test/task-state.test.mjs` |
| `tasks.unified-id-updates` | Task 只公开一个短 `id`;`task.updated` 携带 Adapter 归一化后的增量消息与产物 | `test/gateway-event-schema.test.mjs`、`server/test/task-manager.test.mjs` |
| `messages.citations` | 最终助手 `transcript.final` 可以携带同一轮前台检索产生的规范化 Citation | `test/gateway-event-schema.test.mjs`、`server/test/realtime-presentation-runtime.test.mjs` |
| `frontend.memory-control` | `GET/PATCH /api/memory` 供可替换客户端列出并精确编辑 Realtime 共用的 Provider 记忆文档,不暴露具体存储实现 | `server/test/gateway-application.test.mjs` |
| `realtime.conversation-client-v1` | `WS /api/realtime`、公开事件常量与消息 Schema 共同构成可替换的文本/音频/多模态对话客户端边界 | `test/gateway-event-schema.test.mjs`、`test/custom-conversation-client.test.mjs` |
| `realtime.gateway-client-protocol-v6-handshake` | 同一 WebSocket 可选择以 6.0 `session.hello` 接入,返回有关联关系的 `session.ready`,协商已实现能力,并把 6.0 输入别名归一化到现有业务路径 | `test/gateway-client-protocol.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
| `realtime.gateway-client-protocol-v6-runtime-commands` | 协商后的 6.0 Client 可以通过同一 WebSocket 发布已注册的语义 Client Event,并使用有关联结果的 Task、权限、对话历史和会话输出音色命令;现有 REST 路由调用同一命令服务作为兼容别名 | `test/gateway-client-protocol.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/client-command-runtime.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
Expand Down Expand Up @@ -158,6 +160,8 @@ await orb.load()
| 接口 | 用途 |
| --- | --- |
| `GET /api/health` | 存活、能力探测与运行状态;含 `protocolVersion`、`capabilities`、`gatewayInstanceId`、`voiceConfigured`、`inputSuspension`、`voiceClients`、`backend` |
| `GET /api/memory` | 列出当前 owner 有界、Provider 无关的前台记忆文档 |
| `PATCH /api/memory` | 按 revision 精确编辑这些文档;版本过期返回 `409` |
| `POST /api/input/suspend` | 抢占麦克风:`{ owner, reason?, ttlMs? }`,默认 15 秒,上限 300 秒 |
| `POST /api/input/resume` | 释放抢占:`{ owner }` |
| `GET /api/input` | 当前抢占状态 |
Expand Down
4 changes: 2 additions & 2 deletions docs/i18n-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@

## Current baseline

Updated 2026-08-28 against `github/main` at `8f16eb5`:
Updated 2026-09-02:

- Gateway health contract `5.5.0`;
- Gateway health contract `5.6.0`;
- stable Gateway Client Protocol wire version `6.0.0`;
- GCP1–GCP5 complete;
- WebUI, Desktop, and TUI on the shared reference Client SDK;
Expand Down
14 changes: 14 additions & 0 deletions docs/reference/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,20 @@ several durable changes; the Gateway still produces only one follow-up response.
starts from the latest document, and an exact replacement fails safely when its source fragment
is missing or ambiguous.

## Client Control Plane

Replaceable clients can manage the same memory through two Gateway endpoints:

- `GET /api/memory` returns the current owner's bounded `user` and `memory` documents.
- `PATCH /api/memory` accepts the same exact edits as the Realtime memory tool, including
`expectedRevision`; stale revisions return `409` so a client can reload instead of
overwriting a concurrent change.

This is a document control plane, not a second memory store. It is owner-scoped by the Gateway,
passes writes through `FrontendMemoryRuntime`, and therefore works unchanged with the default
Markdown provider or an injected provider. Clients should render only the formats they
understand and preserve exact source text when issuing a delete or replacement.

## Session Digests and Recall (off by default)

With `QWEN_AUDIO_SESSION_DIGEST=on`, each finished session records its topics and
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/memory.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,18 @@ Realtime 与自动整理都通过同一个记忆服务提交受限 Markdown 变
一句话包含多项持久修改时,Realtime 可在同一轮逐项调用,Gateway 只生成一次
后续回应。写入前会重新读取最新文档,精确替换找不到或匹配多处时安全失败。

## 客户端控制面

可替换客户端可以通过两个 Gateway 接口管理同一份记忆:

- `GET /api/memory` 返回当前 owner 有界的 `user` 与 `memory` 文档。
- `PATCH /api/memory` 接受与 Realtime 记忆工具相同的精确编辑,其中包含
`expectedRevision`;版本过期返回 `409`,客户端应重新读取,而不是覆盖并发修改。

这是一层文档控制面,不是第二套记忆存储。Gateway 负责 owner 隔离,写入统一经过
`FrontendMemoryRuntime`,所以默认 Markdown Provider 与外部注入 Provider 使用同一协议。
客户端只应展示自己理解的格式,删除或替换时必须保留并提交精确原文。

## 会话摘要与回溯(默认关闭)

设 `QWEN_AUDIO_SESSION_DIGEST=on` 后,会话结束时记下这一场的话题与一句不超过 50 字的
Expand Down
3 changes: 3 additions & 0 deletions examples/smart-cockpit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ A preflight validates the Realtime configuration and all four ports before any c
explicitly reports its demo fallback; a deployment replaces only that service.
- Route preference buttons write the authoritative cockpit state. The next route
inherits that preference unless the user explicitly chooses another one.
- The memory settings panel uses the Gateway's provider-neutral memory control
plane. It lists and exactly deletes entries from the same USER/MEMORY documents
used by Realtime, rather than maintaining a cockpit-only memory store.
- Other cockpit work goes through the fixed `spawn_thinking` bridge. The example
backend attaches over A2A, and Qwen3.8-Flash discovers the complete backend MCP
surface for vehicle control, navigation, music, flash-buy, and custom workflows,
Expand Down
2 changes: 2 additions & 0 deletions examples/smart-cockpit/README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ npm run example:smart-cockpit
- 位置查询与导航起点共用 Cockpit Service 的 `vehicleLocation()` 适配入口;
未接车机 GPS 时会明确返回 Demo 默认位置,部署时只需替换该服务。
- 路线偏好按钮写入座舱权威状态;用户未另行指定时,后续导航会继承该偏好。
- 记忆设置面板调用 Gateway 的 Provider 无关记忆控制面,列出并精确删除
Realtime 共用的 USER/MEMORY 文档条目,不维护座舱私有的另一份记忆。
- 其他座舱任务通过固定的 `spawn_thinking` 桥梁提交给后台。示例后台通过 A2A
接入 Gateway,Qwen3.8-Flash 会发现并调用独立的后台 MCP 工具面,完成车控、
导航、音乐、闪购和自定义技能任务,包括有序的多途经点导航。
Expand Down
2 changes: 2 additions & 0 deletions examples/smart-cockpit/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ Web Audio、3D 车辆与业务面板保持完全自主。
- `src/hooks/useVoiceSession.js`:GCP 连接、音频采集/播放、回执、Task 与最近对话恢复。
- `src/hooks/useCockpitState.js`:本示例的 HTTP/SSE 业务状态适配器。
- `src/hooks/useCockpitSkills.js`:按座舱读取、查看和删除持久化的自定义技能。
- `src/hooks/useGatewayMemory.js`:通过 Gateway 通用记忆控制面列出/删除前台记忆。
- `src/config/`:客户端拥有的人设展示和音色选项;不包含实际 Assistant Prompt。
- `src/projections/`:把 GCP/座舱 Service 事件投影为客户端展示状态。
- `src/App.jsx`:页面状态和对话/业务投影,不包含 Agent 或 Realtime Provider 逻辑。

“技能”设置页只负责展示和管理。技能创建与运行仍从语音对话进入 Gateway,
再通过固定的 `spawn_thinking` 桥梁交给后台座舱 Agent。
“记忆”设置页不创建独立座舱存储,而是管理 Realtime 实际使用的前台记忆文档。

从仓库根目录使用 `npm run example:smart-cockpit` 启动完整链路。单独开发 UI 时运行:

Expand Down
24 changes: 23 additions & 1 deletion examples/smart-cockpit/client/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -2554,6 +2554,26 @@ button {
gap: 18px;
}

.memory-toolbar {
display: flex;
align-items: center;
justify-content: space-between;
min-height: 32px;
color: #7f8d93;
font-size: 13px;
}

.memory-toolbar button:disabled {
opacity: 0.55;
cursor: default;
}

.settings-error {
margin: 12px 0 0;
color: #b54a62;
font-size: 13px;
}

.memory-item {
display: grid;
grid-template-columns: 1fr 34px;
Expand All @@ -2571,7 +2591,9 @@ button {
line-height: 1.35;
}

.memory-item time {
.memory-item time,
.memory-meta {
display: block;
color: #9aa6ab;
font-size: 12px;
}
Expand Down
19 changes: 18 additions & 1 deletion examples/smart-cockpit/client/src/App.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import MusicPanel, { PLAYLIST } from './components/MusicPanel'
import FlashBuyPanel from './components/FlashBuyPanel'
import useCockpitState from './hooks/useCockpitState'
import useCockpitSkills from './hooks/useCockpitSkills'
import useGatewayMemory from './hooks/useGatewayMemory'
import useVoiceSession from './hooks/useVoiceSession'
import {
finalUserTranscript,
Expand Down Expand Up @@ -41,7 +42,7 @@ const INITIAL_CAR_STATE = {
acFan: 3,
}

const VALID_TABS = ['persona', 'skills']
const VALID_TABS = ['persona', 'skills', 'memory']
const PERSONA_STORAGE_KEY = 'selectedPersona'
const VOICE_STORAGE_KEY = 'selectedVoice'
const INITIAL_WEATHER_STATE = {
Expand Down Expand Up @@ -107,6 +108,13 @@ export default function App() {
load: loadCustomSkill,
remove: deleteCustomSkill,
} = useCockpitSkills(cockpitId, cockpitActivity)
const {
items: memories,
loading: memoryLoading,
error: memoryError,
load: loadMemories,
remove: deleteMemory,
} = useGatewayMemory()
const [screen, setScreen] = useState('main')
const [settingsTab, setSettingsTab] = useState('persona')
const [selectedPersona, setSelectedPersona] = useState(() => getStoredChoice(
Expand Down Expand Up @@ -410,6 +418,10 @@ export default function App() {
return () => window.removeEventListener('hashchange', onHashChange)
}, [])

useEffect(() => {
if (screen === 'settings' && settingsTab === 'memory') loadMemories()
}, [loadMemories, screen, settingsTab])

return (
<main className="device" aria-label="车机语音交互原型">
<section className="screen">
Expand Down Expand Up @@ -459,6 +471,11 @@ export default function App() {
skillsError={customSkillsError}
onLoadSkill={loadCustomSkill}
onDeleteSkill={deleteCustomSkill}
memories={memories}
memoryLoading={memoryLoading}
memoryError={memoryError}
onDeleteMemory={deleteMemory}
onRefreshMemory={loadMemories}
/>
)}
</div>
Expand Down
39 changes: 39 additions & 0 deletions examples/smart-cockpit/client/src/components/MemoryTab.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
const TrashIcon = () => (
<svg className="icon icon-sm" viewBox="0 0 24 24" aria-hidden="true"><path d="M9 3h6l1 2h4v2H4V5h4l1-2Zm1 6h2v9h-2V9Zm4 0h2v9h-2V9ZM7 9h2v10h8V9h2v10a2 2 0 0 1-2 2H9a2 2 0 0 1-2-2V9Z" fill="currentColor" /></svg>
)

export default function MemoryTab({ items, loading, error, onDelete, onRefresh }) {
return (
<div className="memory-scroll">
<div className="memory-toolbar">
<span>与前台对话共用的长期记忆</span>
<button className="collapse-link" onClick={onRefresh} disabled={loading}>
{loading ? '同步中' : '刷新'}
</button>
</div>
{error && <p className="settings-error" role="status">{error}</p>}
<ul className="memory-list" aria-label="智能体记忆列表">
{items.map(item => (
<li key={item.id} className="memory-item">
<span>
<strong>{item.text}</strong>
<span className="memory-meta">
{item.scopeLabel}{item.section ? ` · ${item.section}` : ''}
</span>
</span>
<button
className="trash-btn"
aria-label={`删除记忆:${item.text}`}
onClick={() => onDelete(item)}
>
<TrashIcon />
</button>
</li>
))}
</ul>
{!loading && items.length === 0 && !error && (
<div className="empty-state" style={{ display: 'block' }}>暂无智能体记忆</div>
)}
</div>
)
}
12 changes: 12 additions & 0 deletions examples/smart-cockpit/client/src/components/SettingsPanel.jsx
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
import PersonaTab from './PersonaTab'
import SkillTab from './SkillTab'
import MemoryTab from './MemoryTab'

const TABS = [
{ id: 'persona', label: '个性化' },
{ id: 'skills', label: '技能' },
{ id: 'memory', label: '记忆' },
]

export default function SettingsPanel({
Expand All @@ -12,6 +14,7 @@ export default function SettingsPanel({
selectedVoice, onSelectVoice,
selectedWake, onSelectWake,
skills, skillsError, onLoadSkill, onDeleteSkill,
memories, memoryLoading, memoryError, onDeleteMemory, onRefreshMemory,
}) {
return (
<section className="settings-panel">
Expand Down Expand Up @@ -43,6 +46,15 @@ export default function SettingsPanel({
onDelete={onDeleteSkill}
/>
</div>
<div className={`tab-page ${activeTab === 'memory' ? 'is-active' : ''}`}>
<MemoryTab
items={memories}
loading={memoryLoading}
error={memoryError}
onDelete={onDeleteMemory}
onRefresh={onRefreshMemory}
/>
</div>
</section>
)
}
13 changes: 13 additions & 0 deletions examples/smart-cockpit/client/src/config/gateway.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
function gatewayOrigin() {
return import.meta.env.VITE_GATEWAY_ORIGIN || window.location.origin
}

export function gatewayHttpUrl(path) {
return new URL(path, gatewayOrigin()).toString()
}

export function gatewayWebSocketUrl(path) {
const url = new URL(path, gatewayOrigin())
url.protocol = url.protocol === 'https:' ? 'wss:' : 'ws:'
return url
}
65 changes: 65 additions & 0 deletions examples/smart-cockpit/client/src/hooks/useGatewayMemory.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import { useCallback, useMemo, useState } from 'react'
import { gatewayHttpUrl } from '../config/gateway'
import { memoryItemsFromDocuments } from '../projections/memory-items'

async function responsePayload(response) {
return response.json().catch(() => ({}))
}

export default function useGatewayMemory() {
const [documents, setDocuments] = useState([])
const [loading, setLoading] = useState(true)
const [error, setError] = useState(null)

const load = useCallback(async () => {
setLoading(true)
try {
const response = await fetch(gatewayHttpUrl('/api/memory'))
const payload = await responsePayload(response)
if (!response.ok) throw new Error(payload.error || `HTTP ${response.status}`)
setDocuments(Array.isArray(payload.documents) ? payload.documents : [])
setError(null)
} catch (reason) {
setError(reason?.message || '记忆服务不可用')
} finally {
setLoading(false)
}
}, [])

const remove = useCallback(async (item) => {
try {
const response = await fetch(gatewayHttpUrl('/api/memory'), {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
changes: [{
document: item.scope,
expectedRevision: item.revision,
edits: [{ old_text: item.oldText, new_text: '' }],
}],
}),
})
const payload = await responsePayload(response)
if (!response.ok) {
throw Object.assign(
new Error(payload.error || `HTTP ${response.status}`),
{ stale: response.status === 409 },
)
}
await load()
return true
} catch (reason) {
setError(reason?.message || '删除记忆失败')
if (reason?.stale) await load()
return false
}
}, [load])

return {
items: useMemo(() => memoryItemsFromDocuments(documents), [documents]),
loading,
error,
load,
remove,
}
}
Loading