diff --git a/CHANGELOG.md b/CHANGELOG.md index fc85223a..3767612c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +- WebUI 新增用户主动触发的一次性拍照输入：照片会在浏览器本地压缩后作为附件发送， + 不会自动调用模型或持久化摄像头画面。 +- WebUI 支持用户主动开启摄像头连续观察：约每秒发送一帧 JPEG，内存中最多保留最近 + 8 帧；不会因观察自动触发模型回复，页面隐藏、断线或停止时会释放摄像头。 - 后台 Session 模型覆盖统一使用 ACP `configOptions` 与 `session/set_config_option`，不再调用私有模型接口或生成后台配置文件；未声明标准 模型选项的 Agent 将沿用自身配置。OpenCode/OpenClaw 一键托管初始化保持不变。 diff --git a/README.md b/README.md index 3170b7f1..b35aa697 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,15 @@ QWEN_AUDIO_AGENT_BACKEND_MODEL=qwen3.7-max ``` > Uses DashScope realtime voice frontend by default; alternatively, switch to a local [speech-to-speech frontend](docs/voice-frontends/speech-to-speech.md), no cloud API Key needed. +> `qwen3.5-omni-flash-realtime` and `qwen3.5-omni-plus-realtime` +> accept text, audio, and image at the model level. The WebUI also supports explicit +> camera observation for these Omni models: it sends one bounded JPEG frame per second, +> keeps at most eight recent frames in memory, and never creates a response from observation +> alone. Native video and observation on legacy Audio models remain unavailable. + +The Desktop app or `qwenaudio config set --realtime-model ` configures the single +Gateway-wide model. Restart the Gateway after a CLI change. WebUI and TUI display the active +model but do not override it. 2. Start the Gateway, then open another terminal to start the TUI (or use `qwenaudio webui` for the browser UI): diff --git a/README_ZH.md b/README_ZH.md index 1e988aa7..bcab8109 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -135,7 +135,13 @@ QWEN_AUDIO_AGENT_BACKEND_MODEL=qwen3.7-max ``` > 默认使用 DashScope 实时语音前台；也可切换为本地 [speech-to-speech 前台](docs/voice-frontends/speech-to-speech.zh.md)，无需云端 API Key。 +> `qwen3.5-omni-flash-realtime` 与 `qwen3.5-omni-plus-realtime` +> 在模型层支持文本、音频和图片输入。WebUI 对这两个 Omni 模型还支持用户显式开启的 +> 摄像头连续观察：每秒发送一帧有界 JPEG，内存中最多保留最近 8 帧，观察本身不会创建 +> 模型回复。原生视频和旧版 Audio 模型的画面观察仍不可用。 +桌面版或 `qwenaudio config set --realtime-model ` 配置 Gateway 统一使用的模型； +CLI 修改后需要重启 Gateway。WebUI 与 TUI 只显示当前生效模型，不单独覆盖模型。 2. 启动 Gateway，另开终端启动 TUI（也可用 `qwenaudio webui` 启动浏览器界面）： diff --git a/desktop/test/input-capabilities.test.mjs b/desktop/test/input-capabilities.test.mjs index 6f1226c4..99254b0b 100644 --- a/desktop/test/input-capabilities.test.mjs +++ b/desktop/test/input-capabilities.test.mjs @@ -10,6 +10,7 @@ test('desktop connection supports the conversation panel inputs', () => { text: true, audio: true, image: true, + observation: true, resource: true, }) assert.equal(supportsComposerInput('desktop'), true) diff --git a/docs/configuration/frontend.md b/docs/configuration/frontend.md index f3a3f7c9..b137b6eb 100644 --- a/docs/configuration/frontend.md +++ b/docs/configuration/frontend.md @@ -45,16 +45,17 @@ The exact supported IDs are: | Model | Model input | Model output | Current client transport | | --- | --- | --- | --- | -| `qwen3.5-omni-flash-realtime` | text, audio, image | text, audio | text, audio | -| `qwen3.5-omni-plus-realtime` | text, audio, image | text, audio | text, audio | +| `qwen3.5-omni-flash-realtime` | text, audio, image | text, audio | text, audio, JPEG observation | +| `qwen3.5-omni-plus-realtime` | text, audio, image | text, audio | text, audio, JPEG observation | | `qwen-audio-3.0-realtime-plus` (default) | text, audio | text, audio | text, audio | | `qwen-audio-3.0-realtime-flash` | text, audio | text, audio | text, audio | -All four profiles support Function Calling. Model capability is not the same as an implemented -client transport: JPEG observation frames and native video are both disabled in this release. -WebUI and TUI read the authoritative profile from Gateway health and only display it. Separate +All four profiles support Function Calling. The two Omni profiles support the WebUI's explicit +JPEG observation transport; frames are sent at about 1 FPS, capped at eight recent in-memory +frames, and do not create model responses by themselves. Native video and observation on the +legacy Audio profiles remain unavailable. WebUI and TUI read the authoritative profile from +Gateway health and use it to gate or display the available inputs. Separate clients cannot select conflicting models on one Gateway. A Desktop attached to a borrowed Gateway, or a later CLI runtime using a conflicting configured model, refuses the mismatch instead of silently changing the running service. To roll back, set the legacy ID above and restart the Gateway. - diff --git a/docs/configuration/frontend.zh.md b/docs/configuration/frontend.zh.md index d93e0761..4d6a9a77 100644 --- a/docs/configuration/frontend.zh.md +++ b/docs/configuration/frontend.zh.md @@ -44,14 +44,14 @@ qwenaudio gateway restart | 模型 | 模型输入 | 模型输出 | 当前客户端传输 | | --- | --- | --- | --- | -| `qwen3.5-omni-flash-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频 | -| `qwen3.5-omni-plus-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频 | +| `qwen3.5-omni-flash-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频、JPEG 观察 | +| `qwen3.5-omni-plus-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频、JPEG 观察 | | `qwen-audio-3.0-realtime-plus`（默认） | 文本、音频 | 文本、音频 | 文本、音频 | | `qwen-audio-3.0-realtime-flash` | 文本、音频 | 文本、音频 | 文本、音频 | -四个档案都支持 Function Calling。模型能力不等于客户端已经实现的传输能力：本版本 -仍关闭 JPEG 观察帧和原生视频传输。WebUI 与 TUI 从 Gateway health 读取权威档案并 -只读展示；同一 Gateway 上的不同客户端不能选择互相冲突的模型。桌面版附着到借用的 +四个档案都支持 Function Calling。两个 Omni 档案支持 WebUI 显式开启的 JPEG 观察传输： +约每秒发送一帧，内存中最多保留最近 8 帧，观察本身不会创建模型回复。原生视频以及 +旧版 Audio 档案的画面观察仍不可用。WebUI 与 TUI 从 Gateway health 读取权威档案，并据此 +限制或展示可用输入；同一 Gateway 上的不同客户端不能选择互相冲突的模型。桌面版附着到借用的 Gateway 时，或后续 CLI 运行时使用了冲突的已配置模型时，会拒绝不一致，而不会静默 修改运行中服务。回滚时设置上表的旧版模型 ID 并重启 Gateway。 - diff --git a/docs/contract.md b/docs/contract.md index 64c301bc..c15718f2 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -27,7 +27,9 @@ runtime-command plane, Agent Delivery, Client Actions, reference Client SDK, and bounded replay all share the same WebSocket. This contract index remains authoritative for implemented behavior. -The current health-contract version is `5.6.0`. The additive `5.6` line exposes +The current health-contract version is `5.7.0`. The additive `5.7` line exposes +explicit WebUI camera observation for vision-capable Realtime frontends without +creating model responses. The additive `5.6` line exposes a provider-neutral frontend memory control plane for replaceable clients. The additive `5.5` line ships the shared reference Client SDK, bounded Task-event replay, and reconnect state @@ -80,6 +82,7 @@ below instead of assuming the old list. | `messages.citations` | Final assistant `transcript.final` events may carry normalized citations collected from frontend retrieval in the same turn | `test/gateway-event-schema.test.mjs`, `server/test/realtime-presentation-runtime.test.mjs` | | `frontend.memory-control` | `GET/PATCH /api/memory` lets replaceable clients list and exactly edit the same provider-backed USER/MEMORY documents used by Realtime, without depending on a storage implementation | `server/test/gateway-application.test.mjs` | | `realtime.conversation-client-v1` | `WS /api/realtime`, published event constants, and message schemas form the replaceable text/audio/multimodal Conversation Client boundary | `test/gateway-event-schema.test.mjs`, `test/custom-conversation-client.test.mjs` | +| `realtime.camera-observation-v1` | After an explicit WebUI opt-in, sends at most one bounded JPEG frame per second and retains at most eight recent frames for a vision-capable Realtime frontend; it never creates a model response and releases the camera on stop, disconnect, or page hide | `server/test/realtime-observation-runtime.test.mjs`, `web/test/camera-input.test.mjs` | | `realtime.gateway-client-protocol-v6-handshake` | The same WebSocket accepts an opt-in 6.0 `session.hello`, returns correlated `session.ready`, negotiates implemented capabilities, and normalizes 6.0 input aliases into the existing business path | `test/gateway-client-protocol.test.mjs`, `server/test/gateway-client-handshake.test.mjs` | | `realtime.gateway-client-protocol-v6-runtime-commands` | Negotiated 6.0 Clients can publish registered semantic Client Events and use correlated Task, permission, conversation-history, and session output-voice commands over the same WebSocket; existing REST routes call the same command service as compatibility aliases | `test/gateway-client-protocol.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/client-command-runtime.test.mjs`, `server/test/gateway-client-handshake.test.mjs` | | `realtime.gateway-client-protocol-v6-agent-delivery` | Client Events, Task results and progress, and permission prompts cross one provider-neutral `AgentDelivery` boundary with `handle`, `context`, `respond`, and `interrupt` modes | `server/test/agent-delivery.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/realtime-provider.test.mjs`, `server/test/announcement-manager.test.mjs` | diff --git a/docs/contract.zh.md b/docs/contract.zh.md index e1866075..46ef4b73 100644 --- a/docs/contract.zh.md +++ b/docs/contract.zh.md @@ -22,7 +22,8 @@ Delivery、Client Action、参考 Client SDK 与有限回放均落在同一条 WebSocket 上。 已实现行为仍以本契约索引为准。 -当前健康契约版本为 `5.6.0`。新增的 `5.6` 能力为可替换客户端提供 +当前健康契约版本为 `5.7.0`。新增的 `5.7` 能力为支持视觉输入的 Realtime +前台提供 WebUI 显式开启的摄像头画面观察，不会创建模型回复。新增的 `5.6` 能力为可替换客户端提供 Provider 无关的前台记忆控制面。`5.5` 能力提供共享参考 Client SDK、有限 Task 事件回放与断线状态恢复。第一方 WebUI、Desktop 和 TUI 已通过同一套一致性测试， Task 控制、权限决策和对话历史不再依赖内部 REST 路由。`5.4` 能力提供有关联关系的 Client Action 与共享 @@ -65,6 +66,7 @@ Task 事件提供与 A2A 对齐的 `submitted`、 | `messages.citations` | 最终助手 `transcript.final` 可以携带同一轮前台检索产生的规范化 Citation | `test/gateway-event-schema.test.mjs`、`server/test/realtime-presentation-runtime.test.mjs` | | `frontend.memory-control` | `GET/PATCH /api/memory` 供可替换客户端列出并精确编辑 Realtime 共用的 Provider 记忆文档，不暴露具体存储实现 | `server/test/gateway-application.test.mjs` | | `realtime.conversation-client-v1` | `WS /api/realtime`、公开事件常量与消息 Schema 共同构成可替换的文本/音频/多模态对话客户端边界 | `test/gateway-event-schema.test.mjs`、`test/custom-conversation-client.test.mjs` | +| `realtime.camera-observation-v1` | WebUI 显式开启后，以约 1 FPS 发送最近最多 8 帧 JPEG 到支持视觉输入的 Realtime 前台；不自动创建模型响应，停止、断线、隐藏页面时释放相机 | `server/test/realtime-observation-runtime.test.mjs`、`web/test/camera-input.test.mjs` | | `realtime.gateway-client-protocol-v6-handshake` | 同一 WebSocket 可选择以 6.0 `session.hello` 接入，返回有关联关系的 `session.ready`，协商已实现能力，并把 6.0 输入别名归一化到现有业务路径 | `test/gateway-client-protocol.test.mjs`、`server/test/gateway-client-handshake.test.mjs` | | `realtime.gateway-client-protocol-v6-runtime-commands` | 协商后的 6.0 Client 可以通过同一 WebSocket 发布已注册的语义 Client Event，并使用有关联结果的 Task、权限、对话历史和会话输出音色命令；现有 REST 路由调用同一命令服务作为兼容别名 | `test/gateway-client-protocol.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/client-command-runtime.test.mjs`、`server/test/gateway-client-handshake.test.mjs` | | `realtime.gateway-client-protocol-v6-agent-delivery` | Client Event、Task 结果与低频进展、权限请求统一跨越 Provider 无关 `AgentDelivery` 边界，并支持 `handle`、`context`、`respond`、`interrupt` 四种模式 | `server/test/agent-delivery.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/realtime-provider.test.mjs`、`server/test/announcement-manager.test.mjs` | diff --git a/docs/i18n-parity.md b/docs/i18n-parity.md index a3ef5d59..b7cd4a76 100644 --- a/docs/i18n-parity.md +++ b/docs/i18n-parity.md @@ -7,7 +7,8 @@ Updated 2026-09-02: -- Gateway health contract `5.6.0`; +- Gateway health contract `5.7.0`; +- WebUI explicit camera observation for vision-capable Realtime frontends; - stable Gateway Client Protocol wire version `6.0.0`; - GCP1–GCP5 complete; - WebUI, Desktop, and TUI on the shared reference Client SDK; diff --git a/examples/smart-cockpit/gateway/test/composition.test.mjs b/examples/smart-cockpit/gateway/test/composition.test.mjs index 819d5fbe..d8fbee4e 100644 --- a/examples/smart-cockpit/gateway/test/composition.test.mjs +++ b/examples/smart-cockpit/gateway/test/composition.test.mjs @@ -42,7 +42,7 @@ test('composes the public Gateway API with the replaceable A2A Agent', async t = assert.equal(response.status, 200) assert.equal(health.ok, true) - assert.equal(health.protocolVersion, '5.6.0') + assert.equal(health.protocolVersion, '5.7.0') assert.equal(health.frontendProfile.name, 'cockpit-example') assert.equal(health.backend.protocol, 'a2a') assert.equal(health.backend.status, 'ready') diff --git a/server/src/core/gateway-protocol.mjs b/server/src/core/gateway-protocol.mjs index ac8bd1e1..3a0c1c16 100644 --- a/server/src/core/gateway-protocol.mjs +++ b/server/src/core/gateway-protocol.mjs @@ -10,6 +10,8 @@ // Every capability listed here is locked by a test (see docs/contract.md); // anything not listed is internal and may change in any release. // +// 5.7.0 adds explicit WebUI camera observation for vision-capable Realtime +// frontends without triggering model responses. // 5.6.0 adds a provider-neutral frontend memory control plane for replaceable // clients to list and edit the same documents used by Realtime. // 5.5.0 adds the GCP5 reference Client SDK, bounded Task-event replay, and @@ -43,7 +45,7 @@ // desktop.settings-window, …) are not part of this contract, and a removed // capability is a breaking change. Hosts migrating from the fork must branch // on the capability list below, never on the version number. -export const GATEWAY_PROTOCOL_VERSION = '5.6.0' +export const GATEWAY_PROTOCOL_VERSION = '5.7.0' export const GATEWAY_CAPABILITIES = Object.freeze([ // The Gateway statically hosts web/dist at its own origin, so a client may @@ -102,6 +104,9 @@ export const GATEWAY_CAPABILITIES = Object.freeze([ // replaceable Conversation Client boundary for audio, text, multimodal // input, transcripts, playback receipts, voice state and Task projections. 'realtime.conversation-client-v1', + // The WebUI may explicitly open a camera and stream bounded JPEG snapshots + // to a vision-capable Realtime frontend without triggering responses. + 'realtime.camera-observation-v1', // session.hello/session.ready negotiate the stable 6.0 Client protocol; // 5.x connect and event names remain compatibility aliases through one // normalization layer. diff --git a/server/src/voice/providers/openai-compatible-protocol.mjs b/server/src/voice/providers/openai-compatible-protocol.mjs index 116abbea..ba421519 100644 --- a/server/src/voice/providers/openai-compatible-protocol.mjs +++ b/server/src/voice/providers/openai-compatible-protocol.mjs @@ -28,6 +28,11 @@ export const openAiCompatibleProtocol = Object.freeze({ audio, }), + imageAppend: image => ({ + type: 'input_image_buffer.append', + image, + }), + // Client-assigned id for a conversation item. The beta dialect accepts one // opaque namespace for every item type. conversationItemId: () => `item_${randomUUID().replaceAll('-', '')}`, diff --git a/server/src/voice/realtime-gateway.mjs b/server/src/voice/realtime-gateway.mjs index 6a469b07..4cb8f42a 100644 --- a/server/src/voice/realtime-gateway.mjs +++ b/server/src/voice/realtime-gateway.mjs @@ -27,6 +27,7 @@ import { ToolCallHandler } from './tools/tool-call-handler.mjs' import { TurnTranscripts } from './tools/turn-transcripts.mjs' import { TurnCitations } from './turn-citations.mjs' import { RealtimeInputRuntime } from './realtime-input-runtime.mjs' +import { RealtimeObservationRuntime } from './realtime-observation-runtime.mjs' import { acceptsPlaybackReceipt, confirmsTaskNotificationOnPlaybackStart, @@ -309,6 +310,7 @@ export function attachRealtimeGateway(server, { const announcedInputs = new Set() let permissionRetryTimer = null let realtimeSession + let observationRuntime const agentDeliveries = new RealtimeAgentDeliveryRuntime({ getFrontend: () => realtimeSession?.frontend, isDeliveryBlocked: () => ( @@ -587,10 +589,13 @@ export function attachRealtimeGateway(server, { announcements.flush() } }, - onDisconnected: () => send(ws, { - type: GatewayServerEvent.VOICE_STATE, - state: 'idle', - }), + onDisconnected: () => { + observationRuntime?.stop('realtime_disconnected') + send(ws, { + type: GatewayServerEvent.VOICE_STATE, + state: 'idle', + }) + }, onReconnected: () => { announcements.flush() progressAnnouncements.flush() @@ -611,6 +616,12 @@ export function attachRealtimeGateway(server, { ? { createFrontend: realtimeFrontendFactory } : {}), }) + observationRuntime = new RealtimeObservationRuntime({ + ensureFrontend: () => realtimeSession.ensure(), + getFrontend: () => realtimeSession.frontend, + send: event => send(ws, event), + onError: reportFrontendError, + }) const voiceClient = { ws, descriptor, @@ -624,6 +635,7 @@ export function attachRealtimeGateway(server, { if (suspend) { // Buffered audio predates the suspension and is no longer wanted. realtimeSession.clearPendingAudio() + observationRuntime?.stop('input_suspended') sleepController?.disable() realtimeSession.cancelResponse() send(ws, { type: GatewayServerEvent.PLAYBACK_CLEAR, reason: 'input_suspended' }) @@ -651,6 +663,7 @@ export function attachRealtimeGateway(server, { sleepController?.disable() inputEnabled = false outputEnabled = false + observationRuntime?.stop('voice_deactivated') announcementWindow.reset() announcements.pause() progressAnnouncements.clear() @@ -1092,6 +1105,7 @@ export function attachRealtimeGateway(server, { const enterSleep = () => { if (sleeping) return + observationRuntime?.stop('sleeping') sleeping = true waking = false announcementWindow.reset() @@ -1478,6 +1492,7 @@ export function attachRealtimeGateway(server, { text: event.inputCapabilities.text === true, audio: event.inputCapabilities.audio === true, image: event.inputCapabilities.image === true, + observation: event.inputCapabilities.observation === true, resource: event.inputCapabilities.resource === true, } : null @@ -1550,6 +1565,29 @@ export function attachRealtimeGateway(server, { return } realtimeSession.appendAudio(event.audio) + } else if (event.type === GatewayClientEvent.OBSERVATION_START) { + if (sleeping || waking) { + send(ws, { + type: GatewayServerEvent.ERROR, + message: `已休眠，请先说“${config.wakeWord}”唤醒。`, + }) + return + } + if (inputSuspended) { + send(ws, { + type: GatewayServerEvent.ERROR, + message: '当前输入正被其他客户端占用，无法进行画面观察。', + }) + return + } + sleepController.recordActivity() + observationRuntime.start() + } else if (event.type === GatewayClientEvent.OBSERVATION_FRAME) { + if (sleeping || waking) return + sleepController.recordActivity() + observationRuntime.frame(event) + } else if (event.type === GatewayClientEvent.OBSERVATION_STOP) { + observationRuntime.stop(event.reason || 'user') } else if ( event.type === GatewayClientEvent.TEXT_MESSAGE || event.type === GatewayClientEvent.INPUT_MESSAGE @@ -1603,6 +1641,7 @@ export function attachRealtimeGateway(server, { }) } } else if (event.type === GatewayClientEvent.MUTE) { + observationRuntime.stop('voice_muted') releaseVoiceClient() sleeping = false waking = false @@ -1614,6 +1653,7 @@ export function attachRealtimeGateway(server, { realtimeSession.close({ notifyDisconnected: true }) } else if (event.type === GatewayClientEvent.INPUT_MUTE) { inputEnabled = false + observationRuntime.stop('input_muted') realtimeSession.clearPendingAudio() } else if (event.type === GatewayClientEvent.SLEEP) { requestExplicitSleep('client') @@ -1651,6 +1691,7 @@ export function attachRealtimeGateway(server, { clearTimeout(permissionRetryTimer) permissionRetryTimer = null sleepController?.close() + observationRuntime?.stop('gateway_disconnected') presenceController.close() realtimeSession.close() // Invisible memory: distil durable personal facts from this session in diff --git a/server/src/voice/realtime-observation-runtime.mjs b/server/src/voice/realtime-observation-runtime.mjs new file mode 100644 index 00000000..61d83ebd --- /dev/null +++ b/server/src/voice/realtime-observation-runtime.mjs @@ -0,0 +1,190 @@ +import { GatewayServerEvent } from '../../../shared/realtime-events.mjs' + +export const OBSERVATION_INTERVAL_MS = 1000 +export const OBSERVATION_MAX_FRAMES = 8 +export const OBSERVATION_MAX_BASE64_BYTES = 256 * 1024 + +const BASE64_PATTERN = /^[A-Za-z0-9+/]*={0,2}$/ + +function normalizeImage(value, maxBytes) { + const image = String(value || '').trim() + if (!image) throw new Error('画面观察缺少图片数据') + if (image.length > maxBytes) { + throw new Error('画面观察图片超过 256 KiB 限制') + } + if (image.length % 4 === 1 || !BASE64_PATTERN.test(image)) { + throw new Error('画面观察图片不是有效的 Base64 数据') + } + return image +} + +function statePayload(state, frames) { + return { + type: GatewayServerEvent.OBSERVATION_STATE, + state, + frames, + } +} + +/** + * Relays explicitly-enabled browser camera frames to a Realtime provider. + * + * This runtime deliberately never calls response.create. The image buffer is + * part of the provider's current audio timeline and is consumed by the next + * user turn. Raw Base64 is kept only in a bounded in-memory ring while this + * connection is observing; stop(), provider disconnects, and socket teardown + * clear it. + */ +export class RealtimeObservationRuntime { + constructor({ + ensureFrontend, + getFrontend, + send, + onError, + now = () => Date.now(), + intervalMs = OBSERVATION_INTERVAL_MS, + maxFrames = OBSERVATION_MAX_FRAMES, + maxBase64Bytes = OBSERVATION_MAX_BASE64_BYTES, + } = {}) { + this.ensureFrontend = ensureFrontend + this.getFrontend = getFrontend + this.send = send + this.onError = onError + this.now = now + this.intervalMs = Math.max(1, Number(intervalMs) || OBSERVATION_INTERVAL_MS) + this.maxFrames = Math.max(1, Math.floor(Number(maxFrames) || OBSERVATION_MAX_FRAMES)) + this.maxBase64Bytes = Math.max( + 4, + Math.floor(Number(maxBase64Bytes) || OBSERVATION_MAX_BASE64_BYTES), + ) + this.state = 'idle' + this.frames = [] + this.lastFrameAt = 0 + this.generation = 0 + this.startPromise = null + } + + snapshot() { + return { + state: this.state, + frames: this.frames.length, + lastFrameAt: this.lastFrameAt, + } + } + + start() { + if (this.state === 'active') return Promise.resolve(true) + if (this.startPromise) return this.startPromise + + const generation = ++this.generation + this.frames = [] + this.lastFrameAt = 0 + this.state = 'starting' + this.#publish() + + const start = Promise.resolve() + .then(() => this.ensureFrontend?.()) + .then(() => { + if (generation !== this.generation) return false + const frontend = this.getFrontend?.() + if (!frontend || frontend.transportCapabilities?.observationInput !== true) { + throw new Error('当前 Realtime 模型不支持画面观察') + } + this.state = 'active' + this.#publish() + return true + }) + .catch(error => { + if (generation !== this.generation) return false + this.state = 'unavailable' + this.frames = [] + this.lastFrameAt = 0 + this.#publish() + this.onError?.(error) + return false + }) + .finally(() => { + if (this.startPromise === start) this.startPromise = null + }) + this.startPromise = start + return start + } + + stop(reason = 'user') { + const safeReason = String(reason || 'user').slice(0, 80) + const hadObservation = ( + this.state !== 'idle' + || Boolean(this.startPromise) + || this.frames.length > 0 + ) + this.generation += 1 + this.startPromise = null + this.state = 'idle' + this.frames = [] + this.lastFrameAt = 0 + if (hadObservation) this.#publish(safeReason) + } + + frame({ image, sequence } = {}) { + if (this.state !== 'active') return false + + let normalized + try { + normalized = normalizeImage(image, this.maxBase64Bytes) + } catch (error) { + this.onError?.(error) + return false + } + + const now = this.now() + if (this.frames.length && now - this.lastFrameAt < this.intervalMs) return false + + const frontend = this.getFrontend?.() + if (!frontend || frontend.transportCapabilities?.observationInput !== true) { + this.stop('provider_unavailable') + this.onError?.(new Error('画面观察的 Realtime 连接不可用')) + return false + } + if (typeof frontend.send !== 'function') { + this.stop('provider_unavailable') + this.onError?.(new Error('Realtime 前台不支持画面观察传输')) + return false + } + + try { + // DashScope Qwen-Omni WebSocket input contract. No response.create is + // sent here, so observation alone never makes the assistant speak. + const payload = typeof frontend.protocol?.imageAppend === 'function' + ? frontend.protocol.imageAppend(normalized) + : { + type: 'input_image_buffer.append', + image: normalized, + } + frontend.send(payload) + } catch (error) { + this.onError?.(error) + return false + } + + this.lastFrameAt = now + this.frames.push({ + image: normalized, + sequence: Number.isInteger(sequence) ? sequence : null, + capturedAt: now, + }) + if (this.frames.length > this.maxFrames) { + this.frames.splice(0, this.frames.length - this.maxFrames) + } + this.#publish() + return true + } + + #publish(reason = '') { + this.send?.({ + ...statePayload(this.state, this.frames.length), + ...(reason ? { reason } : {}), + }) + } +} + +export { normalizeImage as normalizeObservationImage } diff --git a/server/test/gateway-client-protocol-session.test.mjs b/server/test/gateway-client-protocol-session.test.mjs index f5f0eed5..7f7ce537 100644 --- a/server/test/gateway-client-protocol-session.test.mjs +++ b/server/test/gateway-client-protocol-session.test.mjs @@ -50,4 +50,3 @@ test('shares a bounded Task replay cursor across protocol connections', () => { assert.deepEqual(outcome.reply.events.map(event => event.sequence), [2]) assert.equal(outcome.reply.has_more, false) }) - diff --git a/server/test/realtime-observation-runtime.test.mjs b/server/test/realtime-observation-runtime.test.mjs new file mode 100644 index 00000000..9657ab5e --- /dev/null +++ b/server/test/realtime-observation-runtime.test.mjs @@ -0,0 +1,115 @@ +import assert from 'node:assert/strict' +import test from 'node:test' +import { + OBSERVATION_INTERVAL_MS, + OBSERVATION_MAX_FRAMES, + RealtimeObservationRuntime, +} from '../src/voice/realtime-observation-runtime.mjs' +import { openAiCompatibleProtocol } from '../src/voice/providers/openai-compatible-protocol.mjs' + +function fixture({ supported = true } = {}) { + let now = 0 + const providerEvents = [] + const gatewayEvents = [] + const errors = [] + const frontend = { + transportCapabilities: { observationInput: supported }, + send: event => providerEvents.push(event), + } + const runtime = new RealtimeObservationRuntime({ + ensureFrontend: async () => {}, + getFrontend: () => frontend, + send: event => gatewayEvents.push(event), + onError: error => errors.push(error), + now: () => now, + }) + return { + runtime, + frontend, + providerEvents, + gatewayEvents, + errors, + setNow(value) { + now = value + }, + } +} + +const JPEG = '/9j/fake' + +test('encodes a camera frame with the DashScope image-buffer event', () => { + assert.deepEqual(openAiCompatibleProtocol.imageAppend(JPEG), { + type: 'input_image_buffer.append', + image: JPEG, + }) +}) + +test('starts explicitly and forwards observation frames without creating responses', async () => { + const fixtureState = fixture() + assert.equal(await fixtureState.runtime.start(), true) + assert.deepEqual( + fixtureState.gatewayEvents.map(event => event.state), + ['starting', 'active'], + ) + + assert.equal(fixtureState.runtime.frame({ image: JPEG, sequence: 1 }), true) + assert.deepEqual(fixtureState.providerEvents, [{ + type: 'input_image_buffer.append', + image: JPEG, + }]) + assert.equal( + fixtureState.providerEvents.some(event => event.type === 'response.create'), + false, + ) +}) + +test('limits observation to about one frame per second and retains eight frames', async () => { + const fixtureState = fixture() + await fixtureState.runtime.start() + for (let sequence = 0; sequence < OBSERVATION_MAX_FRAMES + 2; sequence += 1) { + fixtureState.setNow(sequence * OBSERVATION_INTERVAL_MS) + assert.equal(fixtureState.runtime.frame({ image: JPEG, sequence }), true) + } + fixtureState.setNow(OBSERVATION_INTERVAL_MS * (OBSERVATION_MAX_FRAMES + 1) + 1) + assert.equal(fixtureState.runtime.frame({ image: JPEG, sequence: 99 }), false) + assert.equal(fixtureState.runtime.snapshot().frames, OBSERVATION_MAX_FRAMES) + assert.equal(fixtureState.providerEvents.length, OBSERVATION_MAX_FRAMES + 2) +}) + +test('rejects malformed or oversized frames without forwarding them', async () => { + const fixtureState = fixture() + await fixtureState.runtime.start() + assert.equal(fixtureState.runtime.frame({ image: 'not base64!' }), false) + assert.equal( + fixtureState.runtime.frame({ image: `A${'a'.repeat(256 * 1024)}` }), + false, + ) + assert.equal(fixtureState.providerEvents.length, 0) + assert.equal(fixtureState.errors.length, 2) +}) + +test('fails closed when the active model does not support observation', async () => { + const fixtureState = fixture({ supported: false }) + assert.equal(await fixtureState.runtime.start(), false) + assert.equal(fixtureState.runtime.snapshot().state, 'unavailable') + assert.equal(fixtureState.errors.length, 1) + assert.equal(fixtureState.errors[0].message, '当前 Realtime 模型不支持画面观察') +}) + +test('stop clears raw frame memory and publishes an idle state', async () => { + const fixtureState = fixture() + await fixtureState.runtime.start() + fixtureState.runtime.frame({ image: JPEG, sequence: 1 }) + fixtureState.runtime.stop('page_hidden') + assert.deepEqual(fixtureState.runtime.snapshot(), { + state: 'idle', + frames: 0, + lastFrameAt: 0, + }) + assert.deepEqual(fixtureState.gatewayEvents.at(-1), { + type: 'observation.state', + state: 'idle', + frames: 0, + reason: 'page_hidden', + }) +}) diff --git a/server/test/realtime-provider.test.mjs b/server/test/realtime-provider.test.mjs index 9fa31f18..1e9e83f9 100644 --- a/server/test/realtime-provider.test.mjs +++ b/server/test/realtime-provider.test.mjs @@ -501,7 +501,7 @@ test('prefers a per-session output voice over the process-wide default', t => { assert.equal(session.voice, 'longanlufeng') }) -test('advertises Omni model vision without admitting unsupported visual transport', t => { +test('advertises Omni model vision and supported observation transport', t => { const originalModel = config.audioModel t.after(() => { config.audioModel = originalModel @@ -514,7 +514,7 @@ test('advertises Omni model vision without admitting unsupported visual transpor assert.equal(profile.modelCapabilities.imageInput, true) assert.equal(profile.modelCapabilities.videoInput, false) assert.equal(profile.transportCapabilities.imageInput, false) - assert.equal(profile.transportCapabilities.observationInput, false) + assert.equal(profile.transportCapabilities.observationInput, true) assert.equal(profile.transportCapabilities.nativeVideoInput, false) assert.equal(frontend.modelProfile, profile) assert.equal(frontend.modelCapabilities, profile.modelCapabilities) diff --git a/shared/client-input-capabilities.mjs b/shared/client-input-capabilities.mjs index b6bfa1f0..2c0980ae 100644 --- a/shared/client-input-capabilities.mjs +++ b/shared/client-input-capabilities.mjs @@ -1,10 +1,22 @@ const PROFILES = Object.freeze({ - web: Object.freeze({ text: true, audio: true, image: true, resource: true }), + web: Object.freeze({ + text: true, + audio: true, + image: true, + observation: true, + resource: true, + }), cli: Object.freeze({ text: true, audio: true, image: true, resource: true }), // The desktop orb and conversation panel are two presentations of the same // client connection. Advertise the panel's inputs for the whole connection // so expanding the window never requires a Realtime reconnect. - desktop: Object.freeze({ text: true, audio: true, image: true, resource: true }), + desktop: Object.freeze({ + text: true, + audio: true, + image: true, + observation: true, + resource: true, + }), }) export function clientInputCapabilities(clientType = 'web') { diff --git a/shared/input-parts.mjs b/shared/input-parts.mjs index 48ba98c5..7cea13e5 100644 --- a/shared/input-parts.mjs +++ b/shared/input-parts.mjs @@ -37,7 +37,7 @@ export function parseDataUrl(value) { function normalizeSource(source) { if (!source || typeof source !== 'object') return undefined - const type = ['clipboard', 'file', 'resource'].includes(source.type) + const type = ['clipboard', 'file', 'resource', 'camera'].includes(source.type) ? source.type : 'file' const text = source.text && typeof source.text === 'object' diff --git a/shared/protocol/gateway-events.mjs b/shared/protocol/gateway-events.mjs index a1da726e..91f61167 100644 --- a/shared/protocol/gateway-events.mjs +++ b/shared/protocol/gateway-events.mjs @@ -174,6 +174,14 @@ const GatewayInputMessagePayloadSchema = z.object({ message: 'a text or parts payload is required', }) +const GatewayObservationFramePayloadSchema = z.object({ + // The browser sends the Base64 body rather than a data URL. Keeping the + // protocol bound close to the provider's 256 KiB encoded-image limit avoids + // accepting a large frame that would only be rejected downstream. + image: z.string().min(1).max(256 * 1024), + sequence: z.number().int().nonnegative().optional(), +}).passthrough() + const GatewayClientPayloadSchemas = Object.freeze({ [GatewayClientEvent.CONNECT]: z.object({ voiceEnabled: z.boolean().optional(), @@ -192,6 +200,7 @@ const GatewayClientPayloadSchemas = Object.freeze({ text: z.boolean().optional(), audio: z.boolean().optional(), image: z.boolean().optional(), + observation: z.boolean().optional(), resource: z.boolean().optional(), }).passthrough().optional(), clientStates: z.array(z.string().min(1)).optional(), @@ -201,6 +210,11 @@ const GatewayClientPayloadSchemas = Object.freeze({ }).passthrough(), [GatewayClientEvent.TEXT_MESSAGE]: GatewayInputMessagePayloadSchema, [GatewayClientEvent.INPUT_MESSAGE]: GatewayInputMessagePayloadSchema, + [GatewayClientEvent.OBSERVATION_START]: z.object({}).passthrough(), + [GatewayClientEvent.OBSERVATION_FRAME]: GatewayObservationFramePayloadSchema, + [GatewayClientEvent.OBSERVATION_STOP]: z.object({ + reason: z.string().max(80).optional(), + }).passthrough(), [GatewayClientEvent.PLAYBACK_STARTED]: z.object({ responseId: z.string().min(1), }).passthrough(), @@ -269,6 +283,11 @@ const GatewayVoicePayloadSchemas = Object.freeze({ [GatewayServerEvent.TRANSCRIPT_DISCARD]: z.object({ role: z.enum(['user', 'assistant']), }).passthrough(), + [GatewayServerEvent.OBSERVATION_STATE]: z.object({ + state: z.enum(['idle', 'starting', 'active', 'unavailable']), + frames: z.number().int().nonnegative().optional(), + reason: z.string().max(80).optional(), + }).passthrough(), [GatewayServerEvent.AGENT_ACTIVITY]: z.object({ activity: z.string().min(1), }).passthrough(), diff --git a/shared/realtime-events.mjs b/shared/realtime-events.mjs index b618ddb8..06ef92f9 100644 --- a/shared/realtime-events.mjs +++ b/shared/realtime-events.mjs @@ -7,6 +7,9 @@ export const GatewayClientEvent = Object.freeze({ AUDIO_APPEND: 'audio.append', TEXT_MESSAGE: 'text.message', INPUT_MESSAGE: 'input.message', + OBSERVATION_START: 'observation.start', + OBSERVATION_FRAME: 'observation.frame', + OBSERVATION_STOP: 'observation.stop', INTERRUPT: 'interrupt', SLEEP: 'sleep', WAKE: 'wake', @@ -43,6 +46,7 @@ export const GatewayServerEvent = Object.freeze({ TRANSCRIPT_DELTA: 'transcript.delta', TRANSCRIPT_FINAL: 'transcript.final', TRANSCRIPT_DISCARD: 'transcript.discard', + OBSERVATION_STATE: 'observation.state', AGENT_ACTIVITY: 'agent.activity', CLIENT_STATE: 'client.state', ERROR: 'error', diff --git a/shared/realtime-model-catalog.mjs b/shared/realtime-model-catalog.mjs index 275a919b..5c74692a 100644 --- a/shared/realtime-model-catalog.mjs +++ b/shared/realtime-model-catalog.mjs @@ -11,7 +11,7 @@ const OMNI_MODEL_CAPABILITIES = Object.freeze({ }) const OMNI_TRANSPORT_CAPABILITIES = Object.freeze({ textInput: true, audioInput: true, imageInput: false, - observationInput: false, nativeVideoInput: false, + observationInput: true, nativeVideoInput: false, }) const LEGACY_MODEL_CAPABILITIES = Object.freeze({ textInput: true, audioInput: true, imageInput: false, videoInput: false, diff --git a/test/gateway-client-conformance.test.mjs b/test/gateway-client-conformance.test.mjs index 909de913..42fd5b8e 100644 --- a/test/gateway-client-conformance.test.mjs +++ b/test/gateway-client-conformance.test.mjs @@ -86,4 +86,3 @@ for (const clientType of Object.values(GatewayReferenceClientType)) { client.stop() }) } - diff --git a/test/realtime-provider-catalog.test.mjs b/test/realtime-provider-catalog.test.mjs index cbc327ed..c207c11a 100644 --- a/test/realtime-provider-catalog.test.mjs +++ b/test/realtime-provider-catalog.test.mjs @@ -27,7 +27,7 @@ const omniTransportCapabilities = { textInput: true, audioInput: true, imageInput: false, - observationInput: false, + observationInput: true, nativeVideoInput: false, } diff --git a/web/src/App.jsx b/web/src/App.jsx index 21dfbe2a..0600be8d 100644 --- a/web/src/App.jsx +++ b/web/src/App.jsx @@ -1027,6 +1027,10 @@ export default function App() { nativeVideo: t('原生视频'), } const modeList = modes => modes.map(mode => inputModeLabels[mode]).join(' / ') + const observationAvailable = ( + modelStatus.metadataStatus === 'current' + && modelStatus.observationInputEnabled === true + ) const resetSession = () => { taskDismissTimers.current.forEach(timer => clearTimeout(timer)) @@ -1487,6 +1491,12 @@ export default function App() { {composerEnabled && } diff --git a/web/src/MultimodalComposer.jsx b/web/src/MultimodalComposer.jsx index bea7e56e..4fce9c4b 100644 --- a/web/src/MultimodalComposer.jsx +++ b/web/src/MultimodalComposer.jsx @@ -1,4 +1,4 @@ -import { useCallback, useRef, useState } from 'react' +import { useCallback, useEffect, useRef, useState } from 'react' import { MAX_INPUT_FILE_BYTES, createInputFilePart, @@ -6,6 +6,16 @@ import { withAttachmentAnchors, } from '../../shared/input-parts.mjs' import { t } from './i18n.js' +import { + CAMERA_IMAGE_TOO_LARGE, + OBSERVATION_INTERVAL_MS, + OBSERVATION_MAX_BYTES, + OBSERVATION_MAX_FRAMES, + appendRecentObservationFrame, + blobToBase64, + captureCameraFrame, + stopCameraStream, +} from './camera-input.js' function filePart(file, index, sourceType = 'file') { return new Promise((resolve, reject) => { @@ -32,16 +42,201 @@ function filePart(file, index, sourceType = 'file') { export default function MultimodalComposer({ onSend, + onObservationStart, + onObservationFrame, + onObservationStop, + observationAvailable = false, + observationState = 'idle', + connectionState = 'connected', compact = false, }) { const [text, setText] = useState('') const [attachments, setAttachments] = useState([]) const [error, setError] = useState('') + const [cameraOpen, setCameraOpen] = useState(false) + const [cameraReady, setCameraReady] = useState(false) + const [observationRequested, setObservationRequested] = useState(false) + const [observationFrameCount, setObservationFrameCount] = useState(0) const picker = useRef(null) + const cameraVideo = useRef(null) + const cameraStream = useRef(null) + const observationRequestedRef = useRef(false) + const observationFramesRef = useRef([]) + const observationSequenceRef = useRef(0) + const previousObservationStateRef = useRef(observationState) const updateAttachments = useCallback(next => { setAttachments(next) }, []) + const stopObservation = useCallback((reason = 'user') => { + if (observationRequestedRef.current) onObservationStop?.(reason) + observationRequestedRef.current = false + observationFramesRef.current = [] + setObservationRequested(false) + setObservationFrameCount(0) + }, [onObservationStop]) + + const closeCamera = useCallback((reason = 'user') => { + stopObservation(reason) + stopCameraStream(cameraStream.current) + cameraStream.current = null + if (cameraVideo.current) { + cameraVideo.current.pause?.() + cameraVideo.current.srcObject = null + } + setCameraReady(false) + setCameraOpen(false) + }, [stopObservation]) + + useEffect(() => { + if (!cameraOpen || !cameraStream.current || !cameraVideo.current) return undefined + const video = cameraVideo.current + const stream = cameraStream.current + video.srcObject = stream + void video.play().catch(() => {}) + return () => { + if (video.srcObject === stream) video.srcObject = null + } + }, [cameraOpen]) + + useEffect(() => { + if (!cameraOpen || !cameraStream.current) return undefined + const stream = cameraStream.current + const onTrackEnded = () => closeCamera('camera_disconnected') + const tracks = stream.getTracks?.() || [] + tracks.forEach(track => track.addEventListener?.('ended', onTrackEnded)) + return () => tracks.forEach(track => ( + track.removeEventListener?.('ended', onTrackEnded) + )) + }, [cameraOpen, closeCamera]) + + useEffect(() => { + const onVisibilityChange = () => { + if (document.hidden) closeCamera('page_hidden') + } + document.addEventListener('visibilitychange', onVisibilityChange) + return () => document.removeEventListener('visibilitychange', onVisibilityChange) + }, [closeCamera]) + + useEffect(() => () => { + if (observationRequestedRef.current) onObservationStop?.('unmount') + observationRequestedRef.current = false + observationFramesRef.current = [] + stopCameraStream(cameraStream.current) + }, [onObservationStop]) + + const startObservation = useCallback(() => { + if (observationRequestedRef.current) return + if (!observationAvailable) { + setError(t('当前模型不支持画面观察')) + return + } + if (!cameraReady || !cameraVideo.current) return + if (onObservationStart && onObservationStart() === false) { + setError(t('画面观察连接不可用')) + return + } + observationRequestedRef.current = true + setObservationRequested(true) + setError('') + }, [cameraReady, observationAvailable, onObservationStart]) + + useEffect(() => { + const previous = previousObservationStateRef.current + previousObservationStateRef.current = observationState + if (!observationRequestedRef.current) return + if (!observationAvailable) { + setError(t('当前模型不支持画面观察')) + closeCamera('model_changed') + return + } + if (['unavailable', 'hidden'].includes(connectionState)) { + setError(t('画面观察连接不可用')) + closeCamera('gateway_disconnected') + return + } + if (observationState === 'unavailable') { + setError(t('画面观察连接不可用')) + closeCamera('provider_unavailable') + return + } + if ( + observationState === 'idle' + && ['starting', 'active', 'unavailable'].includes(previous) + ) { + closeCamera('observation_stopped') + } + }, [closeCamera, connectionState, observationAvailable, observationState]) + + useEffect(() => { + if ( + !observationRequested + || observationState !== 'active' + || !cameraReady + ) return undefined + let disposed = false + let capturing = false + const captureAndSend = async () => { + const video = cameraVideo.current + if (disposed || capturing || !video) return + capturing = true + try { + const blob = await captureCameraFrame(video, { + maxBytes: OBSERVATION_MAX_BYTES, + }) + const image = await blobToBase64(blob) + if (disposed || !observationRequestedRef.current) return + const sequence = observationSequenceRef.current++ + const recent = appendRecentObservationFrame( + observationFramesRef.current, + { image, sequence }, + OBSERVATION_MAX_FRAMES, + ) + observationFramesRef.current = recent + setObservationFrameCount(recent.length) + onObservationFrame?.(image, sequence) + } catch (reason) { + if (disposed) return + setError(reason?.message === CAMERA_IMAGE_TOO_LARGE + ? t('画面观察图片超过 256 KiB 限制') + : t('画面观察捕获失败')) + closeCamera('capture_error') + } finally { + capturing = false + } + } + void captureAndSend() + const timer = setInterval(captureAndSend, OBSERVATION_INTERVAL_MS) + return () => { + disposed = true + clearInterval(timer) + } + }, [cameraReady, closeCamera, observationRequested, observationState, onObservationFrame]) + + const openCamera = useCallback(async () => { + if (cameraStream.current) return + if (!navigator.mediaDevices?.getUserMedia) { + setError(t('相机不可用')) + return + } + try { + const stream = await navigator.mediaDevices.getUserMedia({ + audio: false, + video: { + facingMode: { ideal: 'environment' }, + width: { ideal: 1280 }, + height: { ideal: 720 }, + }, + }) + cameraStream.current = stream + setCameraReady(false) + setCameraOpen(true) + setError('') + } catch { + setError(t('无法打开相机')) + } + }, []) + const addFiles = useCallback(async (fileList, sourceType = 'file') => { const files = [...fileList] if (!files.length) return @@ -51,11 +246,28 @@ export default function MultimodalComposer({ ))) updateAttachments([...attachments, ...next]) setError('') + return true } catch (reason) { setError(reason?.message || String(reason)) + return false } }, [attachments, updateAttachments]) + const capturePhoto = useCallback(async () => { + if (!cameraReady || !cameraVideo.current) return + try { + const blob = await captureCameraFrame(cameraVideo.current) + const file = new File([blob], 'photo.jpg', { type: 'image/jpeg' }) + if (await addFiles([file], 'camera') && !observationRequestedRef.current) { + closeCamera('photo_captured') + } + } catch (reason) { + setError(reason?.message === CAMERA_IMAGE_TOO_LARGE + ? t('照片超过 256 KiB 限制') + : t('无法拍摄照片')) + } + }, [addFiles, cameraReady, closeCamera]) + const submit = event => { event.preventDefault() const content = text.trim() @@ -100,6 +312,18 @@ export default function MultimodalComposer({ aria-label={t('添加图片或文件')} onClick={() => picker.current?.click()} >＋ + + {cameraOpen &&
+
} {error && {error}} } diff --git a/web/src/camera-input.js b/web/src/camera-input.js new file mode 100644 index 00000000..145117f3 --- /dev/null +++ b/web/src/camera-input.js @@ -0,0 +1,125 @@ +export const CAMERA_MAX_WIDTH = 1280 +export const CAMERA_MAX_HEIGHT = 720 +export const CAMERA_MAX_BYTES = 256 * 1024 +// DashScope measures the encoded Base64 payload, so keep the raw JPEG below +// the provider's recommended ~190 KiB margin before converting it to Base64. +export const OBSERVATION_MAX_BYTES = 190 * 1024 +export const OBSERVATION_MAX_BASE64_BYTES = 256 * 1024 +export const OBSERVATION_MAX_FRAMES = 8 +export const OBSERVATION_INTERVAL_MS = 1000 +export const CAMERA_IMAGE_TOO_LARGE = 'camera_image_too_large' + +export const CAMERA_JPEG_QUALITIES = Object.freeze([ + 0.86, + 0.74, + 0.62, + 0.5, + 0.38, + 0.25, + 0.18, + 0.12, + 0.08, +]) + +export function cameraFrameSize( + width, + height, + { maxWidth = CAMERA_MAX_WIDTH, maxHeight = CAMERA_MAX_HEIGHT } = {}, +) { + const sourceWidth = Number(width) + const sourceHeight = Number(height) + if (!Number.isFinite(sourceWidth) || !Number.isFinite(sourceHeight) + || sourceWidth <= 0 || sourceHeight <= 0) { + throw new Error('camera_dimensions_unavailable') + } + const scale = Math.min(1, maxWidth / sourceWidth, maxHeight / sourceHeight) + return { + width: Math.max(1, Math.round(sourceWidth * scale)), + height: Math.max(1, Math.round(sourceHeight * scale)), + } +} + +export function stopCameraStream(stream) { + stream?.getTracks?.().forEach(track => track.stop()) +} + +export function blobToBase64(blob) { + if (!blob || typeof FileReader === 'undefined') { + return Promise.reject(new Error('camera_encoder_unavailable')) + } + return new Promise((resolve, reject) => { + const reader = new FileReader() + reader.onerror = () => reject(reader.error || new Error('camera_encoder_unavailable')) + reader.onload = () => { + const dataUrl = String(reader.result || '') + const separator = dataUrl.indexOf(',') + if (separator < 0) { + reject(new Error('camera_encoder_unavailable')) + return + } + const base64 = dataUrl.slice(separator + 1) + if (base64.length > OBSERVATION_MAX_BASE64_BYTES) { + reject(new Error(CAMERA_IMAGE_TOO_LARGE)) + return + } + resolve(base64) + } + reader.readAsDataURL(blob) + }) +} + +export function appendRecentObservationFrame( + frames, + frame, + maxFrames = OBSERVATION_MAX_FRAMES, +) { + const next = [...(Array.isArray(frames) ? frames : []), frame] + return next.slice(-Math.max(1, Math.floor(Number(maxFrames) || OBSERVATION_MAX_FRAMES))) +} + +export function encodeCameraCanvas( + canvas, + { + maxBytes = CAMERA_MAX_BYTES, + qualities = CAMERA_JPEG_QUALITIES, + } = {}, +) { + if (!canvas || typeof canvas.toBlob !== 'function') { + return Promise.reject(new Error('camera_encoder_unavailable')) + } + const candidates = [...qualities].filter(value => Number.isFinite(value)) + if (!candidates.length) { + return Promise.reject(new Error('camera_encoder_unavailable')) + } + return new Promise((resolve, reject) => { + let index = 0 + const encode = () => { + canvas.toBlob(blob => { + if (blob && blob.size <= maxBytes) { + resolve(blob) + return + } + if (index + 1 >= candidates.length) { + reject(new Error( + blob ? CAMERA_IMAGE_TOO_LARGE : 'camera_encoder_unavailable', + )) + return + } + index += 1 + encode() + }, 'image/jpeg', candidates[index]) + } + encode() + }) +} + +export async function captureCameraFrame(video, options = {}) { + const dimensions = cameraFrameSize(video?.videoWidth, video?.videoHeight, options) + const canvas = document.createElement('canvas') + canvas.width = dimensions.width + canvas.height = dimensions.height + const context = canvas.getContext('2d') + if (!context) throw new Error('camera_encoder_unavailable') + context.drawImage(video, 0, 0, dimensions.width, dimensions.height) + return encodeCameraCanvas(canvas, options) +} diff --git a/web/src/i18n.js b/web/src/i18n.js index b671b0b0..5eb858c0 100644 --- a/web/src/i18n.js +++ b/web/src/i18n.js @@ -103,6 +103,23 @@ const translations = { '输入文字，或粘贴、拖入图片和文件': 'Type a message, or paste and drop images and files', '输入文字或图片': 'Type text or add an image', '添加图片或文件': 'Add images or files', + '拍照': 'Take photo', + '相机预览': 'Camera preview', + '相机不可用': 'Camera is not available', + '无法打开相机': 'Unable to access the camera', + '无法拍摄照片': 'Unable to capture the photo', + '照片超过 256 KiB 限制': 'The photo exceeds the 256 KiB limit', + '拍摄': 'Capture', + '取消': 'Cancel', + '连续观察': 'Continuous observation', + '停止观察': 'Stop observation', + '关闭相机': 'Close camera', + '连续观察中：最近 {count}/8 帧': 'Observing continuously: latest {count}/8 frames', + '正在启动画面观察': 'Starting visual observation', + '当前模型不支持画面观察': 'The current model does not support visual observation', + '画面观察连接不可用': 'Visual observation is unavailable', + '画面观察图片超过 256 KiB 限制': 'The observation image exceeds the 256 KiB limit', + '画面观察捕获失败': 'Could not capture an observation frame', '移除附件': 'Remove attachment', '发送': 'Send', 'Gateway 尚未连接': 'Gateway is not connected yet', diff --git a/web/src/styles.css b/web/src/styles.css index 7a8764ba..92fd7260 100644 --- a/web/src/styles.css +++ b/web/src/styles.css @@ -36,6 +36,7 @@ } .composer-attach, +.composer-camera, .composer-send { min-width: 42px; height: 42px; @@ -46,6 +47,76 @@ cursor: pointer; } +.composer-camera svg { + width: 19px; + height: 19px; + fill: none; + stroke: currentColor; + stroke-linecap: round; + stroke-linejoin: round; + stroke-width: 1.7; + vertical-align: middle; +} + +.camera-capture { + display: grid; + gap: 10px; + margin-top: 10px; + padding: 10px; + border-radius: 14px; + background: rgba(0, 0, 0, 0.28); +} + +.camera-capture video { + width: 100%; + max-height: 320px; + border-radius: 10px; + background: #000; + object-fit: contain; +} + +.camera-actions { + display: flex; + align-items: center; + flex-wrap: wrap; + justify-content: flex-end; + gap: 8px; +} + +.camera-observation-status { + color: #b8b0ff; + font-size: 12px; +} + +.camera-observation { + min-height: 42px; + padding: 0 14px; + border: 1px solid rgba(154, 139, 255, 0.45); + border-radius: 13px; + color: #e5e0ff; + background: rgba(109, 93, 252, 0.16); +} + +.camera-observation.active { + border-color: rgba(126, 229, 197, 0.45); + color: #bff6e3; + background: rgba(48, 148, 117, 0.2); +} + +.camera-actions .ghost { + min-height: 42px; + padding: 0 14px; + border: 1px solid rgba(255, 255, 255, 0.14); + border-radius: 13px; + color: inherit; + background: rgba(255, 255, 255, 0.06); +} + +.camera-actions button:disabled { + cursor: not-allowed; + opacity: 0.5; +} + .composer-send { padding: 0 16px; background: #6d5dfc; @@ -227,6 +298,7 @@ html[data-desktop="orb"] #root { } .desktop-conversation-panel .composer-attach, +.desktop-conversation-panel .composer-camera, .desktop-conversation-panel .composer-send { min-width: 38px; height: 38px; diff --git a/web/src/useRealtimeVoice.js b/web/src/useRealtimeVoice.js index 74bdbdc2..8d008780 100644 --- a/web/src/useRealtimeVoice.js +++ b/web/src/useRealtimeVoice.js @@ -147,6 +147,7 @@ export function realtimeModelStatus(health = {}) { TRANSPORT_INPUT_CAPABILITIES, ), imageInputEnabled: transportCapabilities?.imageInput === true, + observationInputEnabled: transportCapabilities?.observationInput === true, } } @@ -230,6 +231,7 @@ export default function useRealtimeVoice({ createGatewayClientState, ) const [inputReady, setInputReady] = useState(false) + const [observationState, setObservationState] = useState('idle') const [error, setError] = useState('') const [visualError, setVisualError] = useState(false) const { @@ -574,6 +576,7 @@ export default function useRealtimeVoice({ state: 'hidden', }) setInputReady(false) + setObservationState('idle') setError('') setVisualError(false) }, [suspended]) @@ -600,6 +603,9 @@ export default function useRealtimeVoice({ setVisualError(true) } } + if (event.type === GatewayServerEvent.OBSERVATION_STATE) { + setObservationState(event.state || 'idle') + } if (event.type === GatewayServerEvent.TURN_STARTED) { currentTurnId.current = event.turnId || '' if ( @@ -698,6 +704,7 @@ export default function useRealtimeVoice({ } else if (status.state === 'disconnected') { releaseManualInputGuard() stopPlayback() + setObservationState('idle') const disconnectedEvent = { type: GatewayServerEvent.GATEWAY_DISCONNECTED, } @@ -718,6 +725,7 @@ export default function useRealtimeVoice({ return () => { stopPlayback('connection_closed') + setObservationState('idle') client.stop() socketRef.current = null mutedResponses.clear() @@ -745,6 +753,7 @@ export default function useRealtimeVoice({ useEffect(() => { pendingManualInputsRef.current = [] + setObservationState('idle') }, [sessionId]) useEffect(() => { @@ -976,6 +985,27 @@ export default function useRealtimeVoice({ return false }, [holdManualInputGuard, releaseManualInputGuard, sendSocketEvent]) + const sendObservationStart = useCallback(() => ( + sendSocketEvent({ type: GatewayClientEvent.OBSERVATION_START }) + ), [sendSocketEvent]) + + const sendObservationFrame = useCallback((image, sequence) => { + const value = String(image || '').trim() + if (!value) return false + return sendSocketEvent({ + type: GatewayClientEvent.OBSERVATION_FRAME, + image: value, + ...(Number.isInteger(sequence) ? { sequence } : {}), + }) + }, [sendSocketEvent]) + + const sendObservationStop = useCallback((reason = 'user') => ( + sendSocketEvent({ + type: GatewayClientEvent.OBSERVATION_STOP, + reason: String(reason || 'user').slice(0, 80), + }) + ), [sendSocketEvent]) + return { state, visualState: visualVoiceState(state), @@ -983,6 +1013,7 @@ export default function useRealtimeVoice({ error, visualError, connectionState, + observationState, wakeWordActive, ownership, activateAudio, @@ -995,5 +1026,8 @@ export default function useRealtimeVoice({ cancelTask, respondPermission, conversationHistory, + sendObservationStart, + sendObservationFrame, + sendObservationStop, } } diff --git a/web/test/camera-input.test.mjs b/web/test/camera-input.test.mjs new file mode 100644 index 00000000..1720d380 --- /dev/null +++ b/web/test/camera-input.test.mjs @@ -0,0 +1,142 @@ +import assert from 'node:assert/strict' +import test from 'node:test' +import { + CAMERA_IMAGE_TOO_LARGE, + OBSERVATION_MAX_BASE64_BYTES, + appendRecentObservationFrame, + blobToBase64, + cameraFrameSize, + captureCameraFrame, + encodeCameraCanvas, + stopCameraStream, +} from '../src/camera-input.js' + +test('fits camera frames inside the 720p capture bounds', () => { + assert.deepEqual(cameraFrameSize(1920, 1080), { width: 1280, height: 720 }) + assert.deepEqual(cameraFrameSize(1080, 1920), { width: 405, height: 720 }) + assert.deepEqual(cameraFrameSize(640, 480), { width: 640, height: 480 }) +}) + +test('rejects camera frames without usable dimensions', () => { + assert.throws(() => cameraFrameSize(0, 720), /camera_dimensions_unavailable/) + assert.throws(() => cameraFrameSize(Number.NaN, 720), /camera_dimensions_unavailable/) +}) + +test('lowers JPEG quality until the camera image fits the byte limit', async () => { + const qualities = [] + const canvas = { + toBlob(callback, mime, quality) { + qualities.push([mime, quality]) + callback({ size: Math.round(400_000 * quality) }) + }, + } + const blob = await encodeCameraCanvas(canvas, { + maxBytes: 200_000, + qualities: [0.8, 0.4], + }) + assert.equal(blob.size, 160_000) + assert.deepEqual(qualities, [ + ['image/jpeg', 0.8], + ['image/jpeg', 0.4], + ]) +}) + +test('rejects an image that cannot fit the camera byte limit', async () => { + const canvas = { + toBlob(callback) { + callback({ size: 500_000 }) + }, + } + await assert.rejects( + encodeCameraCanvas(canvas, { maxBytes: 200_000, qualities: [0.8, 0.4] }), + error => error.message === CAMERA_IMAGE_TOO_LARGE, + ) +}) + +test('draws the current video frame at the bounded size before encoding', async () => { + const previousDocument = globalThis.document + let canvas + let drawn + globalThis.document = { + createElement(type) { + assert.equal(type, 'canvas') + canvas = { + getContext: () => ({ + drawImage: (...args) => { drawn = args }, + }), + toBlob: callback => callback({ size: 100 }), + } + return canvas + }, + } + try { + await captureCameraFrame( + { videoWidth: 1920, videoHeight: 1080 }, + { maxBytes: 200, qualities: [0.8] }, + ) + assert.equal(canvas.width, 1280) + assert.equal(canvas.height, 720) + assert.deepEqual(drawn, [ + { videoWidth: 1920, videoHeight: 1080 }, + 0, + 0, + 1280, + 720, + ]) + } finally { + globalThis.document = previousDocument + } +}) + +test('stops every track when closing a camera stream', () => { + const stopped = [] + stopCameraStream({ + getTracks: () => [ + { stop: () => stopped.push('video') }, + { stop: () => stopped.push('audio') }, + ], + }) + assert.deepEqual(stopped, ['video', 'audio']) +}) + +test('keeps only the latest eight observation frames', () => { + const frames = Array.from({ length: 8 }, (_, sequence) => ({ sequence })) + const next = appendRecentObservationFrame(frames, { sequence: 8 }) + assert.equal(next.length, 8) + assert.deepEqual(next.map(frame => frame.sequence), [1, 2, 3, 4, 5, 6, 7, 8]) +}) + +test('converts a captured JPEG blob to its Base64 body', async () => { + const previousReader = globalThis.FileReader + class FakeFileReader { + readAsDataURL(blob) { + assert.equal(blob, 'blob') + this.result = 'data:image/jpeg;base64,/9j/fake' + this.onload?.() + } + } + globalThis.FileReader = FakeFileReader + try { + assert.equal(await blobToBase64('blob'), '/9j/fake') + } finally { + globalThis.FileReader = previousReader + } +}) + +test('rejects a Base64 observation body above the provider limit', async () => { + const previousReader = globalThis.FileReader + class FakeFileReader { + readAsDataURL() { + this.result = `data:image/jpeg;base64,${'a'.repeat(OBSERVATION_MAX_BASE64_BYTES + 1)}` + this.onload?.() + } + } + globalThis.FileReader = FakeFileReader + try { + await assert.rejects(blobToBase64('blob'), error => ( + error.message === CAMERA_IMAGE_TOO_LARGE + )) + } finally { + globalThis.FileReader = previousReader + } +}) diff --git a/web/test/client-input-capabilities.test.mjs b/web/test/client-input-capabilities.test.mjs index ec3e3b76..d0c32770 100644 --- a/web/test/client-input-capabilities.test.mjs +++ b/web/test/client-input-capabilities.test.mjs @@ -5,11 +5,12 @@ import { supportsComposerInput, } from '../../shared/client-input-capabilities.mjs' -test('WebUI advertises text, audio, image, and resource input', () => { +test('WebUI advertises text, audio, image, observation, and resource input', () => { assert.deepEqual(clientInputCapabilities('web'), { text: true, audio: true, image: true, + observation: true, resource: true, }) assert.equal(supportsComposerInput('web'), true) diff --git a/web/test/health-capability.test.mjs b/web/test/health-capability.test.mjs index fc467ab6..494c4626 100644 --- a/web/test/health-capability.test.mjs +++ b/web/test/health-capability.test.mjs @@ -25,6 +25,7 @@ function profile(id, label, { imageInput = false, videoInput = false, transportImageInput = false, + transportObservationInput = false, } = {}) { return { id, @@ -40,7 +41,7 @@ function profile(id, label, { textInput: true, audioInput: true, imageInput: transportImageInput, - observationInput: false, + observationInput: transportObservationInput, nativeVideoInput: false, }, } @@ -68,6 +69,7 @@ for (const activeProfile of [flash, plus]) { assert.deepEqual(status.modelInputModes, ['text', 'audio', 'image']) assert.deepEqual(status.transportInputModes, ['text', 'audio']) assert.equal(status.imageInputEnabled, false) + assert.equal(status.observationInputEnabled, false) }) } @@ -82,6 +84,7 @@ test('shows the legacy model without claiming image support', () => { assert.deepEqual(status.modelInputModes, ['text', 'audio']) assert.deepEqual(status.transportInputModes, ['text', 'audio']) assert.equal(status.imageInputEnabled, false) + assert.equal(status.observationInputEnabled, false) }) test('fails closed when model profile metadata is missing', () => { @@ -96,6 +99,7 @@ test('fails closed when model profile metadata is missing', () => { assert.deepEqual(status.modelInputModes, []) assert.deepEqual(status.transportInputModes, []) assert.equal(status.imageInputEnabled, false) + assert.equal(status.observationInputEnabled, false) }) test('rejects stale profile capabilities when the active model changed', () => { @@ -130,6 +134,22 @@ test('enables image controls only for current exact catalog transport truth', () assert.equal(status.imageInputEnabled, true) assert.deepEqual(status.transportInputModes, ['text', 'audio', 'image']) + assert.equal(status.observationInputEnabled, false) +}) + +test('enables continuous observation only for current exact transport truth', () => { + const observable = profile('observable-model', 'Observable model', { + imageInput: true, + transportObservationInput: true, + }) + const status = realtimeModelStatus({ + realtimeModel: observable.id, + realtimeModelProfile: observable, + realtimeModelCatalog: [observable], + }) + + assert.equal(status.observationInputEnabled, true) + assert.deepEqual(status.transportInputModes, ['text', 'audio', 'observation']) }) test('keeps an advertised provider selection separate from the model', () => {