feat(web): add one-shot camera photo input

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Unreleased
 
+- WebUI 新增用户主动触发的一次性拍照输入：照片会在浏览器本地压缩后作为附件发送，
+  不会自动调用模型或持久化摄像头画面。
+- WebUI 支持用户主动开启摄像头连续观察：约每秒发送一帧 JPEG，内存中最多保留最近
+  8 帧；不会因观察自动触发模型回复，页面隐藏、断线或停止时会释放摄像头。
 - 后台 Session 模型覆盖统一使用 ACP `configOptions` 与
   `session/set_config_option`，不再调用私有模型接口或生成后台配置文件；未声明标准
   模型选项的 Agent 将沿用自身配置。OpenCode/OpenClaw 一键托管初始化保持不变。

README.md

@@ -143,6 +143,15 @@ QWEN_AUDIO_AGENT_BACKEND_MODEL=qwen3.7-max
 ```
 
 > Uses DashScope realtime voice frontend by default; alternatively, switch to a local [speech-to-speech frontend](docs/voice-frontends/speech-to-speech.md), no cloud API Key needed.
+> `qwen3.5-omni-flash-realtime` and `qwen3.5-omni-plus-realtime`
+> accept text, audio, and image at the model level. The WebUI also supports explicit
+> camera observation for these Omni models: it sends one bounded JPEG frame per second,
+> keeps at most eight recent frames in memory, and never creates a response from observation
+> alone. Native video and observation on legacy Audio models remain unavailable.
+
+The Desktop app or `qwenaudio config set --realtime-model <id>` configures the single
+Gateway-wide model. Restart the Gateway after a CLI change. WebUI and TUI display the active
+model but do not override it.
 
 2. Start the Gateway, then open another terminal to start the TUI (or use `qwenaudio webui` for the browser UI):
 

README_ZH.md

@@ -135,7 +135,13 @@ QWEN_AUDIO_AGENT_BACKEND_MODEL=qwen3.7-max
 ```
 
 > 默认使用 DashScope 实时语音前台；也可切换为本地 [speech-to-speech 前台](docs/voice-frontends/speech-to-speech.zh.md)，无需云端 API Key。
+> `qwen3.5-omni-flash-realtime` 与 `qwen3.5-omni-plus-realtime`
+> 在模型层支持文本、音频和图片输入。WebUI 对这两个 Omni 模型还支持用户显式开启的
+> 摄像头连续观察：每秒发送一帧有界 JPEG，内存中最多保留最近 8 帧，观察本身不会创建
+> 模型回复。原生视频和旧版 Audio 模型的画面观察仍不可用。
 
+桌面版或 `qwenaudio config set --realtime-model <id>` 配置 Gateway 统一使用的模型；
+CLI 修改后需要重启 Gateway。WebUI 与 TUI 只显示当前生效模型，不单独覆盖模型。
 
 2. 启动 Gateway，另开终端启动 TUI（也可用 `qwenaudio webui` 启动浏览器界面）：
 

desktop/test/input-capabilities.test.mjs

@@ -10,6 +10,7 @@ test('desktop connection supports the conversation panel inputs', () => {
     text: true,
     audio: true,
     image: true,
+    observation: true,
     resource: true,
   })
   assert.equal(supportsComposerInput('desktop'), true)

docs/configuration/frontend.md

@@ -45,16 +45,17 @@ The exact supported IDs are:
 
 | Model | Model input | Model output | Current client transport |
 | --- | --- | --- | --- |
-| `qwen3.5-omni-flash-realtime` | text, audio, image | text, audio | text, audio |
-| `qwen3.5-omni-plus-realtime` | text, audio, image | text, audio | text, audio |
+| `qwen3.5-omni-flash-realtime` | text, audio, image | text, audio | text, audio, JPEG observation |
+| `qwen3.5-omni-plus-realtime` | text, audio, image | text, audio | text, audio, JPEG observation |
 | `qwen-audio-3.0-realtime-plus` (default) | text, audio | text, audio | text, audio |
 | `qwen-audio-3.0-realtime-flash` | text, audio | text, audio | text, audio |
 
-All four profiles support Function Calling. Model capability is not the same as an implemented
-client transport: JPEG observation frames and native video are both disabled in this release.
-WebUI and TUI read the authoritative profile from Gateway health and only display it. Separate
+All four profiles support Function Calling. The two Omni profiles support the WebUI's explicit
+JPEG observation transport; frames are sent at about 1 FPS, capped at eight recent in-memory
+frames, and do not create model responses by themselves. Native video and observation on the
+legacy Audio profiles remain unavailable. WebUI and TUI read the authoritative profile from
+Gateway health and use it to gate or display the available inputs. Separate
 clients cannot select conflicting models on one Gateway. A Desktop attached to a borrowed
 Gateway, or a later CLI runtime using a conflicting configured model, refuses the mismatch
 instead of silently changing the running service. To roll back, set the legacy ID above and
 restart the Gateway.
-

docs/configuration/frontend.zh.md

@@ -44,14 +44,14 @@ qwenaudio gateway restart
 
 | 模型 | 模型输入 | 模型输出 | 当前客户端传输 |
 | --- | --- | --- | --- |
-| `qwen3.5-omni-flash-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频 |
-| `qwen3.5-omni-plus-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频 |
+| `qwen3.5-omni-flash-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频、JPEG 观察 |
+| `qwen3.5-omni-plus-realtime` | 文本、音频、图片 | 文本、音频 | 文本、音频、JPEG 观察 |
 | `qwen-audio-3.0-realtime-plus`（默认） | 文本、音频 | 文本、音频 | 文本、音频 |
 | `qwen-audio-3.0-realtime-flash` | 文本、音频 | 文本、音频 | 文本、音频 |
 
-四个档案都支持 Function Calling。模型能力不等于客户端已经实现的传输能力：本版本
-仍关闭 JPEG 观察帧和原生视频传输。WebUI 与 TUI 从 Gateway health 读取权威档案并
-只读展示；同一 Gateway 上的不同客户端不能选择互相冲突的模型。桌面版附着到借用的
+四个档案都支持 Function Calling。两个 Omni 档案支持 WebUI 显式开启的 JPEG 观察传输：
+约每秒发送一帧，内存中最多保留最近 8 帧，观察本身不会创建模型回复。原生视频以及
+旧版 Audio 档案的画面观察仍不可用。WebUI 与 TUI 从 Gateway health 读取权威档案，并据此
+限制或展示可用输入；同一 Gateway 上的不同客户端不能选择互相冲突的模型。桌面版附着到借用的
 Gateway 时，或后续 CLI 运行时使用了冲突的已配置模型时，会拒绝不一致，而不会静默
 修改运行中服务。回滚时设置上表的旧版模型 ID 并重启 Gateway。
-

docs/contract.md

@@ -27,7 +27,9 @@ runtime-command plane, Agent Delivery, Client Actions, reference Client SDK,
 and bounded replay all share the same WebSocket.
 This contract index remains authoritative for implemented behavior.
 
-The current health-contract version is `5.6.0`. The additive `5.6` line exposes
+The current health-contract version is `5.7.0`. The additive `5.7` line exposes
+explicit WebUI camera observation for vision-capable Realtime frontends without
+creating model responses. The additive `5.6` line exposes
 a provider-neutral frontend memory control plane for replaceable clients. The
 additive `5.5` line ships
 the shared reference Client SDK, bounded Task-event replay, and reconnect state
@@ -80,6 +82,7 @@ below instead of assuming the old list.
 | `messages.citations` | Final assistant `transcript.final` events may carry normalized citations collected from frontend retrieval in the same turn | `test/gateway-event-schema.test.mjs`, `server/test/realtime-presentation-runtime.test.mjs` |
 | `frontend.memory-control` | `GET/PATCH /api/memory` lets replaceable clients list and exactly edit the same provider-backed USER/MEMORY documents used by Realtime, without depending on a storage implementation | `server/test/gateway-application.test.mjs` |
 | `realtime.conversation-client-v1` | `WS /api/realtime`, published event constants, and message schemas form the replaceable text/audio/multimodal Conversation Client boundary | `test/gateway-event-schema.test.mjs`, `test/custom-conversation-client.test.mjs` |
+| `realtime.camera-observation-v1` | After an explicit WebUI opt-in, sends at most one bounded JPEG frame per second and retains at most eight recent frames for a vision-capable Realtime frontend; it never creates a model response and releases the camera on stop, disconnect, or page hide | `server/test/realtime-observation-runtime.test.mjs`, `web/test/camera-input.test.mjs` |
 | `realtime.gateway-client-protocol-v6-handshake` | The same WebSocket accepts an opt-in 6.0 `session.hello`, returns correlated `session.ready`, negotiates implemented capabilities, and normalizes 6.0 input aliases into the existing business path | `test/gateway-client-protocol.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
 | `realtime.gateway-client-protocol-v6-runtime-commands` | Negotiated 6.0 Clients can publish registered semantic Client Events and use correlated Task, permission, conversation-history, and session output-voice commands over the same WebSocket; existing REST routes call the same command service as compatibility aliases | `test/gateway-client-protocol.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/client-command-runtime.test.mjs`, `server/test/gateway-client-handshake.test.mjs` |
 | `realtime.gateway-client-protocol-v6-agent-delivery` | Client Events, Task results and progress, and permission prompts cross one provider-neutral `AgentDelivery` boundary with `handle`, `context`, `respond`, and `interrupt` modes | `server/test/agent-delivery.test.mjs`, `server/test/client-event-router.test.mjs`, `server/test/realtime-provider.test.mjs`, `server/test/announcement-manager.test.mjs` |

docs/contract.zh.md

@@ -22,7 +22,8 @@
 Delivery、Client Action、参考 Client SDK 与有限回放均落在同一条 WebSocket 上。
 已实现行为仍以本契约索引为准。
 
-当前健康契约版本为 `5.6.0`。新增的 `5.6` 能力为可替换客户端提供
+当前健康契约版本为 `5.7.0`。新增的 `5.7` 能力为支持视觉输入的 Realtime
+前台提供 WebUI 显式开启的摄像头画面观察，不会创建模型回复。新增的 `5.6` 能力为可替换客户端提供
 Provider 无关的前台记忆控制面。`5.5` 能力提供共享参考 Client SDK、有限 Task
 事件回放与断线状态恢复。第一方 WebUI、Desktop 和 TUI 已通过同一套一致性测试，
 Task 控制、权限决策和对话历史不再依赖内部 REST 路由。`5.4` 能力提供有关联关系的 Client Action 与共享
@@ -65,6 +66,7 @@ Task 事件提供与 A2A 对齐的 `submitted`、
 | `messages.citations` | 最终助手 `transcript.final` 可以携带同一轮前台检索产生的规范化 Citation | `test/gateway-event-schema.test.mjs`、`server/test/realtime-presentation-runtime.test.mjs` |
 | `frontend.memory-control` | `GET/PATCH /api/memory` 供可替换客户端列出并精确编辑 Realtime 共用的 Provider 记忆文档，不暴露具体存储实现 | `server/test/gateway-application.test.mjs` |
 | `realtime.conversation-client-v1` | `WS /api/realtime`、公开事件常量与消息 Schema 共同构成可替换的文本/音频/多模态对话客户端边界 | `test/gateway-event-schema.test.mjs`、`test/custom-conversation-client.test.mjs` |
+| `realtime.camera-observation-v1` | WebUI 显式开启后，以约 1 FPS 发送最近最多 8 帧 JPEG 到支持视觉输入的 Realtime 前台；不自动创建模型响应，停止、断线、隐藏页面时释放相机 | `server/test/realtime-observation-runtime.test.mjs`、`web/test/camera-input.test.mjs` |
 | `realtime.gateway-client-protocol-v6-handshake` | 同一 WebSocket 可选择以 6.0 `session.hello` 接入，返回有关联关系的 `session.ready`，协商已实现能力，并把 6.0 输入别名归一化到现有业务路径 | `test/gateway-client-protocol.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
 | `realtime.gateway-client-protocol-v6-runtime-commands` | 协商后的 6.0 Client 可以通过同一 WebSocket 发布已注册的语义 Client Event，并使用有关联结果的 Task、权限、对话历史和会话输出音色命令；现有 REST 路由调用同一命令服务作为兼容别名 | `test/gateway-client-protocol.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/client-command-runtime.test.mjs`、`server/test/gateway-client-handshake.test.mjs` |
 | `realtime.gateway-client-protocol-v6-agent-delivery` | Client Event、Task 结果与低频进展、权限请求统一跨越 Provider 无关 `AgentDelivery` 边界，并支持 `handle`、`context`、`respond`、`interrupt` 四种模式 | `server/test/agent-delivery.test.mjs`、`server/test/client-event-router.test.mjs`、`server/test/realtime-provider.test.mjs`、`server/test/announcement-manager.test.mjs` |

docs/i18n-parity.md

@@ -7,7 +7,8 @@
 
 Updated 2026-09-02:
 
-- Gateway health contract `5.6.0`;
+- Gateway health contract `5.7.0`;
+- WebUI explicit camera observation for vision-capable Realtime frontends;
 - stable Gateway Client Protocol wire version `6.0.0`;
 - GCP1–GCP5 complete;
 - WebUI, Desktop, and TUI on the shared reference Client SDK;

examples/smart-cockpit/gateway/test/composition.test.mjs

@@ -42,7 +42,7 @@ test('composes the public Gateway API with the replaceable A2A Agent', async t =
 
   assert.equal(response.status, 200)
   assert.equal(health.ok, true)
-  assert.equal(health.protocolVersion, '5.6.0')
+  assert.equal(health.protocolVersion, '5.7.0')
   assert.equal(health.frontendProfile.name, 'cockpit-example')
   assert.equal(health.backend.protocol, 'a2a')
   assert.equal(health.backend.status, 'ready')

server/src/core/gateway-protocol.mjs

@@ -10,6 +10,8 @@
 // Every capability listed here is locked by a test (see docs/contract.md);
 // anything not listed is internal and may change in any release.
 //
+// 5.7.0 adds explicit WebUI camera observation for vision-capable Realtime
+// frontends without triggering model responses.
 // 5.6.0 adds a provider-neutral frontend memory control plane for replaceable
 // clients to list and edit the same documents used by Realtime.
 // 5.5.0 adds the GCP5 reference Client SDK, bounded Task-event replay, and
@@ -43,7 +45,7 @@
 // desktop.settings-window, …) are not part of this contract, and a removed
 // capability is a breaking change. Hosts migrating from the fork must branch
 // on the capability list below, never on the version number.
-export const GATEWAY_PROTOCOL_VERSION = '5.6.0'
+export const GATEWAY_PROTOCOL_VERSION = '5.7.0'
 
 export const GATEWAY_CAPABILITIES = Object.freeze([
   // The Gateway statically hosts web/dist at its own origin, so a client may
@@ -102,6 +104,9 @@ export const GATEWAY_CAPABILITIES = Object.freeze([
   // replaceable Conversation Client boundary for audio, text, multimodal
   // input, transcripts, playback receipts, voice state and Task projections.
   'realtime.conversation-client-v1',
+  // The WebUI may explicitly open a camera and stream bounded JPEG snapshots
+  // to a vision-capable Realtime frontend without triggering responses.
+  'realtime.camera-observation-v1',
   // session.hello/session.ready negotiate the stable 6.0 Client protocol;
   // 5.x connect and event names remain compatibility aliases through one
   // normalization layer.

server/src/voice/providers/openai-compatible-protocol.mjs

@@ -28,6 +28,11 @@ export const openAiCompatibleProtocol = Object.freeze({
     audio,
   }),
 
+  imageAppend: image => ({
+    type: 'input_image_buffer.append',
+    image,
+  }),
+
   // Client-assigned id for a conversation item. The beta dialect accepts one
   // opaque namespace for every item type.
   conversationItemId: () => `item_${randomUUID().replaceAll('-', '')}`,