Skip to content

feat(web): add one-shot camera photo input - #254

Open
yimi528 wants to merge 12 commits into
QwenAudio:mainfrom
yimi528:feat/131-webui-photo-capture-yimi528
Open

feat(web): add one-shot camera photo input#254
yimi528 wants to merge 12 commits into
QwenAudio:mainfrom
yimi528:feat/131-webui-photo-capture-yimi528

Conversation

@yimi528

@yimi528 yimi528 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an explicit Take photo control to the WebUI composer.
  • Add user-controlled continuous camera observation for Omni Realtime models.
  • Capture bounded JPEG frames at approximately 1 FPS and retain at most the latest 8 frames in memory.
  • Forward observation frames through input_image_buffer.append without creating response.create.
  • Stop camera tracks on cancel, stop, page hide, disconnect, mute, sleep, and component unmount.
  • Add Gateway observation events, model capability discovery, translations, styles, documentation, and tests.

Scope

This PR supports:

  • One-shot photo capture as a normal WebUI attachment.
  • Explicit continuous camera observation after the user clicks “Continuous observation”.
  • JPEG capture up to 1280×720, with Base64 payloads capped at 256 KiB.
  • Omni Realtime transport only; legacy Audio models and native video remain unsupported.

Continuous observation does not proactively produce model responses. Frames become part of the Realtime visual context for subsequent user interaction.

Validation

  • npm run lint
  • npm test --workspace web — 106 tests passed
  • Observation runtime tests — 6 tests passed
  • Gateway contract tests — 13 tests passed
  • npm run build --workspace web
  • git diff --check

Privacy and compatibility

  • Camera permission is requested only after the user presses Take photo.
  • Continuous observation requires explicit user opt-in.
  • Raw camera frames and Base64 data are kept only in bounded in-memory buffers.
  • Frames are not persisted and are not forwarded to the backend ACP Agent.
  • Camera capture stops when the page is hidden, the connection is lost, or the user stops observation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant