Skip to content

feat(studio): give an agent eyes with studio_frame - #3516

Draft
miguel-heygen wants to merge 1 commit into
feat/studio-webmcp-select-seekfrom
feat/studio-webmcp-frame
Draft

feat(studio): give an agent eyes with studio_frame#3516
miguel-heygen wants to merge 1 commit into
feat/studio-webmcp-select-seekfrom
feat/studio-webmcp-frame

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

studio_frame renders the composition to a PNG at a given time and returns its URL.

Stacked on #3515.

Why

This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust.

No agent can judge motion from source. "What does this look like at 2.4 seconds" is not a question a file answers, and it is the question that matters when the thing you are building is a video.

How

Reuses Studio's existing capture endpoint through buildFrameCaptureUrl rather than inventing a second one. The server renders with Puppeteer, so the frame reflects the file on disk.

Two things the tool refuses to fake:

It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion.

It waits before capturing, 150ms by default. The render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the pre-edit composition. That exact staleness was a real bug in this repo once, and the fix is a watcher, not a synchronous invalidation, so the window still exists. An agent reading a stale frame as "my edit failed" would thrash. The wait is on by default, settleMs makes it tunable, and the tool description names the failure mode rather than leaving it to be rediscovered.

It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render.

Test plan

  • Unit tests added/updated
  • Manual testing performed
  • Documentation updated (if applicable)

12 tests in frameTools.test.ts:

  • URL shape, including the time actually captured.
  • Seeks first when given a time; captures where the playhead landed, not what was asked (asserted with a clamping player).
  • Asserts the wait happens BEFORE the probe, by recording call order. Without that this test would pass even if the settle were a no-op.
  • Settle time is honoured, clamped at 5s, defaulted on nonsense input, and skipped entirely at zero.
  • A renderer 500 becomes a failure with a hint, not a dead URL.
  • No project open fails before touching the renderer.
  • Negative and non-finite times are rejected without seeking.

Full package suite 4555 passing across 410 files. bunx tsc --noEmit clean. bunx fallow audit --fail-on-issues clean.

Not yet exercised against a real renderer; that is part of the end-to-end capture in the final unit.

Renders the composition to a PNG at a given time and returns the URL. This is
what turns the tool set from a remote control into a loop: author a change,
capture the instant it affects, look, adjust. No agent can judge motion from
source, because "what does this look like at 2.4 seconds" is not a question a
file answers.

Reuses Studio's existing capture endpoint via `buildFrameCaptureUrl` rather
than inventing a second one.

Two things this does not fake:

It reports the time the playhead LANDED on, not the time requested. The player
clamps, so those differ at the ends, and attaching the wrong time to a frame is
how an agent draws a confident wrong conclusion about motion.

It waits before capturing, by default 150ms. The frame is rendered from the
file on disk, and the render cache is cleared by a file watcher with a 40ms
write-stability threshold, so a capture that beats the watcher renders the
PRE-edit composition. That exact staleness was a real bug here once. An agent
reading a stale frame as "my edit failed" would thrash, so the wait is on by
default, `settleMs` makes it tunable, and the tool description names the
failure rather than leaving it to be rediscovered.

It probes with HEAD before returning, so a URL that 404s comes back as a
failure with a hint instead of as a link the agent cannot render.
@miguel-heygen
miguel-heygen force-pushed the feat/studio-webmcp-select-seek branch from 68a32e9 to f4e938f Compare August 27, 2026 04:48
@miguel-heygen
miguel-heygen force-pushed the feat/studio-webmcp-frame branch from d7ba77c to 57c9bb0 Compare August 27, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant