feat(studio): give an agent eyes with studio_frame - #3516
Draft
miguel-heygen wants to merge 1 commit into
Draft
Conversation
3 tasks
miguel-heygen
marked this pull request as draft
August 27, 2026 01:33
Renders the composition to a PNG at a given time and returns the URL. This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust. No agent can judge motion from source, because "what does this look like at 2.4 seconds" is not a question a file answers. Reuses Studio's existing capture endpoint via `buildFrameCaptureUrl` rather than inventing a second one. Two things this does not fake: It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion. It waits before capturing, by default 150ms. The frame is rendered from the file on disk, and the render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the PRE-edit composition. That exact staleness was a real bug here once. An agent reading a stale frame as "my edit failed" would thrash, so the wait is on by default, `settleMs` makes it tunable, and the tool description names the failure rather than leaving it to be rediscovered. It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render.
miguel-heygen
force-pushed
the
feat/studio-webmcp-select-seek
branch
from
August 27, 2026 04:48
68a32e9 to
f4e938f
Compare
miguel-heygen
force-pushed
the
feat/studio-webmcp-frame
branch
from
August 27, 2026 04:48
d7ba77c to
57c9bb0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
studio_framerenders the composition to a PNG at a given time and returns its URL.Stacked on #3515.
Why
This is what turns the tool set from a remote control into a loop: author a change, capture the instant it affects, look, adjust.
No agent can judge motion from source. "What does this look like at 2.4 seconds" is not a question a file answers, and it is the question that matters when the thing you are building is a video.
How
Reuses Studio's existing capture endpoint through
buildFrameCaptureUrlrather than inventing a second one. The server renders with Puppeteer, so the frame reflects the file on disk.Two things the tool refuses to fake:
It reports the time the playhead LANDED on, not the time requested. The player clamps, so those differ at the ends, and attaching the wrong time to a frame is how an agent draws a confident wrong conclusion about motion.
It waits before capturing, 150ms by default. The render cache is cleared by a file watcher with a 40ms write-stability threshold, so a capture that beats the watcher renders the pre-edit composition. That exact staleness was a real bug in this repo once, and the fix is a watcher, not a synchronous invalidation, so the window still exists. An agent reading a stale frame as "my edit failed" would thrash. The wait is on by default,
settleMsmakes it tunable, and the tool description names the failure mode rather than leaving it to be rediscovered.It probes with HEAD before returning, so a URL that 404s comes back as a failure with a hint instead of as a link the agent cannot render.
Test plan
12 tests in
frameTools.test.ts:Full package suite 4555 passing across 410 files.
bunx tsc --noEmitclean.bunx fallow audit --fail-on-issuesclean.Not yet exercised against a real renderer; that is part of the end-to-end capture in the final unit.