Skip to content

Commit ed7a295

Browse files
miguel-heygenmiguel-heygen
committed
docs: document Studio's WebMCP agent tools
Adds `guides/webmcp`, under Developers > Agent setup. Its first job is to defuse a name collision. `guides/mcp` already exists and covers HeyGen's HOSTED MCP connector, which builds a video from a chat. This page is about an agent working inside Studio on a composition already open in front of you. Different feature, confusingly similar name, so the page says what it is not before it says what it is. Written to DOCS_GUIDELINES: one-sentence intro, outcome before implementation, real values rather than placeholders, and three callouts. The three things a reader most needs are the ones easiest to get wrong: The API is `document.modelContext`, not `navigator.modelContext`. Most published examples use the second, which is a polyfill compatibility shim rather than a spec member, so feature-detecting it misleads. Select first, then edit. Most editing tools act on the current selection, and an agent that skips it gets an error rather than a wrong-element write. Leave Studio visible. Some of Studio's write paths report failure through a toast rather than a return value, so the human is the one who sees it. That is a real property of the co-pilot design, not a nicety, so the page says it plainly. Verified with `npx mint validate` and `npx mint broken-links --check-redirects`, both passing.
1 parent 721c271 commit ed7a295

2 files changed

Lines changed: 131 additions & 1 deletion

File tree

‎docs/docs.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -872,7 +872,8 @@
872872
"group": "Agent setup",
873873
"pages": [
874874
"guides/authentication",
875-
"guides/skills"
875+
"guides/skills",
876+
"guides/webmcp"
876877
]
877878
},
878879
{

‎docs/guides/webmcp.mdx‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
---
2+
title: "Let an agent drive Studio"
3+
sidebarTitle: "Agent tools (WebMCP)"
4+
description: "Studio exposes its editing capabilities as WebMCP tools, so an agent in your browser can see the composition and change it alongside you."
5+
---
6+
7+
Studio registers its own capabilities as WebMCP tools, so an AI agent running in your browser can read what Studio knows and make the same edits you can.
8+
9+
<Note>
10+
This is not the same as [creating through an AI chat](/guides/mcp). That page covers the hosted
11+
HyperFrames MCP connector, which builds and renders a video from a conversation. This page is
12+
about an agent working *inside Studio*, on a composition already open in front of you.
13+
</Note>
14+
15+
## What it looks like
16+
17+
With the tools available, an agent can do this without touching your files:
18+
19+
```text
20+
studio_look -> the project, playhead, selection, and every element
21+
studio_select hf:abc123 -> selects the headline, same as clicking it
22+
studio_inspect -> its resolved styles, text, and animations
23+
studio_set_style {"color":"red"} -> writes it, through Studio's own commit path
24+
studio_frame 2.4 -> a PNG of the composition at 2.4 seconds
25+
```
26+
27+
The last one matters most. It is what lets an agent judge a change instead of guessing at it.
28+
29+
## Turning it on
30+
31+
The tools register automatically when Studio loads. Whether an agent can *reach* them depends on the browser.
32+
33+
| Browser | Status |
34+
| --- | --- |
35+
| Chrome 149 | Origin Trial |
36+
| Edge 150 | Origin Trial |
37+
| ChatGPT Desktop | Shipped |
38+
| Brave (Leo) | Experimental |
39+
| Firefox, Safari | Not yet |
40+
41+
For local development in Chrome, enable the flag and restart:
42+
43+
```text chrome://flags
44+
chrome://flags/#enable-webmcp-testing
45+
```
46+
47+
Then confirm the tools are there from Studio's console:
48+
49+
```javascript
50+
const tools = await document.modelContext.getTools();
51+
console.log(tools.map((tool) => tool.name));
52+
// ["studio_look", "studio_select", "studio_seek", ...]
53+
```
54+
55+
<Warning>
56+
The API is `document.modelContext`, not `navigator.modelContext`. Many published examples use
57+
the second one. It is a compatibility shim some polyfills add, not part of the specification, so
58+
feature-detecting it will mislead you.
59+
</Warning>
60+
61+
On browsers without native support, Studio loads a polyfill so a WebMCP bridge extension can still
62+
connect. Nothing is downloaded on a browser that has the API already.
63+
64+
## What an agent can do
65+
66+
### Read
67+
68+
| Tool | Answers |
69+
| --- | --- |
70+
| `studio_look` | The open project and composition, the playhead, what you have selected, and every element with a handle |
71+
| `studio_inspect` | One element in full: resolved styles, text fields, box, animations, and what it will accept |
72+
| `studio_frame` | A PNG of the composition at any time |
73+
74+
`studio_look` gives every element a **handle**. Pass it back to any tool that edits an element.
75+
76+
### Change
77+
78+
| Tool | Does |
79+
| --- | --- |
80+
| `studio_select` | Selects an element, exactly as clicking it does |
81+
| `studio_seek` | Moves the playhead |
82+
| `studio_set_text` | Rewrites text |
83+
| `studio_set_style` | Sets inline styles |
84+
| `studio_transform` | Moves, resizes or rotates |
85+
| `studio_add_animation` | Adds a GSAP animation at the playhead |
86+
| `studio_update_animation` | Changes a duration, ease or position |
87+
| `studio_add_keyframe` | Adds a keyframe to an animation |
88+
| `studio_delete_animation` | Removes an animation |
89+
90+
Every edit runs through the same commit path a mouse gesture uses, so it lands in your file with the
91+
same undo entry and the same save behaviour. There is no separate agent write path.
92+
93+
## Two rules worth knowing
94+
95+
**Select first, then edit.** Most editing tools act on the current selection rather than taking an
96+
element. That is how Studio itself works: click, then type. An agent that edits without selecting
97+
gets an error telling it to select.
98+
99+
**Check what came back.** Tools report what actually happened, not what was asked for.
100+
`studio_transform` reads the element's box back after writing and tells you which operations took
101+
effect. `studio_frame` reports the time it actually captured. When something could not be verified,
102+
the tool says so rather than claiming success.
103+
104+
## Working alongside an agent
105+
106+
This is built for you and an agent looking at the same composition. Studio shows you every change as
107+
it happens: an agent selecting an element draws the same selection box, and an edit appears in your
108+
undo history under its own name.
109+
110+
That shared view is doing real work. Some of Studio's write paths report a failure through a toast
111+
rather than a return value, so **you** are the one who sees it. Leave Studio visible while an agent
112+
is working.
113+
114+
<Note>
115+
Studio refuses agent writes while auto-save is paused or an external change to the file is waiting
116+
for your decision, and tells the agent why. Resolve the banner and it can continue.
117+
</Note>
118+
119+
## Turning it off
120+
121+
Set `agentToolsEnabled` to `false` in Studio's preferences and reload. The browser also asks your
122+
permission before any agent actually calls a tool, so registering them is not the same as granting
123+
access.
124+
125+
## Related topics
126+
127+
- [Create through an AI chat](/guides/mcp)
128+
- [Install and update agent skills](/guides/skills)
129+
- [Work on the project in Studio](/studio)

0 commit comments

Comments
 (0)