Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 26 additions & 32 deletions apps/docs/guide/work-with-web-ai/webmcp.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# WebMCP authoring

WebMCP lets an agent use LocalStudio's real editor commands while you watch the same canvas. The editor exposes 15 narrowly scoped tools with JSON schemas, bounded results, and visible state changes. The showcase at `/editor/webmcp/` provides an editable card for every tool.
WebMCP lets an agent use LocalStudio's real editor commands while you watch the same canvas. The editor exposes 14 narrowly scoped tools with JSON schemas, bounded results, and visible state changes. The showcase at `/editor/webmcp/` provides an editable card for every tool.

This is an **authoring** integration. Public presentations can contain slide descriptions, transcripts, and authorized recording audio, but the public viewer does not currently register attendee WebMCP tools.

Expand All @@ -12,35 +12,33 @@ This is an **authoring** integration. Public presentations can contain slide des
- For native agent discovery, open `http://localhost:5184/editor/?webmcp=1` directly. WebMCP is document-scoped, so a browser agent does not inherit tools registered by the showcase's child iframe.
- Translation, description generation, image generation, and model preparation may require browser AI support, WebGPU, model downloads, and several gigabytes of browser storage.
- Unsplash and GIPHY search require their respective keys in **Settings > Media integrations**.
- Publishing requires configured S3-compatible remote storage. The local MinIO settings in the project README are suitable for development.

## The 15 tools
## The 14 tools

All inputs are JSON objects. Unknown fields and invalid types are rejected before execution.

| Tool | Input summary | Result or effect |
| ------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------------------------- |
| `create_presentation` | Optional `name`, `width`, `height` | Replaces the editor with a blank presentation. |
| `get_presentation_state` | `detail`, optional slide and pagination fields | Returns bounded project, slide, element, revision, and freshness state. |
| `import_powerpoint_from_url` | Required HTTP(S) `url`; optional safe `.pptx` `fileName` | Starts native PPTX import and font resolution. |
| `translate_deck_and_notes` | Required `targetLanguage`; optional `sourceLanguage` | Starts translation of visible text, notes, and existing descriptions. |
| `generate_deck_detailed_description` | Optional `slideNumbers`, `language`, `force` | Starts fresh, revision-linked semantic descriptions. |
| `list_authoring_catalog` | `kind: "fonts"`; or animations plus `elementType` | Returns bounded usable font or animation metadata. |
| `upsert_slide_content` | Stable `requestId`, slide, mode, and typed elements | Atomically merges or replaces exact slide primitives. |
| `generate_image` | `prompt`; optional dimensions, seed, and steps | Starts image generation and returns an asset ID without placing it. |
| `get_slide_preview` | One-based `slideNumber` | Selects and fits the slide and returns dimensions, count, and render hash. |
| `get_ai_model_status` | `{}` | Reports browser support, providers, model readiness, sizes, and errors. |
| `prepare_ai_models` | Optional `modelIds`; omit to prepare required models | Starts model downloads and preparation. |
| `search_media` | `kind`, `term`, optional `limit` | Returns bounded Unsplash or GIPHY references and attribution. |
| `export_presentation` | `format`, optional `slideRange` and `includeAnimationFrames` | Starts a native PPTX/PDF/PNG/JPEG download. |
| `publish_presentation` | Optional stable `shareId` and `expectedRevision` | Publishes an exact snapshot and returns public/embed URLs and its revision. |
| `get_operation_status` | `operationId`, optional `waitForChangeMs` | Returns queued, running, completed, or failed operation state. |
| Tool | Input summary | Result or effect |
| ------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------------------------------- |
| `create_presentation` | Optional `name`, `width`, `height` | Replaces the editor with a blank presentation. |
| `get_presentation_state` | `detail`, optional slide and pagination fields | Returns bounded project, slide, element, revision, and freshness state. |
| `import_powerpoint_from_url` | Required HTTP(S) `url`; optional safe `.pptx` `fileName` | Starts native PPTX import and font resolution. |
| `translate_deck_and_notes` | Required `targetLanguage`; optional `sourceLanguage` | Starts translation of visible text, notes, and existing descriptions. |
| `generate_deck_detailed_description` | Optional `slideNumbers`, `language`, `force` | Starts fresh, revision-linked semantic descriptions. |
| `list_authoring_catalog` | `kind: "fonts"`; or animations plus `elementType` | Returns bounded usable font or animation metadata. |
| `upsert_slide_content` | Stable `requestId`, slide, mode, and typed elements | Atomically merges or replaces exact slide primitives. |
| `generate_image` | `prompt`; optional dimensions, seed, and steps | Starts image generation and returns an asset ID without placing it. |
| `get_slide_preview` | One-based `slideNumber` | Selects and fits the slide and returns dimensions, count, and render hash. |
| `get_ai_model_status` | `{}` | Reports browser support, providers, model readiness, sizes, and errors. |
| `prepare_ai_models` | Optional `modelIds`; omit to prepare required models | Starts model downloads and preparation. |
| `search_media` | `kind`, `term`, optional `limit` | Returns bounded Unsplash or GIPHY references and attribution. |
| `export_presentation` | `format`, optional `slideRange` and `includeAnimationFrames` | Starts a native PPTX/PDF/PNG/JPEG download. |
| `get_operation_status` | `operationId`, optional `waitForChangeMs` | Returns queued, running, completed, or failed operation state. |

Reader tools are marked read-only. Results containing imported or user-authored content are marked untrusted so slide text and metadata remain evidence rather than instructions.

## Long-running operations

Import, translation, description generation, image generation, model preparation, export, and publishing return an operation ID immediately:
Import, translation, description generation, image generation, model preparation, and export return an operation ID immediately:

```json
{
Expand Down Expand Up @@ -73,37 +71,35 @@ The server must:
- return a successful HTTP status;
- return the PPTX MIME type or `application/octet-stream`;
- provide a safe `.pptx` name in the URL, `Content-Disposition`, or `fileName` input;
- stay within the configured size limit.
- stay within any deployment-specific size limit. LocalStudio does not impose a default PPTX size cap.

LocalStudio streams and bounds the download, then uses the same native parser, mapper, warnings, normalization, and font workflow as visible PowerPoint import. Expired URLs, unreachable servers, CORS failures, wrong MIME types, oversized files, and corrupt packages fail without replacing the current project.
LocalStudio streams the download and enforces a deployment-specific limit when one is configured, then uses the same native parser, mapper, warnings, normalization, and font workflow as visible PowerPoint import. Expired URLs, unreachable servers, CORS failures, wrong MIME types, configured-limit violations, and corrupt packages fail without replacing the current project.

The normal **File > Import > PowerPoint** picker remains available for a person using the editor. It is intentionally not exposed as a WebMCP disk-import tool.

## Manual authoring workflow

1. Open `/editor/webmcp/` and select **Discover tools**. Confirm that 15 tools appear.
1. Open `/editor/webmcp/` and select **Discover tools**. Confirm that 14 tools appear.
2. Run **Create presentation**, then **Upsert slide content**. Confirm the embedded canvas changes.
3. Run **Inspect presentation state** and compare the returned slide revision and elements with the canvas.
4. Optionally import a CORS-enabled PPTX URL. Poll the operation and review page, byte, font, and warning counts.
5. Run **Inspect AI model status**, then **Prepare AI models** with `{}` if a required model is not ready.
6. Run **Translate deck and notes** and **Generate detailed descriptions**. Poll both operations; inspect changed/skipped slides, failures, overflow warnings, description language, generator, timestamp, source revision, reviewed state, and freshness.
7. Run **Focus slide preview** before visual inspection.
8. Run each export format. Inspect the downloaded PPTX/PDF or PNG/JPEG ZIP rather than relying only on the success message.
9. Read the current presentation revision, configure remote storage, then run **Publish presentation** with that value as `expectedRevision`.
10. Open the returned public URL in a clean browser context and confirm its authoring revision matches. The published snapshot includes mirrored fonts, semantic descriptions, transcript context, and raw recording audio only when that recording was authorized for sharing.

### Cross-client check

Run this small check in every supported agent browser:

1. Open `/editor/?webmcp=1` directly and confirm native discovery returns exactly 15 tools.
1. Open `/editor/?webmcp=1` directly and confirm native discovery returns exactly 14 tools.
2. Call `create_presentation`, `upsert_slide_content`, `get_presentation_state`, and `get_slide_preview` through the browser's WebMCP interface.
3. Confirm the state revision and preview render hash agree and visually inspect the same title on the canvas.
4. Open `/editor/webmcp/` separately and confirm its manual bridge finds the same 15 names and exposes 15 editable cards.
4. Open `/editor/webmcp/` separately and confirm its manual bridge finds the same 14 names and exposes 14 editable cards.

The direct editor route proves browser-native WebMCP. The showcase proves the judge-friendly manual control surface; they are complementary checks.

For a short judge demo, show the sequence import → translate → describe → preview → publish. Keep the operation-status card visible, show the canvas changing after each mutation, and finish by opening the returned URL in a clean context. Export, media search, catalogs, AI status, model preparation, and image generation can be shown briefly through their editable cards to demonstrate the complete catalog.
For a short judge demo, show the sequence import → translate → describe → preview → export. Keep the operation status visible in both the editor and the action card, show the canvas changing after each mutation, and finish by inspecting the exported artifact. Media search, catalogs, AI status, model preparation, and image generation can be shown briefly through their editable cards to demonstrate the complete catalog.

## Failures and recovery

Expand All @@ -113,9 +109,7 @@ For a short judge demo, show the sequence import → translate → describe →
- **Import fails:** check URL expiry, HTTP status, CORS, MIME type, filename, size, and whether the file is a valid PPTX package.
- **Media search fails:** configure the matching Unsplash or GIPHY integration.
- **AI preparation or generation fails:** inspect `get_ai_model_status`, browser compatibility, free storage, WebGPU support, and model errors.
- **Publish fails before upload:** configure remote storage. If `expectedRevision` is stale, read state again and retry with the new revision.
- **Publish fails during upload:** verify S3/MinIO endpoint, bucket policy, CORS, public base URL, and writer credentials; retrying the same stable share ID updates the same link.

## Public-viewer roadmap

Attendee-side tools for slide context, transcript search, recording metadata, and navigation are a separate future surface with a different read-only authorization model. They are not part of the shipped 15-tool authoring catalog and should not be presented as current behavior.
Attendee-side tools for slide context, transcript search, recording metadata, and navigation are a separate future surface with a different read-only authorization model. They are not part of the shipped 14-tool authoring catalog and should not be presented as current behavior.
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import type { SlideUpsertBatch, SlideUpsertResult } from './slideUpsertService';
import {
AuthoringOperationRegistry,
type AuthoringOperationProgress,
type AuthoringOperationStatus,
} from './authoringOperationRegistry';

export type AuthoringResult<T> =
Expand Down Expand Up @@ -76,13 +77,6 @@ export interface AuthoringAutomationDelegate {
},
report: AuthoringProgressReporter,
): Promise<unknown>;
publishPresentation?(
input: {
shareId?: string | undefined;
expectedRevision?: string | undefined;
},
report: AuthoringProgressReporter,
): Promise<unknown>;
}

function success<T>(data: T): AuthoringResult<T> {
Expand All @@ -109,13 +103,20 @@ function canonicalize(value: unknown): unknown {
}

class AuthoringAutomationController {
private readonly operations = new AuthoringOperationRegistry();
private readonly operations: AuthoringOperationRegistry;
private readonly upserts = new Map<
string,
{ inputKey: string; promise: Promise<SlideUpsertResult> }
>();

constructor(private readonly delegate: AuthoringAutomationDelegate) {}
constructor(
private readonly delegate: AuthoringAutomationDelegate,
options: {
onOperationStatusChange?: (status: AuthoringOperationStatus) => void;
} = {},
) {
this.operations = new AuthoringOperationRegistry(options.onOperationStatusChange);
}

async createPresentation(input: {
name?: string | undefined;
Expand Down Expand Up @@ -299,17 +300,6 @@ class AuthoringAutomationController {
);
}

publishPresentation(input: {
shareId?: string | undefined;
expectedRevision?: string | undefined;
}) {
if (!this.delegate.publishPresentation) return this.pending('publish_presentation', 177);
const run = this.delegate.publishPresentation.bind(this.delegate);
return operationStarted(
this.operations.start('publishing-presentation', (report) => run(input, report)),
);
}

async getOperationStatus(input: {
operationId: string;
waitForChangeMs?: number | undefined;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ function cloneStatus(status: AuthoringOperationStatus): AuthoringOperationStatus
export class AuthoringOperationRegistry {
private readonly operations = new Map<string, AuthoringOperationStatus>();

constructor(private readonly onStatusChange?: (status: AuthoringOperationStatus) => void) {}

start(stage: string, task: OperationTask): AuthoringOperationStatus {
const operationId = createOperationId();
const timestamp = new Date().toISOString();
Expand All @@ -59,6 +61,7 @@ export class AuthoringOperationRegistry {
warnings: [],
};
this.operations.set(operationId, status);
this.onStatusChange?.(cloneStatus(status));

void Promise.resolve().then(async () => {
this.update(operationId, { state: 'running', progress: 1 });
Expand Down Expand Up @@ -94,14 +97,16 @@ export class AuthoringOperationRegistry {
patch.progress === undefined
? current.progress
: Math.max(current.progress, Math.min(100, Math.round(patch.progress)));
this.operations.set(operationId, {
const nextStatus: AuthoringOperationStatus = {
...current,
...patch,
progress,
percentage: progress,
revision: current.revision + 1,
updatedAt: new Date().toISOString(),
warnings: patch.warnings ? [...patch.warnings] : current.warnings,
});
};
this.operations.set(operationId, nextStatus);
this.onStatusChange?.(cloneStatus(nextStatus));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import type { AuthoringAssetCapabilities } from './createAuthoringAssetCapabilit
import type { deckLocalizationCapability } from './deckLocalizationCapability';
import type { createAuthoringVisualCapability } from './authoringVisualCapability';
import type { PowerPointUrlImportService } from './powerPointUrlImportService';
import type { PresentationPublishingCapability } from './presentationPublishingCapability';
import { authoringRevision } from './getAuthoringSlideRevision';
import {
slideUpsertService,
Expand All @@ -23,7 +22,6 @@ interface CreateAuthoringDelegateOptions {
replaceProject(project: ProjectDocument): void;
applyProject(project: ProjectDocument, activePageId?: string): void;
powerPointUrlImportService?: PowerPointUrlImportService | undefined;
publishingCapability?: PresentationPublishingCapability<unknown> | undefined;
visualCapability?: ReturnType<typeof createAuthoringVisualCapability> | undefined;
}

Expand Down Expand Up @@ -129,7 +127,6 @@ export function createAuthoringAutomationDelegate(
const assetCapabilities = options.assetCapabilities;
const deckLocalization = options.deckLocalization;
const powerPointUrlImportService = options.powerPointUrlImportService;
const publishingCapability = options.publishingCapability;
const visualCapability = options.visualCapability;

function resolveMedia(
Expand Down Expand Up @@ -309,13 +306,5 @@ export function createAuthoringAutomationDelegate(
) => visualCapability.exportPresentation(input, report),
}
: {}),
...(publishingCapability
? {
publishPresentation: (
input: Parameters<NonNullable<AuthoringAutomationDelegate['publishPresentation']>>[0],
report: Parameters<NonNullable<AuthoringAutomationDelegate['publishPresentation']>>[1],
) => publishingCapability.publish(input, report),
}
: {}),
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ export interface DeckDescriptionResult {
language: string;
sourceRevision: string;
freshness: 'fresh';
text: string;
}>;
failures: DeckDescriptionFailure[];
failureCount: number;
Expand All @@ -67,6 +68,7 @@ export interface DeckDescriptionResult {
const descriptionLimits = {
maxCharacters: 12_000,
maxEntries: 100,
maxResultCharacters: 2_000,
maxSceneElements: 100,
maxSceneTextCharacters: 2_000,
} as const;
Expand Down Expand Up @@ -296,6 +298,7 @@ function createCapability(options: DeckLocalizationCapabilityOptions) {
language: descriptionLanguage,
sourceRevision,
freshness: 'fresh',
text: text.slice(0, descriptionLimits.maxResultCharacters),
});
report({
stage: 'describing-slides',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ export interface DeckCapabilityProgress {
current?: number;
total?: number;
detail?: string;
loadedBytes?: number;
totalBytes?: number;
warnings?: string[];
}

Expand Down
Loading
Loading