Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 28 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ LocalStudio.dev runs in the browser without a product backend. Your deck remains
as normal assets, translated text updates in place, presenter controls can run from the companion PWA, spoken sessions
can become accessible transcripts, and project files can be saved to a local folder you control.

| Landing section | What it proves |
| --- | --- |
| Watch the workflow | Import, prompt, generate images, translate, save locally, present, record, and share. |
| Feature showcase | Every AI action returns to editable slide layers inside the same deck. |
| WebMCP Showcase | Host pages and agents can discover editor tools and drive the same local-first surface. |
| Requirements | Chrome-first browser APIs, WebGPU model caches, and local storage expectations. |
| Landing section | What it proves |
| ------------------ | --------------------------------------------------------------------------------------- |
| Watch the workflow | Import, prompt, generate images, translate, save locally, present, record, and share. |
| Feature showcase | Every AI action returns to editable slide layers inside the same deck. |
| WebMCP Showcase | Host pages and agents can discover editor tools and drive the same local-first surface. |
| Requirements | Chrome-first browser APIs, WebGPU model caches, and local storage expectations. |

![LocalStudio prompt-to-slide workflow](apps/landing/public/demo-prompt-to-slides.gif)

Expand All @@ -44,15 +44,15 @@ can become accessible transcripts, and project files can be saved to a local fol
The landing page now walks through the full LocalStudio loop with short product demos. Each step keeps the deck editable
instead of producing a locked screenshot.

| Workflow | Demo |
| --- | --- |
| Bring your own PPT | ![Import an existing presentation into LocalStudio](apps/landing/public/demo-bring-your-ppt.gif) |
| Prompt-to-slide | ![Generate editable slides from a prompt](apps/landing/public/demo-prompt-to-slides.gif) |
| Prompt-to-image | ![Generate an image asset and keep composing the same slide](apps/landing/public/demo-prompt-to-image.gif) |
| Translate | ![Translate slide text in place](apps/landing/public/demo-translate.gif) |
| Work locally | ![Save project files locally and browse version history](apps/landing/public/demo-work-locally.gif) |
| Present with confidence | ![Run a LocalStudio deck in presenter mode](apps/landing/public/demo-present-with-confidence.gif) |
| Share your presentation | ![Publish a portable LocalStudio deck preview](apps/landing/public/demo-share-presentation.gif) |
| Workflow | Demo |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| Bring your own PPT | ![Import an existing presentation into LocalStudio](apps/landing/public/demo-bring-your-ppt.gif) |
| Prompt-to-slide | ![Generate editable slides from a prompt](apps/landing/public/demo-prompt-to-slides.gif) |
| Prompt-to-image | ![Generate an image asset and keep composing the same slide](apps/landing/public/demo-prompt-to-image.gif) |
| Translate | ![Translate slide text in place](apps/landing/public/demo-translate.gif) |
| Work locally | ![Save project files locally and browse version history](apps/landing/public/demo-work-locally.gif) |
| Present with confidence | ![Run a LocalStudio deck in presenter mode](apps/landing/public/demo-present-with-confidence.gif) |
| Share your presentation | ![Publish a portable LocalStudio deck preview](apps/landing/public/demo-share-presentation.gif) |

### Feature showcase

Expand Down Expand Up @@ -112,15 +112,21 @@ need explicit control over the model behind each workflow.

## WebMCP Showcase

WebMCP exposes LocalStudio actions as semantic browser tools, so an external page can discover capabilities, create a
project, generate assets, translate the deck, and read the resulting project snapshot.
WebMCP exposes LocalStudio's production authoring actions as semantic browser tools, so an agent and a person can work
against the same visible editor state. The showcase provides an editable test card for every shipped tool.

- Tool discovery from the editor iframe
- Prompt, image, translate, and snapshot actions
- Same local-first editor surface behind every call
- 15 tools covering creation, bounded inspection, URL-based PPTX import, translation, semantic descriptions, catalogs,
exact slide upserts, image assets, visible previews, AI model status/preparation, stock media, exports, publishing, and
long-running operation status
- Strict JSON schemas, bounded results, read-only/untrusted annotations, and visible editor updates
- Exact-revision public publishing with mirrored fonts, descriptions, transcripts, and authorized recording audio
- Same-origin discovery from the editor iframe, plus a local bridge for manual testing in browsers without WebMCP

[Open the WebMCP showcase](https://localstudio.dev/webmcp/)

[Read the WebMCP authoring guide](apps/docs/guide/work-with-web-ai/webmcp.md) for setup, all tool contracts, operation
polling, the URL-only PowerPoint boundary, the manual/judge workflow, and troubleshooting.

![WebMCP showcase](apps/landing/public/webmcp-showcase.png)

## Requirements
Expand All @@ -130,6 +136,8 @@ LocalStudio runs in the browser, but modern browser AI workflows still need the
- Chrome browser is recommended for Chrome-first browser AI and file system APIs.
- At least 10GB free storage is recommended for model weights, browser-managed caches, generated assets, and local project history.
- Local folder permissions are required for project persistence flows.
- WebMCP requires a supporting in-app browser or an experimental Chrome build with WebMCP testing enabled.
- WebMCP publishing requires S3-compatible remote storage; stock search requires configured Unsplash or GIPHY keys.

## S3-Compatible Project Setup

Expand Down
1 change: 1 addition & 0 deletions apps/docs/.vitepress/docsNavigation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ const guideSidebar: DefaultTheme.SidebarItem[] = [
{ text: 'Prompt to Image', link: '/guide/work-with-web-ai/prompt/image' },
],
},
{ text: 'WebMCP Authoring', link: '/guide/work-with-web-ai/webmcp' },
{ text: 'Translate Decks', link: '/guide/work-with-web-ai/translate-decks' },
{ text: 'Edit Images', link: '/guide/work-with-web-ai/edit-images' },
],
Expand Down
121 changes: 121 additions & 0 deletions apps/docs/guide/work-with-web-ai/webmcp.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# WebMCP authoring

WebMCP lets an agent use LocalStudio's real editor commands while you watch the same canvas. The editor exposes 15 narrowly scoped tools with JSON schemas, bounded results, and visible state changes. The showcase at `/editor/webmcp/` provides an editable card for every tool.

This is an **authoring** integration. Public presentations can contain slide descriptions, transcripts, and authorized recording audio, but the public viewer does not currently register attendee WebMCP tools.

## Requirements

- Use a browser or in-app browser with WebMCP support. In Chrome builds that expose the experimental API, enable the WebMCP testing flag and relaunch.
- Run LocalStudio with `npm run dev`, then open `http://localhost:5184/editor/webmcp/` for local testing.
- Allow the embedded same-origin editor to finish loading before selecting **Discover tools**.
- For native agent discovery, open `http://localhost:5184/editor/?webmcp=1` directly. WebMCP is document-scoped, so a browser agent does not inherit tools registered by the showcase's child iframe.
- Translation, description generation, image generation, and model preparation may require browser AI support, WebGPU, model downloads, and several gigabytes of browser storage.
- Unsplash and GIPHY search require their respective keys in **Settings > Media integrations**.
- Publishing requires configured S3-compatible remote storage. The local MinIO settings in the project README are suitable for development.

## The 15 tools

All inputs are JSON objects. Unknown fields and invalid types are rejected before execution.

| Tool | Input summary | Result or effect |
| ------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------------------------- |
| `create_presentation` | Optional `name`, `width`, `height` | Replaces the editor with a blank presentation. |
| `get_presentation_state` | `detail`, optional slide and pagination fields | Returns bounded project, slide, element, revision, and freshness state. |
| `import_powerpoint_from_url` | Required HTTP(S) `url`; optional safe `.pptx` `fileName` | Starts native PPTX import and font resolution. |
| `translate_deck_and_notes` | Required `targetLanguage`; optional `sourceLanguage` | Starts translation of visible text, notes, and existing descriptions. |
| `generate_deck_detailed_description` | Optional `slideNumbers`, `language`, `force` | Starts fresh, revision-linked semantic descriptions. |
| `list_authoring_catalog` | `kind: "fonts"`; or animations plus `elementType` | Returns bounded usable font or animation metadata. |
| `upsert_slide_content` | Stable `requestId`, slide, mode, and typed elements | Atomically merges or replaces exact slide primitives. |
| `generate_image` | `prompt`; optional dimensions, seed, and steps | Starts image generation and returns an asset ID without placing it. |
| `get_slide_preview` | One-based `slideNumber` | Selects and fits the slide and returns dimensions, count, and render hash. |
| `get_ai_model_status` | `{}` | Reports browser support, providers, model readiness, sizes, and errors. |
| `prepare_ai_models` | Optional `modelIds`; omit to prepare required models | Starts model downloads and preparation. |
| `search_media` | `kind`, `term`, optional `limit` | Returns bounded Unsplash or GIPHY references and attribution. |
| `export_presentation` | `format`, optional `slideRange` and `includeAnimationFrames` | Starts a native PPTX/PDF/PNG/JPEG download. |
| `publish_presentation` | Optional stable `shareId` and `expectedRevision` | Publishes an exact snapshot and returns public/embed URLs and its revision. |
| `get_operation_status` | `operationId`, optional `waitForChangeMs` | Returns queued, running, completed, or failed operation state. |

Reader tools are marked read-only. Results containing imported or user-authored content are marked untrusted so slide text and metadata remain evidence rather than instructions.

## Long-running operations

Import, translation, description generation, image generation, model preparation, export, and publishing return an operation ID immediately:

```json
{
"ok": true,
"data": {
"operationId": "operation-…",
"status": "queued"
}
}
```

Poll with `get_operation_status`. A wait of up to 5,000 ms reduces tight polling:

```json
{
"operationId": "operation-…",
"waitForChangeMs": 1000
}
```

Continue until `data.state` is `completed` or `failed`. Progress can include a stage, percentage, byte totals, slide totals, warnings, and a typed final result. The showcase automatically copies a newly returned operation ID into the status card.

## PowerPoint import is URL-only

`import_powerpoint_from_url` accepts only an authorized HTTP or HTTPS URL. It never accepts base64, raw binary, a disk path, or a staged browser file. Use a presigned MinIO/S3 URL or a localhost HTTP server.

The server must:

- allow browser CORS from the LocalStudio origin;
- return a successful HTTP status;
- return the PPTX MIME type or `application/octet-stream`;
- provide a safe `.pptx` name in the URL, `Content-Disposition`, or `fileName` input;
- stay within the configured size limit.

LocalStudio streams and bounds the download, then uses the same native parser, mapper, warnings, normalization, and font workflow as visible PowerPoint import. Expired URLs, unreachable servers, CORS failures, wrong MIME types, oversized files, and corrupt packages fail without replacing the current project.

The normal **File > Import > PowerPoint** picker remains available for a person using the editor. It is intentionally not exposed as a WebMCP disk-import tool.

## Manual authoring workflow

1. Open `/editor/webmcp/` and select **Discover tools**. Confirm that 15 tools appear.
2. Run **Create presentation**, then **Upsert slide content**. Confirm the embedded canvas changes.
3. Run **Inspect presentation state** and compare the returned slide revision and elements with the canvas.
4. Optionally import a CORS-enabled PPTX URL. Poll the operation and review page, byte, font, and warning counts.
5. Run **Inspect AI model status**, then **Prepare AI models** with `{}` if a required model is not ready.
6. Run **Translate deck and notes** and **Generate detailed descriptions**. Poll both operations; inspect changed/skipped slides, failures, overflow warnings, description language, generator, timestamp, source revision, reviewed state, and freshness.
7. Run **Focus slide preview** before visual inspection.
8. Run each export format. Inspect the downloaded PPTX/PDF or PNG/JPEG ZIP rather than relying only on the success message.
9. Read the current presentation revision, configure remote storage, then run **Publish presentation** with that value as `expectedRevision`.
10. Open the returned public URL in a clean browser context and confirm its authoring revision matches. The published snapshot includes mirrored fonts, semantic descriptions, transcript context, and raw recording audio only when that recording was authorized for sharing.

### Cross-client check

Run this small check in every supported agent browser:

1. Open `/editor/?webmcp=1` directly and confirm native discovery returns exactly 15 tools.
2. Call `create_presentation`, `upsert_slide_content`, `get_presentation_state`, and `get_slide_preview` through the browser's WebMCP interface.
3. Confirm the state revision and preview render hash agree and visually inspect the same title on the canvas.
4. Open `/editor/webmcp/` separately and confirm its manual bridge finds the same 15 names and exposes 15 editable cards.

The direct editor route proves browser-native WebMCP. The showcase proves the judge-friendly manual control surface; they are complementary checks.

For a short judge demo, show the sequence import → translate → describe → preview → publish. Keep the operation-status card visible, show the canvas changing after each mutation, and finish by opening the returned URL in a clean context. Export, media search, catalogs, AI status, model preparation, and image generation can be shown briefly through their editable cards to demonstrate the complete catalog.

## Failures and recovery

- **No tools discovered:** wait for the editor frame, confirm the route is same-origin, and retry. Browsers without WebMCP use the local showcase bridge for manual testing.
- **`invalid_input`:** compare the edited JSON with the tool table; extra fields are rejected.
- **Unknown operation:** use the ID returned by the most recent long-running tool. Operation IDs are page-session state.
- **Import fails:** check URL expiry, HTTP status, CORS, MIME type, filename, size, and whether the file is a valid PPTX package.
- **Media search fails:** configure the matching Unsplash or GIPHY integration.
- **AI preparation or generation fails:** inspect `get_ai_model_status`, browser compatibility, free storage, WebGPU support, and model errors.
- **Publish fails before upload:** configure remote storage. If `expectedRevision` is stale, read state again and retry with the new revision.
- **Publish fails during upload:** verify S3/MinIO endpoint, bucket policy, CORS, public base URL, and writer credentials; retrying the same stable share ID updates the same link.

## Public-viewer roadmap

Attendee-side tools for slide context, transcript search, recording metadata, and navigation are a separate future surface with a different read-only authorization model. They are not part of the shipped 15-tool authoring catalog and should not be presented as current behavior.
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ export class PresentationPublishingCapability<TConfig> {
let share: ShareMetadata;
try {
const options = {
authoringRevision: snapshot.revision,
onProgress: (progress: SharePublishProgress) =>
report({
stage: 'pointer',
Expand Down
2 changes: 2 additions & 0 deletions apps/editor/src/services/contracts/interfaces.ts
Original file line number Diff line number Diff line change
Expand Up @@ -336,11 +336,13 @@ export interface SharePublishProgress {
}

export interface SharePublishOptions {
authoringRevision?: string;
onProgress?: (progress: SharePublishProgress) => void;
}

export interface ShareRecord {
shareId: string;
authoringRevision?: string;
createdAt: string;
updatedAt: string;
project: ProjectDocument;
Expand Down
3 changes: 3 additions & 0 deletions apps/editor/src/services/sharing/shareService.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { publicBasePath } from '../../app/routing/publicBasePath';
import { collectReferencedAssetIds } from '../../domain/assets/assetUsage';
import type { ProjectDocument } from '../../domain/documents/model';
import { authoringRevision } from '../automation/getAuthoringSlideRevision';
import type {
ShareMetadata,
SharePublishOptions,
Expand Down Expand Up @@ -153,6 +154,7 @@ export class BrowserShareService implements ShareService {
if (payload.shareId !== shareId || !payload.project) return null;
return {
shareId,
...(payload.authoringRevision ? { authoringRevision: payload.authoringRevision } : {}),
createdAt: payload.createdAt ?? new Date().toISOString(),
updatedAt: payload.updatedAt ?? payload.createdAt ?? new Date().toISOString(),
project: cloneProject(payload.project),
Expand Down Expand Up @@ -212,6 +214,7 @@ export class BrowserShareService implements ShareService {
const payload: PublicSharePayload = {
schemaVersion: 1,
shareId,
authoringRevision: options?.authoringRevision ?? authoringRevision.getPresentation(project),
createdAt: now,
updatedAt: now,
project: projectForShare,
Expand Down
Loading
Loading