[agent] Opt-in TwelveLabs video understanding (analyze_video tool) by mohit-twelvelabs · Pull Request #198 · palmier-io/palmier-pro

mohit-twelvelabs · 2026-06-27T01:40:17Z

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

Palmier Pro is "the video editor built for AI," so I added an optional bridge to TwelveLabs Pegasus — a cloud video-understanding model that reasons over a whole clip's visuals and audio at once. It's entirely opt-in: with no key set, nothing changes.

What it adds

TwelveLabsClient (Agent/Clients/) — Pegasus video understanding over the v1.3 REST API via URLSession. There's no official Swift SDK, so it talks to the API directly. It mirrors AnthropicClient's auth/error shape, and the key lives in the macOS Keychain via TwelveLabsKeychain (mirroring AnthropicKeychain, including the #if DEBUG env override) — never hardcoded.
analyze_video agent tool (Agent/Tools/, wired into ToolExecutor/ToolDefinitions exactly like the existing tools, so it's also exposed over the MCP server) — the in-app agent can ask Pegasus about a video asset: "summarize this clip," "what is the speaker demonstrating," "find the moment the goal is scored." This complements inspect_media (on-device sampled frames + transcription) by adding whole-clip understanding that isolated frames miss. If no TwelveLabs key is configured, the tool returns a clear notice and no-ops, so the agent simply falls back to inspect_media.
Settings → Agent — a TwelveLabs API key field, stored identically to the Anthropic key (Keychain, with a trash button to clear it).
Docs: a short opt-in capability note in the README and one line in the agent instructions.

Why it fits

A video editor built for AI is the ideal home for video understanding. This keeps the existing on-device path as the default and layers cloud understanding on top only when a user provides their own key.

How it was tested

The TwelveLabs REST contracts the client depends on were live-verified end to end against api.twelvelabs.io/v1.3: multipart asset upload (method=direct, file=) → _id, then POST /v1.3/analyze with {"video":{"type":"asset_id",...},"model_name":"pegasus1.5","stream":false} returning the answer at .data.
I wrote the Swift to match the repo's existing conventions closely (client/keychain/error shape, ToolExecutor extension pattern, AppTheme-only styling in the settings pane, minimal comments per AGENTS.md, async/await + off-main-actor file IO). New files pass swiftc -parse.
Honest caveat: I couldn't run a full swift build to completion on my non-macOS-26 CI — the build reached the Metal-toolchain plugin step (the MetalCIKernelPlugin) before my toolchain blocked there, which is unrelated to these changes. CI here runs on a macos-26 runner and will compile it properly; I'd appreciate a maintainer building it on macOS to confirm. Happy to adjust anything to fit your conventions.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Note

Medium Risk
Uploads user video to a third-party API when enabled; key handling follows existing Anthropic patterns but adds new external dependency and long-running network I/O on agent tool calls.

Overview
Adds optional cloud video understanding via TwelveLabs Pegasus. With no API key, behavior is unchanged.

TwelveLabsClient uploads a local video to TwelveLabs v1.3 (streaming multipart to disk, 200 MB direct-upload cap, 300s timeouts), runs Pegasus pegasus1.5, and returns the answer. TwelveLabsKeychain stores the key like Anthropic (Keychain + DEBUG env override).

New analyze_video agent/MCP tool (mediaRef + prompt) validates video assets on disk, runs upload/analysis off the main actor, and returns a clear notice when TwelveLabs isn’t configured so the agent falls back to inspect_media.

Settings → Agent gets a TwelveLabs key field (save/remove, masked display). README and agent instructions document when to use analyze_video vs on-device inspection.

^{Reviewed by Cursor Bugbot for commit 30bd1e3. Bugbot is set up for automated code reviews on this repo. Configure here.}

Adds a TwelveLabs Pegasus integration behind an optional API key. With no key set, behavior is unchanged. - TwelveLabsClient: Pegasus video understanding over the v1.3 REST API via URLSession (no official Swift SDK). Mirrors AnthropicClient's auth/error shape; the key lives in the Keychain via TwelveLabsKeychain, never hardcoded. - analyze_video agent tool (also exposed over MCP): ask Pegasus a question about a whole video clip — summary, content check, locate a moment — reasoning over motion and audio together, beyond inspect_media's on-device frame sampling. Returns a clear notice and no-ops when no key is configured. - Settings → Agent: TwelveLabs API key field, stored exactly like the Anthropic key (Keychain, DEBUG env override). - README + agent instructions document the opt-in capability.

Mirror inspect_media's generationStatus handling so a missing video file with a .failed status reports the failure message instead of a generic 'Media file not on disk'.

mohit-twelvelabs · 2026-06-27T01:47:18Z

Good catch from Cursor Bugbot 🐛 — fixed in 6c25162. analyze_video now mirrors inspect_media's generationStatus handling: a missing file with .failed(let msg) surfaces "Asset <id> failed: <msg>" (and the downloading/generating/rendering states get their specific guidance too), instead of the generic "Media file not on disk".

— Mohit (@mohit-twelvelabs, TwelveLabs)

- uploadAsset now writes the multipart body to a temp file and uses URLSession.upload(fromFile:), streaming the source clip in 1 MiB chunks instead of loading the whole (multi-GB) video into memory. - Set a 300s timeoutInterval on the upload and analyze requests so whole-clip Pegasus analysis doesn't hit URLSession's default 60s limit.

mohit-twelvelabs · 2026-06-27T01:57:19Z

Thanks Cursor Bugbot 🐛 — both addressed in 6fc9724:

Whole video loaded into memory (High): uploadAsset no longer reads the file into Data/httpBody. It now streams the multipart body to a temp file in 1 MiB chunks and uploads via URLSession.upload(for:fromFile:), so a multi-GB source clip is never fully resident in memory.
Default timeout breaks long analyze (Medium): set request.timeoutInterval = 300 on both the upload and the analyze requests, so whole-clip Pegasus processing won't trip URLSession's default ~60s idle limit.

— Mohit (@mohit-twelvelabs, TwelveLabs)

@mainactor

…-upload limit - analyze_video now runs client.understand in a detached task so the synchronous multipart-body streaming never blocks the @mainactor editor. - uploadAsset rejects files over TwelveLabs' 200MB direct-upload limit with a clear, actionable error instead of an opaque API failure. (Larger files need the multipart/chunked upload flow — a follow-up.)

mohit-twelvelabs · 2026-06-27T02:31:03Z

Thanks again, Cursor Bugbot 🐛 — both addressed in 974225a:

Main-thread multipart build (Medium): analyze_video now runs client.understand(...) inside a Task.detached(priority: .userInitiated), so the synchronous multipart-body streaming happens off the @MainActor ToolExecutor and never blocks the editor.
Large files need multipart API (Medium): confirmed against the TwelveLabs SDK docs — direct local-file upload (method=direct) is capped at 200 MB (public URLs go to 4 GB). uploadAsset now guards that limit and returns a clear, actionable error ("Video is X MB, but TwelveLabs direct upload supports up to 200 MB — export or trim a smaller clip") instead of an opaque API failure on full-res source media. Full multipart/chunked upload for >200 MB local files is a sensible follow-up (the SDK exposes a presigned-chunk flow for it); I kept this PR focused per CONTRIBUTING's note on large PRs, but happy to add it if you'd like it in scope.

— Mohit (@mohit-twelvelabs, TwelveLabs)

…ctually fires attributesOfItem(.size) returns an NSNumber; the as? Int cast is unreliable (notably under non-Apple Foundation), which could let the fileTooLarge guard silently no-op. URL.resourceValues(.fileSizeKey).fileSize returns Int directly.

mohit-twelvelabs · 2026-06-27T02:35:55Z

Good catch, Cursor 🐛 — fixed in 30bd1e3. Switched the size check from attributesOfItem(.size) as? Int (which goes through fragile NSNumber bridging) to fileURL.resourceValues(forKeys: [.fileSizeKey]).fileSize, which returns Int? directly — so the 200 MB fileTooLarge guard now reliably fires before any upload work.

— Mohit (@mohit-twelvelabs, TwelveLabs)

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 30bd1e3. Configure here.}

cursor · 2026-06-27T02:37:47Z

+            // source file from disk, which would otherwise block the editor on @MainActor.
+            answer = try await Task.detached(priority: .userInitiated) {
+                try await client.understand(videoURL: url, prompt: prompt)
+            }.value


Cancel leaves upload running

Medium Severity

analyze_video runs TwelveLabsClient.understand inside Task.detached, so stopping the agent (or cancelling the stream task) does not stop the in-flight upload and analyze. The editor may treat the tool as cancelled while the clip still streams to TwelveLabs and completes analysis.

^{Reviewed by Cursor Bugbot for commit 30bd1e3. Configure here.}

htin1 · 2026-06-27T03:29:39Z

@mohit-twelvelabs can you shoot us an email at founders@palmier.io about this. we want to chat

mohit-twelvelabs · 2026-06-27T04:49:51Z

someone from our team will be in contact, thanks!