Skip to content

feat: shape annotations + animation presets for AI-authored overlays#46

Open
Misterr-H wants to merge 1 commit into
palmier-io:mainfrom
Misterr-H:feat/shape-annotations
Open

feat: shape annotations + animation presets for AI-authored overlays#46
Misterr-H wants to merge 1 commit into
palmier-io:mainfrom
Misterr-H:feat/shape-annotations

Conversation

@Misterr-H

Copy link
Copy Markdown

Addresses #45.

Opening this as a draft per CONTRIBUTING.md — happy to leave it parked until there's room to review, or close it if the direction doesn't fit. The issue lays out the user-problem framing; this PR is the implementation.

Summary

  • New .shape clip type (rect, oval, circle, arrow, line) rendered as CAShapeLayer overlays — same pattern text clips already use, never composition tracks.
  • 17 named animation presets (pop-in, draw-on, shake-{subtle,strong}, spin, fade-{in,out}, slide-{in,out}-{up,down,left,right}, un-draw) that compile to keyframe rows on the existing tracks.
  • Two MCP tools: add_shapes (batch, with enter/exit/loop animations in one call) and apply_animation (preset on an existing clip). Each is one undoable action; strict allowed-keys validation matches add_texts.
  • Reuses every existing keyframe track (position, scale, rotation, opacity); adds one new strokeProgressTrack to drive draw-on / un-draw.

File map

Phase Files Purpose
Data model Models/ClipType.swift, Models/ShapeStyle.swift (new), Models/Timeline.swift, Models/Keyframe.swift, UI/AppTheme.swift .shape case, style payload, Clip.shapeStyle + Clip.strokeProgressTrack, AnimatableProperty.strokeProgress, track color
Renderer Preview/ShapeLayerController.swift (new), Preview/PreviewView.swift, Preview/VideoEngine.swift, Preview/CompositionBuilder.swift, Preview/TimelineRenderer.swift, Export/ExportService.swift CAShapeLayer overlay tree + per-frame state + export bake via AVVideoCompositionCoreAnimationTool; CompositionBuilder filters out shape clips (same as text) so they never become AVAsset-backed composition tracks
VM Editor/ViewModel/EditorViewModel+ClipMutations.swift, EditorViewModel+Keyframes.swift, EditorViewModel+MediaLibrary.swift, EditorViewModel+Tracks.swift placeShapeClips (mirrors placeTextClips), shape-aware property mutation routing (syncShapeLayers instead of composition rebuild)
Presets Editor/Animations/AnimationPresets.swift (new) 17 named presets — pure functions, deterministic, returns keyframe rows ready to merge into a Clip
MCP Agent/Tools/ToolDefinitions.swift, ToolExecutor.swift, ToolExecutor+Shapes.swift (new), ToolExecutor+Generate.swift, ToolExecutor+Timeline.swift, AgentInstructions.swift add_shapes, apply_animation tool definitions + executor; .shape rejected at generate/timeline-inspect paths; agent guidance for tutorial-annotation patterns
Boilerplate switch coverage Generation/Edit/*.swift, Generation/GenerationService.swift, Generation/UI/GenerationView.swift, MediaPanel/MediaTab/MediaTab.swift Adds .shape to existing ClipType exhaustive switches; disambiguates SwiftUI's ShapeStyle protocol where it collides with the new model type

27 files, +1402 / -55, 4 new files.

What the agent call looks like end-to-end

curl -X POST http://127.0.0.1:19789/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{
    "name":"add_shapes",
    "arguments":{"entries":[{
      "kind":"circle",
      "startFrame":15,"durationFrames":120,
      "transform":{"centerX":0.5,"centerY":0.5,"width":0.22,"height":0.22},
      "style":{"strokeColor":"#FF3B30","strokeWidth":10},
      "enterAnim":"pop-in","exitAnim":"fade-out","loopAnim":"shake-subtle"
    }]}
  }}'

# → "Created track 0 ('V1'). Added 1 shape clip: 64F1FBD3-… (circle)
#    on track 0 @ 15 for 120 [pop-in, fade-out, shake-subtle]"

One undoable action; the preview's CAShapeLayer tree updates immediately via syncShapeLayers(); no composition rebuild required.

Test plan

  • swift build clean (no new warnings introduced)
  • swift test — existing tests pass; no behavioral regressions in keyframe sampling, composition build for non-shape timelines, or text-clip placement
  • MCP tools/list exposes add_shapes + apply_animation with their schemas
  • tools/call add_shapes against a live MCP server returns success + clip IDs
  • Visual confirmation in preview — blocked on a pre-existing AVFoundation continuation issue on my dev machine (macOS 26.4.1 / M1 8GB, libconvexmobile.a built for SDK 26.2 vs deployment 26.0). Code paths verified via MCP responses and OSLog. Will fold in a recording on a healthy machine when reviewed.

Out of scope (intentional)

  • Manual draw-on-canvas tool (would land as a PR Show reference thumbnails in AI generation details #2 with ShapeTab inspector mirroring TextTab)
  • Free-form path / pen, gradients, blur/glow, boolean operations between shapes
  • Auto-tracking (shape follows a moving subject) — needs the search/index team's tracking work
  • Per-keyframe easing curves beyond linear / hold / smooth

How I'd split this if you'd rather review smaller chunks

  1. Data model + renderer (Phases 1–2 above) — ~850 LOC
  2. Presets + MCP tools (Phases 3–5) — ~600 LOC

Happy to rebase into that shape on request. Also happy to close this if the direction doesn't fit — no hard feelings either way.

🤖 Generated with Claude Code

Adds a new `.shape` clip type (rect, oval, circle, arrow, line) and a
named-preset animation engine (pop-in, draw-on, shake, spin, slide-{in,out},
fade-{in,out}, un-draw). Shape clips render as `CAShapeLayer` overlays
following the same pattern text clips already use — never as composition
tracks. The composition rebuild path explicitly skips both text and shape
clips, so adding overlays never triggers an AVAsset load.

Two new MCP tools: `add_shapes` (batch placement with optional enter/exit/
loop animations baked into one call) and `apply_animation` (preset by name
on an existing clip). Each tool is one undoable action, follows the strict-
allowed-keys validation pattern from add_texts, and reuses every existing
keyframe track — no new keyframe infrastructure.

Phases:
- Data model: ClipType (.shape), ShapeStyle.swift, Clip.shapeStyle +
  Clip.strokeProgressTrack, AnimatableProperty.strokeProgress
- Renderer: ShapeLayerController (CAShapeLayer overlay tree + export bake
  via AVVideoCompositionCoreAnimationTool), PreviewNSView setShapeRoot,
  VideoEngine syncShapeLayers, TimelineRenderer + ExportService bake hook
- VM: EditorViewModel.placeShapeClips, shape-aware property mutation
  routing (syncShapeLayers instead of composition rebuild)
- Presets: AnimationPresets engine — 17 named presets, pure functions,
  unit-testable, returns keyframe rows that drop into existing tracks
- MCP: add_shapes, apply_animation tool definitions and executor;
  AgentInstructions guidance for tutorial-annotation workflows

Build clean. Existing tests pass. No new keyframe infrastructure, no new
runtime, no new external dependencies.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Misterr-H
Misterr-H marked this pull request as ready for review June 19, 2026 21:21
TimLai666 added a commit to TimLai666/fronda that referenced this pull request Jun 23, 2026
…mier-io#95, palmier-io#94, palmier-io#92, palmier-io#108, palmier-io#46, palmier-io#105, palmier-io#114) in rewrite specs

Adds Upstream change tracking sections to:
- 01-foundation-and-project-model.md: clip data model, project.json
- 03-timeline-editor-and-preview.md: compositor, captions, preview
- 04-export-rendering-and-interchange.md: stall watcher, resolutions
- 05-agent-mcp-and-chat.md: agent tools, transform merge
TimLai666 added a commit to TimLai666/fronda that referenced this pull request Jun 23, 2026
- Add ShapeStyle model (ShapeKind, Stroke, Fill, Arrowhead, Endpoints)
- Add ClipType::Shape variant
- Add shape_style and stroke_progress_track fields to Clip
- Add add_shapes and apply_animation tool definitions
- Add AddShapesInput and ApplyAnimationInput validation structs
- Add serialization tests and tool validation tests
- Update tool count to 39 across all crates
TimLai666 added a commit to TimLai666/fronda that referenced this pull request Jul 3, 2026
The compositor now rasterizes shape clips instead of skipping them: rasterize_shape
draws a filled + border-stroked rectangle or ellipse into an RGBA buffer, and
compose_frame blits it through the same transform/rotation/opacity/blend path as
media (procedural, so fetch_source is not consulted). visible_clips now includes
shape clips (empty media_ref allowed); Text still needs a rasterizer and Arrow/
Line are follow-ups. So shape annotations (PR palmier-io#46) now appear in both the preview
canvas and the exported video.

+3 tests: rect fully filled, oval fills center not corners, a shape clip
composites its fill onto the canvas.
TimLai666 added a commit to TimLai666/fronda that referenced this pull request Jul 4, 2026
The Rust-only stroke_progress_track (palmier-io#46 shape draw-on animation) is a
clip-relative-frame KeyframeTrack, but was omitted from all three places
that maintain the other six tracks: rescale_clip_keyframes (fps change),
clamp_clip_keyframes_to_duration, and split_all_clip_keyframe_tracks. So
a shape's stroke animation would desync on an fps change and leak/duplicate
across a clip split. Also added to count_keyframes for accurate reporting.

No render impact yet (the compositor doesn't sample the track), but the
stored data is now maintained correctly for when it is wired up. Tests
cover the rescale (50→100 at 2x) and split partitioning.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant