Wingman is an AI-powered music production assistant that provides deep Ableton Live control through a streamlined two-component architecture:
- Standalone App (Electron): React UI with AI that runs independently on your desktop
- Remote Script (Python): Ableton Live API integration for total DAW control
Goal: AI can see and control EVERYTHING in Ableton Live - tempo, tracks, clips, plugins, automation, mixer - providing intelligent creative assistance with robust safety guardrails.
┌─────────────────────────────────────────────┐
│ Wingman Standalone App (Electron) │
│ - Full React UI (WingmanMinimal) │
│ - AI logic (OpenAI, Anthropic, etc.) │
│ - Chat interface │
│ - WebSocket server :8765 │
│ - Safety controls & guardrails │
│ - Connection status monitoring │
└──────────────────┬──────────────────────────┘
│
↕ WebSocket (JSON Messages)
│
┌──────────────────┴──────────────────────────┐
│ Ableton Remote Script (Python 3) │
│ - WebSocket client (connects to App) │
│ - Full Live API access (LOM) │
│ - Track/clip/device control │
│ - Scene launching │
│ - Parameter automation │
│ - MIDI clip creation │
│ - Real-time observers for state changes │
└─────────────────────────────────────────────┘
Simplicity: Only 2 components = easier debugging, faster development Reliability: WebSocket provides ordered, reliable message delivery Complete Control: Remote Script has full LOM (Live Object Model) access Safety-First: Guardrails built into app from day one No Plugin Required: Works entirely outside the VST3 ecosystem
❌ Bridge VST3: Unnecessary - Remote Script provides all DAW control we need ❌ UDP Communication: Unreliable (packet loss/reordering) - WebSocket is better for control messages ❌ Complex Port Management: Single WebSocket connection instead of 4+ UDP ports ❌ UID System: Fragile hash-based IDs that break on track renames - use observers instead ❌ Max for Live: Deferred to future phase - not critical for core functionality
The Ableton Remote Script (Python 3 in Live 11+) provides everything we need:
✅ Full Read Access:
- All track properties (name, color, type, arm state, mute, solo, volume, pan)
- Clip matrix (has_clip, is_playing, is_triggered, length, name)
- Device chains and all parameter values
- Transport state (playing, tempo, time signature, position)
- Scene list and properties
✅ Full Control:
- Create/delete tracks and scenes
- Launch clips and scenes
- Arm/mute/solo tracks
- Create MIDI clips and add notes
- Set device parameters (with automation)
- Control transport (play/stop/tempo/locate)
✅ Real-Time Observers:
- Detect any parameter changes instantly
- Track user actions vs. AI actions
- Respond to set changes (tracks added/removed)
- Framework: Electron 28+ (Windows first, Mac later)
- UI: React 18 + TypeScript (existing WingmanMinimal)
- Build: Vite 5
- AI: OpenAI GPT-4, Anthropic Claude, Perplexity, OpenRouter
- Communication: WebSocket server using
wslibrary (port 8765) - Node Version: 18+ LTS
- Language: Python 3.x (Live 11+)
- API: Ableton Live Object Model (LOM) v2 framework
- Location:
%USERPROFILE%\Documents\Ableton\User Library\Remote Scripts\WingmanControl - Communication: WebSocket client using built-in Python modules
- Threading: Python threading for non-blocking WebSocket I/O
Important: Ableton Live 11+ uses Python 3, but it's a stripped-down interpreter:
- No
subprocess,msvcrt,win32apimodules - Limited standard library access
socketandthreadingmodules ARE available
Port: 8765 (app listens, script connects) Format: JSON messages over WebSocket Reliability: TCP-based, ordered delivery, automatic reconnection
AI or user triggers an action in Ableton Live.
{
"type": "command",
"id": "cmd-a3f91b",
"cmd": "clip.launch",
"args": {
"track_index": 0,
"scene_index": 2,
"quantization": 1
},
"meta": {
"origin": "ai",
"timestamp": 1730512345.123
}
}Ableton Live state changes (from observers or user actions).
{
"type": "event",
"event": "tempo_changed",
"data": {
"tempo": 128.0,
"timestamp": 1730512346.456
}
}Response to a command (success or error).
{
"type": "ack",
"id": "cmd-a3f91b",
"success": true,
"result": {
"clip_launched": true,
"is_playing": true
}
}Full DAW state (sent on connection and on-demand).
{
"type": "snapshot",
"data": {
"tempo": 120.0,
"time_signature_numerator": 4,
"time_signature_denominator": 4,
"playing": false,
"current_song_time": 0.0,
"tracks": [
{
"index": 0,
"name": "1-MIDI",
"type": "midi",
"color": 12,
"arm": false,
"mute": false,
"solo": false,
"volume": 0.85,
"pan": 0.0,
"clips": [
{"index": 0, "has_clip": false},
{"index": 1, "has_clip": true, "name": "Bass", "length": 4.0, "playing": false}
]
}
],
"scenes": [
{"index": 0, "name": "Scene 1"},
{"index": 1, "name": "Breakdown"}
]
}
}Transport:
transport.play- Start playbacktransport.stop- Stop playbacktransport.continue- Continue from current positiontransport.locate- Jump to position (args:beats)transport.set_tempo- Change BPM (args:tempo)transport.record- Toggle record mode
Clips:
clip.launch- Fire clip (args:track_index,scene_index,quantization)clip.stop- Stop clip (args:track_index)clip.create- Create new MIDI clip (args:track_index,scene_index,length)clip.delete- Remove clip (args:track_index,scene_index)clip.duplicate- Duplicate clip (args:track_index,scene_index)clip.add_notes- Add MIDI notes (args:track_index,scene_index,notes)clip.clear_notes- Remove all notes (args:track_index,scene_index)
Tracks:
track.create_midi- Create MIDI track (args:name)track.create_audio- Create audio track (args:name)track.delete- Remove track (args:track_index)track.arm- Set arm state (args:track_index,armed)track.mute- Set mute state (args:track_index,muted)track.solo- Set solo state (args:track_index,soloed)track.rename- Change track name (args:track_index,name)track.set_volume- Set volume (args:track_index,volume[0.0-1.0])track.set_pan- Set pan (args:track_index,pan[-1.0-1.0])
Scenes:
scene.launch- Fire scene (args:scene_index)scene.create- Create new scene (args:name)scene.delete- Remove scene (args:scene_index)scene.duplicate- Duplicate scene (args:scene_index)
Devices:
device.param.set- Set device parameter (args:track_index,device_index,param_index,value)device.add- Load device on track (args:track_index,device_name)device.remove- Remove device (args:track_index,device_index)
Transport Events:
tempo_changed- BPM changedtime_signature_changed- Time sig changedplaying_status_changed- Play/stop stateposition_changed- Playhead moved (throttled to 10Hz)
Clip Events:
clip_playing_status_changed- Clip started/stoppedclip_added- New clip createdclip_removed- Clip deleted
Track Events:
track_added- Track createdtrack_removed- Track deletedtrack_name_changed- Track renamedtrack_arm_changed- Arm state changedtrack_mute_changed- Mute state changed
Parameter Events:
device_param_changed- Device parameter value changeduser_touched_param- User manually changed parameter (AI should pause)
Reality: Ableton uses indices that shift when tracks/scenes are reordered or deleted.
Previous Approach (REMOVED): Generate stable UIDs via hashing track properties Why It Failed: UIDs break when users rename tracks, change colors, or modify devices - which happens constantly in music production
New Approach: Accept index-based model and use observers to track changes
- Initial Snapshot: On connection, Remote Script sends full DAW state with indices
- Observer Registration: Script registers observers for all relevant properties
- Change Detection: When anything changes, script sends event with current indices
- App State Sync: Electron app maintains synchronized state model
- Command Validation: Before executing, script validates indices against current state
User renames track 2 → Observer fires → Script sends event:
{
"type": "event",
"event": "track_name_changed",
"data": {
"track_index": 2,
"old_name": "Drums",
"new_name": "Drum Bus"
}
}
App updates its state → AI now knows track 2 is "Drum Bus"
AI sends command:
{
"cmd": "track.mute",
"args": {"track_index": 2, "muted": true}
}
Script validates track 2 exists → Executes → Sends ack
Track Deletion:
User deletes track 3 (of 5 tracks)
→ Track 4 becomes track 3, track 5 becomes track 4
→ Script sends: { "event": "track_removed", "data": {"track_index": 3} }
→ App rebuilds track list
→ In-flight commands to old track 4/5 fail gracefully with error
Track Reordering:
User drags track 1 to position 3
→ Script detects track list change
→ Sends full track list update event
→ App rebuilds state
UI: Large red "PAUSE AI" button always visible Function: Immediately stops all AI-generated commands Scope: User-initiated commands still work
Default-deny for destructive operations:
const DEFAULT_ACL = {
ALLOW_TRACK_DELETE: false, // Requires explicit enable
ALLOW_CLIP_DELETE: false, // Requires explicit enable
ALLOW_SCENE_DELETE: false, // Requires explicit enable
ALLOW_DEVICE_REMOVE: false, // Requires explicit enable
ALLOW_TRANSPORT_CONTROL: true, // Safe operations
ALLOW_CLIP_LAUNCH: true,
ALLOW_PARAMETER_CHANGE: true
};UI: Settings panel with checkboxes for each permission Storage: Persisted to user config file
- Global: Max 10 commands/second from AI
- Per-Parameter: Max 10 parameter changes/second
- Burst Protection: Max 5 commands in 100ms window
Behavior: Excess commands queued or rejected with error
When user manually touches a control:
- Script detects parameter change from user (not from script)
- Sends
user_touched_paramevent to app - App freezes AI control of that parameter for 5 seconds
- Visual indicator in UI: "User controlling [param name]"
Every command validated before execution:
- Track/clip/scene indices checked against current state
- Parameter values clamped to min/max ranges
- Required arguments presence check
- Invalid commands rejected with detailed error
- All AI commands logged with timestamps
- "Undo Last AI Action" button
- Undo stack persisted across sessions
- Max 50 actions in history
- UI: "Preview Mode" toggle
- Function: Shows what AI would do without executing
- Use Case: Test AI understanding before committing
// App side (Electron)
ws.on('close', () => {
connectionStatus = 'disconnected';
showNotification('Lost connection to Ableton Live');
attemptReconnect();
});
function attemptReconnect() {
let retries = 0;
const maxRetries = 10;
const retryInterval = setInterval(() => {
if (retries >= maxRetries) {
clearInterval(retryInterval);
showError('Could not reconnect to Ableton Live');
return;
}
// Try to reconnect...
retries++;
}, 2000); // Retry every 2 seconds
}- Default timeout: 5 seconds per command
- If no ack received, command considered failed
- UI shows timeout error with retry option
- App requests snapshot every 30 seconds (heartbeat)
- If snapshot differs from app state, full resync
- User notified of desync with details
- Script logs errors to Live's Log.txt
- On restart, script sends connection event
- App detects reconnection, requests fresh snapshot
- Previous command queue cleared
- VST3 Plugin Loading: Loads in Ableton Live successfully
- WebView2 Integration: Renders React UI in plugin window
- Basic DAW Context: Reads BPM, time signature, playback state
- AI Integration: OpenAI GPT-4 with DAW context in prompts
- Build System: Automated CMake + Vite build
- Legacy VST3: Kept for backward compatibility during transition
- New Architecture: Electron + Remote Script (actively developing)
- Dual Support: Both architectures will coexist during migration
- Bridge VST3 (minimal plugin) - Not needed
- UDP communication layer - Replaced by WebSocket
- UID generation system - Replaced by index + observers
- Max for Live integration - Deferred to future
Goal: Electron app with safety controls
- Create Electron project structure
- Port React UI from
actual ui/vocal-muse-sidecar-main - Implement WebSocket server (port 8765)
- Add connection status UI
- Build safety controls:
- AI Pause / Kill Switch button
- ACL configuration panel
- Rate limiting engine
- Command validation framework
- Add settings persistence
- Test AI chat works standalone (without DAW)
Goal: Basic Ableton Live control
- Create Remote Script folder structure
- Implement WebSocket client in Python
- Add automatic reconnection logic
- Implement command dispatcher
- Add basic command handlers:
- Transport control (play/stop/tempo)
- Clip launching
- Track mute/solo/arm
- Send snapshot on connection
- Add error handling and logging
- Test with Electron app
Goal: Real-time state synchronization
- Register observers for:
- Tempo changes
- Transport state
- Track list changes
- Clip slots
- Parameter changes
- Implement user touch detection
- Add position update (throttled to 10Hz)
- Test state sync during live editing
Goal: Full DAW control
- Track creation/deletion
- MIDI clip creation
- Add notes to clips
- Device parameter control
- Scene creation/launching
- Volume/pan control
- Test complex workflows
Goal: Robust, production-ready
- Comprehensive error messages
- Undo stack implementation
- Dry run / preview mode
- Connection status heartbeat
- State desync detection & recovery
- Performance optimization
- User documentation
- Video tutorials
Goal: Easy installation
- Electron app installer (.exe for Windows)
- Auto-updater integration
- Remote Script installer script
- Troubleshooting guide
- Beta testing with real users
- Bug fixes based on feedback
C:\wingman\
├── CLAUDE.md # This file
├── CMakeLists.txt # Legacy VST3 build config
├── JUCE/ # JUCE 8.0.9 (legacy)
├── Source/ # Legacy WebView2 VST3 code
├── actual ui/vocal-muse-sidecar-main/ # React UI (will be ported)
└── build/ # Legacy VST3 build output
C:\wingman\
├── CLAUDE.md # This file
│
├── wingman-app/ # Electron standalone app
│ ├── package.json
│ ├── electron-builder.json # Installer config
│ ├── src/
│ │ ├── main/ # Electron main process
│ │ │ ├── index.ts # App entry, window management
│ │ │ ├── websocketServer.ts # WebSocket server
│ │ │ ├── safetyManager.ts # ACL, rate limiting, kill switch
│ │ │ ├── commandValidator.ts # Command validation
│ │ │ └── stateManager.ts # DAW state synchronization
│ │ ├── renderer/ # React UI
│ │ │ ├── App.tsx # Main app component
│ │ │ ├── components/
│ │ │ │ ├── WingmanMinimal.tsx # Main UI
│ │ │ │ ├── SafetyControls.tsx # Kill switch, ACL panel
│ │ │ │ ├── ConnectionStatus.tsx
│ │ │ │ └── UndoPanel.tsx
│ │ │ └── hooks/
│ │ │ ├── useWebSocket.ts # WebSocket hook
│ │ │ └── useDAWState.ts # DAW state hook
│ │ └── preload/
│ │ └── preload.ts # Context bridge
│ ├── .env # API keys
│ └── build/ # Build output
│ └── Wingman Setup.exe
│
├── remote-script/ # Ableton Python control surface
│ └── WingmanControl/
│ ├── __init__.py # Entry point
│ ├── WingmanControl.py # Main Remote Script class
│ ├── websocket_client.py # WebSocket connection
│ ├── command_handler.py # Command dispatcher
│ ├── observer_manager.py # LOM observers
│ ├── state_snapshot.py # Snapshot generator
│ └── config.py # Configuration
│
└── installer/ # Installation tools
├── install_remote_script.py # Auto-install script
└── README.md # Setup instructions
Electron App:
C:\Program Files\Wingman\Wingman.exe
C:\Users\[User]\AppData\Roaming\Wingman\config.json
C:\Users\[User]\AppData\Roaming\Wingman\logs\
Remote Script:
C:\Users\[User]\Documents\Ableton\User Library\Remote Scripts\WingmanControl\
Logs:
C:\Users\[User]\AppData\Roaming\Wingman\logs\app.log
C:\Users\[User]\AppData\Roaming\Ableton\Live X.X\Preferences\Log.txt (Remote Script)
# AI Providers
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...
PERPLEXITY_API_KEY=pplx-...
OPENROUTER_API_KEY=sk-or-...
# WebSocket Configuration
WEBSOCKET_PORT=8765
WEBSOCKET_HOST=localhost
# Safety Defaults
DEFAULT_RATE_LIMIT=10
DEFAULT_ACL_DESTRUCTIVE=false
# Development
NODE_ENV=production
LOG_LEVEL=infoDevelopment:
cd C:\wingman\wingman-app
npm install
npm run dev # Run with hot reloadProduction Build:
cd C:\wingman\wingman-app
npm run build # Build renderer (Vite)
npm run compile # Build main (TypeScript)
npm run dist # Create installerOutput: C:\wingman\wingman-app\build\Wingman Setup.exe
Automated:
cd C:\wingman\installer
python install_remote_script.pyManual:
xcopy "C:\wingman\remote-script\WingmanControl" "%USERPROFILE%\Documents\Ableton\User Library\Remote Scripts\WingmanControl" /E /I /YEnable in Ableton:
- Open Ableton Live preferences
- Go to Link/Tempo/MIDI tab
- In Control Surface dropdown, select "WingmanControl"
- Restart Ableton Live
Reliability: TCP ensures ordered, guaranteed delivery - critical for control messages Simplicity: Single connection instead of multiple UDP ports Error Handling: Built-in connection state management Debugging: Easy to inspect with browser dev tools or Wireshark Firewall: More likely to work through firewalls than UDP
Performance: WebSocket latency (~1-2ms local) is acceptable for control messages (not audio)
Remote Script is Sufficient: LOM provides complete DAW control Complexity Reduction: Fewer components = easier to maintain Cross-Platform: No C++ compilation issues Safer: Runs in separate process, can't crash DAW
Future Consideration: Could add VST3 for visual integration (status widget in DAW), but not for control
Ableton's Reality: Live uses indices internally, fighting this creates problems Observer Pattern: Detect changes in real-time rather than maintain fragile mappings Simplicity: No UID generation/translation logic Performance: No hashing overhead
User Impact: App UI shows track names, AI refers to tracks by name, but commands use validated indices
Live 11+ Requirement: Must use Python 3 syntax
Limited Standard Library: Ableton strips out many modules for safety
Available Modules: socket, threading, json, re, math, time
Not Available: subprocess, urllib, http, msvcrt, win32api
WebSocket Solution: Use socket module directly with WebSocket protocol implementation
- Check Node.js installed:
node --version(v18+ required) - Check port 8765 not in use:
netstat -ano | findstr 8765 - Check firewall allows localhost connections
- Check logs:
%APPDATA%\Wingman\logs\app.log - Try running from terminal to see errors:
Wingman.exe
- Verify installed in correct folder:
%USERPROFILE%\Documents\Ableton\User Library\Remote Scripts\WingmanControl\ - Check selected in Live preferences (Link/Tempo/MIDI → Control Surface)
- Check Python errors in Live's Log.txt:
%APPDATA%\Ableton\Live X.X\Preferences\Log.txt - Verify Wingman app is running first (script connects to app)
- Restart Ableton Live after installing script
- Ensure Wingman app started before Ableton Live
- Check firewall not blocking localhost:8765
- Try restarting both app and Live
- Check app logs for WebSocket errors
- Verify no other program using port 8765
- Check connection status is "Connected" in app
- Verify ACL permissions enabled for command type (Settings → Safety)
- Check AI not paused (red "PAUSE AI" button should not be active)
- Try simple command first: "play the transport"
- Check Live's Log.txt for Python errors
- Verify command targets valid track/clip (check indices)
- Verify connection status is "Connected"
- Check "Last Update" timestamp in app (should update in real-time)
- Try requesting snapshot manually (Settings → Refresh State)
- Check app logs for event parsing errors
- Verify observers are registered (check Live's Log.txt on startup)
- Check event rate throttling (Settings → Performance)
- Disable position updates if not needed (high frequency)
- Reduce AI command rate limit (Settings → Safety)
- Check system resources (Electron app memory usage)
- Clear undo history if very large (> 1000 actions)
- Command Latency: < 20ms (app → script → DAW execution)
- Event Rate:
- Transport: 10 Hz (position updates)
- Parameters: 20 Hz max per parameter
- Other events: Instant (no throttling)
- WebSocket Overhead: < 5ms for round-trip localhost communication
- Snapshot Generation: < 100ms for typical set (50 tracks, 200 clips)
- Memory: < 200 MB for Electron app, < 20 MB for Remote Script
- WebSocket binds to
localhostonly (not 0.0.0.0) - No external network access required
- All data stays on local machine
- Stored in
.envfile (not checked into git) - Electron app uses
dotenvto load securely - Never sent to Remote Script or exposed in UI
- All AI commands validated before execution
- Destructive operations disabled by default
- User can audit all AI actions in undo stack
- "Dry run" mode for testing without executing
- No analytics or tracking
- No crash reporting to external servers
- Logs stored locally only
- Currently only works with Ableton Live 11+ (Python 3)
- No support for other DAWs yet
- Future: Could adapt Remote Script pattern to other DAWs with scripting APIs
- Cannot read audio file waveforms (would require Max for Live)
- Cannot analyze audio in real-time (frequency spectrum, etc.)
- Can only control parameters, not process audio
- Cannot use most Python standard library (Ableton limitation)
- Cannot spawn subprocesses or access Win32 API
- Limited to modules available in Live's Python build
- Initial version Windows only
- Mac support planned after Windows stabilized
- Linux not supported (Electron works, but Ableton is Windows/Mac only)
- Mac support (test Remote Script on Mac Live)
- Advanced MIDI generation (using Magenta, transformers)
- Preset library (common AI commands saved as templates)
- Multi-language support (UI localization)
- Dark/light theme toggle
- Max for Live device (audio analysis, waveform display)
- VST3 widget (show connection status in DAW, optional)
- Cloud sync for presets and settings
- Collaborative sessions (multiple users controlling one DAW)
- Voice control integration (speech-to-text commands)
- Support for other DAWs (FL Studio, Logic Pro, Cubitase, Studio One)
- Advanced AI features (style transfer, arrangement suggestions)
- Marketplace for community presets and templates
- Mobile app (control DAW from phone/tablet)
- Plugin parameter learning (AI watches user, suggests improvements)
Current Branch: claude/code-review-analysis-011CUyxzyvT4tboSrg6C4PEy
Recent Commits:
c77d2a0 Merge pull request #6 (project review feedback)
8cd7fa2 Merge pull request #5 (investigate p variable)
a8a09da Merge pull request #4 (check p)
2579d1d feat: Implement per-chat MIDI isolation and auto-features
c39d3f2 docs: Add comprehensive feature verification reports
- Developer: Sylor
- Project Path:
C:\wingman\ - Logs:
%APPDATA%\Wingman\logs\ - Target Platform: Windows 10/11 (Mac later)
- Node Version: 18+ LTS
- Ableton Version: Live 11+ (Python 3)
A: Yes, the Electron app must run alongside Ableton Live for the Remote Script to connect.
A: Not yet. The Remote Script is Ableton-specific. Future versions may support other DAWs.
A: Safe by default. Destructive operations (track/clip deletion) are disabled until you explicitly enable them in Settings → Safety.
A: Hit the red "PAUSE AI" button immediately. Then use "Undo Last AI Action" to revert. You can also use Ableton's native undo (Ctrl+Z).
A: No. All communication is local (Electron app ↔ Remote Script). AI API calls (OpenAI, etc.) only send text prompts, never audio.
A: VST3 plugins can't control the full DAW - they only see their own track. Ableton's Remote Script API provides complete access.
A: Yes, but test thoroughly first. Use "Dry run" mode to preview AI actions. Consider disabling destructive operations during live sets.
A: Playback continues normally. The app will show "Disconnected" and attempt to reconnect every 2 seconds. No data is lost.
Last Updated: 2025-11-10 Status: Simplified architecture - Electron App + Remote Script only Next Step: Week 1-2 - Foundation & Safety implementation Estimated Completion: 6-8 weeks from start date