feat: add multi-provider agents (claude/codex/gemini) with robust tracking and UI

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## Next
+
+### Features
+
+- Add multi-provider interactive agent support (Claude, Codex, Gemini) with mixed-provider agents in a single workspace.
+- Add provider detection and persisted provider preferences (`defaultProvider`, `askEachTime`).
+- Add `openAgent` message flow while keeping `openClaude` as a backward-compatible alias.
+- Include provider metadata in `agentCreated` and `existingAgents` payloads.
+
+### Migration Notes
+
+- Existing persisted agents without `provider` are restored as `claude`.
+- Claude `/clear` rollover behavior is preserved.
+- Codex and Gemini use provider-specific session file resolution and best-effort tool/turn parsing.
+
 ## v1.0.2
 
 ### Bug Fixes

README.md

@@ -2,7 +2,7 @@
 
 A VS Code extension that turns your AI coding agents into animated pixel art characters in a virtual office.
 
-Each Claude Code terminal you open spawns a character that walks around, sits at desks, and visually reflects what the agent is doing — typing when writing code, reading when searching files, waiting when it needs your attention.
+Each interactive AI terminal you open (Claude Code, Codex, or Gemini) spawns a character that walks around, sits at desks, and visually reflects what the agent is doing — typing when writing code, reading when searching files, waiting when it needs your attention.
 
 This is the source code for the free [Pixel Agents extension for VS Code](https://marketplace.visualstudio.com/items?itemName=pablodelucca.pixel-agents) — you can install it directly from the marketplace with the full furniture catalog included.
 
@@ -11,7 +11,8 @@ This is the source code for the free [Pixel Agents extension for VS Code](https:
 
 ## Features
 
-- **One agent, one character** — every Claude Code terminal gets its own animated character
+- **One agent, one character** — every Claude Code, Codex, or Gemini terminal gets its own animated character
+- **Mixed providers in one workspace** — run Claude, Codex, and Gemini agents side by side in the same panel
 - **Live activity tracking** — characters animate based on what the agent is actually doing (writing, reading, running commands)
 - **Office layout editor** — design your office with floors, walls, and furniture using a built-in editor
 - **Speech bubbles** — visual indicators when an agent is waiting for input or needs permission
@@ -27,7 +28,10 @@ This is the source code for the free [Pixel Agents extension for VS Code](https:
 ## Requirements
 
 - VS Code 1.109.0 or later
-- [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code) installed and configured
+- At least one supported CLI installed and configured:
+  - [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code)
+  - Codex CLI
+  - Gemini CLI
 
 ## Getting Started
 
@@ -48,8 +52,9 @@ Then press **F5** in VS Code to launch the Extension Development Host.
 ### Usage
 
 1. Open the **Pixel Agents** panel (it appears in the bottom panel area alongside your terminal)
-2. Click **+ Agent** to spawn a new Claude Code terminal and its character
-3. Start coding with Claude — watch the character react in real time
+2. Click **+ Agent** to spawn a new agent terminal and its character
+3. Pick a provider (Claude/Codex/Gemini) if prompted, or use your configured default provider
+4. Start coding with your agent and watch the character react in real time
 4. Click a character to select it, then click a seat to reassign it
 5. Click **Layout** to open the office editor and customize your space
 
@@ -81,7 +86,7 @@ The extension will still work without the tileset — you'll get the default cha
 
 ## How It Works
 
-Pixel Agents watches Claude Code's JSONL transcript files to track what each agent is doing. When an agent uses a tool (like writing a file or running a command), the extension detects it and updates the character's animation accordingly. No modifications to Claude Code are needed — it's purely observational.
+Pixel Agents watches provider session transcripts to track what each agent is doing. For Claude and Codex it reads JSONL logs; for Gemini it reads session JSON snapshots. When an agent uses a tool (like writing a file or running a command), the extension detects it and updates the character's animation accordingly. No CLI modifications are required — it's purely observational.
 
 The webview runs a lightweight game loop with canvas rendering, BFS pathfinding, and a character state machine (idle → walk → type/read). Everything is pixel-perfect at integer zoom levels.
 
@@ -92,8 +97,9 @@ The webview runs a lightweight game loop with canvas rendering, BFS pathfinding,
 
 ## Known Limitations
 
-- **Agent-terminal sync** — the way agents are connected to Claude Code terminal instances is not super robust and sometimes desyncs, especially when terminals are rapidly opened/closed or restored across sessions.
-- **Heuristic-based status detection** — Claude Code's JSONL transcript format does not provide clear signals for when an agent is waiting for user input or when it has finished its turn. The current detection is based on heuristics (idle timers, turn-duration events) and often misfires — agents may briefly show the wrong status or miss transitions.
+- **Agent-terminal sync** — the way agents are connected to interactive terminal instances is not super robust and sometimes desyncs, especially when terminals are rapidly opened/closed or restored across sessions.
+- **Provider log format churn** — status/tool tracking depends on local CLI log formats, which may change without notice and require parser updates.
+- **Feature parity across providers** — Claude currently has the richest permission/sub-agent tracking. Codex and Gemini use best-effort parsing where equivalent signals are unavailable.
 - **Windows-only testing** — the extension has only been tested on Windows 11. It may work on macOS or Linux, but there could be unexpected issues with file watching, paths, or terminal behavior on those platforms.
 
 ## Roadmap

src/PixelAgentsViewProvider.ts

@@ -3,6 +3,7 @@ import * as os from 'os';
 import * as path from 'path';
 import * as vscode from 'vscode';
 import type { AgentState } from './types.js';
+import type { AgentProvider } from './types.js';
 import {
 	launchNewTerminal,
 	removeAgent,
@@ -14,9 +15,16 @@ import {
 } from './agentManager.js';
 import { ensureProjectScan } from './fileWatcher.js';
 import { loadFurnitureAssets, sendAssetsToWebview, loadFloorTiles, sendFloorTilesToWebview, loadWallTiles, sendWallTilesToWebview, loadCharacterSprites, sendCharacterSpritesToWebview, loadDefaultLayout } from './assetLoader.js';
-import { WORKSPACE_KEY_AGENT_SEATS, GLOBAL_KEY_SOUND_ENABLED } from './constants.js';
+import {
+	WORKSPACE_KEY_AGENT_SEATS,
+	GLOBAL_KEY_SOUND_ENABLED,
+	GLOBAL_KEY_DEFAULT_PROVIDER,
+	GLOBAL_KEY_ASK_PROVIDER_EACH_TIME,
+	WORKSPACE_KEY_PROVIDER_PREFERENCE_SET,
+} from './constants.js';
 import { writeLayoutToFile, readLayoutFromFile, watchLayoutFile } from './layoutPersistence.js';
 import type { LayoutWatcher } from './layoutPersistence.js';
+import { detectInstalledProviders, getRecommendedProvider } from './providers.js';
 
 export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 	nextAgentId = { current: 1 };
@@ -34,6 +42,7 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 	// /clear detection: project-level scan for new JSONL files
 	activeAgentId = { current: null as number | null };
 	knownJsonlFiles = new Set<string>();
+	claimedSessionFiles = new Set<string>();
 	projectScanTimer = { current: null as ReturnType<typeof setInterval> | null };
 
 	// Bundled default layout (loaded from assets/default-layout.json)
@@ -56,21 +65,73 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 		persistAgents(this.agents, this.context);
 	};
 
+	private getProviderPreference(): { defaultProvider: AgentProvider | null; askEachTime: boolean; preferenceSet: boolean } {
+		const installed = detectInstalledProviders();
+		const recommended = getRecommendedProvider(installed);
+		const preferenceSet = this.context.workspaceState.get<boolean>(WORKSPACE_KEY_PROVIDER_PREFERENCE_SET, false);
+		const saved = this.context.globalState.get<AgentProvider | null>(GLOBAL_KEY_DEFAULT_PROVIDER, null);
+		const savedAskEachTime = this.context.globalState.get<boolean>(GLOBAL_KEY_ASK_PROVIDER_EACH_TIME, false);
+		const installedCount = Object.values(installed).filter(Boolean).length;
+		// First-run behavior: if multiple providers are installed and preference is unset, ask each time.
+		const askEachTime = preferenceSet ? savedAskEachTime : (savedAskEachTime || installedCount > 1);
+
+		let defaultProvider = saved;
+		if (!defaultProvider || !installed[defaultProvider]) {
+			defaultProvider = recommended;
+		}
+		return { defaultProvider, askEachTime, preferenceSet };
+	}
+
+	private ensureDefaultProviderInitialized(): void {
+		const installed = detectInstalledProviders();
+		const saved = this.context.globalState.get<AgentProvider | null>(GLOBAL_KEY_DEFAULT_PROVIDER, null);
+		if (saved && installed[saved]) return;
+		const recommended = getRecommendedProvider(installed);
+		if (recommended) {
+			this.context.globalState.update(GLOBAL_KEY_DEFAULT_PROVIDER, recommended);
+		}
+	}
+
 	resolveWebviewView(webviewView: vscode.WebviewView) {
 		this.webviewView = webviewView;
 		webviewView.webview.options = { enableScripts: true };
 		webviewView.webview.html = getWebviewContent(webviewView.webview, this.extensionUri);
 
 		webviewView.webview.onDidReceiveMessage(async (message) => {
-			if (message.type === 'openClaude') {
+			if (message.type === 'openClaude' || message.type === 'openAgent') {
+				const installed = detectInstalledProviders();
+				const pref = this.getProviderPreference();
+				let provider = (message.provider as AgentProvider | undefined);
+				if (!provider) {
+					provider = pref.defaultProvider || getRecommendedProvider(installed) || 'claude';
+				}
+				if (!installed[provider]) {
+					vscode.window.showErrorMessage(`Pixel Agents: ${provider} CLI is not installed.`);
+					return;
+				}
 				await launchNewTerminal(
 					this.nextAgentId, this.nextTerminalIndex,
 					this.agents, this.activeAgentId, this.knownJsonlFiles,
+					this.claimedSessionFiles,
 					this.fileWatchers, this.pollingTimers, this.waitingTimers, this.permissionTimers,
 					this.jsonlPollTimers, this.projectScanTimer,
 					this.webview, this.persistAgents,
+					provider,
 					message.folderPath as string | undefined,
 				);
+			} else if (message.type === 'saveProviderPreference') {
+				const provider = message.defaultProvider as AgentProvider | null;
+				const askEachTime = !!message.askEachTime;
+				if (provider) {
+					this.context.globalState.update(GLOBAL_KEY_DEFAULT_PROVIDER, provider);
+				}
+				this.context.globalState.update(GLOBAL_KEY_ASK_PROVIDER_EACH_TIME, askEachTime);
+				this.context.workspaceState.update(WORKSPACE_KEY_PROVIDER_PREFERENCE_SET, true);
+				this.webview?.postMessage({
+					type: 'providerPreferenceSaved',
+					defaultProvider: provider,
+					askEachTime,
+				});
 			} else if (message.type === 'focusAgent') {
 				const agent = this.agents.get(message.id);
 				if (agent) {
@@ -91,17 +152,33 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 			} else if (message.type === 'setSoundEnabled') {
 				this.context.globalState.update(GLOBAL_KEY_SOUND_ENABLED, message.enabled);
 			} else if (message.type === 'webviewReady') {
+				this.ensureDefaultProviderInitialized();
+				const installed = detectInstalledProviders();
+				const recommended = getRecommendedProvider(installed);
 				restoreAgents(
 					this.context,
 					this.nextAgentId, this.nextTerminalIndex,
 					this.agents, this.knownJsonlFiles,
+					this.claimedSessionFiles,
 					this.fileWatchers, this.pollingTimers, this.waitingTimers, this.permissionTimers,
 					this.jsonlPollTimers, this.projectScanTimer, this.activeAgentId,
 					this.webview, this.persistAgents,
 				);
 				// Send persisted settings to webview
 				const soundEnabled = this.context.globalState.get<boolean>(GLOBAL_KEY_SOUND_ENABLED, true);
 				this.webview?.postMessage({ type: 'settingsLoaded', soundEnabled });
+				const pref = this.getProviderPreference();
+				this.webview?.postMessage({
+					type: 'providersDetected',
+					installed,
+					recommended,
+				});
+				this.webview?.postMessage({
+					type: 'providerPreferenceLoaded',
+					defaultProvider: pref.defaultProvider,
+					askEachTime: pref.askEachTime,
+					preferenceSet: pref.preferenceSet,
+				});
 
 				// Send workspace folders to webview (only when multi-root)
 				const wsFolders = vscode.workspace.workspaceFolders;
@@ -113,8 +190,8 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 				}
 
 				// Ensure project scan runs even with no restored agents (to adopt external terminals)
-				const projectDir = getProjectDirPath();
 				const workspaceRoot = vscode.workspace.workspaceFolders?.[0]?.uri.fsPath;
+				const projectDir = getProjectDirPath('claude', workspaceRoot);
 				console.log('[Extension] workspaceRoot:', workspaceRoot);
 				console.log('[Extension] projectDir:', projectDir);
 				if (projectDir) {
@@ -225,7 +302,7 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 				}
 				sendExistingAgents(this.agents, this.context, this.webview);
 			} else if (message.type === 'openSessionsFolder') {
-				const projectDir = getProjectDirPath();
+				const projectDir = getProjectDirPath('claude', vscode.workspace.workspaceFolders?.[0]?.uri.fsPath);
 				if (projectDir && fs.existsSync(projectDir)) {
 					vscode.env.openExternal(vscode.Uri.file(projectDir));
 				}
@@ -286,6 +363,7 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 					}
 					removeAgent(
 						id, this.agents,
+						this.claimedSessionFiles,
 						this.fileWatchers, this.pollingTimers, this.waitingTimers, this.permissionTimers,
 						this.jsonlPollTimers, this.persistAgents,
 					);
@@ -327,6 +405,7 @@ export class PixelAgentsViewProvider implements vscode.WebviewViewProvider {
 		for (const id of [...this.agents.keys()]) {
 			removeAgent(
 				id, this.agents,
+				this.claimedSessionFiles,
 				this.fileWatchers, this.pollingTimers, this.waitingTimers, this.permissionTimers,
 				this.jsonlPollTimers, this.persistAgents,
 			);