Default to an installed build, and report missing video tooling

Three problems surfaced running the skill from a fresh checkout.

Nothing ran without a target. With no flags the runner used the build from the
checkout, which only exists after compiling the product, so the documented
starting point failed to launch. Reproducing a reported issue means running the
shipped product anyway, so with no target flag it now finds an installed VS Code
Insiders (falling back to Stable) and logs which one it chose. `--dev` selects the
checkout build, and `--build` still pins an exact install.

A missing ffmpeg was only discovered after the run, as a raw ENOENT, and it threw
out of `runScenario` after the report had been written. The runner now checks for
ffmpeg and ffprobe before launching anything and prints the install command for
the platform, and caption rendering can no longer fail a run that has already
produced its evidence.

The window did not fill the recording. The canvas is 1920x1080 while VS Code
sizes its own window (1440x900 with a workspace, 1200x800 empty), so the capture
showed the window in the top-left corner surrounded by dead space. The window is
now sized to the canvas once recording is on; a window larger than the display
still renders at that size, so this holds on smaller screens.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: adb443eb-11e5-40a1-8608-7f593fa79485

Author: bryanchen-d

.github/skills/validate-ui-scenario/SKILL.md

@@ -18,21 +18,34 @@ step boundary, writes the report, and captions the recording with each step and
 ## Prepare
 
 ```bash
-npm install                        # once
+npm install                             # once
 npm --prefix test/scenario run compile  # after any change under test/scenario
 ```
 
-Add `ffmpeg` and `ffprobe` to `PATH` to get the caption band on the video. Without them the run still
-succeeds and the raw recording is kept.
-
-| Target | Extra flags | Also required | Use for |
-|--------|-------------|---------------|---------|
-| Installed Insiders | `--build <app-root>` | nothing | Reproducing a report against shipped behavior |
-| Dev build from this checkout | *(none)* | `npm run electron`, `npm run transpile-client` | Verifying an unmerged change |
+**Check `ffmpeg` and `ffprobe` are on `PATH` before running.** Without them the scenario still runs
+and keeps the raw recording, but the video is not captioned with step titles. The runner warns at
+startup; if they are missing, tell the user how to install them rather than silently returning an
+unannotated video:
+
+| Platform | Install |
+|----------|---------|
+| Windows | `winget install Gyan.FFmpeg` |
+| macOS | `brew install ffmpeg` |
+| Linux | `sudo apt install ffmpeg` |
+
+A new terminal may be needed for `PATH` to pick them up. An existing run can be annotated afterwards
+with `node test/scenario/out/renderEvidenceChapters.js <run-dir>`.
+
+| Target | Flags | Also required | Use for |
+|--------|-------|---------------|---------|
+| Installed Insiders, else Stable | *(none — the default)* | nothing | Reproducing a report against shipped behavior |
+| Dev build from this checkout | `--dev` | `npm run electron`, `npm run transpile-client` | Verifying an unmerged change |
+| A specific install | `--build <app-root>` | nothing | Pinning an exact build |
 | Web | `--web --headless` | `npm run transpile-client` | Browser-only behavior |
 
-`--build` takes the application root — the install directory on Windows and Linux, or the `.app`
-bundle on macOS:
+With no target flag the runner finds an installed VS Code Insiders (falling back to Stable) and logs
+which one it chose. `--build` takes the application root — the install directory on Windows and
+Linux, or the `.app` bundle on macOS:
 
 ```bash
 # Windows
@@ -41,9 +54,10 @@ bundle on macOS:
 --build "/Applications/Visual Studio Code - Insiders.app"
 ```
 
-An installed build runs with its own profile and extensions directory, so your extensions and
-settings never leak into the recording. Insiders only reproduces **shipped** behavior — to validate
-an unmerged change, run the dev build from a checkout that contains it.
+Every target runs with its own profile and extensions directory, so your extensions and settings
+never leak into the recording, and the window is sized to the recording canvas so the capture has no
+empty margins. An installed build only reproduces **shipped** behavior — to validate an unmerged
+change, use `--dev` in a checkout that contains it.
 
 ## Write the scenario
 
@@ -130,7 +144,7 @@ Each step receives a `context` with `app`, `workbench`, `code`, `page`, and `ski
 ## Run it
 
 ```bash
-node test/scenario/out/runScenario.js <scenario.cjs> --build "<app-root>"
+node test/scenario/out/runScenario.js <scenario.cjs>
 ```
 
 Exit code `0` means every step passed, `1` means the run failed or was aborted, `2` a usage error.
@@ -181,16 +195,17 @@ the issue or pull request by dragging it into the comment box.
   skill when a scenario is not yet covered there, or to iterate locally before proposing one.
 
 <example>
-User: "/validate-ui-scenario reproduce https://github.com/microsoft/vscode/issues/250159 against my
-installed VS Code Insiders, and give me the report and the annotated video."
+User: "/validate-ui-scenario reproduce https://github.com/microsoft/vscode/issues/250159"
 
-1. Read the issue and identify the observable claim: searching `chat confirm` in the Settings editor
+1. Confirm `ffmpeg`/`ffprobe` are available; if not, say so and give the install command before
+   running, so the user is not surprised by a video without step titles.
+2. Read the issue and identify the observable claim: searching `chat confirm` in the Settings editor
    should match **Max Requests**, whose description mentions confirmation.
-2. Add a baseline step (`max requests` finds the setting) so a failure cannot be explained by the
+3. Add a baseline step (`max requests` finds the setting) so a failure cannot be explained by the
    setting being missing from the build.
-3. Write `.build/vscode-playwright-mcp/issue-250159.cjs`, run it with `--build`, and read the
-   printed report path.
-4. Report the outcome per step, link `report.html`, and attach `videos/annotated.mp4`.
+4. Write `.build/vscode-playwright-mcp/issue-250159.cjs` and run it with no target flag, which uses
+   the installed Insiders; read the printed report path.
+5. Report the outcome per step, link `report.html`, and attach `videos/annotated.mp4`.
 
 The run fails at the search step, and that is the answer: the issue reproduces. Report it as a
 successful reproduction, not as a broken scenario.

test/automation/src/playwrightElectron.ts

@@ -37,6 +37,12 @@ export async function launch(options: LaunchOptions): Promise<{ electronProcess:
 async function launchElectron(configuration: IElectronConfiguration, options: LaunchOptions) {
 	const { logger, tracing, snapshots } = options;
 
+	// The recording canvas is fixed, but VS Code sizes its own window (1440x900
+	// with a workspace, 1200x800 empty), so the capture would otherwise show the
+	// window in the top-left corner of a larger frame. The window is resized to
+	// match below, which also renders reliably when it is larger than the screen.
+	const videoSize = { width: 1920, height: 1080 };
+
 	const playwrightImpl = options.playwright ?? playwright;
 	let electron;
 	try {
@@ -46,7 +52,7 @@ async function launchElectron(configuration: IElectronConfiguration, options: La
 			recordVideo: options.videosPath
 				? {
 					dir: options.videosPath,
-					size: { width: 1920, height: 1080 }
+					size: videoSize
 				} : undefined,
 			env: configuration.env as { [key: string]: string },
 			timeout: LAUNCH_TIMEOUT
@@ -63,6 +69,18 @@ async function launchElectron(configuration: IElectronConfiguration, options: La
 			throw enrichLaunchError(error, options);
 		}
 	}
+	if (options.videosPath) {
+		try {
+			await electron.evaluate(({ BrowserWindow }, size) => {
+				const target = BrowserWindow.getAllWindows()[0];
+				target?.setBounds({ x: 0, y: 0, width: size.width, height: size.height });
+			}, videoSize);
+		} catch (error) {
+			// A mismatched window only wastes pixels in the recording, so never fail
+			// a run because the window could not be resized.
+			logger.log(`Playwright (Electron): Failed to size the window to the recording (${error})`);
+		}
+	}
 	// Recording is per page, so sample the origin once the first window exists
 	// rather than when the application finished launching.
 	const videoStartedAt = options.videosPath ? Date.now() : undefined;

test/scenario/src/application.ts

@@ -95,10 +95,61 @@ function parseQuality(): Quality {
 //
 // #### Electron ####
 //
+/**
+ * Locate an installed VS Code Insiders, then Stable.
+ *
+ * Reproducing a reported issue is the common case, and that means running the
+ * shipped product rather than a build from this checkout, so an installed build
+ * is used when the caller did not choose a target.
+ */
+function findInstalledBuild(): string | undefined {
+	const candidates: string[] = [];
+	switch (process.platform) {
+		case 'win32': {
+			const roots = [process.env.LOCALAPPDATA, process.env.ProgramFiles, process.env['ProgramFiles(x86)']].filter((root): root is string => !!root);
+			for (const root of roots) {
+				candidates.push(path.join(root, 'Programs', 'Microsoft VS Code Insiders'), path.join(root, 'Microsoft VS Code Insiders'));
+			}
+			for (const root of roots) {
+				candidates.push(path.join(root, 'Programs', 'Microsoft VS Code'), path.join(root, 'Microsoft VS Code'));
+			}
+			break;
+		}
+		case 'darwin':
+			candidates.push(
+				'/Applications/Visual Studio Code - Insiders.app',
+				path.join(os.homedir(), 'Applications', 'Visual Studio Code - Insiders.app'),
+				'/Applications/Visual Studio Code.app',
+				path.join(os.homedir(), 'Applications', 'Visual Studio Code.app')
+			);
+			break;
+		default:
+			candidates.push('/usr/share/code-insiders', '/opt/visual-studio-code-insiders', '/usr/share/code', '/opt/visual-studio-code');
+			break;
+	}
+	return candidates.find(candidate => {
+		try {
+			return fs.existsSync(candidate) && fs.existsSync(getBuildElectronPath(candidate));
+		} catch {
+			return false; // an incomplete install is not a usable target
+		}
+	});
+}
+
 if (!opts.web) {
 	let testCodePath = opts.build;
 	let electronPath: string | undefined;
 
+	if (!testCodePath && !opts.dev) {
+		testCodePath = findInstalledBuild();
+		if (testCodePath) {
+			// `getApplication` launches whatever `opts.build` names, so record the
+			// choice there rather than only in this block.
+			opts.build = testCodePath;
+			logger.log(`No target given, using the installed build at ${testCodePath}. Pass --dev to run this checkout instead.`);
+		}
+	}
+
 	if (testCodePath) {
 		electronPath = getBuildElectronPath(testCodePath);
 		version = getBuildVersion(testCodePath);
@@ -111,7 +162,7 @@ if (!opts.web) {
 	}
 
 	if (!fs.existsSync(electronPath || '')) {
-		fail(`Cannot find VSCode at ${electronPath}. Please run VSCode once first (scripts/code.sh, scripts\\code.bat) and try again.`);
+		fail(`Cannot find VS Code at ${electronPath}. Install VS Code Insiders, pass --build <app-root>, or build this checkout and pass --dev.`);
 	}
 
 	quality = parseQuality();

test/scenario/src/options.ts

@@ -20,7 +20,8 @@ export const opts = minimist(args, {
 		'web',
 		'headless',
 		'video',
-		'autostart'
+		'autostart',
+		'dev'
 	],
 	default: {
 		verbose: false
@@ -31,6 +32,7 @@ export const opts = minimist(args, {
 	headless?: boolean;
 	web?: boolean;
 	build?: string;
+	dev?: boolean;
 	browser?: 'chromium' | 'webkit' | 'firefox' | 'chromium-msedge' | 'chromium-chrome' | undefined;
 	electronArgs?: string;
 	video?: boolean;

test/scenario/src/renderEvidenceChapters.ts

@@ -299,12 +299,24 @@ function wrap(value: string, limit: number, maxLines: number): string[] {
 	return lines;
 }
 
-if (require.main === module) {
+/**
+ * Render captions without letting a presentation step fail a validation run.
+ *
+ * The raw recording is authoritative, so a missing or failing ffmpeg is reported
+ * and otherwise ignored.
+ */
+export function tryRenderChapters(runRoot: string): void {
 	try {
-		renderChapters(path.resolve(process.argv[2] ?? process.env.RUN_ROOT ?? '.'));
+		renderChapters(runRoot);
 	} catch (error) {
-		// Captions are a presentation aid, so never fail a validation run because
-		// the recording could not be annotated. The raw recording is authoritative.
-		console.warn(`Unable to render evidence captions: ${error instanceof Error ? error.message : error}`);
+		const message = error instanceof Error ? error.message : String(error);
+		const missingTool = /spawnSync (ffmpeg|ffprobe) ENOENT/u.exec(message);
+		console.warn(missingTool
+			? `Unable to render evidence captions: ${missingTool[1]} is not on PATH. Install ffmpeg and re-run this command; the raw recording is unaffected.`
+			: `Unable to render evidence captions: ${message}`);
 	}
 }
+
+if (require.main === module) {
+	tryRenderChapters(path.resolve(process.argv[2] ?? process.env.RUN_ROOT ?? '.'));
+}

test/scenario/src/runScenario.ts

@@ -3,12 +3,45 @@
  *  Licensed under the MIT License. See License.txt in the project root for license information.
  *--------------------------------------------------------------------------------------------*/
 
+import { execFileSync } from 'child_process';
 import type { Page } from '@playwright/test';
 import * as path from 'path';
 import type { Application, Code, Workbench } from '../../automation';
 import { ApplicationService, JSONValue } from './application';
 import { EvidenceService } from './evidence';
-import { renderChapters } from './renderEvidenceChapters';
+import { tryRenderChapters } from './renderEvidenceChapters';
+
+/**
+ * Report missing video tooling before anything is launched.
+ *
+ * Captions are rendered after the run, so a missing ffmpeg is only discovered
+ * once the scenario has already finished. Say so up front, with the command to
+ * fix it, rather than letting the run complete and produce no annotated video.
+ */
+function checkVideoTooling(): void {
+	const missing = ['ffmpeg', 'ffprobe'].filter(tool => {
+		try {
+			execFileSync(process.env[`${tool.toUpperCase()}_PATH`] ?? tool, ['-version'], { stdio: 'ignore' });
+			return false;
+		} catch {
+			return true;
+		}
+	});
+	if (!missing.length) {
+		return;
+	}
+	const install = process.platform === 'win32'
+		? 'winget install Gyan.FFmpeg'
+		: process.platform === 'darwin'
+			? 'brew install ffmpeg'
+			: 'sudo apt install ffmpeg';
+	console.warn(
+		`Warning: ${missing.join(' and ')} not found on PATH, so the recording will not be captioned with step titles.\n` +
+		`         The run still produces the raw video, screenshots, trace and report.\n` +
+		`         To caption it, install ffmpeg (${install}), make sure it is on PATH, and re-run,\n` +
+		`         or annotate the finished run with: node test/scenario/out/renderEvidenceChapters.js <run-dir>`
+	);
+}
 
 /**
  * Runs a UI validation scenario end to end and writes an evidence bundle.
@@ -95,6 +128,7 @@ function loadScenario(scenarioPath: string): Scenario {
 }
 
 export async function runScenario(scenario: Scenario): Promise<{ runPath: string; outcome: 'passed' | 'failed' | 'aborted' }> {
+	checkVideoTooling();
 	const appService = new ApplicationService();
 	const evidence = new EvidenceService(appService);
 	const runPath = await evidence.start(
@@ -151,7 +185,7 @@ export async function runScenario(scenario: Scenario): Promise<{ runPath: string
 
 	const reportPath = await evidence.finish(outcome, notes);
 	console.log(`Report: ${reportPath}`);
-	renderChapters(runPath);
+	tryRenderChapters(runPath);
 	return { runPath, outcome };
 }