Default to an installed build, and report missing video tooling

[Bryan Chen (bryanchen-d)]

Three problems found running the validate-ui-scenario skill from a fresh checkout against #244597.

Nothing ran without a target flag

With no flags the runner used the build from the checkout, which only exists after compiling the product. Verified on unmodified main: the documented starting point fails to launch.

Failed to launch Electron within 60000ms.
Original error: electronApplication.waitForEvent: Target page, context or browser has been closed

Reproducing a reported issue means running the shipped product anyway, so with no target flag the runner now finds an installed VS Code Insiders — falling back to Stable — and logs the choice:

No target given, using the installed build at C:\...\Microsoft VS Code Insiders. Pass --dev to run this checkout instead.

--dev selects the checkout build, and --build <app-root> still pins an exact install. Existing callers that pass --build are unaffected.

A missing ffmpeg was only discovered after the run

Captions are rendered once the scenario finishes, so a missing ffmpeg surfaced as a raw spawnSync ffprobe ENOENT after VS Code had been driven and the report written — and it threw out of runScenario, so the run never printed its outcome.

The runner now checks before launching anything and names the fix for the platform:

Warning: ffmpeg and ffprobe not found on PATH, so the recording will not be captioned with step titles.
         The run still produces the raw video, screenshots, trace and report.
         To caption it, install ffmpeg (winget install Gyan.FFmpeg), make sure it is on PATH, and re-run,
         or annotate the finished run with: node test/scenario/out/renderEvidenceChapters.js <run-dir>

Caption rendering can no longer fail a run that has already produced its evidence, and the post-run message is actionable rather than an ENOENT stack.

The window did not fill the recording

The recording canvas is 1920x1080, but VS Code sizes its own window — DEFAULT_WORKSPACE_WINDOW_SIZE is 1440x900 and DEFAULT_EMPTY_WINDOW_SIZE is 1200x800 — so the capture showed the window in the top-left corner of a larger frame, wasting about 30% of every pixel recorded.

Measured on the reporting machine:

Window viewport	1440 x 900
Video canvas	1920 x 1080
Display work area	1512 x 901

The window is now sized to the canvas once recording is enabled. The display being smaller than the canvas is fine — confirmed by probing before implementing: after setBounds, innerWidth/innerHeight report 1920x1080 and both the screenshot and the video come back full frame on a 1512x901 display. Sizing only happens when video recording is on, so non-recording smoke runs are untouched.

Verification

Against installed Insiders on Windows, with the skill's own documented example:

• no flags, ffmpeg absent: warns up front, runs both steps, writes the report, warns again with the install command, exits 1 for the reproduced failure
• no flags, ffmpeg present: same run, plus videos/annotated.mp4 at 1920x1300 (1080 of workbench + the caption band) with no dead space
• screenshots and raw recording are both 1920x1080, where they were 1440x900 inside a 1920x1080 frame before
• --dev still resolves the checkout build; it fails to launch here only because this checkout is not built, identically to unmodified main

ffmpeg was installed, and still not found

A PATH edit only reaches processes started afterwards, so an editor that was already running never sees it. On the reporting machine ffmpeg was installed and present in the persisted user PATH, yet invisible to the shell the skill ran in — so the run produced no annotated video.

The runner now probes the usual install locations (winget Links and Packages, chocolatey, Program Files, Homebrew, /usr/local/bin) in addition to PATH. Verified with ffmpeg deliberately off PATH:

$ node test/scenario/out/runScenario.js .build/vscode-playwright-mcp/issue-250159.cjs
Captioned 2 steps into videos/annotated.mp4

No warning, no manual setup.

Steps flowed past too quickly to read

A caption is only legible for as long as its step is on screen, and an asserting step can finish in a few hundred milliseconds. Each finished step is now held briefly — stepPauseMs, default 1000, set 0 for timing-sensitive scenarios.

Blocked steps are classified and reported

skip previously recorded only that a step did not run. It now takes needs:

needs	Meaning	Consequence
human	A person is required — hardware, a judgement call, a sign-in that cannot be scripted	Reported so someone can check it by hand
infrastructure	Automatable, but the harness cannot do it yet	Reported as an enhancement request against the skill

The distinction is recorded in manifest.json, highlighted in a Needs attention section of report.html, shown on the caption band, and printed at the end of the run:

  SKIP B-02 Verify the printed page matches [needs human]: Comparing physical print output requires a person with a printer.
Needs a person: B-02 Verify the printed page matches - Comparing physical print output requires a person with a printer.
Outcome: aborted

Both variants were exercised end to end; the caption band renders SKIPPED - NEEDS HUMAN in amber, later steps do not run, and the outcome is aborted rather than passed. test/mcp still compiles against the widened EvidenceService.step signature.

Authored with Copilot

[Copilot]

Pull request overview

Improves UI scenario validation for fresh checkouts and installed VS Code builds.

Changes:

• Defaults to installed Insiders/Stable, with --dev for checkout builds.
• Reports missing video tools and makes caption failures non-fatal.
• Sizes desktop windows to the recording canvas and updates guidance.

Show a summary per file

File	Description
test/scenario/src/runScenario.ts	Adds video-tool checks and safe caption rendering.
test/scenario/src/renderEvidenceChapters.ts	Exposes non-failing caption rendering.
test/scenario/src/options.ts	Adds the --dev option.
test/scenario/src/application.ts	Discovers installed VS Code builds by default.
test/automation/src/playwrightElectron.ts	Resizes recorded Electron windows.
.github/skills/validate-ui-scenario/SKILL.md	Documents new targeting and tooling behavior.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 6/6 changed files
• Comments generated: 4
• Review effort level: Balanced