Skip to content

ci: Parallelize integration and smoke test shards - #334833

Open
Dmitriy Vasyura (dmitrivMS) wants to merge 3 commits into
mainfrom
agents/ci-performance-optimization
Open

ci: Parallelize integration and smoke test shards#334833
Dmitriy Vasyura (dmitrivMS) wants to merge 3 commits into
mainfrom
agents/ci-performance-optimization

Conversation

@dmitrivMS

@dmitrivMS Dmitriy Vasyura (dmitrivMS) commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • run Agent Host E2E in conditional platform jobs beside the remaining Electron integration tests
  • split Electron smoke coverage into complementary core and chat shards on Linux, macOS, and Windows
  • move the macOS PAC and Kerberos proxy scenarios off the main Electron smoke critical path
  • skip repeated prelaunch synchronization in already-prepared test jobs
  • avoid compiling Copilot for Browser smoke jobs, which do not register Chat or Agents Window suites
  • skip login-shell environment probing for deterministic Browser and Remote test servers
  • use Mocha's Dot console reporter for Browser unit tests while preserving full JUnit results

Measured impact

Using #334766 and this PR's first Code OSS attempt as baselines:

  • the initial test sharding moved Agent Host E2E and chat/proxy smoke work off the main Electron paths
  • Browser test servers currently repeat prelaunch synchronization 17-18 times per job
  • a focused Browser smoke run took 15.4s with prelaunch and 12.2s without it, projecting roughly 50-60s saved per Browser job
  • the Browser unit reporter benchmark reduced output from 2,761 to 263 lines and from 177KB to 80KB for the same 2,700-test subset, with identical JUnit coverage
  • Browser-only Copilot compilation removes another 5-8s of unnecessary work

The remaining major setup opportunity is dependency caching: Windows downloads its 1.2GB node_modules cache in about 10s but spends about 71s extracting 6.6GB / 223k files. extensions/copilot/node_modules alone accounts for 1.28GB and 74k files, so job-specific additive caches could remove another 20-30s from Browser jobs, but that needs coordinated cache seeding and is intentionally not mixed into this change.

Validation

  • parsed the root and reusable workflows and asserted caller inputs, check names, artifact uniqueness, prelaunch settings, and Browser-only compile conditions
  • focused Browser smoke run: passed in 12.2s
  • focused Remote smoke run: passed in 15.7s
  • Browser unit benchmark: 2,700 tests passed with the Dot and JUnit reporters
  • compiled test/integration/browser
  • compiled test/automation
  • compiled vscode-test-resolver
  • targeted ESLint for all changed JavaScript and TypeScript files
  • node --check test/smoke/test/index.js
  • git diff --check
  • npm ci

Run Agent Host E2E beside the remaining Electron integration tests, split Electron smoke coverage into core and chat shards, and isolate the macOS proxy scenarios from the main smoke critical path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Skip redundant prelaunch work on prepared CI runners, avoid Browser-only Copilot compilation, disable login-shell probing on test servers, and reduce Browser unit console output while preserving JUnit results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engineering VS Code - Build / issue tracking / etc. vscode-build VS Code build process issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants