Skip to content

feature: add deterministic editor performance lab - #3107

Draft
SimonSiefke wants to merge 9 commits into
mainfrom
agent/autonomous-vscode-performance-lab
Draft

feature: add deterministic editor performance lab#3107
SimonSiefke wants to merge 9 commits into
mainfrom
agent/autonomous-vscode-performance-lab

Conversation

@SimonSiefke

@SimonSiefke SimonSiefke commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • measure Enter-to-DOM-ready and paint confirmation inside the renderer
  • compare fresh-process samples in randomized same-runner ABBA/BAAB blocks
  • disable core AI work, record process manifests, and reject bundled Copilot
  • fingerprint optimized executables, bundles, source maps, commits, and dirty state
  • collect source-mapped deterministic work separately and aggregate hosted replicas hierarchically
  • fail closed on invalid scoring while permitting non-claiming mechanism exploration on underpowered hosts

Fresh-agent trials

The first fresh agent retained no patch. Identical builds appeared 4.69% different with a CI of -13.56% to +3.60%, and bundled Copilot was present despite --disable-extensions.

After refinement, a 24-sample-per-arm A/A correctly became invalid: -2.44%, CI -11.52% to +5.82%. Its 8.67% interval half-width cannot detect a 5% target. Copilot was absent and semantic code/* phases were populated.

A second fresh agent then used deterministic mechanism exploration without delaying behavior. Across three exact runs per arm it found:

  • breadcrumb false focus resets: 3 to 0
  • breadcrumb reveal calls: 4 to 1
  • no-item focus-event allocations: 3 to 0
  • editor option update/clone/migration/registry passes: 3 to 2
  • migration reads: 117/90/51 to 78/60/34

Every claimed counter had 0% variation. The combined action ceiling was below 0.4%, far below local timing resolution, so the patch was saved as a mechanism candidate and reverted rather than described as a speedup.

The trial refined the loop further:

  • deterministic candidates can be queued on noisy hosts, but cannot become proxy-win or UX claims
  • timing is deferred until a compatible batch approaches calibrated detectability
  • tracker arms are grouped to avoid repeated 2.5-3 minute transformations
  • unchanged noisy counters are ignored; unstable counters claiming a delta invalidate the mechanism
  • the scoped optimized desktop build and source-map audit procedure are documented
  • hosted work attribution now includes breadcrumbs, editor configuration/migration, text-file readonly setup, and inline completions

Verdict and validation

A proxy win requires less stable deterministic work and a latency CI upper bound at or below +2%. A UX-confirmed result additionally requires a renderer-latency CI excluding zero. Profiles and instrumented samples never enter scoring.

  • performance-lab: 30 tests passed
  • workflow/build package: 94 tests passed
  • launch-worker: 31 passed, 3 skipped
  • test-worker: 10 passed
  • affected TypeScript checks and repository formatting passed

This PR changes the benchmark and performance lab only. It claims no VS Code speed improvement and retains no VS Code patch.

@SimonSiefke SimonSiefke changed the title feature: add autonomous VS Code performance lab feature: add performance measures Jul 19, 2026
@SimonSiefke SimonSiefke changed the title feature: add performance measures feature: add deterministic editor performance lab Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant