agentHost: Add Copilot runtime regression E2E coverage - #334818
Draft
roblourens wants to merge 5 commits into
Draft
agentHost: Add Copilot runtime regression E2E coverage#334818roblourens wants to merge 5 commits into
roblourens wants to merge 5 commits into
Conversation
Exercise optional custom-agent fields and persistence, empty and image responses, client tool search, native patches, and MCP lifecycle through the real Agent Host. Upgrade desktop and remote runtime/SDK dependencies and preserve freeform tool calls in replay.\n\n(Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The restart tests do not currently assert that selected-agent metadata is actually restored.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
src/vs/platform/agentHost/test/node/e2e/suites/subagentSuite.ts — The restart assertion never checks the persisted message.agent, so both variants still pass if… |
What changed in this PR
Updates the bundled Copilot runtime/SDK and adds deterministic Agent Host regression coverage.
Changes:
- Bumps desktop and remote Copilot dependencies.
- Adds 16 runtime, MCP, subagent, shell, and persistence scenarios.
- Extends replay support for freeform tools and copied-plugin paths.
| File | Description |
|---|---|
subagentSuite.ts |
Adds custom-agent and restoration scenarios. |
managementExtensionsSuite.ts |
Accepts policyHelper diagnostics. |
e2eTestContext.ts |
Exposes recording response injection. |
copilotRuntimeToolsSuite.ts |
Adds runtime tool regressions. |
copilotRuntimeMcpSuite.ts |
Adds MCP lifecycle regressions. |
copilotCoverageSuite.ts |
Adds shell follow-up coverage. |
agentHostE2ESuites.ts |
Registers new suites. |
e2e/README.md |
Documents replay behavior. |
…gpt-5-mini.prompt.md |
Refreshes GPT prompt snapshot. |
…claude-sonnet-4_6.prompt.md |
Refreshes Claude prompt snapshot. |
KNOWN_ISSUES.md |
Updates resolved/current limitations. |
modelRequestProjection.ts |
Preserves freeform tool format. |
capiWireCodec.ts |
Adds custom-tool-call codec support. |
capiReplayProxy.ts |
Resolves copied-plugin replay paths. |
…stopping-an-asynchronous-shell….yaml |
Records shell-stop scenario. |
…shell-output-preserves….yaml |
Records metadata scenario. |
…native-apply-patch….yaml |
Records patch scenario. |
…image-client-tool-results….yaml |
Records image-result scenario. |
…gpt-requests-detailed-reasoning….yaml |
Records reasoning defaults. |
…client-tool-search….yaml |
Records tool-search scenario. |
…accepted-empty-response….yaml |
Records empty-response scenario. |
…server-is-stopped-and-restarted.yaml |
Records MCP restart scenario. |
…plugin-tools-remain-callable….yaml |
Records subagent/MCP scenario. |
…supporting-files-are-readable….yaml |
Records plugin-file scenario. |
…restored-parent-accepts….yaml |
Refreshes restoration fixture. |
…permits-requested-file-output….yaml |
Records output-guidance scenario. |
…without-a-tools-list….yaml |
Records optional-tools scenario. |
…without-a-display-name….yaml |
Records optional-name scenario. |
…selected-on-the-first-turn….yaml |
Records first-turn selection. |
…selected-after-a-default-turn….yaml |
Records later selection. |
capiWireCodec.test.ts |
Tests custom-tool round trips. |
capiReplayProxyPathNormalization.test.ts |
Tests plugin-path binding. |
agent.ts |
Extends diagnostics contract. |
remote/package.json |
Bumps remote dependencies. |
remote/package-lock.json |
Locks remote dependency updates. |
package.json |
Bumps desktop dependencies. |
package-lock.json |
Locks desktop dependency updates. |
Files not reviewed (1)
- remote/package-lock.json: Generated file
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+149
to
+157
| assert.deepStrictEqual({ | ||
| messages: restored.turns.map(turn => turn.message.text), | ||
| response: markdownText({ turns: restored.turns.slice(-1) }).trim(), | ||
| states: restored.turns.map(turn => turn.state), | ||
| }, { | ||
| messages: before.turns.map(turn => turn.message.text), | ||
| response: 'CUSTOM_AGENT_CHILD_OK', | ||
| states: before.turns.map(() => TurnState.Complete), | ||
| }); |
Remove duplicate UNIFIED_WORKSPACE_PICKER_SETTING imports introduced by overlapping main changes so core compilation succeeds again.\n\n(Written by Copilot)\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit fc0a9e9)
…ugin paths Apply streaming configuration to both create and resume, assert it through E2E requests and launcher tests, and surface provider errors without waiting for timeouts. Normalize both decoded and JSON-escaped Windows plugin paths with a portable regression test. (Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve the now-enabled fixed-runtime subagent tests while incorporating upstream changes and resolving the obsolete known-issue gates. (Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The product-build default introduced on main replaced the upgraded runtime with a canary missing five tested fixes. Pin product CI to the same 1.0.84-1 runtime as desktop and remote manifests rather than gating regressions. Follow the new dot-notation lint rule in the added suite. (Written by Copilot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1.0.83-2to1.0.84-1, and the SDK from1.0.13-preview.4to1.0.13-preview.6.policyHelpermanaged-settings diagnostics source.1.0.84-1version. Upstream's canary override lacked five fixes and was correctly rejected by the new tests on all three platforms.Regression evidence
The same 16 scenarios produce 10 failures and 6 passes against the old runtime/SDK pair, and 16 passes against the updated pair.
The six tests that pass both versions are additional contract coverage, not claimed as reproductions of their exact upstream defects. The custom-agent output-guidance baseline is masked by the older optional-field failure. CLI-only, native-runtime-only, platform-dependent, and non-deterministic internal races are not claimed covered by these AHP tests.
Validation
AGENT_HOST_E2E_JOBS=2 npm run test-agent-host-e2e-coverage).mainto resolve overlapping upstream test gates. Type checking and the 16 new scenarios pass after the merge; all three GitHub Electron jobs also passed.Remaining CI issue
The latest GitHub Code OSS run passes compile/hygiene and Windows/Linux Electron. Its macOS Electron job still intermittently fails the existing
cancelling a turn paused for input allows a replacement turntest; the same test passes locally in isolation and on the packaged Azure runs. The failure recurred on retry. Runtime diagnostics end withSession idle finalization suppressedandqueued_lane: trueafter abort, with no replacement model request. The test remains enabled; no timeout increase or success fallback was added.The unrelated macOS Browser failure was a GitHub DNS lookup failure during smoke-test setup and passed on retry. The PR stays draft because the remaining macOS cancellation failure is unresolved.
(Written by Copilot)