Agent Host session listing can starve under catalog invalidation and repeated Git probes

[roblourens]

tl;dr to the best of my understanding:

Brigit's logs show extremely slow listSessions calls, which block sending messages. One single instance took 400 seconds. It looks like there are some specific new-ish issues related to the recent session catalog storage changes. Basically

• listSessions can be slow- there is some known slowness inside the SDK, but also we now spawn a ton of git processes, which is maybe particularly slow on windows. This seems to be a regression from agentHost: make the orchestrator own session enumeration and chat lifecycle #329633
• The listSessions work gets invalidated when certain things happen, including resuming or creating sessions. When this happens during a slow call, it updates the epoch and has to repeat the slow work. This can happen a lot, and we end up looping on this slow work many times.

I have not really followed what all this work is about. Please let me know if I can help. listSessions is already an expensive operation, we should treat it like that, but also we should avoid doing extra expensive work like spawning any git processes.

---

User impact

Opening or creating Agent Host sessions can appear hung for tens of seconds or several minutes. In the reported captures:

• The first message in a new session remained on the new-session screen for roughly a minute before sending.
• A new session in a project took roughly 40 seconds before the first message was dispatched.
• Opening an existing session took roughly 20 seconds in one case.
• A later capture contained a single listSessions request that took 396.6 seconds before returning.
• Once a session and its provider state were warm, subsequent messages in that session were fast.

The problem is a feedback loop between an intrinsically expensive session-list computation and broad catalog invalidation. Listing performs per-session SDK/database/project enrichment, including many Git subprocesses. If a session is created, disposed, or restored while that work is running, the list epoch changes. The completed scan can then be discarded and restarted, while clearing the in-flight entry can allow another scan to start concurrently. Overlapping scans duplicate the same Git work, making each scan slower and increasing the chance of another invalidation.

Environment observed

• VS Code Insiders 1.136.0-insider, commit f83bb625e8aa45c596fecb274a1c6e47b3b5f06c
• Windows ARM64
• Agent Host Copilot provider
• External-session mode: last30Days
• Approximately 700 registered sessions, with roughly 90-100 visible

The invalidation/restart changes described below are also present in VS Code 1.135 Stable.

Evidence from the first log capture

Nine host-side list computations took a cumulative 354.8 seconds:

listSessions computed 89 of 704 session(s) ... in 48075ms
listSessions computed 89 of 704 session(s) ... in 31844ms
listSessions computed 89 of 704 session(s) ... in 32692ms
listSessions computed 91 of 706 session(s) ... in 36583ms
listSessions computed 91 of 706 session(s) ... in 31294ms
listSessions computed 94 of 707 session(s) ... in 59236ms
listSessions computed 94 of 707 session(s) ... in 31937ms
listSessions computed 96 of 709 session(s) ... in 47015ms
listSessions computed 96 of 709 session(s) ... in 36163ms

The computations formed immediate chains such as:

48.075s + 31.844s + 32.692s
36.583s + 31.294s
59.236s + 31.937s
47.015s + 36.163s

One AHP listSessions request took 83.199 seconds and spanned two complete host computations:

request  21:42:12.399 listSessions
compute  47.015s
compute  36.163s
response 21:43:35.598

The same capture contained 899 failed Git probes:

[agentHostGitService] > git rev-parse --show-toplevel failed
fatal: not a git repository (or any of the parent directories): .git

Evidence from the second log capture

A single client AHP request remained pending for 396.602 seconds:

22:32:39.726 listSessions request
22:39:16.328 listSessions response

During that interval, the host ran 11 complete list computations:

78.356s
81.228s
78.270s
84.417s
81.143s
82.195s
72.647s
66.931s
64.145s
38.742s
33.199s

Several computations overlapped in two interleaved chains. Every computation reported the same high-level result:

[AgentService] listSessions computed 99 of 712 session(s) for mode 'last30Days'

The captured client issued, while listing was still pending:

• 25 createSession requests
• 18 disposeSession requests

The host logs contained:

• 27 deferred chat creations
• 9 cold session restores
• Only 1 eventual session materialization
• 1,001 failed git rev-parse --show-toplevel probes

The renderer alternated existing-session restores with fresh untitled composers. Each untitled composer eagerly created a provisional backend session; switching away disposed it. These mutations repeatedly invalidated the list epoch.

While scans overlapped, individual passes took approximately 64-84 seconds. After the create/dispose/restore churn stopped, the final standalone pass took 33.199 seconds. This indicates that concurrent scans were duplicating the same subprocess work and roughly doubling the base scan cost.

Why listing spawns Git processes

The relevant path is:

AgentService.listSessions
  -> _computeSessions
  -> _registeredSessionMetadata (for each registered session)
  -> CopilotAgent.getChatMetadata
  -> SDK getSessionMetadata
  -> _resolveSessionProject when project metadata is unresolved
  -> projectFromCopilotContext
  -> resolveGitProject
  -> AgentHostGitService.getRepositoryRoot
  -> git rev-parse --show-toplevel

A successful repository root may then run git worktree list --porcelain to normalize a linked worktree to its primary checkout.

The project is used to group sessions under a local repository, normalize worktree sessions under the primary repository, and match historical sessions to open local workspaces. It is presentation/enrichment metadata rather than session identity.

Several details amplify the cost:

• AgentHostGitService caches only successful repository roots. Failed rev-parse results are not cached.
• Before #329633, Copilot's batch listing shared one projectByContext map across the entire catalog.
• After #329633, AgentService requests metadata one registered session at a time. Each CopilotAgent.getChatMetadata call creates a fresh limiter and map, losing cross-session project-resolution deduplication.
• Project resolution is persisted only when stored Agent Host metadata already exists. Registered/provider sessions without that stored metadata can repeat the failed lookup on every list.
• The second capture's 1,001 failures across 11 computations are approximately 91 failed probes per complete pass, indicating that the same unresolved/non-repository session set was repeatedly probed.

Why list computations restart

listSessions coalesces an in-flight computation by external-session mode and associates it with a registry epoch. _invalidateSessionList() currently:

this._registryEpoch++;
this._inFlightListSessions.clear();

#331176 introduced registry-first listing, the epoch/coalescing mechanism, and invalidation for create, dispose, discovery, and restore. Its stated purpose was to prevent a caller arriving after a mutation from joining a pre-mutation snapshot.

#331679 added pre/post epoch checks around _computeSessions. If the epoch changes during a scan, the completed caller recursively invokes listSessions again, discarding the result it just computed.

This combination has two effects:

1. Clearing _inFlightListSessions permits a new caller to begin another complete scan while the old one is still running.
2. The old caller can finish, observe a stale epoch, and join or start another scan.

Restore invalidation

A cold restore can register a missing row and hydrate list-visible live state such as title, status, activity, project, working directories, changes, and metadata. The current code invalidates unconditionally before placing the restored summary in AgentHostStateManager.

This is a valid reason to refresh presentation, but it is a broad reason to discard an entire provider/DB/Git traversal. In the captures, restored sessions were already known and every completed list still reported the same visible/total counts.

Provisional create/dispose invalidation

This is the clearest unnecessary invalidation observed:

• Eager new-session backings are created as provisional state with emitNotification: false.
• listSessions explicitly filters idle provisional sessions before per-session metadata work.
• The state-manager overlay also excludes idle provisional sessions.
• Despite being intentionally invisible, every provisional create registers the session and invalidates the list epoch.
• Disposing the composer tombstones/removes the invisible provisional session and invalidates the epoch again.

The second capture therefore repeatedly discarded visible-list computations for create/dispose pairs that did not change the visible result.

Related changes and release inclusion

Relevant changes:

• #329633: moved session enumeration into the orchestrator and changed Copilot from batch listing to per-session getChatMetadata, losing catalog-wide project-resolution deduplication.
• #331176: registry-first listing, epoch/coalescing, and mutation invalidation.
• #331679: restart active list computations when their epoch becomes stale.
• #331606: legacy Copilot CLI migration on open; may introduce additional registry mutation for affected sessions.
• #331730: deleted legacy worktree restoration; relevant to isolated restore-time git worktree list activity, not the repeated rev-parse list probes.
• #332689: later suppresses one metadata-recency invalidation source with the explicit intent of avoiding an in-flight computation becoming a redundant second pass.
• #332984: avoids SDK metadata reads for explicit restore when persisted metadata is complete; it does not apply to passive listSessions metadata reads.

VS Code 1.135 Stable includes #329633, #331176, #331679, #331606, and #331730. It does not include #332689 or #332984.

Telemetry evidence

monacoworkbench/agenthost.startup records the first session-list duration and timeout outcome. It does not include catalog size, external-session mode, internal restart count, Git probe count, or later refresh durations.

Builds containing #331176 but not #331679 improved successful tail latency compared with the available pre-#331176 cohort:

                              before #331176    after #331176, before #331679
p95                           51.8s             41.2s
p99                           96.6s             90.1s
successful lists >= 30s       9.05%             7.05%
requested-list timeout rate   4.59%             4.54%

The first observed build containing #331679 showed a regression relative to the preceding build:

                              before #331679    first after #331679    later after
p95                           34.0s             43.6s                  39.1s
p99                           79.3s             88.5s                  86.7s
successful lists >= 30s       6.10%             7.43%                  6.61%
requested-list timeout rate   2.54%             5.78%                  4.59%

The p95 increase appeared on Windows, macOS, and Linux. These are build-level correlations rather than proof because each build contains multiple changes, but the change matches the restart shape in both supplied captures.

For the affected machine, telemetry shows that the severe slowdown predates the newest build. Successful first-list durations increased from roughly 6-31 seconds on August 17 to frequent 71-133-second durations and two-minute timeouts beginning around August 20-21. The newest observed build was somewhat faster than the immediately preceding build but remained severely slow.

Potential mitigations and fixes

The following are possible mitigations, ordered from containment to architectural fixes:

1. Do not invalidate the visible session list for idle provisional create/dispose. These sessions are explicitly filtered from both the provider list and live overlay until materialization.
2. Keep one computation per mode in flight even when dirty. Record a trailing-refresh bit rather than clearing the in-flight entry and permitting overlapping scans.
3. Avoid recursive full restart after every epoch change. Return the coherent snapshot plus ordered notifications, or run at most one trailing recomputation after the current scan settles.
4. Separate membership and presentation epochs. Restoring live metadata should not necessarily invalidate registry membership enumeration.
5. Restore cross-session project-resolution deduplication. Use a provider/process-lifetime promise cache keyed by normalized cwd, gitRoot, or repository context.
6. Cache negative repository-root results with a bounded TTL. This avoids repeatedly spawning Git for stale or non-repository historical directories without permanently hiding a folder that later becomes a repository.
7. Trust SDK gitRoot or repository context when available. Do not run rev-parse merely to rediscover information the provider already supplied.
8. Filter old external sessions before provider metadata/Git enrichment. Durable registry modifiedTime can reject old rows early in last30Days mode.
9. Remove SDK and Git activation from passive listing. Return host-owned registry/persisted summary metadata immediately; resolve legacy project enrichment once at create/materialize/import/restore or asynchronously, then publish a summary update.
10. Add per-call list telemetry. Record catalog size, returned count, external mode, duration, restart count, concurrent scan count, metadata reads, Git probe/failure counts, and initiator so later-refresh regressions are observable.

A focused regression test or benchmark should use a large catalog with unresolved/non-repository working directories, start listing, then alternate cold restores and provisional composer create/dispose. It should assert that only one scan runs at a time, invisible provisional mutations do not restart it, and the request completes within a bounded number of catalog passes.

(Written by Copilot)

[roblourens]

Actually the 100s of git processes may be for external sessions, covering the case where that info is not already in our own sqlite cache?

[sandy081]

Heads-up: I investigated a similar session-listing performance issue in #331155. The analysis there identified expensive SDK catalog discovery and normal registry hydration during startup. This issue shows another way that the underlying listing cost is amplified: completed list computations are discarded and repeated after registry epoch changes. It is probably worth going through the analysis in #331155 because it provides relevant context for the underlying cost observed here.

I own #331679, so I looked specifically at the two epoch checks introduced by that PR.

#331679 was intended to make session catalog migration resilient when a provider catalog is temporarily unavailable. It added retry coordination, typed catalog failures, and preservation of the last successful client snapshot.

It also added checks before and after normal session hydration:

if (epoch !== this._registryEpoch) {
	return this.listSessions(mode);
}

These checks are not specific to migration. In particular, the final check causes an existing caller to discard a completed result and repeat the complete listing whenever any registry mutation occurs.

In the reported trace, ordinary create, dispose and restore activity repeatedly changes the epoch. As a result, one request traverses 11 complete listing computations and remains pending for approximately 396 seconds. Because invalidation clears the shared in-flight entry without cancelling the existing computation, multiple expensive computations can also overlap.

The epoch and invalidation mechanism originated in #331176. #331679 strengthened that behavior by making an already-running caller restart when its epoch becomes stale. This broader caching and consistency behavior was beyond #331679's intended migration-resilience scope.

I propose removing the two epoch retry checks introduced by #331679, restoring #331176's behavior:

• Callers arriving after invalidation start a fresh computation.
• An already-running caller is allowed to finish.
• One request does not recursively traverse an unbounded sequence of complete listings.

I will leave the decision about whether stronger post-mutation consistency is required—and how to provide it without overlapping or unbounded recomputation—to Benjamin Christopher Simmonds (@benibenj), who introduced the epoch and invalidation mechanism in #331176.

Removing these checks addresses the listing starvation introduced by #331679.

[vritant24]

Closing as indicated addressed in the PR. #333579
cc Benjamin Christopher Simmonds (@benibenj)