Semantic index: guard against large vendored/nested repos in umbrella workspaces

[jonathanKingston]

Summary

Follow-up from the gortex indexing performance work (PR branch claude/gortex-indexing-perf-n6m63e). That PR fixes the common causes of slow/CPU-heavy semantic indexing (gortex v0.60.0 bump, daemon-start race, and excluding gitignored build output). One case remains that those fixes deliberately do not cover: umbrella workspaces that contain large vendored nested git repos of tracked files.

The problem

Some workspaces are an "umbrella" directory holding several embedded git repos. A real example (a DuckDuckGo working tree) breaks down as:

subtree	~files	nature	excludable?
apple-browsers build output	~235k	gitignored (.build/.swiftpm/DerivedData)	✅ handled by the PR
apple-browsers source	~13k	tracked, wanted	keep
shared-web-tests/web-platform-tests	~100k	tracked in a vendored nested repo (WPT corpus)	❌ no .gitignore touches it

After the PR's build-output excludes, gortex still indexes ~109k files, dominated by the ~100k tracked Web Platform Tests fixtures. Because those files are legitimately git-tracked (in WPT's own repo), no .gitignore-based exclude — static or git-derived — can remove them.

Two concrete harms observed:

1. Indexing takes minutes and starves the shared daemon. gortex runs one daemon per userData HOME; every workspace is tracked into it. While the daemon grinds a huge umbrella, every other repo's track --wait can't settle within the app's wait window, so unrelated workspaces get flipped to semantic index: error.
2. Slow-but-progressing indexes surface as failures. When track --wait times out but the daemon is still indexing (and does eventually reach fresh), the app treats the timeout as an error rather than "still building."

Proposed directions (to be decided)

1. Per-nested-repo size guard — when computing what to index, skip an embedded repo whose tracked-file count exceeds a threshold (e.g. keep apple-browsers's ~13k source, skip WPT's ~100k). Most targeted; needs a cheap per-repo file count.
2. Global workspace file-count cap — if a workspace exceeds N files, decline semantic indexing and fall back to text search, so the daemon is never pointed at a pathological tree. Simplest safety net; all-or-nothing.
3. "Still building", not "error" — when track --wait times out but the daemon is still making progress, keep the index status as building and let it settle in the background instead of flipping to error. Worth doing regardless of 1/2.

Directions 2 and 3 are a good safety-net baseline; 1 is the smarter default that preserves semantic search for the parts of an umbrella you actually work in. Likely land 2 + 3 first, then 1.

Notes / where the code lives

• Exclude registration and index orchestration: src/main/services/search/semantic-index.ts (ensureGortexExcludes, ensureGortexIndex, SEMANTIC_INDEX_WAIT_MS/timeout handling).
• Git-derived excludes (the general build-output solution, best-effort): src/main/services/search/git-derived-excludes.ts. Note it currently returns empty on very large umbrella trees (the nested-repo find doesn't complete / FDs exhausted) — related scaling limitation to consider here.
• Index status flip to error: src/main/services/search/index-status.ts + the .catch paths in ensureSemanticIndex/runSemanticIndexUpdate.
• There's also a related scaling issue: the recursive fs.watch over a 525k-file umbrella hits EMFILE. Worth tracking in the same effort since a large-workspace guard could gate the watcher too (src/main/services/search/workspace-index-watcher.ts).

[jonathanKingston]

Status audit (2026-07-27): draft PR #1144 implements the first scale-guard slice: bounded workspace policy, semantic limited/skipped states, watcher/index sequencing, fallback messaging, and focused visual coverage. Nested-repo discovery/exclusions, discovery caching, and timeout-vs-progress lifecycle are explicitly still open. Keeping this issue open against those follow-ups.