[P0] Make large-repository analyze bounded, staged, resumable, and SIGTERM-safe

[100yenadmin]

Parent: #130

Goal

Make large 2,048-dimensional repository analysis bounded, staged, resumable, and cooperative under memory pressure while keeping the canonical index readable.

Non-goals

Do not increase resource limits to hide leaks, restart all embeddings after interruption, auto-SIGKILL analyzers, or mutate the live canonical database before staged validation.

Required behavior

• Automatic heap cap: max(2048, min(6144, floor(effectiveMemoryMb * 0.5))); explicit NODE_OPTIONS wins.
• Stream cached-vector identity, buffer at most 256 vectors for preservation, page at most 512 embeddable nodes, use embedding batch 16 and HTTP sub-batch 8.
• Add analyze --staged with durable promotion journal: prepared, old-backed-up, new-installed, metadata/registry-committed.
• Preserve the 5,000-node pending window and resume only it plus genuine changes.
• Route SIGTERM through bounded checkpoint shutdown and emit sanitized JSONL resource progress via GITNEXUS_ANALYZE_RESOURCE_LOG.

Acceptance and proof

Crash injection at every promotion/checkpoint boundary; 25k x 2,048 CI integration; sequential clean and interrupted/resumed 85k and 125k release soaks; one disposable OpenClaw corpus. Record V8, RSS, and swap as observational telemetry for Voyage-backed runs; acceptance is bounded buffering/checkpoint behavior and no missing or mixed canonical generation, not a host-resource admission threshold.

Evidence: /Volumes/LEXAR/Codex/session-notes/2026-07-20/gitnexus-repository-health-sprint/evals/.

Stop conditions

Stop on unbounded materialization, analyzer crash, unresolved WAL/shadow state, false-success exit, credential/source leakage, or any failure path able to delete both generations.

Exact next action

Create feat/staged-large-repo-analyze from current protected main, run graph impact on heap/cache/pipeline/shutdown symbols, and implement bounded behavior before promotion mechanics.

[evaos-code-review-bot]

evaOS issue enrichment

Issue: #132 - [P0] Make large-repository analyze bounded, staged, resumable, and SIGTERM-safe
State: open; milestone: GitNexus Repository Health & Large-Repo Reliability Sprint

Related issues/PRs: #130.
Existing labels: P0, evidence-gate, release-blocker.
Suggested labels: bug.
Suggested owners: none.

Related context

• EPIC: Restore truthful semantic retrieval and canonical index lifecycle #130 - mentioned in issue metadata; inspect for dependency, duplicate, or prior-decision risk.

Agent-start packet

Problem shape: [P0] Make large-repository analyze bounded, staged, resumable, and SIGTERM-safe - Parent: #130 ## Goal Make large 2,048-dimensional repository analysis bounded, staged, resumable, and cooperative under memory pressure while keeping the canonical index readable. ## Non-goals Do not increase resource
Product fit: Architecture/integration issue; check repo policy, prior decisions, and operational constraints before implementation.
Implementation wedge: Start with a bounded adapter/contract or dry-run packet before changing live runtime behavior.

Build / borrow / buy scan:

• Build: identify the smallest implementation wedge that proves the issue without broad rollout.
• Borrow: inspect same-repo prior art, repo memory, GitNexus context, and allowlisted cross-repo examples first.
• Buy/use: deferred; no product, architecture, UX, library-choice, integration, roadmap, or market-positioning trigger detected.

Candidate sources:

• VISION.md / repo policy / repo-memory.md / GitNexus packet.
• Same-repo issues and PRs referenced by issue metadata.

Acceptance criteria:

• Preserve and execute the issue's stated acceptance criteria.
• Include source citations for any external OSS/library/current-market claims.
• Keep issue-enrichment comments sticky and idempotent.
• Keep labels, owners, reviewers, and roadmap fields suggestion-only.

Proof plan:

• Focused unit/fixture test for the changed planner behavior.
• Dry-run issue-enrichment evidence packet before live comment posting.
• Redaction check for issue body, citations, and generated markdown.
• Contract or degraded-mode proof before runtime promotion.

Known traps:

• Do not treat related links as proof by themselves; each source needs a reason why it matters.
• Do not scan old issue backlogs or exceed repo-level enrichment throttles.

Non-goals:

• Do not auto-apply labels, owners, reviewers, roadmap fields, or milestones.
• Do not bulk-enrich old backlog issues from this planner output.
• Do not claim external market or OSS research was performed unless cited sources are present.

Context-source taxonomy:

• [enabled] vision_repo_policy_memory_gitnexus: Use repo vision, policy, memory, and GitNexus first before external research.
• [enabled] same_repo_issues_prs: Issue metadata contains same-repo references.
• [deferred] allowlisted_cross_repo_github: Deferred until product, UX, market-positioning, or library-choice signal appears.
• [deferred] external_oss_examples: Deferred for low-research-value issue class.
• [deferred] library_api_docs: Deferred unless a library/API decision is present.
• [deferred] current_market_examples: Deferred for implementation-only issues.
• [enabled] internal_mcp_docs: Use internal docs/MCP context when available; missing context is degraded, not blocking.

Validation suggestions

• Confirm owner, acceptance criteria, and validation evidence before implementation.

Triage gaps

• Owner or reviewer signal not detected in issue body.

No labels, owners, reviewers, or roadmap fields were changed by this bot.

[100yenadmin]

Reopening on installed-release runtime evidence. The exact 1.6.10-electric.3 artifact analyzed the approved OpenClaw corpus at 2220e9f841846fab99fef12cd635ee2a32a59705 with --workers 2 --index-only --staged --embeddings 0, parse concurrency 1, and a 6,144 MiB explicit V8 heap. It reached mixed-language scope resolution, then aborted at Swift linking symbols with Reached heap limit / Allocation failed - JavaScript heap out of memory. The canonical 84,501-vector index remained readable; the staged generation and ownership record remain preserved. No host memory/swap admission or termination gate was involved.

This means the issue acceptance condition for a real disposable OpenClaw corpus was not met even though the synthetic 85k/125k embedding soaks passed. Source inspection points to the existing GITNEXUS_DISK_SCOPE_INDEX path being default-off; a preserved staged retry with that path enabled is the next bounded experiment. Do not raise the heap cap as the fix.

Evidence: evals/openclaw-oom-recovery-plan.txt plus the coordinator retry evidence under the sprint packet. A separate defect in the read-only recovery-plan doctor was filed as #153.

[100yenadmin]

The installed .3 disk-scope retry also failed, so the experiment is terminal and source work is required before another OpenClaw run.

Exact result:

• source 2220e9f841846fab99fef12cd635ee2a32a59705
• installed 1.6.10-electric.3
• --workers 2 --index-only --staged --embeddings 0, parse concurrency 1, explicit 6,144 MiB heap
• remote Voyage voyage-code-3 / 2,048 dimensions
• GITNEXUS_DISK_SCOPE_INDEX=1
• exit 134 / V8 heap exhaustion during scopeResolution at 59 percent

No host memory/swap/RSS/free-memory gate admitted or terminated this Voyage run. The serving canonical generation remains untouched and readable at 275,835 nodes, 1,243,701 edges, and 84,501 embeddings. The staged diagnostic generation remains preserved. Its staged lock names dead PID 25699; I have not manually deleted it because the supported next analyze path reclaims stale locks.

Bounded diagnosis:

• parsed-file store: 1.7 GiB
• disk scope-index store: 32 MiB
• 28,737 resource events; parsing max heap/RSS 1,528.3/2,280.8 MiB; logged scope-resolution heap reached 3,874.9 MiB before the final unlogged rise to the V8 cap
• the run reached analyzing types but never resolving references
• current disk seal occurs only after finalizeScopeModel, post-finalize propagation, and resolveReferenceSites, so it cannot bound this finalize-stage peak
• loadParsedFilesForPaths streams shards but retains the full matched language payload, while finalize materializes workspace-wide indexes against it

Disposition: no unchanged retry, no heap increase. Keep canonical serving and implement/test the smallest bounded finalize/materialization seam before the next real OpenClaw attempt.

Evidence: /Volumes/LEXAR/Codex/session-notes/2026-07-20/gitnexus-repository-health-sprint/evals/openclaw-electric3-disk-scope-retry.md (SHA-256 2e3c1ba3fe77fc84c1e45a611b0cd9eb2c3a823ce298a73dbe41bb79ddc4760b).

[100yenadmin]

Large-Swift fix PR: #157, current head c1d4e15b9ea3ae613d0d7aa6f8533d5510e939ec.

The patch replaces O(files x definitions/bindings) same-target fan-out with shared target-level namespaces, preserves target isolation and local-first lookup, and caches one target grouping across the sibling passes. Focused Swift tests and TypeScript pass; GitNexus change impact is LOW with no affected flows. OpenClaw production remains on its healthy 84,501-vector generation and will not be retried until this fix is merged and available in an installed release.

[100yenadmin]

Source fix merged: PR #157 landed on main as merge commit e57b872e3ee596fdc8253f9e251ba3308e3c41e3 after exact-head CI Gate and CodeRabbit approval.

This is merged-source proof only. OpenClaw remains on its healthy 84,501-vector production generation; runtime retry still waits for an installed patch release containing this commit.