feat: StateStore.loadAll({ maxParseBytes }) — stat-only parse budget for sub-linear startup

[ranxianglei]

Closes #185.

Motivation

Startup cost of loadAll grows linearly with corpus size: every boot does readFileSync + JSON.parse over all surviving records, even after startup GC caps the corpus (billion-context#478). At GB-scale corpora that is 100ms~seconds of synchronous parsing per boot, and it keeps growing with session count.

billion-context cannot work around this in-repo: selective loading would require replicating the kernel's canonical/spill (<name>.fb.json) savedAt reconciliation ("freshest wins") — a fragile copy of kernel internals.

What this PR does

Implements suggestion 1 from the issue (purely additive, default behavior unchanged):

export interface LoadAllOptions {
    maxParseBytes?: number;
}
async loadAll(options?: LoadAllOptions): Promise<Map<string, PersistedEnvelope<T>>>

Suggestion 2 (statAll()) was rejected: flat filenames are one-way hashes (sha256(id)[:24].json) and the id lives inside the envelope, so id-keyed metadata would require reading content anyway; a path-keyed variant would push canonical/spill pairing and freshest-wins onto the consumer — exactly the fragile duplication the issue rejects.

Semantics

1. No options ⇒ byte-for-byte identical to current behavior.
2. With maxParseBytes: pass 1 is readdir + stat only (no content reads). Each canonical <stem>.json is paired with its <stem>.fb.json spill under a dir+stem key (same stem in different dirs = different records). Group size = sum of member sizes; group freshness = max mtimeMs. Groups are filled newest-first while the running total stays within budget (skip-and-continue).
3. Hard cap: if even the newest group alone exceeds the budget, nothing is parsed — empty Map plus a warn log [persist] loadAll budget <N> bytes: parsed K/M record groups (skipped ...). Skipped files are never deleted and remain loadable via loadSync(id, hint).
4. mtime is the stat-level proxy for savedAt (savedAt is unknowable without parsing); ordering among parsed records still goes through the unchanged savedAt reconciliation.
5. Non-finite or negative maxParseBytes throws TypeError fail-fast.

The hard constraint from the issue is honored: a record's canonical and spill variants are selected together or not at all, so freshest-wins semantics can't break on a half-parsed pair.

Tests (tests/persist.test.ts, +5)

• Budget selects exactly the newest records within the cap; default loadAll() still parses everything; skipped files stay on disk and remain loadSync-able (skip ≠ delete).
• Canonical+spill all-or-nothing: tight budget skips the whole pair; roomy budget parses both and reconciles by savedAt (spill wins).
• Budget smaller than every record → empty map, files untouched, warn logged.
• -1 / NaN → TypeError.
• Same-stem files in different directories are NOT merged into one oversized group (dir-scoped pairing).

Verification

• npm run typecheck ✅
• npm test — 577 pass / 0 fail ✅
• npm run build ✅

No version bump (feature branch; release workflow handles that separately).

Release ordering (cross-repo rule)

This must land and publish to npm first; billion-context then bumps its exact-version dependency on acp-kernel and consumes loadAll({ maxParseBytes }) from SessionStore.boot() to turn BILI_MAX_SESSIONS into a true load budget.

Per AGENTS.md §6: requires review by at least 2 separate agents before merge.

[github-actions]

📦 Built Package Artifact

Branch: 2026-09-02_loadall-parse-budget (906de16)

Option A — Install from npm PR tag (recommended)

In your adapter project:

npm install acp-kernel@pr-186

Each push to this PR publishes a new version under the pr-186 npm tag.

Option B — Download artifact

1. Download the artifact from the Actions run
2. Extract the tarball and install:

tar xzf acp-kernel-pr186.tgz
npm install ./package

---

This comment is automatically updated on each push.

[ranxianglei]

🤖 ework agent · qwen3.8-27b

[bot] 🏷 独立评审通过(附 2 处已修复的文档/测试问题,已推送到 PR 分支)

查重:本 PR 是 #185 的实现产出,仓库内无其他相似 issue/PR(已按 loadAll budget / statAll / startup cost 检索)。

独立验证(本会话全新上下文,未复用实现时的结论)

• npm run typecheck ✅ · npm test 577 pass / 0 fail ✅ · npm run build ✅ —— 与 PR 描述一致,全部本地复现。
• 通读 src/persist/store.ts 全文核对配对逻辑:spill 由 spillPathFor 生成(store.ts:474-480),只改 basename、同目录 ⇒ dir+stem 分组与 canonical/spill 配对严格等价;"同进同出"硬约束成立。跨目录同 stem 不会误并(key 用 \u0000 分隔 dirname+stem)。
• 对账顺序无关性:loadAll 的 freshest-wins 用 savedAt >= 替换(store.ts:347),文件处理顺序任意(走 walk 序)结果不变——分组内两文件的先后不影响对账正确性。
• 默认路径逐字节等价:options?.maxParseBytes == null 时走原 walkJsonFiles 路径,行为零变化;version 未动(0.0.49),符合 feature 分支规则。
• 边界探测(临时脚本):budget=0 → 空 map 不崩;显式 null(类型非法)→ 按无预算处理,运行时安全。

发现的问题(已修复,commit 906de16 已推到 2026-09-02_loadall-parse-budget)

1. PR 描述第 3 条与实现不符(文档/测试缺口,非逻辑 bug)。 描述写"if even the newest group alone exceeds the budget, nothing is parsed",但实现是 skip-and-continue:最新组超预算时跳过它、继续用更旧的组填充。实测:最新组 200 KB + 两个旧组各 181 B,budget 372 B → 结果是两个旧组都被解析(warn: parsed 2/3 record groups (skipped 1))。

层次判断:实现行为是对的,错的是描述。skip-and-continue 正是 #185 floor 3 批准的语义("从新到旧贪心纳入累计 ≤ 预算的组,其余跳过");真正要保的不变量是"解析总字节 ≤ budget"(硬上限,单个超大文件无法突破),而"nothing is parsed"只在没有任何一组能容纳时才成立。改为 strict-prefix(遇到第一个装不下就停)反而浪费预算、加载更少记录,不符合"在成本约束内尽量加载最近状态"的消费方目标。

处置:不改代码语义,补文档 + 回归测试锁定行为——

• LoadAllOptions.maxParseBytes TSDoc 明确写清"A group that does not fit is skipped individually and selection continues with older groups … only if NO group fits is the result empty";
• 新增测试 loadAll budget skips an oversized newest group and fills from older groups(tests/persist.test.ts),防止将来任一方漂移。
• PR body 无法经 git 修改,建议作者把第 3 条措辞同步更正(或忽略——代码 TSDoc + 测试已是权威语义)。

2. loadAll doc comment 的 "Never throws." 与新契约矛盾。 无效 maxParseBytes 会抛 TypeError fail-fast。已收窄为:文件系统/记录问题永不抛;仅非法预算参数抛 TypeError。

次要观察(不阻塞,无需行动)

• TOCTOU:stat 与 read 之间非原子,并发写入下实际读取字节理论上可略超 stat 时刻的预算。stat-based budget 的固有属性,store 写入本身有 debounce,boot 期并发写概率低,接受现状。
• 被跳过的 namespaced(relPath 自定义)记录需经 loadSync(id, hint) 取回(flat 名 fallback 不可用)——这与现有 loadSync 设计一致,hint 机制本就为此存在,文档已覆盖。

状态

推送后 CI 对新 commit 全绿:test (22) ✅ · test (24) ✅ · build-artifact ✅ · pr-validation ✅,mergeable clean。修复后本地复核:typecheck ✅ · 578 pass / 0 fail(577+1 新测试)· build ✅。

按 AGENTS.md §6 合并前需 ≥2 个独立 agent 评审——本条为第一份独立评审(通过);请再安排一份独立评审或由 owner 确认后合并。发版顺序不变:本仓先合并发布 npm,billion-context 再 bump 精确版本消费。