diff --git a/CHANGELOG.md b/CHANGELOG.md index 14fec20..5a8f2df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## Unreleased (master, since v0.1.38) - **fix(degeneration): thinking/text 单字符退化熔断 + 一次性恢复通知（closes #351）** — 长会话末尾模型偶发退化为单字符长连击（实测：thinking 块末尾 4655 个连续「【」，跨轮升级直至 turn abort、会话停死）。根因链已代码级验证：pi 的 openai-completions 转换把历史 assistant thinking 在**每个后续请求**中回传 provider（`reasoning_content`，或 requiresThinkingAsText 时转纯文本），aborted turn 的部分消息又持久化在会话日志里 → 退化尾部随每轮 prompt 重放 → 模型看到自己上一轮以数千个重复字符结尾 → 续写偏置再次触发退化 → 连环 abort。新增 `src/degeneration.ts`：每个 context 事件对出站视图的 assistant text/thinking 块折叠 ≥`minRun`（默认 200，下限 8，codepoint/代理对安全）的单 codepoint 连击为短标记（保留 ≤3 份样本；纯函数、幂等——标记固定文案无相邻重复码点、fail-safe；持久化历史不改，toolCall 参数不动以免与实际执行脱钩）；当最近一条 assistant 消息已退化时追加一次性 `[ACP recovery notice]`（位置自限：模型产出新 turn 后自动消失，无持久状态不累积，#223 教训）。检测走持久化 originals 而非出站视图：thinking-only aborted turn 会被 projectMessage 丢弃（空文本在 OpenAI 兼容 provider 400），但它仍是模型的"上一轮"，通知必须照发。acp.json 新键 `degenerationGuard`（boolean 或 `{enabled,minRun}`，默认开；`false` 为 kill-switch）。附带修复：`repetitionGuard` 此前不在 user-config KNOWN 白名单内，acp.json 中配置被静默丢弃（dead key），本次补入。测试 `tests/degeneration.test.ts`（31 例：单元 + context transform 端到端 wiring） - **fix(reasoning): 闭合判定改按回合证据——无用户消息的长 agent 会话不再永久保留 compress thinking（closes #348）** — 原门控“compress 调用之后存在真实用户消息才算闭合”在长 agent 会话不可达（整个会话只有开头 1–2 条用户消息，后续 30 个 compress 全部被永久视为活跃回合，观察会话 0 次触发，thinking 地板 20.6K/8.4K/10.6K 字符全部滞留）。现在闭合判定改为：消息内**每个** compress toolCall 的 toolResult（role `toolResult`、`toolCallId` 匹配）已出现在更晚位置，且其后至少还有一条消息（回合已实际推进）。安全门不变：结果未返回或结果仍是最后一条消息（在飞中）绝不动；nudge 在 drop 之后才注入，不可能光当“结果后的消息”闭合在飞回合；per-provider `compress.providers..reasoning.drop=false` 逃生阀保留（GLM 等 reasoning 回显模型）。测试重写 + 新增 #348 场景（无用户消息的助手链闭合、result 悬置、result 在 call 之前、多 toolCall 部分闭合、误 id 不闭合） +- **feat(contract): ACP 压缩块机器可读契约（closes #368）** — sidecar 顶层新增 `schemaVersion`+`producer`（缺失=v1；未知高版本 bcp 侧「宽松加载已知字段 + 每文件记一次日志」，downgrade 不丢状态；仅破坏性变更升版本，其余只增不改）；新增 `billion-context-pi/contract` subpath 导出（`BcpBlockV1` 类型、`SCHEMA_VERSION` 常量、`createSidecarEnvelope`）+ `schema/bcp-block-v1.json`（JSON Schema，随包发布），供下游跨进程/跨重启稳定读取块结构而无需绑定私有字段；README 记录 tmp+rename 原子写承诺（下游可安全用 mtime/size 做水位并整文件替换）。纯加法、零行为变更，不改 sidecar 位置与 pi 集成。测试：`tests/contract.test.ts` + `tests/state.test.ts` - **fix(overflow): output headroom 预留按窗口比例封顶，默认 25%（closes #207）** — `reserveOutputHeadroom` 原按模型注册表 maxTokens **全额**预留输出预算：maxTokens 占窗口比例大的模型（qwen3.8-27b：262144 窗口 / 131072 maxTokens）输入预算被砍半，kernel 75% 强制压缩带在完整窗口 ~37% 处触发（host pct 仅 ~34%，两个口径不同加剧误导）。现在预留量 = min(maxTokens, `outputHeadroomMaxPct` × window)：新增 acp.json 配置键 `outputHeadroomMaxPct`（默认 0.25，接受比例或 `"N%"`；0 完全禁用预留，≥1 恢复旧的全额行为）。小预留不受影响（同窗口 int4 版 32K maxTokens 保持原样），超出预留的超长回复溢出一次后由既有 overflow self-heal（learned window + armed emergency）下一轮恢复。可观测性：`[turn]` 日志新增 `fullWindow` 字段（仅当 limit 被预留削减时出现，= 本轮 recenter 后的完整窗口），消除 pct（完整窗口口径）vs limit（预留后口径）混淆；`output-headroom` 事件日志新增 `cap` 字段；`/acp` 面板与 `acp_status` 分母经 `applyOutputHeadroom` 同步使用同一封顶值（#267 统一口径不回归） - **fix(state): in-memory（无 session 文件）会话压缩后状态不再丢失（closes #322）** — `src/state.ts` 的 cache 更新原本在 no-file early return 之后，导致没有 session 文件的 in-memory 会话压缩后状态不被保存（恢复/重启时丢失）。现在将 cache 更新移到 early return 之前，确保 in-memory 会话也持久化压缩状态。测试：`tests/state.test.ts` + `tests/compress-tool.test.ts` 扩展 - **fix(nudge): 增长基线刻度混用复位（closes #267）** — token 计量在估算值 ↔ provider 真实 usage 锚点间翻转（hostFloor 激活/失活）时，增长增量跨翻转相减是伪度量：旧刻度基线对新刻度 tokenCount 相减，凭空多出 "+35K 假增长"，绕过 cadence 节流乱发 nudge（或反向漏发）。现在翻转时统一重置全部增长基线：kernel 侧 `lastNudgeShownTokens` / `lastPerMessageNudgeTokens` / `lastShownByTier`（0.0.55 per-tier cadence 基线）+ 扩展侧 `clearNudgeTokenStamps()`（#316 引入的同轮 re-inject 刻度戳，旧刻度戳同样会伪造满地板增长触发伪重注入）；usage bands 的 floor-stale 行为保留不动。附带：② `/acp` 面板与 `acp_status` 百分比分母统一为与实时仲裁相同的 `window − maxOutput`（新增 `src/overflow-selfheal.ts` `applyOutputHeadroom`，此前面板按全窗口报告而 nudge bands 按 headroom 后窗口仲裁）；③ auto-update 限流与只读停止标记按安装位置 sha256 隔离（多副本互不干扰）。实现：`src/runtime.ts` `noteTokenScale`/`dropTokenScale`，`src/index.ts` 翻转检测重置，`src/update.ts` `locationKey`；测试 `tests/growth-scale-flip.test.ts` + `tests/update.test.ts` 扩展 diff --git a/README.md b/README.md index a0e45eb..ad99b54 100644 --- a/README.md +++ b/README.md @@ -242,6 +242,31 @@ Restore them next to each other on the target machine. For clone/fork children, > If you import only the `.jsonl`, ACP's log-replay fallback rebuilds the state automatically on the next session (see above). Copying the pair is still preferred — it is exact, while the replay re-derives token snapshots and can only restore what the transcript records. +### Machine-readable contract for compression blocks (#368) + +Downstream tools that read a session's compressed blocks (cross-session search, memory indexes, …) should rely on this stable boundary instead of globbing private fields: + +**Versioned sidecar.** Every `.acp.json` carries a top-level envelope: + +```json +{ "schemaVersion": 1, "producer": { "name": "billion-context-pi", "version": "" }, "blocks": [ … ], … } +``` + +Semantics: +- **Missing `schemaVersion` == v1.** Files written before this field existed are treated as version 1. +- **Unknown / higher version.** billion-context-pi stays *tolerant* when loading a higher `schemaVersion` (reads only the fields it knows, logs `unknown-schema-version` once per file) so a downgrade never silently drops state. **Consumers parsing the file themselves must do the opposite:** skip a file/block whose `schemaVersion` they don't understand, log once, and never rewrite it. +- **Bump only on breakage.** `schemaVersion` changes solely when a listed field is renamed, removed, or re-semanticked. New optional fields are additive and never bump the number. + +**Public exports.** Import the pinned types/constants without pulling in the Pi host integration: + +```ts +import { SCHEMA_VERSION, PRODUCER_NAME, BcpBlockV1, createSidecarEnvelope } from "billion-context-pi/contract"; +``` + +A machine-readable JSON Schema for one block ships at [`schema/bcp-block-v1.json`](./schema/bcp-block-v1.json) (included in the npm package — point your validator at it). Each `blocks[]` element is a `BcpBlockV1`: the required stable core is `blockId`, `summary`, `tier`, `compressedTokens`, `createdAt` (**epoch milliseconds**, not ISO 8601); `topic` / `startRef` / `endRef` / `effectiveMessageIds` are optional coverage metadata. Extra properties are permitted — real blocks also carry internal pruning/index fields that are intentionally outside the contract. + +**Atomic writes.** The sidecar is written as a temp file + `rename()` over the target, so a reader never observes a half-written file: watermark on mtime/size and always replace the whole file. The **file remains the source of truth** across process restarts. + ## Built on acp-kernel The compression engine is [`acp-kernel`](https://github.com/ranxianglei/acp-kernel) — a platform-agnostic, MIT-licensed library with 208 tests. It's bundled inline into `dist/index.js`, so there are zero runtime dependencies. diff --git a/package.json b/package.json index 27fbb80..38774f3 100644 --- a/package.json +++ b/package.json @@ -8,6 +8,10 @@ ".": { "types": "./dist/index.d.ts", "import": "./dist/index.js" + }, + "./contract": { + "types": "./dist/contract.d.ts", + "import": "./dist/contract.js" } }, "pi": { @@ -17,6 +21,7 @@ }, "files": [ "dist", + "schema", "README.md", "LICENSE" ], diff --git a/schema/bcp-block-v1.json b/schema/bcp-block-v1.json new file mode 100644 index 0000000..03617cf --- /dev/null +++ b/schema/bcp-block-v1.json @@ -0,0 +1,50 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://github.com/ranxianglei/billion-context-pi/schema/bcp-block-v1.json", + "title": "BcpBlockV1", + "description": "One ACP compression block as persisted in a .acp.json sidecar (an element of its top-level \"blocks\" array). Only the listed properties are part of the v1 stability contract: evolution is additive-only (new optional fields may appear), and renaming/removing/re-semanticking a listed field requires bumping the sidecar's schemaVersion. additionalProperties are permitted because real blocks carry extra internal pruning/index fields that are intentionally outside the contract.", + "type": "object", + "properties": { + "blockId": { + "type": "string", + "description": "Idempotency key. Stable across restarts; never reused." + }, + "summary": { + "type": "string", + "description": "The final accepted summary text." + }, + "tier": { + "type": "integer", + "enum": [1, 2, 3], + "description": "Compression tier." + }, + "compressedTokens": { + "type": "integer", + "minimum": 0, + "description": "Tokens of original content compressed by this block." + }, + "createdAt": { + "type": "integer", + "description": "Creation time, epoch MILLISECONDS (not an ISO string)." + }, + "topic": { + "type": "string", + "description": "Short human label for the compressed range." + }, + "startRef": { + "type": "string", + "description": "Model-facing start ref, e.g. m00005. Absent when unknown." + }, + "endRef": { + "type": "string", + "description": "Model-facing end ref, e.g. m00020. Absent when unknown." + }, + "effectiveMessageIds": { + "type": "array", + "items": { "type": "string" }, + "description": "Raw message ids covered by this block (authoritative coverage set)." + } + }, + "required": ["blockId", "summary", "tier", "compressedTokens", "createdAt"], + "additionalProperties": true +} diff --git a/src/contract.ts b/src/contract.ts new file mode 100644 index 0000000..e636ecd --- /dev/null +++ b/src/contract.ts @@ -0,0 +1,56 @@ +// Machine-readable contract for ACP compression blocks (#368). Zero runtime deps; +// safe to import across process/restart boundaries. The sidecar is the source of +// truth — this module only pins the versioned, additive-only block shape. + +export const SCHEMA_VERSION = 1 as const; + +export const PRODUCER_NAME = "billion-context-pi"; + +export interface ProducerInfo { + name: string; + /** Semver of the billion-context-pi build that wrote the sidecar. Omitted when unknown (dev/test). */ + version?: string; +} + +export interface SidecarEnvelope { + schemaVersion: number; + producer: ProducerInfo; +} + +/** Build the top-level envelope written to every `.acp.json`. */ +export function createSidecarEnvelope(version?: string): SidecarEnvelope { + return { + schemaVersion: SCHEMA_VERSION, + producer: { name: PRODUCER_NAME, ...(version ? { version } : {}) }, + }; +} + +/** + * One compression block as persisted in a sidecar's `blocks[]` element. + * + * v1 stability contract: listed fields are guaranteed present with stable names + * and semantics; evolution is additive-only (new optional fields may appear). + * Renaming/removing/re-semanticking any field requires bumping `SCHEMA_VERSION`. + * Fields not listed here are internal pruning/index details — not part of the + * contract and may change freely. + */ +export interface BcpBlockV1 { + /** Idempotency key. Stable across restarts; never reused. */ + blockId: string; + /** The final accepted summary text. */ + summary: string; + /** Compression tier (1 | 2 | 3). */ + tier: number; + /** Tokens of original content compressed by this block. */ + compressedTokens: number; + /** Creation time — epoch MILLISECONDS (not an ISO string). */ + createdAt: number; + /** Short human label for the compressed range. */ + topic?: string; + /** Model-facing start ref, e.g. "m00005". Absent when unknown. */ + startRef?: string; + /** Model-facing end ref, e.g. "m00020". Absent when unknown. */ + endRef?: string; + /** Raw message ids covered by this block (authoritative coverage set). */ + effectiveMessageIds?: string[]; +} diff --git a/src/state.ts b/src/state.ts index 7d9ba13..b65f0a3 100644 --- a/src/state.ts +++ b/src/state.ts @@ -2,9 +2,18 @@ import { promises as fs } from "node:fs"; import * as path from "node:path"; import { createInitialState, type CompressionState } from "acp-kernel"; import { logError, logInfo, logWarn } from "./log.js"; +import { createSidecarEnvelope, SCHEMA_VERSION } from "./contract.js"; const STATE_SUFFIX = ".acp.json"; +declare const CURRENT_VERSION: string; + +function producerVersion(): string | undefined { + return typeof CURRENT_VERSION !== "undefined" ? CURRENT_VERSION : undefined; +} + +const warnedUnknownVersions = new Set(); + export interface LiveRefOrigin { rawId: string; identity: string; @@ -60,7 +69,15 @@ export class SessionStateStore { if (file) { try { const raw = await fs.readFile(file, "utf8"); - const parsed = JSON.parse(raw) as CompressionState & { liveRefOrigins?: unknown }; + const parsed = JSON.parse(raw) as CompressionState & { liveRefOrigins?: unknown; schemaVersion?: unknown }; + // Tolerant forward-compat (#368): a higher schemaVersion means a future bcp wrote this + // file — we still load known fields below (refusing would drop state on downgrade) but + // log once per file so the downgrade is observable. Missing schemaVersion == v1. + const sv = typeof parsed?.schemaVersion === "number" ? parsed.schemaVersion : SCHEMA_VERSION; + if (sv > SCHEMA_VERSION && !warnedUnknownVersions.has(file)) { + warnedUnknownVersions.add(file); + logWarn("state", { event: "unknown-schema-version", file, found: sv, supported: SCHEMA_VERSION }); + } if (parsed && Array.isArray(parsed.blocks)) { state = mergeInitialState(parsed); liveRefOrigins = parseLiveRefOrigins(parsed.liveRefOrigins); @@ -100,7 +117,8 @@ export class SessionStateStore { }); const tmp = path.join(dir, `.acp-tmp-${path.basename(file)}`); try { - await fs.writeFile(tmp, JSON.stringify({ ...state, liveRefOrigins }), "utf8"); + const envelope = createSidecarEnvelope(producerVersion()); + await fs.writeFile(tmp, JSON.stringify({ ...envelope, ...state, liveRefOrigins }), "utf8"); await fs.rename(tmp, file); } catch (e) { logError("state", { event: "save-failed", file, error: e instanceof Error ? e.message : String(e) }); diff --git a/tests/contract.test.ts b/tests/contract.test.ts new file mode 100644 index 0000000..446acc4 --- /dev/null +++ b/tests/contract.test.ts @@ -0,0 +1,52 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFile } from "node:fs/promises"; +import * as path from "node:path"; +import { fileURLToPath } from "node:url"; +import { SCHEMA_VERSION, PRODUCER_NAME, createSidecarEnvelope, type BcpBlockV1 } from "../src/contract.js"; + +const here = path.dirname(fileURLToPath(import.meta.url)); + +// Must stay in sync with src/contract.ts BcpBlockV1 required fields and schema/bcp-block-v1.json. +const REQUIRED = ["blockId", "summary", "tier", "compressedTokens", "createdAt"]; +const OPTIONAL = ["topic", "startRef", "endRef", "effectiveMessageIds"]; + +test("SCHEMA_VERSION and PRODUCER_NAME are stable", () => { + assert.equal(SCHEMA_VERSION, 1); + assert.equal(PRODUCER_NAME, "billion-context-pi"); +}); + +test("createSidecarEnvelope: includes version when provided", () => { + assert.deepEqual(createSidecarEnvelope("0.1.65"), { + schemaVersion: 1, + producer: { name: "billion-context-pi", version: "0.1.65" }, + }); +}); + +test("createSidecarEnvelope: omits version when unknown (dev/test build)", () => { + const env = createSidecarEnvelope(undefined); + assert.equal(env.schemaVersion, 1); + assert.equal(env.producer.name, "billion-context-pi"); + assert.ok(!("version" in env.producer), "version key must be absent, not undefined"); +}); + +test("schema/bcp-block-v1.json stays in sync with the BcpBlockV1 contract", async () => { + const raw = await readFile(path.join(here, "..", "schema", "bcp-block-v1.json"), "utf8"); + const schema = JSON.parse(raw) as { + type: string; + required?: string[]; + additionalProperties?: boolean; + properties: Record; + }; + assert.equal(schema.type, "object"); + assert.deepEqual([...(schema.required ?? [])].sort(), [...REQUIRED].sort()); + assert.equal(schema.additionalProperties, true, "internal fields outside the contract must remain allowed"); + for (const key of [...REQUIRED, ...OPTIONAL]) { + assert.ok(schema.properties[key], `schema must define property ${key}`); + } +}); + +test("BcpBlockV1 exposes exactly the documented required core", () => { + const block: BcpBlockV1 = { blockId: "b1", summary: "s", tier: 1, compressedTokens: 10, createdAt: 172 }; + for (const key of REQUIRED) assert.ok(key in block, `BcpBlockV1 requires ${key}`); +}); diff --git a/tests/state.test.ts b/tests/state.test.ts index 049be9c..b1b9514 100644 --- a/tests/state.test.ts +++ b/tests/state.test.ts @@ -52,6 +52,38 @@ test("save then load round-trips state", async () => { await rm(dir, { recursive: true, force: true }); }); +test("save writes schemaVersion + producer envelope to the sidecar (#368)", async () => { + const dir = await tempDir(); + const file = path.join(dir, "session.json"); + const store = new SessionStateStore(); + const state = createInitialState(); + state.blocks.push(makeBlock("b0")); + await store.save(state, file, "sid"); + + const raw = JSON.parse(await fs.readFile(`${file}.acp.json`, "utf8")); + assert.equal(raw.schemaVersion, 1); + assert.equal(raw.producer.name, "billion-context-pi"); + assert.ok(Array.isArray(raw.blocks)); + assert.equal(raw.blocks[0].blockId, "b0"); + await rm(dir, { recursive: true, force: true }); +}); + +test("load tolerates a higher schemaVersion: warns once, still loads known fields (#368)", async () => { + const dir = await tempDir(); + const file = path.join(dir, "session.jsonl"); + await writeAcpState(file, [makeBlock("b0")], 2); + const acpFile = `${file}.acp.json`; + const bumped = JSON.parse(await fs.readFile(acpFile, "utf8")); + bumped.schemaVersion = 999; + await fs.writeFile(acpFile, JSON.stringify(bumped), "utf8"); + + const store = new SessionStateStore(); + const state = await store.load(file, "sid"); + assert.equal(state.blocks.length, 1, "known fields still load despite unknown-higher version"); + assert.equal(state.blocks[0]!.blockId, "b0"); + await rm(dir, { recursive: true, force: true }); +}); + test("load merges forward-compat: missing fields filled from fresh state", async () => { const dir = await tempDir(); const file = path.join(dir, "session.json"); diff --git a/tsup.config.ts b/tsup.config.ts index ef506c5..9c27266 100644 --- a/tsup.config.ts +++ b/tsup.config.ts @@ -6,7 +6,7 @@ const pkg = JSON.parse( ); export default defineConfig({ - entry: ["src/index.ts"], + entry: ["src/index.ts", "src/contract.ts"], format: ["esm"], target: "es2022", dts: false,