wire codecs: silent image loss — openai single remote-URL image dropped; responses image-only user item dropped

[ranxianglei]

来源: ranxianglei/billion-context#488 分析 multimodal payload accounting 时发现（billion-context PR #489 review）

Two silent image-loss gaps in the wire codecs (v0.0.47, dist/wire/index.js). Both drop image content from rebuilt request bodies without any error.

1. openaiToCore: single non-data-URL image gets no sidecar → dropped on re-encode

openaiToCore case "user" (dist/wire/index.js:243-257):

...imgs.length === 1 && firstParsed ? { rawOpenaiContent: imgs[0], imageMediaType: firstParsed.mediaType, imageBase64: firstParsed.base64 }
 : imgs.length > 1 ? { rawOpenaiContentParts: imgs } : {}

For a user message containing exactly ONE image whose url is a remote http(s) URL (not a data: URL): firstParsed = parseDataUrl(url) is undefined → first branch false; imgs.length > 1 false → no sidecar field at all. coreToOpenai then rebuilds content from text alone — the image is silently lost from the forwarded request. Data URLs and multi-image messages are preserved; only the single-remote-URL shape drops.

Repro: {messages:[{role:"user",content:[{type:"text",text:"t"},{type:"image_url",image_url:{url:"https://example.com/a.png"}}]}]} through openaiToCore → coreToOpenai: output content has no image part.

Suggested fix: keep the raw part whenever there is exactly one image, splitting out base64 only when it is a data URL:

...imgs.length === 1
  ? { rawOpenaiContent: imgs[0], ...(firstParsed ? { imageMediaType: firstParsed.mediaType, imageBase64: firstParsed.base64 } : {}) }
  : imgs.length > 1 ? { rawOpenaiContentParts: imgs } : {}

coreToOpenai already re-emits rawOpenaiContent verbatim when present (dist/wire/index.js:355-367), so no encode-side change is needed.

2. responsesToCore: image-only user item produces no core message → dropped from rebuilt input

responsesToCore case "message" (dist/wire/index.js:~505-531): the user branch computes effText = messageContent(item.content) where partText returns "" for non-text parts, and only pushes a core message when if (effText) is truthy. A user message item whose content is exclusively input_image parts (no input_text) yields effText === "" → no core message → coreToResponses never sees it → the item (and its images) is absent from the rebuilt input array.

Repro: {input:[{type:"message",role:"user",content:[{type:"input_image",image_url:"data:image/png;base64,AAA="}]}]} through responsesToCore → zero core messages.

Note the contrast with the Anthropic codec, which handles this correctly: anthropicToCore gives image blocks the placeholder text "[image]" plus a rawAnthropicBlock sidecar, and coreToAnthropic re-emits the raw block unconditionally — so image-only Anthropic content survives the round trip.

Fix needs to span both sides of the Responses codec: e.g. push a placeholder-text core message for image-bearing items with no text while keeping rawResponsesItem, and teach coreToResponses to re-emit the raw item verbatim for it (its current verbatim condition messageContent(raw.content) === message.text cannot match an image-only item because its content text is "").

Impact

For billion-context proxy mode (the only consumer that round-trips through these codecs): any OpenAI-chat or Responses client sending images in these shapes has them silently removed from what is forwarded upstream — the model never sees the image, with no error anywhere. A host-side size estimator counting the raw body will overcount relative to what is actually forwarded (conservative direction for fit-gates, but inconsistent with self-heal learning, which measures the rejected forwarded body).

[ranxianglei]

🤖 ework agent · qwen3.8-27b

[bot] 🏷 已修复并提交 PR：**https://github.com/ranxianglei/acp-kernel/pull/189**（分支 2026-09-03_wire-image-loss-fixes，基于 master v0.0.49）。

Triage 结论

查重：按关键词检索现有 issue/PR，无重复项。

两个 bug 均在 master v0.0.49 上复现确认（不是只凭描述开工），但机制细节与 issue 描述有两处出入：

1. Gap 1 (openai)：issue 指向 imgs.length === 1 && firstParsed 条件，但当前代码在上游还有一层过滤——allImageParts 只收集 url 可解析为 data URL 的图片 part（626f933 为喂 rawOpenaiContentParts 引入）。所以单个 remote-URL 图片根本到不了 sidecar 逻辑（imgs.length === 0）。终态与报告一致，但修复必须同时移除该过滤。且实际缺口比报告的更宽：同一过滤导致多图消息里所有 remote-URL part 都被丢弃——[data, remote] 重建后只剩 [data]，不只是单图场景。
2. Gap 2 (responses)：如描述复现。一个补充：丢失是路径相关的——patchResponsesInput 通过 layout fallback（无 coreId 的 slot → 原样保留）其实保住了这类 item，而 coreToResponses 会丢；且无论哪条路径，未跟踪的 item 对压缩管线不可见，会无界累积。跟踪化修复同时解决两条路径。

层次判断：两处都是 codec 侧 sidecar 遗漏（根因在编码侧，不在宿主或上游），issue 给的方向打到了核心，已采纳并做如下细化。

实现要点

• openai：allImageParts 收集所有 string-url 的 image_url part；单图一律带 rawOpenaiContent 原样 part，仅 data URL 才拆 base64/mediaType（即 issue 建议的形状）；coreToOpenai 无需改动（优先级链本来就优先 verbatim raw）。image-only 消息保持空文本（预存行为，改占位符会移动已工作形状的派生 id）。
• responses：image-only user item 跟踪为 "[image]" 占位文本（anthropic codec 先例）+ rawResponsesItem sidecar，进入压缩空间、id 跨 turn 稳定；coreToResponses 在 kernel 文本仍等于 canonicalUserText(raw) 时原样重发 raw item，被编辑/摘要的消息照旧重建为纯文本。开发中曾踩到一个回归（displayText 从 raw content 而非 split 后的 effText 推导，破坏了 demoted-thinking 的 identity space），已修正并有测试锁定。
• id churn 范围：唯一变化是 ≥2 张图、无文本的 item 从 "\n" join 伪影变为 "[image]"（一次性、确定性）；其余所有已工作形状 id 逐字节不变。

验证

• 新增 9 个回归测试（tests/wire-bili-message-roundtrip.test.ts 5c–5e、6b–6g）：master 上全部失败，修复后通过（用 pristine HEAD 副本验证过失败签名）。
• npm run typecheck ✓ · npm test 581/581 ✓ · npm run build ✓
• 两个独立 agent review（正确性/id 稳定性 + 集成/消费方角度），消费方审计确认无其他 kernel 模块依赖被改行为。

顺带发现（未在本 PR 处理）

• 新 issue：openaiToCore 遇 {type:"image_url", image_url:null} 抛 TypeError（src/wire/openai.ts:271）——预存在、与本修改无关，已单独提 openaiToCore 崩溃：image_url 为 null 的 image_url part 触发 TypeError #188
• 仅报告、待你拍板：assistant-role image-only response item 仍未跟踪（同类缺口反向）；mirror builder 无法表达图片（MirrorBlock 无 image 类型，data-URL 图同样受影响）；字面文本 "[image]" 与占位符共享 base id（ClusterCounter 确定性消歧，message-id 既有取舍）。

PR 等你 review 后由你合并。