fix(responses): scope fallback prompt_cache_key per conversation

[Kizunad]

Problem

The session-ID fallback introduced in #1350 (fixes #1348) works well when one session maps to one conversation. But Claude Code multiplexes many concurrent conversations over a single session_id: every subagent (Task/Explore agent) runs its own conversation, while metadata.user_id carries the same session ID for all of them.

All these conversations therefore go upstream with the same prompt_cache_key and the same prompt head, so the provider routes them to the same cache shard, where their 100k+-token prefixes keep evicting each other.

Measured impact

Same gateway, same upstream channel (OpenAI Responses relay), same model, 3-day window:

client	inbound format	prompt_cache_key	requests	cache hit rate
Codex	openai/responses	per-thread UUID (client-provided)	20,502	96.8%
Claude Code	anthropic/messages	one per session (fallback)	3,744	32.5%

During a Claude Code subagent fan-out (~20 concurrent conversations), nearly every request hit exactly 4,608 or 5,632 cached tokens regardless of prompt size (11k–142k): only the static system+tools prefix survives, and every conversation tail misses on every turn.

With this patch deployed on the same setup (3 parallel subagents): 81% for the whole session including unavoidable cold starts, with steady-state turns at 95–99% (e.g. 152,064 cached / 152,686 prompt). Distinct per-conversation keys were confirmed in the stored outbound bodies.

Change

When the fallback applies (client provided no prompt_cache_key), append a deterministic conversation anchor to the session ID:

prompt_cache_key = <sessionID>-<hex(sha256(leading system/developer messages + first user message))[:16]>

• Later turns of one conversation keep the same key — the anchored head never changes mid-conversation.
• Sibling conversations in the same session diverge in their first user message → distinct keys → distinct shards.
• Client-provided keys are preserved untouched, so Codex and other Responses clients are unaffected.
• No key is emitted where none was emitted before — this only strengthens the existing fallback, so the third-party-backend regression concerns discussed in Draft: Add OpenAI cache identity support #1429 do not apply.

Why this cannot be fixed client-side

Unlike OpenCode (tracing plugin) or Hermes (client-side patch), Claude Code has no mechanism to attach per-conversation identity: metadata.user_id is session-scoped and shared by all subagents, and custom headers are static per process. The protocol translation layer is the only place that holds both the conversation content and the outbound field.

Tests

• Updated TestOutboundTransformer_TransformRequest_UsesSharedSessionIDAsPromptCacheKeyFallback for the new key shape.
• Added TestOutboundTransformer_TransformRequest_PromptCacheKeyScopedPerConversation: same conversation across turns → same key; sibling conversations → distinct keys; client-provided key untouched.
• cd llm && go build ./... && go test ./transformer/openai/responses/ && go vet ./transformer/openai/responses/ all pass.

---

中文摘要

Claude Code 的 subagent 会在同一个 session 里并发跑多个会话。#1350 的兜底让它们共用同一个 prompt_cache_key，上游把所有会话路由到同一个缓存分片，互相驱逐——实测同一上游同一模型：Codex（线程级 key）96.8% vs Claude Code（session 级 key）32.5%，subagent 并发期间每请求不论 prompt 多大都恒定只命中 ~5k token 的静态前缀。

本 PR 在兜底 key 上追加"会话锚点"（开头 system/developer 消息 + 首条 user 消息的 SHA-256 前 16 位十六进制）：同会话逐轮 key 稳定，不同会话自然分流。客户端显式传 key 时不受影响，也不会给原本收不到该字段的后端新增字段（#1429 中讨论的第三方后端回归风险不适用）。补丁部署后同场景实测：整体 81%（含冷启动），稳态轮次 95–99%。

与 OpenCode/Hermes 不同，Claude Code 没有任何客户端机制能按会话（subagent）注入标识——metadata.user_id 是 session 级的、自定义头是进程级静态的——因此只能在协议转换层修。

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR scopes fallback cache keys to individual conversations within a shared session. The main changes are:

• Adds a deterministic hash from leading instructions and the first user message.
• Includes supported text, image, video, document, audio, and compact content.
• Preserves client-provided cache keys.
• Adds tests for key stability, conversation separation, large instructions, and image inputs.

Confidence Score: 5/5

This looks safe to merge.

• The updated hash budget keeps the first user message in the fingerprint.
• Supported non-text content now contributes to conversation separation.
• No blocking issues were found in the changed code.

Important Files Changed

Filename	Overview
llm/transformer/openai/responses/cache_anchor.go	Adds the conversation fingerprint with per-content size limits and non-text content support.
llm/transformer/openai/responses/outbound.go	Scopes session-derived fallback cache keys while leaving explicit keys unchanged.
llm/transformer/openai/responses/outbound_test.go	Tests stable keys across turns, separate keys for sibling conversations, large instructions, image inputs, and explicit keys.

_{Reviews (2): Last reviewed commit: "fix(responses): harden conversation anch..." | Re-trigger Greptile}

[looplj]

只考虑 system 和 第一个 user message 是否足够

[Kizunad]

够的。这个 key 需要在同一会话的每一轮保持不变，而请求里逐轮不变的前缀恰好只有开头的 system/developer 消息 + 第一条 user 消息 —— 再多哈希后续消息，key 会逐轮变化反而破坏缓存。两个会话只有 system + 首条 user 完全一致时才会共用 key，此时它们的真实前缀本就相同，共用分片不会比现状差。

补丁在我的生产网关运行一周后的数据（Claude Code 流量，anthropic/messages 入站）：

窗口	请求数	prompt cache 命中率
修复前（07-16）	3,974	33.4%
修复后（07-19 ~ 07-26）	56,383	78.5%
最近 3 天	23,582	84.4%

≥50k token 的大请求平均命中从 70k 升到 98k（53.4% → 79.3%）。对照：同网关上自带 per-thread key 的 Codex 流量为 96.4%，余下差距主要来自会话首轮冷启动和 agent 重试。