A mid-run change to the selected tool set moves the head of the system prompt and re-bills the conversation

[tinoy1336]

What happened?

A change to the selected tool set made while a run is in progress moves the head of the
system prompt, so the run's next request re-bills from that point. The head is where the
- <tool>: <snippet> bullets are rendered, and it re-derives the selection from the live
set every turn, so adding or removing one selected tool rewrites it.

Measured on one session (72 responses, provider deepseek): five requests followed such a
change, each exactly one tool bullet (266 characters); a sixth head move came from the
injected path described in the comment below. Two of the six re-billed the whole prompt —
156,362 and 192,928 input tokens, with only 1,664 and 128 tokens cached. The other four
re-billed 2.7–17.8% of the prompt.

Steps to reproduce

Validated with pi -ne, so no ambient extensions were loaded. One small extension is
enough — it registers a tool with a promptSnippet, selects it for the run, and removes it
from the live set during the first tool call:

// repro.ts — pi --mode rpc -ne -e ./capture.ts -e ./repro.ts
import { Type } from "typebox";

export default function (pi: any) {
  const TOOL = "probe_enable";
  pi.registerTool({
    name: TOOL,
    description: "Scratch tool that exists so the system prompt can carry a bullet for it.",
    parameters: Type.Object({}, { additionalProperties: false }),
    promptSnippet: `${TOOL} is installed. Call it to enable the scratch tools.`,
    async execute() { return { content: [{ type: "text", text: "ok" }] }; },
  });

  let removed = false;
  pi.on("tool_call", () => {
    if (removed) return undefined;
    removed = true;
    pi.setActiveTools(pi.getActiveTools().filter((n) => n !== TOOL));
    return undefined;
  });

  pi.on("before_agent_start", (event) => {
    const active = pi.getActiveTools();
    if (!active.includes(TOOL)) pi.setActiveTools([...active, TOOL]);
    const selected = event.systemPromptOptions?.selectedTools;
    if (Array.isArray(selected) && !selected.includes(TOOL)) selected.push(TOOL);
    return undefined;
  });
}

Capture what the provider is sent (wrap globalThis.fetch and write init.body to a
file), then send one prompt that makes the model call any tool:

• before the tool call: system prompt 13,796 chars, - probe_enable: … present;
• after it: 13,716 chars, and the only difference in the system prompt is that bullet
  line (79 chars + newline);
• later requests stay there, so it is billed once — but the whole head moved.

Expected behavior

A tool-set change made during a run should not rewrite the head of a prompt that has
already been sent, or — if it must — the cost should not be the conversation behind it.
Rendering the bullets from the same set the request will carry would close the
prompt/payload half; applying selection changes at a run boundary would close the rest.
Happy to prepare a PR if that is the direction you prefer.

Version

0.87.1 (Linux; provider deepseek, model deepseek-flash, openai-completions).

[tinoy1336]

AI-assisted draft (per CONTRIBUTING.md): this report was drafted with an AI coding assistant
running locally, under my direction. The reproduction above is a self-contained extension,
and every number in it comes from captured request bodies or the session's own usage
records.

The injected-path half of the report, with its numbers: a run started by an injected
message (sendMessage(…, {triggerTurn: true}) while the session is idle) fires no
before_agent_start and renders the base options rather than the run's own selection. In a
session that also forced the system prompt, the typed request's head was 170,860 characters
and the injected one's 170,594 — the sole difference being one 266-character tool bullet
line, nothing else in the prompt.

One practical note for reproducing that half: injecting synchronously from an agent_end
handler does not show it, because the run has not settled at that point and the message is
folded into it. Defer until the session is idle (a pi --mode rpc session driven through a
FIFO works well), or wait a few seconds after the run ends.

[github-actions]

This issue was auto-closed. All issues from new contributors are auto-closed by default.

Maintainers review auto-closed issues daily and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.

If a maintainer replies lgtmi on one of your issues, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open. The command must be at the start of the reply (optionally after one or more @username mentions) or at the end.

See CONTRIBUTING.md.

[davidbrai]

to check: maybe we should add supportsMidConvoSystemMessages to deepseek flash

[dtm-beep]

Adding evidence from the llama.cpp / local-LLM side, where this head-rewrite mechanism is fully reproducible, and where the per-model opt-in has a hard constraint.

1. Serving Qwen3.8-Flash-Next (MoE, llama.cpp server, api: openai-completions, pi 0.87.1). On a local server a head move is a full KV-cache re-pay, not just re-billing. One follow-up arrived with f_keep = 0.175: 29,815 tokens re-prefilled at 509 t/s, 58.6 s of pure cache miss in one turn. Server-side context checkpointing is the only mitigation available on our side, and it cannot rescue a rewrite that reaches the first turns.

2. Before adding supportsMidConvoSystemMessages to any llama.cpp model: our chat template rejects mid-conversation messages of that kind outright. Both role:"system" and role:"developer" placed after the first turn fail with Jinja Exception: System message must be at the beginning (verified against the Qwen3.8 template through llama.cpp /v1/chat/completions). So a blanket opt-in for local models would turn cache misses into hard 500s. A per-model flag needs a template-tolerance check first, or the update could be rendered as a user-role message on non-tolerant templates.