Skip to content

mimir's first chat uses tool search too, so the next chat reuses its cache - #2343

Merged
subydoo-ci[bot] merged 1 commit into
mainfrom
fix/first-chat-tool-search
Nov 15, 2026
Merged

subydoo-ci[bot] merged 1 commit into
mainfrom
fix/first-chat-tool-search

Conversation

@subydoo-ci

@subydoo-ci subydoo-ci Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

What this changes

On the mimir cloud connection, the app turned tool search on only after the proxy's first reply said it supported it, and each chat keeps the mode of its first request. So the first chat on a connection went out with every tool in full, and every later chat went out with tool search: a few tools loaded, the rest deferred, and the search tool. The two tool lists differ from the first entry. The prompt cache the first chat wrote was therefore never read by the next one, and the next chat wrote its whole prefix again.

The app now uses tool search from the first request on mimir cloud, so every chat sends the same tool list and the next chat reads the first one's cache.

Fallback for an older proxy. A proxy from before tool search refuses the request ("tools must be a list of at most 128", or the search tool "is not allowed"). On that refusal, the app turns tool search off for that connection, clears the chat's choice, and sends the same request once more the old way, with nothing shown to the person. Later requests on that connection stay the old way. If that proxy later says it takes tool search, the app turns it back on. Any other refusal is handled as before, and a request is never resent more than once. Other providers are unchanged.

Why

In recorded sessions, the short first chat wrote about 75,000 tokens to the cache and the design chat after it wrote about 37,000 again. Measured on one session, the first chat plus the next chat's first request cost about 64% less: 0.106 against 0.291 in dollars, at the API's prices. The next chat read 55,000 tokens from the cache instead of writing them.

For users

A design chat starts a little faster and costs less to run, most of all right after the app opens.

Expectations changed

  • packages/pilot/test/tool-search.test.ts: three tests that pinned the first request without tool search now pin the first request with it, plus the fallback for a proxy from before tool search.
  • packages/pilot/test/sticky-toolsets.test.ts, packages/app/test/mimir-tool-cap.test.ts and packages/app/test/mimir-sticky-toolsets.test.ts: the cases that test the per-turn tool pick now use a connection that refuses tool search the way an older proxy did, with its own refusal text.

How it was tested

New tests in packages/pilot/test/tool-search.test.ts (8):

  • a fresh connection's first request carries the deferred catalog and the search tool, byte for byte the same tool list as a later chat;
  • an older proxy's refusal gets exactly one resend the old way, with no error shown, and later turns, a new chat and a new session on that connection stay the old way;
  • a refusal of the search tool alone is resent once too;
  • other refusals are not resent: a different 400, today's 400 for too many loaded tools, a 502 with the old words, and a 400 with the old words but another code;
  • a request is resent at most once;
  • other providers' first requests are unchanged;
  • the older proxy's refusal is recognized from the desktop and browser connections, and nothing else is;
  • the proxy's tool search event is heard from mimir cloud only, takes back a refusal, and is kept per connection.

On macOS:

  • pnpm -r typecheck passed in every workspace.
  • packages/pilot: 1,748 tests passed and 33 skipped.
  • packages/app tests that mention tool search or the provider: 144 files, 1,424 tests passed and 149 skipped.
  • node scripts/check-dashes.mjs on the changed files passes.

On Linux:

  • Typecheck passed for contracts, mcp, pilot and app.
  • packages/pilot: 1,781 tests passed.
  • packages/app mimir suites: 964 passed and 23 skipped.
  • packages/mcp: 2,352 passed and 42 skipped.

Checklist

  • Commits follow Conventional Commits and are signed off (git commit -s)
  • pnpm lint, pnpm typecheck, pnpm test, cargo clippy, and cargo test pass
  • Tests added or updated
  • Screenshot or recording attached for UI changes
  • Docs updated if behavior or a public interface changed
  • MCP server changes: a file in packages/mcp/changes/, and no edit to packages/mcp/CHANGELOG.md or packages/mcp/TOOLS.md (a release writes both)
  • Any code from another project is credited, and its license is compatible with Apache-2.0
  • No secrets, keys, or private model files in the diff
  • AI tools helped with this change (name them here: ...), and I have read CONTRIBUTING-AI.md

… reuses its cache

On mimir cloud the app turned tool search on only after a reply opened
with the proxy's mimir event, so the first chat after launch went out
with the 128-tool per-turn pick and every later chat with the deferred
catalog. The two tool lists share no prefix, so the next chat wrote the
whole prompt cache again.

Tool search is now on from the first request. A proxy deployed before
tool search refuses that request with a 400 ("tools must be a list of
at most 128", or "tool tool_search_tool_bm25_... is not allowed" for a
short catalog). On that refusal the client marks the transport as plain
and the runtime sends the same step once more with the per-turn pick,
without showing the refusal. It never resends twice, and any other
error behaves as before. The proxy's mimir event still clears the mark.
Other providers are unchanged.

Signed-off-by: subydoo-ci[bot] <343861189+subydoo-ci[bot]@users.noreply.github.com>
@subydoo-ci subydoo-ci Bot added enhancement New feature or request roadmap:cad Roadmap feature: cad labels Nov 15, 2026
@subydoo-ci
subydoo-ci Bot added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 12fe415 Nov 15, 2026
96 checks passed
@subydoo-ci
subydoo-ci Bot deleted the fix/first-chat-tool-search branch November 15, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request roadmap:cad Roadmap feature: cad

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants