Repository navigation
mimir's first chat uses tool search too, so the next chat reuses its cache - #2343
Merged
Merged
Conversation
… reuses its cache
On mimir cloud the app turned tool search on only after a reply opened
with the proxy's mimir event, so the first chat after launch went out
with the 128-tool per-turn pick and every later chat with the deferred
catalog. The two tool lists share no prefix, so the next chat wrote the
whole prompt cache again.
Tool search is now on from the first request. A proxy deployed before
tool search refuses that request with a 400 ("tools must be a list of
at most 128", or "tool tool_search_tool_bm25_... is not allowed" for a
short catalog). On that refusal the client marks the transport as plain
and the runtime sends the same step once more with the per-turn pick,
without showing the refusal. It never resends twice, and any other
error behaves as before. The proxy's mimir event still clears the mark.
Other providers are unchanged.
Signed-off-by: subydoo-ci[bot] <343861189+subydoo-ci[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
On the mimir cloud connection, the app turned tool search on only after the proxy's first reply said it supported it, and each chat keeps the mode of its first request. So the first chat on a connection went out with every tool in full, and every later chat went out with tool search: a few tools loaded, the rest deferred, and the search tool. The two tool lists differ from the first entry. The prompt cache the first chat wrote was therefore never read by the next one, and the next chat wrote its whole prefix again.
The app now uses tool search from the first request on mimir cloud, so every chat sends the same tool list and the next chat reads the first one's cache.
Fallback for an older proxy. A proxy from before tool search refuses the request ("tools must be a list of at most 128", or the search tool "is not allowed"). On that refusal, the app turns tool search off for that connection, clears the chat's choice, and sends the same request once more the old way, with nothing shown to the person. Later requests on that connection stay the old way. If that proxy later says it takes tool search, the app turns it back on. Any other refusal is handled as before, and a request is never resent more than once. Other providers are unchanged.
Why
In recorded sessions, the short first chat wrote about 75,000 tokens to the cache and the design chat after it wrote about 37,000 again. Measured on one session, the first chat plus the next chat's first request cost about 64% less: 0.106 against 0.291 in dollars, at the API's prices. The next chat read 55,000 tokens from the cache instead of writing them.
For users
A design chat starts a little faster and costs less to run, most of all right after the app opens.
Expectations changed
How it was tested
New tests in packages/pilot/test/tool-search.test.ts (8):
On macOS:
pnpm -r typecheckpassed in every workspace.node scripts/check-dashes.mjson the changed files passes.On Linux:
Checklist
git commit -s)pnpm lint,pnpm typecheck,pnpm test,cargo clippy, andcargo testpasspackages/mcp/changes/, and no edit topackages/mcp/CHANGELOG.mdorpackages/mcp/TOOLS.md(a release writes both)