Skip to content

fix: say a refused tool is not available and list only offered tools - #1293

Merged
eous merged 1 commit into
devfrom
fix/1287-unoffered-tool-refusal
Oct 6, 2026
Merged

eous merged 1 commit into
devfrom
fix/1287-unoffered-tool-refusal

Conversation

@eous

@eous eous commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Closes #1287.

Problem

A session's tools can change mid-conversation: an MCP server drops a tool or goes away
(_on_mcp_tools_changed), or another user sends on a shared workstream and the catalog follows the
sender (bind_acting_user). When the model then called such a tool, _prepare_tool_item refused
it with:

Unknown tool: 'X'. Available tools: …. Use one of the listed tool names exactly.

and the operator saw "Model called unknown tool". Both read as if the model had made the name up.
The list was every built-in preparer plus the sender's MCP tools, not what the request offered:
a regular session was told about the coordinator tools, a persona session about tools its persona
hides, a coordinator about tools an operator revoked, and a tool-search session about every
deferred tool.

Change

_unavailable_tool_error builds the refusal:

  • It says the tool is not available now, and names the possible causes: removed since it was
    offered; available only to another user, once the workstream is shared (_shared_workstream);
    misspelled, unless the name was offered exactly.
  • It lists the other tools the request offers outright: the session's active tools without
    deferred and revoked ones, or a task agent's own tools, which
    _prepare_tool_for_principal(offered=...) carries for the agent loop. With tool search on, it
    says more may be found by searching.
  • The operator error reads "Model called a tool that is not available now: 'X'", and the tool card
    header "X: not available".
  • _get_deferred_names, which the refusal and every request read, reads the tool search manager
    once, as _get_active_tools does. A catalog refresh on the MCP thread could drop it between the
    check and the read and raise AttributeError: on a free-threaded build, or on the standard build
    for a caller that passes no caps, since _get_capabilities() then runs in between. Both
    current callers pass caps, so the shipped image could not reach it.

Deferred tools are left to tool search rather than listed. A live probe found that listing them
would be safe (neither native model called an unloaded deferred tool directly: claude-opus-5-5
declined, gpt-6-astra searched first), but it would save nothing, and tool search exists to keep a large catalog out
of the context.

Correction to the issue

The issue says that on OpenAI the replayed tool_search_output makes the API treat a gone tool as
loaded. Live, it does not: gpt-6-astra never called the gone tool in 6 of 6 runs, even when asked
for it by exact name. Its hosted tool search found nothing and it declined, so on that lane the
request's own tools already tell the model, and this refusal is not reached.

Validation

  • Live smoke through real ChatSession turns with a fake MCP server that drops a weather tool
    between turns (temporary SQLite; tools that touch the machine refused in the harness):

    Lane Gone tool called Outcome
    Qwen on vLLM, tool search off yes refusal, accurate reply, 200
    Qwen on vLLM, client-side tool search yes refusal, searched twice, accurate reply, 200
    claude-opus-5-5, native tool search yes (the fix: define every tool a replayed Anthropic tool search names #1286 stub) refusal, accurate reply, 200
    gpt-6-astra, native tool search no, 6 of 6 hosted search found nothing, declined

    Qwen and gpt-6-astra were rerun on the final code. claude-opus-5-5 ran on an earlier revision,
    whose refusal also named another user; the request it sends has not changed since. In 1 of
    the 6 gpt-6-astra runs, once its search came back empty, the model called its own earlier,
    real result invalid. That comes from the OpenAI replay of fix: replay OpenAI hosted search items with their tool namespaces #1285, not from this change; the
    findings are on OpenAI Responses replay omits hosted tool items, so the model loses its own searches and re-runs tool search #1281.

  • An earlier wording named "another user" in every refusal, and a live Qwen reply passed that
    guess on to a one-person chat; it is now named only on shared workstreams, and a live recheck
    shows the reply without it.

  • tests/test_unavailable_tool_refusal.py: the offer list, revoked tools, personas, native and
    client-side tool search, shared and single-user causes, the empty offer, a task agent through
    _run_agent, that an agent's offer does not outlive its call, and deferred names surviving the
    manager being dropped mid-read (fails against the old two-read body). Three tests that pinned
    the old wording are updated.

  • Mutation controls: each of 14 breaks of the new behavior fails a test.

  • Full suite passes; ruff and mypy clean.

Not changed

  • The refusal's list is eventually consistent: it reads the catalog with the same two unlocked
    reads as the request path, so a catalog swap landing between them can skew one refusal's list,
    and the next one reads the new catalog. Locking would put MCP manager calls under the session
    lock, which the MCP code avoids by design.
  • A task agent's call to a name outside its list keeps its own refusal ("not available in agent
    mode").

A session's tools can change mid-conversation: an MCP server drops a tool
or goes away, or another user sends on a shared workstream and the catalog
follows the sender. The model can still call such a tool, from its history
or from a replayed tool search, and dispatch refused it as "Unknown tool"
with a request to use a listed name exactly, as if the model had made the
name up. The list named every built-in preparer and the sender's MCP tools,
not what the request offered: coordinator tools in a regular session, tools
a persona hides or an operator revoked, and deferred tools left to tool
search.

The refusal now says the tool is not available now and gives the possible
causes: removed since it was offered; available only to another user, once
the workstream is shared; or misspelled, unless the name was offered
exactly. It lists the tools the request offers outright, without revoked
ones, and points to tool search when more can be found. A task agent's
refusal lists the agent's own tools. The operator's error and the tool
header say the same.

_get_deferred_names, which the refusal and every request read, now reads
the tool search manager once, as _get_active_tools does. A catalog refresh
on the MCP thread could drop it between the check and the read, raising
AttributeError, on a free-threaded build or for a caller passing no caps.

Validation: live through real ChatSession turns, with a fake MCP server
that drops a tool between turns. Qwen on vLLM (tool search off and
client-side) and claude-opus-5-5 called the gone tool, got the refusal and
explained it accurately, and the requests after it returned 200.
gpt-6-astra never called it (4 of 4): its hosted tool search found nothing
and it declined.
@eous
eous merged commit 9997f3b into dev Oct 6, 2026
15 checks passed
@eous
eous deleted the fix/1287-unoffered-tool-refusal branch October 6, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When the model calls a tool it no longer has, say so instead of "Unknown tool"

1 participant