fix(acp): deliver plain-text replies for self-hosted OpenAI-compatible agents - #4874
fix(acp): deliver plain-text replies for self-hosted OpenAI-compatible agents#4874repudi8or wants to merge 2 commits into
Conversation
buzz-agent's contract is that output is tool calls: streamed assistant text
is observability only and never published. Capable models honour that and
call `send_message`. Small local models served over shared compute do not
always — on multi-step turns they run their tools, then write the answer as
prose and end the turn, so the turn reports success with nothing in the
channel.
Retain the turn's streamed text in AcpClient (bounded at 8 KiB) alongside a
flag for whether a publish tool call was seen, and on EndTurn publish that
text as a threaded channel reply when the agent published nothing itself.
Gated on EndTurn only: MaxTokens / MaxTurnRequests mean the turn was
truncated, so the text is not a deliberate answer. Bare acknowledgements
("OK", "Done") are filtered, and taking the buffer clears it so the same
text can never post twice.
Off by default via --deliver-plain-replies / BUZZ_ACP_DELIVER_PLAIN_REPLIES:
for cloud models, which reliably publish their own replies, posting streamed
text would double-post. The desktop shared-compute preset opts in, so this
stays scoped to the case that needs it.
Verified end to end against a live relay with stub ACP agents that pin each
branch deterministically: a prose-only agent gets its answer delivered, and
an agent that calls send_message while also streaming prose posts exactly
once with the fallback silent.
Forward-ported onto main from the unmerged origin/micspiral/mesh-0-74-gemma
branch (bb4af41). This exact bug was hit in production by the "Local LLM
(llm1)" self-hosted mesh-llm/llama-server agent: turns ended with EndTurn
and correct prose visible in the ACP activity log, but nothing was published
to the channel because send_message was never called.
Signed-off-by: Michael Neale <michael.neale@gmail.com>
with a helping hand from Claude Code
Signed-off-by: Brett Meehan <repudi8or@gmail.com>
…ly delivery
The plain-reply-delivery fallback landed for the built-in relay-mesh
("Buzz shared compute") preset only, gated on provider == "relay-mesh"
in apply_relay_mesh_env. A self-hosted model reached the same way small
local models always are — the "OpenAI-compatible" provider pointed at a
user's own llama.cpp/vLLM/mesh-llm endpoint instead of the built-in
preset — never opted in, so the exact same silent-drop bug reproduced:
the model answers in prose, ACP's activity log shows a correct reply,
and nothing is ever published to the channel.
Default BUZZ_ACP_DELIVER_PLAIN_REPLIES to "true" for effective provider
"openai-compat" too, in the same effective-env assembly step, right
after the relay-mesh translation. Scoped narrowly to "openai-compat":
plain "openai" (the real OpenAI cloud API) is left untouched, since
cloud models reliably call send_message themselves and enabling the
fallback there would risk double-posting. An explicit user-set value
(including an intentional "false") is preserved, matching the emptyness
convention used elsewhere in this module.
Reproduced against the "Local LLM (llm1)" agent in production: provider
"openai-compat", OPENAI_COMPAT_BASE_URL pointed at a remote llama-server
instance — confirmed via the agent's own config screenshots that it is
not on the relay-mesh preset, so relay_mesh.rs's existing default never
applied to it.
with a helping hand from Claude Code
Signed-off-by: Brett Meehan <repudi8or@gmail.com>
|
@wesbillman this fixes a bug affecting self-hosted OpenAI-compatible agents (e.g. local llama.cpp/mesh-llm endpoints) silently dropping their replies instead of publishing them — full context and test results in the PR description. Would appreciate a review/merge when you have a moment. I only have read access to block/buzz directly (opened from a fork), so can't self-merge. |
Live end-to-end repro: confirmed ✅Independently verified the events directly on the relay (not just trusting the report):
This closes the one unchecked item in the test plan: a self-hosted Non-blocking follow-up spotted in the reply content (not the delivery mechanism, which is what this PR fixes): the model looped through many near-duplicate self-summaries for the full turn, and the delivered text is hard-cut mid-word — consistent with hitting the 8KiB |
2 participants
Summary
send_message— the turn reportsEndTurnsuccessfully, the ACP activity log shows the correct reply, but nothing is ever published to the channel. This is what was happening in our mesh-llm deployed "Local LLM (llm1)" agent.crates/buzz-acp: retain each turn's streamed text (bounded at 8 KiB) plus whether a publish tool call was seen; onEndTurn, if nothing was published, deliver that text as a threaded channel reply instead. Gated strictly onEndTurn(notMaxTokens/MaxTurnRequests, which mean the turn was truncated) and filters bare acknowledgements ("OK", "Done"). Off by default via--deliver-plain-replies/BUZZ_ACP_DELIVER_PLAIN_REPLIES, since cloud models reliably callsend_messagethemselves and enabling it there would risk double-posting.desktop/src-tauri: opt the built-in "Buzz shared compute" (relay-mesh) preset into this fallback, and separately default it on for the generalopenai-compatprovider too (a self-hosted endpoint configured directly, not through the built-in preset) — scoped narrowly so plainopenai(the real OpenAI cloud API) is untouched. An explicit user override (including an intentionalfalse) always survives.Test plan
cargo test --manifest-path desktop/src-tauri/Cargo.toml --lib— 2216 passed, 0 failed, 13 ignored (full crate, not scoped to touched files), run independently by two peoplepublish_detection_matches_tool_and_shell_shapes,bare_acknowledgements_are_not_worth_publishing,non_mesh_provider_does_not_opt_into_plain_reply_delivery,openai_compat_provider_opts_into_plain_reply_delivery,real_openai_provider_does_not_opt_into_plain_reply_delivery,explicit_deliver_plain_replies_override_is_preserved_for_openai_compatbuzz-acp(bundled inside the Buzz Desktop app) anddesktop/src-tauri, and there's no way to swap just this agent's binary without patching the signed Desktop app bundle used by every other agent on that machine. Recommend validating live once this ships through the normal Desktop build/update path.Diagnosed and coordinated by the buzz-acp team (architect, dev, tester, conductor) in the
lmlogic-local-llmchannel; core mechanism forward-ported from an unmerged branch (micspiral/mesh-0-74-gemma, commitbb4af4189) by Michael Neale.with a helping hand from Claude Code