fix(responses): keep images in tool results instead of dropping them

[GhostFlying]

The bug

convertToolMessageWithType (outbound_convert.go:301) only copies text parts when turning an internal role:"tool" message back into a Responses function_call_output. An image_url part is skipped, and a tool result that carried nothing but an image then falls through to the empty-string fallback a few lines below:

for _, p := range msg.Content.MultipleContent {
    if p.Type == "text" && p.Text != nil {   // image_url silently skipped
        ...
    }
}

// Some times the tool result is empty, so we need to add an empty string.
if output.Text == nil && len(output.Items) == 0 {
    output.Text = lo.ToPtr("")               // ...and the image becomes ""
}

So the model receives {"type":"function_call_output","call_id":"...","output":""} — a successful, empty tool result, indistinguishable from a tool that genuinely returned nothing.

Every other step of the path is already correct, which is what makes this a one-function bug:

stage	image preserved?
inbound convertContentItemToPart (inbound.go)	yes — input_image → image_url part
internal llm.Message{Role:"tool"} MultipleContent	yes
outbound convertToolMessageWithType	no

convertUserMessage in the same file has a case "image_url" branch, and the Anthropic outbound transformer preserves tool-result images via convertToAnthropicTrivialContent (including data-URL → base64 source). Only this converter drops them.

Spec basis

From the official spec (openai/openai-openapi, main):

• FunctionCallOutputItemParam.output — oneOf a string, or "An array of content outputs (text, image, file) for the function tool call" whose items are InputTextContentParam | InputImageContentParamAutoParam | InputFileContentParam.
• FunctionToolCallOutput.output / CustomToolCallOutput.output — "Can be a string or an list of output content.", items FunctionAndCustomToolCallOutput = InputTextContent | InputImageContent | InputFileContent.

Images in a tool result are a documented, first-class part of the API for both function_call_output and custom_tool_call_output.

Why this went unnoticed

Two things hide it:

1. The failure is silent and looks successful. The gateway returns 200 and the model gets a well-formed empty tool result, so it does what an LLM does with an empty result: it answers anyway. There is no error, no truncation, nothing in the logs.
2. The most common client path is unaffected. A human attaching an image goes through a user message input_image, which works. Only a tool that reads an image itself hits the broken field.

Reproduction

Client: Codex CLI 0.144.3 → /v1/responses, model gpt-5.5. Codex's view_image tool returns a single base64 image with no text, i.e. exactly the shape that degrades to "".

Payload: N solid squares of one colour (N random 2..7, colour random of 6), so a correct answer cannot be guessed — 1/36 by chance. Same PNG generator in all arms.

image delivery	correct
user message input_image (codex -i)	1/1
view_image → function_call_output (before)	0/3
view_image → function_call_output (after this patch)	3/3

Before the patch the wrong answers were stereotyped ("3 red" twice out of three) — no visual information reached the model at all. Also reproduced 0/3 on gpt-5.6-sol before, 2/2 after.

In a real agent loop the effect was worse than a wrong colour: the model called view_image three times per run, received "" each time, and then wrote confident descriptions of a rendered figure it had never seen — including describing a deliberately blank image as readable.

The change

• text-only if becomes a switch, with case "image_url" → input_image, mirroring convertUserMessage.
• detail is now always populated (lo.FromPtrOr(p.ImageURL.Detail, "auto")). This matters because of an asymmetry in the request schemas: a custom_tool_call_output content array resolves to InputImageContent, whose required is ["type","detail"], while function_call_output resolves to InputImageContentParamAutoParam, where detail is optional. Both document auto as the default, so sending it explicitly is valid on either side and avoids branching on the item type.

Tests

TestConvertToolMessage gains: image-only result, image + unsupported parts, a custom_tool_call_output carrying an image. Two new tests assert the serialised wire shape (that output is an array of content parts, and that a custom tool output's image carries detail).

Two existing assertions changed, since they pinned the drop:

• "tool message with multiple content - mixed types (only text extracted)" expected text/image/text to collapse to text/text; it now asserts the image survives in place, and is renamed accordingly.
• "tool message with multiple content but no text parts" expected an image to become ""; it now asserts the image survives, and is renamed.

go test ./... in the llm module: 34 packages ok. gofmt/go vet clean.

Deliberately not in this PR

• Chat Completions outbound has the mirror-image bug. MessageContentFromLLM (openai/outbound_convert.go:217) is role-agnostic and passes image_url straight into tool messages, but ChatCompletionRequestToolMessage.content says "For tool messages, only type text is supported" — so a strict upstream may 400. Whether to drop, annotate or reject there is a design call, and the compatibility surface is different, so it seemed wrong to bundle it here. Happy to open a separate issue.
• Unrepresentable part kinds are still dropped silently. input_audio / video_url / document in a tool result still vanish, and an all-dropped result still degrades to "". Surfacing that properly needs a logger or an error return, which this pure converter has no access to — a separate change. The audio-only case is kept as a test with a comment so the gap is explicit rather than looking intentional.
• input_file. The spec allows it in the output array, but the Responses Item type has no file fields at all (neither inbound nor outbound), so it is a symmetric feature gap rather than an asymmetric drop.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Tool results now preserve image content when sent through the Responses API, including input_image with detail defaulting to "auto".
  • Mixed text and image results maintain their original order (including custom tool outputs).

• Bug Fixes

  • Image-only tool results are no longer discarded.
  • Tool results containing only unsupported content now fall back to an empty-string output instead of showing a placeholder.
  • Truly empty/no-content tool results are still handled correctly.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2ccf1fed-08c1-46af-ab15-31b949eb69fb

📥 Commits

Reviewing files that changed from the base of the PR and between a7399a3 and 94ba2a5.

📒 Files selected for processing (2)

• llm/transformer/openai/responses/outbound_convert.go
• llm/transformer/openai/responses/outbound_convert_test.go

---

📝 Walkthrough

Walkthrough

Tool-result conversion for the Responses API now preserves image_url content as ordered input_image items, defaults detail to "auto", and falls back to empty output when only unsupported content remains. Tests cover conversion and wire serialization.

Changes

Tool output conversion

Layer / File(s)	Summary
Typed content conversion and serialization validation llm/transformer/openai/responses/outbound_convert.go, llm/transformer/openai/responses/outbound_convert_test.go	Tool results preserve ordered text and image parts, map images to input_image with default detail: "auto", retain images alongside unsupported parts, and use empty output for unsupported-only or empty results. Tests verify in-memory conversion and wire-level output arrays.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• looplj/axonhub#2106: Updates the same tool-message conversion path with tool_search_output handling.

Suggested reviewers: looplj, rinstel

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: preserving images in Responses tool results instead of dropping them.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

Tool-result images are now preserved in Responses API output arrays.

• Converts image_url content into input_image items while retaining mixed-content order.
• Supplies detail: "auto" when the source image omits detail.
• Adds struct-level and serialized-wire-shape coverage for function and custom tool outputs.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
llm/transformer/openai/responses/outbound_convert.go	Preserves tool-result images and explicitly defaults image detail, resolving the previously reported custom-tool serialization issue.
llm/transformer/openai/responses/outbound_convert_test.go	Covers image-only and mixed tool results, custom outputs, default detail, and serialized output-array shape.

_{Reviews (2): Last reviewed commit: "fix(responses): default tool image detai..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@llm/transformer/openai/responses/outbound_convert.go`:
- Around line 326-331: Default missing image detail to "auto" in the outbound
image conversion block that builds each input_image Item, while preserving
explicitly provided details. Update the expectations at
llm/transformer/openai/responses/outbound_convert_test.go:142-145, 208-215, and
281-284 to require Detail set to "auto" for omitted details.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 88437236-7ec1-4965-ae5a-d2f5ebc9a260

📥 Commits

Reviewing files that changed from the base of the PR and between 131dc03 and a7399a3.

📒 Files selected for processing (2)

• llm/transformer/openai/responses/outbound_convert.go
• llm/transformer/openai/responses/outbound_convert_test.go