openai-responses: tool-result images in function_call_output dropped by compatible gateways

[MiaM1ku]

Current openai-responses conversion puts tool-result images in function_call_output.output as input_image. That is the official Responses encoding (output is string | (input_text | input_image | input_file)[]). Completions lifts to a user message only because role: "tool" is string-only.

OpenAI accepts the array form. Compatible openai-responses gateways treat function_call_output as a string and drop the images. The model only sees leftover text. Same endpoint accepts input_image on a user message.

Ask: keep the official array form for OpenAI. Fall back to the completions lift (string output + following user input_image) for gateways that cannot take image parts in tool output. Do not always lift.

File: packages/ai/src/api/openai-responses-shared.ts. Issue: #9518

[MiaM1ku]

Official Responses encoding (this is what the current code does, and OpenAI documents it):

pi/packages/ai/src/api/openai-responses-shared.ts

Lines 77 to 104 in 71dca87

	function convertToolResultOutput<TApi extends Api>(
	model: Model<TApi>,
	content: readonly (TextContent | ImageContent)[],
	): string | ToolResultOutputContent {
	const textResult = content
	.filter((c): c is TextContent => c.type === "text")
	.map((c) => c.text)
	.join("\n");
	const images = content.filter((c): c is ImageContent => c.type === "image");
	const hasText = textResult.length > 0;
	if (images.length === 0 || !model.input.includes("image")) {
	return sanitizeSurrogates(hasText ? textResult : images.length > 0 ? "(see attached image)" : "(no tool output)");
	}
	const output: ToolResultOutputContent = [];
	if (hasText) {
	output.push({ type: "input_text", text: sanitizeSurrogates(textResult) });
	}
	for (const image of images) {
	output.push({
	type: "input_image",
	detail: "auto",
	image_url: `data:${image.mimeType};base64,${image.data}`,
	});
	}
	return output;
	}

pi/packages/ai/src/api/openai-responses-shared.ts

Lines 296 to 311 in 71dca87

	} else if (msg.role === "toolResult") {
	const [callId] = msg.toolCallId.split("|");
	const output = convertToolResultOutput(model, msg.content);
	if (options?.grammarToolInputProperties?.has(msg.toolName)) {
	messages.push({
	type: "custom_tool_call_output",
	call_id: callId,
	output,
	});
	} else {
	messages.push({
	type: "function_call_output",
	call_id: callId,
	output,
	});

function_call_output.output is string | (input_text | input_image | input_file)[]. For functions that return images, the array form is the protocol, not a bug: https://developers.openai.com/api/docs/guides/function-calling

Completions has to lift because Chat Completions role: "tool" is string-only:

pi/packages/ai/src/api/openai-completions.ts

Lines 1376 to 1444 in 71dca87

	} else if (msg.role === "toolResult") {
	const imageBlocks: Array<{ type: "image_url"; image_url: { url: string } }> = [];
	const deferredToolNames = new Set<string>();
	let j = i;
	for (; j < transformedMessages.length && transformedMessages[j].role === "toolResult"; j++) {
	const toolMsg = transformedMessages[j] as ToolResultMessage;
	// Extract text and image content
	const textResult = toolMsg.content
	.filter(isTextContentBlock)
	.map((block) => block.text)
	.join("\n");
	const hasImages = toolMsg.content.some((c) => c.type === "image");
	// Always send tool result with text (or placeholder if only images)
	const hasText = textResult.length > 0;
	const toolResultText = hasText ? textResult : hasImages ? "(see attached image)" : "(no tool output)";
	// Some providers require the 'name' field in tool results
	const toolResultMsg: ChatCompletionToolMessageParam = {
	role: "tool",
	content: sanitizeSurrogates(toolResultText),
	tool_call_id: toolMsg.toolCallId,
	};
	if (compat.requiresToolResultName && toolMsg.toolName) {
	(toolResultMsg as any).name = toolMsg.toolName;
	}
	params.push(toolResultMsg);
	if (compat.deferredToolsMode === "kimi") {
	for (const name of toolMsg.addedToolNames ?? []) {
	deferredToolNames.add(name);
	}
	}
	if (hasImages && model.input.includes("image")) {
	for (const block of toolMsg.content) {
	if (isImageContentBlock(block)) {
	imageBlocks.push({
	type: "image_url",
	image_url: {
	url: `data:${block.mimeType};base64,${block.data}`,
	},
	});
	}
	}
	}
	}
	i = j - 1;
	if (imageBlocks.length > 0) {
	if (compat.requiresAssistantAfterToolResult) {
	params.push({
	role: "assistant",
	content: "I have processed the tool results.",
	});
	}
	params.push({
	role: "user",
	content: [
	{
	type: "text",
	text: "Attached image(s) from tool result:",
	},
	...imageBlocks,
	],
	});

Ask: keep the array form for OpenAI. For compatible gateways that stringify function_call_output and drop input_image, fall back to that lift:

1. function_call_output.output stays a string ((see attached image) if there is no text).
2. After consecutive tool results, append a user message with input_text + input_image.

That is still valid Responses input (call pairing stays intact; user input_image is a normal item). It is a compatibility fallback, not a replacement for the official encoding. Always lifting would make the image look like user input instead of tool output.