Tool not executed when using chat completions API #809
Description
Context
The application I am working on for my client is locked behind using the chat completions API because of its landscape, the responses API not being available yet within this landscape.
- Have you read the docs? Yes
- Have you searched for related issues? Yes
Describe the bug
Under those circumstances :
- API is set to chat completions
setOpenAIAPI('chat_completions') - LLM decides to output a text before making a tool call, whithin the same completion
Then this happens : the tool call is not executed, even though the output has a trace of the tool call being made.
Debug information
- Agents SDK version: v0.3.7 (and previous versions)
- Runtime environment: Node 22.20.0 (also happens with Bun 1.3.5 but I believe the runtime environment is not in cause)
Repro steps
Here is a minimal script to reproduce the issue (100% reproducible, no race condition or variability) :
import 'dotenv/config'
import { Agent, run, tool, setOpenAIAPI } from "@openai/agents"
import * as z from "zod"
if (process.env.USE_CHAT_COMPLETIONS_API === '1') {
console.log('Using Chat Completions API')
setOpenAIAPI('chat_completions')
}
const randomNumber = tool({
name: 'random_number',
description: `Generate a random number`,
parameters: z.object({}),
execute: async (args) => {
console.log('Generating a random number...', args)
const num = Math.floor(Math.random() * 100)
console.log('Generated number:', num)
return `The generated random number is :${num}`
}
})
let instructions = ''
if(process.env.USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL === '1') {
console.log('Using ALWAYS OUTPUT BEFORE TOOL CALL instructions')
instructions = 'You MUST ALWAYS say something before you call the tool. You MUST always comment on the result stating a fun fact on the generated number after calling the tool.'
} else {
console.log('Using NEVER OUTPUT BEFORE TOOL CALL instructions')
instructions = 'You MUST NEVER say something before you call the tool. You MUST always comment on the result stating a fun fact on the generated number after calling the tool.'
}
const agent = new Agent({
name: 'Test Agent',
instructions,
tools: [randomNumber]
})
const response = await run(agent, [{ role: 'user', content: 'Please generate a random number for me.' }])
console.log('\nAgent Response Output:', JSON.stringify(response.output, null, 2))Expected behavior
The tool call should always be made, but when the chat completions api is used and the LLM makes some text output before the tool call, the tool is not executed.
In the repro script there are 2 env vars :
USE_CHAT_COMPLETIONS_API -> sets the SDK to use the chat completions API if value is '1'
USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL -> If value is '1', then it will instruct to always make a statement before the tool call. Otherwise instructs to never do it.
Details / Combination outcomes
The last one is the unexpected one
USE_CHAT_COMPLETIONS_API = 0
USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL=0
Behaviour is the expected one: No output before the tool call, tool call is executed :
Using NEVER OUTPUT BEFORE TOOL CALL instructions
Generating a random number... {}
Generated number: 67
Agent Response Output: [
{
"type": "function_call",
"id": "fc_08313c99a30311ef00695cf71ec2b88196bc8ac3a19bcae47b",
"callId": "call_uNH1s9dqaWVEhL5LT8UcQ1AJ",
"name": "random_number",
"status": "completed",
"arguments": "{}",
"providerData": {
"id": "fc_08313c99a30311ef00695cf71ec2b88196bc8ac3a19bcae47b",
"type": "function_call"
}
},
{
"type": "function_call_result",
"name": "random_number",
"callId": "call_uNH1s9dqaWVEhL5LT8UcQ1AJ",
"status": "completed",
"output": {
"type": "text",
"text": "The generated random number is :67"
}
},
{
"id": "msg_08313c99a30311ef00695cf71f73cc819686b59a07ca3e28ca",
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Your random number is 67! Fun fact: 67 is a prime number, which means it cannot be divided by any other number except 1 and itself—pretty exclusive club!",
"annotations": [],
"logprobs": []
}
],
"status": "completed",
"providerData": {}
}
]
USE_CHAT_COMPLETIONS_API = 1
USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL=0
Behaviour is the expected one: No output before the tool call, tool call is executed :
Using Chat Completions API
Using NEVER OUTPUT BEFORE TOOL CALL instructions
Generating a random number... {}
Generated number: 20
Agent Response Output: [
{
"id": "chatcmpl-CuztTz4LjOLQGnJ9sUuecq4WId3a5",
"type": "function_call",
"arguments": "{}",
"name": "random_number",
"callId": "call_iHV64kzrmW6QLSOBSUDQA3Yh",
"status": "completed",
"providerData": {
"type": "function",
"function": {
"name": "random_number",
"arguments": "{}"
}
}
},
{
"type": "function_call_result",
"name": "random_number",
"callId": "call_iHV64kzrmW6QLSOBSUDQA3Yh",
"status": "completed",
"output": {
"type": "text",
"text": "The generated random number is :20"
}
},
{
"id": "chatcmpl-CuztUrhWczcSKktaCtZze11JS6DFk",
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "The generated random number is 20. Fun fact: 20 is the basis for the vigesimal numeral system, and humans typically have 20 fingers and toes combined!",
"providerData": {
"role": "assistant",
"refusal": null,
"annotations": []
}
}
],
"status": "completed"
}
]
USE_CHAT_COMPLETIONS_API = 0
USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL=1
Behaviour is the expected one: There is an output before the tool call, tool call is executed :
Using ALWAYS OUTPUT BEFORE TOOL CALL instructions
Generating a random number... {}
Generated number: 16
Agent Response Output: [
{
"id": "msg_0d1c45130537988200695cf7af0cbc819e84dc08b38972763e",
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Sure! I'll generate a random number for you right now.",
"annotations": [],
"logprobs": []
}
],
"status": "completed",
"providerData": {}
},
{
"type": "function_call",
"id": "fc_0d1c45130537988200695cf7afccd8819e88127e47bf318894",
"callId": "call_9lMm0TvSYEh6YhLb16Dhts6w",
"name": "random_number",
"status": "completed",
"arguments": "{}",
"providerData": {
"id": "fc_0d1c45130537988200695cf7afccd8819e88127e47bf318894",
"type": "function_call"
}
},
{
"type": "function_call_result",
"name": "random_number",
"callId": "call_9lMm0TvSYEh6YhLb16Dhts6w",
"status": "completed",
"output": {
"type": "text",
"text": "The generated random number is :16"
}
},
{
"id": "msg_0d1c45130537988200695cf7b057d0819e9a7d4cb450ec4bc3",
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Your random number is: 16!\n\nFun fact: 16 is a perfect square because it is 4 × 4, and it's also the number of chess pieces each player starts with in a game of chess.",
"annotations": [],
"logprobs": []
}
],
"status": "completed",
"providerData": {}
}
]
USE_CHAT_COMPLETIONS_API = 1
USE_ALWAYS_OUTPUT_BEFORE_TOOL_CALL=1
Behaviour is NOT the one expected: There is an output before the tool call, but the tool call is not executed:
Using Chat Completions API
Using ALWAYS OUTPUT BEFORE TOOL CALL instructions
Agent Response Output: [
{
"id": "chatcmpl-CuzvkCW0bQfoVrViElpMorAugSmr3",
"type": "message",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "Sure! I'll generate a random number for you.\n\n",
"providerData": {
"role": "assistant",
"tool_calls": [
{
"id": "call_jMShJ1qUi6zZbln7mpgfU9ST",
"type": "function",
"function": {
"name": "random_number",
"arguments": "{}"
}
}
],
"refusal": null,
"annotations": []
}
}
],
"status": "completed"
}
]
We can see a trace of the tool call, but it's not registered as such, and thus is never executed.