[Feature][Tools]: Complete Redesign of Tool Calling

[robertgshaw2-redhat]

🚀 The feature, motivation and pitch

• We currently have a patchwork of support for tools in vLLM
• We currently have regexes in tools which can cause noisy-neighbor issues in vLLM
• We would welcome a contributor to redesign the system, improve our CI coverage, and work together with the ecosystem to ensure vLLM's support for tool calling is elite

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[chaunceyjiang]

I’m proposing a generic ToolParser design based on Structured Outputs.

Current Situation:
In vLLM, there are currently many different ToolParser implementations. All of these ToolParsers are doing essentially the same thing—extracting Tool_Call from LLM outputs.
Each LLM has its own tool_call_token format, such as <tool_call></tool_call>, <|tool_call_begin|><|tool_call_end|>, etc.
These parsers typically use regex to extract tool_call, which results in a large number of edge cases to handle. This approach is messy and error-prone.

Proposal:
Use a generic ToolParser based on vLLM’s Structured Outputs.

Structured Outputs in vLLM allow the LLM to produce outputs in a specified format.
As shown in OpenAI function-calling, each function call must follow a JSON Schema
This means we can have the LLM directly output content that conforms to a JSON Schema—rather than having the LLM generate arbitrary text and then using regex to extract the tool_call.

The benefits are significant:

• No need to maintain an ever-growing set of ToolParser implementations.
• No need to maintain very complex code related to tool_parser in serving_chat.py.
• Only one generic ToolParser is needed to extract tool_call.
• No regex extraction required.
• The LLM output can simply be converted into the OpenAI output format.

In OpenAI function-calling, we can use tool_choice to allow the LLM to call different numbers of tools:

• Auto (default): Call zero, one, or multiple functions. → tool_choice: "auto"
• Required: Call one or more functions. → tool_choice: "required"
• Forced Function: Call exactly one specific function. → tool_choice: {"type": "function", "name": "get_weather"}

To implement the above tool_choice behavior, we can add extra properties to the JSON Schema:

1. When tool_choice == Auto: add anyOf + {} to the JSON Schema.
2. When tool_choice == Required: add anyOf + minItems = 1.
3. When tool_choice == Forced Function: add anyOf + minItems = 1 + maxItems = 1.

---

Suppose the user uses the following tools:

tools = [
    {
        "type": "function",
        "function": {
            "name": "calculator",
            "description":
            "Calculator, called when the user needs to perform addition, when call this function ",
            "parameters": {
                "type": "object",
                "properties": {
                    "numbers": {
                        "description": "number list",
                        "items": {
                            "properties": {
                                "number": {
                                    "description": "each number",
                                    "type": "string"
                                }
                            },
                            "type": "object"
                        },
                        "type": "array"
                    }
                },
                "required": ["numbers"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to find the weather for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "unit"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_forecast",
            "description": "Get the weather forecast for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description":
                        "The city to get the forecast for, e.g. 'Vienna'",
                        "default": "Vienna",
                    },
                    "country": {
                        "type":
                        "string",
                        "description":
                        "The country that the city is in, e.g. 'Austria'",
                    },
                    "days": {
                        "type":
                        "integer",
                        "description":
                        "Number of days to get the forecast for (1-7)",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["country", "days", "unit"],
            },
        },
    },
]

We define a helper function to convert the tools provided by the user into a JSON Schema. The code is as follows:

def generate_function_call_array_schema_anyOf(function_defs,
                                              required: bool = False,
                                              tool_name: str | None = None):
    """
    Generate JSON Schema for an array of function calls (anyOf version),
    optionally filtered by a specific tool name.

    Parameters:
        function_defs: List of function definitions. Each item should contain:
            - "function": a dict with keys "name" (str) and "parameters" (dict)
              Optionally, "parameters" may contain a "$defs" dict to extract sub-schemas.
        required: bool
            If True and tool_name=None:
                - The array must have at least one element (minItems=1)
                - Arbitrary values in the array are NOT allowed
        tool_name: str | None
            If specified:
                - Only include this tool's schema in the array items
                - Array must have at least one element (required=True)
                - Arbitrary values are NOT allowed
    
    Returns:
        A JSON Schema dict describing the array structure.
    """
    item_schemas = []
    json_schema_defs = dict[str, dict[str,
                                      Any]]()  # Collect all top-level $defs

    # If a specific tool is requested, automatically enforce required=True
    if tool_name is not None:
        required = True

    # Build item schemas for each function
    for f in function_defs:
        fname = f["function"]["name"]

        # Skip tools that do not match the requested tool_name
        if tool_name and fname != tool_name:
            continue

        # Deep copy parameters to avoid mutating input
        fparams = copy.deepcopy(f["function"].get("parameters", {}))

        # Extract any $defs from this function's parameters
        defs = fparams.pop("$defs", {})
        for def_name, def_schema in defs.items():
            # Ensure no duplicate $defs with different schemas
            if def_name in json_schema_defs and json_schema_defs[
                    def_name] != def_schema:
                raise ValueError(
                    f"Tool definition '{def_name}' has multiple schemas, "
                    "which is not supported.")
            else:
                json_schema_defs[def_name] = def_schema

        # Build the function object schema
        item_schemas.append({
            "type": "object",
            "additionalProperties": False,
            "description": f["function"].get("description", ""),
            "properties": {
                "name": {
                    "const": fname
                },  # Ensure "name" matches this function
                "arguments": fparams  # Function parameters schema
            },
            "required": ["name", "arguments"]
        })

    # If required=False and tool_name is None, allow arbitrary values in the array
    if not required and not tool_name:
        item_schemas.append({})  # {} means any type of value is allowed

    # Build top-level array schema
    schema = {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "title": "FunctionCallArraySchema",
        "type": "array",
        "items": {
            "anyOf": item_schemas
        }  # Each item matches any of the function schemas
    }

    # If required=True, array must have at least one element
    if required:
        schema["minItems"] = 1

    # If tool_name is specified, restrict array to exactly one element
    if tool_name:
        schema["maxItems"] = 1

    # Include top-level $defs if any were collected
    if json_schema_defs:
        schema["$defs"] = json_schema_defs

    return schema

After being processed by this helper function, the tools submitted by the user will be converted into a JSON Schema. The LLM will then produce output that conforms to this schema.

JSON Schema Example

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "FunctionCallArraySchema",
  "type": "array",
  "items": {
    "anyOf": [
      {
        "type": "object",
        "additionalProperties": false,
        "description": "Calculator, called when the user needs to perform addition, when call this function ",
        "properties": {
          "name": {
            "const": "calculator"
          },
          "arguments": {
            "type": "object",
            "properties": {
              "numbers": {
                "description": "number list",
                "items": {
                  "properties": {
                    "number": {
                      "description": "each number",
                      "type": "string"
                    }
                  },
                  "type": "object"
                },
                "type": "array"
              }
            },
            "required": [
              "numbers"
            ]
          }
        },
        "required": [
          "name",
          "arguments"
        ]
      },
      {
        "type": "object",
        "additionalProperties": false,
        "description": "Get the current weather in a given location",
        "properties": {
          "name": {
            "const": "get_current_weather"
          },
          "arguments": {
            "type": "object",
            "properties": {
              "city": {
                "type": "string",
                "description": "The city to find the weather for, e.g. 'Vienna'",
                "default": "Vienna"
              },
              "country": {
                "type": "string",
                "description": "The country that the city is in, e.g. 'Austria'"
              },
              "unit": {
                "type": "string",
                "description": "The unit to fetch the temperature in",
                "enum": [
                  "celsius",
                  "fahrenheit"
                ]
              }
            },
            "required": [
              "country",
              "unit"
            ]
          }
        },
        "required": [
          "name",
          "arguments"
        ]
      },
      {
        "type": "object",
        "additionalProperties": false,
        "description": "Get the weather forecast for a given location",
        "properties": {
          "name": {
            "const": "get_forecast"
          },
          "arguments": {
            "type": "object",
            "properties": {
              "city": {
                "type": "string",
                "description": "The city to get the forecast for, e.g. 'Vienna'",
                "default": "Vienna"
              },
              "country": {
                "type": "string",
                "description": "The country that the city is in, e.g. 'Austria'"
              },
              "days": {
                "type": "integer",
                "description": "Number of days to get the forecast for (1-7)"
              },
              "unit": {
                "type": "string",
                "description": "The unit to fetch the temperature in",
                "enum": [
                  "celsius",
                  "fahrenheit"
                ]
              }
            },
            "required": [
              "country",
              "days",
              "unit"
            ]
          }
        },
        "required": [
          "name",
          "arguments"
        ]
      },
      {}
    ]
  }
}

Prompt

Calculate 1 + 2 + 3 + 4, what is the weather in San Francisco?

LLM Outputs

[
  {
    "name": "calculator",
    "arguments": {
      "numbers": [
        {"number": "1"},
        {"number": "2"},
        {"number": "3"},
        {"number": "4"}
      ]
    }
  },
  {
    "name": "get_current_weather",
    "arguments": {
      "city": "San Francisco",
      "country": "USA",
      "unit": "celsius"
    }
  }
]

Then, based on the LLM Outputs, a very simple conversion is all that’s needed to transform the LLM Outputs into a ChatCompletionMessageToolCall.

[david6666666]

Maybe Using a generic ToolParser based on vLLM's Structured Outputs is a good solution.

[chaunceyjiang]

based on Structured Outputs

When combining text and tool-call outputs, it is quite difficult to use Structured Output

[robertgshaw2-redhat]

Thanks @chaunceyjiang - I will ask a few others to comment on this RFC to ensure this proposal make sense

[robertgshaw2-redhat]

• cc @yeqcharlotte - can you tag the relevant folks from meta
• cc @bbrowning @wseaton - can you guys LMK what you think about this
• cc @casper-hansen - does this jive with your understanding
• cc @russellb - how does this look re:structured outputs

[bbrowning]

@chaunceyjiang As you correctly noted at the start of your proposal, Each LLM has its own tool_call_token format, such as <tool_call></tool_call>, <|tool_call_begin|><|tool_call_end|>, etc. . The purpose of the tool call parsers is to take this specific raw output from the model - which may resemble XML, JSON, Python, begin/end tokens, OpenAI's new Harmony format, etc - and turn that into a well-formatted JSON tool call response to send back to clients.

What this means is we cannot just use structured output parsing to enforce JSON as proposed here, because the models are not necessarily outputting JSON nor trained to output JSON. Some may, but the purpose of the tool call parsing code generally is to take the format the model is trained to output and massage it into the standard OpenAI JSON format linked. Forcing structured output as JSON would break for any models that don't natively produce JSON tool calls, as well as scenarios where they may emit combined text and tool call outputs in a single response, which is not uncommon.

[simon-mo]

Thank you for starting the investigation.

Can you go through how would this work with different kinds of tool structure such as Hermes, Qwen3, and Kimi? They all have different output structure. For example, the output of parameters is not necessarily JSON format, rather combination of XML and plain text.

[wseaton]

I would propose some short term/medium term improvements:

• Allow for model providers to externally vendor parsers (if required), this helps shift maintenance burdens for parser implementations out of tree of vllm
  • A change to support this would be allowing importing tool call parser implementations from external libraries via some type of stable API, example from one of my PoCs (not thinking about changing protocol at all yet, just an example of an externally implemented parser).
• Track RFCs such as those that exist in transformers around storing EBNF parse grammar or json schema for parsing tool calls a model is trained on in the tokenizer config. This would make implementing regex-less generic parsers as a one size fits all approach much easer, but requires model provider buy-in.
• Work on a stable interface that these external libraries can adopt, and then shift the in-tree parsers into a dedicated community contrib package under the vllm-project.

[chaunceyjiang]

@bbrowning You're right. I've been testing locally over the past few days using models with JSON, XML, and Pythonic formats. I found that when forcing structured output to convert the model's response into JSON, and when the user's prompt has weak relevance to the tool call, the model's output quality is not very good.

[bbrowning]

How many different formats do we have for tool calls? Today we have a tool call parser for each model, and each parser implements its own parsing. What if instead we had a single tool call parser for JSON format tool calls, a single one for XML, and a single one for Pythonic? Reduce the actual parser implementations down to one per language that we're parsing as opposed to one per model. That doesn't fix the underlying issues, but it reduces the permutations of actual parser implementations.

Then, for each model, we just need to identify the tokens that represent the start and end boundaries of a tool call so that we can parse tool calls from regular text. Can we do this in some generic way, where each model implementation just has a list of tokens that represent those boundaries? Are there models where we can't identify tool calls based on start and/or stop token boundaries?