|
| 1 | +<!--Copyright 2025 The HuggingFace Team. All rights reserved. |
| 2 | +
|
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with |
| 4 | +the License. You may obtain a copy of the License at |
| 5 | +
|
| 6 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 7 | +
|
| 8 | +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on |
| 9 | +an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the |
| 10 | +specific language governing permissions and limitations under the License. |
| 11 | +
|
| 12 | +⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be |
| 13 | +rendered properly in your Markdown viewer. |
| 14 | +
|
| 15 | +--> |
| 16 | + |
| 17 | +# Response Parsing |
| 18 | + |
| 19 | +It is increasingly common for chat models to generate structured outputs, rather than just a single reply string. |
| 20 | +The most common uses for structured outputs are [tool calling](./chat_extras) and [reasoning models](https://huggingface.co/reasoning-course). |
| 21 | +Tool calling models can output tool calls, containing the name of the tool to call and any arguments to be passed to it, |
| 22 | +while reasoning models often output reasoning steps as a "chain of thought". Some recent models even use both of these, |
| 23 | +and may output reasoning and/or one or more tool calls before their final answer. |
| 24 | + |
| 25 | +Models with structured outputs pose a challenge for chat templating, because the output needs to be parsed before it |
| 26 | +can be appended to the chat. For a concrete example, let's say we ask [GPT-OSS](https://huggingface.co/openai/gpt-oss-120b) |
| 27 | +what the weather is like, and it thinks and decides to call a tool. Here's what the raw model output might look like: |
| 28 | + |
| 29 | +```txt |
| 30 | +<|start|>analysis<|message|>The user asks: "What is the weather like in SF?" We need to get the location of the user? The user explicitly asks about SF (San Francisco). |
| 31 | +So we need to get the current weather in San Francisco, CA. We need to call get_current_weather function. But we need to call function to get weather data. |
| 32 | +So we should call get_current_weather with location "San Francisco, CA". Let's do that. |
| 33 | +We will call function get_current_weather.<|end|><|start|>commentary to=functions.get_current_weather<|channel|>commentary <|constrain|>json<|message|>{"location":"San Francisco, CA"}<|call|> |
| 34 | +} |
| 35 | +``` |
| 36 | + |
| 37 | +But if you want to append this to a chat, you'll need to format it as a chat message dict, like this: |
| 38 | + |
| 39 | +```json |
| 40 | +{ |
| 41 | + "role": "assistant", |
| 42 | + "thinking": "The user asks: \"What is the weather like in SF?\" We need to get the location of the user? The user explicitly asks about SF (San Francisco). So we need to get the current weather in San Francisco, CA. We need to call get_current_weather function. But we need to call function to get weather data. So we should call get_current_weather with location \"San Francisco, CA\". Let's do that.", |
| 43 | + "tool_calls": [ |
| 44 | + { |
| 45 | + "name": "get_current_weather", |
| 46 | + "arguments": { |
| 47 | + "location": "San Francisco, CA" |
| 48 | + } |
| 49 | + } |
| 50 | + ] |
| 51 | +} |
| 52 | +``` |
| 53 | + |
| 54 | +Chat **templates** give us a way to turn messages into formatted input for a model, but we need something else to |
| 55 | +parse model output back into a standard message dict. This is what chat **parsing** is for. |
| 56 | + |
| 57 | +## The [parse_response](~PreTrainedTokenizerBase.parse_response) method |
| 58 | + |
| 59 | +Parsing a chat response on a model that supports it is straightforward. Simply take the raw, decoded output from |
| 60 | +[generate](`~generation.GenerationMixin.generate`), and pass it to the tokenizer's [parse_response](~PreTrainedTokenizerBase.parse_response) method: |
| 61 | + |
| 62 | +```python |
| 63 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 64 | + |
| 65 | +checkpoint = "HuggingFaceTB/SmolLM3-3B" |
| 66 | + |
| 67 | +tokenizer = AutoTokenizer.from_pretrained(checkpoint) |
| 68 | +model = AutoModelForCausalLM.from_pretrained(checkpoint, dtype="auto", device_map="auto") |
| 69 | + |
| 70 | +messages = [ |
| 71 | + { |
| 72 | + "role": "user", |
| 73 | + "content": "Hey! Can you summarize the end of the Cold War as briefly as possible? Like, comically briefly. It should really leave out almost most of the relevant information." |
| 74 | + } |
| 75 | +] |
| 76 | + |
| 77 | +input_ids = tokenizer.apply_chat_template( |
| 78 | + messages, |
| 79 | + add_generation_prompt=True, |
| 80 | + tokenize=True, |
| 81 | + return_tensors="pt" |
| 82 | +).to(model.device) |
| 83 | + |
| 84 | +outputs = model.generate(input_ids, max_new_tokens=1024)[0, input_ids.shape[1]:] |
| 85 | +out_text = tokenizer.decode(outputs) |
| 86 | +parsed = tokenizer.parse_response(out_text) |
| 87 | +print(parsed.keys()) |
| 88 | +``` |
| 89 | + |
| 90 | +And you should get: |
| 91 | + |
| 92 | +```text |
| 93 | +dict_keys(['thinking', 'content']) |
| 94 | +``` |
| 95 | + |
| 96 | +And that's all you need to start using response parsing! `parse_response` should return a complete message dict that is ready to be appended to the chat history. |
| 97 | +When the tokenizer does not support response parsing, `parse_response` will throw an error. We hope to add support |
| 98 | +to more tokenizers over time. |
| 99 | + |
| 100 | +## Developers: Understanding a simple response schema |
| 101 | + |
| 102 | +Under the hood, `parse_response` uses a **JSON schema** to parse the model output. A JSON schema represents |
| 103 | +the structure of the output message dict. The schema is augmented with additional fields that indicate how the |
| 104 | +output message string should be parsed into the expected format. Let's take a look at the schema for a SmolLM response, |
| 105 | +excluding tool calls for now: |
| 106 | + |
| 107 | +```python |
| 108 | +{ |
| 109 | + "x-regex": "(?:<think>\n?(?P<thinking>.+?)\n?</think>)?\s*(?P<content>.+?)?\s*(?:<\|im_end\|>|$)", |
| 110 | + "type": "object", |
| 111 | + "properties": { |
| 112 | + "role": {"const": "assistant"}, |
| 113 | + "content": {"type": "string"}, |
| 114 | + "thinking": {"type": "string"} |
| 115 | + } |
| 116 | +} |
| 117 | +``` |
| 118 | + |
| 119 | +We can see that the schema describes a JSON "object" (a `dict`, in other words) with three keys: `role`, `content`, and `thinking`. |
| 120 | +Because all assistant responses have the role "assistant", the `role` key is a `const`(ant). The other two keys are strings, extracted |
| 121 | +from the named groups in the regex in the `x-regex` field. |
| 122 | + |
| 123 | +Like chat templates, response schemas are set as a property of the tokenizer. To enable response parsing, all you need |
| 124 | +to do is set `tokenizer.response_schema` to a valid schema dict, and `tokenizer.parse_response()` will work! Again, like |
| 125 | +chat templates, this schema will be saved with the processor, so once you set it, you can use `save_pretrained()` or `push_to_hub()` to |
| 126 | +save and share the schema. |
| 127 | + |
| 128 | +## Developers: Complex schemas |
| 129 | + |
| 130 | +Now, let's look at a more complex schema, which includes tool calls, to gain more of an understanding of the parser |
| 131 | +internals. For this, we'll use the `GPT-OSS` schema. GPT-OSS emits both tool calls and thinking blocks, and it uses |
| 132 | +an unusual format where model responses are tagged with one of three "channels": `commentary` for things like |
| 133 | +tool calls, `analysis` for chain of thought blocks, and `final` for messages intended to be sent to the user. |
| 134 | +A full message where the model calls a tool named `get_current_weather` might look like this, with some extra linebreaks added for clarity: |
| 135 | + |
| 136 | +```text |
| 137 | +<|channel|>analysis<|message|> |
| 138 | +The user asks: "What is the weather like in SF?" So we need to get the current weather in San Francisco, CA. |
| 139 | +We need to call get_current_weather function. So we should call get_current_weather with location "San Francisco, CA". |
| 140 | +<|end|> |
| 141 | +<|start|>assistant<|channel|>commentary |
| 142 | +to=functions.get_current_weather <|constrain|>json<|message|> |
| 143 | +{ |
| 144 | + "location": "San Francisco, CA" |
| 145 | +} |
| 146 | +<|call|> |
| 147 | +``` |
| 148 | + |
| 149 | +Parsing proceeds recursively; the output of a regex (or other parser) at one level becomes the input to the nodes below it. |
| 150 | +In other words, don't feel like you have to parse the entire output in one enormous regex! Instead, start with the schema, |
| 151 | +and then add regexes to extract the relevant chunks as you go. Here's a schema that will parse it, with some |
| 152 | +explanatory comments: |
| 153 | + |
| 154 | +```python |
| 155 | +{ |
| 156 | + "type": "object", |
| 157 | + "properties": { |
| 158 | + "role": {"const": "assistant"}, |
| 159 | + # "content" and "thinking" are both similar to the previous example, and just extract a single string |
| 160 | + # However, rather than using a single regex with named groups to extract both, we use a regex in each subkey. |
| 161 | + # When an object node has no parser/regex, the entire input string is passed to all of its children, so |
| 162 | + # parsing can either be done with named groups at the object level, or with separate regexes at the property level. |
| 163 | + "content": {"type": "string", "x-regex": r"<\|channel\|>final<\|message\|>(.*?)(?:<\|end\|>|$)"}, |
| 164 | + "thinking": {"type": "string", "x-regex": r"<\|channel\|>analysis<\|message\|>(.*?)<\|end\|>"}, |
| 165 | + "tool_calls": { |
| 166 | + # "x-regex-iterator" uses re.findall to find multiple possible manages, and returns them as an |
| 167 | + # array/list. You don't need to worry about array handling, though - each item in the array will be |
| 168 | + # parsed by the `items` schema, so just write the schema for a single item. |
| 169 | + "x-regex-iterator": r"<\|channel\|>commentary (to=functions\..*?<\|message\|>.*?)(?:<\|call\|>|$)", |
| 170 | + "type": "array", |
| 171 | + "items": { |
| 172 | + "type": "object", |
| 173 | + "properties": { |
| 174 | + # A const property is a fixed value, and the input has no effect on it. |
| 175 | + "type": {"const": "function"}, |
| 176 | + # Here, we wrap the entire tool call dict in a `{"function": ...}` block. The input string is passed through to it unchanged. |
| 177 | + "function": { |
| 178 | + "type": "object", |
| 179 | + "properties": { |
| 180 | + "name": {"type": "string", "x-regex": r"^to=functions\.(\w+)"}, |
| 181 | + "arguments": { |
| 182 | + "type": "object", |
| 183 | + "x-regex": "<\|message\|>(.*)", |
| 184 | + # The "x-parser" field indicates that the extracted string should be parsed as JSON. |
| 185 | + # The output is then passed to the schema nodes below and recursive parsing continues. |
| 186 | + "x-parser": "json", |
| 187 | + "additionalProperties": {"type": "any"}, |
| 188 | + }, |
| 189 | + }, |
| 190 | + }, |
| 191 | + }, |
| 192 | + }, |
| 193 | + }, |
| 194 | + }, |
| 195 | +} |
| 196 | +``` |
| 197 | + |
| 198 | +## Developers: Understanding the parser logic |
| 199 | + |
| 200 | +The parser follows a few simple rules: |
| 201 | + |
| 202 | +1. Each level of the schema receives input from the level above, applies any regex or parser it has, and then passes the output to its children. |
| 203 | +2. The root level receives the entire decoded model output string as input. |
| 204 | +3. If a node has structured content after parsing (for example, if the regex has named groups and returns a dict, or if the parser returns a dict or list), |
| 205 | + then that structured content is mapped to the node's children, and each child node receives its corresponding value as input. |
| 206 | +4. If an `object` (dict) node has unstructured (string) output, then the entire string is passed to all of its children. This allows child nodes |
| 207 | + to handle parsing individually rather than requiring a single parent regex to extract all keys at once. |
| 208 | +5. If an `array` (list) node has unstructured (string) output, then this throws an error. |
| 209 | + |
| 210 | +There is a small set of allowable `x-` keys that indicate how parsing should be done at each node: |
| 211 | +- `x-regex`: A regex string to apply to the input. If the regex has named groups, the output is a dict of group names to values. Named groups should only be used in `object` nodes. |
| 212 | + Otherwise, the regex must have exactly one unnamed capturing group, and the output is the value of that group as a string. |
| 213 | +- `x-regex-iterator`: A regex string to apply to the input using `re.findall()`. The output is a list of all matches. |
| 214 | + This should only be used in `array` nodes, and the regex must have exactly one unnamed capturing group. The output is distributed to |
| 215 | + the node's `items` schema. |
| 216 | +- `x-parser`: Calls a built-in parser to apply to the input. Currently, the only supported parser is `json`, which parses the input string as JSON. |
| 217 | + The output is passed to the child nodes for further parsing. Note that the `json` parser can return deeply nested output - in this case, the output |
| 218 | + will be progressively unwrapped as it is passed through child nodes. The child nodes do not need additional `x-parser` or `x-regex` fields in this case, |
| 219 | + but their structure must match the structure of the parsed JSON. |
| 220 | +- `x-parser-args`: Only allowed in conjunction with `x-parser`. This is a dict of additional arguments that control parsing. Right now, the only supported |
| 221 | + argument is `transform`, which specifies a `jmespath` transformation to apply to the output. This is useful when the JSON parser returns a structure |
| 222 | + that needs to be modified to match the schema. |
| 223 | +- `x-regex-key-value`: This is rarely necessary, but it can be useful when parsing key-value pairs in non-JSON format where the names of the keys are not known |
| 224 | + in advance, such as when a model emits XML tool calls with arbitrary argument names. The regex must have exactly two named capturing groups, |
| 225 | + `key` and `value`, and the output is a dict mapping keys to values. This should only be used in `object` nodes. |
| 226 | + |
| 227 | +In general, multiple regexes/parsers cannot be combined at the same level. The exception is that `x-regex`, returning a single string, can be combined with the other parsers. In this case, |
| 228 | +`x-regex` is applied first, and then the output is passed to the other parser, either `x-regex-iterator`, `x-parser`, or `x-regex-key-value`. |
| 229 | + |
| 230 | +Putting these ideas together, you can see that the input flows through the schema, being parsed at each level and then distributed to child nodes. Each level |
| 231 | +only needs to extract the input content that is relevant for that part of the schema, and can then let its child nodes handle the rest. Internally, this is handled |
| 232 | +with a parser function that receives input, applies any regexes/parsers at the current level, then maps the result to its child nodes before recursively calling itself on each of them. |
| 233 | +Recursion terminates when it reaches leaf nodes, usually primitive types like `string` or `number`, which simply return the input they receive. |
0 commit comments