ValidationError when using chat history that contains tool message

[off6atomic]

Description

Run the following code:

from typing import Literal

from openai.types.chat import ChatCompletionMessageParam

from mirascope.openai import OpenAICall, OpenAICallParams


def get_current_weather(
    location: str, unit: Literal["celsius", "fahrenheit"] = "fahrenheit"
):
    """Get the current weather in a given location."""
    if "tokyo" in location.lower():
        return f"It is 10 degrees {unit} in Tokyo, Japan"
    elif "san francisco" in location.lower():
        return f"It is 72 degrees {unit} in San Francisco, CA"
    elif "paris" in location.lower():
        return f"It is 22 degress {unit} in Paris, France"
    else:
        return f"I'm not sure what the weather is like in {location}"


class Forecast(OpenAICall):
    prompt_template = """
    MESSAGES: {history}
    USER: {question}
    """

    question: str
    history: list[ChatCompletionMessageParam] = []
    call_params = OpenAICallParams(model="gpt-4-turbo", tools=[get_current_weather])


# Make the first call to the LLM
forecast = Forecast(question="What's the weather in Tokyo Japan?")
response = forecast.call()
history = []
history += [
    {"role": "user", "content": forecast.question},
    response.message.model_dump(),
]

tool = response.tool
if tool:
    print("Tool arguments:", tool.args)
    # > {'location': 'Tokyo, Japan', 'unit': 'fahrenheit'}
    output = tool.fn(**tool.args)
    print("Tool output:", output)
    # > It is 10 degrees fahrenheit in Tokyo, Japan

    # reinsert the tool call into the chat messages through history
    # NOTE: this should be more convenient, e.g. `tool.message_param`
    history += [
        {
            "role": "tool",
            "content": output,
            "tool_call_id": tool.tool_call.id,
            "name": tool.__class__.__name__,
        },
    ]
else:
    print(response.content)  # if no tool, print the content of the response

# Call the LLM again with the history including the tool call
response = Forecast(question="Is that cold or hot?", history=history).call()
print("After Tools Response:", response.content)

And you will get the following error:

---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[27], [line 64](vscode-notebook-cell:?execution_count=27&line=64)
     [61](vscode-notebook-cell:?execution_count=27&line=61)     print(response.content)  # if no tool, print the content of the response
     [63](vscode-notebook-cell:?execution_count=27&line=63) # Call the LLM again with the history including the tool call
---> [64](vscode-notebook-cell:?execution_count=27&line=64) response = Forecast(question="Is that cold or hot?", history=history[:-1]).call()
     [65](vscode-notebook-cell:?execution_count=27&line=65) print("After Tools Response:", response.content)

File ~/Documents/catalyst-ai/.venv/lib/python3.10/site-packages/pydantic/main.py:176, in BaseModel.__init__(self, **data)
    [174](https://file+.vscode-resource.vscode-cdn.net/Users/prithivi/Documents/catalyst-ai/src/v2/~/Documents/catalyst-ai/.venv/lib/python3.10/site-packages/pydantic/main.py:174) # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks
    [175](https://file+.vscode-resource.vscode-cdn.net/Users/prithivi/Documents/catalyst-ai/src/v2/~/Documents/catalyst-ai/.venv/lib/python3.10/site-packages/pydantic/main.py:175) __tracebackhide__ = True
--> [176](https://file+.vscode-resource.vscode-cdn.net/Users/prithivi/Documents/catalyst-ai/src/v2/~/Documents/catalyst-ai/.venv/lib/python3.10/site-packages/pydantic/main.py:176) self.__pydantic_validator__.validate_python(data, self_instance=self)

ValidationError: 11 validation errors for Forecast
history.1.typed-dict.content
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type
history.1.typed-dict.role
  Input should be 'system' [type=literal_error, input_value='assistant', input_type=str]
    For further information visit https://errors.pydantic.dev/2.7/v/literal_error
history.1.typed-dict.content.str
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type
history.1.typed-dict.content.generator[union[typed-dict,typed-dict]]
  Input should be iterable [type=iterable_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/iterable_type
history.1.typed-dict.role
  Input should be 'user' [type=literal_error, input_value='assistant', input_type=str]
    For further information visit https://errors.pydantic.dev/2.7/v/literal_error
history.1.typed-dict.function_call
  Input should be a valid dictionary [type=dict_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/dict_type
history.1.typed-dict.content
  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.7/v/string_type
history.1.typed-dict.role
  Input should be 'tool' [type=literal_error, input_value='assistant', input_type=str]
    For further information visit https://errors.pydantic.dev/2.7/v/literal_error
history.1.typed-dict.tool_call_id
  Field required [type=missing, input_value={'content': None, 'role':...}, 'type': 'function'}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing
history.1.typed-dict.name
  Field required [type=missing, input_value={'content': None, 'role':...}, 'type': 'function'}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing
history.1.typed-dict.role
  Input should be 'function' [type=literal_error, input_value='assistant', input_type=str]
    For further information visit https://errors.pydantic.dev/2.7/v/literal_error

The code above is just a slight modification of code in this file

mirascope/docs/concepts/tools_(function_calling).md

Lines 110 to 174 in b643951

	from typing import Literal
	from openai.types.chat import ChatCompletionMessageParam
	from mirascope.openai import OpenAICall, OpenAICallParams
	def get_current_weather(
	location: str, unit: Literal["celsius", "fahrenheit"] = "fahrenheit"
	):
	"""Get the current weather in a given location."""
	if "tokyo" in location.lower():
	return f"It is 10 degrees {unit} in Tokyo, Japan"
	elif "san francisco" in location.lower():
	return f"It is 72 degrees {unit} in San Francisco, CA"
	elif "paris" in location.lower():
	return f"It is 22 degress {unit} in Paris, France"
	else:
	return f"I'm not sure what the weather is like in {location}"
	class Forecast(OpenAICall):
	prompt_template = """
	MESSAGES: {history}
	USER: {question}
	"""
	question: str
	history: list[ChatCompletionMessageParam] = []
	call_params = OpenAICallParams(model="gpt-4-turbo", tools=[get_current_weather])
	# Make the first call to the LLM
	forecast = Forecast(question="What's the weather in Tokyo Japan?")
	response = forecast.call()
	forecast.history += [
	{"role": "user", "content": forecast.question},
	response.message.model_dump(),
	]
	tool = response.tool
	if tool:
	print("Tool arguments:", tool.args)
	# > {'location': 'Tokyo, Japan', 'unit': 'fahrenheit'}
	output = tool.fn(**tool.args)
	print("Tool output:", output)
	# > It is 10 degrees fahrenheit in Tokyo, Japan
	# reinsert the tool call into the chat messages through history
	# NOTE: this should be more convenient, e.g. `tool.message_param`
	forecast.history += [
	{
	"role": "tool",
	"content": output,
	"tool_call_id": tool.tool_call.id,
	"name": tool.__class__.__name__,
	},
	]
	else:
	print(response.content) # if no tool, print the content of the response
	# Call the LLM again with the history including the tool call
	forecast.question = "Is that cold or hot?"
	response = forecast.call()
	print("After Tools Response:", response.content)

The change I did was about creating a new instance of Forecast instead of reusing the old one. The reason I created a new instance is because I like to not mutate the state of an object after it's created.

Here's the file diff: https://diffcheck.io/y-m0-U_Zn_mDOt7MjX

Python, Mirascope & OS Versions, related packages (not required)

mirascope==0.13.4
pydantic==2.7.1
pydantic-core==2.18.2

[willbakst]

This looks like a bug with openai and how they are typing their message parameters. The following code fails (without mirascope) when it really shouldn't:

from openai.types.chat import ChatCompletionAssistantMessageParam
from pydantic import BaseModel


class MyModel(BaseModel):
    history: list[ChatCompletionAssistantMessageParam]


history = [
    {
        "content": None,
        "role": "assistant",
        "function_call": None,
        "tool_calls": [
            {
                "id": "id",
                "function": {
                    "arguments": '{"location":"Tokyo, Japan"}',
                    "name": "GetCurrentWeather",
                },
                "type": "function",
            }
        ],
    },
]
my_model = MyModel(history=history)
print(my_model)

Taking a deeper look, ChatCompletionAssistantMessageParam has function_call required but they return None in their message, which should be reflected in their typing.

I've filed a bug report on their repo: openai/openai-python#1433

A stop gap that would've ideally worked would've been to use response.message.model_dump(exclude={"function_call"}), but it looks like there is an issue with pydantic that causes another issue.

I've filed a bug report on their repo as well: pydantic/pydantic#9467

For now, it looks like the best solution is to just not use ChatCompletionAssistantMessageParam and instead just use dict[str, Any] or Any. This won't catch issues with the message params, but these other issues with openai and pydantic need to be resolved first anyway, so...

[off6atomic]

Haha the bug is hidden deep in someone else's repo. Thank you for the effort digging @willbakst
I'll mutate the history state similarly to how you do it for now.

Please update us here when the upstream bugs are resolved and integrated into Mirascope.

[willbakst]

Update: the issue has been resolved on the OpenAI side (openai/openai-openapi#269), so once that's been released we're just waiting on Pydantic to fix their issue, which they've added to their v2.8 milestone.

[willbakst]

With #308 this is resolved, the new response.message_param property inserts the correct thing.