Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BaseOllamaChatCompletionClient errors with host arg; incorrectly passes Ollama client constructor host argument to the Ollama client's .chat() method #5663

Open
rylativity opened this issue Feb 22, 2025 · 1 comment · May be fixed by #5674

Comments

@rylativity
Copy link

rylativity commented Feb 22, 2025

What happened?

Describe the bug
For the purpose of this issue, it's important to distinguish between the ollama AsyncClient python client and the Autogen OllamaChatCompletionClient (and BaseOllamaChatCompletionClient), which itself uses the ollama AsyncClient to interact with an ollama server (i.e. amongst other things, the autogen OllamaChatCompletionClient wraps an ollama AsyncClient)

When instantiating an autogen OllamaChatCompletionClient, you can pass an arbitrary kwargs to its constructor, some of which are intended to be passed through to the ollama AsyncClient constructor, and the rest of which are intended to be passed to the instantiated ollama AsyncClient's various generation methods (i.e. what Autogen tends to refer to as create_args) at inference time .

Notice that all of the kwargs are passed to both the AsyncClient constructor through the _ollama_client_from_config() function to instantiate an AsyncClient and eventually to the AsyncClient's inference time arguments (by passing the create_args to the BaseOllamaChatCompletionClient's constructor).

While the ollama AsyncClient's constructor accepts arbitrary kwargs, its .chat() method does not. Therefore, if you pass a host kwarg to the autogen OllamaChatCompletionClient constructor, while it successfully uses that host value when instantiating the underlying ollama AsyncClient, that host kwarg is also eventually passed to the AsyncClient's .chat() method which raises a TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[17], line 1
----> 1 await model_client.create([UserMessage(content="hi", source="user")])

File /workspace/.venv/lib/python3.12/site-packages/autogen_ext/models/ollama/_ollama_client.py:451, in BaseOllamaChatCompletionClient.create(self, messages, tools, json_output, extra_create_args, cancellation_token)
    439     future = asyncio.ensure_future(
    440         self._client.chat(  # type: ignore
    441             # model=self._model_name,
   (...)
    447         )
    448     )
    449 else:
    450     future = asyncio.ensure_future(
--> 451         self._client.chat(  # type: ignore
    452             # model=self._model_name,
    453             messages=ollama_messages,
    454             stream=False,
    455             format=response_format_value,
    456             **create_args_no_response_format,
    457         )
    458     )
    459 if cancellation_token is not None:
    460     cancellation_token.link_future(future)

TypeError: AsyncClient.chat() got an unexpected keyword argument 'host'

To Reproduce

  1. Install autogen-ext directly from the main Github branch here
  2. Import and instantiate an OllamaChatCompletionClient, passing in a host kwarg (even passing http://localhost:11434 if you have Ollama server running on your host machine should still trigger the issue):
from autogen_ext.models.ollama import OllamaChatCompletionClient

client = OllamaChatCompletionClient(
        model="qwen2.5",
        host='http://localhost:11434',
       ...
 )
  1. Call the instantiated client's .create() method:
from autogen_core.models._types import UserMessage

await client.create([UserMessage(content="hi", source="user")])
  1. Observe traceback shown above indicating that host being passed to ollama AsyncClient's .chat() raises TypeError as an unexpected kwarg.

Expected behavior
Calls to an instantiated OllamaChatCompletionClient's .create() method should run successfully, even if a host kwarg value is passed to the OllamaChatCompletionClient's constructor.

Suggested resolution
My initial thoughts for resolving the issue involve either a) removing all kwargs that are used to create the ollama AsyncClient after the client has been instantiated, or b) remove all kwargs except for those expected by the AsyncClient's .chat() method.

One challenge with option a will be that there are arbitrary **kwargs that can be passed to the AsyncClient's constructor that are then passed through to httpx, meaning those would also need to be identified and removed. A challenge with option b is that if new kwargs or options are added to the ollama AsyncClient's .chat() method signature, they would not be supported in the autogen OllamaChatCompletionClient until the list of excepted kwargs is updated. However, I believe the AsyncClient's .chat() method signature is relatively stable, so my instinct would be to go with option b

Which packages was the bug in?

Python Extensions (autogen-ext)

AutoGen library version.

Python dev (main branch)

Other library version.

No response

Model used

ollama qwen2.5, but specific model is not relevant to this issue

Model provider

Ollama

Other model provider

No response

Python version

3.12

.NET version

None

Operating system

Ubuntu

@rylativity
Copy link
Author

Related conversation - #5254 (reply in thread)

@ekzhu ekzhu added this to the 0.4.8-python milestone Feb 23, 2025
rylativity pushed a commit to rylativity/autogen that referenced this issue Feb 23, 2025
…querytime when passing host (and other) kwargs to the OllamaChatCompletionClient constructor
@rylativity rylativity linked a pull request Feb 23, 2025 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants