Unable to serialize unknown type: <class 'PIL.Image.Image> Websockets Autogen > 0.7

[julie2025]

What happened?

Describe the bug
I'm encountering this error while working with Autogen 0.7.xx. I'm using the multimodal agent configured with stream=True and trying to call it inside a WebSocket to get the stream response from the agent. This is the error message: IOWebsockets._handler(): Error in on_connect: Unable to serialize unknown type: <class 'PIL.Image.Image'>
I'm using Autogen's WebSocket configuration, and it works fine with the ConversableAgent to get the stream response, but it doesn't work with the Multimodal ConversableAgent.

To Reproduce
With Autogen =>0.7.xx

1. Set up a WebSocket server with autogen.io.

2. Use a Python client to connect to the WebSocket and request a streamed response from a MultimodalConversableAgent.

3. Send a message containing an image (e.g., <img http://...>).

4. Observe the error when attempting to serialize the PIL.Image.Image object.

Expected behavior
The WebSocket should stream the response chunks from the MultimodalConversableAgent without errors, even when handling image data.

Screenshots
Workaround: Downgrading Autogen version to =<6

Additional context
Add any other context about the problem here.

Which packages was the bug in?

Python Core (autogen-core)

AutoGen library version.

Python dev (main branch)

[jackgerrits]

Can you please provide code to reproduce?

Workaround: Downgrading Autogen version to =<6

Please note that we do not have any packages with that version. There is either the old 0.2 codebase at 0.2.40 (autogen-agentchat==0.2.40), or the new code base at 0.4.7 (autogen-agentchat==0.4.7)

[julie2025]

Hello, is this version of the agent chat autogen-agentchat==0.4.7 and pyautogen==0.7.4

code snippet to reproduce

from vision_assistant import VisionAgent
from autogen.io import IOWebsockets
import json

def on_connect(iostream: IOWebsockets) -> None:
    # msg from client
    initial_msg = iostream.input()
    print('input msg', initial_msg)
    if initial_msg:
        data = json.loads(initial_msg)
        question = data.get('message')
        agent_id = data.get("agentId") 
    agent = None
    if agent_id == "vision_agent":
        agent = VisionAgent(conversation_id)
        agent.initialize_agent()  
    response = agent.process_message(question)
    print("response", response) 

if __name__ == "__main__":
    # start ws server
    with IOWebsockets.run_server_in_thread(on_connect=on_connect, port=8765) as uri:
        print(f"WebSocket server listening on {uri}")
        input("Enter to stop...\n")

### Agent

from autogen.agentchat.contrib.multimodal_conversable_agent import MultimodalConversableAgent
"Need to define VisionAgent and add user proxy and model configuration"

vision_agent = MultimodalConversableAgent(
        name="visionAgent",
        system_message="""
        You are a helpful assistant capable of analysing images.
        """,
        llm_config=llm_config_vision,
    )
def process_message(self, user_message):
    response = self.user_proxy.assist(
            self.vision_agent,
            message= user_message,
            clear_history=False
    )
    return response

Test connection to socket server with Postman

connect to ws://127.0.0.1:8765
message payload:
{
"message": "Explain this image <img https://images-assets.nasa.gov/image/iss071e143785/iss071e143785~medium.jpg>",
"assistantId": "vision_agent"
}

[jackgerrits]

We do not publish the pyautogen package at the moment, see the FAQ.

The code you pasted is not using AutoGen 0.4 it is using the pyautogen dependency. For us to help you please uninstall pyautogen and use just autogen-agentchat either a version in 0.2.x or 0.4.x