feat: add multi-modal attachment propagation to all worker agents thr…

[bittoby]

Enable Image Analysis for All Worker Agents

Problem

Only Multi-Modal Agent could analyze images. Worker agents (Developer, Browser, Document) couldn't see or process image attachments, causing tasks to fail.

Solution

• Added ImageAnalysisToolkit to Developer, Browser, and Document agents
• Registered toolkits with toolkits_to_register_agent parameter
• Updated system prompts with explicit image analysis instructions

Changes

Agent Factories:

• backend/app/agent/factory/developer.py
• backend/app/agent/factory/browser.py
• backend/app/agent/factory/document.py

System Prompts:

• backend/app/agent/prompt.py

Impact

All worker agents can now:

• Analyze screenshots and images
• Extract text from images
• Answer questions about visual content
• Process images alongside text in complex tasks

Testing

✅ Developer Agent: Extracts code from screenshots
✅ Browser Agent: Analyzes webpage screenshots
✅ Document Agent: Ready for document image analysis

Type

• Feature
• Bug Fix

Closes #956

[bittoby]

@Wendong-Fan Could you please review this PR? thanks

[Wendong-Fan]

@Wendong-Fan Could you please review this PR? thanks

thanks @bittoby 's contribution! could @nitpicker55555 and @Zephyroam help checking this?

[bittoby]

@Zephyroam @nitpicker55555 I would appreciate your feedback. please review the PR! thanks

[nitpicker55555]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?

Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

[bittoby]

@nitpicker55555 Obviously, after I tested it correctly I pushed the PR. When I attach a test image and input “As a developer, analyze this screenshot and write code” as prompt, then the developer agent generates HTML that matches the screenshot.

[bittoby]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?

Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

you're right - I haven't added ImageAnalysisToolkit to the worker agents yet. The current implementation works because it relies on the LLM's native vision capability (like gpt-5), but I agree we should add the toolkit for explicit tool calls and non-vision model support. Would you prefer I complete the current approach by adding the toolkit, or switch to your suggestion of modifying the decompose prompt to selectively pass images?

[nitpicker55555]

Obviously, you didn’t test your code. You didn’t register the ImageAnalysisToolkit for the agent—so how is the agent supposed to gain image-reading capability?
Additionally, this design is overly complicated. Why not simply modify the prompt of the decompose agent so that, when breaking down the task, it passes the image location to the corresponding agent? That way, we would only need to adjust the prompt and register the ImageAnalysisToolkit for the agent.

you're right - I haven't added ImageAnalysisToolkit to the worker agents yet. The current implementation works because it relies on the LLM's native vision capability (like gpt-5), but I agree we should add the toolkit for explicit tool calls and non-vision model support. Would you prefer I complete the current approach by adding the toolkit, or switch to your suggestion of modifying the decompose prompt to selectively pass images?

@bittoby Thank you for the explanation, but I still don’t understand: without providing a dedicated tool, how can the model read an image from a passed-in image URL instead of a base64-encoded input? Could you point out which part of your PR implements this functionality?

[bittoby]

Right now it only passes the image file paths along as additional_info. The current setup depends on the model’s built-in vision (which needs base64 data or a URL in the actual message). My PR doesn’t do that conversion or attach the images to the message - it just sends the paths as metadata. I will update PR. I misunderstood. I am clear now.

For this to really work, we need to either:

1. add ImageAnalysisToolkit to the worker agents, or
2. change how we build the LLM request so it includes the images as base64.

Which option do you recommend?

[nitpicker55555]

My idea is to tweak ImageAnalysisToolkit so it supports taking an image path as input and returning the actual image data back to the agent (right now ImageAnalysisToolkit only returns an image description).

The benefits are: the change is relatively small, we’d only need to update the decompose agent’s prompt to pass along the image path. It would also allow the agent to read other image files, not just images uploaded by the user in the prompt, and it could support users providing an image path instead of the image itself.

I also looked into how Claude Code does image reading, and it works in a similar way, via a Read tool.

What do you think @Wendong-Fan @bittoby

[bittoby]

@nitpicker55555 Thanks for your explanation!
Your idea (extending ImageAnalysisToolkit) is a solid quick win, but I see some limitations:

Concerns:

• Mixed responsibility: one toolkit ends up doing both “read” and “analyze”
• Less reusable: e.g., Document Agent might only need to read images for embeddings, but it would still have to pull in the whole analysis toolkit
• Heavier deps everywhere: any agent that needs to read an image would also load the analysis dependencies

My approach (like Claude Code): make a small, separate tool:

• read_image(path) -> base64
• Keep ImageAnalysisToolkit focused on analysis only
• More flexible: agents can read images for embeddings, uploads, etc.

But it requires more code, a bit more wiring

I agree with your approach for now since it's faster to ship. We can refactor to a separate ReadImageTool later if we see reusability issues.

@Wendong-Fan what's your preference?

[nitpicker55555]

@nitpicker55555 Thanks for your explanation!

Your idea (extending ImageAnalysisToolkit) is a solid quick win, but I see some limitations:

Concerns:

• Mixed responsibility: one toolkit ends up doing both “read” and “analyze”

• Less reusable: e.g., Document Agent might only need to read images for embeddings, but it would still have to pull in the whole analysis toolkit

• Heavier deps everywhere: any agent that needs to read an image would also load the analysis dependencies

My approach (like Claude Code): make a small, separate tool:

• read_image(path) -> base64

• Keep ImageAnalysisToolkit focused on analysis only

• More flexible: agents can read images for embeddings, uploads, etc.

But it requires more code, a bit more wiring

I agree with your approach for now since it's faster to ship. We can refactor to a separate ReadImageTool later if we see reusability issues.

@Wendong-Fan what's your preference?

@bittoby what do you mean document agent read image for embedding?

[bittoby]

I mean, here "embedding" = putting the image into a document not analyzing it.
For example:
User uploads screenshot.png and asks: "Create a PDF report with this screenshot",
Document Agent needs to read the png from disk , convert it to base64 data then insert it to pdf. It doesn't need analyze the png, describe or extract text etc

[nitpicker55555]

@bittoby It seems you haven’t reviewed this tool at all. I strongly recommend checking the codebase before continuing the discussion.
ImageAnalysisToolkit has only one dependency—Pillow—which is also required for simple image reading. The difference between this tool and the “read-only image” functionality we need is merely whether we leverage the built-in agent to generate a description. From the agent’s perspective, the only difference lies in the arguments passed in.
We could, of course, implement a separate tool as you suggested, but the supposed benefit of being “more flexible: agents can read images for embeddings, uploads, etc.” does not really exist.

[bittoby]

Okay. @nitpicker55555 I’ll update the PR to follow your approach.

[bittoby]

@nitpicker55555 I updated pr to follow your feedback. please review again