Allow tools to return a list of Parts and submit these parts as-is to the LLM model

[slilichenko]

Currently, it's very non-intuitive to submit non-text Parts to the model, unless the user can construct the request to the model with one of those parts attached. For example, an agent that lists a set of images and let's the user indicate an image to be processed by an LLM currently requires a combination of a special tool to set the image reference in the state and a before-model-callback to check the state and add the Part.from_bytes() to the llm_request.contents and clear the state.

If a FunctionTool could return a list of parts, including the Part.from_function_response() required by the spec, and ADK would add these parts without any conversions to the contents - it would simplify agents that need to process binary files.

[Jacksunwei]

Hi @slilichenko , returning blob from function result would lead to the same blob being sent to LLM in every request. That is not an efficient way of using your quota :)

We actually have load_artifact_tool in google.adk.tools package. In function result, if you have the artifact name returned and the LLM will be able to load the blob on-demand. That's more efficient way of handling blobs.

---

LMK if you think a sample would be helpful for you.

[slilichenko]

Thank you for pointing to that tool! I understand your concern - the function result will be part of session history and will affect the size of the content going forward, unless it gets removed by the content compression.

But load_artifacts only works for a subset of use cases - for files which are available in the artifact store. There is another set of those cases - files that are available as references (stored on Cloud Storage) and which don't need to be brought into the artifact store in order to be processed. See this repo for sample agents that use those references. Sending binary files as Part.from_uri() will not increase the size of the content (assuming the LLM will not re-process them unless they are needed). It can also prevent model confusion where there is an explicit prompt which requires re-processing the image submitted on a previous turn, but the image itself is not available.

IMHO, it's not one or the other - both approaches can be useful, depending on the use case.

[RKest]

@Jacksunwei I can work on this tomorrow if nobody picks this up today🙏

[RKest]

I tried a few approaches, but none of them seem to ideal. @Jacksunwei could you please advise how should we proceed?

1. Naive approach

In __build_response_event, we can handle a special case for when user returned list[Part]

Implementation

def __build_response_event(
    tool: BaseTool,
    function_result: Union[dict[str, object], list[types.Part]],
    tool_context: ToolContext,
    invocation_context: InvocationContext,
) -> Event:
  # New special case ⬇️⬇️⬇️⬇️⬇️⬇️
  if isinstance(function_result, list) and isinstance(function_result[0], types.Part):
    content = types.Content(
        role='user',
        parts=function_result + [types.Part(function_response=
          types.FunctionResponse(name=tool.name, id=tool_context.function_call_id)
        )],
    )
  # Current implementation ⬇️⬇️⬇️⬇️⬇️⬇️
  else:
    if not isinstance(function_result, dict):
      function_result = {'result': function_result}

    part_function_response = types.Part.from_function_response(
        name=tool.name, response=function_result
    )
    part_function_response.function_response.id = tool_context.function_call_id

    content = types.Content(
        role='user',
        parts=[part_function_response],
    )

  function_response_event = Event(
      invocation_id=invocation_context.invocation_id,
      author=invocation_context.agent.name,
      content=content,
      actions=tool_context.actions,
      branch=invocation_context.branch,
  )

  return function_response_event

With this change, the following agent is able to describe what's on the image.

Agent example

from google.adk import Agent
from google.genai import types

def get_image():
  """Get the contents of an image."""
  return [types.Part.from_uri(file_uri="gs://maxind-adk-exp-bucket/totally_a_duck.png")]

root_agent = Agent(
    model='gemini-2.0-flash',
    name='hello_world_agent',
    instruction="""
        Whatever the user says, retrieve an image using the 'get_image' tool, and describe what's on it.
    """,
    tools=[
        get_image,
    ],
)

Issue is this changes the ADK data model, as currently only a single Part with function_response field set, can be a result of a tool call.
I worry this might lead to regressions in other areas, and as a first example, I see the following artifact in how ADK web is rendered.

2. Save parts in FunctionResponse.parts field.

Implementation

def __build_response_event(
    tool: BaseTool,
    function_result: Union[dict[str, object], list[types.Part]],
    tool_context: ToolContext,
    invocation_context: InvocationContext,
) -> Event:
  function_response = types.FunctionResponse(name=tool.name, id=tool_context.function_call_id)
  # 1. Treat empty list as function result with empty list.
  if isinstance(function_result, list) and len(function_result) == 0:
    function_response.response = {"result": []}
  # Branch of interest ⬇️⬇️⬇️⬇️⬇️⬇️⬇️⬇️⬇️⬇️⬇️⬇️
  # 2. If tool returns parts, construct function response from these parts directly.
  elif isinstance(function_result, list) and isinstance(function_result[0], types.FunctionResponsePart):
    function_response.parts = function_result
  # 3. Otherwise if function returns a dict, pass it in function response.
  elif isinstance(function_result, dict):
    function_response.response = function_result
  # 4. Otherwise wrap result in a dict, because spec requires the result to be a dict.
  else:
    function_response.response = {"result": function_result}

  content = types.Content(
      role='user',
      parts=[types.Part(function_response=function_response)],
  )

  function_response_event = Event(
      invocation_id=invocation_context.invocation_id,
      author=invocation_context.agent.name,
      content=content,
      actions=tool_context.actions,
      branch=invocation_context.branch,
  )

  return function_response_event

Agent example

from google.adk import Agent
from google.genai import types

def get_image():
  """Get the contents of an image."""
  return [types.FunctionResponsePart.from_uri(file_uri="gs://maxind-adk-exp-bucket/totally_a_duck.png")]

root_agent = Agent(
    model='gemini-2.0-flash',
    name='hello_world_agent',
    instruction="""
        Whatever the user says, retrieve an image using the 'get_image' tool, and describe what's on it.
    """,
    tools=[
        get_image,
    ],
)

In theory this should work without breaking the data model, but it results in the following exception (summarized):

generic::invalid_argument: The field \'parts\' in function response is only supported with computer use tool.

Full exception (only missing the server's stack trace)

google.genai.errors.ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.DebugInfo', 'detail': '[ORIGINAL ERROR] generic::invalid_argument: The field 'parts' in function response is only supported with computer use tool.; Error raised from operator generate_multi_modal; stmt:7 at g15#0 193211503: "" 294320659: "" 525003295 { 1 { 1 { 1 { 1: "blade:cloud.ai.large_models.vision.videogenerativeservice-prod-us" 2: "VideoGenerativeService" 3: "UnaryMediaGenerateContent" } 2 { 1: 3 2: "generic" 3: "The field \'parts\' in function response is only supported with computer use tool.; Error raised from operator generate_multi_modal; stmt:7 at g15#0" 5 { 1 { 2: 193211503 3: "" } 1 { 2: 294320659 3: "" } 1 { 2: 525003483 3 { 1: 2 2: "[2002:a05:633e:248:b0:46:79cc:447]:4001" 3: ... server stack trace

I haven't found a mitigation.

3. Use url_context_tool

Issue: This tool doesn't support file URIs, only HTML websites

My recommendation

I think the best course of action is to request support for FunctionResponse.parts field.
It is part of the public API, but it's only used in a private native tool, so this may even be considered a bug in the gemini API.

[RKest]

I came up with another option, which is to write a plugin which adds this functionality, it's currently in review🙏