-
Notifications
You must be signed in to change notification settings - Fork 2.2k
feat: Enhance error messages for tool and agent not found errors #3219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
22188c3
feat: Enhance error messages for tool and agent not found errors
jpantsjoha b72e529
refactor: extract shared error message utility for DRY principle
jpantsjoha 62b5f6c
Merge branch 'main' into feat/better-error-messages
jpantsjoha 1806fbb
chore: merge latest main to resolve conflicts
jpantsjoha ea68d60
Merge branch 'feat/better-error-messages' of github.com-jpantsjoha:jp…
jpantsjoha e1249e2
Merge branch 'main' into feat/better-error-messages
jpantsjoha ef9b40f
Merge branch 'main' into feat/better-error-messages
Jacksunwei a4df8bf
Merge branch 'main' into feat/better-error-messages
yyyu-google File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # Copyright 2025 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Utility functions for generating enhanced error messages.""" | ||
|
|
||
| from difflib import get_close_matches | ||
|
|
||
|
|
||
| def format_not_found_error( | ||
| item_name: str, | ||
| item_type: str, | ||
| available_items: list[str], | ||
| causes: list[str], | ||
| fixes: list[str], | ||
| ) -> str: | ||
| """Format an enhanced 'not found' error message with fuzzy matching. | ||
|
|
||
| This utility creates consistent, actionable error messages when tools, | ||
| agents, or other named items cannot be found. It includes: | ||
| - Clear identification of what was not found | ||
| - List of available items (truncated to 20 for readability) | ||
| - Possible causes for the error | ||
| - Suggested fixes | ||
| - Fuzzy matching suggestions for typos | ||
|
|
||
| Args: | ||
| item_name: The name of the item that was not found. | ||
| item_type: The type of item (e.g., 'tool', 'agent', 'function'). | ||
| available_items: List of available item names. | ||
| causes: List of possible causes for the error. | ||
| fixes: List of suggested fixes. | ||
|
|
||
| Returns: | ||
| Formatted error message string with all components. | ||
|
|
||
| Example: | ||
| >>> error_msg = format_not_found_error( | ||
| ... item_name='get_wether', | ||
| ... item_type='tool', | ||
| ... available_items=['get_weather', 'calculate_sum'], | ||
| ... causes=['LLM hallucinated the name', 'Typo in function name'], | ||
| ... fixes=['Check spelling', 'Verify tool is registered'] | ||
| ... ) | ||
| >>> raise ValueError(error_msg) | ||
| """ | ||
| # Truncate available items to first 20 for readability | ||
| if len(available_items) > 20: | ||
| items_preview = ', '.join(available_items[:20]) | ||
| items_msg = ( | ||
| f'Available {item_type}s (showing first 20 of' | ||
| f' {len(available_items)}): {items_preview}...' | ||
| ) | ||
| else: | ||
| items_msg = f"Available {item_type}s: {', '.join(available_items)}" | ||
|
|
||
| # Build error message from parts | ||
| error_parts = [ | ||
| f"{item_type.capitalize()} '{item_name}' is not found.", | ||
| items_msg, | ||
| 'Possible causes:\n' | ||
| + '\n'.join(f' {i+1}. {cause}' for i, cause in enumerate(causes)), | ||
| 'Suggested fixes:\n' + '\n'.join(f' - {fix}' for fix in fixes), | ||
| ] | ||
|
|
||
| # Add fuzzy matching suggestions for typos | ||
| close_matches = get_close_matches(item_name, available_items, n=3, cutoff=0.6) | ||
| if close_matches: | ||
| suggestions = '\n'.join(f' - {match}' for match in close_matches) | ||
| error_parts.append(f'Did you mean one of these?\n{suggestions}') | ||
|
|
||
| return '\n\n'.join(error_parts) | ||
109 changes: 109 additions & 0 deletions
109
tests/unittests/agents/test_llm_agent_error_messages.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Copyright 2025 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Tests for enhanced error messages in agent handling.""" | ||
| from google.adk.agents import LlmAgent | ||
| import pytest | ||
|
|
||
|
|
||
| def test_agent_not_found_enhanced_error(): | ||
| """Verify enhanced error message for agent not found.""" | ||
| root_agent = LlmAgent( | ||
| name='root', | ||
| model='gemini-2.0-flash', | ||
| sub_agents=[ | ||
| LlmAgent(name='agent_a', model='gemini-2.0-flash'), | ||
| LlmAgent(name='agent_b', model='gemini-2.0-flash'), | ||
| ], | ||
| ) | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| root_agent._LlmAgent__get_agent_to_run('nonexistent_agent') | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify error message components | ||
| assert 'nonexistent_agent' in error_msg | ||
| assert 'Available agents:' in error_msg | ||
| assert 'agent_a' in error_msg | ||
| assert 'agent_b' in error_msg | ||
| assert 'Possible causes:' in error_msg | ||
| assert 'Suggested fixes:' in error_msg | ||
|
|
||
|
|
||
| def test_agent_not_found_fuzzy_matching(): | ||
| """Verify fuzzy matching for agent names.""" | ||
| root_agent = LlmAgent( | ||
| name='root', | ||
| model='gemini-2.0-flash', | ||
| sub_agents=[ | ||
| LlmAgent(name='approval_handler', model='gemini-2.0-flash'), | ||
| ], | ||
| ) | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| root_agent._LlmAgent__get_agent_to_run('aproval_handler') # Typo | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify fuzzy matching suggests correct agent | ||
| assert 'Did you mean' in error_msg | ||
| assert 'approval_handler' in error_msg | ||
|
|
||
|
|
||
| def test_agent_tree_traversal(): | ||
| """Verify agent tree traversal helper works correctly.""" | ||
| root_agent = LlmAgent( | ||
| name='orchestrator', | ||
| model='gemini-2.0-flash', | ||
| sub_agents=[ | ||
| LlmAgent( | ||
| name='parent_agent', | ||
| model='gemini-2.0-flash', | ||
| sub_agents=[ | ||
| LlmAgent(name='child_agent', model='gemini-2.0-flash'), | ||
| ], | ||
| ), | ||
| ], | ||
| ) | ||
|
|
||
| available_agents = root_agent._get_available_agent_names() | ||
|
|
||
| # Verify all agents in tree are found | ||
| assert 'orchestrator' in available_agents | ||
| assert 'parent_agent' in available_agents | ||
| assert 'child_agent' in available_agents | ||
| assert len(available_agents) == 3 | ||
|
|
||
|
|
||
| def test_agent_not_found_truncates_long_list(): | ||
| """Verify error message truncates when 100+ agents exist.""" | ||
| # Create 100 sub-agents | ||
| sub_agents = [ | ||
| LlmAgent(name=f'agent_{i}', model='gemini-2.0-flash') for i in range(100) | ||
| ] | ||
|
|
||
| root_agent = LlmAgent( | ||
| name='root', model='gemini-2.0-flash', sub_agents=sub_agents | ||
| ) | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| root_agent._LlmAgent__get_agent_to_run('nonexistent') | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify truncation message | ||
| assert 'showing first 20 of' in error_msg | ||
| assert 'agent_0' in error_msg # First agent shown | ||
| assert 'agent_99' not in error_msg # Last agent NOT shown |
105 changes: 105 additions & 0 deletions
105
tests/unittests/flows/llm_flows/test_functions_error_messages.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| # Copyright 2025 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Tests for enhanced error messages in function tool handling.""" | ||
| from google.adk.flows.llm_flows.functions import _get_tool | ||
| from google.adk.tools import BaseTool | ||
| from google.genai import types | ||
| import pytest | ||
|
|
||
|
|
||
| # Mock tool for testing error messages | ||
| class MockTool(BaseTool): | ||
| """Mock tool for testing error messages.""" | ||
|
|
||
| def __init__(self, name: str = 'mock_tool'): | ||
| super().__init__(name=name, description=f'Mock tool: {name}') | ||
|
|
||
| def call(self, *args, **kwargs): | ||
| return 'mock_response' | ||
|
|
||
|
|
||
| def test_tool_not_found_enhanced_error(): | ||
| """Verify enhanced error message for tool not found.""" | ||
| function_call = types.FunctionCall(name='nonexistent_tool', args={}) | ||
| tools_dict = { | ||
| 'get_weather': MockTool(name='get_weather'), | ||
| 'calculate_sum': MockTool(name='calculate_sum'), | ||
| 'search_database': MockTool(name='search_database'), | ||
| } | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| _get_tool(function_call, tools_dict) | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify error message components | ||
| assert 'nonexistent_tool' in error_msg | ||
| assert 'Available tools:' in error_msg | ||
| assert 'get_weather' in error_msg | ||
| assert 'Possible causes:' in error_msg | ||
| assert 'Suggested fixes:' in error_msg | ||
|
|
||
|
|
||
| def test_tool_not_found_fuzzy_matching(): | ||
| """Verify fuzzy matching suggestions in error message.""" | ||
| function_call = types.FunctionCall(name='get_wether', args={}) # Typo | ||
| tools_dict = { | ||
| 'get_weather': MockTool(name='get_weather'), | ||
| 'calculate_sum': MockTool(name='calculate_sum'), | ||
| } | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| _get_tool(function_call, tools_dict) | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify fuzzy matching suggests correct tool | ||
| assert 'Did you mean' in error_msg | ||
| assert 'get_weather' in error_msg | ||
|
|
||
|
|
||
| def test_tool_not_found_no_fuzzy_match(): | ||
| """Verify error message when no close matches exist.""" | ||
| function_call = types.FunctionCall(name='completely_different', args={}) | ||
| tools_dict = { | ||
| 'get_weather': MockTool(name='get_weather'), | ||
| 'calculate_sum': MockTool(name='calculate_sum'), | ||
| } | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| _get_tool(function_call, tools_dict) | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify no fuzzy matching section when no close matches | ||
| assert 'Did you mean' not in error_msg | ||
|
|
||
|
|
||
| def test_tool_not_found_truncates_long_list(): | ||
| """Verify error message truncates when 100+ tools exist.""" | ||
| function_call = types.FunctionCall(name='nonexistent', args={}) | ||
|
|
||
| # Create 100 tools | ||
| tools_dict = {f'tool_{i}': MockTool(name=f'tool_{i}') for i in range(100)} | ||
|
|
||
| with pytest.raises(ValueError) as exc_info: | ||
| _get_tool(function_call, tools_dict) | ||
|
|
||
| error_msg = str(exc_info.value) | ||
|
|
||
| # Verify truncation message | ||
| assert 'showing first 20 of 100' in error_msg | ||
| assert 'tool_0' in error_msg # First tool shown | ||
| assert 'tool_99' not in error_msg # Last tool NOT shown |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
'from future import annotations' is missing
This import is required to allow forward references in type annotations without quotes.
could you add this import here? And then remember to run autoformat.sh script to make sure the formatter test pass after the change
Thank you