feat: Add caching responses for agents #4363
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manuhortet
approved these changes
Aug 28, 2025
Comment on lines
+8304
to
+8306
| except Exception as e: | ||
|
|
||
| log_warning(f"Error checking model cache: {e}") |
There was a problem hiding this comment.
Suggested change
| except Exception as e: | |
| log_warning(f"Error checking model cache: {e}") | |
| except Exception as e: | |
| log_warning(f"Error checking model response cache: {e}") |
|
|
||
| log_debug(f"Streaming responses cached ({len(streaming_responses)} chunks) to: {cache_file}") | ||
| except Exception as e: | ||
| log_error(f"Error writing streaming cache: {e}") |
There was a problem hiding this comment.
In the case of streaming I would just cache the complete response. And also just deliver that - no need to deliver chunks if we have them all already
Comment on lines
+350
to
+351
| # Time-to-live for cached model responses in seconds | ||
| cache_model_ttl: int = 3600 |
There was a problem hiding this comment.
I would default to never expiring the cache (cache_model_ttl: int = 0)
Con is potential memory growth. Your call
ysolanky
reviewed
Aug 28, 2025
| from agno.models.openai.chat import OpenAIChat | ||
|
|
||
| agent = Agent( | ||
| model=OpenAIChat(id="o3-mini"), cache_model_response=True, debug_mode=True |
There was a problem hiding this comment.
Should cache_model_response not be a param of the model class instead?
ysolanky
reviewed
Aug 28, 2025
| functions=self._functions_for_model, | ||
| tool_choice=self.tool_choice, | ||
| tool_call_limit=self.tool_call_limit, | ||
| model_response = self._get_model_response_with_cache( |
There was a problem hiding this comment.
Here, the default behaviour should continue to be self.model.response but if caching is enabled, we should call this function.
Either way the function name here is misleading
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This adds caching of LLM responses as an opt-in for users. Very useful during development
Type of change
Checklist
./scripts/format.shand./scripts/validate.sh)Additional Notes
Add any important context (deployment instructions, screenshots, security considerations, etc.)