feat(models): add SystemContentBlock support for provider-agnostic caching #1112
Conversation
dbschmigelski
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
dbschmigelski
had a problem deploying
to
auto-approve
GitHub Actions
Failure
dbschmigelski
had a problem deploying
to
auto-approve
GitHub Actions
Failure
dbschmigelski
had a problem deploying
to
auto-approve
GitHub Actions
Failure
| # Concatenate all text elements for backwards compatibility, None if no text found | ||
| text_parts = [block["text"] for block in system_prompt if "text" in block] | ||
| system_prompt_str = "\n".join(text_parts) if text_parts else None | ||
| return system_prompt_str, system_prompt |
There was a problem hiding this comment.
What if instead we just did None, system_prompt? Is there a reason we have to force generate a self.system_prompt when a customer passes in a list? They are opting in to a new feature and so I wouldn't expect it to break them.
There was a problem hiding this comment.
so when we call model.stream we still need to pass system_prompt. Even if a user opted in to passing this additional content, that should not break a model implementation.
So my thinking is that we will need to have the system_prompt_content -> system_prompt transformation regardless. So it is better to keep it and expose it as agent.system_prompt so it is minimally breaking in the event a user is doing "agent.system_prompt" or if a session_manager, hook, model, etc is itself making a call to agent.system_prompt
There was a problem hiding this comment.
Ah gotcha. So rather than requiring them to make changes in multiple places at once, they could do so gradually or be content with the transformation.
| # Use system_prompt_content directly (copy for mutability) | ||
| system_blocks: list[SystemContentBlock] = system_prompt_content.copy() if system_prompt_content else [] | ||
| # Add cache point if configured (backwards compatibility) | ||
| if cache_prompt := self.config.get("cache_prompt"): |
There was a problem hiding this comment.
What happens if someone already specified cache points in their system_prompt_cache? Won't this lead to duplicated entries?
There was a problem hiding this comment.
yes, but it won't it still works. And since it does, I wanted to avoid making assumptions about ordering. Like if cache_prompt is present and the last item is CachePoint then we dedupe. It doesn't break and we emit a warning.
Adding a test to demonstrate this
| system_prompt: Optional[str] = None, | ||
| *, | ||
| tool_choice: ToolChoice | None = None, | ||
| system_prompt_content: list[SystemContentBlock] | None = None, |
There was a problem hiding this comment.
Noting that I do not like this. But we are breaking if we did something like
system_prompt: Optional[str | list[SystemContentBlock]] = None,
Even if we only passed in str if the user passed in str, which was considered, the typing is still breaking for mypy users.
2 participants
Description
This PR introduces support for SystemContentBlock arrays in the Agent constructor, enabling provider-agnostic caching and multi-prompt system configurations.
Design Choices
Users can now pass SystemContentBlock arrays directly to the Agent constructor via the system_prompt parameter, which supports rich content including cache points, multiple text blocks, and future extensibility for other content types.
The system_prompt_content parameter becomes the authoritative source that flows through the entire pipeline from Agent initialization to model providers. When both system_prompt and system_prompt_content are provided, system_prompt_content takes precedence. The existing system_prompt parameter is maintained for backwards compatibility.
Implementation Strategy
The refactoring touches three key areas: Agent initialization logic that processes SystemContentBlock arrays, the streaming event loop that now accepts both parameters with clear precedence rules, and model provider implementations that handle the structured content appropriately.
For Bedrock specifically, the implementation leverages native SystemContentBlock support while deprecating the legacy cache_prompt configuration in favor of explicit cachePoint blocks within the system content. Other providers receive a concatenated string representation for compatibility.
Backwards Compatibility
All existing Agent constructor calls continue to work unchanged. The system_prompt parameter still functions as before, and existing Bedrock cache_prompt configurations are supported with deprecation warnings. The streaming interface maintains the same public API while internally handling both parameter types.
Provider Agnostic Benefits
This approach enables caching across all model providers through a unified interface rather than requiring provider-specific arguments. Users can define cache points, multi-prompt systems, and other advanced system configurations using the same SystemContentBlock format regardless of their chosen model provider.
The follow-up work will implement LiteLLM mappings to extend this provider-agnostic approach to additional model providers, making advanced system prompt features available across the entire ecosystem through this standardized interface.
Related Issues
#937
Documentation PR
ToDo
Type of Change
New feature
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.