Conversation
… config (anthropics#415) - Call _maybe_filter_to_n_most_recent_images in sampling_loop prior to API call - Support filtering images from tool_result content and user messages - Add only_n_most_recent_images to session state defaults and sidebar config - Add unit tests for image filtering and bound application in sampling_loop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #415
Currently in
browser-use-demo,only_n_most_recent_imageswas defined as a parameter insampling_loopand passed fromstreamlit.py, but it was never invoked inside the sampling loop, and the stub_maybe_filter_to_n_most_recent_imagesdid not inspecttool_resultcontent blocks where screenshots are stored. As a result, screenshots accumulated unbounded in the message history, significantly inflating token usage over multi-step interactions.Changes
sampling_loop: Invoke_maybe_filter_to_n_most_recent_images(messages, only_n_most_recent_images)on each turn in the sampling loop before API calls.tool_resultand top-level image blocks: Updated_maybe_filter_to_n_most_recent_imagesto inspect and filter images within bothtool_resultcontent lists (preserving textual tool outputs) and direct message blocks.only_n_most_recent_images(defaulting to 3) in session state defaults and added a sidebar number input for user customization.TestRecentImagesFiltering) intests/test_sampling_loop.pyverifying image bounding, preservation of non-image tool results, and integration withsampling_loop.Verification
Ran test suite:
pytest browser-use-demo/tests/test_sampling_loop.py -k "TestRecentImagesFiltering" -vAll tests passed.