Added benchmarks for Claude Sonnet 3.5 #9
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Generated by copilot:
This pull request includes several updates across different files to improve the functionality and documentation of the forecasting tools and front-end application. The most important changes include updating model limits and descriptions, enhancing search input constraints, and refining benchmark page details.
Updates to model limits and descriptions:
forecasting_tools/ai_models/claude35sonnet.py: Updated theREQUESTS_PER_PERIOD_LIMITto 1,750 andTOKENS_PER_PERIOD_LIMITto 140,000. Added comments about the latest model versions.Enhancements to search input constraints:
forecasting_tools/ai_models/exa_searcher.py: Clarified descriptions forinclude_domainsandexclude_domainsfields to specify that they constrain the search results to only include or exclude specified domains.forecasting_tools/forecasting/llms/smart_searcher.py: Added instruction to return JSON in one combined list instead of separate chunks.Refinements to benchmark page details:
front_end/app_pages/benchmark_page.py: Updated benchmark file descriptions and added a new entry for "Claude 3.5 Sonnet for all tasks". Changed the title to "📈 Benchmarking Forecast Bot".