feat(hillclimb): use evalbench pipeline_debug_info in gap analysis - #213
feat(hillclimb): use evalbench pipeline_debug_info in gap analysis#213wangauone wants to merge 1 commit into
Conversation
…ump evalbench to 1.16.0
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the google-evalbench version from 1.12.0 to 1.16.0 across evals/README.md, plugin/skills/context-engineering-evaluate/SKILL.md, and pyproject.toml. It also adds instructions in plugin/skills/context-engineering-hillclimb/SKILL.md to utilize pipeline_debug_info when available. Feedback is provided to improve the grammatical phrasing of the newly added parenthetical explanation in the hillclimb skill definition for better clarity.
| - Iterate through the failure cases by calling the tool with increasing `offset` (0, 10, 20, ...) until all failed queries are analyzed. | ||
| - **First Batch (offset=0)**: Initialize the report file with the `# Gap Analysis Report - vN` header and `## Summary` section, followed by the analysis of the first batch under `## Failed Queries Detail`. | ||
| - **Subsequent Batches**: Call the tool with the next offset, analyze the new failures, and **append** them to the `## Failed Queries Detail` section. | ||
| - **Use `pipeline_debug_info`**: If a failure case's **Additional Output** contains it, use this generation trace (which context the API retrieved and used) to ground the **Root Cause** and **Proposed Mutation** rather than guessing. |
There was a problem hiding this comment.
The parenthetical phrase (which context the API retrieved and used) is grammatically awkward. It would be clearer and more grammatically correct to phrase it as (showing which context the API retrieved and used) or (the context that the API retrieved and used) to improve readability for the agent/user reading this skill definition.
1 participant
Teaches the hill-climbing gap analysis to use pipeline_debug_info (the Query Data API's generation trace, surfaced in eval report Additional Output) to ground root-cause and proposed mutations. Bumps the evalbench pin to 1.16.0, the release that will include the pipeline_debug_info support (GoogleCloudPlatform/evalbench#570).
This PR won't be merged until we have a new EvalBench release.