Skip to content

feat(hillclimb): use evalbench pipeline_debug_info in gap analysis #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wangauone wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(hillclimb): use evalbench pipeline_debug_info in gap analysis #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@ export EVAL_GCP_PROJECT_REGION="global"

# 5. Navigate to evals/ and execute the evaluation
cd evals/
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.12.0" google-evalbench --experiment_config=core-cujs/run_gemini_cli.yaml
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.16.0" google-evalbench --experiment_config=core-cujs/run_gemini_cli.yaml
```

To run Spanner Graph CUJs:
```bash
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.12.0" google-evalbench --experiment_config=spanner-graph-cujs/run_gemini_cli.yaml
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.16.0" google-evalbench --experiment_config=spanner-graph-cujs/run_gemini_cli.yaml
```

---
Expand Down Expand Up @@ -123,7 +123,7 @@ export EVAL_GCP_PROJECT_REGION="global"

# Execute evalbench from evals/ directory
cd evals
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.12.0" google-evalbench --experiment_config=spanner-graph-cujs/run_gemini_cli.yaml
uvx --default-index https://pypi.org/simple/ --from "google-evalbench==1.16.0" google-evalbench --experiment_config=spanner-graph-cujs/run_gemini_cli.yaml
```

---
Expand Down
2 changes: 1 addition & 1 deletion plugin/skills/context-engineering-evaluate/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Follow these steps exactly in order:

4. **Evalbench Run Integration:**
- Trigger the `run_shell_command` natively to execute the evaluation from the ROOT of the workspace using the following exact command template:
`uvx google-evalbench@1.12.0 --experiment_config=autoctx/experiments/<experiment_name>/eval_configs/run_config.yaml`
`uvx google-evalbench@1.16.0 --experiment_config=autoctx/experiments/<experiment_name>/eval_configs/run_config.yaml`
- Check the command outputs to ensure the evaluation reports materialize in the respective `autoctx/experiments/<experiment_name>/eval_reports/` directory.

## Output
Expand Down
1 change: 1 addition & 0 deletions plugin/skills/context-engineering-hillclimb/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Follow these steps exactly in order:
- Iterate through the failure cases by calling the tool with increasing `offset` (0, 10, 20, ...) until all failed queries are analyzed.
- **First Batch (offset=0)**: Initialize the report file with the `# Gap Analysis Report - vN` header and `## Summary` section, followed by the analysis of the first batch under `## Failed Queries Detail`.
- **Subsequent Batches**: Call the tool with the next offset, analyze the new failures, and **append** them to the `## Failed Queries Detail` section.
- **Use `pipeline_debug_info`**: If a failure case's **Additional Output** contains it, use this generation trace (which context the API retrieved and used) to ground the **Root Cause** and **Proposed Mutation** rather than guessing.

@gemini-code-assist gemini-code-assist Bot Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The parenthetical phrase (which context the API retrieved and used) is grammatically awkward. It would be clearer and more grammatically correct to phrase it as (showing which context the API retrieved and used) or (the context that the API retrieved and used) to improve readability for the agent/user reading this skill definition.


Use the following structure for the report:

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ dev = [

[tool.db-context-engineering]
toolbox_version = "1.4.0"
evalbench_version = "1.12.0"
evalbench_version = "1.16.0"

[tool.ruff]
line-length = 88
Expand Down
Loading