Skip to content

fix: improvements to ensure agent doesn't get stuck on local CUJ run #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
g-lynnzee wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: improvements to ensure agent doesn't get stuck on local CUJ run #167

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc07280
fix: improvements to ensure agent doesn't get stuck on local CUJ run
g-lynnzee Jun 22, 2026
e78fd26
Update evals/README.md
g-lynnzee Jun 22, 2026
9f53aa3
Add in !ENV for proper variable resolution for local runs
g-lynnzee Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions evals/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
results/
.venv/
uv.toml
25 changes: 19 additions & 6 deletions evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This directory contains the evaluation suite for the main functionalities of the

## Overview

The evaluation uses the [evalbench](https://github.com/GoogleCloudPlatform/evalbench) framework with the Gemini CLI and Claude Code orchestrator to run a set of simulated user tasks against the agent.
The evaluation uses the [evalbench](https://github.com/GoogleCloudPlatform/evalbench) framework with the various agent harness orchestrators to run a set of simulated user tasks.

## Configuration Files

Expand Down Expand Up @@ -33,12 +33,25 @@ The suite evaluates the agent across several dimensions using the following scor

## How to Run

From the `evals/` directory, execute the evaluation using latest [evalbench](https://github.com/GoogleCloudPlatform/evalbench/releases):

From the `evals/` directory, run the evaluation using `uvx`:
```bash
cd evals/
evalbench run core-cujs/run_gemini_cli.yaml # Gemini CLI
evalbench run core-cujs/run_claude.yaml # Claude Code
UV_CONFIG_FILE=uv.toml uvx --index-url https://pypi.org/simple/ google-evalbench --experiment_config=core-cujs/run_gemini_cli.yaml
```

Results will be generated in the `results/` directory as CSV reports.
### Local Execution Caveats
Unlike CI (which auto-injects values and queries the metadata server), running locally requires these specific alignments:

1. **Node Version**: Switch your active shell session to Node v20+ before running to support modern regular expressions:
```bash
source ~/.nvm/nvm.sh && nvm use 20
```
2. **Dirty State Cleanup**: If a run crashes midway, wipe the dirty extension installation before retrying:
```bash
rm -rf evals/.venv/fake_home/.gemini/extensions/google-cloud-db-context-engineering
Comment thread
g-lynnzee marked this conversation as resolved.
Outdated
```
3. **Environment Variables**: Before executing a run, ensure you have exported your GCP project ID and the global endpoint location:
```bash
export GOOGLE_CLOUD_PROJECT="your-gcp-project-id"
export GOOGLE_CLOUD_LOCATION="global"
```
2 changes: 2 additions & 0 deletions evals/model_configs/gemini_model.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
generator: gcp_vertex_gemini
vertex_model: gemini-3-flash-preview
gcp_project_id: "${GOOGLE_CLOUD_PROJECT}"
gcp_region: "${GOOGLE_CLOUD_LOCATION:global}"
base_prompt: ""
execs_per_minute: 5
Loading