diff --git a/evals/.gitignore b/evals/.gitignore new file mode 100644 index 00000000..523f67aa --- /dev/null +++ b/evals/.gitignore @@ -0,0 +1,3 @@ +results/ +.venv/ +uv.toml diff --git a/evals/README.md b/evals/README.md index 878f40bb..c4007cf7 100644 --- a/evals/README.md +++ b/evals/README.md @@ -4,7 +4,7 @@ This directory contains the evaluation suite for the main functionalities of the ## Overview -The evaluation uses the [evalbench](https://github.com/GoogleCloudPlatform/evalbench) framework with the Gemini CLI and Claude Code orchestrator to run a set of simulated user tasks against the agent. +The evaluation uses the [evalbench](https://github.com/GoogleCloudPlatform/evalbench) framework with the various agent harness orchestrators to run a set of simulated user tasks. ## Configuration Files @@ -33,12 +33,25 @@ The suite evaluates the agent across several dimensions using the following scor ## How to Run -From the `evals/` directory, execute the evaluation using latest [evalbench](https://github.com/GoogleCloudPlatform/evalbench/releases): - +From the `evals/` directory, run the evaluation using `uvx`: ```bash cd evals/ -evalbench run core-cujs/run_gemini_cli.yaml # Gemini CLI -evalbench run core-cujs/run_claude.yaml # Claude Code +UV_CONFIG_FILE=uv.toml uvx --index-url https://pypi.org/simple/ google-evalbench --experiment_config=core-cujs/run_gemini_cli.yaml ``` -Results will be generated in the `results/` directory as CSV reports. +### Local Execution Caveats +Unlike CI (which auto-injects values and queries the metadata server), running locally requires these specific alignments: + +1. **Node Version**: Switch your active shell session to Node v20+ before running to support modern regular expressions: + ```bash + source ~/.nvm/nvm.sh && nvm use 20 + ``` +2. **Dirty State Cleanup**: If a run crashes midway, wipe the dirty extension installation before retrying: + ```bash + rm -rf .venv/fake_home/.gemini/extensions/google-cloud-db-context-engineering + ``` +3. **Environment Variables**: Before executing a run, ensure you have exported your GCP project ID and the global endpoint location: + ```bash + export GOOGLE_CLOUD_PROJECT="your-gcp-project-id" + export GOOGLE_CLOUD_LOCATION="global" + ``` diff --git a/evals/core-cujs/run_gemini_cli.yaml b/evals/core-cujs/run_gemini_cli.yaml index 48e31ade..84dc5067 100644 --- a/evals/core-cujs/run_gemini_cli.yaml +++ b/evals/core-cujs/run_gemini_cli.yaml @@ -36,4 +36,4 @@ reporting: csv: output_directory: 'results' bigquery: - gcp_project_id: "${EVAL_REPORTING_PROJECT}" + gcp_project_id: !ENV "${EVAL_REPORTING_PROJECT}" diff --git a/evals/model_configs/gemini_model.yaml b/evals/model_configs/gemini_model.yaml index 4bb440a0..96a46ae4 100644 --- a/evals/model_configs/gemini_model.yaml +++ b/evals/model_configs/gemini_model.yaml @@ -1,4 +1,6 @@ generator: gcp_vertex_gemini vertex_model: gemini-3-flash-preview +gcp_project_id: !ENV "${GOOGLE_CLOUD_PROJECT}" +gcp_region: !ENV "${GOOGLE_CLOUD_LOCATION}" base_prompt: "" execs_per_minute: 5