Skip to content

Commit ad5178b

Browse files
prernakakkar-googleprernakakkar-google
committed
Merge branch 'main' into add-evalbench-review-skill
2 parents e5d65ba + 1a305c7 commit ad5178b

27 files changed

Lines changed: 2026 additions & 606 deletions

‎README.md‎

Lines changed: 50 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
11
# EvalBench
22

3-
EvalBench is a flexible framework designed to measure the quality of generative AI (GenAI) workflows around database specific tasks. As of now, it provides a comprehensive set of tools, and modules to evaluate models on NL2SQL tasks, including capability of running and scoring DQL, DML, and DDL queries across multiple supported databases. Its modular, plug-and-play architecture allows you to seamlessly integrate custom components while leveraging a robust evaluation pipeline, result storage, scoring strategies, and dashboarding capabilities.
3+
EvalBench is a flexible framework designed to measure the quality of generative AI (GenAI) workflows. It supports two broad classes of evaluation:
4+
5+
- **NL2SQL / database tasks** — running and scoring DQL, DML, and DDL queries across multiple supported databases (AlloyDB, BigQuery, Spanner, PostgreSQL, MySQL, SQLite, and more).
6+
- **Agentic evaluations** — driving real coding agents and CLIs (Gemini CLI, Claude Code, Codex CLI, Antigravity CLI) through multi-turn scenarios with an LLM-based simulated user, then scoring their tool-call trajectories, goal completion, and behavior.
7+
8+
Its modular, plug-and-play architecture allows you to seamlessly integrate custom components while leveraging a robust evaluation pipeline, result storage, scoring strategies, and dashboarding capabilities.
49

510
---
611

@@ -76,11 +81,40 @@ Start the evaluation process using the provided shell script:
7681

7782
---
7883

84+
## Agentic Evaluations
85+
86+
Beyond single-turn NL2SQL, EvalBench evaluates **agents** — coding CLIs and data agents that reason across multiple turns, call tools, and act on their environment.
87+
88+
Each scenario starts from a prompt and is driven forward by an **LLM-based simulated user** that follows a `conversation_plan` until the goal is met, a terminal state is detected, or `max_turns` is reached. Every turn is captured — text, tool calls, parameters, latency, and tokens — and then scored. Runs are **sandboxed**, so agents never touch your local CLI settings and scenarios can run concurrently.
89+
90+
| Agent | Generator | Orchestrator | Guide |
91+
|---|---|---|---|
92+
| Gemini CLI | `gemini_cli` | `agent` / `geminicli` | [Gemini CLI guide](/docs/gemini_cli_agent_testing.md) |
93+
| Claude Code | `claude_code` | `agent` | [Claude Code guide](/docs/claude_code_agent_testing.md) |
94+
| Codex CLI | `codex_cli` | `agent` | [Codex CLI guide](/docs/codex_cli_agent_testing.md) |
95+
| Antigravity (agy) CLI | `agy_cli` | `agent` | [Antigravity CLI guide](/docs/agy_cli_agent_testing.md) |
96+
| Conversational data agents | `dataagent` | `dataagent` / `interact` | [Data agent spec](/docs/dataagent_spec.md) |
97+
98+
Agents can be evaluated against tools wired up as **MCP servers**, **plugins**, **extensions**, or **skills** — or against a **fake MCP** stub for fast, offline, zero-cost testing. Plugins are installed from a git repo or local directory through each CLI's marketplace and may bundle skills, MCP servers, or both. Trajectories are scored for tool-call accuracy, goal completion, hallucination and clarification behavior, latency, and token cost; see the [scorer reference](/docs/scorers.md#agentic-scorers) for all of them.
99+
100+
To try it without any cloud resources or cost:
101+
102+
```bash
103+
export EVAL_CONFIG=datasets/gemini-cli-tools/example_run_fake_config.yaml
104+
./evalbench/run.sh
105+
```
106+
107+
Read [Agentic evaluations](/docs/agentic-evals.md) for the execution model, sandboxing, and tool paradigms, and the [agentic dataset format](/docs/configs/agentic-dataset-config.md) for how to write scenarios.
108+
109+
---
110+
79111
## Overview
80112

81113
EvalBench's architecture is built around a modular design that supports diverse evaluation needs:
82114
- **Modular and Plug-and-Play:** Easily integrate custom scoring modules, data processors, and dashboard components.
83115
- **Flexible Evaluation Pipeline:** Seamlessly run DQL, DML, and DDL tasks while using a consistent base pipeline.
116+
- **Single-Turn and Agentic:** Use the same pipeline, scorers, and reporting for one-shot NL2SQL generation and for multi-turn agent journeys driven by a simulated user.
117+
- **Sandboxed Agent Execution:** Run real CLIs and MCP servers in isolated environments, in parallel, without touching your local configuration.
84118
- **Result Storage and Reporting:** Store results in various formats (e.g., CSV, BigQuery) and visualize performance with built-in dashboards.
85119
- **Customizability:** Configure and extend EvalBench to measure the performance of GenAI workflows tailored to your specific requirements.
86120

@@ -97,10 +131,21 @@ And allowing digging deeper into the exact details of the improvements and regre
97131
<img width="911" alt="Evalbench Reporting by Databaes / Dialects" src="https://github.com/user-attachments/assets/861696b5-42f1-44c7-a7d0-710f7a32918f" />
98132
<br><br>
99133

100-
A complete guide of Evalbench's available functionality can be found in [run-config documentation](/docs/configs/run-config.md)
134+
---
135+
136+
## Documentation
101137

102-
Please explore the repository to learn more about customizing your evaluation workflows, integrating new metrics, and leveraging the full potential of EvalBench.
138+
Full reference documentation lives in [docs/](/docs/README.md).
103139

140+
| Doc | Contents |
141+
|---|---|
142+
| [Run config](/docs/configs/run-config.md) | The top-level YAML that drives an evaluation run |
143+
| [Scorers](/docs/scorers.md) | Every available scorer and its configuration options |
144+
| [Agentic evaluations](/docs/agentic-evals.md) | Execution model, sandboxing, and tool paradigms |
145+
| [NL2SQL dataset format](/docs/configs/dataset-config.md) | Prompts, golden SQL, and eval queries |
146+
| [Agentic dataset format](/docs/configs/agentic-dataset-config.md) | Scenarios, conversation plans, and expected trajectories |
147+
| [Database config](/docs/configs/db-config.md) | Connection details and supported dialects |
148+
| [Model config](/docs/configs/model-config.md) | Model selection and generation settings |
149+
| [Examples](/docs/examples/) | Runnable notebooks for SQLite, Cloud SQL, and BigQuery |
104150

105-
---
106-
For additional documentation, examples, and support, please refer to the [EvalBench documentation](https://github.com/GoogleCloudPlatform/evalbench). Enjoy evaluating your GenAI models!
151+
Contributions are welcome — see [contributing](/docs/contributing.md). Enjoy evaluating your GenAI models!

‎datasets/agy-cli-tools/example_run_config.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ dataset_format: agent-format
77
# Orchestrator Configuration
88
orchestrator: agent
99
model_config: datasets/model_configs/agy_cli_model.yaml
10-
simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
10+
simulated_user_model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
1111

1212
############################################################
1313
### Scorer Related Configs
@@ -21,11 +21,11 @@ scorers:
2121
# trajectory_matcher:
2222
# filter_native_tools: false
2323
goal_completion:
24-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
24+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2525
behavioral_metrics:
26-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
26+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2727
parameter_analysis:
28-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
28+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2929
turn_count: {}
3030
end_to_end_latency: {}
3131
tool_call_latency: {}

‎datasets/agy-cli-tools/example_run_fake_config.yaml‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ dataset_format: agent-format
77
# Orchestrator Configuration
88
orchestrator: agent
99
model_config: datasets/model_configs/agy_cli_fake_model.yaml
10-
simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
10+
simulated_user_model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
1111

1212
############################################################
1313
### Scorer Related Configs
@@ -21,16 +21,15 @@ scorers:
2121
# trajectory_matcher:
2222
# filter_native_tools: false
2323
goal_completion:
24-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
24+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2525
behavioral_metrics:
26-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
26+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2727
parameter_analysis:
28-
model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
28+
model_config: datasets/model_configs/gemini_3.1_pro_model.yaml
2929
turn_count: {}
3030
end_to_end_latency: {}
3131
tool_call_latency: {}
32-
# token_consumption omitted: agy's transcript exposes no token usage, so the
33-
# scorer would only ever report 0. Re-add if agy starts emitting usage data.
32+
token_consumption: {}
3433

3534
############################################################
3635
### Reporting Related Configs

‎datasets/agy-cli-tools/example_run_skills_config.yaml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,7 @@ scorers:
2525
turn_count: {}
2626
end_to_end_latency: {}
2727
tool_call_latency: {}
28-
# token_consumption omitted: agy's transcript exposes no token usage, so the
29-
# scorer would only ever report 0. Re-add if agy starts emitting usage data.
28+
token_consumption: {}
3029

3130
############################################################
3231
### Reporting Related Configs

‎datasets/model_configs/agy_cli_model.yaml‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,23 @@
1-
# Antigravity (agy) CLI -- self-updating native binary installed via
2-
# https://antigravity.google/cli/install.sh. No npm package and no pinning
3-
# mechanism is exposed by the installer; the `agy` binary on PATH is what runs.
1+
# Antigravity (agy) CLI. See docs/agy_cli_agent_testing.md.
42

53
generator: agy_cli
64

7-
# Model to use. The harness passes this via agy's `--model` flag (agy
8-
# >=1.0.5). The value must be the exact agy UI label (NOT an API id like
9-
# "gemini-2.5-pro") -- list the valid labels with `agy models`. An
10-
# unrecognized label is silently ignored and agy falls back to its default
11-
# model. Omit this key to leave the flag off, so agy uses its own default.
12-
model: "Gemini 3.1 Pro (High)"
5+
# UI label or slug listed by `agy models`. Omit to use agy's default.
6+
model: "gemini-3.1-pro-high"
137

14-
# Timeout for print mode wait (default 5m). Passed via --print-timeout.
8+
# Print-mode timeout (agy default 5m).
159
timeout: "20m"
1610

1711
env:
1812
# Set to your project, or export EVAL_GCP_PROJECT_ID and keep the !ENV form.
19-
# Key stays GOOGLE_CLOUD_PROJECT -- the var the agy subprocess reads.
2013
GOOGLE_CLOUD_PROJECT: !ENV ${EVAL_GCP_PROJECT_ID}
2114
GOOGLE_CLOUD_LOCATION: "global"
2215
GOOGLE_GENAI_USE_VERTEXAI: "true"
2316

2417
setup:
2518
mcp_servers:
2619
"cloud-sql":
27-
# agy's native HTTP endpoint field is "serverUrl"; "url" also works
28-
# as of v1.0.5. A gemini-style "httpUrl" is auto-translated to
29-
# "serverUrl" by the harness (_translate_mcp_config), so it works
30-
# too -- but prefer "serverUrl" here. authProviderType/oauth/headers
31-
# are native agy fields, so Google auth works without Bearer-header
32-
# injection.
33-
serverUrl: "https://sqladmin.googleapis.com/mcp"
20+
httpUrl: "https://sqladmin.googleapis.com/mcp"
3421
authProviderType: google_credentials
3522
oauth:
3623
scopes:

‎datasets/model_configs/agy_cli_skills_model.yaml‎

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,8 @@
1-
# Antigravity (agy) CLI -- self-updating native binary. The `agy` command on
2-
# PATH is what runs.
3-
#
4-
# Model selection: the harness passes the model via agy's `--model` flag
5-
# (agy >=1.0.5). The value must be the exact agy UI label (e.g.
6-
# "Gemini 3.1 Pro (High)"), as listed by `agy models`. Left unset here so the
7-
# flag is omitted and agy uses its own default model.
1+
# Antigravity (agy) CLI. See docs/agy_cli_agent_testing.md.
2+
# No `model` key, so agy uses its default.
83
generator: agy_cli
94
env:
105
# Set to your project, or export EVAL_GCP_PROJECT_ID and keep the !ENV form.
11-
# Key stays GOOGLE_CLOUD_PROJECT -- the var the agy subprocess reads.
126
GOOGLE_CLOUD_PROJECT: !ENV ${EVAL_GCP_PROJECT_ID}
137
GOOGLE_CLOUD_LOCATION: "global"
148
GOOGLE_GENAI_USE_VERTEXAI: "true"
@@ -22,11 +16,8 @@ env:
2216
CLOUD_SQL_POSTGRES_PASSWORD: <password>
2317
CLOUD_SQL_POSTGRES_IP_TYPE: "PUBLIC"
2418
setup:
25-
# `skills` is named for parity with the claude_code/codex_cli harnesses.
26-
# For agy each entry installs a *plugin* (`agy plugin install`), which may
27-
# bundle both skills and its own MCP servers. To attach a standalone MCP
28-
# server not packaged in a plugin, use a separate top-level `mcp_servers`
29-
# block instead.
19+
# Each entry installs a plugin (`agy plugin install`), which may bundle
20+
# skills and its own MCP servers.
3021
skills:
3122
- action: install_from_repo
3223
path: "https://github.com/gemini-cli-extensions/cloud-sql-postgresql.git"

‎docs/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# EvalBench Documentation
2+
3+
Reference documentation for [EvalBench](/README.md). Start with the [main README](/README.md) for installation and a first run.
4+
5+
## Configuration
6+
7+
Every evaluation is driven by a run config that points at a dataset, a database or agent, a model, and a set of scorers.
8+
9+
| Doc | Contents |
10+
|---|---|
11+
| [Run config](/docs/configs/run-config.md) | The top-level YAML for an evaluation run — dataset, generation, setup/teardown, scorers, reporting. |
12+
| [NL2SQL dataset format](/docs/configs/dataset-config.md) | Evaluation items for single-turn SQL evaluations: prompts, golden SQL, eval queries. |
13+
| [Agentic dataset format](/docs/configs/agentic-dataset-config.md) | Scenario evalsets for multi-turn agent evaluations: prompts, conversation plans, expected trajectories. |
14+
| [Database config](/docs/configs/db-config.md) | Connection details per database and the list of supported dialects. |
15+
| [Model config](/docs/configs/model-config.md) | Model selection and generation settings. |
16+
17+
## Scoring
18+
19+
| Doc | Contents |
20+
|---|---|
21+
| [Scorers](/docs/scorers.md) | Full catalog of every scorer — SQL, agentic, skills, transform tooling, and custom Python scorers — with config options. |
22+
| [Judge tools](/docs/judge_tools.md) | Giving LLM-judged scorers function-calling access instead of single-shot prompting. |
23+
| [Summarizer](/docs/summarizer_documentation.md) | How run summaries are aggregated and the rationale behind the formulas. |
24+
25+
## Agentic evaluations
26+
27+
| Doc | Contents |
28+
|---|---|
29+
| [Agentic evaluations](/docs/agentic-evals.md) | Execution model, sandboxing, and tool paradigms. Start here. |
30+
| [Gemini CLI](/docs/gemini_cli_agent_testing.md) | Setup and configuration for evaluating Gemini CLI. |
31+
| [Claude Code](/docs/claude_code_agent_testing.md) | Setup and configuration for evaluating Claude Code. |
32+
| [Codex CLI](/docs/codex_cli_agent_testing.md) | Setup and configuration for evaluating Codex CLI. |
33+
| [Antigravity CLI](/docs/agy_cli_agent_testing.md) | Setup and configuration for evaluating the Antigravity (agy) CLI. |
34+
| [Data agent spec](/docs/dataagent_spec.md) | ADKDataAgent support — multi-turn database agents with clarification turns. |
35+
36+
## Examples
37+
38+
Runnable notebooks in [docs/examples/](/docs/examples/):
39+
40+
- [SQLite example](/docs/examples/sqlite_example.ipynb) — the quickest end-to-end run, no cloud resources needed.
41+
- [GCP Cloud SQL example](/docs/examples/GCP_CloudSQL_Example.ipynb)
42+
- [BigQuery hybrid example](/docs/examples/bigquery_hybrid_example.ipynb)
43+
44+
## Project
45+
46+
| Doc | Contents |
47+
|---|---|
48+
| [Contributing](/docs/contributing.md) | How to submit patches and contributions. |
49+
| [Code of conduct](/docs/code-of-conduct.md) | Community guidelines. |
50+
| [Dependency graph](/docs/architecture.md) | External dependency graph, dependency groups by purpose, and supply-chain risk surfacing. |

0 commit comments

Comments
 (0)