From 3cd09f988dd978b3f73f664906af5977eec2391f Mon Sep 17 00:00:00 2001 From: Omkar Gaikwad Date: Sat, 9 May 2026 15:07:46 +0000 Subject: [PATCH 1/2] docs: add documentation for automated EvalBench evaluations in DEVELOPER.md --- DEVELOPER.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/DEVELOPER.md b/DEVELOPER.md index 63f2a4d..870ac40 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -47,6 +47,31 @@ be successfully installed by the Gemini CLI. The skills themselves are validated using the `skills-validate.yml` workflow. +### Automated Skill Evaluations (EvalBench) + +This repository uses the [EvalBench framework](https://github.com/GoogleCloudPlatform/evalbench) to automatically evaluate the quality, multi-turn conversational capabilities, and skill execution of the extension. + +Evaluations run automatically via Cloud Build (`cloudbuild.yaml`) on pull requests when the `ci:run-evals` or `autorelease: pending` label is applied. Because tests run against a live AlloyDB instance, credentials are securely injected by Secret Manager during CI. + +#### Understanding Evaluation Files + +All evaluation configurations and datasets are located in the [`evals/`](evals/) directory: + +* **Conversational Datasets (`*_dataset.json`):** Define test scenarios for different models (e.g., `gemini_dataset.json`, `claude_dataset.json`). Each scenario contains: + * `starting_prompt`: The initial prompt sent to the agent. + * `conversation_plan`: Instructions for the simulated user LLM to drive multi-turn interactions. + * `expected_trajectory`: The sequence of tool/skill calls expected to successfully complete the task. +* **Run Configurations (`*_run_config.yaml`):** Configure the EvalBench orchestrator, target model configs, and qualitative/performance scorers (e.g., goal completion, behavioral metrics, latency, token consumption). + +#### Maintaining and Adding Scenarios + +When adding new skills or modifying existing behavior, you should add or update corresponding scenarios in the dataset files: + +1. Open `evals/gemini_dataset.json` (and/or `evals/claude_dataset.json`). +2. Add a new scenario block with a unique `id`, a clear `starting_prompt`, a detailed `conversation_plan`, and the `expected_trajectory` of tool calls. +3. Apply the `ci:run-evals` label while creating your pull request to trigger the evaluation pipeline. +4. The evaluation pipeline runs securely via Cloud Build. A maintainer will review the internal logs and results to verify your scenarios pass successfully. + ### Other GitHub Checks * **License Header Check:** A workflow ensures all necessary files contain the From c3b8aa7e0dc3c414852327eb824fee0ee1ee08df Mon Sep 17 00:00:00 2001 From: Omkar Gaikwad Date: Sat, 9 May 2026 16:34:19 +0000 Subject: [PATCH 2/2] docs: update developer guide to reflect consolidated dataset and configuration filenames --- DEVELOPER.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/DEVELOPER.md b/DEVELOPER.md index 870ac40..179c425 100644 --- a/DEVELOPER.md +++ b/DEVELOPER.md @@ -57,17 +57,17 @@ Evaluations run automatically via Cloud Build (`cloudbuild.yaml`) on pull reques All evaluation configurations and datasets are located in the [`evals/`](evals/) directory: -* **Conversational Datasets (`*_dataset.json`):** Define test scenarios for different models (e.g., `gemini_dataset.json`, `claude_dataset.json`). Each scenario contains: +* **Conversational Dataset (`dataset.json`):** Defines test scenarios for the model. Each scenario contains: * `starting_prompt`: The initial prompt sent to the agent. * `conversation_plan`: Instructions for the simulated user LLM to drive multi-turn interactions. * `expected_trajectory`: The sequence of tool/skill calls expected to successfully complete the task. -* **Run Configurations (`*_run_config.yaml`):** Configure the EvalBench orchestrator, target model configs, and qualitative/performance scorers (e.g., goal completion, behavioral metrics, latency, token consumption). +* **Run Configuration (`run_config.yaml`):** Configures the EvalBench orchestrator, target model configs, and qualitative/performance scorers (e.g., goal completion, behavioral metrics, latency, token consumption). #### Maintaining and Adding Scenarios -When adding new skills or modifying existing behavior, you should add or update corresponding scenarios in the dataset files: +When adding new skills or modifying existing behavior, you should add or update corresponding scenarios in the dataset file: -1. Open `evals/gemini_dataset.json` (and/or `evals/claude_dataset.json`). +1. Open `evals/dataset.json`. 2. Add a new scenario block with a unique `id`, a clear `starting_prompt`, a detailed `conversation_plan`, and the `expected_trajectory` of tool calls. 3. Apply the `ci:run-evals` label while creating your pull request to trigger the evaluation pipeline. 4. The evaluation pipeline runs securely via Cloud Build. A maintainer will review the internal logs and results to verify your scenarios pass successfully.