feat: Initial development of seed eval dataset generation by chen0040 · Pull Request #139 · GoogleCloudPlatform/db-context-enrichment

chen0040 · 2026-05-23T18:54:00Z

Title: feat: Initial development of seed eval dataset generation

Description

This PR introduces the initial development of the Seed Evaluation Dataset Generation feature. It provides an autonomous, multi-agent workflow to generate, evaluate, and expand high-quality Natural Language Question (NLQ) and SQL pairs directly from a database schema.

Key Changes

Updated the dataset-generation skill
Added autoctx:generate-dataset to evals/core-cujs/dataset.json to test the new dataset generation capability in the core CUJs.

Testing

Core CUJs updated to include the new dataset generation scenario (autoctx:generate-dataset*).

) Sync with main

chen0040 · 2026-05-23T18:54:23Z

/gcbrun

gemini-code-assist

Code Review

This pull request introduces a comprehensive framework for generating and expanding seed evaluation datasets of Natural Language Question (NLQ) and SQL pairs, including a new MCP tool and crash-resilient state management. The reviewer identified several technical issues and improvement opportunities: the GenAI client requires better exception handling and alignment of retry configurations, while the core generator should replace assert statements with explicit runtime checks and correct docstring return types. Additionally, several magic numbers across the implementation should be refactored into named constants to improve maintainability.

chen0040 · 2026-05-23T19:03:36Z

/gcbrun

chen0040 · 2026-05-25T00:16:59Z

/gcbrun

chen0040 · 2026-05-25T00:52:55Z

/gcbrun

chen0040 · 2026-05-25T00:56:53Z

/gcbrun

chen0040 · 2026-05-25T03:06:35Z

/gcbrun

chen0040 · 2026-05-25T03:57:37Z

/gcbrun

chen0040 · 2026-05-25T04:42:00Z

/gcbrun

chen0040 · 2026-05-25T04:44:08Z

/gcbrun

chen0040 · 2026-05-25T05:25:49Z

/gcbrun

chen0040 · 2026-05-26T19:31:05Z

/gcbrun

chen0040 · 2026-05-27T19:13:46Z

/gcbrun

…e based on its knowledge derived from app code and business context artifacts

chen0040 · 2026-05-28T00:36:27Z

/gcbrun

chen0040 · 2026-05-28T01:03:09Z

/gcbrun

chen0040 · 2026-05-28T03:41:45Z

/gcbrun

wangauone · 2026-06-04T18:20:24Z

@@ -1,14 +1,14 @@
 [project]
 name = "google-cloud-db-context-enrichment"
-version = "0.5.0"


We don't need to bump version by ourselves. This will be handled by the release-please automatically.

wangauone · 2026-06-04T18:22:01Z

@@ -0,0 +1,32 @@
+[
+  {
+    "id": "flight_expert_001",


IIUC, this and the following golden_property_search_advanced.json are part of the eval. We might not want to put these in the root folder.

wangauone · 2026-06-04T18:22:13Z

@@ -1 +1 @@
-3.12
+3.12.13


Why this change?

wangauone · 2026-06-04T18:22:56Z

    standard_file.write_text(json.dumps(standard_content))

    # Call the tool
-    attach_context_set.fn(


I think these has been fixed in the main branch. We can do a quick merge of main to this PR?

wangauone · 2026-06-04T19:58:24Z

+    *   *Level 4 (Expert)*: Analytical operations (Window functions like `RANK()`/`ROW_NUMBER()`, Self-joins, Pivot logic, handling complex JSON/Array data types).

-The standard evaluation format is a JSON object:
+9.  **Generate Dataset Pairs**:


Checking the preview page, it seems there are some formatting issues that these are not showing as bullet points.

wangauone · 2026-06-04T22:05:14Z

+        3. Treat these reverse-translated pairs as your "Seed Pairs" for the rest of the workflow.

-7.  **Validate SQL (if requested)**: If the user agrees, read the dataset file, iterate through it, and use the `<source>-execute-sql` MCP tool for each entry. Report any failures. Overwrite the file with any corrections if the user approves them.
+5.  **Initial Save (If Seed Pairs Provided or Extracted)**: If seed pairs were provided explicitly OR extracted/translated from Query Logs, use the `generate_dataset` MCP tool to save them. You must provide the exact `output_file_path`. Pass the constructed dataset as a JSON string (`dataset_entries_json`).


How to use the tool would better be a tool description.

wangauone · 2026-06-04T22:09:44Z

 ---

-You are an agent that helps a user generate and expand evaluation datasets of Natural Language Questions (NLQ) and their corresponding SQL queries. Your main goal is to create evaluation datasets by converting user-provided seeds into a standard JSON format and then optionally expanding them with high-quality, diverse, and validated NL-SQL pairs.
+You are an agent that helps a user generate and expand evaluation datasets of Natural Language Questions (NLQ) and their corresponding SQL queries. Your main goal is to create high-fidelity evaluation datasets by converting user-provided seeds into a standard JSON format, actively validating them for logical alignment, and expanding them using tunable complexity levels.


Considering only put the high level key info in the skill when the content is getting bigger. We can put details to the skill references to reduce the context consumption here.

wangauone · 2026-06-04T22:11:40Z

+    *   **Action**: Present any discovered flaws to the user and suggest specific corrections. Overwrite the file with user-approved corrections.

-10. **Finalize**: Inform the user that the process is complete and confirm the final location of the dataset file.
+8.  **Prompt for Generation/Expansion & Complexity Tuning**: Ask the user if they want to generate new pairs. If yes, ask them to define the desired **SQL Complexity Level**:


Have we considered to separate the seed generation & dataset expansion to 2 skills?

cc: @helloeve @g-lynnzee

wangauone · 2026-06-04T22:19:25Z

+        3. Treat these reverse-translated pairs as your "Seed Pairs" for the rest of the workflow.

-7.  **Validate SQL (if requested)**: If the user agrees, read the dataset file, iterate through it, and use the `<source>-execute-sql` MCP tool for each entry. Report any failures. Overwrite the file with any corrections if the user approves them.
+5.  **Initial Save (If Seed Pairs Provided or Extracted)**: If seed pairs were provided explicitly OR extracted/translated from Query Logs, use the `generate_dataset` MCP tool to save them. You must provide the exact `output_file_path`. Pass the constructed dataset as a JSON string (`dataset_entries_json`).


I forgot why we want to do initial save here. Wondering if we can save at the end of the workflow?

wangauone · 2026-06-04T22:20:24Z

+5.  **Initial Save (If Seed Pairs Provided or Extracted)**: If seed pairs were provided explicitly OR extracted/translated from Query Logs, use the `generate_dataset` MCP tool to save them. You must provide the exact `output_file_path`. Pass the constructed dataset as a JSON string (`dataset_entries_json`).

-8.  **Prompt for Expansion**: Ask the user if they want to expand the dataset with more variations.
+6.  **Prompt for Validation (If Seed Pairs Provided)**: If a seed was saved, ask the user if they want to validate the `golden_sql` and NLQ alignment in the dataset file. Advise them that this step ensures high-quality evaluation data.


For validation, consider using the validation loop as mentioned in the best practices.

https://agentskills.io/skill-creation/best-practices#validation-loops

chen0040 added 2 commits May 14, 2026 15:54

Refactor project structure, enhance CI/CD, and update dependencies (#121

15fefdb

) Sync with main

initial development of seed eval gen

5fcf7a9

chen0040 added the ci:eval label May 23, 2026

gemini-code-assist Bot reviewed May 23, 2026

View reviewed changes

remove the GEMINI_API_KEY environment variable

d8c8b76

update ci/cd to fix the gcrun issue

6348001

chen0040 added 3 commits May 24, 2026 17:29

revert ci/cd

a845366

update the ci/cd based on main branch changes

b535616

minor update on substitutionOption

b077958

fix the missing build-extension id

4e592e1

make sure that the seed golden appears in the gcbrun output

7c472ec

chen0040 added 3 commits May 24, 2026 20:43

bug fix on ruff check and pytest

459f130

more bug fix on ruff check and pytest

f7ace07

more bug fix on ruff check and pytest

31ccff7

fix cloudbuild

26c4a24

chen0040 added 2 commits May 24, 2026 22:20

address code agent comments

64c399f

address code agent comments

6498f2a

chen0040 requested a review from wangauone May 25, 2026 05:26

debug skill not invoked

7f9ee9a

chen0040 added 4 commits May 27, 2026 11:59

pure skill implementation

ab85419

simplified mock app code

571dfce

simplified mock app code

fbc7ba9

simplified mock app code

a378686

chen0040 added the ci:eval-gemini label May 27, 2026

restructure the cuj folders and update skill to make it more selectiv…

c079ed2

…e based on its knowledge derived from app code and business context artifacts

chen0040 added 2 commits May 27, 2026 17:40

minor update to improve the skill.md

29c08ed

improve the execution guard for edge case

18fad80

separate evals for seed-eval-gen for ci/cd

627ff13

chen0040 requested a review from g-lynnzee June 1, 2026 23:04

wangauone reviewed Jun 4, 2026

View reviewed changes

wangauone force-pushed the feat/golden-dataset-generation branch from da36512 to 9c2a07b Compare July 10, 2026 21:47

		@@ -1 +1 @@
		3.12
		3.12.13

Uh oh!

Conversation

chen0040 commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Title: feat: Initial development of seed eval dataset generation

Description

Key Changes

Testing

Uh oh!

chen0040 commented May 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chen0040 commented May 23, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 25, 2026

Uh oh!

chen0040 commented May 26, 2026

Uh oh!

chen0040 commented May 27, 2026

Uh oh!

chen0040 commented May 28, 2026

Uh oh!

chen0040 commented May 28, 2026

Uh oh!

chen0040 commented May 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chen0040 commented May 23, 2026 •

edited

Loading