skill with descriptive nl-sql generation

[chen0040]

This PR introduces a new skill that is descriptive for nl-sql seed golden pair generation. It updates the comprehensive evaluation test suite for the seed evaluation dataset generation feature.

Key Changes

1. Descriptive Skill for Golden Pair Generation (skills/autoctx-dataset-generation/SKILL.md)

   • Rewrite the skill for golden pair generation to be descriptive and contains the additional phases

2. CI/CD Pipeline Enhancement (.ci/cloudbuild.yaml)

   • Added two new parallel evaluation steps for the eval-dataset-gen test suite
   • Integrated both Gemini CLI and Claude Code orchestrators to test dataset generation
   • Updated upload and completion check logic to include the new evaluation suite

3. New Evaluation Test Suite (evals/eval-dataset-gen/)

   • Created comprehensive test suite with 7 critical user journeys (CUJs) covering diverse dataset generation scenarios:
     • Database Schema Grounding: Generate datasets from database structures directly
     • GitHub Code Grounding: Extract patterns from open-source application code
     • Web Documentation Grounding: Learn from technical documentation and API specifications
     • Local Code Analysis: Analyze provided source code to generate relevant test data
     • Cryptic Schema Resolution: Handle databases with unclear or abbreviated column names using business context
     • Multimodal Grounding: Process ER diagrams and design documents to infer schemas
     • Query Log Analysis: Extract and translate SQL from production query logs into NL-SQL pairs

4. Test Workspace Setup

   • Created isolated workspace directories for each test scenario with:
     • Pre-configured tools.yaml files with database source and execution tool definitions
     • Sample application code and data artifacts (database schemas, ER diagrams, design docs, query logs)
   • Each workspace is designed to simulate realistic user interactions and validate dataset generation quality

5. Evaluation Configuration

   • run_gemini_cli.yaml: Orchestrator configuration for Gemini CLI evaluation
   • run_claude.yaml: Orchestrator configuration for Claude Code evaluation
   • Both configurations include goal completion, turn count, latency, and token consumption scorers

6. Documentation (README.md)

   • Added detailed documentation for each CUJ explaining setup, expected outcomes, and evaluation criteria

Testing

• The evaluation suite validates the dataset generation feature across varied input sources (DB schemas, code, docs, query logs)
• Tests ensure the agent can autonomously discover database structures, understand application intent, and generate quality NL-SQL pairs

[chen0040]

/gcbrun

[gemini-code-assist]

Code Review

This pull request introduces a new evaluation dataset generation workflow, adding CI steps, configuration files, and extensive workspace setups for various grounding scenarios (such as DB schemas, query logs, and local code). It also updates the SKILL.md documentation with detailed instructions for the dataset generation skill. The review feedback highlights a critical runtime crash in hr_agent.py due to iterating over a None return value, corrupted placeholder text in the prompt instructions, a database name mismatch in dataset.json, and minor typos and grammatical errors in SKILL.md.

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun

[chen0040]

/gcbrun