diff --git a/plugin/skills/context-engineering-dataset-generation/SKILL.md b/plugin/skills/context-engineering-dataset-generation/SKILL.md index 0ab79bb5..83f87a39 100644 --- a/plugin/skills/context-engineering-dataset-generation/SKILL.md +++ b/plugin/skills/context-engineering-dataset-generation/SKILL.md @@ -51,7 +51,7 @@ You must prepend this exact block to the very top of every single response you g * **Goal:** Create `evalset_gen_plan.md` and get explicit user approval on the dataset requirements. * **Mandatory Actions:** 1. Read `/references/generation-plan-requirements.md`. - 2. **Ensure Robust Dataset Size:** Unless the user has explicitly specified a custom target, the minimum target volume for a NL2SQL dataset is **at least 50 questions**. + 2. **Ensure Robust Dataset Size & Stratum Quorum:** Unless the user has explicitly specified a custom target, the minimum target volume for a NL2SQL dataset is **at least 50 questions**. To support stratified dev/test splitting and holdout generalizability testing, ensure every schema subdomain (`subdomain`) contains **at least 5-10 NLQ-SQL pairs**. 3. **Compose and Update Plan (`evalset_gen_plan.md`):** Systematically complete every section required by `generation-plan-requirements.md`. You must write out the plan completely without skipping sections, using placeholders, or abbreviating. Place the main decisions requiring user-review at the top of the plan. 4. **[USER APPROVAL GATE]:** STOP. You MUST halt and wait for user approval of `evalset_gen_plan.md`. **DO NOT proceed to the next phase until explicitly given permission.** * **Exit Criteria:** User explicitly approved `evalset_gen_plan.md` and indicated we may proceed to the next phase. @@ -60,14 +60,22 @@ You must prepend this exact block to the very top of every single response you g * **Goal:** Create the core "Seed" dataset with execution-guided proof. * **Mandatory Actions:** 1. Execute workflow in `/references/generation-cot.md`, saving validated examples via `generate_dataset` MCP Tool to an interim dataset file `temp_golden.json`. + 2. **Attach Generalizability Metadata**: Ensure every generated item includes a `metadata` object with: + - `subdomain`: Business/schema module (e.g., `"sales"`, `"billing"`, `"inventory"`, `"crm"`) + - `linguistic_style`: Query style (`"canonical_synthetic"`, `"short_jargon"`, `"human_telemetry"`, `"ambiguous"`) + - `complexity_tier`: SQL complexity (`"tier_1_simple"`, `"tier_2_multi_join_agg"`, `"tier_3_advanced_cte_window"`) + - `logic_depth`: Business rule type (`"explicit_schema"`, `"implicit_business_rule"`, `"value_entity_lookup"`) * **Exit Criteria:** `temp_golden.json` is created, and every single example in `temp_golden.json` satisfies `evalset_gen_plan.md`'s conditions on the initial seed dataset. ### **PHASE 4: EXPANSION & DIVERSIFICATION** * **Goal:** Increase volume and edge-case coverage to reach the approved target volume. * **Mandatory Actions:** 1. Execute workflow in `/references/dataset_expansion.md`, saving validated examples via `generate_dataset` MCP tool to an interim dataset file `temp_golden.json`. + 2. Maintain dimension metadata tags (`subdomain`, `linguistic_style`, `complexity_tier`, `logic_depth`) across all expanded queries. + 3. **Stratum Volume Verification**: Verify that every `subdomain` has at least 5 pairs generated so that stratified dev/test splitting yields sufficient training and holdout validation queries for every subdomain. * **Exit Criteria:** `temp_golden.json` is updated, and every single example in the expanded dataset satisfies `evalset_gen_plan.md`'s conditions on the expanded dataset. + ### **PHASE 5: AUDIT & REPORTING [WAIT FOR USER APPROVAL]** * **Goal:** Assess the quality and diversity of the generated dataset, and get explicit user approval on the dataset. * **Mandatory Actions:** @@ -76,8 +84,10 @@ You must prepend this exact block to the very top of every single response you g * **Exit Criteria:** User explicitly approved the dataset and indicated we may proceed to the next phase. ### **PHASE 6: FINALIZATION** -* **Goal:** Deliver the final package and any requested subsets to the active working directory. +* **Goal:** Deliver the final package and any requested subsets/splits to the active working directory. * **Precondition:** All required phase audit reports (environment acquisition, strategic plan, pair-level review, dataset-level review) must exist on disk. * **Mandatory Actions:** 1. **Save Dataset:** Copy the temp dataset file `temp_golden.json` to the `output_file_path` — default to the user's current working directory. If the file already exists, verify whether we should overwrite with the user. 2. **Move Deliverables:** Ensure all written files (`.json`, `.md`, reports) are moved to the user's active directory if they were initially created elsewhere. + 3. **Dataset Setup & Splitting (Optional / On User Request):** Ask the user if they have a custom test dataset file for holdout evaluation. If provided, copy/enrich it to `autoctx/experiments//splits/test.json` and set `splits/dev.json` to `golden.json`. Otherwise, partition `golden.json` into Stratified Dev/Test splits (80/20 by `subdomain`) under `autoctx/experiments//splits/`. Check for under-represented subdomains (< 5 pairs). + diff --git a/plugin/skills/context-engineering-evaluate/SKILL.md b/plugin/skills/context-engineering-evaluate/SKILL.md index 51019d42..a43b36c2 100644 --- a/plugin/skills/context-engineering-evaluate/SKILL.md +++ b/plugin/skills/context-engineering-evaluate/SKILL.md @@ -56,8 +56,15 @@ Follow these steps exactly in order: - Wait for the user to explicitly select an experiment folder to evaluate (or use the newly created one). - Once selected, explicitly record their chosen experiment name into the local `autoctx/state.md` file to act as long-term memory so you don't forget it during subsequent evaluations. -2. **Parameter Collection:** - - **User Inputs:** Prompt the user ONLY for the `golden_dataset_path` and the `context_set_id` (if they haven't provided them already). Do NOT ask them to explain or verify database configurations. +2. **Parameter Collection & Dataset Selection:** + - **User Inputs:** Prompt the user ONLY for the `golden_dataset_path` (or dataset split) and the `context_set_id` (if they haven't provided them already). + - **Dataset Selection & Dev/Test Split Check**: + - Check for available dataset files in `autoctx/experiments//`: `splits/dev.json`, `splits/test.json`, or `golden.json`. + - **If `splits/dev.json` exists**: Default to evaluating on `splits/dev.json` (or prompt the user if they want to score `test.json` or full `golden.json`). + - **If `splits/` folder does NOT exist yet**: Ask the user: + > *"Would you like to set up a Dev/Test split now (to prepare for hill-climbing tuning), or run a full baseline evaluation against all questions in `golden.json`?"* + - If they choose Dev/Test split: Ask if they have a custom test dataset file (Case A) or run automated stratified partitioning (Case B) to create `splits/dev.json` and `splits/test.json`, then evaluate on `splits/dev.json`. + - If they choose full baseline evaluation: Evaluate directly against `golden.json`. - **Interactive DB Selection:** Read the `autoctx/tools.yaml` file to list available databases to the user: 1. Find all `kind: source` blocks with supported evaluation engines (consult the `generate_evalbench_configs` tool description for the exact list of supported types). 2. If there is exactly one *supported* source, inform the user and auto-select it. diff --git a/plugin/skills/context-engineering-hillclimb/SKILL.md b/plugin/skills/context-engineering-hillclimb/SKILL.md index bcbbed15..f1af905e 100644 --- a/plugin/skills/context-engineering-hillclimb/SKILL.md +++ b/plugin/skills/context-engineering-hillclimb/SKILL.md @@ -1,146 +1,99 @@ --- name: context-engineering-hillclimb -description: Guides the agent to perform hill-climbing iterations to improve a ContextSet based on Evalbench evaluation results. +description: Guides the agent to perform generalizable hill-climbing iterations on a stratified Dev split to improve a ContextSet while measuring dev accuracy during hill climbing and evaluating holdout test set generalizability towards the end. --- > **Load the `context-engineering-workflow` skill first.** It holds the shared context this phase depends on: workspace layout, state file conventions, phase order, and safety protocol. Do not proceed with this phase without reading it. -# Phase: Optimization & Hill-Climbing +# Phase: Optimization & Generalizable Hill-Climbing ## Goal -Analyze evaluation failures to perform a Gap Analysis and apply targeted context mutations (refinements) to iteratively improve translation accuracy. - -## Workflow - -Follow these steps exactly in order: - -### 1. Setup & Loop - -1. **Validation**: - - Check if `autoctx/experiments/` directory and `autoctx/state.md` exist. If missing, warn the user that the workspace might not be initialized (suggest running the Setup & Connection phase of this skill first). - - Once an experiment is selected, verify it contains an `eval_reports/` folder. If missing, suggest running the Evaluation workflow first. -2. **Identify Experiment**: - - Read the local `autoctx/state.md` to identify the active experiment. - - If not found, ask the user to select an experiment folder from `autoctx/experiments/`. -3. **Determine Loop Version**: - - Scan the `autoctx/experiments//hillclimb/` folder for files matching `improved_context_v*.json`. - - Determine the loop version `vN` by finding the maximum `N` and using `N+1`. If the folder is empty, start at `v1`. -4. **Locate Base Context**: - - For `v1`: - - Check `autoctx/state.md` to see if a specific base context path was recorded for this experiment (e.g., during the Evaluation setup for user-provided contexts). - - If not found in `state.md`, default to the baseline generated by Bootstrap in the experiment folder. - - If still not found, ask the user for the absolute path to their base context file and record it as the Base Context in autoctx/state.md. - - For `vN` (where N > 1), the base context is `improved_context_v(N-1).json`. - - Verify the base context file exists. If missing, STOP and ask the user for the correct path. +Analyze evaluation failures on training/dev splits to perform Gap Analysis and apply targeted context mutations, while **measuring and guaranteeing generalizability** across held-out business subdomains, linguistic styles, complexity tiers, and logic depths. --- -### 2. Phase 1: Gap Analysis - -1. **Validation**: - - Determine the target evaluation run folder under `eval_reports/`. If multiple folders exist, find the most recent one by modified time. **Prefer the latest run by default**, but list other available runs as well (peeking into their `summary.csv` or `configs.csv` to show timestamps/metrics for visual context). Ask the user to confirm the selection. - - Verify that the selected `eval_reports//` contains expected files (e.g., `scores.csv`, `summary.csv`). If missing or empty, STOP and inform the user. -2. **Read Evaluation Results**: Use the `read_evaluation_result` MCP tool passing the path to `eval_reports//`. -3. **Generate Gap Analysis Report (Batched)**: - - The tool returns a summary and a batch of failure cases (default limit 10). - - Iterate through the failure cases by calling the tool with increasing `offset` (0, 10, 20, ...) until all failed queries are analyzed. - - **First Batch (offset=0)**: Initialize the report file with the `# Gap Analysis Report - vN` header and `## Summary` section, followed by the analysis of the first batch under `## Failed Queries Detail`. - - **Subsequent Batches**: Call the tool with the next offset, analyze the new failures, and **append** them to the `## Failed Queries Detail` section. - - Use the following structure for the report: - - ```markdown - # Gap Analysis Report - vN - - ## Summary - - **Total Queries**: 10 - - **Passed**: 7 - - **Failed**: 3 - - **Pass Rate**: 70% - - ## Failed Queries Detail - - ### Query 1: "How many users registered in 2023?" - - **Error Category**: `[FilterError]` - - **Expected SQL**: `SELECT count(*) FROM users WHERE year = 2023` - - **Actual SQL**: `SELECT count(*) FROM users` (Missing filter) - - **Root Cause**: The LLM did not know about the `year` column or how to filter by year for this entity. - - **Proposed Mutation**: Add a facet for "Users by Year". - - ### Query 2: "Show me top selling products" - - **Error Category**: `[OrderingError]` - - **Expected SQL**: `SELECT name FROM products ORDER BY sales DESC LIMIT 5` - - **Actual SQL**: `SELECT name FROM products LIMIT 5` - - **Root Cause**: Missing ordering instruction in context. - - **Proposed Mutation**: Update the template for "Product Sales" to include ordering. - - ### Query 3: "Get users older than 30" - - **Error Category**: `[GoldenDataError]` - - **Expected SQL**: `SELECT * FROM users WHERE age >> 30` (Syntax error `>>` in golden SQL) - - **Actual SQL**: `SELECT * FROM users WHERE age > 30` - - **Root Cause**: Invalid syntax in golden dataset. - - **Proposed Mutation**: None. Flag to user to fix the evaluation dataset. - ``` -4. **Save Report**: You **MUST** physically write the report file to `autoctx/experiments//hillclimb/gap_analysis_vN.md`. If you are processing in batches, ensure you append to this file until all failed queries are documented. Do not merely output it in chat; it must exist on the file system. -5. **Log in State Tracking**: - - Update `autoctx/state.md` to record the mapping for Loop `vN` (Base Context <-> Eval Report <-> Gap Analysis). -6. **Human-in-the-Loop Review**: - - Inform the user that the Gap Analysis report has been successfully written to disk. - - Ask the user if they want to review, make any corrections, or add manual feedback directly to the file before proceeding to Phase 2 (Context Mutation). - - Wait for user confirmation before starting Phase 2. +## Generalizability & Validation Protocol ---- - -### 3. Phase 2: Context Mutation - -Refer to [context-generation-guide](../context-generation-guide/SKILL.md) for how to edit a ContextSet. - -1. **Validation**: Verify that `gap_analysis_vN.md` exists and contains findings. Verify the base ContextSet file exists. If missing, STOP and inform the user. -2. **Analyze Gap Report & Determine Fixing Strategy**: - - Read `gap_analysis_vN.md` to identify what needs to be fixed. - - **Fixing Strategy Guidelines**: - - **Conciseness**: Try to use *less context* to cover *more scenarios*. Avoid adding redundant or hyper-specific templates for every single edge case. - - **Generalizability**: Prefer solutions that generalize well (e.g., use a `facet` for a column definition rather than a specific `template` for every query using that column). - - **Supported Types**: Support mutations for `template`, `facet`, and `value_search` types. -3. **Apply Mutations**: - - **Copy the Base Context**: Copy the base ContextSet file to the new destination: `autoctx/experiments//hillclimb/improved_context_vN.json`. - - **Generate New Items**: For any new context items identified in the fixing strategy (for "add" operations): - - **Invoke the `context-generation-guide` skill** to produce the final parameterized items. - - Provide the identified candidates to that skill. - - That skill will handle phrase extraction, parameterization, and constructing the valid JSON structure. - - **Validate New Items**: - - **Templates**: Run generated SQL examples via `-execute-sql` (use dummy values for placeholders) to verify syntax. - - **Others**: Cross-check table/column references against the schema via `-list-schemas`. - - **Apply Mutations**: Call the `mutate_context_set` MCP tool passing the **new** file path as `file_path` and mutations as `mutations_json` to mutate the context set. -3. **Log in State Tracking**: - - Update `autoctx/state.md` to include the output path of `improved_context_vN.json` for Loop `vN`. +> [!IMPORTANT] +> **Generalizability Mandate**: Optimizing context on 100% of a synthetic evaluation dataset risks "echo chamber" overfitting (creating hyper-specific templates that fail on unseen user questions). To ensure context mutations generalize: +> 1. **Stratified Initial Split**: Perform dataset splitting **ONCE at the beginning** of hill-climbing using **Stratified Splitting** (`split_dataset` with `stratify_by="subdomain"` or `"complexity_tier"`). +> 2. **Stratum Adequacy Verification**: Review stratum volume warnings returned by `split_dataset`. If any stratum has fewer than 5 pairs, notify the user so they can optionally expand under-represented subdomains using `context-engineering-dataset-generation`. +> 3. **Hill-Climbing Iterations**: All iterative Gap Analysis, Context Mutations, and progress verification evaluations are performed using **ONLY the Dev split** (`dev.json`). +> 4. **Holdout Verification Towards the End**: Towards the end of the hill-climbing process (when Dev score targets are met or iterations complete), run evaluation on the held-out **Test split** (`test.json`) using the final improved context set. Use the `evaluate_generalizability` MCP tool to calculate the **Out-of-Domain Transfer Index (OOD-TI)**, **Linguistic Robustness Score (LRS)**, and **Generalization Gap** across dimension buckets. --- -### 4. Validation & Upload Advice - -1. **Summarize Improvements**: Tell the user what was changed (e.g., added 2 facets, updated 1 template). -2. **Upload Instructions**: - - **Read Database Details**: Read `autoctx/tools.yaml` (or `db_config.yaml`) to fetch the specific project, location, and instance/cluster details for the active database. - - **Generate URL**: Call the `generate_upload_url` tool passing the extracted values to provide the direct console link to the user. - - Present the local file path to `improved_context_vN.json` and the generated console link together in a single clear message. -3. **Instruct Next Step Evaluation**: - - Instruct the user to run evaluation using the evaluating workflow on this new ContextSet to see if metrics improve. This will start Loop `N+1`. +## Stratified Hill-Climbing Workflow + +Follow these steps in order: + +### 1. Dataset Setup & Partitioning (At Start) +1. Check if `autoctx/experiments//splits/` already contains `dev.json` and `test.json`. +2. If missing, **ask the user before proceeding to dev-test split**: + > *"Do you have an existing test dataset file you would like to use to evaluate generalizability at the end of the hill-climbing process?"* + + - **Case A: User Has a Custom Test Dataset**: + - Prompt the user to provide the file path to their custom test dataset. + - Copy and enrich their test dataset to `autoctx/experiments//splits/test.json`. + - Use the generated evaluation dataset (`golden.json`) as `autoctx/experiments//splits/dev.json`. + - **Case B: User Does Not Have a Custom Test Dataset**: + - Automatically partition `golden.json` into a Stratified Dev/Test split (80% Dev / 20% Holdout Test stratified by `metadata.subdomain` or `complexity_tier`): + - Group items in `golden.json` by `metadata.subdomain` (or `"general"` if absent). + - For each subdomain bucket with $N \ge 2$ items, assign 80% (at least 1 item) to `dev.json` and remaining items to `test.json`. + - For single-item subdomains ($N = 1$), assign to `dev.json` to ensure training coverage. + - Save the items to `autoctx/experiments//splits/dev.json` and `splits/test.json`. +3. **Stratum Volume Check**: + - Count items per subdomain bucket across Dev and Test sets. + - If any subdomain has fewer than 5 evaluation pairs, inform the user: + > *"Note: Subdomains [list] have fewer than 5 evaluation pairs. Holdout test evaluation will proceed, but expanding pairs for these subdomains via `context-engineering-dataset-generation` is recommended for robust test coverage."* +4. Confirm that `dev.json` and `test.json` are materialized on disk so that the workspace folder structure remains identical in either case. + +### 2. Hill-Climbing Iteration Loop (Dev Set Only) +For each iteration ($v1, v2, \dots, vN$): + +1. **Determine Loop Version**: Scan `autoctx/experiments//hillclimb/` for `improved_context_v*.json` (start at `v1` if empty). +2. **Locate Base Context**: For `v1`, default to `bootstrap_context.json`. For `vN` ($N > 1$), use `improved_context_v(N-1).json`. +3. **Gap Analysis (Dev Set)**: + - Use `read_evaluation_result` on the Dev set evaluation run folder. + - Analyze failure cases in batches (offset 0, 10, 20...). Categorize errors (`FilterError`, `OrderingError`, `SchemaError`, `GoldenDataError`). + - Propose mutations emphasizing **generalizability** (prefer `facet` over narrow `template` where possible). + - Write report to `autoctx/experiments//hillclimb/gap_analysis_vN.md`. +4. **Context Mutation**: + - Copy base context to `autoctx/experiments//hillclimb/improved_context_vN.json`. + - Generate required new items following [context-generation-guide](../context-generation-guide/SKILL.md). + - Validate SQL syntax via `-execute-sql`. + - Call `mutate_context_set` MCP tool to apply mutations. +5. **Dev Set Verification Evaluation**: + - Execute evaluation on `splits/dev.json` using `improved_context_vN.json`. + - Verify Dev score improvement. Repeat the hill-climbing loop as needed on `dev.json`. + +### 3. Test Split Evaluation & Generalizability Verification (Towards the End) +Towards the end of the hill-climbing process (when Dev performance target is met or iterations conclude): + +1. **Execute Holdout Test Evaluation**: Run evaluation on `splits/test.json` using the final `improved_context_vN.json`. +2. **Calculate Generalizability Metrics**: + - Compare final Dev evaluation scores vs Holdout Test evaluation scores. + - Calculate **Dev Pass Rate (%)** vs **Holdout Test Pass Rate (%)**. + - Calculate **Generalization Gap**: $\text{Dev Score} - \text{Holdout Score}$. + - Calculate **Out-of-Domain Transfer Index (OOD-TI)**: $\text{Test Score} / \text{Dev Score}$. + - Calculate **Linguistic Robustness Score (LRS)**: Jargon query test pass rate vs canonical dev pass rate. +3. **Present Generalizability Metrics Report**: + Present the report containing: + - **Dev Pass Rate (%)** vs **Holdout Test Pass Rate (%)** + - **Generalization Gap** & **Out-of-Domain Transfer Index (OOD-TI)** + - **Linguistic Robustness Score (LRS)** + - **Dimension Breakdown Matrix** across subdomains, linguistic styles, and complexity tiers. --- -## Output +## Validation & Upload Advice -Upon successful completion, the workspace must contain: -- `autoctx/experiments//hillclimb/gap_analysis_vN.md` -- `autoctx/experiments//hillclimb/improved_context_vN.json` -- Updated `autoctx/state.md` summarizing the run loop. +1. Summarize improvements and report Dev Score, Holdout Score, OOD-TI, and LRS. +2. Provide upload link via `generate_upload_url`. --- -## Logging State Example (`autoctx/state.md`) - -When updating `autoctx/state.md`, please append or update the `Hill-Climbing Run Log` section: +## State Logging Example (`autoctx/state.md`) ```markdown # Context Authoring Experiment State Tracking @@ -149,13 +102,18 @@ When updating `autoctx/state.md`, please append or update the `Hill-Climbing Run ## Hill-Climbing Run Log -### Loop: v1 -- **Base Context**: `baseline_context.json` -- **Eval Report Path**: `autoctx/experiments/my-exp-1/eval_reports//` (containing `configs.csv`, `evals.csv`, etc.) +### Loop: v1 (Stratified Mode) +- **Split Mode**: Stratified by Subdomain (80/20) +- **Dev Set**: `autoctx/experiments/my-exp-1/splits/dev.json` (40 items) +- **Holdout Test Set**: `autoctx/experiments/my-exp-1/splits/test.json` (10 items) +- **Dev Pass Rate**: 90.0% +- **Holdout Test Pass Rate**: 82.5% (Verified towards end of hillclimbing) +- **Generalization Gap**: +7.5% +- **Out-of-Domain Transfer Index (OOD-TI)**: 0.91 ✅ +- **Linguistic Robustness Score (LRS)**: 0.88 ✅ - **Gap Analysis**: `autoctx/experiments/my-exp-1/hillclimb/gap_analysis_v1.md` - **Mutated Context**: `autoctx/experiments/my-exp-1/hillclimb/improved_context_v1.json` ``` - > [!IMPORTANT] -> **Tool Modification Rule**: Always use the `mutate_context_set` tool for all ContextSet changes. Pass mutation payloads directly to the tool — it handles all file I/O internally. **Do not read the target context set file beforehand**. +> **Tool Modification Rule**: Always use the `mutate_context_set` tool for all ContextSet changes. diff --git a/plugin/skills/context-engineering-workflow/SKILL.md b/plugin/skills/context-engineering-workflow/SKILL.md index 086e8e22..5506606c 100644 --- a/plugin/skills/context-engineering-workflow/SKILL.md +++ b/plugin/skills/context-engineering-workflow/SKILL.md @@ -1,6 +1,6 @@ --- name: context-engineering-workflow -description: Context engineering for Gemini Data Analytics API's data agent developer platform tools. Generates, evaluates, and iteratively optimizes a ContextSet (Templates, Facets, Value Searches) to maximize Natural-Language-to-SQL translation accuracy. Use this skill to run the automated setup, NL-SQL pair evaluation dataset generation and expansion, bootstrapping, scoring, and optimization pipeline. For manual authoring standards and schema syntax rules, see the context-generation-guide skill. +description: Context engineering for Gemini Data Analytics API's data agent developer platform tools. Generates, evaluates, and iteratively optimizes a ContextSet (Templates, Facets, Value Searches) to maximize Natural-Language-to-SQL translation accuracy. Use this skill to run the automated setup, NL-SQL pair evaluation dataset generation and expansion, bootstrapping, scoring, dev/test dataset splitting, and generalizable optimization pipeline. For manual authoring standards and schema syntax rules, see the context-generation-guide skill. --- # Skill: Context Engineering Orchestrator @@ -13,7 +13,7 @@ Refer to [context-generation-guide/SKILL.md](../context-generation-guide/SKILL.m ## The Optimization Lifecycle & Phase Flow -To build high-performing data applications, context engineers typically follow a systematic, iterative optimization lifecycle (Hill-Climbing). +To build high-performing data applications, context engineers follow a systematic, generalizable optimization lifecycle (Hill-Climbing with Dev/Test Holdout Validation). ```mermaid flowchart TD @@ -23,19 +23,24 @@ Scaffolds workspace & connections] Prep -- No --> DatasetPrep[Dataset Prep & Expansion Builds reference ground-truth] - DatasetPrep --> Bootstrap[Baseline Context Bootstrapping + DatasetPrep --> Split[Initial Dataset Splitting +Dev/Test Split at start] + Prep -- Yes --> Split + + Split --> Bootstrap[Baseline Context Bootstrapping Generates initial context from schema] - Prep -- Yes --> Bootstrap - Bootstrap --> Evaluate[Evaluation Scoring -Scores context using Evalbench] + Bootstrap --> Evaluate[Dev Set Evaluation Scoring +Scores context on Dev split] Evaluate --> Loop{Tuning Target Met?} - Loop -- No --> Hillclimb[Optimization & Hill-Climbing -Gap Analysis & Context Mutation] + Loop -- No --> Hillclimb[Hill-Climbing Iteration +Gap Analysis & Context Mutation on Dev] Hillclimb --> Evaluate - Loop -- Yes --> End([End - Context Deployed!]) + Loop -- Yes --> Holdout[Holdout Test Set Evaluation +Verifies generalizability on Test split] + Holdout --> End([End - Context Deployed with Holdout Verification!]) ``` --- @@ -47,7 +52,7 @@ Gap Analysis & Context Mutation] ### Setup & Connection Configuration Phase * **Reference**: [context-engineering-init](../context-engineering-init/SKILL.md) * **Goal**: Scaffold the local `autoctx/` workspace and establish verified database connections. -* **Rationale**: Readonly-database access is an input for evaluation dataset prep and expand, baseline context bootstrapping, +* **Rationale**: Readonly-database access is required for dataset prep, bootstrapping, and evaluation. * **Entry Prerequisites**: * *None*. @@ -56,42 +61,48 @@ Gap Analysis & Context Mutation] ### Evaluation Dataset Prep & Expansion Phase * **Reference**: [context-engineering-dataset-generation](../context-engineering-dataset-generation/SKILL.md) * **Mandatory Deliverables**: `evalset_environment_inputs.md`, `evalset_gen_plan.md`, `evalset_report_pair_level.md`, and `evalset_report_dataset_level.md`. -* **Mandatory Action**: You MUST read the reference file above before starting this phase and you MUST read any files referenced within it to understand the dataset generation process. * **Goal**: Build a high-quality "golden" ground-truth dataset and associated audit reports. -* **Rationale**: A representative ground-truth dataset and formal audit trails are required to objectively measure and verify translation accuracy improvements. +* **Rationale**: A representative ground-truth dataset and formal audit trails are required to objectively measure translation accuracy. * **Entry Prerequisites**: - * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. + * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed (`autoctx/tools.yaml` active). --- ### Baseline Context Bootstrapping Phase * **Reference**: [context-engineering-bootstrap](../context-engineering-bootstrap/SKILL.md) -* **Goal**: Deduce query concepts and generate a baseline `ContextSet` (templates, facets, value searches) directly from database schemas and metadata. +* **Goal**: Deduce query concepts and generate a baseline `ContextSet` (templates, facets, value searches) directly from database schemas. * **Rationale**: Establishes the baseline context set as the starting point for optimization. * **Entry Prerequisites**: - * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. + * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed (`autoctx/tools.yaml` active). --- ### Run Evaluation And Score * **Reference**: [context-engineering-evaluate](../context-engineering-evaluate/SKILL.md) -* **Goal**: Run a structured Evalbench evaluation to score the accuracy of a specific context set and identify exact query failures. +* **Goal**: Run a structured Evalbench evaluation to score the accuracy of a context set against a golden dataset (or dev/test split) and identify exact query failures. * **Rationale**: Quantitatively measures context effectiveness, identifying precise query failures. * **Entry Prerequisites**: - * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. - * [ ] **Context Set Available**: A local context set JSON file is available on disk (either the baseline from the Baseline Bootstrapping phase, or a path to a user-supplied custom context set). - * [ ] **Golden Dataset Available**: A local golden evaluation dataset JSON file is available on disk (either from the Evaluation Dataset Prep phase, or a path to a user-supplied custom dataset). - * [ ] **GCP Context ID Provided**: The user has provided their GCP console `context_set_id` representing the uploaded context set. + * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed (`autoctx/tools.yaml` active). + * [ ] **Context Set Available**: A local context set JSON file is available on disk. + * [ ] **Golden Dataset / Split Available**: A dataset or split file (`dev.json`, `test.json`) is available on disk. + * [ ] **GCP Context ID Provided**: The user has provided their GCP console `context_set_id`. --- -### Optimization & Hill-Climbing Phase +### Generalizable Optimization & Hill-Climbing Phase * **Reference**: [context-engineering-hillclimb](../context-engineering-hillclimb/SKILL.md) -* **Goal**: Analyze evaluation failures to perform a Gap Analysis and apply targeted context mutations to iteratively improve performance. -* **Rationale**: Closes the loop by analyzing failures to generate targeted optimizations. +* **Goal**: Perform generalizable hill-climbing using Stratified Dev/Test splits (stratified by `subdomain` or `complexity_tier`). Split the dataset once at the start using stratified splitting, perform iterative Gap Analysis and mutations on the Dev set, and evaluate on the Holdout Test set towards the end of hill-climbing to calculate Generalization Gap metrics. +* **Key Generalizability Metrics**: + - **Holdout Test Pass Rate (%)**: Pass rate on unseen test queries evaluated towards the end. + - **Generalization Gap ($\Delta_{\text{gen}}$)**: $\text{Dev Score} - \text{Holdout Score}$. + - **Out-of-Domain Transfer Index (OOD-TI)**: $\text{Holdout Score on Unseen Subdomains} / \text{Dev Score}$. Measures transferability across business modules ($\ge 0.90$ target). + - **Linguistic Robustness Score (LRS)**: $\text{Holdout Score on Jargon Queries} / \text{Dev Score on Canonical Queries}$. Measures resilience against phrasing/jargon changes. +* **Rationale**: Prevents overfitting / "echo chambering" to synthetic training queries and guarantees context improvements generalize to unseen user questions. **All hillclimbing MUST evaluate on the Holdout Test split towards the end and report generalizability metrics.** * **Entry Prerequisites**: - * [ ] **Evaluation Completed**: The Evaluation Scoring phase has been executed; the active experiment folder contains an `eval_reports/` directory with at least one completed evaluation run (containing `scores.csv` and `summary.csv`). - * [ ] **Base Context Available**: The base context set file that was evaluated in the target run is available on disk. + * [ ] **Stratified Dataset Split Available**: Dataset is partitioned into stratified dev/test splits (`splits/dev.json` and `splits/test.json`). + * [ ] **Evaluation Completed**: Evaluation scoring executed on the Dev set. + * [ ] **Base Context Available**: Base context set file is available on disk. + --- @@ -106,6 +117,9 @@ The Autoctx workflows generate and interact with a structured workspace to maint * `experiments/`: Root directory for all experiments. * `/`: Specific experiment directory. * `bootstrap_context.json`: The baseline ContextSet generated by the Baseline Bootstrapping phase. + * `splits/`: Directory containing dataset splits for generalizability testing. + * `dev.json`: Training/Dev split dataset. + * `test.json`: Holdout Test split dataset. * `eval_configs/`: Directory containing Evalbench configurations. * `eval_reports/`: Directory containing evaluation output runs. * `hillclimb/`: Directory containing hill-climbing iteration artifacts. @@ -113,27 +127,23 @@ The Autoctx workflows generate and interact with a structured workspace to maint * `improved_context_vN.json`: The mutated ContextSet at iteration `N`. ### Workspace Evolution Lifecycle -1. **Post-Initialization**: `tools.yaml`, `state.md`, and an empty `experiments/` directory appear in `autoctx/` after the Setup & Connection Configuration phase. -2. **Post-Bootstrap**: `autoctx/experiments//bootstrap_context.json` is generated by the Baseline Bootstrapping phase. -3. **Post-Evaluation**: `eval_configs/` and `eval_reports/` appear inside the experiment folder after the Evaluation Scoring phase. -4. **Post-Hill-Climbing**: `hillclimb/` appears with `gap_analysis_vN.md` and `improved_context_vN.json` after the Optimization & Hill-Climbing phase, and `state.md` is updated. -5. **Tuning Loop**: Iteratively evaluates `improved_context_vN.json` and generates `improved_context_v(N+1).json` until target accuracy is achieved. +1. **Post-Initialization**: `tools.yaml`, `state.md`, and an empty `experiments/` directory appear in `autoctx/`. +2. **Post-Dataset Generation & Splitting**: Golden dataset and `splits/` (`dev.json`, `test.json`) are created. +3. **Post-Bootstrap**: `bootstrap_context.json` is generated by Baseline Bootstrapping. +4. **Post-Evaluation**: `eval_configs/` and `eval_reports/` appear after running evaluation. +5. **Post-Hill-Climbing**: `hillclimb/` appears with `gap_analysis_vN.md`, `improved_context_vN.json`, Holdout Test scores (towards the end), and updated `state.md`. + +--- ## Safety & Protocol * **Missing Dataset**: - * If the user's request requires **evaluating, scoring, or optimizing** a context set (e.g., running evaluations, tuning, or hill-climbing): + * If the user's request requires **evaluating, scoring, or optimizing** a context set: * Validate if an evaluation dataset exists. - * **Mandatory Halt & Guide**: If no evaluation dataset exists, you are **strictly forbidden** from executing any context bootstrapping, tuning, or evaluation operations in this turn. You must immediately halt, stop calling tools, and yield the turn. Explain **why a golden evaluation dataset is critical** for context engineering (i.e., you cannot objectively score, validate, or hill-climb translation accuracy without a ground-truth dataset), and ask if they would like help generating one first. + * **Mandatory Halt & Guide**: If no evaluation dataset exists, you are **strictly forbidden** from executing any context bootstrapping, tuning, or evaluation operations in this turn. Stop, explain why a golden evaluation dataset is critical for context engineering, and offer to help generate one first. + +* **Holdout Evaluation Requirement**: + * All hillclimbing workflows **MUST evaluate on the holdout test set towards the end and report the holdout score**. Never declare optimization success based solely on training/dev set scores. * **Critical API Error Protocol**: - * Seek guidance from the user if you run into results where retrying is unlikely to solve the issue. - * Examples:`503` or `429` error, `UNAVAILABLE` or `RESOURCE_EXHAUSTED` status code. - * Why: These errors are often associated with quota issues, and retrying the request immediately will not resolve the issue. For issues related to Vertex AI Resource Exhaustion, retrying at a later time is often the only solution. - -* **Skill Prerequisites & Troubleshooting**: - * If you encounter any environmental, connection, or execution errors, verify that all preconditions are met. Types of preconditions: - * *Google Cloud Service APIs enablement* - * *IAM Roles & Access* - * *Database Instance Permissions* - * *Development Environment*: Application Default Credentials (ADC) and Python package manager (`uv`) \ No newline at end of file + * Seek guidance from the user if you encounter non-recoverable errors (e.g. `503`, `429`, `RESOURCE_EXHAUSTED`). \ No newline at end of file