feat(generalizability): dev/test & k-fold splits, holdout evaluation, and generalizability metrics

plugin/skills/context-engineering-dataset-generation/SKILL.md

@@ -51,7 +51,7 @@ You must prepend this exact block to the very top of every single response you g
 *   **Goal:** Create `evalset_gen_plan.md` and get explicit user approval on the dataset requirements.
 *   **Mandatory Actions:**
     1.  Read `<skill_dir>/references/generation-plan-requirements.md`.
-    2.  **Ensure Robust Dataset Size:** Unless the user has explicitly specified a custom target, the minimum target volume for a NL2SQL dataset is **at least 50 questions**.
+    2.  **Ensure Robust Dataset Size & Stratum Quorum:** Unless the user has explicitly specified a custom target, the minimum target volume for a NL2SQL dataset is **at least 50 questions**. To support stratified dev/test splitting and holdout generalizability testing, ensure every schema subdomain (`subdomain`) contains **at least 5-10 NLQ-SQL pairs**.
     3.  **Compose and Update Plan (`evalset_gen_plan.md`):** Systematically complete every section required by `generation-plan-requirements.md`. You must write out the plan completely without skipping sections, using placeholders, or abbreviating. Place the main decisions requiring user-review at the top of the plan.
     4. **[USER APPROVAL GATE]:** STOP. You MUST halt and wait for user approval of `evalset_gen_plan.md`. **DO NOT proceed to the next phase until explicitly given permission.**
 *   **Exit Criteria:** User explicitly approved `evalset_gen_plan.md` and indicated we may proceed to the next phase.
@@ -60,14 +60,22 @@ You must prepend this exact block to the very top of every single response you g
 *   **Goal:** Create the core "Seed" dataset with execution-guided proof.
 *   **Mandatory Actions:**
     1.  Execute workflow in `<skill_dir>/references/generation-cot.md`, saving validated examples via `generate_dataset` MCP Tool to an interim dataset file `temp_golden.json`.
+    2.  **Attach Generalizability Metadata**: Ensure every generated item includes a `metadata` object with:
+        - `subdomain`: Business/schema module (e.g., `"sales"`, `"billing"`, `"inventory"`, `"crm"`)
+        - `linguistic_style`: Query style (`"canonical_synthetic"`, `"short_jargon"`, `"human_telemetry"`, `"ambiguous"`)
+        - `complexity_tier`: SQL complexity (`"tier_1_simple"`, `"tier_2_multi_join_agg"`, `"tier_3_advanced_cte_window"`)
+        - `logic_depth`: Business rule type (`"explicit_schema"`, `"implicit_business_rule"`, `"value_entity_lookup"`)
 *   **Exit Criteria:** `temp_golden.json` is created, and every single example in `temp_golden.json` satisfies `evalset_gen_plan.md`'s conditions on the initial seed dataset. 
 
 ### **PHASE 4: EXPANSION & DIVERSIFICATION**
 *   **Goal:** Increase volume and edge-case coverage to reach the approved target volume.
 *   **Mandatory Actions:**
     1.  Execute workflow in `<skill_dir>/references/dataset_expansion.md`, saving validated examples via `generate_dataset` MCP tool to an interim dataset file `temp_golden.json`.
+    2.  Maintain dimension metadata tags (`subdomain`, `linguistic_style`, `complexity_tier`, `logic_depth`) across all expanded queries.
+    3.  **Stratum Volume Verification**: Verify that every `subdomain` has at least 5 pairs generated so that stratified dev/test splitting yields sufficient training and holdout validation queries for every subdomain.
 *   **Exit Criteria:** `temp_golden.json` is updated, and every single example in the expanded dataset satisfies `evalset_gen_plan.md`'s conditions on the expanded dataset. 
 
+
 ### **PHASE 5: AUDIT & REPORTING [WAIT FOR USER APPROVAL]**
 *   **Goal:** Assess the quality and diversity of the generated dataset, and get explicit user approval on the dataset.
 *   **Mandatory Actions:**
@@ -76,8 +84,10 @@ You must prepend this exact block to the very top of every single response you g
 *   **Exit Criteria:** User explicitly approved the dataset and indicated we may proceed to the next phase.
 
 ### **PHASE 6: FINALIZATION**
-*   **Goal:** Deliver the final package and any requested subsets to the active working directory.
+*   **Goal:** Deliver the final package and any requested subsets/splits to the active working directory.
 *   **Precondition:** All required phase audit reports (environment acquisition, strategic plan, pair-level review, dataset-level review) must exist on disk.
 *   **Mandatory Actions:**
     1.  **Save Dataset:** Copy the temp dataset file `temp_golden.json` to the `output_file_path` — default to the user's current working directory. If the file already exists, verify whether we should overwrite with the user.
     2.  **Move Deliverables:** Ensure all written files (`.json`, `.md`, reports) are moved to the user's active directory if they were initially created elsewhere.
+    3.  **Dataset Setup & Splitting (Optional / On User Request):** Ask the user if they have a custom test dataset file for holdout evaluation. If provided, copy/enrich it to `autoctx/experiments/<exp>/splits/test.json` and set `splits/dev.json` to `golden.json`. Otherwise, partition `golden.json` into Stratified Dev/Test splits (80/20 by `subdomain`) under `autoctx/experiments/<exp>/splits/`. Check for under-represented subdomains (< 5 pairs).
+

plugin/skills/context-engineering-evaluate/SKILL.md

@@ -56,8 +56,15 @@ Follow these steps exactly in order:
    - Wait for the user to explicitly select an experiment folder to evaluate (or use the newly created one).
    - Once selected, explicitly record their chosen experiment name into the local `autoctx/state.md` file to act as long-term memory so you don't forget it during subsequent evaluations.
 
-2. **Parameter Collection:**
-   - **User Inputs:** Prompt the user ONLY for the `golden_dataset_path` and the `context_set_id` (if they haven't provided them already). Do NOT ask them to explain or verify database configurations.
+2. **Parameter Collection & Dataset Selection:**
+   - **User Inputs:** Prompt the user ONLY for the `golden_dataset_path` (or dataset split) and the `context_set_id` (if they haven't provided them already).
+   - **Dataset Selection & Dev/Test Split Check**:
+     - Check for available dataset files in `autoctx/experiments/<experiment_name>/`: `splits/dev.json`, `splits/test.json`, or `golden.json`.
+     - **If `splits/dev.json` exists**: Default to evaluating on `splits/dev.json` (or prompt the user if they want to score `test.json` or full `golden.json`).
+     - **If `splits/` folder does NOT exist yet**: Ask the user:
+       > *"Would you like to set up a Dev/Test split now (to prepare for hill-climbing tuning), or run a full baseline evaluation against all questions in `golden.json`?"*
+       - If they choose Dev/Test split: Ask if they have a custom test dataset file (Case A) or run automated stratified partitioning (Case B) to create `splits/dev.json` and `splits/test.json`, then evaluate on `splits/dev.json`.
+       - If they choose full baseline evaluation: Evaluate directly against `golden.json`.
    - **Interactive DB Selection:** Read the `autoctx/tools.yaml` file to list available databases to the user:
      1. Find all `kind: source` blocks with supported evaluation engines (consult the `generate_evalbench_configs` tool description for the exact list of supported types).
      2. If there is exactly one *supported* source, inform the user and auto-select it.