feat(generalizability): dev/test & k-fold splits, holdout evaluation, and generalizability metrics

[chen0040]

This PR adds dataset splitting and generalizability tooling and updates the hill-climbing and workflow skills so all optimization is measured against a held-out test set. It introduces deterministic dev/test splits, stratified and dimension-holdout splitting, and k‑fold cross validation to support both iterative hill-climbing (with holdout verification) and cross-validated optimization. New MCP tools expose split_dataset and evaluate_generalizability (computing Dev/Holdout pass rates, Generalization Gap, Out-of-Domain Transfer Index (OOD‑TI), and Linguistic Robustness Score (LRS)). Documentation and workflow SKILLs are updated to require holdout evaluation for every hill-climb iteration. Unit tests for splitting modes, k-fold generation, and the generalizability report are included, and the package version is bumped.

Scope:

1. Dev/test split and hillclimbing with holdout reporting

• Added split_dataset (dev_test, stratified, dimension_holdout) and an MCP wrapper so workflows can create dev/test splits.
• Hill-climb flow and SKILL docs updated to require running holdout evaluation; evaluate_generalizability returns holdout scores and generalization metrics for each iteration.

2. K-fold cross validation for cross-validated optimization and analysis

• Added k_fold split mode to split_dataset and K-fold artifact creation (fold_i/dev.json + fold_i/test.json).
• Added guidance and reporting in hillclimb/workflow SKILLs for cross-validated optimization and a cross-validation report in the workspace.

[gemini-code-assist]

Code Review

This pull request introduces a robust generalizability framework for context engineering. It adds dataset splitting capabilities (supporting standard dev/test, stratified, dimension holdout, and K-fold cross-validation) and introduces generalizability metrics such as the Out-of-Domain Transfer Index (OOD-TI) and Linguistic Robustness Score (LRS). These changes are reflected in the updated skills documentation, new MCP tools (split_dataset and evaluate_generalizability), and corresponding unit tests. The review feedback highlights critical edge cases in the dataset splitting and metrics calculation logic, specifically recommending validation to prevent empty test splits in stratified mode, handling missing or empty scores files to avoid misleading metrics, and skipping empty IDs in CSV parsing.

[gemini-code-assist]

In stratified split mode, if the dataset has very few items or if all buckets contain only a single item, test_items will end up completely empty. Writing an empty list to test.json will cause downstream evaluation tools to fail or produce division-by-zero errors. We should validate that both dev_items and test_items are non-empty and return an error message if they are.

            if not dev_items or not test_items:\n                return (\n                    "Error splitting dataset: unable to create non-empty dev and test sets with stratified split. "\n                    "Ensure your dataset has enough items per stratification bucket."\n                )\n\n            dev_path = os.path.join(output_dir, "dev.json")\n            test_path = os.path.join(output_dir, "test.json")

[gemini-code-assist]

If scores.csv is missing, empty, or lacks the expected headers, _load_scores silently returns an empty dictionary {}. This leads to 0.0% pass rates and a misleading 1.00 Out-of-Domain Transfer Index (OOD-TI) with a green checkmark ✅ (High Generalizability). We should explicitly validate that both dev_scores and test_scores are non-empty and return an error message if they are.

    dev_scores = _load_scores(dev_run_folder_path)\n    test_scores = _load_scores(test_run_folder_path)\n\n    if not dev_scores:\n        return f"Error computing metrics: no valid scores found in dev run folder '{dev_run_folder_path}' (check if scores.csv exists and has 'id' and 'score' columns)."\n    if not test_scores:\n        return f"Error computing metrics: no valid scores found in test run folder '{test_run_folder_path}' (check if scores.csv exists and has 'id' and 'score' columns)."

[gemini-code-assist]

If scores.csv contains rows with missing or empty id values, they will overwrite scores[None] or scores[""], which can lead to incorrect metrics. We should skip rows where item_id is missing or empty.

            for row in reader:\n                item_id = row.get("id")\n                if not item_id:\n                    continue\n                try:\n                    score = float(row.get("score", 0))\n                    scores[item_id] = score\n                except (ValueError, TypeError):\n                    pass