feat: Support cross-database evaluation with SQLite ground truth

datasets/bird/example_hybrid_run_config.yaml

@@ -0,0 +1,44 @@
+############################################################
+### Dataset / Eval Items (Hybrid BigQuery -> SQLite Mode)
+############################################################
+# The JSON list of prompts / golden SQLs and eval attributes for the run
+dataset_config: datasets/bird/prompts.json
+
+# Database configs mapping to BigQuery (for executing generated SQL)
+database_configs:
+ - datasets/bat/db_configs/bigquery.yaml
+
+# Filter dialet to bigquery for query execution
+dialects:
+ - bigquery
+query_types:
+ - dql
+dataset_format: bird-standard-format
+
+############################################################
+### Prompt and Generation Modules
+############################################################
+# The YAML config for the model to be used for SQL generation
+model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
+# The prompt generator module id
+prompt_generator: 'SQLGenBasePromptGenerator'
+
+############################################################
+### Scorer Related Configs
+############################################################
+scorers:
+  llmrater:
+    model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
+    # Fallback to local SQLite database when golden query execution fails on BQ
+    hybrid_ground_truth: true
+  python_scorer:
+    # Isolated cross-database Execution Accuracy (XA) judge script
+    script_path: 'evalbench/scorers/judges/hybrid_xa_judge.py'
+    scorer_name: 'hybrid_cross_db'
+
+############################################################
+### Reporting Related Configs
+############################################################
+reporting:
+  csv:
+    output_directory: 'results'

docs/configs/dataset-config.md

@@ -24,6 +24,7 @@ Each evaluation item in the JSON file includes the following keys:
 ## Important Notes
 
 - **Multiple Dialects:** Each SQL-related key (`golden_sql`, `eval_query`, `setup_sql`, `cleanup_sql`) maps dialects to their corresponding queries. This ensures that the evaluation items can be used across different database systems.
+- **Hybrid Engine Evaluations (Cross-Database Mode):** When running hybrid cross-database benchmarks (e.g. generating and executing queries on BigQuery but validating them against SQLite references), the `golden_sql` queries must map to the execution engine's dialect (e.g., `"bigquery"`), even if those reference queries themselves are written in SQLite syntax.
 - **Custom Metadata:** The `other` field is optional and can contain any additional information you deem necessary for reporting or contextual purposes.
 - **Structured Testing:** By defining separate SQL statements for setup, evaluation, and cleanup, this configuration supports robust testing of DDL operations.
 

docs/configs/run-config.md

@@ -77,7 +77,8 @@ The `scorers` section defines various scoring strategies to evaluate the quality
 | `exact_match` | Optional | Evaluates whether the generated SQL query result exactly matches the expected (golden) query result. |
 | `returned_sql` | Optional | Checks that the generated output contains valid SQL code rather than just comments. |
 | `regexp_matcher` | Optional | Uses regular expressions to determine if the generated query satisfies specific patterns. <br><br>**Run Configuration Options:**<br>- `regexp_string_list` (required): A list of regex patterns to match against the generated query.<br>- `invert_results` (Optional, default: `False`): When set to true, non-matching queries score 100 and matching queries score 0.<br>- `match_all_patterns` (Optional, default: `False`): If true, a score of 100 is given only if all regex patterns are matched; otherwise, a match with at least one pattern suffices.<br>- `match_whole_query` (Optional, default: `False`): When true, forces the pattern to match the entire query rather than a substring. |
-| `llmrater` | Optional | Compares the execution results of the golden SQL query with those produced by the model. It scores 100 for concrete positive cases, such as mismatches in column names or extra columns in the generated SQL. This scorer requires its own `model_config` for proper operation. |
+| `llmrater` | Optional | Compares the execution results of the golden SQL query with those produced by the model. It scores 100 for concrete positive cases, such as mismatches in column names or extra columns in the generated SQL. This scorer requires its own `model_config` for proper operation.<br><br>**Run Configuration Options:**<br>- `hybrid_ground_truth` (Optional, default: `False`): When set to true, if the reference (golden) query execution fails on the target BigQuery engine, it dynamically falls back to resolve the correct reference rows from the local SQLite database file. |
+| `python_scorer` | Optional | A generic scorer that executes an external Python script in an isolated sandbox (`uv run --isolated`) to perform custom evaluation logic.<br><br>**Run Configuration Options:**<br>- `script_path` (Required): Path to the Python evaluation script (e.g. `evalbench/scorers/judges/hybrid_xa_judge.py`).<br>- `scorer_name` (Optional): A custom name for the scorer instance (e.g. `hybrid_cross_db`).<br><br>**Included Hybrid Evaluator (`hybrid_xa_judge.py`):**<br>When set to `evalbench/scorers/judges/hybrid_xa_judge.py`, this operates as a cross-database Execution Accuracy (XA) judge. It compares BigQuery execution results against SQLite references by applying strict cell normalization rules: (1) rounding float values to 4 decimal places, (2) sorting rows lexicographically, (3) stripping trailing `.0` string suffixes, and (4) ignoring column headers. |
 | `recall_match` | Optional | Computes the precision and recall by comparing the generated and expected results, ignoring `None` and duplicate values. The default scoring mode is based on recall, where matching results are compared against the expected outputs regardless of their order. |
 | `set_match` | Optional | Measures the execution accuracy by comparing the results of the golden query execution with those of the generated query, as defined by the BIRD methodology. |
 | `exact_match_consistency` | Optional | Evaluates consistency across multiple trials using exact match on execution results. Multiple trials are aggregated at the prompt level using a strict "All-or-Nothing" rule—the prompt is deemed consistent only if ALL trial pairs are consistent. |

evalbench/databases/bigquery.py

@@ -27,6 +27,10 @@ def __init__(self, db_config):
         self.client = bigquery.Client(project=self.project_id)
         self.tmp_users = []
 
+    def ensure_database_exists(self, database_name: str) -> None:
+        # BigQuery datasets are project-scoped; no per-database creation needed.
+        pass
+
     #####################################################
     #####################################################
     # Database Specific Execution Logic and Handling

evalbench/scorers/comparator.py

@@ -34,6 +34,8 @@ def compare(
         generated_execution_result: Any,
         generated_eval_result: Any,
         generated_error: Any,
+        database: str = "",
+        **kwargs,
     ) -> Tuple[float, str]:
         """Abstract method to compare two execution results.
 
@@ -45,6 +47,7 @@ def compare(
           generated_query: The generated query.
           generated_execution_result: The actual execution result, obtained by
             running the generated query.
+          database: Optional database name being evaluated.
 
         Returns:
           Tuple[int, str] containing a score and an analysis of the comparison.

evalbench/scorers/judges/hybrid_xa_judge.py

@@ -0,0 +1,134 @@
+"""Hybrid Execution Accuracy (XA) Cross-Database Evaluator for EvalBench."""
+
+from decimal import Decimal
+import json
+import os
+import sqlite3
+import sys
+from typing import List, Optional
+
+import pandas as pd
+
+
+def get_sqlite_ground_truth(
+    query: str,
+    database: str,
+    db_dir: str = "",
+) -> list:
+    """Resolves candidate SQLite database files and executes query."""
+
+    sqlite_path = os.path.join(db_dir, f"{database}.sqlite")
+    if not os.path.exists(sqlite_path):
+        return []
+    conn = sqlite3.connect(sqlite_path)
+    try:
+        return pd.read_sql_query(query, conn).to_dict(orient="records")
+    finally:
+        conn.close()
+
+
+def compare_result_sets(df_bq: pd.DataFrame, df_sqlite: pd.DataFrame) -> bool:
+    """Compares two DataFrames ignoring column names and row order.
+
+    Normalization rules:
+    1. Floats are rounded to 4 decimal places for cross-engine consistency.
+    2. Rows are sorted lexicographically by string representation.
+    3. Trailing '.0' suffixes are stripped from stringified numeric values.
+    """
+    if df_bq is None or df_sqlite is None:
+        return False
+
+    if df_bq.empty and df_sqlite.empty:
+        return True
+
+    if df_bq.empty != df_sqlite.empty:
+        return False
+
+    def normalize_df(df: pd.DataFrame) -> list[tuple]:
+        rows = []
+        for _, r in df.iterrows():
+            normalized_row = []
+            for val in r:
+                if pd.isna(val):
+                    normalized_row.append(None)
+                elif isinstance(val, (int, float, Decimal)):
+                    try:
+                        normalized_row.append(round(float(val), 4))
+                    except (ValueError, TypeError):
+                        normalized_row.append(str(val))
+                else:
+                    s = str(val).strip().lower()
+                    if s.endswith(".0"):
+                        s = s[:-2]
+                    normalized_row.append(s)
+            rows.append(tuple(normalized_row))
+        rows.sort(key=lambda x: str(x))
+        return rows
+
+    try:
+        bq_rows = normalize_df(df_bq)
+        sqlite_rows = normalize_df(df_sqlite)
+    except Exception:
+        return False
+
+    if len(bq_rows) != len(sqlite_rows):
+        return False
+
+    for r_bq, r_sqlite in zip(bq_rows, sqlite_rows):
+        if len(r_bq) != len(r_sqlite):
+            return False
+        for val_bq, val_sqlite in zip(r_bq, r_sqlite):
+            if val_bq != val_sqlite:
+                return False
+
+    return True
+
+
+def main():
+    try:
+        input_data = json.load(sys.stdin)
+        database = input_data.get("database", "")
+        pred_rows = input_data.get("generated_execution_result")
+        ref_sql = input_data.get("golden_query", "")
+
+        sqlite_db_dir = input_data.get("sqlite_db_dir", "")
+        sqlite_records = get_sqlite_ground_truth(
+            ref_sql, database, sqlite_db_dir
+        )
+        df_sqlite = pd.DataFrame(sqlite_records)
+        sqlite_res_str = json.dumps(sqlite_records)
+
+        gen_err = input_data.get("generated_error")
+        if pred_rows is None or isinstance(pred_rows, str) or gen_err:
+            err_msg = gen_err or "Invalid prediction object"
+            reason = (
+                f"FAIL | BigQuery Error: {err_msg} | "
+                f"SQLite Ground Truth Result: {sqlite_res_str}"
+            )
+            print(json.dumps({"score": 0.0, "reason": reason}))
+            return
+
+        if isinstance(pred_rows, list):
+            df_bq = pd.DataFrame(pred_rows)
+        else:
+            df_bq = pd.DataFrame()
+
+        match = compare_result_sets(df_bq, df_sqlite)
+        score = 100.0 if match else 0.0
+        if match:
+            reason = f"PASS | SQLite Ground Truth Result: {sqlite_res_str}"
+        else:
+            bq_res_str = json.dumps(df_bq.to_dict(orient="records"))
+            reason = (
+                f"FAIL | BQ Prediction: {bq_res_str} vs "
+                f"SQLite Ground Truth: {sqlite_res_str}"
+            )
+        print(json.dumps({"score": score, "reason": reason}))
+
+    except Exception as e:
+        err_reason = f"FAIL: Exception in hybrid judge: {e}"
+        print(json.dumps({"score": 0.0, "reason": err_reason}))
+
+
+if __name__ == "__main__":
+    main()

evalbench/scorers/llmrater.py

@@ -21,8 +21,9 @@
 from scorers import setmatcher
 import logging
 
-from scorers import comparator
+from scorers import comparator, sqlite_bridge
 from .util import make_hashable, with_cache_execute
+from util.config import load_yaml_config
 from databases.util import get_cache_client
 
 ERROR_CATEGORIZATION_PROMPT = """
@@ -166,6 +167,15 @@ def __init__(self, config: dict, global_models):
         if not self.model_config:
             raise ValueError("model_config is required for LLM Rater")
         self.model = get_generator(global_models, self.model_config)
+        self.hybrid_ground_truth = config.get("hybrid_ground_truth", False)
+
+        # Derive SQLite database directory from database_configs list.
+        self.sqlite_db_dir = ""
+        for db_cfg_path in config.get("database_configs", []):
+            db_cfg = load_yaml_config(db_cfg_path)
+            if db_cfg and db_cfg.get("db_type") == "sqlite":
+                self.sqlite_db_dir = db_cfg.get("database_path", "")
+                break
 
     def _is_exact_match(
         self,
@@ -234,6 +244,8 @@ def compare(
         generated_execution_result: list,
         generated_eval_result: str,
         generated_error: str,
+        database: str = "",
+        **kwargs,
     ) -> Tuple[float, str]:
         is_empty_results = len(golden_execution_result) == 0 and len(generated_execution_result) == 0
 
@@ -252,7 +264,22 @@ def compare(
             return 100, "Skipped. Exact Match was found."
 
         if golden_error:
-            return 0, "Golden query failed to execute."
+            # If using hybrid judge, fetch ground truth from SQLite when BQ
+            # fails on golden query syntax (e.g. SQLite functions in reference
+            # queries).
+            if self.hybrid_ground_truth:
+                logging.info(
+                    "Hybrid ground truth: BQ golden query failed, resolving "
+                    "from SQLite reference. query=%s",
+                    golden_query,
+                )
+                golden_execution_result = (
+                    sqlite_bridge.get_sqlite_ground_truth(
+                        golden_query, database, self.sqlite_db_dir
+                    )
+                )
+            else:
+                return 0, "Golden query failed to execute."
         if generated_error:
             return 0, "Generated query failed to execute."
 

evalbench/scorers/pythonscorer.py

@@ -6,6 +6,9 @@
 import os
 
 
+from util.config import load_yaml_config
+
+
 class PythonScorer(comparator.Comparator):
     """
     A general scorer that delegates to an external Python script via `uv run`.
@@ -18,6 +21,14 @@ def __init__(self, config: dict, name: str = "python_scorer"):
         if not self.script_path:
             raise ValueError("script_path is required for PythonScorer")
 
+        # Derive SQLite database directory from database_configs list.
+        self.sqlite_db_dir = ""
+        for db_cfg_path in config.get("database_configs", []):
+            db_cfg = load_yaml_config(db_cfg_path)
+            if db_cfg and db_cfg.get("db_type") == "sqlite":
+                self.sqlite_db_dir = db_cfg.get("database_path", "")
+                break
+
     def compare(
         self,
         nl_prompt: Any,
@@ -30,6 +41,8 @@ def compare(
         generated_execution_result: Any,
         generated_eval_result: Any,
         generated_error: Any,
+        database: str = "",
+        **kwargs,
     ) -> Tuple[float, str]:
 
         # Prepare input data
@@ -44,6 +57,8 @@ def compare(
             "generated_execution_result": generated_execution_result,
             "generated_eval_result": generated_eval_result,
             "generated_error": generated_error,
+            "database": database,
+            "sqlite_db_dir": self.sqlite_db_dir,
         }
 
         try:

evalbench/scorers/score.py

@@ -1,4 +1,4 @@
-"""Performs the compare operation."""
+import inspect
 
 from scorers import comparator
 from scorers import exactmatcher
@@ -50,8 +50,11 @@ def compare(
     if "set_match" in scorers:
         comparators.append(setmatcher.SetMatcher(scorers["set_match"]))
     if "llmrater" in scorers:
-        comparators.append(llmrater.LLMRater(
-            scorers["llmrater"], global_models))
+        llmrater_config = scorers["llmrater"]
+        llmrater_config["database_configs"] = experiment_config.get(
+            "database_configs", []
+        )
+        comparators.append(llmrater.LLMRater(llmrater_config, global_models))
     if "regexp_matcher" in scorers:
         comparators.append(
             generatedqueryregexpmatcher.GeneratedQueryRegexpMatcher(
@@ -154,6 +157,9 @@ def compare(
                     custom_name = os.path.splitext(os.path.basename(script_path))[0].strip()
                 if not custom_name:
                     custom_name = key
+            scorer_config["database_configs"] = experiment_config.get(
+                "database_configs", []
+            )
             comparators.append(pythonscorer.PythonScorer(scorer_config, name=custom_name))
     if "dataform_compile" in scorers:
         comparators.append(
@@ -189,6 +195,15 @@ def compare(
         comparison_result = comparator.ComparisonResult(comp, 0)
         try:
             if eval_output_item["generated_sql"] is not None:
+                # Dynamically inspect signature to only pass the 'database'
+                # parameter to comparators that explicitly support it,
+                # preventing TypeError crashes in other framework scorers.
+                compare_signature = inspect.signature(comp.compare)
+                compare_kwargs = {}
+                if "database" in compare_signature.parameters:
+                    compare_kwargs["database"] = (
+                        eval_output_item.get("database", "")
+                    )
                 score, logs = comp.compare(
                     eval_output_item["nl_prompt"],
                     eval_output_item["golden_sql"],
@@ -200,6 +215,7 @@ def compare(
                     eval_output_item["generated_result"],
                     eval_output_item.get("eval_results", ""),
                     eval_output_item["generated_error"],
+                    **compare_kwargs,
                 )
                 comparison_result.score = score
                 comparison_result.comparison_logs = logs