diff --git a/datasets/bird/example_hybrid_run_config.yaml b/datasets/bird/example_hybrid_run_config.yaml
new file mode 100644
index 00000000..35217eb4
--- /dev/null
+++ b/datasets/bird/example_hybrid_run_config.yaml
@@ -0,0 +1,44 @@
+############################################################
+### Dataset / Eval Items (Hybrid BigQuery -> SQLite Mode)
+############################################################
+# The JSON list of prompts / golden SQLs and eval attributes for the run
+dataset_config: datasets/bird/prompts.json
+
+# Database configs mapping to BigQuery (for executing generated SQL)
+database_configs:
+ - datasets/bat/db_configs/bigquery.yaml
+
+# Filter dialet to bigquery for query execution
+dialects:
+ - bigquery
+query_types:
+ - dql
+dataset_format: bird-standard-format
+
+############################################################
+### Prompt and Generation Modules
+############################################################
+# The YAML config for the model to be used for SQL generation
+model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
+# The prompt generator module id
+prompt_generator: 'SQLGenBasePromptGenerator'
+
+############################################################
+### Scorer Related Configs
+############################################################
+scorers:
+ llmrater:
+ model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
+ # Fallback to local SQLite database when golden query execution fails on BQ
+ hybrid_ground_truth: true
+ python_scorer:
+ # Isolated cross-database Execution Accuracy (XA) judge script
+ script_path: 'evalbench/scorers/judges/hybrid_xa_judge.py'
+ scorer_name: 'hybrid_cross_db'
+
+############################################################
+### Reporting Related Configs
+############################################################
+reporting:
+ csv:
+ output_directory: 'results'
diff --git a/docs/configs/dataset-config.md b/docs/configs/dataset-config.md
index 44157576..ef47da3a 100644
--- a/docs/configs/dataset-config.md
+++ b/docs/configs/dataset-config.md
@@ -24,6 +24,7 @@ Each evaluation item in the JSON file includes the following keys:
## Important Notes
- **Multiple Dialects:** Each SQL-related key (`golden_sql`, `eval_query`, `setup_sql`, `cleanup_sql`) maps dialects to their corresponding queries. This ensures that the evaluation items can be used across different database systems.
+- **Hybrid Engine Evaluations (Cross-Database Mode):** When running hybrid cross-database benchmarks (e.g. generating and executing queries on BigQuery but validating them against SQLite references), the `golden_sql` queries must map to the execution engine's dialect (e.g., `"bigquery"`), even if those reference queries themselves are written in SQLite syntax.
- **Custom Metadata:** The `other` field is optional and can contain any additional information you deem necessary for reporting or contextual purposes.
- **Structured Testing:** By defining separate SQL statements for setup, evaluation, and cleanup, this configuration supports robust testing of DDL operations.
diff --git a/docs/configs/run-config.md b/docs/configs/run-config.md
index 6ec14141..b9cf751a 100644
--- a/docs/configs/run-config.md
+++ b/docs/configs/run-config.md
@@ -77,7 +77,8 @@ The `scorers` section defines various scoring strategies to evaluate the quality
| `exact_match` | Optional | Evaluates whether the generated SQL query result exactly matches the expected (golden) query result. |
| `returned_sql` | Optional | Checks that the generated output contains valid SQL code rather than just comments. |
| `regexp_matcher` | Optional | Uses regular expressions to determine if the generated query satisfies specific patterns.
**Run Configuration Options:**
- `regexp_string_list` (required): A list of regex patterns to match against the generated query.
- `invert_results` (Optional, default: `False`): When set to true, non-matching queries score 100 and matching queries score 0.
- `match_all_patterns` (Optional, default: `False`): If true, a score of 100 is given only if all regex patterns are matched; otherwise, a match with at least one pattern suffices.
- `match_whole_query` (Optional, default: `False`): When true, forces the pattern to match the entire query rather than a substring. |
-| `llmrater` | Optional | Compares the execution results of the golden SQL query with those produced by the model. It scores 100 for concrete positive cases, such as mismatches in column names or extra columns in the generated SQL. This scorer requires its own `model_config` for proper operation. |
+| `llmrater` | Optional | Compares the execution results of the golden SQL query with those produced by the model. It scores 100 for concrete positive cases, such as mismatches in column names or extra columns in the generated SQL. This scorer requires its own `model_config` for proper operation.
**Run Configuration Options:**
- `hybrid_ground_truth` (Optional, default: `False`): When set to true, if the reference (golden) query execution fails on the target BigQuery engine, it dynamically falls back to resolve the correct reference rows from the local SQLite database file. |
+| `python_scorer` | Optional | A generic scorer that executes an external Python script in an isolated sandbox (`uv run --isolated`) to perform custom evaluation logic.
**Run Configuration Options:**
- `script_path` (Required): Path to the Python evaluation script (e.g. `evalbench/scorers/judges/hybrid_xa_judge.py`).
- `scorer_name` (Optional): A custom name for the scorer instance (e.g. `hybrid_cross_db`).
**Included Hybrid Evaluator (`hybrid_xa_judge.py`):**
When set to `evalbench/scorers/judges/hybrid_xa_judge.py`, this operates as a cross-database Execution Accuracy (XA) judge. It compares BigQuery execution results against SQLite references by applying strict cell normalization rules: (1) rounding float values to 4 decimal places, (2) sorting rows lexicographically, (3) stripping trailing `.0` string suffixes, and (4) ignoring column headers. |
| `recall_match` | Optional | Computes the precision and recall by comparing the generated and expected results, ignoring `None` and duplicate values. The default scoring mode is based on recall, where matching results are compared against the expected outputs regardless of their order. |
| `set_match` | Optional | Measures the execution accuracy by comparing the results of the golden query execution with those of the generated query, as defined by the BIRD methodology. |
| `exact_match_consistency` | Optional | Evaluates consistency across multiple trials using exact match on execution results. Multiple trials are aggregated at the prompt level using a strict "All-or-Nothing" ruleāthe prompt is deemed consistent only if ALL trial pairs are consistent. |
diff --git a/evalbench/scorers/comparator.py b/evalbench/scorers/comparator.py
index 29a3abd2..72a97936 100644
--- a/evalbench/scorers/comparator.py
+++ b/evalbench/scorers/comparator.py
@@ -34,6 +34,8 @@ def compare(
generated_execution_result: Any,
generated_eval_result: Any,
generated_error: Any,
+ database: str = "",
+ **kwargs,
) -> Tuple[float, str]:
"""Abstract method to compare two execution results.
@@ -45,6 +47,7 @@ def compare(
generated_query: The generated query.
generated_execution_result: The actual execution result, obtained by
running the generated query.
+ database: Optional database name being evaluated.
Returns:
Tuple[int, str] containing a score and an analysis of the comparison.
diff --git a/evalbench/scorers/judges/hybrid_xa_judge.py b/evalbench/scorers/judges/hybrid_xa_judge.py
new file mode 100644
index 00000000..4440ce34
--- /dev/null
+++ b/evalbench/scorers/judges/hybrid_xa_judge.py
@@ -0,0 +1,134 @@
+"""Hybrid Execution Accuracy (XA) Cross-Database Evaluator for EvalBench."""
+
+from decimal import Decimal
+import json
+import os
+import sqlite3
+import sys
+from typing import List, Optional
+
+import pandas as pd
+
+
+def get_sqlite_ground_truth(
+ query: str,
+ database: str,
+ db_dir: str = "",
+) -> list:
+ """Resolves candidate SQLite database files and executes query."""
+
+ sqlite_path = os.path.join(db_dir, f"{database}.sqlite")
+ if not os.path.exists(sqlite_path):
+ return []
+ conn = sqlite3.connect(sqlite_path)
+ try:
+ return pd.read_sql_query(query, conn).to_dict(orient="records")
+ finally:
+ conn.close()
+
+
+def compare_result_sets(df_bq: pd.DataFrame, df_sqlite: pd.DataFrame) -> bool:
+ """Compares two DataFrames ignoring column names and row order.
+
+ Normalization rules:
+ 1. Floats are rounded to 4 decimal places for cross-engine consistency.
+ 2. Rows are sorted lexicographically by string representation.
+ 3. Trailing '.0' suffixes are stripped from stringified numeric values.
+ """
+ if df_bq is None or df_sqlite is None:
+ return False
+
+ if df_bq.empty and df_sqlite.empty:
+ return True
+
+ if df_bq.empty != df_sqlite.empty:
+ return False
+
+ def normalize_df(df: pd.DataFrame) -> list[tuple]:
+ rows = []
+ for _, r in df.iterrows():
+ normalized_row = []
+ for val in r:
+ if pd.isna(val):
+ normalized_row.append(None)
+ elif isinstance(val, (int, float, Decimal)):
+ try:
+ normalized_row.append(round(float(val), 4))
+ except (ValueError, TypeError):
+ normalized_row.append(str(val))
+ else:
+ s = str(val).strip().lower()
+ if s.endswith(".0"):
+ s = s[:-2]
+ normalized_row.append(s)
+ rows.append(tuple(normalized_row))
+ rows.sort(key=lambda x: str(x))
+ return rows
+
+ try:
+ bq_rows = normalize_df(df_bq)
+ sqlite_rows = normalize_df(df_sqlite)
+ except Exception:
+ return False
+
+ if len(bq_rows) != len(sqlite_rows):
+ return False
+
+ for r_bq, r_sqlite in zip(bq_rows, sqlite_rows):
+ if len(r_bq) != len(r_sqlite):
+ return False
+ for val_bq, val_sqlite in zip(r_bq, r_sqlite):
+ if val_bq != val_sqlite:
+ return False
+
+ return True
+
+
+def main():
+ try:
+ input_data = json.load(sys.stdin)
+ database = input_data.get("database", "")
+ pred_rows = input_data.get("generated_execution_result")
+ ref_sql = input_data.get("golden_query", "")
+
+ sqlite_db_dir = input_data.get("sqlite_db_dir", "")
+ sqlite_records = get_sqlite_ground_truth(
+ ref_sql, database, sqlite_db_dir
+ )
+ df_sqlite = pd.DataFrame(sqlite_records)
+ sqlite_res_str = json.dumps(sqlite_records)
+
+ gen_err = input_data.get("generated_error")
+ if pred_rows is None or isinstance(pred_rows, str) or gen_err:
+ err_msg = gen_err or "Invalid prediction object"
+ reason = (
+ f"FAIL | BigQuery Error: {err_msg} | "
+ f"SQLite Ground Truth Result: {sqlite_res_str}"
+ )
+ print(json.dumps({"score": 0.0, "reason": reason}))
+ return
+
+ if isinstance(pred_rows, list):
+ df_bq = pd.DataFrame(pred_rows)
+ else:
+ df_bq = pd.DataFrame()
+
+ match = compare_result_sets(df_bq, df_sqlite)
+ score = 100.0 if match else 0.0
+ if match:
+ reason = f"PASS | SQLite Ground Truth Result: {sqlite_res_str}"
+ else:
+ bq_res_str = json.dumps(df_bq.to_dict(orient="records"))
+ reason = (
+ f"FAIL | BQ Prediction: {bq_res_str} vs "
+ f"SQLite Ground Truth: {sqlite_res_str}"
+ )
+ print(json.dumps({"score": score, "reason": reason}))
+
+ except Exception as e:
+ err_reason = f"FAIL: Exception in hybrid judge: {e}"
+ print(json.dumps({"score": 0.0, "reason": err_reason}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/evalbench/scorers/llmrater.py b/evalbench/scorers/llmrater.py
index 42112ca3..8be7de38 100644
--- a/evalbench/scorers/llmrater.py
+++ b/evalbench/scorers/llmrater.py
@@ -21,8 +21,9 @@
from scorers import setmatcher
import logging
-from scorers import comparator
+from scorers import comparator, sqlite_bridge
from .util import make_hashable, with_cache_execute
+from util.config import load_yaml_config
from databases.util import get_cache_client
ERROR_CATEGORIZATION_PROMPT = """
@@ -166,6 +167,15 @@ def __init__(self, config: dict, global_models):
if not self.model_config:
raise ValueError("model_config is required for LLM Rater")
self.model = get_generator(global_models, self.model_config)
+ self.hybrid_ground_truth = config.get("hybrid_ground_truth", False)
+
+ # Derive SQLite database directory from database_configs list.
+ self.sqlite_db_dir = ""
+ for db_cfg_path in config.get("database_configs", []):
+ db_cfg = load_yaml_config(db_cfg_path)
+ if db_cfg and db_cfg.get("db_type") == "sqlite":
+ self.sqlite_db_dir = db_cfg.get("database_path", "")
+ break
def _is_exact_match(
self,
@@ -234,6 +244,8 @@ def compare(
generated_execution_result: list,
generated_eval_result: str,
generated_error: str,
+ database: str = "",
+ **kwargs,
) -> Tuple[float, str]:
is_empty_results = len(golden_execution_result) == 0 and len(generated_execution_result) == 0
@@ -252,7 +264,22 @@ def compare(
return 100, "Skipped. Exact Match was found."
if golden_error:
- return 0, "Golden query failed to execute."
+ # If using hybrid judge, fetch ground truth from SQLite when BQ
+ # fails on golden query syntax (e.g. SQLite functions in reference
+ # queries).
+ if self.hybrid_ground_truth:
+ logging.info(
+ "Hybrid ground truth: BQ golden query failed, resolving "
+ "from SQLite reference. query=%s",
+ golden_query,
+ )
+ golden_execution_result = (
+ sqlite_bridge.get_sqlite_ground_truth(
+ golden_query, database, self.sqlite_db_dir
+ )
+ )
+ else:
+ return 0, "Golden query failed to execute."
if generated_error:
return 0, "Generated query failed to execute."
diff --git a/evalbench/scorers/pythonscorer.py b/evalbench/scorers/pythonscorer.py
index 3a1ea561..bc8efc17 100644
--- a/evalbench/scorers/pythonscorer.py
+++ b/evalbench/scorers/pythonscorer.py
@@ -6,6 +6,9 @@
import os
+from util.config import load_yaml_config
+
+
class PythonScorer(comparator.Comparator):
"""
A general scorer that delegates to an external Python script via `uv run`.
@@ -18,6 +21,14 @@ def __init__(self, config: dict, name: str = "python_scorer"):
if not self.script_path:
raise ValueError("script_path is required for PythonScorer")
+ # Derive SQLite database directory from database_configs list.
+ self.sqlite_db_dir = ""
+ for db_cfg_path in config.get("database_configs", []):
+ db_cfg = load_yaml_config(db_cfg_path)
+ if db_cfg and db_cfg.get("db_type") == "sqlite":
+ self.sqlite_db_dir = db_cfg.get("database_path", "")
+ break
+
def compare(
self,
nl_prompt: Any,
@@ -30,6 +41,8 @@ def compare(
generated_execution_result: Any,
generated_eval_result: Any,
generated_error: Any,
+ database: str = "",
+ **kwargs,
) -> Tuple[float, str]:
# Prepare input data
@@ -44,6 +57,8 @@ def compare(
"generated_execution_result": generated_execution_result,
"generated_eval_result": generated_eval_result,
"generated_error": generated_error,
+ "database": database,
+ "sqlite_db_dir": self.sqlite_db_dir,
}
try:
diff --git a/evalbench/scorers/score.py b/evalbench/scorers/score.py
index 7e8db5a2..f677828b 100644
--- a/evalbench/scorers/score.py
+++ b/evalbench/scorers/score.py
@@ -1,4 +1,4 @@
-"""Performs the compare operation."""
+import inspect
from scorers import comparator
from scorers import exactmatcher
@@ -50,8 +50,11 @@ def compare(
if "set_match" in scorers:
comparators.append(setmatcher.SetMatcher(scorers["set_match"]))
if "llmrater" in scorers:
- comparators.append(llmrater.LLMRater(
- scorers["llmrater"], global_models))
+ llmrater_config = scorers["llmrater"]
+ llmrater_config["database_configs"] = experiment_config.get(
+ "database_configs", []
+ )
+ comparators.append(llmrater.LLMRater(llmrater_config, global_models))
if "regexp_matcher" in scorers:
comparators.append(
generatedqueryregexpmatcher.GeneratedQueryRegexpMatcher(
@@ -154,6 +157,9 @@ def compare(
custom_name = os.path.splitext(os.path.basename(script_path))[0].strip()
if not custom_name:
custom_name = key
+ scorer_config["database_configs"] = experiment_config.get(
+ "database_configs", []
+ )
comparators.append(pythonscorer.PythonScorer(scorer_config, name=custom_name))
if "dataform_compile" in scorers:
comparators.append(
@@ -189,6 +195,15 @@ def compare(
comparison_result = comparator.ComparisonResult(comp, 0)
try:
if eval_output_item["generated_sql"] is not None:
+ # Dynamically inspect signature to only pass the 'database'
+ # parameter to comparators that explicitly support it,
+ # preventing TypeError crashes in other framework scorers.
+ compare_signature = inspect.signature(comp.compare)
+ compare_kwargs = {}
+ if "database" in compare_signature.parameters:
+ compare_kwargs["database"] = (
+ eval_output_item.get("database", "")
+ )
score, logs = comp.compare(
eval_output_item["nl_prompt"],
eval_output_item["golden_sql"],
@@ -200,6 +215,7 @@ def compare(
eval_output_item["generated_result"],
eval_output_item.get("eval_results", ""),
eval_output_item["generated_error"],
+ **compare_kwargs,
)
comparison_result.score = score
comparison_result.comparison_logs = logs
diff --git a/evalbench/scorers/sqlite_bridge.py b/evalbench/scorers/sqlite_bridge.py
new file mode 100644
index 00000000..f848d0d1
--- /dev/null
+++ b/evalbench/scorers/sqlite_bridge.py
@@ -0,0 +1,23 @@
+"""SQLite Ground Truth Resolution Adapter for EvalBench."""
+
+import os
+import sqlite3
+
+import pandas as pd
+
+
+def get_sqlite_ground_truth(
+ query: str,
+ database: str,
+ db_dir: str = "",
+) -> list:
+ """Resolves candidate SQLite database files and executes query."""
+
+ sqlite_path = os.path.join(db_dir, f"{database}.sqlite")
+ if not os.path.exists(sqlite_path):
+ return []
+ conn = sqlite3.connect(sqlite_path)
+ try:
+ return pd.read_sql_query(query, conn).to_dict(orient="records")
+ finally:
+ conn.close()
diff --git a/evalbench/test/hybrid_xa_judge_test.py b/evalbench/test/hybrid_xa_judge_test.py
new file mode 100644
index 00000000..256b87e5
--- /dev/null
+++ b/evalbench/test/hybrid_xa_judge_test.py
@@ -0,0 +1,68 @@
+from decimal import Decimal
+import io
+import json
+import unittest
+from unittest.mock import patch
+
+import pandas as pd
+
+from scorers.judges.hybrid_xa_judge import compare_result_sets, main
+
+
+class TestHybridXaJudge(unittest.TestCase):
+
+ def test_compare_result_sets_handles_decimal_vs_float(self):
+ # Decimal vs Float value cell comparison.
+ df_bq = pd.DataFrame(
+ [{"val": Decimal("10.05")}, {"val": Decimal("20.10")}], dtype=object
+ )
+ df_sqlite = pd.DataFrame([{"val": 10.05}, {"val": 20.1}], dtype=object)
+
+ self.assertTrue(compare_result_sets(df_bq, df_sqlite))
+
+ def test_compare_result_sets_ignores_column_names_and_row_order(self):
+ # Dataframe BQ: columns [a, b], rows in order [1, 'Alice'], [2, 'Bob'].
+ df_bq = pd.DataFrame([{"a": 1, "b": "Alice"}, {"a": 2, "b": "Bob"}])
+
+ # Dataframe SQLite: different columns [x, y], shuffled rows
+ # [2, 'Bob'], [1, 'Alice'].
+ df_sqlite = pd.DataFrame([{"x": 2, "y": "Bob"}, {"x": 1, "y": "Alice"}])
+
+ self.assertTrue(compare_result_sets(df_bq, df_sqlite))
+
+ def test_compare_result_sets_different_row_lengths(self):
+ df_bq = pd.DataFrame([{"a": 1}, {"a": 2}])
+ df_sqlite = pd.DataFrame([{"a": 1}])
+
+ self.assertFalse(compare_result_sets(df_bq, df_sqlite))
+
+ def test_compare_result_sets_rounding(self):
+ # Tests that floats are rounded to 4 decimal places.
+ df_bq = pd.DataFrame([{"val": 1.123456}])
+ df_sqlite = pd.DataFrame([{"val": 1.1235}])
+
+ self.assertTrue(compare_result_sets(df_bq, df_sqlite))
+
+ @patch("scorers.judges.hybrid_xa_judge.get_sqlite_ground_truth")
+ def test_hybrid_xa_judge_main_with_matching_results(self, mock_sqlite_gt):
+ input_data = {
+ "database": "mock_db",
+ "golden_query": "SELECT * FROM users",
+ "generated_execution_result": [{"id": 1, "name": "Alice"}],
+ "generated_error": None,
+ "sqlite_db_dir": "/dummy/path",
+ }
+
+ with (
+ patch("sys.stdin", io.StringIO(json.dumps(input_data))),
+ patch("sys.stdout", new_callable=io.StringIO) as mock_stdout,
+ ):
+ mock_sqlite_gt.return_value = [{"id": 1, "name": "Alice"}]
+ main()
+ out_data = json.loads(mock_stdout.getvalue().strip())
+ self.assertEqual(out_data["score"], 100.0)
+ self.assertIn("PASS", out_data["reason"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/evalbench/test/sqlite_bridge_test.py b/evalbench/test/sqlite_bridge_test.py
new file mode 100644
index 00000000..eaff18da
--- /dev/null
+++ b/evalbench/test/sqlite_bridge_test.py
@@ -0,0 +1,45 @@
+import os
+import sqlite3
+import tempfile
+import unittest
+
+from scorers.sqlite_bridge import get_sqlite_ground_truth
+
+
+class TestSqliteBridge(unittest.TestCase):
+
+ def setUp(self):
+ self.test_dir = tempfile.TemporaryDirectory()
+ self.db_dir = self.test_dir.name
+ self.database_name = "test_db"
+
+ # Create a mock sqlite database file.
+ self.db_path = os.path.join(self.db_dir, f"{self.database_name}.sqlite")
+ conn = sqlite3.connect(self.db_path)
+ cursor = conn.cursor()
+ cursor.execute("CREATE TABLE users (id INT, name TEXT)")
+ cursor.execute("INSERT INTO users VALUES (1, 'Alice'), (2, 'Bob')")
+ conn.commit()
+ conn.close()
+
+ def tearDown(self):
+ self.test_dir.cleanup()
+
+ def test_get_sqlite_ground_truth_uses_named_db(self):
+ query = "SELECT * FROM users"
+ results = get_sqlite_ground_truth(query, self.database_name, self.db_dir)
+
+ self.assertEqual(len(results), 2)
+ self.assertEqual(results[0], {"id": 1, "name": "Alice"})
+ self.assertEqual(results[1], {"id": 2, "name": "Bob"})
+
+ def test_get_sqlite_ground_truth_missing_db(self):
+ # Database that does not exist should return empty list.
+ results = get_sqlite_ground_truth(
+ "SELECT * FROM users", "nonexistent_db", self.db_dir
+ )
+ self.assertEqual(results, [])
+
+
+if __name__ == "__main__":
+ unittest.main()