Skip to content

RESULTS.md

Latest commit

 

History

History
207 lines (168 loc) · 13.5 KB

RESULTS.md

File metadata and controls

207 lines (168 loc) · 13.5 KB
Raw

Benchmark results

Real, reproducible results from the bundled benchmark harnesses. Every number below was produced by running the scripts in this directory on the finished engine — no manual tuning, fixed random_state=0, default trainers. Where a figure is retained from an earlier run it is marked prior-cycle result.

Reproduce:

uv sync --extra tabular --extra data --extra validation
uv run python benchmarks/automl_benchmark.py     # Tier-2 (offline, no network)
uv run python benchmarks/amlb_benchmark.py        # Tier-1 (OpenML, needs network)
uv run python benchmarks/beat_baseline.py         # head-to-head vs LogisticRegression (network)
uv run python benchmarks/ablation_eval.py         # component ablation (network)
uv run python benchmarks/scientific_eval.py       # nested CV vs fixed/tuned baselines + optional FLAML (network)
ANTHROPIC_API_KEY=... uv run python benchmarks/genai_value.py   # serve-safe GenAI ablation (real LLM + credit-g)

Tier-2 — offline suite (scikit-learn built-ins)

CI-smoke datasets shipped with scikit-learn; runs in seconds to a minute, no network. AutoML(cv=3, n_trials=1) — the untuned default arm — over the default trainers (RandomForest, Linear, HistGradientBoosting; + XGBoost/LightGBM/CatBoost when installed). The greedy OOF-measured ensemble is default-on (ensemble="auto", served only when it beats the best single model) and TabPFN joins the leaderboard when it is installed and its weights are cached (≤ 1000 rows on CPU).

Dataset Task Metric CV Holdout Winner Seconds
breast_cancer binary roc_auc 0.9972 0.9897 greedy_weighted_ensemble 56.8
iris multiclass accuracy 0.9557 1.0000 tabpfn 5.6
wine multiclass accuracy 1.0000 1.0000 greedy_weighted_ensemble 10.2
diabetes regression rmse −54.74 56.64 greedy_weighted_ensemble 2.8
california_housing regression rmse −0.454 0.442 greedy_weighted_ensemble 21.6

Tier-1 — OpenML-CC18 (AMLB-style)

Real OpenML tasks with genuine categorical data (e.g. credit-g), exercising the dtype-aware preprocessing and string-target encoding. AutoML(cv=5, n_trials=1) — the untuned default arm. Direct comparisons against other AutoML frameworks are claimed only where a rival actually ran (see the FLAML arm in the scientific evaluation below).

OpenML id Dataset Metric CV Holdout Winner Seconds
31 credit-g roc_auc 0.7889 0.8235 greedy_weighted_ensemble 332.6
37 diabetes roc_auc 0.8389 0.8679 tabpfn 55.9
1464 blood-transfusion roc_auc 0.7697 0.7357 greedy_weighted_ensemble 36.4
1480 ilpd roc_auc 0.7592 0.7775 greedy_weighted_ensemble 52.1

The full AMLB (104 tasks), CC18 (72) and CTR23 (35) suites plug into the same run_amlb shape under a nightly compute budget. See docs/benchmarks.md for the three-tier strategy.

Component ablation (OpenML-CC18)

benchmarks/ablation_eval.py turns the engine on one component at a time — legacy (single untuned model) → tuned (Optuna search) → tuned+portfolio (zero-shot/warm-start seeds) → tuned+ensemble (greedy OOF ensemble + auto class-weighting) → full — on identical splits, folds and seeds. Holdout ROC-AUC, verbatim harness output:

# AutoML component ablation (holdout ROC-AUC · identical splits, folds and seeds)

| dataset | legacy | tuned | tuned+portfolio | tuned+ensemble | full |
|---|---:|---:|---:|---:|---:|
| credit-g | 0.7959 | 0.8135 | 0.8248 | 0.8343 | 0.8385 |
| diabetes | 0.8679 | 0.8679 | 0.8679 | 0.8699 | 0.8684 |
| pc4 | 0.9397 | 0.9346 | 0.9383 | 0.9369 | 0.9408 |
| blood-transfusion-service-center | 0.7349 | 0.7349 | 0.7349 | 0.7494 | 0.7494 |
| ilpd | 0.7798 | 0.7772 | 0.7772 | 0.7699 | 0.7685 |
| qsar-biodeg | 0.9409 | 0.9409 | 0.9409 | 0.9406 | 0.9386 |
| wdbc | 0.9971 | 0.9952 | 0.9952 | 0.9948 | 0.9950 |
| climate-model-simulation-crashes | 0.9472 | 0.9465 | 0.9465 | 0.9797 | 0.9871 |
| mean wall-clock | 74.6s | 235.3s | 206.1s | 222.3s | 224.1s |

## Statistics — unit of analysis is the dataset (one score each)

- **full vs legacy**: mean Δ=+0.0104, W/T/L=4/1/3, one-sided Wilcoxon p=0.2305 (n=8)
- **full vs tuned**: mean Δ=+0.0094, W/T/L=4/2/2, one-sided Wilcoxon p=0.125 (n=8)
- **full vs tuned+portfolio**: mean Δ=+0.0076, W/T/L=4/2/2, one-sided Wilcoxon p=0.125 (n=8)
- **full vs tuned+ensemble**: mean Δ=+0.0014, W/T/L=3/2/3, one-sided Wilcoxon p=0.2344 (n=8)

- Friedman p=0.7051 · mean ranks (1=best): legacy=2.75, tuned=3.56, tuned+portfolio=3.19, tuned+ensemble=2.94, full=2.56

Honest reading. The full engine earns the best mean rank (2.56 of 5). The component gains are monotone where the data has structure to exploit — credit-g climbs 0.7959 → 0.8385 as each component is added, and the imbalanced climate-model-simulation-crashes jumps +0.040 (0.9472 → 0.9871) once the ensemble and auto class-weighting engage. On saturated datasets (wdbc, qsar-biodeg) the components tie or trade fractions of a point. At n=8 datasets there is no overall statistical significance (Friedman p=0.71; all Holm-adjusted pairwise p=1) — the ablation is a directional decomposition, not a significance claim, and we state that plainly.

Beating the baseline (head-to-head)

benchmarks/beat_baseline.py pits Firefly's AutoML against a default LogisticRegression — the common single-model reference — on 5-fold cross-validated ROC-AUC. Same data, same folds, same seed: one shuffled, seeded StratifiedKFold(n_splits=5, shuffle=True, random_state=0) is built per dataset and handed to both arms, so the baseline's cross_val_score and Firefly's internal selection score on the identical splits. Firefly runs with ensemble=False, so its best_score is a single-model fold-mean — directly commensurate with the baseline's fold-mean (fold-mean vs fold-mean). The ensemble's contribution is measured separately in the component ablation and nested-CV sections, not in this like-for-like arm.

Dataset Baseline (LogReg) Firefly AutoML Δ Winner Result
credit-g 0.7913 0.8009 +0.0096 random_forest WIN
phoneme 0.8115 0.9641 +0.1526 random_forest WIN
bank-marketing 0.8988 0.9181 +0.0193 random_forest WIN
diabetes 0.8337 0.8446 +0.0109 tabpfn WIN
ilpd 0.7494 0.7624 +0.0130 random_forest WIN
blood-transfusion 0.7521 0.7532 +0.0011 tabpfn WIN

Harness summary: Firefly wins 6, ties 0, losses 0 of 6 · mean ROC-AUC Δ vs baseline = +0.0344. Firefly beats the baseline on every dataset, decisively where the data is non-linear (phoneme +0.153). The margin on blood-transfusion is a bare +0.0011 — a hair over the 0.0005 tie band, effectively a tie on this ~750-row table. Note the winners are now single models (random_forest, tabpfn) rather than the served ensemble: this arm pins ensemble=False for fold-mean-vs-fold-mean comparability.

Note on an earlier artifact. A prior version of this harness scored the baseline on sklearn's unshuffled StratifiedKFold while Firefly used its shuffled, seeded folds — different splits, violating "same folds". On blood-transfusion that manufactured a spurious −0.12 gap (unshuffled LogReg 0.88 vs shuffled ~0.75). With both arms now on the identical shuffled splitter, that artifact is gone and the baseline reads 0.7521 — matching the nested-CV LogReg below.

The comparison above reports Firefly's cross-validated selection score, which is mildly optimistically biased (a max over single models scored on the same folds). Single-model max-selection over just 5 folds of small data is noisy; the unbiased protocol — nested CV — follows.

Scientific evaluation — nested cross-validation

benchmarks/scientific_eval.py uses nested 5-fold CV: an inner CV selects the model on each outer fold's training data only, and the untouched outer fold gives the unbiased estimate. Firefly AutoML is compared against a fixed LogReg, RandomForest and XGBoost, a tuned XGBoost at the same trial budget (n_trials=20), and — when flaml is installed — FLAML at a wall-clock budget matched to Firefly's measured fit time.

Statistics use one aggregated score per dataset per system — the unit of analysis is the dataset (Demšar 2006); the previous pooled-fold Wilcoxon over 25 correlated deltas was retired as statistically invalid for correlated folds. A one-sided Wilcoxon signed-rank test runs across datasets, and Friedman + Holm handle the multi-system comparison. Verbatim harness output:

Nested 5-fold CV · ROC-AUC (mean ± std) · unbiased (selection on inner CV only) · n_trials=20 · FLAML arm on (matched wall-clock)
dataset                         LogReg      RandomForest           XGBoost   XGBoost (tuned)    Firefly AutoML             FLAML   Firefly picks
------------------------------------------------------------------------------------------------------------------------------------------------
credit-g                 0.7913±0.024     0.7902±0.030     0.7761±0.032     0.8021±0.031     0.8036±0.025     0.8021±0.016   greedy_weighted_ensemble×5
phoneme                  0.8115±0.013     0.9639±0.005     0.9573±0.003     0.9603±0.003     0.9651±0.004     0.9708±0.004   greedy_weighted_ensemble×5
diabetes                 0.8337±0.028     0.8194±0.029     0.7836±0.024     0.8317±0.028     0.8370±0.028     0.8206±0.038   greedy_weighted_ensemble×5
ilpd                     0.7494±0.032     0.7701±0.046     0.7292±0.024     0.7417±0.033     0.7588±0.030     0.7357±0.028   greedy_weighted_ensemble×5
blood-transfusion        0.7521±0.031     0.6802±0.068     0.6873±0.039     0.7278±0.049     0.7514±0.034     0.7251±0.041   greedy_weighted_ensemble×4, linear×1

Per-dataset statistics over n=5 datasets (one nested-CV mean per dataset per system)
  Firefly AutoML vs LogReg           mean Δ=+0.0356 | wins 4 / ties 1 / losses 0 | one-sided Wilcoxon (n=5) p=0.0625
  Firefly AutoML vs RandomForest     mean Δ=+0.0184 | wins 4 / ties 0 / losses 1 | one-sided Wilcoxon (n=5) p=0.09375
  Firefly AutoML vs XGBoost          mean Δ=+0.0365 | wins 5 / ties 0 / losses 0 | one-sided Wilcoxon (n=5) p=0.03125
  Firefly AutoML vs XGBoost (tuned)  mean Δ=+0.0105 | wins 5 / ties 0 / losses 0 | one-sided Wilcoxon (n=5) p=0.03125
  Firefly AutoML vs FLAML            mean Δ=+0.0123 | wins 4 / ties 0 / losses 1 | one-sided Wilcoxon (n=5) p=0.09375
  Friedman p=0.03363 · mean ranks (1=best): LogReg=3.20, RandomForest=4.00, XGBoost=5.60, XGBoost (tuned)=3.40, Firefly AutoML=1.60, FLAML=3.20

Honest reading. Firefly AutoML earns the best mean rank (1.60 of 6 systems, Friedman p=0.034). It beats the tuned XGBoost on 5/5 datasets (one-sided Wilcoxon p=0.031), the untuned XGBoost 5/5 (p=0.031), and FLAML on 4/5 at matched wall-clock — because it adapts per dataset (an ensemble on non-linear phoneme, linear where linear genuinely wins on blood-transfusion). At n=5 datasets the power is limited: the per-comparison Wilcoxon tests are one-sided and every Holm-adjusted pairwise test is not significant. We report the full per-dataset table rather than lean on a single headline.

GenAI value — serve-safe controlled ablation (real LLM)

benchmarks/genai_value.py isolates the contribution of GenAI feature engineering end to end and serve-safe (features are fit on raw training data and re-applied at predict time — no leakage). The synthetic task is a retail "high-value customer" problem whose true driver is revenue = unit_price × units — a product withheld from the model that a linear learner cannot derive. Four systems, 8 repeated train/test splits, real anthropic:claude-haiku-4-5:

System ROC-AUC (mean ± std)
linear (raw) 0.9752 ± 0.006
linear + GenAI 0.9957 ± 0.002
Firefly AutoML (raw) 0.9969 ± 0.001
Firefly AutoML + GenAI 0.9969 ± 0.001
  • GenAI lift on a linear model: +0.0205 ROC-AUC — Wilcoxon p = 0.003906 over 8 repeated splits of one dataset (a within-dataset test). The LLM proposed and the gate accepted multiplicative revenue features, rediscovering the withheld driver from the schema alone.
  • GenAI lift on Firefly AutoML: −0.0001 (≈0): the tree-based engine already approximates the interaction, so there is nothing left to add — and the gate keeps only measured train-CV wins (its no-regression guarantee is scoped to the train-CV metric it optimizes; holdout results are reported as measured).

On real data (OpenML credit-g, semantic column names; needs network) the same serve-safe pipeline lifts Firefly's AutoML from 0.8041 ± 0.034 → 0.8105 ± 0.036 ROC-AUC:

System ROC-AUC (mean ± std)
Firefly AutoML (raw) 0.8041 ± 0.034
Firefly AutoML + GenAI 0.8105 ± 0.036

Cost. Reported from the agentic metering only — never invented. On this run the harness printed:

LLM cost : metering unavailable (no tracked LLM usage — deterministic proposer or tracking disabled)

The takeaway: GenAI feature engineering is a Pareto-safe accelerator — it adds measurable, significant value where the data has structure a linear model can't reach on its own, surfaces interpretable domain features, is gated to keep only measured train-CV wins, and its cost is reported from metering rather than asserted.