autoresearch

Domain-agnostic autonomous research framework. Give an AI agent a prediction problem, let it experiment overnight, wake up to results.

Forked from karpathy/autoresearch (ML training optimization) and generalized to work with any domain: sports prediction, time series, classification, regression, whatever has a measurable score.

How it works

Three files that matter:

• prepare.py — data loading + evaluation metric. Fixed. The agent cannot modify this. This is your anti-gaming firewall.
• experiment.py — the single file the agent edits. Algorithm, features, parameters — everything is fair game.
• program.md — instructions for the AI agent. Point your agent here and let it go.

Supporting files:

• config.py — project settings (metric name, direction, time budget, guard rails). Immutable during runs.
• results.tsv — experiment log (created during runs, untracked by git).
• combat_log.md — dead ends tracker (created during runs, prevents re-trying failed approaches).

Quick start

# 1. Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Configure your project
#    Edit config.py — set project name, metric, direction
#    Edit prepare.py — implement load_data() and evaluate()
#    Edit experiment.py — write your baseline algorithm

# 3. Install dependencies
uv sync

# 4. Prepare data (one-time)
uv run prepare.py

# 5. Test manually
uv run experiment.py

# 6. Let the agent loose
#    Point Claude/Codex/etc at this repo and prompt:
#    "Read program.md and let's kick off a new experiment!"

Setting up a new domain

1. Edit config.py

PROJECT_NAME = "march-madness"
METRIC_NAME = "mae"
METRIC_DIRECTION = "lower"    # lower MAE = better
TIME_BUDGET = 120             # 2 min per experiment

2. Fill in prepare.py

Implement three functions:

• prepare() — one-time data download/preprocessing
• load_data() — returns train/test data dict
• evaluate() — computes the ground truth metric

Built-in metric helpers: mae, rmse, accuracy, binary_accuracy, roc_auc, cover_rate. Or write your own.

3. Seed experiment.py

Write a dumb baseline. The agent improves from there.

4. Add domain knowledge to program.md

Fill in the "Domain knowledge" section with what you know about the problem. Add known dead ends so the agent doesn't waste time.

Design philosophy

From Karpathy's original, preserved:

• Single file to modify. The agent only touches experiment.py. Diffs stay reviewable.
• Fixed time budget. Experiments are comparable regardless of what the agent changes.
• Self-contained. No complex infrastructure. One metric, one file, one loop.

Added from tennis-xgboost-autoresearch:

• Immutable evaluation. The evaluate() function is locked. The agent can't game the metric by rewriting the scorer.
• Combat log. Failed experiments document why they failed, preventing the agent from re-trying dead ends.
• Guard rails. Metric bounds in config.py catch degenerate solutions.

New:

• Domain-agnostic. No ML-specific code. Swap data + metric + baseline for any domain.
• Built-in metric library. Common metrics (MAE, RMSE, accuracy, ROC-AUC) included — just pick one.

License

MIT