Extrapolaw is a tool for scaling studies of deep learning pipelines: it searches how architectural and hyperparameter choices behave as you scale models, data, and compute, and fits scaling laws to predict the best choices at larger scales. Use it from a single GPU to a multi-node cluster or even multiple clusters.
Extrapolaw unifies established and novel scaling prescriptions within the ScAn framework and pairs them with a runtime tailored to large-scale models. Extrapolaw is actively maintained and used to run on many different clusters, tuning even billion-parameter scale models with many concurrent trials.
To learn about Extrapolaw, check out the documentation or our examples.
- Search on a budget, predict at scale: scaling laws are integrated directly into the optimization loop, so that results obtained at small scale inform predictions of optimal configurations at larger scales.
- Cost-constrained sampling: optimizers restrict sampling to the feasible region of the search space, i.e., configurations that satisfy a user-specified black-box cost constraint.
- Predicting the optimal configuration at scale: Extrapolaw answers the central question of any scaling study, namely which configuration to train at a target scale where a direct search is prohibitively expensive.
- Tuning distributed models: Extrapolaw works with DDP and FSDP, on a single node or across multiple nodes, out of the box (examples).
- Zero-effort to run many concurrent models: start more workers on the same machine or in a multi-node setup. As long as they share the results directory, they coordinate on their own, with no server to set up.
- Live monitoring and interventions: follow a run with live plots or TensorBoard, and steer it without starting over: add workers, extend the budget, re-run failed trials, or import tuning results from anywhere (even cross-cluster).
- Grounded prescriptions: established prescriptions from the literature, namely Chinchilla (Hoffmann et al., 2022), Kaplan (Kaplan et al., 2020), and CARBS (Fetterman et al., 2023), adapted to budget-aware search and complemented by our novel algorithm, ScAn-BO.
- Model-based search: strategies such as Bayesian optimization choose promising configurations smartly instead of sampling statically (e.g., Chinchilla) or at random.
- Any design space: hyperparameters, architectures, resource allocation, or any component of the pipeline.
- Any scaling dimension: scale model size, dataset size, or any other quantity that sets the compute (
is_compute_scaling=True). - Downstream tasks: follow how downstream performances scale next to the pre-training loss.
- Training or inference cost: predictions can be made with respect to either training or inference compute.
Extrapolaw supports Python 3.11 to 3.14. Install the latest release from PyPI:
pip install extrapolawUsing extrapolaw is based on the following pattern:
- Define an
evaluate_pipelinefunction that submits a training job for one configuration; the job reports its result back withextrapolaw.save_pipeline_results(). - Define a
pipeline_spaceof the parameters to search, marking those that scale the compute withis_compute_scaling=True. - Call
extrapolaw.acquire()with acost_estimatorand a cost budget to acquire data under a cost constraint. - Call
extrapolaw.predict()to extrapolate the results to the target scale(s).
In code, the usage pattern can look like this:
import subprocess
import extrapolaw
ROOT_DIRECTORY = "path/to/save/results" # Replace with the actual path
# 1. Submit one configuration as a multi-GPU job and return immediately (async)
def evaluate_pipeline(pipeline_id: str, **config): # Extrapolaw injects `pipeline_id`
args = " ".join(f"--{name} {value}" for name, value in config.items())
subprocess.run(
f"sbatch --gpus=8 --wrap='torchrun --nproc_per_node=8 train.py "
f"--pipeline_id {pipeline_id} --root_directory {ROOT_DIRECTORY} {args}'",
shell=True,
check=True,
)
return None # The job reports its result itself, see train.py below
# 2. Define the search space; the names match the arguments of the training job
class LMSpace(extrapolaw.PipelineSpace):
lr = extrapolaw.Float(lower=1e-4, upper=1e-2, log=True, prior=3e-4)
batch_size = extrapolaw.Integer(lower=64, upper=1024, log=True, log_base=2)
width = extrapolaw.Integer(lower=256, upper=4096, log=True, is_compute_scaling=True)
n_layers = extrapolaw.Integer(lower=4, upper=48, is_compute_scaling=True)
n_tokens = extrapolaw.Integer(lower=10**8, upper=10**11, log=True, is_compute_scaling=True)
# The training FLOPs of a configuration, known before it is evaluated
def estimate_flops(width: int, n_layers: int, n_tokens: int) -> float:
return 6 * (12 * n_layers * width**2) * n_tokens
# 3. Acquire data under the cost constraint
extrapolaw.acquire(
evaluate_pipeline=evaluate_pipeline,
pipeline_space=LMSpace(),
root_directory=ROOT_DIRECTORY,
cost_estimator=estimate_flops, # Cost of every configuration
max_cost_per_config=1e19, # Never sample a configuration costing more than this
total_cost_to_spend=1e20, # Budget for the whole search
is_async=True, # Keep sampling while earlier jobs are still running
live_plots=True, # Update the data-acquisition plots during the run
)
# 4. Predict the optimal performance and configuration at the target scale(s)
predictions = extrapolaw.predict(
root_directory=ROOT_DIRECTORY,
target_costs=[1e22, 1e23],
) # The plots and their data are also saved to `root_directory/summary/`The training job, train.py, reports its result back to Extrapolaw when it finishes:
from pathlib import Path
import extrapolaw
args = parse_args() # lr, batch_size, width, n_layers, n_tokens, pipeline_id, root_directory
model = build_transformer(width=args.width, n_layers=args.n_layers) # Wrapped in DDP or FSDP
val_loss = train_and_eval(model, lr=args.lr, batch_size=args.batch_size, n_tokens=args.n_tokens)
if is_main_process():
extrapolaw.save_pipeline_results(
user_result={
"objective_to_minimize": val_loss,
"cost": 6 * model.n_params * args.n_tokens, # Training FLOPs
"n_params": model.n_params,
"n_data": args.n_tokens,
},
pipeline_id=args.pipeline_id,
root_directory=Path(args.root_directory),
live_plots=True, # Redraw the scaling plots with this result
)See the evaluate_pipeline reference for the full asynchronous workflow.
- Scaling study of TabPFN: a random search under a FLOP budget, with the scaling law fitted and extrapolated.
- Asynchronous evaluation: submit each configuration as a Slurm job and report its result back with
extrapolaw.save_pipeline_results(). - Scaling study of an OpenCLIP VLM: vision and text towers of varying width and depth, each trained in a Slurm job sized to its GPU memory needs.
- Distributed training: evaluate configurations with DDP or FSDP, in native PyTorch or PyTorch Lightning.
- Getting started
- Reference: running Extrapolaw, search spaces, optimizers, and analysing runs
- Scaling optimizers: cost-constrained search guided by scaling laws
- Algorithms
- API
Please see the documentation for contributors.
To cite Extrapolaw or the papers behind its algorithms, see our citation guide.