FastKernels

A reproducible benchmarking suite for evaluating custom CUDA / Triton / PyTorch kernels across a broad zoo of modern model architectures: dense and MoE LLMs, linear-attention LLMs, diffusion / video / audio models, multimodal and vision encoders, detection / edge networks, 3D / robotics / science models, recommendation models, and world models.

Operators are organized into four levels of abstraction (L1 single-kernel, L2 fused/composite, L3 layer/block, L4 end-to-end pipeline). For each architecture the suite ships a reference implementation (tasks/baseline/) and a slot for candidate replacements (tasks/candidate/); a runner swaps candidates in, validates correctness against the baseline, and measures speedup.

Quick Start

Requires Python 3.10+, CUDA 12+, and a recent NVIDIA GPU (Hopper / Blackwell tested; Ampere supported for a subset of kernels).

git clone https://github.com/Snowflake-AI-Research/fastkernels.git
cd fastkernels
pip install .

This installs the fastkernels CLI plus all benchmark dependencies (PyTorch, Triton, FlashAttention, DeepGEMM, fastsafetensors, etc.).

How to Use FastKernels

There are three main ways to enjoy FastKernels:

1. Optimizing a model for inference: Import a model, capture workload traces, and optimize its operators.
2. Benchmarking agent-generated GPU kernels: Provide a rigorous, sterile environment to test kernels generated by AI coding agents.
3. Using FastKernels as a minimal end-to-end serving framework: Deploy your optimized models directly for inference.

Crucially, optimized kernels developed in the first two functionalities are automatically swapped in and deployed when using FastKernels as a serving framework.

Please read the quick start guide that matches your application:

• Model Optimization Guide
• Benchmarking Agent-Generated Kernels
• End-to-End Serving Framework

If you are contributing to FastKernels directly, please see the Developer Guide.

CLI Overview

Here is a brief illustration of the CLI commands used across these workflows:

# List all available families, architectures, operators, or workloads
fastkernels list [--workloads | --map]

# Capture runtime init/forward type, shape, and dtype metadata for operators
fastkernels capture minimal

# Run a single L1/L2/L3 operator benchmark against its baseline
fastkernels bench --target rms_norm

# Run end-to-end throughput / latency / correctness across scenario models
fastkernels eval full

Citation

If you use FastKernels, please cite our accompanying paper:

@misc{oliaro2026fastkernels,
      title={FastKernels: Benchmarking GPU Kernel Generation in Production}, 
      author={Gabriele Oliaro and Yichao Fu and May Jiang and Owen Lu and Junli Wang and Zhihao Jia and Hao Zhang and Samyam Rajbhandari},
      year={2026},
      eprint={2605.23215},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.23215}, 
}

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.