single-gpu-llm-bench

Reproducible local LLM benchmarks on consumer/prosumer single-GPU hardware — speed, context ceiling, GSM8K, MMLU-Pro, long-context retention, HumanEval, and GAIA Level 1, run with llama.cpp, lm-evaluation-harness, and smolagents.

No model weights are stored in this repo — every model directory documents where to download the weights from and how to reproduce the exact serving config used.

What's here

Path	What it is
comparisons/	Cross-model comparison writeups — one file per comparison run
models/	One self-contained directory per model benchmarked: results, scripts, patched configs
common-scripts/	Latest, most-complete versions of the reusable benchmark scripts — the starting point for benchmarking a new model
BENCHMARKING_GUIDE.md	Accumulated methodology, gotchas, and how to add a new model or hardware setup

This comparison run

Hardware: RTX 5080 (16GB, Blackwell), WSL2 Ubuntu, AMD Threadripper PRO 5955WX

Three 27B-class models at three different quantization levels:

Model	Quantization	Directory
Ternary-Bonsai-27B	Q2_0 (~2-bit)	models/ternary-bonsai-27b/
Bonsai-27B	Q1_0 (~1-bit)	models/bonsai-27b/
Qwen3.6-27B-UD-IQ3_XXS	Unsloth Dynamic (~3-bit)	models/qwen3.6-27b-ud-iq3-xxs/

Full findings + side-by-side comparison table: comparisons/2026-07-27b-3quant-rtx5080-16gb.md

Quick start — reproducing a single model's results

Each directory under models/ is self-contained:

cd models/<model-name>/
cat INSTALL.md    # where to get the model weights + llama.cpp build + Python env
cat README.md     # verified server config, full results, gotchas specific to this model

Install (shared across all models)

python3 -m venv ~/lmeval-env
source ~/lmeval-env/bin/activate
pip install lm-eval smolagents openai rich duckduckgo-search gguf

Each model's own INSTALL.md covers the model-specific pieces (weight download source, which llama.cpp build/fork is needed, any model-specific quirks).

Adding a new model or comparison

See BENCHMARKING_GUIDE.md — it has the full accumulated methodology (verified context-ceiling procedure, required server flags, known pitfalls) and a checklist for benchmarking a new model against this same suite, plus where to add the new comparison writeup.

License

MIT — see LICENSE.