llmfit 1.0 — the release where the numbers became verifiable

[AlexsJones]

The community around llmfit constantly challenges us toward excellence — and rightly disconfirms, critiques, and asks for more. You told us the tok/s estimates weren't accurate enough, and you were right. llmfit 1.0 is the culmination of a lot of work to improve that — and we're calling it 1.0 because this is the release where every number llmfit shows you became verifiable.

Until now, llmfit's speed estimates came from a pure physics-based memory-bandwidth model with fixed assumptions baked in. Starting with 1.0, that model is grounded in runtime sampling and real measured data:

1. Your machine is now sampled, not assumed

llmfit measures your system's effective RAM bandwidth at startup with a short multithreaded sampling sweep (~100 ms), replacing a hardcoded 50 GB/s DDR4 assumption. On modern DDR5/LPDDR5X systems this roughly doubles the accuracy of offload estimates — especially for Mixture-of-Experts architectures, where expert streaming over system RAM is the bottleneck. llmfit system shows you the measured number:

Total RAM: 30.82 GB
Available RAM: 18.57 GB
RAM Bandwidth: ~105 GB/s (measured)

2. Real user data beats formulas

If your GPU matches community benchmark data (RTX 3090/4090/5090, A100, Apple M-series, …), the fit table now shows measured tok/s marked with ✓ — real runs from real users on your hardware, not an estimate.

3. Every estimate now ships its inputs

llmfit info prints the Estimate Basis — the exact bandwidth, efficiency factor, and context assumptions behind the number — plus commands to verify it on your own machine (llmfit bench, or a llama-bench recipe that measures the same quantity the estimate models). No more unexplainable numbers (#292). And when we don't recognize your GPU, we now say so plainly instead of printing a confident guess.

4. Estimates are audited against reality, weekly

CI replays community-measured benchmark runs through the estimator every week; current calibration is median estimated/measured 0.87, and any change that drifts past 2× systematic bias fails the build.

---

What 1.0 means

Going 1.0 is a commitment, not just a number: the --json output schema, exit codes, and environment variables that your scripts and agents depend on are now stable interfaces — breaking them is a major-version event from here on.

Credit where it's due: @RhoMancer's calibration work in #450/#464, @Magolor's #449 report, @kursad-k's #88, and the #292 thread drove this.

Two asks:

• If llmfit tells you your GPU isn't in the bandwidth table, open an issue with the model name — that's one lookup-table entry between you and better estimates.
• If you want your hardware to show measured ✓ numbers, contribute a benchmark run — every submission makes the numbers more honest for everyone. (Update: this now happens straight from llmfit itself — see below.)

---

Get it: llmfit v1.0.0 — binaries for macOS, Linux, and Windows

What's next: we want the measured ✓ data to come from you, with full provenance — the proposal is llmfit bench --all --share, which turns your benchmark run into a pull request under your own GitHub handle. No upload API, no telemetry; every data point a public, auditable commit. The RFC with open design questions is up in #710 — input very welcome.

---

Update (2026-07-20): benchmark & share has shipped

What was "next" above is now live: llmfit bench --share — and the same flow built into the TUI — measures real tok/s on your hardware and contributes it back as a pull request under your own GitHub handle. Merged submissions are embedded into the next release, so anyone on identical hardware gets measured ✓ numbers out of the box. The llmfit community corpus is now the primary source of measured data, with every data point a public, auditable commit.

New: a step-by-step benchmarking guide walks the whole journey with screenshots — download a model, serve it, press b, and share your run — no gh CLI or third-party account needed.

[RhoMancer]

So happy to have contributed to this! Looking forward to see what you've cooked up!

[hugging-cat]

Wow, looking forward to llmfit 1.0! How will the community benchmark data be queried? Will it be via API requests in the app, or pre-scraped data in a JSON file?

[AlexsJones]

It's through PR, - you have a completely optional way of pushing

[AlexsJones]

https://github.com/AlexsJones/llmfit/blob/main/docs/benchmarking.md

[AlexsJones]

@bertsons Neither an API call nor scraping — the community data is embedded in the binary at build time. A build.rs step in llmfit-core aggregates everything under llmfit-core/data/community/**/*.json (the merged benchmark-submission PRs) into a JSON array compiled into the release. So: someone's bench --share PR gets merged → the next release ships it to every user, deterministically, with no network fetch and nothing to go stale at runtime.

On your machine, community runs only surface when their hardware fingerprint (CPU + GPU) matches yours — they appear as llmfit community rows on the benchmark page, and feed measured tok/s into the fit table under the trust order your runs > community runs on identical hardware > localmaxxing medians > formula estimate. Details in #712 (the bench --share RFC implementation) and #721 (the read side).

🤝 Handled by Alex's Repo Steward — replies reviewed by @AlexsJones.

Learn more or run your own: https://github.com/AlexsJones/repo-steward

[stefanlasiewski]

Aside from the tok/s and RAM bandwidth measurements outlined here, are any of these calculations anchored to external industry benchmarks? When I mention llmfit benchmarks to my colleges, the first thing they ask is ,"what is it benchmarking?"

[AlexsJones]

Good question, and the first one worth answering. llmfit bench measures speed, not model quality: real decode throughput (tok/s) and time to first token against the model you're serving locally (Ollama, vLLM, llama-server, MLX or Ferrum), over three runs, on your own hardware. It isn't an MMLU-style capability benchmark. It answers "how fast will this model run on this box".

The estimates are anchored to measurements rather than to an industry suite. The memory-bandwidth formula was checked against published llama.cpp results (Apple Silicon, NVIDIA T4), and wherever real numbers exist they win: your own runs first, then community runs on identical hardware, then localmaxxing.com medians, then the formula. Every fit says which of those it used in estimate_confidence. Details are in how it works and the benchmarking guide.