The same work. Fewer tokens. Lower bill.
60–90% self-host savings · 20× prompt compression · 10× cache hits · 85% smarter routing
Curated open-source projects for cutting LLM bills without cutting capability — self-hosting, quantization, KV cache tricks, token compression, semantic caching, model routing, and cost observability. Every entry earns its place by concretely reducing token spend, inference cost, or API bills. Contribute · guidelines.
You know the bill. Tokens in, dollars out. You spin up another agent, another RAG pipeline, another 200K context window — and somewhere a GPU farm charges you rent.
The fix is not one trick. It is a stack: host your own weights, quantize them, compress the prompts, cache what repeats, route easy work to cheap models, and measure every cent before it compounds.
This list is the map.
Run models on your own hardware or rented GPUs. At scale, self-hosting often beats per-token API pricing by 60–90%.
- Exo - Clusters laptops and desktops into a distributed inference mesh. Splits models across devices so you can run 70B+ locally instead of paying per-token APIs.
- llama.cpp - Efficient C/C++ inference for LLaMA-family models. The foundation many local runners build on; excellent for CPU and quantized models.
- llamafile - Single-file executable models built on llama.cpp. Ship weights + runtime together with zero install — ideal for offline, zero-API-cost deployments.
- LocalAI - OpenAI-compatible API server that runs on CPU, GPU, or Apple Silicon. Broad model and backend support without cloud lock-in.
- Ollama - One-command local model runner. Best for prototyping, single-user workflows, and edge deployments on consumer hardware.
- Tabby - Self-hosted AI coding assistant. Replace Copilot-style subscriptions with a model you control.
Production-grade serving stacks that squeeze more tokens per GPU dollar through PagedAttention, continuous batching, and OpenAI-compatible APIs.
- vLLM - Industry-standard high-throughput inference engine. Up to 10–20× better concurrency than naive serving on the same hardware.
- SGLang - Fast structured generation and serving runtime with RadixAttention for prefix caching across requests.
- TensorRT-LLM - NVIDIA's optimized inference library. Maximum throughput on datacenter GPUs when you need every FLOP.
- TGI (Text Generation Inference) - Hugging Face's production server with continuous batching, quantization, and tensor parallelism.
- LMDeploy - Efficient deployment toolkit from OpenMMLab with TurboMind backend for high-throughput serving.
- Aphrodite Engine - vLLM-compatible inference engine with additional optimizations for open models.
Shrink model weights so the same GPU serves more tokens per dollar. Match the format to your runtime: GGUF for llama.cpp/Ollama, AWQ/GPTQ for vLLM/TGI.
- AutoGPTQ - Post-training 4-bit quantization with GPTQ. Cuts VRAM ~4× on NVIDIA GPUs while keeping perplexity within ~2% of FP16.
- AWQ (llm-awq) - Activation-aware INT3/4 quantization (MLSys 2024 best paper). Preserves salient weight channels for better accuracy at 4-bit on datacenter GPUs.
- bitsandbytes - 4-bit/8-bit loading in PyTorch for research and QLoRA fine-tuning. Lets you fit larger models on consumer GPUs without cloud APIs.
Attack the two biggest inference bottlenecks: KV cache memory on long contexts, and serial token generation. These squeeze more throughput from hardware you already own.
- EAGLE - Speculative decoding via feature extrapolation. A small draft head proposes multiple tokens; the main model verifies in parallel for ~2–3× faster generation with provably identical output distribution.
- kvpress - NVIDIA's unified library of KV cache compression methods (SnapKV, H2O, Finch, and more). Drop-in Hugging Face pipeline for long-context workloads where cache size dominates memory.
- LMCache - Distributed KV cache layer for vLLM and SGLang. Shares prefix KV across instances and tiers so repeated long contexts skip full prefill — lower TTFT and more throughput per GPU.
- Mooncake - KVCache-centric disaggregated serving platform (powers Kimi). Pools and transfers KV across prefill/decode nodes via RDMA — up to 75% more requests at the same SLO on production workloads.
- Medusa - Adds lightweight prediction heads to a frozen base model to speculate multiple tokens per step. No separate draft model required.
- SnapKV - Fine-tuning-free KV cache compression. Observes attention patterns in a prompt window, keeps only clustered important positions per head — up to 8× memory savings on 16K+ contexts.
Shrink prompts and outputs before they hit the bill. These tools target the tokens you never needed in the first place.
- Claw Compactor - 14-stage content-aware compression pipeline with AST-aware code folding, JSON sampling, and reversible storage. Zero LLM inference cost; 15–82% savings depending on content type.
- Caveman - Agent skill that cuts ~65% of output tokens by dropping filler while preserving technical accuracy. Also compresses CLAUDE.md and memory files for compound input savings.
- caveman-code - Standalone CLI that stacks input, tool-call, dedup, and output compression layers for agent workloads.
- LLMLingua - Microsoft Research prompt compressor. Up to 20× compression on long contexts with ~1.5% accuracy loss on benchmarks.
- LongLLMLingua - Long-context variant that mitigates "lost in the middle" while compressing RAG prompts to a fraction of their size.
- LLMLingua-2 - Faster, task-agnostic prompt compression using a distilled model for near-real-time use.
- Ponytail - Agent skill that stops over-engineering via a YAGNI ladder: stdlib, native platform, and one-liners before custom code. Agentic benchmark: ~54% less code, ~22% fewer tokens, ~20% lower cost on real Claude Code sessions.
- Tokenless - Rust toolkit for schema compression, differential responses, TOON encoding, and command-output rewriting. Targets 60–90% savings on agent tool-call loops.
Skip the LLM entirely when a similar question was already answered, or reuse provider-side prefix caches.
- FerroCache - Standalone Rust semantic cache service with fleet-wide sharing, MCP support, and drop-in OpenAI/Anthropic SDK wrappers. Survives app restarts — skip redundant LLM calls across your whole stack.
- GPTCache - Semantic cache for LLM apps. Vector similarity matching returns cached responses for equivalent queries — up to 10× cost reduction on hit.
- LiteLLM - Universal LLM gateway with in-memory, Redis, S3, and semantic caching backends. One integration for caching across providers.
- ModelCache - Multi-level semantic cache with embedding similarity and TTL management for production LLM apps.
- OpenAI Prompt Caching Cookbook - Practical guide to structuring prompts for up to 90% off cached input tokens on OpenAI models.
- RedisVL - Redis vector library for building semantic caches with sub-millisecond lookups at scale.
- semcache - Rust HTTP proxy semantic cache for OpenAI, Anthropic, and Gemini. Drop-in caching with Prometheus metrics and a built-in dashboard — no app code changes required.
Route easy tasks to cheap models and hard tasks to capable ones. Enforce budgets before spend happens.
- Bifrost - High-performance AI gateway with semantic caching, budget limits, and automatic failover across 23+ providers. OpenAI-compatible drop-in with sub-15µs routing overhead.
- Conduit - ML-powered router using Thompson Sampling bandits. Learns which model handles each query type best, balancing cost, quality, and latency from live traffic.
- LiteLLM Proxy - Production proxy with budget limits, rate limiting, load balancing, and fallback chains across 100+ models.
- LLMRouter - Research-grade routing library with 16+ strategies (KNN, MLP, Elo, graph-based, BERT routers). Unified CLI for training cost-aware routers on benchmark data.
- Martian - Python SDK for Martian's LLM gateway and routers. Programmatically manage cost-aware routing across 200+ models via a unified API.
- OpenRouter - Unified API across providers with automatic routing to cheapest available model for a given capability tier.
- ParetoBandit - Cost-aware contextual bandit router with online budget pacing. Adapts when model prices or quality shift — routing decisions in microseconds on CPU.
- Portkey - AI gateway with caching, retries, load balancing, and observability hooks for multi-provider setups.
- RouteLLM - ML-based router that sends queries to strong or weak models based on difficulty. Up to 85% cost reduction with minimal quality loss.
- UncommonRoute - Local OpenAI-compatible proxy that routes each agent step independently. Held-out SWE-bench: matched task quality at 53% lower API cost vs frontier-only.
You can't optimize what you can't measure. These tools attribute token spend to users, features, and prompts.
- CostPilot - Self-hosted LLM cost dashboard with a 3-line Python SDK. Tracks spend, tokens, and latency per model/feature with forecasting and savings recommendations.
- Helicone - LLM observability proxy that logs every request with latency, cost, and cache hit metrics.
- llmwatch - Lightweight SDK instrumentation for OpenAI, Anthropic, Google, and more. Tags costs by feature/user with bundled pricing for 1000+ models — no proxy required.
- Narev - Open-source Vercel AI SDK middleware for usage-based billing. Captures token usage, resolves live model prices, and routes billing events to Stripe, Polar, or Lago.
- Shekel - Budget enforcement and cost tracking for agentic systems. One-line Python integration with OpenTelemetry metrics, circuit breakers, and per-tool spend limits.
Engineering patterns and libraries that reduce context bloat without adding new infrastructure.
- Anthropic Prompt Caching Docs - Official guide to
cache_controlbreakpoints for up to 90% off repeated input prefixes. - DSPy - Declarative prompt optimization from Stanford NLP. Optimizers like MIPROv2 and BootstrapFewShot prune redundant instructions and pick efficient few-shot examples from your data.
- Mem0 - Long-term memory layer for agents. Stores and retrieves only relevant facts instead of replaying full chat history — cuts recurring context tokens on every turn.
- tiktoken - Fast BPE tokenizer. Count tokens before you send them and catch bloated prompts at build time.
- awesome-llm-apps - Curated collection of LLM applications and use cases.
- awesome-local-llm - Resources for running LLMs locally.
- awesome-ai-tools - Broader AI tooling landscape.
Contributions welcome. Please read CONTRIBUTING.md before opening a PR. See CHANGELOG.md for recent updates.
Quick rules:
- Add projects that concretely reduce token spend, inference cost, or API bills.
- Prefer open-source or freely usable tools with active maintenance.
- One line per entry: what it does and why it saves money.
- No paid-only products without a meaningful free tier unless they're industry-standard references (e.g. cloud GPU providers).
CC0 1.0 Universal. To the extent possible under law, contributors waive all copyright and related rights to this list.