Systematic benchmark of inference-time scaling strategies for LLM reasoning. Measures accuracy, latency cost, and compute efficiency across chain-of-thought, repeated sampling, and self-consistency methods.
OpenAI o1, DeepSeek-R1, and QwQ showed that scaling compute at inference time improves reasoning quality. But the tradeoffs are rarely quantified.
This benchmark answers three questions directly:
- How much does each inference-time scaling strategy improve accuracy?
- At what latency cost?
- When do returns diminish to near zero?
It connects to four prior projects in this portfolio:
sampling-strategy-bench -> cost of sampling configurations
output-length-predictor-bench -> output length is highly variable
batched-speculative-decoding -> speedup for long sequences
slo-aware-autoscaling-sim -> capacity planning for high-latency strategies
Without CoT, no strategy produces meaningful accuracy. With CoT at 256 tokens, accuracy jumps from near zero to 65-72%.
| Strategy | Qwen2-0.5B | Qwen2-1.5B | lat_x |
|---|---|---|---|
| direct_greedy | 3.1% | 6.2% | 1.0x |
| repeated_sampling N=8 | 6.2% | 6.2% | 8-9x |
| cot_greedy_128 | 21.9% | 9.4% | 7-8x |
| cot_greedy_256 | 65.6% | 71.9% | 10x |
| cot_greedy_512 | 65.6% | 71.9% | 10x |
| cot_majority N=4 | 50.0% | 84.4% | 37x |
| cot_majority N=8 | 62.5% | 87.5% | 74x |
| Strategy | Qwen2-0.5B acc/lat_x | Qwen2-1.5B acc/lat_x |
|---|---|---|
| cot_greedy_256 | 0.0623 | 0.0758 |
| cot_majority N=2 | 0.0220 | 0.0411 |
| cot_best_of_n N=2 | 0.0251 | 0.0386 |
Models stop generating useful reasoning at 170-180 tokens on these tasks. Allocating 512 tokens is wasted budget.
| Setting | Qwen2-0.5B | Qwen2-1.5B |
|---|---|---|
| cot_greedy_256 (depth) | 0.656 at tok_x=13 | 0.719 at tok_x=9.5 |
| cot_majority N=2 (diversity) | 0.500 at tok_x=27 | 0.750 at tok_x=18 |
For the smaller model, one deep trace beats multiple shorter traces. For the larger model, diverse traces with voting outperform depth.
Qwen2-1.5B, cot_sampling_majority:
N=2 -> 75.0% at 18x latency
N=4 -> 84.4% at 37x latency (+9.4pp at 2x cost)
N=8 -> 87.5% at 74x latency (+3.1pp at 2x cost)
Qwen2-0.5B: direct=0% cot256=0% majority_n4=0%
Qwen2-1.5B: direct=20% cot256=20% majority_n4=40%
Inference-time scaling does not rescue task types the model does not handle at baseline.
Greedy decoding with a direct-answer prompt. Zero-compute baseline.
Sample N short answers. Majority vote. Tests whether diversity over a limited reasoning space helps.
Single chain-of-thought trace, greedy decoding. Tested at 128, 256, and 512 max_new_tokens.
Standard self-consistency. Generate N CoT traces, extract final answer from each, majority vote. The standard inference-time scaling approach from Wang et al. 2023.
Generate N CoT traces, select highest mean logprob. Tests whether model confidence correlates with correctness in reasoning traces.
32 synthetic arithmetic reasoning tasks across 6 types:
- arithmetic_chain — sequential operations on a single value
- inventory — stock accounting with purchases and sales
- group_total — rows times columns minus empty seats
- two_step_word — give and receive narrative problems
- comparison — two parties perform operations, compare results
- conditional — if-else branching based on arithmetic condition
All tasks have exact integer answers, enabling unambiguous quality measurement.
git clone https://github.com/JohnScheuer/inference-time-scaling-bench
cd inference-time-scaling-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python run.py
Runtime: approximately 3-4 hours on an RTX 2070 for the full sweep.
| Plot | Description |
|---|---|
| 01_accuracy_vs_n.png | Accuracy vs N for sampling strategies |
| 02_cot_depth.png | CoT accuracy vs token budget |
| 03_frontier_latency.png | Accuracy vs latency multiplier, all strategies |
| 04_frontier_tokens.png | Accuracy vs token multiplier |
| 05_self_consistency.png | Majority vote vs logprob reranking vs single trace |
| 06_efficiency.png | Accuracy per unit of latency cost |
| 07_per_task_type.png | Accuracy breakdown by task type |
| 08_depth_vs_diversity.png | Single long trace vs N shorter traces |
inference-time-scaling-bench/
├── src/
│ ├── config.py — models, strategies, budgets
│ ├── tasks.py — 6-type synthetic task generation
│ ├── metrics.py — answer parsing and majority voting
│ ├── generator.py — model loading and generation
│ ├── strategies.py — all inference-time scaling strategies
│ ├── bench.py — sweep orchestration
│ └── analysis.py — 8 plots and summary tables
├── results/
├── plots/
├── run.py
├── SUMMARY.txt
├── DESIGN.md
├── LICENSE
└── requirements.txt
- Python 3.10+
- PyTorch >= 2.1.0
- Transformers >= 5.0.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
- NumPy >= 1.26.0
- NVIDIA GPU with >= 8GB VRAM
- Factual recall, code, or open-ended generation
- Models beyond Qwen2-0.5B and 1.5B
- Temperature sensitivity of self-consistency
- Adaptive token budgets that stop at the reasoning plateau
- Batched inference
- DESIGN.md — full design rationale and module descriptions
- SUMMARY.txt — plain-text findings with all numbers
- LICENSE — MIT License
- sampling-strategy-bench
- output-length-predictor-bench
- batched-speculative-decoding
- slo-aware-autoscaling-sim
- draft-model-selection-bench
MIT License — Copyright (c) 2026 João Felipe De Souza
See LICENSE for details.
João Felipe De Souza