qwen3-prefix-cache-bench

Reproducible benchmark of Qwen3-1.7B-Base prefix caching on RTX 3070 Laptop (8 GB, BF16, memory-bound regime). Hand-written reference inference loop + vLLM v1 comparison.

Status: 🚧 work-in-progress — Phase A baseline data coming.

Why

Most LLM inference benchmarks run on A100 / H100. This repo measures what prefix caching actually buys you on consumer-grade hardware (3070 Laptop, 8 GB) in the memory-bound regime that Qwen3-1.7B occupies.

Three questions:

1. What's the break-even prompt length where prefix cache amortization wins over prefill?
2. How does prefix hit ratio shape the TTFT / TPOT trade-off curve?
3. Where's the launch-overhead vs memory-bandwidth bound boundary on this hardware?

Methodology

• Hardware: RTX 3070 Laptop (8 GB VRAM, GDDR6 ~256 GB/s)
• Model: Qwen3-1.7B-Base (BF16), via ModelScope
• Runtime: vLLM v0.x (version pinned in requirements.txt)
• Matrix: concurrency ∈ {1, 2, 4, 8} × prompt_len ∈ {128, 512} × output_len ∈ {64, 128} × prefix_hit_ratio ∈ {0.0, 0.5, 1.0}
• Metrics: TTFT (p50, p99), TPOT (p50, p99), inter-token latency, GPU mem util, SM util

Repo layout

benchmarks/        # benchmark drivers
results/           # data.csv, plots, trace samples
reference/         # hand-written inference loop, for understanding (NOT production)
docs/              # methodology, findings notes

Reproduce

(Filled in after Phase A baseline lands.)

License

MIT