[WIP][DO NOT REVIEW] bench: add router CPU fast-path baseline harness - #290
Draft
jiannan-17 wants to merge 2 commits into
Draft
jiannan-17 wants to merge 2 commits into
jiannan-17 wants to merge 2 commits into
Conversation
Add a reproducible baseline for what the router itself costs on the request path, ahead of the tokenizer/L0 request-path integration (roadmap vllm-project#244, Issue 1). - tests/common/bench_corpus.rs: seeded prompt corpus (hot64, cold, mixed90, short_shared_prefix at 200 B / 2 KiB / 16 KiB) and request builders shared by the benchmark and the harness. - benches/routing_input.rs: criterion groups for extract_text_for_routing, cache_aware / rendezvous_hash worker selection, and Tree insert/prefix_match cost for three candidate routing-key encodings. - tests/router_overhead_bench.rs (#[ignore]): spawns the real vllm-router binary per scenario in front of in-process mock workers, drives closed-loop load, and reports client latency, throughput and the router's own CPU time and peak RSS via wait4(2). A direct_mock scenario gives the floor; means are subtracted, percentiles are reported side by side. - tests/common/bench_mock.rs: a zero-delay worker without the capture store of MockWorker, so it stays flat under load. - tests/bench_corpus_test.rs: CI guard that the corpus and mock build and stay deterministic. - docs/benchmarks/router_overhead.md, scripts/run_benchmarks.py (--bench routing_input, --router-overhead), .buildkite/README.md. No runtime code changes. Signed-off-by: Jiannan Li <april.jiannanli@gmail.com>
28 tasks
- count warmup requests in router CPU per request - draw corpus text from distinct bodies; add long_shared_prefix - add repeats, rate cap, per-worker load share, router-only pinning - avoid Tree drop leak and large-drop memory growth in criterion Signed-off-by: Jiannan Li <april.jiannanli@gmail.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add benchmarks to measure router overhead before the tokenizer/L0 request-path integration in #244, following #270. No runtime behavior changes.
Both use a shared, deterministic corpus. Run instructions and measurement details are in
docs/benchmarks/router_overhead.md. These benchmarks measure router overhead; routing benefits need real-worker measurements.Test Plan
Test Result
Validation recorded during implementation:
Benchmark measurements are pending. The tree-key format comparison also needs review before this is ready.
Essential Elements of an Effective PR Description Checklist