feat(tokenizer): add bounded L0 exact-match encode cache - #270
Open
jiannan-17 wants to merge 1 commit into
Open
Conversation
Add CachedTokenizer, a wrapper around Arc<dyn Tokenizer> that memoizes successful encode() results keyed by the exact input string. Hits return an owned clone of the full Encoding, so token IDs, token strings, offsets and masks are preserved. The cache is bounded by entry count and estimated retained bytes with LRU eviction (lru crate, default features disabled). Results larger than max_entry_bytes are returned but not stored. A single parking_lot::Mutex guards the LRU; tokenization on a miss and the Encoding clone on a hit run outside the lock. encode_batch(), decode and metadata methods delegate to the wrapped tokenizer, and errors are never cached. The wrapped tokenizer must be deterministic with a fixed configuration. Hit, miss, eviction and oversized counters are exported under vllm_tokenizer_cache_*. The entries and bytes gauges hold the total occupancy of every live cache: each instance applies its change to a process-wide aggregate and publishes the totals while holding its own lock, and clear() and Drop subtract its share. Tests use the mock tokenizer and a checked-in byte-level BPE fixture (tests/fixtures/tokenizer/byte_level_bpe.json); gauge semantics are covered with a capturing metrics recorder. A criterion benchmark compares hit, miss, mixed and concurrent workloads against uncached encoding. Construction is library-level only; request-path integration and CLI flags are left for a follow-up. Refs vllm-project#269, vllm-project#244
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
jiannan-17
marked this pull request as draft
September 11, 2026 23:05
jiannan-17
marked this pull request as ready for review
September 11, 2026 23:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Refs #269.
Adds
CachedTokenizerto reuseencode()results for identical inputs. Each cache is tied to a deterministic tokenizer with fixed configuration and returns owned clones of the fullEncoding.lruwith default features disabled.Batch encoding, decoding and metadata calls pass through. This PR adds the library wrapper; request-path integration and CLI options follow separately.
Test Plan
Tests cover encoding metadata, eviction, byte accounting, batch bypass and concurrent gauge updates using mocks and a checked-in BPE fixture.
Test Result
83 tests pass. Formatting passes; Clippy reports no warnings in changed files.
TinyLlama tokenizer on Apple M4 Pro, Criterion 0.5, 100 samples. Point estimates below exclude returned-encoding and per-miss cache destruction.
The 90% hit workload is 7.1× faster than uncached. Concurrent all-hit throughput peaks at 4 threads and drops to ~2.0M encodes/s at 8; sharding is deferred.
A 16 KB input retains an estimated 257,025 bytes per entry. These estimates are not RSS bounds. Router end-to-end performance is not measured.
PR checklist