Repository navigation

perf(tokenizer): byte-level incremental decoding and a cheaper L0 cache - #2765

Open

slin1237 wants to merge 4 commits into

mainfrom

perf/tokenizer-decode

Commits on Oct 4, 2026

perf(tokenizer): decode byte-level vocabularies incrementally without re-decoding
slin1237
committed
facbc0d
perf(tokenizer): count L0 cache entries in atomics and cache token ids only
slin1237
committed
e819d8b
fix(tokenizer): keep the L0 counters sane across clear() and refresh the Sequence docs
slin1237
committed
1dde1b0
fix(tokenizer): keep L0 map mutations and their counters consistent with clear()
slin1237
committed
369b0d1