Skip to content

Pull requests: openai/tiktoken

New pull request New
24 Open 74 Closed
24 Open 74 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Option to use TIKTOKEN_BPE_HOST environment variable for configurable BPE host URL
#357 opened Nov 4, 2024 by AitBaali-Hamza-bcg Loading…
1
make decoder and sorted_token_bytes re-use existing memory
#352 opened Oct 17, 2024 by tmm1 Loading…
use HashSet from rustc-hash too
#351 opened Oct 17, 2024 by tmm1 Loading…
Uses Regex instead of fancy-regex - 6x speedup
#331 opened Aug 5, 2024 by Majdoddin Loading…
3
fix: add encoding for fine-tuned models based on gpt-4o
#329 opened Aug 1, 2024 by hughcrt Loading…
Cache for Encoding - Runtime Boosted by 12%
#319 opened Jul 10, 2024 by Majdoddin Loading…
DOC: Add a link toward PyPI tiktoken package.
#318 opened Jul 9, 2024 by MaxJPRey Loading…
Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer
#314 opened Jun 18, 2024 by LVivona Loading…
Remove dependency on requests
#271 opened Mar 26, 2024 by tristan-jl Loading…
In-code documentation update
#260 opened Feb 19, 2024 by louisbrulenaudet Loading…
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)
#239 opened Jan 15, 2024 by l0rinc Loading…
26
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)
#238 opened Jan 12, 2024 by blaney83 Loading…
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings
#237 opened Jan 6, 2024 by l0rinc Loading…
1
feat: Add option TIKTOKEN_FORCE_CACHE
#170 opened Jul 28, 2023 by Kontrolix Loading…
4
✨ feat: Split code rust, for run as python lib and rust lib
#167 opened Jul 18, 2023 by Miuler Loading…
1
5
fix pycache folder being added instead of encoding info files
#152 opened Jun 23, 2023 by dheerajiiitv Loading…
1
9
Add support for fine-tuned models in encoding_for_model
#135 opened May 23, 2023 by thespino Loading…
2
Option to use env variable for BPE_FILE location
#128 opened May 15, 2023 by csnelsonchu Loading…
update license metadata to use SPDX identifier as per recommendation
#125 opened May 12, 2023 by imrehg Loading…
Allow older versions of dependencies fancy-regex and bstr.
#108 opened Apr 16, 2023 by petterreinholdtsen Loading…
1
Add ruby bindings
#83 opened Mar 28, 2023 by arjun810 Loading…
Enable ppc64le wheels
#65 opened Mar 16, 2023 by mgiessing Loading…
1
Add WASM/JS bindings
#50 opened Mar 6, 2023 by dqbd Loading…
2
WIP: refactor for Java bindings
#40 opened Feb 28, 2023 by eisber Loading…
3
ProTip! Updated in the last three days: updated:>2024-12-16.