-
Notifications
You must be signed in to change notification settings - Fork 51
Issues: zurawiki/tiktoken-rs
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Getting decoder error on specific token id when using GPT2 BPE tokenizer
#97
opened Nov 30, 2024 by
nerdai
Is there a way to cache the bpe when using
num_tokens_from_messages
#81
opened Aug 13, 2024 by
xd009642
Token counts are incorrect when request includes function messages
#40
opened Oct 5, 2023 by
kylewlacy
[Question] Difference between
encode_with_special_tokens and encode_ordinary
#30
opened Aug 11, 2023 by
kojix2
How could you add parallelism to make the encoding faster?
enhancement
New feature or request
help wanted
Extra attention is needed
#7
opened Mar 13, 2023 by
smahm006
ProTip!
no:milestone will show everything without a milestone.