Skip to content

Tokenizer v5: Fix apply chat template + typo #3238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pcuenca merged 2 commits into from
Dec 20, 2025
Merged

Tokenizer v5: Fix apply chat template + typo #3238

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e952214
Update tokenizers.md
qgallouedec Dec 20, 2025
22fa100
Update tokenizers.md
qgallouedec Dec 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tokenizers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,10 +197,10 @@ print(text)
# Give me a brief explanation of gravity in simple terms.<|im_end|>
# <|im_start|>assistant

model_inputs = tokenizer([text], return_tensors="pt")
model_inputs = tokenizer([text], add_special_tokens=False, return_tensors="pt")
Copy link
Member

@pcuenca pcuenca Dec 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree this is dangerous in general (although it does not affect this particular case).

In fact, it'd be useful if we could recommend apply_chat_template with tokenizer=True as the preferred way to prepare conversational inputs cc @ariG23498. (No need to change this example which is illustrative, but we could add a comment in an appropriate point).

```

Notice how the special tokens like `<|im_start>` and `<|im_end>` are applied to the prompt before tokenizing. This is useful for the model to learn where a new sequence starts and ends.
Notice how the special tokens like `<|im_start|>` and `<|im_end|>` are applied to the prompt before tokenizing. This is useful for the model to learn where a new sequence starts and ends.
Copy link
Member

@pcuenca pcuenca Dec 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yes


The `transformers` tokenizer adds everything the raw library lacks:

Expand Down