Tokenizer discard data that exceed max_length #31627
Labels
Core: Tokenization
Internals of the library; Tokenization.
Feature request
Request for a new feature
Feature request
When use tokenizer, it truncate data to max_length, but can't just discard the data.
Motivation
Sometimes we want the sentence to be complete
Your contribution
No
The text was updated successfully, but these errors were encountered: