-
Notifications
You must be signed in to change notification settings - Fork 10
Open
Description
A small improvement on gpt2's bpe ranking code (GPT2Codec.java): byte decoding can be performed at load time on every key from encoder.json and every space-separated pair from vocab.bpe (after invoking split(" ") on the string).
This avoids having to perform the byte conversion every time in memory.
Tested with my own implementation.
Metadata
Metadata
Assignees
Labels
No labels