A simple basic binding of the Tiktoken tokenizer in Haskell.
Currently only supports Linux and only exposes the basic Encode function with the gpt2 tokenizer config, but exposing other functions and configs should be trivial.
Requires cargo
to compile. cabal test
should just work.