Skip to content

zygi/tiktoken-hs-bindings

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A simple basic binding of the Tiktoken tokenizer in Haskell.

Currently only supports Linux and only exposes the basic Encode function with the gpt2 tokenizer config, but exposing other functions and configs should be trivial.

Requires cargo to compile. cabal test should just work.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published