wordchipper/CITATION.cff at main · zspacelabs/wordchipper · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
  - family-names: "Dunnavant"
    given-names: "Crutcher"
    email: "crutcher@zspacelabs.ai"
  - family-names: "Tadjibaev"
    given-names: "Dilshod"
    email: "dilshod@gmail.com"
title: "Wordchipper"
version: 0.9.0
date-released: 2026-03-24
url: "https://github.com/zspacelabs/wordchipper"
repository-code: "https://github.com/zspacelabs/wordchipper"
license: MIT
abstract: "A high-performance Rust byte-pair encoder tokenizer for the OpenAI GPT-2 tokenizer family, achieving 4-9x throughput speedups over tiktoken-rs and tiktoken using custom DFA lexers and parallel processing."
keywords:
  - tokenization
  - bpe
  - large-language-models
  - rust
  - high-performance
  - no-std