Addition of o200k_harmony for OSS models

[ScotterC]

Tiktoken equivalent commit: openai/tiktoken@3591ff1

Side note:
Is it possible to automate this better? What changes are needed in tiktoken_ruby every time a new encoding is added to Tiktoken upstream?

[gjtorikian]

Automation would be nice. Don't have any ideas on how to implement that at the moment, but here are the files that need to be changed: #75

If my eyes don't deceive me it's just gpt-oss- that's missing right now?

[ScotterC]

yes but I think we need to add a o200k_harmony method maybe? That encoding hasn't existed for other models before. Unless the rust bindings take care of that

[gjtorikian]

Yeah; I just added a comment to a PR which adds support for this. We'll see what happens, but I'll otherwise point to a fork if there's no movement upstream.

[davletovalmir]

Added to rust lib in zurawiki/tiktoken-rs#123. Bumping for visibility!