Skip to content

Commit

Permalink
doc up
Browse files Browse the repository at this point in the history
  • Loading branch information
grencez committed Jul 27, 2024
1 parent 5022a3c commit f5f7897
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions example/prompt/assistant_chatml/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,6 @@
This example should be run with [ChatML](https://github.com/openai/openai-python/blob/main/chatml.md)-style models that are tuned to behave like an instruction-following assistant chatbot.

The model typically should have special `<|im_start|>` and `<|im_end|>` tokens, but `setting.sxpb` configures fallbacks that attempt to support any model.
Models that don't support ChatML may produce nonsense, but Gemma seems to behave well, so we specifically try Gemma-style `<start_of_turn>` and `<end_of_turn>` tokens as fallbacks.
When no special tokens are found, we fall back to using BOS and EOS tokens to support jondurbin's Bagel finetunes like [bagel-7b-v0.5](https://huggingface.co/jondurbin/bagel-7b-v0.5).
Gemma is basically the same format but without a `system` role, so we specifically look for Gemma-style `<start_of_turn>` and `<end_of_turn>` tokens as fallbacks.
When no other special tokens are found, we fall back to using the BOS and EOS tokens that all models have.
This is how jondurbin's [bagel-7b-v0.1](https://huggingface.co/jondurbin/bagel-7b-v0.1) finetune supported ChatML, and other instruct-tuned models tend to figure it out.

0 comments on commit f5f7897

Please sign in to comment.