From c514bfd0e96907a7ff0cbd56b61d99854a3b992e Mon Sep 17 00:00:00 2001
From: grencez <git@grencez.dev>
Date: Sat, 27 Jul 2024 15:34:15 -0700
Subject: [PATCH] doc

---
 example/prompt/README.md                    |  9 ++++-----
 example/prompt/assistant_chatml/README.md   |  5 ++++-
 example/prompt/assistant_gemma/README.md    |  2 ++
 example/prompt/assistant_gemma/setting.sxpb | 14 ++------------
 4 files changed, 12 insertions(+), 18 deletions(-)

diff --git a/example/prompt/README.md b/example/prompt/README.md
index cb43a64..684c4f4 100644
--- a/example/prompt/README.md
+++ b/example/prompt/README.md
@@ -8,13 +8,12 @@ In order of interest:
 - [confidant_alpaca](confidant_alpaca/): A camelid that occasionally spits.
   - Demonstrates a method of prompting instruction-tuned models to fill in character dialogue.
 - Instruction-following AI assistants.
+  - For all of these examples, the assistant must end its messages with a special token like EOS.
   - [assistant_alpaca](assistant_alpaca/): Alpaca prompt format.
-  - [assistant_chatml](assistant_chatml/): ChatML prompt format that requires special `<|im_start|>` and `<|im_end|>` tokens.
-  - [assistant_gemma](assistant_gemma/): Gemma prompt format requires special `<start_of_turn>` and `<end_of_turn>` tokens.
+  - [assistant_chatml](assistant_chatml/): ChatML prompt format that typically requires special `<|im_start|>` and `<|im_end|>` tokens but is configured with fallbacks.
+  - [assistant_gemma](assistant_gemma/): Gemma prompt format that requires special `<start_of_turn>` and `<end_of_turn>` tokens.
   - [assistant_mistral](assistant_mistral/): Mistral propmt format that requires special `[INST]` and `[/INST]` tokens.
-- [assistant_vicuna](assistant_vicuna/): Conversational AI assistant.
-  - Minimial prompt that lets a Vicuna-style model do its thing.
-  - Only works with models that end the assistant's message with an EOS token.
+  - [assistant_vicuna](assistant_vicuna/): Vicuna prompt format.
 - [assistant_coprocess](assistant_coprocess/): A simple assistant that can be controlled as a coprocess.
   - Demonstrates the `/puts` and `/gets` commands.
 
diff --git a/example/prompt/assistant_chatml/README.md b/example/prompt/assistant_chatml/README.md
index 3491048..990b08d 100644
--- a/example/prompt/assistant_chatml/README.md
+++ b/example/prompt/assistant_chatml/README.md
@@ -1,4 +1,7 @@
 # ChatML Assistant
 
 This example should be run with [ChatML](https://github.com/openai/openai-python/blob/main/chatml.md)-style models that are tuned to behave like an instruction-following assistant chatbot.
-Most importantly, the model must have special `<|im_start|>` and `<|im_end|>` tokens.
+
+The model typically should have special `<|im_start|>` and `<|im_end|>` tokens, but `setting.sxpb` configures fallbacks that attempt to support any model.
+Models that don't support ChatML may produce nonsense, but Gemma seems to behave well, so we specifically recognize Gemma-style `<start_of_turn>` and `<end_of_turn>` tokens in this example.
+When no special tokens are found, we fall back to using BOS and EOS tokens to support jondurbin's Bagel finetunes like [bagel-7b-v0.5](https://huggingface.co/jondurbin/bagel-7b-v0.5).
diff --git a/example/prompt/assistant_gemma/README.md b/example/prompt/assistant_gemma/README.md
index 6341a71..b582293 100644
--- a/example/prompt/assistant_gemma/README.md
+++ b/example/prompt/assistant_gemma/README.md
@@ -2,3 +2,5 @@
 
 This example should be run with Gemma-style models that are tuned to behave like an instruction-following assistant chatbot.
 Most importantly, the model must have special `<start_of_turn>` and `<end_of_turn>` tokens.
+
+It's like the [assistant_chatml](../assistant_chatml/) example but without a system prompt.
diff --git a/example/prompt/assistant_gemma/setting.sxpb b/example/prompt/assistant_gemma/setting.sxpb
index 2541701..d9403f0 100644
--- a/example/prompt/assistant_gemma/setting.sxpb
+++ b/example/prompt/assistant_gemma/setting.sxpb
@@ -9,18 +9,8 @@
 )
 (substitution
  (special_tokens (())
-  (()
-   (alias "<start_of_turn>")
-   (candidates (())
-    "<start_of_turn>"
-    "<|im_start|>"  ; For ChatML models.
-  ))
-  (()
-   (alias "<end_of_turn>")
-   (candidates (())
-    "<end_of_turn>"
-    "<|im_end|>"  ; For ChatML models.
-  ))
+  (() (alias "<start_of_turn>"))
+  (() (alias "<end_of_turn>"))
  )
 )