Do I need to provide the "chat template" or "prompt format" to llamafile ?

[edouardklein]

Hi all !

I'm using llamafile, as it's the only project that I could compile with AMD GPU support, and because @jart and the other contributors did a wonderful work and this stuff Just Works.

Now, I have trouble with the prompt format.

I have read:

• https://huggingface.co/docs/transformers/main/en/chat_templating which explains what chat templates are:

"Chat templates are part of the tokenizer. They specify how to convert conversations, represented as lists of messages, into a single tokenizable string in the format that the model expects."

and give specific information and examples.

• https://justine.lol/oneliners/ in which @jart sometimes uses prompt template elements in her calls to llamafile (e.g. [INST] or ### User:) and sometimes doesn't (search for "Digital Athena").
• https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/llama.cpp#L18930 llamafile source code, which has a function to apply the templates to the user's input.

I also have run some tests, e.g.

LD_LIBRARY_PATH=$LIBRARY_PATH  ./llamafile-0.8.16 \
    --model ~/Downloads/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
    -p '<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are an AI assistant named Digital Athena which is a large language model trained on the sum of human knowledge.<|eot_id|>' \
    --interactive --interactive-first --color  -ngl 999 --temp 0 \
    --in-suffix '<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>' \
    --in-prefix '<|start_header_id|>user<|end_header_id|>

' \
    --reverse-prompt '<|eot_id|>'

v.s.

LD_LIBRARY_PATH=$LIBRARY_PATH  ./llamafile-0.8.16 \
    --model ~/Downloads/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
-p '
The following is a conversation between a Researcher and their helpful AI assistant Digital Athena which is a large language model trained on the sum of human knowledge.
Researcher: Good morning.
Digital Athena: How can I help you today?
Researcher:' \
--in-prefix ' ' --in-suffix 'Digital Athena:' --reverse-prompt 'Researcher:' \
 --interactive --interactive-first --color  -ngl 999 --temp 0

In this particular case, the version with the prompt format elements yielded the perfect answer to my go-to question ("What would you say are the 4 most important concepts defined and used by Pierre Bourdieu ?"), whereas the second version, without the prompt format yielded a close, but not perfect answer (it forgot "symbolic violence").

So I'm basically lost:

• The code to add the prompt template is hard coded in llama.cpp
• but the author herself is sometimes putting it in via command line arguments
• and doing so seem to yield a slight improvement in quality, but my test is quick and dirty.

To make matters worse, when I use a web ui on top of llamafile, I get the <|eot_id|> string at the end of each response (#630 ) and I don't know if it is a related problem.

If anybody knows what's what, or even have the slightest idea of what's going on, I'd love to hear it !

Thanks :)