Passing chat history to LLM

[pshekhar-esri]

Novice user here, just want to confirm if I'm thinking about this correctly -- when using the ChatHistoy object, the context of the conversation is maintained even if the history variable is not directly passed through the YAML/system prompt, as long as queries + responses are appended to the object. I'm assuming that chat history is still passed to the LLM behind the scenes and therefore the size of the history does contribute to token usage - is this correct?

[sophialagerkranspandey]

Would you mind providing some code snippets of what you're working on and referencing in the questions above? We'd like to understand further before replying.

[pshekhar-esri]

Thank you for getting back to me! Not sure if I'm allowed to share code, but can provide more context:

We have a RAG enabled chat-bot with instructions to always answer user queries based on the knowledge provided and to consider the chat history when responding to queries. These instructions are provided as part of a YAML prompt. There's also a global chat history object (Microsoft.SemanticKernel.ChatCompletion.ChatHistory) to which we append all user queries and assistant responses (including tool calls etc.) at every turn of the conversation. The prompt itself does not contain any chat history passed in as a string/variable so I wanted to know about how it contributes to token usage.

[sophialagerkranspandey]

Everything you send in the request contributes to your token usage. We'd recommend a strategy, such as using the most recent message to help keep your token usage low as opposed to sending the entire chat history.

[pshekhar-esri]

Understood, thank you so much for clarifying!