Caching Problems

[CMSaiko]

So I'm running into some problems regarding the system prompt.

I can't really send the system prompt with each reply, as this causes a certain lag if the system prompt is big. It looks like it doesn't cache the system prompt either.

Example:
const messagesToSend = [ { role: "system", content: systemPrompt }, { role: "user", content: userMessage}, ];

The only thing that worked for me was to keep the whole conversation in an array and hand it over to chat.completions.create().

Example:
const messagesToSend = [ { role: "system", content: this.cachedSystemPrompt }, ...this.conversationHistory];;
const messagesPayload = { messages: messagesToSend, max_tokens: maxTokens, temperature: temperature, stream: true};

And then do:
engine.chat.completions.create(messagesPayload);

The downside to this is that at some point I would have to reload the model since it would take up too much memory. Trimming that array also doesn't work, since the cache completely loses focus, which leads to the system prompt being fully loaded in each chat.completions.create -> which also leads to lag.

How can I make the system prompt be cached the whole time without the engine loading it from scratch every time?