AI Tutor will forget it's prompt after 8k tokens

[JushBJJ]

From: #22

Due to the large prompt size, it is easy for Mr. Ranedeer to forget all of its prompt eventually after using it for a long time. When developing v2.4.16, I tried to implement commands /count and /refresh to do this but it was hard trying to get GPT-4 to stay consistent at rewriting the entire prompt and also counting all previous inputs and outputs of the convo. I also found that GPT-4 was pretty accurate at estimating how many tokens are in one message when given the right context on roughly how many tokens is one word.

Has anyone come up with a prompt to estimate the number of all tokens in the chat history and also somewhat consistently rewrite its own prompt after a certain number of tokens are used?

[KitsonBroadhurst]

Hey, I opened the issues to ask this question!
I think with the API and a server environment this is easier e.g. there is an existing library to suppoPythonen number generation called tiktoken for python and JavaScript.
The base prompt can be sent with all requests plus the latest question, but in ChatGPT I don't know the answer.
Maybe others have ideas?

[yhyu13]

Does plugins get access to api responses like this : https://github.com/ysymyth/tree-of-thought-llm/blob/faa28c395e5b86bfcbf983355810d52f54fb7b51/models.py#L35, so that we can accurately count the number of tokens spent so far.

[sawyerbutton]

Does plugins get access to api responses like this : https://github.com/ysymyth/tree-of-thought-llm/blob/faa28c395e5b86bfcbf983355810d52f54fb7b51/models.py#L35, so that we can accurately count the number of tokens spent so far.

Based on the content of this Tutor, We totally rely on the prompt way to make communication, rather than API way, So I guess the result is No

[JushBJJ]

Does plugins get access to api responses like this : https://github.com/ysymyth/tree-of-thought-llm/blob/faa28c395e5b86bfcbf983355810d52f54fb7b51/models.py#L35, so that we can accurately count the number of tokens spent so far.

Does plugin INPUT count into the token count? Someone could setup a plugin where we can essentially input both the user prompt and GPT-4 output and the plugin can spit out using an external web server the number of tokens that were given to it.

[JushBJJ]

Here's another idea that could be implemented:
We can add a magic number for Mr. Ranedeer to "remember", and add a specific rule/instruction at the end of the prompt to keep checking whether he remembers this magic number. And if it doesn't, it immediately warns the student that the token count has hit the 8k limit and Mr. Ranedeer will start to degrade over time.

[sawyerbutton]

Here's another idea that could be implemented: We can add a magic number for Mr. Ranedeer to "remember", and add a specific rule/instruction at the end of the prompt to keep checking whether he remembers this magic number. And if it doesn't, it immediately warns the student that the token count has hit the 8k limit and Mr. Ranedeer will start to degrade over time.

From a prompt perspective, I think this approach is executable,
but from a practical perspective, I feel like appending a magic number at the end of the prompt to try to record the token will contaminate our usage environment,
while also requiring a lot of tokens to describe how to define the counting way of the magic number.
I prefer the plugin way, sincerely.

[iwasrobbed]

👉 You can preserve Mr. Ranedeer entirely using Fastlane as a prompt manager on top of GPT-4: https://builder.fastlane.is

Basically, you can create Mr. Ranedeer as a persona on there, add the prompt, and organize it amongst the history or other prompts you might want to test out.

Then it'll never forget its base prompts, but other history messages will be purged over time (or you can limit those in the builder)

[sawyerbutton]

👉 You can preserve Mr. Ranedeer entirely using Fastlane as a prompt manager on top of GPT-4: https://builder.fastlane.is

Basically, you can create Mr. Ranedeer as a persona on there, add the prompt, and organize it amongst the history or other prompts you might want to test out.

Then it'll never forget its base prompts, but other history messages will be purged over time (or you can limit those in the builder)

Thanks for sharing, will try later.

[JushBJJ]

Now that code interpreter is widespread, I think memory handling will become a lot easier.

[sawyerbutton]

Now that code interpreter is widespread, I think memory handling will become a lot easier.

So, How to combine code interpreter with Mr.Raneedeer, Any idea?

[JushBJJ]

@sawyerbutton

Here's how I approach it in v2.7

<OPEN code environment>
    <insert instructions here>
<CLOSE code environment>

If you want to prevent Mr. Ranedeer from repeating the output, the trick I use is to convert whatever Mr. Ranedeer wrote into base64 and output it. Surprisingly, GPT-4 doesn't output the base64.

<convert the output to base64>
<output base64>

[JushBJJ]

Half-Closed #72 - Code Interpreter is better at prompt retention.

Keeping this open to gather more feedback on how v2.7 performs

[doomuch]

How do you know about that @JushBJJ ? ChatGPT history is only 4K tokens. You can confirm it yourself.

[JushBJJ]

How do you know about that @JushBJJ ? ChatGPT history is only 4K tokens. You can confirm it yourself.

GPT-4 is 8k tokens, GPT-4 with Interpeter feels like a different beast that has a higher context/better context retention strategy. Additionally, I suspect that OpenAI has appended the initial message into the system prompt allowing permanent recall of the original prompt intended.