The purpose of this project was to experiment with LLM finetuning to see if current state-of-the-art models can be adequately trained to mimic the stream of consciousness generated by humans. Furthermore, would an LLM trained to mimic a stream of consciousness lead to any interesting emergent phenomena such as some development of a sense of self or identity?
To train the model, roughly 450 sentences of my personal streams of consciousness were gathered manually over the course of several days and hours of typing. These sentences were then split into 4 sentence chunks chronologically, which formed the "response" portion of the prompt-response pairs to finetune the model. Each 4 sentence chunk was then examined by GPT 3 to produce a summary of what was thought about which was then crafted into a prompt of the form "Think about...". Over 100 of these prompt-response pairs were generated in total.
The chosen model for fine-tuning was the OpenAI Davinci model. The process of fine-tuning the model cost approximately $13. After the fine-tuning process, a personal model was developed that could be used to generate new thoughts.
Unsurprisingly, the model lost coherence after about 4 sentences since this represents a distribution shift from what it was trained on, but it managed to generate some consistently thought-provoking responses in short formats. However, the model was not very successful at thinking about topics outside it's training data, which ended up being primarily existential observations.
One unforeseen result is that the model occasionally but consistently assumed my identity, calling itself Sevan. In fact, many of the thoughts it generated were more personal to me than I would have expected. Friends and family commented that it actually sounded like me. As a result of this, I began a new project training an LLM on over 10,000 prompt-response pairs of actual text messages I had with friends. However, after initial results lacked promise I put the project on hold.
It is my opinion that models of today are not capable of generating realistic streams of consciousness, even with improvements in the prompt-response pairs. However, this marks the early stage of stream of consciousness LLMs, and I foresee remarkably realistic thought patterns being generated just one or two iterations down the line.
This project is licensed under the MIT License - see the LICENSE file for details