LLM_mechanisms.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3832f058-3452-46d1-94b9-78e79b26cd2b",
   "metadata": {},
   "source": [
    "# Understanding Transformers to Help Build Effective Large Language Model Prompts\n",
    "\n",
    "Here we will explore the mechanisms of an LLM like ChatGPT, which will allow us to gain intuition for the importance of the [key components of a good prompt](https://github.com/teaghan/educational-prompt-engineering/blob/main/key_components.ipynb).\n",
    "\n",
    "## Transformer Architecture Components and Concepts\n",
    "\n",
    "### The Attention Mechanism (Query/Key/Value (QKV)):\n",
    "\n",
    "In transformers, the attention mechanism is a key component that allows the model to focus on different parts of the input sequence when making predictions. It works by assigning weights to different elements in the input sequence based on their relevance to the current element being processed.\n",
    "\n",
    "For LLMs, you can think of an element as one word in, say, a paragraph (the sequence of elements).\n",
    "\n",
    "The attention mechanism involves three main components: Query, Key, and Value. These components are derived from the input sequence and are used to calculate attention scores. The **attention scores determine how much focus should be given to each element in the sequence when processing the current element.**\n",
    "\n",
    "The attention scores are computed using a mathematical operation called the dot product between the Query of the current element and the Key of each element in the sequence. \n",
    "The Values are pre-existing information associated with each element in the input sequence. The softmax operation is used to determine how much attention each element should receive when making predictions based on the current element (query).\n",
    "\n",
    "These scores are then scaled and passed through a softmax function to obtain a probability distribution. The final step involves taking a weighted sum of the Values, where the weights are determined by the attention scores.\n",
    "\n",
    "### Simple Explanation of Attention:\n",
    "\n",
    "Imagine you're reading a paragraph, and you want to understand each word in the context of the one you're currently looking at. The attention mechanism is like giving different levels of importance to each word based on how relevant it is to the word you're focusing on.\n",
    "\n",
    "**Components:**\n",
    "- **Query:** The current element that you are comparing against the rest of the content. Think of it as the word you're currently trying to understand.\n",
    "- **Key:** The Key is the content being examined. Think of it as the other words in the paragraph.\n",
    "- **Value:** The Value is the information associated with each Key. Think of it as the background information on what you're looking at.\n",
    "\n",
    "- **Query:** Think of the Query as the question being asked. It's like the model's way of saying, \"What should I focus on?\"\n",
    "\n",
    "**Process:**\n",
    "1. For each word in the paragraph, compare it to the word you're focusing on (Query).\n",
    "2. Calculate a score based on how relevant each word is to the word you are focusing on (dot product of Query and Key).\n",
    "3. Convert the scores into a kind of percentage (using softmax).\n",
    "4. Use these percentages as weights to access information (Values) that is relevant to each word in the paragraph. This is effectively investigating the secondary information that is relevant to the current word you are focusing on.\n",
    "\n",
    "### Predicting the Next Word\n",
    "\n",
    "The weighted sum calculated by the attention mechanism represents the context or information from the input sequence that the model has deemed relevant for predicting the next word.\n",
    "\n",
    "The weighted sum of values is combined with the internal parameters of the model. This combination is often done through feedforward neural network layers.\n",
    "\n",
    "The combined information goes through a softmax activation function, which turns the output into a probability distribution over the vocabulary. Each word in the vocabulary gets a probability score based on how likely it is to be the next word.\n",
    "\n",
    "The model then samples a word from this probability distribution (stochastic sampling) or chooses the word with the highest probability (greedy decoding). This chosen word becomes the predicted next word.\n",
    "\n",
    "## Relating Attention to the Components of a Prompt\n",
    "\n",
    "Understanding the QKV mechanism provides valuable insights into crafting effective prompts.\n",
    "\n",
    "> In essence, a well-crafted prompt mirrors the interaction between Queries, Keys, and Values in a transformer, guiding the model to focus on the right information and generate contextually appropriate responses.\n",
    "\n",
    "### 1. **Task:**\n",
    "\n",
    "In the QKV mechanism, think of the Query as the task you want the model to perform based on the relevant context (the Key). Crafting a prompt with a clear action verb initiates the task, guiding the model on what specific information to focus on and generate a response accordingly.\n",
    "\n",
    "### 2. **Context:**\n",
    "\n",
    "The context (Key) in the transformer is analogous to providing background information in a prompt. In prompts, offering relevant background information guides the model's understanding and contextualizes the task at hand. It's about presenting the right information to shape the model's responses effectively.\n",
    "\n",
    "### 3. **Including Exemplars:**\n",
    "\n",
    "In QKV terms, exemplars are comparable to examples that shape the attention of the model. When crafting a prompt, providing exemplars helps guide the model's focus on specific details (Keys) during the reasoning process, ensuring more accurate and context-aware responses.\n",
    "\n",
    "### 4. **Persona:**\n",
    "\n",
    "Defining a persona in a prompt instructs the model on how to approach the task based on a predetermined personality or role; a different persona will make different connections (Values), which help determine the type of response. \n",
    "\n",
    "### 5. **Format:**\n",
    "\n",
    "Providing a desired format will influence how the model combines information to generate a response, which will alter the associations (weighting) made with each Key.\n",
    "\n",
    "### 6. **Tone:**\n",
    "\n",
    "The tone plays a fairly similar role to the persona, causing different associations to be made that are in line with the desired tone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d82968d-f624-4781-b566-9c3c6e8abfa2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}