GPT2_instruction_finetuning_from_scratch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "12e91914-5f51-43fa-b65b-625e73b4d17b",
      "metadata": {
        "id": "12e91914-5f51-43fa-b65b-625e73b4d17b"
      },
      "source": [
        "<table style=\"width:100%\">\n",
        "<tr>\n",
        "<td style=\"vertical-align:middle; text-align:left;\">\n",
        "<font size=\"2\">\n",
        "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
        "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
        "</font>\n",
        "</td>\n",
        "<td style=\"vertical-align:middle; text-align:left;\">\n",
        "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
        "</td>\n",
        "</tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf",
      "metadata": {
        "id": "c2520ec3-722f-4f44-bdd1-885b13e7afbf"
      },
      "source": [
        "# Chapter 7: Finetuning To Follow Instructions"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tiktoken"
      ],
      "metadata": {
        "id": "Y27tx970KFva",
        "outputId": "c40c54cd-f67b-42a2-d3c3-8f029aca2a4f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "id": "Y27tx970KFva",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tiktoken\n",
            "  Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2024.5.15)\n",
            "Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2.31.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken) (2024.6.2)\n",
            "Installing collected packages: tiktoken\n",
            "Successfully installed tiktoken-0.7.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "4e19327b-6c02-4881-ad02-9b6d3ec0b1b4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4e19327b-6c02-4881-ad02-9b6d3ec0b1b4",
        "outputId": "b3aca4ba-12d6-4a5e-f05d-cb7c9014bd56"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "matplotlib version: 3.7.1\n",
            "tiktoken version: 0.7.0\n",
            "torch version: 2.3.0+cu121\n",
            "tqdm version: 4.66.4\n",
            "tensorflow version: 2.15.0\n"
          ]
        }
      ],
      "source": [
        "from importlib.metadata import version\n",
        "\n",
        "pkgs = [\n",
        "    \"matplotlib\",  # Plotting library\n",
        "    \"tiktoken\",    # Tokenizer\n",
        "    \"torch\",       # Deep learning library\n",
        "    \"tqdm\",        # Progress bar\n",
        "    \"tensorflow\",  # For OpenAI's pretrained weights\n",
        "]\n",
        "for p in pkgs:\n",
        "    print(f\"{p} version: {version(p)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "264fca98-2f9a-4193-b435-2abfa3b4142f",
      "metadata": {
        "id": "264fca98-2f9a-4193-b435-2abfa3b4142f"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/overview.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813",
      "metadata": {
        "id": "8bbc68e9-75b3-41f1-ac2c-e071c3cd0813"
      },
      "source": [
        "## 7.1 Introduction to instruction finetuning"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab",
      "metadata": {
        "id": "53dba24a-6805-496c-9a7f-c75e2d3527ab"
      },
      "source": [
        "- In chapter 5, we saw that pretraining an LLM involves a training procedure where it learns to generate one word at a time\n",
        "- Hence, a pretrained LLM is good at text completion, but it is not good at following instructions\n",
        "- In this chapter, we teach the LLM to follow instructions better"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18dc0535-0904-44ed-beaf-9b678292ef35",
      "metadata": {
        "id": "18dc0535-0904-44ed-beaf-9b678292ef35"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/instruction-following.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4698b23-12e0-4bd7-a140-ccb3dd71d4e8",
      "metadata": {
        "id": "b4698b23-12e0-4bd7-a140-ccb3dd71d4e8"
      },
      "source": [
        "- The topics covered in this chapter are summarized in the figure below\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-1.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86",
      "metadata": {
        "id": "5384f0cf-ef3c-4436-a5fa-59bd25649f86"
      },
      "source": [
        "## 7.2 Preparing a dataset for supervised instruction finetuning"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f8b34ff8-619f-4e89-bd03-ce513269760d",
      "metadata": {
        "id": "f8b34ff8-619f-4e89-bd03-ce513269760d"
      },
      "source": [
        "- We will work with an instruction dataset I prepared for this chapter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "0G3axLw6kY1N",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0G3axLw6kY1N",
        "outputId": "66ca609f-342c-43cf-805d-ca1fb09cdb3d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of entries: 1100\n"
          ]
        }
      ],
      "source": [
        "import json\n",
        "import os\n",
        "import urllib\n",
        "\n",
        "\n",
        "def download_and_load_file(file_path, url):\n",
        "\n",
        "    if not os.path.exists(file_path):\n",
        "        with urllib.request.urlopen(url) as response:\n",
        "            text_data = response.read().decode(\"utf-8\")\n",
        "        with open(file_path, \"w\", encoding=\"utf-8\") as file:\n",
        "            file.write(text_data)\n",
        "    else:\n",
        "        with open(file_path, \"r\", encoding=\"utf-8\") as file:\n",
        "            text_data = file.read()\n",
        "\n",
        "    with open(file_path, \"r\") as file:\n",
        "        data = json.load(file)\n",
        "\n",
        "    return data\n",
        "\n",
        "\n",
        "file_path = \"instruction-data.json\"\n",
        "url = \"https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch07/01_main-chapter-code/instruction-data.json\"\n",
        "\n",
        "data = download_and_load_file(file_path, url)\n",
        "print(\"Number of entries:\", len(data))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d7af8176-4255-4e92-8c7d-998771733eb8",
      "metadata": {
        "id": "d7af8176-4255-4e92-8c7d-998771733eb8"
      },
      "source": [
        "- Each item in the `data` list we loaded from the JSON file above is a dictionary in the following form"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "-LiuBMsHkzQV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-LiuBMsHkzQV",
        "outputId": "06d8279b-702a-4ace-97cd-887fbbe0b250"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example entry:\n",
            " {'instruction': 'Identify the correct spelling of the following word.', 'input': 'Ocassion', 'output': \"The correct spelling is 'Occasion.'\"}\n"
          ]
        }
      ],
      "source": [
        "print(\"Example entry:\\n\", data[50])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46",
      "metadata": {
        "id": "c5a32b34-485a-4816-a77a-da14f9fe6e46"
      },
      "source": [
        "- Note that the `'input'` field can be empty:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "uFInFxDDk2Je",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uFInFxDDk2Je",
        "outputId": "1be0adb2-a4ca-46e4-da6a-c7a31d8aa68a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Another example entry:\n",
            " {'instruction': \"What is an antonym of 'complicated'?\", 'input': '', 'output': \"An antonym of 'complicated' is 'simple'.\"}\n"
          ]
        }
      ],
      "source": [
        "print(\"Another example entry:\\n\", data[999])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f034799a-6575-45fd-98c9-9d1012d0fd58",
      "metadata": {
        "id": "f034799a-6575-45fd-98c9-9d1012d0fd58"
      },
      "source": [
        "- Instruction finetuning is often referred to as \"supervised instruction finetuning\" because it involves training a model on a dataset where the input-output pairs are explicitly provided\n",
        "- There are different ways to format the entries as inputs to the LLM; the figure below illustrates two example formats that were used for training the Alpaca (https://crfm.stanford.edu/2023/03/13/alpaca.html) and Phi-3 (https://arxiv.org/abs/2404.14219) LLMs, respectively"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10",
      "metadata": {
        "id": "dffa4f70-44d4-4be4-89a9-2159f4885b10"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/prompt-style.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6",
      "metadata": {
        "id": "dd79a74e-befb-491c-be49-f777a6a5b6a6"
      },
      "source": [
        "- In this chapter, we use Alpaca-style prompt formatting, which was the original prompt template for instruction finetuning\n",
        "- Below, we format the input that we will pass as input to the LLM"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "Jhk37nnJnkBh",
      "metadata": {
        "id": "Jhk37nnJnkBh"
      },
      "outputs": [],
      "source": [
        "def format_input(entry):\n",
        "    instruction_text = (\n",
        "        f\"Below is an instruction that describes a task. \"\n",
        "        f\"Write a response that appropriately completes the request.\"\n",
        "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
        "    )\n",
        "\n",
        "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
        "\n",
        "    return instruction_text + input_text"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6",
      "metadata": {
        "id": "011e78b4-e89a-4653-a2ee-7b2739ca04d6"
      },
      "source": [
        "- A formatted response with input field looks like as shown below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "F9UQRfjzo4Js",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F9UQRfjzo4Js",
        "outputId": "bd3b3af9-6201-41fd-b631-54502925d179"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "Identify the correct spelling of the following word.\n",
            "\n",
            "### Input:\n",
            "Ocassion\n",
            "\n",
            "### Response:\n",
            "The correct spelling is 'Occasion.'\n"
          ]
        }
      ],
      "source": [
        "model_input = format_input(data[50])\n",
        "desired_response = f\"\\n\\n### Response:\\n{data[50]['output']}\"\n",
        "\n",
        "print(model_input + desired_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c",
      "metadata": {
        "id": "4dc93ddf-431c-49c0-96f2-fb3a79c4d94c"
      },
      "source": [
        "- Below is a formatted response without an input field"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a3891fa9-f738-41cd-946c-80ef9a99c346",
        "outputId": "3d7ad8e3-e617-4186-a13c-95288ea0e43a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "What is an antonym of 'complicated'?\n",
            "\n",
            "### Response:\n",
            "An antonym of 'complicated' is 'simple'.\n"
          ]
        }
      ],
      "source": [
        "model_input = format_input(data[999])\n",
        "desired_response = f\"\\n\\n### Response:\\n{data[999]['output']}\"\n",
        "\n",
        "print(model_input + desired_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4aa8afd5-2a21-49a5-90c3-6a03865a4771",
      "metadata": {
        "id": "4aa8afd5-2a21-49a5-90c3-6a03865a4771"
      },
      "source": [
        "- Lastly, before we prepare the PyTorch data loaders in the next section, we divide the dataset into a training, validation, and test set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "aFZVopbIlNfx",
      "metadata": {
        "id": "aFZVopbIlNfx"
      },
      "outputs": [],
      "source": [
        "train_portion = int(len(data) * 0.85)  # 85% for training\n",
        "test_portion = int(len(data) * 0.1)   # 10% for testing\n",
        "val_portion = len(data) - train_portion - test_portion  # Remaining 5% for validation\n",
        "\n",
        "train_data = data[:train_portion]\n",
        "test_data = data[train_portion:train_portion + test_portion]\n",
        "val_data = data[train_portion + test_portion:]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "-zf6oht6bIUQ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-zf6oht6bIUQ",
        "outputId": "dce4c87d-f823-4e61-d5da-255d6464082c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training set length: 935\n",
            "Validation set length: 55\n",
            "Test set length: 110\n"
          ]
        }
      ],
      "source": [
        "print(\"Training set length:\", len(train_data))\n",
        "print(\"Validation set length:\", len(val_data))\n",
        "print(\"Test set length:\", len(test_data))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fcaaf606-f913-4445-8301-632ae10d387d",
      "metadata": {
        "id": "fcaaf606-f913-4445-8301-632ae10d387d"
      },
      "source": [
        "## 7.3 Organizing data into training batches"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "233f63bd-9755-4d07-8884-5e2e5345cf27",
      "metadata": {
        "id": "233f63bd-9755-4d07-8884-5e2e5345cf27"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-2.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c149fc1a-7757-4ec8-80cb-e2a3fb007a2c",
      "metadata": {
        "id": "c149fc1a-7757-4ec8-80cb-e2a3fb007a2c"
      },
      "source": [
        "- We tackle this dataset batching in several steps, as summarized in the figure below\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/detailed-batching.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b9af423f-aad9-4b3c-bea5-153021c04862",
      "metadata": {
        "id": "b9af423f-aad9-4b3c-bea5-153021c04862"
      },
      "source": [
        "- First, we implement an `InstructionDataset` class that pre-tokenizes all inputs in the dataset, similar to the `SpamDataset` in chapter 6\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/pretokenizing.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "adc29dc4-f1c7-4c71-937b-95119d6239bb",
      "metadata": {
        "id": "adc29dc4-f1c7-4c71-937b-95119d6239bb"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch.utils.data import Dataset\n",
        "\n",
        "\n",
        "class InstructionDataset(Dataset):\n",
        "    def __init__(self, data, tokenizer):\n",
        "        self.data = data\n",
        "\n",
        "        # Pre-tokenize texts\n",
        "        self.encoded_texts = []\n",
        "        for entry in data:\n",
        "            instruction_plus_input = format_input(entry)\n",
        "            response_text = f\"\\n\\n### Response:\\n{entry['output']}\"\n",
        "            full_text = instruction_plus_input + response_text\n",
        "            self.encoded_texts.append(\n",
        "                tokenizer.encode(full_text)\n",
        "            )\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        return self.encoded_texts[index]\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "384f0e69-4b22-41c0-a25d-f077527eddd1",
      "metadata": {
        "id": "384f0e69-4b22-41c0-a25d-f077527eddd1"
      },
      "source": [
        "- Similar to chapter 6, we want to collect multiple training examples in a batch to accelerate training; this requires padding all inputs to a similar length\n",
        "- Also similar to the previous chapter, we use the `<|endoftext|>` token as a padding token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "ff24fe1a-5746-461c-ad3d-b6d84a1a7c96",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ff24fe1a-5746-461c-ad3d-b6d84a1a7c96",
        "outputId": "f99cf73f-ca94-40dd-d52d-a0870cff6e93"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[50256]\n"
          ]
        }
      ],
      "source": [
        "import tiktoken\n",
        "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
        "\n",
        "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427",
      "metadata": {
        "id": "9e5bd7bc-f347-4cf8-a0c2-94cb8799e427"
      },
      "source": [
        "- In chapter 6, we padded all examples in a dataset to the same length\n",
        "  - Here, we take a more sophisticated approach and develop a custom \"collate\" function that we can pass to the data loader\n",
        "  - This custom collate function pads the training examples in each batch to have the same length (but different batches can have different lengths)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3",
      "metadata": {
        "id": "65c4d943-4aa8-4a44-874e-05bc6831fbd3"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/padding.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "eb4c77dd-c956-4a1b-897b-b466909f18ca",
      "metadata": {
        "id": "eb4c77dd-c956-4a1b-897b-b466909f18ca"
      },
      "outputs": [],
      "source": [
        "def custom_collate_draft_1(\n",
        "    batch,\n",
        "    pad_token_id=50256,\n",
        "    device=\"cpu\"\n",
        "):\n",
        "    # Find the longest sequence in the batch\n",
        "    batch_max_length = max(len(item)+1 for item in batch)\n",
        "\n",
        "    # Pad and prepare inputs\n",
        "    inputs_lst = []\n",
        "\n",
        "    for item in batch:\n",
        "        new_item = item.copy()\n",
        "        # Add an <|endoftext|> token\n",
        "        new_item += [pad_token_id]\n",
        "        # Pad sequences to max_length\n",
        "        # this always adds at least 1 additional padding tokens\n",
        "        padded = new_item + [pad_token_id] * (batch_max_length - len(new_item))\n",
        "        # We remove this extra padded token again here\n",
        "        inputs = torch.tensor(padded[:-1])\n",
        "        inputs_lst.append(inputs)\n",
        "\n",
        "    # Convert list of inputs to tensor and transfer to target device\n",
        "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
        "    return inputs_tensor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "8fb02373-59b3-4f3a-b1d1-8181a2432645",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8fb02373-59b3-4f3a-b1d1-8181a2432645",
        "outputId": "3f1022f1-b56f-4d1e-c0d9-067b8b9b915b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[    0,     1,     2,     3,     4],\n",
            "        [    5,     6, 50256, 50256, 50256],\n",
            "        [    7,     8,     9, 50256, 50256]])\n"
          ]
        }
      ],
      "source": [
        "inputs_1 = [0, 1, 2, 3, 4]\n",
        "inputs_2 = [5, 6]\n",
        "inputs_3 = [7, 8, 9]\n",
        "\n",
        "batch = (\n",
        "    inputs_1,\n",
        "    inputs_2,\n",
        "    inputs_3\n",
        ")\n",
        "\n",
        "print(custom_collate_draft_1(batch))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c46832ab-39b7-45f8-b330-ac9adfa10d1b",
      "metadata": {
        "id": "c46832ab-39b7-45f8-b330-ac9adfa10d1b"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/batching-step-4.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "17769a19-b961-4213-92ef-34f441b2d1d6",
      "metadata": {
        "id": "17769a19-b961-4213-92ef-34f441b2d1d6"
      },
      "source": [
        "- Above, we only returned the inputs to the LLM; however, for LLM training, we also need the target values\n",
        "- Similar to pretraining an LLM, the targets are the inputs shifted by 1 position to the right, so the LLM learns to predict the next token"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef",
      "metadata": {
        "id": "0386b6fe-3455-4e70-becd-a5a4681ba2ef"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/inputs-targets.webp\" width=400px>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "74af192e-757c-4c0a-bdf9-b7eb25bf6ebc",
      "metadata": {
        "id": "74af192e-757c-4c0a-bdf9-b7eb25bf6ebc"
      },
      "outputs": [],
      "source": [
        "def custom_collate_draft_2(\n",
        "    batch,\n",
        "    pad_token_id=50256,\n",
        "    device=\"cpu\"\n",
        "):\n",
        "    # Find the longest sequence in the batch\n",
        "    batch_max_length = max(len(item)+1 for item in batch)\n",
        "\n",
        "    # Pad and prepare inputs\n",
        "    inputs_lst, targets_lst = [], []\n",
        "\n",
        "    for item in batch:\n",
        "        new_item = item.copy()\n",
        "        # Add an <|endoftext|> token\n",
        "        new_item += [pad_token_id]\n",
        "        # Pad sequences to max_length\n",
        "        padded = new_item + [pad_token_id] * (batch_max_length - len(new_item))\n",
        "        inputs = torch.tensor(padded[:-1])  # Truncate the last token for inputs\n",
        "        targets = torch.tensor(padded[1:])  # Shift +1 to the right for targets\n",
        "        inputs_lst.append(inputs)\n",
        "        targets_lst.append(targets)\n",
        "\n",
        "    # Convert list of inputs to tensor and transfer to target device\n",
        "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
        "    targets_tensor = torch.stack(targets_lst).to(device)\n",
        "    return inputs_tensor, targets_tensor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "6eb2bce3-28a7-4f39-9d4b-5e972d69066c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6eb2bce3-28a7-4f39-9d4b-5e972d69066c",
        "outputId": "6d34763d-b58d-4417-e9cf-3665eb9c9a18"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[    0,     1,     2,     3,     4],\n",
            "        [    5,     6, 50256, 50256, 50256],\n",
            "        [    7,     8,     9, 50256, 50256]])\n",
            "tensor([[    1,     2,     3,     4, 50256],\n",
            "        [    6, 50256, 50256, 50256, 50256],\n",
            "        [    8,     9, 50256, 50256, 50256]])\n"
          ]
        }
      ],
      "source": [
        "inputs, targets = custom_collate_draft_2(batch)\n",
        "print(inputs)\n",
        "print(targets)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3bf85703-a0e0-42aa-8f29-cbc28dbf4e15",
      "metadata": {
        "id": "3bf85703-a0e0-42aa-8f29-cbc28dbf4e15"
      },
      "source": [
        "- Next, we introduce an `ignore_index` value to replace all padding token IDs with a new value; the purpose of this `ignore_index` is that we can ignore padding values in the loss function (more on that later)\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/batching-step-5.webp\" width=500px>\n",
        "\n",
        "- Concretely, this means that we replace the token IDs corresponding to `50256` with `-100` as illustrated below"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bd4bed33-956e-4b3f-a09c-586d8203109a",
      "metadata": {
        "id": "bd4bed33-956e-4b3f-a09c-586d8203109a"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/ignore-index.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5346513e-c3f4-44fe-af22-4ebd36497728",
      "metadata": {
        "id": "5346513e-c3f4-44fe-af22-4ebd36497728"
      },
      "source": [
        "- (In addition, we also introduce the `allowed_max_length` in case we want to limit the length of the samples; this will be useful if you plan to work with your own datasets that are longer than the 1024 token context size supported by the GPT-2 model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "41ec6e2d-9eb2-4124-913e-d2af39be4cf2",
      "metadata": {
        "id": "41ec6e2d-9eb2-4124-913e-d2af39be4cf2"
      },
      "outputs": [],
      "source": [
        "def custom_collate_fn(\n",
        "    batch,\n",
        "    pad_token_id=50256,\n",
        "    ignore_index=-100,\n",
        "    allowed_max_length=None,\n",
        "    device=\"cpu\"\n",
        "):\n",
        "    # Find the longest sequence in the batch\n",
        "    batch_max_length = max(len(item)+1 for item in batch)\n",
        "\n",
        "    # Pad and prepare inputs and targets\n",
        "    inputs_lst, targets_lst = [], []\n",
        "\n",
        "    for item in batch:\n",
        "        new_item = item.copy()\n",
        "        # Add an <|endoftext|> token\n",
        "        new_item += [pad_token_id]\n",
        "        # Pad sequences to max_length\n",
        "        padded = new_item + [pad_token_id] * (batch_max_length - len(new_item))\n",
        "        inputs = torch.tensor(padded[:-1])  # Truncate the last token for inputs\n",
        "        targets = torch.tensor(padded[1:])  # Shift +1 to the right for targets\n",
        "\n",
        "        # New: Replace all but the first padding tokens in targets by ignore_index\n",
        "        mask = targets == pad_token_id\n",
        "        indices = torch.nonzero(mask).squeeze()\n",
        "        if indices.numel() > 1:\n",
        "            targets[indices[1:]] = ignore_index\n",
        "\n",
        "        # New: Optionally truncate to maximum sequence length\n",
        "        if allowed_max_length is not None:\n",
        "            inputs = inputs[:allowed_max_length]\n",
        "            targets = targets[:allowed_max_length]\n",
        "\n",
        "        inputs_lst.append(inputs)\n",
        "        targets_lst.append(targets)\n",
        "\n",
        "    # Convert list of inputs and targets to tensors and transfer to target device\n",
        "    inputs_tensor = torch.stack(inputs_lst).to(device)\n",
        "    targets_tensor = torch.stack(targets_lst).to(device)\n",
        "\n",
        "    return inputs_tensor, targets_tensor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "cdf5eec4-9ebe-4be0-9fca-9a47bee88fdc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cdf5eec4-9ebe-4be0-9fca-9a47bee88fdc",
        "outputId": "e2eb7df4-d891-4e62-a770-3164661dea89"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[    0,     1,     2,     3,     4],\n",
            "        [    5,     6, 50256, 50256, 50256],\n",
            "        [    7,     8,     9, 50256, 50256]])\n",
            "tensor([[    1,     2,     3,     4, 50256],\n",
            "        [    6, 50256,  -100,  -100,  -100],\n",
            "        [    8,     9, 50256,  -100,  -100]])\n"
          ]
        }
      ],
      "source": [
        "inputs, targets = custom_collate_fn(batch)\n",
        "print(inputs)\n",
        "print(targets)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "26727c90-0d42-43b3-af21-0a66ad4fbbc7",
      "metadata": {
        "id": "26727c90-0d42-43b3-af21-0a66ad4fbbc7"
      },
      "source": [
        "- Let's see what this replacement by -100 accomplishes\n",
        "- For illustration purposes, let's assume we have a small classification task with 2 class labels, 0 and 1, similar to chapter 6\n",
        "- If we have the following logits values (outputs of the last layer of the model), we calculate the following loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "W2jvh-OP9MFV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W2jvh-OP9MFV",
        "outputId": "d48c0184-af98-491b-fe30-939c2e09f1a2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(1.1269)\n"
          ]
        }
      ],
      "source": [
        "logits_1 = torch.tensor(\n",
        "    [[-1.0, 1.0],  # 1st training example\n",
        "     [-0.5, 1.5]]  # 2nd training example\n",
        ")\n",
        "targets_1 = torch.tensor([0, 1])\n",
        "\n",
        "\n",
        "loss_1 = torch.nn.functional.cross_entropy(logits_1, targets_1)\n",
        "print(loss_1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5edd3244-8886-4505-92e9-367d28529e1e",
      "metadata": {
        "id": "5edd3244-8886-4505-92e9-367d28529e1e"
      },
      "source": [
        "- Now, adding one more training example will, as expected, influence the loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "nvVMuil89v9N",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nvVMuil89v9N",
        "outputId": "2ffbbf15-99e1-45a8-933f-7ca363566d3d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(0.7936)\n"
          ]
        }
      ],
      "source": [
        "logits_2 = torch.tensor(\n",
        "    [[-1.0, 1.0],\n",
        "     [-0.5, 1.5],\n",
        "     [-0.5, 1.5]]  # New 3rd training example\n",
        ")\n",
        "targets_2 = torch.tensor([0, 1, 1])\n",
        "\n",
        "loss_2 = torch.nn.functional.cross_entropy(logits_2, targets_2)\n",
        "print(loss_2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "54dca331-40e0-468b-b690-189fe156ba8f",
      "metadata": {
        "id": "54dca331-40e0-468b-b690-189fe156ba8f"
      },
      "source": [
        "- Let's see what happens if we replace the class label of one of the examples with -100"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "RTyB1vah9p56",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RTyB1vah9p56",
        "outputId": "98625d8d-fcc5-4d9b-9615-7bb36dfa772e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(1.1269)\n",
            "loss_1 == loss_3: tensor(True)\n"
          ]
        }
      ],
      "source": [
        "targets_3 = torch.tensor([0, 1, -100])\n",
        "\n",
        "loss_3 = torch.nn.functional.cross_entropy(logits_2, targets_3)\n",
        "print(loss_3)\n",
        "print(\"loss_1 == loss_3:\", loss_1 == loss_3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cef09d21-b652-4760-abea-4f76920e6a25",
      "metadata": {
        "id": "cef09d21-b652-4760-abea-4f76920e6a25"
      },
      "source": [
        "- As we can see, the resulting loss on these 3 training examples is the same as the loss we calculated from the 2 training examples, which means that the cross entropy loss function ignored the training example with the -100 label\n",
        "- By default, PyTorch has the `cross_entropy(..., ignore_index=-100)` setting to ignore examples corresponding to the label -100\n",
        "- Using this -100 `ignore_index`, we can ignore the additional end-of-text (padding) tokens in the batches that we used to pad the training examples to equal length\n",
        "- However, we don't want to ignore the first instance of the end-of-text (padding) token (50256) because it can help signal to the LLM when the response is complete"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6a4e9c5f-7c49-4321-9f1b-a50468a84524",
      "metadata": {
        "id": "6a4e9c5f-7c49-4321-9f1b-a50468a84524"
      },
      "source": [
        "- In practice, it is also common to mask out the target token IDs that correspond to the instruction, as illustrated in the figure below (this is a recommended reader exercise after completing the chapter)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39",
      "metadata": {
        "id": "fab8f0ed-80e8-4fd9-bf84-e5d0e0bc0a39"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/mask-instructions.webp\" width=600px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bccaf048-ec95-498c-9155-d5b3ccba6c96",
      "metadata": {
        "id": "bccaf048-ec95-498c-9155-d5b3ccba6c96"
      },
      "source": [
        "## 7.4 Creating data loaders for an instruction dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e6b8e656-3af3-4db6-8dde-d8c216a12f50",
      "metadata": {
        "id": "e6b8e656-3af3-4db6-8dde-d8c216a12f50"
      },
      "source": [
        "- In this section, we use the `InstructionDataset` class and `custom_collate_fn` function to instantiate the training, validation, and test data loaders"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9fffe390-b226-4d5c-983f-9f4da773cb82",
      "metadata": {
        "id": "9fffe390-b226-4d5c-983f-9f4da773cb82"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-3.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "932677e9-9317-42e8-b461-7b0269518f97",
      "metadata": {
        "id": "932677e9-9317-42e8-b461-7b0269518f97"
      },
      "source": [
        "- Another additional detail of the previous `custom_collate_fn` function is that we now directly move the data to the target device (e.g., GPU) instead of doing it in the main training loop, which improves efficiency because it can be carried out as a background process when we use the  `custom_collate_fn` as part of the data loader\n",
        "- Using the `partial` function from Python's `functools` standard library, we create a new function with the `device` argument  of the original function pre-filled"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "etpqqWh8phKc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "etpqqWh8phKc",
        "outputId": "e7a0ed2a-e82e-4748-9d38-12ec81c5e1ab"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Device: cuda\n"
          ]
        }
      ],
      "source": [
        "from functools import partial\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(\"Device:\", device)\n",
        "\n",
        "customized_collate_fn = partial(custom_collate_fn, device=device, allowed_max_length=1024)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8ff42c29-8b81-45e5-ae8d-b97cd1cf447a",
      "metadata": {
        "id": "8ff42c29-8b81-45e5-ae8d-b97cd1cf447a"
      },
      "source": [
        "- Next, we instantiate the data loaders similar to previous chapters, except that we now provide our own collate function for the batching process"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "BtWkgir6Hlpe",
      "metadata": {
        "id": "BtWkgir6Hlpe"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "\n",
        "num_workers = 0\n",
        "batch_size = 8\n",
        "\n",
        "torch.manual_seed(123)\n",
        "\n",
        "train_dataset = InstructionDataset(train_data, tokenizer)\n",
        "train_loader = DataLoader(\n",
        "    train_dataset,\n",
        "    batch_size=batch_size,\n",
        "    collate_fn=customized_collate_fn,\n",
        "    shuffle=True,\n",
        "    drop_last=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "1d097dc8-ad34-4f05-b435-e4147965f532",
      "metadata": {
        "id": "1d097dc8-ad34-4f05-b435-e4147965f532"
      },
      "outputs": [],
      "source": [
        "val_dataset = InstructionDataset(val_data, tokenizer)\n",
        "val_loader = DataLoader(\n",
        "    val_dataset,\n",
        "    batch_size=batch_size,\n",
        "    collate_fn=customized_collate_fn,\n",
        "    shuffle=False,\n",
        "    drop_last=False\n",
        ")\n",
        "\n",
        "test_dataset = InstructionDataset(test_data, tokenizer)\n",
        "test_loader = DataLoader(\n",
        "    test_dataset,\n",
        "    batch_size=batch_size,\n",
        "    collate_fn=customized_collate_fn,\n",
        "    shuffle=False,\n",
        "    drop_last=False\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3f67c147-b1a2-4a95-9807-e2d0de0324c0",
      "metadata": {
        "id": "3f67c147-b1a2-4a95-9807-e2d0de0324c0"
      },
      "source": [
        "- Let's see what the dimensions of the resulting input and target batches look like"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "GGs1AI3vHpnX",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GGs1AI3vHpnX",
        "outputId": "af4369a7-c6f5-4db9-e146-72d4d44baac4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train loader:\n",
            "torch.Size([8, 61]) torch.Size([8, 61])\n",
            "torch.Size([8, 76]) torch.Size([8, 76])\n",
            "torch.Size([8, 73]) torch.Size([8, 73])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 72]) torch.Size([8, 72])\n",
            "torch.Size([8, 80]) torch.Size([8, 80])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 62]) torch.Size([8, 62])\n",
            "torch.Size([8, 75]) torch.Size([8, 75])\n",
            "torch.Size([8, 62]) torch.Size([8, 62])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 77]) torch.Size([8, 77])\n",
            "torch.Size([8, 69]) torch.Size([8, 69])\n",
            "torch.Size([8, 79]) torch.Size([8, 79])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 80]) torch.Size([8, 80])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 69]) torch.Size([8, 69])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 60]) torch.Size([8, 60])\n",
            "torch.Size([8, 59]) torch.Size([8, 59])\n",
            "torch.Size([8, 69]) torch.Size([8, 69])\n",
            "torch.Size([8, 63]) torch.Size([8, 63])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 76]) torch.Size([8, 76])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 91]) torch.Size([8, 91])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 75]) torch.Size([8, 75])\n",
            "torch.Size([8, 89]) torch.Size([8, 89])\n",
            "torch.Size([8, 59]) torch.Size([8, 59])\n",
            "torch.Size([8, 88]) torch.Size([8, 88])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 70]) torch.Size([8, 70])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 74]) torch.Size([8, 74])\n",
            "torch.Size([8, 76]) torch.Size([8, 76])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 75]) torch.Size([8, 75])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 69]) torch.Size([8, 69])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 60]) torch.Size([8, 60])\n",
            "torch.Size([8, 60]) torch.Size([8, 60])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 80]) torch.Size([8, 80])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 61]) torch.Size([8, 61])\n",
            "torch.Size([8, 58]) torch.Size([8, 58])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 63]) torch.Size([8, 63])\n",
            "torch.Size([8, 87]) torch.Size([8, 87])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 71]) torch.Size([8, 71])\n",
            "torch.Size([8, 61]) torch.Size([8, 61])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 65]) torch.Size([8, 65])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 60]) torch.Size([8, 60])\n",
            "torch.Size([8, 72]) torch.Size([8, 72])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 70]) torch.Size([8, 70])\n",
            "torch.Size([8, 57]) torch.Size([8, 57])\n",
            "torch.Size([8, 72]) torch.Size([8, 72])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 62]) torch.Size([8, 62])\n",
            "torch.Size([8, 74]) torch.Size([8, 74])\n",
            "torch.Size([8, 80]) torch.Size([8, 80])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 70]) torch.Size([8, 70])\n",
            "torch.Size([8, 91]) torch.Size([8, 91])\n",
            "torch.Size([8, 61]) torch.Size([8, 61])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 80]) torch.Size([8, 80])\n",
            "torch.Size([8, 81]) torch.Size([8, 81])\n",
            "torch.Size([8, 74]) torch.Size([8, 74])\n",
            "torch.Size([8, 82]) torch.Size([8, 82])\n",
            "torch.Size([8, 63]) torch.Size([8, 63])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 68]) torch.Size([8, 68])\n",
            "torch.Size([8, 67]) torch.Size([8, 67])\n",
            "torch.Size([8, 77]) torch.Size([8, 77])\n",
            "torch.Size([8, 91]) torch.Size([8, 91])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 61]) torch.Size([8, 61])\n",
            "torch.Size([8, 75]) torch.Size([8, 75])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 78]) torch.Size([8, 78])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 64]) torch.Size([8, 64])\n",
            "torch.Size([8, 83]) torch.Size([8, 83])\n",
            "torch.Size([8, 66]) torch.Size([8, 66])\n",
            "torch.Size([8, 74]) torch.Size([8, 74])\n",
            "torch.Size([8, 69]) torch.Size([8, 69])\n"
          ]
        }
      ],
      "source": [
        "print(\"Train loader:\")\n",
        "for inputs, targets in train_loader:\n",
        "    print(inputs.shape, targets.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0c8e8dd7-d46a-4cc3-8a7e-c1d31e1b4657",
      "metadata": {
        "id": "0c8e8dd7-d46a-4cc3-8a7e-c1d31e1b4657"
      },
      "source": [
        "- As we can see based on the output above, all batches have a batch size of 8 but a different length, as expected\n",
        "- Let's also double-check that the inputs contain the `<|endoftext|>` padding tokens corresponding to token ID 50256 by printing the contents of the first training example in the `inputs` batch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "id": "21b8fd02-014f-4481-9b71-5bfee8f9dfcd",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "21b8fd02-014f-4481-9b71-5bfee8f9dfcd",
        "outputId": "d27b8807-0cd9-479f-bd0f-a3b69630f8aa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([21106,   318,   281, 12064,   326,  8477,   257,  4876,    13, 19430,\n",
            "          257,  2882,   326, 20431, 32543,   262,  2581,    13,   198,   198,\n",
            "        21017, 46486,    25,   198, 30003,  6525,   262,  6827,  1262,   257,\n",
            "          985,   576,    13,   198,   198, 21017, 23412,    25,   198,   464,\n",
            "         5156,   318,   845, 13779,    13,   198,   198, 21017, 18261,    25,\n",
            "          198,   464,  5156,   318,   355, 13779,   355,   257,  4936,    13,\n",
            "        50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256],\n",
            "       device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "print(inputs[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5f1f3647-8971-4006-89e0-6a2a1ec1d360",
      "metadata": {
        "id": "5f1f3647-8971-4006-89e0-6a2a1ec1d360"
      },
      "source": [
        "- Similarly, we visually double-check that the targets contain the -100 placeholder tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "id": "51649ab4-1a7e-4a9e-92c5-950a24fde211",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "51649ab4-1a7e-4a9e-92c5-950a24fde211",
        "outputId": "1cc35028-316f-4dea-e5be-919110922ee7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([  318,   281, 12064,   326,  8477,   257,  4876,    13, 19430,   257,\n",
            "         2882,   326, 20431, 32543,   262,  2581,    13,   198,   198, 21017,\n",
            "        46486,    25,   198, 30003,  6525,   262,  6827,  1262,   257,   985,\n",
            "          576,    13,   198,   198, 21017, 23412,    25,   198,   464,  5156,\n",
            "          318,   845, 13779,    13,   198,   198, 21017, 18261,    25,   198,\n",
            "          464,  5156,   318,   355, 13779,   355,   257,  4936,    13, 50256,\n",
            "         -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100,  -100],\n",
            "       device='cuda:0')\n"
          ]
        }
      ],
      "source": [
        "print(targets[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d6aad445-8f19-4238-b9bf-db80767fb91a",
      "metadata": {
        "id": "d6aad445-8f19-4238-b9bf-db80767fb91a"
      },
      "source": [
        "## 7.5 Loading a pretrained LLM"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5a5c07d1-4fc9-4846-94cf-b11a085a667b",
      "metadata": {
        "id": "5a5c07d1-4fc9-4846-94cf-b11a085a667b"
      },
      "source": [
        "- In this section, we load a pretrained GPT model using the same code that we used in section 5.5 of chapter 5 and section 6.4 in chapter 6"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d1b438f-88af-413f-96a9-f059c6c55fc4",
      "metadata": {
        "id": "8d1b438f-88af-413f-96a9-f059c6c55fc4"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-4.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8c68eda7-e02e-4caa-846b-ca6dbd396ca2",
      "metadata": {
        "id": "8c68eda7-e02e-4caa-846b-ca6dbd396ca2"
      },
      "source": [
        "- However, instead of loading the smallest 124 million parameter model, we load the medium version with 355 parameters since the 124 million model is too small for achieving qualitatively reasonable results via instruction finetuning"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Copyright (c) Sebastian Raschka under Apache License 2.0 (see LICENSE.txt).\n",
        "# Source for \"Build a Large Language Model From Scratch\"\n",
        "#   - https://www.manning.com/books/build-a-large-language-model-from-scratch\n",
        "# Code: https://github.com/rasbt/LLMs-from-scratch\n",
        "\n",
        "\n",
        "import os\n",
        "import requests\n",
        "import json\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tqdm import tqdm\n",
        "\n",
        "\n",
        "def download_and_load_gpt2(model_size, models_dir):\n",
        "    # Validate model size\n",
        "    allowed_sizes = (\"124M\", \"355M\", \"774M\", \"1558M\")\n",
        "    if model_size not in allowed_sizes:\n",
        "        raise ValueError(f\"Model size not in {allowed_sizes}\")\n",
        "\n",
        "    # Define paths\n",
        "    model_dir = os.path.join(models_dir, model_size)\n",
        "    base_url = \"https://openaipublic.blob.core.windows.net/gpt-2/models\"\n",
        "    filenames = [\n",
        "        \"checkpoint\", \"encoder.json\", \"hparams.json\",\n",
        "        \"model.ckpt.data-00000-of-00001\", \"model.ckpt.index\",\n",
        "        \"model.ckpt.meta\", \"vocab.bpe\"\n",
        "    ]\n",
        "\n",
        "    # Download files\n",
        "    os.makedirs(model_dir, exist_ok=True)\n",
        "    for filename in filenames:\n",
        "        file_url = os.path.join(base_url, model_size, filename)\n",
        "        file_path = os.path.join(model_dir, filename)\n",
        "        download_file(file_url, file_path)\n",
        "\n",
        "    # Load settings and params\n",
        "    tf_ckpt_path = tf.train.latest_checkpoint(model_dir)\n",
        "    settings = json.load(open(os.path.join(model_dir, \"hparams.json\")))\n",
        "    params = load_gpt2_params_from_tf_ckpt(tf_ckpt_path, settings)\n",
        "\n",
        "    return settings, params\n",
        "\n",
        "\n",
        "def download_file(url, destination):\n",
        "    # Send a GET request to download the file in streaming mode\n",
        "    response = requests.get(url, stream=True)\n",
        "\n",
        "    # Get the total file size from headers, defaulting to 0 if not present\n",
        "    file_size = int(response.headers.get(\"content-length\", 0))\n",
        "\n",
        "    # Check if file exists and has the same size\n",
        "    if os.path.exists(destination):\n",
        "        file_size_local = os.path.getsize(destination)\n",
        "        if file_size == file_size_local:\n",
        "            print(f\"File already exists and is up-to-date: {destination}\")\n",
        "            return\n",
        "\n",
        "    # Define the block size for reading the file\n",
        "    block_size = 1024  # 1 Kilobyte\n",
        "\n",
        "    # Initialize the progress bar with total file size\n",
        "    progress_bar_description = url.split(\"/\")[-1]  # Extract filename from URL\n",
        "    with tqdm(total=file_size, unit=\"iB\", unit_scale=True, desc=progress_bar_description) as progress_bar:\n",
        "        # Open the destination file in binary write mode\n",
        "        with open(destination, \"wb\") as file:\n",
        "            # Iterate over the file data in chunks\n",
        "            for chunk in response.iter_content(block_size):\n",
        "                progress_bar.update(len(chunk))  # Update progress bar\n",
        "                file.write(chunk)  # Write the chunk to the file\n",
        "\n",
        "\n",
        "def load_gpt2_params_from_tf_ckpt(ckpt_path, settings):\n",
        "    # Initialize parameters dictionary with empty blocks for each layer\n",
        "    params = {\"blocks\": [{} for _ in range(settings[\"n_layer\"])]}\n",
        "\n",
        "    # Iterate over each variable in the checkpoint\n",
        "    for name, _ in tf.train.list_variables(ckpt_path):\n",
        "        # Load the variable and remove singleton dimensions\n",
        "        variable_array = np.squeeze(tf.train.load_variable(ckpt_path, name))\n",
        "\n",
        "        # Process the variable name to extract relevant parts\n",
        "        variable_name_parts = name.split(\"/\")[1:]  # Skip the 'model/' prefix\n",
        "\n",
        "        # Identify the target dictionary for the variable\n",
        "        target_dict = params\n",
        "        if variable_name_parts[0].startswith(\"h\"):\n",
        "            layer_number = int(variable_name_parts[0][1:])\n",
        "            target_dict = params[\"blocks\"][layer_number]\n",
        "\n",
        "        # Recursively access or create nested dictionaries\n",
        "        for key in variable_name_parts[1:-1]:\n",
        "            target_dict = target_dict.setdefault(key, {})\n",
        "\n",
        "        # Assign the variable array to the last key\n",
        "        last_key = variable_name_parts[-1]\n",
        "        target_dict[last_key] = variable_array\n",
        "\n",
        "    return params\n"
      ],
      "metadata": {
        "id": "8Zu2gMrxLURc"
      },
      "id": "8Zu2gMrxLURc",
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Copyright (c) Sebastian Raschka under Apache License 2.0 (see LICENSE.txt).\n",
        "# Source for \"Build a Large Language Model From Scratch\"\n",
        "#   - https://www.manning.com/books/build-a-large-language-model-from-scratch\n",
        "# Code: https://github.com/rasbt/LLMs-from-scratch\n",
        "#\n",
        "# This file collects all the relevant code that we covered thus far\n",
        "# throughout Chapters 2-6.\n",
        "# This file can be run as a standalone script.\n",
        "\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import tiktoken\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "\n",
        "#####################################\n",
        "# Chapter 2\n",
        "#####################################\n",
        "\n",
        "\n",
        "class GPTDatasetV1(Dataset):\n",
        "    def __init__(self, txt, tokenizer, max_length, stride):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.input_ids = []\n",
        "        self.target_ids = []\n",
        "\n",
        "        # Tokenize the entire text\n",
        "        token_ids = tokenizer.encode(txt, allowed_special={\"<|endoftext|>\"})\n",
        "\n",
        "        # Use a sliding window to chunk the book into overlapping sequences of max_length\n",
        "        for i in range(0, len(token_ids) - max_length, stride):\n",
        "            input_chunk = token_ids[i:i + max_length]\n",
        "            target_chunk = token_ids[i + 1: i + max_length + 1]\n",
        "            self.input_ids.append(torch.tensor(input_chunk))\n",
        "            self.target_ids.append(torch.tensor(target_chunk))\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.input_ids)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        return self.input_ids[idx], self.target_ids[idx]\n",
        "\n",
        "\n",
        "def create_dataloader_v1(txt, batch_size=4, max_length=256,\n",
        "                         stride=128, shuffle=True, drop_last=True):\n",
        "    # Initialize the tokenizer\n",
        "    tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
        "\n",
        "    # Create dataset\n",
        "    dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)\n",
        "\n",
        "    # Create dataloader\n",
        "    dataloader = DataLoader(\n",
        "        dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)\n",
        "\n",
        "    return dataloader\n",
        "\n",
        "\n",
        "#####################################\n",
        "# Chapter 3\n",
        "#####################################\n",
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):\n",
        "        super().__init__()\n",
        "        assert d_out % num_heads == 0, \"d_out must be divisible by n_heads\"\n",
        "\n",
        "        self.d_out = d_out\n",
        "        self.num_heads = num_heads\n",
        "        self.head_dim = d_out // num_heads  # Reduce the projection dim to match desired output dim\n",
        "\n",
        "        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
        "        self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
        "        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)\n",
        "        self.out_proj = nn.Linear(d_out, d_out)  # Linear layer to combine head outputs\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.register_buffer('mask', torch.triu(torch.ones(context_length, context_length), diagonal=1))\n",
        "\n",
        "    def forward(self, x):\n",
        "        b, num_tokens, d_in = x.shape\n",
        "\n",
        "        keys = self.W_key(x)  # Shape: (b, num_tokens, d_out)\n",
        "        queries = self.W_query(x)\n",
        "        values = self.W_value(x)\n",
        "\n",
        "        # We implicitly split the matrix by adding a `num_heads` dimension\n",
        "        # Unroll last dim: (b, num_tokens, d_out) -> (b, num_tokens, num_heads, head_dim)\n",
        "        keys = keys.view(b, num_tokens, self.num_heads, self.head_dim)\n",
        "        values = values.view(b, num_tokens, self.num_heads, self.head_dim)\n",
        "        queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)\n",
        "\n",
        "        # Transpose: (b, num_tokens, num_heads, head_dim) -> (b, num_heads, num_tokens, head_dim)\n",
        "        keys = keys.transpose(1, 2)\n",
        "        queries = queries.transpose(1, 2)\n",
        "        values = values.transpose(1, 2)\n",
        "\n",
        "        # Compute scaled dot-product attention (aka self-attention) with a causal mask\n",
        "        attn_scores = queries @ keys.transpose(2, 3)  # Dot product for each head\n",
        "\n",
        "        # Original mask truncated to the number of tokens and converted to boolean\n",
        "        mask_bool = self.mask.bool()[:num_tokens, :num_tokens]\n",
        "\n",
        "        # Use the mask to fill attention scores\n",
        "        attn_scores.masked_fill_(mask_bool, -torch.inf)\n",
        "\n",
        "        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n",
        "        attn_weights = self.dropout(attn_weights)\n",
        "\n",
        "        # Shape: (b, num_tokens, num_heads, head_dim)\n",
        "        context_vec = (attn_weights @ values).transpose(1, 2)\n",
        "\n",
        "        # Combine heads, where self.d_out = self.num_heads * self.head_dim\n",
        "        context_vec = context_vec.reshape(b, num_tokens, self.d_out)\n",
        "        context_vec = self.out_proj(context_vec)  # optional projection\n",
        "\n",
        "        return context_vec\n",
        "\n",
        "\n",
        "#####################################\n",
        "# Chapter 4\n",
        "#####################################\n",
        "class LayerNorm(nn.Module):\n",
        "    def __init__(self, emb_dim):\n",
        "        super().__init__()\n",
        "        self.eps = 1e-5\n",
        "        self.scale = nn.Parameter(torch.ones(emb_dim))\n",
        "        self.shift = nn.Parameter(torch.zeros(emb_dim))\n",
        "\n",
        "    def forward(self, x):\n",
        "        mean = x.mean(dim=-1, keepdim=True)\n",
        "        var = x.var(dim=-1, keepdim=True, unbiased=False)\n",
        "        norm_x = (x - mean) / torch.sqrt(var + self.eps)\n",
        "        return self.scale * norm_x + self.shift\n",
        "\n",
        "\n",
        "class GELU(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "\n",
        "    def forward(self, x):\n",
        "        return 0.5 * x * (1 + torch.tanh(\n",
        "            torch.sqrt(torch.tensor(2.0 / torch.pi)) *\n",
        "            (x + 0.044715 * torch.pow(x, 3))\n",
        "        ))\n",
        "\n",
        "\n",
        "class FeedForward(nn.Module):\n",
        "    def __init__(self, cfg):\n",
        "        super().__init__()\n",
        "        self.layers = nn.Sequential(\n",
        "            nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n",
        "            GELU(),\n",
        "            nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.layers(x)\n",
        "\n",
        "\n",
        "class TransformerBlock(nn.Module):\n",
        "    def __init__(self, cfg):\n",
        "        super().__init__()\n",
        "        self.att = MultiHeadAttention(\n",
        "            d_in=cfg[\"emb_dim\"],\n",
        "            d_out=cfg[\"emb_dim\"],\n",
        "            context_length=cfg[\"context_length\"],\n",
        "            num_heads=cfg[\"n_heads\"],\n",
        "            dropout=cfg[\"drop_rate\"],\n",
        "            qkv_bias=cfg[\"qkv_bias\"])\n",
        "        self.ff = FeedForward(cfg)\n",
        "        self.norm1 = LayerNorm(cfg[\"emb_dim\"])\n",
        "        self.norm2 = LayerNorm(cfg[\"emb_dim\"])\n",
        "        self.drop_resid = nn.Dropout(cfg[\"drop_rate\"])\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Shortcut connection for attention block\n",
        "        shortcut = x\n",
        "        x = self.norm1(x)\n",
        "        x = self.att(x)   # Shape [batch_size, num_tokens, emb_size]\n",
        "        x = self.drop_resid(x)\n",
        "        x = x + shortcut  # Add the original input back\n",
        "\n",
        "        # Shortcut connection for feed-forward block\n",
        "        shortcut = x\n",
        "        x = self.norm2(x)\n",
        "        x = self.ff(x)\n",
        "        x = self.drop_resid(x)\n",
        "        x = x + shortcut  # Add the original input back\n",
        "\n",
        "        return x\n",
        "\n",
        "\n",
        "class GPTModel(nn.Module):\n",
        "    def __init__(self, cfg):\n",
        "        super().__init__()\n",
        "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
        "        self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
        "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
        "\n",
        "        self.trf_blocks = nn.Sequential(\n",
        "            *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
        "\n",
        "        self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n",
        "        self.out_head = nn.Linear(cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False)\n",
        "\n",
        "    def forward(self, in_idx):\n",
        "        batch_size, seq_len = in_idx.shape\n",
        "        tok_embeds = self.tok_emb(in_idx)\n",
        "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
        "        x = tok_embeds + pos_embeds  # Shape [batch_size, num_tokens, emb_size]\n",
        "        x = self.drop_emb(x)\n",
        "        x = self.trf_blocks(x)\n",
        "        x = self.final_norm(x)\n",
        "        logits = self.out_head(x)\n",
        "        return logits\n",
        "\n",
        "\n",
        "def generate_text_simple(model, idx, max_new_tokens, context_size):\n",
        "    # idx is (B, T) array of indices in the current context\n",
        "    for _ in range(max_new_tokens):\n",
        "\n",
        "        # Crop current context if it exceeds the supported context size\n",
        "        # E.g., if LLM supports only 5 tokens, and the context size is 10\n",
        "        # then only the last 5 tokens are used as context\n",
        "        idx_cond = idx[:, -context_size:]\n",
        "\n",
        "        # Get the predictions\n",
        "        with torch.no_grad():\n",
        "            logits = model(idx_cond)\n",
        "\n",
        "        # Focus only on the last time step\n",
        "        # (batch, n_token, vocab_size) becomes (batch, vocab_size)\n",
        "        logits = logits[:, -1, :]\n",
        "\n",
        "        # Get the idx of the vocab entry with the highest logits value\n",
        "        idx_next = torch.argmax(logits, dim=-1, keepdim=True)  # (batch, 1)\n",
        "\n",
        "        # Append sampled index to the running sequence\n",
        "        idx = torch.cat((idx, idx_next), dim=1)  # (batch, n_tokens+1)\n",
        "\n",
        "    return idx\n",
        "\n",
        "\n",
        "#####################################\n",
        "# Chapter 5\n",
        "#####################################\n",
        "def generate(model, idx, max_new_tokens, context_size, temperature=0.0, top_k=None, eos_id=None):\n",
        "\n",
        "    # For-loop is the same as before: Get logits, and only focus on last time step\n",
        "    for _ in range(max_new_tokens):\n",
        "        idx_cond = idx[:, -context_size:]\n",
        "        with torch.no_grad():\n",
        "            logits = model(idx_cond)\n",
        "        logits = logits[:, -1, :]\n",
        "\n",
        "        # New: Filter logits with top_k sampling\n",
        "        if top_k is not None:\n",
        "            # Keep only top_k values\n",
        "            top_logits, _ = torch.topk(logits, top_k)\n",
        "            min_val = top_logits[:, -1]\n",
        "            logits = torch.where(logits < min_val, torch.tensor(float('-inf')).to(logits.device), logits)\n",
        "\n",
        "        # New: Apply temperature scaling\n",
        "        if temperature > 0.0:\n",
        "            logits = logits / temperature\n",
        "\n",
        "            # Apply softmax to get probabilities\n",
        "            probs = torch.softmax(logits, dim=-1)  # (batch_size, context_len)\n",
        "\n",
        "            # Sample from the distribution\n",
        "            idx_next = torch.multinomial(probs, num_samples=1)  # (batch_size, 1)\n",
        "\n",
        "        # Otherwise same as before: get idx of the vocab entry with the highest logits value\n",
        "        else:\n",
        "            idx_next = torch.argmax(logits, dim=-1, keepdim=True)  # (batch_size, 1)\n",
        "\n",
        "        if idx_next == eos_id:  # Stop generating early if end-of-sequence token is encountered and eos_id is specified\n",
        "            break\n",
        "\n",
        "        # Same as before: append sampled index to the running sequence\n",
        "        idx = torch.cat((idx, idx_next), dim=1)  # (batch_size, num_tokens+1)\n",
        "\n",
        "    return idx\n",
        "\n",
        "\n",
        "def train_model_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
        "                       eval_freq, eval_iter, start_context, tokenizer):\n",
        "    # Initialize lists to track losses and tokens seen\n",
        "    train_losses, val_losses, track_tokens_seen = [], [], []\n",
        "    tokens_seen, global_step = 0, -1\n",
        "\n",
        "    # Main training loop\n",
        "    for epoch in range(num_epochs):\n",
        "        model.train()  # Set model to training mode\n",
        "\n",
        "        for input_batch, target_batch in train_loader:\n",
        "            optimizer.zero_grad()  # Reset loss gradients from previous batch iteration\n",
        "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
        "            loss.backward()  # Calculate loss gradients\n",
        "            optimizer.step()  # Update model weights using loss gradients\n",
        "            tokens_seen += input_batch.numel()\n",
        "            global_step += 1\n",
        "\n",
        "            # Optional evaluation step\n",
        "            if global_step % eval_freq == 0:\n",
        "                train_loss, val_loss = evaluate_model(\n",
        "                    model, train_loader, val_loader, device, eval_iter)\n",
        "                train_losses.append(train_loss)\n",
        "                val_losses.append(val_loss)\n",
        "                track_tokens_seen.append(tokens_seen)\n",
        "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
        "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
        "\n",
        "        # Print a sample text after each epoch\n",
        "        generate_and_print_sample(\n",
        "            model, tokenizer, device, start_context\n",
        "        )\n",
        "\n",
        "    return train_losses, val_losses, track_tokens_seen\n",
        "\n",
        "\n",
        "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
        "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
        "    model.train()\n",
        "    return train_loss, val_loss\n",
        "\n",
        "\n",
        "def generate_and_print_sample(model, tokenizer, device, start_context):\n",
        "    model.eval()\n",
        "    context_size = model.pos_emb.weight.shape[0]\n",
        "    encoded = text_to_token_ids(start_context, tokenizer).to(device)\n",
        "    with torch.no_grad():\n",
        "        token_ids = generate_text_simple(\n",
        "            model=model, idx=encoded,\n",
        "            max_new_tokens=50, context_size=context_size\n",
        "        )\n",
        "        decoded_text = token_ids_to_text(token_ids, tokenizer)\n",
        "        print(decoded_text.replace(\"\\n\", \" \"))  # Compact print format\n",
        "    model.train()\n",
        "\n",
        "\n",
        "def assign(left, right):\n",
        "    if left.shape != right.shape:\n",
        "        raise ValueError(f\"Shape mismatch. Left: {left.shape}, Right: {right.shape}\")\n",
        "    return torch.nn.Parameter(torch.tensor(right))\n",
        "\n",
        "\n",
        "def load_weights_into_gpt(gpt, params):\n",
        "    gpt.pos_emb.weight = assign(gpt.pos_emb.weight, params['wpe'])\n",
        "    gpt.tok_emb.weight = assign(gpt.tok_emb.weight, params['wte'])\n",
        "\n",
        "    for b in range(len(params[\"blocks\"])):\n",
        "        q_w, k_w, v_w = np.split(\n",
        "            (params[\"blocks\"][b][\"attn\"][\"c_attn\"])[\"w\"], 3, axis=-1)\n",
        "        gpt.trf_blocks[b].att.W_query.weight = assign(\n",
        "            gpt.trf_blocks[b].att.W_query.weight, q_w.T)\n",
        "        gpt.trf_blocks[b].att.W_key.weight = assign(\n",
        "            gpt.trf_blocks[b].att.W_key.weight, k_w.T)\n",
        "        gpt.trf_blocks[b].att.W_value.weight = assign(\n",
        "            gpt.trf_blocks[b].att.W_value.weight, v_w.T)\n",
        "\n",
        "        q_b, k_b, v_b = np.split(\n",
        "            (params[\"blocks\"][b][\"attn\"][\"c_attn\"])[\"b\"], 3, axis=-1)\n",
        "        gpt.trf_blocks[b].att.W_query.bias = assign(\n",
        "            gpt.trf_blocks[b].att.W_query.bias, q_b)\n",
        "        gpt.trf_blocks[b].att.W_key.bias = assign(\n",
        "            gpt.trf_blocks[b].att.W_key.bias, k_b)\n",
        "        gpt.trf_blocks[b].att.W_value.bias = assign(\n",
        "            gpt.trf_blocks[b].att.W_value.bias, v_b)\n",
        "\n",
        "        gpt.trf_blocks[b].att.out_proj.weight = assign(\n",
        "            gpt.trf_blocks[b].att.out_proj.weight,\n",
        "            params[\"blocks\"][b][\"attn\"][\"c_proj\"][\"w\"].T)\n",
        "        gpt.trf_blocks[b].att.out_proj.bias = assign(\n",
        "            gpt.trf_blocks[b].att.out_proj.bias,\n",
        "            params[\"blocks\"][b][\"attn\"][\"c_proj\"][\"b\"])\n",
        "\n",
        "        gpt.trf_blocks[b].ff.layers[0].weight = assign(\n",
        "            gpt.trf_blocks[b].ff.layers[0].weight,\n",
        "            params[\"blocks\"][b][\"mlp\"][\"c_fc\"][\"w\"].T)\n",
        "        gpt.trf_blocks[b].ff.layers[0].bias = assign(\n",
        "            gpt.trf_blocks[b].ff.layers[0].bias,\n",
        "            params[\"blocks\"][b][\"mlp\"][\"c_fc\"][\"b\"])\n",
        "        gpt.trf_blocks[b].ff.layers[2].weight = assign(\n",
        "            gpt.trf_blocks[b].ff.layers[2].weight,\n",
        "            params[\"blocks\"][b][\"mlp\"][\"c_proj\"][\"w\"].T)\n",
        "        gpt.trf_blocks[b].ff.layers[2].bias = assign(\n",
        "            gpt.trf_blocks[b].ff.layers[2].bias,\n",
        "            params[\"blocks\"][b][\"mlp\"][\"c_proj\"][\"b\"])\n",
        "\n",
        "        gpt.trf_blocks[b].norm1.scale = assign(\n",
        "            gpt.trf_blocks[b].norm1.scale,\n",
        "            params[\"blocks\"][b][\"ln_1\"][\"g\"])\n",
        "        gpt.trf_blocks[b].norm1.shift = assign(\n",
        "            gpt.trf_blocks[b].norm1.shift,\n",
        "            params[\"blocks\"][b][\"ln_1\"][\"b\"])\n",
        "        gpt.trf_blocks[b].norm2.scale = assign(\n",
        "            gpt.trf_blocks[b].norm2.scale,\n",
        "            params[\"blocks\"][b][\"ln_2\"][\"g\"])\n",
        "        gpt.trf_blocks[b].norm2.shift = assign(\n",
        "            gpt.trf_blocks[b].norm2.shift,\n",
        "            params[\"blocks\"][b][\"ln_2\"][\"b\"])\n",
        "\n",
        "    gpt.final_norm.scale = assign(gpt.final_norm.scale, params[\"g\"])\n",
        "    gpt.final_norm.shift = assign(gpt.final_norm.shift, params[\"b\"])\n",
        "    gpt.out_head.weight = assign(gpt.out_head.weight, params[\"wte\"])\n",
        "\n",
        "\n",
        "def text_to_token_ids(text, tokenizer):\n",
        "    encoded = tokenizer.encode(text, allowed_special={\"<|endoftext|>\"})\n",
        "    encoded_tensor = torch.tensor(encoded).unsqueeze(0)  # add batch dimension\n",
        "    return encoded_tensor\n",
        "\n",
        "\n",
        "def token_ids_to_text(token_ids, tokenizer):\n",
        "    flat = token_ids.squeeze(0)  # remove batch dimension\n",
        "    return tokenizer.decode(flat.tolist())\n",
        "\n",
        "\n",
        "def calc_loss_batch(input_batch, target_batch, model, device):\n",
        "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
        "    logits = model(input_batch)\n",
        "    loss = torch.nn.functional.cross_entropy(logits.flatten(0, 1), target_batch.flatten())\n",
        "    return loss\n",
        "\n",
        "\n",
        "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
        "    total_loss = 0.\n",
        "    if len(data_loader) == 0:\n",
        "        return float(\"nan\")\n",
        "    elif num_batches is None:\n",
        "        num_batches = len(data_loader)\n",
        "    else:\n",
        "        # Reduce the number of batches to match the total number of batches in the data loader\n",
        "        # if num_batches exceeds the number of batches in the data loader\n",
        "        num_batches = min(num_batches, len(data_loader))\n",
        "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
        "        if i < num_batches:\n",
        "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
        "            total_loss += loss.item()\n",
        "        else:\n",
        "            break\n",
        "    return total_loss / num_batches\n",
        "\n",
        "\n",
        "def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses):\n",
        "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
        "\n",
        "    # Plot training and validation loss against epochs\n",
        "    ax1.plot(epochs_seen, train_losses, label=\"Training loss\")\n",
        "    ax1.plot(epochs_seen, val_losses, linestyle=\"-.\", label=\"Validation loss\")\n",
        "    ax1.set_xlabel(\"Epochs\")\n",
        "    ax1.set_ylabel(\"Loss\")\n",
        "    ax1.legend(loc=\"upper right\")\n",
        "\n",
        "    # Create a second x-axis for tokens seen\n",
        "    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis\n",
        "    ax2.plot(tokens_seen, train_losses, alpha=0)  # Invisible plot for aligning ticks\n",
        "    ax2.set_xlabel(\"Tokens seen\")\n",
        "\n",
        "    fig.tight_layout()  # Adjust layout to make room\n",
        "    plt.savefig(\"loss-plot.pdf\")\n",
        "    plt.show()\n"
      ],
      "metadata": {
        "id": "aU36qA8ELd5w"
      },
      "id": "aU36qA8ELd5w",
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "id": "0d249d67-5eba-414e-9bd2-972ebf01329d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0d249d67-5eba-414e-9bd2-972ebf01329d",
        "outputId": "2839340b-e02f-4bb4-92f1-0e85aa1ad49b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "checkpoint: 100%|██████████| 77.0/77.0 [00:00<00:00, 58.4kiB/s]\n",
            "encoder.json: 100%|██████████| 1.04M/1.04M [00:02<00:00, 474kiB/s]\n",
            "hparams.json: 100%|██████████| 91.0/91.0 [00:00<00:00, 63.2kiB/s]\n",
            "model.ckpt.data-00000-of-00001: 100%|██████████| 1.42G/1.42G [03:15<00:00, 7.26MiB/s]\n",
            "model.ckpt.index: 100%|██████████| 10.4k/10.4k [00:00<00:00, 7.53MiB/s]\n",
            "model.ckpt.meta: 100%|██████████| 927k/927k [00:01<00:00, 516kiB/s]\n",
            "vocab.bpe: 100%|██████████| 456k/456k [00:01<00:00, 327kiB/s]\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "\n",
        "\n",
        "BASE_CONFIG = {\n",
        "    \"vocab_size\": 50257,     # Vocabulary size\n",
        "    \"context_length\": 1024,  # Context length\n",
        "    \"drop_rate\": 0.0,        # Dropout rate\n",
        "    \"qkv_bias\": True         # Query-key-value bias\n",
        "}\n",
        "\n",
        "model_configs = {\n",
        "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
        "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
        "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
        "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
        "}\n",
        "\n",
        "CHOOSE_MODEL = \"gpt2-medium (355M)\"\n",
        "\n",
        "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
        "\n",
        "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
        "settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
        "\n",
        "model = GPTModel(BASE_CONFIG)\n",
        "load_weights_into_gpt(model, params)\n",
        "model.eval();"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dbf3afed-bc8e-4d3a-ad9d-eb6f57bb7af5",
      "metadata": {
        "id": "dbf3afed-bc8e-4d3a-ad9d-eb6f57bb7af5"
      },
      "source": [
        "- Before we start finetuning the model in the next section, let's see how it performs on one of the validation tasks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "id": "7bd32b7c-5b44-4d25-a09f-46836802ca74",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7bd32b7c-5b44-4d25-a09f-46836802ca74",
        "outputId": "612515b2-60f9-4efd-cbf3-2191ad5fe6b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction:\n",
            "Convert the active sentence to passive: 'The chef cooks the meal every day.'\n"
          ]
        }
      ],
      "source": [
        "torch.manual_seed(123)\n",
        "\n",
        "input_text = format_input(val_data[0])\n",
        "print(input_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "id": "2e3e68e0-2627-4c65-b4e7-1e0667e4f6fa",
      "metadata": {
        "id": "2e3e68e0-2627-4c65-b4e7-1e0667e4f6fa"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "token_ids = generate(\n",
        "    model=model,\n",
        "    idx=text_to_token_ids(input_text, tokenizer),\n",
        "    max_new_tokens=35,\n",
        "    context_size=BASE_CONFIG[\"context_length\"],\n",
        "    eos_id=50256,\n",
        ")\n",
        "generated_text = token_ids_to_text(token_ids, tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "36e2fda5-f796-4954-8f72-1dd1123e3344",
      "metadata": {
        "id": "36e2fda5-f796-4954-8f72-1dd1123e3344"
      },
      "source": [
        "- Note that the `generate` function we used in previous chapters returns the combined input and output text, which was convenient in the previous section for creating legible text\n",
        "- To isolate the response, we can subtract the length of the instruction from the start of the `generated_text`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "id": "ba4a55bf-a245-48d8-beda-2838a58fb5ba",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ba4a55bf-a245-48d8-beda-2838a58fb5ba",
        "outputId": "8e0cb210-5ddf-4b3e-b3cb-75471e9085bf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "### Response:\n",
            "\n",
            "The chef cooks the meal every day.\n",
            "\n",
            "### Instruction:\n",
            "\n",
            "Convert the active sentence to passive: 'The chef cooks the\n"
          ]
        }
      ],
      "source": [
        "response_text = generated_text[len(input_text):].strip()\n",
        "print(response_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d44080b2-a4c5-4520-a797-549519f66a3e",
      "metadata": {
        "id": "d44080b2-a4c5-4520-a797-549519f66a3e"
      },
      "source": [
        "- As we can see, the model is not capable of following the instructions, yet; it creates a \"Response\" section but it simply repeats the original input sentence as well as the instruction"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "70d27b9d-a942-4cf5-b797-848c5f01e723",
      "metadata": {
        "id": "70d27b9d-a942-4cf5-b797-848c5f01e723"
      },
      "source": [
        "## 7.6 Finetuning the LLM on instruction data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "314b2a39-88b4-44d8-8c85-1c5b0cd6cc4a",
      "metadata": {
        "id": "314b2a39-88b4-44d8-8c85-1c5b0cd6cc4a"
      },
      "source": [
        "- In this section, we finetune the model\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-5.webp\" width=500px>\n",
        "\n",
        "- Note that we can reuse all the loss calculation and training functions that we used in previous chapters"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "00083059-aa41-4d37-8a17-1c72d1b1ca00",
      "metadata": {
        "id": "00083059-aa41-4d37-8a17-1c72d1b1ca00"
      },
      "source": [
        "- Let's calculate the initial training and validation set loss before we start training (as in previous chapters, the goal is to minimize the loss)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "id": "d99fc6f8-63b2-43da-adbb-a7b6b92c8dd5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d99fc6f8-63b2-43da-adbb-a7b6b92c8dd5",
        "outputId": "881eccbf-e06e-497f-af2c-c27db37a6b46"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training loss: 3.825909471511841\n",
            "Validation loss: 3.7619341373443604\n"
          ]
        }
      ],
      "source": [
        "model.to(device)\n",
        "\n",
        "torch.manual_seed(123)\n",
        "\n",
        "with torch.no_grad():\n",
        "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
        "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
        "\n",
        "print(\"Training loss:\", train_loss)\n",
        "print(\"Validation loss:\", val_loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "12a6da8f-15b3-42b0-a136-619b7a35c3e9",
      "metadata": {
        "id": "12a6da8f-15b3-42b0-a136-619b7a35c3e9"
      },
      "source": [
        "- Note that the training is a bit more expensive than in previous chapters since we are using a larger model (355 million instead of 124 million parameters)\n",
        "- The runtimes for various devices are shown for reference below (running this notebook on a compatible GPU device requires no changes to the code)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "db4b57fb-e689-4550-931c-6d34a932487c",
      "metadata": {
        "id": "db4b57fb-e689-4550-931c-6d34a932487c"
      },
      "source": [
        "<div style=\"text-align: left;\">\n",
        "    \n",
        "| Model              | Device                | Runtime for 2 Epochs |\n",
        "|--------------------|-----------------------|----------------------|\n",
        "| gpt2-medium (355M) | CPU (M3 MacBook Air)  | 15.78 minutes        |\n",
        "| gpt2-medium (355M) | GPU (L4)              | 1.83 minutes         |\n",
        "| gpt2-medium (355M) | GPU (A100)            | 0.86 minutes         |\n",
        "| gpt2-small (124M)  | CPU (M3 MacBook Air)  | 5.74 minutes         |\n",
        "| gpt2-small (124M)  | GPU (L4)              | 0.69 minutes         |\n",
        "| gpt2-small (124M)  | GPU (A100)            | 0.39 minutes         |\n",
        "\n",
        "</div>\n",
        "\n",
        "- I ran this notebook using the `\"gpt2-medium (355M)\"` model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "id": "78bcf83a-1fff-4540-97c1-765c4016d5e3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "id": "78bcf83a-1fff-4540-97c1-765c4016d5e3",
        "outputId": "edb4c94b-f44c-4336-ed8f-53827e932b40"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Ep 1 (Step 000000): Train loss 2.637, Val loss 2.626\n",
            "Ep 1 (Step 000005): Train loss 1.174, Val loss 1.103\n",
            "Ep 1 (Step 000010): Train loss 0.872, Val loss 0.944\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-35-5d77fa1263b5>\u001b[0m in \u001b[0;36m<cell line: 11>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0mnum_epochs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m train_losses, val_losses, tokens_seen = train_model_simple(\n\u001b[0m\u001b[1;32m     12\u001b[0m     \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_loader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mval_loader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0mnum_epochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnum_epochs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meval_freq\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meval_iter\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-29-eed68f8edeb3>\u001b[0m in \u001b[0;36mtrain_model_simple\u001b[0;34m(model, train_loader, val_loader, optimizer, device, num_epochs, eval_freq, eval_iter, start_context, tokenizer)\u001b[0m\n\u001b[1;32m    306\u001b[0m             \u001b[0;31m# Optional evaluation step\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    307\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mglobal_step\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0meval_freq\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 308\u001b[0;31m                 train_loss, val_loss = evaluate_model(\n\u001b[0m\u001b[1;32m    309\u001b[0m                     model, train_loader, val_loader, device, eval_iter)\n\u001b[1;32m    310\u001b[0m                 \u001b[0mtrain_losses\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_loss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-29-eed68f8edeb3>\u001b[0m in \u001b[0;36mevaluate_model\u001b[0;34m(model, train_loader, val_loader, device, eval_iter)\u001b[0m\n\u001b[1;32m    325\u001b[0m     \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    326\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 327\u001b[0;31m         \u001b[0mtrain_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcalc_loss_loader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_loader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_batches\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    328\u001b[0m         \u001b[0mval_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcalc_loss_loader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval_loader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_batches\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_iter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    329\u001b[0m     \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-29-eed68f8edeb3>\u001b[0m in \u001b[0;36mcalc_loss_loader\u001b[0;34m(data_loader, model, device, num_batches)\u001b[0m\n\u001b[1;32m    443\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mnum_batches\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    444\u001b[0m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcalc_loss_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 445\u001b[0;31m             \u001b[0mtotal_loss\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    446\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    447\u001b[0m             \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "start_time = time.time()\n",
        "\n",
        "torch.manual_seed(123)\n",
        "\n",
        "optimizer = torch.optim.AdamW(model.parameters(), lr=0.00005, weight_decay=0.1)\n",
        "\n",
        "num_epochs = 2\n",
        "\n",
        "train_losses, val_losses, tokens_seen = train_model_simple(\n",
        "    model, train_loader, val_loader, optimizer, device,\n",
        "    num_epochs=num_epochs, eval_freq=5, eval_iter=5,\n",
        "    start_context=format_input(val_data[0]), tokenizer=tokenizer\n",
        ")\n",
        "\n",
        "end_time = time.time()\n",
        "execution_time_minutes = (end_time - start_time) / 60\n",
        "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ise3wGjlB-iq",
      "metadata": {
        "id": "Ise3wGjlB-iq"
      },
      "source": [
        "- As we can see based on the outputs above, the model trains well, as we can tell based on the decreasing training loss and validation loss values\n",
        "- Furthermore, based on the response text printed after each epoch, we can see that the model correctly follows the instruction to convert the input sentence `'The chef cooks the meal every day.'` into passive voice `'The meal is cooked every day by the chef.'` (We will properly format and evaluate the responses in a later section)\n",
        "- Finally, let's take a look at the training and validation loss curves"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4acd368b-1403-4807-a218-9102e35bfdbb",
      "metadata": {
        "id": "4acd368b-1403-4807-a218-9102e35bfdbb"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
        "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6777e0c4-d82c-46d8-84fb-1376c4f8bae0",
      "metadata": {
        "id": "6777e0c4-d82c-46d8-84fb-1376c4f8bae0"
      },
      "source": [
        "- As we can see, the loss decreases sharply at the beginning of the first epoch, which means the model starts learning quickly\n",
        "- We can see that slight overfitting sets in at around 1 training epoch"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "87b79a47-13f9-4d1f-87b1-3339bafaf2a3",
      "metadata": {
        "id": "87b79a47-13f9-4d1f-87b1-3339bafaf2a3"
      },
      "source": [
        "## 7.7 Extracting and saving responses"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5a25cc88-1758-4dd0-b8bf-c044cbf2dd49",
      "metadata": {
        "id": "5a25cc88-1758-4dd0-b8bf-c044cbf2dd49"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-6.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "17510e9d-7727-4d58-ba9a-d82ec23c1427",
      "metadata": {
        "id": "17510e9d-7727-4d58-ba9a-d82ec23c1427"
      },
      "source": [
        "- In this section, we save the test set responses for scoring in the next section\n",
        "- We also save a copy of the model for future use\n",
        "- But first, let's take a brief look at the responses generated by the finetuned model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VQ2NZMbfucAc",
      "metadata": {
        "id": "VQ2NZMbfucAc"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(123)\n",
        "\n",
        "\n",
        "for entry in test_data[:3]:\n",
        "\n",
        "    input_text = format_input(entry)\n",
        "\n",
        "    token_ids = generate(\n",
        "        model=model,\n",
        "        idx=text_to_token_ids(input_text, tokenizer).to(device),\n",
        "        max_new_tokens=256,\n",
        "        context_size=BASE_CONFIG[\"context_length\"],\n",
        "        eos_id=50256\n",
        "    )\n",
        "    generated_text = token_ids_to_text(token_ids, tokenizer)\n",
        "    response_text = generated_text[len(input_text):].replace(\"### Response:\", \"\").strip()\n",
        "\n",
        "    print(input_text)\n",
        "    print(f\"\\nCorrect response:\\n>> {entry['output']}\")\n",
        "    print(f\"\\nModel response:\\n>> {response_text.strip()}\")\n",
        "    print(\"-------------------------------------\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "49ab64c1-586f-4939-8def-23feeb1b3599",
      "metadata": {
        "id": "49ab64c1-586f-4939-8def-23feeb1b3599"
      },
      "source": [
        "- As we can see based on the test set instructions, given responses, and the model's responses, the model performs relatively well\n",
        "- The answers to the first and last instructions are clearly correct\n",
        "- The second answer is close; the model answers with \"cumulus cloud\" instead of \"cumulonimbus\" (however, note that cumulus clouds can develop into cumulonimbus clouds, which are capable of producing thunderstorms)\n",
        "- Most importantly, we can see that model evaluation is not as straightforward as in the previous chapter, where we just had to calculate the percentage of correct spam/non-spam class labels to obtain the classification accuracy\n",
        "- In practice, instruction-finetuned LLMs such as chatbots are evaluated via multiple approaches\n",
        "  - short-answer and multiple choice benchmarks such as MMLU (\"Measuring Massive Multitask Language Understanding\", [https://arxiv.org/abs/2009.03300](https://arxiv.org/abs/2009.03300)), which test the knowledge of a model\n",
        "  - human preference comparison to other LLMs, such as LMSYS chatbot arena ([https://arena.lmsys.org](https://arena.lmsys.org))\n",
        "  - automated conversational benchmarks, where another LLM like GPT-4 is used to evaluate the responses, such as AlpacaEval ([https://tatsu-lab.github.io/alpaca_eval/](https://tatsu-lab.github.io/alpaca_eval/))\n",
        "\n",
        "- In the next section, we will use an approach similar to AlpaceEval and use another LLM to evaluate the responses of our model; however, we will use our own test set instead of using a publicly available benchmark dataset\n",
        "- For this, we add the model response to the `test_set` dictionary and save it as a `\"instruction-data-with-response.json\"` file for record-keeping so that we can load and analyze it in separate Python sessions if needed"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-PNGKzY4snKP",
      "metadata": {
        "id": "-PNGKzY4snKP"
      },
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "for i, entry in tqdm(enumerate(test_data), total=len(test_data)):\n",
        "\n",
        "    input_text = format_input(entry)\n",
        "\n",
        "    token_ids = generate(\n",
        "        model=model,\n",
        "        idx=text_to_token_ids(input_text, tokenizer).to(device),\n",
        "        max_new_tokens=256,\n",
        "        context_size=BASE_CONFIG[\"context_length\"],\n",
        "        eos_id=50256\n",
        "    )\n",
        "    generated_text = token_ids_to_text(token_ids, tokenizer)\n",
        "    response_text = generated_text[len(input_text):].replace(\"### Response:\", \"\").strip()\n",
        "\n",
        "    test_data[i][\"model_response\"] = response_text\n",
        "\n",
        "\n",
        "with open(\"instruction-data-with-response.json\", \"w\") as file:\n",
        "    json.dump(test_data, file, indent=4)  # \"indent\" for pretty-printing"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "228d6fa7-d162-44c3-bef1-4013c027b155",
      "metadata": {
        "id": "228d6fa7-d162-44c3-bef1-4013c027b155"
      },
      "source": [
        "- Let's double-check one of the entries to see whether the responses have been added to the `test_set` dictionary correctly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "u-AvCCMTnPSE",
      "metadata": {
        "id": "u-AvCCMTnPSE"
      },
      "outputs": [],
      "source": [
        "print(test_data[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c1b2f3f6-8569-405a-9db6-d47cba65608a",
      "metadata": {
        "id": "c1b2f3f6-8569-405a-9db6-d47cba65608a"
      },
      "source": [
        "- Finally, we also save the model in case we want to reuse it in the future"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8cBU0iHmVfOI",
      "metadata": {
        "id": "8cBU0iHmVfOI"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "\n",
        "model.cpu()\n",
        "file_name = f\"{re.sub(r'[ ()]', '', CHOOSE_MODEL) }-sft.pth\"\n",
        "torch.save(model.state_dict(), file_name)\n",
        "print(f\"Model saved as {file_name}\")\n",
        "\n",
        "# Load model via\n",
        "# model.load_state_dict(torch.load(\"gpt2-medium355M-sft.pth\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "obgoGI89dgPm",
      "metadata": {
        "id": "obgoGI89dgPm"
      },
      "source": [
        "## 7.8 Evaluating the finetuned LLM"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "805b9d30-7336-499f-abb5-4a21be3129f5",
      "metadata": {
        "id": "805b9d30-7336-499f-abb5-4a21be3129f5"
      },
      "source": [
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/chapter-overview-7.webp\" width=500px>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "68d2b9d3-b6ff-4533-a89d-7b66079b4fd1",
      "metadata": {
        "id": "68d2b9d3-b6ff-4533-a89d-7b66079b4fd1"
      },
      "source": [
        "- In this section, we automate the response evaluation of the finetuned LLM using another, larger LLM\n",
        "- In particular, we use an instruction-finetuned 8 billion parameter Llama 3 model by Meta AI that can be run locally via ollama ([https://ollama.com](https://ollama.com))\n",
        "- (Alternatively, if you prefer using a more capable LLM like OpenAI's GPT-4 via the ChatGPT API, please see the [../03_model-evaluation/llm-instruction-eval-ollama.ipynb](../03_model-evaluation/llm-instruction-eval-ollama.ipynb) notebook)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ea427a30-36ba-44e3-bb1f-eb0d7008d6e9",
      "metadata": {
        "id": "ea427a30-36ba-44e3-bb1f-eb0d7008d6e9"
      },
      "source": [
        "- Ollama is an application to run LLMs efficiently\n",
        "- It is a wrapper around llama.cpp ([https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)), which implements LLMs in pure C/C++ to maximize efficiency\n",
        "- Note that it is a tool for using LLMs to generate text (inference), not training or finetuning LLMs\n",
        "- Before running the code below, install ollama by visiting [https://ollama.com](https://ollama.com) and following the instructions (for instance, clicking on the \"Download\" button and downloading the ollama application for your operating system)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "747a2fc7-282d-47ec-a987-ed0a23ed6822",
      "metadata": {
        "id": "747a2fc7-282d-47ec-a987-ed0a23ed6822"
      },
      "source": [
        "- For macOS and Windows users, click on the ollama application you downloaded; if it prompts you to install the command line usage, say \"yes\"\n",
        "- Linux users can use the installation command provided on the ollama website\n",
        "\n",
        "- In general, before we can use ollama from the command line, we have to either start the ollama application or run `ollama serve` in a separate terminal\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/ollama-run.webp\" width=700px>\n",
        "\n",
        "\n",
        "- With the ollama application or `ollama serve` running in a different terminal, on the command line, execute the following command to try out the 8 billion parameters Llama 3 model (the model, which takes up 4.7 GB of storage space, will be automatically downloaded the first time you execute this command)\n",
        "\n",
        "```bash\n",
        "# 8B model\n",
        "ollama run llama3\n",
        "```\n",
        "\n",
        "\n",
        "The output looks like as follows\n",
        "\n",
        "```\n",
        "$ ollama run llama3\n",
        "pulling manifest\n",
        "pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB\n",
        "pulling 4fa551d4f938... 100% ▕████████████████▏  12 KB\n",
        "pulling 8ab4849b038c... 100% ▕████████████████▏  254 B\n",
        "pulling 577073ffcc6c... 100% ▕████████████████▏  110 B\n",
        "pulling 3f8eb4da87fa... 100% ▕████████████████▏  485 B\n",
        "verifying sha256 digest\n",
        "writing manifest\n",
        "removing any unused layers\n",
        "success\n",
        "```\n",
        "\n",
        "- Note that `llama3` refers to the instruction finetuned 8 billion Llama 3 model\n",
        "\n",
        "-  Using ollama with the `\"llama3\"` model (a 8B parameter model) requires 16 GB of RAM; if this is not supported by your machine, you can try the smaller model, such as the 3.8B parameter phi-3 model by setting `model = \"phi-3\"`, which only requires 8 Gb of RAM\n",
        "\n",
        "- Alternatively, you can also use the larger 70 billion parameters Llama 3 model, if your machine supports it, by replacing `llama3` with `llama3:70b`\n",
        "\n",
        "- After the download has been completed, you will see a command line prompt that allows you to chat with the model\n",
        "\n",
        "-  Try a prompt like \"What do llamas eat?\", which should return an output similar to the following\n",
        "\n",
        "```\n",
        ">>> What do llamas eat?\n",
        "Llamas are ruminant animals, which means they have a four-chambered\n",
        "stomach and eat plants that are high in fiber. In the wild, llamas\n",
        "typically feed on:\n",
        "1. Grasses: They love to graze on various types of grasses, including tall\n",
        "grasses, wheat, oats, and barley.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7b7b341c-ba0e-40bb-a52c-cb328bbd1fe4",
      "metadata": {
        "id": "7b7b341c-ba0e-40bb-a52c-cb328bbd1fe4"
      },
      "source": [
        "- You can end this session using the input `/bye`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "faaf3e02-8ca0-4edf-be23-60625a5b14e3",
      "metadata": {
        "id": "faaf3e02-8ca0-4edf-be23-60625a5b14e3"
      },
      "source": [
        "- The following code checks whether the ollama session is running correctly before proceeding to use ollama to evaluate the test set responses we generated in the previous section"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "026e8570-071e-48a2-aa38-64d7be35f288",
      "metadata": {
        "id": "026e8570-071e-48a2-aa38-64d7be35f288"
      },
      "outputs": [],
      "source": [
        "import psutil\n",
        "\n",
        "def check_if_running(process_name):\n",
        "    running = False\n",
        "    for proc in psutil.process_iter([\"name\"]):\n",
        "        if process_name in proc.info[\"name\"]:\n",
        "            running = True\n",
        "            break\n",
        "    return running\n",
        "\n",
        "ollama_running = check_if_running(\"ollama\")\n",
        "\n",
        "if not ollama_running:\n",
        "    raise RuntimeError(\"Ollama not running. Launch ollama before proceeding.\")\n",
        "print(\"Ollama running:\", check_if_running(\"ollama\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "723c9b00-e3cd-4092-83c3-6e48b5cf65b0",
      "metadata": {
        "id": "723c9b00-e3cd-4092-83c3-6e48b5cf65b0"
      },
      "outputs": [],
      "source": [
        "# This cell is optional; it allows you to restart the notebook\n",
        "# and only run section 7.7 without rerunning any of the previous code\n",
        "import json\n",
        "from tqdm import tqdm\n",
        "\n",
        "file_path = \"instruction-data-with-response.json\"\n",
        "\n",
        "with open(file_path, \"r\") as file:\n",
        "    test_data = json.load(file)\n",
        "\n",
        "\n",
        "def format_input(entry):\n",
        "    instruction_text = (\n",
        "        f\"Below is an instruction that describes a task. \"\n",
        "        f\"Write a response that appropriately completes the request.\"\n",
        "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
        "    )\n",
        "\n",
        "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
        "\n",
        "    return instruction_text + input_text"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b3464705-d026-4594-977f-fb357e51c3a9",
      "metadata": {
        "id": "b3464705-d026-4594-977f-fb357e51c3a9"
      },
      "source": [
        "- Now, an alternative way to the `ollama run` command we used earlier to interact with the model is via its REST API in Python via the following function\n",
        "- Before you run the next cells in this notebook, make sure that ollama is still running (the previous code cells should print `\"Ollama running: True\"`)\n",
        "- Next, run the following code cell to query the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e3ae0e10-2b28-42ce-8ea2-d9366a58088f",
      "metadata": {
        "id": "e3ae0e10-2b28-42ce-8ea2-d9366a58088f"
      },
      "outputs": [],
      "source": [
        "import urllib.request\n",
        "\n",
        "def query_model(prompt, model=\"llama3\", url=\"http://localhost:11434/api/chat\"):\n",
        "    # Create the data payload as a dictionary\n",
        "    data = {\n",
        "        \"model\": model,\n",
        "        \"seed\": 123,        # for deterministic responses\n",
        "        \"temperature\": 0,   # for deterministic responses\n",
        "        \"messages\": [\n",
        "            {\"role\": \"user\", \"content\": prompt}\n",
        "        ]\n",
        "    }\n",
        "\n",
        "    # Convert the dictionary to a JSON formatted string and encode it to bytes\n",
        "    payload = json.dumps(data).encode(\"utf-8\")\n",
        "\n",
        "    # Create a request object, setting the method to POST and adding necessary headers\n",
        "    request = urllib.request.Request(url, data=payload, method=\"POST\")\n",
        "    request.add_header(\"Content-Type\", \"application/json\")\n",
        "\n",
        "    # Send the request and capture the response\n",
        "    response_data = \"\"\n",
        "    with urllib.request.urlopen(request) as response:\n",
        "        # Read and decode the response\n",
        "        while True:\n",
        "            line = response.readline().decode(\"utf-8\")\n",
        "            if not line:\n",
        "                break\n",
        "            response_json = json.loads(line)\n",
        "            response_data += response_json[\"message\"][\"content\"]\n",
        "\n",
        "    return response_data\n",
        "\n",
        "\n",
        "model = \"llama3\"\n",
        "result = query_model(\"What do Llamas eat?\", model)\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "207ae28f-0f8c-4fda-aeef-e7e3046249cc",
      "metadata": {
        "id": "207ae28f-0f8c-4fda-aeef-e7e3046249cc"
      },
      "source": [
        "- Now, using the `query_model` function we defined above, we can evaluate the responses of our finetuned model; let's try it out on the first 3 test set responses we looked at in a previous section"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "86b839d4-064d-4178-b2d7-01691b452e5e",
      "metadata": {
        "id": "86b839d4-064d-4178-b2d7-01691b452e5e"
      },
      "outputs": [],
      "source": [
        "for entry in test_data[:3]:\n",
        "    prompt = (\n",
        "        f\"Given the input `{format_input(entry)}` \"\n",
        "        f\"and correct output `{entry['output']}`, \"\n",
        "        f\"score the model response `{entry['model_response']}`\"\n",
        "        f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
        "    )\n",
        "    print(\"\\nDataset response:\")\n",
        "    print(\">>\", entry['output'])\n",
        "    print(\"\\nModel response:\")\n",
        "    print(\">>\", entry[\"model_response\"])\n",
        "    print(\"\\nScore:\")\n",
        "    print(\">>\", query_model(prompt))\n",
        "    print(\"\\n-------------------------\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b114fd65-9cfb-45f6-ab74-8331da136bf3",
      "metadata": {
        "id": "b114fd65-9cfb-45f6-ab74-8331da136bf3"
      },
      "source": [
        "- As we can see, the Llama 3 model provides a reasonable evaluation and also gives partial points if a model is not entirely correct, as we can see based on the \"cumulus cloud\" answer\n",
        "- Note that the previous prompt returns very verbose evaluations; we can tweak the prompt to generate integer responses in the range between 0 and 100 (where 100 is best) to calculate an average score for our model\n",
        "- The evaluation of the 110 entries in the test set takes about 1 minute on an M3 MacBook Air laptop"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9d7bca69-97c4-47a5-9aa0-32f116fa37eb",
      "metadata": {
        "id": "9d7bca69-97c4-47a5-9aa0-32f116fa37eb"
      },
      "outputs": [],
      "source": [
        "def generate_model_scores(json_data, json_key):\n",
        "    scores = []\n",
        "    for entry in tqdm(json_data, desc=\"Scoring entries\"):\n",
        "        prompt = (\n",
        "            f\"Given the input `{format_input(entry)}` \"\n",
        "            f\"and correct output `{entry['output']}`, \"\n",
        "            f\"score the model response `{entry[json_key]}`\"\n",
        "            f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
        "            f\"Respond with the integer number only.\"\n",
        "        )\n",
        "        score = query_model(prompt)\n",
        "        try:\n",
        "            scores.append(int(score))\n",
        "        except ValueError:\n",
        "            print(f\"Could not convert score: {score}\")\n",
        "            continue\n",
        "\n",
        "    return scores\n",
        "\n",
        "\n",
        "scores = generate_model_scores(test_data, \"model_response\")\n",
        "print(f\"Number of scores: {len(scores)} of {len(test_data)}\")\n",
        "print(f\"Average score: {sum(scores)/len(scores):.2f}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "407f08d5-9ada-4301-9ebc-f0533c76d3f2",
      "metadata": {
        "id": "407f08d5-9ada-4301-9ebc-f0533c76d3f2"
      },
      "source": [
        "- Our model achieves an average score of above 50, which we can use as a reference point to compare the model to other models or to try out other training settings that may improve the model\n",
        "- Note that ollama is not fully deterministic (as of this writing), so the numbers you are getting might slightly differ from the ones shown above"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6408768b-2784-44f1-b48e-aed0c1eb9b94",
      "metadata": {
        "id": "6408768b-2784-44f1-b48e-aed0c1eb9b94"
      },
      "source": [
        "- For reference, the original\n",
        "  - Llama 3 8B base model achieves a score of 58.51\n",
        "  - Llama 3 8B instruct model achieves a score of 82.65"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "412d7325-284a-446c-92a1-5aa8acc52dee",
      "metadata": {
        "id": "412d7325-284a-446c-92a1-5aa8acc52dee"
      },
      "source": [
        "## 7.9 Conclusions"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tIbNMluCDjVM",
      "metadata": {
        "id": "tIbNMluCDjVM"
      },
      "source": [
        "### 7.9.1 What's next\n",
        "\n",
        "- This marks the final chapter of this book\n",
        "- We covered the major steps of the LLM development cycle: implementing an LLM architecture, pretraining an LLM, and finetuning it\n",
        "\n",
        "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch07_compressed/final-overview.webp\" width=500px>\n",
        "\n",
        "- An optional step that is sometimes followed after instruction finetuning, as described in this chapter, is preference finetuning\n",
        "- Preference finetuning process can be particularly useful for customizing a model to better align with specific user preferences; see the [../04_preference-tuning-with-dpo](../04_preference-tuning-with-dpo) folder if you are interested in this\n",
        "\n",
        "- This GitHub repository also contains a large selection of additional bonus material you may enjoy; for more information, please see the [Bonus Material](https://github.com/rasbt/LLMs-from-scratch?tab=readme-ov-file#bonus-material) section on this repository's README page\n",
        "\n",
        "### 7.9.2 Staying up to date in a fast-moving field\n",
        "\n",
        "- No code in this section\n",
        "\n",
        "### 7.9.3 Final words\n",
        "\n",
        "- I hope you enjoyed this journey of implementing an LLM from the ground up and coding the pretraining and finetuning functions\n",
        "- In my opinion, implementing an LLM from scratch is the best way to understand how LLMs work; I hope you gained a better understanding through this approach\n",
        "- While this book serves educational purposes, you may be interested in using different and more powerful LLMs for real-world applications\n",
        "  - For this, you may consider popular tools such as axolotl ([https://github.com/OpenAccess-AI-Collective/axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)) or LitGPT ([https://github.com/Lightning-AI/litgpt](https://github.com/Lightning-AI/litgpt), which I help developing"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f9853e7f-a81a-4806-9728-be1690807185",
      "metadata": {
        "id": "f9853e7f-a81a-4806-9728-be1690807185"
      },
      "source": [
        "## Summary\n",
        "\n",
        "- No code in this section"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}