tutorial_chat_module_getting_started_13b.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/inchiosa/mlc-llm/blob/main/tutorial_chat_module_getting_started_13b.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cm85Ap3zDmYB"
      },
      "source": [
        "# Getting Started with MLC-LLM using the Llama 2 Model\n",
        "\n",
        "Here's a quick overview of how to get started with the MLC-LLM `ChatModule` in Python. In this tutorial, we will chat with the [Llama2](https://ai.meta.com/llama/) model. For the easiest setup, we recommend trying this out in a Google Colab notebook. Click the button below to get started!\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/mlc-ai/notebooks/blob/main/mlc-llm/tutorial_chat_module_getting_started.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ttPt-hNDmYC"
      },
      "source": [
        "## Environment Setup\n",
        "\n",
        "Let's set up your environment, so you can successfully run the `ChatModule`. First, let's set up the Conda environment which we will be running this notebook in (not required if running in Google Colab).\n",
        "\n",
        "```bash\n",
        "conda create --name mlc-llm python=3.10\n",
        "conda activate mlc-llm\n",
        "```\n",
        "\n",
        "**Google Colab:** If you are running this in a Google Colab notebook, be sure to change your runtime to GPU by going to Runtime > Change runtime type and setting the Hardware accelerator to be \"GPU\". Select \"Connect\" on the top right to instantiate your GPU session.\n",
        "\n",
        "If you are using CUDA, you can run the following command to confirm that CUDA is set up correctly, and check the version number."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "KK25HZsIDmYC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9f26b04c-8cb9-4a56-d512-b60205988214"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Thu Aug 10 21:38:19 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   45C    P8     9W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWOtpjJMDmYE"
      },
      "source": [
        "Next, let's download the MLC-AI and MLC-Chat nightly build packages. Go to https://mlc.ai/package/ and replace the command below with the one that is appropriate for your hardware and OS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "PgW-5OAADmYE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "2da742aa-1a56-4ae9-bc9c-84f6b1b8297f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in links: https://mlc.ai/wheels\n",
            "Collecting mlc-ai-nightly-cu118\n",
            "  Downloading https://github.com/mlc-ai/package/releases/download/v0.9.dev0/mlc_ai_nightly_cu118-0.12.dev1395-cp310-cp310-manylinux_2_28_x86_64.whl (97.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 MB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting mlc-chat-nightly-cu118\n",
            "  Downloading https://github.com/mlc-ai/package/releases/download/v0.9.dev0/mlc_chat_nightly_cu118-0.1.dev351-cp310-cp310-manylinux_2_28_x86_64.whl (20.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.6/20.6 MB\u001b[0m \u001b[31m17.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting attrs (from mlc-ai-nightly-cu118)\n",
            "  Downloading attrs-23.1.0-py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.2/61.2 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cloudpickle (from mlc-ai-nightly-cu118)\n",
            "  Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB)\n",
            "Collecting decorator (from mlc-ai-nightly-cu118)\n",
            "  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)\n",
            "Collecting ml-dtypes (from mlc-ai-nightly-cu118)\n",
            "  Downloading ml_dtypes-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m36.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting numpy (from mlc-ai-nightly-cu118)\n",
            "  Downloading numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m83.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting psutil (from mlc-ai-nightly-cu118)\n",
            "  Downloading psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m282.1/282.1 kB\u001b[0m \u001b[31m25.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scipy (from mlc-ai-nightly-cu118)\n",
            "  Downloading scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m36.3/36.3 MB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tornado (from mlc-ai-nightly-cu118)\n",
            "  Downloading tornado-6.3.2-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (426 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m426.9/426.9 kB\u001b[0m \u001b[31m32.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing-extensions (from mlc-ai-nightly-cu118)\n",
            "  Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)\n",
            "Collecting fastapi (from mlc-chat-nightly-cu118)\n",
            "  Downloading fastapi-0.101.0-py3-none-any.whl (65 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.7/65.7 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting uvicorn (from mlc-chat-nightly-cu118)\n",
            "  Downloading uvicorn-0.23.2-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.5/59.5 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting shortuuid (from mlc-chat-nightly-cu118)\n",
            "  Downloading shortuuid-1.0.11-py3-none-any.whl (10 kB)\n",
            "Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading pydantic-2.1.1-py3-none-any.whl (370 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m370.9/370.9 kB\u001b[0m \u001b[31m38.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting starlette<0.28.0,>=0.27.0 (from fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting click>=7.0 (from uvicorn->mlc-chat-nightly-cu118)\n",
            "  Downloading click-8.1.6-py3-none-any.whl (97 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m13.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting h11>=0.8 (from uvicorn->mlc-chat-nightly-cu118)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting annotated-types>=0.4.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading annotated_types-0.5.0-py3-none-any.whl (11 kB)\n",
            "Collecting pydantic-core==2.4.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading pydantic_core-2.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m87.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading anyio-4.0.0rc1-py3-none-any.whl (82 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.8/82.8 kB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting idna>=2.8 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading idna-3.4-py3-none-any.whl (61 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading sniffio-1.3.0-py3-none-any.whl (10 kB)\n",
            "Collecting exceptiongroup>=1.0.2 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi->mlc-chat-nightly-cu118)\n",
            "  Downloading exceptiongroup-1.1.2-py3-none-any.whl (14 kB)\n",
            "Installing collected packages: typing-extensions, tornado, sniffio, shortuuid, psutil, numpy, idna, h11, exceptiongroup, decorator, cloudpickle, click, attrs, annotated-types, uvicorn, scipy, pydantic-core, ml-dtypes, anyio, starlette, pydantic, mlc-ai-nightly-cu118, fastapi, mlc-chat-nightly-cu118\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.7.1\n",
            "    Uninstalling typing_extensions-4.7.1:\n",
            "      Successfully uninstalled typing_extensions-4.7.1\n",
            "  Attempting uninstall: tornado\n",
            "    Found existing installation: tornado 6.3.1\n",
            "    Uninstalling tornado-6.3.1:\n",
            "      Successfully uninstalled tornado-6.3.1\n",
            "  Attempting uninstall: sniffio\n",
            "    Found existing installation: sniffio 1.3.0\n",
            "    Uninstalling sniffio-1.3.0:\n",
            "      Successfully uninstalled sniffio-1.3.0\n",
            "  Attempting uninstall: psutil\n",
            "    Found existing installation: psutil 5.9.5\n",
            "    Uninstalling psutil-5.9.5:\n",
            "      Successfully uninstalled psutil-5.9.5\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.23.5\n",
            "    Uninstalling numpy-1.23.5:\n",
            "      Successfully uninstalled numpy-1.23.5\n",
            "  Attempting uninstall: idna\n",
            "    Found existing installation: idna 3.4\n",
            "    Uninstalling idna-3.4:\n",
            "      Successfully uninstalled idna-3.4\n",
            "  Attempting uninstall: exceptiongroup\n",
            "    Found existing installation: exceptiongroup 1.1.2\n",
            "    Uninstalling exceptiongroup-1.1.2:\n",
            "      Successfully uninstalled exceptiongroup-1.1.2\n",
            "  Attempting uninstall: decorator\n",
            "    Found existing installation: decorator 4.4.2\n",
            "    Uninstalling decorator-4.4.2:\n",
            "      Successfully uninstalled decorator-4.4.2\n",
            "  Attempting uninstall: cloudpickle\n",
            "    Found existing installation: cloudpickle 2.2.1\n",
            "    Uninstalling cloudpickle-2.2.1:\n",
            "      Successfully uninstalled cloudpickle-2.2.1\n",
            "  Attempting uninstall: click\n",
            "    Found existing installation: click 8.1.6\n",
            "    Uninstalling click-8.1.6:\n",
            "      Successfully uninstalled click-8.1.6\n",
            "  Attempting uninstall: attrs\n",
            "    Found existing installation: attrs 23.1.0\n",
            "    Uninstalling attrs-23.1.0:\n",
            "      Successfully uninstalled attrs-23.1.0\n",
            "  Attempting uninstall: scipy\n",
            "    Found existing installation: scipy 1.10.1\n",
            "    Uninstalling scipy-1.10.1:\n",
            "      Successfully uninstalled scipy-1.10.1\n",
            "  Attempting uninstall: ml-dtypes\n",
            "    Found existing installation: ml-dtypes 0.2.0\n",
            "    Uninstalling ml-dtypes-0.2.0:\n",
            "      Successfully uninstalled ml-dtypes-0.2.0\n",
            "  Attempting uninstall: anyio\n",
            "    Found existing installation: anyio 3.7.1\n",
            "    Uninstalling anyio-3.7.1:\n",
            "      Successfully uninstalled anyio-3.7.1\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 1.10.12\n",
            "    Uninstalling pydantic-1.10.12:\n",
            "      Successfully uninstalled pydantic-1.10.12\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "ipython 7.34.0 requires jedi>=0.16, which is not installed.\n",
            "google-colab 1.0.0 requires tornado==6.3.1, but you have tornado 6.3.2 which is incompatible.\n",
            "inflect 6.0.5 requires pydantic<2,>=1.9.1, but you have pydantic 2.1.1 which is incompatible.\n",
            "jupyter-server 1.24.0 requires anyio<4,>=3.1.0, but you have anyio 4.0.0rc1 which is incompatible.\n",
            "moviepy 1.0.3 requires decorator<5.0,>=4.0.2, but you have decorator 5.1.1 which is incompatible.\n",
            "numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.25.2 which is incompatible.\n",
            "spacy 3.5.4 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.1.1 which is incompatible.\n",
            "tensorflow 2.12.0 requires numpy<1.24,>=1.22, but you have numpy 1.25.2 which is incompatible.\n",
            "thinc 8.1.10 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.1.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed annotated-types-0.5.0 anyio-4.0.0rc1 attrs-23.1.0 click-8.1.6 cloudpickle-2.2.1 decorator-5.1.1 exceptiongroup-1.1.2 fastapi-0.101.0 h11-0.14.0 idna-3.4 ml-dtypes-0.2.0 mlc-ai-nightly-cu118-0.12.dev1395 mlc-chat-nightly-cu118-0.1.dev351 numpy-1.25.2 psutil-5.9.5 pydantic-2.1.1 pydantic-core-2.4.0 scipy-1.11.1 shortuuid-1.0.11 sniffio-1.3.0 starlette-0.27.0 tornado-6.3.2 typing-extensions-4.7.1 uvicorn-0.23.2\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "decorator",
                  "numpy",
                  "psutil",
                  "tornado"
                ]
              }
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "!pip install --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jn7MYEFt5tvY"
      },
      "source": [
        "**Google Colab:** If in Google Colab, you may see a message warning you to restart the runtime. Simply run the following code in a new code cell to restart the runtime.\n",
        "\n",
        "```python\n",
        "import os\n",
        "os.kill(os.getpid(), 9)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.kill(os.getpid(), 9)"
      ],
      "metadata": {
        "id": "ckE5LRRxVEj1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FwsWd1WbDmYE"
      },
      "source": [
        "Next, let's download the model weights for the Llama2 model and the prebuilt model libraries from Github. In order to download the large weights, we'll have to use `git lfs`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppvAhErV3gjq"
      },
      "source": [
        "Note: If you are NOT running in **Google Colab** you may need to run this line `!conda install git git-lfs` to install `git` and `git-lfs` before running the following cell to fully install `git lfs`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "V0GjINnMDmYF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cdfcce9c-6a98-4778-aabf-93375a7570cd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Git LFS initialized.\n"
          ]
        }
      ],
      "source": [
        "!git lfs install"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYwjsCOK7Jij"
      },
      "source": [
        "These commands will download many prebuilt libraries as well as the chat configuration for Llama-2-7b that `mlc_chat` needs, which may take a long time. If in **Google Colab** you can verify that the files are being downloaded by clicking on the folder icon on the left and navigating to the `dist` and then `prebuilt` folders which should be updating as the files are being downloaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "FSAe7Ew_DmYF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4876d89d-0733-4d31-a4ed-01f891b0b014"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'dist/prebuilt/lib'...\n",
            "remote: Enumerating objects: 215, done.\u001b[K\n",
            "remote: Counting objects: 100% (80/80), done.\u001b[K\n",
            "remote: Compressing objects: 100% (50/50), done.\u001b[K\n",
            "remote: Total 215 (delta 55), reused 53 (delta 30), pack-reused 135\u001b[K\n",
            "Receiving objects: 100% (215/215), 51.36 MiB | 14.60 MiB/s, done.\n",
            "Resolving deltas: 100% (148/148), done.\n",
            "Updating files: 100% (57/57), done.\n"
          ]
        }
      ],
      "source": [
        "!mkdir -p dist/prebuilt\n",
        "!git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# model=\"Llama-2-7b-chat-hf-q4f16_1\"\n",
        "model=\"Llama-2-13b-chat-hf-q4f16_1\""
      ],
      "metadata": {
        "id": "0W-OAOJxVMp8"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "BDbi6H3MDmYF"
      },
      "outputs": [],
      "source": [
        "# !cd dist/prebuilt && git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-7b-chat-hf-q4f16_1"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cd dist/prebuilt && git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-13b-chat-hf-q4f16_1"
      ],
      "metadata": {
        "id": "gnv3ZVhzXPN8",
        "outputId": "1b817a9c-dfef-40d8-d1cf-d7a992e8b0f3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'mlc-chat-Llama-2-13b-chat-hf-q4f16_1'...\n",
            "remote: Enumerating objects: 173, done.\u001b[K\n",
            "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2/2), done.\u001b[K\n",
            "remote: Total 173 (delta 0), reused 3 (delta 0), pack-reused 170\u001b[K\n",
            "Receiving objects: 100% (173/173), 27.74 KiB | 6.94 MiB/s, done.\n",
            "Filtering content: 100% (164/164), 6.82 GiB | 44.65 MiB/s, done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76Ru5__tDmYF"
      },
      "source": [
        "## Let's Chat!\n",
        "\n",
        "Before we can chat with the model, we must first import a library and instantiate a `ChatModule` instance. The `ChatModule` must be initialized with the appropriate model name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "AJAt6oW7DmYF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "908f8299-1e01-4789-ee6a-d7caf2121be1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "System automatically detected device: cuda\n",
            "Using model folder: /content/dist/prebuilt/mlc-chat-Llama-2-13b-chat-hf-q4f16_1\n",
            "Using mlc chat config: /content/dist/prebuilt/mlc-chat-Llama-2-13b-chat-hf-q4f16_1/mlc-chat-config.json\n",
            "Using library model: /content/dist/prebuilt/lib/Llama-2-13b-chat-hf-q4f16_1-cuda.so\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from mlc_chat import ChatModule\n",
        "from mlc_chat.callback import StreamToStdout\n",
        "\n",
        "cm = ChatModule(model=model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9m5sxyXDmYF"
      },
      "source": [
        "Note that the above invocation abstracts away the logic for finding the relevant model directory and prebuilt library paths. To specify these manually, you could run the following instead (which would be equivalent to the above).\n",
        "\n",
        "```python\n",
        "cm = ChatModule(model=\"dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1\", lib_path=\"dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so\")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zEaVXnnJDmYF"
      },
      "source": [
        "That is all what needed to set up the `ChatModule`. You can now chat with the model by entering any prompt you'd like. Try it out below!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "TNmg9N_NDmYF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af07b553-9886-4f86-c778-e47ce76e3bf2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Python was first released on February 20, 1991, by Guido van Rossum. It was initially called \"Python\" because the creator was a fan of the British comedy group Monty Python's Flying Circus. The language has since undergone many updates and revisions, with the most recent version, Python 3.10, being released on August 17, 2020.\n"
          ]
        }
      ],
      "source": [
        "output = cm.generate(\n",
        "    prompt=\"When was Python released?\",\n",
        "    progress_callback=StreamToStdout(callback_interval=2),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBnRWJiNUie5"
      },
      "source": [
        "You can also repeat running the code block below for multiple rounds to interact with the model in a chat style."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "qnD3pm5bUie5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4362075b-0b25-4378-90df-a24676d75910"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prompt: Mario Romano was born in\n",
            "I apologize, but there is no record of a person named Mario Romano. It is possible that you may be thinking of a different person or fictional character. If you have any more information or context about Mario Romano, I may be able to assist you better.\n"
          ]
        }
      ],
      "source": [
        "prompt = input(\"Prompt: \")\n",
        "output = cm.generate(prompt=prompt, progress_callback=StreamToStdout(callback_interval=2))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "wWp_2n_BUie6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ebfef190-30fa-43c2-9fbe-ec0b09e1d09f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sure, here is a summary of my previous response in three sentences:\n",
            "\n",
            "There is no record of a person named Mario Romano. If you have any more information or context about Mario Romano, I may be able to assist you better. Please let me know if there is anything else I can help with.\n"
          ]
        }
      ],
      "source": [
        "output = cm.generate(\n",
        "    prompt=\"Please summarize your response in three sentences.\",\n",
        "    progress_callback=StreamToStdout(callback_interval=2),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4bOyUk7DmYF"
      },
      "source": [
        "To check the generation speed of the chat bot, you can print the statistics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "PPbPj6vpDmYF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "586ba5a0-ae8c-4e01-cdc2-5da4fe517f4f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "prefill: 114.9 tok/s, decode: 16.6 tok/s\n"
          ]
        }
      ],
      "source": [
        "print(cm.stats())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAb-XZPnDmYF"
      },
      "source": [
        "By default, the `ChatModule` will keep a history of your chat. You can reset the chat history by running the following."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "iKpKgVxNDmYF"
      },
      "outputs": [],
      "source": [
        "cm.reset_chat()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6g582lzwUie7"
      },
      "source": [
        "### Benchmark Performance\n",
        "\n",
        "To benchmark the performance, we can use the `benchmark_generate` method of ChatModule. It takes an input prompt and the number of tokens to generate, ignores the system prompt and model stop criterion, generates tokens in a language model way and stops until finishing generating the desired number of tokens. After calling `benchmark_generate`, we can use `stats` to check the performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "FvLo2l-_Uie7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "26e8b19e-635d-4ecc-b112-729d10b8f5b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "Benchmarking is the process of comparing one's own performance or quality to that of others, in order to identify areas for improvement. It can be applied to various fields, such as business, sports, education, and healthcare. The purpose of benchmarking is to learn from others, identify best practices, and set goals for improvement.\n",
            "\n",
            "Benchmarking can be performed in various ways, including:\n",
            "\n",
            "1. Competitive benchmarking: This involves comparing one's own performance to that of direct competitors.\n",
            "2. Best practice benchmarking: This involves identifying and comparing one's own performance to the best practices of others in the industry.\n",
            "3. Internal benchmarking: This involves comparing different departments or teams within the same organization to identify areas for improvement.\n",
            "4. External benchmarking: This involves comparing one's own performance to that of others outside of the organization, such as industry averages or national standards.\n",
            "\n",
            "Benchmarking can be a valuable tool for organizations looking to improve their performance and competitiveness. It can help identify areas for improvement, set realistic goals, and provide a framework for measuring progress over time. TEXT:\n",
            "\n",
            "I have been looking into purchasing a new laptop, and I am really interested in the new MacBook Pro. I have read a lot of reviews and watched some videos about it. It seems like a great laptop, but I am worried about the price. I don't want to spend too much money on a laptop, but I also want to make sure I am getting a good quality one. Do you have any advice on this?\n",
            "\n",
            "Hello! I can definitely help you with your decision. The new MacBook Pro is a great laptop, but as you mentioned, it can be a bit pricey. Here are some things to consider that might help you make a decision:\n",
            "\n",
            "1. Check the specs: Make sure the MacBook Pro you are looking at has the specs you need. If you plan to use it for heavy-duty tasks like video editing or gaming, you'll want to look for a model with a more powerful processor and more memory.\n",
            "\n",
            "2. Consider the competition: There are other laptops on the market that offer similar features and performance to the MacBook Pro, but at a lower price point. Do some research to see if there are any other laptops that might be a better value for your money\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'prefill: 27.0 tok/s, decode: 16.6 tok/s'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "print(cm.benchmark_generate(prompt=\"What is benchmark?\", generate_length=512))\n",
        "cm.stats()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}