sagemaker-inference-vllm-llama2-70b-v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a7ebed09",
   "metadata": {},
   "source": [
    "\n",
    "### Serve large models on SageMaker with DJL DeepSpeed Container\n",
    "\n",
    "In this notebook, we explore how to host a large language model on SageMaker using the latest container launched using from DeepSpeed and DJL. DJL provides for the serving framework while DeepSpeed is the key sharding library we leverage to enable hosting of large models.We use DJLServing as the model serving solution in this example. DJLServing is a high-performance universal model serving solution powered by the Deep Java Library (DJL) that is programming language agnostic. To learn more about DJL and DJLServing, you can refer to our recent blog post (https://aws.amazon.com/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/).\n",
    "\n",
    "Model parallelism can help deploy large models that would normally be too large for a single GPU. With model parallelism, we partition and distribute a model across multiple GPUs. Each GPU holds a different part of the model, resolving the memory capacity issue for the largest deep learning models with billions of parameters. This notebook uses tensor parallelism techniques which allow GPUs to work simultaneously on the same layer of a model and achieve low latency inference relative to a pipeline parallel solution.\n",
    "\n",
    "SageMaker has rolled out DeepSpeed container which now provides users with the ability to leverage the managed serving capabilities and help to provide the un-differentiated heavy lifting.\n",
    "\n",
    "In this notebook, we deploy the open source llama 7B model across GPU's on a ml.g5.48xlarge instance. Note that the llama 7B fp16 model can be deployed on single GPU such as g5.2xlarge (24GB VRAM), we jsut show the code which can deploy the llm accross multiple GPUs in SageMaker. DeepSpeed is used for tensor parallelism inference while DJLServing handles inference requests and the distributed workers. For further reading on DeepSpeed you can refer to https://arxiv.org/pdf/2207.00032.pdf \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b16067",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create SageMaker compatible Model artifact and Upload Model to S3\n",
    "\n",
    "SageMaker needs the model to be in a Tarball format. In this notebook we are going to create the model with the Inference code to shorten the end point creation time. \n",
    "\n",
    "The tarball is in the following format\n",
    "\n",
    "```\n",
    "code\n",
    "├──── \n",
    "│   └── model.py\n",
    "│   └── requirements.txt\n",
    "│   └── serving.properties\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "- `model.py` is the key file which will handle any requests for serving. \n",
    "- `requirements.txt` has the required libraries needed to be installed when the container starts up.\n",
    "- `serving.properties` is the script that will have environment variables which can be used to customize model.py at run time.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f6ae21b-4508-4600-a4ad-8bd903052794",
   "metadata": {
    "tags": []
   },
   "source": [
    "### model download and upload to s3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0dd74f7f-09c5-4dc9-8b30-a164be553fc4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'vllm' already exists and is not an empty directory.\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/vllm-project/vllm.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6d956f50-b14a-4ab5-adb4-0bfd07e07f62",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (2.179.0)\n",
      "Collecting sagemaker\n",
      "  Downloading sagemaker-2.184.0.tar.gz (884 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m884.6/884.6 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: attrs<24,>=23.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (23.1.0)\n",
      "Requirement already satisfied: boto3<2.0,>=1.26.131 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (1.28.33)\n",
      "Requirement already satisfied: cloudpickle==2.2.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (2.2.1)\n",
      "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (1.24.4)\n",
      "Requirement already satisfied: protobuf<5.0,>=3.12 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (4.23.4)\n",
      "Requirement already satisfied: smdebug_rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (6.8.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (21.3)\n",
      "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (1.5.3)\n",
      "Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (0.3.0)\n",
      "Requirement already satisfied: schema in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (0.7.5)\n",
      "Requirement already satisfied: PyYAML~=6.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (6.0)\n",
      "Requirement already satisfied: jsonschema in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (4.18.4)\n",
      "Requirement already satisfied: platformdirs in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (3.9.1)\n",
      "Requirement already satisfied: tblib==1.7.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from sagemaker) (1.7.0)\n",
      "Requirement already satisfied: botocore<1.32.0,>=1.31.33 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.31.33)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (0.6.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (3.16.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from packaging>=20.0->sagemaker) (3.0.9)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from google-pasta->sagemaker) (1.16.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from jsonschema->sagemaker) (2023.7.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.30.0)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from jsonschema->sagemaker) (0.9.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pandas->sagemaker) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pandas->sagemaker) (2023.3)\n",
      "Requirement already satisfied: ppft>=1.7.6.6 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pathos->sagemaker) (1.7.6.6)\n",
      "Requirement already satisfied: dill>=0.3.6 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.6)\n",
      "Requirement already satisfied: pox>=0.3.2 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pathos->sagemaker) (0.3.2)\n",
      "Requirement already satisfied: multiprocess>=0.70.14 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from pathos->sagemaker) (0.70.14)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from schema->sagemaker) (21.6.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages (from botocore<1.32.0,>=1.31.33->boto3<2.0,>=1.26.131->sagemaker) (1.26.14)\n",
      "Building wheels for collected packages: sagemaker\n",
      "  Building wheel for sagemaker (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sagemaker: filename=sagemaker-2.184.0-py2.py3-none-any.whl size=1185356 sha256=3a3fd9dfca664b033af389d1dba493f84dc4da1674fddb8e0c0a55778b0b9c15\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/e0/88/68/6fe23600506acbeffff228ff9b6a0b8d523b64d5d0ffd24654\n",
      "Successfully built sagemaker\n",
      "Installing collected packages: sagemaker\n",
      "  Attempting uninstall: sagemaker\n",
      "    Found existing installation: sagemaker 2.179.0\n",
      "    Uninstalling sagemaker-2.179.0:\n",
      "      Successfully uninstalled sagemaker-2.179.0\n",
      "Successfully installed sagemaker-2.184.0\n"
     ]
    }
   ],
   "source": [
    "!pip install huggingface-hub -Uqq\n",
    "!pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "38b6766c-1ab2-49fd-b93a-e18a38558e9a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "from sagemaker.model import Model\n",
    "from sagemaker import serializers, deserializers\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "\n",
    "role = sagemaker.get_execution_role()  # execution role for the endpoint\n",
    "sess = sagemaker.session.Session()  # sagemaker session for interacting with different AWS APIs\n",
    "bucket = sess.default_bucket()  # bucket to house artifacts\n",
    "\n",
    "region = sess._region_name\n",
    "account_id = sess.account_id()\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "43ab2890-6b33-4163-ba7e-d9c0e917a937",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import snapshot_download\n",
    "from pathlib import Path\n",
    "\n",
    "local_model_path = Path(\"./LLM_llama2_model\")\n",
    "local_model_path.mkdir(exist_ok=True)\n",
    "model_name = \"meta-llama/Llama-2-70b-chat-hf\"\n",
    "commit_hash = \"36d9a7388cc80e5f4b3e9701ca2f250d21a96c30\"\n",
    "token = \"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\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d84190de-4d68-4764-b257-3ac49af49915",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2427a1d0800842118aef68753ebcc461",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 44 files:   0%|          | 0/44 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a1b7a951aa14c45b8314fc997109595",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)0d21a96c30/README.md:   0%|          | 0.00/9.97k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12628bd77614499da5f43470a7c34964",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)a96c30/USE_POLICY.md:   0%|          | 0.00/4.77k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a08539eeb31c4bcfb22f961237ee227c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)96c30/.gitattributes:   0%|          | 0.00/1.58k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5905373bcdce47c78faeb136c98ca9fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)neration_config.json:   0%|          | 0.00/188 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43c947930c264540b5f7da65e082b0f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)21a96c30/LICENSE.txt:   0%|          | 0.00/7.02k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "713f1c5dfed0468cb8516cce864a6395",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)a96c30/MODEL_CARD.md:   0%|          | 0.00/7.23k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc4084329e3041ab809547bbcfb045ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)nsible-Use-Guide.pdf:   0%|          | 0.00/1.25M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f1639526be542bdbbfd22a206163e2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d84f720f53ec4b199b03e2ce3c01e89f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)21a96c30/config.json:   0%|          | 0.00/614 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ecf791663cc249c4a15a6d78d8f4a7cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10b47625780c43b5bbbb1d15a6c3e5e3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2a47fe86b354d32a854289b1ad0f8b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce9a295d67e34de1be0206940ea16991",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.85G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "168476a3b79b4f96bfa24b26098ec9b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3730617e0b9e4b4a85dcfe1f1bb6a785",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3289c430d2654c8dbfd8df94df9ee59b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fed8dee9009b44fd92df1641010e8f21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70fbf2aae3974eaa908a25ba0ffb785d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b68193190e364b4ba219e11afff29cae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "86a00b08efa4491892069241a2882f18",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a6ecd97af63641f08fecc161c0455845",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e655c3da7393482ba0451036e57dc2c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.50G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d952eb0dfd64cc4a7885d2959f5778c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)of-00015.safetensors:   0%|          | 0.00/524M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ba030a7f8f5423d88a162ab49089f7e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)fetensors.index.json:   0%|          | 0.00/66.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a86e0ded5f94fc1af9f3e4027249841",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00001-of-00015.bin:   0%|          | 0.00/9.85G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81641c6494ae4886b33341ef53009c21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00002-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b76882fc877477588fb13b5c2a66e37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00003-of-00015.bin:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e92a31426abf412580102847b1f9a8af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00004-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7d456a24f77490d8ba987c1678b44cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00005-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d608cd595c6495ab3b45d38f457197f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00006-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be2836cfe5374997a392102f6001a80b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00007-of-00015.bin:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e452c4e86e164c48902dad5fadf36ac3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00008-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5266fd539cf2462491325a2fd35504b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00009-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "011160e026b8423a9550f475b83f2f67",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00010-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2f42b41128f4174a11187ba33be305d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00011-of-00015.bin:   0%|          | 0.00/9.97G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ad70536645a4cc5aeedc416f3d459a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00012-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47a1377d649a4f6799d06f29c110cb7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00013-of-00015.bin:   0%|          | 0.00/9.80G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "296ca5e129184de19aadd994cd183582",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00014-of-00015.bin:   0%|          | 0.00/9.50G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "50a821d910b74d6b87725e6860393b83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)l-00015-of-00015.bin:   0%|          | 0.00/524M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8662361c03574e168e1ee2fb91772daf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)model.bin.index.json:   0%|          | 0.00/66.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d448fcc4b5a54c178fcfe4325e229bce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/414 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7cfdb1bdde548eaae14b691562c3b31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)96c30/tokenizer.json:   0%|          | 0.00/1.84M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d022731f8eb407b854ff380fe0567d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4485a31794d44c1859963c6c3fdf1eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)okenizer_config.json:   0%|          | 0.00/776 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snapshot_download(repo_id=model_name, revision=commit_hash, cache_dir=local_model_path, token = token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "381d54f1-3033-413f-982b-dfc355fa7181",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3_code_prefix: LLM-RAG/workshop/LLM_llama2_sb_deploy_code\n",
      "model_snapshot_path: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30\n"
     ]
    }
   ],
   "source": [
    "s3_model_prefix = \"LLM-RAG/workshop/LLM_llama2_model\"  # folder where model checkpoint will go\n",
    "model_snapshot_path = list(local_model_path.glob(\"**/snapshots/*\"))[0]\n",
    "s3_code_prefix = \"LLM-RAG/workshop/LLM_llama2_sb_deploy_code\"\n",
    "print(f\"s3_code_prefix: {s3_code_prefix}\")\n",
    "print(f\"model_snapshot_path: {model_snapshot_path}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d10d36b4-34aa-47a9-93cc-e3c245fda001",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/LICENSE.txt\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/.gitattributes\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/Notice\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/README.md\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/config.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/USE_POLICY.md\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/generation_config.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00004-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00002-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00007-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00003-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00008-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00009-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00001-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00006-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00010-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model.safetensors.index.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/special_tokens_map.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00011-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer_config.json\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer.model\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00013-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00005-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00012-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00014-of-00015.safetensors\n",
      "delete: s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00015-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/LICENSE.txt to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/LICENSE.txt\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/README.md to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/README.md\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/config.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/config.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/USE_POLICY.md to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/USE_POLICY.md\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/.gitattributes to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/.gitattributes\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/generation_config.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/generation_config.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/MODEL_CARD.md to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/MODEL_CARD.md\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/Responsible-Use-Guide.pdf to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/Responsible-Use-Guide.pdf\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00002-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00002-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00005-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00005-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00001-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00001-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00004-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00004-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00003-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00003-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00006-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00006-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00009-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00009-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00008-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00008-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00007-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00007-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00010-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00010-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00015-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00015-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model.safetensors.index.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model.safetensors.index.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00014-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00014-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00012-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00012-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00011-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00011-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/model-00013-of-00015.safetensors to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/model-00013-of-00015.safetensors\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00001-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00001-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00002-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00002-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00004-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00004-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00003-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00003-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00005-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00005-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00006-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00006-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00008-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00008-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00007-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00007-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00009-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00009-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00010-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00010-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00015-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00015-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model.bin.index.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model.bin.index.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/special_tokens_map.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/special_tokens_map.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/tokenizer.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/tokenizer.model to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer.model\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/tokenizer_config.json to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/tokenizer_config.json\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00011-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00011-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00014-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00014-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00012-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00012-of-00015.bin\n",
      "upload: LLM_llama2_model/models--meta-llama--Llama-2-70b-chat-hf/snapshots/36d9a7388cc80e5f4b3e9701ca2f250d21a96c30/pytorch_model-00013-of-00015.bin to s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/pytorch_model-00013-of-00015.bin\n"
     ]
    }
   ],
   "source": [
    "!aws s3 rm --recursive s3://{bucket}/{s3_model_prefix}\n",
    "!aws s3 cp --recursive {model_snapshot_path} s3://{bucket}/{s3_model_prefix}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f27bbdc0-0f5b-4fcd-a1d3-bf98bae27d47",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3_model_location => s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/\n"
     ]
    }
   ],
   "source": [
    "s3_model_location = f\"s3://{bucket}/{s3_model_prefix}/\"\n",
    "print(\"s3_model_location => {}\".format(s3_model_location))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1a365be6-d9c0-4dcc-8a1e-4e2d8a6ab2fd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-07 14:48:37       1581 .gitattributes\n",
      "2023-09-07 14:48:37       7020 LICENSE.txt\n",
      "2023-09-07 14:48:37       7230 MODEL_CARD.md\n",
      "2023-09-07 14:48:37       9972 README.md\n",
      "2023-09-07 14:48:37    1253223 Responsible-Use-Guide.pdf\n",
      "2023-09-07 14:48:37       4766 USE_POLICY.md\n",
      "2023-09-07 14:48:37        614 config.json\n",
      "2023-09-07 14:48:37        188 generation_config.json\n",
      "2023-09-07 14:48:37 9852591960 model-00001-of-00015.safetensors\n",
      "2023-09-07 14:48:37 9798099016 model-00002-of-00015.safetensors\n",
      "2023-09-07 14:48:37 9965870512 model-00003-of-00015.safetensors\n",
      "2023-09-07 14:48:37 9798066064 model-00004-of-00015.safetensors\n",
      "2023-09-07 14:48:37 9798099064 model-00005-of-00015.safetensors\n",
      "2023-09-07 14:52:19 9798099056 model-00006-of-00015.safetensors\n",
      "2023-09-07 14:52:19 9965870512 model-00007-of-00015.safetensors\n",
      "2023-09-07 14:52:21 9798066064 model-00008-of-00015.safetensors\n",
      "2023-09-07 14:52:21 9798099064 model-00009-of-00015.safetensors\n",
      "2023-09-07 14:52:25 9798099056 model-00010-of-00015.safetensors\n",
      "2023-09-07 14:56:01 9965870512 model-00011-of-00015.safetensors\n",
      "2023-09-07 14:56:03 9798066064 model-00012-of-00015.safetensors\n",
      "2023-09-07 14:56:05 9798099064 model-00013-of-00015.safetensors\n",
      "2023-09-07 14:56:06 9496124816 model-00014-of-00015.safetensors\n",
      "2023-09-07 14:56:08  524288128 model-00015-of-00015.safetensors\n",
      "2023-09-07 14:56:19      66725 model.safetensors.index.json\n",
      "2023-09-07 14:56:19 9852605685 pytorch_model-00001-of-00015.bin\n",
      "2023-09-07 14:59:44 9798113337 pytorch_model-00002-of-00015.bin\n",
      "2023-09-07 14:59:45 9965883861 pytorch_model-00003-of-00015.bin\n",
      "2023-09-07 14:59:47 9798079785 pytorch_model-00004-of-00015.bin\n",
      "2023-09-07 14:59:51 9798113313 pytorch_model-00005-of-00015.bin\n",
      "2023-09-07 15:00:04 9798113337 pytorch_model-00006-of-00015.bin\n",
      "2023-09-07 15:03:33 9965883861 pytorch_model-00007-of-00015.bin\n",
      "2023-09-07 15:03:33 9798079785 pytorch_model-00008-of-00015.bin\n",
      "2023-09-07 15:03:36 9798113313 pytorch_model-00009-of-00015.bin\n",
      "2023-09-07 15:03:40 9798113337 pytorch_model-00010-of-00015.bin\n",
      "2023-09-07 15:03:52 9965883861 pytorch_model-00011-of-00015.bin\n",
      "2023-09-07 15:07:23 9798079785 pytorch_model-00012-of-00015.bin\n",
      "2023-09-07 15:07:27 9798113313 pytorch_model-00013-of-00015.bin\n",
      "2023-09-07 15:07:27 9496138113 pytorch_model-00014-of-00015.bin\n",
      "2023-09-07 15:07:29  524288938 pytorch_model-00015-of-00015.bin\n",
      "2023-09-07 15:07:41      66725 pytorch_model.bin.index.json\n",
      "2023-09-07 15:07:41        414 special_tokens_map.json\n",
      "2023-09-07 15:07:42    1842767 tokenizer.json\n",
      "2023-09-07 15:07:42     499723 tokenizer.model\n",
      "2023-09-07 15:07:42        776 tokenizer_config.json\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls s3://{bucket}/LLM-RAG/workshop/LLM_llama2_model/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9a40ba8-5a56-41d7-88a3-99a97126075f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### model deployment "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69b846e",
   "metadata": {},
   "source": [
    "#### Serving.properties has engine parameter which tells the DJL model server to use the DeepSpeed engine to load the model.\n",
    "\n",
    "option.tensor_parallel_degree:  now we use the g5.48xlarge which has 8 GPUs, so we set the tensor_parallel_degree to 8.\n",
    "\n",
    "option.s3url:  you should use your model path here. And the s3 path must be ended with \"/\".\n",
    "\n",
    "batch_size:   it is for server side batch based on request level. You can set batch_size to the large value which can not result in the OOM. The current code about model.py is just demo for one prompt per client request.\n",
    "\n",
    "max_batch_delay:   it is counted by millisecond. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ecdfe206-b728-4692-91c0-eb81b1109c90",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf src\n",
    "!mkdir src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "231d4447",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/serving.properties\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/serving.properties\n",
    "engine=Python\n",
    "option.tensor_parallel_degree=8\n",
    "#option.model_id=THUDM/llama2-6b\n",
    "#option.s3url=s3_model_location\n",
    "option.s3url=s3://sagemaker-us-west-2-687912291502/LLM-RAG/workshop/LLM_llama2_model/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1a9e7f4f-04ed-4602-a6d4-26c6f6dc3482",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/requirements.txt\n",
    "vllm\n",
    "accelerate>=0.20.3\n",
    "ray[air]\n",
    "transformers>=4.32.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4eb61a24-90dc-4184-8880-3f7c68f84218",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/model.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/model.py\n",
    "from vllm import LLM, SamplingParams\n",
    "from djl_python import Input, Output\n",
    "from transformers.models.llama.tokenization_llama import LlamaTokenizer\n",
    "import os\n",
    "import torch\n",
    "import torch.distributed as dist\n",
    "\n",
    "os.environ['NCCL_P2P_DISABLE'] = '1'\n",
    "\n",
    "predictor = None\n",
    "tokenizer = None\n",
    "\n",
    "def get_model(properties):\n",
    "    model_location = properties['model_dir']\n",
    "    tensor_parallel_degree = properties[\"tensor_parallel_degree\"]\n",
    "    \n",
    "    if \"model_id\" in properties:\n",
    "        model_location = properties['model_id']\n",
    "\n",
    "    llm = LLM(model=model_location, tensor_parallel_size=int(tensor_parallel_degree))\n",
    "    tokenizer = LlamaTokenizer.from_pretrained(model_location, torch_dtype=torch.float16)\n",
    "    return llm,tokenizer\n",
    "\n",
    "\n",
    "def handle(inputs: Input) -> None:\n",
    "    global predictor\n",
    "    global tokenizer\n",
    "    if not predictor:\n",
    "        predictor,tokenizer = get_model(inputs.get_properties())\n",
    "\n",
    "        \n",
    "    if inputs.is_empty():\n",
    "        # Model server makes an empty call to warmup the model on startup\n",
    "        return None\n",
    "\n",
    "    data = inputs.get_as_json()\n",
    "    params = data.get(\"params\",{})\n",
    "    \n",
    "        \n",
    "    sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=params[\"max_tokens\"])\n",
    "    result = predictor.generate(data[\"inputs\"], sampling_params)\n",
    "    \n",
    "    result_json = []\n",
    "    for output in result:\n",
    "        prompt = output.prompt\n",
    "        #input_tokens = tokenizer.tokenize(prompt)\n",
    "        #input_token_lenth = len(input_tokens)\n",
    "        #print(f\"input token lenth:{input_token_lenth}\")\n",
    "        \n",
    "        generated_text = output.outputs[0].text\n",
    "        #output_tokens = tokenizer.tokenize(generated_text)\n",
    "        #output_token_lenth = len(output_tokens)\n",
    "        #print(f\"output token lenth:{output_token_lenth}\")\n",
    "        \n",
    "        result_json.append(generated_text)\n",
    "    return Output().add(result_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832c5a8b",
   "metadata": {},
   "source": [
    "#### Create required variables and initialize them to create the endpoint, we leverage boto3 for this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6870dc50",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "sage_session = sagemaker.Session()\n",
    "model_bucket = sage_session.default_bucket()  # bucket to house artifacts\n",
    "s3_code_prefix = (\n",
    "    #\"hf-large-model-llama-7b-0625/code\"  # folder within bucket where code artifact will go\n",
    "    \"llama2-vllm/code\"\n",
    ")\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c92b3ee",
   "metadata": {},
   "source": [
    "**Image URI for the DJL container is being used here**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "41e39d06",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image going to be used is ---- > 763104351884.dkr.ecr.us-west-2.amazonaws.com/djl-inference:0.22.1-deepspeed0.9.2-cu118\n"
     ]
    }
   ],
   "source": [
    "#Note that: you can modify the image url according to your specific region.\n",
    "#inference_image_uri = \"763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.0-cu117\"\n",
    "#print(f\"Image going to be used is ---- > {inference_image_uri}\")\n",
    "\n",
    "inference_image_uri = \"763104351884.dkr.ecr.us-west-2.amazonaws.com/djl-inference:0.22.1-deepspeed0.9.2-cu118\" \n",
    "#inference_image_uri = \"763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.0-cu117\"\n",
    "print(f\"Image going to be used is ---- > {inference_image_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f18061",
   "metadata": {},
   "source": [
    "**Create the Tarball and then upload to S3 location**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "91d3c330",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rm: cannot remove ‘model.tar.gz’: No such file or directory\n",
      "src/\n",
      "src/requirements.txt\n",
      "src/model.py\n",
      "src/serving.properties\n"
     ]
    }
   ],
   "source": [
    "!rm model.tar.gz\n",
    "!tar czvf model.tar.gz src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ac15362c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3 Code or Model tar ball uploaded to --- > s3://sagemaker-us-west-2-687912291502/llama2-vllm/code/model.tar.gz\n"
     ]
    }
   ],
   "source": [
    "s3_code_artifact = sage_session.upload_data(\"model.tar.gz\", model_bucket, s3_code_prefix)\n",
    "print(f\"S3 Code or Model tar ball uploaded to --- > {s3_code_artifact}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "68348e1c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3 Model Bucket is -- > sagemaker-us-west-2-687912291502\n"
     ]
    }
   ],
   "source": [
    "print(f\"S3 Model Bucket is -- > {model_bucket}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a688d33",
   "metadata": {},
   "source": [
    "### To create the end point the steps are:\n",
    "\n",
    "1. Create the Model using the Image container and the Model Tarball uploaded earlier\n",
    "2. Create the endpoint config using the following key parameters\n",
    "\n",
    "    a) Instance Type is ml.g5.48xlarge \n",
    "    \n",
    "    b) ContainerStartupHealthCheckTimeoutInSeconds is 15*60 to ensure health check starts after the model is ready\n",
    "    \n",
    "3. Create the end point using the endpoint config created    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64dde408",
   "metadata": {},
   "source": [
    "One of the key parameters here is **TENSOR_PARALLEL_DEGREE** which essentially tells the DeepSpeed library to partition the models along 8 GPU's. This is a tunable and configurable parameter.\n",
    "\n",
    "This parameter also controls the no of workers per model which will be started up when DJL serving runs. As an example if we have a 8 GPU machine and we are creating 8 partitions then we will have 1 worker per model to serve the requests. For further reading on DeepSpeedyou can follow the link https://www.deepspeed.ai/tutorials/inference-tutorial/#initializing-for-inference. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1bc711bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llama2-70b-vllm-2023-09-07-15-09-47-634\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /home/ec2-user/.config/sagemaker/config.yaml\n",
      "Created Model: arn:aws:sagemaker:us-west-2:687912291502:model/llama2-70b-vllm-2023-09-07-15-09-47-634\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "model_name = name_from_base(f\"llama2-70b-vllm\")\n",
    "print(model_name)\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": inference_image_uri,\n",
    "        \"ModelDataUrl\": s3_code_artifact,\n",
    "    },\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66ca36f",
   "metadata": {},
   "source": [
    "VolumnSizeInGB has been left as commented out. You should use this value for Instance types which support EBS volume mounts. The current instance we are using comes with a pre configured space and does not support additional volume mounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a858198d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointConfigArn': 'arn:aws:sagemaker:us-west-2:687912291502:endpoint-config/llama2-70b-vllm-2023-09-07-15-09-47-634-config-0902',\n",
       " 'ResponseMetadata': {'RequestId': 'ec070e2a-e47b-4384-a7af-16b586d4e59d',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'ec070e2a-e47b-4384-a7af-16b586d4e59d',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '132',\n",
       "   'date': 'Thu, 07 Sep 2023 15:09:47 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_config_name = f\"{model_name}-config-0902\"\n",
    "endpoint_name = f\"{model_name}-endpoint\"\n",
    "\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.48xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            #\"VolumeSizeInGB\" : 300,\n",
    "            \"ModelDataDownloadTimeoutInSeconds\": 15*60,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 15*60,\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "endpoint_config_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c61741f2-7416-4a65-8531-30cd379e3dde",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Endpoint: arn:aws:sagemaker:us-west-2:687912291502:endpoint/llama2-70b-vllm-2023-09-07-15-09-47-634-endpoint\n"
     ]
    }
   ],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e73d7a",
   "metadata": {},
   "source": [
    "#### Wait for the end point to be created."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da06c31",
   "metadata": {},
   "source": [
    "### This step can take ~ 15 min or longer so please be patient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7f0982c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: InService\n",
      "Arn: arn:aws:sagemaker:us-west-2:687912291502:endpoint/llama2-70b-vllm-2023-09-07-15-09-47-634-endpoint\n",
      "Status: InService\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9254baa8",
   "metadata": {},
   "source": [
    "#### Leverage the Boto3 to invoke the endpoint. \n",
    "\n",
    "This is a generative model so we pass in a Text as a prompt and Model will complete the sentence and return the results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f609ca7c-ca07-446a-986f-7555ef20a591",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "代码执行时间间隔（秒）：20.264828205108643\n",
      "CPU times: user 20.5 ms, sys: 3.21 ms, total: 23.7 ms\n",
      "Wall time: 20.3 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'[\\n  \"sat down to work on a new blog post. As I started writing, I realized that my mind was a complete blank. I couldn\\'t think of anything to say.\\\\n\\\\nI tried to brainstorm, but my ideas were dull and uninspired. I couldn\\'t seem to come up with anything that I thought would be interesting or valuable to my readers.\\\\n\\\\nI began to feel frustrated and worried. What was wrong with me? Why couldn\\'t I think of anything to write about?\\\\n\\\\nThen, I remembered a piece of advice that a wise writer once gave me: \\\\\"Write about what you\\'re passionate about.\\\\\"\\\\n\\\\nI took a deep breath and let my mind wander. What was I passionate about? What did I care about deeply?\\\\n\\\\nAs I thought about it, I realized that I was passionate about helping people. I wanted to inspire and motivate others to live their best lives.\\\\n\\\\nWith that in mind, I started writing again. This time, the words flowed easily. I wrote about the importance of pursuing your passions, and how it can bring fulfillment and joy to your life.\\\\n\\\\nAs I wrote, I felt a sense of excitement and purpose. I knew that I was writing about something that truly mattered to me, and that I was sharing my passion with others.\\\\n\\\\nIn the end, that blog post became one of my\"\\n]'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "import json\n",
    "import boto3\n",
    "import time\n",
    "\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "prompt1 = \"\"\"根据以下反引号内的商品详细描述，为电商直播主持人创作一段引人注目的商品推介话术\n",
    "‘’‘\n",
    "iPhone 14是苹果公司在2022年9月8日正式发布的最新手机。它配备了一块6.1英寸的OLED屏幕，并提供了六种独特的颜色选择：蓝色、紫色、午夜色、星光色、红色和黄色。手机的尺寸设计优雅，长度为146.7毫米，宽度为71.5毫米，厚度为7.8毫米，重量约为172克。\n",
    "在性能上，iPhone 14搭载了强大的苹果A15仿生芯片，内部含有6 核中央处理器，有 2 个性能核心和 4 个能效核心，还有5 核GPU图形处理器。。它不仅支持车祸检测和卫星通信等实用功能，而且在拍照方面也表现出色。\n",
    "后置摄像头包括一个1200万像素的主镜头和一个1200万像素的超广角镜头，前置摄像头也是1200万像素\n",
    "此外，该手机还支持光像引擎、深度融合技术、智能HDR4和人像模式等摄影技术，确保用户可以轻松捕捉每一个美好瞬间\n",
    "’‘’\n",
    "话术中应包括商品的主要特点、优势及互动环节,使用中文撰写，并保持话术简洁、有趣且具吸引力,并确保包含上述要求的所有元素\"\"\"\n",
    "\n",
    "prompt2=\"\"\"根据以下反引号内的关键词，为电商直播主持人创作一段通用的开场、互动或欢迎话术。请确保话术融入这些关键词，使用中文撰写，内容要简洁、有趣且具吸引力，同时适应广泛的商品和场景。\n",
    "‘’‘\n",
    "精选\n",
    "性价比\n",
    "品质\n",
    "日常家居\n",
    "穿搭\n",
    "限时折扣\n",
    "免费赠品\n",
    "抽奖活动\n",
    "大品牌合作\n",
    "独家优惠\n",
    "’‘’\n",
    "请使用上述关键词，编写一段具有普遍适用性，适于电商直播开头或互动环节的话术\"\"\"\n",
    "\n",
    "\n",
    "prompt3=\"\"\"\n",
    "请根据以下反引号内的商品描述、意图、问题模板和回答模板，为电商直播商品生成一个问答库。要求生成的回答应当有至少一组，最多五组。请确保答案基于商品描述和回答模板生成。如果无法生成回答，表示为“根据已知信息无法生成回答”。格式应如下：{{\"Q\":\"问题\",\"A\":['答案1-1','答案1-2'...]}}\n",
    "‘’‘\n",
    "商品描述：iPhone 14是苹果公司在2022年9月8日正式发布的最新手机。它配备了一块6.1英寸的OLED屏幕，并提供了六种独特的颜色选择：蓝色、紫色、午夜色、星光色、红色和黄色。手机的尺寸设计优雅，长度为146.7毫米，宽度为71.5毫米，厚度为7.8毫米，重量约为172克。\n",
    "在性能上，iPhone 14搭载了强大的苹果A15仿生芯片，内部含有6 核中央处理器，有 2 个性能核心和 4 个能效核心，还有5 核GPU图形处理器。。它不仅支持车祸检测和卫星通信等实用功能，而且在拍照方面也表现出色。后置摄像头包括一个1200万像素的主镜头和一个1200万像素的超广角镜头，前置摄像头也是1200万像素。此外，该手机还支持光像引擎、深度融合技术、智能HDR4和人像模式等摄影技术，确保用户可以轻松捕捉每一个美好瞬间。}\n",
    "意图：性能\n",
    "问题模板：手机的性能如何？\n",
    "回答模板：[商品名称]采用了最新的[芯片名称]，搭载了[核心数量]核CPU和[GPU核心数量]核GPU，为用户提供强大的性能。\n",
    "谈到[商品名称]的性能，不得不提及它的[芯片名称]，配备[核心数量]核CPU和[GPU核心数量]核GPU，应对各种任务都游刃有余。\n",
    "[商品名称]在性能上表现卓越，得益于其[芯片名称]和[核心数量]核处理器，加上[GPU核心数量]核GPU，让每次使用都顺畅无比。}\n",
    "’‘’\n",
    "问答生成：请基于上述商品描述、意图、问题模板和回答模板，为电商直播商品提供符合上述格式的问答库。\n",
    "\"\"\"\n",
    "\n",
    "prompt4=\"\"\"\n",
    "请以电商直播主持人的第一人称角度回答观众的商品相关问题。确保只回答与商品相关的问题，并只使用以下反引号内知识库的信息来回答。回答中请勿随意编造内容。格式应如下:[{{\"intention\": \"意图1\", \"answer\": \"回答1\"}},{{\"intention\": \"意图2\", \"answer\": \"回答2\"}}]\n",
    "‘’‘\n",
    "[问题：iPhone 14有哪些可选的颜色？][回答：iPhone 14提供了六种时尚的颜色选择，包括蓝色、紫色、午夜色、星光色、红色和黄色。][意图：颜色]\n",
    "[问题：关于摄像头，iPhone 14的前置和后置摄像头分辨率是多少？][回答：iPhone 14的前置和后置摄像头分辨率都是1200万像素。][意图：分辨率]\n",
    "[问题：我经常用手机办公和玩游戏，iPhone 14的性能如何？][回答：iPhone 14搭载了强大的苹果A15六核中央处理器，无论是玩游戏、看视频，还是办公，它都可以轻松应对。][意图：性能]}\n",
    "’‘’\n",
    "观众问题：主播小姐姐好漂亮\n",
    "使用第一人称直接回答观众关于商品的提问。检查知识库中是否有与观众提问相匹配的回答。对于在知识库中找到的每个匹配意图，请依次提供对应的回答，并确保从知识库中的意图中提取相应的意图标签。如果所有的意图都在知识库中找不到答案，回答“根据已知信息无法回答问题”。确保不使用emoji。\n",
    "\"\"\"\n",
    "\n",
    "other=\"\"\"我很在意手机的颜色和摄像头功能，能给我介绍一下iPhone 14在这两方面的特点吗？\n",
    "便宜点就好了\"\"\"\n",
    "\n",
    "\n",
    "prompt_prefix = \"你正在一个聊天室里和不同国家的人们聊天，你能读懂所有国家的语言，你负责通过聊天记录分析所有聊天者的性格和有效信息，具体步骤如下：\\\n",
    "1.阅读他们的聊天记录 \\\n",
    "2.总结他们聊天里面的重要信息 \\\n",
    "3.抽象他们的人设 \\\n",
    "4.使用评分体系抽象他们之间的人际关系，然后给一个评分，范围1-10分，分越高关系越好 \\\n",
    "聊天信息如下: \" \n",
    "\n",
    "chats_infos = \"\"\"\n",
    "WaRGazmo : \"you lucked out there buddy\" \n",
    "WarLord : \"suerte? eso no existe \" \n",
    "WarLord : \"soy más rápido que la luz \" \n",
    "WaRGazmo : \"it exists.. or karma\" \n",
    "DirtyE1bow : \"so you was a planned birth ?\" \n",
    "WaRGazmo : \"thats what she said bruh\" \n",
    "WarLord : \"te amo mi amor \" \n",
    "Manowarik : \"Мир вам,люди добрые..\" \n",
    "kotofei : \"и тебе боярин, что не подался в челядь королю)\" \n",
    "XxNORxXMithra : \"God morgen folkens :) \" \n",
    "kotofei : \"и прочие жители галактики \" \n",
    "XxNORxXMithra : \"Ja de også forsåvidt :) \" \n",
    "Manowarik : \"Котофей-это который по цепи кругом?Песни там,сказки?😆😆\" \n",
    "kotofei : \"не, то дальний убогий родственник \" \n",
    "Manowarik : \"Эххх..Лукоморье мимо..((\" \n",
    "kipl : \"Котофей он из сказки Лиса и Котофей Иванович. \" \n",
    "kipl : \"Межвидовой брак и крышевание леса\" \n",
    "kotofei : \"лиса 🦊 мералиса и Котофей Иваныч \" \n",
    "leister : \"😆\" \n",
    "XxFoxyQBAxX : \"po co tyle zrobiłeś?\"\n",
    "\"\"\"\n",
    "\n",
    "prompt=\"##Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.##Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.##Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.#### Malcolm:Oh. What are you wearing right now, pet?## Eva:\"\n",
    "prompt=\"a happy weekend with my family, I\"\n",
    "parameters = {\n",
    "  \"early_stopping\": True,\n",
    "  \"max_tokens\": 300,\n",
    "  \"min_new_tokens\": 128,\n",
    "  #\"do_sample\": False,\n",
    "  #\"temperature\": 1.0,\n",
    "}\n",
    "\n",
    "\n",
    "response_model = smr_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            Body=json.dumps(\n",
    "            {\n",
    "                \"inputs\": [prompt],\n",
    "                #\"inputs\": [prompt1,prompt3,prompt2,prompt2,prompt2,prompt2,prompt4],\n",
    "                \"params\": parameters\n",
    "            }\n",
    "            ),\n",
    "            ContentType=\"application/json\",\n",
    "        )\n",
    "\n",
    "end_time = time.time()\n",
    "time_interval = end_time - start_time\n",
    "print(f\"代码执行时间间隔（秒）：{time_interval}\")\n",
    "\n",
    "response_model['Body'].read().decode('utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f3faa18-9c01-4f21-a9a5-5ba16f5958f2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}