sagemaker-inference-vllm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a7ebed09",
   "metadata": {},
   "source": [
    "\n",
    "### Serve large models on SageMaker with DJL DeepSpeed Container\n",
    "\n",
    "In this notebook, we explore how to host a large language model on SageMaker using the latest container launched using from DeepSpeed and DJL. DJL provides for the serving framework while DeepSpeed is the key sharding library we leverage to enable hosting of large models.We use DJLServing as the model serving solution in this example. DJLServing is a high-performance universal model serving solution powered by the Deep Java Library (DJL) that is programming language agnostic. To learn more about DJL and DJLServing, you can refer to our recent blog post (https://aws.amazon.com/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/).\n",
    "\n",
    "Model parallelism can help deploy large models that would normally be too large for a single GPU. With model parallelism, we partition and distribute a model across multiple GPUs. Each GPU holds a different part of the model, resolving the memory capacity issue for the largest deep learning models with billions of parameters. This notebook uses tensor parallelism techniques which allow GPUs to work simultaneously on the same layer of a model and achieve low latency inference relative to a pipeline parallel solution.\n",
    "\n",
    "SageMaker has rolled out DeepSpeed container which now provides users with the ability to leverage the managed serving capabilities and help to provide the un-differentiated heavy lifting.\n",
    "\n",
    "In this notebook, we deploy the open source llama 7B model across GPU's on a ml.g5.48xlarge instance. Note that the llama 7B fp16 model can be deployed on single GPU such as g5.2xlarge (24GB VRAM), we jsut show the code which can deploy the llm accross multiple GPUs in SageMaker. DeepSpeed is used for tensor parallelism inference while DJLServing handles inference requests and the distributed workers. For further reading on DeepSpeed you can refer to https://arxiv.org/pdf/2207.00032.pdf \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b16067",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create SageMaker compatible Model artifact and Upload Model to S3\n",
    "\n",
    "SageMaker needs the model to be in a Tarball format. In this notebook we are going to create the model with the Inference code to shorten the end point creation time. \n",
    "\n",
    "The tarball is in the following format\n",
    "\n",
    "```\n",
    "code\n",
    "├──── \n",
    "│   └── model.py\n",
    "│   └── requirements.txt\n",
    "│   └── serving.properties\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "- `model.py` is the key file which will handle any requests for serving. \n",
    "- `requirements.txt` has the required libraries needed to be installed when the container starts up.\n",
    "- `serving.properties` is the script that will have environment variables which can be used to customize model.py at run time.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69b846e",
   "metadata": {},
   "source": [
    "#### Serving.properties has engine parameter which tells the DJL model server to use the DeepSpeed engine to load the model.\n",
    "\n",
    "option.tensor_parallel_degree:  now we use the g5.48xlarge which has 8 GPUs, so we set the tensor_parallel_degree to 8.\n",
    "\n",
    "option.s3url:  you should use your model path here. And the s3 path must be ended with \"/\".\n",
    "\n",
    "batch_size:   it is for server side batch based on request level. You can set batch_size to the large value which can not result in the OOM. The current code about model.py is just demo for one prompt per client request.\n",
    "\n",
    "max_batch_delay:   it is counted by millisecond. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "ecdfe206-b728-4692-91c0-eb81b1109c90",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf src\n",
    "!mkdir src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "231d4447",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/serving.properties\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/serving.properties\n",
    "engine=Python\n",
    "option.tensor_parallel_degree=4\n",
    "#option.model_id=huggyllama/llama-7b\n",
    "option.model_id=huggyllama/llama-13b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "1a9e7f4f-04ed-4602-a6d4-26c6f6dc3482",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/requirements.txt\n",
    "vllm==0.1.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "4eb61a24-90dc-4184-8880-3f7c68f84218",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./src/model.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./src/model.py\n",
    "from vllm import LLM, SamplingParams\n",
    "from djl_python import Input, Output\n",
    "import os\n",
    "\n",
    "os.environ['NCCL_P2P_DISABLE'] = '1'\n",
    "\n",
    "predictor = None\n",
    "\n",
    "def get_model(properties):\n",
    "    model_name = properties[\"model_id\"]\n",
    "    tensor_parallel_degree = int(properties[\"tensor_parallel_degree\"])\n",
    "    llm = LLM(model=model_name, tensor_parallel_size=tensor_parallel_degree)\n",
    "    return llm\n",
    "\n",
    "\n",
    "def handle(inputs: Input) -> None:\n",
    "    global predictor\n",
    "    if not predictor:\n",
    "        predictor = get_model(inputs.get_properties())\n",
    "\n",
    "    if inputs.is_empty():\n",
    "        # Model server makes an empty call to warmup the model on startup\n",
    "        return None\n",
    "\n",
    "    data = inputs.get_as_json()\n",
    "    params = data.get(\"params\",{})\n",
    "    sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=params[\"max_tokens\"])\n",
    "    result = predictor.generate(data[\"inputs\"], sampling_params)\n",
    "    result_json = []\n",
    "    for output in result:\n",
    "        prompt = output.prompt\n",
    "        generated_text = output.outputs[0].text\n",
    "        result_json.append(generated_text)\n",
    "    return Output().add(result_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832c5a8b",
   "metadata": {},
   "source": [
    "#### Create required variables and initialize them to create the endpoint, we leverage boto3 for this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "6870dc50",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "sage_session = sagemaker.Session()\n",
    "model_bucket = sage_session.default_bucket()  # bucket to house artifacts\n",
    "s3_code_prefix = (\n",
    "    \"hf-large-model-llama-7b-0625/code\"  # folder within bucket where code artifact will go\n",
    ")\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c92b3ee",
   "metadata": {},
   "source": [
    "**Image URI for the DJL container is being used here**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "41e39d06",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image going to be used is ---- > 763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.22.1-deepspeed0.9.2-cu118\n"
     ]
    }
   ],
   "source": [
    "#Note that: you can modify the image url according to your specific region.\n",
    "#inference_image_uri = \"763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.0-cu117\"\n",
    "#print(f\"Image going to be used is ---- > {inference_image_uri}\")\n",
    "\n",
    "inference_image_uri = \"763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.22.1-deepspeed0.9.2-cu118\" \n",
    "#inference_image_uri = \"763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:0.21.0-deepspeed0.8.0-cu117\"\n",
    "print(f\"Image going to be used is ---- > {inference_image_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f18061",
   "metadata": {},
   "source": [
    "**Create the Tarball and then upload to S3 location**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "91d3c330",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "src/\n",
      "src/model.py\n",
      "src/requirements.txt\n",
      "src/.ipynb_checkpoints/\n",
      "src/.ipynb_checkpoints/serving-checkpoint.properties\n",
      "src/.ipynb_checkpoints/model-checkpoint.py\n",
      "src/.ipynb_checkpoints/requirements-checkpoint.txt\n",
      "src/serving.properties\n"
     ]
    }
   ],
   "source": [
    "!rm model.tar.gz\n",
    "!tar czvf model.tar.gz src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "ac15362c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3 Code or Model tar ball uploaded to --- > s3://sagemaker-us-east-1-514385905925/hf-large-model-llama-7b-0625/code/model.tar.gz\n"
     ]
    }
   ],
   "source": [
    "s3_code_artifact = sage_session.upload_data(\"model.tar.gz\", model_bucket, s3_code_prefix)\n",
    "print(f\"S3 Code or Model tar ball uploaded to --- > {s3_code_artifact}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "68348e1c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3 Model Bucket is -- > sagemaker-us-east-1-514385905925\n"
     ]
    }
   ],
   "source": [
    "print(f\"S3 Model Bucket is -- > {model_bucket}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a688d33",
   "metadata": {},
   "source": [
    "### To create the end point the steps are:\n",
    "\n",
    "1. Create the Model using the Image container and the Model Tarball uploaded earlier\n",
    "2. Create the endpoint config using the following key parameters\n",
    "\n",
    "    a) Instance Type is ml.g5.48xlarge \n",
    "    \n",
    "    b) ContainerStartupHealthCheckTimeoutInSeconds is 15*60 to ensure health check starts after the model is ready\n",
    "    \n",
    "3. Create the end point using the endpoint config created    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64dde408",
   "metadata": {},
   "source": [
    "One of the key parameters here is **TENSOR_PARALLEL_DEGREE** which essentially tells the DeepSpeed library to partition the models along 8 GPU's. This is a tunable and configurable parameter.\n",
    "\n",
    "This parameter also controls the no of workers per model which will be started up when DJL serving runs. As an example if we have a 8 GPU machine and we are creating 8 partitions then we will have 1 worker per model to serve the requests. For further reading on DeepSpeedyou can follow the link https://www.deepspeed.ai/tutorials/inference-tutorial/#initializing-for-inference. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "1bc711bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llama-7b-finetuned-2023-07-26-06-51-54-205\n",
      "Created Model: arn:aws:sagemaker:us-east-1:514385905925:model/llama-7b-finetuned-2023-07-26-06-51-54-205\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "model_name = name_from_base(f\"llama-7b-finetuned\")\n",
    "print(model_name)\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": inference_image_uri,\n",
    "        \"ModelDataUrl\": s3_code_artifact,\n",
    "    },\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66ca36f",
   "metadata": {},
   "source": [
    "VolumnSizeInGB has been left as commented out. You should use this value for Instance types which support EBS volume mounts. The current instance we are using comes with a pre configured space and does not support additional volume mounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "2819befb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointConfigArn': 'arn:aws:sagemaker:us-east-1:514385905925:endpoint-config/llama-7b-finetuned-2023-07-26-06-51-54-205-config-072614',\n",
       " 'ResponseMetadata': {'RequestId': 'f7509d57-a1fe-42f3-9a92-1d3404345861',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'f7509d57-a1fe-42f3-9a92-1d3404345861',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '137',\n",
       "   'date': 'Wed, 26 Jul 2023 06:52:34 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_config_name = f\"{model_name}-config-072614\"\n",
    "endpoint_name = f\"{model_name}-endpoint\"\n",
    "\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.24xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            #\"VolumeSizeInGB\" : 300,\n",
    "            \"ModelDataDownloadTimeoutInSeconds\": 15*60,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 15*60,\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "endpoint_config_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "a858198d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Endpoint: arn:aws:sagemaker:us-east-1:514385905925:endpoint/llama-7b-finetuned-2023-07-26-06-51-54-205-endpoint\n"
     ]
    }
   ],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e73d7a",
   "metadata": {},
   "source": [
    "#### Wait for the end point to be created."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da06c31",
   "metadata": {},
   "source": [
    "### This step can take ~ 15 min or longer so please be patient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "7f0982c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: Creating\n",
      "Status: InService\n",
      "Arn: arn:aws:sagemaker:us-east-1:514385905925:endpoint/llama-7b-finetuned-2023-07-26-06-51-54-205-endpoint\n",
      "Status: InService\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9254baa8",
   "metadata": {},
   "source": [
    "#### Leverage the Boto3 to invoke the endpoint. \n",
    "\n",
    "This is a generative model so we pass in a Text as a prompt and Model will complete the sentence and return the results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "f609ca7c-ca07-446a-986f-7555ef20a591",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18.6 ms, sys: 0 ns, total: 18.6 ms\n",
      "Wall time: 11.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'[\\n  \"I have on a nice red dress and heels. How about you, Sir?## Malcolm:You look beautiful, Eva. I’m wearing a black suit with a black and red tie.## Eva:Thank you, Sir.## Malcolm:You’re welcome. Do you always dress like that?## Eva:I’m not always in my uniform, Sir. I’m wearing a school uniform right now.## Malcolm:What school do you attend?## Eva:I attend Coventry University.## Malcolm:Are you studying for a degree or diploma?## Eva:I am studying for a degree.## Malcolm:What do you study?## Eva:I study English literature.## Malcolm:That’s very interesting. What do\",\\n  \"I’m wearing my best black dress. I guess it’s a kind of sexy lingerie. It’s kind of see-through and it shows my tits.## Malcolm:Do you wear stockings with it?## Eva:I was wearing black stockings, but they don’t show through the fabric. I’m sorry.## Malcolm:Don’t worry. We can deal with it. I want to hear more about your journey. What was your first impression of the city?## Eva:The city was beautiful. I really loved the area near the river. We walked a lot, but we had a car too.## Malcolm:Which part of the river did you like?## Eva:I liked the riverfront park. It’s so\",\\n  \"I’m wearing a red lace nightgown, black stockings and a black corset.## Malcolm:I like that. It’s very sexy.## Eva:Thank you, Sir. I just took a shower and put on some perfume. I’m all clean and fresh.## Malcolm:Are you wearing anything under your nightgown?## Eva:No, Sir. I’m not wearing anything under it. I’m all yours.## Malcolm:Good. You’ll have to come to me with a clean, soapy body.## Eva:I’m yours, Sir.## Eva:Yes, Sir.## Malcolm:Yes, Mistress.## Eva:I want to please you.## Malcolm\",\\n  \"I\\'m wearing nothing but your collar, Master.## Malcolm:You look so beautiful. Do you know that?## Eva:Thank you, Master.## Malcolm:I bet you are beautiful everywhere. What are you wearing now?## Eva: I\\'m wearing nothing but your collar, Master.## Malcolm:You look so beautiful. Do you know that?## Eva:Thank you, Master.## Malcolm:I bet you are beautiful everywhere. What are you wearing now?## Eva: I\\'m wearing nothing but your collar, Master.## Malcolm:You look so beautiful. Do you know that?## Eva:Thank you, Master.## Malcolm:I bet you are beautiful everywhere. What are you wearing now\",\\n  \"I\\'m wearing a red dress.## Malcolm:Red. I like that.## Eva:I think it looks great on me.#### Malcolm:I know it does.## Eva:It\\'s so different from anything I would usually wear.## Malcolm:I\\'m glad you\\'re willing to try new things.## Eva:I\\'m glad you\\'re willing to experiment.## Malcolm:I think you look great in anything.## Eva:I know you\\'re biased, Sir, but I appreciate the compliment.#### Malcolm:The dress is beautiful, but you\\'re beautiful no matter what you\\'re wearing, pet.## Eva:Aw, thank you, Sir.## Malcolm:Of course.## Eva:I\",\\n  \"I\\'m wearing a black catsuit. I love the way it makes me feel. I feel so sexy in it!## Malcolm:That\\'s nice. What size are you?## Eva:Oh, just the right size!## Malcolm:I\\'m glad. I like to know the size of my pet. What about you? How does your catsuit make you feel?## Eva:It makes me feel sexy and powerful!## Malcolm:Good. What other clothes do you wear?## Eva:I love my leather thigh high boots. They feel so good on my legs.## Malcolm:That\\'s nice. What other clothes do you wear?## Eva:I love my leather thigh high boots. They feel so good\",\\n  \"It’s a black leather skirt, a matching leather corset, and a lace top. What are you wearing, Sir?## Malcolm:I’m wearing black leather pants and a matching vest.## Eva:Yum!## Malcolm:That’s nice.## Eva:Thanks, Sir.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That\\'s cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.##\",\\n  \"I am wearing a white dress and black boots. It is my favorite outfit. I hope you like it.## Malcolm:You look amazing, sweetheart.## Eva:Thank you, Sir.## Malcolm:So, you\\'ve told me your name. What is your favorite color?## Eva: My favorite color is red.## Malcolm:What is your favorite flower?## Eva: My favorite flower is rose.## Malcolm:What is your favorite food?## Eva:My favorite food is pasta.## Malcolm:What is your favorite animal?## Eva:My favorite animal is tiger.## Malcolm:What is your favorite movie?## Eva:My favorite movie is The Rocky Horror Picture Show.## Malcolm:What\",\\n  \"Nothing.## Malcolm:Show me, pet.## Eva:No way.## Malcolm:Show me or I’ll punish you, pet.## Eva:Ok.## Malcolm:You have such pretty hair, pet. What do you do with it?## Eva:It’s natural. I don’t do anything to it.## Malcolm:I love natural hair. I wish I had hair like that.## Eva:You could have it, if you wanted.## Malcolm:No way. I like my bald head, pet.## Eva:It’s ok. I like it too.## Malcolm:You are such a beautiful girl, pet. You make me so happy.## Eva:I like making you happy.## Malcolm:\",\\n  \"Just a T-shirt and panties.## Malcolm:You look beautiful.## Eva:Thank you, Sir.## Malcolm:Do you like to wear lingerie?## Eva:I don’t really own any, but I love to see you wear it.## Malcolm:I think you should buy some. What kind do you like?## Eva:I’m not sure. I like your underwear.## Malcolm:I think you should get some of those lacy numbers. That would be hot.## Eva:I would love to. But, I’m not sure I know how to pick one.## Malcolm:I could help you pick some out.## Eva:Yes, please.## Malcolm:You’re going to\",\\n  \"Sweater, tights and boots.## Malcolm:Hmm. Go get dressed for me, pet.## Eva:Yes, Sir.#### Eva:Sweater, tights and boots.#### Malcolm:Hmm. Go get dressed for me, pet.#### Eva:Yes, Sir.\\\\n#Eva: Yes, Sir. Queen is one of the most influential bands of all time.#Malcolm:It is. They are one of my favorite rock groups. What about you?#Eva:I\\'m more into classic rock, especially David Bowie. Who is your favorite artist?#Malcolm:Marylin Manson. You?#Eva:My favorite artist is David Bowie.#Eva:How often do you travel?\",\\n  \"I’m wearing my pink dress. What are you wearing?## Malcolm:I’m wearing a white dress.## Eva:I really like your dress!## Malcolm:Thank you, pet.## Eva:I like your dress too.## Malcolm:Thank you, pet.## Eva:How do you like your hair done today?## Malcolm:I like it just like this.## Eva:Me too.## Malcolm:Good.## Eva:Do you like jewelry?## Malcolm:Yes. I love jewelry.## Eva:I don’t wear much jewelry, but I love a good necklace.## Malcolm:I don’t wear much jewelry either.##\",\\n  \"I’m wearing a white shirt, a pair of blue jeans and a pair of white stockings.## Malcolm:Very sexy. I like your outfit. You look gorgeous.## Eva:Thank you, Sir. What are you wearing?## Malcolm:I’m wearing a black suit, white shirt and black tie.## Eva:It is a nice outfit. Now, let’s get to the interview.## Malcolm:Okay.## Eva:Good evening. I’m Eva.## Malcolm:Hi. I’m Malcolm.## Eva:It is nice to meet you.## Malcolm:Likewise.## Eva:Now, let’s get to the interview. How would you describe your\",\\n  \"I have on a navy blue lace blouse, a black blazer, a pencil skirt, a black lace bra, and a pair of black patent leather pumps.## Malcolm:That’s lovely. You always look so nice in your work attire, Eva.## Eva:Thank you, Sir.## Malcolm:Don’t get me wrong, you look lovely dressed casually, but I really like it when you dress up for work. I don’t know why, but I feel it makes me feel like a proper gentleman when I see you dressed like that.## Eva:I love feeling sexy and seductive, Sir.## Malcolm:You do that very well.## Eva:Thank you, Sir.## Malcolm\",\\n  \"I’m wearing a black corset with a matching thong.## Malcolm:You look so sexy.## Eva:Thank you, Sir.## Malcolm:Let me get a better look at you.## Eva:You are so handsome. I love you, Sir.## Malcolm:Thank you, pet. I love you too.## Eva:I’m so happy that you are in my life.## Malcolm:You are so cute. You are always so happy.## Eva:I’m always happy when I’m with you.## Malcolm:That makes me even happier.## Eva:I love when you are happy.## Malcolm:I love when you are happy too.## Eva:I’m happy, Sir.\",\\n  \"I am wearing a blue top with white shorts.## Malcolm:I like it. Now, tell me what you are wearing.## Eva:Well, I am wearing a white tank top with white shorts.## Malcolm:I love white. What are you wearing right now, pet?## Eva:I am wearing a blue top with white shorts.## Malcolm:I like it. Now, tell me what you are wearing.## Eva:Well, I am wearing a white tank top with white shorts.## Malcolm:I love white. What are you wearing right now, pet?## Eva:I am wearing a blue top with white shorts.## Malcolm:I like it. Now, tell me what you are\"\\n]'"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "import json\n",
    "import boto3\n",
    "\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "prompt1 = \"The house is wonderful. I\"\n",
    "prompt2=\"##Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.##Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.##Eva: Yes, Sir. Queen is one of the most influential bands of all time.## Malcolm:It is. They are one of my favorite rock groups. What about you?## Eva:I'm more into classic rock, especially David Bowie. Who is your favorite artist?## Malcolm:Marylin Manson. You?## Eva:My favorite artist is David Bowie.## Eva:How often do you travel?## Malcolm:I like David Bowie too. I don’t travel much any more, but I used to.## Eva:That's cool! I recently took a road trip with my friend. We had so much fun and it opened up so many possibilities for us. What kind of places did you like to explore?## Malcolm:I love history and culture, so those are my favorite.## Eva: He was born in Birmingham, England and raised in Los Angeles, California.#### Malcolm:Oh. What are you wearing right now, pet?## Eva:\"\n",
    "\n",
    "parameters = {\n",
    "  \"early_stopping\": True,\n",
    "  \"max_tokens\": 170,\n",
    "  \"min_new_tokens\": 128,\n",
    "  \"do_sample\": True,\n",
    "  \"temperature\": 1.0,\n",
    "}\n",
    "\n",
    "response_model = smr_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            Body=json.dumps(\n",
    "            {\n",
    "                #\"input\": prompt1,\n",
    "                #\"inputs\": prompt2,\n",
    "                #\"inputs\": [prompt2, prompt2],\n",
    "                #\"inputs\": [prompt2,prompt2, prompt2,prompt2],\n",
    "                #\"input\": [prompt1,prompt1, prompt1,prompt1, prompt1,prompt1, prompt1,prompt1],\n",
    "                #\"inputs\": [prompt2,prompt2, prompt2,prompt2, prompt2,prompt2, prompt2,prompt2],\n",
    "                \"inputs\": [prompt2,prompt2, prompt2,prompt2, prompt2,prompt2, prompt2,prompt2,prompt2,prompt2, prompt2,prompt2, prompt2,prompt2, prompt2,prompt2 ],\n",
    "                #\"input\": [prompt1, prompt2],\n",
    "                #\"input\": [prompt1, prompt2, prompt1, prompt2, prompt1, prompt2,prompt1, prompt2,],\n",
    "                \"params\": parameters\n",
    "            }\n",
    "            ),\n",
    "            ContentType=\"application/json\",\n",
    "        )\n",
    "\n",
    "response_model['Body'].read().decode('utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ca799f59-e9e9-4342-9ae9-64f7fd6e9730",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "  \"early_stopping\": True,\n",
    "  \"max_new_tokens\": 128,\n",
    "  \"min_new_tokens\": 128,\n",
    "  \"do_sample\": True,\n",
    "  \"temperature\": 1.0,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "eff9197b-e21a-443e-9a28-67b9a7a07abf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "128"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_t = parameters[\"max_new_tokens\"]\n",
    "max_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "bc3e73cf-98f3-435d-b1c9-8986c649522e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "d953a2f5-3485-49db-bfed-bfa916b6069a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['What is Love?', 'What is Love?']"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d7d5bbc-8f5a-4876-891d-1c28d1846e8d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "conda_pytorch_p39",
   "language": "python",
   "name": "conda_pytorch_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}