LLM-Custom-Decoding-Falcon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9fb44f6d",
   "metadata": {},
   "source": [
    "## Text Generation using Different Decoding Strategies with Amazon SageMaker JumpStart SDK and Falcon 7B Instruct Language Model\n",
    "\n",
    "---\n",
    "This Amazon SageMaker Studio Notebook demonstrates how to use the SageMaker Python SDK with very little efforts to firstly deploy Falcon-7B-Instruct Large Language Model and then generate text using different decoding methods. \n",
    "\n",
    "This notebook has the following prerequisites:\n",
    "- Select an AWS region where [Amazon SageMaker JumpStart](https://aws.amazon.com/sagemaker/jumpstart) is available. \n",
    "- [Setup Amazon SageMaker Domain](https://docs.aws.amazon.com/sagemaker/latest/dg/onboard-quick-start.html).\n",
    "- [Available service queta for \"ml.g5.2xlarge for endpoint usage\"](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html).\n",
    "- Less than $2 per hour to spend on Amazon SageMaker JumpStart model deployment and Amazon SageMaker Studio notebook usage.  \n",
    "\n",
    "This notebook is based on the following references:\n",
    "- [Amazon SageMaker JumpStart SDK](https://sagemaker.readthedocs.io/en/v2.82.0/overview.html#use-prebuilt-models-with-sagemaker-jumpstart), providing pretrained models for a wide range of problem types to help you get started with machine learning.\n",
    "- [Falcon-7B-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct), an open source model with 7B parameters decoder-only model built by TII.\n",
    "- Public articles ([Link 1](https://huggingface.co/blog/how-to-generate), [Link 2](https://huggingface.co/docs/transformers/generation_strategies), [Link 3](https://huggingface.co/blog/sagemaker-huggingface-llm#4-run-inference-and-chat-with-our-model)) published on Hugging Face, an open source community and data science platform for machine learning models and datasets. \n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b05b931-992e-4526-978d-f03196874a3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade pip --quiet --disable-pip-version-check --root-user-action=ignore\n",
    "!pip install --upgrade sagemaker --quiet --root-user-action=ignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08508ffa-90f4-4c8c-9c69-b8002bb37f12",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "# Define SageMaker JumpStart Model using model id, instance type, and endpoint timeout\n",
    "my_model = JumpStartModel(model_id=\"huggingface-llm-falcon-7b-instruct-bf16\",\n",
    "                          instance_type=\"ml.g5.2xlarge\",\n",
    "                          env={'ENDPOINT_SERVER_TIMEOUT':'300'})\n",
    "\n",
    "# Take a look at the JumpStart Model parameters printed by this cell\n",
    "print(\"Model id =\", my_model.model_id)\n",
    "print(\"Model name =\", my_model.name)\n",
    "print(\"Model version =\", my_model.model_version)\n",
    "print(\"Instance type =\", my_model.instance_type)\n",
    "print(\"Instance number of GPUs =\", my_model.env[\"SM_NUM_GPUS\"])\n",
    "print(\"Model maximum input length =\", my_model.env[\"MAX_INPUT_LENGTH\"])\n",
    "print(\"Model maximum total tokens =\", my_model.env[\"MAX_TOTAL_TOKENS\"])\n",
    "print(\"Server endpoint timeout =\", my_model.env[\"ENDPOINT_SERVER_TIMEOUT\"], \"seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Host the model on the instance and deploy an inference endpoint\n",
    "# Because the model size is >80GB, expecy deploy() to take 15 min!\n",
    "predictor = my_model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471b13e2-9d0b-4158-8841-a6c07a7d3b45",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "**Decoding Strategies**\n",
    "\n",
    "Large Language Models are designed to \"guess\" the next token having read all the previous ones based on a predefined sampling method. There are several methods (decoding strategies) we can configure for picking this output token, such as Greedy Search, Beam Search, and Contrastive Search.\n",
    "\n",
    "We start by defining the promote which will enable us to test different decoding strategies for the same input text. We will also fix the values of some hyperparameters across all decoding strategies to make easy for us to compare the generated text:\n",
    "- *Temperature* is used to control the randomness of predictions by scaling the logits before applying softmax. The softmax layer in the transformer architecture turns the logits into probabilities (between 0 and 1). A low temperature (below 1) sharpens the probabilities of the predicted words resulting in more conservative and predictable text. A high temperature (above 1) makes the model generates more creative and diverse text resulting in unusual or unexpected words.\n",
    "- *stop* provides a list of input tokens to the model to stop the generation. The generation will stop when one of the tokens is generated.\n",
    "- *max_new_tokens* defines the maximum number of tokens to be generated by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77efa7b9-4e50-400b-b7c7-8a8ea07d82a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompt = \"What are the top 5 ingredients for a delicious lemon cheesecake?\"\n",
    "#prompt = \"What is the secret ingredient for a delicious lemon cheesecake?\"\n",
    "stop_keywords = [\"<|endoftext|>\", \"</s>\"]\n",
    "max_new_tokens = 150\n",
    "temperature = 0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd22952a-4992-41ab-b4d2-2cfb15559652",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Strategy 1: Greedy Search**\n",
    "\n",
    "Greedy search is a deterministic method that simply selects the word with the highest probability as its next word. To configure greedy decoding, we set the *do_sample* hyperparameter to false and make sure the *num_beams* hyperparameter is set to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1a55aa5-f2ad-4db8-9718-b76f969cffbe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "greedy_search_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": False,\n",
    "        \"num_beams\": 1,\n",
    "    }\n",
    "}\n",
    "greedy_search_response = predictor.predict(greedy_search_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Greedy Search Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(greedy_search_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e153983-9964-4e8f-b148-5a2e968d61cc",
   "metadata": {},
   "source": [
    "**Strategy 2: Beam Search**\n",
    "\n",
    "Beam search is another deterministic method that reduces the risk of missing hidden high probability word sequences by keeping a fixed number (beam) of active candidates at each time step and eventually choosing the hypothesis that has the overall highest probability. To configure beam decoding, we set the *do_sample* hyperparameter to false and set the *num_beams* hyperparameter to a value above 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "008e8836-7a49-407f-bf04-8dcf59de8dbf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "beam_search_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": False,\n",
    "        \"num_beams\": 4,\n",
    "    }\n",
    "}\n",
    "beam_search_response = predictor.predict(beam_search_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Beam Search Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(beam_search_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf74caa-c563-4b1e-9585-a28237dc539f",
   "metadata": {},
   "source": [
    "**Strategy 3: Multinomial Sampling**\n",
    "\n",
    "Solely maximizing the output probability in deterministic methods can lead to dullness and repetitions. Conversely, stochastic methods try to solve the problem by introducing randomness to the sampling process. \n",
    "\n",
    "Multinomial sampling randomly selects the next token based on the probability distribution over the entire vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the risk of repetition. To configure multinomial sampling, we set the *do_sample* hyperparameter to true and set the *num_beams* hyperparameter to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b253a50c-a024-4608-9502-00591764eb5c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "multinomial_sampling_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"num_beams\": 1\n",
    "    }\n",
    "}\n",
    "multinomial_sampling_response = predictor.predict(multinomial_sampling_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Multinomial Sampling Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(multinomial_sampling_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850ad903-7381-466a-b0e1-33cd61532786",
   "metadata": {},
   "source": [
    "**Strategy 4: Top-k Sampling**\n",
    "\n",
    "Top- k sampling means sorting by probability and zeroing out the probabilities for anything below the k'th token. To configure Top-k sampling, we set the *do_sample* hyperparameter to true and set the *top_k* hyperparameter to a value above 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e829e375-9ac3-46ba-a4fa-8753196f2a2f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "top_k_sampling_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"top_k\": 7\n",
    "    }\n",
    "}\n",
    "top_k_sampling_response = predictor.predict(top_k_sampling_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Top-k Search Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(top_k_sampling_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "573bd4c2-9283-4573-915d-2a9aa9d548bf",
   "metadata": {},
   "source": [
    "**Strategy 5: Top-p Sampling**\n",
    "\n",
    "Top k sampling (or nucleus sampling) chooses from the smallest possible set of words whose cumulative probability exceeds the probability p. To configure Top p sampling, we set the *do_sample* hyperparameter to true and set the *top_p* hyperparameter to a value less than 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "608ad83e-d6f8-4cf0-b3b9-4c65d31b1aba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "nucleus_sampling_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.8\n",
    "    }\n",
    "}\n",
    "nucleus_sampling_response = predictor.predict(nucleus_sampling_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Nucleus search Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(nucleus_sampling_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e210091e-6014-415d-9d3b-525237dd7d47",
   "metadata": {},
   "source": [
    "**Strategy 6: Contrastive Sampling**\n",
    "\n",
    "Contrastive search selects from the most probable candidates predicted by the model while taking into account the degeneration penalty computed from the previous context. This decoding strategy tries to maintain the semantic coherence in the generated text while reducing repetitions. To configure Contrastive search, we set the *top_k* hyperparameter to a value above 1 and set the *repetition_penalty* hyperparameter to a value between 0 and 1. When *repetition_penalty* is close to zero, contrastive search degenerates to the greedy search method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5057878-116d-459d-a8c1-dd33ca709ccf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "contrastive_search_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"repetition_penalty\": 0.9,\n",
    "        \"top_k\": 7\n",
    "    }\n",
    "}\n",
    "contrastive_search_response = predictor.predict(contrastive_search_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Contrastive Search Response:\", \">\"*40, \"\\033[95m\")\n",
    "print(contrastive_search_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab4d4dec-7df9-40de-8795-ea836f2829a4",
   "metadata": {},
   "source": [
    "**Strategy 7: Combining Multiple Methods**\n",
    "\n",
    "Combining multiple methods (such as top-k and top-p sampling) can sometimes improve the diversity and fluency of the generated text. Try to configure your own decoding strategy including changing the temperature hyperparameter and see the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31039442-f500-4c01-ad17-45a9e71fadf8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "multiple_strategies_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.8,\n",
    "        \"top_k\": 7\n",
    "    }\n",
    "}\n",
    "multiple_strategies_response = predictor.predict(multiple_strategies_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Multiple Strategies Response 1:\", \">\"*40, \"\\033[95m\")\n",
    "print(multiple_strategies_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d54c6bf-d3f5-4db5-83e4-743fe55e80b9",
   "metadata": {},
   "source": [
    "Beam search can be used in conjunction with sampling by choosing the next token at random from the set of possible tokens while building up the beams. Let us try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58d991f2-8ded-4d9d-bdcc-d4a6ba15a724",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "multiple_strategies_payload = {\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\n",
    "        \"stop\": stop_keywords,\n",
    "        \"temperature\": temperature,\n",
    "        \"max_new_tokens\": max_new_tokens,\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.8,\n",
    "        \"top_k\": 7,\n",
    "        \"num_beams\": 3\n",
    "    }\n",
    "}\n",
    "multiple_strategies_response = predictor.predict(multiple_strategies_payload)\n",
    "\n",
    "print(prompt)\n",
    "print(\"Multiple Strategies Response 2:\", \">\"*40, \"\\033[95m\")\n",
    "print(multiple_strategies_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04d99e2",
   "metadata": {},
   "source": [
    "### SageMaker Clean up "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3b60c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}