project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sports Image Classification\n",
    "Dataset: https://www.kaggle.com/code/littlebughenrylee/100-sports-classification-resnet-93-yolo-98\n",
    "\n",
    "<p>Collection of sports images covering 100 different sports.. Images are 224,224,3 jpg format. Data is separated into train, test and valid directories. Additionallly a csv file is included for those that wish to use it to create there own train, test and validation datasets.</p>\n",
    "\n",
    "Images were gathered from internet searches. The images were scanned with a duplicate image detector program I wrote. Any duplicate images were removed to prevent bleed through of images between the train, test and valid data sets. All images were then resized to 224 X224 X 3 and converted to jpg format. A csv file is included that for each image file contains the relative path to the image file, the image file class label and the dataset (train, test or valid) that the image file resides in. This is a clean dataset.\n",
    "\n",
    "<img src=\"https://t3.ftcdn.net/jpg/02/78/42/76/360_F_278427683_zeS9ihPAO61QhHqdU1fOaPk2UClfgPcW.jpg\" class=\"center\">\n",
    "<style>.center {\n",
    "  display: block;\n",
    "  margin-left: auto;\n",
    "  margin-right: auto;\n",
    "  width: 50%;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
    "import torch\n",
    "from torch import nn as nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import time\n",
    "import torchvision\n",
    "from torchvision import transforms\n",
    "\n",
    "from torchinfo import summary\n",
    "import kaggle\n",
    "from plotly.subplots import make_subplots\n",
    "import os\n",
    "import glob\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Dataset from Kaggle\n",
    "link: https://www.kaggle.com/datasets/gpiosenka/sports-classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Already Exist\n"
     ]
    }
   ],
   "source": [
    "if \"sports-classification\" in os.listdir():\n",
    "    print('Data Already Exist')\n",
    "else:\n",
    "  print(\"Downloading Data From Kaggle\")\n",
    "  !kaggle datasets download -d gpiosenka/sports-classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example usage:\n",
    "directory_path = os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Downloaded ZIP File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['d:\\\\sports-classification.zip']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract the filenames without extension\n",
    "data_folder_name = glob.glob(os.path.join(directory_path, '*.zip'))\n",
    "filenames_without_extension = [os.path.splitext(os.path.basename(file))[0] for file in data_folder_name]\n",
    "data_folder_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder_name = filenames_without_extension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_folder_name = data_folder_name[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine If Dataset is unzipped or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory 'sports-classification' already exists.\n"
     ]
    }
   ],
   "source": [
    "if not os.path.exists(data_folder_name):\n",
    "    os.makedirs(data_folder_name)\n",
    "    print(f\"Directory '{data_folder_name}' created.\")\n",
    "    !unzip sports-classification.zip -d 'sports-classification/'\n",
    "else:\n",
    "    print(f\"Directory '{data_folder_name}' already exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Model Results Directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_RESULTS_DIR = 'models_results'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directory 'models_results' already exists.\n"
     ]
    }
   ],
   "source": [
    "if not os.path.exists(MODEL_RESULTS_DIR):\n",
    "    os.makedirs(MODEL_RESULTS_DIR)\n",
    "    print(f\"Directory '{MODEL_RESULTS_DIR}' created.\")\n",
    "else:\n",
    "    print(f\"Directory '{MODEL_RESULTS_DIR}' already exists.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Dataset and get total number of classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_csv('sports-classification/sports.csv')\n",
    "NUM_CLASSES = data['class id'].nunique()\n",
    "print(NUM_CLASSES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Device either GPU or CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cuda'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Device\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "# hardcode :\n",
    "# loss_fn -> CrossEntropyLoss\n",
    "# optimizer -> Adam(lr = 0.0005)\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define ImageDataset\n",
    "The ImageDataset class is a custom dataset implementation for handling image data in PyTorch. It is designed to work with a specific directory structure where each sub-directory corresponds to a distinct class, and the images belonging to that class are stored within that sub-directory. This structure is commonly used in image classification datasets.\n",
    "\n",
    "The class supports image data in the JPG format. \n",
    "\n",
    "Attributes: The class has attributes root_dir, transform, images, labels, and class_labels.\n",
    "\n",
    "- `root_dir`: The root directory path where the image data is stored.\n",
    "- `transform`: A function for image transformations (e.g., resizing, normalization).\n",
    "- `images`: A list to store the file paths of all the images in the dataset.\n",
    "- `labels`: A list to store the corresponding class labels (as integers) for each image.\n",
    "- `class_labels`: A dictionary mapping class directory names to integer labels.\n",
    "\n",
    "`__len__` Method: Implements the `__len__` method to return the total number of samples (images) in the dataset. It returns the length of the images list.\n",
    "\n",
    "`__getitem__` Method: Implements the `__getitem__` method to access a specific sample (image) and its associated label by index. It loads the image from the file path using PIL, applies the specified transformation (if any), and returns the transformed image along with its label.\n",
    "\n",
    "Overall, the ImageDataset class simplifies the process of working with image datasets in PyTorch, providing a flexible and robust solution for handling image data in machine learning projects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Raw dataset\n",
    "class ImageDataset(Dataset):\n",
    "    def __init__(self, root_dir, transform=None):\n",
    "\n",
    "        \"\"\"\n",
    "        Initializes the ImageDataset object.\n",
    "\n",
    "        Parameters:\n",
    "            root_dir (str): The root directory path containing sub-directories, each representing a class.\n",
    "            transform (optional, callable): A callable function to transform the images (e.g., resizing, normalization).\n",
    "        \"\"\"\n",
    "\n",
    "        self.root_dir = root_dir\n",
    "        self.transform = transform\n",
    "        self.images = []\n",
    "        self.labels = []\n",
    "        self.class_labels = {}\n",
    "\n",
    "        # Create a mapping of class labels to integers\n",
    "        self.class_labels = {}\n",
    "        class_idx = 0\n",
    "\n",
    "        # Iterate over sub-directories\n",
    "        for class_dir in os.listdir(self.root_dir):\n",
    "            class_dir_path = os.path.join(self.root_dir, class_dir)\n",
    "            if os.path.isdir(class_dir_path):\n",
    "                self.class_labels[class_dir] = class_idx\n",
    "                class_idx += 1\n",
    "\n",
    "                # Iterate over images in the sub-directory\n",
    "                for img_filename in os.listdir(class_dir_path):\n",
    "                    if img_filename.endswith(\".jpg\"):\n",
    "                        img_path = os.path.join(class_dir_path, img_filename)\n",
    "                        self.images.append(img_path)\n",
    "                        self.labels.append(self.class_labels[class_dir])\n",
    "\n",
    "    def __len__(self):\n",
    "\n",
    "        \"\"\"\n",
    "        Returns the total number of samples in the dataset.\n",
    "\n",
    "        Returns:\n",
    "            int: The number of samples in the dataset.\n",
    "        \"\"\"\n",
    "        \n",
    "        return len(self.images)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "\n",
    "        \"\"\"\n",
    "        Retrieves a sample from the dataset.\n",
    "\n",
    "        Parameters:\n",
    "            idx (int): The index of the sample to retrieve.\n",
    "\n",
    "        Returns:\n",
    "            PIL.Image.Image: The image sample.\n",
    "            int: The label associated with the image.\n",
    "        \"\"\"\n",
    "\n",
    "        image = Image.open(self.images[idx])        \n",
    "        label = self.labels[idx]\n",
    "        \n",
    "        if image.mode == \"L\":\n",
    "            image = Image.merge(\"RGB\", (image, image, image))\n",
    "        if self.transform:\n",
    "            image = self.transform(image)\n",
    "            \n",
    "        return image, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch DataLoader Function for Image Classification\n",
    "\n",
    "### Description:\n",
    "The `model_dataloader` function is designed to create PyTorch data loaders for training, validation, and testing datasets in an image classification task. It uses a custom dataset, named `ImageDataset`, which is assumed to handle a specific directory structure where each sub-directory represents a distinct class, and the images belonging to that class are stored within the respective sub-directory. This function takes weights and a transformation function as input and returns three PyTorch data loaders for the training, validation, and testing datasets.\n",
    "\n",
    "### Parameters:\n",
    "- `weights`: A variable (or value) that represents the weights used in the dataset (this variable is defined outside the function).\n",
    "- `transform`: A transformation function for image preprocessing, data augmentation, or resizing. The `transform` function is applied to the images loaded from the dataset.\n",
    "\n",
    "### Data Folder Structure:\n",
    "- The function assumes that the image data is stored in the \"sports-classification\" folder in the following structure:\n",
    "\n",
    "        sports-classification/ <br>\n",
    "        ├── train/ <br>\n",
    "        ├── valid/ <br>\n",
    "        └── test/ <br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pytorch dataloader\n",
    "def model_dataloder(weights, transform):\n",
    "    \"\"\"\n",
    "    Returns three PyTorch DataLoaders for training, validation, and testing.\n",
    "    \n",
    "    Parameters:\n",
    "        weights (list): A list of weights used for data sampling in DataLoader (optional).\n",
    "        transform (torchvision.transforms): Image transformation to be applied to the datasets.\n",
    "        \n",
    "    Returns:\n",
    "        train_dataloader (DataLoader): DataLoader for the training dataset.\n",
    "        val_dataloader (DataLoader): DataLoader for the validation dataset.\n",
    "        test_dataloader (DataLoader): DataLoader for the test dataset.\n",
    "    \"\"\"\n",
    "    weights = weights\n",
    "    \n",
    "    data_folder = \"sports-classification\"\n",
    "\n",
    "    train_folder = data_folder + \"/train\"\n",
    "    val_folder = data_folder + \"/valid\"\n",
    "    test_folder = data_folder + \"/test\"\n",
    "\n",
    "    # Images from internet\n",
    "    image_urls = \"testing_images\"\n",
    "    \n",
    "    # pytorch dataset\n",
    "    train_dataset = ImageDataset(train_folder, transform = transform)\n",
    "    val_dataset = ImageDataset(val_folder, transform = transform)\n",
    "    test_dataset = ImageDataset(test_folder, transform = transform)\n",
    "\n",
    "    # test downloaded  images \n",
    "    custom_images = ImageDataset(image_urls, transform = transform)\n",
    "    \n",
    "    # pytorch dataloader\n",
    "    train_dataloader = DataLoader(dataset = train_dataset, batch_size = 32, shuffle = True)\n",
    "    val_dataloader = DataLoader(dataset = val_dataset, batch_size = 32, shuffle = False)\n",
    "    test_dataloader = DataLoader(dataset = test_dataset, batch_size = 32, shuffle = False)\n",
    "    custom_dataloader = DataLoader(dataset = custom_images, batch_size = 32, shuffle = False)\n",
    "    \n",
    "    return train_dataloader, val_dataloader, test_dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Function\n",
    "\n",
    "### Description:\n",
    "The `train` function is used to perform one training epoch for a given model using a specified data loader. It computes the training loss and accuracy during the training process.\n",
    "\n",
    "### Parameters:\n",
    "- `model`: The PyTorch model to be trained.\n",
    "- `dataloader`: The PyTorch data loader containing the training dataset.\n",
    "- `loss_fn`: The loss function used to compute the training loss.\n",
    "- `optimizer`: The optimizer used to update the model's parameters during training.\n",
    "- `device`: The device on which to perform computations (e.g., \"cuda\" for GPU or \"cpu\" for CPU).\n",
    "\n",
    "### Returns:\n",
    "- `train_loss`: The average training loss computed over the training dataset.\n",
    "- `train_acc`: The average training accuracy computed over the training dataset.\n",
    "\n",
    "## Validation Function\n",
    "\n",
    "### Description:\n",
    "The `val` function is used to evaluate a trained model on a validation dataset. It computes the validation loss and accuracy.\n",
    "\n",
    "### Parameters:\n",
    "- `model`: The PyTorch model to be evaluated.\n",
    "- `dataloader`: The PyTorch data loader containing the validation dataset.\n",
    "- `loss_fn`: The loss function used to compute the validation loss.\n",
    "- `device`: The device on which to perform computations (e.g., \"cuda\" for GPU or \"cpu\" for CPU).\n",
    "\n",
    "### Returns:\n",
    "- `val_loss`: The average validation loss computed over the validation dataset.\n",
    "- `val_acc`: The average validation accuracy computed over the validation dataset.\n",
    "\n",
    "## Test Accuracy Function\n",
    "\n",
    "### Description:\n",
    "The `test_accuracy_resnet` function is used to evaluate the accuracy of a trained model on a test dataset. It computes the accuracy of the model's predictions compared to the actual labels.\n",
    "\n",
    "### Parameters:\n",
    "- `model`: The PyTorch model to be evaluated.\n",
    "- `dataloader`: The PyTorch data loader containing the test dataset.\n",
    "- `device`: The device on which to perform computations (e.g., \"cuda\" for GPU or \"cpu\" for CPU).\n",
    "\n",
    "### Returns:\n",
    "- `accuracy`: The accuracy of the model's predictions on the test dataset, represented as a percentage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train -> train_loss, train_acc\n",
    "def train (model, dataloader, loss_fn, optimizer, device):\n",
    "    train_loss, train_acc = 0, 0\n",
    "    \n",
    "    model.to(device)\n",
    "    model.train()\n",
    "    \n",
    "    for batch, (x, y) in enumerate (dataloader):\n",
    "        x, y = x.to(device), y.to(device)\n",
    "        \n",
    "        train_pred = model(x)\n",
    "        \n",
    "        loss = loss_fn(train_pred, y)\n",
    "        train_loss = train_loss + loss.item()\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        train_pred_label = torch.argmax(torch.softmax(train_pred, dim = 1), dim = 1)\n",
    "        train_acc = train_acc + (train_pred_label == y).sum().item() / len(train_pred)\n",
    "    \n",
    "    train_loss = train_loss / len(dataloader)\n",
    "    train_acc = train_acc / len(dataloader)\n",
    "    \n",
    "    return train_loss, train_acc\n",
    "\n",
    "# Val -> val_loss, val_acc\n",
    "def val (model, dataloader, loss_fn, device):\n",
    "    val_loss, val_acc = 0, 0\n",
    "    \n",
    "    model.to(device)\n",
    "    model.eval()\n",
    "    \n",
    "    with torch.inference_mode():\n",
    "        for batch, (x, y) in enumerate(dataloader):\n",
    "            x, y = x.to(device), y.to(device)\n",
    "            \n",
    "            val_pred = model(x)\n",
    "            \n",
    "            loss = loss_fn(val_pred, y)\n",
    "            val_loss = val_loss + loss.item()\n",
    "            \n",
    "            val_pred_label = torch.argmax(torch.softmax(val_pred, dim = 1), dim = 1)\n",
    "            val_acc = val_acc + (val_pred_label == y).sum().item() / len(val_pred)\n",
    "        \n",
    "        val_loss = val_loss / len(dataloader)\n",
    "        val_acc = val_acc / len(dataloader)\n",
    "        \n",
    "        return val_loss, val_acc\n",
    "    \n",
    "def test_accuracy_resnet(model,dataloader,device):\n",
    "    # empty list store labels\n",
    "    predict_label_list = []\n",
    "    actual_label_list = []\n",
    "\n",
    "    # eval mode\n",
    "    model.eval()\n",
    "\n",
    "    for images, labels in dataloader: \n",
    "        \n",
    "        for label in labels:\n",
    "            label = label.item()\n",
    "            actual_label_list.append(label)\n",
    "        \n",
    "        for image in images:\n",
    "            with torch.inference_mode():\n",
    "                image = image.to(device)\n",
    "                # add batch_size and device\n",
    "                image = image.unsqueeze(dim = 0)\n",
    "                # logits\n",
    "                logits = model(image)\n",
    "                # lables\n",
    "                label = torch.argmax(logits).item()\n",
    "                predict_label_list.append(label)\n",
    "\n",
    "    accuracy = accuracy_score(actual_label_list, predict_label_list)\n",
    "    return accuracy*100\n",
    "\n",
    "def classify_custom_images(model,dataloader,device,df):\n",
    "    \n",
    "    pred_labels = []\n",
    "    # eval mode\n",
    "    model.eval()\n",
    "\n",
    "    for images, labels in dataloader: \n",
    "        \n",
    "        for image in images:\n",
    "            with torch.inference_mode():\n",
    "                image = image.to(device)\n",
    "                # add batch_size and device\n",
    "                image = image.unsqueeze(dim = 0)\n",
    "                # logits\n",
    "                logits = model(image)\n",
    "                # lables\n",
    "                label = torch.argmax(logits).item()\n",
    "                text_label = df[df['class id']==label]['labels'].iloc[0]\n",
    "                pred_labels.append(text_label)\n",
    "    return pred_labels\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Loop Function\n",
    "\n",
    "### Description:\n",
    "The `training_loop` function is responsible for training a given PyTorch model on a training dataset and evaluating its performance on a validation dataset. It allows you to control the number of training epochs and implements early stopping based on the `patience` parameter. The function records and returns the training and validation losses, accuracies, and training time for each epoch. Additionally, it displays two line charts showing the loss and accuracy trends over epochs.\n",
    "\n",
    "### Parameters:\n",
    "- `model`: The PyTorch model to be trained and evaluated.\n",
    "- `train_dataloader`: The PyTorch data loader containing the training dataset.\n",
    "- `val_dataloader`: The PyTorch data loader containing the validation dataset.\n",
    "- `device`: The device on which to perform computations (e.g., \"cuda\" for GPU or \"cpu\" for CPU).\n",
    "- `epochs`: The number of epochs for training.\n",
    "- `patience`: The number of consecutive epochs with no improvement in validation loss to trigger early stopping.\n",
    "\n",
    "### Returns:\n",
    "- `model_results`: A pandas DataFrame containing the results for each epoch, including training loss, training accuracy, validation loss, validation accuracy, and time taken for each epoch.\n",
    "- `training_time`: The total time taken for the entire training process.\n",
    "\n",
    "### Training and Early Stopping:\n",
    "The function trains the model for the specified number of epochs. If the validation loss does not improve for `patience` consecutive epochs, early stopping is triggered, and the training process stops to avoid overfitting.\n",
    "\n",
    "### Line Charts:\n",
    "The function creates two line charts using Plotly to visualize the loss and accuracy trends over epochs for both training and validation datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def training_loop(model, train_dataloader, val_dataloader, device, epochs, patience):\n",
    "    # empty dict for restore results\n",
    "    results = {\"train_loss\":[], \"train_acc\":[], \"val_loss\":[], \"val_acc\":[]}\n",
    "    \n",
    "    # hardcode loss_fn and optimizer\n",
    "    loss_fn = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr = 0.0005)\n",
    "\n",
    "    # variable to hold the training time\n",
    "    training_time = 0.0\n",
    "    epoch_run_time = []\n",
    "    # loop through epochs\n",
    "    for epoch in range(epochs):\n",
    "\n",
    "         # record the start time for each epoch\n",
    "        epoch_start_time = time.time()\n",
    "\n",
    "        train_loss, train_acc = train(model = model, \n",
    "                                      dataloader = train_dataloader,\n",
    "                                      loss_fn = loss_fn,\n",
    "                                      optimizer = optimizer,\n",
    "                                      device = device)\n",
    "        \n",
    "        val_loss, val_acc = val(model = model,\n",
    "                                dataloader = val_dataloader,\n",
    "                                loss_fn = loss_fn,\n",
    "                                device = device)\n",
    "        \n",
    "        # record the end time for each epoch\n",
    "        epoch_end_time = time.time()\n",
    "        \n",
    "        # calculate the time taken for this epoch\n",
    "        epoch_time = epoch_end_time - epoch_start_time\n",
    "        epoch_run_time.append(epoch_time)\n",
    "        training_time += epoch_time\n",
    "        \n",
    "        # print results for each epoch\n",
    "        print(f\"Epoch: {epoch+1}\\n\"\n",
    "              f\"Train loss: {train_loss:.4f} | Train accuracy: {(train_acc*100):.3f}%\\n\"\n",
    "              f\"Val loss: {val_loss:.4f} | Val accuracy: {(val_acc*100):.3f}%\\n\"\n",
    "              f\"| Epoch time: {epoch_time:.2f} seconds\")\n",
    "        \n",
    "        # record results for each epoch\n",
    "        results[\"train_loss\"].append(train_loss)\n",
    "        results[\"train_acc\"].append(train_acc)\n",
    "        results[\"val_loss\"].append(val_loss)\n",
    "        results[\"val_acc\"].append(val_acc)\n",
    "        \n",
    "        # calculate average \"val_loss\" for early_stopping\n",
    "        mean_val_loss = np.mean(results[\"val_loss\"])\n",
    "        best_val_loss = float(\"inf\")\n",
    "        num_no_improvement = 0\n",
    "        if np.mean(mean_val_loss > best_val_loss):\n",
    "            best_val_loss = mean_val_loss\n",
    "        \n",
    "            model_state_dict = model.state_dict()\n",
    "            best_model.load_state_dict(model_state_dict)\n",
    "        else:\n",
    "            num_no_improvement +=1\n",
    "    \n",
    "        if num_no_improvement == patience:\n",
    "            break\n",
    "    \n",
    "    # Saving Results for model\n",
    "    dic = {\n",
    "        \"epochs\": list(range(1,epochs+1)),\n",
    "        'Train_loss':results[\"train_loss\"],\n",
    "        'Train_Accuracy': results['train_acc'],\n",
    "        'Validation_loss':results[\"val_loss\"],\n",
    "        'Validation_Accuracy': results['val_acc'],\n",
    "        'Time_Taken':epoch_run_time\n",
    "\n",
    "    }\n",
    "    model_results = pd.DataFrame(dic)\n",
    "    \n",
    "\n",
    "    # Create the figure for the chart\n",
    "    fig = go.Figure()\n",
    "        # Add the 'Train loss' and 'Val loss' traces as lines\n",
    "    fig.add_trace(go.Scatter(x=list(range(1, len(results[\"train_loss\"]) + 1)), \n",
    "                            y=results[\"train_loss\"], mode='lines', name='Train loss'))\n",
    "    fig.add_trace(go.Scatter(x=list(range(1, len(results[\"val_loss\"]) + 1)), \n",
    "                            y=results[\"val_loss\"], mode='lines', name='Val loss'))\n",
    "    \n",
    "\n",
    "    # Update the layout for better visualization\n",
    "    fig.update_layout(title=\"Loss over Epochs\",\n",
    "                    xaxis_title=\"Epochs\",\n",
    "                    yaxis_title=\"Loss\",\n",
    "                    legend=dict(x=0.05, y=1.1),\n",
    "                    width=800, height=400)\n",
    "    \n",
    "    # Create the figure for the chart\n",
    "    fig2 = go.Figure()\n",
    "        # Add the 'Train loss' and 'Val loss' traces as lines\n",
    "    fig2.add_trace(go.Scatter(x=list(range(1, len(results[\"train_acc\"]) + 1)), \n",
    "                            y=results[\"train_acc\"], mode='lines', name='Train Accuracy'))\n",
    "    fig2.add_trace(go.Scatter(x=list(range(1, len(results[\"val_acc\"]) + 1)), \n",
    "                            y=results[\"val_acc\"], mode='lines', name='Val Accuracy'))\n",
    "    \n",
    "\n",
    "    # Update the layout for better visualization\n",
    "    fig2.update_layout(title=\"Accuracy over Epochs\",\n",
    "                    xaxis_title=\"Epochs\",\n",
    "                    yaxis_title=\"Accuracy\",\n",
    "                    legend=dict(x=0.05, y=1.1),\n",
    "                    width=800, height=400)\n",
    "\n",
    "    # combined_fig.show()\n",
    "    fig.show()\n",
    "    fig2.show()\n",
    "    \n",
    "    return model_results, training_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to Add Custom Layers to a PyTorch Model\n",
    "\n",
    "### Description:\n",
    "The `add_custom_layers` function is designed to modify a pre-trained PyTorch model by adding custom layers to the model's architecture. The function is useful for fine-tuning the model for a specific task, such as image classification.\n",
    "\n",
    "### Parameters:\n",
    "- `model`: The pre-trained PyTorch model to which the custom layers will be added. The model should be an instance of a PyTorch model, such as ResNet-50.\n",
    "\n",
    "### Returns:\n",
    "- `model`: The modified PyTorch model with custom layers added.\n",
    "\n",
    "### Custom Layers:\n",
    "The function first sets `requires_grad` to `False` for all parameters in the provided model. This step is typically done to freeze the weights of the pre-trained layers, preventing them from being updated during training.\n",
    "\n",
    "Next, the function defines a custom sequence of layers using the `nn.Sequential` module. You can modify this part of the code as per your specific requirements. In the provided example, the custom layers consist of three fully connected (linear) layers followed by ReLU activation and dropout layers.\n",
    "\n",
    "- The first fully connected layer takes as input the number of features in the last layer of the original model (`model.fc.in_features`). The number of output features for this layer is set to 1024.\n",
    "- A ReLU activation function is applied after the first fully connected layer.\n",
    "- A dropout layer with a dropout probability of 0.5 is added after the first ReLU activation.\n",
    "- The second fully connected layer takes 1024 input features and outputs 512 features.\n",
    "- Another ReLU activation and dropout layer are applied after the second fully connected layer.\n",
    "- The third and final fully connected layer takes 512 input features and outputs the number of classes (assuming `NUM_CLASSES` is defined for the specific task).\n",
    "\n",
    "Finally, the function replaces the last layer of the provided `model` (e.g., the fully connected layer of ResNet-50) with the custom layers defined above. The modified model is then returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_custom_layers(model):\n",
    "    for param in model.parameters():\n",
    "        param.requires_grad = False\n",
    "    \n",
    "    \n",
    "    num_features = model.fc.in_features\n",
    "    \n",
    "    # Define your custom layers (you can modify this part as per your requirements)\n",
    "    custom_layers = nn.Sequential(\n",
    "        nn.Linear(num_features, 1024),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(0.5),\n",
    "        nn.Linear(1024, 512),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(0.5),\n",
    "        nn.Linear(512, NUM_CLASSES)  # Assuming you have defined num_classes for your specific task\n",
    "    )\n",
    "\n",
    "    # Replace the last layer of the ResNet-50 model with the custom layers\n",
    "    model.fc = custom_layers\n",
    "\n",
    "    return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to Get ResNet Data Loaders\n",
    "\n",
    "### Description:\n",
    "The `get_resnet_dataloaders` function is designed to obtain PyTorch data loaders with data augmentation for training, validation, and testing datasets. It utilizes the `model_dataloader` function and applies specific data augmentation transformations to the images to enhance the training process for ResNet models.\n",
    "\n",
    "### Parameters:\n",
    "- `model_weights`: The weights or any other variable needed for the `model_dataloader` function (this variable is defined outside the function).\n",
    "\n",
    "### Returns:\n",
    "- `resnet_train_dataloader`: The PyTorch data loader containing the training dataset with data augmentation.\n",
    "- `resnet_val_dataloader`: The PyTorch data loader containing the validation dataset with data augmentation.\n",
    "- `resnet_test_dataloader`: The PyTorch data loader containing the testing dataset without data augmentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_resnet_dataloaders(model_weights):\n",
    "    \n",
    "    # resnet_weight.transforms()\n",
    "    resnet_transform = transforms.Compose([\n",
    "        transforms.Resize(size = 232),\n",
    "        transforms.ColorJitter(brightness = (0.8, 1.2)),\n",
    "        transforms.RandomHorizontalFlip(p = 0.5),\n",
    "        transforms.RandomRotation(degrees = 15),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])\n",
    "    ])\n",
    "    resnet_train_dataloader, resnet_val_dataloader, resnet_test_dataloader = model_dataloder(weights = model_weights, \n",
    "                                                                                         transform = resnet_transform\n",
    "                                                                                        )\n",
    "    \n",
    "    return resnet_train_dataloader, resnet_val_dataloader, resnet_test_dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Model Performance dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_times = pd.DataFrame(columns = ['Model','Testing Accuracy','Training_Time(Minutes)'])\n",
    "training_times.to_csv('models_results/training_time.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ResNet Model\n",
    "- ResNet, short for \"Residual Network,\" is a deep neural network architecture that was introduced by Kaiming He et al. from Microsoft Research in 2015\n",
    "- The primary motivation behind ResNet was to address the problem of vanishing gradients in very deep neural networks. \n",
    "- The ResNet architecture comes in various depths, such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, among others. \n",
    "\n",
    "\n",
    "The model is based on the [ResNet](https://pytorch.org/hub/pytorch_vision_resnet/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ResNet50 download weights and Pretrained Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_weight_50 = torchvision.models.ResNet50_Weights.DEFAULT\n",
    "\n",
    "resnet_model_50 = torchvision.models.resnet50(weights = resnet_weight_50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add some custom layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "=================================================================\n",
       "Layer (type:depth-idx)                   Param #\n",
       "=================================================================\n",
       "ResNet                                   --\n",
       "├─Conv2d: 1-1                            (9,408)\n",
       "├─BatchNorm2d: 1-2                       (128)\n",
       "├─ReLU: 1-3                              --\n",
       "├─MaxPool2d: 1-4                         --\n",
       "├─Sequential: 1-5                        --\n",
       "│    └─Bottleneck: 2-1                   --\n",
       "│    │    └─Conv2d: 3-1                  (4,096)\n",
       "│    │    └─BatchNorm2d: 3-2             (128)\n",
       "│    │    └─Conv2d: 3-3                  (36,864)\n",
       "│    │    └─BatchNorm2d: 3-4             (128)\n",
       "│    │    └─Conv2d: 3-5                  (16,384)\n",
       "│    │    └─BatchNorm2d: 3-6             (512)\n",
       "│    │    └─ReLU: 3-7                    --\n",
       "│    │    └─Sequential: 3-8              (16,896)\n",
       "│    └─Bottleneck: 2-2                   --\n",
       "│    │    └─Conv2d: 3-9                  (16,384)\n",
       "│    │    └─BatchNorm2d: 3-10            (128)\n",
       "│    │    └─Conv2d: 3-11                 (36,864)\n",
       "│    │    └─BatchNorm2d: 3-12            (128)\n",
       "│    │    └─Conv2d: 3-13                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-14            (512)\n",
       "│    │    └─ReLU: 3-15                   --\n",
       "│    └─Bottleneck: 2-3                   --\n",
       "│    │    └─Conv2d: 3-16                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-17            (128)\n",
       "│    │    └─Conv2d: 3-18                 (36,864)\n",
       "│    │    └─BatchNorm2d: 3-19            (128)\n",
       "│    │    └─Conv2d: 3-20                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-21            (512)\n",
       "│    │    └─ReLU: 3-22                   --\n",
       "├─Sequential: 1-6                        --\n",
       "│    └─Bottleneck: 2-4                   --\n",
       "│    │    └─Conv2d: 3-23                 (32,768)\n",
       "│    │    └─BatchNorm2d: 3-24            (256)\n",
       "│    │    └─Conv2d: 3-25                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-26            (256)\n",
       "│    │    └─Conv2d: 3-27                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-28            (1,024)\n",
       "│    │    └─ReLU: 3-29                   --\n",
       "│    │    └─Sequential: 3-30             (132,096)\n",
       "│    └─Bottleneck: 2-5                   --\n",
       "│    │    └─Conv2d: 3-31                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-32            (256)\n",
       "│    │    └─Conv2d: 3-33                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-34            (256)\n",
       "│    │    └─Conv2d: 3-35                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-36            (1,024)\n",
       "│    │    └─ReLU: 3-37                   --\n",
       "│    └─Bottleneck: 2-6                   --\n",
       "│    │    └─Conv2d: 3-38                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-39            (256)\n",
       "│    │    └─Conv2d: 3-40                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-41            (256)\n",
       "│    │    └─Conv2d: 3-42                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-43            (1,024)\n",
       "│    │    └─ReLU: 3-44                   --\n",
       "│    └─Bottleneck: 2-7                   --\n",
       "│    │    └─Conv2d: 3-45                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-46            (256)\n",
       "│    │    └─Conv2d: 3-47                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-48            (256)\n",
       "│    │    └─Conv2d: 3-49                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-50            (1,024)\n",
       "│    │    └─ReLU: 3-51                   --\n",
       "├─Sequential: 1-7                        --\n",
       "│    └─Bottleneck: 2-8                   --\n",
       "│    │    └─Conv2d: 3-52                 (131,072)\n",
       "│    │    └─BatchNorm2d: 3-53            (512)\n",
       "│    │    └─Conv2d: 3-54                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-55            (512)\n",
       "│    │    └─Conv2d: 3-56                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-57            (2,048)\n",
       "│    │    └─ReLU: 3-58                   --\n",
       "│    │    └─Sequential: 3-59             (526,336)\n",
       "│    └─Bottleneck: 2-9                   --\n",
       "│    │    └─Conv2d: 3-60                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-61            (512)\n",
       "│    │    └─Conv2d: 3-62                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-63            (512)\n",
       "│    │    └─Conv2d: 3-64                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-65            (2,048)\n",
       "│    │    └─ReLU: 3-66                   --\n",
       "│    └─Bottleneck: 2-10                  --\n",
       "│    │    └─Conv2d: 3-67                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-68            (512)\n",
       "│    │    └─Conv2d: 3-69                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-70            (512)\n",
       "│    │    └─Conv2d: 3-71                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-72            (2,048)\n",
       "│    │    └─ReLU: 3-73                   --\n",
       "│    └─Bottleneck: 2-11                  --\n",
       "│    │    └─Conv2d: 3-74                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-75            (512)\n",
       "│    │    └─Conv2d: 3-76                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-77            (512)\n",
       "│    │    └─Conv2d: 3-78                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-79            (2,048)\n",
       "│    │    └─ReLU: 3-80                   --\n",
       "│    └─Bottleneck: 2-12                  --\n",
       "│    │    └─Conv2d: 3-81                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-82            (512)\n",
       "│    │    └─Conv2d: 3-83                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-84            (512)\n",
       "│    │    └─Conv2d: 3-85                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-86            (2,048)\n",
       "│    │    └─ReLU: 3-87                   --\n",
       "│    └─Bottleneck: 2-13                  --\n",
       "│    │    └─Conv2d: 3-88                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-89            (512)\n",
       "│    │    └─Conv2d: 3-90                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-91            (512)\n",
       "│    │    └─Conv2d: 3-92                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-93            (2,048)\n",
       "│    │    └─ReLU: 3-94                   --\n",
       "├─Sequential: 1-8                        --\n",
       "│    └─Bottleneck: 2-14                  --\n",
       "│    │    └─Conv2d: 3-95                 (524,288)\n",
       "│    │    └─BatchNorm2d: 3-96            (1,024)\n",
       "│    │    └─Conv2d: 3-97                 (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-98            (1,024)\n",
       "│    │    └─Conv2d: 3-99                 (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-100           (4,096)\n",
       "│    │    └─ReLU: 3-101                  --\n",
       "│    │    └─Sequential: 3-102            (2,101,248)\n",
       "│    └─Bottleneck: 2-15                  --\n",
       "│    │    └─Conv2d: 3-103                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-104           (1,024)\n",
       "│    │    └─Conv2d: 3-105                (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-106           (1,024)\n",
       "│    │    └─Conv2d: 3-107                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-108           (4,096)\n",
       "│    │    └─ReLU: 3-109                  --\n",
       "│    └─Bottleneck: 2-16                  --\n",
       "│    │    └─Conv2d: 3-110                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-111           (1,024)\n",
       "│    │    └─Conv2d: 3-112                (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-113           (1,024)\n",
       "│    │    └─Conv2d: 3-114                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-115           (4,096)\n",
       "│    │    └─ReLU: 3-116                  --\n",
       "├─AdaptiveAvgPool2d: 1-9                 --\n",
       "├─Sequential: 1-10                       --\n",
       "│    └─Linear: 2-17                      2,098,176\n",
       "│    └─ReLU: 2-18                        --\n",
       "│    └─Dropout: 2-19                     --\n",
       "│    └─Linear: 2-20                      524,800\n",
       "│    └─ReLU: 2-21                        --\n",
       "│    └─Dropout: 2-22                     --\n",
       "│    └─Linear: 2-23                      51,300\n",
       "=================================================================\n",
       "Total params: 26,182,308\n",
       "Trainable params: 2,674,276\n",
       "Non-trainable params: 23,508,032\n",
       "================================================================="
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resnet_model_50 = add_custom_layers(resnet_model_50)\n",
    "summary(resnet_model_50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform Data using model Data Loaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_train_dataloader, resnet_val_dataloader, resnet_test_dataloader = get_resnet_dataloaders(resnet_weight_50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1\n",
      "Train loss: 1.3481 | Train accuracy: 60.770%\n",
      "Val loss: 0.5813 | Val accuracy: 85.117%\n",
      "| Epoch time: 89.49 seconds\n",
      "Epoch: 2\n",
      "Train loss: 0.8833 | Train accuracy: 73.870%\n",
      "Val loss: 0.4373 | Val accuracy: 87.344%\n",
      "| Epoch time: 69.63 seconds\n",
      "Epoch: 3\n",
      "Train loss: 0.6921 | Train accuracy: 79.431%\n",
      "Val loss: 0.3509 | Val accuracy: 90.469%\n",
      "| Epoch time: 70.79 seconds\n",
      "Epoch: 4\n",
      "Train loss: 0.5773 | Train accuracy: 82.715%\n",
      "Val loss: 0.3294 | Val accuracy: 91.875%\n",
      "| Epoch time: 72.29 seconds\n",
      "Epoch: 5\n",
      "Train loss: 0.4954 | Train accuracy: 85.090%\n",
      "Val loss: 0.2843 | Val accuracy: 91.562%\n",
      "| Epoch time: 79.76 seconds\n",
      "Epoch: 6\n",
      "Train loss: 0.4531 | Train accuracy: 86.309%\n",
      "Val loss: 0.2832 | Val accuracy: 90.703%\n",
      "| Epoch time: 72.06 seconds\n",
      "Epoch: 7\n",
      "Train loss: 0.4187 | Train accuracy: 87.125%\n",
      "Val loss: 0.2481 | Val accuracy: 91.680%\n",
      "| Epoch time: 72.93 seconds\n",
      "Epoch: 8\n",
      "Train loss: 0.3741 | Train accuracy: 88.513%\n",
      "Val loss: 0.2713 | Val accuracy: 91.992%\n",
      "| Epoch time: 72.61 seconds\n",
      "Epoch: 9\n",
      "Train loss: 0.3492 | Train accuracy: 89.328%\n",
      "Val loss: 0.2677 | Val accuracy: 93.828%\n",
      "| Epoch time: 72.56 seconds\n",
      "Epoch: 10\n",
      "Train loss: 0.3222 | Train accuracy: 89.972%\n",
      "Val loss: 0.2460 | Val accuracy: 92.852%\n",
      "| Epoch time: 72.46 seconds\n",
      "Epoch: 11\n",
      "Train loss: 0.2886 | Train accuracy: 90.872%\n",
      "Val loss: 0.2641 | Val accuracy: 92.578%\n",
      "| Epoch time: 72.58 seconds\n",
      "Epoch: 12\n",
      "Train loss: 0.2793 | Train accuracy: 91.431%\n",
      "Val loss: 0.2389 | Val accuracy: 92.852%\n",
      "| Epoch time: 73.59 seconds\n",
      "Epoch: 13\n",
      "Train loss: 0.2692 | Train accuracy: 91.294%\n",
      "Val loss: 0.1847 | Val accuracy: 93.516%\n",
      "| Epoch time: 72.54 seconds\n",
      "Epoch: 14\n",
      "Train loss: 0.2608 | Train accuracy: 91.839%\n",
      "Val loss: 0.1818 | Val accuracy: 95.586%\n",
      "| Epoch time: 72.61 seconds\n",
      "Epoch: 15\n",
      "Train loss: 0.2490 | Train accuracy: 91.924%\n",
      "Val loss: 0.2250 | Val accuracy: 93.633%\n",
      "| Epoch time: 71.58 seconds\n",
      "Epoch: 16\n",
      "Train loss: 0.2395 | Train accuracy: 92.245%\n",
      "Val loss: 0.2117 | Val accuracy: 94.023%\n",
      "| Epoch time: 72.30 seconds\n",
      "Epoch: 17\n",
      "Train loss: 0.2155 | Train accuracy: 93.398%\n",
      "Val loss: 0.2164 | Val accuracy: 93.750%\n",
      "| Epoch time: 72.65 seconds\n",
      "Epoch: 18\n",
      "Train loss: 0.2177 | Train accuracy: 93.181%\n",
      "Val loss: 0.2608 | Val accuracy: 92.461%\n",
      "| Epoch time: 71.83 seconds\n",
      "Epoch: 19\n",
      "Train loss: 0.2179 | Train accuracy: 93.020%\n",
      "Val loss: 0.2758 | Val accuracy: 92.266%\n",
      "| Epoch time: 72.60 seconds\n",
      "Epoch: 20\n",
      "Train loss: 0.2073 | Train accuracy: 93.472%\n",
      "Val loss: 0.2525 | Val accuracy: 93.633%\n",
      "| Epoch time: 72.78 seconds\n",
      "Epoch: 21\n",
      "Train loss: 0.1958 | Train accuracy: 93.750%\n",
      "Val loss: 0.2678 | Val accuracy: 93.633%\n",
      "| Epoch time: 72.93 seconds\n",
      "Epoch: 22\n",
      "Train loss: 0.1913 | Train accuracy: 93.723%\n",
      "Val loss: 0.2081 | Val accuracy: 93.438%\n",
      "| Epoch time: 74.90 seconds\n",
      "Epoch: 23\n",
      "Train loss: 0.1883 | Train accuracy: 94.134%\n",
      "Val loss: 0.2524 | Val accuracy: 94.141%\n",
      "| Epoch time: 121.84 seconds\n",
      "Epoch: 24\n",
      "Train loss: 0.1725 | Train accuracy: 94.397%\n",
      "Val loss: 0.2029 | Val accuracy: 93.750%\n",
      "| Epoch time: 74.00 seconds\n",
      "Epoch: 25\n",
      "Train loss: 0.1754 | Train accuracy: 94.565%\n",
      "Val loss: 0.2271 | Val accuracy: 94.922%\n",
      "| Epoch time: 70.71 seconds\n",
      "Epoch: 26\n",
      "Train loss: 0.1692 | Train accuracy: 94.708%\n",
      "Val loss: 0.1990 | Val accuracy: 94.414%\n",
      "| Epoch time: 71.81 seconds\n",
      "Epoch: 27\n",
      "Train loss: 0.1670 | Train accuracy: 94.723%\n",
      "Val loss: 0.2649 | Val accuracy: 92.773%\n",
      "| Epoch time: 70.60 seconds\n",
      "Epoch: 28\n",
      "Train loss: 0.1751 | Train accuracy: 94.772%\n",
      "Val loss: 0.2075 | Val accuracy: 94.336%\n",
      "| Epoch time: 70.42 seconds\n",
      "Epoch: 29\n",
      "Train loss: 0.1566 | Train accuracy: 95.078%\n",
      "Val loss: 0.1944 | Val accuracy: 93.711%\n",
      "| Epoch time: 71.17 seconds\n",
      "Epoch: 30\n",
      "Train loss: 0.1708 | Train accuracy: 94.445%\n",
      "Val loss: 0.2070 | Val accuracy: 95.195%\n",
      "| Epoch time: 71.23 seconds\n"
     ]
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          1.3480680121629725,
          0.8833459618532262,
          0.692072065061585,
          0.5772530896615643,
          0.4954153271965788,
          0.453094252654444,
          0.418717446096983,
          0.3741211166949634,
          0.34923230674806366,
          0.3221783692026025,
          0.28860971090600107,
          0.2792521447419979,
          0.26915062030433057,
          0.2608437033687934,
          0.2489897339158996,
          0.2394809110787525,
          0.21554711575827326,
          0.21765065288074067,
          0.217928714412419,
          0.20725561049877184,
          0.195797999941201,
          0.19132100843043195,
          0.18829325555439738,
          0.17251371386708136,
          0.17540927117765479,
          0.16915238045500283,
          0.16703038798618655,
          0.1751354774096054,
          0.15664498288608134,
          0.17077792813489434
         ]
        },
        {
         "mode": "lines",
         "name": "Val loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.5813413187861443,
          0.43726950511336327,
          0.3509037373587489,
          0.32940409588627517,
          0.28430355060845613,
          0.283223828766495,
          0.24809253681451082,
          0.27130098082125187,
          0.2677333443425596,
          0.24604486930184066,
          0.2640923170838505,
          0.2389390486641787,
          0.18471944716293365,
          0.1817703563719988,
          0.22495795134454966,
          0.21166604850441217,
          0.21635676606092602,
          0.26080447249114513,
          0.27577771828509867,
          0.2525363254826516,
          0.2678362485021353,
          0.20810159490793012,
          0.2523773925786372,
          0.20289449213305488,
          0.22713888005819172,
          0.1989917066966882,
          0.2649021811084822,
          0.20753241528291255,
          0.19442398729734123,
          0.20703681139275432
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Loss over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Loss"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.6077014218009479,
          0.738699644549763,
          0.7943127962085308,
          0.8271475118483412,
          0.8509034360189573,
          0.8630924170616114,
          0.871252962085308,
          0.8851303317535545,
          0.8932760663507109,
          0.8997186018957345,
          0.9087233412322274,
          0.9143068720379146,
          0.9129443127962085,
          0.9183945497630331,
          0.9192387440758294,
          0.922452606635071,
          0.9339751184834123,
          0.9318127962085309,
          0.9301984597156397,
          0.9347156398104265,
          0.9375,
          0.9372334123222749,
          0.9413359004739336,
          0.9439721563981043,
          0.9456457345971564,
          0.9470823459715639,
          0.9472304502369668,
          0.9477191943127962,
          0.9507849526066351,
          0.9444460900473933
         ]
        },
        {
         "mode": "lines",
         "name": "Val Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.851171875,
          0.8734375,
          0.9046875,
          0.91875,
          0.915625,
          0.90703125,
          0.916796875,
          0.919921875,
          0.93828125,
          0.928515625,
          0.92578125,
          0.928515625,
          0.93515625,
          0.955859375,
          0.936328125,
          0.940234375,
          0.9375,
          0.924609375,
          0.92265625,
          0.936328125,
          0.936328125,
          0.934375,
          0.94140625,
          0.9375,
          0.94921875,
          0.944140625,
          0.927734375,
          0.943359375,
          0.937109375,
          0.951953125
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Accuracy over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Accuracy"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Actual training ResNet model\n",
    "resnet_results,training_time = training_loop(model = resnet_model_50,\n",
    "                               train_dataloader = resnet_train_dataloader,\n",
    "                               val_dataloader = resnet_val_dataloader,\n",
    "                               device = device,\n",
    "                               epochs = 30,\n",
    "                               patience = 5\n",
    "                              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Model Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_results.to_csv('models_results/resnet_model_50.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calcluate Model Testing Accuracy and Training Time Measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing Accuracy is 95.6%\n"
     ]
    }
   ],
   "source": [
    "test_accuracy = test_accuracy_resnet(resnet_model_50,resnet_test_dataloader,device)\n",
    "print(f\"Testing Accuracy is {test_accuracy}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Training Time 37.3207 Minitues\n"
     ]
    }
   ],
   "source": [
    "print(f'Model Training Time {round(training_time/60,4)} Minitues')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Test Accuracy and Training Time in csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_times = pd.read_csv('models_results/training_time.csv')\n",
    "row = {'Model':'ResNet50','Testing Accuracy':test_accuracy,'Training_Time(Minutes)':round(training_time/60,4)}\n",
    "\n",
    "# Use the loc method to add the new row to the DataFrame\n",
    "training_times.loc[len(training_times)] = row\n",
    "training_times.to_csv('models_results/training_time.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Trained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(resnet_model_50, 'trained_models/resnet_model_50.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ResNet152\n",
    "### ResNet152 download weights and Pretrained Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_weight_152 = torchvision.models.ResNet152_Weights.DEFAULT\n",
    "\n",
    "resnet_model_152 = torchvision.models.resnet152(weights = resnet_weight_152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add some custom layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "=================================================================\n",
       "Layer (type:depth-idx)                   Param #\n",
       "=================================================================\n",
       "ResNet                                   --\n",
       "├─Conv2d: 1-1                            (9,408)\n",
       "├─BatchNorm2d: 1-2                       (128)\n",
       "├─ReLU: 1-3                              --\n",
       "├─MaxPool2d: 1-4                         --\n",
       "├─Sequential: 1-5                        --\n",
       "│    └─Bottleneck: 2-1                   --\n",
       "│    │    └─Conv2d: 3-1                  (4,096)\n",
       "│    │    └─BatchNorm2d: 3-2             (128)\n",
       "│    │    └─Conv2d: 3-3                  (36,864)\n",
       "│    │    └─BatchNorm2d: 3-4             (128)\n",
       "│    │    └─Conv2d: 3-5                  (16,384)\n",
       "│    │    └─BatchNorm2d: 3-6             (512)\n",
       "│    │    └─ReLU: 3-7                    --\n",
       "│    │    └─Sequential: 3-8              (16,896)\n",
       "│    └─Bottleneck: 2-2                   --\n",
       "│    │    └─Conv2d: 3-9                  (16,384)\n",
       "│    │    └─BatchNorm2d: 3-10            (128)\n",
       "│    │    └─Conv2d: 3-11                 (36,864)\n",
       "│    │    └─BatchNorm2d: 3-12            (128)\n",
       "│    │    └─Conv2d: 3-13                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-14            (512)\n",
       "│    │    └─ReLU: 3-15                   --\n",
       "│    └─Bottleneck: 2-3                   --\n",
       "│    │    └─Conv2d: 3-16                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-17            (128)\n",
       "│    │    └─Conv2d: 3-18                 (36,864)\n",
       "│    │    └─BatchNorm2d: 3-19            (128)\n",
       "│    │    └─Conv2d: 3-20                 (16,384)\n",
       "│    │    └─BatchNorm2d: 3-21            (512)\n",
       "│    │    └─ReLU: 3-22                   --\n",
       "├─Sequential: 1-6                        --\n",
       "│    └─Bottleneck: 2-4                   --\n",
       "│    │    └─Conv2d: 3-23                 (32,768)\n",
       "│    │    └─BatchNorm2d: 3-24            (256)\n",
       "│    │    └─Conv2d: 3-25                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-26            (256)\n",
       "│    │    └─Conv2d: 3-27                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-28            (1,024)\n",
       "│    │    └─ReLU: 3-29                   --\n",
       "│    │    └─Sequential: 3-30             (132,096)\n",
       "│    └─Bottleneck: 2-5                   --\n",
       "│    │    └─Conv2d: 3-31                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-32            (256)\n",
       "│    │    └─Conv2d: 3-33                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-34            (256)\n",
       "│    │    └─Conv2d: 3-35                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-36            (1,024)\n",
       "│    │    └─ReLU: 3-37                   --\n",
       "│    └─Bottleneck: 2-6                   --\n",
       "│    │    └─Conv2d: 3-38                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-39            (256)\n",
       "│    │    └─Conv2d: 3-40                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-41            (256)\n",
       "│    │    └─Conv2d: 3-42                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-43            (1,024)\n",
       "│    │    └─ReLU: 3-44                   --\n",
       "│    └─Bottleneck: 2-7                   --\n",
       "│    │    └─Conv2d: 3-45                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-46            (256)\n",
       "│    │    └─Conv2d: 3-47                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-48            (256)\n",
       "│    │    └─Conv2d: 3-49                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-50            (1,024)\n",
       "│    │    └─ReLU: 3-51                   --\n",
       "│    └─Bottleneck: 2-8                   --\n",
       "│    │    └─Conv2d: 3-52                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-53            (256)\n",
       "│    │    └─Conv2d: 3-54                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-55            (256)\n",
       "│    │    └─Conv2d: 3-56                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-57            (1,024)\n",
       "│    │    └─ReLU: 3-58                   --\n",
       "│    └─Bottleneck: 2-9                   --\n",
       "│    │    └─Conv2d: 3-59                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-60            (256)\n",
       "│    │    └─Conv2d: 3-61                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-62            (256)\n",
       "│    │    └─Conv2d: 3-63                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-64            (1,024)\n",
       "│    │    └─ReLU: 3-65                   --\n",
       "│    └─Bottleneck: 2-10                  --\n",
       "│    │    └─Conv2d: 3-66                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-67            (256)\n",
       "│    │    └─Conv2d: 3-68                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-69            (256)\n",
       "│    │    └─Conv2d: 3-70                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-71            (1,024)\n",
       "│    │    └─ReLU: 3-72                   --\n",
       "│    └─Bottleneck: 2-11                  --\n",
       "│    │    └─Conv2d: 3-73                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-74            (256)\n",
       "│    │    └─Conv2d: 3-75                 (147,456)\n",
       "│    │    └─BatchNorm2d: 3-76            (256)\n",
       "│    │    └─Conv2d: 3-77                 (65,536)\n",
       "│    │    └─BatchNorm2d: 3-78            (1,024)\n",
       "│    │    └─ReLU: 3-79                   --\n",
       "├─Sequential: 1-7                        --\n",
       "│    └─Bottleneck: 2-12                  --\n",
       "│    │    └─Conv2d: 3-80                 (131,072)\n",
       "│    │    └─BatchNorm2d: 3-81            (512)\n",
       "│    │    └─Conv2d: 3-82                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-83            (512)\n",
       "│    │    └─Conv2d: 3-84                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-85            (2,048)\n",
       "│    │    └─ReLU: 3-86                   --\n",
       "│    │    └─Sequential: 3-87             (526,336)\n",
       "│    └─Bottleneck: 2-13                  --\n",
       "│    │    └─Conv2d: 3-88                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-89            (512)\n",
       "│    │    └─Conv2d: 3-90                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-91            (512)\n",
       "│    │    └─Conv2d: 3-92                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-93            (2,048)\n",
       "│    │    └─ReLU: 3-94                   --\n",
       "│    └─Bottleneck: 2-14                  --\n",
       "│    │    └─Conv2d: 3-95                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-96            (512)\n",
       "│    │    └─Conv2d: 3-97                 (589,824)\n",
       "│    │    └─BatchNorm2d: 3-98            (512)\n",
       "│    │    └─Conv2d: 3-99                 (262,144)\n",
       "│    │    └─BatchNorm2d: 3-100           (2,048)\n",
       "│    │    └─ReLU: 3-101                  --\n",
       "│    └─Bottleneck: 2-15                  --\n",
       "│    │    └─Conv2d: 3-102                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-103           (512)\n",
       "│    │    └─Conv2d: 3-104                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-105           (512)\n",
       "│    │    └─Conv2d: 3-106                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-107           (2,048)\n",
       "│    │    └─ReLU: 3-108                  --\n",
       "│    └─Bottleneck: 2-16                  --\n",
       "│    │    └─Conv2d: 3-109                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-110           (512)\n",
       "│    │    └─Conv2d: 3-111                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-112           (512)\n",
       "│    │    └─Conv2d: 3-113                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-114           (2,048)\n",
       "│    │    └─ReLU: 3-115                  --\n",
       "│    └─Bottleneck: 2-17                  --\n",
       "│    │    └─Conv2d: 3-116                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-117           (512)\n",
       "│    │    └─Conv2d: 3-118                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-119           (512)\n",
       "│    │    └─Conv2d: 3-120                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-121           (2,048)\n",
       "│    │    └─ReLU: 3-122                  --\n",
       "│    └─Bottleneck: 2-18                  --\n",
       "│    │    └─Conv2d: 3-123                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-124           (512)\n",
       "│    │    └─Conv2d: 3-125                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-126           (512)\n",
       "│    │    └─Conv2d: 3-127                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-128           (2,048)\n",
       "│    │    └─ReLU: 3-129                  --\n",
       "│    └─Bottleneck: 2-19                  --\n",
       "│    │    └─Conv2d: 3-130                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-131           (512)\n",
       "│    │    └─Conv2d: 3-132                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-133           (512)\n",
       "│    │    └─Conv2d: 3-134                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-135           (2,048)\n",
       "│    │    └─ReLU: 3-136                  --\n",
       "│    └─Bottleneck: 2-20                  --\n",
       "│    │    └─Conv2d: 3-137                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-138           (512)\n",
       "│    │    └─Conv2d: 3-139                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-140           (512)\n",
       "│    │    └─Conv2d: 3-141                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-142           (2,048)\n",
       "│    │    └─ReLU: 3-143                  --\n",
       "│    └─Bottleneck: 2-21                  --\n",
       "│    │    └─Conv2d: 3-144                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-145           (512)\n",
       "│    │    └─Conv2d: 3-146                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-147           (512)\n",
       "│    │    └─Conv2d: 3-148                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-149           (2,048)\n",
       "│    │    └─ReLU: 3-150                  --\n",
       "│    └─Bottleneck: 2-22                  --\n",
       "│    │    └─Conv2d: 3-151                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-152           (512)\n",
       "│    │    └─Conv2d: 3-153                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-154           (512)\n",
       "│    │    └─Conv2d: 3-155                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-156           (2,048)\n",
       "│    │    └─ReLU: 3-157                  --\n",
       "│    └─Bottleneck: 2-23                  --\n",
       "│    │    └─Conv2d: 3-158                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-159           (512)\n",
       "│    │    └─Conv2d: 3-160                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-161           (512)\n",
       "│    │    └─Conv2d: 3-162                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-163           (2,048)\n",
       "│    │    └─ReLU: 3-164                  --\n",
       "│    └─Bottleneck: 2-24                  --\n",
       "│    │    └─Conv2d: 3-165                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-166           (512)\n",
       "│    │    └─Conv2d: 3-167                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-168           (512)\n",
       "│    │    └─Conv2d: 3-169                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-170           (2,048)\n",
       "│    │    └─ReLU: 3-171                  --\n",
       "│    └─Bottleneck: 2-25                  --\n",
       "│    │    └─Conv2d: 3-172                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-173           (512)\n",
       "│    │    └─Conv2d: 3-174                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-175           (512)\n",
       "│    │    └─Conv2d: 3-176                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-177           (2,048)\n",
       "│    │    └─ReLU: 3-178                  --\n",
       "│    └─Bottleneck: 2-26                  --\n",
       "│    │    └─Conv2d: 3-179                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-180           (512)\n",
       "│    │    └─Conv2d: 3-181                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-182           (512)\n",
       "│    │    └─Conv2d: 3-183                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-184           (2,048)\n",
       "│    │    └─ReLU: 3-185                  --\n",
       "│    └─Bottleneck: 2-27                  --\n",
       "│    │    └─Conv2d: 3-186                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-187           (512)\n",
       "│    │    └─Conv2d: 3-188                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-189           (512)\n",
       "│    │    └─Conv2d: 3-190                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-191           (2,048)\n",
       "│    │    └─ReLU: 3-192                  --\n",
       "│    └─Bottleneck: 2-28                  --\n",
       "│    │    └─Conv2d: 3-193                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-194           (512)\n",
       "│    │    └─Conv2d: 3-195                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-196           (512)\n",
       "│    │    └─Conv2d: 3-197                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-198           (2,048)\n",
       "│    │    └─ReLU: 3-199                  --\n",
       "│    └─Bottleneck: 2-29                  --\n",
       "│    │    └─Conv2d: 3-200                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-201           (512)\n",
       "│    │    └─Conv2d: 3-202                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-203           (512)\n",
       "│    │    └─Conv2d: 3-204                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-205           (2,048)\n",
       "│    │    └─ReLU: 3-206                  --\n",
       "│    └─Bottleneck: 2-30                  --\n",
       "│    │    └─Conv2d: 3-207                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-208           (512)\n",
       "│    │    └─Conv2d: 3-209                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-210           (512)\n",
       "│    │    └─Conv2d: 3-211                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-212           (2,048)\n",
       "│    │    └─ReLU: 3-213                  --\n",
       "│    └─Bottleneck: 2-31                  --\n",
       "│    │    └─Conv2d: 3-214                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-215           (512)\n",
       "│    │    └─Conv2d: 3-216                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-217           (512)\n",
       "│    │    └─Conv2d: 3-218                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-219           (2,048)\n",
       "│    │    └─ReLU: 3-220                  --\n",
       "│    └─Bottleneck: 2-32                  --\n",
       "│    │    └─Conv2d: 3-221                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-222           (512)\n",
       "│    │    └─Conv2d: 3-223                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-224           (512)\n",
       "│    │    └─Conv2d: 3-225                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-226           (2,048)\n",
       "│    │    └─ReLU: 3-227                  --\n",
       "│    └─Bottleneck: 2-33                  --\n",
       "│    │    └─Conv2d: 3-228                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-229           (512)\n",
       "│    │    └─Conv2d: 3-230                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-231           (512)\n",
       "│    │    └─Conv2d: 3-232                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-233           (2,048)\n",
       "│    │    └─ReLU: 3-234                  --\n",
       "│    └─Bottleneck: 2-34                  --\n",
       "│    │    └─Conv2d: 3-235                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-236           (512)\n",
       "│    │    └─Conv2d: 3-237                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-238           (512)\n",
       "│    │    └─Conv2d: 3-239                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-240           (2,048)\n",
       "│    │    └─ReLU: 3-241                  --\n",
       "│    └─Bottleneck: 2-35                  --\n",
       "│    │    └─Conv2d: 3-242                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-243           (512)\n",
       "│    │    └─Conv2d: 3-244                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-245           (512)\n",
       "│    │    └─Conv2d: 3-246                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-247           (2,048)\n",
       "│    │    └─ReLU: 3-248                  --\n",
       "│    └─Bottleneck: 2-36                  --\n",
       "│    │    └─Conv2d: 3-249                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-250           (512)\n",
       "│    │    └─Conv2d: 3-251                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-252           (512)\n",
       "│    │    └─Conv2d: 3-253                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-254           (2,048)\n",
       "│    │    └─ReLU: 3-255                  --\n",
       "│    └─Bottleneck: 2-37                  --\n",
       "│    │    └─Conv2d: 3-256                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-257           (512)\n",
       "│    │    └─Conv2d: 3-258                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-259           (512)\n",
       "│    │    └─Conv2d: 3-260                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-261           (2,048)\n",
       "│    │    └─ReLU: 3-262                  --\n",
       "│    └─Bottleneck: 2-38                  --\n",
       "│    │    └─Conv2d: 3-263                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-264           (512)\n",
       "│    │    └─Conv2d: 3-265                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-266           (512)\n",
       "│    │    └─Conv2d: 3-267                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-268           (2,048)\n",
       "│    │    └─ReLU: 3-269                  --\n",
       "│    └─Bottleneck: 2-39                  --\n",
       "│    │    └─Conv2d: 3-270                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-271           (512)\n",
       "│    │    └─Conv2d: 3-272                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-273           (512)\n",
       "│    │    └─Conv2d: 3-274                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-275           (2,048)\n",
       "│    │    └─ReLU: 3-276                  --\n",
       "│    └─Bottleneck: 2-40                  --\n",
       "│    │    └─Conv2d: 3-277                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-278           (512)\n",
       "│    │    └─Conv2d: 3-279                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-280           (512)\n",
       "│    │    └─Conv2d: 3-281                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-282           (2,048)\n",
       "│    │    └─ReLU: 3-283                  --\n",
       "│    └─Bottleneck: 2-41                  --\n",
       "│    │    └─Conv2d: 3-284                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-285           (512)\n",
       "│    │    └─Conv2d: 3-286                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-287           (512)\n",
       "│    │    └─Conv2d: 3-288                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-289           (2,048)\n",
       "│    │    └─ReLU: 3-290                  --\n",
       "│    └─Bottleneck: 2-42                  --\n",
       "│    │    └─Conv2d: 3-291                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-292           (512)\n",
       "│    │    └─Conv2d: 3-293                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-294           (512)\n",
       "│    │    └─Conv2d: 3-295                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-296           (2,048)\n",
       "│    │    └─ReLU: 3-297                  --\n",
       "│    └─Bottleneck: 2-43                  --\n",
       "│    │    └─Conv2d: 3-298                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-299           (512)\n",
       "│    │    └─Conv2d: 3-300                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-301           (512)\n",
       "│    │    └─Conv2d: 3-302                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-303           (2,048)\n",
       "│    │    └─ReLU: 3-304                  --\n",
       "│    └─Bottleneck: 2-44                  --\n",
       "│    │    └─Conv2d: 3-305                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-306           (512)\n",
       "│    │    └─Conv2d: 3-307                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-308           (512)\n",
       "│    │    └─Conv2d: 3-309                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-310           (2,048)\n",
       "│    │    └─ReLU: 3-311                  --\n",
       "│    └─Bottleneck: 2-45                  --\n",
       "│    │    └─Conv2d: 3-312                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-313           (512)\n",
       "│    │    └─Conv2d: 3-314                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-315           (512)\n",
       "│    │    └─Conv2d: 3-316                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-317           (2,048)\n",
       "│    │    └─ReLU: 3-318                  --\n",
       "│    └─Bottleneck: 2-46                  --\n",
       "│    │    └─Conv2d: 3-319                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-320           (512)\n",
       "│    │    └─Conv2d: 3-321                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-322           (512)\n",
       "│    │    └─Conv2d: 3-323                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-324           (2,048)\n",
       "│    │    └─ReLU: 3-325                  --\n",
       "│    └─Bottleneck: 2-47                  --\n",
       "│    │    └─Conv2d: 3-326                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-327           (512)\n",
       "│    │    └─Conv2d: 3-328                (589,824)\n",
       "│    │    └─BatchNorm2d: 3-329           (512)\n",
       "│    │    └─Conv2d: 3-330                (262,144)\n",
       "│    │    └─BatchNorm2d: 3-331           (2,048)\n",
       "│    │    └─ReLU: 3-332                  --\n",
       "├─Sequential: 1-8                        --\n",
       "│    └─Bottleneck: 2-48                  --\n",
       "│    │    └─Conv2d: 3-333                (524,288)\n",
       "│    │    └─BatchNorm2d: 3-334           (1,024)\n",
       "│    │    └─Conv2d: 3-335                (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-336           (1,024)\n",
       "│    │    └─Conv2d: 3-337                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-338           (4,096)\n",
       "│    │    └─ReLU: 3-339                  --\n",
       "│    │    └─Sequential: 3-340            (2,101,248)\n",
       "│    └─Bottleneck: 2-49                  --\n",
       "│    │    └─Conv2d: 3-341                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-342           (1,024)\n",
       "│    │    └─Conv2d: 3-343                (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-344           (1,024)\n",
       "│    │    └─Conv2d: 3-345                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-346           (4,096)\n",
       "│    │    └─ReLU: 3-347                  --\n",
       "│    └─Bottleneck: 2-50                  --\n",
       "│    │    └─Conv2d: 3-348                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-349           (1,024)\n",
       "│    │    └─Conv2d: 3-350                (2,359,296)\n",
       "│    │    └─BatchNorm2d: 3-351           (1,024)\n",
       "│    │    └─Conv2d: 3-352                (1,048,576)\n",
       "│    │    └─BatchNorm2d: 3-353           (4,096)\n",
       "│    │    └─ReLU: 3-354                  --\n",
       "├─AdaptiveAvgPool2d: 1-9                 --\n",
       "├─Sequential: 1-10                       --\n",
       "│    └─Linear: 2-51                      2,098,176\n",
       "│    └─ReLU: 2-52                        --\n",
       "│    └─Dropout: 2-53                     --\n",
       "│    └─Linear: 2-54                      524,800\n",
       "│    └─ReLU: 2-55                        --\n",
       "│    └─Dropout: 2-56                     --\n",
       "│    └─Linear: 2-57                      51,300\n",
       "=================================================================\n",
       "Total params: 60,818,084\n",
       "Trainable params: 2,674,276\n",
       "Non-trainable params: 58,143,808\n",
       "================================================================="
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resnet_model_152 = add_custom_layers(resnet_model_152)\n",
    "summary(resnet_model_152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform Data using model Data Loaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_train_dataloader, resnet_val_dataloader, resnet_test_dataloader = get_resnet_dataloaders(resnet_weight_152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1\n",
      "Train loss: 2.6988 | Train accuracy: 33.988%\n",
      "Val loss: 0.9109 | Val accuracy: 75.156%\n",
      "| Epoch time: 125.14 seconds\n",
      "Epoch: 2\n",
      "Train loss: 1.0857 | Train accuracy: 68.205%\n",
      "Val loss: 0.5149 | Val accuracy: 85.391%\n",
      "| Epoch time: 127.26 seconds\n",
      "Epoch: 3\n",
      "Train loss: 0.7806 | Train accuracy: 76.638%\n",
      "Val loss: 0.4604 | Val accuracy: 87.500%\n",
      "| Epoch time: 125.88 seconds\n",
      "Epoch: 4\n",
      "Train loss: 0.6444 | Train accuracy: 80.296%\n",
      "Val loss: 0.3340 | Val accuracy: 89.727%\n",
      "| Epoch time: 126.72 seconds\n",
      "Epoch: 5\n",
      "Train loss: 0.5382 | Train accuracy: 83.586%\n",
      "Val loss: 0.3082 | Val accuracy: 89.492%\n",
      "| Epoch time: 126.68 seconds\n",
      "Epoch: 6\n",
      "Train loss: 0.4919 | Train accuracy: 84.907%\n",
      "Val loss: 0.3027 | Val accuracy: 89.844%\n",
      "| Epoch time: 127.62 seconds\n",
      "Epoch: 7\n",
      "Train loss: 0.4390 | Train accuracy: 86.647%\n",
      "Val loss: 0.2093 | Val accuracy: 93.359%\n",
      "| Epoch time: 129.38 seconds\n",
      "Epoch: 8\n",
      "Train loss: 0.3952 | Train accuracy: 87.958%\n",
      "Val loss: 0.2672 | Val accuracy: 91.797%\n",
      "| Epoch time: 127.86 seconds\n",
      "Epoch: 9\n",
      "Train loss: 0.3701 | Train accuracy: 88.586%\n",
      "Val loss: 0.2377 | Val accuracy: 93.242%\n",
      "| Epoch time: 125.79 seconds\n",
      "Epoch: 10\n",
      "Train loss: 0.3479 | Train accuracy: 89.434%\n",
      "Val loss: 0.2505 | Val accuracy: 92.383%\n",
      "| Epoch time: 126.69 seconds\n",
      "Epoch: 11\n",
      "Train loss: 0.3175 | Train accuracy: 90.102%\n",
      "Val loss: 0.2656 | Val accuracy: 92.383%\n",
      "| Epoch time: 125.92 seconds\n",
      "Epoch: 12\n",
      "Train loss: 0.2982 | Train accuracy: 90.865%\n",
      "Val loss: 0.2148 | Val accuracy: 94.023%\n",
      "| Epoch time: 126.41 seconds\n",
      "Epoch: 13\n",
      "Train loss: 0.2786 | Train accuracy: 91.222%\n",
      "Val loss: 0.2263 | Val accuracy: 92.773%\n",
      "| Epoch time: 126.22 seconds\n",
      "Epoch: 14\n",
      "Train loss: 0.2703 | Train accuracy: 91.657%\n",
      "Val loss: 0.2297 | Val accuracy: 92.578%\n",
      "| Epoch time: 126.63 seconds\n",
      "Epoch: 15\n",
      "Train loss: 0.2696 | Train accuracy: 91.546%\n",
      "Val loss: 0.2416 | Val accuracy: 92.969%\n",
      "| Epoch time: 134.54 seconds\n",
      "Epoch: 16\n",
      "Train loss: 0.2435 | Train accuracy: 92.552%\n",
      "Val loss: 0.2057 | Val accuracy: 92.969%\n",
      "| Epoch time: 127.53 seconds\n",
      "Epoch: 17\n",
      "Train loss: 0.2397 | Train accuracy: 92.307%\n",
      "Val loss: 0.2562 | Val accuracy: 91.211%\n",
      "| Epoch time: 126.59 seconds\n",
      "Epoch: 18\n",
      "Train loss: 0.2347 | Train accuracy: 92.608%\n",
      "Val loss: 0.2635 | Val accuracy: 93.164%\n",
      "| Epoch time: 127.12 seconds\n",
      "Epoch: 19\n",
      "Train loss: 0.2236 | Train accuracy: 92.876%\n",
      "Val loss: 0.2035 | Val accuracy: 93.555%\n",
      "| Epoch time: 127.45 seconds\n",
      "Epoch: 20\n",
      "Train loss: 0.2087 | Train accuracy: 93.575%\n",
      "Val loss: 0.2368 | Val accuracy: 93.945%\n",
      "| Epoch time: 126.72 seconds\n",
      "Epoch: 21\n",
      "Train loss: 0.2125 | Train accuracy: 93.482%\n",
      "Val loss: 0.2275 | Val accuracy: 92.266%\n",
      "| Epoch time: 127.53 seconds\n",
      "Epoch: 22\n",
      "Train loss: 0.2025 | Train accuracy: 93.513%\n",
      "Val loss: 0.2196 | Val accuracy: 93.555%\n",
      "| Epoch time: 126.30 seconds\n",
      "Epoch: 23\n",
      "Train loss: 0.2036 | Train accuracy: 93.538%\n",
      "Val loss: 0.2301 | Val accuracy: 92.578%\n",
      "| Epoch time: 127.53 seconds\n",
      "Epoch: 24\n",
      "Train loss: 0.1846 | Train accuracy: 94.379%\n",
      "Val loss: 0.2424 | Val accuracy: 92.461%\n",
      "| Epoch time: 151.17 seconds\n",
      "Epoch: 25\n",
      "Train loss: 0.2061 | Train accuracy: 93.626%\n",
      "Val loss: 0.2154 | Val accuracy: 94.219%\n",
      "| Epoch time: 150.55 seconds\n",
      "Epoch: 26\n",
      "Train loss: 0.1833 | Train accuracy: 94.172%\n",
      "Val loss: 0.2348 | Val accuracy: 94.609%\n",
      "| Epoch time: 148.99 seconds\n",
      "Epoch: 27\n",
      "Train loss: 0.1707 | Train accuracy: 94.582%\n",
      "Val loss: 0.2512 | Val accuracy: 93.164%\n",
      "| Epoch time: 156.31 seconds\n",
      "Epoch: 28\n",
      "Train loss: 0.1849 | Train accuracy: 94.234%\n",
      "Val loss: 0.2299 | Val accuracy: 93.164%\n",
      "| Epoch time: 154.29 seconds\n",
      "Epoch: 29\n",
      "Train loss: 0.1663 | Train accuracy: 94.742%\n",
      "Val loss: 0.2406 | Val accuracy: 93.750%\n",
      "| Epoch time: 110.50 seconds\n",
      "Epoch: 30\n",
      "Train loss: 0.1696 | Train accuracy: 94.636%\n",
      "Val loss: 0.2221 | Val accuracy: 93.633%\n",
      "| Epoch time: 106.86 seconds\n"
     ]
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          2.698774123643812,
          1.0856792332154315,
          0.7806488237392281,
          0.6444337479946738,
          0.5382257551307926,
          0.49186121294566243,
          0.4389922821726562,
          0.3951998638492342,
          0.3700920538756096,
          0.3478518171827375,
          0.31746513187285,
          0.29819614653833104,
          0.2785764014463148,
          0.2703046841255579,
          0.2695705392022763,
          0.243473356706642,
          0.23968514590855175,
          0.23473371721592276,
          0.22360818763815254,
          0.2086756124748226,
          0.21253763013011753,
          0.20248731727363162,
          0.20360046677441465,
          0.18459734901503363,
          0.20609171928890863,
          0.18326832600582338,
          0.17072862252963805,
          0.18493406281815356,
          0.16631446551902318,
          0.16961508301435418
         ]
        },
        {
         "mode": "lines",
         "name": "Val loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.9108989723026752,
          0.5148525689728558,
          0.46035976335406303,
          0.33395227137953043,
          0.308201537001878,
          0.3027209732681513,
          0.20930384600069374,
          0.2672356250695884,
          0.23773601651191711,
          0.25054825004190207,
          0.2656432413496077,
          0.21481352020055056,
          0.22625416639493778,
          0.2297444095602259,
          0.2415792082902044,
          0.2056860621087253,
          0.2562047285027802,
          0.2634591106325388,
          0.2034971808316186,
          0.23680261912522838,
          0.22746757458662614,
          0.21956495894119143,
          0.2300563412718475,
          0.24239460297394544,
          0.21535133046563715,
          0.23477748350705951,
          0.25122828281018883,
          0.2299242252483964,
          0.24060031969565898,
          0.2220753093715757
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Loss over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Loss"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.33988447867298577,
          0.6820497630331753,
          0.7663803317535546,
          0.8029620853080569,
          0.8358560426540285,
          0.849066943127962,
          0.8664691943127961,
          0.8795764218009479,
          0.8858560426540285,
          0.8943424170616114,
          0.9010219194312796,
          0.908649289099526,
          0.9122186018957347,
          0.9165728672985781,
          0.9154620853080568,
          0.92551836492891,
          0.9230746445497631,
          0.9260811611374407,
          0.9287618483412322,
          0.9357523696682464,
          0.9348193127962084,
          0.9351303317535545,
          0.9353821090047393,
          0.9437944312796208,
          0.9362559241706162,
          0.941720971563981,
          0.9458234597156397,
          0.942343009478673,
          0.9474229857819905,
          0.94635663507109
         ]
        },
        {
         "mode": "lines",
         "name": "Val Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.7515625,
          0.85390625,
          0.875,
          0.897265625,
          0.894921875,
          0.8984375,
          0.93359375,
          0.91796875,
          0.932421875,
          0.923828125,
          0.923828125,
          0.940234375,
          0.927734375,
          0.92578125,
          0.9296875,
          0.9296875,
          0.912109375,
          0.931640625,
          0.935546875,
          0.939453125,
          0.92265625,
          0.935546875,
          0.92578125,
          0.924609375,
          0.9421875,
          0.94609375,
          0.931640625,
          0.931640625,
          0.9375,
          0.936328125
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Accuracy over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Accuracy"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Actual training ResNet model\n",
    "resnet_results,training_time = training_loop(model = resnet_model_152,\n",
    "                               train_dataloader = resnet_train_dataloader,\n",
    "                               val_dataloader = resnet_val_dataloader,\n",
    "                               device = device,\n",
    "                               epochs = 30,\n",
    "                               patience = 5\n",
    "                              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Model Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_results.to_csv('models_results/resnet_model_152.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calcluate Model Testing Accuracy and Training Time Measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing Accuracy is 95.19999999999999%\n"
     ]
    }
   ],
   "source": [
    "test_accuracy = test_accuracy_resnet(resnet_model_152,resnet_test_dataloader,device)\n",
    "print(f\"Testing Accuracy is {test_accuracy}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Training Time 65.0694 Minitues\n"
     ]
    }
   ],
   "source": [
    "print(f'Model Training Time {round(training_time/60,4)} Minitues')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Test Accuracy and Training Time in csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_times = pd.read_csv('models_results/training_time.csv')\n",
    "row = {'Model':'ResNet50','Testing Accuracy':test_accuracy,'Training_Time(Minutes)':round(training_time/60,4)}\n",
    "# Use the loc method to add the new row to the DataFrame\n",
    "training_times.loc[len(training_times)] = row\n",
    "training_times.to_csv('models_results/training_time.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Testing Accuracy</th>\n",
       "      <th>Training_Time(Minutes)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ResNet50</td>\n",
       "      <td>95.6</td>\n",
       "      <td>37.3207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ResNet152</td>\n",
       "      <td>95.2</td>\n",
       "      <td>65.0694</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Model  Testing Accuracy  Training_Time(Minutes)\n",
       "0   ResNet50              95.6                 37.3207\n",
       "1  ResNet152              95.2                 65.0694"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_times = pd.read_csv('models_results/training_time.csv')\n",
    "training_times.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Trained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(resnet_model_152, 'trained_models/resnet_model_152.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ResNet101\n",
    "### ResNet101 download weights and Pretrained Models\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_weight_101 = torchvision.models.ResNet101_Weights.DEFAULT\n",
    "\n",
    "resnet_model_101 = torchvision.models.resnet101(weights = resnet_weight_101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add some custom layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "=================================================================\n",
       "Layer (type:depth-idx)                   Param #\n",
       "=================================================================\n",
       "ResNet                                   --\n",
       "├─Conv2d: 1-1                            9,408\n",
       "├─BatchNorm2d: 1-2                       128\n",
       "├─ReLU: 1-3                              --\n",
       "├─MaxPool2d: 1-4                         --\n",
       "├─Sequential: 1-5                        --\n",
       "│    └─Bottleneck: 2-1                   --\n",
       "│    │    └─Conv2d: 3-1                  4,096\n",
       "│    │    └─BatchNorm2d: 3-2             128\n",
       "│    │    └─Conv2d: 3-3                  36,864\n",
       "│    │    └─BatchNorm2d: 3-4             128\n",
       "│    │    └─Conv2d: 3-5                  16,384\n",
       "│    │    └─BatchNorm2d: 3-6             512\n",
       "│    │    └─ReLU: 3-7                    --\n",
       "│    │    └─Sequential: 3-8              16,896\n",
       "│    └─Bottleneck: 2-2                   --\n",
       "│    │    └─Conv2d: 3-9                  16,384\n",
       "│    │    └─BatchNorm2d: 3-10            128\n",
       "│    │    └─Conv2d: 3-11                 36,864\n",
       "│    │    └─BatchNorm2d: 3-12            128\n",
       "│    │    └─Conv2d: 3-13                 16,384\n",
       "│    │    └─BatchNorm2d: 3-14            512\n",
       "│    │    └─ReLU: 3-15                   --\n",
       "│    └─Bottleneck: 2-3                   --\n",
       "│    │    └─Conv2d: 3-16                 16,384\n",
       "│    │    └─BatchNorm2d: 3-17            128\n",
       "│    │    └─Conv2d: 3-18                 36,864\n",
       "│    │    └─BatchNorm2d: 3-19            128\n",
       "│    │    └─Conv2d: 3-20                 16,384\n",
       "│    │    └─BatchNorm2d: 3-21            512\n",
       "│    │    └─ReLU: 3-22                   --\n",
       "├─Sequential: 1-6                        --\n",
       "│    └─Bottleneck: 2-4                   --\n",
       "│    │    └─Conv2d: 3-23                 32,768\n",
       "│    │    └─BatchNorm2d: 3-24            256\n",
       "│    │    └─Conv2d: 3-25                 147,456\n",
       "│    │    └─BatchNorm2d: 3-26            256\n",
       "│    │    └─Conv2d: 3-27                 65,536\n",
       "│    │    └─BatchNorm2d: 3-28            1,024\n",
       "│    │    └─ReLU: 3-29                   --\n",
       "│    │    └─Sequential: 3-30             132,096\n",
       "│    └─Bottleneck: 2-5                   --\n",
       "│    │    └─Conv2d: 3-31                 65,536\n",
       "│    │    └─BatchNorm2d: 3-32            256\n",
       "│    │    └─Conv2d: 3-33                 147,456\n",
       "│    │    └─BatchNorm2d: 3-34            256\n",
       "│    │    └─Conv2d: 3-35                 65,536\n",
       "│    │    └─BatchNorm2d: 3-36            1,024\n",
       "│    │    └─ReLU: 3-37                   --\n",
       "│    └─Bottleneck: 2-6                   --\n",
       "│    │    └─Conv2d: 3-38                 65,536\n",
       "│    │    └─BatchNorm2d: 3-39            256\n",
       "│    │    └─Conv2d: 3-40                 147,456\n",
       "│    │    └─BatchNorm2d: 3-41            256\n",
       "│    │    └─Conv2d: 3-42                 65,536\n",
       "│    │    └─BatchNorm2d: 3-43            1,024\n",
       "│    │    └─ReLU: 3-44                   --\n",
       "│    └─Bottleneck: 2-7                   --\n",
       "│    │    └─Conv2d: 3-45                 65,536\n",
       "│    │    └─BatchNorm2d: 3-46            256\n",
       "│    │    └─Conv2d: 3-47                 147,456\n",
       "│    │    └─BatchNorm2d: 3-48            256\n",
       "│    │    └─Conv2d: 3-49                 65,536\n",
       "│    │    └─BatchNorm2d: 3-50            1,024\n",
       "│    │    └─ReLU: 3-51                   --\n",
       "├─Sequential: 1-7                        --\n",
       "│    └─Bottleneck: 2-8                   --\n",
       "│    │    └─Conv2d: 3-52                 131,072\n",
       "│    │    └─BatchNorm2d: 3-53            512\n",
       "│    │    └─Conv2d: 3-54                 589,824\n",
       "│    │    └─BatchNorm2d: 3-55            512\n",
       "│    │    └─Conv2d: 3-56                 262,144\n",
       "│    │    └─BatchNorm2d: 3-57            2,048\n",
       "│    │    └─ReLU: 3-58                   --\n",
       "│    │    └─Sequential: 3-59             526,336\n",
       "│    └─Bottleneck: 2-9                   --\n",
       "│    │    └─Conv2d: 3-60                 262,144\n",
       "│    │    └─BatchNorm2d: 3-61            512\n",
       "│    │    └─Conv2d: 3-62                 589,824\n",
       "│    │    └─BatchNorm2d: 3-63            512\n",
       "│    │    └─Conv2d: 3-64                 262,144\n",
       "│    │    └─BatchNorm2d: 3-65            2,048\n",
       "│    │    └─ReLU: 3-66                   --\n",
       "│    └─Bottleneck: 2-10                  --\n",
       "│    │    └─Conv2d: 3-67                 262,144\n",
       "│    │    └─BatchNorm2d: 3-68            512\n",
       "│    │    └─Conv2d: 3-69                 589,824\n",
       "│    │    └─BatchNorm2d: 3-70            512\n",
       "│    │    └─Conv2d: 3-71                 262,144\n",
       "│    │    └─BatchNorm2d: 3-72            2,048\n",
       "│    │    └─ReLU: 3-73                   --\n",
       "│    └─Bottleneck: 2-11                  --\n",
       "│    │    └─Conv2d: 3-74                 262,144\n",
       "│    │    └─BatchNorm2d: 3-75            512\n",
       "│    │    └─Conv2d: 3-76                 589,824\n",
       "│    │    └─BatchNorm2d: 3-77            512\n",
       "│    │    └─Conv2d: 3-78                 262,144\n",
       "│    │    └─BatchNorm2d: 3-79            2,048\n",
       "│    │    └─ReLU: 3-80                   --\n",
       "│    └─Bottleneck: 2-12                  --\n",
       "│    │    └─Conv2d: 3-81                 262,144\n",
       "│    │    └─BatchNorm2d: 3-82            512\n",
       "│    │    └─Conv2d: 3-83                 589,824\n",
       "│    │    └─BatchNorm2d: 3-84            512\n",
       "│    │    └─Conv2d: 3-85                 262,144\n",
       "│    │    └─BatchNorm2d: 3-86            2,048\n",
       "│    │    └─ReLU: 3-87                   --\n",
       "│    └─Bottleneck: 2-13                  --\n",
       "│    │    └─Conv2d: 3-88                 262,144\n",
       "│    │    └─BatchNorm2d: 3-89            512\n",
       "│    │    └─Conv2d: 3-90                 589,824\n",
       "│    │    └─BatchNorm2d: 3-91            512\n",
       "│    │    └─Conv2d: 3-92                 262,144\n",
       "│    │    └─BatchNorm2d: 3-93            2,048\n",
       "│    │    └─ReLU: 3-94                   --\n",
       "│    └─Bottleneck: 2-14                  --\n",
       "│    │    └─Conv2d: 3-95                 262,144\n",
       "│    │    └─BatchNorm2d: 3-96            512\n",
       "│    │    └─Conv2d: 3-97                 589,824\n",
       "│    │    └─BatchNorm2d: 3-98            512\n",
       "│    │    └─Conv2d: 3-99                 262,144\n",
       "│    │    └─BatchNorm2d: 3-100           2,048\n",
       "│    │    └─ReLU: 3-101                  --\n",
       "│    └─Bottleneck: 2-15                  --\n",
       "│    │    └─Conv2d: 3-102                262,144\n",
       "│    │    └─BatchNorm2d: 3-103           512\n",
       "│    │    └─Conv2d: 3-104                589,824\n",
       "│    │    └─BatchNorm2d: 3-105           512\n",
       "│    │    └─Conv2d: 3-106                262,144\n",
       "│    │    └─BatchNorm2d: 3-107           2,048\n",
       "│    │    └─ReLU: 3-108                  --\n",
       "│    └─Bottleneck: 2-16                  --\n",
       "│    │    └─Conv2d: 3-109                262,144\n",
       "│    │    └─BatchNorm2d: 3-110           512\n",
       "│    │    └─Conv2d: 3-111                589,824\n",
       "│    │    └─BatchNorm2d: 3-112           512\n",
       "│    │    └─Conv2d: 3-113                262,144\n",
       "│    │    └─BatchNorm2d: 3-114           2,048\n",
       "│    │    └─ReLU: 3-115                  --\n",
       "│    └─Bottleneck: 2-17                  --\n",
       "│    │    └─Conv2d: 3-116                262,144\n",
       "│    │    └─BatchNorm2d: 3-117           512\n",
       "│    │    └─Conv2d: 3-118                589,824\n",
       "│    │    └─BatchNorm2d: 3-119           512\n",
       "│    │    └─Conv2d: 3-120                262,144\n",
       "│    │    └─BatchNorm2d: 3-121           2,048\n",
       "│    │    └─ReLU: 3-122                  --\n",
       "│    └─Bottleneck: 2-18                  --\n",
       "│    │    └─Conv2d: 3-123                262,144\n",
       "│    │    └─BatchNorm2d: 3-124           512\n",
       "│    │    └─Conv2d: 3-125                589,824\n",
       "│    │    └─BatchNorm2d: 3-126           512\n",
       "│    │    └─Conv2d: 3-127                262,144\n",
       "│    │    └─BatchNorm2d: 3-128           2,048\n",
       "│    │    └─ReLU: 3-129                  --\n",
       "│    └─Bottleneck: 2-19                  --\n",
       "│    │    └─Conv2d: 3-130                262,144\n",
       "│    │    └─BatchNorm2d: 3-131           512\n",
       "│    │    └─Conv2d: 3-132                589,824\n",
       "│    │    └─BatchNorm2d: 3-133           512\n",
       "│    │    └─Conv2d: 3-134                262,144\n",
       "│    │    └─BatchNorm2d: 3-135           2,048\n",
       "│    │    └─ReLU: 3-136                  --\n",
       "│    └─Bottleneck: 2-20                  --\n",
       "│    │    └─Conv2d: 3-137                262,144\n",
       "│    │    └─BatchNorm2d: 3-138           512\n",
       "│    │    └─Conv2d: 3-139                589,824\n",
       "│    │    └─BatchNorm2d: 3-140           512\n",
       "│    │    └─Conv2d: 3-141                262,144\n",
       "│    │    └─BatchNorm2d: 3-142           2,048\n",
       "│    │    └─ReLU: 3-143                  --\n",
       "│    └─Bottleneck: 2-21                  --\n",
       "│    │    └─Conv2d: 3-144                262,144\n",
       "│    │    └─BatchNorm2d: 3-145           512\n",
       "│    │    └─Conv2d: 3-146                589,824\n",
       "│    │    └─BatchNorm2d: 3-147           512\n",
       "│    │    └─Conv2d: 3-148                262,144\n",
       "│    │    └─BatchNorm2d: 3-149           2,048\n",
       "│    │    └─ReLU: 3-150                  --\n",
       "│    └─Bottleneck: 2-22                  --\n",
       "│    │    └─Conv2d: 3-151                262,144\n",
       "│    │    └─BatchNorm2d: 3-152           512\n",
       "│    │    └─Conv2d: 3-153                589,824\n",
       "│    │    └─BatchNorm2d: 3-154           512\n",
       "│    │    └─Conv2d: 3-155                262,144\n",
       "│    │    └─BatchNorm2d: 3-156           2,048\n",
       "│    │    └─ReLU: 3-157                  --\n",
       "│    └─Bottleneck: 2-23                  --\n",
       "│    │    └─Conv2d: 3-158                262,144\n",
       "│    │    └─BatchNorm2d: 3-159           512\n",
       "│    │    └─Conv2d: 3-160                589,824\n",
       "│    │    └─BatchNorm2d: 3-161           512\n",
       "│    │    └─Conv2d: 3-162                262,144\n",
       "│    │    └─BatchNorm2d: 3-163           2,048\n",
       "│    │    └─ReLU: 3-164                  --\n",
       "│    └─Bottleneck: 2-24                  --\n",
       "│    │    └─Conv2d: 3-165                262,144\n",
       "│    │    └─BatchNorm2d: 3-166           512\n",
       "│    │    └─Conv2d: 3-167                589,824\n",
       "│    │    └─BatchNorm2d: 3-168           512\n",
       "│    │    └─Conv2d: 3-169                262,144\n",
       "│    │    └─BatchNorm2d: 3-170           2,048\n",
       "│    │    └─ReLU: 3-171                  --\n",
       "│    └─Bottleneck: 2-25                  --\n",
       "│    │    └─Conv2d: 3-172                262,144\n",
       "│    │    └─BatchNorm2d: 3-173           512\n",
       "│    │    └─Conv2d: 3-174                589,824\n",
       "│    │    └─BatchNorm2d: 3-175           512\n",
       "│    │    └─Conv2d: 3-176                262,144\n",
       "│    │    └─BatchNorm2d: 3-177           2,048\n",
       "│    │    └─ReLU: 3-178                  --\n",
       "│    └─Bottleneck: 2-26                  --\n",
       "│    │    └─Conv2d: 3-179                262,144\n",
       "│    │    └─BatchNorm2d: 3-180           512\n",
       "│    │    └─Conv2d: 3-181                589,824\n",
       "│    │    └─BatchNorm2d: 3-182           512\n",
       "│    │    └─Conv2d: 3-183                262,144\n",
       "│    │    └─BatchNorm2d: 3-184           2,048\n",
       "│    │    └─ReLU: 3-185                  --\n",
       "│    └─Bottleneck: 2-27                  --\n",
       "│    │    └─Conv2d: 3-186                262,144\n",
       "│    │    └─BatchNorm2d: 3-187           512\n",
       "│    │    └─Conv2d: 3-188                589,824\n",
       "│    │    └─BatchNorm2d: 3-189           512\n",
       "│    │    └─Conv2d: 3-190                262,144\n",
       "│    │    └─BatchNorm2d: 3-191           2,048\n",
       "│    │    └─ReLU: 3-192                  --\n",
       "│    └─Bottleneck: 2-28                  --\n",
       "│    │    └─Conv2d: 3-193                262,144\n",
       "│    │    └─BatchNorm2d: 3-194           512\n",
       "│    │    └─Conv2d: 3-195                589,824\n",
       "│    │    └─BatchNorm2d: 3-196           512\n",
       "│    │    └─Conv2d: 3-197                262,144\n",
       "│    │    └─BatchNorm2d: 3-198           2,048\n",
       "│    │    └─ReLU: 3-199                  --\n",
       "│    └─Bottleneck: 2-29                  --\n",
       "│    │    └─Conv2d: 3-200                262,144\n",
       "│    │    └─BatchNorm2d: 3-201           512\n",
       "│    │    └─Conv2d: 3-202                589,824\n",
       "│    │    └─BatchNorm2d: 3-203           512\n",
       "│    │    └─Conv2d: 3-204                262,144\n",
       "│    │    └─BatchNorm2d: 3-205           2,048\n",
       "│    │    └─ReLU: 3-206                  --\n",
       "│    └─Bottleneck: 2-30                  --\n",
       "│    │    └─Conv2d: 3-207                262,144\n",
       "│    │    └─BatchNorm2d: 3-208           512\n",
       "│    │    └─Conv2d: 3-209                589,824\n",
       "│    │    └─BatchNorm2d: 3-210           512\n",
       "│    │    └─Conv2d: 3-211                262,144\n",
       "│    │    └─BatchNorm2d: 3-212           2,048\n",
       "│    │    └─ReLU: 3-213                  --\n",
       "├─Sequential: 1-8                        --\n",
       "│    └─Bottleneck: 2-31                  --\n",
       "│    │    └─Conv2d: 3-214                524,288\n",
       "│    │    └─BatchNorm2d: 3-215           1,024\n",
       "│    │    └─Conv2d: 3-216                2,359,296\n",
       "│    │    └─BatchNorm2d: 3-217           1,024\n",
       "│    │    └─Conv2d: 3-218                1,048,576\n",
       "│    │    └─BatchNorm2d: 3-219           4,096\n",
       "│    │    └─ReLU: 3-220                  --\n",
       "│    │    └─Sequential: 3-221            2,101,248\n",
       "│    └─Bottleneck: 2-32                  --\n",
       "│    │    └─Conv2d: 3-222                1,048,576\n",
       "│    │    └─BatchNorm2d: 3-223           1,024\n",
       "│    │    └─Conv2d: 3-224                2,359,296\n",
       "│    │    └─BatchNorm2d: 3-225           1,024\n",
       "│    │    └─Conv2d: 3-226                1,048,576\n",
       "│    │    └─BatchNorm2d: 3-227           4,096\n",
       "│    │    └─ReLU: 3-228                  --\n",
       "│    └─Bottleneck: 2-33                  --\n",
       "│    │    └─Conv2d: 3-229                1,048,576\n",
       "│    │    └─BatchNorm2d: 3-230           1,024\n",
       "│    │    └─Conv2d: 3-231                2,359,296\n",
       "│    │    └─BatchNorm2d: 3-232           1,024\n",
       "│    │    └─Conv2d: 3-233                1,048,576\n",
       "│    │    └─BatchNorm2d: 3-234           4,096\n",
       "│    │    └─ReLU: 3-235                  --\n",
       "├─AdaptiveAvgPool2d: 1-9                 --\n",
       "├─Linear: 1-10                           2,049,000\n",
       "=================================================================\n",
       "Total params: 44,549,160\n",
       "Trainable params: 44,549,160\n",
       "Non-trainable params: 0\n",
       "================================================================="
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resnet_model_101 = add_custom_layers(resnet_model_101)\n",
    "summary(resnet_model_101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transform Data using model Data Loaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_train_dataloader, resnet_val_dataloader, resnet_test_dataloader = get_resnet_dataloaders(resnet_weight_101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1\n",
      "Train loss: 2.7710 | Train accuracy: 31.943%\n",
      "Val loss: 0.9804 | Val accuracy: 74.062%\n",
      "| Epoch time: 84.57 seconds\n",
      "Epoch: 2\n",
      "Train loss: 1.1481 | Train accuracy: 66.598%\n",
      "Val loss: 0.6465 | Val accuracy: 80.234%\n",
      "| Epoch time: 94.07 seconds\n",
      "Epoch: 3\n",
      "Train loss: 0.8434 | Train accuracy: 74.818%\n",
      "Val loss: 0.4742 | Val accuracy: 85.820%\n",
      "| Epoch time: 93.41 seconds\n",
      "Epoch: 4\n",
      "Train loss: 0.6943 | Train accuracy: 78.922%\n",
      "Val loss: 0.3933 | Val accuracy: 88.164%\n",
      "| Epoch time: 93.74 seconds\n",
      "Epoch: 5\n",
      "Train loss: 0.6055 | Train accuracy: 81.709%\n",
      "Val loss: 0.3505 | Val accuracy: 90.312%\n",
      "| Epoch time: 94.25 seconds\n",
      "Epoch: 6\n",
      "Train loss: 0.5515 | Train accuracy: 83.257%\n",
      "Val loss: 0.3462 | Val accuracy: 89.727%\n",
      "| Epoch time: 94.04 seconds\n",
      "Epoch: 7\n",
      "Train loss: 0.4921 | Train accuracy: 85.255%\n",
      "Val loss: 0.2975 | Val accuracy: 91.094%\n",
      "| Epoch time: 94.33 seconds\n",
      "Epoch: 8\n",
      "Train loss: 0.4489 | Train accuracy: 86.229%\n",
      "Val loss: 0.3028 | Val accuracy: 90.195%\n",
      "| Epoch time: 93.86 seconds\n",
      "Epoch: 9\n",
      "Train loss: 0.4176 | Train accuracy: 87.328%\n",
      "Val loss: 0.2658 | Val accuracy: 91.680%\n",
      "| Epoch time: 93.63 seconds\n",
      "Epoch: 10\n",
      "Train loss: 0.3920 | Train accuracy: 87.583%\n",
      "Val loss: 0.3089 | Val accuracy: 91.484%\n",
      "| Epoch time: 93.79 seconds\n",
      "Epoch: 11\n",
      "Train loss: 0.3732 | Train accuracy: 88.201%\n",
      "Val loss: 0.2924 | Val accuracy: 91.680%\n",
      "| Epoch time: 94.41 seconds\n",
      "Epoch: 12\n",
      "Train loss: 0.3499 | Train accuracy: 89.031%\n",
      "Val loss: 0.2665 | Val accuracy: 91.992%\n",
      "| Epoch time: 123.96 seconds\n",
      "Epoch: 13\n",
      "Train loss: 0.3297 | Train accuracy: 89.554%\n",
      "Val loss: 0.2735 | Val accuracy: 91.797%\n",
      "| Epoch time: 108.59 seconds\n",
      "Epoch: 14\n",
      "Train loss: 0.3166 | Train accuracy: 90.216%\n",
      "Val loss: 0.2921 | Val accuracy: 91.211%\n",
      "| Epoch time: 137.48 seconds\n",
      "Epoch: 15\n",
      "Train loss: 0.3059 | Train accuracy: 90.324%\n",
      "Val loss: 0.2699 | Val accuracy: 91.289%\n",
      "| Epoch time: 127.04 seconds\n",
      "Epoch: 16\n",
      "Train loss: 0.3054 | Train accuracy: 90.538%\n",
      "Val loss: 0.2677 | Val accuracy: 91.758%\n",
      "| Epoch time: 94.16 seconds\n",
      "Epoch: 17\n",
      "Train loss: 0.2933 | Train accuracy: 91.087%\n",
      "Val loss: 0.3109 | Val accuracy: 91.016%\n",
      "| Epoch time: 161.71 seconds\n",
      "Epoch: 18\n",
      "Train loss: 0.2661 | Train accuracy: 91.404%\n",
      "Val loss: 0.2728 | Val accuracy: 91.406%\n",
      "| Epoch time: 162.42 seconds\n",
      "Epoch: 19\n",
      "Train loss: 0.2637 | Train accuracy: 91.928%\n",
      "Val loss: 0.2871 | Val accuracy: 90.898%\n",
      "| Epoch time: 150.41 seconds\n",
      "Epoch: 20\n",
      "Train loss: 0.2603 | Train accuracy: 91.588%\n",
      "Val loss: 0.2698 | Val accuracy: 92.656%\n",
      "| Epoch time: 134.33 seconds\n",
      "Epoch: 21\n",
      "Train loss: 0.2465 | Train accuracy: 92.303%\n",
      "Val loss: 0.2439 | Val accuracy: 92.773%\n",
      "| Epoch time: 145.59 seconds\n",
      "Epoch: 22\n",
      "Train loss: 0.2479 | Train accuracy: 92.216%\n",
      "Val loss: 0.2666 | Val accuracy: 92.383%\n",
      "| Epoch time: 96.95 seconds\n",
      "Epoch: 23\n",
      "Train loss: 0.2488 | Train accuracy: 92.223%\n",
      "Val loss: 0.2478 | Val accuracy: 92.578%\n",
      "| Epoch time: 96.29 seconds\n",
      "Epoch: 24\n",
      "Train loss: 0.2456 | Train accuracy: 92.487%\n",
      "Val loss: 0.2492 | Val accuracy: 92.188%\n",
      "| Epoch time: 96.08 seconds\n",
      "Epoch: 25\n",
      "Train loss: 0.2249 | Train accuracy: 93.027%\n",
      "Val loss: 0.2376 | Val accuracy: 92.266%\n",
      "| Epoch time: 97.52 seconds\n",
      "Epoch: 26\n",
      "Train loss: 0.2324 | Train accuracy: 92.924%\n",
      "Val loss: 0.2640 | Val accuracy: 92.461%\n",
      "| Epoch time: 143.50 seconds\n",
      "Epoch: 27\n",
      "Train loss: 0.2318 | Train accuracy: 92.981%\n",
      "Val loss: 0.2668 | Val accuracy: 92.734%\n",
      "| Epoch time: 157.36 seconds\n",
      "Epoch: 28\n",
      "Train loss: 0.2214 | Train accuracy: 93.092%\n",
      "Val loss: 0.3248 | Val accuracy: 91.484%\n",
      "| Epoch time: 160.34 seconds\n",
      "Epoch: 29\n",
      "Train loss: 0.2061 | Train accuracy: 93.363%\n",
      "Val loss: 0.2360 | Val accuracy: 92.773%\n",
      "| Epoch time: 160.63 seconds\n",
      "Epoch: 30\n",
      "Train loss: 0.2066 | Train accuracy: 93.751%\n",
      "Val loss: 0.2863 | Val accuracy: 93.555%\n",
      "| Epoch time: 160.21 seconds\n"
     ]
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          2.7709886589886454,
          1.148050290713378,
          0.8434178333429364,
          0.6943373067119111,
          0.6055253223644048,
          0.5515132427074333,
          0.4920664708619999,
          0.44886644219900196,
          0.41762272920900895,
          0.39201705424132716,
          0.37318505292926074,
          0.34986005919451396,
          0.3297327327414005,
          0.3165907151423289,
          0.305870268723411,
          0.30536417326761933,
          0.2933317249027297,
          0.26605554817959454,
          0.2636918961798763,
          0.2603412508540809,
          0.24653214162824702,
          0.24787160232927985,
          0.24877814431476156,
          0.24558637233812944,
          0.22494212139653863,
          0.23237725562676434,
          0.23182212082907486,
          0.22138906293879682,
          0.20608754205667543,
          0.20657173677908158
         ]
        },
        {
         "mode": "lines",
         "name": "Val loss",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.9803752973675728,
          0.6465298850089312,
          0.47420581616461277,
          0.3933233702555299,
          0.350454268977046,
          0.3462189817801118,
          0.29746143938973546,
          0.3027863819152117,
          0.26582270208746195,
          0.3089098962955177,
          0.2924361969344318,
          0.26648234063759446,
          0.27348831854760647,
          0.2920840095030144,
          0.26992253260686994,
          0.26766924327239394,
          0.3109332653693855,
          0.2727603712119162,
          0.28714548680000007,
          0.26976233115419745,
          0.24386616423726082,
          0.2665919157443568,
          0.24784119380638003,
          0.24923405866138637,
          0.23763226019218564,
          0.26404125010594726,
          0.26678439090028405,
          0.3247700307983905,
          0.23597871023230255,
          0.2863301149336621
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Loss over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Loss"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "lines",
         "name": "Train Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.3194312796208531,
          0.6659804502369668,
          0.748178317535545,
          0.789218009478673,
          0.8170912322274881,
          0.8325681279620853,
          0.8525473933649289,
          0.862292654028436,
          0.8732819905213269,
          0.875829383886256,
          0.8820053317535546,
          0.890313981042654,
          0.8955420616113744,
          0.9021623222748815,
          0.9032434834123223,
          0.9053761848341233,
          0.9108708530805687,
          0.9140402843601896,
          0.9192831753554502,
          0.9158767772511849,
          0.9230302132701422,
          0.9221563981042653,
          0.9222304502369668,
          0.9248667061611374,
          0.9302725118483413,
          0.9292357819905213,
          0.9298133886255925,
          0.9309241706161138,
          0.9336344786729858,
          0.9375148104265404
         ]
        },
        {
         "mode": "lines",
         "name": "Val Accuracy",
         "type": "scatter",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30
         ],
         "y": [
          0.740625,
          0.80234375,
          0.858203125,
          0.881640625,
          0.903125,
          0.897265625,
          0.9109375,
          0.901953125,
          0.916796875,
          0.91484375,
          0.916796875,
          0.919921875,
          0.91796875,
          0.912109375,
          0.912890625,
          0.917578125,
          0.91015625,
          0.9140625,
          0.908984375,
          0.9265625,
          0.927734375,
          0.923828125,
          0.92578125,
          0.921875,
          0.92265625,
          0.924609375,
          0.92734375,
          0.91484375,
          0.927734375,
          0.935546875
         ]
        }
       ],
       "layout": {
        "height": 400,
        "legend": {
         "x": 0.05,
         "y": 1.1
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Accuracy over Epochs"
        },
        "width": 800,
        "xaxis": {
         "title": {
          "text": "Epochs"
         }
        },
        "yaxis": {
         "title": {
          "text": "Accuracy"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Actual training ResNet model\n",
    "resnet_results,training_time = training_loop(model = resnet_model_101,\n",
    "                               train_dataloader = resnet_train_dataloader,\n",
    "                               val_dataloader = resnet_val_dataloader,\n",
    "                               device = device,\n",
    "                               epochs = 30,\n",
    "                               patience = 5\n",
    "                              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Model Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_results.to_csv('models_results/resnet_model_101.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calcluate Model Testing Accuracy and Training Time Measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing Accuracy is 94.0%\n"
     ]
    }
   ],
   "source": [
    "resnet_model_101 = torch.load('trained_models/resnet_model_152.pth')\n",
    "test_accuracy = test_accuracy_resnet(resnet_model_101,resnet_test_dataloader,device)\n",
    "print(f\"Testing Accuracy is {test_accuracy}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Training Time 58.9779 Minitues\n"
     ]
    }
   ],
   "source": [
    "print(f'Model Training Time {round(training_time/60,4)} Minitues')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Test Accuracy and Training Time in csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "row = {'Model':'ResNet101','Testing Accuracy':test_accuracy,'Training_Time(Minutes)':round(training_time/60,4)}\n",
    "\n",
    "# Use the loc method to add the new row to the DataFrame\n",
    "training_times.loc[len(training_times)] = row\n",
    "training_times.to_csv('models_results/training_time.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Testing Accuracy</th>\n",
       "      <th>Training_Time(Minutes)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ResNet50</td>\n",
       "      <td>95.6</td>\n",
       "      <td>37.3207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ResNet152</td>\n",
       "      <td>95.2</td>\n",
       "      <td>65.0694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ResNet101</td>\n",
       "      <td>94.0</td>\n",
       "      <td>58.9779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Model  Testing Accuracy  Training_Time(Minutes)\n",
       "0   ResNet50              95.6                 37.3207\n",
       "1  ResNet152              95.2                 65.0694\n",
       "2  ResNet101              94.0                 58.9779"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_times = pd.read_csv('models_results/training_time.csv')\n",
    "training_times.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Trained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(resnet_model_101, 'trained_models/resnet_model_101.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Performance Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('models_results/training_time.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Testing Accuracy</th>\n",
       "      <th>Training_Time(Minutes)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ResNet50</td>\n",
       "      <td>95.6</td>\n",
       "      <td>37.3207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ResNet152</td>\n",
       "      <td>95.2</td>\n",
       "      <td>65.0694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ResNet101</td>\n",
       "      <td>94.0</td>\n",
       "      <td>58.9779</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Model  Testing Accuracy  Training_Time(Minutes)\n",
       "0   ResNet50              95.6                 37.3207\n",
       "1  ResNet152              95.2                 65.0694\n",
       "2  ResNet101              94.0                 58.9779"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "marker": {
          "color": "royalblue"
         },
         "text": [
          95.6,
          95.2,
          94
         ],
         "textposition": "auto",
         "type": "bar",
         "width": 0.5,
         "x": [
          "ResNet50",
          "ResNet152",
          "ResNet101"
         ],
         "y": [
          95.6,
          95.2,
          94
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Testing Accuracy of Different Models"
        },
        "xaxis": {
         "title": {
          "text": "Model"
         }
        },
        "yaxis": {
         "title": {
          "text": "Testing Accuracy in %"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Bar plot\n",
    "fig_bar = go.Figure(go.Bar(x=df.Model, y=df['Testing Accuracy'], text=df['Testing Accuracy'],\n",
    "                           textposition='auto', marker=dict(color='royalblue'), width = 0.5))\n",
    "fig_bar.update_layout(title='Testing Accuracy of Different Models',\n",
    "                      xaxis_title='Model', yaxis_title='Testing Accuracy in %')\n",
    "fig_bar.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "marker": {
          "color": "green",
          "size": 12
         },
         "mode": "lines+markers+text",
         "text": [
          "ResNet50",
          "ResNet152",
          "ResNet101"
         ],
         "type": "scatter",
         "x": [
          "ResNet50",
          "ResNet152",
          "ResNet101"
         ],
         "y": [
          37.3207,
          65.0694,
          58.9779
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Testing Accuracy vs Training Time for Different Models"
        },
        "xaxis": {
         "title": {
          "text": "Training Time (Minutes)"
         }
        },
        "yaxis": {
         "title": {
          "text": "Testing Accuracy"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Scatter plot\n",
    "fig_scatter = go.Figure(go.Scatter(y=df['Training_Time(Minutes)'], x=df.Model,\n",
    "                                  mode='lines+markers+text', text=df.Model,\n",
    "                                  marker=dict(size=12, color='green')))\n",
    "fig_scatter.update_layout(title='Testing Accuracy vs Training Time for Different Models',\n",
    "                          xaxis_title='Training Time (Minutes)', yaxis_title='Testing Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overall observations of all trained models\n",
    "\n",
    "1. ResNet50 achieved the highest accuracy (95.6%), outperforming ResNet152 (95.2%) and ResNet101 (94%).\n",
    "\n",
    "2. Model complexity didn't linearly correlate with accuracy, as ResNet152 had more parameters but only slightly higher accuracy than ResNet50.\n",
    "\n",
    "3. ResNet50 was computationally efficient, training in the least time (37.32 minutes), while ResNet152 took the longest (65.07 minutes).\n",
    "\n",
    "4. ResNet50 strikes a balance between training time and accuracy, making it a practical choice.\n",
    "\n",
    "5. Actual generalization performance should be validated on unseen data to confirm these trends.\n",
    "\n",
    "6. Dataset characteristics can influence results, so these observations are dataset-specific."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## YOLO For Image Classification\n",
    "\n",
    "YOLO, or You Only Look Once, is an object detection algorithm known for its real-time and efficient performance. It revolutionized object detection by simultaneously predicting object bounding boxes and class probabilities in a single pass through the neural network. Key points about YOLO include:\n",
    "\n",
    "- *Real-time Detection:* YOLO processes images in real-time, making it suitable for applications that require rapid object detection.\n",
    "\n",
    "- *Single Pass Prediction:* Unlike traditional methods, YOLO predicts objects directly from the entire image in a single forward pass.\n",
    "\n",
    "- *Bounding Box Accuracy:* YOLO's bounding box predictions can be less precise for small objects compared to other methods, but its overall speed and efficiency make it a popular choice for various computer vision tasks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install ultralytics\n",
    "from ultralytics import YOLO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-cls.pt to 'yolov8s-cls.pt'...\n",
      "100%|██████████| 12.2M/12.2M [00:00<00:00, 14.6MB/s]\n"
     ]
    }
   ],
   "source": [
    "yolo_model = YOLO(\"yolov8s-cls.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "New https://pypi.org/project/ultralytics/8.0.148 available  Update with 'pip install -U ultralytics'\n",
      "Ultralytics YOLOv8.0.145  Python-3.9.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3060 Laptop GPU, 6144MiB)\n",
      "\u001b[34m\u001b[1mengine\\trainer: \u001b[0mtask=classify, mode=train, model=yolov8s-cls.pt, data=sports-classification, epochs=30, patience=50, batch=16, imgsz=256, save=True, save_period=-1, cache=False, device=None, workers=8, project=None, name=None, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, vid_stride=1, line_width=None, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, boxes=True, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, tracker=botsort.yaml, save_dir=runs\\classify\\train\n",
      "Overriding model.yaml nc=1000 with nc=100\n",
      "\n",
      "                   from  n    params  module                                       arguments                     \n",
      "  0                  -1  1       928  ultralytics.nn.modules.conv.Conv             [3, 32, 3, 2]                 \n",
      "  1                  -1  1     18560  ultralytics.nn.modules.conv.Conv             [32, 64, 3, 2]                \n",
      "  2                  -1  1     29056  ultralytics.nn.modules.block.C2f             [64, 64, 1, True]             \n",
      "  3                  -1  1     73984  ultralytics.nn.modules.conv.Conv             [64, 128, 3, 2]               \n",
      "  4                  -1  2    197632  ultralytics.nn.modules.block.C2f             [128, 128, 2, True]           \n",
      "  5                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]              \n",
      "  6                  -1  2    788480  ultralytics.nn.modules.block.C2f             [256, 256, 2, True]           \n",
      "  7                  -1  1   1180672  ultralytics.nn.modules.conv.Conv             [256, 512, 3, 2]              \n",
      "  8                  -1  1   1838080  ultralytics.nn.modules.block.C2f             [512, 512, 1, True]           \n",
      "  9                  -1  1    786020  ultralytics.nn.modules.head.Classify         [512, 100]                    \n",
      "YOLOv8s-cls summary: 99 layers, 5208836 parameters, 5208836 gradients\n",
      "Transferred 156/158 items from pretrained weights\n",
      "\u001b[34m\u001b[1mTensorBoard: \u001b[0mStart with 'tensorboard --logdir runs\\classify\\train', view at http://localhost:6006/\n",
      "\u001b[34m\u001b[1mAMP: \u001b[0mrunning Automatic Mixed Precision (AMP) checks with YOLOv8n...\n",
      "Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8n.pt to 'yolov8n.pt'...\n",
      "100%|██████████| 6.23M/6.23M [00:00<00:00, 16.3MB/s]\n",
      "\u001b[34m\u001b[1mAMP: \u001b[0mchecks passed \n",
      "\u001b[34m\u001b[1malbumentations: \u001b[0mRandomResizedCrop(p=1.0, height=256, width=256, scale=(0.5, 1.0), ratio=(0.75, 1.3333333333333333), interpolation=1), HorizontalFlip(p=0.5), ColorJitter(p=0.5, brightness=[0.6, 1.4], contrast=[0.6, 1.4], saturation=[0.30000000000000004, 1.7], hue=[-0.015, 0.015]), Normalize(p=1.0, mean=(0.0, 0.0, 0.0), std=(1.0, 1.0, 1.0), max_pixel_value=255.0), ToTensorV2(always_apply=True, p=1.0, transpose_mask=False)\n",
      "\u001b[34m\u001b[1moptimizer:\u001b[0m AdamW(lr=0.000714, momentum=0.9) with parameter groups 26 weight(decay=0.0), 27 weight(decay=0.0005), 27 bias(decay=0.0)\n",
      "Image sizes 256 train, 256 val\n",
      "Using 8 dataloader workers\n",
      "Logging results to \u001b[1mruns\\classify\\train\u001b[0m\n",
      "Starting training for 30 epochs...\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       1/30     0.761G     0.6908          4        256: 100%|██████████| 844/844 [00:34<00:00, 24.29it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 42.73it/s]\n",
      "                   all      0.872      0.988\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       2/30     0.705G     0.2082          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.25it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.05it/s]\n",
      "                   all      0.884      0.992\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       3/30     0.734G     0.1633          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.12it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 68.13it/s]\n",
      "                   all      0.896      0.996\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       4/30     0.734G     0.1298          4        256: 100%|██████████| 844/844 [00:24<00:00, 35.13it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.13it/s]\n",
      "                   all      0.934          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       5/30     0.734G    0.09684          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.84it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.76it/s]\n",
      "                   all      0.954      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       6/30     0.734G    0.07843          4        256: 100%|██████████| 844/844 [00:24<00:00, 35.05it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.67it/s]\n",
      "                   all      0.952          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       7/30     0.734G    0.05872          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.64it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.47it/s]\n",
      "                   all       0.94      0.996\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       8/30     0.734G    0.04585          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.26it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.66it/s]\n",
      "                   all       0.96          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "       9/30     0.734G    0.03842          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.33it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.47it/s]\n",
      "                   all      0.966          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      10/30     0.734G    0.03534          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.44it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.48it/s]\n",
      "                   all       0.97      0.996\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      11/30     0.734G    0.03157          4        256: 100%|██████████| 844/844 [00:25<00:00, 33.68it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 65.29it/s]\n",
      "                   all      0.962          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      12/30     0.734G    0.02474          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.65it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.32it/s]\n",
      "                   all      0.968      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      13/30     0.734G    0.02579          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.84it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.62it/s]\n",
      "                   all      0.974          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      14/30     0.734G    0.02262          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.62it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.63it/s]\n",
      "                   all       0.97          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      15/30     0.732G    0.01983          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.22it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 68.70it/s]\n",
      "                   all      0.968          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      16/30     0.732G    0.01601          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.64it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.61it/s]\n",
      "                   all       0.97      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      17/30     0.734G    0.01309          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.50it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.44it/s]\n",
      "                   all      0.984          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      18/30     0.734G    0.01368          4        256: 100%|██████████| 844/844 [00:23<00:00, 36.00it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.90it/s]\n",
      "                   all       0.97          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      19/30     0.734G    0.01245          4        256: 100%|██████████| 844/844 [00:23<00:00, 36.35it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.64it/s]\n",
      "                   all       0.97      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      20/30     0.734G   0.009883          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.52it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 68.00it/s]\n",
      "                   all      0.982      0.998\n",
      "Closing dataloader mosaic\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      21/30     0.734G   0.008549          4        256: 100%|██████████| 844/844 [00:22<00:00, 37.30it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 71.55it/s]\n",
      "                   all      0.972          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      22/30     0.732G   0.009427          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.27it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 65.71it/s]\n",
      "                   all      0.976      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      23/30     0.734G   0.007002          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.89it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.31it/s]\n",
      "                   all      0.978      0.998\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      24/30     0.734G   0.005605          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.86it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.19it/s]\n",
      "                   all      0.978          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      25/30     0.734G   0.006086          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.32it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.81it/s]\n",
      "                   all       0.98          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      26/30     0.734G   0.004629          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.21it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.36it/s]\n",
      "                   all       0.98          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      27/30     0.734G   0.004798          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.63it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 67.08it/s]\n",
      "                   all      0.982          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      28/30     0.734G   0.003365          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.21it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 66.53it/s]\n",
      "                   all      0.982          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      29/30     0.734G   0.003309          4        256: 100%|██████████| 844/844 [00:24<00:00, 34.71it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 64.98it/s]\n",
      "                   all       0.98          1\n",
      "\n",
      "      Epoch    GPU_mem       loss  Instances       Size\n",
      "      30/30     0.734G   0.003061          4        256: 100%|██████████| 844/844 [00:23<00:00, 35.20it/s]\n",
      "               classes   top1_acc   top5_acc: 100%|██████████| 16/16 [00:00<00:00, 46.90it/s]\n",
      "                   all       0.98          1\n",
      "\n",
      "30 epochs completed in 0.218 hours.\n",
      "Optimizer stripped from runs\\classify\\train\\weights\\last.pt, 10.5MB\n",
      "Optimizer stripped from runs\\classify\\train\\weights\\best.pt, 10.5MB\n",
      "Results saved to \u001b[1mruns\\classify\\train\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "yolo_results = yolo_model.train(data = 'sports-classification', \n",
    "                                epochs = 30,\n",
    "                                imgsz = 256 \n",
    "                                # similar with ResNet model -> but 256 (YOLO need multiple 32)\n",
    "                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>train/loss</th>\n",
       "      <th>metrics/accuracy_top1</th>\n",
       "      <th>metrics/accuracy_top5</th>\n",
       "      <th>val/loss</th>\n",
       "      <th>lr/pg0</th>\n",
       "      <th>lr/pg1</th>\n",
       "      <th>lr/pg2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>25</td>\n",
       "      <td>0.00463</td>\n",
       "      <td>0.980</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7812</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>0.000149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>26</td>\n",
       "      <td>0.00480</td>\n",
       "      <td>0.982</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7817</td>\n",
       "      <td>0.000125</td>\n",
       "      <td>0.000125</td>\n",
       "      <td>0.000125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>27</td>\n",
       "      <td>0.00337</td>\n",
       "      <td>0.982</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7812</td>\n",
       "      <td>0.000101</td>\n",
       "      <td>0.000101</td>\n",
       "      <td>0.000101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>28</td>\n",
       "      <td>0.00331</td>\n",
       "      <td>0.980</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7817</td>\n",
       "      <td>0.000078</td>\n",
       "      <td>0.000078</td>\n",
       "      <td>0.000078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>29</td>\n",
       "      <td>0.00306</td>\n",
       "      <td>0.980</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7808</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>0.000054</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      epoch               train/loss    metrics/accuracy_top1  \\\n",
       "25                       25                  0.00463                    0.980   \n",
       "26                       26                  0.00480                    0.982   \n",
       "27                       27                  0.00337                    0.982   \n",
       "28                       28                  0.00331                    0.980   \n",
       "29                       29                  0.00306                    0.980   \n",
       "\n",
       "      metrics/accuracy_top5                 val/loss                   lr/pg0  \\\n",
       "25                      1.0                   1.7812                 0.000149   \n",
       "26                      1.0                   1.7817                 0.000125   \n",
       "27                      1.0                   1.7812                 0.000101   \n",
       "28                      1.0                   1.7817                 0.000078   \n",
       "29                      1.0                   1.7808                 0.000054   \n",
       "\n",
       "                     lr/pg1                   lr/pg2  \n",
       "25                 0.000149                 0.000149  \n",
       "26                 0.000125                 0.000125  \n",
       "27                 0.000101                 0.000101  \n",
       "28                 0.000078                 0.000078  \n",
       "29                 0.000054                 0.000054  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yolo_result_csv = \"runs/classify/train/results.csv\"\n",
    "\n",
    "pd.read_csv(yolo_result_csv).tail(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Yolo Model Results Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='runs/classify/train/results.png' height=500 width = 500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"image-container\">\n",
    "    <img src=\"runs/classify/train/val_batch2_labels.jpg\" alt=\"Image 1\">\n",
    "    <img src=\"runs/classify/train/val_batch2_pred.jpg\" alt=\"Image 2\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}