IA_DNN_ImageObjectDetection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "IA_DNN_ImageObjectDetection.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MedicalImageAnalysisTutorials/DeepLearning4All/blob/main/IA_DNN_ImageObjectDetection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "figzmLKBxWC8"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "\n",
        "**The Object Detection Problem:** involves classification and localization. It takes an image as input and produces one or more bounding boxes with the class label attached to each bounding box. \\\\\n",
        "\n",
        "**The Object Recognition Problem:** is to identity the objects in images or videos. \\\\\n",
        "\n",
        "**Draft version**\n",
        "In this notebook, I will try to provide a practical tutorial for deep learning using simple examples. I will try to use simple implementation and avoid using built-in functions to give clear idea about the concept. You need basic programming knowledge. \n",
        "I made the code flexible so one can try different approaches, datsaets, optimisers, loss functions based on if else statements. \n",
        "\n",
        "**The general code structure:**\n",
        "\n",
        "    1. Read and pre-process the data\n",
        "    2. Create the model, optimiser, loss function, and metrics\n",
        "    3. Start the training loop\n",
        "    4. Evaluate the final model\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LTipttMz22A0"
      },
      "source": [
        "**The Object Detection Problem:** \\\\\n",
        "When we’re shown an image, our brain instantly recognizes the objects contained in it. On the other hand, it takes a lot of time and training data for a machine to identify these objects. But with the recent advances in hardware and deep learning, this computer vision field has become a whole lot easier and more intuitive.\n",
        "First try to understand what is object detection problem: \\\\\n",
        "\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/I1_2009_09_08_drive_0012_001351-768x223.png)\\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/06/I1_2009_09_08_drive_0012_001351-copy-768x223.png) \\\\\n",
        "Our objective behind doing object detection is two folds:\n",
        "\n",
        "1.   To identify what all objects are present in the image and where they’re located\n",
        "2.   Filter out the object of attention\n",
        "\\\\\n",
        "**Then we can introduce what is object detection using deep learning. We will start from RCNN series net work to Yolo** \\\\\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7GQNa5MnGgM6"
      },
      "source": [
        "### RCNN\n",
        "the RCNN algorithm proposes a bunch of boxes in the image and checks if any of these boxes contain any object. **RCNN uses selective search to extract these boxes from an image (these boxes are called regions).**\\\\\n",
        "\n",
        "\n",
        "*   First, an image is taken as an input: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-14-59-02.png)\n",
        "*   Then, we get the Regions of Interest (ROI) using some proposal method (for example, selective search as seen above): \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-00-09.png)\n",
        "*   All these regions are then reshaped as per the input of the CNN, and each region is passed to the ConvNet: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-01-56.png)\n",
        "*   CNN then extracts features for each region and SVMs are used to divide these regions into different classes: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-03-02.png)\n",
        "*   Finally, a bounding box regression (Bbox reg) is used to predict the bounding boxes for each identified region: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-06-33.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GI3K0cF0JSLc"
      },
      "source": [
        "##  Fast RCNN \n",
        "Instead of running a CNN 2,000 times per image, we can run it just once per image and get all the regions of interest (regions containing some object).\n",
        "\n",
        "Ross Girshick, the author of RCNN, came up with this idea of running the CNN just once per image and then finding a way to share that computation across the 2,000 regions. In Fast RCNN, we feed the input image to the CNN, which in turn generates the convolutional feature maps. Using these maps, the regions of proposals are extracted. We then use a RoI pooling layer to reshape all the proposed regions into a fixed size, so that it can be fed into a fully connected network. \\\\\n",
        "\n",
        "The differences with RCNN network are as follows: \\\\\n",
        "*   The input image is passed to a ConvNet which returns the region of interests accordingly. Then we apply the RoI pooling layer on the extracted regions of interest to make sure all the regions are of the same size: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-45-26.png)\n",
        "*   Finally, these regions are passed on to a fully connected network which classifies them, as well as returns the bounding boxes using softmax and linear regression layers simultaneously: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-08-15-47-18.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HjO3YhWeMHpv"
      },
      "source": [
        "##  Faster RCNN\n",
        "Faster RCNN is the modified version of Fast RCNN. The major difference between them is that Fast RCNN uses selective search for generating Regions of Interest, while Faster RCNN uses “Region Proposal Network”, aka RPN. RPN takes image feature maps as an input and generates a set of object proposals, each with an objectness score as output. The main steps are as follows: \\\\ \n",
        "\n",
        "Finally, the proposals are passed to a fully connected layer which has a softmax layer and a linear regression layer at its top, to classify and output the bounding boxes for objects.\n",
        "\n",
        "1.   We take an image as input and pass it to the ConvNet which returns the feature map for that image.\n",
        "2.   Region proposal network is applied on these feature maps. This returns the object proposals along with their objectness score.\n",
        "3.   A RoI pooling layer is applied on these proposals to bring down all the proposals to the same size.\n",
        "4.   Finally, the proposals are passed to a fully connected layer which has a softmax layer and a linear regression layer at its top, to classify and output the bounding boxes for objects.\n",
        "\n",
        "\n",
        "*   The input image is passed to a ConvNet which returns the region of interests accordingly. Then we apply the RoI pooling layer on the extracted regions of interest to make sure all the regions are of the same size: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/Screenshot-from-2018-10-09-14-15-36.png)\n",
        "*   Let me briefly explain how this Region Proposal Network (RPN) actually works. To begin with, Faster RCNN takes the feature maps from CNN and passes them on to the Region Proposal Network. RPN uses a sliding window over these feature maps, and at each window, it generates k Anchor boxes of different shapes and sizes:: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/10/0n6pZEyvW47nlcdQz.png)\n",
        "\n",
        "Anchor boxes are fixed sized boundary boxes that are placed throughout the image and have different shapes and sizes. For each anchor, RPN predicts two things:\n",
        "\n",
        "*   The first is the probability that an anchor is an object (it does not consider which class the object belongs to)\n",
        "*   Second is the bounding box regressor for adjusting the anchors to better fit the object"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31PpH2ziQPvX"
      },
      "source": [
        "##  YOLO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gSMEPFtYQcCT"
      },
      "source": [
        "Then we are going to introduce another famous deep learning object detection model: Yolo\n",
        "\n",
        "The YOLO framework (You Only Look Once) deals with object detection in a different way. It takes the entire image in a single instance and predicts the bounding box coordinates and class probabilities for these boxes. The biggest advantage of using YOLO is its superb speed – it’s incredibly fast and can process 45 frames per second. YOLO also understands generalized object representation.\n",
        "\n",
        "This is one of the best algorithms for object detection and has shown a comparatively similar performance to the R-CNN algorithms. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P2XfzTP-Fgj5"
      },
      "source": [
        "**Yolo network structure**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYOcRV7i5QDi"
      },
      "source": [
        "Our model will be trained as follows:\n",
        "\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-18-46-10.png)\n",
        "\n",
        "*   The framework first take the raw image with annotation. Then seperate images as grid(3X3 in our sample): \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-17-46-32.png) \\\\\n",
        "*   Image classification and localization are applied on each grid. YOLO then predicts the bounding boxes and their corresponding class probabilities for objects (if any are found, of course). \n",
        "\n",
        "The labelled data can be encoded as follows: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-18-01-24.png) \\\\\n",
        "Here,\n",
        "*   pc defines whether an object is present in the grid or not (it is the probability)\n",
        "*   bx, by, bh, bw specify the bounding box if there is an object\n",
        "*   c1, c2, c3 represent the classes, obviously, it can have more classes in different datasets. So, if the object is a car, c2 will be 1 and c1 & c3 will be 0, and so on. \\\\\n",
        "\n",
        "Let's take an example, the grid in which we have a car (c2 = 1): \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-19-35-31.png) \\\\\n",
        "\n",
        "The bounding box can be encoded as follows: \\\\\n",
        "![image1](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-18-27-25.png) \\\\\n",
        "\n",
        "pc will be equal to 1. \\\\\n",
        "bx, by, bh, and bw will be calculated relative to this grid only. bx and by are the center point of the object and bh and bw are the ratio of width and height of the bounding box. \\\\\n",
        "In this case, it will be (around) bx = 0.4, by = 0.3, bh = 0.9, bw = 0.5: \\\\\n",
        "\n",
        "![image17](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-15-19-39-51.png) \\\\\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FqzNLvqwCChS"
      },
      "source": [
        "**Intersection over Union and Non-Max Suppression**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ts_ZQKrXCIoh"
      },
      "source": [
        "This is where Intersection over Union comes into the picture. It calculates the intersection over union of the actual bounding box and the predicted bonding box. \\\\\n",
        "\n",
        "![image18](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-16-13-12-02.png) \\\\\n",
        "\n",
        "IoU = Area of the intersection / Area of the union, i.e.\n",
        "\n",
        "IoU = Area of yellow box / Area of green box \\\\\n",
        "\n",
        "One of the most common problems with object detection algorithms is that is that an object might be detected multiple times. To improve the output of YOLO significantly, Non-Max Suppression is used. \\\\\n",
        "\n",
        "![image19](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-16-13-32-40.png) \\\\\n",
        "\n",
        "1. Discard all the boxes having probabilities less than or equal to a pre-defined threshold (for example, 0.5)\n",
        "2. For the remaining boxes:\n",
        "Pick the box with the highest probability and take that as the output \n",
        "prediction. Discard any other box which has IoU greater than the threshold with the output box from the above\n",
        "3. Repeat step 2 until all the boxes are either taken as the output prediction or discarded\n",
        "\n",
        "![image20](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-17-12-21-31.png) \\\\\n",
        "\n",
        "The output are the boxes with maximum probability and suppressing the close-by boxes with non-max probabilities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xP4gE04MF9tv"
      },
      "source": [
        "**Anchor Boxes**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mn6yEBUQGJEw"
      },
      "source": [
        "Till now, each grid can only identify one object. Considering there may be multiple objects in a single grid, anchor boxes are used. \\\\\n",
        "\n",
        "Each object is assigned to the corresponding grid based on the midpoint of the object and its location. In the example below, we divide the image into 3x3 grid and 2 anchor boxes in each grid.\n",
        "\n",
        "![image21](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-17-13-20-41.png) \\\\\n",
        "\n",
        "So the y label for YOLO with 2 anchor boxes will be: \\\\\n",
        "\n",
        "![image22](https://cdn.analyticsvidhya.com/wp-content/uploads/2018/12/Screenshot-from-2018-11-17-13-33-31.png) \\\\\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EI4Jnc91fWdC"
      },
      "source": [
        "# Notebook setting"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nyuCgS3MOAWM"
      },
      "source": [
        "#TODOS:\n",
        "#  correct rcnn, yolo,...  diagrams\n",
        "#  try using pytorch code directly (using same dataset fron the tutorial)\n",
        "#  organise the code\n",
        "#  use on spine 2d data            img       seg            loc       loc    \n",
        "#         download 3d dataset: img.nii.gz img_seg.nii.gz img.fcsv img.json\n",
        "#         1. resize all images  (resize image and segmentation)\n",
        "#         2. extract 2d images (3 views) (from image and segmentation) \n",
        "#         3. compute the center (based on segmentation)\n",
        "#            image vtLbael1 center x,y       \n",
        "#         4. cretae yolo labels (your task)\n",
        "#         5. augmentation\n",
        "#    \n",
        "#  use on spine 3d data \n",
        "\n",
        "usePytorch = 1\n",
        "datasetID  = 4  # 1: minst is selected by default, for cifar10 use 2, for facial use 3, for VOC use 4\n",
        "NNID = 5  # 1: NN is by default, for DNN use 2,or 3, for 3D use 4, for Yolo use 5  \n",
        "number_of_classes = 20\n",
        "\n",
        "batch_size = 2 # you can try larger batch size e.g. 1024 * 6\n",
        "# if you have large GPU memory you can combine the images to batches \n",
        "# for faster training.\n",
        "# It is good to try different values\n",
        "\n",
        "lossFunctionID = 4\n",
        "# 1: SparseCategoricalCrossentropy is by default, for MeanSquaredError use 2\n",
        "# for CategoricalCrossentropy use 3, for yolo_loss use 4\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NooQ5tm85ZDb",
        "outputId": "c807300f-a8a1-4d82-8709-3c1c845c1c31"
      },
      "source": [
        "# TODO:\n",
        "# complete the introduction\n",
        "# add the introduction with the figures to this notebook https://github.com/idhamari/Deep-Learning-Coursera/blob/master/Convolutional%20Neural%20Networks/Week3/Car%20detection%20for%20Autonomous%20Driving/Autonomous_driving_application_Car_detection_v3a.ipynb\n",
        "# define yolo model and its related functions \n",
        "# find dataset and use it \n",
        "\n",
        "import sys\n",
        "# Setup \n",
        "doInstall = 1\n",
        "if doInstall:\n",
        "    !pip install SimpleITK\n",
        "    !pip install labelImg \n",
        "\n",
        "\n",
        "import os, random, time, math, colorsys, imghdr, PIL, csv\n",
        "\n",
        "from glob import glob\n",
        "\n",
        "import numpy as np\n",
        "import scipy.io\n",
        "import scipy.misc\n",
        "import pandas as pd\n",
        "\n",
        "from skimage.transform import resize\n",
        "import cv2 \n",
        "import SimpleITK as sitk \n",
        "from PIL import Image, ImageDraw, ImageFont\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import datasets, layers, models\n",
        "\n",
        "from keras import backend as K\n",
        "from keras.layers import Input, Lambda, Conv2D, BatchNormalization\n",
        "from keras.models import load_model, Model\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision.transforms as transforms\n",
        "import torch.optim as optim\n",
        "import torchvision.transforms.functional as FT\n",
        "from tqdm import tqdm\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.patches as patches\n",
        "from collections import Counter\n",
        "\n",
        "# Yolo\n",
        "# from yad2k.yad2k.models.keras_yolo import yolo_head, yolo_boxes_to_corners, preprocess_true_boxes, yolo_loss, yolo_body\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "# to reproduce the same results given same input\n",
        "np.random.seed(1)               \n",
        "!ls\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting SimpleITK\n",
            "  Downloading SimpleITK-2.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (48.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 48.4 MB 10 kB/s \n",
            "\u001b[?25hInstalling collected packages: SimpleITK\n",
            "Successfully installed SimpleITK-2.1.1\n",
            "Collecting labelImg\n",
            "  Downloading labelImg-1.8.6.tar.gz (247 kB)\n",
            "\u001b[K     |████████████████████████████████| 247 kB 12.6 MB/s \n",
            "\u001b[?25hCollecting pyqt5\n",
            "  Downloading PyQt5-5.15.6-cp36-abi3-manylinux1_x86_64.whl (8.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 8.3 MB 40.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from labelImg) (4.2.6)\n",
            "Collecting PyQt5-sip<13,>=12.8\n",
            "  Downloading PyQt5_sip-12.9.0-cp37-cp37m-manylinux1_x86_64.whl (317 kB)\n",
            "\u001b[K     |████████████████████████████████| 317 kB 46.9 MB/s \n",
            "\u001b[?25hCollecting PyQt5-Qt5>=5.15.2\n",
            "  Downloading PyQt5_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl (59.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 59.9 MB 61 kB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: labelImg\n",
            "  Building wheel for labelImg (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for labelImg: filename=labelImg-1.8.6-py2.py3-none-any.whl size=261540 sha256=e5a915b50d0b33aa04bc3deca3f20c68f41bc167fb023d6118ef5ded31aa0f47\n",
            "  Stored in directory: /root/.cache/pip/wheels/6e/83/14/df9548a7a658185c419c9bb21eeae66b13307c28881a37e151\n",
            "Successfully built labelImg\n",
            "Installing collected packages: PyQt5-sip, PyQt5-Qt5, pyqt5, labelImg\n",
            "Successfully installed PyQt5-Qt5-5.15.2 PyQt5-sip-12.9.0 labelImg-1.8.6 pyqt5-5.15.6\n",
            "sample_data\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q7WAn3WH3Lpr"
      },
      "source": [
        "# Image Object Detection\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoTv78FC4P_J"
      },
      "source": [
        "## Reading and exploring the datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M78Lx7yugJbM"
      },
      "source": [
        "### Using dataset minst / cifar10"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dzj-Tvek4V8s"
      },
      "source": [
        "# minst dataset\n",
        "if datasetID == 1:\n",
        "  (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "  class_names = range(10)\n",
        "\n",
        "\n",
        "# cifar10 dataset\n",
        "if datasetID == 2:\n",
        "  # The CIFAR10 dataset contains 60,000 color images in 10 classes, \n",
        "  # with 6,000 images in each class.\n",
        "  # The dataset is divided into 50,000 training images and 10,000 testing images.\n",
        "  # The classes are mutually exclusive and there is no overlap between them.\n",
        "\n",
        "  (x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()\n",
        "  class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n",
        "              'dog', 'frog', 'horse', 'ship', 'truck']\n",
        "\n",
        "\n",
        "if NNID == 4:\n",
        "  # 3D dataset\n",
        "  # TODO: fix this \n",
        "  showSamples =0\n",
        "  \n",
        "  x_train = x_train.reshape(-1,32*32*3) # (32 * 32 * 3)        \n",
        "  x_train = np.resize(x_train,(x_train.shape[0],15,15,15))        \n",
        "  x_train = x_train.reshape(-1,15,15,15)\n",
        "  x_test = x_test.reshape(-1,32*32*3) # (32 * 32 * 3)\n",
        "  x_test = np.resize(x_test,(x_test.shape[0],15,15,15))\n",
        "  x_test = x_test.reshape(-1,15,15,15)\n",
        "  x_train  =  x_train[..., np.newaxis] # np.reshape(x_train, (-1, h,w,1))\n",
        "  y_train  =  y_train[..., np.newaxis] # np.reshape(y_train, (-1, h,w,1))\n",
        "  x_test   =  x_test[..., np.newaxis]  # np.reshape(x_test,  (-1, h,w,1))\n",
        "  y_test   =  y_test[..., np.newaxis]  # np.reshape(y_test,  (-1, h,w,1))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YM16gpuBWAqz"
      },
      "source": [
        "### Using facial dataset\n",
        "\n",
        "\n",
        "We are using Face Detection Data Set and Benchmark (FDDB) [dataset](http://vis-www.cs.umass.edu/fddb/README.txt) from university of Massachusetts.  \n",
        "\n",
        "\n",
        "FDDB-folds contains files with names: FDDB-fold-xx.txt and FDDB-fold-xx-ellipseList.txt, where xx = {01, 02, ..., 10} represents the fold-index.\n",
        "\n",
        "Each line in the \"FDDB-fold-xx.txt\" file specifies a path to an image in the above-mentioned data set. For instance, the entry \"2002/07/19/big/img_130\" corresponds to \"originalPics/2002/07/19/big/img_130.jpg.\"\n",
        "\n",
        "The corresponding annotations are included in the file \"FDDB-fold-xx-ellipseList.txt\" in the following \n",
        "format:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "<image name i>\n",
        "<number of faces in this image =im>\n",
        "<face i1>\n",
        "<face i2>\n",
        "...\n",
        "<face im>\n",
        "```\n",
        "\n",
        "Here, each face is denoted by:\n",
        "\n",
        "            major_axis_radius minor_axis_radius angle center_x center_y 1\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vkWAKxh3WELE"
      },
      "source": [
        "showSamples = 1\n",
        "doDownload = 0\n",
        "\n",
        "# Facial dataset\n",
        "# more info http://vis-www.cs.umass.edu/fddb/README.txt\n",
        "\n",
        "if datasetID == 3:\n",
        "  doDownload = 1\n",
        "\n",
        "if doDownload:\n",
        "  !wget http://vis-www.cs.umass.edu/fddb/originalPics.tar.gz\n",
        "  !tar zxvf  originalPics.tar.gz\n",
        "\n",
        "  !wget http://vis-www.cs.umass.edu/fddb/FDDB-folds.tgz\n",
        "  !tar zxvf FDDB-folds.tgz\n",
        "  !mkdir originalPics\n",
        "  !mv 2002 originalPics\n",
        "  !mv 2003 originalPics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BIDkZloTgOkA"
      },
      "source": [
        "if datasetID == 3:\n",
        "  trnRatio = 0.90\n",
        "  tstRatio = 1 - trnRatio\n",
        "  img_fnms = sorted([y for x in os.walk(\"originalPics\") for y in glob(os.path.join(x[0], '*.jpg'))])\n",
        "  lbls_fnms = sorted([y for x in os.walk(\"FDDB-folds\") for y in glob(os.path.join(x[0], '*ellipseList.txt'))])\n",
        "  # print((img_fnms[0:2]))\n",
        "  # print(len(img_fnms))\n",
        "  # print((lbls_fnms[0:2]))\n",
        "  # print(len(lbls_fnms))\n",
        "  # print(\"--------------------------\")\n",
        "  lblInfo = []\n",
        "  numObj = 0\n",
        "  for fnm in lbls_fnms:\n",
        "      f = open(fnm,'r')\n",
        "      lines = f.readlines()\n",
        "      imgLoc=[]\n",
        "      for x in lines:\n",
        "          if \"img\" in x: \n",
        "              imgPath = x.strip()\n",
        "          elif \".\" in x:     \n",
        "              imgLoc.append([float(y) for y in x.split()])\n",
        "              if len(imgLoc) == numObj:\n",
        "                lblInfo.append([imgPath,numObj,imgLoc,])                   \n",
        "                imgLoc=[]\n",
        "                numObj=0\n",
        "          else:\n",
        "              numObj = int(x)\n",
        "\n",
        "  # print(\"---------------------------\")\n",
        "  # annotations for 5171 faces\n",
        "  # 2845 images  ???\n",
        "  # print(len(lblInfo))\n",
        "  s = 0 \n",
        "  for x in lblInfo:\n",
        "  #     print(x[0])\n",
        "  #     print(x[1])\n",
        "        s = s  + x[1]\n",
        "  #     for i in range (x[1]):\n",
        "  #         print(x[2][i])\n",
        "  # print(s)\n",
        "  # imgLst = [1             , 2               ]\n",
        "  # lblLst = [[1,3,[loc]]   , [1,3,[loc]]     ]\n",
        "\n",
        "  #(x_train, y_train), (x_test, y_test)\n",
        "  # class_names = range(10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbCxzquYXLAQ"
      },
      "source": [
        "### Using VOC dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZckvGJONXSOE",
        "outputId": "4694a87e-e798-4b9e-e90e-69db4010a6be"
      },
      "source": [
        "if datasetID == 4:\n",
        "  if not usePytorch:\n",
        "    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar\n",
        "    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar\n",
        "\n",
        "    !tar xvf VOCtrainval_06-Nov-2007.tar\n",
        "    !tar xvf VOCtest_06-Nov-2007.tar\n",
        "\n",
        "    !rm VOCtrainval_06-Nov-2007.tar\n",
        "    !rm VOCtest_06-Nov-2007.tar\n",
        "  \n",
        "  else:\n",
        "    # DOWNLOAD FROM HERE (FASTER DOWNLOAD)                                          \n",
        "    # VOC2007 DATASET                                                              \n",
        "    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar\n",
        "    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar # \n",
        "\n",
        "    # VOC2012 DATASET                                                              \n",
        "    !wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar\n",
        "\n",
        "    # Extract tar files\n",
        "    !tar xf VOCtrainval_11-May-2012.tar\n",
        "    !tar xf VOCtrainval_06-Nov-2007.tar\n",
        "    !tar xf VOCtest_06-Nov-2007.tar\n",
        "\n",
        "    # Need voc_label.py to clean up data from xml files\n",
        "    !wget https://pjreddie.com/media/files/voc_label.py\n",
        "\n",
        "    # Run python file to clean data from xml files\n",
        "    !python voc_label.py\n",
        "\n",
        "    # Get train by using train+val from 2007 and 2012\n",
        "    # Then we only test on 2007 test set\n",
        "    # Unclear from paper what they actually just as a dev set\n",
        "    !cat 2007_train.txt 2007_val.txt 2012_*.txt > train.txt\n",
        "    !cp 2007_test.txt test.txt\n",
        "\n",
        "    # Move txt files we won't be using to clean up a little bit\n",
        "    !mkdir old_txt_files\n",
        "    !mv 2007* 2012* old_txt_files/\n",
        "\n",
        "    read_train = open(\"train.txt\", \"r\").readlines()\n",
        "\n",
        "    with open(\"train.csv\", mode=\"w\", newline=\"\") as train_file:\n",
        "        for line in read_train:\n",
        "            image_file = line.split(\"/\")[-1].replace(\"\\n\", \"\")\n",
        "            text_file = image_file.replace(\".jpg\", \".txt\")\n",
        "            data = [image_file, text_file]\n",
        "            writer = csv.writer(train_file)\n",
        "            writer.writerow(data)\n",
        "\n",
        "    read_train = open(\"test.txt\", \"r\").readlines()\n",
        "\n",
        "    with open(\"test.csv\", mode=\"w\", newline=\"\") as train_file:\n",
        "        for line in read_train:\n",
        "            image_file = line.split(\"/\")[-1].replace(\"\\n\", \"\")\n",
        "            text_file = image_file.replace(\".jpg\", \".txt\")\n",
        "            data = [image_file, text_file]\n",
        "            writer = csv.writer(train_file)\n",
        "            writer.writerow(data)\n",
        "\n",
        "    !mkdir data\n",
        "    !mkdir data/images\n",
        "    !mkdir data/labels                                                                       \n",
        "                                                                                            \n",
        "    !mv VOCdevkit/VOC2007/JPEGImages/*.jpg data/images/                                      \n",
        "    !mv VOCdevkit/VOC2012/JPEGImages/*.jpg data/images/                                      \n",
        "    !mv VOCdevkit/VOC2007/labels/*.txt data/labels/                                          \n",
        "    !mv VOCdevkit/VOC2012/labels/*.txt data/labels/ \n",
        "\n",
        "    # We don't need VOCdevkit folder anymore, can remove\n",
        "    # in order to save some space \n",
        "    !rm -rf VOCdevkit/\n",
        "    !mv test.txt old_txt_files/\n",
        "    !mv train.txt old_txt_files/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-11-05 08:08:43--  http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar\n",
            "Resolving host.robots.ox.ac.uk (host.robots.ox.ac.uk)... 129.67.94.152\n",
            "Connecting to host.robots.ox.ac.uk (host.robots.ox.ac.uk)|129.67.94.152|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 460032000 (439M) [application/x-tar]\n",
            "Saving to: ‘VOCtrainval_06-Nov-2007.tar’\n",
            "\n",
            "VOCtrainval_06-Nov- 100%[===================>] 438.72M   113MB/s    in 3.5s    \n",
            "\n",
            "2021-11-05 08:08:46 (124 MB/s) - ‘VOCtrainval_06-Nov-2007.tar’ saved [460032000/460032000]\n",
            "\n",
            "--2021-11-05 08:08:46--  http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar\n",
            "Resolving host.robots.ox.ac.uk (host.robots.ox.ac.uk)... 129.67.94.152\n",
            "Connecting to host.robots.ox.ac.uk (host.robots.ox.ac.uk)|129.67.94.152|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 451020800 (430M) [application/x-tar]\n",
            "Saving to: ‘VOCtest_06-Nov-2007.tar’\n",
            "\n",
            "VOCtest_06-Nov-2007 100%[===================>] 430.13M   183MB/s    in 2.3s    \n",
            "\n",
            "2021-11-05 08:08:49 (183 MB/s) - ‘VOCtest_06-Nov-2007.tar’ saved [451020800/451020800]\n",
            "\n",
            "--2021-11-05 08:08:49--  http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar\n",
            "Resolving host.robots.ox.ac.uk (host.robots.ox.ac.uk)... 129.67.94.152\n",
            "Connecting to host.robots.ox.ac.uk (host.robots.ox.ac.uk)|129.67.94.152|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1999639040 (1.9G) [application/x-tar]\n",
            "Saving to: ‘VOCtrainval_11-May-2012.tar’\n",
            "\n",
            "VOCtrainval_11-May- 100%[===================>]   1.86G  93.0MB/s    in 13s     \n",
            "\n",
            "2021-11-05 08:09:01 (152 MB/s) - ‘VOCtrainval_11-May-2012.tar’ saved [1999639040/1999639040]\n",
            "\n",
            "--2021-11-05 08:09:38--  https://pjreddie.com/media/files/voc_label.py\n",
            "Resolving pjreddie.com (pjreddie.com)... 128.208.4.108\n",
            "Connecting to pjreddie.com (pjreddie.com)|128.208.4.108|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2042 (2.0K) [application/octet-stream]\n",
            "Saving to: ‘voc_label.py’\n",
            "\n",
            "voc_label.py        100%[===================>]   1.99K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-11-05 08:09:39 (397 MB/s) - ‘voc_label.py’ saved [2042/2042]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UU1oYI7WViwe"
      },
      "source": [
        "## Data preparation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rosb9MwaVsY2"
      },
      "source": [
        "if datasetID < 4:\n",
        "  # get size \n",
        "  h = x_train.shape[1] # image height\n",
        "  w = x_train.shape[2] # image width\n",
        "  # check for rgb \n",
        "  try:\n",
        "    # number of channels\n",
        "    c =  x_train.shape[3]\n",
        "  except:\n",
        "    # number of channels\n",
        "    c =  1\n",
        "    # if there is no number of channels, add 1\n",
        "    x_train  =  x_train[..., np.newaxis] # np.reshape(x_train, (-1, h,w,1))\n",
        "    y_train  =  y_train[..., np.newaxis] # np.reshape(y_train, (-1, h,w,1))\n",
        "    x_test   =  x_test[..., np.newaxis]  # np.reshape(x_test,  (-1, h,w,1))\n",
        "    y_test   =  y_test[..., np.newaxis]  # np.reshape(y_test,  (-1, h,w,1))\n",
        "\n",
        "\n",
        "  # Reserve 10,000 samples for validation.\n",
        "  x_val = x_train[-10000:]\n",
        "  y_val = y_train[-10000:]\n",
        "  x_train = x_train[:-10000]\n",
        "  y_train = y_train[:-10000]\n",
        "\n",
        "  number_of_pixels = h * w * c\n",
        "\n",
        "\n",
        "  # print(\"dataset shape   : \",x_train.shape)\n",
        "  # print(\"number of images: \",x_train.shape[0])\n",
        "  # print(\"image size      : \",x_train[0].shape)\n",
        "  # print(\"image data type : \",type(x_train[0][0][0][0]))\n",
        "  # print(\"image max  value: \",np.max(x_train[0]))\n",
        "  # print(\"image min  value: \",np.min(x_train[0]))\n",
        "  # if c==1:\n",
        "  #    print(\"gray or binary image (not color image)\")\n",
        "  # elif c==3:\n",
        "  #    print(\"rgb color image (or probably non-color image represented with 3 channels)\")\n",
        "\n",
        "\n",
        "  # display sample images \n",
        "  if showSamples:\n",
        "    plt.figure(figsize=(10,10))\n",
        "    for i in range(25):\n",
        "      plt.subplot(5,5,i+1)\n",
        "      plt.xticks([])\n",
        "      plt.yticks([])\n",
        "      plt.grid(False)\n",
        "      #plt.imshow(x_train[i])\n",
        "      plt.imshow(cv2.cvtColor(x_train[i], cv2.COLOR_BGR2RGB))\n",
        "\n",
        "      # The CIFAR labels happen to be arrays, \n",
        "      # which is why you need the extra index\n",
        "      if datasetID==1:\n",
        "        plt.xlabel(y_train[i])\n",
        "      elif datasetID==2:\n",
        "        plt.xlabel(class_names[y_train[i][0]])\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "  # normalisation\n",
        "  x_train = np.array([ x/255.0 for x in x_train])\n",
        "  x_val   = np.array([ x/255.0 for x in x_val])\n",
        "  x_test  = np.array([ x/255.0 for x in x_test])\n",
        "  #y_train = y_train.astype(np.float32)\n",
        "\n",
        "  # for NN we need 1D \n",
        "  if NNID == 1:\n",
        "    x_train = np.reshape(x_train, (-1, number_of_pixels))\n",
        "    x_val = np.reshape(x_val, (-1, number_of_pixels))\n",
        "    x_test = np.reshape(x_test, (-1, number_of_pixels))\n",
        "\n",
        "  # Prepare the training dataset.\n",
        "  # print(x_train.shape,y_train.shape)\n",
        "  train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "  train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n",
        "  #train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size)\n",
        "\n",
        "  # Prepare the validation dataset.\n",
        "  val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "  val_dataset = val_dataset.batch(batch_size)\n",
        "\n",
        "  # Prepare the test dataset.\n",
        "  tst_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
        "  tst_dataset = tst_dataset.batch(batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PXR-ArTJsUoQ"
      },
      "source": [
        "import argparse\n",
        "import xml.etree.ElementTree as ET\n",
        "import os\n",
        "import cv2 as cv\n",
        "import numpy as np\n",
        "from tensorflow import keras\n",
        "\n",
        "if datasetID == 4:\n",
        "  if not usePytorch:\n",
        "    parser = argparse.ArgumentParser(description='Build Annotations.')\n",
        "    parser.add_argument('dir', default='..', help='Annotations.')\n",
        "\n",
        "    sets = [('2007', 'train'), ('2007', 'val'), ('2007', 'test')]\n",
        "\n",
        "    classes_num = {'aeroplane': 0, 'bicycle': 1, 'bird': 2, 'boat': 3, 'bottle': 4, 'bus': 5,\n",
        "                  'car': 6, 'cat': 7, 'chair': 8, 'cow': 9, 'diningtable': 10, 'dog': 11,\n",
        "                  'horse': 12, 'motorbike': 13, 'person': 14, 'pottedplant': 15, 'sheep': 16,\n",
        "                  'sofa': 17, 'train': 18, 'tvmonitor': 19}\n",
        "\n",
        "\n",
        "    def convert_annotation(year, image_id, f):\n",
        "        in_file = os.path.join('VOCdevkit/VOC%s/Annotations/%s.xml' % (year, image_id))\n",
        "        tree = ET.parse(in_file)\n",
        "        root = tree.getroot()\n",
        "\n",
        "        for obj in root.iter('object'):\n",
        "            difficult = obj.find('difficult').text\n",
        "            cls = obj.find('name').text\n",
        "            classes = list(classes_num.keys())\n",
        "            if cls not in classes or int(difficult) == 1:\n",
        "                continue\n",
        "            cls_id = classes.index(cls)\n",
        "            xmlbox = obj.find('bndbox')\n",
        "            b = (int(xmlbox.find('xmin').text), int(xmlbox.find('ymin').text),\n",
        "                int(xmlbox.find('xmax').text), int(xmlbox.find('ymax').text))\n",
        "            f.write(' ' + ','.join([str(a) for a in b]) + ',' + str(cls_id))\n",
        "\n",
        "    for year, image_set in sets:\n",
        "      print(year, image_set)\n",
        "      with open(os.path.join('VOCdevkit/VOC%s/ImageSets/Main/%s.txt' % (year, image_set)), 'r') as f:\n",
        "          image_ids = f.read().strip().split()\n",
        "      with open(os.path.join(\"VOCdevkit\", '%s_%s.txt' % (year, image_set)), 'w') as f:\n",
        "          for image_id in image_ids:\n",
        "              f.write('%s/VOC%s/JPEGImages/%s.jpg' % (\"VOCdevkit\", year, image_id))\n",
        "              convert_annotation(year, image_id, f)\n",
        "              f.write('\\n')\n",
        "\n",
        "    # load data into batches\n",
        "    def read(image_path, label):\n",
        "        image = cv.imread(image_path)\n",
        "        image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n",
        "        image_h, image_w = image.shape[0:2]\n",
        "        image = cv.resize(image, (448, 448))\n",
        "        image = image / 255.\n",
        "\n",
        "        label_matrix = np.zeros([7, 7, 30])\n",
        "        for l in label:\n",
        "            l = l.split(',')\n",
        "            l = np.array(l, dtype=np.int)\n",
        "            xmin = l[0]\n",
        "            ymin = l[1]\n",
        "            xmax = l[2]\n",
        "            ymax = l[3]\n",
        "            cls = l[4]\n",
        "            x = (xmin + xmax) / 2 / image_w\n",
        "            y = (ymin + ymax) / 2 / image_h\n",
        "            w = (xmax - xmin) / image_w\n",
        "            h = (ymax - ymin) / image_h\n",
        "            loc = [7 * x, 7 * y]\n",
        "            loc_i = int(loc[1])\n",
        "            loc_j = int(loc[0])\n",
        "            y = loc[1] - loc_i\n",
        "            x = loc[0] - loc_j\n",
        "\n",
        "            if label_matrix[loc_i, loc_j, 24] == 0:\n",
        "                label_matrix[loc_i, loc_j, cls] = 1\n",
        "                label_matrix[loc_i, loc_j, 20:24] = [x, y, w, h]\n",
        "                label_matrix[loc_i, loc_j, 24] = 1  # response\n",
        "\n",
        "        return image, label_matrix"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "upeoNXVDszVO"
      },
      "source": [
        "class My_Custom_Generator(keras.utils.Sequence) :\n",
        "  \n",
        "  def __init__(self, images, labels, batch_size) :\n",
        "    self.images = images\n",
        "    self.labels = labels\n",
        "    self.batch_size = batch_size\n",
        "    \n",
        "    \n",
        "  def __len__(self) :\n",
        "    return (np.ceil(len(self.images) / float(self.batch_size))).astype(np.int)\n",
        "  \n",
        "  \n",
        "  def __getitem__(self, idx) :\n",
        "    batch_x = self.images[idx * self.batch_size : (idx+1) * self.batch_size]\n",
        "    batch_y = self.labels[idx * self.batch_size : (idx+1) * self.batch_size]\n",
        "\n",
        "    train_image = []\n",
        "    train_label = []\n",
        "\n",
        "    for i in range(0, len(batch_x)):\n",
        "      img_path = batch_x[i]\n",
        "      label = batch_y[i]\n",
        "      image, label_matrix = read(img_path, label)\n",
        "      train_image.append(image)\n",
        "      train_label.append(label_matrix)\n",
        "    return np.array(train_image), np.array(train_label)\n",
        "\n",
        "if datasetID == 4:\n",
        "  if not usePytorch:\n",
        "    train_datasets = []\n",
        "    val_datasets = []\n",
        "\n",
        "    with open(os.path.join(\"VOCdevkit\", '2007_train.txt'), 'r') as f:\n",
        "        train_datasets = train_datasets + f.readlines()\n",
        "    with open(os.path.join(\"VOCdevkit\", '2007_val.txt'), 'r') as f:\n",
        "        val_datasets = val_datasets + f.readlines()\n",
        "\n",
        "    X_train = []\n",
        "    Y_train = []\n",
        "\n",
        "    X_val = []\n",
        "    Y_val = []\n",
        "\n",
        "    for item in train_datasets:\n",
        "      item = item.replace(\"\\n\", \"\").split(\" \")\n",
        "      X_train.append(item[0])\n",
        "      arr = []\n",
        "      for i in range(1, len(item)):\n",
        "        arr.append(item[i])\n",
        "      Y_train.append(arr)\n",
        "\n",
        "    for item in val_datasets:\n",
        "      item = item.replace(\"\\n\", \"\").split(\" \")\n",
        "      X_val.append(item[0])\n",
        "      arr = []\n",
        "      for i in range(1, len(item)):\n",
        "        arr.append(item[i])\n",
        "      Y_val.append(arr)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qe5De7Wp7nES"
      },
      "source": [
        "class VOCDataset(torch.utils.data.Dataset):\n",
        "    def __init__(\n",
        "        self, csv_file, img_dir, label_dir, S=7, B=2, C=20, transform=None,\n",
        "    ):\n",
        "        self.annotations = pd.read_csv(csv_file)\n",
        "        self.img_dir = img_dir\n",
        "        self.label_dir = label_dir\n",
        "        self.transform = transform\n",
        "        self.S = S\n",
        "        self.B = B\n",
        "        self.C = C\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.annotations)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        label_path = os.path.join(self.label_dir, self.annotations.iloc[index, 1])\n",
        "        boxes = []\n",
        "        with open(label_path) as f:\n",
        "            for label in f.readlines():\n",
        "                class_label, x, y, width, height = [\n",
        "                    float(x) if float(x) != int(float(x)) else int(x)\n",
        "                    for x in label.replace(\"\\n\", \"\").split()\n",
        "                ]\n",
        "\n",
        "                boxes.append([class_label, x, y, width, height])\n",
        "\n",
        "        img_path = os.path.join(self.img_dir, self.annotations.iloc[index, 0])\n",
        "        image = Image.open(img_path)\n",
        "        boxes = torch.tensor(boxes)\n",
        "\n",
        "        if self.transform:\n",
        "            # image = self.transform(image)\n",
        "            image, boxes = self.transform(image, boxes)\n",
        "\n",
        "        # Convert To Cells\n",
        "        label_matrix = torch.zeros((self.S, self.S, self.C + 5 * self.B))\n",
        "        for box in boxes:\n",
        "            class_label, x, y, width, height = box.tolist()\n",
        "            class_label = int(class_label)\n",
        "\n",
        "            # i,j represents the cell row and cell column\n",
        "            i, j = int(self.S * y), int(self.S * x)\n",
        "            x_cell, y_cell = self.S * x - j, self.S * y - i\n",
        "\n",
        "            \"\"\"\n",
        "            Calculating the width and height of cell of bounding box,\n",
        "            relative to the cell is done by the following, with\n",
        "            width as the example:\n",
        "            \n",
        "            width_pixels = (width*self.image_width)\n",
        "            cell_pixels = (self.image_width)\n",
        "            \n",
        "            Then to find the width relative to the cell is simply:\n",
        "            width_pixels/cell_pixels, simplification leads to the\n",
        "            formulas below.\n",
        "            \"\"\"\n",
        "            width_cell, height_cell = (\n",
        "                width * self.S,\n",
        "                height * self.S,\n",
        "            )\n",
        "\n",
        "            # If no object already found for specific cell i,j\n",
        "            # Note: This means we restrict to ONE object\n",
        "            # per cell!\n",
        "            if label_matrix[i, j, 20] == 0:\n",
        "                # Set that there exists an object\n",
        "                label_matrix[i, j, 20] = 1\n",
        "\n",
        "                # Box coordinates\n",
        "                box_coordinates = torch.tensor(\n",
        "                    [x_cell, y_cell, width_cell, height_cell]\n",
        "                )\n",
        "\n",
        "                label_matrix[i, j, 21:25] = box_coordinates\n",
        "\n",
        "                # Set one hot encoding for class_label\n",
        "                label_matrix[i, j, class_label] = 1\n",
        "\n",
        "        return image, label_matrix"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bP0E_Pzs04y_"
      },
      "source": [
        "## Dataset augmentation\n",
        "\n",
        "It is important to train the model on different variations of the dataset. It is also important to have large datset for training.\n",
        "\n",
        "Using dataset augmentation helps to achieve both of the above goals. From one image, one can generate hundred thousands of images using image transformation.\n",
        "\n",
        "The image transformation could be [spatial transform]() or point transform where we move the points of the image to new locations e.g. shifting, flipping, and/or rotating the imag. \n",
        "\n",
        "Another type of transformation is intensity transform or pixel transform where we change the color values of the pixels in the image e.g. invert the color, add more brightness or darkness. \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jb2KdoTR07L2"
      },
      "source": [
        "# TODO\n",
        "doAug = 0\n",
        "\n",
        "def imagePixelTransforms(img):    \n",
        "    images = []\n",
        "    # let's make 3 simple transformations\n",
        "    img1   = 1.0- img # invert color\n",
        "    img2   = img +0.3 # more brightness\n",
        "    img3   = img -0.3 # more darkness\n",
        "    images = np.array([img1,img2,img3])\n",
        "    images = [ img.reshape(img.shape) for img in images]\n",
        "\n",
        "    # plt.figure() ;    plt.imshow(img)\n",
        "    # plt.figure() ;    plt.imshow(img1)\n",
        "    # plt.figure() ;    plt.imshow(img2)\n",
        "    # plt.figure() ;    plt.imshow(img3)    \n",
        "    return images\n",
        "\n",
        "def imagePointTransforms(img):\n",
        "    images = []\n",
        "    # let's make 3 simple transformations\n",
        "    # Perform the rotation\n",
        "    center  = (img.shape[0] / 2, img.shape[1] / 2)\n",
        "    sz      = (img.shape[1], img.shape[0])\n",
        "    tMatrix = cv2.getRotationMatrix2D(center, 45, 1)\n",
        "    img1 = cv2.warpAffine(img, tMatrix, sz)\n",
        "    img1 = img1[...,np.newaxis] if img1.shape !=img.shape else img1\n",
        "    tMatrix = cv2.getRotationMatrix2D(center, 90, 1)\n",
        "    img2 = cv2.warpAffine(img, tMatrix, sz)\n",
        "    img2 = img2[...,np.newaxis] if img2.shape !=img.shape else img2\n",
        "    tMatrix = cv2.getRotationMatrix2D(center, 270, 1)\n",
        "    img3 = cv2.warpAffine(img, tMatrix, sz)\n",
        "    img3 = img3[...,np.newaxis] if img3.shape !=img.shape else img3\n",
        "\n",
        "    images = np.array([img1,img2,img3])\n",
        "    # plt.figure() ;    plt.imshow(img)\n",
        "    # plt.figure() ;    plt.imshow(img1)\n",
        "    # plt.figure() ;    plt.imshow(img2)\n",
        "    # plt.figure() ;    plt.imshow(img3)    \n",
        "    #plt.figure() ;    plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))\n",
        "    # plt.figure() ;    plt.imshow(cv2.cvtColor(img1, cv2.COLOR_BGR2RGB))\n",
        "    return images\n",
        "\n",
        "\n",
        "# define a function for sitk transform\n",
        "def resample(img_array, transform):\n",
        "    # Output image Origin, Spacing, Size, Direction are taken from the reference\n",
        "    # image in this call to Resample\n",
        "    image = sitk.GetImageFromArray(img_array)\n",
        "    reference_image = image\n",
        "    interpolator = sitk.sitkCosineWindowedSinc\n",
        "    default_value = 100.0\n",
        "    resampled_img = sitk.Resample(image, reference_image, transform,\n",
        "                         interpolator, default_value)\n",
        "    resampled_array = sitk.GetArrayFromImage(resampled_img)\n",
        "    return resampled_array\n",
        "\n",
        "def affine_rotate(transform, degrees):\n",
        "    parameters = np.array(transform.GetParameters())\n",
        "    new_transform = sitk.AffineTransform(transform)\n",
        "    dimension =3 \n",
        "    matrix = np.array(transform.GetMatrix()).reshape((dimension,dimension))\n",
        "    radians = -np.pi * degrees / 180.\n",
        "    rotation = np.array([[1  ,0,0], \n",
        "                         [0, np.cos(radians), -np.sin(radians)],\n",
        "                         [0, np.sin(radians), np.cos(radians)]]\n",
        "                        )\n",
        "    new_matrix = np.dot(rotation, matrix)\n",
        "    new_transform.SetMatrix(new_matrix.ravel())\n",
        "    return new_transform\n",
        "\n",
        "\n",
        "def imagePoint3DTransforms(img):\n",
        "    #print(\"imagePoint3DTransforms\")\n",
        "    images = []\n",
        "    # let's make 3 simple transformations\n",
        "    # Perform the rotation\n",
        "    # In SimpleITK resampling convention, the transformation maps points \n",
        "    # from the fixed image to the moving image,\n",
        "    # so inverse of the transform is applied\n",
        "\n",
        "    center = (img.shape[0] /2, img.shape[1] /2,img.shape[1] /2)\n",
        "    rotation_around_center = sitk.AffineTransform(3)\n",
        "    rotation_around_center.SetCenter(center)\n",
        "    \n",
        "    rotation_around_center = affine_rotate(rotation_around_center, -45)\n",
        "    img1 = resample(img, rotation_around_center)\n",
        "\n",
        "    rotation_around_center = affine_rotate(rotation_around_center, -90)\n",
        "    img2 = resample(img, rotation_around_center)\n",
        "\n",
        "    rotation_around_center = affine_rotate(rotation_around_center, -90)\n",
        "    img3 = resample(img, rotation_around_center)\n",
        "\n",
        "    images = np.array([img1,img2,img3])\n",
        "    # plt.figure() ;    plt.imshow(img)\n",
        "    # plt.figure() ;    plt.imshow(img1)\n",
        "    # plt.figure() ;    plt.imshow(img2)\n",
        "    # plt.figure() ;    plt.imshow(img3)    \n",
        "    #plt.figure() ;    plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))\n",
        "    # plt.figure() ;    plt.imshow(cv2.cvtColor(img1, cv2.COLOR_BGR2RGB))\n",
        "    return images\n",
        "\n",
        "def doAugmentation(images,labels,batch_size):\n",
        "    # input is an image or a batch e.g. list of images \n",
        "    # get numpy arrays from the tensor    \n",
        "    images = images.numpy()\n",
        "    labels = labels.numpy()\n",
        "    # if 1d convert back to 2d\n",
        "    #print(images.shape)\n",
        "    rgb = 0 ; is3d = 0\n",
        "    if len(images.shape) == 2:\n",
        "       try: \n",
        "          img2d_shape = int(math.sqrt(images.shape[1])) # gray or binary image\n",
        "          images =images.reshape(-1,img2d_shape,img2d_shape)\n",
        "       except:\n",
        "          try: \n",
        "            img2d_shape = int(math.sqrt(images.shape[1]/3)) # rgb image\n",
        "            images =images.reshape(-1,img2d_shape,img2d_shape,3)\n",
        "            rgb = 1  \n",
        "          except:\n",
        "            pass  \n",
        "            # img3d_shape = int(math.sqrt(images.shape[1]/3)) # rgb image\n",
        "            # images =images.reshape(-1,img2d_shape,img2d_shape,3)\n",
        "            # is3d = 1  \n",
        "\n",
        "\n",
        "\n",
        "    x_outputs = [] ; y_outputs = []\n",
        "    i = 0\n",
        "    for img in images:\n",
        "        #print(\"-------------------------\", i ,\"--------------------\")\n",
        "        if NNID==4:\n",
        "           img = img.squeeze() \n",
        "        # from each images we generate 6 images\n",
        "        # 64 batch will generate 448\n",
        "        x_outputs.extend([img])\n",
        "        imgs1 = imagePoint3DTransforms(img)\n",
        "        imgs2 = imagePixelTransforms(img)\n",
        "        #if not rgb:\n",
        "           #imgs1 = np.array( x[...,np.newaxis] for x in imgs1 if len(x.shape)<3) \n",
        "           #imgs2 = np.array( x[...,np.newaxis] for x in imgs2 if len(x.shape)<3)\n",
        "        x_outputs.extend(imgs1) # 3 images\n",
        "        x_outputs.extend(imgs2) # 3 images\n",
        "        # print(img.shape)\n",
        "        # print(imgs1[0].shape)\n",
        "        # print(imgs2[0].shape)\n",
        "        # assign the same label to all transformed images\n",
        "        for j in range ( len(imgs1) +len(imgs2)+1):\n",
        "            y_outputs.extend([labels[i]])\n",
        "\n",
        "        i = i +1\n",
        "    x_outputs = np.array(x_outputs)\n",
        "    if NNID==4:\n",
        "       x_outputs = np.array([x[...,np.newaxis] for x in x_outputs])\n",
        "    y_outputs = np.array(y_outputs)\n",
        "\n",
        "    if (not rgb) and (NNID==1):\n",
        "       x_outputs = np.reshape(x_outputs, (-1,img2d_shape*img2d_shape,1))\n",
        "    elif (rgb) and (NNID==1):\n",
        "       x_outputs = np.reshape(x_outputs, (-1,img2d_shape*img2d_shape*3))   \n",
        "\n",
        "    new_train_dataset = tf.data.Dataset.from_tensor_slices((x_outputs, y_outputs))\n",
        "    new_train_dataset = new_train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n",
        "\n",
        "    return new_train_dataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MmSF08Oq4ae6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27e98271-3300-4a37-de65-94be5923f565"
      },
      "source": [
        "# NN TensorFlow\n",
        "def getNNModel(number_of_pixels,number_of_classes):\n",
        "    inputs = keras.Input(shape=(number_of_pixels,), name=\"digits\")\n",
        "    x = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "    x = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x)\n",
        "    outputs = layers.Dense(number_of_classes, name=\"predictions\")(x)\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    return model\n",
        "print(\"NN model is defined ...\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NN model is defined ...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tpxH1ntsqfuZ"
      },
      "source": [
        "# TODO\n",
        "# NN pytorch\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HtEO2ORyBFeu"
      },
      "source": [
        "## Define optimiser and loss function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uQbxXtlvGj1r"
      },
      "source": [
        "# defined loss functions of Yolo\n",
        "\n",
        "def xywh2minmax(xy, wh):\n",
        "  xy_min = xy - wh / 2\n",
        "  xy_max = xy + wh / 2\n",
        "\n",
        "  return xy_min, xy_max\n",
        "\n",
        "\n",
        "def iou(pred_mins, pred_maxes, true_mins, true_maxes):\n",
        "  intersect_mins = K.maximum(pred_mins, true_mins)\n",
        "  intersect_maxes = K.minimum(pred_maxes, true_maxes)\n",
        "  intersect_wh = K.maximum(intersect_maxes - intersect_mins, 0.)\n",
        "  intersect_areas = intersect_wh[..., 0] * intersect_wh[..., 1]\n",
        "\n",
        "  pred_wh = pred_maxes - pred_mins\n",
        "  true_wh = true_maxes - true_mins\n",
        "  pred_areas = pred_wh[..., 0] * pred_wh[..., 1]\n",
        "  true_areas = true_wh[..., 0] * true_wh[..., 1]\n",
        "\n",
        "  union_areas = pred_areas + true_areas - intersect_areas\n",
        "  iou_scores = intersect_areas / union_areas\n",
        "\n",
        "  return iou_scores\n",
        "\n",
        "\n",
        "def yolo_head(feats):\n",
        "  # Dynamic implementation of conv dims for fully convolutional model.\n",
        "  conv_dims = K.shape(feats)[1:3]  # assuming channels last\n",
        "  # In YOLO the height index is the inner most iteration.\n",
        "  conv_height_index = K.arange(0, stop=conv_dims[0])\n",
        "  conv_width_index = K.arange(0, stop=conv_dims[1])\n",
        "  conv_height_index = K.tile(conv_height_index, [conv_dims[1]])\n",
        "\n",
        "  # TODO: Repeat_elements and tf.split doesn't support dynamic splits.\n",
        "  # conv_width_index = K.repeat_elements(conv_width_index, conv_dims[1], axis=0)\n",
        "  conv_width_index = K.tile(K.expand_dims(conv_width_index, 0), [conv_dims[0], 1])\n",
        "  conv_width_index = K.flatten(K.transpose(conv_width_index))\n",
        "  conv_index = K.transpose(K.stack([conv_height_index, conv_width_index]))\n",
        "  conv_index = K.reshape(conv_index, [1, conv_dims[0], conv_dims[1], 1, 2])\n",
        "  conv_index = K.cast(conv_index, K.dtype(feats))\n",
        "\n",
        "  conv_dims = K.cast(K.reshape(conv_dims, [1, 1, 1, 1, 2]), K.dtype(feats))\n",
        "\n",
        "  box_xy = (feats[..., :2] + conv_index) / conv_dims * 448\n",
        "  box_wh = feats[..., 2:4] * 448\n",
        "\n",
        "  return box_xy, box_wh\n",
        "\n",
        "def yolo_loss(y_true, y_pred):\n",
        "  # S is the split size of image\n",
        "  # B is the num of boxes\n",
        "  # C is the num of classes\n",
        "\n",
        "  \n",
        "  y_pred = K.cast(y_pred, K.dtype(y_true))\n",
        "  \n",
        "  label_class   = y_true[..., :20]  # ? * 7 * 7 * 20\n",
        "  label_box     = y_true[..., 20:24]  # ? * 7 * 7 * 4\n",
        "  response_mask = y_true[..., 24]  # ? * 7 * 7\n",
        "  response_mask = K.expand_dims(response_mask)  # ? * 7 * 7 * 1\n",
        "\n",
        "  predict_class  = y_pred[..., :20]  # ? * 7 * 7 * 20\n",
        "  predict_trust  = y_pred[..., 20:22]  # ? * 7 * 7 * 2\n",
        "\n",
        "  predict_box    = y_pred[..., 22:]  # ? * 7 * 7 * 8\n",
        "\n",
        "  _label_box    = K.reshape(label_box, [-1, 7, 7, 1, 4])\n",
        "  _predict_box  = K.reshape(predict_box, [-1, 7, 7, 2, 4])\n",
        "\n",
        "  label_xy, label_wh = yolo_head(_label_box)  # ? * 7 * 7 * 1 * 2, ? * 7 * 7 * 1 * 2\n",
        "  label_xy = K.expand_dims(label_xy, 3)  # ? * 7 * 7 * 1 * 1 * 2\n",
        "  label_wh = K.expand_dims(label_wh, 3)  # ? * 7 * 7 * 1 * 1 * 2\n",
        "  label_xy_min, label_xy_max = xywh2minmax(label_xy, label_wh)  # ? * 7 * 7 * 1 * 1 * 2, ? * 7 * 7 * 1 * 1 * 2\n",
        "\n",
        "  predict_xy, predict_wh = yolo_head(_predict_box)  # ? * 7 * 7 * 2 * 2, ? * 7 * 7 * 2 * 2\n",
        "  predict_xy = K.expand_dims(predict_xy, 4)  # ? * 7 * 7 * 2 * 1 * 2\n",
        "  predict_wh = K.expand_dims(predict_wh, 4)  # ? * 7 * 7 * 2 * 1 * 2\n",
        "  predict_xy_min, predict_xy_max = xywh2minmax(predict_xy, predict_wh)  # ? * 7 * 7 * 2 * 1 * 2, ? * 7 * 7 * 2 * 1 * 2\n",
        "\n",
        "  iou_scores = iou(predict_xy_min, predict_xy_max, label_xy_min, label_xy_max)  # ? * 7 * 7 * 2 * 1\n",
        "  best_ious = K.max(iou_scores, axis=4)  # ? * 7 * 7 * 2\n",
        "  best_box = K.max(best_ious, axis=3, keepdims=True)  # ? * 7 * 7 * 1\n",
        "\n",
        "  box_mask = K.cast(best_ious >= best_box, K.dtype(best_ious))  # ? * 7 * 7 * 2\n",
        "\n",
        "  \n",
        "  response_mask = K.cast(response_mask, K.dtype(box_mask))\n",
        "\n",
        "  no_object_loss = 0.5 * (1 - box_mask * response_mask)\n",
        "  no_object_loss = no_object_loss * K.square(0 - predict_trust)\n",
        "  object_loss = box_mask * response_mask * K.square(1 - predict_trust)\n",
        "  confidence_loss = no_object_loss + object_loss\n",
        "  confidence_loss = K.sum(confidence_loss)\n",
        "\n",
        "  class_loss = response_mask * K.square(label_class - predict_class)\n",
        "  class_loss = K.sum(class_loss)\n",
        "\n",
        "  _label_box = K.reshape(label_box, [-1, 7, 7, 1, 4])\n",
        "  _predict_box = K.reshape(predict_box, [-1, 7, 7, 2, 4])\n",
        "\n",
        "  label_xy, label_wh = yolo_head(_label_box)  # ? * 7 * 7 * 1 * 2, ? * 7 * 7 * 1 * 2\n",
        "  predict_xy, predict_wh = yolo_head(_predict_box)  # ? * 7 * 7 * 2 * 2, ? * 7 * 7 * 2 * 2\n",
        "\n",
        "  box_mask = K.expand_dims(box_mask)\n",
        "  response_mask = K.expand_dims(response_mask)\n",
        "\n",
        "  box_loss  = 5 * box_mask * response_mask * K.square((label_xy - predict_xy) / 448)\n",
        "  box_loss += 5 * box_mask * response_mask * K.square((K.sqrt(label_wh) - K.sqrt(predict_wh)) / 448)\n",
        "  box_loss = K.sum(box_loss)\n",
        "\n",
        "  loss = confidence_loss + class_loss + box_loss\n",
        "\n",
        "  return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D2CBVAaABKU4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "38eee568-c3b3-48a1-f3c5-bfe8fa5f7f8c"
      },
      "source": [
        "# Instantiate an optimizer to train the model.\n",
        "\n",
        "optimiserID = 1 # SGD by default for ADAM use 2 \n",
        "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
        "if optimiserID ==2:\n",
        "   optimizer = keras.optimizers.Adam()#learning_rate=0.0001\n",
        "# Instantiate a loss function.\n",
        "\n",
        "if lossFunctionID == 1: # SparseCategoricalCrossentropy by default for MSE use 2 \n",
        "  loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
        "\n",
        "# Prepare the metrics.\n",
        "train_acc_metric = keras.metrics.SparseCategoricalAccuracy()\n",
        "val_acc_metric   = keras.metrics.SparseCategoricalAccuracy()\n",
        "tst_acc_metric   = keras.metrics.SparseCategoricalAccuracy()\n",
        "if lossFunctionID==2:\n",
        "   loss_fn = keras.losses.MeanSquaredError()\n",
        "   # Prepare the metrics.\n",
        "   train_acc_metric = keras.metrics.MeanSquaredError()\n",
        "   val_acc_metric   = keras.metrics.MeanSquaredError()\n",
        "   tst_acc_metric   = keras.metrics.MeanSquaredError()\n",
        "\n",
        "elif lossFunctionID==3:\n",
        "   loss_fn = keras.losses.CategoricalCrossentropy()\n",
        "   # Prepare the metrics.\n",
        "   train_acc_metric = keras.metrics.CategoricalCrossentropy()\n",
        "   val_acc_metric   = keras.metrics.CategoricalCrossentropy()\n",
        "   tst_acc_metric   = keras.metrics.CategoricalCrossentropy()\n",
        "elif lossFunctionID==4:\n",
        "   print(\"loss: yolo_loss\")\n",
        "   loss_fn = yolo_loss\n",
        "  #  # Prepare the metrics.\n",
        "  #  train_acc_metric = keras.metrics.CategoricalCrossentropy()\n",
        "  #  val_acc_metric   = keras.metrics.CategoricalCrossentropy()\n",
        "  #  tst_acc_metric   = keras.metrics.CategoricalCrossentropy()\n",
        "print(\"optimiser, loss, and metrics are defined .... \")\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "loss: yolo_loss\n",
            "optimiser, loss, and metrics are defined .... \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AmAbPxuDqrVV"
      },
      "source": [
        "# TODO\n",
        "# Pytorch\n",
        "class YoloLoss(nn.Module):\n",
        "    \"\"\"\n",
        "    Calculate the loss for yolo (v1) model\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, S=7, B=2, C=20):\n",
        "        super(YoloLoss, self).__init__()\n",
        "        self.mse = nn.MSELoss(reduction=\"sum\")\n",
        "\n",
        "        \"\"\"\n",
        "        S is split size of image (in paper 7),\n",
        "        B is number of boxes (in paper 2),\n",
        "        C is number of classes (in paper and VOC dataset is 20),\n",
        "        \"\"\"\n",
        "        self.S = S\n",
        "        self.B = B\n",
        "        self.C = C\n",
        "\n",
        "        # These are from Yolo paper, signifying how much we should\n",
        "        # pay loss for no object (noobj) and the box coordinates (coord)\n",
        "        self.lambda_noobj = 0.5\n",
        "        self.lambda_coord = 5\n",
        "\n",
        "    def forward(self, predictions, target):\n",
        "        # predictions are shaped (BATCH_SIZE, S*S(C+B*5) when inputted\n",
        "        predictions = predictions.reshape(-1, self.S, self.S, self.C + self.B * 5)\n",
        "\n",
        "        # Calculate IoU for the two predicted bounding boxes with target bbox\n",
        "        iou_b1 = intersection_over_union(predictions[..., 21:25], target[..., 21:25])\n",
        "        iou_b2 = intersection_over_union(predictions[..., 26:30], target[..., 21:25])\n",
        "        ious = torch.cat([iou_b1.unsqueeze(0), iou_b2.unsqueeze(0)], dim=0)\n",
        "\n",
        "        # Take the box with highest IoU out of the two prediction\n",
        "        # Note that bestbox will be indices of 0, 1 for which bbox was best\n",
        "        iou_maxes, bestbox = torch.max(ious, dim=0)\n",
        "        exists_box = target[..., 20].unsqueeze(3)  # in paper this is Iobj_i\n",
        "\n",
        "        # ======================== #\n",
        "        #   FOR BOX COORDINATES    #\n",
        "        # ======================== #\n",
        "\n",
        "        # Set boxes with no object in them to 0. We only take out one of the two \n",
        "        # predictions, which is the one with highest Iou calculated previously.\n",
        "        box_predictions = exists_box * (\n",
        "            (\n",
        "                bestbox * predictions[..., 26:30]\n",
        "                + (1 - bestbox) * predictions[..., 21:25]\n",
        "            )\n",
        "        )\n",
        "\n",
        "        box_targets = exists_box * target[..., 21:25]\n",
        "\n",
        "        # Take sqrt of width, height of boxes to ensure that\n",
        "        box_predictions[..., 2:4] = torch.sign(box_predictions[..., 2:4]) * torch.sqrt(\n",
        "            torch.abs(box_predictions[..., 2:4] + 1e-6)\n",
        "        )\n",
        "        box_targets[..., 2:4] = torch.sqrt(box_targets[..., 2:4])\n",
        "\n",
        "        box_loss = self.mse(\n",
        "            torch.flatten(box_predictions, end_dim=-2),\n",
        "            torch.flatten(box_targets, end_dim=-2),\n",
        "        )\n",
        "\n",
        "        # ==================== #\n",
        "        #   FOR OBJECT LOSS    #\n",
        "        # ==================== #\n",
        "\n",
        "        # pred_box is the confidence score for the bbox with highest IoU\n",
        "        pred_box = (\n",
        "            bestbox * predictions[..., 25:26] + (1 - bestbox) * predictions[..., 20:21]\n",
        "        )\n",
        "\n",
        "        object_loss = self.mse(\n",
        "            torch.flatten(exists_box * pred_box),\n",
        "            torch.flatten(exists_box * target[..., 20:21]),\n",
        "        )\n",
        "\n",
        "        # ======================= #\n",
        "        #   FOR NO OBJECT LOSS    #\n",
        "        # ======================= #\n",
        "\n",
        "        #max_no_obj = torch.max(predictions[..., 20:21], predictions[..., 25:26])\n",
        "        #no_object_loss = self.mse(\n",
        "        #    torch.flatten((1 - exists_box) * max_no_obj, start_dim=1),\n",
        "        #    torch.flatten((1 - exists_box) * target[..., 20:21], start_dim=1),\n",
        "        #)\n",
        "\n",
        "        no_object_loss = self.mse(\n",
        "            torch.flatten((1 - exists_box) * predictions[..., 20:21], start_dim=1),\n",
        "            torch.flatten((1 - exists_box) * target[..., 20:21], start_dim=1),\n",
        "        )\n",
        "\n",
        "        no_object_loss += self.mse(\n",
        "            torch.flatten((1 - exists_box) * predictions[..., 25:26], start_dim=1),\n",
        "            torch.flatten((1 - exists_box) * target[..., 20:21], start_dim=1)\n",
        "        )\n",
        "\n",
        "        # ================== #\n",
        "        #   FOR CLASS LOSS   #\n",
        "        # ================== #\n",
        "\n",
        "        class_loss = self.mse(\n",
        "            torch.flatten(exists_box * predictions[..., :20], end_dim=-2,),\n",
        "            torch.flatten(exists_box * target[..., :20], end_dim=-2,),\n",
        "        )\n",
        "\n",
        "        loss = (\n",
        "            self.lambda_coord * box_loss  # first two rows in paper\n",
        "            + object_loss  # third row in paper\n",
        "            + self.lambda_noobj * no_object_loss  # forth row\n",
        "            + class_loss  # fifth row\n",
        "        )\n",
        "\n",
        "        return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dxyj5W7P4d2M"
      },
      "source": [
        "\n",
        "## Define required functions\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C8dFra_t4i8d"
      },
      "source": [
        "# define training parameters and file paths \n",
        "\n",
        "# model log files path\n",
        "modelPath   = \"./modelClassification.h5\"\n",
        "logFilePath = \"./training_log.csv\"\n",
        "figPath     = \"./training_log.png\"\n",
        "\n",
        "logFile = open(logFilePath,'w')\n",
        "logFile.write(\"epoch \\t trnLoss \\t valLoss \\t trnAcc \\t valAcc \\t time \\n\" )\n",
        "logFile.close()\n",
        "# Using optimised tensorflow functions provides more speed\n",
        "\n",
        "@tf.function\n",
        "def train_step(model,x, y):\n",
        "    print(\"train_step\")\n",
        "\n",
        "    with tf.GradientTape() as tape:\n",
        "        #print(\"get result\")\n",
        "        model_output = model(x, training=True)\n",
        "        # print(\"model_output: \", model_output.shape)\n",
        "        # print(\"y: \", y.shape)\n",
        "        #print(\"get loss value \")\n",
        "        #y = keras.utils.to_categorical(y)\n",
        "        loss_value = loss_fn(y, model_output)\n",
        "        # print(\"loss_value: \", loss_value)\n",
        "    grads = tape.gradient(loss_value, model.trainable_weights)\n",
        "    # print(\"grads\")\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
        "    # print(\"optimizer\")\n",
        "    # print(\"y: \", y.shape)\n",
        "    # print(\"model_output: \", model_output.shape)\n",
        "    # train_acc_metric.update_state(y, model_output)\n",
        "    # print(\"update_state\")\n",
        "    return loss_value\n",
        "\n",
        "@tf.function\n",
        "def val_step(model,x, y):\n",
        "    val_logits = model(x, training=False)\n",
        "    #y = keras.utils.to_categorical(y)\n",
        "    loss_value = loss_fn(y, val_logits)\n",
        "    # val_acc_metric.update_state(y, val_logits)\n",
        "    return loss_value\n",
        "\n",
        "# plotting function to monitor the curves\n",
        "def iaPlotLoss(logPath,figPath=None):\n",
        "    f = open(logPath,'r')\n",
        "    lst = f.readlines()\n",
        "    # first line is labels:\n",
        "    labels = lst[0].split()[1:-2]\n",
        "    x  = [ int(  ln.split()[0]) for ln in lst[1:]] # epoch\n",
        "    y1 = [ float(ln.split()[1]) for ln in lst[1:]] # lossTrain\n",
        "    y2 = [ float(ln.split()[2]) for ln in lst[1:]] # lossValidation\n",
        "    y3 = [ float(ln.split()[3]) for ln in lst[1:]] # accTrain\n",
        "    y4 = [ float(ln.split()[4]) for ln in lst[1:]] # accValidation\n",
        "    #plotting    \n",
        "    plt.clf()\n",
        "    fig, ax = plt.subplots()    \n",
        "    l1, = ax.plot(x, y1) ;     l2, = ax.plot(x, y2) ;\n",
        "    l3, = ax.plot(x, y3) ;     l4, = ax.plot(x, y4) ;\n",
        "    ax.legend((l1, l2,l3,l4), labels, loc='upper right', shadow=True)\n",
        "    plt.xlabel('epoch')\n",
        "    if figPath:\n",
        "        plt.savefig(figPath, bbox_inches='tight')\n",
        "    else:\n",
        "        plt.show()\n",
        "        plt.close()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlx1hVMQGAbR"
      },
      "source": [
        "if NNID==1:\n",
        "    # Load the saved model \n",
        "    model = keras.models.load_model(modelPath, compile=False)\n",
        "\n",
        "    start_time = time.time() \n",
        "    # Run a validation loop at the end of each epoch.\n",
        "    for x_batch_tst, y_batch_tst in tst_dataset:\n",
        "        output = model.predict(x_batch_tst)\n",
        "        #y = keras.utils.to_categorical(y_batch_tst)\n",
        "        tst_acc_metric.update_state(y_batch_tst, output)\n",
        "\n",
        "    tst_acc = tst_acc_metric.result()\n",
        "\n",
        "    # compute time required for each epoch\n",
        "    end_time = time.time() - start_time\n",
        "\n",
        "    print(\"test accuracy : %.4f \\t time:  %.2f\" % (  float(tst_acc), end_time))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WpZmXOUfrDho"
      },
      "source": [
        "# TODO Pytorch\n",
        "def intersection_over_union(boxes_preds, boxes_labels, box_format=\"midpoint\"):\n",
        "    \"\"\"\n",
        "    Calculates intersection over union\n",
        "    Parameters:\n",
        "        boxes_preds (tensor): Predictions of Bounding Boxes (BATCH_SIZE, 4)\n",
        "        boxes_labels (tensor): Correct labels of Bounding Boxes (BATCH_SIZE, 4)\n",
        "        box_format (str): midpoint/corners, if boxes (x,y,w,h) or (x1,y1,x2,y2)\n",
        "    Returns:\n",
        "        tensor: Intersection over union for all examples\n",
        "    \"\"\"\n",
        "\n",
        "    if box_format == \"midpoint\":\n",
        "        box1_x1 = boxes_preds[..., 0:1] - boxes_preds[..., 2:3] / 2\n",
        "        box1_y1 = boxes_preds[..., 1:2] - boxes_preds[..., 3:4] / 2\n",
        "        box1_x2 = boxes_preds[..., 0:1] + boxes_preds[..., 2:3] / 2\n",
        "        box1_y2 = boxes_preds[..., 1:2] + boxes_preds[..., 3:4] / 2\n",
        "        box2_x1 = boxes_labels[..., 0:1] - boxes_labels[..., 2:3] / 2\n",
        "        box2_y1 = boxes_labels[..., 1:2] - boxes_labels[..., 3:4] / 2\n",
        "        box2_x2 = boxes_labels[..., 0:1] + boxes_labels[..., 2:3] / 2\n",
        "        box2_y2 = boxes_labels[..., 1:2] + boxes_labels[..., 3:4] / 2\n",
        "\n",
        "    if box_format == \"corners\":\n",
        "        box1_x1 = boxes_preds[..., 0:1]\n",
        "        box1_y1 = boxes_preds[..., 1:2]\n",
        "        box1_x2 = boxes_preds[..., 2:3]\n",
        "        box1_y2 = boxes_preds[..., 3:4]  # (N, 1)\n",
        "        box2_x1 = boxes_labels[..., 0:1]\n",
        "        box2_y1 = boxes_labels[..., 1:2]\n",
        "        box2_x2 = boxes_labels[..., 2:3]\n",
        "        box2_y2 = boxes_labels[..., 3:4]\n",
        "\n",
        "    x1 = torch.max(box1_x1, box2_x1)\n",
        "    y1 = torch.max(box1_y1, box2_y1)\n",
        "    x2 = torch.min(box1_x2, box2_x2)\n",
        "    y2 = torch.min(box1_y2, box2_y2)\n",
        "\n",
        "    # .clamp(0) is for the case when they do not intersect\n",
        "    intersection = (x2 - x1).clamp(0) * (y2 - y1).clamp(0)\n",
        "\n",
        "    box1_area = abs((box1_x2 - box1_x1) * (box1_y2 - box1_y1))\n",
        "    box2_area = abs((box2_x2 - box2_x1) * (box2_y2 - box2_y1))\n",
        "\n",
        "    return intersection / (box1_area + box2_area - intersection + 1e-6)\n",
        "\n",
        "\n",
        "def non_max_suppression(bboxes, iou_threshold, threshold, box_format=\"corners\"):\n",
        "    \"\"\"\n",
        "    Does Non Max Suppression given bboxes\n",
        "    Parameters:\n",
        "        bboxes (list): list of lists containing all bboxes with each bboxes\n",
        "        specified as [class_pred, prob_score, x1, y1, x2, y2]\n",
        "        iou_threshold (float): threshold where predicted bboxes is correct\n",
        "        threshold (float): threshold to remove predicted bboxes (independent of IoU) \n",
        "        box_format (str): \"midpoint\" or \"corners\" used to specify bboxes\n",
        "    Returns:\n",
        "        list: bboxes after performing NMS given a specific IoU threshold\n",
        "    \"\"\"\n",
        "\n",
        "    assert type(bboxes) == list\n",
        "\n",
        "    bboxes = [box for box in bboxes if box[1] > threshold]\n",
        "    bboxes = sorted(bboxes, key=lambda x: x[1], reverse=True)\n",
        "    bboxes_after_nms = []\n",
        "\n",
        "    while bboxes:\n",
        "        chosen_box = bboxes.pop(0)\n",
        "\n",
        "        bboxes = [\n",
        "            box\n",
        "            for box in bboxes\n",
        "            if box[0] != chosen_box[0]\n",
        "            or intersection_over_union(\n",
        "                torch.tensor(chosen_box[2:]),\n",
        "                torch.tensor(box[2:]),\n",
        "                box_format=box_format,\n",
        "            )\n",
        "            < iou_threshold\n",
        "        ]\n",
        "\n",
        "        bboxes_after_nms.append(chosen_box)\n",
        "\n",
        "    return bboxes_after_nms\n",
        "\n",
        "\n",
        "def mean_average_precision(\n",
        "    pred_boxes, true_boxes, iou_threshold=0.5, box_format=\"midpoint\", num_classes=20\n",
        "):\n",
        "    \"\"\"\n",
        "    Calculates mean average precision \n",
        "    Parameters:\n",
        "        pred_boxes (list): list of lists containing all bboxes with each bboxes\n",
        "        specified as [train_idx, class_prediction, prob_score, x1, y1, x2, y2]\n",
        "        true_boxes (list): Similar as pred_boxes except all the correct ones \n",
        "        iou_threshold (float): threshold where predicted bboxes is correct\n",
        "        box_format (str): \"midpoint\" or \"corners\" used to specify bboxes\n",
        "        num_classes (int): number of classes\n",
        "    Returns:\n",
        "        float: mAP value across all classes given a specific IoU threshold \n",
        "    \"\"\"\n",
        "\n",
        "    # list storing all AP for respective classes\n",
        "    average_precisions = []\n",
        "\n",
        "    # used for numerical stability later on\n",
        "    epsilon = 1e-6\n",
        "\n",
        "    for c in range(num_classes):\n",
        "        detections = []\n",
        "        ground_truths = []\n",
        "\n",
        "        # Go through all predictions and targets,\n",
        "        # and only add the ones that belong to the\n",
        "        # current class c\n",
        "        for detection in pred_boxes:\n",
        "            if detection[1] == c:\n",
        "                detections.append(detection)\n",
        "\n",
        "        for true_box in true_boxes:\n",
        "            if true_box[1] == c:\n",
        "                ground_truths.append(true_box)\n",
        "\n",
        "        # find the amount of bboxes for each training example\n",
        "        # Counter here finds how many ground truth bboxes we get\n",
        "        # for each training example, so let's say img 0 has 3,\n",
        "        # img 1 has 5 then we will obtain a dictionary with:\n",
        "        # amount_bboxes = {0:3, 1:5}\n",
        "        amount_bboxes = Counter([gt[0] for gt in ground_truths])\n",
        "\n",
        "        # We then go through each key, val in this dictionary\n",
        "        # and convert to the following (w.r.t same example):\n",
        "        # ammount_bboxes = {0:torch.tensor[0,0,0], 1:torch.tensor[0,0,0,0,0]}\n",
        "        for key, val in amount_bboxes.items():\n",
        "            amount_bboxes[key] = torch.zeros(val)\n",
        "\n",
        "        # sort by box probabilities which is index 2\n",
        "        detections.sort(key=lambda x: x[2], reverse=True)\n",
        "        TP = torch.zeros((len(detections)))\n",
        "        FP = torch.zeros((len(detections)))\n",
        "        total_true_bboxes = len(ground_truths)\n",
        "        \n",
        "        # If none exists for this class then we can safely skip\n",
        "        if total_true_bboxes == 0:\n",
        "            continue\n",
        "\n",
        "        for detection_idx, detection in enumerate(detections):\n",
        "            # Only take out the ground_truths that have the same\n",
        "            # training idx as detection\n",
        "            ground_truth_img = [\n",
        "                bbox for bbox in ground_truths if bbox[0] == detection[0]\n",
        "            ]\n",
        "\n",
        "            num_gts = len(ground_truth_img)\n",
        "            best_iou = 0\n",
        "\n",
        "            for idx, gt in enumerate(ground_truth_img):\n",
        "                iou = intersection_over_union(\n",
        "                    torch.tensor(detection[3:]),\n",
        "                    torch.tensor(gt[3:]),\n",
        "                    box_format=box_format,\n",
        "                )\n",
        "\n",
        "                if iou > best_iou:\n",
        "                    best_iou = iou\n",
        "                    best_gt_idx = idx\n",
        "\n",
        "            if best_iou > iou_threshold:\n",
        "                # only detect ground truth detection once\n",
        "                if amount_bboxes[detection[0]][best_gt_idx] == 0:\n",
        "                    # true positive and add this bounding box to seen\n",
        "                    TP[detection_idx] = 1\n",
        "                    amount_bboxes[detection[0]][best_gt_idx] = 1\n",
        "                else:\n",
        "                    FP[detection_idx] = 1\n",
        "\n",
        "            # if IOU is lower then the detection is a false positive\n",
        "            else:\n",
        "                FP[detection_idx] = 1\n",
        "\n",
        "        TP_cumsum = torch.cumsum(TP, dim=0)\n",
        "        FP_cumsum = torch.cumsum(FP, dim=0)\n",
        "        recalls = TP_cumsum / (total_true_bboxes + epsilon)\n",
        "        precisions = torch.divide(TP_cumsum, (TP_cumsum + FP_cumsum + epsilon))\n",
        "        precisions = torch.cat((torch.tensor([1]), precisions))\n",
        "        recalls = torch.cat((torch.tensor([0]), recalls))\n",
        "        # torch.trapz for numerical integration\n",
        "        average_precisions.append(torch.trapz(precisions, recalls))\n",
        "\n",
        "    return sum(average_precisions) / len(average_precisions)\n",
        "\n",
        "\n",
        "def plot_image(image, boxes):\n",
        "    \"\"\"Plots predicted bounding boxes on the image\"\"\"\n",
        "    im = np.array(image)\n",
        "    height, width, _ = im.shape\n",
        "\n",
        "    # Create figure and axes\n",
        "    fig, ax = plt.subplots(1)\n",
        "    # Display the image\n",
        "    ax.imshow(im)\n",
        "\n",
        "    # box[0] is x midpoint, box[2] is width\n",
        "    # box[1] is y midpoint, box[3] is height\n",
        "\n",
        "    # Create a Rectangle potch\n",
        "    for box in boxes:\n",
        "        box = box[2:]\n",
        "        assert len(box) == 4, \"Got more values than in x, y, w, h, in a box!\"\n",
        "        upper_left_x = box[0] - box[2] / 2\n",
        "        upper_left_y = box[1] - box[3] / 2\n",
        "        rect = patches.Rectangle(\n",
        "            (upper_left_x * width, upper_left_y * height),\n",
        "            box[2] * width,\n",
        "            box[3] * height,\n",
        "            linewidth=1,\n",
        "            edgecolor=\"r\",\n",
        "            facecolor=\"none\",\n",
        "        )\n",
        "        # Add the patch to the Axes\n",
        "        ax.add_patch(rect)\n",
        "\n",
        "    plt.show()\n",
        "\n",
        "def get_bboxes(\n",
        "    loader,\n",
        "    model,\n",
        "    iou_threshold,\n",
        "    threshold,\n",
        "    pred_format=\"cells\",\n",
        "    box_format=\"midpoint\",\n",
        "    device=\"cuda\",\n",
        "):\n",
        "    all_pred_boxes = []\n",
        "    all_true_boxes = []\n",
        "\n",
        "    # make sure model is in eval before get bboxes\n",
        "    model.eval()\n",
        "    train_idx = 0\n",
        "\n",
        "    for batch_idx, (x, labels) in enumerate(loader):\n",
        "        x = x.to(device)\n",
        "        labels = labels.to(device)\n",
        "\n",
        "        with torch.no_grad():\n",
        "            predictions = model(x)\n",
        "\n",
        "        batch_size = x.shape[0]\n",
        "        true_bboxes = cellboxes_to_boxes(labels)\n",
        "        bboxes = cellboxes_to_boxes(predictions)\n",
        "\n",
        "        for idx in range(batch_size):\n",
        "            nms_boxes = non_max_suppression(\n",
        "                bboxes[idx],\n",
        "                iou_threshold=iou_threshold,\n",
        "                threshold=threshold,\n",
        "                box_format=box_format,\n",
        "            )\n",
        "\n",
        "\n",
        "            #if batch_idx == 0 and idx == 0:\n",
        "            #    plot_image(x[idx].permute(1,2,0).to(\"cpu\"), nms_boxes)\n",
        "            #    print(nms_boxes)\n",
        "\n",
        "            for nms_box in nms_boxes:\n",
        "                all_pred_boxes.append([train_idx] + nms_box)\n",
        "\n",
        "            for box in true_bboxes[idx]:\n",
        "                # many will get converted to 0 pred\n",
        "                if box[1] > threshold:\n",
        "                    all_true_boxes.append([train_idx] + box)\n",
        "\n",
        "            train_idx += 1\n",
        "\n",
        "    model.train()\n",
        "    return all_pred_boxes, all_true_boxes\n",
        "\n",
        "\n",
        "\n",
        "def convert_cellboxes(predictions, S=7):\n",
        "    \"\"\"\n",
        "    Converts bounding boxes output from Yolo with\n",
        "    an image split size of S into entire image ratios\n",
        "    rather than relative to cell ratios. Tried to do this\n",
        "    vectorized, but this resulted in quite difficult to read\n",
        "    code... Use as a black box? Or implement a more intuitive,\n",
        "    using 2 for loops iterating range(S) and convert them one\n",
        "    by one, resulting in a slower but more readable implementation.\n",
        "    \"\"\"\n",
        "\n",
        "    predictions = predictions.to(\"cpu\")\n",
        "    batch_size = predictions.shape[0]\n",
        "    predictions = predictions.reshape(batch_size, 7, 7, 30)\n",
        "    bboxes1 = predictions[..., 21:25]\n",
        "    bboxes2 = predictions[..., 26:30]\n",
        "    scores = torch.cat(\n",
        "        (predictions[..., 20].unsqueeze(0), predictions[..., 25].unsqueeze(0)), dim=0\n",
        "    )\n",
        "    best_box = scores.argmax(0).unsqueeze(-1)\n",
        "    best_boxes = bboxes1 * (1 - best_box) + best_box * bboxes2\n",
        "    cell_indices = torch.arange(7).repeat(batch_size, 7, 1).unsqueeze(-1)\n",
        "    x = 1 / S * (best_boxes[..., :1] + cell_indices)\n",
        "    y = 1 / S * (best_boxes[..., 1:2] + cell_indices.permute(0, 2, 1, 3))\n",
        "    w_y = 1 / S * best_boxes[..., 2:4]\n",
        "    converted_bboxes = torch.cat((x, y, w_y), dim=-1)\n",
        "    predicted_class = predictions[..., :20].argmax(-1).unsqueeze(-1)\n",
        "    best_confidence = torch.max(predictions[..., 20], predictions[..., 25]).unsqueeze(\n",
        "        -1\n",
        "    )\n",
        "    converted_preds = torch.cat(\n",
        "        (predicted_class, best_confidence, converted_bboxes), dim=-1\n",
        "    )\n",
        "\n",
        "    return converted_preds\n",
        "\n",
        "\n",
        "def cellboxes_to_boxes(out, S=7):\n",
        "    converted_pred = convert_cellboxes(out).reshape(out.shape[0], S * S, -1)\n",
        "    converted_pred[..., 0] = converted_pred[..., 0].long()\n",
        "    all_bboxes = []\n",
        "\n",
        "    for ex_idx in range(out.shape[0]):\n",
        "        bboxes = []\n",
        "\n",
        "        for bbox_idx in range(S * S):\n",
        "            bboxes.append([x.item() for x in converted_pred[ex_idx, bbox_idx, :]])\n",
        "        all_bboxes.append(bboxes)\n",
        "\n",
        "    return all_bboxes\n",
        "\n",
        "def save_checkpoint(state, filename=\"my_checkpoint.pth.tar\"):\n",
        "    print(\"=> Saving checkpoint\")\n",
        "    torch.save(state, filename)\n",
        "\n",
        "\n",
        "def load_checkpoint(checkpoint, model, optimizer):\n",
        "    print(\"=> Loading checkpoint\")\n",
        "    model.load_state_dict(checkpoint[\"state_dict\"])\n",
        "    optimizer.load_state_dict(checkpoint[\"optimizer\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RPkWBAJP47f1"
      },
      "source": [
        "## Creating the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5kIdWca6uuMv"
      },
      "source": [
        "# define the output layer of YOLO\n",
        "\n",
        "class Yolo_Reshape(tf.keras.layers.Layer):\n",
        "  def __init__(self, target_shape):\n",
        "    super(Yolo_Reshape, self).__init__()\n",
        "    self.target_shape = tuple(target_shape)\n",
        "\n",
        "  def get_config(self):\n",
        "    config = super().get_config().copy()\n",
        "    config.update({\n",
        "        'target_shape': self.target_shape\n",
        "    })\n",
        "    return config\n",
        "\n",
        "  def call(self, input):\n",
        "    # grids 7x7\n",
        "    S = [self.target_shape[0], self.target_shape[1]]\n",
        "    # classes\n",
        "    C = number_of_classes\n",
        "    # no of bounding boxes per grid\n",
        "    B = 2\n",
        "\n",
        "    idx1 = S[0] * S[1] * C\n",
        "    idx2 = idx1 + S[0] * S[1] * B\n",
        "    \n",
        "    # class probabilities\n",
        "    class_probs = K.reshape(input[:, :idx1], (K.shape(input)[0],) + tuple([S[0], S[1], C]))\n",
        "    class_probs = K.softmax(class_probs)\n",
        "\n",
        "    #confidence\n",
        "    confs = K.reshape(input[:, idx1:idx2], (K.shape(input)[0],) + tuple([S[0], S[1], B]))\n",
        "    confs = K.sigmoid(confs)\n",
        "\n",
        "    # boxes\n",
        "    boxes = K.reshape(input[:, idx2:], (K.shape(input)[0],) + tuple([S[0], S[1], B * 4]))\n",
        "    boxes = K.sigmoid(boxes)\n",
        "\n",
        "    outputs = K.concatenate([class_probs, confs, boxes])\n",
        "    return outputs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f2a1zuKu4_ZR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b2ce9a60-d8d3-49c7-abd7-910d6c6c9343"
      },
      "source": [
        "# Simple DNN\n",
        "# just two conolution layers followed by dense layer\n",
        "def getSimpleDNNModel(input_shape,number_of_pixels,number_of_classes):\n",
        "    nF        = 16 # number of filters\n",
        "    inputs = keras.Input(shape=input_shape, name=\"images\") \n",
        "    # Create CNN model\n",
        "    x11  = layers.Conv2D(nF, (3, 3), activation='relu', input_shape=input_shape) (inputs)\n",
        "    x13  = layers.MaxPooling2D((2, 2)) (x11)\n",
        "    x21  = layers.Conv2D(2*nF, (3, 3), activation='relu') (x13)\n",
        "    x23  = layers.MaxPooling2D((2, 2))(x21)\n",
        "    #dense layer for classification\n",
        "    x31 = layers.Flatten()(x23)# convert from 3d to 1d\n",
        "    outputs = layers.Dense(number_of_classes, name=\"predictions\")(x31)\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    return model\n",
        "\n",
        "# this is a better model for CIFAR10\n",
        "def getDNNModel(input_shape,number_of_pixels,number_of_classes):\n",
        "    nF        = 64 # number of filters\n",
        "    inputs = keras.Input(shape=input_shape, name=\"images\") \n",
        "    # Create CNN model\n",
        "    x11  = layers.Conv2D(nF, (3, 3), activation='relu', input_shape=input_shape) (inputs)\n",
        "    x12  = layers.BatchNormalization()(x11)\n",
        "    x13  = layers.MaxPooling2D((2, 2)) (x12)\n",
        "    x14  = layers.Dropout(0.25)(x13)\n",
        "    x21  = layers.Conv2D(2*nF, (3, 3), activation='relu') (x14)\n",
        "    x22  = layers.BatchNormalization()(x21)\n",
        "    x23  = layers.MaxPooling2D((2, 2))(x22)\n",
        "    x24  = layers.Dropout(0.25)(x23)\n",
        "    x31  = layers.Conv2D(2*nF, (3, 3), activation='relu')(x24)\n",
        "    #dense layer for classification\n",
        "    x41 = layers.Flatten()(x31)# convert from 3d to 1d\n",
        "    #x7 = layers.Dense(2*nF, activation='relu')(x6)\n",
        "    #x8 = layers.Dense(2*nF, activation='relu')(x7)\n",
        "    x42  = layers.Dropout(0.50)(x41)\n",
        "    outputs = layers.Dense(number_of_classes, name=\"predictions\")(x42)\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    return model\n",
        "\n",
        "def getSimpleDNN3DModel(input_shape,number_of_pixels,number_of_classes):\n",
        "    nF        = 16 # number of filters\n",
        "    inputs = keras.Input(shape=input_shape, name=\"images\") \n",
        "    # Create CNN model\n",
        "    x11  = layers.Conv3D(nF, (3, 3, 3), activation='relu', input_shape=input_shape) (inputs)\n",
        "    x13  = layers.MaxPooling3D((2, 2, 2)) (x11)\n",
        "    x21  = layers.Conv3D(2*nF, (3, 3, 3), activation='relu') (x13)\n",
        "    x23  = layers.MaxPooling3D((2, 2 ,2))(x21)\n",
        "    #dense layer for classification\n",
        "    x31 = layers.Flatten()(x23)# convert from 3d to 1d\n",
        "    outputs = layers.Dense(number_of_classes, name=\"predictions\")(x31)\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    return model\n",
        "\n",
        "#========================================\n",
        "#              Yolo  \n",
        "#========================================\n",
        "\n",
        "def getYoloDNNModel(input_shape,number_of_pixels,number_of_classes):\n",
        "    # input_shape = [batch_size, width, height, num of channels]\n",
        "    lrelu = tf.keras.layers.LeakyReLU(alpha=0.1)\n",
        "    nF        = 16 # number of filters\n",
        "    lrelu = layers.LeakyReLU(alpha=0.1)\n",
        "    inputs = keras.Input(shape=input_shape, name=\"images\")\n",
        "    # Create Yolo model\n",
        "\n",
        "    x011  = layers.Conv2D(nF*4, (7, 7), 2, padding = 'same', activation=lrelu, input_shape=input_shape) (inputs)\n",
        "    x012  = layers.MaxPooling2D((2, 2), 2, padding = 'same') (x011)\n",
        "   \n",
        "    x021  = layers.Conv2D(nF*12, (3, 3), 1, padding = 'same', activation=lrelu) (x012)\n",
        "    x022  = layers.MaxPooling2D((2, 2), 2, padding = 'same') (x021)\n",
        "    \n",
        "    x030  = layers.Conv2D(nF*8, (1, 1), 1, padding = 'same', activation=lrelu)  (x022)\n",
        "    x040  = layers.Conv2D(nF*16, (3, 3), 1, padding = 'same', activation=lrelu) (x030)\n",
        "    x050  = layers.Conv2D(nF*16, (1, 1), 1, padding = 'same', activation=lrelu) (x040)\n",
        "    \n",
        "    x061  = layers.Conv2D(nF*32, (3, 3), 1, padding = 'same', activation=lrelu) (x050)\n",
        "    x062  = layers.MaxPooling2D((2, 2), padding = 'same') (x061)\n",
        "    \n",
        "    x070  = layers.Conv2D(nF*16, (1, 1), 1, padding = 'same', activation=lrelu) (x062)\n",
        "    x080  = layers.Conv2D(nF*32, (3, 3), 1, padding = 'same', activation=lrelu) (x070)\n",
        "    x090  = layers.Conv2D(nF*16, (1, 1), 1, padding = 'same', activation=lrelu) (x080)\n",
        "    x100  = layers.Conv2D(nF*32, (3, 3), 1, padding = 'same', activation=lrelu) (x090)\n",
        "    x110  = layers.Conv2D(nF*16, (1, 1), 1, padding = 'same', activation=lrelu) (x100)\n",
        "    x120  = layers.Conv2D(nF*32, (3, 3), 1, padding = 'same', activation=lrelu) (x110)\n",
        "    x130  = layers.Conv2D(nF*16, (1, 1), 1, padding = 'same', activation=lrelu) (x120)\n",
        "    x140  = layers.Conv2D(nF*32, (3, 3), 1, padding = 'same', activation=lrelu) (x130)\n",
        "    x150  = layers.Conv2D(nF*32, (1, 1), 1, padding = 'same', activation=lrelu) (x140)\n",
        "\n",
        "    x161  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x150)\n",
        "    x162  = layers.MaxPooling2D((2, 2), 2, padding = 'same') (x161)\n",
        "\n",
        "    x170  = layers.Conv2D(nF*32, (1, 1), 1, padding = 'same', activation=lrelu) (x162)\n",
        "    x180  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x170)\n",
        "    x190  = layers.Conv2D(nF*32, (1, 1), 1, padding = 'same', activation=lrelu) (x180)\n",
        "    x200  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x190)\n",
        "    x210  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x200)\n",
        "    x220  = layers.Conv2D(nF*64, (3, 3), 2, padding = 'same', activation=lrelu) (x210)\n",
        "    x230  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x220)\n",
        "    x240  = layers.Conv2D(nF*64, (3, 3), 1, padding = 'same', activation=lrelu) (x230)\n",
        "\n",
        "\n",
        "    # x011  = layers.Conv2D(filters=64, kernel_size= (7, 7), strides=(1, 1), input_shape=input_shape, padding = 'same', activation=lrelu) (inputs)\n",
        "    # x012  = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding = 'same') (x011)\n",
        "\n",
        "    # x021  = layers.Conv2D(filters=192, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x012)\n",
        "    # x022  = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding = 'same') (x021)\n",
        "\n",
        "    # x030  = layers.Conv2D(filters=128, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x022)\n",
        "    # x040  = layers.Conv2D(filters=256, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x030)\n",
        "    # x050  = layers.Conv2D(filters=256, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x040)\n",
        "    # x061  = layers.Conv2D(filters=512, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x050)\n",
        "    # x062  = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding = 'same') (x061)\n",
        "\n",
        "    # x070  = layers.Conv2D(filters=256, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x062)\n",
        "    # x080  = layers.Conv2D(filters=512, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x070)\n",
        "    # x090  = layers.Conv2D(filters=256, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x080)\n",
        "    # x100  = layers.Conv2D(filters=512, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x090)\n",
        "    # x110  = layers.Conv2D(filters=256, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x100)\n",
        "    # x120  = layers.Conv2D(filters=512, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x110)\n",
        "    # x130  = layers.Conv2D(filters=256, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x120)\n",
        "    # x140  = layers.Conv2D(filters=512, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x130)\n",
        "    # x150  = layers.Conv2D(filters=512, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x140)\n",
        "    # x161  = layers.Conv2D(filters=1024, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x150)\n",
        "    # x162  = layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding = 'same') (x161)\n",
        "\n",
        "    # x170  = layers.Conv2D(filters=512, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x162)\n",
        "    # x180  = layers.Conv2D(filters=1024, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x170)\n",
        "    # x190  = layers.Conv2D(filters=512, kernel_size= (1, 1), padding = 'same', activation=lrelu) (x180)\n",
        "    # x200  = layers.Conv2D(filters=1024, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x190)\n",
        "    # x210  = layers.Conv2D(filters=1024, kernel_size= (3, 3), padding = 'same', activation=lrelu) (x200)\n",
        "    # x220  = layers.Conv2D(filters=1024, kernel_size= (3, 3), strides=(2, 2), padding = 'same') (x210)\n",
        "\n",
        "    # x230  = layers.Conv2D(filters=1024, kernel_size= (3, 3), activation=lrelu) (x220)\n",
        "    # x240  = layers.Conv2D(filters=1024, kernel_size= (3, 3), activation=lrelu) (x230)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    x250   = layers.Flatten()(x240)# convert from 3d to 1d\n",
        "    x260   = layers.Dense(512)(x250)\n",
        "    x270   = layers.Dense(1024)(x260)\n",
        "    x280 = layers.Dense(1470, activation='sigmoid')(x270)\n",
        "    outputs = Yolo_Reshape(target_shape=(7,7,30))(x280)\n",
        "\n",
        "\n",
        "    #dense layer for classification\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    # print(model.summary())\n",
        "\n",
        "    return model    \n",
        "\n",
        "\n",
        "print(\"DNN model is defined ...\")    \n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DNN model is defined ...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pptczs0E8G0v"
      },
      "source": [
        "# TODO\n",
        "# Pytorch\n",
        "\"\"\" \n",
        "Information about architecture config:\n",
        "Tuple is structured by (kernel_size, filters, stride, padding) \n",
        "\"M\" is simply maxpooling with stride 2x2 and kernel 2x2\n",
        "List is structured by tuples and lastly int with number of repeats\n",
        "\"\"\"\n",
        "\n",
        "architecture_config = [\n",
        "    (7, 64, 2, 3),\n",
        "    \"M\",\n",
        "    (3, 192, 1, 1),\n",
        "    \"M\",\n",
        "    (1, 128, 1, 0),\n",
        "    (3, 256, 1, 1),\n",
        "    (1, 256, 1, 0),\n",
        "    (3, 512, 1, 1),\n",
        "    \"M\",\n",
        "    [(1, 256, 1, 0), (3, 512, 1, 1), 4],\n",
        "    (1, 512, 1, 0),\n",
        "    (3, 1024, 1, 1),\n",
        "    \"M\",\n",
        "    [(1, 512, 1, 0), (3, 1024, 1, 1), 2],\n",
        "    (3, 1024, 1, 1),\n",
        "    (3, 1024, 2, 1),\n",
        "    (3, 1024, 1, 1),\n",
        "    (3, 1024, 1, 1),\n",
        "]\n",
        "\n",
        "\n",
        "class CNNBlock(nn.Module):\n",
        "    def __init__(self, in_channels, out_channels, **kwargs):\n",
        "        super(CNNBlock, self).__init__()\n",
        "        self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)\n",
        "        self.batchnorm = nn.BatchNorm2d(out_channels)\n",
        "        self.leakyrelu = nn.LeakyReLU(0.1)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.leakyrelu(self.batchnorm(self.conv(x)))\n",
        "\n",
        "\n",
        "class Yolov1(nn.Module):\n",
        "    def __init__(self, in_channels=3, **kwargs):\n",
        "        super(Yolov1, self).__init__()\n",
        "        self.architecture = architecture_config\n",
        "        self.in_channels = in_channels\n",
        "        self.darknet = self._create_conv_layers(self.architecture)\n",
        "        self.fcs = self._create_fcs(**kwargs)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.darknet(x)\n",
        "        return self.fcs(torch.flatten(x, start_dim=1))\n",
        "\n",
        "    def _create_conv_layers(self, architecture):\n",
        "        layers = []\n",
        "        in_channels = self.in_channels\n",
        "\n",
        "        for x in architecture:\n",
        "            if type(x) == tuple:\n",
        "                layers += [\n",
        "                    CNNBlock(\n",
        "                        in_channels, x[1], kernel_size=x[0], stride=x[2], padding=x[3],\n",
        "                    )\n",
        "                ]\n",
        "                in_channels = x[1]\n",
        "\n",
        "            elif type(x) == str:\n",
        "                layers += [nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2))]\n",
        "\n",
        "            elif type(x) == list:\n",
        "                conv1 = x[0]\n",
        "                conv2 = x[1]\n",
        "                num_repeats = x[2]\n",
        "\n",
        "                for _ in range(num_repeats):\n",
        "                    layers += [\n",
        "                        CNNBlock(\n",
        "                            in_channels,\n",
        "                            conv1[1],\n",
        "                            kernel_size=conv1[0],\n",
        "                            stride=conv1[2],\n",
        "                            padding=conv1[3],\n",
        "                        )\n",
        "                    ]\n",
        "                    layers += [\n",
        "                        CNNBlock(\n",
        "                            conv1[1],\n",
        "                            conv2[1],\n",
        "                            kernel_size=conv2[0],\n",
        "                            stride=conv2[2],\n",
        "                            padding=conv2[3],\n",
        "                        )\n",
        "                    ]\n",
        "                    in_channels = conv2[1]\n",
        "\n",
        "        return nn.Sequential(*layers)\n",
        "\n",
        "    def _create_fcs(self, split_size, num_boxes, num_classes):\n",
        "        S, B, C = split_size, num_boxes, num_classes\n",
        "\n",
        "        # In original paper this should be\n",
        "        # nn.Linear(1024*S*S, 4096),\n",
        "        # nn.LeakyReLU(0.1),\n",
        "        # nn.Linear(4096, S*S*(B*5+C))\n",
        "\n",
        "        return nn.Sequential(\n",
        "            nn.Flatten(),\n",
        "            nn.Linear(1024 * S * S, 496),\n",
        "            nn.Dropout(0.0),\n",
        "            nn.LeakyReLU(0.1),\n",
        "            nn.Linear(496, S * S * (C + B * 5)),\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oD_BI-0C4_kr"
      },
      "source": [
        "## Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fKpCUhIT5EWp"
      },
      "source": [
        "if not usePytorch:\n",
        "    epochs = 10 # number of iterations\n",
        "\n",
        "    if NNID>=2:\n",
        "      # input_shape = [h,w,c] \n",
        "\n",
        "      if NNID==2:\n",
        "        model = getSimpleDNNModel(input_shape, number_of_pixels,number_of_classes)\n",
        "      elif NNID==3: # advanced \n",
        "        model = getDNNModel(input_shape, number_of_pixels,number_of_classes)\n",
        "      elif NNID==4: # 3D \n",
        "        input_shape = [h,w,c,1] \n",
        "        model = getSimpleDNN3DModel(input_shape, number_of_pixels,number_of_classes)\n",
        "      elif NNID==5: # Yolo\n",
        "        grid_w=7\n",
        "        grid_h=7\n",
        "        cell_w=64\n",
        "        cell_h=64\n",
        "        img_w=grid_w*cell_w\n",
        "        img_h=grid_h*cell_h\n",
        "        channel=3\n",
        "        input_shape = [img_w,img_h,channel]\n",
        "        number_of_classes = 20\n",
        "        number_of_pixels = img_w * img_h * channel\n",
        "        model = getYoloDNNModel(input_shape, number_of_pixels, number_of_classes)\n",
        "        \n",
        "\n",
        "        print(\"===================================================\")\n",
        "        print(\"               Training Loop           \")\n",
        "        print(\"===================================================\")\n",
        "        total_time_start = time.time()\n",
        "        # we loop number of iterations\n",
        "        # for each iteration, we loop through all the training samples\n",
        "        for epoch in range(epochs):\n",
        "            #print(\"\\nStart of epoch %d\" % (epoch,))\n",
        "            start_time = time.time()\n",
        "\n",
        "            # Iterate over the batches of the dataset.\n",
        "            train_dataset = My_Custom_Generator(X_train, Y_train, batch_size)\n",
        "            for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
        "                #print(train_dataset.shape)\n",
        "                #print(x_batch_train.shape,y_batch_train.shape)\n",
        "                doAug = 0\n",
        "                if doAug: \n",
        "                    #do augmentation\n",
        "                    new_train_batch = doAugmentation(x_batch_train , y_batch_train , batch_size)                \n",
        "                    for stp, (new_x_batch_train, new_y_batch_train) in enumerate(new_train_batch):\n",
        "                        #print(stp)\n",
        "                        #print(new_train_batch.shape)\n",
        "                        #print(new_x_batch_train.shape,new_y_batch_train.shape)\n",
        "                        #model.summary()\n",
        "                        loss_value = train_step(model,new_x_batch_train, new_y_batch_train)\n",
        "                        train_acc = train_acc_metric.result()\n",
        "                        train_acc_metric.reset_states()\n",
        "                        print(\"   epoch:%d \\t stp %d trnLoss: %.4f \" % (epoch, stp, float(loss_value)))\n",
        "                else:                    \n",
        "                    # print(step)\n",
        "                    # print(x_batch_train.shape)\n",
        "                    # print(y_batch_train.shape)\n",
        "                    # model.summary()\n",
        "                    lossFunctionID = 4\n",
        "                    loss_value = train_step(model,x_batch_train, y_batch_train)\n",
        "                    train_acc = train_acc_metric.result()\n",
        "                    train_acc_metric.reset_states()\n",
        "\n",
        "            # Run a validation loop at the end of each epoch.\n",
        "            val_dataset = My_Custom_Generator(X_val, Y_val, batch_size)\n",
        "            for x_batch_val, y_batch_val in val_dataset:\n",
        "                val_loss_value = val_step(model, x_batch_val, y_batch_val)\n",
        "\n",
        "            val_acc = val_acc_metric.result()\n",
        "            val_acc_metric.reset_states()\n",
        "            \n",
        "            # compute time required for each epoch\n",
        "            end_time = time.time() - start_time\n",
        "\n",
        "            print(\"epoch:%d \\t trnLoss: %.4f \\t valLoss: %.4f \\t trnAcc: %.4f \\t valAcc: %.4f \\t time:  %.2f\" % (epoch, float(loss_value),float(val_loss_value), float(train_acc), float(val_acc), end_time))\n",
        "            logFile = open(logFilePath,'a')\n",
        "            logFile.write(\"%d \\t %.4f \\t  %.4f \\t %.4f \\t  %.4f \\t  %.2f \\n\" % (epoch, float(loss_value),float(val_loss_value), float(train_acc), float(val_acc), end_time))\n",
        "            logFile.close()\n",
        "            if epoch % 5 ==0:\n",
        "            # plot the result        \n",
        "              iaPlotLoss(logFilePath)\n",
        "              model.save(modelPath)      \n",
        "        # save the final model\n",
        "        model.save(modelPath)     \n",
        "\n",
        "        # plot the result        \n",
        "        iaPlotLoss(logFilePath)\n",
        "        total_time_end = time.time() - total_time_start\n",
        "        print(\"Training this dataset took \", total_time_end,\" seconds!\") \n",
        "        print(\"Training this dataset took \", total_time_end/60.0,\" minutes!\") \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IK-U7PBvXGif",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 529
        },
        "outputId": "38f27295-13c6-420c-e9ed-9c689d1d8240"
      },
      "source": [
        "# py \n",
        "if usePytorch:\n",
        "  if NNID == 5:\n",
        "    seed = 123\n",
        "    torch.manual_seed(seed)\n",
        "\n",
        "    # Hyperparameters etc. \n",
        "    LEARNING_RATE = 2e-5\n",
        "    DEVICE = \"cuda\" if torch.cuda.is_available else \"cpu\"\n",
        "    BATCH_SIZE = 1 # 64 in original paper but I don't have that much vram, grad accum?\n",
        "    WEIGHT_DECAY = 0\n",
        "    EPOCHS = 300\n",
        "    NUM_WORKERS = 2\n",
        "    PIN_MEMORY = True\n",
        "    LOAD_MODEL = False\n",
        "    LOAD_MODEL_FILE = \"overfit.pth.tar\"\n",
        "    IMG_DIR = \"data/images\"\n",
        "    LABEL_DIR = \"data/labels\"\n",
        "\n",
        "    class Compose(object):\n",
        "      def __init__(self, transforms):\n",
        "          self.transforms = transforms\n",
        "\n",
        "      def __call__(self, img, bboxes):\n",
        "          for t in self.transforms:\n",
        "              img, bboxes = t(img), bboxes\n",
        "\n",
        "          return img, bboxes\n",
        "\n",
        "\n",
        "    transform = Compose([transforms.Resize((448, 448)), transforms.ToTensor(),])\n",
        "\n",
        "\n",
        "    def train_fn(train_loader, model, optimizer, loss_fn):\n",
        "        loop = tqdm(train_loader, leave=True)\n",
        "        mean_loss = []\n",
        "\n",
        "        for batch_idx, (x, y) in enumerate(loop):\n",
        "            x, y = x.to(DEVICE), y.to(DEVICE)\n",
        "            out = model(x)\n",
        "            loss = loss_fn(out, y)\n",
        "            mean_loss.append(loss.item())\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "            # update progress bar\n",
        "            loop.set_postfix(loss=loss.item())\n",
        "\n",
        "        print(f\"Mean loss was {sum(mean_loss)/len(mean_loss)}\")\n",
        "\n",
        "\n",
        "    model = Yolov1(split_size=7, num_boxes=2, num_classes=20).to(DEVICE)\n",
        "    optimizer = optim.Adam(\n",
        "        model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY\n",
        "    )\n",
        "    loss_fn = YoloLoss()\n",
        "\n",
        "    if LOAD_MODEL:\n",
        "        load_checkpoint(torch.load(LOAD_MODEL_FILE), model, optimizer)\n",
        "\n",
        "    train_dataset = VOCDataset(\n",
        "        \"train.csv\",\n",
        "        transform=transform,\n",
        "        img_dir=IMG_DIR,\n",
        "        label_dir=LABEL_DIR,\n",
        "    )\n",
        "\n",
        "    test_dataset = VOCDataset(\n",
        "        \"test.csv\", transform=transform, img_dir=IMG_DIR, label_dir=LABEL_DIR,\n",
        "    )\n",
        "\n",
        "    train_loader = DataLoader(\n",
        "        dataset=train_dataset,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        num_workers=NUM_WORKERS,\n",
        "        pin_memory=PIN_MEMORY,\n",
        "        shuffle=True,\n",
        "        drop_last=True,\n",
        "    )\n",
        "\n",
        "    test_loader = DataLoader(\n",
        "        dataset=test_dataset,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        num_workers=NUM_WORKERS,\n",
        "        pin_memory=PIN_MEMORY,\n",
        "        shuffle=True,\n",
        "        drop_last=True,\n",
        "    )\n",
        "\n",
        "    for epoch in range(EPOCHS):\n",
        "        # for x, y in train_loader:\n",
        "        #    x = x.to(DEVICE)\n",
        "        #    for idx in range(8):\n",
        "        #        bboxes = cellboxes_to_boxes(model(x))\n",
        "        #        bboxes = non_max_suppression(bboxes[idx], iou_threshold=0.5, threshold=0.4, box_format=\"midpoint\")\n",
        "        #        plot_image(x[idx].permute(1,2,0).to(\"cpu\"), bboxes)\n",
        "\n",
        "        #    import sys\n",
        "        #    sys.exit()\n",
        "\n",
        "        pred_boxes, target_boxes = get_bboxes(\n",
        "            train_loader, model, iou_threshold=0.5, threshold=0.4\n",
        "        )\n",
        "\n",
        "        mean_avg_prec = mean_average_precision(\n",
        "            pred_boxes, target_boxes, iou_threshold=0.5, box_format=\"midpoint\"\n",
        "        )\n",
        "        # print(\"epoch:%d \\t trnLoss: %.4f \\t valLoss: %.4f \\t trnAcc: %.4f \\t valAcc: %.4f \\t time:  %.2f\" % (epoch, float(loss_value),float(val_loss_value), float(train_acc), float(val_acc), end_time))\n",
        "        print(f\"Epoch: {epoch} Train mAP: {mean_avg_prec}\")\n",
        "\n",
        "        #if mean_avg_prec > 0.9:\n",
        "        #    checkpoint = {\n",
        "        #        \"state_dict\": model.state_dict(),\n",
        "        #        \"optimizer\": optimizer.state_dict(),\n",
        "        #    }\n",
        "        #    save_checkpoint(checkpoint, filename=LOAD_MODEL_FILE)\n",
        "        #    import time\n",
        "        #    time.sleep(10)\n",
        "\n",
        "        train_fn(train_loader, model, optimizer, loss_fn)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)\n",
            "  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 0 Train mAP: 0.0\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 16550/16550 [15:43<00:00, 17.54it/s, loss=5.31]"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mean loss was 13.303371761479047\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 1 Train mAP: 6.750193279003724e-05\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 16550/16550 [15:55<00:00, 17.32it/s, loss=7.64]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean loss was 11.759896752603824\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 2 Train mAP: 0.00020331663836259395\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 13%|█▎        | 2195/16550 [02:10<14:10, 16.87it/s, loss=14.7]\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-18-c72351ce9148>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m    119\u001b[0m         \u001b[0;31m#    time.sleep(10)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 121\u001b[0;31m         \u001b[0mtrain_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_loader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-18-c72351ce9148>\u001b[0m in \u001b[0;36mtrain_fn\u001b[0;34m(train_loader, model, optimizer, loss_fn)\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mbatch_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloop\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m             \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDEVICE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDEVICE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloss_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m             \u001b[0mmean_loss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-16-46fd1ba225d7>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 52\u001b[0;31m         \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdarknet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     53\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfcs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflatten\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_dim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    137\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    138\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 139\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    140\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-16-46fd1ba225d7>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     38\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mleakyrelu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatchnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    441\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    442\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 443\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_conv_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    444\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    445\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0mConv3d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_ConvNd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py\u001b[0m in \u001b[0;36m_conv_forward\u001b[0;34m(self, input, weight, bias)\u001b[0m\n\u001b[1;32m    438\u001b[0m                             _pair(0), self.dilation, self.groups)\n\u001b[1;32m    439\u001b[0m         return F.conv2d(input, weight, bias, self.stride,\n\u001b[0;32m--> 440\u001b[0;31m                         self.padding, self.dilation, self.groups)\n\u001b[0m\u001b[1;32m    441\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    442\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "deIAKC2E5HcT"
      },
      "source": [
        "## Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iwUVTMeg5Ks5"
      },
      "source": [
        "if NNID==2:\n",
        "    # Load the saved model \n",
        "    model = keras.models.load_model(modelPath, compile=False)\n",
        "\n",
        "    start_time = time.time() \n",
        "    # Run a validation loop at the end of each epoch.\n",
        "    for x_batch_tst, y_batch_tst in tst_dataset:\n",
        "        output = model.predict(x_batch_tst)\n",
        "        #y = keras.utils.to_categorical(y_batch_tst)\n",
        "        tst_acc_metric.update_state(y_batch_tst, output)\n",
        "\n",
        "    tst_acc = tst_acc_metric.result()\n",
        "\n",
        "    # compute time required for each epoch\n",
        "    end_time = time.time() - start_time\n",
        "\n",
        "    print(\"test accuracy : %.4f \\t time:  %.2f\" % (  float(tst_acc), end_time))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j-YaiYcJP028"
      },
      "source": [
        "#TODO: show examples using the above datasets\n",
        "# https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jeRdRNJF2kwU"
      },
      "source": [
        "# More resources:\n",
        "\n",
        "* 3Blue1Brown Neural Network [video tutorials](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi) \n",
        "* Deep Learning Video Lectures by Prof. Andreas Maier [Winter 20/21](https://www.youtube.com/watch?v=SCFToE1vM2U&list=PLpOGQvPCDQzvJEPFUQ3mJz72GJ95jyZTh)\n",
        "* Some of the code in this notebook is taken from [here](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch)\n",
        "* Calculating number of parameters in [CNN](https://towardsdatascience.com/understanding-and-calculating-the-number-of-parameters-in-convolution-neural-networks-cnns-fc88790d530d)\n",
        "* Some of the code in this notebook is taken from [here](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/cnn.ipynb)\n",
        "* https://nihcc.app.box.com/v/ChestXray-NIHCC\n",
        "* https://www.tensorflow.org/datasets/catalog/patch_camelyon\n",
        "* https://danielnouri.org/notes/2014/12/17/using-convolutional-neural-nets-to-detect-facial-keypoints-tutorial/\n",
        "* https://elix-tech.github.io/ja/2016/06/02/kaggle-facial-keypoints-ja.html\n",
        "* https://fairyonice.github.io/achieving-top-23-in-kaggles-facial-keypoints-detection-with-keras-tensorflow.html\n",
        "* https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html\n",
        "* https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/object_detection.ipynb\n",
        "* https://github.com/nicknochnack/TFODCourse\n",
        "* https://www.analyticsvidhya.com/blog/2018/10/a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1/\n",
        "* https://github.com/enggen/Deep-Learning-Coursera\n",
        "* https://github.com/prateeshreddy/Deep-Learning-Coursera\n",
        "* https://github.com/JudasDie/deeplearning.ai\n",
        "* https://github.com/aladdinpersson/Machine-Learning-Collection/tree/master/ML/Pytorch/object_detection/YOLO\n",
        "* https://youtu.be/n9_XyCGr-MI\n",
        "* https://www.maskaravivek.com/post/yolov1/"
      ]
    }
  ]
}