Darth_Vader_Training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "Darth Vader - Training.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/thecognifly/AIYVisionKit_Utils/blob/master/Darth_Vader_Training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFGkmfIQ0u-q"
      },
      "source": [
        "# Using transfer learning to detect my Darth Vader clock\n",
        "\n",
        "So, in this notebook I show how I trained an object detector for the [Google AYI Vision Kit](https://aiyprojects.withgoogle.com/vision). This hardware is already old, but I like it because it is a hat for the RPI Zero W that, in addition to the neural net accelerator it has PWM and ADC, and it doesn't use a lot of power (less than 2W in my tests while running the object detector). Moreover, it weighs less than 30g! However, its age brings some problems because Tensorflow kept evolving since they launched the AIY Vision Kit, so I could only train the object detection model that fits the kit using Tensorflow 1.15.2. After training, it will be necessary to export (freeze) the model before you can compile (convert) it for the Google AIY Bonnet, but that step only works on Tensorflow 1.14, lol!\n",
        "\n",
        "BTW, I'm not trying to train a super-duper Darth Vader detector (actually, the latest checkpoints are overfitting because the dataset I collected doesn't have a lot of diversity and I didn't stop training when I should...), these notebooks are only simple step-by-step instructions to allow you (and myself in the future) to train different object detectors that are capable of running on the Google AIY Vision Kit.\n",
        "\n",
        "In fact, this model is tiny, so it may be useful for running somewhere else... maybe directly in a RPI?!?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "esok7HPu7EDa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3ee5074d-f62b-46de-ebbf-546be38946b1"
      },
      "source": [
        "!pip install git+git://github.com/ricardodeazambuja/colab_utils.git --upgrade"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting git+git://github.com/ricardodeazambuja/colab_utils.git\n",
            "  Cloning git://github.com/ricardodeazambuja/colab_utils.git to /tmp/pip-req-build-0gy74x8m\n",
            "  Running command git clone -q git://github.com/ricardodeazambuja/colab_utils.git /tmp/pip-req-build-0gy74x8m\n",
            "Collecting ffmpeg-python\n",
            "  Downloading https://files.pythonhosted.org/packages/d7/0c/56be52741f75bad4dc6555991fabd2e07b432d333da82c11ad701123888a/ffmpeg_python-0.2.0-py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from colab-utils==0.2) (1.4.1)\n",
            "Requirement already satisfied, skipping upgrade: pillow in /usr/local/lib/python3.6/dist-packages (from colab-utils==0.2) (7.0.0)\n",
            "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from colab-utils==0.2) (1.19.4)\n",
            "Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from ffmpeg-python->colab-utils==0.2) (0.16.0)\n",
            "Building wheels for collected packages: colab-utils\n",
            "  Building wheel for colab-utils (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for colab-utils: filename=colab_utils-0.2-cp36-none-any.whl size=19461 sha256=3392de27fb1f102358be039901e1f105cffbc9538072f1258a90a9134ccf1443\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-_rq62zeb/wheels/21/75/32/38aeb76b2424385f43eae9fd28c98e084308f4f6d9cb0a4f97\n",
            "Successfully built colab-utils\n",
            "Installing collected packages: ffmpeg-python, colab-utils\n",
            "Successfully installed colab-utils-0.2 ffmpeg-python-0.2.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TKclUf8k7EUr"
      },
      "source": [
        "from colab_utils import copy2clipboard, webcam2numpy, labelImage, imshow"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h0WqKct01fQW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa8ff28e-b77b-410b-9f39-d2cf88ec3957"
      },
      "source": [
        "import os\n",
        "\n",
        "# The lines below will allow me to easily switch between colab and local notebooks\n",
        "MYNOTEBOOKPATH = '/content/' # inside the notebook\n",
        "os.environ['MYNOTEBOOKPATH'] = MYNOTEBOOKPATH # in the shell commands\n",
        "\n",
        "!echo \"MYNOTEBOOKPATH=\"$MYNOTEBOOKPATH\n",
        "print(f\"MYNOTEBOOKPATH={MYNOTEBOOKPATH}\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MYNOTEBOOKPATH=/content/\n",
            "MYNOTEBOOKPATH=/content/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8a21B89v1AXN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c245fa2-c1b2-46fd-f7f1-bd50e0540599"
      },
      "source": [
        "from IPython import get_ipython\n",
        "\n",
        "is_colab = False\n",
        "\n",
        "try:\n",
        "  #%tensorflow_version 1.x\n",
        "  get_ipython().magic('tensorflow_version 1.x')\n",
        "  is_colab = True\n",
        "except:\n",
        "  print(\"not in colab...\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 1.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCz0xuhTtO7n"
      },
      "source": [
        "If the current version available within collab doesn't work anymore, just install the 1.15.2. First, uninstall the one installed:  \n",
        "`!pip uninstall tensorflow -y`  \n",
        "After that, if things haven't changed, you can install using pip:  \n",
        "`!pip install tensorflow==1.15.2 > /dev/null`  \n",
        "One problem: the pip one will not be optimized and that is bad for training...\n",
        "\n",
        "Luckly, when I was creating this notebook, the current version was 1.15.2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3H2tguxb1GhN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e42cafe4-00a9-4189-fc52-c1f6889e302e"
      },
      "source": [
        "!python -c \"import tensorflow as tf; print(tf.__version__)\" # python version for the shell commands\n",
        "# FutureWarning seems to be a Numpy problem when using tf < 1.15: https://stackoverflow.com/a/58546467"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.15.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6bcOIVSY0u-w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f35d0f2a-f49d-4f92-d08a-0ae8115aa96f"
      },
      "source": [
        "!uname -a\n",
        "!lsb_release -a\n",
        "!nvidia-smi"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Linux 9d86d16cbafc 4.19.112+ #1 SMP Thu Jul 23 08:00:38 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux\n",
            "No LSB modules are available.\n",
            "Distributor ID:\tUbuntu\n",
            "Description:\tUbuntu 18.04.5 LTS\n",
            "Release:\t18.04\n",
            "Codename:\tbionic\n",
            "Mon Dec 21 13:41:29 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 455.45.01    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   43C    P8     9W /  70W |      0MiB / 15079MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QYaogEya0u-3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f1e87bc-dc7e-4693-e007-6deff62aca79"
      },
      "source": [
        "!cd $MYNOTEBOOKPATH && git clone https://github.com/cocodataset/cocoapi.git"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'cocoapi'...\n",
            "remote: Enumerating objects: 975, done.\u001b[K\n",
            "remote: Total 975 (delta 0), reused 0 (delta 0), pack-reused 975\u001b[K\n",
            "Receiving objects: 100% (975/975), 11.72 MiB | 32.54 MiB/s, done.\n",
            "Resolving deltas: 100% (575/575), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vd0O0dckn-kp"
      },
      "source": [
        "The cell below will prints lots of warnings, but don't worry... they will not bite you back later ;)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "dCr4LNoN0u-6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c423fd91-8cea-40b8-8ad1-12e890bd3f28"
      },
      "source": [
        "!cd $MYNOTEBOOKPATH/cocoapi/PythonAPI && make > /dev/null #redirects stdout to the null hole, but stderr will show..."
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /content/cocoapi/PythonAPI/pycocotools/_mask.pyx\n",
            "  tree = Parsing.p_module(s, pxd, full_module_name)\n",
            "\u001b[01m\u001b[K../common/maskApi.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KrleDecode\u001b[m\u001b[K’:\n",
            "\u001b[01m\u001b[K../common/maskApi.c:46:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "       \u001b[01;35m\u001b[Kfor\u001b[m\u001b[K( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}\n",
            "       \u001b[01;35m\u001b[K^~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:46:49:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’\n",
            "       for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; \u001b[01;36m\u001b[Kv\u001b[m\u001b[K=!v; }}\n",
            "                                                 \u001b[01;36m\u001b[K^\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KrleFrPoly\u001b[m\u001b[K’:\n",
            "\u001b[01m\u001b[K../common/maskApi.c:166:3:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "   \u001b[01;35m\u001b[Kfor\u001b[m\u001b[K(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];\n",
            "   \u001b[01;35m\u001b[K^~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:166:54:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’\n",
            "   for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); \u001b[01;36m\u001b[Kx\u001b[m\u001b[K[k]=x[0];\n",
            "                                                      \u001b[01;36m\u001b[K^\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:167:3:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "   \u001b[01;35m\u001b[Kfor\u001b[m\u001b[K(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];\n",
            "   \u001b[01;35m\u001b[K^~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:167:54:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kfor\u001b[m\u001b[K’\n",
            "   for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); \u001b[01;36m\u001b[Ky\u001b[m\u001b[K[k]=y[0];\n",
            "                                                      \u001b[01;36m\u001b[K^\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KrleToString\u001b[m\u001b[K’:\n",
            "\u001b[01m\u001b[K../common/maskApi.c:212:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "       \u001b[01;35m\u001b[Kif\u001b[m\u001b[K(more) c |= 0x20; c+=48; s[p++]=c;\n",
            "       \u001b[01;35m\u001b[K^~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:212:27:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’\n",
            "       if(more) c |= 0x20; \u001b[01;36m\u001b[Kc\u001b[m\u001b[K+=48; s[p++]=c;\n",
            "                           \u001b[01;36m\u001b[K^\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KrleFrString\u001b[m\u001b[K’:\n",
            "\u001b[01m\u001b[K../common/maskApi.c:220:3:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kwhile\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "   \u001b[01;35m\u001b[Kwhile\u001b[m\u001b[K( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;\n",
            "   \u001b[01;35m\u001b[K^~~~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:220:22:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kwhile\u001b[m\u001b[K’\n",
            "   while( s[m] ) m++; \u001b[01;36m\u001b[Kcnts\u001b[m\u001b[K=malloc(sizeof(uint)*m); m=0;\n",
            "                      \u001b[01;36m\u001b[K^~~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:228:5:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n",
            "     \u001b[01;35m\u001b[Kif\u001b[m\u001b[K(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;\n",
            "     \u001b[01;35m\u001b[K^~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:228:34:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’\n",
            "     if(m>2) x+=(long) cnts[m-2]; \u001b[01;36m\u001b[Kcnts\u001b[m\u001b[K[m++]=(uint) x;\n",
            "                                  \u001b[01;36m\u001b[K^~~~\u001b[m\u001b[K\n",
            "\u001b[01m\u001b[K../common/maskApi.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KrleToBbox\u001b[m\u001b[K’:\n",
            "\u001b[01m\u001b[K../common/maskApi.c:141:31:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kxp\u001b[m\u001b[K’ may be used uninitialized in this function [\u001b[01;35m\u001b[K-Wmaybe-uninitialized\u001b[m\u001b[K]\n",
            "       if(j%2==0) xp=x; else if\u001b[01;35m\u001b[K(\u001b[m\u001b[Kxp<x) { ys=0; ye=h-1; }\n",
            "                               \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NCMRffdsoHkr"
      },
      "source": [
        "**Important Step: I only tested it using my modified version of the TensorFlow Object Detection API. I fixed some silly bugs, but I have no idea whether or not the official latest version will work. So, we need to clone my fork!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sonbmHut0u-9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c7c2071-dc55-43dc-8627-bf3e504f5cf7"
      },
      "source": [
        "!cd $MYNOTEBOOKPATH && git clone https://github.com/ricardodeazambuja/models.git && cd $MYNOTEBOOKPATH/models && git checkout laptop_rtx"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'models'...\n",
            "remote: Enumerating objects: 34169, done.\u001b[K\n",
            "remote: Total 34169 (delta 0), reused 0 (delta 0), pack-reused 34169\u001b[K\n",
            "Receiving objects: 100% (34169/34169), 518.62 MiB | 37.47 MiB/s, done.\n",
            "Resolving deltas: 100% (22328/22328), done.\n",
            "Checking out files: 100% (2502/2502), done.\n",
            "Branch 'laptop_rtx' set up to track remote branch 'laptop_rtx' from 'origin'.\n",
            "Switched to a new branch 'laptop_rtx'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GZDjtR4c0u_B"
      },
      "source": [
        "!cp -r $MYNOTEBOOKPATH/cocoapi/PythonAPI/pycocotools $MYNOTEBOOKPATH/models/research/"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1GkA29H0u_E"
      },
      "source": [
        "os.chdir(f\"{MYNOTEBOOKPATH}models/research/\") # for changing the current working directory everywhere\n",
        "                                              # the normal cd (without !) would do the same job... but it doesn't accept variables"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rDkxp_61i32k",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d8ee24ab-6c74-4f32-b4e4-d8d21cbc9729"
      },
      "source": [
        "# yup, I keep checking things...\n",
        "pwd"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'/content/models/research'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jqpx8mAo0u_H"
      },
      "source": [
        "!protoc object_detection/protos/*.proto --python_out=."
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SUKrpwsK0u_K",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "76c3e452-c29a-4723-83c2-fc3ea9854a41"
      },
      "source": [
        "!echo $PYTHONPATH"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/tensorflow-1.15.2/python3.6:/env/python\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NAaOVJgDBEC9"
      },
      "source": [
        "If you change environmental variables using something like `!export MYVAR = 'something'` the changes will only affect that shell and all the commands will need to be executed in tandem, e.g.: `!export MYVAR = 'something' && echo $MYVAR`.  \n",
        "However, there are some caveats. `PYTHONPATH` needs to be set **BEFORE** the interpreter is started, but in a notebook the interpreter must be started to use the notebooks itself, so the changes made to the `PYTHONPATH` will not be seen by the interpreter. A solution in this situation is to change the system path instead."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x3xryDsT5BYn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b116895d-39f1-4b14-bbf3-fc2ffbf60049"
      },
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "RESEARCHPATH = os.path.join(MYNOTEBOOKPATH,\"models/research\")\n",
        "if RESEARCHPATH not in os.environ['PYTHONPATH']:\n",
        "  print(f\"Changing the $PYTHONPATH from:\\n{os.environ['PYTHONPATH']}\")\n",
        "  os.environ['PYTHONPATH'] = os.environ['PYTHONPATH'] + \":\" + RESEARCHPATH + \":\" + os.path.join(RESEARCHPATH, \"slim\")\n",
        "  print(f\"To:\\n{os.environ['PYTHONPATH']}\")\n",
        "\n",
        "if RESEARCHPATH not in \"\".join(sys.path):\n",
        "  print(f\"Changing the system path from:\\n{sys.path}\")\n",
        "  sys.path.append(RESEARCHPATH)\n",
        "  sys.path.append(os.path.join(RESEARCHPATH, \"slim\"))\n",
        "  print(f\"To:\\n{sys.path}\")"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Changing the $PYTHONPATH from:\n",
            "/tensorflow-1.15.2/python3.6:/env/python\n",
            "To:\n",
            "/tensorflow-1.15.2/python3.6:/env/python:/content/models/research:/content/models/research/slim\n",
            "Changing the system path from:\n",
            "['/tensorflow-1.15.2/python3.6', '', '/env/python', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.6/dist-packages/IPython/extensions', '/root/.ipython']\n",
            "To:\n",
            "['/tensorflow-1.15.2/python3.6', '', '/env/python', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.6/dist-packages/IPython/extensions', '/root/.ipython', '/content/models/research', '/content/models/research/slim']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FwPFCx7oosfw"
      },
      "source": [
        "The cell below, if it works as it should (that means, you didn't forget to execute anything above...), it will give you 'OK' at the end."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pNgcIyiz0u_Z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c35f4769-0c68-4767-fc8d-f83175d95e29"
      },
      "source": [
        "!python $MYNOTEBOOKPATH/models/research/object_detection/builders/model_builder_test.py "
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "Running tests under Python 3.6.9: /usr/bin/python3\n",
            "[ RUN      ] ModelBuilderTest.test_create_experimental_model\n",
            "[       OK ] ModelBuilderTest.test_create_experimental_model\n",
            "[ RUN      ] ModelBuilderTest.test_create_faster_rcnn_model_from_config_with_example_miner\n",
            "[       OK ] ModelBuilderTest.test_create_faster_rcnn_model_from_config_with_example_miner\n",
            "[ RUN      ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
            "[       OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
            "[ RUN      ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
            "[       OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
            "[ RUN      ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
            "[       OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
            "[ RUN      ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
            "[       OK ] ModelBuilderTest.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
            "[ RUN      ] ModelBuilderTest.test_create_rfcn_model_from_config\n",
            "[       OK ] ModelBuilderTest.test_create_rfcn_model_from_config\n",
            "[ RUN      ] ModelBuilderTest.test_create_ssd_fpn_model_from_config\n",
            "[       OK ] ModelBuilderTest.test_create_ssd_fpn_model_from_config\n",
            "[ RUN      ] ModelBuilderTest.test_create_ssd_models_from_config\n",
            "[       OK ] ModelBuilderTest.test_create_ssd_models_from_config\n",
            "[ RUN      ] ModelBuilderTest.test_invalid_faster_rcnn_batchnorm_update\n",
            "[       OK ] ModelBuilderTest.test_invalid_faster_rcnn_batchnorm_update\n",
            "[ RUN      ] ModelBuilderTest.test_invalid_first_stage_nms_iou_threshold\n",
            "[       OK ] ModelBuilderTest.test_invalid_first_stage_nms_iou_threshold\n",
            "[ RUN      ] ModelBuilderTest.test_invalid_model_config_proto\n",
            "[       OK ] ModelBuilderTest.test_invalid_model_config_proto\n",
            "[ RUN      ] ModelBuilderTest.test_invalid_second_stage_batch_size\n",
            "[       OK ] ModelBuilderTest.test_invalid_second_stage_batch_size\n",
            "[ RUN      ] ModelBuilderTest.test_session\n",
            "[  SKIPPED ] ModelBuilderTest.test_session\n",
            "[ RUN      ] ModelBuilderTest.test_unknown_faster_rcnn_feature_extractor\n",
            "[       OK ] ModelBuilderTest.test_unknown_faster_rcnn_feature_extractor\n",
            "[ RUN      ] ModelBuilderTest.test_unknown_meta_architecture\n",
            "[       OK ] ModelBuilderTest.test_unknown_meta_architecture\n",
            "[ RUN      ] ModelBuilderTest.test_unknown_ssd_feature_extractor\n",
            "[       OK ] ModelBuilderTest.test_unknown_ssd_feature_extractor\n",
            "----------------------------------------------------------------------\n",
            "Ran 17 tests in 0.163s\n",
            "\n",
            "OK (skipped=1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JSk-kGML_fY9"
      },
      "source": [
        "I strongly suggest you to have a look at this to learn more about the tfrecord: https://www.tensorflow.org/tutorials/load_data/tfrecord"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DLzxDA836n_C",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2ee69ec8-882c-4c0a-82ff-4304b2abc8b7"
      },
      "source": [
        "#\n",
        "# Just one \"tiny\" detail here: it will mount your gdrive, so if you mess with it\n",
        "# you may destroy other data you saved there ;)\n",
        "#\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0dLw_iQppEIR"
      },
      "source": [
        "I hope it's clear the dataset used here was created [converting the videos saved in the RPI](https://github.com/thecognifly/AIYVisionKit_Utils/blob/master/Darth_Vader_Convert_RPI_Videos_2_PNG_images.ipynb) and [annotating them](https://github.com/thecognifly/AIYVisionKit_Utils/blob/master/Darth_Vader_Annotate_Images.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Oe5kFnZBA5O"
      },
      "source": [
        "DATASETPATH = \"/content/drive/My Drive/Datasets/DarthVader/\""
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUDf4CMo0yL4"
      },
      "source": [
        "### Generate the TFRecords (this doesn't need to be repeated if it's already available...)\n",
        "\n",
        "The object detection API used expects the dataset to be following the Pascal VOC standards. And that's what we do in the next cells."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lUniwWs3DKP5"
      },
      "source": [
        "os.chdir(DATASETPATH)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ghQ4S5DlpPjP"
      },
      "source": [
        "import json\n",
        "\n",
        "filename = \"/content/drive/My Drive/Datasets/DarthVader/annotations.json\"\n",
        "with open(filename, 'r') as f:\n",
        "  read_annotations = json.load(f)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2r4osE7PrwrM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6058b8ba-2db2-4189-9709-487d85b0dd31"
      },
      "source": [
        "# As always, I will test if it worked...\n",
        "read_annotations['darthvader1.png'][0][1]"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'darthvader'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBZnrxftpUTh"
      },
      "source": [
        "from PIL import Image\n",
        "\n",
        "# !pip install natsort\n",
        "from natsort import natsorted # https://natsort.readthedocs.io/en/master/\n",
        "\n",
        "# \n",
        "# This step will convert the annotations to the format used by Pascal VOC\n",
        "#\n",
        "with open('darthvader_dataset.csv', 'w') as f:\n",
        "  f.write(\"filename,width,height,class,xmin,ymin,xmax,ymax\\n\")\n",
        "  for filename in natsorted(read_annotations.keys()):\n",
        "    img = Image.open(filename)\n",
        "    w,h = img.size\n",
        "    box = read_annotations[filename][0][0]\n",
        "    obj_class = read_annotations[filename][0][1]\n",
        "    xmin = int(min((box[0],box[0]+box[2]))*w)\n",
        "    ymin = int(min((box[1],box[1]+box[3]))*h)\n",
        "    xmax = int(max((box[0],box[0]+box[2]))*w)\n",
        "    ymax = int(max((box[1],box[1]+box[3]))*h)\n",
        "    f.write(\",\".join([filename,str(int(w)),str(int(h)),obj_class,str(xmin),str(ymin),str(xmax),str(ymax)])+'\\n')"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SYj9awxJte9U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb7c02fc-8e86-4c16-fdab-3d2f1e05e443"
      },
      "source": [
        "# testing stuff again...\n",
        "!head darthvader_dataset.csv"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "filename,width,height,class,xmin,ymin,xmax,ymax\n",
            "darthvader1.png,256,256,darthvader,119,147,246,229\n",
            "darthvader2.png,256,256,darthvader,107,153,240,237\n",
            "darthvader3.png,256,256,darthvader,122,184,255,255\n",
            "darthvader4.png,256,256,darthvader,106,134,250,252\n",
            "darthvader5.png,256,256,darthvader,62,87,246,255\n",
            "darthvader6.png,256,256,darthvader,67,147,198,253\n",
            "darthvader7.png,256,256,darthvader,56,72,195,249\n",
            "darthvader8.png,256,256,darthvader,87,71,230,249\n",
            "darthvader9.png,256,256,darthvader,119,79,239,241\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Nu3ywTfvD5I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e52ef8a2-6e4f-42c2-8aec-020f525b6dde"
      },
      "source": [
        "# Confirming the number of samples is correct by counting the lines (wc -l)\n",
        "!cat darthvader_dataset.csv | wc -l"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "95\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mXdAZ8C4vgpE"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "rdn = np.random.RandomState(seed=999) # to make sure the sets will not change \n",
        "                                      # when this cell is executed again\n",
        "\n",
        "#\n",
        "# Couldn't we simply use np.random.seed? Yes and no. \n",
        "# If you use np.random.seed with other libraries, the somewhere else\n",
        "# something may change the seed. The method above is bullet proof ;)\n",
        "#\n",
        "\n",
        "n_samples = 95-1\n",
        "indices = np.arange(n_samples)\n",
        "rdn.shuffle(indices)\n",
        "\n",
        "with open(\"darthvader_dataset.csv\", 'r') as f:\n",
        "  header = f.readline()\n",
        "  all_lines = f.readlines()\n",
        "  with open(\"darthvader_dataset_train.csv\", 'w') as ftrain:\n",
        "    ftrain.write(header)\n",
        "    with open(\"darthvader_dataset_test.csv\", 'w') as ftest:\n",
        "      ftest.write(header)\n",
        "      for li,line in enumerate(all_lines):\n",
        "        if li in indices[:int(n_samples*0.9)]:\n",
        "          ftrain.write(line)\n",
        "        else:\n",
        "          ftest.write(line)"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RzaQrruzGW9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9b61cf3-f7c3-43d2-840b-b920bb9ca4d2"
      },
      "source": [
        "# More testing... (remember, the first line is the header, so it's 10+84=>94)\n",
        "!wc -l darthvader_dataset_test.csv\n",
        "!wc -l darthvader_dataset_train.csv"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "11 darthvader_dataset_test.csv\n",
            "85 darthvader_dataset_train.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JRkprTnpqH47"
      },
      "source": [
        "It's always a good idea to have a look at the first lines at least..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OZY5CGxzxEgs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7d09fa90-7d7a-489b-d217-e121d3e34013"
      },
      "source": [
        "!head darthvader_dataset_test.csv"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "filename,width,height,class,xmin,ymin,xmax,ymax\n",
            "darthvader9.png,256,256,darthvader,119,79,239,241\n",
            "darthvader18.png,256,256,darthvader,106,60,175,174\n",
            "darthvader35.png,256,256,darthvader,83,154,155,255\n",
            "darthvader53.png,256,256,darthvader,86,183,212,255\n",
            "darthvader57.png,256,256,darthvader,81,139,198,253\n",
            "darthvader71.png,256,256,darthvader,51,122,190,253\n",
            "darthvader80.png,256,256,darthvader,108,62,222,236\n",
            "darthvader98.png,256,256,darthvader,16,27,173,254\n",
            "darthvader101.png,256,256,darthvader,89,26,246,252\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GXL8eT4KxHI6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "977e7fd2-b7c8-43d0-be67-7dfa49e42e36"
      },
      "source": [
        "!head darthvader_dataset_train.csv"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "filename,width,height,class,xmin,ymin,xmax,ymax\n",
            "darthvader1.png,256,256,darthvader,119,147,246,229\n",
            "darthvader2.png,256,256,darthvader,107,153,240,237\n",
            "darthvader3.png,256,256,darthvader,122,184,255,255\n",
            "darthvader4.png,256,256,darthvader,106,134,250,252\n",
            "darthvader5.png,256,256,darthvader,62,87,246,255\n",
            "darthvader6.png,256,256,darthvader,67,147,198,253\n",
            "darthvader7.png,256,256,darthvader,56,72,195,249\n",
            "darthvader8.png,256,256,darthvader,87,71,230,249\n",
            "darthvader10.png,256,256,darthvader,67,114,159,242\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gn7bVaQky06"
      },
      "source": [
        "By using the magic `%%file darthvader_object-detection.pbtxt`, the cell below will write its content to the file `darthvader_object-detection.pbtxt`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JJrru93V4ciR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd243fb1-2a38-4297-b35f-dd82dbd78204"
      },
      "source": [
        "%%file darthvader_object-detection.pbtxt\n",
        "item {\n",
        "  id: 1\n",
        "  name: 'darthvader'\n",
        "}"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting darthvader_object-detection.pbtxt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mrXoshJdk_GD",
        "outputId": "2567bf2c-7576-4530-86e1-6502885098f3"
      },
      "source": [
        "# If you don't believe me...\n",
        "!cat darthvader_object-detection.pbtxt"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "item {\n",
            "  id: 1\n",
            "  name: 'darthvader'\n",
            "}"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2i3yGPranEPx"
      },
      "source": [
        "Finally, we will create the TFRecords"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyZmjkgAPIia"
      },
      "source": [
        "#\n",
        "# Heavily inspired by (copied from) https://github.com/datitran/raccoon_dataset\n",
        "#\n",
        "import os\n",
        "import io\n",
        "import tensorflow as tf\n",
        "\n",
        "from PIL import Image\n",
        "from object_detection.utils import dataset_util\n",
        "\n",
        "def readCSV(filename, debug=False):\n",
        "  \"\"\"Read a CSV file and return a bunch of dictionaries\"\"\"\n",
        "  with open(filename, 'r') as f:\n",
        "    header = f.readline().strip('\\n').split(',')\n",
        "    if debug: print(header)\n",
        "    samples = []\n",
        "\n",
        "    for l in f.readlines():\n",
        "      if l:\n",
        "        current_sample = l.strip('\\n').split(',')\n",
        "        if debug: print(current_sample)\n",
        "        samples.append(dict(zip(header, current_sample)))\n",
        "  return samples\n",
        "\n",
        "def getClassesIds(filename):\n",
        "  \"\"\"Read a pbtxt file to return a dictionary with classes / ids\"\"\"\n",
        "  ids = []\n",
        "  classes = []\n",
        "  with open(filename, 'r') as f:\n",
        "    for l in f.readlines():\n",
        "      if 'id' in l:\n",
        "        ids.append(int(l.strip('\\n').split(':')[-1]))\n",
        "      if 'name' in l:\n",
        "        classes.append(''.join(c for c in l.split(':')[-1].strip('\\n') if c not in \" '\"))\n",
        "  return dict(zip(classes,ids))\n",
        "\n",
        "\n",
        "def create_tf_example(sample, path, class_id):\n",
        "  with tf.io.gfile.GFile(os.path.join(path, f\"{sample['filename']}\"), 'rb') as fid:\n",
        "    encoded_jpg = fid.read()\n",
        "  image = Image.open(io.BytesIO(encoded_jpg))\n",
        "  width, height = image.size\n",
        "\n",
        "  filename = sample['filename'].encode('utf8')\n",
        "  image_format = image.format.lower().encode('utf-8')\n",
        "  xmins = [float(sample['xmin']) / width]\n",
        "  xmaxs = [float(sample['xmax']) / width]\n",
        "  ymins = [float(sample['ymin']) / height]\n",
        "  ymaxs = [float(sample['ymax']) / height]\n",
        "  classes_text = [sample['class'].encode('utf8')]\n",
        "  classes = [class_id[sample['class']]]\n",
        "\n",
        "  tf_example = tf.train.Example(features=tf.train.Features(feature={\n",
        "    'image/height': dataset_util.int64_feature(height),\n",
        "    'image/width': dataset_util.int64_feature(width),\n",
        "    'image/filename': dataset_util.bytes_feature(filename),\n",
        "    'image/source_id': dataset_util.bytes_feature(filename),\n",
        "    'image/encoded': dataset_util.bytes_feature(encoded_jpg),\n",
        "    'image/format': dataset_util.bytes_feature(image_format),\n",
        "    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),\n",
        "    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),\n",
        "    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),\n",
        "    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),\n",
        "    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),\n",
        "    'image/object/class/label': dataset_util.int64_list_feature(classes),\n",
        "  }))\n",
        "  return tf_example\n",
        "\n",
        "\n",
        "def generateTFRecordFiles(OUTPUT_PATH, IMG_PATH, CSV_PATH, PBTXT_PATH):\n",
        "  class_id = getClassesIds(PBTXT_PATH)\n",
        "  with tf.python_io.TFRecordWriter(OUTPUT_PATH) as writer:\n",
        "    path = os.path.join(IMG_PATH)\n",
        "    samples = readCSV(CSV_PATH)\n",
        "    for sample in samples:\n",
        "      tf_example = create_tf_example(sample, path, class_id)\n",
        "      writer.write(tf_example.SerializeToString())\n",
        "\n",
        "  print(f'Successfully created the TFRecords: {OUTPUT_PATH}')"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L1yQyoo70u_k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "99d6107c-cd91-4624-c0b4-0fe51a62755e"
      },
      "source": [
        "IMG_PATH = DATASETPATH\n",
        "PBTXT_PATH = os.path.join(DATASETPATH,\"darthvader_object-detection.pbtxt\")\n",
        "\n",
        "OUTPUT_PATH = os.path.join(DATASETPATH,\"test.record\")\n",
        "CSV_PATH = os.path.join(DATASETPATH,\"darthvader_dataset_test.csv\")\n",
        "generateTFRecordFiles(OUTPUT_PATH, IMG_PATH, CSV_PATH, PBTXT_PATH)\n",
        "\n",
        "OUTPUT_PATH = os.path.join(DATASETPATH,\"train.record\")\n",
        "CSV_PATH = os.path.join(DATASETPATH,\"darthvader_dataset_train.csv\")\n",
        "generateTFRecordFiles(OUTPUT_PATH, IMG_PATH, CSV_PATH, PBTXT_PATH)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Successfully created the TFRecords: /content/drive/My Drive/Datasets/DarthVader/test.record\n",
            "Successfully created the TFRecords: /content/drive/My Drive/Datasets/DarthVader/train.record\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUXFkZVZqZ8c"
      },
      "source": [
        "Now that our data is inside those two TFRecords files, it's necessary to set the config file for the Object Detection API:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vkxNhUeM0u_p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0affbdd8-12e8-499d-aad6-b6d3d2d6f0b8"
      },
      "source": [
        "%%file ssd_mobilenetv1_256x256_AIY_VOC2012.config\n",
        "\n",
        "# Nice explanations can be found here:\n",
        "# https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html\n",
        "#\n",
        "# Or directly from the source:\n",
        "# https://github.com/tensorflow/models/blob/master/research/object_detection/utils/config_util.py\n",
        "#\n",
        "\n",
        "# Embedded SSD with Mobilenet v1 configuration for MSCOCO Dataset.\n",
        "# Users should configure the fine_tune_checkpoint field in the train config as\n",
        "# well as the label_map_path and input_path fields in the train_input_reader and\n",
        "# eval_input_reader. Search for \"PATH_TO_BE_CONFIGURED\" to find the fields that\n",
        "# should be configured.\n",
        "\n",
        "model {\n",
        "  ssd {\n",
        "    inplace_batchnorm_update: true\n",
        "    freeze_batchnorm: false\n",
        "    num_classes: 1 # if you have more classes (I have only Darth Vader clock...)\n",
        "    box_coder {\n",
        "      faster_rcnn_box_coder {\n",
        "        y_scale: 10.0\n",
        "        x_scale: 10.0\n",
        "        height_scale: 5.0\n",
        "        width_scale: 5.0\n",
        "      }\n",
        "    }\n",
        "    matcher {\n",
        "      argmax_matcher {\n",
        "        matched_threshold: 0.5\n",
        "        unmatched_threshold: 0.5\n",
        "        ignore_thresholds: false\n",
        "        negatives_lower_than_unmatched: true\n",
        "        force_match_for_each_row: true\n",
        "        use_matmul_gather: true\n",
        "      }\n",
        "    }\n",
        "    similarity_calculator {\n",
        "      iou_similarity {\n",
        "      }\n",
        "    }\n",
        "    anchor_generator {\n",
        "      ssd_anchor_generator {\n",
        "        num_layers: 5\n",
        "        min_scale: 0.2\n",
        "        max_scale: 0.95\n",
        "        aspect_ratios: 1.0\n",
        "        aspect_ratios: 2.0\n",
        "        aspect_ratios: 0.5\n",
        "        aspect_ratios: 3.0\n",
        "        aspect_ratios: 0.3333\n",
        "      }\n",
        "    }\n",
        "    # This model already has a image resizer (256x256) embedded\n",
        "    image_resizer {\n",
        "      fixed_shape_resizer {\n",
        "        height: 256\n",
        "        width: 256\n",
        "      }\n",
        "    }\n",
        "    box_predictor {\n",
        "      convolutional_box_predictor {\n",
        "        min_depth: 0\n",
        "        max_depth: 0\n",
        "        num_layers_before_predictor: 0\n",
        "        use_dropout: false\n",
        "        dropout_keep_probability: 0.8\n",
        "        kernel_size: 1\n",
        "        box_code_size: 4\n",
        "        apply_sigmoid_to_scores: false\n",
        "        conv_hyperparams {\n",
        "          activation: RELU_6,\n",
        "          regularizer {\n",
        "            l2_regularizer {\n",
        "              weight: 0.00004\n",
        "            }\n",
        "          }\n",
        "          initializer {\n",
        "            truncated_normal_initializer {\n",
        "              stddev: 0.03\n",
        "              mean: 0.0\n",
        "            }\n",
        "          }\n",
        "          batch_norm {\n",
        "            train: true,\n",
        "            scale: true,\n",
        "            center: true,\n",
        "            decay: 0.9997,\n",
        "            epsilon: 0.001,\n",
        "          }\n",
        "        }\n",
        "      }\n",
        "    }\n",
        "    feature_extractor {\n",
        "      type: 'embedded_ssd_mobilenet_v1' # the embedded version is the only one that worked for me\n",
        "      min_depth: 16\n",
        "      depth_multiplier: 0.125 # anything bigger than this will be too big for the AIY Vision Kit\n",
        "      override_base_feature_extractor_hyperparams: true\n",
        "      conv_hyperparams {\n",
        "        activation: RELU_6,\n",
        "        regularizer {\n",
        "          l2_regularizer {\n",
        "            weight: 0.00004\n",
        "          }\n",
        "        }\n",
        "        initializer {\n",
        "          truncated_normal_initializer {\n",
        "            stddev: 0.03\n",
        "            mean: 0.0\n",
        "          }\n",
        "        }\n",
        "        batch_norm {\n",
        "          train: true,\n",
        "          scale: true,\n",
        "          center: true,\n",
        "          decay: 0.9997,\n",
        "          epsilon: 0.001,\n",
        "        }\n",
        "      }\n",
        "    }\n",
        "    loss {\n",
        "      classification_loss {\n",
        "        weighted_sigmoid {\n",
        "        }\n",
        "      }\n",
        "      localization_loss {\n",
        "        weighted_smooth_l1 {\n",
        "        }\n",
        "      }\n",
        "      hard_example_miner {\n",
        "        num_hard_examples: 3000\n",
        "        iou_threshold: 0.99\n",
        "        loss_type: CLASSIFICATION\n",
        "        max_negatives_per_positive: 3\n",
        "        min_negatives_per_image: 0\n",
        "      }\n",
        "      classification_weight: 1.0\n",
        "      localization_weight: 1.0\n",
        "    }\n",
        "    normalize_loss_by_num_matches: true\n",
        "    normalize_loc_loss_by_codesize: true\n",
        "    post_processing {\n",
        "      batch_non_max_suppression {\n",
        "        score_threshold: 1e-8\n",
        "        iou_threshold: 0.6\n",
        "        max_detections_per_class: 3 # changed from 100 to 3\n",
        "        max_total_detections: 3 # changed from 100 to 3\n",
        "      }\n",
        "      score_converter: SIGMOID\n",
        "    }\n",
        "  }\n",
        "}\n",
        "train_config: {\n",
        "  batch_size: 32 # this number allowed me to train without memory problems. If you are having problems, make it smaller\n",
        "  optimizer {\n",
        "    rms_prop_optimizer: {\n",
        "      learning_rate: {\n",
        "        exponential_decay_learning_rate {\n",
        "          initial_learning_rate: 0.004\n",
        "          decay_steps: 800720\n",
        "          decay_factor: 0.95\n",
        "        }\n",
        "      }\n",
        "      momentum_optimizer_value: 0.9\n",
        "      decay: 0.9\n",
        "      epsilon: 1.0\n",
        "    }\n",
        "  }\n",
        "  fine_tune_checkpoint_type: 'detection'\n",
        "  # The directory below needs to match your own system. This one reflects a directory Checkpoints inside my google drive\n",
        "  # where there's another directory called VOC2012_Trained with the checkpoint\n",
        "  fine_tune_checkpoint: \"/content/drive/My Drive/Checkpoints/VOC2012_Trained/model.ckpt-870894\"\n",
        "  # the pre-trained model came from: https://github.com/google/aiyprojects-raspbian/issues/314#issuecomment-391414207\n",
        "  # https://drive.google.com/file/d/1_MeZ8kvmpNibPZvSJGnwKNRATeuyxNtu/view?usp=sharing\n",
        "  from_detection_checkpoint: true #DEPRECATED????\n",
        "\n",
        "  # I went crazy with the data augmentation...\n",
        "  data_augmentation_options {\n",
        "    random_horizontal_flip {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_vertical_flip {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_adjust_contrast {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_adjust_saturation {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_rotation90 {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_adjust_brightness {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_distort_color {\n",
        "    }\n",
        "  } \n",
        "  data_augmentation_options {\n",
        "    random_jitter_boxes {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_image_scale {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    random_black_patches {\n",
        "    }\n",
        "  }\n",
        "  data_augmentation_options {\n",
        "    ssd_random_crop {\n",
        "    }\n",
        "  }\n",
        "}\n",
        "\n",
        "train_input_reader: {\n",
        "  tf_record_input_reader {\n",
        "    # If you saved your stuff in a different directory, change it here and below (and above!)\n",
        "    input_path: \"/content/drive/My Drive/Datasets/DarthVader/train.record\"\n",
        "  }\n",
        "  label_map_path: \"/content/drive/My Drive/Datasets/DarthVader/darthvader_object-detection.pbtxt\"\n",
        "}\n",
        "\n",
        "eval_config: {\n",
        "  # (Optional): Uncomment the line below if you installed the Coco evaluation tools\n",
        "  # and you want to also run evaluation (if you are following my notebooks, you did install)\n",
        "  metrics_set: \"coco_detection_metrics\"\n",
        "  # https://github.com/tensorflow/models/issues/2225#issuecomment-325596658\n",
        "  use_moving_averages: true\n",
        "  # (Optional): Set this to the number of images in your train set\n",
        "  # if you want to also run evaluation\n",
        "  num_examples: 11\n",
        "  num_visualizations: 11\n",
        "  eval_interval_secs: 60\n",
        "  # Note: The below line limits the evaluation process to 10 evaluations.\n",
        "  # Remove the below line to evaluate indefinitely.\n",
        "  # max_evals: 10\n",
        "}\n",
        "\n",
        "eval_input_reader: {\n",
        "  tf_record_input_reader {\n",
        "    input_path: \"/content/drive/My Drive/Datasets/DarthVader/test.record\"\n",
        "  }\n",
        "  label_map_path: \"/content/drive/My Drive/Datasets/DarthVader/darthvader_object-detection.pbtxt\"\n",
        "  shuffle: false\n",
        "  num_readers: 1\n",
        "}"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Overwriting ssd_mobilenetv1_256x256_AIY_VOC2012.config\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KlbUGQDZ1kqD"
      },
      "source": [
        "### Training will start soon..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZKawRRqV6vKO"
      },
      "source": [
        "os.chdir(f\"{MYNOTEBOOKPATH}models/research/\")"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KlOV_gDfHA89"
      },
      "source": [
        "**Where do I set the number of steps or time that takes until an event file is created?**\n",
        "- eventsout...\n",
        "The only thing I noticed is: it will save when I stop it and that looks like some cache algorithm."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVBheBmfQzpR"
      },
      "source": [
        "The nice thing about this setup is that you can stop and restart and it will automatically pick up from the last checkpoint if you pass the same folder (there's a text file called \"checkpoint\" that keeps track of that and you can even edit it) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydqv0jaV1sLU"
      },
      "source": [
        "!mkdir \"/content/drive/My Drive/Checkpoints/DarthVader/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "izovzf0fnwYw"
      },
      "source": [
        "Training really starts below. Currently, it's saving 50 checkpoints (save_checkpoints_steps) and it will print a lot of stuff on the output. Jupyter notebooks slow down if a cell becomes too big, therefore you can use the `> /dev/null` to avoid that.\n",
        "\n",
        "Even without Tensorboard, you can check how your training is doing by reading the `INFO:tensorflow:loss = ????, step = ???? (???? sec)` lines in the output. The first time takes a while.\n",
        "\n",
        "If you want to use Tensorboard, I prepared a notebook exactly for that:  \n",
        "https://github.com/thecognifly/AIYVisionKit_Utils/blob/master/Tensorboard_from_GDrive.ipynb"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_UspXLJ40u_u"
      },
      "source": [
        "!MODELDIR=\"/content/drive/My Drive/Checkpoints/DarthVader/\" && \\\n",
        "python object_detection/model_main.py \\\n",
        "    --pipeline_config_path=\"/content/drive/My Drive/Datasets/DarthVader/ssd_mobilenetv1_256x256_AIY_VOC2012.config\" \\\n",
        "    --model_dir=\"${MODELDIR}\" \\\n",
        "    --num_train_steps=30000 \\\n",
        "    --sample_1_of_n_eval_examples=1 \\\n",
        "    --alsologtostderr \\\n",
        "    --save_checkpoints_steps=50#> /dev/null"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TQcJ24ni9Keq"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}