PROD_Overview_managed_ml_automl.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PROD_Overview_managed_ml_automl.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "Hat17x80OVRW"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/noahgift/managed_ml_systems_and_iot/blob/master/PROD_Overview_managed_ml_automl.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "Hat17x80OVRW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Managed Machine Learning Systems and Internet of Things\n",
        "\n",
        "![hero](https://user-images.githubusercontent.com/58792/45260894-08e67a00-b3a8-11e8-941f-e539cb10f8e1.jpg)"
      ]
    },
    {
      "metadata": {
        "id": "4h6NFsfUOaUc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "kaAAnCxZTerK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)"
      ]
    },
    {
      "metadata": {
        "id": "jVxkbCuXOZbg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lEr3vxxDh1xY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Managed and AutoML Systems Overview\n",
        "\n",
        "\n",
        "**Sections**\n",
        "\n",
        "\n",
        "\n",
        "*   [Overview](https://colab.research.google.com/drive/13rfkBM5R3VBIv9Ci5gbUke_HFqIV9NvK#scrollTo=clBbP1yD182S&line=3&uniqifier=1)\n",
        "\n",
        "*   [References](https://colab.research.google.com/drive/13rfkBM5R3VBIv9Ci5gbUke_HFqIV9NvK#scrollTo=rNLiUbJbh9Ch)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "clBbP1yD182S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Overview\n",
        "\n",
        "A Few Examples of Automl and Managed ML Systems\n",
        "\n",
        "### Commercial AutoML\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "KOs2dBmbWJUm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* [AWS Machine Learning](https://docs.aws.amazon.com/machine-learning/latest/dg/tutorial.html?icmpid=docs_machinelearning_console)\n",
        "\n",
        "![AWS AutoMl](https://user-images.githubusercontent.com/58792/49121094-13583780-f264-11e8-9e17-5e0c20e23841.png)\n",
        "\n",
        "* [Google AutoML](https://cloud.google.com/automl/)\n",
        "   - \"Create your own custom vision, natural language, and translation models with minimum machine learning skills required\"\n",
        "![Google AutoML](https://user-images.githubusercontent.com/58792/49597100-99ddda80-f930-11e8-9d4a-2b54d800fc41.png)   \n",
        "   \n",
        "* [Azure/Microsoft:  Custom Vision](https://customvision.ai/)\n",
        "  - \"Full AutoML on Microsoft\"\n",
        "![ms_automl](https://user-images.githubusercontent.com/58792/49519518-fa4c1980-f855-11e8-92f3-975e0a881196.png)\n",
        "\n",
        "*  [ H20](https://www.h2o.ai/products/h2o-driverless-ai/)\n",
        " - Automatic Feature Engineering \n",
        " - ML interpretability\n",
        " - automatic scoring pipelines\n",
        " \n",
        "*   [DataRobot](https://www.datarobot.com/)\n",
        "\n",
        "  - Automated machine learning\n",
        "  - Automates entire modeling lifecycle\n",
        "  - coding and ml skills are optional\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "vEfjV0__WOGO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Open Source AutoML\n"
      ]
    },
    {
      "metadata": {
        "id": "v5D-0rBMFNYm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Auto-Keras"
      ]
    },
    {
      "metadata": {
        "id": "5s-mKmehFRVY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[autokeras](https://autokeras.com/)"
      ]
    },
    {
      "metadata": {
        "id": "iNQSs7TEE778",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### H20 Automl"
      ]
    },
    {
      "metadata": {
        "id": "n7KTQXjUFB66",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[H20 AutoML](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html)"
      ]
    },
    {
      "metadata": {
        "id": "l2Reh1nKEutI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Auto-Sklearn\n"
      ]
    },
    {
      "metadata": {
        "id": "OdA6v_giE1Gl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[auto-sklearn](http://automl.github.io/auto-sklearn/stable/)"
      ]
    },
    {
      "metadata": {
        "id": "NvoiEwiAWrWy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Ludwig (Open Source AutoML)"
      ]
    },
    {
      "metadata": {
        "id": "jbnbFKcTXNOn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Github Project URL**:  https://uber.github.io/ludwig/\n",
        "\n",
        "![alt text](https://user-images.githubusercontent.com/58792/52920000-f78d8c00-32bc-11e9-8e5e-adf53f1b8a37.png)"
      ]
    },
    {
      "metadata": {
        "id": "Aa-KnZKcfvkV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### Install Ludwig"
      ]
    },
    {
      "metadata": {
        "id": "Q3FrtesdfyV9",
        "colab_type": "code",
        "outputId": "2a01a8ab-b163-4775-a101-5457c8c980d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1169
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install --upgrade numpy #must restart colab runtime\n",
        "!pip install --upgrade scikit-image\n",
        "!pip install -q ludwig\n",
        "!python -m spacy download en "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting numpy\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/35/d5/4f8410ac303e690144f0a0603c4b8fd3b986feb2749c435f7cdbb288f17e/numpy-1.16.2-cp36-cp36m-manylinux1_x86_64.whl (17.3MB)\n",
            "\u001b[K    100% |████████████████████████████████| 17.3MB 2.2MB/s \n",
            "\u001b[31mfeaturetools 0.4.1 has requirement pandas>=0.23.0, but you'll have pandas 0.22.0 which is incompatible.\u001b[0m\n",
            "\u001b[31malbumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.8 which is incompatible.\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: numpy\n",
            "  Found existing installation: numpy 1.14.6\n",
            "    Uninstalling numpy-1.14.6:\n",
            "      Successfully uninstalled numpy-1.14.6\n",
            "Successfully installed numpy-1.16.2\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Collecting scikit-image\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/24/06/d560630eb9e36d90d69fe57d9ff762d8f501664ce478b8a0ae132b3c3008/scikit_image-0.14.2-cp36-cp36m-manylinux1_x86_64.whl (25.3MB)\n",
            "\u001b[K    100% |████████████████████████████████| 25.3MB 833kB/s \n",
            "\u001b[?25hCollecting pillow>=4.3.0 (from scikit-image)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/85/5e/e91792f198bbc5a0d7d3055ad552bc4062942d27eaf75c3e2783cf64eae5/Pillow-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 10.1MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (1.11.0)\n",
            "Collecting dask[array]>=1.0.0 (from scikit-image)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a3/79/41d27ad703e782a422636dc8e0ce2f7624ef541b7219bd93a4af0b0d799c/dask-1.1.3-py2.py3-none-any.whl (703kB)\n",
            "\u001b[K    100% |████████████████████████████████| 706kB 22.9MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: cloudpickle>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (0.6.1)\n",
            "Requirement already satisfied, skipping upgrade: matplotlib>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (3.0.3)\n",
            "Requirement already satisfied, skipping upgrade: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (1.1.0)\n",
            "Requirement already satisfied, skipping upgrade: networkx>=1.8 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (2.2)\n",
            "Requirement already satisfied, skipping upgrade: PyWavelets>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image) (1.0.2)\n",
            "Requirement already satisfied, skipping upgrade: toolz>=0.7.3; extra == \"array\" in /usr/local/lib/python3.6/dist-packages (from dask[array]>=1.0.0->scikit-image) (0.9.0)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.11.0; extra == \"array\" in /usr/local/lib/python3.6/dist-packages (from dask[array]>=1.0.0->scikit-image) (1.16.2)\n",
            "Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->scikit-image) (2.3.1)\n",
            "Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->scikit-image) (2.5.3)\n",
            "Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->scikit-image) (0.10.0)\n",
            "Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.0.0->scikit-image) (1.0.1)\n",
            "Requirement already satisfied, skipping upgrade: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=1.8->scikit-image) (4.3.2)\n",
            "Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib>=2.0.0->scikit-image) (40.8.0)\n",
            "\u001b[31mfeaturetools 0.4.1 has requirement pandas>=0.23.0, but you'll have pandas 0.22.0 which is incompatible.\u001b[0m\n",
            "\u001b[31malbumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.8 which is incompatible.\u001b[0m\n",
            "Installing collected packages: pillow, dask, scikit-image\n",
            "  Found existing installation: Pillow 4.1.1\n",
            "    Uninstalling Pillow-4.1.1:\n",
            "      Successfully uninstalled Pillow-4.1.1\n",
            "  Found existing installation: dask 0.20.2\n",
            "    Uninstalling dask-0.20.2:\n",
            "      Successfully uninstalled dask-0.20.2\n",
            "  Found existing installation: scikit-image 0.13.1\n",
            "    Uninstalling scikit-image-0.13.1:\n",
            "      Successfully uninstalled scikit-image-0.13.1\n",
            "Successfully installed dask-1.1.3 pillow-5.4.1 scikit-image-0.14.2\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL",
                  "dask"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K    8% |██▊                             | 10kB 19.5MB/s eta 0:00:01\r\u001b[K    16% |█████▍                          | 20kB 2.3MB/s eta 0:00:01\r\u001b[K    25% |████████▏                       | 30kB 3.3MB/s eta 0:00:01\r\u001b[K    33% |██████████▉                     | 40kB 2.2MB/s eta 0:00:01\r\u001b[K    42% |█████████████▌                  | 51kB 2.6MB/s eta 0:00:01\r\u001b[K    50% |████████████████▎               | 61kB 3.1MB/s eta 0:00:01\r\u001b[K    59% |███████████████████             | 71kB 3.6MB/s eta 0:00:01\r\u001b[K    67% |█████████████████████▋          | 81kB 4.0MB/s eta 0:00:01\r\u001b[K    76% |████████████████████████▍       | 92kB 4.5MB/s eta 0:00:01\r\u001b[K    84% |███████████████████████████     | 102kB 3.5MB/s eta 0:00:01\r\u001b[K    93% |█████████████████████████████▊  | 112kB 3.6MB/s eta 0:00:01\r\u001b[K    100% |████████████████████████████████| 122kB 4.6MB/s \n",
            "\u001b[?25h  Building wheel for ludwig (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25hRequirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in /usr/local/lib/python3.6/dist-packages (2.0.0)\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
            "\n",
            "    You can now load the model via spacy.load('en')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yVLSSAUViyiX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### Basic Ideas"
      ]
    },
    {
      "metadata": {
        "id": "qHaRqAN5i1iV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Training Models**\n",
        "* **Prediction (Inference)**\n",
        "* **Datatypes**\n",
        " - binary\n",
        " - numerical\n",
        " - category\n",
        " - set\n",
        " - bag\n",
        " - sequence\n",
        " - text\n",
        " - timeseries\n",
        " - image\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "W3G5sZ3yo-yK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### Classification of Book Title Categories (Project)"
      ]
    },
    {
      "metadata": {
        "id": "aIbXYrxU8ySd",
        "colab_type": "code",
        "outputId": "b882ab80-bd11-4cad-cc71-8bc12be6f1a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 414
        }
      },
      "cell_type": "code",
      "source": [
        "!wget https://raw.githubusercontent.com/uchidalab/book-dataset/master/Task1/book30-listing-train.csv\n",
        "!wget https://raw.githubusercontent.com/noahgift/recommendations/master/model_definition.yaml"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-03-06 18:33:48--  https://raw.githubusercontent.com/uchidalab/book-dataset/master/Task1/book30-listing-train.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 9728786 (9.3M) [text/plain]\n",
            "Saving to: ‘book30-listing-train.csv’\n",
            "\n",
            "book30-listing-trai 100%[===================>]   9.28M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2019-03-06 18:33:48 (86.7 MB/s) - ‘book30-listing-train.csv’ saved [9728786/9728786]\n",
            "\n",
            "--2019-03-06 18:33:48--  https://raw.githubusercontent.com/noahgift/recommendations/master/model_definition.yaml\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 180 [text/plain]\n",
            "Saving to: ‘model_definition.yaml’\n",
            "\n",
            "model_definition.ya 100%[===================>]     180  --.-KB/s    in 0s      \n",
            "\n",
            "2019-03-06 18:33:49 (22.2 MB/s) - ‘model_definition.yaml’ saved [180/180]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "v-w5Zkzcumoi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### Ingest"
      ]
    },
    {
      "metadata": {
        "id": "Ef8dbaV4tHrz",
        "colab_type": "code",
        "outputId": "94a149d2-4ca0-4bb7-e117-12cbfd59cb8c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv(\"https://media.githubusercontent.com/media/noahgift/recommendations/master/data/book30-listing-train-with-headers.csv\")\n",
        "df = df.drop(\"Unnamed: 0\", axis=1)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ASIN</th>\n",
              "      <th>FILENAME</th>\n",
              "      <th>IMAGE URL</th>\n",
              "      <th>TITLE</th>\n",
              "      <th>AUTHOR</th>\n",
              "      <th>CATEGORYID</th>\n",
              "      <th>CATEGORY</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1404803335</td>\n",
              "      <td>1404803335.jpg</td>\n",
              "      <td>http://ecx.images-amazon.com/images/I/51UJnL3T...</td>\n",
              "      <td>Magnets: Pulling Together, Pushing Apart (Amaz...</td>\n",
              "      <td>Natalie M. Rosinsky</td>\n",
              "      <td>4</td>\n",
              "      <td>Children's Books</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1446276082</td>\n",
              "      <td>1446276082.jpg</td>\n",
              "      <td>http://ecx.images-amazon.com/images/I/51MGUKhk...</td>\n",
              "      <td>Energy Security (SAGE Library of International...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10</td>\n",
              "      <td>Engineering &amp; Transportation</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1491522666</td>\n",
              "      <td>1491522666.jpg</td>\n",
              "      <td>http://ecx.images-amazon.com/images/I/51qKvjsi...</td>\n",
              "      <td>An Amish Gathering: Life in Lancaster County</td>\n",
              "      <td>Beth Wiseman</td>\n",
              "      <td>9</td>\n",
              "      <td>Christian Books &amp; Bibles</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>970096410</td>\n",
              "      <td>0970096410.jpg</td>\n",
              "      <td>http://ecx.images-amazon.com/images/I/51qoUENb...</td>\n",
              "      <td>City of Rocks Idaho: A Climber's Guide (Region...</td>\n",
              "      <td>Dave Bingham</td>\n",
              "      <td>26</td>\n",
              "      <td>Sports &amp; Outdoors</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>8436808053</td>\n",
              "      <td>8436808053.jpg</td>\n",
              "      <td>http://ecx.images-amazon.com/images/I/41aDW5pz...</td>\n",
              "      <td>Como vencer el insomnio. Tecnicas, reglas y co...</td>\n",
              "      <td>Choliz Montanes</td>\n",
              "      <td>11</td>\n",
              "      <td>Health, Fitness &amp; Dieting</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         ASIN        FILENAME  \\\n",
              "0  1404803335  1404803335.jpg   \n",
              "1  1446276082  1446276082.jpg   \n",
              "2  1491522666  1491522666.jpg   \n",
              "3   970096410  0970096410.jpg   \n",
              "4  8436808053  8436808053.jpg   \n",
              "\n",
              "                                           IMAGE URL  \\\n",
              "0  http://ecx.images-amazon.com/images/I/51UJnL3T...   \n",
              "1  http://ecx.images-amazon.com/images/I/51MGUKhk...   \n",
              "2  http://ecx.images-amazon.com/images/I/51qKvjsi...   \n",
              "3  http://ecx.images-amazon.com/images/I/51qoUENb...   \n",
              "4  http://ecx.images-amazon.com/images/I/41aDW5pz...   \n",
              "\n",
              "                                               TITLE               AUTHOR  \\\n",
              "0  Magnets: Pulling Together, Pushing Apart (Amaz...  Natalie M. Rosinsky   \n",
              "1  Energy Security (SAGE Library of International...                  NaN   \n",
              "2       An Amish Gathering: Life in Lancaster County         Beth Wiseman   \n",
              "3  City of Rocks Idaho: A Climber's Guide (Region...         Dave Bingham   \n",
              "4  Como vencer el insomnio. Tecnicas, reglas y co...      Choliz Montanes   \n",
              "\n",
              "   CATEGORYID                      CATEGORY  \n",
              "0           4              Children's Books  \n",
              "1          10  Engineering & Transportation  \n",
              "2           9      Christian Books & Bibles  \n",
              "3          26             Sports & Outdoors  \n",
              "4          11     Health, Fitness & Dieting  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "metadata": {
        "id": "efw0icqJ_0Bq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "df.to_csv(\"book30-listing-train-with-headers.csv\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wR_L2OPkuqH4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### EDA"
      ]
    },
    {
      "metadata": {
        "id": "IUGz5U5duj-D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Columns**"
      ]
    },
    {
      "metadata": {
        "id": "KVYJIiwHuhiT",
        "colab_type": "code",
        "outputId": "82b7a90f-0eed-456f-849f-58d72fa04a04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "cell_type": "code",
      "source": [
        "df.columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['ASIN', 'FILENAME', 'IMAGE URL', 'TITLE', 'AUTHOR', 'CATEGORYID',\n",
              "       'CATEGORY'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "ir0Viy2zuyr1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Shape**"
      ]
    },
    {
      "metadata": {
        "id": "-kaJsKyruyAl",
        "colab_type": "code",
        "outputId": "7b139d22-63ee-40d7-d3f2-f75e25dcbd1f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "df.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(51299, 7)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "metadata": {
        "id": "JJNSA3n0u3Zf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "###### Training w/Ludwig"
      ]
    },
    {
      "metadata": {
        "id": "bldBWuL2Nwmh",
        "colab_type": "code",
        "outputId": "b93bec4c-f589-4fb3-8463-18fbb4627b9f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        }
      },
      "cell_type": "code",
      "source": [
        "!head book30-listing-train-with-headers.csv"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ",ASIN,FILENAME,IMAGE URL,TITLE,AUTHOR,CATEGORYID,CATEGORY\n",
            "0,1404803335,1404803335.jpg,http://ecx.images-amazon.com/images/I/51UJnL3Tx6L.jpg,\"Magnets: Pulling Together, Pushing Apart (Amazing Science)\",Natalie M. Rosinsky,4,Children's Books\n",
            "1,1446276082,1446276082.jpg,http://ecx.images-amazon.com/images/I/51MGUKhkyhL.jpg,Energy Security (SAGE Library of International Security),,10,Engineering & Transportation\n",
            "2,1491522666,1491522666.jpg,http://ecx.images-amazon.com/images/I/51qKvjsi3ML.jpg,An Amish Gathering: Life in Lancaster County,Beth Wiseman,9,Christian Books & Bibles\n",
            "3,970096410,0970096410.jpg,http://ecx.images-amazon.com/images/I/51qoUENb1CL.jpg,City of Rocks Idaho: A Climber's Guide (Regional Rock Climbing Series),Dave Bingham,26,Sports & Outdoors\n",
            "4,8436808053,8436808053.jpg,http://ecx.images-amazon.com/images/I/41aDW5pzZBL.jpg,\"Como vencer el insomnio. Tecnicas, reglas y consejos practicos para dormir mejor (BIBLIOTECA PRACTICA) (Spanish Edition)\",Choliz Montanes,11,\"Health, Fitness & Dieting\"\n",
            "5,1848291388,1848291388.jpg,http://ecx.images-amazon.com/images/I/51Lpg7xmrBL.jpg,John Martin Littlejohn: An Enigma of Osteopathy,John O'Brien,16,Medical Books\n",
            "6,73402656,0073402656.jpg,http://ecx.images-amazon.com/images/I/51WccSzFUrL.jpg,Chemistry: The Molecular Nature of Matter and Change,Martin Silberberg,23,Science & Math\n",
            "7,323045979,0323045979.jpg,http://ecx.images-amazon.com/images/I/51rJir5EpnL.jpg,\"Mosby's Oncology Nursing Advisor: A Comprehensive Guide to Clinical Practice, 1e\",Susan Newton MS  RN  AOCN  AOCNS,16,Medical Books\n",
            "8,1847176968,1847176968.jpg,http://ecx.images-amazon.com/images/I/61KoC743OzL.jpg,Ireland's Wild Atlantic Way,Carsten Krieger,29,Travel\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ous6EqC8Nocg",
        "colab_type": "code",
        "outputId": "ea63b3b8-625d-4eb4-99df-0ea14882292c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "cell_type": "code",
      "source": [
        "!cat model_definition.yaml"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "input_features:\n",
            "    -\n",
            "        name: TITLE\n",
            "        type: text\n",
            "        encoder: parallel_cnn\n",
            "        level: word\n",
            "\n",
            "output_features:\n",
            "    -\n",
            "        name: CATEGORY\n",
            "        type: category"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WpVA2fyXLRoK",
        "colab_type": "code",
        "outputId": "a24ea754-289e-46d6-ce91-32641969fd19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 18997
        }
      },
      "cell_type": "code",
      "source": [
        "!ludwig experiment --data_csv book30-listing-train-with-headers.csv --model_definition_file model_definition.yaml\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " _         _        _      \n",
            "| |_  _ __| |_ __ _(_)__ _ \n",
            "| | || / _` \\ V  V / / _` |\n",
            "|_|\\_,_\\__,_|\\_/\\_/|_\\__, |\n",
            "                     |___/ \n",
            "ludwig v0.1.0 - Experiment\n",
            "\n",
            "Experiment name: experiment\n",
            "Model name: run\n",
            "Output path: results/experiment_run_0\n",
            "\n",
            "ludwig_version: '0.1.0'\n",
            "command: ('/usr/local/bin/ludwig experiment --data_csv '\n",
            " 'book30-listing-train-with-headers.csv --model_definition_file '\n",
            " 'model_definition.yaml')\n",
            "dataset_type: 'book30-listing-train-with-headers.csv'\n",
            "model_definition: {   'combiner': {'type': 'concat'},\n",
            "    'input_features': [   {   'encoder': 'parallel_cnn',\n",
            "                              'level': 'word',\n",
            "                              'name': 'TITLE',\n",
            "                              'tied_weights': None,\n",
            "                              'type': 'text'}],\n",
            "    'output_features': [   {   'dependencies': [],\n",
            "                               'loss': {   'class_distance_temperature': 0,\n",
            "                                           'class_weights': 1,\n",
            "                                           'confidence_penalty': 0,\n",
            "                                           'distortion': 1,\n",
            "                                           'labels_smoothing': 0,\n",
            "                                           'negative_samples': 0,\n",
            "                                           'robust_lambda': 0,\n",
            "                                           'sampler': None,\n",
            "                                           'type': 'softmax_cross_entropy',\n",
            "                                           'unique': False,\n",
            "                                           'weight': 1},\n",
            "                               'name': 'CATEGORY',\n",
            "                               'reduce_dependencies': 'sum',\n",
            "                               'reduce_input': 'sum',\n",
            "                               'top_k': 3,\n",
            "                               'type': 'category'}],\n",
            "    'preprocessing': {   'bag': {   'fill_value': '',\n",
            "                                    'format': 'space',\n",
            "                                    'lowercase': 10000,\n",
            "                                    'missing_value_strategy': 'fill_with_const',\n",
            "                                    'most_common': False},\n",
            "                         'binary': {   'fill_value': 0,\n",
            "                                       'missing_value_strategy': 'fill_with_const'},\n",
            "                         'category': {   'fill_value': '<UNK>',\n",
            "                                         'lowercase': False,\n",
            "                                         'missing_value_strategy': 'fill_with_const',\n",
            "                                         'most_common': 10000},\n",
            "                         'force_split': False,\n",
            "                         'image': {'missing_value_strategy': 'backfill'},\n",
            "                         'numerical': {   'fill_value': 0,\n",
            "                                          'missing_value_strategy': 'fill_with_const'},\n",
            "                         'sequence': {   'fill_value': '',\n",
            "                                         'format': 'space',\n",
            "                                         'lowercase': False,\n",
            "                                         'missing_value_strategy': 'fill_with_const',\n",
            "                                         'most_common': 20000,\n",
            "                                         'padding': 'right',\n",
            "                                         'padding_symbol': '<PAD>',\n",
            "                                         'sequence_length_limit': 256,\n",
            "                                         'unknown_symbol': '<UNK>'},\n",
            "                         'set': {   'fill_value': '',\n",
            "                                    'format': 'space',\n",
            "                                    'lowercase': False,\n",
            "                                    'missing_value_strategy': 'fill_with_const',\n",
            "                                    'most_common': 10000},\n",
            "                         'split_probabilities': (0.7, 0.1, 0.2),\n",
            "                         'stratify': None,\n",
            "                         'text': {   'char_format': 'characters',\n",
            "                                     'char_most_common': 70,\n",
            "                                     'char_sequence_length_limit': 1024,\n",
            "                                     'fill_value': '',\n",
            "                                     'lowercase': True,\n",
            "                                     'missing_value_strategy': 'fill_with_const',\n",
            "                                     'padding': 'right',\n",
            "                                     'padding_symbol': '<PAD>',\n",
            "                                     'unknown_symbol': '<UNK>',\n",
            "                                     'word_format': 'space_punct',\n",
            "                                     'word_most_common': 20000,\n",
            "                                     'word_sequence_length_limit': 256},\n",
            "                         'timeseries': {   'fill_value': '',\n",
            "                                           'format': 'space',\n",
            "                                           'missing_value_strategy': 'fill_with_const',\n",
            "                                           'padding': 'right',\n",
            "                                           'padding_value': 0,\n",
            "                                           'timeseries_length_limit': 256}},\n",
            "    'training': {   'batch_size': 128,\n",
            "                    'bucketing_field': None,\n",
            "                    'decay': False,\n",
            "                    'decay_rate': 0.96,\n",
            "                    'decay_steps': 10000,\n",
            "                    'dropout_rate': 0.0,\n",
            "                    'early_stop': 3,\n",
            "                    'epochs': 200,\n",
            "                    'gradient_clipping': None,\n",
            "                    'increase_batch_size_on_plateau': 0,\n",
            "                    'increase_batch_size_on_plateau_max': 512,\n",
            "                    'increase_batch_size_on_plateau_patience': 5,\n",
            "                    'increase_batch_size_on_plateau_rate': 2,\n",
            "                    'learning_rate': 0.001,\n",
            "                    'learning_rate_warmup_epochs': 5,\n",
            "                    'optimizer': {   'beta1': 0.9,\n",
            "                                     'beta2': 0.999,\n",
            "                                     'epsilon': 1e-08,\n",
            "                                     'type': 'adam'},\n",
            "                    'reduce_learning_rate_on_plateau': 0,\n",
            "                    'reduce_learning_rate_on_plateau_patience': 5,\n",
            "                    'reduce_learning_rate_on_plateau_rate': 0.5,\n",
            "                    'regularization_lambda': 0,\n",
            "                    'regularizer': 'l2',\n",
            "                    'staircase': False,\n",
            "                    'validation_field': 'combined',\n",
            "                    'validation_measure': 'loss'}}\n",
            "\n",
            "Using full raw csv, no hdf5 and json file with the same name have been found\n",
            "Building dataset (it may take a while)\n",
            "Loading NLP pipeline\n",
            "Writing dataset\n",
            "Writing train set metadata with vocabulary\n",
            "Training set: 36059\n",
            "Validation set: 5042\n",
            "Test set: 10198\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n",
            "\n",
            "╒══════════╕\n",
            "│ TRAINING │\n",
            "╘══════════╛\n",
            "\n",
            "2019-03-06 18:36:27.959991: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
            "2019-03-06 18:36:27.960345: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x15c8100 executing computations on platform Host. Devices:\n",
            "2019-03-06 18:36:27.960387: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>\n",
            "2019-03-06 18:36:28.022493: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-03-06 18:36:28.023062: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x15c92e0 executing computations on platform CUDA. Devices:\n",
            "2019-03-06 18:36:28.023105: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
            "2019-03-06 18:36:28.023572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "totalMemory: 11.17GiB freeMemory: 10.99GiB\n",
            "2019-03-06 18:36:28.023609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0\n",
            "2019-03-06 18:36:37.541446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-03-06 18:36:37.541513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 \n",
            "2019-03-06 18:36:37.541534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N \n",
            "2019-03-06 18:36:37.541874: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2019-03-06 18:36:37.541950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10646 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "\n",
            "Epoch   1\n",
            "Training:   0% 0/282 [00:00<?, ?it/s]2019-03-06 18:36:38.317612: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally\n",
            "Training: 100% 282/282 [00:19<00:00, 14.41it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.34it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 58.45it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 55.50it/s]\n",
            "Took 26.4015s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.3077 │     0.0791 │      0.1855 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.3093 │     0.0768 │      0.1813 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.3204 │     0.0757 │      0.1757 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   2\n",
            "Training: 100% 282/282 [00:17<00:00, 17.09it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.76it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 59.62it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 60.34it/s]\n",
            "Took 23.7353s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2798 │     0.0851 │      0.2012 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2836 │     0.0841 │      0.1987 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2976 │     0.0782 │      0.1903 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   3\n",
            "Training: 100% 282/282 [00:17<00:00, 17.10it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.24it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 60.97it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 59.40it/s]\n",
            "Took 23.7937s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2645 │     0.0932 │      0.2105 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2703 │     0.0908 │      0.2061 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2895 │     0.0855 │      0.1908 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   4\n",
            "Training: 100% 282/282 [00:17<00:00, 17.07it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.75it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 59.92it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 59.73it/s]\n",
            "Took 23.8098s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2555 │     0.0930 │      0.2159 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2681 │     0.0863 │      0.2075 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2855 │     0.0847 │      0.1936 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   5\n",
            "Training: 100% 282/282 [00:17<00:00, 16.99it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.74it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 60.59it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 60.42it/s]\n",
            "Took 23.7983s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2494 │     0.0977 │      0.2169 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2630 │     0.0896 │      0.2100 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2852 │     0.0861 │      0.1954 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   6\n",
            "Training: 100% 282/282 [00:17<00:00, 16.93it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.28it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 60.40it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 59.97it/s]\n",
            "Took 23.8555s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2394 │     0.1008 │      0.2227 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2568 │     0.0928 │      0.2156 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2816 │     0.0899 │      0.2009 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Validation loss on combined improved, model saved\n",
            "\n",
            "\n",
            "Epoch   7\n",
            "Training: 100% 282/282 [00:17<00:00, 16.73it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.63it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 60.79it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 59.47it/s]\n",
            "Took 23.7988s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2358 │     0.1029 │      0.2247 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2589 │     0.0930 │      0.2130 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2836 │     0.0885 │      0.2027 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Last improvement of loss on combined happened 1 epoch ago\n",
            "\n",
            "\n",
            "Epoch   8\n",
            "Training: 100% 282/282 [00:17<00:00, 16.85it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 59.38it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 60.19it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 59.76it/s]\n",
            "Took 23.8339s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2308 │     0.1033 │      0.2277 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2631 │     0.0922 │      0.2132 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2841 │     0.0906 │      0.2068 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Last improvement of loss on combined happened 2 epochs ago\n",
            "\n",
            "\n",
            "Epoch   9\n",
            "Training: 100% 282/282 [00:17<00:00, 16.90it/s]\n",
            "Evaluation train: 100% 282/282 [00:04<00:00, 60.34it/s]\n",
            "Evaluation vali : 100% 40/40 [00:00<00:00, 61.08it/s]\n",
            "Evaluation test : 100% 80/80 [00:01<00:00, 60.57it/s]\n",
            "Took 23.7050s\n",
            "╒════════════╤════════╤════════════╤═════════════╕\n",
            "│ CATEGORY   │   loss │   accuracy │   hits_at_k │\n",
            "╞════════════╪════════╪════════════╪═════════════╡\n",
            "│ train      │ 3.2243 │     0.1059 │      0.2311 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ vali       │ 3.2594 │     0.0960 │      0.2114 │\n",
            "├────────────┼────────┼────────────┼─────────────┤\n",
            "│ test       │ 3.2824 │     0.0904 │      0.2051 │\n",
            "╘════════════╧════════╧════════════╧═════════════╛\n",
            "Last improvement of loss on combined happened 3 epochs ago\n",
            "\n",
            "EARLY STOPPING due to lack of validation improvement, it has been 3 epochs since last validation accuracy improvement\n",
            "\n",
            "Best validation model epoch: 6\n",
            "Best validation model loss on validation set combined: 3.2567949578763\n",
            "Best validation model loss on test set combined: 3.2816265680014234\n",
            "\n",
            "╒═════════╕\n",
            "│ PREDICT │\n",
            "╘═════════╛\n",
            "\n",
            "Evaluation: 100% 80/80 [00:01<00:00, 58.31it/s]\n",
            "\n",
            "===== CATEGORY =====\n",
            "accuracy: 0.09040988429103745\n",
            "hits_at_k: 0.205138262404393\n",
            "loss: 3.2823813798077364\n",
            "overall_stats: { 'avg_f1_score_macro': 0.06090545474213738,\n",
            "  'avg_f1_score_micro': 0.09040988429103747,\n",
            "  'avg_f1_score_weighted': 0.0607478077092019,\n",
            "  'avg_precision_macro': 0.0644176382815366,\n",
            "  'avg_precision_micro': 0.09040988429103745,\n",
            "  'avg_precision_weighted': 0.09040988429103745,\n",
            "  'avg_recall_macro': 0.09169381528053014,\n",
            "  'avg_recall_micro': 0.09040988429103745,\n",
            "  'avg_recall_weighted': 0.09040988429103745,\n",
            "  'kappa_score': 0.05942286707355582,\n",
            "  'overall_accuracy': 0.09040988429103745}\n",
            "per_class_stats: {<UNK>: {   'accuracy': 1.0,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.0,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 0,\n",
            "    'false_omission_rate': 0.0,\n",
            "    'false_positive_rate': 0.0,\n",
            "    'false_positives': 0,\n",
            "    'hit_rate': 0,\n",
            "    'informedness': 0.0,\n",
            "    'markedness': 0.0,\n",
            "    'matthews_correlation_coefficient': 0,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 1.0,\n",
            "    'positive_predictive_value': 0,\n",
            "    'precision': 0,\n",
            "    'recall': 0,\n",
            "    'sensitivity': 0,\n",
            "    'specificity': 1.0,\n",
            "    'true_negative_rate': 1.0,\n",
            "    'true_negatives': 10198,\n",
            "    'true_positive_rate': 0,\n",
            "    'true_positives': 0},\n",
            "  Children's Books: {   'accuracy': 0.9494999019415572,\n",
            "    'f1_score': 0.061930783242258654,\n",
            "    'fall_out': 0.030880288750751994,\n",
            "    'false_discovery_rate': 0.9476923076923077,\n",
            "    'false_negative_rate': 0.9241071428571429,\n",
            "    'false_negatives': 207,\n",
            "    'false_omission_rate': 0.02096627164995446,\n",
            "    'false_positive_rate': 0.030880288750751994,\n",
            "    'false_positives': 308,\n",
            "    'hit_rate': 0.07589285714285714,\n",
            "    'informedness': 0.04501256839210521,\n",
            "    'markedness': 0.03134142065773782,\n",
            "    'matthews_correlation_coefficient': 0.037560056454459236,\n",
            "    'miss_rate': 0.9241071428571429,\n",
            "    'negative_predictive_value': 0.9790337283500455,\n",
            "    'positive_predictive_value': 0.052307692307692305,\n",
            "    'precision': 0.052307692307692305,\n",
            "    'recall': 0.07589285714285714,\n",
            "    'sensitivity': 0.07589285714285714,\n",
            "    'specificity': 0.969119711249248,\n",
            "    'true_negative_rate': 0.969119711249248,\n",
            "    'true_negatives': 9666,\n",
            "    'true_positive_rate': 0.07589285714285714,\n",
            "    'true_positives': 17},\n",
            "  Engineering & Transportation: {   'accuracy': 0.9555795253971366,\n",
            "    'f1_score': 0.029978586723768737,\n",
            "    'fall_out': 0.030369411530419166,\n",
            "    'false_discovery_rate': 0.9775641025641025,\n",
            "    'false_negative_rate': 0.9548387096774194,\n",
            "    'false_negatives': 148,\n",
            "    'false_omission_rate': 0.014970665587699772,\n",
            "    'false_positive_rate': 0.030369411530419166,\n",
            "    'false_positives': 305,\n",
            "    'hit_rate': 0.04516129032258064,\n",
            "    'informedness': 0.014791878792161484,\n",
            "    'markedness': 0.007465231848197584,\n",
            "    'matthews_correlation_coefficient': 0.010508320734252603,\n",
            "    'miss_rate': 0.9548387096774194,\n",
            "    'negative_predictive_value': 0.9850293344123002,\n",
            "    'positive_predictive_value': 0.022435897435897436,\n",
            "    'precision': 0.022435897435897436,\n",
            "    'recall': 0.04516129032258064,\n",
            "    'sensitivity': 0.04516129032258064,\n",
            "    'specificity': 0.9696305884695808,\n",
            "    'true_negative_rate': 0.9696305884695808,\n",
            "    'true_negatives': 9738,\n",
            "    'true_positive_rate': 0.04516129032258064,\n",
            "    'true_positives': 7},\n",
            "  Christian Books & Bibles: {   'accuracy': 0.9601882722102373,\n",
            "    'f1_score': 0.01932367149758454,\n",
            "    'fall_out': 0.034142490625616695,\n",
            "    'false_discovery_rate': 0.9885714285714285,\n",
            "    'false_negative_rate': 0.9375,\n",
            "    'false_negatives': 60,\n",
            "    'false_omission_rate': 0.006092607636068226,\n",
            "    'false_positive_rate': 0.034142490625616695,\n",
            "    'false_positives': 346,\n",
            "    'hit_rate': 0.0625,\n",
            "    'informedness': 0.028357509374383305,\n",
            "    'markedness': 0.005335963792503229,\n",
            "    'matthews_correlation_coefficient': 0.012301001718042272,\n",
            "    'miss_rate': 0.9375,\n",
            "    'negative_predictive_value': 0.9939073923639318,\n",
            "    'positive_predictive_value': 0.011428571428571429,\n",
            "    'precision': 0.011428571428571429,\n",
            "    'recall': 0.0625,\n",
            "    'sensitivity': 0.0625,\n",
            "    'specificity': 0.9658575093743833,\n",
            "    'true_negative_rate': 0.9658575093743833,\n",
            "    'true_negatives': 9788,\n",
            "    'true_positive_rate': 0.0625,\n",
            "    'true_positives': 4},\n",
            "  Sports & Outdoors: {   'accuracy': 0.9661698372229849,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.032881821751079676,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 10,\n",
            "    'false_omission_rate': 0.001013890297069886,\n",
            "    'false_positive_rate': 0.032881821751079676,\n",
            "    'false_positives': 335,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.032881821751079676,\n",
            "    'markedness': -0.001013890297069886,\n",
            "    'matthews_correlation_coefficient': -0.005773955318791468,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9989861097029301,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9671181782489203,\n",
            "    'true_negative_rate': 0.9671181782489203,\n",
            "    'true_negatives': 9853,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Health, Fitness & Dieting: {   'accuracy': 0.9627377917238674,\n",
            "    'f1_score': 0.005235602094240837,\n",
            "    'fall_out': 0.03356960031502265,\n",
            "    'false_discovery_rate': 0.9970760233918129,\n",
            "    'false_negative_rate': 0.975,\n",
            "    'false_negatives': 39,\n",
            "    'false_omission_rate': 0.0039569805194805685,\n",
            "    'false_positive_rate': 0.03356960031502265,\n",
            "    'false_positives': 341,\n",
            "    'hit_rate': 0.025,\n",
            "    'informedness': -0.00856960031502263,\n",
            "    'markedness': -0.001033003911293462,\n",
            "    'matthews_correlation_coefficient': -0.002975303454042867,\n",
            "    'miss_rate': 0.975,\n",
            "    'negative_predictive_value': 0.9960430194805194,\n",
            "    'positive_predictive_value': 0.0029239766081871343,\n",
            "    'precision': 0.0029239766081871343,\n",
            "    'recall': 0.025,\n",
            "    'sensitivity': 0.025,\n",
            "    'specificity': 0.9664303996849773,\n",
            "    'true_negative_rate': 0.9664303996849773,\n",
            "    'true_negatives': 9817,\n",
            "    'true_positive_rate': 0.025,\n",
            "    'true_positives': 1},\n",
            "  Medical Books: {   'accuracy': 0.9299862718180035,\n",
            "    'f1_score': 0.07989690721649484,\n",
            "    'fall_out': 0.031157117966588044,\n",
            "    'false_discovery_rate': 0.9074626865671642,\n",
            "    'false_negative_rate': 0.9297052154195011,\n",
            "    'false_negatives': 410,\n",
            "    'false_omission_rate': 0.041569502179864104,\n",
            "    'false_positive_rate': 0.031157117966588044,\n",
            "    'false_positives': 304,\n",
            "    'hit_rate': 0.07029478458049887,\n",
            "    'informedness': 0.03913766661391094,\n",
            "    'markedness': 0.05096781125297167,\n",
            "    'matthews_correlation_coefficient': 0.044662749633889935,\n",
            "    'miss_rate': 0.9297052154195011,\n",
            "    'negative_predictive_value': 0.9584304978201359,\n",
            "    'positive_predictive_value': 0.09253731343283582,\n",
            "    'precision': 0.09253731343283582,\n",
            "    'recall': 0.07029478458049887,\n",
            "    'sensitivity': 0.07029478458049887,\n",
            "    'specificity': 0.968842882033412,\n",
            "    'true_negative_rate': 0.968842882033412,\n",
            "    'true_negatives': 9453,\n",
            "    'true_positive_rate': 0.07029478458049887,\n",
            "    'true_positives': 31},\n",
            "  Science & Math: {   'accuracy': 0.9626397332810355,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.03660451422963695,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 8,\n",
            "    'false_omission_rate': 0.0008142493638676473,\n",
            "    'false_positive_rate': 0.03660451422963695,\n",
            "    'false_positives': 373,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.03660451422963695,\n",
            "    'markedness': -0.0008142493638676473,\n",
            "    'matthews_correlation_coefficient': -0.005459414110155727,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9991857506361324,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.963395485770363,\n",
            "    'true_negative_rate': 0.963395485770363,\n",
            "    'true_negatives': 9817,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Travel: {   'accuracy': 0.9690135320651108,\n",
            "    'f1_score': 0.006289308176100628,\n",
            "    'fall_out': 0.03032384690873402,\n",
            "    'false_discovery_rate': 0.9967741935483871,\n",
            "    'false_negative_rate': 0.875,\n",
            "    'false_negatives': 7,\n",
            "    'false_omission_rate': 0.0007079288025889641,\n",
            "    'false_positive_rate': 0.03032384690873402,\n",
            "    'false_positives': 309,\n",
            "    'hit_rate': 0.125,\n",
            "    'informedness': 0.09467615309126609,\n",
            "    'markedness': 0.002517877649024003,\n",
            "    'matthews_correlation_coefficient': 0.01543965575277066,\n",
            "    'miss_rate': 0.875,\n",
            "    'negative_predictive_value': 0.999292071197411,\n",
            "    'positive_predictive_value': 0.0032258064516129032,\n",
            "    'precision': 0.0032258064516129032,\n",
            "    'recall': 0.125,\n",
            "    'sensitivity': 0.125,\n",
            "    'specificity': 0.969676153091266,\n",
            "    'true_negative_rate': 0.969676153091266,\n",
            "    'true_negatives': 9881,\n",
            "    'true_positive_rate': 0.125,\n",
            "    'true_positives': 1},\n",
            "  Business & Money: {   'accuracy': 0.9679348891939596,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.03082965144820815,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 13,\n",
            "    'false_omission_rate': 0.0013152569809793402,\n",
            "    'false_positive_rate': 0.03082965144820815,\n",
            "    'false_positives': 314,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.03082965144820815,\n",
            "    'markedness': -0.0013152569809793402,\n",
            "    'matthews_correlation_coefficient': -0.006367802940450986,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9986847430190207,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9691703485517918,\n",
            "    'true_negative_rate': 0.9691703485517918,\n",
            "    'true_negatives': 9871,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Cookbooks, Food & Wine: {   'accuracy': 0.9616591488527162,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.03521888834235121,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 33,\n",
            "    'false_omission_rate': 0.0033536585365853133,\n",
            "    'false_positive_rate': 0.03521888834235121,\n",
            "    'false_positives': 358,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.03521888834235121,\n",
            "    'markedness': -0.0033536585365853133,\n",
            "    'matthews_correlation_coefficient': -0.010867940261998726,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9966463414634147,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9647811116576488,\n",
            "    'true_negative_rate': 0.9647811116576488,\n",
            "    'true_negatives': 9807,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Politics & Social Sciences: {   'accuracy': 0.9595018631104139,\n",
            "    'f1_score': 0.004819277108433736,\n",
            "    'fall_out': 0.03643884183572976,\n",
            "    'false_discovery_rate': 0.9973045822102425,\n",
            "    'false_negative_rate': 0.9772727272727273,\n",
            "    'false_negatives': 43,\n",
            "    'false_omission_rate': 0.00437569960313422,\n",
            "    'false_positive_rate': 0.03643884183572976,\n",
            "    'false_positives': 370,\n",
            "    'hit_rate': 0.022727272727272728,\n",
            "    'informedness': -0.013711569108457056,\n",
            "    'markedness': -0.0016802818133767605,\n",
            "    'matthews_correlation_coefficient': -0.004799927104217274,\n",
            "    'miss_rate': 0.9772727272727273,\n",
            "    'negative_predictive_value': 0.9956243003968658,\n",
            "    'positive_predictive_value': 0.0026954177897574125,\n",
            "    'precision': 0.0026954177897574125,\n",
            "    'recall': 0.022727272727272728,\n",
            "    'sensitivity': 0.022727272727272728,\n",
            "    'specificity': 0.9635611581642702,\n",
            "    'true_negative_rate': 0.9635611581642702,\n",
            "    'true_negatives': 9784,\n",
            "    'true_positive_rate': 0.022727272727272728,\n",
            "    'true_positives': 1},\n",
            "  Crafts, Hobbies & Home: {   'accuracy': 0.9663659541086488,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.03135443286809514,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 24,\n",
            "    'false_omission_rate': 0.002429395687822633,\n",
            "    'false_positive_rate': 0.03135443286809514,\n",
            "    'false_positives': 319,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.03135443286809514,\n",
            "    'markedness': -0.002429395687822633,\n",
            "    'matthews_correlation_coefficient': -0.008727675750385966,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9975706043121774,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9686455671319049,\n",
            "    'true_negative_rate': 0.9686455671319049,\n",
            "    'true_negatives': 9855,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Religion & Spirituality: {   'accuracy': 0.964600902137674,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.03511525257479153,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 3,\n",
            "    'false_omission_rate': 0.00030487804878043256,\n",
            "    'false_positive_rate': 0.03511525257479153,\n",
            "    'false_positives': 358,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.03511525257479153,\n",
            "    'markedness': -0.00030487804878043256,\n",
            "    'matthews_correlation_coefficient': -0.003271982531652095,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9996951219512196,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9648847474252085,\n",
            "    'true_negative_rate': 0.9648847474252085,\n",
            "    'true_negatives': 9837,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Literature & Fiction: {   'accuracy': 0.9330260835457933,\n",
            "    'f1_score': 0.09054593874833555,\n",
            "    'fall_out': 0.03334013050570961,\n",
            "    'false_discovery_rate': 0.9058171745152355,\n",
            "    'false_negative_rate': 0.9128205128205128,\n",
            "    'false_negatives': 356,\n",
            "    'false_omission_rate': 0.03618989529328043,\n",
            "    'false_positive_rate': 0.03334013050570961,\n",
            "    'false_positives': 327,\n",
            "    'hit_rate': 0.08717948717948718,\n",
            "    'informedness': 0.053839356673777594,\n",
            "    'markedness': 0.057992930191484104,\n",
            "    'matthews_correlation_coefficient': 0.055877563056532756,\n",
            "    'miss_rate': 0.9128205128205128,\n",
            "    'negative_predictive_value': 0.9638101047067196,\n",
            "    'positive_predictive_value': 0.09418282548476455,\n",
            "    'precision': 0.09418282548476455,\n",
            "    'recall': 0.08717948717948718,\n",
            "    'sensitivity': 0.08717948717948718,\n",
            "    'specificity': 0.9666598694942904,\n",
            "    'true_negative_rate': 0.9666598694942904,\n",
            "    'true_negatives': 9481,\n",
            "    'true_positive_rate': 0.08717948717948718,\n",
            "    'true_positives': 34},\n",
            "  Humor & Entertainment: {   'accuracy': 0.9677387723082957,\n",
            "    'f1_score': 0,\n",
            "    'fall_out': 0.0315014720314033,\n",
            "    'false_discovery_rate': 1.0,\n",
            "    'false_negative_rate': 1.0,\n",
            "    'false_negatives': 8,\n",
            "    'false_omission_rate': 0.0008099625392326004,\n",
            "    'false_positive_rate': 0.0315014720314033,\n",
            "    'false_positives': 321,\n",
            "    'hit_rate': 0.0,\n",
            "    'informedness': -0.0315014720314033,\n",
            "    'markedness': -0.0008099625392326004,\n",
            "    'matthews_correlation_coefficient': -0.005051238687304228,\n",
            "    'miss_rate': 1.0,\n",
            "    'negative_predictive_value': 0.9991900374607674,\n",
            "    'positive_predictive_value': 0.0,\n",
            "    'precision': 0.0,\n",
            "    'recall': 0.0,\n",
            "    'sensitivity': 0.0,\n",
            "    'specificity': 0.9684985279685967,\n",
            "    'true_negative_rate': 0.9684985279685967,\n",
            "    'true_negatives': 9869,\n",
            "    'true_positive_rate': 0.0,\n",
            "    'true_positives': 0},\n",
            "  Law: {   'accuracy': 0.9497940772700529,\n",
            "    'f1_score': 0.02661596958174905,\n",
            "    'fall_out': 0.034128330505937554,\n",
            "    'false_discovery_rate': 0.9799426934097422,\n",
            "    'false_negative_rate': 0.96045197740113,\n",
            "    'false_negatives': 170,\n",
            "    'false_omission_rate': 0.017260635597522556,\n",
            "    'false_positive_rate': 0.034128330505937554,\n",
            "    'false_positives': 342,\n",
            "    'hit_rate': 0.03954802259887006,\n",
            "    'informedness': 0.005419692092932582,\n",
            "    'markedness': 0.002796670992735395,\n",
            "    'matthews_correlation_coefficient': 0.003893211484861977,\n",
            "    'miss_rate': 0.96045197740113,\n",
            "    'negative_predictive_value': 0.9827393644024774,\n",
            "    'positive_predictive_value': 0.02005730659025788,\n",
            "    'precision': 0.02005730659025788,\n",
            "    'recall': 0.03954802259887006,\n",
            "    'sensitivity': 0.03954802259887006,\n",
            "    'specificity': 0.9658716694940624,\n",
            "    'true_negative_rate': 0.9658716694940624,\n",
            "    'true_negatives': 9679,\n",
            "    'true_positive_rate': 0.03954802259887006,\n",
            "    'true_positives': 7},\n",
            "  Computers & Technology: {   'accuracy': 0.952441655226515,\n",
            "    'f1_score': 0.039603960396039604,\n",
            "    'fall_out': 0.035295287333465875,\n",
            "    'false_discovery_rate': 0.9726027397260274,\n",
            "    'false_negative_rate': 0.9285714285714286,\n",
            "    'false_negatives': 130,\n",
            "    'false_omission_rate': 0.013220787145326929,\n",
            "    'false_positive_rate': 0.035295287333465875,\n",
            "    'false_positives': 355,\n",
            "    'hit_rate': 0.07142857142857142,\n",
            "    'informedness': 0.03613328409510563,\n",
            "    'markedness': 0.014176473128645561,\n",
            "    'matthews_correlation_coefficient': 0.022632775592577734,\n",
            "    'miss_rate': 0.9285714285714286,\n",
            "    'negative_predictive_value': 0.9867792128546731,\n",
            "    'positive_predictive_value': 0.0273972602739726,\n",
            "    'precision': 0.0273972602739726,\n",
            "    'recall': 0.07142857142857142,\n",
            "    'sensitivity': 0.07142857142857142,\n",
            "    'specificity': 0.9647047126665341,\n",
            "    'true_negative_rate': 0.9647047126665341,\n",
            "    'true_negatives': 9703,\n",
            "    'true_positive_rate': 0.07142857142857142,\n",
            "    'true_positives': 10},\n",
            "  Test Preparation: {   'accuracy': 0.9214551872916258,\n",
            "    'f1_score': 0.17337461300309598,\n",
            "    'fall_out': 0.025327053898482443,\n",
            "    'false_discovery_rate': 0.7423312883435582,\n",
            "    'false_negative_rate': 0.869362363919129,\n",
            "    'false_negatives': 559,\n",
            "    'false_omission_rate': 0.05662479740680715,\n",
            "    'false_positive_rate': 0.025327053898482443,\n",
            "    'false_positives': 242,\n",
            "    'hit_rate': 0.13063763608087092,\n",
            "    'informedness': 0.10531058218238853,\n",
            "    'markedness': 0.20104391424963453,\n",
            "    'matthews_correlation_coefficient': 0.14550619111864346,\n",
            "    'miss_rate': 0.869362363919129,\n",
            "    'negative_predictive_value': 0.9433752025931929,\n",
            "    'positive_predictive_value': 0.25766871165644173,\n",
            "    'precision': 0.25766871165644173,\n",
            "    'recall': 0.13063763608087092,\n",
            "    'sensitivity': 0.13063763608087092,\n",
            "    'specificity': 0.9746729461015176,\n",
            "    'true_negative_rate': 0.9746729461015176,\n",
            "    'true_negatives': 9313,\n",
            "    'true_positive_rate': 0.13063763608087092,\n",
            "    'true_positives': 84},\n",
            "  Arts & Photography: {   'accuracy': 0.9494018434987253,\n",
            "    'f1_score': 0.033707865168539325,\n",
            "    'fall_out': 0.031925540432345856,\n",
            "    'false_discovery_rate': 0.9725609756097561,\n",
            "    'false_negative_rate': 0.9563106796116505,\n",
            "    'false_negatives': 197,\n",
            "    'false_omission_rate': 0.01995947315096247,\n",
            "    'false_positive_rate': 0.031925540432345856,\n",
            "    'false_positives': 319,\n",
            "    'hit_rate': 0.043689320388349516,\n",
            "    'informedness': 0.01176377995600375,\n",
            "    'markedness': 0.007479551239281346,\n",
            "    'matthews_correlation_coefficient': 0.009380180965661622,\n",
            "    'miss_rate': 0.9563106796116505,\n",
            "    'negative_predictive_value': 0.9800405268490375,\n",
            "    'positive_predictive_value': 0.027439024390243903,\n",
            "    'precision': 0.027439024390243903,\n",
            "    'recall': 0.043689320388349516,\n",
            "    'sensitivity': 0.043689320388349516,\n",
            "    'specificity': 0.9680744595676541,\n",
            "    'true_negative_rate': 0.9680744595676541,\n",
            "    'true_negatives': 9673,\n",
            "    'true_positive_rate': 0.043689320388349516,\n",
            "    'true_positives': 9},\n",
            "  Parenting & Relationships: {   'accuracy': 0.9264561678760541,\n",
            "    'f1_score': 0.12383177570093457,\n",
            "    'fall_out': 0.0331377997324277,\n",
            "    'false_discovery_rate': 0.8586666666666667,\n",
            "    'false_negative_rate': 0.8898128898128899,\n",
            "    'false_negatives': 428,\n",
            "    'false_omission_rate': 0.04357121042451395,\n",
            "    'false_positive_rate': 0.0331377997324277,\n",
            "    'false_positives': 322,\n",
            "    'hit_rate': 0.1101871101871102,\n",
            "    'informedness': 0.07704931045468255,\n",
            "    'markedness': 0.09776212290881947,\n",
            "    'matthews_correlation_coefficient': 0.08679000033823284,\n",
            "    'miss_rate': 0.8898128898128899,\n",
            "    'negative_predictive_value': 0.956428789575486,\n",
            "    'positive_predictive_value': 0.14133333333333334,\n",
            "    'precision': 0.14133333333333334,\n",
            "    'recall': 0.1101871101871102,\n",
            "    'sensitivity': 0.1101871101871102,\n",
            "    'specificity': 0.9668622002675723,\n",
            "    'true_negative_rate': 0.9668622002675723,\n",
            "    'true_negatives': 9395,\n",
            "    'true_positive_rate': 0.1101871101871102,\n",
            "    'true_positives': 53},\n",
            "  Romance: {   'accuracy': 0.8741910178466366,\n",
            "    'f1_score': 0.14409606404269512,\n",
            "    'fall_out': 0.026097534004202116,\n",
            "    'false_discovery_rate': 0.686046511627907,\n",
            "    'false_negative_rate': 0.9064935064935065,\n",
            "    'false_negatives': 1047,\n",
            "    'false_omission_rate': 0.10625126852039779,\n",
            "    'false_positive_rate': 0.026097534004202116,\n",
            "    'false_positives': 236,\n",
            "    'hit_rate': 0.09350649350649351,\n",
            "    'informedness': 0.06740895950229131,\n",
            "    'markedness': 0.20770221985169535,\n",
            "    'matthews_correlation_coefficient': 0.11832578132646722,\n",
            "    'miss_rate': 0.9064935064935065,\n",
            "    'negative_predictive_value': 0.8937487314796022,\n",
            "    'positive_predictive_value': 0.313953488372093,\n",
            "    'precision': 0.313953488372093,\n",
            "    'recall': 0.09350649350649351,\n",
            "    'sensitivity': 0.09350649350649351,\n",
            "    'specificity': 0.9739024659957979,\n",
            "    'true_negative_rate': 0.9739024659957979,\n",
            "    'true_negatives': 8807,\n",
            "    'true_positive_rate': 0.09350649350649351,\n",
            "    'true_positives': 108},\n",
            "  History: {   'accuracy': 0.9407727005295156,\n",
            "    'f1_score': 0.08206686930091187,\n",
            "    'fall_out': 0.030993618960802216,\n",
            "    'false_discovery_rate': 0.9189189189189189,\n",
            "    'false_negative_rate': 0.916923076923077,\n",
            "    'false_negatives': 298,\n",
            "    'false_omission_rate': 0.030207805372529095,\n",
            "    'false_positive_rate': 0.030993618960802216,\n",
            "    'false_positives': 306,\n",
            "    'hit_rate': 0.08307692307692308,\n",
            "    'informedness': 0.05208330411612083,\n",
            "    'markedness': 0.050873275708551935,\n",
            "    'matthews_correlation_coefficient': 0.05147473448315956,\n",
            "    'miss_rate': 0.916923076923077,\n",
            "    'negative_predictive_value': 0.9697921946274709,\n",
            "    'positive_predictive_value': 0.08108108108108109,\n",
            "    'precision': 0.08108108108108109,\n",
            "    'recall': 0.08307692307692308,\n",
            "    'sensitivity': 0.08307692307692308,\n",
            "    'specificity': 0.9690063810391978,\n",
            "    'true_negative_rate': 0.9690063810391978,\n",
            "    'true_negatives': 9567,\n",
            "    'true_positive_rate': 0.08307692307692308,\n",
            "    'true_positives': 27},\n",
            "  Comics & Graphic Novels: {   'accuracy': 0.9567562267111198,\n",
            "    'f1_score': 0.2194690265486726,\n",
            "    'fall_out': 0.02855711422845686,\n",
            "    'false_discovery_rate': 0.8213256484149856,\n",
            "    'false_negative_rate': 0.7155963302752293,\n",
            "    'false_negatives': 156,\n",
            "    'false_omission_rate': 0.01583595574053398,\n",
            "    'false_positive_rate': 0.02855711422845686,\n",
            "    'false_positives': 285,\n",
            "    'hit_rate': 0.28440366972477066,\n",
            "    'informedness': 0.25584655549631385,\n",
            "    'markedness': 0.16283839584448034,\n",
            "    'matthews_correlation_coefficient': 0.2041118386555654,\n",
            "    'miss_rate': 0.7155963302752293,\n",
            "    'negative_predictive_value': 0.984164044259466,\n",
            "    'positive_predictive_value': 0.1786743515850144,\n",
            "    'precision': 0.1786743515850144,\n",
            "    'recall': 0.28440366972477066,\n",
            "    'sensitivity': 0.28440366972477066,\n",
            "    'specificity': 0.9714428857715431,\n",
            "    'true_negative_rate': 0.9714428857715431,\n",
            "    'true_negatives': 9695,\n",
            "    'true_positive_rate': 0.28440366972477066,\n",
            "    'true_positives': 62},\n",
            "  Reference: {   'accuracy': 0.9580309864679349,\n",
            "    'f1_score': 0.03603603603603603,\n",
            "    'fall_out': 0.034039184642786435,\n",
            "    'false_discovery_rate': 0.9772727272727273,\n",
            "    'false_negative_rate': 0.9130434782608696,\n",
            "    'false_negatives': 84,\n",
            "    'false_omission_rate': 0.00853138330286407,\n",
            "    'false_positive_rate': 0.034039184642786435,\n",
            "    'false_positives': 344,\n",
            "    'hit_rate': 0.08695652173913043,\n",
            "    'informedness': 0.05291733709634405,\n",
            "    'markedness': 0.014195889424408747,\n",
            "    'matthews_correlation_coefficient': 0.027408186113894065,\n",
            "    'miss_rate': 0.9130434782608696,\n",
            "    'negative_predictive_value': 0.9914686166971359,\n",
            "    'positive_predictive_value': 0.022727272727272728,\n",
            "    'precision': 0.022727272727272728,\n",
            "    'recall': 0.08695652173913043,\n",
            "    'sensitivity': 0.08695652173913043,\n",
            "    'specificity': 0.9659608153572136,\n",
            "    'true_negative_rate': 0.9659608153572136,\n",
            "    'true_negatives': 9762,\n",
            "    'true_positive_rate': 0.08695652173913043,\n",
            "    'true_positives': 8},\n",
            "  Teen & Young Adult: {   'accuracy': 0.9467542655422632,\n",
            "    'f1_score': 0.04903677758318739,\n",
            "    'fall_out': 0.03406472297364993,\n",
            "    'false_discovery_rate': 0.96045197740113,\n",
            "    'false_negative_rate': 0.935483870967742,\n",
            "    'false_negatives': 203,\n",
            "    'false_omission_rate': 0.02062169849654616,\n",
            "    'false_positive_rate': 0.03406472297364993,\n",
            "    'false_positives': 340,\n",
            "    'hit_rate': 0.06451612903225806,\n",
            "    'informedness': 0.030451406058608077,\n",
            "    'markedness': 0.018926324102323866,\n",
            "    'matthews_correlation_coefficient': 0.02400694025561542,\n",
            "    'miss_rate': 0.935483870967742,\n",
            "    'negative_predictive_value': 0.9793783015034538,\n",
            "    'positive_predictive_value': 0.03954802259887006,\n",
            "    'precision': 0.03954802259887006,\n",
            "    'recall': 0.06451612903225806,\n",
            "    'sensitivity': 0.06451612903225806,\n",
            "    'specificity': 0.9659352770263501,\n",
            "    'true_negative_rate': 0.9659352770263501,\n",
            "    'true_negatives': 9641,\n",
            "    'true_positive_rate': 0.06451612903225806,\n",
            "    'true_positives': 14},\n",
            "  Self-Help: {   'accuracy': 0.8597764267503432,\n",
            "    'f1_score': 0.10062893081761005,\n",
            "    'fall_out': 0.026990704446186564,\n",
            "    'false_discovery_rate': 0.7507788161993769,\n",
            "    'false_negative_rate': 0.9369582348305753,\n",
            "    'false_negatives': 1189,\n",
            "    'false_omission_rate': 0.12038068239343935,\n",
            "    'false_positive_rate': 0.026990704446186564,\n",
            "    'false_positives': 241,\n",
            "    'hit_rate': 0.06304176516942474,\n",
            "    'informedness': 0.03605106072323827,\n",
            "    'markedness': 0.12884050140718362,\n",
            "    'matthews_correlation_coefficient': 0.06815303910936646,\n",
            "    'miss_rate': 0.9369582348305753,\n",
            "    'negative_predictive_value': 0.8796193176065606,\n",
            "    'positive_predictive_value': 0.24922118380062305,\n",
            "    'precision': 0.24922118380062305,\n",
            "    'recall': 0.06304176516942474,\n",
            "    'sensitivity': 0.06304176516942474,\n",
            "    'specificity': 0.9730092955538134,\n",
            "    'true_negative_rate': 0.9730092955538134,\n",
            "    'true_negatives': 8688,\n",
            "    'true_positive_rate': 0.06304176516942474,\n",
            "    'true_positives': 80},\n",
            "  Calendars: {   'accuracy': 0.8386938615414787,\n",
            "    'f1_score': 0.1948115516397455,\n",
            "    'fall_out': 0.014044612297887449,\n",
            "    'false_discovery_rate': 0.37421383647798745,\n",
            "    'false_negative_rate': 0.8846376811594203,\n",
            "    'false_negatives': 1526,\n",
            "    'false_omission_rate': 0.1544534412955466,\n",
            "    'false_positive_rate': 0.014044612297887449,\n",
            "    'false_positives': 119,\n",
            "    'hit_rate': 0.11536231884057971,\n",
            "    'informedness': 0.10131770654269223,\n",
            "    'markedness': 0.47133272222646605,\n",
            "    'matthews_correlation_coefficient': 0.2185276880271912,\n",
            "    'miss_rate': 0.8846376811594203,\n",
            "    'negative_predictive_value': 0.8455465587044534,\n",
            "    'positive_predictive_value': 0.6257861635220126,\n",
            "    'precision': 0.6257861635220126,\n",
            "    'recall': 0.11536231884057971,\n",
            "    'sensitivity': 0.11536231884057971,\n",
            "    'specificity': 0.9859553877021126,\n",
            "    'true_negative_rate': 0.9859553877021126,\n",
            "    'true_negatives': 8354,\n",
            "    'true_positive_rate': 0.11536231884057971,\n",
            "    'true_positives': 199},\n",
            "  Science Fiction & Fantasy: {   'accuracy': 0.940184349872524,\n",
            "    'f1_score': 0.0729483282674772,\n",
            "    'fall_out': 0.02893694791349377,\n",
            "    'false_discovery_rate': 0.9223300970873787,\n",
            "    'false_negative_rate': 0.9312320916905444,\n",
            "    'false_negatives': 325,\n",
            "    'false_omission_rate': 0.03286479927191832,\n",
            "    'false_positive_rate': 0.02893694791349377,\n",
            "    'false_positives': 285,\n",
            "    'hit_rate': 0.06876790830945559,\n",
            "    'informedness': 0.0398309603959619,\n",
            "    'markedness': 0.044805103640703114,\n",
            "    'matthews_correlation_coefficient': 0.04224488499984121,\n",
            "    'miss_rate': 0.9312320916905444,\n",
            "    'negative_predictive_value': 0.9671352007280817,\n",
            "    'positive_predictive_value': 0.07766990291262135,\n",
            "    'precision': 0.07766990291262135,\n",
            "    'recall': 0.06876790830945559,\n",
            "    'sensitivity': 0.06876790830945559,\n",
            "    'specificity': 0.9710630520865062,\n",
            "    'true_negative_rate': 0.9710630520865062,\n",
            "    'true_negatives': 9564,\n",
            "    'true_positive_rate': 0.06876790830945559,\n",
            "    'true_positives': 24},\n",
            "  Mystery, Thriller & Suspense: {   'accuracy': 0.9052755442243577,\n",
            "    'f1_score': 0.12021857923497267,\n",
            "    'fall_out': 0.02871675320546785,\n",
            "    'false_discovery_rate': 0.8041543026706232,\n",
            "    'false_negative_rate': 0.9132720105124836,\n",
            "    'false_negatives': 695,\n",
            "    'false_omission_rate': 0.07047966737653377,\n",
            "    'false_positive_rate': 0.02871675320546785,\n",
            "    'false_positives': 271,\n",
            "    'hit_rate': 0.08672798948751642,\n",
            "    'informedness': 0.05801123628204863,\n",
            "    'markedness': 0.12536602995284296,\n",
            "    'matthews_correlation_coefficient': 0.08527976539213014,\n",
            "    'miss_rate': 0.9132720105124836,\n",
            "    'negative_predictive_value': 0.9295203326234662,\n",
            "    'positive_predictive_value': 0.19584569732937684,\n",
            "    'precision': 0.19584569732937684,\n",
            "    'recall': 0.08672798948751642,\n",
            "    'sensitivity': 0.08672798948751642,\n",
            "    'specificity': 0.9712832467945322,\n",
            "    'true_negative_rate': 0.9712832467945322,\n",
            "    'true_negatives': 9166,\n",
            "    'true_positive_rate': 0.08672798948751642,\n",
            "    'true_positives': 66},\n",
            "  Biographies & Memoirs: {   'accuracy': 0.8841929790154932,\n",
            "    'f1_score': 0.11269722013523667,\n",
            "    'fall_out': 0.030467309985904856,\n",
            "    'false_discovery_rate': 0.7893258426966292,\n",
            "    'false_negative_rate': 0.9230769230769231,\n",
            "    'false_negatives': 900,\n",
            "    'false_omission_rate': 0.09144482828693357,\n",
            "    'false_positive_rate': 0.030467309985904856,\n",
            "    'false_positives': 281,\n",
            "    'hit_rate': 0.07692307692307693,\n",
            "    'informedness': 0.04645576693717213,\n",
            "    'markedness': 0.11922932901643724,\n",
            "    'matthews_correlation_coefficient': 0.07442371880565375,\n",
            "    'miss_rate': 0.9230769230769231,\n",
            "    'negative_predictive_value': 0.9085551717130664,\n",
            "    'positive_predictive_value': 0.21067415730337077,\n",
            "    'precision': 0.21067415730337077,\n",
            "    'recall': 0.07692307692307693,\n",
            "    'sensitivity': 0.07692307692307693,\n",
            "    'specificity': 0.9695326900140951,\n",
            "    'true_negative_rate': 0.9695326900140951,\n",
            "    'true_negatives': 8942,\n",
            "    'true_positive_rate': 0.07692307692307693,\n",
            "    'true_positives': 75}}\n",
            "\n",
            "Finished: experiment_run\n",
            "Saved to: results/experiment_run_0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "21q0EGuO5dpd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "0ExRA9v55cr6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JHaOaZkrER6f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "1OX02OwOEM2-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "CNsgrxh_TnT1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Managed ML\n",
        "\n",
        "*   AWS Sagemaker\n",
        "*   AWS ML Service\n",
        "*   Google Cloud Machine Learning Engine\n",
        "*   Azure Machine Learning Studio\n"
      ]
    },
    {
      "metadata": {
        "id": "P2aormwaFY2V",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "fDJ3UIy1WStP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### ML enabled BI Systems"
      ]
    },
    {
      "metadata": {
        "id": "YzudQSfFWlb_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### AWS Quicksite"
      ]
    },
    {
      "metadata": {
        "id": "7DJqsEMaX9lg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* [AWS Quicksite](https://aws.amazon.com/blogs/big-data/amazon-quicksight-announces-ml-insights-in-preview/)\n",
        "\n",
        "* ML insights embedded into visualization tools\n",
        "\n",
        "![AWS Quicksite](https://d2908q01vomqb2.cloudfront.net/b6692ea5df920cad691c20319a6fffd7a4a766b8/2018/11/27/QuickSightMLInsights4.png)"
      ]
    },
    {
      "metadata": {
        "id": "d5OljB0HYvBY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Google Big Query ML"
      ]
    },
    {
      "metadata": {
        "id": "YnRsJXfeYxsw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[BigQueryML](https://cloud.google.com/bigquery/docs/bigqueryml-intro)\n",
        "\n",
        "![BigQueryML](https://user-images.githubusercontent.com/58792/49598306-31442d00-f933-11e8-81a9-7e611653a30c.png)"
      ]
    },
    {
      "metadata": {
        "id": "hksa_zjN2c49",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## ASICs and Managed and AutoML\n",
        "\n",
        "Most if not all systems tie back to ASICS:  \n",
        "\n",
        "* GPU\n",
        "* TPU\n",
        "* Vendor Specific ASIC:  example/A11 from Apple, AWS Infernetia\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Cx0H8qm8laFP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Potential Endgame?\n",
        "\n",
        "\n",
        "Vertically integrated AI Stack:  Chip + Software + Cloud\n",
        "\n",
        "Example:  [Edge TPU](https://cloud.google.com/edge-tpu/)\n",
        "\n",
        "![Edge TPU](https://cloud.google.com/images/products/edge-tpu/penny-edge-tpu.png)\n",
        "\n",
        "\n",
        "Also AWS has similar strategy (full vertically integrated AI)"
      ]
    },
    {
      "metadata": {
        "id": "rNLiUbJbh9Ch",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## References\n"
      ]
    },
    {
      "metadata": {
        "id": "fkfk6yssiA7O",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*   [Pragmatic AI: An Introduction to Cloud-Based Machine Learning-Physical Book](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   [Pragmatic AI: Pragmatic AI: An Introduction to Cloud-Based Machine Learning-SafariOnline Book](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118)\n",
        "* [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*  [noahgift.com](https://noahgift.com/)\n",
        "*  [paiml.com](https://paiml.com/)\n",
        "*   [Azure Machine Learning Studio Documentation](https://docs.microsoft.com/en-us/azure/machine-learning/studio/)\n",
        "* [ml_studio_overview](http://download.microsoft.com/download/C/4/6/C4606116-522F-428A-BE04-B6D3213E9E52/ml_studio_overview_v1.1.pdf)\n",
        "* [aws_automl](https://alexisperrier.com/aws/2017/12/04/automl_aws_data_science.html)\n",
        "* [data robot](https://www.datarobot.com/)\n",
        "* [auto-weka](https://www.cs.ubc.ca/labs/beta/Projects/autoweka/)\n",
        "* [auto-ml-sklearn](https://github.com/automl/auto-sklearn)\n",
        "* [h2o-driverless-ai](https://www.h2o.ai/products/h2o-driverless-ai/)\n",
        "* [automl challenge](https://www.kdnuggets.com/2016/08/winning-automl-challenge-auto-sklearn.html)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Mh3-LClRh-8W",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}