Lab2-LSTM-For-Timeseries (2).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/header.png\" alt=\"Header\" style=\"width: 400px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">Predictive Maintenance using NVIDIA RAPIDS and Deep Learning Models</h1>\n",
    "<h4 align=\"center\">Part 2: Training GPU LSTM models using Keras+Tensorflow for Time Series</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous lab, we utilized accelerated XGBoost for classification of faulty hard drives. In this lab, we deal with data points gathered from hard drives as time series and use recurrent neural networks to predict hard disk failures. \n",
    "\n",
    "Companies are looking at implementing \"Predictive Maintenance Procedures\" to avoid downtimes by predicting the failure of a device before it happens so they can repair/replace the device, preventing costly downtimes.  \n",
    "\n",
    "In this exercise, we will leverage the Backblaze Hard Drive SMART data to train an LSTM model that will predict potential future failures.  Unlike the previous exercise, where the XGBoost model predicted based on current data, the LSTM model will leverage time series data to look for trends prior to a failure.\n",
    "\n",
    "One observation that comes from this exercise is that most of the work to train this model revolves around the data preparation.  We will walk through creating the \"normal\" and \"failing\" sequences that will indicate when we expect to see early warning signs of a disk failure.  Then we will use that data to train our model.\n",
    "\n",
    "This lab covers the following topics:\n",
    "\n",
    "* [Recurrent Networks: Long Short-Term Memory](#1)\n",
    "* [Vanishing Gradient Problem and LSTMs](#2)\n",
    "* [LSTM Cell Structure](#3)\n",
    "* [LSTM Applications](#4)\n",
    "* [Preparing the Environment](#5)\n",
    "* [Load Datasets for Data Preparation](#6)\n",
    "* [Preparing the Training Data](#7)\n",
    "* [Creating X and y labels](#8)\n",
    "    * [ Exercise 1: Create class labels](#e1)\n",
    "    * [ Exercise 2: Reshaping the training data](#e2)\n",
    "* [Model Design](#9)\n",
    "* [Training the Model](#10)\n",
    "* [Confusion Matrix](#11)\n",
    "* [Optional:  CNN LSTM Architecture](#12)\n",
    "\n",
    "Let's first discuss LSTM models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"1\"></a>\n",
    "## Recurrent Networks: Long Short-Term Memory\n",
    "\n",
    "In this section, we present a brief overview of Recurrent Neural Networks (RNNs) and the Long short-term memory (LSTM) variation of RNNs, which is used in this lab to predict hard disk failures. Note that an in-depth understanding of the internal operations of LSTMs is not required to follow the application presented. However, at the end of this section, you are provided with external links for more in-depth analysis of such models and we encourage the students to read through them in order to have a better grasp of the architecture behind RNN networks.\n",
    "\n",
    "RNNs were created to solve the memory-persistence issue found in conventional feedforward networks. Inspired by the human mind, which does not initiate the thinking process from scratch and instead uses memory-persistence based on the sequential order of events, RNNs were devised to possess such a characteristic. To accomplish this, RNNs maintain a loop that allows the information to be persistent in the network. Figure 1 depicts an RNN network where at each time step $t$, the network receives input $X_t$ and outputs a value of $y_t$. The left side of the figure shows an RNN cell with input $X$ and output $y$. There is also a loop-back of the state, which represents the memory. At a given timestep $t$, the output also depends on all previous timesteps $t-1, t-2, ..., 0$. This can also be thought of as an unrolled network with each timestep being input to a different cell and horizontal connections between cells. These connections represent the state that is passed on to the next timestep. Each timestep generates its own output that can be fed into another layer. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/lstm seq.png\" style=\"margin-top:10px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 1. The unrolled recurrent neural network</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the conventional RNNs tend to look at information stored a long time ago, often we are interested in more recent data points, which in most cases suffice to predict the next state. Another well-studied issue with RNNs is the gradient vanishing-exploding problem. In a multi-layer network, gradients are calculated as a product of gradients of many activation functions. Consequently, when those gradients are close to zero, the gradient vanishes. On the other hand, when the previous gradients are bigger, it explodes. We will return to this topic later in this section.\n",
    "\n",
    "__LSTM__ is a type of Recurrent Neural Network especially designed to prevent the neural network output for a given input from either decaying or exploding as it cycles through the feedback loops. The feedback loops are what allow recurrent networks to be better at pattern recognition than other neural networks. \n",
    " \n",
    "Length of memory of past input is critical for solving sequence learning tasks and Long short-term memory networks provide better performance compared to other RNN architectures by alleviating what is called the vanishing gradient problem.\n",
    "\n",
    "Similar to the RNN model shown above, the LSTM Architecture consists of linear units with a self-connection that allows the flow of a value (forward pass) or gradient (backward pass) to be preserved and subsequently retrieved at the required time step. The self-recurrent unit, the memory cell, is capable of storing information which lies a dozen of time-steps in the past. This is very powerful for many tasks. For example, for text data, an LSTM unit can store information contained in the previous paragraph and apply this information to a sentence in the current paragraph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"2\"></a>\n",
    "## Vanishing Gradient Problem and LSTMs\n",
    "\n",
    "A simple RNN model only has a single hidden RNN layer while a stacked RNN model (needed for advanced applications) has multiple RNN hidden layers. A common problem in deep networks is the “vanishing gradient” problem, where the gradient gets smaller and smaller with each layer until it is too small to affect the deepest layers. \n",
    "\n",
    "This is because small gradients or weights (values less than 1) are multiplied many times over through the multiple time steps, and the gradients shrink asymptotically to zero. This means the weights of those earlier layers won’t be changed significantly and, therefore, the network won’t learn long-term dependencies.\n",
    "\n",
    "The gradients of neural networks are found during backpropagation.  When we multiply lots of -1 << real numbers << 1 together, we end up with a vanishing product, which leads to a very small value and hence practically no learning of the weight values – the predictive power of the neural network then plateaus.\n",
    "\n",
    "With the memory cell in LSTMs, we have continuous gradient flow (errors maintain their value) which thus eliminates the vanishing gradient problem and enables learning from sequences which are hundreds of time steps long."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTM models are powerful enough to learn the most important past behaviours and understand whether or not those past behaviours are important features in making future predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"3\"></a>\n",
    "## LSTM Cell Structure\n",
    "\n",
    "An LSTM cell block is made of various components called the cell state, the input gate, the forget gate, and the output gate.  The cell state runs along the entire length of the cell and interacts with each of the gates.  The gates are a way to control the information that gets through to the cell state.\n",
    "\n",
    "Here is a graphical representation of the LSTM cell:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"img/LSTM cell.png\" style=\"margin-top:10px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 2. LSTM Cell</p> <p style=\"text-align: center;color:gray;font-size:12px;\"> Image from book: <i>Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems</i>\n",
    " by Aurélien Géron </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice first, on the bottom, we have our new sequence value $x_t$ being concatenated to the previous output from the cell $h_{t-1}$. The first step for this combined input is for it to be squashed via a $tanh$ layer. Next, the output of the $tanh$ operation is passed through an input gate. An input gate is a layer of sigmoid activated nodes whose output is multiplied by the squashed input. These gate sigmoids can act to “kill off” any elements of the input vector that aren’t required and need to be forgotten. A sigmoid function outputs values between 0 and 1, so the weights connecting the input to these nodes can be trained to output values close to zero to “switch off” certain input values (or, conversely, outputs close to 1 to “pass through” other values).\n",
    "\n",
    "The next step in the flow of data through this cell is the internal state / forget gate loop. LSTM cells have an internal state variable $c_t$. This variable lagged one time step, i.e. $c_{t-1}$, is added to the input data to create an effective layer of recurrence. This addition operation, instead of a multiplication operation, helps to reduce the risk of vanishing gradients. However, this recurrence loop is controlled by a forget gate. This works the same as the input gate but instead helps the network learn which state variables should be “remembered” or “forgotten”.\n",
    "\n",
    "As the final gate, we have an output layer $tanh$ squashing function (re-scale the numbers between -1 and 1), the output of which is controlled by an output gate. This gate determines which values are actually allowed as an output from the cell $h_t$.\n",
    "\n",
    "Last, running along the top of the cell is the cell state, that leverage pointwise multiplication operations to interact with each of the gates.  \n",
    "\n",
    "\n",
    "To have a more in-depth understanding of LSTM networks and relative analysis, we recommend the curious audience to read the following articles:\n",
    "\n",
    " - [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n",
    " \n",
    " - [Tutorial for Implementing Long Short-Term Memory](https://heartbeat.fritz.ai/a-beginners-guide-to-implementing-long-short-term-memory-networks-lstm-eb7a2ff09a27)\n",
    " \n",
    " - [LSTM: A Search Space Odyssey](https://arxiv.org/pdf/1503.04069.pdf)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"4\"></a>\n",
    "## LSTM Applications\n",
    "\n",
    "LSTMs, due to their ability to learn long term dependencies, are applicable to a number of sequence learning problems including language modelling and translation, acoustic modelling of speech, speech synthesis, speech recognition, audio and video data analysis, handwriting recognition and generation, sequence prediction, and protein secondary structure prediction.\n",
    "\n",
    "In this lab, we apply the LSTM model to the Backblaze data we introduced in the previous lab to create a model that predicts the hard disk failure based on the recorded data-points at least a few days before the actual failure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"5\"></a>\n",
    "## Preparing the Environment\n",
    "\n",
    "Below we are importing the required libraries and tools we use throughout this notebook:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "import os\n",
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)\n",
    "\n",
    "# Suppress TF logs\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "# prevent keras/tf from allocating all gpu memory\n",
    "gpu=2 # set to number of GPU we want to leverage for training\n",
    "tf.config.set_soft_device_placement(True) # set soft device placement to enabled\n",
    "tf.debugging.set_log_device_placement(True) # turns logging for device placement decisions on\n",
    "for each_device in tf.config.experimental.list_physical_devices('GPU'): \n",
    "    tf.config.experimental.set_memory_growth(each_device, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"6\"></a>\n",
    "## Load Datasets for Data Preparation\n",
    "\n",
    "For your convenience, we have filtered and stored the `ST4000DM000` model data points in a pickle file. In the following cell, we are reloading the pickle file into the memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading in training and test SMART data stored in previous exercise...\n",
      "(12237899, 53)\n",
      "(3196552, 53)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data_dir = './data/'\n",
    "\n",
    "# Files from Lab 1\n",
    "train_pkl_file = 'Lab1-2017-Full-ST4000DM000.pkl'\n",
    "test_pkl_file = 'Lab1-2016-Q4-ST4000DM000.pkl'\n",
    "\n",
    "# Load our training and test set from previous step\n",
    "print(\"Reading in training and test SMART data stored in previous exercise...\")\n",
    "df_train = pd.read_pickle(data_dir+train_pkl_file)\n",
    "df_test = pd.read_pickle(data_dir+test_pkl_file)\n",
    "\n",
    "print(df_train.shape)\n",
    "print(df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take another look at the data samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>model</th>\n",
       "      <th>capacity_bytes</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_normalized</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_3_normalized</th>\n",
       "      <th>smart_3_raw</th>\n",
       "      <th>smart_4_normalized</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_198_normalized</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_normalized</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_normalized</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_normalized</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_normalized</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-02-01</td>\n",
       "      <td>Z305B2QN</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>117.0</td>\n",
       "      <td>124084032.0</td>\n",
       "      <td>91.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>9690.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>3.179327e+10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>8.596073e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2017-02-01</td>\n",
       "      <td>Z302A0YH</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>22057080.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>17310.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.169678e+10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1.860347e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2017-02-01</td>\n",
       "      <td>Z305BT0W</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>117.0</td>\n",
       "      <td>119535272.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>8514.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.755353e+10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1.631456e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2017-02-01</td>\n",
       "      <td>Z302A0YE</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>115.0</td>\n",
       "      <td>90592144.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>564.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>18051.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.150714e+10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.376942e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2017-02-01</td>\n",
       "      <td>Z302PGH8</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4000787030016</td>\n",
       "      <td>0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>240109536.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>200.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>14303.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>1.645696e+10</td>\n",
       "      <td>100.0</td>\n",
       "      <td>7.311286e+10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 53 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         date serial_number        model  capacity_bytes  failure  \\\n",
       "4  2017-02-01      Z305B2QN  ST4000DM000   4000787030016        0   \n",
       "7  2017-02-01      Z302A0YH  ST4000DM000   4000787030016        0   \n",
       "8  2017-02-01      Z305BT0W  ST4000DM000   4000787030016        0   \n",
       "12 2017-02-01      Z302A0YE  ST4000DM000   4000787030016        0   \n",
       "13 2017-02-01      Z302PGH8  ST4000DM000   4000787030016        0   \n",
       "\n",
       "    smart_1_normalized  smart_1_raw  smart_3_normalized  smart_3_raw  \\\n",
       "4                117.0  124084032.0                91.0          0.0   \n",
       "7                109.0   22057080.0                92.0          0.0   \n",
       "8                117.0  119535272.0                93.0          0.0   \n",
       "12               115.0   90592144.0                92.0          0.0   \n",
       "13               120.0  240109536.0                92.0          0.0   \n",
       "\n",
       "    smart_4_normalized  ...  smart_198_normalized  smart_198_raw  \\\n",
       "4                100.0  ...                 100.0            0.0   \n",
       "7                100.0  ...                 100.0            0.0   \n",
       "8                100.0  ...                 100.0            0.0   \n",
       "12               100.0  ...                 100.0            0.0   \n",
       "13               100.0  ...                 100.0            0.0   \n",
       "\n",
       "    smart_199_normalized  smart_199_raw  smart_240_normalized  smart_240_raw  \\\n",
       "4                  200.0            0.0                 100.0         9690.0   \n",
       "7                  200.0            0.0                 100.0        17310.0   \n",
       "8                  200.0            0.0                 100.0         8514.0   \n",
       "12                 200.0          564.0                 100.0        18051.0   \n",
       "13                 200.0            0.0                 100.0        14303.0   \n",
       "\n",
       "    smart_241_normalized  smart_241_raw  smart_242_normalized  smart_242_raw  \n",
       "4                  100.0   3.179327e+10                 100.0   8.596073e+09  \n",
       "7                  100.0   2.169678e+10                 100.0   1.860347e+11  \n",
       "8                  100.0   2.755353e+10                 100.0   1.631456e+10  \n",
       "12                 100.0   2.150714e+10                 100.0   2.376942e+11  \n",
       "13                 100.0   1.645696e+10                 100.0   7.311286e+10  \n",
       "\n",
       "[5 rows x 53 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to further process the data before the training phase. Each \"raw\" data column has a duplicated \"normalized\" version (example. smart_1_raw and smart_2_normalized) and while values of the pairs are different because of normalization, they do not possess extra information. Similar to the previous lab, we choose to eliminate the \"normalized\" columns.\n",
    "\n",
    "We also need to sort the columns based on the `serial_number` and `date` columns. Since we are dealing with time series data, the only meaningful way to process is to have them sorted, first by the related serial_number and second by the date recorded. The `prepare_dataframe` function is intended to accomplish these steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_dataframe(df):\n",
    "    print(\"Removing unnecessary columns\")\n",
    "    \n",
    "    # find and drop all normalized columns\n",
    "    cols = [c for c in df.columns if (c.lower().find(\"normalized\")!=-1)]\n",
    "    df=df.drop(columns=cols)\n",
    "\n",
    "    # also drop the model and capacity columns as we don't need them\n",
    "    df = df.drop(columns=['model','capacity_bytes'])\n",
    "\n",
    "    # convert string dates to datetime format\n",
    "    df['date'] = pd.to_datetime(df['date'])\n",
    "\n",
    "    # sort data by serial number and dates so ready for sequence creation\n",
    "    print(\"Sorting the data frame based on serial number and date\")\n",
    "    df = df.sort_values(by=['serial_number', 'date'], axis=0, ascending=True)\n",
    "    df = df.reset_index(drop=True)\n",
    "    \n",
    "    # ensure no NaN's in data\n",
    "    df = df.fillna(0)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the preparation step over both train and test datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing train set...\n",
      "Removing unnecessary columns\n",
      "Sorting the data frame based on serial number and date\n",
      "\n",
      "Processing test set...\n",
      "Removing unnecessary columns\n",
      "Sorting the data frame based on serial number and date\n"
     ]
    }
   ],
   "source": [
    "print(\"Processing train set...\")\n",
    "df_train = prepare_dataframe(df_train)\n",
    "\n",
    "print(\"\")\n",
    "print(\"Processing test set...\")\n",
    "df_test = prepare_dataframe(df_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create sequences from data sets\n",
    "\n",
    "In this section, we create sequences of data based on the serial numbers. To have a better understanding of this process, we need to elaborate on the architecture of the deep learning model that consumes the data. We start with a high-level architecture at this section, and later we will elaborate the model in more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"img/lstm_simple.png\" style=\"margin-top:10px;width:600px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 3. Basic RNN model. </p>\n",
    "\n",
    "In figure 3, a simplified version of the RNN architecture is shown. Here we have `n` time steps representing each data point in the DataFrame. Also, only the last RNN (LSTM) cell is outputting the value $y$, which in our model indicates whether the data sequence ends up in a failure state or not.\n",
    "Note that the input values $x_1, x_2, …, x_n$ are multi-dimensional values and not single scalars.\n",
    "\n",
    "One important question is how to choose the value $n$. Our dataset contains lengthy time series sequences for each hard drive. Below, we choose a sequence length of _5_ and later ask the students to experiment with other values and justify their value of choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, we'd create sequences from all of the data that we have.  Even with the small data sets that we have, that can take more than 4 hours.  For this lab, we will simplify the process to make the time reasonable.\n",
    "\n",
    "To create 'sequences' for training our Time Series models, we will first sort the data by serial number and date (which we did above).  This will allow us to grab groups of sequential records to create each sequence. \n",
    "\n",
    "Next, we reviewed all the records to ensure that we had data reported for every day so we could create consistent sequences.  In the case of this data set, for 2017, there were 2 holes where data wasn't reported, and we used a fill-forward technique to recreate this data.\n",
    "\n",
    "To create models to predict into the future, we will use a \"lookahead_days\" parameter to see how far in the future we want the model to be trained to identify failures.  Determining how many days to lookahead is very dependent on your environment and the nature of the failures.  For the hard drives, during the creation of the lab, we tried 14 days, but we the models we created, with the minimal amount of data we use for this lab, were not very accurate.  In looking at the visualizations from Lab 1, some of the anomalous sensor values seemed to only occur 1-2 days prior to the failure, and we have selected 2 days.  In practice, this is another parameter that can be tuned.\n",
    "\n",
    "With the data all prepared, we can create sequences.\n",
    "\n",
    "Failed Records:\n",
    "\n",
    "Next, we create failed record sequences.  To do this, we will find all of the failed disk records.  Then, we will move back \"lookahead_days\" as this will be the last record in the sequence we are creating.  Then, we will take the previous $sequence length-1$ records and combine it with this last sequence record to create a sequence of records that ends up with a failure \"lookahead_days\" later.  In the case of our data set, there are so few failed disks (~1600) that we will use all of them to train our model.\n",
    "\n",
    "Normal Records:\n",
    "\n",
    "For normal records, we will first identify all disks (by serial_number) that did not have a failure.  Then, we will select a subset of the disks (to shorten the time) to create sequences for.  To create the sequence, we will walk sequentially through the records for a given disk and create sequences from that record and the next $sequence length-1$ records.  So, our first sequence (for length 5) would be Day 1-Day 5.  Then, we will go to the next day and create a sequence of Day 2-Day6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Routine to return serial numbers of good and bad disks\n",
    "\n",
    "def get_disk_serials(df, num_disks):\n",
    "    # Get failed serial numbers\n",
    "    failed_serials = df[df['failure'] == 1]['serial_number'].unique().tolist()\n",
    "\n",
    "    # Get serial numbers for disks that didn't fail - first remove failed disks\n",
    "    df_tmp = df[~df.serial_number.isin(failed_serials)]\n",
    "    normal_serials = df_tmp.serial_number.value_counts()[:num_disks].index.tolist()\n",
    "\n",
    "    print('Normal Disk Serials:',len(normal_serials))\n",
    "    print('Failed Disk Serials:',len(failed_serials))\n",
    "\n",
    "    return normal_serials, failed_serials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine checks all the records for a given serial number to ensure we are not skipping any days.  For consistency in training our model, we need to ensure that there is a data record for each date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def adjust_dates(df_loc): \n",
    "    df_mod = pd.DataFrame()\n",
    "    cur_serial = df_loc['serial_number'].unique().tolist()[0]\n",
    "    col_list = df_loc.columns.tolist()\n",
    "    cur_dates = df_loc['date'].values\n",
    "    \n",
    "    # determine number of days between last record and first record\n",
    "    num_date_range = int((cur_dates[-1] - cur_dates[0]).astype('timedelta64[D]')/ np.timedelta64(1, 'D'))+1\n",
    "    \n",
    "    # do we have records for each day or are there holes ?  If so, fill them.\n",
    "    if num_date_range > cur_dates.shape[0]:\n",
    "        i_low = 0\n",
    "        \n",
    "        # step through all days to ensure next date correct\n",
    "        for i in range(cur_dates.shape[0]-1): \n",
    "            \n",
    "            # calculate number of days between current data and next data - should be 1 day\n",
    "            diff_days = int((cur_dates[i+1] - cur_dates[i]).astype('timedelta64[D]')/ np.timedelta64(1, 'D'))\n",
    "            \n",
    "            # if not 1 day, fill in missing days with forward fill\n",
    "            if diff_days > 1:\n",
    "                df_mod = df_mod.append(df_loc.iloc[i_low:i+1])\n",
    "                tmp_array = np.empty((diff_days-1,len(col_list),))\n",
    "                tmp_array[:] = np.nan\n",
    "                df_add = pd.DataFrame(tmp_array,columns=col_list)\n",
    "                df_add['date'] = [ cur_dates[i] + np.timedelta64(1, 'D')*j for j in range(1,diff_days)]\n",
    "                df_mod = df_mod.append(df_add)\n",
    "                i_low = i+1\n",
    "\n",
    "        # add missing records and use forward fill to update missing sensor data\n",
    "        df_mod = df_mod.append(df_loc.iloc[i_low:])\n",
    "        df_mod = df_mod.fillna(method=\"ffill\")\n",
    "    else:\n",
    "        df_mod = df_loc \n",
    "    \n",
    "    return df_mod "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine fills in missing data to ensure we have consistent sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fix_date_gaps(df, normal_serials=None, failed_serials=None):\n",
    "    df_fixed = pd.DataFrame()\n",
    "\n",
    "    serials_list = normal_serials + failed_serials \n",
    "    for i, cur_serial in enumerate(serials_list): \n",
    "        df_fixed = df_fixed.append(adjust_dates(df[df['serial_number'] == cur_serial]))\n",
    "        \n",
    "    return df_fixed.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine builds failed sequences by finding the failure and then going back \"lookahead\" days and building a sequence based on the previous \"sequence_length\" days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Routine to return failed sequences \n",
    "def create_failed_sequences(df, sequence_length, lookahead):\n",
    "\n",
    "    failed_serials = df.serial_number.unique().tolist()\n",
    "    print(\"Number of failed serials : \", len(failed_serials)) \n",
    "\n",
    "    failed_seq_list = []\n",
    "    for serial in failed_serials:\n",
    "        df_tmp = df[df['serial_number'] == serial]\n",
    "        df_tmp = df_tmp.reset_index(drop=True)\n",
    "        num_recs = df_tmp.index.size\n",
    "        \n",
    "        # if enough records, add failed sequence\n",
    "        if num_recs > (sequence_length+lookahead): \n",
    "            # find first failure\n",
    "            df_failed = df_tmp[df_tmp['failure'] == 1]\n",
    "            \n",
    "            # find end of sequence - going back \"lookahead\" days from failure\n",
    "            idx2 = df_failed.index[0] - lookahead + 1\n",
    "            \n",
    "            # find beginning of sequence\n",
    "            idx1 = idx2 - sequence_length\n",
    "            \n",
    "            if idx1 > 0: \n",
    "                failed_seq_list.append(df_tmp.iloc[idx1:idx2,:])\n",
    "    \n",
    "    print(\"Number of failed sequences :\", len(failed_seq_list)) \n",
    "    \n",
    "    return pd.concat(failed_seq_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Routine to pick some serial_numbers and create all sequences from those disks up to num_normal sequences\n",
    "def create_normal_sequences(df, sequence_length, num_normal, lookahead, day_step=1):\n",
    "    normal_seq_list = []\n",
    "    num_seq = 0\n",
    "    \n",
    "    # ensure no failed sequences\n",
    "    if df[df['failure'] == 1].index.size > 0: return None \n",
    "    \n",
    "    # get list of normal serial numbers\n",
    "    normal_serials = df.serial_number.unique().tolist()\n",
    "    \n",
    "    print(\"Number of normal serials : \", len(normal_serials)) \n",
    "    \n",
    "    for serial in normal_serials:\n",
    "        df_tmp = df[df['serial_number'] == serial]\n",
    "        num_recs = df_tmp.shape[0]\n",
    "        \n",
    "        # \n",
    "        for i in range(0, num_recs-(sequence_length+lookahead)+1, day_step):\n",
    "            if (num_seq < num_normal):\n",
    "                normal_seq_list.append(df_tmp.iloc[i:i+sequence_length])\n",
    "                num_seq += 1\n",
    "    \n",
    "    print(\"Number of normal sequences :\", len(normal_seq_list))\n",
    "    \n",
    "    return pd.concat(normal_seq_list)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Routine to add column \"sequence_label\" indicating whether this was a normal or failed sequence.\n",
    "# We will use this later for as label for training.\n",
    "def label_sequence(df, label):\n",
    "    df.insert(2, 'sequence_label', np.full(df.shape[0], label), True)\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify key parameters - you can change and re-run to test different configurations\n",
    "sequence_length = 5          # Number of days in sequence to train to detect and predict failure\n",
    "lookahead_days = 1           # Number of days in future to predict failure\n",
    "num_normal_disks = 20        # Maximum number of normal disks to look at\n",
    "max_normal_sequences = 4000  # Maximum number of normal sequences to create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Determining good and bad disk serial numbers\n",
      "Normal Disk Serials: 20\n",
      "Failed Disk Serials: 1061\n",
      "Total Records before fixing date gaps : (189830, 27)\n",
      "Total Records after fixing date gaps  : (194895, 27)\n",
      "Creating sequences for normal disks\n",
      "Number of normal serials :  20\n",
      "Number of normal sequences : 4000\n",
      "Creating sequences for failed disks\n",
      "Number of failed serials :  1061\n",
      "Number of failed sequences : 1041\n"
     ]
    }
   ],
   "source": [
    "# get pools of serial numbers for good and bad disks\n",
    "print(\"Determining good and bad disk serial numbers\")\n",
    "normal_serials, bad_serials = get_disk_serials(df_train, num_normal_disks)\n",
    "\n",
    "df_selected = df_train[df_train.serial_number.isin(normal_serials + bad_serials)]\n",
    "print('Total Records before fixing date gaps :', df_selected.shape)\n",
    "\n",
    "df_selected = fix_date_gaps(df_selected, normal_serials, bad_serials)\n",
    "print('Total Records after fixing date gaps  :', df_selected.shape)\n",
    "    \n",
    "# get sequences\n",
    "print(\"Creating sequences for normal disks\")\n",
    "normal_sequences = create_normal_sequences(df_selected[df_selected.serial_number.isin(normal_serials)], \n",
    "                                           sequence_length, max_normal_sequences, lookahead_days)\n",
    "\n",
    "# add field to indicate normal sequence\n",
    "label_sequence(normal_sequences, 0)\n",
    "\n",
    "print(\"Creating sequences for failed disks\")\n",
    "failure_sequences = create_failed_sequences(df_selected[df_selected.serial_number.isin(bad_serials)],\n",
    "                                            sequence_length, lookahead_days)\n",
    "\n",
    "# add field to indicate failing sequence\n",
    "label_sequence(failure_sequences, 1)\n",
    "\n",
    "# combine to creating training set\n",
    "train_samples = pd.concat([normal_sequences, failure_sequences]).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a few normal disk sequences.  You should see 'sequence_length' records in a row that are sequential days for the same disk.  You'll see the first sequence contains day 1-day 5 (for sequence_length 5) and the second sequence contains day 2-day 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>sequence_label</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_3_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_191_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01-01</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18329664.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28388666.0</td>\n",
       "      <td>30392.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30310.0</td>\n",
       "      <td>2.954271e+10</td>\n",
       "      <td>1.651592e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01-02</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22850040.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29206958.0</td>\n",
       "      <td>30416.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30333.0</td>\n",
       "      <td>2.954980e+10</td>\n",
       "      <td>1.651971e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01-03</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58107392.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30048343.0</td>\n",
       "      <td>30440.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30358.0</td>\n",
       "      <td>2.955665e+10</td>\n",
       "      <td>1.652328e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01-04</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>98703136.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30929894.0</td>\n",
       "      <td>30464.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30382.0</td>\n",
       "      <td>2.956570e+10</td>\n",
       "      <td>1.652671e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01-05</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>234334048.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31823917.0</td>\n",
       "      <td>30488.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30405.0</td>\n",
       "      <td>2.957306e+10</td>\n",
       "      <td>1.653034e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01-02</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22850040.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29206958.0</td>\n",
       "      <td>30416.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30333.0</td>\n",
       "      <td>2.954980e+10</td>\n",
       "      <td>1.651971e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01-03</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58107392.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30048343.0</td>\n",
       "      <td>30440.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30358.0</td>\n",
       "      <td>2.955665e+10</td>\n",
       "      <td>1.652328e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01-04</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>98703136.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30929894.0</td>\n",
       "      <td>30464.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30382.0</td>\n",
       "      <td>2.956570e+10</td>\n",
       "      <td>1.652671e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01-05</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>234334048.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31823917.0</td>\n",
       "      <td>30488.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30405.0</td>\n",
       "      <td>2.957306e+10</td>\n",
       "      <td>1.653034e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017-01-06</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>162557080.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32768268.0</td>\n",
       "      <td>30513.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30430.0</td>\n",
       "      <td>2.958124e+10</td>\n",
       "      <td>1.653404e+11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        date serial_number  sequence_label  failure  smart_1_raw  smart_3_raw  \\\n",
       "0 2017-01-01      Z300GQK1               0      0.0   18329664.0          0.0   \n",
       "1 2017-01-02      Z300GQK1               0      0.0   22850040.0          0.0   \n",
       "2 2017-01-03      Z300GQK1               0      0.0   58107392.0          0.0   \n",
       "3 2017-01-04      Z300GQK1               0      0.0   98703136.0          0.0   \n",
       "4 2017-01-05      Z300GQK1               0      0.0  234334048.0          0.0   \n",
       "1 2017-01-02      Z300GQK1               0      0.0   22850040.0          0.0   \n",
       "2 2017-01-03      Z300GQK1               0      0.0   58107392.0          0.0   \n",
       "3 2017-01-04      Z300GQK1               0      0.0   98703136.0          0.0   \n",
       "4 2017-01-05      Z300GQK1               0      0.0  234334048.0          0.0   \n",
       "5 2017-01-06      Z300GQK1               0      0.0  162557080.0          0.0   \n",
       "\n",
       "   smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  ...  smart_191_raw  \\\n",
       "0         12.0          0.0   28388666.0      30392.0  ...            0.0   \n",
       "1         12.0          0.0   29206958.0      30416.0  ...            0.0   \n",
       "2         12.0          0.0   30048343.0      30440.0  ...            0.0   \n",
       "3         12.0          0.0   30929894.0      30464.0  ...            0.0   \n",
       "4         12.0          0.0   31823917.0      30488.0  ...            0.0   \n",
       "1         12.0          0.0   29206958.0      30416.0  ...            0.0   \n",
       "2         12.0          0.0   30048343.0      30440.0  ...            0.0   \n",
       "3         12.0          0.0   30929894.0      30464.0  ...            0.0   \n",
       "4         12.0          0.0   31823917.0      30488.0  ...            0.0   \n",
       "5         12.0          0.0   32768268.0      30513.0  ...            0.0   \n",
       "\n",
       "   smart_192_raw  smart_193_raw  smart_194_raw  smart_197_raw  smart_198_raw  \\\n",
       "0            1.0        19064.0           28.0            0.0            0.0   \n",
       "1            1.0        19064.0           28.0            0.0            0.0   \n",
       "2            1.0        19064.0           28.0            0.0            0.0   \n",
       "3            1.0        19064.0           29.0            0.0            0.0   \n",
       "4            1.0        19064.0           29.0            0.0            0.0   \n",
       "1            1.0        19064.0           28.0            0.0            0.0   \n",
       "2            1.0        19064.0           28.0            0.0            0.0   \n",
       "3            1.0        19064.0           29.0            0.0            0.0   \n",
       "4            1.0        19064.0           29.0            0.0            0.0   \n",
       "5            1.0        19064.0           26.0            0.0            0.0   \n",
       "\n",
       "   smart_199_raw  smart_240_raw  smart_241_raw  smart_242_raw  \n",
       "0            0.0        30310.0   2.954271e+10   1.651592e+11  \n",
       "1            0.0        30333.0   2.954980e+10   1.651971e+11  \n",
       "2            0.0        30358.0   2.955665e+10   1.652328e+11  \n",
       "3            0.0        30382.0   2.956570e+10   1.652671e+11  \n",
       "4            0.0        30405.0   2.957306e+10   1.653034e+11  \n",
       "1            0.0        30333.0   2.954980e+10   1.651971e+11  \n",
       "2            0.0        30358.0   2.955665e+10   1.652328e+11  \n",
       "3            0.0        30382.0   2.956570e+10   1.652671e+11  \n",
       "4            0.0        30405.0   2.957306e+10   1.653034e+11  \n",
       "5            0.0        30430.0   2.958124e+10   1.653404e+11  \n",
       "\n",
       "[10 rows x 28 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normal_sequences.head(2*sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look \"lookahead\" days past the first failed sequence to see the actual failure in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Failure Sequence\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>sequence_label</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_3_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_191_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>2017-02-19</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>83295096.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>531725607.0</td>\n",
       "      <td>25281.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25210.0</td>\n",
       "      <td>2.316110e+10</td>\n",
       "      <td>8.881304e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>2017-02-20</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>73725216.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>532565600.0</td>\n",
       "      <td>25305.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25234.0</td>\n",
       "      <td>2.316667e+10</td>\n",
       "      <td>8.884915e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>2017-02-21</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>41438696.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>533423914.0</td>\n",
       "      <td>25329.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25258.0</td>\n",
       "      <td>2.317391e+10</td>\n",
       "      <td>8.888442e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>2017-02-22</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25482720.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>534282220.0</td>\n",
       "      <td>25353.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25282.0</td>\n",
       "      <td>2.318148e+10</td>\n",
       "      <td>8.892227e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>2017-02-23</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>233703728.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>535136227.0</td>\n",
       "      <td>25377.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25306.0</td>\n",
       "      <td>2.318899e+10</td>\n",
       "      <td>8.895852e+10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         date serial_number  sequence_label  failure  smart_1_raw  \\\n",
       "49 2017-02-19      S30070F5               1      0.0   83295096.0   \n",
       "50 2017-02-20      S30070F5               1      0.0   73725216.0   \n",
       "51 2017-02-21      S30070F5               1      0.0   41438696.0   \n",
       "52 2017-02-22      S30070F5               1      0.0   25482720.0   \n",
       "53 2017-02-23      S30070F5               1      0.0  233703728.0   \n",
       "\n",
       "    smart_3_raw  smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  ...  \\\n",
       "49          0.0          8.0          0.0  531725607.0      25281.0  ...   \n",
       "50          0.0          8.0          0.0  532565600.0      25305.0  ...   \n",
       "51          0.0          8.0          0.0  533423914.0      25329.0  ...   \n",
       "52          0.0          8.0          0.0  534282220.0      25353.0  ...   \n",
       "53          0.0          8.0          0.0  535136227.0      25377.0  ...   \n",
       "\n",
       "    smart_191_raw  smart_192_raw  smart_193_raw  smart_194_raw  smart_197_raw  \\\n",
       "49            0.0            2.0         8249.0           30.0            0.0   \n",
       "50            0.0            2.0         8249.0           29.0            0.0   \n",
       "51            0.0            2.0         8249.0           29.0            0.0   \n",
       "52            0.0            2.0         8249.0           30.0            0.0   \n",
       "53            0.0            2.0         8249.0           29.0            0.0   \n",
       "\n",
       "    smart_198_raw  smart_199_raw  smart_240_raw  smart_241_raw  smart_242_raw  \n",
       "49            0.0            0.0        25210.0   2.316110e+10   8.881304e+10  \n",
       "50            0.0            0.0        25234.0   2.316667e+10   8.884915e+10  \n",
       "51            0.0            0.0        25258.0   2.317391e+10   8.888442e+10  \n",
       "52            0.0            0.0        25282.0   2.318148e+10   8.892227e+10  \n",
       "53            0.0            0.0        25306.0   2.318899e+10   8.895852e+10  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('First Failure Sequence')\n",
    "failure_sequences.head(sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's check the record \"lookahead\" days in the future to see that we did encounter a failure on that date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We should see a failure below 1 days after the sequence above.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_3_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>smart_10_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_191_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16559</th>\n",
       "      <td>2017-02-24</td>\n",
       "      <td>S30070F5</td>\n",
       "      <td>1</td>\n",
       "      <td>184511376.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>535953309.0</td>\n",
       "      <td>25401.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8249.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25330.0</td>\n",
       "      <td>2.319425e+10</td>\n",
       "      <td>8.899058e+10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            date serial_number  failure  smart_1_raw  smart_3_raw  \\\n",
       "16559 2017-02-24      S30070F5        1  184511376.0          0.0   \n",
       "\n",
       "       smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  smart_10_raw  ...  \\\n",
       "16559          8.0          0.0  535953309.0      25401.0           0.0  ...   \n",
       "\n",
       "       smart_191_raw  smart_192_raw  smart_193_raw  smart_194_raw  \\\n",
       "16559            0.0            2.0         8249.0           29.0   \n",
       "\n",
       "       smart_197_raw  smart_198_raw  smart_199_raw  smart_240_raw  \\\n",
       "16559            0.0            0.0            0.0        25330.0   \n",
       "\n",
       "       smart_241_raw  smart_242_raw  \n",
       "16559   2.319425e+10   8.899058e+10  \n",
       "\n",
       "[1 rows x 27 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_fail_serial = failure_sequences['serial_number'].iloc[0]\n",
    "df_tmp = df_train[df_train['serial_number'] == first_fail_serial]\n",
    "first_fail_index = df_tmp[df_tmp['failure'] == 1].iloc[0]\n",
    "\n",
    "print(\"We should see a failure below\",lookahead_days, \"days after the sequence above.\")\n",
    "df_tmp[df_tmp['failure'] == 1].head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's output how many training records we have.  It's not a lot of data, but the process we go through is that same whether it's 1000's or 1M's of records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Sequences: 5041\n"
     ]
    }
   ],
   "source": [
    "# check samples\n",
    "num_seq = int(train_samples.shape[0] / sequence_length)\n",
    "print('Training Sequences:', num_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our objects are huge and in order to prevent memory leaks, we are going to store the data, clear the objects, and retrieve them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gc\n",
    "\n",
    "# Store filesnames for later use\n",
    "lab2_train_file = data_dir + 'Lab2-LSTM_train_data_' + str(sequence_length) + '.pkl'\n",
    "lab2_test_file =  data_dir + 'Lab2-LSTM_test_data_' + str(sequence_length) + '.pkl'\n",
    "\n",
    "# Save away training data\n",
    "train_samples.to_pickle(lab2_train_file)\n",
    "del (train_samples)\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we repeat the same process for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Determining good and bad disk serial numbers\n",
      "Normal Disk Serials: 20\n",
      "Failed Disk Serials: 234\n",
      "Total Records before fixing date gaps : (12605, 27)\n",
      "Total Records after fixing date gaps  : (12605, 27)\n",
      "Creating sequences for normal disks\n",
      "Number of normal serials :  20\n",
      "Number of normal sequences : 1740\n",
      "Creating sequences for failed disks\n",
      "Number of failed serials :  234\n",
      "Number of failed sequences : 216\n"
     ]
    }
   ],
   "source": [
    "# get pools of serial numbers for good and bad disks\n",
    "print(\"Determining good and bad disk serial numbers\")\n",
    "normal_serials, bad_serials = get_disk_serials(df_test, num_normal_disks)\n",
    "\n",
    "df_selected = df_test[df_test.serial_number.isin(normal_serials + bad_serials)]\n",
    "print('Total Records before fixing date gaps :', df_selected.shape)\n",
    "\n",
    "df_selected = fix_date_gaps(df_selected, normal_serials, bad_serials)\n",
    "print('Total Records after fixing date gaps  :', df_selected.shape)\n",
    "    \n",
    "# get sequences\n",
    "print(\"Creating sequences for normal disks\")\n",
    "normal_sequences = create_normal_sequences(df_selected[df_selected.serial_number.isin(normal_serials)], \n",
    "                                           sequence_length, max_normal_sequences, lookahead_days)\n",
    "\n",
    "# add field to indicate normal sequence\n",
    "label_sequence(normal_sequences, 0)\n",
    "\n",
    "print(\"Creating sequences for failed disks\")\n",
    "failure_sequences = create_failed_sequences(df_selected[df_selected.serial_number.isin(bad_serials)],\n",
    "                                            sequence_length, lookahead_days)\n",
    "\n",
    "# add field to indicate failure sequence\n",
    "label_sequence(failure_sequences, 1)\n",
    "\n",
    "# combine to creating training set\n",
    "test_samples = pd.concat([normal_sequences, failure_sequences]).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've created samples for testing let's store them away."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# store away training and test samples\n",
    "test_samples.to_pickle(lab2_test_file)\n",
    "del (test_samples)\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"8\"></a>\n",
    "## Creating X and y labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to cleanly read the LSTM training and test data sets in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>sequence_label</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_3_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_191_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01-01</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18329664.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28388666.0</td>\n",
       "      <td>30392.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30310.0</td>\n",
       "      <td>2.954271e+10</td>\n",
       "      <td>1.651592e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01-02</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22850040.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29206958.0</td>\n",
       "      <td>30416.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30333.0</td>\n",
       "      <td>2.954980e+10</td>\n",
       "      <td>1.651971e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01-03</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58107392.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30048343.0</td>\n",
       "      <td>30440.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30358.0</td>\n",
       "      <td>2.955665e+10</td>\n",
       "      <td>1.652328e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01-04</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>98703136.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30929894.0</td>\n",
       "      <td>30464.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30382.0</td>\n",
       "      <td>2.956570e+10</td>\n",
       "      <td>1.652671e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01-05</td>\n",
       "      <td>Z300GQK1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>234334048.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31823917.0</td>\n",
       "      <td>30488.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19064.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30405.0</td>\n",
       "      <td>2.957306e+10</td>\n",
       "      <td>1.653034e+11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        date serial_number  sequence_label  failure  smart_1_raw  smart_3_raw  \\\n",
       "0 2017-01-01      Z300GQK1               0      0.0   18329664.0          0.0   \n",
       "1 2017-01-02      Z300GQK1               0      0.0   22850040.0          0.0   \n",
       "2 2017-01-03      Z300GQK1               0      0.0   58107392.0          0.0   \n",
       "3 2017-01-04      Z300GQK1               0      0.0   98703136.0          0.0   \n",
       "4 2017-01-05      Z300GQK1               0      0.0  234334048.0          0.0   \n",
       "\n",
       "   smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  ...  smart_191_raw  \\\n",
       "0         12.0          0.0   28388666.0      30392.0  ...            0.0   \n",
       "1         12.0          0.0   29206958.0      30416.0  ...            0.0   \n",
       "2         12.0          0.0   30048343.0      30440.0  ...            0.0   \n",
       "3         12.0          0.0   30929894.0      30464.0  ...            0.0   \n",
       "4         12.0          0.0   31823917.0      30488.0  ...            0.0   \n",
       "\n",
       "   smart_192_raw  smart_193_raw  smart_194_raw  smart_197_raw  smart_198_raw  \\\n",
       "0            1.0        19064.0           28.0            0.0            0.0   \n",
       "1            1.0        19064.0           28.0            0.0            0.0   \n",
       "2            1.0        19064.0           28.0            0.0            0.0   \n",
       "3            1.0        19064.0           29.0            0.0            0.0   \n",
       "4            1.0        19064.0           29.0            0.0            0.0   \n",
       "\n",
       "   smart_199_raw  smart_240_raw  smart_241_raw  smart_242_raw  \n",
       "0            0.0        30310.0   2.954271e+10   1.651592e+11  \n",
       "1            0.0        30333.0   2.954980e+10   1.651971e+11  \n",
       "2            0.0        30358.0   2.955665e+10   1.652328e+11  \n",
       "3            0.0        30382.0   2.956570e+10   1.652671e+11  \n",
       "4            0.0        30405.0   2.957306e+10   1.653034e+11  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read datasets back in cleanly\n",
    "df_train = pd.read_pickle(lab2_train_file)\n",
    "df_test = pd.read_pickle(lab2_test_file)\n",
    "\n",
    "df_train.head(sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before creating X and y labels, there are certain columns that we need to get rid of. As an example, we can safely remove the `date` column. While the date column values helped us creating the sequences, they cannot be used for training/inference purposes (why?).\n",
    "\n",
    "Also, the `serial_number` and `failure` columns will not be used. Discuss why these two columns are not required."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Discuss your answers here:  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e1\"></a>\n",
    "### Exercise 1: Create class labels\n",
    "\n",
    "To train the model, we need to have the class labels for each sample. Complete the \"TODO\" sections below to store the class labels for `df_train` and `df_test` DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# index data will hold disk serial numbers for relevant samples. We do not need serial numbers as training or testing features\n",
    "drop_columns_list = ['date','serial_number', 'failure', 'sequence_label']\n",
    "\n",
    "# x_train = <<< TO DO >>>\n",
    "# x_test = <<< TO DO >>>\n",
    "x_train = df_train.drop(columns=drop_columns_list)\n",
    "x_test = df_test.drop(columns=drop_columns_list)\n",
    "\n",
    "y_train = df_train['sequence_label']\n",
    "y_test = df_test['sequence_label']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get stuck, click [here](#a1) for an answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously, we removed the normalized columns and kept the raw ones. The reason behind this was that the normalized columns are generated based on much more samples from the original set and hence, do not represent the distribution accurately. In this section, we are going to normalize values in both test and train sets. We will utilize [preprocessing.StandardScaler().fit()](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) method for this purpose, but first we need to convert the Pandas DataFrame to NumPy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "# X train data\n",
    "x_train = x_train.values.astype(np.float64)\n",
    "scaler = preprocessing.StandardScaler().fit(x_train)\n",
    "x_train = scaler.transform(x_train)\n",
    "\n",
    "# X test data\n",
    "x_test = x_test.values.astype(np.float64)\n",
    "x_test = scaler.transform(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e2\"></a>\n",
    "### Exercise 2: Reshaping the training data\n",
    "\n",
    "Next, we need to reshape the array to get 3D data (num_samples, timesteps, num_features) needed to feed LSTM. Fix to TODO sections below to reshape the data accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reshaping training and test data for LSTM\n",
      "- Original Train Shape: (25205, 24)\n",
      "- Final Train Shape:    (5041, 5, 24)\n"
     ]
    }
   ],
   "source": [
    "print(\"Reshaping training and test data for LSTM\")\n",
    "print(\"- Original Train Shape:\", x_train.shape)\n",
    "\n",
    "# x_train = <<< TO DO >>>\n",
    "# x_test = <<< TO DO >>>\n",
    "x_train = x_train.reshape(int(x_train.shape[0]/sequence_length), sequence_length, x_train.shape[1])\n",
    "x_test = x_test.reshape(int(x_test.shape[0]/sequence_length), sequence_length, x_test.shape[1])\n",
    "\n",
    "print(\"- Final Train Shape:   \", x_train.shape)\n",
    "data_shape = x_train.shape[1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get stuck, click [here](#a2) for an answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we create the y labels (failed or not) for both test and train sets (last column of `y_train` and `y_test`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Y train data\n",
    "y_train = y_train.values[0::sequence_length]\n",
    "\n",
    "# Y test data\n",
    "y_test = y_test.values[0::sequence_length]\n",
    "\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"9\"></a>\n",
    "## Model Design\n",
    "We've finally arrived at the neural network design! Previously, we provided a simplistic overview of the network. In this section, we are going to discuss the full network design first and then build the model using tf.keras.\n",
    "\n",
    "In figure 3, we showed how we utilize the output of the last LSTM node in the sequence to create the relevant output. In reality, one-layer LSTM nodes are barely enough to capture the information encoded within a sequence. Alternatively, [stacked LSTMs](https://machinelearningmastery.com/stacked-long-short-term-memory-networks/) allow capturing more data complexity. In our model, we are going to use three layers of stacked LSTMs, as shown in figure 4.\n",
    "\n",
    "\n",
    "<img src=\"img/lstm_stacked.png\" style=\"margin-top:10px;width:600px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 4.  Stacked LSTM layers. </p>\n",
    "\n",
    "There is an important difference between the bottom two layers and the top one. While all cells of the bottom two layers send their outputs to their upper layer, the topmost layer has only one output which is used for classification purposes. This is achieved by setting the `return_sequences` in the LSTM definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op RandomUniform in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Sub in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Mul in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Add in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarIsInitializedOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op LogicalNot in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Assert in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarIsInitializedOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op LogicalNot in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op Assert in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op RandomStandardNormal in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Qr in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op DiagPart in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Sign in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Transpose in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op ConcatV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import LSTM\n",
    "\n",
    "# some parameters to control model\n",
    "dp_lvl = 0.2\n",
    "regularizer_lvl = 0.002\n",
    "units = 64\n",
    "\n",
    "# distribute training across multiple GPUs\n",
    "strategy = tf.distribute.MirroredStrategy(devices=[f'/gpu:{device_id}' for device_id in range(gpu)])\n",
    "\n",
    "# network design\n",
    "with strategy.scope():\n",
    "    model = Sequential()\n",
    "    model.add(LSTM(units, input_shape=(x_train.shape[1], x_train.shape[2]),dropout = dp_lvl,recurrent_dropout = dp_lvl, return_sequences =  True ))\n",
    "    model.add(LSTM(units, dropout = dp_lvl,recurrent_dropout = dp_lvl, return_sequences =  True ))\n",
    "    model.add(LSTM(units, dropout = dp_lvl,recurrent_dropout = dp_lvl, return_sequences =  False ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `units` variable holds the number of output dimensionality for each cell. At the next level, we add three fully-connected (FC) layers. The first FC layer is followed by a [Dropout layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout). The [Dropout method](https://www.analyticsvidhya.com/blog/2018/04/fundamentals-deep-learning-regularization-techniques/) is a regularization technique to reduce the possibility of overfitting during training. Using this method, some randomly selected neurons are ignored during training. During the forward pass, the selected neurons do not contribute to the activation function, and their weights are also not updated within the backward phase. The last layer has a single output which indicates whether the output of the sequence is a faulty hard drive or not. Let's add those three layers to our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "\n",
    "with strategy.scope():\n",
    "    model.add(Dense(units, activation='tanh',activity_regularizer=regularizers.l2(regularizer_lvl)))\n",
    "\n",
    "    # Dropout is a technique used to tackle Overfitting.\n",
    "    model.add(Dropout (0.2))\n",
    "    model.add(Dense(units, activation='tanh',activity_regularizer=regularizers.l2(regularizer_lvl)))\n",
    "    model.add(Dense(1, activation='tanh',activity_regularizer=regularizers.l2(regularizer_lvl)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that, our model is complete and ready to train. Let's first take an overall look at the model we have built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm (LSTM)                  (None, 5, 64)             22784     \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 5, 64)             33024     \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 64)                33024     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 97,217\n",
      "Trainable params: 97,217\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"10\"></a>\n",
    "## Training the Model\n",
    "\n",
    "\n",
    "There are a few parameters that we need to define before proceeding with the training. We set the [learning rate](https://machinelearningmastery.com/understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks/), the number of epochs, checkpoint file locations, [decay](https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1), and more importantly define a numerical optimizer, [Adam](https://arxiv.org/abs/1412.6980), to train the network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training using 2 GPU(s)\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.utils import multi_gpu_model\n",
    "\n",
    "epochs_num = 50\n",
    "learning_rate = 0.001\n",
    "decay_rate = 3 * learning_rate / epochs_num\n",
    "optimizer = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, epsilon=None,decay = decay_rate)\n",
    "\n",
    "print('Training using %d GPU(s)' % strategy.num_replicas_in_sync)\n",
    "\n",
    "with strategy.scope(): \n",
    "    model.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=['accuracy', 'mae'])\n",
    "\n",
    "# checkpoint\n",
    "model_fn = \"best_model.h5\"\n",
    "model_filepath=data_dir+model_fn\n",
    "\n",
    "checkpoint = ModelCheckpoint(model_filepath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='max')\n",
    "es = EarlyStopping(patience=20, verbose=1)\n",
    "\n",
    "callbacks_list = [es, checkpoint]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the model defined and data loaded, let's go ahead and train the model.\n",
    "\n",
    "Remember - Our model is learning the sequences that lead up to failure a few days before the failure occurs (imposed through the way data is labeled).  This can be used as guidance on when we might perform predictive maintenance.  In our case, that would be replacing a disk driver prior to an expected failure.\n",
    "\n",
    "<span style=\"background-color:red;color:white;padding:1px,3px\"><b>Note:</b></span> At the beginning, the weights are initialized randomly. Sometimes, specific weight initialization and network settings lead to early overfitting. Since our training data is small, this may occur more often. If you see an early convergence of accuracy to 1.00, e.g.:\n",
    "\n",
    "```\n",
    "Epoch 00002: val_acc did not improve from 1.00000\n",
    "```\n",
    "\n",
    "you need to start the training over. In the above example, notice that the epoch number 2 is still very early in the training and the network has already overfitted.\n",
    "\n",
    "Now start the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op RangeDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op DatasetCardinality in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RebatchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op AutoShardDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Train on 4536 samples, validate on 505 samples\n",
      "Epoch 1/50\n",
      "Executing op OptimizeDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ModelDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MultiDeviceIterator in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MultiDeviceIteratorInit in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MultiDeviceIteratorToStringHandle in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op AnonymousIteratorV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op MakeIterator in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op AnonymousIteratorV2 in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op MakeIterator in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op AssignVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op Fill in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "INFO:tensorflow:batch_all_reduce: 15 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).\n",
      "Executing op __inference_initialize_variables_8919 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "INFO:tensorflow:batch_all_reduce: 15 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10\n",
      "Executing op __inference_distributed_function_15138 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 1.4311 - accuracy: 0.8735 - mae: 0.2403Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_17897 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "\n",
      "Epoch 00001: val_accuracy improved from -inf to 0.05941, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 20s 4ms/sample - loss: 1.4296 - accuracy: 0.8735 - mae: 0.2406 - val_loss: 1.2930 - val_accuracy: 0.0594 - val_mae: 0.6666\n",
      "Epoch 2/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.4059 - accuracy: 0.8927 - mae: 0.2826\n",
      "Epoch 00002: val_accuracy improved from 0.05941 to 0.41188, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.4056 - accuracy: 0.8926 - mae: 0.2825 - val_loss: 1.7435 - val_accuracy: 0.4119 - val_mae: 0.5641\n",
      "Epoch 3/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3869 - accuracy: 0.8980 - mae: 0.2681\n",
      "Epoch 00003: val_accuracy did not improve from 0.41188\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3863 - accuracy: 0.8981 - mae: 0.2680 - val_loss: 4.6937 - val_accuracy: 0.2733 - val_mae: 0.7179\n",
      "Epoch 4/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.4013 - accuracy: 0.9335 - mae: 0.2763\n",
      "Epoch 00004: val_accuracy improved from 0.41188 to 0.42970, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.4006 - accuracy: 0.9336 - mae: 0.2764 - val_loss: 6.7667 - val_accuracy: 0.4297 - val_mae: 0.7154\n",
      "Epoch 5/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.4023 - accuracy: 0.9099 - mae: 0.2792\n",
      "Epoch 00005: val_accuracy did not improve from 0.42970\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.4017 - accuracy: 0.9101 - mae: 0.2791 - val_loss: 6.6436 - val_accuracy: 0.4079 - val_mae: 0.6873\n",
      "Epoch 6/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3666 - accuracy: 0.9331 - mae: 0.2231\n",
      "Epoch 00006: val_accuracy improved from 0.42970 to 0.48713, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3660 - accuracy: 0.9332 - mae: 0.2229 - val_loss: 4.9802 - val_accuracy: 0.4871 - val_mae: 0.5852\n",
      "Epoch 7/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3437 - accuracy: 0.9357 - mae: 0.1996\n",
      "Epoch 00007: val_accuracy did not improve from 0.48713\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3435 - accuracy: 0.9356 - mae: 0.1996 - val_loss: 4.9817 - val_accuracy: 0.4812 - val_mae: 0.5806\n",
      "Epoch 8/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3400 - accuracy: 0.9216 - mae: 0.2319\n",
      "Epoch 00008: val_accuracy did not improve from 0.48713\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3401 - accuracy: 0.9215 - mae: 0.2319 - val_loss: 5.0890 - val_accuracy: 0.3624 - val_mae: 0.6469\n",
      "Epoch 9/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2955 - accuracy: 0.9455 - mae: 0.1935\n",
      "Epoch 00009: val_accuracy improved from 0.48713 to 0.50495, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2952 - accuracy: 0.9455 - mae: 0.1934 - val_loss: 5.8158 - val_accuracy: 0.5050 - val_mae: 0.5998\n",
      "Epoch 10/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3367 - accuracy: 0.9291 - mae: 0.2247\n",
      "Epoch 00010: val_accuracy improved from 0.50495 to 0.52079, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3362 - accuracy: 0.9292 - mae: 0.2247 - val_loss: 5.7874 - val_accuracy: 0.5208 - val_mae: 0.5774\n",
      "Epoch 11/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2751 - accuracy: 0.9488 - mae: 0.1777\n",
      "Epoch 00011: val_accuracy improved from 0.52079 to 0.54653, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2748 - accuracy: 0.9489 - mae: 0.1777 - val_loss: 4.4163 - val_accuracy: 0.5465 - val_mae: 0.4968\n",
      "Epoch 12/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3338 - accuracy: 0.9198 - mae: 0.2214\n",
      "Epoch 00012: val_accuracy did not improve from 0.54653\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3333 - accuracy: 0.9200 - mae: 0.2214 - val_loss: 4.9058 - val_accuracy: 0.2396 - val_mae: 0.7328\n",
      "Epoch 13/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.4491 - accuracy: 0.8514 - mae: 0.2953\n",
      "Epoch 00013: val_accuracy improved from 0.54653 to 0.57030, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.4484 - accuracy: 0.8516 - mae: 0.2950 - val_loss: 1.0599 - val_accuracy: 0.5703 - val_mae: 0.4389\n",
      "Epoch 14/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3878 - accuracy: 0.9375 - mae: 0.2300\n",
      "Epoch 00014: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3872 - accuracy: 0.9376 - mae: 0.2299 - val_loss: 6.1629 - val_accuracy: 0.5188 - val_mae: 0.6077\n",
      "Epoch 15/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3356 - accuracy: 0.9382 - mae: 0.2281\n",
      "Epoch 00015: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3353 - accuracy: 0.9381 - mae: 0.2281 - val_loss: 1.2215 - val_accuracy: 0.5663 - val_mae: 0.4271\n",
      "Epoch 16/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3366 - accuracy: 0.9494 - mae: 0.2155\n",
      "Epoch 00016: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3395 - accuracy: 0.9493 - mae: 0.2157 - val_loss: 6.0282 - val_accuracy: 0.5248 - val_mae: 0.5700\n",
      "Epoch 17/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3294 - accuracy: 0.9368 - mae: 0.2194\n",
      "Epoch 00017: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3289 - accuracy: 0.9369 - mae: 0.2193 - val_loss: 6.0089 - val_accuracy: 0.5089 - val_mae: 0.6076\n",
      "Epoch 18/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3355 - accuracy: 0.9494 - mae: 0.2204\n",
      "Epoch 00018: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3350 - accuracy: 0.9495 - mae: 0.2205 - val_loss: 5.7713 - val_accuracy: 0.5406 - val_mae: 0.5717\n",
      "Epoch 19/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2871 - accuracy: 0.9428 - mae: 0.2178\n",
      "Epoch 00019: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2907 - accuracy: 0.9422 - mae: 0.2181 - val_loss: 1.3114 - val_accuracy: 0.5327 - val_mae: 0.4660\n",
      "Epoch 20/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3119 - accuracy: 0.9231 - mae: 0.2218\n",
      "Epoch 00020: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3121 - accuracy: 0.9233 - mae: 0.2220 - val_loss: 1.0816 - val_accuracy: 0.3168 - val_mae: 0.6129\n",
      "Epoch 21/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3557 - accuracy: 0.9167 - mae: 0.2559\n",
      "Epoch 00021: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3552 - accuracy: 0.9169 - mae: 0.2556 - val_loss: 5.5882 - val_accuracy: 0.5010 - val_mae: 0.5785\n",
      "Epoch 22/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3156 - accuracy: 0.9371 - mae: 0.2068\n",
      "Epoch 00022: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3151 - accuracy: 0.9372 - mae: 0.2067 - val_loss: 6.2058 - val_accuracy: 0.5149 - val_mae: 0.6113\n",
      "Epoch 23/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3231 - accuracy: 0.9468 - mae: 0.2246\n",
      "Epoch 00023: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3260 - accuracy: 0.9466 - mae: 0.2246 - val_loss: 4.1420 - val_accuracy: 0.5663 - val_mae: 0.4571\n",
      "Epoch 24/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2901 - accuracy: 0.9492 - mae: 0.1922\n",
      "Epoch 00024: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2897 - accuracy: 0.9493 - mae: 0.1923 - val_loss: 4.0838 - val_accuracy: 0.5703 - val_mae: 0.4702\n",
      "Epoch 25/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2862 - accuracy: 0.9490 - mae: 0.2148\n",
      "Epoch 00025: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2859 - accuracy: 0.9491 - mae: 0.2147 - val_loss: 5.8075 - val_accuracy: 0.5366 - val_mae: 0.5691\n",
      "Epoch 26/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2778 - accuracy: 0.9545 - mae: 0.2007\n",
      "Epoch 00026: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2778 - accuracy: 0.9544 - mae: 0.2006 - val_loss: 5.2580 - val_accuracy: 0.5525 - val_mae: 0.5123\n",
      "Epoch 27/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3266 - accuracy: 0.9329 - mae: 0.2146\n",
      "Epoch 00027: val_accuracy did not improve from 0.57030\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3263 - accuracy: 0.9330 - mae: 0.2146 - val_loss: 5.7457 - val_accuracy: 0.5366 - val_mae: 0.5568\n",
      "Epoch 28/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3311 - accuracy: 0.9302 - mae: 0.2476\n",
      "Epoch 00028: val_accuracy improved from 0.57030 to 0.58416, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3315 - accuracy: 0.9301 - mae: 0.2476 - val_loss: 1.1329 - val_accuracy: 0.5842 - val_mae: 0.4472\n",
      "Epoch 29/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2919 - accuracy: 0.9532 - mae: 0.2083\n",
      "Epoch 00029: val_accuracy did not improve from 0.58416\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2914 - accuracy: 0.9533 - mae: 0.2083 - val_loss: 5.3200 - val_accuracy: 0.5663 - val_mae: 0.5416\n",
      "Epoch 30/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2956 - accuracy: 0.9567 - mae: 0.2314\n",
      "Epoch 00030: val_accuracy did not improve from 0.58416\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2953 - accuracy: 0.9568 - mae: 0.2313 - val_loss: 5.8838 - val_accuracy: 0.5564 - val_mae: 0.5645\n",
      "Epoch 31/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3031 - accuracy: 0.9532 - mae: 0.2274\n",
      "Epoch 00031: val_accuracy did not improve from 0.58416\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.3034 - accuracy: 0.9530 - mae: 0.2277 - val_loss: 5.4315 - val_accuracy: 0.5564 - val_mae: 0.5377\n",
      "Epoch 32/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2838 - accuracy: 0.9549 - mae: 0.2182\n",
      "Epoch 00032: val_accuracy did not improve from 0.58416\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2836 - accuracy: 0.9548 - mae: 0.2183 - val_loss: 5.4265 - val_accuracy: 0.5743 - val_mae: 0.5127\n",
      "Epoch 33/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2977 - accuracy: 0.9536 - mae: 0.2136\n",
      "Epoch 00033: val_accuracy improved from 0.58416 to 0.70297, saving model to ./data/best_model.h5\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2978 - accuracy: 0.9535 - mae: 0.2138 - val_loss: 0.7937 - val_accuracy: 0.7030 - val_mae: 0.3084\n",
      "Epoch 34/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2730 - accuracy: 0.9448 - mae: 0.1932\n",
      "Epoch 00034: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2726 - accuracy: 0.9449 - mae: 0.1932 - val_loss: 4.2369 - val_accuracy: 0.5564 - val_mae: 0.4833\n",
      "Epoch 35/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2457 - accuracy: 0.9576 - mae: 0.1567\n",
      "Epoch 00035: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2455 - accuracy: 0.9575 - mae: 0.1568 - val_loss: 5.3411 - val_accuracy: 0.5604 - val_mae: 0.5053\n",
      "Epoch 36/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2644 - accuracy: 0.9567 - mae: 0.1552\n",
      "Epoch 00036: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2640 - accuracy: 0.9568 - mae: 0.1551 - val_loss: 5.1534 - val_accuracy: 0.5624 - val_mae: 0.4878\n",
      "Epoch 37/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2184 - accuracy: 0.9549 - mae: 0.1434\n",
      "Epoch 00037: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2181 - accuracy: 0.9550 - mae: 0.1432 - val_loss: 1.3293 - val_accuracy: 0.5743 - val_mae: 0.4217\n",
      "Epoch 38/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2214 - accuracy: 0.9572 - mae: 0.1257\n",
      "Epoch 00038: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2212 - accuracy: 0.9572 - mae: 0.1256 - val_loss: 0.9409 - val_accuracy: 0.5663 - val_mae: 0.4279\n",
      "Epoch 39/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2133 - accuracy: 0.9605 - mae: 0.1342\n",
      "Epoch 00039: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2130 - accuracy: 0.9605 - mae: 0.1342 - val_loss: 4.3395 - val_accuracy: 0.5584 - val_mae: 0.4714\n",
      "Epoch 40/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2267 - accuracy: 0.9485 - mae: 0.1333\n",
      "Epoch 00040: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2263 - accuracy: 0.9486 - mae: 0.1332 - val_loss: 1.6343 - val_accuracy: 0.5663 - val_mae: 0.4455\n",
      "Epoch 41/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2250 - accuracy: 0.9578 - mae: 0.1256\n",
      "Epoch 00041: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2247 - accuracy: 0.9579 - mae: 0.1255 - val_loss: 3.9988 - val_accuracy: 0.5624 - val_mae: 0.4715\n",
      "Epoch 42/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2334 - accuracy: 0.9583 - mae: 0.1270\n",
      "Epoch 00042: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2331 - accuracy: 0.9583 - mae: 0.1269 - val_loss: 4.3176 - val_accuracy: 0.5822 - val_mae: 0.4446\n",
      "Epoch 43/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1989 - accuracy: 0.9598 - mae: 0.1197\n",
      "Epoch 00043: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.1987 - accuracy: 0.9599 - mae: 0.1196 - val_loss: 2.9291 - val_accuracy: 0.5782 - val_mae: 0.4437\n",
      "Epoch 44/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2297 - accuracy: 0.9598 - mae: 0.1236\n",
      "Epoch 00044: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2294 - accuracy: 0.9599 - mae: 0.1235 - val_loss: 3.2768 - val_accuracy: 0.5762 - val_mae: 0.4386\n",
      "Epoch 45/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2610 - accuracy: 0.9605 - mae: 0.1317\n",
      "Epoch 00045: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2605 - accuracy: 0.9605 - mae: 0.1317 - val_loss: 5.0857 - val_accuracy: 0.5663 - val_mae: 0.4738\n",
      "Epoch 46/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2804 - accuracy: 0.9600 - mae: 0.1448\n",
      "Epoch 00046: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2833 - accuracy: 0.9599 - mae: 0.1451 - val_loss: 5.4214 - val_accuracy: 0.5743 - val_mae: 0.4909\n",
      "Epoch 47/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2393 - accuracy: 0.9596 - mae: 0.1319\n",
      "Epoch 00047: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2389 - accuracy: 0.9597 - mae: 0.1318 - val_loss: 5.1795 - val_accuracy: 0.5663 - val_mae: 0.4712\n",
      "Epoch 48/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2318 - accuracy: 0.9651 - mae: 0.1092\n",
      "Epoch 00048: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2349 - accuracy: 0.9649 - mae: 0.1094 - val_loss: 5.4850 - val_accuracy: 0.5663 - val_mae: 0.4717\n",
      "Epoch 49/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2123 - accuracy: 0.9644 - mae: 0.1280\n",
      "Epoch 00049: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2119 - accuracy: 0.9645 - mae: 0.1280 - val_loss: 5.3403 - val_accuracy: 0.5941 - val_mae: 0.4632\n",
      "Epoch 50/50\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2528 - accuracy: 0.9567 - mae: 0.1746\n",
      "Epoch 00050: val_accuracy did not improve from 0.70297\n",
      "4536/4536 [==============================] - 6s 1ms/sample - loss: 0.2524 - accuracy: 0.9568 - mae: 0.1745 - val_loss: 1.8794 - val_accuracy: 0.6139 - val_mae: 0.4030\n",
      "Executing op DestroyResourceOp in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op DeleteIterator in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op DeleteIterator in device /job:localhost/replica:0/task:0/device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "# fit network\n",
    "history = model.fit(x_train, y_train, epochs=epochs_num, batch_size=16, validation_split=0.1, verbose=1, shuffle=True, callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the accuracy level during the training epochs. One point to observe is that at the beginning, the accuracy rate jumps pretty quickly and afterwards fluctuates heavily. This behaviour is due to our small and non-diverse training dataset. With a larger dataset which is also more uniformly distributed, you will be able to experience a smoother convergence path and higher accuracy rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xV9fnA8c+TkLD3ClM2AiorDEVcBQVFkIo422r7Ey3aYq2zw2pbW+veRWvBgSI4EAcK1IUKCAEJG9kQZgIkECCEJM/vj++95Ca5SW7GzU1ynvfrdV43955zz3lOAuc533lEVTHGGONdUZEOwBhjTGRZIjDGGI+zRGCMMR5nicAYYzzOEoExxnicJQJjjPE4SwTGmBITkRtF5NtIx2HKhyUCY4zxOEsEpkoRp8r8u61q8Rpvsn+gpsRE5D4R2SwiR0RkrYiMzbf+ZhFZF7C+n+/zdiLyvogki8gBEXne9/mDIjIt4PsdRERFpIbv/Vci8rCIfAccAzqJyE0Bx9giIrfki2GMiKwQkcO+WEeIyFUisizfdr8XkQ8KOc8SH6OIeM8RkaUikuZ7PSdgPzf69n9ERLaKyPW+z7uIyNe+76SIyIwi/iaDRWShiKSKSKKIXBCw7isR+aeILPHta7aINAlYP1pE1vi++5WI9AhYF/RvFrD+cRE55It7ZHHnZCopVbXFlhItwFVAa9yNxNXAUaBVwLpdwABAgC7AaUA0kAg8BdQFagHn+r7zIDAtYP8dAAVq+N5/BewAegE1gBjgMqCz7xjn4y64/XzbDwTSgOG+GNsApwM1gYNAj4Bj/QBcWch5lvgYhcTbEjgE/Mz3/lrf+6a+38VhoLvvu62AXr6fpwN/9O3/1O8rSJxtgAPApb5th/veNw+IZxdwhu947/l/30A3399vuO/3eg+wCYgt5m92I3ASuNm33a+B3b7fVaHnZEvlXCIegC1VfwFWAGN8P88FJgXZ5mwg2X9xz7fuQYpPBH8tJoYP/McFXgKeKmS7fwMP+37u5bsg1wzxPEM9Rp54fQlgSb5tFvkupnWBVOBKoHa+bV4HXgbaFhPXvcAb+T6bC/wiIJ5HAtb1BDJ9F/A/AzMD1kX5ksYFxfzNbgQ2Bbyv4/ubxRV1TrZUzsWqhkyJicjPfVUiqSKSirvTbOZb3Q7YHORr7YDtqppVysPuzBfDSBFZLCIHfTFcGkIMAK8B14mI4C7QM1X1RLANy3CM/PG2BrbnW78daKOqR3GlqluBPSLyiYic7tvmHtwd9hJf1c0vCznWacBV/r+HL9ZzcXfiweLZjrv7b5Y/NlXN8W3bhuL/ZnsDvnfM92O9Ys7JVEKWCEyJiMhpwH+A24GmqtoIWI27YIG7iHQO8tWdQHt/vX8+R3F3lH5xQbY5NU2uiNTEVW88DrT0xTAnhBhQ1cW4u+GhwHXAG8G2K8sx8seLqzI5Ld/69rg7b1R1rqoOx1241+N+v6jqXlW9WVVbA7cAL4pIlyDH2okrETQKWOqq6iMB27TLd+yTQEr+2HwJsp0vtqL+ZkUq7JxM5WSJwJRUXdxFLhlcgyquROD3CnCXiPQXp4sveSwB9gCPiEhdEaklIkN831kBnCci7UWkIXB/MTHE4ur7k4EsXyPlxQHr/wvcJCI/EZEoEWmT7470deB5IEtVC+sLX9ZjBJoDdBOR60Skhohcjaue+VhEWvoaa+sCJ4B0IBvA17jd1rePQ7jfe3aQ/U8DLheRS0Qk2ve7vSDguwA3iEhPEakD/BV4V1WzgZnAZb7ziAF+74tjIUX/zQpV1DmZyskSgSkRVV0LPIGr494HnAl8F7D+HeBh4C3gCK5evYnvonM5rvF4B5CEqz5AVecDM4CVwDLg42JiOAL8FncRO4S7s/8wYP0S4CZcI2ca8DV578jfwCWvoKWBcjpG4L4OAKNwF9kDuCqfUaqagvs/+HvcnflBXKP0RN9XBwDfi0i679iTVHVrkP3vBMYAf8Alrp3A3eT9//0G8CquOqeW79xQ1Q3ADcBzuBLC5cDlqppZ1N+sGEWdk6mERNUeTGO8RURqA/txPYA2RjqecBORr3CN8a9EOhZTOVmJwHjRr4GlXkgCxoSixI1AxlRlIrIN1+B7RYRDMabSsKohY4zxuLBVDYnIFBHZLyKrC1kvIvKsiGwSkZXim4bAGGNMxQpn1dCruC56rxeyfiTQ1bcMwo34HFTcTps1a6YdOnQonwiNMcYjli1blqKqzYOtC1siUNUFItKhiE3GAK+rq5taLCKNRKSVqu4par8dOnQgISGhHCM1xpjqT0Tyj24/JZK9htqQd9h7ku8zY4wxFSiSiUCCfBa05VpEJohIgogkJCcnhzksY4zxlkgmgiTyzn/SFjcSsQBVfVlV41U1vnnzoFVcxhhjSimSieBD4Oe+3kODgbTi2geMMcaUv7A1FovIdNyc5s1EJAn4C27qW1R1Mm4irktxD8E4hpu3xRhjTAULZ6+ha4tZr8Bt4Tq+McaY0NhcQ8YY43E215AxxlSQk9knWZ+ynsR9iWw6uIkoiaJGVI08S60atRjYZiBntTyLKKmYe3VLBMaYiMvRHPYf3c+OtB3sTNvJjrQdHD5xmCt7XskZLc4ofgdB9peVk3VqOX7yOEcyj3D4xOE8y7GTx4J+X1U5mXOSzOzMAku0RBMbHVtgqRFVgyiJQkSIkqhTy4FjB0jcl8iKvStYk7yGzOzMkM6hRd0WDOs0jOGdhjO803DaNAjfMKsqN+lcfHy82shiU5mdzD7JzsM7ad+wPTWiyn6vlZ2TzZ70PWxP3c621G1sS93G9rTt7EnfQ8u6LenUuBOdGneiY6OOdGrciWZ1miEiZGZnkpaRRmpG6qlFUerF1iuw1K5RG/eUyqKlZqTyyvJXeD3xdTKyMk5dBGOiY079XD+2Po1qNaJhzYY0qtXo1KIoKcdSTi0Hjh8g5VgKe9P3knQ4qdAL5HmnncdtA25j7OljiYmOybMuR3NYuHMh7659lw/Wf8DuI7vJyslCgw9JKpNoiSYmOoYczQn5Yu7Xom4L+sT1oXfL3vSJ60OfuD50a9oNQfIkrKycLA6fOMyC7QuYv2U+87fMZ//R/QD0bN6TOwffya/6/apU8YvIMlWND7rOEoEJp8zsTGKjY0PePiMrg6ycLOrF1gtjVCVzIusEaSfSEPLe6UVJFCeyT7Bm/5pTd3yJ+xJZvX81mdmZnN32bGaMm0G7hu2KP0gQOZrDwwse5h/f/oOMrIw861rUbUFcvTj2pe9j39F9edbVjamLooXe7QbTuFZjRnYdyehuoxnRZQQNazXMs37zwc088/0zTPlhCkdPHuXc9ufSrkG7AnfLJ7JPkJ6ZTmpGKmkZaaSdSCtwrNjoWJrXaU6zOs1oWqcpLeq2oH2D9rRv6JZ2DdvRvmF7VJUpP0zh3wn/ZmvqVlrVa8WE/hP4Zd9fsvHARt5b9x6z1s9ib/peakbX5OLOF9OreS9iomMKVLfUjK5Jw1oNqR9bnwY1G5xa6sTUKTQBBt7tx0TFEB0VfWqdqpKVk5XnvLNzssnRHHI0B0VP/Vwvth5x9YI9hrt4OZrDqn2rmL9lPvM2z+PKHldyS/wtpdqXJYJq5MCxA6RmpNK5SVHPTXdyNIdnFj/Dowsf5cmLn+TaM4vsyFVuTmaf5MMNH/Kf5f9h3uZ5nNPuHCYNmsTYHmMLvUPekbaD575/jv8s/w/pmen0b92fizpcxIUdL2RIuyHUja0b1pjTMtL4cMOH/LD3B/am782zHMo4FNI+mtdpfupur2ntpjz8zcPERMfw+hWvc1m3y0oUT3pmOr/44Be8v+59ruxxJcM6DaNDow50aNSB9g3bUyemzqltj2YeZVvqNrYc2sLW1K1sPbSV6KjoPHfj/jv0KIkiPTO9wLIuZR2fbPyElGMp1IiqwXmnncfobqPp2rQrLy97mQ83fEiNqBpce+a13DHoDvq26hvSeWTnZHMk8wiHjh9CRGhWpxl1Y+qGVPoI3Mdnmz7jhaUv8Nmmz07d7deuUZtLu17KuJ7juKzrZdSvWb9Ev2OvsURQTSzfs5zR00ezJ30PE/pN4K8X/pXmdYOPtN6Wuo0bP7iRr7d/TfM6zTlw/ADTr5zO+F7jwxbfpoObeGX5K0xdMZX9R/fTtkFbxnQfw6ebPmXLoS20a9CO2wbcxs39b6ZJ7SYALE5azFOLn+K9te8BMK7nOLo26cqX277k+13fk5WTRUxUDIPaDmJ0t9H8dtBvqVmjZrnE67/4z1w7k3mb55GZnUndmLq0qt+KuHpxtKzbkrh6ccTVi6NxrcYAp+7y/EuURNGjeQ96t+xNXL24PBe4jQc2ctU7V5G4L5F7zrmHv1/09wJVG8FsObSFK96+gjXJa3ji4ieYNGhSiS6cpZWdk83ipMV89ONHfPTjR6xNXgtA09pN+XX8r5k4YCKt6rcKexxF2XxwMzPXzKRb026M6DIi7DcI1YklgkoqKyeLtIw0mtRuUux/9A/Wf8D1719P09pNubTrpbyy/BXqxdbjgfMf4PaBt5+qflFVXl3xKpM+mwTAsyOfZVzPcVz65qUs3LmQGeNmcGXPK0sV78p9K3kj8Q1O5pwsUK+5NXUrC7YvIFqiGdVtFDf3u5kRXUYQHRVNdk42n2z8hKcXP82X276kdo3aXHvGtaxNWcvipMU0rNmQCf0ncPvA22nfsP2p46VnpvPdju/4ctuXfL71cxJ2J3BGizOYOmYq8a2D/nsulqoye8Ns/vvDf09d/Ns1aMdVPa/iql5XMbDNwHLtqZGRlcHvPvsdk5dNZki7IUy/cnqRVUVfbP2Cq965ClVlxrgZDO88vNxiKanNBzezLmUdP+n4E2rH1I5YHKZ8WCIoZ9NXTWfXkV3cefadIV00dh3exeMLHyfpSBL7j+4/tRw8fhCAAa0H8MD5D3BZ18sKJARV5bGFj3Hf/+5jQJsBzL5mNnH14liXvI47593JZ5s+o2uTrjxx8RMMajuICR9NYPaG2Zx/2vm8esWrdGjUAYAjJ44w4s0RLNm1hHevepcxp48p8XmPmDaC+VvmUz+2foE62Ia1GnJ1r6u5sc+NtK7futB9rNy3kme/f5Y3V71Jm/ptmDRoEjf1vSmkNoE5G+dw80c3sy99H/cOuZcHzn+gRKWD1IxUJn4ykemrp4f14h/M9FXTmfDxBGpG12TSoEm0a9iO1vVb07p+a1rVa0WT2k14fsnz/G7u7+jerDuzr5lNlyZdwhqT8RZLBOXof1v+xyXTLiFHcxjdfTTTxk4rsm5ycdJifjrjpxw4foBOjTvRom4Lt9RxrzHRMfxn+X/YlrqN/q3688D5D3B5t8tP9fq49eNbmbpiKuN7jefVMa8WuDObs3EOd869kw0HNlCrRi1UlX/85B/cMfiOAhe3wycOc/EbF7N8z3Lev/p9RnUbFfJ5Hzx+kJaPt+T3Z/+eR4Y9UrJfWhAnsk4QEx1T4gtwakYqv5v7O15d8Sq9mvfi1SteDal08O2Ob7nh/RtIOpzEQxc8xH3n3pen8a8i/HjgR65//3oSdhf89xsbHUtmdiaju4/mjbFv0KBmgwqNzVR/RSUCVLVKLf3799dI2XJwizb5VxM948Uz9PHvHtfoh6L1zBfP1K2HtgbdfuoPUzX2b7Ha6ZlOumrfqkL3m5mVqVOWT9HOz3RWHkT7TO6j01dN1/Onnq88iP75iz9rdk52kd9/etHTevlbl+vqfauLPIfU46k64OUBGvu3WP3kx09COm9V1f8u/6/yIJqwKyHk74TTJz9+oq2faK3RD0Xr3fPu1mW7l+nJ7JMFtsvMytQ/ff4njXooSjs/01kX71wcgWjzOpZ5TDcf3KzfbP9GZ6yeoU8tekrvmXePPv/980X+nY0pCyBBC7muWokgREczjzJkyhC2p20n4eYEOjfpzPzN8xn/7nhqRNVg1tWzOLf9uYCr+79r3l088/0zDOs0jBnjZpxqHC1KVk4Wb658k79/83c2HdxEbHQsU0ZP4fqzri/Xczl0/BDD3hjGmv1rmP+z+Qw9bWix3xn55kg2pGxg8283V0jDZShSM1K5c+6dTF0xFYA6MXUY2GYg57Q9h3PanUOr+q2Y+MlEvt/1PTf2uZFnRzxrPUuMZ1nVUBmpKte+dy0z18xkzvVzGNFlxKl1Px74kcunX87WQ1uZPGoyY7qP4ep3r+bzrZ9zx6A7eOzix0o8qCgrJ4tZ62bRqXEn+rfuX96nA7iqnt6Te9OtaTc+//nnRW576PghWjzegjsH38m/hv8rLPGUxY60HSzcuZCFOxeyKGkRP+z5gWzNBqBRrUa8NOqlsPaWMqYqsKqhMnr020eVB9FHvnkk6PqDxw7q8NeHKw+iTf/VVGP/FqtTf5hasUGWwj8W/EN5EF27f22R2039YaryILokaUkFRVY2RzOP6tfbvtYXlrygO1J3RDocYyoFrGqo9OZvns+IN0cwruc43r7y7UKrRbJysrh73t18vPFj3hj7BoPbDq6wGEsr+WgybZ9qyy39b+HZkc8Wut1lb13G2uS1bPntlkpTLWSMKZmiSgQ2DXURthzawtXvXk2v5r2YMnpKkRfBGlE1eGrEU2z8zcYqkQQAmtdtzvhe43kt8TXSM9ODbpOakcr8zfMZ12OcJQFjqilLBD6qyu4ju/nkx0/429d/Y+yMsQx6ZRAAH1zzQbUdwTgxfiKHTxzmzZVvBl3/4YYPOZlzknE9x1VwZMaYiuL5aag3pGzg/s/vZ+HOhXkm7+rWtBs/6fgTfjf4d3Rq3CmCEYbX4LaD6RvXlxeWvsCE/hMK3PW/s/Yd2jdsz8A2AyMUoTEm3DybCLJzsnnm+2f44xd/pHaN2ow5fQx94/rSr1U/erfs7ZluhiLCxAETufmjm/l2x7d5upKmZaQxb/M8bh9wu1ULGVONeTIRbDq4iZtm38S3O75ldPfRvDTqpVJPE1sdXHfmddw9/25eTHgxTyL4cMOHZGZnclWvqyIYnTEm3DzVRpCjOTy/5Hl6T+7Nqn2reO2K1/jg6g88nQTADcS6qc9NvLf2Pfam7z31+bvr3qVtg7ZWLWRMNeeZRLAtdRvDXh/Gbz79Deeddh6rJ67m571/blUePr+O/zUnc07yyvJXADcv0dxNcxnXY1yFPTfVGBMZnvkfvnr/ahJ2J/DK5a8w57o5tG3QNtIhVSpdm3bl4s4X89Kyl8jKyeKjDR9xIvuEVQsZ4wGeaSMY1W0UWydtpWmdppEOpdKaGD+RK2ZcwUcbPuKdte/Qpn6bKjMmwhhTep5JBIAlgWKM6jaK9g3b89jCx1i+Zzm39L/FqoWM8QD7X25OiY6K5tb+t7IoaZFVCxnjIZYITB6/6vcrYqNjaVWvFee0OyfS4RhjKoCnqoZM8VrUbcHjwx+nUa1GVi1kjEdYIjAF/GbQbyIdgjGmAtktnzHGeJwlAmOM8ThLBMYY43GWCIwxxuMsERhjjMdZIjDGGI8LayIQkREiskFENonIfUHWNxSRj0QkUUTWiMhN4YzHGGNMQWFLBCISDbwAjAR6AteKSM98m90GrFXV3sAFwBMiEhuumIwxxhQUzhLBQGCTqm5R1UzgbWBMvm0UqC/uoQD1gINAVhhjMsYYk084E0EbYGfA+yTfZ4GeB3oAu4FVwCRVzcm/IxGZICIJIpKQnJwcrniNMcaTwpkIgj36S/O9vwRYAbQG+gDPi0iDAl9SfVlV41U1vnnz5uUfqTHGeFg4E0ES0C7gfVvcnX+gm4D31dkEbAVOD2NMxhhj8glnIlgKdBWRjr4G4GuAD/NtswP4CYCItAS6A1vCGJMxxph8wjb7qKpmicjtwFwgGpiiqmtE5Fbf+snA34BXRWQVrirpXlVNCVdMxhhjCgrrNNSqOgeYk++zyQE/7wYuDmcMxhhjimYji40xxuMsERhjjMdZIjDGGI+zRGCMMR5nicAYYzzOEoExxnicJQJjjPE4SwTGGONxlgiMMcbjLBEYY4zHWSIwxhiPs0RgjDEeZ4nAGGM8zhKBMcZ4nCUCY7wuLQ1+/DHSUZgIskRgjNf94Q8waBBkZ0c6EhMhlgiM8bpvvoHUVNi4MdKRmAixRGCMlx0+DKtXu5+XL49sLCZiLBEY42VLloCq+/mHHyIbi4kYSwTGeNnixe61WzcrEXiYJQJjvGzRIujRAy680CUCf+nAeIolAmO8StWVCM4+G/r1cw3G27ZFOioTAZYIjPGqjRvh4EEYPBj69nWfWTuBJ9UobIWI9Cvqi6pqFYrGVGWLFrnXs8+GLl0gOtpVD/30p5GNy1S4QhMB8EQR6xS4qJxjMcZUpMWLoUED6NkToqKgVy9rMPaoQhOBql5YkYEYYyrYokUwcKBLAuCqh+bOjWxMJiKKbSMQkToi8icRedn3vquIjAp/aMaYsDlyBFatctVCfv36wd69sGdP5OIyERFKY/FUIBM4x/c+Cfh72CIyxoRfQgLk5BRMBGDVQx4USiLorKqPAicBVPU4IGGNyhgTXv6G4kGDcj/r3RtELBF4UCiJIFNEauMaiBGRzsCJsEZljAmvxYuhe3do0iT3s/r1oWtX60LqQaEkggeBz4B2IvIm8DlwTziDMsaEkaorEQweXHBdv35WIvCgYhOBqs4DfgrcCEwH4lX1q/CGZYwJm82bISUlb/uAX9++sH07HDhQ8XGZiAml19CHwMXAV6r6saqmhD8sY0zY+CeaC5YI/A3GVj3kKaFUDT0BDAXWisg7IjJORGqFOS5jTLgsWgT16rkBZPnZVBOVV2am6+kVBqFUDX2tqhOBTsDLwHhgf1iiMcaE3+LFbiBZdHTBdU2bQvv21k5Q2axe7f5mL7wQlt2HNOmcr9fQlcCtwADgtRC/N0JENojIJhG5r5BtLhCRFSKyRkS+DjVwY0wpHD0KiYnBG4r9rMG48sjJgSefhP793UC/Tp3Ccpii5hoCQERmAINwPYdewLUVFFs+EZFo3/bDcYPQlorIh6q6NmCbRsCLwAhV3SEiLUp3GsaYkCQkuIfUB2sf8OvXD2bPdqOP69evuNhMXjt2wI03wpdfwpgx8J//QPPmYTlUqCOLO6vqrar6RShJwGcgsElVt6hqJvA2MCbfNtcB76vqDgBVtSonY8LJ31BcVImgb1/XxTQxsWJiMnmpwrRpcOaZsHQp/Pe/MGtW2JIAhJYIFgD3l2KuoTbAzoD3Sb7PAnUDGovIVyKyTER+HmxHIjJBRBJEJCE5OTmEQxtjglq0yE053axZ4dvYVBORc/QoXHMN/OxnLhEkJsIvf+lGfIdROOcaChZ5/ufg1QD6A5cBlwB/FpFuBb6k+rKqxqtqfPMwZkVjqrXAJ5IVpVUraNnSEkFFO3IERoyAd9+Ff/wDvv46bG0C+YVzrqEkoF3A+7bA7iDbfKaqR33jExYAvUPYtzHVS1ISLFwY3mNs2wb79hWfCERc9ZB1Ia04qalw8cWuxDZ9Otx/f/BeXWESzrmGlgJdRaSjiMQC1wAf5ttmNjBURGqISB1co/S6kKM3pjrIyoLLLoMhQ2D8+PBNA+2faK6o9gG/fv1gzRrIyAhPLBVl714YOxZat4Zzz3WNr3//u7vYLl0Kycmu8TySDh6EYcNg2TJXGhg/vsJDKLbXEPAX8s41NAQ33USRVDVLRG4H5gLRwBRVXSMit/rWT1bVdSLyGbASyAFeUdXVpTsVY6qoF1+ElStd3fCsWe7hMI88ArfckvvQmPKweDHUqePqnovTr5+7QK5aBQMGlP3YKSlw/Di0a1f8tuVl1iy4+WZX7/7Tn8KuXfC//8FrQXq/N2zoJuBr0gQaN4YWLWDkSNdbJ5SeU9u3Q2ysq1YriZQUlwTWrXPxXnZZyb5fXlS12AVoiqvHHwU0C+U74Vr69++vJsx+/FF1+/ZIRxF5WVmq27apfv656uLF4TnG3r2qDRqoDh+umpOjumGD6kUXqYLq4MGqK1eW/RgnT6o+9ZRq3bruOKHYssXFMHly6Y+bk6P67beq116rGhPj9te9u+pvf6v6ySeqR4+Wft9FSUtTvekmd7x+/VTXrs27/uhR1dWrVT/4QPWZZ1QffNDFdMMNqpdeqnr22aqtWrnv166tOn686qxZqhkZufs4ckT1o49Ub79dtUsXty2onnWW6t13q/7vf3m3D2bvXtVevVRr1VKdO7f8fw/5AAlayHVV3PqCKuvD6+Pj4zUhISESh/aOQYNccXXNGneXU5zUVJgzB66+ukLrNctdYiJMneomZdu0CbZsccP6/aZNg+uvL99j3ngjvPWWu/Pu3t19pgpvvAF33glpaXDHHXDrrdC5c8n3v2wZTJjgGn5HjoSXXgrtrlzV3R1ffTVMnlyyYx49Cm++6Uo6iYnubvvGG+G002DePPjqK1flFBsL550Hl1ziljPOKLp3TE6Oa7dYsMA9a7lLF7e0apVbcvrmG/j5z10f/PvvhwceCO3fcLBjLVzoqpDeecdVITVsCJdfDrt3u+OcPOlKWBdc4Or3MzJcae7bb3PXXXihGwwW7P/FW2/Bzp3w0UdwUfgfAS8iy1Q1PujKwjIE8GURyxeFfS/ci5UIKkDr1u7u5tlnQ9v+uuvc9o8+Gt64wunoUdW2bd3d2Vlnqf70p6r33KP68suuRHDBBaqxse4Ot7x8+637vd13X/D1ycmqN96Ye7c5YIDqk0+q7tpV/L4PH1a94w7VqCjVuDjVmTPdHXpJXHihO2Z+aWmqK1a4u96ZM12p4Z//dHfCP/uZasOGuXfHL72kmp6e9/vHjrk74DvvdHfE/vNr3drdyb/9tmpKitt2zx7V115z/8aaN8/dNnCpVcvtZ9gwVRHVTp1Uv/uuZOdalJMnVT/7TPUXv1Bt3Fj1zDNV77pLdf581ePHC25/5Ijqhx+q3nZb3tJC/qVZM9UFC8ovzmJQmhJBZWUlgjBTdXdQWVlu3pnNm92dUGG+/trdETVt6rq/LVvm7uyqmn/8A/74R3c+551XcP3Bg66R9dAh+P77snfry8qC+Hg33fP69VC3buHb7tgBM2a4u9MffnB3zeefD1ddFXyQ0aFD8Le/uTrxW29159aoUcljvOsueP55+GvlQkgAAB+YSURBVMMfXAlp0yb372F/IeM+a9Z0/w7OPx9uuw3OOSe0/u87d7qSwty5rg7/0CH3vQ4dYOtWt03z5u6u+5JL4Cc/gRMn8sa0aZPb9txz4V//cpPqVRaFTRQnEvbxAXkPV3iJwBKByevQIVclcMMNrirk3ntdw2UwJ0+6bobp6e4COmAAtG3rGiVLUxyPlP37XRXDRRfBBx8Uvt2PP7pk0KqVqzYoLEFmZLgpGvr0ya3uye/55+E3v4GZM90FPVQbNriEMH26i6cwZ5wBL79cfFfRonzwgetxI+L+rv6qmM6dXSJs2TK3cbVJE6hdu/TH8svOdr155s6FFSvcRGuXXOJ+l+XZcO5BpaoaqqyLVQ2VQnJy6NuuX++KrdOmuWJ+zZqFNxw/+aTbdtYs9/799937P/+57DFXpIkTVaOj3bkX54svVGvUUL34YldlECg7W/XNN1Xbt3e/h+hoVz2wf3/e7fbtc9Unw4aVvLrGLydHddMm1+iZf1m7tmBspT3Gjh3Bqz9MlUMRVUMRv7CXdLFEUAI5Oap/+Yv7M4daZ7pggdt+3jyXAGrWdL0p8tu9W7V+fdURI/JezH7+c3cBXLKkXE4h7Navd/FOnBj6d155xf2OJk7MPfcFC1x9Oqj27as6e3ZugmnQQPWRR3IvqDfd5HrRrFtX/udjTCHKlAiA93BdR6OK27YiFksEIcrKUp0wQU81TL36amjfe/ddt/2KFe79ffe598uW5d3uhhtc4+mPP+b9/NAh1+h6+umuUbAksrNVb7lFdcgQ1Ycecl02s7JKto+SGjPGJbR9+0r2vbvvdr+XP/1JdexY93ObNq5hMzs7d7u1a1Uvv9ytb9/enRe4hmhjKlBZE8Ew4E1gM/AIcHpx3wnnYokgBMeP516c7rjDvT7ySGjfffFFt/2ePe59aqrr3XDhhXnvfkH1j38Mvo/583OPHaqcnNxYe/RwvT9AtUkT1auvVp0yxfW7Lk9ff+2O8fDDJf9uVpZLIqBar57q3/9edL/4zz93JQV/wjhypPRxG1MK5VI1BDTEPZhmJ7AQuAmICfX75bVYIijGoUOqQ4e6C+kzz7jP6tUL/aL84IPun0VgHfNzz7nPPv7YfX7mme7utqgL3223ue98+WVox33sMbf9pEkuKSQnq771luuyFxenp7rblbZOPb/sbFeV06ZN6Qc2paer/vvfoSeo7GxX4lq+vHTHM6YMypwIcCOLJwEJuPmCrgaewz2kxhJBZbFrl7tIx8S4vth+nTu70Z2hmDhRtWnTvJ9lZqp27eru1J94wv2zef/9oveTnu6+c9ppxVe7vPGG2+f48XmrVfxyctxdO7i2ifIwfbqWqMrMmCquqERQbH8sEXkf+AaoA1yuqqNVdYaq/gaoRJ11PW7JEtdve+vW3FG+fnFxbvKtUOzf7+ZZCRQT4/pmr1vn+pZfcglccUXR+6lbF15/3Y3C7NoVHn00+ARm8+bBTTe5EZivvx68i6B/Nkxwo33L6sQJN+q0d2/XTdYYjwulY+7zqtpTVf+pqnmmRdTC+qSairN9O1x3nZsWIjPT9ecfNizvNmVNBOAu/OeeCzVqwLPPhjYQZvBgN8XAeee58Qinn+76v/sH2CQkuMnAevVyE27VrFn4vvwDuMojETz3nJuS+fHHq/aUGMaUk1ASQQ/fs4UBEJHGIjIxjDGZUKSluYtr9+7uIvrHP7rBRv2CTBHVsmXZE4GIGyS1dCl0K/DsoML16OHmUvn8czfo6LrrXIKYMcPNtNisGXz6adGjl8HNUyOSO9K0NJYvd4nn7rvdvDv5E6YxHhVKIrhZVVP9b1T1EHBz+EIyRTp50o1K7dIFHnvMVQFt3OjmWC9suty4ODdi+EQIj5EoLBGAu5D3LuVzgy66yJUAXn3VVRddc40bRTp3bmhT99aqBW3alK5EsHixSzr9+8MXX7iJyKZPL/l+jKmmQkkEUSK59QAiEg1UofkDqpl773VTE5x5pruwvvaaG/5flLg491rYHDF+J0+6OXUKSwRlFRUFv/iFmxrhuefcRbmwKRiC6dixZIlg4UIYPtxNs/D99/Dww64q7aGHii+BGOMhoTyYZi4wU0Qm455SdivuQTUmEubPd5NvffZZ6BNW+RPB3r1FT0GckuJew5UI/OrUgdtvL/n3OnVyk5KFYt8+1wDduLErOd16a+WaiMyYSiSURHAvcAvwa9yziucBr4QzKFOIo0dh7drcicBC1bKle923r+jt/CWGcCeC0urUyVUrZWS4qqKirFrlGs/feqtC5no3piorNhGoag7wb99iSmvTJleFU9wFrCiJia7HTf/+JfteYImgKFUhEai66p3iqpTWr3evPXqEPy5jqrhQxhF0FZF3RWStiGzxLxURXLUxZ47rS1+/vruI//rXMGUKrF5dsgdnL1vmXuNL2GvXXyIoLhH4SwyVNRF07OheQ2kn2LDB/b79SdAYU6hQGoun4koDWcCFwOvAG+EMqtqZOtV1k7zrLldn/dZb8KtfuQbfRo1cfX8oli1zF/XWrUt2/Jo13XGrQ4kAQksE69e7cQsV+OAPY6qqUBJBbVX9HPcQm+2q+iBgla6hSktz/eivvRb++c/cJzCtX++eS1ujhns4SSiWLXMlitJc3Fq2DK2NICam8vaoiYtzVWuhjCXYsKFkPZKM8bBQGoszRCQK2CgitwO7gEp6y1gJvfee678fOJVBVJS7SHXvDu++6x6EXZxjx3IbiksjlNHF/jEElfUuWiS0LqRHj7rHH55+esXEZUwVF0qJ4A7cPEO/BfoDNwC/CGdQ1cq0aa59YMCA4OuHDnUNyXv2BF/vV9qGYr+SJILKrFOn4hOB/xGOViIwJiRFJgLf4LHxqpquqkmqepOqXqmqiysovqpt50746itXGijsLnvoUPdaXKnA31Bc2kQQyjQTVSkRaBHP2vb3GLISgTEhKTIRqGo20D9wZLEpgenT3QXr+usL36ZvXzfAKpRE0KKFm2ahNOLi3EPmjx4tfJuqkgiOHHEjoAuzYYNLvF26VFxcxlRhobQR/ADMFpF3gFNXEVV9P2xRVRfTprnpDTp3LnybmBg3fXQoiaC0DcWQ241y377c3jeBVKtGIgjsQtq0afBt1q9325VlzIYxHhJKG0ET4ACup9DlvmVUOIOqFlaudKNbQ5nvfuhQt31qavD1x47BmjWlrxaC4geVHT0Kx49X/kQQShdS6zFkTImEMrL4pooIpNqZNs11DR0/vvhthw51d+TffedmycyvrA3FUPygsso+hsCvuEFlOTkuEVx4YcXFZEwVV2wiEJGpuMnm8lDVX4YlouogO9sNGhs50g0kK86gQa6K6JtvgieC0o4oDhRYNRRMVUkE9eq5GAsbS7BzpyvZWEOxMSELpY3g44CfawFjgd3hCaea+Ppr2LULnnwytO3r1HEX+cLaCcraUAzQvLlrX6jqJQIoeizBhg3u1aqGjAlZsW0EqvpewPImMB44I/yhVWJz5sBTT0FWVvD106a5eW4uvzz0fQ4d6p7+dfx4wXVlbSgGV03VvHn1SARFjSWwrqPGlFgojcX5dQXal3cgVcpDD8Gdd7rnAuSvajl+3I0WHjcOatcOfZ9Dh7oHw3z/fcH9rV1btvYBv6KmmfAngubNy36ccOvUCXbsCJ6IN2xwU2RUhYRmTCURyuyjR0TksH8BPsI9o8CbsrNdb6D+/d0jEPv2hQULctd/9JHr5x5Kb6FAQ4a4O/781UOJie6Y5ZEIihpdvH+/K8WUJHlFSqdO7neyc2fBdTbZnDElFkrVUH1VbRCwdFPV9yoiuEpp0yZ3l3777S4R1KvnHnzy6KOu58+0aa4u//zzS7bfxo3dbKSBSQXKPqI4UHGJoKrcRRfVc8i6jhpTYqGUCMaKSMOA941E5IpQdi4iI0Rkg4hsEpH7ithugIhki8i40MIuR5mZ8LOf5TYyFmflSvfauzecdZZ7bvDYse5ZwqNGwaefwnXXQXR0yWMZOhQWLcpb5bFsmauuKe65xKHwTzMRbHqGqpQIChtLcOSIa6S39gFjSiSUNoK/qGqa/42qpgJ/Ke5LvnmKXgBGAj2Ba0WkZyHb/Qv3bOSKt3Gju4ufPj207RMT3UXe/+SrBg3cNNJPPw3z5rmLeEmrhfzOO88N7Prhh9zPyqOh2C8uzs2EevhwwXVVKRG0besav/MnAptszphSCSURBNsmlG6nA4FNqrpFVTOBt4ExQbb7DfAesD+EfZa/5GT3unRpaNuvXOnuOAOnLxCBSZPcgLAXX3QlhdLwT0Dnrx46frzsI4oDFTW6uColguho6NCh4FgC6zFkTKmEkggSRORJEeksIp1E5ClgWQjfawMEtuYl+T47RUTa4MYlTA414HKXkuJeExKKntHSLzHRVQsFM3CgewxlabVq5eYl8jcYl2dDMRSeCHJyXEKsKokAgo8l2LDBPeuhqLmdjDEFhJIIfgNkAjOAmcBx4LYQvhesLiP/lfZp4F7fLKeF70hkgogkiEhCsv8Ovrz497d/f/BeKIFSU123xdLe8Ydi6FD49lt3cS6PEcWBCptm4uBBd7yqlAiCjSVYv959XrNmZGIypooKZa6ho0ChDb1FSALaBbxvS8ERyfHA275ZrpsBl4pIlqp+kC+Gl4GXAeLj40O4bS+BwMSydCm0L2KIRGBDcbicdx68+qq7qJVnQzEUPs1EVRpM5tepExw44No7GjRwn61fb+0DxpRCKL2G5otIo4D3jUUklIbdpUBXEekoIrHANcCHgRuoakdV7aCqHYB3gYn5k0DYpaRA3bpurp+EhKK3TUx0r+EuEYBrJyjPhmKAJk1cI2v+EkFVTQSQ206Qne0a/q19wJgSC6VqqJmvpxAAqnqIEJ5ZrKpZwO243kDrgJmqukZEbhWRW0sbcLlLTnZ182eeWXyD8cqVbhK5Vq3CF0/nzu7Off788m0oBld/3qJF9UgE+ccS7NgBGRlWIjCmFELp/ZMjIu1VdQeAiJxGkNlIg1HVOcCcfJ8FbRhW1RtD2We5S0521S9nnQVvv+0ajAu7A09MdNuFc9SqiKseeu+98m0o9gs2qKwqJoL8Ywn840CsRGBMiYVSIvgj8K2IvCEibwALgPvDG1YFSklxiSA+HtLS3MjhYLKzYfXq8LYP+A0d6o4H4UkEwdoIoqIKf+JXZdS4MTRqlJsIrOuoMaUWyhQTnwH9yO011F9VIzP4KxySk111z4AB7n1h7QT+qSXC2T7g528naNYM2rUretuSKqxE0KxZ6UZDR1LHjrltBBs2uOQQyvMfjDF5hDr7aDZuwFca0FNEzgtfSBVINbdqqGdPN0issHYCf0NxRZQIzjjD3e2WZ0Oxn38G0pyc3M+q0mCyQIFdSG2yOWNKLZQnlP0fMAnX/XMFMBhYhHuGcdV25Iib+rl5c9drqG/fwhPBypXujrlngVkyyl90tJvyIhyN0nFxrtrpwIHcKaerciL46KPcx1NeckmkIzKmSgqlRDAJGABsV9ULgb5AOY/qihD/GAJ/dcKAAbB8eW79fKDERHfHWVGDlUaMCE/pI9hYgqqcCDIzXWlgzx5rHzCmlEJJBBmqmgEgIjVVdT1QPfro+ROB/844Ph6OHYN16wpuu3JlxVQLhVuwaSaqaiLwdyH97DP3al1HjSmVUBJBkm9A2QfAfBGZTXV5ZrF/niF/IiiswfjQofBPLVFR8k8zceKE6y1VFROBvwvpHF8PZSsRGFMqofQaGquqqar6IPBn4L9ASM8jqPTyVw116+ae0pW/nWDVKvdanUoE/qoh/++gKiaC005zjcMLFrh2FX9iMMaUSCgDyk5R1a/DFUhE5K8aiopyPXXylwgqYmqJitKggesd5S8RVMXBZH6xsa577Y4dLonHxkY6ImOqpNI8vL76SElxF8W6dXM/i4+HFStcI6RfYmL4p5aoKCK5TyqDqp0IILedwNoHjCk1bycC/2CywL7nAwa4JOCvDoLchuLq0kc9cFBZVU8E/uogax8wptQsEfirhfz8c//7q4f8U0tUh2ohv8BpJqpLIrASgTGl5u1E4J9nKFDHjm66Zn+DsX9qierQUOyXv0RQqxbUqxfZmEqrSxf36n+GtDGmxLydCPxVQ4FEXKnAXyKoTg3Ffi1buiR48mTuGIKqWu01diy8+SacfXakIzGmyrJEkL9EAK6dYPVqN7isIqeWqChxcbnzLFXVwWR+NWvCdddV3URmTCXg3URw4oSba6iwRJCd7UoDFT21REUIHEtQ1ROBMabMvJsI/KOKg01b7G8wXrrUJYLq1D4AeUcXWyIwxvO8mwjyDyYL1KaNGzMwbx7s3Fn9EoG/RLBnjyUCY4yHE0H+eYbyi4+HTz91P1enhmLILRFs3OiqyCwRGONp3k0E+ecZym/AgNyHt1S3EkHdum5OJf+gOUsExniaJYKiSgTgEoW/KqU6adnS9YgCSwTGeJx3E0FKiptkrnHj4Ov9iaA6TS0RKC7OtX+AJQJjPK5Es49WK8nJbgRxYQ9sb97cPfrwsssqNq6KEljKsURgjKd5OxEUVi3k53/yVXUUmAiK+z0YY6o1b1cNefkC6O851KiRzeNvjMd5NxEEm2fIS/wlAqsWMsbzvJ0IvFwisERgjPHxZiLIyYEDB7ydCPxVQ5YIjPE8byaCQ4dcMrCqIUsExhiPJoLiBpN5QcuWULs2nHZapCMxxkSYN7uPFjfPkBfExrrZVTt0iHQkxpgI82YiKG6eIa/o1SvSERhjKgGrGjLGGI/zZiIo6qE0xhjjMd5MBMnJUK8e1KoV6UiMMSbiwpoIRGSEiGwQkU0icl+Q9deLyErfslBEKmbif68PJjPGmABhSwQiEg28AIwEegLXikjPfJttBc5X1bOAvwEvhyuePLw+z5AxxgQIZ4lgILBJVbeoaibwNjAmcANVXaiqh3xvFwNtwxhPLq/PM2SMMQHCmQjaADsD3if5PivMr4BPg60QkQkikiAiCcn+Hj9lYVVDxhhzSjgTQbDHemnQDUUuxCWCe4OtV9WXVTVeVeObl/UCrmqJwBhjAoRzQFkS0C7gfVtgd/6NROQs4BVgpKoeCGM8zrFjkJFhVUPGGOMTzkSwFOgqIh2BXcA1wHWBG4hIe+B94Geq+mMYY8llg8mM8aSTJ0+SlJRERkZGpEMJq1q1atG2bVtiYmJC/k7YEoGqZonI7cBcIBqYoqprRORW3/rJwANAU+BFcQ+Iz1LV+HDFBFgiMMajkpKSqF+/Ph06dMB3val2VJUDBw6QlJREx44dQ/5eWOcaUtU5wJx8n00O+Pn/gP8LZwwF2KhiYzwpIyOjWicBABGhadOmlLRTjfdGFluJwBjPqs5JwK8052iJwBhjPM57iSAlBWJioEGDSEdijPGQ1NRUXnzxxRJ/79JLLyU1NTUMEeXyXiLwjyr2QBHRGFN5FJYIsrOzi/zenDlzaNSoUbjCArz4YBobTGaM593x2R2s2LuiXPfZJ64PT494utD19913H5s3b6ZPnz7ExMRQr149WrVqxYoVK1i7di1XXHEFO3fuJCMjg0mTJjFhwgQAOnToQEJCAunp6YwcOZJzzz2XhQsX0qZNG2bPnk3t2rXLHLv3SgQpKdZjyBhT4R555BE6d+7MihUreOyxx1iyZAkPP/wwa9euBWDKlCksW7aMhIQEnn32WQ4cKDi+duPGjdx2222sWbOGRo0a8d5775VLbN4sEfTrF+kojDERVNSde0UZOHBgnr7+zz77LLNmzQJg586dbNy4kaZNm+b5TseOHenTpw8A/fv3Z9u2beUSizcTgVUNGWMirG7duqd+/uqrr/jf//7HokWLqFOnDhdccEHQEdA1a9Y89XN0dDTHjx8vl1i8VTV08iSkplrVkDGmwtWvX58jR44EXZeWlkbjxo2pU6cO69evZ/HixRUam7dKBP46NysRGGMqWNOmTRkyZAhnnHEGtWvXpmXLlqfWjRgxgsmTJ3PWWWfRvXt3Bg8eXKGxeSsR2GAyY0wEvfXWW0E/r1mzJp9+GvRxLKfaAZo1a8bq1atPfX7XXXeVW1zeqhqyeYaMMaYAbyUCKxEYY0wBlgiMMcbjvJUI/FVDTZpENg5jjKlEvJUIkpOhcWM36ZwxxhjAi4nAqoWMMSYPbyUCm2fIGBMhpZ2GGuDpp5/m2LFj5RxRLm8lAisRGGMipDInAu8NKBs0KNJRGGMi7Y47YEX5TkNNnz7wdGjTUA8fPpwWLVowc+ZMTpw4wdixY3nooYc4evQo48ePJykpiezsbP785z+zb98+du/ezYUXXkizZs348ssvyzduvJQIVK1qyBgTMY888girV69mxYoVzJs3j3fffZclS5agqowePZoFCxaQnJxM69at+eSTTwA3B1HDhg158skn+fLLL2kWpuuXdxJBWhpkZVnVkDGmyDv3ijBv3jzmzZtH3759AUhPT2fjxo0MHTqUu+66i3vvvZdRo0YxdOjQConHO4nABpMZYyoJVeX+++/nlltuKbBu2bJlzJkzh/vvv5+LL76YBx54IOzxeKex2OYZMsZEUOA01JdccglTpkwhPT0dgF27drF//352795NnTp1uOGGG7jrrrtYvnx5ge+Gg5UIjDGmAgROQz1y5Eiuu+46zj77bADq1avHtGnT2LRpE3fffTdRUVHExMTw73//G4AJEyYwcuRIWrVqFZbGYlHVct9pOMXHx2tCQkLJv/jdd/DUU/Dcc9CqVfkHZoyp1NatW0ePHj0iHUaFCHauIrJMVeODbe+dEsGQIW4xxhiTh3faCIwxxgRlicAY4xlVrSq8NEpzjpYIjDGeUKtWLQ4cOFCtk4GqcuDAAWrVqlWi73mnjcAY42lt27YlKSmJZH8PwmqqVq1atG3btkTfsURgjPGEmJgYOnbsGOkwKiWrGjLGGI+zRGCMMR5nicAYYzyuyo0sFpFkYHspv94MSCnHcKoSr567nbe32HkX7jRVDTrHTpVLBGUhIgmFDbGu7rx67nbe3mLnXTpWNWSMMR5nicAYYzzOa4ng5UgHEEFePXc7b2+x8y4FT7URGGOMKchrJQJjjDH5WCIwxhiP80wiEJERIrJBRDaJyH2RjidcRGSKiOwXkdUBnzURkfkistH32jiSMYaDiLQTkS9FZJ2IrBGRSb7Pq/W5i0gtEVkiIom+837I93m1Pm8/EYkWkR9E5GPf+2p/3iKyTURWicgKEUnwfVam8/ZEIhCRaOAFYCTQE7hWRHpGNqqweRUYke+z+4DPVbUr8LnvfXWTBfxeVXsAg4HbfH/j6n7uJ4CLVLU30AcYISKDqf7n7TcJWBfw3ivnfaGq9gkYO1Cm8/ZEIgAGAptUdYuqZgJvA2MiHFNYqOoC4GC+j8cAr/l+fg24okKDqgCqukdVl/t+PoK7OLShmp+7Oum+tzG+Ranm5w0gIm2By4BXAj6u9uddiDKdt1cSQRtgZ8D7JN9nXtFSVfeAu2ACLSIcT1iJSAegL/A9Hjh3X/XICmA/MF9VPXHewNPAPUBOwGdeOG8F5onIMhGZ4PusTOftlecRSJDPrN9sNSQi9YD3gDtU9bBIsD999aKq2UAfEWkEzBKRMyIdU7iJyChgv6ouE5ELIh1PBRuiqrtFpAUwX0TWl3WHXikRJAHtAt63BXZHKJZI2CcirQB8r/sjHE9YiEgMLgm8qarv+z72xLkDqGoq8BWujai6n/cQYLSIbMNV9V4kItOo/ueNqu72ve4HZuGqvst03l5JBEuBriLSUURigWuADyMcU0X6EPiF7+dfALMjGEtYiLv1/y+wTlWfDFhVrc9dRJr7SgKISG1gGLCean7eqnq/qrZV1Q64/89fqOoNVPPzFpG6IlLf/zNwMbCaMp63Z0YWi8iluDrFaGCKqj4c4ZDCQkSmAxfgpqXdB/wF+ACYCbQHdgBXqWr+BuUqTUTOBb4BVpFbZ/wHXDtBtT13ETkL1zgYjbuxm6mqfxWRplTj8w7kqxq6S1VHVffzFpFOuFIAuKr9t1T14bKet2cSgTHGmOC8UjVkjDGmEJYIjDHG4ywRGGOMx1kiMMYYj7NEYIwxHmeJwBhARP4pIheIyBWRmp1WRL4SEc89eN1EniUCY5xBuDEH5+PGIxjjGZYIjKeJyGMishIYACwC/g/4t4g8EGTb5iLynogs9S1DfJ8/KCJviMgXvvngb/Z9Lr79r/bNH391wL7u8X2WKCKPBBzmKt/zBX4UkaFhPXljfLwy6ZwxQanq3SLyDvAz4E7gK1UdUsjmzwBPqeq3ItIemAv08K07C/cchLrADyLyCXA27hkBvXEjvZeKyALfZ1cAg1T1mIg0CThGDVUd6BsJ/xfclBHGhJUlAmPclNUrgNOBtUVsNwzoGTCjaQP/vC/AbFU9DhwXkS9xE4GdC0z3zQ66T0S+xpU8zgemquoxgHxTAfgny1sGdCjriRkTCksExrNEpA/uiW5tgRSgjvtYVgBn+y7sgaKCfe5LDPnnalGCT3+O7/PC5nY54XvNxv5/mgpibQTGs1R1har2AX7EPcL0C+AS3yMA8ycBgHnA7f43vkTiN8b3/OCmuEn/lgILgKt9D45pDpwHLPHt55ciUse3n8CqIWMqnCUC42m+C/QhVc0BTlfVoqqGfgvEi8hKEVkL3BqwbgnwCbAY+JtvzvhZwEogEZdk7lHVvar6GW7a4ARf6eOucj8xY0rAZh81poxE5EEgXVUfj3QsxpSGlQiMMcbjrERgjDEeZyUCY4zxOEsExhjjcZYIjDHG4ywRGGOMx1kiMMYYj/t/aQ04Zi/eGAkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# show plot accuracy changes during training\n",
    "plt.plot(history.history['accuracy'],'g')\n",
    "plt.plot(history.history['val_accuracy'],'r')\n",
    "plt.title('accuracy across epochs')\n",
    "plt.ylabel('accuracy level')\n",
    "plt.xlabel('# epoch')\n",
    "plt.legend(['train', 'test'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to evaluate the trained model against our test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op ReadVariableOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_84066 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "1956/1 [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 3s 2ms/sample - loss: 1.1509 - accuracy: 0.4954 - mae: 0.5304\n",
      "accuracy: 49.54% mae: 53.04%\n"
     ]
    }
   ],
   "source": [
    "# load weights from best model\n",
    "model.load_weights(model_filepath)\n",
    "\n",
    "# evaluate model against test set\n",
    "with strategy.scope(): \n",
    "    model.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=['accuracy', 'mae']) \n",
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print(\"%s: %.2f%% %s: %.2f%%\" % (model.metrics_names[1], scores[1]*100, model.metrics_names[2], scores[2]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"11\"></a>\n",
    "## Confusion Matrix - Reminder From the Previous Lab:\n",
    "\n",
    "---\n",
    "In this section, we are going to plot the confusion matrix. While accuracy rate provides us with an overall outlook on how well our model is performing, we need more insights into the accuracy. This is especially true since our dataset is small.  We need to compromise some aspects of accuracy rate in favour of others. The `classification_report` provides us with the accuracy stats broken down into `precision`, `recall`, `f1-score`, and `support`. Before introducing these measures, let's define some abbreviations:\n",
    "\n",
    "For brevity, we are using the following abbreviations: \n",
    "- __FP__: False Positive\n",
    "- __TP__: True  Positive\n",
    "- __FN__: False Negative\n",
    "- __TN__: True  Negative\n",
    "\n",
    "The first measure is focused on identifying positive cases and is called __recall__. We define recall as the ability of the model to identify all true positive samples of the dataset. In mathematical terms, recall is the ratio of true positives over true positives plus false negatives. By other means, recall tells us, among all the test samples belonging to the output class, how many of them are identified correctly by the model:\n",
    "\n",
    "\\begin{equation*}\n",
    "recall = \\frac{TP}{TP+FN}\n",
    "\\end{equation*}\n",
    "\n",
    "The next measure, is called __precision__ and is the ability of the model to identify the relevant samples only, and is defined as the ratio of true positives over true positives plus false positives:\n",
    "\n",
    "\\begin{equation*}\n",
    "precision = \\frac{TP}{TP+FP}\n",
    "\\end{equation*}\n",
    "\n",
    "\n",
    "Selecting a proper threshold usually stems from a good balance between the precision and recall values. A well-known measure that provides such a balance is `F1 score`, which is a harmonic mean of precision and recall, and defined as:\n",
    "\n",
    "\\begin{equation*}\n",
    "{F_1 \\: score} = 2*\\frac{precision*recall}{precision+recall}\n",
    "\\end{equation*}\n",
    "\n",
    "---\n",
    "\n",
    "One very important measure is the recall rate on defective samples. We are interested in models that retrieve as many defective hard drives as possible. The downside is that the high recall rate usually results in a lower precision rate when the dataset is small. Here we are willing to accept this pitfall in favour of a higher recall rate, given the limitation on the dataset size. We try to predict at least half of the defective hard drives. Let's measure how our model is performing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RebatchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op AutoShardDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op OptimizeDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ModelDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MultiDeviceIterator in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op AnonymousIteratorV2 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op AnonymousIteratorV2 in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_85770 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.46      0.62      1740\n",
      "           1       0.15      0.78      0.25       216\n",
      "\n",
      "    accuracy                           0.50      1956\n",
      "   macro avg       0.55      0.62      0.44      1956\n",
      "weighted avg       0.86      0.50      0.58      1956\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "y_pred = model.predict_classes(x_test)\n",
    "conf_test = y_test.reshape(y_test.shape[0],1)\n",
    "print(classification_report(conf_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op PrefetchDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op TensorDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op RepeatDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ZipDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_87349 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.95      0.94      4000\n",
      "           1       0.79      0.72      0.76      1041\n",
      "\n",
      "    accuracy                           0.90      5041\n",
      "   macro avg       0.86      0.84      0.85      5041\n",
      "weighted avg       0.90      0.90      0.90      5041\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict_classes(x_train)\n",
    "conf_train = y_train.reshape(y_train.shape[0],1)\n",
    "print(classification_report(conf_train, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"12\"></a>\n",
    "## Optional:  CNN LSTM Architecture\n",
    "\n",
    "In this section, we are going to introduce a more advanced architecture, officially known as a __CNN LSTM Sequence Prediction__ model. This model is designed specifically for sequential prediction with spatial inputs (e.g. videos), however, could be applied to time series prediction as well.\n",
    "\n",
    "The CNN LSTM uses a convolutional neural layer to extract features from the sequence on top of the LSTM layers which also inherently extract features. Since we are dealing with one dimensional time series, we are going to use 1D convolutional network, starting with a kernel size of 12 and 50 filters. We also use a [max pooling](https://computersciencewiki.org/index.php/Max-pooling_/_Pooling) layer of size 2 to shrink the data size. Figure 5 shows the architecture of the new model.\n",
    "\n",
    "\n",
    "<img src=\"img/lstm_conv.png\" style=\"margin-top:10px;width:800px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 5.  CNN LSTM layers. </p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Training using 2 GPU(s)\n",
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv1d (Conv1D)              (None, 5, 64)             4672      \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 5, 64)             12352     \n",
      "_________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D) (None, 2, 64)             0         \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (None, 2, 64)             33024     \n",
      "_________________________________________________________________\n",
      "lstm_4 (LSTM)                (None, 64)                33024     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 87,297\n",
      "Trainable params: 87,297\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Conv1D, MaxPooling1D\n",
    "\n",
    "# model parameters\n",
    "dp_lvl = 0.2\n",
    "regularizer_lvl = 0.002\n",
    "data_shape = x_train.shape[1:]\n",
    "\n",
    "kernel_s = 3 # should be smaller than sequence length\n",
    "filter_s = 64\n",
    "\n",
    "with strategy.scope(): \n",
    "    # network design\n",
    "    model = Sequential()\n",
    "    model.add(Conv1D(kernel_size = kernel_s, filters = filter_s, padding='same', input_shape=data_shape, activation='relu'))\n",
    "    model.add(Conv1D(kernel_size = kernel_s, filters = filter_s, padding='same' ,activation='relu'))\n",
    "    model.add(MaxPooling1D(pool_size=2))\n",
    "    model.add(LSTM(filter_s, dropout = dp_lvl,recurrent_dropout = dp_lvl, return_sequences = True ))\n",
    "    model.add(LSTM(filter_s, dropout = dp_lvl,recurrent_dropout = dp_lvl, return_sequences = False ))\n",
    "    model.add(Dense(filter_s, activation='tanh',activity_regularizer=regularizers.l2(regularizer_lvl)))\n",
    "    model.add(Dropout (0.2))\n",
    "    model.add(Dense(1, activation='relu',activity_regularizer=regularizers.l2(regularizer_lvl)))\n",
    "\n",
    "# training parameters\n",
    "epochs_num = 100\n",
    "learning_rate = 0.001\n",
    "decay_rate = 3 * learning_rate / epochs_num\n",
    "optimizer = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, epsilon=None,decay = decay_rate)\n",
    "\n",
    "print('Training using %d GPU(s)' % strategy.num_replicas_in_sync)\n",
    "\n",
    "# print out model\n",
    "model.summary()\n",
    "\n",
    "# checkpoint\n",
    "model_fn = \"best_1D_model.h5\"\n",
    "model_filepath=data_dir+model_fn\n",
    "\n",
    "checkpoint = ModelCheckpoint(model_filepath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='auto')\n",
    "\n",
    "es = EarlyStopping(patience=20, verbose=1)\n",
    "\n",
    "callbacks_list = [es, checkpoint]\n",
    "\n",
    "with strategy.scope(): \n",
    "    model.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=['accuracy', 'mae'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the model built, let’s train the new model to determine its accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Train on 4536 samples, validate on 505 samples\n",
      "Epoch 1/100\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op VarHandleOp in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10\n",
      "Executing op __inference_initialize_variables_94291 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "INFO:tensorflow:batch_all_reduce: 14 all-reduces with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10\n",
      "Executing op __inference_distributed_function_98852 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.3897 - accuracy: 0.9293 - mae: 0.1276Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_101259 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "\n",
      "Epoch 00001: val_accuracy improved from -inf to 0.60990, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 16s 3ms/sample - loss: 0.3890 - accuracy: 0.9295 - mae: 0.1275 - val_loss: 3.3595 - val_accuracy: 0.6099 - val_mae: 0.7509\n",
      "Epoch 2/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2644 - accuracy: 0.9711 - mae: 0.1128\n",
      "Epoch 00002: val_accuracy did not improve from 0.60990\n",
      "4536/4536 [==============================] - 3s 603us/sample - loss: 0.2640 - accuracy: 0.9711 - mae: 0.1131 - val_loss: 5.8216 - val_accuracy: 0.5802 - val_mae: 1.1678\n",
      "Epoch 3/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2640 - accuracy: 0.9739 - mae: 0.1394\n",
      "Epoch 00003: val_accuracy did not improve from 0.60990\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.2636 - accuracy: 0.9740 - mae: 0.1392 - val_loss: 5.6188 - val_accuracy: 0.5842 - val_mae: 1.0361\n",
      "Epoch 4/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2297 - accuracy: 0.9735 - mae: 0.0994\n",
      "Epoch 00004: val_accuracy improved from 0.60990 to 0.62376, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 616us/sample - loss: 0.2293 - accuracy: 0.9735 - mae: 0.0995 - val_loss: 4.6637 - val_accuracy: 0.6238 - val_mae: 0.9379\n",
      "Epoch 5/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2010 - accuracy: 0.9801 - mae: 0.1230\n",
      "Epoch 00005: val_accuracy improved from 0.62376 to 0.66931, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 612us/sample - loss: 0.2006 - accuracy: 0.9802 - mae: 0.1228 - val_loss: 3.9399 - val_accuracy: 0.6693 - val_mae: 1.0612\n",
      "Epoch 6/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1781 - accuracy: 0.9821 - mae: 0.1228\n",
      "Epoch 00006: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 607us/sample - loss: 0.1778 - accuracy: 0.9821 - mae: 0.1230 - val_loss: 4.3489 - val_accuracy: 0.6535 - val_mae: 0.9836\n",
      "Epoch 7/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1612 - accuracy: 0.9784 - mae: 0.1032\n",
      "Epoch 00007: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1609 - accuracy: 0.9784 - mae: 0.1032 - val_loss: 4.6831 - val_accuracy: 0.6356 - val_mae: 0.8310\n",
      "Epoch 8/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1972 - accuracy: 0.9713 - mae: 0.1078\n",
      "Epoch 00008: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1975 - accuracy: 0.9709 - mae: 0.1081 - val_loss: 1.4997 - val_accuracy: 0.0000e+00 - val_mae: 0.6679\n",
      "Epoch 9/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2663 - accuracy: 0.9649 - mae: 0.0898\n",
      "Epoch 00009: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.2658 - accuracy: 0.9649 - mae: 0.0897 - val_loss: 4.8720 - val_accuracy: 0.6198 - val_mae: 0.7015\n",
      "Epoch 10/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2231 - accuracy: 0.9773 - mae: 0.0852\n",
      "Epoch 00010: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 609us/sample - loss: 0.2227 - accuracy: 0.9773 - mae: 0.0850 - val_loss: 5.3546 - val_accuracy: 0.6040 - val_mae: 0.6050\n",
      "Epoch 11/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1880 - accuracy: 0.9828 - mae: 0.0822\n",
      "Epoch 00011: val_accuracy did not improve from 0.66931\n",
      "4536/4536 [==============================] - 3s 603us/sample - loss: 0.1877 - accuracy: 0.9828 - mae: 0.0821 - val_loss: 4.9091 - val_accuracy: 0.6475 - val_mae: 0.8493\n",
      "Epoch 12/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1869 - accuracy: 0.9837 - mae: 0.0956\n",
      "Epoch 00012: val_accuracy improved from 0.66931 to 0.67129, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 615us/sample - loss: 0.1866 - accuracy: 0.9837 - mae: 0.0957 - val_loss: 4.4072 - val_accuracy: 0.6713 - val_mae: 0.8079\n",
      "Epoch 13/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2015 - accuracy: 0.9803 - mae: 0.0702\n",
      "Epoch 00013: val_accuracy improved from 0.67129 to 0.68515, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 614us/sample - loss: 0.2017 - accuracy: 0.9799 - mae: 0.0705 - val_loss: 3.8060 - val_accuracy: 0.6851 - val_mae: 0.6842\n",
      "Epoch 14/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1736 - accuracy: 0.9841 - mae: 0.0685\n",
      "Epoch 00014: val_accuracy did not improve from 0.68515\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1733 - accuracy: 0.9841 - mae: 0.0684 - val_loss: 4.0229 - val_accuracy: 0.6733 - val_mae: 0.6535\n",
      "Epoch 15/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2294 - accuracy: 0.9770 - mae: 0.0673\n",
      "Epoch 00015: val_accuracy improved from 0.68515 to 0.73267, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.2291 - accuracy: 0.9771 - mae: 0.0673 - val_loss: 2.5755 - val_accuracy: 0.7327 - val_mae: 0.6184\n",
      "Epoch 16/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1617 - accuracy: 0.9870 - mae: 0.0763\n",
      "Epoch 00016: val_accuracy did not improve from 0.73267\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.1614 - accuracy: 0.9870 - mae: 0.0763 - val_loss: 3.2494 - val_accuracy: 0.7188 - val_mae: 0.6052\n",
      "Epoch 17/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2219 - accuracy: 0.9773 - mae: 0.0727\n",
      "Epoch 00017: val_accuracy did not improve from 0.73267\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.2215 - accuracy: 0.9773 - mae: 0.0729 - val_loss: 6.3184 - val_accuracy: 0.5287 - val_mae: 0.6503\n",
      "Epoch 18/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1778 - accuracy: 0.9845 - mae: 0.0983\n",
      "Epoch 00018: val_accuracy did not improve from 0.73267\n",
      "4536/4536 [==============================] - 3s 606us/sample - loss: 0.1809 - accuracy: 0.9843 - mae: 0.0985 - val_loss: 5.1457 - val_accuracy: 0.6218 - val_mae: 0.9465\n",
      "Epoch 19/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1427 - accuracy: 0.9870 - mae: 0.1061\n",
      "Epoch 00019: val_accuracy did not improve from 0.73267\n",
      "4536/4536 [==============================] - 3s 607us/sample - loss: 0.1425 - accuracy: 0.9870 - mae: 0.1061 - val_loss: 3.7939 - val_accuracy: 0.6911 - val_mae: 0.8172\n",
      "Epoch 20/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1339 - accuracy: 0.9879 - mae: 0.0936\n",
      "Epoch 00020: val_accuracy improved from 0.73267 to 0.75644, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 617us/sample - loss: 0.1336 - accuracy: 0.9879 - mae: 0.0936 - val_loss: 2.7805 - val_accuracy: 0.7564 - val_mae: 0.7233\n",
      "Epoch 21/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1356 - accuracy: 0.9896 - mae: 0.0839\n",
      "Epoch 00021: val_accuracy improved from 0.75644 to 0.75842, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 615us/sample - loss: 0.1354 - accuracy: 0.9896 - mae: 0.0841 - val_loss: 3.1856 - val_accuracy: 0.7584 - val_mae: 0.7160\n",
      "Epoch 22/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1572 - accuracy: 0.9872 - mae: 0.0730\n",
      "Epoch 00022: val_accuracy did not improve from 0.75842\n",
      "4536/4536 [==============================] - 3s 607us/sample - loss: 0.1569 - accuracy: 0.9872 - mae: 0.0729 - val_loss: 3.0612 - val_accuracy: 0.7485 - val_mae: 0.5258\n",
      "Epoch 23/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1559 - accuracy: 0.9867 - mae: 0.0634\n",
      "Epoch 00023: val_accuracy improved from 0.75842 to 0.77822, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.1590 - accuracy: 0.9866 - mae: 0.0636 - val_loss: 2.7436 - val_accuracy: 0.7782 - val_mae: 0.5806\n",
      "Epoch 24/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1873 - accuracy: 0.9826 - mae: 0.0734\n",
      "Epoch 00024: val_accuracy did not improve from 0.77822\n",
      "4536/4536 [==============================] - 3s 602us/sample - loss: 0.1873 - accuracy: 0.9826 - mae: 0.0735 - val_loss: 0.5656 - val_accuracy: 0.7426 - val_mae: 0.3369\n",
      "Epoch 25/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1313 - accuracy: 0.9841 - mae: 0.0691\n",
      "Epoch 00025: val_accuracy did not improve from 0.77822\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1312 - accuracy: 0.9841 - mae: 0.0692 - val_loss: 3.5828 - val_accuracy: 0.7089 - val_mae: 0.6204\n",
      "Epoch 26/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1521 - accuracy: 0.9843 - mae: 0.0621\n",
      "Epoch 00026: val_accuracy improved from 0.77822 to 0.81980, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.1518 - accuracy: 0.9843 - mae: 0.0620 - val_loss: 1.2666 - val_accuracy: 0.8198 - val_mae: 0.4414\n",
      "Epoch 27/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1330 - accuracy: 0.9883 - mae: 0.0648\n",
      "Epoch 00027: val_accuracy did not improve from 0.81980\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1327 - accuracy: 0.9883 - mae: 0.0646 - val_loss: 2.7127 - val_accuracy: 0.7663 - val_mae: 0.6652\n",
      "Epoch 28/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0947 - accuracy: 0.9912 - mae: 0.0799\n",
      "Epoch 00028: val_accuracy improved from 0.81980 to 0.82970, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 610us/sample - loss: 0.0945 - accuracy: 0.9912 - mae: 0.0797 - val_loss: 1.6389 - val_accuracy: 0.8297 - val_mae: 0.6572\n",
      "Epoch 29/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0951 - accuracy: 0.9920 - mae: 0.0825\n",
      "Epoch 00029: val_accuracy did not improve from 0.82970\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0950 - accuracy: 0.9921 - mae: 0.0826 - val_loss: 3.0320 - val_accuracy: 0.7564 - val_mae: 0.7142\n",
      "Epoch 30/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0843 - accuracy: 0.9934 - mae: 0.0763\n",
      "Epoch 00030: val_accuracy improved from 0.82970 to 0.83168, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.0842 - accuracy: 0.9934 - mae: 0.0763 - val_loss: 1.8509 - val_accuracy: 0.8317 - val_mae: 0.6175\n",
      "Epoch 31/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1313 - accuracy: 0.9750 - mae: 0.0661\n",
      "Epoch 00031: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 602us/sample - loss: 0.1311 - accuracy: 0.9751 - mae: 0.0660 - val_loss: 2.4192 - val_accuracy: 0.7228 - val_mae: 0.3979\n",
      "Epoch 32/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.2400 - accuracy: 0.9589 - mae: 0.0882\n",
      "Epoch 00032: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 602us/sample - loss: 0.2397 - accuracy: 0.9590 - mae: 0.0884 - val_loss: 1.0746 - val_accuracy: 0.7822 - val_mae: 0.3768\n",
      "Epoch 33/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.1257 - accuracy: 0.9872 - mae: 0.0504\n",
      "Epoch 00033: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 605us/sample - loss: 0.1255 - accuracy: 0.9872 - mae: 0.0503 - val_loss: 2.6922 - val_accuracy: 0.7545 - val_mae: 0.5398\n",
      "Epoch 34/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0896 - accuracy: 0.9892 - mae: 0.0616\n",
      "Epoch 00034: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0894 - accuracy: 0.9892 - mae: 0.0615 - val_loss: 2.0281 - val_accuracy: 0.7861 - val_mae: 0.4744\n",
      "Epoch 35/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0656 - accuracy: 0.9927 - mae: 0.0564\n",
      "Epoch 00035: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0655 - accuracy: 0.9927 - mae: 0.0564 - val_loss: 1.8844 - val_accuracy: 0.8079 - val_mae: 0.5603\n",
      "Epoch 36/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0557 - accuracy: 0.9929 - mae: 0.0651\n",
      "Epoch 00036: val_accuracy did not improve from 0.83168\n",
      "4536/4536 [==============================] - 3s 603us/sample - loss: 0.0556 - accuracy: 0.9929 - mae: 0.0651 - val_loss: 2.1333 - val_accuracy: 0.7881 - val_mae: 0.5649\n",
      "Epoch 37/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0454 - accuracy: 0.9949 - mae: 0.0706\n",
      "Epoch 00037: val_accuracy improved from 0.83168 to 0.83960, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 610us/sample - loss: 0.0453 - accuracy: 0.9949 - mae: 0.0707 - val_loss: 1.4638 - val_accuracy: 0.8396 - val_mae: 0.6070\n",
      "Epoch 38/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0412 - accuracy: 0.9967 - mae: 0.0681\n",
      "Epoch 00038: val_accuracy did not improve from 0.83960\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0412 - accuracy: 0.9967 - mae: 0.0685 - val_loss: 1.6813 - val_accuracy: 0.8218 - val_mae: 0.5946\n",
      "Epoch 39/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9936 - mae: 0.0584\n",
      "Epoch 00039: val_accuracy did not improve from 0.83960\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0474 - accuracy: 0.9936 - mae: 0.0584 - val_loss: 1.3336 - val_accuracy: 0.8396 - val_mae: 0.5273\n",
      "Epoch 40/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0380 - accuracy: 0.9956 - mae: 0.0562\n",
      "Epoch 00040: val_accuracy improved from 0.83960 to 0.86535, saving model to ./data/best_1D_model.h5\n",
      "4536/4536 [==============================] - 3s 611us/sample - loss: 0.0379 - accuracy: 0.9956 - mae: 0.0561 - val_loss: 1.3183 - val_accuracy: 0.8653 - val_mae: 0.5176\n",
      "Epoch 41/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0314 - accuracy: 0.9971 - mae: 0.0614\n",
      "Epoch 00041: val_accuracy did not improve from 0.86535\n",
      "4536/4536 [==============================] - 3s 604us/sample - loss: 0.0313 - accuracy: 0.9971 - mae: 0.0613 - val_loss: 1.3605 - val_accuracy: 0.8554 - val_mae: 0.5582\n",
      "Epoch 42/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0311 - accuracy: 0.9976 - mae: 0.0671\n",
      "Epoch 00042: val_accuracy did not improve from 0.86535\n",
      "4536/4536 [==============================] - 3s 603us/sample - loss: 0.0310 - accuracy: 0.9976 - mae: 0.0672 - val_loss: 1.6400 - val_accuracy: 0.8574 - val_mae: 0.5642\n",
      "Epoch 43/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0371 - accuracy: 0.9976 - mae: 0.0557\n",
      "Epoch 00043: val_accuracy did not improve from 0.86535\n",
      "4536/4536 [==============================] - 3s 603us/sample - loss: 0.0371 - accuracy: 0.9976 - mae: 0.0557 - val_loss: 1.5714 - val_accuracy: 0.8594 - val_mae: 0.4843\n",
      "Epoch 44/100\n",
      "4528/4536 [============================>.] - ETA: 0s - loss: 0.0720 - accuracy: 0.9925 - mae: 0.0427\n",
      "Epoch 00044: val_accuracy did not improve from 0.86535\n",
      "4536/4536 [==============================] - 3s 602us/sample - loss: 0.0719 - accuracy: 0.9925 - mae: 0.0426 - val_loss: 2.8431 - val_accuracy: 0.7604 - val_mae: 0.4895\n",
      "Epoch 00044: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1e07a799e8>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit network\n",
    "model.fit(x_train, y_train, epochs=epochs_num, batch_size=16, validation_split=0.1, verbose=1, shuffle=True, callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start out by reviewing the training loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xV9fnA8c+TkLD3ClM2AiorDEVcBQVFkIo422r7Ey3aYq2zw2pbW+veRWvBgSI4EAcK1IUKCAEJG9kQZgIkECCEJM/vj++95Ca5SW7GzU1ynvfrdV43955zz3lOAuc533lEVTHGGONdUZEOwBhjTGRZIjDGGI+zRGCMMR5nicAYYzzOEoExxnicJQJjjPE4SwTGmBITkRtF5NtIx2HKhyUCY4zxOEsEpkoRp8r8u61q8Rpvsn+gpsRE5D4R2SwiR0RkrYiMzbf+ZhFZF7C+n+/zdiLyvogki8gBEXne9/mDIjIt4PsdRERFpIbv/Vci8rCIfAccAzqJyE0Bx9giIrfki2GMiKwQkcO+WEeIyFUisizfdr8XkQ8KOc8SH6OIeM8RkaUikuZ7PSdgPzf69n9ERLaKyPW+z7uIyNe+76SIyIwi/iaDRWShiKSKSKKIXBCw7isR+aeILPHta7aINAlYP1pE1vi++5WI9AhYF/RvFrD+cRE55It7ZHHnZCopVbXFlhItwFVAa9yNxNXAUaBVwLpdwABAgC7AaUA0kAg8BdQFagHn+r7zIDAtYP8dAAVq+N5/BewAegE1gBjgMqCz7xjn4y64/XzbDwTSgOG+GNsApwM1gYNAj4Bj/QBcWch5lvgYhcTbEjgE/Mz3/lrf+6a+38VhoLvvu62AXr6fpwN/9O3/1O8rSJxtgAPApb5th/veNw+IZxdwhu947/l/30A3399vuO/3eg+wCYgt5m92I3ASuNm33a+B3b7fVaHnZEvlXCIegC1VfwFWAGN8P88FJgXZ5mwg2X9xz7fuQYpPBH8tJoYP/McFXgKeKmS7fwMP+37u5bsg1wzxPEM9Rp54fQlgSb5tFvkupnWBVOBKoHa+bV4HXgbaFhPXvcAb+T6bC/wiIJ5HAtb1BDJ9F/A/AzMD1kX5ksYFxfzNbgQ2Bbyv4/ubxRV1TrZUzsWqhkyJicjPfVUiqSKSirvTbOZb3Q7YHORr7YDtqppVysPuzBfDSBFZLCIHfTFcGkIMAK8B14mI4C7QM1X1RLANy3CM/PG2BrbnW78daKOqR3GlqluBPSLyiYic7tvmHtwd9hJf1c0vCznWacBV/r+HL9ZzcXfiweLZjrv7b5Y/NlXN8W3bhuL/ZnsDvnfM92O9Ys7JVEKWCEyJiMhpwH+A24GmqtoIWI27YIG7iHQO8tWdQHt/vX8+R3F3lH5xQbY5NU2uiNTEVW88DrT0xTAnhBhQ1cW4u+GhwHXAG8G2K8sx8seLqzI5Ld/69rg7b1R1rqoOx1241+N+v6jqXlW9WVVbA7cAL4pIlyDH2okrETQKWOqq6iMB27TLd+yTQEr+2HwJsp0vtqL+ZkUq7JxM5WSJwJRUXdxFLhlcgyquROD3CnCXiPQXp4sveSwB9gCPiEhdEaklIkN831kBnCci7UWkIXB/MTHE4ur7k4EsXyPlxQHr/wvcJCI/EZEoEWmT7470deB5IEtVC+sLX9ZjBJoDdBOR60Skhohcjaue+VhEWvoaa+sCJ4B0IBvA17jd1rePQ7jfe3aQ/U8DLheRS0Qk2ve7vSDguwA3iEhPEakD/BV4V1WzgZnAZb7ziAF+74tjIUX/zQpV1DmZyskSgSkRVV0LPIGr494HnAl8F7D+HeBh4C3gCK5evYnvonM5rvF4B5CEqz5AVecDM4CVwDLg42JiOAL8FncRO4S7s/8wYP0S4CZcI2ca8DV578jfwCWvoKWBcjpG4L4OAKNwF9kDuCqfUaqagvs/+HvcnflBXKP0RN9XBwDfi0i679iTVHVrkP3vBMYAf8Alrp3A3eT9//0G8CquOqeW79xQ1Q3ADcBzuBLC5cDlqppZ1N+sGEWdk6mERNUeTGO8RURqA/txPYA2RjqecBORr3CN8a9EOhZTOVmJwHjRr4GlXkgCxoSixI1AxlRlIrIN1+B7RYRDMabSsKohY4zxuLBVDYnIFBHZLyKrC1kvIvKsiGwSkZXim4bAGGNMxQpn1dCruC56rxeyfiTQ1bcMwo34HFTcTps1a6YdOnQonwiNMcYjli1blqKqzYOtC1siUNUFItKhiE3GAK+rq5taLCKNRKSVqu4par8dOnQgISGhHCM1xpjqT0Tyj24/JZK9htqQd9h7ku8zY4wxFSiSiUCCfBa05VpEJohIgogkJCcnhzksY4zxlkgmgiTyzn/SFjcSsQBVfVlV41U1vnnzoFVcxhhjSimSieBD4Oe+3kODgbTi2geMMcaUv7A1FovIdNyc5s1EJAn4C27qW1R1Mm4irktxD8E4hpu3xRhjTAULZ6+ha4tZr8Bt4Tq+McaY0NhcQ8YY43E215AxxlSQk9knWZ+ynsR9iWw6uIkoiaJGVI08S60atRjYZiBntTyLKKmYe3VLBMaYiMvRHPYf3c+OtB3sTNvJjrQdHD5xmCt7XskZLc4ofgdB9peVk3VqOX7yOEcyj3D4xOE8y7GTx4J+X1U5mXOSzOzMAku0RBMbHVtgqRFVgyiJQkSIkqhTy4FjB0jcl8iKvStYk7yGzOzMkM6hRd0WDOs0jOGdhjO803DaNAjfMKsqN+lcfHy82shiU5mdzD7JzsM7ad+wPTWiyn6vlZ2TzZ70PWxP3c621G1sS93G9rTt7EnfQ8u6LenUuBOdGneiY6OOdGrciWZ1miEiZGZnkpaRRmpG6qlFUerF1iuw1K5RG/eUyqKlZqTyyvJXeD3xdTKyMk5dBGOiY079XD+2Po1qNaJhzYY0qtXo1KIoKcdSTi0Hjh8g5VgKe9P3knQ4qdAL5HmnncdtA25j7OljiYmOybMuR3NYuHMh7659lw/Wf8DuI7vJyslCgw9JKpNoiSYmOoYczQn5Yu7Xom4L+sT1oXfL3vSJ60OfuD50a9oNQfIkrKycLA6fOMyC7QuYv2U+87fMZ//R/QD0bN6TOwffya/6/apU8YvIMlWND7rOEoEJp8zsTGKjY0PePiMrg6ycLOrF1gtjVCVzIusEaSfSEPLe6UVJFCeyT7Bm/5pTd3yJ+xJZvX81mdmZnN32bGaMm0G7hu2KP0gQOZrDwwse5h/f/oOMrIw861rUbUFcvTj2pe9j39F9edbVjamLooXe7QbTuFZjRnYdyehuoxnRZQQNazXMs37zwc088/0zTPlhCkdPHuXc9ufSrkG7AnfLJ7JPkJ6ZTmpGKmkZaaSdSCtwrNjoWJrXaU6zOs1oWqcpLeq2oH2D9rRv6JZ2DdvRvmF7VJUpP0zh3wn/ZmvqVlrVa8WE/hP4Zd9fsvHARt5b9x6z1s9ib/peakbX5OLOF9OreS9iomMKVLfUjK5Jw1oNqR9bnwY1G5xa6sTUKTQBBt7tx0TFEB0VfWqdqpKVk5XnvLNzssnRHHI0B0VP/Vwvth5x9YI9hrt4OZrDqn2rmL9lPvM2z+PKHldyS/wtpdqXJYJq5MCxA6RmpNK5SVHPTXdyNIdnFj/Dowsf5cmLn+TaM4vsyFVuTmaf5MMNH/Kf5f9h3uZ5nNPuHCYNmsTYHmMLvUPekbaD575/jv8s/w/pmen0b92fizpcxIUdL2RIuyHUja0b1pjTMtL4cMOH/LD3B/am782zHMo4FNI+mtdpfupur2ntpjz8zcPERMfw+hWvc1m3y0oUT3pmOr/44Be8v+59ruxxJcM6DaNDow50aNSB9g3bUyemzqltj2YeZVvqNrYc2sLW1K1sPbSV6KjoPHfj/jv0KIkiPTO9wLIuZR2fbPyElGMp1IiqwXmnncfobqPp2rQrLy97mQ83fEiNqBpce+a13DHoDvq26hvSeWTnZHMk8wiHjh9CRGhWpxl1Y+qGVPoI3Mdnmz7jhaUv8Nmmz07d7deuUZtLu17KuJ7juKzrZdSvWb9Ev2OvsURQTSzfs5zR00ezJ30PE/pN4K8X/pXmdYOPtN6Wuo0bP7iRr7d/TfM6zTlw/ADTr5zO+F7jwxbfpoObeGX5K0xdMZX9R/fTtkFbxnQfw6ebPmXLoS20a9CO2wbcxs39b6ZJ7SYALE5azFOLn+K9te8BMK7nOLo26cqX277k+13fk5WTRUxUDIPaDmJ0t9H8dtBvqVmjZrnE67/4z1w7k3mb55GZnUndmLq0qt+KuHpxtKzbkrh6ccTVi6NxrcYAp+7y/EuURNGjeQ96t+xNXL24PBe4jQc2ctU7V5G4L5F7zrmHv1/09wJVG8FsObSFK96+gjXJa3ji4ieYNGhSiS6cpZWdk83ipMV89ONHfPTjR6xNXgtA09pN+XX8r5k4YCKt6rcKexxF2XxwMzPXzKRb026M6DIi7DcI1YklgkoqKyeLtIw0mtRuUux/9A/Wf8D1719P09pNubTrpbyy/BXqxdbjgfMf4PaBt5+qflFVXl3xKpM+mwTAsyOfZVzPcVz65qUs3LmQGeNmcGXPK0sV78p9K3kj8Q1O5pwsUK+5NXUrC7YvIFqiGdVtFDf3u5kRXUYQHRVNdk42n2z8hKcXP82X276kdo3aXHvGtaxNWcvipMU0rNmQCf0ncPvA22nfsP2p46VnpvPdju/4ctuXfL71cxJ2J3BGizOYOmYq8a2D/nsulqoye8Ns/vvDf09d/Ns1aMdVPa/iql5XMbDNwHLtqZGRlcHvPvsdk5dNZki7IUy/cnqRVUVfbP2Cq965ClVlxrgZDO88vNxiKanNBzezLmUdP+n4E2rH1I5YHKZ8WCIoZ9NXTWfXkV3cefadIV00dh3exeMLHyfpSBL7j+4/tRw8fhCAAa0H8MD5D3BZ18sKJARV5bGFj3Hf/+5jQJsBzL5mNnH14liXvI47593JZ5s+o2uTrjxx8RMMajuICR9NYPaG2Zx/2vm8esWrdGjUAYAjJ44w4s0RLNm1hHevepcxp48p8XmPmDaC+VvmUz+2foE62Ia1GnJ1r6u5sc+NtK7futB9rNy3kme/f5Y3V71Jm/ptmDRoEjf1vSmkNoE5G+dw80c3sy99H/cOuZcHzn+gRKWD1IxUJn4ykemrp4f14h/M9FXTmfDxBGpG12TSoEm0a9iO1vVb07p+a1rVa0WT2k14fsnz/G7u7+jerDuzr5lNlyZdwhqT8RZLBOXof1v+xyXTLiFHcxjdfTTTxk4rsm5ycdJifjrjpxw4foBOjTvRom4Lt9RxrzHRMfxn+X/YlrqN/q3688D5D3B5t8tP9fq49eNbmbpiKuN7jefVMa8WuDObs3EOd869kw0HNlCrRi1UlX/85B/cMfiOAhe3wycOc/EbF7N8z3Lev/p9RnUbFfJ5Hzx+kJaPt+T3Z/+eR4Y9UrJfWhAnsk4QEx1T4gtwakYqv5v7O15d8Sq9mvfi1SteDal08O2Ob7nh/RtIOpzEQxc8xH3n3pen8a8i/HjgR65//3oSdhf89xsbHUtmdiaju4/mjbFv0KBmgwqNzVR/RSUCVLVKLf3799dI2XJwizb5VxM948Uz9PHvHtfoh6L1zBfP1K2HtgbdfuoPUzX2b7Ha6ZlOumrfqkL3m5mVqVOWT9HOz3RWHkT7TO6j01dN1/Onnq88iP75iz9rdk52kd9/etHTevlbl+vqfauLPIfU46k64OUBGvu3WP3kx09COm9V1f8u/6/yIJqwKyHk74TTJz9+oq2faK3RD0Xr3fPu1mW7l+nJ7JMFtsvMytQ/ff4njXooSjs/01kX71wcgWjzOpZ5TDcf3KzfbP9GZ6yeoU8tekrvmXePPv/980X+nY0pCyBBC7muWokgREczjzJkyhC2p20n4eYEOjfpzPzN8xn/7nhqRNVg1tWzOLf9uYCr+79r3l088/0zDOs0jBnjZpxqHC1KVk4Wb658k79/83c2HdxEbHQsU0ZP4fqzri/Xczl0/BDD3hjGmv1rmP+z+Qw9bWix3xn55kg2pGxg8283V0jDZShSM1K5c+6dTF0xFYA6MXUY2GYg57Q9h3PanUOr+q2Y+MlEvt/1PTf2uZFnRzxrPUuMZ1nVUBmpKte+dy0z18xkzvVzGNFlxKl1Px74kcunX87WQ1uZPGoyY7qP4ep3r+bzrZ9zx6A7eOzix0o8qCgrJ4tZ62bRqXEn+rfuX96nA7iqnt6Te9OtaTc+//nnRW576PghWjzegjsH38m/hv8rLPGUxY60HSzcuZCFOxeyKGkRP+z5gWzNBqBRrUa8NOqlsPaWMqYqsKqhMnr020eVB9FHvnkk6PqDxw7q8NeHKw+iTf/VVGP/FqtTf5hasUGWwj8W/EN5EF27f22R2039YaryILokaUkFRVY2RzOP6tfbvtYXlrygO1J3RDocYyoFrGqo9OZvns+IN0cwruc43r7y7UKrRbJysrh73t18vPFj3hj7BoPbDq6wGEsr+WgybZ9qyy39b+HZkc8Wut1lb13G2uS1bPntlkpTLWSMKZmiSgQ2DXURthzawtXvXk2v5r2YMnpKkRfBGlE1eGrEU2z8zcYqkQQAmtdtzvhe43kt8TXSM9ODbpOakcr8zfMZ12OcJQFjqilLBD6qyu4ju/nkx0/429d/Y+yMsQx6ZRAAH1zzQbUdwTgxfiKHTxzmzZVvBl3/4YYPOZlzknE9x1VwZMaYiuL5aag3pGzg/s/vZ+HOhXkm7+rWtBs/6fgTfjf4d3Rq3CmCEYbX4LaD6RvXlxeWvsCE/hMK3PW/s/Yd2jdsz8A2AyMUoTEm3DybCLJzsnnm+2f44xd/pHaN2ow5fQx94/rSr1U/erfs7ZluhiLCxAETufmjm/l2x7d5upKmZaQxb/M8bh9wu1ULGVONeTIRbDq4iZtm38S3O75ldPfRvDTqpVJPE1sdXHfmddw9/25eTHgxTyL4cMOHZGZnclWvqyIYnTEm3DzVRpCjOTy/5Hl6T+7Nqn2reO2K1/jg6g88nQTADcS6qc9NvLf2Pfam7z31+bvr3qVtg7ZWLWRMNeeZRLAtdRvDXh/Gbz79Deeddh6rJ67m571/blUePr+O/zUnc07yyvJXADcv0dxNcxnXY1yFPTfVGBMZnvkfvnr/ahJ2J/DK5a8w57o5tG3QNtIhVSpdm3bl4s4X89Kyl8jKyeKjDR9xIvuEVQsZ4wGeaSMY1W0UWydtpWmdppEOpdKaGD+RK2ZcwUcbPuKdte/Qpn6bKjMmwhhTep5JBIAlgWKM6jaK9g3b89jCx1i+Zzm39L/FqoWM8QD7X25OiY6K5tb+t7IoaZFVCxnjIZYITB6/6vcrYqNjaVWvFee0OyfS4RhjKoCnqoZM8VrUbcHjwx+nUa1GVi1kjEdYIjAF/GbQbyIdgjGmAtktnzHGeJwlAmOM8ThLBMYY43GWCIwxxuMsERhjjMdZIjDGGI8LayIQkREiskFENonIfUHWNxSRj0QkUUTWiMhN4YzHGGNMQWFLBCISDbwAjAR6AteKSM98m90GrFXV3sAFwBMiEhuumIwxxhQUzhLBQGCTqm5R1UzgbWBMvm0UqC/uoQD1gINAVhhjMsYYk084E0EbYGfA+yTfZ4GeB3oAu4FVwCRVzcm/IxGZICIJIpKQnJwcrniNMcaTwpkIgj36S/O9vwRYAbQG+gDPi0iDAl9SfVlV41U1vnnz5uUfqTHGeFg4E0ES0C7gfVvcnX+gm4D31dkEbAVOD2NMxhhj8glnIlgKdBWRjr4G4GuAD/NtswP4CYCItAS6A1vCGJMxxph8wjb7qKpmicjtwFwgGpiiqmtE5Fbf+snA34BXRWQVrirpXlVNCVdMxhhjCgrrNNSqOgeYk++zyQE/7wYuDmcMxhhjimYji40xxuMsERhjjMdZIjDGGI+zRGCMMR5nicAYYzzOEoExxnicJQJjjPE4SwTGGONxlgiMMcbjLBEYY4zHWSIwxhiPs0RgjDEeZ4nAGGM8zhKBMcZ4nCUCY7wuLQ1+/DHSUZgIskRgjNf94Q8waBBkZ0c6EhMhlgiM8bpvvoHUVNi4MdKRmAixRGCMlx0+DKtXu5+XL49sLCZiLBEY42VLloCq+/mHHyIbi4kYSwTGeNnixe61WzcrEXiYJQJjvGzRIujRAy680CUCf+nAeIolAmO8StWVCM4+G/r1cw3G27ZFOioTAZYIjPGqjRvh4EEYPBj69nWfWTuBJ9UobIWI9Cvqi6pqFYrGVGWLFrnXs8+GLl0gOtpVD/30p5GNy1S4QhMB8EQR6xS4qJxjMcZUpMWLoUED6NkToqKgVy9rMPaoQhOBql5YkYEYYyrYokUwcKBLAuCqh+bOjWxMJiKKbSMQkToi8icRedn3vquIjAp/aMaYsDlyBFatctVCfv36wd69sGdP5OIyERFKY/FUIBM4x/c+Cfh72CIyxoRfQgLk5BRMBGDVQx4USiLorKqPAicBVPU4IGGNyhgTXv6G4kGDcj/r3RtELBF4UCiJIFNEauMaiBGRzsCJsEZljAmvxYuhe3do0iT3s/r1oWtX60LqQaEkggeBz4B2IvIm8DlwTziDMsaEkaorEQweXHBdv35WIvCgYhOBqs4DfgrcCEwH4lX1q/CGZYwJm82bISUlb/uAX9++sH07HDhQ8XGZiAml19CHwMXAV6r6saqmhD8sY0zY+CeaC5YI/A3GVj3kKaFUDT0BDAXWisg7IjJORGqFOS5jTLgsWgT16rkBZPnZVBOVV2am6+kVBqFUDX2tqhOBTsDLwHhgf1iiMcaE3+LFbiBZdHTBdU2bQvv21k5Q2axe7f5mL7wQlt2HNOmcr9fQlcCtwADgtRC/N0JENojIJhG5r5BtLhCRFSKyRkS+DjVwY0wpHD0KiYnBG4r9rMG48sjJgSefhP793UC/Tp3Ccpii5hoCQERmAINwPYdewLUVFFs+EZFo3/bDcYPQlorIh6q6NmCbRsCLwAhV3SEiLUp3GsaYkCQkuIfUB2sf8OvXD2bPdqOP69evuNhMXjt2wI03wpdfwpgx8J//QPPmYTlUqCOLO6vqrar6RShJwGcgsElVt6hqJvA2MCbfNtcB76vqDgBVtSonY8LJ31BcVImgb1/XxTQxsWJiMnmpwrRpcOaZsHQp/Pe/MGtW2JIAhJYIFgD3l2KuoTbAzoD3Sb7PAnUDGovIVyKyTER+HmxHIjJBRBJEJCE5OTmEQxtjglq0yE053axZ4dvYVBORc/QoXHMN/OxnLhEkJsIvf+lGfIdROOcaChZ5/ufg1QD6A5cBlwB/FpFuBb6k+rKqxqtqfPMwZkVjqrXAJ5IVpVUraNnSEkFFO3IERoyAd9+Ff/wDvv46bG0C+YVzrqEkoF3A+7bA7iDbfKaqR33jExYAvUPYtzHVS1ISLFwY3mNs2wb79hWfCERc9ZB1Ia04qalw8cWuxDZ9Otx/f/BeXWESzrmGlgJdRaSjiMQC1wAf5ttmNjBURGqISB1co/S6kKM3pjrIyoLLLoMhQ2D8+PBNA+2faK6o9gG/fv1gzRrIyAhPLBVl714YOxZat4Zzz3WNr3//u7vYLl0Kycmu8TySDh6EYcNg2TJXGhg/vsJDKLbXEPAX8s41NAQ33USRVDVLRG4H5gLRwBRVXSMit/rWT1bVdSLyGbASyAFeUdXVpTsVY6qoF1+ElStd3fCsWe7hMI88ArfckvvQmPKweDHUqePqnovTr5+7QK5aBQMGlP3YKSlw/Di0a1f8tuVl1iy4+WZX7/7Tn8KuXfC//8FrQXq/N2zoJuBr0gQaN4YWLWDkSNdbJ5SeU9u3Q2ysq1YriZQUlwTWrXPxXnZZyb5fXlS12AVoiqvHHwU0C+U74Vr69++vJsx+/FF1+/ZIRxF5WVmq27apfv656uLF4TnG3r2qDRqoDh+umpOjumGD6kUXqYLq4MGqK1eW/RgnT6o+9ZRq3bruOKHYssXFMHly6Y+bk6P67beq116rGhPj9te9u+pvf6v6ySeqR4+Wft9FSUtTvekmd7x+/VTXrs27/uhR1dWrVT/4QPWZZ1QffNDFdMMNqpdeqnr22aqtWrnv166tOn686qxZqhkZufs4ckT1o49Ub79dtUsXty2onnWW6t13q/7vf3m3D2bvXtVevVRr1VKdO7f8fw/5AAlayHVV3PqCKuvD6+Pj4zUhISESh/aOQYNccXXNGneXU5zUVJgzB66+ukLrNctdYiJMneomZdu0CbZsccP6/aZNg+uvL99j3ngjvPWWu/Pu3t19pgpvvAF33glpaXDHHXDrrdC5c8n3v2wZTJjgGn5HjoSXXgrtrlzV3R1ffTVMnlyyYx49Cm++6Uo6iYnubvvGG+G002DePPjqK1flFBsL550Hl1ziljPOKLp3TE6Oa7dYsMA9a7lLF7e0apVbcvrmG/j5z10f/PvvhwceCO3fcLBjLVzoqpDeecdVITVsCJdfDrt3u+OcPOlKWBdc4Or3MzJcae7bb3PXXXihGwwW7P/FW2/Bzp3w0UdwUfgfAS8iy1Q1PujKwjIE8GURyxeFfS/ci5UIKkDr1u7u5tlnQ9v+uuvc9o8+Gt64wunoUdW2bd3d2Vlnqf70p6r33KP68suuRHDBBaqxse4Ot7x8+637vd13X/D1ycmqN96Ye7c5YIDqk0+q7tpV/L4PH1a94w7VqCjVuDjVmTPdHXpJXHihO2Z+aWmqK1a4u96ZM12p4Z//dHfCP/uZasOGuXfHL72kmp6e9/vHjrk74DvvdHfE/vNr3drdyb/9tmpKitt2zx7V115z/8aaN8/dNnCpVcvtZ9gwVRHVTp1Uv/uuZOdalJMnVT/7TPUXv1Bt3Fj1zDNV77pLdf581ePHC25/5Ijqhx+q3nZb3tJC/qVZM9UFC8ovzmJQmhJBZWUlgjBTdXdQWVlu3pnNm92dUGG+/trdETVt6rq/LVvm7uyqmn/8A/74R3c+551XcP3Bg66R9dAh+P77snfry8qC+Hg33fP69VC3buHb7tgBM2a4u9MffnB3zeefD1ddFXyQ0aFD8Le/uTrxW29159aoUcljvOsueP55+GvlQkgAAB+YSURBVMMfXAlp0yb372F/IeM+a9Z0/w7OPx9uuw3OOSe0/u87d7qSwty5rg7/0CH3vQ4dYOtWt03z5u6u+5JL4Cc/gRMn8sa0aZPb9txz4V//cpPqVRaFTRQnEvbxAXkPV3iJwBKByevQIVclcMMNrirk3ntdw2UwJ0+6bobp6e4COmAAtG3rGiVLUxyPlP37XRXDRRfBBx8Uvt2PP7pk0KqVqzYoLEFmZLgpGvr0ya3uye/55+E3v4GZM90FPVQbNriEMH26i6cwZ5wBL79cfFfRonzwgetxI+L+rv6qmM6dXSJs2TK3cbVJE6hdu/TH8svOdr155s6FFSvcRGuXXOJ+l+XZcO5BpaoaqqyLVQ2VQnJy6NuuX++KrdOmuWJ+zZqFNxw/+aTbdtYs9/799937P/+57DFXpIkTVaOj3bkX54svVGvUUL34YldlECg7W/XNN1Xbt3e/h+hoVz2wf3/e7fbtc9Unw4aVvLrGLydHddMm1+iZf1m7tmBspT3Gjh3Bqz9MlUMRVUMRv7CXdLFEUAI5Oap/+Yv7M4daZ7pggdt+3jyXAGrWdL0p8tu9W7V+fdURI/JezH7+c3cBXLKkXE4h7Navd/FOnBj6d155xf2OJk7MPfcFC1x9Oqj27as6e3ZugmnQQPWRR3IvqDfd5HrRrFtX/udjTCHKlAiA93BdR6OK27YiFksEIcrKUp0wQU81TL36amjfe/ddt/2KFe79ffe598uW5d3uhhtc4+mPP+b9/NAh1+h6+umuUbAksrNVb7lFdcgQ1Ycecl02s7JKto+SGjPGJbR9+0r2vbvvdr+XP/1JdexY93ObNq5hMzs7d7u1a1Uvv9ytb9/enRe4hmhjKlBZE8Ew4E1gM/AIcHpx3wnnYokgBMeP516c7rjDvT7ySGjfffFFt/2ePe59aqrr3XDhhXnvfkH1j38Mvo/583OPHaqcnNxYe/RwvT9AtUkT1auvVp0yxfW7Lk9ff+2O8fDDJf9uVpZLIqBar57q3/9edL/4zz93JQV/wjhypPRxG1MK5VI1BDTEPZhmJ7AQuAmICfX75bVYIijGoUOqQ4e6C+kzz7jP6tUL/aL84IPun0VgHfNzz7nPPv7YfX7mme7utqgL3223ue98+WVox33sMbf9pEkuKSQnq771luuyFxenp7rblbZOPb/sbFeV06ZN6Qc2paer/vvfoSeo7GxX4lq+vHTHM6YMypwIcCOLJwEJuPmCrgaewz2kxhJBZbFrl7tIx8S4vth+nTu70Z2hmDhRtWnTvJ9lZqp27eru1J94wv2zef/9oveTnu6+c9ppxVe7vPGG2+f48XmrVfxyctxdO7i2ifIwfbqWqMrMmCquqERQbH8sEXkf+AaoA1yuqqNVdYaq/gaoRJ11PW7JEtdve+vW3FG+fnFxbvKtUOzf7+ZZCRQT4/pmr1vn+pZfcglccUXR+6lbF15/3Y3C7NoVHn00+ARm8+bBTTe5EZivvx68i6B/Nkxwo33L6sQJN+q0d2/XTdYYjwulY+7zqtpTVf+pqnmmRdTC+qSairN9O1x3nZsWIjPT9ecfNizvNmVNBOAu/OeeCzVqwLPPhjYQZvBgN8XAeee58Qinn+76v/sH2CQkuMnAevVyE27VrFn4vvwDuMojETz3nJuS+fHHq/aUGMaUk1ASQQ/fs4UBEJHGIjIxjDGZUKSluYtr9+7uIvrHP7rBRv2CTBHVsmXZE4GIGyS1dCl0K/DsoML16OHmUvn8czfo6LrrXIKYMcPNtNisGXz6adGjl8HNUyOSO9K0NJYvd4nn7rvdvDv5E6YxHhVKIrhZVVP9b1T1EHBz+EIyRTp50o1K7dIFHnvMVQFt3OjmWC9suty4ODdi+EQIj5EoLBGAu5D3LuVzgy66yJUAXn3VVRddc40bRTp3bmhT99aqBW3alK5EsHixSzr9+8MXX7iJyKZPL/l+jKmmQkkEUSK59QAiEg1UofkDqpl773VTE5x5pruwvvaaG/5flLg491rYHDF+J0+6OXUKSwRlFRUFv/iFmxrhuefcRbmwKRiC6dixZIlg4UIYPtxNs/D99/Dww64q7aGHii+BGOMhoTyYZi4wU0Qm455SdivuQTUmEubPd5NvffZZ6BNW+RPB3r1FT0GckuJew5UI/OrUgdtvL/n3OnVyk5KFYt8+1wDduLErOd16a+WaiMyYSiSURHAvcAvwa9yziucBr4QzKFOIo0dh7drcicBC1bKle923r+jt/CWGcCeC0urUyVUrZWS4qqKirFrlGs/feqtC5no3piorNhGoag7wb99iSmvTJleFU9wFrCiJia7HTf/+JfteYImgKFUhEai66p3iqpTWr3evPXqEPy5jqrhQxhF0FZF3RWStiGzxLxURXLUxZ47rS1+/vruI//rXMGUKrF5dsgdnL1vmXuNL2GvXXyIoLhH4SwyVNRF07OheQ2kn2LDB/b79SdAYU6hQGoun4koDWcCFwOvAG+EMqtqZOtV1k7zrLldn/dZb8KtfuQbfRo1cfX8oli1zF/XWrUt2/Jo13XGrQ4kAQksE69e7cQsV+OAPY6qqUBJBbVX9HPcQm+2q+iBgla6hSktz/eivvRb++c/cJzCtX++eS1ujhns4SSiWLXMlitJc3Fq2DK2NICam8vaoiYtzVWuhjCXYsKFkPZKM8bBQGoszRCQK2CgitwO7gEp6y1gJvfee678fOJVBVJS7SHXvDu++6x6EXZxjx3IbiksjlNHF/jEElfUuWiS0LqRHj7rHH55+esXEZUwVF0qJ4A7cPEO/BfoDNwC/CGdQ1cq0aa59YMCA4OuHDnUNyXv2BF/vV9qGYr+SJILKrFOn4hOB/xGOViIwJiRFJgLf4LHxqpquqkmqepOqXqmqiysovqpt50746itXGijsLnvoUPdaXKnA31Bc2kQQyjQTVSkRaBHP2vb3GLISgTEhKTIRqGo20D9wZLEpgenT3QXr+usL36ZvXzfAKpRE0KKFm2ahNOLi3EPmjx4tfJuqkgiOHHEjoAuzYYNLvF26VFxcxlRhobQR/ADMFpF3gFNXEVV9P2xRVRfTprnpDTp3LnybmBg3fXQoiaC0DcWQ241y377c3jeBVKtGIgjsQtq0afBt1q9325VlzIYxHhJKG0ET4ACup9DlvmVUOIOqFlaudKNbQ5nvfuhQt31qavD1x47BmjWlrxaC4geVHT0Kx49X/kQQShdS6zFkTImEMrL4pooIpNqZNs11DR0/vvhthw51d+TffedmycyvrA3FUPygsso+hsCvuEFlOTkuEVx4YcXFZEwVV2wiEJGpuMnm8lDVX4YlouogO9sNGhs50g0kK86gQa6K6JtvgieC0o4oDhRYNRRMVUkE9eq5GAsbS7BzpyvZWEOxMSELpY3g44CfawFjgd3hCaea+Ppr2LULnnwytO3r1HEX+cLaCcraUAzQvLlrX6jqJQIoeizBhg3u1aqGjAlZsW0EqvpewPImMB44I/yhVWJz5sBTT0FWVvD106a5eW4uvzz0fQ4d6p7+dfx4wXVlbSgGV03VvHn1SARFjSWwrqPGlFgojcX5dQXal3cgVcpDD8Gdd7rnAuSvajl+3I0WHjcOatcOfZ9Dh7oHw3z/fcH9rV1btvYBv6KmmfAngubNy36ccOvUCXbsCJ6IN2xwU2RUhYRmTCURyuyjR0TksH8BPsI9o8CbsrNdb6D+/d0jEPv2hQULctd/9JHr5x5Kb6FAQ4a4O/781UOJie6Y5ZEIihpdvH+/K8WUJHlFSqdO7neyc2fBdTbZnDElFkrVUH1VbRCwdFPV9yoiuEpp0yZ3l3777S4R1KvnHnzy6KOu58+0aa4u//zzS7bfxo3dbKSBSQXKPqI4UHGJoKrcRRfVc8i6jhpTYqGUCMaKSMOA941E5IpQdi4iI0Rkg4hsEpH7ithugIhki8i40MIuR5mZ8LOf5TYyFmflSvfauzecdZZ7bvDYse5ZwqNGwaefwnXXQXR0yWMZOhQWLcpb5bFsmauuKe65xKHwTzMRbHqGqpQIChtLcOSIa6S39gFjSiSUNoK/qGqa/42qpgJ/Ke5LvnmKXgBGAj2Ba0WkZyHb/Qv3bOSKt3Gju4ufPj207RMT3UXe/+SrBg3cNNJPPw3z5rmLeEmrhfzOO88N7Prhh9zPyqOh2C8uzs2EevhwwXVVKRG0besav/MnAptszphSCSURBNsmlG6nA4FNqrpFVTOBt4ExQbb7DfAesD+EfZa/5GT3unRpaNuvXOnuOAOnLxCBSZPcgLAXX3QlhdLwT0Dnrx46frzsI4oDFTW6uColguho6NCh4FgC6zFkTKmEkggSRORJEeksIp1E5ClgWQjfawMEtuYl+T47RUTa4MYlTA414HKXkuJeExKKntHSLzHRVQsFM3CgewxlabVq5eYl8jcYl2dDMRSeCHJyXEKsKokAgo8l2LDBPeuhqLmdjDEFhJIIfgNkAjOAmcBx4LYQvhesLiP/lfZp4F7fLKeF70hkgogkiEhCsv8Ovrz497d/f/BeKIFSU123xdLe8Ydi6FD49lt3cS6PEcWBCptm4uBBd7yqlAiCjSVYv959XrNmZGIypooKZa6ho0ChDb1FSALaBbxvS8ERyfHA275ZrpsBl4pIlqp+kC+Gl4GXAeLj40O4bS+BwMSydCm0L2KIRGBDcbicdx68+qq7qJVnQzEUPs1EVRpM5tepExw44No7GjRwn61fb+0DxpRCKL2G5otIo4D3jUUklIbdpUBXEekoIrHANcCHgRuoakdV7aCqHYB3gYn5k0DYpaRA3bpurp+EhKK3TUx0r+EuEYBrJyjPhmKAJk1cI2v+EkFVTQSQ206Qne0a/q19wJgSC6VqqJmvpxAAqnqIEJ5ZrKpZwO243kDrgJmqukZEbhWRW0sbcLlLTnZ182eeWXyD8cqVbhK5Vq3CF0/nzu7Off788m0oBld/3qJF9UgE+ccS7NgBGRlWIjCmFELp/ZMjIu1VdQeAiJxGkNlIg1HVOcCcfJ8FbRhW1RtD2We5S0521S9nnQVvv+0ajAu7A09MdNuFc9SqiKseeu+98m0o9gs2qKwqJoL8Ywn840CsRGBMiYVSIvgj8K2IvCEibwALgPvDG1YFSklxiSA+HtLS3MjhYLKzYfXq8LYP+A0d6o4H4UkEwdoIoqIKf+JXZdS4MTRqlJsIrOuoMaUWyhQTnwH9yO011F9VIzP4KxySk111z4AB7n1h7QT+qSXC2T7g528naNYM2rUretuSKqxE0KxZ6UZDR1LHjrltBBs2uOQQyvMfjDF5hDr7aDZuwFca0FNEzgtfSBVINbdqqGdPN0issHYCf0NxRZQIzjjD3e2WZ0Oxn38G0pyc3M+q0mCyQIFdSG2yOWNKLZQnlP0fMAnX/XMFMBhYhHuGcdV25Iib+rl5c9drqG/fwhPBypXujrlngVkyyl90tJvyIhyN0nFxrtrpwIHcKaerciL46KPcx1NeckmkIzKmSgqlRDAJGABsV9ULgb5AOY/qihD/GAJ/dcKAAbB8eW79fKDERHfHWVGDlUaMCE/pI9hYgqqcCDIzXWlgzx5rHzCmlEJJBBmqmgEgIjVVdT1QPfro+ROB/844Ph6OHYN16wpuu3JlxVQLhVuwaSaqaiLwdyH97DP3al1HjSmVUBJBkm9A2QfAfBGZTXV5ZrF/niF/IiiswfjQofBPLVFR8k8zceKE6y1VFROBvwvpHF8PZSsRGFMqofQaGquqqar6IPBn4L9ASM8jqPTyVw116+ae0pW/nWDVKvdanUoE/qoh/++gKiaC005zjcMLFrh2FX9iMMaUSCgDyk5R1a/DFUhE5K8aiopyPXXylwgqYmqJitKggesd5S8RVMXBZH6xsa577Y4dLonHxkY6ImOqpNI8vL76SElxF8W6dXM/i4+HFStcI6RfYmL4p5aoKCK5TyqDqp0IILedwNoHjCk1bycC/2CywL7nAwa4JOCvDoLchuLq0kc9cFBZVU8E/uogax8wptQsEfirhfz8c//7q4f8U0tUh2ohv8BpJqpLIrASgTGl5u1E4J9nKFDHjm66Zn+DsX9qierQUOyXv0RQqxbUqxfZmEqrSxf36n+GtDGmxLydCPxVQ4FEXKnAXyKoTg3Ffi1buiR48mTuGIKqWu01diy8+SacfXakIzGmyrJEkL9EAK6dYPVqN7isIqeWqChxcbnzLFXVwWR+NWvCdddV3URmTCXg3URw4oSba6iwRJCd7UoDFT21REUIHEtQ1ROBMabMvJsI/KOKg01b7G8wXrrUJYLq1D4AeUcXWyIwxvO8mwjyDyYL1KaNGzMwbx7s3Fn9EoG/RLBnjyUCY4yHE0H+eYbyi4+HTz91P1enhmLILRFs3OiqyCwRGONp3k0E+ecZym/AgNyHt1S3EkHdum5OJf+gOUsExniaJYKiSgTgEoW/KqU6adnS9YgCSwTGeJx3E0FKiptkrnHj4Ov9iaA6TS0RKC7OtX+AJQJjPK5Es49WK8nJbgRxYQ9sb97cPfrwsssqNq6KEljKsURgjKd5OxEUVi3k53/yVXUUmAiK+z0YY6o1b1cNefkC6O851KiRzeNvjMd5NxEEm2fIS/wlAqsWMsbzvJ0IvFwisERgjPHxZiLIyYEDB7ydCPxVQ5YIjPE8byaCQ4dcMrCqIUsExhiPJoLiBpN5QcuWULs2nHZapCMxxkSYN7uPFjfPkBfExrrZVTt0iHQkxpgI82YiKG6eIa/o1SvSERhjKgGrGjLGGI/zZiIo6qE0xhjjMd5MBMnJUK8e1KoV6UiMMSbiwpoIRGSEiGwQkU0icl+Q9deLyErfslBEKmbif68PJjPGmABhSwQiEg28AIwEegLXikjPfJttBc5X1bOAvwEvhyuePLw+z5AxxgQIZ4lgILBJVbeoaibwNjAmcANVXaiqh3xvFwNtwxhPLq/PM2SMMQHCmQjaADsD3if5PivMr4BPg60QkQkikiAiCcn+Hj9lYVVDxhhzSjgTQbDHemnQDUUuxCWCe4OtV9WXVTVeVeObl/UCrmqJwBhjAoRzQFkS0C7gfVtgd/6NROQs4BVgpKoeCGM8zrFjkJFhVUPGGOMTzkSwFOgqIh2BXcA1wHWBG4hIe+B94Geq+mMYY8llg8mM8aSTJ0+SlJRERkZGpEMJq1q1atG2bVtiYmJC/k7YEoGqZonI7cBcIBqYoqprRORW3/rJwANAU+BFcQ+Iz1LV+HDFBFgiMMajkpKSqF+/Ph06dMB3val2VJUDBw6QlJREx44dQ/5eWOcaUtU5wJx8n00O+Pn/gP8LZwwF2KhiYzwpIyOjWicBABGhadOmlLRTjfdGFluJwBjPqs5JwK8052iJwBhjPM57iSAlBWJioEGDSEdijPGQ1NRUXnzxxRJ/79JLLyU1NTUMEeXyXiLwjyr2QBHRGFN5FJYIsrOzi/zenDlzaNSoUbjCArz4YBobTGaM593x2R2s2LuiXPfZJ64PT494utD19913H5s3b6ZPnz7ExMRQr149WrVqxYoVK1i7di1XXHEFO3fuJCMjg0mTJjFhwgQAOnToQEJCAunp6YwcOZJzzz2XhQsX0qZNG2bPnk3t2rXLHLv3SgQpKdZjyBhT4R555BE6d+7MihUreOyxx1iyZAkPP/wwa9euBWDKlCksW7aMhIQEnn32WQ4cKDi+duPGjdx2222sWbOGRo0a8d5775VLbN4sEfTrF+kojDERVNSde0UZOHBgnr7+zz77LLNmzQJg586dbNy4kaZNm+b5TseOHenTpw8A/fv3Z9u2beUSizcTgVUNGWMirG7duqd+/uqrr/jf//7HokWLqFOnDhdccEHQEdA1a9Y89XN0dDTHjx8vl1i8VTV08iSkplrVkDGmwtWvX58jR44EXZeWlkbjxo2pU6cO69evZ/HixRUam7dKBP46NysRGGMqWNOmTRkyZAhnnHEGtWvXpmXLlqfWjRgxgsmTJ3PWWWfRvXt3Bg8eXKGxeSsR2GAyY0wEvfXWW0E/r1mzJp9+GvRxLKfaAZo1a8bq1atPfX7XXXeVW1zeqhqyeYaMMaYAbyUCKxEYY0wBlgiMMcbjvJUI/FVDTZpENg5jjKlEvJUIkpOhcWM36ZwxxhjAi4nAqoWMMSYPbyUCm2fIGBMhpZ2GGuDpp5/m2LFj5RxRLm8lAisRGGMipDInAu8NKBs0KNJRGGMi7Y47YEX5TkNNnz7wdGjTUA8fPpwWLVowc+ZMTpw4wdixY3nooYc4evQo48ePJykpiezsbP785z+zb98+du/ezYUXXkizZs348ssvyzduvJQIVK1qyBgTMY888girV69mxYoVzJs3j3fffZclS5agqowePZoFCxaQnJxM69at+eSTTwA3B1HDhg158skn+fLLL2kWpuuXdxJBWhpkZVnVkDGmyDv3ijBv3jzmzZtH3759AUhPT2fjxo0MHTqUu+66i3vvvZdRo0YxdOjQConHO4nABpMZYyoJVeX+++/nlltuKbBu2bJlzJkzh/vvv5+LL76YBx54IOzxeKex2OYZMsZEUOA01JdccglTpkwhPT0dgF27drF//352795NnTp1uOGGG7jrrrtYvnx5ge+Gg5UIjDGmAgROQz1y5Eiuu+46zj77bADq1avHtGnT2LRpE3fffTdRUVHExMTw73//G4AJEyYwcuRIWrVqFZbGYlHVct9pOMXHx2tCQkLJv/jdd/DUU/Dcc9CqVfkHZoyp1NatW0ePHj0iHUaFCHauIrJMVeODbe+dEsGQIW4xxhiTh3faCIwxxgRlicAY4xlVrSq8NEpzjpYIjDGeUKtWLQ4cOFCtk4GqcuDAAWrVqlWi73mnjcAY42lt27YlKSmJZH8PwmqqVq1atG3btkTfsURgjPGEmJgYOnbsGOkwKiWrGjLGGI+zRGCMMR5nicAYYzyuyo0sFpFkYHspv94MSCnHcKoSr567nbe32HkX7jRVDTrHTpVLBGUhIgmFDbGu7rx67nbe3mLnXTpWNWSMMR5nicAYYzzOa4ng5UgHEEFePXc7b2+x8y4FT7URGGOMKchrJQJjjDH5WCIwxhiP80wiEJERIrJBRDaJyH2RjidcRGSKiOwXkdUBnzURkfkistH32jiSMYaDiLQTkS9FZJ2IrBGRSb7Pq/W5i0gtEVkiIom+837I93m1Pm8/EYkWkR9E5GPf+2p/3iKyTURWicgKEUnwfVam8/ZEIhCRaOAFYCTQE7hWRHpGNqqweRUYke+z+4DPVbUr8LnvfXWTBfxeVXsAg4HbfH/j6n7uJ4CLVLU30AcYISKDqf7n7TcJWBfw3ivnfaGq9gkYO1Cm8/ZEIgAGAptUdYuqZgJvA2MiHFNYqOoC4GC+j8cAr/l+fg24okKDqgCqukdVl/t+PoK7OLShmp+7Oum+tzG+Ranm5w0gIm2By4BXAj6u9uddiDKdt1cSQRtgZ8D7JN9nXtFSVfeAu2ACLSIcT1iJSAegL/A9Hjh3X/XICmA/MF9VPXHewNPAPUBOwGdeOG8F5onIMhGZ4PusTOftlecRSJDPrN9sNSQi9YD3gDtU9bBIsD999aKq2UAfEWkEzBKRMyIdU7iJyChgv6ouE5ELIh1PBRuiqrtFpAUwX0TWl3WHXikRJAHtAt63BXZHKJZI2CcirQB8r/sjHE9YiEgMLgm8qarv+z72xLkDqGoq8BWujai6n/cQYLSIbMNV9V4kItOo/ueNqu72ve4HZuGqvst03l5JBEuBriLSUURigWuADyMcU0X6EPiF7+dfALMjGEtYiLv1/y+wTlWfDFhVrc9dRJr7SgKISG1gGLCean7eqnq/qrZV1Q64/89fqOoNVPPzFpG6IlLf/zNwMbCaMp63Z0YWi8iluDrFaGCKqj4c4ZDCQkSmAxfgpqXdB/wF+ACYCbQHdgBXqWr+BuUqTUTOBb4BVpFbZ/wHXDtBtT13ETkL1zgYjbuxm6mqfxWRplTj8w7kqxq6S1VHVffzFpFOuFIAuKr9t1T14bKet2cSgTHGmOC8UjVkjDGmEJYIjDHG4ywRGGOMx1kiMMYYj7NEYIwxHmeJwBhARP4pIheIyBWRmp1WRL4SEc89eN1EniUCY5xBuDEH5+PGIxjjGZYIjKeJyGMishIYACwC/g/4t4g8EGTb5iLynogs9S1DfJ8/KCJviMgXvvngb/Z9Lr79r/bNH391wL7u8X2WKCKPBBzmKt/zBX4UkaFhPXljfLwy6ZwxQanq3SLyDvAz4E7gK1UdUsjmzwBPqeq3ItIemAv08K07C/cchLrADyLyCXA27hkBvXEjvZeKyALfZ1cAg1T1mIg0CThGDVUd6BsJ/xfclBHGhJUlAmPclNUrgNOBtUVsNwzoGTCjaQP/vC/AbFU9DhwXkS9xE4GdC0z3zQ66T0S+xpU8zgemquoxgHxTAfgny1sGdCjriRkTCksExrNEpA/uiW5tgRSgjvtYVgBn+y7sgaKCfe5LDPnnalGCT3+O7/PC5nY54XvNxv5/mgpibQTGs1R1har2AX7EPcL0C+AS3yMA8ycBgHnA7f43vkTiN8b3/OCmuEn/lgILgKt9D45pDpwHLPHt55ciUse3n8CqIWMqnCUC42m+C/QhVc0BTlfVoqqGfgvEi8hKEVkL3BqwbgnwCbAY+JtvzvhZwEogEZdk7lHVvar6GW7a4ARf6eOucj8xY0rAZh81poxE5EEgXVUfj3QsxpSGlQiMMcbjrERgjDEeZyUCY4zxOEsExhjjcZYIjDHG4ywRGGOMx1kiMMYYj/t/aQ04Zi/eGAkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show plot accuracy changes during training\n",
    "plt.plot(history.history['accuracy'],'g')\n",
    "plt.plot(history.history['val_accuracy'],'r')\n",
    "plt.title('accuracy across epochs')\n",
    "plt.ylabel('accuracy level')\n",
    "plt.xlabel('# epoch')\n",
    "plt.legend(['train', 'test'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's check out the overall accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_159660 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "1956/1 [========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 2s 1ms/sample - loss: 4.0404 - accuracy: 0.4141 - mae: 0.8210\n",
      "accuracy: 41.41% mae: 82.10%\n"
     ]
    }
   ],
   "source": [
    "# load weights from best model\n",
    "model.load_weights(model_filepath)\n",
    "\n",
    "# evaluate model against test set\n",
    "with strategy.scope(): \n",
    "    model.compile(optimizer=optimizer,loss='binary_crossentropy',metrics=['accuracy', 'mae']) \n",
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print(\"%s: %.2f%% %s: %.2f%%\" % (model.metrics_names[1], scores[1]*100, model.metrics_names[2], scores[2]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the previous model, we are going to build the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing op MapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op FlatMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op ParallelMapDataset in device /job:localhost/replica:0/task:0/device:CPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "Executing op GeneratorDataset in device /job:localhost/replica:0/task:0/device:GPU:1\n",
      "Executing op __inference_distributed_function_161012 in device /job:localhost/replica:0/task:0/device:GPU:0\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.35      0.52      1740\n",
      "           1       0.15      0.90      0.25       216\n",
      "\n",
      "    accuracy                           0.41      1956\n",
      "   macro avg       0.56      0.63      0.39      1956\n",
      "weighted avg       0.88      0.41      0.49      1956\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# confusion matrix on test set\n",
    "y_pred = model.predict_classes(x_test)\n",
    "conf_test = y_test.reshape(y_test.shape[0],1)\n",
    "print(classification_report(conf_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leveraging the model for calculating \"Remaining Useful Life\" (RUL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case above, we developed a \"classifier\" which will indicate, based on SMART drive data, that a given hard drive will fail within the \"sequnce_length\".  By increasing the sequence_length, we could potentially learn to detect failures further in the future. In addition, different failure types could have different failure windows, so an ensemble of models, each with different windows and sequences could be leveraged for more advanced RUL predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# free up memory for upcoming exercises\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Group Discussion Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: What do you think about the model accuracy ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer noicee"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: Discuss some ways that you could improve the performance of the model above ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer add layers "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assess\"></a>\n",
    "## Assessment – Predicting Time Series\n",
    "\n",
    "The RNN model we created in this lab was designed to output a single value, indicating whether the subject time series results in a \"failed\" hard drive or not. To accomplish this, a simple class label is returned by the model through the below line of tf.keras code:\n",
    "\n",
    "```\n",
    "model.add(Dense(1, activation='relu',activity_regularizer=regularizers.l2(regularizer_lvl)))\n",
    "```\n",
    "\n",
    "In this assignment, we intend to perform a slightly different approach. For example, while previously we were given with sequences of length \"5\" (the original value), now we assume that the length of the sequence is one less, e.g. 4 (these are arbitrary lengths for the sake of example) in our case. Predicting the 5th element of the sequence is the subject of our model. By other means, we want the model to return a single (multi-dimensional) vector, (a vector of size [1 x 47] for the above DataFrame. You should determine the vector size by the array shape). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image](img/Lab2-Assessment-Example.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the above criteria, there are some other changes required for the architecture of the model:\n",
    "\n",
    "1- The model starts with two stacked LSTM layers with a unit size of 128.\n",
    "\n",
    "2- The last output of the second LSTM layer is fed into a Dense model with the following properties: \n",
    "    - The number of outputs is 256\n",
    "    - The activation function is `tanh`\n",
    "    \n",
    "3- The fourth layer is a dropout layer with dropout probability of 0.2\n",
    "\n",
    "4- Next is another dense model with:\n",
    "    - The number of outputs is 128\n",
    "    - The activation function is `tanh`\n",
    "    \n",
    "5- The last layer is a Dense layer where the number of outputs is equal to the dimension of a single row of data. (Remember that we are trying to predict the values of a multi dimension vector).\n",
    "    \n",
    "The function should simply return the above model.\n",
    "        \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "__To do so, open [this python file](/lab/edit/assignment.py)__  and complete the `get_model()` function by filling in the __<<< TODO >>>__ section according to the criteria described above.\n",
    "\n",
    "\n",
    "Once done, you can use the following block of code to verify that your implementation is correct. Note that this is __NOT__ the last step. After you succeed the test below, you need to click on assess test button on the course page. For more details, read the red note below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-07-01 07:31:57.548788: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: out of memory\n",
      "Aborted (core dumped)\n"
     ]
    }
   ],
   "source": [
    "!python try_assessment.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"background-color:red;color:white;padding:1px,3px\"><b>Note:</b></span> __Once the code is completed, go back to the assessment page (where you clicked on the current notebook) and click on the `ASSESS TASK` button. If you have correctly implemented the code, you will pass this section.__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTMs provide a powerful way to analyze time series data.  While XGBoost was able to predict failure for a given set of SMART data, the LSTM gave us the ability to predict future disk failures before they happen.\n",
    "\n",
    "One thing to remember: The amount of data, model complexity, the number of features, and the number of epochs have been reduced in this tutorial to reduce the computational (and time) burden.  If we were leveraging this model for production usage, we would train on a much larger data set.  \n",
    "\n",
    "This training would take hours to days, and we would regularly retrain the model leveraging additional data as it became available.  Also, a key thing to remember is that the data is device and environment-dependent, so any major changes would require retraining to ensure the model was still appropriate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"All-answers\"></a>\n",
    "## Answers to selected exercises:\n",
    "---\n",
    "\n",
    "<a name=\"a1\"></a>\n",
    "**Exercise 1: Create class labels:**"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "x_train = df_train.drop(columns=drop_columns_list)\n",
    "x_test = df_test.drop(columns=drop_columns_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](#e1) to go back"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a2\"></a>\n",
    "**Exercise 2: Reshaping the training data**"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "x_train = x_train.reshape(int(x_train.shape[0]/sequence_length), sequence_length, x_train.shape[1])\n",
    "x_test = x_test.reshape(int(x_test.shape[0]/sequence_length), sequence_length, x_test.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](#e2) to go back"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}