Lab3-AE-For-Anomaly-Detection (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/header.png\" alt=\"Header\" style=\"width: 400px;\"/>\n",
    "<h1 align=\"center\">Predictive Maintenance using NVIDIA RAPIDS and Deep Learning Models</h1>\n",
    "<h4 align=\"center\">Part 3: Training Autoencoder for Anomaly Detection</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous lab, we used LSTMs to predict hard drive failures. While timeseries prediction is a very well-studied approach in predictive maintenance, in this section we focus on a different deep learning technique, which is called __anomaly detection__. In a nutshell, considering the fact that only a small portion of the hard drives are defective in our use case, we can treat those as anomalies. For example, out of three million samples we dealt with in our training set within the previous lab, only 256 samples were defective. Those are \"anomalies\" among the big pool of well-functioning hard drives. Now, if we could have a model capable of learning what a proper functioning hard drive looks like, it could detect hard drives with a different behavior pattern.\n",
    "\n",
    "\n",
    "This lab covers the following topics:\n",
    "\n",
    "* [Autoencoders](#1)\n",
    "    * [Exercise 1: Retrieve the output labels](#e1)\n",
    "    * [Exercise 2: Filtering the labels](#e2)\n",
    "    * [Testing Models](#2)\n",
    "        * [LSTM Autoencoder Model](#3)\n",
    "        * [1D Convolution Model](#3b)\n",
    "* [Hyperparameter Optimization](#e4)\n",
    "    * [Exercise 3: Experimenting with Hyperparameters](#e4)\n",
    "    * [Exercise 4: Update code to use specified validation data](#e5)\n",
    "    * [Exercise 5: Create HPO result plots](#e6)\n",
    "\n",
    "In this lab, we will leverage the same Backblaze Hard Drive SMART data to train an Autoencoder model that will detect anomaly activity.  We'll leverage the time series sequences that we created in the previous step to train an Autoencoder Model to identify when there are significant variances from expected sensor data.  These variances can be used to alert support teams that there are potential issues with the system.\n",
    "\n",
    "You'll see that autoencoders (AEs) are relatively easy to use, but let's first walk through the basics of using autoencoders.\n",
    "\n",
    "We can use unsupervised learning algorithms like a deep autoencoder network to perform anomaly detection. A deep autoencoder takes an input X and attempts to map it back to itself; in essence, the autoencoder is trying to learn the identity function of the input data. Since learning the identity function is pretty trivial, we force the hidden layers of our autoencoder to be of lower dimensionality than the previous layer. This constraint forces the autoencoder to learn patterns in the data, and really learn efficient representations. \n",
    "\n",
    "<img src=\"img/AE_img-2.png\" style=\"margin-top:10px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 1. Using an autoencoder to map a timeseries data back to itself</p>\n",
    "\n",
    "To train this type of neural network, we can calculate the difference between the input and the output and use that loss, typically called a reconstruction loss, as our loss function for training our model. The idea is that our autoencoder is essentially learning the data generating distribution for a unique dataset, and when we feed data generated from a different distribution into our autoencoder, our autoencoder will do a very poor job of reconstructing the input from the second source and our reconstruction error will be high; this indicates that our input is anomalous instead of just an extreme value.\n",
    "\n",
    "<img src=\"img/AE_img-3.png\" style=\"margin-top:10px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 2. The predicted and ground-truth timeseries comparison after applying the autoencoder</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"1\"></a>\n",
    "## Autoencoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Autoencoders are a subset of neural network architectures where the output dimension is the same as the input dimension. Autoencoders have two networks, an encoder and a decoder.  The encoder encodes its input data into a smaller dimensional space, called the latent space.  The decoder network tries to reconstruct the original data from the latent encoding. Typically, the encoder and decoder are symmetric, and the latent space is a bottleneck. The autoencoder has to learn essential characteristics of the data to be able to do a high-quality reconstruction of the data during decode.\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"img/AE_img-1.png\" style=\"margin-top:10px;\"/>\n",
    "<p style=\"text-align: center;color:gray\"> Figure 3.  Structure of an auto-encoder model</p>\n",
    "\n",
    "Autoencoders can be built using many different layer types based on the desire goal of the model.  In this lab, we'll experiment using two different autoencoder models.  One built with LSTM layers and one built on top of 1D-Convolution.  You'll have the opportunity to test both.  Additionally, you'll see that we leverage Dropout layers as a way to control overfitting by randomly omitting subsets of features at each iteration of a training procedure. To begin, let’s import some libraries that we are going to use throughout this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Import libraries that will be needed for the lab\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import logging\n",
    "import importlib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import collections\n",
    "from tqdm import tqdm\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_curve, auc,precision_recall_fscore_support, average_precision_score, precision_recall_curve, auc, confusion_matrix,accuracy_score\n",
    "\n",
    "from tensorflow.keras import initializers, regularizers\n",
    "from tensorflow.keras.models import Model, Sequential\n",
    "from tensorflow.keras.optimizers import Adam, RMSprop\n",
    "from tensorflow.keras.layers import *\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from tensorflow.keras.layers import Input, LSTM, BatchNormalization, LeakyReLU, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise, MaxPooling2D, concatenate\n",
    "\n",
    "from tensorflow.keras import backend as K\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import *\n",
    "\n",
    "import pickle\n",
    "import gc\n",
    "\n",
    "import random\n",
    "\n",
    "random.seed(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"1b\"></a>\n",
    "### Input Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 16-day sequence data set\n"
     ]
    }
   ],
   "source": [
    "data_dir = \"./data/\"\n",
    "\n",
    "print('Using 16-day sequence data set')\n",
    "sequence_length = 16        # Number of days in sequence to train\n",
    "wsize = 16\n",
    "train_fname = data_dir + 'Lab3-AE-train-16day-8step.pkl'\n",
    "test_fname = data_dir + 'Lab3-AE-test-16day-8step.pkl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Set Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we experienced in the previous exercises, preparing large data sets can take some time.  So, for this exercise, we have created larger training and validation datasets which we will use for our training and testing.  \n",
    "\n",
    "The datasets consists of 50000 sequences.  Each sequence is for an individual drive and tracks the SMART data features over 7 continuous days.  Since we will be training an Autoencoder to identify sequences for normal hard drive operation, we have not include any sequences that ended in drive failures.\n",
    "\n",
    "When we created these sequences, we assigned them a sequence id (field: \"ae_seq_id\") and a day number in the sequence (field: \"ae_seq_data_id\" which goes from 0-15)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Training and Validation Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Original shape: (74000, 46)\n"
     ]
    }
   ],
   "source": [
    "# Reading larger training and test data set with only ST4000DM000 data\n",
    "df = pd.read_pickle(train_fname)\n",
    "print('Train Original shape:', df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our data set.  We'll print out the first 2 sequences.  You'll see the sequence id fields we discussed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ae_seq_id</th>\n",
       "      <th>ae_seq_data_id</th>\n",
       "      <th>date</th>\n",
       "      <th>serial_number</th>\n",
       "      <th>model</th>\n",
       "      <th>capacity_bytes</th>\n",
       "      <th>failure</th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_184_normalized</th>\n",
       "      <th>smart_187_normalized</th>\n",
       "      <th>smart_188_normalized</th>\n",
       "      <th>smart_189_normalized</th>\n",
       "      <th>smart_190_normalized</th>\n",
       "      <th>smart_193_normalized</th>\n",
       "      <th>smart_194_normalized</th>\n",
       "      <th>smart_197_normalized</th>\n",
       "      <th>smart_198_normalized</th>\n",
       "      <th>smart_199_normalized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>195468</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2017-07-28</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>169386552.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195469</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2017-07-29</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>220437952.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195470</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2017-07-30</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12172112.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195471</th>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2017-07-31</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33350928.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195472</th>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2017-08-01</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>98402824.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195473</th>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2017-08-02</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>173407640.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195474</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2017-08-03</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>59425848.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195475</th>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2017-08-04</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>178379120.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195476</th>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2017-08-05</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21791640.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195477</th>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2017-08-06</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>151298144.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195478</th>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2017-08-07</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29705424.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195479</th>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2017-08-08</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>103469832.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195480</th>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2017-08-09</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>171503104.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195481</th>\n",
       "      <td>0.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2017-08-10</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>202534848.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>71.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195482</th>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2017-08-11</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>243275368.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195483</th>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2017-08-12</td>\n",
       "      <td>Z300HM1N</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>76567144.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200466</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2017-04-07</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>116265744.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>87.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200467</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2017-04-08</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>154691184.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200468</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2017-04-09</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>201368272.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200469</th>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2017-04-10</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>243208416.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200470</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2017-04-11</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42572824.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200471</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2017-04-12</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>80559216.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200472</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2017-04-13</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120589400.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200473</th>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2017-04-14</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160877680.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200474</th>\n",
       "      <td>1.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2017-04-15</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>198030056.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200475</th>\n",
       "      <td>1.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2017-04-16</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>237670544.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200476</th>\n",
       "      <td>1.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2017-04-17</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>32806016.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200477</th>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>2017-04-18</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>74222592.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200478</th>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2017-04-19</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>118242064.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200479</th>\n",
       "      <td>1.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>2017-04-20</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>159253192.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>79.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200480</th>\n",
       "      <td>1.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2017-04-21</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>199694096.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200481</th>\n",
       "      <td>1.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2017-04-22</td>\n",
       "      <td>W300AZR3</td>\n",
       "      <td>ST4000DM000</td>\n",
       "      <td>4.000787e+12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>239649088.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>78.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32 rows × 46 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        ae_seq_id  ae_seq_data_id       date serial_number        model  \\\n",
       "195468        0.0             0.0 2017-07-28      Z300HM1N  ST4000DM000   \n",
       "195469        0.0             1.0 2017-07-29      Z300HM1N  ST4000DM000   \n",
       "195470        0.0             2.0 2017-07-30      Z300HM1N  ST4000DM000   \n",
       "195471        0.0             3.0 2017-07-31      Z300HM1N  ST4000DM000   \n",
       "195472        0.0             4.0 2017-08-01      Z300HM1N  ST4000DM000   \n",
       "195473        0.0             5.0 2017-08-02      Z300HM1N  ST4000DM000   \n",
       "195474        0.0             6.0 2017-08-03      Z300HM1N  ST4000DM000   \n",
       "195475        0.0             7.0 2017-08-04      Z300HM1N  ST4000DM000   \n",
       "195476        0.0             8.0 2017-08-05      Z300HM1N  ST4000DM000   \n",
       "195477        0.0             9.0 2017-08-06      Z300HM1N  ST4000DM000   \n",
       "195478        0.0            10.0 2017-08-07      Z300HM1N  ST4000DM000   \n",
       "195479        0.0            11.0 2017-08-08      Z300HM1N  ST4000DM000   \n",
       "195480        0.0            12.0 2017-08-09      Z300HM1N  ST4000DM000   \n",
       "195481        0.0            13.0 2017-08-10      Z300HM1N  ST4000DM000   \n",
       "195482        0.0            14.0 2017-08-11      Z300HM1N  ST4000DM000   \n",
       "195483        0.0            15.0 2017-08-12      Z300HM1N  ST4000DM000   \n",
       "200466        1.0             0.0 2017-04-07      W300AZR3  ST4000DM000   \n",
       "200467        1.0             1.0 2017-04-08      W300AZR3  ST4000DM000   \n",
       "200468        1.0             2.0 2017-04-09      W300AZR3  ST4000DM000   \n",
       "200469        1.0             3.0 2017-04-10      W300AZR3  ST4000DM000   \n",
       "200470        1.0             4.0 2017-04-11      W300AZR3  ST4000DM000   \n",
       "200471        1.0             5.0 2017-04-12      W300AZR3  ST4000DM000   \n",
       "200472        1.0             6.0 2017-04-13      W300AZR3  ST4000DM000   \n",
       "200473        1.0             7.0 2017-04-14      W300AZR3  ST4000DM000   \n",
       "200474        1.0             8.0 2017-04-15      W300AZR3  ST4000DM000   \n",
       "200475        1.0             9.0 2017-04-16      W300AZR3  ST4000DM000   \n",
       "200476        1.0            10.0 2017-04-17      W300AZR3  ST4000DM000   \n",
       "200477        1.0            11.0 2017-04-18      W300AZR3  ST4000DM000   \n",
       "200478        1.0            12.0 2017-04-19      W300AZR3  ST4000DM000   \n",
       "200479        1.0            13.0 2017-04-20      W300AZR3  ST4000DM000   \n",
       "200480        1.0            14.0 2017-04-21      W300AZR3  ST4000DM000   \n",
       "200481        1.0            15.0 2017-04-22      W300AZR3  ST4000DM000   \n",
       "\n",
       "        capacity_bytes  failure  smart_1_raw  smart_4_raw  smart_5_raw  ...  \\\n",
       "195468    4.000787e+12      0.0  169386552.0         24.0          0.0  ...   \n",
       "195469    4.000787e+12      0.0  220437952.0         24.0          0.0  ...   \n",
       "195470    4.000787e+12      0.0   12172112.0         24.0          0.0  ...   \n",
       "195471    4.000787e+12      0.0   33350928.0         24.0          0.0  ...   \n",
       "195472    4.000787e+12      0.0   98402824.0         24.0          0.0  ...   \n",
       "195473    4.000787e+12      0.0  173407640.0         24.0          0.0  ...   \n",
       "195474    4.000787e+12      0.0   59425848.0         24.0          0.0  ...   \n",
       "195475    4.000787e+12      0.0  178379120.0         24.0          0.0  ...   \n",
       "195476    4.000787e+12      0.0   21791640.0         24.0          0.0  ...   \n",
       "195477    4.000787e+12      0.0  151298144.0         24.0          0.0  ...   \n",
       "195478    4.000787e+12      0.0   29705424.0         24.0          0.0  ...   \n",
       "195479    4.000787e+12      0.0  103469832.0         24.0          0.0  ...   \n",
       "195480    4.000787e+12      0.0  171503104.0         25.0          0.0  ...   \n",
       "195481    4.000787e+12      0.0  202534848.0         25.0          0.0  ...   \n",
       "195482    4.000787e+12      0.0  243275368.0         25.0          0.0  ...   \n",
       "195483    4.000787e+12      0.0   76567144.0         25.0          0.0  ...   \n",
       "200466    4.000787e+12      0.0  116265744.0         20.0          0.0  ...   \n",
       "200467    4.000787e+12      0.0  154691184.0         20.0          0.0  ...   \n",
       "200468    4.000787e+12      0.0  201368272.0         20.0          0.0  ...   \n",
       "200469    4.000787e+12      0.0  243208416.0         20.0          0.0  ...   \n",
       "200470    4.000787e+12      0.0   42572824.0         20.0          0.0  ...   \n",
       "200471    4.000787e+12      0.0   80559216.0         20.0          0.0  ...   \n",
       "200472    4.000787e+12      0.0  120589400.0         20.0          0.0  ...   \n",
       "200473    4.000787e+12      0.0  160877680.0         20.0          0.0  ...   \n",
       "200474    4.000787e+12      0.0  198030056.0         20.0          0.0  ...   \n",
       "200475    4.000787e+12      0.0  237670544.0         20.0          0.0  ...   \n",
       "200476    4.000787e+12      0.0   32806016.0         20.0          0.0  ...   \n",
       "200477    4.000787e+12      0.0   74222592.0         20.0          0.0  ...   \n",
       "200478    4.000787e+12      0.0  118242064.0         20.0          0.0  ...   \n",
       "200479    4.000787e+12      0.0  159253192.0         20.0          0.0  ...   \n",
       "200480    4.000787e+12      0.0  199694096.0         20.0          0.0  ...   \n",
       "200481    4.000787e+12      0.0  239649088.0         20.0          0.0  ...   \n",
       "\n",
       "        smart_184_normalized  smart_187_normalized  smart_188_normalized  \\\n",
       "195468                 100.0                 100.0                 100.0   \n",
       "195469                 100.0                 100.0                 100.0   \n",
       "195470                 100.0                 100.0                 100.0   \n",
       "195471                 100.0                 100.0                 100.0   \n",
       "195472                 100.0                 100.0                 100.0   \n",
       "195473                 100.0                 100.0                 100.0   \n",
       "195474                 100.0                 100.0                 100.0   \n",
       "195475                 100.0                 100.0                 100.0   \n",
       "195476                 100.0                 100.0                 100.0   \n",
       "195477                 100.0                 100.0                 100.0   \n",
       "195478                 100.0                 100.0                 100.0   \n",
       "195479                 100.0                 100.0                 100.0   \n",
       "195480                 100.0                 100.0                 100.0   \n",
       "195481                 100.0                 100.0                 100.0   \n",
       "195482                 100.0                 100.0                 100.0   \n",
       "195483                 100.0                 100.0                 100.0   \n",
       "200466                 100.0                 100.0                 100.0   \n",
       "200467                 100.0                 100.0                 100.0   \n",
       "200468                 100.0                 100.0                 100.0   \n",
       "200469                 100.0                 100.0                 100.0   \n",
       "200470                 100.0                 100.0                 100.0   \n",
       "200471                 100.0                 100.0                 100.0   \n",
       "200472                 100.0                 100.0                 100.0   \n",
       "200473                 100.0                 100.0                 100.0   \n",
       "200474                 100.0                 100.0                 100.0   \n",
       "200475                 100.0                 100.0                 100.0   \n",
       "200476                 100.0                 100.0                 100.0   \n",
       "200477                 100.0                 100.0                 100.0   \n",
       "200478                 100.0                 100.0                 100.0   \n",
       "200479                 100.0                 100.0                 100.0   \n",
       "200480                 100.0                 100.0                 100.0   \n",
       "200481                 100.0                 100.0                 100.0   \n",
       "\n",
       "        smart_189_normalized  smart_190_normalized  smart_193_normalized  \\\n",
       "195468                  94.0                  75.0                  72.0   \n",
       "195469                  94.0                  76.0                  72.0   \n",
       "195470                  94.0                  75.0                  72.0   \n",
       "195471                  94.0                  75.0                  72.0   \n",
       "195472                  94.0                  75.0                  72.0   \n",
       "195473                  94.0                  75.0                  72.0   \n",
       "195474                  94.0                  75.0                  72.0   \n",
       "195475                  94.0                  75.0                  72.0   \n",
       "195476                  94.0                  75.0                  72.0   \n",
       "195477                  94.0                  75.0                  72.0   \n",
       "195478                  94.0                  75.0                  72.0   \n",
       "195479                  94.0                  75.0                  72.0   \n",
       "195480                  94.0                  75.0                  71.0   \n",
       "195481                  94.0                  76.0                  71.0   \n",
       "195482                  94.0                  76.0                  70.0   \n",
       "195483                  94.0                  75.0                  70.0   \n",
       "200466                  97.0                  75.0                  87.0   \n",
       "200467                  97.0                  75.0                  86.0   \n",
       "200468                  97.0                  74.0                  86.0   \n",
       "200469                  97.0                  74.0                  85.0   \n",
       "200470                  97.0                  75.0                  85.0   \n",
       "200471                  97.0                  74.0                  84.0   \n",
       "200472                  97.0                  75.0                  83.0   \n",
       "200473                  97.0                  75.0                  83.0   \n",
       "200474                  97.0                  75.0                  82.0   \n",
       "200475                  97.0                  75.0                  81.0   \n",
       "200476                  97.0                  75.0                  81.0   \n",
       "200477                  97.0                  75.0                  80.0   \n",
       "200478                  97.0                  76.0                  80.0   \n",
       "200479                  97.0                  76.0                  79.0   \n",
       "200480                  97.0                  77.0                  78.0   \n",
       "200481                  97.0                  77.0                  78.0   \n",
       "\n",
       "        smart_194_normalized  smart_197_normalized  smart_198_normalized  \\\n",
       "195468                  25.0                 100.0                 100.0   \n",
       "195469                  24.0                 100.0                 100.0   \n",
       "195470                  25.0                 100.0                 100.0   \n",
       "195471                  25.0                 100.0                 100.0   \n",
       "195472                  25.0                 100.0                 100.0   \n",
       "195473                  25.0                 100.0                 100.0   \n",
       "195474                  25.0                 100.0                 100.0   \n",
       "195475                  25.0                 100.0                 100.0   \n",
       "195476                  25.0                 100.0                 100.0   \n",
       "195477                  25.0                 100.0                 100.0   \n",
       "195478                  25.0                 100.0                 100.0   \n",
       "195479                  25.0                 100.0                 100.0   \n",
       "195480                  25.0                 100.0                 100.0   \n",
       "195481                  24.0                 100.0                 100.0   \n",
       "195482                  24.0                 100.0                 100.0   \n",
       "195483                  25.0                 100.0                 100.0   \n",
       "200466                  25.0                 100.0                 100.0   \n",
       "200467                  25.0                 100.0                 100.0   \n",
       "200468                  26.0                 100.0                 100.0   \n",
       "200469                  26.0                 100.0                 100.0   \n",
       "200470                  25.0                 100.0                 100.0   \n",
       "200471                  26.0                 100.0                 100.0   \n",
       "200472                  25.0                 100.0                 100.0   \n",
       "200473                  25.0                 100.0                 100.0   \n",
       "200474                  25.0                 100.0                 100.0   \n",
       "200475                  25.0                 100.0                 100.0   \n",
       "200476                  25.0                 100.0                 100.0   \n",
       "200477                  25.0                 100.0                 100.0   \n",
       "200478                  24.0                 100.0                 100.0   \n",
       "200479                  24.0                 100.0                 100.0   \n",
       "200480                  23.0                 100.0                 100.0   \n",
       "200481                  23.0                 100.0                 100.0   \n",
       "\n",
       "        smart_199_normalized  \n",
       "195468                 200.0  \n",
       "195469                 200.0  \n",
       "195470                 200.0  \n",
       "195471                 200.0  \n",
       "195472                 200.0  \n",
       "195473                 200.0  \n",
       "195474                 200.0  \n",
       "195475                 200.0  \n",
       "195476                 200.0  \n",
       "195477                 200.0  \n",
       "195478                 200.0  \n",
       "195479                 200.0  \n",
       "195480                 200.0  \n",
       "195481                 200.0  \n",
       "195482                 200.0  \n",
       "195483                 200.0  \n",
       "200466                 200.0  \n",
       "200467                 200.0  \n",
       "200468                 200.0  \n",
       "200469                 200.0  \n",
       "200470                 200.0  \n",
       "200471                 200.0  \n",
       "200472                 200.0  \n",
       "200473                 200.0  \n",
       "200474                 200.0  \n",
       "200475                 200.0  \n",
       "200476                 200.0  \n",
       "200477                 200.0  \n",
       "200478                 200.0  \n",
       "200479                 200.0  \n",
       "200480                 200.0  \n",
       "200481                 200.0  \n",
       "\n",
       "[32 rows x 46 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2*sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation Routines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To work with the data, we are going to build a more structured set of functions that are going to help us with data curation. As stated in the previous lab, the dataset contains duplicated columns with \"normalized\" data. Our first function, `remove_normalized`, is aimed to remove those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the \"normalized\" columns which are simply nomralized copies of the \"raw\" column\n",
    "\n",
    "def remove_normalized(df):\n",
    "    cols = [c for c in df.columns if (c.lower().find(\"normalized\")==-1)]\n",
    "    df=df[cols]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e1\"></a>\n",
    "\n",
    "### Exercise 1: Retrieve the output labels\n",
    "\n",
    "We would like to get the failure column in NumPy format. Please complete the following function to perform return the appropriate labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_Labels(df):\n",
    "    # return the proper column in NumPy format\n",
    "#     << TO DO >>\n",
    "    y_labels = df['failure'].values\n",
    "    return y_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](#a1) for answer\n",
    "\n",
    "### Sort data based on `ae_seq_id` and `ae_seq_data_id` columns\n",
    "\n",
    "Prior to training, we need to ensure that all the data is properly sorted so that sequences are presented in the correct order.  Typically, we would sort the data by unique id and date/time.  When we created the data set for this exercise, we added two fields:\n",
    "<ul>\n",
    "    <li>ae_seq_id: Id number of unique sequence</li>\n",
    "    <li>ae_seq_data_id: Day number of record in sequence (day 0 - day (sequence_length-1))</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort data frame to ensure proper order\n",
    "\n",
    "def sort_dataframe(df):\n",
    "    print(\"Sorting the data frame based on pre-created sequence id and sequence day id\")\n",
    "    df['date'] = pd.to_datetime(df['date'])\n",
    "    \n",
    "    #sort dataframe by 'ae_seq_id' and 'ae_seq_data_id'\n",
    "    df = df.sort_values(by=['ae_seq_id', 'ae_seq_data_id'], axis=0, ascending=True)\n",
    "    df = df.reset_index(drop=True)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we have created some \"helper\" functions to:\n",
    "<ul>\n",
    "    <li>Remove columns that possess no useful information for training</li>\n",
    "    <li>Normalize the columns</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove columns unnecessary for training\n",
    "\n",
    "def remove_columns(df, drop_columns_list = ['date','serial_number',\n",
    "                                               'model', \n",
    "                                               'failure', \n",
    "                                               'capacity_bytes']):\n",
    "    print(\"removing unnecessary columns\")\n",
    "    df = df.drop(columns=drop_columns_list)\n",
    "    df = df.reset_index(drop=True)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This routine removes the normalized features and removes unnecessary columns\n",
    "\n",
    "def clean_dataframe(df,column_list=None):\n",
    "        \n",
    "    #sort the dataframe\n",
    "    df = sort_dataframe(df)\n",
    "\n",
    "    #remove normalized columns\n",
    "    df = remove_normalized(df)\n",
    "\n",
    "    #remove other unnecessary columns\n",
    "    if column_list is not None:\n",
    "        df = remove_columns(df, column_list)\n",
    "    \n",
    "    return df\n",
    "\n",
    "# This routine scales values for training and reshapes data as appropriate for LSTM or CONV1D training\n",
    "\n",
    "def normalize_df(df, ae_type=None,scaler=None):\n",
    "    \n",
    "    print(\"Normalizing input for \"+ae_type)\n",
    "    #normalize data with max/min values \n",
    "    df.fillna(0, inplace=True)\n",
    "    x = df.values\n",
    "    x = x.astype(np.float64)\n",
    "    if scaler is None:\n",
    "        scaler = preprocessing.StandardScaler().fit(x)\n",
    "    x = scaler.transform(x)\n",
    "    \n",
    "    #reshape to obtain appropriate input for autoencoder\n",
    "    if ae_type == \"lstm\": \n",
    "        x = x.reshape(int(x.shape[0]/sequence_length), sequence_length, x.shape[1])\n",
    "    elif ae_type == \"conv1d\": \n",
    "        tshape = x.shape\n",
    "        print(tshape, \"dropping \", tshape[0]-int(tshape[0]/w_size)*w_size, end = \" : reshaping to \")       \n",
    "        x = x[:int(tshape[0]/w_size)*w_size,:].reshape(int(tshape[0]/w_size), w_size, tshape[1])\n",
    "\n",
    "    print(x.shape)\n",
    "    \n",
    "    return x, scaler \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to load both training and test sets below, starting with the training set. Note that unlike the LSTM models, here we do not require the `failure` columns. The class labels are not what we learn, rather we learn the training data pattern. In that sense, autoencoders are unsupervised models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing the train set\n",
      "Sorting the data frame based on pre-created sequence id and sequence day id\n",
      "removing unnecessary columns\n",
      "Normalizing input for lstm\n",
      "(4625, 16, 21)\n"
     ]
    }
   ],
   "source": [
    "print(\"processing the train set\")\n",
    "\n",
    "df = clean_dataframe(df,['ae_seq_id', \n",
    "                            'ae_seq_data_id', \n",
    "                            'date','serial_number',\n",
    "                            'model', \n",
    "                            'failure', \n",
    "                           'capacity_bytes'])\n",
    "\n",
    "x_train, scaler = normalize_df(df,\"lstm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at a few samples of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>smart_12_raw</th>\n",
       "      <th>smart_183_raw</th>\n",
       "      <th>smart_184_raw</th>\n",
       "      <th>smart_187_raw</th>\n",
       "      <th>smart_188_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_190_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>169386552.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11790406.0</td>\n",
       "      <td>34692.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57729.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34450.0</td>\n",
       "      <td>3.177469e+10</td>\n",
       "      <td>1.425277e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>220437952.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12531683.0</td>\n",
       "      <td>34716.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34474.0</td>\n",
       "      <td>3.177673e+10</td>\n",
       "      <td>1.425649e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12172112.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13249488.0</td>\n",
       "      <td>34740.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34498.0</td>\n",
       "      <td>3.177876e+10</td>\n",
       "      <td>1.426023e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33350928.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13970733.0</td>\n",
       "      <td>34764.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34522.0</td>\n",
       "      <td>3.178076e+10</td>\n",
       "      <td>1.426377e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>98402824.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14725252.0</td>\n",
       "      <td>34788.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34546.0</td>\n",
       "      <td>3.178302e+10</td>\n",
       "      <td>1.426726e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>173407640.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15477836.0</td>\n",
       "      <td>34812.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34570.0</td>\n",
       "      <td>3.178512e+10</td>\n",
       "      <td>1.427067e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>59425848.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16264486.0</td>\n",
       "      <td>34836.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34594.0</td>\n",
       "      <td>3.178752e+10</td>\n",
       "      <td>1.427406e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>178379120.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17053336.0</td>\n",
       "      <td>34860.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34618.0</td>\n",
       "      <td>3.178998e+10</td>\n",
       "      <td>1.427768e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>21791640.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17824256.0</td>\n",
       "      <td>34884.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34642.0</td>\n",
       "      <td>3.179215e+10</td>\n",
       "      <td>1.428109e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>151298144.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18616634.0</td>\n",
       "      <td>34908.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34666.0</td>\n",
       "      <td>3.179427e+10</td>\n",
       "      <td>1.428451e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>29705424.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19396235.0</td>\n",
       "      <td>34932.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57731.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34690.0</td>\n",
       "      <td>3.179640e+10</td>\n",
       "      <td>1.428825e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>103469832.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20150989.0</td>\n",
       "      <td>34956.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>57732.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34714.0</td>\n",
       "      <td>3.179845e+10</td>\n",
       "      <td>1.429180e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>171503104.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20594596.0</td>\n",
       "      <td>34981.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>58409.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34737.0</td>\n",
       "      <td>3.179961e+10</td>\n",
       "      <td>1.429477e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>202534848.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20683853.0</td>\n",
       "      <td>35005.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>59661.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34755.0</td>\n",
       "      <td>3.179972e+10</td>\n",
       "      <td>1.429751e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>243275368.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20778485.0</td>\n",
       "      <td>35029.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>60884.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34774.0</td>\n",
       "      <td>3.179984e+10</td>\n",
       "      <td>1.430094e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>76567144.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21192916.0</td>\n",
       "      <td>35053.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>61772.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>34795.0</td>\n",
       "      <td>3.180088e+10</td>\n",
       "      <td>1.430484e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>116265744.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120332000.0</td>\n",
       "      <td>33461.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>26818.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33350.0</td>\n",
       "      <td>3.084112e+10</td>\n",
       "      <td>1.919088e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>154691184.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120425993.0</td>\n",
       "      <td>33484.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>28084.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33369.0</td>\n",
       "      <td>3.084124e+10</td>\n",
       "      <td>1.919419e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>201368272.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120522492.0</td>\n",
       "      <td>33508.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29240.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33389.0</td>\n",
       "      <td>3.084135e+10</td>\n",
       "      <td>1.919757e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>243208416.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120617730.0</td>\n",
       "      <td>33532.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>30410.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33409.0</td>\n",
       "      <td>3.084147e+10</td>\n",
       "      <td>1.920092e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>42572824.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120716059.0</td>\n",
       "      <td>33556.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>31656.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33429.0</td>\n",
       "      <td>3.084159e+10</td>\n",
       "      <td>1.920439e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>80559216.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120810048.0</td>\n",
       "      <td>33580.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>32938.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33448.0</td>\n",
       "      <td>3.084171e+10</td>\n",
       "      <td>1.920758e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>120589400.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120904269.0</td>\n",
       "      <td>33604.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>34195.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33467.0</td>\n",
       "      <td>3.084182e+10</td>\n",
       "      <td>1.921093e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>160877680.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121001196.0</td>\n",
       "      <td>33628.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>35548.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33487.0</td>\n",
       "      <td>3.084194e+10</td>\n",
       "      <td>1.921437e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>198030056.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121095069.0</td>\n",
       "      <td>33652.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>36779.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33506.0</td>\n",
       "      <td>3.084206e+10</td>\n",
       "      <td>1.921758e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>237670544.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121189166.0</td>\n",
       "      <td>33676.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>38030.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33525.0</td>\n",
       "      <td>3.084218e+10</td>\n",
       "      <td>1.922090e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>32806016.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121286158.0</td>\n",
       "      <td>33701.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>39338.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33545.0</td>\n",
       "      <td>3.084230e+10</td>\n",
       "      <td>1.922431e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>74222592.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121380839.0</td>\n",
       "      <td>33724.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>40607.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33564.0</td>\n",
       "      <td>3.084241e+10</td>\n",
       "      <td>1.922767e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>118242064.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121476046.0</td>\n",
       "      <td>33748.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>41828.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33584.0</td>\n",
       "      <td>3.084253e+10</td>\n",
       "      <td>1.923092e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>159253192.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121570379.0</td>\n",
       "      <td>33772.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>43101.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33603.0</td>\n",
       "      <td>3.084265e+10</td>\n",
       "      <td>1.923429e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>199694096.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121667225.0</td>\n",
       "      <td>33796.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>44376.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33623.0</td>\n",
       "      <td>3.084277e+10</td>\n",
       "      <td>1.923776e+11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>239649088.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>121761333.0</td>\n",
       "      <td>33820.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>45637.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33642.0</td>\n",
       "      <td>3.084288e+10</td>\n",
       "      <td>1.924095e+11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    smart_1_raw  smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  \\\n",
       "0   169386552.0         24.0          0.0   11790406.0      34692.0   \n",
       "1   220437952.0         24.0          0.0   12531683.0      34716.0   \n",
       "2    12172112.0         24.0          0.0   13249488.0      34740.0   \n",
       "3    33350928.0         24.0          0.0   13970733.0      34764.0   \n",
       "4    98402824.0         24.0          0.0   14725252.0      34788.0   \n",
       "5   173407640.0         24.0          0.0   15477836.0      34812.0   \n",
       "6    59425848.0         24.0          0.0   16264486.0      34836.0   \n",
       "7   178379120.0         24.0          0.0   17053336.0      34860.0   \n",
       "8    21791640.0         24.0          0.0   17824256.0      34884.0   \n",
       "9   151298144.0         24.0          0.0   18616634.0      34908.0   \n",
       "10   29705424.0         24.0          0.0   19396235.0      34932.0   \n",
       "11  103469832.0         24.0          0.0   20150989.0      34956.0   \n",
       "12  171503104.0         25.0          0.0   20594596.0      34981.0   \n",
       "13  202534848.0         25.0          0.0   20683853.0      35005.0   \n",
       "14  243275368.0         25.0          0.0   20778485.0      35029.0   \n",
       "15   76567144.0         25.0          0.0   21192916.0      35053.0   \n",
       "16  116265744.0         20.0          0.0  120332000.0      33461.0   \n",
       "17  154691184.0         20.0          0.0  120425993.0      33484.0   \n",
       "18  201368272.0         20.0          0.0  120522492.0      33508.0   \n",
       "19  243208416.0         20.0          0.0  120617730.0      33532.0   \n",
       "20   42572824.0         20.0          0.0  120716059.0      33556.0   \n",
       "21   80559216.0         20.0          0.0  120810048.0      33580.0   \n",
       "22  120589400.0         20.0          0.0  120904269.0      33604.0   \n",
       "23  160877680.0         20.0          0.0  121001196.0      33628.0   \n",
       "24  198030056.0         20.0          0.0  121095069.0      33652.0   \n",
       "25  237670544.0         20.0          0.0  121189166.0      33676.0   \n",
       "26   32806016.0         20.0          0.0  121286158.0      33701.0   \n",
       "27   74222592.0         20.0          0.0  121380839.0      33724.0   \n",
       "28  118242064.0         20.0          0.0  121476046.0      33748.0   \n",
       "29  159253192.0         20.0          0.0  121570379.0      33772.0   \n",
       "30  199694096.0         20.0          0.0  121667225.0      33796.0   \n",
       "31  239649088.0         20.0          0.0  121761333.0      33820.0   \n",
       "\n",
       "    smart_12_raw  smart_183_raw  smart_184_raw  smart_187_raw  smart_188_raw  \\\n",
       "0           24.0            1.0            0.0            0.0            0.0   \n",
       "1           24.0            1.0            0.0            0.0            0.0   \n",
       "2           24.0            1.0            0.0            0.0            0.0   \n",
       "3           24.0            1.0            0.0            0.0            0.0   \n",
       "4           24.0            1.0            0.0            0.0            0.0   \n",
       "5           24.0            1.0            0.0            0.0            0.0   \n",
       "6           24.0            1.0            0.0            0.0            0.0   \n",
       "7           24.0            1.0            0.0            0.0            0.0   \n",
       "8           24.0            1.0            0.0            0.0            0.0   \n",
       "9           24.0            1.0            0.0            0.0            0.0   \n",
       "10          24.0            1.0            0.0            0.0            0.0   \n",
       "11          24.0            1.0            0.0            0.0            0.0   \n",
       "12          25.0            1.0            0.0            0.0            0.0   \n",
       "13          25.0            1.0            0.0            0.0            0.0   \n",
       "14          25.0            1.0            0.0            0.0            0.0   \n",
       "15          25.0            1.0            0.0            0.0            0.0   \n",
       "16          20.0            1.0            0.0            0.0            0.0   \n",
       "17          20.0            1.0            0.0            0.0            0.0   \n",
       "18          20.0            1.0            0.0            0.0            0.0   \n",
       "19          20.0            1.0            0.0            0.0            0.0   \n",
       "20          20.0            1.0            0.0            0.0            0.0   \n",
       "21          20.0            1.0            0.0            0.0            0.0   \n",
       "22          20.0            1.0            0.0            0.0            0.0   \n",
       "23          20.0            1.0            0.0            0.0            0.0   \n",
       "24          20.0            1.0            0.0            0.0            0.0   \n",
       "25          20.0            1.0            0.0            0.0            0.0   \n",
       "26          20.0            1.0            0.0            0.0            0.0   \n",
       "27          20.0            1.0            0.0            0.0            0.0   \n",
       "28          20.0            1.0            0.0            0.0            0.0   \n",
       "29          20.0            1.0            0.0            0.0            0.0   \n",
       "30          20.0            1.0            0.0            0.0            0.0   \n",
       "31          20.0            1.0            0.0            0.0            0.0   \n",
       "\n",
       "    ...  smart_190_raw  smart_192_raw  smart_193_raw  smart_194_raw  \\\n",
       "0   ...           25.0           14.0        57729.0           25.0   \n",
       "1   ...           24.0           14.0        57731.0           24.0   \n",
       "2   ...           25.0           14.0        57731.0           25.0   \n",
       "3   ...           25.0           14.0        57731.0           25.0   \n",
       "4   ...           25.0           14.0        57731.0           25.0   \n",
       "5   ...           25.0           14.0        57731.0           25.0   \n",
       "6   ...           25.0           14.0        57731.0           25.0   \n",
       "7   ...           25.0           14.0        57731.0           25.0   \n",
       "8   ...           25.0           14.0        57731.0           25.0   \n",
       "9   ...           25.0           14.0        57731.0           25.0   \n",
       "10  ...           25.0           14.0        57731.0           25.0   \n",
       "11  ...           25.0           14.0        57732.0           25.0   \n",
       "12  ...           25.0           14.0        58409.0           25.0   \n",
       "13  ...           24.0           14.0        59661.0           24.0   \n",
       "14  ...           24.0           14.0        60884.0           24.0   \n",
       "15  ...           25.0           14.0        61772.0           25.0   \n",
       "16  ...           25.0            3.0        26818.0           25.0   \n",
       "17  ...           25.0            3.0        28084.0           25.0   \n",
       "18  ...           26.0            3.0        29240.0           26.0   \n",
       "19  ...           26.0            3.0        30410.0           26.0   \n",
       "20  ...           25.0            3.0        31656.0           25.0   \n",
       "21  ...           26.0            3.0        32938.0           26.0   \n",
       "22  ...           25.0            3.0        34195.0           25.0   \n",
       "23  ...           25.0            3.0        35548.0           25.0   \n",
       "24  ...           25.0            3.0        36779.0           25.0   \n",
       "25  ...           25.0            3.0        38030.0           25.0   \n",
       "26  ...           25.0            3.0        39338.0           25.0   \n",
       "27  ...           25.0            3.0        40607.0           25.0   \n",
       "28  ...           24.0            3.0        41828.0           24.0   \n",
       "29  ...           24.0            3.0        43101.0           24.0   \n",
       "30  ...           23.0            3.0        44376.0           23.0   \n",
       "31  ...           23.0            3.0        45637.0           23.0   \n",
       "\n",
       "    smart_197_raw  smart_198_raw  smart_199_raw  smart_240_raw  smart_241_raw  \\\n",
       "0             0.0            0.0            0.0        34450.0   3.177469e+10   \n",
       "1             0.0            0.0            0.0        34474.0   3.177673e+10   \n",
       "2             0.0            0.0            0.0        34498.0   3.177876e+10   \n",
       "3             0.0            0.0            0.0        34522.0   3.178076e+10   \n",
       "4             0.0            0.0            0.0        34546.0   3.178302e+10   \n",
       "5             0.0            0.0            0.0        34570.0   3.178512e+10   \n",
       "6             0.0            0.0            0.0        34594.0   3.178752e+10   \n",
       "7             0.0            0.0            0.0        34618.0   3.178998e+10   \n",
       "8             0.0            0.0            0.0        34642.0   3.179215e+10   \n",
       "9             0.0            0.0            0.0        34666.0   3.179427e+10   \n",
       "10            0.0            0.0            0.0        34690.0   3.179640e+10   \n",
       "11            0.0            0.0            0.0        34714.0   3.179845e+10   \n",
       "12            0.0            0.0            0.0        34737.0   3.179961e+10   \n",
       "13            0.0            0.0            0.0        34755.0   3.179972e+10   \n",
       "14            0.0            0.0            0.0        34774.0   3.179984e+10   \n",
       "15            0.0            0.0            0.0        34795.0   3.180088e+10   \n",
       "16            0.0            0.0            0.0        33350.0   3.084112e+10   \n",
       "17            0.0            0.0            0.0        33369.0   3.084124e+10   \n",
       "18            0.0            0.0            0.0        33389.0   3.084135e+10   \n",
       "19            0.0            0.0            0.0        33409.0   3.084147e+10   \n",
       "20            0.0            0.0            0.0        33429.0   3.084159e+10   \n",
       "21            0.0            0.0            0.0        33448.0   3.084171e+10   \n",
       "22            0.0            0.0            0.0        33467.0   3.084182e+10   \n",
       "23            0.0            0.0            0.0        33487.0   3.084194e+10   \n",
       "24            0.0            0.0            0.0        33506.0   3.084206e+10   \n",
       "25            0.0            0.0            0.0        33525.0   3.084218e+10   \n",
       "26            0.0            0.0            0.0        33545.0   3.084230e+10   \n",
       "27            0.0            0.0            0.0        33564.0   3.084241e+10   \n",
       "28            0.0            0.0            0.0        33584.0   3.084253e+10   \n",
       "29            0.0            0.0            0.0        33603.0   3.084265e+10   \n",
       "30            0.0            0.0            0.0        33623.0   3.084277e+10   \n",
       "31            0.0            0.0            0.0        33642.0   3.084288e+10   \n",
       "\n",
       "    smart_242_raw  \n",
       "0    1.425277e+11  \n",
       "1    1.425649e+11  \n",
       "2    1.426023e+11  \n",
       "3    1.426377e+11  \n",
       "4    1.426726e+11  \n",
       "5    1.427067e+11  \n",
       "6    1.427406e+11  \n",
       "7    1.427768e+11  \n",
       "8    1.428109e+11  \n",
       "9    1.428451e+11  \n",
       "10   1.428825e+11  \n",
       "11   1.429180e+11  \n",
       "12   1.429477e+11  \n",
       "13   1.429751e+11  \n",
       "14   1.430094e+11  \n",
       "15   1.430484e+11  \n",
       "16   1.919088e+11  \n",
       "17   1.919419e+11  \n",
       "18   1.919757e+11  \n",
       "19   1.920092e+11  \n",
       "20   1.920439e+11  \n",
       "21   1.920758e+11  \n",
       "22   1.921093e+11  \n",
       "23   1.921437e+11  \n",
       "24   1.921758e+11  \n",
       "25   1.922090e+11  \n",
       "26   1.922431e+11  \n",
       "27   1.922767e+11  \n",
       "28   1.923092e+11  \n",
       "29   1.923429e+11  \n",
       "30   1.923776e+11  \n",
       "31   1.924095e+11  \n",
       "\n",
       "[32 rows x 21 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2*sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to load the test set. Unlike, the train set, we require the `failure` column for the test set to evaluate the model accuracy at the end. Again, you can see that we are using the processed dataset from the previous lab for our test purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sorting the data frame based on pre-created sequence id and sequence day id\n",
      "removing unnecessary columns\n",
      "Normalizing input for lstm\n",
      "(834, 16, 21)\n"
     ]
    }
   ],
   "source": [
    "df_test = pd.read_pickle(test_fname)\n",
    "y_labels = get_Labels(df_test)\n",
    "\n",
    "df_test = clean_dataframe(df_test, ['ae_seq_id', 'ae_seq_data_id', 'date','serial_number','model', \n",
    "                            'failure', \n",
    "                            'capacity_bytes'])\n",
    "\n",
    "x_test, _ = normalize_df(df_test,'lstm',scaler=scaler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>smart_1_raw</th>\n",
       "      <th>smart_4_raw</th>\n",
       "      <th>smart_5_raw</th>\n",
       "      <th>smart_7_raw</th>\n",
       "      <th>smart_9_raw</th>\n",
       "      <th>smart_12_raw</th>\n",
       "      <th>smart_183_raw</th>\n",
       "      <th>smart_184_raw</th>\n",
       "      <th>smart_187_raw</th>\n",
       "      <th>smart_188_raw</th>\n",
       "      <th>...</th>\n",
       "      <th>smart_190_raw</th>\n",
       "      <th>smart_192_raw</th>\n",
       "      <th>smart_193_raw</th>\n",
       "      <th>smart_194_raw</th>\n",
       "      <th>smart_197_raw</th>\n",
       "      <th>smart_198_raw</th>\n",
       "      <th>smart_199_raw</th>\n",
       "      <th>smart_240_raw</th>\n",
       "      <th>smart_241_raw</th>\n",
       "      <th>smart_242_raw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20931656.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>445083804.0</td>\n",
       "      <td>10497.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19298.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10389.0</td>\n",
       "      <td>3.606082e+10</td>\n",
       "      <td>8.751856e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>172359472.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>445791739.0</td>\n",
       "      <td>10521.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19298.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10413.0</td>\n",
       "      <td>3.609658e+10</td>\n",
       "      <td>8.776299e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21705248.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>447586660.0</td>\n",
       "      <td>10545.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19298.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10437.0</td>\n",
       "      <td>3.614011e+10</td>\n",
       "      <td>8.817362e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>129069792.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>448820783.0</td>\n",
       "      <td>10568.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19298.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10461.0</td>\n",
       "      <td>3.617421e+10</td>\n",
       "      <td>8.851069e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68452888.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>449512275.0</td>\n",
       "      <td>10593.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19298.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10485.0</td>\n",
       "      <td>3.621012e+10</td>\n",
       "      <td>8.869208e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>59189440.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>449985857.0</td>\n",
       "      <td>10617.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19300.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10509.0</td>\n",
       "      <td>3.624462e+10</td>\n",
       "      <td>8.878416e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>117643832.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>451076087.0</td>\n",
       "      <td>10641.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19300.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10533.0</td>\n",
       "      <td>3.628199e+10</td>\n",
       "      <td>8.901264e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>210523216.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>451546733.0</td>\n",
       "      <td>10665.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>21.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19300.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10557.0</td>\n",
       "      <td>3.631089e+10</td>\n",
       "      <td>8.910142e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2369744.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>451908456.0</td>\n",
       "      <td>10689.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19300.0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10581.0</td>\n",
       "      <td>3.633408e+10</td>\n",
       "      <td>8.916714e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>130718904.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>452575870.0</td>\n",
       "      <td>10713.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10605.0</td>\n",
       "      <td>3.638426e+10</td>\n",
       "      <td>8.928378e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>118628456.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>453641142.0</td>\n",
       "      <td>10737.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10629.0</td>\n",
       "      <td>3.641227e+10</td>\n",
       "      <td>8.952648e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>182296152.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>454176955.0</td>\n",
       "      <td>10761.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10653.0</td>\n",
       "      <td>3.645253e+10</td>\n",
       "      <td>8.961123e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>47644936.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>455687902.0</td>\n",
       "      <td>10785.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10677.0</td>\n",
       "      <td>3.649212e+10</td>\n",
       "      <td>8.984780e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>209559408.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>456866368.0</td>\n",
       "      <td>10809.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10701.0</td>\n",
       "      <td>3.654030e+10</td>\n",
       "      <td>9.006662e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>52369144.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>457597063.0</td>\n",
       "      <td>10833.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10725.0</td>\n",
       "      <td>3.660039e+10</td>\n",
       "      <td>9.017427e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>179911504.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>458086103.0</td>\n",
       "      <td>10857.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19303.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10749.0</td>\n",
       "      <td>3.663817e+10</td>\n",
       "      <td>9.024461e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>170489672.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>544653734.0</td>\n",
       "      <td>13739.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30424.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13582.0</td>\n",
       "      <td>3.857965e+10</td>\n",
       "      <td>2.475205e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>218930264.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>545302736.0</td>\n",
       "      <td>13763.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13606.0</td>\n",
       "      <td>3.861341e+10</td>\n",
       "      <td>2.477526e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>121983304.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>546610816.0</td>\n",
       "      <td>13787.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13630.0</td>\n",
       "      <td>3.864594e+10</td>\n",
       "      <td>2.481269e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>84711632.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>547290773.0</td>\n",
       "      <td>13811.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13654.0</td>\n",
       "      <td>3.867881e+10</td>\n",
       "      <td>2.483501e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>109898208.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>548939592.0</td>\n",
       "      <td>13834.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13678.0</td>\n",
       "      <td>3.873947e+10</td>\n",
       "      <td>2.487543e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>104488776.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>551032632.0</td>\n",
       "      <td>13858.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13702.0</td>\n",
       "      <td>3.879835e+10</td>\n",
       "      <td>2.491775e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>181122920.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>551724773.0</td>\n",
       "      <td>13883.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13726.0</td>\n",
       "      <td>3.883539e+10</td>\n",
       "      <td>2.493543e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>36315568.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>552278534.0</td>\n",
       "      <td>13906.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13750.0</td>\n",
       "      <td>3.885619e+10</td>\n",
       "      <td>2.495545e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>82759384.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>552843651.0</td>\n",
       "      <td>13930.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13774.0</td>\n",
       "      <td>3.888197e+10</td>\n",
       "      <td>2.497250e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>151460032.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>554130031.0</td>\n",
       "      <td>13954.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13797.0</td>\n",
       "      <td>3.891351e+10</td>\n",
       "      <td>2.500865e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>182983096.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>554784945.0</td>\n",
       "      <td>13979.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13822.0</td>\n",
       "      <td>3.894760e+10</td>\n",
       "      <td>2.502570e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>112044808.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>556215819.0</td>\n",
       "      <td>14002.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13846.0</td>\n",
       "      <td>3.897927e+10</td>\n",
       "      <td>2.506768e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>204594816.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>556815879.0</td>\n",
       "      <td>14026.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13870.0</td>\n",
       "      <td>3.900667e+10</td>\n",
       "      <td>2.508617e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>31216360.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>557416355.0</td>\n",
       "      <td>14050.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13893.0</td>\n",
       "      <td>3.903526e+10</td>\n",
       "      <td>2.510276e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>194976032.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>558093153.0</td>\n",
       "      <td>14075.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13918.0</td>\n",
       "      <td>3.906552e+10</td>\n",
       "      <td>2.512448e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>75671016.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>558691985.0</td>\n",
       "      <td>14099.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30427.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13942.0</td>\n",
       "      <td>3.909157e+10</td>\n",
       "      <td>2.514321e+10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>32 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    smart_1_raw  smart_4_raw  smart_5_raw  smart_7_raw  smart_9_raw  \\\n",
       "0    20931656.0          5.0          0.0  445083804.0      10497.0   \n",
       "1   172359472.0          5.0          0.0  445791739.0      10521.0   \n",
       "2    21705248.0          5.0          0.0  447586660.0      10545.0   \n",
       "3   129069792.0          5.0          0.0  448820783.0      10568.0   \n",
       "4    68452888.0          5.0          0.0  449512275.0      10593.0   \n",
       "5    59189440.0          5.0          0.0  449985857.0      10617.0   \n",
       "6   117643832.0          5.0          0.0  451076087.0      10641.0   \n",
       "7   210523216.0          5.0          0.0  451546733.0      10665.0   \n",
       "8     2369744.0          5.0          0.0  451908456.0      10689.0   \n",
       "9   130718904.0          5.0          0.0  452575870.0      10713.0   \n",
       "10  118628456.0          5.0          0.0  453641142.0      10737.0   \n",
       "11  182296152.0          5.0          0.0  454176955.0      10761.0   \n",
       "12   47644936.0          5.0          0.0  455687902.0      10785.0   \n",
       "13  209559408.0          5.0          0.0  456866368.0      10809.0   \n",
       "14   52369144.0          5.0          0.0  457597063.0      10833.0   \n",
       "15  179911504.0          5.0          0.0  458086103.0      10857.0   \n",
       "16  170489672.0          1.0          0.0  544653734.0      13739.0   \n",
       "17  218930264.0          1.0          0.0  545302736.0      13763.0   \n",
       "18  121983304.0          1.0          0.0  546610816.0      13787.0   \n",
       "19   84711632.0          1.0          0.0  547290773.0      13811.0   \n",
       "20  109898208.0          1.0          0.0  548939592.0      13834.0   \n",
       "21  104488776.0          1.0          0.0  551032632.0      13858.0   \n",
       "22  181122920.0          1.0          0.0  551724773.0      13883.0   \n",
       "23   36315568.0          1.0          0.0  552278534.0      13906.0   \n",
       "24   82759384.0          1.0          0.0  552843651.0      13930.0   \n",
       "25  151460032.0          1.0          0.0  554130031.0      13954.0   \n",
       "26  182983096.0          1.0          0.0  554784945.0      13979.0   \n",
       "27  112044808.0          1.0          0.0  556215819.0      14002.0   \n",
       "28  204594816.0          1.0          0.0  556815879.0      14026.0   \n",
       "29   31216360.0          1.0          0.0  557416355.0      14050.0   \n",
       "30  194976032.0          1.0          0.0  558093153.0      14075.0   \n",
       "31   75671016.0          1.0          0.0  558691985.0      14099.0   \n",
       "\n",
       "    smart_12_raw  smart_183_raw  smart_184_raw  smart_187_raw  smart_188_raw  \\\n",
       "0            5.0            0.0            0.0            0.0            0.0   \n",
       "1            5.0            0.0            0.0            0.0            0.0   \n",
       "2            5.0            0.0            0.0            0.0            0.0   \n",
       "3            5.0            0.0            0.0            0.0            0.0   \n",
       "4            5.0            0.0            0.0            0.0            0.0   \n",
       "5            5.0            0.0            0.0            0.0            0.0   \n",
       "6            5.0            0.0            0.0            0.0            0.0   \n",
       "7            5.0            0.0            0.0            0.0            0.0   \n",
       "8            5.0            0.0            0.0            0.0            0.0   \n",
       "9            5.0            0.0            0.0            0.0            0.0   \n",
       "10           5.0            0.0            0.0            0.0            0.0   \n",
       "11           5.0            0.0            0.0            0.0            0.0   \n",
       "12           5.0            0.0            0.0            0.0            0.0   \n",
       "13           5.0            0.0            0.0            0.0            0.0   \n",
       "14           5.0            0.0            0.0            0.0            0.0   \n",
       "15           5.0            0.0            0.0            0.0            0.0   \n",
       "16           1.0            0.0            0.0            0.0            0.0   \n",
       "17           1.0            0.0            0.0            0.0            0.0   \n",
       "18           1.0            0.0            0.0            0.0            0.0   \n",
       "19           1.0            0.0            0.0            0.0            0.0   \n",
       "20           1.0            0.0            0.0            0.0            0.0   \n",
       "21           1.0            0.0            0.0            0.0            0.0   \n",
       "22           1.0            0.0            0.0            0.0            0.0   \n",
       "23           1.0            0.0            0.0            0.0            0.0   \n",
       "24           1.0            0.0            0.0            0.0            0.0   \n",
       "25           1.0            0.0            0.0            0.0            0.0   \n",
       "26           1.0            0.0            0.0            0.0            0.0   \n",
       "27           1.0            0.0            0.0            0.0            0.0   \n",
       "28           1.0            0.0            0.0            0.0            0.0   \n",
       "29           1.0            0.0            0.0            0.0            0.0   \n",
       "30           1.0            0.0            0.0            0.0            0.0   \n",
       "31           1.0            0.0            0.0            0.0            0.0   \n",
       "\n",
       "    ...  smart_190_raw  smart_192_raw  smart_193_raw  smart_194_raw  \\\n",
       "0   ...           19.0            0.0        19298.0           19.0   \n",
       "1   ...           19.0            0.0        19298.0           19.0   \n",
       "2   ...           19.0            0.0        19298.0           19.0   \n",
       "3   ...           20.0            0.0        19298.0           20.0   \n",
       "4   ...           20.0            0.0        19298.0           20.0   \n",
       "5   ...           20.0            0.0        19300.0           20.0   \n",
       "6   ...           21.0            0.0        19300.0           21.0   \n",
       "7   ...           21.0            0.0        19300.0           21.0   \n",
       "8   ...           26.0            0.0        19300.0           26.0   \n",
       "9   ...           23.0            0.0        19303.0           23.0   \n",
       "10  ...           22.0            0.0        19303.0           22.0   \n",
       "11  ...           23.0            0.0        19303.0           23.0   \n",
       "12  ...           24.0            0.0        19303.0           24.0   \n",
       "13  ...           24.0            0.0        19303.0           24.0   \n",
       "14  ...           25.0            0.0        19303.0           25.0   \n",
       "15  ...           24.0            0.0        19303.0           24.0   \n",
       "16  ...           28.0            0.0        30424.0           28.0   \n",
       "17  ...           24.0            0.0        30427.0           24.0   \n",
       "18  ...           23.0            0.0        30427.0           23.0   \n",
       "19  ...           22.0            0.0        30427.0           22.0   \n",
       "20  ...           25.0            0.0        30427.0           25.0   \n",
       "21  ...           25.0            0.0        30427.0           25.0   \n",
       "22  ...           25.0            0.0        30427.0           25.0   \n",
       "23  ...           24.0            0.0        30427.0           24.0   \n",
       "24  ...           24.0            0.0        30427.0           24.0   \n",
       "25  ...           24.0            0.0        30427.0           24.0   \n",
       "26  ...           24.0            0.0        30427.0           24.0   \n",
       "27  ...           23.0            0.0        30427.0           23.0   \n",
       "28  ...           24.0            0.0        30427.0           24.0   \n",
       "29  ...           23.0            0.0        30427.0           23.0   \n",
       "30  ...           24.0            0.0        30427.0           24.0   \n",
       "31  ...           23.0            0.0        30427.0           23.0   \n",
       "\n",
       "    smart_197_raw  smart_198_raw  smart_199_raw  smart_240_raw  smart_241_raw  \\\n",
       "0             0.0            0.0            0.0        10389.0   3.606082e+10   \n",
       "1             0.0            0.0            0.0        10413.0   3.609658e+10   \n",
       "2             0.0            0.0            0.0        10437.0   3.614011e+10   \n",
       "3             0.0            0.0            0.0        10461.0   3.617421e+10   \n",
       "4             0.0            0.0            0.0        10485.0   3.621012e+10   \n",
       "5             0.0            0.0            0.0        10509.0   3.624462e+10   \n",
       "6             0.0            0.0            0.0        10533.0   3.628199e+10   \n",
       "7             0.0            0.0            0.0        10557.0   3.631089e+10   \n",
       "8             0.0            0.0            0.0        10581.0   3.633408e+10   \n",
       "9             0.0            0.0            0.0        10605.0   3.638426e+10   \n",
       "10            0.0            0.0            0.0        10629.0   3.641227e+10   \n",
       "11            0.0            0.0            0.0        10653.0   3.645253e+10   \n",
       "12            0.0            0.0            0.0        10677.0   3.649212e+10   \n",
       "13            0.0            0.0            0.0        10701.0   3.654030e+10   \n",
       "14            0.0            0.0            0.0        10725.0   3.660039e+10   \n",
       "15            0.0            0.0            0.0        10749.0   3.663817e+10   \n",
       "16            0.0            0.0            0.0        13582.0   3.857965e+10   \n",
       "17            0.0            0.0            0.0        13606.0   3.861341e+10   \n",
       "18            0.0            0.0            0.0        13630.0   3.864594e+10   \n",
       "19            0.0            0.0            0.0        13654.0   3.867881e+10   \n",
       "20            0.0            0.0            0.0        13678.0   3.873947e+10   \n",
       "21            0.0            0.0            0.0        13702.0   3.879835e+10   \n",
       "22            0.0            0.0            0.0        13726.0   3.883539e+10   \n",
       "23            0.0            0.0            0.0        13750.0   3.885619e+10   \n",
       "24            0.0            0.0            0.0        13774.0   3.888197e+10   \n",
       "25            0.0            0.0            0.0        13797.0   3.891351e+10   \n",
       "26            0.0            0.0            0.0        13822.0   3.894760e+10   \n",
       "27            0.0            0.0            0.0        13846.0   3.897927e+10   \n",
       "28            0.0            0.0            0.0        13870.0   3.900667e+10   \n",
       "29            0.0            0.0            0.0        13893.0   3.903526e+10   \n",
       "30            0.0            0.0            0.0        13918.0   3.906552e+10   \n",
       "31            0.0            0.0            0.0        13942.0   3.909157e+10   \n",
       "\n",
       "    smart_242_raw  \n",
       "0    8.751856e+09  \n",
       "1    8.776299e+09  \n",
       "2    8.817362e+09  \n",
       "3    8.851069e+09  \n",
       "4    8.869208e+09  \n",
       "5    8.878416e+09  \n",
       "6    8.901264e+09  \n",
       "7    8.910142e+09  \n",
       "8    8.916714e+09  \n",
       "9    8.928378e+09  \n",
       "10   8.952648e+09  \n",
       "11   8.961123e+09  \n",
       "12   8.984780e+09  \n",
       "13   9.006662e+09  \n",
       "14   9.017427e+09  \n",
       "15   9.024461e+09  \n",
       "16   2.475205e+10  \n",
       "17   2.477526e+10  \n",
       "18   2.481269e+10  \n",
       "19   2.483501e+10  \n",
       "20   2.487543e+10  \n",
       "21   2.491775e+10  \n",
       "22   2.493543e+10  \n",
       "23   2.495545e+10  \n",
       "24   2.497250e+10  \n",
       "25   2.500865e+10  \n",
       "26   2.502570e+10  \n",
       "27   2.506768e+10  \n",
       "28   2.508617e+10  \n",
       "29   2.510276e+10  \n",
       "30   2.512448e+10  \n",
       "31   2.514321e+10  \n",
       "\n",
       "[32 rows x 21 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head(2*sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='e2'></a>\n",
    "### Exercise 2: Filtering the labels\n",
    "\n",
    "As for class labels, there is one extra filtering step. Take some time and try to figure the purpose behind the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_labels_seq = y_labels[sequence_length-1::sequence_length]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To label a particular sequence as normal or failed, we will use the 'failure' column from the last day of the sequence.  The above code finds the last day failure label to label the entire sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 - LSTM based autoencoder or seq2seq model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"2\"></a>\n",
    "## Defining the LSTM Autoencoder Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we are going to define two autoencoder models. The first model uses LSTMs as autoencoders. In this model, we create the autoencoder for time sequences using LSTM encoder-decoder models. \n",
    "\n",
    "An LSTM unit could be thought of an encoder unit where the input data are represented as a vector of a hidden states. We use this concept to encode the input into 1024 __bottleneck__ vectors. Afterwards we use another set of two LSTM units to reconstruct the original series.\n",
    "\n",
    "For the second model, we use a convolutional autoencoder. In this model, in order to encode the information, we use down-sampling (MaxPooling1D) and up-sampling (UpSampling1D) methods.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 16, 21)]          0         \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 128)               76800     \n",
      "_________________________________________________________________\n",
      "repeat_vector (RepeatVector) (None, 16, 128)           0         \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 16, 128)           131584    \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 16, 21)            12600     \n",
      "=================================================================\n",
      "Total params: 220,984\n",
      "Trainable params: 220,984\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Model #1 - Wide LSTM-based Encoder-Decoder\n",
    "data_shape = x_train.shape[1:]   \n",
    "    \n",
    "## Some parameters to change\n",
    "dp_lvl = 0.2\n",
    "lstm_width = 128\n",
    "\n",
    "inputs = Input(shape=(data_shape))\n",
    "encoded = LSTM(lstm_width, dropout = dp_lvl, recurrent_dropout = dp_lvl, return_sequences =  False, activation='relu')(inputs)\n",
    "decoded = RepeatVector(data_shape[0])(encoded)\n",
    "decoded = LSTM(lstm_width, return_sequences=True, activation='relu')(decoded)\n",
    "decoded = LSTM(data_shape[1], activation='tanh', return_sequences=True)(decoded)\n",
    "\n",
    "# Build entire model\n",
    "autoencoder = Model(inputs, decoded)\n",
    "\n",
    "# Parameters to control learning rate\n",
    "learning_rate = 0.001\n",
    "\n",
    "optimizer = Adam(lr=learning_rate)\n",
    "autoencoder.compile(optimizer=optimizer, loss='mae')\n",
    "    \n",
    "print(autoencoder.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We setup callback routines to two callback routines\n",
    "<ul>\n",
    "    <li>ModelCheckpoint - stores a copy of our model weights for our best training epoch.  This can be used for later deployments.</li>\n",
    "    <li>EarlyStopping - stops the training early if the model is no longer improving.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "\n",
    "# checkpoint\n",
    "checkpoint = ModelCheckpoint(data_dir+\"best_model_ae-lstm.h5\", monitor='val_loss', \n",
    "                             verbose=1, save_best_only=True, mode='min')\n",
    "es = EarlyStopping(patience=20, verbose=1)\n",
    "\n",
    "callbacks_list = [es, checkpoint]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"3\"></a>\n",
    "## Training the LSTM model\n",
    "Now that we've created our model, we'll go ahead and train it.  Then we'll evaluate the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4162 samples, validate on 463 samples\n",
      "Epoch 1/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.3901\n",
      "Epoch 00001: val_loss improved from inf to 0.29392, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 6s 2ms/sample - loss: 0.3871 - val_loss: 0.2939\n",
      "Epoch 2/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2829\n",
      "Epoch 00002: val_loss improved from 0.29392 to 0.24939, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2811 - val_loss: 0.2494\n",
      "Epoch 3/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2515\n",
      "Epoch 00003: val_loss improved from 0.24939 to 0.22464, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2503 - val_loss: 0.2246\n",
      "Epoch 4/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2347\n",
      "Epoch 00004: val_loss improved from 0.22464 to 0.21053, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2335 - val_loss: 0.2105\n",
      "Epoch 5/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2250\n",
      "Epoch 00005: val_loss improved from 0.21053 to 0.20431, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 191us/sample - loss: 0.2241 - val_loss: 0.2043\n",
      "Epoch 6/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2174\n",
      "Epoch 00006: val_loss improved from 0.20431 to 0.19901, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 189us/sample - loss: 0.2184 - val_loss: 0.1990\n",
      "Epoch 7/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2163\n",
      "Epoch 00007: val_loss improved from 0.19901 to 0.19530, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.2152 - val_loss: 0.1953\n",
      "Epoch 8/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2116\n",
      "Epoch 00008: val_loss improved from 0.19530 to 0.19511, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.2123 - val_loss: 0.1951\n",
      "Epoch 9/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2089\n",
      "Epoch 00009: val_loss improved from 0.19511 to 0.18861, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2078 - val_loss: 0.1886\n",
      "Epoch 10/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2057\n",
      "Epoch 00010: val_loss did not improve from 0.18861\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.2051 - val_loss: 0.1892\n",
      "Epoch 11/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2056\n",
      "Epoch 00011: val_loss improved from 0.18861 to 0.18574, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2051 - val_loss: 0.1857\n",
      "Epoch 12/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2029\n",
      "Epoch 00012: val_loss improved from 0.18574 to 0.18551, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 189us/sample - loss: 0.2033 - val_loss: 0.1855\n",
      "Epoch 13/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2023\n",
      "Epoch 00013: val_loss improved from 0.18551 to 0.18429, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.2016 - val_loss: 0.1843\n",
      "Epoch 14/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2006\n",
      "Epoch 00014: val_loss did not improve from 0.18429\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.2006 - val_loss: 0.1848\n",
      "Epoch 15/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1989\n",
      "Epoch 00015: val_loss did not improve from 0.18429\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1998 - val_loss: 0.1853\n",
      "Epoch 16/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.2020\n",
      "Epoch 00016: val_loss improved from 0.18429 to 0.18407, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.2014 - val_loss: 0.1841\n",
      "Epoch 17/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1973\n",
      "Epoch 00017: val_loss improved from 0.18407 to 0.18170, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 185us/sample - loss: 0.1976 - val_loss: 0.1817\n",
      "Epoch 18/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1972\n",
      "Epoch 00018: val_loss improved from 0.18170 to 0.18119, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.1966 - val_loss: 0.1812\n",
      "Epoch 19/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1954\n",
      "Epoch 00019: val_loss improved from 0.18119 to 0.17985, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.1962 - val_loss: 0.1798\n",
      "Epoch 20/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1950\n",
      "Epoch 00020: val_loss did not improve from 0.17985\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1952 - val_loss: 0.1816\n",
      "Epoch 21/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1931\n",
      "Epoch 00021: val_loss improved from 0.17985 to 0.17920, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.1939 - val_loss: 0.1792\n",
      "Epoch 22/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1924\n",
      "Epoch 00022: val_loss improved from 0.17920 to 0.17629, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.1922 - val_loss: 0.1763\n",
      "Epoch 23/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1924\n",
      "Epoch 00023: val_loss did not improve from 0.17629\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1922 - val_loss: 0.1784\n",
      "Epoch 24/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1899\n",
      "Epoch 00024: val_loss did not improve from 0.17629\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1902 - val_loss: 0.1773\n",
      "Epoch 25/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1899\n",
      "Epoch 00025: val_loss did not improve from 0.17629\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1893 - val_loss: 0.1769\n",
      "Epoch 26/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1907\n",
      "Epoch 00026: val_loss improved from 0.17629 to 0.17575, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.1896 - val_loss: 0.1757\n",
      "Epoch 27/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1885\n",
      "Epoch 00027: val_loss did not improve from 0.17575\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1884 - val_loss: 0.1761\n",
      "Epoch 28/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1864\n",
      "Epoch 00028: val_loss improved from 0.17575 to 0.17369, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.1874 - val_loss: 0.1737\n",
      "Epoch 29/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1876\n",
      "Epoch 00029: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 179us/sample - loss: 0.1879 - val_loss: 0.1763\n",
      "Epoch 30/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1869\n",
      "Epoch 00030: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1872 - val_loss: 0.1760\n",
      "Epoch 31/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1897\n",
      "Epoch 00031: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1891 - val_loss: 0.1779\n",
      "Epoch 32/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1881\n",
      "Epoch 00032: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 182us/sample - loss: 0.1891 - val_loss: 0.1755\n",
      "Epoch 33/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1886\n",
      "Epoch 00033: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1879 - val_loss: 0.1750\n",
      "Epoch 34/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1876\n",
      "Epoch 00034: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 184us/sample - loss: 0.1869 - val_loss: 0.1753\n",
      "Epoch 35/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1850\n",
      "Epoch 00035: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 183us/sample - loss: 0.1858 - val_loss: 0.1748\n",
      "Epoch 36/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1855\n",
      "Epoch 00036: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 182us/sample - loss: 0.1858 - val_loss: 0.1760\n",
      "Epoch 37/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1878\n",
      "Epoch 00037: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1873 - val_loss: 0.1754\n",
      "Epoch 38/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1874\n",
      "Epoch 00038: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1869 - val_loss: 0.1763\n",
      "Epoch 39/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1871\n",
      "Epoch 00039: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 183us/sample - loss: 0.1865 - val_loss: 0.1777\n",
      "Epoch 40/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1888\n",
      "Epoch 00040: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1885 - val_loss: 0.1818\n",
      "Epoch 41/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1913\n",
      "Epoch 00041: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 182us/sample - loss: 0.1909 - val_loss: 0.1802\n",
      "Epoch 42/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1873\n",
      "Epoch 00042: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 183us/sample - loss: 0.1875 - val_loss: 0.1762\n",
      "Epoch 43/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1854\n",
      "Epoch 00043: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 183us/sample - loss: 0.1854 - val_loss: 0.1746\n",
      "Epoch 44/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1856\n",
      "Epoch 00044: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1850 - val_loss: 0.1755\n",
      "Epoch 45/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1841\n",
      "Epoch 00045: val_loss did not improve from 0.17369\n",
      "4162/4162 [==============================] - 1s 182us/sample - loss: 0.1835 - val_loss: 0.1740\n",
      "Epoch 46/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1832\n",
      "Epoch 00046: val_loss improved from 0.17369 to 0.17354, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 186us/sample - loss: 0.1835 - val_loss: 0.1735\n",
      "Epoch 47/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1841\n",
      "Epoch 00047: val_loss improved from 0.17354 to 0.17249, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 189us/sample - loss: 0.1826 - val_loss: 0.1725\n",
      "Epoch 48/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1830\n",
      "Epoch 00048: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1836 - val_loss: 0.1755\n",
      "Epoch 49/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1848\n",
      "Epoch 00049: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1854 - val_loss: 0.1752\n",
      "Epoch 50/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1842\n",
      "Epoch 00050: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1832 - val_loss: 0.1741\n",
      "Epoch 51/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1816\n",
      "Epoch 00051: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1821 - val_loss: 0.1732\n",
      "Epoch 52/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1822\n",
      "Epoch 00052: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1821 - val_loss: 0.1744\n",
      "Epoch 53/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1812\n",
      "Epoch 00053: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1820 - val_loss: 0.1742\n",
      "Epoch 54/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1816\n",
      "Epoch 00054: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1810 - val_loss: 0.1741\n",
      "Epoch 55/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1796\n",
      "Epoch 00055: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1809 - val_loss: 0.1731\n",
      "Epoch 56/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1805\n",
      "Epoch 00056: val_loss did not improve from 0.17249\n",
      "4162/4162 [==============================] - 1s 180us/sample - loss: 0.1808 - val_loss: 0.1743\n",
      "Epoch 57/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1806\n",
      "Epoch 00057: val_loss improved from 0.17249 to 0.17237, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.1807 - val_loss: 0.1724\n",
      "Epoch 58/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1799\n",
      "Epoch 00058: val_loss improved from 0.17237 to 0.17195, saving model to ./data/best_model_ae-lstm.h5\n",
      "4162/4162 [==============================] - 1s 187us/sample - loss: 0.1808 - val_loss: 0.1720\n",
      "Epoch 59/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1821\n",
      "Epoch 00059: val_loss did not improve from 0.17195\n",
      "4162/4162 [==============================] - 1s 182us/sample - loss: 0.1811 - val_loss: 0.1736\n",
      "Epoch 60/60\n",
      "3968/4162 [===========================>..] - ETA: 0s - loss: 0.1817\n",
      "Epoch 00060: val_loss did not improve from 0.17195\n",
      "4162/4162 [==============================] - 1s 181us/sample - loss: 0.1828 - val_loss: 0.1749\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "\n",
    "# number of training epochs - feel free to change\n",
    "#\n",
    "# Time: 60 epochs ~= 3 minutes for LSTM\n",
    "\n",
    "epochs_num = 60\n",
    "history = autoencoder.fit(x_train, x_train, batch_size=128, epochs=epochs_num, validation_split=0.1, verbose=1, callbacks=callbacks_list) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the loss function to see how accurate our training was.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./data/best_model_ae-lstm.h5\n"
     ]
    }
   ],
   "source": [
    "# load weights from best model\n",
    "print(data_dir+\"best_model_ae-lstm.h5\")\n",
    "autoencoder.load_weights(data_dir+\"best_model_ae-lstm.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV1b3//9cn8wgCYZIpYTAMMogoAg6gMjmgXCvO2qpVq962v9a26rfW2vbe2/a2VmttvdZaqxUUtSoKioogVkUGmQeZMZEpzCQkkGH9/lg75CScQAI5OQl5P9v9ODn77L3PWsGc99lr7b2WOecQERGpKibaBRARkYZJASEiImEpIEREJCwFhIiIhKWAEBGRsBQQIiISlgJCRCLGzJ4zs19FuxxyfBQQctzMbKOZXVzNaw+a2QYzyzezXDN7OVi/PFiXb2alZlYU8vxBM/ummTkze7TK8a4M1j9XzfsNN7PcOq+kSBOmgJA6Z2a3ADcBFzvn0oBBwAwA51wf51xasP5j4N7y5865/w4OsQ64xsziQg57M7C6/mpRv6rUVaRBUEBIJJwFTHfOrQNwzm11zj1di/23AkuB0QBm1hIYCkw5nsKYWS8zm2Vme4IzmHEhr11iZivMbL+ZfW1m9wXrM8zs7WCfXWb2sZmF/Xsxs8fNLMfM9pnZAjM7L+S12ODMaF3wHgvMrFPwmjOze8xsDbAmWPdtM1sbvOcUMzs1WG9m9gcz225me81siZmdfrQ6VFPWW81spZntNrPpZtYl5DVnZt81s/VmtsPM/re8zmYWY2Y/NbNNQRmeN7PmIfuea2afBr+vHDP7ZsjbtjCzqUH5PjezbseqkzQMCgiJhDnAzWb2IzMbZGaxx3GM5/FnDQDXAm8CB2t7EDOLB94C3gPaAP8JvGhm2cEmfwPudM6lA6cDHwbrfwjkAq2BtsCDQHXj0swDBgAtgYnAK2aWFLz2A+A64BKgGXArcCBk3yuBwUBvM7sQ+B9gAtAe2AS8FGw3CjgfOA04BbgG2HmMOlT9XVwZ1OM/gnp9DEyqstl4/BnfQOCKoLwA3wyWEUBXIA34U3DczsA7wBPBcQcAi0KOeR3wCNACWAv8Vw3qJA2Bc06LluNagI34ZqRwr90AfAAU4P/o7w+zzSzg9irrvgn8G0gGtgHN8YEzDPgV8Fw17zccyA2z/jz8GUlMyLpJwM+Dn78C7gSaVdnvF/hQ6n4cv5fdQP/g5y+BK6rZzgEXhjz/G/DbkOdpQDGQCVyIb2I7J7QuR6tDmPd7B7gt5HkMPqy6hJRnTMjrdwMzgp9nAHeHvJYdlC0OeAB4vZr3fA54JuT5JcCq4Odq66SlYSw6g5CIcM696Jy7GP/N8C7gF2Y2uhb7FwJTgZ8CGc65T46zKKcCOc65spB1m4AOwc9X4T+0NpnZR2Y2JFj/v/hvu+8FTS73V/cGZvbDoNlmr5ntwYdaRvByJ3yfSnVyqpR1U/kT51w+Plw7OOc+xH9jfxLYZmZPm1mzY9Shqi7A40Ez0B5gF2Ahv4uq5dkUlOmIsgU/x+HPro5Vx60hPx/ABx/HqJM0AAoIiSjnXLFz7hVgCb75ozaexzf1vHACRdgMdKrSf9AZ+Doo3zzn3BX45qc3gMnB+v3OuR8657oClwM/MLOLqh486G/4Cb5ZqIVz7hRgL/6DF/wHbrejlC+02Woz/kO8/NipQKuQsv7ROXcm0AffLPOjo9UhjBx8U9QpIUuyc+7TkG06Vfk9bQ5XtuC1EvxZ3rHqWK3q6iQNgwJCTlS8mSWFLHHmL1W91MzSg87NsfgPgM9reeyPgJH4tu0aqVKWJGAuvpnrx2YWb2bD8R/4L5lZgpndYGbNnXPFwD6gNDjOZWbW3cwsZH1pmLdMx39Q5gFxZvYzfF9DuWeAX5pZj6BTtp+Ztaqm+BOBb5nZADNLBP4b+Nw5t9HMzjKzwUGfSgFQBJQerQ5hPAU8YGZ9gjo2N7Orq2zzIzNrYb4j/XvAy8H6ScD/Z2ZZZpYWlO1l51wJ8CJwsZlNCP79W5nZgGrKcFh1dTrWflJ/FBByoqYBhSHLz/EfUg/i28b3AL8FvuOc+3dtDuy8Gc65XTXcpUOVshTivxGPA8YCO4A/Azc751YF+9wEbDSzffimsBuD9T3wfSj5wGfAn51zs8K853R82/5qfLNLEZWbaR7Ff6N/D/97+Ru+fyVcfWcADwGvAVvw38qvDV5uBvwV37+xCd/09Ltj1KHq8V8HfoMPx33AsuD3EupNYAG+k3lqUF6AZ/FncrOBDUE9/zM47lf4Jq4f4putFgH9w5WhiqPVSRoAc04TBomIv8wV6OGcWxvtskjDoDMIEREJSwEhIiJhqYlJRETC0hmEiIiEdVINEJaRkeEyMzOjXQwRkUZjwYIFO5xzrcO9dlIFRGZmJvPnz492MUREGg0z21Tda2piEhGRsBQQIiISlgJCRETCUkCIiEhYCggREQlLASEiImEpIEREJCwFBPDLj37J9LXTo10MEZEGRQEB/PbT3/Lu2nejXQwRkQZFAQE0S2zG/kP7o10MEZEGRQEBpCeks+/gvmgXQ0SkQVFA4M8gFBAiIpUpIFATk4hIOAoIID1RTUwiIlUpIAjOIA7qDEJEJJQCAnVSi4iEo4Cgog9C83OLiFRQQODPIErKSigqKYp2UUREGgwFBP4MAtCVTCIiIRQQVASE+iFERCooIPCXuYICQkQklAKCkCYmXeoqInKYAgLfSQ06gxARCaWAQJ3UIiLhKCBQH4SISDgKCNQHISISjgICSI1PxTCdQYiIhFBAAGZGemK6+iBEREIoIAKaNEhEpDIFREAjuoqIVKaACGhWORGRyhQQAc0qJyJSmQIioFnlREQqU0AE1AchIlKZAiKgPggRkcoUEIHyMwhNOyoi4ikgAs0Sm1HmyigsKYx2UUREGgQFRECzyomIVKaACGhEVxGRyiIaEGY2xsy+NLO1ZnZ/mNevMLMlZrbIzOab2bkhr200s6Xlr0WynKARXUVEqoqL1IHNLBZ4EhgJ5ALzzGyKc25FyGYzgCnOOWdm/YDJQM+Q10c453ZEqoyhNKuciEhlkTyDOBtY65xb75w7BLwEXBG6gXMu31VcNpQKRO0SIs0qJyJSWSQDogOQE/I8N1hXiZmNN7NVwFTg1pCXHPCemS0wszuqexMzuyNonpqfl5d33IVVH4SISGWRDAgLs+6IMwTn3OvOuZ7AlcAvQ14a5pwbCIwF7jGz88O9iXPuaefcIOfcoNatWx93YdUHISJSWSQDIhfoFPK8I7C5uo2dc7OBbmaWETzfHDxuB17HN1lFjC5zFRGpLJIBMQ/oYWZZZpYAXAtMCd3AzLqbmQU/DwQSgJ1mlmpm6cH6VGAUsCyCZSU5LpkYi1EfhIhIIGJXMTnnSszsXmA6EAs865xbbmZ3Ba8/BVwF3GxmxUAhcE1wRVNb4PUgO+KAic65dyNVVvDTjmpWORGRChELCADn3DRgWpV1T4X8/BvgN2H2Ww/0j2TZwtGIriIiFXQndQiN6CoiUkEBEUKzyomIVFBAhNCsciIiFRQQIdQHISJSQQERQn0QIiIVFBAhdJmriEgFBUSI9IR09h/cr2lHRURQQFTSLLEZDkdBcUG0iyIiEnUKiBAa0VVEpIICIoRGdBURqaCACKFZ5UREKiggQmhWORGRCgqIEJoTQkSkggIiRHkntfogREQUEJXoDEJEpIICIoQ6qUVEKiggQiTFJREXE6dOahERFBCVmJlGdBURCSggqtCIriIingKiCs0qJyLiKSCq0KxyIiKeAqIKzQkhIuIpIKpIT0hXH4SICAqII+gMQkTEU0BUoctcRUQ8BUQVzRKbkX8onzJXFu2iiIhElQKiivIB+/IP5Ue5JCIi0aWAqEKzyomIeAqIKjRgn4iIp4CoQrPKiYh4CogqNCeEiIingKhCs8qJiHgKiCp0BiEi4ikgqlAntYiIp4CoQp3UIiKeAqKKxLhE4mPidQYhIk2eAiIMzQkhIhLhgDCzMWb2pZmtNbP7w7x+hZktMbNFZjbfzM6t6b6R1CyxGfsO6QxCRJq2iAWEmcUCTwJjgd7AdWbWu8pmM4D+zrkBwK3AM7XYN2LSE9N1BiEiTV4kzyDOBtY659Y75w4BLwFXhG7gnMt3zrngaSrgarpvJGlOCBGRyAZEByAn5HlusK4SMxtvZquAqfiziBrvG+x/R9A8NT8vL69OCq5Z5UREIhsQFmadO2KFc68753oCVwK/rM2+wf5PO+cGOecGtW7d+rgLG0pnECIikQ2IXKBTyPOOwObqNnbOzQa6mVlGbfeta5pVTkQksgExD+hhZllmlgBcC0wJ3cDMupuZBT8PBBKAnTXZN5J0mauICMRF6sDOuRIzuxeYDsQCzzrnlpvZXcHrTwFXATebWTFQCFwTdFqH3TdSZa0qPTGdguICSstKiY2Jra+3FRFpUCIWEADOuWnAtCrrngr5+TfAb2q6b0SUlUHfvnDjjfDAA0DFcBv5h/JpntQ84kUQEWmIdCd1TAzs2werVh1epRFdRUQUEF5WFmzcePhp+YiuutRVRJoyBQRAZmalgNAZhIiIAsLLzITcXCguBjSrnIgIKCC8zEzfWZ3jb97WGYSISA0DwsxSzSwm+Pk0MxtnZvGRLVo9ysryj0Ezk2aVExGp+RnEbCDJzDrgR2D9FvBcpApV7zIz/WMQEJpVTkSk5gFhzrkDwH8ATzjnxuOH4T45dOzoL3fdsAGo6IPQGYSINGU1DggzGwLcgB91FSJ8k129io/3IRGcQSTEJpAYm6hOahFp0moaEN8HHgBeD4bL6ArMjFyxoqDKvRAa0VVEmroanQU45z4CPgIIOqt3OOe+G8mC1bvMTJgx4/DT9ETNCSEiTVtNr2KaaGbNzCwVWAF8aWY/imzR6llmJnz9NRw8CEDzxObsKtwV3TKJiERRTZuYejvn9uEn9ZkGdAZuilipoiErC5w7fC/Eaa1OY+WOlVEulIhI9NQ0IOKD+x6uBN50zhVTzQxvjVaVS137te3Hxj0b2Vu0N2pFEhGJppoGxP8BG4FUYLaZdQFOrh7c8oAILnXt37Y/AEu3L41SgUREoqtGAeGc+6NzroNz7hLnbQJGRLhs9atDB4iNPXwG0b+dD4jFWxdHsVAiItFT007q5mb2qJnND5bf488mTh5xcdC58+GA6JDegRZJLVi8TQEhIk1TTZuYngX2AxOCZR/w90gVKmpChv02M/q368+SbUuiWiQRkWipaUB0c8497JxbHyyPAF0jWbCoyMw83AcBvh9i6fallJaVRq9MIiJRUtOAKDSzc8ufmNkwoDAyRYqizEzYsgWKigB/JdOB4gOs270uuuUSEYmCmgbEXcCTZrbRzDYCfwLujFipoqV82O+vvgIqrmRSR7WINEU1vYppsXOuP9AP6OecOwO4MKIli4Yq90L0adOHWItVP4SINEm1mlHOObcvuKMa4AcRKE90VbkXIikuieyMbF3JJCJN0olMOWp1VoqG4tRT/dDfIaO69mvbTwEhIk3SiQTEyTXUBvgb5ULuhQDfD/HV3q/YU7QneuUSEYmCowaEme03s31hlv3AqfVUxvoV5lJXQP0QItLkHDUgnHPpzrlmYZZ059zJM6NcqJCb5cA3MYGuZBKRpudEmphOTllZsG0bFPrbPE5NP5VWya3UDyEiTY4CoqryK5k2bQI05IaINF0KiKqqXOoKvh9i2fZlGnJDRJoUBURVVW6WA98PUVhSyJpda6JSJBGRaFBAVNW+PSQkHHGpK6ijWkSaFgVEVTEx0KVLpSam3q17a8gNEWlyFBDhVLnUNTEukZ4ZPXUlk4g0KQqIcLKyKgUE+ClIFRAi0pQoIMLJzIS8PCgoOLyqf9v+5O7LZVfhruiVS0SkHikgwqnmSibQkBsi0nRENCDMbIyZfWlma83s/jCv32BmS4LlUzPrH/LaRjNbamaLzGx+JMt5hDABoSuZRKSpidh4SmYWCzwJjARygXlmNsU5tyJksw3ABc653WY2FngaGBzy+gjn3I5IlbFa5TPLhQREu7R2tE5prX4IEWkyInkGcTaw1jm33jl3CHgJuCJ0A+fcp8653cHTOUDHCJan5tq2haSkSgFRPuTGoq2LolcuEZF6FMmA6ADkhDzPDdZV5zbgnZDnDnjPzBaY2R3V7WRmd5jZfDObn5eXd0IFDjmovxdi3bpKq8/vfD4Lty7k631f1837iIg0YJEMiHAzzoWdZMjMRuAD4ichq4c55wYCY4F7zOz8cPs65552zg1yzg1q3br1iZa5Qv/+ML9y18fVfa4G4NUVr9bd+4iINFCRDIhcoFPI847A5qobmVk/4BngCufczvL1zrnNweN24HV8k1X9GTYMcnL8EuiZ0ZN+bfvxyopX6rUoIiLREMmAmAf0MLMsM0sArgWmhG5gZp2BfwE3OedWh6xPNbP08p+BUcCyCJb1SEOH+sfPPqu0+ureV/NJzifk7sut1+KIiNS3iAWEc64EuBeYDqwEJjvnlpvZXWZ2V7DZz4BWwJ+rXM7aFvi3mS0G5gJTnXPvRqqsYfXvD8nJ8OmnlVZf3ds3M7224rV6LY6ISH0z58J2CzRKgwYNcvPn1+EtE8OHw4EDMHdupdX9n+pPWkIan9z6Sd29l4hIFJjZAufcoHCv6U7qoxk6FBYu9CERYkLvCXya86mamUTkpKaAOJqhQ6GkRFcziUiTpIA4mnPO8Y9V+iFOa3Ua/dv219VMInJSU0AcTUYGZGcfERAAE/r4ZqacvTlhdhQRafwUEMcydKgPiCqd+eVXM6mZSUROVgqIYxk2DHbuhNWrK63u0aoHA9oNUDOTiJy0FBDHUn7DXLhmpt4T+Cz3MzUzichJSQFxLNnZ0KJF2IDQ1UwicjJTQBxLTAwMGRI2ILq37M4Z7c5g8orJUSiYiEhkKSBqYuhQWLECdu8+4qXr+17PnNw5fLzp4ygUTEQkchQQNVHeDzFnzhEv3X3W3XRu3pl7pt1DSVlJPRdMRCRyFBA1cdZZEBsbtpkpJT6FP4z+A0u3L+Uv8/4ShcKJiESGAqIm0tL86K5hAgJgfM/xjOo2iodmPsS2/G31XDgRkchQQNTU0KHw+ed+bKYqzIw/jvkjB4oP8MCMB6JQOBGRuqeAqKmhQ6GgAJYuDftydkY2PxjyA/6+6O98lvNZ2G1ERBoTBURNDRvmH6tpZgL46fk/pUN6B+59515Ky0rrqWAiIpGhgKipTp2gQ4ejBkRaQhq/H/V7vtjyBX/94q/1WDgRkbqngKgpM38WMWNG2H6IchP6TGBE5ggenPEgX+39qh4LKCJStxQQtXH99bBtG0ydWu0mZsafL/0zZa6MEf8YoVnnRKTRUkDUxqWXQvv28NejNx/1zOjJ9Bunk1eQx4X/uJDN+zfXUwFFROqOAqI24uLgttvgnXfgq6M3Hw3uOJh3bniHzfs3c9HzF+n+CBFpdBQQtXXbbX7yoGefPeamwzoPY9oN0/hq71dc+PyF5BXk1UMBRUTqhgKitjIzYfRo+NvfjtpZXe78Lufz1nVvsX73ei56/iJ1XItIo6GAOB533AG5ufDuuzXa/MKsC5ly7RTW7lpL9p+yeejDh8g/lB/hQoqInBgFxPG47DJo1w6efrrGu4zsNpKV96zkyp5X8quPf0X2n7J5fvHzlLmyCBZUROT4KSCOR3w83Hqrv9w1t+aXsXY5pQuTrprEJ7d+Qof0Dtzyxi0MfmYwMzfMjGBhRUSOjwLieN1+O5SV1aizuqqhnYYy5/Y5vDD+Bbbs38KFz1/IiH+MYPam2REoqIjI8VFAHK+sLBg1Cp55BkprP+5SjMVwY78bWfOfa3hs9GOs2rGKC567gIufv5hPvvokAgUWEakdBcSJ+Pa3IScHpk8/7kMkxyfzvXO+x7rvruP3o37P0u1LOffv53Lda9dRcKigDgsrIlI7CogTMW4ctGlTq87q6qTEp/CDIT9g/XfX88jwR5i8fDJD/jaE9bvX10FBRURqTwFxIhIS/I1zU6bAG2/UySFTE1L52QU/450b3iF3Xy6Dnh7E++ver5Nji4jUhgLiRP2//wdnnw3XXQf//nedHXZUt1HM+/Y8OjTrwJgXx/C7T3+Hc67Oji8iciwKiBOVmgpvvw1dusDll8OyZXV26G4tu/HZbZ8xvud4fvT+j7jipStYtHVRnR1fRORoFBB1ISPDd1QnJ8OYMcccyK820hLSeOXqV/jtxb9l1sZZnPF/Z3DZxMs0ramIRJydTM0WgwYNcvPnz49eAZYsgfPPh1NP9c1NLVvW6eF3F+7myXlP8ticx9hZuJMRmSO4feDtpMSnYBhmhmGkJaQxpNMQkuKS6vT9ReTkY2YLnHODwr6mgKhjH33kB/MbOBDeew/S0ur8LfIP5fPXBX/ld5/9rtq5JtIS0ri0x6X8R6//YGz3saQnptd5OUSk8VNA1Ld//QsmTICzzoJp06BFi4i8zcGSg6zasYoyV4bD4ZzD4diav5UpX07hjVVvkHcgj8TYREZ2G8m408Zx2WmX0T69fUTKIyKNT9QCwszGAI8DscAzzrlfV3n9BuAnwdN84DvOucU12TecBhMQ4C97veYa6NnTn0m0bVvvRSgtK+XTnE/518p/8fqq19m0dxMAZ516FpefdjmXZ19Ov7b9iDF1RYk0VVEJCDOLBVYDI4FcYB5wnXNuRcg2Q4GVzrndZjYW+LlzbnBN9g2nQQUEwPvvw5VXQocO8MEH0Llz1IrinGPZ9mVM+XIKb61+i7lfz8XhSI1PpU+bPvRt09cvbfvSp3Uf2qS2wcyOOM7aXWuZunoq09ZOo8yV8eioR+nbtm8UaiRSN4qLi8nNzaWoqCjaRYmopKQkOnbsSHx8fKX10QqIIfgP/NHB8wcAnHP/U832LYBlzrkOtd23XIMLCIBPP4VLLoFmzXxInHZatEsEwLb8bby79l0Wbl3I0u1LWbptKXkHKma8a57YnJ4ZPcnOyKZnq55sL9jOtLXTWL1zNQDZrbLZVbiLPUV7ePiCh/nJuT8hLiYuWtUROW4bNmwgPT2dVq1ahf1SdDJwzrFz5072799PVlZWpdeOFhCR/IvuAOSEPM8FBh9l+9uAd2q7r5ndAdwB0DmK39CrNXQozJzpB/Y77zyYPBkuuCDapaJtWltuGXALt3DL4XXb8rexdPtSVuatZNWOVXy580tmrJ/B84ufJzE2keGZw7n3rHu5pMcldGvZjbyCPO59515+OvOnvPHlGzx3xXP0adMnirUSqb2ioiIyMzNP2nAAMDNatWpFXl7tpj2OZECE+22HPV0xsxH4gDi3tvs6554GngZ/BlH7YtaDM86Ajz+GSy+F4cPhnnvg17+OyBVOJ6JtWlvaprXl4q4XV1q//+B+YmNiSYlPqbS+dWprXv7Gy3yj1ze4e9rdDHx6IPcPu5/R3UfTr20/0hIaVv1EqnMyh0O546ljJAMiF+gU8rwjcMQ1mWbWD3gGGOuc21mbfRuVnj39fRIPPghPPOEnG/rb3+DCC6NdsmM61iWyV/e5mgsyL+Ceaffwi9m/4Bezf4FhdG/ZnQHtBtC3TV/apLahRXILWiS1oEVyC1omt6Rz885qlhJpwCL51zkP6GFmWcDXwLXA9aEbmFln4F/ATc651bXZt1FKTYXHH4err/Yz0l10Edx5p7/aKSnJ34ldvnTsCDGN5+qiNqlteOXqV/hq71cs2rqIRVsXsXjbYhZsWcArK14Ju09ibCJ92vShX9t+9GvTj35t+3HmqWdyStIp9Vx6kejZs2cPEydO5O67767VfpdccgkTJ07klFMi9/cS6ctcLwEew1+q+qxz7r/M7C4A59xTZvYMcBWwKdilpLyzJNy+x3q/BtlJXZ0DB+BnP4NHH4Vw/wYXXeTvoUhIqP+y1bHC4kJ2Fe5id9FudhfuZnfRbnYc2MHKvJUs2b6ExVsXs61gGwCG0bt1b4Z1GsbQTkMZ1nkY3Vp0axJNABIdK1eupFevXlF7/40bN3LZZZexrMo4bqWlpcTGxtbpe4Wrq26Ua8jWrfOTDhUVQWGhX1avhkcegZtugn/8A5rAh+O2/G0s2baEOblz+DT3Uz7L+Yy9B/cC0KlZJ8Zlj2Nc9jiGZw4nIbbxh6Y0HKEfmt9/9/t1PiDmgHYDeGzMY9W+fu211/Lmm2+SnZ1NfHw8aWlptG/fnkWLFrFixQquvPJKcnJyKCoq4nvf+x533HEHAJmZmcyfP5/8/HzGjh3Lueeey6effkqHDh148803SU5OPmpdy0XrKiapiW7d/FJVTAw8/LB/7eGH679c9axtWltGpo1kZLeRAJS5MlbmreTfX/2bd9e9y98X/Z0n5z1JekI6Y3uMZUjHIcRaLDEWg5kRYzG0TG7JmO5jaJbYLMq1Eam5X//61yxbtoxFixYxa9YsLr30UpYtW3b4ctRnn32Wli1bUlhYyFlnncVVV11Fq1atKh1jzZo1TJo0ib/+9a9MmDCB1157jRtvvPGEy6aAaKgeegjWr4ef/xy6dvVnE01IjMXQp00f+rTpw52D7qSwuJAZG2bw5qo3eWv1W0xePjnsfklxSVx+2uVc3/d6xnYfS2JcYj2XvOnYeWAnRSVFdGjWIdpFqTNH+6ZfX84+++xK9yr88Y9/5PXXXwcgJyeHNWvWHBEQWVlZDBgwAIAzzzyTjRs31klZFBANlZmfyjQnx89a16mTv0S2iUqOT+ay0y7jstMuo8yVsadoD845ylzZ4bGo1u9ez0vLXuLl5S/zyopXaJ7YnCt6XsHAdgP9DX8ZPencvPMRQ4uUlpVSUFxAekK6+jpqYOm2pTz++eP8c8k/KSkr4buDv8sjwx/RgJB1JDU19fDPs2bN4oMPPuCzzz4jJSWF4cOHh73jOzGx4otQbGwshYWFdVIWBURDlpAAr73mb7YbPx5efhmKiyE3t2JJS/OXzrZvOgPwlTcnVdUurR1DOw3l0dGP8uGGD5m4dCJvrX6L5xc/f3ibpLgkurboSmlZKfsO7mPfwX0UFBcA0DK5JYM7DGZIxyGc0/Eczu5wNs2TmtdJmYtLi5mxYQbT1uKfJWwAABSlSURBVEzjYMlB4mLiiI2JJdZiiY+Np1uLbgxsP5C+bfs2yGHaS8tKmbpmKo9//jgfbviQ5Lhkvjngm5S5Mv4w5w9MXj6Zx8Y8xlW9rlLI1lJ6ejr79+8P+9revXtp0aIFKSkprFq1ijlz5tRr2RQQDd0pp/irmQYP9sOIl4uJ8aGQlwcvvAD//d/+ktk6vuqhMYqLiWNUt1GM6jYK5xw7Duw4fGf4qh2rWLd7HQmxCTRLaEazRL+kxKfw5c4vmZM7h3fXvovDHb6Xo0+bPvRp7ZfT25xOy+SW7CzcSV5BHjsO7GDHgR0UlxXTpXkXurboSlaLLJolNqPMlfFZzmdMXDqRySsms+PADlLjU0lLSKPUlVJaVkpJWQmHSg9xsPTg4bL3bt2bM9ufyRntzmBAuwH0b9c/Kv0qzjkWbV3EpGWTeGnZS+Tsy6Fjs478+qJf8+0zv304pG8941buevsurn7lasZ0H8Ofxv6Jbi3D9KtJWK1atWLYsGGcfvrpJCcn0zZkYM8xY8bw1FNP0a9fP7KzsznnnHPqtWy6iqmxyMmBefP8wH8dO/rRYePiYM0a+M53YMYMPzf2U0/5O7fluO0t2svcr+cyJ3cOi7ctZnnectbsXEOpK63xMVoltyIhNoEt+VtIjktmXPY4ru97PaO7jT6iX8Q5x8Y9G/liyxd+2foFCzYvqDQ2VvlNhwPaDvCP7QZwavqpR/22XubKWLhlIe+vf58P1n/A3K/nkpqQSkZKBq1TWpORkkFGSgYtk1vSMrklLZL8DYzpiel8vOljJi2bxJc7vyQuJo7R3UZzc/+bGd9zPPGx8Ue8V0lZCU/OfZKHZj7EgeIDjO4+mpv63cQV2VeQHH/k1TQNSbQvc61Pusz1ZA2Io3EOJk6EH/wAduyA22/3gwKmpPib81JS/GCBffseX1NUYaEPo/gjPxiaioMlB1m9czXL85azt2gvrVMrPmAzUjKItVg27tnIhj0bWL97Pet3r2fvwb1c2uNSrsi+otbt8845tuRvYdHWRSzcspBF2/zjut3rDm+TkZLBgHYD6HpKV2IsBhcyGs2OAzuYuXEmuwp3AdC3TV/O63wexWXF5B2oOPPJK8hjd9FuylxZpfc3jAsyL+C606/jql5X0Sqlcqdodb7e9zVPzH2CF5e+SO6+XNIT0vlG729w7enXknVKFq1SWnFK0in1NsR8aVkpW/O3HrUjXQGhgGgadu+G++/3Q3iUVvNtt1Mn31w1eLCf0Mg52LatYtm+/cglPx8SE/2ZSfm+gwdDVlaTuEejIdl3cB9Lti3xd6pvXcyibYvYtGfT4TMJC4YxS4lP4fwu5zOy60gu6noR7dLaVXvMMlfG/oP7K93M2DOj5wldnVTmypi1cRYvLHmBV1e8Sv6h/MOvxVgMLZJa0Dq1Ned0PIeRXUdycdeLaZPa5rjfL1RRSREfrP+AN1a9wZQvp5B3II8B7QZw2xm3cX3f64/ov1JAKCCaltJSf6d2QUHF465d8MUX8PnnMGcObNp05H6xsZCR4Zuv2rSpWFq39vt//jksWODPKAC6dPH9Hrff7rcRCeNA8QE+3vQx2wu2s7NwJzsP7GRn4U4279/Mx199fPgsp3/b/ozsOjJsMGWkZNCndR96ZvQ8oskqNDRnb5rNO2vfIf9QPukJ6Vx62qX0a9OPV1a8wsKtC0mMTWR8r/F8a8C3OL/L+STFJSkgFBByhG3bfGAkJlYEQqtWxx7/qbgYli3zIfPqq/Dhh/4Y114L994Lg8L+dyYSVmlZKQu3LuT9de/z3vr3+OSrTyguK652e8Po2qIrvVv3Jj42nsVbF1dqdmuf1p5x2eO4sueVjMgcUam/Z+GWhTy78FleXPoiu4t2ExcTx+ltTueJQU+Q3TOblPiUagePNOzwGNOGERsTG/WZGEvKSth3cB97i/ZSXFbMaa2OPdeMAkIBUb9WrIAnn/RDghQU+H6Orl196LRtC+3awamn+uasDtU0WRQW+k72996D7t3hlluged1cXip1YNcuuO8+f1n1qFEwdiz07h2R5sWDJQcpLKl8DX95f8zy7ctZkbeCFTtWsHz7cg6VHqJ/u/616rgH3wT13rr3mJM7h/mb53Nfj/to2fnIy6aPJSkuieS4ZFLiU0iOTyYxNhEzo/x//v92+G5/w2p9CXD5vT6lzl/xVn7Pzt6iveQfysfhiLVYmic1J+uUrGMeXwGhgIiOvXvh+efhrbdg61a/7NhReSDCrl39pEnnnefPNBYsgClTfDAUFvoRbYuKfMf6TTf5eTNOPz16dRI/j8kNN/h/z+7dYeVKv75jRxgzxp85XnRRdMt4glauXElWjywOFB84orO+XOjnpMNRXFpMYUkhhcWFhy9RronykCjvKzraB3p5OLgwU+EkxyXTPKk5zRObk5aQVuPgUUAoIBqOkhIfEps2+alXP/7YLzt2VGzTuTOMG+eXCy6ApUv9GcmkST4sLrjAX511+eXqEK9PJSXwq1/BL3/pg33SJB/qOTkwfTq8+66fc33fPvj2t+EPf/DB3gidaB9EaVkphSWFHCo9hHMOhzscKI7gQz7kw758m3J79uxhyitTuOn2iuF0yj/wYyyGWIs9fGNlXEwcSbFJJMT5ASsfe+wx7rjjDlJSKk/mVZu6Hi0gfGFPkuXMM8900sCVlTm3YoVzL7zg3KJF/nk4O3Y499vfOpeZ6Rw4N2iQc+++G377deuce/hh58aNc+4nP3Fu0iT/HiUlEa1KrRUUODd7tn9syDZtcu7cc/3v/eabndu3L/x2RUXO3X+/c2bO9ejh3Ny59VvOOrJixYqovv+GDRtcnz59jmvfLl26uLy8vBpvH66uwHxXzWeqziCkYSsp8XeKP/KIPxM57zz/zXbAAHjlFd/38fHH/uyiRw/YsMF3pIOfeKlXL3/vR+vW/gqt1q19h3zfvr75KjHCg/kVFflv2y+/7JvfCgqgRQt/5dc99/grwaJtzx7/O5w1Cz76CBYu9PfO/OUvUJMRQWfNgptvhi1b/OCS99/fqO7or/St+vvfh0V1O9w3AwbAYzUb7nvkyJG0adOGyZMnc/DgQcaPH88jjzxCQUEBEyZMIDc3l9LSUh566CG2bdvGfffdR3Z2NhkZGcycOfOYRdFw33JyiYuDb30Lrr/e39/xq1/5ZqeEBDh0CLKz/TAjN97o7/E4dMi3ky9aBIsX+070zZv9z3l5cDCkvTg+Hvr1800nZ57p+0D27au89OrlP/yaVTPUxZYtfgrZ2bN9IKWl+SU1Ffbvh7ff9sfJyPBlHD7cj6/16KPw+9/DlVfCd78L55xz4mFVWgrLl/srzNau9RcIdOrk+ws6dfJXqW3Y4H8/5cuyZb5Zzzn//uecAz/9qb9QoGvXmr3v8OH+9/ud7/h933zT9yFdeunRj3HwYN0GtHO+Lyw5OfLBX4dCh/t+7733ePXVV5k7dy7OOcaNG8fs2bPJy8vj1FNPZerUqYAfo6l58+Y8+uijzJw5k4yMjIiUTWcQ0rgUFsL//Z8/m7jmGn/DXk37Jpzz3+C3bPEBMn++XxYs8B8soeLj/Qf97t3+8eab/Tf+3r396ytW+A/4f/7Tn7EMGQJlZf74BQX+5kIzuOQS35E7YkTlO9G/+sp/Q3/6aX+VEEB6uj/DKT/bMfPHKV/27/fHCL1CrG1bH4qff+6HYinwAw8SF+fPvo6mSxcfgEOG+NAdPNiH5PFyDl580fdbrA5mEM7O9kFx3nnw9deVw2nLFt/xffHFvqN7xAgfYsdj5kx/5jJ3rn8eH+9/n2lp0LKlP2McONAvAwZUCvyj9kGUlPh/15iYiqWOhc4od9999/Hqq68enkY0Pz+fBx54gPPOO4/Ro0czYcIELrvsMs477zxwzk8aNH06Gc2b12iUBHVSKyCktsrKYONG/9i8uf/wKP8GOn8+/OlP8NJL/hvviBG++WXqVP9h+q1v+U707t2P770PHIA33vDf7PPyKpYdO3xAlJ+RlC+HDlXc9b51q982Ntbf5X7OORVLVpZvOsrJ8Uturt82M9OHXHZ2ZDuV1671g0xOneqboA4d8uvT030o9erlA2rhQv/6/v2+vgMH+rKVNwdmZPglM9M3CVYNsEWLfDBMn+7PlO6803+I79/vl/x8PxrA4sX+TLJc9+7+eL17s3L8eHr17u3/zQ8d8vuUh3zVobXN/PHNjlxiYyvmlE9J8Y+xsf6/m/IvDgUF/pjp6b6p8ZRT2JiTczggfvjDH3Laaadx5513HvEr3bVzJ9Nef52nnn6aUUOG8LNvfpPMSy9l/vPPk1HebKrLXKungJCIycvzTVx//rM/i7n3Xrj77ujfQV7+Dbchz12en++bsTp39vfEVP0QKy72Zz8zZvgbL3NyfEBWPauLjYWePX0YDhjgb/ScONF/0D74oD/DCzPN5mFbt/pA+uIL/7h8OaxZw8q336ZX1SaauDgfoGlp/ueyMr+UlvpH34VfeSkp8f9thJ65xcT47ct/TknxQbRvn693TAw7y8oYOH48m1av5r2ZM3nokUeYMWMGaWlpfL1pE/GFhZTs3k3LmBiSYmN5Y9Ysnps2jTeefZa+l1zClH/9i6zs7BqdSSsgFBASSeV/L7rkNvIOHYKdO304r1njzxYWLvSPX3/tw+D734cf/9gPi3+c77Fy1Sp6dezov9knJPhQSEw8vn9j5/wHf2GhPzssLvblTE31j+XHdM4H565dsHs3199/P0vWrGHs0KF0bNOGZ6ZMASAtOZl//uIXrN28mR/98Y/ExMcTn5jIX556ikGDBvHEE0/w5JNP0r59+4h0UisgRKTxycvz/QzHGwwhoj4WU3nf2KFD/uwjdElM9M2eqal18qVEVzGJyMkv2k17dam8r6kBiu5oUyIi0mApIESkyTuZmtqrczx1VECISJOWlJTEzp07T+qQcM6xc+dOkmp5n4v6IESkSevYsSO5ubnk5eUde+NGLCkpiY4dO9ZqHwWEiDRp8fHxZGVlRbsYDZKamEREJCwFhIiIhKWAEBGRsE6qO6nNLA/YdJy7ZwA7jrlV43Ay1QVUn4bsZKoLnFz1qWldujjnwt55eFIFxIkws/nV3W7e2JxMdQHVpyE7meoCJ1d96qIuamISEZGwFBAiIhKWAqLC09EuQB06meoCqk9DdjLVBU6u+pxwXdQHISIiYekMQkREwlJAiIhIWE0+IMxsjJl9aWZrzez+aJentszsWTPbbmbLQta1NLP3zWxN8NgimmWsKTPrZGYzzWylmS03s+8F6xtrfZLMbK6ZLQ7q80iwvlHWB8DMYs1soZm9HTxvzHXZaGZLzWyRmc0P1jXm+pxiZq+a2argb2jIidanSQeEmcUCTwJjgd7AdWbWO7qlqrXngDFV1t0PzHDO9QBmBM8bgxLgh865XsA5wD3Bv0djrc9B4ELnXH9gADDGzM6h8dYH4HvAypDnjbkuACOccwNC7hdozPV5HHjXOdcT6I//dzqx+jjnmuwCDAGmhzx/AHgg2uU6jnpkAstCnn8JtA9+bg98Ge0yHme93gRGngz1AVKAL4DBjbU+QMfgQ+ZC4O1gXaOsS1DejUBGlXWNsj5AM2ADwYVHdVWfJn0GAXQAckKe5wbrGru2zrktAMFjmyiXp9bMLBM4A/icRlyfoElmEbAdeN8515jr8xjwY6AsZF1jrQuAA94zswVmdkewrrHWpyuQB/w9aAJ8xsxSOcH6NPWAsDDrdN1vlJlZGvAa8H3n3L5ol+dEOOdKnXMD8N++zzaz06NdpuNhZpcB251zC6Jdljo0zDk3EN/EfI+ZnR/tAp2AOGAg8Bfn3BlAAXXQPNbUAyIX6BTyvCOwOUplqUvbzKw9QPC4PcrlqTEzi8eHw4vOuX8Fqxttfco55/YAs/D9RY2xPsOAcWa2EXgJuNDM/knjrAsAzrnNweN24HXgbBpvfXKB3OAMFeBVfGCcUH2aekDMA3qYWZaZJQDXAlOiXKa6MAW4Jfj5FnxbfoNnZgb8DVjpnHs05KXGWp/WZnZK8HMycDGwikZYH+fcA865js65TPzfyYfOuRtphHUBMLNUM0sv/xkYBSyjkdbHObcVyDGz7GDVRcAKTrA+Tf5OajO7BN+2Ggs865z7rygXqVbMbBIwHD+07zbgYeANYDLQGfgKuNo5tytaZawpMzsX+BhYSkU794P4fojGWJ9+wD/w/23FAJOdc78ws1Y0wvqUM7PhwH3Oucsaa13MrCv+rAF888xE59x/Ndb6AJjZAOAZIAFYD3yL4L87jrM+TT4gREQkvKbexCQiItVQQIiISFgKCBERCUsBISIiYSkgREQkLAWEyFGY2f+Y2XAzuzJao/2a2SwzO6HJ50WOhwJC5OgG4+/DuAB/j4ZIk6GAEAnDzP7XzJYAZwGfAbcDfzGzn4XZtrWZvWZm84JlWLD+52b2gpl9GIzH/+1gvQXHXxbMR3BNyLF+HKxbbGa/Dnmbq4O5JVab2XkRrbxIIC7aBRBpiJxzPzKzV4CbgB8As5xzw6rZ/HHgD865f5tZZ2A60Ct4rR9+botUYKGZTcUPMz8AP2Z/BjDPzGYH664EBjvnDphZy5D3iHPOnR3c+f8wftgOkYhSQIhU7wxgEdATP65NdS4GevuhpABoVj7OD/Cmc64QKDSzmfgB4c4FJjnnSvGDqX2EP1O5APi7c+4AQJUhEcoHLlyAn/9DJOIUECJVBGPaPIcf3XcHfrIfC+Z1GBJ84IeKCbc+CIyqY9k4wg8zT7C+urFvDgaPpejvVuqJ+iBEqnDOLQrmcFiNn4r2Q2C081NTVg0HgPeAe8ufBAFT7grzc1O3wg+qOA+YDVwTTCbUGjgfmBsc51YzSwmOE9rEJFLvFBAiYQQf3Ludc2VAT+fc0ZqYvgsMMrMlZrYCuCvktbnAVGAO8MtgDoLXgSXAYnz4/Ng5t9U59y5+eOb5wdnKfXVeMZFa0GiuIhFiZj8H8p1zv4t2WUSOh84gREQkLJ1BiIhIWDqDEBGRsBQQIiISlgJCRETCUkCIiEhYCggREQnr/wcwHka/x72rwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show plot accuracy changes during training\n",
    "\n",
    "plt.plot(history.history['loss'],'g')\n",
    "plt.plot(history.history['val_loss'],'r')\n",
    "plt.title('LSTM Loss across epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('# epoch')\n",
    "plt.legend(['train', 'test'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is starting to get trained, but at this point, the accuracy may not be that great.  In the real world, we train on even larger data sets for longer periods of time until the accurracy met our business requirements.  We also review the model and can increase it's complexity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and use our model to make predictions on the test data.  Remember, this is a reconstruction model, so the predictions, ideally, should look similar to the original data for normal disks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use model to predict on test data set\n",
    "y_pred = autoencoder.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell will plot a comparison of the original data to the predicted data for a random record. It compares the features for the first day in the sequence. Run it a few times to see how well your model came out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This record is from an anomalous disk.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEWCAYAAABfdFHAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZgkVZ3u8e9bvSjV7HSrQNNVgugozrC1CDOIDG7AqKDiXLFUwKUGGFyvjkt7FUfbwXF3FLRUQOwSdxEdFEFvD3AVpZtNEFSQqu5i62YRhVKhu3/3j3PKzqrOzMrMqqjIrH4/z5NPZp6IE3Fi/UWcOBGhiMDMzKwoXWUXwMzMZjcHGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQjnQdBhJSyQ9KGnOdPbbwLDOlfSBqQ7HGpOX254FDPdJkq6R9EdJbyhg+IdLGpnu4U4nSdtI+p6kByR9o+zybA3aOtBIermkVXmju1PSDyQdWna5yhQRayJi24jYOJ39ToWkEyVdUeQ4ylR0kJW0UtJrK9PycvtdAaP7N2BlRGwXEZ8qYPiFyfuDYUkPSbpA0s4tDuo44LHALhHx0mksYseRdI6kkPSEirR/lvRTSaOSVk7o/xl5f1z5CUkvqTeetg00kt4CfAL4IGmlWAKcCRxTZrkmI2luJw67nXTadHZYeXuAG1vJWOZ0StoH+BzwStL+YJS0P2hFD/CbiNgwTcXrSPmgfa8qne4j7XvPmNghIi7PB0HbRsS2wPOBB4Ef1h1ZRLTdB9ghF/6ldfp5VJ4Zd+TPJ4BH5W6HAyOko7d1wJ3AscDRwG/yjHxXxbBOB74JfA34I3A1sG9F93cAt+ZuvwJeVNHtROD/AR/Pw/1ALttHgDXA3cBngW1qTEcX8G5gOJf1PGCH3K0XCOA1eViXVaTNzf08Pqf/EbgU+AywYkL+sX5XAu/P5f0j8CNgYUVZvgHcBTyQh7lPRbdzgQ9UKf+TgT8DG/My+33FMjwPWJ+n7d1AV415MDb/VwB/AF6b838xL7vb83ydU5HndcBNFcvkgIryrAR+T9qhvnDCNHwG+O+c7+fAXrmb8jJcl6f/euCpQD/wCPBwnr7v5f6HgLfn/v4CzM3z+gm15hnpIOnaPI23AkcCy/O8+3Me/qdzv38dVr15SVr/riCtb/cDtwFH1ZjPP5kwric2MOxx63aVYW6Tp/P+vBzeBoxMtu2QtpH7gL+t6PcxwJ+ARVXG80HgKxX/98rLZLsa01p1PQDel/M9kufBa6rkPQhYlZfT3cDHKrodDPw0D/c64PCKbo8H/idP6yXAp8nbYpVxHE7aR/1vNu+jTqrovhJ47YT9zBUV/wM4FfhtHt/78zz5WS7314H5dfafc4FrgL9jwnpb0c9rSWe/9fbV5wDnTLpPn6yHMj6kDXADeQdZo59/B67MK+eivPDfX7EQNwDvAeaRdkrrga8A2wH7kDa2PXP/p+cV77jc/1tJG+y83P2lwG6koPC/gIeAXStWgA3A6/PC24YU9C4Eds7j+x7wHzWm49XALcCewLbAt4Ev5269eSU4D1iQhz2WNhY8fkbaycwHDs0rWb1AcytpB7NN/n/GhLJsx+Ygfm1Ft3OpsqOpthHktPOA7+bh9ZIC/BYb9YT5f2yex9sAF5COYBfkZfwL4F8qlsftwNNIAeIJpKPUeXlevivPjyNIG+GTKqbhPtKOZC4wCHw1d3sesBrYMQ/zyRXLeItpJwWaa4E9yAcR1Ak0eZwPAM/J07g78DfVdioTh1VvXuZ5/whpHZ8DnEI68FKNeT1uXA0Me9y6XWV4ZwCXk9b1PYAbGB9o6m07ZwIfquj3jeRAXmU83wXePiHtQeDAKv1Oth6cTo0AULFNvTL/3hY4OP/eHbiXdMDalZflveTAmPN9jLT9HJbHWS/QbCDtx+blYY4CO9VYTieyZaC5ENietD/7C/Bj0n5kB1JQP6HONL4N+GS19bain7qBBujO03j4pPv0yXoo4wP0AXdN0s+twNEV/58HDFUsxD+Rj4DzRhTA0yv6Xw0cW7HiXVnRrYt0hPGMGuO+FjimYgVYU9FNeWPaqyLtEOC2GsP6MXBqxf8nkXYcc9kcKPas6D6WNpdUnbgB6K7ovoL6gebdFf2eCvywRrl2zHnHzq7OpcFAQ9rh/QV4SkXav9RaafP8v6zi/2Nz/m0q0o4H/m/+fTHwxirDeQbpjKyrIu184PSKafhCRbejgZvz7yNIO9mDmXDmVW3aSYHm1RPS6gWazwEfrzH9K6kRaCabl3ne31LRrTvnfdxk42pw2GuqDaei/98BR1b876ci0Eyy7TwdWMvmM6hVwD/X2U5OnpB2O1V2cg2sB6dTP9BcRjrzWTgh/e3kg8CKtIuBE9i8LS6o6PaVWuNh8z5qbkXaOjYHtXHrBNUDzT9U/F9NRSAGPgp8osa49yAF4h0qhtVKoHkl6YC86kFN5addr9HcCyycpE54N9Kp/pjhnPbXYcTmi+B/yt93V3T/E+loZczasR8RsYl0WrsbgKRXSbpW0u8l/Z5UpbKwWl7S2VU3sLqi/x/m9EanYy5pZ1tt+BPz3hcRow30O+auit+j5HkgaY6kMyTdKukPpB0pjJ/ORi0kHUlOnK7d6+SpLPfY2cmdFfPwc6QzG0gbyq1VhrEbsDYvv1rjrTr9EfETUlXHZ4C7JQ1I2r5OeSeWeTK1yjyZRublX6epYl2oXLenMuzJpnG3Cf1UDqvuthMRPycdlD1T0t+QAuuFNcbzIOnovdL2pCPqqmWaZD2o5zWks/6bJV0l6fk5vQd46di05Ok5FNg1j/P+iHhowjjruTfGXyf66/rYoIn7s3r7t0qfAP49Ih5oYlzVnACcFznq1NOugeZnpKqtY+v0cwdpwY9ZktNatcfYD0ldwGLgDkk9wOeB00itVHYkVQ+oIm/ljL6HtJD3iYgd82eHSBfOGp2ODYxfaWotyDuBnSV1V5uOJr2cdA3h2aRT796crloZ6pTvHtJZ2cTpur3BYawlHWkvrJiH20fEPhXdq13EvAPYIy+/Rse7uQARn4qIA0lVEU8kVS9MLFutMkPaUVQui8dV/K5V5nrDh9bmZaMaGfZkO5E7Gb/OLRn70eC28yXgFaSj429GxJ9rjOdGYN+KYe9JqqL6TZV+p7oe/DYijicd2HwI+KakBaRl+OWKdXLHiFgQEWeQ5sNOub/KcbbqIWqvS1P1LODDku6SNHaQ8jNJL290AJL2IJ2VnddI/20ZaHKkfQ/wGUnHSuqWNE/SUZL+M/d2PvBuSYskLcz9r5jCaA+U9OJ8FvUm0o7uStI1giBd40HSSaSjslpl30TauD4u6TE5z+6Snlcjy/nAmyU9XtK2pIueX4sGWsRExDCpuuF0SfMlHQK8oLHJ3cJ2pGm+l7SCf7CJvHcDiyXNz+XaSLoYuVzSdnmH8xYaXD4RcSepocJHJW0vqUvSXpKemXv5AvBWSQcqeUIex9gR8r/l9eVw0vz46mTjlPQ0SU+XNC8PY6yBw9j0NXJPy7XAy/PZ4ZHAMyu6fRE4SdKz8vTsno/i6w5/qvOynmka9teBd0raSdJi0vWcMY1sO18GXkQKNvV2WoPAC3Lz2gWkaxvfjohqZzQtrwe5nK+QtChvy7/PyRtJ8+UFkp6Xl/Gjle4bWlyxLb4vb4uH0vq2CGldenHe9z2BdJY1XZ5ICtr75Q+ksn4H/lq78WhSzUpXns55E4bxSuCnEdHQWXpbBhqAiPgYaaV/N2lFXUs6Mrog9/IB0oK9HvglqaXYVO51+C7pYuX9pJn44oh4JCJ+Rarv/Blph/C3pJY49bydVAd6Za6GupR07aWas0kb22Wk+s4/M35jnUwf6RrQvaTp/xopYDTrPNKp/u2kC4lXNpH3J6Qjzrsk3ZPTXk/a2H9HahX1FdK0NupVpGqdX5GWyTdJVRRExDdIrbW+Qqo6uQDYOSIeBl4IHEU6Wj8TeFVE3NzA+LYnHSDcT5oP95IaWUAKEk/J1SUX1MgP6WL2C0g7pz42r6tExC+Ak0gtuB4gtU4aO5P4JHCcpPslVbu3Zarzsp6pDvt9pPl1G+ng4MtjHRrZdiJihLTtBqlRQVURcSNwMingrCMdGJ1ao9+prAeQGiPdKOlB0rJ5WUT8OSLWks7638XmfdLb2LwffTnputN9wHtp8Gi/ho+TWsfdTTrrG5zCsMaJiHURcdfYJyffExFjlxheSaqVOYt0vetPpG2j0qtyuRqiBqrXZj1Jp5Muhr2i7LJMlaSvkS5wv7fsspg1QtLZwB0R8e6yyzKdZtN+Zao66UYzq0LS00hHULcBzyUdcW1xo5VZO5LUC7wY2L/ckliR2rbqzBr2OFJTyAeBTwGnRMQ1pZbIrAGS3k9qHPDhiLit7PJYcVx1ZmZmhfIZjZmZFWpWXqNZuHBh9Pb2ll0MM7OOsXr16nsiotaN5VMyKwNNb28vq1atKrsYZmYdQ9JkTzJoWalVZ5LOlrRO0g01uh+u9HKia/PnPTNdRjMzm5qyz2jOJT1fqt6NTZdHxPPrdDczszZW6hlNRFxGugfEzMxmqU5odXaIpOuUXuO8T62eJPUrvfZ51fr162eyfGZmVke7B5qrgZ6I2Bf4LyqeHTVRRAxExNKIWLpoUSENJ8zMrAVtHWgi4g8R8WD+fREwLz+p2WzrMTgIvb3Q1ZW+B6ft+YrF6tRy27QruzFAXZIeB9wdESHpIFJgvLfkYpnNnMFB6O+H0fw+s+Hh9B+gr6+8ck2mU8tthSj1ETSSzie9PGch6XHY7yW9WZGI+Kyk00jvQN9AelT1WyLip5MNd+nSpeH7aGxW6O1NO+mJenpgaGimS9O4Ti33VkzS6ohYWsiwZ+OzzhxobNbo6oJq26gEmzZtmd4uOrXcW7EiA01bX6Mx2+otqfE24Frp7aJTy22FcKAxa2fLl0N39/i07u6U3s46tdxWCAcas3bW1wcDA+nahpS+Bwba/4J6p5bbCuFrNGZm5ms0ZmbWuRxozMysUA40ZmZWKAcaMzMrlAONmZkVyoHGzMwK5UBjZmaFcqAxM7NCOdCYmVmhHGjMzKxQDjRmZp2uzd9m2tZv2DQzs0l0wNtMfUZjZtbJli3bHGTGjI6m9DbhQGNm1snWrGkuvQQONGZmnawD3mbqQGNm1sk64G2mDjRmZp2sA95m6lZnZmadrq+vrQLLRD6jMTOzQjnQmJlZoRxozMysUA40ZmZWKAcaMzMrlAONmZkVyoHGzMwK5UBjZmaFcqAxM7NCOdCYmVmhHGjMzKxQDjRmZlYoBxozMytUqYFG0tmS1km6oUZ3SfqUpFskXS/pgJkuo5mZTU3ZZzTnAkfW6X4UsHf+9ANnzUCZzMxsGpUaaCLiMuC+Or0cA5wXyZXAjpJ2nZnSmZnZdCj7jGYyuwNrK/6P5LQtSOqXtErSqvXr189I4czMbHLtHmhUJS2q9RgRAxGxNCKWLlq0qOBimZlZo9o90IwAe1T8XwzcUVJZzMysBe0eaC4EXpVbnx0MPBARd5ZdKDMza9zcMkcu6XzgcGChpBHgvcA8gIj4LHARcDRwCzAKnFROSc3MrFWlBpqIOH6S7gH86wwVx8zMCtDuVWdmZtbhHGjMzKxQDjRmZlYoBxozMyuUA42ZmRXKgcbMzArlQGNmZoVyoDEzs0I50JiZWaEcaMzMrFAONGZmVigHGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQjnQmJlZoRxozMysUA40ZmZWKAcaMzMrlAONmRVicBB6e6GrK30PDpZdIiuLA41Zm+vEHfbgIPT3w/AwRKTv/v7OKLtNPwcaszbWqTvsZctgdHR82uhoSretjwONWRvr1B32mjXNpdvs5kBj1sY6dYe9ZElz6TY17V696kBj1sY6dYe9fDl0d49P6+5O6Ta9OqF61YHGWjJ46hX0zh2hS5vonTvC4KlXlF2kWalTd9h9fTAwAD09IKXvgYGUbtOrE6pXFRFll2HaLV26NFatWlV2MWatwVOvoP+s/RllwV/TunmIgVOuoe/MQ0ss2ew0OJh2GmvWpDOZ5cu9w7bNurrSmcxEEmza1PhwJK2OiKXTV7KKYTvQWLN6544wvHHxFuk9c0YY2rBlupkVp7c3VZdN1NMDQ0OND6fIQOOqM2vamo27NZVuZsXphOpVBxpr2pI5dzSVbmbF6YTrYQ401rTl/UN089C4tG4eYnn/UDkFMtvK9fWlarJNm9J3OwUZcKCxFvSdeSgDp1xDz5wRxCZ65oy4IYCZ1eTGAGZm5sYA7a7d78o1MyvT3LIL0OnG7sodu2Fq7K5caL96UjOzMpR6RiPpSEm/lnSLpHdU6X64pAckXZs/7ymjnPV0wl25ZmZlKu2MRtIc4DPAc4AR4CpJF0bEryb0enlEPH/GC9igTn3ooZnZTCnzjOYg4JaI+F1EPAx8FTimxPK0pFMfemhmNlPKDDS7A2sr/o/ktIkOkXSdpB9I2qfWwCT1S1oladX69eunu6w1LT/6iur3lBzth0yamUG5gUZV0ia2tb4a6ImIfYH/Ai6oNbCIGIiIpRGxdNGiRdNYzPr6LnoFA7yOHobSPSUMMcDr6LvoFTNWBjOzdlZmq7MRYI+K/4uBcc8wiYg/VPy+SNKZkhZGxD0zVMbJrVlDH8P0cf6E9Gpx1Mxs61PmGc1VwN6SHi9pPvAy4MLKHiQ9TpLy74NI5b13xktajy/SmJnV1XCgkbRg8r4aFxEbgNOAi4GbgK9HxI2STpZ0cu7tOOAGSdcBnwJeFu32KINOeHSqmVmJJn0EjaS/B74AbBsRSyTtC/xLRJw6EwVsxYw/gsZvpjKzDlfkI2gauUbzceB55GqtiLhO0mFFFKZj9fU5sJiZ1dBQ1VlErJ2QtLGAspiZ2SzUyBnN2lx9Fvmi/RtI11TMzMwm1cgZzcnAv5JuphwB9sv/zczMJjXpGU2+Z8UXIMzMrCWTBhpJ57DlHftExKsLKZGZmc0qjVyj+X7F70cDL2LCHfxmZma1NFJ19q3K/5LOBy4trERmZjartPIImr0BP1/FzMwa0sg1mj+SrtEof98FvL3gcpmZ2SzRSNXZdjNREDMzm51qBhpJB9TLGBFXT39xzMxstql3RvPROt0COGKay2JmZrNQzUATEf84kwUxM7PZqaE3bEp6KvAU0n00AETEeUUVyszMZo9GWp29FzicFGguAo4CrgAcaMzMbFKN3EdzHPAs4K6IOAnYF3hUoaUyM7NZo5FA8+eI2ARskLQ9sA7Ys9himZnZbFGvefOngfOBX0jaEfg8sBp4EPjFzBTPzMw6Xb1rNL8FPgLsRgou5wPPAbaPiOtnoGxmZjYL1Kw6i4hPRsQhwGHAfcA5wA+AYyXtPUPlMzOzDjfpNZqIGI6ID0XE/sDLSa8JuLnwkpmZ2awwaaCRNE/SCyQNks5ofgO8pPCSmZnZrFCvMcBzgOOBfyJd/P8q0B8RD81Q2czMbBaod0bzLuBnwJMj4gURMTibg8zgIPT2QldX+h4cLLtEZmazg591Rgoq/f0wOpr+Dw+n/wB9feWVy8xsNmjlDZuzzrJlm4PMmNHRlG5mZlPjQAOsWdNcupmZNc6BBliypLl0MzNrnAMNsHw5dHePT+vuTulmZjY1DjSkC/4DA9DTA1L6HhhwQwAzs+nQ0IvPtgZ9fQ4sZmZF8BmNmZkVyoHGzMwK5UBjZmaFKjXQSDpS0q8l3SLpHVW6S9KncvfrJR1QRjnNzKx1pQUaSXOAzwBHAU8Bjpf0lAm9HQXsnT/9wFkzWkgzM5uyMs9oDgJuiYjfRcTDpKdDHzOhn2OA8yK5EthR0q4zXdB2NXjqFfTOHaFLm+idO8LgqVeUXSSzzucn7E67MgPN7sDaiv8jOa3ZfgCQ1C9plaRV69evn9aCtqPBU6+g/6z9Gd64mKCL4Y2L6T9rfwcbs6kYHGTwpEvpHV5JV2ygd3glgydd6mAzRWUGGlVJixb6SYkRAxGxNCKWLlq0aMqFa3fLBnoZZcG4tFEWsGygt5wCmc0Cg2/8Of2PfJphetMBHL30P/JpBt/487KL1tHKDDQjwB4V/xcDd7TQz1ZpzcbdmkpvJ67ys3a17N63VD+Au/ctJZVodigz0FwF7C3p8ZLmAy8DLpzQz4XAq3Lrs4OBByLizpkuaJFa3ekumVM93tZKbxeu8rN2tobqT9KtlW6NKS3QRMQG4DTgYuAm4OsRcaOkkyWdnHu7CPgdcAvweeDUUgpbkKnsdJf3D9HN+BeedvMQy/uHCirt9HCVn7WzJbuMNpVujVFE1UseHW3p0qWxatWqsosxqd65IwxvXLxFes+cEYY2bJk+0eCpV7BsoJc1G3djyZw7WN4/RN+ZhxZR1GnTpU1EleMbsYlN4fuHrVyDg9D/6g2MPrz5MZDd8zcwcPbcWf8sREmrI2JpEcP2ll2iqV5n6TvzUIY2LGZTdDG0YXHbBxmYepWfr+9Ykfr6YODsueOf5L4VBJmiOdCUqFOvs0zFVKr8fH3HZkJfHwwNwaZN6dtBZuocaErUqddZpqLvzEMZOOUaeuaMIDbRM2eEgVOuaehszNd3th5TOnP1DZftJyJm3efAAw+MTrHilMujZ87aEBujZ87aWHHK5WUXqW2JjQGxxUdsLLtoNo1WnHJ5dPPguGXczYONbRsrVsSKeSdGD7elbYrbYsW8EyNWrCi+4B0OWBUF7ZPdGMA6xlQbT1hnmMpyHlz4Bvrv/Y9xZ77dPMTALu+k755PTXtZZxM3BjBj66xq3BpNpZGMb7hsTw401jGmcn3HOsdUGsn4hsv25EBjHaXMJt1uWj0zpnLm6hsu25MDjVkD3LR65kzlzHX5J7ele/6GcWnd8zew/JPbFlVca4AbA5g1wA0ROsfgICxbBmvWwJIlsHy574VpRJGNARxozBrgR+fYbOdWZ2Yl2xqf4mA2XRxobKvS6k3jblpt1joHGttqjD2Zd3g43W8+PJz+NxJs3LTarHW+RjPGVxBnvd6FDzJ875atj3p2eZChe9wqybZuRV6jmTt5L1uBwUHo74fR3NZ+eDj9BwebWWTNvd1NpZvZ9HDVGaQzmdEJN3SNjqZ0mzWWsKapdDObHg40kKrLmkm3jrR8l49Vv6C/y8dKKlEH8CP3bRo40EC6JtNMunWkvk8+nYF5p9HDULqgzxAD806j75NPL7tohZlSnBgcZPCkS+kdXklXbKB3eCWDJ13qYGPNK+r9A2V+mn4fzYoVEd3d419y0t3td1jMRitWRPT0REjpe4aWcRmjXbEionv+I+NX6/mPNDzuFbu8vvp7YXZ5fbEFt1Lg99E0x63OrJ2MNasefXhz25vu+RsKfxf9VFvZ9WqIYXq3zM8QQ7FlunU2P4KmSX4EjbWTsppVT/WxOX7sztbFj6Ax62BlNaueais7P3LfposDjVnBympWPdVWdn7kvk0XBxqzgpXVrHqqrez6+mDg7Ln09IAEPT0Ufl3JZicHGrOCldasuq+PvnOezVDP4WzSXIZ6DqfvnGc31cilrw+GhmDTpvTtIGOtcGMAs5ngVo3W5vysM7NO19fnwGJbLVedmZlZoRxozMysUA40ZmZWKAcaMzMrlAONmZkVyoHGzGy6+P09VTnQbM28UZhNn7FXwg8Pp7cqjL0S3tuVA81WyxuF2fTyK+FrKiXQSNpZ0iWSfpu/d6rR35CkX0q6VpJv9Z9O3ijMpteaNQxyPL3cRhcb6eU2Bjner4SnvDOadwA/joi9gR/n/7X8Y0TsV9SjEbZa3ijMptXgzqfRz+cZppegi2F66efzDO58WtlFK11ZgeYY4Ev595eAY0sqx1bLG4XZ9FrGBxllwbi0URawjA82lH82XzIt5aGakn4fETtW/L8/IraoPpN0G3A/EMDnImKgzjD7gX6AJUuWHDg8PDz9BZ9Fynrro9ls1dWVLndOJKWnX9czdsm0sja7uxsGBmbuEXkd+SpnSZcCj6vSaRnwpQYDzW4RcYekxwCXAK+PiMsmG7ef3jy5qWwUZral3t7Upmainp70ioWi8k6XjnyVc0Q8OyKeWuXzXeBuSbsC5O91NYZxR/5eB3wHOKio8m5tlixpLt3M6lu+PJ2FVOruTumTqXVpdLZcMi3rGs2FwAn59wnAdyf2IGmBpO3GfgPPBW6YsRLOclPZKMxsS319qapr3BtJG6z6mu0HfmUFmjOA50j6LfCc/B9Ju0m6KPfzWOAKSdcBvwD+OyJ+WEppZ6GpbBRmVl2rbySd7Qd+fsOmmVkbKPslrH7DppnZLDebX8LqR9CYmVmhHGjMzKxQDjRmZlYoBxozMyuUA42ZmRXKgcbMzArlQGNmZoVyoDEzs0I50JiZWaEcaMzMrFAONGZmVigHGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTNrB4OD0NsLXV3pe3Cw7BJNG7/4zMysbIOD0N8Po6Pp//Bw+g+z4m1oPqMxMyvbsmWbg8yY0dGUPgs40JiZlW3NmubSO4wDjZlZ2ZYsaS69wzjQmJmVbfly6O4en9bdndJnAQcaM7Oy9fXBwAD09ICUvgcGZkVDAHCrMzOz9tDXN2sCy0Q+ozEzs0I50JiZWaEcaMzMrFAONGZmVigHGjMzK5QiouwyTDtJ64HhFrMvBO7ZivKWOW5Pc2fkLXPcnuaZy9sTEYtazFtfRPhT8QFWbU15O7XcnmbPL0/z9E9zUR9XnZmZWaEcaMzMrFAONFsa2MryljluT3Nn5C1z3J7mmctbmFnZGMDMzNqHz2jMzKxQDjRmZlYoB5pM0pGSfi3pFknvaDLv2ZLWSbqhhfHuIen/SrpJ0o2S3thE3kdL+oWk63Le97Uw/jmSrpH0/RbyDkn6paRrJa1qMu+Okr4p6eY87Yc0mO9JeXxjnz9IelMT431znlc3SDpf0qObyPvGnO/GRsZZbb2QtLOkSyT9Nn/v1ETel+Zxb5K0tMnxfjjP6+slfUfSjk3kfX/Od62kH0narZlxV3R7q6SQtLCJcZ8u6faK5X10M+OV9Pq8Xd8o6T+bGO/XKsY5JOnaJvLuJ+nKse1C0kHV8tbJv6+kn+Vt63uStq69lWcAAAfvSURBVK+Rt+q+o9F1bEaV3b66HT7AHOBWYE9gPnAd8JQm8h8GHADc0MK4dwUOyL+3A37T6LgBAdvm3/OAnwMHNzn+twBfAb7fQtmHgIUtzvMvAa/Nv+cDO7a43O4i3WjWSP+7A7cB2+T/XwdObDDvU4EbgG7S6zUuBfZudr0A/hN4R/79DuBDTeR9MvAkYCWwtMnxPheYm39/qMnxbl/x+w3AZ5sZd07fA7iYdCN11XWmxrhPB97awPKplvcf83J6VP7/mGbKXNH9o8B7mhjvj4Cj8u+jgZVNlvsq4Jn596uB99fIW3Xf0eg6NpMfn9EkBwG3RMTvIuJh4KvAMY1mjojLgPtaGXFE3BkRV+fffwRuIu0QG8kbEfFg/jsvfxpu3SFpMfBPwBeaKvQU5SO0w4AvAkTEwxHx+xYG9Szg1oho5ikQc4FtJM0lBY07Gsz3ZODKiBiNiA3A/wAvqpehxnpxDCnIkr+PbTRvRNwUEb+erKA18v4olxvgSmBxE3n/UPF3AXXWsTrbwseBf2sx76Rq5D0FOCMi/pL7WdfseCUJ+Gfg/CbyBjB2FrIDddaxGvmfBFyWf18CvKRG3lr7jobWsZnkQJPsDqyt+D9Cgzv76SSpF9ifdGbSaJ45+bR+HXBJRDScF/gEaePf1ESeSgH8SNJqSf1N5NsTWA+ck6vtviBpQQvjfxk1dgDVRMTtwEeANcCdwAMR8aMGs98AHCZpF0ndpCPVPZosL8BjI+LOXJ47gce0MIypejXwg2YySFouaS3QB7ynybwvBG6PiOuayVfhtFx1d3aT1UBPBJ4h6eeS/kfS01oY9zOAuyPit03keRPw4Ty/PgK8s8lx3gC8MP9+KQ2sZxP2He2wjo3jQJOoStqMtvuWtC3wLeBNE44g64qIjRGxH+kI9SBJT21wfM8H1kXE6pYKnPxDRBwAHAX8q6TDGsw3l1RdcFZE7A88RDrFb5ik+aSN8RtN5NmJdLT3eGA3YIGkVzSSNyJuIlU5XQL8kFS9uqFupjYkaRmp3IPN5IuIZRGxR853WhPj6waW0WRwqnAWsBewH+ng4KNN5J0L7AQcDLwN+Ho+Q2nG8TRxMJOdArw5z683k8/cm/Bq0va0mlQl9nC9nlvdd8wkB5pkhPFHDYtpvEplyiTNI60ogxHx7VaGkaueVgJHNpjlH4AXShoiVRUeIWlFk+O8I3+vA75DqoJsxAgwUnH29U1S4GnGUcDVEXF3E3meDdwWEesj4hHg28DfN5o5Ir4YEQdExGGk6o5mjnLH3C1pV4D8XbU6pwiSTgCeD/RFrsBvwVeoUZVTw16kwH5dXtcWA1dLelwjmSPi7nwwtQn4PI2vY5DWs2/nKuZfkM7cqzZEqCZXr74Y+FoT4wQ4gbRuQToQaqbMRMTNEfHciDiQFORurVPGavuO0taxWhxokquAvSU9Ph8pvwy4cCZGnI+wvgjcFBEfazLvorHWQ5K2Ie1Ib24kb0S8MyIWR0QvaXp/EhENHd3n8S2QtN3Yb9LF5oZa3UXEXcBaSU/KSc8CftXouLNWjjTXAAdL6s7z/Vmkeu2GSHpM/l5C2gE1O35I69UJ+fcJwHdbGEbTJB0JvB14YUSMNpl374q/L6TBdQwgIn4ZEY+JiN68ro2QLmDf1eC4d634+yIaXMeyC4Aj8nCeSGp00syTjZ8N3BwRI03kgXSQ+sz8+wiaPCCpWM+6gHcDn63RX619RynrWF1lt0Zolw+pzv03pKOHZU3mPZ90Wv8IaUN6TRN5DyVV010PXJs/RzeY9++Aa3LeG6jRMqaB4RxOk63OSNdZrsufG1uYZ/sBq3LZLwB2aiJvN3AvsEML0/o+0o7yBuDL5BZJDea9nBQQrwOe1cp6AewC/Ji08/kxsHMTeV+Uf/8FuBu4uIm8t5CuQ46tY1VbjtXI+608v64Hvgfs3uq2QJ2WijXG/WXgl3ncFwK7NpF3PrAil/1q4IhmygycC5zcwjI+FFid15OfAwc2mf+NpH3Rb4AzyE9wqZK36r6j0XVsJj9+BI2ZmRXKVWdmZlYoBxozMyuUA42ZmRXKgcbMzArlQGNmZoVyoDFrgKSNGv/U6N4WhnGspKdMf+nM2tvcsgtg1iH+FOlRP1NxLPB9mrg5VdLc2PwwTLOO5DMasxZJOjA/rHG1pIsrHvvxOklXKb0n6Fv5SQR/T7qr/sP5jGgvSSuV3ysjaWF+RAuSTpT0DUnfIz20dEF+oORV+SGkDT9Z3KwdONCYNWabimqz7+RnTP0XcFykZ1KdDSzP/X47Ip4WEfuSHnHzmoj4KenO9rdFxH4RUfP5VdkhwAkRcQTpoZQ/iYinkd6x8uEWn3ZtVgpXnZk1ZlzVWX5K9lOBS/IDgeeQHiUC8FRJHwB2BLYlvfCrWZdExNh7Sp5LegDqW/P/RwNLaOI5bWZlcqAxa42AGyOi2iuozwWOjYjrJJ1IepZcNRvYXKsw8ZXSD00Y10uigZeembUjV52ZtebXwCJJh0B6XLukfXK37YA7c/VaX0WeP+ZuY4aAA/Pv4+qM62Lg9WPvUpG0/9SLbzZzHGjMWhDpld/HAR+SdB3pyblj77b5P6Sn9l7C+EfqfxV4W76gvxfp7YunSPop9d+T8n7Sa7qvl3RD/m/WMfz0ZjMzK5TPaMzMrFAONGZmVigHGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQv1/jG+VUmQ8LDgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create horizontal positions for scatter plot\n",
    "x_pos = np.arange(x_test.shape[2])\n",
    "\n",
    "# select random record\n",
    "rec_num = random.randint(0, y_pred.shape[0])\n",
    "\n",
    "# plot predicted points\n",
    "plt.scatter(x_pos, x_test[rec_num,0,:], c=\"#ff0000\")\n",
    "plt.scatter(x_pos, y_pred[rec_num,0,:], c=\"#0000ff\")\n",
    "\n",
    "# label graph\n",
    "plt.xlabel(\"Feature\")\n",
    "plt.ylabel(\"Value\")\n",
    "plt.xticks(range(x_test.shape[2]))\n",
    "plt.title(\"Compare original to reconstruction for day 0 of seq num \" + str(rec_num))\n",
    "\n",
    "if y_labels_seq[rec_num] == 0:\n",
    "    print(\"This record is from a normal disk.\")\n",
    "else:\n",
    "    print(\"This record is from an anomalous disk.\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see that once you have a trained model, you need to determine a \"threshold\" of difference in detecting an anomaly. In our case, we know the number of normal disks and the number of anomalous disks. We'll calculate the MSE for each reconstruction and set our initial threshold based on that. This should be a good point to start the threshold at. In real world usage, you'll adjust the model and threshold to obtain accuracy to meet your business needs. In our case we'd like the recall to be as high as possible to identify as many failing disks as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Suggested Threshold: 0.1517269995824294\n"
     ]
    }
   ],
   "source": [
    "# Calcuate MSE for all sequences and predictions\n",
    "import statistics\n",
    "\n",
    "# determine median mse to split data into normal vs anomaly\n",
    "pred_mse = []  # calculated mse for test data\n",
    "pred_res = []  # result: 0 normal, 1 anomaly\n",
    "\n",
    "for i in range(len(x_test)):\n",
    "    pred_mse.append(mean_squared_error(y_pred[i,:,:], x_test[i,:,:]))\n",
    "\n",
    "# Find number of disks that we know are normal and use that value as splitting point for data\n",
    "num_normal_disks = list(y_labels_seq).count(0)\n",
    "\n",
    "sort_mse = pred_mse.copy()\n",
    "sort_mse.sort()\n",
    "suggested_threshold = sort_mse[num_normal_disks]\n",
    "print(\"Suggested Threshold:\", suggested_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Used threshold: 0.1517269995824294\n",
      "Resulting in the following classification of predicted data:\n",
      "- Normal Disks:    601\n",
      "- Anomalous Disks: 233\n"
     ]
    }
   ],
   "source": [
    "# Routine to specify whether or not a sequence is anomalous\n",
    "\n",
    "# specify override threshold or leave 0 to use suggested threshold\n",
    "override_threshold = 0\n",
    "\n",
    "if override_threshold == 0:\n",
    "    threshold = suggested_threshold\n",
    "else:\n",
    "    threshold = override_threshold\n",
    "\n",
    "for mse in pred_mse:\n",
    "    if (mse <= threshold):\n",
    "        pred_res.append(0)\n",
    "    else:\n",
    "        pred_res.append(1)\n",
    "        \n",
    "print(\"Used threshold:\", threshold)\n",
    "print(\"Resulting in the following classification of predicted data:\")\n",
    "print(\"- Normal Disks:   \", pred_res.count(0))\n",
    "print(\"- Anomalous Disks:\", pred_res.count(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"3b\"></a>\n",
    "## Training the 1D Convolutional model\n",
    "Now that we've tested our LSTM model, let's create and train a 1D-Convolutional model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint2 = ModelCheckpoint(data_dir+\"best_model_ae-conv1d.h5\", monitor='val_loss', verbose=1, save_best_only=True, mode='min')\n",
    "es2 = EarlyStopping(patience=20, verbose=1)\n",
    "\n",
    "callbacks_list2 = [es2, checkpoint2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build out a 1D Convolution Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         [(None, 16, 21)]          0         \n",
      "_________________________________________________________________\n",
      "conv1d (Conv1D)              (None, 16, 160)           20320     \n",
      "_________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D) (None, 8, 160)            0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 8, 160)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 8, 80)             76880     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 4, 80)             0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 4, 80)             0         \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 4, 4)              1924      \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 4, 4)              0         \n",
      "_________________________________________________________________\n",
      "up_sampling1d (UpSampling1D) (None, 8, 4)              0         \n",
      "_________________________________________________________________\n",
      "conv1d_3 (Conv1D)            (None, 8, 80)             2000      \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 8, 80)             0         \n",
      "_________________________________________________________________\n",
      "up_sampling1d_1 (UpSampling1 (None, 16, 80)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_4 (Conv1D)            (None, 16, 160)           76960     \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 16, 160)           0         \n",
      "_________________________________________________________________\n",
      "conv1d_5 (Conv1D)            (None, 16, 21)            3381      \n",
      "=================================================================\n",
      "Total params: 181,465\n",
      "Trainable params: 181,465\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Traditional Conv1D model\n",
    "\n",
    "# Model parameter settings\n",
    "sample_size = x_train.shape[1]\n",
    "num_signals = x_train.shape[2]\n",
    "dropout_rate = 0.2\n",
    "outside_filters = 160\n",
    "inside_filters = 80\n",
    "ksize=6\n",
    "\n",
    "# TTraditional 1D-Convolution Autoencoder\n",
    "\n",
    "input_data = Input(shape=(sample_size, num_signals))\n",
    "# --- Encode ---\n",
    "encoded = Conv1D(filters=outside_filters, kernel_size=ksize, activation=\"relu\", padding=\"same\")(input_data)\n",
    "encoded = MaxPooling1D(pool_size=2)(encoded)\n",
    "encoded = Dropout(rate=dropout_rate)(encoded)\n",
    "encoded = Conv1D(filters=inside_filters, kernel_size=ksize, activation=\"relu\", padding=\"same\")(encoded)\n",
    "encoded = MaxPooling1D(pool_size=2)(encoded)\n",
    "encoded = Dropout(rate=dropout_rate)(encoded)\n",
    "# --- Encodings ---\n",
    "encoded = Conv1D(filters=4, kernel_size=ksize, activation=\"relu\", padding=\"same\")(encoded)\n",
    "encoded = Dropout(rate=dropout_rate)(encoded)\n",
    "# --- Decode ---\n",
    "decoded = UpSampling1D(size=2)(encoded)\n",
    "decoded = Conv1D(filters=inside_filters, kernel_size=ksize, activation=\"relu\", padding=\"same\")(decoded)\n",
    "decoded = Dropout(rate=dropout_rate)(decoded)\n",
    "decoded = UpSampling1D(size=2)(decoded)\n",
    "decoded = Conv1D(filters=outside_filters, kernel_size=ksize, activation=\"relu\", padding=\"same\")(decoded)\n",
    "decoded = Dropout(rate=dropout_rate)(decoded)\n",
    "# --- Output ---\n",
    "output_layer = Conv1D(filters=num_signals, kernel_size=1, activation=\"linear\", padding=\"same\")(decoded)\n",
    "autoencoder2 = Model(input_data, output_layer)\n",
    "lr = 1e-3 # learning rate\n",
    "autoencoder2.compile(optimizer = \"adam\", loss = \"mse\", metrics = [\"accuracy\"])\n",
    "autoencoder2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4162 samples, validate on 463 samples\n",
      "Epoch 1/100\n",
      "4096/4162 [============================>.] - ETA: 0s - loss: 0.9154 - accuracy: 0.1481\n",
      "Epoch 00001: val_loss improved from inf to 0.46912, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 1s 356us/sample - loss: 0.9089 - accuracy: 0.1477 - val_loss: 0.4691 - val_accuracy: 0.2208\n",
      "Epoch 2/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.7889 - accuracy: 0.1562\n",
      "Epoch 00002: val_loss improved from 0.46912 to 0.38698, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.7695 - accuracy: 0.1547 - val_loss: 0.3870 - val_accuracy: 0.1833\n",
      "Epoch 3/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.6830 - accuracy: 0.2252\n",
      "Epoch 00003: val_loss improved from 0.38698 to 0.33731, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.7029 - accuracy: 0.2227 - val_loss: 0.3373 - val_accuracy: 0.2937\n",
      "Epoch 4/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6954 - accuracy: 0.2576\n",
      "Epoch 00004: val_loss did not improve from 0.33731\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.6797 - accuracy: 0.2600 - val_loss: 0.3486 - val_accuracy: 0.3510\n",
      "Epoch 5/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6619 - accuracy: 0.3148\n",
      "Epoch 00005: val_loss improved from 0.33731 to 0.32743, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.6571 - accuracy: 0.3227 - val_loss: 0.3274 - val_accuracy: 0.3957\n",
      "Epoch 6/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6149 - accuracy: 0.3888\n",
      "Epoch 00006: val_loss improved from 0.32743 to 0.30366, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.6346 - accuracy: 0.3891 - val_loss: 0.3037 - val_accuracy: 0.4603\n",
      "Epoch 7/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6755 - accuracy: 0.4203\n",
      "Epoch 00007: val_loss improved from 0.30366 to 0.28761, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.6117 - accuracy: 0.4201 - val_loss: 0.2876 - val_accuracy: 0.5054\n",
      "Epoch 8/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6520 - accuracy: 0.4382\n",
      "Epoch 00008: val_loss improved from 0.28761 to 0.27106, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.6079 - accuracy: 0.4409 - val_loss: 0.2711 - val_accuracy: 0.5234\n",
      "Epoch 9/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5855 - accuracy: 0.4507\n",
      "Epoch 00009: val_loss improved from 0.27106 to 0.25306, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.5716 - accuracy: 0.4512 - val_loss: 0.2531 - val_accuracy: 0.5437\n",
      "Epoch 10/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6293 - accuracy: 0.4734\n",
      "Epoch 00010: val_loss did not improve from 0.25306\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.5593 - accuracy: 0.4690 - val_loss: 0.2739 - val_accuracy: 0.5088\n",
      "Epoch 11/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.6072 - accuracy: 0.4944\n",
      "Epoch 00011: val_loss improved from 0.25306 to 0.25279, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.5459 - accuracy: 0.4946 - val_loss: 0.2528 - val_accuracy: 0.5607\n",
      "Epoch 12/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4432 - accuracy: 0.4903\n",
      "Epoch 00012: val_loss improved from 0.25279 to 0.23878, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.5084 - accuracy: 0.4916 - val_loss: 0.2388 - val_accuracy: 0.5045\n",
      "Epoch 13/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.4964 - accuracy: 0.5171\n",
      "Epoch 00013: val_loss did not improve from 0.23878\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4842 - accuracy: 0.5181 - val_loss: 0.2441 - val_accuracy: 0.5782\n",
      "Epoch 14/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5101 - accuracy: 0.5106\n",
      "Epoch 00014: val_loss improved from 0.23878 to 0.23208, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4856 - accuracy: 0.5126 - val_loss: 0.2321 - val_accuracy: 0.5309\n",
      "Epoch 15/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5109 - accuracy: 0.5217\n",
      "Epoch 00015: val_loss did not improve from 0.23208\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.5009 - accuracy: 0.5182 - val_loss: 0.2397 - val_accuracy: 0.5856\n",
      "Epoch 16/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5741 - accuracy: 0.5175\n",
      "Epoch 00016: val_loss improved from 0.23208 to 0.23146, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.5143 - accuracy: 0.5191 - val_loss: 0.2315 - val_accuracy: 0.5004\n",
      "Epoch 17/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5201 - accuracy: 0.5291\n",
      "Epoch 00017: val_loss did not improve from 0.23146\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.5131 - accuracy: 0.5246 - val_loss: 0.2436 - val_accuracy: 0.5267\n",
      "Epoch 18/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5227 - accuracy: 0.5042\n",
      "Epoch 00018: val_loss did not improve from 0.23146\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.5171 - accuracy: 0.5020 - val_loss: 0.2773 - val_accuracy: 0.5140\n",
      "Epoch 19/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.5102 - accuracy: 0.5187\n",
      "Epoch 00019: val_loss improved from 0.23146 to 0.22942, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4770 - accuracy: 0.5243 - val_loss: 0.2294 - val_accuracy: 0.5466\n",
      "Epoch 20/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5072 - accuracy: 0.5283\n",
      "Epoch 00020: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4795 - accuracy: 0.5272 - val_loss: 0.2359 - val_accuracy: 0.4970\n",
      "Epoch 21/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4733 - accuracy: 0.5128\n",
      "Epoch 00021: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4574 - accuracy: 0.5125 - val_loss: 0.2356 - val_accuracy: 0.5846\n",
      "Epoch 22/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5159 - accuracy: 0.5014\n",
      "Epoch 00022: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4834 - accuracy: 0.5034 - val_loss: 0.2477 - val_accuracy: 0.5015\n",
      "Epoch 23/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.4384 - accuracy: 0.5209\n",
      "Epoch 00023: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4708 - accuracy: 0.5229 - val_loss: 0.2297 - val_accuracy: 0.5144\n",
      "Epoch 24/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4451 - accuracy: 0.5248\n",
      "Epoch 00024: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4309 - accuracy: 0.5241 - val_loss: 0.2355 - val_accuracy: 0.4976\n",
      "Epoch 25/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4731 - accuracy: 0.5317\n",
      "Epoch 00025: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4728 - accuracy: 0.5241 - val_loss: 0.2431 - val_accuracy: 0.4891\n",
      "Epoch 26/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5190 - accuracy: 0.5076\n",
      "Epoch 00026: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4913 - accuracy: 0.5103 - val_loss: 0.2480 - val_accuracy: 0.5765\n",
      "Epoch 27/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4576 - accuracy: 0.5264\n",
      "Epoch 00027: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4766 - accuracy: 0.5262 - val_loss: 0.2355 - val_accuracy: 0.5088\n",
      "Epoch 28/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4917 - accuracy: 0.5288\n",
      "Epoch 00028: val_loss did not improve from 0.22942\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4552 - accuracy: 0.5298 - val_loss: 0.2317 - val_accuracy: 0.5483\n",
      "Epoch 29/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4169 - accuracy: 0.5500\n",
      "Epoch 00029: val_loss improved from 0.22942 to 0.22360, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4330 - accuracy: 0.5511 - val_loss: 0.2236 - val_accuracy: 0.5606\n",
      "Epoch 30/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4208 - accuracy: 0.5426\n",
      "Epoch 00030: val_loss did not improve from 0.22360\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4298 - accuracy: 0.5354 - val_loss: 0.2296 - val_accuracy: 0.5838\n",
      "Epoch 31/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4486 - accuracy: 0.5424\n",
      "Epoch 00031: val_loss did not improve from 0.22360\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4418 - accuracy: 0.5447 - val_loss: 0.2261 - val_accuracy: 0.6151\n",
      "Epoch 32/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3886 - accuracy: 0.5521\n",
      "Epoch 00032: val_loss did not improve from 0.22360\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4144 - accuracy: 0.5524 - val_loss: 0.2356 - val_accuracy: 0.5472\n",
      "Epoch 33/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3535 - accuracy: 0.5444\n",
      "Epoch 00033: val_loss improved from 0.22360 to 0.21465, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3619 - accuracy: 0.5457 - val_loss: 0.2146 - val_accuracy: 0.5252\n",
      "Epoch 34/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4281 - accuracy: 0.5388\n",
      "Epoch 00034: val_loss did not improve from 0.21465\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4054 - accuracy: 0.5414 - val_loss: 0.2296 - val_accuracy: 0.5786\n",
      "Epoch 35/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4201 - accuracy: 0.5444\n",
      "Epoch 00035: val_loss did not improve from 0.21465\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4298 - accuracy: 0.5480 - val_loss: 0.2300 - val_accuracy: 0.5377\n",
      "Epoch 36/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.5111 - accuracy: 0.5485\n",
      "Epoch 00036: val_loss did not improve from 0.21465\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4934 - accuracy: 0.5431 - val_loss: 0.2302 - val_accuracy: 0.5100\n",
      "Epoch 37/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4372 - accuracy: 0.5247\n",
      "Epoch 00037: val_loss improved from 0.21465 to 0.20944, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4070 - accuracy: 0.5299 - val_loss: 0.2094 - val_accuracy: 0.5211\n",
      "Epoch 38/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4074 - accuracy: 0.5782\n",
      "Epoch 00038: val_loss did not improve from 0.20944\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3837 - accuracy: 0.5765 - val_loss: 0.2104 - val_accuracy: 0.5694\n",
      "Epoch 39/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4556 - accuracy: 0.5530\n",
      "Epoch 00039: val_loss improved from 0.20944 to 0.20864, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4196 - accuracy: 0.5518 - val_loss: 0.2086 - val_accuracy: 0.5714\n",
      "Epoch 40/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.4507 - accuracy: 0.5528\n",
      "Epoch 00040: val_loss did not improve from 0.20864\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4278 - accuracy: 0.5491 - val_loss: 0.2185 - val_accuracy: 0.5400\n",
      "Epoch 41/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3708 - accuracy: 0.5505\n",
      "Epoch 00041: val_loss improved from 0.20864 to 0.19969, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4399 - accuracy: 0.5446 - val_loss: 0.1997 - val_accuracy: 0.5512\n",
      "Epoch 42/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4823 - accuracy: 0.5629\n",
      "Epoch 00042: val_loss improved from 0.19969 to 0.19740, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.4346 - accuracy: 0.5635 - val_loss: 0.1974 - val_accuracy: 0.6070\n",
      "Epoch 43/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4053 - accuracy: 0.5660\n",
      "Epoch 00043: val_loss did not improve from 0.19740\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4196 - accuracy: 0.5650 - val_loss: 0.2047 - val_accuracy: 0.5811\n",
      "Epoch 44/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4085 - accuracy: 0.5633\n",
      "Epoch 00044: val_loss improved from 0.19740 to 0.19344, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3769 - accuracy: 0.5648 - val_loss: 0.1934 - val_accuracy: 0.6056\n",
      "Epoch 45/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3926 - accuracy: 0.5715\n",
      "Epoch 00045: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4094 - accuracy: 0.5725 - val_loss: 0.1973 - val_accuracy: 0.6292\n",
      "Epoch 46/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4526 - accuracy: 0.5714\n",
      "Epoch 00046: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4491 - accuracy: 0.5692 - val_loss: 0.2032 - val_accuracy: 0.5877\n",
      "Epoch 47/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4399 - accuracy: 0.5612\n",
      "Epoch 00047: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4244 - accuracy: 0.5629 - val_loss: 0.2082 - val_accuracy: 0.5826\n",
      "Epoch 48/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3535 - accuracy: 0.5531\n",
      "Epoch 00048: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3810 - accuracy: 0.5564 - val_loss: 0.2007 - val_accuracy: 0.5860\n",
      "Epoch 49/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4474 - accuracy: 0.5744\n",
      "Epoch 00049: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4244 - accuracy: 0.5729 - val_loss: 0.2018 - val_accuracy: 0.6142\n",
      "Epoch 50/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4331 - accuracy: 0.5698\n",
      "Epoch 00050: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3964 - accuracy: 0.5708 - val_loss: 0.2117 - val_accuracy: 0.5713\n",
      "Epoch 51/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.4375 - accuracy: 0.5590\n",
      "Epoch 00051: val_loss did not improve from 0.19344\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.4104 - accuracy: 0.5542 - val_loss: 0.1970 - val_accuracy: 0.5239\n",
      "Epoch 52/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3465 - accuracy: 0.5698\n",
      "Epoch 00052: val_loss improved from 0.19344 to 0.19339, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 43us/sample - loss: 0.3565 - accuracy: 0.5704 - val_loss: 0.1934 - val_accuracy: 0.5800\n",
      "Epoch 53/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3460 - accuracy: 0.5809\n",
      "Epoch 00053: val_loss did not improve from 0.19339\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3292 - accuracy: 0.5742 - val_loss: 0.2126 - val_accuracy: 0.5383\n",
      "Epoch 54/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3690 - accuracy: 0.5706\n",
      "Epoch 00054: val_loss did not improve from 0.19339\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3473 - accuracy: 0.5724 - val_loss: 0.1953 - val_accuracy: 0.5827\n",
      "Epoch 55/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3640 - accuracy: 0.5865\n",
      "Epoch 00055: val_loss improved from 0.19339 to 0.19322, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3551 - accuracy: 0.5874 - val_loss: 0.1932 - val_accuracy: 0.5996\n",
      "Epoch 56/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3028 - accuracy: 0.5829\n",
      "Epoch 00056: val_loss improved from 0.19322 to 0.18657, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3362 - accuracy: 0.5803 - val_loss: 0.1866 - val_accuracy: 0.5701\n",
      "Epoch 57/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3461 - accuracy: 0.5822\n",
      "Epoch 00057: val_loss did not improve from 0.18657\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3675 - accuracy: 0.5834 - val_loss: 0.2223 - val_accuracy: 0.5930\n",
      "Epoch 58/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3918 - accuracy: 0.5705\n",
      "Epoch 00058: val_loss did not improve from 0.18657\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3850 - accuracy: 0.5724 - val_loss: 0.1889 - val_accuracy: 0.5544\n",
      "Epoch 59/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3570 - accuracy: 0.5695\n",
      "Epoch 00059: val_loss did not improve from 0.18657\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3521 - accuracy: 0.5738 - val_loss: 0.1867 - val_accuracy: 0.5846\n",
      "Epoch 60/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3153 - accuracy: 0.5809\n",
      "Epoch 00060: val_loss improved from 0.18657 to 0.18438, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3161 - accuracy: 0.5813 - val_loss: 0.1844 - val_accuracy: 0.5684\n",
      "Epoch 61/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3867 - accuracy: 0.5760\n",
      "Epoch 00061: val_loss did not improve from 0.18438\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3822 - accuracy: 0.5768 - val_loss: 0.2059 - val_accuracy: 0.5679\n",
      "Epoch 62/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3771 - accuracy: 0.5771\n",
      "Epoch 00062: val_loss did not improve from 0.18438\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3615 - accuracy: 0.5821 - val_loss: 0.1877 - val_accuracy: 0.5977\n",
      "Epoch 63/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3799 - accuracy: 0.5864\n",
      "Epoch 00063: val_loss improved from 0.18438 to 0.17863, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3457 - accuracy: 0.5866 - val_loss: 0.1786 - val_accuracy: 0.6204\n",
      "Epoch 64/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2927 - accuracy: 0.5594\n",
      "Epoch 00064: val_loss did not improve from 0.17863\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3236 - accuracy: 0.5566 - val_loss: 0.2134 - val_accuracy: 0.5805\n",
      "Epoch 65/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3817 - accuracy: 0.5679\n",
      "Epoch 00065: val_loss did not improve from 0.17863\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3894 - accuracy: 0.5640 - val_loss: 0.1907 - val_accuracy: 0.5738\n",
      "Epoch 66/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3451 - accuracy: 0.5795\n",
      "Epoch 00066: val_loss did not improve from 0.17863\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3358 - accuracy: 0.5810 - val_loss: 0.1984 - val_accuracy: 0.6011\n",
      "Epoch 67/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3195 - accuracy: 0.5997\n",
      "Epoch 00067: val_loss did not improve from 0.17863\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3278 - accuracy: 0.5964 - val_loss: 0.1793 - val_accuracy: 0.6222\n",
      "Epoch 68/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3389 - accuracy: 0.5902\n",
      "Epoch 00068: val_loss did not improve from 0.17863\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3299 - accuracy: 0.5930 - val_loss: 0.1846 - val_accuracy: 0.6183\n",
      "Epoch 69/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3028 - accuracy: 0.6094\n",
      "Epoch 00069: val_loss improved from 0.17863 to 0.17520, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.2971 - accuracy: 0.6110 - val_loss: 0.1752 - val_accuracy: 0.6188\n",
      "Epoch 70/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3301 - accuracy: 0.5879\n",
      "Epoch 00070: val_loss did not improve from 0.17520\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3450 - accuracy: 0.5894 - val_loss: 0.1912 - val_accuracy: 0.6262\n",
      "Epoch 71/100\n",
      "3712/4162 [=========================>....] - ETA: 0s - loss: 0.3156 - accuracy: 0.6085\n",
      "Epoch 00071: val_loss improved from 0.17520 to 0.17517, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3118 - accuracy: 0.6085 - val_loss: 0.1752 - val_accuracy: 0.6154\n",
      "Epoch 72/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3110 - accuracy: 0.5960\n",
      "Epoch 00072: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3044 - accuracy: 0.5971 - val_loss: 0.1823 - val_accuracy: 0.6096\n",
      "Epoch 73/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.3456 - accuracy: 0.5905\n",
      "Epoch 00073: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3383 - accuracy: 0.5903 - val_loss: 0.2002 - val_accuracy: 0.5883\n",
      "Epoch 74/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3536 - accuracy: 0.5825\n",
      "Epoch 00074: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3494 - accuracy: 0.5840 - val_loss: 0.1846 - val_accuracy: 0.6151\n",
      "Epoch 75/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3286 - accuracy: 0.5959\n",
      "Epoch 00075: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3309 - accuracy: 0.5976 - val_loss: 0.1873 - val_accuracy: 0.6282\n",
      "Epoch 76/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3220 - accuracy: 0.5990\n",
      "Epoch 00076: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3038 - accuracy: 0.5978 - val_loss: 0.1826 - val_accuracy: 0.5868\n",
      "Epoch 77/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3087 - accuracy: 0.5750\n",
      "Epoch 00077: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3265 - accuracy: 0.5720 - val_loss: 0.1842 - val_accuracy: 0.5985\n",
      "Epoch 78/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3497 - accuracy: 0.5911\n",
      "Epoch 00078: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3448 - accuracy: 0.5852 - val_loss: 0.1774 - val_accuracy: 0.6030\n",
      "Epoch 79/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.3742 - accuracy: 0.5898\n",
      "Epoch 00079: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3677 - accuracy: 0.5927 - val_loss: 0.2196 - val_accuracy: 0.5938\n",
      "Epoch 80/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3331 - accuracy: 0.5941\n",
      "Epoch 00080: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3299 - accuracy: 0.5934 - val_loss: 0.1815 - val_accuracy: 0.6335\n",
      "Epoch 81/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3289 - accuracy: 0.5907\n",
      "Epoch 00081: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3085 - accuracy: 0.5919 - val_loss: 0.1859 - val_accuracy: 0.5883\n",
      "Epoch 82/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2882 - accuracy: 0.6073\n",
      "Epoch 00082: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3195 - accuracy: 0.6073 - val_loss: 0.1828 - val_accuracy: 0.6340\n",
      "Epoch 83/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3298 - accuracy: 0.6096\n",
      "Epoch 00083: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3423 - accuracy: 0.6121 - val_loss: 0.1773 - val_accuracy: 0.6515\n",
      "Epoch 84/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3160 - accuracy: 0.6111\n",
      "Epoch 00084: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3130 - accuracy: 0.6116 - val_loss: 0.1806 - val_accuracy: 0.5935\n",
      "Epoch 85/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3698 - accuracy: 0.6005\n",
      "Epoch 00085: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3449 - accuracy: 0.6011 - val_loss: 0.1766 - val_accuracy: 0.6425\n",
      "Epoch 86/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3554 - accuracy: 0.6114\n",
      "Epoch 00086: val_loss did not improve from 0.17517\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3364 - accuracy: 0.6138 - val_loss: 0.1786 - val_accuracy: 0.6565\n",
      "Epoch 87/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.3096 - accuracy: 0.6056\n",
      "Epoch 00087: val_loss improved from 0.17517 to 0.17282, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.3193 - accuracy: 0.6090 - val_loss: 0.1728 - val_accuracy: 0.6253\n",
      "Epoch 88/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2808 - accuracy: 0.6040\n",
      "Epoch 00088: val_loss did not improve from 0.17282\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.2851 - accuracy: 0.6057 - val_loss: 0.1749 - val_accuracy: 0.6250\n",
      "Epoch 89/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3050 - accuracy: 0.6066\n",
      "Epoch 00089: val_loss did not improve from 0.17282\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3139 - accuracy: 0.6100 - val_loss: 0.1785 - val_accuracy: 0.6405\n",
      "Epoch 90/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2548 - accuracy: 0.6167\n",
      "Epoch 00090: val_loss did not improve from 0.17282\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.2812 - accuracy: 0.6093 - val_loss: 0.1835 - val_accuracy: 0.5926\n",
      "Epoch 91/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2819 - accuracy: 0.6122\n",
      "Epoch 00091: val_loss improved from 0.17282 to 0.16920, saving model to ./data/best_model_ae-conv1d.h5\n",
      "4162/4162 [==============================] - 0s 42us/sample - loss: 0.2950 - accuracy: 0.6104 - val_loss: 0.1692 - val_accuracy: 0.6592\n",
      "Epoch 92/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3227 - accuracy: 0.6028\n",
      "Epoch 00092: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3156 - accuracy: 0.6041 - val_loss: 0.1779 - val_accuracy: 0.6257\n",
      "Epoch 93/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3296 - accuracy: 0.6145\n",
      "Epoch 00093: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3043 - accuracy: 0.6161 - val_loss: 0.1829 - val_accuracy: 0.6474\n",
      "Epoch 94/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3350 - accuracy: 0.6180\n",
      "Epoch 00094: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3284 - accuracy: 0.6143 - val_loss: 0.1763 - val_accuracy: 0.5940\n",
      "Epoch 95/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3168 - accuracy: 0.6075\n",
      "Epoch 00095: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3049 - accuracy: 0.6072 - val_loss: 0.1838 - val_accuracy: 0.6339\n",
      "Epoch 96/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2928 - accuracy: 0.6163\n",
      "Epoch 00096: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3105 - accuracy: 0.6159 - val_loss: 0.1893 - val_accuracy: 0.6479\n",
      "Epoch 97/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.3145 - accuracy: 0.6165\n",
      "Epoch 00097: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3099 - accuracy: 0.6215 - val_loss: 0.1754 - val_accuracy: 0.6282\n",
      "Epoch 98/100\n",
      "3584/4162 [========================>.....] - ETA: 0s - loss: 0.2687 - accuracy: 0.6088\n",
      "Epoch 00098: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3054 - accuracy: 0.6092 - val_loss: 0.1746 - val_accuracy: 0.6238\n",
      "Epoch 99/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.3166 - accuracy: 0.6157\n",
      "Epoch 00099: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.3115 - accuracy: 0.6118 - val_loss: 0.1794 - val_accuracy: 0.5957\n",
      "Epoch 100/100\n",
      "3456/4162 [=======================>......] - ETA: 0s - loss: 0.2899 - accuracy: 0.6143\n",
      "Epoch 00100: val_loss did not improve from 0.16920\n",
      "4162/4162 [==============================] - 0s 35us/sample - loss: 0.2947 - accuracy: 0.6106 - val_loss: 0.1781 - val_accuracy: 0.6080\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "\n",
    "# number of training epochs - feel free to change\n",
    "#\n",
    "#       100 epochs ~= 3 minutes for Conv 1D\n",
    "\n",
    "epochs_num = 100\n",
    "history2 = autoencoder2.fit(x_train, x_train, batch_size=128, epochs=epochs_num, validation_split=0.1, verbose=1, callbacks=callbacks_list2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./data/best_model_ae-conv1d.h5\n"
     ]
    }
   ],
   "source": [
    "# load weights from best model\n",
    "print(data_dir+\"best_model_ae-conv1d.h5\")\n",
    "autoencoder2.load_weights(data_dir+\"best_model_ae-conv1d.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3gV1dbA4d9KQiChl9BC7x0EpCgCUqSIKFawV8CL13btBSOKol5FFLxgwYIfRSnSghQBEZASEKRL6KG3hBICKev7Y05CykmDHA5w1vs85yEzs2fPmgCzzuw9e4+oKsYYY3yXn7cDMMYY412WCIwxxsdZIjDGGB9nicAYY3ycJQJjjPFxlgiMMcbHWSIwxlwyIrJQRB73dhwmLUsEPkxEnhKRCBE5KyLfpdvWXkSSROSU6xMlIj+JyLU5qLeFiISLSLSIHBORFSLyiMdOxH0MD4vI4kt5TGOuVJYIfNs+4F1gdGbbVbUQUBhoBWwG/hCRjplVKCKtgfnA70ANoCTwJNAtD+O+6olIgLdjML7DEoEPU9XJqvoLcDSbcqqqUao6EPga+CCL4h8B36vqB6p6xLXvKlW9O7mAiDwhIpGuu4VpIlI+1TYVkf4islVEjovICHHkd91hNEhVNkREzohI6dyct4hcJyIrRSTG9ed1qbY9LCLbReSkiOwQkftc62uIyO+ufY6IyIQs6v9ZRA64yi4SkfqptgWJyMcissu1fbFrXRXXuT8mIruB+SLiJyJvuMoeEpEfRKSoq54CIvKjiBx1/V5WikiZrM7BTZx+IvKKiGxz1fOTiJRwbUuOp6+I7BOR/SLyn1T75heRT13b9rl+zp9q+60iskZETrjq75rq0JVFZIkrvjkiUiq7czIepqr28fEPzl3Bd+nWtQei3JTtACQBBd1sCwYSgRuzOFYH4AjQFMgPfA4sSrVdgRlAMaAScBjo6to2GhicquwA4NdMjvMwsNjN+hLAceABIADo41ouCRQETgC1XWXLAfVdP48DXsf58lQAaJPFOT6KcxeVH/gUWJNq2whgIRAK+APXucpVcZ37D644glz1RALVgELAZGCMq55+wHTX79wfaAYUyeoc3MT5LLAMqOCKYRQwzrUtOZ5xrjobuv4uOrm2D3LtWxoIAZYC77i2tQBigM6u31coUMe1bSGwDajlOseFwJCszsnb/z984eP1AOzj/Q+5SwR1XBeIUDfbQl3b6mRxrG+AD1MtFwLigSquZU19kQV+Al5x/dwJ2J5q2xLgwUyO8zDuE8EDwIp06/50lS8IRAN3AEHpyvwAfAlUyOXvtpjrnIq6LopngMZuyiVfeKulWvcb8K9Uy7Vdv6sAnCSxFGiUrp5Mz8HNMTcBHVMtl0tVf3I8dVJt/xD4xvXzNqB7qm1dgJ2un0cBQzM55kLgjVTL/8KVzDM7J/t4/mNNQya3ki/20SLyWqrO5JE436yTcC4omSkP7EpeUNVTOE1ToanKHEj1cyxOsgCn7yFIRFqKSGWgCTAll/GnOb7LLpzEdhq4B+gP7BeRmSJSx1XmJUCAFSKyQUQedVe5iPiLyBBXc8gJYKdrUynXpwDORTQze7KIdRfORboMMAaYDYx3Nc18KCL5sjmH9CoDU1zNMNE4iSHRVb+7eHa5YsostuRtFbM5x8z+ft2eUxb1mDxiicDkVi9gtaqeVtX3VLWQ69NfVWNxvl3fkcX++3AuQACISEGcZpm92R1YVZNw7hD6APcCM1T1ZC7jT3N8l0rJx1fV2araGSeZbQa+cq0/oKpPqGp5nCaML0Skhpv67wVuxbl7KYrzzRqcJHIEiAOqZ3WaWcRaCUgADqpqvKq+rar1cJqXegAPZnUObuwBuqlqsVSfAqqa+u+iYrrj78situRte7I5R7eyOifjWZYIfJiIBIhIAZz2WH9XZ12Gp1VcnbWhIvIW8DjwWhbVvgQ8LCIvikhJ1/6NRWS8a/tY4BERaeLqXHwPWK6qO3MY9licb7z3uX7O5hSlQOoPEA7UEpF7Xed/D1APmCEiZUSkpys5nQVO4XxDRkTuEpEKrnqP41ywE90cs7Br36M4bd3vJW9wJbLRwCciUt5199A6dSdrOuOA50SkqogUctU1QVUTRORGEWkoIv44fQLxQGJW5+DGSGCw6+4qufP91nRl3hSRYHE6vB8BkjvJxwFvuPYpBQwEfnRt+wbn77ijq0M6NIu7khSZnVN2+5k84O22Kft47wOE4VzQUn/CXNva4zTznAJO43zbmwi0ykG9LYBZOB2Gx4DlpGrLx2m22ObaNoNU7e6uGGqkWv4OeDdd/ZGufQOziOFhN+emOE0rbYBVrvhW4eqTwPkG/btrfTROe3Y917YPce4aTrli75vJcQsBU4GTOM0lD6Y+J5wO0k9ddcUAi1zrqiTHl6ouP5wL7B6cjtofgeKubX2ALa6/m4PAZ65zy/Qc3MTqBzzvquek67zec21Ljqev6+/+APBSqn0LuI653/X5DCiQansv4G9XvZFAF9f6hcDj6f6eFmd1Tt7+f+ILH3H9BRhjTAoRqQLsAPKpaoJ3ozGeZk1Dxhjj4ywRGGOMj7OmIWOM8XF2R2CMMT7uipvYqlSpUlqlShVvh2GMMVeUVatWHVHVEHfbrrhEUKVKFSIiIrwdhjHGXFFEJP2I+hTWNGSMMT7OEoExxvg4SwTGGOPjLBEYY4yPs0RgjDE+zhKBMcb4OEsExhjj4zyaCESkq4hsEedF5a+42V5cRKaIyN8iskJSvZg8r60/tJ7Xf3udo7FZvqfdGGN8jscSgevlEiOAbjgv/ugjIvXSFXsN58XejXDmbR/mqXi2Ht3Ke4vfY3fMbk8dwhhjrkievCNoAUSq6nZVPQeMx3mFX2r1cF7QjapuBqqISBk8IKSgM7L60OlDnqjeGGOuWJ5MBKGkffF1FGlfUA6wFrgdQERa4LwDtQIeEBLsJILDsYc9Ub0xxlyxPJkIxM269HNeDwGKi8ga4N/AXzgv505bkUhfEYkQkYjDhy/sQp58R3D4tCUCY4xJzZOTzkUBFVMtV8B592kKVT2B80JsRERwXo23I31Fqvol8CVA8+bNL+gFCsUKFCPAL8DuCIwxJh1P3hGsBGqKSFURCQR6A9NSFxCRYq5tAI8Di1zJIc/5iR+lgkvZHYExxqTjsTsCVU0QkaeA2YA/MFpVN4hIf9f2kUBd4AcRSQQ2Ao95Kh5w+gnsjsAYY9Ly6PsIVDUcCE+3bmSqn/8EanoyhtRCCloiMMaY9HxqZHFIcIg1DRljTDo+lwhsHIExxqTlW4mgYAgxZ2M4l3jO26EYY8xlw7cSgWtQ2ZHYI16OxBhjLh8+lQhKFywN2KAyY4xJzacSQcroYntyyBhjUvhWIgi2aSaMMSY930oEdkdgjDEZ+FQiKBFUAj/xszsCY4xJxacSgZ/4UTKopI0lMMaYVHwqEYBNM2GMMen5XCIoXbC0JQJjjEnF5xKBzTdkjDFp+WYisDsCY4xJ4XuJoGAIx84cIyEpwxsxjTHGJ/leInANKjsae9TLkRhjzOXB9xKBa1CZPUJqjDEO30sEwTa62BhjUvNoIhCRriKyRUQiReQVN9uLish0EVkrIhtE5BFPxgOpppmwJ4eMMQbwYCIQEX9gBNANqAf0EZF66YoNADaqamOgPfCxiAR6KiZINRW13REYYwzg2TuCFkCkqm5X1XPAeODWdGUUKCwiAhQCjgEefZynZFBJBLE7AmOMcfFkIggF9qRajnKtS204UBfYB6wDnlHVpPQViUhfEYkQkYjDhy/uAu7v50+JoBJ2R2CMMS6eTATiZp2mW+4CrAHKA02A4SJSJMNOql+qanNVbR4SEnLRgdl8Q8YYc54nE0EUUDHVcgWcb/6pPQJMVkcksAOo48GYAJtmwhhjUvNkIlgJ1BSRqq4O4N7AtHRldgMdAUSkDFAb2O7BmADnjsDGERhjjMNjiUBVE4CngNnAJuAnVd0gIv1FpL+r2DvAdSKyDvgNeFlVj3gqpmQ235AxxpwX4MnKVTUcCE+3bmSqn/cBN3kyBndKFyzN0dijJCYl4u/nf6kPb4wxlxWfG1kMzh2Bohw7c8zboRhjjNf5ZiKwl9gbY0wKn0wEZQqWAWDfyfQPMRljjO/xyURQp5TzhOrGwxu9HIkxxnifTyaCsoXKUjKoJOsPrfd2KMYY43U+mQhEhAalG1giMMYYfDQRACmJQDX9rBfGGONbfDoRnDx3kj0n9mRf2BhjrmI+mwgalm4IwLqD67wciTHGeJfPJoL6pesDWD+BMcbn+WwiKFagGBWKVGD9YUsExhjf5rOJALAnh4wxBl9PBCEN2HR4EwlJHn07pjHGXNZ8OhE0LNOQs4lniTwW6e1QjDHGa3w6ETQo3QCwDmNjjG/z6URQt1RdBLFEYIzxaT6dCILyBVGjRA3WHbKxBMYY3+XTiQCcfgK7IzDG+DKPJgIR6SoiW0QkUkRecbP9RRFZ4/qsF5FEESnhyZjSaxDSgMhjkZyJP3MpD2uMMZcNjyUCEfEHRgDdgHpAHxGpl7qMqn6kqk1UtQnwKvC7ql7S90c2KN2AJE1i85HNl/Kwxhhz2fDkHUELIFJVt6vqOWA8cGsW5fsA4zwYj1vNyzcHYPKmyZf60MYYc1nwZCIIBVJP7RnlWpeBiAQDXYFJmWzvKyIRIhJx+HDevme4avGq3FH3Dj5b8RnHzxzP07qNMeZK4MlEIG7WZTb5/y3AksyahVT1S1VtrqrNQ0JC8izAZAPbDeTE2RMMWz4sz+s2xpjLnScTQRRQMdVyBSCzt8X3xgvNQskalWlErzq9+HTZp0THRXsrDGOM8QpPJoKVQE0RqSoigTgX+2npC4lIUaAdMNWDsWTrzbZvEnM2hs+Wf+bNMIwx5pLzWCJQ1QTgKWA2sAn4SVU3iEh/EemfqmgvYI6qnvZULDlxTblr6Fm7J0OXDSUmLsaboRhjzCXl0XEEqhquqrVUtbqqDnatG6mqI1OV+U5Ve3syjpwa2HYg0XHRdldgjPEpPj+yOLVm5ZvRs3ZPPv7zY+srMMb4DEsE6QxqP4iYszF88ucn3g7FGGMuCUsE6TQu25g7693Jp8s+5WjsUW+HY4wxHmeJwI2wdmGcOneK/y79r7dDMcYYj7NE4Eb90vXp3aA3n6/4nEOnD3k7HGOM8ShLBJl4q91bnEk4w4dLPvR2KMYY41GWCDJRu1Rt7m90PyNWjmD/yf3eDscYYzzGEkEW3mz7JvGJ8QxZPMTboRhjjMdYIshCjRI1eLjJw4xaNYqoE1HeDscYYzzCEkE23mj7BkmaxHt/vOftUIwxxiMsEWSjSrEqPHbNY3y9+mt2Re/ydjjGGJPnLBHkwGs3vIaI8O6id70dijHG5DlLBDlQsWhF+jbty3drv2PH8R3eDscYY/KUJYIcevWGV/EXf7srMMZcdSwR5FD5wuXp16wf36/9nm3Htnk7HGOMyTOWCHLhlTavkM8/H+/+YXcFxpirhyWCXChXuBz9m/VnzNoxRB6L9HY4xhiTJywR5NLLbV4m0D+QVl+3ou/0vszZNoeEpARvh2WMMRfMo4lARLqKyBYRiRSRVzIp015E1ojIBhH53ZPx5IWyhcoy54E5dK7emXHrx9Hlxy70n9E/+x2NMeYyJarqmYpF/IF/gM5AFLAS6KOqG1OVKQYsBbqq6m4RKa2qWc773Lx5c42IiPBIzLl1Jv4Mj09/nF82/8KRF48QlC/I2yEZY4xbIrJKVZu72+bJO4IWQKSqblfVc8B44NZ0Ze4FJqvqboDsksDlJihfEA83fpjY+Fjmbp/r7XCMMeaCeDIRhAJ7Ui1HudalVgsoLiILRWSViDzoriIR6SsiESIScfjwYQ+Fe2HaVWlH0fxFmbJ5irdDMcaYC+LJRCBu1qVvhwoAmgE3A12AN0WkVoadVL9U1eaq2jwkJCTvI70Igf6B9KjVg+lbplunsTHmiuTJRBAFVEy1XAHY56bMr6p6WlWPAIuAxh6MySN61enF0TNHWbx7sbdDMcaYXPNkIlgJ1BSRqiISCPQGpqUrMxW4QUQCRCQYaAls8mBMHtG1RlcKBBRgyiZrHjLGXHk8lghUNQF4CpiNc3H/SVU3iEh/EenvKrMJ+BX4G1gBfK2q6z0Vk6cUDCzITdVv4pctv+Cpp7CMMcZTAjxZuaqGA+Hp1o1Mt/wR8JEn47gUetXpxbQt01i9fzXNyjfzdjjGGJNjNrI4j9xS6xb8xd+eHjLGXHEsEeSRksEl6VStE58t/4yIfZfHgDdjjMmJHCUCESkoIn6un2uJSE8RyefZ0K483/T8hpLBJen6Y1c2Ht6Y/Q7GGHMZyOkdwSKggIiEAr8BjwDfeSqoK1VokVDmPTCPQP9AOv3Qie3Ht3s7JGOMyVZOE4GoaixwO/C5qvYC6nkurCtX9RLVmfvAXM4lnqPzmM4cPn15jYQ2xpj0cpwIRKQ1cB8w07XOo08cXcnql67PzHtnsu/kPm6bcBtxCXHeDskYYzKV00TwLPAqMMU1FqAasMBzYV35WlZoyZheY1i6ZymPTn3UxhcYYy5bOUoEqvq7qvZU1Q9cncZHVPVpD8d2xbuz3p283/F9xq0fx9u/v+3tcIwxxq2cPjU0VkSKiEhBYCOwRURe9GxoV4eXr3+ZPg368P7i9zlx9oS3wzHGmAxy2jRUT1VPALfhjBSuBDzgsaiuIiLCk82f5FziOeZsm+PtcIwxJoOcJoJ8rnEDtwFTVTWejFNKm0y0rtiakkElmbplqrdDMcaYDHKaCEYBO4GCwCIRqQxYO0cOBfgF0KNWD2b+M9PeWWCMuezktLP4M1UNVdXu6tgF3Ojh2K4qPWv35HjccXtngTHmspPTzuKiIvJJ8usiReRjnLsDk0M3Vb+J/P75mbYl/SsZjDHGu3LaNDQaOAnc7fqcAL71VFBXo0KBhehYrSNTt0y1MQXGmMtKThNBdVV9S1W3uz5vA9U8GdjVqGetnmw/vj3XE9KdSzyXJnnsO7mPYcuG0f3/urNi74q8DtMY42NyOk3EGRFpo6qLAUTkeuCM58K6Ot1S+xb6z+zP1C1TqV+6fo72mbp5Kn0m9SFJkyhfuDxFCxRl7YG1KIoglC1UlhahLTwcuTHmapbTO4L+wAgR2SkiO4HhQL/sdhKRriKyRUQiReQVN9vbi0iMiKxxfQbmKvorTPnC5bm2/LU5fox01b5V3Dv5XuqUqsPTLZ+mdcXWhASHENY+jM0DNnN3/buZFTmLJE3ycOTGmKtZju4IVHUt0FhEiriWT4jIszjvGnZLRPyBEUBnIApYKSLTVDV9u8gfqtrjgqK/At1R9w5e+e0V/j74N43KNMq03J6YPdwy7hZCgkMIvy+csoXKZijTrUY3JmyYwNoDa7mm3DWeDNsYcxXL1RvKVPWEa4QxwPPZFG8BRLr6FM4B44FbLyDGq0rfZn0pFFiIIYuHZFrm5NmT9BjXg9Pxp5l570y3SQCga42uAIRvDXe73RhjcuJiXlUp2WwPBfakWo5yrUuvtYisFZFZIpKzhvMrWPGg4jzZ/EkmbJhA5LHIDNtVlX4z+rH+0Hp+vuvnLPsSyhQqQ7NyzZgVOcuTIRtjrnIXkwiyewbSXaJIv89qoLKqNgY+B35xW5FI3+QxDIcPX/kvenmu1XPk88vHR0s+yrDty1VfMm79ON658R1uqn5TtnV1r9mdP6P+5NiZY54I1WvWHljLqn2rvB2GMT4hy0QgIidF5ISbz0mgfDZ1RwEVUy1XAPalLuBqajrl+jkcZ06jUukrUtUvVbW5qjYPCQnJyXld1soVLsej1zzKd2u/Y9/J87+Sv/b/xTO/PkPXGl15pU2GvnW3utXoRpImMXfb3DyPc/z68Vw/+noOnT6U53Vnp//M/vSd0feSH9cYX5RlIlDVwqpaxM2nsKpm19G8EqgpIlVFJBDoDaQZVisiZUVEXD+3cMVz9MJP58rx4nUvkpiUyJDFQ9h4eCPhW8O56+e7KBVcijG9xuAnObtZaxHaghJBJQiPzNt+ggOnDtB/Rn+W7llK74m9L+kcSUmaxN8H/ybyWKQNvjPmEriYpqEsqWoC8BQwG9gE/OR6u1l/EenvKnYnsF5E1gKfAb3VR/7nVy1elT4N+/D5is+p/0V9bh57M1Enohh/53hKBWe4KcqUv58/Xap3YdbW7B8j/XDJh7zz+zs5urg+P/t5ziSc4c22b7Jg5wLemP9GjmO6WNuObSM2PpYTZ09cdU1exlyOPPreYVdzT3i6dSNT/TwcZ0yCT/qw04dcU/YayhUqR5ViVahVshYlg0vmup7uNbszbv04RkaM5GjsUSL2R/BQ44e4ve7tKWX2ntjL6/NfJyEpgURNJKx9WKb1zdk2h3Hrx/FWu7cIax/GodOH+GDJB7QMbUmvur0u5FRz5e+D559K3n58+wX9TowxOWcvoPeicoXL8Xzr7J7CzV6X6l3wEz8GhA9AEAoGFmTF3hV0rdGV4HzBAAxfMZwkTeLW2rfy9u9vUyq4FE+1eCpDXWfiz/Cvmf+iZomaKf0Uw7oOY/X+1Tz0y0O0rNCS8oWz6x66OOkTwbWh13r0eMb4Oo81DV2Wzp2DpKtvFG5IwRDm3D+HOffP4fjLx5l13ywOnDrA8BXOzdbpc6cZtWoUver0YuLdE+lZuydPz3qaCesnZKjr8xWfs+34Nkb2GEmBgAIA5A/Iz7g7xhGXEMeg3wd5/HzWHlxLpaKVACcRGGM8y3cSwYQJEBQEO3Z4OxKP6FitI52rd6ZogaK0qdSGbjW68cGSD4iJi+H7td9zPO44z7V6jgC/AMbfMZ5rQ6/lhbkvZOgv+GXzL7QIbUGHqh3SrK9eojr9m/fn69Vfs+XIFo+ey98H/6Z1hdaUKVjGEoExl4DvJILy5Z27gciMg7iuRu92eJdjZ47x8Z8f8+myT7m2/LVcV/E6AILyBdG3aV+iTkSx/tD6lH2OnznO8r3L6VK9i9s632j7BkH5gnh9/usei/vE2RPsiN5BozKNqFa8GtujLREY42m+kwhq1HD+9JFE0LRcU+6oeweD/xjM1mNbeb7187ie1AWgW81uQNrpKebvmE+SJmU6kK10wdK80PoFJm2axLKoZR6Je93BdQDnE4HdERjjcb6TCMqWheBgn0kEAINuHISqUqFIBe6oe0eabeULl6dJ2SZpxh/M2TaHIvmL0DK0ZaZ1Pt/6eUoXLM3L8172yDP+yR3Fjcs0plrxauyO2U18YnyeH8cYc57vJAIRqF7dpxJBvZB6/O/m//Fljy/J558vw/ZuNbqxZPcSouOiUVVmb5tNh6od3JZNVjh/YQa2HciiXYuYvW12nsf898G/KVagGBWKVKBa8WokaRK7Y3bn+XGMMef5TiIAp3nIhxIBQL/m/VKagdLrXrM7iZrIvO3ziDwWya6YXdxULfv5jZ5o9gRVilXhjflv5PldwdqDa2lUphEiQrXizkvwLnXz0JDFQ1i9f/UlPaYx3uR7iWD7dkhM9HYkl4VWFVpRrEAxwreGM2fbHIAcTXQX6B/IW+3eYtX+Vfyy2e08gRckSZNYd2gdjcs0BvBKIth2bBuv/vYqA8IH2PQWxmf4XiI4dw727vV2JJeFAL8Abqp+E7MiZ/Hrtl+pXrw61UtUz9G+9ze6n9ola/PmgjdJTMqYWGPiYnh+9vNsO7Ytx/HsOL6DU+dOpbywp3zh8gT6B7LteM7rAGcq79ST+eVG8pTey6KWsXj34guqw5grje8lAvC55qGsdK/RnQOnDhC+NTxHdwPJAvwCGHTjIDYc3sD49ePTbDt17hTdx3Zn6LKhDFyY87ePpu4oBvATP6oWq5qrOwJV5elZTxP6SSjtv2vPzH9m5upVnuFbw6larCqlgkvxwZIPcryfMVcySwQ+LvktZ1k9NpqZO+vdSeMyjXlr4VvsP7kfcKao6DmuJ8ujltO6Qmt+3vAzB08dzFF9fx/8G0HSvIwnt4+QvvbbawxfOZxba9/KtuPb6DGuB01GNmFX9K5s942Nj2XBzgXOyOsWTzNz68w04yyMuVr5ViKoUAHy57dEkEryW878xZ8bq9yYq339xI/3O77P9uPbKf9JeRqPbEy779qxcOdCvr/te7677Tvik+L5avVXOapv7cG11CxZM2V+JHASwbbj23LUXv/eH+8xZMkQ+jXrx5R7prD96e2M6TWGXTG7uHvi3ZxLPJfl/gt3LiQuIY7uNbszoMUACuYryIdLPsxR7MZcyXwrEfj5QdWqlgjSGdhuIINuHETRAkVzvW+3mt1Y9+Q6Puj0ASWDSrIzeidf3fIV9zW6j1ola9G5WmdGrRqV7fsMTpw9wfwd8zOMYahWvBonzp7geNzxTPc9l3iOl+e+zOvzX+e+hvfxxc1fICLk88/H/Y3u59tbv2XF3hW8MOeFLGMI3xpOcL5g2lZuS4mgEjzR9AnGrR9nj6+aq5+qXlGfZs2a6UXp0UO1UaOLq8Pk2NTNU5UwdNLGSVmWG/LHECUMjdgbkWb9lE1TlDB05d6VbvfbdHiTNh3VVAlD+07rq/GJ8W7LPffrc0oYOn7deLfbk5KStMqnVfSWsbekrNsdvVsDBgXoS3NeylC+z8Q+2nti7yzPyZjLCRChmVxXfeuOAM6PJbBHAy+Jm2veTOWilRmxckSmZeIS4hi6bCidq3WmWflmabZl9Qjpb9t/o+mopuyK3sWUe6Yw6pZRBPi5n1n9g04fcF3F63h8+uNu+wu2HN3CzuiddK/ZPWVdxaIVaV+lfcqTRMnOxJ9h8qbJTFg/wabAMFcF30wEsbFw4IC3I/EJ/n7+9G/en/k75rPx8Ea3Zb5b8x0HTx/k1TavZthWtVhVwH0i+GTZJ5QMLsnfT/7NbXVuyzKOfP75GNNrDKfOneLnjT9n2J4851K3GmkH33Ws2pF1h9Zx4NT5fy9L9yzlbOJZFGVUxKgsj2vMlcA3EwFYP8El9HjTx/EXf8atG5dhW0JSAh8t/YgWoS1oX6V9hu2F8xcmJDgkQyKIiYth3vZ53FP/nhy/KKda8WrULVWXudvnZtgWvjWc+iH1qVyscpr1nap1ApwJ+ZLN2z6PAL8AOmXmVZUAACAASURBVFfrzOg1ozmbcDZlW3RcdK7GThhzOfBoIhCRriKyRUQiReSVLMpdKyKJInKnJ+MBLBF4QangUjQu25hlezPOWPrzhp/Zfnw7r7Z5Nc3sqKm5e4Q0fGs45xLPpXkdZ050rtaZRbsWEZcQl7IuJi6GRbsWZbgbALim7DUUL1Cc37b/lrJu3o55tKrQiheve5EjsUeYuHEiACfPnuSGb2+g3XftbFSyuaJ4LBGIiD8wAugG1AP6iEi9TMp9gPOSe8+rXBkCAiwRXGKtQluxPGp5hlHIH//5MXVL1aVn7Z6Z7lsvpB4r960kOi46Zd3kzZMpW6gsrSq0ylUcnat3Ji4hjiW7l6Ssm7RpEvFJ8dxZL+P3EH8/fzpU7cDc7XNRVY6dOcaqfavoVLUTHat1pEaJGvwv4n8kaRIPTHmA9YfWs/fkXnbFZD9uwZjLhSfvCFoAkaq6XVXPAeOBW92U+zcwCTjkwVjOCwhwksE2u32/lFpVaMXJcyfZdGRTyroDpw6wav8qHmr8EH6S+T/Ff7f4NyfOnuDz5Z8DTmdt+NZwetXpleV+7rSr3I4Av4A0zUNj142levHqtAht4XafjlU7sufEHiKPRbJw50IUpWO1jviJH/2b9WfJniXcO+lepm6ZyiNNHgFgedTyXMVljDd5MhGEAntSLUe51qUQkVCgFzAyq4pEpK+IRIhIxOHDhy8+Mh+chdTbkr+5p36hTXJzS3I7fGauKXcNt9S6haHLhnLi7AnmbJtDbHxsrpuFwOlzaF2hdUoi2HdyH/N3zOe+hvdl2jSVHN+87fOYt30ehQILpYx3eLjJw+T3z8+EDRN4/JrHGdVjFPn987Ni74pcx2aMt3gyEbj7X5W+4fRT4GVVzXI6UFX9UlWbq2rzkJCQi4/MHiG95GqUqEGJoBJpEsG8HfMoEVSCJmWbZLv/wHYDOR53nOErhjN582SKFyhOu8rtLiiWztU689f+vzgSe4Tx68ejKPc2vDfL2CsVrcS8HU4iaFe5Xco7G0oGl+Sl61/ijrp3MOLmEeTzz0fTck1ZvtfuCMyVw5OJIAqomGq5ApB+SsjmwHgR2QncCXwhIlk/B5gXatSAmBg4etTjhzIOEaFVhVYpiUBVmbd9Hh2qdsDfzz/b/ZuXb073mt35+M+PmbZlGj1r98zyBTpZ6Vy9M4ry2/bfGLtuLM3KNaN2qdpZxt6paidmbZ3F1mNbM9zBDLpxEBPvnkigfyAALUNbsmr/qsv+zWpLdi/h46UfezsMcxnwZCJYCdQUkaoiEgj0BqalLqCqVVW1iqpWASYC/1LVvJvgPjPJTw5t2eLxQ5nzWoW2YuPhjcTExbD12FaiTkTRqWrWzUKpDWw7kGNnjhEdF31BzULJmpdvTtH8RRm+cjir9q/ivob3ZbtPp2qdOJNwJuXnrLSs0JK4hLg8m7AuSZMY/ddojsQeyZP6kn224jNemvcSp86dytN6zZXHY4lAVROAp3CeBtoE/KSqG0Skv4j099Rxc6RpU+fP5Xb7fim1qtAKRVmxdwXzts8Dsr+optayQku6VO9CocBCdK7W+YLjCPALoEPVDizevRg/8aN3g97Z7tOhagcAyhQsQ/2Q+lmWTe50zqvmoVERo3hs2mN8sDjrabEHLxrMhPUTclzvuoPrSNIkIvZFXGyI5grn0XEEqhquqrVUtbqqDnatG6mqGTqHVfVhVZ3oyXhSlC/vTD63ZEn2ZU2eaRHaAkFYFrWMedvnUblo5ZQpJHLqx9t/ZOmjSwnKF3RRsSQnkg5VO1CucLlsy5cpVIa2ldtyR907Mu1UTpb8PoO86DDeFb2Ll+a9BMD4DeMzfbdCYlIi7/7xLv1m9MvRncPZhLP8c/QfwJ5wMr44sjjZ9dc7icA6jC+ZogWKUi+kHkv2LGHBzgV0qtYp24tqeqWCS9GwTMOLjqVbzW4E+gfy2DWP5XifhQ8tZHj34dmWExFahLa46DsCVeWJ6U+gqgzuMJioE1Fpxj+ktv34duIS4og5G8PbC9/Otu7NRzaT6HpGwzq2jW8ngoMHnXcYm0umVYVWzN0+l+i46Fw1C+W1KsWqcPCFgzlqFkomIjlOXC1DW7Lp8CZOnD1xoSHy7Zpvmbt9Lh92/pCnWz5NUEAQY9eNdVt23aF1gPP7HblqJFuOZN3/lVz+mrLXWCIwPpwI2rRx/lxs76W9lFpVaJXSvJHc7u4txQoU81jdLUNbougFt78fiT3C87Ofp13ldvRv3p9CgYXoWbsnP2/82e3TSOsPrUcQxt4+lqCAoJTmpMysO7iOfH75eKDRA+w7uY+oE1EXFOelMmbtGG4ee7NN3eEhvpsI6tWDYsWsn+ASSx5Y1qhMI0oXLO3laDzn2tBrgQtvfx+7biwxZ2P4rNtnKaOn7214L0fPHHU7ad66Q+uoVrwaVYtX5dU2rzJtyzQW7lyYaf3rDq2jbkhdrq90/UXFeSnExsfy4twXCd8azl8H/vJ2OFcl300Efn7QurUlgkusbqm6lC9cnltru5tt5OpRIqgENUvUvOBmlx/W/kDTck1pVKZRyrou1btQrEAxxq3POIvruoPraFC6AQDPtnqWikUq8q+Z/yI2PtZt/esOraNh6YY0LtOYQP/Ay7p5aGTESA6edt57PW3LtGxKmwvhu4kAnOahjRvh2DFvR+Iz/P382TxgMwPbDfR2KB7XskJLlkUty/RJH4CoE1F8sfKLNO9T3nBoA6v2r+LBRg+mKZs/ID931r2TKZumpLnAxyXEsfXYVhqWdjrRg/IFMfrW0Ww+spnnfn0uwzGj46KJOhFFw9INyR+Q/7LuJ4iNj+WDJR/QsWpH2lRqY4nAQ3w7EVzv3BazdKl34/AxhfMXzvRNYleTW2rdwsHTB/k18tcM21SVMWvH0OCLBgwIH5BmhO+Yv8fgL/70adgnw359GvbhdPxppm+ZnrJu0+FNJGlSmqepOlXrxEvXv8SXq79MmSY7WfJAt+Q7iJahLYnYF5Hte6W9YWTESA6dPkRY+zB61urJXwf+Yk/Mnux3NLni24ng2mud2UiTm4eiouCuu2D1au/GZa4Kver0olyhcgxfkfaR0+i4aG7/6XYe/OVBGpZpyE3Vb+KdRe+wM3oniUmJ/Pj3j3Sr2c1tH0q7yu0oU7AMkzZNSlmX/sKe7J0b36FFaAuemP5EmtdzrjvoPDGUnDhaVmhJbHwsGw5tyJsTd2NPzB6afdmMgQtyfieYfDfQqVon2lRqkzJV+fR/pmezp8kt304EwcHOKOPFi2H3bmjXDiZOhH/9y8YXmIuWzz8f/Zr1Y1bkLCKPnZ/tdkD4AGb8M4OPOn/EwocW8vUtX+Mnfjzz6zMs2LmAvSf3ZmgWSubv589tdW4jfGs4Z+KdKS/WHVpHoH8gNUvUzHD8sbePJTEpkUenPZryxM26Q+somr8oFYs4U4Elz6Sam+aho7FH6TG2R47e2bzj+A7afteW1ftXM/iPwTl+kmrEihEcOn2It9q9BUDtUrWpVbJWjhNB6jfHpV738C8PM379+BzV4St8OxGA00+wcqWTBI4ehWeecaaemDQp+32NyUbfZn0J8Avgi5VfAPBr5K+MXTeW1294nReuewF/P38qFq3IwHYDmbZlGk+FP0XR/EW5pfYtmdZ5e93bOR1/OuXpofWH1lOnVB23k/BVL1GdwR0GM3/H/JRpPdYdcjqWk8dEVCtejZJBJXP15NCsyFnM3DqTkRFZziDPP0f/4YZvbyAmLobfHvyNMgXL0G9Gv2ybofbE7GHQokF0r9mdNpXapKzvWasn83fM5+TZk1nuv3j3YooMKcLi3WkfD5/xzwy+X/s9fSb14bGpj3H63OlsztQ3WCK4/no4exaio2HePPj4Y6hfH157DeIv79kjzeWvXOFy3FnvTkb/NZpDpw/x5MwnqVOqDq+2eTVNuWdbPUvdUnXZcnQLd9e/mwIBBTKts32V9hQrUIzJmyYD558AykzfZn2pVLQSr89/HVVl/aH1aZqRkkdCL9u7LMfP6S/atQiACRsmZNoZHp8YT6cfOnEu8RwLH15Ih6od+LTrp6zev5oRK0ZkWreqMiB8AEmaxPBuaZvVbql9C+cSzzFn25ws9399/uucSzzHiJVpj/Pjuh8pW6gsr7V5jW/XfMu1X13L1qNbc3TOVzNLBDfdBP36wfz50Lw5+PvDkCGwdSt89ZW3ozNXgaeufYqYszG0+64dO6N38tUtX5E/IH+aMoH+gYzsMZJCgYXo26xvlvUF+gdyS61bmP7PdA6fPkzUiagM/QOp5Q/Iz8C2A1m5byX/i/gf0XHRGRJHx6od2Xh4I70n9ebYmeyfovtj9x8E5wtmd8xu/tzzp9sy87bPY8+JPXx5y5cpj8HeVe8uutboyhsL3sh0ENvkTZOZ/s903m7/NlWLV02z7bqK11EiqATT/sn86aEFOxewaNciKhapyORNk1PmXjoae5SZ/8zk3gb3MrjjYOY+MJdDpw9x98S7L/spwz1OVa+oT7NmzdTjkpJU27ZVLV1a9eRJzx/PXNWSkpK0ycgmShjab3q/bMvmxJRNU5QwNGxBmBKGztgyI8vy8YnxWvOzmhr0bpAShv6+8/cM2wcvGqwBgwK0/MfldfTq0frfJf/Vh6Y8pLeNv01PxJ1IKXvw1EElDH3jtze0wLsFdMDMAW6P+dCUh7To+0U1Lj4uzfptx7ZpgXcL6H2T7suwT/SZaC3333LaZGQTjU+Md1vvA5Mf0JIflMxQr6rz+2szuo2GfhyqK/euVMLQoX8OVVXV/638nxKGrt63OqV88u/xrQVvuf/FXaRTZ0/pvG3zPFJ3bgERmsl11esX9tx+LkkiUFVdtsz59bzzzqU5nrmqTd44WVt/3VqPnzmeJ/WdPndagwcHa7EhxZQwdFf0rmz3Gfv3WCUMJQw9GnvUbZlV+1ZpvRH1UsqV/qi0EoaOXj06pcykjZOUMHTp7qV65093aumPSme4aMfFx2nR94vqQ1MecnucZ2c9qwGDAnTfiX1p1j8d/rT6ve2nK/euzPQ85m6bq4Shw5YNy3Tb8OXDVVW15Vcttd6IepqUlKTXfXOd1h9RP0OyvX/y/RowKEBX7VuV6TEv1LOznlXC0D92/ZHndeeWJYIL1aOHaokSdldgLkt3TLhDCUOLvF8kR3cSiUmJ2vCLhlrxk4pZljsTf0aX7Vmmh08f1qSkJK0+rLp2+qFTyvZnZj2jQe8G6dmEsylJYU7knDR1TN08VQlDw/8Jd3uMrUe3Zvgmvit6l+YblE/7TuubZXxJSUna4fsOWurDUhoTF5Nm/XXfXKcVPqmQcrfw9aqvlTD0x7U/KmHo+3+8n6G+Y7HHtNx/y2mDLxq4vcu4UIdOHUq5A7tpzE15Vu+FyioRWB9BVl5/3Rl1PDLrJyOM8Ybkt7SlfgIoK37ix7Q+0/ild9YvASwQUICWFVpSKrgUIsK9De9l/o75HDh1AHD6B1pWaEmgfyDda3anSP4iGaa9+GnDTxQvUDzTGWZrlKhBtxrdGLVqVMqo6sGLBiMivNH2jSzjExGGdBzCkdgjaQbijVo1iqV7lvJam9dS+mDuaXAPhQIL0W9GPwC3b6MrHlScr275ivWH1vPJn59keezcGLZ8GHEJcTza5FHmbJuTJ++n8BRLBFlp1Qo6dYL//hfOnPF2NMakcXPNmwn0D6RxmcY53qdKsSo0Ldc0V8fp06APSZrETxt+4sTZE6w5sIa2ldoCTtLoVacXkzdNTnlu/0z8GaZumcrtdW/P8r3S/27xbw6cOsCkjZPYGb2T0WtG80TTJ6hYtGKm+yS7NvRa7qp3Fx//+TEHTx1k2LJhPDnzSbrV6MZjTc+/Y6JQYCF61+/N6fjTtK/SPtO6b651M+2rtOf7td87TSWpLNy5MNtHa4f+OZSWX7dM6QCPiYth+Irh3F73dj7t+iklgkrwzqJ3sj0vb7FEkJ3XX3feW/DNN96OxJg0ihYoyu8P/+7xeZvqhtSlSdkmjF03lqV7lpKkSdxQ+YaU7X0a9CHmbAyvzHuFuIQ4ZkXO4tS5U9xT/54s6+1Sows1StRg+MrhvLvoXfzFP8NjtVl5t8O7xCXE0fGHjjw7+1lur3s7U+6ZQqB/YJpy/Zr3QxAeafJIlvXdU/8ethzdkvKuBnDmcbrjpzt4YMoDGRJEsj0xe3ht/mus2LuCdt+1Y3fMbv4X8T9izsbwaptXKZy/MM+1eo4Z/8zgr/3O7Kmr9q3iP7P/w4tzXuTthW8zbNkwth3bllJnkibx84afuX709UzZNCXHv5MLJZmdXJ5ULtIVGAb4A1+r6pB0228F3gGSgATgWVXN8gUBzZs314iIS/iOVVW44QZn5HFkJAQGZr+PMVeZj5Z8xEvzXuLu+nczedNkol+OpmBgQcB5TWbf6X0ZvWY0tUrWomRQSSKPRbLvP/uynVPq02Wf8tzs5/ATP5669imGdRuWq7ienPEkI1eN5L6G9/Hdbd9lerxtx7ZRrXi1LJvQDp0+RLmPy/Fqm1d5t8O7gDMd+H2TneaklU+spHn55hn2e3DKg/y04SfG9BrDE9OfoHhQcU6fO03Tck359X5nnqmYuBgqf1qZJmWbEJwvmFmRs8jvnx8/8eNMwvnWhhur3MittW/lh79/YPX+1eTzy0fBwIKsf3I9oUVCc/W7SU9EVqlqxhMAz3UW41z8twHVgEBgLVAvXZlCnE9GjYDN2dV7STuLk82a5fSrf/HFpT+2MZeB3dG7U54kavFVC7dl5kTO0SqfVlHC0P7T++eo3ugz0VpwcEEt8G6BDE8Q5cTJsyd14oaJmpCYkOt93en4fUet9XmtlM73dt+204qfVNR8g/Lp878+n6F8xN4IJQx9ee7LKcvFhxRXwtBFOxelKfvm/DeVMLTUh6X0vUXvafSZaFVVTUhM0F3Ru3TwosFabVg1JQyt8mkV/X7N97r58GYNHhysXX/smuNHizODN54aAloDs1Mtvwq8mk35TdnV65VEkJSk2r69aqFCqpGRl/74xlwG2n7bVglDX5j9QqZlTp09pcOXD8/VRX306tFpHk/1plERo5QwdM3+Nbr58OaUJ41uHXerlv+4fJqEk5SUpO2+bachH4akXNRVVdcfXK9fr/o6Q91n4s/opI2T9NTZU5kePzEpUdcfXJ/m6aXhy4crYejIlSMv6ty8lQjuxGkOSl5+ABjuplwvYDNwDGidSV19gQggolKlShf1y7hgu3apFium2qKF6rlz3onBGC8auXKkEoZO3TzV26F4zKFTh9T/bX99bd5r+p/Z/9GAQQG6/+R+nbB+ghKGzt8+P6Vs8mC0L1Z4tqUgMSlRO/3QSQsOLqiRRy/8i2hWicCTncXuGuMydEio6hRVrQPchtNfkHEn1S9VtbmqNg8JCcnjMHOoUiX48ktYsQLefts7MRjjRQ83eZhvb/2Wm2ve7O1QPCakYAgdqnZg/IbxfLfmO26tfStlC5WlR60eFAosxNh1YwFnRtUnpj9B/ZD6PNHsCY/G5Cd+jO45mgC/gDx9vDXNMTxSqyMKSP2sVgVgX2aFVXURUF1ESnkwpotz113w6KPw3nvw++/ejsaYSyp/QH4ebvIw/n7+3g7Fo+6ufzfbj2/n6JmjKfM+BecLpledXkzcNJEjsUfoOb4nCUkJTL5n8iV5yVLFohVZ9MiiXHem55QnE8FKoKaIVBWRQKA3kGamKBGpIa5ufBFpitOpfNSDMV28YcOgenV48EGIifF2NMaYPNarTi/8xZ8qxaqkGRB3b8N7iY6LptXXrdh0eBM/3/UztUrWumRxNSrTyGNJx2OpTFUTROQpYDbOE0SjVXWDiPR3bR8J3AE8KCLxwBngHldb1uWrUCH48Udn+uqnn4bvv/d2RMaYPFQyuCT/vem/VC9eHT85/125Y9WOhASHsO34NoZ3G57pqOkrkUfHEXjCJR9HkJmwMKev4Oef4c47L/3xExLg3/+GJ5+ERo0u/fGN8UH/9/f/ceDUAf5z3X+8HUquZTWOwBLBhYqPd+4Ktm2D9euhXLm027/6ytk2ZIj7/S/W7NnQtatzVzLMM+2GxpirR1aJwPO9HFerfPlgzBi45hp44gmYPh2SRy3u2uV8Wz971rlYt2+f98efMMH5c9GivK/bmKtQfHw8UVFRxMXFeTsUjypQoAAVKlQgX77M53lKz+4ILtann8Jzz8G4cdC7t7PugQdg4kQoXhxCQ513IPvlYb/82bNQpgzExjpNRMePQ9GieVe/MVehHTt2ULhwYUqWLJmj2VqvRKrK0aNHOXnyJFWrpn27W1Z3BDbp3MX697/h2mudJpqjR2HVKqcz+dln4f33ISLC6UfIS3PmOE8sPfusMxfSkiV5W78xV6G4uLirOgmAM0V3yZIlc33XY4ngYvn7w9dfO9/KX3gBXnwRSpWCV16B++93OnJfew3Oncu7Y06YACVKODOj5ssHf/yRd3UbcxW7mpNAsgs5R0sEeaFRI3jpJfjuO1iwAN56y2mq8feHDz6A7dth1Ki8OdaZMzB1Ktx+u3OM5s2tn8AYc1EsEeSVN9+E2rWdT79+59d36QIdO8LAgc401hcrPBxOnTrfH3HDDbBypb04x5jLXHR0NF988UWu9+vevTvR0dEeiOg8SwR5pUABZx6iZcuc5ppkIs6jpP7+0LMnnDhxcceZMAFKl4Z27Zzltm2dR1mXZ/0GJZKSnLuWnD7Ouno1PPKIk2j277+4mI0xmSaCxMTELPcLDw+nWLFingoLsMdH81aRIu7XV63qPEXUuTPce6/TtLN3L3zyCaxd63Quh+bgpROHDsGMGc58RwGuv7rrr3eSzR9/ZP6YqioMGHD+3ctt2jgfd377zbl7WboUChZ09u3SxZlbqXjx7GM05grw7K/PsubAmjyts0nZJnza9dNMt7/yyits27aNJk2akC9fPgoVKkS5cuVYs2YNGzdu5LbbbmPPnj3ExcXxzDPP0LevM89RlSpViIiI4NSpU3Tr1o02bdqwdOlSQkNDmTp1KkFBQRcdu90RXCrt28Nnn8HMmdC6tTNf0YgRzjf5Tp2ci3yyFSugb1/YsuX8uuho54IMaZueihWDhg0z7ydQde4ERo50HnOtUsUZ93D2bNpyMTHO+k6dYN8+GDoUoqKcpLVlC/To4Tyuaoy5IEOGDKF69eqsWbOGjz76iBUrVjB48GA2btwIwOjRo1m1ahURERF89tlnHD2acdq1rVu3MmDAADZs2ECxYsWYNGlS3gSX2fzUl+vHKy+myUtPPaUaHKz6zDPOOw4WLVINClJt1Eh13z7Vl19W9fNzXhURHKz61VeqJ0+qtm6tmi+f6q+/uq+zYEHnPQkHD6q+8IJqnz6qDz+s2quXU9eAAc4Ldn791VkeONDZNzFR9eefVStUcI770kuqsbFp6//5Z1UR1U6dVJcvd+rJjbg41aFDVbdtu7DfmTF5YOPGjV49/o4dO7R+/fqqqrpgwQJt3759mu1vvfWWNmrUSBs1aqRFihTRP//8U1VVK1eurIcPH9YdO3ZojRo1UsoPGTJE33nnHbfHcneueOPFNJ76XPGJICkp44tt5sxRDQx0LvSg+vjjqhs3qnbo4CyXKaPq7686aZL7OidMcMr17atapIhqQIBqjRqqFSuqli7tJIrExPPl77vPOdbw4U4CAtUGDZyLfGa++up8fJUrOwnj+PHsz/fAAdXrr3f2a9vWfRJJSlI9c0b12DHV/ftV9+xR3bFDNT4++/qNyaHLLRHcfPPNKdsWLFig119/vZ4+fVpVVdu1a6cLFixQ1bSJIHl/VdWPPvpI33rrLbfHym0isKahS00kbWcyOH0HkyY5A9Nmz3Y6l+vWhblz4cMPnWacb791Hhl154YbnD+//NLpRF6/HrZuhd274eBB+PzztCObhw51+jOeesp52mjMGFizBlq0yDzuxx936vr2W6hfHz7+2HlsduFCZ/vx4854iTJloEMHZ8T17NnOOa1eDX36OM1X06efrzM21onX3x+CgpyxEeXKQcWKTr9Ku3ZOJ7cxV4HChQtz8uRJt9tiYmIoXrw4wcHBbN68mWXLll3S2Kyz+HLRo4fzSc3Pzxmg9sIL5+cxcqdcOfjmG6hc2XlUNTshIU6n886dzsypATn8Z1C8ODz8sPNZvtyZSqNDB6eO5NHON98MO3Y4/REAFSrA4sVOP8bq1fDyy9C9u3Px79/f6eR+9lknpqAg5+mrgADnUdsPPnA60h98MGfxGXMZK1myJNdffz0NGjQgKCiIMmXKpGzr2rUrI0eOpFGjRtSuXZtWrVpd2uAyu1W4XD9XfNPQ1eTUKdV+/Zxmnx49VNesOb9t61bVH35wmoaS/fKLU3bkSOcDqmFh7utOTFRt3tzpu3DdLvuUXbucZr2FC70dyeXh5EnVf/65qCq83TR0KVnTkLl0ChZ0nkY6ftxp8mnc+Py2GjWcO4ZU33ro2dNpxnr9dWdupq5dnYF47vj5Oc1PUVFOM1NWkpKcO40aNZzHcq8GP/3kPEn21VfejuTyMHAgNGniDKY0ec4Sgbl4OR3sIgIffeRMzle2rNPsk9WsrG3bwm23OZP3HTzovkxsrNM09eGHzlQezz+f+/gvVlwcZNL2myImBubPz3mdyY8FTp2at4/tzplzfjzJlSQ83Pk9zJ3r7UiuSpYIzKXVsqVzkZs3D0qWzL78Bx84F9oePZw7iKZNnW+G99zjzOnUrp1zsRw2zHlr3E8/ORe7vLJrV9YX+c2bnc7zBg3g8OHMyz34oNN/k7qzPDN79zoj1Dt1cr4Bz5yZ+7gz85//OIMLN23Kuzo9bdeu82Nqpk3Luqy5IB5NBCLSVUS2iEikiLziZvt9IvK367NURBq7q8dcZW6/HWrWzFnZWrWcpqRDh5w7ifLlnVHYq1bBu+86F+KpU52mbTGvegAAECxJREFUppdfduodMMBJHuklJDhPLiUkZH3MxESnzg4dnAF4tWrBlCkZy82f7wwOPHnSuWPp08fZN70ZM5wLWHCwM1Dw2LGsj598rGHDnKa18eOzLp/s22+znoBw82bnibKkJBg0KGd1Xg6S7wKaNnWSYjZTMpgLkFnnwcV+cF5Yvw2oBgQCa4F66cpcBxR3/dwNWJ5dvdZZbFKcOZOxI3nuXPed0H/9pdqsmbPtjjtUz57NWF9Cgur336tWr+6Uq1hR9e23VZs0cZbvukt18mTVL75Qff55Z7xGvXrOmIdvvnHKvPpq2jpjY1WrVlWtW9cZpxEQoPrAA1mf1403OvWqqv7736r586vGxGS9z19/OccvUcIZVOjOO+84ZR580BkguG5d1nXmhdjY3A9ATO+uu1RDQ1XHjXPiX7r0gqqxzmIvDCgDWgOzUy2/CryaRfniwN7s6rVEYLLVu7czAK9lS9V//csZUOfv7wzM69/f+Wd/663OiGdV52I1dqxqnTrOtmuucUZTJw9oO3dO9d13nUF/zqQdzijsHj3SDqp74gln2/jx5y9+YWHOut9+c5YHDnSWp01zH/vhw07db7zhLC9Z4pT/4Yesz7lrV9WiRZ1Bf/fd575Mo0aq112neuSIauHCqnfeeX7bjBmqgwZd/EU7tago1VKlnIGOFyohQbV4cdVHHnF+1wEBGZNtDlki8E4iuBP4OtXyA8DwLMq/kLp8um19gQggolKlSnn32zJXp6NHVV980RnJXLiw88/8sceckcuqzohqUO3c2ZmKo1AhZ7lePdWJE9OOwk5tzx7VVaucqUASEjJuj4tTvfZap66qVZ0ElD+/6j33nC9z9qxqw4bON/dHHnGm3vjjj/MX4K+/dvb/6y9nOTFRtVIl1e7dMz/fBQucfT788HyiST8VyZYtzvqhQ53lN990lhcscO5QkhNcdgknp5KSnGSbXO+PP15YPcuXO/uPG+csd+igmmp0bW54OxEcP35cR4wYcUH7Dh06NGXUcU5cTongLjeJ4PNMyt4IbAJKZlev3RGYXElMdD8VxqhRzj//UqWcb/Jz5ri/uOdWTIzql186F+78+Z1EtGdP2jIbNqh26eKME0i+ULZt61z8u3d3kkjqb+YvveR8Ez5yJOPxkpJUW7RwxlvExjrNZbVrq1ap4ozzSDZ4sHOc3bud5WPHnDsIcO6WBg5UbdXKiSknU4ccPOh8089s/qiJE52633tPtU0bJ9leyDiAd95xmrEOH3aWhw516t2+PddVeTsRpJkiIinp/B1pDiRPM5FTuU0EHnt5vYi0BsJUtYtr+VUAVX0/XblGwBSgm6r+k129l93L682VKyrKeYw1pyOrc+vkSeeRx9RjKdI7eNDpHH7zTaczXMR5BPajj86X+esvp6O0cGFnrEaTJtCsmfPZvBnuvtt5Xepjjznlf//dme32kUecaUcCAuCaa5xR23/+eb7e//2P/2/vboOjKq8Ajv9P3ggCUSGoYJAEgRCoBTGKgAWsVgVU8INKHR21wFRtR1ReBnRQmY5jHbWVcSi1kyoCBaYFJIx2kAoIVFoBNaRIAHktEeUlBKkgAsnph3OXLJANBFhW9p7fzJ3s3ntz9zmb5J7c59nnXKZNs3ka11xjM78LC+0+3OPHx25zdbXNIJ83z/b/6CPIyKjZXlkJHTvajPfly+1+Fp07W9mQ6dNtVvqyZfbeDx5ss89j6dXL3sPI3/zGjTZfZPx4+4DAN99Aaam9P40axT4OUFZWRkFBgT154gkrq3I2delS55yXQYMGUVxcTH7btvyssJBLsrL465IlfF9VxV133cW4cePYv38/99xzD+Xl5VRVVTF27Fh27NjBiBEjyM/PJzs7m0WLFp20KcfEGqjr5vXxvCJIAzYBedQMFnc6bp8rgA1Aj1M9rl8RuKRUWan65JOqzZqprlp14vbiYqsg27NnTVdWZCkoOLFA39NP27Y+fWrGGV599eTtePRRG6MoKVFdu9YKILZpc2xX02uv6dHBc1AdPbpmW3W1dcOlplo3WkRkVnlkadLE/tNPTVUdOFD1uedsbKNbN3u+dq1dXdU2JtCxo43jPPCAVe4FGxvp3Vv1xRdjDpYf81/ysGG2f32WXr2sCnCPHrVvHzbM4t+3z65gjuti3LxqlXa68krVFSv0/Tfe0KF3363Vy5drVWWl9u/fXxcvXqwzZ87UIUOGHP2evXv3qmr8rwjilgjsdekHrMc+PfRMsO4R4JHgcRFQCZQES8yGRhZPBC70qqpUy8pUp0yxkuOxqsZOmWInytRU+1PfsuXkx66osO6y5s3tRJ2ZaV1VKSmqL79sCSIjQ/WOO+ykN3So7bdwoZUVueUWe62RI0889uTJqhMmqJaWWgwbN1rZ9exsO0br1vaJqawsO7H366dHxzGijRlj67OybPB/1izrPot8uqthQ/u01eef27ahQ1WvukrXLF1qyaW6uqbibWXlidWAo1VXW3mLTZtUV65UXbHClsh4U/TPZNcue83IPqtW2ft56JDqxo26ubhYO7Vtq7pnjw5/6ilt3bq1ds7P187t2+uVbdpoUVGRrlu3TnNzc3XUqFG65IMPbDxq+3Zt3aqV7tq8ue62RvnBdA3Fi3cNOVcPJSU2byMvz+4+dypmzLBihw8/bBVqGzWyQoMzZ9rjJk2sO6Z5c9i/37qVdu+2yW8ZGfDCC/DYY1ZY8FQcOWJzAxo0sOc7d8KYMfDmm/Z6FRU128Bu97p0Kdx4o83NiLZunU1CnDKlZr5IVhZcey1lI0dS0KyZdZVVV9dUthWx279GugkPHbLuqH377IZQhw/bDPjsbFu2brWqvR062OsfPGiz2g8csO63Sy+196G83PYLCkZuOXKE2wcPZvXq1QwfPpz27dvzy4cessl9aWn2fqamsmfvXv4+Zw5/nDqVW7p149mhQ8m9805WTp5Mdn6+Vec9ifp2DXkicC7ZVVXZySwz8/SPoWon+JdegtmzrXR6xKef2gzvvn2tj7xlyzNvM9ikwf37bZygvrZtgzlzbGyie3dIT6dszRoKWrSwk3tamlW7zciwRBMZn0lJqZmwlpICF15oJVQuuqgmsR06ZCfvlBQbB9m2zdbn5tp+kUrBqpYg9+2Dli2pOHCArl27snXrVubPn8/YsWNZsGABjVX5ctky0kU4UlVF06wsMhs3Zs7KlUwqLmbOO+9wVefOzJ02jbx27azdJ+GJwDkXP9XVtdeHqqo69SuABKnt5HjUwYN2JaJq/+U3bGhfY9XC+vZbu/pQtauWNm2OvWqJ4b777qO0tJS+ffuSk5NDUVERAI0bN2bqpElsWL+ekWPGkJKWRnp6OhMnTqSwsJDXX3+dCRMm0KJFi7gMFnsicM6FQp2J4HTs3WvdQZddVnfxxASobyLwG9M459zpiHQZJYEfVhpzzjl3znkicM6FxvnWFX46TidGTwTOuVDIzMykoqIiqZOBqlJRUUFmPT8h5mMEzrlQyMnJoby8nF113UAoCWRmZpJTV9mOWngicM6FQnp6Onl5eYluxg+Sdw0551zIeSJwzrmQ80TgnHMhd97NLBaRXcDW0/z2bGD3WWzO+SKMcYcxZghn3GGMGeofd2tVbV7bhvMuEZwJEVkZa4p1Mgtj3GGMGcIZdxhjhrMbt3cNOedcyHkicM65kAtbIvhTohuQIGGMO4wxQzjjDmPMcBbjDtUYgXPOuROF7YrAOefccTwROOdcyIUmEYjIbSKyTkQ2iMjoRLcnHkSklYgsEpEyEflcRIYF65uKyD9E5Ivg68WJbuvZJiKpIvKZiLwbPA9DzBeJyEwRWRv8zLuHJO4ng9/v1SIyXUQyky1uEXlTRHaKyOqodTFjFJExwbltnYjcWt/XC0UiEJFUYALQF+gI/FxEOia2VXFxBBiuqgXA9cCvgjhHAwtUtR2wIHiebIYBZVHPwxDzeGCeqnYAOmPxJ3XcInI58DhQqKo/AlKBQSRf3JOA245bV2uMwd/4IKBT8D1/CM55pywUiQC4DtigqptU9RAwAxiQ4Daddar6lap+Gjz+H3ZiuByL9e1gt7eBgYlpYXyISA7QHyiKWp3sMWcBvYA/A6jqIVXdS5LHHUgDGopIGnABsJ0ki1tVlwB7jlsdK8YBwAxV/V5VNwMbsHPeKQtLIrgc2Bb1vDxYl7REJBe4GvgYuFRVvwJLFsAliWtZXLwGjAKqo9Yle8xtgF3AW0GXWJGINCLJ41bVL4FXgP8CXwHfqOp8kjzuQKwYz/j8FpZEILWsS9rPzYpIY2AW8ISq7kt0e+JJRG4HdqrqJ4luyzmWBnQFJqrq1cB+zv/ukJMK+sUHAHlAS6CRiNyf2FYl3Bmf38KSCMqBVlHPc7DLyaQjIulYEviLqs4OVu8QkRbB9hbAzkS1Lw56AneKyBasy++nIjKV5I4Z7He6XFU/Dp7PxBJDssd9M7BZVXep6mFgNtCD5I8bYsd4xue3sCSCFUA7EckTkQxsYGVugtt01omIYH3GZar6u6hNc4EHg8cPAsXnum3xoqpjVDVHVXOxn+tCVb2fJI4ZQFW/BraJSH6w6iZgDUkeN9YldL2IXBD8vt+EjYUle9wQO8a5wCARaSAieUA7YHm9jqyqoViAfsB6YCPwTKLbE6cYb8AuCUuBkmDpBzTDPmXwRfC1aaLbGqf4+wDvBo+TPmagC7Ay+HnPAS4OSdzjgLXAamAK0CDZ4gamY2Mgh7H/+AfXFSPwTHBuWwf0re/reYkJ55wLubB0DTnnnIvBE4FzzoWcJwLnnAs5TwTOORdyngiccy7kPBE4B4jIiyLSR0QGJqo6rYh8KCKhuwm7SzxPBM6Zblhdpt7A0gS3xblzyhOBCzUReVlESoFrgX8BQ4CJIvJsLfs2F5FZIrIiWHoG658XkSkisjCoFT80WC/B8VeLyH9E5N6oY40K1q0Skd9GvczdIrJcRNaLyE/iGrxzgbREN8C5RFLVkSLyN+AB4CngQ1XtGWP38cDvVfWfInIF8D5QEGz7MXYPiEbAZyLyHtAdm/3bGcgGVojIkmDdQKCbqh4QkaZRr5GmqteJSD/gOay2jnNx5YnAOSvXXQJ0wOr1xHIz0NFK3ACQJSJNgsfFqvod8J2ILMLqwd8ATFfVKqxg2GLsyqM38JaqHgBQ1ei685FCgZ8AuWcamHOnwhOBCy0R6YLdCSoH2I3d5EREpAToHpzYo6XUtj5IDMfXalFqLw9MsD5WbZfvg69V+N+nO0d8jMCFlqqWqGoXrBhhR2AhcKuqdqklCQDMB34deRIkkogBwb1zm2HF71YAS4B7xe6n3By7o9jy4Di/EJELguNEdw05d855InChFpygK1W1GuigqnV1DT0OFIpIqYisAR6J2rYceA/4N/AbVd0OvINVBl2FJZlRqvq1qs7DSgevDK4+Rpz1wJyrB68+6twZEpHngW9V9ZVEt8W50+FXBM45F3J+ReCccyHnVwTOORdyngiccy7kPBE451zIeSJwzrmQ80TgnHMh939xVzFTKcQdPwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show plot accuracy changes during training\n",
    "\n",
    "plt.plot(history2.history['loss'],'g')\n",
    "plt.plot(history2.history['val_loss'],'r')\n",
    "plt.title('1D-Conv Loss across epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('# epoch')\n",
    "plt.legend(['train', 'test'], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and use our model to make predictions on the test data. Remember, this is a reconstruction model, so the predictions, ideally, should look similar to the original data for normal disks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use model to predict on test data set\n",
    "y_pred2 = autoencoder2.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell will plot a comparison of the original data to the predicted data for a random record. It only shows compares the features for the first day in the sequence. Run it a few times to see how well your model came out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This record is from a normal disk.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEWCAYAAABfdFHAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZgkVZ3u8e/bC0qxyNKNLE1XAQIjMFeBFmFERUQFRtkGZ8BSWdQawQV1dETaqzjaM7gvg1upiNgF7rI4KgKKwB0RuhEQBJGleqFZ2kaRplRo+nf/OKfsrCIzKzOroiKz6v08Tz6ZGREnzon1F3Ei4oQiAjMzs6LMKLsAZmY2tTnQmJlZoRxozMysUA40ZmZWKAcaMzMrlAONmZkVyoGmw0iaL2mtpJkTOWwD4zpX0ofGOx5rTF5uOxcw3t0l/UrSI5LeWsD4D5K0cqLHO5EkbSzpEkkPS/p22eWZDto60Eh6laQleaO7T9KPJB1YdrnKFBHLI2LTiHhiIocdD0knSrqmyDzKVHSQlXSlpNdXdsvL7e4Csvt34MqI2CwiPlPA+AuT9wfLJD0q6UJJW7U4qmOBpwNbR8QrJ7CIHUHSP0q6RtIfJd0v6UuSNqvof2ve5w5/1km6JPfbTdJFklZLekjSpZJ2HyvPtg00kt4BfAr4T9JKMR/4HHBkmeUai6RZnTjudtJp09lh5e0Gbm0lYZnTKWlP4IvAa0j7gyHS/qAV3cAdEbFugorXaZ4GfAjYHngmMA/46HDPiNgzH+hsCmwGLAeGz/y2AC4Gdicth+uAi8bMMSLa7pNnxFrglXWGeQopEK3Kn08BT8n9DgJWko7eHgTuA44CDgfuAB4CzqgY15nAd4BvAo8ANwDPquh/OnBX7vcb4OiKficC/w/4ZB7vh3LZPpYX0APAF4CNa0zHDOC9wLJc1vOAp+V+PUAAr8vjuqqi26w8zE65+yPA5cBngcWj0g8PeyXwwVzeR4CfAHMqyvJt4H7g4TzOPSv6nQt8qEr5nwn8BXgiL7M/VizD84DVedreC8yoMQ+G5/9i4E/A63P6r+Rld2+erzMr0rwBuK1imexTUZ4rgT+SdqhHjJqGzwL/k9P9Etgl91Nehg/m6b8Z2AvoAx4HHsvTd0kefhB4dx7ur8CsPK+fUWuekQ6SbszTeBdwKLAoz7u/5PGfnYf927jqzUvS+ncNaX37A3APcFiN+fzTUXnt1sC4R6zbVca5cZ7OP+Tl8C5g5VjbDmkbeQj4+4phtwH+DMytks9/AudX/N8lL5PNakxr1fUA+EBO93ieB6+rknY/YEleTg8An6jotz/wv3m8NwEHVfTbCfh5ntbLgLPJ22KVPA4i7aP+jQ37qJMq+l8JvH7Ufuaaiv8BnAr8Luf3wTxPfpHL/S1gowb3t8cAv67R74V5Pm1So/9WuSxb182jkYJM9oe0Aa4j7yBrDPMfwLV55ZybF/4HKxbiOuB9wGzSTmk1cD4pQu9J2th2zsOfmVe8Y/Pw7yRtsLNz/1eSov8M4F+AR4HtKlaAdcBbSDubjUlB7+K8EDYDLgH+q8Z0nAzcCewMbAp8D/h67teTF+J5wCZ53MPdhoPHL0g7mY2AA/NKVi/Q3EXawWyc/581qiybsSGI31jR71yq7GiqbQS523mkI53NcjnuoMpGPWr+H5Xn8cbAhaQj2E3yMr4O+NeK5XEv8BxSgHgG6Sh1dp6XZ+T5cTBpI9y9YhoeIu1IZgEDwDdyv5cBS0lHbCLtqLarNe2kQHMjsCP5III6gSbn+TDwkjyNOwB/V22nMnpc9eZlnvePk9bxmcAppAMv1ZjXI/JqYNwj1u0q4zsLuJq0ru8I3MLIQFNv2/kc8OGKYU8jB/Iq+VwEvHtUt7XAvlWGHWs9OJMaAaBim3pN/r0psH/+vQOwhnTAOiMvyzXkwJjTfYK0/bwg51kv0Kwj7cdm53EOAVvWWE4n8uRAczGwOWl/9lfgCtJ+5GmkoH5Cg/vbT5G3gyr9zgHOrZP2KOC+MfNopCCT/QF6gfvHGOYu4PCK/y8DBisW4p/JR8B5IwrguRXDLwWOqljxrq3oN4N0hPH8GnnfCBxZsQIsr+invDHtUtHtAOCeGuO6Aji14v/upB3HLDYEip0r+g93m0WqTlwHdFX0X0z9QPPeimFPBX5co1xb5LTDZ1fn0mCgIe3w/grsUdHtX0nXBqqlPxO4quL/03P6jSu6HQ/8LP++FDitynieTzojm1HR7QLgzIpp+HJFv8OB2/Pvg0k72f0ZdeZVbdpJgebkUd3qBZovAp+sMf1XUiPQjDUv87y/s6JfV0677Vh5NTju5dXGUzH83cChFf/7qAg0Y2w7zwVWsOEMagnwz3W2kzeO6nYvFWcUTawHZ1I/0FxFOvOZM6r7u8kHgRXdLgVOYMO2uElFv/Nr5cOGfdSsim4PsiGojVgnqB5onlfxfykVgRj4OPCpessuD/cS0tnoblX6dZEOXJ80j3P/eXkZHD9WPu16jWYNMGeMOuHtSaf6w5blbn8bR2y4CP7n/P1ARf8/k45Whq0Y/hER60mntdsDSHqtpBvzxbM/kqpU5lRLSzq76gKWVgz/49y90emYRdrZVhv/6LQPRcRQA8MOu7/i9xB5HkiaKeksSXdJ+hNpRwojp7NRc0hHkqOna4c6aSrLPXx2cl/FPPwi6cwG0pHzXVXGsT2wIi+/WvlWnf6I+CmpquOzwAOS+iVtXqe8o8s8llplHksj8/Jv01SxLlSu2+MZ91jTuP2oYSrHVXfbiYhfkg7KXijp70iB9eIa+awlHb1X2px01lC1TGOsB/W8jnTWf7uk6yW9PHfvBl45PC15eg4Etst5/iEiHh2VZz1rYuR1or+tjw0avT+rt397Ekn7k4LhsRFxR5VBjiHVAPy8Stq5pKr3z0XEBWMVtF0DzS9IVVtH1RlmFWnBD5ufu7Vqx+EfkmaQovUqSd3Al4A3k+ohtyBVD6gibVT8/j1pIe8ZEVvkz9MiXVhrdDrWMXKlCaq7D9hKUle16WjSq0jXEA4hnXr35O6qlaBO+X5POisbPV33NjiOFaQj7TkV83DziNizov8uVcaxCtgxL79G891QgIjPRMS+pKqI3UjXG0aXrVaZIe0oKpfFthW/a5W53vihtXnZqEbGXa9skNbBynVu/vCPBredrwGvJl3k/05E/KVGPrcCz6oY986kKqpqO8jxrge/i4jjSQc2Hwa+I2kT0jL8esU6uUVEbBIRZ5Hmw5Z5uMo8W/UotdelcZO0NymonxwRV9QY7ATgvMinLxVptyQFmYsjYlEj+bVloImIh0nXVz4r6ShJXZJmSzpM0kfyYBcA75U0V9KcPPzicWS7r6Rj8lnU20g7umtJ1wiCdI0HSSeRjspqlX09aeP6pKRtcpodJL2sRpILgLdL2knSpqSLnt+MBu6IiYhlpOqGMyVtJOkA4BWNTe6TbEaa5jWkFfw/m0j7ADBP0ka5XE+QLkYukrRZ3uG8gwaXT0TcR1qRPy5pc0kzJO0i6YV5kC8D75S0r5Jn5DyGj5D/Pa8vB5HmxzfGylPScyQ9V9LsPI7hGxyGp6+RZ1puBF6Vzw4PJV1IHfYV4CRJL87Ts0M+iq87/vHOy3omaNzfAt4jaUtJ80jXc4Y1su18HTiaFGzOq5PPAPAKSc/PO/P/AL4XEdXOaFpeD3I5Xy1pbt6W/5g7P0GaL6+Q9LK8jJ+q9NzQvIpt8QN5WzyQ1rdFSOvSMXnf9wzSWdaEkLQXqZblLRFxSY1h5gEvIh0IVHbfnFRd+P8i4vRG82zLQAMQEZ8grfTvJa2oK0hHRhfmQT5EWrA3A78m3Sk2nmcdLiJdrPwD6ejqmIh4PCJ+Q6rv/AVph/D3pDtx6nk36WLktbka6nLStZdqziFtbFeRbkD4CyM31rH0kq4BrSFN/zdJAaNZ55FO9e8lXUi8tom0PyUdcd4v6fe521tIG/vdpLuizidNa6NeS6rW+Q1pmXyHVEVBRHybdLfW+aSqkwuBrSLiMeAI4DDS0frngNdGxO0N5Lc56QDhD6T5sIZ0kwWkILFHri65sEZ6SBezX0HaOfWyYV0lIq4DTiLdwfUwqTpi+Ezi08Cxkv4gqdqzLeOdl/WMd9wfIM2ve0gHB18f7tHIthMRK0nbbpBuKqgqIm4F3kgKOA+SDoxOrTHseNYDSDcj3SppLWnZHBcRf4mIFaSz/jPYsE96Fxv2o68iXXd6CHg/9QPnWD5JujvuAdLOfmAc4xrt30hV+V/RhmdlRt/y/hrgFxExurr3aNJNOCdp5LM2dc/eNOqsaFqSdCbpIu6ryy7LeEn6JukC9/vLLotZIySdA6yKiPeWXZaJNJX2K+PVSQ+aWRWSnkM6groHeCnpiOusUgtl1iBJPaSLznuXWxIrUttWnVnDtiXdCrkW+AxwSkT8qtQSmTVA0gdJNwd8NCLuKbs8VhxXnZmZWaF8RmNmZoWaktdo5syZEz09PWUXw8ysYyxduvT3EVHrwfJxmZKBpqenhyVLlpRdDDOzjiFprJYMWlZa1ZmkHSX9TNJtSu8/OK3KMJL0GUl3SrpZ0j5llNXMzFpX5hnNOuDfIuIGpZfuLJV0WX7Ia9hhwK7581zg8/nbzMw6RGlnNBFxX0TckH8/Qnq3yOhG744kt7UTEdcCW0jabpKLamZm49AWd53lh7b2JrVRVGkHRrYMu5IaLbBK6lN67fOS1atXF1FMMzNrQemBJjck+V3gbRHxp9G9qySp+uBPRPRHxIKIWDB3biE3TpiZWQtKDTS5pdzvAgMR8b0qg6xkZBPk8xjfqwDMOs/AAPT0wIwZ6XtgIttXLFCnltsmXJl3nYnUKu5tuaXmai4GXpvvPtsfeDg3IW82PQwMQF8fLFsGEem7r6/9d9qdWm4rRGlN0OT3NVxNauJ/+E14Z5BfFhQRX8jB6GxSs91DwEkRMeYDMgsWLAg/R2NTQk9P2kmP1t0Ng4OTXZrGdWq5pzFJSyNiQRHjLu325oi4hjHe3pjf7PamySmRWRtavry57u2iU8tthSj9ZgAzq2N+jfdJ1ereLjq13FYIBxqzdrZoEXR1jezW1ZW6t7NOLbcVwoHGrJ319kJ/f7q2IaXv/v7UvZ11armtEFPyfTS+GcDMrDlF3gzgMxozMyuUA42ZmRXKgcbMzArlQGNmZoVyoDEzs0I50JiZWaEcaMzMrFAONGZmVigHGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQjnQmJlZoRxozMysUA40ZmZWqFIDjaRzJD0o6ZYa/Q+S9LCkG/PnfZNdRjMzG59ZJed/LnA2cF6dYa6OiJdPTnHMzGyilXpGExFXAQ+VWQYzMytWJ1yjOUDSTZJ+JGnPsgtjZmbNKbvqbCw3AN0RsVbS4cCFwK7VBpTUB/QBzJ8/f/JKaGZmdbX1GU1E/Cki1ubfPwRmS5pTY9j+iFgQEQvmzp07qeU0M7Pa2jrQSNpWkvLv/UjlXVNuqczMrBmlVp1JugA4CJgjaSXwfmA2QER8ATgWOEXSOuDPwHERESUV18zMWlBqoImI48fofzbp9mczM+tQbV11ZmYwMAA9PTBjRvoeGCi7RGbNcaAxa2MDA9DXB8uWQUT67uvrjGDjAGnDHGjM2tjChTA0NLLb0FDq3s46OUDaxHOgMWtjy5c3171ddGqAtGI40Ji1sVrPHrf7M8mdGiCtGA40Zm1s0SLo6hrZrasrdW9nnRogrRgONGZtrLcX+vuhuxuk9N3fn7q3s04NkFaMdm/rzGza6+1t/8Ay2nB5Fy5M1WXz56cg02nTYRPDgcbMCtGJAdKK4aozMzMrlAONmZkVyoHGzMwK5UBjZmaFcqAxM7NCOdCYmVmhHGjMzKxQDjRmZlYoBxozMyuUA42ZmRXKgcbMzArlQGNmZoUqNdBIOkfSg5JuqdFfkj4j6U5JN0vaZ7LLaGZm41P2Gc25wKF1+h8G7Jo/fcDnJ6FMZmY2gUoNNBFxFfBQnUGOBM6L5FpgC0nbTU7pzMxsIpR9RjOWHYAVFf9X5m5mZtYh2j3QqEq3qDqg1CdpiaQlq1evLrhYZmbWqHYPNCuBHSv+zwNWVRswIvojYkFELJg7d+6kFM7MzMbW7oHmYuC1+e6z/YGHI+K+sgtlZmaNm1Vm5pIuAA4C5khaCbwfmA0QEV8AfggcDtwJDAEnlVNSMzNrVamBJiKOH6N/AG+apOKYmVkB2r3qzMzMOpwDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQjnQmJlZoRxozMysUA40ZmZWKAcaMzMrlAONmZkVyoHGzMwK5UBjZmaFcqAxM7NCOdCYmVmhHGjMzKxQDjRmZlYoBxozMyuUA42ZmRXKgcbMzArlQGNmZoVyoDEzs0KVGmgkHSrpt5LulHR6lf4HSXpY0o35874yymlmZq2bVVbGkmYCnwVeAqwErpd0cUT8ZtSgV0fEyye9gGZmNiHKPKPZD7gzIu6OiMeAbwBHllgeMzMrQJmBZgdgRcX/lbnbaAdIuknSjyTtOTlFMzOziVJa1RmgKt1i1P8bgO6IWCvpcOBCYNeqI5P6gD6A+fPnT2Q5zcxsHMo8o1kJ7Fjxfx6wqnKAiPhTRKzNv38IzJY0p9rIIqI/IhZExIK5c+cWVWYzM2tSmYHmemBXSTtJ2gg4Dri4cgBJ20pS/r0fqbxrJr2kZmbWsoarziRtEhGPTlTGEbFO0puBS4GZwDkRcaukN+b+XwCOBU6RtA74M3BcRIyuXjMzszamsfbbkv4B+DKwaUTMl/Qs4F8j4tTJKGArFixYEEuWLCm7GGZmHUPS0ohYUMS4G6k6+yTwMnKVVUTcBLygiMKYmdnU09A1mohYMarTEwWUxczMpqBGrtGsyNVnkS/avxW4rdhimZnZVNHIGc0bgTeRHqZcCTw7/zczMxvTmGc0EfF7oHcSymJmZlPQmIFG0ld58hP7RMTJhZTIzMymlEau0fyg4vdTgaMZ9QS/mZlZLY1UnX238r+kC4DLCyuRmZlNKa00QbMr4FYrzcysIY1co3mEdI1G+ft+4N0Fl8vMzKaIMc9oImKziNi84nu30dVpU8HAAPT0wIwZ6XtgoOwSmZlNDTXPaCTtUy9hRNww8cUpx8AA9PXB0FD6v2xZ+g/Q6xu7zczGpWajmpJ+ViddRMTBxRRp/JptVLOnJwWX0bq7YXBwwoplZta2imxUs+YZTUS8qIgM29Hy5c11NzOzxjX0PhpJewF7kJ6jASAiziuqUJNt/vzqZzR+I7SZ2fiNeTOApPcD/50/LwI+AhxRcLkm1aJF0NU1sltXV+puZmbj08hzNMcCLwbuj4iTgGcBTym0VJOstxf6+9M1GSl99/f7RgAzs4nQSNXZXyJivaR1kjYHHgR2Lrhck66314HFzKwI9W5vPhu4ALhO0hbAl4ClwFrguskpnpmZdbp6ZzS/Az4GbE8KLhcALwE2j4ibJ6FsZmY2BdS8RhMRn46IA4AXAA8BXwV+BBwladdJKp+ZmXW4RpqgWRYRH46IvYFXkV4TcPtEZC7pUEm/lXSnpNOr9Jekz+T+N4/VWoGZmbWfRm5vni3pFZIGSGc0dwD/NN6MJc0EPgscRnpG53hJe4wa7DBSa9G7An3A58ebb00lNXY2HdtYm47TbDad1Qw0kl4i6RxgJWkn/0Ngl4j4l4i4cALy3g+4MyLujojHgG8AR44a5kjgvEiuBbaQtN0E5D3ScGNny5ZBxIbGzgreA44324FTr6Fn1kpmaD09s1YycOo1hZZ3IpQ0q80a5yOhiRcRVT/Az4A3AFvVGmY8H9LzOV+u+P8a4OxRw/wAOLDi/xXAghrj6wOWAEvmz58fTenujkj7vZGf7u7mxtOk8WS7+JSro4u1I9J1sTYWn3J1oWUer5JmtVljFi+OxbNPjG7uCfFEdHNPLJ59YsTixWWXrHDAkihgXx8RdW8GeFFEfCkiHiooxqlati0MkzpG9EfEgohYMHfu3OZKUlJjZ+PJdmF/D0NsMqLbEJuwsL9n/AUrUCe3K+cD3alv4LRf0vf42Syjh2AGy+ih7/GzGTjtl2UXraO18obNibIS2LHi/zxgVQvDjF+tRs0Kbuxs/lZrm+peafkT2zfVvV2MZ5rL5Cq/6WHhmndUP4Bb846SSjQ1lBlorgd2lbSTpI2A44CLRw1zMfDafPfZ/sDDEXHfhJekpMbOFnEGXTw6MlseZRFnjJl2/szq8bZW93YxnmmG8s4qFi7c8L6iYUNDqbtNHctrvKW+VndrUFF1co18gMNJd7HdBSzM3d4IvDH/FunOtLuAX1Pj+szoz7777tt8BeXixelCgZS+J6NOVorFHD+yPpjjUxnG0KnXaMY1zYsjujZ6fOQ0b/T4ZC2qqteWGii2dZDurR+pfg1x60fKLlrhKPAaTamBpqhPS4GmDOO8Mr74lKuje+aKtMOeuaL9g0zEuKa5zJ2Ab2KYZOM58BtH2jIPZsrmQDNVA83ixRFdXSP3XF1dU/sOl3FMs3ii+lkFT7Rzsa1Z45nZE7CgyqjcaAcONFM10ERMz7W6xWnu5p7qZxXcU2Rp/2Y6LqpSdHdXr15t5PTRp54tKzLQlHkzgEF6N8HgIKxfn76nw7sKWpzmRVt/ovqNBFt/YuLLWEUvAwzSw3pmMEgPvUz9W87KuPliYNnz6ONLI28x5ksMLHve2ImXL2eA4+nhHmbwBD3cwwDHd8b981NZURGszE9HndFY48p8mG4a1p2VNcndM1dUPymZuWLsMm/9luo3yWz9lmILPQHKPmPGVWcONJaVtTVOwyqZsiZZrK9xLW792GXu0LvG2uE4pshAozT+qWXBggWxZMmSsothbWhgID37snx5eh530aIGa+5mzEjb/2hSqgKcgsqa5J6e9EDsaN3dqaa1nk5dTOOZ5okiaWlELChi3L5GY9PGwAD0nbxu5NP9J69r7LpDSa1HlKmsSR7P89Odupg6uWmmRjjQ2LSx8LS1DD028qWyQ4/NYuFpDTR/U1LrEWUqa5J7e6G/Px3NS+m7v7+xM89OXUydGiAbVlSdXJkfX6Oxasb9HE7ZV2tL0ImT3Kll9jWaDuNrNFZNjwZZRs+TunczyGA8uftU0PI1KZt0ZS8rX6MxmwBlP4cz2cZ1Tcom3VR+TsuBxqaN3k8/l/7Zb6abQcR6uhmkf/ab6f30cwvPu4y3oY7rmpRNrin+HgpXndn0UkL9xMCp19D3+b1HvOeki0fpP+VX9H7uwMLynaH1VGv8Q6xnffgYs620wf3NRVadOdCYFaxn1kqWPTHvSd27Z65kcN2Tu09YvtPwmlTHaoMHgHyNxqyDlfU21Ol2TaqjTfH7mx1ozApW1ttQy7wmZU3q1AeAGuRAY1awRX2D1c8s+gaLzbi3l96vHsJg90Gs1ywGuw+i96uH+P7mdjSep1Q7gK/RmE2CgVOvYWF/D8uf2J75M1exqG+w0BsBzJrlmwGa5EBjZtYc3wxgZmYdy4HGzMwKNWvsQSaepK2AbwI9wCDwzxHxhyrDDQKPAE8A64o6rTMzs+KUdUZzOnBFROwKXJH/1/KiiHh2OweZMt6rbmbWKcoKNEcCX8u/vwYcVVI5xm2KN1FkZjZuZQWap0fEfQD5e5sawwXwE0lLJfXVG6GkPklLJC1ZvXr1BBe3toULYWhoZLehodTdrOP5dN0mQGHXaCRdDmxbpVczu+DnRcQqSdsAl0m6PSKuqjZgRPQD/ZBub266wC2a6q9gtWls+HR9+Ehq+HQdpsyDhDY5CjujiYhDImKvKp+LgAckbQeQvx+sMY5V+ftB4PvAfkWVt1Xzt6re5Hqt7mYdw6frNkHKqjq7GDgh/z4BuGj0AJI2kbTZ8G/gpcAtk1bCBi3ijOrNi3BGSSUymyA+XbcJUlagOQt4iaTfAS/J/5G0vaQf5mGeDlwj6SbgOuB/IuLHpZS2jt6HzqafN4xsuJA30PvQ2WUXzWx8pniLwjZ5Sgk0EbEmIl4cEbvm74dy91URcXj+fXdEPCt/9oyI9mzGdP58ermAQXZiPTMZZCd6uWDqb4y+SDz1TfEWhW3yuGWA8ZqOG+PAAAMnXU7PsiuZEevoWXYlAydd7mAz1UzxFoVt8rhRzYlQwuuByzQw5630rfmvJ7+aeOv30Pv7z5RYMjNrlRvVbHe9vem93uvXp+8pHGQAFq55x4ggAzDEJixc846SSmQ2vbV7TXYpbZ1ZZ1tO9etPtbqbWXE64XEnn9FY0+ZvPdRUdzMrTic87uRAY01b9OlN6dpo3YhuXRutY9GnNy2pRGbTVyc87uRAY03r7YX+c2aNvBnpnFltc5puNp10wuNODjTWkml2/4NZ2+qEJywcaMzMOlgnPO7ku87MzDpcb297BZbRfEZjZmaFcqAxM7NCOdBMY+3+NLFZp/E2VZ2v0UxTnfA0sVkn8TZVmxvVnKZ6etKGMFp3d7pd2cya0+nblBvVtAnXCU8Tm3USb1O1OdBMU53wNLFZJ/E2VZsDzTTVCU8Tm3USb1O1OdBMU53wNLFZJ/E2VZtvBjAzM98MYGZmnauUQCPplZJulbReUs0IKulQSb+VdKek0yezjGZmNjHKOqO5BTgGuKrWAJJmAp8FDgP2AI6XtMfkFM/MzCZKKS0DRMRtAJLqDbYfcGdE3J2H/QZwJPCbwgtoZmYTpp2v0ewArKj4vzJ3q0pSn6QlkpasXr268MKZmbWNNm9krbAzGkmXA9tW6bUwIi5qZBRVutW8RS4i+oF+SHedNVRIM7NO1wGNrBUWaCLikHGOYiWwY8X/ecCqcY7TzGxqWbhwQ5AZNjSUurdJoGnnqrPrgV0l7SRpI+A44OKSy2Rm1l46oJG1sm5vPlrSSuAA4H8kXZq7by/phwARsQ54M3ApcBvwrYi4tYzympm1rQ5oZK2UQBMR34+IeRHxlIh4ekS8LHdfFRGHVwz3w4jYLSJ2iQi3GGRmNloHNLLWzlVnZmY2lg5oZM1v2DQz63S9vW0VWEbzGY2ZmRXKgcbMzArlQGNmZoVyoDEzs0I50JiZWaEcaMzMrFAONGZmVigHGjMzK5QDjZmZFcqBxszMCuVAY6GoUQkAAAq2SURBVGZmhXKgMTOzQjnQmJlNlIEB6OmBGTPS98BA2SVqCw4005k3CrOJMzAAfX2wbBlEpO++Pm9XONBMX94ozCbWwoUwNDSy29BQ6j7NOdBMV94ozCbW8uXNdZ9GHGimK28UZhNr/vzmuk8jDjTTlTcKs4m1aBF0dY3s1tWVuk9zDjTTlTcKs4nV2wv9/dDdDVL67u9v61csT5ZSAo2kV0q6VdJ6SQvqDDco6deSbpS0ZDLLOOV5ozCbeL29MDgI69enb29PAMwqKd9bgGOALzYw7Isi4vcFl2d66u31hmBmhSsl0ETEbQCSysjezMwmUbtfowngJ5KWSuqrN6CkPklLJC1ZvXr1JBXPzMzGUtgZjaTLgW2r9FoYERc1OJrnRcQqSdsAl0m6PSKuqjZgRPQD/QALFiyIlgptZmYTrrBAExGHTMA4VuXvByV9H9gPqBpozMysPbVt1ZmkTSRtNvwbeCnpJgIzM+sgipj8WiZJRwP/DcwF/gjcGBEvk7Q98OWIOFzSzsD3c5JZwPkR0dBDHpJWA8taLN4coNW73DoxbZl5e5o7I22ZeXuaJy9td0TMbTFtfRHhT8UHWDKd0nZquT3Nnl+e5omf5qI+bVt1ZmZmU4MDjZmZFcqB5sn6p1naMvP2NHdG2jLz9jRPXtrClHIzgJmZTR8+ozEzs0I50JiZWaEcaDJJh0r6raQ7JZ3eZNpzJD0oqekHSiXtKOlnkm7Lr044rYm0T5V0naSbctoPtJD/TEm/kvSDFtK2/BoHSVtI+o6k2/O0H9Bgut1zfsOfP0l6WxP5vj3Pq1skXSDpqU2kPS2nu7WRPKutF5K2knSZpN/l7y2bSNvo6zWqpf1ontc3S/q+pC2aSPvBnO5GST/Jz7s1nHdFv3dKCklzmsj7TEn3Vizvw5vJV9Jb8nZ9q6SPNJHvNyvyHJR0YxNpny3p2uHtQtJ+1dLWSf8sSb/I29YlkjavkbbqvqPRdWxSlX1/dTt8gJnAXcDOwEbATcAeTaR/AbAPcEsLeW8H7JN/bwbc0WjegIBN8+/ZwC+B/ZvM/x3A+cAPWij7IDCnxXn+NeD1+fdGwBYtLrf7SQ+aNTL8DsA9wMb5/7eAExtMuxepZYou0gPElwO7NrteAB8BTs+/Twc+3ETaZwK7A1cCC5rM96XArPz7w03mu3nF77cCX2gm79x9R+BS0oPUVdeZGnmfCbyzgeVTLe2L8nJ6Sv6/TTNlruj/ceB9TeT7E+Cw/Ptw4Momy3098ML8+2TggzXSVt13NLqOTebHZzTJfsCdEXF3RDwGfAM4stHEkRr6fKiVjCPivoi4If9+BLiNtENsJG1ExNr8d3b+NHx3h6R5wD8CX26q0OOUj9BeAHwFICIei4g/tjCqFwN3RUQzrUDMAjaWNIsUNFY1mO6ZwLURMRQR64CfA0fXS1BjvTiSFGTJ30c1mjYibouI345V0Bppf5LLDXAtMK+JtH+q+LsJddaxOtvCJ4F/bzHtmGqkPQU4KyL+mod5sNl8JQn4Z+CCJtIGMHwW8jTqrGM10u/OhjYdLwP+qUbaWvuOhtaxyeRAk+wArKj4v5IGd/YTSVIPsDfpzKTRNDPzaf2DwGUR0XBa4FOkjX99E2kqNfwah1F2BlYDX83Vdl9Was+uWcdRYwdQTUTcC3wMWA7cBzwcET9pMPktwAskbS2pi3SkumOT5QV4ekTcl8tzH7BNC+MYr5OBHzWTQNIiSSuAXuB9TaY9Arg3Im5qJl2FN+equ3OarAbaDXi+pF9K+rmk57SQ9/OBByLid02keRvw0Ty/Pga8p8k8bwGOyL9fSQPr2ah9RzusYyM40CTV3sA2qfd9S9oU+C7wtlFHkHVFxBMR8WzSEep+kvZqML+XAw9GxNKWCpw8LyL2AQ4D3iTpBQ2mm0WqLvh8ROwNPEo6xW+YpI1IG+O3m0izJelobydge2ATSa9uJG2kl/V9mHSE+WNS9eq6uonakKSFpHIPNJMuIhZGxI453ZubyK8LWEiTwanC54FdgGeTDg4+3kTaWcCWwP7Au4Bv5TOUZhxPEwcz2SnA2/P8ejv5zL0JJ5O2p6WkKrHH6g3c6r5jMjnQJCsZedQwj8arVMZN0mzSijIQEd9rZRy56ulK4NAGkzwPOELSIKmq8GBJi5vM82+vcSA1gFrzoucoK4GVFWdf3yEFnmYcBtwQEQ80keYQ4J6IWB0RjwPfA/6h0cQR8ZWI2CciXkCq7mjmKHfYA5K2A8jfVatziiDpBODlQG/kCvwWnE+NqpwadiEF9pvyujYPuEFStXdVPUlEPJAPptYDX6LxdQzSeva9XMV8HenMveqNCNXk6tVjgG82kSfACaR1C9KBUDNlJiJuj4iXRsS+pCB3V50yVtt3lLaO1eJAk1wP7Cppp3ykfBxw8WRknI+wvgLcFhGfaDLt3OG7hyRtTNqR3t5I2oh4T0TMi4ge0vT+NCIaOrrP+bX8GoeIuB9YIWn33OnFwG8azTtr5UhzObC/pK48319MqtduiNIL+JA0n7QDajZ/SOvVCfn3CUCjLwEcF0mHAu8GjoiIoSbT7lrx9wgaXMcAIuLXEbFNRPTkdW0l6QL2/Q3mvV3F36Np7lUhFwIH5/HsRrrppJmWjQ8Bbo+IlU2kgXSQ+sL8+2CaPCCpWM9mAO8FvlBjuFr7jlLWsbrKvhuhXT6kOvc7SEcPC5tMewHptP5x0ob0uibSHkiqprsZuDF/Dm8w7f8BfpXT3kKNO2MaGM9BNHnXGek6y035c2sL8+zZwJJc9guBLZtI2wWsAZ7WwrR+gLSjvAX4OvmOpAbTXk0KiDcBL25lvQC2Bq4g7XyuALZqIu3R+fdfgQeAS5tIeyfpOuTwOlb1zrEaab+b59fNwCXADq1uC9S5U7FG3l8Hfp3zvhjYrom0GwGLc9lvAA5upszAucAbW1jGBwJL83ryS2DfJtOfRtoX3QGcRW7BpUraqvuORtexyfy4CRozMyuUq87MzKxQDjRmZlYoBxozMyuUA42ZmRXKgcbMzArlQGPWAElPaGSr0T0tjOMoSXtMfOnM2tussgtg1iH+HKmpn/E4CvgBTTycKmlWbGgM06wj+YzGrEWS9s2NNS6VdGlFsx9vkHS90nuCvptbIvgH0lP1H81nRLtIulL5vTKS5uQmWpB0oqRvS7qE1GjpJrlByetzI6QNtyxu1g4caMwas3FFtdn3cxtT/w0cG6lNqnOARXnY70XEcyLiWaQmbl4XEf9LerL9XRHx7Iio2X5VdgBwQkQcTGqU8qcR8RzSO1Y+2mJr12alcNWZWWNGVJ3lVrL3Ai7LDQLPJDUlArCXpA8BWwCbkl741azLImL4PSUvJTWA+s78/6nAfJpop82sTA40Zq0RcGtEVHsF9bnAURFxk6QTSW3JVbOODbUKo18p/eiovP4pGnjpmVk7ctWZWWt+C8yVdACk5tol7Zn7bQbcl6vXeivSPJL7DRsE9s2/j62T16XAW4bfpSJp7/EX32zyONCYtSDSK7+PBT4s6SZSy7nD77b5v6RWey9jZJP63wDelS/o70J6++Ipkv6X+u9J+SDpNd03S7ol/zfrGG692czMCuUzGjMzK5QDjZmZFcqBxszMCuVAY2ZmhXKgMTOzQjnQmJlZoRxozMysUP8fdHdoFVrEfgYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create horizontal positions for scatter plot\n",
    "x_pos = np.arange(x_test.shape[2])\n",
    "\n",
    "# select random record\n",
    "rec_num = random.randint(0, y_pred.shape[0])\n",
    "\n",
    "# plot predicted points\n",
    "plt.scatter(x_pos, x_test[rec_num,0,:], c=\"#ff0000\")\n",
    "plt.scatter(x_pos, y_pred2[rec_num,0,:], c=\"#0000ff\")\n",
    "\n",
    "# label graph\n",
    "plt.xlabel(\"Feature\")\n",
    "plt.ylabel(\"Value\")\n",
    "plt.xticks(range(x_test.shape[2]))\n",
    "plt.title(\"Compare original to reconstruction for day 0 of seq num \" + str(rec_num))\n",
    "\n",
    "if y_labels_seq[rec_num] == 0:\n",
    "    print(\"This record is from a normal disk.\")\n",
    "else:\n",
    "    print(\"This record is from an anomalous disk.\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like before, we need to determine a \"threshold\" of difference in detecting an anomaly. We'll use the same technique on the new data to determine our threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Suggested Threshold: 0.1596888519125517\n"
     ]
    }
   ],
   "source": [
    "# Calcuate MSE for all sequences and predictions\n",
    "import statistics\n",
    "\n",
    "# determine median mse to split data into normal vs anomaly\n",
    "pred_mse2 = []  # calculated mse for test data\n",
    "pred_res2 = []  # result: 0 normal, 1 anomaly\n",
    "\n",
    "for i in range(len(x_test)):\n",
    "    pred_mse2.append(mean_squared_error(y_pred2[i,:,:], x_test[i,:,:]))\n",
    "\n",
    "# Find number of disks that we know are normal and use that value as splitting point for data\n",
    "num_normal_disks = list(y_labels_seq).count(0)\n",
    "\n",
    "sort_mse2 = pred_mse2.copy()\n",
    "sort_mse2.sort()\n",
    "suggested_threshold2 = sort_mse2[num_normal_disks]\n",
    "print(\"Suggested Threshold:\", suggested_threshold2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Used threshold: 0.1596888519125517\n",
      "Resulting in the following classification of predicted data:\n",
      "- Normal Disks:    601\n",
      "- Anomalous Disks: 233\n"
     ]
    }
   ],
   "source": [
    "# Routine to specify whether or not a sequence is anomalous\n",
    "\n",
    "# specify override threshold or leave 0 to use suggested threshold\n",
    "override_threshold2 = 0\n",
    "\n",
    "if override_threshold2 == 0:\n",
    "    threshold2 = suggested_threshold2\n",
    "else:\n",
    "    threshold2 = override_threshold2\n",
    "\n",
    "for mse in pred_mse2:\n",
    "    if (mse <= threshold2):\n",
    "        pred_res2.append(0)\n",
    "    else:\n",
    "        pred_res2.append(1)\n",
    "        \n",
    "print(\"Used threshold:\", threshold2)\n",
    "print(\"Resulting in the following classification of predicted data:\")\n",
    "print(\"- Normal Disks:   \", pred_res2.count(0))\n",
    "print(\"- Anomalous Disks:\", pred_res2.count(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROC Curve\n",
    "\n",
    "The ROC curve plots the False Positive rate along the horizontal axis and the True Positive rate along the vertical axis. False Positive rate and True Positive rate will change based on the various threshold values. \n",
    "\n",
    "The threshold is high at the y-intercept, resulting in a 0% False Positive rate and a low True Positive rate. As the threshold decreases, the True Positive rate will increase but so will the False Positive rate. \n",
    "\n",
    "Perfect classification occurs if the distribution of anomalous disks and normal disks are completely separated. This would yield a point in the upper left corner of the ROC space, where the False Positive rate is 0% and the True Positive rate is 100%. \n",
    "\n",
    "https://en.wikipedia.org/wiki/Receiver_operating_characteristic\n",
    "\n",
    "Check the ROC curve of the LSTM model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUZfbA8e9JgUAINSDSe5dmECw06cXFjspiLxGxILr4w4aKi1hApAiIu+iKfVlRUFBUUEGQIiBdikKQ3iGBtPP7496EIaRMQqYkOZ/nmSdz57YzNzP3zPu+976vqCrGGGNMVkICHYAxxpjgZonCGGNMtixRGGOMyZYlCmOMMdmyRGGMMSZbliiMMcZkyxJFISEi60SkU6DjCDQRmSwiT/t5n9NFZKQ/9+krIjJARL7O47qF9jMoIioi9QIdR6CI3UeR/0TkD+ACIAU4AcwFBqvqiUDGVdiIyO3A3ap6RYDjmA7EqepTAY5jBFBPVf/uh31NJwjes7+IiAL1VXVLoGMJBCtR+M5VqloKaAm0Av4vwPHkmoiEFcV9B5IdcxOUVNUe+fwA/gC6eky/DMzxmG4HLAaOAKuBTh7zygP/Bv4CDgOfeczrC6xy11sMNM+4T6AKkACU95jXCjgAhLvTdwIb3O3PA2p6LKvAA8DvwPYs3t/fgHVuHAuAxhni+D9gvbv9fwMRuXgPw4A1wGkgDHgC2Aocd7d5jbtsY+AUZ0ptR9zXpwMj3eedgDhgKLAP2A3c4bG/CsAXwDFgGTAS+Cmb/+sVHv+3ncDtHvucCMxx41wK1PVYb5y7/DFgBdDeY94I4FPgPXf+3cAlwM/ufnYDE4BiHus0Bb4BDgF7geFATyARSHKPx2p32TLA2+52drnvMdSddzuwCBjrbmuk+9pP7nxx5+0Djrr/l2bAve5+Et19fZHxcw+EunGl/e9WANWzOK6Zfh+Ay3A+t9Xd6RbuMo3c6Uw/G5m8tyPANnd7t7v/i33AbR7LTwcmu8f1OLCQc78X9dznxYFXgR3u8Z8MlAj0ecen57RAB1AYHxm+MNWA34Bx7nRV4CDQG6dE182drujOnwN8BJQDwoGO7uut3Q93W/dLeJu7n+KZ7PM74B6PeF4BJrvPrwa24Jxow4CngMUey6r7ZSmf2YcfaACcdOMOB/7hbq+YRxxrgeruNhZx5sTtzXtY5a5bwn3tBpzkFwL0d/d9oTvvdjKc2Dk3USQDz7ux9gbigXLu/A/dR0mgCc4JJNNEAdTAOYHc7G6rAtDSY5+HcE7wYcAM4EOPdf/uLh+Gk7T24CZPnESR5P5fQoASwMU4J88woBZOUn/EXT4K56Q/FIhwp9t6bOu9DHF/BkwBIoFKwC/AfR7HLxl40N1XCc5OFD1wTvBlcZJGY49jn36cs/jcP47zuW/ortsCqJDJcc3p+/Aizue5BE6iGuyxbk6fjWTgDpzP2kicE/tEnBN9d/f/Wcrj/RwHOrjzx3l+Fjg7UbwOfI7z+Y7C+bExKtDnHZ+e0wIdQGF8uF+YE+4HT4FvgbLuvGHAfzIsPw/npHkhkIp7IsuwzJvACxle28SZROL5Jb0b+M59LjgnwA7u9FfAXR7bCME5edZ0pxW4Mpv39jTwcYb1d3HmV+AfQKzH/N7A1ly8hztzOLargH7u89vJOVEkAGEe8/fhnIRDcU7QDT3mZVmiwCkl/S+LedOBaRne88Zs3sNhoIX7fATwQw7v+ZG0feMkql+zWG4EHokCp53sNB4J313/e4/jtyPDNtKPKXAlsNk9XiFZHecMn/u0z+CmtP9TDu8ty++D+zwcJ1n9htPWJ7n4bPzuMe8inM/2BR6vHeTsZO+Z3EvhlFbTSjMK1MP5Pp3k7BLjpWRR+i4sD2uj8J2rVTUK52TVCIh2X68J3CAiR9IeOFUaF+L8kj6kqocz2V5NYGiG9arj/KLK6FPgUhGpgvMLSYEfPbYzzmMbh3A+/FU91t+ZzfuqAvyZNqGqqe7yWa3/p0eM3ryHs/YtIreKyCqP5Ztx5lh646CqJntMx+OcBCri/Ir23F9277s6TjVHVvZksg8ARGSoiGwQkaPueyjD2e8h43tuICKzRWSPiBwD/umxfE5xeKqJc6Ld7XH8puCULDLdtydV/Q6n2msisFdEpopIaS/37W2c2X0fUNUknJN4M+A1dc/M4NVnY6/H8wR3exlfK+UxnX4s1Lnw5BDnfr8q4pRAV3jsd677eqFlicLHVHUhzgf9VfelnTi/oMp6PCJV9SV3XnkRKZvJpnYCL2ZYr6SqfpDJPo8AXwM3ArcAH3h8wXbiVD14bqeEqi723EQ2b+kvnC83ACIiOCeFXR7LVPd4XsNdx9v34HkiqAm8BQzGqbYoi1OtJV7EmZP9OFUT1bKIO6OdQN3c7kRE2uP8ar4Rp6RYFqe+XzwWy/g+3gQ24lxlUxqnrj9t+eziyLidnTglimiP411aVZtms87ZG1R9Q1UvxmkXaYBTpZTjejnEmXG5rL4PiEhV4Fmctq7XRKS4+3pOn428SP//i0gpnKqlvzIscwAnwTT1iLeMOheuFFqWKPzjdaCbiLTEabS8SkR6iEioiESISCcRqaaqu3GqhiaJSDkRCReRDu423gJiRaStOCJFpI+IRGWxz/eBW4Hr3OdpJgP/JyJNAUSkjIjckIv38jHQR0S6iEg4Tl35aZzGyDQPiEg1ESmPc5L7KI/vIRLnhLTfjfUOnF+NafYC1USkWC7iB0BVU4CZwAgRKSkijXCOV1ZmAF1F5EYRCRORCu7/MydROAlpPxAmIs8AOf0qj8Jp2D7hxnW/x7zZQGUReUREiotIlIi0deftBWqJSIj7Hnfj/GB4TURKi0iIiNQVkY5exI2ItHH/V+E41S1pFw+k7atONqtPA14Qkfru/7q5iFTIZLksvw/uj5DpOI3xd+G0zbzgrpfTZyMveovIFe7n6QVgqaqeVeJyS9BvAWNFpJK776oi0uM89x3ULFH4garuB94FnnY/eP1wTqD7cX5RPc6Z/8VAnLrzjTj16Y+421gO3INTFXAYpwH59mx2+zlQH9irqqs9YvkfMBr40K3WWAv0ysV72YTTODse59fVVTiXAid6LPY+zglqm/sYmZf3oKrrgddwrgDai1PPvMhjke9wrr7aIyIHvH0PHgbjVAPtAf4DfICT9DKLZQdO28NQnCqJVTgNtDmZh5P8N+NUw50i+yougMdwSoLHcU5KaYkWVT2O0+B7lRv370Bnd/Yn7t+DIrLSfX4rUIwzV6F9ilut44XS7v4Pu7Ef5EzJ+G2giVv98lkm647B+VHxNU7SexunQfosOXwfHsJpZ3naLRHfAdwhIu29+Gzkxfs4pZdDOBcUDMhiuWE4n90l7ndoPk6jfaFlN9yZfCXOzYZ3q+r8QMeSWyIyGqisqrcFOhbjX1LEbiDMLStRmCJLRBq5VSIiIpfgVG/8L9BxGRNs7E5MU5RF4VQ3VcGp5nsNmBXQiIwJQlb1ZIwxJltW9WSMMSZbBa7qKTo6WmvVqhXoMIwxpkBZsWLFAVXN042BBS5R1KpVi+XLlwc6DGOMKVBE5M+cl8qcVT0ZY4zJliUKY4wx2bJEYYwxJluWKIwxxmTLEoUxxphsWaIwxhiTLZ8lChH5l4jsE5G1WcwXEXlDRLaIyBoRae2rWIwxxuSdL0sU03EGfM9KL5xusOvjDNb+pg9jMcaYIisxMSXnhbLhsxvuVPUHEamVzSL9gHfdfuaXiEhZEbnQHWzFGGNMXs3sA9u/BGDcj22ZtvT8KmwCeWd2Vc4ewCXOfe2cRCEi9+KUOqhRo4ZfgjPGmALFIzl4anHhXtbvPb8hvQPZmJ3Z2LaZdmWrqlNVNUZVYypWLNRjmBtjTO55JImdR0rz5uIYqN0bhiqd3tzOlq2PnNfmA1miiOPsweyrce5A5sYYY7LjJonklBDeWH87z3xal5Mnk2j2yO20dxepXbvcee0ikInic2CwiHwItAWOWvuEMcbk0vYvWfpnVe774hZW/xEJJHHddY2pU+f8koMnnyUKEfkA6AREi0gczqDl4QCqOhn4Emew+i1APM7A6cYYYzLKov3hcHwEw7/qw5QlMagKtWqVZcKEXvTp0yBfd+/Lq55uzmG+Ag/4av/GGFPgZZEg0jz3TScm/9yGsNBUHnv8Cp5+uiMlS4bnexgFbjwKY4wpMjyTRO3ecO0ckpNTCQtzrkN66rZ4tt/1OS++eCXNmlXyWRiWKIwxJthkLEkMVU6dSmb0cwv47LNNLF16N8WKhRIdXZJZs27yeTiWKIwxJtCyq2Kq3Ztvv93G/ffP4fffDwEwb94Wrrqqod/Cs04BjTEmkLJKErV7s/fvx/n7zLvo2vU//P77IRo3jmbBgtv8miTAShTGGBMYGROE2waR5r331vBgo4kcOXKKiIgwnnmmA0OHXkaxYqF+D9UShTHG+FsOSQIgNVU5cuQUPXvWY+LE3vl6X0RuWaIwxhh/yKyKySNBnDiRyM8/76Rbt7oADBzYnCpVoujSpTYimfV45D/WRmGMMb6WQ5L47LONNG48kauu+oAtW5wGaxGha9c6AU8SYCUKY4zJf9k0UHtWMf355xEeemgun3++CYCYmCqcPp3sryi9ZonCGGPyKoc7p8/ikSSSklJ4/fUljBixkPj4JKKiivHPf3bh/vtjCA0NvooeSxTGGJNRbhJAVjJpoE7z0ENfMXnyCgBuvLEpY8f2oEqVqPPbnw9ZojDGmIxykySySQhZeeSRdixc+CdjxvSgZ896uQzO/yxRGGNMmky6zjhfqsp7763hyy+38P771yIiNGwYzdq1gwgJCXxDtTcsURhjih5vqpZq9z7v3WzadID775/D99//ATiXvPbuXR+gwCQJsERhjCkK8tjonFcJCUmMGvUTo0cvIjExhQoVSvDaa93p1Sv4q5kyY4nCGFP4eFtiOM+EkJn587cRGzubrVsPA3DXXa0YPborFSqUzPd9+YslCmNM4ZFDL6y+SAwZLV68k61bD9O0aUUmT+7LFVfU8Pk+fc0ShTGmYPPy5jZfSUlJZcuWQzRsGA3AsGGXEx1dkrvvbh2QDvx8wRKFMaZgCnCCAPj1193Exs5h27bDbNo0mPLlS1C8eBiDBrXxy/79xRKFMaZgyqH3VV86fvw0zzzzPW+88QupqUrVqlFs3XqI8uWr+i0Gf7JEYYwp2PLhXgdvqSozZ27g4YfnsmvXcUJChCFD2vHcc52Iiirutzj8zRKFMaZgyY/uNfLokUfm8sYbvwDQpk0VpkzpS6tWFwYkFn8Kvt6njDEmK5kN+ONH11zTmDJlijNxYm9+/vmuIpEkwEoUxpiCwIsR4Xzhp5928P3323n66Y4AdOpUix07hlC6dOGtZsqMJQpjTPDKYcAfXzl4MJ5hw+bz9tu/AtClSx0uu6w6QJFLEmCJwhgTbAJ42auq8u67q3nssW84cCCe8PAQnnjiClq1quzT/QY7SxTGmOASgBIEwIYN+7n//jksXPgnAJ0712LSpD40ahTt830HO0sUxpjAye4KJj9e9gowZszPLFz4JxUrlmTMmB4MGHBRUIxXHQwsURhj/C+nS1z9dDXT0aOnKFMmAoBRo7oSGVmMZ57pSPnyJfyy/4LCEoUxxv8CeFc1wF9/HWfIkHmsWbOX1atjKVYslOjokrz+ek+/xlFQWKIwxgSOn6uXUlJSmTRpGU8++R3HjydSsmQ4K1fupl27an6No6CxRGGM8Y8A3lENsGLFX9x332xWrNgNwN/+1pDx43tRo0aZgMVUUPg0UYhIT2AcEApMU9WXMswvA7wH1HBjeVVV/+3LmIwxfpTT+BB+MmLEAl544QdSU5Xq1Uszfnwv+vVr5Lf9F3Q+SxQiEgpMBLoBccAyEflcVdd7LPYAsF5VrxKRisAmEZmhqom+issYk49yW0oIQHsEQJ065RCBoUMvZcSITpQqVczvMRRkvixRXAJsUdVtACLyIdAP8EwUCkSJcw1aKeAQkOzDmIwx+cXbJBGA5LBt22GWLdtF//7NABg4sDlt21ZNH1zI5I4vE0VVYKfHdBzQNsMyE4DPgb+AKKC/qqZm3JCI3AvcC1CjRsEfVtCYAi1A/S55IzExhVdfXcwLL/yAqnLxxVWoV688ImJJ4jz4svfYzO5UyXiJQw9gFVAFaAlMEJHS56ykOlVVY1Q1pmLFivkfqTHGe0GaJH744U9atpzMk09+x6lTyVx/fZMi2S+TL/iyRBEHVPeYroZTcvB0B/CSqiqwRUS2A42AX3wYlzEmtzKrZvLzpa1ZOXAgnscf/4bp01cBUL9+ed58sw9dutQJcGSFhy8TxTKgvojUBnYBNwG3ZFhmB9AF+FFELgAaAtt8GJMxJrey6sE1SMTGzua//91A8eKhDB/enn/843IiIuzK//zks6OpqskiMhiYh3N57L9UdZ2IxLrzJwMvANNF5DecqqphqnrAVzEZU6Sd730MQVTNlJqqhIQ4tdsvvnglCQnJvP56D+rXrxDgyAoncWp9Co6YmBhdvnx5oMMwpuB57Tw6uAuSJBEfn8QLLyxk1aq9fPnlLdZpXy6IyApVjcnLulY+M6YwyE1pIUjaFnJrzpzNDB78FX/8cQQR+OWXXbRta11v+IMlCmMKA2+TRBC1LXgrLu4YDz88l5kzNwDQosUFTJ7c15KEH1miMKagm9nnzPMCWlrIyqRJyxg2bD4nTiQSGRnOCy905sEH2xIW5ssr+01GliiMCTZ5bXQugKWFnBw4EM+JE4lcc00jxo3rSfXq1oFfIFiiMCaQ8qtH1SBpbD5fR46cYuPGA+ndfg8bdjmXXFKVnj3rBTiyos0ShTH+FMT9IwWSqvLRR+sYMmQeKSmpbNw4mPLlS1C8eJgliSBgicIYX/ImMRSxpJDRli2HeOCBL/n6660AXHZZdY4ePWXDkQYRSxTG+EJO4zAU4cSQ5vTpZF5+eREvvvgjp0+nUK5cBC+/3I0772yVfjOdCQ5eJwoRiVTVk74MxpgCL6vuLiwxnKN//0+ZNWsTALfe2oJXXulGpUqRAY7KZCbHRCEilwHTcMaLqCEiLYD7VHWQr4MzJqjlVK1kCSJbjzzSjk2bDjJpUm86d64d6HBMNrwpUYzF6Q78cwBVXS0iHXwalTEFgVUteS01VfnXv35lw4b9vPZaDwA6darF2rX3Expq90QEO6+qnlR1Z4Y+VVJ8E44xBUDGkkQhu8ktv/32215iY+eweLEzjtmtt7agRYvKAJYkCghvEsVOt/pJRaQY8BCwwbdhGRMEvL1iyWTq5MlEnntuIWPG/ExKilK5cilef70HzZtfEOjQTC55kyhigXE4Q5vGAV8D1j5hCj9rf8izL77YxODBX7Fjx1FE4IEH2vDii1dSpkxEoEMzeeBNomioqgM8XxCRy4FFvgnJmACzqqXz9tlnG9mx4yitWlVmypS+tGlTNdAhmfPgTaIYD7T24jVjCo7c3CFtcpScnMquXceoWbMsAKNHd6NVqwuJjY2xDvwKgSwThYhcClwGVBSRRz1mlcYZsc6Y4FWIRnMLdkuWxBEbO5vTp1NYvTqWYsVCiY4uyeDBlwQ6NJNPsitRFMO5dyIMiPJ4/RhwvS+DMibPcpMgLBmcl8OHExg+/FumTFmBKtSqVZY//jhCgwY2HGlhk2WiUNWFwEIRma6qf/oxJmO8Z11l+J2q8sEHaxkyZB779p0kLCyExx+/jKee6kDJkuGBDs/4gDdtFPEi8grQFEi/ZEFVr/RZVMbkxBJEwAwYMJMPPlgLQPv2NXjzzT40bVopwFEZX/ImUcwAPgL64lwqexuw35dBGZMjzyRhicGvevasx9dfb+WVV7px220trQO/IsCbRFFBVd8WkYc9qqMW+jowY7JUiIf+DEbz529j69ZD3HdfDAADBzanb98G1g14EeJNokhy/+4WkT7AX4CNam4CJ600YZeu+tTevSd49NGvef/93yhePJSuXetQt255RMSSRBHjTaIYKSJlgKE490+UBh7xaVSmaPP2yiWrbvKJ1FRl6tQVPPHEfI4ePU1ERBjPPNPBxqsuwnJMFKo62316FOgM6XdmG+MbdiNcwKxevYf77pvN0qW7AOjVqx4TJvSmTp1yAY7MBFJ2N9yFAjfi9PE0V1XXikhfYDhQAmjlnxBNkWXtD373j3/MZ+nSXVSpEsW4cT257rrGZOg52hRB2ZUo3gaqA78Ab4jIn8ClwBOq+pk/gjNFkGdDtfE5VSU+PonIyGIAvPFGTyZPXs5zz3WmdOniAY7OBIvsEkUM0FxVU0UkAjgA1FPVPf4JzRQ5nm0TVrXkc3/+eYQHH/yKkyeTmD9/ICJCw4bRjB3bM9ChmSCTXaJIVNVUAFU9JSKbLUmY8+btGA/WUO0zSUkpjB27hOeeW0h8fBJRUcX4/fdD1vWGyVJ2iaKRiKxxnwtQ150WQFW1uc+jM4VHbnprtSThM4sW7SA2dg5r1+4DoH//powZ04MqVaJyWNMUZdklisZ+i8IUPtbFRtB58MEvmTBhGQB16pRj4sTe9OxZL8BRmYIgu04BrSNA4x0rLRQIFStGEh4ewrBhlzN8eHtKlLAO/Ix3vLnhLs9EpCfOMKqhwDRVfSmTZToBrwPhwAFV7ejLmMx5yO0YD5YYAmrjxgPs2HGU7t3rAjBs2OXceGNTGjWKDnBkpqDxWaJw78OYCHTDGWt7mYh8rqrrPZYpC0wCeqrqDhGxLiiDkTVAFygJCUn8858/Mnr0IsqWjWDjxsGUL1+C4sXDLEmYPPEqUYhICaCGqm7KxbYvAbao6jZ3Gx8C/YD1HsvcAsxU1R0AqrovF9s3vpZZgrCEENS+/norgwbNYevWwwD87W8NsfvlzPnKMVGIyFXAqzgj3tUWkZbA86r6txxWrQrs9JiOA9pmWKYBEC4iC3BG0Runqu96GbvxpYxJwhJEUNu9+zhDhszjo4/WAdC0aUUmT+7LFVfUCHBkpjDwpkQxAqd0sABAVVeJSC0v1svsd0zGPhnCgIuBLjjdgvwsIktUdfNZGxK5F7gXoEYN++D7XMYb3yxBBL1rr/2YJUviKFEijBEjOjFkSDvCw21oe5M/vEkUyap6NA/9vcThdAGSphpOF+UZlzmgqieBkyLyA9ACOCtRqOpUYCpATEyMdQDkK1aKKFBUNb0fppde6sKrr/7M+PG9qFWrbIAjM4WNN4lirYjcAoSKSH3gIWCxF+stA+qLSG1gF3ATTpuEp1nABBEJw6naaguM9TZ4k0+sLaJAOX78NM888z0nTyYxdepVAHTsWIuOHWsFNjBTaHmTKB4EngROA+8D84CROa2kqskiMthdPhT4l6quE5FYd/5kVd0gInOBNUAqziW0a/P2VkyuWYIoUFSVmTM38PDDc9m16zhhYSEMH97eShDG50Q1+5ocEWmlqr/6KZ4cxcTE6PLlywMdRuHwmkd1oiWIoLZ9+2EGD/6KL7/8HYBLLqnK5Ml9aNXqwgBHZgoKEVmhqjF5WdebEsUYEbkQ+AT4UFXX5WVHJojZuA9BS1V5+eVFPPfcQhISkilTpjijRnXh3nsvJjQ0JNDhmSLCmxHuOotIZZxBjKaKSGngI1XNsfrJBDEb96FAEBE2bz5IQkIyN9/cjDFjelC5cqlAh2WKmByrns5aWOQi4B9Af1Ut5rOosmFVT/kkrdrJqpyCzoED8ezZc4JmzSqlT//66266dasb4MhMQXY+VU85ll1FpLGIjBCRtcAEnCuequVlZyYIzOxzdtuEJYmgoapMn76KRo0mcMMNn5CYmAJAdHRJSxImoLxpo/g38AHQXVUz3gdhCpqM90mYoLBhw35iY+fwww9Op80tWlTm8OEELrjAqplM4HnTRtHOH4EYH8iuMz9rwA4K8fFJvPjiD7zyymKSklKpWLEkY8b0YMCAi8jDTa7G+ESWiUJEPlbVG0XkN87uesNGuCsoshs4yAScqnLlle+wdOkuAO6772JGjepCuXIlAhyZMWfLrkTxsPu3rz8CMfnM86omKz0EJRFh0KA2xMcnMWVKXy69tHrOKxkTAFk2ZqvqbvfpIFX90/MBDPJPeCbPPDv1M0EhJSWV8eOXMmbMz+mvDRzYnBUr7rUkYYKaN3fsdMvktV75HYjxEbuqKSgsX/4XbdtO46GH5jJ8+Lf89ddxwClVWC+vJthl10ZxP07JoY6IrPGYFQUs8nVgxhQGR4+e4qmnvmPixGWoQvXqpRk/vhdVqkQFOjRjvJZdG8X7wFfAKOAJj9ePq+ohn0Zlzo/ddR1wqsonn6znkUfmsnv3CUJDhSFD2vHss50oVSog96oak2fZJQpV1T9E5IGMM0SkvCWLIJTZeBImYKZMWcHu3Sdo164akyf3oUWLyoEOyZg8yalE0RdYgXN5rOdF3QrU8WFcJi9s0KGAOn06mSNHTnHBBaUQESZN6s2CBX9wzz0XExJi90SYgivLRKGqfd2/tf0XjsmTjCUJuxzW7xYu/IPY2DlUqRLF/PkDEREaNoymYcPoQIdmzHnzpq+ny0Uk0n3+dxEZIyI2cHWwsOqmgNq//yS33/4ZnTq9w8aNB9i58yh7954MdFjG5CtvLo99E4gXkRY4Pcf+CfzHp1EZ73gmidq9nZKEVTf5RWqq8vbbK2nUaCLvvLOa4sVDee65TqxZc791A24KHW86BUxWVRWRfsA4VX1bRG7zdWAmBxmThCUIv1FVevR4j/nztwHQtWsdJk3qTf36FQIcmTG+4U2iOC4i/wcMBNqLSCgQ7tuwTKZsjOugICK0b1+D337by9ixPbjppmbWgZ8p1LwZM7sycAuwTFV/dNsnOqnqu/4IMKMiPXDRaxlORpYk/GbOnM0kJaVy9dWNAOcKp4SEZMqWjQhwZMZ4x6djZqvqHhGZAbQRkb7AL4FKEkWWXdUUMHFxx3j44bnMnLmB6OiSdOhQk/LlS1C8eBjFi3tTIDem4Mvxky4iNwKvAAtw7qUYLyKPq+qnPo7NZFXVZHwuOdnpwO+ZZxZw4kQikZHhDB9+BaVLFw90aMb4nTc/iZ4E2qjqPgARqQjMByxR+Iq1RQTUL7/s4r77ZrNq1R4ArrmmEXkHeSYAACAASURBVOPG9aR69TIBjsyYwPAmUYSkJQnXQby7rNbkRlaj0VmC8KvUVOWOO2axfv1+atQow4QJvbjqqoaBDsuYgPImUcwVkXk442YD9AeyGDrNeCW7IUrTWILwG1Xl9OkUIiLCCAkRJk7szVdf/c4zz3QkMtI68DMmx6ueAETkWuAKnDaKH1T1f74OLCuF4qqnjFcvpbHk4Hdbthxi0KA5VK9emrff7hfocIzxGZ9c9SQi9YFXgbrAb8BjqrorbyGaTNnVSwFz+nQyo0cv4p///JHTp1MoX74EL78cT4UKJQMdmjFBJ7u2hn8Bs4HrcHqQHe+XiIzxse++207z5pN59tkFnD6dwm23tWDjxgcsSRiThezaKKJU9S33+SYRWemPgAo1b9omjM+kpKRyxx2z+M9/nAEbGzaswOTJfenUqVZgAzMmyGWXKCJEpBVnxqEo4TmtqpY4cst6eQ2o0NAQwsJCiIgI46mn2vPYY5fZTXPGeCHLxmwR+T6b9VRVr/RNSNkrsI3ZnqUJa5vwm99+28upU8m0aVMVgIMH4zly5BR165YPcGTG+JdPGrNVtXPeQzLpbLyIgDh5MpERIxYwduwS6tevwOrVsRQrFkqFCiWtLcKYXLJyty9lliTs8lef+/zzTTz44Ffs2HEUEejatTZJSSkUKxYa6NCMKZB8mihEpCcwDggFpqnqS1ks1wZYAvQvVH1I2XgRfrVjx1EeeugrZs3aBEDr1hcyZUpfYmKqBDgyYwo2nyUKd9yKiUA3IA5YJiKfq+r6TJYbDczzVSwBZ0nC51JSUunUaTrbtx8hKqoYI0deyaBBbQgLs95mjDlf3vQeK8AAoI6qPu+OR1FZVX/JYdVLgC2qus3dzodAP2B9huUeBP4LtMlt8MaoKiJCaGgII0Z04osvNvP66z2oWrV0oEMzptDw5ufWJOBS4GZ3+jhOSSEnVYGdHtNx7mvpRKQqcA0wObsNici9IrJcRJbv37/fi10H2Mw+WXfTYfLF4cMJxMbO5p///DH9tYEDm/PJJzdYkjAmn3lT9dRWVVuLyK8AqnpYRLzpKS2zM2XG60JfB4apakp2Q0mq6lRgKjiXx3qxb//K7kY6u8opX6kq77//G48++jX79p0kKqoYgwdfQpkyETYcqTE+4k2iSHLbERTSx6NI9WK9OKC6x3Q14K8My8QAH7pf8Gigt4gkq+pnXmw/OFj34H6zefNBBg2aw7ffbgegffsavPlmH8qUseFIjfElbxLFG8D/gEoi8iJwPfCUF+stA+qLSG1gF3ATztjb6VS1dtpzEZkOzC6wScISg88kJ6cycuQPjBr1E4mJKVSoUIJXXunG7be3tFKEMX7gzZjZM0RkBdAFpzrpalXd4MV6ySIyGOdqplDgX6q6TkRi3fnZtksUCJYk/CI0VPjxxx0kJqZw550tGT26G9HRdtOcMf6S43gU7lVO51DVHT6JKAdB1YVHWoO1dcmR7/buPcGpU8nUrFkWgN9/P8ju3Sfo0KFmgCMzpmDySRceHubgtE8IEAHUBjYBTfOyQ2Oyk5qqTJ26gieemE9MTBW++WYgIkL9+hWoX79CoMMzpkjypurpIs9pEWkN3OeziEyRtWrVHmJjZ7N0qTM+VrFioZw4kUhUVPEAR2ZM0ZbrO7NVdaXb5UbRNrNPoCMoNI4fP82zzy5g3LilpKYqVapEMW5cT667rrE1VhsTBLy5M/tRj8kQoDVQAO568zHPhmyTZ4mJKbRuPZUtWw4REiI8/HBbnn++M6VLWynCmGDhTYkiyuN5Mk6bxX99E04BZFc7nZdixUIZOLA5X3yxmcmT+3DxxdaBnzHBJttE4d5oV0pVH/dTPKaQS0pKYezYJdSoUYabbmoGwBNPXMGTT7YnNNQ68DMmGGWZKEQkzL0XorU/AzKF16JFO4iNncPatfuoWLEkffs2oFSpYjZOhDFBLrsSxS847RGrRORz4BPgZNpMVZ3p49iCU3b9OplMHTqUwLBh3zBt2q8A1KlTjkmTelOqlDddhhljAs2bNorywEHgSs7cT6FA0UsUNqxprqgq//nPGoYO/ZoDB+IJDw9h2LDLGT68PSVKhAc6PGOMl7JLFJXcK57WciZBpCkatyJbh3/nJSkplVGjfuLAgXg6dqzJm2/2oXHjioEOyxiTS9klilCgFN51F144WZLItYSEJBITUyhTJoJixUKZOrUv27Yd5tZbW9g9EcYUUNklit2q+rzfIgkmGUsS1peTV+bN28KgQV/SqVNN3n67HwDt29ekfXvrn8mYgiy7RFF0f/5ZO0Su7N59nCFD5vHRR+sAiIwMJz4+iZIlrR3CmMIgu0TRxW9RBAsrSeRKSkoqb765nCef/I5jx05TokQYI0Z0YsiQdoSH2yWvxhQWWSYKVT3kz0ACzq5oypVTp5Lp0OHfLFvmDFrYt28Dxo/vRa1aZQMcmTEmv+W6U8BCywYhypWIiDCaNavE7t0neOONnlx9dSNrrDamkLJEkZEliUypKjNnbuCCC0pxxRXOWFZjxvQgNFSsG3BjCrmilyjszupc2779MIMHf8WXX/5Oo0bRrFp1H8WLh1G2bESgQzPG+EHRSxTZJQlrlzhLYmIKr722mBde+IGEhGTKlCnOww+3JSzMOu8zpigpeokijV3RlK0ff/yT2Ng5rF/vDD1yyy0X8dpr3alcuVSAIzPG+FvRShQ2Kp1XEhKSuP76T9i37yT16pVn0qTedOtWN9BhGWMCpGglChuVLkuqSkqKEhYWQokS4YwZ053Nmw/yf//XnoiIovUxMcacreicATxLE3Zl01nWr99PbOxsunWrw9NPdwRgwIDmAY7KGBMsik6rpJUmzhEfn8Tw4d/SosVkfvxxB9Om/crp08mBDssYE2SKRonCShPn+Oqr33nggS/Zvv0IAPfddzGjRnWhePGi8ZEwxnivaJwVrDSR7uTJRG6/fRaffroegObNL2Dy5D5cemn1AEdmjAlWRSNRpLHSBCVLhnPoUAKRkeE891wnHn64nd0XYYzJVtFKFEXU8uV/UbZsBPXqlUdEmDbtKkJDQ6hRo0ygQzPGFACF/6dkEb534ujRUzz44JdccslbxMbORtW5ybB27XKWJIwxXivcJQrPfp2KUPuEqvLxx+t45JF57NlzgtBQoXXrC0lOTrVxIowxuVa4E0UR7Dp869ZDPPDAl8ybtxWASy+txuTJfWne/IIAR2aMKagKb6IogpfEHj9+mpiYtzhy5BRly0YwenRX7r67NSEhNk6EMSbvfJooRKQnMA4IBaap6ksZ5g8AhrmTJ4D7VXX1ee+4iFY5RUUVZ8iQdmzZcohXX+1OpUqRgQ7JGFMI+CxRiEgoMBHoBsQBy0Tkc1Vd77HYdqCjqh4WkV7AVKBtrneW1RgThbzKaf/+kzz++Dd06VKbgQNbAPD00x1spDljTL7y5VVPlwBbVHWbqiYCHwL9PBdQ1cWqetidXAJUy9OeiliSSE1Vpk1bScOGE3jnndU8+eR3JCWlAFiSMMbkO19WPVUFdnpMx5F9aeEu4KvMZojIvcC9ADVq1Mh6C0VgjIm1a/cRGzubRYucQ9u1ax0mTeptVzMZY3zGl4kis5+2mZ7JRaQzTqK4IrP5qjoVp1qKmJiYwp8NMpGQkMSIEQsYM2YJycmpXHBBJGPH9uCmm5pZKcIY41O+TBRxgGcHQtWAvzIuJCLNgWlAL1U96MN4CrSQEOHzzzeTkpLKoEExvPhiFxuz2hjjF75MFMuA+iJSG9gF3ATc4rmAiNQAZgIDVXWzD2MpkOLijlGyZDjly5egePEwpk93mnjats1bU44xxuSFzxqzVTUZGAzMAzYAH6vqOhGJFZFYd7FngArAJBFZJSLLfRVPQZKcnMrYsT/TuPFEHn/86/TX27atZknCGON3Pr2PQlW/BL7M8Npkj+d3A3f7MoaCZunSOO67bzarV+8F4OjR0yQnp1oPr8aYgCm8d2YXMEeOnGL48G+ZPHk5qlCzZhkmTOhN374NAh2aMaaIs0QRBA4fTqBJk0ns2XOCsLAQhg69lKef7kBkZLFAh2aMMZYogkG5ciXo1asemzcf5M03+3DRRdaBnzEmeFiiCIDTp5MZPXoRHTvWpGPHWgBMmNCbiIgw68DPGBN0LFH42Xffbef+++ewefNBGjeO5rff7ic0NISSJcMDHZoxxmTKEoWf7Nt3kqFDv+a999YA0KhRNJMm9SE01K5mMsYEN0sUPpbWgd+wYfM5cuQUERFhPPVUex5//HKKFbP+mYwxwc8ShY8dPXqKJ5/8jiNHTtGjR10mTuxN3brlAx2WMcZ4zRKFD5w8mUhYWAjFi4dRrlwJJk/uQ0qKcsMNTawDP2NMgWMV5Pns88830aTJJF5+eVH6a9dd14Qbb2xqScIYUyBZosgnO3Yc5eqrP6Rfvw/ZseMo8+ZtJTW1SPaIbowpZCxRnKekpBRefXUxjRtPZNasTURFFWPcuJ4sXHi73RNhjCkUrI3iPBw4EE+XLu+yZo3Tgd8NNzRh7NgeVK1aOsCRGWNM/rFEcR4qVChBdHRJatcuy4QJvendu36gQzJZSEpKIi4ujlOnTgU6FGN8KiIigmrVqhEenn838Rb8RDGzj992parMmPEbl1xSlQYNKiAivPfeNZQpE2F3Vge5uLg4oqKiqFWrll1UYAotVeXgwYPExcVRu3btfNtuwUsUe1fAa5l80Wv39uluN206wKBBX/Ldd9vp0qU233wzEBHhwgujfLpfkz9OnTplScIUeiJChQoV2L9/f75ut+AliszU7g3XzvHJpk+dSmbUqB956aVFJCamUKFCCf7+9+Y+2ZfxLUsSpijwxee8YCaKof657HT+/G3cf/8ctmw5BMCdd7bk5Ze7UaFCSb/s3xhjgoFdHpuFvXtP0Lfv+2zZcogmTSryww+38/bb/SxJmDwrVarUOa9t2rSJTp060bJlSxo3bsy9997LvHnzaNmyJS1btqRUqVI0bNiQli1bcuutt7JgwQJEhLfffjt9G7/++isiwquvvnrO9keMGEHVqlVp2bIlTZo04YMPPkifp6qMHDmS+vXr06BBAzp37sy6devS5584cYL77ruPunXr0rRpUzp06MDSpUvz+aicv+uvv55t27YFOowszZ07l4YNG1KvXj1eeumlLJdbsGABLVu2pGnTpnTs2DH99bFjx9K0aVOaNWvGzTffnH5BxmOPPcZ3333n8/gB58NSkB4XV0N9JSUlVVNTU9OnR4/+SUeN+lFPn0722T6Nf6xfvz7QIWhkZOQ5r3Xv3l0/++yz9Ok1a9acNb9jx466bNmy9Onvv/9eL7roIu3WrVv6a//4xz+0RYsW+sorr5yz/WeffTb99c2bN2tUVJQmJiaqqur48eO1V69eevLkSVVVnTdvntapU0cTEhJUVbV///76xBNPaEpKiqqqbt26VWfPnp2n956Z1NTU9G3n1dq1a/Xqq6/O1TrJyf77PicnJ2udOnV069atevr0aW3evLmuW7funOUOHz6sjRs31j///FNVVffu3auqqnFxcVqrVi2Nj49XVdUbbrhB//3vf6uq6h9//HHW58BTZp93YLnm8bxbMKuefGDVqj3Exs7mgQfaMHBgCwD+8Y/LAxyV8YnMLobID3moEt29ezfVqlVLn77oootyXKdGjRocO3aMvXv3UqlSJebOnUvv3jlfzFG/fn1KlizJ4cOHqVSpEqNHj2bBggWULOmUkrt3785ll13GjBkz6NSpE0uXLmXGjBmEhDgVD3Xq1KFOnTrnbHfu3LkMHz6clJQUoqOj+fbbbxkxYgSlSpXiscceA6BZs2bMnj0bgF69etG5c2d+/vlnrr76ak6ePMnLL78MwPTp01mxYgXjx4/nvffe44033iAxMZG2bdsyadIkQkPP7nF5xowZ9OvXL336/vvvZ9myZSQkJHD99dfz3HPPAVCrVi3uvPNOvv76awYPHkz58uV59tlnOX36NHXr1uXf//43pUqV4vnnn+eLL74gISGByy67jClTppxXnf8vv/xCvXr10o/bTTfdxKxZs2jSpMlZy73//vtce+211KhRA4BKlSqlz0tOTiYhIYHw8HDi4+OpUqUKADVr1uTgwYPs2bOHypUr5zlGbxT5qqfjx0/z6KPzuPjiqSxduosxY5bgJF9jfG/IkCFceeWV9OrVi7Fjx3LkyBGv1rv++uv55JNPWLx4Ma1bt6Z48eI5rrNy5Urq169PpUqVOHbsGCdPnqRu3bpnLRMTE8O6detYt24dLVu2POfEnNH+/fu55557+O9//8vq1av55JNPcoxj06ZN3Hrrrfz6668MGjSImTNnps/76KOP6N+/Pxs2bOCjjz5i0aJFrFq1itDQUGbMmHHOthYtWsTFF1+cPv3iiy+yfPly1qxZw8KFC1mzZk36vIiICH766Se6du3KyJEjmT9/PitXriQmJoYxY8YAMHjwYJYtW8batWtJSEhIT26eZsyYkV416Pm4/vrrz1l2165dVK9ePX26WrVq7Nq165zlNm/ezOHDh+nUqRMXX3wx7777LgBVq1blscceo0aNGlx44YWUKVOG7t27p6/XunVrFi1adM728luRLVGoKp99tpGHHppLXNwxQkKEhx9uy/PPd7arYwo7P10M4Y077riDHj16MHfuXGbNmsWUKVNYvXp1jif+G2+8kf79+7Nx40ZuvvlmFi9enOWyY8eO5a233mLbtm3MnTs32+2qaq4+/0uWLKFDhw7p1+yXL59zF/o1a9akXbt2AFSsWJE6deqwZMkS6tevz6ZNm7j88suZOHEiK1asoE2bNgAkJCSc9Ss7ze7du6lYsWL69Mcff8zUqVNJTk5m9+7drF+/nubNnasU+/fvnx7z+vXrufxyp8YgMTGRSy+9FIDvv/+el19+mfj4eA4dOkTTpk256qqrztrngAEDGDBggFfHJ7MfnZkd3+TkZFasWMG3335LQkICl156Ke3ataNixYrMmjWL7du3U7ZsWW644Qbee+89/v73vwNOyeOvv/7yKpbzUSQTxYED8dxxxyxmz94MQExMFaZM6Uvr1hcGODJTFFWpUoU777yTO++8k2bNmrF27dqzfiVnpnLlyoSHh/PNN98wbty4bBPFkCFDeOyxx5g5cya33norW7dupXTp0kRGRrJt27azqpNWrlxJx44dadq0KatXryY1NTW96ikzWSWWsLAwUlNT06c974iPjIw8a9n+/fvz8ccf06hRI6655hpEBFXltttuY9SoUdkehxIlSqRve/v27bz66qssW7aMcuXKcfvtt2e6X1WlW7duZzXsp8U4aNAgli9fTvXq1RkxYkSmd/LPmDGDV1555ZzX69Wrx6effnrWa9WqVWPnzp3p03FxcelVRxmXi46OJjIyksjISDp06MDq1asBqF27dnoyvPbaa1m8eHF6ojh16hQlSpTI9hjlhyJZ9RQVVYwtWw5RunRxJkzoxZIld1mSMAExd+5ckpKSANizZw8HDx6katWqXq37/PPPM3r06Byrh9Jce+21xMTE8M477wDw+OOP89BDD5GQkADA/Pnz+emnn7jllluoW7cuMTExPPvss+m/in///XdmzZp11jYvvfRSFi5cyPbt2wE4dMi5lLxWrVqsXLkScJJP2vys4vrss8/44IMP0n/1d+nShU8//ZR9+/alb/fPP/88Z93GjRuzZcsWAI4dO0ZkZCRlypRh7969fPXVV5nur127dixatCh9vfj4eDZv3pyeFKKjozlx4sQ5J/00AwYMYNWqVec8Mlu+TZs2/P7772zfvp3ExEQ+/PBD/va3v52zXL9+/fjxxx9JTk4mPj6epUuX0rhxY2rUqMGSJUuIj49HVfn2229p3Lhx+nqbN2+mWbNmWR7b/FJkShSLFu2gUaNoKlQoSfHiYXz44XVUqhRpd1Ybv4mPjz+r4frRRx8lLi6Ohx9+mIiICABeeeUVrxsmL7vsslzH8Mwzz3DLLbdwzz338OCDD3L48GEuuugiQkNDqVy5MrNmzUr/hTpt2jSGDh1KvXr1KFmyJBUqVDjnl3TFihWZOnUq1157LampqVSqVIlvvvmG6667jnfffZeWLVvSpk0bGjRokGVM5cqVo0mTJqxfv55LLrkEgCZNmjBy5Ei6d+9Oamoq4eHhTJw4kZo1a561bp8+fViwYAFdu3alRYsWtGrViqZNm1KnTp30qqWMKlasyPTp07n55ps5ffo0ACNHjqRBgwbcc889XHTRRdSqVSu92ut8hIWFMWHCBHr06EFKSgp33nknTZs2BWDy5MkAxMbG0rhxY3r27Enz5s0JCQnh7rvvTk8A119/Pa1btyYsLIxWrVpx7733Ak7/ZVu2bCEmJua848yJFLSG25jqost3eh/zwYPxPPHEfKZN+5W77mrFtGnnZnNT+G3YsOGsX2KmcEhISKBz584sWrTI65JVYfG///2PlStX8sILL5wzL7PPu4isUNU8ZZVCW/WkqrzzzioaNZrItGm/Eh4eQpUqUXZFkzGFSIkSJXjuuecyvZKosEtOTmbo0KF+2VehrHrauPEAsbGzWbjQqdPs1KkWb77Zh0aNogMcmTEmv/Xo0SPQIQTEDTfc4Ld9FbpEERd3jBYtJpOYmEJ0dElee607Awc2t0teTa4v/TSmIPJFrUmhSxTVqpVm4MDmhIQIL73UlfLlfX/pmAl+ERERHDx4kAoVKliyMIWWuuNRpF0ckV8KfGP27t3HGTJkHrGxMXTqVAuA1FS18arNWWyEO1NUZDXC3fk0ZhfYEkVKSipvvrmcJ5/8jmPHTrNlyyGWLbsHEbEkYc4RHh6eryN+GVOU+PSqJxHpKSKbRGSLiDyRyXwRkTfc+WtEpLU32125cjft2r3Ngw9+xbFjp7nqqgb89783WpWCMcb4gM9KFCISCkwEugFxwDIR+VxV13ss1guo7z7aAm+6f7O080hp2rR5i9RUpVq10owf34t+/RpakjDGGB/xZYniEmCLqm5T1UTgQ6BfhmX6Ae+63aUvAcqKSLZ9aRyKL4EIPPpoOzZseICrr25kScIYY3zIl20UVYGdHtNxnFtayGyZqsBuz4VE5F7gXnfyNDy7dswYcHsGLsqigQOBDiJI2LE4w47FGXYszmiY1xV9mSgy+5mf8RIrb5ZBVacCUwFEZHleW+4LGzsWZ9ixOMOOxRl2LM4QkeV5XdeXVU9xQHWP6WpAxo7TvVnGGGNMAPkyUSwD6otIbREpBtwEfJ5hmc+BW92rn9oBR1V1d8YNGWOMCRyfVT2parKIDAbmAaHAv1R1nYjEuvMnA18CvYEtQDxwhxebnuqjkAsiOxZn2LE4w47FGXYszsjzsShwd2YbY4zxr0Lbzbgxxpj8YYnCGGNMtoI2Ufiq+4+CyItjMcA9BmtEZLGItAhEnP6Q07HwWK6NiKSIyPX+jM+fvDkWItJJRFaJyDoRWejvGP3Fi+9IGRH5QkRWu8fCm/bQAkdE/iUi+0RkbRbz83beVNWge+A0fm8F6gDFgNVAkwzL9Aa+wrkXox2wNNBxB/BYXAaUc5/3KsrHwmO573Aulrg+0HEH8HNRFlgP1HCnKwU67gAei+HAaPd5ReAQUCzQsfvgWHQAWgNrs5ifp/NmsJYofNL9RwGV47FQ1cWqetidXIJzP0ph5M3nAuBB4L/APn8G52feHItbgJmqugNAVQvr8fDmWCgQJU5/P6VwEkWyf8P0PVX9Aee9ZSVP581gTRRZde2R22UKg9y+z7twfjEURjkeCxGpClwDTPZjXIHgzeeiAVBORBaIyAoRudVv0fmXN8diAtAY54be34CHVTXVP+EFlTydN4N1PIp86/6jEPD6fYpIZ5xEcYVPIwocb47F68AwVU0p5J1FenMswoCLgS5ACeBnEVmiqpt9HZyfeXMsegCrgCuBusA3IvKjqh7zdXBBJk/nzWBNFNb9xxlevU8RaQ5MA3qp6kE/xeZv3hyLGOBDN0lEA71FJFlVP/NPiH7j7XfkgKqeBE6KyA9AC6CwJQpvjsUdwEvqVNRvEZHtQCPgF/+EGDTydN4M1qon6/7jjByPhYjUAGYCAwvhr0VPOR4LVa2tqrVUtRbwKTCoECYJ8O47MgtoLyJhIlISp/fmDX6O0x+8ORY7cEpWiMgFOD2pbvNrlMEhT+fNoCxRqO+6/yhwvDwWzwAVgEnuL+lkLYQ9Znp5LIoEb46Fqm4QkbnAGiAVmKaqmV42WZB5+bl4AZguIr/hVL8MU9VC1/24iHwAdAKiRSQOeBYIh/M7b1oXHsYYY7IVrFVPxhhjgoQlCmOMMdmyRGGMMSZbliiMMcZkyxKFMcaYbFmiMEHJ7fl1lcejVjbLnsiH/U0Xke3uvlaKyKV52MY0EWniPh+eYd7i843R3U7acVnr9oZaNoflW4pI7/zYtym67PJYE5RE5ISqlsrvZbPZxnRgtqp+KiLdgVdVtfl5bO+8Y8ppuyLyDrBZVV/MZvnbgRhVHZzfsZiiw0oUpkAQkVIi8q37a/83ETmn11gRuVBEfvD4xd3efb27iPzsrvuJiOR0Av8BqOeu+6i7rbUi8oj7WqSIzHHHNlgrIv3d1xeISIyIvASUcOOY4c474f79yPMXvluSuU5EQkXkFRFZJs44Afd5cVh+xu3QTUQuEWcskl/dvw3du5SfB/q7sfR3Y/+Xu59fMzuOxpwj0P2n28MemT2AFJxO3FYB/8PpRaC0Oy8a587StBLxCffvUOBJ93koEOUu+wMQ6b4+DHgmk/1Nxx27ArgBWIrTod5vQCRO19TrgFbAdcBbHuuWcf8uwPn1nh6TxzJpMV4DvOM+L4bTk2cJ4F7gKff14sByoHYmcZ7weH+fAD3d6dJAmPu8K/Bf9/ntwASP9f8J/N19Xhan36fIQP+/7RHcj6DswsMYIEFVW6ZN+FBiRAAAAkBJREFUiEg48E8R6YDTHUVV4AJgj8c6y4B/uct+pqqrRKQj0ARY5HZvUgznl3hmXhGRp4D9OL3wdgH+p06neojITKA9MBd4VURG41RX/ZiL9/UV8IaIFAd6Aj+oaoJb3dVczozIVwaoD2zPsH4JEVkF1AJWAN94LP+OiNTH6Q00PIv9dwf+JiKPudMRQA0KZx9QJp9YojAFxQCckckuVtUkEfkD5ySXTlV/cBNJH+A/IvIKcBj4RlVv9mIfj6vqp2kTItI1s4VUdbOIXIzTZ84oEflaVZ/35k2o6ikRWYDT7XV/4IO03QEPquq8HDaRoKotRaQMMBt4AHgDpy+j71X1Grfhf0EW6wtwnapu8iZeY8DaKEzBUQbY5yaJzkDNjAuISE13mbeAt3GGhFwCXC4iaW0OJUWkgZf7/AG42l0nEqfa6EcRqQLEq+p7wKvufjJKcks2mfkQpzO29jgd2eH+vT9tHRFp4O4zU6p6FHgIeMxdpwywy519u8eix3Gq4NLMAx4Ut3glIq2y2ocxaSxRmIJiBhAjIstxShcbM1mmE7BKRH7FaUcYp6r7cU6cH4jIGpzE0cibHarqSpy2i19w2iymqeqvwEXAL24V0JPAyExWnwqsSWvMzuBrnLGN56szdCc4Y4msB1aKyFpgCjmU+N1YVuN0q/0yTulmEU77RZrvgSZpjdk4JY9wN7a17vT/t2sHJAAAAAzC+re2gQm2EnI4LPdYAJZFAcASCgCWUACwhAKAJRQALKEAYAkFACswrN86qMwDJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "\n",
    "def roc_auc_show(pred_y, y_label, figure_label_name='LSTM'):\n",
    "    fpr, tpr, _ = roc_curve(y_label, pred_y)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "\n",
    "    plt.figure()\n",
    "    lw = 2\n",
    "    plt.plot(fpr, tpr, color='darkorange',\n",
    "             lw=lw, label=figure_label_name + ' ROC curve (area = %0.2f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.title('Receiver operating characteristic example')\n",
    "    plt.legend(loc=\"lower right\")\n",
    "    plt.show()\n",
    "\n",
    "roc_auc_show(pred_mse, y_labels_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the ROC curve of the 1D-Conv model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3gUVffA8e8hCYQSOqjU0Ks0Q7HQBKQqdlRebO+rRAQVUfGHDRULFhCpIioWFBsKgoKiAoqCFAEpghTFICC9htTz+2OGsISUTchmS87nefbJzk47O9mds/femXtFVTHGGGMyU8jfARhjjAlsliiMMcZkyRKFMcaYLFmiMMYYkyVLFMYYY7JkicIYY0yWLFGECBFZJyId/B2Hv4nIJBF5LJ/3OVVERuTnPn1FRPqKyNe5XDdkP4MioiJS299x+IvYfRR5T0T+BM4BUoCjwFxgoKoe9WdcoUZEbgX+p6qX+DmOqUCcqj7q5ziGA7VV9T/5sK+pBMB7zi8iokAdVd3s71j8wUoUvnO5qpYAmgHNgf/zczw5JiLhBXHf/mTH3AQkVbVHHj+AP4HOHtMvAHM8ptsAPwEHgdVAB495ZYG3gH+AA8DnHvN6Aavc9X4CmqTfJ1AJiAfKesxrDuwFItzp24EN7vbnAdU9llXgbuAPYFsm7+8KYJ0bxwKgQbo4/g9Y727/LSAyB+9hKLAGSADCgYeBLcARd5tXucs2AE5wqtR20H19KjDCfd4BiAOGAP8CO4HbPPZXDvgCOAwsA0YAP2bxf73E4//2N3Crxz7HA3PcOJcCtTzWG+MufxhYAbT1mDcc+AR4z53/P6AV8LO7n53AOKCwxzqNgG+A/cBuYBjQDUgEktzjsdpdthTwhrudHe57DHPn3QosBka72xrhvvajO1/cef8Ch9z/S2PgTnc/ie6+vkj/uQfC3LhO/u9WAFUzOa4Zfh+Ai3A+t1Xd6abuMvXd6Qw/Gxm8t4PAVnd7t7r/i3+BWzyWnwpMco/rEWAhZ34varvPiwAvAdvd4z8JKOrv845Pz2n+DiAUH+m+MFWA34Ax7nRlYB/QA6dE18WdruDOnwN8CJQBIoD27ust3A93a/dLeIu7nyIZ7PM74A6PeF4EJrnPrwQ245xow4FHgZ88llX3y1I2ow8/UBc45sYdATzkbq+wRxxrgaruNhZz6sTtzXtY5a5b1H3tOpzkVwjo4+77PHferaQ7sXNmokgGnnJj7QEcB8q486e7j2JAQ5wTSIaJAqiGcwK50d1WOaCZxz7345zgw4FpwHSPdf/jLh+Ok7R24SZPnESR5P5fCgFFgQtwTp7hQDROUr/PXT4K56Q/BIh0p1t7bOu9dHF/DrwGFAcqAr8A/T2OXzIwyN1XUU5PFF1xTvClcZJGA49jn3acM/ncP4jzua/nrtsUKJfBcc3u+/AMzue5KE6iGuixbnafjWTgNpzP2gicE/t4nBP9Ze7/s4TH+zkCtHPnj/H8LHB6ongFmIXz+Y7C+bHxnL/POz49p/k7gFB8uF+Yo+4HT4FvgdLuvKHAu+mWn4dz0jwPSMU9kaVbZiLwdLrXNnIqkXh+Sf8HfOc+F5wTYDt3+ivgvx7bKIRz8qzuTitwaRbv7THgo3Tr7+DUr8A/gViP+T2ALTl4D7dnc2xXAb3d57eSfaKIB8I95v+LcxIOwzlB1/OYl2mJAqeU9Fkm86YCU9K959+zeA8HgKbu8+HAomze830n942TqH7NZLnheCQKnHayBDwSvrv+9x7Hb3u6baQdU+BSYJN7vApldpzTfe5PfgY3nvw/ZfPeMv0+uM8jcJLVbzhtfZKDz8YfHvPOx/lsn+Px2j5OT/aeyb0ETmn1ZGlGgdo436djnF5ivJBMSt+h8rA2Ct+5UlWjcE5W9YHy7uvVgetE5ODJB06Vxnk4v6T3q+qBDLZXHRiSbr2qOL+o0vsEuFBEKuH8QlLgB4/tjPHYxn6cD39lj/X/zuJ9VQL+Ojmhqqnu8pmt/5dHjN68h9P2LSI3i8gqj+Ubc+pYemOfqiZ7TB/HOQlUwPkV7bm/rN53VZxqjszsymAfAIjIEBHZICKH3PdQitPfQ/r3XFdEZovILhE5DDzrsXx2cXiqjnOi3elx/F7DKVlkuG9PqvodTrXXeGC3iEwWkZJe7tvbOLP6PqCqSTgn8cbAy+qemcGrz8Zuj+fx7vbSv1bCYzrtWKhz4cl+zvx+VcApga7w2O9c9/WQZYnCx1R1Ic4H/SX3pb9xfkGV9ngUV9Xn3XllRaR0Bpv6G3gm3XrFVPWDDPZ5EPgauB64CfjA4wv2N07Vg+d2iqrqT56byOIt/YPz5QZARATnpLDDY5mqHs+ruet4+x48TwTVgdeBgTjVFqVxqrXEizizswenaqJKJnGn9zdQK6c7EZG2OL+ar8cpKZbGqe8Xj8XSv4+JwO84V9mUxKnrP7l8VnGk387fOCWK8h7Hu6SqNspindM3qPqqql6A0y5SF6dKKdv1sokz/XKZfR8QkcrAEzhtXS+LSBH39ew+G7mR9v8XkRI4VUv/pFtmL06CaeQRbyl1LlwJWZYo8scrQBcRaYbTaHm5iHQVkTARiRSRDiJSRVV34lQNTRCRMiISISLt3G28DsSKSGtxFBeRniISlck+3wduBq5xn580Cfg/EWkEICKlROS6HLyXj4CeItJJRCJw6soTcBojT7pbRKqISFmck9yHuXwPxXFOSHvcWG/D+dV40m6giogUzkH8AKhqCjADGC4ixUSkPs7xysw0oLOIXC8i4SJSzv1/ZicKJyHtAcJF5HEgu1/lUTgN20fduO7ymDcbOFdE7hORIiISJSKt3Xm7gWgRKeS+x504PxheFpGSIlJIRGqJSHsv4kZEWrr/qwic6paTFw+c3FfNLFafAjwtInXc/3UTESmXwXKZfh/cHyFTcRrj/4vTNvO0u152n43c6CEil7ifp6eBpap6WonLLUG/DowWkYruviuLSNez3HdAs0SRD1R1D/AO8Jj7weuNcwLdg/OL6kFO/S/64dSd/45Tn36fu43lwB04VQEHcBqQb81it7OAOsBuVV3tEctnwEhgulutsRbonoP3shGncXYszq+ry3EuBU70WOx9nBPUVvcxIjfvQVXXAy/jXAG0G6eeebHHIt/hXH21S0T2evsePAzEqQbaBbwLfICT9DKKZTtO28MQnCqJVTgNtNmZh5P8N+FUw50g6yougAdwSoJHcE5KJxMtqnoEp8H3cjfuP4CO7uyP3b/7RGSl+/xmoDCnrkL7BLdaxwsl3f0fcGPfx6mS8RtAQ7f65fMM1h2F86Pia5yk9wZOg/Rpsvk+3IPTzvKYWyK+DbhNRNp68dnIjfdxSi/7cS4o6JvJckNxPrtL3O/QfJxG+5BlN9yZPCXOzYb/U9X5/o4lp0RkJHCuqt7i71hM/pICdgNhTlmJwhRYIlLfrRIREWmFU73xmb/jMibQ2J2YpiCLwqluqoRTzfcyMNOvERkTgKzqyRhjTJas6skYY0yWgq7qqXz58hodHe3vMIwxJqisWLFir6rm6sbAoEsU0dHRLF++3N9hGGNMUBGRv7JfKmNW9WSMMSZLliiMMcZkyRKFMcaYLFmiMMYYkyVLFMYYY7JkicIYY0yWfJYoRORNEflXRNZmMl9E5FUR2Swia0Skha9iMcYYk3u+LFFMxRnwPTPdcbrBroMzWPtEH8ZijDEFVmJiSvYLZcFniUJVF+H0656Z3sA76lgClBYRb/vJN8YYk50ZPRlzZRsuqD7orDbjzzuzK3P6AC5x7ms70y8oInfilDqoVq1avgRnjDF+NaMnbPvyrDfT9Lxo1u8+uyG9/ZkoMhrbNsOubFV1MjAZICYmxrq7NcYEnzw68Wfn74Mlmb2+Lndd5HR11KFLQzY/dB81az6V6236M1HEcfpg9lU4cyBzY4wJDblJEjV6wNVzvFo0OTmVV19dyuOjv+fYsSQa3/cJbdtWdzaT8z2fxp+JYhYwUESmA62BQ+5g8MYYE1pm9Dz1fEjeV4osXRpH//6zWb16NwDXXNOAmjXL5Nn2fZYoROQDoANQXkTicAYtjwBQ1UnAlziD1W8GjuMMnG6MMcHH22qlGj3ydLcHDsQzbNi3vPbaClQhOro048Z1p2fPunm6H58lClW9MZv5Ctztq/0bY0y+yEmS8LIayVtPPrmQSZNWEB5eiAceuJDHHmtPsWIReboPCMLxKIwxxu8ySg4+SAQZSU5OJTzcubPh0UfbsW3bQZ555lIaN67os31aojDGmMz4sbSQ3okTyYwc+SOff76RpUv/R+HCYZQvX4yZM2/w6X7BEoUxpiDKi0tV86kEAfDtt1u56645/PGHcw/zvHmbufzyevmyb7BEYYwpaHKaJPIxIaS3e/dRhgz5mmnTfgOgQYPyTJzYk/bto/M1DksUxpjQ58c2hdx67701DBr0FQcPniAyMpzHH2/HkCEXUbhwWL7HYonCGBO8cluFFOBJAiA1VTl48ATdutVm/PgeeXpfRE5ZojDGBK8gqULyxtGjifz889906VILgH79mlCpUhSdOtVAJKMej/KPJQpjTPDIrAThg7ud89Pnn//OoEFfsWfPMdauHUDt2mURETp3runv0ABLFMaYYJBVFVMe3+2cn/766yD33DOXWbM2AhATU4mEhGQ/R3UmSxTGmMAQQPcs+FpSUgqvvLKE4cMXcvx4ElFRhXn22U7cdVcMYWGBN0K1JQpjTP4423sXQiBBnHTPPV8xadIKAK6/vhGjR3elUqUoP0eVOUsUxpj8UUBKC9647742LFz4F6NGdaVbt9r+DidbliiMMb7n4262A5mq8t57a/jyy828//7ViAj16pVn7doBFCrk36uZvGWJwhjjO+mrm4K44Tk3Nm7cy113zeH77/8EnEtee/SoAxA0SQIsURhjfCWjJFEAqpUA4uOTeO65Hxk5cjGJiSmUK1eUl1++jO7dA7+aKSOWKIwxeScIu8rIa/PnbyU2djZbthwA4L//bc7IkZ0pV66YnyPLPUsUxpi8U8CTBMBPP/3Nli0HaNSoApMm9eKSS6r5O6SzZonCGJO53F7SWoAarFNSUtm8eT/16pUHYOjQiylfvhj/+18Lv3Tg5wuBd2eHMSZw5LbDvQLi1193ctFFb3LJJW+xf388AEWKhDNgQMuQSRJgJQpjTGYK8CWt2TlyJIHHH/+eV1/9hdRUpXLlKLZs2U/ZspX9HZpPWKIwxpySWWO0AZx7ImbM2MC9985lx44jFCokDB7chief7EBUVBF/h+czliiMKeiy63CvgDVGZ+W+++by6qu/ANCyZSVee60XzZuf5+eofM8ShTEFkSWHXLnqqga8/fZqnn22E/37XxCQHfj5giUKYwqSzBKEJYcM/fjjdr7/fhuPPdYegA4dotm+fTAlS4ZuNVNGLFEYU1AU4Dulc2rfvuMMHTqfN974FYBOnWpy0UVVAQpckgBLFMaEJqtayhVV5Z13VvPAA9+wd+9xIiIK8fDDl9C8+bn+Ds2vLFEYE2osSeTKhg17uOuuOSxc+BcAHTtGM2FCT+rXL+/fwAKAJQpjQolnkrCkkCOjRv3MwoV/UaFCMUaN6krfvucjEjw9vPqSJQpjQoG1P+TKoUMnKFUqEoDnnutM8eKFefzx9pQtW9TPkQWWgnFtlzGhzJJEjv3zzxH69PmENm3eIDExBYDy5YvxyivdLElkwEoUxgQzq2rKkZSUVCZMWMYjj3zHkSOJFCsWwcqVO2nTpoq/QwtoliiMCVaWJHJkxYp/6N9/NitW7ATgiivqMXZsd6pVK+XnyAKfTxOFiHQDxgBhwBRVfT7d/FLAe0A1N5aXVPUtX8ZkTFCzgYFyZfjwBTz99CJSU5WqVUsydmx3eveu7++wgobPEoWIhAHjgS5AHLBMRGap6nqPxe4G1qvq5SJSAdgoItNUNdFXcRkTNLwZC8KShFdq1iyDCAwZciHDh3egRInC/g4pqPiyRNEK2KyqWwFEZDrQG/BMFApEiXMNWglgP5Dsw5iMCTw5HRzIkkO2tm49wLJlO+jTpzEA/fo1oXXrymmDC5mc8WWiqAz87TEdB7ROt8w4YBbwDxAF9FHV1PQbEpE7gTsBqlUL/mEFTQGVm9HiLCnkSGJiCi+99BNPP70IVeWCCypRu3ZZRMSSxFnwZaLI6E6V9KOfdAVWAZcCtYBvROQHVT182kqqk4HJADExMTaCiglMuR02FCwh5IFFi/4iNnY2GzbsBaBv3/MLZL9MvuDLRBEHVPWYroJTcvB0G/C8qiqwWUS2AfWBX3wYlzF5KycJwhJCntu79zgPPvgNU6euAqBOnbJMnNiTTp1q+jmy0OHLRLEMqCMiNYAdwA3ATemW2Q50An4QkXOAesBWH8ZkTO54mwwsEeS72NjZfPrpBooUCWPYsLY89NDFREbalf95yWdHU1WTRWQgMA/n8tg3VXWdiMS68ycBTwNTReQ3nKqqoaq611cxGeM1a2AOaKmpSqFCTu32M89cSnx8Mq+80pU6dcr5ObLQJE6tT/CIiYnR5cuX+zsME4qySw6WDPzu+PEknn56IatW7ebLL2+yTvtyQERWqGpMbta18pkxYDeyBYE5czYxcOBX/PnnQUTgl1920Lq1db2RHyxRmILLkkNQiIs7zL33zmXGjA0ANG16DpMm9bIkkY8sUZiCxQb1CSoTJixj6ND5HD2aSPHiETz9dEcGDWpNeLh1fJ2fLFGY0GbtDkFt797jHD2ayFVX1WfMmG5UrWod+PmDJQoT2jJKEpYcAtbBgyf4/fe9ad1+Dx16Ma1aVaZbt9p+jqxgs0RhQteMnqeeDwmuq/sKGlXlww/XMXjwPFJSUvn994GULVuUIkXCLUkEAKvoM6Ep/VgNJmBt3ryfbt2mceONn7Jr11Hq1CnHoUMn/B2W8WAlChM67CqmoJKQkMwLLyzmmWd+ICEhhTJlInnhhS7cfnvztJvpTGDwOlGISHFVPebLYIzJNUsSQadPn0+YOXMjADff3JQXX+xCxYrF/RyVyUi2iUJELgKm4IwXUU1EmgL9VXWAr4Mzxis2JGhQuu++NmzcuI8JE3rQsWMNf4djsuBNiWI0TnfgswBUdbWItPNpVMZkx0oQQSU1VXnzzV/ZsGEPL7/cFYAOHaJZu/YuwsKsqTTQeVX1pKp/p+tTJcU34RjjJUsSQeO333YTGzuHn35yxjG7+eamNG16LoAliSDhTaL4261+UhEpDNwDbPBtWMZ4yS57DVjHjiXy5JMLGTXqZ1JSlHPPLcErr3SlSZNz/B2aySFvEkUsMAZnaNM44GvA2ieM/3jeH2EC0hdfbGTgwK/Yvv0QInD33S155plLKVUq0t+hmVzwJlHUU9W+ni+IyMXAYt+EZAq8nAwSZALS55//zvbth2je/Fxee60XLVtW9ndI5ix4kyjGAi28eM2Y3LOBgoJacnIqO3Ycpnr10gCMHNmF5s3PIzY2xjrwCwGZJgoRuRC4CKggIvd7zCqJM2KdMXnHGqeD1pIlccTGziYhIYXVq2MpXDiM8uWLMXBgK3+HZvJIViWKwjj3ToQDUR6vHwau9WVQpoCxPpmC0oED8Qwb9i2vvbYCVYiOLs2ffx6kbl0bjjTUZJooVHUhsFBEpqrqX/kYkylIrE+moKOqfPDBWgYPnse//x4jPLwQDz54EY8+2o5ixSL8HZ7xAW/aKI6LyItAIyDtkgVVvdRnUZmCw+6oDjp9+87ggw/WAtC2bTUmTuxJo0YV/RyV8SVvWpmmAb8DNYAngT+BZT6MyYS6GT3hZXEeJ1mSCBrdutWmXLmivPnmFSxYcKsliQLAmxJFOVV9Q0Tu9aiOWujrwEwIy6jh2gSs+fO3smXLfvr3jwGgX78m9OpVl7Jli/o5MpNfvEkUSe7fnSLSE/gHsFHNzdmzhuuAtnv3Ue6//2vef/83ihQJo3PnmtSqVRYRsSRRwHiTKEaISClgCM79EyWB+3walQlddld1wEtNVSZPXsHDD8/n0KEEIiPDefzxdjZedQGWbaJQ1dnu00NAR0i7M9sY76W/oc6qmwLS6tW76N9/NkuX7gCge/fajBvXg5o1y/g5MuNPWd1wFwZcj9PH01xVXSsivYBhQFGgef6EaEJC+iRhjdcB6aGH5rN06Q4qVYpizJhuXHNNA9L1HG0KoKxKFG8AVYFfgFdF5C/gQuBhVf08P4IzIcJuqAtYqsrx40kUL14YgFdf7cakSct58smOlCxZxM/RmUCRVaKIAZqoaqqIRAJ7gdqquit/QjMhw26oC0h//XWQQYO+4tixJObP74eIUK9eeUaP7ubv0EyAySpRJKpqKoCqnhCRTZYkzFmx6qaAkJSUwujRS3jyyYUcP55EVFRh/vhjv3W9YTKVVaKoLyJr3OcC1HKnBVBVbeLz6IwxeWrx4u3Exs5h7dp/AejTpxGjRnWlUqWobNY0BVlWiaJBvkVhQkdOuws3+WbQoC8ZN87pVKFmzTKMH9+Dbt1q+zkqEwyy6hTQOgI02bNBhoJGhQrFiYgoxNChFzNsWFuKFrUO/Ix3vLnhLtdEpBvOMKphwBRVfT6DZToArwARwF5Vbe/LmEweySpB2OWvAeH33/eyffshLrusFgBDh17M9dc3on798n6OzAQbnyUK9z6M8UAXnLG2l4nILFVd77FMaWAC0E1Vt4uI9S4W6DJKEJYYAkp8fBLPPvsDI0cupnTpSH7/fSBlyxalSJFwSxImV7xKFCJSFKimqhtzsO1WwGZV3epuYzrQG1jvscxNwAxV3Q6gqv/mYPsmv2V0d7UliIDy9ddbGDBgDlu2HADgiivqYffLmbOVbaIQkcuBl3BGvKshIs2Ap1T1imxWrQz87TEdB7ROt0xdIEJEFuCMojdGVd/xMnaT32zsiIC1c+cRBg+ex4cfrgOgUaMKTJrUi0suqebnyEwo8KZEMRyndLAAQFVXiUi0F+tl9Dsm/W254cAFQCecbkF+FpElqrrptA2J3AncCVCtmn3w8136koQliYBz9dUfsWRJHEWLhjN8eAcGD25DRIQNbW/yhjeJIllVD+Wiv5c4nC5ATqqC00V5+mX2quox4JiILAKaAqclClWdDEwGiImJsT4g8kNmjdV29VLAUNW0fpief74TL730M2PHdic6urSfIzOhxptEsVZEbgLCRKQOcA/wkxfrLQPqiEgNYAdwA06bhKeZwDgRCcep2moNjPY2eJOHsrvM1aqbAsaRIwk8/vj3HDuWxOTJlwPQvn007dtH+zcwE7K8SRSDgEeABOB9YB4wIruVVDVZRAa6y4cBb6rqOhGJdedPUtUNIjIXWAOk4lxCuzZ3b8XkmCWHoKKqzJixgXvvncuOHUcIDy/EsGFtrQRhfE5Us67JEZHmqvprPsWTrZiYGF2+fLm/wwgNL6erTrTEELC2bTvAwIFf8eWXfwDQqlVlJk3qSfPm5/k5MhMsRGSFqsbkZl1vShSjROQ84GNguqquy82OTICxrr+DgqrywguLefLJhcTHJ1OqVBGee64Td955AWFhhfwdnikgvBnhrqOInIsziNFkESkJfKiq2VY/mQDlWeVkjdMBTUTYtGkf8fHJ3HhjY0aN6sq555bwd1imgMm26um0hUXOBx4C+qhqYZ9FlQWresqFrK5gsqqmgLN373F27TpK48YV06Z//XUnXbrU8nNkJpidTdVTtmVXEWkgIsNFZC0wDueKpyq52ZnxA0sSQUNVmTp1FfXrj+O66z4mMTEFgPLli1mSMH7lTRvFW8AHwGWqmv4+CBPo7G7qoLBhwx5iY+ewaJHTaXPTpudy4EA855xj1UzG/7xpo2iTH4GYPJRRKcKSREA6fjyJZ55ZxIsv/kRSUioVKhRj1Kiu9O17Prm4ydUYn8g0UYjIR6p6vYj8xuldb9gId4Euo95dTcBRVS699G2WLt0BQP/+F/Dcc50oU6aonyMz5nRZlSjudf/2yo9AjA/YZa8BTUQYMKAlx48n8dprvbjwwqrZr2SMH2Q1wt1O9+kAVR3qOU9ERgJDz1zL+I0NQRrwUlJSmTBhGUlJqdx//4UA9OvXhBtvbGwd+JmA5k1jdhfOTArdM3jN+IN13hcUli//h9jY2axYsZMiRcK44YbGVKoUhYhYkjABL6s2iruAAUBNEVnjMSsKWOzrwIwXbCChgHfo0AkeffQ7xo9fhipUrVqSsWO7U6lSlL9DM8ZrWZUo3ge+Ap4DHvZ4/Yiq7vdpVCZrliACnqry8cfrue++uezceZSwMGHw4DY88UQHSpTwy72qxuRaVolCVfVPEbk7/QwRKWvJwg9svOqg8tprK9i58yht2lRh0qSeNG16rr9DMiZXsitR9AJW4Fwe63lRtwI1fRiX8WQJIigkJCRz8OAJzjmnBCLChAk9WLDgT+644wIKFbJ7Ikzwyuqqp17u3xr5F445g1UzBYWFC/8kNnYOlSpFMX9+P0SEevXKU69eeX+HZsxZy/aqJxG5GFilqsdE5D9AC+AVVd3u8+gKMksQQWHPnmM8+OA3vP32asC5BHb37mPWw6sJKd50aD8ROC4iTXF6jv0LeNenURlLEgEuNVV5442V1K8/nrffXk2RImE8+WQH1qy5y5KECTne3EeRrKoqIr2BMar6hojc4uvAjMvurg44qkrXru8xf/5WADp3rsmECT2oU6ecnyMzxje8SRRHROT/gH5AWxEJAyJ8G5YxgUtEaNu2Gr/9tpvRo7tyww2NrQM/E9K8qXrqAyQAt6vqLqAy8KJPozImwMyZs4nPP/89bXro0Iv5/feB3Hij9fJqQp833YzvEpFpQEsR6QX8oqrv+D60Asr6bAoocXGHuffeucyYsYHy5YvRrl11ypYtSpEi4RQp4k2B3Jjg580Id9cDvwDX4YybvVRErvV1YAVSRlc6Gb9ITk5l9OifadBgPDNmbKB48QiGDbuEkiWL+Ds0Y/KdNz+JHgFaquq/ACJSAZgPfOLLwEJOTkoKdpWTX/3yyw7695/NqlW7ALjqqvqMGdONqlVL+TkyY/zDm0RR6GSScO3Du7YN48mSRFBITVVuu20m69fvoVq1Uowb153LL6/n76RIw1MAACAASURBVLCM8StvEsVcEZmHM242OI3bVonurfQlCbvcNeCoKgkJKURGhlOokDB+fA+++uoPHn+8PcWLWwd+xnjTmP2giFwNXILT39NkVf3M55GFCmtzCGibN+9nwIA5VK1akjfe6A1Ahw7RdOgQ7d/AjAkgWY1HUQd4CagF/AY8oKo78iuwoGcliYCWkJDMyJGLefbZH0hISKFs2aK88MJxypUr5u/QjAk4WZUo3gTeARYBlwNjgavzI6iglllPryZgfPfdNu66aw6bNu0D4JZbmvLii10sSRiTiawSRZSqvu4+3ygiK/MjoKBlXYEHvJSUVG67bSbvvusM2FivXjkmTepl1UzGZCOrRBEpIs05NQ5FUc9pVbXEcZL19BoUwsIKER5eiMjIcB59tC0PPHCR3TRnjBdENeO6cxH5Pov1VFUv9U1IWYuJidHly5f7Y9dnsgQR8H77bTcnTiTTsmVlAPbtO87BgyeoVausnyMzJn+JyApVjcnNulkNXNQx9yEVEJYkAtaxY4kMH76A0aOXUKdOOVavjqVw4TDKlStmbRHG5JCVu3NrRs9Tz+2KpoAya9ZGBg36iu3bDyECnTvXICkphcKFw/wdmjFByad3WItINxHZKCKbReThLJZrKSIpQdWH1MnShF3RFDC2bz/ElVdOp3fv6WzffogWLc7jl1/uYOzYHnbjnDFnwWclCnfcivFAFyAOWCYis1R1fQbLjQTm+SqWPOdZmrDqpoCQkpJKhw5T2bbtIFFRhRkx4lIGDGhJeLj1NmPM2fJmzGwB+gI1VfUpEakGnKuqv2Szaitgs6pudbczHegNrE+33CDgU6BlToPPV3Z/REBSVUSEsLBCDB/egS++2MQrr3SlcuWS/g7NmJDhzc+tCcCFwI3u9BGckkJ2KgN/e0zHua+lEZHKwFXApKw2JCJ3ishyEVm+Z88eL3btA3aPREA5cCCe2NjZPPvsD2mv9evXhI8/vs6ShDF5zJuqp9aq2kJEfgVQ1QMi4k2Fb0bDfqVv9X0FGKqqKVmNEqaqk4HJ4Fwe68W+fccarv1KVXn//d+4//6v+fffY0RFFWbgwFaUKhVpI80Z4yPeJIoktx1BIW08ilQv1osDqnpMVwH+SbdMDDDd/YKXB3qISLKqfu7F9k0Bs2nTPgYMmMO3324DoG3bakyc2JNSpSL9HJkxoc2bRPEq8BlQUUSeAa4FHvVivWVAHRGpAewAbgBu8lxAVWucfC4iU4HZAZkkPBuvTb5LTk5lxIhFPPfcjyQmplCuXFFefLELt97azEoRxuQDb7oZnyYiK4BOONVJV6rqBi/WSxaRgThXM4UBb6rqOhGJdedn2S4RMDwbsa3x2i/CwoQffthOYmIKt9/ejJEju1C+vN00Z0x+ybQLj7QFnKuczqCq230SUTbytQuP9EnCGq/zze7dRzlxIpnq1UsD8Mcf+9i58yjt2lX3c2TGBCefdOHhYQ5O+4QAkUANYCPQKDc7DCqWJPJdaqoyefIKHn54PjExlfjmm36ICHXqlKNOnXL+Ds+YAsmbqqfzPadFpAXQ32cRBSJLEvli1apdxMbOZulSZ3yswoXDOHo0kaioIn6OzJiCLcd3ZqvqShEJ7JvjTFA5ciSBJ55YwJgxS0lNVSpVimLMmG5cc00Da6w2JgB4c2f2/R6ThYAWgJ/uestHdqVTvkhMTKFFi8ls3ryfQoWEe+9tzVNPdaRkSStFGBMovClRRHk8T8Zps/jUN+EEELvSKV8ULhxGv35N+OKLTUya1JMLLqjk75CMMelkmSjcG+1KqOqD+RRPYLBO/3wmKSmF0aOXUK1aKW64oTEADz98CY880pawMOvAz5hAlGmiEJFw916IFvkZkN/ZfRM+s3jxdmJj57B27b9UqFCMXr3qUqJEYRsnwpgAl1WJ4hec9ohVIjIL+Bg4dnKmqs7wcWz5y4Y19Zn9++MZOvQbpkz5FYCaNcswYUIPSpSwMSKMCQbetFGUBfYBl3LqfgoFgj9RZNR1OFiSyCOqyrvvrmHIkK/Zu/c4ERGFGDr0YoYNa0vRohH+Ds8Y46WsEkVF94qntZxKECcFZxeqmSWGkyxB5KmkpFSee+5H9u49Tvv21Zk4sScNGlTwd1jGmBzKKlGEASXwrrvwwGelh3wRH59EYmIKpUpFUrhwGJMn92Lr1gPcfHNTuyfCmCCVVaLYqapP5VskvmR9NuWLefM2M2DAl3ToUJ033ugNQNu21Wnb1vpnMiaYZZUoQufnnyUJn9q58wiDB8/jww/XAVC8eATHjydRrJi1QxgTCrK6cL1TvkWRXyxJ5KmUlFTGjfuF+vXH8+GH6yhaNJyRIzuzYsWdliSMCSGZlihUdX9+BuIz1hWHT5w4kUy7dm+xbJkzaGGvXnUZO7Y70dGl/RyZMSav5bhTwKBiN8/5TGRkOI0bV2TnzqO8+mo3rryyvjVWGxOiQjtRWNtEnlFVZszYwDnnlOCSS5yxrEaN6kpYmFg34MaEuNBNFNZfU57Ztu0AAwd+xZdf/kH9+uVZtao/RYqEU7p0pL9DM8bkg9BNFFbldNYSE1N4+eWfePrpRcTHJ1OqVBHuvbc14eHWeZ8xBUloJgorTZy1H374i9jYOaxf7ww9ctNN5/Pyy5dx7rkl/ByZMSa/hWaisNLEWYmPT+Laaz/m33+PUbt2WSZM6EGXLrX8HZYxxk9CL1FYaSJXVJWUFCU8vBBFi0YwatRlbNq0j//7v7ZERobex8QY473QOwNYaSLH1q/fQ2zsbLp0qcljj7UHoG/fJn6OyhgTKEKnVXJGT3jZ4zp+K01k6/jxJIYN+5amTSfxww/bmTLlVxISkv0dljEmwIRGiSKjQYdMlr766g/uvvtLtm07CED//hfw3HOdKFIkND4Sxpi8ExpnBbuxzmvHjiVy660z+eST9QA0aXIOkyb15MILq/o5MmNMoAqNRHGSJYlsFSsWwf798RQvHsGTT3bg3nvb2H0RxpgshVaiMBlavvwfSpeOpHbtsogIU6ZcTlhYIapVK+Xv0IwxQcB+SoawQ4dOMGjQl7Rq9TqxsbNRdQYmrFGjjCUJY4zXrEQRglSVjz5ax333zWPXrqOEhQktWpxHcnIqERFh/g7PGBNkLFGEmC1b9nP33V8yb94WAC68sAqTJvWiSZNz/ByZMSZYWaIIIUeOJBAT8zoHD56gdOlIRo7szP/+14JChWycCGNM7vk0UYhIN2AMEAZMUdXn083vCwx1J48Cd6nq6hztxEawSxMVVYTBg9uwefN+XnrpMipWLO7vkIwxIcBniUJEwoDxQBcgDlgmIrNUdb3HYtuA9qp6QES6A5OB1l7vpICPYLdnzzEefPAbOnWqQb9+TQF47LF2NtKcMSZP+fKqp1bAZlXdqqqJwHSgt+cCqvqTqh5wJ5cAVXK0hwJ6o11qqjJlykrq1RvH22+v5pFHviMpKQXAkoQxJs/5suqpMvC3x3QcWZcW/gt8ldEMEbkTuBOgWrVqZy5QgJLE2rX/Ehs7m8WLnUPbuXNNJkzoYVczGWN8xpeJIqOftprhgiIdcRLFJRnNV9XJONVSxMTE6Bl9OxUA8fFJDB++gFGjlpCcnMo55xRn9Oiu3HBDYytFGGN8ypeJIg7w7ECoCvBP+oVEpAkwBeiuqvuy3erBP2DbilPTBaRtolAhYdasTaSkpDJgQAzPPNPJxqw2xuQLXyaKZUAdEakB7ABuAG7yXEBEqgEzgH6qusmrrSYcdv4WgHaJuLjDFCsWQdmyRSlSJJypU50mntatc9aUY4wxZ8NnjdmqmgwMBOYBG4CPVHWdiMSKSKy72ONAOWCCiKwSkeVe7yCEk0RyciqjR/9MgwbjefDBr9Neb926iiUJY0y+8+l9FKr6JfBlutcmeTz/H/C/HG84hKubli6No3//2axevRuAQ4cSSE5OtR5ejTF+E5x3ZodgaeLgwRMMG/YtkyYtRxWqVy/FuHE96NWrrr9DM8YUcMGZKELMgQPxNGw4gV27jhIeXoghQy7kscfaUbx4YX+HZowxligCQZkyRenevTabNu1j4sSenH++deBnjAkclij8ICEhmZEjF9O+fXXat48GYNy4HkRGhlsHfsaYgGOJIp9999027rprDps27aNBg/L89ttdhIUVolixCH+HZowxGbJEkU/+/fcYQ4Z8zXvvrQGgfv3yTJjQk7Awu5rJGBPYLFH42MkO/IYOnc/BgyeIjAzn0Ufb8uCDF1O4sPXPZIwJfJYofOzQoRM88sh3HDx4gq5dazF+fA9q1Srr77CMMcZrlih84NixRMLDC1GkSDhlyhRl0qSepKQo113X0DrwM8YEHasgz2OzZm2kYcMJvPDC4rTXrrmmIddf38iShDEmKFmiyCPbtx/iyiun07v3dLZvP8S8eVtITc2wV3VjjAkqlijOUlJSCi+99BMNGoxn5syNREUVZsyYbixceKvdE2GMCQnWRnEW9u49TqdO77BmjdOB33XXNWT06K5UrlzSz5EZY0zesURxFsqVK0r58sWoUaM048b1oEePOv4OKSglJSURFxfHiRMn/B2KMUEvMjKSKlWqEBGRdzfxWqLIAVVl2rTfaNWqMnXrlkNEeO+9qyhVKtLurD4LcXFxREVFER0dbQ3+xpwFVWXfvn3ExcVRo0aNPNuutVF4aePGvXTu/C79+n3GgAFzUHUaqs87L8qSxFk6ceIE5cqVsyRhzFkSEcqVK5fnpXMrUWTjxIlknnvuB55/fjGJiSmUK1eU//ynib/DCjmWJIzJG774LlmiyML8+Vu56645bN68H4Dbb2/GCy90oVy5Yn6OzBhj8o9VPWVi9+6j9Or1Pps376dhwwosWnQrb7zR25JEiNq1axc33HADtWrVomHDhvTo0YNNmzbx559/IiKMHTs2bdmBAwcydepUAG699VYqV65MQkICAHv37iU6OjrDfZQoUeKM1zZu3EiHDh1o1qwZDRo04M4772TevHk0a9aMZs2aUaJECerVq0ezZs24+eabWbBgASLCG2+8kbaNX3/9FRHhpZdeOmP7w4cPp3LlyjRr1oyGDRvywQcfpM1TVUaMGEGdOnWoW7cuHTt2ZN26dWnzjx49Sv/+/alVqxaNGjWiXbt2LF26NEfHNT9ce+21bN261d9hZGru3LnUq1eP2rVr8/zzz2e4zIsvvpj2P2/cuDFhYWHs37+fEydO0KpVK5o2bUqjRo144okn0tZ54IEH+O677/LnTahqUD0uqIL6SkpKqqampqZNjxz5oz733A+akJDss30a1fXr1/t1/6mpqdqmTRudOHFi2mu//vqrLlq0SLdt26YVK1bUWrVqaUJCgqqq3n333frWW2+pquott9yiVatW1QkTJqiq6p49e7R69eoZ7qd48eJnvHbZZZfp559/nja9Zs2a0+a3b99ely1bljb9/fff6/nnn69dunRJe+2hhx7Spk2b6osvvnjG9p944om01zdt2qRRUVGamJioqqpjx47V7t2767Fjx1RVdd68eVqzZk2Nj49XVdU+ffroww8/rCkpKaqqumXLFp09e3aG7y03UlNT07adW2vXrtUrr7wyR+skJ+ff9zk5OVlr1qypW7Zs0YSEBG3SpImuW7cuy3VmzZqlHTt2VFXnGB05ckRVVRMTE7VVq1b6888/q6rqn3/+edrnwFNG3ylguebyvGtVT65Vq3YRGzubu+9uSb9+TQF46KGL/RxVAfSyj9oqhmR+l/z3339PREQEsbGxaa81a9YMgD///JMKFSpw8cUX8/bbb3PHHXecsf59993H6NGjM5yXnZ07d1KlSpW06fPPPz/bdapVq8bhw4fZvXs3FStWZO7cufTo0SPb9erUqUOxYsU4cOAAFStWZOTIkSxYsIBixZxS8mWXXcZFF13EtGnT6NChA0uXLmXatGkUKuRUPNSsWZOaNWuesd25c+cybNgwUlJSKF++PN9++y3Dhw+nRIkSPPDAAwA0btyY2bNnA9C9e3c6duzIzz//zJVXXsmxY8d44YUXAJg6dSorVqxg7NixvPfee7z66qskJibSunVrJkyYQFjY6T0uT5s2jd69e6dN33XXXSxbtoz4+HiuvfZannzySQCio6O5/fbb+frrrxk4cCBly5bliSeeICEhgVq1avHWW29RokQJnnrqKb744gvi4+O56KKLeO21186qzv+XX36hdu3aacfthhtuYObMmTRs2DDTdT744ANuvPFGwGlvOFkSTUpKIikpKS2e6tWrs2/fPnbt2sW5556b6xi9UeCrno4cSeD+++dxwQWTWbp0B6NGLUm7oskUDGvXruWCCy7IcpmHH36Yl19+mZSUlDPmVatWjUsuuYR33303x/sePHgwl156Kd27d2f06NEcPHjQq/WuvfZaPv74Y3766SdatGhBkSJFsl1n5cqV1KlTh4oVK3L48GGOHTtGrVq1TlsmJiaGdevWsW7dOpo1a3bGiTm9PXv2cMcdd/Dpp5+yevVqPv7442zj2LhxIzfffDO//vorAwYMYMaMGWnzPvzwQ/r06cOGDRv48MMPWbx4MatWrSIsLIxp06adsa3Fixef9r975plnWL58OWvWrGHhwoWsWbMmbV5kZCQ//vgjnTt3ZsSIEcyfP5+VK1cSExPDqFGjAKdacdmyZaxdu5b4+Pi05OZp2rRpadVEno9rr732jGV37NhB1apV06arVKnCjh07Mj02x48fZ+7cuVxzzTVpr6WkpNCsWTMqVqxIly5daN26ddq8Fi1asHjx4ow2lacKbIlCVfn889+55565xMUdplAh4d57W/PUUx3tChx/yuKXvz/VqFGDVq1a8f7772c4f9iwYVxxxRX07NkzR9u97bbb6Nq1K3PnzmXmzJm89tprrF69OtsT//XXX0+fPn34/fffufHGG/npp58yXXb06NG8/vrrbN26lblz52a5XVXN0ed/yZIltGvXLu2a/bJls+9Cv3r16rRp0waAChUqULNmTZYsWUKdOnXYuHEjF198MePHj2fFihW0bNkSgPj4eCpWrHjGtnbu3EmFChXSpj/66CMmT55McnIyO3fuZP369TRp4lyl2KdPn7SY169fz8UXOzUGiYmJXHjhhYBTunzhhRc4fvw4+/fvp1GjRlx++eWn7bNv37707dvXq+OT0Y/OrI7vF198wcUXX3zacQwLC2PVqlUcPHiQq666irVr19K4cWMAKlasyD///ONVLGejQCaKvXuPc9ttM5k9exMAMTGVeO21XrRocZ6fIzP+0KhRIz755JNslxs2bBjXXnst7dq1O2Ne7dq1adasGR999FGO91+pUiVuv/12br/9dho3buxVCefcc88lIiKCb775hjFjxmSZKAYPHswDDzzAjBkzuPnmm9myZQslS5akePHibN269bTqpJUrV9K+fXsaNWrE6tWrSU1NTat6ykhmiSU8PJzU1NS0ac/r+osXL37asn369OGjjz6ifv36XHXVVYgIqsott9zCc889l+VxKFq0aNq2t23bxksvvcSyZcsoU6YMt956a4b7VVW6dOlyWsP+yRgHDBjA8uXLqVq1KsOHD8/wfoRp06bx4osvnvF67dq1z/gcValShb///jttOi4ujkqVKmX6fqZPn55W7ZRe6dKl6dChA3Pnzk1LFCdOnKBo0aKZbi+vFMiqp6iowmzevJ+SJYswblx3liz5ryWJAuzSSy8lISGB119/Pe21ZcuWsXDhwtOWq1+/Pg0bNsywOgLgkUceyfDKo6zMnTuXpKQkwLnyat++fVSuXNmrdZ966ilGjhyZbfXQSVdffTUxMTG8/fbbADz44IPcc889xMfHAzB//nx+/PFHbrrpJmrVqkVMTAxPPPFE2q/iP/74g5kzZ562zQsvvJCFCxeybds2APbvdy4lj46OZuXKlYCTfE7Ozyyuzz//nA8++CDtV3+nTp345JNP+Pfff9O2+9dff52xboMGDdi8eTMAhw8fpnjx4pQqVYrdu3fz1VdfZbi/Nm3asHjx4rT1jh8/zqZNm9KSQvny5Tl69GimPx769u3LqlWrznhktHzLli35448/2LZtG4mJiUyfPp0rrrgiw+0eOnSIhQsXntbmsmfPnrTqyPj4eObPn0/9+vXT5m/atCktafhSgSlRLF68nfr1y1OuXDGKFAln+vRrqFixOOedF+Xv0IyfiQifffYZ9913H88//zyRkZFER0fzyiuvnLHsI488QvPmzTPcTqNGjWjRokXaCTK948ePn9Zwff/99xMXF8e9995LZGQk4Fwm6W3D5EUXXeTVcp4ef/xxbrrpJu644w4GDRrEgQMHOP/88wkLC+Pcc89l5syZab9Qp0yZwpAhQ6hduzbFihWjXLlyZ/ySrlChApMnT+bqq68mNTWVihUr8s0333DNNdfwzjvv0KxZM1q2bEndunUzjalMmTI0bNiQ9evX06pVKwAaNmzIiBEjuOyyy0hNTSUiIoLx48dTvXr109bt2bMnCxYsoHPnzjRt2pTmzZvTqFEjatasmVa1lF6FChWYOnUqN954Y9plzSNGjKBu3brccccdnH/++URHR6dVe52N8PBwxo0bR9euXUlJSeH222+nUaNGAEyaNAkg7SKKzz77jMsuu+y0EtfOnTu55ZZbSElJITU1leuvv55evXoBTuP25s2biYmJOes4syPB1nAbU1V0+d/ex7xv33Eefng+U6b8yn//25wpUzLO5sZ/NmzYQIMGDfwdhglC8fHxdOzYkcWLF3tdsgoVn332GStXruTpp58+Y15G3ykRWaGqucoqIVv1pKq8/fYq6tcfz5QpvxIRUYhKlaLsiiZjQkjRokV58skns7ySKFQlJyczZMiQfNlXSFY9/f77XmJjZ7NwoVOn2aFDNBMn9qR+/fJ+jswYk9e6du3q7xD84rrrrsu3fYVcooiLO0zTppNITEyhfPlivPzyZfTr18QueQ1wOb0s0xiTMV/UmoRcoqhSpST9+jWhUCHh+ec7U7as7y8dM2cnMjKSffv2WVfjxpwldcejOHlxRF4J+sbsnTuPMHjwPGJjY+jQIRqA1FS18aqDiI1wZ0zeyWyEu7NpzA7aEkVKSioTJy7nkUe+4/DhBDZv3s+yZXcgIpYkgkxERESejsZljMlbPr3qSUS6ichGEdksIg9nMF9E5FV3/hoRaeHNdleu3EmbNm8waNBXHD6cwOWX1+XTT6+3agtjjPEBn5UoRCQMGA90AeKAZSIyS1XXeyzWHajjPloDE92/mfr7YElatnyd1FSlSpWSjB3bnd6961mSMMYYH/FliaIVsFlVt6pqIjAd6J1umd7AO2536UuA0iKSZV8a+48XRQTuv78NGzbczZVX1rckYYwxPuTLNorKwN8e03GcWVrIaJnKwE7PhUTkTuBOdzIBnlg7ahS4PQMXZOWBvf4OIkDYsTjFjsUpdixOqZfbFX2ZKDL6mZ/+EitvlkFVJwOTAURkeW5b7kONHYtT7FicYsfiFDsWp4jI8tyu68uqpzigqsd0FSB9x+neLGOMMcaPfJkolgF1RKSGiBQGbgBmpVtmFnCze/VTG+CQqu5MvyFjjDH+47OqJ1VNFpGBwDwgDHhTVdeJSKw7fxLwJdAD2AwcB27zYtOTfRRyMLJjcYodi1PsWJxix+KUXB+LoLsz2xhjTP4K2W7GjTHG5A1LFMYYY7IUsInCV91/BCMvjkVf9xisEZGfRKSpP+LMD9kdC4/lWopIiohcm5/x5SdvjoWIdBCRVSKyTkQWZrRMKPDiO1JKRL4QkdXusfCmPTToiMibIvKviKzNZH7uzpuqGnAPnMbvLUBNoDCwGmiYbpkewFc492K0AZb6O24/HouLgDLu8+4F+Vh4LPcdzsUS1/o7bj9+LkoD64Fq7nRFf8ftx2MxDBjpPq8A7AcK+zt2HxyLdkALYG0m83N13gzUEoVPuv8IUtkeC1X9SVUPuJNLcO5HCUXefC4ABgGfAv/mZ3D5zJtjcRMwQ1W3A6hqqB4Pb46FAlHi9PdTAidRJOdvmL6nqotw3ltmcnXeDNREkVnXHjldJhTk9H3+F+cXQyjK9liISGXgKmBSPsblD958LuoCZURkgYisEJGb8y26/OXNsRgHNMC5ofc34F5VTc2f8AJKrs6bgToeRZ51/xECvH6fItIRJ1Fc4tOI/MebY/EKMFRVU0K8s0hvjkU4cAHQCSgK/CwiS1R1k6+Dy2feHIuuwCrgUqAW8I2I/KCqh30dXIDJ1XkzUBOFdf9xilfvU0SaAFOA7qq6L59iy2/eHIsYYLqbJMoDPUQkWVU/z58Q842335G9qnoMOCYii4CmQKglCm+OxW3A8+pU1G8WkW1AfeCX/AkxYOTqvBmoVU/W/ccp2R4LEakGzAD6heCvRU/ZHgtVraGq0aoaDXwCDAjBJAHefUdmAm1FJFxEiuH03rwhn+PMD94ci+04JStE5BycnlS35muUgSFX582ALFGo77r/CDpeHovHgXLABPeXdLKGYI+ZXh6LAsGbY6GqG0RkLrAGSAWmqGqGl00GMy8/F08DU0XkN5zql6GqGnLdj4vIB0AHoLyIxAFPABFwdudN68LDGGNMlgK16skYY0yAsERhjDEmS5YojDHGZMkShTHGmCxZojDGGJMlSxQmILk9v67yeERnsezRPNjfVBHZ5u5rpYhcmIttTBGRhu7zYenm/XS2MbrbOXlc1rq9oZbOZvlmItIjL/ZtCi67PNYEJBE5qqol8nrZLLYxFZitqp+IyGXAS6ra5Cy2d9YxZbddEXkb2KSqz2Sx/K1AjKoOzOtYTMFhJQoTFESkhIh86/7a/01Ezug1VkTOE5FFHr+427qvXyYiP7vrfiwi2Z3AFwG13XXvd7e1VkTuc18rLiJz3LEN1opIH/f1BSISIyLPA0XdOKa58466fz/0/IXvlmSu7eK0rQAAA01JREFUEZEwEXlRRJaJM05Afy8Oy8+4HbqJSCtxxiL51f1bz71L+SmgjxtLHzf2N939/JrRcTTmDP7uP90e9sjoAaTgdOK2CvgMpxeBku688jh3lp4sER91/w4BHnGfhwFR7rKLgOLu60OBxzPY31TcsSuA64ClOB3q/QYUx+maeh3QHLgGeN1j3VLu3wU4v97TYvJY5mSMVwFvu88L4/TkWRS4E3jUfb0IsByokUGcRz3e38dAN3e6JBDuPu8MfOo+vxUY57H+s8B/3Oelcfp9Ku7v/7c9AvsRkF14GAPEq2qzkxMiEgE8KyLtcLqjqAycA+zyWGcZ8Ka77OequkpE2gMNgcVu9yaFcX6JZ+RFEXkU2IPTC28n4DN1OtVDRGYAbYG5wEsiMhKnuuqHHLyvr4BXRaQI0A1YpKrxbnVXEzk1Il8poA6wLd36RUVkFRANrAC+8Vj+bRGpg9MbaEQm+78MuEJEHnCnI4FqhGYfUCaPWKIwwaIvzshkF6hqkoj8iXOSS6Oqi9xE0hN4V0ReBA4A36jqjV7s40FV/eTkhIh0zmghVd0kIhfg9JnznIh8rapPefMmVPWEiCzA6fa6D/DByd0Bg1R1XjabiFfVZiJSCpgN3A28itOX0feqepXb8L8gk/UFuEZVN3oTrzFgbRQmeJQC/nWTREegevoFRKS6u8zrwBs4Q0IuAS4WkZNtDsVEpK6X+1wE/9/eHapUEEQBGP5PMBnsPoCIYBCsgo/gI1gNmmxWk1iMilHRV1AEg1gucq9eLuILmEwiWAxjOLMgso5W4f/KsuwOZ9IezpxlhrU6ZppcNrqJiFngvZRyAuzXON991Mqmzzm5GdsKuZEd9brRjYmIuRqzVynlFdgCtuuYGeC5Pl7/8uobuQTXuQA2o5ZXEbH0UwypY6LQf3EKLEfEHVldPPW8swrcR8SI7CMclFJeyA/nWUSMycQx/5eApZQh2bsYkD2L41LKCFgEBnUJaAfY7Rl+BIy7ZvY3l+TZxlclj+6EPEvkERhGxAQ45JeKv87lgdxWe4+sbm7J/kXnGljomtlk5TFV5zap91KTv8dKkpqsKCRJTSYKSVKTiUKS1GSikCQ1mSgkSU0mCklSk4lCktT0CZ4Z/xfQnMYiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "roc_auc_show(pred_mse2, y_labels_seq, figure_label_name='CNN LSTM')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance comparison of LSTM and 1D-Conv models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you've tested both LSTM and the 1D Convultional anomaly detectors, how do you think their accuracy and performance compared?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For LSTM AE\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.80      0.80      0.80       600\n",
      "           1       0.49      0.49      0.49       234\n",
      "\n",
      "    accuracy                           0.71       834\n",
      "   macro avg       0.64      0.64      0.64       834\n",
      "weighted avg       0.71      0.71      0.71       834\n",
      "\n",
      "For 1D Conv AE\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.81      0.81       600\n",
      "           1       0.51      0.51      0.51       234\n",
      "\n",
      "    accuracy                           0.73       834\n",
      "   macro avg       0.66      0.66      0.66       834\n",
      "weighted avg       0.73      0.73      0.73       834\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "y_test = []\n",
    "\n",
    "for i in np.arange(0, len(y_labels), sequence_length):\n",
    "    y_test.append(max(y_labels[i:i+sequence_length]))\n",
    "\n",
    "print('For LSTM AE')\n",
    "print(classification_report(y_test, pred_res))\n",
    "print('For 1D Conv AE')\n",
    "print(classification_report(y_test, pred_res2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STOP HERE -- END OF SESSION 3\n",
    "We will complete Exercise 3 as part of next session of DLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e4\"></a>\n",
    "### Exercise 3: Experimenting with Hyperparameters\n",
    "\n",
    "Now that you've worked your through the entire notebook, feel free to go back and adjust the models to see if you can improve accuracy.  You can add layers, increase filters, and change hyperparameters.  This is where the real art of Deep Learning happens...\n",
    "\n",
    "However, it is possible to take a systematic approach to hyperparameter optimization by sweeping the space of all parameter values and finding those which maximize some metric you're interested in.  We can then define a function to construct our model based on a given set of parameters, as well as define a scoring function which creates and trains the model and evaluates it's performance. There are many approaches to perform hyperparameter optimization using Deep Learning models. We will explore one concept called Bayesian Optimization which is a fairly common approach for this type of optimization.  \n",
    "\n",
    "In this example, we'll continue with the LSTM model designed previously and set up the parameter search space to include adjusting the dropout rate and LSTM width along with a factor describing how much of the suggested threshold to use when classifying a defective disk.  In more complicated models, other hyperparameters such as the number of stacked LSTM layers or convolution kernel width could be included in the search space.  Just keep in mind that each new parameter will expand the search space and require more time to maximize.\n",
    "\n",
    "![Bayesian Optimization](https://raw.githubusercontent.com/fmfn/BayesianOptimization/master/examples/bayesian_optimization.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the python package for optimizing the architecture for the LSTM autoencoder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bayes_opt import BayesianOptimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a class **Hyperparameter** to create inputs for the optimization and provide a mapping between unscaled parameters required by optimizater and scaled parameters used in the method **lstm_autoencoder** defined below:   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Hyperparameter: \n",
    "    \n",
    "    def __init__(self,inputs=None):\n",
    "        \n",
    "        _defaults = {\"min\": 0, \"max\": 1.0, \"count\": 11, \"scale\" : \"linear\", \"type\" : \"float\" }\n",
    "        \n",
    "        for key in sorted(_defaults.keys()): \n",
    "            try:\n",
    "                _defaults[key] = inputs[key]\n",
    "            except:\n",
    "                pass\n",
    "        \n",
    "        if inputs is not None: \n",
    "            for key in sorted(_defaults.keys()):\n",
    "                try:\n",
    "                    eval(\"self.set_\"+key)(_defaults[key])\n",
    "                except:\n",
    "                    print(key + \"not defined for Hyperparameter \" )\n",
    "            self.update()\n",
    "            \n",
    "        print(self)\n",
    "            \n",
    "    def set_min(self,val=None):\n",
    "        self.min = val \n",
    "        \n",
    "    def set_max(self,val=None):\n",
    "        self.max = val \n",
    "        \n",
    "    def set_count(self,val=None):\n",
    "        self.count = val \n",
    "        \n",
    "    def set_step(self,val=None):\n",
    "        self.step = val \n",
    "                     \n",
    "    def set_type(self,val=None):\n",
    "        self.type = val \n",
    "\n",
    "    def set_scale(self,val=None):\n",
    "        self.scale = val \n",
    "        \n",
    "    def update(self):\n",
    "        if self.scale == \"log\":\n",
    "             self.step = (np.log(self.max) - np.log(self.min))/(self.count-1)\n",
    "        else:\n",
    "             self.step = (self.max - self.min)/(self.count-1)\n",
    "        if self.type == \"int\": \n",
    "            self.step = int(self.step)\n",
    "        \n",
    "    def from_unscaled(self,val=None): \n",
    "        if self.scale == \"log\":\n",
    "            _tval =  np.exp(np.log(self.min) + self.step*val)\n",
    "        else:\n",
    "            _tval =  self.min + self.step*val\n",
    "        \n",
    "        if self.type == \"int\":\n",
    "            _tval = int(_tval)\n",
    "        \n",
    "        return _tval \n",
    "    \n",
    "    def to_unscaled(self,val=None): \n",
    "        if self.scale == \"log\":\n",
    "            _tval =  (np.log(val) - np.log(self.min))/self.step\n",
    "        else:\n",
    "            _tval =  (val - self.min)/self.step\n",
    "        \n",
    "        if self.type == \"int\":\n",
    "            _tval = int(_tval)\n",
    "        \n",
    "        return _tval \n",
    "    \n",
    "    def __repr__(self):\n",
    "        \n",
    "        _cur_str = ' Hyperparameter (min: {}, max: {}, scale: {}, step: {}, count: {}, type: {})'.format(self.min,\n",
    "                                         self.max,\n",
    "                                         self.scale,\n",
    "                                         self.step, \n",
    "                                         self.count, \n",
    "                                         self.type )\n",
    "        return  _cur_str \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Method to plot the training history of each architecture "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_history(history,result,lw,lr):\n",
    "    plt.figure()\n",
    "    plt.plot(history['loss'],'g')\n",
    "    plt.plot(history['val_loss'],'r')\n",
    "\n",
    "    plt.ylabel('Loss')\n",
    "    plt.xlabel('# epoch')\n",
    "    plt.legend(['train', 'test'], loc='lower right')\n",
    "    plt.ylim([0.15,0.55])\n",
    "    plt.title(\"LSTM Loss across epochs \\n lstm width {:7d} learning rate {:7.4f} objective {:7.4f}\" .format(lw,lr,result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Next define a function to train the model and return the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model #1 - Wide LSTM-based Encoder-Decoder\n",
    "data_shape = x_train.shape[1:]  \n",
    "\n",
    "dp_lvl = 0.2\n",
    "\n",
    "def lstm_autoencoder(data_shape, params_list):\n",
    "    \n",
    "    \n",
    "    mod_params = {} \n",
    "    mod_params['lstm_width'] = lstm_width.from_unscaled(int(np.round(params_list[0])))\n",
    "    mod_params['learning_rate'] = learning_rate.from_unscaled(params_list[1])\n",
    "    mod_params['num_epochs'] = 20\n",
    "    mod_params['batch_size'] = 128 \n",
    "    \n",
    "    inputs = Input(shape=(data_shape))\n",
    "    encoded = LSTM(int(mod_params[\"lstm_width\"]), dropout = dp_lvl, recurrent_dropout = dp_lvl, return_sequences =  False, activation='relu')(inputs)\n",
    "    decoded = RepeatVector(data_shape[0])(encoded)\n",
    "    decoded = LSTM(int(mod_params[\"lstm_width\"]), return_sequences=True, activation='relu')(decoded)\n",
    "    decoded = LSTM(data_shape[1], activation='tanh', return_sequences=True)(decoded)\n",
    "\n",
    "    # Build entire model\n",
    "    autoencoder = Model(inputs, decoded)\n",
    "    \n",
    "    filename = \"case_%.3e_%d_best_model_ae-lstm.h5\" %  \\\n",
    "                                 (mod_params[\"learning_rate\"],mod_params[\"lstm_width\"])\n",
    "    if print_flag == 1:\n",
    "        print(autoencoder.summary())\n",
    "    return autoencoder, filename, mod_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utility methods for plotting and reporting resutls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_history(num_step, mod_params, loss, result, num_train, num_non_train, run_time):\n",
    "\n",
    "    if num_step == 0: \n",
    "        print(\"\\n| {:>9s} | {:>10s} | {:>13s} | {:>6s} | {:>9s} | {:>9s} | {:>13s} | {:>15s} | \\n|\".format(\"Iteration\", \"lstm width\",\n",
    "                                                                         \"learning rate\",\n",
    "                                                                         \"loss\",\"objective\", \"trainable\", \"non trainable\",\n",
    "                                                                         \"run time (mins)\")+'-'*107+\"|\")\n",
    "        \n",
    "    print( \"| {:>9d} | {:>10d} | {:>13.6f} | {:>6.4f} | {:>9.4f} | {:>9d} | {:>13d} | {:>15.4f} |\".format( num_step + 1,\n",
    "        mod_params['lstm_width'],mod_params['learning_rate'],loss, result, num_train, num_non_train, run_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_model_loss(idx_list,header): \n",
    "    \n",
    "    print(\" \"*20 + header)\n",
    "    for idx in idx_list:\n",
    "        cur_iter = iter_history[idx]\n",
    "        iter_history[idx][\"model\"].summary()\n",
    "        plot_history(cur_iter[\"history\"],cur_iter['objective'],\n",
    "                     cur_iter['params'][\"lstm_width\"],cur_iter['params'][\"learning_rate\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_loss_map(results_list):\n",
    "    results_list_shifted = [r - 0.9*min_res for r in results_list]\n",
    "    plt.figure(figsize=(6,4))\n",
    "    plt.scatter([c['params']['lstm_width'] for c in iter_history],\n",
    "                [c['params']['learning_rate'] for c in iter_history],\n",
    "                s=[r*500 for r in results_list_shifted],\n",
    "                c=[r for r in results_list_shifted ])\n",
    "\n",
    "    for i,c in enumerate(iter_history):\n",
    "        plt.text(c['params']['lstm_width'], c['params']['learning_rate'], r'%d'%(i),fontsize = 14)\n",
    "    plt.xlabel('lstm_width',fontsize=14)\n",
    "    plt.ylabel('learning_rate',fontsize=14)\n",
    "    plt.tick_params(labelsize=12)\n",
    "    plt.yscale('log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Main method for building the autoencoder\n",
    "This method builds the autoencoder and trains using provided data using \n",
    "* method **lstm_autoencoder**\n",
    "* print history of training with the inputs to the architecture of the LSTM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "\n",
    "print_flag = 0 \n",
    "iter_history = []\n",
    "\n",
    "def count_params(model, param_type = \"train\"): # Compute number of params in a model (the actual number of floats)\n",
    "    if param_type == \"train\":\n",
    "        return sum([np.prod(K.get_value(w).shape) for w in model.trainable_weights])\n",
    "    else: \n",
    "        return sum([np.prod(K.get_value(w).shape) for w in model.non_trainable_weights])\n",
    "        \n",
    "def train_and_fit_AE(param1, param2):\n",
    "    \n",
    "    time_start = time.time()\n",
    "    autoencoder, filename, params  = lstm_autoencoder(data_shape, [param1, param2])\n",
    "   \n",
    "    # checkpoint\n",
    "    checkpoint = ModelCheckpoint(data_dir+filename, monitor='val_loss', \n",
    "                                 verbose=print_flag, save_best_only=True, mode='min')\n",
    "    es = EarlyStopping(patience=15, verbose=print_flag)\n",
    "\n",
    "    optimizer = Adam(lr=params['learning_rate'])\n",
    "    autoencoder.compile(optimizer=optimizer, loss='mae')\n",
    "\n",
    "    history = autoencoder.fit(x_train, x_train, batch_size=params['batch_size'], epochs=params['num_epochs'], \n",
    "                              validation_split=0.1, verbose=print_flag, callbacks=[es, checkpoint])\n",
    "    loss = history.history[\"val_loss\"][-1]\n",
    "    if np.isnan(loss) : loss = 1e8\n",
    "    result = np.exp(-20*loss**2) \n",
    "\n",
    "    print_history(len(iter_history), params, loss, result, \n",
    "                  count_params(autoencoder,\"train\"), \n",
    "                  count_params(autoencoder,\"non_train\"),\n",
    "                  (time.time()-time_start)/60.0)\n",
    "    iter_history.append({'objective':result, 'params': params, 'history':history.history, \"model\" : autoencoder})\n",
    "    \n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the parameter space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lstm_width = Hyperparameter({\"min\": 32, \"max\": 128, \"count\": 15, \"type\" : \"int\" })\n",
    "learning_rate = Hyperparameter({\"min\": 0.0005, \"max\": 0.1, \"count\": 15, \"scale\" : \"log\"})\n",
    "\n",
    "pbounds = {\"param1\"       : (0,lstm_width.count-1),\n",
    "           \"param2\"    : (0,learning_rate.count-1)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run one test of our \"train_and_fit\" function to ensure it runs correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print_flag = 1\n",
    "train_and_fit_AE(lstm_width.to_unscaled(80),learning_rate.to_unscaled(0.003317))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's the define Bayesian Optimization environment using the our model function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = BayesianOptimization(\n",
    "    f=train_and_fit_AE,\n",
    "    pbounds=pbounds,\n",
    "    verbose=0, # verbose = 1 prints only when a maximum is observed, verbose = 0 is silent\n",
    "    random_state=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main call to optimizer\n",
    "Run the optimizer to explore the parameter space. Each call to the method can take a few minutes. So the total time is almost 15-20 minutes. How can this be improved ? \n",
    "\n",
    "Suggestions for  experimentation (outside of current DLI): \n",
    "* **init_points** specifies the initial number of calls to the **lstm_autoencoder** using random parameters. This is used to get initial values to populate our search space. This can be done in parallel. \n",
    "* **n_iter** specifies the subsequent number of steps to leverage the prior results to search through new architectures using the most optimal possible parameters.  This is where we leverage Bayesian Optimization to optimize our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print_flag = 0\n",
    "iter_history = []\n",
    "\n",
    "init_points = 8 \n",
    "num_steps = 8 \n",
    "\n",
    "optimizer.maximize(\n",
    "    init_points=init_points,\n",
    "    n_iter=num_steps,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> \n",
    "\n",
    "### Example output from optimization \n",
    "\n",
    "<img src=\"./images/optresult.png\" alt=\"optresult\"/>\n",
    "\n",
    "<!-- ![image.png](attachment:image.png) -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> \n",
    "\n",
    "### Processing the results\n",
    "Bayesian optimization has provided several options for improved architectures. Some of the architectures are similar while there are quite a few which are different. It is useful to sort the architectures and take a look at the best performing ones. The next few steps sort and plot the best architectures. \n",
    "#### Sort the architectures with loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_list = [cur_iter['objective'] for cur_iter in iter_history]\n",
    "\n",
    "idx_sorted = sorted(range(len(results_list)), key=results_list.__getitem__)\n",
    "min_res = results_list[idx_sorted[0]]\n",
    "max_res = results_list[idx_sorted[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a3\"></a>\n",
    "#### Plot the loss per epoch for the best three cases "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[-1-i] for i in range(3)],\"Best cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> \n",
    "\n",
    "**An example loss curve for a good architecture**\n",
    "![Loss example 1](./images/loss_crv_ex1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a4\"></a>\n",
    "#### Plot the loss per epoch for the worst three cases "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[i] for i in range(3)],\"Worst cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**A typical loss curve for a bad architecture**\n",
    "![Loss example 2](./images/loss_crv_ex2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a5\"></a>\n",
    "#### Plot different architectures in a 2D space representating the different architectures with the minimum loss as the size "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_loss_map(results_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**An example map for the loss with the inputs for the lstm autoencoder** \n",
    "![Loss example 1](./images/loss_map.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e5\"></a>\n",
    "### Exercise 4: Update code to use specified validation data\n",
    "\n",
    "The test data can also be specified as a separate dataset rather than being specified as a fraction of the dataset. This is useful especially in situations where it is important to have a certain structure in the test data representative of the training dataset, rather than a randomly sampled section of the main dataset. **train_and_fit_AE** method needs to be specified. \n",
    "\n",
    "As an exercise, please update the << TO DO >> below with the appropriate validation data using the test data set as the features and label.\n",
    "\n",
    "<a name=\"q1\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "\n",
    "print_flag = 0 \n",
    "iter_history = []\n",
    "\n",
    "def train_and_fit_AE(param1, param2):\n",
    "    \n",
    "    time_start = time.time()\n",
    "    autoencoder, filename, params  = lstm_autoencoder(data_shape, [param1, param2])\n",
    "   \n",
    "    # checkpoint\n",
    "    checkpoint = ModelCheckpoint(data_dir+filename, monitor='val_loss', \n",
    "                                 verbose=print_flag, save_best_only=True, mode='min')\n",
    "    es = EarlyStopping(patience=15, verbose=print_flag)\n",
    "\n",
    "    optimizer = Adam(lr=params['learning_rate'])\n",
    "    autoencoder.compile(optimizer=optimizer, loss='mae')\n",
    "\n",
    "    history = autoencoder.fit(x_train, x_train, batch_size=params['batch_size'], epochs=params['num_epochs'], \n",
    "                              validation_data=<<TO DO>>, verbose=print_flag, callbacks=[es, checkpoint])\n",
    "    \n",
    "    loss = history.history[\"val_loss\"][-1]\n",
    "    if np.isnan(loss) : loss = 1e8\n",
    "    result = np.exp(-20*loss**2) \n",
    "\n",
    "    print_history(len(iter_history), params, loss, result, \n",
    "                  count_params(autoencoder,\"train\"), \n",
    "                  count_params(autoencoder,\"non_train\"),\n",
    "                  (time.time()-time_start)/60.0)\n",
    "    iter_history.append({'objective':result, 'params': params, 'history':history.history, \"model\" : autoencoder})\n",
    "        \n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](#a2) to see answer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "optimizer = BayesianOptimization(\n",
    "    f=train_and_fit_AE,\n",
    "    pbounds=pbounds,\n",
    "    verbose=0, # verbose = 1 prints only when a maximum is observed, verbose = 0 is silent\n",
    "    random_state=1,\n",
    ")\n",
    "\n",
    "print_flag = 0\n",
    "iter_history = []\n",
    "\n",
    "init_points = 8 \n",
    "num_steps = 8 \n",
    "\n",
    "optimizer.maximize(\n",
    "    init_points=init_points,\n",
    "    n_iter=num_steps,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"e6\"></a>\n",
    "### Exercise 5: Create HPO result plots \n",
    "Now the results of this new hyperparameter optimization can be analyzed in a similar way as the previous particular dataset.\n",
    "* sort the architectures \n",
    "* loss vs epoch for best three architectures \n",
    "* loss vs epoch for worst three architectures \n",
    "* map of training and test loss with key parameters of the architectures - learning rate and lstm width \n",
    "\n",
    "The code required below is used from the plotting above. <br>\n",
    "Use the hyperlinks to go to the appropriate session and scroll down to this section to complete the exercise. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sort the architectures with loss   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_list = [cur_iter['objective'] for cur_iter in iter_history]\n",
    "\n",
    "idx_sorted = sorted(range(len(results_list)), key=results_list.__getitem__)\n",
    "min_res = results_list[idx_sorted[0]]\n",
    "max_res = results_list[idx_sorted[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the loss per epoch for the best three cases**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[-1-i] for i in range(3)],\"Best cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the loss per epoch for the worst three cases**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[i] for i in range(3)],\"Worst cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot different architectures in a 2D space representating the different architectures with the minimum loss as the size**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_loss_map(results_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Results of specifying a different validation dataset\n",
    "It is important to understand the role of data in prediction quality of models. In the first exercise of hyperparameter optimization, a fraction of the main dataset is used which is randomly sampled. In the second exercise of hyperparameter optimization, the test dataset for XGBoost in Lab1 is used. The distribution of the sensor data is different from the main dataset. A histogram of the both datasets is shown below. \n",
    "<br>\n",
    "\n",
    "This leads to the difference in training of the autoencoder models and the loss for each model and, eventually an impact on the direction of hyperparameter optimization. \n",
    "\n",
    "<img src=\"./images/histogram_lab1.png\" alt=\"histogram lab 1\" width=\"700\"/>\n",
    "\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Exercise\n",
    "\n",
    "Update **train_and_fit_AE** to optimize the autoencoder based on 1D convolutions example discussed in this lab. \n",
    "* What are the key components to perform the hyperparameter optimization ?\n",
    "* What can be inputs to the optimization ? \n",
    "* How should the objective function be structured ?   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Define the method to create the autoencoder model with input parameters using tf.keras "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model #2 -Conv1D-based Encoder-Decoder\n",
    "data_shape = x_train.shape[1:]  \n",
    "\n",
    "dp_lvl = 0.2\n",
    "\n",
    "def conv1d_autoencoder(data_shape, params_list):\n",
    "    \n",
    "    \n",
    "    mod_params = {} \n",
    "    #convert from unscaled inputs provided by optimizer using params_list \n",
    "    \n",
    "    inputs = Input(shape=(data_shape))\n",
    "    #add code for Conv1D \n",
    "    << TO DO>> \n",
    "\n",
    "    # Build entire model\n",
    "    autoencoder = Model(inputs, <<TO DO>> )\n",
    "    \n",
    "    # provide an output filename for storing model weights for lowest loss for the current architecture\n",
    "    filename = << TO DO>> \n",
    "     \n",
    "    if print_flag == 1:\n",
    "        print(autoencoder.summary())\n",
    "    return autoencoder, filename, mod_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Define the method to create the autoencoder model with input parameters using tf.keras. No change in needed but the cell below must be run "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_flag = 0 \n",
    "iter_history = []\n",
    "\n",
    "def train_and_fit_AE(param1, param2):\n",
    "    \n",
    "    time_start = time.time()\n",
    "    autoencoder, filename, params  = conv1d_autoencoder(data_shape, [param1, param2])\n",
    "   \n",
    "    # checkpoint\n",
    "    checkpoint = ModelCheckpoint(data_dir+filename, monitor='val_loss', \n",
    "                                 verbose=print_flag, save_best_only=True, mode='min')\n",
    "    es = EarlyStopping(patience=15, verbose=print_flag)\n",
    "\n",
    "    optimizer = Adam(lr=params['learning_rate'])\n",
    "    autoencoder.compile(optimizer=optimizer, loss='mae')\n",
    "\n",
    "    history = autoencoder.fit(x_train, x_train, batch_size=params['batch_size'], epochs=params['num_epochs'], \n",
    "                              validation_split=0.1, verbose=print_flag, callbacks=[es, checkpoint])\n",
    "    loss = history.history[\"val_loss\"][-1]\n",
    "    if np.isnan(loss) : loss = 1e8\n",
    "    result = np.exp(-20*loss**2) \n",
    "\n",
    "    print_history(len(iter_history), params, loss, result, \n",
    "                  count_params(autoencoder,\"train\"), \n",
    "                  count_params(autoencoder,\"non_train\"),\n",
    "                  (time.time()-time_start)/60.0)\n",
    "    iter_history.append({'objective':result, 'params': params, 'history':history.history, \"model\" : autoencoder})\n",
    "        \n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Define the method to create the autoencoder model with input parameters using tf.keras "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv1d_param1 = Hyperparameter({\"min\": <<TO DO>>, \"max\": <<TO DO>>, \"count\": <<TO DO>>, \"type\" : <<TO DO>> })\n",
    "conv1d_param2 = Hyperparameter({\"min\": <<TO DO>>, \"max\": <<TO DO>>, \"count\": <<TO DO>>, \"scale\" : <<TO DO>> })\n",
    "\n",
    "pbounds = {\"param1\"    : (0,conv1d_param1.count-1),\n",
    "           \"param2\"    : (0,conv1d_param2.count-1)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Define the method to create the autoencoder model with input parameters using tf.keras "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_flag = 1\n",
    "train_and_fit_AE(conv1d_param1.to_unscaled(<<TO DO>>),conv1d_param2.to_unscaled(<<TO DO>>))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the optimizer using the new method to call Conv1d based autoencoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = BayesianOptimization(\n",
    "    f=train_and_fit_AE,\n",
    "    pbounds=pbounds,\n",
    "    verbose=0, # verbose = 1 prints only when a maximum is observed, verbose = 0 is silent\n",
    "    random_state=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run optimization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print_flag = 0\n",
    "iter_history = []\n",
    "\n",
    "init_points = 8 \n",
    "num_steps = 8 \n",
    "\n",
    "optimizer.maximize(\n",
    "    init_points=init_points,\n",
    "    n_iter=num_steps,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sort architectures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_list = [cur_iter['objective'] for cur_iter in iter_history]\n",
    "\n",
    "idx_sorted = sorted(range(len(results_list)), key=results_list.__getitem__)\n",
    "min_res = results_list[idx_sorted[0]]\n",
    "max_res = results_list[idx_sorted[-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the loss per epoch for the best three cases**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[-1-i] for i in range(3)],\"Best cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the loss per epoch for the worst three cases**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model_loss([idx_sorted[i] for i in range(3)],\"Worst cases\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot different architectures in a 2D space representating the different architectures with the minimum loss as the size**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_loss_map(results_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this course, we tackled the predictive maintenance problem using different approaches. Mainly, we used a machine-learning-based approach called XGBoost to classify the defective hard drive samples. Next, we used recurrent neural networks to predict hard drive feature sequences and indicate whether the timeseries ends in a fail state or not. Finally, we used an anomaly detection approach to classify samples into defective/normal bins.\n",
    "One important factor affecting the outcome of all the models experimented in this course, is the number of data sequences. While we had the liberty of choosing among a dataset of 6M samples, the defective hard drives were significantly less. This challenge affected the outcome of the models discussed so far. To overcome similar issues in industrial applications, the reader is encouraged to investigate data augmentation techniques to increase the training set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"Updating the autoencoder validation data\"></a>\n",
    "## Answers to selected exercises:\n",
    "---\n",
    "\n",
    "<a name=\"a1\"></a>\n",
    "Exercise 1: Updating the autoencoder validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_Labels(df):\n",
    "    # return the proper column in NumPy format\n",
    "    y_labels = df['failure'].values\n",
    "    return y_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a1\"></a>\n",
    "Click [here](#e1) to go back"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise 4: Updating the autoencoder validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_data=(x_test, x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"a2\"></a>\n",
    "Click [here](#q1) to go back"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answers to Optional Exercise\n",
    "\n",
    "Click [here](Lab3-AE-For-Anomaly-Detection-soln.ipynb) to go through the complete solution of this lab"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}