t81_558_class_09_4_transfer_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_09_4_transfer_nlp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Transfer Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Material\n",
    "\n",
    "* Part 9.1: Introduction to Keras Transfer Learning [[Video]](https://www.youtube.com/watch?v=WLlP6S-Z8Xs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_1_keras_transfer.ipynb)\n",
    "* Part 9.2: Popular Pretrained Neural Networks for Keras [[Video]](https://www.youtube.com/watch?v=ctVA1_46YEE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_2_popular_transfer.ipynb)\n",
    "* Part 9.3: Transfer Learning for Computer Vision and Keras [[Video]](https://www.youtube.com/watch?v=61vMUm_XBMI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_3_transfer_cv.ipynb)\n",
    "* **Part 9.4: Transfer Learning for Languages and Keras** [[Video]](https://www.youtube.com/watch?v=ajmAAg9FxXA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_4_transfer_nlp.ipynb)\n",
    "* Part 9.5: Transfer Learning for Keras Feature Engineering [[Video]](https://www.youtube.com/watch?v=Dttxsm8zpL8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_5_transfer_feature_eng.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9.4: Transfer Learning for Languages and Keras\n",
    "\n",
    "You will commonly use transfer learning in conjunction with Natural Language Processing (NLP).  This course has an entire module that covers NLP.  However, we will look at how you can load a network into Keras for NLP via transfer learning.  The following three sources were helpful for the creation of this section.\n",
    "\n",
    "* Universal sentence encoder [[Cite:cer2018universal]](https://arxiv.org/abs/1803.11175). arXiv preprint arXiv:1803.11175)\n",
    "* Deep Transfer Learning for Natural Language Processing: Text Classification with Universal Embeddings [[Cite:howard2018universal]](https://towardsdatascience.com/deep-transfer-learning-for-natural-language-processing-text-classification-with-universal-1a2c69e5baa9)\n",
    "* [Keras Tutorial: How to Use Google's Universal Sentence Encoder for Spam Classification](http://hunterheidenreich.com/blog/google-universal-sentence-encoder-in-keras/)\n",
    "\n",
    "These examples make use of TensorFlow Hub, which allows pretrained models to be loaded into TensorFlow easily.  To install TensorHub use the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow_hub\n",
      "  Downloading tensorflow_hub-0.9.0-py2.py3-none-any.whl (103 kB)\n",
      "Requirement already satisfied: six>=1.12.0 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_hub) (1.15.0)\n",
      "Requirement already satisfied: protobuf>=3.8.0 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_hub) (3.12.3)\n",
      "Requirement already satisfied: numpy>=1.12.0 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_hub) (1.18.5)\n",
      "Requirement already satisfied: setuptools in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from protobuf>=3.8.0->tensorflow_hub) (49.2.0.post20200714)\n",
      "Installing collected packages: tensorflow-hub\n",
      "Successfully installed tensorflow-hub-0.9.0\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow_hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also necessary to install TensorFlow Datasets, which you can install \\with the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow_datasets\n",
      "  Downloading tensorflow_datasets-3.2.1-py3-none-any.whl (3.4 MB)\n",
      "Requirement already satisfied: requests>=2.19.0 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (2.24.0)\n",
      "Requirement already satisfied: wrapt in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (1.12.1)\n",
      "Collecting dill\n",
      "  Downloading dill-0.3.2.zip (177 kB)\n",
      "Requirement already satisfied: attrs>=18.1.0 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (19.3.0)\n",
      "Requirement already satisfied: absl-py in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (0.9.0)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (3.12.3)\n",
      "Collecting promise\n",
      "  Downloading promise-2.3.tar.gz (19 kB)\n",
      "Requirement already satisfied: numpy in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (1.18.5)\n",
      "Collecting tensorflow-metadata\n",
      "  Downloading tensorflow_metadata-0.23.0-py3-none-any.whl (43 kB)\n",
      "Requirement already satisfied: future in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (0.18.2)\n",
      "Requirement already satisfied: six in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (1.15.0)\n",
      "Requirement already satisfied: tqdm in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (4.47.0)\n",
      "Requirement already satisfied: termcolor in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from tensorflow_datasets) (1.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from requests>=2.19.0->tensorflow_datasets) (2020.6.20)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from requests>=2.19.0->tensorflow_datasets) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from requests>=2.19.0->tensorflow_datasets) (3.0.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from requests>=2.19.0->tensorflow_datasets) (2.10)\n",
      "Requirement already satisfied: setuptools in c:\\users\\jeffh\\miniconda3\\envs\\tensorflow\\lib\\site-packages (from protobuf>=3.6.1->tensorflow_datasets) (49.2.0.post20200714)\n",
      "Collecting googleapis-common-protos\n",
      "  Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)\n",
      "Building wheels for collected packages: dill, promise\n",
      "  Building wheel for dill (setup.py): started\n",
      "  Building wheel for dill (setup.py): finished with status 'done'\n",
      "  Created wheel for dill: filename=dill-0.3.2-py3-none-any.whl size=78977 sha256=c9ee55462820f4e66c44c76f46eb499453725a7339436b18eb0947153d0d2592\n",
      "  Stored in directory: c:\\users\\jeffh\\appdata\\local\\pip\\cache\\wheels\\72\\6b\\d5\\5548aa1b73b8c3d176ea13f9f92066b02e82141549d90e2100\n",
      "  Building wheel for promise (setup.py): started\n",
      "  Building wheel for promise (setup.py): finished with status 'done'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: tensorflow-metadata 0.23.0 has requirement absl-py<0.9,>=0.7, but you'll have absl-py 0.9.0 which is incompatible.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=21499 sha256=e1f17d3a5a840906a2af17e7a80c4013d8cf29812f014a8ad6413bc21f126faa\n",
      "  Stored in directory: c:\\users\\jeffh\\appdata\\local\\pip\\cache\\wheels\\29\\93\\c6\\762e359f8cb6a5b69c72235d798804cae523bbe41c2aa8333d\n",
      "Successfully built dill promise\n",
      "Installing collected packages: dill, promise, googleapis-common-protos, tensorflow-metadata, tensorflow-datasets\n",
      "Successfully installed dill-0.3.2 googleapis-common-protos-1.52.0 promise-2.3 tensorflow-datasets-3.2.1 tensorflow-metadata-0.23.0\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow_datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the Internet Movie DataBase (IMDB) reviews data set. This example is based on a TensorFlow example that you can [find here](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_text_classification.ipynb#scrollTo=2ew7HTbPpCJH)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\\Users\\jeffh\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0...\u001b[0m\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e91454a3cfa4da29e7031f9e4dc46ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Completed...', max=1.0, style=Progre…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f73bca3621f740c1a35b98eb37e0b8fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Size...', max=1.0, style=ProgressSty…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to C:\\Users\\jeffh\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0.incomplete7JB40W\\imdb_reviews-train.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd991f960f174653a16e79e770d49fa4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to C:\\Users\\jeffh\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0.incomplete7JB40W\\imdb_reviews-test.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "afe4be83b8594ef28a2007815080787a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shuffling and writing examples to C:\\Users\\jeffh\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0.incomplete7JB40W\\imdb_reviews-unsupervised.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05a0bdf8c11144638ba463bebec52b3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset imdb_reviews downloaded and prepared to C:\\Users\\jeffh\\tensorflow_datasets\\imdb_reviews\\plain_text\\1.0.0. Subsequent calls will reuse this data.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "train_data, test_data = tfds.load(name=\"imdb_reviews\", \n",
    "                                  split=[\"train\", \"test\"], \n",
    "                                  batch_size=-1, as_supervised=True)\n",
    "\n",
    "train_examples, train_labels = tfds.as_numpy(train_data)\n",
    "test_examples, test_labels = tfds.as_numpy(test_data)\n",
    "\n",
    "# /Users/jheaton/tensorflow_datasets/imdb_reviews/plain_text/0.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a pretrained embedding model called [gnews-swivel-20dim](https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1).  This network was trained by Google on GNEWS data and can convert RAW text into vectors.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\"\n",
    "hub_layer = hub.KerasLayer(model, output_shape=[20], input_shape=[], \n",
    "                           dtype=tf.string, trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following three movie reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([b\"This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like Christopher Walken's good name. I could barely sit through it.\",\n",
       "       b'I have been known to fall asleep during films, but this is usually due to a combination of things including, really tired, being warm and comfortable on the sette and having just eaten a lot. However on this occasion I fell asleep because the film was rubbish. The plot development was constant. Constantly slow and boring. Things seemed to happen, but with no explanation of what was causing them or why. I admit, I may have missed part of the film, but i watched the majority of it and everything just seemed to happen of its own accord without any real concern for anything else. I cant recommend this film at all.',\n",
       "       b'Mann photographs the Alberta Rocky Mountains in a superb fashion, and Jimmy Stewart and Walter Brennan give enjoyable performances as they always seem to do. <br /><br />But come on Hollywood - a Mountie telling the people of Dawson City, Yukon to elect themselves a marshal (yes a marshal!) and to enforce the law themselves, then gunfighters battling it out on the streets for control of the town? <br /><br />Nothing even remotely resembling that happened on the Canadian side of the border during the Klondike gold rush. Mr. Mann and company appear to have mistaken Dawson City for Deadwood, the Canadian North for the American Wild West.<br /><br />Canadian viewers be prepared for a Reefer Madness type of enjoyable howl with this ludicrous plot, or, to shake your head in disgust.'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_examples[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding layer can convert each to 20-number vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 20), dtype=float32, numpy=\n",
       "array([[ 1.765786  , -3.882232  ,  3.9134233 , -1.5557289 , -3.3362343 ,\n",
       "        -1.7357955 , -1.9954445 ,  1.2989551 ,  5.081598  , -1.1041286 ,\n",
       "        -2.0503852 , -0.72675157, -0.65675956,  0.24436149, -3.7208383 ,\n",
       "         2.0954835 ,  2.2969332 , -2.0689783 , -2.9489717 , -1.1315987 ],\n",
       "       [ 1.8804485 , -2.5852382 ,  3.4066997 ,  1.0982676 , -4.056685  ,\n",
       "        -4.891284  , -2.785554  ,  1.3874227 ,  3.8476458 , -0.9256538 ,\n",
       "        -1.896706  ,  1.2113281 ,  0.11474707,  0.76209456, -4.8791065 ,\n",
       "         2.906149  ,  4.7087674 , -2.3652055 , -3.5015898 , -1.6390051 ],\n",
       "       [ 0.71152234, -0.6353217 ,  1.7385626 , -1.1168286 , -0.5451594 ,\n",
       "        -1.1808156 ,  0.09504455,  1.4653089 ,  0.66059524,  0.79308075,\n",
       "        -2.2268345 ,  0.07446612, -1.4075904 , -0.70645386, -1.907037  ,\n",
       "         1.4419787 ,  1.9551861 , -0.42660055, -2.8022065 ,  0.43727064]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hub_layer(train_examples[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add additional layers to attempt to classify the movie reviews as either positive or negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "keras_layer (KerasLayer)     (None, 20)                400020    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                336       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 400,373\n",
      "Trainable params: 400,373\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "model.add(hub_layer)\n",
    "model.add(tf.keras.layers.Dense(16, activation='relu'))\n",
    "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split and train the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = train_examples[:10000]\n",
    "partial_x_train = train_examples[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/40\n",
      "15000/15000 [==============================] - 3s 189us/sample - loss: 0.6388 - accuracy: 0.6433 - val_loss: 0.5910 - val_accuracy: 0.6937\n",
      "Epoch 2/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.5626 - accuracy: 0.7191 - val_loss: 0.5495 - val_accuracy: 0.7295\n",
      "Epoch 3/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.5173 - accuracy: 0.7573 - val_loss: 0.5138 - val_accuracy: 0.7585\n",
      "Epoch 4/40\n",
      "15000/15000 [==============================] - 2s 145us/sample - loss: 0.4774 - accuracy: 0.7839 - val_loss: 0.4809 - val_accuracy: 0.7832\n",
      "Epoch 5/40\n",
      "15000/15000 [==============================] - 2s 146us/sample - loss: 0.4393 - accuracy: 0.8106 - val_loss: 0.4546 - val_accuracy: 0.7973\n",
      "Epoch 6/40\n",
      "15000/15000 [==============================] - 2s 146us/sample - loss: 0.4038 - accuracy: 0.8305 - val_loss: 0.4239 - val_accuracy: 0.8155\n",
      "Epoch 7/40\n",
      "15000/15000 [==============================] - 2s 144us/sample - loss: 0.3702 - accuracy: 0.8500 - val_loss: 0.3997 - val_accuracy: 0.8288\n",
      "Epoch 8/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.3392 - accuracy: 0.8642 - val_loss: 0.3811 - val_accuracy: 0.8371\n",
      "Epoch 9/40\n",
      "15000/15000 [==============================] - 2s 144us/sample - loss: 0.3133 - accuracy: 0.8787 - val_loss: 0.3601 - val_accuracy: 0.8462\n",
      "Epoch 10/40\n",
      "15000/15000 [==============================] - 2s 144us/sample - loss: 0.2876 - accuracy: 0.8895 - val_loss: 0.3457 - val_accuracy: 0.8534\n",
      "Epoch 11/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.2653 - accuracy: 0.9009 - val_loss: 0.3341 - val_accuracy: 0.8602\n",
      "Epoch 12/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.2454 - accuracy: 0.9103 - val_loss: 0.3267 - val_accuracy: 0.8635\n",
      "Epoch 13/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.2280 - accuracy: 0.9178 - val_loss: 0.3174 - val_accuracy: 0.8676\n",
      "Epoch 14/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.2117 - accuracy: 0.9259 - val_loss: 0.3119 - val_accuracy: 0.8701\n",
      "Epoch 15/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.1978 - accuracy: 0.9321 - val_loss: 0.3074 - val_accuracy: 0.8715\n",
      "Epoch 16/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.1847 - accuracy: 0.9371 - val_loss: 0.3059 - val_accuracy: 0.8714\n",
      "Epoch 17/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.1725 - accuracy: 0.9426 - val_loss: 0.3024 - val_accuracy: 0.8739\n",
      "Epoch 18/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.1616 - accuracy: 0.9471 - val_loss: 0.3079 - val_accuracy: 0.8725\n",
      "Epoch 19/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.1508 - accuracy: 0.9519 - val_loss: 0.3022 - val_accuracy: 0.8747\n",
      "Epoch 20/40\n",
      "15000/15000 [==============================] - 2s 145us/sample - loss: 0.1406 - accuracy: 0.9567 - val_loss: 0.3026 - val_accuracy: 0.8745\n",
      "Epoch 21/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.1317 - accuracy: 0.9593 - val_loss: 0.3043 - val_accuracy: 0.8760\n",
      "Epoch 22/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.1237 - accuracy: 0.9629 - val_loss: 0.3058 - val_accuracy: 0.8763\n",
      "Epoch 23/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.1154 - accuracy: 0.9668 - val_loss: 0.3081 - val_accuracy: 0.8765\n",
      "Epoch 24/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.1077 - accuracy: 0.9697 - val_loss: 0.3116 - val_accuracy: 0.8773\n",
      "Epoch 25/40\n",
      "15000/15000 [==============================] - 2s 140us/sample - loss: 0.1011 - accuracy: 0.9737 - val_loss: 0.3153 - val_accuracy: 0.8759\n",
      "Epoch 26/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0945 - accuracy: 0.9761 - val_loss: 0.3179 - val_accuracy: 0.8762\n",
      "Epoch 27/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.0886 - accuracy: 0.9787 - val_loss: 0.3224 - val_accuracy: 0.8757\n",
      "Epoch 28/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0826 - accuracy: 0.9805 - val_loss: 0.3275 - val_accuracy: 0.8750\n",
      "Epoch 29/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0768 - accuracy: 0.9828 - val_loss: 0.3322 - val_accuracy: 0.8744\n",
      "Epoch 30/40\n",
      "15000/15000 [==============================] - 2s 140us/sample - loss: 0.0718 - accuracy: 0.9851 - val_loss: 0.3376 - val_accuracy: 0.8743\n",
      "Epoch 31/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0672 - accuracy: 0.9860 - val_loss: 0.3430 - val_accuracy: 0.8747\n",
      "Epoch 32/40\n",
      "15000/15000 [==============================] - 2s 140us/sample - loss: 0.0625 - accuracy: 0.9885 - val_loss: 0.3506 - val_accuracy: 0.8723\n",
      "Epoch 33/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0583 - accuracy: 0.9889 - val_loss: 0.3546 - val_accuracy: 0.8734\n",
      "Epoch 34/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0546 - accuracy: 0.9903 - val_loss: 0.3619 - val_accuracy: 0.8732\n",
      "Epoch 35/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.0509 - accuracy: 0.9916 - val_loss: 0.3679 - val_accuracy: 0.8713\n",
      "Epoch 36/40\n",
      "15000/15000 [==============================] - 2s 142us/sample - loss: 0.0473 - accuracy: 0.9923 - val_loss: 0.3751 - val_accuracy: 0.8721\n",
      "Epoch 37/40\n",
      "15000/15000 [==============================] - 2s 141us/sample - loss: 0.0441 - accuracy: 0.9938 - val_loss: 0.3817 - val_accuracy: 0.8707\n",
      "Epoch 38/40\n",
      "15000/15000 [==============================] - 2s 143us/sample - loss: 0.0410 - accuracy: 0.9945 - val_loss: 0.3888 - val_accuracy: 0.8700\n",
      "Epoch 39/40\n",
      "15000/15000 [==============================] - 2s 144us/sample - loss: 0.0384 - accuracy: 0.9949 - val_loss: 0.3966 - val_accuracy: 0.8707\n",
      "Epoch 40/40\n",
      "15000/15000 [==============================] - 2s 145us/sample - loss: 0.0359 - accuracy: 0.9955 - val_loss: 0.4033 - val_accuracy: 0.8684\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=40,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 4s 167us/sample - loss: 0.4406 - accuracy: 0.8523\n",
      "[0.44060500403404235, 0.85228]\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(test_data, test_labels)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhU5ZXH8e9hk12UxYVmM4IEBRpsEQUJJpq4RRQxQjooYlQ0alxGRUkUdZhJIpMYo8ZBjZqIQUcjQYOaoBJwibJqRBFRQVpcAAVBEAHP/PHepoumqnqr21Xd9fs8Tz1Vd6lbp29331Pves3dERGR/NUg2wGIiEh2KRGIiOQ5JQIRkTynRCAikueUCERE8pwSgYhInlMikIwysyfN7KxM75tNZrbCzI6J4bhuZgdGr+80s59XZt9qfE6xmf29unGmOe5QMyvJ9HGl9jXKdgCSfWa2KWGxObAV2BEtn+/uUyt7LHc/Po596zt3H5eJ45hZV+A9oLG7b4+OPRWo9O9Q8o8SgeDuLUtfm9kK4MfuPqv8fmbWqPTiIiL1h6qGJKXSor+ZXW1mHwH3mtleZvaEma0xs8+i1wUJ75ltZj+OXo8xs+fNbHK073tmdnw19+1mZnPMbKOZzTKz283sgRRxVybGm8zsheh4fzezdgnbR5vZSjNbZ2YT0pyfgWb2kZk1TFh3qpm9Fr0eYGYvmdl6M/vQzG4zsyYpjnWfmf1nwvKV0XtWm9nYcvueaGaLzOxzM1tlZhMTNs+Jnteb2SYzO6L03Ca8/0gzm2dmG6LnIyt7btIxs29G719vZkvM7OSEbSeY2RvRMT8ws/+I1reLfj/rzexTM5trZrou1TKdcKnIvsDeQBfgPMLfzL3RcmdgC3BbmvcfDrwFtAN+BdxjZlaNfR8EXgHaAhOB0Wk+szIx/hA4G+gANAFKL0y9gN9Hx98/+rwCknD3fwFfAN8ud9wHo9c7gMuin+cI4DvAhWniJorhuCieY4HuQPn2iS+AM4E2wInABWZ2SrRtSPTcxt1buvtL5Y69N/A34NboZ/s18Dcza1vuZ9jt3FQQc2PgceDv0fsuBqaa2UHRLvcQqhlbAYcAz0brrwBKgPbAPsC1gOa9qWVKBFKRr4Hr3X2ru29x93Xu/qi7b3b3jcAk4Ftp3r/S3e9y9x3A/cB+hH/4Su9rZp2Bw4Dr3P0rd38emJHqAysZ473uvszdtwAPA4XR+hHAE+4+x923Aj+PzkEqfwZGAZhZK+CEaB3uvsDd/+Xu2919BfC/SeJI5gdRfK+7+xeExJf4881293+7+9fu/lr0eZU5LoTE8ba7/ymK68/AUuD7CfukOjfpDARaAr+IfkfPAk8QnRtgG9DLzFq7+2fuvjBh/X5AF3ff5u5zXROg1TolAqnIGnf/snTBzJqb2f9GVSefE6oi2iRWj5TzUekLd98cvWxZxX33Bz5NWAewKlXAlYzxo4TXmxNi2j/x2NGFeF2qzyJ8+x9uZnsAw4GF7r4yiqNHVO3xURTHfxFKBxXZJQZgZbmf73Azey6q+toAjKvkcUuPvbLcupVAx4TlVOemwpjdPTFpJh73NEKSXGlm/zSzI6L1NwPLgb+b2btmNr5yP4ZkkhKBVKT8t7MrgIOAw929NWVVEamqezLhQ2BvM2uesK5Tmv1rEuOHiceOPrNtqp3d/Q3CBe94dq0WglDFtBToHsVxbXViIFRvJXqQUCLq5O57AncmHLeib9OrCVVmiToDH1QiroqO26lc/f7O47r7PHcfRqg2mk4oaeDuG939Cnc/gFAqudzMvlPDWKSKlAikqloR6tzXR/XN18f9gdE37PnARDNrEn2b/H6at9QkxkeAk8xscNSweyMV/588CFxCSDj/Vy6Oz4FNZtYTuKCSMTwMjDGzXlEiKh9/K0IJ6UszG0BIQKXWEKqyDkhx7JlADzP7oZk1MrMzgF6EapyaeJnQdnGVmTU2s6GE39G06HdWbGZ7uvs2wjnZAWBmJ5nZgVFbUOn6Hck/QuKiRCBVdQvQDFgL/At4qpY+t5jQ4LoO+E/gIcJ4h2SqHaO7LwF+Qri4fwh8RmjMTOfPwFDgWXdfm7D+PwgX6Y3AXVHMlYnhyehneJZQbfJsuV0uBG40s43AdUTfrqP3bia0ibwQ9cQZWO7Y64CTCKWmdcBVwEnl4q4yd/8KOJlQMloL3AGc6e5Lo11GAyuiKrJxwI+i9d2BWcAm4CXgDnefXZNYpOpM7TJSF5nZQ8BSd4+9RCJS36lEIHWCmR1mZt8wswZR98phhLpmEakhjSyWumJf4C+EhtsS4AJ3X5TdkETqB1UNiYjkOVUNiYjkuTpXNdSuXTvv2rVrtsMQEalTFixYsNbd2yfbVucSQdeuXZk/f362wxARqVPMrPyI8p1UNSQikueUCERE8pwSgYhInqtzbQQiUvu2bdtGSUkJX375ZcU7S1Y1bdqUgoICGjduXOn3KBGISIVKSkpo1aoVXbt2JfV9hSTb3J1169ZRUlJCt27dKv2+vKgamjoVunaFBg3C81TdxlukSr788kvatm2rJJDjzIy2bdtWueRW70sEU6fCeefB5uiWJitXhmWA4uLsxSVS1ygJ1A3V+T3V+xLBhAllSaDU5s1hvYiI5EEieP/9qq0Xkdyzbt06CgsLKSwsZN9996Vjx447l7/66qu0750/fz6XXHJJhZ9x5JFHZiTW2bNnc9JJJ2XkWLWl3ieCzuVv8lfBehGpuUy3y7Vt25bFixezePFixo0bx2WXXbZzuUmTJmzfvj3le4uKirj11lsr/IwXX3yxZkHWYfU+EUyaBM2b77quefOwXkQyr7RdbuVKcC9rl8t0J40xY8Zw+eWXc/TRR3P11VfzyiuvcOSRR9KvXz+OPPJI3nrrLWDXb+gTJ05k7NixDB06lAMOOGCXBNGyZcud+w8dOpQRI0bQs2dPiouLKZ2leebMmfTs2ZPBgwdzySWXVPjN/9NPP+WUU06hT58+DBw4kNdeew2Af/7znztLNP369WPjxo18+OGHDBkyhMLCQg455BDmzp2b2ROWRr1vLC5tEJ4wIVQHde4ckoAaikXika5dLtP/d8uWLWPWrFk0bNiQzz//nDlz5tCoUSNmzZrFtddey6OPPrrbe5YuXcpzzz3Hxo0bOeigg7jgggt263O/aNEilixZwv7778+gQYN44YUXKCoq4vzzz2fOnDl069aNUaNGVRjf9ddfT79+/Zg+fTrPPvssZ555JosXL2by5MncfvvtDBo0iE2bNtG0aVOmTJnC9773PSZMmMCOHTvYXP4kxqjeJwIIf3y68IvUjtpslzv99NNp2LAhABs2bOCss87i7bffxszYtm1b0veceOKJ7LHHHuyxxx506NCBjz/+mIKCgl32GTBgwM51hYWFrFixgpYtW3LAAQfs7J8/atQopkyZkja+559/fmcy+va3v826devYsGEDgwYN4vLLL6e4uJjhw4dTUFDAYYcdxtixY9m2bRunnHIKhYWFNTo3VVHvq4ZEpHbVZrtcixYtdr7++c9/ztFHH83rr7/O448/nrIv/R577LHzdcOGDZO2LyTbpzo38Ur2HjNj/Pjx3H333WzZsoWBAweydOlShgwZwpw5c+jYsSOjR4/mj3/8Y5U/r7qUCEQko7LVLrdhwwY6duwIwH333Zfx4/fs2ZN3332XFStWAPDQQw9V+J4hQ4YwNWocmT17Nu3ataN169a888479O7dm6uvvpqioiKWLl3KypUr6dChA+eeey7nnHMOCxcuzPjPkIoSgYhkVHExTJkCXbqAWXieMiX+6tmrrrqKa665hkGDBrFjx46MH79Zs2bccccdHHfccQwePJh99tmHPffcM+17Jk6cyPz58+nTpw/jx4/n/vvvB+CWW27hkEMOoW/fvjRr1ozjjz+e2bNn72w8fvTRR/npT3+a8Z8hlTp3z+KioiLXjWlEatebb77JN7/5zWyHkXWbNm2iZcuWuDs/+clP6N69O5dddlm2w9pNst+XmS1w96Jk+6tEICJSSXfddReFhYUcfPDBbNiwgfPPPz/bIWVErL2GzOw44LdAQ+Bud/9Fkn2GArcAjYG17v6tOGMSEamuyy67LCdLADUVWyIws4bA7cCxQAkwz8xmuPsbCfu0Ae4AjnP3982sQ1zxiIhIcnFWDQ0Alrv7u+7+FTANGFZunx8Cf3H39wHc/ZMY4xERkSTiTAQdgVUJyyXRukQ9gL3MbLaZLTCzM5MdyMzOM7P5ZjZ/zZo1MYUrIpKf4kwEySbFLt9FqRFwKHAi8D3g52bWY7c3uU9x9yJ3L2rfvn21A0ox0FBEJK/FmQhKgE4JywXA6iT7POXuX7j7WmAO0DeOYJ54Arp1g9XlIxCRnDd06FCefvrpXdbdcsstXHjhhWnfU9rV/IQTTmD9+vW77TNx4kQmT56c9rOnT5/OG2/sbNrkuuuuY9asWVUJP6lcmq46zkQwD+huZt3MrAkwEphRbp+/AkeZWSMzaw4cDrwZRzC9esGaNXDttXEcXUTiNGrUKKZNm7bLumnTplVq4jcIs4a2adOmWp9dPhHceOONHHPMMdU6Vq6KLRG4+3bgIuBpwsX9YXdfYmbjzGxctM+bwFPAa8ArhC6mr8cRzwEHwGWXwf33w7x5cXyCiMRlxIgRPPHEE2zduhWAFStWsHr1agYPHswFF1xAUVERBx98MNdff33S93ft2pW1a9cCMGnSJA466CCOOeaYnVNVQxgjcNhhh9G3b19OO+00Nm/ezIsvvsiMGTO48sorKSws5J133mHMmDE88sgjADzzzDP069eP3r17M3bs2J3xde3aleuvv57+/fvTu3dvli5dmvbny/Z01bGOI3D3mcDMcuvuLLd8M3BznHGUuvZauO8+uPRSeP75MPxdRKrm0kth8eLMHrOwEG65JfX2tm3bMmDAAJ566imGDRvGtGnTOOOMMzAzJk2axN57782OHTv4zne+w2uvvUafPn2SHmfBggVMmzaNRYsWsX37dvr378+hhx4KwPDhwzn33HMB+NnPfsY999zDxRdfzMknn8xJJ53EiBEjdjnWl19+yZgxY3jmmWfo0aMHZ555Jr///e+59NJLAWjXrh0LFy7kjjvuYPLkydx9990pf75sT1edVyOLW7cOE1+9+CJUYr4oEckhidVDidVCDz/8MP3796dfv34sWbJkl2qc8ubOncupp55K8+bNad26NSeffPLOba+//jpHHXUUvXv3ZurUqSxZsiRtPG+99RbdunWjR4/Qv+Wss85izpw5O7cPHz4cgEMPPXTnRHWpPP/884wePRpIPl31rbfeyvr162nUqBGHHXYY9957LxMnTuTf//43rVq1SnvsysiL+xEkGjMGbr8drroKTj5591kSRSS9dN/c43TKKadw+eWXs3DhQrZs2UL//v157733mDx5MvPmzWOvvfZizJgxKaefLmUpqgLGjBnD9OnT6du3L/fddx+zZ89Oe5yK5mkrnco61VTXFR2rdLrqE088kZkzZzJw4EBmzZq1c7rqv/3tb4wePZorr7ySM89M2vO+0vKqRADQsCH89rewahVU0FlARHJIy5YtGTp0KGPHjt1ZGvj8889p0aIFe+65Jx9//DFPPvlk2mMMGTKExx57jC1btrBx40Yef/zxnds2btzIfvvtx7Zt23ZOHQ3QqlUrNm7cuNuxevbsyYoVK1i+fDkAf/rTn/jWt6o3Q062p6vOu0QAcNRRcPrp8MtfQklJ5m+0LSLxGDVqFK+++iojR44EoG/fvvTr14+DDz6YsWPHMmjQoLTv79+/P2eccQaFhYWcdtppHHXUUTu33XTTTRx++OEce+yx9OzZc+f6kSNHcvPNN9OvXz/eeeedneubNm3Kvffey+mnn07v3r1p0KAB48aNq9bPle3pqvN2GuoVK6BnTygqgkWLdr3HavPmtTN/ukhdoWmo6xZNQ11JXbvCFVfACy+kvtG2iEg+yNtEAHDNNam3xXGjbRGRXJTXiaBlS2jbNvm2OG60LVKX1bVq5HxVnd9TXicCgN/8JjQSJ6qNG22L1CVNmzZl3bp1SgY5zt1Zt24dTZs2rdL78m4cQXmjR8Py5XDjjWG5S5eQBNRQLFKmoKCAkpISNA187mvatCkFBQVVek/eJwKAG26AZctg+nR4+mk46KBsRySSWxo3bky3bt2yHYbEJO+rhkr9+tfQrBmMHQs7dmQ7GhGR2qNEENlvP7j11jAP0W9/m+1oRERqjxJBguLiMP/QhAmQMDutiEi9pkSQwAzuvDNUEZ19tqqIRCQ/KBGUU1pF9NJLqiISkfygRJCEqohEJJ8oESShKiIRySdKBCkkVhGdeaamqRaR+kuJII3iYujfHx58EFauBPfwfN55SgYiUn8oEaRhBh9/vPt6TVMtIvWJEkEFVq9Ovl7TVItIfaFEUIFU01FrmmoRqS+UCCowaVKYljpRs2aaplpE6o9YE4GZHWdmb5nZcjMbn2T7UDPbYGaLo8d1ccZTHcXF4f7FXbqUrRs4UNNUi0j9EVsiMLOGwO3A8UAvYJSZ9Uqy61x3L4weN8YVT00UF4eb3bvD1VfDc8/BI49kOyoRkcyIs0QwAFju7u+6+1fANGBYjJ9XK266CQ47DM49Vw3GIlI/xJkIOgKrEpZLonXlHWFmr5rZk2Z2cLIDmdl5ZjbfzOZn+w5JjRvDn/8M27fDj36kUcciUvfFmQgsybryNzxdCHRx977A74DpyQ7k7lPcvcjdi9q3b5/hMKvuG9+AO+6AuXPVaCwidV+ciaAE6JSwXADs0ivf3T93903R65lAYzNrF2NMGTN6dGg7uOEGeOGFbEcjIlJ9cSaCeUB3M+tmZk2AkcCMxB3MbF8zs+j1gCiedTHGlFF33BHmHvrhD+Gzz7IdjYhI9cSWCNx9O3AR8DTwJvCwuy8xs3FmNi7abQTwupm9CtwKjHT38tVHOat16zAP0QcfQEFBmJJCk9KJSF3TKM6DR9U9M8utuzPh9W3AbXHGELfly8OspJs3h+XSSelAYw1EpG7QyOIamjABtm3bdZ0mpRORukSJoIZSjSXQGAMRqSuUCGoo1eRz++5bu3GIiFSXEkENJZuUDmDrVli1avf1IiK5RomghhInpTMLz5MmhZHHxx0Hn36a7QhFRNKLtddQvigu3r2H0JFHwve+B9//PvzjH8lLDSIiuUAlgpgMHRrGE7z0EowcGUoIIiK5SIkgRiNGwG23weOPw/nnh2msRURyjaqGYnbhhfDRR2H66gMPhGuuyXZEIiK7UomgFtxwQ6ge+tnP4Jlnsh2NiMiulAhqgRncdRf07BkSgrqVikguUSKoBVOnwiGHwBtvwLp1cPTRYZyBiEguUCKI2dSpYRK6lSvDsju88w6cdFJ24xIRKaVEELMJE8pmJk00axY88EDtxyMiUp4SQczSTT533nnw73/XXiwiIskoEcQs1aR0BQXQpg0MHw4bNtRuTCIiiZQIYpZsUrrmzeEXv4CHH4YVK2DMGA02E5HsUSKIWbJJ6aZMCesHD4abb4bp0+FXv8p2pCKSr6wO3SIYgKKiIp8/f362w8gYdxg1Cv7v/0IJ4bTTsh2RiNRHZrbA3YuSbVOJIMvM4O67YeDAMNjsr3/NdkQikm+UCHJAy5bw5JPQvz+cfjr87W/ZjkhE8okSQY5o3Rqefhr69Ak9iZ5+OtsRiUi+UCLIsqlToWtXaNAACgvh3HOhVy845RRNUCcitSPWRGBmx5nZW2a23MzGp9nvMDPbYWYj4own1yROP+Eeni+/PNy7oHv3cHez2bOzHaWI1HexJQIzawjcDhwP9AJGmVmvFPv9Esi7ypBk009s3hzGGMyaBd26wYknwty52YlPRHLD1q3hmrBkSTzHj7NEMABY7u7vuvtXwDRgWJL9LgYeBT6JMZaclGr6ifffhw4dQtVQp05wwgnw4ou1G5uIZNc778Dtt4cJKvfeG449Fn7/+3g+K85E0BFInHm/JFq3k5l1BE4F7owxjpyVavqJ0vX77gvPPgv77Qff/W4YayAi9dPmzTBzJlx8cagaPvBAuOgiePNNOPtseOIJ+OUv4/nsOBOBJVlXfvTaLcDV7r4j7YHMzjOz+WY2f82aNRkLMNtSTT8xaVLZ8v77h3aCPn3gBz+AK66AbdtqNUwRicnGjfDQQ6HbePv2oSr4nnvgoIPgd7+Dt98OJYPbbgvbWrSIJ44471lcAnRKWC4AVpfbpwiYZmYA7YATzGy7u09P3MndpwBTIIwsji3iWlZcHJ4nTAjVQZ07hyRQur5UaTK44gr49a9h3rwwCnnffWs9ZBGpoQ0b4PHH4ZFHQjfxL78M/8tjxoTegkcdBU2b1m5MsU0xYWaNgGXAd4APgHnAD909aXOHmd0HPOHuj6Q7bn2bYqKqSnsatW4dqooGD852RCJSkY8+CgNFH3sM/vEP+Oor6NgRRowI08oceSQ0bBhvDOmmmIitRODu283sIkJvoIbAH9x9iZmNi7bnZbtATRUXlw06O/roMGndT38apqoQkdzgDosXh3r9xx8PpXgIk05efHFIAAMGhPFDuUCTztVRGzaEouT06XDGGWG+opYtsx2VSP7avDlU4T7+eEgAJSXhC9rhh4eeP9//PvTunb0vbVkpEUi89twT/vKXMH31tdeG/sUzZ4bupiISP/dwh8G//z3U9c+dG/r7t2gRevndeGPo+r3PPtmOtGI5UjCRVBKnoOjaNSyXMoOrr4anngqNzUccoVtfisTpk0/gwQfhrLNCJ46+feHKK+HDD+HCC8P/4tq14Uva2WfXjSQAKhHktNKG4dLRxytXhmXYtWfRsceGbyMnnBAajx97DL797dqPV6S+cQ/9+P/6V5gxA15+Oaxr2zb83333u+HRsWPFx8plaiPIYV27hot/eV26hFtclrdqFRx/PCxbBvfdBz/8YcwBitRD27fD88+HC/+MGaEfP0BRUajnP/74MGV83L18Mk1tBHVUuikokunUKfwBn3pqKDGsWgVXXaUeRSLpuMO778Jzz4XHk0/CZ59Bkybwne+Eqp+TTqr73/rTUSLIYZ07Jy8RpJqaAqBNm1BPOWYMjB8fksFvf1v3vr2IxKX0wj97dtmjpCRs22cfOPnk8Pjud/OnJ54SQQ6bNGnXNgLYfQqKZPbYI7QvdOoUxhl88EFo4GrWLN54RXLVF1+EgVwzZoRZPFdFs6B16ABDh4YxOUOHhqkd8rEEXalEYGYtgC3u/rWZ9QB6Ak+6u2a9iVFlp6BIpkGD0LW0U6cw4KxfvzBfyTHHxBuzSK5YvTr05y+9+G/dGkrMxx4L11wTLvw9e+bnhb+8SjUWm9kC4ChgL+BfwHxgs7tX4pKUWfnUWJwp//gHXHBBaPQ64wz4n/+p3/Wdkp/Wrg3jaebODRf/0tG83brBsGGhumfwYGjcOLtxZksmGovN3Teb2TnA79z9V2a2KHMhSpyOPRZefz2UEP77v8OcJzfcEIa65+s/hdRd69eHC/6SJeHvuvT1xx+H7aWjef/rv8LFv1cvfeuvSKUTgZkdARQD51TxvZIDmjaF666DH/0ILrkkzGR6771wxx1htkORXLZsWRik9Ze/lH3Th9CY26tXGENz8MFwyCGhGrRDh+zFWhdVdmTxpcA1wGPRxHEHAM/FF5ZUVrqRx8kccECYC2X69DAX+pAhYZRkaa8JkVzgDq++CtdfH+bnOeigUK9vBjfdFOr+V6wIc269/DL84Q/hy833vqckUB1VHlBmZg2Alu7+eTwhpac2gjLlRx5D6FU0ZUrlGpQ3bw6NzzffHJbPPjtMWXHAAfHEK5LO2rXwr3+F7pzTp4c2rQYNQol1+PAwPkZzaVVfujaCyjYWPwiMA3YAC4A9gV+7+82ZDLQylAjKVHXkcSorV4b2g7vvhh07QhK59trwLUwkDl9/DUuXhntxlz7eeitsa9w4DOQaPjw08uobfmZkIhEsdvdCMysGDgWuBha4e5/MhloxJYIyDRqEInR5ZuEfrapWr4bJk+HOO8Ndk37wg5AQ+tT6b1nqm40b4ZVXwgX/pZfCY/36sK1t23BjltJHUdHut3CVmstEr6HGZtYYOAW4zd23mVndmqSoHqrOyON09t8/3Apz/Hj4zW/CuIOHHgrzq/zoR2GOlVatahaz1H/u8N57Zd/0X3oJXnut7MtJr17hHr2lF/7u3dWrJ9sqmwj+F1gBvArMMbMuQFbaCKRMdUceV6RDh9DN9Morww20b7stNDA3aRIGpJ1ySkgOumeyuIcvIwsXwoIF4bFwIaxZE7a3ahW6cv7sZ+Gif/jhYVCX5JZqzz5qZo3cfXuG46mQqoZ2NXVq9UYeV8X27eGb3fTp4fHee+Eb3BFHhKRw6qlw4IGZ/UzJPe6hd9krr8D8+WUX/XXrwvZGjUIXzv794bDDYNCgsKx5rnJDJtoI9gSuB4ZEq/4J3OjuGzIWZSUpEWSXexjEU5oUFi4M6486Cn7843AvVtXv1g/r14cL/iuvhMfLL4ebsENo0D3kEDj00PDo3z+0JTVtmt2YJbVMJIJHgdeB+6NVo4G+7j48Y1FWkhJBblm5Ev78Z7jnHli+HFq3DiWSH/84XBykbli7FhYtCjdcX7QoJPjSXjwQepANGFD26Ns3TG4odUfGeg1VtK42KBFUTW1UHUEoKcyZE7qgPvJI6HXUrx+ccw6MHBl6hkh2ff11uNViSUnoXvzqq2UX/g8+KNuvU6fwuyu96BcVwV57ZS1syZBMJIKXgCvd/floeRAw2d2PyGiklaBEUHk1HXBWXevXh2mv77orXGggVCMMGhQm/Ro8OIx1UE+RzPr663CRX7o0fJt///2wXFISpl1evRq2JcwX3KABfPObUFgYLvyFheGhpF0/ZSIR9AX+SBhIBvAZcJa7v5axKCtJiaDyMjXgrCYWLoSZM+GFF0KD8+dRX7OOHcsSw4ABYRoBtS1UzhdfwNtvh4v90qVlj7fegi1byvZr2hQKCnZ/dOoUnr/5Td2jIp/UOBEkHKg1gLt/bmaXuvstGYqx0pQIKi/TA85qaseO0ND8/PMhMcydWzbHUYMGoUgYef8AAA+4SURBVB669Ftp6SNfR5V+9VX4Rr9sWbjAL1tW9jqxGscsTLN80EFhbv3E5332UalLymQsEZQ76PvunnbokpkdB/wWaAjc7e6/KLd9GHAT8DWwHbi0tPopFSWCysuFEkFF3n8/lBpK66oXL971nsz77Rcebdokf+y9d7gQ9ugRqjTSXfi++iocv7R08sIL4WYlgwaFXk9DhoQG7jim5t66NVSZffZZeKxfH7pdfvTRro8PPwzPn3666/v32itc3Hv0KHsufainjlRGXIlglbunnALKzBoCy4BjgRJgHjDK3d9I2Kcl8IW7u5n1AR52957pPleJoPKy1UZQU59+GhoyFy0KI1LXrg0XzsTHF1/s/r42bcKFsXv3sucWLcJEZi+8EKYvLq066dIlJICmTUPJ5O23w/rmzWHgwJAUjjoqVGFt3x7q1ss/b9sWZr/89NPUj9ILf2KVTXlNm5YlvH33LXsUFJRd9Nu1y/x5lvySiSkmkqkogwwAlrv7u1EQ04BhwM5E4O6bEvZvUYljShXU5FaX2bT33uEeskcfnXqfbdtCe8OaNWGWyrffDlUnb78dLuwPPlhWLdaoUWgMPf/8smkNyt+h7aOPwvvmzg29n264IXm1WjrNmoXY9947lE569Ajf5EsfbdrsurzXXuHi36qVqnAku9KWCMxsI8kvzgY0c/eUicTMRgDHufuPo+XRwOHuflG5/U4F/hvoAJzo7i8lOdZ5wHkAnTt3PnRlsvoOkQRbtsC774Zv7IWFVW+IXr8+zJHz2WchkTRunPx5zz3DhX+vvdTwKrmt2iUCd6/JFGPJvuPsllTc/THgMTMbQmgv2O326u4+BZgCoWqoBjFJObU1zqC2NWsWpjeorjZtwiR7IvkgzttNlgCJbQgFwOpUO7v7HDP7hpm1c/e1McYlkfJtCCtXhmWoH8lARCqnsreqrI55QHcz62ZmTYCRwIzEHczsQLNQO2pm/YEmwLoYY5IEEybs2pAMYXnChOzEIyLZEVuJwN23m9lFwNOE7qN/iO53PC7afidwGnCmmW0DtgBneHW7MUmVJXbTrMx6Eamfqt19NFvUfTRz6sI4AxHJjHSNxXFWDUmOmzRp9940mbixjYjULUoEeay4OAwuK50ArkuX3B9sJiKZF2evIakDiot14RfJdyoRSEpTp4Z2hAYNwvPUqdmOSETioBKBJKUxBiL5QyUCSUpjDETyhxKBJKUxBiL5Q4lAkuqc4k4TqdaLSN2lRCBJaYyBSP5QIpCkKjPGQL2KROoH9RqSlNKNMVCvIpH6QyUCqRb1KhKpP5QIpFrUq0ik/lAikGpRryKR+kOJQKpFvYpE6g8lAqkW9SoSqT/Ua0iqTb2KROoHlQgkFupVJFJ3KBFILNSrSKTuUCKQWKhXkUjdoUQgsahMryI1JovkBiUCiUVFvYpKG5NXrgT3ssZkJQOR2mfunu0YqqSoqMjnz5+f7TCkhrp2DRf/8rp0gRUrajsakfrPzBa4e1GybSoRSFaoMVkkd8SaCMzsODN7y8yWm9n4JNuLzey16PGimfWNMx7JHWpMFskdsSUCM2sI3A4cD/QCRplZr3K7vQd8y937ADcBU+KKR3KLGpNFckecJYIBwHJ3f9fdvwKmAcMSd3D3F939s2jxX0BBjPFIDlFjskjuiDMRdARWJSyXROtSOQd4MtkGMzvPzOab2fw1a9ZkMETJpuLi0DD89dfhOXHqCY1MFqk9cSYCS7IuaRclMzuakAiuTrbd3ae4e5G7F7Vv3z6DIUquUmOySO2JMxGUAJ0SlguA1eV3MrM+wN3AMHdfF2M8UodU1Jis9gORzIkzEcwDuptZNzNrAowEZiTuYGadgb8Ao919WYyxSB2TrjFZ7QcimRVbInD37cBFwNPAm8DD7r7EzMaZ2bhot+uAtsAdZrbYzDRSTID0jclqPxDJLI0sljqnQYNQEijPLDQ8i8juNLJY6pXKDEZTG4JI5SkRSJ1T0WA0tSGIVI0SgdQ5FQ1GUxuCSNWojUDqHbUhiOxObQSSV9SGIFI1SgRS76gNQaRqlAik3lEbgkjVKBFIvZRuQrvKzGOkqiPJJ0oEkncqM4+Rqo4knygRSN6pqA1BVUeSb5QIJO9U1IagKbAl3ygRSF5K14ag7qeSb5QIRMpR91PJN0oEIuVkovupSgxSl2iKCZEqqmgKi9ISQ2KyaN5812QiUts0xYRIBlXUhqBeR1LXKBGIVFFFbQgV9TpStZHkGiUCkSqqqA0hXYlBDc2Si9RGIJJh6doIJkwIF//yunQJ3VhF4qI2ApFalK7EoHmOJBc1ynYAIvVRcXHyHkKdOycvEZSf56i0NFFadVR6TJE4qEQgUosyMc+RSgySaUoEIrWopvMcqbFZ4qDGYpEc0rVr+sbkiraLpJK1xmIzO87M3jKz5WY2Psn2nmb2kpltNbP/iDMWkbqgpmMUQFVHUnWxJQIzawjcDhwP9AJGmVmvcrt9ClwCTI4rDpG6pCZjFEBVR1I9cZYIBgDL3f1dd/8KmAYMS9zB3T9x93nAthjjEKlT0k2RrcZmiUOciaAjsCphuSRaV2Vmdp6ZzTez+WvWrMlIcCJ1kRqbJQ5xJgJLsq5aLdPuPsXdi9y9qH379jUMS6Ruq8lNdSoqMai0kJ/iTAQlQKeE5QJgdYyfJ5L3atLYrNJC/oozEcwDuptZNzNrAowEZsT4eSJ5ryaNzWpfyF+xjiMwsxOAW4CGwB/cfZKZjQNw9zvNbF9gPtAa+BrYBPRy989THVPjCESqL92EeKNH64Y79VnWxhG4+0x37+Hu33D3SdG6O939zuj1R+5e4O6t3b1N9DplEhCRmklXYsjEDXdUYqibNMWESJ5J1diciRvuVNTGoESRm5QIRASo+WC2yvRIUmN0blIiEJGdajKYraISg6qWcpcSgYhUSk1LDDWtWlKSiI9uTCMilZbqhjsQSgbJehWVlhgquilPRSUG3bAnPioRiEhGVFRiqEnVkqqV4qVEICIZk66NoSZVS+qxFC8lAhGpNdVtjK6NHkv5nCiUCEQkJ6QrMcTdYynvE4W716nHoYce6iKSfx54wL1LF3ez8PzAA2XbunRxD5fwXR9duoTtZsm3m1Xu/Q884N68+a7bmjcviyFdbLkCmO8prqtZv7BX9aFEICLlVXShjjNRVPTZpfFlO1GkSwSqGhKROq+mPZZqMgaiXlQ7pcoQufpQiUBEqiPdt/KalCjirnaqKPbKQlVDIiLpVTdRZLt9orLSJQJVDYmIUP0xEHFWO0HlBtPVlBKBiEglpEoU2WyfyBQlAhGRGqrJiOqaJopMUCIQEYlZnIkiEzT7qIhIlqWb1bV0/YQJoTqoc+eQBDI566oSgYhIjkuXKDJBVUMiInlOiUBEJM8pEYiI5DklAhGRPKdEICKS5yxMQVF3mNkaIMktsHdqB6ytpXCqSrFVj2KrHsVWPfU1ti7u3j7ZhjqXCCpiZvPdvSjbcSSj2KpHsVWPYquefIxNVUMiInlOiUBEJM/Vx0QwJdsBpKHYqkexVY9iq568i63etRGIiEjV1McSgYiIVIESgYhInqs3icDMjjOzt8xsuZmNz3Y8icxshZn928wWm9n8LMfyBzP7xMxeT1i3t5n9w8zejp73yqHYJprZB9G5W2xmJ2Qptk5m9pyZvWlmS8zsp9H6rJ+7NLFl/dyZWVMze8XMXo1iuyFanwvnLVVsWT9vCTE2NLNFZvZEtBzLeasXbQRm1hBYBhwLlADzgFHu/kZWA4uY2QqgyN2zPkjFzIYAm4A/uvsh0bpfAZ+6+y+iJLqXu1+dI7FNBDa5++TajqdcbPsB+7n7QjNrBSwATgHGkOVzlya2H5Dlc2dmBrRw901m1hh4HvgpMJzsn7dUsR1HDvzNAZjZ5UAR0NrdT4rrf7W+lAgGAMvd/V13/wqYBgzLckw5yd3nAJ+WWz0MuD96fT/hIlLrUsSWE9z9Q3dfGL3eCLwJdCQHzl2a2LLOg03RYuPo4eTGeUsVW04wswLgRODuhNWxnLf6kgg6AqsSlkvIkX+EiAN/N7MFZnZetoNJYh93/xDCRQXokOV4yrvIzF6Lqo6yUm2VyMy6Av2Al8mxc1cuNsiBcxdVbywGPgH+4e45c95SxAY5cN6AW4CrgK8T1sVy3upLIrAk63ImswOD3L0/cDzwk6gKRCrn98A3gELgQ+B/shmMmbUEHgUudffPsxlLeUliy4lz5+473L0QKAAGmNkh2YgjmRSxZf28mdlJwCfuvqA2Pq++JIISoFPCcgGwOkux7MbdV0fPnwCPEaqycsnHUT1zaX3zJ1mOZyd3/zj6Z/0auIssnruoHvlRYKq7/yVanRPnLllsuXTuonjWA7MJdfA5cd5KJcaWI+dtEHBy1L44Dfi2mT1ATOetviSCeUB3M+tmZk2AkcCMLMcEgJm1iBrwMLMWwHeB19O/q9bNAM6KXp8F/DWLseyi9I8+cipZOndRw+I9wJvu/uuETVk/d6liy4VzZ2btzaxN9LoZcAywlNw4b0ljy4Xz5u7XuHuBu3clXM+edfcfEdd5c/d68QBOIPQcegeYkO14EuI6AHg1eizJdmzAnwnF3W2EktQ5QFvgGeDt6HnvHIrtT8C/gdeif4L9shTbYEJ142vA4uhxQi6cuzSxZf3cAX2ARVEMrwPXRetz4bylii3r561cnEOBJ+I8b/Wi+6iIiFRffakaEhGRalIiEBHJc0oEIiJ5TolARCTPKRGIiOQ5JQKRiJntSJhxcrFlcBZbM+tqCbOqiuSSRtkOQCSHbPEw3YBIXlGJQKQCFu4n8cto7vpXzOzAaH0XM3smmpzsGTPrHK3fx8wei+a5f9XMjowO1dDM7ormvv97NJoVM7vEzN6IjjMtSz+m5DElApEyzcpVDZ2RsO1zdx8A3EaYFZLo9R/dvQ8wFbg1Wn8r8E937wv0J4woB+gO3O7uBwPrgdOi9eOBftFxxsX1w4mkopHFIhEz2+TuLZOsXwF8293fjSZ3+8jd25rZWsL0A9ui9R+6ezszWwMUuPvWhGN0JUxz3D1avhpo7O7/aWZPEW7IMx2Y7mVz5IvUCpUIRCrHU7xOtU8yWxNe76Csje5E4HbgUGCBmantTmqVEoFI5ZyR8PxS9PpFwsyQAMWEWx1CmAzsAth545PWqQ5qZg2ATu7+HOEmJG2A3UolInHSNw+RMs2iu1WVesrdS7uQ7mFmLxO+PI2K1l0C/MHMrgTWAGdH638KTDGzcwjf/C8gzKqaTEPgATPbk3CDpd94mBtfpNaojUCkAlEbQZG7r812LCJxUNWQiEieU4lARCTPqUQgIpLnlAhERPKcEoGISJ5TIhARyXNKBCIiee7/AX4TMyFQu3iUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history_dict['accuracy']\n",
    "val_acc = history_dict['val_accuracy']\n",
    "loss = history_dict['loss']\n",
    "val_loss = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV5Z3H8c+PIEKQVSkiUYJWRRRZTKm4oXUpitWqwwjS1m3KuNRanbrVabV27Ght1TraUdwXLFotjrtWq9VqW4kKCggKiBJxCVDZt5Df/PGcm9xczl1IcnNvku/79Tqve896fzlJzu8+z3Oe55i7IyIikqpDoQMQEZHipAQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQnJmZs+Y2anNvW0hmdkiMzsiD8d1M/tq9P5WM/tpLts24nMmmtnzjY1TJBNTP4i2zcxWJ82WAhuAzdH8v7v7lJaPqniY2SLg39z9hWY+rgO7u/v85trWzMqBD4Ft3L2mOeIUyaRjoQOQ/HL37RLvM10MzayjLjpSLPT3WBxUxdROmdmhZlZlZpeY2WfA3WbWy8yeNLNqM/tn9L4saZ+XzezfovenmdlfzezX0bYfmtnRjdx2oJm9YmarzOwFM7vFzB5IE3cuMf7CzF6Ljve8me2QtP67ZvaRmS0zs8sznJ/9zewzMytJWnaCmb0TvR9pZn8zsy/N7FMzu9nMOqU51j1m9l9J8xdF+ywxszNSth1rZm+b2UozW2xmVyatfiV6/dLMVpvZqMS5Tdr/ADObbmYrotcDcj03W3mee5vZ3dHP8E8zeyxp3fFmNiP6GRaY2ZhoeYPqPDO7MvF7NrPyqKrtTDP7GPhztPwP0e9hRfQ3snfS/l3M7DfR73NF9DfWxcyeMrPzUn6ed8zs23E/q6SnBNG+7Qj0BgYAkwh/D3dH87sA64CbM+z/dWAesAPwK+BOM7NGbPsg8AawPXAl8N0Mn5lLjKcApwNfAToBPwYws8HA/0bH3yn6vDJiuPvfgTXAN1KO+2D0fjNwQfTzjAIOB87JEDdRDGOieI4EdgdS2z/WAN8DegJjgbOTLmyHRK893X07d/9byrF7A08BN0U/2/XAU2a2fcrPsMW5iZHtPN9PqLLcOzrWDVEMI4H7gIuin+EQYFG68xFjNLAX8M1o/hnCefoK8BaQXCX6a2A/4ADC3/HFQC1wL/CdxEZmNhToDzy9FXEIgLtraicT4R/1iOj9ocBGoHOG7YcB/0yaf5lQRQVwGjA/aV0p4MCOW7Mt4eJTA5QmrX8AeCDHnykuxv9Mmj8HeDZ6/zNgatK6rtE5OCLNsf8LuCt6341w8R6QZtsfAdOS5h34avT+HuC/ovd3AdckbbdH8rYxx70RuCF6Xx5t2zFp/WnAX6P33wXeSNn/b8Bp2c7N1pxnoB/hQtwrZrvbEvFm+vuL5q9M/J6TfrZdM8TQM9qmByGBrQOGxmy3LbCc0K4DIZH8rqX/39rCpBJE+1bt7usTM2ZWama3RUX2lYQqjZ7J1SwpPku8cfe10dvttnLbnYDlScsAFqcLOMcYP0t6vzYppp2Sj+3ua4Bl6T6LUFo40cy2BU4E3nL3j6I49oiqXT6L4vgloTSRTYMYgI9Sfr6vm9lLUdXOCuCsHI+bOPZHKcs+Inx7Tkh3bhrIcp53JvzO/hmz687AghzjjVN3bsysxMyuiaqpVlJfEtkhmjrHfZa7bwAeBr5jZh2ACYQSj2wlJYj2LfUWtv8A9gS+7u7dqa/SSFdt1Bw+BXqbWWnSsp0zbN+UGD9NPnb0mdun29jd5xAusEfTsHoJQlXVXMK31O7ATxoTA6EElexB4HFgZ3fvAdyadNxstxwuIVQJJdsF+CSHuFJlOs+LCb+znjH7LQZ2S3PMNYTSY8KOMdsk/4ynAMcTquF6EEoZiRiWAuszfNa9wERC1d9aT6mOk9woQUiyboRi+5dRffYV+f7A6Bt5JXClmXUys1HAt/IU4yPAsWZ2UNSgfBXZ/wceBH5IuED+ISWOlcBqMxsEnJ1jDA8Dp5nZ4ChBpcbfjfDtfH1Un39K0rpqQtXOrmmO/TSwh5mdYmYdzexkYDDwZI6xpcYRe57d/VNC28DvosbsbcwskUDuBE43s8PNrIOZ9Y/OD8AMYHy0fQXwLznEsIFQyisllNISMdQSquuuN7OdotLGqKi0R5QQaoHfoNJDoylBSLIbgS6Eb2d/B55toc+dSGjoXUao93+IcGGI0+gY3X02cC7hov8p8E+gKstuvye01/zZ3ZcmLf8x4eK9Crg9ijmXGJ6JfoY/A/Oj12TnAFeZ2SpCm8nDSfuuBa4GXrNw99T+KcdeBhxL+Pa/jNBoe2xK3LnKdp6/C2wilKK+ILTB4O5vEBrBbwBWAH+hvlTzU8I3/n8CP6dhiSzOfYQS3CfAnCiOZD8G3gWmE9ocrqXhNe0+YAihTUsaQR3lpOiY2UPAXHfPewlG2i4z+x4wyd0PKnQsrZVKEFJwZvY1M9stqpIYQ6h3fizbfiLpRNV35wCTCx1La6YEIcVgR8ItmKsJ9/Cf7e5vFzQiabXM7JuE9prPyV6NJRmoiklERGKpBCEiIrHa1GB9O+ywg5eXlxc6DBGRVuPNN99c6u594ta1qQRRXl5OZWVlocMQEWk1zCy1930dVTGJiEgsJQgREYmlBCEiIrHy1gZhZncRuv1/4e77xKw34LfAMYRRJU9z97eidWOidSXAHe5+TWPj2LRpE1VVVaxfvz77xtLiOnfuTFlZGdtss02hQxGRFPlspL6H8ICR+9KsP5rwIJDdCQ+T+V/g69FwwrcQHqhSBUw3s8ejkTW3WlVVFd26daO8vJz0z7KRQnB3li1bRlVVFQMHDix0OCKSIm9VTO7+CmEArXSOB+7z4O+Eseb7ASMJD5dZ6O4bganRto2yfv16tt9+eyWHImRmbL/99irdiTTSlClQXg4dOoTXKVOy7bF1CtkG0Z+GD06pipalWx7LzCaZWaWZVVZXV6fbpunRSl7odyNtXbaLeKb12dZNmgQffQTu4XXSpOZNEoVMEHFXBs+wPJa7T3b3Cnev6NMntq+HiEijNfUCn+kinml9tn0vvxzWrm0Yy9q1YXmzyefzTAlPgJqVZt1twISk+XmEZ92OAp5LWn4ZcFkun7fffvt5qjlz5myxrKUsXbrUhw4d6kOHDvW+ffv6TjvtVDe/YcOGjPtOnz7dzzvvvKyfMWrUqOYKt2AK+TsSeeAB9wED3M3C6wMPNFxXWuoeLtFhKi2t3ybb+gEDGq5LTAMGZF+fbV+z+PVmW/fzA5We7hqebkVzTFkSxFjCU6kM2J/oYeuEhvOFwECgEzAT2DuXz2uOBJHpj6UprrjiCr/uuusaLNu0aVPzHLyVU4KQpsr2f5tufT4v8O7ZL+KZ1mfbN9tn56ogCYLwJK5PCU+dqgLOJDyA/axovRHuVlpAeCpURdK+xwDvR+suz/Uzm5ogsv2xNEUiQZx66ql+wQUX+KGHHuoXXnih/+Mf//BRo0b5sGHDfNSoUT537lx3d3/ppZd87NixdfuefvrpPnr0aB84cKD/9re/rTtu165d67YfPXq0n3TSSb7nnnv6Kaec4rW1te7u/tRTT/mee+7pBx54oJ933nl1x0324Ycf+kEHHeTDhw/34cOH+2uvvVa37tprr/V99tnH9913X7/kkkvc3f2DDz7www8/3Pfdd18fPny4z58/v9HnRglCmqIp3/LzeYF3z28JormuVwUrQbT01NQE0VwZOU5yghg7dqzX1NS4u/uKFSvqShJ/+tOf/MQTT3T3LRPEqFGjfP369V5dXe29e/f2jRs3unvDBNG9e3dfvHixb9682ffff39/9dVXfd26dV5WVuYLFy50d/fx48fHJog1a9b4unXr3N39/fff98S5fPrpp33UqFG+Zs0ad3dftmyZu7uPHDnS//jHP7q7+7p16+rWN4YShDS2BODetItwPi/wibgbm7xySQDNUeORKUGoJ3WSjz/euuWNNW7cOEpKSgBYsWIF48aNY5999uGCCy5g9uzZsfuMHTuWbbfdlh122IGvfOUrfP7551tsM3LkSMrKyujQoQPDhg1j0aJFzJ07l1133bWun8GECRNij79p0ya+//3vM2TIEMaNG8ecOaHbyQsvvMDpp59OaWkpAL1792bVqlV88sknnHDCCUDo7JZYLxInXw25kP3/NtP6XXaJX5dYfvXVkPqnXVoalueyfuJEmDwZBgwAs/A6eXJYnm19tn0T+y9aBLW14TV5XXNQgkiS7Y+luXTt2rXu/U9/+lMOO+wwZs2axRNPPJG2T8C2225b976kpISampqctglfELK74YYb6Nu3LzNnzqSyspKNGzcCoYSZeitqrscUgabfjZNtfbb/20zr83mBT8h2Ec+0Pt8JIBsliCTZ/ljyYcWKFfTvH7p53HPPPc1+/EGDBrFw4UIWLVoEwEMPPZQ2jn79+tGhQwfuv/9+Nm/eDMBRRx3FXXfdxdroP3T58uV0796dsrIyHnssPDZ6w4YNdeul/UpXSsh2gW9KCQCa9i0/3xf41k4JIkkufyzN7eKLL+ayyy7jwAMPrLsoN6cuXbrwu9/9jjFjxnDQQQfRt29fevToscV255xzDvfeey/7778/77//fl0pZ8yYMRx33HFUVFQwbNgwfv3rXwNw//33c9NNN7HvvvtywAEH8NlnnzV77FJcGltNlO0C35QSADT9W35bvsA3WbrGidY4FVs/iGKxatUqd3evra31s88+26+//voCR9SQfkfFoSn9AfJ5N04+7y6UzI3UBb+oN+ekBBHv+uuv96FDh/pee+3lp5xySpPuOMoH/Y5aRr4SgHvmu4Ga426cfPVPEiWIxp85aRH6HeVfPhNALvvrAl+8MiUItUGItANNbShu6u2gqudvnZQgRNqITI3ILZEAWvoGD8k/JQiRVqIpnc1aIgGolNAGpat7ao2T2iBaJ/2OsmtqG0JLDdsgrQ9qgyicQw89lOeee67BshtvvJFzzjkn4z6VlZUAHHPMMXz55ZdbbHPllVfW9UlI57HHHqsbMgPgZz/7GS+88MLWhC9FoqltCCoBSGMoQeTZhAkTmDp1aoNlU6dOTTsmUqqnn36anj17NuqzUxPEVVddxRFHHNGoY0nLSFeN1NQ2BFACkK2nBJFn//Iv/8KTTz7Jhg0bAFi0aBFLlizhoIMO4uyzz6aiooK9996bK664Inb/8vJyli5dCsDVV1/NnnvuyRFHHMG8efPqtrn99tv52te+xtChQznppJNYu3Ytr7/+Oo8//jgXXXQRw4YNY8GCBZx22mk88sgjALz44osMHz6cIUOGcMYZZ9TFV15ezhVXXMGIESMYMmQIc+fO3SKmRYsWcfDBBzNixAhGjBjB66+/XrfuV7/6FUOGDGHo0KFceumlAMyfP58jjjiCoUOHMmLECBYsWNAMZ7btydSO0NQ2BJFGSVf31BqnbG0Q55/vPnp0807nn5+lgs/djznmGH/sscfc3f2///u//cc//rG71w+dXVNT46NHj/aZM2e6u/vo0aN9+vTp7u4+YMAAr66u9srKSt9nn318zZo1vmLFCt9tt93qHkC0dOnSus+6/PLL/aabbnJ391NPPdX/8Ic/1K1LzCeGAJ83b567u3/3u9/1G264oe7zEvvfcsstfuaZZ27x8zT30ODtqQ2iscNWqw1B8gW1QRRWcjVTcvXSww8/zIgRIxg+fDizZ89uUB2U6tVXX+WEE06gtLSU7t27c9xxx9WtmzVrFgcffDBDhgxhypQpaYcMT5g3bx4DBw5kjz32AODUU0/llVdeqVt/4oknArDffvvVDfKXTEODp9eUO40yVSOpDUEKoWOhA2hJN95YmM/99re/zYUXXshbb73FunXrGDFiBB9++CG//vWvmT59Or169eK0005LO9R3Quqw2wmnnXYajz32GEOHDuWee+7h5Zdfznic8KUhvcSw4emGFU8eGry2tpbOnTvXHbc9Dw2eSACJxuREAoBwsc7U0DxxYqgu+uijLY+bPCidLvrSklSCaAHbbbcdhx56KGeccUZd6WHlypV07dqVHj168Pnnn/PMM89kPMYhhxzCtGnTWLduHatWreKJJ56oW7dq1Sr69evHpk2bmJL0lbVbt26sWrVqi2MNGjSIRYsWMX/+fCCMzDp69Oicfx4NDR6vqXcaqR1Bio0SRAuZMGECM2fOZPz48QAMHTqU4cOHs/fee3PGGWdw4IEHZtx/xIgRnHzyyQwbNoyTTjqJgw8+uG7dL37xC77+9a9z5JFHMmjQoLrl48eP57rrrmP48OENGoY7d+7M3Xffzbhx4xgyZAgdOnTgrLPOyvlnac9Dg+ezt7J6I0vRSdc40RwTMAaYB8wHLo1Z3wuYBrwDvAHsk7RuEfAuMIMMjSjJkzrKtU6t5XfUEp3VRFpaputr3koQZlYC3AIcDQwGJpjZ4JTNfgLMcPd9ge8Bv01Zf5i7D3P3inzFKZIsUwkhWxWSxiuStiafVUwjgfnuvtDdNwJTgeNTthkMvAjg7nOBcjPrm8eYRNJqyl1GoDuNpO3JZ4LoDyxOmq+KliWbCZwIYGYjgQFAWbTOgefN7E0zm5TuQ8xskplVmllldXV17Dbeju6kaW2K6XeTrYSg3srS3uQzQcTdk5l6NbgG6GVmM4DzgLeBxH2VB7r7CEIV1blmdkjch7j7ZHevcPeKPn36bLG+c+fOLFu2rKguRBK4O8uWLau7TbalNHY4C91lJO1NPvtBVAE7J82XAUuSN3D3lcDpABZuoP8wmnD3JdHrF2Y2jVBl9QpbqaysjKqqKtKVLqSwOnfuTFlZWfYNm0mmvgq59EOAUKL4+OOw/OqrVUqQtsvy9c3azDoC7wOHA58A04FT3H120jY9gbXuvtHMvg8c7O7fM7OuQAd3XxW9/xNwlbs/m+kzKyoqPDEKqkic8vL4JDBgQLjYJycPCCUENSRLW2Zmb6a7EShvVUzuXgP8AHgOeA942N1nm9lZZpa46X4vYLaZzSVUJZ0fLe8L/NXMZhJuf30qW3IQSWhsXwXdZSTSUN5KEIWgEoSkViFBw1JAphJEzLBTIm1eQUoQIoXQ1L4KIlJPCUJanaYMd6FqJJHctavRXKX1yzZiarY7kRLbKSGIZKcShLQqqkISaTlKEFJ0VIUkUhxUxSRFRVVIIsVDJQgpKqpCEikeShBSVFSFJFI8VMUkRUVVSCLFQyUIaXGZGqFVhSRSPJQgpEVleyiPqpBEiofGYpIWpbGQRIqLxmKSopGtEVpEiocShORFunaGXB7bKSLFQXcxSbPL1Nkt3UN51AgtUnyUIKTZZerslmhn0GM7RYqfGqml2XXoEO5QSmUGtbUtH4+IpKdGamlRamcQaRuUIKRR1NlNpO3La4IwszFmNs/M5pvZpTHre5nZNDN7x8zeMLN9ct1XCked3UTah7y1QZhZCfA+cCRQBUwHJrj7nKRtrgNWu/vPzWwQcIu7H57LvnHUBtEy1NlNpO0oVBvESGC+uy90943AVOD4lG0GAy8CuPtcoNzM+ua4rxSIOruJtA/5TBD9gcVJ81XRsmQzgRMBzGwkMAAoy3Ffov0mmVmlmVVWV1c3U+iSiRqhRdqHfCYIi1mWWp91DdDLzGYA5wFvAzU57hsWuk929wp3r+jTp09T4pUcqRFapH3IZ0e5KmDnpPkyYEnyBu6+EjgdwMwM+DCaSrPtK4WTaGxWZzeRti2fJYjpwO5mNtDMOgHjgceTNzCzntE6gH8DXomSRtZ9Jb8y3cYKIRksWhQ6vi1apOQg0hblrQTh7jVm9gPgOaAEuMvdZ5vZWdH6W4G9gPvMbDMwBzgz0775ilUayjSWkhKBSPuhoTZkC7qNVaT90FAbslV0G6uIgBKExNBtrCICShASQ7exiggoQbRbme5S0lhKIgJ6YFC7lMtdShMnKiGItHcqQbRDmZ74Jq3T5s3w5Zch2S9YAGvWFDoiaQtUgmiHdJdSy3KH5cvhk0+gqiq8rloFGzfCpk3xrzU14aJfU1M/JeY3bgz7r1hRP61eveXndusGO+3UcOrXD7p3T/+5GzdCx47QpUvDqbS0/v1224VjJ0+dOm35+dL6KUG0Q7vsEt/Pob3cpfTllzB3LnTtGi6avXuHtpZ0amvDhX3uXHjvvfBaVQUlJeFimnhNTCUloURWVVWfENavT398s3CB3Wab+tfU4yXPd+wYLvL9+kGPHmHq3r3+fUkJfP45LFkSpk8/hddfD+83bEgfR0lJ+OzNm0PC2BqdOoVEsd12YeraNX7q0iV8jllo/0qeEo+k3bAhTOvXb/l+++1hyBDYZ58w7bhj5t+dNI0SRDt09dUN2yCgbd6lVFMDH3wA77zTcEotKXXqVP/tOvFNu0cPWLgwJIN58xpW2fTsGRru3Rt+s0/+xt+lC5SVwde+BiecEN7371//2qNHfTIoKWmZ8+EekuOqVQ0TUuJ9h6QK55oaWLcuTGvXNny/enWYVq1qOCWWrVkTptWrQ3JMzK9ZE45RWxsm9/hnlJeUQOfOsO22YUp+//e/w91312+7/fb1yWLw4HC8pUvrp+rq+vfLl4dkki6pl5aGGzZ2263hNHBg+H2mnsvVq0PpbeXK8Nq5c/jb6dOn4blszdSTup2aMqW4B9vbuBE+/LD+G3Di23Dyt+JVq7b8Bpr8vqqq/htzx44waBDsu2+YBg8O30pTj5uYVq4MSWDQoDDttVf9a58++tbanBKJInEp6pjla+sXX8Ds2TBrVsNp5cr6bXr3hh12CFOfPuG1V6/we0utsktMq1eHLwULFoS/rWT9+4djJhLCypXxyS0Rf9++W1bt9epVXyWXWk1XWhqSZ2rSTSTe2lrYY4/wd1tW1rx/f5l6UitBSEG5h4t96rf8994L/7TJSkvDP2rim3737vUXl7ipf//6hDBoUPgGmquamuwXKike7iGxb7NNuJA35XfnHkocCxbUJ4wFC0LpK7U6L/G+e/fwZSTui8ySJbBsWfP9rN26hS8qe+8dEkZiStyWvrWUINqhYi0huMO778KTT8KLL8LMmQ3/ecrK6i/qe+0V5hPfwrp10zd3aZ02bAiljrgquVWrQtVdly7xpYtu3cL/zbx5MGdOKD3NmROmzz4Lx+/VK/wfNXeC0HekNqjYRmNduxb+/Gd46qkwLY6eFThsWKifTySEIUPCtz+RtmbbbUNVV1Oeada3LxxySMNly5aF0vbSpfn58qQSRBtU6NFYa2vDH+1f/gJPPx1KCuvXh7tYjjwSjj0Wjj46lApEpLBUgmhnWrqfw8aN8Oab8Ne/wquvwmuvhTtGAHbdNZRexo6F0aO3rh1ARApLCaINaol+Dl9+CbfdBs8+C//4R7gDA8KdFiecAAcdBAcfHBKE2g1EWicliDYon/0cli2DG2+Em24KjW4jRsC//3tIBgceGOpJRaRtUIJogxIN0c15F9MXX8BvfgO/+124++Kkk+A//zM0NItI26QE0UY112isn34K110Ht94aGprHjw+JZ++9m35sESluWTuEm9mxZtaojuNmNsbM5pnZfDO7NGZ9DzN7wsxmmtlsMzs9ad0iM3vXzGaYmW5NipHpmQ5N9dln8MMfhmEGbroJxo0LdyY9+KCSg0h7kUsJYjzwWzN7FLjb3d/L5cBmVgLcAhwJVAHTzexxd5+TtNm5wBx3/5aZ9QHmmdkUd98YrT/M3Zfm/NO0I/nq67BsGfzqV/A//xPuTjrtNPjJT0Jjs4i0L1lLBu7+HWA4sAC428z+ZmaTzKxbll1HAvPdfWF0wZ8KHJ96eKCbmRmwHbAcSBlgQeI09zMdVqyAK68MJYbrrgttDHPnwh13KDmItFc5VR25+0rgUcJFvh9wAvCWmZ2XYbf+wOKk+apoWbKbgb2AJcC7wPnunhgCy4HnzexNM5uU7kOiZFVpZpXV1dW5/DhtQnP1dVizBq69NiSBn/8cjjoqDIVx//3w1a82PU4Rab1yaYP4lplNA/4MbAOMdPejgaHAjzPtGrMstdv2N4EZwE7AMOBmM+serTvQ3UcARwPnmllKJ/PogO6T3b3C3Sv6NKUfeyuTrk9Drn0d3MOwybvtBpdeCvvvD5WV8MgjamMQkSCXEsQ44AZ339fdr3P3LwDcfS1wRob9qoCdk+bLCCWFZKcDf/RgPvAhMCg6/pLo9QtgGqHKSiJXXx36NiTLta/DokXwzW/CGWeEUsJf/xrGSNpvv7yEKiKtVC4J4grgjcSMmXUxs3IAd38xw37Tgd3NbKCZdSI0dj+ess3HwOHRcfsCewILzaxroo3DzLoCRwGzcvmB2ouJE2Hy5PohfgcMCPOZGqhra+Hmm8PDVf72N7jlFnjlldDBTUQkVS53Mf0BOCBpfnO07GuZdnL3GjP7AfAcUALc5e6zzeysaP2twC+Ae8zsXUKV1CXuvtTMdgWmhbZrOgIPuvuzW/ejtX1b09fh/ffhzDNDaeGb3wzDZAwYkN/4RKR1yyVBdEy67RR33xiVCLJy96eBp1OW3Zr0fgmhdJC630JCG4c0UU0N3HAD/Oxn4ZGId98Np56q8ZFEJLtcqpiqzey4xIyZHQ+ob0Ir8PbbcMABcPHFodQwZ07o16DkICK5yKUEcRYwxcxuJlQDLQa+l9eopEmWL4ef/jQMj7H99jB1KvzrvyoxiMjWyaWj3AJ33x8YDAx29wOiO44kz7Z2KI3a2tCxbc89Q3I499zQ9nDyyUoOIrL1chqsz8zGAnsDnaOGY9z9qjzG1e5t7VAalZUhIbzxRngWw803w1C14ohIE+TSUe5W4GTgPEIV0zhA97/kWa5DaSxdGhLHyJGhF/X994dbV5UcRKSpcmmkPsDdvwf8091/DoyiYQc4yYNchtL4+99h0CC46y740Y9g3jz4zndUnSQizSOXKqb10etaM9sJWAYMzF9IAtkfG/ryy/Ctb4UnuL38cuj8JiLSnHIpQTxhZj2B64C3gEXA7/MZlGQeSuPZZ+Hoo0OyePVVJQcRyY+MJYjoQUEvuvuXwKNm9iTQ2d1XtEh07Vi6x4aWlsJxxxS7f5MAABLOSURBVIWk8Nxz0I7GJxSRFpaxBBENvf2bpPkNSg4tZ+LEMLBebW14NQtPdttvP/jzn5UcRCS/cqliet7MTjJT02ch3XFHaIA++GB4/nno2bPQEYlIW5dLI/WFQFegxszWE251dXfvnnk3aS433QTnnx/aHR59FLp0KXREItIeZE0Q7p7t0aKSR9deGx7oc+KJ8OCDsO22hY5IRNqLXDrKHRI3tURwbV22oTSmTQvJYcIEeOghJQcRaVm5VDFdlPS+M+HJbm8C38hLRO1EtqE0FiwII69+7WthiO6OOQ2KIiLSfMw99THRWXYw2xn4lbtPyE9IjVdRUeGVlZWFDiMn5eXxHeEGDID33gvDdH/0Ebz1VthWRCQfzOxNd6+IW9eY76VVgLpmNVGmoTTOPx9mzIAnn1RyEJHCyZogzOx/gEQxowMwDJiZz6Dag3RDafTuDbffDpddBmPHtnxcIiIJuZQgkutsaoDfu/treYqn3bj66oZtEBAeCbpqFRx6KFylwdRFpMBy6Sj3CPCAu9/r7lOAv5tZabadAMxsjJnNM7P5ZnZpzPoeZvaEmc00s9lmdnqu+7Z2EyfC5MmhzcEMdt45dH7r3Rt+/3s1SotI4eWSIF4EkrtmdQFeyLaTmZUAtwBHE55GN8HMBqdsdi4wx92HAocCvzGzTjnu2+olhtLYvBlGjYIvvgiPB91xx0JHJiKSW4Lo7O6rEzPR+1xKECOB+e6+0N03AlOB41O2caBbNIzHdsByQjVWLvu2GbfcAg8/HKqdRo8udDQiIkEuCWKNmY1IzJjZfsC6HPbrDyxOmq+KliW7GdgLWAK8C5wfDRCYy76JeCaZWaWZVVZXV+cQVnH5xz/gwgvh2GPh4osLHY2ISL1carp/BPzBzJZE8/0IjyDNJm5wv9ROF98EZhA63e0G/MnMXs1x37DQfTIwGUI/iBziKhrvvReG7u7fH+69N/SoFhEpFrmMxTTdzAYBexIu3HPdfVMOx66i4aNJywglhWSnA9d46K0338w+BAbluG+rNm8efOMboYH6mWdC47SISDHJZSymc4Gu7j7L3d8FtjOzc3I49nRgdzMbaGadgPHA4ynbfAwcHn1OX0ISWpjjvq3WBx/AYYeF5zy89FJ4rrSISLHJpVLj+9ET5QBw938C38+2k7vXAD8AngPeAx5299lmdpaZnRVt9gvgADN7l3C31CXuvjTdvlvzgxWrBQtCcti0KTz0Z6+9Ch2RiEi8XNogOpiZRdVAidtXO+VycHd/Gng6ZdmtSe+XAEflum9rt3BhSA7r14eSw957FzoiEZH0cilBPAc8bGaHm9k3gN8Dz+Q3rLYheTjvsjIYORLWrIEXX4QhQwodnYhIZrmUIC4BJgFnExqp3ybcySQZpA7n/ckn4fUXv4ChQwsXl4hIrrKWIKJ+CX8nNB5XEBqV38tzXK3e5Zc3HGcp4Y47Wj4WEZHGSFuCMLM9CHcPTQCWAQ8BuPthLRNa65ZpOG8RkdYgUwliLqG08C13P8jd/wfY3DJhtX677LJ1y0VEik2mBHES8BnwkpndbmaHE9/DWWJcdVXoBJestDSMtyQi0hqkTRDuPs3dTyb0bH4ZuADoa2b/a2axt6ZKvbVrwR369AmJYsCAMLz3xImFjkxEJDdb9UxqM+sNjANOdvdv5C2qRiqWZ1KvWAG77w6DB4f+DqklCRGRYpHpmdRbNTycuy9399uKMTkUk1/+EpYuheuvV3IQkdZL44c2sw8/hBtvhO99D0aMyL69iEixUoJoZpdeCiUlaowWkdZPCaIZvf56eDLcxReHZzyIiLRmShDNpLYWLrgA+vWDiy4qdDQiIk2Xy1hMkoOHHoI33oC774auXQsdjYhI06kE0QzWrYNLLoHhw0PjtIhIW6AE0QSJ4bxLS2HxYjj6aD1XWkTaDl3OGikxnPdHH9Uvu/HGsFxEpC1QgmikuOG8164Ny0VE2gIliEbScN4i0tblNUGY2Rgzm2dm883s0pj1F5nZjGiaZWabo/GeMLNFZvZutK7wAyyl0HDeItLW5S1BmFkJcAtwNDAYmGBmg5O3cffr3H2Yuw8DLgP+4u7LkzY5LFofO5BUIcVVJWk4bxFpS/JZghgJzHf3he6+EZgKHJ9h+wnA7/MYT7NaujS87rijhvMWkbYpnx3l+gOLk+argK/HbWhmpcAY4AdJix143swcuM3dJ6fZdxIwCWCXFqrfWbsWbrgBxoyBZ55pkY8UEWlx+SxBxA10ne7hE98CXkupXjrQ3UcQqqjONbND4nZ098nuXuHuFX369GlaxDm64w6oroaf/KRFPk5EpCDymSCqgJ2T5suAJWm2HU9K9ZK7L4levwCmEaqsCm7jRrjuOjj44DCJiLRV+UwQ04HdzWygmXUiJIHHUzcysx7AaOD/kpZ1NbNuiffAUcCsPMaas/vvh6oq9XcQkbYvb20Q7l5jZj8AngNKgLvcfbaZnRWtvzXa9ATgeXdfk7R7X2CahcexdQQedPdn8xVrrmpq4JprYL/94Cg9lVtE2ri8jubq7k8DT6csuzVl/h7gnpRlC4Gh+YytMf7wB5g/H/74Rz1KVETaPvWkzlFtbXjW9ODBcHymm3VFRNoIPQ8iR08+CbNmhTYIjdgqIu2BLnU5cA89pHfdFcaPL3Q0IiItQyWIHLz4Ynha3G23QUedMRFpJ1SCyMEvfwk77QSnnlroSEREWo4SRBZvvQUvvQT/8R+w7baFjkZEpOUoQWRxxx3QuTOccUahIxERaVlKEBmsXQsPPgjjxkHPnoWORkSkZSlBZPDoo7BiBTz/fLi1tbxcz5wWkfZD9+Rk8Mtfhh7Tn38e5j/6CCZNCu/13AcRaetUgkjjgw9g7tzQByLZ2rUaqE9E2gcliDTuuiv9uo8/brk4REQKRQkiRk0N3HMPdOkSv76FHlwnIlJQShAxnn4aPvsMzj4bSksbristDcNuiIi0dUoQMe64A/r1g2uvhcmTYcCA0Fg9YECYVwO1iLQHuospxZIloQRx8cVh3KWJE5UQRKR9Ugkixb33wubN6jktIqIEkcQd7rwTDj0UvvrVQkcjIlJYShBJ/vIXWLAAzjyz0JGIiBReXhOEmY0xs3lmNt/MLo1Zf5GZzYimWWa22cx657JvPtx5J/ToASed1BKfJiJS3PKWIMysBLgFOBoYDEwws8HJ27j7de4+zN2HAZcBf3H35bns29y+/BIeeSQ0SKfr/yAi0p7kswQxEpjv7gvdfSMwFTg+w/YTgN83ct8me/BBWL9e1UsiIgn5TBD9gcVJ81XRsi2YWSkwBni0EftOMrNKM6usrq5udLB33gnDh8OIEY0+hIhIm5LPBGExyzxmGcC3gNfcffnW7uvuk929wt0r+vTp04gw4e23w5PjVHoQEamXzwRRBeycNF8GLEmz7Xjqq5e2dt8mu/PO8NS4U07J1yeIiLQ++UwQ04HdzWygmXUiJIHHUzcysx7AaOD/tnbf5rBuHTzwQLhzqVevfHyCiEjrlLehNty9xsx+ADwHlAB3uftsMzsrWn9rtOkJwPPuvibbvvmIc5tt4L77NEKriEgq89Qn4rRiFRUVXllZWegwRERaDTN7090r4tapJ7WIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEiuvCcLMxpjZPDObb2aXptnmUDObYWazzewvScsXmdm70To9aFpEpIV1zNeBzawEuAU4EqgCppvZ4+4+J2mbnsDvgDHu/rGZfSXlMIe5+9J8xSgiIunlswQxEpjv7gvdfSMwFTg+ZZtTgD+6+8cA7v5FHuMREZGtkM8E0R9YnDRfFS1LtgfQy8xeNrM3zex7SesceD5aPindh5jZJDOrNLPK6urqZgteRKS9y2eCsJhlnjLfEdgPGAt8E/ipme0RrTvQ3UcARwPnmtkhcR/i7pPdvcLdK/r06bPVQU6ZAuXl0KFDeJ0yZasPISLSJuWtDYJQYtg5ab4MWBKzzVJ3XwOsMbNXgKHA++6+BEK1k5lNI1RZvdKcAU6ZApMmwdq1Yf6jj8I8wMSJzflJIiKtTz5LENOB3c1soJl1AsYDj6ds83/AwWbW0cxKga8D75lZVzPrBmBmXYGjgFnNHeDll9cnh4S1a8NyEZH2Lm8lCHevMbMfAM8BJcBd7j7bzM6K1t/q7u+Z2bPAO0AtcIe7zzKzXYFpZpaI8UF3f7a5Y/z4461bLiLSnph7arNA61VRUeGVlbl3mSgvD9VKqQYMgEWLmi0sEZGiZWZvuntF3Lp23ZP66quhtLThstLSsFxEpL1r1wli4kSYPDmUGMzC6+TJaqAWEYH83sXUKkycqIQgIhKnXZcgREQkPSUIERGJpQQhIiKxlCBERCSWEoSIiMRqUx3lzKwaiOn6BsAOQLE+W0KxNY5iaxzF1jhtNbYB7h470mmbShCZmFllut6ChabYGkexNY5ia5z2GJuqmEREJJYShIiIxGpPCWJyoQPIQLE1jmJrHMXWOO0utnbTBiEiIlunPZUgRERkKyhBiIhIrDafIMxsjJnNM7P5ZnZpoeNJZWaLzOxdM5thZrk/7Sg/sdxlZl+Y2aykZb3N7E9m9kH02quIYrvSzD6Jzt0MMzumAHHtbGYvmdl7ZjbbzM6Plhf8vGWIrRjOW2cze8PMZkax/TxaXgznLV1sBT9vSTGWmNnbZvZkNJ+X89am2yDMrAR4HzgSqCI8J3uCu88paGBJzGwRUOHuBe+AY2aHAKuB+9x9n2jZr4Dl7n5NlGB7ufslRRLblcBqd/91S8eTFFc/oJ+7vxU9R/1N4NvAaRT4vGWI7V8p/HkzoKu7rzazbYC/AucDJ1L485YutjEU+LwlmNmFQAXQ3d2Pzdf/aVsvQYwE5rv7QnffCEwFji9wTEXL3V8BlqcsPh64N3p/L+EC0+LSxFZw7v6pu78VvV8FvAf0pwjOW4bYCs6D1dHsNtHkFMd5SxdbUTCzMmAscEfS4ryct7aeIPoDi5PmqyiSf5AkDjxvZm+a2aRCBxOjr7t/CuGCA3ylwPGk+oGZvRNVQRWk+ivBzMqB4cA/KLLzlhIbFMF5i6pJZgBfAH9y96I5b2ligyI4b8CNwMVAbdKyvJy3tp4gLGZZ0XwTiBzo7iOAo4Fzo6oUyc3/ArsBw4BPgd8UKhAz2w54FPiRu68sVBxxYmIrivPm7pvdfRhQBow0s30KEUecNLEV/LyZ2bHAF+7+Zkt8XltPEFXAzknzZcCSAsUSy92XRK9fANMI1WLF5POoLjtRp/1FgeOp4+6fR//ItcDtFOjcRfXUjwJT3P2P0eKiOG9xsRXLeUtw9y+Blwl1/EVx3hKSYyuS83YgcFzUdjkV+IaZPUCezltbTxDTgd3NbKCZdQLGA48XOKY6ZtY1ajzEzLoCRwGzMu/V4h4HTo3enwr8XwFjaSDxDxE5gQKcu6hB807gPXe/PmlVwc9butiK5Lz1MbOe0fsuwBHAXIrjvMXGVgznzd0vc/cydy8nXM/+7O7fIV/nzd3b9AQcQ7iTaQFweaHjSYltV2BmNM0udHzA7wlF502E0teZwPbAi8AH0WvvIortfuBd4J3oH6RfAeI6iFBt+Q4wI5qOKYbzliG2Yjhv+wJvRzHMAn4WLS+G85YutoKft5Q4DwWezOd5a9O3uYqISOO19SomERFpJCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghDJwsw2J43gOcOacVRgMyu3pBFqRYpJx0IHINIKrPMw7IJIu6IShEgjWXiWx7XRswPeMLOvRssHmNmL0aBuL5rZLtHyvmY2LXrOwEwzOyA6VImZ3R49e+D5qPcuZvZDM5sTHWdqgX5MaceUIESy65JSxXRy0rqV7j4SuJkwyibR+/vcfV9gCnBTtPwm4C/uPhQYQeg9D7A7cIu77w18CZwULb8UGB4d56x8/XAi6agntUgWZrba3beLWb4I+Ia7L4wGxfvM3bc3s6WEYRg2Rcs/dfcdzKwaKHP3DUnHKCcMJ717NH8JsI27/5eZPUt4SNJjwGNe/4wCkRahEoRI03ia9+m2ibMh6f1m6tsGxwK3APsBb5qZ2gylRSlBiDTNyUmvf4vev04YaRNgIuGRlRAGUTsb6h5I0z3dQc2sA7Czu79EeDhMT2CLUoxIPukbiUh2XaKniyU86+6JW123NbN/EL5sTYiW/RC4y8wuAqqB06Pl5wOTzexMQknhbMIItXFKgAfMrAfhwVc3eHg2gUiLURuESCNFbRAV7r600LGI5IOqmEREJJZKECIiEkslCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFY/w/DutVGSH9EOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # clear figure\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}