13. RNN Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Representing Data for Training RNNs in Keras\n",
    "When we use recurrent networks, the most common form of data is text. In order to represent a sentence, we will need to make a few decisions. \n",
    "- Will our network fundamentally represent characters? Words? What are the building blocks for our structure?\n",
    "- What is the maximum sequence length? \n",
    "- What is my maximum vocabulary size?\n",
    "- What style of RNN should I use?\n",
    "- Can I incorporate an Embedding? What size?\n",
    "- What is my problem type? Many to one? Many to many? Sequence-to-Sequence?\n",
    "\n",
    "A great tutorial that this notebook also uses as base code:\n",
    "- http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/\n",
    "\n",
    "Let's start with the example above and see what the representation in Keras should look like. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ericlarson/opt/anaconda3/envs/mlenv2020/lib/python3.7/site-packages/tensorflow_core/python/keras/datasets/imdb.py:129: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "/Users/ericlarson/opt/anaconda3/envs/mlenv2020/lib/python3.7/site-packages/tensorflow_core/python/keras/datasets/imdb.py:130: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "import os\n",
    "os.environ['KMP_DUPLICATE_LIB_OK']='True'\n",
    "\n",
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 1000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words)\n",
    "\n",
    "# truncate and pad input sequences to be the same length\n",
    "max_review_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=max_review_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=max_review_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'> (25000, 500)\n",
      "<class 'numpy.ndarray'> (500,)\n",
      "Vocabulary size: 999\n",
      "(25000,) 0 1\n"
     ]
    }
   ],
   "source": [
    "print(type(X_train), X_train.shape)\n",
    "print(type(X_train[0]), X_train[0].shape)\n",
    "print('Vocabulary size:', np.max(X_train))\n",
    "print(y_train.shape, np.min(y_train), np.max(y_train))\n",
    "NUM_CLASSES = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IMDB ratings data contains: a number of different reviews and the sentiment of the review as positive or negative. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# show example without the FOR loop\n",
    "from tensorflow.keras.models import Sequential, Model\n",
    "from tensorflow.keras.layers import Dense, Input, SimpleRNN\n",
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "EMBED_SIZE = 50\n",
    "RNN_STATESIZE = 100\n",
    "rnns = []\n",
    "input_holder = Input(shape=(X_train.shape[1], ))\n",
    "shared_embed = Embedding(top_words, # input dimension (max int of OHE)\n",
    "                EMBED_SIZE, # output dimension size\n",
    "                input_length=max_review_length)(input_holder) # number of words in each sequence\n",
    "\n",
    "\n",
    "x = SimpleRNN(RNN_STATESIZE, dropout=0.2, recurrent_dropout=0.2)(shared_embed)\n",
    "x = Dense(NUM_CLASSES, activation='sigmoid')(x)\n",
    "simple_rnn_model = Model(inputs=input_holder,outputs=x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 500)]             0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 500, 50)           50000     \n",
      "_________________________________________________________________\n",
      "simple_rnn (SimpleRNN)       (None, 100)               15100     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 65,201\n",
      "Trainable params: 65,201\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 500)]             0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 500, 50)           50000     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 100)               60400     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 110,501\n",
      "Trainable params: 110,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 500)]             0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 500, 50)           50000     \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, 100)               45600     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 101       \n",
      "=================================================================\n",
      "Total params: 95,701\n",
      "Trainable params: 95,701\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import LSTM, GRU\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.optimizers.schedules import ExponentialDecay\n",
    "\n",
    "# create LSTM\n",
    "x = LSTM(RNN_STATESIZE, dropout=0.2, recurrent_dropout=0.2)(shared_embed)\n",
    "x = Dense(NUM_CLASSES, activation='sigmoid')(x)\n",
    "lstm_model = Model(inputs=input_holder,outputs=x)\n",
    "\n",
    "# create GRU\n",
    "x = GRU(RNN_STATESIZE, dropout=0.2, recurrent_dropout=0.2)(shared_embed)\n",
    "x = Dense(NUM_CLASSES, activation='sigmoid')(x)\n",
    "gru_model = Model(inputs=input_holder,outputs=x)\n",
    "\n",
    "# lr_schedule = ExponentialDecay(\n",
    "#     initial_learning_rate=0.1,\n",
    "#     decay_steps=10000,\n",
    "#     decay_rate=0.95) \n",
    "\n",
    "opt = Adam(lr=0.0001, epsilon=0.0001, clipnorm=1.0)\n",
    "\n",
    "simple_rnn_model.compile(loss='binary_crossentropy', \n",
    "              optimizer= opt, \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "lstm_model.compile(loss='binary_crossentropy', \n",
    "              optimizer= opt, \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "gru_model.compile(loss='binary_crossentropy', \n",
    "              optimizer= opt, \n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print(simple_rnn_model.summary())\n",
    "print(lstm_model.summary())\n",
    "print(gru_model.summary())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2UAAABoCAIAAAD+Ys9HAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO2dd1gU1/rH32VBFBA0llggKrEgxH5BsMSKxoKaSFGsoIC911z15vHx6mPFaKyYECsRUCMqdookgiioKLYooQUQEJQinfn9cW7mt9kyzMy22d3389fumdlz3vKdM2dnzpwRURQFCIIgCIIgCKIAI20bgCAIgiAIgggaHC8iCIIgCIIgTOB4EUEQBEEQBGECx4sIgiAIgiAIE8aSX+Lj4/fs2aMtUxDdIiwsTNsm/A/ULaJWVqxY4eLiom0r/oeHh4e2TUB0D9QwwgMXF5cVK1bQX/9xfTErKys8PFzjJiE6RnZ2tqB0grpF1Ed4eHhWVpa2rfh/wsPDs7OztW0FokughhEeJCQkxMfHS5YYy+4knOtGiDAJDQ318vLSthXSoG4RdSASibRtgjTLly/39PTUthWIzoAaRnggexkY5y8iCIIgCIIgTOB4EUEQBEEQBGECx4sIgiAIgiAIEzheRBAEQRAEQZjA8SKCIAiCIAjChJzno3mQlpa2ZcuWzZs3W1tbq6RCVZGXl/fixYuhQ4ey2fnOnTt//fUX/bVZs2ZjxoxRl2V/c+PGjXfv3tFfe/bs6eDgoO5GEX6UlZXdvn370aNH//nPf3jXEB0d/dtvv23fvr3BrRo4rKTkJ8WECRPMzc3Z16Z/8UHYUF1dHRcXd/nyZVdX17Fjx2q4dZZdaHV19cmTJ588eWJjYzNo0KDmzZu/e/fOxcUlPj4+PT1dUeWmpqbffPNNVFTU27dvAUAkEnl4eIjFYtk94+Li6DViJk6caGZmpqRfiCTMPYMWUWaMYWJi0qpVq3bt2nXp0kVd9qkU1VxfTE5ODg4OfvLkiUpqUwkFBQWrVq2ytbW9cOECy584Ozs3adLE29vb29u7sLCQpQKUpE+fPgkJCd7e3jNmzGjTpo2u6MYwCQ8Pnzt3bkhICO8arl27tmTJkl9++YXNVg0cVrT8Vq1aVVVVVVdXV1dXV1pa+uDBAx8fn5ycHE616V98EDY8ffo0NDR07969XAWjEth0oR8/fnRycgoLC3Nzc2vRosX69eu7detG1pYLDAxctWpVcnJyXl5ebGyst7f30aNHCwoKXr58GRgYOGfOHAAYMGBARUWFt7f31KlTz507J1t/eXn5xIkTvb29d+7c2bNnTxwsqhzmnkEr8Bhj9OzZ882bN97e3rNnzy4pKSkoKLh06ZKXl1enTp02bNhQU1OjVoNVACXB2bNnpUrYU1BQwO+H7Dl+/Dj7nRMTEx8/fgwAS5YsYf+r+vr6Zs2aAUBRURF3Azkg6cuDBw8AoF+/fmptUYUooxN1oEl7vvrqq27duilTg6enp62tLcutGjisiPy+/PJLqfJVq1Y9ffqUa236Fx8AOHv2rLpbYY/Q7CGQzjYoKIh5N059OHsa7EK3bt1qZGSUlZVFl/j7+69cuZKiqAkTJjx79owUXr58GQCWLVtGvlZWVnbv3p18Li8vNzY2BoB//etfsvUfOHCgdevWALB+/XpVOaVChKYZfvYw9wyah98YgyycTuuKoqj6+vqwsDBLS0tXV9eSkhI1WMoTd3d3d3d3yRKVzV9s2bKlqqqSS1RU1Pr169nv7+joaGdnx7UVkUjUtGlTALCysuL6W/ZI+UJa5HTjD9EWYrFYycVvjYyMjIwUHndSW9V9WMHf8pNl2bJlPO7z6l98EDaQsRRz6rn24expsAt99OhRfX19SUkJXbJt2zZyF3vQoEHdu3eX+ytTU1NfX1/y2czMzM7Ozt7e/sGDB9HR0ZK7URR15MiRuXPnguKjCVEe5p5B8/AbY1haWkqViEQid3f3o0eP3rx5c/DgwdXV1SoyUPWoZv5ifX19bGyshYWFo6MjAGRlZZ0/f37x4sXPnj27ePHiZ599Nm3aNDrT2dnZERER8+fPj42NvX79evv27efMmdOkSZNLly69efPGwsJi7ty5paWlJ06cqKmpadu2rZeXV3R09KRJk0Qi0ZEjR9q1a+fm5sbb1MLCwqCgIF9f308//ZTN/gy+KHIEANThy6tXrxISElJSUgYOHPj1118DwO3bt8mfFTLJxtTUNDEx8dmzZ82bN584cSIA5OTkXLt2LTs7e+DAgSNGjKCrKi4uDgkJWbBgwdWrV1NSUlauXEm6e71HbkAqKiouXrw4YcKE/Pz8yMhIkhSxWPz27duIiAgjIyMPDw/Zg/zu3bvXr1/v2bPn5MmTmesHgKKiovDw8PT0dHJxQuq0qmgrp8MKAMrKyk6ePJmZmdmlSxcnJ6fu3buTiVZcNQ8A165dc3JyIv+a9D4+iMqhKCo2NvbRo0disdjOzs7V1VW23+OkKx4almTUqFGhoaGzZs26cOEC+Rf0ySefkBfjrl69muGHq1atoj8bGRmtXLnSx8dn586dw4YNo8uvXr3q6OjIzzCEGYaeQVFnwtAPyMqSuSp+8NOql5fXiRMnIiMjExMTBw0axGCYNh2UvNjI775eamqqu7s7ABw6dIiiqIiIiFatWgFAYGCgj4/P+PHjAWDr1q1k51OnTjVv3rxJkybz5s3z9fUlk6MdHR2rq6spinJwcLC2tiZ7lpSUWFpauri4UBT18OHDgQMHtmrVKjo6+uHDhywNq6qqAplrxUFBQQCwb98+Rb+ysbEBgLq6OmZfmB3h5MvLly9B3g1BSQIDA4cOHVpfX//nn3927Njx4MGDFEWVl5eTmd1v3ryh97Szs3v58iVFUVFRUX5+fsnJyaGhoRYWFgsWLCA7/Pzzz2ZmZsbGxvv37+/VqxcAPH78mGVICTp6P1puQGJiYsiEp927d/v7+69Zs8bMzGzy5MlBQUHTpk2bMmWKSCRyc3OjKxk3blynTp3Gjx8/btw4clli+vTpDPVTFPXixQtHR8e7d+/W1NQcOXLE1NS0a9eudIWKtnI6rCiKKioq6tq16507d8rKysjfCUdHR3JnjVnzsvKrqakZPHhwZmamgcSHGdCLe3nqJjU1FQCOHTtGvn777bfk3vT9+/ednJwomX6Pq664aliK8vLyzz77DABatWp14sQJRbtJ3Y+WomfPnlVVVe3btweAlJQUutzV1TU1NfX777+XkpxwEJpmWNrD0DMo6kyY+wFZWTJUxQYeY4wPHz7AP+9H02zevFnSWrmGadJB2fvRqpm/mJKSQnfcFEWtW7cOAG7dukW+9u3bV3JmyfTp00UiET01auPGjQBw+PBhYh89xiI/JGMsiqImTZpkY2PDySq5uSwrKztz5gzDLAHJ8SKzLwyOcPKFzXixc+fOCxcupH8+duxY8jkiIgIkpg3l5OSQBJeWltra2paVlZFyMms7Pj6efJ02bRoAnD9/nqKo58+fM7QrF10cLzIEZM+ePQAQFhZGNpGMnzt3jnz997//bWpqSuth3LhxjRo1evHiBUVR9fX15DpuZGQkQ/39+/dfvXo1Ka+vr7e1tZUcDzFs5XRYrV+/vkOHDuRzUlIS6VDIV2bNE/k1a9Zs+PDhw4cPHzJkCJmJRcaLhhAfZnT0XKthJMeL9fX1LVu2jI6OJpu2bNlCPkj1e5x0xUbDzF3o27dvv/rqK3KVxNXVVXIuI02D40WKonbu3AkAM2fOJIVPnjwZM2YMRVE4XmQPS3sUHfvMZzdF/YBcWTJX1SA8xhgM48Xz588DAJETg2Eac1Bd8xdNTU0lv5J7svStfXt7+8zMTHqrubm5sbExveTBunXrjI2N79y502ArKnlpurm5+dSpU9nPMmHwhbcjwN2XmJiYLVu2AMCzZ8+ysrL++OMPUj5+/Pju3bvv2bOHoigAOHPmzMyZMwEgJCSkoqJizZo1CxcuXLhwYW5u7ueff/769Wvyq3bt2gEAOZfzmIGhizAEhNx17dGjB9mzW7duAECuvAKAnZ1dVVWV5IOfDg4OZB+RSDR//nwAuHLliqL6o6Ki7t27R9+9EolEjo6OdPaZt3I6rN68eVNQUEDmvvTq1cvc3JzMVQB2mu/Zs+ft27dv374dExOTkZHRv39/epPexwdRLSKRqFu3bl5eXhcvXoR/3tKV7Pc46Yprvy1L69atr169GhIS0qpVq5s3b/bp0+fRo0c86vH397eysgoJCSGr53z//fcrV67kbRWiCIZjn/nspqgfkCtL5qr4wVurZWVl8Pc0XAbDtOigJmaticViMpqRi5mZmbW1dUFBQYP1qGS8qCQMvrB3BLj70r59+xs3bly+fHnIkCGff/45uUBC6lm9erWvr29kZOS4ceNu3bq1dOlSAEhNTW3btu2BAwfk1kbmOghq7rC6YQ6IJI0bN5b8amJiAgDl5eVyd3Z2djYyMsrJyTE2NpZbf2BgIAB88cUXdIlk6snjdYq2MiMlxWHDhoWGhv7222/Dhw8vLi6urq6mJ69wpXHjxt9++y3pleRulfxqgPFBGuSHH37w8PCYNGnSiBEjTp8+TU/kYkgfJ13xZsqUKSNHjpw6deqtW7dWr1598+ZNrjVYWloGBATs2LFj796969ate/r0qfIz3hBZGI599p05/LMfkJUlp6rUTXJyMgCQ/+rsDdOkg9ofMVRVVeXl5dna2ja4pxDGiwywdwRY+5Kfn0+ueG/cuHHLli3bt2+fPHmy1CT9adOmtW/ffvfu3ampqQ4ODuTJFbFY/PLlSx1Yz0lTqCkglpaWFhYWtra2iuonj2Teu3dPspDOPvNWTsydO3flypXz5s0LCwvbtGnTtm3b6LtvPJgwYULLli3fv39fW1vLuxLQ3/ggzPTu3Ts5OXnBggUxMTF9+/YtKioi5Zrvw/Pz81++fEmuuBBatmz5008/icXimJiY9+/f86hz6dKljRo1Onr06Pbt2xcsWKA6Y5H/h+HY592Zy8pSOCdKiqLi4uLEYjH5H8vPMHU7qP3xYkJCQmVlJZm2aWxsXFlZKXc3kUhUV1enWdO4IekIqMgXPz8/sVj8559/btmyZfr06eSST319veQ+jRo1WrZsWXR09OrVq318fEhhr169ysvLDx8+TO/2/v37gwcP8vBLP1BTQB4+fFhSUjJmzBhF9ZPbbVFRUXJ/zryVE+QCXnBwcM+ePQMDA1Vyj4w8qqJMDfodH0QuVVVVJ0+ebNq06YEDB65cuZKbm0smZmmlD/fz82vduvXy5cvJH2+CjY0Nuf0tNaWBQe0URX38+JF8bteu3fTp00tLS0NCQqZMmaIeww0dhmOfX2cuV5bCOVEuX748KSlp586dZEoGD8M04KBqxovkUCwsLCRfyT8DehmhwsLCqqoqyUOxtrb2+fPn5HN4ePiQIUPIMGvUqFGFhYXBwcHl5eXBwcHv3r1LS0srLi4GgLZt2+bl5aWlpb1584blTQryQ6lBW1JSkpOTU0xMjKJfEePplbqYfVHkCCdfMjIyJJsgfPz4ccmSJcbGxsbGxmRaQ0hISElJSVxc3J07d4qLi8vKykpLS8nOAQEBVlZWhYWF9GRKLy8vGxubVatW7dy58/nz56Ghof7+/jNmzCBbSQAZXgSnfzAEhISRPp2QaNNXREisJE82ZWVl9JA9LCzMy8trxIgRiuqfMGGCnZ3dyZMnycTWnJyc2NjY7OzslJSU2tpa5q2cDqtDhw6Fh4fX1NRUV1dnZmbS2oCGNE/kJ3WhpaKiYvny5SKRyMTERO/jgygPmcVPtEFRFHnsDwBGjRrVsmVLskymVL/HSVdsNKyoC23evPnHjx8DAgLo2p48efLs2bMZM2ZITbogRwHxRYrc3Ny//vqLPpusWrVKJBItXryY3DeHv083xBJEeRiO/cmTJzOc3RT1A3JlyXyibBAeYwzy8smKigrJkoULF+7bt2/x4sXLly8nhQyGadJBaSQffuH33GtCQgJZ2OKLL764fPlyTEwMuSc7d+7c3NzckJAQsobWd999V1NTQ1FUQECAWCxetGjR6tWrp0yZ4ubmRj9JVFpa6uzsDADdu3c/f/78N998M3r0aPLwb3R0tLGxcbNmzRiWwpEkMjLSy8sLAFq3bh0UFJSbm0vKz507JxKJ5L6H4ObNm2TNVQD45ptvzp07x+wLgyPsfTl9+rSTkxMAiESi/v37jxgxYsCAAQ4ODqQbOnr0KKnN19fX2Ni4c+fOhw8fDg8Pb9So0fDhw9+9e0c3N2/evAMHDki68+zZs65duxJ3HBwckpOTSfmxY8fIkhCenp737t1jleN/oovPR1MKAnL37l3yf27WrFlpaWnR0dF9+/YFgHHjxqWmpt69e5ck0dPT89WrVxRF3bhxo0+fPiNHjvzuu+8CAgLIS5wY6qco6s8//yQLBNra2np7e7u5uQ0aNOjQoUMVFRUMW+/cucPpsLpw4YLUesUjR44ksmfQPC0/AOjXr9/w4cOHDh3aq1cvct1l7969hhAfZkA3ny3VJPfu3Rs9ejQA9OnTJzIysqKiom3btlOmTAkLC9u1a9emTZvIbpL9HlddsdEwQxc6YsSIyZMnDxo0aPHixX5+fi1atFiwYEF5eTldSXV19f79++3t7QHAyspqy5YtkouUhYWFffnllwDg6uoaFRVFCr29vYuLiymKKi8v37NnD1nWsWXLlhs3bpSsWQgITTMs7WHoGRR1Jgz9QGlpqVxZKqqqQXiMMSIiIuj3DLu4uLi6uo4bN27ixIkrV668f/++1M5yDdOkg+paT4cTAQEBJiYmFEVlZmZ++PBBdof8/HzygZwwaN6/f6+St+XIbZQHDTpCqdQXyf0rKyultrq6upLOS4r09PSMjAxODTWIjo4XCSoJyMePH+nlZljWn5+fTxY1KC0t5bqVDTdu3AgODv7jjz/i4uKuX79+/vx5b2/vbdu2ka2q0jxLdC4+DOjouVa71NTUVFVVySZamT5cGQ3n5OSQD5mZmcnJybxVpKMITTOc7GE49rl25opkyaMqZlTY33IyTLUOyo4XtflWD7LSoSxkOUqQeWKOfkcfwxRjf3//3r17M7cr+y4KJVHkCLDwhT2Sz+dLTbt5/Pixra0tefO1FB06dODakH6jkoA0adJEUdIV1U8rwcLCguvWBklKSpo9e3ZmZqZYLO7cuTMpJE8Ek88q1zwzOhcfRLWQp+7IEtmSKPOeVWU03LZtW/LBxsaGobtGBAjDsc+1M1ckS7lVKTPMUGF/y8lHTg7yQAvjxY8fP9bW1paVlfHr+gFA8l1MUtDa0gDKO6IkSUlJa9as6dGjR0xMzK+//qoVGxAhkJKSkpube+zYsZEjR3bo0CE9PT0xMTElJUVNr+vVOTA+CIJwQiDDDEGh6fHi6dOnb9y4QVHU2rVr/fz8GrwWKBcPDw+VG8YVlTiiJPX19ffv309KSgoKCurYsaPmDUAEwuzZs4uLi3/55ZelS5caGxv36NHDx8dn8+bNjRo10rZpggDjgyAIJ4QwzBAamh4vkhfLks9SN1V1CyE44ujoWFRUZGRkZFArbyOyiESiFStWrFixoqamhn5gE6HB+CAIgiiJpseLysxfERQCcYTMV0AQAg6GmMH4IAiC8AOvSyEIgiAIgiBM4HgRQRAEQRAEYQLHiwiCIAiCIAgTOF5EEARBEARBmJDztIRIJNK8HQiiJKhbxEDw8vIiLyJDEB0FNawTkHeu0sgZL5K3qxksgYGBAEC/9huRJT4+fu/evdq2QhoD1y0DJF8YH34I8Ky2bNkyFxcXbVuhfbCvZglqWOUYgvaIj5LIGS96enpqxBiBEhYWBgYfhAYR4HgRU8bA3r17MT78EOC51sXFBbMJ2FezBjWscgxBe8RHSXD+IoIgCIIgCMIEjhcRBEEQBEEQJnC8iCAIgiAIgjCB40UEQRAEQRCECRwvIgiCIAiCIEyoZbyYlpbm6+ubnZ2tjsoRRAOghhH9BhWOCBnUpwBRy3gxOTk5ODj4yZMn6qgcQTQAahjRb1DhiJBBfQoQtYwX3d3dCwoKxowZo47KaU6cOKHW+rWIqlzT9RCdOHHixYsXWmlaMxoG3c8RD1Desly6dCkhIUGTLaLCVQKKmebw4cOZmZmqqg31KUBpqWv+YsuWLdVUMyEqKmr9+vVqbUJbqMo1PQjRvn37unfv3qNHj127dmVlZWm4dXVrGPQiR1xBecvl0qVLLi4uNjY2GzZsSE1N1UyjqHAlQTFLsmHDho4dO7q4uBw6dKiwsFD5Cg1Zn8KUlpz3uyhPfX19bGyshYWFo6MjAGRlZZ0/f37x4sXPnj27ePHiZ599Nm3aNCMjIwDIzs6OiIiYP39+bGzs9evX27dvP2fOnCZNmgDApUuX3rx5Y2FhMXfu3NLS0hMnTtTU1LRt29bLyys6OnrSpEkikejIkSPt2rVzc3MrLCwMCgry9fX99NNP1eERP0pLSyMjI58/f25jYzNq1CgbGxtg9AsAZF1TFCKu9QgzRMzU19cDQGpq6vr169esWePs7Dxz5kwPD48WLVpooGlJDQMvGet3juTKGzgeufrdA3BCLBZnZ2fv2LHjv//9r52d3axZs6ZOndqhQwc1NYcKl4Jrdy3rHfAKlG5FiYH6+nqKou7du3f//v3FixePGDFixowZkyZNsrCw4FebfuhTr/pJSgLyhllKOVJTU8k7qg8dOkRRVERERKtWrQAgMDDQx8dn/PjxALB161aKok6dOtW8efMmTZrMmzfP19d37NixAODo6FhdXU2qcnBwsLa2Jp9LSkosLS1dXFwoinr48OHAgQNbtWoVHR398OFDiqKCgoIAYN++fUoaT1GUu7u7u7u78vU8evSoR48e586dy8/P37Vrl4WFxfHjx8kmRX5RMq4xh4h9PZRKQ6QSnbChT58+kloViURisdjIyGjYsGHHjx8vKSlRkz1SGqaUkLG2ciSJOvLFIG+K9ZErzB5ACgA4e/asyquVws/Pz9j4H//eTUxMAKBnz5579+7Nzc1VrT36oXBV9dUUr+5a1jt+gdKAmDWj4ebNm0sKmPTVJiYmY8eOPX78eHl5OXt7hK9PltrT6X5S1kfVjxcpikpJSZHM9Lp16wDg1q1b5Gvfvn379etHPk+fPl0kEj19+pR83bhxIwAcPnyYNpeOAvkhnelJkybZ2NjQm8rKys6cOUMPIJRBJX1QVVWVnZ3dpk2b6BJvb+9GjRqlpqZSjH5RMq4xhIhTPSoMkbbGi3J7otDQ0NOnT6vcHikNU3xlrK0cSaLyfDHLm+Jy5AqwB5BCW+NFqf9Izs7OR44c+fDhg6rs0QOFq2q8yLu7lvKO4hsodYtZK+NFyX8+IpHI3Nx8+vTpERERNTU1bOwRuD7ZaE/X+0lZH9VyP9rU1FTyK7lqamdnR77a29tfv36dfDY3Nzc2NnZwcCBf161bt23btjt37gQEBDTYikgkoj+bm5tPnTpVJcarhGvXrr148cLZ2ZkuGT169JkzZ3788cfdu3c3+HMp14QZIg28aj0vL09ueV1dHQDU19ffuHHj6tWrRG/R0dFDhgwhtyeUR0rDYJAyVoQhyFuSwMDA8PBwNVVOSEtLk1tOURRRe2JiYmJi4uLFi8nnCRMmNG7cWJkWUeE0yuhZ0jtQIlDqjtKpU6fUrWEyEFRUXl5efvbs2VOnTpHZRK9fv2auTQ/0qX/9pBbW6xaLxRRFyd1kZmZmbW1dUFDAph6pA1VQPHv2DAAk520MHjwYAJ4/f87m5wyu6U2IdB1DkLEiUN6GgOEoXBk9M3vHPlDCj5LQEL4+9a+fVMv1Rd5UVVXl5eWNHj2azc5CPsA++eQTAIiPjyf6AIAOHTqYmJgoumIvBYNrwglRaGio+ion9O3bNzc3V7ac9BRisdjV1XX27Nk1NTXTpk0bNmyYuu1hiXBypCYMQd6SLF++XN1X0/39/R8/fixbLhKJjIyMKIpycnLy8fGZMmWKlZWVk5OTkhcXlUSYaeKNMnpm9o59oNQdpenTp6tbwySMspiYmNTW1pqZmX399deenp5jxowxMTHp3Lmz+iwRiD71r58U1vsAExISKisryVRWADA2Nq6srJS7p0gkIrdphEn//v0B4M6dO3TJ06dPa2pqXFxcgNEvaMg1yRApU48uQs/l+vLLL4ODg9+9e3flyhUPDw+5E7+0iN7niFneoMSRqzc9gPKQ51169Oixe/fuv/76Kz4+3t/f39LSUtt2Aeidwnl31w16xzJQOhElrtCzzF1dXX/++ef8/PyTJ0+6ublpoLsWiD71r59Uy3ixqqoKAOgVmEpKSgCgurqafC0sLKyqqqKvJNfW1tKXZ8PDw4cMGUJHYdSoUYWFhcHBweXl5WR8kJaWVlxcDABt27bNy8tLS0t78+ZNeXl5UlKSk5NTTEyMOtzhQa9evWbNmnXnzh16/dLffvutS5cu/v7+wOgXyLgGikPEqR6hhYg9IpHI2NhYJBI5Ozv/8MMP+fn5UVFRM2fObNq0qfoaldIw8JWxXuaIWd7A5cgFPe0BeEC0RIaJdnZ2mzdvTk9Pf/z48dKlS9u0aaPy5lDhNLy7a1kxA69A6USUWEL+1YvF4hEjRhw/fryoqOjKlSszZ840MzPjVI8e6FP/+knVjxfv3bu3efNmADh79uyVK1diY2MvXLgAAFu3bs3Ly/vll1/i4uJKS0s3b95cW1sLAEZGRgcPHlyzZs3UqVMzMjIuXbpEV+Xh4eHs7Ozr6+vo6NisWbN+/fr17t373LlzZBNFUf369YuMjDQ3N8/IyHjw4EGDU2g1yeHDh2fOnEmWEvjxxx8jIyNv377dqFEjYPQLZFwDxSHiVI8AQ9Qg5OEVBweHbdu2ZWRk3L17d968eRpYfFFKwwDAW8b6miMGeQOXIxf0twfgSl1dnbW19Zo1a54+ffr8+fN169apb/FFVLgU/LprWTEDr0DpSpSYMTIyEolE/fv3379/f15e3vXr16dPn85v8UW90ae+9ZOSD0trbJ0UmoCAABMTE4qiMjMzP3z4IHef/Px88qGiokKy/P3795KPhSv6OZnf2bYAAAz/SURBVFdUuKYXRVHv37///fffs7KyZDcp8ov6p2sNhohlPZTqQqQxnRw/fvz58+fCsUcRAsyRJOqLD4O8KXZHrgB7AClAI2uRRERExMfHC8ceKYSpcNX21RSv7lrKO96BUquYNaOZQ4cOZWRkaMUezeuTk/Z0tJ/U0Ho6PKAXPZeFrNIJAFJTvK2srCS/CmRmjxRWVlYDBgyQu0mRXyDjGkFRiNjXI8wQMTBz5kxtm8ANQ8sRg7yB9ZFL0NcegCXk7SDCR78VzqO7litm4B4oHYqSIubNm6dtEwSqT73pJ7X8vMvHjx9ra2vLysq0a4aQwRAJH8wRbzB0OgGmiSUYKK2g92EXiIPaHC+ePn36xo0bFEWtXbv20aNHWrREsGCIhA/miDcYOp0A08QSDJRW0PuwC8dBbd6PHj9+/Lhx48hn2cXcEcAQ6QKYI95g6HQCTBNLMFBaQe/DLhwHtTleVDTzA6HBEAkfzBFvMHQ6AaaJJRgoraD3YReOg8JarxtBEARBEAQRGjheRBAEQRAEQZjA8SKCIAiCIAjChJz5i6GhoZq3QzhkZ2eDwQeBmfj4eG2bIAdMmSJIvjA+eoMwD0DNg3217qLrGjYE7WVnZ1tbW/+jSHLxbvIeCARhA8s14jUA6hZRK5p/nwoD2g4GopOghhEeNPx+F0ynJKGhoV5eXhgTSUhMtG2FNAaeIw8PDwAICwvTtiH6hkgk0rYJ0pw9e9bT01PbVugwIpHIoGKIGtYuOqo3ck6RBOcvIgiCIAiCIEzgeBFBEARBEARhAseLCIIgCIIgCBM4XkQQBEEQBEGYwPEigiAIgiAIwgSOFxEEQRAEQRAm5Kyn0yDp6en0Yptdu3bt168fvam2tjYxMXHAgAEAkJOTc+bMmfz8/NGjRw8dOlQsFrOs/8qVKyUlJeRzVlbWokWLzMzMAKCqqio2NvbRo0eDBg3q37+/VIVytyYnJ7do0aJDhw70bmlpaffu3SOfu3Xr1rdvXx4RkAVjootIpgb4ZkdRaoAxOyxTowGEpk9lmlNUp1YCKwSEpnCWcJKTupOrkhgCQF5e3osXL4YOHSpVrhO9hBYxtPirz18VWC65GCNZ97jBxTZPnToFACEhIbm5uSUlJXT5+/fvt27dSkqePn06f/78nJyc+Pj4AQMGtGvXLiMjg81Kns+fP5dcLGrKlCmk/O3bt506dQoKCiooKFi9evW4ceNqa2vpXynaWlNTM2/evNjYWHrPsrKy9PT0uLg4ExOT5cuXN2gPxoR3TDQGP3skU0PxzY6i1FCM2WGfGva4u7tLra3KBgHqk3dzDHUqE1gQ3lrHLO0RoMLZwFVOPJKr4Rjm5+evXLmySZMmS5YskdqkmV7CwDWsQ/FXq79cLZc9p/AfL75//16yMDs7283NjS6cOnVqYGAg+RwdHQ0AixYtYmOin59fdHR05t9UVFRQFFVXVzdo0KAJEyaQfWprazt06LB27VrylXlrbW3tmDFjUlJSpBrq2LGjyseLGBOtwMMeqdRQfLMjNzUUY/z5paZBeIwXBatPHs01WCfvwOrouVawCufXomqTq+EYJiYmPn78GACkzt8a6yUMXMO6En91+8vVcjWOFz09PX/66Sf6q4+Pj4ODA/lcUVEBAJMnT26w5tzc3P79+2dlZUmVk0hdunSJLtm0aZO5uXlZWVmDWymKunnzprOzs1SdGhgvGnhMNAYPe6RSQ/HKjqLUUIzx55eaBuExXhSyPrk2x6ZOfoHV0XOtkBXOo0XVJleTMSRUVVXJnr811ksYsoYJOhF/dftLYG+57DlFNc+7JCYmXrlyxd3dnS45ePDglStXyOeMjAwAGDZsWIP17N+//969ezY2Nra2tj///DP19xvezp8/DwA9evSg9/ziiy/Ky8sjIyMb3AoAI0eOLC0tJbtpDIyJYJFNDfDKjqLUAGP8BZIageuTa3Ns6jQczQtc4Txa1HxyVRVDBoTfS2gRQ4u/BvwlKGO5asaLO3bscHFxadq0KV3SuHFjelrlr7/+am9v7+fn12A9Q4YMWb169aBBg7Kzs318fEaNGlVXVwcAr1+/BoC2bdvSe7Zu3RoAXr161eBWwsCBA7ds2aKUkxzBmAgW2dQAr+woSg0wxl8gqRG+Pjk1x7JOA9G88BXOtUXNJ1dVMWRA+L2EFjG0+GvAXxrelqtmvJiSktKuXTu5myiKCg4OPnbsWKNGjRqsZ9SoUTt27IiLi7t//76dnd2tW7d27twJAG/fvhWLxZI1kMflcnNzG9xKcHBwePLkSXV1NU8PuYMxESwMqQEu2VGUGmCMv0BSI3x9cmqOZZ0GonnhK5xri5pPrqpiyIDwewktYmjx14C/NLwtV8F4sbq6Oi0tTXIkLsmtW7dGjx7t4uLCqc5evXolJSVZW1uHhIQAgIWFhdQO5E9nmzZtGtxKsLKyqq2tJX8aNADGRLAwpwZ4ZUcqNcAYfyGkRif0yak5lnUaguZ1QuFcW9RwctURQ1kE3ktoEUOLv2b8peFtuQrGi0VFRXV1dU2aNJG7NSoqavPmzTyqNTMzmzhx4h9//AEANjY2dXV1ZAonobS0FADs7e0b3Eog6c/OzuZhCQ8wJoKFOTXANzuSqQHG+AshNTqhT07NsazTEDSvEwrn2qKGk6umGEoh8F5Cixha/DXjLw1vy1UwXmzTpk2zZs1INGXp2LGjlZUVv5rt7Oy6du0KAN27dweArKwselNhYSH8nTzmrYTi4mIAsLGx4WcJVzAmgoU5NaBEdujUAGP8hZAandAnp+ZY1mkImtcJhXNtUcPJVV8MJRF4L6FFDC3+mvGXhrflqpm/6ODgkJ+fL3dTQEAA72ovXLgwceJEAJgzZ46pqenvv/9Ob0pKSurduzfpSpi3EnJzc0UiUadOnXgbwxWMiWBhSA0okR06NcAYf4GkRvj65NQcyzoNRPPCVzjXFjWfXDXFUBLh9xJaxNDirwF/aXhbrprx4uDBg588eSJbHhcXN378+MzMTKlyf3//sWPHvn37VrLw1atXy5Yte/jwIfmamppaXl6+YcMGAGjTps2iRYt27txJlleorKy8dOnSjz/+aGRk1OBWQnp6+qhRoxo3bqwSf9mAMREsilIDCrLDNTXAGH+BpEbg+lRtzOkaDETzAle43OaYW9R8clUSQxpyRaeyslKyUPi9hBYxtPhrwF8a/pZLLsbIe23qoqKi1q1bv379WmrPXbt2iUSiqKgoqfLPP/8cAHbt2iVZmJSURK64Dhs2bO3atdu3b//48SO9tb6+fu3atePHj9+3b9/69etPnDgh+VvmrVVVVS1atLh586ZkobrX68aYaAyu9ihKDaUgOzxSQzHGn0dqGoTret0C16fKY07xDSzo4FrHAle43OYabFGFydVYDAmRkZFeXl4A0Lp166CgoNzcXDZOadhfTYLx15a/nCxX4/tdDh8+vHDhQtmd3717J1tYWVl59uzZixcvypa/evUqOztbUdO1tbV5eXlct4aGhk6cOFGqUAPvdzHwmGgMHvYoSg0lLzu8U0MxZodTahqEx/tdhKxPdcScX2B18VxLCVvhippj06JKkqvJGLJB3b2EIWuYDQKJv2b8ZW+5Kt/vIvnoEAD4+fm9e/eOvpVA88knn8j9bXx8/NixY6XKTU1Nu3Tp0r59e0WNisXiTz/9lNPWFy9enD59ml4GgoZeeFaFYEx0BUWpAXnZ4Z0aYMwOp9SoA8HqU5nmFNWpycAKAcEqnKE5Ni1qMrkqiSEbhNxLaBFDi78G/FXWcsnBI8vrNGfPnrW0tHRzc9u6davkVc3Kyko/P7/ExMQGa7h586bUrQ01kZ6ePnPmTMmx+ZMnT3bu3Ll48WIrK6v169c3WAPGRBY9uL5I6Uhq2MPj+iIlyCCoozllAgu6eW2GwuQqRqdjqFZ/NQPGXxFq9Zer5aq5H81MRkaGkjWokJycnPr6emVqwJjIoh/jRYLepIbfeJEgqCCoA2UCq7vnWgImVxadjqEG/FU3GH9m1OQvV8tlzynGPC9LKuazzz5TeZ28YVgwXZNgTAQLpgYEFgR1YMiax+Qqj6BiaIBiNrT4q8lf5S1XzXo6CIIgCIIgiL6C40UEQRAEQRCECRwvIgiCIAiCIEzgeBFBEARBEARhQs7zLh4eHpq3Q7BkZ2cDxuSfkJgIDQPPUUJCAhh8EAyEwMDAsLAwbVuh22AMtYuhxV8X/U1ISHB2dpYsEX/33Xf0l5KSkg8fPmjaKGFjaWlpb2+vbSuEBYmJp6entg35H6hbALC2tra2tta2FXqIvb39V199ZWNjo21D/kdqaqqlpaW2rdBt7O3tDSqGqGHtoqN6s7a2dnFxcXFxoUtEFEVp0SAEQRAEQRBE4OD8RQRBEARBEIQJHC8iCIIgCIIgTOB4EUEQBEEQBGECx4sIgiAIgiAIE/8HVzPDtLqEr2AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# you will need to install pydot properly on your machine to get this running\n",
    "plot_model(\n",
    "    lstm_model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 246s 10ms/sample - loss: 0.7162 - accuracy: 0.5032 - val_loss: 0.6961 - val_accuracy: 0.5017\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 244s 10ms/sample - loss: 0.7130 - accuracy: 0.4996 - val_loss: 0.6928 - val_accuracy: 0.5120\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 245s 10ms/sample - loss: 0.7075 - accuracy: 0.5064 - val_loss: 0.6923 - val_accuracy: 0.5165\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 609s 24ms/sample - loss: 0.6923 - accuracy: 0.5310 - val_loss: 0.6909 - val_accuracy: 0.5660\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 611s 24ms/sample - loss: 0.6838 - accuracy: 0.5788 - val_loss: 0.6763 - val_accuracy: 0.6073\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 608s 24ms/sample - loss: 0.6577 - accuracy: 0.6342 - val_loss: 0.6259 - val_accuracy: 0.7017\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 645s 26ms/sample - loss: 0.6680 - accuracy: 0.6124 - val_loss: 0.6573 - val_accuracy: 0.6486\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 642s 26ms/sample - loss: 0.5713 - accuracy: 0.7234 - val_loss: 0.6039 - val_accuracy: 0.6953\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 657s 26ms/sample - loss: 0.4982 - accuracy: 0.7674 - val_loss: 0.5727 - val_accuracy: 0.7153\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fe77c8ee290>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# takes about 2 hours to run\n",
    "simple_rnn_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))\n",
    "lstm_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))\n",
    "gru_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 244s 10ms/sample - loss: 0.7020 - accuracy: 0.5172 - val_loss: 0.6843 - val_accuracy: 0.5565\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 244s 10ms/sample - loss: 0.6996 - accuracy: 0.5235 - val_loss: 0.6821 - val_accuracy: 0.5795\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 250s 10ms/sample - loss: 0.6948 - accuracy: 0.5278 - val_loss: 0.6841 - val_accuracy: 0.5426\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 605s 24ms/sample - loss: 0.5038 - accuracy: 0.7801 - val_loss: 0.4623 - val_accuracy: 0.7990\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 605s 24ms/sample - loss: 0.4548 - accuracy: 0.8014 - val_loss: 0.4145 - val_accuracy: 0.8173\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 603s 24ms/sample - loss: 0.4299 - accuracy: 0.8128 - val_loss: 0.3902 - val_accuracy: 0.8276\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/3\n",
      "25000/25000 [==============================] - 650s 26ms/sample - loss: 0.4377 - accuracy: 0.8047 - val_loss: 0.5565 - val_accuracy: 0.7218\n",
      "Epoch 2/3\n",
      "25000/25000 [==============================] - 638s 26ms/sample - loss: 0.4273 - accuracy: 0.8115 - val_loss: 0.5551 - val_accuracy: 0.7225\n",
      "Epoch 3/3\n",
      "25000/25000 [==============================] - 637s 25ms/sample - loss: 0.4324 - accuracy: 0.8063 - val_loss: 0.5709 - val_accuracy: 0.7027\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fe77caeae90>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# takes about 2 hours to run\n",
    "simple_rnn_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))\n",
    "lstm_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))\n",
    "gru_model.fit(X_train, y_train, epochs=3, batch_size=64, validation_data=(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this operation can take time \n",
    "yhat_rnn = simple_rnn_model.predict(X_test)\n",
    "yhat_lstm = lstm_model.predict(X_test)\n",
    "yhat_gru = gru_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAARcklEQVR4nO3df4zceX3f8eere3FaAgkktwnU9mErODmcBq6wNU2VlEujA18gmGsSxUdaRBJkGcVtEwl6rqqStlQKJ9KQhDNdWcQhP5S4aUvACgumTXRAAkjeC+YuPmK0cgBvHMTekZIepfh89+4f8z0yzM3ufHeZ9eLPPR/SSPP5fN/znfd61i9//J3vdyZVhSTp2ve3troBSdJ0GOiS1AgDXZIaYaBLUiMMdElqxHVb9cTXX3997dq1a6ueXpKuSffcc88DVTU7btuWBfquXbtYXFzcqqeXpGtSkk+tts1DLpLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IheV4om2Q/8MjADvK2q3jiy/ZuA3wJu6Pb5C1X1a1PuVdewXUffvdUtNOuTb3zJVregrxETV+hJZoBjwK3AXuD2JHtHyn4auL+qngvcDPznJNum3KskaQ19DrnsA5aq6kJVXQZOAgdGagp4SpIATwY+B1yZaqeSpDX1CfTtwMWh8XI3N+wu4NnAJeA+4F9V1aOjO0pyKMliksWVlZUNtixJGqdPoGfM3Og3S78YOAv8XeAm4K4k3/i4B1Udr6q5qpqbnR376Y+SpA3qE+jLwM6h8Q4GK/FhPwG8owaWgD8HbpxOi5KkPvoE+hlgT5Ld3RudB4FTIzWfBn4AIMm3Ad8JXJhmo5KktU08bbGqriQ5ApxmcNriiao6l+Rwt30eeAPw9iT3MThEc0dVPbCJfUuSRvQ6D72qFoCFkbn5ofuXgBdNtzVJ0np4pagkNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRG9Aj3J/iTnkywlOTpm++uSnO1uf5rkkSTfPP12JUmrmRjoSWaAY8CtwF7g9iR7h2uq6k1VdVNV3QT8G+D9VfW5TehXkrSKPiv0fcBSVV2oqsvASeDAGvW3A78zjeYkSf31+U7R7cDFofEy8IJxhUmeBOwHjqyy/RBwCOCGG25YV6OSrq5dR9+91S0065NvfMmm7LfPCj1j5mqV2h8C/ni1wy1Vdbyq5qpqbnZ2tm+PkqQe+gT6MrBzaLwDuLRK7UE83CJJW6JPoJ8B9iTZnWQbg9A+NVqU5JuAFwLvmm6LkqQ+Jh5Dr6orSY4Ap4EZ4ERVnUtyuNs+35XeBryvqr6wad1KklbV501RqmoBWBiZmx8Zvx14+7QakyStj1eKSlIjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiN6BXqS/UnOJ1lKcnSVmpuTnE1yLsn7p9umJGmSiV9Bl2QGOAbcAiwDZ5Kcqqr7h2qeCrwV2F9Vn07yrZvUryRpFX1W6PuApaq6UFWXgZPAgZGaVwDvqKpPA1TVZ6fbpiRpkj6Bvh24ODRe7uaGfQfwtCR3J7knySvH7SjJoSSLSRZXVlY21rEkaaw+gZ4xczUyvg54PvAS4MXAv0vyHY97UNXxqpqrqrnZ2dl1NytJWt3EY+gMVuQ7h8Y7gEtjah6oqi8AX0jyAeC5wCem0qUkaaI+K/QzwJ4ku5NsAw4Cp0Zq3gV8X5LrkjwJeAHw8em2Kklay8QVelVdSXIEOA3MACeq6lySw932+ar6eJL3AvcCjwJvq6o/3czGJUlfqc8hF6pqAVgYmZsfGb8JeNP0WpMkrYdXikpSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1Ihe56F/rdl19N1b3UKzPvnGl2x1C5I2yBW6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqRG9Aj3J/iTnkywlOTpm+81JPp/kbHd7/fRblSStZeKVoklmgGPALQy+DPpMklNVdf9I6Qer6qWb0KMkqYc+K/R9wFJVXaiqy8BJ4MDmtiVJWq8+gb4duDg0Xu7mRn1Pko8leU+S7xq3oySHkiwmWVxZWdlAu5Kk1fQJ9IyZq5HxnwDPrKrnAm8B3jluR1V1vKrmqmpudnZ2XY1KktbWJ9CXgZ1D4x3ApeGCqvrrqnqou78AfF2S66fWpSRpoj6BfgbYk2R3km3AQeDUcEGSpydJd39ft98Hp92sJGl1E89yqaorSY4Ap4EZ4ERVnUtyuNs+D/wI8JokV4AvAgeravSwjCRpE/X6govuMMrCyNz80P27gLum25okaT28UlSSGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIa0SvQk+xPcj7JUpKja9T9gySPJPmR6bUoSepjYqAnmQGOAbcCe4Hbk+xdpe5OBt89Kkm6yvqs0PcBS1V1oaouAyeBA2Pq/gXwP4DPTrE/SVJPfQJ9O3BxaLzczX1Zku3AbcA8a0hyKMliksWVlZX19ipJWkOfQM+YuRoZ/xJwR1U9staOqup4Vc1V1dzs7GzPFiVJfVzXo2YZ2Dk03gFcGqmZA04mAbge+MEkV6rqndNoUpI0WZ9APwPsSbIb+AvgIPCK4YKq2v3Y/SRvB37fMJekq2tioFfVlSRHGJy9MgOcqKpzSQ5329c8bi5Jujr6rNCpqgVgYWRubJBX1au++rYkSevllaKS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEb0CvQk+5OcT7KU5OiY7QeS3JvkbJLFJN87/VYlSWuZ+I1FSWaAY8AtDL4w+kySU1V1/1DZHwCnqqqSPAf4XeDGzWhYkjRenxX6PmCpqi5U1WXgJHBguKCqHqqq6obfABSSpKuqT6BvBy4OjZe7ua+Q5LYkfwa8G/jJcTtKcqg7JLO4srKykX4lSavoE+gZM/e4FXhV/V5V3Qi8HHjDuB1V1fGqmququdnZ2XU1KklaW59AXwZ2Do13AJdWK66qDwDfnuT6r7I3SdI69An0M8CeJLuTbAMOAqeGC5I8K0m6+88DtgEPTrtZSdLqJp7lUlVXkhwBTgMzwImqOpfkcLd9Hvhh4JVJHga+CPzY0JukkqSrYGKgA1TVArAwMjc/dP9O4M7ptiZJWg+vFJWkRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RG9Ar0JPuTnE+ylOTomO0/nuTe7vahJM+dfquSpLVMDPQkM8Ax4FZgL3B7kr0jZX8OvLCqngO8ATg+7UYlSWvrs0LfByxV1YWqugycBA4MF1TVh6rqr7rhR4Ad021TkjRJn0DfDlwcGi93c6v5KeA94zYkOZRkMcniyspK/y4lSRP1CfSMmauxhcn3Mwj0O8Ztr6rjVTVXVXOzs7P9u5QkTXRdj5plYOfQeAdwabQoyXOAtwG3VtWD02lPktRXnxX6GWBPkt1JtgEHgVPDBUluAN4B/POq+sT025QkTTJxhV5VV5IcAU4DM8CJqjqX5HC3fR54PfAtwFuTAFypqrnNa1uSNKrPIReqagFYGJmbH7r/auDV021NkrQeXikqSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWpEr0BPsj/J+SRLSY6O2X5jkg8n+VKS106/TUnSJBO/sSjJDHAMuIXBF0afSXKqqu4fKvsc8C+Bl29Gk5Kkyfqs0PcBS1V1oaouAyeBA8MFVfXZqjoDPLwJPUqSeugT6NuBi0Pj5W5u3ZIcSrKYZHFlZWUju5AkraJPoGfMXG3kyarqeFXNVdXc7OzsRnYhSVpFn0BfBnYOjXcAlzanHUnSRvUJ9DPAniS7k2wDDgKnNrctSdJ6TTzLpaquJDkCnAZmgBNVdS7J4W77fJKnA4vANwKPJvkZYG9V/fXmtS5JGjYx0AGqagFYGJmbH7r/GQaHYiRJW8QrRSWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRvQI9yf4k55MsJTk6ZnuS/Eq3/d4kz5t+q5KktUwM9CQzwDHgVmAvcHuSvSNltwJ7utsh4L9MuU9J0gR9Vuj7gKWqulBVl4GTwIGRmgPAb9TAR4CnJnnGlHuVJK2hz5dEbwcuDo2XgRf0qNkO/OVwUZJDDFbwAA8lOb+ubq9d1wMPbHUTfeTOre7ga4av2bXlmnm94Kt+zZ652oY+gZ4xc7WBGqrqOHC8x3M2JcliVc1tdR/qz9fs2uLrNdDnkMsysHNovAO4tIEaSdIm6hPoZ4A9SXYn2QYcBE6N1JwCXtmd7fIPgc9X1V+O7kiStHkmHnKpqitJjgCngRngRFWdS3K42z4PLAA/CCwB/xf4ic1r+Zr0hDvM1ABfs2uLrxeQqscd6pYkXYO8UlSSGmGgS1IjDPSekvzbJOe6jzY4m+QFSd425qrZje7/oWnsR4837s82yXcmubt7LT+e5HiSF3fjs0ke6j7u4myS30hyc5JK8lND+/j73dxrr+5P9MSS5NuS/HaSC0nuSfLhJLd1r8nnk3w0yZ8l+YWhx/z70dclySeTXH/1f4Krp8956E94Sb4HeCnwvKr6UvdLsa2qXr3FrWnjfgV4c1W9CyDJd1fVfQze/CfJ3cBrq2qxG98M3Af8GPCr3T4OAh+7ql0/wSQJ8E7g16vqFd3cM4GXAX8FfLCqXprk7wAfTfJ7VfXHW9bwFnOF3s8zgAeq6ksAVfVAVV3qVnhzMFgFJrmzW0H8ryT7uu0Xkrysq3lVkncleW+3+vu5cU+W5HVJznT/G/gPV+2nfGJ5BoPrJwDownySTwN/u1sxBtgPvGeT+tPAPwEud2fTAVBVn6qqtwwXVdUXgbMMrlB/wjLQ+3kfsDPJJ5K8NckLx9R8A3B3VT0f+D/AfwJuAW4D/uNQ3T7gx4GbgB997B+ExyR5EYMPOdvX1Tw/yT+e7o8j4M3AHyZ5T5KfTfLUno/778CPAv8I+BPgS5vUnwa+i8Gf85qSPI3B35sPbHpHX8MM9B6q6iHg+Qw+h2YF+K9JXjVSdhl4b3f/PuD9VfVwd3/XUN3/rKoHuxXFO4DvHdnPi7rbRxn8It/I4BdVU1RVvwY8G/hvwM3AR5J8fY+H/i6DQL8d+J1Na1BjJTmW5GNJznRT35fkXuAzwO9X1We6+dXOx276PG0DvaeqeqSq7q6qnwOOAD88UvJw/c1J/Y/Srdyq6lG+8r2K0V+ocZ+L8/NVdVN3e1ZV/Sqauqq6VFUnquoAcAX4ez0e8xngYQb/+/qDTW5RcA748vcrVNVPAz8AzHZTH6yq5wDfDbwmyU3d/IPA00b29RTgf29ms1vNQO+hOyNieJV8E/CpDe7uliTf3L2J83Jg9A2c08BPJnly99zbk3zrBp9Lq8jgS1u+rrv/dOBbgL/o+fDXA3dU1SOb1Z++7A8ZvG/xmqG5J40WVdUngJ8H7uimPgC8LMlTAJL8U+Bjrb9mnuXSz5OBt3THWa8w+IiDQwyOp67XHwG/CTwL+O3HzqJ4TFW9L8mzgQ8P3nfjIeCfAZ/dcPd6UpLlofEvMvgAuV9O8v+6udcN/Xd9TVX1oWk3qPGqqpK8HHhzkn/N4JDnF/ib4B42D7w2ye6qujfJXcAfJSkGf3+aPyvNS/+vou64+1xVHdnqXiS1x0MuktQIV+iS1AhX6JLUCANdkhphoEtSIwx0SWqEgS5Jjfj/qTrKp/mVcrIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics as mt\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "acc = [mt.accuracy_score(y_test,np.round(yhat_rnn)),\n",
    "       mt.accuracy_score(y_test,np.round(yhat_lstm)),\n",
    "       mt.accuracy_score(y_test,np.round(yhat_gru)),\n",
    "      ]\n",
    "\n",
    "plt.bar([1,2,3],acc)\n",
    "plt.xticks([1,2,3],['Simple','LSTM','GRU'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_____\n",
    "\n",
    "# RNNs with Pre-processing in Keras\n",
    "\n",
    "## Moving to 20 Newsgroups\n",
    "So we should be able to convert a new dataset into the same format as above. Let's do this from scratch, converting the 20 news groups data. \n",
    "- http://qwone.com/~jason/20Newsgroups/\n",
    "\n",
    "We looked at this a while back when we created the tf-idf and bag-of-words models. This time, we are not going to get rid of the sequence of words for classification. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# convert 20 newsgroups into keras sequence format\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "bunch = fetch_20newsgroups(subset='train', shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------Random Document---------\n",
      "================================\n",
      "Document Label:  talk.religion.misc\n",
      "================================\n",
      "From: lanphi872@moscow.uidaho.edu (Rob Lanphier)\n",
      "Subject: Re: A KIND and LOVING God!!\n",
      "Organization: University of Idaho\n",
      "Lines: 104\n",
      "Distribution: world\n",
      "NNTP-Posting-Host: snake.cs.uidaho.edu\n",
      "X-Newsreader: TIN [version 1.1 PL8]\n",
      "\n",
      "Malcolm Lee (mlee@post.RoyalRoads.ca) wrote:\n",
      ": I will clarify my earlier quote.  God's laws were originally written for \n",
      ": the Israelites.  Jesus changed that fact by now making the Law applicable to\n",
      ": all people, not just the Jews.  Gentiles could be part of the kingdom of\n",
      ": Heaven through the saving grace of God.  I never said that the Law was made\n",
      ": obsolete by Jesus.\n",
      "\n",
      "Just for reference, here's the earlier quote:\n",
      "Malcolm Lee (mlee@post.RoyalRoads.ca) wrote in reference to Leviticus 21:9\n",
      "and Deuteronomy 22:20-25:\n",
      ": These laws written for the Israelites, God's chosen people whom God had\n",
      ": expressly set apart from the rest of the world.  The Israelites were a\n",
      ": direct witness to God's existence.  To disobey God after KNOWing that God\n",
      ": is real would be an outright denial of God and therefore immediately\n",
      ": punishable.\n",
      ": Remember, these laws were written for a different time and applied only to\n",
      ": God's chosen people.  But Jesus has changed all of that.  We are living in\n",
      ": the age of grace.  Sin is no longer immediately punishable by death.  There\n",
      ": is repentance and there is salvation through our Lord Jesus Christ.  And\n",
      ": not just for a few chosen people.  Salvation is available to everyone, Jew\n",
      ": and Gentile alike.\n",
      "\n",
      "These are two conflicting statements.  To say one is a clarification of the\n",
      "other is a breach of logic.  I don't mind people shifting their position on\n",
      "an issue.  It irritates me when it is said under the premise that no change\n",
      "was made.  What about Deuteronomy 22:20-25?  Is it wrong now?  Did Jesus\n",
      "change that?\n",
      "\n",
      ": If anything, He clarified the Law such as in that quote you made.  In the\n",
      ": following verses, Jesus takes several portions of the Law and expounds upon\n",
      ": the Law giving clearer meaning to what God intended.\n",
      "\n",
      "Sure he does this.  However, he doesn't address the notion of stoning\n",
      "non-virgin brides, because this needs no clarification.  Are you going to\n",
      "deny that Deuteronomy 22:20-25 is not patently clear in its intent?\n",
      "\n",
      ": I think you will agree with me that there are in today's world, a lot of\n",
      ": modern-day Pharisees who know the bible from end to end but do not believe\n",
      ": in it.  What good is head knowledge if there is nothing in the heart?\n",
      "\n",
      "I'll agree that there is a lot of modern day Pharisees that know the Bible\n",
      "from end to end and don't believe in it.  Depending on how they use this\n",
      "knowledge, they can be scary.  They can argue any position they desire, and\n",
      "back it up with selected parts of the Bible.  Such Pharisees include David\n",
      "Koresh and Adolph Hitler.  I will qualify this by saying *I don't know* if\n",
      "they actually believed what they were preaching, but the ends certainly\n",
      "made the means look frightening.\n",
      "\n",
      "However, just as scary are those that don't know much of the Bible, but\n",
      "believe every word.  In fact, this is probably scarier, since there are far\n",
      "more of these people, from what I've seen.  In addition, they are very easy\n",
      "to manipulate by the aforementioned Pharisees, since they don't know enough\n",
      "to debate with these people.\n",
      "\n",
      ": Christianity is not just a set of rules; it's a lifestyle that changes one's\n",
      ": perspectives and personal conduct.  And it demands obedience to God's will.\n",
      "\n",
      "No, it demands obedience to a book.  If God came down and personally told\n",
      "me how I should behave, then I would say that I would be doing God's will\n",
      "by doing it.  However, if preachers, pastors, and evangelists tell me to\n",
      "obey the will of a book written by people who have been dead for close to\n",
      "two millenia (even longer for the OT), even if I follow everything in it\n",
      "with my heart, I could scarcely be honest with myself by saying I'm doing\n",
      "the will of God.\n",
      "\n",
      ": Some people can live by it, but many others cannot or will not.  That is\n",
      ": their choice and I have to respect it because God respects it too.\n",
      "\n",
      "Well, if God respects it so much, how come there is talk in the Bible about\n",
      "eternal damnation for non-believers?  I see little respect eminating from\n",
      "the god of the Bible.  I see a selfish and spiteful god.\n",
      "\n",
      ": God be with you,\n",
      "\n",
      "Not yours, thanks ;)\n",
      "\n",
      ": Malcolm Lee  :)\n",
      "\n",
      "Rob Lanphier\n",
      "lanphi872@snake.cs.uidaho.edu\n",
      "lanph872@uidaho.edu\n",
      "\n",
      "And for the curious, here is my earlier post:\n",
      "> Hmm, for a book that only applied to the Israelites (Deuteronomy), Jesus sure\n",
      "> quoted it a lot (Mt 4: 4,7,10).  In addition, he alludes to it in several\n",
      "> other places (Mt 19:7-8; Mk 10:3-5; Jn 5:46-47).  And, just in case it isn't\n",
      "> clear Jesus thought the Old Testament isn't obsolete, I'll repeat the\n",
      "> verse in Matthew which gets quoted on this group a lot:\n",
      "> \n",
      "> \"Do not think that I have come to abolish the Law or the Prophets; I have\n",
      "> not come to abolish them but to fulfill them.  I tell you the truth, until\n",
      "> heaven and earth disappear, not the smallest letter, not the least stroke\n",
      "> of a pen, will by any means disappear from the Law until everything is\n",
      "> accomplished.  Anyone who breaks one of the least of these commandments\n",
      "> and teaches others to do the same will be called least in the kingdom of\n",
      "> heaven, but whoever practices and teaches these commands will be called\n",
      "> great in the kingdom of heaven.  For I tell you that unless your\n",
      "> righteousness surpasses that of the Pharisees and the teachers of the law,\n",
      "> you will certainly not enter the kingdom of heaven.\"  (Mt 5:17-20 NIV, in\n",
      "> pretty red letters, so that you know it's Jesus talking)\n",
      " \n",
      "> This causes a serious dilemma for Christians who think the Old Testament\n",
      "> doesn't apply to them.  I think that's why Paul Harvey likes quoting it so\n",
      "> much ;).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# display a random document and label\n",
    "idx = round(np.random.rand()*len(bunch.data))\n",
    "print('--------Random Document---------')\n",
    "print('================================')\n",
    "print('Document Label: ',bunch.target_names[bunch.target[idx]])\n",
    "print('================================')\n",
    "print(\"\\n\".join(bunch.data[idx].split(\"\\n\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 134142 unique tokens. Distilled to 134142 top words.\n",
      "Shape of data tensor: (11314, 1000)\n",
      "Shape of label tensor: (11314, 20)\n",
      "134142\n",
      "CPU times: user 4.77 s, sys: 577 ms, total: 5.35 s\n",
      "Wall time: 6.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "NUM_TOP_WORDS = None # use entire vocabulary!\n",
    "MAX_ART_LEN = 1000 # maximum and minimum number of words\n",
    "\n",
    "#tokenize the text\n",
    "tokenizer = Tokenizer(num_words=NUM_TOP_WORDS)\n",
    "tokenizer.fit_on_texts(bunch.data)\n",
    "# save as sequences with integers replacing words\n",
    "sequences = tokenizer.texts_to_sequences(bunch.data)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "NUM_TOP_WORDS = len(word_index) if NUM_TOP_WORDS==None else NUM_TOP_WORDS\n",
    "top_words = min((len(word_index),NUM_TOP_WORDS))\n",
    "print('Found %s unique tokens. Distilled to %d top words.' % (len(word_index),top_words))\n",
    "\n",
    "X = pad_sequences(sequences, maxlen=MAX_ART_LEN)\n",
    "\n",
    "y_ohe = keras.utils.to_categorical(bunch.target)\n",
    "print('Shape of data tensor:', X.shape)\n",
    "print('Shape of label tensor:', y_ohe.shape)\n",
    "print(np.max(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that's it! The representation is now:\n",
    "- each word is converted to an integer \n",
    "- each article is a series of integers that represent the correct ordering of words\n",
    "- the target is one hot encoded\n",
    "\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train Shape: (9051, 1000) Label Shape: (9051, 20)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAF3CAYAAABKeVdaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABQ9klEQVR4nO2dedzlY/3/n6+ZZB9DljDGIOk7sjZE/CrRSrY2vllCUfky0oK+FfL1JaGir1DSIKTsJCRLGDRjm8aSvUSoLGNJlvfvj/d17vvc95zl+nw+Z7ln7vfz8TiP+z6f+7zPdZ1zn3O9r+u9yswIgiAIAoAx/Z5AEARBMHIIpRAEQRAMEEohCIIgGCCUQhAEQTBAKIUgCIJggFAKQRAEwQBdVQqSHpY0S9Ltkmaka0tJulLSfennknWPP0jS/ZLulfTBbs4tCIIgmJtenBQ2M7N1zWxKun8gcJWZrQ5cle4jaTKwA7Am8CHgBEljezC/IAiCIPGGPoy5DfDe9Ps04BrggHT9bDN7GXhI0v3AhsD0Zk+09NJL26RJk7o51yAIgvmOmTNn/t3Mlmn0t24rBQOukGTASWZ2MrCcmT0OYGaPS1o2PXZF4KY62UfTtaZMmjSJGTNmdGHaQRAE8y+SHmn2t24rhU3M7LG08F8p6Z4Wj1WDa3PV4JC0J7AnwMSJEzszyyAIggDosk/BzB5LP58EzsfNQU9IWh4g/XwyPfxRYKU68QnAYw2e82Qzm2JmU5ZZpuHpJwiCIChJ15SCpEUlLV77HfgA8EfgImDX9LBdgQvT7xcBO0haUNIqwOrALd2aXxAEQTA33TQfLQecL6k2zplm9htJfwDOkbQH8GfgEwBmNlvSOcBdwKvA3mb2WhfnFwRBEAyja0rBzB4E1mlw/R/A5k1kDgcO79acgiAIgtZERnMQBEEwQCiFIAiCYIBQCkEQBMEA/choDoJRw6QDLy0s8/CRW3ZhJkGQRyiFYFRQdHGuX5iryAbFife7v4RSCOYJRutC0a/XHSec0UsohT4wr37h5tV5j1b6qUhHqxKfHwilMA8SX7ggCLpFRB8FQRAEA8RJoSSxWy9OvGdBMPKJk0IQBEEwQJwURhmxWw+CoBVxUgiCIAgGCKUQBEEQDBBKIQiCIBgglEIQBEEwQDiagyAIEhGIkXFSSL2Wx6Tf3yppa0kLdH9qQRAEQa/JMR9dBywkaUXgKmA34GfdnFQQBEHQH3KUgszsRWB74Hgz2w6Y3N1pBUEQBP0gSylI2hj4NFAzuIUvIgiCYD4kRynsBxwEnG9msyWtClzd1VkFQRAEfaHtjt/MrgWuBUgO57+b2b7dnlgQBEFRoudHdXKij86UNE7SosBdwL2Svtr9qQVBEAS9Jsd8NNnMngO2BX4NTAR27uakgiAIgv6QoxQWSHkJ2wIXmtkrgHV1VkEQBEFfyFEKJwEPA4sC10laGXium5MKgiAI+kOOo/k44Li6S49I2qx7UwqCIAj6RVa+gaQtgTWBheouf7srMwqCIAj6Rk700YnAp4B9AAGfAFbu8ryCIAiCPpDjU3iXme0CPG1mhwIbAyt1d1pBEARBP8hRCi+lny9KWgF4BVile1MKgiAI+kWOT+ESSeOB7wK34uGoP+nmpIIgCIL+kBN9dFj69VxJlwALmdmz3Z1WEARB0A+aKgVJ27f4G2Z2Xs4AksYCM4C/mtlWkpYCfgFMwvMfPmlmT6fHHgTsAbwG7Gtml2e+jiAIgqADtDopfLTF3wzIUgrAVOBuYFy6fyBwlZkdKenAdP8ASZOBHfDQ1xWA30p6q5m9ljlOEARBUJGmSsHMdqv65JImAFsChwP7p8vbAO9Nv08DrgEOSNfPNrOXgYck3Q9sCEyvOo8gCIIgj6bRR5L2l7RHg+v7SNov8/m/D3wNeL3u2nJm9jhA+rlsur4i8Je6xz2argVBEAQ9olVI6u7A6Q2un5z+1hJJWwFPmtnMzLmowbW5Cu9J2lPSDEkznnrqqcynDoIgCHJopRTMzP7d4OLLNF7Ah7MJsLWkh4GzgfdJOgN4QtLyAOnnk+nxjzI0KW4C8FiD8U82sylmNmWZZZbJmEYQBEGQS8vkNUnL5VxrhJkdZGYTzGwS7kD+nZntBFwE7JoetitwYfr9ImAHSQtKWgVYHbgl61UEQRAEHaGVUvgucKmk90haPN3eC1wMHF1hzCOB90u6D3h/uo+ZzQbOwbu7/QbYOyKPgiAIekur6KPTJD2FV0N9O27fnw0cbGaXFRnEzK7Bo4wws38Amzd53OF4pFIQBEHQB1pmNKfFv5ACCIIgCOZdcgriBUEQBKOErCY78yuTDry00OMfPnLLLs0kCIJgZBAnhSAIgmCAticFSQsCH8ML2A083syiHWcQBMF8Ro756ELgWWAm8HJ3pxMEQRD0kxylMMHMPtT1mQRBEAR9J0cp3ChpLTOb1fXZBEEQzKMUDVyBkRm80qrJziw8Ye0NwG6SHsTNR8LrIq3dmykGQRAEvaLVSWGrns0iCIIgGBE0DUk1s0fM7BFgeeCfdff/Cby5VxMMgiAIekdOnsKPgOfr7r+QrgVBEATzGTlKQWY20OzGzF5nlGdCB0EQzK/kKIUHJe0raYF0mwo82O2JBUEQBL0nRyl8HngX8Fe8O9o7gT27OakgCIKgP7Q1A5nZk3jntCAIgmA+J6f20ULAHsCawEK162a2exfnFQRBEPSBHPPR6XgI6geBa4EJwJxuTioIgiDoDzlK4S1m9k3gBTObBmwJrNXdaQVBEAT9IEcpvJJ+PiPp7cASeBntIAiCYD4jJ9/gZElLAt8ALgIWA77V1VkFQRAEfSEn+ugn6dfrgFW7O50gCIKgnzQ1H0n6ft3vU4f97Wfdm1IQBEHQL1r5FN5d9/uuw/4WZbODIAjmQ1opBTX5PQiCIJhPaeVTGJMczGPqfq8ph7Fdn1kQBEHQc1ophSWAmQwqglvr/mZzPzwIgiCY12mqFMxsUg/nEQRBEIwAcpLXgiAIglFCKIUgCIJggFAKQRAEwQBZbTUljQWWq3+8mf25W5MKgiAI+kNOP4V9gIOBJ4DX02UjEtiCIAjmO3JOClOBNczsH92eTBAEQdBfcnwKfwGeLfrEkhaSdIukOyTNlnRour6UpCsl3Zd+Llknc5Ck+yXdK+mDRccMgiAIqpFzUngQuEbSpcDLtYtmdmwbuZeB95nZ85IWAK6XdBmwPXCVmR0p6UDgQOAASZPxXtBrAisAv5X0VjN7rfjLCoIgCMqQc1L4M3Al8EZg8bpbS8x5Pt1dIN0M2AaYlq5PA7ZNv28DnG1mL5vZQ8D9wIZ5LyMIgiDoBDn9FA4t++Qpamkm8Bbg/8zsZknLmdnj6bkfl7RseviKwE114o+ma0EQBPM9kw68tNDjHz5yy67Mo6lSkPR9M9tP0sU0qHVkZlu3e/Jk+llX0njg/NTOs+mQjZ6iwbz2BPYEmDhxYrspBEEQBAVodVI4Pf08uuogZvaMpGuADwFPSFo+nRKWB55MD3sUWKlObALwWIPnOhk4GWDKlClRmC8IgqCDtCqINzP9vLbME0taBnglKYSFgS2A7+B9nncFjkw/L0wiFwFnSjoWdzSvDtxSZuwgCIKgHFkZzSVZHpiW/ApjgHPM7BJJ04FzJO2BO7E/AWBmsyWdA9wFvArsHZFHQRAEvaVrSsHM7gTWa3D9H8DmTWQOBw7v1pyCIAiC1hQqiCdpjKRx3ZpMEARB0F/aKgVJZ0oaJ2lR3LRzr6Svdn9qQRAEQa/JOSlMNrPn8CSzXwMTgZ27OakgCIKgP+QohQVSmYptgQvN7BWiR3MQBMF8SY5SOAl4GFgUuE7SysBz3ZxUEARB0B9yylwcBxxXd+kRSZt1b0pBEARBv8hxNC8n6ZRU4ZRUzXTXrs8sCIIg6Dk55qOfAZfjWcYAfwL269J8giAIgj6SoxSWNrNzSK04zexVIDKNgyAI5kNylMILkt5EijiStBElOrEFQRAEI5+cMhf748XqVpN0A7AM8PGuzioIgiDoCznRR7dKeg+wBt7z4N6UqxAEQRDMZ+REH+0NLGZms83sj8Bikr7Y/akFQRAEvSbHp/A5M3umdsfMngY+17UZBUEQBH0jRymMkTTQKjP1R3hj96YUBEEQ9IscR/PleFOcE/EIpM8Dv+nqrIIgCIK+kKMUDgD2Ar6AO5qvAH7SzUkFQRAE/SEn+uh14EfpFgRBEMzHtFUKkjYBDgFWTo8XYGa2anenFgRBEPSaHPPRKcCXgJlEeYsgCIL5mhyl8KyZXdb1mQRBEAR9J0cpXC3pu8B5wMu1i2Z2a9dmFQRBEPSFHKXwzvRzSt01A97X+ekEQRAE/SQn+ii6rAVBEIwSck4KSNoSWBNYqHbNzL7drUkFQRAE/SGnIN6JwKeAffBw1E/g4alBEATBfEZO7aN3mdkuwNNmdiiwMbBSd6cVBEEQ9IMcpfBS+vmipBWAV4BVujelIAiCoF/k+BQukTQe+C5wKx55FLWPgiAI5kNylMJRZvYycK6kS3Bn87+6O60gCIKgH+SYj6bXfjGzl83s2fprQRAEwfxD05OCpDcDKwILS1oPjzwCGAcs0oO5BUEQBD2mlfnog8BngAnAMQwqhTnA17s7rSAIgqAfNFUKZjYNmCbpY2Z2bg/nFARBEPSJHJ/CBEnj5PxE0q2SPtD1mQVBEAQ9J0cp7G5mzwEfAJYFdgOObCckaSVJV0u6W9JsSVPT9aUkXSnpvvRzyTqZgyTdL+leSR8s+ZqCIAiCkuQohZov4SPAqWZ2R921VrwKfNnM/gPYCNhb0mTgQOAqM1sduCrdJ/1tB7zG0oeAEySNLfJigiAIgmrkKIWZkq7AlcLlkhYHXm8nZGaP13oumNkc4G48mmkbYFp62DRg2/T7NsDZKez1IeB+YMMCryUIgiCoSE7y2h7AusCDZvaipDfhJqRsJE0C1gNuBpYzs8fBFYekZdPDVgRuqhN7NF0b/lx7AnsCTJw4scg0giAIgjbk9FN4XdITwGRJWaW265G0GHAusJ+ZPSc1tTw1+oM1mM/JwMkAU6ZMmevvQRAEQXnaLvKSvoOXzr4LeC1dNuC6DNkFcIXwczM7L11+QtLy6ZSwPPBkuv4oQ6uvTgAey3oVQRAEQUfI2flvC6yR6h9lIz8SnALcbWbH1v3pImBXPIJpV+DCuutnSjoWWAFYHbilyJhBEARBNXKUwoPAAkAhpQBsAuwMzJJ0e7r2dVwZnCNpD+DPeNMezGy2pHPwE8mrwN5m9tpczxoEQRB0jRyl8CJwu6SrqFMMZrZvKyEzu57moaubN5E5HDg8Y05BEARBF8hRChelWxAEQTCfkxN9NK3dY4IgCIL5g1als2fRICS0hpmt3ZUZBUEQBH2j1Ulhq57NIgiCIBgRtCqd/UgvJxIEQRD0n5zaR0EQBMEoIZRCEARBMEBTpZDyEmplLoIgCIJRQCtH8/KS3gNsLelshiWi1cpiB0EQBPMPrZTCt/AGOBOAY4f9zYD3dWtSQRAEQX9oFX30K+BXkr5pZof1cE5BEARBn8jJaD5M0tbAu9Ola8zsku5OKwiCIOgHbaOPJB0BTMWrl94FTE3XgiAIgvmMnIJ4WwLrmtnrAJKmAbcBB3VzYkEQBEHvyc1TGF/3+xJdmEcQBEEwAsg5KRwB3Cbpajws9d3EKSEIgmC+JMfRfJaka4ANcKVwgJn9rdsTC4IgCHpPzkkBM3ucaLQTBEEw3xO1j4IgCIIBQikEQRAEA7RUCpLGSPpjryYTBEEQ9JeWSiHlJtwhaWKP5hMEQRD0kRxH8/LAbEm3AC/ULprZ1l2bVRAEQdAXcpTCoV2fRRAEQTAiyMlTuFbSysDqZvZbSYsAY7s/tSAIgqDX5BTE+xzwK+CkdGlF4IIuzikIgiDoEzkhqXsDmwDPAZjZfcCy3ZxUEARB0B9ylMLLZvbv2h1Jb8A7rwVBEATzGTlK4VpJXwcWlvR+4JfAxd2dVhAEQdAPcpTCgcBTwCxgL+DXwDe6OakgCIKgP+REH72eGuvcjJuN7jWzMB8FQRDMh7RVCpK2BE4EHsBLZ68iaS8zu6zbkwuCIAh6S07y2jHAZmZ2P4Ck1YBLgVAKQRAE8xk5PoUnawoh8SDwZJfmEwRBEPSRpkpB0vaStsfrHv1a0mck7YpHHv2h3RNL+qmkJ+urrEpaStKVku5LP5es+9tBku6XdK+kD1Z8XUEQBEEJWp0UPppuCwFPAO8B3otHIi3ZXGyAnwEfGnbtQOAqM1sduCrdR9JkYAdgzSRzgqQopREEQdBjmvoUzGy3Kk9sZtdJmjTs8ja4YgGYBlwDHJCun21mLwMPSbof2BCYXmUOQRAEQTFyoo9WAfYBJtU/vmTp7OVSv2fM7HFJtXIZKwI31T3u0XSt0Xz2BPYEmDgx2jwEQRB0kpzoowuAU3BfwutdmocaXGuYC2FmJwMnA0yZMiXyJYIgCDpIjlL4l5kd16HxnpC0fDolLM9gFNOjwEp1j5sAPNahMYMgCIJMckJSfyDpYEkbS1q/dis53kXArun3XYEL667vIGnBZK5aHbil5BhBEARBSXJOCmsBOwPvY9B8ZOl+UySdhTuVl5b0KHAwcCRwjqQ9gD8DnwAws9mSzgHuAl4F9jaz1wq/miAIgqASOUphO2DV+vLZOZjZjk3+tHmTxx8OHF5kjCAIgqCz5JiP7gDGd3keQRAEwQgg56SwHHCPpD8AL9culgxJDYIgCEYwOUrh4K7PIgiCIBgR5PRTuLYXEwmCIAj6T05G8xwGE8neCCwAvGBm47o5sSAIgqD35JwUFq+/L2lbvC5REARBMJ+RE300BDO7gDY5CkEQBMG8SY75aPu6u2OAKTSpSxQEQRDM2+REH3207vdXgYfxUtdBEATBfEaOT6FSX4UgCIJg3qGpUpD0rRZyZmaHdWE+QRAEQR9pdVJ4ocG1RYE9gDcBoRSCIAjmM1q14zym9rukxYGpwG7A2cAxzeSCIAiCeZeWPgVJSwH7A5/Geyqvb2ZP92JiQRAEQe9p5VP4LrA93vpyLTN7vmezCoIgCPpCq+S1LwMrAN8AHpP0XLrNkfRcb6YXBEEQ9JJWPoXC2c5BEATBvE0s/EEQBMEAoRSCIAiCAUIpBEEQBAOEUgiCIAgGCKUQBEEQDBBKIQiCIBgglEIQBEEwQCiFIAiCYIBQCkEQBMEAoRSCIAiCAUIpBEEQBAOEUgiCIAgGCKUQBEEQDBBKIQiCIBgglEIQBEEwQCiFIAiCYIBQCkEQBMEAI04pSPqQpHsl3S/pwH7PJwiCYDQxopSCpLHA/wEfBiYDO0qa3N9ZBUEQjB5GlFIANgTuN7MHzezfwNnANn2eUxAEwahBZtbvOQwg6ePAh8zss+n+zsA7zey/6h6zJ7BnursGcG8XprI08Pd5THa0jh3zHj1jx7w7x8pmtkyjP7yhwwNVRQ2uDdFaZnYycHJXJyHNMLMp85LsaB075j16xo5594aRZj56FFip7v4E4LE+zSUIgmDUMdKUwh+A1SWtIumNwA7ARX2eUxAEwahhRJmPzOxVSf8FXA6MBX5qZrP7MJUq5ql+yY7WsWPeo2fsmHcPGFGO5iAIgqC/jDTzURAEQdBHQikEQRAEA4RSCIIgCAYIpRAE8wCSNsm5FgRVCaUASBovaV9Jx0o6rnYrIH+UpHGSFpB0laS/S9opU3ZvSePr7i8p6YuZsstJOkXSZen+ZEl75M47yawsaYv0+8KSFs+Um6smlaT3Fhk7ySxaQmZqer+VXv+tkj5QQP4Ttdcp6RuSzpO0fhuZ/VvdMsfdSlLZ79zxmdeajV3mNS/V6pY5riTtJOlb6f5ESRvmzjvJjJW0QpKdKGliptxqkhZMv783fcfHZ8p+J+daC/lVJC1Ud39hSZMyZRet/5xIGiNpkdyxqxJKwfk1MAmYBcysu+XyATN7DtgKT8B7K/DVTNnPmdkztTtm9jTwuUzZn+Hhuyuk+38C9suURdLngF8BJ6VLE4ALMsXPkXRA+tIvLOl44IgCY79L0l3A3en+OpJOyBTfPb3fHwCWAXYDjswdG/immc2RtCnwQWAa8KM2Mou3ueWwA3Bf2kT8R46ApI0lfRlYZpgiOgQP286lzGueCcxIP5/CP1/3pd9zvx8nABsDO6b7c/Cil1lI2gd4ArgSuDTdLskUPxd4TdJbgFOAVYAzM2Xf3+DahzNlAX4JvF53/7V0LYergHolsAjw2wJjV2JE5Sn0kYXMLGu314QF0s+PAGeZ2T+lRhU7GjJGkizFBssrxb4xU3ZpMztH0kEwkOfxWoF5740XIbw5yd8nadlM2XcC3wFuxBfFnwNFzBnfwxeni9LYd0h6d6Zs7c39CHBqks1+w/EvKMCWwI/M7MK0yDbFzA4t8PzNnmMnSePwBfJUSQacin9m5jQReyOwGP5drVc+zwEfLzB8mde8CoCkE4GLzOzX6f6HgS0yx32nma0v6bb0nE/LE1NzmQqsYWb/KCBT4/X0ndgO+L6ZHV+bRzMkfQH4IrCqpDvr/rQ4cEOBsd+QinoCYGb/LvC6FzKz5+tkn+/lSSGUgnN62jVfArxcu2hm/8yUv1jSPcBLwBclLQP8K1P2cnzXfSJe5+nzwG8yZV+Q9KYkh6SNgGczZQFeTh9WkvwbGFZrqgWv4K93YWAh4CEze721yFDM7C/D1vJchTZT0hX4zu+gZBYpMvZfJZ2EL2zfSSaGlqdmtTEnmtm+OQOb2XOSzsXft/2A7YCvSjrOzOYyB5nZtcC1kn5mZo8kpWItlEgzCr/mOjYws8/XzekySYdlyr6SNjq1z+gyFPtf/YVin+nhY+8I7Ap8NF1boMXjwU8Sl+Gn3vp+LnMKrAcAT0na2swuApC0DflF7V6QtL6Z3Zpk34F/13pCJK/hdn3gcOAZBhdFM7NVCzzHksBzZvaa3E6+uJn9LUNuDF71dQt8B3wF8BMza7tAJpvw8cDbgT/ippSPm9mdLQUH5Y/CX/MuwD74DukuM/vvDNk7gAuBw4A34SaoV8wsa/cq6VfAscAPgY2AfYEpZrZDhuwYYF3gQTN7JinGFQu87kWADwGz0uloeWAtM7uihcyurZ7TzKZljPtRYHdgNeB0YJqZPZnmc7eZrdxCdgp+qqidFp7FzWhZZpwyr7lO9nLg98AZ+PdjJ+DdZvbBDNlPA58C1sdNVh8HvmFmWaYUSafg1ZAvZeiG7dgM2cn4Jmu6mZ0laRXgU2bW1tQoaTXgUTN7We4rWxs4rd7UmyH/c9y0K1y57WJm92fIboC3DajVfVs+zbuISbs0oRQASQ/gx9xS5WmTUvl57QOTFMSOZpZrI689z1LAhJzFLe2+9sWVwhr4B+9eM3ulwHgCPovb5oWfWn5iGR8KSVPMbMawazub2emZYy8N/IChynBqrplA0tZAzdx0rZldnCNXJ78psLqZnZp2r4uZ2UNFnqMokk7D39/rGvxtczO7qoXsncDeZvb7dH9T4AQzWztz7I2A2bUTRjpdTTazmzNklwIOxt9vA64Dvp27c5b0NmBz/P98lZndnSOXZA9udL0T5rw2494OTMF9jZfjZs41zOwjBZ9nMXydLXSyk7QAg9/re4p8rytjZqP+hv/DF6kgf3uDa7dlyl4DjAOWAv6MO/COzZWtMOcxwB/7/d6XnPuRuDNu93S7EjiigPzBwMXAn9L9FYAbMmWXAY7GgxN+V7sVGHtlYIv0+8L4iTJHbq755c45PfY20iaw7v9/a8H3fbES/6uN6l8jftJ5Z48+J5ukz8afgAeBh/DTZY7srenn14B9au9hgbGnpu+1gJ8At+IBKTmyn6i9Z8A3gPOA9XvxnplZ+BQSrwG3S7qaoUfULDsx1ZzFS5jbmT+LO00PHubgasUNkn4I/AJ4oW7et7YTNLPXJd0haaKZ/TlzvJZIusTMtmrzmONp4bfIfM8/AqxryYchaRq+6B2UOdXtgPXwLypm9pgyQ3Fxk8AvcIft53F79VM5gslvtSe+AVgNj/Y6Ed9Ft+OW5BM4C3//PgVck0yIOf/zgc9nevzryYeUM+934QvbYsBESesAe5lZTuj0j3DTUY0XGlxrNfYy+MK8Ju67qs3/fRnipwBfwjdaRQIwYNAfsQv5/oh6djezH0j6ILAsHiF3Kn4ibsc3zeyXGowUOxp/z95ZYPzShFJwLiA/FLMRVZzFb0j23U8CbW35w3hX+vntumsG5HxhwG2VsyXdwlClsnXBedTICaWd0f4hWYwHauaLJQrK/tvMLEX/FM2VeJOZnSJpqg06ga/NlK0S7bVu+jncnPIu8v7nD0ral8Ew1C/iu+ccKkWKlVVGiZoS3oqCShh41swuKzBWPbul8Q43s4eSP+KMAvJVIuQKR4p1klAKDHUSJn/ASpbptEwcAOwFfIE6Z3Gm7LdxpXK9mf1B0qp4LHjOvDcrMMdGdMQuW+Q9swyHbAZHALelk51wW/fXC8ifk3bd49PufXfgx5myNdvu45K2xJ2BEzJlq0R7bWEZwQct+DxwHG6OMNz8tmdLiTqsfKRYFWUE1ZTw1ZK+i5tf6i0AOSfpu3CfXe3+QxTLhakSIVclUqwy4WgGJF0DbI0rydvxnci1Vi13oSekhWn40frbzSU6Nu41VHjPklngAGAyxc0CpNPVBrhSuNkyIr2Gyb+fOge7mV2ZKbcVHomzEu7kHwccain0sI1slWivh/BEw59aAUdtJ6gYKbYsrozex6Ay2s/Mnswc+yYz2yhFQB2HK+FfmdlqGbJXN7hsOZ8xSavjm4/hn8+siMQqEXJVIsU6QSgFQNJtZrZesuuvVLPrW5vIDknnmNknJc2iwY6vlbykr5nZUc1s7Dm29WSuWgTYDD+ZfBy4xcyySl2kiJTjgf/AfSBjgRfMbFyGbKn3rE7+Ctws8BXqzAJmdkCG7FVmtnm7ayONtFDswVBllHVCSTvNHXCzxhjgp8DZ5pndreQ68TmrFClWhSpKuOK41+Omuu/hPoXd8PWyYTRUndzbzOweNSkh0uqUImlc8i82LCFixfIkShPmI6esXX9q+tnSudqE2m6vio39XWa2dlqMD5V0DH5UzuWH+ELzSzz8bhdg9UzZKr4QKGEWkNeSWQRYOpmsavaMcQyW+mglPwdfGMXQBVL4DjJHGa6KL5Ab4+aA6cCXzCzHJHKImX2LZKqS1/T5uZl9up2geUjjj4EfJ3v+WcD30i7+MGse/175c2Yeqt12jo1I/7M9mPs0u3vm2LWSFs/im58iY3+ryXPmnKQXNrOrUgDJI8Ahkn7P3D6d4eyPm+WOaTQ0rX0/Z+JryUwGP6f1stl5U1UIpeCUsuub2ePp5yNFB7QUV1/Rxl7LcnxR0grAP3AbZpF53C9pbLJXnyrpxkzR0r6QRBnb/F54FvAKpMihxHNk1NMxs9wIo1acmcbaLt3fAV+gcyJDJko6yMyOkJc8+CUeNdUWeUTblviOdRK+6Pwc+H94eOxbG8l14nOWnKz7pHEH1ozMgITTgXtwR/W3ceXS1vzViRMOdcETuELaKmfsxL/Sye4+eYvgv+JRRC0xsz3Tz8L+PkuRe5bKi/SLMB91AEnb43WAlsW1e5Gd5xR8p70yQ79wbc0wkr6JH6k3xxcqA36cdqM5874ONwn8BPgb8DjwGTNbJ0N2ITPLLeXRSL6KbX4fa1ASosDYwxO5FgPWtLxErpvN7J3Drt1kZhtlyApfyGfhu97LzOx7mXN+ELgaOMXMbhz2t+PaLZKS3oqb6iYx9HOWY1+/Aw/vnEWdszSd8NrJ1syMd6ZT7QK42azluJI+amYXq0kmeRkllxy2F1leJvYGuAIZj2ftjwO+a2Y3ZY5VU+KTGPp+t83ETvJrN5AtYgUozahWCh3ajSDpfuCjZRyAku7FK6oO/8IVOn2kD/xCZpZdJ0bSyngFyjfi8dxL4FmyOan49yfZ3+MZrjcUGbsKknZpdN3MTsuUvw1PBqqFpI4BZphZ09j5Ojvv13Bn8dkM5gssaGZNawENsy8vgJcEuQFfaNtGw6QF5r+rBBCkhf1EhsXsW0bphEaKsMC4t5jZhmkD8kV883FLAYftJ2xYSYxG1zKfa8k0dq6JFEmLmtkL7R85l9yv8fpnw7/XbSP+JP0UL6sxu07Wck1uVRntSqEjuxFJN5hZqYYnkq43s01LytYW5N/ji3LRVPr3ATeZ2Yslx5+Imy82weOxnzGzdTNljwL+BzeB/QZYB49KaRsLnpR4jYXwk9Ktll936fbh82znJJdH/wy389awVoucGkfB1Mvm7NavLmOSqJOfaWbvKCn7n7iv6QoKhnamQIRz8UXuVDwB7ptmdlJLwUH5W4cr60bXmsjWB4CMxbPRv21mP8yQ3RhX2ouZWdGEvbafpzayd5nZXP1KesWoVgrDKborSGYjgPcAb8YT4Oq/NG2Pe5I2x0spX1VCdlVgU3xh3ijJ/97MvpQ5/9OS3D9wxfJ73EfwdIbshDTue/AF/Z9JNqunQm1hlpc13hY/qVydY7pq8FxLAKdn2riRdB5eXqQ+dn4zM9u26Ni9QtLh+EmucPZ6kj8EeBI4n4KVgCUdAewMPMDQnWtukmRh5OW5P4IHMvyi7k/j8JpNbRv1pJNwjVeBJ8zs1czxb8aj+S4ys/XStT+a2dsz5b+D13kqHEYqLwJ4jHmuRM8JRzNDdwUUS+P/aN3vL+KhhjWMvEig3YC34WaFgS9cjqyZPSjpJeDf6bYZHl6ahZntApCc1B/H/RIrkPe5+DPwB+B/ra6scgGq9KAYzovkR01BhUQueQz5/sBEM9tTHs++hg1GybSS/V/gKBtaOPHLZvaNjKGrZq/XTsP1zZ9yI1q2A1a1uv4Aucjj8w/BT5OGbzwOs/bhrI/hEVNbM7Shzxx8A5HD8JPzOElzLLO4nJVP2AO4CTg/mSZfoYCfEa8mO13S33AFXpMtdfIoSpwUqL4rqDj2LDNbq6TsA3iN9jPxL9vtVqCngbxl6P8D1krPcz1+0pieIbsOfkp5NzARjzy61sxOyRz7CHyxeQkv/TAeuCTHdi3pYgbNAmPwBKNzzOzA5lJD5JfK2SE3kf0FvkjtYmZvl7QwXpp53QzZ22qfr7prWaaQfpJe8z6WmXA2TPZK3MRZMwt+GnivmWU16ZG0QG0RV8FqA5IexgMZnsYX1vF4MMWTeMfDpv4UVUjYS/IP4ifgWVZwkU3+uv2p6GcsjfWo8t5IvuEZsVBXBRG4o4D8qnjVzafwD9yFwCqZsj/Gj8Nl5j0VD2u8CW/NuRuwWgH5v+N1eHYDJpUYfzE88/Jw4BHg4Uy5MfjOd0lgbLq2KPDmTPn31N02wcuNF5n3fel9+zAMVg7NlJ1R9rMC3Ik7pWv3F8ajoHJk/xcYX3d/SeB/Csx7AXxh+1W6/RewQKbsNbh5sFZC+iJ8A5UjO7PZe1hg7LJVhE8EPlh3/wP4Qr9R7TvfQnZpPFLsifSdPgPPrcmd9+XAmCKfrTrZ7Kq73bjFSYGO7Apuwk0vZ6VLO+A7q5xd7914xcyHKHlUTCGVu+EhhxPMLLt3r6Q18d3+prgJ5l4z2zlDbgawIN6O83rgOiuwk5E03cw2zn18J0mhoVvgNY82xG3WPzOzP2XI3og7tm8wbzO5Gm7+yrFxfw03h5yKn3R2xxfXozJkb7MKpwxJP8EVQy14YmfgNTP7bIbsexpdt7yQ1KNxM9A56dLH8fDfdklgNfnbrGTmvKQZZjal0bVGwQadRNLP8M3iZRRvDnQCfqq5mIJ+xk4QPgXn83iW6orAo3iUxd4F5GVDm8ucIU94yeFDBcYZOqhnMG+K79inA9/CzUi58uNw08/KeEz0EuQX7fqwmeVWq2zEFZI+BpxnBXcmqlCeA1zj4nX2r5S0Gb4L/GIK2zzQWpvPDsajpVaSVOtL/ZnMcY+Sl0WvmU4OM7PLc2SBsZIWNLOXAZLZasFMWfCWmvVO/N+l19uWnMW/BXvhppDT8Q3PGLzd5P7k2dirZM7/U9IBePgwePjw0/IQ34afc3WmtDv4Ju8h/PNZpCc1+AnyZcr5KCsTSoFqafyJqyUdyNDY9UuVYtuthf26yO66ATfhjssnSspfX3f7oZk9WkD235KOpa77GR7ul5ursD9uMnotOcuLOOIaled4S+7Ek/NzJ3y3/ASerXsRXsDsl7TICjezKyXdip8ohdcAKtKx7zZ8x25kZjMnzgCuklR/yiiSwPWapNXM7AEYiFwrXXVV0smWsndbYdWzyKtkzv8nrsQvSPevT9fG4kqmEbVyIJvgvqpa5NMnGOrwbolV6AxnZruVle0EYT4C5BU7P8fcGYRZySIphr0ZZgV6Pafna9msRk2KbdUNmBWmWAV58/k/MtQcsY6Zbd9cqmNj10wAA2YESTea2bvayabH/gnfuZ46XBFKOsDMvtNCdhPcof9CctSvD/wgR7lL+iTwXdxOLtzJ/1Uz+1XmvD9EXVG6AqeMWk7Kz/Cy1cJPh7uZWascilbP9w7LS3xr9H593zrU2KlbyHNLPmCDTu4F8Pe8Sq7InmZ2cknZrSwjwq0TxEnBuRA3u/yWErsn63ytknbNamrFthbCd8p34F/0tXHHcctkuGHRO3NhefH+q5nZx+ruHyrva5uNhvZZvqbAh/5Fee2g2+VJcI/jp45c1mhmsmqlEBI/AtZJ0VdfxauVnoY7vdvx37gZ50kY2Iz8Fnf8tsXMfkN+86YBkrlkHdxnVN/39+WWgq3nkrtrrn+/voaHfp9Om/dLHao20OB5cxfmFfDWobVT/mJkFF1sN3wF2Q2AUAo9ZBHLKNncCklvZ+7a61llF+qeIyvkrrZbkXQ2sKeZzaqbw1cyhjo6/dweT7qrhQvuCDycOd2XJG1qZtensTdhsEBfWyQdiX/Qf54uTU3PlxNWujNum/4vPGZ9JeBjLSWGcoW8VMIzaS5L4mWo29bEAV41M5O0DXCceaXXhhnxDRhjQ8M6/0GF5ikFTDivSdravM5SdvMoSd83s/2abSIyNw/179cPCrxfnagi3IjchflIBhs5gSuxQ6oMbJlZ3E1ksxzznSDMR4Ck/wFuNLNfl5Q/GHgvrhR+jYc6Xm8ZZRdUoVlNowiKIlEVkq4zs3e3u9ZEdl3cdLQE/kX7J15ML8t5mRyu69pgn+WxeJhn1xN0mrxvc0X3NJG9Ft+t74afcp7CzSNtc03kXcDWZjBK7VPAnWU3JLkmnPTYwhnRteevGH1U5f0aCxxpZl9t99huIOnNDFa/LdTISdJUPMpsDl5wcj08iCErw1neF3sSQ83ZhTaZZRnVJwUNra//dUkvUzz7EDzMbh18UdtN0nLkt+NcwryxxmdxG/fBacHM4Z4UanhGeh07kV8aGGAZSata6gUgL5G8TI6gmd2OmwXGpfstm700YTzl+ywPQdIhZnZI5sNfkzSxZteWl0PI3R19CndW7mFmf5PXf/pujqCZfTVFXG2Cf8ZONrPzM8dt9HzZjk9KZETXPf8M4KVhCjw38qnK+/WapFL1mqDcwqy5m+T8Jf1cQdIKBfx1u5vZDyR9EP9O7Zbm0lYpSDodD1O/nUFztuFmyq4zqpVCByIjarxk3pD81bRIPkl+Q4wqIXefwftCT033r2Ownk8O+wHXyLMvwXcmLc0RKZSw0XUgvzQwjfssH5Qp24giC+TXges12NTn3WSWuUi7xWPr7v+ZAl9WMzsXLxCXRYdMOOCL8pBGQCmSJ4ercAf38+n+wvji1taxX/X9wj8jF+FRYfUnnJzwzDILc5UmOfXUzFQfwTd7d0jZdVym4AmtfTHjjGqlUEPV2zvOkDQez06eiX95bsmULRVyl3Zrl5iXC/he5lj18mPw3fnqeO0lyHM+1hTpGrhPoNb/4KO4Umo37iZmdgMec30Ng32WDyhyPB+OpWYyGePXXvf6DIaVfskyw0rVOEfieTNre9JRub4btfyXo1s8Jodf4a+5nl8COTvxhcysphAws+flNaBKkesLSSyF+17qF+PcmP3CC7NVaJIzjJnylrOrAAfJ26nm5gD9Eff1PV5xDuWwPqZT9/uGO4XfhEfvLIl/AJfCd8x3Zz6HcOdw7f4kYO0ic6gw/4tw81NZ+esqyF4BLF53f3HgNxlyM9PPWyuMPY25Sz78tEevewaeE3EbrhB2w4sC5sjeD/xHyXEXpa5sQhp7kQy5t+FO+AfwwILa7TPkl9i4Ae8/Ubs/Ba/3VPY9fEdZ2YLj1E4F9+FtXBenQdmNJrJjcV/fvvjpYX9g/wJjj8GV8Ph0/0256wLeTOlpSpQV6cRttJ8U6ts7zmRwZ5HV3hF8myfpAtKOy8weLjiHP0oq26zmX8AsedGx+qN1brjelZK+wtzOx5xicRPxyqw1/o0rxHa8Ik/AWlHSccP/mDn3tS1FDiWZpyW1dRLXUeV1Y+VbmD5hJRoxJcqacNbA21COZ2hV3zm0D32uMRX4paTH8F36CrivoBRWwBeiajlEe+AJiQ+a2YvypMXcxLCLadAkpwDb4DWMat/l13CTco6/8JAS43WMUa0UzOwHwA9Usb0jcJOkDczsDyXm8BYNNqvZCjhB0jOWF0F0abqVpfbFqi/pYeT5Q04HbpF0fpLZjjxb8Vb44vY+ivkB6hkjaUlLfR/kmeNFPstVXnfhHAkN9t2YIa84egHFa9qUMuGY2YXAhZI2tozqt01YBXfSTsT/zxvRxjHfQV9IlRyiKgvzBKsWCXew1QURmNkzKUrxgnaCZnZtClbZIF26xUpUqC1LhKQmVCHPQNJdeOP0R/CdZ3ZRO1VsVtNPUmRILVHuOjPLLtsgaR3LDF9tILsL7pT+Fb7gfBI43IbWn+oKKtHCNJ2MmmE5u15JN+BFFm9N96cAx1tmUUFJ0/CSHM+k+0vijVxyxq71V94Ur9Z6DPB1a1HwsRPhrOl5bs/cIGXJFgg9Lt0kJ8nPVbRPmWXyVTHzvSqhFKiWZ5DkV2503fJKH7zOYLOaC3PnnGRXx6N4hiuz7LIaFZXhWGA5hh7rS5cvUIFUfkmT8dOG8C9vdpcqecmCL1CXTQ2cZJnNV9JJ4W24QrrXSjSfKUpSAr/Am88MmHByTTGNFsMCC+Rt5pVKj8D7A5xZQHZRGoSzWmYLWFXIIaq4MG+Hh3qXaZKDvM/yM7gZ2vD6Wkua2WcyZO8A3m/DMt+tRFfCMpTOppzP+DheDvlv5sWo1qFYBco5DW6PZcquh5td/lPSdEmnSdojU/ZUPAT1VWCz9DzZu+WkDI9Pt82Ao3DnWo7sPviO+Uo8/f5Sqqfhb9D+IQMshVdGPR54Sp5jkcuPcB/QCen2DjJDeSVtiTttj8ML890vbx1ZCklNa1wNo2bC+QL+nt9Lfm4FJJNb3bhFTG5/lXQSfiL7taQFyV87rsKdvDUWxk1BLZE0R9JzuD/jEkkvSXqu7noOMyQdK2k1SatK+h75JstjgI1xZ/44M1s8VyEk9sH9bL/Ao7z+RX7l5Y5mvhemF97skX7DbXbgH5hx+K4gKzIjyT2M2yv/nv6Br+EluG8lI9KC8s1qapE8s+qu/b7AvGfhH7Y70v3lgIszZe+nQNORDv+/DsYdgX9K91fAHfS58nM1xWl0rYnsPcBb6u6vhofyln0th2Y+7s70c1M8IGEb2jSKGSa/C57YeBgeBn0PsHOm7CJ4xNLq6f7yeLG4HNnbc6516XOyKF6uYkb6bh8BLJopW7pJTgfm/d00/mfS7TLgO70af1Q7muuokmcAnsZ/vqWqlZI+gC/y5+A70Va21+HNat5t+eW0/5Xi7u+T92/4Kx4Dn0uVpLu/ALlRUnMhaW/g5zbUxr2jmZ2QIb4dvmu+FcDMHktx4LlUKSP9pA31HzyIv29tUV0/hDr+N3Pc2vy2BE40swslHZIpi5mdlj5rNZPb9pZpcjM39ZxXd/9x8mPoX5C0vg31hRSpkbUddc7i9D19r5ldkDHvF4CsFq0NeBxP7CzUJKcTDnbrcOZ7UcKnMAxJk4BxltkHNsmU7vAkaRkr2axG0gb47m88vgNcAu+vcFOm/Al4du8OwJdxZXi7ZdRzl3QKHu54KQU7SyX5ud6XAnbqW8xsQ6XOY8luPd0yo0UkbY6b3urLSO9uZr9rIVOLIHp/evw5+Jf+E7hf4csZ487VKa3RtSayl+BKfwvc3PUSfsLNtjMnR/HqZnZqslMvZmYP5cqXoQO+kMKfk04szMm0OhfWpk9Cpxzs/SROCsOw4nkGUKLDUx2lm9XYYAjs8+THX9fLfzH9eqKk31BMGf453cp0lgK3ccvSriS9V22fR5JwG/NJwHhJn8NDTH9cYOzrGVZGOkOmPsb/CQZLPz+FJ8+1mvOb8a5+C8vzKWr5MOMYam9vxSfx0+fR5uGNy+Olu7NIi9wU/DWfijf6OQPfjXaTwuGsw2hkS2+3blXOAm+3+LeQqym7dc1D3geQ12JqqhQkXW9mm2qwJtvAnyhWi60ScVJoQu6uNT12adzOXQvPvB632z4LTLTW4Yqlm9VIeiu+MKzM0Aig3PosSFqxgXzbchVVkVcMnYQ3Vze8JepfcnfcwAF4u0IBl5vZlQXGLr1jL4O8VPRn8EX5DwxNkpxmPei9K+91sR6eSb5eupbV67jiuIXDWYfJV4nimdpoYR5+LRcVaJLT5DOWvab0kzgpNKHIP8+8bs4+Tf7cVCEkqjSr+SW+qP6YEs2B5LHYnwLuYmg1xqZKoRNH88QBeOGxL+CL5BXkV5adDjxjBUsqd2jH3uh5W4bSmtk0eeXLHc3s580e12X+bWYmqXYyK9KUqAqVfCH49+qbDLbFvAL4Rqbsrnjv9Xo+0+BaLm0L2knaEa8Ku4q8kF+NxfEglPaDpDa+w5hjmSHTVQmlgC+ONqymfaNrBZ8zd1dRpVnNq2ZWpCrqcLbFu5AV6cDVkQJt5nHrJ+Kmq6XwDNJcxbYZsJekWrJg7Tnb7Xo/iC8KE6ir3ImHEH89c+xGtO2KlRz6ezHYVKjXnFPR5FaWWjjrFsB3VCyctZSzuBMLc5O55DTJuRF3Ui/N0Eqrc8hvcHQr3jjqaVwRjQcel/Qk8Llcf0xZwnxE06NepaO1pL1yPkQq0aymbiexLx75cj5Dnb1ZNXxSZMUnrK58Qi7ynr83WWYSUgP5ayjfXKh0smCS/5h5CevCNIogahJV1Ej2m7jCL1VzqSqS3k9Jk1uFMRfBfSGzzOy+5AtZy0pmCqfnbLnhSp+PVfAQ1HqFMgcP7X01Y4ypVGiSUwVJJ9I8mvEHuaa30uOPZqUg6QvAF/EwzAfq/rQ4Hve+Uw/nkt2sRtJDDDYHGo5Zm4xmDfa9XRFP1LuKoUqlbVE6SafhTsN/4LVpfo9ngT/dTjbJ32aeJftZvMrswb2wcaexxwPfooRzv2IEUaNIn7b/r2AouRuuimPcYWbryHsx7I2bsE5t93/uhLNYFaIZO8FoNx+diSeGzLWjKLJ7k1dfPASP5DCSo9nMmh5VVaFZjZkVyd5tRK3v7UwG+yEUwsx2AZC0Ap4R/n94uGHuZ6pKc6GqnII79z+Z7u+M7wqbOvc74Y/owP+tMA0Wp4E/0cOIlk7STiF0KIqnVJMcM9s0/azSwKtKNGNlRrtSGItHgMyVfi5pqQKK4WzcOVtzGH8aNxFs0UKmUrOaZkh6s7VpVmNm01r9PXOcnfBCXWvhmdw/xE8LuZRqLtQhyjj36/0RxzA0gijLH6GKNZfKUHFx6jvJB/Ex5i6d/e1mMh1amEs1yWniJK6fW86a8p94NOMF+Ofs+nRtLIMbma4x2s1HNTMMpF1E/e+5x3pJM83sHcOuzXUEbCJ7BfAxM5uT7i8O/NLMPpT5MoY/36VmtmUZ2SR/iGX0Opb0d9zkdiJwtZXL7+gLkqbjVSfrnftHW5uKo/Ls8dIRRPJ+2gswNPz4NTP7bJnnKzF+ffLa0niTpK4mr1VFnj/zLH6qHQhEMLNG7TJrMpUX5vS/XhfvxfBMsgasaG3yeKqadkcCo/qkUH+cTx+k1amrFlqAqyXtgDuCwM0puX0OyjaraUgVhZDIimwws6UlrYnveg+XV2y918x2biUn6WtmdlSdX2P48+Y2CKrCF4BpkoY499sJdSCCaAMbmoH8O3lFzK6juZPX3khvkteqMqHEBmkmLRZm8kq5lOrFUMVEqM6Fe1diVCuFGsnZORU3DdyOO1BvxCun5rAX3q7vjHR/DF7zZX/a2zDLNqtB3i949rBTxmQzuzlz3nNh+b2Ox+EKbWVciS1Bnr2z1nlsBsUyWzuGmd0OrFPEuV9Hla5tVWouVaVqvah+caOktcxsVq5Ah3w3B1vJJjk1JG1NnanQ2peF71Q/7kqMavNRDUmzcLv+TWa2rqS34dUrS7ccLDh+qWY1km7De+fWEpLGADNyImHS4ys1XsFtndenOT+aM2ad/Aa4LX4Sg5sT62H00S7MbafOiboqHUGkxjWXdjOzq7MmXgFVrBfVL+QNrN4CPIRHyGU3sEryRRfmmlzpXgzpsUfia0rtVLkj/t08KEe+n8RJwfmXmf1LUi3m/B5JaxR5AklrM/cik1u+4HY84eUN6bkmWl6zmoHaQWm81yUV+Z+W7nXcgcXkDLxER9keuFX4NXBTmbGr7ELN7KpkZhuouWTFEger0K/ktapU6VUxfGGeKmmTzIV5hrwmWX15jSJJYx/B6x/VmgtNA27DOwY2m+8sWkeK9USBh1JwHk27xwtw88DT5DfJqdVnWRuYzeAiY9SVG24huw8eafAEbkqoObxzPgAPStqXwQYxX8R3oblU7XU8BBWoDQM8ZWalwmE7wEKWkSTXiCoRRJIWwv9Hm+L/499LOtHM/lVmLkUws6PlyWvP4UrpW9aD5LWqmNkjktbBI93A+4Xk+mEKL8x11JfXqJVhyW2SU2M87q8CN6+2I7fhUlcJ89Ew5CVvlwB+Y5ltFiXdZWaTS453P/BOa5HT0EJ2WbwDWK0A3m+B/SyzybeG9joGLwNdutexCiQVJVPKjsydONeL4nBfwivLXkLBTPAqEUSSzsEzZGu+px3x4m6fKPQCRhHyzOLPMbjB2g7vL3B8huydeO+Ff6b7S+EmpF6YKHfEG/xcjSuVdwMHmdnZLQUH5VfGI8V+K2lh4A0132G3CaXQAeS9BY6xAn2C62Svxvuxtk297wYpgmgzKN7ruOK4Z+B9joecrnL8GR0Ye2+8y90zDB7Xc/0CdwyLIGp4rdOy3UDSyWa2Zz/GziUt7Bub10CiiC+kzMLcyQggeXJmrcXsLdYmf6hO7nN4scilzGy1ZHI80cxyA18qEeajzjANmC7pbxR3hj2Id3gq3KwmRa/8gMEa9dOBL5lZtgnJzGZLeooUipvrz1CJLO5hrJPrtOsC++MtNf9eQrZKBNFtkjay1ARJ0juBG0rMoVN0tVREhxBD39+aibUtZnaWvMZWbWE+IGNh7mQE0MYMmgrH4jXKctgb2BC4GcC8ZlSRjoqVCKXQGX6KmxHKOE2rNKs5E3eEbZfu7wCcRYv2n/WkyIxj8PIUT+LRMHcDa2aIl8nirucmSZN7dTIZxmygVCE/3Dl+taQhEUSZsu8EdpFUU7oTgbtrDsZeRwJZl6ttdohTgZtTyDZ4Zd9TCsgXWpitQpOcYY89AY+aOitd2kvSFmaW45d42cz+rVRVIwWP9MykE+ajDiDpd1agsU0Hx73ZhlVMlHSTmW2UKX8H7o/4rXlxus3wjN22JgVVyOJOj70bb3pfKtSwCmmBWRM3KxQqBJjkF6REBJGaVHetGz+3N3c2nTSH9AtJ6+MLuygWsj18Yf4U8EDOwqyKTXIkzQbeXosOTOHis8ys7YZL0lG4aXMX3OH9ReAuM+tJjbA4KXSGeySdCVxMptO0Q1/WqyUdiO/aDf/QX5ocajmO01fM7B+SxkgaY2ZXyxvv5FAlixu8FHC/uIACSUj1VIwgegPwqJm9LOm9eITZafVhwV1gRCREFUXSODN7Ln2WH0632t9y65K9h6EL8zT8NN9q3E71YrgXPwnWFP1K5PdTOAD4bJrrXngIdW4DqsrESaEDSDq1weWWTlN1oMG3GidS1Y/froT2b/Hj+BF4U5An8VIM78oYew6wKIPmsjEMZviazYPVN3OoEkEkL7o3Bc9nuRwvgriGmX2kK5MdOvaiwEt14ZljgQWtZD+MbiPpEjPbSkPrkwH5dckknYf72B5J91cGjjSzHVvIVO7FkJ7nWtyXcUu6tAHu83sRmm/60oniTjN7e8443SCUQp9RxWY1FcdeFPgX/kX7NB6K+/My4bHzOsovBFgl+qiWTfw1fIE+vohJogqSbgK2sNRQSdJiwBU5G4B5lbILc4fGbrjZq9Fq0yfp53iUVE4Ca8cJ81EHkDQBOJ6hkThTLa/0w2fwlpSlmtU0mEvb0tk16sL8xuGmr6JjVcniHmnkOl2rRBC9kswTu+Al0sFzHnrBQlbXYc/Mnpd3RRvRJH/CcJ4FHsnYtX+rxHid6MWQddJvwfLAbEm3MLS+Vk/8P3FS6ACSrsQjgWr2252AT5vZ+ws8R61ZzVeAFcyslMJWgdLZ8oqf38ZbRL5OsaN5wyzuViaz+YHkIF8DjxiDFEGEvwctHeWSJgOfx+Psz5K0CvApMzuyy9NG0g3APmZ2a7o/BTje2pQL7zfphLM+bo8X3r/jDuBNwOetB+0xO0VuXkgVk3InCKXQAdSgRV6ja01khzeruR5P5Z/ehakOH/s+PDGocLy+KmRx9xtVKwTY8wiiTpCUwC/w8i2GhyF/aqSHpUo6GzjMzGan+5PxsODDgPNyvmPDnq/lwqzONMlp9tzvGOnvN4T5qFP8PS3utdC3HcmPVPg+FZrVpAVtJYaacG7NFH+A8vH60/uYZ1CV0oUA6XAEUa4vowOsgpfOnojntdQSHkc6b6spBAAzu0vSemb2oNp3x2xEu4S9TvRiQNKkBt/lMTmyTZ6vZ9nnoRQ6w+54O8rv4R+cG8lMaLKSzWoAJB2G+yQeoK5cA4O1kNpxEF6v/maKx+tXyeLuN1UKAZ4LTJH0FjyJ6iLcdFg2gqhXO8dvmtkv5YUf348nLf6IzETHPnKvpB8xtF/xn1KuSMsihGUWZutcH+3zJH3UzP6a5vIefI0om8Xfs+zzUAqd4TBg12GLzNG4smiJyjerAe/XupplFu5rwEnA7yiXiV0li7vfHIMrw1/hSvSTeC2kHF43s1clbQ98vxZBVHYiltnUqAPUSkVsidfRuVDSIT0auwqfwfNC9oOBfsVfwRXCZm1kKy3MKtmLIbEXcIGkj+I+kf8lc+PQ6VNGYcwsbhVvwG0515rI3gmcgCfMTCg47rnAshXmfWMF2d/1+32v+D+bDPwXnjE6uYDczbh58I/AKunaHzNlpwHj6+4vCfy0R6/3EnwT8ABe0nlB4I5+/x+6/Jo3AP4AvBlfkG8HVsqUPRKv4Lt7ul0JHFFw/I3T9/sWYJkCcrfi/aBr99+DZ0P35H0LR3MHkJeLeK8NPSlcawULvhUJJ02PnwJciC9Q9eafrNA1SYfjGZfDM7FzSkifgC8u2VncIwkNbWK/DLCYZTSxrxJB1CgnoYd5CovgWeSzzAusLQ+sZSM0ekfSOWb2STVpPGP5ndc2xpXhv4AtzeypTLk7GdqLYSy+0Ws5ruauUDAZb6D1dJp32++mvCvhCXjYcu2U8VEz+0vO3KsSSqEDaGhfggFzhBXsS6AG9VbaPH42/oEfYsKxzNC1JhnRZnkhqYWzuEcKqmtib2ZvTeHAvzSzrjax79TmYTQgaXkze7xZtJe1iPLq0MJcqhdDlaS1Yc9TSpl1gvApdAAzO03SDNzBK2B7KxeVUzSc4u9mdlyJcYDKrSVzK4OORDraxL5ABFEVX8aowsweTz/LhPh2os7TEXii4pBeDO2Echf9RjRQZovgiXqnSMq2AFQllEKHSEqganhm0Z65MyUdgUfA1JtwckNS5yLXhFUxi7vf/NvMTFKtUNqiFZ8vK4Kog5uH+Z5hGcW1zVItVNSsRWZxlYW57jnK9GIYPu8hf6J9RvSIKFoY5qM+I2k1Ssa9p13McMwqlPHOzYjuRBZ3P5AHt38TWBEPzTwCdySeaRktHjswfilfRpBPxYW5/nm2Z7Aa7vVmltskZ54mlEKfUR8rZ1ahShZ3v5F0K16e+AP4QnG5ZTaxr5gN3RdfxrzOMEW6NLB4txWpKvRiGPY8y5K6GgJYiyJ3nVJmVQnzUf+pxb1vR8G4d3lLzIOp281QoCVmlVMK1bK4+8104Bkz+2oJ2SrZ0B31ZYwG6hUp3oXtjXjZ8mxFWmRhrqNwL4ZhYxbuamhmI+Kz0LuEiKAZtcqZu+Kx5JBfOfNs4Cm8JebH0++/KDD2uXjP4Vp27iq4SSiH3XFH6d/wyI6Pk9+Wst9shmdjPyDpztotU3ZMOh0AhbOh/50WmU75MkYD2wFbk6qFmtljeMObtkjaWl7f6yG8jebDwGWZ49aa5NQo0iQHPKF1I+BPKaBjcwr245a0rKSJtVsR2SrESaH/7IbHvR9uZg+luPcz2sjUWMrMDqu7/z+Sti0wdulTChWyuEcAH64gWyqCKPkyLpF0EjBe0ufw96pocMFoo0pQQG1hHtJuNlP2TXj/7CG9GJS6sWVEApXualjmlNFJQin0mRR9sm/d/YfwbMocqrbErD+lFK3vv7bV9Xwws38WMKP0lZJhjjXZUhFEaWHbFvdlPIebQ76V68sYxZxTQZFWaTdbuBfDMJ6RNzK6Dvi5pCdpU6upjirKrDKhFPqMpK3wD8HK+P+jrVOpziElYH8GI4DGAs/jfoYcqpxSqhSVm9dZCnihFkEkaZVMx2cVX8aoI52ufgG8jXKKtPTC3IGw1jvwCsRfYrCr4WKZslWUWWVGy5d4JPN9YHu8/EBWKFgHHVKrAvtZSuUveEoZlYlYDRyfC5Dv+NwM2EvSIwztqDUvVJbtOel0dYGZvQOvPVSUKgvzXKhY+erN0vfqdbzmVS1LOocqp4zKhFLoP3/BC6plxwZLepuZ3aPGrQqLJK/tAPxA0rnAqWZ2d+4cRnEiVpUIoiq+jNHKTZI2MLM/lJCtsjA3om35aklfwKu6rjZsrMXJdzR3VJkVJfIU+oy8+NVheHREfVbysS1kTjazPTuRvCYv3b0jbkoyfPd7lpnNyX2O0YSkW8xsQ6U6VcnxOT12+91B0l3AW/HCjS+Q0bejfmEG7q/70+LADWa2U8a4k2xY+eoc5SRpCbz67RHAgXV/mmOZXdvUoAaapDt79RkLpdBnJF2B+wGGF7U7tIdzWBrPSN4Pj3J4C3BcLzJ85yX6nQ09GlG5gngdWZjxyqRDejFYF4sXdkKZdWQeoRT6i6QZZjalg8+XXX5b3gBkd/xDeDowzcyelJdZvtvMWvYjHo2oQjZ0MO+gPpSv7oQy68g8Qin0F0lH4g1rOlLXXpm1i9JjTwN+YmbXNfjb5mZ2VSfmND8h6f+An5W0cQcdQNIlZrZVD8bpW/nqfhJKoc+k8NJFcX/CK/Swzkmyh79kZq9Leise+neZmfUs0mFeo4GNG4gIol6i1GuhS89duRfDvE4ohXkYVatdhKSZwP/Dj6w3ATOAF83s012Z8HxAGRt3UJ76jUu6PwZYyMxe7NJ4HWmSMy8TSmEeRhUrrNZF0OwDLGxmR6lH7SGDIAdJNwFbmNnz6f5iwBVm9q7+zmz+JQrijUCSMzOH183sVTx2/vtm9iVg+WJDaWM8FrpWHiNyV4KRxEI1hQCQfl+kW4NJmiPpuQa3OZKe69a4I4lYAEYgw2OUW1CldhHAVLzF4PlmNlvSqkCj3Icg6BcvSFq/lpAp6R3AS90arIPVAuZZwnzUZ6o4eyVNxmsXTTdvH7gK8Ckzyy1VEQQjmhQaejbwWLq0PP4Zz2qB2oHxy/RimKcJpdBnqjh7UzG9X9eccB2az55mdnKnni8IqiJpAbzWlIB7ehEd16x8tZn1pHx1PwmfQv9RiqTYHjjezLbDw+By2AG4T9JRkv6jU/Pp0PMEQWWSQvgCcAhe/XevdK3bVG6SM68SSqH/lHb2prT39YAHgFMlTZe0Z4ECbY2es23RryDoIT8C3oFnF5+Qfv9RD8Z9xbyt7UD5amDdHozbd8LR3H/2o4Kz18yeS1VOF07PtR3wVUltaxfJezwfgpd9LtzjOQh6wAZmtk7d/d9JuqMH4/a1fHU/iZNCnzGza1OW5AmSFjOzB81s37aCeO0iSecDv8OjjjY0sw8D6wBfyXiKs3F7adkez0HQbV5LSZoApE3Taz0Yt7589W/w0/g9PRi374Sjuc9IWgs4De/mJXxh3sXMZmfIVqpdJGlmamBSf62jBfqCoAqS3gf8DHgwXZoE7JbMOd0ct6/lq/tJmI/6z0nA/rUPeSpX8WMgJ2PzC6SY7eHhrJnF7Kr2eA6CbvMm4O24MtgG/148263BOtQkZ54mTgp9RtIdw2ymDa81ka1Uu6iuGF8tpHUMg0XeelKULwhaUdudS9oUL199DPB1M3tnl8YbEeWr+0mcFPrPg5K+ifczAG92k9MEHlI4q6Q98HDWoyTdljtwZG8G8wA1/8GWwIlmdqGkQ7o1mJk9i59EduzWGCOdUAr9Z3fgUOC8dP86vDVmDvXhrHuka4X+p5LWxo/mA3Jmdl5TgSDoLX+VdBKwBfAdSQsSATJdJcxH8zCS3o1HGd1gZt9JkRn7FYhe+ilebns2gyYkM7PduzLhIChI6gL4IWCWmd0naXlgrU41pQrmJpTCCKRXpSYk3WVmudnTQRCMAuIYNjIpXWpC0p4FHj49FdULgiAAwqcwIqlYaqKIQpmGK4a/4e1Aa61A5/tY7CAIGhPmoz7Tz1ITku4H9gdmMehTiNaSQTCKiZNC/zkbjzj6WLr/abzUxBbtBDugUP5sZhcVnXAQBPMvcVLoM1VKTUi6ElcoZ6RLnwbea2ZtFUqSPwEYD1yMm4+ACEkNgtFMnBT6T5VSE0uZ2WF19/9H0rYFxl4YVwYfqLtmDOZMBEEwyoiTQp+pUmpC0tF4aYt6hbKmmR3cpekGQTCfE0phHqZq7SJJE4DjGeqTmGpmj3ZnxkEQjHRCKYwA+lVqIvkkzmRo3aVPm9n7uz12EAQjk1AKfaZqqYkqCkXS7Wa2brtrQRCMHsLR3H82KltqoplCId9R/HdJOwFnpfs7AtGKMwhGMaEU+s90SZPN7K4SsqUVSmJ34IfA93BlciP5FVqDIJgPCaXQf6qUmqiiUAAOA3Y1s6cBJC0FHI0riyAIRiGhFPrPT4GdGVZqIpOqtYvWrikEXPCfktYrOIcgCOYjQin0nyqlJqooFIAxkpYcdlKIz0QQjGJiAeg/90g6k3KlJqrWLjoGuFHSr3CfwieBwys8XxAE8zgRktpnJJ3a4HJWSGonahelfgrvw01PV1XwTwRBMB8QSmEepopCCYIgaEQohT4TpSaCIBhJRDvO/nMqcBGwArAibgpqdAKYC0kTJJ0v6UlJT0g6NymZIAiCUoRS6D/LmNmpZvZquv0MWCZTtrRCCYIgaEQohf7zd0k7SRqbbjuRX2qiikIJgiCYi1AK/Wd3PBT0b8DjeE+E3FITVRRKEATBXISjuc9ImgbsN7zURGZI6kS8dtHGDNYu2tfM/tzFKQdBMB8TyWv9p0qpiahdFARBRwnzUf8ZI2nJ2p2CpSbmUihA1C4KgqA0cVLoP1VKTUTtoiAIOkosIH3GzE6TNIPBUhPbFyg1EbWLgiDoKOFonseJ2kVBEHSSUApBEATBAOFoDoIgCAYIpRAEQRAMEEohCIIgGCCUQhAEQTBAKIUgCIJggP8PflUTI1RfGfUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics as mt\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Split it into train / test subsets\n",
    "X_train, X_test, y_train_ohe, y_test_ohe = train_test_split(X, y_ohe, test_size=0.2,\n",
    "                                                            stratify=bunch.target, \n",
    "                                                            random_state=42)\n",
    "NUM_CLASSES = 20\n",
    "\n",
    "# print some stats of the data\n",
    "print(\"X_train Shape:\",X_train.shape, \"Label Shape:\", y_train_ohe.shape)\n",
    "uniq_classes = np.sum(y_train_ohe,axis=0)\n",
    "plt.bar(list(range(20)),uniq_classes)\n",
    "plt.xticks(list(range(20)), bunch.target_names, rotation='vertical')\n",
    "plt.ylabel(\"Number of Instances in Each Class\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the embedding\n",
    "But this is going to be a more involved process. Maybe we can speed up the training by loading up a pre-trained embedding of the words?! Recall that the GloVe embedding uses pre-trained word embeddings in order to group contextually similar words together. \n",
    "\n",
    "A number of versions of the GloVe embedding matrix are pre-trained using various document. Let's use a version trained from 6B different documents from wikipedia (this is actually one of the smaller datasets trained upon--so the vocabulary might be a bit limited).\n",
    "\n",
    "Let's use the GloVe word embedding in keras. We will follow the example at:\n",
    "- https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html\n",
    "\n",
    "You can download the nearly 1GB pretrained embeddings here:\n",
    "- http://nlp.stanford.edu/data/glove.6B.zip\n",
    "\n",
    "Let's take a quick look at the format of the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m.\u001b[m\u001b[m                 glove.6B.100d.txt glove.6B.300d.txt\r\n",
      "\u001b[34m..\u001b[m\u001b[m                glove.6B.200d.txt glove.6B.50d.txt\r\n"
     ]
    }
   ],
   "source": [
    "!ls -a \"large_data/glove/\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581\r\n",
      ", 0.013441 0.23682 -0.16899 0.40951 0.63812 0.47709 -0.42852 -0.55641 -0.364 -0.23938 0.13001 -0.063734 -0.39575 -0.48162 0.23291 0.090201 -0.13324 0.078639 -0.41634 -0.15428 0.10068 0.48891 0.31226 -0.1252 -0.037512 -1.5179 0.12612 -0.02442 -0.042961 -0.28351 3.5416 -0.11956 -0.014533 -0.1499 0.21864 -0.33412 -0.13872 0.31806 0.70358 0.44858 -0.080262 0.63003 0.32111 -0.46765 0.22786 0.36034 -0.37818 -0.56657 0.044691 0.30392\r\n",
      ". 0.15164 0.30177 -0.16763 0.17684 0.31719 0.33973 -0.43478 -0.31086 -0.44999 -0.29486 0.16608 0.11963 -0.41328 -0.42353 0.59868 0.28825 -0.11547 -0.041848 -0.67989 -0.25063 0.18472 0.086876 0.46582 0.015035 0.043474 -1.4671 -0.30384 -0.023441 0.30589 -0.21785 3.746 0.0042284 -0.18436 -0.46209 0.098329 -0.11907 0.23919 0.1161 0.41705 0.056763 -6.3681e-05 0.068987 0.087939 -0.10285 -0.13931 0.22314 -0.080803 -0.35652 0.016413 0.10216\r\n",
      "of 0.70853 0.57088 -0.4716 0.18048 0.54449 0.72603 0.18157 -0.52393 0.10381 -0.17566 0.078852 -0.36216 -0.11829 -0.83336 0.11917 -0.16605 0.061555 -0.012719 -0.56623 0.013616 0.22851 -0.14396 -0.067549 -0.38157 -0.23698 -1.7037 -0.86692 -0.26704 -0.2589 0.1767 3.8676 -0.1613 -0.13273 -0.68881 0.18444 0.0052464 -0.33874 -0.078956 0.24185 0.36576 -0.34727 0.28483 0.075693 -0.062178 -0.38988 0.22902 -0.21617 -0.22562 -0.093918 -0.80375\r\n",
      "to 0.68047 -0.039263 0.30186 -0.17792 0.42962 0.032246 -0.41376 0.13228 -0.29847 -0.085253 0.17118 0.22419 -0.10046 -0.43653 0.33418 0.67846 0.057204 -0.34448 -0.42785 -0.43275 0.55963 0.10032 0.18677 -0.26854 0.037334 -2.0932 0.22171 -0.39868 0.20912 -0.55725 3.8826 0.47466 -0.95658 -0.37788 0.20869 -0.32752 0.12751 0.088359 0.16351 -0.21634 -0.094375 0.018324 0.21048 -0.03088 -0.19722 0.082279 -0.09434 -0.073297 -0.064699 -0.26044\r\n",
      "and 0.26818 0.14346 -0.27877 0.016257 0.11384 0.69923 -0.51332 -0.47368 -0.33075 -0.13834 0.2702 0.30938 -0.45012 -0.4127 -0.09932 0.038085 0.029749 0.10076 -0.25058 -0.51818 0.34558 0.44922 0.48791 -0.080866 -0.10121 -1.3777 -0.10866 -0.23201 0.012839 -0.46508 3.8463 0.31362 0.13643 -0.52244 0.3302 0.33707 -0.35601 0.32431 0.12041 0.3512 -0.069043 0.36885 0.25168 -0.24517 0.25381 0.1367 -0.31178 -0.6321 -0.25028 -0.38097\r\n",
      "in 0.33042 0.24995 -0.60874 0.10923 0.036372 0.151 -0.55083 -0.074239 -0.092307 -0.32821 0.09598 -0.82269 -0.36717 -0.67009 0.42909 0.016496 -0.23573 0.12864 -1.0953 0.43334 0.57067 -0.1036 0.20422 0.078308 -0.42795 -1.7984 -0.27865 0.11954 -0.12689 0.031744 3.8631 -0.17786 -0.082434 -0.62698 0.26497 -0.057185 -0.073521 0.46103 0.30862 0.12498 -0.48609 -0.0080272 0.031184 -0.36576 -0.42699 0.42164 -0.11666 -0.50703 -0.027273 -0.53285\r\n",
      "a 0.21705 0.46515 -0.46757 0.10082 1.0135 0.74845 -0.53104 -0.26256 0.16812 0.13182 -0.24909 -0.44185 -0.21739 0.51004 0.13448 -0.43141 -0.03123 0.20674 -0.78138 -0.20148 -0.097401 0.16088 -0.61836 -0.18504 -0.12461 -2.2526 -0.22321 0.5043 0.32257 0.15313 3.9636 -0.71365 -0.67012 0.28388 0.21738 0.14433 0.25926 0.23434 0.4274 -0.44451 0.13813 0.36973 -0.64289 0.024142 -0.039315 -0.26037 0.12017 -0.043782 0.41013 0.1796\r\n",
      "\" 0.25769 0.45629 -0.76974 -0.37679 0.59272 -0.063527 0.20545 -0.57385 -0.29009 -0.13662 0.32728 1.4719 -0.73681 -0.12036 0.71354 -0.46098 0.65248 0.48887 -0.51558 0.039951 -0.34307 -0.014087 0.86488 0.3546 0.7999 -1.4995 -1.8153 0.41128 0.23921 -0.43139 3.6623 -0.79834 -0.54538 0.16943 -0.82017 -0.3461 0.69495 -1.2256 -0.17992 -0.057474 0.030498 -0.39543 -0.38515 -1.0002 0.087599 -0.31009 -0.34677 -0.31438 0.75004 0.97065\r\n",
      "'s 0.23727 0.40478 -0.20547 0.58805 0.65533 0.32867 -0.81964 -0.23236 0.27428 0.24265 0.054992 0.16296 -1.2555 -0.086437 0.44536 0.096561 -0.16519 0.058378 -0.38598 0.086977 0.0033869 0.55095 -0.77697 -0.62096 0.092948 -2.5685 -0.67739 0.10151 -0.48643 -0.057805 3.1859 -0.017554 -0.16138 0.055486 -0.25885 -0.33938 -0.19928 0.26049 0.10478 -0.55934 -0.12342 0.65961 -0.51802 -0.82995 -0.082739 0.28155 -0.423 -0.27378 -0.007901 -0.030231\r\n"
     ]
    }
   ],
   "source": [
    "!head \"large_data/glove/glove.6B.50d.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n",
      "Embedding Shape: (134143, 100) \n",
      " Total words found: 56914 \n",
      " Percentage: 42.427856839343086\n",
      "CPU times: user 7.28 s, sys: 229 ms, total: 7.51 s\n",
      "Wall time: 7.54 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "EMBED_SIZE = 100\n",
    "# the embed size should match the file you load glove from\n",
    "embeddings_index = {}\n",
    "f = open('large_data/glove/glove.6B.100d.txt')\n",
    "# save key/array pairs of the embeddings\n",
    "#  the key of the dictionary is the word, the array is the embedding\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))\n",
    "\n",
    "# now fill in the matrix, using the ordering from the\n",
    "#  keras word tokenizer from before\n",
    "found_words = 0\n",
    "embedding_matrix = np.zeros((len(word_index) + 1, EMBED_SIZE))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be ALL-ZEROS\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "        found_words = found_words+1\n",
    "\n",
    "print(\"Embedding Shape:\",embedding_matrix.shape, \"\\n\",\n",
    "      \"Total words found:\",found_words, \"\\n\",\n",
    "      \"Percentage:\",100*found_words/embedding_matrix.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "# save this embedding now\n",
    "embedding_layer = Embedding(len(word_index) + 1,\n",
    "                            EMBED_SIZE,\n",
    "                            weights=[embedding_matrix],# here is the embedding getting saved\n",
    "                            input_length=MAX_ART_LEN,\n",
    "                            trainable=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 1000, 100)         13414300  \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 100)               80400     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 20)                2020      \n",
      "=================================================================\n",
      "Total params: 13,496,720\n",
      "Trainable params: 82,420\n",
      "Non-trainable params: 13,414,300\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.layers import LSTM\n",
    "\n",
    "rnn = Sequential()\n",
    "rnn.add(embedding_layer)\n",
    "rnn.add(LSTM(100,dropout=0.2, recurrent_dropout=0.2))\n",
    "rnn.add(Dense(NUM_CLASSES, activation='sigmoid'))\n",
    "rnn.compile(loss='categorical_crossentropy', \n",
    "              optimizer='rmsprop', \n",
    "              metrics=['accuracy'])\n",
    "print(rnn.summary())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/6\n",
      "9051/9051 [==============================] - 363s 40ms/sample - loss: 2.8493 - accuracy: 0.1138 - val_loss: 2.6135 - val_accuracy: 0.1578\n",
      "Epoch 2/6\n",
      "9051/9051 [==============================] - 357s 39ms/sample - loss: 2.6393 - accuracy: 0.1505 - val_loss: 2.5079 - val_accuracy: 0.1648\n",
      "Epoch 3/6\n",
      "9051/9051 [==============================] - 358s 40ms/sample - loss: 2.5182 - accuracy: 0.1632 - val_loss: 2.4845 - val_accuracy: 0.1829\n",
      "Epoch 4/6\n",
      "9051/9051 [==============================] - 347s 38ms/sample - loss: 2.3814 - accuracy: 0.1921 - val_loss: 2.2054 - val_accuracy: 0.2205\n",
      "Epoch 5/6\n",
      "9051/9051 [==============================] - 363s 40ms/sample - loss: 2.2085 - accuracy: 0.2432 - val_loss: 2.0055 - val_accuracy: 0.2965\n",
      "Epoch 6/6\n",
      "9051/9051 [==============================] - 382s 42ms/sample - loss: 2.0587 - accuracy: 0.2943 - val_loss: 1.8944 - val_accuracy: 0.3305\n"
     ]
    }
   ],
   "source": [
    "history = []\n",
    "tmp = rnn.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe), epochs=6, batch_size=64)\n",
    "history.append( tmp )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/6\n",
      "9051/9051 [==============================] - 380s 42ms/sample - loss: 1.9154 - accuracy: 0.3324 - val_loss: 1.7110 - val_accuracy: 0.3955\n",
      "Epoch 2/6\n",
      "9051/9051 [==============================] - 381s 42ms/sample - loss: 1.7238 - accuracy: 0.4002 - val_loss: 1.4770 - val_accuracy: 0.4887\n",
      "Epoch 3/6\n",
      "9051/9051 [==============================] - 358s 40ms/sample - loss: 1.5282 - accuracy: 0.4736 - val_loss: 1.3275 - val_accuracy: 0.5369\n",
      "Epoch 4/6\n",
      "9051/9051 [==============================] - 348s 38ms/sample - loss: 1.3424 - accuracy: 0.5435 - val_loss: 1.1766 - val_accuracy: 0.6186\n",
      "Epoch 5/6\n",
      "9051/9051 [==============================] - 345s 38ms/sample - loss: 1.2069 - accuracy: 0.5905 - val_loss: 1.0639 - val_accuracy: 0.6403\n",
      "Epoch 6/6\n",
      "9051/9051 [==============================] - 347s 38ms/sample - loss: 1.0899 - accuracy: 0.6358 - val_loss: 0.9566 - val_accuracy: 0.6911\n"
     ]
    }
   ],
   "source": [
    "# let's extend the training by a number of epochs\n",
    "tmp = rnn.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe), epochs=6, batch_size=64)\n",
    "history.append( tmp )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAFNCAYAAABSRs15AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAB8+klEQVR4nOzdd3hUZfrG8e+TRgiE3nuXXiT0joqAKNgpNiyIil1X3d/q6u66umtDEUVUxAKCiiIqKkWU3nsn9NA7oaa9vz/O6EYEDJCZk3J/risXmTlnzvuMQt7cc95izjlEREREREQk+wvzuwARERERERHJHAp4IiIiIiIiOYQCnoiIiIiISA6hgCciIiIiIpJDKOCJiIiIiIjkEAp4IiIiIiIiOYQCnkgmMbPhZvavDJ67ycwuDXZNIiIi2V1m9a/nch2R7EwBT0REREREJIdQwBOR3zGzCL9rEBEREZHzo4AnuUpg6MbjZrbUzI6a2ftmVtLMvjezRDObZGaF051/lZmtMLODZvazmdVKd6yRmS0MvG40EH1KW93MbHHgtTPNrH4Ga7zCzBaZ2WEz22pmz55yvHXgegcDx28LPJ/XzF4xs81mdsjMpgeea29mCaf573Bp4PtnzewLM/vEzA4Dt5lZUzObFWhjh5m9aWZR6V5fx8wmmtl+M9tlZn81s1JmdszMiqY7r7GZ7TGzyIy8dxERyZ6yQ/96mprvMrP4QF82zszKBJ43M3vNzHYH+tOlZlY3cKyrma0M1LbNzB47r/9gIkGkgCe50bXAZUAN4Erge+CvQDG8fxMPAJhZDeBT4CGgODAe+MbMogJhZyzwMVAE+DxwXQKvvRgYBtwNFAXeAcaZWZ4M1HcUuAUoBFwB3GNmPQLXrRCod1CgpobA4sDrXgYaAy0DNf0FSMvgf5PuwBeBNkcAqcDDeP9NWgCXAPcGaogFJgE/AGWAasBk59xO4GfghnTXvQkY5ZxLzmAdIiKSfWX1/vU3ZtYReAGvzyoNbAZGBQ53AtoG3kch4EZgX+DY+8DdzrlYoC7w07m0KxIKCniSGw1yzu1yzm0DpgFznHOLnHMnga+ARoHzbgS+c85NDASUl4G8eAGqORAJDHTOJTvnvgDmpWvjLuAd59wc51yqc+5D4GTgdWflnPvZObfMOZfmnFuK1wm2CxzuA0xyzn0aaHefc26xmYUBtwMPOue2BdqcGXhPGTHLOTc20OZx59wC59xs51yKc24TXgf6aw3dgJ3OuVeccyecc4nOuTmBYx/ihTrMLBzohddJi4hIzpel+9dT9AGGOecWBup7CmhhZpWAZCAWqAmYc26Vc25H4HXJQG0zK+CcO+CcW3iO7YoEnQKe5Ea70n1//DSP8we+L4P3iR4Azrk0YCtQNnBsm3POpXvt5nTfVwQeDQwfOWhmB4HygdedlZk1M7MpgaGNh4D+eJ9+ErjG+tO8rBjeEJbTHcuIrafUUMPMvjWznYFhm//OQA0AX+N1fFXwPsU95Jybe541iYhI9pKl+9dTnFrDEby7dGWdcz8BbwKDgV1mNtTMCgROvRboCmw2s1/MrMU5tisSdAp4Ime2Ha8jAbwx+XidyDZgB1A28NyvKqT7fivwvHOuULqvGOfcpxlodyQwDijvnCsIDAF+bWcrUPU0r9kLnDjDsaNATLr3EY43JCY9d8rjt4HVQHXnXAG8ITZ/VgPOuRPAZ3ifjN6M7t6JiMgf+dW/nq2GfHhDPrcBOOfecM41BurgDdV8PPD8POdcd6AE3lDSz86xXZGgU8ATObPPgCvM7JLAIiGP4g0DmQnMAlKAB8wswsyuAZqme+27QP/A3Tgzs3zmLZ4Sm4F2Y4H9zrkTZtYU6J3u2AjgUjO7IdBuUTNrGPj0cxjwqpmVMbNwM2sRmJOwFogOtB8J/A34s7kKscBh4IiZ1QTuSXfsW6CUmT1kZnnMLNbMmqU7/hFwG3AV8EkG3q+IiOQufvWv6Y0E+ppZw0Bf+W+8IaWbzKxJ4PqReB+SngBSA3ME+5hZwcDQ0sN4c9ZFshQFPJEzcM6twZtPNgjvDtmVwJXOuSTnXBJwDV6QOYA3n+DLdK+djzdP4M3A8fjAuRlxL/APM0sEniHdp4POuS14Q0MeBfbjLbDSIHD4MWAZ3lyF/cB/gDDn3KHANd/D+2TyKPC7VTVP4zG8YJmI15mOTldDIt7wyyuBncA6oEO64zPwFndZGJi/JyIi8hsf+9f0NUwGngbG4N01rAr0DBwugNf3HcAbxrkPb54geKNTNgWmL/QPvA+RLMV+P8RZROTCmdlPwEjn3Ht+1yIiIiKSmyjgiUimMrMmwES8OYSJftcjIiIikptoiKaIZBoz+xBvj7yHFO5EREREQk938ERERERERHII3cETERERERHJIRTwREREREREcogIvws4V8WKFXOVKlXyuwwREQmBBQsW7HXOFfe7juxCfaSISO5wtv4x2wW8SpUqMX/+fL/LEBGREDCzzX7XkJ2ojxQRyR3O1j9qiKaIiIiIiEgOoYAnIiIiIiKSQyjgiYiIiIiI5BBBnYNnZp2B14Fw4D3n3IunHH8c6JOullpAcefc/nNpJzk5mYSEBE6cOJEJVWdt0dHRlCtXjsjISL9LERGRbCC39JHqH0VEPEELeGYWDgwGLgMSgHlmNs45t/LXc5xzLwEvBc6/Enj4XMMdQEJCArGxsVSqVAkzy5w3kAU559i3bx8JCQlUrlzZ73JERCQbyA19pPpHEZH/CeYQzaZAvHNug3MuCRgFdD/L+b2AT8+noRMnTlC0aNEc23H9yswoWrRojv8UVkREMk9u6CPVP4qI/E8wA15ZYGu6xwmB5/7AzGKAzsCY820sJ3dc6eWW9ykiIpknN/QdueE9iohkRDAD3ul+0roznHslMONMwzPNrJ+ZzTez+Xv27Mm0AjPLwYMHeeutt875dV27duXgwYOZX5CIiEgWoT5SRCS0ghnwEoDy6R6XA7af4dyenGV4pnNuqHMuzjkXV7z4aTds99WZOq/U1NSzvm78+PEUKlQoSFWJiIj4T32kiEhoBTPgzQOqm1llM4vCC3HjTj3JzAoC7YCvg1hLUD355JOsX7+ehg0b0qRJEzp06EDv3r2pV68eAD169KBx48bUqVOHoUOH/va6SpUqsXfvXjZt2kStWrW46667qFOnDp06deL48eN+vR0RkQtzbD/MGgzuTIM2JKs6cjKFIyeSM/Wa6iNFREIraAHPOZcCDAB+BFYBnznnVphZfzPrn+7Uq4EJzrmjwaol2F588UWqVq3K4sWLeemll5g7dy7PP/88K1d6C4YOGzaMBQsWMH/+fN544w327dv3h2usW7eO++67jxUrVlCoUCHGjDnv6YgiIv6JnwRvt4SJz8DulX9+vmQZzjl2HT7Bxr3H2H80KdOuqz5SRCS0groPnnNuPDD+lOeGnPJ4ODA8s9p87psVrNx+OLMuB0DtMgX4+5V1Mnx+06ZNf7dM8xtvvMFXX30FwNatW1m3bh1Fixb93WsqV65Mw4YNAWjcuDGbNm264LpFREIm6agX6ua9B8VrQu/RUDLjPzclNDLSR55ITiU1zREVEUZk+J9/Dqw+UkQkawlqwMut8uXL99v3P//8M5MmTWLWrFnExMTQvn370y7jnCdPnt++Dw8P1/ATEck+EubDl/1g/wZoMQA6Pg2R0X5XJecpOjKckylpJKWkkeYgT0TmDvZRHykiElw5LuCdy6eImSU2NpbExMTTHjt06BCFCxcmJiaG1atXM3v27BBXJyISJKnJ8Mt/YdorUKAM3PoNVG7jd1VyFhntI51z7E48ya7DJ8ifJ4KKRWMIDzu/oKc+UkQktHJcwPND0aJFadWqFXXr1iVv3ryULFnyt2OdO3dmyJAh1K9fn4suuojmzZv7WKmISCbZs8a7a7djMTToDV1ehOiCflclmcTMKFkgmsjwMLYdOM76PUepVDQfUedxN099pIhIaJnLZqucxcXFufnz5//uuVWrVlGrVi2fKgq93PZ+RSQLSUuDue/ApGchMgaufB1qXxW05sxsgXMuLmgN5DDB6CMTTySzZd8xwsKMSkVjyBuVdT8bVv8oIrnF2frHrPtTWkREspZDCTD2Xtj4C1S/HK4aBLEl//x1kq3FRkdSpUR+Nu09yvo9R6lYNIbY6Ei/yxIRkTNQwBMRkbNzDpZ+BuMfh7QU767dxbeCmd+VSYjkjQynWiDkbdp7jLKFoymSL8+fv1BEREJOAU9ERM7s2H749mFYORbKN4Orh0CRKn5XJT6IDA+jSvH8bNl/jIQDx0lKSaNkgWhMQV9EJEtRwBMRkdNbNxG+HgDH9sElf4dWD0JYuN9ViY/CA/Pwth08zu7EkySlOsoVzkuYQp6ISJahgCciIr+XdBQm/A3mD4PitaDP51C6vt9VZTtmVh74CCgFpAFDnXOvn3LO40CfwMMIoBZQ3Dm338w2AYlAKpCSVRabMTPKFspLVHgYOw+fIDk1jYpFYojIwKboIiISfAp4IiLyP1vnwld3w/6N2rT8wqUAjzrnFppZLLDAzCY651b+eoJz7iXgJQAzuxJ42Dm3P901Ojjn9oa06gwwM0oUiCYqIoytgW0UKheLISpCd3hFRPymj9sywcGDB3nrrbfO67UDBw7k2LFjmVyRiMg5SkmCyf+EYZdDagrc9i1c/rzC3QVwzu1wzi0MfJ8IrALKnuUlvYBPQ1FbZikUE0XlYvlISUsjfvdRjiWl/OEc9ZEiIqGlgJcJ1HmJSLa2ezW8dwlMexka9IJ7ZkCl1n5XlaOYWSWgETDnDMdjgM7AmHRPO2CCmS0ws35nuXY/M5tvZvP37NmTiVVnTP48EVQtnp8wgw17jnL4ePLvjquPFBEJLQ3RzARPPvkk69evp2HDhlx22WWUKFGCzz77jJMnT3L11Vfz3HPPcfToUW644QYSEhJITU3l6aefZteuXWzfvp0OHTpQrFgxpkyZ4vdbEZHcJC0N5rwNk56DPPnhxhFQq5vfVeU4ZpYfL7g95Jw7fIbTrgRmnDI8s5VzbruZlQAmmtlq59zUU1/onBsKDAVvo/NMLj9DoiPDqVoiP5v2HWXzvqOUKZSXovm9bRTUR4qIhJYCXiZ48cUXWb58OYsXL2bChAl88cUXzJ07F+ccV111FVOnTmXPnj2UKVOG7777DoBDhw5RsGBBXn31VaZMmUKxYsV8fhcikqsc3Apj74FN06BGF7jqDchfwu+qchwzi8QLdyOcc1+e5dSenDI80zm3PfDnbjP7CmgK/CHgZRWR4WFUKZafrfuPse3gcZJS0yhVIFp9pIhIiOW8gPf9k7BzWeZes1Q96PJihk6dMGECEyZMoFGjRgAcOXKEdevW0aZNGx577DGeeOIJunXrRps2bTK3RhGRjHAOloyC7/8CLg2uGgSNbtam5UFg3gZx7wOrnHOvnuW8gkA74KZ0z+UDwpxziYHvOwH/uOCigtxHhocZFYvGsP3QCfYkniQpJY20tP/dVFQfKSISfDkv4PnMOcdTTz3F3Xff/YdjCxYsYPz48Tz11FN06tSJZ555xocKRSTXOroPvn0QVn0DFVpAj7ehSGW/q8rJWgE3A8vMbHHgub8CFQCcc0MCz10NTHDOHU332pLAV4FNxCOAkc65H0JR9IUyM8oUjCYqPIwdh46zf///5tCpjxQRCb6cF/AyeKctM8XGxpKYmAjA5ZdfztNPP02fPn3Inz8/27ZtIzIykpSUFIoUKcJNN91E/vz5GT58+O9eq+EnIhJUa3/0Ni0/fgAufQ5a3q9Ny4PMOTcd+NNbo8654cDwU57bADTI9KJC1EeaGcVj8xAVbhw6GMOBg4c5mZKqPlJEJARyXsDzQdGiRWnVqhV169alS5cu9O7dmxYtWgCQP39+PvnkE+Lj43n88ccJCwsjMjKSt99+G4B+/frRpUsXSpcurQnkIpL5Th6BH/8KCz+EEnXg5i+9IXUiIVAwJoqLq5enYZNm1K1bjyu6qo8UEQk2c86XBbfOW1xcnJs/f/7vnlu1ahW1atXyqaLQy23vV0TO05bZ3qblBzZ7d+w6/g0i8vhd1TkxswXOuTi/68gusmofeTI5lY37jpKS6ihfJC8F80YFpZ2s8F5FRELhbP2j7uCJiOQkKUmweQasGgcLhkPBcnDbd1Cpld+VSS6WJzKcasXzs2nfMTbvO0bpgo7isdnrwwYRkexCAU9EJLs7shvWTYC1P8D6nyEpEcLzeKtjdvoXRBcIeUnOOUwrc0o6EeFhVCmWj60HjrHj0HGSU9MoXTBaf09ERDKZAp6ISHaTlgY7l3gLp6z9EbYv9J6PLQP1roXql0OVdhCVL+SlOef4ZukO3p++kZF3NiNfHnUz8j9hYUaFIjHsOHSCvUe8bRQqFIkhLEwhT0Qks+SYnje3fFqc3eZMikgmOZkIG372At26CXBkF2BQLs6bW1f9cm/xFB9/Dm7ae5Snv17OtHV7qVe2IPuPJingZRFZqY80M8oUyktURBjbDx5nw96jVCwaQ2R42AVdV/2jiIgnR/S80dHR7Nu3j6JFi2aZDiwYnHPs27eP6Ohov0sRkVDYvwHWBoZebp4BqUmQpwBUu8QLdNUvg3z+Lx9/MiWVIT9vYPDP8USFh/HcVXW4qXlFwnVXJkvIqn1ksfx5iAwPY+v+Y6zfc4RKRfMRHXl+W3eofxQR+Z8cEfDKlStHQkICe/bs8buUoIuOjqZcuXJ+lyEiwZCaDFtm/W/o5b513vPFakDTflCjM1RoDuGR/taZzoz4vTw9djkb9h6lW/3SPN2tNiUL6JfsrCSr95GpKWnsPnqS7RuhSP485Ik4vzt56h9FRDw5IuBFRkZSuXJlv8sQETl3R/fCuomBBVJ+gpOHITwKKrWGJndCjU5QpIrfVf7BnsSTPP/dSsYu3k7FojF8eHtT2tUo7ndZchrZoY/csu8Ytw2fS8L+49zUvCL921ehRKw+KBAROR85IuCJiGQbzsHOpd7Qy3U/QsJ8wEH+UlC7u3eXrkp7yJPf70pPKzXNMXLuFv77w2pOJqfxQMdq3Nuh2nkPrRMBqFA0hi/vacm/vlvFh7M2MXLuZm5qVpG721XVdgoiIudIAU9EJNiSjsKGX7y7dOsmQuJ27/myjaH9U1DjcihVH8IubJGJYFu+7RD/N3Y5S7YepGXVovyzR12qFs+aQVSyn0IxUbx8fQMGdKjGoJ/iGTZjI5/M2czNzb2gVyy/gp6ISEYo4ImIBENKEiz6GNaMh43TIPUkRMVC1Q7eXbrql0H+En5XmSFHTqbw6oS1DJ+5kSL5ohh4Y0O6NyyTpRbskEwy5x1vxdbGfSFfUV9KqFQsH6/c0IABHasxaPI63p++kU9mb+GWFhXp17YKRRX0RETOSgFPRCSzpaXB1/fCss+hSNX/zaWr0BIiovyuLsOcc/ywfCfPfbOSXYkn6N20An+5vCYFY7LOIi+SybYtgKWjYepLUP9GaH4vlKjpSymVi+Xj1Rsbcl/Harz5UzzvTtvAR7M2c0vLitzdtipF8mWff0siIqFk2W3fmLi4ODd//ny/yxARObMf/w9mvQmXPANtHvW7mvOydf8xnvl6OVPW7KFW6QI8f3VdLq5QOOR1mNkC51xcyBvOpjKlj9y9Cma/7QW9lBNQtaMX9Kpe4usw4vjdRxj00zrGLdlO3shwbm1ZibvaVFHQE5Fc6Wz9owKeiEhmmjkIJvwNmt4NXf7j68bj5yMpJY13p23gjcnriAgzHul0Ebe2qEjEBW5Cfb4U8M5NpvaRR/fBgg9g7rtwZKe3XUez/tCgJ0Tly5w2zkP87kTemBzPN0u3E5Mu6BVW0BORXEQBT0QkFJZ+Dl/eCbV7wHXDICx7rSw5e8M+/jZ2OfG7j9ClbimeubI2pQvm9bUmBbxzE5Q+MiUJVo6FWYNhx2KILgSNb/P2ZixYNnPbOgfrdiXy+uR1fLdsBzGR4dzWygt6hWIU9EQk51PAExEJtvU/wYgbvI3I+3wBkdlnD699R07y7/GrGbMwgXKF8/LP7nXpUDNrLACjgHdugtpHOgdb53hBb/W3gEGdHt7wzXL+/S9a+2vQW7qD/Hki6NuqEne2rqK5oiKSo/kW8MysM/A6EA6855x78TTntAcGApHAXudcu7NdUwFPRLKc7YtgeDcoXAn6jofogn5XlCFpaY7P5m/lhe9XcywphX5tqzCgQ3XyRmWdO48KeOcmZH3kgc0wdygs/AhOHoZyTaD5PVCrO4T7s37bmp2JvD55LeOX7SQ2EPTuUNATkRzKl4BnZuHAWuAyIAGYB/Ryzq1Md04hYCbQ2Tm3xcxKOOd2n+26CngikqXs3wDvd4KIvHDHBChQ2u+KMmTVjsP831fLWLjlIE0rF+H5HnWpXjLW77L+QAHv3IS8jzyZCItHeouyHNgIBcpB07ug8a2QN/SL8oD3d/uNyev4fvlOYqMjuL1VZW5vXZmCeRX0RCTn8CvgtQCedc5dHnj8FIBz7oV059wLlHHO/S2j11XAE5Es48geeP8yOHHIC3fFqvtd0Z86ejKF1wN7ixXMG8lfu9bi2ovLZtk97RTwzo1vfWRaKqybALPfgo1TITIGGvb2FmXx6d/Fyu2HeX3yWn5csYvY6AjuaO0FvQLRCnoikv2drX8M5jiKssDWdI8TgGannFMDiDSzn4FY4HXn3EenXsjM+gH9ACpUqBCUYkVEzsnJIzDiOkjcCbd+ky3C3YQVO3l23Aq2HzpBr6bl+cvlNbXyoGSOsHC4qIv3tXMZzB7iDd+c9x5U7+QN36zSIaSrytYuU4B3bo5jxfZDvD5pHQMnrWPY9I3c2aYKfVtVIlZBT0RyqGAGvNP9FD/1dmEE0Bi4BMgLzDKz2c65tb97kXNDgaHgfToZhFpFRDIuJQk+u9n7RbbnSCjfxO+KzirhwDGeHbeSSat2UbNULG/0akRcpSJ+lyU5Val60GMwXPp3mD/MC3kfXw3Fa3lBr/4NEBm61VnrlCnI0FviWL7tEAMnrePViWt5f/pG7mxdmdsU9EQkBwrmxkYJQPl0j8sB209zzg/OuaPOub3AVKBBEGsSEbkwaWkwboC3auaVr8NFnf2u6IySU9MY8st6Lnt1KjPi9/LXrjX55v7WCncSGvlLQPsn4eEV0ONtCIuAbx6A1+rAT//y7n6HUN2yBXnv1ji+GdCaJpUK88rEtbT57xQGT4nnyMmUkNYiIhJMwQx484DqZlbZzKKAnsC4U875GmhjZhFmFoM3hHNVEGsSEbkwk5+FpaOh49/g4pv9ruaM5m/aT7c3pvPi96tpXb0Ykx5tR7+2VYn0acPy3MjMypvZFDNbZWYrzOzB05zT3swOmdniwNcz6Y51NrM1ZhZvZk+GtvpMFJHHm4/Xfxrc+i2Ubw5TX4bX6sKX/bxVaEOoXrmCvHdrE8YNaMXFFQrz0o9raP2fnxg8JZ6jCnoikgMEbYimcy7FzAYAP+JtkzDMObfCzPoHjg9xzq0ysx+ApUAa3lYKy4NVk4jIBZn1Fsx4HZrcCW0e87ua00o8kcx/fljNJ7O3ULZQXt69JY7Lapf0u6zcKgV41Dm30MxigQVmNjH9atIB05xz3dI/EViJejDpVqI2s3GneW32YQaV23hf+zfAnHdg0SfeByYVWnrDN2te4c3nC4H65Qox7LYmLN56kNcnreWlH9fwwYxNPH55Da5rXJ7wsKy58JCIyJ/RRuciIhmx7AsYcwfUuhKu/zBkv4Sei4krd/H02OXsTjxB31aVeeSyGuTL48+eZJklJ62iaWZfA2865yame6498NhpAt6frkR9OtmujzxxyAt5c4bAwS0QWwaKXwT5S0JsSchf6n9//vpcnuBs57FwywH+9e1KFm45SO3SBXi6W21aVC0alLZERC6UX6toiojkDBt+ga/6Q8VWcM17WS7c7U48wXPjVvLdsh3ULBXLkJsb07B8Ib/LknTMrBLQCJhzmsMtzGwJ3jz1x5xzK8jYStTZX3RBaHGft53CmvGw7HM4tA32xcORXZCa9MfXROZLF/pKQOyv4e+UP/MWgbCMD0m+uEJhxtzTkm+W7uA/36+m17uz6VS7JH/tWotKxfJl4psWEQkuBTwRkbPZsRRG9fG2Qeg5EiKj/a7oN845Pp+fwL++W8mJlDQev/wi+rWtonl2WYyZ5QfGAA855w6fcnghUNE5d8TMugJjgepkbCXqX6+f/bcSCgv37o7XuvJ/zzkHxw94QS9x5+//PLILEnfBruUQPxmSEk9zzQgv6P0W+kr8/o5gbMn/HQ/3VtI0M65qUIZOtUvy/vSNDJ4Sz2Wv/cJtLSsxoGN1bZYuItmCAp6IyJkc2OTtdRddEPp8AXkL+V3RbzbvO8pTXy5j5vp9NK1chBeuqUfV4vn9LktOYWaReOFuhHPuy1OPpw98zrnxZvaWmRUjYytR//q6nLmVkBnEFPG+StQ6+7lJRwPhbzcc2emFv9/+3OUN/9w6F47tPf3rY4p6oe/iW6B5f6Ijw7mvQzWub1yOlyes4b3pGxmzcBsPX1aDXk3KE6EPUUQkC1PAExE5naN74eNrIOUk3D4OCpb1uyIAUlLTeG/6Rl6buJao8DD+fXU9ejYpT5gWhMhyzMyA94FVzrlXz3BOKWCXc86ZWVO81a33AQcJrEQNbMNbibp3SArPjqLyQdGq3tfZpCb/LwQe2f37O4O7lsMPT3gf6DTsBUCJAtH897oG3NKiEv/8diVPj13Ox7M28X9X1KZdjeIheGMiIudOAU9E5FQnj8CI6+HwNrjlayhR0++KAFi+7RBPjFnKiu2H6VS7JP/oXpdSBbPOkFH5g1bAzcAyM1sceO6vQAXwVpMGrgPuMbMU4DjQ03mrn512JeoQ15/zhEd6H9ac7gOblCQYcS2Mux8KlvNW+wyoW7Ygo/o158cVu3jh+1XcOmwuHS4qzv9dUYtqJYKz6IuIyPnSKpoiIumlJsOnvWD9ZLjxE2/Zdp8dT0pl4KS1vDd9I0XyRfHP7nXoXLe032WFRE5aRTMU1EdeoOMHYdjlkLgD7pjoreh5ipMpqXw4cxODJsdzLDmVm5pV4KFLa1A4X1To6xWRXOts/aMGkYuI/Mo5GPcAxE+EK17NEuFuZvxeOr8+lXembuCGuHJMerhdrgl3IiGXtxD0/gzCo7y7+Ef2/OGUPBHh9GtblZ8fb0+vpuX5ePZm2r00hfenbyQpJS30NYuInEIBT0TkV5P/AUtGQvunIK6vr6UcPJbEX75YQu/35hBmxqd3NeeFa+pTMEar+IkEVeGK0Gu0N0fv056QfPy0pxXNn4d/9ajH9w+2pUH5Qvzz25VcPnAqk1buIruNjhKRnEUBT0QEYM47MP1VaHwbtHvCtzKcc3y7dDuXvvoLYxZu4972Vfn+wTbacFkklMo1hmvfhW0L4Mt+kHbmO3MXlYrlo9ub8sFtTTCDOz+az03vz2HVjlN3xBARCQ0FPBGRFV/B90/ARVdA11e85dl9sOPQce76aD4DRi6idMG8jBvQir90rkl0ZNbaWF0kV6h1JXT6F6waB5OeOeupZkaHmiX48aG2PHtlbVZsP8wVb0zjqS+XsffIyRAVLCLi0SqaIpK7bZzmfUJfvilc9z6Eh/7HYlqaY8SczfznhzWkpKXxtytqcVvLStprS8RvLe6DAxth5iAoXBma3HHW0yPDw7itVWV6NCrLG5Pj+WjWJr5Zsp0BHavRt1Ul8kTowxoRCT4FPBHJvXYuh1G9vV/ceo2CyLwhLyF+dyJPjlnG/M0HaFO9GM/3qEeFojEhr0NETsMMOv/H2yh9/GNQqAJUv+xPX1YoJopnrqxNn+YVeGH8Kl78fjUj5mzmqS616FK3FObTKAERyR308bCI5E4Ht8An10JUfrj5S4gpEtLmk1LSeGPyOrq+Pp11u4/w8vUN+Oj2pgp3IllNeARc9wGUrAOf3wY7l2X4pVWL5+e9W5vwyR3NyBcVwb0jFnLjO7NZlnAoePWKSK6ngCciuc+x/fDxNZBy3At3BcuFtPkFmw/QbdA0Xp24lsvrlmLSI+24rnE5faovklXlye9tn5CnAIy4AQ5vP6eXt65ejO8eaMO/r67H+j1HuGrwdB79bAm7Dp8IUsEikpsp4IlI7pJ0DEbe4N3B6zUKStQKWdNHTqbw7LgVXDdkJkdOpDDstjgG9WpE8dg8IatBRM5TgTLQ5zM4edj7GXIy8ZxeHh5m9G5WgSmPt6df2yp8s2Q77V/6mTcmr+N4UmqQihaR3EgBT0Ryj9QU+KKvt/T5de9DxZYha3rK6t1c/tpUPpy1iVuaV2TCI+3oWLNkyNoXkUxQqh5cPxx2rYQvbvd+ppyjAtGRPNWlFpMeaUeHmsV5deJaLnnlZ75evE3754lIplDAE5HcwTn49kFY+wN0fdlbAj0E9h05yQOfLqLv8HnERIXzRf+WPNe9LvnzaI0rkWyp+mXQ9SVYNwF+eML72XIeKhSN4a0+jRndrzlF8kfx4KjF3DtiIQePJWVywSKS2+g3DBHJHaY8D4s+gbZ/+dOlzjODc46vFm3jn9+u5MjJFB66tDr3tK+qZdJFcoImd/x++4SWA877Us2qFGXcfa15b/oGXvpxDYtfP8irNzSkRdWimViwiOQmuoMnIjnf3Hdh6kvQ6Gbo8NegN5eW5njum5U88tkSKhfLx/gH2vDQpTUU7kRykkv/AbWuggl/g1XfXNClwsKMfm2r8uU9rcgbGU7v92bz0o+rSU5Ny6RiRSQ3UcATkZxt5TgY/zjU6AzdBnr7WgVRaprjyS+XMnzmJm5vVZkv+rekesnYoLYpIj4IC4NrhkLZxjDmLkhYcMGXrFeuIN/c35obGpdn8JT1XDdkFpv3Hc2EYkUkN1HAE5Gca+NUGHMnlIvz9rEKD+6o9OTUNB4ctYjP5ifwwCXVebpbLcLCtPWBSI4VmddbjTd/Cfj0Rjiw+YIvmS9PBP+5rj5v9bmYjXuO0PX1aYxZkKAFWEQkwxTwRCTnST4OE56Gj7pD4Yre/lVRwd1A/ERyKvd8soBvl+7gqS41eeSyGtrXTiQ3yF8c+nwOqUkw4no4fjBTLtu1Xml+eKgtdcoW5NHPl/DgqMUcPpGcKdcWkZxNAU9EcpbNs+DtVjDzDWh0E9wxEWKKBLXJoydTuOPDeUxatZt/9qjL3e2qBrU9Ecliil8EN46A/Rvgs5shJXNWwixTKC+f3tWcxzrV4LtlO+gycBoLNu/PlGuLSM6lgCciOcPJIzD+L/BBF0hLhpvHwlWDIG+hoDZ76Hgytwyby6z1+3jl+gbc3LxiUNsTkSyqchu46g1vaPi3D5339gmnCg8zBnSszuf9WxAeZlw/ZBYDJ60lRQuwiMgZKOCJSPa34Wd4uwXMfQea3gX3zIKqHYLe7L4jJ+n97myWJhxkcO+LubZxuaC3KSJZWMPe0O4JWDwCpr6cqZe+uEJhvnugNT0almXgpHX0HDqbrfuPZWobIpIzKOCJSPZ14hCMe8CbaxcWCX2/9zYgzpM/6E3vOnyCG4fOJn73Ed69JY4u9UoHvU0RyQbaPwX1b4Qp/4Kln2fqpWOjI3n1xoYMvLEhq3cm0vX1aYxbsj1T2xCR7E8bnYtI9rT2R/jmITiyE1o+4O1vF5k3JE1v3X+MPu/NYd+Rk3x4e1OaV9GGxCISYOYNDz+UAF/fCwXLQsWWmdpEj0ZlaVyxMA+MWsQDny7ilzV7eK57HfLn0a91IqI7eCKS3RzbD1/2g5E3QHRBuGMSdPpnyMLd+j1HuOGdWRw8lsQndzZTuBORP4rIAzd+AoUqwqjesDc+05soXySGz+9uwQMdq/HVogSueGMai7cezPR2RCT7UcATkexj5dcwuCksH+PNc7n7FyjXOGTNr9pxmBvfmUVyahqj725BowqFQ9a2iGQzMUWgz2dgYTDiOji6N9ObiAgP45FOFzGqXwuSU9K47u2ZDJ4ST2qa9swTyc0U8EQk6zuyGz67xfuKLQ39fvaGZEbkCVkJi7cepOfQ2USGhzH67hbUKl0gZG2LSDZVpIq3Efrh7d6dvOQTQWmmaeUifP9gWy6vW4qXflxDn/dms+PQ8aC0JSJZnwKeiGRdzsGS0d5duzXfwyXPwF0/Qal6IS1j9oZ99Hl3NgXzRvLZ3S2oWjz4i7hI9mdm5c1sipmtMrMVZvbgac7pY2ZLA18zzaxBumObzGyZmS02s/mhrV4yTfmmcM07sHUOjL0H0oKzvUHBmEje7NWI/15Xn6UJh+g8cBo/LN8RlLZEJGvTbFwRyZoObYNvH4Z1P0K5JtB9sLeZcIhNWbOb/h8voEKRGD65sxklC0SHvAbJtlKAR51zC80sFlhgZhOdcyvTnbMRaOecO2BmXYChQLN0xzs45zJ/bJ+EVp2r4cAmmPQsFK4El/49KM2YGTfEladJpSI8OGoR/T9ZSK+mFXi6Wy1iovQrn0huoX/tIpK1OAcLP4IJf4PUZLj8BWh2N4SFh7yU75ft4IFRi6hRMpaP72hGkXxRIa9Bsi/n3A5gR+D7RDNbBZQFVqY7Z2a6l8wGtJliTtXqIdi/Eaa/6oW8xrcGranKxfLxRf+WvDpxLe9MXc+cjft4o2cj6pYtGLQ2RSTrCOoQTTPrbGZrzCzezJ48zfH2ZnYoMPxksZk9E8x6RCSLO7AJPu4B3zwApRvAPTOgxb2+hLsxCxK4b+RC6pcrxMi7mivcyQUxs0pAI2DOWU67A/g+3WMHTDCzBWbWL4jlSSiYwRWvQNWO3uiE9T8FtbmoiDCe7FKTT+5oxtGTKVz91gzem7aBNC3AIpLjBS3gmVk4MBjoAtQGeplZ7dOcOs051zDw9Y9g1SMiWVhaGswZCm+1hIT5cMWrcMs4KFrVl3I+nr2ZRz9fQouqRfno9qYUzBvpSx2SM5hZfmAM8JBz7vAZzumAF/CeSPd0K+fcxXj96H1m1vYMr+1nZvPNbP6ePXsyuXrJVOGRcP2HULwmfHYr7Fr556+5QK2qFeOHB9vS4aIS/Ou7Vdz6wVx2Hw7OYi8ikjUE8w5eUyDeObfBOZcEjAK6B7E9EcmO9sbD8K7w/eNQoTncOxua3AFh/qwB9c4v63l67HIurVWC929tQj5tHCwXwMwi8cLdCOfcl2c4pz7wHtDdObfv1+edc9sDf+4GvsLrV//AOTfUORfnnIsrXrx4Zr8FyWzRBbztEyJjYMT1kLgz6E0WzhfFOzc35vmr6zJv0346vz6Nyat2Bb1dEfFHMH+DKgtsTfc4IfDcqVqY2RIz+97M6gSxHhHJSlJTYMbrMKQV7F4JPd6Gm8ZAofK+lOOc49WJa3nh+9V0q1+at29qTHRk6IeGSs5hZga8D6xyzr16hnMqAF8CNzvn1qZ7Pl9gYRbMLB/QCVge/KolJAqWg96j4fgBGHkjJB0NepNmRp9mFflmQGtKFojmjg/n8/evl3MiOTXobYtIaAUz4Nlpnjt14PdCoKJzrgEwCBh72gtp+IlIzrJrJbx/GUx8BqpeAvfNhYa9vTkqPnDO8a/vVvHG5HXcEFeO13s2IjJcu8jIBWsF3Ax0TDfXvKuZ9Tez/oFzngGKAm+dsh1CSWC6mS0B5gLfOed+CPk7kOAp0xCuGwY7l8IXd0BaaIJW9ZKxfHVvS25vVZkPZ22m+5szWLMzMSRti0homHPBmWxrZi2AZ51zlwcePwXgnHvhLK/ZBMSdbUnouLg4N3++tgMSyZZSk2H6a/DLf71hSl1fgjrX+BbsAFLTHH8bu4xP527ltpaVeKZbbcLC/KtHfs/MFjjn4vyuI7tQH5kNzRnqDVFv1h86vxjSn4c/r9nNY58v4fCJFB7rVIPbW1UmQh9uiWQLZ+sfg/mveB5Q3cwqm1kU0BMYd0phpQJDWDCzpoF69v3hSiKS/W1fDEM7wJTnofZV3l27utf6Gu6SU9N45LPFfDp3KwM6VOPvVyrciUiINesHze+FOUPg6wGQcjJkTbe/qATfP9iWttWL8e/xq7nyzRks2nIgZO2LSHAEbfUA51yKmQ0AfgTCgWHOuRW/Dktxzg0BrgPuMbMU4DjQ0wXrlqKI+CP5BPzyH2++Xb7i0HMk1LzC76o4mZLKgJGLmLhyF3/pfBH3tq/md0kiklt1eh6i8sPU/8L+9XDDx5A/NAvmFI/Nw7u3xPHD8p08+80Krnl7Jjc1q8jjnS+iQLRWEBbJjoI2RDNYNPxEJBtJ3AkfdYc9q6HhTXD5vyBvYb+r4lhSCnd/vIBp6/by3FV1uLVlJb9LkjPQEM1zoz4ym1s+BsbeC/lKQO9RUDK0a88lnkjmlQlr+WjWJormz8Mz3WrTrX5pzMeRFiJyen4N0RSR3Cwlydvn6eAW6DMGegzOEuHu8Ilkbh02lxnxe3npuvoKdyKSddS9Fvp+D2nJ8H4nWD0+pM3HRkfy7FV1GHtfK0oWyMP9ny7i1g/msWXfsZDWISIXRgFPRIJjwt9g62y4ahBUv9TvagA4cDSJPu/OYdGWgwzqdTHXx/mzJYOIyBmVvRju+gmKVYdRvWHaqxDi0Vb1yxVi7L2teKZbbRZs2s9lr/3C4CnxJKWkhbQOETk/CngikvmWjIa570Dz+6DedX5XA8Duwye4cegs1uxKZOgtjbmifmm/SxIROb0CZbw7eXWvgcnPwVd3e/OZQygiPIzbW1dm0qPt6HBRCV76cQ1XvDGNeZv2h7QOETl3Cngikrl2LIVvHoSKreCy5/yuBoCEA8e44Z1ZJBw4zvC+TehYs6TfJYmInF1kXrj2fejwf7B0NHzYDRJ3hbyM0gXzMuTmxrx/axzHklK5fsgsnvhiKQePJYW8FhHJGAU8Eck8x/bD6JsgbyG4fjiE+78C28a9R7lhyCz2HU3i4zua0bJqMb9LEhHJGDNo9xe44SPYtQLe7QA7lvhSyiW1SjLxkbb0a1uFLxYm0PGVXxizIIHstlifSG6ggCcimSMtFb68Cw5vDyzxXcLvilizM5Hrh8ziZEoao/o1p3FF/xd5ERE5Z7W7w+0/eN8P6wwrv/aljJioCP7atRbfDGhNxaIxPPr5Evq8N4f1e474Uo+InJ4Cnohkjp9fhPhJ0PW/UL6J39Wwasdher07m/AwGH13c+qUKeh3SSIi5690A7hrird1wme3wC//DfniK7+qXaYAY/q35F896rJs2yG6DJzGaxPXciI51Zd6ROT3FPBE5MKtHu9t0NvwJmjc1+9qWLXjMH3em0NUeBij+7WgWolYv0sSEblwsSXh1m+h/o0w5Xn44nZIPu5LKWFhxk3NKzL50XZ0rluK1yevo+vr05gZv9eXekTkfxTwROTC7I33Vngr3RCueNmbM+Kj9OFuVL/mVCqWz9d6REQyVWQ0XP0OXPosrPgKPugCh3f4Vk6J2Gje6NWIj25vSkqao/d7c3h49GL2HjnpW00iuZ0Cnoicv5NHvEVVwiLgxo+9Vd98pHAnIrmCGbR+GHqOhD1rvcVXti3wtaS2NYoz4eG2DOhQjW+XbueSV35h1NwtpKVpERaRUFPAE5Hz4xyMGwB718B1w6BQBV/LUbgTkVynZle4YwKERcIHXWH5GF/LiY4M57HLL+L7B9twUalYnvxyGTcOncXaXYm+1iWS2yjgicj5mfWmNzzokmegagdfS1G4E5Fcq1Rd6DcFyjTy5uT99DykpflaUrUSsYzu15z/Xlef+N1H6Pr6NP7zw2qOJ2kRFpFQUMATkXO3cSpM/DvUugpaPeRrKQp3IpLr5SsGt3ztLXQ19b/w+a2QdNTXksyMG+LKM/nR9vRoVJa3f15Pp4G/8POa3b7WJZIbKOCJyLk5lACf94WiVaHHW74uqqJwJyISEJEHur8JnZ6H1d/CsMu9n9c+K5Ivipevb8CndzUnMjyM2z6Yx30jF7L78Am/SxPJsRTwRCTjUk56+y+lnIQbR0Ae/7YfULgTETmFGbQcAL1Gw4HNMLQDbJ3nd1UAtKhalO8fbMMjl9Vg4spdXPLKL3w0axOpWoRFJNMp4IlIxn3/F2+ltqvfhuI1fCtD4U5E5CxqdII7JkJUDAy/ApaM8rsiAPJEhPPAJdX58aG2NChfiGe+XsE1b89kxfZDfpcmkqMo4IlIxiz4EBYMh9aPQK0rfStD4U5EJANK1IS7pkD5pt5epRP/7vviK7+qXCwfH9/RlNd7NmTbgWNcOWg6f/96OYeOJ/tdmkiOkKGAZ2ZjzOwKM1MgFMmNEhbA+MegSgfo+DffylC4ExE5BzFF4OavoHFfmDEQRveBk1ljywIzo3vDskx+pD03Na/Ix7M30/Hln/ls3lbtnSdygTIa2N4GegPrzOxFM6sZxJpEJCs5utebd5e/FFz7PoSF+1KGwp2IyHkIj4Rur0GX/8LaH+D9y735eVlEwZhI/tG9Lt/c35rKxfLxlzFLuebtmSxNOOh3aSLZVoYCnnNuknOuD3AxsAmYaGYzzayvmUUGs0AR8VFqCnzRF47thRs/hnxFfSlD4U5E5AKYQbO7oc8X3sqa73aAzbP8rup36pQpyOf9W/DqDQ1IOHCc7oNn8NSXyzhwNMnv0kSynQwPuTSzosBtwJ3AIuB1vMA3MSiViYj/Jj/n7XnX7TUo09CXEhTuJLsys/JmNsXMVpnZCjN78DTnmJm9YWbxZrbUzC5Od6yzma0JHHsytNVLjlTtErhrMkQXgg+vhEWf+F3R75gZ11xcjp8ea8ftrSrz2fytdHjlZz6ZvVmrbYqcg4zOwfsSmAbEAFc6565yzo12zt0P5A9mgSLikxVfwcw3IO4OaNjblxIU7iSbSwEedc7VApoD95lZ7VPO6QJUD3z1w5sSgZmFA4MDx2sDvU7zWpFzV6y6F/IqtYKv74Mf/w/SUv2u6ncKREfydLfafP9gG2qWiuVvY5fTffB0Fmw+4HdpItlCRu/gvemcq+2ce8E5tyP9AedcXBDqEhE/7V4NY++Dck2g84u+lKBwJ9mdc26Hc25h4PtEYBVQ9pTTugMfOc9soJCZlQaaAvHOuQ3OuSRgVOBckQuXtzD0GQNN+8GsN2HkjXA864WnGiVj+fSu5gzq1Yi9iUlc+/ZMHvt8CXsST/pdmkiWltGAV8vMCv36wMwKm9m9wSlJRHx14pC30lpUPrjhI4iICnkJCneS05hZJaARMOeUQ2WBrekeJwSeO9PzIpkjPAK6vgRXvAobpsDA+jDxGTi8489fG0JmxpUNyjD50Xb0b1eVrxdvo+MrP/PBjI2kpGaNbR9EspqMBry7nHMHf33gnDsA3BWUikTEP2lp8NU9sH8jXD8cCpQJeQkKd5LTmFl+YAzwkHPu8KmHT/MSd5bnT3f9fmY238zm79mz58KKldynyR3efnnVLoWZg2BgPW/o5p41flf2O/nyRPBkl5r88FBbGpYvxHPfrKTboOnM2bDP79JEspyMBrwwM/utswnMDQj9x/oiElzTX4U138Hlz3vzM0JM4U5ymsBK02OAEc65L09zSgJQPt3jcsD2szz/B865oc65OOdcXPHixTOncMldSteH6z+A+xdC41th2RcwuCl82gu2zPa7ut+pWjw/H93elCE3NSbxRAo3Dp3Ng6MWsevwCb9LE8kyMhrwfgQ+M7NLzKwj8CnwQ/DKEpGQi58EP/0L6l0PzfqHvHmFO8lpAh+Mvg+scs69eobTxgG3BFbTbA4cCsx1nwdUN7PKZhYF9AycKxI8RSrDFa/Awyug3ROwZRYMuxze7wSrx3ujPLIAM6Nz3VJMeqQdD3SsxvfLd9Lx5Z8ZOnU9SSlZo0YRP5lzf77srJmFAXcDl+ANG5kAvOecC/myS3FxcW7+/PmhblYkZzuwCd5pBwXLwR0TvPl3IaRwJ2diZguy62JeZtYabwXqZcCvv3X+FagA4JwbEgiBbwKdgWNAX+fc/MDruwIDgXBgmHPu+T9rU32kZKqko95WCjPfhENboFgNaPkA1L8BIvL4Xd1vNu87yj++Wcnk1bupWjwfz11Vl9bVi/ldlkhQna1/zFDAy0rUeYlksqRjMKwTHNwC/X6GIlVC2rzCnZxNdg54flAfKUGRmgIrx8KMgbBzGeQvBc3vgbi+EF3Q7+p+M3nVLp77ZiVb9h+ja71S/N8VtSlbKK/fZYkExdn6x4zug1fdzL4ws5VmtuHXr8wtU0RCzjn49mHYuRyueU/hTkRE/ig8AupdB3dPg5u/ghI1YdLf4dU6MOFpOHza6aEhd0mtkkx4uC2PXlaDn1bv5tJXfmHwlHhOpmStff5Egi2jc/A+wNt8NQXoAHwEfBysokQkROa9B0tHQfunoEankDatcCfZiZk9aGYFAnPl3jezhWYW2n80In4zg6od4Zavod8vXr8x601vi4Wx93l7qPosOjKc+y+pzqRH2tGuRnFe+nENl782lSmrd/tdmkjIZDTg5XXOTcYb0rnZOfcs0DF4ZYlI0G2ZDT88CTU6Q9vHQ9q0wp1kQ7cHtjjoBBQH+gIv+luSiI/KNITrhsEDi7yhmsvHwFvNYGRP2DzLGyHio3KFYxhyc2M+ur0pYWFG3+HzuPPDeWzZd8zXukRCIaMB70RgoZV1ZjbAzK4GSgSxLhEJpsSd8NmtULA8XP0OhGX0R8GFU7iTbOrXrYK6Ah8455Zw+r3qRHKXwpW8DdMfXuGNBtk6Bz7o7K28uepb31febFujOD882JYnu9Rk5vp9XPraL7w6cS0nkjVsU3KujP5W9xAQAzwANAZuAm4NUk0iEkwpSV64O3kYeo6AvIVC1rTCnWRjC8xsAl7A+9HMYvnfypgikq8otH/SC3pdX4YjO2F0H28/vQUfQspJ30qLigijf7uqTH60HZfXKcUbk9dx6au/8OOKnWS3xQZFMuJPA15gU/MbnHNHnHMJzrm+zrlrnXN/uvOlmXU2szVmFm9mT57lvCZmlmpm151j/SJyrib8DbbOhqsGQck6IWtW4U6yuTuAJ4EmzrljQCTeME0RSS8qBpreBfcvgmvfh8i88M0DMLAeTHsVjh/0rbTSBfMyqFcjPr2rOTFR4dz98QJu+2AeCQc0bFNylj8NeIG97hoH9urJsEAwHAx0AWoDvcys9hnO+w/eZuoiEkxLRsPcd6D5fd6KaCGicCc5QAtgjXPuoJndBPwNOORzTSJZ128rb071FmUpWQcmPwev1fU+aDy0zbfSWlQtyncPtOHpbrVZsPkAVw6azoz4vb7VI5LZMjpEcxHwtZndbGbX/Pr1J69pCsQ75zY455KAUUD305x3PzAG0PJGIsG0Yyl88yBUbA2XPReyZhXuJId4GzhmZg2AvwCb8VaUFpGzMYMq7b3tFe6eCjUuh1lvwesNYOy9sHuVL2VFhodxR+vKjBvQimL583Dz+3MYOnW9hmxKjpDRgFcE2Ie3cuaVga9uf/KassDWdI8TAs/9xszKAlcDQzJYh4icj2P7YfRNkLcwXP8BhEeGpNllCYcU7iSnSHHeb37dgdedc68DsT7XJJK9lG4A170PDyyEuNth+ZfwVnMY1QeO7PGlpCrF8/PVfa24vE4p/j1+Nfd/uohjSSm+1CKSWSIycpJz7nzmGZxuSOepH4sMBJ5wzqWebQSomfUD+gFUqFDhPEoRycXSUuHLu7yNaPt+D/lDswDuzPV76ffRAgrmjWTEnc0U7iS7SzSzp4CbgTaB6QWh+aREJKcpXAm6/hfaPQHz3oXpr8E7beC6D6Bii5CXkz9PBG/1uZi3f1nPSz+uIX73Ed65uTEVi6rfkuwpQ3fwzOwDMxt26tefvCwBKJ/ucTlg+ynnxAGjzGwTcB3wlpn1OPVCzrmhzrk451xc8eLFM1KyiIC3PPUPT0L8JK8zLd8kJM3+uGIntw2bR+mC0Yy5p6XCneQENwIn8fbD24k3IuUlf0sSyeZ+XXnzzkneYizDr4AZr/uyh56ZcW/7agzv25Qdh05w5aDp/LxGs4cke8roEM1vge8CX5OBAsCRP3nNPKC6mVU2syigJzAu/QnOucrOuUrOuUrAF8C9zrmxGS9fRM4o+QSMuR3mDoUWA6BxaBb8+2z+Vu75ZAF1yhbg8/4tKFUwOiTtigRTINSNAAqaWTfghHNOc/BEMkOpetDvF6jVDSY+A6N6w/EDvpTSrkZxvhnQmjKF8tJ3+DwGT4nXvDzJdjIU8JxzY9J9jQBuAOr+yWtSgAF4q2OuAj5zzq0ws/5m1v9CCxeRszh+AD65BlZ8BZ3+5X2d20K452Xo1PX85YultKpWjBF3NqNQTFTQ2xQJBTO7AZgLXI/XB87R1j4imSi6AFz/IXT+D6ybAO+0he2LfCmlQtEYvry3JVfWL8NLP67hnk8WcuSk5uVJ9mHn86mEmV0EfOecq5b5JZ1dXFycmz9/fqibFck+Dm6FT66FAxuhx9sh2Q7BOcd/fljDkF/W061+aV69oSFRERkdICByZma2wDkXlwXqWAJc5pzbHXhcHJjknGvgb2W/pz5ScoSt8+Dz2+Dobuj8AsTdEZIPKU/lnOP96Rv59/hVVCmen3dubkzV4vlDXofI6Zytf8zoHLxEMzv86xfwDfBEZhYpIplgx1J471JI3Ak3fRmScJeSmsaTY5Yx5Jf13NS8Aq/3bKRwJzlR2K/hLmAfGZ/mICLnonwT6D8NKreD7x6FMXfCyT+bGZT5zIw721Thkzuasf9oEj3enMGklbtCXofIucroEM1Y51yBdF81nHNjgl2ciJyD9T/BB10hLBzu+BEqtwl6kyeSUxkwchGj52/lgY7V+Gf3uoSHhf5TVpEQ+MHMfjSz28zsNrw56eN9rkkk54opAr0/g45Pw4ov4d0OsGulL6W0rFaMcQNaUbFYDHd+NJ/XJq4lLU3z8iTryugdvKvNrGC6x4VOt9qliPhkySgYcT0UruitRlaiVtCbPHIyhduHz+OHFTt5plttHul0EWfb7kQkO3POPQ4MBeoDDYChzjmNZBEJprAwaPsY3PI1HD8I73aExZ/6Ukq5wjF80b8l11xcltcnr+Ouj+Zz+ESyL7WI/JmMDi/5u3Pu0K8PnHMHgb8HpSIRyTjnYOrL8NXdULEV9B0PBcoEvdl9R07S+93ZzNm4n9dubMDtrSsHvU0RvwUWGnvEOfewc+4rv+sRyTUqt/WGbJZtDGP7w7j7Ifl4yMuIjgznlesb8NxVdfhl7R56vDmDdbsSQ16HyJ/JaMA73XkZ2iRdRIIkNQW+ewR++ifUuwH6fAHRBf/8dRdo28HjXP/OLNbsTOTdWxpzdaNyQW9TxC+nzkFP95UYmJMuIqEQW8q7k9fmUVj4Ebx/GexbH/IyzIxbW1Zi5F3NOXwimR6DZ/DD8h0hr0PkbDIa8Oab2atmVtXMqpjZa8CCYBYmImeRdBRG3wTzh0Hrh+GaoRAR/C0J4ncnct3bM9mTeJKP72hGx5olg96miJ9OMwf9169Y51wBv+sTyVXCI+CSZ6D353AoAYa2h5Xj/vRlwdC0chG+vb8N1UvG0v+Thfz3h9Wkal6eZBEZDXj3A0nAaOAz4DhwX7CKEpGzOLIHPrwS1v0IXV+GS58NyfLRS7Ye5Pohs0hOdYzu14KmlYsEvU0REZE/qNEJ7p4KxarDZzfDD09BSlLIyyhVMJrRdzenV9PyvPXzevoOn8fBY6GvQ+RUGV1F86hz7knnXFzg66/OuaPBLk5ETrFvvTcsZddKuPETaHpXSJqdEb+XXu/OJn90BGPuaUHtMrpxISIiPipUAfr+AE3vhtlvwfCu3l29EMsTEc4L19Tn31fXY9b6vVz15gxW7dDobfFXRlfRnGhmhdI9LmxmPwatKhH5o4T5Xrg7eRhu/QZqXhGSZr9ftoO+H8yjQpEYxvRvScWi+ULSroiIyFlFREHX/8L1w2H3ahjSBtZN8qWU3s0qMKpfC04kp3LNWzMZt2S7L3WIQMaHaBYLrJwJgHPuAFAiKBWJyB+t+R6Gd4M8sXDHRG8T2BD4dO4W7hu5kHrlCjK6XwtKFIgOSbsiIiIZVudq6Pezt4r0iOvgp39BWmrIy2hcsTDf3t+aOmUK8MCni/j3+FWkpKaFvA6RjAa8NDOr8OsDM6sEaCapSCjMHwajekOJml64K1o16E0653jr53ie+nIZbWsU55M7mlEwJjLo7YrkJGY2zMx2m9nyMxx/3MwWB76Wm1mqmRUJHNtkZssCx+aHtnKRbKhYNW8f2EZ9YOpL8HEPOLI75GWUKBDNyLuac0uLigyduoFbP5jL/qOalyehldGA93/AdDP72Mw+Bn4BngpeWSKCczD5H/Dtw1DtMrjtO8gf/Bvnzjn+PX4V//1hDd0bluHdW+LIGxUe9HZFcqDhQOczHXTOveSca+ica4jXp/7inNuf7pQOgeNxwS1TJIeIzAvdB0P3t2DrPG/I5qYZIS8jKiKMf3Svy0vX1WfepgNcOWg6y7cd+vMXimSSjC6y8gMQB6zBW0nzUbyVNEUkGFKSYOw9MO0VuPhW6DkSooI/9y0lNY3Hv1jKu9M2cmuLirx2Q0MiwzP6OZCIpOecmwrs/9MTPb2AT4NYjkju0agP3DUZ8uSHD7vBtFchLfRDJa+PK88X/VvgnOPat2fy5cLQLwIjuVNGF1m5E5iMF+weBT4Gng1eWSK52InDMPJ6WPIpdPgbXPm6t/dPsJtNTuWeEQv5YkECD19ag2evqkNYWPC3XxDJ7cwsBu9O35h0TztggpktMLN+/lQmko2VrAN3TYHa3WHyczCqFxzL6Octmad+uUKMu781jSoU4pHPlvDsuBUka16eBFlGP5p/EGgCbHbOdQAaAXuCVpVIbnV4B3zQFTZN94aYtHs8JHvcHT6RzK3D5jJp1S7+0b0OD15aHQtBuyICwJXAjFOGZ7Zyzl0MdAHuM7O2Z3qxmfUzs/lmNn/PHnXNIr+JLgDXfQBdXoL4yfBOO9i2IORlFMufh0/uaMYdrSszfOYm+rw3hz2JJ0Neh+QeGQ14J5xzJwDMLI9zbjVwUfDKEsmFdq/2tkE4sBF6j/aGmITA3iMn6TV0Ngs2H2DgjQ25pUWlkLQrIr/pySnDM51z2wN/7ga+Apqe6cXOuaG/7lNbvHjxoBYqku2YQbN+cHtgd6/3L4c5Q7157iEUER7G091qM/DGhixNOMiVg6azeOvBkNYguUdGA15CYB+8scBEM/sa0AYfIpll0wwY1glSk6DveKh2aUia3br/GNcPmcX6PUd479Y4ujcsG5J2RcRjZgWBdsDX6Z7LZ2axv34PdAJOuxKniGRQucZw9y9QtSN8/zh8cTucTAx5GT0alWXMPS2JCDduGDKLz+ZtDXkNkvNlaGKPc+7qwLfPmtkUoCDwQ9CqEslNVnwFX/aDwpWgzxdQuGJIml27K5Fb3p/LsaQURtzZjMYVi4SkXZHcwsw+BdoDxcwsAfg7EAngnBsSOO1qYIJz7mi6l5YEvgoMk44ARgYWOxORCxFTBHqNghkD4ad/ws6lcMNH3ny9EKpTpiDfDGjNA6MW8ZcxS1mx/RB/61Zbi5pJpjEX4lvUFyouLs7Nn68tgSSHmDUYfvwrVGjhrZQZE5qQtXDLAfp+MI88EWF8dEdTapYqEJJ2Rc6VmS3QNgEZpz5SJIM2TYcv7oATh+CaoVD7qpCXkJKaxn9+WM270zbSvEoR3urTmCL5okJeh2RPZ+sf9VGBiB/S0uCHp7xwV+squHlsyMLd1LV76PPuHArFRDLmnpYKdyIikvtUag39p0GpevD5bbDsi5CXEBEexv9dUZtXb2jAwi3evLyV2w+HvA7JeRTwREIt+QR8cRvMfgua3QPXfwiR0SFp+tul27njw3lUKpaPL/q3pHyRmJC0KyIikuXkLwE3f+WNovnyLlg80pcyrrm4HJ/f3YLUNG+/vO+W7vClDsk5FPBEQunYfvi4B6z8Gi7/N3R5EcJC88/w49mbuf/TRTQqX5hR/ZpTPDZPSNoVERHJsvLkhz6fQ+V2MPZeWDDclzIalC/EuPtbUat0LPeNXMgrE9aQlpa9plFJ1qGAJxIqB7fAsM7eHjzXfQAt7gtJs845Bk1ex9Njl3NJzRJ8dEdTCuaNDEnbIiIiWV5UjLf4SvXL4JsHvW0UfFAiNppP+zXnxrjyDPopnn4fzyfxRLIvtUj2poAnEgo7lsB7l8KRnd58u7rXhKTZjXuPMuDTRbwycS3XXFyWt29qTHRkeEjaFhERyTYio+HGT6BmN28bhZmDfCkjT0Q4L15bj+euqsOUNXu4+q2ZbNx79M9fKJKOAp5IMO3fAN8+Au93grBIb6PVSq2C3uy2g8d54oulXPrqL/y0ajcPX1qDl69roCWYRUREziQiD1w/HOpcDRP+BlNf9qUMM+PWlpX4+I6m7Dtyku5vTueXtXt8qUWypwztgyci52j7IpjxujfXLiwCGvSCDn+F2FJBbXZ34gnemrKekXO2AHBLi4rc276a5tuJiIhkRHgkXPMehEd5e+WlJkH7p8DblzKkWlYtxrgBrbnro/n0/WAuT3apyV1tqmA+1CLZiwKeSGZxDjZMgekDYeMvkKcAtHoQmvUPerA7cDSJIVPX8+HMTSSnOm6IK8f9HatTplDeoLYrIiKS44RHQI+3vbD3y38g5SRc+qwvIa98kRi+vLclj32+hH+PX83K7Yd58dr6mm4hZ6WAJ3KhUlNg5Vjvjt3OpRBbGi77JzS+DaKDu8dc4olk3p++kfenbeRIUgrdG5ThoUtrUKlYvqC2KyIikqOFhcOVgyA8D8wY6IW8zi/4EvJioiIY3PtiBk+J5+UJa1m/5yhDb2lM6YL6EFdOTwFP5HwlHYPFI7yJ2Ac3Q7EacNWbUP8Gbxx/EB1PSuXDWZsY8st6Dh5LpnOdUjzSqQY1SsYGtV0REZFcIywMrnjF69Nnv+UN1+z6csi2N0rPzBjQsToXlSrAw6MXc+WgGQy56WLiKhUJeS2S9SngiZyrY/th7rsw9x04tg/KNfU+1avRJeg/9E+mpDJq7lbenBLPnsSTtKtRnMc6XUS9cgWD2q6IiEiuZObtWxse5d3JSz0JV77h3eHzwWW1S/LVvS2566P59Hp3Nv/oXpdeTSv4UotkXQp4Ihl1cAvMGgwLP4LkY16ga/UgVGwR9KZTUtMYszCBNybHs+3gcZpVLsJbfS6miT65ExERCS4zbw5eRB5vTl5qMnR/y5ur54PqJWP5+r7W3D9qEU99uYxVOw7zdLfaWilbfqOAJ/Jndi735tctH+P9kK93A7R6AErUCnrTaWmOb5ZuZ+CkdWzce5QG5Qry4rX1aF2tmFbREhERCRUzbzXsX1fXTDkJ177nLcTig4IxkXxwWxP++8Nq3pm6gTU7E3mrz8UUza9Vs0UBT+T0nINN073hGPGTICo/NL/H+ypYLgTNOyau3MWrE9eyemciNUvFMvTmxlxWu6SCnYiIiF/aPubdyZvwN+9O3vUfBH3e/ZmEhxlPda1FzdKxPDFmGVe9OYOhtzSmThlN28jtFPBE0ktLhdXfenfsti2AfMWh49PQ5A7IWzjozTvnmLZuL69MWMOShENULpaPN3o1olu90oSFKdiJiIj4ruX93uqa3z8Oo2+CGz6GyGjfyrm6UTmqFs9Pv48WcN3bs3j5+gZcUb+0b/WI/4Ia8MysM/A6EA6855x78ZTj3YF/AmlACvCQc256MGsSOa3kE7DkU29FzP3roUgV6PYaNOgdsh/a8zbt56Uf1zB3437KFsrLf6+tzzUXlyVCY+pFRESylmb9vOGZ3z4Mn94IPT+FqBjfyqlfrhDj7m/FPZ8s5L6RC1m1oxqPXFZDHw7nUkELeGYWDgwGLgMSgHlmNs45tzLdaZOBcc45Z2b1gc+AmsGqSeQPjh+E+e/D7CFwdDeUaQTXfwi1rgzZClnLEg7x8oQ1/LJ2D8Vj8/DcVXXo2bQ8eSK0iamIiEiWFdfXG5759X0w4nroPRry5PetnBKx0Yy8qxl//3oFb06JZ9WOw7zWsyEFov2ZJyj+CeYdvKZAvHNuA4CZjQK6A78FPOfckXTn5wNcEOsR+Z/D270VMRcMh6QjUPUSaP0QVGoTsk1M1+5K5NUJa/lhxU4KxUTyVJea3NKiEnmjFOxERESyhYa9vYVXvuwHn1wDfT6HaP/mwOWJCOeFa+pRp0wBnvtmJVcPnsG7t8RRpbh/wVNCL5gBryywNd3jBKDZqSeZ2dXAC0AJ4Iog1iMCe9Z48+uWfgYuDepeAy0fgNL1Q1bCpr1HGThpLV8v2U6+qAgeurQ6d7SuTKw+YRMREcl+6l3nDdf84nb4qAfc/GVI5u2fiZlxc4tKVCsRy30jF9J98AwG9WpE+4tK+FaThFYwA97pboP84Q6dc+4r4Csza4s3H+/SP1zIrB/QD6BCBW3mKOdh33pvxas14yEiL8TdDi3ug8IVQ1bC9oPHGfTTOj6bn0BkuNGvbRX6t61K4XxRIatBREREgqB2d7jxE/jsFvjwSrj5a8hX1NeSWlQtytf3taLfxwu4ffg8nuhck35tq2g17lwgmAEvASif7nE5YPuZTnbOTTWzqmZWzDm395RjQ4GhAHFxcRrGKedm/wYYfoW3OXm7J6Fpv5D+0D16MoVXJ67l41mbAbi5eUXu7VCVErH+rbglIiIimeyiLt5iK6P7wIfd4JavIb+/d83KF4lhzD0tePzzpbzw/WpW7jjMf66tT3SkpoPkZMEMePOA6mZWGdgG9AR6pz/BzKoB6wOLrFwMRAH7gliT5DYHt8CHV3kbkvb9AUrWDmnzC7cc4JHRi9m8/xg3NC7PA5dWp2yhvCGtQUREREKk+qXQ+zP4tKf34fIt46CAv1sWxERF8GbvRtT+uQAvT1jDhj1HeefmxpTR7yM5VtDWX3fOpQADgB+BVcBnzrkVZtbfzPoHTrsWWG5mi/FW3LzROac7dJI5Dm/3wt2Jw3DzVyENd8mpabw6YQ3XvT2T5FTHqLua85/r6ivciYiI5HRV2sFNY7zfQz7oAge3/vlrgszMuK9DNd67JY6Ne49y1ZvTmbdpv99lSZBYdstTcXFxbv78+X6XIVndkd3wQVdI3OENkSgXF7Km1+85wsOjF7M04RDXXlyOZ6+qrQVURM6TmS1wzoXuH3A2pz5SJAvZOg8+uRbyFoRbv4HClfyuCID43Ync9dECEg4c45lutbmpeUXNy8uGztY/agdlyXmO7fdWsTq8zVuuOEThzjnHR7M2ccUb09i6/xhv97mYV25ooHAnkkuZ2TAz221my89wvL2ZHTKzxYGvZ9Id62xma8ws3syeDF3VIpJpyjeBW7/2RhJ90NVb8C0LqFYilrH3taJ1tWI8/fUKHhq9mGNJKX6XJZlIAU9yluMH4eMesC8een0KFVuGpNldh09w6wfzeObrFTSvUpQfH2pLl3r+jrkXEd8NBzr/yTnTnHMNA1//ADCzcLxpC12A2kAvMwvtBGIRyRxlGsFt30LKCW+45p41flcEQMG8kbx/axMe61SDcUu20/3NGcTvPvLnL5RsQQFPco6TiTDiOti1EnqOgCrtQ9Ls+GU7uHzgVOZu3Mc/e9Tlg9uaUKKAVsgUye2cc1OB85nk0hSId85tcM4lAaOA7planIiETql6cNt34Jx3J2/naW/qh1xYmDGgY3U+vr0Z+44m0f3N6Xy3dIffZUkmUMCTnCHpKIy4AbYthOuHQ/XLgt7k4RPJPDJ6MfeOWEjFIjGMf6ANN2scu4icmxZmtsTMvjezOoHnygLpV2VICDwnItlViVrQ93sIj/K2UNi+2O+KftO6ejG+e6A1F5XyNkb/xzcrSU5N87ssuQAKeJL9JZ+AUb1h62y49l2o1S3oTc7esI8uA6fx9ZLtPHhJdb64pyVViucPersikqMsBCo65xoAg4CxgedP9ynRGVdEM7N+ZjbfzObv2bMn86sUkcxRrBr0/Q6i8sNHV0FC1lkQqXTBvIzq14LbWlZi2IyN9Bo6m52HTvhdlpwnBTzJ3lKS4LObYcPP0H0w1L02qM2dTEnl3+NX0evd2URFhPFF/xY8fFkNIsP1T0lEzo1z7rBz7kjg+/FApJkVw7tjVz7dqeWA7We5zlDnXJxzLq548eJBrVlELlCRKtB3POQt7C0It3mW3xX9JioijGevqsOgXo1YueMw3QZNY2b8Xr/LkvOg30ol+0pNhi/6wroJ0G0gNOwd1OZW7ThM9zdnMHTqBvo0q8B3D7SmUYXCQW1TRHIuMytlgTHdZtYUr0/eB8wDqptZZTOLAnoC4/yrVEQyVaEK3nDN2JLecM2Jf4ekY35X9ZsrG5Rh3IBWFIqJ4qb35zB4SjxpadlrW7XcTgFPsqe0VPjqblj9LXT+D8T1DVpTqWmOd35ZT/c3Z7D3SBIf3NaEf/WoR0xURNDaFJHsz8w+BWYBF5lZgpndYWb9zax/4JTrgOVmtgR4A+jpPCnAAOBHYBXwmXNuhR/vQUSCpEAZuGMi1O8JMwbCW81g7QS/q/pNtRKxfH1fK7rVL8NLP67hro/mc+hYst9lSQZpo3PJftLS4Ov7YMlIuPQ5aP1Q0JpKOHCMRz9bwpyN+7m8TkleuKY+RfJFBa09Efk9bXR+btRHimRDm2bAtw/D3jVQuzt0ftELgFmAt8fvZv713UpKFYzm7T6NqVu2oN9lCdroXHIS5+C7R7xw1/6poIU75xxjFiTQZeA0Vmw/zEvX1WfITY0V7kRERCRzVWoF/adDx6dh7Y/wZlOYPcQbreQzM+PWlpUYfXcLUlId17w9k9HztvhdlvwJBTzJPpyDH56CBR9A64eh3RNBaebA0STuG7mQRz9fQq3SBfj+wTZcH1de2x+IiIhIcEREQdvH4N5ZUL4p/PAEvNsRti/yuzIALq5QmG/vb03TSkV4YswyHv98CSeS/Q+gcnoKeJI9OAeTnoU5b0Oze+CSv0MQAtfPa3Zz+cCpTFy5iye71OTTfs0pXyQm09sRERER+YMiVeCmMXDdMEjc4YW875+AE4f9royi+fPw4e1NeaBjNT5fkMDVb81k096jfpclp6GAJ9nDL//xJiHH3Q6dX8j0cHc8KZWnxy7ntg/mUSgmkrH3taJ/u6qEh+munYiIiISQmbft04B5EHcHzHkHBjeFFWO9D7x9FB5mPNLpIj64rQnbDx7nyjenM2HFTl9rkj9SwJOsb/pA+PkFaNgHur6S6eFuydaDXPHGND6evZk7W1dm3IDW1CmjCcQiIiLio+iCcMXLcOdkyFcMPr8VRt4ABzb7XRkdapbg2/tbU6loPvp9vIAXvl9FSmqa32VJgAKeZG2zh8Ckv3ufZF01CMIy769sSmoar09axzVvz+REcioj72zG37rVJjoyPNPaEBEREbkg5RrDXT/D5f/2Vtwc3Aymv+btB+yj8kVi+Lx/C3o3q8A7v2zgpvfnsDvxhK81iUcBT7Ku+R94k4xrdoOr34GwzAteG/ce5bohs3ht0lqurF+a7x9qS8tqxTLt+iIiIiKZJjwCWtwHA+ZCtUu8dQmGtIEts30tKzoynH9fXY9Xrm/A4q0H6fbGdOZu3O9rTaKAJ1nV4pHenjDVL4frPoDwyEy5rHOOEXM20/X1aWzce5RBvRoxsGcjCubNnOuLiIiIBE3BctBzBPT8FJKOwLDLYdz9cMzfUHVt43KMva8V+fJE0Ovd2bw7dQPZba/tnEQBT7Ke5WO8jcyrtIMbPvKWDs4EuxNPcMeH8/m/r5YTV6kwPz7UlisbZI2NREVEREQyrGZXuHc2tLwfFo2AN5vAklG+LsJSs1QBvh7QiktrleD58au455OFHD7h7zDS3EoBT7KWVd/AmLugfHPoORIiozPlsj+u2EnngdOYEb+XZ6+szYd9m1KqYOZcW0RERCTk8uSHTv+Cu3+BIpXhq7vho6tg7zrfSioQHcmQmxrzf11rMXHVLrq/OYPVO/3f4iG3UcCTrGPtBPi8L5S9GPp8BlH5LviSiSeSefzzJdz98QLKFIrmuwdac1uryoRp+wMRERHJCUrVg9snQLfXYMcSeLslTPk3JPuz4ImZcVfbKoy8sxlHTqbQY/AMvlyY4EstuZUCnmQNG36G0TdBydrQ5wvIE3vBl5y7cT9dXp/GmIUJ3N+xGl/e04pqJS78uiIiIiJZSliYt1fwgPlQu7u3f/DbLWH9FN9KalalKN890Jr65QrxyGdL+OtXyziRnOpbPbmJAp74b/NMGNkTilaDm8dC3kIXdLmTKam88P0qbhw6i/Aw4/P+LXm000VEReivu4iIiORg+UvAte/BzV8BDj7u4U19ObLbl3JKxEYz8s5m3N2uCiPnbOH6IbPYuv+YL7XkJvqNV/y1dR6MuB4KlYdbxkJMkQu63Oqdh+n+5gze+WUDPZtUYPwDbWhcsXDm1CoiIiKSHVTtCPfMgnZPwMqx8Gact/1UWug3I48ID+OpLrV45+bGbNp7lG6DpjNltT+BM7dQwBP/bF8Mn1wL+YrDLV97nzqdp7Q0x7tTN3DVoBnsPXKS92+N44Vr6pEvT0Tm1SsiIiKSXURGQ4e/Qv8ZUKo+fPuQt63CzuW+lHN5nVJ8c39rSheMpu/wefx7/CqOnEzxpZacTgFP/LFrBXx8NUQXgFvHQYHz364g4cAxer83m+fHr6L9RcX58aG2XFKrZCYWKyIiIpJNFa8Bt34DPYbA/vXwTluY8DQkHQ15KZWK5eOre1vRs0l5hk7dQPuXpvDp3C2kpmnPvMykgCeht2ctfNQdIvJ44a5QhfO6jHOOMQsS6DJwGsu3Heal6+rzzs2NKZo/TyYXLCIiIpKNmUHDXt4iLA17w8w3YHBzWPNDyEvJGxXOi9fWZ+x9rahUNB9PfbmMK96YxrR1e0JeS06lgCehtX+Dt0cLwC3joEiV87vM0STuHbGQRz9fQq3SBfj+wTZcH1ceM21/ICIiInJaMUWg+5vQ9weIioFPb4TPboHDO0JeSsPyhfi8fwve6nMxR5NSuPn9ufT9YC7rdiWGvJacRgFPQufgFvjwKkg56YW74jXO6zJT1uzm8oFTmbRqF092qcmn/ZpTvkhMJhcrIiIikkNVbAF3T4OOT3t38QY3hbnvQlpotzEwM7rWK82kR9rx1641mb/pAJ1fn8bTY5ez78jJkNaSkyjgSfClJsOC4fB+Jzhx2Fu6t2Ttc77MsaQU/jZ2GX0/mEeRmCi+vq81/dtVJVyblouIiIicm4goaPsY3DsLyjSC8Y95v6v5sAhLnohw+rWtys+Pt6dPswqMnLuF9i/9zDu/rOdkivbOO1cKeBI8qSmwaAQMagzfPOgtpHLrOCjT8JwvtWjLAa54Yzoj5mzhrjaV+XpAK2qXKZD5NYuIiIjkJkWrequZXz0UDmz0FmGZ+AwkhX6/uqL58/CP7nX58aE2NKlchBe+X82lr/7Cd0t34JwWYskorSEvmS8tFZZ/Cb+8CPvivaV5e42GGpd7k3zPQXJqGm/+FM+bU+IpVSCakXc2p0XVokEqXERERCQXMoMGN0L1y2Di0zDjdVjxFVzxGlS/NOTlVCsRy7DbmjBt3R6e/24V941cSOOKhXm6W20ali8U8nqyG93Bk8yTlgYrxsLbLeHLOyE8Cm78BO6eChd1Pudwt37PEa57eyavT15H94Zl+P6hNgp3IiIiIsESUwS6D4bbvoOIaBhxLXzeFxJ3+VJOm+rF+e6BNrx4TT027ztGj8EzeHDUIrYdPO5LPdmF7uDJhXMO1oyHKS/ArmVQrAZcNwxqXw1h5/4ZgnOOj2dv5t/jVxEdGc5bfS6ma73SQShcRCR4zGwY0A3Y7Zyre5rjfYAnAg+PAPc455YEjm0CEoFUIMU5FxeSokVEACq1hv7TYfpAmPYyrJ8Mlz4HF996Xr/bXYjwMKNn0wp0a1CGIT+v591pG/hh+U7ubFOZe9pXI38exZlTWXYbzxoXF+fmz5/vdxkCXrCLnwRTnofti6BwZWj/JNS7HsLCz+uSuw6f4PEvljJ17R7a1SjOS9fVp0SB6EwuXESyCzNbkF3DjZm1xQtuH50h4LUEVjnnDphZF+BZ51yzwLFNQJxzbu+5tKk+UkQy3d518O3DsGkalG8G3Qae12J5mWXbweO89MNqxi7eTrH8eXi0Uw1uiCuf6xbdO1v/GNQIbmadzWyNmcWb2ZOnOd7HzJYGvmaaWYNg1iOZxDnY8DMMuxxGXAdH98FVg2DAPGjQ87zD3XdLd3D5wKnM3biPf/aoy/C+TRTuRCTbcs5NBfaf5fhM59yBwMPZQLmQFCYici6KVYdbv4Eeb3th7502MPkfkOzPMMmyhfIysGejwEbpMdoo/TSCFvDMLBwYDHQBagO9zOzUuL8RaOecqw/8ExgarHokk2yeCcO7wUfd4eBWuOJVuH8BXHwLhEee1yUPHU/m4dGLuW/kQioWiWH8A224uXlFbVouIrnJHcD36R47YIKZLTCzfj7VJCLiMYOGvWHAfKh3A0x7Bd5qAet/8q2kM22UHr9bG6UHc9BqUyDeObcBwMxGAd2Blb+e4Jybme58fXqZlSXMh5/+BRumQL4S0Pk/0Pg2iLywO2wz4/fy2OdL2JV4kocurc59HaoRGa61f0Qk9zCzDngBr3W6p1s557abWQlgopmtDtwRPN3r+wH9ACpUqBD0ekUkF8tXFK5+2xux9e3D8PHVXuC7/N+Qv3jIy/l1o/RLapXgw5mbGDQ5nssHTqNPswo8eEl1iubPE/KasoJg/iZdFtia7nFC4LkzOfXTS8kKti+GETfAe5fAzqXQ6V/w4BJo3v+Cwt2J5FT++e1Ker83h+jIcL68pyUPXVpD4U5EchUzqw+8B3R3zu379Xnn3PbAn7uBr/A+ND0t59xQ51yccy6uePHQ/4IlIrlQlXZwz0xo+xdvO4U342DhR96K6j44daP0EXO20P7lnxk6NXdulB7M36ZPN77utCu6pPv08okzHO9nZvPNbP6ePRpfGxI7l8OoPjC0HWydA5c8Aw8uhZb3Q1TMBV16+bZDXPXmdN6fvpFbWlTkuwfa0EB7mohILmNmFYAvgZudc2vTPZ/PzGJ//R7oBCz3p0oRkTOIjIaO/wf3zIAStWHc/TD8CtizxreSfrdReqUi/Hu8t1H6+GW5a6P0YA7RTADKp3tcDth+6knpPr3skv7Ty/Scc0MJzM+Li4vLPf93/LBnDfz8gvdpTJ4C0P4paH4PRBe84Eunpjnembqe1yaupXBMFMP7NqH9RSUyoWgRkazHzD4F2gPFzCwB+DsQCeCcGwI8AxQF3grMOf51O4SSwFeB5yKAkc65H0L+BkREMqL4Rd6+eYs/gQlPw9utoPXD0ObRC57Kc75O3Sj93hELiatYmL/lko3Sg7ZNgplFAGuBS4BtwDygt3NuRbpzKgA/AbecMh/vjLQEdJDsWw+//AeWfQ4Reb0hmC0GeBteXqDjSal8t2wHH87cxLJth+harxTP96hH4XxRmVC4iORk2XmbBD+ojxQRXx3ZAz/+FZZ9BkWqQrfXvOGcPkpNc3w+fysvT1jL3iMn6dGwDI93rknZQnl9retCna1/DNodPOdcipkNAH4EwoFhzrkVZtY/cPxsn15KqBzYDFP/C4s/hfAoL9S1ehDyFbvgS6/YfohRc7cydvE2Ek+kULlYPgbe2JDuDctohUwRERGRnCZ/cbj2XWjYC759BD66Chr09tZwyFfUl5JOt1H698t30qtpBfq1rUKZbB70TkcbnedWh7bBtJe9CbEWDnG3e7fTY0te0GUTTyQzbsl2Rs/bytKEQ0RFhNG1bil6Nq1As8pFFOxE5JzoDt65UR8pIllG8nGY+hLMeN2b9tPpX95WCz7/Lrjt4HFen7SWLxduwwyua1yee9pVpULRC1tjItTO1j8q4OU2iTth2quw4ANvw/KLb/HGSBc82wKnZ+ecY9HWg4yau4Vvl+7gWFIqNUvF0rNJeXo0KkuhGA3FFJHzo4B3btRHikiWs2slfPuQt2hfpTbesM1i1f2uioQDx3jnlw2MnreVVOfo0bAs93aoStXi+f0uLUMU8HKz1GTYsQQ2TYfNM2DjVO+5hr2h3V+g0PnvmXTwWBJfLdrGqLlbWbMrkZiocK6sX4aeTcvTsHwh3a0TkQumgHdu1EeKSJaUlgYLh8PEZyHlOLR5DFo/BBH+71O36/AJhk7dwIg5mzmZksYV9UozoGM1apYq4HdpZ6WAl5ukJMH2hf8LdFvmQPJR71jR6lClvbcqZtGq53V55xyzN+xn9LwtjF++k6SUNBqUK8iNTSpwVcMy5M8TzIVZRSS3UcA7N+ojRSRLS9wFPz4Fy8d4NxlaPgAN+1zwFlyZYe+Rk7w/fSMfzdzE0aRUOtUuyf0dq1Ov3IWvJB8MCng5WfIJ2LbAC3ObpsPWud4nIwDFa0GlVlAx8HUB8+v2JJ5kzMIERs/bysa9R4mNjuDqRmXp2aQCtctk7U84RCT7UsA7N+ojRSRbiJ/sbcuVMA9iinmrtze5E/IW9rsyDh5LYvjMTQybvpHDJ1Jof1Fx7u9YjcYVL3xl+cykgJeTJB/3QtzmGbBphvcPI/Wkd6xkXS/I/RrqLnAlzLQ0x7T4vYyau4WJK3eRkuZoWqkINzYpT9d6pckbFZ4Jb0hE5MwU8M5Nru8jRST7cA42z4Tpr0H8RIjKD41vgxb3QYEyfldH4olkPp69mfembWT/0SRaVCnK/R2r0aJq0SwxDUkBLztLOupNSt00wwt12xZAahJYGJSqBxVbe4GuQotM2bMOYMeh43w+37tbt+3gcYrki+Lai8tyY5MKVCuRPSaeikjOoIB3bnJdHykiOcPOZd5qm8vHeKu7N+jpbduVBRZjOZaUwsg5Wxg6dQO7E0/SuGJhBnSsRvsaxX0Negp42cmJw4FAF5hDt30RpKV4f9lLN/DCXKU2UKE5RGfemOCU1DR+Wr2bUfO28vOa3aQ5aF2tGD2bluey2iXJE6G7dSISegp45ybH95EikrMd2AQz34RFH0PKSajVDVo9DOUa+10ZJ5JT+Xz+Vob8soFtB49Tr2xBBnSsxmW1ShIWFvqgp4CXlR0/CFtmw+bp3l26HYvBpUFYBJS5ODDcsjVUaAZ5YjO9+S37jjF6/hY+n5/A7sSTlIjNw/Vx5bgxrkK22w9ERHIeBbxzk+P6SBHJnY7sgTlDYN67cOKQd3Oj9cNQtaPv++glpaTx1aIE3vp5PZv3HeOikrHc17EaV9QrTXgIg54CXlaRmgxHdsH2xf9bFGXnMsBBeBSUbRyYQ9cayjeFqHxBKeNkSioTVuxi1LwtzIjfR5hBh4tK0LNpBTpcVJyI8LCgtCsicq4U8M5Ntu4jRUROdeIwLBgOs9+CxB1Qqr63vULtHhDm7+iylNQ0vl26gzenxBO/+whViuXj3g7V6N6wDJEh+F1aAS/YTiZ6y74e2QVHdqb7fpe3sfiR3d7zx/b97zXhebwQ9+uiKOWaQGTeoJSXlubYdvA4q3cmMnvDPr5atI39R5MoWygvNzYpz/Vx5ShdMDhti4hcCAW8c5Ml+0gRkQuVchKWjvbm6e2Lh8KVodUD0KA3REb7WlpamuOHFTsZ9FM8q3YcpnyRvNzTrhrXNi4b1ClOCnjnIy0Nju0NhLRAcPvd97v/F95+3WcuvbBIyF/S25ogfynIXwJiS3nPFavh3a3L5L+Qzjn2HDnJ2p1HWLMrkTU7D7Nm1xHW7UrkWFIqABFhRqc6JenZpAKtqxXzZcywiEhGKeCdGwU8EcnR0lJh9XfeypvbF0K+Et7+zk3uyNS1Kc6Hc47Jq3Yz6Kd1LEk4ROmC0dzdtgo9m1YgOjLzg54CXnqpyXB4+yl32Hb9Mbwd2Q0u9Y+vz1MgENwCoS1/qUCIK5nu+ZLePh5BHCN8+EQy63YlsnpnImt3JgYCXSIHjiX/dk6x/FHUKBlLjZKx1CwVS41S3vfajFxEsgsFvHOjgCciuYJzsHEqzBgI63/yfj+Pu90Le7GlfC7NMW3dXgb9tI55mw5QLH8e7mpTmZuaVyRfJv4OroCXXvxk+OSa3z9nYZCveLqQ9utdt1PuwOUvCVGhXXjkRHIq8buPsHbX/0Lc2p2JbD904rdz8kWFU6NUIMSVjOWikl6YK5Y/T0hrFRHJbAp450YBT0Ryne2LvaC38mtvkcKGvaHlA1C0qt+VMWfDPgb9FM/0+L0UionkjlaVuaVlJQrmjbzgayvgpXdkN6z98fd34PIVyxITNTfvP8banYG7coFAt2nvUdIC/4uiwsOoWiI/F5XM/7tAV7ZQ3iyx4aKISGZTwDs3CngikmvtWw8zB8HiEd4WY7Wu8hZkKdPI78pYuOUAg3+KZ/Lq3cTmieDeDtW4p/2FBdCz9Y+5b6xe/hJw8c2+Ne+cY8ehE6wJDKv8dXjlut1HSEpJA7yRnZWK5uOikrF0q1+Gi0rGclGp/FQqmk8rXIqIiIiInKpoVbhyILR/Ema/DfOHwcqxUKWDF/Qqt/Nti4WLKxTm/duasHzbIQZPied4UkpQ28t9d/B8kpyaxleLtjF4Sjyb9x377fnSBaO9YZWlYgNBLpZqJfIHZTKmiEh2ozt45ya79pEiIpnuxCEv5M16C47u9u7ktX4YanbzfeSec+6CR9/pDp6Pfg12b/4Uz5b9x6hXtiD/7F6HmqULUKNELAVjLnwMroiIiIiIpBNd0At0ze6BJSNhxhvw2S1QtJo3R6/+jb5tsRDsqVUKeEGSnJrGVwu3MWjKOrbuP069sgV5/9Y4OtYsoflyIiIiIiKhEBntrbB58a3eQizTX4NvHoCf/glN7vK2WMhXzO8qM5UCXiY7XbB79tY6CnYiIiIiIn4JC4e610Cdq2HDzzBrMPz8b5j+qnc3r8V9UPwiv6vMFAp4meTUYFe/XEGeu6oOHS5SsBMRERERyRLMoGoH72v3apj9FiwZBQs/hOqdvKDn44IsmUEB7wIlp6bx5cIE3pwSr2AnIiIiIpJdlKgJV70BHZ/2FmSZ9y581B1K1vWCXt1rISL77SutgHeefg12g36KJ+GAgp2IiIiISLaUvzi0fwJaPQjLPvOGb469ByY9B03v8ubwxRTxu8oM06Zq5yg5NY3R87bQ4eWfeWLMMormi+KD25rw9X2t6FizpMKdiIgAYGbDzGy3mS0/w3EzszfMLN7MlprZxemOdTazNYFjT4auahGRXCwyGi6+Be6dDTeNgZK1vcVYXq0N3z4Ce+P9rjBDdAcvg5JT0xizwBuKmXDgOA3KFeSf3evS/qLiCnUiInI6w4E3gY/OcLwLUD3w1Qx4G2hmZuHAYOAyIAGYZ2bjnHMrg16xiIh48++qXep97VrhzdNb9LE3jLNGZ2g5ACq2yrLz9BTw/oSCnYiInA/n3FQzq3SWU7oDHznnHDDbzAqZWWmgEhDvnNsAYGajAucq4ImIhFrJOtB9MHR8Bua9B/Pfh+FXQOkG0GKAtypneNba11oB7wySUv63eIqCnYiIBEFZYGu6xwmB5073fLMQ1iUiIqeKLQkd/w/aPOKtujn7LfjyLpj4d2jWDxrfBnkL+10loID3B0kpaYxZmMCbP8Wz7eBxGpQvxD971KV9DQU7ERHJVKfrVNxZnj/9Rcz6Af0AKlSokDmViYjI6UXmhbi+3sbp8ZNg1psw6Vn45SVodBM07w9FqvhaogJewOmC3b+uVrATEZGgSQDKp3tcDtgORJ3h+dNyzg0FhgLExcWdMQiKiEgmCguDGp28r53LvJU35w+DuUOh5hXe8M0KzX2Zp5frA56CnYiI+GQcMCAwx64ZcMg5t8PM9gDVzawysA3oCfT2sU4RETmbUvXg6iFwyd+9vfTmvQ+rv4UyF3v76dXuAeGhi125NuAlpaTxxYIEBk/xgl3D8oV4/uq6tFOwExGRTGBmnwLtgWJmlgD8HYgEcM4NAcYDXYF44BjQN3AsxcwGAD8C4cAw59yKkL8BERE5NwVKwyXPQJtHYcmnMOstGHOHN4Sz2d3eFgzRBYNehnmLd2UfcXFxbv78+ef9+tMFu4cura5gJyKSBZnZAudcnN91ZBcX2keKiEgmSkuDtT94wzc3T4eo/F7Ia3Y3FK50QZc+W/+Y6+7g/bJ2D3/9apnu2ImIiIiISPCEhUHNrt7X9kXeHb25Q2H/Bug9OmjN5rqAd0nNEoy8qxktqhRVsBMRERERkeAr0wiufRcufRaSjwW1qVwX8MLCjJZVi/ldhoiIiIiI5DYFywa9ibBgXtzMOpvZGjOLN7MnT3O8ppnNMrOTZvZYMGsRERERERHJ6YJ2B8/MwoHBwGV4e/3MM7NxzrmV6U7bDzwA9AhWHSIiIiIiIrlFMO/gNQXinXMbnHNJwCige/oTnHO7nXPzgOQg1iEiIiIiIpIrBDPglQW2pnucEHjunJlZPzObb2bz9+zZkynFiYiIiIiI5DTBDHinW6LyvDbdc84Ndc7FOefiihcvfoFliYiIiIiI5EzBDHgJQPl0j8sB24PYnoiIiIiISK4WzIA3D6huZpXNLAroCYwLYnsiIiIiIiK5WtBW0XTOpZjZAOBHIBwY5pxbYWb9A8eHmFkpYD5QAEgzs4eA2s65w8GqS0REREREJKcK6kbnzrnxwPhTnhuS7vudeEM3RURERERE5AIFdaNzERERERERCR1z7rwWtvSNme0BNl/gZYoBezOhnOxI7z130nvPnXLCe6/onNPyyRmkPvKC6b3nTnrvuU9OeN9n7B+zXcDLDGY23zkX53cdftB713vPbfTec+d7l/OXm//e6L3rvec2ufW95/T3rSGaIiIiIiIiOYQCnoiIiIiISA6RWwPeUL8L8JHee+6k95475eb3LucvN/+90XvPnfTec58c/b5z5Rw8ERERERGRnCi33sETERERERHJcXJdwDOzzma2xszizexJv+sJFTMrb2ZTzGyVma0wswf9rinUzCzczBaZ2bd+1xJKZlbIzL4ws9WB//8t/K4pFMzs4cDf9eVm9qmZRftdUzCZ2TAz221my9M9V8TMJprZusCfhf2sUbI29Y/qH9U/5o7+EXJXH5kb+8dcFfDMLBwYDHQBagO9zKy2v1WFTArwqHOuFtAcuC8XvfdfPQis8rsIH7wO/OCcqwk0IBf8NzCzssADQJxzri4QDvT0t6qgGw50PuW5J4HJzrnqwOTAY5E/UP+o/pFc0DecRq7rHyFX9pHDyWX9Y64KeEBTIN45t8E5lwSMArr7XFNIOOd2OOcWBr5PxPshVtbfqkLHzMoBVwDv+V1LKJlZAaAt8D6Acy7JOXfQ16JCJwLIa2YRQAyw3ed6gso5NxXYf8rT3YEPA99/CPQIZU2Srah/RP2j37WEUi7vHyEX9ZG5sX/MbQGvLLA13eMEctEP8V+ZWSWgETDH51JCaSDwFyDN5zpCrQqwB/ggMPzmPTPL53dRweac2wa8DGwBdgCHnHMT/K3KFyWdczvA+yUWKOFzPZJ1qX9E/aPPdYRaruwfQX1kQI7uH3NbwLPTPJerlhE1s/zAGOAh59xhv+sJBTPrBux2zi3wuxYfRAAXA2875xoBR8lhwxBOJzCWvjtQGSgD5DOzm/ytSiRLU/+o/jG3yZX9I6iPzA1yW8BLAMqne1yOHHxL+lRmFonXeY1wzn3pdz0h1Aq4ysw24Q076mhmn/hbUsgkAAnOuV8/jf4Cr0PL6S4FNjrn9jjnkoEvgZY+1+SHXWZWGiDw526f65GsS/2j+kf1j7mjfwT1kZDD+8fcFvDmAdXNrLKZReFNKB3nc00hYWaGN858lXPuVb/rCSXn3FPOuXLOuUp4/89/cs7lik+qnHM7ga1mdlHgqUuAlT6WFCpbgOZmFhP4u38JuWTy/CnGAbcGvr8V+NrHWiRrU/+o/lH9Y+7oH0F9JOTw/jHC7wJCyTmXYmYDgB/xVgwa5pxb4XNZodIKuBlYZmaLA8/91Tk33r+SJETuB0YEfmnbAPT1uZ6gc87NMbMvgIV4K+QtAob6W1VwmdmnQHugmJklAH8HXgQ+M7M78Dr06/2rULIy9Y/qH3OpXNc/Qu7rI3Nj/2jO5aoh9iIiIiIiIjlWbhuiKSIiIiIikmMp4ImIiIiIiOQQCngiIiIiIiI5hAKeiIiIiIhIDqGAJyIiIiLy/+3bz4tOcRTH8fcHJYz8KDYWhA3Kz51JKf+AxUhhkrWNnRQpe0vFcsRCZDZWYjFloRH5UbKymlI2GlEkjsV8F0MheeaZcef9Wj3P6XtP37O4nc793it1hAOe1BFJ9ie5M9v7kCRpLrE/ar5xwJMkSZKkjnDAk/osybEk40meJrmSZGGSD0kuJnmS5H6SNW3tziQPkzxPMppkVYtvTnIvybN2zaaWfiDJrSSvklxPklkrVJKkv2B/lHrDAU/qoyRbgMPAYFXtBL4CR4FlwJOq2g2MAefbJVeB01W1HXgxLX4duFRVO4C9wJsW3wWcArYCG4HBGS5JkqR/Zn+UemfRbG9AmmcOAHuAR+3h4RLgLfANuNHWXANuJ1kBrKyqsRYfAW4mWQ6sq6pRgKr6BNDyjVfVRPv/FNgAPJjxqiRJ+jf2R6lHHPCk/gowUlVnfggm535aV3/I8Sufp/3+ive4JOn/YH+UesRXNKX+ug8MJVkLkGR1kvVM3YtDbc0R4EFVTQLvkuxr8WFgrKreAxNJDrYci5Ms7WcRkiT1mP1R6hGfXkh9VFUvk5wF7iZZAHwBTgIfgW1JHgOTTH2HAHAcuNwa1GvgRIsPA1eSXGg5DvWxDEmSesr+KPVOqn530i2pH5J8qKqB2d6HJElzif1R+nu+oilJkiRJHeEJniRJkiR1hCd4kiRJktQRDniSJEmS1BEOeJIkSZLUEQ54kiRJktQRDniSJEmS1BEOeJIkSZLUEd8Ba+GOg2CQQuwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# combine all the history from training together\n",
    "combined = dict()\n",
    "for key in ['accuracy','val_accuracy','loss','val_loss']:\n",
    "    combined[key] = np.hstack([x.history[key] for x in history])\n",
    "    \n",
    "# summarize history for accuracy\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.plot(combined['accuracy'])\n",
    "plt.plot(combined['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "\n",
    "# summarize history for loss\n",
    "plt.subplot(122)\n",
    "plt.plot(combined['loss'])\n",
    "plt.plot(combined['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are these converged? No! Not even close!!! We need to run these for many more epochs. \n",
    "\n",
    "## [Back to Slides]\n",
    "\n",
    "# From Recurrent back to Convolutional\n",
    "Since all the sequences are now the same length, it is possible to construct a convolutional network on top of the embedded sequence outputs. Once embedded, there really is not too much difference in representation about the data and an image (though the data clearly has a different intuition). We will follow these steps:\n",
    "- use an embedding layer to translate to a dense representation of the entire sequence\n",
    "- train 1D filters to convolve with the output of the embedded sequence\n",
    " - note: 1D filters doesn't mean they are only one dimension, it means the filters are looped through in a single dimension. The size of the filter is [`kernel_size x embed_size`]. It loops through the entire embedding, over a number of words at a time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 1000)]            0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 1000, 100)         13414300  \n",
      "_________________________________________________________________\n",
      "conv1d (Conv1D)              (None, 996, 128)          64128     \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 996, 128)          0         \n",
      "_________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D) (None, 199, 128)          0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 199, 128)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 195, 128)          82048     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 39, 128)           0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 39, 128)           0         \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 35, 128)           82048     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_2 (MaxPooling1 (None, 1, 128)            0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 20)                2580      \n",
      "=================================================================\n",
      "Total params: 13,661,616\n",
      "Trainable params: 247,316\n",
      "Non-trainable params: 13,414,300\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/2\n",
      "9051/9051 [==============================] - 81s 9ms/sample - loss: 3.0401 - acc: 0.0863 - val_loss: 2.6299 - val_acc: 0.1670\n",
      "Epoch 2/2\n",
      "9051/9051 [==============================] - 76s 8ms/sample - loss: 2.4299 - acc: 0.1999 - val_loss: 2.1595 - val_acc: 0.2766\n"
     ]
    }
   ],
   "source": [
    "# but wait, if we are just making the sequences of this all the same size\n",
    "#  can't we just use a 1-D convolutional network? \n",
    "\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Conv1D, MaxPooling1D, Flatten, Dense\n",
    "from tensorflow.keras.layers import Dense, Input, Dropout\n",
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "EMBED_SIZE = 100 \n",
    "sequence_input = Input(shape=(MAX_ART_LEN,), dtype='int32')\n",
    "# starting size: 1000\n",
    "embedded_sequences = embedding_layer(sequence_input) # from previous embedding\n",
    "x = Conv1D(128, 5, activation='relu',\n",
    "           kernel_initializer='he_uniform')(embedded_sequences)\n",
    "\n",
    "# after conv, size becomes: 1000-4=996\n",
    "x = Dropout(0.05)(x)\n",
    "x = MaxPooling1D(5)(x)# after max pool, 996/5 = 199\n",
    "x = Dropout(0.15)(x)\n",
    "x = Conv1D(128, 5, activation='relu',\n",
    "           kernel_initializer='he_uniform')(x)\n",
    "\n",
    "# new size is 195\n",
    "x = MaxPooling1D(5)(x) # after max pool, size is 195/5 = 39\n",
    "x = Dropout(0.2)(x)\n",
    "x = Conv1D(128, 5, activation='relu',\n",
    "           kernel_initializer='he_uniform')(x)\n",
    "\n",
    "# after convolution, size becomes 35 elements long\n",
    "x = MaxPooling1D(35)(x) # this is the size to globally flatten \n",
    "# flattened vector max pools across each of the 35 elements\n",
    "# so vectors is now 128 dimensions (same as num output filters)\n",
    "x = Flatten()(x)\n",
    "x = Dropout(0.25)(x)\n",
    "x = Dense(128, activation='relu',\n",
    "          kernel_initializer='he_uniform')(x)\n",
    "\n",
    "preds = Dense(NUM_CLASSES, activation='softmax',\n",
    "              kernel_initializer='glorot_uniform')(x)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "print(model.summary())\n",
    "histories = []\n",
    "tmp = model.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe),\n",
    "          epochs=2, batch_size=128)\n",
    "histories.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 1.9841 - acc: 0.3192 - val_loss: 1.6240 - val_acc: 0.4719\n",
      "Epoch 2/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 1.6381 - acc: 0.4341 - val_loss: 1.4983 - val_acc: 0.4613\n",
      "Epoch 3/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 1.3915 - acc: 0.5127 - val_loss: 1.2825 - val_acc: 0.5449\n",
      "Epoch 4/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 1.2228 - acc: 0.5736 - val_loss: 1.1901 - val_acc: 0.5833\n",
      "Epoch 5/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 1.0899 - acc: 0.6211 - val_loss: 0.9687 - val_acc: 0.6712\n",
      "Epoch 6/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.9510 - acc: 0.6695 - val_loss: 1.0514 - val_acc: 0.6270\n",
      "Epoch 7/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.8733 - acc: 0.6967 - val_loss: 0.9065 - val_acc: 0.6977\n",
      "Epoch 8/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.7789 - acc: 0.7334 - val_loss: 0.9630 - val_acc: 0.6677\n",
      "Epoch 9/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.6970 - acc: 0.7618 - val_loss: 0.8464 - val_acc: 0.7114\n",
      "Epoch 10/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.6220 - acc: 0.7850 - val_loss: 0.7328 - val_acc: 0.7631\n"
     ]
    }
   ],
   "source": [
    "tmp = model.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe),\n",
    "          epochs=10, batch_size=128)\n",
    "histories.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.5751 - acc: 0.8027 - val_loss: 0.8596 - val_acc: 0.7278\n",
      "Epoch 2/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.5151 - acc: 0.8230 - val_loss: 0.7820 - val_acc: 0.7521\n",
      "Epoch 3/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.4766 - acc: 0.8398 - val_loss: 0.6707 - val_acc: 0.7905\n",
      "Epoch 4/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.4235 - acc: 0.8565 - val_loss: 0.7614 - val_acc: 0.7698\n",
      "Epoch 5/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.3922 - acc: 0.8640 - val_loss: 0.6525 - val_acc: 0.7928\n",
      "Epoch 6/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.3573 - acc: 0.8778 - val_loss: 0.7147 - val_acc: 0.7795\n",
      "Epoch 7/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.3359 - acc: 0.8852 - val_loss: 0.6493 - val_acc: 0.8020\n",
      "Epoch 8/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.2847 - acc: 0.9009 - val_loss: 0.6356 - val_acc: 0.8202\n",
      "Epoch 9/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.2848 - acc: 0.9055 - val_loss: 0.6622 - val_acc: 0.8153\n",
      "Epoch 10/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.2487 - acc: 0.9205 - val_loss: 0.7026 - val_acc: 0.8011\n"
     ]
    }
   ],
   "source": [
    "tmp = model.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe),\n",
    "          epochs=10, batch_size=128)\n",
    "histories.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/10\n",
      "9051/9051 [==============================] - 76s 8ms/sample - loss: 0.1314 - acc: 0.9560 - val_loss: 0.8257 - val_acc: 0.8016\n",
      "Epoch 2/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.1182 - acc: 0.9624 - val_loss: 0.7264 - val_acc: 0.8290\n",
      "Epoch 3/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1247 - acc: 0.9606 - val_loss: 0.8586 - val_acc: 0.8042\n",
      "Epoch 4/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1150 - acc: 0.9616 - val_loss: 0.9350 - val_acc: 0.8095\n",
      "Epoch 5/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1186 - acc: 0.9634 - val_loss: 0.7631 - val_acc: 0.8294\n",
      "Epoch 6/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1017 - acc: 0.9674 - val_loss: 0.6935 - val_acc: 0.8374\n",
      "Epoch 7/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1087 - acc: 0.9674 - val_loss: 0.7646 - val_acc: 0.8316\n",
      "Epoch 8/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1050 - acc: 0.9659 - val_loss: 0.7737 - val_acc: 0.8392\n",
      "Epoch 9/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0953 - acc: 0.9686 - val_loss: 0.6940 - val_acc: 0.8506\n",
      "Epoch 10/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1073 - acc: 0.9696 - val_loss: 0.7011 - val_acc: 0.8405\n"
     ]
    }
   ],
   "source": [
    "tmp = model.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe),\n",
    "          epochs=10, batch_size=128)\n",
    "histories.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 9051 samples, validate on 2263 samples\n",
      "Epoch 1/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0866 - acc: 0.9745 - val_loss: 0.7282 - val_acc: 0.8365\n",
      "Epoch 2/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.1049 - acc: 0.9688 - val_loss: 0.7922 - val_acc: 0.8352\n",
      "Epoch 3/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0889 - acc: 0.9712 - val_loss: 0.8328 - val_acc: 0.8347\n",
      "Epoch 4/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0919 - acc: 0.9697 - val_loss: 0.7734 - val_acc: 0.8347\n",
      "Epoch 5/10\n",
      "9051/9051 [==============================] - 75s 8ms/sample - loss: 0.0801 - acc: 0.9744 - val_loss: 0.8722 - val_acc: 0.8175\n",
      "Epoch 6/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0885 - acc: 0.9740 - val_loss: 0.7272 - val_acc: 0.8484\n",
      "Epoch 7/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0906 - acc: 0.9748 - val_loss: 0.7958 - val_acc: 0.8392\n",
      "Epoch 8/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0806 - acc: 0.9753 - val_loss: 0.7773 - val_acc: 0.8378\n",
      "Epoch 9/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0858 - acc: 0.9741 - val_loss: 0.8967 - val_acc: 0.8219\n",
      "Epoch 10/10\n",
      "9051/9051 [==============================] - 74s 8ms/sample - loss: 0.0796 - acc: 0.9748 - val_loss: 0.8384 - val_acc: 0.8378\n"
     ]
    }
   ],
   "source": [
    "tmp = model.fit(X_train, y_train_ohe, validation_data=(X_test, y_test_ohe),\n",
    "          epochs=10, batch_size=128)\n",
    "histories.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAAFNCAYAAABSRs15AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAB9aElEQVR4nO3dd5xU5fXH8c+Z2d5hC23pRXpRwILGjmLvvUZjTDRqEpNoEo3p5pfEJEZjr7HHXlCx9wIoHaQJ7NLb9r7z/P54BtiFXVhgZ2fL9/16zWt27ty5c+a6ePfM8zznmHMOERERERERafsC0Q5AREREREREmocSPBERERERkXZCCZ6IiIiIiEg7oQRPRERERESknVCCJyIiIiIi0k4owRMREREREWknlOCJNBMze9jM/tDEfZeZ2VGRjklERKSta67r6+4cR6QtU4InIiIiIiLSTijBE5F6zCwm2jGIiIiIyJ5RgicdSnjqxs/MbJaZlZrZA2bWxcxeN7NiM3vbzDrV2f8kM5trZgVm9r6ZDanz3Bgz+yr8uqeBhO3e6wQzmxF+7admNrKJMR5vZl+bWZGZ5ZnZLds9f3D4eAXh5y8Jb080s7+b2XIzKzSzj8PbDjOz/AbOw1Hhn28xs2fN7DEzKwIuMbPxZvZZ+D1Wm9kdZhZX5/XDzOwtM9tkZmvN7Jdm1tXMyswss85++5nZejOLbcpnFxGRtqktXF8biPl7ZrY4fC172cy6h7ebmf3DzNaFr6ezzGx4+LnjzGxeOLaVZnb9Hp0wkQhSgicd0enA0cAg4ETgdeCXQBb+38Q1AGY2CHgSuA7IBiYDr5hZXDjZeRH4L9AZ+F/4uIRfuy/wIPB9IBO4B3jZzOKbEF8pcBGQARwP/MDMTgkft1c43n+HYxoNzAi/7m/AfsBB4Zh+DoSaeE5OBp4Nv+fjQC3wY/w5ORA4EvhhOIZU4G3gDaA7MAB4xzm3BngfOKvOcS8AnnLOVTcxDhERabta+/V1KzM7Avgz/prVDVgOPBV+eiLwnfDnyADOBjaGn3sA+L5zLhUYDry7O+8r0hKU4ElH9G/n3Frn3ErgI+AL59zXzrlK4AVgTHi/s4HXnHNvhROUvwGJ+ATqACAW+Kdzrto59ywwtc57fA+4xzn3hXOu1jn3CFAZft1OOefed87Nds6FnHOz8BfBQ8NPnw+87Zx7Mvy+G51zM8wsAHwXuNY5tzL8np+GP1NTfOacezH8nuXOuenOuc+dczXOuWX4C+iWGE4A1jjn/u6cq3DOFTvnvgg/9wg+qcPMgsC5+Iu0iIi0f636+rqd84EHnXNfheO7ETjQzPoA1UAqMBgw59x859zq8OuqgaFmluac2+yc+2o331ck4pTgSUe0ts7P5Q08Tgn/3B3/jR4AzrkQkAf0CD+30jnn6rx2eZ2fewM/DU8fKTCzAqBn+HU7ZWb7m9l74amNhcCV+G8/CR9jSQMvy8JPYWnouabI2y6GQWb2qpmtCU/b/FMTYgB4CX/h64f/FrfQOfflHsYkIiJtS6u+vm5n+xhK8KN0PZxz7wJ3AHcCa83sXjNLC+96OnAcsNzMPjCzA3fzfUUiTgmeSONW4S8kgJ+Tj7+IrARWAz3C27boVefnPOCPzrmMOrck59yTTXjfJ4CXgZ7OuXTgbmDL++QB/Rt4zQagopHnSoGkOp8jiJ8SU5fb7vFdwAJgoHMuDT/FZlcx4JyrAJ7BfzN6IRq9ExGRHUXr+rqzGJLxUz5XAjjnbnfO7QcMw0/V/Fl4+1Tn3MlADn4q6TO7+b4iEacET6RxzwDHm9mR4SIhP8VPA/kU+AyoAa4xsxgzOw0YX+e19wFXhkfjzMySzRdPSW3C+6YCm5xzFWY2HjivznOPA0eZ2Vnh9800s9Hhbz8fBG4zs+5mFjSzA8NrEhYCCeH3jwV+DexqrUIqUASUmNlg4Ad1nnsV6Gpm15lZvJmlmtn+dZ5/FLgEOAl4rAmfV0REOpZoXV/regK41MxGh6+Vf8JPKV1mZuPCx4/Ff0laAdSG1wieb2bp4amlRfg16yKtihI8kUY4577Bryf7N36E7ETgROdclXOuCjgNn8hsxq8neL7Oa6fh1wncEX5+cXjfpvgh8DszKwZups63g865FfipIT8FNuELrIwKP309MBu/VmET8Bcg4JwrDB/zfvw3k6VAvaqaDbgen1gW4y+mT9eJoRg//fJEYA2wCDi8zvOf4Iu7fBVevyciIrJVFK+vdWN4B7gJeA4/atgfOCf8dBr+2rcZP41zI36dIPjZKcvCyxeuDH8OkVbF6k9xFhHZe2b2LvCEc+7+aMciIiIi0pEowRORZmVm44C38GsIi6Mdj4iIiEhHoimaItJszOwRfI+865TciYiIiLQ8jeCJiIiIiIi0ExrBExERERERaSeU4ImIiIiIiLQTMdEOYHdlZWW5Pn36RDsMERFpAdOnT9/gnMuOdhxtha6RIiIdw86uj20uwevTpw/Tpk2LdhgiItICzGx5tGNoS3SNFBHpGHZ2fYzYFE0ze9DM1pnZnEaeNzO73cwWm9ksM9s3UrGIiIiIiIh0BJFcg/cwcOxOnp8EDAzfrgDuimAsIiIiIiIi7V7EEjzn3IfApp3scjLwqPM+BzLMrFuk4hEREREREWnvorkGrweQV+dxfnjb6u13NLMr8KN89OrVa4cDVVdXk5+fT0VFRWQibUUSEhLIzc0lNjY22qGIiEgb0FGukbo+ioh40UzwrIFtDXZdd87dC9wLMHbs2B32yc/PJzU1lT59+mDW0GHbB+ccGzduJD8/n759+0Y7HBERaQM6wjVS10cRkW2i2QcvH+hZ53EusGpPDlRRUUFmZma7vXBtYWZkZma2+29hRUSk+XSEa6SujyIi20QzwXsZuChcTfMAoNA5t8P0zKZqzxeuujrK5xQRkebTEa4dHeEziog0RSTbJDwJfAbsY2b5ZnaZmV1pZleGd5kMLAUWA/cBP4xULJFWUFDAf/7zn91+3XHHHUdBQUHzByQiItJK6BopItKyIllF81znXDfnXKxzLtc594Bz7m7n3N3h551z7irnXH/n3AjnXJvtzNrYxau2tnanr5s8eTIZGRkRikpERCT6dI0UEWlZ0Syy0m7ccMMNLFmyhNGjRxMbG0tKSgrdunVjxowZzJs3j1NOOYW8vDwqKiq49tprueKKKwDo06cP06ZNo6SkhEmTJnHwwQfz6aef0qNHD1566SUSExOj/MlEZG+VVtawsaSK3E6JBAKRnUJWXRtiVUE5AAEzYoJG0IxAwIgJv3dlTYjK6hCVNbX+55paKqpDVNWGqKl11IZCVNc6akOO6toQtSFHbDBAcnyQxLgYkuOCJMYFSY6LITEuSE3IUVUT2nar9cetqglxQL9MEmKDEf3M0ryKK6oJOUd6YlyzHVPXSBGRlqUErxnceuutzJkzhxkzZvD+++9z/PHHM2fOnK2VvB588EE6d+5MeXk548aN4/TTTyczM7PeMRYtWsSTTz7Jfffdx1lnncVzzz3HBRdcEI2PIyJ7qKSyhnmripi9spA5KwuZvbKQJetLcA4ykmLZr1cn9uvTibG9OzMyN73B5Ke8qpb1xZWsL/HFIlITYklNiCE1IZbkuODWdUZlVTXMX13EvFVFzA3fvllbTFVNqEU/8858esMRdM/QH+HbM7ME4EMgHn8dftY595vt9jHgX8BxQBlwiXPuq0jHtrGkiqraULMmeLpGioi0rHaX4P32lbnMW1XUrMcc2j2N35w4rMn7jx8/vl6Z5ttvv50XXngBgLy8PBYtWrTDxatv376MHj0agP32249ly5btddwiHdmm0iryNpURcg12XyEmECAtMYa0cAIVE6w/Y905x+ayalYVlLO6sILVhf6+uKKasqpaKqprKauqpbyqlvLqWorKq1m+qYwtb9clLZ4RPdI5YWQ3slPjmZVXyLTlm3hnwToAYoPG8B7pdM9IZENxpU/qiisprqxp9DMFDFLiY0iKi2FtccXW98pIimVY9zQuOagPA3JSiAkYNSFHKOSodX40rjbkcA4SYoPExwSIjw0QHxP+OSZAbEyA2ECAmKAf7YsJBogJGMGAUV0borSylvLqGkoraymrqqGsyn/+mIARHxsgLhgkLibgb0F/n5nSfElCO1MJHOGcKzGzWOBjM3vdOfd5nX0mAQPDt/2Bu8L3e2VX18iqmhA1oRBJcU3/80DXSBGR1qXdJXitQXJy8taf33//fd5++20+++wzkpKSOOywwxos4xwfH7/152AwSHl5eYvEKtKWVVTXsqm0im83lLJobTGL1pWwaF0Ji9eVsKm0areOlRIfQ1pCDGmJsVRU17K6sILK7UbDYoNGakIsibF+muKW+05JcfTslMQpY3owokc6I3qkk5OWUO+154f/NN9UWsX05ZuZtnwT05dtZt6qIrJT4xnSPY3vpMSTkxZPdko82anxmBnFFdUUV9RQXFFNUbm/L62qpWenJIZ2T2NY9zS6pSeogmAb4pxzQEn4YWz4tv03EScDj4b3/dzMMsys295Um24KM3DOBxOp3yhdI0VEIqvdJXi78y1ic0lNTaW4uLjB5woLC+nUqRNJSUksWLCAzz//vMH9RNqzvE1lfLJ4A58s2cjidSWkJ8bQOTmOTknhW3IcnZNjiQsGKa+upbyqhvItI2TVfpSsqLyaTWXVbC6tYlNpFQVlVZRW1S/SkJYQw8AuqUwc2oUBOSn0zkwmJtjwn6nVNSGKK2ooqqimsNwnT1t+josJcPTQLnRLT6R7RgLd0hPplpFAVnL8Xq+j65wcx9FDu3D00C57dRxp28wsCEwHBgB3Oue+2G6XHkBencf54W17leDt6hpZVF7Nso2lDMhJ2a1RvJ3RNVJEpGW1uwQvGjIzM5kwYQLDhw8nMTGRLl22/eF27LHHcvfddzNy5Ej22WcfDjjggChGKtK8nHOEHIScI+T8FMDakKO0soYvvt3Ep0s28PHiDeRt8t+2Z6fGM6x7GqWVNXyzppjNZdUUlFURangWJQDxMQES44KkJcTSKTmOzJQ4Buak0Ck5jk5JflvfzGQGdEkhOyVeI1nSJjjnaoHRZpYBvGBmw51zc+rs0tAvcoP/UszsCuAKgF69eu1VXHExfqpyVU2IpGaaYatrpIhIyzLXyPqU1mrs2LFu2rT6HRXmz5/PkCFDohRRy+ton1eaT0llDSs2lrFiUynLN5axfFMZeZvKMDOOHJzDxGF+1KoxNbUhPlu6kVdmruLNuWspLK/e6fulJsRwQL9MJvTPZMKALAbkpOyQgIVCjqKKajaVVlFd60iKC5IQG9x6H4xw5Ulp3cxsunNubLTjiCQz+w1Q6pz7W51t9wDvO+eeDD/+BjhsV1M09/YaGQo55qwqpGtawg7TjNsCXR9FpKPY2fVRI3gi7VRBWRUz8gqYkVfA1ysKmLuqkA0l9delZSTF0qtzEiWVNfzm5bn85uW5jOqZwTHDunDMsK70z04hFHJMW76ZV2auYvLs1WwsrSIlPoajh3ahV+ckAmYEDAIB2/pzXEyAMb06Mbx72g7FS7YXCBgZSXFkNNdwgUgrZ2bZQLVzrsDMEoGjgL9st9vLwNVm9hS+uEphpNffgf/3GBMMtKpqrCIisnuU4Im0A845lm4o5bMlG/l6RQFf521m6fpSwBdNGJSTymH75NA3K5nemUn07pxMr8wk0hNjtx5j8boS3py7hilz1/B/b3zD/73xDQNyUiitrGF1YQUJsQGOHNKFE0d257B9stXfTGTPdQMeCa/DCwDPOOdeNbMrAZxzdwOT8S0SFuPbJFzaUsHFBQNU1irBExFpq5TgibRR+ZvL+HTJRj5bspFPl2xgbVElAFkpcYzu2YnT981lTM8MRuSmk5oQu4ujwYCcFAbkDOCqwwewqqCct+at5e35a0mIDXLDpMEcNaQLyfH6X4bI3nLOzQLGNLD97jo/O+Cqloxri/iYAKU7adchIiKtm/5aE2nlampDrNhUxpL1pSxeV8KidcVMW7aZFZvKAMhMjuPA/pkc1D+LA/tn0iczaa8LjXTPSOTig/pw8UF9muETiEhbEhsToKosRMg5AipaJCLS5ijBE2lFCsqqmJlfyIwVBSxYU8TidSUs21hKde22YkjZqfGMys3g0gl9OKh/FoO67Fi4RERkT8WF181W14SI11RsEZE2RwmeSJRU1YSYu6qQmeFCKDPzC/l2w7Z1c30yk+mfncKRQ7rQPzuZATkp9MtOqbduTkSkucVvaZVQqwRPRKQtUoLXDAoKCnjiiSf44Q9/uNuv/ec//8kVV1xBUlJSBCKT1qQ25Ji3qohPl2zg0yUbmbpsE2XhRt05qfGM7pnBmWNzGZ3b9HVzIiLNbcsIXnNV0tQ1UkSkZSnBawYFBQX85z//2eOL1wUXXKCLVztRXRuioKyazWVVbC6tYnNZFSsLKvhi6UY+X7qRogpfuGBATgpn7JfLAf0yGdMrg65pCZpmKSKtQkzQMDOqmqmSpq6RIiItSwleM7jhhhtYsmQJo0eP5uijjyYnJ4dnnnmGyspKTj31VH77299SWlrKWWedRX5+PrW1tdx0002sXbuWVatWcfjhh5OVlcV7770X7Y8iO1FYVs3CdcWsLapgTWEFa4sqWFtUyZqiCtYVVbCxtIriioYrz/XsnMik4d04aEAmB/bLbJMNhEWkYzAz4pqxF56ukSIiLUsJXjO49dZbmTNnDjNmzGDKlCk8++yzfPnllzjnOOmkk/jwww9Zv3493bt357XXXgOgsLCQ9PR0brvtNt577z2ysrKi/CmkMTW1IR79bDm3vbWQkjqlw+NiAnRNS6BrWgLDe6STlRJPp6Q4OifHkpEUR+fkODolxZGVGkdOqhI6EWk74mMCVDZTgqdrpIhIy2p/Cd7rN8Ca2c17zK4jYNKtTdp1ypQpTJkyhTFjfIujkpISFi1axCGHHML111/PL37xC0444QQOOeSQ5o1RIuKrFZv59QtzmLe6iO8MyubSg/rQLcMndemJsZpWKSJtSxOvkd1qaqkJOVxcEGMX/5/TNVJEpFVpfwlelDnnuPHGG/n+97+/w3PTp09n8uTJ3HjjjUycOJGbb745ChFKU2wureL/3lzAk1/m0TUtgbvO35djh3dVQiciHYKZ4XutNy9dI0VEIq/9JXhN/BaxOaWmplJcXAzAMcccw0033cT5559PSkoKK1euJDY2lpqaGjp37swFF1xASkoKDz/8cL3XavpJ6xAKOZ79Kp9bX19AYXk13zukL9ceNYiU+Pb3T0VEOqAmXiMryqtZtrGUATkpJMXt3f//dI0UEWlZ+qu1GWRmZjJhwgSGDx/OpEmTOO+88zjwwAMBSElJ4bHHHmPx4sX87Gc/IxAIEBsby1133QXAFVdcwaRJk+jWrZsWkLew2pDj2w2lzFtdxNxVhcxbVcTcVUVsKq1iv96d+MMpwxnSLS3aYYqItLi4mG2tEpLi9u5YukaKiLQsi8QUjEgaO3asmzZtWr1t8+fPZ8iQIVGKqOV1tM/bnJxzvD5nDQ98/C3zVhVRXu370MUGjUFdUhnWPY0JA7I4cWR3AgFNxxSJNjOb7pwbG+042ormukbWhhxzVxXSNS2hTVX91fVRRDqKnV0fNYInHcaawgpuemkOb81by8CcFM4Z35Oh3dIY1j2dATkpW7+xFhHp6IIBIybQfK0SRESk5SjBk3YvFHI8NTWPP0+eT3UoxC+PG8x3J/QlJqiETkSkMXExgWZrdi4iIi1HCZ60a0vXl3Dj87P54ttNHNQ/kz+fNoLemcnRDktEpNWLiwlQVqf3p4iItA3tJsFzznWIEvZtbc1ktKwvruTpqSu4/d3FxMcE+MvpIzhrbM8O8TsiIrK9PblGxgUDFNaGCDlHoA38v1PXRxERr10keAkJCWzcuJHMzMx2/Qe8c46NGzeSkNB2Fry3FOccC9eW8Pb8tbw9fy0z8gpwDiYN78pvTxrWpooEiIg0pz29RsbFBHBAdU2I+Nhg5AJsBro+iohs0y4SvNzcXPLz81m/fn20Q4m4hIQEcnNzox1GqzF9+SZenbWat+evJW9TOQAjc9P58VGDOGpIF4Z2V5sDEenY9vQaWVkTYn1xJbWb4kho5Qke6PooIrJFu0jwYmNj6du3b7TDkBa0ZH0Jf3ptPu8sWEd8TICDB2Txg0MHcOSQHLpotE5EZKs9vUauKazglD+/w+9PGc6FB/SOQGQiIhIJ7SLBk46joKyKf769iMc+X05CbJAbJg3mogN7kxSnX2URkeaUkxpPXEyAvE1l0Q5FRER2g/4qljahujbEfz9bzr/eWURxRTXnjO/FT44eRFZKfLRDExFplwIBo2enRFZsVIInItKWKMGTVq2wrJq356/lzvcWs3RDKYcMzOJXxw9hcFetrRMRibRenZNYoRE8EZE2RQmetDp5m8p4e/5a3pq3li+/3URNyNEvO5kHLxnL4fvktOtKqSIirUmvzklMW7a5w7QiEhFpD5TgSauwurCcJ7/MY8rcNSxYUwzAwJwUrvhOP44a2oXRuRkEAvrjQkSkJfXsnERxZQ0FZdV0So6LdjgiItIESvAkqpxzPP/VSm55ZS6llTWM7d2ZXx03hKOHdqFPVnK0wxMR6VhKN0L5ZsgaAPgRPIAVm8qU4ImItBFK8CRqNpRU8svnZzNl3lrG9+nMX88cSe9MJXXSCjkHyz+BaQ9Bp96w36WQ0TPaUe2eUAgqCyGxU7QjkdbsfxdDdTl87x0AemVuS/BG9cyIYmAiItJUSvAkKt6Ys4ZfvTCb4soafnXcEL57cF+CmoIprU0oBAvfgI9vg/ypkJAOFUXw8T9gn+Ng3OXQ7zBo7WuTnIMXfwCznob+R8CY82Gf4yFWPSNlO12GwdeP+d/9QICenbYleCIi0jYowZMWVVhezS0vz+WFr1cyvEcaT541mkFdUqMdlkh9tdUw5zn4+J+wfj5k9ILj/gZjLoCSdTD9IfjqUVjwKmQO8IneqHMhMSPakTfsi3tg1lMwaBKsmQ3PfhcSMmDEmT7Z6za69Sep0jJyhkBVCRTmQafeJMfHkJUSp154IiJtiBI8aTEfLVrPz5+dxbriSq45ciA/OmIAscFAtMOS9qy6YvdGqTYs9knb1AegcAXkDIXT7oNhp0Ew/L/LTr3hqFvg0Btg3ksw9T544wZ4+xa/f/Y+4dtgf5/RGwLBSHy6pln2CUz5lR+xO/sxwMG3H8DXj/skdep90GW4H4lMSIf41O1u6dB1OMTsRs/JUC0s/xT6HhKpTyWRkjPU36+b53/X8YVWNIInItJ2KMGTiCuvquXPr8/n0c+W0z87med/cJDWckhkOAfrF8C8l33ytW4uZA6E3gdCrwOh1wHQqe+20apQCFZOgwWvwTeTYcNCv73nAXDcX2HgRAg08iVEbAKMOtvfVs2AWc/491v6Acx8ctt+MQnQqY8fMYtPqZM4pfn72EQIxEIgxieRgVgIhh9bI+8diAknZLvoB1m0Cv53iX//U+/a9ln6H+Fv5Zv9SOWMJ3xSW1Pe8HE694eT74DeB+38/QDWLYCXr4b8afCDT/yUP2k7sgf7+3XzYJ9JAPTunMTUZZujGJSIiOwOJXgSUV+v2MxPn5nJ0g2lfHdCX35+7D4kxEZxNKMj2ZLsLJoCC6fA5m9h7KWw/w98orEnxyvfDKXr/TTFyiLo+x2fpESTc7Bmlk/q5r8cTtLMJ3MH/8T/oTrvJT9aBZDS1T8XlwKL3vSfJxADfQ72Uy33meSnZO6O7qP9bYuKQli/0J//Dd/A5mVQWezPX8EK/3NlsZ8Kt6cyesNp9/rP0pCaSnjmIqgqhYtf8aNz20vs5D/zuMv949rqbXFtibEwH975HTw0ye931C0N/zevrfZTWj/8P39uT7tv22iQtB0JaZDeC9bN37qpV+ckXp65iqqaEHExmnUhItLaKcGTiKiqCfHvdxdx53uL6ZaeyBOX789BA7KiHVb7V1UK337kE5dFb/l1NOCn4GUNhHf/4NdjHXK9T/Yam3ZXUwVL34e5L/gEaUtSF6quv1+vA+GilyGmieXTl34As/8Hx/yx4YRjd636Gp6/wid1FvBJ2vgrYMiJkNp1236hkE+2VnwWvn3uk7ABR8Hg4/19c66fS0iHnuP8bWdCtb5iYaja/1xbHf65BmprwIUafl1RPrz6E590HfJTOPQXftSvrjdu9IVhznwYcgY3Le5gLCR19re69pnkf3c+vwsWvgkn/tOfsy1WzYCXroa1s/101kn/BynZTXtPaX1yhtRL8Hp2TiLkYFVBudrXiIi0AUrwpNktXFvMT56ZwZyVRZyxXy43nziUtITYXb9Q9lyo1idu7/4BqkshNhn6Hw7fuR4GHA3pPfx+eVPhnd/CG7+Az+7wicGoc/3UwFAtLPvYT9mb/7IfbUrIgNxxPkFMyYbkHEjJgeRs2LgYXvuJv530710X6Vj5FTx5ro9v/QK44PldTzHcmdnPwktX+VhOvN0nasmNfIkQCECXof427jK/zbnoFxYJBPdsNDVnMFz5Mbz+C/jwr7DkXT9iltnfP//14zDtATjoGhh26t7HGZcMx/7ZH+ulq+Gx02HUeXDkzfDlPfDJ7f6/w9mPw5AT9v79JLpyhvjfqdpqCMbW64WnBE9EpPVTgifN6qUZK/n5s7NIiY/hngv345hhXXf9opZWXeErCWb233Gkoi1a/43/ozv/S79m7MCr/MhaQ6NzPcf56XpL3/fT7l6+Gj75lx/5+mYylKz10+v2OQ6Gn+7XaTU2OtfvUL/G66O/+XVWB/yg8Rg3LYUnzoLkTDj4D/Da9fD4mXDBs7s/xTNU62P/5J/Q6yA469E9Gy2KdnK3txLS/Lq6QRPhlevg7kN8EtZtJLz6Yz999sjfNO979hwPV34EH/yfbxUx80nA+eqiE/+gHntNZGY9gUeBrkAIuNc596/t9jkMeAn4Nrzpeefc71okwJyhfiR54xLIGVyvF56IiLR+SvCkWYRCjn+8vZB/v7uY8X07c+d5+5KduhtV91rKhkW+6MTaOf5xRi9fIr776PD9mD1L+sI9o1pUbbVPcj6os+ZpxJm7TlzM/Ohev8N8xch3/+CLbAw6xid1AydCXFLTYjj8V3407s1fQtYgGHDkjvuUbvAjPqEaP2qXNRASO/tS/Y+f5ZO8uCaOClQUwnOX+3WF+13qpwI2dXpoezXsVMgdDy9eCa9c44u6JGfDGQ9tq/zZnGLi4cibYOjJ8OntfgS4of/usjM1wE+dc1+ZWSow3czecs7N226/j5xzLT8kmjPE36+bBzmD6ZKaQFwwoFYJIiJthBI82WtlVTX89JmZvD5nDWeP7cnvTxneOhfiz3zaj2zExPspfeWbYfUMv35o/svb9us60o9G9Dt018esqfRTHT+6DQ6+Dr7zswgFv53VM/30xDWz/R/4k/66+6NYZn6t2uATfPK1/RqupggE4NR74IGJ8OylcPm7kDVg2/NVpX7krmiVHznMGui3DzsFXK1P1p44G857ZtdJ5YbF8OQ5vljM8bdtm2opfgruhS/B53fCtAfh9Psbn67aXLqN9O8ju805txpYHf652MzmAz2A7RO86MgaBBbcug4vEDByOydqBE9EpI2IaIJnZscC/wKCwP3OuVu3ez4deAzoFY7lb865hyIZkzSv1YXlXP7INOavLuLXxw/hsoP7Yq1t6ltVKUz+Ocx4zE/pO/3+bWvStigv8EnTqq/92qVHT/LJz8Q/+BLzDVn8tj/upiW+ouG7f/AlxoecGLnPEgrB+3+Gj/7u/4BvjjVPZnuW3G0RnwLnPgn3HQ5Png2Xv+MLltTWwP8u9ef07Mf89L66hp/uP88LV/jXnft0w0le2Sa/HujVn/gRqYte8lNKpb5AAA76kb9Jm2FmfYAxwBcNPH2gmc0EVgHXO+fmNnKMK4ArAHr12s0KsA2JTfBT2Ndtyzd7qReeiEibEbEEz8yCwJ3A0UA+MNXMXt5uCspVwDzn3Ilmlg18Y2aPO+eqIhWXNJ+vV2zmiv9Op7yqlgcuHsfhg3OiHdKO1s7zUzI3LITv/DxcbbCBX/vEDD9i1+9Q2P/K8Kjc3317gYOu9uX2txTDKFjhKxQueNX3Bzv/OZ9wPHw8PP99uLxfZHp/VVfAC9+HeS/6aXHH/rn1rHnq1BvO+q9PjJ/9rh+Re+3HvprnCf/wBVAaMvJMP5L3wpXw1Hlw9n/9up+V03wftfxpsHGR37fLCDjn8a3Nl0XaOjNLAZ4DrnPOFW339FdAb+dciZkdB7wIDGzoOM65e4F7AcaOHeuaJbicIX6GQFivzklMX7YZ51zr+xJPRETqieQI3nhgsXNuKYCZPQWcTP0pKA5INX+1SAE24dcmSCv34tcr+flzs+iSFs/jl+/PoC5R7oW2Ped837PXf+4bSl/0ol9z1hSxCb765Ojz4O1bfKI34wlfsKIw3z828xUED7x6WzGTsx+Dew/zlSKveL95C7iUb4anzofln/hRxQOvbn1FQvpMgOP/Dq9c68/D2tl+yurY7+78daPO8YVTXroKbu21rTVAcrav4Dn6XH/fc//G2zqItDFmFotP7h53zj2//fN1Ez7n3GQz+4+ZZTnnNrRIgDlDfW/JqjKIS6JX5ySKK2soLK8mI6mDr3sVEWnlIpng9QDy6jzOB/bfbp87gJfx009SgbOda6zxk7QGZVU13PLyXJ6Zls/4vp25+4L96Jzcyi72zsFbN8Gn//ZJ3Wn3+dL+uyutu28kPe5yX47+xSv99qEnw8Q/QkbP7fbv5pO8h4/zo4YXPN88RS4K8uDxM/zI1ukPwIgz9v6YkbLfJX7U9Mt7YPT5vghLU4w534+iLv/UF7rJHecL4LS2JFakGYS/1HwAmO+cu62RfboCa51zzszGAwFgY4sFmTMUcLDhG+g+hp51WiUowRMRad0imeA19JfZ9lNHjgFmAEcA/YG3zOyj7aeqNPv6Atkjc1YWcs1TX/PthlKuPnwA1x41kNhgKyumUlvjR5BmPAbjvgeT/uJ7je2NnuP9urJvJvspkX0m7GTfcX5K4ktX+STz2D/v/Ni7qr65Zo5P7qpK4cLnfen71u6YP8Hg46D3hN1L0AYf3/hUTpH2ZQJwITDbzGaEt/0Svx4d59zdwBnAD8ysBigHznHONc/0y6bIGerv182H7mO29sJbvrGMkbkZLRaGiIjsvkgmePlA3SGOXPxIXV2XAreGL1qLzexbYDDwZd2dIrK+QJrMOceDnyzjL68voFNyLI9fvj8H9d/DCn27207AOZh6P1QU+LVxO+uZVl0Bz13m18YdegMcdkPzjQAFAk0vZjLmAp+Yff4f3yB8zPn1n9+0FOY8728bFvpqhD3G+lGr3P2gU18f99IP4OkLfAuE774RmXV9kRCMafp0WJEOyDn3MQ1/CVp3nzvws1yio3NfCMbDWl/XpW6zcxERad0imeBNBQaaWV9gJXAOcN52+6wAjgQ+MrMuwD7A0gjGJLtpQ0kl1/9vJu9/s56jhnTh/84YuedTMue/Ci/+AI75I+x70a73dw7eutn32gL44h5fJGW/S3as+lhR5It0LPsIjv0LHHDlnsXYXCb+AdbNhVevg+x9ILUbzH0B5jwHq77y+/Q6EMZ/zxcy+PoxP60RICnT9+T79kPIHOD7xKXnRuuTiEhHFAj6/3eFWyUkx8eQnRrP0vWlUQ5MRER2JWIJnnOuxsyuBt7Et0l40Dk318yuDD9/N/B74GEzm43/NvMXLbaAXHbp0yUbuObJGRRVVPO7k4dx4QG997x6WvlmeO0nUFMBL/8INi6GI29pfDQvVAuv/RSmPwRjL/NVI9++BSZf70fGjrwZhp7iR7q2NNJeM9uvtxt51h5+4mYUjIEzH/HFRh45EarD33p3H+OTv2Gn1k/aamtg/fxtlSNXTvMNx0/5j1+bJiLS0roM8zMJwgZ3TWXBmu2LfYqISGsT0T54zrnJwOTttt1d5+dVwMRIxiB75s25a/jRE1/Ts3Mi/71sPEO6pe3dAd++xSdil70FMx6HT/7lpyqeeu+Ovc9qq33Z/DnPwsE/9tUrzeCSV2HRFHjrN76ISY+xMOEaeOf3UJjne7ENOmbv4mxOSZ3h3Kdgyq98/73hp/neUg0JxkDXEf429tKWjVNEpCE5Q2Dmk/4LusRODOmWxsOfLKOmNkRMa1t/LSIiW0U0wZO26aUZK/nJMzMZ3iOdRy4dt/cV05Z9AtMf9g2Yc/eDHvtC1kDfS67wOJ8EpXb1+1ZX+ORt4es+sTvkJ9uOY+YTuAFH+T863v0jPHORb4Nw4QvQ+6C9izMSugz1sYmItDV1C630Pogh3VKpqg2xdENp62uNIyIiW+krOKnnyS9XcN3TMxjbuxOPX77/3id3NZW+qmVGLzjsRr/NDA74gR9xW78Q7jvCT6+sLPYVIxe+4fup1U3u6goEfSGTa77yFSsvm9I6kzsRkbYsZ4i/X+fb1w7u6mdyzF+taZoiIq2ZEjzZ6oGPv+XG52fznYHZPHzpeFLim2GA96O/w8ZFPhGLS67/3D6TfHVI5+DBY/1t+adw6j2+99yuxCb6Jtpb/ggREZHmk9YD4tO3Flrpn51CbNCYv7o4yoGJiMjOKMETnHP8+51F/P7VeRw7rCv3XrQfiXF72TsOYN0C+Og2GHGWn1bZkG4j4Xvv+rVpGxbC2f+FUWfv/XuLiMjeMfNfoIUTvLiYAANyUjWCJyLSymkNXgfnnOPWNxZwzwdLOW1MD/7vjJHbFs9vXgZlG6HHfrt/4FAIXrkG4lN23ew7rZsvvlK2yf8sIiKtQ84Q3+LFOTBjSNdUPl6sYtciIq2ZRvA6uL9N+YZ7PljK+fv34m9njtqW3IVC8MQ58OCkrd/e7pbpD0HeF3DMnyC5CU3RY+KV3ImItDY5Q6GiAIrXADCkWxrriivZWFIZ3bhERKRRSvA6sGem5nHne0s4d3xP/nDKcAKBOj3uFr3p+7K5Wnj+CqipavqBi1b5tgh9D/X960REpG3artDKlpY5C9ZoHZ6ISGulBK+D+mTxBn75wmwOGZjF704eXr+BuXN+7VxGLzj9AVgzCz74S9MO7BxM/hnUVvnCKnvaGF1ERKKvbqsEYHA33x5B6/BERFovJXgd0KK1xVz52HT6Z6dw5/n7Ert9w9rln0D+l3DQNTDsFBh9Pnx8G+RN3fmBnfMjdwte9S0RGmvqLSIibUNyJqR02TqCl5UST3ZqvCppioi0YkrwOpj1xZVc+vBUEmKDPHjpONISYnfc6aPbIDnb95oDOPZWSMuFF66AqtKGD+wcvP0b+OSfMPYynxyKiEjblzNka4IHfpqmRvBERFovJXgdSHlVLZc/Oo0NJZU8cPFYemQk7rjTqhmw5B3fiDw2/HxCGpzyH9j0LUy5acfXOAdv3Qyf/Msnd8f/HQL61RIRaRdyhvq2N6EQAEO6prJ4XQnVtaEoByYiIg3RX+EdRCjk+MkzM5iVX8C/zhnDyNyMhnf85J8Qn7Zjo/G+h8CBV8G0B2Dx29u2Owdv3QSf3u5fc/zfte5ORKQ9yRkKNeVQsAzwI3hVtSGWrm9kRoeIiESVErwO4i9vLuD1OWv41XFDOGZY14Z32rgE5r0E4y6DhPQdnz/iJsgeAi9e5XvWOQdTfg2f/hvGfQ+O+5uSOxGR9mZLoZW19StpapqmiEjrpASvA3hjzmru+WApFx7Qm8sO7tv4jp/8E4JxcMAPG34+NgFOuwfKNsBrP/XJ3Wd3wPgr4Li/KrkTEWmPsvfx9+FKmv2yk4kLBpTgiYi0UjHRDkAia1NpFb9+cQ7De6Rx84lD67dDqKtoFcx4Eva7GFJyGj9gt1Fw2A3w7h/84/Hfh0l/UXInItJexadARu+thVZigwEG5KQwX73wRERaJSV47dxvXp5LYXk1j12+/47tEOr67E5wITjoR7s+6IQfw9q50Lmfn7ap5E5EpH3rMmzrCB74aZofLlofxYBERKQxmqLZjr0xZzWvzFzFNUcMZHDXtMZ3LNsE0x6C4adDpz67PnAwBs58GI68WcmdiEhHkDMENi6CmioAhnRLZX1xJRtKKqMcmIiIbE8JXju1uc7UzCsP20XD8S/vg+pSOPjHLROciIi0LTlDIVTjkzy2FVpZoIbnIiKtjhK8dmrL1My/njFq51Mzq0rhi7th0CToMrTlAhQRkbYjZ4i/D0/THNw1FVAlTRGR1kgJXjv0xpw1vDxzFT86YuDWb1kbNfUBKN8Eh/ykZYITEZG2J3MgBGL8+msgMyWenNR4JXgiIq2QErx2ZsvUzGHd0/jBrqZmlm2Cj/4GA46GnuNbJkAREWl7YuJ8Ya0NC7duGtItTZU0RURaISV47cwtr8yloKyKv525i6mZAB/8BSqLYeLvWyY4ERFpuzIHwMbFWx8O6ZbG4nXFVNWEohiUiIhsTwleO/L+1Bm8OiOvaVMzNyyGqffDvhdvW1shIiLSmMwBsGkphGoBX0mzutaxZH1JlAMTEZG6lOC1E6tW5bP/axN5OuWf/PDQPrt+wdu/gZgEOPyXEY9NRETagayBUFsFBSuAOpU012gdnohIa6IErx3YXFrFk4/cSSKVjK2ZTuz7f9j5C5Z9AgtehYOvg5ScFolRRETauMyB/j48TbNfVjJxwQDz1SpBRKRVUYLXxpVX1fLdR6ZyUPkHlKf1hbGXwSf/hNnPNvyCUAim/ArSesABV7VorCIi0oZlhRO8Db4XXkwwwMAuKaqkKSLSyijBa8NqakNc/cRXrMr7lgMC80gcczYceyv0ngAvXQWrvt7xRXOe9duPvBniklo+aBERaZuSMiEhY2uzcwhX0tQInohIq6IEr41yzvGrF+bwzoJ13DFqOYaD4af7UtZnPgJJWfDU+VCybtuLqsvhnd9Bt1Ew4qzoBS8iIm2PmS+0sqF+grehpJL1xZVRDExEROpSgtdG/eOthTw9LY9rjhjAuJL3oOsIyB7kn0zJhnMe933unrkIaqr89s/vgsI8mPhHCOg/vYiI7KasgbBxydaHQ7qmAmiapohIK6K/8tugxz5fzu3vLubssT358dg4yJ/qR+/q6j4aTr4DVnwGr/8cStbDR7fBPsdB30OiEreIiICZ9TSz98xsvpnNNbNrG9jHzOx2M1tsZrPMbN9oxLqDzAFQvAoqfWsEVdIUEWl9lOC1RjOegCfOgdqaHZ56c+4abn5pDkcOzuGPpw7H5r7gnxh22o7HGXEGHPxjmP4QPHoSVJfBUb+NcPAiIrILNcBPnXNDgAOAq8xs6Hb7TAIGhm9XAHe1bIiNyKpfSbNTchxd0xK0Dk9EpBVRgtcaTXsQFr4OMx6rt7m0soYbn5/N8B7p3HHevsQEAzDnecgdD516N3ysI26CgRNh3TwY+91t0zhFRCQqnHOrnXNfhX8uBuYDPbbb7WTgUed9DmSYWbcWDnVH27VKABjcLVVTNEVEWhEleK1N+WZYOR0sAO/9aes0GIBHPlvGptIqfnvSMBLjgrD+G1g7e8fpmXUFgnD6/X7k7ohft8AHEBGRpjKzPsAY4IvtnuoB5NV5nM+OSWDL69wXsB0KrSxeV0JVTSh6cYmIyFZK8Fqbpe+DC8HRv4eStfDpvwEoqazh3g+Xctg+2Yzp1cnvO+c5nwgOO2Xnx0xI903NEzMiGLiIiOwOM0sBngOuc85tPwRmDbzENXKcK8xsmplNW79+fXOHWV9sImT03KFVQk3IsXhdyU5eKCIiLUUJXmuz+B2IT4f9r4ShJ8Ont0PxGh75dBkFZdVcd1R4iqVzPsHrczCkdo1uzCIislvMLBaf3D3unHu+gV3ygZ51HucCqxo6lnPuXufcWOfc2Ozs7OYPdnuZA+tN0RzazVfSnLOqMPLvLSIiu6QErzVxDpa8C/0OhWAMHPkbqK2m6u0/cO+HSzlicA6je2b4fdfM8hfYnU3PFBGRVsfMDHgAmO+cu62R3V4GLgpX0zwAKHTOrW6xIHdmS6sE5wcU+2WlkJ4Yy/Rlm6McmIiIgBK81mX9N1C0EgYc6R9n9odxlxMz83FyKr7luqMGbtt39rMQiIEhJ0UnVhER2VMTgAuBI8xsRvh2nJldaWZXhveZDCwFFgP3AT+MUqw7yhwAVSVQ7PPNQMAY27sTU5dvinJgIiICEBPtAKSOJe/4+/5Hbt1UfMCPcV88wt8znmNk7vf9xlAI5r7g90vqHIVARURkTznnPqbhNXZ193HAVS0T0W7a0iphwyJI6w7A2D6deWfBOjaWVJKZEh/F4ERERCN4rcnidyBrkF/AHvbg9CLuqD6ZkeVf+AIs4BubF+ZpeqaIiLS8zAH+vk6hlXF9fPGvacs1TVNEJNqU4LUW1eWw/JN6o3eF5dXc//FS8gdeBOm9YMqv/ejdnGchJgH2mRTFgEVEpENK7Q6xSbBhW6GVEbnpxMUEmLZM0zRFRKJNCV5rsfxTqKnYtv4OeODjbymuqOHqicPhyJthzWyY+YSfnjlwIiSkRTFgERHpkAIBv0a8zghefEyQUbnpTFWhFRGRqFOC11oseReC8dB7AgCFZdU89PG3HDusK0O7p/npmN1Gw2vXQ+l6GHFGdOMVEZGOa7tWCeDX4c1ZWUh5VW2UghIREVCC13osfgd6HwhxSQDc//FSiitruHZL5cxAACb+AWrKIS7Fj+CJiIhEQ9ZAKFgBNZVbN43v05makGNGXkH04hIRESV4rULhSlg/f+v6u4KyKh76ZBnHjejKkG51pmH2PQTGXgYHXg2xiVEKVkREOrzMgeBCsGnp1k379uqEGVqHJyISZWqT0Bosedffh9ff3ffRUkqrarj2yEE77ntCYz1xRUREWkhmf3+/YRHkDAEgPSmWfbqkMlWVNEVEokojeK3BkncgtRvkDGVjSSUPfbKM40d0Y5+uqdGOTEREZEcNtEoAGNunE18t30xtyEUhKBERASV40ReqhSXvQf8jwIy7P1hCRXUt1x3VwOidiIhIa5CQBild67VKABjXpzMllTXMX10UpcBERCSiCZ6ZHWtm35jZYjO7oZF9DjOzGWY218w+iGQ8rdKqr6GiAPofwbqiCh79bDmnjOnBgJyUaEcmIiLSuKyGK2mC1uGJiERTxBI8MwsCdwKTgKHAuWY2dLt9MoD/ACc554YBZ0YqnlZr8TuAQf8juPO9xdSEHNceOTDaUYmIiOxc5oAdpmj2yEike3qC1uGJiERRJEfwxgOLnXNLnXNVwFPAydvtcx7wvHNuBYBzbl0E42mdlrwD3cewsiqRJ7/M46yxufTOTI52VCIiIjuXOQDKN0Ppxnqbx/bpzLRlm3BO6/BERKKhSQmemT1nZseb2e4khD2AvDqP88Pb6hoEdDKz981supldtBvHb/vKCyB/Ggw4kjve9d+CXn2ERu9ERKQNyApfr7YbxRvXpxNriyrJ31wehaBERKSpCdtd+NG2RWZ2q5kNbsJrrIFt23+dFwPsBxwPHAPcZGY7VBcxsyvMbJqZTVu/fn0TQ24Dvv0AXC1rsg/if9PyOXd8T3pkqL+diIi0AVsqaW7YLsHr69fhTdU6PBGRqGhSguece9s5dz6wL7AMeMvMPjWzS80stpGX5QM96zzOBVY1sM8bzrlS59wG4ENgVAPvf69zbqxzbmx2dnZTQm4bFr8D8Wn8bV4awYBx1eEDoh2RiIhI02T0hkDsDiN4g3JSSU2IYeoyrcMTEYmGJk+5NLNM4BLgcuBr4F/4hO+tRl4yFRhoZn3NLA44B3h5u31eAg4xsxgzSwL2B+bv1idoq5yDJe9S0n0Cz89Yy0UH9iYnLSHaUYmIiDRNMAY694ONS+ptDgSMsb07qZKmiEiUNHUN3vPAR0AScKJz7iTn3NPOuR8BDdbzd87VAFcDb+KTtmecc3PN7EozuzK8z3zgDWAW8CVwv3Nuzt5+qDZhwyIozOOV0sEkxAa58tD+0Y5IRERk92QN3GGKJvhCK4vWlbC5tCoKQYmIdGwxTdzvDufcuw094Zwb29iLnHOTgcnbbbt7u8d/Bf7axDjaj0//hbMg/17Rl0sP70NmSny0IxIREdk9mf1h4ZtQW+NH9MLGbemHt3wzRw/tEq3oREQ6pKZO0RwS7lkHgJl1MrMfRiakDuDbD+Hrx3gj/SyK47vyvUP6RTsiERGR3Zc5EELVULC83uaRuenEBQOapikiEgVNTfC+55wr2PLAObcZ+F5EImrvqsvhlWupTOvNdWuO4bJD+pKRFBftqERERHbf1lYJi+ttTogNMiI3XZU0RUSioKkJXsDMtrY9MLMgoKxkT3z4V9i0lEc6X0cwLpHvHtw32hGJiIjsmcxwgtfgOrxOzF5ZSEV1bQsHJSLSsTU1wXsTeMbMjjSzI4An8cVRZHesnQuf/Iuq4efwjyXdOXl0d9ISGusyISIi0solZ0Jipx1aJQCM692Z6lrHzLyClo9LRKQDa2qC9wvgXeAHwFXAO8DPIxVUuxSqhZevgYR0Xsj+AeXVtZw7vle0oxIREdk7mQN3aJUAsF/vToAvtCIiIi2nSVU0nXMh4K7wTfbE1Adg5TTcqffy0HtFDOuexoge6dGOSkREZO9kDoAlOxba7pQcx6AuKVqHJyLSwpraB2+gmT1rZvPMbOmWW6SDazcK8+Gd30L/I5mRcTQL1hRz3v69qLOsUUREpG3KGgAla6CiaIenxvbpzPTlm6kNuSgEJiLSMTV1iuZD+NG7GuBw4FHgv5EKql1xDib/DFwITriNJ6fmkRQX5KRR3aMdmYiINAMzu9bM0sx7wMy+MrOJ0Y6rxWQ2XEkTYP++nSmuqGFmfkHLxiQi0oE1NcFLdM69A5hzbrlz7hbgiMiF1Y7Mfxm+mQyH/5KixB68MnM1J4/uTqqKq4iItBffdc4VAROBbOBS4NbohtSCcob4+8Vv7/DUYYNyiA0ab8xZ08JBiYh0XE1N8CrMLAAsMrOrzexUICeCcbUPVaUw+efQbRTs/wNe+nqliquIiLQ/W+bbHwc85JybWWdb+5c1EIacCB/+DTbUH8VLT4plwoAsJs9ejXOapiki0hKamuBdByQB1wD7ARcAF0copvZjybt+XcJRv8UFgjz+xQoVVxERaX+mm9kUfIL3ppmlAqEox9SyjvsbxCbAK9dAqP5HP25EN/I3lzN7ZWGUghMR6Vh2meCFm5qf5Zwrcc7lO+cudc6d7pz7vAXia9sWvgnx6dDnYGbkFai4iohI+3QZcAMwzjlXBsTip2l2HKldYeIfYPknMP2hek9NHNqFmIAxebamaYqItIRdJnjOuVpgP1NWsnucg0VvQf/DIRjLk1+uUHEVEZH26UDgG+dcgZldAPwa6HjDVWMuhL7fgbd+A4Urt27OSIrjoAFZvD5H0zRFRFpCU6dofg28ZGYXmtlpW26RDKzNWzPLT88cdAxFFdUqriIi0n7dBZSZ2Sjg58ByfLXpjsUMTrwdQjXw2k/8F51hxw3vyvKNZcxdtWMrBRERaV5NTfA6AxvxlTNPDN9OiFRQ7cLCKf5+wFEqriIi0r7VOD80dTLwL+fcv4DUKMcUHZ37whG/hoVvwJzntm6eOKwrwYDx+pzVUQxORKRjaFKCF153t/3tu5EOrk1b9CZ03xeXnM3jX6xgeI80RuZmRDsqERFpfsVmdiNwIfBaeO36TqdrmNmDZrbOzOY08vxhZlZoZjPCt5sjEHdkHPAD6L4vvP4LKN0IQOfkOA7sl8nk2Ws0TVNEJMKalOCZ2UPhi1G9W6SDa7NKN0D+NBh0zNbiKhq9ExFpt84GKvH98NYAPYC/7uI1DwPH7mKfj5xzo8O33+19mC0kEIST/g0VBfDmjVs3TxrRlW83lLJgTXH0YhMR6QCaOkXzVeC18O0dIA0oiVRQbd7itwEHAyequIqISDsXTuoeB9LN7ASgwjm30zV4zrkPgU0tEV9UdB0OB/8EZj3tC44BxwzrSsDg9dmapikiEklNnaL5XJ3b48BZwPDIhtaGLZoCyTkUdx6m4ioiIu2cmZ0FfAmcib8+fmFmZzTDoQ80s5lm9rqZDWuG47Ws71wPWfvAK9dBVRlZKfHs3zeT19T0XEQkopo6gre9gYDmHDaktsaP4A08mvcWbqS8upbT9s2NdlQiIhI5v8L3wLvYOXcRMB64aS+P+RXQ2zk3Cvg38GJjO5rZFWY2zcymrV+/fi/fthnFxMOkW6EoHxa8CsBxI7qyZH0pi9ZpEpCISKQ0dQ1esZkVbbkBrwC/iGxobVT+l1BRCAMn8ta8tWSlxLFvr07RjkpERCIn4JxbV+fxRvb8C1QAnHNFzrmS8M+TgVgzy2pk33udc2Odc2Ozs7P35m2bX9/DIKM3fP0YAMcM74oZTNY0TRGRiGnqFM1U51xandsg59xzu35lB7TwTQjEUNX7UN5fsI4jB3chGFCPeBGRduwNM3vTzC4xs0vw69Un780BzayrmVn45/H46/XGvY60pQUCMPo8+PZDKFhBTmoC4/p0VoInIhJBTR3BO9XM0us8zjCzUyIWVVu26C3odSCfr6qhuLKGicO6RDsiERGJIOfcz4B7gZHAKOBe59xOZ7mY2ZPAZ8A+ZpZvZpeZ2ZVmdmV4lzOAOWY2E7gdOMe11YVro84FHMx8CvBNzxeuLWHxOlXTFBGJhKZOIfmNc65wywPnXAHwm4hE1JYV5MG6uTBwIlPmrSExNsiEAQ3OqBERkXYkXITsJ865HzvnXmjC/uc657o552Kdc7nOuQecc3c75+4OP3+Hc26Yc26Uc+4A59ynkf8UEdKpN/Q5BGY8Ds4xaUQ3AF6fvSbKgYmItE9NTfAa2i+mOQNpFxZNASA0YCJvz1vHoYOySYgNRjkoERGJhO3Xp9e5FYfXq8sWYy6Azctg+ad0SUtgbO9OvKZpmiIiEdHUBG+amd1mZv3NrJ+Z/QOYHsnA2qRFUyCjN7Mru7CmqIKjh2p6pohIe9XA+vQtt1TnXFq042tVhpwIcakw4wkAJo3oxoI1xSxdr2qaIiLNrakJ3o+AKuBp4BmgHLgqUkG1SdXlsPQDGHQMb81fRzBgHDE4J9pRiYiIRF9cMgw7Bea+AJUlTBreFYDX52iapohIc2tqFc1S59wNW8owO+d+6ZwrjXRwbcqyT6CmfOv6u3F9OtEpOS7aUYmIiLQOo8+H6lKY/zLdMxIZ0yuDV2ep6bmISHNrahXNt8wso87jTmb2ZsSiaosWvQkxiSxPHcPCtSVMHNo12hGJiIi0Hr0OgM79tk7TPH3fXOavLuKzJW2v+4OISGvW1CmaWeHKmQA45zYDHWv+4ZrZMPtZqK7Y8TnnfP+7focyZaFfV6/1dyIiInWY+Z54yz6CTd9yxn655KTG8+93F0c7MhGRdqWpCV7IzHpteWBmfYCONadiyk3w3GXwj2Hw7h+huM66gQ0LoWD51umZQ7ql0bNzUvRiFRERaY1GnQsYzHyKhNggV3ynH58t3cj05ZujHZmISLvR1ATvV8DHZvZfM/sv8AFwY+TCaoUKlkOPsZA7Fj78K/xjODx/Baz8yo/eAZt6HMb05Zs1eiciItKQ9Fzod5ifphkKcd7+veiUFMud72kUT0SkuTS1yMobwFjgG3wlzZ/iK2l2DKGQb2Le9xA472n40XQYdxkseA3uOxze+yPkDOXtlXGEHExUgiciItKw0edD4QpY/jFJcTFcdnBf3l2wjjkrC6MdmYhIu9DUIiuXA+/gE7ufAv8FbolcWK1MyRoIVUNGeJZqZn+Y9Bf4yXw49la/aHy/S5kyby09MhIZ1l3tj0RERBo05ASIT9tabOXCA/uQGh/Df97XKJ6ISHNo6hTNa4FxwHLn3OHAGGB9xKJqbQpW+PuMXvW3J6TBAT+AH35G2Zjv8tGi9Rw9tAtm1vIxioiItAWxiTD8NJj3ElQWk54Yy0UH9eb1OWtYvK442tGJiLR5TU3wKpxzFQBmFu+cWwDsE7mwWpmtCV7vRnf5aNEGKmtCWn8nIiKyK6MvgOoymPsiAN+d0JeEmCD/eX9JdOMSEWkHmprg5Yf74L0IvGVmLwGrIhVUq1Ow3N+n5za6y5S5a0lLiGF8384tFJSIiEgblTsWMgfC9IcgVEtmSjzn7d+Ll2asYsXGsmhHJyLSpjW1yMqpzrkC59wtwE3AA8ApEYyrdSlYAck5flpJA2pqQ7y7YC1HDM4hNtjUnFlERKSDMoMJ18DK6fDmrwC44jv9CJpx94caxRMR2Ru7nY045z5wzr3snKuKRECtUsGKHdff1TFt+WY2l1UzcVjXFgxKRESkDdv3Ijjgh/DFXfDZnXRJS+DMsbk8Oy2fNYUV0Y5ORKTN0nBTU+wiwXtr3lriggG+Myi7BYMSERFp4yb+EYac5Efx5r7IlYf2p9Y57v1wabQjExFps5Tg7cqWHng7SfDemb+WgwZkkhIf04KBiYiItHGBAJx2L/QcD89fQc/imZwyugdPfLmcjSWV0Y5ORKRNUoK3K9v3wNvOhpJKlm0sY0L/rBYOTEREpB2ITYRzn4KMnvDUuVw72lFZE+L+j7+NdmQiIm2SErxd2UWLhJl5BQCM6pnRMvGIiIi0N0md4fxnIRBDr8kXcsGweB74+FuWbSiNdmQiIm2OErxdaazJedjMvAKCAWN4j7QWDEpERKSd6dwXznsaSjdwc/HvyAhWcfPLc3HORTsyEZE2RQnermzpgZfRs8Gnv84rYFCXVJLitP5ORERkr/TYD854iNh1s3msx/N8uHA9k2eviXZUIiJtihK8XdlJDzznHDPzChjdMz0KgYmIiLRD+xwL+13CwHVvMLZbDL97dS7FFdXRjkpEpM2IaIJnZsea2TdmttjMbtjJfuPMrNbMzohkPHtkJy0Slm0so6iihlG5GS0bk4iISHs2+nyspoLbhi9jXXEl/3x7UbQjEhFpMyKW4JlZELgTmAQMBc41s6GN7PcX4M1IxbJXdpLgzcjbDMDoXhktGJCIiEg712M/yBxArxUvc974Xjz86TLmrSqKdlQiIm1CJEfwxgOLnXNLnXNVwFPAyQ3s9yPgOWBdBGPZM7vogTczr5CkuCADc1JbODAREZF2zAxGnQPLP+YXBySRkRjLr1+cTSikgisiIrsSyQSvB5BX53F+eNtWZtYDOBW4O4Jx7Lld9MD7Oq+A4T3SCQashQMTERFp50aeDUDawuf55XFD+GpFAc9My9vFi0REJJIJXkNZz/Zfvf0T+IVzrnanBzK7wsymmdm09evXN1d8u7aTHniVNbXMX1XEGPW/ExERaX4ZvaDPITDzSU4b053xfTtz6xsL2FRaFe3IRERatUgmePlA3d4CucCq7fYZCzxlZsuAM4D/mNkp2x/IOXevc26sc25sdnZ2hMJtwE564C1YXUxVbUgNzkVERCJl1DmwaQm2cjp/OGU4JRU13Pr6/GhHJSLSqkUywZsKDDSzvmYWB5wDvFx3B+dcX+dcH+dcH+BZ4IfOuRcjGNPu2UkPvBl5BQCMVoInIiISGUNOgpgEmPkkg7qkctkhfXlmWj7Tlm2KdmQiIq1WxBI851wNcDW+OuZ84Bnn3Fwzu9LMrozU+zarnfTAm5lXQHZqPN3SE6IQmIiISAeQkAaDT4A5z0FNJdceOZDu6Qnc/NJcalVwRUSkQRHtg+ecm+ycG+Sc6++c+2N4293OuR2KqjjnLnHOPRvJeHbbzlok5BcwKjcDMxVYERERiZhR50JFASx8k6S4GH55/BDmrS7i6akquCIi0pCIJnhtXiMJXmFZNUvXlzJG/e9EREQiq99hkNIFZj0NwPEjurF/38789c0FFJZVb9uvpgoWvgm1NdGJU0SklVCC15id9MCbtbIAgFG5GS0bk4iISEcTjIERZ/rkrXQjZsYtJw2jsLyaf7y90O9TWw3PXgpPnAWf3RHdeEVEokwJXmN20gNvZrjAyojc9BYOSkRE2gMze9DM1pnZnEaeNzO73cwWm9ksM9u3pWNsVUad66/Jc58HYEi3NM7fvzf//Xw536zcBM9+Fxa8Cp36wEd/h9IN0Y1XRCSKlOA1Zic98GbkFdA/O5n0xNgWDkpERNqJh4Fjd/L8JGBg+HYFcFcLxNR6dR0OXUbAzCe3bvrJ0YPIiDc2PXYJzH8Zjr0VzvsfVJXC+3+OXqwiIlGmBK8xjfTAc84xI69Q/e9ERGSPOec+BHZW6/9k4FHnfQ5kmFm3lomulRp1DqycDuv9tMxOiUGe7fooB5Z/wIIRP4MDfgDZg2Dsd2HaQ7D+mygHLCISHUrwGtNID7xVhRVsKKlU/zsREYmkHkDdMpH54W0d14gzwQIw6ykI1cJLV9F39WQeSriIyxYdRHlVrd/vsBsgLhmm3BTdeEVEokQJXmMa6YE3Y0UBoAbnIiISUQ314Gmw8ZuZXWFm08xs2vr16yMcVhSldoH+R8LMp+GVa/x0zcN/xZCzbmFlQTn3fLjE75ecBd+5Hha9CUvei27MIiJRoASvMY20SJiZX0BcMMDgrmlRCEpERDqIfKDuFJJcYFVDOzrn7nXOjXXOjc3Ozm6R4KJm1DlQlA9fPwbf+Tkc+nMO6JfJ8SO7cdf7S8jfXOb3G/99fw2f8ms/2rcrTk3TRaT9UILXmEYSvBl5BQztnkZcjE6diIhEzMvAReFqmgcAhc651dEOKuoGHw9dR8Khv4DDf7l18y+PG4IZ/HnyAr8hNgGO+i2snQMzHm/8eBsWwwMT4f6joKYywsGLiLQMZSkNaaQHXk1tiNn5hZqeKSIie8XMngQ+A/Yxs3wzu8zMrjSzK8O7TAaWAouB+4AfRinU1iU2Ea78yCd3tm0Wa4+MRH542ABem72aZ6aGly4OOxVyx8O7f4DKkvrHcQ6m3g93Hwzr5sPKafDeH1vwg4iIRI4SvIY00gNv0boSyqtrleCJiMhecc6d65zr5pyLdc7lOucecM7d7Zy7O/y8c85d5Zzr75wb4ZybFu2YW7srvtOPQwZm8fPnZvHoZ8t8AnjMn6BkLXzyr207Fq+Bx8+A134KvQ+Cq76E/S6BT26HZR9HK3zZHWU7K0DbwayeCa/f4NuDiIQpwWtIIz3wZoQbnKtFgoiISOuSEBvk/ovHcvTQLtz80lzu/mAJ9BwHw0+HT/8NhSth3kvwnwNg2Sdw3N/ggucgrRtM/KNvkv7ClVBRGO2PIjuz4nP46wCY/2q0I4m+5Z/BwyfAF3fB9IejHY20IkrwGtJID7yZeQWkJ8bSJzMpCkGJiIjIzsTHBPnP+fty4qju3Pr6Am6b8g3uyJvBhfxau2cugk59/TTP8d/bNs0zPgVOuxeKVvrREGmdQiF44wZwtTDr6WhHE12L34b/ngopXaD7vvDpHVpHKlspwWtIIz3wZuQVMKpnBmYNVa8WERGRaIsNBvjn2aM5a2wut7+7mD9+UoY76EdQvBoOvQEumwJZA3d8Yc/xcMj1MPMJP9Inrc/s/8Gqr32Svuitjjstcd5L8MQ5/vf4u2/AEb+C4lVKeluTtXNh7otRq9CrBK8hDfTAK62sYeHaYq2/ExERaeWCAePW00ZyyUF9uP/jb/lV4cmEfroQDr8RgrGNv/DQn0P3MfDKdX6tnrQeVWXwzm/9f58T/wU15bBoSrSjanlfPwb/uwR67AcXv+L7PvY/0leX/fifTWsLIpFVvhkeOx3+dzE8dT6UbmzxEJTgNaSBFglzVhYScjC6Z3qUghIREZGmCgSM35w4lCsP7c8TX+Zx/eR8QqFdfJsejIXT7oPqcnjpKvXHa00+u8NPoT3mT9DnYEjO7ngjrZ/f5X8v+x0GFz4PiRl+uxkc8hPYtATmvxzNCFu31TNh9azIv8/rv4CSdXDg1f5LiLsnwNL3I/++dSjBa0gDCd7M/AIARuVmtHw8IiIistvMjF8cuw8/PmoQz3+1kr+8uWDXL8oaCBN/79c4TXtg99+0ugIWTtFISnMqWg0f/wOGnOQrnwaCMOREWPimH9nrCD74P7/+cMiJcO5TEJdc//khJ0HmAPjoNn0x0ZDVM+GBY+C/p0S2kNK8l/1U2e/8DI75I3zvHYhPhUdPgbduhpqqyL13HUrwthcKQWH+DgnejLwCcjslkpkSH6XAREREZHeZGdccOYALDujFPR8s5YkvVuz6ReMu99Pe3vw1bFi0e2/41s3wxJnw/BVQW71nQUt97/4BQjVw9G+3bRt6ClSXweK3ohZWi1nwmu/TOOpcOONhiGngb9FAECZcC2tmwZJ3WjzEVq1kHTx5ni+mVLYRPvp7hN5nPbz6Yz9d9jvX+23dRsEVH4RbsfwLHjgaNiyOzPvXoQRveyVrobZqhwRvVn6h2iOIiIi0QWbGLScO47B9srnppTl8sHD9rl4AJ98JsQnw3OVN/9Y970v48l7/B96cZ+GZi1XZcG+tmgEzHof9vw+d+23b3nsCJGW2/2malSUw+eeQMxRO+jcEYxrfd+Q5kNodPvpHy8UXDc5B8dqm7VtTCU9f4BO785/1SfLnd8Hm5c0f02s/hsoiOPWe+mt945LgxH/C2Y/5Qo73fMevpYwgJXjba6BFwqbSKvI3lzOyh9bfiYiItEUxwQB3nLcvg7qkctXjX7FgTdHOX5DWzf9BvXqGHz3ZlZoqePlHkJ4Ll06GSX+Fb16DJ8/pONMIm5tzMOXXkNTZVzitKxjjpyt+84ZfM9lWLP8MFkxu+v4f/AWK8uGEf+y8QBBATBwcdDUs/9h/2dDeOOenP993BPx9kE98qyt2vv9rP4G8L+CU/0D30XDETWBBX7CnOc3+H8x/BY74NXQZ2vA+Q06EKz+BHvvC2nnN+/7bUYK3vQYSvFnh9XcjcpXgiYiItFUp8TE8eMlYkuODfPehqawr2skfh+D/INv3Yj+16tsPd77vx/+A9Qv8H+LxqbD/FX4UcOn78PgZUFncbJ+jw1jwGiz7CA67cVtBkbqGngzVpbC4jUxJrK4IV1Y8zyemu7J2Hnz+HxhzIfQ6oGnvse/FkNjJr8VrL5zz6y3vO9xPfy7bACPOgi/vgQeOanwa9Rd3+5Gy7/wchp/mt6X3gIN+BHOeg7ypzRNf0SqYfD303N8XVtmZ9B5w0Utw1C3N896NUIK3vS098NK39cCbne8XY47QCJ6IiEib1i09kQcuHkdBeTWXPTKNsqqanb/g2D/74hXPfx/KNjW8z7oF8OFfYcSZMPDobdvHXACn3+9HEB492ZdPl6apqYK3boKsfWC/Sxvep88hkNgZ5r3YoqHtsZlP+qVAaT381N/13zS+byjkR5/i0+Do3zX9PeJTYP8rYeHrER8lirh6id1ZfprlSf+GH30Fp98H5z4NhSvhnkPh68frF5dZ/A68+UsYfIL/gqCuCdf6BvFv/rJpBWl2tpbWOXjpar/PKXf5tZC7Egj60dYIUoK3vYIVvvRuXNLWTTPzC+mXnUxqwi6GxkVERKTVG94jnTvOG8PcVYVc8+QManfWPiEu2SdppevhlWt2/IMwFPLb41Ph2FsbeLPT4az/wprZ8PCJvhCDeJuWwlePwuxnYf6r/o/y5Z/5dXcf3+afP+aPja87C8bC4OPD0zR3MRobbaFaPxLcfV+47E3fa/nJcxpP+mc8Dis+88ldUufde6/xV0Bssh9Vbqs2LoH7j9oxsdv3om1TVfc5Fn4QnvL40g/h+e9BRZEvYvLspZA9xK+HC2yX7sSn+KmU+V/u+suBL+6FP3WH24b6/oOf3wUrv9qW9E1/2Be1Ofp3kNm/mU/CntvJSs0OqoEWCbNXFnBQ/6woBSQiIiLN7YjBXbjlpGHc/NJcfvfKXG45aRhm1vDO3UfDkTf5CplfPQr7XbztuWkP+BG6U+/xTacbMvg4OO9pX8nvkRPg8nf8H5kdWdkmeGCiT5wb0/8IGHDUzo8z7BT4+r+w5F1/nlureS/B5m99IpCe6wtuPHw8/O9SX/yjbhJbtsn/rvU6EEafv/vvldQZxl7qk5EjfgWd+ux9/DWVfipidTnkDPGFiCLlmzd8FdpAAE68HUaf1/j6w7TufsrjR7fB+3+C/Gl+hCwQA+c+2fi/s9Hnwxf3wFu/gX2O27EyaW2Nb0sx9T7/e5jY2f87n/uCfz4m0TebX/U19D0Uxl7WfJ+/GSjB217BCug2cuvDtUUVrC2q1PRMERGRduaiA/uQt6mM+z76lriYAL88bkjjSd6BP/IjTG/c4HuxZQ30bZXe/q3/A3Dk2Tt/s/5H+D84/3sqTP4ZnHpX0wMt3eDXVTVl+ldb8cYNfvTq4lchtatveVBdXue+3Df03lUi0fdQSMjwCVRrTfCc86NpmQP9lEGAXvv79ZovX+2noh775237v3Wzr8Z4/G07jj411YFX+QTmk9vhhN1Yj+cczHzKFxcqzPfN5QtXQum6bfuMu9wXEdrT2BoTCsEHt/rCMt1G+ZHvTr13/bpAEA79GfQ52E99LczzSd/OXhsI+n6X/z3Vn6cJ12x7rqLQJ95L3vHr9Y767bZ/e4Ur/cjfii98wpec5dfaNve52EtK8OoKhfwvxZATtm6aFV5/N1IFVkRERNqdXx43hKqaEPd99C0BM26YNLjhJC8QgFPvhrsOgucug8vehtd+Cq7W/6HelBGN/of7Bsgf/p//eeRZu37N8k99k+R9JsGZD0d25KSlLHzTN4M+9BfQ95C9O1Yw1idN81/2o0wN9YiLtiXv+v50J91RPxHY90JYO8cXUukyzK/ZXPG5H5GccG3j1RibIq27P/60B3yic9A1u/7d2VIJdtZTEJfqC4Kk9YAuw/2oY1oP/zm+vNdPUTzhn82X2JRv9qN2i6b40bXj/+6nse6O3gfCDz/1fe+yBu56//5HwMCJ8OHf/HsmZ8LmZfDE2bBxsR89rDtaD/6cpJ8Kw07dvdhamBK8uhrogTc7v4CAwbDuSvBERETaGzPjlpOGUesc93y4lEDA+Pkx+zSc5KV193+kP30+PHqSXyM18Y+7NwXu0F/4ypCv/thP8drZup218/w6rZh4v1Zo6v0w/nu7+xFbl4pCeOU6vz5q+9YHe2royTDjMV+xdNAxzXPM5vTxP3x/uoYS+ol/9NVXX/0xdOrrqzGm9/S/J3vrmD9BeYEfEdy4xCdNjU11LC/w/eKWfQSH/9o36m7o34Bzfr3pR3/36wpPun3vR5bXzPH/pgpX+lHLsd/d8y8yEtL9ramO/r3/0uaDW2H4Gb7CaagaLnge+h26ZzG0Aq1rPDHatrZI2DakO2tlIYO6pJIY146mRYiIiMhWZsbvThrO+fv34q73l/C3Kd/gGquuN+QE2O8Sn9x1H+MrFu6OYAycdp9fI/TcZY03US/Mh8dO92t9rvwYBhztq/6tnrl779favHUzlKzx09qaq5Jgv8MgPh3mvtg8x2tO+dN80nTgVQ2PLgZj4IyH/OjYoyfBunkw6S++uM/eik2E0x+AQ34KXz0Cj5/pE+ztFeTBg8f63+lT7vbTHRtLsMx8L7lDb/BJ9Ys/9Inenqgo8l9a3H+UH329dDKMu6xlR6lzBvt/z1Mf8OtjE9Lg8nfbdHIHSvDq264HnnOOWfmFWn8nIiLSzgUCxu9PHs6543tx53tLuO2thY0necf8yU95O+3+xis87kxGTzj5Dl+goaGGy+WbfXJXVQIXPOen2J16DyRl+Up+Fbto0g6+ufrKr3Y/tkha+oGvOnjgVZC7X/MdNybOr7/75rXGE+Zo+fgffo3g9lP96krqDOc+5ZP5wSf4yqDNJRCAI2/2CfWyj3xhm83Ltz2/epZPsIpW+t+10efu+phmcPiNfqRv1lO+emXtLtqNbFFRBLOe8QWH/jrAT3PusS9c8QH0HL9nn3FvHXajH/XLHe8LIGUNiE4czUhTNOvatMTfh3vgrSwoZ1NpFSN7ZkQvJhEREWkRgYDxx1OG45zj3+8uJmDGj48etOOOccm+QMPeGHKir7z32R1+BGpL/7zqcnjyXN8i4ILnoOtwvz05E854AB4+AV69zo/MNDbSsX4hPHMRrJ8P477nC3g0NjWvOdRU+rVkXUc1nvBWlfr1XZ37w+G/av4Yhp7i+8x9+0H9XoR7Y8XnPkErL/DnLxgXvoV/Tkj3yWpj02zXfwMLXvWNtuNTd/5eOYPhulm73m9PjbnA/337zIVw/5G+h1z5Zt94PSEdvvvm7q/5O/Rn/r/327dAqMb/Tm7/e1ZV6iul5n3pR1gXvw21lX7K6rjL/H+33HHRLVKSkg3Xzfb/rtvDGleU4NWXPw2yB2/tgbelwflIjeCJiIh0CIGA8adTRxByjn+9s4jakOOnEwc1Xl1zbxzzR59EvHCl7+eVnO2rAK74HM54EPp+p/7+vQ+Cw38J7/7eN/ke20AD8NnPwsvX+Ol5o8/3Zd7XzYMzH/F/yDan6gpfEOTjf/gRoE59YMJ1vqz99tMR3/k9FCyHS1/f/eIZTdH/cN8UfO6Le5/grV/oR1YXvOobYmcP9kVFqgt9rYbaan9fvBpmPOFHyPb//o5r0T75lx+V2//7TXvf3e13t7v6HQqXveWnaj58nP8cOUPh/Gf8+tI9cfCPIRALU34FRashKdMndFtu1WXb9k3t7tfXDTs1+knd9tpZ2xIleFuEQpA/1X+jFjYzv5DYoDG4W4S+TREREZFWJxAwbj1tJAEz7nhvMQXlVfz2pOEEA82c5MUm+kTu3sN8BcHMAT6pOPZWGH5aw685+Cew/BN4/ReQOxa6jvDbayphyq99hcOeB8CZD/k/2vsd7kfO7j0MznnMrxvcW9UVvh/gx/+A4lX+/Q7+sU92Xr0OPvg/X15+v4v9qMiKL+CLu/1oYu+D9v79GxIT7yuNLngFvjnBj4rubiJZvAbev9V/ttgkv9bsgB9u/eJ/B0Wr/ed980bfpuHkO7dN7yvM95VCx17WeH/EaMjex09DfO67PiE+9e69HzU86Gp//j+53Sd0ydnQuZ+/T8n291mDoMfY1pXUtWPW6PzyVmrs2LFu2rRpzX/g9QvhznFw0r9h34sAOP/+zykqr+GVHx3c/O8nIiK7ZGbTnXNjox1HWxGxa2QH5Zzj1jcWcM8HSzlhZDduO2s0cTER+AN1+sPwyrX+5wnX+mbYO1OyHu4+2I86XPF+eKrdJbByOhx4NRx1S/2pcqtnwlPn+xGVE/8Fo87Zsziry2H6I/DJP/3oVa+D4LBf+F50Zr7C4tL34MO/w/KP/WjOAT+AmU/7BPSHn0ZuCiJA/nR47FRfSCQ2GQYc6b+4HzgREjMafo1z/vx9cTd8+m8/Mjf2Mjj0501LzJzzidzrP/ef8Yib/Gee8mvfX+3aGfWqs4s0l51dHzWCt0X+l/4+1y/w3FJg5cRRezhkLSIiIm2amXHjpCF0Sorj1tcXUFRRw90X7EtSXDP/+bTvxX69FgZH3rLr/VOy4fT7fdXFp86DNbN9JcOz/gtDT9px/26jfCL4v0vghe/7hO/o3zetQExtjR8xnPeS7zVXuh56T4DT7vXTROtOXTXzvcX6H+GnmX74N3j3D/65C56PbHIHvnDL9Yt9MZEFr/nb/Jd9xdI+h0C3kVC20TeOL1m3bRphTYV//bBTfYK2s9YV2zPzCXO/w3z7hym/8udq7RwYcaaSO4kKJXhb5H3pF5lm+cXUyzaWUVxRwyg1OBcREenQrjy0P52SYrnx+dmcf/8XPHTJODKSmqnEP/gk4dg/795r+h7iq/+990foMgLOemTniUlyFlz4Aky5yTfWXvSW78OXM9j3pMsZDOm9/BS62mpfrGTeSz5JKtvo15INPBrGX9G05uS9DoALnoVVM6BolR9Nawkxcf69BhwJx/0NVn3lp73Of9UnfsnZ225Zg7ZNIexziK/muKdSu8K5T8Ls/8Hkn/mpihOubb7PJbIblOBtkT+13tzgWfkFAIzokRG9mERERKRVOHtcL9IT47jmya85657PePS7+9M1PSG6QR1yvU9Kek9o2nqzYCxMutWv3ZvxOHz7oS9zv0VsMmQNhM3LoKIA4lJg0LF+VHDAUXvWm637aH+LhkDAf9bcsX7aqnORrZJo5puZ9zsMNn27+1UpRZqJEjzwc7XXzYehJ2/dNDu/kPiYAAO7tK+qOiIiIrJnjh3elYcvHcf3Hp3G6Xd9yn/O35dR0WylFAj4xGt3jTjD38C3AFi/wP8dtH6Bv+VM8n8T9TscYqOcxDanliqBn5LjbyJRogQP/KJknC/ZGjYrv5Bh3dOIDaraj4iIiHgHDcjiySsO4LJHpnHynZ9w5n65/OzYfchJbaOJUGKGn07Z64BoRyIizUTZC0DeVMD8ED5QG3LMWVXIyNyMqIYlIiIirc/I3Aze/emhfP87/XhxxkoO/+v73P3BEipraqMdmoiIEjzAV9DMHuyLrABL15dQVlXLCDU4FxERkQakJsRy43FDmPLjQzmwfya3vr6Aif/4kClz19DWWlCJSPuiBG9Lg/Oe26ZnzswvBGBUTyV4IiIi0ri+Wcncf/E4HvnueGKDAa7473QuevBLVhaURzs0EemglOBtXOSLrIT73wHMzi8gOS5I3ywVWBEREZFdO3RQNq9fewi/OXEoXy3fzHH/+ogpc9dEOywR6YCU4OWFG5z33JbgzVpZyPAe6QQDLVRtSURERNq82GCASyf05bVrDqFn50Su+O90bnl5rtbmiUiLUoKXH25wnjkQgOraEPNWFTFSDc5FRERkD/TJSua5HxzEpRP68PCnyzjtP5/y7YbSaIclIh2EEry8qb49QrjB+cK1xVTWhBihCpoiIhIhZnasmX1jZovN7IYGnj/MzArNbEb4dnM04pQ9Fx8T5DcnDuO+i8aysqCcE27/iBe/XhntsESkA+jYCV5FoW/oWW/9XbjAikbwREQkAswsCNwJTAKGAuea2dAGdv3IOTc6fPtdiwYpzebooV2YfM0hDO2exnVPz+C6p75m+UaN5olI5HTsBC9/GuB2qKCZlhBDr85J0YtLRETas/HAYufcUudcFfAUcHKUY5II6p6RyJPfO4BrjhjA5NlrOPxv73PNk18zf3VRtEMTkXYoogleE6agnG9ms8K3T81sVCTj2UF+uMF5j7FbN81eWcDI3AzMVGBFREQiogeQV+dxfnjb9g40s5lm9rqZDWvsYGZ2hZlNM7Np69evb+5YpZnEBAP8ZOI+fPSLw7n8kH68M38tk/71EZc9PJXpyzdFOzwRaUciluA1cQrKt8ChzrmRwO+BeyMVT4PyvoScIZCQBkBFdS3frClWgRUREYmkhr5B3L4z9ldAb+fcKODfwIuNHcw5d69zbqxzbmx2dnbzRSkR0SUtgV8eN4RPbjiCnxw9iK9WbOb0uz7j7Hs+44ulG6Mdnoi0A5EcwdvlFBTn3KfOuc3hh58DuRGMp75QyE/RzN02PXPBmmKqa50SPBERiaR8oGedx7nAqro7OOeKnHMl4Z8nA7FmltVyIUqkZSTFcc2RA/nkhiO46YShLN9Yxtn3fs4vX5hNcUV1tMMTkTYskgleU6egbHEZ8HoE46lvw0KoLKzX/252fgGAKmiKiEgkTQUGmllfM4sDzgFerruDmXW18FoBMxuPv15reKcdSoqL4bKD+/Le9YfxvUP68tSXK5j4jw95b8G6aIcmIm1UJBO8pkxB8TuaHY5P8H7RyPPNv74gP9zgvE4Fza/zCshKiad7ekLzvIeIiMh2nHM1wNXAm8B84Bnn3Fwzu9LMrgzvdgYwx8xmArcD5zjnGryGSvuQGBfkV8cP5bkfHERKfAyXPjyVnzw9g82lVdEOTUTamJgIHnuXU1AAzGwkcD8wyTnX4LeTzrl7Ca/PGzt2bPNc4PK+hIQMyBywddOMvAJG91SBFRERiazwtMvJ2227u87PdwB3tHRcEn1jenXi1WsO5s53F/Of95fw4aL1/P7k4Uwa0S3aoYlIGxHJEbymTEHpBTwPXOicWxjBWHaUX7/BeWFZNUvXlzKmV0aLhiEiIiJSV3xMkJ9M3IeXrz6YrukJ/ODxrzj1P5/w1JcrKKmsiXZ4ItLKRSzBa+IUlJuBTOA/ZjbDzKZFKp56ygt8g/M66+9mhtffje6Z0SIhiIiIiOzM0O5pvPjDCdxy4lBKKmq44fnZjPvD21z/v5lMXbYJzdoVkYZEcopmU6agXA5cHskYGrQynEfWqaA5I68AM1RBU0RERFqNmGCASyb05eKD+vB1XgH/m5bHyzNW8ez0fPplJXPm2J6cNTaXzJT4aIcqIq1ERBO8VitvS4Pz/bZumpFXwIDsFFITYqMXl4iIiEgDzIx9e3Vi316duOmEobw2azXPTMvjL28s4B9vL+SkUd255KA+DO+hL6pFOrqOmeDlfwk5Q7c2OHfOMSOvgCMH50Q5MBEREZGdS4qL4cyxPTlzbE8WryvmkU+X89xX+Tw7PZ+xvTtxyYQ+HDOsK7HBSJZaEJHWquMleKEQ5E+H4adu3ZS3qZxNpVWMVoEVERERaUMG5KTy+1OGc/0x+/C/aXk8+tlyrn7ia7qmJXDO+J4cMTiHYd3TCQZUIVyko+h4Cd6Gb3yD83r97zYDKrAiIiIibVN6YiyXH9KPSyf05b0F63jks2X88+1F/PPtRaQnxnJgv0wmDMzi4AFZ9MlMUksokXas4yV4VaU+ueu5/9ZNM/IKSIwNsk+X1CgGJiIiIrJ3ggHjqKFdOGpoF9YVV/DZko18vGgDnyzewBtz1wDQPT2BSSO6ce1RA0lT7QGRdqfjJXi5Y+Hyt+ptmpFXwIge6cRorrqIiIi0EzmpCZw8ugcnj+6Bc45lG8v4ZPEGPlq0noc++ZZXZq7ilpOGMWl4V43oibQjHT6jqaoJMXdVkdbfiYiISLtlZvTNSuaCA3pzz4VjefGqCWSlxPPDx7/i8kemsbKgPNohikgz6fAJ3vzVRVTVhLT+TkRERDqMkbkZvHz1BH513BA+XbKRo2/7gPs/WkpNbSjaoYnIXup4UzS3MyOvAIBRSvBERESkA4kJBvjed/px7PCu3PzSHP7w2nxenLGS08bksmXGpnPb9g8GjO8MyqZvVnJ0AhaRJlGCl1dAdmo83dMToh2KiIiISIvr2TmJBy8Zx2uzV/PbV+bxu1fnNbqvGRw5uAuXH9KX/ft21to9kVZICV5eAaN7Zuh/UCIiItJhmRknjOzOMcO6UlJRU2d7+B6juLKaZ6bl89jnyznn3rUM75HGZQf35fgR3YmL6fCrfkRajQ6d4BWUVfHthlLO2C832qGIiIiIRF1sMECn5LgGn0tPiuUnRw/ih4f154WvV/LAx9/y46dncuvrCzh//94M6ZZG94wEuqcnkpEUqy/PRaKkQyd4W9bfjdH6OxEREZEmSYgNcu74Xpw9ticfLlrPAx9/y21vLay3T2JskG4ZCfTISKRfVjJHDunCAf0yNdIn0gI6fIJnBiNy06MdioiIiEibEggYh+2Tw2H75LChpJKVm8tZVVDOqsIKVhWUs7qwnJUFFTwzLZ9HPltOakIMRwzOYeLQrhy2TzbJ8R36z1CRiOnQ/7Jm5BUwMCeF1ITYaIciIiIi0mZlpcSTlRLfYFXyiupaPlm8gTfnruHt+et4acYq4mICHDwgi4P6Z9I/J4UB2Sl0z0gkGNC0TpG91WETPOccM/MKOHpol2iHIiIiItJuJcQGOXJIF44c0oWa2hDTl2/mzblrmTJvDe8uWLd1v7iYAP2ykumXnUy/rBQykmKJjwkQGwzfYgLEBQPExwYYkJ1CbqdErfMTaUCHTfCWbyxjc1k1o3t2inYoIiIiIh1CTDDA/v0y2b9fJjefOJRNpVUsWV/CknUlLN1QypJ1JcxbVcQbc9YQcjs/VqekWIb3SGdkbjojeqQzIjeD7ukJSvqkw+uwCd6WAiujVWBFREREJCo6J8fRObkz4/p0rre9qiZEeXUt1bUhf6txVIV/Lq2sYcGaYmbnFzJrZSF3f7CU2nA2mJkcx6ieGYzumeHvczNIT9JSHOlYOnSClxgbZFCXlGiHIiIiIiJ1xMUEdlpxc2ydhLCiupb5q4uYs7KQmfmFzMgr4L1v1uHCI4D9spIZ3TODMb07cWC/zvTPTtEon7RrHTbB+zqvgBG56cQEVa5XREREpK1KiA0yplcnxvTqxIXhbUUV1cwOJ3sz8gr4aPEGnv96JeALwhzQrzMH9MvkwP6Z9MtKbjThq64NsbGkirVFFf5WXMn6ogrWFlVSUF5Fr85JDOySyqAuqQzMSVFlUGkVOuRvYWVNLfNXFXHphD7RDkVEREREmllaQiwTBmQxYUAW4IvrrdhUxudLN/LZko18tnQjr85aDUBOajzd0hMor66lvLqWiuoQFVX+55oGFgIGDLJT40lNiOW9b9ZTVRPa+lxup0T26ZLKgC4p9O6cTO/MJHp1TlKFUGlRHTLBm7eqiKrakNbfiYiIiHQAZkbvzGR6ZyZz9rheOOdYtrGMz5Zs5ItvN1JYXk332CCJsUHiw/cJsQESY4N0TomjS2oCXdIS6JIWT2ZK/NZkrTbkE8dv1hSzaG0x36wtZtHaEj5ctJ7q2m3JYWzQyO3kk72+WckMyElhYE4KA3JSyEyJ32nsoZCjtKqG5LgYAkoSpQk6ZIK3tcBKr4yoxiEiIiIiLc/M6JuVTN+sZM7bv9ceHycY2HacY4d33bq9NuRYXVjOio1lLN9UxvKNZazYVMryjWVMXbaJsqrarft2To5jQHYK/XNSCAZgU2lVnVs1m8uqqA054mICdE9PoEenRHpkJNI9w99np8ZTWROirKqGsqpayqtqKa2spay6hvhggOE90hndM4OctIS9OmfSdnTYBK9LWjzd0hOjHYqIiIiItDPBgB+xy+2UxEHbPeecY1VhBYvXlYRvxSxeV8Ibc1ZjZnRKiiUzOZ6+Wcns1zuOzslxpCXEsqm0ivyCclYVlPP+N+tZV1y50xjiYgLU1Ia2tpvompbAyNx0RvXMYFRuBumJsWworWRjSRUbSyrZWFrFhpJKNpVWERMwEuNiSI4LkhgXJDkuhqT4ICnxMXRJS6BHOLnMSIrdYf2ic471JZV8u76Ubzf4m5mxf7/OjO/TeZfrFGtDjoVri5m7qoguafEM755Op+S43f1P0KF12ARP0zNFREREpKWZ2dYE6dBB2Xt8nMqaWtYUVrChpJL4mCDJ8TEkhROypNggMcEA5VW1zF3lq4vOyi9gZl4BU+atbfB4cTEBslPi6ZQci3NQVlVLaWWNHxGsqmmwL2FSXHDrSGJKQgwrNpbx7YZSSipr6h3XOcfdHywhJmCMzE3nwP6ZHNQ/i/16d6KqNsSMFQVMX76Zr1Zs5usVBfVeD9AjI5ERPdIZ3iON4T3SGdQllZBzfr1kdW345n8OBoxemUn07JS000qs25/L8qpaqmpDVNWEqK514fsQtSFHdmo8XdIS2sw6yg6X4G0qrWL5xjLOGbfnw/EiIiIiItEUHxPcuq6wMYlxQcb26VyvrURhWTWzVhZQXlVLVmo8mclxZKbEkxwXbLSaqHOOypoQxRU1rCmsYGVBGSsLKli52Y8oriwoZ/nGUnplJrNf705bp632zUqme0YiVTUhpi/fzGdLN/Dpko3c/cFS7nxvCbFBoybkcM4Xr9mnaxqnjOnOfr07MaJHOmuLKpm9spA54dsbc9c0+fwEAz6R7pOVTN/MJPpkJRMbDLCuqII14UqoW6qjbi6r3uXxYgJGt4wEcjOS6NEpkdxOiWSmxFNSUUNRRTWF5f5WFL6FHGQkxdI5OY5OSeFbciydkuLom5XM8B7pTf4su6vDJXgz1eBcRERERDqo9KRYDhm4eyOHZkZCbJCE2CDZqfGMyN295CQxLsjBA7M4eKCvalpcUc20ZZv5/NuNJMXGsF/vTozqmU5qQv2m9ANyUrdWQgXf/mLuyiKWbighNhjwMcX4+8S4IAkxQapqa1m2oYxlG/300GUbS/lq+eato4JmkJ3iR+RyOyWxX+9OdElLICU+htiYAPHBALExRlwwSGzQCJixttgns/mbfTL70SI/RXZLr8XYoJGeGEtaQixpibGkJ8URMNhcVs2KTWVsKq2iuGLbqOSJo7rz73PH7NY53B0dLsE7aEAmL141gcFdU6MdioiIiIhIh5OaEMvhg3M4fHDObr0uLSGWA/v7/oU7s1/vzvUeb1kXWBtyZKfEN0sf7MqaWgrLqklNiCUhNtDo6OcW1bUhCsqqKSirIjbCfbg7XIIXHxPU6J2IiIiISAdhZuSkNm8V0fiYIDlpwSbvHxsMkJ0aT3bqzttiNIfIpo8iIiKyAzM71sy+MbPFZnZDA8+bmd0efn6Wme0bjThFRKTtUYInIiLSgswsCNwJTAKGAuea2dDtdpsEDAzfrgDuatEgRUSkzVKCJyIi0rLGA4udc0udc1XAU8DJ2+1zMvCo8z4HMsysW0sHKiIibY8SPBERkZbVA8ir8zg/vG139xEREdmBEjwREZGW1VCpte1bCDdlH7+j2RVmNs3Mpq1fv36vgxMRkbZNCZ6IiEjLygd61nmcC6zag30AcM7d65wb65wbm529e72tRESk/VGCJyIi0rKmAgPNrK+ZxQHnAC9vt8/LwEXhapoHAIXOudUtHaiIiLQ9Ha4PnoiISDQ552rM7GrgTSAIPOicm2tmV4afvxuYDBwHLAbKgEujFa+IiLQtSvBERERamHNuMj6Jq7vt7jo/O+Cqlo5LRETaPk3RFBERERERaSfMf0nYdpjZemD5Xh4mC9jQDOG0Fzof9el81KfzUZ/OR32RPh+9nXOqHNJEukZGhM5HfTof2+hc1KfzUV/Uro9tLsFrDmY2zTk3NtpxtBY6H/XpfNSn81Gfzkd9Oh/tj/6b1qfzUZ/OxzY6F/XpfNQXzfOhKZoiIiIiIiLthBI8ERERERGRdqKjJnj3RjuAVkbnoz6dj/p0PurT+ahP56P90X/T+nQ+6tP52Ebnoj6dj/qidj465Bo8ERERERGR9qijjuCJiIiIiIi0Ox0uwTOzY83sGzNbbGY3RDuelmZmD5rZOjObU2dbZzN7y8wWhe87RTPGlmRmPc3sPTObb2Zzzeza8PYOd07MLMHMvjSzmeFz8dvw9g53Luoys6CZfW1mr4Yfd9jzYWbLzGy2mc0ws2nhbR32fLQ3Hf36CLpG1qXrY326Ru5I18f6WtM1skMleGYWBO4EJgFDgXPNbGh0o2pxDwPHbrftBuAd59xA4J3w446iBvipc24IcABwVfh3oiOek0rgCOfcKGA0cKyZHUDHPBd1XQvMr/O4o5+Pw51zo+uUfu7o56Nd0PVxq4fRNXILXR/r0zVyR7o+7qhVXCM7VIIHjAcWO+eWOueqgKeAk6McU4tyzn0IbNpu88nAI+GfHwFOacmYosk5t9o591X452L8/6h60AHPifNKwg9jwzdHBzwXW5hZLnA8cH+dzR32fDRC56N96PDXR9A1si5dH+vTNbI+XR+bLCrnpKMleD2AvDqP88PbOrouzrnV4P+HDuREOZ6oMLM+wBjgCzroOQlPt5gBrAPecs512HMR9k/g50CozraOfD4cMMXMppvZFeFtHfl8tCe6Pjauw/+O6/ro6RpZzz/R9XF7reYaGdMSb9KKWAPbVEZUMLMU4DngOudckVlDvyrtn3OuFhhtZhnAC2Y2PMohRY2ZnQCsc85NN7PDohxOazHBObfKzHKAt8xsQbQDkmaj66M0SNfHbXSN9HR9bFSruUZ2tBG8fKBnnce5wKooxdKarDWzbgDh+3VRjqdFmVks/uL1uHPu+fDmDn1OnHMFwPv4tSgd9VxMAE4ys2X46WpHmNljdNzzgXNuVfh+HfACflpfhz0f7Yyuj43rsL/juj42TNdIXR8b0pqukR0twZsKDDSzvmYWB5wDvBzlmFqDl4GLwz9fDLwUxVhalPmvIh8A5jvnbqvzVIc7J2aWHf5WEjNLBI4CFtABzwWAc+5G51yuc64P/v8V7zrnLqCDng8zSzaz1C0/AxOBOXTQ89EO6frYuA75O67rY326Rm6j6+OOWts1ssM1Ojez4/DzhoPAg865P0Y3opZlZk8ChwFZwFrgN8CLwDNAL2AFcKZzbvtF5u2SmR0MfATMZts88l/i1xl0qHNiZiPxC4CD+C9/nnHO/c7MMulg52J74Sko1zvnTuio58PM+uG/kQQ/vf8J59wfO+r5aI86+vURdI2sS9fH+nSNbJiuj15ru0Z2uARPRERERESkvepoUzRFRERERETaLSV4IiIiIiIi7YQSPBERERERkXZCCZ6IiIiIiEg7oQRPRERERESknVCCJ9JOmNlhZvZqtOMQERFpTXR9lI5GCZ6IiIiIiEg7oQRPpIWZ2QVm9qWZzTCze8wsaGYlZvZ3M/vKzN4xs+zwvqPN7HMzm2VmL5hZp/D2AWb2tpnNDL+mf/jwKWb2rJktMLPHzcyi9kFFRER2g66PIs1DCZ5ICzKzIcDZwATn3GigFjgfSAa+cs7tC3wA/Cb8kkeBXzjnRgKz62x/HLjTOTcKOAhYHd4+BrgOGAr0AyZE+COJiIjsNV0fRZpPTLQDEOlgjgT2A6aGvzxMBNYBIeDp8D6PAc+bWTqQ4Zz7ILz9EeB/ZpYK9HDOvQDgnKsACB/vS+dcfvjxDKAP8HHEP5WIiMje0fVRpJkowRNpWQY84py7sd5Gs5u228/t4hiNqazzcy36Ny4iIm2Dro8izURTNEVa1jvAGWaWA2Bmnc2sN/7f4hnhfc4DPnbOFQKbzeyQ8PYLgQ+cc0VAvpmdEj5GvJklteSHEBERaWa6Poo0E317IdKCnHPzzOzXwBQzCwDVwFVAKTDMzKYDhfh1CAAXA3eHL1BLgUvD2y8E7jGz34WPcWYLfgwREZFmpeujSPMx53Y20i0iLcHMSpxzKdGOQ0REpDXR9VFk92mKpoiIiIiISDuhETwREREREZF2QiN4IiIiIiIi7YQSPBERERERkXZCCZ6IiIiIiEg7oQRPRERERESknVCCJyIiIiIi0k4owRMREREREWkn/h91BZ5S44FeEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics as mt\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# combine all the history from training together\n",
    "combined = dict()\n",
    "for key in ['acc','val_acc','loss','val_loss']:\n",
    "    combined[key] = np.hstack([x.history[key] for x in histories])\n",
    "    \n",
    "# summarize history for accuracy\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.plot(combined['acc'])\n",
    "plt.plot(combined['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "\n",
    "# summarize history for loss\n",
    "plt.subplot(122)\n",
    "plt.plot(combined['loss'])\n",
    "plt.plot(combined['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay. So we seem to be maxing out the performance on the validation set. The results are pretty good--I would expect these to be about a top 10 submission on a class-based Kaggle competition. Nice!\n",
    "- https://inclass.kaggle.com/c/cs5740-20-newsgroups-classification/leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[End of Lecture Demo]**\n",
    "\n",
    "Below is an archive of discussions from previous semesters. While it is interesting, it is not critical to the understanding of recurrent neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# =============[ARCHIVED]=============\n",
    "# But others have done much better on 20 News groups! \n",
    "\n",
    "Why are we not getting the 95% on the data? Let's take a look at the original posting by the Keras blog. They claim that they get validation accuracy of 95% after two epochs!\n",
    "\n",
    "Why are we not getting that same accuracy?  \n",
    "- https://github.com/kimardenmiller/NLP_CNN/blob/master/Embeddings/0.0.4%20word%20index%20save.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Indexing word vectors.\n",
      "Processing text data set\n",
      "Found 19997 texts.\n",
      "Xref: cantaloupe.srv.cs.cmu.edu alt.atheism:49960 alt.atheism.moderated:713 news.answers:7054 alt.answers:126\n",
      "Path: cantaloupe.srv.cs.cmu.edu!crabapple.srv.cs.cmu.edu!bb3.andrew.cmu.edu!news.sei.cmu.edu!cis.ohio-state.edu!magnus.acs.ohio-state.edu!usenet.ins.cwru.edu!agate!spool.mu.edu!uunet!pipex!ibmpcug!mantis!mathew\n",
      "From: mathew <mathew@mantis.co.uk>\n",
      "Newsgroups: alt.atheism,alt.atheism.moderated,news.answers,alt.answers\n",
      "Subject: Alt.Atheism FAQ: Atheist Resources\n",
      "Summary: Books, addresses, music -- anything related to atheism\n",
      "Keywords: FAQ, atheism, books, music, fiction, addresses, contacts\n",
      "Message-ID: <19930329115719@mantis.co.uk>\n",
      "Date: Mon, 29 Mar 1993 11:57:19 GMT\n",
      "Expires: Thu, 29 Apr 1993 11:57:19 GMT\n",
      "Followup-To: alt.atheism\n",
      "Distribution: world\n",
      "Organization: Mantis Consultants, Cambridge. UK.\n",
      "Approved: news-answers-request@mit.edu\n",
      "Supersedes: <19930301143317@mantis.co.uk>\n",
      "Lines: 290\n",
      "\n",
      "Archive-name: atheism/resources\n",
      "Alt-atheism-archive-name: resources\n",
      "Last-modified: 11 December 1992\n",
      "Version: 1.0\n",
      "\n",
      "                              Atheist Resources\n",
      "\n",
      "                      Addresses of Atheist Organizations\n",
      "\n",
      "                                     USA\n",
      "\n",
      "FREEDOM FROM RELIGION FOUNDATION\n",
      "\n",
      "Darwin fish bumper stickers and assorted other atheist paraphernalia are\n",
      "available from the Freedom From Religion Foundation in the US.\n",
      "\n",
      "Write to:  FFRF, P.O. Box 750, Madison, WI 53701.\n",
      "Telephone: (608) 256-8900\n",
      "\n",
      "EVOLUTION DESIGNS\n",
      "\n",
      "Evolution Designs sell the \"Darwin fish\".  It's a fish symbol, like the ones\n",
      "Christians stick on their cars, but with feet and the word \"Darwin\" written\n",
      "inside.  The deluxe moulded 3D plastic fish is $4.95 postpaid in the US.\n",
      "\n",
      "Write to:  Evolution Designs, 7119 Laurel Canyon #4, North Hollywood,\n",
      "           CA 91605.\n",
      "\n",
      "People in the San Francisco Bay area can get Darwin Fish from Lynn Gold --\n",
      "try mailing <figmo@netcom.com>.  For net people who go to Lynn directly, the\n",
      "price is $4.95 per fish.\n",
      "\n",
      "AMERICAN ATHEIST PRESS\n",
      "\n",
      "AAP publish various atheist books -- critiques of the Bible, lists of\n",
      "Biblical contradictions, and so on.  One such book is:\n",
      "\n",
      "\"The Bible Handbook\" by W.P. Ball and G.W. Foote.  American Atheist Press.\n",
      "372 pp.  ISBN 0-910309-26-4, 2nd edition, 1986.  Bible contradictions,\n",
      "absurdities, atrocities, immoralities... contains Ball, Foote: \"The Bible\n",
      "Contradicts Itself\", AAP.  Based on the King James version of the Bible.\n",
      "\n",
      "Write to:  American Atheist Press, P.O. Box 140195, Austin, TX 78714-0195.\n",
      "      or:  7215 Cameron Road, Austin, TX 78752-2973.\n",
      "Telephone: (512) 458-1244\n",
      "Fax:       (512) 467-9525\n",
      "\n",
      "PROMETHEUS BOOKS\n",
      "\n",
      "Sell books including Haught's \"Holy Horrors\" (see below).\n",
      "\n",
      "Write to:  700 East Amherst Street, Buffalo, New York 14215.\n",
      "Telephone: (716) 837-2475.\n",
      "\n",
      "An alternate address (which may be newer or older) is:\n",
      "Prometheus Books, 59 Glenn Drive, Buffalo, NY 14228-2197.\n",
      "\n",
      "AFRICAN-AMERICANS FOR HUMANISM\n",
      "\n",
      "An organization promoting black secular humanism and uncovering the history of\n",
      "black freethought.  They publish a quarterly newsletter, AAH EXAMINER.\n",
      "\n",
      "Write to:  Norm R. Allen, Jr., African Americans for Humanism, P.O. Box 664,\n",
      "           Buffalo, NY 14226.\n",
      "\n",
      "                                United Kingdom\n",
      "\n",
      "Rationalist Press Association          National Secular Society\n",
      "88 Islington High Street               702 Holloway Road\n",
      "London N1 8EW                          London N19 3NL\n",
      "071 226 7251                           071 272 1266\n",
      "\n",
      "British Humanist Association           South Place Ethical Society\n",
      "14 Lamb's Conduit Passage              Conway Hall\n",
      "London WC1R 4RH                        Red Lion Square\n",
      "071 430 0908                           London WC1R 4RL\n",
      "fax 071 430 1271                       071 831 7723\n",
      "\n",
      "The National Secular Society publish \"The Freethinker\", a monthly magazine\n",
      "founded in 1881.\n",
      "\n",
      "                                   Germany\n",
      "\n",
      "IBKA e.V.\n",
      "Internationaler Bund der Konfessionslosen und Atheisten\n",
      "Postfach 880, D-1000 Berlin 41. Germany.\n",
      "\n",
      "IBKA publish a journal:\n",
      "MIZ. (Materialien und Informationen zur Zeit. Politisches\n",
      "Journal der Konfessionslosesn und Atheisten. Hrsg. IBKA e.V.)\n",
      "MIZ-Vertrieb, Postfach 880, D-1000 Berlin 41. Germany.\n",
      "\n",
      "For atheist books, write to:\n",
      "\n",
      "IBDK, Internationaler B\"ucherdienst der Konfessionslosen\n",
      "Postfach 3005, D-3000 Hannover 1. Germany.\n",
      "Telephone: 0511/211216\n",
      "\n",
      "\n",
      "                               Books -- Fiction\n",
      "\n",
      "THOMAS M. DISCH\n",
      "\n",
      "\"The Santa Claus Compromise\"\n",
      "Short story.  The ultimate proof that Santa exists.  All characters and \n",
      "events are fictitious.  Any similarity to living or dead gods -- uh, well...\n",
      "\n",
      "WALTER M. MILLER, JR\n",
      "\n",
      "\"A Canticle for Leibowitz\"\n",
      "One gem in this post atomic doomsday novel is the monks who spent their lives\n",
      "copying blueprints from \"Saint Leibowitz\", filling the sheets of paper with\n",
      "ink and leaving white lines and letters.\n",
      "\n",
      "EDGAR PANGBORN\n",
      "\n",
      "\"Davy\"\n",
      "Post atomic doomsday novel set in clerical states.  The church, for example,\n",
      "forbids that anyone \"produce, describe or use any substance containing...\n",
      "atoms\". \n",
      "\n",
      "PHILIP K. DICK\n",
      "\n",
      "Philip K. Dick Dick wrote many philosophical and thought-provoking short \n",
      "stories and novels.  His stories are bizarre at times, but very approachable.\n",
      "He wrote mainly SF, but he wrote about people, truth and religion rather than\n",
      "technology.  Although he often believed that he had met some sort of God, he\n",
      "remained sceptical.  Amongst his novels, the following are of some relevance:\n",
      "\n",
      "\"Galactic Pot-Healer\"\n",
      "A fallible alien deity summons a group of Earth craftsmen and women to a\n",
      "remote planet to raise a giant cathedral from beneath the oceans.  When the\n",
      "deity begins to demand faith from the earthers, pot-healer Joe Fernwright is\n",
      "unable to comply.  A polished, ironic and amusing novel.\n",
      "\n",
      "\"A Maze of Death\"\n",
      "Noteworthy for its description of a technology-based religion.\n",
      "\n",
      "\"VALIS\"\n",
      "The schizophrenic hero searches for the hidden mysteries of Gnostic\n",
      "Christianity after reality is fired into his brain by a pink laser beam of\n",
      "unknown but possibly divine origin.  He is accompanied by his dogmatic and\n",
      "dismissively atheist friend and assorted other odd characters.\n",
      "\n",
      "\"The Divine Invasion\"\n",
      "God invades Earth by making a young woman pregnant as she returns from\n",
      "another star system.  Unfortunately she is terminally ill, and must be\n",
      "assisted by a dead man whose brain is wired to 24-hour easy listening music.\n",
      "\n",
      "MARGARET ATWOOD\n",
      "\n",
      "\"The Handmaid's Tale\"\n",
      "A story based on the premise that the US Congress is mysteriously\n",
      "assassinated, and fundamentalists quickly take charge of the nation to set it\n",
      "\"right\" again.  The book is the diary of a woman's life as she tries to live\n",
      "under the new Christian theocracy.  Women's right to own property is revoked,\n",
      "and their bank accounts are closed; sinful luxuries are outlawed, and the\n",
      "radio is only used for readings from the Bible.  Crimes are punished\n",
      "retroactively: doctors who performed legal abortions in the \"old world\" are\n",
      "hunted down and hanged.  Atwood's writing style is difficult to get used to\n",
      "at first, but the tale grows more and more chilling as it goes on.\n",
      "\n",
      "VARIOUS AUTHORS\n",
      "\n",
      "\"The Bible\"\n",
      "This somewhat dull and rambling work has often been criticized.  However, it\n",
      "is probably worth reading, if only so that you'll know what all the fuss is\n",
      "about.  It exists in many different versions, so make sure you get the one\n",
      "true version.\n",
      "\n",
      "                             Books -- Non-fiction\n",
      "\n",
      "PETER DE ROSA\n",
      "\n",
      "\"Vicars of Christ\", Bantam Press, 1988\n",
      "Although de Rosa seems to be Christian or even Catholic this is a very\n",
      "enlighting history of papal immoralities, adulteries, fallacies etc.\n",
      "(German translation: \"Gottes erste Diener. Die dunkle Seite des Papsttums\",\n",
      "Droemer-Knaur, 1989)\n",
      "\n",
      "MICHAEL MARTIN\n",
      "\n",
      "\"Atheism: A Philosophical Justification\", Temple University Press,\n",
      " Philadelphia, USA.\n",
      "A detailed and scholarly justification of atheism.  Contains an outstanding\n",
      "appendix defining terminology and usage in this (necessarily) tendentious\n",
      "area.  Argues both for \"negative atheism\" (i.e. the \"non-belief in the\n",
      "existence of god(s)\") and also for \"positive atheism\" (\"the belief in the\n",
      "non-existence of god(s)\").  Includes great refutations of the most\n",
      "challenging arguments for god; particular attention is paid to refuting\n",
      "contempory theists such as Platinga and Swinburne.\n",
      "541 pages. ISBN 0-87722-642-3 (hardcover; paperback also available)\n",
      "\n",
      "\"The Case Against Christianity\", Temple University Press\n",
      "A comprehensive critique of Christianity, in which he considers\n",
      "the best contemporary defences of Christianity and (ultimately)\n",
      "demonstrates that they are unsupportable and/or incoherent.\n",
      "273 pages. ISBN 0-87722-767-5\n",
      "\n",
      "JAMES TURNER\n",
      "\n",
      "\"Without God, Without Creed\", The Johns Hopkins University Press, Baltimore,\n",
      " MD, USA\n",
      "Subtitled \"The Origins of Unbelief in America\".  Examines the way in which\n",
      "unbelief (whether agnostic or atheistic)  became a mainstream alternative\n",
      "world-view.  Focusses on the period 1770-1900, and while considering France\n",
      "and Britain the emphasis is on American, and particularly New England\n",
      "developments.  \"Neither a religious history of secularization or atheism,\n",
      "Without God, Without Creed is, rather, the intellectual history of the fate\n",
      "of a single idea, the belief that God exists.\" \n",
      "316 pages. ISBN (hardcover) 0-8018-2494-X (paper) 0-8018-3407-4\n",
      "\n",
      "GEORGE SELDES (Editor)\n",
      "\n",
      "\"The great thoughts\", Ballantine Books, New York, USA\n",
      "A \"dictionary of quotations\" of a different kind, concentrating on statements\n",
      "and writings which, explicitly or implicitly, present the person's philosophy\n",
      "and world-view.  Includes obscure (and often suppressed) opinions from many\n",
      "people.  For some popular observations, traces the way in which various\n",
      "people expressed and twisted the idea over the centuries.  Quite a number of\n",
      "the quotations are derived from Cardiff's \"What Great Men Think of Religion\"\n",
      "and Noyes' \"Views of Religion\".\n",
      "490 pages. ISBN (paper) 0-345-29887-X.\n",
      "\n",
      "RICHARD SWINBURNE\n",
      "\n",
      "\"The Existence of God (Revised Edition)\", Clarendon Paperbacks, Oxford\n",
      "This book is the second volume in a trilogy that began with \"The Coherence of\n",
      "Theism\" (1977) and was concluded with \"Faith and Reason\" (1981).  In this\n",
      "work, Swinburne attempts to construct a series of inductive arguments for the\n",
      "existence of God.  His arguments, which are somewhat tendentious and rely\n",
      "upon the imputation of late 20th century western Christian values and\n",
      "aesthetics to a God which is supposedly as simple as can be conceived, were\n",
      "decisively rejected in Mackie's \"The Miracle of Theism\".  In the revised\n",
      "edition of \"The Existence of God\", Swinburne includes an Appendix in which he\n",
      "makes a somewhat incoherent attempt to rebut Mackie.\n",
      "\n",
      "J. L. MACKIE\n",
      "\n",
      "\"The Miracle of Theism\", Oxford\n",
      "This (posthumous) volume contains a comprehensive review of the principal\n",
      "arguments for and against the existence of God.  It ranges from the classical\n",
      "philosophical positions of Descartes, Anselm, Berkeley, Hume et al, through\n",
      "the moral arguments of Newman, Kant and Sidgwick, to the recent restatements\n",
      "of the classical theses by Plantinga and Swinburne.  It also addresses those\n",
      "positions which push the concept of God beyond the realm of the rational,\n",
      "such as those of Kierkegaard, Kung and Philips, as well as \"replacements for\n",
      "God\" such as Lelie's axiarchism.  The book is a delight to read - less\n",
      "formalistic and better written than Martin's works, and refreshingly direct\n",
      "when compared with the hand-waving of Swinburne.\n",
      "\n",
      "JAMES A. HAUGHT\n",
      "\n",
      "\"Holy Horrors: An Illustrated History of Religious Murder and Madness\",\n",
      " Prometheus Books\n",
      "Looks at religious persecution from ancient times to the present day -- and\n",
      "not only by Christians.\n",
      "Library of Congress Catalog Card Number 89-64079. 1990.\n",
      "\n",
      "NORM R. ALLEN, JR.\n",
      "\n",
      "\"African American Humanism: an Anthology\"\n",
      "See the listing for African Americans for Humanism above.\n",
      "\n",
      "GORDON STEIN\n",
      "\n",
      "\"An Anthology of Atheism and Rationalism\", Prometheus Books\n",
      "An anthology covering a wide range of subjects, including 'The Devil, Evil\n",
      "and Morality' and 'The History of Freethought'.  Comprehensive bibliography.\n",
      "\n",
      "EDMUND D. COHEN\n",
      "\n",
      "\"The Mind of The Bible-Believer\", Prometheus Books\n",
      "A study of why people become Christian fundamentalists, and what effect it\n",
      "has on them.\n",
      "\n",
      "                                Net Resources\n",
      "\n",
      "There's a small mail-based archive server at mantis.co.uk which carries\n",
      "archives of old alt.atheism.moderated articles and assorted other files.  For\n",
      "more information, send mail to archive-server@mantis.co.uk saying\n",
      "\n",
      "   help\n",
      "   send atheism/index\n",
      "\n",
      "and it will mail back a reply.\n",
      "\n",
      "\n",
      "mathew\n",
      "ÿ\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/keras/preprocessing/text.py:89: UserWarning: The `nb_words` argument in `Tokenizer` has been renamed `num_words`.\n",
      "  warnings.warn('The `nb_words` argument in `Tokenizer` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 214873 unique tokens.\n",
      "Shape of data tensor: (19997, 1000)\n",
      "Sample data tensor: [  447   119   489   755    76   696  2362    10    76   118  3007    83\n",
      "   814     4   216    76  7208 16127  7084   126     1   516    25     4\n",
      "    83  9295 12513  7994     5  3773  7097     5   393     4   719     6\n",
      "   933     3     5  1891  2154     3  2977     5  7449    18 13741     1\n",
      "    97     1  7097  4952     3  2842   807    18     1  7994  1249     9\n",
      "  3477     3 15061     5 10707     6  7469  9926     5     4   760    14\n",
      "   234  2234     4     5   489   480   188     1 11088  8086    14     1\n",
      "  4304 11974     4  1131   226  1483     9  3096   186   126  2129    47\n",
      "     5  6103  2438  5693     4  3144    35  1768  3639  3058    76     9\n",
      " 13222    47   126 18603     6  2056  1226     6 15022   108  3903  3798\n",
      "     1  3639  5090   216   719    47   742     5  1044  2066  8364    28\n",
      "   423  5236    18   325  2062   159  1509   423     9  5744     6   314\n",
      "    23 11655    47     5   859   451  1472  2129     9  9934     3   280\n",
      "  2771   971  4159  1684     1  7192     5  1143   480    19     1  5352\n",
      "    10     1   155  1502     9     6 10814  2431   295  1956     4     1\n",
      "  2183     3   391    13   206   374     1   539     9     1     4     5\n",
      " 13919   399    28   423  5215     3   684   341     1   148   202  7477\n",
      "   206     3   309  2151     9     6   102  2231  3584    25  1824  7226\n",
      "    25 16844     6     1   841     9   115   244    14 12155    18     1\n",
      "   659  3260    25  6937  2924    84  3860   674  5574     7     1   438\n",
      "   193    25   340     6 13978  1702  2337     9  1477     3   116   244\n",
      "     3    46   203    35     1  7192 11180    94     6    94 19607    28\n",
      "    13   919    19  1104  3919     1   659    20  2529 10607     6   256\n",
      "    75   696   125 19180   369    13     9   388  1085   914    34   115\n",
      "    71    10  1348   110    54    56     1 19935     9    67    13  1285\n",
      "     7   200   381  2145    71   198   322    17   116     1    61   383\n",
      "   350   882   454  4544   855   218 11773     4   816   844  2039   755\n",
      "   218 11773   385     3    23   202    31   176  2141    20     9     5\n",
      "   183   590     4 16139 19907   318  2086  2790  1286  1337  2778   534\n",
      "  2257   357     5  6192  4278  1931    99   844  3013   284     5  3943\n",
      "     6  9497  4278     4   357  1694    50 12208  8157  9676     6  3157\n",
      "     7    20  1344   726 13771   307    14  3848   357     8   151     1\n",
      "   454  1299     7     1  1296     4   216   129     6   132    14  2343\n",
      "   357     1  1299     7     1   454  1296     4   216   129  1139   408\n",
      "     4     1   211 15517  1822    14   216   967  2279     9  1902     3\n",
      "  5150   213    28     6  1887  5183    65 17667    68 18169 11806   132\n",
      "   348     1   331   371  1131  1931    99   844     5  4682 14620     4\n",
      "  1131     7    90    76  7976     1   380  9404     4  1131     6  6220\n",
      "  8758    10    37    25     6    31 10491  1887  5183    65   104   801\n",
      "  5965   319   216   319  8195     1  6287  4172    99   844  3124  1677\n",
      "   284     1  1321     4     7  1388     1   189     7    90   538  9931\n",
      "    31 17778  2273     5  6723  2121   193   775    19     1   974 10801\n",
      "     6   310  1932  3080     6  5356     1  5291     9    19   511     6\n",
      "  1848   148  2915 12609  1343     5   831   590     4    31   357   319\n",
      "   216   319  8195     9   447     1  5275   590     4     1  6252     4\n",
      "     5   772   554     1  1299    10   216  1285  9740  1887  5183 18169\n",
      "    65    59  1390    65   103  1174  2503     1   408  2454   882   148\n",
      "   892   284     5  3400     4 11874     4     5   381   566    19  1885\n",
      "     6  5003    90  4780    31 13659  1138     1  8367  2766     6   193\n",
      "   775  1139  7971     6   696 12994   547    18   200    96    14    83\n",
      "  2003  2967 11069     1   189     7    90  1104    96  1779     6  9528\n",
      "     1   554   245     1  4812   512     5   327     4     1 11874    25\n",
      "  4978    18    54   408   634   120     4   188     6  1377     4   188\n",
      "  1887  5183  1390    65 10650    59  1116     1  1296     4   216  6428\n",
      "  3061  5409    20   539     9     1   437  1595     7     5    10  2528\n",
      "    27     1     4  4076  5818     6    33  7141    27   807     6   456\n",
      "  5982     7    20   256  3174     3  9614     5  1003     4 14806  1822\n",
      "    14     1  1296     4   216   126  1822    90    25  2529     6  4609\n",
      "   928     1     4  1556  8045  2113   922   202  1084     6     3     5\n",
      "   216    90     9  3432    28   854    28    51    23 11868   100  5120\n",
      "     7     1  5333     4  4076     7     1  6428  3061     4     1  1296\n",
      "     4   216  1139    50     7    90    76   588     5  2529  1562     3\n",
      "  9756   264   253  9756     1  5333     4  4076  5409    20  1595  1694\n",
      "     5  4682  2000     4     1  8646  1822    14     6   371     1  1296\n",
      "     4   216    13 10051    18     1  9029  6192  3466     4  6840   561\n",
      "  1882   990   333     1   890  1822     4  9245     6     3     1  1530\n",
      "     4     1  9029    47     6    13   132  3003   190  3466    90  3170\n",
      "     1  2067     4   216  1738     1  9654     4     1  3407   213    28\n",
      "   190     4     6  3337    28   165    28 12997    14   216   213    28\n",
      "     1   539     9     5     3   308   533     6   298   923   119   666\n",
      "     6  1886    97  2106    27     1   695 10643     4   801     5  1974\n",
      " 13328    50 15275   590     4   831  1583     6 13009   882  1177    46\n",
      "   831  5795    18  2718   556     3     1  1138   356     6    22   115\n",
      "    47   751  1393     4  1502  4781   450   327  2776  1799  8414   141\n",
      "  2159  2450  3281   511    50   179     1  5170    14  3281  1565    14\n",
      "   470  1804  4178    50     4   357     6   882    50  6331     5  1792\n",
      "  1649     4  5741   715  5857  5190  1542     6     6  5857   590     4\n",
      "  4682  8980   170  7833     1   700     4     1   659  6860   882     5\n",
      "   979     4   181    96   916   202 10814     6    54  1008    13    75\n",
      "    19   117    32  1700   658     5   629   227   480  1598   612    46\n",
      "  1327   358   194    90  6668  3702     4   438   127   357  9193  1823\n",
      "     6 15022   108   494    14    94   290   518   227     3  1598   612\n",
      "  1327   358   194   597   292   518   357  3297     6    13    62   227\n",
      "   277     5   240  1899]\n",
      "Shape of label tensor: (19997, 20)\n",
      "Sample label tensor: [[ 1.  0.  0. ...,  0.  0.  0.]\n",
      " [ 1.  0.  0. ...,  0.  0.  0.]\n",
      " [ 1.  0.  0. ...,  0.  0.  0.]\n",
      " ..., \n",
      " [ 1.  0.  0. ...,  0.  0.  0.]\n",
      " [ 1.  0.  0. ...,  0.  0.  0.]\n",
      " [ 1.  0.  0. ...,  0.  0.  0.]]\n",
      "Preparing embedding matrix.\n",
      "Training model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/ipykernel/__main__.py:148: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15998 samples, validate on 3999 samples\n",
      "Epoch 1/2\n",
      "15998/15998 [==============================] - 281s - loss: 1.9110 - acc: 0.3264 - val_loss: 0.9826 - val_acc: 0.638\n",
      "Epoch 2/2\n",
      "15998/15998 [==============================] - 293s - loss: 0.7430 - acc: 0.7523 - val_loss: 1.3303 - val_acc: 0.56\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x136316080>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from __future__ import print_function\n",
    "import os\n",
    "import numpy as np\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from keras.layers import Dense, Input, Flatten\n",
    "from keras.layers import Conv1D, MaxPooling1D, Embedding\n",
    "from keras.models import Model\n",
    "import sys\n",
    "import pandas as pd\n",
    "import pickle\n",
    "from keras.models import load_model\n",
    "\n",
    "np.random.seed(1337)\n",
    "\n",
    "BASE_DIR = ''\n",
    "GLOVE_DIR = 'large_data/glove'\n",
    "TEXT_DATA_DIR = 'large_data/20news-bydate/20news-bydate-train/'\n",
    "TEXT_DATA_DIR = 'large_data/20_newsgroups/'\n",
    "MAX_SEQUENCE_LENGTH = 1000\n",
    "MAX_NB_WORDS = 20000\n",
    "EMBEDDING_DIM = 100\n",
    "VALIDATION_SPLIT = 0.2\n",
    "\n",
    "# first, build index mapping words in the embeddings set to their embedding vector\n",
    "print('Indexing word vectors.')\n",
    "\n",
    "embeddings_index = {}\n",
    "\n",
    "f = open('large_data/glove/glove.6B.100d.txt')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "# second, prepare text samples and their labels\n",
    "print('Processing text data set')\n",
    "\n",
    "texts = []  # list of text samples\n",
    "labels_index = {}  # dictionary mapping label news_group to numeric id\n",
    "labels = []  # list of label ids\n",
    "for news_group in sorted(os.listdir(TEXT_DATA_DIR)):\n",
    "    path = os.path.join(TEXT_DATA_DIR, news_group)\n",
    "    if os.path.isdir(path):\n",
    "        label_id = len(labels_index)\n",
    "        labels_index[news_group] = label_id\n",
    "        for post in sorted(os.listdir(path)):\n",
    "            if post.isdigit():\n",
    "                post_path = os.path.join(path, post)\n",
    "                if sys.version_info < (3,):\n",
    "                    f = open(post_path)\n",
    "                else:\n",
    "                    f = open(post_path, encoding='latin-1')\n",
    "                texts.append(f.read())\n",
    "                f.close()\n",
    "                labels.append(label_id)\n",
    "\n",
    "print('Found %s texts.' % len(texts))\n",
    "print(texts[0])\n",
    "\n",
    "# override with sklearn\n",
    "# texts = bunch.data\n",
    "# labels = bunch.target\n",
    "# labels_index = bunch.target_names\n",
    "\n",
    "# finally, vectorize the text samples into a 2D integer tensor\n",
    "tokenizer = Tokenizer(nb_words=MAX_NB_WORDS)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "pickle.dump(tokenizer, open('large_data/tokenizer.004.p', 'wb'))\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))\n",
    "# print('Word index', word_index.values())\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Sample data tensor:', data[0, 0:])\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "print('Sample label tensor:', labels[690:750, 0:])\n",
    "# sys.exit()\n",
    "\n",
    "# split the data into a training set and a validation set\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "nb_validation_samples = int(VALIDATION_SPLIT * data.shape[0])\n",
    "\n",
    "x_train = data[:-nb_validation_samples]\n",
    "y_train = labels[:-nb_validation_samples]\n",
    "x_val = data[-nb_validation_samples:]\n",
    "y_val = labels[-nb_validation_samples:]\n",
    "\n",
    "print('Preparing embedding matrix.')\n",
    "\n",
    "# prepare embedding matrix\n",
    "nb_words = min(MAX_NB_WORDS, len(word_index))\n",
    "embedding_matrix = np.zeros((nb_words + 1, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    if i > MAX_NB_WORDS:\n",
    "        continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "\n",
    "# load pre-trained word embeddings into an Embedding layer\n",
    "# trainable = False so as to keep the embeddings fixed\n",
    "embedding_layer = Embedding(nb_words + 1,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=False)\n",
    "\n",
    "print('Training model.')\n",
    "\n",
    "# train a 1D convnet with global maxpooling\n",
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "x = Conv1D(128, 5, activation='relu')(embedded_sequences)\n",
    "x = MaxPooling1D(5)(x)\n",
    "x = Conv1D(128, 5, activation='relu')(x)\n",
    "x = MaxPooling1D(5)(x)\n",
    "x = Conv1D(128, 5, activation='relu')(x)\n",
    "x = MaxPooling1D(35)(x)\n",
    "x = Flatten()(x)\n",
    "x = Dense(128, activation='relu')(x)\n",
    "preds = Dense(len(labels_index), activation='softmax')(x)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=2, batch_size=128)\n",
    "\n",
    "#  Dec 30, 2016  loss: 0.3069 - acc: 0.8908 - val_loss: 0.1421 - val_acc: 0.9549\n",
    "\n",
    "# # First evaluation of the model\n",
    "# scores = model.evaluate(x_val, y_val, verbose=0)\n",
    "# print(\"Accuracy of Run Model: %.2f%%\" % (scores[1]*100))\n",
    "\n",
    "# model.save('../saved_models/0.0.4_model.h5')\n",
    "# np.save('../saved_models/0.0.4_word_index.npy', word_index)\n",
    "\n",
    "# del model  # deletes the existing model\n",
    "\n",
    "# # returns a compiled model identical to the previous one\n",
    "# model = load_model('../saved_models/0.0.4_model.h5')\n",
    "\n",
    "# # Final evaluation of the model\n",
    "# scores = model.evaluate(x_val, y_val, verbose=0)\n",
    "# print(\"Accuracy of Disk-Loaded Model: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/eclarson/anaconda/envs/MLEnv/lib/python3.5/site-packages/ipykernel/__main__.py:2: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15998 samples, validate on 3999 samples\n",
      "Epoch 1/2\n",
      "15998/15998 [==============================] - 255s - loss: 0.3346 - acc: 0.9032 - val_loss: 0.2218 - val_acc: 0.93\n",
      "Epoch 2/2\n",
      "15998/15998 [==============================] - 244s - loss: 0.2170 - acc: 0.9320 - val_loss: 0.2049 - val_acc: 0.94\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x12a96f4e0>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, validation_data=(x_val, y_val),\n",
    "          nb_epoch=2, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}