lstm_tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training an LSTM network on the Penn Tree Bank (PTB) dataset - Part III: TensorFlow graph execution \n",
    "---\n",
    "\n",
    "[1]: http://people.idsia.ch/~juergen/\n",
    "[2]: http://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
    "[3]: https://www.coursera.org/specializations/deep-learning\n",
    "[4]: http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf\n",
    "[5]: https://arxiv.org/abs/1503.04069\n",
    "[6]: http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz\n",
    "[7]: https://arxiv.org/abs/1409.2329\n",
    "[8]: https://www.python.org/\n",
    "[9]: http://www.numpy.org/\n",
    "[10]: https://www.tensorflow.org/\n",
    "[11]: https://www.tensorflow.org/guide/eager\n",
    "[12]: https://colab.research.google.com/notebooks/welcome.ipynb\n",
    "[13]: https://www.tensorflow.org/guide/graphs\n",
    "[14]: https://gist.github.com/karpathy/d4dee566867f8291f086\n",
    "[15]: https://github.com/tensorflow/models/tree/master/tutorials/rnn/ptb\n",
    "[16]: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/rnn_ptb\n",
    "\n",
    "\n",
    "## Implementing an LSTM network with TensorFlow low-level APIs\n",
    "The implementation is similar to [this][15] TensorFlow tutorial example, but differs in the following respects:\n",
    "1. This implementation depends on TensorFlow API that requires an Nvidia GPU to run. The tutorial example switches API depending on whether GPU is available or not, and so can run on either GPU or CPU. \n",
    "2. The tutorial example builds separate dataflow graphs for training, evaluation, and testing. This implementation builds only one dataflow graph and uses it for all three purposes.\n",
    "3. This implementation uses `tf.data.Dataset` for the input pipeline whereas the tutorial predates the Dataset API.\n",
    "\n",
    "There is only one class, PTBModel, which is responsible for:\n",
    "1. Building the dataflow graph. The `__init__` method of the PTBModel class builds the dataflow graph. It defines instance atttributes to hold onto those ops/tensors which have to be run at training or evaluation time.\n",
    "2. Running an epoch of training or evaluation by executing the correct ops/tensors in the graph.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "b_Kgm9eYKaR_"
   },
   "outputs": [],
   "source": [
    "#MIT License - Copyright (c) 2018 tmatha\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import logging\n",
    "import time\n",
    "import math\n",
    "import datetime\n",
    "import collections\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S4ctjFCxKaSL"
   },
   "outputs": [],
   "source": [
    "#MIT License - Copyright (c) 2018 tmatha\n",
    "\n",
    "class PTBModel(object):\n",
    "  \"\"\"\n",
    "  LSTM network for language modeling for training/evaluation/testing on the PTB\n",
    "  dataset. Model similar to the TensorFlow tutorial example at\n",
    "  https://github.com/tensorflow/models/tree/master/tutorials/rnn/ptb. The main \n",
    "  differences are:\n",
    "    \n",
    "  -The tutorial uses tf.contrib.cudnn_rnn.CudnnLSTM when running on Nvidia GPU, \n",
    "   and tf.contrib.rnn.BasicLSTMCell or tf.contrib.rnn.LSTMBlockCell when GPU is  \n",
    "   not available. This model only uses tf.contrib.cudnn_rnn.CudnnLSTM and \n",
    "   so Nvidia GPU is always required.\n",
    "  -The tutorial uses separate computational graphs for training, evaluation and\n",
    "   testing. This model builds only one computational graph and uses it for \n",
    "   all three.\n",
    "  -This model uses tf.data.Dataset for the input pipe line where as the tutorial\n",
    "   predates the Dataset API.\n",
    "  \n",
    "  Ref: \n",
    "    Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals, \n",
    "    \"Recurrent Neural Network Regularization\", ICLR 2015\n",
    "    \n",
    "  Imports:\n",
    "    tf\n",
    "    tf.nn\n",
    "    tf.math\n",
    "    tf.contrib.cudnn_rnn.CudnnLSTM - requires GPU\n",
    "    tf.layers.Dense\n",
    "    tf.train.GradientDescentOptimizer\n",
    "  \n",
    "  \"\"\"\n",
    "  \n",
    "  def __init__(\n",
    "      self, \n",
    "      vocab_size, \n",
    "      embedding_dim, \n",
    "      hidden_dim, \n",
    "      num_layers, \n",
    "      initializer,\n",
    "      dropout_ratio, \n",
    "      clip_norm,\n",
    "      learning_rate,\n",
    "      inp,\n",
    "      target,\n",
    "      batch_size):\n",
    "    \"\"\"Initalizes a model instance. Creates the nodes (ops) and edges (tensors)\n",
    "    of the dataflow graph and builds the graph. Data enters the graph via  \n",
    "    inp and target tensors. Model instance attributes hold references to all ops \n",
    "    endpoints which have to be run at training or evaluation time. \n",
    "    Kernels, biases and state are stored in tf.Variables.\n",
    "    \n",
    "    Args:\n",
    "      vocab_size: int;\n",
    "      embedding_dim: int; \n",
    "      hidden_dim: int; hidden state size, cell state size\n",
    "      num_layers: int; number of layers\n",
    "      initializer: initializer object for initializing kernels and biases\n",
    "      dropout_ratio: float; drop out ratio\n",
    "      clip_norm: float; maximum global norm for clipping gradients\n",
    "      learning_rate: tf.float32, 0-d tensor; \n",
    "      inp: input (features) tf.int64 2-d tensor of shape (seq_len,batch_size)\n",
    "      target: target (labels) tf.int64 2-d tensor of shape (seq_len,batch_size)\n",
    "      batch_size: int; batch size\n",
    "\n",
    "    Returns:\n",
    "      a model instance\n",
    "\n",
    "    Raises:\n",
    "      \n",
    "    \"\"\"\n",
    "    \n",
    "    #create ops and tensors;\n",
    "    embedding=tf.get_variable(\n",
    "        'embedding',\n",
    "        shape=(vocab_size, embedding_dim),\n",
    "        initializer=initializer,\n",
    "        trainable=True)\n",
    "    \n",
    "    #crashes when input_mode='skip_input' or 'auto_select'\n",
    "    rnn=tf.contrib.cudnn_rnn.CudnnLSTM(\n",
    "        num_layers=num_layers, \n",
    "        num_units=hidden_dim, \n",
    "        dropout=dropout_ratio,\n",
    "        kernel_initializer=initializer,\n",
    "        input_mode='linear_input')\n",
    "\n",
    "    dense=tf.layers.Dense(\n",
    "        units=vocab_size,        \n",
    "        kernel_initializer=initializer)\n",
    "    \n",
    "    h=tf.get_variable('h',shape=(num_layers,batch_size,hidden_dim),\n",
    "                     initializer=tf.zeros_initializer,trainable=False)\n",
    "    c=tf.get_variable('c',shape=(num_layers,batch_size,hidden_dim),\n",
    "                     initializer=tf.zeros_initializer,trainable=False)\n",
    "   \n",
    "    State=collections.namedtuple('State',['h','c'])\n",
    "    self._state=State(h=h,c=c)\n",
    "    \n",
    "    #build dataflow graph connecting the ops nodes with tensor \n",
    "    #edges starting with 'inp' and 'target';\n",
    "    #model instance attributes hold references to the ops endpoints which have \n",
    "    #to be run at training or evaluation time;\n",
    "    y=tf.nn.embedding_lookup(embedding, inp)\n",
    "    \n",
    "    #tf.contrib.cudnn_rnn.CudnnLSTM requires input tensor to be of shape \n",
    "    #(seq_len,batch_size,embedding_dim), where as tf.keras.layers.CuDNNLSTM \n",
    "    #requires input tensor to be of shape (batch_size,seq_len,embedding_dim);\n",
    "    #see _CudnnRNN(base_layer.Layer).call(self, inputs, initial_state=None, \n",
    "    #training=True)\n",
    "    y,(h,c)=rnn(y, initial_state=self._state, training=True)\n",
    "    self._update=State(h=self._state.h.assign(h),c=self._state.c.assign(c))\n",
    "    \n",
    "    y=dense(y)\n",
    "    self._loss=tf.math.reduce_sum(tf.nn.sparse_softmax_cross_entropy_with_logits\n",
    "        (labels=target,logits=y))\n",
    "    \n",
    "    optimizer=tf.train.GradientDescentOptimizer(learning_rate)\n",
    "    grad_var=optimizer.compute_gradients(self._loss)\n",
    "    gradients,variables=zip(*grad_var)\n",
    "    clipped, global_norm=tf.clip_by_global_norm(gradients, clip_norm)\n",
    "    grad_var=zip(clipped, variables)\n",
    "    self._training_op=optimizer.apply_gradients(grad_var)\n",
    "\n",
    "  def run_epoch(self, session, steps, is_training=False):\n",
    "    \"\"\"Runs an epoch of training or evaluation.\n",
    "        \n",
    "    Args:\n",
    "      session: the session object to run ops\n",
    "      steps: int; number of steps in an epoch\n",
    "      is_training: boolean; if True, train; if False, evaluate\n",
    "      \n",
    "    Returns:\n",
    "      A python list of losses, one loss (float) per step\n",
    "      \n",
    "    Raises:\n",
    "    \n",
    "    \"\"\"\n",
    "    session.run({'h':self._state.h.initializer,'c':self._state.c.initializer})\n",
    "    losses=[]\n",
    "    fetches={'update':self._update, 'loss':self._loss}\n",
    "    if is_training:\n",
    "      fetches['training_op']=self._training_op\n",
    "    \n",
    "    for step in range(steps):\n",
    "      fetched=session.run(fetches)\n",
    "      losses.append(fetched['loss'])\n",
    "    return losses\n",
    "\n",
    "  @classmethod\n",
    "  def instance(cls, model_type='small', vocab_size=10000, clip_norm=5,\n",
    "               learning_rate=None, inp=None, target=None):\n",
    "    \"\"\"Returns a model instance.\n",
    "    \n",
    "    Args:\n",
    "      model_type: string; 'small', 'medium' or 'large'; only 'small' is\n",
    "        implemented presently\n",
    "      vocab_size: int;\n",
    "      clip_norm: float; maximum global norm for clipping gradients\n",
    "      learning_rate: tf.float32 0-d tensor\n",
    "      inp: input (features) tf.int64 2-d tensor of shape (seq_len,batch_size)\n",
    "      target: target (labels) tf.int64 2-d tensor of shape (seq_len,batch_size)\n",
    "\n",
    "    Returns:\n",
    "      a model instance\n",
    "\n",
    "    Raises:\n",
    "      \n",
    "    \"\"\"\n",
    "    \n",
    "    return {\n",
    "        'small':PTBModel(\n",
    "            vocab_size=vocab_size,\n",
    "            embedding_dim=200,\n",
    "            hidden_dim=200,\n",
    "            num_layers=2,\n",
    "            initializer=tf.random_uniform_initializer(minval=-0.1,maxval=0.1),\n",
    "            dropout_ratio=0.,\n",
    "            clip_norm=clip_norm,\n",
    "            learning_rate=learning_rate,\n",
    "            inp=inp,\n",
    "            target=target,\n",
    "            batch_size=20),\n",
    "        'medium':None,\n",
    "        'large':None\n",
    "        }.get(model_type,None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Data shaping\n",
    "In language modeling, both input (feature) and target (label) sequences are formed from the same original sequence. The target sequence is simply the input sequence itself advanced by one time step. Both input and target data have to be fed to the model in chunks or mini batches of the requisite shape. The requisite shape differes from API to API; some (e.g. `tf.keras.layers.CuDNNLSTM`) require mini batches of shape (batch size, sequence length), whereas others (e.g. `tf.contrib.cudnn_rnn.CudnnLSTM`) call for mini batches of shape (sequence length, batch size). Here, sequence length is the length of the RNN layer or subsequence, and batch size is the number of subsequences processed together in one step of gradient descent. In other words, gradients are computed and applied for a full mini batch at each step of gradient descent, and it takes \"steps\" number of mini batches to cover an epoch. If the data doesn't divide cleanly into an integral number of mini batches, the last remaining data is dropped. The following function takes a 1-d data sequence and returns a tuple of input and target arrays to be fed to a Dataset which in turn has to feed correctly-shaped mini batches to the model. The original sequential order is preserved throughout except for having to split the original sequence into the requisite number of subsequences to be processed together. This entails in-memory data array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mzvbdhVNKaSH"
   },
   "outputs": [],
   "source": [
    "#MIT License - Copyright (c) 2018 tmatha\n",
    "\n",
    "def features_labels(data_array,batch_size,seq_len,batch_first=True):\n",
    "  \"\"\"Splits the sequential data into batch_size number of sub_sequences and \n",
    "  folds them into the requisite shape. This procedure is applied to the data to \n",
    "  derive the features array. This procedure is repeated to derive the labels \n",
    "  array also, except in this case the data is shifted by one time step. \n",
    "  Returns a named tuple of features and labels.\n",
    "  \n",
    "  Args:\n",
    "    data_array: np.int64 1-d numpy array of shape (size,)\n",
    "    batch_size: int;\n",
    "    seq_len: int; length of the rnn layer\n",
    "    batch_first: boolean; the returned numpy arrays will be of shape \n",
    "      (batch_size*steps, seq_len) if True and \n",
    "      (seq_len*steps, batch_size) if False\n",
    "      \n",
    "  Returns:\n",
    "    named tuple of features and labels, features and labels are np.int64 2-d \n",
    "      numpy arrays of shape (batch_size*steps, seq_len) if batch_first is True \n",
    "      and (seq_len*steps, batch_size) if batch_first is False\n",
    "    steps: int; number of mini batches in an epoch\n",
    "      \n",
    "  Raises:\n",
    "    ValueError: If input data_array is not 1-d\n",
    "\n",
    "  \"\"\"\n",
    "  if len(data_array.shape) != 1:\n",
    "    raise ValueError('Expected 1-d data array, '\n",
    "                     'instead data array shape is {} '.format(data_array.shape))\n",
    "  \n",
    "  def fold(used_array):\n",
    "    shaped_array=np.reshape(used_array,(batch_size,seq_len*steps),order='C')\n",
    "    \n",
    "    if batch_first:\n",
    "      return np.concatenate(np.split(shaped_array,steps,axis=1),axis=0)\n",
    "    else:\n",
    "      return np.transpose(shaped_array)\n",
    "\n",
    "  steps=(data_array.shape[0]-1)//(batch_size*seq_len)\n",
    "  used=batch_size*seq_len*steps\n",
    "  \n",
    "  features=fold(data_array[:used])\n",
    "  labels=fold(data_array[1:used+1])\n",
    "  \n",
    "  Data=collections.namedtuple('Data',['features','labels'])\n",
    "  return Data(features=features,labels=labels),steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Training and evaluation\n",
    "The non-regularized LSTM network, as in [Zaremba et al. (2015)][7], is configured and trained on the [Penn Tree Bank dataset][6]. Training the network for 13 epochs in [Colaboratory][12] with GPU acceleration takes less than 7 minutes. The trained network is shown to replicate word-level perplexities previously reported.\n",
    "\n",
    "[6]: http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz\n",
    "[7]: https://arxiv.org/abs/1409.2329\n",
    "[12]: https://colab.research.google.com/notebooks/welcome.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1005
    },
    "colab_type": "code",
    "id": "5USjmkBUbE0J",
    "outputId": "7d941314-88e6-42b7-9e66-b42428339ed3"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:date 2018-11-02 07:26:24.193743\n",
      "INFO:root:device /device:GPU:0\n",
      "INFO:root:TensorFlow vers. 1.12.0-rc2\n",
      "INFO:root:['aer', 'banknote', 'berlitz', 'calloway', 'centrust', 'cluett', 'fromstein', 'gitano', 'guterman', 'hydro-quebec']\n",
      "INFO:root:['consumers', 'may', 'want', 'to', 'move', 'their', 'telephones', 'a', 'little', 'closer']\n",
      "INFO:root:['no', 'it', 'was', \"n't\", 'black', 'monday', '<eos>', 'but', 'while', 'the']\n",
      "INFO:root:size_train 929589, size_valid 73760, size_test 82430\n",
      "INFO:root:vocab_train 10000, vocab_valid 6022, vocab_test 6049\n",
      "INFO:root:<tf.Variable 'lr:0' shape=() dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'embedding:0' shape=(10000, 200) dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'h:0' shape=(2, 20, 200) dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'c:0' shape=(2, 20, 200) dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'cudnn_lstm/opaque_kernel:0' shape=<unknown> dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'dense/kernel:0' shape=(200, 10000) dtype=float32_ref>\n",
      "INFO:root:<tf.Variable 'dense/bias:0' shape=(10000,) dtype=float32_ref>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                        TRAINING\n",
      "time      epochs         loss               perplexity\n",
      "                    train    valid       train       valid\n",
      "======    ======    =====    =====       =====       =====\n",
      "0:00:42     1.00     5.59     5.21      268.98      182.69\n",
      "0:01:12     2.00     4.91     5.00      135.28      148.40\n",
      "0:01:42     3.00     4.64     4.92      103.92      136.81\n",
      "0:02:13     4.00     4.48     4.90       87.94      133.68\n",
      "0:02:43     5.00     4.22     4.82       67.92      124.21\n",
      "0:03:13     6.00     4.03     4.80       56.08      122.05\n",
      "0:03:44     7.00     3.91     4.80       50.09      122.02\n",
      "0:04:14     8.00     3.85     4.80       47.09      122.01\n",
      "0:04:45     9.00     3.82     4.80       45.54      121.78\n",
      "0:05:15    10.00     3.80     4.80       44.69      121.36\n",
      "0:05:46    11.00     3.79     4.80       44.21      120.92\n",
      "0:06:16    12.00     3.78     4.79       43.95      120.60\n",
      "0:06:47    13.00     3.78     4.79       43.80      120.42\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfgAAAFnCAYAAABKGFvpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8U1X/B/DPzWo60t3KUOYjyEZE\nWbIdFZ5H2SCIezMVREBURARlb0VU4IciCAIWFIpsVEChiEBBRGWXLuhuaZvk90fatGlu0psmt0mT\nz/v14kVy7jjfm6b93nvuuecIRqPRCCIiIvIqCncHQERERK7HBE9EROSFmOCJiIi8EBM8ERGRF2KC\nJyIi8kJM8ERERF5I5e4AiKqrxo0bo06dOlAqlTAajQgKCsL48ePRoUMHWev966+/MGrUKABAdnY2\nsrOzUaNGDQBA37598dJLL7mknsaNG2P//v3mfTurR48emDVrFtq2bWt3vaysLMyfPx8//fQTBEGA\nUqnEgAED8Mwzz0AQBJfEQuQLmOCJnLBmzRpzAjx27BheeeUV7NixA+Hh4bLVeeedd2LHjh0AgE2b\nNiE2NharVq2Srb6qZDAY8MILL6Bhw4bYunUr/Pz8cP36dYwYMQIZGRl47bXX3B0iUbXBBE/kIvfc\ncw/q1KmD48ePo2fPnti1axcWLlyI3Nxc1K1bF3PmzEF4eDgWL16MmzdvIikpCWfPnkVYWBiWLVuG\n6OhoDB8+HD169MDOnTtx5coV3HvvvZg7d65DV65HjhzB/Pnzcdttt0GlUmHu3Lk2YykoKMCsWbNw\n8OBBFBYWYtCgQXj55ZfN+9q2bRu+++47ZGVl4YUXXsCwYcMAAEuXLkVsbCz0ej0aNmyI2bNnIzg4\n2O6xlbVhwwZ88803WLNmDbRarbn8wIEDSEpKwpo1a6BWqwEANWrUwPz585Geng4AuHbtGt5++21c\nuXIFarUazz//PPr06YMrV65gyJAh6NWrFxISEvDhhx/i0UcfxauvvootW7YgPT0dU6dOxQMPPIDF\nixfj+vXr+OCDDwDA4v327duxdOlS6PV6qFQqTJkyBe3atavcl4LIjXgPnsiFioqKoNFocPnyZUyY\nMAFz587F7t270a5dO0ydOtW83o4dOzB58mTs2rULERER+Pbbb83L9uzZg5UrVyIuLg6HDx9GfHy8\nw3EkJCRgyJAhmDt3rt1YVqxYgfPnz2Pr1q3Ytm0b4uLisHfvXvN+rl27hq1bt+Lzzz/HRx99hBs3\nbuDUqVP46quv8O2332Lnzp0oKCjAl19+KenYAODo0aNYsWIFPv74Y4vkDgC//vorOnXqZE7uJerU\nqYOWLVsCAN5++23cd999iIuLw/LlyzF9+nRcuXIFAJCeno4mTZqY48nJyYEgCNi2bRtmzZqFKVOm\noKioyO5n995772H58uXYvn073n33XezZs8eBT57IczDBE7nI/v37kZqaijZt2uDAgQO477770KhR\nIwDAkCFDsGfPHuj1egBA27ZtUbt2bQiCgCZNmiAxMdG8n5iYGGi1WgQEBKBevXoWy6TSarXmvgD2\nYtm7dy+GDh0KjUaDgIAAPPbYY9i5c6d5P3369AEANGzYEA0aNMCpU6fQvHlz7Nu3D0FBQVAoFLj7\n7rtx+fJl8zb2ji0xMRGTJk3C4sWLERkZaRV3RkYGIiIibB5XYWEhfvnlFwwdOhQAULt2bbRr1w6H\nDx82L3/wwQctthkwYAAAoGPHjigqKsLFixftfnYRERFYt24drl69irZt22LSpEl21yfyVGyiJ3LC\n8OHDzZ3sateujRUrViAwMBBZWVk4evQoYmJizOsGBQWZm5l1Op25XKlUmhN/yXq2lkkVEhJifm0v\nlqysLMycORPz5s0DABQUFJivlAEgLCzM/Fqn0yEzMxN5eXmYOXMmjhw5AsCUlLt162axnq34Z8yY\nAYVCYTOJh4WFITk52eZxpaenw2g0WtQRHByMGzdumOsr+/kJgmDxWQQHByMjI8Pm/gHg448/xscf\nf4x+/fqhZs2amDx5Mu677z672xB5IiZ4IieU7WRXVnR0NDp27IhFixa5ISrpsURHR+PZZ59F9+7d\nRbfNyMjAHXfcYX4dEhKC1atX48KFC9i0aRMCAwMxf/58JCUlSYrljTfewL///ot33nkHy5Yts1re\nrl07TJw4Efn5+RbN95cuXcLu3bsxfPhwKBQKcyyAKenbOmEwGo24efOm+USlZDuFQgGDwWBxnCXq\n1KmDmTNnwmAwYMuWLRg3bhwOHjwo6fiIPAmb6IlkcP/99+Po0aPmpus//vgD06dP97hYevbsiQ0b\nNkCv18NoNGLZsmU4cOCAedtt27YBAP7++29cunQJLVq0QFpaGho0aIDAwEBcvXoV+/fvR25urqRY\n6tSpg1GjRuHSpUvYvHmzaKwNGjTAhAkTkJ2dDQC4fv06xo4di6KiIqhUKtx///1Yv349AFPiP3r0\nKDp27GizzpJj+Omnn6DValG/fn1ER0fj3LlzMBgMuHHjhvmYb9y4gWeeeQbZ2dlQKBRo1aoVH82j\naotX8EQyiI6Oxvvvv48RI0agsLAQgYGBmDx5ssfFMnToUFy5cgW9e/eG0WhE8+bN8dRTT5m3rV27\nNh577DFkZmbirbfeQmhoKIYMGYLRo0fj4YcfRuPGjTFx4kSMGjVK8qN6Go0GH374IZ5//nl06NDB\nogVEEAR88sknmD9/Pvr06QOVSgV/f38MGzbMfC/9vffew5QpU7Bp0yao1WpMnz4dNWvWNHe0K0up\nVKKwsBC9e/dGRkYGpk+fDoVCgZiYGMTGxuKBBx5AgwYNEBMTg7S0NISHh6Nz587o378/lEol1Gq1\nuac9UXUjcD54IvJGV65cwUMPPYSEhAR3h0LkFmyiJyIi8kKyNdEfOXIEY8aMwZ133gkAaNSoEZ5/\n/nlMmDABer0eUVFRmD17NjQaDWJjY7F69WooFAoMGjQIAwcOlCssIiIinyBbE/2RI0fw1VdfWfTc\nnTRpErp06YJHHnkE8+bNQ40aNdCnTx/07dsXGzduhFqtxoABA/Dll18iNDRUjrCIiIh8QpU20R85\ncgQ9e/YEAHTv3h2HDh3CiRMn0KJFC+h0Omi1WrRp06ZSI3cRERFRKVl70Z8/fx4vv/wyMjIyMHLk\nSOTl5UGj0QAwjRaVkpKC1NRUi4k5wsPDkZKSImdYREREXk+2BF+vXj2MHDkSjzzyCC5fvownn3zS\nYkQrW3cGpNwxMBqNEBQVND5Mnw689ZZDMRMREXkL2RL8bbfdhl69egEwDW4RGRmJkydPmkeoSkpK\nQnR0NKKjo5GammreLjk5Ga1bt7a7bykDT+TkFiA3Jcu5g/BQUVE6pHjpsUnhy8fvy8cO8Ph5/L57\n/FFRuopXKke2e/CxsbH4/PPPAQApKSlIS0tDv379EBcXBwDYuXMnOnfujFatWuHkyZPIzMxETk4O\n4uPj0bZtW7nCIiIi8gmyXcH36NED48ePx+7du1FYWIipU6eiSZMmePPNN7F+/XrUqlULffr0gVqt\nxrhx4/Dcc89BEASMGDHCYiKJSuP4PURE5MNkS/BBQUH45JNPrMpXrlxpVRYTE2Mx0xURERE5hyPZ\nEREReSHvTfBsoiciIh/GBE9EROSFvDfBExER+TDvTfC8gici8nn79u2WtN7ChXNx7dpVm8snTnzd\nVSFVGe9N8ERE5NMSE69h1644SeuOGTMOtWrVtrn8ww/nuSqsKiPrWPRyMkRFQ5GS7O4wiIjIQ82b\n9xHOnDmNzp3vxUMPPYLExGtYsGAZZs6chpSUZOTl5eHZZ19Ep06dMXLki3j99QnYu3c3cnKycenS\nRVy9egWjR49Dhw6d0Lt3T3z//W6MHPki7r23HeLjjyI9PR0ffTQfkZGRmDbtbVy/nogWLVpiz55d\n2Lz5B3cfPq/giYjIOz3++HC0bt0GTz/9PIqKCrFs2WfIycnGffe1x5Iln2LatJn4/PPlVtslJydh\nzpxFGDNmPGJjN1ktDwwMxMKFH6N9+444cGAPDh/+BQUFt/Dpp6vQps29SE31jAnTqu0VfIV4D56I\nyKOEdWkH1dkzTu0jqszrorua4OaBI5K2a9KkGQBApwvGmTOnERu7CYKgQGZmhtW6LVua5kOJjo5G\ndna21fJWre42L8/IyMDFi/+iRYtWAIAOHTpBqVQ6ckiy8d4ET0REHkVqMrbFmclm1Go1AODHH3cg\nMzMTS5d+hszMTDz//HCrdcsmaLEZTssvNxqNUChMZYIgSJoQrSp4bxN9YaG7IyAiIjdSKBQW05QD\nQHp6OmrWrAWFQoH9+/eg0AW5onbt2/HnnwkAgF9/PWxVp7t4bYLXHNzv7hCIiMiN6tatjz//PIuc\nnNJm9m7deuCXXw5izJhX4O/vj+joaKxcucKpejp27IycnBy88spzOHHiOIKDQ5wN3SUEo1j7QzVQ\n1LQZVGcSbC4vvOdepG+X9vxjdePLcyIDvn38vnzsAI+fx++Zx5+ZmYH4+KPo1q0nUlKSMWbMK1i7\n9luX1lGZ+eCr7T34oiZN7SZ4oFqetxARUTUTEBCIPXt2Ye3aNTAaDRg1yjMGxam2CR7wjE4MRETk\n21QqFaZNm+nuMKx47T14IiIiX8YET0RE5IWY4ImIiLyQFyd43qMnIiLf5b0J3kNGEiIiIs82YMD/\nkJubizVrVuHUqT8sluXm5mLAgP/Z3b5kStofftiK/fv3yhano6ptgjeEh9tfoXo+3k9E5LM2b1ah\na9cA1KwZhK5dA7B5c9U+6DV8+NNo3rylQ9uUnZK2V6//oWvX7nKEVinV9jG5wm49gM+sZwGyx3/R\nfNx6rC8MdevJExQREVXK5s0qvPSSv/n9mTPK4vd56Nu3qFL7fPbZYZgxYy5q1KiB69cTMWnSOERF\nRSMvLw/5+fl47bU30LRpc/P6H3wwFd269UTr1nfjrbcmoKCgwDzxDADs3LkdGzeuh1KpQL16DfHm\nm2+Zp6RduXIFDAYDQkND0b//YCxbthAnT55AUZEe/fsPQkxMb9GpZmvUqFHpz6wi1fYKvjJN8EHT\n34X2q/+TIRgiInLGggUa0fKFC8XLpejSpTt+/vkAAODgwf3o0qU7/vvfPli8eDlefnkkvvpqteh2\ncXHb0aBBQyxb9hnuvLORuTwvLw9z5y7Gxx9/gUuXLuDvv8+bp6R95pkXzOv9/ns8/vnnb3z88RdY\ntOgTfPHFp8jNzQFgPdWsnKrtFTwREVUvXboE4OxZx6ZSTUhQIjq67DCtpa/vukuPAwdy7dTXHUuW\nLED//oPw00/7MXLka1i3bg2+/noNCgsLodVqRbe7cOEftG59DwDg7rvvMZcHBwdj0qRxAICLF/9F\nRka66PZnzyagdes2AAB/f3/Uq9cAly9fBmA91aycmOCJiKhK2EvGXbsG4MwZ6+TftKke+/aZtnN0\nLPoGDRoiLS0FSUnXkZWVhYMH9yEyMhpvv/0+zp5NwJIlC0S3MxoBhcLUSmwwmPpzFRYWYt68WVi1\nai0iIiIxYcJYm/UKgmDRDayoqNC8v4qmonWl6ttET0REXmPs2ALR8jFjxMul6tDhfnz66TJ07twV\nGRnpqF37dgDA/v17UVQkfm+/Tp26OHv2DAAgPv4oACA3NwdKpRIREZFISrqOs2fPoKioSHRK2rvu\naobjx48Vb5eLq1ev4Pbb6zh1HJXhxQmeveiJiKqLvn2LsHx5Hpo21UOlMqJpUz2WL698B7sSXbt2\nx65dcejWrSdiYnpj/fqv8NprI9CsWXOkpaXh++9jrbaJiemN06dPYsyYV3D58kUIgoCQkFDce287\nPP/8k1i5cgWGDh2ORYvmmaekXbRornn7Vq1ao3HjuzBixAt47bURePnlkfD397eqR27VdrrYjLUb\nEDJskM3lhfe0Rfp2yw4MUdHByBk7HrmT35E7PFl56pSJVcWXj9+Xjx3g8fP4fff4KzNdrBdfwRMR\nEfkuJngiIiIvVH0TfIXPwXOoWiIi8l3VNsEbFY49S0lERORLqm2Ch6L6hk5ERCQ3ZkkiIiIv5LUJ\nXn3sN3eHQERE5DZem+CJiIh8GRM8ERGRF2KCJyIi8kJM8ERERF6ICZ6IiMgLMcETERF5IZ9L8EL1\nnDyPiIjIIV6d4JVnEhA4+Q13h0FERFTlvDrBh3dtj4DPlrs7DCIioirn1QmeiIjIVzHBExEReSEm\neCIiIi/EBE9EROSFmOCJiIi8EBM8ERGRF2KCJyIi8kKyJvj8/Hw88MAD2LRpExITEzF8+HAMHToU\nY8aMQUFBAQAgNjYW/fv3x8CBA7FhwwY5wyEiIvIZsib4jz/+GCEhIQCARYsWYejQoVi7di3q1q2L\njRs3Ijc3F0uXLsWqVauwZs0arF69Gunp6S6PQ334F5fvk4iIyJPJluD//vtvnD9/Ht26dQMAHDly\nBD179gQAdO/eHYcOHcKJEyfQokUL6HQ6aLVatGnTBvHx8S6PJfTRGJfvk4iIyJPJluA/+ugjTJw4\n0fw+Ly8PGo0GABAREYGUlBSkpqYiPDzcvE54eDhSUlLkComIiMhnqOTY6ZYtW9C6dWvccccdosuN\nNmZ0s1UuJjQ0wKGYoqJ0AICAAA0Cil9XZ1FecAzO8OXj9+VjB3j8PH7fPn5HyJLg9+3bh8uXL2Pf\nvn24fv06NBoNAgICkJ+fD61Wi6SkJERHRyM6Ohqpqanm7ZKTk9G6dWtJdaSn5yLUgZhSUrIQBSA3\ntwA5KVmOHZCHiYrSIaWaH4MzfPn4ffnYAR4/j993j78yJzayJPgFCxaYXy9evBi1a9fG8ePHERcX\nh8ceeww7d+5E586d0apVK0yZMgWZmZlQKpWIj4/H5MmT5QgJMBjk2S8REZEHkiXBixk1ahTefPNN\nrF+/HrVq1UKfPn2gVqsxbtw4PPfccxAEASNGjIBOJ0/zi/8nS2XZLxERkSeSPcGPGjXK/HrlypVW\ny2NiYhATI38vd8W1K7LXQURE5Cl8byS7/Dx3R0BERCQ7n0nwilTT43cBy5e5ORIiIiL5+UyC127a\n6O4QiIiIqky1TfCGyCh3h0BEROSxqm2C1zdvAWPxyHhERERkqdomeAC4ceR3d4dARETkkap1gjfU\nvt3dIRAREXmkap3giYiISBwTPBERkRdigiciIvJCTPBEREReiAmeiIjICzHBExEReSEmeCIiIi/k\nkwk+KjoYflu+dXcYREREsvHJBA8Ayj/PujsEIiIi2fhsgiciIvJmTPBEREReiAmeiIjICzHBExER\neSEmeCIiIi/EBF/i1i0ImRnujoKIiMglqn2CNyqVLtmP7rWRiPzPHS7ZFxERkbtV/wTvH1DJDY0W\nb5UX/nVBNERERJ6h2id4IiIissYET0RE5IWY4EuUa7InIiKqzpjgiYiIvBATPBERkRdigjdjEz0R\nEXmP6p/gXXXvnPfgiYjIi1T/BE9ERERWfDjB84qdiIi8lw8n+HLYRE9ERF6ECZ6IiMgLMcETERF5\nISZ4IiIiL8QEX4L34ImIyIv4bIIPnDfb5jL1zwerMBIiIiLXq/YJ3lCrlmt2VOYCPrRvbwjJya7Z\nLxERkRtU+wSf/v2P7g6BiIjI41T7BG8MDXPRjngPnoiIvEe1T/BERERkjQneDvXhX6A8ddLdYRAR\nETlM5e4APIXiRppVWeijMdDXqIkbf/zphoiIiIgqj1fwxZRXLrs7BCIiIpdhgreFne6IiKga84oE\nf6vX/9wdAhERkUfxigQPQXB3BERERB7FKxK8ITjYZftSXLvqsn0RERG5i1ck+JwPPnLZviJaN3HZ\nvoiIiNxFtsfk8vLyMHHiRKSlpeHWrVt49dVXcdddd2HChAnQ6/WIiorC7NmzodFoEBsbi9WrV0Oh\nUGDQoEEYOHCgQ3UZg3QyHQUREVH1JFuC37t3L5o3b44XXngBV69exbPPPos2bdpg6NCheOSRRzBv\n3jxs3LgRffr0wdKlS7Fx40ao1WoMGDAADz74IEJDQ+UKzUx5/i8oUlNQ2L6j7HURERFVJdma6Hv1\n6oUXXngBAJCYmIjbbrsNR44cQc+ePQEA3bt3x6FDh3DixAm0aNECOp0OWq0Wbdq0QXx8vFxhWQh+\n9gmEPhojukwAH5MjIqLqS/aR7IYMGYLr16/jk08+wTPPPAONRgMAiIiIQEpKClJTUxEeHm5ePzw8\nHCkpKRXuNyrK+WZ5ldJ0fhM1ZbzVsoiIIACAUiG4pC5X88SYqpIvH78vHzvA4+fx+/bxO0L2BL9u\n3TqcOXMGb7zxBoxlBo8x2hhIxlZ5eSkpWRbvoyoRW5HeYPoAPv3UallaWjYiAOgNRtwoU5fy3J8w\nqtUw1G9QiRpdIypKZ3X8vsSXj9+Xjx3g8fP4fff4K3NiI1sT/alTp5CYmAgAaNKkCfR6PQIDA5Gf\nnw8ASEpKQnR0NKKjo5GammreLjk5GdHR0XKFZakSz8+H338vwh7u5vpYiIiIXEi2BH/06FF88cUX\nAIDU1FTk5uaiY8eOiIuLAwDs3LkTnTt3RqtWrXDy5ElkZmYiJycH8fHxaNu2rVxhuYbe4O4IiIiI\n7JKtiX7IkCF46623MHToUOTn5+Odd95B8+bN8eabb2L9+vWoVasW+vTpA7VajXHjxuG5556DIAgY\nMWIEdLqqucci3Lhhc5kRHB2PiIiqL9kSvFarxdy5c63KV65caVUWExODmBjx3uxyUiZdr/I6iYiI\nqoJXjGQnB7uPyXGmOSIi8nBM8BUR6YinyM5ikiciIo8mKcF//fXXyM7OljsWz2QjkSv/Pl/FgRAR\nEUknKcH/+eefePTRR/Hmm2/i6NGjcsdERERETpLUyW7q1KkwGAw4cuQIYmNjMWfOHPTs2RODBg1C\nSEiI3DESERGRgyTfg1coFKhTpw5q1KiBgoICnD59GsOGDcOuXbvkjI+IiIgqQdIV/ObNm7Fp0yak\np6dj4MCBWLlyJUJCQpCZmYknnngCDzzwgNxxeh4+Jk9ERB5MUoL/+eefMWbMGIsR5vLz8xEcHIyn\nnnpKtuDcqqRzXXEv+uDnnoTf1i1uDIiIiEg6SU30N2/etBo+dtiwYQCA/v37uz4qD6D9crXpRXGi\nZ3InIqLqxO4VfGxsLJYuXYrExER069bNXF5YWIjIyEi5Y3Mr9c8H7a9QiYlqiIiIqordBP/oo4+i\nd+/eeOuttzBq1ChzuUKhqLoZ3yRKSc5EZJ1oCMWz1REREfkyu030CQkJUCqVeOyxx3Dp0iXzvwsX\nLuDXX3+tqhglK2rRynU740h1RERUjdm9gt+yZQuaNm2KZcuWWS0TBAEdOnSQLTB30/zyk+lFJZri\ng598HFmLlsEYGubiqIiIiKSxm+AnT54MAFi+fDkCAgIsliUlJckXVTVgbzpZvx3fI+/sGRS271iF\nEREREZWS1It+wIABFkPUfvfdd3jiiSdkC8qjsKmeiIiqIUnPwS9ZsgTTpk1D48aNkZiYCLVajXXr\n1skdm+M8PRnn5gLlWkKIiIjkIOkKvkGDBhg9ejS2b9+Ov/76C6NHj0ZERITcsXm2iu7Ni5xsRNWr\nAdXxYzIFREREVErSFfzbb7+NCxcu4Msvv0R6ejpee+01PPjgg3jllVfkjs/tlNcTXbo/RWqKS/dH\nREQkRtIVfMOGDfF///d/qFOnDlq2bOm588Nz8BkiIiIAEhP8008/jf379+PLL78EYOpBP378eFkD\nqxSZ7sGrDh+yLuTJBBEReTBJCX727NnYuHEjNm3aBADYunUrpk+fLmtgnkT9x3HrsmO/uSESIiIi\naSQl+N9++w1LlixBYGAgAGDEiBE4ffq0rIFVShVeVStupFVZXURERI6SlOD9/PwAmEavAwC9Xg+9\nXi9fVN7A0x/ZIyIiryYpwbdp0waTJk1CcnIyVq5ciSeeeAL33Xef3LE5rgqTqrH4ZCcqOhiKy5eq\nrF4iIiIpJD0m99prr2HHjh3QarW4fv06nnnmGTz00ENyx+bRBKMRQvFwvcrLl2C4o460DXllT0RE\nVcBugr98+bL5dbNmzdCsWTOLZXfccYd8kVVGFSdP7fq1AADVyRNQxR9D3sgxpQvZy56IiNzIboJ/\n6qmnIAgCjCKJUxAE7N69W7bAPIrI8RsFwTzdTNDbkwDAMsHzSp2IiNzIboLfs2dPVcXhGjJdNQfM\nn21Vpps8AdlT3pOlPiIiImdJugd//vx5LFq0COfPn4cgCGjcuDFGjx6NevXqyRyeZ1DcuGFjichV\nen6+rLEQERFJIakX/cSJE9GlSxcsWbIEixYtQvv27TFhwgS5Y6uWFGmpphcONtELqakQUjhOPRER\nuYakK3h/f38MGDDA/L5hw4aIi4uTLajqImj6VOvCoqJK7Suse0cIBgPSTp93KiYiIiJA4hV8+/bt\nsWvXLuTl5SEnJwe7d+/G3XffDaPRCIPBIHeM0nlAz/WgqVNML2zFYqNcmXQdipRkmaIiIiJfI+kK\nftmyZaIj1y1ZsgSCIODMmTMuD6xSPKDnut/3saYXtmLxgBiJiMj7SUrwJ0+ehEIh6WLfvdycPLUr\nP3Nr/URERCUkZe2nnnpK7ji8gu7N163KVEd/hWb3TjdEQ0REvkzSFXyTJk2wcOFC3H333VCr1eby\nDh06yBZYpXjAPfjygp8dDuX1RKQkZ7o7FCIi8iGSEnzJPfajR4+aywRB8LwEX80ZPfAEhYiIqidJ\nCX7NmjUAAKPRaJ4ylion5InBtq/m+dkSEZGLSLoHf/bsWfTr1w+PPPIIAGDp0qU4ceKErIF5DQlJ\nW3E9UfK6REREUkhK8NOmTcOMGTMQFRUFAOjVqxdmzpwpa2DVXkmP/uL/1XvtTMxTWFj62mBA4OQ3\nZAyMiIh8gaQEr1KpcNddd5nf169fHyqVpNb9KlXUoqW7Q7BWPBBQ6OC+ttcp83ifkJuDgM+Wyx0V\nERF5OckJ/vLly+b77/v37xedQtbdbj1qJ4m6iTI5SfrKbKInIiIXkXQZ/uabb+LVV1/Fv//+i3vu\nuQe1a9fGrFmz5I7N9whChYP1KE/+AbRoBEBbNTEREVG1ZDfBZ2dnY+nSpfj333/x2GOPoV+/ftBo\nNAgKCqqq+BzjaVfAOTnS1ivp4/yAAAAgAElEQVRJ6hLiD+95P/C//wGff+VEYERE5O3sNtFPnToV\ngiBg8ODB+Pvvv7FmzRrPTe4eSDfmVcc3knLro2ynPCIiIhF2r+CvXr2KOXPmAAC6dOmCp59+uipi\nqjwP6xegcOT+ewWEjHQYtf4u2x8REXk3u1fwZXvKK5VK2YPxJkJ+HtTxRyte0WIj2030kXfWge71\nUaY3Fcw5r/7pgGP1EhGR17Gb4MuPWsdR7KTTbP0OQkGBYxtV0MlOeemi6cWuXXZ3E9rvvxWeBBAR\nkXez20R//PhxdOvWzfw+LS0N3bp1Mw9Zu2/fPpnDc5AHnYAIer30lR3oZCc9AM/5LIiIqOrZTfA7\nduyoqji8juaHbY5vZDQCjpwYVLQvIiLyWXYTfO3atasqDpcoatbc3SGYKbKzHN5GyM9H5F31ZYjG\nRLPjBxQ8/Aiv7omIfICs483OmjULx44dQ1FREV566SW0aNECEyZMgF6vR1RUFGbPng2NRoPY2Fis\nXr0aCoUCgwYNwsCBAytVnzEk1MVHIA/luT+hv7NRaaKVerXt5FV5yJNDkPrnBRjDwp3aDxEReT5J\nQ9VWxuHDh/HXX39h/fr1+OyzzzBjxgwsWrQIQ4cOxdq1a1G3bl1s3LgRubm5WLp0KVatWoU1a9Zg\n9erVSE9PlyssjxB+/71QHzlkdx3V7/HyVM6rdyIinyBbgr/33nuxcOFCAEBwcDDy8vJw5MgR9OzZ\nEwDQvXt3HDp0CCdOnECLFi2g0+mg1WrRpk0bxMfLlNw8SV4ehMwMIDtbdLHquJOfAe/BExH5NNma\n6JVKJQICAgAAGzduRJcuXfDTTz9Bo9EAACIiIpCSkoLU1FSEh5c2GYeHhyMlJaXC/UdF6eQJvIqE\nhvgDne8DatYEvvnGarlOp4Wu3DGqc0tPBio6/qgoHaBWW5VHRuqAsOr92QHV/+fvDF8+doDHz+P3\n7eN3hOxzvu7atQsbN27EF198gYceeshcbms2Oqmz1KWkiHdi0w16HNpvvnY80CqWnp6L0MREGLJz\ncDMtGxHllmdl5SO/zDFGAcCpU+b3to6/ZN2U5Eyg+GSqbHlqahaMRZ431a8joqJ0do/fm/nysQM8\nfh6/7x5/ZU5sZGuiB4CDBw/ik08+wYoVK6DT6RAQEID8/HwAQFJSEqKjoxEdHY3U1FTzNsnJyYiO\njpYzLI+gPvxL6RuRkxrNrjgoT5+CkH7TsR2zaZ6IiCBjgs/KysKsWbOwfPlyhIaaerd37NgRcXFx\nAICdO3eic+fOaNWqFU6ePInMzEzk5OQgPj4ebdu2lSssjxG4wDTGvyIrU3S5349xCO/eEUGT3qhc\nBbYSPU8AiIh8gmxttT/88ANu3ryJsWPHmss+/PBDTJkyBevXr0etWrXQp08fqNVqjBs3Ds899xwE\nQcCIESOg0zlxj8XLeolLGe42aNJ4oLAI2XMWVEFERERUHciW4AcPHozBgwdbla9cudKqLCYmBjEx\nMa6puBomeGf7DGhXfgbBYGCCJyIiM1nvwZM0gfNm2VxmlHDCIhgMIhuyKZ6IyJcxwXsbJnYiIoIX\nJngpV7yVtQ6D0RInoEIhWuIE1sH6FoTLVXA8QpknEABAdfKE6UVlE31ODvw/XlK5bYmIyGN4XYKX\nyzoMxuNYh5NoCT1UOImWeBzr5E/yFZyvRDZtYPE+4MPpTlWnOfwzgt6d7NQ+iIjI/ZjgJZoB8aQ3\nE5NkrVeZmOjQ+n67fzS9YFM9EZFPY4KXKAFNHSp3lYompalw+317oN6zq8L1hLQ0IDe3whOD0Ed6\nQHX0V6diIiIi+XlfgpfpHnxTJIiW++EW/kQjWep0SnGiDh3UB6FD+lW4emST+gge9XKF66mPHYVm\n/1676whpaeLlN8TLiYjI9bwvwctkMmaIlg/ABtyPnzADk1Ao/9D+slIkXpPWtG9nHeW5PxHZpL71\nvi9fQuRd1uVERCQPJniJhmA9vsYQi170X2MIVuFZHEVbHERn3IvfcBT3VF1Q9pKxI/fgb92qeB2D\nQdI+BRtD7wo5OdLjISIipzHBO2AI1uMEWqMQGpxAawzBegBAXVzCD+iFNzAb/8U2jMds5MLfdRWL\nJEf1TwesyoKffLzifYkkab/vY00vBMFmEo+qEQq/9Wtt7sOsGo4kSETkjbwvwbspwQgAhmEtTqIF\nrqMGWuAkdqOHS/atOnfWqiy033+h+vWIRZnfju/LxFOJXvQVfHaqc386vk8iInIL70vwbhaFVHyJ\n4ViMUXgWX+BZfI4bCJOlroBFcx3eRsgUaUIvuSKvIMErrl52uD4iInIPr07wRU2bu63uXtiOU2iO\nIGSjOU5hAwZU5pravgruiQc/Maj0TXEzf8R9rWzvThBgL0jt5m8dCo+IiNzH6xK8MSAAAFDQoRNu\n7vvFrbHokI1FGIONGIB38R76YjOuopbrKqigk53fzh3mt6qz4o/52WSv411l7sHz3jwRUZXyugSf\nM+kdd4dgpSMO4TjuRmv8jtb4HcvxIgwVjUErgfLaNZvLtKu+sHgvFBZWvMMyneyU5/+yuVrgnA/t\n74OIiNzO6xI8goJM/3tYovFDAabiPexFd6zEM+iOvTiHO6VtbOOKWXXmtM1NlOfPWbzXbPuuwmqE\n/Dxp8RARkcfzvgTv4ZrjNH5GJ/THt+iIXzATEyscIMdvW6zT9QZ8+nGF66iPx0vucCcbo9H0zD0R\nETmFCd4NlDBgNBbjKNpiP7pWOECO/xefOlyH4MhAN2XW1fy4w86K8vP79htE1Qh1awxERN7A5xK8\nQRfs7hDM6uEituMRjMcc9Mb3eAOzRAfIEXJzqywm5eVLxZVW8greyU529u79ExGRdN6b4G0klPxn\nnq/iQOwTADyBr3ASLXAVtV02QI7fFgceaRP7rBxJ8AYDoNfbX4fT1xIRVSnvTfC2EoqHJppopGAt\nhmERRuMZrMRz+Aw3UfmmaiE/X/rKZZvoS2aKcyDBR9UIRVRN02A+urEjpddbTHX0V+hGvOjwdkRE\nZJv3Jvhqqjd+wGk0gz/y0AynsRH9XT9ATjHlqZNQnTwhvuzPM5Xap+r0SYv3iqTrQHa23W20334D\n7YZ1pjc2Tiy0n38K5OQgKjoYSHDwmX4iIh/EBO+BdMjGEozCBgzE23gf/bAJ11AT6zDYYja7dRhc\nqf1rtpoemQvv0QlhPTuLrqPdsgmKf/6G4npipY8DACJaNELwq87fFtFNGg/NgX2mN4nOxURE5Auq\n9wTmDipq0sxjm+jFdMIv+B2t8QHewl04iyyUdhA8iZZ4HKar3pJZ7aQKeW44Ui5cN78X0m+KrhfR\n/m4U3dkIN38+WonoSymSkyQ1+SuuXPa48QuIiKor772Cd7bjmIfwQwGm4V3UwHXR5TMxqVL7FYpK\nR7bTTZ4gsoLps1L+fb5S+7fcl/2vmbG4roClC6XtrxqdpBERuYv3Jngv8w8aiJafRAsswBgcQnvk\nw8/l9QqeNOgMEzsRkWRM8NVEU4h3LLsdV3AOjTASSxCBNLTDYYzGQqzF4/gbDWx30HNBa4aQkQ4h\nLc328ps3AADqY785sFMJcT37rPT9ERH5KK+9B2+oaT1rmzEw0A2RuMZkzDDfcy9rFiaY78Hnwh/H\ncA+OoB02oR/exEfIhxbtcRjtcATtcRj34jeEINP5q2GjEaF9elv0mg9vdZfFKmE97rdYX+p+K3SZ\n89ITEVXEKxN8asI/5mSeNXsB9P+5E4bbasAQGobgF55yc3SVU5LEZ2ISEtAUTZGASZhp0cEuAHno\njJ/QGT+Zy66gNo6gHY6gHabhHcSjDeriIu5+Q4cueAHtcRjNcBpKONYUHzjtHatH4pSJlrPbKa9e\nKY1t/mzrnWRnA2VPuoxGCHm2J7wJmPuRQzESEfkyr0zwxshI8+v8pyybczU/H6zqcFxmCNY73GP+\ndlzF7diE/tgEACiECqfQHIc3t8cv6Ih5eB3XUAv34Bja47D5aj9UpKl8HQZjBiYjAU3RZOVFvIXr\nkuPxK57NTnn2DPR3NQEARDWohczFn1iuaKeJXm3jmX0LhYVQ/nkW+uYtJMVFROStvDLBk21qFOFu\n/I678TtegSm53kQofsV9OIz2+AQv4xmshO7HInTAILTDETT+TYFUPIGnsMa8n1O5De0/pldu/nmh\noAAAoF27BjnTZpjLlVevmJO66vgxFHbq4tTxadesgm7iOKQkZ1ouMBohZKTDGBrm1P6JiKoLdrIj\nhCEdD2Mn3sU0bEcvpCEC2zq8h174AefxH0yapMXTWC267WTMwM/oiFNohsu4HZnQwQAButdHia4f\n8MkSy4Iy99zVx+MBhe2vZNmBfrp2DcDmzdbnp7aG6NXs3IHIRnVt7puIyNvwCr4Mgy4YiqzMilf0\ncgKAO3XX0QKb8CTWIGXXk6gZ7Q+9yPngBdTDG5iNDISY/+UiALqNuQjBjDKlpf/U0zUIDgZq4WVo\nE1oipCAF0eiEEGRAkxWKmtDBYLDM9esw2KKT4ZkzwEsv+QPIQ9++RWWCF2/iVyQnuejTISKqHpjg\ny1KyQcOWpkjASbS0Km+Bk/gFnSzK9FAgrVkX5P7xr0VqT0coMhCCa4HAzZsCktEa6duVyFA0QSZm\nmdb5pjYyMRa5tYIQFAQEBxuh0xlxEZ+LxrVwocYywcssYP5s5I4dXy0HTSIi38IEX0ZBtx7Qbtnk\n7jA8girhVOmb7Gybj+lNwkyrMiUMCFNlIRoXRfed8tpM08QxS18GyufmHNN/169mQt2oOa69Mgep\nHXrhwR7ig/gkJCjwyCMBaNpUj6ZNDWh94Q60c2IWvooEznwfua+OBvxcP6gQEZEr8ZK1jJxJ77g7\nBI+h+uuc+XVUg1oYgvX4GkMsJrv5GkNs96IvKBQvB6B78WloDv9st36lEojIuoS7tsxF8+YGmwP9\nNG5swDsTMtC0UQESEhR4d1c31MEltG4diMcf98e0aRps3KjCqWsRKIC64gMnIvISPnsFb1SpIBRZ\nXj4a6osPB0smjjympz71h81l2i2bcGvwUGmV6k0/I1stCK+/XoDer7dCTIP/IOPbWPgv/xIBb09G\n/HeZOHNGgYQEJeLiVFhwKAaXkIH6XTVo0sSApk0NaNpUjyZNDKhVy+hYizuHzCWiasBnE3zq5RRE\n1eQjU+5ilJhR1fHHAFgP9HNXZApGfRCGvn2LoHzpisWgOgoYUbeuEXXr6hETowcAaL9cC9Xrb+CX\nxalISDAl/uXLNUhIUKCgQECTJqYm/iZNDGjSxJT4dTrLWDZvVmEJTiChfiQaNTJg7NiCKr3/T0Tk\nCN9tolcq3R0B2aH49x+rsiFYjxNojUJocPjJBabk6sDVtD/y0bKlAUOGFGHatFvYsCEPp0/n4Jdf\ncjB+fAHq1zcgPl6JKVO0aNEiCG3bBuLJJ7WYOVODt97yw0sv+eMkWkKvF3DmjBIvveQv+qieHDZv\nVqFr1wCoVLD5iGBV1F+zZpBb6+fx++bxU+Xwp0QeSfWPtGlqlWfPWBZUond7VJQRUVF6dOmiB2Dq\nO6DXAxcuCEhIUCIhQYF168Tv348ercWKFQYEBBgREGCEvz/M//v7GxEQYPq/pLzkffnysuuXHwpg\n82ZV8SOBJiUnF1aPCMqE9bN+d9ZfEsOCBRqcOwc0ahRQ5a1npfUrqlXrHRM8VW96vbT1HEz8SiXQ\nsKERDRsW4X//AxYs0IiuV1QEvPvuLeTmAnl5AvLygNxc0/95eQJyc4H0dIVoecn65d/7+aFM4jfi\nyhXxhrbx47XYsqXI7uGJlTtatnev+J+JN97Q4ocf5P8jZ6v+CRO0iIsrgiDA4p9CUfLaaFEOlF0m\n/d+6deL1T5nih3/+UUChKN2vQmE0v7csN32nTGVGi3Jb65esN326+BMbM2f6oWZNo1WdSqXYvq3r\nLInHcl2jVRzz54t/96vqEVV3n2C4u35nMMGXo69bD8qLF9wdhg+oIOFW2PRuY/v8W5Xcn32NGhlw\n5oz1bZ3GjQ1o107iSYYERiPMSb/kpKBLlwDRdXNzgUGDimwemjMT+JUt27ZN/M9EdjbQu7f8f+C2\nbhWvPysL6NnTdPwl/wDT/waDYFFe9p/BULqereWm16Z93Lgh/l1LSxNQWGhav/Sfwry9wWA6/yxZ\nVlJmMAgW25SWo1y5ab0rV8Trv3hRwPTpGvOx2q7Psk6j0XI9e7EZDICNwSGRkKBAzZpBVicEgOX7\nkhOf8idh5dcpf4JWss3Fi+InuGPGaLFkicHmiVmJ0jLrEz6xbcq//u038du5VT0GR2UwwZdz4/Bx\ndr6rJhTXExHeo3SQHf9F8xA0fWrp8n//gW78WORMfMvpusaOLbA4iy8xZkyB0/suSxCAgADT1buJ\nEY0b2z65qIoEO2+eeP133WVAnz7y1z9/vu36Bw6Uv/5vv1XZrH/iRNf+/MV07RogWn+TJgZs22Z7\n9kW562/a1IAff8y1OJEQO1Eqe7JSfrnlOqUnX2W36dZN/AS3sBCYNy9f9CQNKPtesCqzXG57ewA4\ncMD69x4Azp3z/C5sTPDlsfNdtaC4eQN+W761KCub3AEgol1r0wu1Crf++5hT9ZnO1POw9KVzSEBT\nNGqqwJgxIvfhbt2CkJEBY3S0U/WVVVUnF6yf9Ttav1ry0BKVaUEzbWOv9axVK8emua4MWyfYjRrJ\nX7ezPP8UxMVu9X7Uqe31tW93USS+Tbv+K6e291/5meR1NXt2mW6WOyHw/XfR99Fb5l78+/blijbP\nBb07GZHN/2NVrl2zChH1a1Wq7r59i7B8eR6aNciGCoVo2lSP5cur7v6fuf5Gt6BS6N1Wf9OmeqhU\ncHP9Rh5/Fdc/dqz4iUxVnuC4s35n+NwVvNHZIUbtzHZG0mk3f1vxSsX8F84VX+DARYFuwmvSVxYR\nsHg+ckeMrnA9RZL4pDaqo79CkZNd6fr79i3CwAbHEfZgV6Tsq/oJkfr2LcLjaZ9AN3mC2+rv9+BN\nRN7KREpEbbfU7877rSX1R0XpkJKS67b63aGk9WzhQg3OnVOiUSO9eOtZldRv6kVflfU7g9nKUZxk\npEoIGRnm10EfvCe6jmbPjy6pK3DaOwjr1hEA4L90kfm1Paqjv4ov4PdDNoHvTAaaNHFb/YpLF6E8\nIz5kMsmrb98i7NuXi8JC2Gw9q4r6r13Ldkv9leWDCd76ss8Q6sjkJNZ/wI28qnc5IbPiq0TN/r0u\nqUu9b495ch3N7p2WE+2UVearo8hId6wSb0j8bj4GITvLrfWH9vsvwru2d1v9gW9NAObPd1v9mp3b\nobh8yW31q479BhR4frO4J/G9zFTumaBbMb1wq99Ap3ZpiL7Nqe1JRBWO9y6U1GUwQMiSlkTKDrUb\n/OxwKM//JUdolrzhJMEZ7p4CwMl+HM4KWPEJMG+e2+oPeWIwAqe5b0KusEd6Ap9J73tDvpjgyylq\n0cruH86CTp2tygrbdZAzJAKgmzjOuR3kVnyfMqTffyGkpZnfa/b8CPWJ4w5X5bftOwQPKz5JdGTE\nmWrHG47BCZ7wM3T3REfurp9X8A6RNcGfO3cODzzwAL788ksAQGJiIoYPH46hQ4dizJgxKCj+YcXG\nxqJ///4YOHAgNmzYIGdIyB/6JPKeeAoAkDV7AXLHT4QhJMTm+sawcMsCAUiP3SFniOQCUfVqiJb7\nL15gfq356YBFc7xmx3aH6xGyTLcSVCJj51uu6MLkYLDzeE7Jw8VycHeCc3dy8QT8DMgBsiX43Nxc\nvP/+++jQofRqd9GiRRg6dCjWrl2LunXrYuPGjcjNzcXSpUuxatUqrFmzBqtXr0Z6uoP3Nx1Q2LU7\nsuctBgDkP/UsIAjIfW0C0n49Ib5+67stC8oPkwQgd+IUWWIl1/P7bpNlQZk/mP7/94UDezJ9B/y+\nsZ7CVnx11yXHkL69bS4LfuFphHWvuJNgdSTIdeIiOQAPuIJ3M8HdJxjurr+akS3BazQarFixAtFl\nBvw4cuQIevbsCQDo3r07Dh06hBMnTqBFixbQ6XTQarVo06YN4uPj5QpLnJ8fDPXqm98WdOsBANDX\nq4+8UZaPV4lNc5o/dLi88ZHLqP/4HUFjR5QWOPIHo4J1Ff/+I3kaXGdoDv1s8ypddeQQVHL19Pb1\nBOcJx88ERw6QLcGrVCpotVqLsry8PGg0pokLIiIikJKSgtTUVISHlzaDh4eHIyUlRa6wJMl78ZXS\nN+V/qW38kqckZYiWk+fx276t9I0jfzDLTmwj8j0wj5wnwn/NKun1SCDk5thYIP79VP0eD81Ox29B\neBQmN/dz98/A3fVXM24b6MZo4wdlq7y8qCidK8OxFGIa+1ipVFjVo1IprcpkjYVcTnHzpvlnFlqU\nCyitk6LYzzSyxZ3m16GhAUCUDsixvJ2k9VNZbi8IQJl+JU59V8ICS2OJ1AE6kX0pBPF6Rr4InDvn\n3B/IYH/xfZf1119AYCBQq3Kj9tmlUVZcv5yUCvfWDwBGo1vr99NY//2rUm4+/uqmShN8QEAA8vPz\nodVqkZSUhOjoaERHRyM1NdW8TnJyMlq3tn0lVCIlRb5nYjWZeQgBoC/S40ZKFqLKLMtr2w7Z5cpK\nYilbRp4tpeRnOHQoipo2t/pFKP/9Kv+zTc/IQ2FKFqKmT7cozy/QQ1u8vfrQzwgFkHPsBEpSszPf\nW9XNHJRMg5SamgVj8SxfUdHBuLH/MPRNmiLcCChF6gnTG6Bysn5tVj50ACAISEkWH6cgqlEjFDVq\njJs//VbpemwJzi+EH+T93bcn3GAU/WyrShQAGI1urf/WrSJkuvP44b7P390qc2JTpY/JdezYEXFx\ncQCAnTt3onPnzmjVqhVOnjyJzMxM5OTkID4+Hm3btq3KsBxiFLtqompHs3WL+bWtgW2iooOhtjFa\nXuBHH0Bx9YpVubZMJ77Qxx4x1RX3gzOhijKWe2QtcM6Hphce8JiekCf/DGfu4QH34N3N3U3k7q6/\nmpHtCv7UqVP46KOPcPXqVahUKsTFxWHOnDmYOHEi1q9fj1q1aqFPnz5Qq9UYN24cnnvuOQiCgBEj\nRkDn5iRqiHLdTGDkmVSnT0paT/nP3yjs8aBVufrYb/D7brOkfaiP2+80KmRmAAoFjEEVfO/tJem8\nXLsjvalcMRCP1JMEkT/Cml1xMAbpUNjeiR7+xfsNfaAL0ncdEF0l9MGuyHtlpNODV4nyhPzOBEcO\nkC3BN2/eHGvWrLEqX7lypVVZTEwMYmJi5ArFYUUtW6Oga3cU3n2P1TJjgPjcxORdzMnSTlILmDfL\nJXWFde0AY3Awbu4/7PC22lWfAwD8du2E+r5WMKo1lvtu1xoZ38nYuc5ohHbV58h/5nm7q4UMHQhD\nRATSzvzrTGUATE9C2KI+cRz6H+PkSfDFtGtWIX/406LLlH+ehf4/d3LaafIIPj+SnS0ZG75D7mTL\nYRlv/PQbcse+IVudTs90Ry6juHy54nUybT85oXv1Bcl1Ka9eMT3alpNjGoEvW/qsc/6fLy+Np0xf\nlhKqf/+B8s+zkvdnl9jJjl4P3ZuvWw7Va+sq09mrTye2D35qKPy2SJ/BUFTx8evG2Z5VMLzzffDb\nIHFshMpITobupWdsLg58/10gP1+26tX799qeaAmA8tyfstUNADh+HMKNNJuLSwaeIhMmeAfoGzUG\n/P1l23/W7AUVr0Quodmzy/4KTo47rt243uFtFOk3EVWvBqIaWPZAF9JvIio6WHyjwkLLfaSKPGLq\nqmZdO60Zws0brq9PAiEpCWHtyw1GJVK/3/Zt8NskwyiZ+flQHT9mGZPIMMmB096B4uIFl1RpMdWy\n0WgxJkLA4vlQ/nXOJfWIUeRkI+TJx8UX5uUh/P57ZasbALB2LYLeFB/GWnXiOCIb3i5v/dUME7yT\nCjp3M7+uaIz6ov/caXc5VR317/bHnBeKm4N1k1zbYqM8ewbqX34SXaY6+YdoudiVud/unYBeD9U/\nf1uUC2USvv/ypaYXcibc4vpC//cwAidbf1aBb08sbZGQ4Qpe9ecZq8/Ake2d5b/iE4Q93L3C9QKW\nLJDcZ8MRAR99gMg6FfcZiooOhkLq5+QAv2+/Md8msjdEcljX9i77/IUy49ErT500d3YtO6+EhVu3\n4L9ssUvqrm6Y4CXIXPSxzWUZ38aaX+fZuC9XQl+3HgwRETaXW417T+4jU1IM79IOoX16AQBUvx2x\nWBby5BDJ+wl+4WlodtjvnR/09iQAgN/3Wx2M0gaRK/iS6VMFgwF+O+OslgcsXwbh1i3TGzma6IuT\niurwIQjmE6HS9YSszNJyJ+sXG6VQkWZ98mW7Htd/p9S/x1skPKD05LQ8RXKyy+sPmvA6dBOKR/u0\n08Ij1+iK4T06IWTYILv1q/74HUFT35Klfk/HBC+FjfneC5u3tHh/a1Bp01VBjwfEd2XrLBNAwUOe\n09HQ51VBM3NY7wel3S+18Xcz5JlhkupxbIx9xygvlHaaU166YHpR/rNTuKj7ucjPJGDhXABA2KMP\nW/0RV/z7DyIb3i75iYnKUJ00zWERcWcdqH4veVrCFKdw8wb8F80vHUFQhu9Uya0m3ciXrGdQLCqC\nkJYGVfxRl9dbQlF8zzvgw+k2E6yiZCImmX6nVAmn4PftNzbrV4o8zuormOClsPHFNNx+h81N9LeV\nzmZW8thdhRM1SHgMKWvW/ArXIeeVTVyykumPnq3bAGI0P+6QNJ+9ssw9ZP/FC+w+lgfA+t50QaGN\nNaURbllPFar5+WDpm5LPsvj/kuRj6wTd8QCsfz81B/ebqshIh6rktk9x/Zq47Qia/q7NWy+upP3m\nayjLJdKAhXMR2aQ+Ame8D8D2lb3DRD6HwHmzrMoD5s1CyOC+CHuwq0VcctQf/MrzVuX+C+dC+/ly\nBL9ou1Oit2OCl6ISTW7GMs/SZ3z1jeti8YQJL3yA9v9WmV8HPzXU5fsv6e2rPiQ9ETui5DaAFCHD\nBiG8o/UjoeWVbaIOeh3DwvwAACAASURBVP8dhLdoLLqeMvEaAJjvTZc8UWBz/HyJNAf3mV9rv1xt\ntVyRnCS+Yckja04mmArv9Zfbv3n2O1edYDhIce1acSDFBXK3SpU7wfL7bhM0e3dDKHLuxM6mCjrC\nBn3wHoLenypP3dUEE7wz7P3ClE3ExesZAwJtrOwAJvgqUTaZWExO4yK60a8CAILetX1vUHnqpOV3\nLMe6d7ajwrp3guqY7WFkg597Ev6fLrMoU1y6iOBhA62e7Vbk2Hmcr0yHK7/d4qMBOkP3+iirFgLN\n/r2mF0YjhOwshD3QBQCg/Ps8AEB1ynVN9SH9H7UqK3mCwdxSV/IZyPE7W9KvoaxyCdZtrOoXbJS7\nuB7+bbTCBO8MqV/Y4lnIDOG2O9hJxi+xVxDSbwIAVHaeUQ/v0QnKs2fMP/OwRx92ul7V6ZNQ/yze\nahAyuC/8tm5B0JSJFuXqnw/C78c4QCV9XKyQx/tLj+n4Mev7xxLY6r2u/u1Xi6t583PrLpxPXnNw\nHzS7d1qUaVeX9nVQXE+E7vVRpnW3fgfAtXOpR90RBaFcpzn/FeU6Axd/b1TxlidCzlKkJEO9d7dV\n/xHNruJOlubjNNVvbrlxVS/6mzdMT2aU25/qjxMu2b83YYKXopIDd9z80XR/DnrrPyyGkFDxbXbu\nsx8LE7x3kPhztNns7ATtRvGBWDR7d4tvUBKrA6OzqY8ckrxu2MPdEVWvRsUrSqS8ekU0qSmTrrus\nDgAIeXyAZUHx5yRkZUG9b4+5WF3cEc/VV7CRzf9j8V5bMpCP0Qjt6i/MnSvNfRFcWH/o4L6IuKe5\nRVnIc0+W1r9mFVRnTrusvrI0h35GyNPDEN7Jcs6SoGlvm1+XPBbn7G2h6o4JXiZGQUBRK9MAHIJB\nb72CjZ7FRa3b2N+xSGJI+0Pm0aOoyilLmpMFwWpiGWepzp4xvXDwD75RIT3Biw32AgDhze+EkCR+\n0qK4fAkhj8ZAXX4Qolu3oLh2VXLdABA4a4bkdYOfGOTS56QDP5wOVfFtAYvyme+7rA4xZSf5Ebua\ndfXTBIoUG4/dGY1Wt3kAQLjluhH2NAf2WnT6LM9XH4srjwlegoL/Pmo1ylzWzDnIHT/Rxhbl6K0T\nfO7I18yvDXYmGSk/8Y2xXDOpMSAQhho1bU7f6Qj9HXWc3gdJI2TYHuYWMI1qZ3ohz6+o4tJFRN0W\nIsu+7VEmJ0H1r3hnteAXnoLm8C/wX7nCojxwzoeIaN3EoXqEHPErN7FhVv127jAnBPXe3UCBdW99\nScr8bvqVmVWwIpH1aiBo/NjK1Ski7KFuUJ2zvvUT9NabousLGekVPhHhUP0PdBb9m+e/ZKHL6rAn\nYPZM0XLVYemtSt6CCV4Coy4Y+U89a1GW/9yLKBKZjEY00ZZ82ctcfRuiTLMbZ3y+BtnzFiF39Ovi\ndZe7YpdzshtD9G0VrnMrprds9fsSc7OtLSVX14Xy9EAuGfxGyuNxrhY4xcaJcfF33S/OcnIcQWz4\n3QrYuroM6/WAzd7Xfuu+QujgvvD7PtaiXEi/Ka0FocwtDEFk9EEACJg/26pMyM01N6cHPz3MJWPJ\nKyTejlBcu4rIO+sgtJdp3A5bcTtCdSahdHCjMgSRTpmKf/9BVHQwtF+ssFpWWYE2ErwiI9267OoV\nRDSuW9pi5mWY4OVSthN9WJjIctMKBf99FLf69EfOlKmiu8la5rovfkUKHn6kwnWMAfKNxU9llEyN\nOrivLP0ugt6dDAC2H48zGuH/yRKLIuWFf1xSt63Z4NTH5BuQpSyljVHVSpqwFdctk2PI8CGSWhCE\nG6Xj8StsXBHba6ZX/XoEfj9stUrOiiuXJSfsErbGcSg/p4Fws7iz59kzEJKTEdm0gUP12Kz/8iWr\nMvXRXxEw9yOLspInH/xXroDy5B+251xwRUx/nSszGJFJ4PSpUNy8Cb+tm6HeswuhD3WVrX53YIKX\nWcq/iShq2RpZ85cgd0SZWahK/mhX8Me7sFPnciXydbLLHTtewlrs5FcVAhaVDmgU9N7bdtaUh+Li\nBQS9M9mizF/k2fPKUh/6GYFTxJuM5RY88kXR8oDlpvvGJSc/JezNXlaWraRenq2mYuU/pvv25fsv\nRLRpZr7CdgkbfS+UV8VnUAzp/z9o11pP/e0o9bGjCPzoA9Flqj/PQvWPdb8FwDSngqJ4bAVnBE17\nG2EPdRNdFjh/DjS7d4rOUaE+sM+lT2BUJSZ4megbF5/xB5qefc8f9iQM9eqXriD1qkxsvU6dzC/F\nxseWU96ro6q0Pl+lObDX/NrvBxeNJe9I/WV6gcvRghD62CMI+PRjWa/YbCoshCZuu9VjZja5uPe7\ndvMGy9n3igWPfgWAqZNeeRX12XCExeN9ZY6t5LFDxRXLRK85uB+abd+5rH6h7DTLZerXjXrZqgww\nzamgXfeVy+q3UKaugBWfiK4SOuBRh54K8SRM8DJISc7ErX4D7a8k4Zni/AGDLTpZGSKjoG/WDPip\n9Dnmohal4+FnfPGl+XXuGOspFbOnf1hhnWm/2r83bIiMsiorqu+aZj3yHBbT6brj0cy8PAROnyrL\nrlXn/0LI8MGIbP4fRNatuN+Jy+kNiGxcD+q9u0XveQsi94pdeZIhpKcjKjrYMtGWoT70s8vqEqNI\nSkJ4yS2PMsclFPc9UFy6KHP9182jK0rm7sGDKokJvooVdO8JACi08zhcSc/58vfl0xL+hr5B6bOv\nOa9PQMbG0g5B+rr1zK8L773Per+6iq+WLFoZJChs1wHpP+5HUcP/VLwyVR9yDS8qUeAHUxGwaJ7s\n9ZR9tMyCwQAhxfHOfVKoTpiagUMH9xW/5y13c3DxCZvmh20I79GpgpVlqP7GDSivXYWQkgLtxvVV\nXr/654PQblwPIf0mVAmnqrz+qsQEX8VuPdYPBZ27wVC/AVLPineESTt9HinJmTDUqg0AyH+sn/jO\n/PwAjab0fUVXWjaWZ366Eunrvq0wdrF9GLVaGINDcPNQvI0NpMt+p+LnhAs6VP0fJF9U0kQf2vvB\n0klMqlDAp2VGZXPD1ZP/x0sQ2ayhPDuv6NfUYIDyn/Ols9ChzGA1LqnfFEDJLQFRRUUWT3CI9Yp3\nVmSzhlCLPLYo+vN2ZStS8b4iG9UtHRPCSzHBV7H8ocPNc8gbJQ5da7Qx6p1dZX4h0n63/yUu6Pkg\nCns86HgdDtLXqGl3ed7IMbLHQNIIxY+SqX87gsByPZ99QdB7U8yvy8+37rQKRgQU0lIRPGwQQp4Y\n7Np6pTIaEfJ4/9JZ4FA6a55LSEjWupeftXiKI3DGtCqtP2jsCIvOkLrRryBgpgtjqCJM8N6k3Bc3\n+wPTH2ZjUJDNTTIXLoPRTtN95qcr7dbhiIwNtjvqGLVaSfso7NCx0vVT9RPw0QdQuLCDWWXYbMav\n9A7t/9lV/X1edCQ819Vv/3c48MPpUB07Kl/ztY1RPEtoDu6HdtNGaL/+0u56clEkXoP/2jXQbigd\n0ll56SIC589xSzzOYIL3JuWatvJesNMEV+xW/0F2lxsquOp2KOEXdxgsatrcalHJsL45r0+wu4vc\niZaPjBW26yDa8Y+8Q+Dcj6wGnqlKmm2xLn9S5f/bu+/AKMq8D+DfSTaFhIQUEjoREEITKVJCL+GA\nU3oLMSgevHfKIdiIEOGQlyMk3J0HKCdI8b0DA0S6d3AgaBTOAK/oCwpiAUGKkmSzJdnN1pn3j9ky\nuztbEpKd7O7v89dmdnaeZ0LY3zzt94h2S/tR+DXxPAC29+/c9nnJX124m9xnFXHhHF8PD6loH4iX\n5FG25E8Sz0OpDxTgA4AxYzDYpCSv5zGcYHKO2JeS4Ji1de8whu8LhkH5jz/bfjT16m17rct60utn\nAXgcU9UuW+H2PTHmtu2gWbq8Vp8hxFfNfpOD8AbY8MdX8XPrv5u+Ni3R+shs5ywhy8tOg5bvB3f7\nGTyo+N+L50FwFnHuswYp358owAcA/fRZkF+7WfcLWAIrF2vvqufqmOOcbRpnW9uvPPQvaF5bZXtP\nOJNe8cFJl8+6m1zEpqTC+JjrrH9fcNHRYNu1q9NnCWnsnNP2+lt9ZbarFUHLOSFzuP/Lt6xikP14\nA00ECacCke8bPJNGz92uY1xEJCo/uwhzx05gY5siTFMNXzLSld9XOSzZccmzzzCOPQWCsUVzly4e\na+r2ms5nhoWB8bBsiGuWAEPmOJTfuo8UKdY0E0LqVfTB/bbX7tIaN6SEmZNtr5v+cZWHMxs/asEH\nMDbOaRc6QbA1p/Hr2cvL1EB0NMwPd+bHwC3nmDv4sN6dYTzP+HXuahf2Cgjqot72P/zp1geAWix7\n0uXM8/i+bQJhE8qRTwghQhTgA5T83JfQzVvgcIyL52fDl5epYe4s3oJWfnACin9/BOPosSj/RSRj\nloV+jH3ZnKlbd/GTnMf5nQK8dhG/Baa5RSvH82sR4M3t7VvYVpZe9PlzDUVZfNjhZ2O/xySqCSHE\n3xpiTkJDogAfoNiOnVzS3bLt0xwmwIkx9+gJU19LUHIzDl9V+Aa0r75W+0o5LX/RzcwCAJgGZfAH\nahngKz89D8gibD+bO3XmXxQX248Jsvc9CDbex73RIyIcftTkvgadl5UIhBAiBQrwwcYyAe5B6J5Z\nAJMgla6pczo4X7rAPUzc0z7/Itg2bfkfOA41c3K8Xs7ctRtqnlng+saMGQCAiu9uQT/DPsvYbMn8\n5451tj0XFeXyHtuuPcpve09NyqY6jfMzjMf7JoQEES9r+Bsb+mYiXlVtfgcVV657PY+zjNebuvUA\nF+P4oKFZuRqIiID2v55FzcLF0OU8De2C33kv3PJg4fCAYV0VEN3EYZjAdWtdR9oXl/IvRCbtmdun\nAVFRXif9sc5ZBRkGhmEjvD5cNBTjgEGSlGtFexCQkBJgD/OBVVsijagoQCQbnkvq2Ui+Zaz4pBSI\niADbsqXLZzRr10P35FMw9R8ITf6fRIszdeuBqoK/OB6sZX7qqg2bXQ9ahzTMZtuhim9vQj9mLLQL\nF7u9lgOR/+D6rCdR6SUdcL1JdkxvrHqvuN6GKerCONjzQ1VDq1ov7TImVdH7kpZf/Zq0s7ytG2OF\nioZam99QKMCTOikvU/PzAAR0s7Md1r9ziUleW8RiFJ+UQvcb+3aOxv4DYRw8tFbX0M1wnyBEuOyO\nS0yCes8BmAb62BKOcJz3wNXDkEitvOuYOpiLiHTpLfEn7YuvSFY2AJjTu0pbfscG2pDGR4aJk72f\n1IA0uXmSlm9un+bfAi1b2gYKCvCk/kRG+h4oa0H5wQmo9ojsdufcghe28gUZ+tR/21ZvdeGaJUD9\n5hYAgPzzr2CqY4IeZ6aHO9ftgwwDZfFhyM99WS/1qC22rbRJhowS7y5o7iBtgBduHy0F3exsScuv\nzl/v3wKpi56QeiZYv+/AU4AHoP/VeACAYdhIh+NsfDOYOviWocssEsCsSxBZp9aDt5z4NfP5FJna\nJS+7vKd9KRfld+WoXrHap3rZhIeDa9HCpTeloZjbtXc5piw+jKp14sMtQa+e89QHHJm0udI8bZTV\nICjAE9Lw5F9ccVmy5syQOQ4AwDklBNJPnurTF7Pi1KdQ7T0IAGCTklB+x/MaWOGOeCaRgFu9zpID\nXDAHwDo5zzB2HBARgZrFL3quVEaG7WV5mbr2ewk8IP3kaS7HjCNHQy9hS67i+h1bMiUpKA98ILoy\nw1+0v1soWdkID4fJTc6Nhmbs2QumTp3BNm/utzK5Zj4up20kKMCTgCTaNWxpwVvH/XWzs6HcewCI\nibEfmzYD+mkzUZ2/3usELdMjj8LcJd32ujbBVOGpy1wwB6Dy/75BeZkanPPsfHeaN0fVurptW1mT\nPRcAPzHPWcW3N6F+e7vXaxgECZCE8yvcpUmud8Ptuckrvr3Jlx0X77p80Q+siZeMw0ZA88oyv5ev\n+OeHAADNmgIYe/bye/mqovcBhoHiP5/7vWwA0KxZBy41FfKrN/xWps//TxsJCvAkaNT89jloXn7V\nfqBJExhHj3U4p2rLThiHDINx9Fjo5s13ey3DiFHuW/m1PO7SrW1pwbNOM+KduU++4z1RECeSYrja\n8kAjnBhmGDmaPz++Gb+p0ZeetxI1DhkG3aSprm+IrLJwe42+/VyOVXx3y7dW8N/+ZnvJJdp3WGQT\nEn0uX4yvrUDdVH4nNHPbdvbES4BDQiZ/0E2dDtOAgfYDgt4jf6henW/rIZOC4sTHXpfF1jfOz7/j\n+kABngQNU9/H6paBz4m5bTtUvbXV8aAgpnKRbgKRJcBXWlo01kx+1avzHa+fZhm795LRz9wlXXx9\nvQ+ZACvFWlWW+jHV1QD4BwhV8WG+JW59IBAMH9S36j8WAAC0gu19TZYeEi4hERU/lUHz0lLPF+nR\nQ3xehLt0yiLElhXKr95A9X/nu57spGqrZRWD07+Bbu7TPpcvpvxepddz2ORkQKEQLb+68C8in7DT\nT5zi8X1vD3YA3wNUeYbfy55zWk3ibe6It5wXql37vJav2vEP4KOP+B+cxsLNXnpwaiyrclzyWFho\nfNimWnH6rNdzGhsK8IQ4qfziCtgW9jX86q07Hb78zd178Gl0nZhbtwEXFmabhKfJzYP6nXdh+NV4\nhy8wrlkCqtb/1T4m74bqvWKoDhzlr/XKMluXOONDgOdSRCb8WRMENYkBABgHu85AZ4wG22vNS7mi\n1/alfLXlAYlNErSyLS1uYx++Ba9ZuhyKMxfs6ZUZBmzrtm6vaWvheymfjW9ma2nrx//adlyzfCVf\n/qDBAPg19FVvvAlTj0cAADXPLoJu2ky31zUMG2H/wam3hhP0thgf7WN7bd2PAQBMTkv6rHUEAMhk\nkF/+1uN91Tw9H0hIEC3fLJg0qp/whO111Z828PXzMueEbdMWqp27PZ6jnzTFvizR6XomwZ4Mwt9h\n9dpCvn7p3fhyYsV7egzjJkD77CKP5RtHjgZGjRItXzffvse7btYc+3UtPVT6J/jlhEZhr4e17l27\nQftSLgxeluK629+jMaMAT4gX+qkzYO7ew36AYWDu2s3lPHXR+5ALMv6xaQ9BP2U6EBkJw7gJAADl\nwX9CP3kadPPmQz91htsyDaMzwSUm2buBhTsFtmzt9nNsYiIqrlwHFxcPo2UJn3rT247XkPGtdU5s\nCMBospfjlKXO16x1XGSk/bMO2wlbHjCS+KEJLiqaPybMJSCYn6Dc67Q00hrYvT1gyMLBxVsDof0r\nLvKjUwCA6tfXAuATKulynobi4//Yq2iwP+Bon3ve6cK+zTNgW7WGqRv/98IKhhFk314DAFsvRdXm\nbSi/cc82l4B1ThzlzEOQ5sLsQzJGwdJNU7rr36lVeZkalWcuQH7xawCA4YlJnsv38f5rBJtgWVew\nWLu3+a2q7eWr3iu2la/x1oPi44qFqo32YRxrz4W1JyzqwxMO5Vet/yuUh44BAFSHj/l0/UBCAZ6Q\nesI1jQPnZVzdOHS418l6bPMUqLb/w+ni9qBmeGISKm7cdfmcfuIU6LKfsrXeVUXvQ375W+iznnTo\nhufCwiH//CvXbIEAYDTarzdjNhRH+S9E1Z79UJR+AQAeZ00bRo6G/MoPMPXrD2OfvjAMGwHV9r9D\n+9vnbBMWrUw9H3G9gGCIgIt1XP1g6tHT8ob7AG9u1x7Vq/OhXbSEPzUqEmxCAkzdesBs6ZWx/RuJ\n5RXX1dheauo43GMcNNjeirQEJeG8A23ua/b3mjZ1mEsg5JIkylOAE8y5YNu2hbFff4fPcIKeFKVl\nZQjAJwpiRZY+AoD8fy+7L8852AvqxgnmY7CtLA8tgr95YXe9Yex4t+VbW/+268L1YVH0Z8FrU29L\nb4pgxY0wOY5u3ny3/2elzBBZX6RdxEgIcSG/6pr33yElKMOAaxrnco56h+NDAZeQ6DIdr/L0WbAP\ndXBbtnXLYeX7RwCGse0EyAlawtrcPGhfcM1gp5s8DfppM20zjZUnSmzvGSZNBaNU2H52l+GQYe0B\n3jRwECpLSpE0MgPyS9fANktACgDV7mKEqVWin9csXW5bsld5+izYtm35PQsARJ7+ENFHDwEAFMdO\nwWQNgsLyhZnKBEGpZk4OjKMz+fv/3e/t8yicKI6ftl03/Pr30P/6CRhGjgabkorYv65Hk53bgLAw\nPm9ALddUWydHmtunwThkuOOblgBfceMuuNim0E+ehuYdWsH8UAdU/WkDDGPGIvLUSYTfuQ1jxhDb\nxkuesMmOEw+Fy01dMvhZgqr8q+/AtmiJiht30bxjG3DNElCTPdchF0XN8y+C85IzAoCtF8S5DMD9\nWLr886+AsDAojp9G4oQxMD3yKAwZQ2xDbqaHO6Pq7e2IuHDOa/m6OTmILfij1/MaM4bjarE5dyNS\nXl4ldRUkk5ISR/cfQvfPyOXgEhOBsDC/3DtTVgYu1f5AkZIaD+W+QzCOGvPg11YqwHmY8R5280ck\nD3iUH2aw9ESkpMaj4vodcHHxHu8/7Od7/Be5m8DJqFVosmUztB7Sq8auXI6YrZtRfl8FMAzCv7mK\npBGDfEq5HP3uduiefMptD03EpyWIXbMKyg8/cXuNlFT+AauypBTm7j0QXbQLcS/8HuV35YBMhpTU\nePH75zgk/HoMlMc/cnvtmII1iDp6GIrPLnotX/X3PTBMeBwJE0Yj4uLnkP/fN2A9bKjElJUh8fFM\nVHpo9SdMHAeOYaA6+m+v5au37oR+ynSktOCHkSo/PQ9z1278v/+9SpccGLLPLyDu5cVQfOI+cDdv\nlQjdnBxUv/Gm1/KVew/COHgoUtqngk1IgOr9IzAJ5lZIISXF9aHeG+qiJ6SR45KT/ZpBSxjcAX4v\nAFP3nvVzbS/L2diHOqDi6g3XSYI+jL+yrVp7/D1x8c08BncA0Ly2CsaBGfbyRHYedEf3zAKPwy/G\n4SM9BnfAvrbdNufDZJkTERHh+XfAMB6DOwBol630GNwB2NbzGyY8ztfDMgfEU3AH+L8ZT8EdAJSH\nj0F16F8ezzH25yfB6afO4Oe6WLrJHea8iCS4Mj02wGNwB4CKG/dsS0U91qFnL763xrosTiaTPLjX\nFXXRE0I8Uv7rQ7+WxzmtSVft2S86JNEgoqOh/MA+EYtt1cqvWepMAwY69Bawbfy7DbF26XJoBasn\nvKVfrhWR3AzOlEeOO8wDYZOSEH7rZv2UL9xy2o3ym7+4PEi5Gw4IBNRFH4BCrYvaWSjffyjfOxCi\n96/X81s2Q4L71+sRplQ4LBv1J6ZSDkans/Ug+Pv+w27/BC4q2qVXSwp16aKnFjwhhDRmEua5R1SU\nZMEd4JdUStkCdTfDP1DQGDwhhBAShCjAE0IIIUGIAjwhhBAShCjAE0IIIUGo0Uyyy8/Px6VLl8Aw\nDPLy8tCrl//3NyaEEEKCRaMI8BcuXMCtW7ewb98+XL9+HXl5edi3z/v2gYQQQggR1yi66EtLS5GZ\nyed57tSpE1QqFaqrq718ihBCCCHuNIoAX1FRgcREewrLpKQklJeXS1gjQgghJLA1ii56Z74k16tL\nVp9gQvcfuvcfyvcO0P3T/Yf2/ddGo2jBp6amoqKiwvZzWVkZUpw3myCEEEKIzxpFgB8yZAhOnOA3\neLhy5QpSU1PRtGlTiWtFCCGEBK5G0UXft29f9OjRA1lZWWAYBqtWrZK6SoQQQkhAC9jd5AghhBDi\nXqPooieEEEJI/aIATwghhAShgAvw+fn5mD17NrKysnD58mWpq+N369evx+zZszF9+nScPHlS6ur4\nnU6nQ2ZmJg4ePCh1Vfzu6NGjmDRpEqZNm4aSkhKpq+NXGo0GixYtwty5c5GVlYUzZ85IXSW/+O67\n75CZmYndu3cDAH7++WfMnTsX2dnZWLJkCQwGg8Q1bFhi9z9v3jzk5ORg3rx5QZ8vxfn+rc6cOYP0\n9HSvnw+oAC9Mabt27VqsXbtW6ir51blz5/D9999j37592L59O/Lz86Wukt+9/fbbaNasmdTV8DuF\nQoHNmzejqKgIW7ZswenTp6Wukl8dOnQIHTp0wK5du7Bx48aQ+L+v1WqxZs0aZGRk2I5t2rQJ2dnZ\nKCoqQlpaGvbv3y9hDRuW2P1v2LABs2bNwu7duzF27Fi8++67EtawYYndPwDo9Xq88847Pi0lD6gA\nH+opbfv374+NGzcCAOLj41FTUwOz2Sxxrfzn+vXr+OGHHzBy5Eipq+J3paWlyMjIQNOmTZGamoo1\na9ZIXSW/SkxMhFKpBACo1WqHzJfBKjIyEtu2bUNqaqrt2Pnz5zFmzBgAwKhRo1BaWipV9Rqc2P2v\nWrUK48aNA+D4NxGMxO4fALZs2YLs7GxERkZ6vUZABfhQT2kbHh6OmJgYAMD+/fsxfPhwhIeHS1wr\n/yksLMSyZcukroYk7ty5A51Oh2effRbZ2dlB/cUu5vHHH8e9e/cwduxY5OTk4NVXX5W6Sg1OJpMh\nOjra4VhNTY3tiz05OTmov//E7j8mJgbh4eEwm80oKirCxIkTJapdwxO7/x9//BHXrl3DhAkTfLtG\nQ1TMX0J1hd+pU6ewf/9+7Ny5U+qq+M3hw4fRu3dvtGvXTuqqSEapVOKtt97CvXv38NRTT+Hjjz8G\nwzBSV8svjhw5gtatW2PHjh24du0a8vLyQnIehlCofv+ZzWbk5uZi0KBBLt3XwW7dunVYsWKFz+cH\nVICnlLb85IotW7Zg+/btiIsLnZzMJSUluH37NkpKSvDLL78gMjISLVu2xODBg6Wuml8kJyejT58+\nkMlkaN++PWJjY1FZWYnk5GSpq+YXX3zxBYYOHQoA6Nq1K8rKymA2m0OqBwvgW7A6nQ7R0dG4f/++\nS/dtKFi+fDnS0tKwaNEiqaviV/fv38eNGzfwyiuvAODjX05OjssEPKGA6qIP9ZS2VVVVWL9+PbZu\n3YqEhASpq+NXGzZswIEDB1BcXIyZM2di4cKFIRPcAWDo0KE4d+4cWJaFQqGAVqsNiXFoq7S0NFy6\ndAkAcPfuXcTGEw6ICwAAA7VJREFUxoZccAeAwYMH274DT548iWHDhklcI/86evQoIiIisHjxYqmr\n4nctWrTAqVOnUFxcjOLiYqSmpnoM7kCAteBDPaXtsWPHoFAo8MILL9iOFRYWonXr1hLWivhDixYt\nMG7cOMyaNQsAsGLFCoSFBdTz+QOZPXs28vLykJOTA5PJhNdff13qKjW4r7/+GoWFhbh79y5kMhlO\nnDiBP//5z1i2bBn27duH1q1bY8qUKVJXs8GI3b9cLkdUVBTmzp0LgJ9sHax/C2L3/+abb9aqcUep\nagkhhJAgFDpNAEIIISSEUIAnhBBCghAFeEIIISQIUYAnhBBCghAFeEIIISQIBdQyOUJI3d25cwfj\nx49Hnz59HI6PGDECCxYseODrnz9/Hhs2bMCePXse+FqEkAdHAZ6QEJKUlIRdu3ZJXQ1CiB9QgCeE\noHv37li4cCHOnz8PjUaDgoICdOnSBZcuXUJBQQFkMhkYhsEf/vAHPPzww7h58yZWrlwJlmURFRWF\ndevWAQBYlsWqVavwzTffIDIyElu3bgUAvPzyy1Cr1TCZTBg1ahSee+45KW+XkJBAY/CEEJjNZnTu\n3Bm7du3CnDlzsGnTJgBAbm4uli9fjl27duGZZ57B6tWrAfDbds6fPx/vvfcepk+fjuPHjwPgt/R9\n/vnnUVxcDJlMhrNnz+Kzzz6DyWRCUVER9u7di5iYGLAsK9m9EhIqqAVPSAiprKy0pfm0Wrp0KQDY\nNnPp27cvduzYAbVaDblcjl69egEABgwYgJdeegkAcPnyZQwYMAAAv5UrwI/Bd+zYEc2bNwcAtGzZ\nEmq1GqNHj8amTZuwZMkSjBgxAjNnzgypNLuESIUCPCEhxNMYvDBrNcMwLlvROme1FmuFi20Ak5yc\njCNHjuDLL7/E6dOnMX36dBw6dMhlr2tCSP2ix2hCCADg3LlzAICLFy8iPT0dcXFxSElJse3iVlpa\nit69ewPgW/lnzpwBwG+C9MYbb7i97tmzZ1FSUoJ+/fohNzcXMTExkMvlDXw3hBBqwRMSQsS66Nu2\nbQsAuHr1Kvbs2QOVSoXCwkIA/G6FBQUFCA8PR1hYmG3nrpUrV2LlypUoKiqCTCZDfn4+fvrpJ9Ey\nO3TogGXLlmH79u0IDw/H0KFD0aZNm4a7SUIIANpNjhACID09HVeuXIFMRs/8hAQL6qInhBBCghC1\n4AkhhJAgRC14QgghJAhRgCeEEEKCEAV4QgghJAhRgCeEEEKCEAV4QgghJAhRgCeEEEKC0P8DU1Io\nEChmUfYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f218aeb9d30>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:execution time - 0:06:48s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss/perplexity on test set after 13 epochs:     4.74/  114.29\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#MIT License - Copyright (c) 2018 tmatha\n",
    "#*******************************************************************************\n",
    "# ---------------------- set up -----------------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "start_time=time.time()\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "logging.info('date {}'.format(datetime.datetime.now()))\n",
    "logging.info('device {}'.format(tf.test.gpu_device_name()))\n",
    "logging.info('TensorFlow vers. {}'.format(tf.__version__))\n",
    "assert tf.test.is_gpu_available()\n",
    "\n",
    "#*******************************************************************************\n",
    "# -------------- --- hyper parameters -----------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "batch_size=20\n",
    "seq_len=20\n",
    "clip_norm=5\n",
    "learning_rate=1.\n",
    "decay=0.5\n",
    "epochs=13\n",
    "epochs_no_decay=4\n",
    "\n",
    "#*******************************************************************************\n",
    "# ---------------- data: Python lists of strings ------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "with open('ptb.train.txt','r') as f1,open('ptb.valid.txt','r') as f2,open(\n",
    "    'ptb.test.txt','r') as f3:\n",
    "    seq_train=f1.read().replace('\\n','<eos>').split(' ')\n",
    "    seq_valid=f2.read().replace('\\n','<eos>').split(' ')\n",
    "    seq_test=f3.read().replace('\\n','<eos>').split(' ')\n",
    "\n",
    "seq_train=list(filter(None,seq_train))\n",
    "seq_valid=list(filter(None,seq_valid))\n",
    "seq_test=list(filter(None,seq_test))\n",
    "\n",
    "logging.info(seq_train[:10])\n",
    "logging.info(seq_valid[:10])\n",
    "logging.info(seq_test[:10])\n",
    "\n",
    "size_train=len(seq_train)\n",
    "size_valid=len(seq_valid)\n",
    "size_test=len(seq_test)\n",
    "logging.info('size_train {}, size_valid {}, size_test {}'.format(\n",
    "    size_train,size_valid,size_test))\n",
    "\n",
    "vocab_train=set(seq_train)\n",
    "vocab_valid=set(seq_valid)\n",
    "vocab_test=set(seq_test)\n",
    "\n",
    "assert vocab_valid.issubset(vocab_train)\n",
    "assert vocab_test.issubset(vocab_train)\n",
    "logging.info('vocab_train {}, vocab_valid {}, vocab_test {}'.format(\n",
    "    len(vocab_train),len(vocab_valid),len(vocab_test)))\n",
    "\n",
    "vocab_train=sorted(vocab_train)#must have deterministic ordering, so word2id \n",
    "                               #dictionary is reproducible across invocations\n",
    "word2id={w:i for i,w in enumerate(vocab_train)}\n",
    "id2word={i:w for i,w in enumerate(vocab_train)}\n",
    "\n",
    "#*******************************************************************************\n",
    "# -- data: np.int64 1-d numpy arrays -> np.int64 2-d numpy arrays of shape ----*\n",
    "# -- (seq_len*steps, batch_size) ----------------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "#Note tf.contrib.cudnn_rnn.CudnnLSTM requires input tensor to be of shape \n",
    "#(seq_len,batch_size,embedding_dim), where as tf.keras.layers.CuDNNLSTM \n",
    "#requires input tensor to be of shape (batch_size,seq_len,embedding_dim)\n",
    "ids_train=np.array([word2id[word] for word in seq_train],copy=False,order='C')\n",
    "ids_valid=np.array([word2id[word] for word in seq_valid],copy=False,order='C')\n",
    "ids_test=np.array([word2id[word] for word in seq_test],copy=False,order='C')\n",
    "\n",
    "data_train,steps_train=features_labels(\n",
    "    ids_train,batch_size,seq_len,batch_first=False)\n",
    "data_valid,steps_valid=features_labels(\n",
    "    ids_valid,batch_size,seq_len,batch_first=False)\n",
    "data_test,steps_test=features_labels(\n",
    "    ids_test,batch_size,seq_len,batch_first=False)\n",
    "\n",
    "#*******************************************************************************\n",
    "# ----------------- configure dataflow graph ----------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "#A model instance encapsulates all nodes (ops) and edges (tensors) of the \n",
    "#dataflow graph except:\n",
    "#  features_placeholder: tf.int64 2-d tensor of shape (seq_len*steps,batch_size)\n",
    "#  labels_placeholder:   tf.int64 2-d tensor of shape (seq_len*steps,batch_size)\n",
    "#  lr: tf.float32 0-d tensor\n",
    "#  learning_rate_decay: op to decay learning rate by a factor\n",
    "tf.reset_default_graph()\n",
    "\n",
    "features_placeholder=tf.placeholder(\n",
    "    data_train.features.dtype, (None,batch_size))#data_train.features.shape\n",
    "labels_placeholder=tf.placeholder(\n",
    "    data_train.labels.dtype, (None,batch_size))#data_train.labels.shape\n",
    "dataset=tf.data.Dataset.from_tensor_slices(\n",
    "    (features_placeholder, labels_placeholder))\n",
    "#\"An initializable iterator requires you to run an explicit  \n",
    "#iterator.initializer operation before using it. In exchange for this  \n",
    "#inconvenience, it enables you to parameterize the definition of the dataset, \n",
    "#using one or more tf.placeholder() tensors that can be fed when you \n",
    "#initialize the iterator.\" - https://www.tensorflow.org/guide/datasets\n",
    "iterator=dataset.batch(seq_len,drop_remainder=True).make_initializable_iterator()\n",
    "features,labels=iterator.get_next()\n",
    "\n",
    "lr=tf.get_variable('lr',initializer=learning_rate,trainable=False)\n",
    "learning_rate_decay=lr.assign(lr*decay)\n",
    "\n",
    "model=PTBModel.instance(model_type='small',\n",
    "                        vocab_size=len(word2id),\n",
    "                        clip_norm=clip_norm,\n",
    "                        learning_rate=lr,\n",
    "                        inp=features,\n",
    "                        target=labels)\n",
    "\n",
    "#kernels, biases, state and lr are tf.Variables\n",
    "for var in tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES):\n",
    "  logging.info(var)\n",
    "\n",
    "#*******************************************************************************\n",
    "# ------------------------- initialize ----------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "perplexity_train=[]\n",
    "perplexity_valid=[]\n",
    "\n",
    "print('\\n'+' '*24+'TRAINING'+'\\n'+\n",
    "      'time'+' '*6+\n",
    "      'epochs'+' '*9+\n",
    "      'loss'+' '*15+\n",
    "      'perplexity'+'\\n'+\n",
    "      ' '*20+\n",
    "      'train'+' '*4+\n",
    "      'valid'+' '*7+\n",
    "      'train'+' '*7+\n",
    "      'valid'+'\\n'+\n",
    "      '======'+' '*4+\n",
    "      '======'+' '*4+\n",
    "      '====='+' '*4+\n",
    "      '====='+' '*7+\n",
    "      '====='+' '*7+\n",
    "      '=====')\n",
    "  \n",
    "session=tf.Session()\n",
    "session.run(tf.initializers.global_variables())\n",
    "\n",
    "#*******************************************************************************\n",
    "# ---------------------- train and evaluate -----------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "for epoch in range(epochs):\n",
    "  #train\n",
    "  session.run(iterator.initializer,\n",
    "              feed_dict={features_placeholder:data_train.features,\n",
    "                         labels_placeholder:data_train.labels})\n",
    "  losses_train=model.run_epoch(session,steps_train,is_training=True)\n",
    "  assert len(losses_train)==steps_train\n",
    "  loss_train_avg=sum(losses_train)/(len(losses_train)*seq_len*batch_size)\n",
    "  perplexity_train+=[(epoch+(step+1)/len(losses_train),math.exp(loss/(\n",
    "      seq_len*batch_size))) for step,loss in enumerate(losses_train)]\n",
    "  \n",
    "  #evaluate\n",
    "  session.run(iterator.initializer,\n",
    "              feed_dict={features_placeholder:data_valid.features,\n",
    "                         labels_placeholder:data_valid.labels})\n",
    "  losses_valid=model.run_epoch(session,steps_valid,is_training=False)\n",
    "  assert len(losses_valid)==steps_valid\n",
    "  loss_valid_avg=sum(losses_valid)/(len(losses_valid)*seq_len*batch_size)\n",
    "  perplexity_valid.append((epoch+1,math.exp(loss_valid_avg)))\n",
    "  \n",
    "  if epoch>epochs_no_decay-2:\n",
    "    session.run(learning_rate_decay)\n",
    "\n",
    "  print('{:}'.format(datetime.timedelta(seconds=round(time.time()-start_time))),\n",
    "        '{:5.2f}'.format(epoch+1),\n",
    "        '{:5.2f}'.format(loss_train_avg),\n",
    "        '{:5.2f}'.format(loss_valid_avg),\n",
    "        '{:8.2f}'.format(math.exp(loss_train_avg)),\n",
    "        '{:8.2f}'.format(perplexity_valid[-1][1]),sep=' '*4)\n",
    "\n",
    "#*******************************************************************************\n",
    "# ----------------------- plot ------------------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "plt.plot([loss[0] for loss in perplexity_train],\n",
    "         [loss[1] for loss in perplexity_train],\n",
    "         linewidth=1,color='red',label='training')\n",
    "plt.plot([loss[0] for loss in perplexity_valid],\n",
    "         [loss[1] for loss in perplexity_valid],\n",
    "         linewidth=1,color='blue',label='validation', marker='o')\n",
    "plt.grid(True,which='both',axis='both')\n",
    "plt.title('Penn Treebank Corpus')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Perplexity')\n",
    "plt.axis([0,14,0,500])\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "#*******************************************************************************\n",
    "# ------------------------ test -----------------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "session.run(iterator.initializer,\n",
    "            feed_dict={features_placeholder:data_test.features,\n",
    "                       labels_placeholder:data_test.labels})\n",
    "losses_test=model.run_epoch(session,steps_test,is_training=False)\n",
    "assert len(losses_test)==steps_test\n",
    "loss_test_avg=sum(losses_test)/(len(losses_test)*seq_len*batch_size)\n",
    "print(\"\\nLoss/perplexity on test set after {} epochs: {:8.2f}/{:8.2f}\\n\".format(\n",
    "    epoch+1,loss_test_avg,math.exp(loss_test_avg)))\n",
    "\n",
    "#*******************************************************************************\n",
    "# ----------------------- finish ----------------------------------------------*\n",
    "#                                                                              *\n",
    "#*******************************************************************************\n",
    "session.close()\n",
    "logging.info(\"execution time - {}s\".format(\n",
    "    datetime.timedelta(seconds=round(time.time()-start_time))))"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "lstm_tf3_(3) (1).ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}