8. RNN Baiscs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of RNN_PT.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/avinregmi/PyTorch-Lessons/blob/master/8.%20RNN%20Baiscs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVq5mz9s8CPL",
        "colab_type": "text"
      },
      "source": [
        "## Building RNNs is Fun with PyTorch and Google Colab\n",
        "In this tutorial, I will first teach you how to build a recurrent neural network (RNN) with a single layer, consisting of one single neuron, with PyTorch and Google Colab. I will also show you how to implement a simple RNN-based model for image classification.\n",
        "\n",
        "We will be using Google Colab so we need to manually install the PyTorch library first. You can do this by using the following command:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hicY8Xs6_dI4",
        "colab_type": "code",
        "outputId": "f8664d59-1e81-451e-cd07-6a5bbe3e334e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        }
      },
      "source": [
        "!pip3 install torch torchvision"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (519.5MB)\n",
            "\u001b[K    100% |████████████████████████████████| 519.5MB 29kB/s \n",
            "tcmalloc: large alloc 1073750016 bytes == 0x5a19a000 @  0x7f4f985111c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54e4c8\n",
            "\u001b[?25hCollecting torchvision\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 13.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n",
            "Collecting pillow>=4.1.1 (from torchvision)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d1/24/f53ff6b61b3d728b90934bddb4f03f8ab584a7f49299bf3bde56e2952612/Pillow-5.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 3.6MB/s \n",
            "\u001b[?25hInstalling collected packages: torch, pillow, torchvision\n",
            "  Found existing installation: Pillow 4.0.0\n",
            "    Uninstalling Pillow-4.0.0:\n",
            "      Successfully uninstalled Pillow-4.0.0\n",
            "Successfully installed pillow-5.2.0 torch-0.4.1 torchvision-0.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXs_oWct-qxY",
        "colab_type": "text"
      },
      "source": [
        "Now we can import the necessary libraries we will use in the tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BBuAr4-L8CPN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import os\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EMQfRrRl8CPV",
        "colab_type": "text"
      },
      "source": [
        "### RNN with A Single Neuron\n",
        "\n",
        "\n",
        "The idea of this tutorial is to show you the basic operations necessary for building an RNN architecture using PyTorch. This guide assumes you have knowledge of basic RNNs and that you have read the tutorial on [building neural networks from scratch using PyTorch](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0). I will try to review RNNs wherever possible for those that need a refresher but I will keep it minimal.\n",
        "\n",
        "First, let's build the computation graph for a single-layer RNN. Again, we are not concerned with the math for now, I just want to show you the PyTorch operations needed to build your RNN models.\n",
        "\n",
        "For illustration purposes, this is the architecture we are building:\n",
        "\n",
        "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vQXBLYvvI1dqAHdLA0hQdsP1PojmCfuSCMK2DXEL0uTvRUqvD1eYK8fsECcNCoekxCbgWJ-k7QF_1s4/pub?w=600&h=400)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kHjE4j2x8CPW",
        "colab_type": "text"
      },
      "source": [
        "And here is the code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qiY3EllI8CPY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SingleRNN(nn.Module):\n",
        "    def __init__(self, n_inputs, n_neurons):\n",
        "        super(SingleRNN, self).__init__()\n",
        "        \n",
        "        self.Wx = torch.randn(n_inputs, n_neurons) # 4 X 1\n",
        "        self.Wy = torch.randn(n_neurons, n_neurons) # 1 X 1\n",
        "        \n",
        "        self.b = torch.zeros(1, n_neurons) # 1 X 1\n",
        "        \n",
        "    def forward(self, X0, X1): #X0:4x4 X1:4x4\n",
        "        self.Y0 = torch.tanh(torch.mm(X0, self.Wx) + self.b) # 4 X 1\n",
        "        \n",
        "        self.Y1 = torch.tanh(torch.mm(self.Y0, self.Wy) +\n",
        "                            torch.mm(X1, self.Wx) + self.b) # 4 X 1\n",
        "        \n",
        "        return self.Y0, self.Y1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cP94VBHE8CPc",
        "colab_type": "text"
      },
      "source": [
        "In the above code, I have implemented a simple one layer, one neuron RNN. I initialized two weight matrices, `Wx` and `Wy` with values from a normal distribution. `Wx` contains connection weights for the inputs of the current time step, while `Wy` contains connection weights for the outputs of the previous time step. We added a bias `b`. The `forward` function computes two outputs -- one for each time step... two in this case. Note that we are using `tanh` as the nonlinearity (activation function).\n",
        "\n",
        "As for the input, we are providing 4 instances, with each instance containing two input sequences.\n",
        "\n",
        "For illustration purposes, this is how the data is being fed into the RNN model: \n",
        "\n",
        "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vRpQYtOzO1U_3yQLf1885kMaja6MsXtJ8QnlqxrfpTgZmb4WpewJXphGdmotYXDB1VE6zlW6cBY_WqR/pub?w=600&h=600)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ydSxau_eFpwH",
        "colab_type": "text"
      },
      "source": [
        "And this is the code to test the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7a7-kIhj8CPe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "N_INPUT = 4\n",
        "N_NEURONS = 1\n",
        "\n",
        "X0_batch = torch.tensor([[0,1,2,0], [3,4,5,0], \n",
        "                         [6,7,8,0], [9,0,1,0]],\n",
        "                        dtype = torch.float) #t=0 => 4 X 4\n",
        "\n",
        "X1_batch = torch.tensor([[9,8,7,0], [0,0,0,0], \n",
        "                         [6,5,4,0], [3,2,1,0]],\n",
        "                        dtype = torch.float) #t=1 => 4 X 4\n",
        "\n",
        "model = SingleRNN(N_INPUT, N_NEURONS)\n",
        "\n",
        "Y0_val, Y1_val = model(X0_batch, X1_batch)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNaFkJUP8CPj",
        "colab_type": "text"
      },
      "source": [
        "After we have fed the input into the computation graph, we obtain outputs for each timestep (`Y0`, `Y1`), which we can now print as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-teLNoV8CPl",
        "colab_type": "code",
        "outputId": "29b3a9e2-237d-4273-95f6-cf81678c4f07",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "print(Y0_val)\n",
        "print(Y1_val)"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.8383],\n",
            "        [-0.9879],\n",
            "        [-0.9992],\n",
            "        [ 1.0000]])\n",
            "tensor([[-0.9927],\n",
            "        [-0.0235],\n",
            "        [-0.9007],\n",
            "        [-0.0942]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6tcX6ii8CP0",
        "colab_type": "text"
      },
      "source": [
        "### Increasing Neurons in RNN Layer\n",
        "Next, I will show you how to generalize the RNN we have just build to let the single layer support an `n` amount of neurons. In terms of the architecture, nothing really changes since we have already parameterized the number of neurons in the computation graph we have built. However, the size of the output changes since we have changed the size of number of units (i.e., neurons) in the RNN layer. \n",
        "\n",
        "Here is an illustration of what we will build:\n",
        "\n",
        "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vQov6BGg1fXOb7Bg5zenPh7R5j6VsZJh_D6JevQ_sm_fCxmXORxad3qLIFGG1FojzJig0qdcAQoGYoN/pub?w=600&h=404)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t5e2Eh5dGvnB",
        "colab_type": "text"
      },
      "source": [
        "And here is the code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zcEqoDfP8CP2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BasicRNN(nn.Module):\n",
        "    def __init__(self, n_inputs, n_neurons):\n",
        "        super(SingleRNN, self).__init__()\n",
        "        \n",
        "        self.Wx = torch.randn(n_inputs, n_neurons) # n_inputs X n_neurons\n",
        "        self.Wy = torch.randn(n_neurons, n_neurons) # n_neurons X n_neurons\n",
        "        \n",
        "        self.b = torch.zeros(1, n_neurons) # 1 X n_neurons\n",
        "    \n",
        "    def forward(self, X0, X1):\n",
        "        self.Y0 = torch.tanh(torch.mm(X0, self.Wx) + self.b) # batch_size X n_neurons\n",
        "        \n",
        "        self.Y1 = torch.tanh(torch.mm(self.Y0, self.Wy) +\n",
        "                            torch.mm(X1, self.Wx) + self.b) # batch_size X n_neurons\n",
        "        \n",
        "        return self.Y0, self.Y1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xlc4vEf88CP5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "N_INPUT = 3 # number of features in input\n",
        "N_NEURONS = 5 # number of units in layer\n",
        "\n",
        "X0_batch = torch.tensor([[0,1,2], [3,4,5], \n",
        "                         [6,7,8], [9,0,1]],\n",
        "                        dtype = torch.float) #t=0 => 4 X 3\n",
        "\n",
        "X1_batch = torch.tensor([[9,8,7], [0,0,0], \n",
        "                         [6,5,4], [3,2,1]],\n",
        "                        dtype = torch.float) #t=1 => 4 X 3\n",
        "\n",
        "model = SingleRNN(N_INPUT, N_NEURONS)\n",
        "\n",
        "Y0_val, Y1_val = model(X0_batch, X1_batch)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9VNN7jf8CP9",
        "colab_type": "text"
      },
      "source": [
        "Now when we print the outputs produced for each time step, it is of size (`4 X 5`), which represents the batch size and number of neurons, respectively."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r1kiEzop8CP-",
        "colab_type": "code",
        "outputId": "199c162b-9a22-454c-8065-07ca8a35be79",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "print(Y0_val)\n",
        "print(Y1_val)"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.9302,  0.8259, -0.9346,  0.0900, -0.8302],\n",
            "        [-1.0000,  1.0000, -0.9740, -1.0000, -0.9990],\n",
            "        [-1.0000,  1.0000, -0.9898, -1.0000, -1.0000],\n",
            "        [-0.9999,  1.0000,  1.0000, -1.0000,  1.0000]])\n",
            "tensor([[-1.0000,  1.0000,  0.9946, -1.0000, -1.0000],\n",
            "        [ 0.3098, -0.6453,  0.9932, -0.6578, -0.9666],\n",
            "        [-1.0000,  1.0000,  0.9986, -1.0000, -1.0000],\n",
            "        [-0.9355,  0.9999, -0.7902, -1.0000, -0.9998]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BRKjsv2t8CQG",
        "colab_type": "text"
      },
      "source": [
        "### PyTorch Built-in RNN Cell"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rcu2H6D88CQH",
        "colab_type": "text"
      },
      "source": [
        "If you take a closer look at the `BasicRNN` computation graph we have just built, it has a serious flaw. What if we wanted to build an architecture that supports extremely large inputs and outputs. The way it is currently built, it would require us to individually compute the outputs for every time step, increasing the lines of code needed to implement the desired computation graph. Below I will show you how to consolidate and implement this more efficiently and cleanly using the built-in RNNCell module.\n",
        "\n",
        "Let's first try to implement this informally to analyze the role `RNNCell` plays:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dp0Wjh4Z8CQI",
        "colab_type": "code",
        "outputId": "ff700261-a628-44cf-ef1d-3f57f8ebd62c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "rnn = nn.RNNCell(3, 5) # n_input X n_neurons\n",
        "\n",
        "X_batch = torch.tensor([[[0,1,2], [3,4,5], \n",
        "                         [6,7,8], [9,0,1]],\n",
        "                        [[9,8,7], [0,0,0], \n",
        "                         [6,5,4], [3,2,1]]\n",
        "                       ], dtype = torch.float) # X0 and X1\n",
        "\n",
        "hx = torch.randn(4, 5) # m X n_neurons\n",
        "output = []\n",
        "\n",
        "# for each time step\n",
        "for i in range(2):\n",
        "    hx = rnn(X_batch[i], hx)\n",
        "    output.append(hx)\n",
        "\n",
        "print('X_batch size: ', X_batch.shape)\n",
        "print(output)"
      ],
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_batch size:  torch.Size([2, 4, 3])\n",
            "[tensor([[-0.8115,  0.6101, -0.9623, -0.9366, -0.2288],\n",
            "        [-0.9831,  0.9961, -0.9998, -0.9773, -0.6530],\n",
            "        [-1.0000,  0.9968, -1.0000, -0.9999, -0.9997],\n",
            "        [ 0.9102,  0.9982, -0.9949,  0.7214,  0.8398]], grad_fn=<TanhBackward>), tensor([[-0.9997,  0.9999, -1.0000, -0.9995, -0.9968],\n",
            "        [ 0.4001,  0.5743, -0.2412, -0.3060,  0.4213],\n",
            "        [-0.9787,  0.9981, -0.9998, -0.9838, -0.9271],\n",
            "        [-0.8936,  0.5191, -0.9781, -0.6428, -0.3726]], grad_fn=<TanhBackward>)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cRNUCsXl8CQN",
        "colab_type": "text"
      },
      "source": [
        "With the above code, we have basically implemented the same model that was implemented in `BasicRNN`. `torch.RNNCell(...)` does all the magic of creating and maintaining the necessary weights and biases for us. `torch.RNNCell` accepts a tensor as input and outputs the next hidden state for each element in the batch. Read more about this module [here](https://pytorch.org/docs/stable/nn.html?highlight=rnncell#torch.nn.RNNCell).\n",
        "\n",
        "Now, let's formally build the computation graph using the same information we used above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Opob45Zj8CQP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CleanBasicRNN(nn.Module):\n",
        "    def __init__(self, batch_size, n_inputs, n_neurons):\n",
        "        super(CleanBasicRNN, self).__init__()\n",
        "        \n",
        "        rnn = nn.RNNCell(n_inputs, n_neurons)\n",
        "        self.hx = torch.randn(batch_size, n_neurons) # initialize hidden state\n",
        "        \n",
        "    def forward(self, X):\n",
        "        output = []\n",
        "\n",
        "        # for each time step\n",
        "        for i in range(2):\n",
        "            self.hx = rnn(X[i], self.hx)\n",
        "            output.append(self.hx)\n",
        "        \n",
        "        return output, self.hx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oL1yBNis8CQa",
        "colab_type": "code",
        "outputId": "d9e482e0-5970-47af-9aba-f0a67ca699f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "FIXED_BATCH_SIZE = 4 # our batch size is fixed for now\n",
        "N_INPUT = 3\n",
        "N_NEURONS = 5\n",
        "\n",
        "X_batch = torch.tensor([[[0,1,2], [3,4,5], \n",
        "                         [6,7,8], [9,0,1]],\n",
        "                        [[9,8,7], [0,0,0], \n",
        "                         [6,5,4], [3,2,1]]\n",
        "                       ], dtype = torch.float) # X0 and X1\n",
        "\n",
        "\n",
        "model = CleanBasicRNN(FIXED_BATCH_SIZE, N_INPUT, N_NEURONS)\n",
        "output_val, states_val = model(X_batch)\n",
        "print(output_val) # contains all output for all timesteps\n",
        "print(states_val) # contain values for final state or final timestep, i.e., t=1"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[tensor([[-0.5875,  0.9017, -0.9148, -0.8548,  0.0085],\n",
            "        [-0.9974,  0.9939, -0.9998, -0.9898, -0.8220],\n",
            "        [-0.9999,  0.9999, -1.0000, -0.9994, -0.9799],\n",
            "        [ 0.5357,  0.9693, -0.9960,  0.5890,  0.3850]], grad_fn=<TanhBackward>), tensor([[-0.9998,  0.9998, -1.0000, -0.9995, -0.9980],\n",
            "        [ 0.4429,  0.6220, -0.2390, -0.2724,  0.4610],\n",
            "        [-0.9789,  0.9981, -0.9998, -0.9839, -0.9281],\n",
            "        [-0.8255,  0.7146, -0.9778, -0.6223, -0.2146]], grad_fn=<TanhBackward>)]\n",
            "tensor([[-0.9998,  0.9998, -1.0000, -0.9995, -0.9980],\n",
            "        [ 0.4429,  0.6220, -0.2390, -0.2724,  0.4610],\n",
            "        [-0.9789,  0.9981, -0.9998, -0.9839, -0.9281],\n",
            "        [-0.8255,  0.7146, -0.9778, -0.6223, -0.2146]], grad_fn=<TanhBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iv_HcbFhT47d",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "000d743a-6d72-4ccd-bac1-9b63f5c43c36"
      },
      "source": [
        "X_batch.shape"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 4, 3])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l240cp4g8CQh",
        "colab_type": "text"
      },
      "source": [
        "You can see how the code is much cleaner since we don't need to explicitly operate on the weights as shown in the previous code snippet  --  everything is handled implicitly and eloquently behind the scenes by PyTorch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n3zn-Ydx8CQi",
        "colab_type": "text"
      },
      "source": [
        "### RNN for Image Classification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eTJHs-S-8CQk",
        "colab_type": "text"
      },
      "source": [
        "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vSiMstqkE9hTYmhPD3KMeFRNNKYA2NnrCayahBOEL1TalRqaWF7rH8a7O-nP9c-mKOdZRsWtmAGZfNN/pub?w=969&h=368)\n",
        "\n",
        "Now that you have learned how to build a simple RNN from scratch and using the built-in `RNNCell` module provided in PyTorch, let's do something more sophisticated and special.\n",
        "\n",
        "Let's try to build an image classifier using the MNIST dataset. The MNIST dataset consists of images that contain hand-written numbers from 1–10. Essentially, we want to build a classifier to predict the numbers displayed by a set of images. I know this sounds strange but you will be surprised by how well RNNs perform on this image classification task.\n",
        "\n",
        "In addition, we will also be using the `RNN` module instead of the `RNNCell` module since we want to generalize the computation graph to be able to support an `n` number of layers as well. We will only use one layer in the following computation graph, but you can experiment with the code later on by adding more layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vnhd2vR8CQo",
        "colab_type": "text"
      },
      "source": [
        "#### Importing the dataset \n",
        "Before building the RNN-based computation graph, let's import the MNIST dataset, split it into test and train portions, do a few transformations, and further explore it. You will need the following PyTorch libraries and lines of code to download and import the MNIST dataset to Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6FViVH8w8CQr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torchvision\n",
        "import torchvision.transforms as transforms"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yrNdklIv8CQv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BATCH_SIZE = 64\n",
        "\n",
        "# list all transformations\n",
        "transform = transforms.Compose(\n",
        "    [transforms.ToTensor()])\n",
        "\n",
        "# download and load training dataset\n",
        "trainset = torchvision.datasets.MNIST(root='./data', train=True,\n",
        "                                        download=True, transform=transform)\n",
        "trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE,\n",
        "                                          shuffle=True, num_workers=2)\n",
        "\n",
        "# download and load testing dataset\n",
        "testset = torchvision.datasets.MNIST(root='./data', train=False,\n",
        "                                       download=True, transform=transform)\n",
        "testloader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE,\n",
        "                                         shuffle=False, num_workers=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ixg3XnDM8CQy",
        "colab_type": "text"
      },
      "source": [
        "The code above loads and prepares the dataset to be fed into the computation graph we will build later on. Take a few minutes to play around with the code and understand what is happening. Notice that we needed to provide a batch size. This is because `trainloader` and `testloader` are iterators which will make it easier when we are iterating on the dataset and training our RNN model with minibatches."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2f0fVaIV8CQ3",
        "colab_type": "text"
      },
      "source": [
        "#### Exloring the dataset\n",
        "Here is a few lines of code to explore the dataset. I won't cover much of what's going on here, but you can take some time and look at it by yourself."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TEKzVTOY8CQ4",
        "colab_type": "code",
        "outputId": "36e75b76-8136-4675-eca4-633b57b6472e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "# functions to show an image\n",
        "def imshow(img):\n",
        "    #img = img / 2 + 0.5     # unnormalize\n",
        "    npimg = img.numpy()\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "\n",
        "# get some random training images\n",
        "dataiter = iter(trainloader)\n",
        "images, labels = dataiter.next()\n",
        "\n",
        "# show images\n",
        "imshow(torchvision.utils.make_grid(images))"
      ],
      "execution_count": 138,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQEAAAD8CAYAAAB3lxGOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOxdd1gUV/c+SxFFMRQRFRBj4VPsXQSV\nxG40NkyMYjRqsBsjtlixxv6ZWKOxmyjGHnusUVTEimJHRLqAgnTYue/vD5z57cICW2bgA+c8z/uw\nzMyeM3P33nduOfccBQCSRRZZPl4xKu4bkEUWWYpXZBKQRZaPXGQSkEWWj1xkEpBFlo9cZBKQRZaP\nXGQSkEWWj1wkIwGFQtFNoVA8VSgULxQKxQyp7MgiiyyGiUIKPwGFQmFMRM+IqDMRRRBRIBF9A+CR\n6MZkkUUWg0SqnkArInoB4CWALCLaR0S9JbIliyyyGCAmEum1J6Jwlf8jiKh1fhcrFArZbVEWWaSX\neAC2uQ9KRQKFikKh8CYi7+KyL4ssH6GEaTooFQlEEpGjyv8OH44JAmAzEW0mknsCsshSnCLVnEAg\nEdVRKBSfKhSKMkQ0kIiOSWRLFllkMUAk6QkAUCoUivFEdIaIjIloG4BgKWzJIossholkfgIATgJw\nBlALwGKp7EgtFSpUoGHDhhWpTR8fH/L39ydnZ+citVtaxd3dnQBQkyZNivtW/jcFQLGDiGAobty4\ngW7duhmsh0elSpXg5eUFf39/ZGdnY/DgwWrnHRwcUKNGDdSoUQNly5YVzS4RgeM4MMbw9u1bfPbZ\nZ6Lq/thgamqKLVu2gDEGJyenYr8ffaBQKNC5c2ds2LABe/fuxfXr13Hy5EmkpaWhc+fOuui6pbH9\nFTcBiEECZcuWxa1bt9CoUSNRCr1GjRq4ffs2OI5DdnY2srOzMXfuXOG8h4cH4uPjkZ2dDY7jsHXr\nVsybN08U2xUrVgQAnDt3Dnfu3EFqaioOHDgAGxubYq+MJRE1atQQSNXOzs5gfU5OThgzZgxOnToF\njuMEjBkzBmPGjEHr1q0NtmFrawt/f38B165dU7Olirdv38LBwUFb3SWbBCpWrIhnz56hXbt2ec75\n+PiA4zjRegI3b94UGnhycjImTJgAIoKnpydCQkIEAuCv4T9XqVLFYNsXL17EuXPnUKdOHVhZWWHZ\nsmVITk7GqVOnYGJiImoDcXR0xJEjR8AYEzB27FhRdDs5OeHgwYN49eoVKlWqJOp964IpU6aA4zhE\nRETgk08+MVjfF198AaVSmS+io6Nx7949jfVUW9jZ2eHp06fIzMwUGntQUBAmT56MyZMnw9LSEubm\n5jhw4AA4jkPt2rW11V2ySWDw4MFgjGH8+PF5zvn6+opGAn369EFiYqLQwH/66ScQEdzc3LBkyRJ8\n++23ahg6dCguX76MgwcP4vLlywbZHj9+PBhjeX5UDw8PJCcnY9GiRaI1jg4dOiAkJATjxo2Dg4MD\nHBwccO3aNZw6dUoU/ZMmTYJSqURycjJmzpyJevXqiXbv+cHGxga//fYbJk+eLBy7d+8eGGPo1KmT\nKDa6du2Kt2/fIiUlBdeuXYOXlxeOHDmCxMRENTLo0aOHwbbc3NzQsWNHdOzYEdWqVctzfs6cOeA4\nDr1799ZWp0wChcHLy0utq6V6btu2bRg5cmS+3+XvQV/b5ubmePPmDRhjGs/PmjULiYmJ8PDwMLhy\nNW7cGElJSVi8eLHa8bCwMNFIoEWLFkI3XPXvpk2bsGnTJnTt2lX4y5d7v379DLI5Y8YMcByHX375\nBUSEcuXK4cmTJwb9LppgaWmJDh06YO7cucI8g7u7u+gkUBjs7e2RmppaYL3MhZJNAuvXrwcA9O/f\nP8+53377DYwxg0ngwoULyM7Oxq1bt9C+fXudvjtv3jxkZ2frbXvHjh1gjOH27dsazzds2BDp6el4\n9OiRQc9oZWWFoKAgnDt3DqampsJxb29vcByHb775xuDKWa9ePRw8eBBKpRKxsbHYuHEjOI4TGgj/\nOfffwMBAg+wGBgYiIyMDbdu2BVEO2fFdaUOfqTDcunVLeL7IyEi0aNFCcptlypTBlStXEBwcrO13\nSjYJnDhxAowxWFtb5zn39OlTUXoCPAnwcwC6wBASaNKkiTAm79KlS77XrVmzBmFhYahYsaLez+jt\n7Y3Xr1+jTp06wjEHBwc8ffoUf/31F8zMzAwqwxYtWiA2NhaMMcTGxqqdq1evHry9vTFr1iwEBgbi\n0aNH4DgOABAbG2vQkKFTp05gjOHOnTvCscmTJ4PjOPj6+hr0TAWhTJkymD59ukAACQkJus7Y643p\n06eD4zisWLFC2++UXBIwNjbGxYsXCyWBK1eu4Pfff9erQD09PZGSkoLs7Gy9lpIMIYE9e/aAMYbf\nfvsNRkZG+V7n7u4OxhiaNm2qd8U5ffo0AgMDUbNmTRARWrdujRs3boAxhqlTpxpUKWfNmoXY2Fgo\nlUpcunQJzZo1K/B6Ly8voSfwww8/GGT7zJkzYIzBx8dHOHbr1i1ERUUZpLcwzJkzB0qlEi9evMC0\nadPQoUMHSe21b98ePXv2xJ9//onU1FQkJCToYrPkkgDR//cEmjVrhooVK2Lw4MGIiIhQG8Mb0hMA\nIOjRlQRU5xJ0tTtz5kwwxjB//vxCr23VqhUYY3B3dzeoItnY2KBNmzbw8PBA48aNYWJigqCgIIwZ\nM8YgvZUqVULdunULva5evXoIDQ0Fx3FYsGCBwasH48ePB8dxePToESwsLITjHMdh2LBhwv8WFhb4\n/PPPcfLkSSQkJOg9V1CmTBls2bJFePsvW7bMoPvXFr1791ar75s3b4aVlZUuOko2CcyePRsA1Jaz\n3rx5g3Xr1iE5OdngOQHVpT5dSYAfRvArCdrCxMQEz549A8dxWs1B8CSwfPlyUSuXi4sLGGMYMGBA\nkVTmbt26CZOFCxcuNFjf3bt3wXEchgwZonb8+fPnsLCwQKdOnXDmzBm8fPlSaEA7d+5Er1699LJX\nvXp1tUnAixcvFkm5ERGaN28uDAM4jsOsWbN0+X7JJgEzMzMsWbIE165dw7Vr17B9+3aB9cWYEzh4\n8KBeJODr64uQkBC9hgLNmzcHY0zrLitPAmLPPI8ZM6ZISYAfMsTGxqJ69eoG6erWrRsYY4iIiFA7\nPmTIECQkJGD//v3CS+PgwYPw9PQU5Rn69OmDs2fPCo1RLP8KbaBQKFCnTh28e/cOSqVSl0nIkk0C\nBYEngaVLl+qtY9iwYQIJFDQ5p4oVK1ao9SB0tblixYo849iCMHnyZISHh+vaBSwUS5cuBWNMmCeQ\nErt37xaWCw1dEiQi4e3eq1cvNG7cGAsWLMDx48fx7t07pKSkYM+ePejbty/s7OwKnG/RB5aWlti6\ndSvCw8ORlJSE69evo0aNGpKXoWp9+GgmBgvDsWPHDO4JqJJAREQE3NzcCrx+xYoVAvlkZ2fjwIED\nOtvkSaBly5aFXuvi4oLk5GRcuHBB9Mq0ZcsWxMXFoVy5cpJXXH4u4OHDh6Loi4qKAsdxePHihUAu\nPMRyIy8MLVu2RGZmJpRKJVasWCE62eQHIyMjHDlyBAkJCdp+p/SSAO+oY0hPgCjHZVe1EuWegHN3\nd8/jUAQAO3fu1MseTwKjR48u8Dpra2sEBATg1atXovi/58aDBw+wa9euIqm4/FxAYSsH2sLf31/4\nLa5cuYIdO3bAz89PdAehwrB8+XJhjsDR0bHI7P71119ITk7W9vrSSwLe3t5gjGH37t0G6enYsaPa\nvoCoqChcuHBBQFRUlNr57Oxs7Ny5E5aWlnrZa9++PRhjCA0NRfny5TVe06FDBwQEBIAxhlWrVole\nierVqwelUonp06dLXmH79esHpVKJAwcOwNzcXDI7q1atytfzUiqMHj1aIIEdO3YUiU0jIyMcO3YM\nly5d0vY7pZcELC0t8fLlS5w4ccLg5aYOHTqo7RBUbfC5/9dnCKAKc3NzBAcHgzGG06dPY+rUqXB0\ndIS9vT26dOmCzZs3IykpCfHx8Zg2bZqah59YmDlzJrKysorEt3/37t2Sv6GrVauGmJgY7Nu3T/Ln\nUUVwcLDBzkJVq1bV6fqdO3fKcwK5K7MYQwKinB7BuXPn8jT6yMhInDt3DhMmTICTkxMqVKhgsK0G\nDRrg1atXakufPDIzM3Hnzh3Y2tpKVnnj4uJw/vx5yfSrIjAwULS5gPwwa9YscBxXJD0bVfC9AEM2\neQ0aNAjPnj3DrFmz4OLiorG3VKZMGcycORPPnj1DVlYWjhw5osuEbukmAaKcXVchISEavQplaEZs\nbKxBbsja4vLly1AqlcX+vGKhSpUquHPnDpRKJcLCwtCwYUPRdA8bNgzr168XsGnTJrV5qGXLlum7\nClH6SUCG7rh165bkNvi5j6KerJMSS5YsEbwF69evX+z3oyVkEpCRFzNnzpTcRrt27cBxHP76669i\nf96PHBpJQJJchLqKnHdAFlmKRG4DaJH7oJyaXBZZPnKRSUAWWT5ykUlAFlk+cpFJQBZZPnKRSUAW\nWT5ykUlAliIVMzMz2r9/P0VGRlLdunWL+3b0kr59+1JgYCAxxggABQYGFvctGSbF7SOgr59AxYoV\n0aRJEzVI6V6bH3799VcAKPINKyUVfCgwXeIo6IOGDRvi3r17yMzMxKJFi7B06VKsXr3a4ECqa9eu\nRXZ2NjIyMpCYmIgXL14IGYiKu2y1QOlwFho9ejSuXbuGR48e5fG1l2KvvTYVQt/4gh8jZs+eDY7j\nkJ6ejq5du0piY+TIkcjMzNS4H+Pzzz83SPfOnTuxYMECuLi4oGbNmrCwsMCGDRvAcRzmzJkDY2Pj\nYi/jAlCySaBTp05ISkqCUqkEYwwAEB0djWfPnuHw4cNIT09HfHw8atWqVSQFOmbMGDUCOHr0qKT2\n+vfvj++//x69evXC9OnT1WCI3itXroAxhuvXr6N79+6SlxsfD/D06dOS2fDz8wNjDBcvXsT169cx\ndepUhIWFgTFmcHKVBg0aoEyZMmrHLCwshN7N7NmzJS0/ExOTPERjZGSEZs2aoVOnTvDx8Smoh1Wy\nSYAPy3358mX89ddfWLVqlZD7z9raWogso2vSEH1Qu3ZttaCVHMfhiy++EE3/pEmTMGnSJAQHByM5\nORnJyckC4WRlZam92bSJUqwJdnZ2WLFiBVJSUnDkyBGcPn0ab9680ZjuSkzw9y1WWjBNOHz4MBhj\nQpJYY2NjgQT++ecfSWzyMS0YY5L1cIhytmPfvHkT8+bNQ4MGDbBu3TocO3ZMsP3+/Xvs2bMnv++X\nbBIoW7YsKlWqlIeFiXJiADDGkJWVhTZt2khaiU1MTLB9+3Y1Arh06ZIoyUhzNxQAiIqKUttRtn79\nenTu3Bk2NjawsbHRKySYiYkJDh06BI7j1AKinDlzBm3btoWrq6vBY+f8wHGcpHv9nZychHRufLZe\nT09PoUwLCxtnCPjAo+fOnZNE/7fffovo6GgcP348zzAnJCQE27dvR+PGjQvSUbJJID8YGRnh9OnT\nYIzpko5JLzg6OmLXrl1qBJCZmYnmzZuLaqdq1aoCxM7oa2pqimvXriE9PV0tr6GzszNSU1ORlpYG\nxhiGDh0qevmZmZlJvtffyclJGDJaW1ujadOmiI+PB2MMe/fu1fgSEQsWFha4dOmSMBkppu5GjRoh\nJiYGkZGRMDY2hr29PTp37ozZs2fD3t5e2+hWpY8ETExMhMxEGRkZoqUmzw98wktV6JAMUiNGjx6N\n0NBQURKNaoMdO3YgMzMTo0aNUivHhQsXguM4bNmyBYwxjBgxQnTbXbp0AcdxkqfpOn/+PBhjqFq1\nKs6dOye8LYsikOry5cvBcZxoIcaMjY3RpUsXxMTE4P79+5gyZYoh+kofCWzatEn4gRcsWCDpj+vr\n6yukzOIxZMgQfNgBqTeioqLAGMP69eslr6CLFy8GAHz99dfCMRsbGyxatAgpKSmYP38+atSoAcYY\nmjRpIrr9Xbt2SZ4WjChnBp8xJvQAOI6Dh4eHwb+VNujbty8YY3oHn82NhQsXCnVchCXwkk0C9erV\nww8//IDIyEiNSz9SzmwPGjRIrfEHBQWJNgcwadIkYbkzLi5OlKzAmuDp6YmEhAQhIo2joyM2b96M\nyMhIDBw4EEQ53fWQkBBcvXpVknvYuXNnkaTssrCwQGJiohDEVcoewCeffKI2W89nQhajJ1CrVi08\nefIEISEheP/+PSIiIoS8kTzWrl2rSyQt8UmAiF4R0QMiuscbICJrIvqHiJ5/+GtlKAk4OzsLqcZ4\nvH79Gi1atBAiy8bFxam94cRC1apV8fDhQ3Ach5SUFFy4cEH0kNLVqlXD33//jYyMDDDGRIldmBt/\n/PEHtmzZAqIcR6vffvsNWVlZwuQZUU5YK7FXOlRx5cqVIiGBxo0bC2UpJQnUrFkTkZGRav4pYpIA\nXzdsbW3h4uIi/FatWrVCq1atsGrVKjx48AAXLlzQdkJcMhKolOvYciKa8eHzDCJaZigJDBo0CIwx\nREdH49ixY5gwYYLgD6BQKDBo0CCkpaXhwYMHov7IVapUQVBQkNADuH37tqSVl+/6rV69WnTdfn5+\nwtLqs2fPkJCQkCffwfTp0xEbGytJFp1q1aqBMYY+ffpIWoZEOU5c/MsiIyNDtNRjqqhXrx4iIyMR\nFBSEjIwMfPvttyD6/7RoYg0HCsOIESPAGFNLvFoAiowEnhJR1Q+fqxLRU0NJoEKFCmjfvr3aWys3\nAgMDkZKSUtgSidaoUqUK7t+/LxBARkYG5syZI+kP6uDggGXLluHSpUv55iHQFwMHDhSWHc+dO4fa\ntWurna9WrRrCw8NFn9XmsXr1ajDGYGJiImkZDh06NM9Q8fLly6LaKFeuHO7duweO4+Di4oKBAwci\nNTUVnTp1En1isCBUqVJFGB4MGjRIm+9IQgKhRHSHiG4TkfeHY4kq5xWq/xsyJ1AY1q1bB8aYaIV/\n4MABtXkAscilMAwZMgSMMV2STGoFExMTNGnSBA4ODhpdW/k4gFLNSfDLuFKWXa1atRARESE4BfEO\nQllZWaLa4bv8qq7iaWlpSEpKQkxMDDiOw6ZNm/TSvXbtWrRr167Aa8qVK4cRI0Zg27ZtSElJwe+/\n/w57e3tt9EtCAvYf/lYmovtE1J5yNXoiepfPd72J6NYHGPzDHDp0CIwxeHl5GaxrwoQJyMzMFH7o\ne/fuFZlPeMOGDcEYw4wZM4rEHo9du3YhNDRUsk1YPAnUr19fMoedM2fOgDGGM2fOwMTEBDdv3gRj\nDM+ePRPVjiYS8PT0FDYTGbIM+vz58zxzTmXKlIGDgwMcHBzQvHlzwZfjwoULeVLlFQJpVweIyJeI\nppAEw4HCwL8Bnj59arCn2+eff47U1FS1lYDvvvtOkkqbGxYWFjh+/Dhu3boFGxubIrFJlDNRmJWV\nhV9//VUyG2fOnAEAycbLS5YsEZK1NG3aFET/v4QsNgnUrFkTSUlJYIyhS5cuQp3btWsXGGMGRVXm\n7/fQoUMCeL8Hfjh3+fJlTJs2TZ/0d+KSABGVJyILlc/XiKgbEa0g9YnB5VKRgJWVFbZs2YLQ0FDh\nDWDIjztx4kQ8fvxYIID3799r280yGL179xY28wwePLhIbPJo1aoVOI7DZ599Jol+Z2dnpKengzGG\n8+fPo27duqLbmDdvHhhj2LJlC6pVq4Z69erh/PnzUCqV+Pvvv0W3N27cOGHT0MKFC9GnTx/h5fHT\nTz/prbdnz57YuHGjgBs3biAuLk7439fX15BeqegkUJNyhgD3iSiYiGZ9OG5DROcpZ4nwHBFZG0oC\n5cuXV8O4ceOwZs0aYfyXkZGBwMBAWFlZ5atDmzx+b968UZsInDt3riSNgoe5uTn69u2LjRs3Ij09\nHQkJCQZVIH2xdOlSREZGomzZspLo57M9M8bg4uIiiY358+cLOwdv3bolvDnDw8MlsWdhYSHkkeTr\nDGMMBw8eFHVJ0tjYWMzJ1JLtLFQU4EkgNDQUzs7OktubNm0aOnToUKzP7ObmhqysLPTu3bvYy98Q\nuLi4IDo6Gq9evcLVq1exZs2aYr+n/0HIyUdkySsrV64kDw8PatEiT04KWUqfyMlHZMkr9evXJy8v\nr+K+DVmKUeSegCyyfDwi9wRkkUWWvCKTgCyyfOQik4AssnzkIpOALLJ85CKTgCyyfORSakhg7Nix\n9ObNGwJAv/zyC1lYWBT3Lcnykconn3xCb968oXnz5ommc/DgwcRxHI0dO1Y0nbyUGhIYMWIE2djY\nEGOMxo8fT+XLlxfdxvfffy94WTHGKDw8nKpWrSq6naKWli1bUp8+fdTQuXNnyew5OjpSq1atKDIy\nkjiOo/fv35O7u7tk9opa1q1bR7a2tqRQKETTOX78eFq3bh3FxcVRy5YtRdNLRFTsLsNiuA3Xr19f\n2MfNQ8w8AET/n9oqd7ThmTNnimrHzc0NkydPLtLcdnv37s3zXImJiaKHa3N2dsbWrVvBcRyUSqUA\njuMM2nn3vwQrKyvExMTg1KlTom4/X7VqFVxdXWFiYmJItiuNbsMlvidgZmZGEydOJFtbW+HY3Llz\n6f3796LZUCgUNHv2bDIxMclzzsHBQRQb9vb2NG/ePDp79iy5urpSamoqERH16tWLVq5cSZ9++qko\ndnhp1qwZtW/fnhISEqhPnz55zltYWFDv3r2pbdu2otgzNzen8ePH09ChQ0XR978qPj4+VLlyZVq9\nejVxHCea3tTUVKpZsyYplUpydHQUTS8RUbH3AgztCXTu3FntDSbF9t9atWrleVPy2LBhg0G67e3t\nsWjRIkRGRoLjONy5c0c4161bN7x+/RocxyEhIQEjR46EkZGRQfaqVq2KlStXIioqKt9nUkVcXJwo\nZRgZGan25lftCSQlJUmaGUgTvL29sXr1agQGBoqq9/r163j48KFWu1Z1wYIFC4RNUap1REeUvl2E\n5ubm+P3339W6sD179hS18D/77DOEhIQINg4dOoRnz56JQgJGRkY4ceKEkKV3+/btsLOzU7umcuXK\nGDZsmLBXffPmzXpvLa1atSpu3rypVePn8fbtW4PKz9nZGW/fvgUAcByHpKQkfP/999i/f79gY/Lk\nyaL+ZjwWLlyIFi1aYNGiRdi0aZMQZITjOOF+VKMD6YJFixblCfXt4OCArKwsdOzYUfRnGTRokBAv\n448//tBXT+kjAU9PT7UK++eff4pe+MuWLVPLOThixAi1uQF9MxAZGRlhwYIF4DgOr1+/Rp06dQq8\nvmrVqggPDwfHcXpFI65evTquXLlSaKMPDw/Hv//+q3bMkPJzdnZGQkKC8PZv3Lgx1qxZg7CwMKEn\nIPa2bVtbW5w6dUqt15HfX33mIlq3bg2lUpln+/XWrVuRmJgoSSAaJycnREdHw8XFxZDoyaWPBK5f\nvy5U1ISEBPTo0UPUgh8+fDiio6PBcRzi4+PRqlUruLu7qzWQ+vXr66W7TJkyAgFoq8PX11d4Vl1s\nWVlZITAwME+Df//+Pf744w+0aNECLVq0wObNm9G2bVvY29vj+fPnwnWGZnresGED9u3bhyFDhmDu\n3LlqQ4NevXqJ3mAWLVoEjuMQHByMwMBA7Nq1C97e3vD29ka9evWE6/r166dXrsfevXuDMYbFixcL\nx5ycnJCZmal3gFFt4Ofnh+DgYEPiQJY+ElCt0L/88ovohc7PZHMcJ7C+GCRgamqKO3fuIDs7W6eA\nlOXKlRN6Idp+59NPP4W/v38eAti3b1+Bs/8PHjwQbd6DR8+ePdV6BY8ePcInn3wi6m82a9YsQb+5\nubnodYKI8PLlSzDGcPbsWeEYn8NRypTrTZs2NTRic+kiAdWhgL+/v6hZe6ytrREbG6vWyyDKabx8\n+mmO4/Dw4UOdgz1++eWX4DgO9+/f1+vevL29dSIB1fBX/LMMGDCg0O+pkoC+WZednZ2xZs0aJCcn\nq00G8p9jY2NRuXJlg38vvvufu6u/cOFC0eoEUU6Yu9evXyM1NTXPfT9+/BgnTpwQ1V5uuLi4AIAh\nyWtLDwl07doV8fHx4DgOsbGx+Pzzz0Ut7Nzr5j/++COICO3bt1c7rk8SUX5iTt9x8KeffmoQCfCZ\ncgrCwIED8fbtW4PnBDp06IA3b96oNcyjR4/iyZMnUCqVePPmjUEk0Lx5c2zatAktWrQAx+XE+IuN\njUVgYCCSk5OxceNGUetFv379wBjT2OUHIGkvgCiHBBhjCAsL03dIUHpIQLWRij0MMDExUZtAO3Dg\ngLAsx0/kcRyH7OxsveYg+Eapb6ovNzc3g0igZs2ahX6HT1OuLwl8//33Qu9h7dq1UCqVuHDhAj77\n7DMYGxtj1apVQq+goKxSheHy5ctQKpVYtGgRFi5ciIULF6J69eqwtbVFaGgofvjhB1HrBp/MJCws\nDIGBgfD19YWLi4uQYk01YYy+vaeC4OLiIqQ8O378uD46SgcJ9OjRA+/evQPHcQgICBA9XVeDBg3U\nJs769esHopylutu3bwvnAgIC9NJvCAnUrVsXp0+f/p8mAXd3dyEmf36z2Js3bxZ0G0IC/LPNmjVL\n7Xi3bt3AcRyaNWsmat148+aNWnozICePQmpqKgAgKCgI169fx/Xr1xEaGiqqbaIcErhx4waICP/+\n+68+IeJLPglUqFBB7S2tzdhWV3z11VeCftUEpMePH1drGE5OTnrp5yfpkpOT0bVrV/To0QPVqlUD\nUc7YVlOjcHZ2xu7du5GRkYGsrCydwpID0JoEatSogQcPHiApKUm4PjMzU2tbFhYWuH//PpRKJU6f\nPq3xWZo3by4MDfbv329QeG5eDz/jb2tri9WrVyM2NlaSicE6derA3d1dwJdffolff/0Vv/76K2Ji\nYnDkyBHh//Hjx4teN42NjXHr1i0Q5eQhfPXqla7lV7JJwNzcHNOnT1er0FKMwTZu3Cjo//7770GU\ns6afnZ2t1gvQ1yOsQ4cOwnwGj6ysLBw8eBBxcXHgOA4HDx5UQ1ZWFjguZw1/+vTpOtnL3RNYu3Zt\nnnIbOHAgFi5cqHEZURePQVUSyC+nYVhYmNB4x44da9BvxT/bokWLcPDgQeH/lJQU7Nq1S/S6UVDj\n9PX1LRJbd+/eFT4HBwfruvqQ+SEAACAASURBVHelZJNAr1698lRQQ1OOacKcOXOEN2CbNm1gZ2eH\n2bNnCzaDg4MN7mZ6eHhgw4YNePjwoVrmY47L8RzcsGED3r59iw0bNmDDhg0YMmQI6tevL/QYdIEm\n9+CYmBg8ePBAgOokII+MjAwsX75c52xB/v7+UCqVsLCw0HieX+K8efMmypQpY1A5BgcH53EAevjw\nIfr27VskDZJHgwYNiixhrWoyleDgYCxYsECX75dsEli/fr1aJY2OjpakkBcvXizYOHHihOC7z/cA\nunTpIpotY2NjGBsbw9TUVA1EJFrWmQYNGiAoKChPIy8Ily9f1isharVq1RAaGgqO49Tcty0sLLBq\n1So8efIEHMfh0aNHojwbvzTIOwSpOgIVJXr27FnkJGBqaooXL17oulxYukiAHxuJDS8vL40N49y5\nc2jdunWxVDJD4ezsjFevXqklWs2N+Ph4xMXFoVevXnmy4mqLatWq4erVq1AqlUhJSYGfnx/8/Pxw\n584dtSXCosjuVJTYuXNnkZHAsWPHYGtri23btuH9+/e69oZLNgkUJSpUqIDly5eD4zjs378fQ4YM\nKfZ7EhPOzs748ssv0ahRI0n0V61aVWj0SqUSN2/exMqVKyVLeCpDa8hpyGSR5SMXOfmILLLIkldk\nEpBFlo9cZBKQRZaPXGQSkEWWj1xkEpBFlo9cZBKQRZaPXGQS+B+Unj17EgDy9vYu7luR5WOQ4nYU\nMsRZiOM4dOvWrbgdMESFlZUVGGP4/fffJbeluhehYcOG6NKlCyZNmoQrV64AgNpmFRn5o2/fvti7\ndy8YY3j37l2x308B0M9jkIi2EdEbInqocsyaiP4houcf/lp9OK4gol+J6AURBRFRMylJQKlUomvX\nrsVdsKLC1dUVjDHJyc3b2xvv3r1DVFQUoqKi8rgUM8aQmZkJLy+vYi+T/1XUqFEDkydPRnJysrCD\nMSsrC8uXL5dkc5sI0JsE2hNRM1IngeVENOPD5xlEtOzD5x5EdIpyyKANEQVIRQK2tralkgR69+6N\n+/fvi5rCKjfKly+Px48fq20zVv2ckpKCqKgo/P3333BxcTHI1pQpU8AYQ1RUFA4cOADGGNauXVvs\n5WwoHB0d8fDhQ43lx3H6h4+TGBpJIG9erVwC4F+FQlEj1+HeROTx4fNOIrpERNM/HN+FnJZ9Q6FQ\nWCoUiqoAoguzo6vMmjVLbJVqYmtrS9WrVyciovbt29N//vMftc9Dhw6lQ4cOUVpamqh2PT09KTY2\nVtQUVrnFwsKCnJ2d1Y75+/tTSEgInThxgh4/fkwPHz402E737t3J19eXUlNTqUyZMuTh4UHHjx+n\ngwcPGqxbGylbtixVqFCBevToQWXLliU7OzuKiYmhL774goyMjMjFxYUyMjJo5MiRdOPGDa311q5d\nm44dOybUCU2ycuVK+vLLL8V4jDwyduxYcnJy4l+gRER08uRJ+vfff/VTqOWbugap9wQSVT4r+P+J\n6DgRuaucO09ELUracMDW1lYIsMHvU1f9zP8VO3xV69atkZ2dLflGm/79+wshsqSyUaFCBTx9+hSR\nkZGoVauWpM+jCmNjY/Tr1w+rV6/G06dPkZqaisTERCQkJODevXu4d+8efH198eOPP+Lzzz9H48aN\ndYpAVLt2bWFLtCpCQ0ORmJgo/H/jxg1RIilrwrt37/L0PPhgNIWEcNd/FyEVQAIf/n+nKwkQkTcR\n3foAvQpDKhJQjV7L/3316hVCQ0PVotfoGnCjMGzevFnShkmUMxl44cIF4bmkssPHZRA7a3N+sLCw\nwODBg/Hq1SsAEKICe3l5oXLlyrCxsTHYhqOjIx49eqQxAEuDBg0wevRoteNiJ8OpVasWXr16BcYY\nnj17JkSIsrS0xM8//wwACAgIyJMeTQWiksBTIqr64XNVInr64fNvRPSNpuuk6glIMXHVvHlztbf+\ngQMHUKlSJVSqVEk4JnYiS1dXV2RnZ0vaMM3MzDBixAi1cWzdunVRpUoVWFlZiWbH0tISERERePLk\niah6C8KlS5eE3s2+ffv0zgxVEPbs2ZOHAG7fvo3hw4eDKG9iGjFJYNSoUYiJiRF6GS1btsxzzb17\n98BxHL788sv89IhKAitIfWJw+YfPX5D6xOBNqSYGiXJI4PHjx6L/2Ll7AnzEGm9vb+GYt7e3qDZn\nz54NxpjGjLOVKlVCv379MH/+fHTo0EFvG126dMmzAqAaS3DJkiWiNNpp06aB4ziMGzdOONa4cWM0\na9YMzZo1yzf0mCHYvn07AGDixImi6+YxadIkobzev3+P3bt3q5VXbhKYN2+eKHZHjBgBxhjS09Mx\nbtw4tG3bVuN133zzjTQkQER7iSiaiLKJKIKIRhCRDeV09Z8T0TkislaZH1hPRCFE9IC0mA8whAQ4\njsPTp09F/7FVewIPHz4Ujq9evVroCeiTFDQ/tG3bVugF8IlOatWqhblz52LZsmWIjo4W3nLZ2dl6\nJdHknysjI0MtbLYmvHjxQu9Izk2bNkVaWho4jsPWrVuxYsUKBAcHqwVqDQkJ0Sn9mjYICAjAw4cP\nJV1VMTExgZubG9zc3NRyDPDITQJXr141yJ6FhQXWrFmDjIwMvHz5Ughtlx8J8DkpfHx88tNZ+iIL\nSdUTyK8BqQ4R2rVrJ4reLl26IDs7G2lpaejfvz+ICOnp6WCM4eDBg3kmrT777DPJhgxVq1bF/Pnz\n8fbtWzDGkJKSgoEDB+qsp2zZsrC0tESZMmVQtmzZPOcrVaokZD5etWqVKPdub28vzNns3bsXtWvX\nLpJ64ezsjDFjxgjOQpoQEBCAqlWr6qR31KhRYIzh5cuXQr0oCP379wcAZGZmwt3dPb/rSicJLFq0\nqEh+bD6hBWMMBw4cEE3vtWvXwBgTMsqUKVMGjDE8ePBA46y1vb295JOHVatWxaZNm8BxHKKioiSx\nUbFiRbx8+RLZ2dmi5Y+wsLDAkSNHwBhDcnKy3lmeCkOrVq3w5Zdf4v79+0IIdU2z9bmDt2qrv23b\ntkhPT0d6enqhgW3NzMwwd+5cpKSkgDFWWA+19JFAUboN8wktxFwadHd3h1KpBGNMmETy8PBAVlZW\nnoZhbGwMNzc3+Pv7IyMjQ/Ln7dKli+DkI5WNoUOHgjFmcLdZFUZGRmjcuDEmTZqE+Ph4jd12fWFj\nY4M9e/YICVbzm1/JD9raefXqVZ45FU2YNGkSrl+/LujftGlTYYl5SxcJuLi4FKnHIP8jnzp1SjSd\n3bt3FyrQv//+i9GjR+PcuXO4f/8+rK2t4e7ujtGjR8PHxwdXr14FYwxKpRJLly6V7DlNTU3Rp08f\nIYW4lCRQr149IdKxFKsId+/exdGjR0XR5ejoKERN1gQxSYBPbdakSRON5318fODv7y8kpQkICEDv\n3r210V26SGDNmjW4ceMGKlWqJFklVQU/FyD2qsAvv/wiVKKCkJGRgVu3bhU06WMwunXrhp07d6pV\narETvqqifPnyQp6CKlWqiK7/2rVrQu4+Q/Dtt98iOjq6wAbOl1dmZiaOHTuGRo0aoXHjxjh27JiQ\ncEVbe7yukydPYvjw4Rg+fDhGjBiB4cOHw9fXV7B39+5dzJ07V5dsWKWLBNq3b4+QkBDMmTNHskqq\nWln5H0YK/cOGDVObtU9NTUVqaipiYmKwfft2zJgxQ+/EGtOnTy/wLVuuXDkMGTIEZ86cQXp6ulql\n3rhxoyTLeaoIDg5Gamqqvqm2CwRjDFu2bDFYz6xZswp9y/P148WLFxq/r0uC0piYmDwvAQDC5xkz\nZui7N6F0kQCRdEuEucEnJFFdLhQT/PibMYbg4GBRdR84cACBgYHo0qVLHvj6+mrsdVy5ckWSPI+5\nUalSJcTExODKlSuS6GeM4fDhw6Lo0YYEQkJCDN5wRZSTbPQ///lPHsycOVPr/I35kETpIwGlUonX\nr1+je/fuohR+fti0aRMYY1i4cKEk+hs1aoSMjAycOnVK9KUt1XFs7sqcexdhUFAQunfvXpj/uWhw\ndXUFx3GYNGmSJPoZY4VOrmkDAIWSwJ49e1CnTp0iKTcDUPpIIDw8XMhyI9WwoF69eoiNjUVsbCyq\nV69e3D+izuAzKeVHAlFRUdi1axf69OkjSZe8IOzbtw8cx0m2lBcRESHKvv6ffvop3xRu/Mak4v6d\ntUTpI4GiAL+bUCznIBk5zkS8n3uvXr1E1//JJ58gJCSk2J/zfxAyCegKW1tbhIaGAjk3KUMEODs7\nCw42a9euFX3o0bJlS0RFRUnuUFVCoV9QkY9Z4uLiKD4+nsLCwor7VkqNPHv2jJycnCTTn5qaSuHh\n4TR9+nTJbJQ2kROSyiLLxyNyQlJZZJElr8gkIIssH7nIJCCLLB+5yCQgiywfucgkIIssH7nIJCCL\nLB+5lBoSmDlzpuQ2qlSpQiEhIZSdnU3Z2dm0bds2yW3KUvxiZGREFhYWBcLHx4csLCzIxER61xtf\nX1+NTnceHh76KSxub0FdPAaNjIxQqVIlGBkZqR2vWbMmUlNTJfe4qlGjBrKzswVs27ZNdBumpqY4\nfvw4ACAjIwO+vr75BpcQA507d8abN28AAEeOHClVad0cHR0LiryrNRo3bizsUSkMI0aMkDQPoYeH\nB1TF19cXRCT8LQQl3224UqVKYIzlSSRRt27dInETlZoEXFxccPToUSEi7/Dhw+Hm5iZZBp8qVark\n2RDz6tUrSSMXFSX4KMc7d+5E//794evri3nz5qF69eo6NVRtCYCHVER68eJFofF7eHjoo6Pkuw0P\nHjy4uG9BEjEyMqJp06bRqFGjqHr16jR69Gg6e/Ys2dnZUVhYGMXGxopqr3z58jRr1iyaNGkSERGF\nh4dTZmYmWVhYkKOjI/n4+JCdnR0NGDCAGjduTCEhIQXqmzdvHtWvX18tL+OrV6/oypUrRERUq1Yt\nQcf58+dFfZb8ZNWqVUKuwMGDB9PgwYNJoVAQAJozZw7duHGDBg4cSBEREUVyP4bKxYsXhe7+pUuX\n6NKlS+IpL+5egC49gWnTphVrT+DixYtqb83t27cbrLN69eo4e/asoPPIkSMgIvz444/gOA43b97U\nOVx1QShXrhwOHTok2Bs3bhwqVqwIopyewf3799WeUZv4BkuXLsWTJ0+QlJRUaJi0169f4/79+5g7\ndy6qVasmye80fPhwpKWl5XlD8yHieHz11Vda6VP9TkJCAu7fv1+kPQHVHgBg0Ga2kj0cqFOnDhIT\nE3Hnzp08seyXLVtmaOEUim7duiE+Pl5tOLB8+XKDdNaoUQPbtm0TGtyxY8eEBsmTAMdxesX+1wQ+\nlBivd9++fXmu+fnnn3UmAR7Ozs5o06YNhg8fDk9PT6xduxYbNmzAlClTsGHDBmzYsAEpKSnIzMwE\nYwxxcXE4evSoKHkCiXK2EK9bt07rYKD379/XSu/Zs2fx/v17/P3331i6dCm6d+9eZCSgOgdw8eJF\nQ/WVbBLo2LFjvsFD1qxZgydPnohW8Jpw6NAhNQIQI/cAn4AjOzsbffv2Rfny5YVzUpDA5MmTBZ3J\nycmwt7fPc80ff/whXHPnzh21exILjRs3xoABA4QcAWJFbPrpp58KbJy5ewIHDx7USq+VlRX++9//\nCv83bdoUe/fuxY0bNyQlAVUCyD3x5+HhIUAHnSWbBMLCwpCYmKgWldbMzAwTJkwQgnQ6OTmJXmF5\n8I01OzsbcXFxhSaFKAy1atVCVFQUOI7D2bNn85zno+7wIaUNvf9+/foJ6cE4jhOSaPIwNjbGggUL\nhMaSkpKCNm3aSFaeRDk9h8zMTLx588ag0FzlypVDbGws0tPTkZ2djejoaPzzzz95Guf+/fsxd+5c\neHt7Q6lUYu3atQbdv7u7O/z9/SUjgdwTgR4eHvD19UVuuXjxorZkUHInBi0tLcnR0ZGIiJ4+fUpZ\nWVlERKRQKMja2lq4rlOnTrR161bJ7+f+/ft09uxZg3TUqVOH7OzsiIiob9++ec4/fPiQBgwYQERE\npqamBtkiImrbti2ZmZkRUc79+/n5qZ1ftGgRTZs2Tfg/NTWVbty4YbDdgiQsLIw4jiMrKyuqUqUK\nPX/+XGcdlpaWtH//fqpUqRIBoLi4OKpatSoREXl6eqpde+DAAXJ1daXNmzcTEVFUVJRe912jRg36\n8ssvDdJRmHh4eKit+8+bNy9fPwDVCUO9pLh7Adr0BCZPnoyrV68KGDt2LNq1a4cRI0ZgxIgRSE5O\nxr///ivZG+vkyZNqPQGxkmn+/fffQhf16NGjApKSktTGru/evTM461GXLl2QmJgo6OzVqxdatmyJ\nI0eOaIybV0A+O1FQoUIFvHjxQkiPruv3mzVrphb37+HDh/m+gcuXL4+vv/4aQE7A0KysLJw5cwbW\n1tZ63XtRzAloeuPzPYLc16oOGwrRW7KHAwUhLi5OlPjymuDh4YH4+HiBBM6dOydaSCxbW1tcuXJF\n4yTW8+fPha7506dPRcnQ061bN9y+fbvQyLlihOkuDHxi1devX+uVQObMmTNCo3v69KnG+Q1HR0cs\nW7YMd+7cUZsT2LFjh973Xbt2bRw5cqRAEjh8+DB++eUX/PLLLwYlx9G2yy+TAOWQwNy5c0WvqJ06\ndRJWBDiOQ2hoqOgx8aysrNCgQQNs27ZNgI+PD+zs7LB+/XpwHIeIiAh8+umnotjr2rWrWoNPTU1F\nRESE8P/u3bslTe9NRLC2tsaNGzfAGMOQIUP00sE3uIcPH8LR0VHt3CeffIKJEyfi3bt3ao0zKSkJ\nU6dOzeNxqgvatWunk+PQ/v37sXXrVmzdulVnW/wcQGHjfdW5g0J0lm4SWLduneiV9cKFC8IQgOM4\n/PTTT5I2jtx4+/atMEsvlk4zMzN4eXnhjz/+wB9//IGOHTuq9Ub06ZrriiZNmgAAAgICYGJiopcO\npVKJt2/fwsHBQU3v1KlTERYWlqcxXr9+XRTPS11JQBVSladqb6GQa0s3CUyfPl3Ugt2yZYvaG3Pk\nyJGSN47c4HPYSZUPsGzZshg/frzwjFOmTMGHeI+SYuHChUhJSUGjRo301qFUKpGZmYkHDx7gwYMH\nePjwId6/f5+n4f36669wcXERzZ//f4kEPDw8dHUlLr0kEB8fDzc3N9EKt1KlSmoTgdnZ2ZI3DE3g\nSUAsP4HcWLZsGV6/fg2O45CUlIT69etL/kzGxsY4fvw4oqOjDdJTmCNQfHy8JD78UpOAtg5BuScO\nDdlAVCKWCAsTAPTNN9+Qv7+/KPqWL18uip7/ZTEyMqLatWuTvb09ERHt3buXgoODJbfbvXt36tGj\nB/n4+Bik58PLI8+xzMxM2rRpE23atEmvJcfC5O7duzRx4kT69ddfRddNlLPcp/psly5dosuXL6v9\n7+HhQfPmzROOzZ8/n3x9ffU3Wty9ALGGA2LuHeBdeflewO+//y75G1ITpOwJdOvWTXhrhoeH6z02\n1xVBQUEICgoy2N7OnTsRHR0tYN68eRg8eLBoLsgFwdzcHDVr1lSDmMOB/JYHc4sOTkI8Sm9PgIjo\nn3/+IWdnZ3r27JnouhcuXCi6zuKWjRs3EhFRbGwsdevWjZRKpeQ2jY2NydnZmYYOHWqwvaFDh4p0\nV7pLWloavXz5Uu1YtWrVaM2aNfTVV1+pHXd1ddU5eY2vr68AXjp06KDWIzDozZ9btHhLbyOiN0T0\nUOWYLxFFEtG9D+ihcu4nInpBRE+JqGtR9ARKK6TqCbi7uyMjIwMcx+ntMKMPFi1alF/KbBlFA/0m\nBomoPRE1o7wkMEXDtS5EdJ+IzIjoUyIKISJjmQRkEBGOHj1a7PfwkUMjCRQaYxDAv0T0trDrPkhv\nItoHIBNAKOX0CFpp+V1ZSrnExcUV9y3IokEMCTQ6XqFQBCkUim0KhcLqwzF7IgpXuSbiwzFZZKGR\nI0cW9y3IokH0JYGNRFSLiJoQUTQRrdJVgUKh8FYoFLcUCsUtPe9BFllkEUH0IgEAsQA4AIyIttD/\nd/kjichR5VKHD8c06dgMoAU0ZEmVRRZZik70IgGFQlFV5d++RPTww+djRDRQoVCYKRSKT4moDhHd\nNOwWZZFFFimlUD8BhUKxl4g8iKiSQqGIIKJ5ROShUCiaUM6M4ysiGkVEBCBYoVDsJ6JHRKQkonEA\nOGluXRZZZBFDFNDgflnkN5GzaUUWWWSRVm5rGn6XqDRku3fvpuzsbGKMEWNMCL9VnGJhYUEBAQEa\nfdm1kZEjR9Lp06fp4sWLpFQqBXAcR0qlkm7evEmDBg0SQoPJolkuXryolYu6qJ52RFS3bl3q2LEj\nNWvWTDg2fvx4ysjIoJ9//pkaNGggqj2FQkFnz54lADR9+nRSKBSGK9XFx18qkBaODnv27MkTwz47\nOxurVq3SOn68FNi8ebPgg6/rd0eOHImUlBSNkXBz/y9FyrOGDRvC1dUVrq6uOHnyJPz9/TFq1Ci4\nuroWt1OLTtDW1x7QerddgahcuTK2bt2K169f4/3792CMIT09HUePHkWDBg3g5+cn1FE/Pz/RntPa\n2hr79+9XawOqAVW0CAZTcrcST5w4UXjoR48eYf369Vi/fr1w7MWLF6hevXqRVjxra2v4+voKrr3p\n6ek66zh9+nSeRr906VJUrlwZdnZ2GDZsGEJCQkQnAkdHR2zevBlv3rzRSDjx8fGil5ejoyMWLFiQ\np1HeuXMH1tbWBscxyK+x+/r6qu25F4MEjh8/DsYYsrKykJKSgkOHDiE1NRVBQUEIDw/HrVu3hK3N\n+kZOUoWRkRG+/vprvH37FowxHDx4EF9//TXWrl0La2tr1KhRAwcPHsTx48dhampakK6SSQJNmzYV\nHv7Ro0dqzLd161aBCO7fvy86EXh5ecHS0lLjuW+//VYtRFePHj101j9gwABs3boVUVFRmDJlClxc\nXDRed+/ePXAcZ/AefEtLS6xcubLQXsekSZNEK0NjY2MMHz4cu3btKjCuoWooeX1Q0G46MUnAzs4O\nSUlJCAsLw8qVK9XOWVhY4MmTJ0KdvHv3rsHlZ2ZmhlmzZgk6IyIiUKFCBeH8hAkTkJ6eLpxXPacB\nJZME5syZIzxg7vBQjRs3xps3b4TzYmSg5fHNN98gKytLYwBTLy8vtUi3p06dEs2uJty9exdKpRJR\nUVF6fd/Z2Rlr1qwR7heAWgPM/T9/7M6dOwZtXnJycsrT+PmeE8dxSEtLE0K3rVq1SpKyu5grhZee\niTwF+Pr6gjGGOXPmoEyZMnnOX7t2DYwxBAUFqYU+0wf29vZ4/PixUL937NgBCwsL4fyECROQlpYG\nxhiSkpKwefPmwiIolUwSiI+Px+PHj+Hn56dWADz8/f2FQpo5c6YoFWfAgAEICQkRQlnb2dkJ52xs\nbHDr1i2hIk+dOlXjfYkJngQSEhJ0zkvo7OwsRC5WffOHhoZi7969sLe3h729PXr06IG9e/di7969\nePTokdA7SEtL02vnn5OTEx4/fiyUU0ZGBpYtW4br16+rZVYKDQ0Fx3G4e/eu6Cm9cxOAtlF78oNC\nocCRI0eQlZWlMQqTk5MTkpOTwRjDjz/+aJAtFxcXBAcHgzGG5ORk+Pr6CinqiP4/mQw/H/Gf//xH\nG70lkwQaNmxYYFexc+fOQlCRe/fuiVJ54uLihIqaO+1Zy5YthXNhYWGiRx/WBJ4EOI6Dl5eXTt99\n+fJlnuAWixcvLjDjj5OTE65evYpHjx5BqVRi9erVOtk0MTHB3r17hXK6d+8eevXqBSJCTEwMOI7D\njRs3ULFiRYEEzpw5U9h4Vidomiw0VKe5uTkYY0hMTNQY/Xnp0qVgjMHf3x9ffPGFQba+++47MMaQ\nkpKCVq1aqZ3r378/UlNTBQLQIUdEySSBwjBo0CBhhvavv/4yqODr1KmDn3/+GVlZWUIjV41n37Rp\nUzx79kyIyVdUqxI8CcTExORJxloYVMf7sbGxOgUtdXBwwPXr13WKimNsbIwZM2YIBBAZGYlOnTqp\n3Q/HcVi3bh3c3d3x/v17UYcDud/+YhEAUQ658ZN+y5YtUzu3ceNGvH//HtnZ2fj8888NtuXo6Cgk\nbx0xYoTaOT0JAFRaSUB1OGDonIDqcl9YWBiaNm2qdt7NzU04P3HiRFEqlo2NDapUqaIG1aUeZ2dn\nxMbG6j0xqEoCnp6eOn/f1dVVJxKwsbFRI4Dck7X8uZMnT6oNDcQI554fAQCiZPQFEWH79u1gjOHc\nuXMoV66ccJyvg2JGve7cuTOCgoKQmZmJvXv3olq1aujfvz8YY0hLS9NnvqZ0kMCIESNw//59MMbQ\nsmVL4bMYWXP4t7w2OHHihN52+vbtmyeRZWEz9kqlEunp6di9e7dOyVANIYGePXsiLS1NaxIwMzOD\nv78/OI7Dy5cv870fVWzZskXUuQBfX1+1FQApiGDatGlgjCE0NBRdunTBwIED8ejRI0kyOBMRTE1N\nceHCBYFoVHtWOqLkk8B3332nthzCGAMAMMZECS/96aefwtfXFwEBAQgJCcHdu3cRERGBgIAAtfyA\nly9fNihXn6YGrg0JqB7TduJJ9Ts+Pj5a3+OMGTPw+PFjKJVKnbrq//3vf4Vl0/Hjx6NDhw7o0KED\nNmzYgLNnzwplGBMTg19++UXn4Y2uyD03IIZOOzs7tVl7xhjGjBkj2TOYmpriypUrgq3jx4+jdu3a\n+ugq+SQQHx+fx2uQx7///ovu3buLVvDVq1eHubm5kJHn6tWr4LiclN2GLjOJQQLv37/XylZCQoLa\nd/z8/ODo6Jivd1mtWrXg5+enZkuXtOE8CRQGMX+rgiAFCRARGjRooFb/nj9/Dh8fH0mc1v773/8K\ncwCff/45UlNT8eLFC32IoHSTAGMMb9++Ra9evQpzmNALfCJPQ4YBqhVT07q8pv9v376NKVOmYMqU\nKahSpYpw/sKFC1rZ+vHHHzWSyaZNm7By5UqsXLkSq1atEj7zb3+eBF68eKHTs9WtW1ftjc8jISFB\nyMLMcVyRJDohUh8O4QSAOQAAHftJREFUiEkCq1atEurd+fPnhc9ZWVmiul07OzsLvV+eOCdOnCgQ\nQX7ObPmg9JMAj0uXLqFXr16ijtH4yivGsKNcuXLYs2cPEhMT833zR0REYNGiRWp+AeXKlYOXlxcO\nHz6sU6ry3bt3a0U4uY+NHTtWr+czNTVF06ZN1cD7GohFAtr0xnJPFBrag+Px3XffISsrC4wxnDp1\nCsbGxrCxscHUqVORnJyM4OBgdOjQQRRbY8aMAWMM4eHhas5JS5cuRVpaGv7880+NTkv5oOSTwHff\nfYeXL18iLi4OcXFxWLBgAUaPHo0lS5bg3bt3eeYKxPTk4zgu3xTY+mLJkiUCEfAkEBsbi969e6NG\njRqi2SlXrhzc3Nzw6tWrfIcevPMQD3t7e0myE4tBAqpd/NwTgZquAcSbFCQirFu3Tuiet2vXTu0c\n7zFoqLMQDzs7OyQkJCA7Oxvz589Xc0zbsGEDGGO6OJCVfBIorLACAgIEhuaxZs0arFmzxiDdjRs3\nBsdxOH36tOiNok6dOmjYsCEaNWqEhg0bipaCPD9brVu3xuLFi3H8+HFcuXIF06ZNQ5s2bXQa9xsC\nsUlAVXJvFpKCAIhI8BXIvUPQzc0NCQkJYIyhb9++otmztbXFgQMH8uwdqFu3LtLS0jBv3jxtdZVu\nEuDRu3dvLF68GEqlUthufPz4cYN1chyHzz77rEgaSmkGTwK6ej7mhrYixq7B3Pjhhx+QnJyM7Oxs\nLF68GK6urmjXrp3QGz137pzoqx5GRkb45ptv8OjRIwQEBMDHxwc9e/bEmzdvdCG5j4MEePTq1Quz\nZ88WZZKmV69eePbsWZFvVy6NiIqKAsdxuHnzpq6TWnng4eFRIAFI+Rx9+vQRPPf4DTz8JKGUe0ks\nLS3zzIHl9l4sAB8XCcj430TdunURHR0NjsvZUXj58mVJVnNkaIR+GYhkkUVMefLkCXXu3Jni4+PJ\nxMSE3N3dqUKFCsV9Wx+1yIFGZZHl45GSH2hUFllkEV9kEpBFlo9cZBKQRZaPXGQSkEWWj1xkEpBF\nlo9cZBKQRZYSJBYWFpSVlUXDhg0TTWeJIgFjY2OysrKi/fv3E2OMgoKC6OTJk9SlSxeysrIiU1NT\nSe3PmjWLGGM0ZMgQSe189dVXNGDAAOGvq6urpPaIiCpWrEht2rTRiJo1a4pqa8+ePaRUKmn16tWi\n6s0t5cuXp6lTp9Lbt2+F1HXr1q0zOHWXp6cnJSYmUmpqKu3du5cWL14s0h3nL19//TUxxigpKYnG\njBlDO3bsEE95cXsL6uIxWLNmzQK3EK9ZswZubm6SeVwtXrxYCDMulQ1XV1dwHCfE48/OzkZYWBj2\n7dsnqh0TExOYmprC19cXfn5+OH36dL7lunnzZlHt+vn5QalUqiWSkQKxsbEan8cQD8UdO3aA47g8\nOtPT07Fz506MGzdOkmhJEyZMEGwZEOG65LsN79q1S63gQ0JCkJycjJSUFOFYZGQkrl27hmHDhhlU\n6BUrVsTChQthY2MjHDt8+LBkJODo6IgBAwYIz8FXtNx/V6xYYbCtRo0a4cqVK7hy5QoCAgIEm2Fh\nYTh//jweP34s4Pz58+jZs6doz+nu7i5sYZaKBMzMzODm5pYvqelLAr179xY2pqWmpuLSpUu4dOkS\nHj9+DACCfik2LZ06dUrQr5p/QEeUbBJo2bIlXr9+LRTE2bNnQZSTKGTQoEH4+++/1X7of/75xyAi\n4KPgbNiwQTgmJQn4+/sLb//cPYHcfw21dfToUTDGMGPGDJiZmaFWrVqoVauWQHgVK1YUIPZz3rp1\nS3ISWLFiRYE9Rn1JICIiQnjRqG69Njc3h6enJ6Kjo8EYQ0BAgKjPY25ujqCgIOH+TUxM9NVVckmg\nYsWKiImJEQohODgY1tbWatdYWlqiTZs2CA8PF65LTU3VuaCMjIywadMmMMaQkZGhFnacZ3uxScDV\n1RUA1LqZHMfh9evXCAsLw7Vr19R6BNeuXdPbVvXq1fH8+XOMHDlS1GQf2kI1mIkUJNC6dWtERUUV\nGILO3NxcZ70WFhZITEwU4llquqZy5cpYtmyZ3hGZCoJqT8AAPSWXBKysrNR+yMmTJ+d7bYsWLQTG\n1qfA2rRpI7yNW7durVYJ+EYoZsAIPz8/vHr1Su1tz3Ecli9fjjZt2qBNmzZwcHBA//791a7Rx1aX\nLl2QmJiIgIAA0VN+aQueAM6dOyd6T6NmzZpqBPD8+fM80an1TVXHB/tMS0szOLuQrvjss8+E9GYy\nCTCGly9fFrqvv1GjRggNDdWr6zRlyhRwHIfAwEC18Fq9e/cWSEDLvG+FQrUHoNoTyO961bkBfeyd\nOXMGoaGhaNu2LYhIVDLTBi1btoRSqcTp06clmTyztLRUGxYOGzZM2OfPGMPJkyd1icenBj7Jzc6d\nO4VjxsbGQi5HHpUqVRL9uX766SfhGTIyMgzRVXK3Ent7ewuf9+zZQ69fvy7w+qCgIPrjjz+IiGjy\n5Mk62eK3tZYrV46MjY11vFPtxdXVlfbu3SssXfG4du1avt8BQIwxnjh1ktq1a1OrVq3IycmJzp8/\nT4mJifTnn3/SqFGj6O+//6auXbuSs7OzIY9UoNSqVYsOHz5MREQ7duygjIwM0W0kJibSt99+S9Om\nTaPXr1/TmjVryMLCQjh/9epVysrK0llvmTJlqGnTpkREZG5uTmPHjqV169bR1atXKTw8XA3Pnz+n\ny5cv06hRo8jc3Fy0Z+MlPT1ddJ3F3gsorCdQvXp13L59W+gFaPtmNzExwatXr3Dr1i2dIgJZWFgI\nEzxRUVHo1KkTrKys0Lt3b2FOoH79+ga/yTTN/F+9elXr7+hqz9LSEhEREXj69Cl+//13DBkyBEuW\nLAFjDIcOHcKyZctw9OhR0d9iPBo2bAilUoknT57omzijUNSvXx8bN24U4vyp4sSJE3oPgXr16qVx\nWZBPspKamoq0tDQhXT1//urVq6LMu6j2BN69eyccNzc3R6dOnbB27Vp4enpq83z6DQeIyJGILhLR\nIyIKJqIfPhy3JqJ/iOj5h79WH44riOhXInpBREFE1MwQEvDy8lJbEtS24Fq0aIG4uDhMmTJF78Kv\nUKEChgwZgqlTpwpLaTExMfjiiy/0jsS7atUqnRu/o6Oj0B01ZDhQ2LN+//33QoNxcnISTXflypWF\npKoNGjQQ/d47duyoVTj6grJbF4Y+ffrg8OHD8PX11Wp14eeffxbs3rlzx+BnHD16tKBv5cqVaj4Q\nHMchMTERM2bMKEyP3iRQlT40ZCKyIKJnRORCRMuJaMaH4zOIaNmHzz2I6BTlkEEbIgooahIwNTXF\nmjVrwJhBjhVq4NfwHz16ZJCe3Mt9V69ehYODQ4HfUV0+NGRisDCUKVMGx44dA2M6xa0rFCNGjBAm\nBA2NK5gb1apV0/jm14TcaealhJGRERYvXgzGchyJDMgfCHd3d2Flgkd6ejqePHkCd3d3NG/eHLVq\n1dIm9Lg4E4NEdJSIOhPRUyKqqkIUTz98/o2IvlG5XriuKEigZcuWWL58ufAdsUiAHw4YSgKqE4CF\n9QAGDBiQpwfAcVyBqyOGonz58oiKisKhQ4dE0/n27VuBBMS+X9U3bm6cPn1aLWv1unXrJCs3TbC3\ntxcmJo8dO6a3HlUnMn5YrCeZGk4CRFSDiF4TUUUiSlQ5ruD/J6LjROSucu48EbUoChIwNTXF8uXL\nhbF7amqqaAk0fHx8DP4xidR7AvmRgJ+fH/bt2yeKs9Bff/2FP//8U+sEFU2aNEFqamqJIIFOnTrl\neUOqwszMDAMHDlQbT4vdEykMv/32m1AX9dUxbtw4tedSTX5auXJl9O7dG3/99Re+//77wnQZRgJE\nVIGIbhNRvw//J+Y6/04XEiAibyK69QH53niDBg3w9OlT4UdctGgRbG1t81zXokULYQjAWI5H4dCh\nQ0X7MX///XcwxrB06VKD9Li6uiIsLCzP233AgAGFug37+fmhTZs2Otnjx8rx8fHw8vLK42SlCiMj\nI/j6+oIxhq+++kqUchs2bBgyMjKgVCrx66+/itrA1q9fny8BREREgIjUSIAxaVx6C8L27dsNJoEy\nZcpg586dwjPY2NjA1NQUtWrVwpkzZ4Tjks0JfGiwpkR0hogma+rmk4TDASJSa9z8D7xo0SK0bdsW\nkyZNQmhoKOLi4oTzqampohIA0f9nnTGUBIiowLe8JrdhT09P9O/fXy9bVlZW2LFjh1A2sbGxmD9/\nPipXrpznWtWY9mI5xCxZskTIrajqfWkorK2t1eL+q+LIkSNo0qQJiHJSsL18+RKM5aSmS0lJETVH\nZUHw9PQU7vHmzZsG6XJyckLoB9+X/CDlxKCCiHYR0Zpcx1eQ+sTg8g+fvyD1icGbWtgotCJrM/HD\nQwrf/ri4OAAQhQSINC8R5nYbZozB09PTYFsKhQINGzaEn5+foD8pKQmrVq1C7dq1Ubt2bbi5ueHk\nyZNgjGHPnj16O9WowtHREampqVAqlbh7966ov4emXY8xMTHw8fHJsyw3atQotesMzXNgamqar7ej\nmZkZGjZsiP9r7+qjoqr69bNRQLxgImSZvKWhlNHHa9ylJvimZHR1uUDNMknf16WmSTc1V5JXzLCV\nS5OlqanlxXK9ft0CM6+JH2V+YeLrR35yUYaQQtMGFFTkyznnuX/MzGkGZoBhzjgDc561nsVwzl57\n/84+Z5757X32/v3WrVunCEBlZaUqSXB69uzJzMzMetctSRJ/+eWXprzRabYIxJgqOAvgtIlDAYTA\n6OrrAOwF0MlCNFYB+AXAOTQyH9AUERBC8L777lPGV7b466+/cseOHUxKSnJJIk29Xk9Zlrlv374G\nXeqm0vzr3pAnMGrUqEbfHDhCPz8/Dho0yGrnoC2qlWnpueeeU+YCxo0bp+r9qGuzXq+362lERUXx\n5s2bPHnyJBctWuT0kulZs2axqKiIc+fOZVJSEpOSkrh582bOnz+fp0+fVmy6e/cui4uLGR8fr9p1\n+/r6MjMzkxkZGVy7di1v3LjhyGvwlrts2MzAwEAmJiZyzpw5Vg/A0qVLrdb5u4LmRKfHjh1T5Vey\nLs2TgTNnzmzW2N8RBgcH85NPPuGZM2esePz4cY4cOZI+Pj6qtPP666/TYDAwJyfHqXf0tmiePykr\nK+OHH37IZ555psHyvr6+qv04fP/99416ozU1NVY7UD2ENkVASz6iQYP3QEs+okGDhvrQRECDBi+H\nJgIaNHg5NBHQoMHLoYmABg1eDk0ENGjwcmgioEGDl0MTAQ0avBwtXgQmT54Mg8FgFYdQTWzduhUl\nJSVWcQBLSkowZswYl7TX2jFlyhSlH/v3769q3Z07d0ZaWhoWL16MjIwMrFy5EmlpaUhOTsaDDz7o\n8jR1LRbuXjLsyLLhupw0aZKSfWjatGmqL7PMzMy0GVvOvGlj8eLFziSC8EpWVVUp+yPMUY/V4IgR\nI1hRUUFJkqxyG1j+/9RTTxEwLiEePHiwS0KdhYWFMTU1lWfOnFF2ba5atYq9evVSrY0nnniCb7zx\nBtPS0pQdkk38DthcNtwWLRhvvfWWSyK6muHn56ckrzx58iTCw8OxadMm5ObmYsWKFXj33Xexfft2\nHD582Kl2unfvjhkzZuCVV15BVFQUgoKCUFZWhpKSEqVMZGQkfH198dprryE7OxtZWVmN1pucnFzv\nWFxcHPz9/fHdd98px+5FclAAGDx4MPz8/FSvNyoqCuvWrUO7du3www8/QK/XY8OGDcr5+++/H8uW\nLUO3bt1w+/ZtrFu3Dr1790ZsbKzDbfn7+yMsLAwvv/wyJkyYgLNnz+LEiRP48ssvMXHiREycOBE9\nevSwanvq1KkYN24cOnTo4PS1xsfHY9OmTVbP/c2bN9GhQ4fm1+9uL8AZT+DUqVPKbi214wcAf6br\nkmWZ7dq1Y8eOHRkQEEDAGKyitLSUpaWljIyMbHYb7du3588//6y0c+rUKd6+fZuFhYU8cuSIwpqa\nGqXMTz/91KS6zb+4TaFer1c4b9481XMSJCYmUqfT8fr166p7Ahs3bmw0ctHSpUtZWlrKy5cv886d\nO0xISHC4nSeffJJHjx5tdPNQTk4OU1JSGB8fz/j4eL766qtcvny5U9c4dOhQvv/++7xz5w6rq6t5\n6NAhrlixgv369WPfvn1ZUlJCvV7f2A7Jlr+LsC7NIlBcXKzqAwuAvXv35tWrV0mS586ds1mmX79+\nrKqq4jvvvNPsdizTpjWVrhCBuqyqquKxY8cYHh7udF8uXLiQ1dXVlCSJsbGxqovA7t27GxWBOXPm\nKGWaGyIuIyNDuQeXLl1ieno6169fz1u3binHKysrlYAmarFXr168fPmy0m/dunWzOj9s2DBKksTi\n4uLGdrhqIuAICwsLSRpjFTYUeXfHjh2srKxkRESEQ/UHBgYyOTnZ6he+LgsKCpwSAUtP4siRI5w7\ndy5TU1M5YMAATps2jXPnzrU6X1tbW08MCgoKnOrH/v37s6ioiLIsMzMzk/3791euQy0R2Lp1q2Jv\n3XNt2rSxEh5nwovl5eVRlo35DM25E/r06cPS0lJu376da9euZVlZGdesWaPqs3j06FFKksRbt27Z\n9NDMArFgwYLG6mq9IjB27FhVO33EiBGsrq6mLMvMyMhosOy8efMoy8ZY8E2tv3fv3ty/f3+9L3d1\ndTV37drF0NBQhoaGMjAwUPn8448/KuXMGZnVZnx8PEeOHKkkQLX3xWoqIyMjee3aNUqSxPnz5zMg\nIICDBg1Shh9q5TaIi4tTfuXrDguTk5NpMBhYXl7OGTNmOJU0Zvz48Vb364svvlACnZrLjBo1SrWg\nLAB44MABVlRUMC8vr176O19fX3777besqalhRUUFo6OjG6uv9YlAUVERr1+/ruosc8eOHXnhwgXK\nsjEmXWJiYoPlIyIiHBaBzZs3Wz1MJFlQUNDgvMaVK1coyzINBgNfeOEF1a63Ln19ffnpp58qAlBZ\nWdmseh555BHlLcpXX32luKnJycmUJIm5ubmq2RwSEsK9e/fSYDCwqqqK06dPZ0hIiJJXsry8XJU5\njjZt2vDFF1/kZ599ZuXBXbp0ySX3Yvbs2ayqqmJlZWW96ETR0dHMysqiJEm8fv16UwPDtj4RkGV1\nsrtYsmfPnsrNXb58eaPx6IKDg3njxg2HRODIkSNKG1euXOHUqVMZHBxst/ywYcNoMBgoyzI///xz\nlzxwZq5Zs8bKC2hOBqfOnTtTp9NRlmWeP3/eKu2YWQR0Oh0ff/xxfvzxx9y5cyf1ej1PnDihTLw6\nyujoaB47dkx5Jbh3715WV1c3exKwMcbExFgJ+eLFi1Wt38/PjzqdjpIkMSUlxercrFmzeOPGDeUe\nOTAp3vpEgKTqIrBkyRLlxjYlNtxjjz3m8LxEx44dmzTh5uPjw40bNyr2TJo0SfWH2ZK//fablQA4\nuvYiLCxMCc0uyzK3bNnC6OhofvDBB8p787rMz89ndnY2U1NTmx1RuS7Xr1/v9FDGEXbq1MkqrNiE\nCROcrrN9+/Z2J23NHtY333zjaL2tSwTatm2rxPxT84aaXwuWlJQwJCSk0fLmePA6nU71h2vMmDHK\nw1VbW8vY2FiXPMQ+Pj5ctGiRVaDTnTt3OhxQ1fwrX/dhtcXi4mIuXLhQtQxRZvbo0YM6nU6ZIxg6\ndKhL+qwuIyIiePbsWeV+DR8+3Ok6w8LCmJWVxYsXL/LUqVP86KOPOGrUKJJkeXl5cxK7ti4RmD59\nOmVZdirHmy2aJ+DMySsaYt++fXn79m3Ksqx6Yo3g4GAePnzYkdVgzebbb79t9QXNyspiaGiow/VY\nziXYEoHCwkLls5o5CCxZWFhIg8HAP/74wyUJTxpiQECAsuYjPz+fXbt2VaXeoKAgZZg0bdo0yrLM\n0aNHNyfgrSYCTaF5ONCQCISFhXHs2LFKGHK9Xq96tGPLhCsHDhxwWcKM7t27My8vT/ly5uTkNMkD\nssXa2lqWl5cri4JmzZql8KGHHmJgYKDSzqOPPqr6tURGRlKSJJaUlHDgwIE0GAysqalRLZFKU9ih\nQwfm5ORQlmW+9957qtY9fPhwFhUVOZMXovWJwLVr1/j000+r2tGJiYnKLLytCaWHH36Yubm5VuPa\nhQsXqmpDeHi4soiooqKi0XDaztBSAGpra5163Zqens64uDgGBATYdVXNbbkiQYz5bcDkyZMJQHk9\n2VjiV7WZkJCgPBtq1dmnTx/u3r3b5mIhB9i6RGDZsmXcs2eP6jewa9euSqrrq1evKqmdEhMTuWXL\nFmXW28w9e/ao5vaZaU55Jssy09PTXfrAWrrrak3M2eMDDzygXJcaGXksGRAQwKysLJJUhMy84lOt\n7EepqamMiYlptJzaIhAWFsY9e/ZQkiTu3r3bmbpalwjU1NS4RAQAcMCAAaysrFQ8grpffFk2JviM\njY11yS5Ccxv79++vl1JLTR4+fFgRgOPHjzu1kKYpTEpKUtpr7pDDHseNG6fsGjSLwO+//05JklR7\ng3ThwgVu27at0XJpaWmUZZmHDh1yus22bdvy4MGDyupNR1em1qFNEWjR8QQyMzNdUm92djZWrVqF\nsrIy+Pj4IDw8XDlXVVWFDRs2YODAgdi3bx8MBoOqbZv3vEuShIsXL+Lu3buq1m9GSkoKoqKiABh3\nSMbFxaG6utolbZlxr2IwREZGIiIiQvUdiyQRHh6O4OBgm+eDgoKwcuVKTJkyBQaDAV9//bXTbS5b\ntgwxMTEAgCFDhiA/P9/pOuvB3V5Acz2BrVu3qr5Roy6ff/555b336tWruXr1ape7zG+++aYyzHBV\nG6NHj1Y29OTm5rr8mszMzs6mXq/nrl27mr0oyB6Dg4N54MABZbHQtWvXFM9g48aNqrSRnJysLIDa\ntm0bExISOGTIEHbp0oXbtm1TUs7Lsuys207AOPyoqamhJEkcMWIETZm6nGHrGg60Ro4ePVoZBqiV\nD9AWp0+ffk8X07Q29unThwsWLODatWt58OBBpqenMz09nePHj3dm0s6K5sVCM2fOdDqLsgU1EfBk\n+vv78/jx45RlmS+99JJL24qKimJZWRmXLFni9uvWeE+pJSTVoMHLoSUk1aBBQ31oIqBBg5fDUwKN\nlgK4Y/rbkhAKzWZXo6XZC3iuzY/YOugRcwIAIIQ4YWu84snQbHY9Wpq9QMuzWRsOaNDg5dBEQIMG\nL4cnicB/u9uAZkCz2fVoafYCLcxmj5kT0KBBg3vgSZ6ABg0a3AC3i4AQ4j+EEBeFEAVCiNnutsce\nhBBFQohzQojTQogTpmOdhBA/CCF0pr+2t5fdOxu/FELohRDnLY7ZtFEYscLU72eFEM96kM2pQogr\npr4+LYQYanHuv0w2XxRCvOQmm/8ihNgvhPg/IUSuEGK66bhH97VduHnPQBsAvwB4FIAfgDMAnnD3\nXgY7thYBCK1zbDGA2abPswF87GYb/wbgWQDnG7MRwFAAuwAIAP0A/MuDbE4F8K6Nsk+YnhF/AN1N\nz04bN9jcBcCzps9BAPJNtnl0X9ujuz2BPgAKSBaSrAXwFYAEN9vkCBIA/NP0+Z8AhrvRFpA8BOBG\nncP2bEwAsJ5GHAXQUQjR5d5Y+ifs2GwPCQC+IllD8hKAAhifoXsKkldJ/mz6fBtAHoCu8PC+tgd3\ni0BXAMUW/182HfNEEMD3QoiTQojJpmMPkLxq+nwNwAPuMa1B2LPR0/v+P02u85cWwyyPs1kI0Q1A\nbwD/Qgvta3eLQEtCDMlnAQwB8JYQ4m+WJ2n0+zz6VUtLsNGEzwCEA/grgKsAlrjXHNsQQgQC+AbA\nDJK3LM+1oL52uwhcAfAXi//DTMc8DiSvmP7qAXwLoxv6h9mtM/3Vu89Cu7Bno8f2Pck/SEokZQDp\n+NPl9xibhRC+MArAJpJbTYdbXF8D7heB4wB6CiG6CyH8ALwGYLubbaoHIcS/CSGCzJ8BxAE4D6Ot\n/zAV+weA/3WPhQ3Cno3bAfzdNHPdD8BNC1fWragzXh4BY18DRptfE0L4CyG6A+gJ4Jgb7BMAvgCQ\nR3KpxakW19cA3Pt2wGLmNB/Gmd4Ud9tjx8ZHYZyVPgMg12wngBAAPwLQAdgLoJOb7fwfGN3nuzCO\nOyfasxHGmepVpn4/B+DfPcjmDSabzsL4BepiUT7FZPNFAEPcZHMMjK7+WQCnTRzq6X1tj9qKQQ0a\nvBzuHg5o0KDBzdBEQIMGL4cmAho0eDk0EdCgwcuhiYAGDV4OTQQ0aPByaCKgQYOXQxMBDRq8HP8P\nDFKPe3NbWUQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7NloDy08CQ9",
        "colab_type": "text"
      },
      "source": [
        "#### Model\n",
        "Let's construct the computation graph. Below are the parameters:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dqjWHTo08CQ-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# parameters \n",
        "N_STEPS = 28\n",
        "N_INPUTS = 28\n",
        "N_NEURONS = 150\n",
        "N_OUTPUTS = 10\n",
        "N_EPHOCS = 10"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gpFu75e88CRB",
        "colab_type": "text"
      },
      "source": [
        "And finally, here is a figure of the RNN-based classification model we are building:\n",
        "\n",
        "![alt txt](https://docs.google.com/drawings/d/e/2PACX-1vQWhELhewvq_bHgqwf4vwDb5B9DN9-jAxeTF9Y73zr-OsW6OXC-ngxAfojivXyZEhjzLXceTZU2Ncz3/pub?w=550&h=600)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HukX7ZjP8CRC",
        "colab_type": "text"
      },
      "source": [
        "And here is the code for the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "14jNZxeg8CRD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# parameters \n",
        "N_STEPS = 28\n",
        "N_INPUTS = 28\n",
        "N_NEURONS = 150\n",
        "N_OUTPUTS = 10\n",
        "\n",
        "N_EPHOCS = 10\n",
        "\n",
        "BATCH_SIZE = 64\n",
        "#model = ImageRNN(BATCH_SIZE, N_STEPS, N_INPUTS, N_NEURONS, N_OUTPUTS) #64,28,28,150,10\n",
        "#logits = model(images.view(-1, 28,28)) #64,28,28\n",
        "\n",
        "\n",
        "class ImageRNN(nn.Module):\n",
        "    def __init__(self, batch_size, n_steps, n_inputs, n_neurons, n_outputs):\n",
        "        super(ImageRNN, self).__init__()\n",
        "        \n",
        "        self.n_neurons = n_neurons\n",
        "        self.batch_size = batch_size\n",
        "        self.n_steps = n_steps\n",
        "        self.n_inputs = n_inputs\n",
        "        self.n_outputs = n_outputs\n",
        "        \n",
        "        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons) #28,150\n",
        "        \n",
        "        self.FC = nn.Linear(self.n_neurons, self.n_outputs) #150, 10\n",
        "        \n",
        "    def init_hidden(self,):\n",
        "        # (num_layers, batch_size, n_neurons)\n",
        "        return (torch.zeros(1, self.batch_size, self.n_neurons)) # 1, 64, 150\n",
        "        \n",
        "    def forward(self, X):\n",
        "        # transforms X to dimensions: n_steps X batch_size X n_inputs\n",
        "        X = X.permute(1, 0, 2)  #28x 64 x 28\n",
        "        self.batch_size = X.size(1) #64\n",
        "        self.hidden = self.init_hidden() #1, 64, 150\n",
        "        \n",
        "        # lstm_out => n_steps, batch_size, n_neurons (hidden states for each time step)\n",
        "        # self.hidden => 1, batch_size, n_neurons (final state from each lstm_out)\n",
        "        print(X.shape, self.hidden.shape)\n",
        "        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)      \n",
        "        out = self.FC(self.hidden)\n",
        "        \n",
        "        return out.view(-1, self.n_outputs) # batch_size X n_output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Yw8fFlp8CRJ",
        "colab_type": "text"
      },
      "source": [
        "The `ImageRNN` model is doing the following:\n",
        "- The initialization function `__init__(...)` declares a few variables, and then a basic RNN layer `basic_rnn` followed by a fully-connected layer `self.FC`.\n",
        "- The `init_hidden` function initializes hidden weights with zero values. \n",
        "The `forward` function accepts an input of size `n_steps X batch_size X n_neurons`. Then the data flows through the RNN layer and then through the fully-connected layer. \n",
        "- The output are the log probabilities of the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bJUCKm_U8CRK",
        "colab_type": "text"
      },
      "source": [
        "#### Testing the model with some samples\n",
        "A very good practice encouraged by PyTorch developers throughout their documentation, and which I really like and highly recommend, is to always test the model with a portion of the dataset before actual training. This is to ensure that you have the correct dimension specified and that the model is outputing the information you expect. Below I show an example of how to test your model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5G6kDd5Jv8WD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "78dd416a-1404-475a-bb7f-3441ab524477"
      },
      "source": [
        "rnn = nn.RNN(10, 20, 2) #input size, hidden size, number of layers\n",
        "input = torch.randn(5, 3, 10) #sqe length, batch, input_size\n",
        "h0 = torch.randn(2, 3, 20)   # num_layers, batch, hidden_size\n",
        "output, hn = rnn(input, h0) # \n",
        "output.shape # seq_len, batch, hidden_size\n"
      ],
      "execution_count": 132,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([5, 3, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_K408jbaZ7u-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "88b13a29-4d5d-4db6-9265-6c019cc9fa13"
      },
      "source": [
        "dataiter = iter(trainloader)\n",
        "images, labels = dataiter.next()\n",
        "X = images.view(-1, 28,28).shape\n",
        "X"
      ],
      "execution_count": 146,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([64, 28, 28])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P6W0jwK08CRM",
        "colab_type": "code",
        "outputId": "7aff8f27-0b1f-45da-f7ce-2cd9dd3d8db0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "dataiter = iter(trainloader)\n",
        "images, labels = dataiter.next()\n",
        "model = ImageRNN(BATCH_SIZE, N_STEPS, N_INPUTS, N_NEURONS, N_OUTPUTS)\n",
        "logits = model(images.view(-1, 28,28))\n",
        "#print(logits[0:10])"
      ],
      "execution_count": 150,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([28, 64, 28]) torch.Size([1, 64, 150])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BKDkckp98CRQ",
        "colab_type": "text"
      },
      "source": [
        "#### Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46Qke-c08CRR",
        "colab_type": "text"
      },
      "source": [
        "Now let's look at the code for training the image classification model. But first, let's declare a few helper functions needed to train the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BoDWoBuy8CRS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "# Device\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Model instance\n",
        "model = ImageRNN(BATCH_SIZE, N_STEPS, N_INPUTS, N_NEURONS, N_OUTPUTS)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "def get_accuracy(logit, target, batch_size):\n",
        "    ''' Obtain accuracy for training round '''\n",
        "    corrects = (torch.max(logit, 1)[1].view(target.size()).data == target.data).sum()\n",
        "    accuracy = 100.0 * corrects/batch_size\n",
        "    return accuracy.item()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uvCzj28o8CRX",
        "colab_type": "text"
      },
      "source": [
        "Before training a model in PyTorch, you can programatically specify what device you want to use during training; the `torch.device(...)` function tells the program that we want to use the GPU if one is available, otherwise the CPU will be the default device.\n",
        "\n",
        "Then we create an instance of the model, `ImageRNN(...)``, with the proper parameters. The criterion represents the function we will use to compute the loss of the model. The `nn.CrossEntropyLoss()` function basically applies a log softmax followed by a negative log likelihood loss operation over the output of the model. To compute the loss, the function needs both the log probabilities and targets. We will see later in our code how to provide this to the criterion.\n",
        "\n",
        "For training, we also need an optimization algorithm which helps to update weights based on the current loss. This is achieved with the `optim.Adam` optimization function, which requires the model parameters and a learning rate. Alternatively, you can also use `optim.SGD` or any other optimization algorithm that's available. \n",
        "\n",
        "The `get_accuracy(...)` function simply computes the accuracy of the model given the log probabilities and target values. As an exercise, you can write code to test this function as we did with the model before.\n",
        "\n",
        "Let's put everything together and train our image classification model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LaAmGhZT8CRX",
        "colab_type": "code",
        "outputId": "af2d6cbf-1fb7-41a9-9e44-2c2a20f9a746",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "for epoch in range(N_EPHOCS):  # loop over the dataset multiple times\n",
        "    train_running_loss = 0.0\n",
        "    train_acc = 0.0\n",
        "    model.train()\n",
        "    \n",
        "    # TRAINING ROUND\n",
        "    for i, data in enumerate(trainloader):\n",
        "         # zero the parameter gradients\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        # reset hidden states\n",
        "        model.hidden = model.init_hidden() \n",
        "        \n",
        "        # get the inputs\n",
        "        inputs, labels = data\n",
        "        inputs = inputs.view(-1, 28,28) \n",
        "\n",
        "        # forward + backward + optimize\n",
        "        outputs = model(inputs)\n",
        "\n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        train_running_loss += loss.detach().item()\n",
        "        train_acc += get_accuracy(outputs, labels, BATCH_SIZE)\n",
        "         \n",
        "    model.eval()\n",
        "    print('Epoch:  %d | Loss: %.4f | Train Accuracy: %.2f' \n",
        "          %(epoch, train_running_loss / i, train_acc/i))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch:  0 | Loss: 0.7149 | Train Accuracy: 75.81\n",
            "Epoch:  1 | Loss: 0.2770 | Train Accuracy: 91.46\n",
            "Epoch:  2 | Loss: 0.2099 | Train Accuracy: 93.58\n",
            "Epoch:  3 | Loss: 0.1766 | Train Accuracy: 94.50\n",
            "Epoch:  4 | Loss: 0.1638 | Train Accuracy: 94.94\n",
            "Epoch:  5 | Loss: 0.1457 | Train Accuracy: 95.44\n",
            "Epoch:  6 | Loss: 0.1347 | Train Accuracy: 95.81\n",
            "Epoch:  7 | Loss: 0.1299 | Train Accuracy: 95.87\n",
            "Epoch:  8 | Loss: 0.1228 | Train Accuracy: 96.06\n",
            "Epoch:  9 | Loss: 0.1140 | Train Accuracy: 96.45\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "93-DMqi58CRf",
        "colab_type": "text"
      },
      "source": [
        "We can also compute accuracy on the testing dataset to test how well the model performs on the image classification task. As you can see below, our RNN model is performing very well on the MNIST classification task.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzeqYT4N8CRi",
        "colab_type": "code",
        "outputId": "57dbeaa9-e43c-47d8-ddc6-2e32e3c0e2fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "test_acc = 0.0\n",
        "for i, data in enumerate(testloader, 0):\n",
        "    inputs, labels = data\n",
        "    inputs = inputs.view(-1, 28, 28)\n",
        "\n",
        "    outputs = model(inputs)\n",
        "\n",
        "    test_acc += get_accuracy(outputs, labels, BATCH_SIZE)\n",
        "        \n",
        "print('Test Accuracy: %.2f'%( test_acc/i))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Accuracy: 96.97\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FD98PsQi8CRv",
        "colab_type": "text"
      },
      "source": [
        "### Final Words\n",
        "Please notice that we are not using GPU in this tutorial since the models we are building are relatively simple. As an exercise, you can take a look at the [PyTorch documentation](https://pytorch.org/docs/stable/notes/cuda.html) to learn how to program specific operations to execute on the GPU. You can then try to optimize the code to run on the GPU. If you need help with this, reach out to me on [Twitter](https://twitter.com/omarsar0).\n",
        "\n",
        "That's it for this tutorial. Congratulations! You are now able to implement a basic RNN in PyTorch. You also learned how to apply RNNs to solve a real-world, image classification problem.\n",
        "\n",
        "In the next tutorial, we will do more advanced things with RNNs and try to solve even more complex problems, such as sarcasm detection and sentiment classification. Until next time!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jKAm1xB08CRx",
        "colab_type": "text"
      },
      "source": [
        "### References\n",
        "- [A Simple Neural Network from Scratch with PyTorch and Google Colab](https://github.com/omarsar/pytorch_intro_neural_network/blob/master/nn.ipynb)\n",
        "- [Hands on Machine Learning with Scikit-learn and Tensorflow](http://shop.oreilly.com/product/0636920052289.do)"
      ]
    }
  ]
}