13_Transfer_learning_part2_and_Fine_tuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tYmg8Bx21LKJ"
      },
      "source": [
        "<center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">به نام خدا</div></center>\n",
        "<h1><center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\"> Transfer Learning - Part 2</h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z1TZTJBZ1LKM"
      },
      "source": [
        "# Transfer Learning - Part 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DCqhM-Qz1LKN"
      },
      "source": [
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "در این نوت بوک Fine Tuning بررسی خواهد شد.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "collapsed": true,
        "id": "JBVXV2UI1LKP",
        "outputId": "00339a95-6dd4-431a-a03f-c2bbf3a60789",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5\n",
            "58889256/58889256 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "from tensorflow import keras\n",
        "from keras.applications import VGG16\n",
        "\n",
        "conv_base = VGG16(weights='imagenet',\n",
        "                  include_top=False,\n",
        "                  input_shape=(150, 150, 3))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "collapsed": true,
        "id": "AogLq3uI1LKQ"
      },
      "outputs": [],
      "source": [
        "from keras import models\n",
        "from keras import layers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(conv_base)\n",
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(256, activation='relu'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7bFAufO1LKQ"
      },
      "source": [
        "This is what our model looks like now:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "yCGi7X1n1LKR",
        "outputId": "52b1c11c-8911-4de3-e41d-36ef5352a9f6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " vgg16 (Functional)          (None, 4, 4, 512)         14714688  \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 8192)              0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 256)               2097408   \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,812,353\n",
            "Trainable params: 16,812,353\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "9Nnpm8gX1LKS",
        "outputId": "bd8ce0c3-26e8-42c2-bd48-3e53c6a83675",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " vgg16 (Functional)          (None, 4, 4, 512)         14714688  \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 8192)              0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 256)               2097408   \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,812,353\n",
            "Trainable params: 16,812,353\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "collapsed": true,
        "id": "BIxIfKss1LKT"
      },
      "outputs": [],
      "source": [
        "conv_base.trainable = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "4iS_E_HB1LKU",
        "outputId": "f49a8875-9cd1-46ef-d38d-109951a31e05",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " vgg16 (Functional)          (None, 4, 4, 512)         14714688  \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 8192)              0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 256)               2097408   \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,812,353\n",
            "Trainable params: 2,097,665\n",
            "Non-trainable params: 14,714,688\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "base_dir = 'D:/dataset/catDog/catVsdog'\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "test_dir = os.path.join(base_dir, 'test')"
      ],
      "metadata": {
        "id": "v3B4z1iD1iiK"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#or"
      ],
      "metadata": {
        "id": "7VqwDamJ1svE"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os, shutil\n",
        "#for run in Colab\n",
        "\n",
        "!mkdir ~/.kaggle\n",
        "!mv kaggle.json ~/.kaggle\n",
        "! kaggle competitions download -c dogs-vs-cats\n",
        "!unzip dogs-vs-cats.zip\n",
        "!unzip -qq train.zip\n",
        "!rm -r dogs-vs-cats.zip sampleSubmission.csv test1.zip train.zip sample_data/\n",
        "\n",
        "# The path to the directory where the original\n",
        "# dataset was uncompressed\n",
        "original_dataset_dir = '/content/train'\n",
        "\n",
        "# The directory where we will\n",
        "# store our smaller dataset\n",
        "base_dir = '/content/subset'\n",
        "os.makedirs(base_dir, exist_ok=True)\n",
        "\n",
        "\n",
        "# Directories for our training,\n",
        "# validation and test splits\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "os.mkdir(train_dir)\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "os.mkdir(validation_dir)\n",
        "test_dir = os.path.join(base_dir, 'test')\n",
        "os.mkdir(test_dir)\n",
        "# Directory with our training cat pictures\n",
        "train_cats_dir = os.path.join(train_dir, 'cats')\n",
        "os.mkdir(train_cats_dir)\n",
        "\n",
        "# Directory with our training dog pictures\n",
        "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
        "os.mkdir(train_dogs_dir)\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
        "os.mkdir(validation_cats_dir)\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
        "os.mkdir(validation_dogs_dir)\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "test_cats_dir = os.path.join(test_dir, 'cats')\n",
        "os.mkdir(test_cats_dir)\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "test_dogs_dir = os.path.join(test_dir, 'dogs')\n",
        "os.mkdir(test_dogs_dir)\n",
        "\n",
        "# Copy first 1000 cat images to train_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(train_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 cat images to validation_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(validation_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 cat images to test_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(test_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy first 1000 dog images to train_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(train_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 dog images to validation_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(validation_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 dog images to test_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(test_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)"
      ],
      "metadata": {
        "id": "ItlhuoOO1ttp",
        "outputId": "6000fb08-4ed1-46db-b7f7-fba10376ab1f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /root/.kaggle/kaggle.json'\n",
            "Downloading dogs-vs-cats.zip to /content\n",
            " 99% 801M/812M [00:07<00:00, 176MB/s]\n",
            "100% 812M/812M [00:08<00:00, 101MB/s]\n",
            "Archive:  dogs-vs-cats.zip\n",
            "  inflating: sampleSubmission.csv    \n",
            "  inflating: test1.zip               \n",
            "  inflating: train.zip               \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(train_dir, validation_dir, test_dir)"
      ],
      "metadata": {
        "id": "14AFs-VK17m7",
        "outputId": "493877e8-bc57-488e-e0fc-090e46f74e22",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/subset/train /content/subset/validation /content/subset/test\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SmHxJxW61LKV",
        "outputId": "939b29ba-a0e5-4ad0-a721-564c0ed15410"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 2000 images belonging to 2 classes.\n",
            "Found 1000 images belonging to 2 classes.\n",
            "Epoch 1/30\n",
            "74s - loss: 0.4465 - acc: 0.7810 - val_loss: 0.2056 - val_acc: 0.9120\n",
            "Epoch 2/30\n",
            "72s - loss: 0.2738 - acc: 0.8905 - val_loss: 0.1239 - val_acc: 0.9550\n",
            "Epoch 3/30\n",
            "72s - loss: 0.2088 - acc: 0.9145 - val_loss: 0.1194 - val_acc: 0.9560\n",
            "Epoch 4/30\n",
            "72s - loss: 0.1835 - acc: 0.9280 - val_loss: 0.1025 - val_acc: 0.9550\n",
            "Epoch 5/30\n",
            "72s - loss: 0.1642 - acc: 0.9330 - val_loss: 0.0903 - val_acc: 0.9680\n",
            "Epoch 6/30\n",
            "72s - loss: 0.1360 - acc: 0.9410 - val_loss: 0.0794 - val_acc: 0.9740\n",
            "Epoch 7/30\n",
            "72s - loss: 0.1426 - acc: 0.9465 - val_loss: 0.0968 - val_acc: 0.9560\n",
            "Epoch 8/30\n",
            "72s - loss: 0.1013 - acc: 0.9580 - val_loss: 0.1411 - val_acc: 0.9430\n",
            "Epoch 9/30\n",
            "72s - loss: 0.1177 - acc: 0.9500 - val_loss: 0.2105 - val_acc: 0.9310\n",
            "Epoch 10/30\n",
            "72s - loss: 0.0949 - acc: 0.9620 - val_loss: 0.0900 - val_acc: 0.9710\n",
            "Epoch 11/30\n",
            "72s - loss: 0.0915 - acc: 0.9655 - val_loss: 0.1204 - val_acc: 0.9630\n",
            "Epoch 12/30\n",
            "72s - loss: 0.0782 - acc: 0.9645 - val_loss: 0.0995 - val_acc: 0.9650\n",
            "Epoch 13/30\n",
            "72s - loss: 0.0717 - acc: 0.9755 - val_loss: 0.1269 - val_acc: 0.9580\n",
            "Epoch 14/30\n",
            "72s - loss: 0.0670 - acc: 0.9715 - val_loss: 0.0994 - val_acc: 0.9680\n",
            "Epoch 15/30\n",
            "71s - loss: 0.0718 - acc: 0.9735 - val_loss: 0.0558 - val_acc: 0.9790\n",
            "Epoch 16/30\n",
            "72s - loss: 0.0612 - acc: 0.9780 - val_loss: 0.0870 - val_acc: 0.9690\n",
            "Epoch 17/30\n",
            "71s - loss: 0.0693 - acc: 0.9765 - val_loss: 0.0972 - val_acc: 0.9720\n",
            "Epoch 18/30\n",
            "71s - loss: 0.0596 - acc: 0.9785 - val_loss: 0.0832 - val_acc: 0.9730\n",
            "Epoch 19/30\n",
            "71s - loss: 0.0497 - acc: 0.9800 - val_loss: 0.1160 - val_acc: 0.9610\n",
            "Epoch 20/30\n",
            "71s - loss: 0.0546 - acc: 0.9780 - val_loss: 0.1057 - val_acc: 0.9660\n",
            "Epoch 21/30\n",
            "71s - loss: 0.0568 - acc: 0.9825 - val_loss: 0.2012 - val_acc: 0.9500\n",
            "Epoch 22/30\n",
            "71s - loss: 0.0493 - acc: 0.9830 - val_loss: 0.1384 - val_acc: 0.9610\n",
            "Epoch 23/30\n",
            "71s - loss: 0.0328 - acc: 0.9905 - val_loss: 0.1281 - val_acc: 0.9640\n",
            "Epoch 24/30\n",
            "71s - loss: 0.0524 - acc: 0.9860 - val_loss: 0.0846 - val_acc: 0.9760\n",
            "Epoch 25/30\n",
            "71s - loss: 0.0422 - acc: 0.9845 - val_loss: 0.1002 - val_acc: 0.9670\n",
            "Epoch 26/30\n",
            "71s - loss: 0.0617 - acc: 0.9825 - val_loss: 0.0858 - val_acc: 0.9760\n",
            "Epoch 27/30\n",
            "71s - loss: 0.0568 - acc: 0.9830 - val_loss: 0.0889 - val_acc: 0.9700\n",
            "Epoch 28/30\n",
            "71s - loss: 0.0296 - acc: 0.9915 - val_loss: 0.1406 - val_acc: 0.9620\n",
            "Epoch 29/30\n",
            "71s - loss: 0.0432 - acc: 0.9890 - val_loss: 0.1535 - val_acc: 0.9650\n",
            "Epoch 30/30\n",
            "71s - loss: 0.0354 - acc: 0.9885 - val_loss: 0.1832 - val_acc: 0.9510\n"
          ]
        }
      ],
      "source": [
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "from keras import optimizers\n",
        "\n",
        "\n",
        "train_datagen = ImageDataGenerator(\n",
        "      rescale=1./255,\n",
        "      rotation_range=40,\n",
        "      width_shift_range=0.2,\n",
        "      height_shift_range=0.2,\n",
        "      shear_range=0.2,\n",
        "      zoom_range=0.2,\n",
        "      horizontal_flip=True,\n",
        "      fill_mode='nearest')\n",
        "\n",
        "# Note that the validation data should not be augmented!\n",
        "test_datagen = ImageDataGenerator(rescale=1./255)\n",
        "\n",
        "train_generator = train_datagen.flow_from_directory(\n",
        "        # This is the target directory\n",
        "        train_dir,\n",
        "        # All images will be resized to 150x150\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        # Since we use binary_crossentropy loss, we need binary labels\n",
        "        class_mode='binary')\n",
        "\n",
        "validation_generator = test_datagen.flow_from_directory(\n",
        "        validation_dir,\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        class_mode='binary')\n",
        "\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer=optimizers.RMSprop(lr=2e-5),\n",
        "              metrics=['acc'])\n",
        "\n",
        "history = model.fit(\n",
        "      train_generator,\n",
        "      epochs=30,\n",
        "      validation_data=validation_generator,\n",
        "      verbose=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vQhA6GLP1LKY"
      },
      "source": [
        "Let's plot our results again:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dgHn_a-I1LKY",
        "outputId": "59c8b356-2dee-4383-e0ef-66ed4f3248dc"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FdX5wPHvCwIh7CQIArKoVERljaAVBFzRirjgAnGv\nxVpRsdr+VKhQFbu41KVWxYpLjSBqqdAKFAMWlaqEQkBABBUwbIZFZFMIeX9/nElyE+9N5ib35m7v\n53nuk3tnzsw9M3PzzplzzpwRVcUYY0xqqBPrDBhjjKk9FvSNMSaFWNA3xpgUYkHfGGNSiAV9Y4xJ\nIRb0jTEmhVjQT0EiUldE9ohIh0imjSUROUZEIt7/WETOFJF1AZ9Xi8gAP2mr8V1/FZF7qru8MX4c\nFusMmKqJyJ6Aj+nA98Ah7/ONqpoTzvpU9RDQONJpU4GqHhuJ9YjIDcCVqjooYN03RGLdxlTGgn4C\nUNXSoOuVJG9Q1XdCpReRw1S1qDbyZkxV7PcYX6x6JwmIyAMi8pqITBGR3cCVInKKiHwoIt+IyGYR\neUJE6nnpDxMRFZFO3udXvPmzRGS3iPxXRDqHm9abf66IfCYiu0TkSRH5QESuDZFvP3m8UUTWishO\nEXkiYNm6IvInEdkuIl8AQyrZP2NFZGqFaU+JyKPe+xtEZJW3PZ97pfBQ6yoQkUHe+3QR+ZuXtxVA\nnwppx4nIF956V4jIBd70E4E/AwO8qrNtAft2QsDyP/e2fbuI/ENEjvCzb8LZzyX5EZF3RGSHiGwR\nkV8HfM9vvH3yrYjkiUjbYFVpIvJ+yXH29ucC73t2AONEpIuIzPe+Y5u335oFLN/R28ZCb/7jIpLm\n5fm4gHRHiMg+EckItb2mCqpqrwR6AeuAMytMewA4AAzFncgbAicB/XBXc0cBnwGjvfSHAQp08j6/\nAmwDsoB6wGvAK9VIeziwGxjmzfslcBC4NsS2+MnjW0AzoBOwo2TbgdHACqA9kAEscD/noN9zFLAH\naBSw7q+BLO/zUC+NAKcD+4Hu3rwzgXUB6yoABnnvHwbeBVoAHYGVFdJeBhzhHZORXh5ae/NuAN6t\nkM9XgAne+7O9PPYE0oC/APP87Jsw93MzYCtwG9AAaAr09ebdDeQDXbxt6Am0BI6puK+B90uOs7dt\nRcBNQF3c7/FHwBlAfe938gHwcMD2fOLtz0Ze+lO9eZOAiQHfcwcwPdb/h4n8inkG7BXmAQsd9OdV\nsdydwOve+2CB/JmAtBcAn1Qj7fXAewHzBNhMiKDvM48nB8z/O3Cn934BrpqrZN55FQNRhXV/CIz0\n3p8LrK4k7T+Bm733lQX9DYHHAvhFYNog6/0E+In3vqqg/xLwYMC8prh2nPZV7Zsw9/NVwKIQ6T4v\nyW+F6X6C/hdV5GF4yfcCA4AtQN0g6U4FvgTE+7wUuDjS/1ep9LLqneTxVeAHEekqIv/yLte/Be4D\nMitZfkvA+31U3ngbKm3bwHyo+y8tCLUSn3n09V3A+kryC/AqMMJ7P9L7XJKP80XkI6/q4RtcKbuy\nfVXiiMryICLXiki+V0XxDdDV53rBbV/p+lT1W2An0C4gja9jVsV+PhIX3IOpbF5VKv4e24jINBHZ\n6OXhxQp5WKeu00A5qvoB7qqhv4icAHQA/lXNPBmsTj+ZVOyu+CyuZHmMqjYF7sWVvKNpM64kCoCI\nCOWDVEU1yeNmXLAoUVWX0mnAmSLSDlf99KqXx4bAG8DvcFUvzYF/+8zHllB5EJGjgKdxVRwZ3no/\nDVhvVd1LN+GqjErW1wRXjbTRR74qqmw/fwUcHWK5UPP2enlKD5jWpkKaitv3B1yvsxO9PFxbIQ8d\nRaRuiHy8DFyJuyqZpqrfh0hnfLCgn7yaALuAvV5D2I218J3/BHqLyFAROQxXT9wqSnmcBowRkXZe\no97/VZZYVbfgqiBexFXtrPFmNcDVMxcCh0TkfFzds9883CMizcXdxzA6YF5jXOArxJ3/foYr6ZfY\nCrQPbFCtYArwUxHpLiINcCel91Q15JVTJSrbzzOADiIyWkQaiEhTEenrzfsr8ICIHC1OTxFpiTvZ\nbcF1GKgrIqMIOEFVkoe9wC4RORJXxVTiv8B24EFxjeMNReTUgPl/w1UHjcSdAEwNWNBPXncA1+Aa\nVp/FNbhGlapuBS4HHsX9Ex8NLMGV8CKdx6eBXGA5sAhXWq/Kq7g6+tKqHVX9BrgdmI5rDB2OO3n5\nMR53xbEOmEVAQFLVZcCTwMdemmOBjwKWnQusAbaKSGA1Tcnys3HVMNO95TsA2T7zVVHI/ayqu4Cz\ngEtwJ6LPgIHe7IeAf+D287e4RtU0r9ruZ8A9uEb9YypsWzDjgb64k88M4M2APBQB5wPH4Ur9G3DH\noWT+Otxx/l5VF4a57aaCksYRYyLOu1zfBAxX1fdinR+TuETkZVzj8IRY5yXR2c1ZJqJEZAiup8x+\nXJe/g7jSrjHV4rWPDANOjHVekoFV75hI6w98gavLPge4yBreTHWJyO9w9wo8qKobYp2fZGDVO8YY\nk0KqLOmLyGQR+VpEPgkxX7zbrdeKyDIR6R0w7xoRWeO9rolkxo0xxoSvypK+iJyGu338ZVU9Icj8\n84BbcHdE9gMeV9V+XteuPNzt+gosBvqo6s7Kvi8zM1M7depUjU0xxpjUtXjx4m2qWlkXacBHQ66q\nLhBvsK0QhuFOCAp86PVZPgIYBMxV1R0AIjIXNyjWlMq+r1OnTuTl5VWVLWOMMQFEpKq70oHINOS2\no/wt1wXetFDTf0BERnkj+OUVFhZGIEvGGGOCiYveO6o6SVWzVDWrVasqr06MMcZUUySC/kbKjz/S\n3psWaroxxpgYiUTQnwFc7fXiORnYpaqbgTnA2SLSQkRa4EYunBOB7zPGGFNNVTbkisgUXKNspogU\n4MbQqAegqs8Ab+N67qzFDe96nTdvh4jcjxsXBeC+kkZdY4wxseGn986IKuYrcHOIeZOBydXLmjHG\nxJ+cHBg7FjZsgA4dYOJEyK7uUHgxEBcNucaY5JWTA506QZ067m9OTqxzVH05OTBqFKxfD6ru76hR\nibVNFvSNMVGTDEEy0NixsG9f+Wn79rnpicKCvjEmapIhSAbaEGLIt1DT/V7l1ObVkA2tbIyJmnCD\nZLzr0MFdrQSbXlHJVU7JSa/kKgfKtwH4TRcpVtI3xkRNsGAYanoi1P1PnAjp6eWnpae76RX5vcqp\n7ashC/rGmKjxGyQTpe4/OxsmTYKOHUHE/Z00KXiJ3O9VTm1fDVnQN8ZEjd8gGW5pN5ZXBdnZsG4d\nFBe7v6GqYPxe5YRzNRQJFvSNMWELJ+j6CZLhlHajdVUQ6ROJ36uccKqMIkJV4+rVp08fNcbExiuv\nqHbsqCri/r7ySvA06emqLuS6V3p68LR+dexYfn0lr44da5bWz/ZEa5vC/X4/6SoD5KmPGBvzIF/x\nZUHfmNjwG/jCCbqR/m5VFxiDfb9I9dcZjW2qbX6DvlXvGJOgIl0d4bdePRoNj+E0kPqtAw+nnSDZ\nupZWxoK+MXEknJt5Il2v7TfwRavh0W8Dqd868HACeW03psaSBX1jqinSJe1wAnk4pVi/+fQb+Gq9\n4bECv1cF4QTyWG9TrfJTB1SbL6vTN7G2fr3qpZeqzp8fOk24DX9+GurCqVeORr12uGlr2vAYbdE4\nRvEMa8g1qSDS/6hffqnaqZP7z6hfX/W114KnC7cHiZ/g4zeQh/P94TZQJnrgqyjZtqcyFvRN0ot0\nN7vPP1ft0EG1RQvVuXNVTz3VBYvHHvth2lgH6GicSExi8xv0rU7fJKxIjlmydi0MHAh798Kdd8IN\nN8AHH0BaGowZA7/+tWtgLBFOfbHfBsVw6pWjUa8da0VFcPXVcPrpsDFCT9PeuRMuvBD69oU5c9wp\nL+X5OTPU5stK+mWKi1WLimKdi/gVqVLs6tWqbduqZmaqPvjgD0vQhx3m/mZnq37/vVsmWn3AI10d\nEa2bjiKtuFj1uutc/ho0UG3TRvWDD2q2zk8+UT3mGNV69VSPPNKte/Bg1Y8/jkye4w1WvZO4iotV\nZ85UPfFE1R/9SHXPnljnKD5FIpiuXOkCzOGHqy5fHnqdzZu7v2eeqfrtt5WvM9h3+w28RUWqW7eq\nfvddDXeOj22PF8XFqr/8pdsv48e7YH300S5YT5pUvXVOn67auLFq69aq77/v9ufjj7sTO6gOH+5O\n9snEgn6CWrhQdcCAsuAFqr/6VaxzFZ/8BtNQ6X7/exfs27RxwV+18quHF15QrVtXtVcv1c2bw8vr\nyy+rHnGEW1dGhuoVV7hAl53tTiTdu7sAVaeOS9Ohg+q6dRHZTXHvgQfcNt96qzsBqKru2KF6zjlu\n+s9/XnaFVZVDh1Tvvdct17evakFB+fm7drn5jRq5Y3njjaqbNkV2e2LFgn6CWblS9cIL3RFp3Vr1\nL39RPXBA9ac/dT/O/PxY5zC0b79V3bgxNt9dk66Qdeq4QPzpp1WnLbl6ePttd8Lo3LnqkuKXX7qS\n6qWXukBfcZ0l6zn5ZNVhw1RHjVIdN0714YfdlUWXLq7Un8z+/Ge3L66+2gXsQEVFqv/3f25+//6q\nW7ZUvq5du1QvuMClv/Za1f37Q6fdskV19GhXdZeernrPParffFPz7YklC/oJ4quvXGCvU0e1SRPV\n++9X3b27bP62be6S9OSTf/hPES+uvtpdij/8cHzmMVTpHVQ/+6x8Wj9XDx9/rNqqlTsuH35YNn3H\nDtU333Ql06OPLlu+bVvVa65xpf0PPlBdu7b8MQ7mgw9UGzZU7dlTdefOiO2KuPLKK27/XHCB6sGD\nodNNmeL2Rbt2oevjV69W7drVFZCeeKLsiqEqa9eqjhjh8tGypeojj7jCViKyoB/ntm931TZpaa4/\n+JgxqoWFwdO+9JI7Us8+W7t59KOoyHVxbNrU5fGss+LvcjlU6b1t2+Dp/Vw9rFmjetRR7oRw662u\nKqGkaqZxY9Xzz3d1yCtX+g9AFc2e7U6m/fur7t1bvXXEq5kzXYAePLjyEnmJJUvcsWjQwP0/BPrn\nP93vLzOz8hvqKrN4serZZ7vjd9JJ7vgmGgv6caq4uOzyXUT1qqtcNUBVywwe7Jap6hK3tv33v+5X\nNGWK6jPPuBJZZqbqjBmxzlmZYKX3tLSaN2hu2eICRN26qqec4uqK33svsiXFadPc7+Tcc/3Xa8e7\nd991+z8rq6xR3I/CQvd/AKq33eb288SJbv/06hWZNpA33nCFmEaNVCdPrv4JOxYs6Mep8ePdXj/3\n3PDq6T/91F0RZGdHLWvVMmGC+6fbts19XrnSVUmA6i9+obpvX1naWPYi+dWvyqp52rWL3HcXF0e/\nFD5pksv35ZcnfhfevDxXjXnccaGvbCtz8KC7KoayhvERIyJ7DDZsUB00yK37sstctV0isKAfh0qq\naa67rnoliJJeCXPnRj5v1XXyyar9+pWf9t13ZV3wunVzJ7dYjuvy4ouuwa5nz/irevLrj3/U0p4s\n4f52vvvO/fZefz06efNr1Sp3Fdixo2vLqomXXnI9rx56KDql8aIi1d/9zv1ujjxS9T//ifx3RJoF\n/Tgzf76rnz399Opfpu/f72426dLFXz1otG3f7uqx7703+Pw5c1x3yPr13SVzVX3qDxxw3Sjr1/d3\ncqhKcbG72QpUzzjD9e5IZHfd5bbl7rv9pd+2zXUMaN1aS3sr1fSGp+pav161fXuXl0SqL//4Y/c/\nV6eO61kVTtVdQYErcIwZE34X3+qwoB9HVq1y9fHHHVfznhhz57qjFirQ1qbXXnN5adMmdKn8669V\nhw4NHvBLXldf7UrhFYN9VTdcVaaoSPXmm92yI0cmR314cbHrVw6u5B/K6tXuiqBhQ5f2nHNU33rL\nDSR31FHh1aPXNL8rV7reNEcf7f4Hli6tne+OpN27y+4W7tfP9fgJ5ttvXVvWrbe6//XA3++Pfxz9\n36AF/Tjx9dfuH+3ww1W/+CIy6xw50gXIwP7lsTBw4A+Dc7BSeXGx6w4XKqC3bas6ZIjqr39d+cnB\nb5/1/ftVL7nELXPHHfHZjbS6iopc3T6oPvdc2fTiYtdAOnSoOwHXr696/fXuLuMSCxa4eTfcEL38\nbdmimpPj+sm3b1927Lp0id1VRqS89ppqs2aud9bLL7v2hQ8+cO1a/fuXDdfRsKE70T70kDvJTZ2q\npVVz0WRBPw7s3+96daSlle/PXVNbtrhS06BBsetdUFzseq34LZW/8orbD4HpGjRQffrp8ulCda8E\nV0X0179WHsR37Ci7o/nRRyO5xfHj++9dRwAR1VdfdUG2Tx+3zRkZqr/5TeheXiVVRG+9FZm87N2r\nOmuWO7l2717+WA0f7roZR6qwEw/Wry/7fZVcSYm4nkh3362amxu86rXkJrPAE3WkWdCPsUOHykpk\nb7wR+fU/84xbd8U+y7Vl+fLQwTnUgGd+GmhDNfj+4Q+qp53mPg8YUDZsQqANG1SPP961nUyZEsmt\njT9797qhn0v20bHHut9EYG+pYL7/3nVvbNWqZt1/Dx1y3SZLquTq13ftVQ8+qLpoUeL3MqpMUZHq\nn/7kSu7TppX1XKtqmbPPdvspkgXAQBb0Y+yee7TKuteaOHTIXUVkZvr70alGtlfMQw+FDvrh1r/7\nzeehQ6rPP+9KkfXquRJtSalq+XLXFbNJE1faSgU7d7peUjNnhleFtWKFu+r6yU+qd6VYVOSqb8D9\nnT07+W4ei4bt292wG23bRqdh14J+DD3/vNuzo0ZFt/olP99Vsfz0p1WnjfQQu2ee6epsYzFs79at\nqldeqaV1xU884aq7jjgiMRsKY+Hxx93+e+aZ8JYrKnI3FIKryzbhyc93/yP9+0e+YdeCfozMnesa\ndM4+u3bG8PjVr9xRXLCg8nThPjavMnv2uMvUO+6I7Q1Xc+eWjXFz7LGpMyplJBw65E7c6ek/HH8o\nlIMHy8apuf/+6OYvmU2Z4vbhL34R2fVa0K+h4mJXbfLpp24EST8B/JNP3BggJ5xQeyP27dnjgm23\nbpWXHCL52Lx//cst++9/VzvbEbNvn+tJsX17rHOSeAoKXFVZv36VD3im6n7/l17qjvvvflc7+Utm\nJYW155+P3Dr9Bv3DavtJXfFg71748EPYsgW2bg3++vpr9/i2QBkZ0Lp12evww8veZ2S4x+qlp8O/\n/gXNmtXOtjRqBH/+Mwwd6h7p99hjwdN16ADr1wefHq7Zs6FhQxgwIPxlI61hQ7jqqljnIjG1awfP\nPAOXX+4eyTh+fPB0Bw7AiBHw97/DQw+5x0mamnnwQViyBG66CU44wT3OsbaIO0HEj6ysLM3Ly4vq\nd9x0k/uxl6hfv3wwDwzqmZmwZ0/ok8Pu3WXrSU+HBQugT5+oZj+oMWPg8cfhySdh9Ogfzs/JgVGj\nyj9TNj09+HNVq3LssXD00fD22zXLs4kPV10FU6a4ZwL361d+3oEDcNll8NZb8Kc/ud+ZiYzt2yEr\nCw4ehMWLXcypCRFZrKpZVSb0czlQm6/aqN7p3NnVZ376qesBUZPG1n373CiZH37ougzGSlGRuzGn\nTh031Gwwkah//+ILd1n6+OM1ya2JJ998457U1aVL+UdzfvedGyIaVJ98Mnb5S2ZLlrj+/gMG1LwN\nkEjW6QNDgNXAWuCuIPM7ArnAMuBdoH3AvEPAUu81o6rvinbQLwlaTzwR1a+Jid27VXv3dsPCLllS\ns3WFOkE8/bTbf8n2fNFU9+677liX3DW6f7+7AQzcU9xM9Lz6qtvPt9xSs/VELOgDdYHPgaOA+kA+\n0K1CmteBa7z3pwN/C5i3x09GSl7RDvrPPee2esWKqH5NzGzc6EYFbNu2+iMZVta9c9gwN4ZLIo0z\nbvwpaVx8/fWyB4pU98HkJjx33OH29wsvVH8dkQz6pwBzAj7fDdxdIc0K4EjvvQDfBsyLq6A/YoQb\nICyZg1Z+vrtJqWfP6g2uFap7Z4cObr3RHkPExMZ335UNpSAS2Z4lpnIHD7o7mvv2rf5YUX6Dfh0f\n7QPtgK8CPhd40wLlAxd77y8CmohIhvc5TUTyRORDEbkw2BeIyCgvTV5hYaGPLFWPKsybB6efDiJR\n+5qY694dXn8dli+HK674YS+kqmzYEHr67t1wzjk1z6OJPw0auAb/Y4+FF1+E66+PdY5Sx2GHuf/Z\nefOgjp+oXAORWv2dwEARWQIMBDbi6vIBOqprUR4JPCYiR1dcWFUnqWqWqma1atUqQln6oZUrXY+b\nM86I2lfERE4OdOrkfiydOrnP55wDTz3letiMGeNOeH6F6sbZtKn7cZ5+eiRybeLRCSfAp5/C1VfH\nOiepp2VL1wU72vz0098IHBnwub03rZSqbsIr6YtIY+ASVf3Gm7fR+/uFiLwL9MK1EdS63Fz3N5mC\nVsWumOvXu88AN94Ia9fCww/DMcf47243cWLw7p0tWkDPni74G2MSk5+S/iKgi4h0FpH6wBXAjMAE\nIpIpIiXruhuY7E1vISINStIApwIrI5X5cM2bB0cd5UrDyWLs2PLBGdznsWPd+z/8AS6+GH75S9fX\n2o/sbNd/v2NHVw3WsaO7KWf9ehgyJLL5N8bUriqDvqoWAaOBOcAqYJqqrhCR+0TkAi/ZIGC1iHwG\ntAYmetOPA/JEJB+YD/xeVWMS9IuK4N13k6uUD5XXv4Or8vnb3+Ckk2DkSPB731t2NqxbB8XF7m+T\nJm661ecbk9h8DcOgqm8Db1eYdm/A+zeAN4IstxA4sYZ5jIglS2DXruSrz/czvEJ6OsyY4e62HDoU\nPvoo/OEX5sxxdyj37Fmz/BpjYivK7cTxo6Q+f/Dg2ObDr2CNs8FMnOiCeqD0dDc9UOvWrlF3/344\n7zzYscN/XoqLXdA/++zo9ywwxkRXyvwLz5vneibUdHyL2lDSOLt+vet1U9I4GyzwB6t/DzWeTrdu\nMH06rFnjAv+ePf7ys2QJbNtm9fnGJIOUCPrffw/vv5849flVNc5WVLH+vbIB1AYPhtdeg0WL4KKL\n3L6pyuzZ7u9ZZ/nJvTEmnqVE0P/wQ1etkSj1+VU1ztbUhRfC5Mnwzjuucbeqm7fmzHEjhx5+eGS+\n3xgTOykR9HNzXV30aafFOif+hGpkrc7Y96Fcc40be//vf3dVR6Fu3tq1CxYutF47xiSLlAn6WVnQ\nvHmsc+KP38bZmrrtNrj3XnjhBfdgjGCBf948OHTI6vONSRZJH/R374aPP06cqh0Ir3G2piZMgFtu\ngUcfdU/zqWj2bNdH/+STI//dxpjal/RB/733XJ11vDTi+u2KGU7jbE2IuGqeq66CcePgL38pm6fq\n6vPPPBPq1YvO9xtjalfSPyN33jz3OMRTT411TiofJydaQd2POnXg+edd/f3o0a4abORIWL3a5fHu\nu2OXN2NMZCV9ST83F378Y/cA7VgLtytmbapXz3XlHDjQjbD4z3+6Uj5YI64xySSpg/727bB0afzU\n50e7K2ZNpaW54Rp69YJLL4Wnn4auXZNrgDpjUl1SB/35893feKnPr42umDXVpAnMmuUC/erVVso3\nJtkkddCfNw8aN3YjTEZTpMfJibXMTJg7F4YPd2PyG2OSR1I35ObmuhuyotnzJJzG2ZLPY8e6Kp0O\nHVzAj2Ujbijt27vHtxljkotoOM/RqwVZWVma53fQ90oUFMCRR8Ijj7gHiERLp07Bhzbu2NF1tTTG\nmNogIou9R9NWKmmrd+bNc3+jXZ8f742zxhgTKKmDfkYGdO8e3e9JhMZZY4wpkZRBX9XV5w8eHP2H\nfiRK46wxxkCSBv21a12dfm30z6/NcXKMMaamkrL3TsmjEWvrpqzsbAvyxpjEkJQl/dxc1+XwmGNi\nnRNjjIkvSRf0i4vdnbhnnOGqW4wxxpRJuqC/bJkbcydehl4wxph4knRBv7b65xtjTCJKuqCfmws/\n+pGr0zfGGFNeUgX9gwdhwYLI9drxO5CaMcYkiqTqsrloEezZE5mqnXh9ypUxxtREUpX0S+rzBw+u\n+bri+SlXxhhTXUkV9HNzoWdPN+ZOTdlAasaYZJQ0QX//fli4MHL1+TaQmjEmGSVN0N+1C664As4/\nPzLrs4HUjDHJKGkactu0gZdeitz6EukpV8YY41fSBP1osIHUjDHJJmmqd4wxxlTNgr4xxqQQC/rG\nGJNCLOgbY0wKsaBvjDEpxFfQF5EhIrJaRNaKyF1B5ncUkVwRWSYi74pI+4B514jIGu91TSQzb4wx\nJjxVBn0RqQs8BZwLdANGiEi3CskeBl5W1e7AfcDvvGVbAuOBfkBfYLyItIhc9o0xxoTDT0m/L7BW\nVb9Q1QPAVGBYhTTdAG+4M+YHzD8HmKuqO1R1JzAXGFLzbBtjjKkOP0G/HfBVwOcCb1qgfOBi7/1F\nQBMRyfC5LCIySkTyRCSvsLDQb96NMcaEKVINuXcCA0VkCTAQ2Agc8ruwqk5S1SxVzWrVqlWEsmSM\nMaYiP8MwbASODPjc3ptWSlU34ZX0RaQxcImqfiMiG4FBFZZ9twb5NcYYUwN+SvqLgC4i0llE6gNX\nADMCE4hIpoiUrOtuYLL3fg5wtoi08Bpwz/amGWOMiYEqg76qFgGjccF6FTBNVVeIyH0icoGXbBCw\nWkQ+A1oDE71ldwD3404ci4D7vGnGGGNiQFQ11nkoJysrS/Py8mKdDWOMSSgislhVs6pKl3J35Obk\nQKdOUKeO+5uTE+scGWNM7Ump8fRzcmDUqLIHnq9f7z6DjZtvjEkNKVXSHzu2LOCX2LfPTTfGmFSQ\nUkF/w4bwphtjTLJJqaDfoUN4040xJtmkVNCfOBHS08tPS093040xJhWkVNDPzoZJk6BjRxBxfydN\nskZcY0zqSKneO+ACvAV5Y0yqSqmSvjHGpDoL+sYYk0Is6BtjTAqxoG+MMSnEgr4xxqQQC/rGGJNC\nLOgbY0wKsaBvjDEpxIK+McakEAv6xhiTQizoG2NMCrGgb4wxKcSCvjHGpBAL+sYYk0Is6BtjTAqx\noG+MMSnEgr4xxqQQC/rGGJNCLOgbY0wKsaBvjDEpxIK+McakEAv6xhiTQizoG2NMCrGgb4wxKcSC\nvjHGpBBbJzS3AAARfUlEQVQL+sYYk0Is6BtjTAqxoG+MMSnEgr4xxqQQC/rGGJNCfAV9ERkiIqtF\nZK2I3BVkfgcRmS8iS0RkmYic503vJCL7RWSp93om0htgjDHGv8OqSiAidYGngLOAAmCRiMxQ1ZUB\nycYB01T1aRHpBrwNdPLmfa6qPSObbWOMMdXhp6TfF1irql+o6gFgKjCsQhoFmnrvmwGbIpdFY4wx\nkeIn6LcDvgr4XOBNCzQBuFJECnCl/FsC5nX2qn3+IyIDgn2BiIwSkTwRySssLPSfe2OMMWGJVEPu\nCOBFVW0PnAf8TUTqAJuBDqraC/gl8KqINK24sKpOUtUsVc1q1apVhLJkjDGmIj9BfyNwZMDn9t60\nQD8FpgGo6n+BNCBTVb9X1e3e9MXA58CPapppY4wx1eMn6C8CuohIZxGpD1wBzKiQZgNwBoCIHIcL\n+oUi0sprCEZEjgK6AF9EKvPGGGPCU2XvHVUtEpHRwBygLjBZVVeIyH1AnqrOAO4AnhOR23GNuteq\nqorIacB9InIQKAZ+rqo7orY1xhhjKiWqGus8lJOVlaV5eXmxzoYxxiQUEVmsqllVpbM7co0xJoVY\n0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaFWNA3xpgUYkHfGGNSiAV9Y4xJIRb0jTEmhVjQN8aYFGJB\n3xhjUogFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaFWNA3xpgUYkHfGGNSiAV9\nY4xJIRb0jTEmhVjQN8aYFGJB3xhjUogFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSN\nMSaFWNA3xpgUYkHfGGNSyGGxzoAxJn4cPHiQgoICvvvuu1hnxYSQlpZG+/btqVevXrWWt6BvjClV\nUFBAkyZN6NSpEyIS6+yYClSV7du3U1BQQOfOnau1DqveMcaU+u6778jIyLCAH6dEhIyMjBpdiVnQ\nN8aUYwE/vtX0+FjQN8aYFGJB3xhTbTk50KkT1Knj/ubk1Gx927dvp2fPnvTs2ZM2bdrQrl270s8H\nDhzwtY7rrruO1atXV5rmqaeeIqemmU1Q1pBrjKmWnBwYNQr27XOf1693nwGys6u3zoyMDJYuXQrA\nhAkTaNy4MXfeeWe5NKqKqlKnTvAy6wsvvFDl99x8883Vy2AS8FXSF5EhIrJaRNaKyF1B5ncQkfki\nskRElonIeQHz7vaWWy0i50Qy88aY2Bk7tizgl9i3z02PtLVr19KtWzeys7M5/vjj2bx5M6NGjSIr\nK4vjjz+e++67rzRt//79Wbp0KUVFRTRv3py77rqLHj16cMopp/D1118DMG7cOB577LHS9HfddRd9\n+/bl2GOPZeHChQDs3buXSy65hG7dujF8+HCysrJKT0iBxo8fz0knncQJJ5zAz3/+c1QVgM8++4zT\nTz+dHj160Lt3b9atWwfAgw8+yIknnkiPHj0YG42dVYUqg76I1AWeAs4FugEjRKRbhWTjgGmq2gu4\nAviLt2w37/PxwBDgL976jDEJbsOG8KbX1Keffsrtt9/OypUradeuHb///e/Jy8sjPz+fuXPnsnLl\nyh8ss2vXLgYOHEh+fj6nnHIKkydPDrpuVeXjjz/moYceKj2BPPnkk7Rp04aVK1fym9/8hiVLlgRd\n9rbbbmPRokUsX76cXbt2MXv2bABGjBjB7bffTn5+PgsXLuTwww9n5syZzJo1i48//pj8/HzuuOOO\nCO0d//yU9PsCa1X1C1U9AEwFhlVIo0BT730zYJP3fhgwVVW/V9UvgbXe+owxCa5Dh/Cm19TRRx9N\nVlZW6ecpU6bQu3dvevfuzapVq4IG/YYNG3LuuecC0KdPn9LSdkUXX3zxD9K8//77XHHFFQD06NGD\n448/Puiyubm59O3blx49evCf//yHFStWsHPnTrZt28bQoUMBd0NVeno677zzDtdffz0NGzYEoGXL\nluHviBryE/TbAV8FfC7wpgWaAFwpIgXA28AtYSyLiIwSkTwRySssLPSZdWNMLE2cCOnp5aelp7vp\n0dCoUaPS92vWrOHxxx9n3rx5LFu2jCFDhgTtu16/fv3S93Xr1qWoqCjouhs0aFBlmmD27dvH6NGj\nmT59OsuWLeP666+P+7uZI9V7ZwTwoqq2B84D/iYivtetqpNUNUtVs1q1ahWhLBljoik7GyZNgo4d\nQcT9nTSp+o244fj2229p0qQJTZs2ZfPmzcyZMyfi33Hqqacybdo0AJYvXx70SmL//v3UqVOHzMxM\ndu/ezZtvvglAixYtaNWqFTNnzgTcTW/79u3jrLPOYvLkyezfvx+AHTt2RDzfVfHTe2cjcGTA5/be\ntEA/xdXZo6r/FZE0INPnssaYBJWdXTtBvqLevXvTrVs3unbtSseOHTn11FMj/h233HILV199Nd26\ndSt9NWvWrFyajIwMrrnmGrp168YRRxxBv379Sufl5ORw4403MnbsWOrXr8+bb77J+eefT35+PllZ\nWdSrV4+hQ4dy//33RzzvlZGSluaQCUQOAz4DzsAF7EXASFVdEZBmFvCaqr4oIscBubhqnG7Aq7h6\n/Lbe9C6qeijU92VlZWleXl6NNsoYUz2rVq3iuOOOi3U24kJRURFFRUWkpaWxZs0azj77bNasWcNh\nh8W+p3uw4yQii1U1K8QiparMvaoWichoYA5QF5isqitE5D4gT1VnAHcAz4nI7bhG3WvVnU1WiMg0\nYCVQBNxcWcA3xph4sWfPHs444wyKiopQVZ599tm4CPg15WsLVPVtXANt4LR7A96vBIJeX6nqRCBK\nTTvGGBMdzZs3Z/HixbHORsTZMAzGGJNCLOgbY0wKsaBvjDEpxIK+McakEAv6xpi4MXjw4B/caPXY\nY49x0003Vbpc48aNAdi0aRPDhw8PmmbQoEFU1R38scceY1/AKHLnnXce33zzjZ+sJwwL+saYuDFi\nxAimTp1abtrUqVMZMWKEr+Xbtm3LG2+8Ue3vrxj03377bZo3b17t9cWjxO90aoyJijFjIMhIwjXS\nsyd4IxoHNXz4cMaNG8eBAweoX78+69atY9OmTQwYMIA9e/YwbNgwdu7cycGDB3nggQcYNqz82I/r\n1q3j/PPP55NPPmH//v1cd9115Ofn07Vr19KhDwBuuukmFi1axP79+xk+fDi//e1veeKJJ9i0aROD\nBw8mMzOT+fPn06lTJ/Ly8sjMzOTRRx8tHaXzhhtuYMyYMaxbt45zzz2X/v37s3DhQtq1a8dbb71V\nOqBaiZkzZ/LAAw9w4MABMjIyyMnJoXXr1uzZs4dbbrmFvLw8RITx48dzySWXMHv2bO655x4OHTpE\nZmYmubm5ETsGFvSNMXGjZcuW9O3bl1mzZjFs2DCmTp3KZZddhoiQlpbG9OnTadq0Kdu2bePkk0/m\nggsuCPnM2Keffpr09HRWrVrFsmXL6N27d+m8iRMn0rJlSw4dOsQZZ5zBsmXLuPXWW3n00UeZP38+\nmZmZ5da1ePFiXnjhBT766CNUlX79+jFw4EBatGjBmjVrmDJlCs899xyXXXYZb775JldeeWW55fv3\n78+HH36IiPDXv/6VP/7xjzzyyCPcf//9NGvWjOXLlwOwc+dOCgsL+dnPfsaCBQvo3LlzxMfnsaBv\njAmqshJ5NJVU8ZQE/eeffx5wY97fc889LFiwgDp16rBx40a2bt1KmzZtgq5nwYIF3HrrrQB0796d\n7t27l86bNm0akyZNoqioiM2bN7Ny5cpy8yt6//33ueiii0pH+rz44ot57733uOCCC+jcuTM9e/YE\nQg/fXFBQwOWXX87mzZs5cOAAnTt3BuCdd94pV53VokULZs6cyWmnnVaaJtLDLydNnX6kn9VpjImN\nYcOGkZuby//+9z/27dtHnz59ADeAWWFhIYsXL2bp0qW0bt26WsMYf/nllzz88MPk5uaybNkyfvKT\nn9RoOOSSYZkh9NDMt9xyC6NHj2b58uU8++yzMR1+OSmCfsmzOtevB9WyZ3Va4Dcm8TRu3JjBgwdz\n/fXXl2vA3bVrF4cffjj16tVj/vz5rF+/vtL1nHbaabz66qsAfPLJJyxbtgxwwzI3atSIZs2asXXr\nVmbNmlW6TJMmTdi9e/cP1jVgwAD+8Y9/sG/fPvbu3cv06dMZMGCA723atWsX7dq5R4m89NJLpdPP\nOussnnrqqdLPO3fu5OSTT2bBggV8+eWXQOSHX06KoF+bz+o0xkTfiBEjyM/PLxf0s7OzycvL48QT\nT+Tll1+ma9eula7jpptuYs+ePRx33HHce++9pVcMPXr0oFevXnTt2pWRI0eWG5Z51KhRDBkyhMGD\nB5dbV+/evbn22mvp27cv/fr144YbbqBXr16+t2fChAlceuml9OnTp1x7wbhx49i5cycnnHACPXr0\nYP78+bRq1YpJkyZx8cUX06NHDy6//HLf3+NHlUMr17bqDK1cp44r4VckAsXFEcqYMSnAhlZODDUZ\nWjkpSvq1/axOY4xJVEkR9Gv7WZ3GGJOokiLox/JZncYkm3ir8jXl1fT4JE0//Vg9q9OYZJKWlsb2\n7dvJyMgIedOTiR1VZfv27aSlpVV7HUkT9I0xNde+fXsKCgooLCyMdVZMCGlpabRv377ay1vQN8aU\nqlevXumdoCY5JUWdvjHGGH8s6BtjTAqxoG+MMSkk7u7IFZFCoPJBNSqXCWyLUHbiQbJtDyTfNiXb\n9kDybVOybQ/8cJs6qmqrqhaKu6BfUyKS5+dW5ESRbNsDybdNybY9kHzblGzbA9XfJqveMcaYFGJB\n3xhjUkgyBv1Jsc5AhCXb9kDybVOybQ8k3zYl2/ZANbcp6er0jTHGhJaMJX1jjDEhWNA3xpgUkjRB\nX0SGiMhqEVkrInfFOj+RICLrRGS5iCwVkfAeJxYHRGSyiHwtIp8ETGspInNFZI33t0Us8xiuENs0\nQUQ2esdpqYicF8s8hkNEjhSR+SKyUkRWiMht3vSEPE6VbE8iH6M0EflYRPK9bfqtN72ziHzkxbzX\nRKS+r/UlQ52+iNQFPgPOAgqARcAIVV0Z04zVkIisA7JUNSFvKhGR04A9wMuqeoI37Y/ADlX9vXdy\nbqGq/xfLfIYjxDZNAPao6sOxzFt1iMgRwBGq+j8RaQIsBi4EriUBj1Ml23MZiXuMBGikqntEpB7w\nPnAb8Evg76o6VUSeAfJV9emq1pcsJf2+wFpV/UJVDwBTgWExzlPKU9UFwI4Kk4cBL3nvX8L9QyaM\nENuUsFR1s6r+z3u/G1gFtCNBj1Ml25Ow1NnjfaznvRQ4HXjDm+77GCVL0G8HfBXwuYAEP9AeBf4t\nIotFZFSsMxMhrVV1s/d+C9A6lpmJoNEissyr/kmIqpCKRKQT0Av4iCQ4ThW2BxL4GIlIXRFZCnwN\nzAU+B75R1SIvie+YlyxBP1n1V9XewLnAzV7VQtJQV7eY+PWL8DRwNNAT2Aw8EtvshE9EGgNvAmNU\n9dvAeYl4nIJsT0IfI1U9pKo9gfa4mo2u1V1XsgT9jcCRAZ/be9MSmqpu9P5+DUzHHexEt9Wrdy2p\nf/06xvmpMVXd6v1TFgPPkWDHyasnfhPIUdW/e5MT9jgF255EP0YlVPUbYD5wCtBcREoehOU75iVL\n0F8EdPFas+sDVwAzYpynGhGRRl5DFCLSCDgb+KTypRLCDOAa7/01wFsxzEtElARHz0Uk0HHyGgmf\nB1ap6qMBsxLyOIXangQ/Rq1EpLn3viGuw8oqXPAf7iXzfYySovcOgNcF6zGgLjBZVSfGOEs1IiJH\n4Ur34B5r+WqibZOITAEG4YaA3QqMB/4BTAM64IbQvkxVE6ZhNMQ2DcJVGyiwDrgxoD48rolIf+A9\nYDlQ7E2+B1cPnnDHqZLtGUHiHqPuuIbauriC+jRVvc+LEVOBlsAS4EpV/b7K9SVL0DfGGFO1ZKne\nMcYY44MFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaF/D8bKEGQJUzoLwAAAABJ\nRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd55306e400>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FOX9wPHPFwg3clo8OD0qdzgiaBERQYpSpVi0HB5o\nbdRqsaW2UrAWsbRqqVIs9Se2Wi0BpGoUKkitogityCGCCAjFgAFEQEAwKCT5/v54dnOxSWaz9+T7\nfr3y2t3Z2Zlndjbfeeb7PPOMqCrGGGP8pUaiC2CMMSb6LLgbY4wPWXA3xhgfsuBujDE+ZMHdGGN8\nyIK7Mcb4kAV3E5KI1BSRoyLSJprzJpKInCMiUe/7KyKDRCSnxOstItLPy7xVWNdfRGRiVT9fwXJ/\nIyJ/i/ZyTeLUSnQBTHSIyNESL+sDXwMFgde3qmpWOMtT1QKgYbTnrQ5U9bxoLEdEbgGuU9VLSiz7\nlmgs2/ifBXefUNWi4BqoGd6iqv8ub34RqaWq+fEomzEm/iwtU00ETrufE5G5InIEuE5ELhSRd0Tk\nkIjsEZEZIpIWmL+WiKiItAu8nh14f7GIHBGR/4pI+3DnDbx/uYh8JCKHReQxEVkhImPLKbeXMt4q\nIttE5KCIzCjx2Zoi8qiIHBCR7cCQCr6fSSIyr8y0mSLySOD5LSKyKbA9/wvUqstbVq6IXBJ4Xl9E\n/h4o20agV5l57xWR7YHlbhSRqwLTuwJ/AvoFUl77S3y3k0t8/rbAth8QkZdE5HQv301lRGR4oDyH\nROQNETmvxHsTRWS3iHwhIptLbOsFIrI2MH2viPze6/pMDKiq/fnsD8gBBpWZ9hvgOHAl7qBeDzgf\n6IM7gzsL+Ai4MzB/LUCBdoHXs4H9QAaQBjwHzK7CvN8AjgDDAu+NB04AY8vZFi9lfBloDLQDPg9u\nO3AnsBFoBTQHlrmffMj1nAUcBRqUWPZnQEbg9ZWBeQS4FDgGdAu8NwjIKbGsXOCSwPNpwJtAU6At\n8GGZea8FTg/sk9GBMrQMvHcL8GaZcs4GJgeeDw6UsTtQF/gz8IaX7ybE9v8G+FvgecdAOS4N7KOJ\nwJbA887ADuC0wLztgbMCz1cBowLPGwF9Ev2/UJ3/rOZevSxX1YWqWqiqx1R1laquVNV8Vd0OzAL6\nV/D551V1taqeALJwQSXceb8DrFPVlwPvPYo7EITksYy/U9XDqpqDC6TBdV0LPKqquap6AHiwgvVs\nBz7AHXQALgMOqurqwPsLVXW7Om8ArwMhG03LuBb4jaoeVNUduNp4yfXOV9U9gX0yB3dgzvCwXIAx\nwF9UdZ2qfgVMAPqLSKsS85T33VRkJLBAVd8I7KMHcQeIPkA+7kDSOZDa+zjw3YE7SJ8rIs1V9Yiq\nrvS4HSYGLLhXL5+UfCEiHUTkFRH5VES+AKYALSr4/KclnudRcSNqefOeUbIcqqq4mm5IHsvoaV24\nGmdF5gCjAs9HB14Hy/EdEVkpIp+LyCFcrbmi7yro9IrKICJjReT9QPrjENDB43LBbV/R8lT1C+Ag\ncGaJecLZZ+UttxC3j85U1S3Az3D74bNAmu+0wKw3AZ2ALSLyrohc4XE7TAxYcK9eynYDfAJXWz1H\nVU8B7sOlHWJpDy5NAoCICKWDUVmRlHEP0LrE68q6as4HBonImbga/JxAGesBzwO/w6VMmgD/8liO\nT8srg4icBTwO3A40Dyx3c4nlVtZtczcu1RNcXiNc+meXh3KFs9wauH22C0BVZ6tqX1xKpibue0FV\nt6jqSFzq7Q/ACyJSN8KymCqy4F69NQIOA1+KSEfg1jis859ATxG5UkRqAXcBp8aojPOBn4jImSLS\nHLinoplV9VNgOfA3YIuqbg28VQeoDewDCkTkO8DAMMowUUSaiLsO4M4S7zXEBfB9uOPcD3E196C9\nQKtgA3IIc4EfiEg3EamDC7Jvq2q5Z0JhlPkqEbkksO6f49pJVopIRxEZEFjfscBfIW4DrheRFoGa\n/uHAthVGWBZTRRbcq7efATfi/nGfwDV8xpSq7gW+DzwCHADOBt7D9cuPdhkfx+XGN+Aa+5738Jk5\nuAbSopSMqh4Cfgpk4xolR+AOUl78GncGkQMsBp4tsdz1wGPAu4F5zgNK5qlfA7YCe0WkZHol+PlX\ncemR7MDn2+Dy8BFR1Y247/xx3IFnCHBVIP9eB3gY107yKe5MYVLgo1cAm8T1xpoGfF9Vj0daHlM1\n4lKexiSGiNTEpQFGqOrbiS6PMX5hNXcTdyIyJJCmqAP8CtfL4t0EF8sYX7HgbhLhImA77pT/28Bw\nVS0vLWOMqQJLyxhjjA9Zzd0YY3woYQOHtWjRQtu1a5eo1RtjTEpas2bNflWtqPswkMDg3q5dO1av\nXp2o1RtjTEoSkcqutAYsLWOMMb5kwd0YY3zIgrsxxviQ3YnJmGrixIkT5Obm8tVXXyW6KMaDunXr\n0qpVK9LSyhtaqGIW3I2pJnJzc2nUqBHt2rXDDcZpkpWqcuDAAXJzc2nfvn3lHwghpdIyWVnQrh3U\nqOEes8K65bMx1dtXX31F8+bNLbCnABGhefPmEZ1lpUzNPSsLMjMhL8+93rHDvQYYE/E4eMZUDxbY\nU0ek+yplau6TJhUH9qC8PDfdGGNMaSkT3HfuDG+6MSa5HDhwgO7du9O9e3dOO+00zjzzzKLXx497\nG/b9pptuYsuWLRXOM3PmTLKilLO96KKLWLduXVSWFW8pk5Zp08alYkJNN8ZEX1aWOzPeudP9n02d\nGlkKtHnz5kWBcvLkyTRs2JC777671DyqiqpSo0boeufTTz9d6XruuOOOqhfSR1Km5j51KtSvX3pa\n/fpuujEmuoJtXDt2gGpxG1csOjFs27aNTp06MWbMGDp37syePXvIzMwkIyODzp07M2XKlKJ5gzXp\n/Px8mjRpwoQJE0hPT+fCCy/ks88+A+Dee+9l+vTpRfNPmDCB3r17c9555/Gf//wHgC+//JLvfe97\ndOrUiREjRpCRkVFpDX327Nl07dqVLl26MHHiRADy8/O5/vrri6bPmDEDgEcffZROnTrRrVs3rrvu\nuqh/Z16kTM09WGOIZk3CGBNaRW1csfif27x5M88++ywZGRkAPPjggzRr1oz8/HwGDBjAiBEj6NSp\nU6nPHD58mP79+/Pggw8yfvx4nnrqKSZMmHDSslWVd999lwULFjBlyhReffVVHnvsMU477TReeOEF\n3n//fXr27Flh+XJzc7n33ntZvXo1jRs3ZtCgQfzzn//k1FNPZf/+/WzYsAGAQ4cOAfDwww+zY8cO\nateuXTQt3lKm5g7uR5WTA4WF7tECuzGxEe82rrPPPrsosAPMnTuXnj170rNnTzZt2sSHH3540mfq\n1avH5ZdfDkCvXr3IyckJueyrr776pHmWL1/OyJEjAUhPT6dz584Vlm/lypVceumltGjRgrS0NEaP\nHs2yZcs455xz2LJlC+PGjWPJkiU0btwYgM6dO3PdddeRlZVV5YuQIpVSwd0YEx/ltWXFqo2rQYMG\nRc+3bt3KH//4R9544w3Wr1/PkCFDQvb3rl27dtHzmjVrkp+fH3LZderUqXSeqmrevDnr16+nX79+\nzJw5k1tvvRWAJUuWcNttt7Fq1Sp69+5NQUFBVNfrhQV3Y8xJEtnG9cUXX9CoUSNOOeUU9uzZw5Il\nS6K+jr59+zJ//nwANmzYEPLMoKQ+ffqwdOlSDhw4QH5+PvPmzaN///7s27cPVeWaa65hypQprF27\nloKCAnJzc7n00kt5+OGH2b9/P3llc1xxkDI5d2NM/CSyjatnz5506tSJDh060LZtW/r27Rv1dfz4\nxz/mhhtuoFOnTkV/wZRKKK1ateKBBx7gkksuQVW58sorGTp0KGvXruUHP/gBqoqI8NBDD5Gfn8/o\n0aM5cuQIhYWF3H333TRq1Cjq21CZhN1DNSMjQ+1mHcbEz6ZNm+jYsWOii5EU8vPzyc/Pp27dumzd\nupXBgwezdetWatVKrvpuqH0mImtUNaOcjxRJri0xxpg4OHr0KAMHDiQ/Px9V5Yknnki6wB4pf22N\nMcZ40KRJE9asWZPoYsSUNagaY4wPWXA3xhgfsuBujDE+ZMHdGGN8yIK7MSYuBgwYcNIFSdOnT+f2\n22+v8HMNGzYEYPfu3YwYMSLkPJdccgmVda2ePn16qYuJrrjiiqiM+zJ58mSmTZsW8XKizYK7MSYu\nRo0axbx580pNmzdvHqNGjfL0+TPOOIPnn3++yusvG9wXLVpEkyZNqry8ZGfB3RgTFyNGjOCVV14p\nujFHTk4Ou3fvpl+/fkX9znv27EnXrl15+eWXT/p8Tk4OXbp0AeDYsWOMHDmSjh07Mnz4cI4dO1Y0\n3+233140XPCvf/1rAGbMmMHu3bsZMGAAAwYMAKBdu3bs378fgEceeYQuXbrQpUuXouGCc3Jy6Nix\nIz/84Q/p3LkzgwcPLrWeUNatW8cFF1xAt27dGD58OAcPHixaf3AI4OCAZW+99VbRzUp69OjBkSNH\nqvzdhmL93I2phn7yE4j2DYa6d4dAXAypWbNm9O7dm8WLFzNs2DDmzZvHtddei4hQt25dsrOzOeWU\nU9i/fz8XXHABV111Vbn3EX388cepX78+mzZtYv369aWG7J06dSrNmjWjoKCAgQMHsn79esaNG8cj\njzzC0qVLadGiRallrVmzhqeffpqVK1eiqvTp04f+/fvTtGlTtm7dyty5c3nyySe59tpreeGFFyoc\nn/2GG27gscceo3///tx3333cf//9TJ8+nQcffJCPP/6YOnXqFKWCpk2bxsyZM+nbty9Hjx6lbt26\nYXzblfNUcxeRISKyRUS2icjJAyYXz/c9EVERqfTSWGNM9VMyNVMyJaOqTJw4kW7dujFo0CB27drF\n3r17y13OsmXLioJst27d6NatW9F78+fPp2fPnvTo0YONGzdWOijY8uXLGT58OA0aNKBhw4ZcffXV\nvP322wC0b9+e7t27AxUPKwxufPlDhw7Rv39/AG688UaWLVtWVMYxY8Ywe/bsoith+/bty/jx45kx\nYwaHDh2K+hWylS5NRGoCM4HLgFxglYgsUNUPy8zXCLgLWBnVEhpjoq6iGnYsDRs2jJ/+9KesXbuW\nvLw8evXqBUBWVhb79u1jzZo1pKWl0a5du5DD/Fbm448/Ztq0aaxatYqmTZsyduzYKi0nKDhcMLgh\ngytLy5TnlVdeYdmyZSxcuJCpU6eyYcMGJkyYwNChQ1m0aBF9+/ZlyZIldOjQocplLctLzb03sE1V\nt6vqcWAeMCzEfA8ADwFV/yaNMb7WsGFDBgwYwM0331yqIfXw4cN84xvfIC0tjaVLl7Ij1A2TS7j4\n4ouZM2cOAB988AHr168H3HDBDRo0oHHjxuzdu5fFixcXfaZRo0Yh89r9+vXjpZdeIi8vjy+//JLs\n7Gz69esX9rY1btyYpk2bFtX6//73v9O/f38KCwv55JNPGDBgAA899BCHDx/m6NGj/O9//6Nr167c\nc889nH/++WzevDnsdVbEy3nAmcAnJV7nAn1KziAiPYHWqvqKiPy8vAWJSCaQCdDG7mxtTLU0atQo\nhg8fXqrnzJgxY7jyyivp2rUrGRkZldZgb7/9dm666SY6duxIx44di84A0tPT6dGjBx06dKB169al\nhgvOzMxkyJAhnHHGGSxdurRoes+ePRk7diy9e/cG4JZbbqFHjx4VpmDK88wzz3DbbbeRl5fHWWed\nxdNPP01BQQHXXXcdhw8fRlUZN24cTZo04Ve/+hVLly6lRo0adO7cueiuUtFS6ZC/IjICGKKqtwRe\nXw/0UdU7A69rAG8AY1U1R0TeBO5W1Qo7ndqQv8bElw35m3oiGfLXS1pmF9C6xOtWgWlBjYAuwJsi\nkgNcACywRlVjjEkcL8F9FXCuiLQXkdrASGBB8E1VPayqLVS1naq2A94Brqqs5m6MMSZ2Kg3uqpoP\n3AksATYB81V1o4hMEZGrYl1AY0z0JOrOayZ8ke4rTx0rVXURsKjMtPvKmfeSiEpkjImJunXrcuDA\nAZo3b17uxUEmOagqBw4ciOjCJrtC1ZhqolWrVuTm5rJv375EF8V4ULduXVq1alXlz1twN6aaSEtL\no3379okuhokTGzjMGGN8yIK7Mcb4kAV3Y4zxIQvuxhjjQxbcjTHGhyy4G2OMD1lwN8YYH7Lgbowx\nPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA9ZcDfGGB+y4G6MMT5kwd0YY3zI\ngrsxxviQBXdjjPEhC+7GGONDFtyNMcaHLLgbY4wPWXA3xhgfsuBujDE+ZMHdGGN8yIK7Mcb4kAV3\nY4zxIQvuxhjjQxbcjTHGhyy4G2OMD1lwN8YYH7LgbowxPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY\n40MW3I0xxoc8BXcRGSIiW0Rkm4hMCPH+bSKyQUTWichyEekU/aIaY4zxqtLgLiI1gZnA5UAnYFSI\n4D1HVbuqanfgYeCRqJfUGGOMZ15q7r2Bbaq6XVWPA/OAYSVnUNUvSrxsAGj0imiMMSZctTzMcybw\nSYnXuUCfsjOJyB3AeKA2cGmoBYlIJpAJ0KZNm3DLaowxxqOoNaiq6kxVPRu4B7i3nHlmqWqGqmac\neuqp0Vq1McaYMrwE911A6xKvWwWmlWce8N1ICmWMMSYyXoL7KuBcEWkvIrWBkcCCkjOIyLklXg4F\ntkaviMYYY8JVac5dVfNF5E5gCVATeEpVN4rIFGC1qi4A7hSRQcAJ4CBwYywLbYwxpmJeGlRR1UXA\nojLT7ivx/K4ol8sYY0wE7ApVY4zxIQvuxhjjQxbcjTHGh3wb3LOyoF07qFHDPWZlJbpExhgTP54a\nVFNNVhZkZkJennu9Y4d7DTBmTOLKZYwx8eLLmvukScWBPSgvz003xpjqwJfBfefO8KYbY4zf+DK4\nlzcmmY1VZoypLnwZ3KdOhfr1S0+rX99NN8aY6sCXwX3MGJg1C9q2BRH3OGuWNaYaY6oPX/aWARfI\nLZgbY6orX9bcjTGmurPgbowxPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA9Z\ncDfGGB+y4G6MMT5kwd0YY3zIgrsxxviQBXdjjPEhC+7GGONDFtyNMcaHLLgbY4wPWXA3xhgfsuBu\njDE+ZMHdGGN8yIK7Mcb4kAV3Y4zxIQvuxhjjQxbcjTHGh6p9cM/KgnbtoEYN95iVlegSGWNM5Kp1\ncM/KgsxM2LEDVN1jZmb1CvD/+x88+WSiS2GMibZqHdwnTYK8vNLT8vLc9Opi8mR3QNu/P9ElMcZE\nU8oF9wMH4NVXo7OsnTvDm+43x4/DP//pnr//fmLLYoyJLk/BXUSGiMgWEdkmIhNCvD9eRD4UkfUi\n8rqItI1+UZ3HH4crrohOTbNNm/Cm+82bb8KhQ+65BXdj/KXS4C4iNYGZwOVAJ2CUiHQqM9t7QIaq\ndgOeBx6OdkGDLrvM5cdffz3yZU2dCvXrl55Wv76bXh28+CI0aACnnmrBvbrJz4cvvkh0KUwseam5\n9wa2qep2VT0OzAOGlZxBVZeqajB7/Q7QKrrFLJaRAU2awL/+FfmyxoyBWbOgbVsQcY+zZrnpfldQ\nAC+9BEOHQs+eFtyrmylT4Lzz4OuvE10SEytegvuZwCclXucGppXnB8DiUG+ISKaIrBaR1fv27fNe\nyhJq1oSBA+G111wNPlJjxkBODhQWusfqENgB3nkH9u6F4cMhPR0+/NDl4E31sGQJfPopLFqU6JKY\nWIlqg6qIXAdkAL8P9b6qzlLVDFXNOPXUU6u8nsGD4ZNP4KOPqryIau/FF6F2bdd+0b07nDgBmzcn\nulQmHvLyYO1a93zOnMSWxcSOl+C+C2hd4nWrwLRSRGQQMAm4SlVjerJ32WXuMRqpmepIFbKz3fd4\nyimu5g6Wmqku3n3X5dzPOw8WLrTcu195Ce6rgHNFpL2I1AZGAgtKziAiPYAncIH9s+gXs7T27eHs\ns11qxoTv/ffh449dSgbgm9+EOnUsuFcXK1a4x0cfdTn37OzElsfERqXBXVXzgTuBJcAmYL6qbhSR\nKSJyVWC23wMNgX+IyDoRWVDO4qJm8GBYutSlE0x4XnzRDbdwVWDv1aoFXbpYcK8uli+Hzp1hyBA4\n66zqdUV2deIp566qi1T1m6p6tqpODUy7T1UXBJ4PUtWWqto98HdVxUuM3GWXwdGjrmEwXvwyDk12\nNvTr57pABqWnu+AejUZqk7wKC+G//4WLLnI9xEaPdt2KP/000SUz0ZZyV6gGDRjggmy8UjN+GYfm\no4/ggw/g6qtLT09Ph3377J/c7zZuhMOHoW9f93r0aBfw589PbLlM9KVscG/SBPr0iV9w98s4NMH8\najDfHmSNqtXD8uXuMRjcO3Z0vaWs10x8fP01fO97sHJl7NeVssEdXGrm3Xfh4MHYr8sv49BkZ7sL\nwVq3Lj29Wzf3aMHd31asgNNPd50SgkaPdsFm27bElau6+MUvXJtXPM6QUzq4Dx7sTimXLo39uvww\nDk1urvsnLpuSAWja1G2LBXd/W77c1dpFiqeNGuVez52buHJVB9nZMGMG/OQnMGxY5fNHKqWDe+/e\n0KhRfPq7+2Ecmpdeco9lUzJBwUZV40+7drm2omBKJqhVK7j4Ytd+ZA3qsZGTAzff7M6aH3ooPutM\n6eCeluYaVuORd/fDODTZ2S7H2qFD6PfT02HLFvjqq/iWy8RHsH/7RRed/N7o0W7fr1sX3zJVB8eP\nw/e/77IMzz3nrgyPh5QO7uBSM9u3uzsKxVoqj0Ozfz+89VbolExQerobUGzjxviVy8TP8uXubDPY\neF7SiBGuspRqvb9SwS9/6doGn3rKXVcQLykf3INDEdjVqhVbuNAF7vJSMmA9ZvxuxQq44AIXxMtq\n1gwuv9zl3QsK4l82v1q4EB55BO64w/WSiaeUD+7nnusaApMpuCfjxU7Z2e576tmz/HnOPtuN727B\n3X+OHHEpl7L59pJGj4bdu+Htt+NXLj/buRNuvBF69IBp0+K//pQP7iIuNfPGG24wpERLxoudjhxx\njc5XX126l0RZNWpA164W3P1o5UqXTqwouF95pTu4W5/3yJ044Xoh5ee7C8Tq1o1/GVI+uINLzRw6\nBKtXJ7okyXmx0+LF7uKJilIyQTYMgT+tWOEO3hdeWP489eu738g//mE38YjUr34F//mP63RxzjmJ\nKYMvgvvAga5GmgypmWS82Ck7240jU1GtLSg93R0oP/mk8nlN6li+3J2VnXJKxfONGeP2f7RuQl8d\nLV7sujtmZsLIkYkrhy+Ce/Pm0KtXcozvnmwXO331Ffzzn+6iiZo1K58/2KhqXeL8Iz/fDbAXqgtk\nWQMHuoqApWaqZtcuuOEGd8X39OmJLYsvgju41Mw777j8ciIl28VOr7/uRs+sqAtkSV27ukfLu/vH\nhg3uN+DlzC0tDa69FhYsSPz/UqrJz3d59mPHXJ69Xr3Elsc3wX3wYPflvvlmYsuRbBc7ZWe7U/FL\nL/U2f6NGrteMBXf/KDtYWGXGjHFnfHYTj/Dcf7/rafR//+fucpVovgnuF17oasjJkJpJloud8vPh\n5Zdh6FB3pyWvbBgCf1mxwg0U5zU1eMEFrguvpWa8mz/fnZ3ffDNcd12iS+P4JrjXqQP9+ydHo2qy\nWL7cXZnqNSUTlJ7urvg9ejQ25TLxo1o8WJhXwZt4/PvfsHdv7MrmB5s3u5vMf//7rj/7Y48lukTF\nfBPcwaVmtmxJvWF4YyU72/WvHTIkvM+lp7ugsGFDbMpl4mfnTtfI56UxtaTRo92VqnYTj9AOHYKf\n/cy1Ua1YAX/4g7vDVdn2tkTyVXD381AE4V71qurGjR48GBo2DG9dNgyBfwQHCwun5g7uHqvp6Zaa\nKaugAJ580t1U/tFHYexYd3ez8ePjNyCYV74K7p06wRln+C+4V+Wq19Wr3fjt4aZkwDUCN25swd0P\nli93jeTBXlDhGD3a9UDbvj365UpFb78N55/v/ve++U1YtcoF+pYtE12y0HwV3EVc7f3f/3aNmX5R\nlates7Ndv/Yrrwx/fSKun64F99S3YoXrbODlGoeyghfgVPebeOzc6b6Liy929xmeO9cF+l69El2y\nivkquIML7gcOwHvvJbok3nhJt4R71asqvPACXHKJG+2vKtLTYf16fx0kq5vDh127SbgpmaA2baBf\nv+p7Ew9V+N3v3P0PXn4Z7rvPNaCOHFnxGE3JwnfBfdAg95gKqRmv6ZZwr3pdvNjlASPpgpmeDl9+\naafkqey//3W/q6oGd3C/oU2bXHqmulm2DCZOdO1Wmze7fuwNGiS6VN75Lri3bOkCUzL0d6+M13RL\nOFe9qsLkye4sIJL+tt27u0dLzaSuFStcOqZPn6ovY8QI1/7Srx9cf3316kE1Y4Y7850717VDpRrf\nBXdwqZkVK04OnMnGa7olnKteFy92DT333hv6pgxede7sUkUW3FPX8uXuIB1ub6mSmjd3AX3cONeO\n062buyhu2TJ/p2p27nT3HP7hDxM/jEBV+TK4Dx7s7lu4bFmiS1KxcNItXq56LVlrv+GGyMpWr567\nhNqCe2o6ccKN4R5u//ZQWrd2dxPauRMeeMBVHvr3h299ywVAP7bL/PnP7vFHP0psOSLhy+B+0UXu\nitVkT81Ee5CxYK190qTIau1BNgxB6lq3zg1gFUm+vaxmzdwZ4Y4dMHOmu3p1+HDXBfmvf/XPGPDH\njrkujt/9buJGc40GXwb3evVcjjDZG1WjOchYNGvtQenp7h/50KHoLM/ET7iDhYWjXj1Xo/3oI5g3\nz1VIbrnF3fz5ySejv754mzMHPv/cpaJSmS+DO7jUzAcfwJ49iS5JxaI1yNirrxbX2qN1pVzwStX1\n66OzvEj8/veuX/FnnyW6JKlhxQpo395d1BcrtWq5MVXWrHEVqfbtXW+vZPi9VJWqGx+ma1fXrz2V\n+Ta4B4cieOgh1+f9xInElieWYlFrh+QZhuCll+AXv4C1a+Gaa/y9L6NB1QX3WNTaQxFxXZAXLnTD\nS0+eHJ+UTh41AAASr0lEQVT1xsLbb7vf+7hxqdGXvSK+De7durmj7x//CD17ukuwL7gA7rwTnnkG\nNm5040T4wauvwrvvhl9rr+wCqtNPhxYtEhvcP/zQdcHr3Rueeso1ko8fn7jypILt2+HTT6PTmBqO\npk3dvsnOdgfiVDRjhtuO0aMTXZIoUNWE/PXq1UtjrbBQdds21XnzVO++W7V/f9WGDVVd3Ua1QQPV\niy5S/elPVRcscPOnmsJC1d69Vdu1U/36a++fmz1btX794u8C3OvZs0vPN3CgakZGeGWaN0916FDV\nPXvC+1xZBw+qnnOOasuWqrm5btrPfubK+tRTkS3bz555xn1HGzbEf92HDqk2bar6ne/Ef92R2rFD\ntWZN1V/8ItElqRiwWj3EWF8H91AKClQ3bVJ99lnVceNUv/Ut1Xr13DcxapTqkSMJKVaVLVrkyj5r\nVnifa9u2dGAP/rVtW3q+8eNV69ZVPXHC23I3biz+Ps85RzUnJ7xyBeXnq15+uWpamury5cXTT5xQ\nHTRItXZt1Xfeqdqy/e6HP1Rt0sT91hNh6lS3/1euTMz6q2rCBNUaNar+m40XC+5hOHHC/SBr1FDt\n0CExNZ6qCNba27YNr9auqioSOriLlJ4vWAv88MPKl5mXp9q1q+qpp6q++KILMK1bq27ZEl7ZVN0/\nGqg+8cTJ7+3fr9q+veoZZ0R+dpAK5s9XHTZM9dVXvZ1dduqkesUVsS9Xeb74QrV5c9UhQxJXhnDl\n5bkyDx+e6JJUzoJ7FbzxhksB1Kvnglqyq2qtXdV7zX3dOjd97tzKl3nHHW7eRYvc6/fec4H+G99w\ny/Hquefccm67rfx51q1zaaRvfSv8A1sq+fhjl0qsUcN9J927q86ZU/6Z1IEDbr6pU+NazJM89JAr\nx4oViS2HV3/9qyvv0qWJLknlLLhX0e7dLjcPqrfc4o7oySiSWruq95z711+71MiECRUvLzvbLWP8\n+NLTN29WbdXK1eL/+9/KyxUM2n37Vr5dwYPArbdWvtxUVFCgOmCAC+5btrgA1KGD2+b27VX/9CfV\nL78s/ZmFC937b72VmDIHHT3qDuoDBya2HF4UFrqDZpcuqdHuZsE9AidOqE6c6L6d9HTVjz5KdIlO\nFkmtPWj2bHdwEHGPZQN7ULduLv9dnp07XSNar16hA3JOjsu/N2ig+vrr5S9n/37XMHzmmd7TLRWl\nb1Ldn/508j4uKFB96SXVCy9077VooXr//e67U3XfR1paclRKHnnElfHNNxNdkootWxb5/1I8WXCP\ngldeUW3WTLVRI9Xnn090aYpFWmsP1/XXu/x2KCdOqPbr52qXFR0E9+xxNaM6dVzPpFDLGTjQvf/u\nu97Llp/vcrtpaamTAvBi61Z3BjNkSOjaZGGhC0pDhxafdd11l2qPHqp9+sS/vKHk5amefrrqxRcn\nd434mmtc5aTsWVCyimpwB4YAW4BtwIQQ718MrAXygRFelpkKwV3V1Tp793bf1F13JUd+d/Hiimsa\nXmvkXk2b5tb32Wcnvzd5snvv2WcrX86BA6rnn++6m82ZU/q9n/7ULacqbR2ff6569tmqp51W3GUy\nleXnuy66jRurfvJJ5fNv2OAOwLVqacjUWCI99pgr07//neiShLZzp/s9/vzniS6Jd1EL7kBN4H/A\nWUBt4H2gU5l52gHdgGf9FtxVXUAfN859W336uL7ziVJY6MrQpk3oA43XXHo4XnvNLee110pPf+st\n19B3/fXel/XFF65NQ6Q4lfLss8UHz6r64AOX9undW/XYsaovJxkE0xl/+1t4n9uxQ/W3v3WPyeLY\nMdfm8q1vJWftfeJE9xv++ONEl8S7aAb3C4ElJV7/EvhlOfP+zY/BPegf/3ApmrQ0V9M8cCD+ZQjW\n2svLMXvtBROOzz5zy5g2rXja/v3un/acc1zADkdenuuqB6o//rHrRz9ggOrx41Uvo6rqCy+4Zd58\nc3IGEi82b3bfx5VXpu42lPX4426/vPpq+J89eNBdlxILwe6P3/1ubJYfK9EM7iOAv5R4fT3wp3Lm\n9XVwV3Wn/Tff7GqeTZq4gPfVV/FZd2W1dlXv/dfD1aSJqxmLuPX36uUOcqtXV215X3/tcp3BA0+o\nlE9V3Htv1Wq9yeDECbd/mzXzV//9r792+/j888M7YL36qsvZi7hKQLQvMHzqKfdbeeON6C431pIy\nuAOZwGpgdZs2beLxPcTM+vWusQtcD485c2J/ReDcuRXX2lVjU3OfPbu4n3XJv9Gjq75MVZdbfuwx\nV1uNloICl5pp0yZ+B92gDz5Qvekm1csuq9rVsw8+6L7Xsu0RfvCXv7htW7iw8nnz8orToJ07q2Zm\nFrcfLVkSnfIUFrrG51Tp/liSpWXi5F//ct0lwdVMYtHtq6BA9YEH3A+8R4+KG3VjkXMv74AR6vgc\n7cbcqliyxJXvz3+O/bqCvVa+853i77ply+LapteU1YYNbkiF730v9YKNF8ePq551lvv9VrR9a9e6\nK2xB9Sc/KW4/Wb68uI//2LGRp0TffrvyilKyimZwrwVsB9qXaFDtXM681S64q7oa6N/+5nLQoHrV\nVdHLEx48WBw4Ro92F4dUJtoB1muqJxYHlqooLHQXQZ15ZuwaV/PzXY7/ggu0qL/5lCmuLeLwYdU7\n73TfT6tWqi+/XPGyjh9X7dnTXc0brfRUMgoOZfHiiye/l5/vrmpNS3Pdbv/1r5PnOXZMddIk17ul\nZcvIuidfe61LNXr5f0o20e4KeQXwUaDXzKTAtCnAVYHn5wO5wJfAAWBjZcv0U3APystzvRUaNXI/\nwFtvjaznwvvvuy5+tWq59EWianReUz2xSAlV1euvu3XPmBHd5R475mp7557rln/WWe4MIVQf6Xfe\ncWPtgOrVV5ffTfP++908yXQtRSycOKH6zW+676RkCjMnp/iq8BEjii/IKs9777kzgOD3unt3eOX4\n5BP3/3n33WFvQlKwi5gSaO9eN85KWpr7y8wMv6vV7NlujJvTT0/8xTnBslRWI49VY25VFBa6gHHa\nadG5WvPzz914LS1bum3KyHADeuXnV/y548dVf/c71wPmlFNUZ84sHdjee88dvEeNiryMqSAry31/\n8+e717Nnu++lUSNXs/dagTlxwrVR1KnjauBPP33yZ7/6yp1BL1yo+uij7n/y2992Z3Qiqtu3R3XT\n4saCexLYsUP1Rz9yudRatVwvm61bK/7M11+7XC24K/uSpdfE7Nkux15RqieZau6qrv0DXL/xSOzY\n4dIu4IZhWLo0/LOobdvcUMXghg7YsMHt627d3AEoEd1qEyE/3+XUO3ZUHTnSfR99+1Y90G7e7K6Q\nBneF8623usdgWrLk77BxY3dQHjkytRutLbgnkdxc1/pft27xRT+heojs2uUu9gheZRhpv+94S5ac\ne0kDB7oBrKqaW83PdwfZhg1VV62KrCyFhe6CrebN3cH+oovUcw8SP5k/3213rVqqv/mN93sFlKeg\nwKXGGjd2322fPqpjxqj++tfut/fOOy7V45eGagvuSWjPHncnofr1Xa1i5EjXfU7VXe3ZsqXrS/7c\nc4ktZyTCacyNR8+a5cvdr/zhh6v2+d/+VqPeb37fPtUbbtCinh/VTUGB6h/+EPnBsiy/BO/KWHBP\nYnv3qt5zjwvkwdPJmjVVzzvP3cmoOohnLf/b33Y1unCvpF21ytUur7kmNoFjy5bUOzsziec1uIub\nN/4yMjJ09erVCVl3sjhwAKZPdzflvewydwPoU05JdKnio1072LHj5Olt20JOTnTXtXKluzn6b38L\nv/ylt898+aW7sXpenrtBeLNm0S2TMVUlImtUNaOy+WrEozAmtObN4YEH4PPP4fnnq09gB9i50/v0\nrCx3MKhRwz1mZYW3rj59YOhQ+P3v4YsvvC1z/HjYuhWefdYCu0lNFtyTQM2aiS5B/LVp4216VhZk\nZrpavqp7zMwMP8Dffz8cPAh//GPly3z5ZZg1C+6+GwYM8L6OSA9CxkSVl9xNLP6qc87deM+5h9u9\nsqJG2mHDim/aXd4yd+923R579AhvbJpk7Clk/AlrUDXJzktvmXAujKoswL73Xuhllfz79rddl9UP\nPwxvW5Ktj7/xL6/B3RpUTVILp+HVy7wjRsCLL7rQW1bTpi51M3Mm/OhH4ZWzRo3QyxSBwsLwlmVM\nRaxB1fjC1KlQv37pafXru+lleWmknTzZPdaqVXqeunXh6FHX8Hr77aXf85JL99qGYGLD2jtC8FK9\nj8WfpWWMV14vdvKaGvn+913qpVWr4puPtG7trmTdu/fkdXvJpYebc0+GoZH9orq1d2A5d1PdeP0n\n37jRBdV77nGvgzfnfuWVk5cZTi7da8BOdDDy24ElFvsomVlwN9WS13/e0aOLAyq4EQNDicVIl7Fq\nfPWy7Yk+sMRCqt1vIFIW3I2pwObNxbcO7Nix/GGBYxGIY3HAiFXX0liIdu05Fe83EAkL7sZUYuxY\nNxzze++VP088b1sYSRrB6zITPeZ+LL5Pr8tM9LZHiwV3Yypx/Li3O2VFu6YZi0Zar4ErmheFVUUi\nU1JWc7fgbkzMRTsYeZ03nANGLGrZiaw9W87dgrsxSSGaV+eWnTeaqZ5wJLr2HIveMvHugWPB3ZgU\nl+gUSiIbfoPzJnu3xUScDVhwNybFJTqNYF02K5eIMxGvwd3GljEmiWVlwaRJbgiFNm3csAtjxsRv\n3ZmZ7oYlQfXru+GQY12GeN7MJRKJGFPIxpYxxgfGjHHBrLDQPcYrsAfXPWuWC6gi7jEegR3Cu5lL\nIoU7plA8x8Cx4G6MKZfXg0u0g1aqDMQWzsB20brxjFcW3I0xEYlF0AonaCZSOGc3kyaVTnGBez1p\nUmzKZsHdGBORWAStRKaEgryejXg9u4l3qskaVI0xEfHjjUpi0ZgcrUZia1A1xsRFquTHwxGLs5F4\np5osuBtjIpIq+fFwxCKFEu9UkwV3Y0xEkiE/Hm2xOhuJZ9dWC+7GmIglsj9+LPjhbMSCuzHGlOGH\ns5Falc9ijDHVz5gxqRXMy7KauzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA8lbGwZEdkHhBhp\nwZMWwP4oFicZ+G2b/LY94L9t8tv2gP+2KdT2tFXVUyv7YMKCeyREZLWXgXNSid+2yW/bA/7bJr9t\nD/hvmyLZHkvLGGOMD1lwN8YYH0rV4D4r0QWIAb9tk9+2B/y3TX7bHvDfNlV5e1Iy526MMaZiqVpz\nN8YYUwEL7sYY40MpF9xFZIiIbBGRbSIyIdHliZSI5IjIBhFZJyIpeVNZEXlKRD4TkQ9KTGsmIq+J\nyNbAY9NEljEc5WzPZBHZFdhP60TkikSWMVwi0lpElorIhyKyUUTuCkxPyf1Uwfak7H4Skboi8q6I\nvB/YpvsD09uLyMpAzHtORGp7Wl4q5dxFpCbwEXAZkAusAkap6ocJLVgERCQHyFDVlL3wQkQuBo4C\nz6pql8C0h4HPVfXBwEG4qarek8hyelXO9kwGjqrqtESWrapE5HTgdFVdKyKNgDXAd4GxpOB+qmB7\nriVF95OICNBAVY+KSBqwHLgLGA+8qKrzROT/gPdV9fHKlpdqNffewDZV3a6qx4F5wLAEl6naU9Vl\nwOdlJg8Dngk8fwb3j5cSytmelKaqe1R1beD5EWATcCYpup8q2J6Upc7RwMu0wJ8ClwLPB6Z73kep\nFtzPBD4p8TqXFN+huJ33LxFZIyKZiS5MFLVU1T2B558CLRNZmCi5U0TWB9I2KZG+CEVE2gE9gJX4\nYD+V2R5I4f0kIjVFZB3wGfAa8D/gkKrmB2bxHPNSLbj70UWq2hO4HLgjkBLwFXW5v9TJ/4X2OHA2\n0B3YA/whscWpGhFpCLwA/ERVvyj5XirupxDbk9L7SVULVLU70AqXqehQ1WWlWnDfBbQu8bpVYFrK\nUtVdgcfPgGzcDvWDvYG8aDA/+lmCyxMRVd0b+McrBJ4kBfdTII/7ApClqi8GJqfsfgq1PX7YTwCq\neghYClwINBGR4C1RPce8VAvuq4BzA63HtYGRwIIEl6nKRKRBoDEIEWkADAY+qPhTKWMBcGPg+Y3A\nywksS8SCATBgOCm2nwKNdX8FNqnqIyXeSsn9VN72pPJ+EpFTRaRJ4Hk9XMeRTbggPyIwm+d9lFK9\nZQACXZumAzWBp1R1aoKLVGUichautg7uZuVzUnF7RGQucAlueNK9wK+Bl4D5QBvc0M7XqmpKNFKW\nsz2X4E71FcgBbi2Rq056InIR8DawASgMTJ6Iy1On3H6qYHtGkaL7SUS64RpMa+Iq3vNVdUogTswD\nmgHvAdep6teVLi/VgrsxxpjKpVpaxhhjjAcW3I0xxocsuBtjjA9ZcDfGGB+y4G6MMT5kwd0YY3zI\ngrsxxvjQ/wPcz2v1dLH/YgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd544300160>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(len(acc))\n",
        "\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.legend()\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NithgimW1LKZ"
      },
      "source": [
        "As you can see, we reach a validation accuracy of about 96%. This is much better than our small convnet trained from scratch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWFbwvb21LKZ"
      },
      "source": [
        "## Fine-tuning\n",
        "\n",
        "Another widely used technique for model reuse, complementary to feature extraction, is _fine-tuning_.\n",
        "Fine-tuning consists in unfreezing a few of the top layers\n",
        "of a frozen model base used for feature extraction, and jointly training both the newly added part of the model (in our case, the\n",
        "fully-connected classifier) and these top layers. This is called \"fine-tuning\" because it slightly adjusts the more abstract\n",
        "representations of the model being reused, in order to make them more relevant for the problem at hand.\n",
        "\n",
        "![fine-tuning VGG16](https://s3.amazonaws.com/book.keras.io/img/ch5/vgg16_fine_tuning.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zogAczrK1LKa"
      },
      "source": [
        "We have stated before that it was necessary to freeze the convolution base of VGG16 in order to be able to train a randomly initialized\n",
        "classifier on top. For the same reason, it is only possible to fine-tune the top layers of the convolutional base once the classifier on\n",
        "top has already been trained. If the classified wasn't already trained, then the error signal propagating through the network during\n",
        "training would be too large, and the representations previously learned by the layers being fine-tuned would be destroyed. Thus the steps\n",
        "for fine-tuning a network are as follow:\n",
        "\n",
        "* 1) Add your custom network on top of an already trained base network.\n",
        "* 2) Freeze the base network.\n",
        "* 3) Train the part you added.\n",
        "* 4) Unfreeze some layers in the base network.\n",
        "* 5) Jointly train both these layers and the part you added.\n",
        "\n",
        "We have already completed the first 3 steps when doing feature extraction. Let's proceed with the 4th step: we will unfreeze our `conv_base`,\n",
        "and then freeze individual layers inside of it.\n",
        "\n",
        "As a reminder, this is what our convolutional base looks like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "ekwobOar1LKa",
        "outputId": "47a5dd55-45ad-445a-f916-0fc04efcff48",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"vgg16\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 150, 150, 3)]     0         \n",
            "                                                                 \n",
            " block1_conv1 (Conv2D)       (None, 150, 150, 64)      1792      \n",
            "                                                                 \n",
            " block1_conv2 (Conv2D)       (None, 150, 150, 64)      36928     \n",
            "                                                                 \n",
            " block1_pool (MaxPooling2D)  (None, 75, 75, 64)        0         \n",
            "                                                                 \n",
            " block2_conv1 (Conv2D)       (None, 75, 75, 128)       73856     \n",
            "                                                                 \n",
            " block2_conv2 (Conv2D)       (None, 75, 75, 128)       147584    \n",
            "                                                                 \n",
            " block2_pool (MaxPooling2D)  (None, 37, 37, 128)       0         \n",
            "                                                                 \n",
            " block3_conv1 (Conv2D)       (None, 37, 37, 256)       295168    \n",
            "                                                                 \n",
            " block3_conv2 (Conv2D)       (None, 37, 37, 256)       590080    \n",
            "                                                                 \n",
            " block3_conv3 (Conv2D)       (None, 37, 37, 256)       590080    \n",
            "                                                                 \n",
            " block3_pool (MaxPooling2D)  (None, 18, 18, 256)       0         \n",
            "                                                                 \n",
            " block4_conv1 (Conv2D)       (None, 18, 18, 512)       1180160   \n",
            "                                                                 \n",
            " block4_conv2 (Conv2D)       (None, 18, 18, 512)       2359808   \n",
            "                                                                 \n",
            " block4_conv3 (Conv2D)       (None, 18, 18, 512)       2359808   \n",
            "                                                                 \n",
            " block4_pool (MaxPooling2D)  (None, 9, 9, 512)         0         \n",
            "                                                                 \n",
            " block5_conv1 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_conv2 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_conv3 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_pool (MaxPooling2D)  (None, 4, 4, 512)         0         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 14,714,688\n",
            "Trainable params: 0\n",
            "Non-trainable params: 14,714,688\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "conv_base.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncdhAU-s1LKb"
      },
      "source": [
        "\n",
        "We will fine-tune the last 3 convolutional layers, which means that all layers up until `block4_pool` should be frozen, and the layers\n",
        "`block5_conv1`, `block5_conv2` and `block5_conv3` should be trainable.\n",
        "\n",
        "Why not fine-tune more layers? Why not fine-tune the entire convolutional base? We could. However, we need to consider that:\n",
        "\n",
        "* Earlier layers in the convolutional base encode more generic, reusable features, while layers higher up encode more specialized features. It is\n",
        "more useful to fine-tune the more specialized features, as these are the ones that need to be repurposed on our new problem. There would\n",
        "be fast-decreasing returns in fine-tuning lower layers.\n",
        "* The more parameters we are training, the more we are at risk of overfitting. The convolutional base has 15M parameters, so it would be\n",
        "risky to attempt to train it on our small dataset.\n",
        "\n",
        "Thus, in our situation, it is a good strategy to only fine-tune the top 2 to 3 layers in the convolutional base.\n",
        "\n",
        "Let's set this up, starting from where we left off in the previous example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "collapsed": true,
        "id": "0qBGqYbp1LKb"
      },
      "outputs": [],
      "source": [
        "conv_base.trainable = True\n",
        "\n",
        "set_trainable = False\n",
        "for layer in conv_base.layers:\n",
        "    if layer.name == 'block5_conv1':\n",
        "        set_trainable = True\n",
        "    if set_trainable:\n",
        "        layer.trainable = True\n",
        "    else:\n",
        "        layer.trainable = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MqIkZc921LKc"
      },
      "source": [
        "Now we can start fine-tuning our network. We will do this with the RMSprop optimizer, using a very low learning rate. The reason for using\n",
        "a low learning rate is that we want to limit the magnitude of the modifications we make to the representations of the 3 layers that we are\n",
        "fine-tuning. Updates that are too large may harm these representations.\n",
        "\n",
        "Now let's proceed with fine-tuning:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fO6pSmmB1LKc",
        "outputId": "c52e869d-ba6c-4c10-f733-5818c1d6e6cc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "100/100 [==============================] - 32s - loss: 0.0215 - acc: 0.9935 - val_loss: 0.0980 - val_acc: 0.9720\n",
            "Epoch 2/100\n",
            "100/100 [==============================] - 32s - loss: 0.0131 - acc: 0.9960 - val_loss: 0.1247 - val_acc: 0.9700\n",
            "Epoch 3/100\n",
            "100/100 [==============================] - 32s - loss: 0.0140 - acc: 0.9940 - val_loss: 0.1044 - val_acc: 0.9790\n",
            "Epoch 4/100\n",
            "100/100 [==============================] - 33s - loss: 0.0102 - acc: 0.9965 - val_loss: 0.1259 - val_acc: 0.9770\n",
            "Epoch 5/100\n",
            "100/100 [==============================] - 33s - loss: 0.0137 - acc: 0.9945 - val_loss: 0.1036 - val_acc: 0.9800\n",
            "Epoch 6/100\n",
            "100/100 [==============================] - 33s - loss: 0.0183 - acc: 0.9935 - val_loss: 0.1260 - val_acc: 0.9750\n",
            "Epoch 7/100\n",
            "100/100 [==============================] - 33s - loss: 0.0141 - acc: 0.9945 - val_loss: 0.1575 - val_acc: 0.9690\n",
            "Epoch 8/100\n",
            "100/100 [==============================] - 33s - loss: 0.0094 - acc: 0.9965 - val_loss: 0.0935 - val_acc: 0.9780\n",
            "Epoch 9/100\n",
            "100/100 [==============================] - 33s - loss: 0.0079 - acc: 0.9985 - val_loss: 0.1452 - val_acc: 0.9760\n",
            "Epoch 10/100\n",
            "100/100 [==============================] - 33s - loss: 0.0127 - acc: 0.9970 - val_loss: 0.1027 - val_acc: 0.9790\n",
            "Epoch 11/100\n",
            "100/100 [==============================] - 33s - loss: 0.0097 - acc: 0.9965 - val_loss: 0.1463 - val_acc: 0.9720\n",
            "Epoch 12/100\n",
            "100/100 [==============================] - 33s - loss: 0.0055 - acc: 0.9980 - val_loss: 0.1361 - val_acc: 0.9720\n",
            "Epoch 13/100\n",
            "100/100 [==============================] - 33s - loss: 0.0274 - acc: 0.9955 - val_loss: 0.1446 - val_acc: 0.9740\n",
            "Epoch 14/100\n",
            "100/100 [==============================] - 33s - loss: 0.0043 - acc: 0.9985 - val_loss: 0.1123 - val_acc: 0.9790\n",
            "Epoch 15/100\n",
            "100/100 [==============================] - 33s - loss: 0.0057 - acc: 0.9975 - val_loss: 0.1912 - val_acc: 0.9700\n",
            "Epoch 16/100\n",
            "100/100 [==============================] - 33s - loss: 0.0144 - acc: 0.9960 - val_loss: 0.1415 - val_acc: 0.9780\n",
            "Epoch 17/100\n",
            "100/100 [==============================] - 33s - loss: 0.0048 - acc: 0.9990 - val_loss: 0.1231 - val_acc: 0.9780\n",
            "Epoch 18/100\n",
            "100/100 [==============================] - 33s - loss: 0.0188 - acc: 0.9965 - val_loss: 0.1551 - val_acc: 0.9720\n",
            "Epoch 19/100\n",
            "100/100 [==============================] - 33s - loss: 0.0160 - acc: 0.9970 - val_loss: 0.2155 - val_acc: 0.9740\n",
            "Epoch 20/100\n",
            "100/100 [==============================] - 33s - loss: 0.0047 - acc: 0.9965 - val_loss: 0.1559 - val_acc: 0.9730\n",
            "Epoch 21/100\n",
            "100/100 [==============================] - 33s - loss: 0.0132 - acc: 0.9980 - val_loss: 0.1518 - val_acc: 0.9740\n",
            "Epoch 22/100\n",
            "100/100 [==============================] - 33s - loss: 0.0086 - acc: 0.9965 - val_loss: 0.1517 - val_acc: 0.9790\n",
            "Epoch 23/100\n",
            "100/100 [==============================] - 33s - loss: 0.0070 - acc: 0.9980 - val_loss: 0.1887 - val_acc: 0.9670\n",
            "Epoch 24/100\n",
            "100/100 [==============================] - 33s - loss: 0.0044 - acc: 0.9985 - val_loss: 0.1818 - val_acc: 0.9740\n",
            "Epoch 25/100\n",
            "100/100 [==============================] - 33s - loss: 0.0159 - acc: 0.9970 - val_loss: 0.1860 - val_acc: 0.9680\n",
            "Epoch 26/100\n",
            "100/100 [==============================] - 33s - loss: 0.0056 - acc: 0.9980 - val_loss: 0.1657 - val_acc: 0.9740\n",
            "Epoch 27/100\n",
            "100/100 [==============================] - 33s - loss: 0.0118 - acc: 0.9980 - val_loss: 0.1542 - val_acc: 0.9760\n",
            "Epoch 28/100\n",
            "100/100 [==============================] - 33s - loss: 0.0031 - acc: 0.9990 - val_loss: 0.1493 - val_acc: 0.9770\n",
            "Epoch 29/100\n",
            "100/100 [==============================] - 33s - loss: 0.0114 - acc: 0.9965 - val_loss: 0.1921 - val_acc: 0.9680\n",
            "Epoch 30/100\n",
            "100/100 [==============================] - 33s - loss: 0.0031 - acc: 0.9990 - val_loss: 0.1188 - val_acc: 0.9830\n",
            "Epoch 31/100\n",
            "100/100 [==============================] - 33s - loss: 0.0068 - acc: 0.9985 - val_loss: 0.1814 - val_acc: 0.9740\n",
            "Epoch 32/100\n",
            "100/100 [==============================] - 33s - loss: 0.0096 - acc: 0.9985 - val_loss: 0.2034 - val_acc: 0.9760\n",
            "Epoch 33/100\n",
            "100/100 [==============================] - 33s - loss: 0.0072 - acc: 0.9985 - val_loss: 0.1970 - val_acc: 0.9730\n",
            "Epoch 34/100\n",
            "100/100 [==============================] - 33s - loss: 0.0047 - acc: 0.9990 - val_loss: 0.2349 - val_acc: 0.9680\n",
            "Epoch 35/100\n",
            "100/100 [==============================] - 33s - loss: 0.0066 - acc: 0.9990 - val_loss: 0.1865 - val_acc: 0.9740\n",
            "Epoch 36/100\n",
            "100/100 [==============================] - 33s - loss: 0.0115 - acc: 0.9975 - val_loss: 0.1933 - val_acc: 0.9750\n",
            "Epoch 37/100\n",
            "100/100 [==============================] - 33s - loss: 0.0101 - acc: 0.9980 - val_loss: 0.1779 - val_acc: 0.9780\n",
            "Epoch 38/100\n",
            "100/100 [==============================] - 33s - loss: 0.0101 - acc: 0.9975 - val_loss: 0.1887 - val_acc: 0.9700\n",
            "Epoch 39/100\n",
            "100/100 [==============================] - 33s - loss: 0.0093 - acc: 0.9980 - val_loss: 0.2159 - val_acc: 0.9720\n",
            "Epoch 40/100\n",
            "100/100 [==============================] - 33s - loss: 0.0049 - acc: 0.9990 - val_loss: 0.1412 - val_acc: 0.9790\n",
            "Epoch 41/100\n",
            "100/100 [==============================] - 33s - loss: 0.0052 - acc: 0.9985 - val_loss: 0.2066 - val_acc: 0.9690\n",
            "Epoch 42/100\n",
            "100/100 [==============================] - 33s - loss: 0.0043 - acc: 0.9990 - val_loss: 0.1860 - val_acc: 0.9770\n",
            "Epoch 43/100\n",
            "100/100 [==============================] - 33s - loss: 0.0031 - acc: 0.9985 - val_loss: 0.2361 - val_acc: 0.9680\n",
            "Epoch 44/100\n",
            "100/100 [==============================] - 33s - loss: 0.0012 - acc: 0.9995 - val_loss: 0.2440 - val_acc: 0.9680\n",
            "Epoch 45/100\n",
            "100/100 [==============================] - 33s - loss: 0.0035 - acc: 0.9985 - val_loss: 0.1428 - val_acc: 0.9820\n",
            "Epoch 46/100\n",
            "100/100 [==============================] - 33s - loss: 0.0111 - acc: 0.9970 - val_loss: 0.1822 - val_acc: 0.9720\n",
            "Epoch 47/100\n",
            "100/100 [==============================] - 33s - loss: 0.0047 - acc: 0.9990 - val_loss: 0.1726 - val_acc: 0.9720\n",
            "Epoch 48/100\n",
            "100/100 [==============================] - 33s - loss: 0.0039 - acc: 0.9995 - val_loss: 0.2164 - val_acc: 0.9730\n",
            "Epoch 49/100\n",
            "100/100 [==============================] - 33s - loss: 0.0060 - acc: 0.9970 - val_loss: 0.1856 - val_acc: 0.9810\n",
            "Epoch 50/100\n",
            "100/100 [==============================] - 33s - loss: 0.0126 - acc: 0.9980 - val_loss: 0.1824 - val_acc: 0.9720\n",
            "Epoch 51/100\n",
            "100/100 [==============================] - 33s - loss: 0.0155 - acc: 0.9965 - val_loss: 0.1867 - val_acc: 0.9710\n",
            "Epoch 52/100\n",
            "100/100 [==============================] - 33s - loss: 0.0059 - acc: 0.9985 - val_loss: 0.2287 - val_acc: 0.9700\n",
            "Epoch 53/100\n",
            "100/100 [==============================] - 33s - loss: 0.0046 - acc: 0.9980 - val_loss: 0.2337 - val_acc: 0.9650\n",
            "Epoch 54/100\n",
            "100/100 [==============================] - 33s - loss: 0.0087 - acc: 0.9970 - val_loss: 0.1168 - val_acc: 0.9820\n",
            "Epoch 55/100\n",
            "100/100 [==============================] - 33s - loss: 0.0046 - acc: 0.9985 - val_loss: 0.1496 - val_acc: 0.9790\n",
            "Epoch 56/100\n",
            "100/100 [==============================] - 33s - loss: 0.0067 - acc: 0.9985 - val_loss: 0.1615 - val_acc: 0.9750\n",
            "Epoch 57/100\n",
            "100/100 [==============================] - 33s - loss: 0.0066 - acc: 0.9975 - val_loss: 0.2520 - val_acc: 0.9630\n",
            "Epoch 58/100\n",
            "100/100 [==============================] - 33s - loss: 0.0017 - acc: 0.9990 - val_loss: 0.1899 - val_acc: 0.9740\n",
            "Epoch 59/100\n",
            "100/100 [==============================] - 33s - loss: 0.0022 - acc: 0.9990 - val_loss: 0.2321 - val_acc: 0.9680\n",
            "Epoch 60/100\n",
            "100/100 [==============================] - 33s - loss: 0.0091 - acc: 0.9975 - val_loss: 0.1416 - val_acc: 0.9790\n",
            "Epoch 61/100\n",
            "100/100 [==============================] - 33s - loss: 0.0054 - acc: 0.9985 - val_loss: 0.1749 - val_acc: 0.9720\n",
            "Epoch 62/100\n",
            "100/100 [==============================] - 33s - loss: 0.0028 - acc: 0.9995 - val_loss: 0.2065 - val_acc: 0.9740\n",
            "Epoch 63/100\n",
            "100/100 [==============================] - 33s - loss: 0.0058 - acc: 0.9985 - val_loss: 0.1749 - val_acc: 0.9750\n",
            "Epoch 64/100\n",
            "100/100 [==============================] - 33s - loss: 0.0076 - acc: 0.9980 - val_loss: 0.1542 - val_acc: 0.9760\n",
            "Epoch 65/100\n",
            "100/100 [==============================] - 33s - loss: 0.0081 - acc: 0.9980 - val_loss: 0.2627 - val_acc: 0.9660\n",
            "Epoch 66/100\n",
            "100/100 [==============================] - 33s - loss: 0.0107 - acc: 0.9980 - val_loss: 0.1748 - val_acc: 0.9740\n",
            "Epoch 67/100\n",
            "100/100 [==============================] - 33s - loss: 5.1450e-04 - acc: 1.0000 - val_loss: 0.1922 - val_acc: 0.9710\n",
            "Epoch 68/100\n",
            "100/100 [==============================] - 33s - loss: 0.0054 - acc: 0.9985 - val_loss: 0.2299 - val_acc: 0.9680\n",
            "Epoch 69/100\n",
            "100/100 [==============================] - 33s - loss: 0.0091 - acc: 0.9965 - val_loss: 0.1533 - val_acc: 0.9730\n",
            "Epoch 70/100\n",
            "100/100 [==============================] - 33s - loss: 0.0190 - acc: 0.9965 - val_loss: 0.2232 - val_acc: 0.9690\n",
            "Epoch 71/100\n",
            "100/100 [==============================] - 33s - loss: 0.0058 - acc: 0.9985 - val_loss: 0.1773 - val_acc: 0.9710\n",
            "Epoch 72/100\n",
            "100/100 [==============================] - 33s - loss: 0.0072 - acc: 0.9990 - val_loss: 0.1660 - val_acc: 0.9760\n",
            "Epoch 73/100\n",
            "100/100 [==============================] - 33s - loss: 0.0036 - acc: 0.9990 - val_loss: 0.2354 - val_acc: 0.9700\n",
            "Epoch 74/100\n",
            "100/100 [==============================] - 33s - loss: 0.0023 - acc: 0.9995 - val_loss: 0.1904 - val_acc: 0.9700\n",
            "Epoch 75/100\n",
            "100/100 [==============================] - 33s - loss: 0.0137 - acc: 0.9955 - val_loss: 0.1363 - val_acc: 0.9810\n",
            "Epoch 76/100\n",
            "100/100 [==============================] - 33s - loss: 0.0055 - acc: 0.9990 - val_loss: 0.2119 - val_acc: 0.9710\n",
            "Epoch 77/100\n",
            "100/100 [==============================] - 33s - loss: 9.2985e-04 - acc: 1.0000 - val_loss: 0.1777 - val_acc: 0.9760\n",
            "Epoch 78/100\n",
            "100/100 [==============================] - 33s - loss: 0.0032 - acc: 0.9990 - val_loss: 0.2568 - val_acc: 0.9710\n",
            "Epoch 79/100\n",
            "100/100 [==============================] - 33s - loss: 0.0112 - acc: 0.9970 - val_loss: 0.2066 - val_acc: 0.9720\n",
            "Epoch 80/100\n",
            "100/100 [==============================] - 33s - loss: 0.0055 - acc: 0.9975 - val_loss: 0.1793 - val_acc: 0.9700\n",
            "Epoch 81/100\n",
            "100/100 [==============================] - 33s - loss: 0.0045 - acc: 0.9980 - val_loss: 0.2082 - val_acc: 0.9690\n",
            "Epoch 82/100\n",
            "100/100 [==============================] - 33s - loss: 0.0095 - acc: 0.9985 - val_loss: 0.1985 - val_acc: 0.9700\n",
            "Epoch 83/100\n",
            "100/100 [==============================] - 33s - loss: 0.0032 - acc: 0.9985 - val_loss: 0.1992 - val_acc: 0.9710\n",
            "Epoch 84/100\n",
            "100/100 [==============================] - 33s - loss: 0.0019 - acc: 0.9995 - val_loss: 0.2285 - val_acc: 0.9690\n",
            "Epoch 85/100\n",
            "100/100 [==============================] - 33s - loss: 0.0066 - acc: 0.9970 - val_loss: 0.1833 - val_acc: 0.9760\n",
            "Epoch 86/100\n",
            "100/100 [==============================] - 33s - loss: 0.0066 - acc: 0.9990 - val_loss: 0.2054 - val_acc: 0.9690\n",
            "Epoch 87/100\n",
            "100/100 [==============================] - 33s - loss: 0.0072 - acc: 0.9980 - val_loss: 0.2084 - val_acc: 0.9730\n",
            "Epoch 88/100\n",
            "100/100 [==============================] - 33s - loss: 0.0043 - acc: 0.9980 - val_loss: 0.1840 - val_acc: 0.9690\n",
            "Epoch 89/100\n",
            "100/100 [==============================] - 33s - loss: 0.0101 - acc: 0.9980 - val_loss: 0.1953 - val_acc: 0.9700\n",
            "Epoch 90/100\n",
            "100/100 [==============================] - 33s - loss: 0.0092 - acc: 0.9990 - val_loss: 0.1996 - val_acc: 0.9740\n",
            "Epoch 91/100\n",
            "100/100 [==============================] - 33s - loss: 0.0081 - acc: 0.9965 - val_loss: 0.2136 - val_acc: 0.9750\n",
            "Epoch 92/100\n",
            "100/100 [==============================] - 33s - loss: 0.0055 - acc: 0.9985 - val_loss: 0.1775 - val_acc: 0.9790\n",
            "Epoch 93/100\n",
            "100/100 [==============================] - 33s - loss: 3.4598e-04 - acc: 1.0000 - val_loss: 0.2102 - val_acc: 0.9760\n",
            "Epoch 94/100\n",
            "100/100 [==============================] - 33s - loss: 0.0109 - acc: 0.9985 - val_loss: 0.2081 - val_acc: 0.9760\n",
            "Epoch 95/100\n",
            "100/100 [==============================] - 33s - loss: 0.0028 - acc: 0.9985 - val_loss: 0.2033 - val_acc: 0.9720\n",
            "Epoch 96/100\n",
            "100/100 [==============================] - 33s - loss: 0.0074 - acc: 0.9990 - val_loss: 0.2118 - val_acc: 0.9750\n",
            "Epoch 97/100\n",
            "100/100 [==============================] - 33s - loss: 0.0034 - acc: 0.9990 - val_loss: 0.2621 - val_acc: 0.9740\n",
            "Epoch 98/100\n",
            "100/100 [==============================] - 33s - loss: 0.0054 - acc: 0.9975 - val_loss: 0.2589 - val_acc: 0.9650\n",
            "Epoch 99/100\n",
            "100/100 [==============================] - 33s - loss: 0.0060 - acc: 0.9990 - val_loss: 0.2242 - val_acc: 0.9700\n",
            "Epoch 100/100\n",
            "100/100 [==============================] - 33s - loss: 0.0010 - acc: 0.9995 - val_loss: 0.2403 - val_acc: 0.9750\n"
          ]
        }
      ],
      "source": [
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer=optimizers.RMSprop(learning_rate=1e-5),\n",
        "              metrics=['acc'])\n",
        "\n",
        "history = model.fit(\n",
        "      train_generator,\n",
        "      epochs=100,\n",
        "      validation_data=validation_generator)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nb2J9Xj41LKh",
        "outputId": "52ed570f-fd2d-42f9-fd9d-434cff473346"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 1000 images belonging to 2 classes.\n",
            "test acc: 0.967999992371\n"
          ]
        }
      ],
      "source": [
        "test_generator = test_datagen.flow_from_directory(\n",
        "        test_dir,\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        class_mode='binary')\n",
        "\n",
        "test_loss, test_acc = model.evaluate_generator(test_generator, steps=50)\n",
        "print('test acc:', test_acc)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5RjUB4v1LKi"
      },
      "source": [
        "\n",
        "Here we get a test accuracy of 97%. In the original Kaggle competition around this dataset, this would have been one of the top results.\n",
        "However, using modern deep learning techniques, we managed to reach this result using only a very small fraction of the training data\n",
        "available (about 10%). There is a huge difference between being able to train on 20,000 samples compared to 2,000 samples!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jldMDzGs1LKj"
      },
      "source": [
        "## Take-aways: using convnets with small datasets\n",
        "\n",
        "Here's what you should take away from the exercises of these past two sections:\n",
        "\n",
        "* Convnets are the best type of machine learning models for computer vision tasks. It is possible to train one from scratch even on a very\n",
        "small dataset, with decent results.\n",
        "* On a small dataset, overfitting will be the main issue. Data augmentation is a powerful way to fight overfitting when working with image\n",
        "data.\n",
        "* It is easy to reuse an existing convnet on a new dataset, via feature extraction. This is a very valuable technique for working with\n",
        "small image datasets.\n",
        "* As a complement to feature extraction, one may use fine-tuning, which adapts to a new problem some of the representations previously\n",
        "learned by an existing model. This pushes performance a bit further.\n",
        "\n",
        "Now you have a solid set of tools for dealing with image classification problems, in particular with small datasets."
      ]
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}