ner_BERT_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bqbS2gLMhNog"
   },
   "source": [
    "# Named entity recognition with Bert PyTorch\n",
    "\n",
    "Named Enity Recognition is one of the most common NLP problems. The goal is classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. What can you use it for? Here are a few ideas - social media, chatbot, customer support tickets, survey responses, and data mining!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MObExv0FhXmd"
   },
   "source": [
    "Let's start by importing some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 803,
     "status": "ok",
     "timestamp": 1595575333846,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "oNUyGjRQCZ3o"
   },
   "outputs": [],
   "source": [
    "files_path = \"./\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2583,
     "status": "ok",
     "timestamp": 1595575336606,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "tG83Y0AhC05p"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "import tensorflow as tf\n",
    "\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DB-FhT-hEefx"
   },
   "source": [
    "# Training Parameters\n",
    "We'll set the hyperparameters for training our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3287,
     "status": "ok",
     "timestamp": 1595575337872,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "uZlKveosDFtS"
   },
   "outputs": [],
   "source": [
    "# Hyperparams if GPU is available\n",
    "if len(tf.config.list_physical_devices('GPU')) > 0:\n",
    "    BATCH_SIZE = 512  # Number of examples used in each iteration\n",
    "    EPOCHS = 5  # Number of passes through entire dataset\n",
    "    MAX_LEN = 75  # Max length of review (in words)\n",
    "    EMBEDDING = 40  # Dimension of word embedding vector\n",
    "\n",
    "    \n",
    "# Hyperparams for CPU training\n",
    "else:\n",
    "    BATCH_SIZE = 32\n",
    "    EPOCHS = 5\n",
    "    MAX_LEN = 75\n",
    "    EMBEDDING = 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8d7tHjXbE6G4"
   },
   "source": [
    "# Data\n",
    "\n",
    "Let's take a look at data. [Annotated Corpus for Named Entity Recognition](https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus?select=ner_dataset.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 427
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5413,
     "status": "ok",
     "timestamp": 1595575343770,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "1S3bXmbXEm5g",
    "outputId": "dde5d60a-f8b1-4abd-d2fc-6e1fd696fe7c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences:  47959\n",
      "Number of words in the dataset:  35178\n",
      "Tags: ['B-per', 'I-art', 'I-gpe', 'B-art', 'O', 'I-per', 'B-geo', 'I-tim', 'I-nat', 'I-geo', 'B-nat', 'B-org', 'B-gpe', 'I-org', 'I-eve', 'B-tim', 'B-eve']\n",
      "Number of Labels:  17\n",
      "What the dataset looks like:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentence #</th>\n",
       "      <th>Word</th>\n",
       "      <th>POS</th>\n",
       "      <th>Tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>Thousands</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>of</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>demonstrators</td>\n",
       "      <td>NNS</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>have</td>\n",
       "      <td>VBP</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>marched</td>\n",
       "      <td>VBN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>through</td>\n",
       "      <td>IN</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>London</td>\n",
       "      <td>NNP</td>\n",
       "      <td>B-geo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>to</td>\n",
       "      <td>TO</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>protest</td>\n",
       "      <td>VB</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Sentence: 1</td>\n",
       "      <td>the</td>\n",
       "      <td>DT</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Sentence #           Word  POS    Tag\n",
       "0  Sentence: 1      Thousands  NNS      O\n",
       "1  Sentence: 1             of   IN      O\n",
       "2  Sentence: 1  demonstrators  NNS      O\n",
       "3  Sentence: 1           have  VBP      O\n",
       "4  Sentence: 1        marched  VBN      O\n",
       "5  Sentence: 1        through   IN      O\n",
       "6  Sentence: 1         London  NNP  B-geo\n",
       "7  Sentence: 1             to   TO      O\n",
       "8  Sentence: 1        protest   VB      O\n",
       "9  Sentence: 1            the   DT      O"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(f\"{files_path}ner_dataset.csv\", encoding=\"latin1\")\n",
    "data = data.fillna(method=\"ffill\")\n",
    "\n",
    "print(\"Number of sentences: \", len(data.groupby(['Sentence #'])))\n",
    "\n",
    "words = list(set(data[\"Word\"].values))\n",
    "n_words = len(words)\n",
    "print(\"Number of words in the dataset: \", n_words)\n",
    "\n",
    "tags = list(set(data[\"Tag\"].values))\n",
    "print(\"Tags:\", tags)\n",
    "n_tags = len(tags)\n",
    "print(\"Number of Labels: \", n_tags)\n",
    "\n",
    "print(\"What the dataset looks like:\")\n",
    "# Show the first 10 rows\n",
    "data.head(n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9307,
     "status": "ok",
     "timestamp": 1595575348759,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "HrIVsOi2Eu7o",
    "outputId": "5409f465-5ada-411b-c39d-9188fd50812e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is what a sentence looks like:\n",
      "[('Thousands', 'NNS', 'O'), ('of', 'IN', 'O'), ('demonstrators', 'NNS', 'O'), ('have', 'VBP', 'O'), ('marched', 'VBN', 'O'), ('through', 'IN', 'O'), ('London', 'NNP', 'B-geo'), ('to', 'TO', 'O'), ('protest', 'VB', 'O'), ('the', 'DT', 'O'), ('war', 'NN', 'O'), ('in', 'IN', 'O'), ('Iraq', 'NNP', 'B-geo'), ('and', 'CC', 'O'), ('demand', 'VB', 'O'), ('the', 'DT', 'O'), ('withdrawal', 'NN', 'O'), ('of', 'IN', 'O'), ('British', 'JJ', 'B-gpe'), ('troops', 'NNS', 'O'), ('from', 'IN', 'O'), ('that', 'DT', 'O'), ('country', 'NN', 'O'), ('.', '.', 'O')]\n"
     ]
    }
   ],
   "source": [
    "class SentenceGetter(object):\n",
    "    \"\"\"Class to Get the sentence in this format:\n",
    "    [(Token_1, Part_of_Speech_1, Tag_1), ..., (Token_n, Part_of_Speech_1, Tag_1)]\"\"\"\n",
    "    def __init__(self, data):\n",
    "        \"\"\"Args:\n",
    "            data is the pandas.DataFrame which contains the above dataset\"\"\"\n",
    "        self.n_sent = 1\n",
    "        self.data = data\n",
    "        self.empty = False\n",
    "        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s[\"Word\"].values.tolist(),\n",
    "                                                           s[\"POS\"].values.tolist(),\n",
    "                                                           s[\"Tag\"].values.tolist())]\n",
    "        self.grouped = self.data.groupby(\"Sentence #\").apply(agg_func)\n",
    "        self.sentences = [s for s in self.grouped]\n",
    "    \n",
    "    def get_next(self):\n",
    "        \"\"\"Return one sentence\"\"\"\n",
    "        try:\n",
    "            s = self.grouped[\"Sentence: {}\".format(self.n_sent)]\n",
    "            self.n_sent += 1\n",
    "            return s\n",
    "        except:\n",
    "            return None\n",
    "        \n",
    "getter = SentenceGetter(data)\n",
    "sent = getter.get_next()\n",
    "print('This is what a sentence looks like:')\n",
    "print(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9050,
     "status": "ok",
     "timestamp": 1595575348759,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "k4VggbtJEvLJ",
    "outputId": "d0899345-4b29-44cd-c976-4699be970624"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcy0lEQVR4nO3debQmVXnv8e/PBhlEaYaWkIbYIEQv3EQgHcTIjQS4jCrGOOA12hK8naxlHJMomgGjcoNDRIhTUFAkiiIaaYeABMFoiECDiAyy7EATmiA0MojiBDz3j9pH3z726Xobz3vG72etd52qXbuqnjrV/T5n167alapCkqQNecR0ByBJmvlMFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplstCsleSAJGumOw5pPjBZaEZI8v2Bz0NJfjgw/8Lpjm+2SfLGJP803XFo7thkugOQAKpqq7HpJKuBl1bVv05fRJMvSYBU1UPTHYu0sWxZaEZLslmSdyX57/Z5V5LNJqj7iiTXJdmprfeOJP+V5PYk70+yRat3QJI1Sf4syR1JbktyzAZiuDjJ3yW5LMn3kpybZNuB5fsluSTJPUm+keSAceuekOTfgfuBXdez/dcluTXJfUluSHJQK39EkuOS/GeS7yY5e2y/SZYkqSTL2jHemeQv27LDgDcAz28ts2+08q2TnNaO99Ykb0myoC17SZKvtt/Z3UluSnL4QIzbJvlQOwd3J/nMwLKnJ7mqHf8lSX5ziFOrWcZkoZnuL4H9gL2AJwH7An81vlKSvwFeAjytqtYAJwK/3tbbDVgM/M3AKr8CbN3KjwXek2SbDcTxYuCPgB2BB4BT2n4XA58H3gJsC/w58KkkiwbWfRGwHHg0cPO4uJ8A/Cnw21X1aOBQYHVb/HLgWcDTgF8F7gbeMy6u/YEnAAcBf5Pkf1TVecD/Az5RVVtV1ZNa3Q+32HcD9gYOAV46sK0nAzcA2wNvA05rrSGAM4EtgT2BxwIntfj3Bk4H/hjYDvhHYMVECV2zWFX58TOjPnRflge36f8EjhhYdiiwuk0fANwKvBP4KrB1Kw/wA+DxA+s9BbhpYL0fApsMLL8D2G+CeC4GThyY3wP4CbAAeB1w5rj65wPLBtZ90waOdbe274OBTcctux44aGB+R+CndJePlwAF7DSw/DLg6Db9RuCfBpbtAPwY2GKg7AXARW36JcCqgWVbtu3/StvvQ8A264n/fcCbx5XdQJe0p/3fkp/J+9hnoZnuV1n3r/GbW9mYhXR/tT+/qu5tZYvovuyu+PkfxoTuy33Md6vqgYH5+4GtmNgt42LYlO4v8McBz03yjIHlmwIXTbDuOqpqVZJX0X2575nkfOA1VfXfbdv/nGSwj+NBui/+Md8Z8hge1+K6beB38ohxsf1sW1V1f6u3FV2L6a6qunuC7S5L8vKBskey7jnSHOBlKM10Y1+aY36tlY25G3g68KEkT21ld9K1HPasqoXts3UNdKI/DDuPi+GnbT+30LUsFg58HlVVJw7U3+DQzlX1saran+44C3hrW3QLcPi4bW9eVbcOEe/4fd5C17LYfmBbj6mqPYfY1i3AtkkWTrDshHExbllVZw2xXc0iJgvNdGcBf5VkUZLt6fod1rkltKouBl4IfDrJvtXdbfQB4KQkj4WubyHJob9EHH+YZI8kWwJvAs6pqgdbLM9IcmiSBUk2bx3oOw2z0SRPSHJgu8b/I7okN9aSeD9wQpLHtbqLkhw1ZLy3A0uSPAKgqm4Dvgj8fZLHtM7zxyd5Wt+G2rr/Arw3yTZJNk3yu23xB4A/SfLkdB6V5Mgkjx4yTs0SJgvNdG8BVgJXA98Ermxl66iqC+g6oD+bZB+6voRVwNeSfA/4V7qO4IfrTLoO4u8AmwOvaPu9BTiK7u6jtXR/af8Fw//f2oyuM/7Otu3HAq9vy04GVgBfTHIf8DW6TuhhfLL9/G6SK9v0i+kuEV1H1yI7h64/YhgvomtNfYuuj+VVAFW1Evi/wLvbNlfR9X9ojkmVLz+SNiTJxXSdxR+c7lik6WLLQpLUy2QhSerlZShJUi9bFpKkXnPyobztt9++lixZMt1hSNKscsUVV9xZVYvWt2ykySLd6KH30T11+kBVLW0DoX2CbriC1cDzquruNgbNycARdE+ivqSqrmzbWcbPxwN6S1WdsaH9LlmyhJUrV07+AUnSHJbk5omWTcVlqN+rqr2qammbPw64sKp2By5s8wCHA7u3z3K6MWdoyeV4uvvL9wWO7xnwTZI0yaajz+IoYKxlcAbdqJpj5R+pzteAhUl2pBs47oKqGhub5gLgsKkOWpLms1Eni6J7+vSKJMtb2Q5t+ADonlgdGxRtMesOaramlU1Uvo4ky5OsTLJy7dq1k3kMkjTvjbqDe/+qurWNz3NBkm8NLqyqSjIp9+5W1anAqQBLly71fmBJmkQjbVmMjY5ZVXcA/0zX53B7u7xE+3lHq34r647suVMrm6hckjRFRpYs2uiTjx6bpnsr1zV0A6Mta9WWAee26RXAi9vIlfsB97bLVecDh7TRLrdp2zl/VHFLkn7RKC9D7UD34pax/Xysqs5LcjlwdpJj6V4i87xW/wt0t82uort19hiAqroryZuBy1u9N1XVXSOMW5I0zpwc7mPp0qXlcxaStHGSXDHwmMM6HO5DktRrTg73ofVbctzn11u++sQjpzgSSbONLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m4oTXiXFHinlKSOLQtJUi+ThSSpl8lCktTLZCFJ6mWykCT18m6oOWhDdzdJ0sNhy0KS1MtkIUnqZbKQJPUyWUiSetnBrQ3yhUmSwJaFJGkIJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqNPFkkWZDk60k+1+Z3SXJpklVJPpHkka18sza/qi1fMrCN17fyG5IcOuqYJUnrmoqWxSuB6wfm3wqcVFW7AXcDx7byY4G7W/lJrR5J9gCOBvYEDgPem2TBFMQtSWpGmiyS7AQcCXywzQc4EDinVTkDeFabPqrN05Yf1OofBXy8qn5cVTcBq4B9Rxm3JGldo25ZvAt4LfBQm98OuKeqHmjza4DFbXoxcAtAW35vq/+z8vWs8zNJlidZmWTl2rVrJ/s4JGleG9k7uJM8Hbijqq5IcsCo9jOmqk4FTgVYunRpjXp/M8FE78eWpMk2smQBPBV4ZpIjgM2BxwAnAwuTbNJaDzsBt7b6twI7A2uSbAJsDXx3oHzM4DqSpCkwsstQVfX6qtqpqpbQdVB/qapeCFwEPKdVWwac26ZXtHna8i9VVbXyo9vdUrsAuwOXjSpuSdIvGmXLYiKvAz6e5C3A14HTWvlpwJlJVgF30SUYquraJGcD1wEPAC+rqgenPmxJmr+mJFlU1cXAxW36RtZzN1NV/Qh47gTrnwCcMLoIJUkb4hPckqReJgtJUq/p6LPQHDDRbburTzxyiiORNBVsWUiSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSLwcSnAV817ak6WbLQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVKv3mSR5PFJNmvTByR5RZKFow9NkjRTDPPyo08BS5PsBpwKnAt8DDhilIFpdproRU2rTzxyiiORNJmGuQz1UFU9APw+8A9V9RfAjqMNS5I0kwyTLH6a5AXAMuBzrWzTvpWSbJ7ksiTfSHJtkr9t5bskuTTJqiSfSPLIVr5Zm1/Vli8Z2NbrW/kNSQ7d2IOUJP1yhkkWxwBPAU6oqpuS7AKcOcR6PwYOrKonAXsBhyXZD3grcFJV7QbcDRzb6h8L3N3KT2r1SLIHcDSwJ3AY8N4kC4Y9QEnSL683WVTVdcDrgCvb/E1V9dYh1quq+n6b3bR9CjgQOKeVnwE8q00f1eZpyw9Kklb+8ar6cVXdBKwC9h3i2CRJk2SYu6GeAVwFnNfm90qyYpiNJ1mQ5CrgDuAC4D+Be1ofCMAaYHGbXgzcAtCW3wtsN1i+nnUG97U8ycokK9euXTtMeJKkIQ1zGeqNdH/J3wNQVVcBuw6z8ap6sKr2AnZq23jiwwtzqH2dWlVLq2rpokWLRrUbSZqXhurgrqp7x5U9tDE7qap7gIvo+j4WJhm7ZXcn4NY2fSuwM0BbvjXw3cHy9awjSZoCwySLa5P8H2BBkt2T/ANwSd9KSRaNPbyXZAvgfwPX0yWN57Rqy+ie2wBY0eZpy79UVdXKj253S+0C7A5cNtTRSZImxTDJ4uV0dyL9GDgL+B7wqiHW2xG4KMnVwOXABVX1ObrO8tckWUXXJ3Faq38asF0rfw1wHEBVXQucDVxH12/ysqp6cLjDkyRNht4nuKvqfuAv22doVXU1sPd6ym9kPXczVdWPgOdOsK0TgBM2Zv+SpMkzYbJI8lm6W13Xq6qeOZKIJEkzzoZaFu+YsigkSTPahMmiqr48Nt2G5HgiXUvjhqr6yRTENu9MNAifJE233j6LJEcC76d7oC7ALkn+uKr+ZdTBSZJmhmGGKP974PeqahV077cAPg+YLCRpnhjm1tn7xhJFcyNw34jikSTNQMO0LFYm+QLdsw5Fd3vr5UmeDVBVnx5hfJKkGWCYZLE5cDvwtDa/FtgCeAZd8jBZSNIcN8xDecdMRSCSpJlrmLuhdqEb8mPJYH0fypOk+WOYy1CfoRu36bNs5GizkqS5YZhk8aOqOmXkkUiSZqxhksXJSY4Hvkg38iwAVXXlyKKSJM0owySL3wBeRPfu7LHLUGPv0pYkzQPDJIvnArs6HpQkzV/DPMF9DbBw1IFIkmauYVoWC4FvJbmcdfssvHVWkuaJYZLF8SOPQpI0ow3zBPeX++pIfSZ6V8fqE4+c4kgkPRy9fRZJ9ktyeZLvJ/lJkgeTfG8qgpMkzQzDdHC/G3gB8G26AQRfCrxnlEFJkmaWYZIF7X0WC6rqwar6EHDYaMOSJM0kw3Rw39/ewX1VkrcBtzFkkpEkzQ3DfOm/qNX7U+AHwM7AH4wyKEnSzDLM3VA3t8kfJTkF2Hnca1YlSXPcMHdDXZzkMUm2Ba4EPpDknaMPTZI0UwxzGWrrqvoe8GzgI1X1ZODg0YYlSZpJhkkWmyTZEXge8LkRxyNJmoGGSRZvAs4HVlXV5Ul2pXvmQpI0TwzTwf1J4JMD8zfi3VCSNK8M85yFJtlE4yRJ0kzlw3WSpF4mC0lSr2Ges/irgenNht1wkp2TXJTkuiTXJnllK982yQVJvt1+btPKk+SUJKuSXJ1kn4FtLWv1v51k2cYdoiTplzVhskjyuiRPAZ4zUPwfG7HtB4A/q6o9gP2AlyXZAzgOuLCqdgcubPMAhwO7t89y4H0tjm3pXsD0ZGBf4PixBCNJmhoball8C3gusGuSryT5ALBdkicMs+Gquq2qrmzT9wHXA4uBo4AzWrUzgGe16aPoHvqrqvoasLA933EocEFV3VVVdwMX4Ki3kjSlNpQs7gHeAKwCDgBObuXHJblkY3aSZAmwN3ApsENV3dYWfQfYoU0vBm4ZWG1NK5uofPw+lidZmWTl2rVrNyY8SVKPDSWLQ4HPA48H3kl3GegHVXVMVf3OsDtIshXwKeBVbdiQn6mqAmqjo16Pqjq1qpZW1dJFixZNxiYlSc2EyaKq3lBVBwGrgTOBBcCiJF9N8tlhNp5kU7pE8dGq+nQrvr1dXqL9vKOV30o3/PmYnVrZROWSpCkyzK2z51fVyqo6FVhTVfsDx/StlCTAacD1VTU4Su0KYOyOpmXAuQPlL253Re0H3NsuV50PHJJkm9axfUgrkyRNkWGG+3jtwOxLWtmdQ2z7qXQvTvpmkqta2RuAE4GzkxwL3Ew3QCHAF4Aj6PpI7qclpKq6K8mbgctbvTdV1V1D7F+SNEk2ariPqvrGRtT9KpAJFh+0nvoFvGyCbZ0OnD7sviVJk8snuCVJvUwWkqReJgtJUi+ThSSpl++z0Iw00Ts/Vp945BRHIglsWUiShmCykCT18jKUppWvmJVmB1sWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1MllIknqZLCRJvUwWkqReJgtJUi+ThSSpl8lCktTLZCFJ6mWykCT1cojyEXL4bUlzhS0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb1MFpKkXiYLSVIvk4UkqZfJQpLUa2TJIsnpSe5Ics1A2bZJLkjy7fZzm1aeJKckWZXk6iT7DKyzrNX/dpJlo4pXkjSxUbYsPgwcNq7sOODCqtoduLDNAxwO7N4+y4H3QZdcgOOBJwP7AsePJRhJ0tQZWbKoqn8D7hpXfBRwRps+A3jWQPlHqvM1YGGSHYFDgQuq6q6quhu4gF9MQJKkEZvqPosdquq2Nv0dYIc2vRi4ZaDemlY2UfkvSLI8ycokK9euXTu5UUvSPDdtHdxVVUBN4vZOraqlVbV00aJFk7VZSRJTnyxub5eXaD/vaOW3AjsP1NuplU1ULkmaQlOdLFYAY3c0LQPOHSh/cbsraj/g3na56nzgkCTbtI7tQ1qZJGkKjey1qknOAg4Atk+yhu6uphOBs5McC9wMPK9V/wJwBLAKuB84BqCq7kryZuDyVu9NVTW+01ySNGIjSxZV9YIJFh20nroFvGyC7ZwOnD6JoUmSNpJPcEuSepksJEm9TBaSpF4mC0lSL5OFJKmXyUKS1MtkIUnqNbLnLOaTJcd9frpDkKSRsmUhSeply0KzykStuNUnHjnFkUjziy0LSVIvk4UkqZfJQpLUy2QhSeplspAk9TJZSJJ6mSwkSb18zkJzgs9fSKNly0KS1MtkIUnqZbKQJPUyWUiSepksJEm9TBaSpF4mC0lSL5+z0Jzm8xfS5LBlIUnqZbKQJPUyWUiSetlnsREmuv6t2ce+DGnj2LKQJPUyWUiSenkZShrg5Slp/WxZSJJ6zZqWRZLDgJOBBcAHq+rEaQ5J88jDubnB1ojmklnRskiyAHgPcDiwB/CCJHtMb1SSNH/MlpbFvsCqqroRIMnHgaOA60axM2+R1WSYrH9HE7VQ7F/RVJotyWIxcMvA/BrgyYMVkiwHlrfZ7ye5YSP3sT1w58OOcHbxWGeRvHXoqtsDd25E/dls1p/XjTCVx/q4iRbMlmTRq6pOBU59uOsnWVlVSycxpBnLY52bPNa5aaYc66zoswBuBXYemN+plUmSpsBsSRaXA7sn2SXJI4GjgRXTHJMkzRuz4jJUVT2Q5E+B8+lunT29qq6d5N087EtYs5DHOjd5rHPTjDjWVNV0xyBJmuFmy2UoSdI0MllIknrN+2SR5LAkNyRZleS46Y5nMiXZOclFSa5Lcm2SV7bybZNckOTb7ec20x3rZEmyIMnXk3yuze+S5NJ2fj/RbpCY9ZIsTHJOkm8luT7JU+bqeU3y6vbv95okZyXZfC6d1ySnJ7kjyTUDZes9l+mc0o776iT7TFWc8zpZzINhRB4A/qyq9gD2A17Wju844MKq2h24sM3PFa8Erh+YfytwUlXtBtwNHDstUU2+k4HzquqJwJPojnnOndcki4FXAEur6n/S3eByNHPrvH4YOGxc2UTn8nBg9/ZZDrxvimKc38mCgWFEquonwNgwInNCVd1WVVe26fvovlAW0x3jGa3aGcCzpifCyZVkJ+BI4INtPsCBwDmtypw41iRbA78LnAZQVT+pqnuYo+eV7q7NLZJsAmwJ3MYcOq9V9W/AXeOKJzqXRwEfqc7XgIVJdpyKOOd7sljfMCKLpymWkUqyBNgbuBTYoapua4u+A+wwTWFNtncBrwUeavPbAfdU1QNtfq6c312AtcCH2iW3DyZ5FHPwvFbVrcA7gP+iSxL3AlcwN8/roInO5bR9Z833ZDEvJNkK+BTwqqr63uCy6u6dnvX3Tyd5OnBHVV0x3bFMgU2AfYD3VdXewA8Yd8lpDp3Xbej+mt4F+FXgUfziJZs5baacy/meLOb8MCJJNqVLFB+tqk+34tvHmq7t5x3TFd8keirwzCSr6S4nHkh3XX9hu3wBc+f8rgHWVNWlbf4cuuQxF8/rwcBNVbW2qn4KfJruXM/F8zpoonM5bd9Z8z1ZzOlhRNo1+9OA66vqnQOLVgDL2vQy4Nypjm2yVdXrq2qnqlpCdx6/VFUvBC4CntOqzZVj/Q5wS5IntKKD6Ibrn3Pnle7y035Jtmz/nseOdc6d13EmOpcrgBe3u6L2A+4duFw1UvP+Ce4kR9Bd6x4bRuSEaQ5p0iTZH/gK8E1+fh3/DXT9FmcDvwbcDDyvqsZ3sM1aSQ4A/ryqnp5kV7qWxrbA14E/rKofT2d8kyHJXnQd+Y8EbgSOofvjb86d1yR/Czyf7u6+rwMvpbtOPyfOa5KzgAPohiK/HTge+AzrOZctYb6b7lLc/cAxVbVySuKc78lCktRvvl+GkiQNwWQhSeplspAk9TJZSJJ6mSwkSb1MFpqRknx/BNvcIsmX2wCSI5NkdZLtR7mPtp+3t9FY3z6u/IAkvzPE+h9O8py+ekNs5x1JDvxlt6OZbVa8VlWaJH8EfLqqHpzuQCaSZJOBMY/6LAe2Xc/xHAB8H7hkMmPbgH8APgB8aYr2p2lgy0KzRpLHJzkvyRVJvpLkia38w22M/0uS3LiBv5ZfSHsStv31ffHAOyE+2h54WqdlkGRpkovb9BuTnNH2fXOSZyd5W5Jvtrg2HdjXa1v5ZUl2a+svSvKpJJe3z1MHtntmkn8Hzhx3zGktiGva9p7fylcAWwFXjJW18iXAnwCvTnJVkv+VZEmSL7X3H1yY5NfW87t9c/s9LkjyFy2+q9sDcbRtXJ/kA60188UkWwBU1c3Adkl+ZdhzqdnHZKHZ5FTg5VX1W8CfA+8dWLYjsD/wdODE8Su24Vx2rarVA8V7A6+ie5fJrnRjDvV5PN24U88E/gm4qKp+A/gh3fDoY+5t5e+mGyEAurGqTqqq3wb+gDaUerMHcHBVvWDc/p4N7EX3zoqDgbcn2bGqngn8sKr2qqpPjFVux/f+tp+9quordH/5n1FVvwl8FDhl3O/m7cAiuqfAD6J7V8K+bb+/leR3W9XdgfdU1Z7APe0YxlzJcL8/zVJehtKskG7k3N8BPtkaAACbDVT5TFU9BFyXZH1Dc29P9wU36LKqWtO2fxWwBPhqTyj/UlU/TfJNuiFizmvl32zrjzlr4OdJbfpgYI+B+B/TjgtgRVX9cD372x84q11quj3Jl4HfZuPGMHsKXdKBruXytoFlfw1cWlXLAZIcAhxCN4QGdK2X3enGaLqpqq5q5Vew7vHeQTcqrOYok4Vmi0fQvcNgrwmWD44LlPUs/yGw+QbWeZCf/394gJ+3ute7TlU9lOSn9fPxch5i3f9PtZ7pRwD7VdWPBjfYkscP1hPzVLicrvWwbRtHKsDfVdU/DlZql7fG/762GJjfnO53rDnKy1CaFdp7OG5K8lz42bX8J23E+ncDC5KM//Jfn9XAb7XpP9hAvQ15/sDP/2jTXwRePlahDQbY5yvA81tfwiK6N+Rd1rPOfcCjB+YvoRuJF7p+m68MLDuP7rLd55M8Gjgf+KOxFk+SxUkeO0Scvw5c01tLs5bJQjPVlknWDHxeQ/dFd2ySbwDXsvGvwP0i3WWdPn8LnJxkJd1f0A/HNkmupnsn+Ktb2SuApa3j+Dq6jug+/wxcDXyD7m6j17Yhyjfks8Dvj3Vw0yWoY1o8L2ox/UxVfZLubqYVdInkY8B/tEtt57Bu4vkFrWN/N2BKRj/V9HDUWc0bSfYBXl1VL5ruWOaSJL8P7FNVfz3dsWh0bFlo3qiqK4GLMuKH8uahTYC/n+4gNFq2LCRJvWxZSJJ6mSwkSb1MFpKkXiYLSVIvk4Ukqdf/B9dWWtejw9LFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get all the sentences\n",
    "sentences = getter.sentences\n",
    "\n",
    "# Plot sentence by lenght\n",
    "plt.hist([len(s) for s in sentences], bins=50)\n",
    "plt.title('Token per sentence')\n",
    "plt.xlabel('Len (number of token)')\n",
    "plt.ylabel('# samples')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PprAvnVuZTkE"
   },
   "source": [
    "# Data Preprocessing\n",
    "Before feeding the data into the model, we have to preprocess the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 420
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6527,
     "status": "ok",
     "timestamp": 1595575349172,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "c_EVfMnwXpK4",
    "outputId": "466d3f3e-ebbf-4a84-9c6c-cd5f117f63fd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Thousands',\n",
       " 'of',\n",
       " 'demonstrators',\n",
       " 'have',\n",
       " 'marched',\n",
       " 'through',\n",
       " 'London',\n",
       " 'to',\n",
       " 'protest',\n",
       " 'the',\n",
       " 'war',\n",
       " 'in',\n",
       " 'Iraq',\n",
       " 'and',\n",
       " 'demand',\n",
       " 'the',\n",
       " 'withdrawal',\n",
       " 'of',\n",
       " 'British',\n",
       " 'troops',\n",
       " 'from',\n",
       " 'that',\n",
       " 'country',\n",
       " '.']"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = [[word[0] for word in sentence] for sentence in getter.sentences]\n",
    "sentences[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UIF-RGh1s8sV"
   },
   "source": [
    "The sentences are annotated with the BIO-schema and the labels look like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 818,
     "status": "ok",
     "timestamp": 1595575364053,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "2urGT7sos9NE",
    "outputId": "f8bac363-d1b3-4847-c08b-6d56da92959f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['O', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-gpe', 'O', 'O', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "labels = [[s[2] for s in sentence] for sentence in getter.sentences]\n",
    "print(labels[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 782,
     "status": "ok",
     "timestamp": 1595575372167,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "XDzayq2wtAJk"
   },
   "outputs": [],
   "source": [
    "tag_values = list(set(data[\"Tag\"].values))\n",
    "tag_values.append(\"PAD\")\n",
    "tag2idx = {t: i for i, t in enumerate(tag_values)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jstjozhDtCJL"
   },
   "source": [
    "### Prepare the sentences and labels\n",
    "\n",
    "Before we can start fine-tuning the model, we have to prepare the data set for the use with pytorch and BERT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jFyzlVmotI9r"
   },
   "outputs": [],
   "source": [
    "! pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1177,
     "status": "ok",
     "timestamp": 1595575441712,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "eodz4NE6tFl9",
    "outputId": "4a9a64db-290b-41c5-f7f3-ea0923180309"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'1.5.1+cu101'"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "from transformers import BertTokenizer, BertConfig\n",
    "\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "torch.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "85HJRKkCtK4d"
   },
   "source": [
    "Here we fix some configurations. We will limit our sequence length to 75 tokens and we will use a batch size of 32 as suggested by the Bert paper. Note, that Bert supports sequences of up to 512 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 759,
     "status": "ok",
     "timestamp": 1595575453905,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "NSKPc4RvtGKd"
   },
   "outputs": [],
   "source": [
    "MAX_LEN = 75\n",
    "bs = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 787,
     "status": "ok",
     "timestamp": 1595575458763,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "8EUMSXJftUGk"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "n_gpu = torch.cuda.device_count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 758,
     "status": "ok",
     "timestamp": 1595575465618,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "G_e5F4mttW9z",
    "outputId": "c2ff0a47-347d-493c-f25c-8fdf4dbe6a62"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'Tesla P100-PCIE-16GB'"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cuda.get_device_name(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8BdCtarTtZ6c"
   },
   "source": [
    "The Bert implementation comes with a pretrained tokenizer and a definied vocabulary. We load the one related to the smallest pre-trained model bert-base-cased. We use the cased variate since it is well suited for NER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "a4258a337320400db9729f301d3b115b",
      "c38d9e21ae1944429ab950691bb64505",
      "42aeeabfba234ed9a68e0dd36cff9e84",
      "f396604c499740659071205caa3b7afc",
      "5bae04f105594ca7b1cd006f90e07213",
      "11a9aef6dece41d48af2ed66b8f4b806",
      "06dd2f195f5846a1a9c5e83828255f29",
      "b906cf5884f04e01af30129bf05fa76d"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1594,
     "status": "ok",
     "timestamp": 1595575486291,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "VEkG3q5rtXuE",
    "outputId": "ebc17b33-9632-4e20-f4c0-f876528e1242"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4258a337320400db9729f301d3b115b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=213450.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W-eNEbXytdl7"
   },
   "source": [
    "Now we tokenize all sentences. Since the BERT tokenizer is based a Wordpiece tokenizer it will split tokens in subword tokens. For example ‘gunships’ will be split in the two tokens ‘guns’ and ‘##hips’. We have to deal with the issue of splitting our token-level labels to related subtokens. In practice you would solve this by a specialized data structure based on label spans, but for simplicity I do it explicitly here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 801,
     "status": "ok",
     "timestamp": 1595575500886,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "1VjwuRH3tcPM"
   },
   "outputs": [],
   "source": [
    "def tokenize_and_preserve_labels(sentence, text_labels):\n",
    "    tokenized_sentence = []\n",
    "    labels = []\n",
    "\n",
    "    for word, label in zip(sentence, text_labels):\n",
    "\n",
    "        # Tokenize the word and count # of subwords the word is broken into\n",
    "        tokenized_word = tokenizer.tokenize(word)\n",
    "        n_subwords = len(tokenized_word)\n",
    "\n",
    "        # Add the tokenized word to the final tokenized word list\n",
    "        tokenized_sentence.extend(tokenized_word)\n",
    "\n",
    "        # Add the same label to the new list of labels `n_subwords` times\n",
    "        labels.extend([label] * n_subwords)\n",
    "\n",
    "    return tokenized_sentence, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 37860,
     "status": "ok",
     "timestamp": 1595575545631,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "8rBSnSw0thcr"
   },
   "outputs": [],
   "source": [
    "tokenized_texts_and_labels = [\n",
    "    tokenize_and_preserve_labels(sent, labs)\n",
    "    for sent, labs in zip(sentences, labels)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 31713,
     "status": "ok",
     "timestamp": 1595575545633,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "J_yDLkLgti8D"
   },
   "outputs": [],
   "source": [
    "tokenized_texts = [token_label_pair[0] for token_label_pair in tokenized_texts_and_labels]\n",
    "labels = [token_label_pair[1] for token_label_pair in tokenized_texts_and_labels]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J1N0slBptlPD"
   },
   "source": [
    "Next, we cut and pad the token and label sequences to our desired length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2755,
     "status": "ok",
     "timestamp": 1595575548390,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "TQzu2ahKtkpz"
   },
   "outputs": [],
   "source": [
    "input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n",
    "                          maxlen=MAX_LEN, dtype=\"long\", value=0.0,\n",
    "                          truncating=\"post\", padding=\"post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3510,
     "status": "ok",
     "timestamp": 1595575549151,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "doZ7CWaato7U"
   },
   "outputs": [],
   "source": [
    "tags = pad_sequences([[tag2idx.get(l) for l in lab] for lab in labels],\n",
    "                     maxlen=MAX_LEN, value=tag2idx[\"PAD\"], padding=\"post\",\n",
    "                     dtype=\"long\", truncating=\"post\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LN-HWu7NtqeN"
   },
   "source": [
    "The Bert model supports something called attention_mask, which is similar to the masking in keras. So here we create the mask to ignore the padded elements in the sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7920,
     "status": "ok",
     "timestamp": 1595575563203,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "3Mt1CGmpts-v"
   },
   "outputs": [],
   "source": [
    "attention_masks = [[float(i != 0.0) for i in ii] for ii in input_ids]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wQwraVdstuid"
   },
   "source": [
    "Now we split the dataset to use 10% to validate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 803,
     "status": "ok",
     "timestamp": 1595575568258,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "q_vF6QGBtvD_"
   },
   "outputs": [],
   "source": [
    "tr_inputs, val_inputs, tr_tags, val_tags = train_test_split(input_ids, tags,\n",
    "                                                            random_state=2018, test_size=0.1)\n",
    "tr_masks, val_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
    "                                             random_state=2018, test_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q26m-qHutyCj"
   },
   "source": [
    "Since we’re operating in pytorch, we have to convert the dataset to torch tensors.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 799,
     "status": "ok",
     "timestamp": 1595575582374,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "nhpzyC9otwp7"
   },
   "outputs": [],
   "source": [
    "tr_inputs = torch.tensor(tr_inputs)\n",
    "val_inputs = torch.tensor(val_inputs)\n",
    "tr_tags = torch.tensor(tr_tags)\n",
    "val_tags = torch.tensor(val_tags)\n",
    "tr_masks = torch.tensor(tr_masks)\n",
    "val_masks = torch.tensor(val_masks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sb_XrVN9t1nA"
   },
   "source": [
    "The last step is to define the dataloaders. We shuffle the data at training time with the RandomSampler and at test time we just pass them sequentially with the SequentialSampler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 893,
     "status": "ok",
     "timestamp": 1595575596800,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "Vf8m5v3_t0Fz"
   },
   "outputs": [],
   "source": [
    "train_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=bs)\n",
    "\n",
    "valid_data = TensorDataset(val_inputs, val_masks, val_tags)\n",
    "valid_sampler = SequentialSampler(valid_data)\n",
    "valid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CY_pRzZLt5Jb"
   },
   "source": [
    "The transformer package provides a BertForTokenClassification class for token-level predictions. BertForTokenClassification is a fine-tuning model that wraps BertModel and adds token-level classifier on top of the BertModel. The token-level classifier is a linear layer that takes as input the last hidden state of the sequence. We load the pre-trained bert-base-cased model and provide the number of possible labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 816,
     "status": "ok",
     "timestamp": 1595575612209,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "eHYjV1S8t5u2",
    "outputId": "cce9f080-85cd-48e6-83ea-ec2f3525deea"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'3.0.2'"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import transformers\n",
    "from transformers import BertForTokenClassification, AdamW\n",
    "\n",
    "transformers.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 218,
     "referenced_widgets": [
      "a22593cc69dd47de80298f953624efab",
      "30b688358e1c42fcb761a42ecd18fcc1",
      "279e766e8b8642779aaa03a884f80c87",
      "1b76b947f86042978a787cca37fcef7a",
      "231f4852683c417aa529bb7955ea3e45",
      "dfe1476d0f76491c81630b304009e5ba",
      "96efaef3e123490d89cc2e1b74db7e51",
      "666e97b5542a493bb3deb026cc690910",
      "3a581d6821e6496d88a1a1790100503a",
      "af31b747552747ef8eff1b8db557a55c",
      "ed2efdde5f4d4bb095401934940afb2d",
      "c7d109df95ee4c74868b800c5a5024a2",
      "4900e38bbc684cab96b63dd34df16d85",
      "8959a967f46948b1beef30456b8eeab9",
      "260be31f9c64401ea38426f6e0fe173a",
      "d027e472beca4cc09e24871affa463e9"
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12713,
     "status": "ok",
     "timestamp": 1595575629403,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "PKoDLsvPt5rz",
    "outputId": "a209ef32-b3b8-42e1-e5ab-52b9b88e1672"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a22593cc69dd47de80298f953624efab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a581d6821e6496d88a1a1790100503a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=435779157.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
      "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = BertForTokenClassification.from_pretrained(\n",
    "    \"bert-base-cased\",\n",
    "    num_labels=len(tag2idx),\n",
    "    output_attentions = False,\n",
    "    output_hidden_states = False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v1qEYGtst-lT"
   },
   "source": [
    "Now we have to pass the model parameters to the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10581,
     "status": "ok",
     "timestamp": 1595575642814,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "A0K5EMEzt9LD"
   },
   "outputs": [],
   "source": [
    "model.cuda();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4KpvSgcguC2j"
   },
   "source": [
    "Before we can start the fine-tuning process, we have to setup the optimizer and add the parameters it should update. A common choice is the AdamW optimizer. We also add some weight_decay as regularization to the main weight matrices. If you have limited resources, you can also try to just train the linear classifier on top of BERT and keep all other weights fixed. This will still give you a good performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 808,
     "status": "ok",
     "timestamp": 1595575655913,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "TTUV2Fs-uD0m"
   },
   "outputs": [],
   "source": [
    "FULL_FINETUNING = True\n",
    "if FULL_FINETUNING:\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'gamma', 'beta']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "         'weight_decay_rate': 0.01},\n",
    "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "         'weight_decay_rate': 0.0}\n",
    "    ]\n",
    "else:\n",
    "    param_optimizer = list(model.classifier.named_parameters())\n",
    "    optimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\n",
    "\n",
    "optimizer = AdamW(\n",
    "    optimizer_grouped_parameters,\n",
    "    lr=3e-5,\n",
    "    eps=1e-8\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AAYtrP-ouHGr"
   },
   "source": [
    "We also add a scheduler to linearly reduce the learning rate throughout the epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 788,
     "status": "ok",
     "timestamp": 1595575669453,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "q0vyM8J6uHmD"
   },
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup\n",
    "\n",
    "epochs = 3\n",
    "max_grad_norm = 1.0\n",
    "\n",
    "# Total number of training steps is number of batches * number of epochs.\n",
    "total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "# Create the learning rate scheduler.\n",
    "scheduler = get_linear_schedule_with_warmup(\n",
    "    optimizer,\n",
    "    num_warmup_steps=0,\n",
    "    num_training_steps=total_steps\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CDCMZhz8uLZD"
   },
   "source": [
    "## Fit BERT for named entity recognition\n",
    "First we define some metrics, we want to track while training. We use the f1_score from the seqeval package. You ca find more details here. And we use simple accuracy on a token level comparable to the accuracy in keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 302
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6545,
     "status": "ok",
     "timestamp": 1595575708013,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "vXyOyNhAuQLU",
    "outputId": "09ca92bd-e1fa-4056-ee2c-f45211d52ee7"
   },
   "outputs": [],
   "source": [
    "! pip install seqeval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 850,
     "status": "ok",
     "timestamp": 1595575762154,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "AXEDeIK0uJe7"
   },
   "outputs": [],
   "source": [
    "from seqeval.metrics import f1_score, accuracy_score\n",
    "from tqdm import tqdm, trange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3fz0xnLluVS1"
   },
   "source": [
    "Finally, we can finetune the model. A few epochs should be enougth. The paper suggest 3-4 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 286
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 737192,
     "status": "ok",
     "timestamp": 1595576879552,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "7qTpjf2iuO_L",
    "outputId": "ee0a9873-247c-439b-a448-396ce128e8c8"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch:   0%|          | 0/3 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average train loss: 0.1933022626186186\n",
      "Validation loss: 0.1399431469043096\n",
      "Validation Accuracy: 0.9564909507106039\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch:  33%|███▎      | 1/3 [06:12<12:25, 372.83s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation F1-Score: 0.8059171597633136\n",
      "\n",
      "Average train loss: 0.11022734615242066\n",
      "Validation loss: 0.1297031193971634\n",
      "Validation Accuracy: 0.9597598932859048\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch:  67%|██████▋   | 2/3 [12:24<06:12, 372.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation F1-Score: 0.8224928582168095\n",
      "\n",
      "Average train loss: 0.08266287265805247\n",
      "Validation loss: 0.13044918569425742\n",
      "Validation Accuracy: 0.961571397987583\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 100%|██████████| 3/3 [18:35<00:00, 371.89s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation F1-Score: 0.8274398445673292\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "## Store the average loss after each epoch so we can plot them.\n",
    "loss_values, validation_loss_values = [], []\n",
    "\n",
    "for _ in trange(epochs, desc=\"Epoch\"):\n",
    "    # ========================================\n",
    "    #               Training\n",
    "    # ========================================\n",
    "    # Perform one full pass over the training set.\n",
    "\n",
    "    # Put the model into training mode.\n",
    "    model.train()\n",
    "    # Reset the total loss for this epoch.\n",
    "    total_loss = 0\n",
    "\n",
    "    # Training loop\n",
    "    for step, batch in enumerate(train_dataloader):\n",
    "        # add batch to gpu\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "        # Always clear any previously calculated gradients before performing a backward pass.\n",
    "        model.zero_grad()\n",
    "        # forward pass\n",
    "        # This will return the loss (rather than the model output)\n",
    "        # because we have provided the `labels`.\n",
    "        outputs = model(b_input_ids, token_type_ids=None,\n",
    "                        attention_mask=b_input_mask, labels=b_labels)\n",
    "        # get the loss\n",
    "        loss = outputs[0]\n",
    "        # Perform a backward pass to calculate the gradients.\n",
    "        loss.backward()\n",
    "        # track train loss\n",
    "        total_loss += loss.item()\n",
    "        # Clip the norm of the gradient\n",
    "        # This is to help prevent the \"exploding gradients\" problem.\n",
    "        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)\n",
    "        # update parameters\n",
    "        optimizer.step()\n",
    "        # Update the learning rate.\n",
    "        scheduler.step()\n",
    "\n",
    "    # Calculate the average loss over the training data.\n",
    "    avg_train_loss = total_loss / len(train_dataloader)\n",
    "    print(\"Average train loss: {}\".format(avg_train_loss))\n",
    "\n",
    "    # Store the loss value for plotting the learning curve.\n",
    "    loss_values.append(avg_train_loss)\n",
    "\n",
    "\n",
    "    # ========================================\n",
    "    #               Validation\n",
    "    # ========================================\n",
    "    # After the completion of each training epoch, measure our performance on\n",
    "    # our validation set.\n",
    "\n",
    "    # Put the model into evaluation mode\n",
    "    model.eval()\n",
    "    # Reset the validation loss for this epoch.\n",
    "    eval_loss, eval_accuracy = 0, 0\n",
    "    nb_eval_steps, nb_eval_examples = 0, 0\n",
    "    predictions , true_labels = [], []\n",
    "    for batch in valid_dataloader:\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        b_input_ids, b_input_mask, b_labels = batch\n",
    "\n",
    "        # Telling the model not to compute or store gradients,\n",
    "        # saving memory and speeding up validation\n",
    "        with torch.no_grad():\n",
    "            # Forward pass, calculate logit predictions.\n",
    "            # This will return the logits rather than the loss because we have not provided labels.\n",
    "            outputs = model(b_input_ids, token_type_ids=None,\n",
    "                            attention_mask=b_input_mask, labels=b_labels)\n",
    "        # Move logits and labels to CPU\n",
    "        logits = outputs[1].detach().cpu().numpy()\n",
    "        label_ids = b_labels.to('cpu').numpy()\n",
    "\n",
    "        # Calculate the accuracy for this batch of test sentences.\n",
    "        eval_loss += outputs[0].mean().item()\n",
    "        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n",
    "        true_labels.extend(label_ids)\n",
    "\n",
    "    eval_loss = eval_loss / len(valid_dataloader)\n",
    "    validation_loss_values.append(eval_loss)\n",
    "    print(\"Validation loss: {}\".format(eval_loss))\n",
    "    pred_tags = [tag_values[p_i] for p, l in zip(predictions, true_labels)\n",
    "                                 for p_i, l_i in zip(p, l) if tag_values[l_i] != \"PAD\"]\n",
    "    valid_tags = [tag_values[l_i] for l in true_labels\n",
    "                                  for l_i in l if tag_values[l_i] != \"PAD\"]\n",
    "    print(\"Validation Accuracy: {}\".format(accuracy_score(pred_tags, valid_tags)))\n",
    "    print(\"Validation F1-Score: {}\".format(f1_score(pred_tags, valid_tags)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5NAaeGw5xVev"
   },
   "source": [
    "# Visualize the training loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 481
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1200,
     "status": "ok",
     "timestamp": 1595576985409,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "ys2sp-koxVQ4",
    "outputId": "c3b482b8-ac8b-48e0-dada-75cd140e3e2e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvAAAAGaCAYAAABpIXfbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUVf7H8fdMeiGFEBIglEAg1NCREkQRkY6hiIhUxYi7WMAV/IkQVHQBFZVVF5FQQhFpBpQivYgYeu8dAiQQSAFSJ78/skRCAmRCYJLweT3PPsuce++53zkJ8pk7555rSE9PT0dERERERAoFo6ULEBERERGR3FOAFxEREREpRBTgRUREREQKEQV4EREREZFCRAFeRERERKQQUYAXERERESlEFOBFROS+WrZsSe/evS1dhoiIoAAvIvLQ/PXXX/j7+zNlyhRLlyIiIkWItaULEBGRgm/58uWWLkFERP5HV+BFRB4jKSkpJCUlmX2cra0ttra2D6GigikhIcHSJYiI3JUCvIhIAXDq1Cn+9a9/ERgYSM2aNWnZsiVjx47lxo0bWfY7fvw4ISEhtG/fnrp161K7dm26dOnCvHnzsvU5ceJE/P39OXr0KJ999hlPPvkkAQEB7Nq1i4ULF+Lv78+ff/7JlClTaNWqFTVr1uS5555j0aJF2frKaQ78rbbjx4/z2muvUbduXerXr8+bb75JdHR0tj4OHTrEgAEDqFOnDk888QTDhg0jJiYGf39/hg8fnqtxSk5OZvLkyXTu3JnatWtTv359unTpwsyZMzP3GT58OP7+/jkef+e5zp07h7+/PxMnTmTp0qV06dKFgIAAPvnkE8aPH4+/vz+HDh3K1k98fDwBAQG88cYbWdo3b97MgAEDaNCgAbVq1aJjx47MmTMnV+9NRCS3NIVGRMTC9u3bR9++fXFxcaFHjx54eXlx6NAhwsLC2LlzJ2FhYdjY2AAQERHBtm3beOqpp/Dx8eHmzZssX76cESNGEBMTQ3BwcLb+3333Xezt7RkwYAAAnp6enD9/HoAJEyaQmJhIjx49sLW1Zc6cOQwfPpxy5cpRv379+9Z+6dIl+vTpQ6tWrXjvvfc4dOgQc+fOJSEhgdDQ0Mz9Tp06Ra9evTCZTPTu3RsvLy/Wr1/Pq6++mutxSk5O5pVXXiEiIoLAwEA6deqEnZ0dR44c4ffff+fll1/OdV93WrVqFWFhYfTs2ZMXX3wRZ2dnqlSpwo8//kh4eDhVq1bNsv+yZctISkoiKCgos23u3LmMGjWKOnXq8Prrr+Pg4MDmzZsJCQnhzJkzDBs2LM/1iYjcTgFeRMTC/u///g9PT0/mz5+Ps7NzZnuTJk345z//yZIlS+jSpQsAnTt3pmfPnlmO79evH3379uWHH35gwIABmWH/FhcXF6ZOnYq19d//yd+1axeQEYrnz5+fOT2mTZs2PPPMM8yaNStXAf706dNMmDCBdu3aZbYZjUZmz57NiRMnqFixIpDxQSEhIYHZs2dn9vvyyy/z9ttvs3///lyN0/Tp04mIiCA4OJghQ4Zk2WYymXLVx90cO3aMxYsXU6lSpSztNWvWZMmSJbz77rtYWVlltv/yyy+4ubnRokULAKKiovjkk09o3749X3zxReZ+vXr14pNPPmHatGm89NJLlC1b9oHqFBEBTaEREbGow4cPc/jwYTp06EBycjIxMTGZ/6tfvz6Ojo788ccfmfs7Ojpm/jkpKYmrV69y7do1mjVrRkJCAidOnMh2jr59+2YJ77d76aWXssxt9/LywtfXl1OnTuWq/pIlS2YJ7wCNGzcGMsI9QFpaGhs2bCAgICDbh4Jb3wrkxpIlS3B1deUf//hHtm1G44P9c9aiRYts4R0gKCiI6OjoLD+Ds2fPsmPHDjp06JA5ditWrCA5OZlu3bpl+RnGxMTQsmVLTCYTmzdvfqAaRURu0RV4ERELOn78OJAxX33ixIk57nP58uXMP1+/fp3//Oc/LFu2jAsXLmTbNy4uLltbhQoV7nr+nK4Iu7m5ZU6xuZ+7HQ9w7do1AGJiYrhx4wa+vr7Z9s2p7W5Onz5NtWrVsLOzy/UxuXW3MWrfvj3//ve/CQ8P58knnwQgPDyc9PR0OnfunLnfrZ9jv3797nqO23+OIiIPQgFeRKQAGDBgAM2bN89xm4uLS+afhw4dyrp163jhhRdo2LAhbm5uWFlZsX79eqZNm5bjVBJ7e/u7nvdBr1zfPq3kTunp6Q/Ud14ZDIYc21NTU+96jIODQ47t7u7utGjRglWrVpGQkICzszPh4eFUqlSJgICAzP1uvdexY8dSsmTJHPvS9BkRyS8K8CIiFlS+fHkgI0g3bdr0nvvGxcWxbt06OnfuzEcffZRlW0GenlG8eHEcHR05efJktm05td1NhQoVOHHiBMnJyfdc0tLV1RXI+Abg1rcBkDH1JS+CgoJYtWoVy5cvx9fXlzNnzjB06NBstUFG4L/fz1FE5EFpDryIiAVVr16dKlWq8NNPP+UYMFNTUzOnoty6Wn7nle2oqKgcl5EsKKysrGjevDl79uxh+/btWbbdvlLN/XTs2JHY2Fi+++67bNtuH5NbYfrODzVTp041o+q/tWjRAnd3d8LDwwkPD8doNGaZPgPQtm1bbG1tmThxIomJidn6iI+PJzk5OU/nFxG5k67Ai4g8ZH/++WeOD09yd3enZ8+ejBs3jr59+9KpUye6du2Kn58fiYmJnD59mpUrVzJkyBC6dOmCs7MzzZo1Y/Hixdjb21OrVi3Onz/P3Llz8fHxyQz6BdHbb7/Npk2bePXVV3n55Zfx9vZm3bp1xMTEAHef9nK7Pn36sHbtWr7//nv27t1LYGAgtra2HDt2jJMnTzJt2jQAOnTowIQJExg5ciQnTpzAzc2NjRs3cvXq1TzVbmNjQ4cOHZg5cyb79u2jadOmeHl5ZdnH29ubkJAQRowYQbt27ejUqRNlypQhJiaGI0eOsGrVKn777Td8fHzyVIOIyO0U4EVEHrKNGzeycePGbO2+vr707NmTatWqsWjRIiZNmsSaNWv46aefcHJyokyZMgQFBdGkSZPMY8aPH88XX3zBmjVrWLRoERUqVOCdd97B2tqa999//1G+LbNUrFiRWbNmMXbsWGbMmIGdnR1PPfUUI0eOpFWrVrm6MdXW1pbQ0FBCQ0P59ddf+fLLL7Gzs6N8+fKZy2wCODs788MPP/DZZ58xadIkHB0dad26NePHj6dhw4Z5qv/5558nLCyMGzduZLv6fkvXrl2pUKECoaGhzJ07l/j4eNzc3PD19eWtt97C09MzT+cWEbmTId1SdxmJiMhjb9++fXTt2pWhQ4fy2muvWbocEZFCQXPgRUTkkbhzbnh6ejo//vgjgG78FBExg6bQiIjII9G5c2caN25MlSpVuHnzJmvXrmXbtm20a9eOmjVrWro8EZFCQ1NoRETkkRg3bhxr167l4sWLpKam4uPjQ8eOHRk4cCA2NjaWLk9EpNBQgBcRERERKUQ0B15EREREpBBRgBcRERERKUR0E6uZrl69jsn06GcdeXg4c+VKwiM/b2Gl8TKPxss8Gi/zaLzMo/Eyj8bLfBoz81hivIxGA+7uTnfdrgBvJpMp3SIB/ta5Jfc0XubReJlH42UejZd5NF7m0XiZT2NmnoI2XppCIyIiIiJSiCjAi4iIiIgUIgrwIiIiIiKFiAK8iIiIiEghogAvIiIiIlKIaBUaERERKXRSU1O4fj2OK1ciSUlJsXQ5hUpUlBGTyWTpMgqN/B4vKysbnJ1dcXC4+zKR96MALyIiIoVKamoKMTGXcHQshqurG+npRgwGg6XLKjSsrY2kpirA51Z+jld6ejopKUlcu3YZa2sbbGxs89SPptCIiIhIoXL9ehyOjsVwdnbF2tpG4V0KDYPBgK2tPU5OriQkXMtzPwrwIiIiUqgkJd3E3j7v0w9ELM3e3oGUlOQ8H68pNAXcn/svsnD9cWLikijuYkeXFpVoUsPb0mWJiIhYjMmUhpWVlaXLEMkzo9EKkyktz8crwBdgf+6/yPRlh0j+37yrK3FJTF92CEAhXkREHmuaNiOF2YP+/moKTQG2cP3xzPB+S3KqiYXrj1uoIhERERGxNAX4AuxKXJJZ7SIiIiJ3ExjYgClTJj3yYx/Ejh3bCAxswI4d2x75uQsyBfgCzMPFLsd2ZwebR1yJiIiIPGz79u1lypRJxMfHW7oUKeAU4AuwLi0qYWud9UdkABJupjBj+SGSU/J+84OIiIgULAcO7GXq1MkkJDycAL969R/07fvKIz9W8p9uYi3Abt2oevsqNM83r0jk5ess++sMR8/H8nqnGpTxdLZwpSIiIvIopaWlkZaWhq1t7h8EZGeX8zf7D/tYyX8K8AVckxreNKnhjadnMaKj//5EXq28Oz/+eoCPp2/jxVaVaVG7tO7IFxERKaSmTJnE1KmTAejevVNm+7x5iylVqjSBgQ3o3r0nVar4ExY2lfPnzzFhwrfUq9eA2bPD2LBhLWfOnCYxMZEKFXzp3bsfTz/dKss5AgMb0L//QIKDB2U5588/hzNlyiQ2bVoPQIsWLRkyZBj29vbZjn3llWCzj01KSuT77yeycuVykpNTqFevPu+++z5BQe2y9GmO1at/Z+bMaZw+fQpHRyeaNWvOoEFv4ubmlrnP2bNn+O9/J7J37x4SEuJxdXUjIKA2//rXBzg7Z1z8XLVqBbNnh3H27BkMBgPe3t506PA8L7zQ0+yaHiUF+EKqZkUPRg9oxI+/HmDG8sMcOHWVfm38cbTX/HgREZG8uPXslStxSXg84mevtGjRksjIc6xYsYw33xyCq2tGEHVzc8/cZ+vWLaxZ8ztBQd0pVqwYJUqUAGD+/J9o1uxJnn22DampKaxa9TsffjicceO+omnTwPuee8SI9yhd2ofXXx/MkSOHWLLkF9zc3HnjjTfz5dgxY0azZs1K2rbtQLVqNdi1awf/+tfb5g5RpqVLl/Dpp6OpUaMWgwa9SVTUJRYsmMvBg/uZPHkGdnZ2pKSkMGTIYKysjPTo8RKurq5cunSJzZs3kZAQj7OzM1u3biEk5ANatHiaTp2CSEtL49Spk+zdu1sBXh4eV2c73ulRhxV/nWHhhhOcjIzj9c41qFTG1dKliYiIFCqWfvaKn19l/P2rsWLFMpo3f4pSpUpn2+fs2TOEhf1MuXLls7TPmbMAO7u/r3h37dqDAQN6MXfurFwF+KpVq/Peex9kvo6NjeW338JzFeDvd+zhw4dYs2YlPXv25h//eAuALl268+mnozl27Mh9+79Tamoq338/ET+/KkycOClzCpG/f1VCQj5gyZJFdOv2IqdOneDChfNMnjydatVqZB5/+9X+zZv/wNe3ImPGjDe7DktTgC/kjAYDbRuXp0o5NyaF7+ezmTsIetKXto3LY9SUGhEReYz8sfcCm/ZcyNOxxyNjSU1Lz9KWnGpi6tKDbNgVaVZfgQGlaFarVJ7quJd69RpkC+9AlvAeFxeHyWQiIKAuq1atyFW/zz/fNcvr2rXrsGHDWq5fT8DJ6d732d3v2L/+2gxAUFC3LPt17dqDpUuX5Kq+2x06dICrV2MYOHBQlvn/LVs+y7fffs3mzX/QrduLmXX/8cdG/PyqYGOTfYaCs7MzUVGX2L9/HzVq1DS7FktSgC8iKpV2JaR/I2asOMSC9Sc4ePoqAztUx9VZN52IiIjcz53h/X7tlpDTVXnICKnTp0/h2LEjJCcnZ7bn9t44L6+s3zAUK+YCQHx8/H0D/P2OvXjxAlZWVnh7Z/1A4+Pjk6va7nTxYsYHtDs/yBiNRnx8ynLpUsb20qXL0KNHL6ZN+5G5c2dTt249mjZtTuvWbXB0dAIyvglYu3YVwcH9KFWqDA0bNuLpp1vRsOETeartUVKAL0Ic7a0J7lSD6hWKM3vlEUaGRvBqh+rUquhh6dJEREQeuma18n7l+1/f/ZHjgxI9XOwY1qveg5aWL26/0n7L7t07GT58CLVr12XIkGF4eJTA2tqapUuXsHLl8lz1azRa5dienn7/Dy8PcuzDNnjwO7Rv35GNG9cTEbGFL78cy4wZoUyaNBVPz5K4uxdn6tTZRERsYcuWzWzZspnFixfRvn0n3n9/pKXLvyetA1/EGAwGnqxdmg/7NcTVyZYJP+/m5zXHSE0zWbo0ERGRAiunZ6/YWhvp0qLSI6zC/Kmv69atwdbWli+//A8dOnSmSZNmBeoKsrd3KdLS0jKvnN9y7ty5PPcHcObM6Szt6enpnDt3Fi+vrB/gKlb0o2/fV/j228l8/30oUVGX+OWXBZnbbWxsaNasOUOHDuPnn3+hS5fu/PbbYs6fz1t9j4oCfBFVpoQTI/o04Ol6ZVgecYbPZm4n6uoNS5clIiJSIDWp4U3ftlUzn4Lu4WJH37ZVH9kqNAAODg4AZj3IyWg0YjAYMJn+vlB34UIkGzeuy+/y8qRRoyYALFo0P0v7ggVz89Rf1arVcXcvzi+/zCclJSWzfe3a1URHR9G0aTMArl9PIDU1NcuxFStWwsrKKnOaUWzstSzbDQYDlSpVBiApKfu3MQWJptAUYbY2VvRu7U/18sWZuvQgIVO30qeNP42rP7r/GImIiBQWt569Yin+/lUB+OGH73jmmdZYW1vTrNmTmcE+J02bBjJ37iyGDh3Ms88+x9WrV1m4cB5lypTl+PGjj6r0u6patRpPPdWSOXPCuHbtauYykmfPZlxBN/cZNtbW1gwaNJhPPx3N4MHBtGrVmqioS8yfP5eKFSvRsWMQANu3b2PChHE89dQzlCtXHpMpjRUrlmEwGGjRoiUA//73J8THx1GvXgNKlizJpUsZ/VSuXIUKFXzzdyDymQL8Y6C+vycVvIsxacl+flh8gAMnr9Lr2SrY2eY8b01EREQevSpVqhIc/A8WLpzHX3/9iclkYt68xfcM8PXrN2T48A+ZOXM633zzJaVKlWbQoMFcuBBZIAI8wIgRH1G8uAerVv3OunVraNCgEaNHf8ZLL3U160myt7Rr1xFbW1tmzZrOt99+jZOTE88+24bXXx+c+cRYP7/KNGrUmM2bNxIevhB7e3v8/Crz+effULNmLQCee64tixcvYtGi+SQkxFO8uActW7ZiwIDXMBoL9iQVQ3pBuMugELlyJQGT6dEP2Z1PYs2LNJOJxZtO8evmU3gVd+T1zjUo51UsnyosWPJjvB4nGi/zaLzMo/Eyj8br/i5ePI23d8YqJNbWRlJTdZ+XOQrCmB09epj+/XsxcuTHtG7d1qK13M/DGq/bf4/vZDQa8PC4+wpABfvjheQrK6ORoCcr8m7PuiQmp/LJjG2s3n6uQNwpLiIiIkVTUlJitraff56D0Wikdu26Fqio8NMUmsdQtfLuhAxoROhvB5m18ggHTsXQv101nB2yP+RARERE5EGEhU3j2LEj1KvXAIPByF9/ZSzZ2KlTULZ15CV3FOAfUy6OtrzVLYCV284xb+0xRoVG8FrH6viXc7d0aSIiIlKE1KwZwPbtEUyd+iM3b97Ay8ubV14Jpnfv/pYurdBSgH+MGQwGWjcsi39ZN/4bvo9xc3bSqZkvHZtWwGg0fy1aERERkTs1btyUxo2bWrqMIkVz4IXy3sUY2a8hjat7E77pJOPm7CQmLvt8NRERERGxPAV4AcDBzpqBHavzaodqnL4Yz6jQCHYejbZ0WSIiIiJyBwV4yaJpzVKE9G9ICVcHJi7Yy6yVR0hJTbN0WSIiIiLyPwrwko1XcUf+r3d9nm1QltXbz/HJjO1cuHLd0mWJiIiICArwchc21kZ6tqrMW90CuBqfxOhpW9m4J1JrxouIiIhYmAK83FNtvxKMHtCIiqVcmLr0EJOXHOBmUqqlyxIRERF5bCnAy325F7Pj3RfrEvRkRSIORjF66lZOXoizdFkiIiIijyUFeMkVo9FAx6YVGNarLmkmE5+GbWf5X2cwaUqNiIhIgTRmTAjdunXMfH3hQiSBgQ349dfFZh+bH6ZMmURgYIN87TM3duzYRmBgA3bs2PbIz/2wKMCLWSr7uBEyoBF1/Erw89pjfDVvN3HXky1dloiIiBQQYWHT2LBhnaXLKNIU4MVsTvY2vBFUk97P+XPo9DVGhUaw/1SMpcsSERGRe/D2LsXq1X/Qtm37h3qeWbOmsXHjumztffu+wurVfzzUcz8uFOAlTwwGA0/XLcPIvg1wtLfmy592MX/dcVLTTJYuTURERHJgMBiws7PDysrKIue3trbGzs7OIucuaqwtefLk5GS+/vprwsPDiYuLo2rVqrzzzjs0adLknsft2bOHhQsXsmfPHo4cOUJKSgqHDx/Ocd+oqCi++eYbNm/ezJUrV/Dy8qJ169a89tpruLi4PIy39VjxKenMyH4NmbPqKEu3nObwmasEd6pBCTcHS5cmIiJSaKxZs4qRI4fz3Xc/EhBQJ8u2mTOnMWnSt8yfvwQvL292797JvHk/ceDAPq5ejcHdvThPPdWS4OB/YGdnf9dzXLgQSffunRgxIoQ2bTpktm/YsI7Jk7/j/PlzlCnjw6uvDsrx+Nmzw9iwYS1nzpwmMTGRChV86d27H08/3Spzn1tz3Jct+5Vly34FoG3bDnzwQQhTpkxi6tTJbNr091z01NRUZswIZdmy37h8OQpPz5K0a9eR3r37Z/mgERjYgO7de1K7dh1+/HES58+fpUyZsvzzn2/TuHFTM0b6b6tX/87MmdM4ffoUjo5ONGvWnEGD3sTNzS1zn7NnzzBp0n/Ys2c3CQnxuLq6ERBQm3/96wOcnZ0BWLVqBbNnh3H27BkMBgPe3t506PA8L7zQM0915YZFA/zw4cP5/fff6dOnD+XLl2fRokUMHDiQsLAw6tate9fj1q9fz7x58/D396ds2bKcOHEix/1u3LjBiy++yI0bN+jVqxfe3t4cOHCAqVOnsmPHDmbPnv2w3tpjxc7Gin5tq1K9gjvTlx9i1NSt9GtblYZVS1q6NBERkVyL27KZywsXkBpzBeviHpTo0hWXPIZDczVtGoiDgwNr1qzKFuDXrFlFzZq18PLyBmDt2lUkJSUSFNQNFxdXDh7cz4IFPxMVFcUnn4w167wREVsYMeI9fH0rEhz8D2JjY/nss9F4emb/N3z+/J9o1uxJnn22DampKaxa9TsffjicceO+omnTQAA+/PAjxo//FH//anTqFARAmTI+dz3/2LGfsGzZrzzzTGsCAnqxZ89Ofvzxv1y6dJFhw0Zk2XfXru2sXbuKoKBuODg4Mn/+T4wY8R4LFvyKq6vbXc6Qs6VLl/Dpp6OpUaMWgwa9SVTUJRYsmMvBg/uZPHkGdnZ2pKSkMGTIYKysjPTo8RKurq5cunSJzZs3kZAQj7OzM1u3biEk5ANatHiaTp2CSEtL49Spk+zdu7toBvg9e/bw22+/8f7779OvXz8Ann/+eTp06MDnn3/OrFmz7npsz549GThwIPb29owZM+auAX7dunWcP3+eSZMm8dRTT2W229vbExoaytmzZylbtmx+vq3HWqNqXviWcmHS4v18/8s+9tcuTc9WlbGzscxXdSIiIrkVt2Uzl2ZMIz05Y2GG1JgrXJoxDeCRhHh7e3uaNg1k3brVvPXWUAwGAwDnz5/jyJFDvPXWu5n7Dho0OMuV9s6du1CmTFl++OFbLl68iLe3d67P+/3331CihCfffz8FR0cnAOrWrcc77/wTb+9SWfadM2dBlvN27dqDAQN6MXfurMwA/9xz7ZgwYRylS5fhuefa3fPcR48eYdmyX3n++a68++77/+vzBZydixEevpCuXXvg51c5c//Tp08xc+Y8SpcuA0C9eg3o168nq1atoGvXHrl+z6mpqXz//UT8/KowceIkbG1tAfD3r0pIyAcsWbKIbt1e5NSpE1y4cJ7Q0DCqVKmWefwrrwRn/nnz5j/w9a3ImDHjc33+/GCxAL98+XJsbGzo3r17ZpudnR3dunVjwoQJREVFUbJkzldwS5QokatzJCQkAODh4ZHj8fb2d/+aSfLG082B4b3q8cvGkyzbcppj52N5vXMNfDydLV2aiIgUcXGb/yB204Y8HZt44jjpqVkfVJienMylaaHEblhvVl+ugU/i0rSZ2TW0bPksq1evZM+eXdSunTETYc2alRiNxizTVG4P0Tdv3iQpKYlatQJIT0/n6NFDuQ7wly9f5ujRI/Tt+0pmeAdo2LAxFSpUJDHxZpb9bz9vXFwcJpOJgIC6rFq1wuz3CrBlS8YNrT169MrS3qPHS4SHL+TPP//IEuAbNWqcGd4B/Pwq4+TkRGTkebPOe+jQAa5ejWHgwEGZ4R0yxv/bb79m8+Y/6NbtRZycMrLLpk0b8PX1w8bGJltfzs7OREVdYv/+fdSoUdOsOh6ExQL8wYMH8fX1xcnJKUt7QEDGL+DBgwfvGuBzq379+hiNRsaMGcPw4cOzTKHp0qULnp6eD9S/5Mzayki3pypRrbw7k389wMfTt/HiM5V5qk7pzCsKIiIiBcmd4f1+7Q9D48bNcHR0YvXq37ME+ICAOlkuXl68eJEpU/7Lpk0biI/P+mDFWxcvc+PixQsA+Phkn41Qrlx5jhw5lKXtjz82Mn36FI4dO0Jy8t9LSOf13/aLFy9gZWWVbYpNmTJlsbKy4tKlC1nab00hul2xYi7Ex8ebfV7IeI+3MxqN+PiUzTxv6dJl6NGjF6Ghk5kzZxZ169ajadPmtG7dJvMDT5cu3Vm7dhXBwf0oVaoMDRs24umnW9Gw4RNm1WQuiwX46OhovLy8srXfCtVRUVEPfI5KlSrx0UcfMW7cOHr0+PurlR49ehASEvLA/cu91fAtzugBjZjy6wHCVhzmwKkY+rWtipN99k+wIiIiD8qlabM8XfkGOPHeUFJjrmRrty7uQdn33n/Q0nLFzs6OZs2as379Gt5++19ERp7n6NEjDBkyLHOftLQ03nnnDeLj4+jVqw/ly1fA3t6By5ejGTMmhPSH9IDF3bt3Mnz4EGrXrsuQIcPw8CiBtbU1S5cuYeXK5Q/lnHcyGnOekvuw3jPA4MHv0KlTZ9atW0tExBa+/HIsM2aEMmnSVDw9S+LuXpypU2cTEbGFLVs2s2XLZhYvXjGqbsoAACAASURBVET79p14//2RD60uiwX4xMTEHL+KuLW8UFJSUr6cx9vbm9q1a/Pkk09SunRptm3bRlhYGK6urgwdOtTs/jw8LDcVxNOzmMXOnVeenjDmjUB+WX+cGUsP8FHUNt7tVZ/qvh73P/iBz134xsuSNF7m0XiZR+NlHo3XvUVFGbG2/nsl7Nv/nFclu3XnwrTQzDnwAAZbW0p2654v/efWs8+2ZuXK5ezdu4u9e3djZWVFq1bPZtZw4sRRzp49w8iRH9Gu3d8ryfz11xYg48npt/a9dWX81msrq6xj5uNTGoDIyHPZ3uPZs6ezHLthwxpsbe345pvvskw7ubXSzO3HGwwGDAZDtj6Nxqz1lC5dmrS0NC5ejKRcuXKZ+0VGniUtLY3SpUtn6cNozPlnndO5bnfrfVtZZfzelCmT8b7PnTtDgwZ/Pxk2PT2dc+fOUqlSpSz9VarkR6VKfrzyykD27dvLq6/2ZfHihQQHv/G/92NHixYtaNGiBenp6XzxxVjmz/+Z/v1fyfHbjb/fjzHPf9ctFuDt7e1JSUnJ1n4ruOfHOqHbt2/n9ddfZ/78+VSrlnHzQatWrXB2duY///kPQUFBVKxY0aw+r1xJwGR6eJ/07sbTsxjR0eZ9RVSQNK/pRZniDkxavI/3v/2Dzs19ad+4fOZf5vxW2MfrUdN4mUfjZR6Nl3k0XvdnMplITc147oi1tTHzzw/CuVFjvEymbKvQODdqnC/951aDBo1xcnJi5coV7Nu3lzp16uPi4pZZQ3p6xr+baWmm29rSmTs3Y2U9kyk9SzuQ+Trttme1pKaacHPzoHLlKvz22xJ69uydOS1k69YtnDx5Am/vUre9dyMGAyQnp2I0ZsTHCxci2bBhbZZzANjbOxAfH5dt3G7lp1vtjRo15fvv/8OcObMYOvTvbxnmzMl4L0880SxLHyYTOf4s0tPT7/kzuvW+b41Z5crVcHcvzsKF83juufaZF5TXrFlFdHQUvXr1ITXVxPXrCdjZ2WNvb5vZf/nyFbGysiIxMYnUVBOxsdeyrYDj6+sHwI0bifesy2Qy3fXvutFouOdFY4sFeE9PzxynyURHRwM88Px3gLlz51KyZMnM8H5Ly5YtmThxIrt27TI7wEveVSztQkj/RsxYcZhFG05w8FQMAzvWwL2YHuogIiKW59K46SNbNvJubG1tCQxswYoVy7h58wbvvfdBlu3ly1egTBkfvv32K6Kjo3BycmLdujVmzwO/JTj4n7z33tsMGvQK7dp1JC4ujgUL5uLrW5GbN/++ibVp00Dmzp3F0KGDefbZ57h69SoLF86jTJmyHD9+NEuf/v5V2bYtgp9+mkmJEp6UKlUmxxs8K1euQtu2HVi0aB7x8XEEBNRhz55drFq1gg4dOlOpkl+e3tP9WFtbM2jQYD79dDSDBwfTqlVroqIuMX/+XCpWrETHjhnLX27fvo0JE8bRsuUz+PiUx2RKY8WKZRgMBlq0aAnAv//9CfHxcdSr14CSJUty6VJGP5UrV6FCBd+HUj9YMMBXrVqVsLAwrl+/nuVG1t27d2duf1BXrlwhLS0tW3vq/25IyWmbPFwOdta81rE61Su4M2vlEUaFRvBqh2oEVMrdykIiIiJF3TPPtGbFiqVYWVnRosXTWbZZW1szduwEvvpqPGFh07Czs6V586fp2vUF+vUzf93xxo2b8vHH/2by5O+ZNOlbSpf24f33R7Fp03p27tyeuV/9+g0ZPvxDZs6czjfffEmpUqUZNGgwFy5EZgvw//jH24wd+wmTJ39PUlISbdt2uOsKLcOGjaBUqdIsW/Yr69atxtOzJK+++jq9e/c3+72Yo127jtja2jJr1nS+/fZrnJycePbZNrz++uDMWSB+fpVp1Kgxf/yxkaiohdjb2+PnV5nPP/+GmjVrAfDcc21ZvHgRixbNJyEhnuLFPWjZshUDBryG0fjwpl4Z0h/mzP972L17Ny+88EKWdeCTk5Pp0KEDHh4ezJkzB4DIyEhu3rxJpUqVcuxnzJgxzJgxI8cnsX788cfMnDmTWbNmZZnj9OmnnzJ9+nQWLFhAzZrmLfmjKTT558KV63z/y37ORSfQumFZuj1VCWur/PllL4rj9TBpvMyj8TKPxss8Gq/7u3jxNN7eGSuI5NcUmseJxsw8D2u8bv89vlOBnUJTu3Zt2rRpw+eff050dDTlypVj0aJFREZG8tlnn2XuN2zYMCIiIrIE9PPnzxMeHg7A3r17Afjuu++AjCv3LVtmfK3Rq1cvFi5cSHBwMC+//DKlSpVi69at/PrrrzRv3tzs8C75q5SHEx/2rc/Pa47z+9azHD57jdc718DL3dHSpYmIiIgUWBYL8ADjxo3jq6++Ijw8nNjYWPz9/fnhhx+oX7/+PY87d+4cX3/9dZa2W6+DgoIyA3zFihVZsGBB5jkuX75MyZIlefXVVxk8ePDDeVNiFhtrK3q1rkL1Cu6ELj1IyNSt9GntT5OauX+KnIiIiMjjxGJTaAorTaF5eGLiEvlh8X6OnIulWU1verWugr1t3j5jPg7jlZ80XubReJlH42Uejdf9aQrNg9GYmacgTqF5dAubitxHcRd7/vVSXTo1q8Dm/RcZPW0bpy/qHzERERGR2ynAS4FiZTTyfPOKvNezLskpaYwJ28bKrWcf6lPWRERERAoTBXgpkPzLuTN6QCNq+nowZ/VRJi7YS/yN5PsfKCIiIlLEKcBLgeXsYMPgrrV4qVVl9p28wqjQCA6dvmrpskREREQsSgFeCjSDwUCrBmUZ0acB9rbWjJ+zk0UbTpBm0s03IiKPM02tlMLsQX9/FeClUCjnVYyR/RrQtJY3SzafYtzsncTEJVq6LBERsQArKxtSUpIsXYZInqWkJGNllffV3BXgpdCwt7XmlfbVea1jdc5GJTAqNIIdR6ItXZaIiDxizs6uXLt2mevX40lNTdXVeCk00tPTSU5O4tq1aJyd3fLcj0Uf5CSSF41reFOxtAv/Dd/Pfxbu5el6ZXixpR821laWLk1ERB4BBwcnrK1tSEi4RlJSAsnJWuTAHEajEZOmouZafo+XlZU1xYq54+DglOc+FOClUCrp7sj/9a7PgvXHWRFxlqNnY3m9cw1Kl8j7XwYRESk8bGxscXcvqQdf5YHGzDwFcbw0hUYKLWsrIz1aVubt7rWJvZ7ER9O3smF3pL5KFRERkSJNAV4KvYBKHowe0IhKpV2ZtuwQkxbv5/rNFEuXJSIiIvJQKMBLkeDmbMfQHnXo2qIi2w5F89aX6zgRGWfpskRERETynQK8FBlGo4H2TSowvFc90tPT+WzmdpZtOY1JU2pERESkCFGAlyLHz8eVr4c+Td3KJZi37jgTft5N7HWtUCAiIiJFgwK8FEnODjYMer4mfdr4c+TsNUaFRrDv5BVLlyUiIiLywBTgpcgyGAw8VacMH/ZtQDEHG76cu5t5a4+Rmqa1b0VERKTwUoCXIs/H05kRfRvwVJ3SLPvrDJ/N3EHUtZuWLktEREQkTxTg5bFgZ2NFnzZVeeP5mlyMucHoqRFEHLxk6bJEREREzKYAL4+VBlVLMrp/Q0qXcOK/4fuZuvQgSclpli5LREREJNcU4OWxU8LNgWEv1aN9k/Js2nOBj6Zv5WxUgqXLEhEREckVBXh5LFlbGenaohJDX6zDjcRUPp6+jTU7zpGuNeNFRESkgFOAl8da9QrFGT2gEdXKuzPz9yN8u2gfCTdTLF2WiIiIyF0pwMtjz8XJlre6B9CjpR+7j10mZGoER85es3RZIiIiIjlSgBcBjAYDzzUqx//1ro+10cjY2TtY/MdJTCZNqREREZGCRQFe5Da+pVwY1b8hT1T34peNJ/n8p51cjU+ydFkiIiIimRTgRe7gYGfNwA7VeaV9NU5eiGdUaAS7jl22dFkiIiIigAK8SI4MBgPNapViZL8GFC9mxzfz9zB71RFSUk2WLk1EREQecwrwIvdQysOJD/o0oFV9H1ZtO8eYsG1cjLlh6bJERETkMaYAL3IfNtZGXnq2Cm92DSAmLonRU7fyx94Lli5LREREHlMK8CK5VKdyCUYPaEQF72JM+e0gk5fs52ZSqqXLEhERkceMAryIGdyL2fGvnnV5PtCXLQcuMXraVk5djLN0WSIiIvIYUYAXMZPRaKBToC/DXqpHSqqJMTO283vEGdLTtWa8iIiIPHwK8CJ5VKWsG6MHNCKgkgc/rTnG1/P3EHcj2dJliYiISBGnAC/yAJwdbPhnl1r0erYKB05dZVRoBAdPX7V0WSIiIlKEKcCLPCCDwcAz9X0Y0ac+jnbWfD5nJws3HCfNpDXjRUREJP8pwIvkk3JexRjZtyGBAaX4dfNpxs7ayeXYm5YuS0RERIoYBXiRfGRna0X/dtUI7lSDc9EJhIRuZfvhKEuXJSIiIkWIArzIQ/BEdS9CBjTCq7gj3y7ax4wVh0lOSbN0WSIiIlIEKMCLPCQl3Rx4/+V6tH2iHOt2nufjGds4H51g6bJERESkkLN4gE9OTmb8+PEEBgYSEBDACy+8wJ9//nnf4/bs2UNISAhdunShZs2a+Pv733P/kydP8vbbb9O4cWMCAgJo27YtkydPzq+3IZIjaysj3Z/2Y8gLtYm/nszH07exftd5rRkvIiIieWbxAD98+HCmT59Op06d+OCDDzAajQwcOJCdO3fe87j169czb948AMqWLXvPfffv30+3bt04f/48wcHBjBgxglatWnHx4sV8ex8i91KzogejBzSiso8r05cf5vvw/dxITLF0WSIiIlIIGdIteClwz549dO/enffff59+/foBkJSURIcOHShZsiSzZs2667GXL1/G2dkZe3t7xowZw4wZMzh8+HC2/dLS0ujUqRO+vr588803GI0P9pnlypUETKZHP2SensWIjo5/5OctrArqeJnS01nx1xkWbjiBezE7gjvVoFIZV0uXVWDHq6DSeJlH42UejZd5NF7m05iZxxLjZTQa8PBwvvv2R1hLNsuXL8fGxobu3btnttnZ2dGtWze2b99OVNTdV+8oUaIE9vb29z3Hpk2bOHbsGO+88w5Go5Hr169j0vrcYiFGg4G2jcsz/OV6AHw2cwe//XkKk6bUiIiISC5ZNMAfPHgQX19fnJycsrQHBASQnp7OwYMHH/gcf/75J87Ozly6dInnnnuOevXqUa9ePUaMGMHNm1qjWyyjUmlXQvo3okFVTxasP8GXc3cRm5Bk6bJERESkELBogI+OjqZkyZLZ2j09PQHueQU+t06fPk1aWhpvvPEGgYGBTJw4kZ49ezJ//nyGDh36wP2L5JWjvTXBnWrQr21Vjp2LZWRoBHtPXLF0WSIiIlLAWVvy5ImJidjY2GRrt7OzAzLmwz+oGzducPPmTV588UU+/PBDAFq3bo3BYGDKlCkcOnSIqlWr5rq/e81Hetg8PYtZ7NyFUWEZr66tXGhYsxTjZ25nws+7CXrKj95tq2Fj/Wg/XxeW8SooNF7m0XiZR+NlHo2X+TRm5ilo42XRAG9vb09KSvaVOG4F91tB/kHPAdChQ4cs7Z06dWLKlCls377drACvm1gLh8I2Xg5WBoa/VJe5a4+xaN0xdh2+RHCnGpR0d3wk5y9s42VpGi/zaLzMo/Eyj8bLfBoz8+gm1jt4enrmOE0mOjoaIMfpNXk5B4CHh0eW9luv4+LiHvgcIvnB1saK3q39+UdQLS7F3CRk6la2HNBSpyIiIpKVRQN81apVOXnyJNevX8/Svnv37sztD6pGjRoAXLp0KUv7rTXgixcv/sDnEMlP9f09CRnQEJ+Szvyw+AChvx0kKTnN0mWJiIhIAWHRAN+mTRtSUlIyH8gEGU9mXbhwIfXq1cPLywuAyMhIjh8/nqdztGzZEhsbG+bPn5+lfd68eRgMBho3bpz3NyDykJRwdWDYS3Xp0LQCf+y9wOhpWzlzSV93ioiIiIXnwNeuXZs2bdrw+eefEx0dTbly5Vi0aBGRkZF89tlnmfsNGzaMiIiILA9qOn/+POHh4QDs3bsXgO+++w7IuHLfsmVLALy8vHjttdf49ttvSUlJoXHjxuzcuZPFixfz0ksvUb58+Uf1dkXMYmU00uXJilQr787kJfv5ZMY2erSsTMt6ZTAYDJYuT0RERCzEogEeYNy4cXz11VeEh4cTGxuLv78/P/zwA/Xr17/ncefOnePrr7/O0nbrdVBQUGaABxg8eDAuLi7Mnj2bNWvWULJkSd5++22Cg4Pz/w2J5LNq5d0JGdCI0N8OMmvlEQ6ciqF/u2o4O2RfwUlERESKPkN6uh4BaQ6tQlM4FMXxSk9PZ+W2c8xbewwXJ1te61gd/3Lu+dJ3URyvh0njZR6Nl3k0XubReJlPY2YerUIjInlmMBho3bAsI/o0wNbayLg5OwnfdNIiHyhFRETEchTgRQqZ8t7FGNmvIY2rexO+6STj5uwkJi7R0mWJiIjII6IAL1IIOdhZM7BjdV7tUI3TF+MZFRrBzqPRli5LREREHgEFeJFCrGnNUoT0b0gJVwcmLtjLrJVHSEnVmvEiIiJFmQK8SCHnVdyR/+tdn2cblGX19nN8MmM7F65cv/+BIiIiUigpwIsUATbWRnq2qsxb3QK4Gp/E6Glb2bgnEi0yJSIiUvQowIsUIbX9SjB6QCMqlnJh6tJDTF5ygJtJqZYuS0RERPKRArxIEeNezI53X6xL0JMViTgYxeipWzl5Ic7SZYmIiEg+UYAXKYKMRgMdm1ZgWK+6pJlMfBq2neV/ncGkKTUiIiKFngK8SBFW2ceNkAGNqONXgp/XHuOrebuJu55s6bJERETkASjAixRxTvY2vBFUk97P+XPo9DVGhUZw4FSMpcsSERGRPFKAF3kMGAwGnq5bhpF9G+Bob80XP+1iwfrjpKaZLF2aiIiImEkBXuQx4lPSmZH9GtK8dml++/M0Y2ft4PK1m5YuS0RERMygAC/ymLGzsaJf26q83rkGkVeuM2rqVjbtPm/pskRERCSXFOBFHlONqnkR0r8RpTwcGTtjG9OXHyIpJc3SZYmIiMh9KMCLPMY83RwY3qse3VpWZsOuSD6Zvo1z0QmWLktERETuQQFe5DFnbWWkb/vqDOlRh/ibKXw8fRvrdp4nXWvGi4iIFEgK8CICQA3f4owe0Aj/sm7MWHGY737Zx/XEFEuXJSIiIndQgBeRTK5Otrz9Qm1eeNqPXUcvExIawbFzsZYuS0RERG6jAC8iWRgNBto8UY73X66P0Wjg37N28OvmU5hMmlIjIiJSECjAi0iOKpZ2IaR/IxpWK8nCDSf4Yu4ursYnWbosERGRx54CvIjclYOdNa91rE7/dlU5HhnLqNAI9hy/bOmyREREHmsK8CJyTwaDgeYBpRnVryFuznZ8NW8PP60+SmqaydKliYiIPJYU4EUkV0p5OPFh3/o8U8+H37eeZUzYdi5dvWHpskRERB47CvAikms21lb0al2FwV1qcfnaTUKmbuXPfRctXZaIiMhjRQFeRMxWt4onowc0onxJZyb/eoApvx4gMTnV0mWJiIg8FhTgRSRPirvY86+X6tKpWQU277/I6GnbOH0x3tJliYiIFHkK8CKSZ1ZGI883r8h7PeuSnJLGmLBtrNx6lvR0rRkvIiLysCjAi8gD8y/nzugBjajp68Gc1UeZuGAv8TeSLV2WiIhIkaQALyL5wtnBhsFda/FSq8rsO3mFUaERHDp91dJliYiIFDkK8CKSbwwGA60alGVEnwbY21ozfs5OFm04QZpJa8aLiIjkFwV4Ecl35byKMbJfA5rW8mbJ5lOMm72TmLhES5clIiJSJCjAi8hDYW9rzSvtqzOwY3XORCUwKjSCHUeiLV2WiIhIoacALyIPVZMa3oT0b0gJNwf+s3AvYb8fJiU1zdJliYiIFFoK8CLy0Hm5O/JB7/o816gsa3ec5+Pp24m8fN3SZYmIiBRK+RLgU1NTWbFiBT///DPR0fqKXESys7Yy0qNlZd7uXpvY60l8NH0rG3ZHas14ERERM1mbe8C4ceP466+/WLBgAQDp6en079+fbdu2kZ6ejpubGz///DPlypXL92JFpPALqOTB6AGNmLzkANOWHeLAqRj6PFcVR3uz/3MkIiLyWDL7CvzGjRtp0KBB5us1a9awdetWXnnlFb744gsAfvjhh/yrUESKHDdnO4b2qEPXFhXZdiiakKkRnIiMs3RZIiIihYLZl7wuXrxI+fLlM1+vXbsWHx8f3n33XQCOHj3KkiVL8q9CESmSjEYD7ZtUwL+sO5MW7+ezmdvp8mRFnnuiHEaDwdLliYiIFFhmX4FPSUnB2vrv3P/XX3/RtGnTzNdly5bVPHgRyTU/H1dGD2hI3colmLfuOBN+3k3s9WRLlyUiIlJgmR3gvb292blzJ5Bxtf3s2bM0bNgwc/uVK1dwdHTMVV/JycmMHz+ewMBAAgICeOGFF/jzzz/ve9yePXsICQmhS5cu1KxZE39//1ydb+nSpfj7+2eZAiQiludob8Og52vSp40/R85eY1RoBPtPxli6LBERkQLJ7ADfvn17fvnlF4KDgwkODsbZ2ZkWLVpkbj948GCub2AdPnw406dPp1OnTnzwwQcYjUYGDhyY+QHhbtavX8+8efOAjCv+uZGYmMj48eNz/eFCRB4tg8HAU3XK8GHfBhRzsOGLubuYt+4YqWkmS5cmIiJSoJgd4IODgwkKCmLXrl0YDAbGjh2Li4sLAPHx8axZs4YmTZrct589e/bw22+/8e677/Lee+/Ro0cPpk+fTqlSpfj888/veWzPnj3Zvn07CxcuJDAwMFd1T548GVtbW1q2bJmr/UXEMnw8nRnRtwFP1SnNsi1n+PesHURfu2npskRERAoMs29itbW15dNPP81xm5OTE5s2bcLe3v6+/SxfvhwbGxu6d++e2WZnZ0e3bt2YMGECUVFRlCxZMsdjS5QoYVbNkZGR/Pjjj0yYMIHff//drGMtLW7LZi4vXMCRqzFYuxenRJeuuDRuev8DRQoxOxsr+rSpSvUKxZm67BAhUyPo26Yqjap5Wbo0ERERi8vXJ7GmpqZSrFgxbGxs7rvvwYMH8fX1xcnJKUt7QEAA6enpHDx4MN/qGjt2LHXr1i10V9/jtmzm0oxppMZcgfR0UmOucGnGNOK2bLZ0aSKPRIOqJRndvyGlSzjx3/D9TFt2kKSUNEuXJSIiYlFmB/j169czceLELG2zZs2iXr161KlTh6FDh5KSknLffqKjo3O8wu7p6QlAVFSUuaXlKCIigpUrVzJ8+PB86e9RurxwAenJWVfjSE9OJmrWTGL/2Mj1vXtIPH2KlKtXSU9NtVCVIg9XCTcHhr1Uj/ZNyrNx9wU+mraVc1EJli5LRETEYsyeQjNlyhQ8PDwyXx8/fpxPP/2UsmXL4uPjw9KlS6lVqxb9+vW7Zz+JiYk5Xqm3s7MDICkpydzSsklLS+OTTz6hS5cuVK1a9YH7A/DwcM6XfnLjyNWcV+Ew3bzBpalTsrVbF3PGxs0NWzc3bNxcs/3ZxtU187UxF9+SFHaensUsXUKhUtDH6/VudWgSUIYvZm/n4xnbeLVzTdo2qYDBQmvGF/TxKmg0XubReJlH42U+jZl5Ctp4mR3gT5w4kWXVmaVLl2JnZ8f8+fNxdnZm6NCh/PLLL/cN8Pb29jleqb8V3G8F+Qcxd+5czp07R2ho6AP3dcuVKwmYTOn51t+9WLsXz5g+k0O7z3vDSYuLIy0ultS4ONLi4kiNiyUtNpakuDhuXDpGWlwspsTEHPs2Ojpi5eKCtYsrVi6uWLu43PbaJaPNNaPNaGP7sN9qvvP0LEZ0dLylyyg0Cst4lXa3Z1S/hkz57SDfL9jDX3sv0K9tVZwdHu0H0sIyXgWFxss8Gi/zaLzMpzEzjyXGy2g03POisdkBPjY2Fnd398zXmzdvpnHjxjg7Z5ykUaNGrF+//r79eHp65jhN5tZDoO52A2tuJScn880339ClSxcSExM5d+4cADdu3MBkMnHu3DkcHR0pXrz4A53nYSrRpSuXZkzLMo3GYGtLia7dsPUsCZ73HyNTcnL2kH978I+NJensGW7ExWK6mfNKH0YHh+zhPsv//73NmA8fvETuxcXJlre6B7By61nmrztOyNQIXutYgypl3SxdmoiIyCNhdoB3d3cnMjISgISEBPbu3cuQIUMyt6emppKWdv+bzKpWrUpYWBjXr1/PciPr7t27M7c/iMTERK5evUpYWBhhYWHZtj/zzDO0a9eOCRMmPNB5HqZbq81cXriA1DyuQmO0tcVYwhObEp733deUkkxaXPxtgf/v4H/rz8nnz5N68CCmG9dz7MNgZ5811Lv+HfLvDP7GXKxWJJITo8HAc43KUaWsG5PC9zN29g6eD/SlfZMKGI2WmVIjIiLyqJgd4OvUqcNPP/2En58fGzZsIC0tjSeffDJz++nTp3N19bxNmzaEhoYyb968zOk2ycnJLFy4kHr16uHllbFcXGRkJDdv3qRSpUpm1eng4MC3336brX3GjBns2bOHzz//PPMcBZlL46a4NG76SL6+MdrYYvTwwOa2exzuJj019e5X9f/XlnzpIqlHD2NKyPmGQ4Ot7W1X9bMH/tuv+Bvt7S0211kKLt9SLozq35Cw3w+zaONJDp6+ysCONXAvpm+CRESk6DI7wL/55pv06dOHt99+G4CgoCD8/PwASE9PZ9WqVTzxxBP37ad27dq0adOGzz//nOjoaMqVK8eiRYuIjIzks88+y9xv2LBhREREcPjw4cy28+fPEx4eDsDevXsB+O6774CMK/ctW7bExsaGVq1aZTvvqlWrOHDgQI7bJPcM1tbYFC+OTS6mIKWnppKWEP/3Vf3YuGyBPyU6msTjx0hLSID07PcYGGxs7gj1dwb+v4O/0UFP232cONhZM7BDdWpUKM7M348wKjSCAe2rUcfPvOdF3X5FFwAAIABJREFUiIiIFBZmB3g/Pz+WLl3Kjh07KFasGA0bNszcFhcXR9++fXMV4AHGjRvHV199RXh4OLGxsfj7+/PDDz9Qv379ex537tw5vv766yxtt14HBQUVuvXeizqDtTXWbu5Yu7nfd9/0tDTSEuL/dxU/LsfpPKkxV0g8dZK0uLicw761Nafd3DA4F8t2Nf/OefxGJydd2S8CDAYDzWqVomJpFyaF7+eb+Xto1cCH7k/5YWOdr4+7EBERsThDenoOCUju6lGuQnM73TGeXbrJRFpCQpaQfyv42yTfICHqyt9TfOLjIad7M6ysMkJ+sRxuynXNukqP0ckJg7FohsGi9PuVkmpi3tpjrNp+jnJezrzeuSbexfP3W5miNF6PgsbLPBov82i8zKcxM0+RWIXmljNnzrB69WrOnj0LQNmyZXnmmWcoV65cXrsUMYvBaMTaxQVrFxfunPF851+2dJMJ0/Xrd72q//dNuudIjYvNOewbjVgVc/n/9u48Lspy/x//656NAQZmWIZ9FQUUZHHXNreKY7aezKMWqWVWnzrlqfPR8nNO2ylb7KTHU7+yNNOvablFmpmarS6ZqJCKoIgCojCigKwDzP37Y+CGkQEdWYaB1/Px6BFcc99zX/fV7fjqnut+X1bm6LecziPXaHps2O/ulAoZpt4aiQFhnli+NQOvfPo7HrwtEjcM9Ld314iIiDrEdQX4RYsW4eOPP25Rbeadd97B7Nmz8cwzz3RI54g6iiCTQe7mBrmbGxAY2Oa2oijCVFmJutJSi7v6VwZ/47lzqC8rtb4KriCYj2el1Kb0b23D/wBo3CDI5Z105r1XQj9vvDJzGJZ+fRTLvsnAsdMX8eBtUXB2uu77FkRERN2CzX+TrV+/Hh9++CESExPx6KOPol+/fgCAEydOYNmyZfjwww8RHByM++67r8M7S9QVBEGA3NUVcldXICCgzW1FUYSpqqpZNZ6W03nqy0phLCpEfVmZRU3/ZgeEXKOxCPtWg79Waw77CgbQa+Xh5oS/T0nElj2nkbI7B9kFZXj87hiE+bnbu2tERETXzeY58Pfddx+USiVWr14NxRVBoq6uDtOmTUNtbS02btzYoR3tLjgH3jF0x/ESRRFiTbVUhaeurMTK3f2mn8WGVYmvJNNoWl1I68o7/dca9rvjeHW0rLwSfPT1UZRVGDFpdARuHRp83Q8w94bx6kgcL9twvGzD8bIdx8w2PWIOfHZ2Nv72t7+1CO8AoFAoMGHCBPz73/+29W2JejxBECConaFSOwPXsAaBqabGssZ+acvpPNWnT6O+rBSm6mqr7yFzcW3zrr7cXQuF1h0mbc+vmx4ZrMMrM4fh060ZWLvrJI6duYSZd/SHu4vK3l0jIiKyic0BXqlUorKystXXKyoqoFQq29UpIgJkTk5Q6X0A/dUXRjMZjS0W0qovtZzOU5OXi8qyUpiqqlrsnwNA5uzcyjSelsFf5uSYgV/jrMRT9w3EroNn8cWuk3hp+X48dmcM+odevcQpERFRd2FzgB84cCC++OILTJo0Cd7elgulFBcX48svv0R8fHyHdZCIrk6mUkHmrYfSW3/VbU21RtSXXbaYr+9UV42yc4amOftnz6IuIwOmygqr7yE4qaWHcK8M+VcGf5la3dGn2y6CIGDc4CD0C9Lio6+PYuGaQ7hjVCjuvjEcclYOIiIiB2BzgH/yyScxffp0TJgwAX/+85+lVVhPnjyJjRs3oqKiAgsXLuzwjhJRx5ApVZB5eUHp5SW1tTa/T6yrs7yrL03faZrOYzx/DnVZmTCVl1s9nqBSNU3ZsbJybvPpPDK1ussW1grxdcM/Hx6Kz3dmYcueMzh+pgSP3TUA3lrnLjk+ERHR9bI5wA8dOhRLlizBa6+9hk8//dTitYCAALz11lsYMmRIh3WQiOxHUCig9PSE0tPzqtuKdXWoL7/cNEe/tKxF4K8tKkL1yROoLy+3voquUmm91KaV6TwyZ5d2h30nlRwzJvTHgDBPfLbtOF5e/jtmTIjG4KirT1siIiKyl+uqRzd27FiMHj0aR44cQX5+PgDzQk4xMTH48ssvMWHCBGzdurVDO0pE3ZugUECh84BCd/X55GJ9PerLL7daY7++tBS1xcWozjllXkXXWthXKFp5OLfldB6Zq2ubYX/4AF+EB7jjo5SjeH/TEYxODMRfxvaFSsn6/ERE1P1cd0FpmUyGuLg4xMXFWbRfunQJOTk57e4YEfVcglwOhVYHhVbXYhXdK4kmE+rLy1utsV9XVoa6S5dQfeYM6i+XASZTyzeRy82h3q1Z4L9iOo/WXYv/vacfvk4twrf783AivwSP3xWDQH3rZbyIiIjsgSvCEFG3JshkUDQ8LHstYd9UUWH9rn7zVXTP5qGurAy4YjVpAIiXyxHn7IoL+QocTVfDEOSLwDA/i+k8LqH+qKtXQK7RQOCDr0RE1MUY4ImoxxBkMsjd3CB3cwMCA9vcVhRFi7B/5V19p0uXUJhXhNrsLBSfSIfM1BT2z0oHFMzHa23l3ObTeTRuEOSckkNERO3HAE9EvZIgCJBrNJBrNEBAgNVtQkQR3/2Wi6U/ZcPXRcD0mwMR4CzCFUZczC9sEfyNRYWoLyuDaDRaO6D5eFZLbl4R+DVu17yKLhER9T78G4KIqBUyQcCfRoQiMkSHj1KO4s1tZ3HvzeFInjgYYrH1spmiKMJUXd0s3Le8u19fVobaU9moKyuDWFNj/dgaTRslN92hcNdJdfgZ9omIOl7Zvj24sHEDsi5dhMLDE973/RnuI0bZu1sArjHAX1kusi0HDx687s4QEXVHEQFavDxjGFZ+dxwbfjqF7ILLePj2SGg1LWflC4IAubMz5M7OgK/fVd/bVF3dYs7+lYG/+vRp1JeVwlRdbfU9ZC6u1u/qN6ux3ziHX8aVsomIrqps3x4UrlwhfaNad7EYhStXAEC3CPHXFODfeustm960qxZiISLqKi5qBWbfFYMBYZ74fOcJ/HP5fjw6cQAG9vG6+s5tkKnVUKnVgM/Va8+bamrauKtvbqvJy0VlWSlMVVXWj+fs3ObKuc2Dv8zpao8Nt607370ios4himJT6V+Tyfw7RMDU2C5CbPy54R/zPiZAhMXPEBv2b/jZcvsr92/42dSwPwCYRIjSzyapX6L0c2t9EVG09vMW0yFFoxEXNm7oFp9j1xTgV65c2dn9ICLq9gRBwM3xARga6483VuzHe1+mIWlYCO67pQ8U8s6vRiNzcoJMr4dSr7/qtqZaI+rLLqOutLTV6TzGs2dRl5EBU2WF9eOp1ZA3PpR7Rci/MvjL1GqLfbv73averK3wY+13y4AlNuSeK8KUqSGkNQaohp8b25tvYzVg2dAfy4AnAhonlJVVNesTmt7TJEJsoy+WAa+pX2IrYc+8fVv9MzVs2jyQNvWrKZBeETib9bGpv4196ZhA2vS7CbkyGerr6q328ar/PSz6a2q6phpL+FpZt6MnqbtYbO8uALjGAD9s2LDO7gcRkcMI8XPHP5KH4ItdJ7Ftfy4y8y5h9l0x8PFwsXfXJDKlCjIvLyi9rv4NgVhXJ83NrysraSi5aRn4jefPoS4rE6Zy63P/BScnKdDL3d1Reeyo1btXRWs+h1hvgvXA0hCAgKYwdS0BqNXfr+2OnrS9tYBnLWBdEaAsA+01BKCr9DdPBtTV1V/9fK+pv413QGER+HqSc11xkMaZBTJZ0ywDmayhXYAgE6SfIRMatjH/DKHh92b/mH9v2F8ABOnntra38ntDH2QKJSA09FOQmfvTrI9Cs74AAtTOStQY66+hv7KG/jXvrw39a3G+aPi98WeZ9eNf0V9pfKX3a3ovCLKGzWzpXyv7N4ybIMiQv2gh6ktLW1wKCs/2fevaUQRR7IF/mjtRcXE5TKauHzK93g0Gw+UuP66j4njZhuNlm+bjlZpZhE+3HodJFJGcFIURA64+792RiXV1qC+/3DR9p9Tyrr4U+M/md23HrgxYbfzlfbW/0C3CxbXuL2sIIILQtDZAY8BCQ0BrDFWNAauV91OrVagx1l17ABIEiwDTan9b6WPTz5YBRtpe1nYgujIMWQbShvNta/9rHa9WwqanpwaXSipbCZzNj2+9L9ccQHsQfuZf3ZXfIgKAoFLBN3l6l3yLKJMJ8PJqfSFBli4gImqHwVE+CPVzw9LNx7D062M4dvoSpo2PhJOqZ9Z8FxQKKHQeUOg82tzu1P8+Z/WrZrlOh+C5L6KtO3pthePGUCk0C3s9LWAxXNnGRe+GCjXHizpWY0i/sHED6rrhczwM8ERE7eStdcbcqYlI+fU0vtlzGtlnSzH7rhiE+LrZu2t2433fn63evdLf/wBU+qs/sEtEZG/uI0bBfcSobvk/1VwDnIioA8hlMtx3cx88PyURVTV1+NfKVHyfmo/eOkvRfcQo+CZPN88XFQQoPL267KtnIqKejnfgiYg6UP9QD7w8cxiWf5OB1TuycOz0RcyY0B8a595Xf707370iInJkvANPRNTB3F1UeOb+OPxlXD+kZxfjpeX7kZVXYu9uERFRD8EAT0TUCQRBwG1Dg/F/yUOgUsjw1ucHkfJrjl2qWBERUc/CAE9E1IlC/dzwz+lDMWKAH1J+zcE7aw7hYlm1vbtFREQOjAGeiKiTOTspMOvOAXh0Yn+cPn8ZLy3fj0MnDPbuFhEROSgGeCKiLjIq1h8vzxgKb60zlmz4A6t3ZKG2rt7e3SIiIgfDAE9E1IV8PV3w4kODceuQYHyfmo/XV6biXHGFvbtFREQOhAGeiKiLKRUyTBnfD8/cH4eLl2vw6ooD+DX9XK+tGU9ERLZhgCcispP4vt54ZeYwhPu7YfnWDHy8+Riqaurs3S0iIurmGOCJiOzIw80Jz/8lEffe3Af7M4rwyqe/I+dcmb27RURE3RgDPBGRnclkAu4cFYa50xJRbzLhjVWp2PZbLkycUkNERFYwwBMRdRP9gnR4eeYwJPT1xpc/nMSidWkoqzDau1tERNTNMMATEXUjrmolnrw3Fg/dHoXjZ0rw0vL9OHb6or27RURE3QgDPBFRNyMIAsYkBuKfDw+Bi1qBd9cexoafslFXb7J314iIqBtggCci6qaCfDT45/ShuCk+AN/sPYO3Vh/EhZIqe3eLiIjsjAGeiKgbc1LKMf1P0Xj87hgUFFfgpU9/x+/Hi+zdLSIisiOFPQ9uNBqxePFipKSkoKysDNHR0ZgzZw5GjhzZ5n7p6enYuHEj0tPTkZWVhdraWmRmZrbYLjs7Gxs2bMDu3buRm5sLV1dXxMTE4K9//StiYmI667SIiDrcsP6+CPd3x0dfH8X/99URHEsIwF/G9YOTUm7vrhERURez6x34efPm4bPPPsNdd92F+fPnQyaTYdasWTh06FCb+/30009Yt24dACA4OLjV7davX49169YhNjYW8+bNw/Tp03Hq1Ck88MAD2LdvX4eeCxFRZ9PrnDFv2iBMGBGKnw8X4F+fHUC+odze3SIioi4miHZauzs9PR2TJk3CCy+8gOnTpwMAampqMHHiRPj4+GD16tWt7nvhwgVoNBqo1Wq8/vrrWLlypdU78EeOHEF4eDhcXV2ltkuXLmHChAno27cvVq1aZXO/i4vLYTJ1/ZDp9W4wGC53+XEdFcfLNhwv23SH8TqacxEfbzGv3DplXD/ckhAAQRDs2qfWdIfxciQcL9twvGzHMbONPcZLJhPg5aVp/fUu7IuFbdu2QalUYtKkSVKbk5MT7r//fqSmpqKoqPU5nt7e3lCr1Vc9RmxsrEV4BwAPDw8MGTIE2dnZ1995IiI7iwn3xCszhyEqWIeV32Xig6+OoKK61t7dIiKiLmC3AJ+RkdHi7jgAxMXFQRRFZGRkdNqxDQYDPDw8Ou39iYi6gtZVhWcfiMcDY/ri8IkLeHn5fpzML7V3t4iIqJPZLcAbDAb4+Pi0aNfr9QDQ5h349jhw4AAOHz6MP/3pT53y/kREXUkmCEgaHoIXHhwMmUzAm6sPYsue03aZ6kdERF3DblVoqquroVQqW7Q7OTkBMM+H72jFxcV47rnnEBISgpkzZ17Xe7Q1H6mz6fVudju2I+J42YbjZZvuNl56vRsGRvng/fVp2PjzKZwsKMPfpg6Cl9bZ3l0D0P3Gq7vjeNmG42U7jpltutt42S3Aq9Vq1Na2nK/ZGNwbg3xHqaysxOzZs1FVVYVly5bBxcXlut6HD7E6Bo6XbThetunO4/XwbZGI8HfD6h1ZeOqdH/DoxP6Ii/C2a5+683h1Rxwv23C8bMcxsw0fYm1Gr9dbnSZjMBgAwOr0mutlNBrx9NNPIysrCx988AH69u3bYe9NRNSdCIKAm+IC8NL0odBpnLBoXTrWfn8CdfUme3eNiIg6iN0CfHR0NHJyclBRUWHRnpaWJr3eEUwmE+bOnYu9e/fi3//+N4YMGdIh70tE1J35e7niHw8PxrhBQdj+ex5eX5WKwkuV9u4WERF1ALsF+KSkJNTW1koLMgHmO+UbN27EoEGD4OvrCwAoKChoV8nH1157DVu3bsVLL72E8ePHt7vfRESOQqmQY9ptkXj6voG4UFKFlz/9HXuPnLd3t4iIqJ3sNgc+Pj4eSUlJWLhwIQwGA0JCQrBp0yYUFBRgwYIF0nZz587F/v37LRZqOnv2LFJSUgAAf/zxBwDggw8+AGC+cz927FgAwIoVK/D5558jMTERarVa2qfR3Xff3annSETUHSRG6vGKnxuWfn0UH285hmOnL2LabZFQq+z2VwAREbWDXT+93377bSxatAgpKSkoLS1FVFQUli5disGDB7e5X35+PhYvXmzR1vj7vffeKwX448ePAwAOHTqEQ4cOtXgfBngi6i083dX4+9REbN59Gpv3nMbJgjI8flcMQv26V2UFIiK6OkEURRYLtgGr0DgGjpdtOF62cfTxysy9hKWbj+FypRGTxvTF+MFBEASh047n6OPV1ThetuF42Y5jZhtWoSEiIruLCvHAKzOHITbcC2t2nsCSDX/gcqXR3t0iIqJrxABPRNQLaZyVePrPAzF1fD8cySnGy5/+jszcS/buFhERXQMGeCKiXkoQBIwfEoz5Dw2BSinH22sO4atfTqHexJrxRETdGQM8EVEvF+rnhpemD8GoWD98vfs03vn8EC6WVdu7W0RE1AoGeCIiglqlwCN3DMCsOwfgTFE5Xlq+H4eyDPbuFhERWcEAT0REkpExfnh5xlB465yxZOMf+H/bM1FbV2/vbhERUTMM8EREZMHXwwXzHxqM24cFY9fBs3jts1QUXKiwd7eIiKgBAzwREbWgkMsweWw/PDspHqUVNXj1s9/xS1oBuHQIEZH9McATEVGr4iK88MrMYYgI0OLTb4/jo6+PorK6zt7dIiLq1RjgiYioTTqNE56bnIA/39IHB44b8PKn+3GqoMze3SIi6rUY4ImI6KpkMgF3jAzDvGmDIIrAgv+Xim9/OwMTp9QQEXU5BngiIrpmfYO0eGXmUCT288a6H7Kx6Ms0lFYY7d0tIqJehQGeiIhs4qJW4ol7YpGcFIXMvBK8tHw/juZctHe3iIh6DQZ4IiKymSAIGJ0QiH88PARuzkq8+8VhrPvxJOrqTfbuGhFRj8cAT0RE1y1Ir8H/PTwEoxMC8O2+XLy5+iAMJVX27hYRUY/GAE9ERO3ipJQjOSkaT94Ti3PFlXj50/3Yn1Fo724REfVYCnt3gIiIeoYh0T4I83PDR5uP4sOUo/jhYD4MJdW4dLkGnu5OuO+WCIyM8bN3N4mIHB7vwBMRUYfx1jlj7tRBSOjrhcy8Uly8XAMRQHFZDT779jj2Hj1v7y4SETk8BngiIupQCrkMeUXlLdqNdSZ8viMLRZcqIbJ+PBHRdeMUGiIi6nDFZTVW2yuq6zDvo33QaVSIDNYhKliHyGAd/L1dIROELu4lEZFjYoAnIqIO5+XuZDXEazUq3HVDOLLySpCZewn7M4oAABpnJfoFaREV4oGoYB2CfTSQyRjoiYisYYAnIqIOd98tEfjs2+Mw1jXVhVcpZHhgTF+MjPHDmMRAiKIIQ2k1snJLkJVn/ufQiQsAAGcnOfoG6hAZrEVUsAfC/N2gkHPWJxERwABPRESdoLHazMafsnGxzHoVGkEQ4KNzho/OGTfG+QMALpZVIyu/BFl5pcjKK8GGn4oBmMN/RKAWkQ1TbvoEuMNJKe/6EyMi6gYY4ImIqFOMjPHDyBg/6PVuMBguX9M+nu5qjBjghxEDzEG/rNKIEw1hPjPvEr7+NQciALlMQLi/uxTo+wVp4ezEv9KIqHfgpx0REXVb7i4qDI7SY3CUHgBQWV2Hk2dLkNkw5ea7/bnYuu8MBAEI8XWTHortF6SFm4vKzr0nIuocDPBEROQwXNQKxEV4Iy7CGwBQY6zHqYJSKdD/cOgstv+eBwAI9HaV7tBHBuvg4eZkz64TEXUYBngiInJYTio5+od5on+YJwCgts6E0+fLGqbclGDP0fP44dBZAICPh7NF6UpvrRoCS1cSkQNigCcioh5DqZChX5AO/YJ0uGMkUG8yIa+oHJkNlW4OZRnwa/o5AICHm5M5zIeYQ72fpwsDPRE5BAZ4IiLqseQyGcL83BHm547bh4XAJIoouFAhla3MOHMJ+44VAgDcXJTSdJuoYB2C9KxFT0TdEwM8ERH1GjJBQJBegyC9BmMHBUEURRSVVEl36LPySpCaaQAAODspzItLNdylD/VlLXoi6h4Y4ImIqNcSBAG+Hi7w9XDBzfEBAIDi0sZa9OZ/0rMbatErZejbUIs+qqEWvVLBWvRE1PUY4ImIiJrx0qoxUusnLTpVWmHEibym0pUpv5hr0SvkAvr4uyMyxDztJiKAteiJqGvwk4aIiKgNWlcVhkT7YEi0DwCgoroWJ/JLpTv0W/fmYsueM5AJAkL9NM0Wl9JB46y0c++JqCdigCciIrKBq1qJhL7eSOhrrkVfbaxD9tky6Q7996ln8d3+PAgAAvUaaQ59ZJAWWg1r0RNR+zHAExERtYNapUBMuCdiwhtr0dcj59xlKdD/+sc5fH8wHwDg6+mCqODGefQe8NKq7dl1InJQDPBEREQdSKmQS9NoAKCu3oTcwnJpys2B4wb8nGauRe/lrjaH+YZ59L4ezvbsOhE5CAZ4IiKiTqSQy9AnwB19AtyRNNxci/6soQKZuZeQlVeCoznF2Hv0PADA3VWFgX29EeZjnnoToHeFjItLEdEVGOCJiIi6kEwQEOyjQbCPBuOHBEMURZy/WCndoc88cwm70woAAK5qBfoF6aS79CG+GshlrEVP1NsxwBMREdmRIAjw93KFv5crbkkIhF7vhoyTReYw37DA1OGTFwAATiq5RS36cH93KBUM9ES9DQM8ERFRN+OtdYa31hmjYv0BACXlNdId+qy8Emz6+RQA8/SciAB385z7EB36BmjhpOLiUkQ9nV0DvNFoxOLFi5GSkoKysjJER0djzpw5GDlyZJv7paenY+PGjUhPT0dWVhZqa2uRmZlpdVuTyYRly5ZhzZo1MBgMCAsLwxNPPIEJEyZ0xikRERF1OJ3GCcP6+2JYf18AQHlVrcXiUlv2noa4B5DLBIT6uZlLVwbr0C9ICxc1a9ET9TR2DfDz5s3D9u3bkZycjNDQUGzatAmzZs3CqlWrkJiY2Op+P/30E9atW4eoqCgEBwfj1KlTrW773nvvYenSpZg8eTJiY2Px/fffY86cOZDJZEhKSuqM0yIiIupUGmclEiP1SIzUAwCqauqQfbZUCvQ7DuTh299yIQAI9mlaXCoyWAd3V5V9O09E7SaIoija48Dp6emYNGkSXnjhBUyfPh0AUFNTg4kTJ8LHxwerV69udd8LFy5Ao9FArVbj9ddfx8qVK63egS8sLMS4ceMwZcoUzJ8/HwAgiiIefPBBnDt3Djt37oTMxoeBiovLYTJ1/ZDp9W4wGC53+XEdFcfLNhwv23C8bMPxsk1HjJexth6nCsrM8+jzSpB9thTGOhMAwN/LRbpDHxmsg6e7Y9ei5/VlO46ZbewxXjKZAC8vTauv2+0O/LZt26BUKjFp0iSpzcnJCffffz/ee+89FBUVwcfHx+q+3t7e13SMnTt3ora2FlOnTpXaBEHAlClT8NxzzyE9PR0JCQntOxEiIqJuRqWUIzrUA9GhHgDMtejPnL8sBfrfMgrx42FzpRtvrbop0Ifo4KNzhsDSlUTdmt0CfEZGBsLDw+Hq6mrRHhcXB1EUkZGR0WqAt+UYGo0G4eHhLY4BAMeOHWOAJyKiHk8hlyEiUIuIQC3+NCIUJpOIvKKmxaXSsoux+4i5Fr1Oo5Kq3EQG6+DvzVr0RN2N3QK8wWCAr69vi3a93jyfr6ioqEOOYe1ufUceg4iIyNHIGh52DfVzw61DzbXozxVXSnfos/JKsD/D/HekxlmJfkFac6AP0SHExw0yGQM9kT3ZLcBXV1dDqWz5ZLyTkxMA83z4jjiGStXyYZ32HKOt+UidTa93s9uxHRHHyzYcL9twvGzD8bKNPcbLx8cd8f39AJifFyu8WIkj2cU4esr8z6ET5lr0zk4KDAj3REwfL8T28UbfYJ3da9Hz+rIdx8w23W287Bbg1Wo1amtrW7Q3hurGkN3eYxiNxg49Bh9idQwcL9twvGzD8bINx8s23WW85ADiwz0QH+4BjOuLS5ebatFn5pUg9bj5Dr1KIUOfhlr0USEe6BPgDidl19Wi7y7j5Ug4ZrbhQ6zN6PV6q1NYDAYDALR7/nvjMQ4cONCpxyAiIuoNPNycMHyAL4YPME9/Las04kReqRTqN+85ja93n4ZcJiDc312qctMvSAtnJ64bSdSR7PYnKjo6GqtWrUJFRYXFg6xpaWnS6+3Vv39/rFu3Djk5ORYPsjYeo3///u0+BhERUW/k7qLC4Cg9BkeZnyurrK7DybNNc+i/25+LrfvOQBCAEB+3hjv+pKXTAAAgAElEQVT05kDv5sJa9ETtYbcAn5SUhOXLl2PdunVSHXij0YiNGzdi0KBB0gOuBQUFqKqqQkREhM3HGDduHBYsWIDPP//cog782rVrERAQgPj4+A47HyIiot7MRa1AXIQ34iLMxSNqautxqtniUj8ePosdB/IAAIHerhaLS3m4tX/aLFFvYrcAHx8fj6SkJCxcuBAGgwEhISHYtGkTCgoKsGDBAmm7uXPnYv/+/RYLNZ09exYpKSkAgD/++AMA8MEHHwAw37kfO3YsAMDPzw/JyclYvnw5ampqMHDgQOzcuRMHDhzAe++9Z/MiTkRERHRtnJRy9A/zRP8wTwBAbZ0Jp883LS619+h5/HDoLADAR+eMyJCm0pXeWjVr0RO1wa6T0t5++20sWrQIKSkpKC0tRVRUFJYuXYrBgwe3uV9+fj4WL15s0db4+7333isFeAB4/vnnodVq8cUXX2Djxo0IDw/Hu+++iwkTJnT8CREREZFVSoUM/YJ06Bekwx0jgXqTyVyLPtcc6A9lGfBr+jkA5vn2zVeL9fdyYaAnakYQRbHrS6o4MFahcQwcL9twvGzD8bINx8s2vXW8TKKIggsVTZVucktQWmGuJOfmopTCfFSwDkF6jVSLvreOV3twzGzDKjREREREVsgEAUF6DYL0GowdFARRFFFUUoWs3GalKzPNVeScnRTS4lLD4gKgdZJDIee0WOo9GOCJiIio2xEEAb4eLvD1cMFN8QEAgOLSamTll0h36dOzi7Hux2yolDL0DdRKd+jD/d2h6sJa9ERdjQGeiIiIHIKXVo2RWj+MjDGvGFtaYURhWQ0OHDmHzLwSpPySAxGAQt5Uiz4qWIeIQNaip56FVzMRERE5JK2rCn3DvBDpb17mvqK6FifymxaX+nZfLr7ZewYyQUCon6bZ4lI6aJyVdu490fVjgCciIqIewVWtREJfbyT0NdeirzbWIbugDJkN8+i/Tz2L7/aba9EH6V0bFpfyQGSQFloNa9GT42CAJyIioh5JrVIgJswTMVIt+nrknLssLS61+4/z2HXQXIve19MFUcFa6S69t9bZnl0nahMDPBEREfUKSoVcCugAUFdvrkXfeIf+wHEDfk4z16L3cndqukMfrIOvhzNr0VO3wQBPREREvZJCLkO4vzvC/d2RNDwEJlHEWUOFVLbyaM5F7D1aCABwd1VJD8VGBusQqHeFjIGe7IQBnoiIiAjmWvTBPhoE+2gwbrC5Fv35i5VNi0vlleDA8SIAgKtagX5BDYtLhegQ4quBXMZa9NQ1GOCJiIiIrBAEAf5ervD3csUtCYEAgAulVc0CfSkOn7wAAHBSya+oRe8GpYK16KlzMMATERERXSNvrTO8tc4YFesPACgpr5ECfVZeCTb9fAqAeXpORIC5Fn1kiA59A7RwUjHQU8dggCciIiK6TjqNE4b198Ww/r4AgPKqWpxoWC02M7cEW/aehrgHkMsEhPq5SQ/RRgZp4aJmLXq6PgzwRERERB1E46xEYj89EvvpAQBVNXXIPlsqla7ceSAP237LhQAg2KdpcanIYB3cXVX27Tw5DAZ4IiIiok7i7KRAbB8vxPbxAgAYa+uRc64MmQ136H9OK8DO1HwAgL+XixTmo4J18HRX27Pr1I0xwBMRERF1EZVSjqgQD0SFeAA3mGvRnzl/Wapysz+jCD8dLgAAeGvVUtnKyBAdfHSsRU9mDPBEREREdqKQyxARqEVEoBZ/GhEKk0lEvqHcPOUmtwRp2cXYfeQ8AECrUUmBPipYB39v1qLvrRjgiYiIiLoJmUxAiK8bQnzdcOuQYIiiiHPFlrXo92eYa9FrnJXoF6Q1h/oQHYJ9WIu+t2CAJyIiIuqmBEFAgLcrArxdMToxEKIowlBajazcptKVh06Ya9GrVXL0bQj0UcEeCPN3g0LOQN8TMcATEREROQhBEOCjc4aPzhk3xplr0V+6bFmLfsNP5lr0SkVTLfqoYB36BGrhpGQt+p6AAZ6IiIjIgXm4OWH4AF8MH2CuRX+50oisvFIp0G/ecxpfi+Za9GH+bkiM8kWQlwv6BmrhomYUdET8r0ZERETUg7i5qDA4So/BUeZa9JXVdTh5trRhDv0lbPrxJOpNIgQBCPFptrhUsBZuLqxF7wgY4ImIiIh6MBe1AnERXoiLMNeid9M6Y3/aWWlxqR8Pn8WOA3kAgEBvV4vFpTzcnOzZdWoFAzwRERFRL6JWKdA/zBP9wzwBALV15lr0mXmXkJlXgr1Hz+OHQ2cBAD4656ZAH6KDXqtmLfpugAGeiIiIqBdTKmToG6RF3yAt7hgJ1JtMyCsqR1auuWzloRMG/PrHOQDm+fZRze7Q+3u5MNDbAQM8EREREUnkMhnC/NwR5ueO24aFwCSKOHehQqpDn5F7CfuOFQIA3FyUiAxqWFwqRIcgvQYyGQN9Z2OAJyIiIqJWyQQBgXoNAvUajBkUBFEUUVRSJdWiz8wrQWqWAQDg7KRoWlwqWIdQP9ai7wwM8ERERER0zQRBgK+HC3w9XHBTfAAA4GJZtfRQbFZeCdKziwEAKqUMEQENi0uF6BDu7w4Va9G3GwM8EREREbWLp7saI2P8MDLGDwBQVmG0WFwq5dcciAAUcgHh/k2LS0UEauHsxDhqK44YEREREXUod1cVhkT7YEi0DwCgsroWWflNi0t9uy8X3+w9A5kgIMRXg6gQ85SbfkE6aJyVdu5998cAT0RERESdykWtREJfbyT09QYAVBvrkF1QJs2j/z71LL7bb65FH6RvqkUfFayDVsNa9FdigCciIiKiLqVWKRAT5okYqRZ9PXLOXZbm0e/+4zx2HTTXovf1dEFUsFYK9d5aZ3t2vVtggCciIiIiu1Iq5FJAB8y16HMLy5HZcIc+NdOAn9PMtei93J0sVov18+x9tegZ4ImIiIioW5HLZAj3d0e4vzuShptr0Z81NNWiP3r6EvYeNdeid3dVSdNtIoN1CNS7QtbDAz0DPBERERF1azJBQLCPBsE+GowbbK5FX3ipyhzocy8hM68EB44XAQBc1Qr0a7a4VIivBnJZz6pFzwBPRERERA5FEAT4ebrAz9MFNzfUor9QWiVVucnMK8XhkxcAAE5KOfoGaaW79OH+blAqHLsWPQM8ERERETk8b60zvLXOGBXrDwAoLa9BVn4pMnMvISuvBJt+PgUAUMhl6BPQVIu+b6AWTqqWgX7v0fPY+FM2LpbVwNPdCffdEiHVubc3BngiIiIi6nG0GicMjfbB0IZa9OVVtTiR37S41Na9Z7Blz2nIZQJC/dykh2L7BWmRnl2Mz749DmOdCQBQXFaDz749DgDdIsQzwBMRERFRj6dxViKxnx6J/fQAgKqaOmSfLZVKV+48kIdtv+VCACCTCag3iRb7G+tM2PhTNgM8EREREZE9ODspENvHC7F9vAAAxtp65JwrQ2ZeCb76JcfqPsVlNV3ZxVb1rEdyiYiIiIiug0opR1SIB+66IRxe7tZXf22tvavZNcAbjUa88847uPHGGxEXF4cHHngAe/fuvaZ9CwsL8cwzz2DIkCEYNGgQnnzySeTl5bXY7vLly3jrrbdw2223IS4uDmPHjsU///lPFBYWdvTpEBEREVEPcN8tEVApLGOySiHDfbdE2KlHluw6hWbevHnYvn07kpOTERoaik2bNmHWrFlYtWoVEhMTW92voqICycnJqKiowOOPPw6FQoEVK1YgOTkZX331FbRaLQDAZDLhkUcewYkTJzBlyhSEh4cjJycHa9aswb59+7BlyxaoVKquOl0iIiIicgCN89xZheYK6enp+Oabb/DCCy9g+vTpAIB77rkHEydOxMKFC7F69epW9/38889x5swZbNy4EQMGDAAA3HTTTbjzzjuxYsUKPPPMMwCAP/74A2lpafjnP/+JadOmSfsHBATgtddew8GDBzFixIjOO0kiIiIickgjY/wwMsYPer0bDIbL9u6OBbtNodm2bRuUSiUmTZoktTk5OeH+++9HamoqioqKWt33u+++Q0JCghTeASAiIgIjR47Et99+K7WVl5cDALy8vCz29/b2BgCo1eoOORciIiIioq5itwCfkZGB8PBwuLq6WrTHxcVBFEVkZGRY3c9kMiEzMxOxsbEtXhs4cCBOnz6NqqoqAEBMTAxcXFywePFi7N27F4WFhdi7dy8WL16M4cOHIz4+vuNPjIiIiIioE9ktwBsMBvj4+LRo1+vNtTlbuwNfUlICo9EobXflvqIowmAwAAB0Oh3ee+89XL58GdOnT8fNN9+M6dOnIzQ0FEuXLoUgCB14RkREREREnc9uc+Crq6uhVCpbtDs5mcvz1NRYr7PZ2G7t4dPGfaurq6U2T09PxMbGIjExERERETh+/Dg++eQTvPjii/j3v/9tc7+9vDQ279NR9Ho3ux3bEXG8bMPxsg3HyzYcL9twvGzD8bIdx8w23W287Bbg1Wo1amtrW7Q3BvTGMH6lxnaj0djqvo1z2/Py8pCcnIyFCxdi/PjxAIDx48cjMDAQ8+bNw5///GfccMMNNvW7uLgcpitW5uoK3fEBiu6M42UbjpdtOF624XjZhuNlG46X7ThmtrHHeMlkQps3je02hUav11udJtM4/cXa9BrAPC1GpVJJ2125ryAI0vSajRs3wmg04pZbbrHYbuzYsQCAgwcPtusciIiIiIi6mt0CfHR0NHJyclBRUWHRnpaWJr1ujUwmQ2RkJI4cOdLitfT0dISGhsLZ2RkAUFxcDFEUIYqWd8zr6uos/k1ERERE5CjsFuCTkpJQW1uLdevWSW1GoxEbN27EoEGD4OvrCwAoKChAdna2xb633347Dh8+jGPHjkltp06dwr59+5CUlCS1hYWFwWQyWZSWBIAtW7YAgEUZSiIiIiIiR2C3OfDx8fFISkrCwoULYTAYEBISgk2bNqGgoAALFiyQtps7dy7279+PzMxMqW3q1KlYt24dHnvsMcyYMQNyuRwrVqyAXq+XFoUCgHvvvRfLly/H/PnzceTIEfTt2xdHjx7F+vXrERUVJU2lISIiIiJyFHYL8ADw9ttvY9GiRUhJSUFpaSmioqKwdOlSDB48uM39NBoNVq1ahTfeeAMffPABTCYThg8fjvnz58PDw0PazsPDAxs2bMDixYuxa9curFmzBjqdDvfffz/mzJljtQrO1chk9is9ac9jOyKOl204XrbheNmG42UbjpdtOF6245jZpqvH62rHE8QrJ4gTEREREVG3Zbc58EREREREZDsGeCIiIiIiB8IAT0RERETkQBjgiYiIiIgcCAM8EREREZEDYYAnIiIiInIgDPBERERERA6EAZ6IiIiIyIEwwBMRERERORAGeCIiIiIiB6Kwdwd6C6PRiMWLFyMlJQVlZWWIjo7GnDlzMHLkyKvuW1hYiDfeeAO7d++GyWTCiBEj8MILLyA4OLjFtuvWrcPy5cuRn5+PgIAAJCcnY9q0aZ1xSp3qesdr+/bt2Lp1K9LT01FcXAx/f3+MGTMGTz75JNzc3Cy2jYqKsvoeL7/8MqZMmdJh59IVrne8lixZgv/+978t2r29vbF79+4W7b39+ho7dizOnj1r9bXQ0FBs375d+r0nXV9FRUVYuXIl0tLScOTIEVRWVmLlypUYPnz4Ne2fnZ2NN954AwcPHoRSqcSYMWMwd+5ceHp6WmxnMpmwbNkyrFmzBgaDAWFhYXjiiScwYcKEzjitTnO942UymbBp0ybs2LEDGRkZKC0tRVBQECZOnIiZM2dCpVJJ2+bn52PcuHFW3+fjjz/GzTff3KHn1Jnac33NmzcPmzZtatEeHx+PL7/80qKtt19fQOufSwAwatQofPrppwB6zvWVnp6OTZs24bfffkNBQQF0Oh0SExPx7LPPIjQ09Kr7d+f8xQDfRebNm4ft27cjOTkZoaGh2LRpE2bNmoVVq1YhMTGx1f0qKiqQnJyMiooKPP7441AoFFixYgWSk5Px1VdfQavVStuuXbsWL730EpKSkjBjxgwcOHAAr776KmpqajBz5syuOM0Oc73j9Y9//AM+Pj64++67ERAQgMzMTKxatQq//PILNmzYACcnJ4vtb7zxRtx1110WbfHx8Z1yTp3peser0auvvgq1Wi393vznRry+gBdffBEVFRUWbQUFBVi0aBFuuOGGFtv3lOsrJycHH3/8MUJDQxEVFYVDhw5d877nz5/HtGnT4O7ujjlz5qCyshLLly9HVlYWvvzySyiVSmnb9957D0uXLsXkyZMRGxuL77//HnPmzIFMJkNSUlJnnFqnuN7xqqqqwosvvoiEhAT85S9/gZeXFw4dOoTFixdj3759WLFiRYt97rrrLtx4440WbdHR0R1xGl2mPdcXADg7O+OVV16xaLvyfw4BXl8A8Pbbb7doO3LkCFauXGn1M8zRr69PPvkEBw8eRFJSEqKiomAwGLB69Wrcc889WL9+PSIiIlrdt9vnL5E6XVpamhgZGSl++umnUlt1dbU4fvx4cerUqW3uu3TpUjEqKko8evSo1Hby5Emxf//+4qJFi6S2qqoqcdiwYeITTzxhsf9zzz0nJiYmimVlZR1zMl2gPeO1b9++Fm2bNm0SIyMjxQ0bNli0R0ZGiv/61786pM/21J7x+s9//iNGRkaKpaWlbW7H66t177//vhgZGSmmpqZatPeU60sURfHy5cvixYsXRVEUxR07doiRkZFW/6xZ89JLL4kJCQni+fPnpbbdu3eLkZGR4rp166S28+fPizExMRZjZjKZxKlTp4pjxowR6+vrO+hsOt/1jldNTU2L60gURXHJkiUt3iMvL6/Fdeyo2nN9zZ07Vxw8ePBVt+P11boXX3xRjIqKEs+dOye19ZTrKzU1VaypqbFoy8nJEWNjY8W5c+e2uW93z1+cA98Ftm3bBqVSiUmTJkltTk5OuP/++5GamoqioqJW9/3uu++QkJCAAQMGSG0REREYOXIkvv32W6ntt99+Q0lJCaZOnWqx/7Rp01BRUYGff/65A8+oc7VnvKx9hTh+/HgA5q/xramurkZNTU07e20/7RmvRqIoory8HKIoWn2d11frtmzZgqCgIAwaNMjq645+fQGARqOBh4fHde27fft2jB07Fr6+vlLbqFGjEBYWZvEZtnPnTtTW1lpcY4IgYMqUKTh79izS09Ov/wS62PWOl0qlsnod3XrrrQBa/wyrrKyE0Wi0+XjdRXuur0b19fUoLy9v9XVeX9YZjUZs374dQ4cOhZ+fn9VtHPn6GjRokMXUMwAICwtDv379Wv3z1Ki75y8G+C6QkZGB8PBwuLq6WrTHxcVBFEVkZGRY3c9kMiEzMxOxsbEtXhs4cCBOnz6NqqoqAMCxY8cAoMW2MTExkMlk0uuO4HrHqzUXLlwAAKsfeOvXr0dCQgLi4uJw5513YseOHdffcTvpiPEaPXo0Bg8ejMGDB+OFF15ASUmJxeu8vqw7duwYsrOzMXHiRKuv94Trqz0KCwtRXFxs9TMsLi7OYqwzMjKg0WgQHh7eYjsADnWNdbS2PsMWL16MxMRExMXFYfLkyfj999+7unt2V1FRIX1+DR8+HAsWLGjxP828vqz76aefUFZW1mKqX6OeeH2JoogLFy60+T9BjpC/OAe+CxgMBou7T430ej0AtHrHr6SkBEajUdruyn1FUYTBYEBISAgMBgNUKhV0Op3Fdo1ttt5VtKfrHa/WfPzxx5DL5bjtttss2hMTEzFhwgQEBQXh3LlzWLlyJZ566im8++67rQay7qg94+Xu7o6HHnoI8fHxUCqV2LdvH7744gscO3YM69atk+5c8PqybvPmzQBg9S+/nnJ9tUfjWLb2GVZcXIz6+nrI5XIYDAZ4e3tb3a75e/VGn3zyCdzc3CzmIstkMtx444249dZb4ePjgzNnzmDZsmWYMWMGVqxYgSFDhtixx11Hr9fj0UcfRf/+/WEymfDDDz9gxYoVyM7OxieffCJtx+vLus2bN0OlUuH222+3aO/J19fXX3+NwsJCzJkzp9VtHCF/McB3gerqaosHtRo1PlDZ2tfrje1Xfv3TfN/q6uo2j9G4rSN9hX+942XN5s2bsX79esyePRshISEWr61du9bi93vvvRcTJ07EO++8gzvuuAOCIFxH77tee8br4Ycftvg9KSkJ/fr1w6uvvoqvvvoKDzzwQJvHaDxOb7y+TCYTvvnmGwwYMMDqg1A95fpqj2v9DHN1dUV1dXWb2znSNdaRPvzwQ+zZswevvvqqRSWtgIAALFu2zGLbCRMm4I477sDChQtbXH891XPPPWfx+8SJE+Hr64tly5Zh9+7d0oOZvL5aKi8vx48//ohbbrkF7u7uFq/11OsrOzsbr776KgYPHoy777671e0cIX9xCk0XUKvVqK2tbdHe+B/1ysoojRrbrc09a9y3sVqIWq1udY5aTU1Nq8fojq53vK504MABzJ8/H6NHj8Yzzzxz1e1dXFzwl7/8BefPn8epU6ds67QdddR4NZoyZQqcnZ2xd+9ei2Pw+rK0f/9+FBYW4s4777ym7R31+mqPjvgMu97ruCfYunUrFi1ahMmTJ2Py5MlX3d7X1xd33HEH0tLSpK/3e6PGqh/X8hnWm6+v7777DjU1Ndf8Gebo15fBYMDs2bOh1WqxePFiyGStR2BHyF8M8F1Ar9db/QrFYDAAAHx8fKzup9PpoFKppO2u3FcQBOnrHb1ej9ra2hZzl41GI0pKSlo9Rnd0vePV3PHjx/HEE08gKioK7733HuRy+TUd29/fHwBQWlpqQ4/tqyPGqzmZTAZfX1+LMeD11dLmzZshk8lwxx13XPOxHfH6ao/GsWztM8zLy0v6s6nX66W53ldu1/y9eovdu3fjf//3fzFmzBi89NJL17yfv78/TCYTysrKOrF33Zu3tzeUSmWLzzBeX5Y2b94MNzc3jBkz5pr3cdTr6/Lly5g1axYuX76MTz75xOrUmOYcIX8xwHeB6Oho5OTktKgfnZaWJr1ujUwmQ2RkJI4cOdLitfT0dISGhsLZ2RkA0L9/fwBose2RI0dgMpmk1x3B9Y5Xo9zcXDz66KPw9PTERx99BBcXl2s+dl5eHgDrNYS7q/aO15Vqa2tx7tw5iwd8eH1ZaqzcMGzYMKvz6VvjiNdXe/j6+sLT07PVz7Dm103//v1RXl6OnJwci+0a/7s40jXWXmlpaXjqqacwcOBAm25AAOZrTC6XW9So7m3Onz+P2tpaiz9nvL4sFRUV4bfffsNtt91mdZpIaxzx+qqpqcHjjz+O06dP46OPPkKfPn2uuo8j5C8G+C6QlJSE2tparFu3TmozGo3YuHEjBg0aJAWAgoKCFmWNbr/9dhw+fNjiKeZTp05h3759FgtPjBgxAjqdDp9//rnF/mvWrIGLi4vDrJoGtG+8DAYDZs6cCUEQsGzZslaD0sWLF1u0Xbp0CZ9//jmCgoIQFhbWcSfUydozXtbGYdmyZaipqcFNN90ktfH6stRYuaG1r5570vVli9zcXOTm5lq03Xbbbdi1axcKCwultr179+L06dMWn2Hjxo2DUqm0uMZEUcTatWsREBDgkAtgXY218crOzsZjjz2GwMBAfPjhh1YXVQOsX2NnzpzBN998gyFDhrS6nyO7crxqamqslo784IMPAMDioV9eX5a2bt0Kk8lk02eYI15f9fX1ePbZZ3H48GEsXrwYCQkJVrdzxPzFh1i7QHx8PJKSkrBw4ULpqeVNmzahoKAACxYskLabO3cu9u/fj8zMTKlt6tSpWLduHR577DHMmDEDcrkcK1asgF6vx/Tp06Xt1Go1/vrXv+LVV1/FM888gxtvvBEHDhzA119/jeeff77FAyrdWXvG69FHH0VeXh4effRRpKamIjU1VXotJCREWmVz9erV+P777zF69GgEBASgsLAQX3zxBS5evIj333+/6062A7RnvMaMGYMJEyYgMjISKpUKv/32G7777jsMHjzYolIKry9LrVVuaNSTrq9GjaGo8S+5lJQUpKamwt3dHQ8++CAASJ9Ju3btkvZ7/PHHsW3bNiQnJ+PBBx9EZWUlli1bhujoaIuHyPz8/JCcnIzly5ejpqYGAwcOxM6dO3HgwAG89957bc5X7Y6uZ7zKy8vxyCOPoKysDI888gh+/PFHi/eMioqSviF65513kJeXhxEjRsDHxwe5ubnSg4Vz587t7NPrcNczXgaDQXo4vE+fPlIVmr1792LChAkYOnSo9P68vix9/fXX8PHxsbp2CtBzrq8333wTu3btwpgxY1BSUoKUlBTpNVdXV2mdGEfMX4LY2sot1KFqamqwaNEibN68GaWlpYiKisLf/vY3jBo1StrmoYceshoYzp8/jzfeeAO7d++GyWTC8OHDMX/+fAQHB7c4zpdffonly5cjPz8f/v7+eOihh5CcnNzp59fRrne8oqKiWn3Pe++9F2+++SYA4Ndff8WyZcuQlZWF0tJSuLi4ICEhAbNnz8bgwYM778Q6yfWO1//93//h4MGDOHfuHGpraxEYGIgJEyZg9uzZVu+w9PbrCzCHrFGjRuGWW27BkiVLrL5/T7u+gNb/bAUGBkoBYezYsQBaBoYTJ07gzTffRGpqKpRKJUaPHo0XXnihxTdkJpMJH3/8Mb744gsUFRUhPDwcs2fPdsiym9czXvn5+Rg3blyr7/nUU0/h6aefBmBeQGzt2rU4efIkLl++DHd3dwwbNgxPPfUU+vXr15Gn0iWuZ7zKysrw2muvIS0tDUVFRTCZTAgLC8O9996L5OTkFlOPevv11ejUqVP405/+hBkzZmDevHlW36enXF+Nn+PWNB8rR8xfDPBERERERA7Esb4zIiIiIiLq5RjgiYiIiIgcCAM8EREREZEDYYAnIiIiInIgDPBERERERA6EAZ6IiIiIyIEwwBMRERERORAGeCIi6vYeeughaWEaIqLeTmHvDhARkX389qCAnZEAAAStSURBVNtvba4UKJfLcezYsS7sERERXQsGeCKiXm7ixIm4+eabW7TLZPySloioO2KAJyLq5QYMGIC7777b3t0gIqJrxNsrRETUpvz8fERFRWHJkiXYsmUL7rzzTgwcOBCjR4/GkiVLUFdX12Kf48eP43/+538wfPhwDBw4EBMmTMDHH3+M+vr6FtsaDAb861//wrhx4xAbG4uRI0dixowZ2L17d4ttCwsL8be//Q1Dhw5FfHw8HnnkEeTk5HTKeRMRdVe8A09E1MtVVVXh4sWLLdpVKhU0Go30+65du5CXl4dp06bB29sbu3btwn//+18UFBRgwYIF0nZ//PEHHnroISgUCmnbH374AQsXLsTx48fx7rvvStvm5+djypQpKC4uxt13343Y2FhUVVUhLS0Ne/bswQ033CBtW1lZiQcffBDx8fGYM2cO8vPzsXLlSjz55JPYsmUL5HJ5J40QEVH3wgBPRNTLLVmyBEuWLGnRPnr0aHz00UfS78ePH8f69esRExMDAHjwwQfx1FNPYePGjZg8eTISEhIAAK+//jqMRiPWrl2L6Ohoadtnn30WW7Zswf3334+RI0cCAF555RUUFRXhk08+wU033WRxfJPJZPH7pUuX8Mgjj2DWrFlSm6enJ9555x3s2bOnxf5ERD0VAzwRUS83efJkJCUltWj39PS0+H3UqFFSeAcAQRDw6KOPYufOndixYwcSEhJQXFyMQ4cO4dZbb5XCe+O2TzzxBLZt24YdO3Zg5MiRKCkpwS+//IKbbrrJavi+8iFamUzWomrOiBEjAABnzpxhgCeiXoMBnoiolwsNDcWoUaOuul1ERESLtr59+wIA8vLyAJinxDRvb65Pnz6QyWTStrm5uRBFEQMGDLimfvr4+MDJycmiTafTAQBKSkqu6T2IiHoCPsRKREQOoa057qIodmFPiIjsiwGeiIiuSXZ2dou2kydPAgCCg4MBAEFBQRbtzZ06dQomk0naNiQkBIIgICMjo7O6TETUIzHAExHRNdmzZw+OHj0q/S6KIj755BMAwPjx4wEAXl5eSExMxA8//ICsrCyLbZcuXQoAuPXWWwGYp7/cfPPN+Pnnn7Fnz54Wx+NddSIi6zgHnoiolzt27BhSUlKsvtYYzAEgOjoaDz/8MKZNmwa9Xo/vv/8ee/bswd13343ExERpu/nz5+Ohhx7CtGnTMHXqVOj1evzwww/49ddfMXHiRKkCDQD84x//wLFjxzBr1izcc889iImJQU1NDdLS0hAYGIi///3vnXfiREQOigGeiKiX27JlC7Zs2WL1te3bt0tzz8eOHYvw8HB89NFHyMnJgZeXF5588kk8+eSTFvsMHDgQa9euxX/+8x+sWbMGlZWVCA4OxvPPP4+ZM2dabBscHIwNGzbg/fffx88//4yUlBS4u7sjOjoakydP7pwTJiJycILI7yiJiKgN+fn5GDduHJ566ik8/fTT9u4OEVGvxznwREREREQOhAGeiIiIiMiBMMATERERETkQzoEnIiIiInIgvANPRERERORAGOCJiIiIiBwIAzwRERERkQNhgCciIiIiciAM8EREREREDoQBnoiIiIjIgfz/dggp0NwPWAMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "\n",
    "# Use plot styling from seaborn.\n",
    "sns.set(style='darkgrid')\n",
    "\n",
    "# Increase the plot size and font size.\n",
    "sns.set(font_scale=1.5)\n",
    "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
    "\n",
    "# Plot the learning curve.\n",
    "plt.plot(loss_values, 'b-o', label=\"training loss\")\n",
    "plt.plot(validation_loss_values, 'r-o', label=\"validation loss\")\n",
    "\n",
    "# Label the plot.\n",
    "plt.title(\"Learning curve\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "veYNOx0PxauJ"
   },
   "source": [
    "## Apply the model to a new sentence\n",
    "\n",
    "Finally we want our model to identify named entities in new text. I just took this sentence from the recent New York Times frontpage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 950,
     "status": "ok",
     "timestamp": 1595576988949,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "TCjVCxt5xYqu"
   },
   "outputs": [],
   "source": [
    "test_sentence = \"\"\"\n",
    "Mr. Trump’s tweets began just moments after a Fox News report by Mike Tobin, a \n",
    "reporter for the network, about protests in Minnesota and elsewhere. \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wv8iM0yexjow"
   },
   "source": [
    "We first tokenize the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 927,
     "status": "ok",
     "timestamp": 1595576991024,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "0DA5FwsgxjJv"
   },
   "outputs": [],
   "source": [
    "tokenized_sentence = tokenizer.encode(test_sentence)\n",
    "input_ids = torch.tensor([tokenized_sentence]).cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9C1LT4xXxnL5"
   },
   "source": [
    "Then we can run the sentence through the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 882,
     "status": "ok",
     "timestamp": 1595576993174,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "9jnH3oHmxldP"
   },
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    output = model(input_ids)\n",
    "label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 878,
     "status": "ok",
     "timestamp": 1595576994929,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "yquu1R74xqyw"
   },
   "outputs": [],
   "source": [
    "# join bpe split tokens\n",
    "tokens = tokenizer.convert_ids_to_tokens(input_ids.to('cpu').numpy()[0])\n",
    "new_tokens, new_labels = [], []\n",
    "for token, label_idx in zip(tokens, label_indices[0]):\n",
    "    if token.startswith(\"##\"):\n",
    "        new_tokens[-1] = new_tokens[-1] + token[2:]\n",
    "    else:\n",
    "        new_labels.append(tag_values[label_idx])\n",
    "        new_tokens.append(token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 571
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 606,
     "status": "ok",
     "timestamp": 1595576996133,
     "user": {
      "displayName": "Saikumar Dev",
      "photoUrl": "",
      "userId": "03015931960176323674"
     },
     "user_tz": -330
    },
    "id": "5c84jVkPxs7n",
    "outputId": "6d495757-63e4-47a4-af30-b0b40d328a75"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O\t[CLS]\n",
      "B-per\tMr\n",
      "B-per\t.\n",
      "I-per\tTrump\n",
      "O\t’\n",
      "O\ts\n",
      "O\ttweets\n",
      "O\tbegan\n",
      "O\tjust\n",
      "O\tmoments\n",
      "O\tafter\n",
      "O\ta\n",
      "B-org\tFox\n",
      "I-org\tNews\n",
      "O\treport\n",
      "O\tby\n",
      "B-per\tMike\n",
      "I-per\tTobin\n",
      "O\t,\n",
      "O\ta\n",
      "O\treporter\n",
      "O\tfor\n",
      "O\tthe\n",
      "O\tnetwork\n",
      "O\t,\n",
      "O\tabout\n",
      "O\tprotests\n",
      "O\tin\n",
      "B-geo\tMinnesota\n",
      "O\tand\n",
      "O\telsewhere\n",
      "O\t.\n",
      "O\t[SEP]\n"
     ]
    }
   ],
   "source": [
    "for token, label in zip(new_tokens, new_labels):\n",
    "    print(\"{}\\t{}\".format(label, token))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gxaE7feYxxUv"
   },
   "source": [
    "this works amazing! This approach will give you very strong performing models for named entity recognition. Since BERT is available as a multilingual model in 102 languages, you can use it for a wide variety of tasks. Try it for your problems and let me know how it works for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DUjTp5E_5ULF"
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uqISik0s5qwd"
   },
   "source": [
    "[Named entity recognition with Bert\n",
    "](https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)\n",
    "\n",
    "[Name Entity Recognition with BERT in TensorFlow](https://androidkt.com/name-entity-recognition-with-bert-in-tensorflow/)\n",
    "\n",
    "[Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.](https://github.com/huggingface/transformers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MJxcp60S5ZSI"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyMu2J0d3WoUueCMvWqYg4um",
   "collapsed_sections": [],
   "mount_file_id": "1R7wkvg4DnKWplBeBbLg8R_n-u6-X0SaI",
   "name": "ner_BERT_pytorch.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "06dd2f195f5846a1a9c5e83828255f29": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11a9aef6dece41d48af2ed66b8f4b806": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1b76b947f86042978a787cca37fcef7a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_666e97b5542a493bb3deb026cc690910",
      "placeholder": "​",
      "style": "IPY_MODEL_96efaef3e123490d89cc2e1b74db7e51",
      "value": " 433/433 [00:00&lt;00:00, 1.33kB/s]"
     }
    },
    "231f4852683c417aa529bb7955ea3e45": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "260be31f9c64401ea38426f6e0fe173a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "279e766e8b8642779aaa03a884f80c87": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dfe1476d0f76491c81630b304009e5ba",
      "max": 433,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_231f4852683c417aa529bb7955ea3e45",
      "value": 433
     }
    },
    "30b688358e1c42fcb761a42ecd18fcc1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3a581d6821e6496d88a1a1790100503a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ed2efdde5f4d4bb095401934940afb2d",
       "IPY_MODEL_c7d109df95ee4c74868b800c5a5024a2"
      ],
      "layout": "IPY_MODEL_af31b747552747ef8eff1b8db557a55c"
     }
    },
    "42aeeabfba234ed9a68e0dd36cff9e84": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_11a9aef6dece41d48af2ed66b8f4b806",
      "max": 213450,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_5bae04f105594ca7b1cd006f90e07213",
      "value": 213450
     }
    },
    "4900e38bbc684cab96b63dd34df16d85": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "5bae04f105594ca7b1cd006f90e07213": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "666e97b5542a493bb3deb026cc690910": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8959a967f46948b1beef30456b8eeab9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "96efaef3e123490d89cc2e1b74db7e51": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a22593cc69dd47de80298f953624efab": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_279e766e8b8642779aaa03a884f80c87",
       "IPY_MODEL_1b76b947f86042978a787cca37fcef7a"
      ],
      "layout": "IPY_MODEL_30b688358e1c42fcb761a42ecd18fcc1"
     }
    },
    "a4258a337320400db9729f301d3b115b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_42aeeabfba234ed9a68e0dd36cff9e84",
       "IPY_MODEL_f396604c499740659071205caa3b7afc"
      ],
      "layout": "IPY_MODEL_c38d9e21ae1944429ab950691bb64505"
     }
    },
    "af31b747552747ef8eff1b8db557a55c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b906cf5884f04e01af30129bf05fa76d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c38d9e21ae1944429ab950691bb64505": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c7d109df95ee4c74868b800c5a5024a2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d027e472beca4cc09e24871affa463e9",
      "placeholder": "​",
      "style": "IPY_MODEL_260be31f9c64401ea38426f6e0fe173a",
      "value": " 436M/436M [00:07&lt;00:00, 58.2MB/s]"
     }
    },
    "d027e472beca4cc09e24871affa463e9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dfe1476d0f76491c81630b304009e5ba": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ed2efdde5f4d4bb095401934940afb2d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8959a967f46948b1beef30456b8eeab9",
      "max": 435779157,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4900e38bbc684cab96b63dd34df16d85",
      "value": 435779157
     }
    },
    "f396604c499740659071205caa3b7afc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b906cf5884f04e01af30129bf05fa76d",
      "placeholder": "​",
      "style": "IPY_MODEL_06dd2f195f5846a1a9c5e83828255f29",
      "value": " 213k/213k [00:00&lt;00:00, 852kB/s]"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}