4. Mood Detection of Tweets with BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Mood Detection of Tweets - Word Embeddings and LSTM\n",
    "\n",
    "Now that we have our cleaned dataset, we will now continue to train a neural network to classify a tweet's mood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing for CUDA-enabled TF with GPU Support\n",
    "\n",
    "**What this means:**\n",
    "Deep Learning requires a lot of matrix calculations and your CPU is not really meant for that sort of task. However, graphics cards (the hardware that your computer uses to make games run well) are able to perform these sort of tasks very well, so we need to check if the Jupyter Notebook has access to your (Nvidia) GPU, in order to train the model much more efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0507 14:45:44.569529 140372727723776 __init__.py:56] Some hub symbols are not available because TensorFlow version is less than 1.14\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import os\n",
    "\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\"\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"2, 3\"\n",
    "\n",
    "config=tf.ConfigProto(device_count={'GPU':  2}, intra_op_parallelism_threads=1, inter_op_parallelism_threads=1, allow_soft_placement=True)\n",
    "config.gpu_options.allow_growth=True\n",
    "\n",
    "sess=tf.Session(graph=tf.get_default_graph(),config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default GPU Device: /device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "if tf.test.gpu_device_name():\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))\n",
    "else:\n",
    "    print(\"Please install GPU version of TF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "2.1.6-tf\n",
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "print(tf.test.is_built_with_cuda())\n",
    "print(tf.keras.__version__)\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing our mood tweets dataset\n",
    "\n",
    "Dataset can also be accessed at: [future Kaggle link for my dataset] \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the CSV dataset with pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman ab...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>@TheDumbMedico Ameen. Or ye suicide wali bat a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>You showed the leak in @HouseofCommons today @...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>The number in my bio is for a Suicide hotline ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>cw: suicidal ideation it's unsurprisingly hard...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class                                              tweet\n",
       "0  suicidal  \"He was lost & scared\" says a Newport woman ab...\n",
       "1  suicidal  @TheDumbMedico Ameen. Or ye suicide wali bat a...\n",
       "2  suicidal  You showed the leak in @HouseofCommons today @...\n",
       "3  suicidal  The number in my bio is for a Suicide hotline ...\n",
       "4  suicidal  cw: suicidal ideation it's unsurprisingly hard..."
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"datasets/mood_tweets.csv\")\n",
    "df.drop(df.columns[0], inplace=True, axis=1)\n",
    "df[\"tweet\"] = df[\"tweet\"].astype(str)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>25758</td>\n",
       "      <td>25758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>6</td>\n",
       "      <td>25758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>@Kkeke_99 truth bc i didnt know about them unt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>5885</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           class                                              tweet\n",
       "count      25758                                              25758\n",
       "unique         6                                              25758\n",
       "top     cheerful  @Kkeke_99 truth bc i didnt know about them unt...\n",
       "freq        5885                                                  1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's analyse how the dataset is lain out and roughly how the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18115</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college &amp; university. I honestly wish I had started this years ago.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4418</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4417</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Glad to see that Craig continues to keep the orientalism industry alive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4416</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>@_Kazma8 No problem dude. Glad she was found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4415</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Sign up for this awesome contest!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          class  \\\n",
       "18115  cheerful   \n",
       "4418   cheerful   \n",
       "4417   cheerful   \n",
       "4416   cheerful   \n",
       "4415   cheerful   \n",
       "\n",
       "                                                                                                                                                                                                                                                          tweet  \n",
       "18115  You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college & university. I honestly wish I had started this years ago.                         \n",
       "4418   Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.  \n",
       "4417   Glad to see that Craig continues to keep the orientalism industry alive                                                                                                                                                                                   \n",
       "4416   @_Kazma8 No problem dude. Glad she was found                                                                                                                                                                                                              \n",
       "4415   Sign up for this awesome contest!                                                                                                                                                                                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14794</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>But why does my client commit suicide if I press the ranked ladder tab?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14793</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14792</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8106</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>@liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          class  \\\n",
       "14794  suicidal   \n",
       "14793  suicidal   \n",
       "14792  suicidal   \n",
       "8106   suicidal   \n",
       "0      suicidal   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                      tweet  \n",
       "14794  But why does my client commit suicide if I press the ranked ladder tab?                                                                                                                                                                                                               \n",
       "14793  Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg                                                                                                                                         \n",
       "14792  The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight                                                                                                       \n",
       "8106   @liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?                                                                     \n",
       "0      \"He was lost & scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array(['cheerful', 'depressed', 'happy', 'overjoyed', 'sad', 'suicidal'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1) # option to be set so that the tweet's texts won't be truncated\n",
    "df.sort_values(\"class\", inplace=True)\n",
    "display(df.head())\n",
    "display(df.tail())\n",
    "df[\"class\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now change the classes into a numerical representation to make it easier for deep learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18115</th>\n",
       "      <td>4</td>\n",
       "      <td>You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college &amp; university. I honestly wish I had started this years ago.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4418</th>\n",
       "      <td>4</td>\n",
       "      <td>Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4417</th>\n",
       "      <td>4</td>\n",
       "      <td>Glad to see that Craig continues to keep the orientalism industry alive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4416</th>\n",
       "      <td>4</td>\n",
       "      <td>@_Kazma8 No problem dude. Glad she was found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4415</th>\n",
       "      <td>4</td>\n",
       "      <td>Sign up for this awesome contest!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class  \\\n",
       "18115  4      \n",
       "4418   4      \n",
       "4417   4      \n",
       "4416   4      \n",
       "4415   4      \n",
       "\n",
       "                                                                                                                                                                                                                                                          tweet  \n",
       "18115  You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college & university. I honestly wish I had started this years ago.                         \n",
       "4418   Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.  \n",
       "4417   Glad to see that Craig continues to keep the orientalism industry alive                                                                                                                                                                                   \n",
       "4416   @_Kazma8 No problem dude. Glad she was found                                                                                                                                                                                                              \n",
       "4415   Sign up for this awesome contest!                                                                                                                                                                                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14794</th>\n",
       "      <td>0</td>\n",
       "      <td>But why does my client commit suicide if I press the ranked ladder tab?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14793</th>\n",
       "      <td>0</td>\n",
       "      <td>Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14792</th>\n",
       "      <td>0</td>\n",
       "      <td>The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8106</th>\n",
       "      <td>0</td>\n",
       "      <td>@liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class  \\\n",
       "14794  0      \n",
       "14793  0      \n",
       "14792  0      \n",
       "8106   0      \n",
       "0      0      \n",
       "\n",
       "                                                                                                                                                                                                                                                                                      tweet  \n",
       "14794  But why does my client commit suicide if I press the ranked ladder tab?                                                                                                                                                                                                               \n",
       "14793  Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg                                                                                                                                         \n",
       "14792  The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight                                                                                                       \n",
       "8106   @liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?                                                                     \n",
       "0      \"He was lost & scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classes_index = {\n",
    "    0 : \"suicidal\",\n",
    "    1 : \"depressed\",\n",
    "    2 : \"sad\",\n",
    "    3 : \"happy\",\n",
    "    4 : \"cheerful\",\n",
    "    5 : \"overjoyed\",\n",
    "}\n",
    "\n",
    "for val, class_name in classes_index.items():\n",
    "    df.loc[(df[\"class\"] == class_name), \"class\"] = val\n",
    "\n",
    "display(df.head())\n",
    "display(df.tail())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bert\n",
    "from bert import run_classifier\n",
    "from bert import optimization\n",
    "from bert import tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we'll set an output directory location to store our model output and checkpoints. This can be a local directory, in which case you'd set OUTPUT_DIR to the name of the directory you'd like to create. If you're running this code in Google's hosted Colab, the directory won't persist after the Colab session ends.\n",
    "\n",
    "Alternatively, if you're a GCP user, you can store output in a GCP bucket. To do that, set a directory name in OUTPUT_DIR and the name of the GCP bucket in the BUCKET field.\n",
    "\n",
    "Set DO_DELETE to rewrite the OUTPUT_DIR if it exists. Otherwise, Tensorflow will load existing model checkpoints from that directory (if they exist)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Model output directory: bert_models *****\n"
     ]
    }
   ],
   "source": [
    "# Set the output directory for saving model file\n",
    "# Optionally, set a GCP bucket location\n",
    "\n",
    "OUTPUT_DIR = 'bert_models'\n",
    "\n",
    "# Whether or not to clear/delete the directory and create a new one\n",
    "DO_DELETE = False\n",
    "\n",
    "if not os.path.exists(OUTPUT_DIR):\n",
    "    os.mkdir(OUTPUT_DIR)\n",
    "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DATA_COLUMN = 'tweet'\n",
    "LABEL_COLUMN = 'class'\n",
    "# label_list is the list of labels, i.e. True, False or 0, 1 or 'dog', 'cat'\n",
    "label_list = list(classes_index.keys())\n",
    "label_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing\n",
    "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create  `InputExample`'s using the constructor provided in the BERT library.\n",
    "\n",
    "- `text_a` is the text we want to classify, which in this case, is the `Request` field in our Dataframe. \n",
    "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
    "- `label` is the label for our example, i.e. True, False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the InputExample class from BERT's run_classifier code to create examples from the data\n",
    "df_InputExamples = df.apply(lambda x: bert.run_classifier.InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
    "                                                                   text_a = x[DATA_COLUMN], \n",
    "                                                                   text_b = None, \n",
    "                                                                   label = x[LABEL_COLUMN]), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18115    <bert.run_classifier.InputExample object at 0x7faa88168d30>\n",
       "4418     <bert.run_classifier.InputExample object at 0x7faa88168f98>\n",
       "4417     <bert.run_classifier.InputExample object at 0x7faa88168fd0>\n",
       "4416     <bert.run_classifier.InputExample object at 0x7faa8817d048>\n",
       "4415     <bert.run_classifier.InputExample object at 0x7faa8817d080>\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_InputExamples.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to preprocess our data so that it matches the data BERT was trained on. For this, we'll need to do a couple of things (but don't worry--this is also included in the Python library):\n",
    "\n",
    "\n",
    "1. Lowercase our text (if we're using a BERT lowercase model)\n",
    "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
    "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
    "4. Map our words to indexes using a vocab file that BERT provides\n",
    "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
    "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
    "\n",
    "Happily, we don't have to worry about most of these details.\n",
    "\n",
    "To start, we'll need to load a vocabulary file and lowercasing information directly from the BERT tf hub module:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.009019 140372727723776 tf_logging.py:115] Saver not created because there are no variables in the graph to restore\n"
     ]
    }
   ],
   "source": [
    "# This is a path to an uncased (all lowercase) version of BERT\n",
    "BERT_MODEL_HUB = \"https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1\"\n",
    "\n",
    "def create_tokenizer_from_hub_module():\n",
    "  \"\"\"Get the vocab file and casing info from the Hub module.\"\"\"\n",
    "  with tf.Graph().as_default():\n",
    "    bert_module = hub.Module(BERT_MODEL_HUB)\n",
    "    tokenization_info = bert_module(signature=\"tokenization_info\", as_dict=True)\n",
    "    with tf.Session() as sess:\n",
    "      vocab_file, do_lower_case = sess.run([tokenization_info[\"vocab_file\"],\n",
    "                                            tokenization_info[\"do_lower_case\"]])\n",
    "      \n",
    "  return bert.tokenization.FullTokenizer(\n",
    "      vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
    "\n",
    "tokenizer = create_tokenizer_from_hub_module()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great--we just learned that the BERT model we're using expects lowercase data (that's what stored in tokenization_info[\"do_lower_case\"]) and we also loaded BERT's vocab file. We also created a tokenizer, which breaks words into word pieces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this',\n",
       " 'here',\n",
       " \"'\",\n",
       " 's',\n",
       " 'an',\n",
       " 'example',\n",
       " 'of',\n",
       " 'using',\n",
       " 'the',\n",
       " 'bert',\n",
       " 'token',\n",
       " '##izer']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using our tokenizer, we'll call `run_classifier.convert_examples_to_features` on our InputExamples to convert them into features BERT understands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 25758\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.577533 140372727723776 tf_logging.py:115] Writing example 0 of 25758\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.580434 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.581424 140372727723776 tf_logging.py:115] guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] you know what ' s awesome ? after a month of doing yoga i ' m getting most of my range of motion and flexibility back in the knee that i blew out playing rugby in college & university . i honestly wish i had started this years ago . [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.582406 140372727723776 tf_logging.py:115] tokens: [CLS] you know what ' s awesome ? after a month of doing yoga i ' m getting most of my range of motion and flexibility back in the knee that i blew out playing rugby in college & university . i honestly wish i had started this years ago . [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2017 2113 2054 1005 1055 12476 1029 2044 1037 3204 1997 2725 13272 1045 1005 1049 2893 2087 1997 2026 2846 1997 4367 1998 16991 2067 1999 1996 6181 2008 1045 8682 2041 2652 4043 1999 2267 1004 2118 1012 1045 9826 4299 1045 2018 2318 2023 2086 3283 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.583408 140372727723776 tf_logging.py:115] input_ids: 101 2017 2113 2054 1005 1055 12476 1029 2044 1037 3204 1997 2725 13272 1045 1005 1049 2893 2087 1997 2026 2846 1997 4367 1998 16991 2067 1999 1996 6181 2008 1045 8682 2041 2652 4043 1999 2267 1004 2118 1012 1045 9826 4299 1045 2018 2318 2023 2086 3283 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.584352 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.585269 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.586126 140372727723776 tf_logging.py:115] label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.588286 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.589180 140372727723776 tf_logging.py:115] guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] just rolled over 10 ##k awesome miles in @ corbett and my @ tesla 3 . @ el ##on ##mus ##k one suggestion : can i pl ##z ha ##z headlights on by default ? every time i drive during the day they start off and i think they improve visibility for other drivers as i pass them . [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.590080 140372727723776 tf_logging.py:115] tokens: [CLS] just rolled over 10 ##k awesome miles in @ corbett and my @ tesla 3 . @ el ##on ##mus ##k one suggestion : can i pl ##z ha ##z headlights on by default ? every time i drive during the day they start off and i think they improve visibility for other drivers as i pass them . [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2074 4565 2058 2184 2243 12476 2661 1999 1030 24119 1998 2026 1030 26060 1017 1012 1030 3449 2239 7606 2243 2028 10293 1024 2064 1045 20228 2480 5292 2480 18167 2006 2011 12398 1029 2296 2051 1045 3298 2076 1996 2154 2027 2707 2125 1998 1045 2228 2027 5335 16476 2005 2060 6853 2004 1045 3413 2068 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.590997 140372727723776 tf_logging.py:115] input_ids: 101 2074 4565 2058 2184 2243 12476 2661 1999 1030 24119 1998 2026 1030 26060 1017 1012 1030 3449 2239 7606 2243 2028 10293 1024 2064 1045 20228 2480 5292 2480 18167 2006 2011 12398 1029 2296 2051 1045 3298 2076 1996 2154 2027 2707 2125 1998 1045 2228 2027 5335 16476 2005 2060 6853 2004 1045 3413 2068 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.591941 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.592838 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.593694 140372727723776 tf_logging.py:115] label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.594990 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.595895 140372727723776 tf_logging.py:115] guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] glad to see that craig continues to keep the oriental ##ism industry alive [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.596774 140372727723776 tf_logging.py:115] tokens: [CLS] glad to see that craig continues to keep the oriental ##ism industry alive [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 5580 2000 2156 2008 7010 4247 2000 2562 1996 11481 2964 3068 4142 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.597675 140372727723776 tf_logging.py:115] input_ids: 101 5580 2000 2156 2008 7010 4247 2000 2562 1996 11481 2964 3068 4142 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.598592 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.599516 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.600365 140372727723776 tf_logging.py:115] label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.601540 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.602408 140372727723776 tf_logging.py:115] guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] @ _ ka ##z ##ma ##8 no problem dude . glad she was found [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.603272 140372727723776 tf_logging.py:115] tokens: [CLS] @ _ ka ##z ##ma ##8 no problem dude . glad she was found [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1030 1035 10556 2480 2863 2620 2053 3291 12043 1012 5580 2016 2001 2179 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.604159 140372727723776 tf_logging.py:115] input_ids: 101 1030 1035 10556 2480 2863 2620 2053 3291 12043 1012 5580 2016 2001 2179 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.605065 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.605959 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.606816 140372727723776 tf_logging.py:115] label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.607909 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.608771 140372727723776 tf_logging.py:115] guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] sign up for this awesome contest ! [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.609630 140372727723776 tf_logging.py:115] tokens: [CLS] sign up for this awesome contest ! [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 3696 2039 2005 2023 12476 5049 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.610503 140372727723776 tf_logging.py:115] input_ids: 101 3696 2039 2005 2023 12476 5049 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.611419 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.612292 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:51.613139 140372727723776 tf_logging.py:115] label: 4 (id = 4)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 10000 of 25758\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:45:57.385486 140372727723776 tf_logging.py:115] Writing example 10000 of 25758\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 20000 of 25758\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:02.568547 140372727723776 tf_logging.py:115] Writing example 20000 of 25758\n"
     ]
    }
   ],
   "source": [
    "# We'll set sequences to be at most 128 tokens long.\n",
    "MAX_SEQ_LENGTH = 128\n",
    "# Convert our train and test features to InputFeatures that BERT understands.\n",
    "df_features = bert.run_classifier.convert_examples_to_features(df_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a model\n",
    "\n",
    "Now that we've prepared our data, let's focus on building a model. `create_model` does just this below. First, it loads the BERT tf hub module again (this time to extract the computation graph). Next, it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_model(is_predicting, input_ids, input_mask, segment_ids, labels,\n",
    "                 num_labels):\n",
    "  \"\"\"Creates a classification model.\"\"\"\n",
    "\n",
    "  bert_module = hub.Module(\n",
    "      BERT_MODEL_HUB,\n",
    "      trainable=True)\n",
    "  bert_inputs = dict(\n",
    "      input_ids=input_ids,\n",
    "      input_mask=input_mask,\n",
    "      segment_ids=segment_ids)\n",
    "  bert_outputs = bert_module(\n",
    "      inputs=bert_inputs,\n",
    "      signature=\"tokens\",\n",
    "      as_dict=True)\n",
    "\n",
    "  # Use \"pooled_output\" for classification tasks on an entire sentence.\n",
    "  # Use \"sequence_outputs\" for token-level output.\n",
    "  output_layer = bert_outputs[\"pooled_output\"]\n",
    "\n",
    "  hidden_size = output_layer.shape[-1].value\n",
    "\n",
    "  # Create our own layer to tune for politeness data.\n",
    "  output_weights = tf.get_variable(\n",
    "      \"output_weights\", [num_labels, hidden_size],\n",
    "      initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
    "\n",
    "  output_bias = tf.get_variable(\n",
    "      \"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
    "\n",
    "  with tf.variable_scope(\"loss\"):\n",
    "\n",
    "    # Dropout helps prevent overfitting\n",
    "    output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
    "\n",
    "    logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
    "    logits = tf.nn.bias_add(logits, output_bias)\n",
    "    log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
    "\n",
    "    # Convert labels into one-hot encoding\n",
    "    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
    "\n",
    "    predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))\n",
    "    # If we're predicting, we want predicted labels and the probabiltiies.\n",
    "    if is_predicting:\n",
    "      return (predicted_labels, log_probs)\n",
    "\n",
    "    # If we're train/eval, compute loss between predicted and actual label\n",
    "    per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
    "    loss = tf.reduce_mean(per_example_loss)\n",
    "    return (loss, predicted_labels, log_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll wrap our model function in a `model_fn_builder` function that adapts our model to work for training, evaluation, and prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_fn_builder actually creates our model function\n",
    "# using the passed parameters for num_labels, learning_rate, etc.\n",
    "def model_fn_builder(num_labels, learning_rate, num_train_steps,\n",
    "                     num_warmup_steps):\n",
    "  \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
    "  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument\n",
    "    \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
    "\n",
    "    input_ids = features[\"input_ids\"]\n",
    "    input_mask = features[\"input_mask\"]\n",
    "    segment_ids = features[\"segment_ids\"]\n",
    "    label_ids = features[\"label_ids\"]\n",
    "\n",
    "    is_predicting = (mode == tf.estimator.ModeKeys.PREDICT)\n",
    "    \n",
    "    # TRAIN and EVAL\n",
    "    if not is_predicting:\n",
    "\n",
    "      (loss, predicted_labels, log_probs) = create_model(\n",
    "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
    "\n",
    "      train_op = bert.optimization.create_optimizer(\n",
    "          loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu=False)\n",
    "\n",
    "      # Calculate evaluation metrics. \n",
    "      def metric_fn(label_ids, predicted_labels):\n",
    "        accuracy = tf.metrics.accuracy(label_ids, predicted_labels)\n",
    "        #f1_score = tf.contrib.metrics.f1_score(\n",
    "        #    label_ids,\n",
    "        #    predicted_labels)\n",
    "        #auc = tf.metrics.auc(\n",
    "        #    label_ids,\n",
    "        #    predicted_labels)\n",
    "        recall = tf.metrics.recall(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        precision = tf.metrics.precision(\n",
    "            label_ids,\n",
    "            predicted_labels) \n",
    "        true_pos = tf.metrics.true_positives(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        true_neg = tf.metrics.true_negatives(\n",
    "            label_ids,\n",
    "            predicted_labels)   \n",
    "        false_pos = tf.metrics.false_positives(\n",
    "            label_ids,\n",
    "            predicted_labels)  \n",
    "        false_neg = tf.metrics.false_negatives(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        return {\n",
    "            \"eval_accuracy\": accuracy,\n",
    "            #\"f1_score\": f1_score,\n",
    "            #\"auc\": auc,\n",
    "            \"precision\": precision,\n",
    "            \"recall\": recall,\n",
    "            \"true_positives\": true_pos,\n",
    "            \"true_negatives\": true_neg,\n",
    "            \"false_positives\": false_pos,\n",
    "            \"false_negatives\": false_neg\n",
    "        }\n",
    "\n",
    "      eval_metrics = metric_fn(label_ids, predicted_labels)\n",
    "\n",
    "      if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        return tf.estimator.EstimatorSpec(mode=mode,\n",
    "          loss=loss,\n",
    "          train_op=train_op)\n",
    "      else:\n",
    "          return tf.estimator.EstimatorSpec(mode=mode,\n",
    "            loss=loss,\n",
    "            eval_metric_ops=eval_metrics)\n",
    "    else:\n",
    "      (predicted_labels, log_probs) = create_model(\n",
    "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
    "\n",
    "      predictions = {\n",
    "          'probabilities': log_probs,\n",
    "          'labels': predicted_labels\n",
    "      }\n",
    "      return tf.estimator.EstimatorSpec(mode, predictions=predictions)\n",
    "\n",
    "  # Return the actual model function in the closure\n",
    "  return model_fn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute train and warmup steps from batch size\n",
    "# These hyperparameters are copied from this colab notebook (https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\n",
    "BATCH_SIZE = 32\n",
    "LEARNING_RATE = 2e-5\n",
    "NUM_TRAIN_EPOCHS = 3.0\n",
    "# Warmup is a period of time where hte learning rate \n",
    "# is small and gradually increases--usually helps training.\n",
    "WARMUP_PROPORTION = 0.1\n",
    "# Model configs\n",
    "SAVE_CHECKPOINTS_STEPS = 500\n",
    "SAVE_SUMMARY_STEPS = 100\n",
    "VALIDATION_SPLIT = 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25758"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5151"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "# we shuffle our sorted data, to make it random first\n",
    "random.shuffle(df_features)\n",
    "\n",
    "# before we begin to split them\n",
    "num_validation_samples = int(VALIDATION_SPLIT * len(df_features))\n",
    "num_validation_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_features = df_features[:num_validation_samples]\n",
    "train_features = df_features[num_validation_samples:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute # train and warmup steps from batch size\n",
    "num_train_steps = int(len(train_features) / BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
    "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify output directory and number of checkpoint steps to save\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    model_dir=OUTPUT_DIR,\n",
    "    save_summary_steps=SAVE_SUMMARY_STEPS,\n",
    "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': 'bert_models', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7faa31b08128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:06.531221 140372727723776 tf_logging.py:115] Using config: {'_model_dir': 'bert_models', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7faa31b08128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "model_fn = model_fn_builder(\n",
    "  num_labels=len(label_list),\n",
    "  learning_rate=LEARNING_RATE,\n",
    "  num_train_steps=num_train_steps,\n",
    "  num_warmup_steps=num_warmup_steps)\n",
    "\n",
    "estimator = tf.estimator.Estimator(\n",
    "  model_fn=model_fn,\n",
    "  config=run_config,\n",
    "  params={\"batch_size\": BATCH_SIZE})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we create an input builder function that takes our training feature set (`train_features`) and produces a generator. This is a pretty standard design pattern for working with Tensorflow [Estimators](https://www.tensorflow.org/guide/estimators)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an input function for training. drop_remainder = True for using TPUs.\n",
    "train_input_fn = bert.run_classifier.input_fn_builder(\n",
    "    features=train_features,\n",
    "    seq_length=MAX_SEQ_LENGTH,\n",
    "    is_training=True,\n",
    "    drop_remainder=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we train our model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beginning Training!\n",
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:18.953002 140372727723776 tf_logging.py:115] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:22.185296 140372727723776 tf_logging.py:115] Saver not created because there are no variables in the graph to restore\n",
      "/home/gzhenfun/anaconda3/envs/bert/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:31.217102 140372727723776 tf_logging.py:115] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Create CheckpointSaverHook.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:31.219996 140372727723776 tf_logging.py:115] Create CheckpointSaverHook.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Graph was finalized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:36.089108 140372727723776 tf_logging.py:115] Graph was finalized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:41.170332 140372727723776 tf_logging.py:115] Running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:41.321447 140372727723776 tf_logging.py:115] Done running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 0 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:46:55.777678 140372727723776 tf_logging.py:115] Saving checkpoints for 0 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 1.7532601, step = 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:47:06.765743 140372727723776 tf_logging.py:115] loss = 1.7532601, step = 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.47515\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:48:14.555107 140372727723776 tf_logging.py:115] global_step/sec: 1.47515\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.1330314, step = 101 (67.792 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:48:14.558142 140372727723776 tf_logging.py:115] loss = 0.1330314, step = 101 (67.792 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85797\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:49:08.377319 140372727723776 tf_logging.py:115] global_step/sec: 1.85797\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.01043457, step = 201 (53.823 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:49:08.381397 140372727723776 tf_logging.py:115] loss = 0.01043457, step = 201 (53.823 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85203\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:50:02.372009 140372727723776 tf_logging.py:115] global_step/sec: 1.85203\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.059115604, step = 301 (53.994 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:50:02.375793 140372727723776 tf_logging.py:115] loss = 0.059115604, step = 301 (53.994 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.84755\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:50:56.497850 140372727723776 tf_logging.py:115] global_step/sec: 1.84755\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.04622001, step = 401 (54.125 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:50:56.500905 140372727723776 tf_logging.py:115] loss = 0.04622001, step = 401 (54.125 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 500 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:51:50.026949 140372727723776 tf_logging.py:115] Saving checkpoints for 500 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.73681\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:51:54.074596 140372727723776 tf_logging.py:115] global_step/sec: 1.73681\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.16945139, step = 501 (57.576 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:51:54.076485 140372727723776 tf_logging.py:115] loss = 0.16945139, step = 501 (57.576 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.84998\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:52:48.129420 140372727723776 tf_logging.py:115] global_step/sec: 1.84998\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.005248884, step = 601 (54.056 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:52:48.132100 140372727723776 tf_logging.py:115] loss = 0.005248884, step = 601 (54.056 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85489\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:53:42.041126 140372727723776 tf_logging.py:115] global_step/sec: 1.85489\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.14718188, step = 701 (53.913 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:53:42.044778 140372727723776 tf_logging.py:115] loss = 0.14718188, step = 701 (53.913 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.8506\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:54:36.077679 140372727723776 tf_logging.py:115] global_step/sec: 1.8506\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0026476844, step = 801 (54.036 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:54:36.081133 140372727723776 tf_logging.py:115] loss = 0.0026476844, step = 801 (54.036 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.84994\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:55:30.133446 140372727723776 tf_logging.py:115] global_step/sec: 1.84994\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0036297175, step = 901 (54.055 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:55:30.136150 140372727723776 tf_logging.py:115] loss = 0.0036297175, step = 901 (54.055 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 1000 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:56:23.623900 140372727723776 tf_logging.py:115] Saving checkpoints for 1000 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.75364\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:56:27.157443 140372727723776 tf_logging.py:115] global_step/sec: 1.75364\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.058256775, step = 1001 (57.023 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:56:27.159082 140372727723776 tf_logging.py:115] loss = 0.058256775, step = 1001 (57.023 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85188\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:57:21.156697 140372727723776 tf_logging.py:115] global_step/sec: 1.85188\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0023932336, step = 1101 (54.000 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:57:21.159393 140372727723776 tf_logging.py:115] loss = 0.0023932336, step = 1101 (54.000 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85292\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:58:15.125372 140372727723776 tf_logging.py:115] global_step/sec: 1.85292\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0020429878, step = 1201 (53.968 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:58:15.127703 140372727723776 tf_logging.py:115] loss = 0.0020429878, step = 1201 (53.968 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.84996\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:59:09.180814 140372727723776 tf_logging.py:115] global_step/sec: 1.84996\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.017281398, step = 1301 (54.056 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 14:59:09.183718 140372727723776 tf_logging.py:115] loss = 0.017281398, step = 1301 (54.056 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85169\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:00:03.185542 140372727723776 tf_logging.py:115] global_step/sec: 1.85169\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.008529071, step = 1401 (54.005 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:00:03.188677 140372727723776 tf_logging.py:115] loss = 0.008529071, step = 1401 (54.005 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 1500 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:00:56.767282 140372727723776 tf_logging.py:115] Saving checkpoints for 1500 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.74969\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:01:00.338342 140372727723776 tf_logging.py:115] global_step/sec: 1.74969\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0019483108, step = 1501 (57.152 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:01:00.340330 140372727723776 tf_logging.py:115] loss = 0.0019483108, step = 1501 (57.152 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85044\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:01:54.379629 140372727723776 tf_logging.py:115] global_step/sec: 1.85044\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.027974296, step = 1601 (54.042 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:01:54.382384 140372727723776 tf_logging.py:115] loss = 0.027974296, step = 1601 (54.042 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.8494\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:02:48.451101 140372727723776 tf_logging.py:115] global_step/sec: 1.8494\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.001222246, step = 1701 (54.071 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:02:48.453061 140372727723776 tf_logging.py:115] loss = 0.001222246, step = 1701 (54.071 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.85036\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:03:42.494772 140372727723776 tf_logging.py:115] global_step/sec: 1.85036\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0018905711, step = 1801 (54.044 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:03:42.497460 140372727723776 tf_logging.py:115] loss = 0.0018905711, step = 1801 (54.044 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:global_step/sec: 1.84937\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:04:36.567366 140372727723776 tf_logging.py:115] global_step/sec: 1.84937\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:loss = 0.0015011837, step = 1901 (54.073 sec)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:04:36.570474 140372727723776 tf_logging.py:115] loss = 0.0015011837, step = 1901 (54.073 sec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving checkpoints for 1931 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:04:52.789425 140372727723776 tf_logging.py:115] Saving checkpoints for 1931 into bert_models/model.ckpt.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loss for final step: 0.0012503576.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:04:56.128183 140372727723776 tf_logging.py:115] Loss for final step: 0.0012503576.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took time  0:18:49.375330\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "print(f'Beginning Training!')\n",
    "current_time = datetime.now()\n",
    "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
    "print(\"Training took time \", datetime.now() - current_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use our test data to see how well our model did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_input_fn = run_classifier.input_fn_builder(\n",
    "    features=test_features,\n",
    "    seq_length=MAX_SEQ_LENGTH,\n",
    "    is_training=False,\n",
    "    drop_remainder=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:04:58.973592 140372727723776 tf_logging.py:115] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:02.991786 140372727723776 tf_logging.py:115] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:11.962692 140372727723776 tf_logging.py:115] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Starting evaluation at 2019-05-07-07:05:11\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:11.986277 140372727723776 tf_logging.py:115] Starting evaluation at 2019-05-07-07:05:11\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Graph was finalized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:13.390394 140372727723776 tf_logging.py:115] Graph was finalized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:13.396456 140372727723776 tf_logging.py:115] Restoring parameters from bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:15.408491 140372727723776 tf_logging.py:115] Running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:15.624501 140372727723776 tf_logging.py:115] Done running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Finished evaluation at 2019-05-07-07:05:44\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:44.891416 140372727723776 tf_logging.py:115] Finished evaluation at 2019-05-07-07:05:44\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving dict for global step 1931: eval_accuracy = 0.9846632, false_negatives = 10.0, false_positives = 13.0, global_step = 1931, loss = 0.07248825, precision = 0.9971194, recall = 0.9977827, true_negatives = 628.0, true_positives = 4500.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:44.894817 140372727723776 tf_logging.py:115] Saving dict for global step 1931: eval_accuracy = 0.9846632, false_negatives = 10.0, false_positives = 13.0, global_step = 1931, loss = 0.07248825, precision = 0.9971194, recall = 0.9977827, true_negatives = 628.0, true_positives = 4500.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1931: bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.362723 140372727723776 tf_logging.py:115] Saving 'checkpoint_path' summary for global step 1931: bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'eval_accuracy': 0.9846632,\n",
       " 'false_negatives': 10.0,\n",
       " 'false_positives': 13.0,\n",
       " 'loss': 0.07248825,\n",
       " 'precision': 0.9971194,\n",
       " 'recall': 0.9977827,\n",
       " 'true_negatives': 628.0,\n",
       " 'true_positives': 4500.0,\n",
       " 'global_step': 1931}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator.evaluate(input_fn=test_input_fn, steps=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's write code to make predictions on new sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getPrediction(in_sentences):\n",
    "  labels = list(classes_index.items())\n",
    "  input_examples = [run_classifier.InputExample(guid=\"\", text_a = x, text_b = None, label = 0) for x in in_sentences] # here, \"\" is just a dummy label\n",
    "  input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
    "  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=False)\n",
    "  predictions = estimator.predict(predict_input_fn)\n",
    "  return [(sentence, prediction['probabilities'], labels[prediction['labels']]) for sentence, prediction in zip(in_sentences, predictions)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_sentences = [\n",
    "  \"If I just disappear from the world right now, will anyone even care?\",\n",
    "  \"Nobody can understand the stress and anxiety that exists within me right now. Nobody cares.\",\n",
    "  \"When I'm training my neural network, I can't watch netflix because it lags like crazy. Sigh\",\n",
    "  \"Today seems like a fine day. I can't wait to go to school today.\",\n",
    "  \"Wow your scrambled eggs are delicious!\",\n",
    "  \"I am so excited for Captain Marvel actually\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.570571 140372727723776 tf_logging.py:115] Writing example 0 of 6\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.573590 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.578235 140372727723776 tf_logging.py:115] guid: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] if i just disappear from the world right now , will anyone even care ? [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.579765 140372727723776 tf_logging.py:115] tokens: [CLS] if i just disappear from the world right now , will anyone even care ? [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2065 1045 2074 10436 2013 1996 2088 2157 2085 1010 2097 3087 2130 2729 1029 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.581306 140372727723776 tf_logging.py:115] input_ids: 101 2065 1045 2074 10436 2013 1996 2088 2157 2085 1010 2097 3087 2130 2729 1029 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.582887 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.584558 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.586024 140372727723776 tf_logging.py:115] label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.588223 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.589659 140372727723776 tf_logging.py:115] guid: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] nobody can understand the stress and anxiety that exists within me right now . nobody cares . [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.591088 140372727723776 tf_logging.py:115] tokens: [CLS] nobody can understand the stress and anxiety that exists within me right now . nobody cares . [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 6343 2064 3305 1996 6911 1998 10089 2008 6526 2306 2033 2157 2085 1012 6343 14977 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.592565 140372727723776 tf_logging.py:115] input_ids: 101 6343 2064 3305 1996 6911 1998 10089 2008 6526 2306 2033 2157 2085 1012 6343 14977 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.594082 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.595478 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.596847 140372727723776 tf_logging.py:115] label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.598833 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.600208 140372727723776 tf_logging.py:115] guid: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] when i ' m training my neural network , i can ' t watch netflix because it la ##gs like crazy . sigh [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.601533 140372727723776 tf_logging.py:115] tokens: [CLS] when i ' m training my neural network , i can ' t watch netflix because it la ##gs like crazy . sigh [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2043 1045 1005 1049 2731 2026 15756 2897 1010 1045 2064 1005 1056 3422 20907 2138 2009 2474 5620 2066 4689 1012 6682 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.602911 140372727723776 tf_logging.py:115] input_ids: 101 2043 1045 1005 1049 2731 2026 15756 2897 1010 1045 2064 1005 1056 3422 20907 2138 2009 2474 5620 2066 4689 1012 6682 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.604394 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.605806 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.607162 140372727723776 tf_logging.py:115] label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.608933 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.610297 140372727723776 tf_logging.py:115] guid: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] today seems like a fine day . i can ' t wait to go to school today . [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.611660 140372727723776 tf_logging.py:115] tokens: [CLS] today seems like a fine day . i can ' t wait to go to school today . [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2651 3849 2066 1037 2986 2154 1012 1045 2064 1005 1056 3524 2000 2175 2000 2082 2651 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.612961 140372727723776 tf_logging.py:115] input_ids: 101 2651 3849 2066 1037 2986 2154 1012 1045 2064 1005 1056 3524 2000 2175 2000 2082 2651 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.614266 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.615558 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.616789 140372727723776 tf_logging.py:115] label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.618367 140372727723776 tf_logging.py:115] *** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.619591 140372727723776 tf_logging.py:115] guid: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] wow your scrambled eggs are delicious ! [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.620784 140372727723776 tf_logging.py:115] tokens: [CLS] wow your scrambled eggs are delicious ! [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 10166 2115 13501 6763 2024 12090 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.621952 140372727723776 tf_logging.py:115] input_ids: 101 10166 2115 13501 6763 2024 12090 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.623135 140372727723776 tf_logging.py:115] input_mask: 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.624360 140372727723776 tf_logging.py:115] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.625548 140372727723776 tf_logging.py:115] label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:47.664012 140372727723776 tf_logging.py:115] Calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:51.673863 140372727723776 tf_logging.py:115] Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done calling model_fn.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:51.889468 140372727723776 tf_logging.py:115] Done calling model_fn.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Graph was finalized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:52.396250 140372727723776 tf_logging.py:115] Graph was finalized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:52.400949 140372727723776 tf_logging.py:115] Restoring parameters from bert_models/model.ckpt-1931\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:53.185207 140372727723776 tf_logging.py:115] Running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I0507 15:05:53.234896 140372727723776 tf_logging.py:115] Done running local_init_op.\n"
     ]
    }
   ],
   "source": [
    "predictions = getPrediction(pred_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila! We have a sentiment classifier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('If I just disappear from the world right now, will anyone even care?',\n",
       "  array([-0.05108946, -3.2444801 , -4.8545737 , -6.6508336 , -7.8408146 ,\n",
       "         -6.6161776 ], dtype=float32),\n",
       "  (0, 'suicidal')),\n",
       " ('Nobody can understand the stress and anxiety that exists within me right now. Nobody cares.',\n",
       "  array([-0.53542405, -0.89993197, -6.3480854 , -5.847459  , -6.9001694 ,\n",
       "         -6.0589876 ], dtype=float32),\n",
       "  (0, 'suicidal')),\n",
       " (\"When I'm training my neural network, I can't watch netflix because it lags like crazy. Sigh\",\n",
       "  array([-5.0027833, -0.5956425, -3.074578 , -4.671892 , -1.0175464,\n",
       "         -3.6877565], dtype=float32),\n",
       "  (1, 'depressed')),\n",
       " (\"Today seems like a fine day. I can't wait to go to school today.\",\n",
       "  array([-3.9587896 , -4.2296805 , -6.3801646 , -0.16406982, -4.8696446 ,\n",
       "         -2.222822  ], dtype=float32),\n",
       "  (3, 'happy')),\n",
       " ('Wow your scrambled eggs are delicious!',\n",
       "  array([-3.9031124 , -4.7588186 , -5.7492213 , -0.19345228, -2.4745243 ,\n",
       "         -2.8176239 ], dtype=float32),\n",
       "  (3, 'happy')),\n",
       " ('I am so excited for Captain Marvel actually',\n",
       "  array([-2.2635143, -3.7895002, -3.0956917, -1.0930314, -1.230438 ,\n",
       "         -1.6054376], dtype=float32),\n",
       "  (3, 'happy'))]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}