NLP - text classification on top of language model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP - text classification on top of language model for the IMDb dataset\n",
    "\n",
    "**Language model: Given a few words of a sentence, build a model that allows you to predict what the next word is going to be.**\n",
    "\n",
    "Task: predict whether a review on IMDb is positive or negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.learner import *\n",
    "\n",
    "# Pytorch's NLP library:\n",
    "import torchtext\n",
    "from torchtext.datasets import language_modeling\n",
    "from torchtext import vocab, data\n",
    "\n",
    "from fastai.rnn_reg import *\n",
    "from fastai.rnn_train import *\n",
    "from fastai.nlp import *\n",
    "from fastai.lm_rnn import *\n",
    "\n",
    "import dill as pickle # Imported dill as pickle because pickle can't handle `TEXT.vocab` correctly!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan: pretrain a language model to predict the next word in a sequence of words and then fine tune the language model to classify sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset: IMDb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "README  imdb.vocab  imdbEr.txt  \u001b[0m\u001b[01;34mtest\u001b[0m/  \u001b[01;34mtrain\u001b[0m/\r\n"
     ]
    }
   ],
   "source": [
    "PATH = 'data/aclImdb/'\n",
    "\n",
    "TRN_PATH = 'train/all/'\n",
    "VAL_PATH = 'test/all/'\n",
    "TRN = f'{PATH}{TRN_PATH}'\n",
    "VAL = f'{PATH}{VAL_PATH}'\n",
    "\n",
    "%ls {PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0_0.txt', '0_3.txt', '0_9.txt', '10000_0.txt', '10000_4.txt']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_files = !ls {TRN}\n",
    "trn_files[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Bromwell High is a cartoon comedy. It ran at the same time as some other programs about school life, such as \"Teachers\". My 35 years in the teaching profession lead me to believe that Bromwell High\\'s satire is much closer to reality than is \"Teachers\". The scramble to survive financially, the insightful students who can see right through their pathetic teachers\\' pomp, the pettiness of the whole situation, all remind me of the schools I knew and their students. When I saw the episode in which a student repeatedly tried to burn down the school, I immediately recalled ......... at .......... High. A classic line: INSPECTOR: I\\'m here to sack one of your teachers. STUDENT: Welcome to Bromwell High. I expect that many adults of my age think that Bromwell High is far fetched. What a pity that it isn\\'t!']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review = !cat {TRN}{trn_files[2]}\n",
    "review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How mandy words are in the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17486270\r\n"
     ]
    }
   ],
   "source": [
    "!find {TRN} -name '*.txt' | xargs cat | wc -w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5686609\r\n"
     ]
    }
   ],
   "source": [
    "!find {VAL} -name '*.txt' | xargs cat | wc -w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "spacy_tok = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Bromwell High is a cartoon comedy . It ran at the same time as some other programs about school life , such as \" Teachers \" . My 35 years in the teaching profession lead me to believe that Bromwell High \\'s satire is much closer to reality than is \" Teachers \" . The scramble to survive financially , the insightful students who can see right through their pathetic teachers \\' pomp , the pettiness of the whole situation , all remind me of the schools I knew and their students . When I saw the episode in which a student repeatedly tried to burn down the school , I immediately recalled ......... at .......... High . A classic line : INSPECTOR : I \\'m here to sack one of your teachers . STUDENT : Welcome to Bromwell High . I expect that many adults of my age think that Bromwell High is far fetched . What a pity that it is n\\'t !'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([word.string.strip() for word in spacy_tok(review[0])])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice for example how 'isn\\'t' became 'is n\\'t' during tokenization!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Torchtext *field* definition describes how to preprocess a chunk of text\n",
    "1. convert to lowercase\n",
    "1. tokenize using spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = data.Field(lower=True, tokenize='spacy')  # Part of torchtext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. **bs** batch size, how many sequences of words are considered at the same time? Entire dataset will be split into *bs* batches.\n",
    "1. **bptt** (*backprop through time*) How long is each sequence of words in a minibatch? Number of layers the model will 'backprop' through.\n",
    "1. **min_freq** When converting the words to integers, call every word *unknown* that appears less than *min_freq* times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64\n",
    "bptt = 70"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ModelData` object fills the `TEXT` object with the crucial attribute `Text.vocab`. Stores which words/tokens have been found in text and the respective mapping to a unique integer value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.makedirs(f'{PATH}models', exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(TEXT, open(f'{PATH}models/TEXT.pk1', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4583, 37392, 20540756)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of batches, number of unique tokens, number of words\n",
    "len(md.trn_dl), md.nt, len(md.trn_ds[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the mapping from tokens to ints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<unk>', '<pad>', 'the', ',', '.', 'and', 'a', 'of', 'to', 'is']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# int-to-string, sorted according to frequency apart from first two\n",
    "TEXT.vocab.itos[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEXT.vocab.stoi['<unk>']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\"', 'planet', 'of', 'the', 'apes', '\"', 'an', 'awesome', ',', 'classic']"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.trn_ds[0].text[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable containing:\n",
       "   15\n",
       " 1359\n",
       "    7\n",
       "    2\n",
       " 5355\n",
       "   15\n",
       "   44\n",
       " 1186\n",
       "    3\n",
       "  387\n",
       "[torch.cuda.LongTensor of size 10x1 (GPU 0)]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEXT.numericalize([md.trn_ds[0].text[:10]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'planet'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEXT.vocab.itos[1359]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's take a look at the *data loader* of the *LanguageModelData* object\n",
    "Creates batches with bs=64 columns and bptt~80 rows. Note, that the second tensor the following command returns is shifted by 1. Reason: we aim to predict the next word. Each column contains many sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Variable containing:\n",
       "     15      3    229  ...      74     13    205\n",
       "   1359     24      9  ...    1866      9     62\n",
       "      7     12      6  ...       4     75      5\n",
       "         ...            ⋱           ...         \n",
       "     18   1512   1279  ...       8    136   4779\n",
       "   2494     10      3  ...     120     23   1413\n",
       "     13      2    589  ...      66     12     42\n",
       " [torch.cuda.LongTensor of size 68x64 (GPU 0)], Variable containing:\n",
       "   1359\n",
       "     24\n",
       "      9\n",
       "   ⋮   \n",
       "    941\n",
       "     35\n",
       "  15906\n",
       " [torch.cuda.LongTensor of size 4352 (GPU 0)])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(md.trn_dl))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each token gets its own embedding vector giving us an embedding matrix of 37392 x 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "em_sz = 200  # size of each embedding vector. Rule of thumb: between 50 and 600\n",
    "nh = 500     # number of hidden activations per layer\n",
    "nl = 3       # number of layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer and regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_fn = partial(optim.Adam, betas=(0.7, 0.99))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Researchs have found that the default amount of momentum for Adam (0.9) is too high for these models.\n",
    "\n",
    "Regularization through dropout according to [Merity et al. 2017](https://arxiv.org/pdf/1708.02182.pdf). Increase the dropout values when overfitting, decrease when underfitting..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner = md.get_model(opt_fn=opt_fn, emb_sz=em_sz, n_hid=nh, n_layers=nl,\n",
    "                      dropouti=0.05, dropout=0.05, wdrop=0.1, dropoute=0.02, dropouth=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regularization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient clipping:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.clip=0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee8539bda9fd4375b0c1cd86ce9011e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=15), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                \n",
      "    0      4.836045   4.717827  \n",
      "    1      4.637386   4.502882                                \n",
      "    2      4.523966   4.424905                                \n",
      "    3      4.565808   4.427167                                \n",
      "    4      4.46659    4.358337                                \n",
      "    5      4.41535    4.309937                                \n",
      "    6      4.352574   4.297426                                \n",
      "    7      4.488351   4.365648                                \n",
      "    8      4.450402   4.339508                                \n",
      "    9      4.395329   4.309739                                \n",
      "    10     4.371314   4.284916                                \n",
      "    11     4.3584     4.261108                                \n",
      "    12     4.319931   4.243292                                \n",
      "    13     4.297283   4.234079                                \n",
      "    14     4.265207   4.231984                                \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.23198])]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit(lrs=3e-3, n_cycle=4, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the encoder of the encoder-decoder model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save_encoder('enc1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.load_encoder('enc1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0e7a3ca4df04da2a9fe9477115c8f9b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss                                \n",
      "    0      4.510499   4.377362  \n",
      "    1      4.500967   4.384203                                \n",
      "    2      4.48165    4.361681                                \n",
      "    3      4.453777   4.339405                                \n",
      "    4      4.422646   4.316233                                \n",
      "    5      4.394123   4.293792                                \n",
      "    6      4.366147   4.272203                                \n",
      "    7      4.313716   4.256799                                \n",
      "    8      4.31329    4.249368                                \n",
      "    9      4.303805   4.247358                                \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([4.24736])]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.fit(lrs=3e-3, n_cycle=1, wds=1e-6, cycle_len=10)\n",
    "# wds weight decay parameter (L2 reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**At this point, we could train this further (we are not overfitting) if we actually planned to use the network. However, it takes quite some time..**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.save_encoder('enc2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "learner.load_encoder('enc2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy of language models is often measured using the metric *perplexity* (`exp()` of the loss)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69.9205781465884"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "math.exp(4.24736)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(TEXT, open(f'{PATH}models/TEXT.pk1', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = learner.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\". so , it was n't quite was i was expecting , but i really liked it anyway ! the best\""
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string = \"\"\". So, it wasn't quite was I was expecting, but I really liked it anyway! The best\"\"\"\n",
    "string_tok = [TEXT.preprocess(string)]\n",
    "' '.join(string_tok[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "string_num = TEXT.numericalize(string_tok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the batch size of the model to 1\n",
    "m[0].bs = 1\n",
    "# Turn off dropout for evaluation\n",
    "m.eval()\n",
    "# Reset the hidden state\n",
    "m.reset()\n",
    "# Get prediction from model\n",
    "res, *_ = m(string_num)\n",
    "# Reset the batchsize \n",
    "m[0].bs = bs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['part',\n",
       " 'thing',\n",
       " 'scene',\n",
       " 'way',\n",
       " 'parts',\n",
       " 'of',\n",
       " 'line',\n",
       " 'aspect',\n",
       " 'performance',\n",
       " 'scenes']"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next_words = torch.topk(res[-1], 10)[1]\n",
    "next_words\n",
    "[TEXT.vocab.itos[o] for o in next_words.data.cpu().numpy()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**All of these could be the next word of our test sentence!!**\n",
    "\n",
    "Let's let our model write some more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ". So, it wasn't quite was I was expecting, but I really liked it anyway! The best \n",
      "\n",
      "part of the movie . the movie is a bit of a mess , but it 's not a bad movie . it 's a very good movie , but it 's not a bad movie . it 's a good movie , but it 's not a good movie "
     ]
    }
   ],
   "source": [
    "print(string, \"\\n\")\n",
    "for i in range(50):\n",
    "    n=res[-1].topk(2)[1]\n",
    "    n = n[1] if n.data[0]==0 else n[0]\n",
    "    print(TEXT.vocab.itos[n.data[0]], end = ' ')\n",
    "    res, *_ = m(n[0].unsqueeze(0))  # Adds a dimension (for minibatch assumably)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, ahaha, the language model could use some more training probably but right now I don't want to invest the time.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment analysis\n",
    "### Fine tune the language model to do classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course we need the same vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = pickle.load(open(f'{PATH}models/TEXT.pk1', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sequential: tells torchtext to not tokenize as \n",
    "# we want to store only 'pos' or 'neg' \n",
    "IMDB_LABEL = data.Field(sequential=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use a dataset built into torchtext. Refer to the arxiv.ipynb to see how a torchtext dataset is defined.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading aclImdb_v1.tar.gz\n"
     ]
    }
   ],
   "source": [
    "splits = torchtext.datasets.IMDB.splits(text_field=TEXT, label_field=IMDB_LABEL, root='data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = splits[0].examples[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('pos',\n",
       " 'when you come across a gem of a movie like this , you realize why the \\' 80s were the greatest decade to live thru . the rock music ruled , & so did movies ... especially horror movies . filmmakers knew how to entertain us , & \" trick')"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.label, ' '.join(test.text[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a fastai ModelData object from torchtext splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "md2 = TextData.from_splits(PATH, splits=splits, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3 = md2.get_model(opt_fn, max_sl=1500, bptt=bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl, dropout=0.1, dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the pretrained *encoder*. We use differential learning rates as we are fine-tuning a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3.load_encoder(f'enc2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3.clip = 25.\n",
    "lrs = np.array([1e-4, 1e-4, 1e-4, 1e-3, 1e-2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88dc3cb206d9459dbce60a4af29fc8b1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.661577   0.487363   0.761779  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.48736]), 0.7617787532059408]"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m3.freeze_to(-1)\n",
    "m3.fit(lrs/2, 1, metrics=[accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "m3.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a59146b447f046bf966ecf68f1c709cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.464077   0.315161   0.876894  \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.31516]), 0.8768944070587855]"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m3.fit(lrs, 1, metrics=[accuracy], cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "67e4963a6dfb40caa570a595e6ab8e5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=14), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.425103   0.260042   0.897343  \n",
      "    1      0.391833   0.274921   0.89697                     \n",
      "    2      0.368948   0.286929   0.892561                    \n",
      "    3      0.357182   0.261986   0.901059                    \n",
      "    4      0.341577   0.265945   0.901818                    \n",
      "    5      0.350868   0.253552   0.905013                    \n",
      "    6      0.333761   0.255058   0.90734                     \n",
      "    7      0.327202   0.255533   0.908164                    \n",
      "    8      0.31469    0.287294   0.903437                    \n",
      "    9      0.296798   0.27291    0.906724                    \n",
      "    10     0.306673   0.27883    0.913195                    \n",
      "    11     0.283881   0.284334   0.907709                    \n",
      "    12     0.307085   0.252115   0.915737                    \n",
      "    13     0.276472   0.271925   0.91426                     \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([0.27192]), 0.9142602107326157]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m3.fit(lrs, 7, metrics=[accuracy], cycle_len=2, cycle_save_name='imdb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds, targs = m3.predict_with_targs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.argmax(preds, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = (preds==targs).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91888"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:deeplearning]",
   "language": "python",
   "name": "conda-env-deeplearning-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}