12_text_jcat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Preprocessing on the IMDb (Internet Movie Review Database)\n",
    "### The IMDB consists of 50,000 labeled reviews of movies (positive or negative) and 50,000 unlabelled ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from exp.nb_11a import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Define a class `TextList` that will directly read text from filenames\n",
    "### `TextList` is a subclass of `ItemList` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def read_file(fn): \n",
    "    with open(fn, 'r', encoding = 'utf8') as f: return f.read()\n",
    "    \n",
    "class TextList(ItemList):\n",
    "    @classmethod\n",
    "    def from_files(cls, path, extensions='.txt', recurse=True, include=None, **kwargs):\n",
    "        return cls(get_files(path, extensions, recurse=recurse, include=include), path, **kwargs)\n",
    "    \n",
    "    def get(self, i):\n",
    "        if isinstance(i, Path): return read_file(i)\n",
    "        return i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Import the IMDb Data directly into a `TextList`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=4964)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = datasets.untar_data(datasets.URLs.IMDB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/imdb.vocab'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/ld.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/ll_clas.pkl'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/README'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/tmp_clas'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/tmp_lm'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/train'),\n",
       " WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/unsup')]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just in case there are some text log files, we restrict the ones we take to the training, test, and unsupervised folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "il = TextList.from_files(path, include=['train', 'test', 'unsup'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. A little Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should expect a total of 100,000 texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(il.items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the first item in the list as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Once again Mr. Costner has dragged out a movie for far longer than necessary. Aside from the terrific sea rescue sequences, of which there are very few I just did not care about any of the characters. Most of us have ghosts in the closet, and Costner's character are realized early on, and then forgotten until much later, by which time I did not care. The character we should really care about is a very cocky, overconfident Ashton Kutcher. The problem is he comes off as kid who thinks he's better than anyone else around him and shows no signs of a cluttered closet. His only obstacle appears to be winning over Costner. Finally when we are well past the half way point of this stinker, Costner tells us all about Kutcher's ghosts. We are told why Kutcher is driven to be the best with no prior inkling or foreshadowing. No magic here, it was all I could do to keep from turning it off an hour in.\""
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt = il[0]\n",
    "txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s t r'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xx = 'str'\n",
    "yy = \" \".join(xx)\n",
    "yy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Once again Mr. Costner has dragged out a movie for far longer than necessary. Aside from the terrific sea rescue sequences, of which there are very few I just did not care about any of the characters. Most of us have ghosts in the closet, and Costner'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt[:250]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "900\n"
     ]
    }
   ],
   "source": [
    "# first text element in il (same as txt) has length of 900\n",
    "print(len(il[:100][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Tokenizing using the `spaCy` library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to tokenize the dataset first, which is splitting a sentence in individual tokens. Those tokens are the basic words or punctuation signs with a few tweaks: don't for instance is split between do and n't. We will use a processor for this, in conjunction with the [spaCy library](https://spacy.io/). spaCy describes itself as a purveyor of `Industrial Strength Natural Language Processing`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import spacy,html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=5070)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Helper functions\n",
    "Before and after tokenization, we will use helper functions to do a bit of processing. These helper functions manipulate text using the powerful syntax of `regular expressions` (sometimes abbreviated `regex`). Python's `re` library implements the language of `regular expressions`. On first encounter, the compact syntax of `re` can be a bit off-putting. It can be annoying, like trying to read a foreign language about which you have no clue. The brain becomes confused at its inability to parse as when reading words of its native language.\n",
    "\n",
    "What each of these helper functions do is made clear by the documentation. So on first pass, think of the `re` operations as black boxes implementing the documentation. For now you needn't worry about the details of regular expressions. \n",
    "\n",
    "At some point, you will eventually need to bite the bullet and understand `regular expressions` to go further in NLP. Here are a few helpful resources once you are ready to learn more:\n",
    "\n",
    "https://scotch.io/tutorials/an-introduction-to-regex-in-python\n",
    "\n",
    "https://www.w3schools.com/python/python_regex.asp\n",
    "\n",
    "http://marvin.cs.uidaho.edu/Handouts/regex.html\n",
    "\n",
    "http://flockhart.virtualave.net/RBIF0100/regexp.html\n",
    "\n",
    "https://jakevdp.github.io/WhirlwindTourOfPython/14-strings-and-regular-expressions.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1.1 Pre-proccessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before tokenizeing, we will apply a bit of preprocessing on the texts to clean them up (we saw the one up there had some HTML code). These rules are applied before we split the sentences in tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "#special tokens\n",
    "UNK, PAD, BOS, EOS, TK_REP, TK_WREP, TK_UP, TK_MAJ = \"xxunk xxpad xxbos xxeos xxrep xxwrep xxup xxmaj\".split()\n",
    "\n",
    "def sub_br(t):\n",
    "    \"Replaces the <br /> by \\n\"\n",
    "    re_br = re.compile(r'<\\s*br\\s*/?>', re.IGNORECASE)\n",
    "    return re_br.sub(\"\\n\", t)\n",
    "\n",
    "def spec_add_spaces(t):\n",
    "    \"Add spaces around / and #\"\n",
    "    return re.sub(r'([/#])', r' \\1 ', t)\n",
    "\n",
    "def rm_useless_spaces(t):\n",
    "    \"Remove multiple spaces\"\n",
    "    return re.sub(' {2,}', ' ', t)\n",
    "\n",
    "def replace_rep(t):\n",
    "    \"Replace repetitions at the character level: cccc -> TK_REP 4 c\"\n",
    "    def _replace_rep(m:Collection[str]) -> str:\n",
    "        c,cc = m.groups()\n",
    "        return f' {TK_REP} {len(cc)+1} {c} '\n",
    "    re_rep = re.compile(r'(\\S)(\\1{3,})')\n",
    "    return re_rep.sub(_replace_rep, t)\n",
    "    \n",
    "def replace_wrep(t):\n",
    "    \"Replace word repetitions: word word word -> TK_WREP 3 word\"\n",
    "    def _replace_wrep(m:Collection[str]) -> str:\n",
    "        c,cc = m.groups()\n",
    "        return f' {TK_WREP} {len(cc.split())+1} {c} '\n",
    "    re_wrep = re.compile(r'(\\b\\w+\\W+)(\\1{3,})')\n",
    "    return re_wrep.sub(_replace_wrep, t)\n",
    "\n",
    "def fixup_text(x):\n",
    "    \"Various messy things we've seen in documents\"\n",
    "    re1 = re.compile(r'  +')\n",
    "    x = x.replace('#39;', \"'\").replace('amp;', '&').replace('#146;', \"'\").replace(\n",
    "        'nbsp;', ' ').replace('#36;', '$').replace('\\\\n', \"\\n\").replace('quot;', \"'\").replace(\n",
    "        '<br />', \"\\n\").replace('\\\\\"', '\"').replace('<unk>',UNK).replace(' @.@ ','.').replace(\n",
    "        ' @-@ ','-').replace('\\\\', ' \\\\ ')\n",
    "    return re1.sub(' ', html.unescape(x))\n",
    "    \n",
    "default_pre_rules = [fixup_text, replace_rep, replace_wrep, spec_add_spaces, rm_useless_spaces, sub_br]\n",
    "default_spec_tok = [UNK, PAD, BOS, EOS, TK_REP, TK_WREP, TK_UP, TK_MAJ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' xxrep 4 c '"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace_rep('cccc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' xxwrep 5 word  '"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace_wrep('word word word word word ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1.2 Post-proccessing\n",
    "After tokenization we process the tokens to remove capitalization, but adding marker tokens to flag the start and end of the text and to preserve information about where the capitalization was in the original text.\n",
    "\n",
    "`TK_UP` indicates that the next token was originally in all caps\n",
    "\n",
    "`TK_MAJ` indicates that the next token was originally capitalized\n",
    "\n",
    "`BOS` indicates the beginning of a string\n",
    "\n",
    "`EOS` indicates the end of a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def replace_all_caps(x):\n",
    "    \"Replace tokens in ALL CAPS by their lower version and add `TK_UP` before, if length > 1\"\n",
    "    res = []\n",
    "    for t in x:\n",
    "        if t.isupper() and len(t) > 1: res.append(TK_UP); res.append(t.lower())\n",
    "        else: res.append(t)\n",
    "    return res\n",
    "\n",
    "def deal_caps(x):\n",
    "    \"Replace all Capitalized tokens by their lower version and add `TK_MAJ` before.\"\n",
    "    res = []\n",
    "    for t in x:\n",
    "        if t == '': continue\n",
    "        if t[0].isupper() and len(t) > 1 and t[1:].islower(): res.append(TK_MAJ)\n",
    "        res.append(t.lower())\n",
    "    return res\n",
    "\n",
    "\"Brackets each token with BOS and EOS tokens\"\n",
    "def add_eos_bos(x): return [BOS] + x + [EOS]\n",
    "\n",
    "default_post_rules = [deal_caps, replace_all_caps, add_eos_bos]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', 'xxup', 'am', 'C', 'xxup', 'shouting']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "replace_all_caps(['I', 'AM', 'C' ,'SHOUTING'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxmaj', 'my', 'xxmaj', 'name', 'is', 'xxmaj', 'jeremy']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deal_caps(['My', 'Name', 'is', 'Jeremy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Parallellizing the Tokenization Process\n",
    "Since tokenizing and applying those rules takes a bit of time, we'll parallelize it using `ProcessPoolExecutor` to go faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from spacy.symbols import ORTH\n",
    "from concurrent.futures import ProcessPoolExecutor\n",
    "\n",
    "def parallel(func, arr, max_workers=4):\n",
    "    # should specify what are the inputs func and arr?\n",
    "    if max_workers<2: results = list(progress_bar(map(func, enumerate(arr)), total=len(arr)))\n",
    "    else:\n",
    "        # use context manager to handle parallel processing case\n",
    "        with ProcessPoolExecutor(max_workers=max_workers) as ex:\n",
    "            return list(progress_bar(ex.map(func, enumerate(arr)), total=len(arr)))\n",
    "    if any([o is not None for o in results]): return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class TokenizeProcessor(Processor):\n",
    "    # initialize max_workers to 1, because Windows 10 won't allow max_workers > 1\n",
    "    #def __init__(self, lang=\"en\", chunksize=2000, pre_rules=None, post_rules=None, max_workers=4): \n",
    "    def __init__(self, lang=\"en\", chunksize=2000, pre_rules=None, post_rules=None, max_workers=1): \n",
    "        self.chunksize,self.max_workers = chunksize,max_workers\n",
    "        # using spacy's tokenizer\n",
    "        self.tokenizer = spacy.blank(lang).tokenizer\n",
    "        for w in default_spec_tok:\n",
    "            # dictionary of default_spec_tok\n",
    "            self.tokenizer.add_special_case(w, [{ORTH: w}])\n",
    "        self.pre_rules  = default_pre_rules  if pre_rules  is None else pre_rules\n",
    "        self.post_rules = default_post_rules if post_rules is None else post_rules\n",
    "\n",
    "    def proc_chunk(self, args):\n",
    "        # specify inputs: what are i and chunk?\n",
    "        i,chunk = args\n",
    "        \n",
    "        # pre-process\n",
    "        chunk = [compose(t, self.pre_rules) for t in chunk]\n",
    "        # what does .pipe do?\n",
    "        docs = [[d.text for d in doc] for doc in self.tokenizer.pipe(chunk)]\n",
    "        \n",
    "        # post-process\n",
    "        docs = [compose(t, self.post_rules) for t in docs]\n",
    "        return docs\n",
    "\n",
    "    def __call__(self, items): \n",
    "        toks = []\n",
    "        if isinstance(items[0], Path): items = [read_file(i) for i in items]\n",
    "        chunks = [items[i: i+self.chunksize] for i in (range(0, len(items), self.chunksize))]\n",
    "        toks = parallel(self.proc_chunk, chunks, max_workers=self.max_workers)\n",
    "        return sum(toks, [])\n",
    "    \n",
    "    def proc1(self, item): return self.proc_chunk([item])[0]\n",
    "    \n",
    "    # what do these deprocessing functions do?\n",
    "    def deprocess(self, toks): return [self.deproc1(tok) for tok in toks]\n",
    "    # this one inserts blank space between characters \n",
    "    def deproc1(self, tok):    return \" \".join(tok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 Instantiate the TokenizeProcessor() and explore the data a bit more"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "tp = TokenizeProcessor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "il_short = il[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'q', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'j', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', \"'\", 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'z', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', \"'\", 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', \"'\", 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'y', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'v', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'b', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', ',', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'w', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 's', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'c', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'l', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'd', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'k', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'e', 'xxeos'],\n",
       " ['xxbos', 'p', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'm', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', 'g', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 't', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', 'f', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'a', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'h', 'xxeos'],\n",
       " ['xxbos', 'o', 'xxeos'],\n",
       " ['xxbos', 'u', 'xxeos'],\n",
       " ['xxbos', 'r', 'xxeos'],\n",
       " ['xxbos', ' ', 'xxeos'],\n",
       " ['xxbos', 'i', 'xxeos'],\n",
       " ['xxbos', 'n', 'xxeos'],\n",
       " ['xxbos', '.', 'xxeos']]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# hmmmm.... this is weird What's going on here?\n",
    "#      Makes a list with an item for each of the 900 characters in the text.\n",
    "#           Each item is a list with 3 elements: \n",
    "#                the character, bracketed by 'xxbos' and 'xxeos'\n",
    "#                so xxbos and xxeos flag the beginning and end of each item\n",
    "# Aha! This is the result of add_eos_bos(x), the last function in the post-processing default_post_rules!\n",
    "tp(il[:100][0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "207\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['xxbos',\n",
       " 'xxmaj',\n",
       " 'once',\n",
       " 'again',\n",
       " 'xxmaj',\n",
       " 'mr.',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " 'has',\n",
       " 'dragged',\n",
       " 'out',\n",
       " 'a',\n",
       " 'movie',\n",
       " 'for',\n",
       " 'far',\n",
       " 'longer',\n",
       " 'than',\n",
       " 'necessary',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'aside',\n",
       " 'from',\n",
       " 'the',\n",
       " 'terrific',\n",
       " 'sea',\n",
       " 'rescue',\n",
       " 'sequences',\n",
       " ',',\n",
       " 'of',\n",
       " 'which',\n",
       " 'there',\n",
       " 'are',\n",
       " 'very',\n",
       " 'few',\n",
       " 'i',\n",
       " 'just',\n",
       " 'did',\n",
       " 'not',\n",
       " 'care',\n",
       " 'about',\n",
       " 'any',\n",
       " 'of',\n",
       " 'the',\n",
       " 'characters',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'most',\n",
       " 'of',\n",
       " 'us',\n",
       " 'have',\n",
       " 'ghosts',\n",
       " 'in',\n",
       " 'the',\n",
       " 'closet',\n",
       " ',',\n",
       " 'and',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " \"'s\",\n",
       " 'character',\n",
       " 'are',\n",
       " 'realized',\n",
       " 'early',\n",
       " 'on',\n",
       " ',',\n",
       " 'and',\n",
       " 'then',\n",
       " 'forgotten',\n",
       " 'until',\n",
       " 'much',\n",
       " 'later',\n",
       " ',',\n",
       " 'by',\n",
       " 'which',\n",
       " 'time',\n",
       " 'i',\n",
       " 'did',\n",
       " 'not',\n",
       " 'care',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'the',\n",
       " 'character',\n",
       " 'we',\n",
       " 'should',\n",
       " 'really',\n",
       " 'care',\n",
       " 'about',\n",
       " 'is',\n",
       " 'a',\n",
       " 'very',\n",
       " 'cocky',\n",
       " ',',\n",
       " 'overconfident',\n",
       " 'xxmaj',\n",
       " 'ashton',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'the',\n",
       " 'problem',\n",
       " 'is',\n",
       " 'he',\n",
       " 'comes',\n",
       " 'off',\n",
       " 'as',\n",
       " 'kid',\n",
       " 'who',\n",
       " 'thinks',\n",
       " 'he',\n",
       " \"'s\",\n",
       " 'better',\n",
       " 'than',\n",
       " 'anyone',\n",
       " 'else',\n",
       " 'around',\n",
       " 'him',\n",
       " 'and',\n",
       " 'shows',\n",
       " 'no',\n",
       " 'signs',\n",
       " 'of',\n",
       " 'a',\n",
       " 'cluttered',\n",
       " 'closet',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'his',\n",
       " 'only',\n",
       " 'obstacle',\n",
       " 'appears',\n",
       " 'to',\n",
       " 'be',\n",
       " 'winning',\n",
       " 'over',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'finally',\n",
       " 'when',\n",
       " 'we',\n",
       " 'are',\n",
       " 'well',\n",
       " 'past',\n",
       " 'the',\n",
       " 'half',\n",
       " 'way',\n",
       " 'point',\n",
       " 'of',\n",
       " 'this',\n",
       " 'stinker',\n",
       " ',',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " 'tells',\n",
       " 'us',\n",
       " 'all',\n",
       " 'about',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " \"'s\",\n",
       " 'ghosts',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'we',\n",
       " 'are',\n",
       " 'told',\n",
       " 'why',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " 'is',\n",
       " 'driven',\n",
       " 'to',\n",
       " 'be',\n",
       " 'the',\n",
       " 'best',\n",
       " 'with',\n",
       " 'no',\n",
       " 'prior',\n",
       " 'inkling',\n",
       " 'or',\n",
       " 'foreshadowing',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'no',\n",
       " 'magic',\n",
       " 'here',\n",
       " ',',\n",
       " 'it',\n",
       " 'was',\n",
       " 'all',\n",
       " 'i',\n",
       " 'could',\n",
       " 'do',\n",
       " 'to',\n",
       " 'keep',\n",
       " 'from',\n",
       " 'turning',\n",
       " 'it',\n",
       " 'off',\n",
       " 'an',\n",
       " 'hour',\n",
       " 'in',\n",
       " '.',\n",
       " 'xxeos']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the 900 characters of the first text item are mapped into 207 tokens, including words, special tokens and punctuation\n",
    "#      note that the beginning and end tokens are 'xxbos' and 'xxeos'\n",
    "print(len(tp(il[:100])[0]))\n",
    "tp(il[:100])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "207\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "['xxbos',\n",
       " 'xxmaj',\n",
       " 'once',\n",
       " 'again',\n",
       " 'xxmaj',\n",
       " 'mr.',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " 'has',\n",
       " 'dragged',\n",
       " 'out',\n",
       " 'a',\n",
       " 'movie',\n",
       " 'for',\n",
       " 'far',\n",
       " 'longer',\n",
       " 'than',\n",
       " 'necessary',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'aside',\n",
       " 'from',\n",
       " 'the',\n",
       " 'terrific',\n",
       " 'sea',\n",
       " 'rescue',\n",
       " 'sequences',\n",
       " ',',\n",
       " 'of',\n",
       " 'which',\n",
       " 'there',\n",
       " 'are',\n",
       " 'very',\n",
       " 'few',\n",
       " 'i',\n",
       " 'just',\n",
       " 'did',\n",
       " 'not',\n",
       " 'care',\n",
       " 'about',\n",
       " 'any',\n",
       " 'of',\n",
       " 'the',\n",
       " 'characters',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'most',\n",
       " 'of',\n",
       " 'us',\n",
       " 'have',\n",
       " 'ghosts',\n",
       " 'in',\n",
       " 'the',\n",
       " 'closet',\n",
       " ',',\n",
       " 'and',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " \"'s\",\n",
       " 'character',\n",
       " 'are',\n",
       " 'realized',\n",
       " 'early',\n",
       " 'on',\n",
       " ',',\n",
       " 'and',\n",
       " 'then',\n",
       " 'forgotten',\n",
       " 'until',\n",
       " 'much',\n",
       " 'later',\n",
       " ',',\n",
       " 'by',\n",
       " 'which',\n",
       " 'time',\n",
       " 'i',\n",
       " 'did',\n",
       " 'not',\n",
       " 'care',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'the',\n",
       " 'character',\n",
       " 'we',\n",
       " 'should',\n",
       " 'really',\n",
       " 'care',\n",
       " 'about',\n",
       " 'is',\n",
       " 'a',\n",
       " 'very',\n",
       " 'cocky',\n",
       " ',',\n",
       " 'overconfident',\n",
       " 'xxmaj',\n",
       " 'ashton',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'the',\n",
       " 'problem',\n",
       " 'is',\n",
       " 'he',\n",
       " 'comes',\n",
       " 'off',\n",
       " 'as',\n",
       " 'kid',\n",
       " 'who',\n",
       " 'thinks',\n",
       " 'he',\n",
       " \"'s\",\n",
       " 'better',\n",
       " 'than',\n",
       " 'anyone',\n",
       " 'else',\n",
       " 'around',\n",
       " 'him',\n",
       " 'and',\n",
       " 'shows',\n",
       " 'no',\n",
       " 'signs',\n",
       " 'of',\n",
       " 'a',\n",
       " 'cluttered',\n",
       " 'closet',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'his',\n",
       " 'only',\n",
       " 'obstacle',\n",
       " 'appears',\n",
       " 'to',\n",
       " 'be',\n",
       " 'winning',\n",
       " 'over',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'finally',\n",
       " 'when',\n",
       " 'we',\n",
       " 'are',\n",
       " 'well',\n",
       " 'past',\n",
       " 'the',\n",
       " 'half',\n",
       " 'way',\n",
       " 'point',\n",
       " 'of',\n",
       " 'this',\n",
       " 'stinker',\n",
       " ',',\n",
       " 'xxmaj',\n",
       " 'costner',\n",
       " 'tells',\n",
       " 'us',\n",
       " 'all',\n",
       " 'about',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " \"'s\",\n",
       " 'ghosts',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'we',\n",
       " 'are',\n",
       " 'told',\n",
       " 'why',\n",
       " 'xxmaj',\n",
       " 'kutcher',\n",
       " 'is',\n",
       " 'driven',\n",
       " 'to',\n",
       " 'be',\n",
       " 'the',\n",
       " 'best',\n",
       " 'with',\n",
       " 'no',\n",
       " 'prior',\n",
       " 'inkling',\n",
       " 'or',\n",
       " 'foreshadowing',\n",
       " '.',\n",
       " 'xxmaj',\n",
       " 'no',\n",
       " 'magic',\n",
       " 'here',\n",
       " ',',\n",
       " 'it',\n",
       " 'was',\n",
       " 'all',\n",
       " 'i',\n",
       " 'could',\n",
       " 'do',\n",
       " 'to',\n",
       " 'keep',\n",
       " 'from',\n",
       " 'turning',\n",
       " 'it',\n",
       " 'off',\n",
       " 'an',\n",
       " 'hour',\n",
       " 'in',\n",
       " '.',\n",
       " 'xxeos']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(tp(il[:100])[0]))\n",
    "tp(il[:100])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'xxbos • xxmaj • once • again • xxmaj • mr. • xxmaj • costner • has • dragged • out • a • movie • for • far • longer • than • necessary • . • xxmaj • aside • from • the • terrific • sea • rescue • sequences • , • of • which • there • are • very • few • i • just • did • not • care • about • any • of • the • characters • . • xxmaj • most • of • us • have • ghosts • in • the • closet • , • and • xxmaj'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' • '.join(tp(il[:100])[0])[:400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'xxbos • xxmaj • once • again • xxmaj • mr. • xxmaj • costner • has • dragged • out • a • movie • for • far • longer • than • necessary • . • xxmaj • aside • from • the • terrific • sea • rescue • sequences • , • of • which • there • are • very • few • i • just • did • not • care • about • any • of • the • characters • . • xxmaj • most • of • us • have • ghosts • in • the • closet • , • and • xxmaj'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' • '.join(tp(il[:100])[0])[:400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'xxbos • xxmaj • once • again • xxmaj • mr. • xxmaj • costner • has • dragged • out • a • movie • for • far • longer • than • necessary • . • xxmaj • aside • from • the • terrific • sea • rescue • sequences • , • of • which • there • are • very • few • i • just • did • not • care • about • any • of • the • characters • . • xxmaj • most • of • us • have • ghosts • in • the • closet • , • and • xxmaj'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' • '.join(tp(il[:100])[0])[:400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "1451"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(' • '.join(tp(il[:100])[0])[:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Numericalizing the text data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have tokenized our texts, we replace each token by an individual number, this is called numericalizing. Again, we do this with a processor (not so different from the `CategoryProcessor`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=5491)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import collections\n",
    "\n",
    "class NumericalizeProcessor(Processor):\n",
    "    def __init__(self, vocab=None, max_vocab=60000, min_freq=2): \n",
    "        self.vocab,self.max_vocab,self.min_freq = vocab,max_vocab,min_freq\n",
    "    \n",
    "    def __call__(self, items):\n",
    "        #The vocab is defined on the first use.\n",
    "        if self.vocab is None:\n",
    "            freq = Counter(p for o in items for p in o)\n",
    "            # include a word only if it occurs more than self.min_freq times in the text\n",
    "            self.vocab = [o for o,c in freq.most_common(self.max_vocab) if c >= self.min_freq]\n",
    "            for o in reversed(default_spec_tok):\n",
    "                if o in self.vocab: self.vocab.remove(o)\n",
    "                self.vocab.insert(0, o)\n",
    "        if getattr(self, 'otoi', None) is None:\n",
    "            self.otoi = collections.defaultdict(int,{v:k for k,v in enumerate(self.vocab)}) \n",
    "        return [self.proc1(o) for o in items]\n",
    "    def proc1(self, item):  return [self.otoi[o] for o in item]\n",
    "    \n",
    "    def deprocess(self, idxs):\n",
    "        assert self.vocab is not None\n",
    "        return [self.deproc1(idx) for idx in idxs]\n",
    "    def deproc1(self, idx): return [self.vocab[i] for i in idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Splitting the data into training and validation sets\n",
    "For text classification, we will split by the grand parent folder as before, but for language modeling, we take all the texts and just put 10% aside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "sd = SplitData.split_by_func(il, partial(random_splitter, p_valid=0.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SplitData\n",
       "Train: TextList (90052 items)\n",
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/0_2.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10000_4.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10002_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10003_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10004_2.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10005_2.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10006_2.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10007_4.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10009_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/1000_3.txt')...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       "Valid: TextList (9948 items)\n",
       "[WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10001_1.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10008_4.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10013_4.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10028_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10062_4.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10068_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10078_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10095_3.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/10098_1.txt'), WindowsPath('C:/Users/cross-entropy/.fastai/data/imdb/test/neg/100_4.txt')...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Labeling\n",
    "When we do language modeling, we will infer the labels from the text during training, so there's no need to label. The training loop expects labels however, so we need to add dummy ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# proc_tok,proc_num = TokenizeProcessor(max_workers=8),NumericalizeProcessor()\n",
    "proc_tok,proc_num = TokenizeProcessor(max_workers=1),NumericalizeProcessor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='46' class='' max='46', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [46/46 09:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='5' class='' max='5', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [5/5 01:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 10min 43s\n"
     ]
    }
   ],
   "source": [
    "%time ll = label_by_func(sd, lambda x: 0, proc_x = [proc_tok,proc_num])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the items have been processed they will become list of numbers. We can still access the underlying raw data in `x_obj` for the text and `y_obj` for the targets (which in this case are all dummies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method LabeledData.x_obj of LabeledData\n",
       "x: TextList (90052 items)\n",
       "[[2, 7, 301, 193, 7, 611, 7, 5508, 61, 3423, 60, 12, 29, 28, 247, 1140, 93, 1727, 9, 7, 1202, 51, 8, 1362, 1598, 2092, 834, 10, 13, 79, 54, 38, 70, 189, 18, 56, 87, 37, 480, 59, 120, 13, 8, 121, 9, 7, 110, 13, 201, 41, 2952, 17, 8, 4891, 10, 11, 7, 5508, 22, 123, 38, 1722, 435, 34, 10, 11, 115, 1536, 382, 94, 330, 10, 47, 79, 75, 18, 87, 37, 480, 9, 7, 8, 123, 90, 155, 83, 480, 59, 15, 12, 70, 8765, 10, 37775, 7, 8442, 7, 11059, 9, 7, 8, 460, 15, 39, 292, 143, 26, 538, 49, 1212, 39, 22, 146, 93, 273, 345, 208, 108, 11, 293, 74, 3688, 13, 12, 16726, 4891, 9, 7, 40, 82, 13583, 737, 14, 42, 1770, 142, 7, 5508, 9, 7, 449, 68, 90, 38, 88, 518, 8, 331, 116, 243, 13, 19, 3962, 10, 7, 5508, 715, 201, 44, 59, 7, 11059, 22, 2952, 9, 7, 90, 38, 590, 154, 7, 11059, 15, 2122, 14, 42, 8, 139, 27, 74, 2676, 17143, 55, 12061, 9, 7, 74, 1326, 149, 10, 16, 25, 44, 18, 95, 58, 14, 404, 51, 1618, 16, 143, 48, 564, 17, 9, 3], [2, 7, 19, 15, 48, 514, 13, 154, 8, 2136, 13, 213, 124, 38, 8, 186, 9, 7, 3732, 11, 364, 10, 54, 22, 83, 180, 296, 168, 149, 9, 12, 622, 474, 13, 8, 115, 1180, 23, 15760, 1988, 13, 7, 1742, 23, 2315, 11, 7, 1742, 7, 6027, 10, 49, 161, 274, 14248, 128, 232, 142, 20, 46, 38, 2120, 13, 137, 10, 11, 137, 88, 9, 7, 58, 35, 1343, 27, 19, 43, 10, 159, 84, 7, 183, 7, 689, 7, 512, 10, 7, 32130, 55, 126, 7, 183, 7, 791, 7, 5298, 28, 7, 1742, 23, 2315, 10, 55, 7, 15085, 2805, 8, 7, 2560, 10, 7, 2029, 7, 2869, 55, 7, 2139, 28, 7, 1742, 7, 6027, 11, 84, 8, 164, 831, 9, 7, 1742, 23, 2315, 22, 2271, 7540, 429, 642, 178, 19, 31, 17144, 45, 8, 2811, 10, 11, 18, 167, 152, 1555, 64, 8, 2088, 7, 960, 7, 7960, 25, 422, 17, 19, 31, 66, 7, 11, 154, 8, 2088, 91, 39, 234, 317, 8, 2449, 186, 123, 66, 7, 51, 7, 1704, 19628, 10, 191, 31, 18, 161, 131, 27, 7, 960, 7, 7960, 61, 108, 419, 8, 2449, 186, 2191, 123, 10, 11, 45, 241, 17, 7, 1704, 40, 123, 1095, 10, 79, 113, 16, 666, 16520, 92, 24, 7, 462, 10, 19, 15, 349, 23, 1001, 213, 1225, 9, 7, 54, 38, 3539, 146, 124, 14, 84, 10, 11, 63, 32, 83, 205, 14, 84, 19, 43, 10, 126, 7, 8816, 7, 329, 10, 79, 15, 2357, 12, 11291, 990, 30, 61, 146, 137, 11, 12, 146, 246, 9, 7, 8, 82, 169, 20, 113, 19, 45, 44, 296, 168, 25, 12, 548, 519, 34, 8, 377, 23, 8, 672, 25, 242, 2361, 10, 79, 292, 517, 14, 253, 72, 28, 8, 508, 31, 413, 23, 30, 37, 203, 9, 222, 125, 184, 9, 3], [2, 7, 37, 76, 8, 7, 4777, 95, 940, 770, 312, 451, 10, 11, 284, 7, 2696, 7, 2096, 15, 74, 24409, 23, 367, 39, 22, 349, 14, 619, 68, 16, 292, 14, 217, 2791, 213, 118, 9, 7, 8, 8519, 409, 11, 1085, 36128, 86, 2459, 17, 239, 11, 31, 10, 8, 17594, 13, 8, 7, 11370, 670, 29, 341, 25, 17, 388, 6495, 10, 8, 137, 573, 12, 162, 2024, 14, 274, 145, 22, 16309, 14509, 1218, 11, 9092, 137, 9, 7, 19, 25, 43, 13, 8, 128, 1742, 23, 2315, 118, 18, 235, 26, 12, 538, 11, 481, 10, 82, 14, 126, 111, 330, 11, 3322, 9, 7, 960, 7, 7960, 11, 7, 986, 7, 20961, 38, 16310, 27, 1147, 475, 382, 12, 3250, 1257, 9736, 59, 14, 159, 72, 17, 7889, 926, 142, 12, 5168, 27, 1642, 9298, 9, 18, 519, 16, 14, 7, 2696, 28, 949, 11, 6694, 937, 72, 8, 304, 121, 11, 1522, 9, 7, 30, 18, 2041, 312, 585, 28, 1618, 60, 7, 873, 23, 1440, 376, 9, 7, 1742, 23, 2315, 11, 6027, 224, 41, 98, 802, 897, 45, 19, 75, 10, 11, 153, 18, 161, 523, 218, 80, 3736, 26, 13369, 10, 17, 77, 677, 46, 1616, 1081, 17, 19, 29, 9, 7, 16, 22, 59, 7361, 249, 13, 43, 230, 3733, 1618, 40, 162, 34, 8, 102, 230, 14, 8, 243, 32, 187, 668, 3130, 17, 2587, 1581, 13, 2902, 9, 7, 166, 19, 15, 12, 29, 10, 112, 37, 12, 633, 52, 18, 491, 1164, 474, 77, 75, 17803, 44, 8, 394, 130, 1290, 17, 19, 29, 10, 30, 54, 86, 128, 10, 11, 46, 1628, 1276, 9, 18, 207, 8, 577, 168, 19, 20, 312, 34, 287, 25, 0, 13, 1458, 11, 56, 419, 200, 143, 8, 24410, 9, 7, 54, 38, 127, 200, 18, 152, 389, 59, 16, 10, 43, 2117, 12, 150, 27, 12, 11967, 11, 8, 102, 15, 7, 20961, 22, 674, 3712, 7335, 9, 7, 1356, 368, 19, 29, 15, 53, 9145, 22, 6998, 9, 7, 1156, 1907, 1876, 16, 63, 18, 167, 4833, 11, 18, 58, 35, 250, 53, 6797, 10, 7, 30, 18, 167, 88, 1864, 16, 4686, 53, 629, 9, 379, 426, 10, 19342, 9, 3], [2, 7, 8443, 1310, 36, 118, 15, 37, 12, 3770, 686, 28, 111, 33, 83, 38, 666, 22121, 9, 7, 80, 9407, 1105, 2337, 38, 2773, 13, 3255, 325, 744, 264, 3679, 9, 7, 30, 490, 7, 8443, 1310, 38, 822, 124, 27, 8, 11770, 13, 1694, 14, 2765, 796, 28, 142, 127, 621, 50, 7, 17, 19, 46, 1989, 3379, 10, 80, 9907, 10, 30, 269, 1810, 11, 21719, 93, 9093, 10, 2550, 2337, 698, 25547, 9, 24, 7, 44702, 62, 3712, 15, 12, 1007, 13, 12, 31, 47, 7, 18284, 7, 9094, 27, 8, 186, 439, 11, 7, 26823, 7, 20962, 11, 7, 7025, 7, 8567, 17, 8, 473, 9, 7, 8, 233, 715, 12, 81, 13, 4819, 133, 11, 3887, 17, 11, 208, 7, 6496, 316, 8, 7, 1040, 1606, 13, 6207, 9, 7, 8443, 1657, 8, 213, 51, 8, 4820, 104, 8, 3626, 1090, 10, 6826, 14, 42, 2449, 10, 52, 54, 38, 7, 15761, 48192, 10, 348, 17, 332, 7514, 10, 7, 1045, 7178, 55, 8, 19343, 15762, 13, 8, 23942, 9, 7, 30, 16, 15, 56, 1822, 4810, 10, 8, 5230, 1959, 15, 356, 20209, 9, 24, 7, 2109, 7, 0, 310, 8, 1611, 12062, 268, 49, 747, 28, 8, 8862, 8639, 230, 49, 234, 1418, 34, 117, 94, 10623, 9, 7, 71, 15, 48, 1611, 10, 9247, 10, 88, 4007, 7, 1040, 560, 11, 708, 725, 8, 844, 9, 7, 57, 1887, 2003, 13, 2898, 2916, 36, 58551, 3181, 10, 10274, 10, 2870, 10, 1127, 10, 731, 92, 11, 13102, 33, 38, 8, 139, 313, 14, 126, 19, 453, 11, 296, 127, 426, 9, 7, 71, 14794, 19, 676, 4443, 541, 27, 48, 4788, 1142, 13, 3997, 9, 18, 658, 65, 83, 67, 524, 238, 361, 28, 57, 9, 7, 71, 83, 1031, 16, 9, 3], [2, 12, 172, 169, 578, 14, 89, 153, 168, 21, 7, 22948, 21, 96, 34, 8, 43, 519, 10, 8, 576, 15, 12, 5066, 23, 5041, 11, 8, 170, 15, 483, 2089, 14, 114, 201, 410, 154, 39, 91, 64, 39, 91, 36, 22552, 14249, 92, 5738, 10, 1411, 89, 10, 4911, 33, 163, 40, 1220, 9, 7, 34, 8, 102, 519, 10, 8, 7, 646, 353, 45, 241, 15, 70, 947, 2364, 9, 7, 52, 18, 275, 558, 4764, 54, 85, 98, 218, 69, 11, 360, 429, 45, 8, 186, 75, 50, 7, 19, 31, 15, 394, 36, 172, 109, 19, 230, 61, 5001, 14, 191, 7845, 11, 20963, 17, 40, 500, 33, 11, 8725, 36, 135, 73, 90, 42, 17, 12, 1951, 44703, 229, 279, 2255, 2279, 150, 66, 33, 9, 7, 37, 14, 762, 8, 21, 752, 21, 1266, 36, 461, 10, 109, 0, 92, 7, 693, 16, 9, 36, 194, 33, 3], [2, 7, 19, 7, 1045, 202, 31, 61, 14, 42, 43, 13, 8, 12242, 18, 41, 131, 9, 24, 18, 25, 37, 1864, 13, 120, 1978, 225, 520, 2502, 11, 16727, 10, 30, 19, 15, 457, 452, 713, 12, 315, 123, 9, 24, 7, 279, 576, 15, 5066, 11, 5041, 26, 12, 934, 13, 2724, 15086, 45, 8, 926, 13, 40, 354, 9, 39, 103, 61, 12, 3437, 9146, 10, 30, 18, 78, 37, 855, 60, 135, 20, 409, 51, 9, 7, 40, 1129, 23, 3010, 187, 60, 11, 9502, 108, 2047, 9, 24, 7, 316, 8, 272, 12, 3689, 23, 1432, 12243, 10, 11, 45, 329, 39, 2033, 104, 36129, 11, 3438, 5826, 11, 5095, 8, 556, 13, 352, 546, 9, 7, 46, 38, 44, 1611, 10, 13, 285, 10, 345, 90, 73, 35, 480, 59, 8, 211, 20, 39, 639, 1698, 80, 4088, 204, 14, 8, 10485, 9, 7, 39, 16311, 869, 14, 177, 816, 10, 11, 20, 15, 64, 240, 108, 1034, 9, 24, 7, 53, 18, 324, 10, 12, 70, 669, 29, 20, 15, 479, 11, 70, 579, 26, 7, 9971, 7, 52613, 132, 2214, 11, 56, 2334, 40, 75, 2574, 556, 9, 3], [2, 7, 129, 12, 216, 23, 75, 355, 13, 7, 985, 31, 10, 18, 863, 69, 93, 19, 9, 18, 197, 35, 83, 42, 2590, 14, 940, 14, 94, 10, 26, 19, 29, 15, 56, 52, 358, 9, 7, 8, 81, 251, 42, 8, 16521, 752, 138, 158, 144, 10, 2293, 18, 95, 35, 776, 8, 399, 137, 10, 8, 950, 46, 456, 1804, 10, 11, 8, 1242, 21, 2584, 21, 7, 985, 81, 9, 7, 63, 32, 161, 1966, 109, 128, 7, 985, 118, 381, 121, 10, 1668, 11, 1290, 20, 327, 117, 21, 286, 21, 10, 0, 52, 10, 115, 5866, 734, 13, 19, 29, 9, 7, 638, 10, 12, 15087, 176, 95, 41, 590, 32, 109, 19, 29, 25, 182, 14, 828, 361, 10, 11, 20, 22, 37, 12, 67, 169, 17, 77, 309, 9, 24, 7, 467, 13, 21, 7, 1469, 21, 7, 13103, 96, 40, 192, 17, 19, 29, 15, 37, 83, 69, 93, 12, 2042, 10, 11, 919, 32, 199, 12, 8367, 355, 10, 32, 58, 35, 387, 14, 2709, 163, 19, 474, 13, 31, 9, 24, 263, 125, 184, 3], [2, 21, 7, 6568, 7, 542, 21, 715, 13, 12, 3670, 343, 176, 7, 985, 255, 49, 747, 17, 53, 27, 12, 145, 129, 7243, 47, 57, 219, 18797, 49, 15, 12, 859, 9, 7, 19, 873, 510, 15, 59, 1337, 1375, 5272, 11, 1337, 1375, 710, 10, 710, 10, 11, 69, 710, 9, 7, 32, 259, 99, 14, 84, 8, 360, 93, 4350, 196, 13, 297, 26, 46, 710, 34, 8, 2535, 10, 710, 11, 317, 398, 1477, 10, 710, 11, 99, 12, 9737, 10, 710, 11, 1249, 11, 1249, 11, 710, 10, 710, 34, 2753, 7179, 10, 2875, 60, 11, 710, 10, 530, 9, 26, 32, 373, 2260, 1065, 28, 158, 14, 601, 9, 7, 8, 1585, 41984, 13, 12, 81, 15, 37, 6968, 14, 1422, 12, 31, 27, 380, 148, 392, 1068, 10, 12, 11869, 196, 10, 11, 74, 213, 10, 74, 888, 10, 74, 369, 55, 1028, 10, 74, 1252, 463, 92, 56, 11870, 0, 9, 36, 11968, 33, 3], [2, 7, 2827, 310, 12, 1930, 11, 240, 325, 9908, 2444, 9, 7, 2450, 7, 11522, 7, 177, 61, 14, 58, 12, 133, 131, 27, 8, 110, 7721, 11, 4482, 450, 13, 10486, 2130, 17, 7, 359, 9, 18, 1223, 24411, 153, 168, 19, 31, 9, 7, 13, 285, 18, 85, 8, 15088, 10, 30, 20, 91, 37, 408, 19, 31, 87, 37, 6141, 14, 8, 6543, 17, 8, 0, 9, 24, 7, 154, 197, 35, 7, 1852, 7, 2827, 317, 158, 39, 78, 83, 1485, 143, 53, 12, 97, 302, 9, 7, 1852, 7, 2827, 49, 42, 547, 17, 12, 231, 26, 12, 97, 302, 9, 7, 2088, 1274, 95, 58, 16, 146, 9, 24, 7, 8, 1631, 786, 28, 7, 1852, 22, 4274, 27, 7, 14636, 55, 25, 16, 7, 3571, 7, 20964, 51, 21, 7, 329, 7, 1138, 7, 14, 7, 2104, 21, 9, 3], [2, 7, 107, 13, 44, 10, 18, 73, 53, 14, 156, 20, 18, 257, 12, 355, 13, 44, 13, 8, 171, 20, 977, 17, 19, 31, 11, 45, 8, 75, 20, 18, 1573, 16, 10, 18, 487, 14, 53, 16, 9, 24, 18, 122, 20, 8, 304, 313, 20, 18, 25, 52, 683, 25, 20, 8, 984, 884, 4930, 89, 12, 792, 704, 9, 7, 17, 77, 542, 10, 12, 792, 704, 28, 7, 711, 118, 15, 53, 158, 60, 13, 12, 7, 3416, 7, 37776, 696, 10, 158, 20, 61, 12, 188, 13, 479, 989, 11, 526, 11, 887, 8, 522, 27, 48, 291, 20, 15, 2713, 14, 42, 1536, 6676, 539, 9, 24, 7, 19, 29, 687, 60, 27, 8, 2385, 877, 13, 129, 160, 12, 31, 9, 7, 90, 41, 279, 304, 123, 10, 20, 3283, 12, 145, 20, 39, 91, 37, 53, 10, 13, 129, 585, 17, 12, 617, 11, 502, 20, 527, 8, 587, 13, 43, 13, 40, 7961, 9, 7, 40, 8817, 1865, 14, 42, 227, 10, 30, 8, 337, 20, 40, 334, 251, 42, 585, 10, 91, 37, 3508, 14, 108, 382, 20, 71, 10348, 14, 108, 20, 71, 25, 12, 192, 13, 8, 780, 9, 24, 7, 8, 784, 13, 12, 67, 792, 704, 86, 17, 288, 10, 45, 19, 243, 10, 30, 51, 54, 10, 18, 469, 20, 8, 31, 573, 12, 286, 471, 11, 901, 242, 65, 454, 13, 12, 3689, 1965, 1471, 59, 49, 511, 14, 42, 27, 49, 11, 64, 8, 133, 13, 12, 164, 634, 15, 9, 7, 8, 31, 251, 41, 98, 758, 14, 89, 10, 63, 8, 984, 884, 85, 3409, 13, 12, 2309, 1394, 22, 4660, 11, 85, 37, 98, 8399, 26, 792, 704, 9, 7, 19, 484, 14, 42, 69, 13, 12, 1965, 1471, 81, 11, 8, 477, 937, 484, 14, 42, 12, 3689, 6695, 14, 8, 315, 1553, 13, 8, 31, 9, 18, 469, 53, 19, 31, 95, 41, 244, 12, 237, 188, 146, 93, 16, 87, 9, 18, 469, 53, 16, 810, 998, 8, 522, 72, 14, 12, 219, 1607, 20, 132, 27558, 9, 7, 52, 10, 18, 41, 14, 220, 16, 12, 2470, 706, 93, 18, 73, 41, 451, 14, 11, 156, 20, 16, 1616, 363, 13, 77, 1444, 9, 3]...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       "y: ItemList (90052 items)\n",
       "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       ">"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Numericalized text lists\n",
    "ll.train.x_obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"xxbos xxmaj once again xxmaj mr. xxmaj costner has dragged out a movie for far longer than necessary . xxmaj aside from the terrific sea rescue sequences , of which there are very few i just did not care about any of the characters . xxmaj most of us have ghosts in the closet , and xxmaj costner 's character are realized early on , and then forgotten until much later , by which time i did not care . xxmaj the character we should really care about is a very cocky , overconfident xxmaj ashton xxmaj kutcher . xxmaj the problem is he comes off as kid who thinks he 's better than anyone else around him and shows no signs of a cluttered closet . xxmaj his only obstacle appears to be winning over xxmaj costner . xxmaj finally when we are well past the half way point of this stinker , xxmaj costner tells us all about xxmaj kutcher 's ghosts . xxmaj we are told why xxmaj kutcher is driven to be the best with no prior inkling or foreshadowing . xxmaj no magic here , it was all i could do to keep from turning it off an hour in . xxeos\""
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here is the text in the first review\n",
    "ll.train.x_obj(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Save the labels`\n",
    "Since the preprocessing takes time, we save the intermediate result using pickle. \n",
    "\n",
    "Don't use`lambda` functions in your processors or they won't be able to pickle!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(ll, open(path/'ld.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "ll = pickle.load(open(path/'ld.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Batching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Batching for Language Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a bit of work to convert our `LabelList` in a `DataBunch` as we don't just want batches of IMDB reviews. We want to stream through all the concatenated texts. We also have to prepare the targets that are the newt words in the text. All of this is done with the next object called `LM_PreLoader`. At the beginning of each epoch, it'll shuffle the articles (if `shuffle=True`) and create a big stream by concatenating all of them. We divide this big stream in `bs` smaller streams. That we will read in chunks of bptt length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=5565)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just using those for illustration purposes, they're not used otherwise.\n",
    "from IPython.display import display,HTML\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say our stream is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stream = \"\"\"\n",
    "In this notebook, we will go back over the example of classifying movie reviews we studied in part 1 and dig deeper under the surface. \n",
    "First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we'll have another example of the Processor used in the data block API.\n",
    "Then we will study how we build a language model and train it.\\n\n",
    "\"\"\"\n",
    "tokens = np.array(tp([stream])[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to split the data into 6 batches of 15 tokens each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>xxbos</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>in</td>\n",
       "      <td>this</td>\n",
       "      <td>notebook</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>go</td>\n",
       "      <td>back</td>\n",
       "      <td>over</td>\n",
       "      <td>the</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>classifying</td>\n",
       "      <td>movie</td>\n",
       "      <td>reviews</td>\n",
       "      <td>we</td>\n",
       "      <td>studied</td>\n",
       "      <td>in</td>\n",
       "      <td>part</td>\n",
       "      <td>1</td>\n",
       "      <td>and</td>\n",
       "      <td>dig</td>\n",
       "      <td>deeper</td>\n",
       "      <td>under</td>\n",
       "      <td>the</td>\n",
       "      <td>surface</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>first</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>look</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>processing</td>\n",
       "      <td>steps</td>\n",
       "      <td>necessary</td>\n",
       "      <td>to</td>\n",
       "      <td>convert</td>\n",
       "      <td>text</td>\n",
       "      <td>into</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>numbers</td>\n",
       "      <td>and</td>\n",
       "      <td>how</td>\n",
       "      <td>to</td>\n",
       "      <td>customize</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>by</td>\n",
       "      <td>doing</td>\n",
       "      <td>this</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>'ll</td>\n",
       "      <td>have</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>another</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>processor</td>\n",
       "      <td>used</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>data</td>\n",
       "      <td>block</td>\n",
       "      <td>api</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>then</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>study</td>\n",
       "      <td>how</td>\n",
       "      <td>we</td>\n",
       "      <td>build</td>\n",
       "      <td>a</td>\n",
       "      <td>language</td>\n",
       "      <td>model</td>\n",
       "      <td>and</td>\n",
       "      <td>train</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bs,seq_len = 6,15\n",
    "d_tokens = np.array([tokens[i*seq_len:(i+1)*seq_len] for i in range(bs)])\n",
    "df = pd.DataFrame(d_tokens)\n",
    "display(HTML(df.to_html(index=False,header=None)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>in</td>\n",
       "      <td>this</td>\n",
       "      <td>notebook</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>go</td>\n",
       "      <td>back</td>\n",
       "      <td>over</td>\n",
       "      <td>the</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>classifying</td>\n",
       "      <td>movie</td>\n",
       "      <td>reviews</td>\n",
       "      <td>we</td>\n",
       "      <td>studied</td>\n",
       "      <td>in</td>\n",
       "      <td>part</td>\n",
       "      <td>1</td>\n",
       "      <td>and</td>\n",
       "      <td>dig</td>\n",
       "      <td>deeper</td>\n",
       "      <td>under</td>\n",
       "      <td>the</td>\n",
       "      <td>surface</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>first</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>look</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>processing</td>\n",
       "      <td>steps</td>\n",
       "      <td>necessary</td>\n",
       "      <td>to</td>\n",
       "      <td>convert</td>\n",
       "      <td>text</td>\n",
       "      <td>into</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>numbers</td>\n",
       "      <td>and</td>\n",
       "      <td>how</td>\n",
       "      <td>to</td>\n",
       "      <td>customize</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>by</td>\n",
       "      <td>doing</td>\n",
       "      <td>this</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>'ll</td>\n",
       "      <td>have</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>another</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>processor</td>\n",
       "      <td>used</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>data</td>\n",
       "      <td>block</td>\n",
       "      <td>api</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>then</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>study</td>\n",
       "      <td>how</td>\n",
       "      <td>we</td>\n",
       "      <td>build</td>\n",
       "      <td>a</td>\n",
       "      <td>language</td>\n",
       "      <td>model</td>\n",
       "      <td>and</td>\n",
       "      <td>train</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1        2      3          4          5      6      7  \\\n",
       "0        xxbos       \\n    xxmaj     in       this   notebook      ,     we   \n",
       "1  classifying    movie  reviews     we    studied         in   part      1   \n",
       "2           \\n    xxmaj    first     we       will       look     at    the   \n",
       "3      numbers      and      how     to  customize         it      .  xxmaj   \n",
       "4      another  example       of    the      xxmaj  processor   used     in   \n",
       "5         then       we     will  study        how         we  build      a   \n",
       "\n",
       "            8      9         10     11       12       13     14  \n",
       "0        will     go       back   over      the  example     of  \n",
       "1         and    dig     deeper  under      the  surface      .  \n",
       "2  processing  steps  necessary     to  convert     text   into  \n",
       "3          by  doing       this      ,       we      'll   have  \n",
       "4         the   data      block    api        .       \\n  xxmaj  \n",
       "5    language  model        and  train       it        .   \\n\\n  "
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can also view the data frame like this:\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting each batch of 15 tokens into 3 sub-batches of 5 tokens each can be done like this\n",
    "Here, `bptt` means `back-propagation through time`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxbos</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>in</td>\n",
       "      <td>this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>classifying</td>\n",
       "      <td>movie</td>\n",
       "      <td>reviews</td>\n",
       "      <td>we</td>\n",
       "      <td>studied</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>first</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>numbers</td>\n",
       "      <td>and</td>\n",
       "      <td>how</td>\n",
       "      <td>to</td>\n",
       "      <td>customize</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>another</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>then</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>study</td>\n",
       "      <td>how</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1        2      3          4\n",
       "0        xxbos       \\n    xxmaj     in       this\n",
       "1  classifying    movie  reviews     we    studied\n",
       "2           \\n    xxmaj    first     we       will\n",
       "3      numbers      and      how     to  customize\n",
       "4      another  example       of    the      xxmaj\n",
       "5         then       we     will  study        how"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>notebook</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>will</td>\n",
       "      <td>go</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>in</td>\n",
       "      <td>part</td>\n",
       "      <td>1</td>\n",
       "      <td>and</td>\n",
       "      <td>dig</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>look</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>processing</td>\n",
       "      <td>steps</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>xxmaj</td>\n",
       "      <td>by</td>\n",
       "      <td>doing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>processor</td>\n",
       "      <td>used</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>we</td>\n",
       "      <td>build</td>\n",
       "      <td>a</td>\n",
       "      <td>language</td>\n",
       "      <td>model</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0      1      2           3      4\n",
       "0   notebook      ,     we        will     go\n",
       "1         in   part      1         and    dig\n",
       "2       look     at    the  processing  steps\n",
       "3         it      .  xxmaj          by  doing\n",
       "4  processor   used     in         the   data\n",
       "5         we  build      a    language  model"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>back</td>\n",
       "      <td>over</td>\n",
       "      <td>the</td>\n",
       "      <td>example</td>\n",
       "      <td>of</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>deeper</td>\n",
       "      <td>under</td>\n",
       "      <td>the</td>\n",
       "      <td>surface</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>necessary</td>\n",
       "      <td>to</td>\n",
       "      <td>convert</td>\n",
       "      <td>text</td>\n",
       "      <td>into</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>this</td>\n",
       "      <td>,</td>\n",
       "      <td>we</td>\n",
       "      <td>'ll</td>\n",
       "      <td>have</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>block</td>\n",
       "      <td>api</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n</td>\n",
       "      <td>xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>and</td>\n",
       "      <td>train</td>\n",
       "      <td>it</td>\n",
       "      <td>.</td>\n",
       "      <td>\\n\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0      1        2        3      4\n",
       "0       back   over      the  example     of\n",
       "1     deeper  under      the  surface      .\n",
       "2  necessary     to  convert     text   into\n",
       "3       this      ,       we      'll   have\n",
       "4      block    api        .       \\n  xxmaj\n",
       "5        and  train       it        .   \\n\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bs,bptt = 6,5\n",
    "for k in range(3):\n",
    "    d_tokens = np.array([tokens[i*seq_len + k*bptt:i*seq_len + (k+1)*bptt] for i in range(bs)])\n",
    "    df = pd.DataFrame(d_tokens)\n",
    "    #display(HTML(df.to_html(index=False,header=None)))\n",
    "    display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class LM_PreLoader():\n",
    "    def __init__(self, data, bs=64, bptt=70, shuffle=False):\n",
    "        self.data,self.bs,self.bptt,self.shuffle = data,bs,bptt,shuffle\n",
    "        total_len = sum([len(t) for t in data.x])\n",
    "        self.n_batch = total_len // bs\n",
    "        self.batchify()\n",
    "    \n",
    "    def __len__(self): return ((self.n_batch-1) // self.bptt) * self.bs\n",
    "    \n",
    "    def batchify(self):\n",
    "        texts = self.data.x\n",
    "        if self.shuffle: texts = texts[torch.randperm(len(texts))]\n",
    "        stream = torch.cat([tensor(t) for t in texts])\n",
    "        self.batched_data = stream[:self.n_batch * self.bs].view(self.bs, self.n_batch)\n",
    "        \n",
    "    def __getitem__(self, idx):\n",
    "        source = self.batched_data[idx % self.bs] # % is the mod() operation\n",
    "        seq_idx = (idx // self.bs) * self.bptt\n",
    "        return source[seq_idx:seq_idx+self.bptt],source[seq_idx+1:seq_idx+self.bptt+1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate a text DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = DataLoader(LM_PreLoader(ll.valid, shuffle=True), batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it all works ok: `x1`, `y1`, `x2` and `y2` should all be of size `bs`  by `bptt`. The texts in each row of `x1` should continue in `x2`. `y1` and `y2` should have the same texts as their `x` counterpart, shifted of one position to the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "iter_dl = iter(dl)\n",
    "x1,y1 = next(iter_dl)\n",
    "x2,y2 = next(iter_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 70]), torch.Size([64, 70]))"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1.size(),y1.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = proc_num.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xxbos ok , so , xxmaj chuck xxmaj norris somehow found a way to get this sequel produced . i have one question - why ? xxmaj who read this script and said , \" xxmaj sound \\'s great ! xxmaj original ! \" xxmaj this movie is regurgitated crap that i would n\\'t tarnish my toilet bowel with . xxmaj of course it is another story , following in'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join(vocab[o] for o in x1[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ok , so , xxmaj chuck xxmaj norris somehow found a way to get this sequel produced . i have one question - why ? xxmaj who read this script and said , \" xxmaj sound \\'s great ! xxmaj original ! \" xxmaj this movie is regurgitated crap that i would n\\'t tarnish my toilet bowel with . xxmaj of course it is another story , following in the'"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join(vocab[o] for o in y1[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"the tradition of mia , xxmaj invasion usa , xxmaj walker , xxmaj texas xxmaj ranger , xxmaj delta xxmaj force and so on in which xxmaj chuck xxmaj norris , of course , is the one man who can same xxmaj america from some bad guys . xxmaj he does n't even need a gun most times . xxmaj it 's stupid patriotic jargon that xxunk the mind .\""
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join(vocab[o] for o in x2[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's prepare some convenience function to do this quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_lm_dls(train_ds, valid_ds, bs, bptt, **kwargs):\n",
    "    return (DataLoader(LM_PreLoader(train_ds, bs, bptt, shuffle=True), batch_size=bs, **kwargs),\n",
    "            DataLoader(LM_PreLoader(valid_ds, bs, bptt, shuffle=False), batch_size=2*bs, **kwargs))\n",
    "\n",
    "def lm_databunchify(sd, bs, bptt, **kwargs):\n",
    "    return DataBunch(*get_lm_dls(sd.train, sd.valid, bs, bptt, **kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,bptt = 64,70\n",
    "data = lm_databunchify(ll, bs, bptt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batching for classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we will want to tackle classification, gathering the data will be a bit different: first we will label our texts with the folder they come from, and then we will need to apply padding to batch them together. To avoid mixing very long texts with very short ones, we will also use `Sampler` to sort (with a bit of randomness for the training set) our samples by length.\n",
    "\n",
    "First the data block API calls shold look familiar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Jump_to lesson 12 video](https://course.fast.ai/videos/?lesson=12&t=5877)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc_cat = CategoryProcessor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='13' class='' max='13', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [13/13 02:31<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='13' class='' max='13', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [13/13 02:30<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "il = TextList.from_files(path, include=['train', 'test'])\n",
    "sd = SplitData.split_by_func(il, partial(grandparent_splitter, valid_name='test'))\n",
    "ll = label_by_func(sd, parent_labeler, proc_x = [proc_tok, proc_num], proc_y=proc_cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SplitData\n",
       "Train: LabeledData\n",
       "x: TextList (25000 items)\n",
       "[[2, 7, 81, 13, 12, 145, 49, 61, 7391, 1427, 28, 12, 4409, 9, 7, 533, 60, 27, 12, 652, 150, 20, 15, 12, 1362, 514, 13, 1916, 223, 9, 12, 11729, 7048, 328, 15, 670, 104, 48, 2055, 10, 1059, 2487, 47, 8, 976, 0, 13, 16, 22, 5993, 9, 7, 490, 16, 2968, 1916, 8, 237, 75, 27, 74, 767, 1429, 869, 253, 16, 56, 117, 143, 1517, 9, 7, 76, 165, 51, 8, 970, 155, 42, 670, 143, 9, 7, 8, 12793, 429, 73, 114, 7, 2303, 327, 753, 14, 12, 850, 12545, 9, 7, 34, 12, 1960, 656, 16, 22, 146, 93, 32, 251, 122, 27, 65, 67, 672, 47, 714, 101, 7, 37206, 7, 39035, 9, 7, 714, 426, 7, 3532, 7, 21243, 11, 7, 11415, 7, 6512, 78, 42, 131, 3361, 9, 3], [2, 7, 4154, 62, 12324, 533, 26, 12, 3412, 183, 8866, 14567, 1584, 15, 4992, 72, 27, 4662, 6558, 198, 160, 11638, 14, 959, 4915, 7, 3591, 7, 4513, 36, 7, 626, 7, 1853, 33, 49, 15, 1676, 111, 198, 12, 756, 13, 25258, 22, 14, 40, 3920, 17, 9655, 13, 16, 129, 3208, 14, 8, 1032, 26, 12, 4754, 10, 103, 34, 1793, 15, 7, 4513, 544, 7, 2018, 36, 7, 7894, 7, 34655, 33, 198, 57, 478, 9, 7, 8, 8866, 59644, 323, 143, 26, 3676, 30, 2247, 23, 914, 8, 1584, 15, 5267, 23, 20441, 47, 8, 1129, 23, 1733, 7, 12888, 36, 7, 607, 7, 39802, 33, 198, 40, 127, 13049, 22, 7, 12563, 36, 7, 13548, 7, 22866, 33, 198, 7, 2360, 36, 7, 535, 7, 32851, 33, 49, 3339, 8, 5259, 198, 885, 60, 27, 2857, 2426, 10, 46, 1309, 14, 2177, 8, 4662, 10224, 198, 1302, 34, 12, 29217, 1584, 3273, 34, 48, 3872, 942, 30, 153, 253, 40, 5281, 7, 12888, 242, 1798, 48, 2923, 11489, 17, 8, 7, 3377, 198, 2107, 1135, 13, 8, 1584, 5154, 16, 6534, 104, 8, 1598, 135, 16, 8112, 14, 8, 1356, 227, 3924, 17, 8, 661, 13, 8, 7, 21657, 7, 4660, 9, 7, 27, 914, 17, 363, 5730, 10, 1002, 24463, 17, 198, 277, 14435, 142, 6356, 2354, 143, 285, 8, 705, 11929, 28, 8, 4687, 22, 26, 46, 11883, 366, 27, 75, 718, 650, 60, 92, 24, 7, 103, 561, 472, 8, 1064, 286, 28427, 7, 4154, 6086, 19, 349, 699, 14, 8, 6809, 23, 617, 1532, 704, 7, 4154, 36, 2732, 33, 25, 534, 47, 7, 1734, 7, 16689, 198, 153, 301, 193, 53, 16, 22, 8732, 18, 197, 35, 156, 7, 4154, 62, 12324, 15, 120, 454, 13, 1536, 390, 16, 15, 436, 284, 37, 2809, 28, 8, 227, 1004, 9, 7, 60, 13, 8, 297, 7, 4154, 124, 18, 41, 131, 52, 247, 18, 177, 451, 19, 43, 8, 139, 10, 56, 9, 7, 16, 61, 77, 1651, 130, 13, 8, 297, 27, 12, 357, 2247, 23, 914, 5267, 23, 30516, 198, 115, 8, 6534, 36, 87, 35, 39, 84, 8, 2923, 11489, 66, 33, 198, 6914, 13, 8, 14567, 36, 294, 8, 1188, 86, 280, 14, 1518, 8, 233, 7, 4154, 27, 175, 1080, 1532, 510, 13, 8, 826, 7, 8, 7, 17914, 7, 1149, 36, 5019, 33, 33, 198, 17702, 15, 135, 16, 2968, 382, 8, 148, 27, 12, 4484, 5871, 4268, 165, 2339, 973, 10, 370, 37958, 68, 8, 914, 1159, 60, 55, 10847, 26, 8, 14567, 20363, 55, 63, 120, 13, 8, 3892, 38, 3208, 198, 16, 22, 12, 548, 337, 20, 95, 41, 113, 28, 12, 101, 138, 1532, 510, 30, 97, 6866, 123, 22, 10, 769, 429, 10, 12996, 287, 23, 1324, 198, 12, 164, 620, 13, 2401, 55, 792, 55, 1117, 813, 19, 15, 12, 1121, 1436, 9, 7, 153, 8, 269, 10585, 130, 944, 43, 2463, 28, 23100, 1066, 249, 37, 20, 94, 569, 119, 8, 1584, 8112, 198, 54, 22, 37, 26, 94, 12123, 26, 18, 217, 54, 155, 41, 98, 9, 7, 76, 68, 8, 7, 3491, 443, 585, 200, 58, 35, 1195, 72, 20, 94, 27, 12, 189, 679, 13, 674, 4824, 198, 9979, 1676, 59, 30, 54, 22, 56, 158, 1817, 149, 9, 7, 732, 7, 3039, 26, 8, 55833, 13732, 3576, 7, 932, 7, 30728, 15, 162, 30, 82, 240, 12, 384, 13, 157, 198, 1180, 76, 568, 256, 14774, 14, 56, 185, 3752, 17, 8, 956, 9, 24, 7, 8, 365, 398, 198, 2284, 353, 13, 7, 4154, 62, 12324, 502, 23100, 249, 153, 8, 201, 264, 2510, 724, 48, 1554, 564, 13, 891, 593, 12, 183, 652, 920, 721, 10, 128, 69, 157, 27, 7, 732, 7, 3039, 26, 7, 30728, 10, 2080, 14, 2130, 60, 123, 22, 10, 1140, 2092, 157, 198, 8, 3656, 55, 175, 384, 13, 352, 2058, 593, 8, 23806, 9, 7, 153, 18, 73, 53, 14, 84, 19, 1554, 891, 18, 257, 37, 271, 18, 95, 864, 163, 12, 748, 297, 564, 615, 13, 7, 4154, 62, 12324, 9, 7, 26, 863, 8, 31, 61, 1970, 947, 27, 508, 8180, 198, 6899, 1702, 2598, 10, 18, 105, 156, 74, 69, 102, 93, 8, 3175, 1584, 2173, 308, 38, 35, 101, 370, 9, 7, 361, 27, 8, 102, 127, 7, 4154, 2054, 19, 323, 3547, 13, 288, 17, 8, 7, 11407, 7, 1447, 22, 7, 2261, 13, 7, 906, 284, 18, 78, 122, 13, 742, 13, 459, 124, 93, 19, 52, 18, 10898, 20, 22, 12, 138, 2582, 9, 7, 8, 213, 157, 38, 12, 138, 769, 490, 10, 8, 1083, 15, 579, 198, 37, 94, 2432, 55, 1117, 15, 4960, 79, 15, 12, 906, 26, 18, 10898, 19, 95, 41, 98, 12, 206, 67, 31, 63, 113, 2838, 9, 24, 7, 8, 392, 1199, 38, 2882, 63, 180, 2115, 9, 7, 8, 137, 15, 35, 101, 10, 127, 75, 7, 861, 2267, 7, 689, 7, 4910, 61, 324, 254, 16, 25, 12, 1425, 14, 341, 17, 19, 10, 43, 75, 7, 861, 2267, 7, 626, 7, 1853, 307, 176, 198, 11917, 10, 103, 43, 75, 7, 861, 2267, 7, 856, 7, 1596, 307, 1930, 153, 7, 2703, 7, 1572, 7, 856, 15, 383, 138, 14, 58, 198, 54, 38, 983, 13, 102, 1082, 1437, 14, 185, 60, 28, 117, 9, 24, 7, 4154, 62, 12324, 15, 8, 110, 1532, 19670, 13, 8, 297, 7, 4154, 124, 52, 247, 198, 18, 451, 8, 1026, 540, 16, 76, 63, 46, 86, 12, 245, 673, 10, 8, 392, 198, 1993, 471, 91, 35, 366, 173, 198, 12, 31, 59, 12, 18399, 1584, 56, 155, 35, 42, 19, 364, 55, 12996, 9, 7, 1609, 47, 7, 8, 7, 13575, 92, 7, 4154, 62, 13174, 36, 5016, 33, 9, 3], [2, 7, 19, 31, 3577, 158, 18, 95, 35, 300, 77, 3859, 34, 45, 107, 96, 3228, 34, 8, 192, 13, 8, 998, 560, 9, 7, 19, 5205, 6074, 14, 620, 13, 1221, 68, 71, 5214, 8, 278, 27, 57, 998, 145, 9, 7, 76, 8, 752, 157, 409, 618, 26, 129, 1562, 8, 171, 45, 317, 9, 7, 16, 95, 70, 88, 41, 98, 8, 170, 49, 40657, 64, 39, 929, 51, 8, 171, 9, 18, 56, 58, 35, 140, 9, 24, 7, 30, 95, 16, 41, 98, 8, 910, 66, 7, 56, 627, 49, 25, 8, 7005, 17, 133, 27, 66, 7, 39, 484, 69, 15610, 13, 40, 29860, 1938, 11, 3074, 10, 11, 1134, 13, 333, 11, 40, 6393, 6234, 10, 93, 13, 1796, 55, 256, 345, 9, 7, 39, 132, 2545, 89, 39, 25, 17, 133, 27, 8, 2597, 9, 24, 18, 25, 683, 17, 19, 29, 9, 7, 30, 10, 58, 35, 852, 16, 25, 2482, 28, 48, 7, 861, 10, 52, 1639, 28, 668, 9, 3], [2, 7, 772, 312, 10, 10, 10, 18, 140, 19, 15, 457, 14, 42, 48, 21, 555, 21, 31, 10, 10, 30, 1387, 10, 46, 155, 41, 2698, 60, 1831, 45, 8, 2643, 52, 100, 95, 2213, 80, 3874, 60, 11, 37, 126, 9, 7, 284, 8, 150, 1702, 11, 13942, 471, 25, 338, 10, 19, 81, 15, 117, 1366, 14, 126, 9, 7, 8, 3828, 13, 12, 498, 1433, 25, 1775, 9, 7, 8, 4850, 4, 222, 1440, 4, 374, 2805, 3210, 679, 86, 117, 216, 9, 7, 109, 216, 78, 32, 126, 127, 100, 56, 1305, 54, 11, 688, 66, 7, 282, 68, 8, 429, 15, 127, 100, 5129, 9, 18, 83, 85, 12, 266, 75, 56, 402, 163, 19, 31, 9, 7, 8, 376, 86, 338, 10, 30, 109, 94, 13, 20, 479, 10, 16267, 10, 3217, 10, 541, 78, 32, 212, 66, 7, 8, 82, 169, 18, 451, 25, 7, 8132, 7, 18935, 11, 57, 802, 2387, 11, 1230, 150, 9, 7, 886, 19, 25, 12, 6803, 13, 7, 3401, 9, 7, 11, 18, 167, 74, 355, 2254, 40, 370, 9, 18, 122, 273, 49, 568, 46, 523, 385, 385, 125, 263, 621, 13, 19, 15, 10, 10, 88, 10, 3161, 9, 3], [2, 7, 68, 18, 25, 138, 77, 722, 573, 89, 361, 14, 8, 803, 14, 84, 7, 8934, 9, 7, 16, 25, 43, 13, 128, 118, 18, 321, 27, 77, 722, 10, 30, 19, 25, 8, 82, 43, 90, 2411, 60, 13, 9, 7, 254, 115, 18, 85, 132, 131, 7, 8934, 382, 56, 1053, 10, 11, 18, 95, 41, 1382, 60, 8, 395, 13, 77, 136, 229, 16, 9, 7, 64, 12, 2168, 10, 12766, 10, 11, 2280, 364, 450, 13, 1344, 22, 4730, 11, 2847, 4571, 9, 7, 2265, 7, 1448, 15, 43, 13, 77, 543, 992, 30, 7, 8934, 15, 47, 247, 8, 270, 450, 13, 629, 13, 40, 623, 9, 7, 17, 8, 14536, 415, 13, 7, 12405, 7, 23902, 10, 7, 1448, 428, 201, 12, 479, 10, 25031, 10, 10228, 10, 2661, 17, 14, 8, 475, 13, 12, 252, 13322, 47, 8, 2009, 4091, 2319, 47, 4132, 10, 29924, 10, 623, 10, 133, 10, 698, 23, 133, 10, 0, 10, 857, 9, 7, 8, 31, 10, 5182, 10, 61, 74, 727, 2083, 10, 74, 239, 10, 11, 15, 13819, 17, 9994, 6338, 9, 7, 19, 31, 415, 78, 42, 139, 4795, 26, 16932, 17, 958, 10, 804, 48, 21354, 4076, 13, 429, 14, 14638, 12, 21, 69, 11454, 1144, 13, 1191, 11, 830, 21, 9, 7, 30, 7, 2265, 7, 1448, 15, 74, 7, 12405, 7, 3401, 9, 7, 8, 31, 15, 2280, 579, 11, 769, 9, 7, 30, 691, 20, 10, 18, 346, 85, 74, 1978, 27, 55, 2408, 28, 120, 13, 8, 121, 9, 7, 322, 18, 469, 82, 6898, 28, 19, 5676, 13, 20679, 10, 7202, 10, 29586, 13764, 10, 32277, 17, 12, 11826, 2908, 28, 1963, 9, 7, 8214, 12, 3510, 13, 20807, 41143, 11, 7606, 7, 31563, 42085, 8, 81, 18974, 53, 12, 6779, 17, 8, 663, 9, 7, 312, 2416, 17, 3929, 19364, 11, 11553, 1084, 225, 8164, 9, 7, 312, 15, 21, 437, 21, 11, 21, 2729, 21, 10, 1767, 14, 187, 471, 55, 1741, 55, 857, 11, 16, 56, 289, 34, 11, 34, 14, 8, 243, 135, 32, 56, 205, 14, 4192, 44, 13, 111, 9, 7, 16, 22, 132, 59, 3704, 10, 16, 22, 82, 59, 11675, 13577, 13751, 9, 7, 16, 15, 180, 69, 93, 12, 2009, 463, 598, 14, 48, 1586, 691, 8, 328, 22, 1250, 14, 3804, 9, 7, 2265, 7, 1448, 2686, 14, 114, 121, 52, 12135, 17, 532, 90, 250, 336, 60, 9, 7, 11, 28, 20, 313, 18, 275, 19, 29, 2280, 554, 4922, 11, 16389, 16650, 9, 18, 84, 64, 39, 25, 182, 28, 30, 40, 13035, 34, 9278, 40, 754, 163, 7, 25340, 15182, 11, 8261, 31, 3227, 42532, 16, 518, 8, 243, 13, 6735, 9, 18, 566, 406, 19, 43, 63, 32, 199, 577, 12, 138, 117, 664, 11, 387, 158, 14, 3059, 32, 13, 342, 9, 7, 886, 10, 305, 22, 56, 3826, 19, 31, 132, 578, 9, 3], [2, 21, 7, 16, 737, 20, 128, 1502, 187, 8, 337, 13, 12, 7, 2265, 7, 1448, 463, 37780, 9, 21, 7, 11, 28, 67, 313, 96, 46, 38, 10162, 1670, 11, 2168, 13895, 13, 7, 3401, 9, 7, 11, 305, 22, 37, 538, 3424, 96, 1502, 86, 685, 8369, 13, 7, 1448, 22, 7, 3401, 10990, 10, 7, 1448, 22, 7202, 13601, 14, 8, 4009, 10208, 9, 7, 64, 18, 58, 35, 99, 15, 19, 96, 154, 25, 7, 1448, 1260, 11378, 28, 40, 2665, 17, 9454, 7, 3401, 10, 30, 8, 32898, 7, 1942, 12190, 25, 0, 28, 21, 5695, 143, 21, 7, 1509, 17, 40, 792, 125, 202, 124, 66, 7, 17, 7, 1538, 7, 1739, 22, 667, 10, 16, 22, 12, 669, 840, 13, 2768, 20782, 9, 18, 73, 41, 14, 1046, 27, 20, 9, 3], [2, 7, 8, 349, 591, 47, 12, 7, 183, 7, 791, 2872, 17, 360, 93, 184, 174, 14, 114, 12, 21, 7, 3589, 21, 31, 23, 8, 107, 129, 7, 2540, 7, 0, 22, 21, 7, 609, 7, 4054, 21, 36, 79, 25, 113, 17, 7, 7436, 10, 27, 7, 3589, 171, 10, 74, 360, 50, 33, 7, 8, 1969, 66, 7, 861, 7, 9446, 324, 16, 139, 10, 17, 2789, 14, 7, 6463, 62, 21, 7, 8, 7, 176, 7, 3317, 7, 2062, 21, 96, 21, 7, 43, 73, 41, 14, 41, 12, 507, 13, 1595, 37, 14, 447, 60, 1347, 45, 8, 342, 13, 7, 138, 7, 11880, 9, 21, 7, 206, 94, 8, 186, 169, 149, 9, 21, 7, 8934, 21, 15, 9326, 388, 13, 31485, 55834, 21968, 9, 36, 21, 18, 167, 1662, 13, 77, 2679, 9, 21, 7, 281, 104, 8, 661, 3825, 96, 21, 18, 58, 35, 53, 49, 18, 167, 1545, 9, 21, 33, 7, 8, 3713, 28162, 36, 14, 381, 12, 9461, 2593, 33, 51, 7, 3401, 38, 517, 14, 1943, 9, 7, 8, 979, 554, 23, 585, 252, 404, 7061, 201, 13, 109, 547, 11, 1049, 46, 38, 10, 14, 8, 243, 13, 22158, 9, 36, 21, 18, 373, 12, 5936, 13, 6654, 8, 102, 272, 9, 7, 16, 25, 17, 23, 18, 58, 35, 140, 23, 7, 8, 7, 183, 7, 13262, 9, 21, 21, 7, 461, 9, 7, 20, 25, 48, 176, 5936, 9, 18, 18552, 16, 9, 21, 33, 7, 247, 51, 37, 3100, 59, 151, 100, 10, 215, 10, 18, 275, 111, 203, 606, 9, 7, 94, 13, 8, 739, 15, 627, 53, 8, 172, 541, 51, 7, 1448, 22, 922, 124, 23, 82, 39, 22, 534, 40, 171, 14, 317, 8, 427, 703, 9, 7, 277, 37, 196, 333, 17, 8, 29, 10, 39, 61, 358, 7, 1182, 7, 6903, 7, 1511, 990, 44, 13, 40, 12157, 22481, 10, 39031, 10, 11, 5650, 10023, 10, 1618, 57, 104, 48, 2257, 13214, 36, 94, 53, 7, 5860, 7, 6100, 17, 21, 7, 4241, 21, 33, 9, 24, 7, 8, 1147, 130, 23, 5430, 252, 27, 4985, 14136, 401, 23, 209, 14, 42, 4990, 69, 55, 360, 51, 7, 3401, 22, 21, 7, 3973, 7, 649, 10, 21, 8, 1147, 252, 2578, 9622, 72, 27, 12, 188, 13, 8678, 4872, 9, 7, 16, 44, 292, 163, 17, 8, 35653, 1105, 125, 20287, 3164, 96, 8, 24632, 7251, 13, 12, 14118, 34, 1967, 10, 8, 12612, 6952, 20, 11199, 8, 100, 1374, 34, 8, 1881, 9, 530, 10, 530, 9, 24, 7, 1448, 22, 330, 21, 624, 21, 124, 38, 360, 2257, 10, 30, 103, 247, 360, 436, 9, 18, 259, 212, 21, 7, 8934, 9, 21, 7, 2265, 22, 1699, 113, 12, 2628, 29, 9, 3], [2, 18, 58, 35, 140, 49, 14, 1688, 10, 8, 10989, 935, 55, 8, 4962, 170, 9, 7, 16, 484, 14, 42, 43, 13, 165, 118, 135, 52, 94, 25, 1527, 14, 8, 426, 36, 7, 8511, 10, 7, 1231, 10, 7, 9903, 10, 7, 14237, 11, 7, 3687, 33, 20, 54, 25, 35, 214, 336, 14, 83, 114, 12, 29, 9, 7, 19, 95, 41, 98, 70, 436, 10, 30, 54, 25, 12, 14227, 13, 39212, 10, 76, 18743, 10, 20, 5195, 142, 274, 150, 9, 7, 254, 16, 207, 48, 2118, 706, 581, 154, 25, 8, 11775, 7557, 5086, 150, 339, 27, 12, 0, 23, 176, 268, 11, 37, 7, 8511, 7, 13963, 66, 7, 154, 91, 7, 4757, 20617, 163, 4135, 897, 1474, 27, 127, 13, 8, 110, 340, 11, 1262, 1439, 17, 8, 195, 66, 7, 63, 46, 86, 82, 281, 28, 936, 154, 37, 196, 7, 9424, 7, 5964, 11, 7, 4363, 7, 55835, 322, 66, 7, 19, 25, 52, 733, 18, 25, 777, 14, 187, 20, 8, 170, 25, 35, 12, 757, 343, 176, 9, 7, 64, 12, 474, 10, 37, 56, 28, 8, 796, 30, 28, 8, 171, 26, 88, 9, 3], [2, 7, 19, 31, 15, 1558, 45, 139, 9, 7, 8511, 7, 13963, 15, 26, 172, 26, 12, 2904, 13, 22816, 9, 7, 57, 8118, 8620, 51, 21, 7, 1112, 11, 7, 643, 21, 2798, 142, 17, 12, 1215, 591, 45, 223, 9, 7, 1231, 7, 4757, 15, 8, 82, 43, 14, 238, 60, 14522, 17, 19, 508, 1190, 23, 223, 9, 7, 8, 82, 1186, 169, 14, 238, 60, 13, 19, 950, 15, 7, 1231, 11, 7, 9903, 22, 1412, 9, 7, 2252, 20, 793, 6482, 146, 1969, 9, 3], [2, 7, 8, 31, 15, 97, 9, 7, 54, 15, 74, 102, 116, 14, 156, 16, 9, 7, 8, 81, 15, 866, 11, 6971, 10, 282, 28, 19, 657, 9, 18, 58, 35, 122, 110, 100, 140, 64, 12, 21, 3404, 21, 15, 55, 105, 83, 480, 9, 18, 469, 26, 63, 18, 25, 168, 12, 29, 51, 8, 1344, 22, 9, 7, 8, 874, 25, 56, 37, 838, 28, 8, 343, 4253, 10, 76, 129, 287, 17, 9341, 9, 18, 122, 19, 13258, 315, 28, 312, 345, 49, 321, 16, 117, 26, 8, 8446, 86, 380, 11, 1534, 45, 8, 148, 9, 7, 110, 87, 35, 790, 28, 8, 13070, 370, 9, 24, 18, 58, 35, 122, 7, 14215, 83, 217, 8, 31, 60, 1421, 13, 75, 9, 7, 128, 13, 8, 157, 484, 14, 42, 615, 363, 26, 63, 46, 86, 132, 1832, 55, 39, 56, 87, 35, 140, 109, 14, 1443, 111, 9, 7, 39, 5366, 51, 43, 150, 14, 8, 386, 11, 32, 85, 14, 371, 11, 855, 60, 55, 493, 64, 25, 182, 34, 9, 18, 83, 87, 35, 99, 7, 2265, 22, 36, 7, 3304, 33, 1954, 136, 55, 1369, 370, 9, 7, 64, 86, 44, 8, 21, 1759, 21, 895, 9785, 11, 1720, 1310, 36, 51, 7, 4293, 12600, 33, 59, 66, 7, 64, 25, 39, 553, 66, 18, 122, 16, 25, 40, 70, 358, 591, 45, 280, 14, 1029, 19, 479, 1954, 19778, 136, 28, 7, 2265, 22, 123, 36, 7, 537, 33, 9, 7, 16, 87, 35, 181, 9, 7, 16, 87, 35, 76, 327, 14, 114, 295, 83, 9, 24, 7, 8, 82, 67, 169, 59, 19, 31, 25, 7, 2265, 7, 10899, 9, 7, 39, 276, 40, 123, 36, 7, 537, 33, 12818, 9, 7, 32, 83, 87, 99, 12, 101, 295, 13, 64, 12, 21, 3404, 21, 226, 41, 98, 53, 36, 156, 1701, 174, 625, 33, 9, 7, 39, 25, 101, 11, 110, 1325, 105, 132, 99, 4260, 28, 16, 9, 24, 7, 26, 28, 7, 5814, 10, 7, 5631, 11, 7, 9367, 92, 7, 364, 9, 24, 7, 58, 35, 84, 16, 50, 7, 16, 15, 1366, 50, 7, 919, 32, 38, 12, 315, 7, 10899, 355, 9, 3]...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       "y: ItemList (25000 items)\n",
       "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       "\n",
       "Valid: LabeledData\n",
       "x: TextList (25000 items)\n",
       "[[2, 7, 301, 193, 7, 611, 7, 5508, 61, 3423, 60, 12, 29, 28, 247, 1140, 93, 1727, 9, 7, 1202, 51, 8, 1362, 1598, 2092, 834, 10, 13, 79, 54, 38, 70, 189, 18, 56, 87, 37, 480, 59, 120, 13, 8, 121, 9, 7, 110, 13, 201, 41, 2952, 17, 8, 4891, 10, 11, 7, 5508, 22, 123, 38, 1722, 435, 34, 10, 11, 115, 1536, 382, 94, 330, 10, 47, 79, 75, 18, 87, 37, 480, 9, 7, 8, 123, 90, 155, 83, 480, 59, 15, 12, 70, 8765, 10, 37775, 7, 8442, 7, 11059, 9, 7, 8, 460, 15, 39, 292, 143, 26, 538, 49, 1212, 39, 22, 146, 93, 273, 345, 208, 108, 11, 293, 74, 3688, 13, 12, 16726, 4891, 9, 7, 40, 82, 13583, 737, 14, 42, 1770, 142, 7, 5508, 9, 7, 449, 68, 90, 38, 88, 518, 8, 331, 116, 243, 13, 19, 3962, 10, 7, 5508, 715, 201, 44, 59, 7, 11059, 22, 2952, 9, 7, 90, 38, 590, 154, 7, 11059, 15, 2122, 14, 42, 8, 139, 27, 74, 2676, 17143, 55, 12061, 9, 7, 74, 1326, 149, 10, 16, 25, 44, 18, 95, 58, 14, 404, 51, 1618, 16, 143, 48, 564, 17, 9, 3], [2, 7, 19, 15, 48, 514, 13, 154, 8, 2136, 13, 213, 124, 38, 8, 186, 9, 7, 3732, 11, 364, 10, 54, 22, 83, 180, 296, 168, 149, 9, 12, 622, 474, 13, 8, 115, 1180, 23, 15760, 1988, 13, 7, 1742, 23, 2315, 11, 7, 1742, 7, 6027, 10, 49, 161, 274, 14248, 128, 232, 142, 20, 46, 38, 2120, 13, 137, 10, 11, 137, 88, 9, 7, 58, 35, 1343, 27, 19, 43, 10, 159, 84, 7, 183, 7, 689, 7, 512, 10, 7, 32130, 55, 126, 7, 183, 7, 791, 7, 5298, 28, 7, 1742, 23, 2315, 10, 55, 7, 15085, 2805, 8, 7, 2560, 10, 7, 2029, 7, 2869, 55, 7, 2139, 28, 7, 1742, 7, 6027, 11, 84, 8, 164, 831, 9, 7, 1742, 23, 2315, 22, 2271, 7540, 429, 642, 178, 19, 31, 17144, 45, 8, 2811, 10, 11, 18, 167, 152, 1555, 64, 8, 2088, 7, 960, 7, 7960, 25, 422, 17, 19, 31, 66, 7, 11, 154, 8, 2088, 91, 39, 234, 317, 8, 2449, 186, 123, 66, 7, 51, 7, 1704, 19628, 10, 191, 31, 18, 161, 131, 27, 7, 960, 7, 7960, 61, 108, 419, 8, 2449, 186, 2191, 123, 10, 11, 45, 241, 17, 7, 1704, 40, 123, 1095, 10, 79, 113, 16, 666, 16520, 92, 24, 7, 462, 10, 19, 15, 349, 23, 1001, 213, 1225, 9, 7, 54, 38, 3539, 146, 124, 14, 84, 10, 11, 63, 32, 83, 205, 14, 84, 19, 43, 10, 126, 7, 8816, 7, 329, 10, 79, 15, 2357, 12, 11291, 990, 30, 61, 146, 137, 11, 12, 146, 246, 9, 7, 8, 82, 169, 20, 113, 19, 45, 44, 296, 168, 25, 12, 548, 519, 34, 8, 377, 23, 8, 672, 25, 242, 2361, 10, 79, 292, 517, 14, 253, 72, 28, 8, 508, 31, 413, 23, 30, 37, 203, 9, 222, 125, 184, 9, 3], [2, 7, 107, 13, 44, 18, 731, 165, 5112, 13369, 10, 49, 32690, 504, 63, 46, 85, 12, 923, 6731, 465, 80, 29705, 9, 7, 44, 46, 58, 15, 2687, 11, 1192, 274, 102, 11, 137, 53, 0, 353, 13, 3945, 9, 24, 7, 8, 29, 91, 35, 212, 69, 93, 757, 249, 14, 1265, 64, 15, 182, 34, 179, 90, 199, 496, 45, 8, 7545, 7, 54, 15, 37, 12, 694, 2232, 123, 17, 19, 29, 10, 567, 28, 8, 4735, 230, 10, 49, 15, 103, 8, 82, 43, 27, 331, 12, 1208, 9, 24, 7, 960, 7, 7960, 11, 7, 986, 7, 20961, 38, 218, 2096, 53807, 11, 7, 0, 123, 15, 56, 26, 94, 12, 1019, 26, 8, 3945, 9, 18, 20737, 53, 108, 227, 51, 8, 391, 9, 24, 7, 8, 29, 15, 1060, 27, 1136, 559, 11, 7, 2696, 7, 3047, 15479, 96, 100, 1467, 163, 5744, 27, 2849, 1676, 2581, 9, 7, 54, 15, 206, 94, 74, 130, 11, 16, 15, 12, 219, 460, 68, 32, 3946, 28, 74, 23, 43, 9, 7, 1293, 1510, 10, 567, 51, 7, 7960, 11, 8, 4735, 230, 11, 1293, 99, 64, 46, 1780, 9, 24, 7, 8, 82, 127, 332, 100, 20, 78, 504, 15, 8, 4735, 230, 11, 8, 8077, 30, 46, 199, 171, 47, 5931, 10, 37, 640, 1540, 1208, 352, 13369, 9, 24, 7, 790, 262, 51, 19, 629, 11, 126, 7640, 621, 385, 11, 263, 322, 9, 7, 45, 10117, 46, 41, 121, 32, 480, 59, 10, 12, 295, 13, 470, 11, 180, 30, 164, 171, 17, 8, 196, 9, 3], [2, 7, 37, 76, 8, 7, 4777, 95, 940, 770, 312, 451, 10, 11, 284, 7, 2696, 7, 2096, 15, 74, 24409, 23, 367, 39, 22, 349, 14, 619, 68, 16, 292, 14, 217, 2791, 213, 118, 9, 7, 8, 8519, 409, 11, 1085, 36128, 86, 2459, 17, 239, 11, 31, 10, 8, 17594, 13, 8, 7, 11370, 670, 29, 341, 25, 17, 388, 6495, 10, 8, 137, 573, 12, 162, 2024, 14, 274, 145, 22, 16309, 14509, 1218, 11, 9092, 137, 9, 7, 19, 25, 43, 13, 8, 128, 1742, 23, 2315, 118, 18, 235, 26, 12, 538, 11, 481, 10, 82, 14, 126, 111, 330, 11, 3322, 9, 7, 960, 7, 7960, 11, 7, 986, 7, 20961, 38, 16310, 27, 1147, 475, 382, 12, 3250, 1257, 9736, 59, 14, 159, 72, 17, 7889, 926, 142, 12, 5168, 27, 1642, 9298, 9, 18, 519, 16, 14, 7, 2696, 28, 949, 11, 6694, 937, 72, 8, 304, 121, 11, 1522, 9, 7, 30, 18, 2041, 312, 585, 28, 1618, 60, 7, 873, 23, 1440, 376, 9, 7, 1742, 23, 2315, 11, 6027, 224, 41, 98, 802, 897, 45, 19, 75, 10, 11, 153, 18, 161, 523, 218, 80, 3736, 26, 13369, 10, 17, 77, 677, 46, 1616, 1081, 17, 19, 29, 9, 7, 16, 22, 59, 7361, 249, 13, 43, 230, 3733, 1618, 40, 162, 34, 8, 102, 230, 14, 8, 243, 32, 187, 668, 3130, 17, 2587, 1581, 13, 2902, 9, 7, 166, 19, 15, 12, 29, 10, 112, 37, 12, 633, 52, 18, 491, 1164, 474, 77, 75, 17803, 44, 8, 394, 130, 1290, 17, 19, 29, 10, 30, 54, 86, 128, 10, 11, 46, 1628, 1276, 9, 18, 207, 8, 577, 168, 19, 20, 312, 34, 287, 25, 0, 13, 1458, 11, 56, 419, 200, 143, 8, 24410, 9, 7, 54, 38, 127, 200, 18, 152, 389, 59, 16, 10, 43, 2117, 12, 150, 27, 12, 11967, 11, 8, 102, 15, 7, 20961, 22, 674, 3712, 7335, 9, 7, 1356, 368, 19, 29, 15, 53, 9145, 22, 6998, 9, 7, 1156, 1907, 1876, 16, 63, 18, 167, 4833, 11, 18, 58, 35, 250, 53, 6797, 10, 7, 30, 18, 167, 88, 1864, 16, 4686, 53, 629, 9, 379, 426, 10, 19342, 9, 3], [2, 7, 8443, 1310, 36, 118, 15, 37, 12, 3770, 686, 28, 111, 33, 83, 38, 666, 22121, 9, 7, 80, 9407, 1105, 2337, 38, 2773, 13, 3255, 325, 744, 264, 3679, 9, 7, 30, 490, 7, 8443, 1310, 38, 822, 124, 27, 8, 11770, 13, 1694, 14, 2765, 796, 28, 142, 127, 621, 50, 7, 17, 19, 46, 1989, 3379, 10, 80, 9907, 10, 30, 269, 1810, 11, 21719, 93, 9093, 10, 2550, 2337, 698, 25547, 9, 24, 7, 44702, 62, 3712, 15, 12, 1007, 13, 12, 31, 47, 7, 18284, 7, 9094, 27, 8, 186, 439, 11, 7, 26823, 7, 20962, 11, 7, 7025, 7, 8567, 17, 8, 473, 9, 7, 8, 233, 715, 12, 81, 13, 4819, 133, 11, 3887, 17, 11, 208, 7, 6496, 316, 8, 7, 1040, 1606, 13, 6207, 9, 7, 8443, 1657, 8, 213, 51, 8, 4820, 104, 8, 3626, 1090, 10, 6826, 14, 42, 2449, 10, 52, 54, 38, 7, 15761, 48192, 10, 348, 17, 332, 7514, 10, 7, 1045, 7178, 55, 8, 19343, 15762, 13, 8, 23942, 9, 7, 30, 16, 15, 56, 1822, 4810, 10, 8, 5230, 1959, 15, 356, 20209, 9, 24, 7, 2109, 7, 0, 310, 8, 1611, 12062, 268, 49, 747, 28, 8, 8862, 8639, 230, 49, 234, 1418, 34, 117, 94, 10623, 9, 7, 71, 15, 48, 1611, 10, 9247, 10, 88, 4007, 7, 1040, 560, 11, 708, 725, 8, 844, 9, 7, 57, 1887, 2003, 13, 2898, 2916, 36, 58551, 3181, 10, 10274, 10, 2870, 10, 1127, 10, 731, 92, 11, 13102, 33, 38, 8, 139, 313, 14, 126, 19, 453, 11, 296, 127, 426, 9, 7, 71, 14794, 19, 676, 4443, 541, 27, 48, 4788, 1142, 13, 3997, 9, 18, 658, 65, 83, 67, 524, 238, 361, 28, 57, 9, 7, 71, 83, 1031, 16, 9, 3], [2, 12, 172, 169, 578, 14, 89, 153, 168, 21, 7, 22948, 21, 96, 34, 8, 43, 519, 10, 8, 576, 15, 12, 5066, 23, 5041, 11, 8, 170, 15, 483, 2089, 14, 114, 201, 410, 154, 39, 91, 64, 39, 91, 36, 22552, 14249, 92, 5738, 10, 1411, 89, 10, 4911, 33, 163, 40, 1220, 9, 7, 34, 8, 102, 519, 10, 8, 7, 646, 353, 45, 241, 15, 70, 947, 2364, 9, 7, 52, 18, 275, 558, 4764, 54, 85, 98, 218, 69, 11, 360, 429, 45, 8, 186, 75, 50, 7, 19, 31, 15, 394, 36, 172, 109, 19, 230, 61, 5001, 14, 191, 7845, 11, 20963, 17, 40, 500, 33, 11, 8725, 36, 135, 73, 90, 42, 17, 12, 1951, 44703, 229, 279, 2255, 2279, 150, 66, 33, 9, 7, 37, 14, 762, 8, 21, 752, 21, 1266, 36, 461, 10, 109, 0, 92, 7, 693, 16, 9, 36, 194, 33, 3], [2, 7, 19, 7, 1045, 202, 31, 61, 14, 42, 43, 13, 8, 12242, 18, 41, 131, 9, 24, 18, 25, 37, 1864, 13, 120, 1978, 225, 520, 2502, 11, 16727, 10, 30, 19, 15, 457, 452, 713, 12, 315, 123, 9, 24, 7, 279, 576, 15, 5066, 11, 5041, 26, 12, 934, 13, 2724, 15086, 45, 8, 926, 13, 40, 354, 9, 39, 103, 61, 12, 3437, 9146, 10, 30, 18, 78, 37, 855, 60, 135, 20, 409, 51, 9, 7, 40, 1129, 23, 3010, 187, 60, 11, 9502, 108, 2047, 9, 24, 7, 316, 8, 272, 12, 3689, 23, 1432, 12243, 10, 11, 45, 329, 39, 2033, 104, 36129, 11, 3438, 5826, 11, 5095, 8, 556, 13, 352, 546, 9, 7, 46, 38, 44, 1611, 10, 13, 285, 10, 345, 90, 73, 35, 480, 59, 8, 211, 20, 39, 639, 1698, 80, 4088, 204, 14, 8, 10485, 9, 7, 39, 16311, 869, 14, 177, 816, 10, 11, 20, 15, 64, 240, 108, 1034, 9, 24, 7, 53, 18, 324, 10, 12, 70, 669, 29, 20, 15, 479, 11, 70, 579, 26, 7, 9971, 7, 52613, 132, 2214, 11, 56, 2334, 40, 75, 2574, 556, 9, 3], [2, 7, 129, 12, 216, 23, 75, 355, 13, 7, 985, 31, 10, 18, 863, 69, 93, 19, 9, 18, 197, 35, 83, 42, 2590, 14, 940, 14, 94, 10, 26, 19, 29, 15, 56, 52, 358, 9, 7, 8, 81, 251, 42, 8, 16521, 752, 138, 158, 144, 10, 2293, 18, 95, 35, 776, 8, 399, 137, 10, 8, 950, 46, 456, 1804, 10, 11, 8, 1242, 21, 2584, 21, 7, 985, 81, 9, 7, 63, 32, 161, 1966, 109, 128, 7, 985, 118, 381, 121, 10, 1668, 11, 1290, 20, 327, 117, 21, 286, 21, 10, 0, 52, 10, 115, 5866, 734, 13, 19, 29, 9, 7, 638, 10, 12, 15087, 176, 95, 41, 590, 32, 109, 19, 29, 25, 182, 14, 828, 361, 10, 11, 20, 22, 37, 12, 67, 169, 17, 77, 309, 9, 24, 7, 467, 13, 21, 7, 1469, 21, 7, 13103, 96, 40, 192, 17, 19, 29, 15, 37, 83, 69, 93, 12, 2042, 10, 11, 919, 32, 199, 12, 8367, 355, 10, 32, 58, 35, 387, 14, 2709, 163, 19, 474, 13, 31, 9, 24, 263, 125, 184, 3], [2, 21, 7, 6568, 7, 542, 21, 715, 13, 12, 3670, 343, 176, 7, 985, 255, 49, 747, 17, 53, 27, 12, 145, 129, 7243, 47, 57, 219, 18797, 49, 15, 12, 859, 9, 7, 19, 873, 510, 15, 59, 1337, 1375, 5272, 11, 1337, 1375, 710, 10, 710, 10, 11, 69, 710, 9, 7, 32, 259, 99, 14, 84, 8, 360, 93, 4350, 196, 13, 297, 26, 46, 710, 34, 8, 2535, 10, 710, 11, 317, 398, 1477, 10, 710, 11, 99, 12, 9737, 10, 710, 11, 1249, 11, 1249, 11, 710, 10, 710, 34, 2753, 7179, 10, 2875, 60, 11, 710, 10, 530, 9, 26, 32, 373, 2260, 1065, 28, 158, 14, 601, 9, 7, 8, 1585, 41984, 13, 12, 81, 15, 37, 6968, 14, 1422, 12, 31, 27, 380, 148, 392, 1068, 10, 12, 11869, 196, 10, 11, 74, 213, 10, 74, 888, 10, 74, 369, 55, 1028, 10, 74, 1252, 463, 92, 56, 11870, 0, 9, 36, 11968, 33, 3], [2, 7, 2793, 1736, 25903, 17, 7, 19702, 7, 19413, 41, 12, 216, 23, 2005, 74, 23, 10783, 7506, 27, 8, 7, 49664, 13, 7, 3985, 10, 30, 64, 569, 68, 8, 30313, 7, 5130, 478, 747, 28, 12, 764, 7, 2200, 27, 1860, 542, 10, 2169, 1128, 10, 11, 65, 10083, 1124, 34, 8, 943, 1802, 66, 3626, 7, 1090, 23, 7, 1671, 738, 61, 12, 8048, 16674, 261, 209, 12, 245, 29646, 208, 8, 8038, 9, 7, 16, 15, 70, 1252, 34, 8, 10189, 10, 35315, 23, 53, 1230, 10, 10279, 1585, 34, 81, 9, 7, 3084, 7, 17025, 36, 48, 31107, 564, 6958, 28, 7, 2492, 7, 8731, 33, 428, 16, 12, 8272, 10, 76, 173, 71, 22, 8645, 27, 11082, 7, 1692, 7, 18239, 36, 17, 12363, 114, 23, 72, 11, 3965, 1128, 33, 9, 7, 103, 67, 96, 7, 4282, 7, 18367, 26, 7, 3084, 22, 26410, 3003, 10, 12, 26230, 2049, 49, 22, 1370, 2457, 34, 8, 319, 10, 11, 7, 5742, 7, 7559, 419, 817, 36, 49, 345, 66, 33, 9, 7, 8, 1830, 679, 13, 7, 9906, 247, 57755, 8, 213, 749, 34, 8, 7, 1671, 25291, 10, 11, 65, 13, 8, 759, 376, 38, 203, 399, 9, 7, 47, 8, 75, 13, 8, 219, 0, 1991, 10, 110, 796, 105, 41, 85, 214, 9, 194, 385, 125, 263, 51, 4, 222, 194, 3]...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n",
       "y: ItemList (25000 items)\n",
       "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0...]\n",
       "Path: C:\\Users\\cross-entropy\\.fastai\\data\\imdb\n"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ll"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(ll, open(path/'ll_clas.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "ll = pickle.load(open(path/'ll_clas.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the labels seem consistent with the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(\"xxbos xxmaj airport ' 77 starts as a brand new luxury 747 plane is loaded up with valuable paintings & such belonging to rich businessman xxmaj philip xxmaj stevens ( xxmaj james xxmaj stewart ) who is flying them & a bunch of vip 's to his estate in preparation of it being opened to the public as a museum , also on board is xxmaj stevens daughter xxmaj julie ( xxmaj kathleen xxmaj quinlan ) & her son . xxmaj the luxury jetliner takes off as planned but mid - air the plane is hi - jacked by the co - pilot xxmaj chambers ( xxmaj robert xxmaj foxworth ) & his two accomplice 's xxmaj banker ( xxmaj monte xxmaj markham ) & xxmaj wilson ( xxmaj michael xxmaj pataki ) who knock the passengers & crew out with sleeping gas , they plan to steal the valuable cargo & land on a disused plane strip on an isolated island but while making his descent xxmaj chambers almost hits an oil rig in the xxmaj ocean & loses control of the plane sending it crashing into the sea where it sinks to the bottom right bang in the middle of the xxmaj bermuda xxmaj triangle . xxmaj with air in short supply , water leaking in & having flown over 200 miles off course the problems mount for the survivor 's as they await help with time fast running out ... \\n\\n xxmaj also known under the slightly different tile xxmaj airport 1977 this second sequel to the smash - hit disaster thriller xxmaj airport ( 1970 ) was directed by xxmaj jerry xxmaj jameson & while once again like it 's predecessors i ca n't say xxmaj airport ' 77 is any sort of forgotten classic it is entertaining although not necessarily for the right reasons . xxmaj out of the three xxmaj airport films i have seen so far i actually liked this one the best , just . xxmaj it has my favourite plot of the three with a nice mid - air hi - jacking & then the crashing ( did n't he see the oil rig ? ) & sinking of the 747 ( maybe the makers were trying to cross the original xxmaj airport with another popular disaster flick of the period xxmaj the xxmaj poseidon xxmaj adventure ( 1972 ) ) & submerged is where it stays until the end with a stark dilemma facing those trapped inside , either suffocate when the air runs out or drown as the 747 floods or if any of the doors are opened & it 's a decent idea that could have made for a great little disaster flick but bad unsympathetic character 's , dull dialogue , lethargic set - pieces & a real lack of danger or suspense or tension means this is a missed opportunity . xxmaj while the rather sluggish plot keeps one entertained for 108 odd minutes not that much happens after the plane sinks & there 's not as much urgency as i thought there should have been . xxmaj even when the xxmaj navy become involved things do n't pick up that much with a few shots of huge ships & helicopters flying about but there 's just something lacking here . xxmaj george xxmaj kennedy as the jinxed airline worker xxmaj joe xxmaj patroni is back but only gets a couple of scenes & barely even says anything preferring to just look worried in the background . \\n\\n xxmaj the home video & theatrical version of xxmaj airport ' 77 run 108 minutes while the us tv versions add an extra hour of footage including a new opening credits sequence , many more scenes with xxmaj george xxmaj kennedy as xxmaj patroni , flashbacks to flesh out character 's , longer rescue scenes & the discovery or another couple of dead bodies including the navigator . xxmaj while i would like to see this extra footage i am not sure i could sit through a near three hour cut of xxmaj airport ' 77 . xxmaj as expected the film has dated badly with horrible fashions & interior design choices , i will say no more other than the toy plane model effects are n't great either . xxmaj along with the other two xxmaj airport sequels this takes pride of place in the xxmaj razzie xxmaj award 's xxmaj hall of xxmaj shame although i can think of lots of worse films than this so i reckon that 's a little harsh . xxmaj the action scenes are a little dull unfortunately , the pace is slow & not much excitement or tension is generated which is a shame as i reckon this could have been a pretty good film if made properly . \\n\\n xxmaj the production values are alright if nothing spectacular . xxmaj the acting is n't great , two time xxmaj oscar winner xxmaj jack xxmaj lemmon has said since it was a mistake to star in this , one time xxmaj oscar winner xxmaj james xxmaj stewart looks old & frail , also one time xxmaj oscar winner xxmaj lee xxmaj grant looks drunk while xxmaj sir xxmaj christopher xxmaj lee is given little to do & there are plenty of other familiar faces to look out for too . \\n\\n xxmaj airport ' 77 is the most disaster orientated of the three xxmaj airport films so far & i liked the ideas behind it even if they were a bit silly , the production & bland direction does n't help though & a film about a sunken plane just should n't be this boring or lethargic . xxmaj followed by xxmaj the xxmaj concorde ... xxmaj airport ' 79 ( 1979 ) . xxeos\",\n",
       "  'neg'),\n",
       " (\"xxbos xxmaj to all the miserable people who have done everything from complain about the dialogue , the budget , the this and the that xxrep 4 . who wants to hear it ? if you missed the point of this beyond - beautiful movie , that 's your loss . xxmaj the rest of us who deeply love this movie do not care what you think . i am a xxunk guy who has seen thousands of movies in my life , and this one stands in its own entity , in my book . xxmaj it was not supposed to be a documentary , or a completely factual account of what happened that night . xxmaj it is the most amazing love story ever attempted . i know that it is the cynical 90 's and the millennium has everyone in a tizzy , but come on . xxmaj someone on this comments board complained that it made too much money ! xxmaj how lame is that ? xxmaj it made bundles of money in every civilized country on the planet , and is the top grossing film in the planet . i will gladly side with the majority this time around . xxmaj okay , cynics , time to crawl back under your rock , i am done . xxeos\",\n",
       "  'pos')]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(ll.train.x_obj(i), ll.train.y_obj(i)) for i in [1,12552]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw samplers in notebook 03. For the validation set, we will simply sort the samples by length, and we begin with the longest ones for memory reasons (it's better to always have the biggest tensors first)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from torch.utils.data import Sampler\n",
    "\n",
    "class SortSampler(Sampler):\n",
    "    def __init__(self, data_source, key): self.data_source,self.key = data_source,key\n",
    "    def __len__(self): return len(self.data_source)\n",
    "    def __iter__(self):\n",
    "        return iter(sorted(list(range(len(self.data_source))), key=self.key, reverse=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the training set, we want some kind of randomness on top of this. So first, we shuffle the texts and build megabatches of size `50 * bs`. We sort those megabatches by length before splitting them in 50 minibatches. That way we will have randomized batches of roughly the same length.\n",
    "\n",
    "Then we make sure to have the biggest batch first and shuffle the order of the other batches. We also make sure the last batch stays at the end because its size is probably lower than batch size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class SortishSampler(Sampler):\n",
    "    def __init__(self, data_source, key, bs):\n",
    "        self.data_source,self.key,self.bs = data_source,key,bs\n",
    "\n",
    "    def __len__(self) -> int: return len(self.data_source)\n",
    "\n",
    "    def __iter__(self):\n",
    "        idxs = torch.randperm(len(self.data_source))\n",
    "        megabatches = [idxs[i:i+self.bs*50] for i in range(0, len(idxs), self.bs*50)]\n",
    "        sorted_idx = torch.cat([tensor(sorted(s, key=self.key, reverse=True)) for s in megabatches])\n",
    "        batches = [sorted_idx[i:i+self.bs] for i in range(0, len(sorted_idx), self.bs)]\n",
    "        max_idx = torch.argmax(tensor([self.key(ck[0]) for ck in batches]))  # find the chunk with the largest key,\n",
    "        batches[0],batches[max_idx] = batches[max_idx],batches[0]            # then make sure it goes first.\n",
    "        batch_idxs = torch.randperm(len(batches)-2)\n",
    "        sorted_idx = torch.cat([batches[i+1] for i in batch_idxs]) if len(batches) > 1 else LongTensor([])\n",
    "        sorted_idx = torch.cat([batches[0], sorted_idx, batches[-1]])\n",
    "        return iter(sorted_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Padding: we had the padding token (that has an id of 1) at the end of each sequence to make them all the same size when batching them. Note that we need padding at the end to be able to use `PyTorch` convenience functions that will let us ignore that padding (see 12c)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def pad_collate(samples, pad_idx=1, pad_first=False):\n",
    "    max_len = max([len(s[0]) for s in samples])\n",
    "    res = torch.zeros(len(samples), max_len).long() + pad_idx\n",
    "    for i,s in enumerate(samples):\n",
    "        if pad_first: res[i, -len(s[0]):] = LongTensor(s[0])\n",
    "        else:         res[i, :len(s[0]) ] = LongTensor(s[0])\n",
    "    return res, tensor([s[1] for s in samples])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64\n",
    "train_sampler = SortishSampler(ll.train.x, key=lambda t: len(ll.train[int(t)][0]), bs=bs)\n",
    "train_dl = DataLoader(ll.train, batch_size=bs, sampler=train_sampler, collate_fn=pad_collate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "iter_dl = iter(train_dl)\n",
    "x,y = next(iter_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([3311, 2212, 2206, 1621, 1577], 1000)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lengths = []\n",
    "for i in range(x.size(0)): lengths.append(x.size(1) - (x[i]==1).sum().item())\n",
    "lengths[:5], lengths[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last one is the minimal length. This is the first batch so it has the longest sequence, but if look at the next one that is more random, we see lengths are roughly the sames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([297, 297, 297, 296, 296], 287)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x,y = next(iter_dl)\n",
    "lengths = []\n",
    "for i in range(x.size(0)): lengths.append(x.size(1) - (x[i]==1).sum().item())\n",
    "lengths[:5], lengths[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the padding at the end:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[   2,    7,   54,  ..., 7052,    9,    3],\n",
       "        [   2,   18,  167,  ..., 1210,    9,    3],\n",
       "        [   2,    7, 2215,  ...,  272,    9,    3],\n",
       "        ...,\n",
       "        [   2,   18,  321,  ...,    1,    1,    1],\n",
       "        [   2,    7,   19,  ...,    1,    1,    1],\n",
       "        [   2,    7,   19,  ...,    1,    1,    1]])"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we add a convenience function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_clas_dls(train_ds, valid_ds, bs, **kwargs):\n",
    "    train_sampler = SortishSampler(train_ds.x, key=lambda t: len(train_ds.x[t]), bs=bs)\n",
    "    valid_sampler = SortSampler(valid_ds.x, key=lambda t: len(valid_ds.x[t]))\n",
    "    return (DataLoader(train_ds, batch_size=bs, sampler=train_sampler, collate_fn=pad_collate, **kwargs),\n",
    "            DataLoader(valid_ds, batch_size=bs*2, sampler=valid_sampler, collate_fn=pad_collate, **kwargs))\n",
    "\n",
    "def clas_databunchify(sd, bs, **kwargs):\n",
    "    return DataBunch(*get_clas_dls(sd.train, sd.valid, bs, **kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,bptt = 64,70\n",
    "data = clas_databunchify(ll, bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 12_text.ipynb to exp\\nb_12.py\n"
     ]
    }
   ],
   "source": [
    "!python notebook2script.py 12_text.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}