t81_558_class_10_3_text_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_10_3_text_generation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 10: Time Series in Keras**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 10 Material\n",
    "\n",
    "* Part 10.1: Time Series Data Encoding for Deep Learning [[Video]](https://www.youtube.com/watch?v=dMUmHsktl04&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_1_timeseries.ipynb)\n",
    "* Part 10.2: Programming LSTM with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=wY0dyFgNCgY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_2_lstm.ipynb)\n",
    "* **Part 10.3: Text Generation with Keras and TensorFlow** [[Video]](https://www.youtube.com/watch?v=6ORnRAz3gnA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_3_text_generation.ipynb)\n",
    "* Part 10.4: Image Captioning with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=NmoW_AYWkb4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_4_captioning.ipynb)\n",
    "* Part 10.5: Temporal CNN in Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=i390g8acZwk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_5_temporal_cnn.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 10.3: Text Generation with LSTM\n",
    "\n",
    "Recurrent neural networks are also known for their ability to generate text.  As a result, the output of the neural network can be free-form text.  In this section, we will see how to train an LSTM can  on a textual document, such as classic literature, and learn to output new text that appears to be of the same form as the training material.  If you train your LSTM on [Shakespeare](https://en.wikipedia.org/wiki/William_Shakespeare), it will learn to crank out new prose similar to what Shakespeare had written. \n",
    "\n",
    "Don't get your hopes up.  You are not going to teach your deep neural network to write the next [Pulitzer Prize for Fiction](https://en.wikipedia.org/wiki/Pulitzer_Prize_for_Fiction).  The prose generated by your neural network will be nonsensical.  However, it will usually be nearly grammatically and of a similar style as the source training documents. \n",
    "\n",
    "A neural network generating nonsensical text based on literature may not seem useful at first glance.  However, this technology gets so much interest because it forms the foundation for many more advanced technologies.  The fact that the LSTM will typically learn human grammar from the source document opens a wide range of possibilities. You can use similar technology to complete sentences when a user is entering text.  Simply the ability to output free-form text becomes the foundation of many other technologies.  In the next part, we will use this technique to create a neural network that can write captions for images to describe what is going on in the picture. \n",
    "\n",
    "### Additional Information\n",
    "\n",
    "The following are some of the articles that I found useful in putting this section together.\n",
    "\n",
    "* [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "* [Keras LSTM Generation Example](https://keras.io/examples/lstm_text_generation/)\n",
    "\n",
    "### Character-Level Text Generation\n",
    "\n",
    "There are several different approaches to teaching a neural network to output free-form text.  The most basic question is if you wish the neural network to learn at the word or character level.  In many ways, learning at the character level is the more interesting of the two.  The LSTM is learning to construct its own words without even being shown what a word is.  We will begin with character-level text generation.  In the next module, we will see how we can use nearly the same technique to operate at the word level.  We will implement word-level automatic captioning in the next module.\n",
    "\n",
    "We begin by importing the needed Python packages and defining the sequence length, named **maxlen**.  Time-series neural networks always accept their input as a fixed-length array.  Because you might not use all of the sequence elements, it is common to fill extra elements with zeros.  You will divide the text into sequences of this length, and the neural network will train to predict what comes after this sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import LambdaCallback\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.layers import LSTM\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "from tensorflow.keras.utils import get_file\n",
    "import numpy as np\n",
    "import random\n",
    "import sys\n",
    "import io\n",
    "import requests\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this simple example, we will train the neural network on the classic children's book [Treasure Island](https://en.wikipedia.org/wiki/Treasure_Island).  We begin by loading this text into a Python string and displaying the first 1,000 characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ï»¿The Project Gutenberg EBook of Treasure Island, by Robert Louis Stevenson\r\n",
      "\r\n",
      "This eBook is for the use of anyone anywhere at no cost and with\r\n",
      "almost no restrictions whatsoever.  You may copy it, give it away or\r\n",
      "re-use it under the terms of the Project Gutenberg License included\r\n",
      "with this eBook or online at www.gutenberg.net\r\n",
      "\r\n",
      "\r\n",
      "Title: Treasure Island\r\n",
      "\r\n",
      "Author: Robert Louis Stevenson\r\n",
      "\r\n",
      "Illustrator: Milo Winter\r\n",
      "\r\n",
      "Release Date: January 12, 2009 [EBook #27780]\r\n",
      "\r\n",
      "Language: English\r\n",
      "\r\n",
      "\r\n",
      "*** START OF THIS PROJECT GUTENBERG EBOOK TREASURE ISLAND ***\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "Produced by Juliet Sutherland, Stephen Blundell and the\r\n",
      "Online Distributed Proofreading Team at http://www.pgdp.net\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      " THE ILLUSTRATED CHILDREN'S LIBRARY\r\n",
      "\r\n",
      "\r\n",
      "         _Treasure Island_\r\n",
      "\r\n",
      "       Robert Louis Stevenson\r\n",
      "\r\n",
      "          _Illustrated by_\r\n",
      "            Milo Winter\r\n",
      "\r\n",
      "\r\n",
      "           [Illustration]\r\n",
      "\r\n",
      "\r\n",
      "           GRAMERCY BOOKS\r\n",
      "              NEW YORK\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      " Foreword copyright Â© 1986 by Random House V\n"
     ]
    }
   ],
   "source": [
    "r = requests.get(\"https://data.heatonresearch.com/data/t81-558/text/\"\\\n",
    "                 \"treasure_island.txt\")\n",
    "raw_text = r.text\n",
    "print(raw_text[0:1000])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will extract all unique characters from the text and sort them.  This technique allows us to assign a unique ID to each character.  Because we sorted the characters, these IDs should remain the same.  If we add new characters to the original text, then the IDs would change.  We build two dictionaries.  The first **char2idx** is used to convert a character into its ID.  The second **idx2char** converts an ID back into its character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_text = raw_text.lower()\n",
    "processed_text = re.sub(r'[^\\x00-\\x7f]',r'', processed_text) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus length: 397400\n",
      "total chars: 60\n"
     ]
    }
   ],
   "source": [
    "print('corpus length:', len(processed_text))\n",
    "\n",
    "chars = sorted(list(set(processed_text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to build the actual sequences.  Just like previous neural networks, there will be an $x$ and $y$.  However, for the LSTM, $x$ and $y$ will both be sequences.  The $x$ input will specify the sequences where $y$ are the expected output.  The following code generates all possible sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nb sequences: 132454\n"
     ]
    }
   ],
   "source": [
    "# cut the text in semi-redundant sequences of maxlen characters\n",
    "maxlen = 40\n",
    "step = 3\n",
    "sentences = []\n",
    "next_chars = []\n",
    "for i in range(0, len(processed_text) - maxlen, step):\n",
    "    sentences.append(processed_text[i: i + maxlen])\n",
    "    next_chars.append(processed_text[i + maxlen])\n",
    "print('nb sequences:', len(sentences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the project gutenberg ebook of treasure ',\n",
       " ' project gutenberg ebook of treasure isl',\n",
       " 'oject gutenberg ebook of treasure island',\n",
       " 'ct gutenberg ebook of treasure island, b',\n",
       " 'gutenberg ebook of treasure island, by r',\n",
       " 'enberg ebook of treasure island, by robe',\n",
       " 'erg ebook of treasure island, by robert ',\n",
       " ' ebook of treasure island, by robert lou',\n",
       " 'ook of treasure island, by robert louis ',\n",
       " ' of treasure island, by robert louis ste',\n",
       " ' treasure island, by robert louis steven',\n",
       " 'easure island, by robert louis stevenson',\n",
       " 'ure island, by robert louis stevenson\\r\\n\\r',\n",
       " ' island, by robert louis stevenson\\r\\n\\r\\nth',\n",
       " 'land, by robert louis stevenson\\r\\n\\r\\nthis ',\n",
       " 'd, by robert louis stevenson\\r\\n\\r\\nthis ebo',\n",
       " 'by robert louis stevenson\\r\\n\\r\\nthis ebook ',\n",
       " 'robert louis stevenson\\r\\n\\r\\nthis ebook is ',\n",
       " 'ert louis stevenson\\r\\n\\r\\nthis ebook is for',\n",
       " ' louis stevenson\\r\\n\\r\\nthis ebook is for th',\n",
       " 'uis stevenson\\r\\n\\r\\nthis ebook is for the u',\n",
       " ' stevenson\\r\\n\\r\\nthis ebook is for the use ',\n",
       " 'evenson\\r\\n\\r\\nthis ebook is for the use of ',\n",
       " 'nson\\r\\n\\r\\nthis ebook is for the use of any',\n",
       " 'n\\r\\n\\r\\nthis ebook is for the use of anyone',\n",
       " '\\r\\nthis ebook is for the use of anyone an',\n",
       " 'his ebook is for the use of anyone anywh',\n",
       " ' ebook is for the use of anyone anywhere',\n",
       " 'ook is for the use of anyone anywhere at',\n",
       " ' is for the use of anyone anywhere at no',\n",
       " ' for the use of anyone anywhere at no co',\n",
       " 'r the use of anyone anywhere at no cost ',\n",
       " 'he use of anyone anywhere at no cost and',\n",
       " 'use of anyone anywhere at no cost and wi',\n",
       " ' of anyone anywhere at no cost and with\\r',\n",
       " ' anyone anywhere at no cost and with\\r\\nal',\n",
       " 'yone anywhere at no cost and with\\r\\nalmos',\n",
       " 'e anywhere at no cost and with\\r\\nalmost n',\n",
       " 'nywhere at no cost and with\\r\\nalmost no r',\n",
       " 'here at no cost and with\\r\\nalmost no rest',\n",
       " 'e at no cost and with\\r\\nalmost no restric',\n",
       " 't no cost and with\\r\\nalmost no restrictio',\n",
       " 'o cost and with\\r\\nalmost no restrictions ',\n",
       " 'ost and with\\r\\nalmost no restrictions wha',\n",
       " ' and with\\r\\nalmost no restrictions whatso',\n",
       " 'd with\\r\\nalmost no restrictions whatsoeve',\n",
       " 'ith\\r\\nalmost no restrictions whatsoever. ',\n",
       " '\\r\\nalmost no restrictions whatsoever.  yo',\n",
       " 'lmost no restrictions whatsoever.  you m',\n",
       " 'st no restrictions whatsoever.  you may ',\n",
       " 'no restrictions whatsoever.  you may cop',\n",
       " 'restrictions whatsoever.  you may copy i',\n",
       " 'trictions whatsoever.  you may copy it, ',\n",
       " 'ctions whatsoever.  you may copy it, giv',\n",
       " 'ons whatsoever.  you may copy it, give i',\n",
       " ' whatsoever.  you may copy it, give it a',\n",
       " 'atsoever.  you may copy it, give it away',\n",
       " 'oever.  you may copy it, give it away or',\n",
       " 'er.  you may copy it, give it away or\\r\\nr',\n",
       " '  you may copy it, give it away or\\r\\nre-u',\n",
       " 'ou may copy it, give it away or\\r\\nre-use ',\n",
       " 'may copy it, give it away or\\r\\nre-use it ',\n",
       " ' copy it, give it away or\\r\\nre-use it und',\n",
       " 'py it, give it away or\\r\\nre-use it under ',\n",
       " 'it, give it away or\\r\\nre-use it under the',\n",
       " ' give it away or\\r\\nre-use it under the te',\n",
       " 've it away or\\r\\nre-use it under the terms',\n",
       " 'it away or\\r\\nre-use it under the terms of',\n",
       " 'away or\\r\\nre-use it under the terms of th',\n",
       " 'y or\\r\\nre-use it under the terms of the p',\n",
       " 'r\\r\\nre-use it under the terms of the proj',\n",
       " 're-use it under the terms of the project',\n",
       " 'use it under the terms of the project gu',\n",
       " ' it under the terms of the project guten',\n",
       " ' under the terms of the project gutenber',\n",
       " 'der the terms of the project gutenberg l',\n",
       " ' the terms of the project gutenberg lice',\n",
       " 'e terms of the project gutenberg license',\n",
       " 'erms of the project gutenberg license in',\n",
       " 's of the project gutenberg license inclu',\n",
       " 'f the project gutenberg license included',\n",
       " 'he project gutenberg license included\\r\\nw',\n",
       " 'project gutenberg license included\\r\\nwith',\n",
       " 'ject gutenberg license included\\r\\nwith th',\n",
       " 't gutenberg license included\\r\\nwith this ',\n",
       " 'utenberg license included\\r\\nwith this ebo',\n",
       " 'nberg license included\\r\\nwith this ebook ',\n",
       " 'rg license included\\r\\nwith this ebook or ',\n",
       " 'license included\\r\\nwith this ebook or onl',\n",
       " 'ense included\\r\\nwith this ebook or online',\n",
       " 'e included\\r\\nwith this ebook or online at',\n",
       " 'ncluded\\r\\nwith this ebook or online at ww',\n",
       " 'uded\\r\\nwith this ebook or online at www.g',\n",
       " 'd\\r\\nwith this ebook or online at www.gute',\n",
       " 'with this ebook or online at www.gutenbe',\n",
       " 'h this ebook or online at www.gutenberg.',\n",
       " 'his ebook or online at www.gutenberg.net',\n",
       " ' ebook or online at www.gutenberg.net\\r\\n\\r',\n",
       " 'ook or online at www.gutenberg.net\\r\\n\\r\\n\\r\\n',\n",
       " ' or online at www.gutenberg.net\\r\\n\\r\\n\\r\\ntit',\n",
       " ' online at www.gutenberg.net\\r\\n\\r\\n\\r\\ntitle:',\n",
       " 'line at www.gutenberg.net\\r\\n\\r\\n\\r\\ntitle: tr',\n",
       " 'e at www.gutenberg.net\\r\\n\\r\\n\\r\\ntitle: treas',\n",
       " 't www.gutenberg.net\\r\\n\\r\\n\\r\\ntitle: treasure',\n",
       " 'ww.gutenberg.net\\r\\n\\r\\n\\r\\ntitle: treasure is',\n",
       " 'gutenberg.net\\r\\n\\r\\n\\r\\ntitle: treasure islan',\n",
       " 'enberg.net\\r\\n\\r\\n\\r\\ntitle: treasure island\\r\\n',\n",
       " 'erg.net\\r\\n\\r\\n\\r\\ntitle: treasure island\\r\\n\\r\\na',\n",
       " '.net\\r\\n\\r\\n\\r\\ntitle: treasure island\\r\\n\\r\\nauth',\n",
       " 't\\r\\n\\r\\n\\r\\ntitle: treasure island\\r\\n\\r\\nauthor:',\n",
       " '\\r\\n\\r\\ntitle: treasure island\\r\\n\\r\\nauthor: ro',\n",
       " '\\ntitle: treasure island\\r\\n\\r\\nauthor: rober',\n",
       " 'tle: treasure island\\r\\n\\r\\nauthor: robert l',\n",
       " ': treasure island\\r\\n\\r\\nauthor: robert loui',\n",
       " 'reasure island\\r\\n\\r\\nauthor: robert louis s',\n",
       " 'sure island\\r\\n\\r\\nauthor: robert louis stev',\n",
       " 'e island\\r\\n\\r\\nauthor: robert louis stevens',\n",
       " 'sland\\r\\n\\r\\nauthor: robert louis stevenson\\r',\n",
       " 'nd\\r\\n\\r\\nauthor: robert louis stevenson\\r\\n\\r\\n',\n",
       " '\\n\\r\\nauthor: robert louis stevenson\\r\\n\\r\\nill',\n",
       " 'author: robert louis stevenson\\r\\n\\r\\nillust',\n",
       " 'hor: robert louis stevenson\\r\\n\\r\\nillustrat',\n",
       " ': robert louis stevenson\\r\\n\\r\\nillustrator:',\n",
       " 'obert louis stevenson\\r\\n\\r\\nillustrator: mi',\n",
       " 'rt louis stevenson\\r\\n\\r\\nillustrator: milo ',\n",
       " 'louis stevenson\\r\\n\\r\\nillustrator: milo win',\n",
       " 'is stevenson\\r\\n\\r\\nillustrator: milo winter',\n",
       " 'stevenson\\r\\n\\r\\nillustrator: milo winter\\r\\n\\r',\n",
       " 'venson\\r\\n\\r\\nillustrator: milo winter\\r\\n\\r\\nre',\n",
       " 'son\\r\\n\\r\\nillustrator: milo winter\\r\\n\\r\\nrelea',\n",
       " '\\r\\n\\r\\nillustrator: milo winter\\r\\n\\r\\nrelease ',\n",
       " '\\nillustrator: milo winter\\r\\n\\r\\nrelease dat',\n",
       " 'lustrator: milo winter\\r\\n\\r\\nrelease date: ',\n",
       " 'trator: milo winter\\r\\n\\r\\nrelease date: jan',\n",
       " 'tor: milo winter\\r\\n\\r\\nrelease date: januar',\n",
       " ': milo winter\\r\\n\\r\\nrelease date: january 1',\n",
       " 'ilo winter\\r\\n\\r\\nrelease date: january 12, ',\n",
       " ' winter\\r\\n\\r\\nrelease date: january 12, 200',\n",
       " 'nter\\r\\n\\r\\nrelease date: january 12, 2009 [',\n",
       " 'r\\r\\n\\r\\nrelease date: january 12, 2009 [ebo',\n",
       " '\\r\\nrelease date: january 12, 2009 [ebook ',\n",
       " 'elease date: january 12, 2009 [ebook #27',\n",
       " 'ase date: january 12, 2009 [ebook #27780',\n",
       " ' date: january 12, 2009 [ebook #27780]\\r\\n',\n",
       " 'te: january 12, 2009 [ebook #27780]\\r\\n\\r\\nl',\n",
       " ' january 12, 2009 [ebook #27780]\\r\\n\\r\\nlang',\n",
       " 'nuary 12, 2009 [ebook #27780]\\r\\n\\r\\nlanguag',\n",
       " 'ry 12, 2009 [ebook #27780]\\r\\n\\r\\nlanguage: ',\n",
       " '12, 2009 [ebook #27780]\\r\\n\\r\\nlanguage: eng',\n",
       " ' 2009 [ebook #27780]\\r\\n\\r\\nlanguage: englis',\n",
       " '09 [ebook #27780]\\r\\n\\r\\nlanguage: english\\r\\n',\n",
       " '[ebook #27780]\\r\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r',\n",
       " 'ook #27780]\\r\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r\\n**',\n",
       " ' #27780]\\r\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r\\n*** s',\n",
       " '7780]\\r\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r\\n*** star',\n",
       " '0]\\r\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r\\n*** start o',\n",
       " '\\n\\r\\nlanguage: english\\r\\n\\r\\n\\r\\n*** start of t',\n",
       " 'language: english\\r\\n\\r\\n\\r\\n*** start of this',\n",
       " 'guage: english\\r\\n\\r\\n\\r\\n*** start of this pr',\n",
       " 'ge: english\\r\\n\\r\\n\\r\\n*** start of this proje',\n",
       " ' english\\r\\n\\r\\n\\r\\n*** start of this project ',\n",
       " 'glish\\r\\n\\r\\n\\r\\n*** start of this project gut',\n",
       " 'sh\\r\\n\\r\\n\\r\\n*** start of this project gutenb',\n",
       " '\\n\\r\\n\\r\\n*** start of this project gutenberg',\n",
       " '\\r\\n*** start of this project gutenberg eb',\n",
       " '** start of this project gutenberg ebook',\n",
       " 'start of this project gutenberg ebook tr',\n",
       " 'rt of this project gutenberg ebook treas',\n",
       " 'of this project gutenberg ebook treasure',\n",
       " 'this project gutenberg ebook treasure is',\n",
       " 's project gutenberg ebook treasure islan',\n",
       " 'roject gutenberg ebook treasure island *',\n",
       " 'ect gutenberg ebook treasure island ***\\r',\n",
       " ' gutenberg ebook treasure island ***\\r\\n\\r\\n',\n",
       " 'tenberg ebook treasure island ***\\r\\n\\r\\n\\r\\n\\r',\n",
       " 'berg ebook treasure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n',\n",
       " 'g ebook treasure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\npro',\n",
       " 'book treasure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduc',\n",
       " 'k treasure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced ',\n",
       " 'reasure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by ',\n",
       " 'sure island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by jul',\n",
       " 'e island ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by juliet',\n",
       " 'sland ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by juliet su',\n",
       " 'nd ***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by juliet suthe',\n",
       " '***\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by juliet sutherla',\n",
       " '\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nproduced by juliet sutherland,',\n",
       " '\\n\\r\\n\\r\\n\\r\\nproduced by juliet sutherland, st',\n",
       " '\\r\\n\\r\\nproduced by juliet sutherland, steph',\n",
       " '\\nproduced by juliet sutherland, stephen ',\n",
       " 'oduced by juliet sutherland, stephen blu',\n",
       " 'ced by juliet sutherland, stephen blunde',\n",
       " ' by juliet sutherland, stephen blundell ',\n",
       " ' juliet sutherland, stephen blundell and',\n",
       " 'liet sutherland, stephen blundell and th',\n",
       " 't sutherland, stephen blundell and the\\r\\n',\n",
       " 'utherland, stephen blundell and the\\r\\nonl',\n",
       " 'erland, stephen blundell and the\\r\\nonline',\n",
       " 'and, stephen blundell and the\\r\\nonline di',\n",
       " ', stephen blundell and the\\r\\nonline distr',\n",
       " 'tephen blundell and the\\r\\nonline distribu',\n",
       " 'hen blundell and the\\r\\nonline distributed',\n",
       " ' blundell and the\\r\\nonline distributed pr',\n",
       " 'undell and the\\r\\nonline distributed proof',\n",
       " 'ell and the\\r\\nonline distributed proofrea',\n",
       " ' and the\\r\\nonline distributed proofreadin',\n",
       " 'd the\\r\\nonline distributed proofreading t',\n",
       " 'he\\r\\nonline distributed proofreading team',\n",
       " '\\nonline distributed proofreading team at',\n",
       " 'line distributed proofreading team at ht',\n",
       " 'e distributed proofreading team at http:',\n",
       " 'istributed proofreading team at http://w',\n",
       " 'ributed proofreading team at http://www.',\n",
       " 'uted proofreading team at http://www.pgd',\n",
       " 'd proofreading team at http://www.pgdp.n',\n",
       " 'roofreading team at http://www.pgdp.net\\r',\n",
       " 'freading team at http://www.pgdp.net\\r\\n\\r\\n',\n",
       " 'ading team at http://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r',\n",
       " 'ng team at http://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n',\n",
       " 'team at http://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r',\n",
       " 'm at http://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n',\n",
       " 't http://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r',\n",
       " 'ttp://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n t',\n",
       " '://www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the ',\n",
       " 'www.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the ill',\n",
       " '.pgdp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illust',\n",
       " 'dp.net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrat',\n",
       " 'net\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrated ',\n",
       " '\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrated chi',\n",
       " '\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrated childr',\n",
       " \"\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrated children'\",\n",
       " \"\\n\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n the illustrated children's l\",\n",
       " \"\\r\\n\\r\\n\\r\\n\\r\\n the illustrated children's libr\",\n",
       " \"\\n\\r\\n\\r\\n the illustrated children's library\",\n",
       " \"\\r\\n the illustrated children's library\\r\\n\\r\",\n",
       " \"the illustrated children's library\\r\\n\\r\\n\\r\\n\",\n",
       " \" illustrated children's library\\r\\n\\r\\n\\r\\n   \",\n",
       " \"lustrated children's library\\r\\n\\r\\n\\r\\n      \",\n",
       " \"trated children's library\\r\\n\\r\\n\\r\\n         \",\n",
       " \"ted children's library\\r\\n\\r\\n\\r\\n         _tr\",\n",
       " \" children's library\\r\\n\\r\\n\\r\\n         _treas\",\n",
       " \"ildren's library\\r\\n\\r\\n\\r\\n         _treasure\",\n",
       " \"ren's library\\r\\n\\r\\n\\r\\n         _treasure is\",\n",
       " \"'s library\\r\\n\\r\\n\\r\\n         _treasure islan\",\n",
       " 'library\\r\\n\\r\\n\\r\\n         _treasure island_\\r',\n",
       " 'rary\\r\\n\\r\\n\\r\\n         _treasure island_\\r\\n\\r\\n',\n",
       " 'y\\r\\n\\r\\n\\r\\n         _treasure island_\\r\\n\\r\\n   ',\n",
       " '\\r\\n\\r\\n         _treasure island_\\r\\n\\r\\n      ',\n",
       " '\\n         _treasure island_\\r\\n\\r\\n       ro',\n",
       " '       _treasure island_\\r\\n\\r\\n       rober',\n",
       " '    _treasure island_\\r\\n\\r\\n       robert l',\n",
       " ' _treasure island_\\r\\n\\r\\n       robert loui',\n",
       " 'reasure island_\\r\\n\\r\\n       robert louis s',\n",
       " 'sure island_\\r\\n\\r\\n       robert louis stev',\n",
       " 'e island_\\r\\n\\r\\n       robert louis stevens',\n",
       " 'sland_\\r\\n\\r\\n       robert louis stevenson\\r',\n",
       " 'nd_\\r\\n\\r\\n       robert louis stevenson\\r\\n\\r\\n',\n",
       " '\\r\\n\\r\\n       robert louis stevenson\\r\\n\\r\\n   ',\n",
       " '\\n       robert louis stevenson\\r\\n\\r\\n      ',\n",
       " '     robert louis stevenson\\r\\n\\r\\n         ',\n",
       " '  robert louis stevenson\\r\\n\\r\\n          _i',\n",
       " 'obert louis stevenson\\r\\n\\r\\n          _illu',\n",
       " 'rt louis stevenson\\r\\n\\r\\n          _illustr',\n",
       " 'louis stevenson\\r\\n\\r\\n          _illustrate',\n",
       " 'is stevenson\\r\\n\\r\\n          _illustrated b',\n",
       " 'stevenson\\r\\n\\r\\n          _illustrated by_\\r',\n",
       " 'venson\\r\\n\\r\\n          _illustrated by_\\r\\n  ',\n",
       " 'son\\r\\n\\r\\n          _illustrated by_\\r\\n     ',\n",
       " '\\r\\n\\r\\n          _illustrated by_\\r\\n        ',\n",
       " '\\n          _illustrated by_\\r\\n           ',\n",
       " '        _illustrated by_\\r\\n            mi',\n",
       " '     _illustrated by_\\r\\n            milo ',\n",
       " '  _illustrated by_\\r\\n            milo win',\n",
       " 'illustrated by_\\r\\n            milo winter',\n",
       " 'ustrated by_\\r\\n            milo winter\\r\\n\\r',\n",
       " 'rated by_\\r\\n            milo winter\\r\\n\\r\\n\\r\\n',\n",
       " 'ed by_\\r\\n            milo winter\\r\\n\\r\\n\\r\\n   ',\n",
       " 'by_\\r\\n            milo winter\\r\\n\\r\\n\\r\\n      ',\n",
       " '\\r\\n            milo winter\\r\\n\\r\\n\\r\\n         ',\n",
       " '           milo winter\\r\\n\\r\\n\\r\\n           [',\n",
       " '        milo winter\\r\\n\\r\\n\\r\\n           [ill',\n",
       " '     milo winter\\r\\n\\r\\n\\r\\n           [illust',\n",
       " '  milo winter\\r\\n\\r\\n\\r\\n           [illustrat',\n",
       " 'ilo winter\\r\\n\\r\\n\\r\\n           [illustration',\n",
       " ' winter\\r\\n\\r\\n\\r\\n           [illustration]\\r\\n',\n",
       " 'nter\\r\\n\\r\\n\\r\\n           [illustration]\\r\\n\\r\\n\\r',\n",
       " 'r\\r\\n\\r\\n\\r\\n           [illustration]\\r\\n\\r\\n\\r\\n  ',\n",
       " '\\r\\n\\r\\n           [illustration]\\r\\n\\r\\n\\r\\n     ',\n",
       " '\\n           [illustration]\\r\\n\\r\\n\\r\\n        ',\n",
       " '         [illustration]\\r\\n\\r\\n\\r\\n           ',\n",
       " '      [illustration]\\r\\n\\r\\n\\r\\n           gra',\n",
       " '   [illustration]\\r\\n\\r\\n\\r\\n           gramer',\n",
       " '[illustration]\\r\\n\\r\\n\\r\\n           gramercy ',\n",
       " 'lustration]\\r\\n\\r\\n\\r\\n           gramercy boo',\n",
       " 'tration]\\r\\n\\r\\n\\r\\n           gramercy books\\r',\n",
       " 'tion]\\r\\n\\r\\n\\r\\n           gramercy books\\r\\n  ',\n",
       " 'n]\\r\\n\\r\\n\\r\\n           gramercy books\\r\\n     ',\n",
       " '\\n\\r\\n\\r\\n           gramercy books\\r\\n        ',\n",
       " '\\r\\n           gramercy books\\r\\n           ',\n",
       " '          gramercy books\\r\\n              ',\n",
       " '       gramercy books\\r\\n              new',\n",
       " '    gramercy books\\r\\n              new yo',\n",
       " ' gramercy books\\r\\n              new york\\r',\n",
       " 'amercy books\\r\\n              new york\\r\\n\\r\\n',\n",
       " 'rcy books\\r\\n              new york\\r\\n\\r\\n\\r\\n\\r',\n",
       " ' books\\r\\n              new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n',\n",
       " 'oks\\r\\n              new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n fo',\n",
       " '\\r\\n              new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n forew',\n",
       " '             new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword',\n",
       " '          new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword co',\n",
       " '       new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyr',\n",
       " '    new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyrigh',\n",
       " ' new york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyright  ',\n",
       " 'w york\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyright  198',\n",
       " 'ork\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyright  1986 b',\n",
       " '\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n foreword copyright  1986 by r',\n",
       " '\\n\\r\\n\\r\\n\\r\\n foreword copyright  1986 by rand',\n",
       " '\\r\\n\\r\\n foreword copyright  1986 by random ',\n",
       " '\\n foreword copyright  1986 by random hou',\n",
       " 'oreword copyright  1986 by random house ',\n",
       " 'word copyright  1986 by random house val',\n",
       " 'd copyright  1986 by random house value ',\n",
       " 'opyright  1986 by random house value pub',\n",
       " 'right  1986 by random house value publis',\n",
       " 'ht  1986 by random house value publishin',\n",
       " ' 1986 by random house value publishing\\r\\n',\n",
       " '86 by random house value publishing\\r\\n co',\n",
       " 'by random house value publishing\\r\\n color',\n",
       " 'random house value publishing\\r\\n color il',\n",
       " 'dom house value publishing\\r\\n color illus',\n",
       " ' house value publishing\\r\\n color illustra',\n",
       " 'use value publishing\\r\\n color illustratio',\n",
       " ' value publishing\\r\\n color illustrations ',\n",
       " 'lue publishing\\r\\n color illustrations by ',\n",
       " ' publishing\\r\\n color illustrations by mil',\n",
       " 'blishing\\r\\n color illustrations by milo w',\n",
       " 'shing\\r\\n color illustrations by milo wint',\n",
       " 'ng\\r\\n color illustrations by milo winter ',\n",
       " '\\n color illustrations by milo winter cop',\n",
       " 'olor illustrations by milo winter copyri',\n",
       " 'r illustrations by milo winter copyright',\n",
       " 'llustrations by milo winter copyright  1',\n",
       " 'strations by milo winter copyright  1915',\n",
       " 'ations by milo winter copyright  1915, 1',\n",
       " 'ons by milo winter copyright  1915, 1943',\n",
       " ' by milo winter copyright  1915, 1943 by',\n",
       " ' milo winter copyright  1915, 1943 by ra',\n",
       " 'lo winter copyright  1915, 1943 by rand\\r',\n",
       " 'winter copyright  1915, 1943 by rand\\r\\n m',\n",
       " 'ter copyright  1915, 1943 by rand\\r\\n mcna',\n",
       " ' copyright  1915, 1943 by rand\\r\\n mcnally',\n",
       " 'pyright  1915, 1943 by rand\\r\\n mcnally & ',\n",
       " 'ight  1915, 1943 by rand\\r\\n mcnally & com',\n",
       " 't  1915, 1943 by rand\\r\\n mcnally & compan',\n",
       " '1915, 1943 by rand\\r\\n mcnally & company\\r\\n',\n",
       " '5, 1943 by rand\\r\\n mcnally & company\\r\\n al',\n",
       " '1943 by rand\\r\\n mcnally & company\\r\\n all r',\n",
       " '3 by rand\\r\\n mcnally & company\\r\\n all righ',\n",
       " 'y rand\\r\\n mcnally & company\\r\\n all rights ',\n",
       " 'and\\r\\n mcnally & company\\r\\n all rights res',\n",
       " '\\r\\n mcnally & company\\r\\n all rights reserv',\n",
       " 'mcnally & company\\r\\n all rights reserved.',\n",
       " 'ally & company\\r\\n all rights reserved.\\r\\n\\r',\n",
       " 'y & company\\r\\n all rights reserved.\\r\\n\\r\\n t',\n",
       " ' company\\r\\n all rights reserved.\\r\\n\\r\\n this',\n",
       " 'mpany\\r\\n all rights reserved.\\r\\n\\r\\n this 20',\n",
       " 'ny\\r\\n all rights reserved.\\r\\n\\r\\n this 2002 ',\n",
       " '\\n all rights reserved.\\r\\n\\r\\n this 2002 edi',\n",
       " 'll rights reserved.\\r\\n\\r\\n this 2002 editio',\n",
       " 'rights reserved.\\r\\n\\r\\n this 2002 edition p',\n",
       " 'hts reserved.\\r\\n\\r\\n this 2002 edition publ',\n",
       " ' reserved.\\r\\n\\r\\n this 2002 edition publish',\n",
       " 'served.\\r\\n\\r\\n this 2002 edition published ',\n",
       " 'ved.\\r\\n\\r\\n this 2002 edition published by ',\n",
       " '.\\r\\n\\r\\n this 2002 edition published by gra',\n",
       " '\\r\\n this 2002 edition published by gramer',\n",
       " 'this 2002 edition published by gramercy ',\n",
       " 's 2002 edition published by gramercy boo',\n",
       " '002 edition published by gramercy books,',\n",
       " ' edition published by gramercy books, an',\n",
       " 'ition published by gramercy books, an im',\n",
       " 'on published by gramercy books, an impri',\n",
       " 'published by gramercy books, an imprint ',\n",
       " 'lished by gramercy books, an imprint of ',\n",
       " 'hed by gramercy books, an imprint of ran',\n",
       " ' by gramercy books, an imprint of random',\n",
       " ' gramercy books, an imprint of random\\r\\n ',\n",
       " 'amercy books, an imprint of random\\r\\n hou',\n",
       " 'rcy books, an imprint of random\\r\\n house ',\n",
       " ' books, an imprint of random\\r\\n house val',\n",
       " 'oks, an imprint of random\\r\\n house value ',\n",
       " ', an imprint of random\\r\\n house value pub',\n",
       " 'n imprint of random\\r\\n house value publis',\n",
       " 'mprint of random\\r\\n house value publishin',\n",
       " 'int of random\\r\\n house value publishing, ',\n",
       " ' of random\\r\\n house value publishing, a d',\n",
       " ' random\\r\\n house value publishing, a divi',\n",
       " 'ndom\\r\\n house value publishing, a divisio',\n",
       " 'm\\r\\n house value publishing, a division o',\n",
       " ' house value publishing, a division of r',\n",
       " 'use value publishing, a division of rand',\n",
       " ' value publishing, a division of random ',\n",
       " 'lue publishing, a division of random hou',\n",
       " ' publishing, a division of random house,',\n",
       " 'blishing, a division of random house, in',\n",
       " 'shing, a division of random house, inc.,',\n",
       " 'ng, a division of random house, inc., 28',\n",
       " ' a division of random house, inc., 280 p',\n",
       " 'division of random house, inc., 280 park',\n",
       " 'ision of random house, inc., 280 park\\r\\n ',\n",
       " 'on of random house, inc., 280 park\\r\\n ave',\n",
       " 'of random house, inc., 280 park\\r\\n avenue',\n",
       " 'random house, inc., 280 park\\r\\n avenue, n',\n",
       " 'dom house, inc., 280 park\\r\\n avenue, new ',\n",
       " ' house, inc., 280 park\\r\\n avenue, new yor',\n",
       " 'use, inc., 280 park\\r\\n avenue, new york, ',\n",
       " ', inc., 280 park\\r\\n avenue, new york, ny ',\n",
       " 'nc., 280 park\\r\\n avenue, new york, ny 100',\n",
       " ', 280 park\\r\\n avenue, new york, ny 10017.',\n",
       " '80 park\\r\\n avenue, new york, ny 10017.\\r\\n\\r',\n",
       " 'park\\r\\n avenue, new york, ny 10017.\\r\\n\\r\\n g',\n",
       " 'k\\r\\n avenue, new york, ny 10017.\\r\\n\\r\\n gram',\n",
       " ' avenue, new york, ny 10017.\\r\\n\\r\\n gramerc',\n",
       " 'enue, new york, ny 10017.\\r\\n\\r\\n gramercy i',\n",
       " 'e, new york, ny 10017.\\r\\n\\r\\n gramercy is a',\n",
       " 'new york, ny 10017.\\r\\n\\r\\n gramercy is a re',\n",
       " ' york, ny 10017.\\r\\n\\r\\n gramercy is a regis',\n",
       " 'rk, ny 10017.\\r\\n\\r\\n gramercy is a register',\n",
       " ' ny 10017.\\r\\n\\r\\n gramercy is a registered ',\n",
       " ' 10017.\\r\\n\\r\\n gramercy is a registered tra',\n",
       " '017.\\r\\n\\r\\n gramercy is a registered tradem',\n",
       " '.\\r\\n\\r\\n gramercy is a registered trademark',\n",
       " '\\r\\n gramercy is a registered trademark an',\n",
       " 'gramercy is a registered trademark and t',\n",
       " 'mercy is a registered trademark and the ',\n",
       " 'cy is a registered trademark and the col',\n",
       " 'is a registered trademark and the coloph',\n",
       " 'a registered trademark and the colophon ',\n",
       " 'egistered trademark and the colophon is ',\n",
       " 'stered trademark and the colophon is a t',\n",
       " 'red trademark and the colophon is a trad',\n",
       " ' trademark and the colophon is a tradema',\n",
       " 'ademark and the colophon is a trademark ',\n",
       " 'mark and the colophon is a trademark of\\r',\n",
       " 'k and the colophon is a trademark of\\r\\n r',\n",
       " 'nd the colophon is a trademark of\\r\\n rand',\n",
       " 'the colophon is a trademark of\\r\\n random ',\n",
       " ' colophon is a trademark of\\r\\n random hou',\n",
       " 'lophon is a trademark of\\r\\n random house,',\n",
       " 'hon is a trademark of\\r\\n random house, in',\n",
       " ' is a trademark of\\r\\n random house, inc.\\r',\n",
       " ' a trademark of\\r\\n random house, inc.\\r\\n\\r\\n',\n",
       " 'trademark of\\r\\n random house, inc.\\r\\n\\r\\n pr',\n",
       " 'demark of\\r\\n random house, inc.\\r\\n\\r\\n print',\n",
       " 'ark of\\r\\n random house, inc.\\r\\n\\r\\n printed ',\n",
       " ' of\\r\\n random house, inc.\\r\\n\\r\\n printed and',\n",
       " '\\r\\n random house, inc.\\r\\n\\r\\n printed and bo',\n",
       " 'random house, inc.\\r\\n\\r\\n printed and bound',\n",
       " 'dom house, inc.\\r\\n\\r\\n printed and bound in',\n",
       " ' house, inc.\\r\\n\\r\\n printed and bound in th',\n",
       " 'use, inc.\\r\\n\\r\\n printed and bound in the u',\n",
       " ', inc.\\r\\n\\r\\n printed and bound in the unit',\n",
       " 'nc.\\r\\n\\r\\n printed and bound in the united ',\n",
       " '\\r\\n\\r\\n printed and bound in the united sta',\n",
       " '\\n printed and bound in the united states',\n",
       " 'rinted and bound in the united states of',\n",
       " 'ted and bound in the united states of am',\n",
       " ' and bound in the united states of ameri',\n",
       " 'd bound in the united states of america\\r',\n",
       " 'ound in the united states of america\\r\\n\\r\\n',\n",
       " 'd in the united states of america\\r\\n\\r\\n co',\n",
       " 'n the united states of america\\r\\n\\r\\n cover',\n",
       " 'he united states of america\\r\\n\\r\\n cover de',\n",
       " 'united states of america\\r\\n\\r\\n cover desig',\n",
       " 'ted states of america\\r\\n\\r\\n cover design b',\n",
       " ' states of america\\r\\n\\r\\n cover design by j',\n",
       " 'ates of america\\r\\n\\r\\n cover design by judy',\n",
       " 's of america\\r\\n\\r\\n cover design by judy fu',\n",
       " 'f america\\r\\n\\r\\n cover design by judy fucci',\n",
       " 'merica\\r\\n\\r\\n cover design by judy fucci, s',\n",
       " 'ica\\r\\n\\r\\n cover design by judy fucci, stud',\n",
       " '\\r\\n\\r\\n cover design by judy fucci, studio ',\n",
       " '\\n cover design by judy fucci, studio gra',\n",
       " 'over design by judy fucci, studio graphi',\n",
       " 'r design by judy fucci, studio graphix, ',\n",
       " 'esign by judy fucci, studio graphix, inc',\n",
       " 'gn by judy fucci, studio graphix, inc.\\r\\n',\n",
       " 'by judy fucci, studio graphix, inc.\\r\\n\\r\\n ',\n",
       " 'judy fucci, studio graphix, inc.\\r\\n\\r\\n ran',\n",
       " 'y fucci, studio graphix, inc.\\r\\n\\r\\n random',\n",
       " 'ucci, studio graphix, inc.\\r\\n\\r\\n random ho',\n",
       " 'i, studio graphix, inc.\\r\\n\\r\\n random house',\n",
       " 'studio graphix, inc.\\r\\n\\r\\n random house\\r\\n ',\n",
       " 'dio graphix, inc.\\r\\n\\r\\n random house\\r\\n new',\n",
       " ' graphix, inc.\\r\\n\\r\\n random house\\r\\n new yo',\n",
       " 'aphix, inc.\\r\\n\\r\\n random house\\r\\n new york ',\n",
       " 'ix, inc.\\r\\n\\r\\n random house\\r\\n new york  to',\n",
       " ' inc.\\r\\n\\r\\n random house\\r\\n new york  toron',\n",
       " 'c.\\r\\n\\r\\n random house\\r\\n new york  toronto ',\n",
       " '\\n\\r\\n random house\\r\\n new york  toronto  lo',\n",
       " ' random house\\r\\n new york  toronto  londo',\n",
       " 'ndom house\\r\\n new york  toronto  london  ',\n",
       " 'm house\\r\\n new york  toronto  london  syd',\n",
       " 'ouse\\r\\n new york  toronto  london  sydney',\n",
       " 'e\\r\\n new york  toronto  london  sydney  a',\n",
       " ' new york  toronto  london  sydney  auck',\n",
       " 'w york  toronto  london  sydney  aucklan',\n",
       " 'ork  toronto  london  sydney  auckland\\r\\n',\n",
       " '  toronto  london  sydney  auckland\\r\\n ww',\n",
       " 'oronto  london  sydney  auckland\\r\\n www.r',\n",
       " 'nto  london  sydney  auckland\\r\\n www.rand',\n",
       " '  london  sydney  auckland\\r\\n www.randomh',\n",
       " 'ondon  sydney  auckland\\r\\n www.randomhous',\n",
       " 'on  sydney  auckland\\r\\n www.randomhouse.c',\n",
       " ' sydney  auckland\\r\\n www.randomhouse.com\\r',\n",
       " 'dney  auckland\\r\\n www.randomhouse.com\\r\\n\\r\\n',\n",
       " 'y  auckland\\r\\n www.randomhouse.com\\r\\n\\r\\n\\r\\n ',\n",
       " 'auckland\\r\\n www.randomhouse.com\\r\\n\\r\\n\\r\\n lib',\n",
       " 'kland\\r\\n www.randomhouse.com\\r\\n\\r\\n\\r\\n librar',\n",
       " 'nd\\r\\n www.randomhouse.com\\r\\n\\r\\n\\r\\n library o',\n",
       " '\\n www.randomhouse.com\\r\\n\\r\\n\\r\\n library of c',\n",
       " 'ww.randomhouse.com\\r\\n\\r\\n\\r\\n library of cong',\n",
       " 'randomhouse.com\\r\\n\\r\\n\\r\\n library of congres',\n",
       " 'domhouse.com\\r\\n\\r\\n\\r\\n library of congress c',\n",
       " 'house.com\\r\\n\\r\\n\\r\\n library of congress cata',\n",
       " 'se.com\\r\\n\\r\\n\\r\\n library of congress catalog',\n",
       " 'com\\r\\n\\r\\n\\r\\n library of congress cataloging',\n",
       " '\\r\\n\\r\\n\\r\\n library of congress cataloging-in',\n",
       " '\\n\\r\\n library of congress cataloging-in-pu',\n",
       " ' library of congress cataloging-in-publi',\n",
       " 'brary of congress cataloging-in-publicat',\n",
       " 'ry of congress cataloging-in-publication',\n",
       " 'of congress cataloging-in-publication da',\n",
       " 'congress cataloging-in-publication data\\r',\n",
       " 'gress cataloging-in-publication data\\r\\n\\r\\n',\n",
       " 'ss cataloging-in-publication data\\r\\n\\r\\n st',\n",
       " 'cataloging-in-publication data\\r\\n\\r\\n steve',\n",
       " 'aloging-in-publication data\\r\\n\\r\\n stevenso',\n",
       " 'ging-in-publication data\\r\\n\\r\\n stevenson, ',\n",
       " 'g-in-publication data\\r\\n\\r\\n stevenson, rob',\n",
       " 'n-publication data\\r\\n\\r\\n stevenson, robert',\n",
       " 'ublication data\\r\\n\\r\\n stevenson, robert lo',\n",
       " 'ication data\\r\\n\\r\\n stevenson, robert louis',\n",
       " 'tion data\\r\\n\\r\\n stevenson, robert louis, 1',\n",
       " 'n data\\r\\n\\r\\n stevenson, robert louis, 1850',\n",
       " 'ata\\r\\n\\r\\n stevenson, robert louis, 1850-18',\n",
       " '\\r\\n\\r\\n stevenson, robert louis, 1850-1894.',\n",
       " '\\n stevenson, robert louis, 1850-1894.\\r\\n ',\n",
       " 'tevenson, robert louis, 1850-1894.\\r\\n    ',\n",
       " 'enson, robert louis, 1850-1894.\\r\\n     tr',\n",
       " 'on, robert louis, 1850-1894.\\r\\n     treas',\n",
       " ' robert louis, 1850-1894.\\r\\n     treasure',\n",
       " 'bert louis, 1850-1894.\\r\\n     treasure is',\n",
       " 't louis, 1850-1894.\\r\\n     treasure islan',\n",
       " 'ouis, 1850-1894.\\r\\n     treasure island/r',\n",
       " 's, 1850-1894.\\r\\n     treasure island/robe',\n",
       " '1850-1894.\\r\\n     treasure island/robert ',\n",
       " '0-1894.\\r\\n     treasure island/robert lou',\n",
       " '894.\\r\\n     treasure island/robert louis ',\n",
       " '.\\r\\n     treasure island/robert louis ste',\n",
       " '     treasure island/robert louis steven',\n",
       " '  treasure island/robert louis stevenson',\n",
       " 'reasure island/robert louis stevenson; i',\n",
       " 'sure island/robert louis stevenson; illu',\n",
       " 'e island/robert louis stevenson; illustr',\n",
       " 'sland/robert louis stevenson; illustrate',\n",
       " 'nd/robert louis stevenson; illustrated i',\n",
       " 'robert louis stevenson; illustrated in c',\n",
       " 'ert louis stevenson; illustrated in colo',\n",
       " ' louis stevenson; illustrated in color b',\n",
       " 'uis stevenson; illustrated in color by\\r\\n',\n",
       " ' stevenson; illustrated in color by\\r\\n   ',\n",
       " 'evenson; illustrated in color by\\r\\n   mil',\n",
       " 'nson; illustrated in color by\\r\\n   milo w',\n",
       " 'n; illustrated in color by\\r\\n   milo wint',\n",
       " 'illustrated in color by\\r\\n   milo winter.',\n",
       " 'ustrated in color by\\r\\n   milo winter.\\r\\n ',\n",
       " 'rated in color by\\r\\n   milo winter.\\r\\n    ',\n",
       " 'ed in color by\\r\\n   milo winter.\\r\\n       ',\n",
       " 'in color by\\r\\n   milo winter.\\r\\n       p. ',\n",
       " 'color by\\r\\n   milo winter.\\r\\n       p. cm.',\n",
       " 'or by\\r\\n   milo winter.\\r\\n       p. cm.--(',\n",
       " 'by\\r\\n   milo winter.\\r\\n       p. cm.--(ill',\n",
       " '\\n   milo winter.\\r\\n       p. cm.--(illust',\n",
       " ' milo winter.\\r\\n       p. cm.--(illustrat',\n",
       " 'lo winter.\\r\\n       p. cm.--(illustrated ',\n",
       " 'winter.\\r\\n       p. cm.--(illustrated chi',\n",
       " 'ter.\\r\\n       p. cm.--(illustrated childr',\n",
       " \".\\r\\n       p. cm.--(illustrated children'\",\n",
       " \"       p. cm.--(illustrated children's l\",\n",
       " \"    p. cm.--(illustrated children's libr\",\n",
       " \" p. cm.--(illustrated children's library\",\n",
       " \" cm.--(illustrated children's library)\\r\\n\",\n",
       " \".--(illustrated children's library)\\r\\n   \",\n",
       " \"(illustrated children's library)\\r\\n     o\",\n",
       " \"lustrated children's library)\\r\\n     orig\",\n",
       " \"trated children's library)\\r\\n     origina\",\n",
       " \"ted children's library)\\r\\n     originally\",\n",
       " \" children's library)\\r\\n     originally pu\",\n",
       " \"ildren's library)\\r\\n     originally publi\",\n",
       " \"ren's library)\\r\\n     originally publishe\",\n",
       " \"'s library)\\r\\n     originally published: \",\n",
       " 'library)\\r\\n     originally published: new',\n",
       " 'rary)\\r\\n     originally published: new yo',\n",
       " 'y)\\r\\n     originally published: new york:',\n",
       " '\\n     originally published: new york: ch',\n",
       " '   originally published: new york: child',\n",
       " 'originally published: new york: children',\n",
       " \"ginally published: new york: children's \",\n",
       " \"ally published: new york: children's cla\",\n",
       " \"y published: new york: children's classi\",\n",
       " \"ublished: new york: children's classics,\",\n",
       " \"ished: new york: children's classics, 19\",\n",
       " \"ed: new york: children's classics, 1986.\",\n",
       " \" new york: children's classics, 1986.\\r\\n \",\n",
       " \"w york: children's classics, 1986.\\r\\n    \",\n",
       " \"ork: children's classics, 1986.\\r\\n     su\",\n",
       " \": children's classics, 1986.\\r\\n     summa\",\n",
       " \"hildren's classics, 1986.\\r\\n     summary:\",\n",
       " \"dren's classics, 1986.\\r\\n     summary: wh\",\n",
       " \"n's classics, 1986.\\r\\n     summary: while\",\n",
       " ' classics, 1986.\\r\\n     summary: while go',\n",
       " 'assics, 1986.\\r\\n     summary: while going',\n",
       " 'ics, 1986.\\r\\n     summary: while going th',\n",
       " ', 1986.\\r\\n     summary: while going throu',\n",
       " '986.\\r\\n     summary: while going through ',\n",
       " '.\\r\\n     summary: while going through the',\n",
       " '     summary: while going through the po',\n",
       " '  summary: while going through the posse',\n",
       " 'ummary: while going through the possessi',\n",
       " 'ary: while going through the possessions',\n",
       " ': while going through the possessions of',\n",
       " 'hile going through the possessions of a ',\n",
       " 'e going through the possessions of a dec',\n",
       " 'oing through the possessions of a deceas',\n",
       " 'g through the possessions of a deceased ',\n",
       " 'hrough the possessions of a deceased gue',\n",
       " 'ugh the possessions of a deceased guest\\r',\n",
       " ' the possessions of a deceased guest\\r\\n  ',\n",
       " 'e possessions of a deceased guest\\r\\n   wh',\n",
       " 'ossessions of a deceased guest\\r\\n   who o',\n",
       " 'essions of a deceased guest\\r\\n   who owed',\n",
       " 'ions of a deceased guest\\r\\n   who owed th',\n",
       " 's of a deceased guest\\r\\n   who owed them ',\n",
       " 'f a deceased guest\\r\\n   who owed them mon',\n",
       " ' deceased guest\\r\\n   who owed them money,',\n",
       " 'ceased guest\\r\\n   who owed them money, th',\n",
       " 'sed guest\\r\\n   who owed them money, the m',\n",
       " ' guest\\r\\n   who owed them money, the mist',\n",
       " 'est\\r\\n   who owed them money, the mistres',\n",
       " '\\r\\n   who owed them money, the mistress o',\n",
       " '  who owed them money, the mistress of t',\n",
       " 'ho owed them money, the mistress of the ',\n",
       " 'owed them money, the mistress of the inn',\n",
       " 'd them money, the mistress of the inn an',\n",
       " 'hem money, the mistress of the inn and h',\n",
       " ' money, the mistress of the inn and her ',\n",
       " 'ney, the mistress of the inn and her son',\n",
       " ', the mistress of the inn and her son fi',\n",
       " 'he mistress of the inn and her son find ',\n",
       " 'mistress of the inn and her son find a\\r\\n',\n",
       " 'tress of the inn and her son find a\\r\\n   ',\n",
       " 'ss of the inn and her son find a\\r\\n   tre',\n",
       " 'of the inn and her son find a\\r\\n   treasu',\n",
       " 'the inn and her son find a\\r\\n   treasure ',\n",
       " ' inn and her son find a\\r\\n   treasure map',\n",
       " 'n and her son find a\\r\\n   treasure map th',\n",
       " 'nd her son find a\\r\\n   treasure map that ',\n",
       " 'her son find a\\r\\n   treasure map that lea',\n",
       " ' son find a\\r\\n   treasure map that leads ',\n",
       " 'n find a\\r\\n   treasure map that leads the',\n",
       " 'ind a\\r\\n   treasure map that leads them t',\n",
       " ' a\\r\\n   treasure map that leads them to a',\n",
       " '\\n   treasure map that leads them to a pi',\n",
       " ' treasure map that leads them to a pirat',\n",
       " \"easure map that leads them to a pirate's\",\n",
       " \"ure map that leads them to a pirate's fo\",\n",
       " \" map that leads them to a pirate's fortu\",\n",
       " \"p that leads them to a pirate's fortune.\",\n",
       " \"hat leads them to a pirate's fortune.\\r\\n \",\n",
       " \" leads them to a pirate's fortune.\\r\\n    \",\n",
       " \"ads them to a pirate's fortune.\\r\\n     is\",\n",
       " \" them to a pirate's fortune.\\r\\n     isbn \",\n",
       " \"em to a pirate's fortune.\\r\\n     isbn 0-5\",\n",
       " \"to a pirate's fortune.\\r\\n     isbn 0-517-\",\n",
       " \"a pirate's fortune.\\r\\n     isbn 0-517-221\",\n",
       " \"irate's fortune.\\r\\n     isbn 0-517-22114-\",\n",
       " \"te's fortune.\\r\\n     isbn 0-517-22114-4\\r\\n\",\n",
       " 's fortune.\\r\\n     isbn 0-517-22114-4\\r\\n   ',\n",
       " 'ortune.\\r\\n     isbn 0-517-22114-4\\r\\n     [',\n",
       " 'une.\\r\\n     isbn 0-517-22114-4\\r\\n     [1. ',\n",
       " '.\\r\\n     isbn 0-517-22114-4\\r\\n     [1. bur',\n",
       " '     isbn 0-517-22114-4\\r\\n     [1. buried',\n",
       " '  isbn 0-517-22114-4\\r\\n     [1. buried tr',\n",
       " 'sbn 0-517-22114-4\\r\\n     [1. buried treas',\n",
       " ' 0-517-22114-4\\r\\n     [1. buried treasure',\n",
       " '517-22114-4\\r\\n     [1. buried treasure--f',\n",
       " '-22114-4\\r\\n     [1. buried treasure--fict',\n",
       " '114-4\\r\\n     [1. buried treasure--fiction',\n",
       " '-4\\r\\n     [1. buried treasure--fiction. 2',\n",
       " '\\n     [1. buried treasure--fiction. 2. p',\n",
       " '   [1. buried treasure--fiction. 2. pira',\n",
       " '[1. buried treasure--fiction. 2. pirates',\n",
       " ' buried treasure--fiction. 2. pirates--f',\n",
       " 'ried treasure--fiction. 2. pirates--fict',\n",
       " 'd treasure--fiction. 2. pirates--fiction',\n",
       " 'reasure--fiction. 2. pirates--fiction. 3',\n",
       " 'sure--fiction. 2. pirates--fiction. 3. a',\n",
       " 'e--fiction. 2. pirates--fiction. 3. adve',\n",
       " 'fiction. 2. pirates--fiction. 3. adventu',\n",
       " 'tion. 2. pirates--fiction. 3. adventure\\r',\n",
       " 'n. 2. pirates--fiction. 3. adventure\\r\\n  ',\n",
       " '2. pirates--fiction. 3. adventure\\r\\n   an',\n",
       " 'pirates--fiction. 3. adventure\\r\\n   and a',\n",
       " 'ates--fiction. 3. adventure\\r\\n   and adve',\n",
       " 's--fiction. 3. adventure\\r\\n   and adventu',\n",
       " 'fiction. 3. adventure\\r\\n   and adventures',\n",
       " 'tion. 3. adventure\\r\\n   and adventures--f',\n",
       " 'n. 3. adventure\\r\\n   and adventures--fict',\n",
       " '3. adventure\\r\\n   and adventures--fiction',\n",
       " 'adventure\\r\\n   and adventures--fiction. 4',\n",
       " 'enture\\r\\n   and adventures--fiction. 4. c',\n",
       " 'ure\\r\\n   and adventures--fiction. 4. cari',\n",
       " '\\r\\n   and adventures--fiction. 4. caribbe',\n",
       " '  and adventures--fiction. 4. caribbean ',\n",
       " 'nd adventures--fiction. 4. caribbean are',\n",
       " 'adventures--fiction. 4. caribbean area--',\n",
       " 'entures--fiction. 4. caribbean area--his',\n",
       " 'ures--fiction. 4. caribbean area--histor',\n",
       " 's--fiction. 4. caribbean area--history--',\n",
       " 'fiction. 4. caribbean area--history--18t',\n",
       " 'tion. 4. caribbean area--history--18th\\r\\n',\n",
       " 'n. 4. caribbean area--history--18th\\r\\n   ',\n",
       " '4. caribbean area--history--18th\\r\\n   cen',\n",
       " 'caribbean area--history--18th\\r\\n   centur',\n",
       " 'ibbean area--history--18th\\r\\n   century--',\n",
       " 'ean area--history--18th\\r\\n   century--fic',\n",
       " ' area--history--18th\\r\\n   century--fictio',\n",
       " 'ea--history--18th\\r\\n   century--fiction.]',\n",
       " '-history--18th\\r\\n   century--fiction.] i.',\n",
       " 'story--18th\\r\\n   century--fiction.] i. wi',\n",
       " 'ry--18th\\r\\n   century--fiction.] i. winte',\n",
       " '-18th\\r\\n   century--fiction.] i. winter, ',\n",
       " 'th\\r\\n   century--fiction.] i. winter, mil',\n",
       " '\\n   century--fiction.] i. winter, milo, ',\n",
       " ' century--fiction.] i. winter, milo, 188',\n",
       " 'ntury--fiction.] i. winter, milo, 1888-1',\n",
       " 'ry--fiction.] i. winter, milo, 1888-1956',\n",
       " '-fiction.] i. winter, milo, 1888-1956, i',\n",
       " 'ction.] i. winter, milo, 1888-1956, ill.',\n",
       " 'on.] i. winter, milo, 1888-1956, ill. ii',\n",
       " '] i. winter, milo, 1888-1956, ill. ii. t',\n",
       " '. winter, milo, 1888-1956, ill. ii. titl',\n",
       " 'inter, milo, 1888-1956, ill. ii. title.\\r',\n",
       " 'er, milo, 1888-1956, ill. ii. title.\\r\\n  ',\n",
       " ' milo, 1888-1956, ill. ii. title.\\r\\n   ii',\n",
       " 'lo, 1888-1956, ill. ii. title.\\r\\n   iii. ',\n",
       " ' 1888-1956, ill. ii. title.\\r\\n   iii. ser',\n",
       " '88-1956, ill. ii. title.\\r\\n   iii. series',\n",
       " '1956, ill. ii. title.\\r\\n   iii. series.\\r\\n',\n",
       " '6, ill. ii. title.\\r\\n   iii. series.\\r\\n\\r\\n ',\n",
       " 'ill. ii. title.\\r\\n   iii. series.\\r\\n\\r\\n   p',\n",
       " '. ii. title.\\r\\n   iii. series.\\r\\n\\r\\n   pz7.',\n",
       " 'i. title.\\r\\n   iii. series.\\r\\n\\r\\n   pz7.s84',\n",
       " 'title.\\r\\n   iii. series.\\r\\n\\r\\n   pz7.s8482 ',\n",
       " 'le.\\r\\n   iii. series.\\r\\n\\r\\n   pz7.s8482 tr ',\n",
       " '\\r\\n   iii. series.\\r\\n\\r\\n   pz7.s8482 tr 200',\n",
       " '  iii. series.\\r\\n\\r\\n   pz7.s8482 tr 2002\\r\\n',\n",
       " 'ii. series.\\r\\n\\r\\n   pz7.s8482 tr 2002\\r\\n   ',\n",
       " ' series.\\r\\n\\r\\n   pz7.s8482 tr 2002\\r\\n   [fi',\n",
       " 'ries.\\r\\n\\r\\n   pz7.s8482 tr 2002\\r\\n   [fic]-',\n",
       " 's.\\r\\n\\r\\n   pz7.s8482 tr 2002\\r\\n   [fic]--dc',\n",
       " '\\n\\r\\n   pz7.s8482 tr 2002\\r\\n   [fic]--dc21\\r',\n",
       " '   pz7.s8482 tr 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n',\n",
       " 'pz7.s8482 tr 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n   ',\n",
       " '.s8482 tr 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n      ',\n",
       " '482 tr 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n         ',\n",
       " ' tr 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n            ',\n",
       " ' 2002\\r\\n   [fic]--dc21\\r\\n\\r\\n               ',\n",
       " '02\\r\\n   [fic]--dc21\\r\\n\\r\\n                  ',\n",
       " '\\n   [fic]--dc21\\r\\n\\r\\n                     ',\n",
       " ' [fic]--dc21\\r\\n\\r\\n                        ',\n",
       " 'ic]--dc21\\r\\n\\r\\n                           ',\n",
       " '--dc21\\r\\n\\r\\n                              ',\n",
       " 'c21\\r\\n\\r\\n                                 ',\n",
       " '\\r\\n\\r\\n                                    ',\n",
       " '\\n                                       ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                       2',\n",
       " '                                    2002',\n",
       " '                                 2002023',\n",
       " '                              2002023301',\n",
       " '                           2002023301\\r\\n ',\n",
       " '                        2002023301\\r\\n 9 8',\n",
       " '                     2002023301\\r\\n 9 8 7 ',\n",
       " '                  2002023301\\r\\n 9 8 7 6 5',\n",
       " '               2002023301\\r\\n 9 8 7 6 5 4 ',\n",
       " '            2002023301\\r\\n 9 8 7 6 5 4 3 2',\n",
       " '         2002023301\\r\\n 9 8 7 6 5 4 3 2 1\\r',\n",
       " '      2002023301\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n',\n",
       " '   2002023301\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\nt',\n",
       " '2002023301\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntran',\n",
       " '2023301\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscr',\n",
       " '3301\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscribe',\n",
       " \"1\\r\\n 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's\",\n",
       " \" 9 8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's no\",\n",
       " \"8 7 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's note:\",\n",
       " \" 6 5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's note:\\r\\n\\r\",\n",
       " \"5 4 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's note:\\r\\n\\r\\n  \",\n",
       " \" 3 2 1\\r\\n\\r\\n\\r\\ntranscriber's note:\\r\\n\\r\\n    m\",\n",
       " \"2 1\\r\\n\\r\\n\\r\\ntranscriber's note:\\r\\n\\r\\n    mino\",\n",
       " \"\\r\\n\\r\\n\\r\\ntranscriber's note:\\r\\n\\r\\n    minor s\",\n",
       " \"\\n\\r\\ntranscriber's note:\\r\\n\\r\\n    minor spel\",\n",
       " \"transcriber's note:\\r\\n\\r\\n    minor spellin\",\n",
       " \"nscriber's note:\\r\\n\\r\\n    minor spelling a\",\n",
       " \"riber's note:\\r\\n\\r\\n    minor spelling and \",\n",
       " \"er's note:\\r\\n\\r\\n    minor spelling and typ\",\n",
       " 's note:\\r\\n\\r\\n    minor spelling and typogr',\n",
       " 'ote:\\r\\n\\r\\n    minor spelling and typograph',\n",
       " ':\\r\\n\\r\\n    minor spelling and typographica',\n",
       " '\\r\\n    minor spelling and typographical e',\n",
       " '   minor spelling and typographical erro',\n",
       " 'minor spelling and typographical errors ',\n",
       " 'or spelling and typographical errors hav',\n",
       " 'spelling and typographical errors have b',\n",
       " 'lling and typographical errors have been',\n",
       " 'ng and typographical errors have been co',\n",
       " 'and typographical errors have been corre',\n",
       " ' typographical errors have been correcte',\n",
       " 'pographical errors have been corrected w',\n",
       " 'raphical errors have been corrected with',\n",
       " 'hical errors have been corrected without',\n",
       " 'al errors have been corrected without\\r\\n ',\n",
       " 'errors have been corrected without\\r\\n    ',\n",
       " 'ors have been corrected without\\r\\n    not',\n",
       " ' have been corrected without\\r\\n    note. ',\n",
       " 've been corrected without\\r\\n    note. dia',\n",
       " 'been corrected without\\r\\n    note. dialec',\n",
       " 'n corrected without\\r\\n    note. dialect a',\n",
       " 'orrected without\\r\\n    note. dialect and ',\n",
       " 'ected without\\r\\n    note. dialect and var',\n",
       " 'ed without\\r\\n    note. dialect and varian',\n",
       " 'without\\r\\n    note. dialect and variant s',\n",
       " 'hout\\r\\n    note. dialect and variant spel',\n",
       " 't\\r\\n    note. dialect and variant spellin',\n",
       " '    note. dialect and variant spellings ',\n",
       " ' note. dialect and variant spellings hav',\n",
       " 'te. dialect and variant spellings have b',\n",
       " ' dialect and variant spellings have been',\n",
       " 'alect and variant spellings have been re',\n",
       " 'ct and variant spellings have been retai',\n",
       " 'and variant spellings have been retained',\n",
       " ' variant spellings have been retained, w',\n",
       " 'riant spellings have been retained, whil',\n",
       " 'nt spellings have been retained, whilst\\r',\n",
       " 'spellings have been retained, whilst\\r\\n  ',\n",
       " 'llings have been retained, whilst\\r\\n    i',\n",
       " 'ngs have been retained, whilst\\r\\n    inco',\n",
       " ' have been retained, whilst\\r\\n    inconsi',\n",
       " 've been retained, whilst\\r\\n    inconsiste',\n",
       " 'been retained, whilst\\r\\n    inconsistent ',\n",
       " 'n retained, whilst\\r\\n    inconsistent hyp',\n",
       " 'etained, whilst\\r\\n    inconsistent hyphen',\n",
       " 'ined, whilst\\r\\n    inconsistent hyphenati',\n",
       " 'd, whilst\\r\\n    inconsistent hyphenation ',\n",
       " 'whilst\\r\\n    inconsistent hyphenation has',\n",
       " 'lst\\r\\n    inconsistent hyphenation has be',\n",
       " '\\r\\n    inconsistent hyphenation has been ',\n",
       " '   inconsistent hyphenation has been sta',\n",
       " 'inconsistent hyphenation has been standa',\n",
       " 'onsistent hyphenation has been standardi',\n",
       " 'istent hyphenation has been standardised',\n",
       " 'ent hyphenation has been standardised. c',\n",
       " ' hyphenation has been standardised. colo',\n",
       " 'phenation has been standardised. color p',\n",
       " 'nation has been standardised. color plat',\n",
       " 'ion has been standardised. color plates ',\n",
       " ' has been standardised. color plates hav',\n",
       " 's been standardised. color plates have\\r\\n',\n",
       " 'een standardised. color plates have\\r\\n   ',\n",
       " ' standardised. color plates have\\r\\n    be',\n",
       " 'andardised. color plates have\\r\\n    been ',\n",
       " 'ardised. color plates have\\r\\n    been rep',\n",
       " 'ised. color plates have\\r\\n    been reposi',\n",
       " 'd. color plates have\\r\\n    been repositio',\n",
       " 'color plates have\\r\\n    been repositioned',\n",
       " 'or plates have\\r\\n    been repositioned ac',\n",
       " 'plates have\\r\\n    been repositioned accor',\n",
       " 'tes have\\r\\n    been repositioned accordin',\n",
       " ' have\\r\\n    been repositioned according t',\n",
       " 've\\r\\n    been repositioned according to t',\n",
       " '\\n    been repositioned according to thei',\n",
       " '  been repositioned according to their c',\n",
       " 'een repositioned according to their capt',\n",
       " ' repositioned according to their caption',\n",
       " 'positioned according to their captions; ',\n",
       " 'itioned according to their captions; the',\n",
       " \"oned according to their captions; the 'c\",\n",
       " \"d according to their captions; the 'colo\",\n",
       " \"ccording to their captions; the 'color p\",\n",
       " \"rding to their captions; the 'color plat\",\n",
       " \"ng to their captions; the 'color plates'\",\n",
       " \"to their captions; the 'color plates'\\r\\n \",\n",
       " \"their captions; the 'color plates'\\r\\n    \",\n",
       " \"ir captions; the 'color plates'\\r\\n    lis\",\n",
       " \"captions; the 'color plates'\\r\\n    listin\",\n",
       " \"tions; the 'color plates'\\r\\n    listing r\",\n",
       " \"ns; the 'color plates'\\r\\n    listing rema\",\n",
       " \" the 'color plates'\\r\\n    listing remains\",\n",
       " \"e 'color plates'\\r\\n    listing remains as\",\n",
       " \"color plates'\\r\\n    listing remains as pr\",\n",
       " \"or plates'\\r\\n    listing remains as print\",\n",
       " \"plates'\\r\\n    listing remains as printed \",\n",
       " \"tes'\\r\\n    listing remains as printed to \",\n",
       " \"'\\r\\n    listing remains as printed to ind\",\n",
       " '    listing remains as printed to indica',\n",
       " ' listing remains as printed to indicate ',\n",
       " 'sting remains as printed to indicate the',\n",
       " 'ng remains as printed to indicate the or',\n",
       " 'remains as printed to indicate the origi',\n",
       " 'ains as printed to indicate the original',\n",
       " 's as printed to indicate the original lo',\n",
       " 's printed to indicate the original locat',\n",
       " 'rinted to indicate the original location',\n",
       " 'ted to indicate the original locations.\\r',\n",
       " ' to indicate the original locations.\\r\\n\\r\\n',\n",
       " ' indicate the original locations.\\r\\n\\r\\n\\r\\n\\r',\n",
       " 'dicate the original locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\n',\n",
       " 'ate the original locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncon',\n",
       " ' the original locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\nconten',\n",
       " 'e original locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r',\n",
       " 'riginal locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n',\n",
       " 'inal locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n ',\n",
       " 'l locations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n    ',\n",
       " 'ocations.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n       ',\n",
       " 'tions.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n          ',\n",
       " 'ns.\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n             ',\n",
       " '\\r\\n\\r\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n                ',\n",
       " '\\n\\r\\n\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n                   ',\n",
       " '\\r\\n\\r\\ncontents\\r\\n\\r\\n\\r\\n                      ',\n",
       " '\\ncontents\\r\\n\\r\\n\\r\\n                         ',\n",
       " 'ntents\\r\\n\\r\\n\\r\\n                            ',\n",
       " 'nts\\r\\n\\r\\n\\r\\n                               ',\n",
       " '\\r\\n\\r\\n\\r\\n                                  ',\n",
       " '\\n\\r\\n                                     ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                        ',\n",
       " '                                     pag',\n",
       " '                                  page\\r\\n',\n",
       " '                               page\\r\\n _t',\n",
       " '                            page\\r\\n _to t',\n",
       " '                         page\\r\\n _to the ',\n",
       " '                      page\\r\\n _to the hes',\n",
       " '                   page\\r\\n _to the hesita',\n",
       " '                page\\r\\n _to the hesitatin',\n",
       " '             page\\r\\n _to the hesitating p',\n",
       " '          page\\r\\n _to the hesitating purc',\n",
       " '       page\\r\\n _to the hesitating purchas',\n",
       " '    page\\r\\n _to the hesitating purchaser_',\n",
       " ' page\\r\\n _to the hesitating purchaser_   ',\n",
       " 'ge\\r\\n _to the hesitating purchaser_      ',\n",
       " '\\n _to the hesitating purchaser_         ',\n",
       " 'to the hesitating purchaser_            ',\n",
       " 'the hesitating purchaser_               ',\n",
       " ' hesitating purchaser_                  ',\n",
       " 'sitating purchaser_                     ',\n",
       " 'ating purchaser_                        ',\n",
       " 'ng purchaser_                           ',\n",
       " 'purchaser_                             _',\n",
       " 'chaser_                             _vii',\n",
       " 'ser_                             _viii_\\r',\n",
       " '_                             _viii_\\r\\n _',\n",
       " '                           _viii_\\r\\n _lis',\n",
       " '                        _viii_\\r\\n _list o',\n",
       " '                     _viii_\\r\\n _list of c',\n",
       " '                  _viii_\\r\\n _list of colo',\n",
       " '               _viii_\\r\\n _list of color p',\n",
       " '            _viii_\\r\\n _list of color plat',\n",
       " '         _viii_\\r\\n _list of color plates_',\n",
       " '      _viii_\\r\\n _list of color plates_   ',\n",
       " '   _viii_\\r\\n _list of color plates_      ',\n",
       " '_viii_\\r\\n _list of color plates_         ',\n",
       " 'ii_\\r\\n _list of color plates_            ',\n",
       " '\\r\\n _list of color plates_               ',\n",
       " '_list of color plates_                  ',\n",
       " 'st of color plates_                     ',\n",
       " 'of color plates_                        ',\n",
       " 'color plates_                           ',\n",
       " 'or plates_                              ',\n",
       " ...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorization...\n"
     ]
    }
   ],
   "source": [
    "print('Vectorization...')\n",
    "x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        x[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(132454, 40, 60)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(132454, 60)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dummy variables for $y$ are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False,  True, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False,  True, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False,  True, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False,  True, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False,  True, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False,  True, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "         True, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False,  True, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False,  True, False,\n",
       "        False, False, False, False, False, False],\n",
       "       [False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False, False, False, False, False, False, False, False, False,\n",
       "        False,  True, False, False, False, False]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create the neural network.  This neural network's primary feature is the LSTM layer, which allows the sequences to be processed.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n"
     ]
    }
   ],
   "source": [
    "# build the model: a single LSTM\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n",
    "model.add(Dense(len(chars), activation='softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm (LSTM)                  (None, 128)               96768     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 60)                7740      \n",
      "=================================================================\n",
      "Total params: 104,508\n",
      "Trainable params: 104,508\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LSTM will produce new text character by character.  We will need to sample the correct letter from the LSTM predictions each time.  The **sample** function accepts the following two parameters:\n",
    "\n",
    "* **preds** - The output neurons.\n",
    "* **temperature** - 1.0 is the most conservative, 0.0 is the most confident (willing to make spelling and other errors).\n",
    "\n",
    "The sample function below is essentially performing a [softmax]() on the neural network predictions.  This causes each output neuron to become a probability of its particular letter.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample(preds, temperature=1.0):\n",
    "    # helper function to sample an index from a probability array\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras calls the following function at the end of each training Epoch.  The code generates sample text generations that visually demonstrate the neural network better at text generation.  As the neural network trains, the generations should look more realistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_epoch_end(epoch, _):\n",
    "    # Function invoked at end of each epoch. Prints generated text.\n",
    "    print(\"******************************************************\")\n",
    "    print('----- Generating text after Epoch: %d' % epoch)\n",
    "\n",
    "    start_index = random.randint(0, len(processed_text) - maxlen - 1)\n",
    "    for temperature in [0.2, 0.5, 1.0, 1.2]:\n",
    "        print('----- temperature:', temperature)\n",
    "\n",
    "        generated = ''\n",
    "        sentence = processed_text[start_index: start_index + maxlen]\n",
    "        generated += sentence\n",
    "        print('----- Generating with seed: \"' + sentence + '\"')\n",
    "        sys.stdout.write(generated)\n",
    "\n",
    "        for i in range(400):\n",
    "            x_pred = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(sentence):\n",
    "                x_pred[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(x_pred, verbose=0)[0]\n",
    "            next_index = sample(preds, temperature)\n",
    "            next_char = indices_char[next_index]\n",
    "\n",
    "            generated += next_char\n",
    "            sentence = sentence[1:] + next_char\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to train.  It can take up to an hour to train this network, depending on how fast your computer is.  If you have a GPU available, please make sure to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 132454 samples\n",
      "Epoch 1/60\n",
      "   128/132454 [..............................] - ETA: 35:39******************************************************\n",
      "----- Generating text after Epoch: 0\n",
      "----- temperature: 0.2\n",
      "----- Generating with seed: \"im shouting.\n",
      "\n",
      "but you may suppose i pa\"\n",
      "im shouting.\n",
      "\n",
      "but you may suppose i pa"
     ]
    },
    {
     "ename": "UnknownError",
     "evalue": " [_Derived_]  Fail to find the dnn implementation.\n\t [[{{node CudnnRNN}}]]\n\t [[sequential/lstm/StatefulPartitionedCall]] [Op:__inference_distributed_function_3394]\n\nFunction call stack:\ndistributed_function -> distributed_function -> distributed_function\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mUnknownError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-16-29d39d1d557e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     11\u001b[0m           \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m128\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m           \u001b[0mepochs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m60\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 13\u001b[1;33m           callbacks=[print_callback])\n\u001b[0m",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[0;32m    817\u001b[0m         \u001b[0mmax_queue_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmax_queue_size\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    818\u001b[0m         \u001b[0mworkers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mworkers\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 819\u001b[1;33m         use_multiprocessing=use_multiprocessing)\n\u001b[0m\u001b[0;32m    820\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    821\u001b[0m   def evaluate(self,\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[0;32m    395\u001b[0m                       total_epochs=1)\n\u001b[0;32m    396\u001b[0m                   cbks.make_logs(model, epoch_logs, eval_result, ModeKeys.TEST,\n\u001b[1;32m--> 397\u001b[1;33m                                  prefix='val_')\n\u001b[0m\u001b[0;32m    398\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    399\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhistory\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[1;34m(self, type, value, traceback)\u001b[0m\n\u001b[0;32m    128\u001b[0m                 \u001b[0mvalue\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    129\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 130\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mthrow\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtraceback\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    131\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    132\u001b[0m                 \u001b[1;31m# Suppress StopIteration *unless* it's the same exception that\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2.py\u001b[0m in \u001b[0;36mon_epoch\u001b[1;34m(self, epoch, mode)\u001b[0m\n\u001b[0;32m    769\u001b[0m       \u001b[1;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mModeKeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTRAIN\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    770\u001b[0m         \u001b[1;31m# Epochs only apply to `fit`.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 771\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_epoch_end\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mepoch_logs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    772\u001b[0m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mprogbar\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_epoch_end\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mepoch_logs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    773\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\callbacks.py\u001b[0m in \u001b[0;36mon_epoch_end\u001b[1;34m(self, epoch, logs)\u001b[0m\n\u001b[0;32m    300\u001b[0m     \u001b[0mlogs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlogs\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    301\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0mcallback\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 302\u001b[1;33m       \u001b[0mcallback\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_epoch_end\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlogs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    303\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    304\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0mon_train_batch_begin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlogs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-15-6e1e333ca3e9>\u001b[0m in \u001b[0;36mon_epoch_end\u001b[1;34m(epoch, _)\u001b[0m\n\u001b[0;32m     19\u001b[0m                 \u001b[0mx_pred\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mt\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mchar_indices\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mchar\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m1.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     20\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 21\u001b[1;33m             \u001b[0mpreds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx_pred\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     22\u001b[0m             \u001b[0mnext_index\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msample\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtemperature\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     23\u001b[0m             \u001b[0mnext_char\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mindices_char\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mnext_index\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training.py\u001b[0m in \u001b[0;36mpredict\u001b[1;34m(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[0;32m   1011\u001b[0m         \u001b[0mmax_queue_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmax_queue_size\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1012\u001b[0m         \u001b[0mworkers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mworkers\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1013\u001b[1;33m         use_multiprocessing=use_multiprocessing)\n\u001b[0m\u001b[0;32m   1014\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1015\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0mreset_metrics\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2.py\u001b[0m in \u001b[0;36mpredict\u001b[1;34m(self, model, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[0;32m    496\u001b[0m         \u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mModeKeys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mPREDICT\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    497\u001b[0m         \u001b[0msteps\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0msteps\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmax_queue_size\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmax_queue_size\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 498\u001b[1;33m         workers=workers, use_multiprocessing=use_multiprocessing, **kwargs)\n\u001b[0m\u001b[0;32m    499\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    500\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2.py\u001b[0m in \u001b[0;36m_model_iteration\u001b[1;34m(self, model, mode, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, **kwargs)\u001b[0m\n\u001b[0;32m    473\u001b[0m               \u001b[0mmode\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmode\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    474\u001b[0m               \u001b[0mtraining_context\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtraining_context\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 475\u001b[1;33m               total_epochs=1)\n\u001b[0m\u001b[0;32m    476\u001b[0m           \u001b[0mcbks\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmake_logs\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mepoch_logs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    477\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2.py\u001b[0m in \u001b[0;36mrun_one_epoch\u001b[1;34m(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)\u001b[0m\n\u001b[0;32m    126\u001b[0m         step=step, mode=mode, size=current_batch_size) as batch_logs:\n\u001b[0;32m    127\u001b[0m       \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 128\u001b[1;33m         \u001b[0mbatch_outs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mexecution_function\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    129\u001b[0m       \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mStopIteration\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mOutOfRangeError\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    130\u001b[0m         \u001b[1;31m# TODO(kaftan): File bug about tf function and errors.OutOfRangeError?\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\training_v2_utils.py\u001b[0m in \u001b[0;36mexecution_function\u001b[1;34m(input_fn)\u001b[0m\n\u001b[0;32m     96\u001b[0m     \u001b[1;31m# `numpy` translates Tensors to values in Eager mode.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     97\u001b[0m     return nest.map_structure(_non_none_constant_value,\n\u001b[1;32m---> 98\u001b[1;33m                               distributed_function(input_fn))\n\u001b[0m\u001b[0;32m     99\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    100\u001b[0m   \u001b[1;32mreturn\u001b[0m \u001b[0mexecution_function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, *args, **kwds)\u001b[0m\n\u001b[0;32m    566\u001b[0m         \u001b[0mxla_context\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mExit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    567\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 568\u001b[1;33m       \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    569\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    570\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mtracing_count\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\def_function.py\u001b[0m in \u001b[0;36m_call\u001b[1;34m(self, *args, **kwds)\u001b[0m\n\u001b[0;32m    636\u001b[0m               *args, **kwds)\n\u001b[0;32m    637\u001b[0m       \u001b[1;31m# If we did not create any variables the trace we have is good enough.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 638\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_concrete_stateful_fn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcanon_args\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcanon_kwds\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# pylint: disable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    639\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    640\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mfn_with_cond\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0minner_args\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0minner_kwds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[1;34m(self, args, kwargs)\u001b[0m\n\u001b[0;32m   1609\u001b[0m          if isinstance(t, (ops.Tensor,\n\u001b[0;32m   1610\u001b[0m                            resource_variable_ops.BaseResourceVariable))),\n\u001b[1;32m-> 1611\u001b[1;33m         self.captured_inputs)\n\u001b[0m\u001b[0;32m   1612\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1613\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[1;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[0;32m   1690\u001b[0m       \u001b[1;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1691\u001b[0m       return self._build_call_outputs(self._inference_function.call(\n\u001b[1;32m-> 1692\u001b[1;33m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[0;32m   1693\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n\u001b[0;32m   1694\u001b[0m         \u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\function.py\u001b[0m in \u001b[0;36mcall\u001b[1;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[0;32m    543\u001b[0m               \u001b[0minputs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    544\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"executor_type\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexecutor_type\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"config_proto\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 545\u001b[1;33m               ctx=ctx)\n\u001b[0m\u001b[0;32m    546\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    547\u001b[0m           outputs = execute.execute_with_cancellation(\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow_core\\python\\eager\\execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[1;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[0;32m     65\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     66\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 67\u001b[1;33m     \u001b[0msix\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcore\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     68\u001b[0m   \u001b[1;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     69\u001b[0m     keras_symbolic_tensors = [\n",
      "\u001b[1;32m~\\miniconda3\\envs\\tensorflow\\lib\\site-packages\\six.py\u001b[0m in \u001b[0;36mraise_from\u001b[1;34m(value, from_value)\u001b[0m\n",
      "\u001b[1;31mUnknownError\u001b[0m:  [_Derived_]  Fail to find the dnn implementation.\n\t [[{{node CudnnRNN}}]]\n\t [[sequential/lstm/StatefulPartitionedCall]] [Op:__inference_distributed_function_3394]\n\nFunction call stack:\ndistributed_function -> distributed_function -> distributed_function\n"
     ]
    }
   ],
   "source": [
    "# Ignore useless W0819 warnings generated by TensorFlow 2.0.  Hopefully can remove this ignore in the future.\n",
    "# See https://github.com/tensorflow/tensorflow/issues/31308\n",
    "import logging, os\n",
    "logging.disable(logging.WARNING)\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"\n",
    "\n",
    "# Fit the model\n",
    "print_callback = LambdaCallback(on_epoch_end=on_epoch_end)\n",
    "\n",
    "model.fit(x, y,\n",
    "          batch_size=128,\n",
    "          epochs=60,\n",
    "          callbacks=[print_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}