Character Level RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# THIS IS STILL IN PROGRESS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we're going to be looking at how you can generate text that is similar to the input text that you give the network. This work is inspired by the great [blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) by Andrej Karpathy and most of the code is adapted from [this tutorial](http://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/). The network that we're going to be building is a **character level recurrent neural network**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load in Input Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be loading in a book from Thomas Paine, obtained from this [website](http://www.textfiles.com/etext/NONFICTION/). We'll need to clean up the sentences in the file, mainly removing the extra spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Removes punctuation, parentheses, question marks, etc., and leaves only alphanumeric characters\n",
    "import re\n",
    "strip_special_chars = re.compile(\"[^A-Za-z0-9 ]+\")\n",
    "\n",
    "def cleanSentences(string):\n",
    "    string = string.lower().replace(\"       \", \" \")\n",
    "    string = string.lower().replace(\"     \", \" \")\n",
    "    string = string.lower().replace(\"    \", \" \")\n",
    "    string = string.lower().replace(\"   \", \" \")\n",
    "    string = string.lower().replace(\"  \", \" \")\n",
    "    string = string.lower().replace(\"   \", \" \")\n",
    "    return re.sub(strip_special_chars, \"\", string.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7280\n"
     ]
    }
   ],
   "source": [
    "allText = \"\"\n",
    "with open(\"Data/paine.txt\", \"r\", encoding='utf-8') as f:\n",
    "    lines=f.readlines()\n",
    "    numWords = 0\n",
    "    for line in lines:\n",
    "        allText += (cleanSentences(line))\n",
    "        numWords += len(line.split())\n",
    "    chars = sorted(list(set(allText)))\n",
    "    print (numWords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "40064\n",
      "37\n"
     ]
    }
   ],
   "source": [
    "nChars = len(allText)\n",
    "nVocab = len(chars)\n",
    "seqLength = 100\n",
    "print (nChars)\n",
    "print (nVocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ',\n",
       " '0',\n",
       " '1',\n",
       " '2',\n",
       " '3',\n",
       " '4',\n",
       " '5',\n",
       " '6',\n",
       " '7',\n",
       " '8',\n",
       " '9',\n",
       " 'a',\n",
       " 'b',\n",
       " 'c',\n",
       " 'd',\n",
       " 'e',\n",
       " 'f',\n",
       " 'g',\n",
       " 'h',\n",
       " 'i',\n",
       " 'j',\n",
       " 'k',\n",
       " 'l',\n",
       " 'm',\n",
       " 'n',\n",
       " 'o',\n",
       " 'p',\n",
       " 'q',\n",
       " 'r',\n",
       " 's',\n",
       " 't',\n",
       " 'u',\n",
       " 'v',\n",
       " 'w',\n",
       " 'x',\n",
       " 'y',\n",
       " 'z']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "charToInt = dict((c, i) for i, c in enumerate(chars))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Examples:  39964\n"
     ]
    }
   ],
   "source": [
    "# prepare the dataset of input to output pairs encoded as integers\n",
    "dataX = []\n",
    "dataY = []\n",
    "for i in range(0, nChars - seqLength, 1):\n",
    "    seq_in = allText[i:i + seqLength]\n",
    "    seq_out = allText[i + seqLength]\n",
    "    dataX.append([charToInt[char] for char in seq_in])\n",
    "    dataY.append(charToInt[seq_out])\n",
    "nExamples = len(dataX)\n",
    "print (\"Total Examples: \", nExamples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reshape X to be [samples, time steps, features]\n",
    "X = np.reshape(dataX, (nExamples, seqLength, 1))\n",
    "# normalize\n",
    "X = X / float(nVocab)\n",
    "# one hot encode the output variable\n",
    "y = np.zeros([nExamples, nVocab])\n",
    "for i, example in enumerate(dataY):\n",
    "    lis = np.zeros(nVocab)\n",
    "    lis[example] = 1\n",
    "    y[i] = lis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batchSize = 24\n",
    "lstmUnits = 48\n",
    "iterations = 100000\n",
    "numDimensions = 1\n",
    "numClasses = nVocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "\n",
    "labels = tf.placeholder(tf.float32, [None, numClasses])\n",
    "input_data = tf.placeholder(tf.float32, [None, seqLength, numDimensions])\n",
    "\n",
    "lstmCell = tf.contrib.rnn.BasicLSTMCell(lstmUnits)\n",
    "lstmCell = tf.contrib.rnn.DropoutWrapper(cell=lstmCell, output_keep_prob=0.85)\n",
    "value, _ = tf.nn.dynamic_rnn(lstmCell, input_data, dtype=tf.float32)\n",
    "\n",
    "weight = tf.Variable(tf.truncated_normal([lstmUnits, numClasses]))\n",
    "bias = tf.Variable(tf.constant(0.1, shape=[numClasses]))\n",
    "value = tf.transpose(value, [1, 0, 2])\n",
    "last = tf.gather(value, int(value.get_shape()[0]) - 1)\n",
    "prediction = (tf.matmul(last, weight) + bias)\n",
    "\n",
    "correctPred = tf.equal(tf.argmax(prediction,1), tf.argmax(labels,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correctPred, tf.float32))\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=labels))\n",
    "optimizer = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "\n",
    "sess = tf.InteractiveSession()\n",
    "tf.summary.scalar('Loss', loss)\n",
    "tf.summary.scalar('Accuracy', accuracy)\n",
    "merged = tf.summary.merge_all()\n",
    "logdir = \"tensorboard/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\") + \"/\"\n",
    "writer = tf.summary.FileWriter(logdir, sess.graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import randint\n",
    "def getTrainBatch():\n",
    "    num = randint(0,nExamples - batchSize - 1)\n",
    "    labels = y[num:num+batchSize]\n",
    "    arr = X[num:num+batchSize]\n",
    "    return arr, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "saver = tf.train.Saver()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(iterations):\n",
    "    #Next Batch of reviews\n",
    "    nextBatch, nextBatchLabels = getTrainBatch();\n",
    "    sess.run(optimizer, {input_data: nextBatch, labels: nextBatchLabels})\n",
    "   \n",
    "    #Write summary to Tensorboard\n",
    "    if (i % 25 == 0):\n",
    "        summary = sess.run(merged, {input_data: nextBatch, labels: nextBatchLabels})\n",
    "        writer.add_summary(summary, i)\n",
    "\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Cannot feed value of shape (100, 1) for Tensor 'Placeholder_1:0', which has shape '(24, 100, 1)'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-49-b6f18a016eb8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnextBatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnextBatchLabels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetTrainBatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mpred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprediction\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0minput_data\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mnextBatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mnextBatchLabels\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpred\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/adit/anaconda/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    765\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    766\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 767\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    768\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    769\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/adit/anaconda/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    942\u001b[0m                 \u001b[0;34m'Cannot feed value of shape %r for Tensor %r, '\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    943\u001b[0m                 \u001b[0;34m'which has shape %r'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 944\u001b[0;31m                 % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))\n\u001b[0m\u001b[1;32m    945\u001b[0m           \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgraph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_feedable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msubfeed_t\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    946\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Tensor %s may not be fed.'\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0msubfeed_t\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Cannot feed value of shape (100, 1) for Tensor 'Placeholder_1:0', which has shape '(24, 100, 1)'"
     ]
    }
   ],
   "source": [
    "nextBatch, nextBatchLabels = getTrainBatch();\n",
    "pred = sess.run(prediction, {input_data: nextBatch[0], labels: nextBatchLabels[0]})\n",
    "p = (tf.argmax(pred,1).eval())\n",
    "print ([chars[x] for x in p])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What You Can Do (IN PROGRESS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Really cool stuff right? Now that you know about the model, and how it works, you can try this generate new text based on your own datasets. In order to try the model with your own datasets, save the txt file in the Data folder, and then "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:py3]",
   "language": "python",
   "name": "conda-env-py3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}