t81_558_class_11_03_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_03_embedding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Getting Started with Spacy in Python [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* Part 11.2: Word2Vec and Text Classification [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* **Part 11.3: What are Embedding Layers in Keras** [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.3: What are Embedding Layers in Keras\n",
    "\n",
    "[Embedding Layers](https://keras.io/layers/embeddings/) are a handy feature of Keras that allows the program to automatically insert additional information into the data flow of your neural network.  In the previous section, you saw that Word2Vec could expand words to a 300 dimension vector.  An embedding layer would allow you to insert these 300-dimension vectors in the place of word-indexes automatically.  \n",
    "\n",
    "Programmers often use embedding layers with Natural Language Processing (NLP); however, they can be used in any instance where you wish to insert a lengthier vector in an index value place.  In some ways, you can think of an embedding layer as dimension expansion. However, the hope is that these additional dimensions provide more information to the model and provide a better score.\n",
    "\n",
    "### Simple Embedding Layer Example\n",
    "\n",
    "* **input_dim** = How large is the vocabulary?  How many categories are you encoding? This parameter is the number of items in your \"lookup table.\"\n",
    "* **output_dim** = How many numbers in the vector that you wish to return. \n",
    "* **input_length** = How many items are in the input feature vector that you need to transform?\n",
    "\n",
    "Now we create a neural network with a vocabulary size of 10, which will reduce those values between 0-9 to 4 number vectors.  Each feature vector coming in will have two such features.  This neural network does nothing more than pass the embedding on to the output.  But it does let us see what the embedding is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding\n",
    "import numpy as np\n",
    "\n",
    "model = Sequential()\n",
    "embedding_layer = Embedding(input_dim=10, output_dim=4, input_length=2)\n",
    "model.add(embedding_layer)\n",
    "model.compile('adam', 'mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the structure of this neural network so that we can see what is happening inside it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 2, 4)              40        \n",
      "=================================================================\n",
      "Total params: 40\n",
      "Trainable params: 40\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this neural network, which is just an embedding layer, the input is a vector of size 2. These two inputs are integer numbers from 0 to 9 (corresponding to the requested input_dim quantity of 10 values). Looking at the summary above, we see that the embedding layer has 40 parameters.  This value comes from the embedded lookup table that contains four amounts (output_dim) for each of the 10 (input_dim) possible integer values for the two inputs. The output is 2 (input_length) length 4 (output_dim) vectors, resulting in a total output size of 8, which corresponds to the Output Shape given in the summary above.\n",
    "\n",
    "Now, let us query the neural network with two rows.  The input is two integer values, as was specified when we created the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[[ 0.00050902 -0.0099237  -0.02883428 -0.00821529]\n",
      "  [ 0.02421514  0.03112716  0.02453538 -0.01354214]]]\n"
     ]
    }
   ],
   "source": [
    "input_data = np.array([\n",
    "    [1,2]\n",
    "])\n",
    "\n",
    "pred = model.predict(input_data)\n",
    "\n",
    "print(input_data.shape)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see two length-4 vectors that Keras looked up for each of the input integers.  Recall that Python arrays are zero-based.  Keras replaced the value of 1 with the second row of the 10 x 4 lookup matrix.  Similarly, Keras replaced the value of 2 by the third row of the lookup matrix. The following code displays the lookup matrix in its entirety. The embedding layer performs no mathematical operations other than inserting the correct row from the lookup table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[ 3.9767805e-02,  1.3598096e-02, -1.1770856e-02, -3.6321927e-02],\n",
       "        [ 5.0902367e-04, -9.9236965e-03, -2.8834283e-02, -8.2152858e-03],\n",
       "        [ 2.4215136e-02,  3.1127159e-02,  2.4535384e-02, -1.3542138e-02],\n",
       "        [ 4.6041872e-02,  4.6050500e-02,  2.2079099e-02, -4.1100323e-02],\n",
       "        [-1.9084716e-02,  1.5681457e-02,  1.9137934e-04,  1.2393482e-03],\n",
       "        [ 5.9496611e-05, -4.3054130e-02,  3.0203927e-02, -3.3005968e-02],\n",
       "        [ 1.4646780e-02, -2.4961460e-02, -8.0889687e-03,  3.7561730e-04],\n",
       "        [ 3.7584487e-02, -3.7326049e-02,  3.9304320e-02, -4.2805064e-02],\n",
       "        [ 1.3853524e-02, -1.7934263e-02, -4.2281806e-02, -3.8661052e-02],\n",
       "        [-4.2616617e-02,  1.4965128e-02, -3.5379924e-02, -3.9788373e-03]],\n",
       "       dtype=float32)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_layer.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values above are random parameters that Keras generated as starting points.  Generally, we will either transfer an embedding or train these random values into something useful.  The next section demonstrates how to embed a hand-coded embedding. \n",
    "\n",
    "### Transferring An Embedding\n",
    "\n",
    "Now, we see how to hard-code an embedding lookup that performs a simple one-hot encoding.  One-hot encoding would transform the input integer values of 0, 1, and 2 to the vectors $[1,0,0]$, $[0,1,0]$, and $[0,0,1]$ respectively. The following code replaced the random lookup values in the embedding layer with this one-hot coding inspired lookup table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding\n",
    "import numpy as np\n",
    "\n",
    "embedding_lookup = np.array([\n",
    "    [1,0,0],\n",
    "    [0,1,0],\n",
    "    [0,0,1]\n",
    "])\n",
    "\n",
    "model = Sequential()\n",
    "embedding_layer = Embedding(input_dim=3, output_dim=3, input_length=2)\n",
    "model.add(embedding_layer)\n",
    "model.compile('adam', 'mse')\n",
    "\n",
    "embedding_layer.set_weights([embedding_lookup])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have the following parameters to the Embedding layer:\n",
    "    \n",
    "* input_dim=3 - There are three different integer categorical values allowed.\n",
    "* output_dim=3 - Per one-hot encoding, three columns represent a categorical value with three possible values.\n",
    "* input_length=2 - The input vector has two of these categorical values.\n",
    "\n",
    "Now we query the neural network with two categorical values to see the lookup performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[[1. 0. 0.]\n",
      "  [0. 1. 0.]]]\n"
     ]
    }
   ],
   "source": [
    "input_data = np.array([\n",
    "    [0,1]\n",
    "])\n",
    "\n",
    "pred = model.predict(input_data)\n",
    "\n",
    "print(input_data.shape)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The given output shows that we provided the program with two rows from the one-hot encoding table. This encoding is a correct one-hot encoding for the values 0 and 1, where there are up to 3 unique values possible. \n",
    "\n",
    "The next section demonstrates how to train this embedding lookup table.\n",
    "\n",
    "### Training an Embedding\n",
    "\n",
    "First, we make use of the following imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import array\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Flatten, Embedding, Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a neural network that classifies restaurant reviews according to positive or negative.  This neural network can accept strings as input, such as given here.  This code also includes positive or negative labels for each review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define 10 resturant reviews.\n",
    "reviews = [\n",
    "    'Never coming back!',\n",
    "    'Horrible service',\n",
    "    'Rude waitress',\n",
    "    'Cold food.',\n",
    "    'Horrible food!',\n",
    "    'Awesome',\n",
    "    'Awesome service!',\n",
    "    'Rocks!',\n",
    "    'poor work',\n",
    "    'Couldn\\'t have done better']\n",
    "\n",
    "# Define labels (1=negative, 0=positive)\n",
    "labels = array([1,1,1,1,1,0,0,0,0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the second to the last label is incorrect.  Errors such as this are not too out of the ordinary, as most training data could have some noise.\n",
    "\n",
    "We define a vocabulary size of 50 words.  Though we do not have 50 words, it is okay to use a value larger than needed.  If there are more than 50 words, the least frequently used words in the training set are automatically dropped by the embedding layer during training.  For input, we one-hot encode the strings.  Note that we use the TensorFlow one-hot encoding method here, rather than Scikit-Learn. Scikit-learn would expand these strings to the 0's and 1's as we would typically see for dummy variables.  TensorFlow translates all of the words to index values and replaces each word with that index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded reviews: [[38, 15, 15], [16, 20], [24, 15], [18, 24], [16, 24], [35], [35, 20], [7], [37, 32], [21, 3, 27, 16]]\n"
     ]
    }
   ],
   "source": [
    "VOCAB_SIZE = 50\n",
    "encoded_reviews = [one_hot(d, VOCAB_SIZE) for d in reviews]\n",
    "print(f\"Encoded reviews: {encoded_reviews}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The program one-hot encodes these reviews to word indexes; however, their lengths are different.  We pad these reviews to 4 words and truncate any words beyond the fourth word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[38 15 15  0]\n",
      " [16 20  0  0]\n",
      " [24 15  0  0]\n",
      " [18 24  0  0]\n",
      " [16 24  0  0]\n",
      " [35  0  0  0]\n",
      " [35 20  0  0]\n",
      " [ 7  0  0  0]\n",
      " [37 32  0  0]\n",
      " [21  3 27 16]]\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 4\n",
    "\n",
    "padded_reviews = pad_sequences(encoded_reviews, maxlen=MAX_LENGTH, \\\n",
    "                               padding='post')\n",
    "print(padded_reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each review is padded by appending zeros at the end, as specified by the padding=post setting.\n",
    "\n",
    "Next, we create a neural network to learn to classify these reviews. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 4, 8)              400       \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 433\n",
      "Trainable params: 433\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "embedding_layer = Embedding(VOCAB_SIZE, 8, input_length=MAX_LENGTH)\n",
    "model.add(embedding_layer)\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network accepts four integer inputs that specify the indexes of a padded movie review.  The first embedding layer converts these four indexes into four vectors of length 8.  These vectors come from the lookup table that contains 50 (VOCAB_SIZE) rows of vectors of length 8. This encoding is evident by the 400 (8 times 50) parameters in the embedding layer. The size of the output from the embedding layer is 32 (4 words expressed as 8-number embedded vectors).  A single output neuron is connected to the embedding layer by 33 weights (32 from the embedding layer and a single bias neuron).  Because this is a single-class classification network, we use the sigmoid activation function and binary_crossentropy.\n",
    "\n",
    "The program now trains the neural network.  Both the embedding lookup and dense 33 weights are updated to produce a better score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x63f0cd210>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the model\n",
    "model.fit(padded_reviews, labels, epochs=100, verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the learned embeddings.  Think of each word's vector as a location in 8 dimension space where words associated with positive reviews are close to other words with positive reviews.  Similarly, training places negative reviews close to each other.  In addition to the training setting these embeddings, the 33 weights between the embedding layer and output neuron similarly learn to transform these embeddings into an actual prediction.  You can see these embeddings here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50, 8)\n",
      "[array([[-1.03855960e-01, -5.59105724e-02, -6.04345910e-02,\n",
      "         1.50698468e-01,  8.98860618e-02,  8.96284208e-02,\n",
      "         1.65250570e-01,  4.28558849e-02],\n",
      "       [ 2.30258144e-02,  4.66232412e-02, -3.89982834e-02,\n",
      "         2.99915113e-02, -2.06652526e-02,  3.44550945e-02,\n",
      "        -1.44930705e-02, -4.07445915e-02],\n",
      "       [-4.72828634e-02,  1.85899399e-02,  8.94228369e-03,\n",
      "         9.16576385e-03, -1.39516592e-03,  3.86941768e-02,\n",
      "        -2.00331211e-04, -2.19582673e-02],\n",
      "       [-1.38059288e-01, -6.69398904e-02, -1.37284666e-01,\n",
      "         8.37527588e-02,  8.94882604e-02,  9.02104154e-02,\n",
      "         6.08446635e-02,  1.74611673e-01],\n",
      "       [-3.90188769e-03,  2.26297863e-02, -3.32948118e-02,\n",
      "        -7.31148571e-03,  3.02187465e-02, -4.14650813e-02,\n",
      "        -4.23040651e-02,  1.48720481e-02],\n",
      "       [-2.69240271e-02,  4.56026904e-02, -1.25192292e-02,\n",
      "        -2.45216247e-02, -4.73748930e-02, -2.03959588e-02,\n",
      "        -4.59431186e-02,  3.84237655e-02],\n",
      "       [-1.44931301e-02,  3.59132998e-02, -2.16722973e-02,\n",
      "         3.91978659e-02, -2.92063002e-02, -1.60443559e-02,\n",
      "         3.41716073e-02, -4.44576517e-02],\n",
      "       [ 1.02968447e-01, -1.24651685e-01, -7.43341371e-02,\n",
      "        -1.05752744e-01,  5.67097403e-02, -1.00757137e-01,\n",
      "        -1.13389604e-01,  5.45799546e-02],\n",
      "       [-6.00196421e-04,  7.01391697e-03,  4.14869301e-02,\n",
      "        -4.44086567e-02, -5.56975603e-03, -1.35170110e-02,\n",
      "        -3.44227552e-02, -5.79791144e-03],\n",
      "       [-3.81987467e-02,  3.17828991e-02, -4.03899774e-02,\n",
      "         3.31121124e-02,  3.75297926e-02, -1.16758235e-02,\n",
      "        -1.49350390e-02, -4.27979715e-02],\n",
      "       [ 1.43063106e-02, -1.90679785e-02,  4.09963243e-02,\n",
      "         4.85630371e-02, -2.82872319e-02,  9.44522768e-03,\n",
      "         2.85794400e-02,  6.19646162e-03],\n",
      "       [-1.14040747e-02,  2.47441605e-03, -1.04773268e-02,\n",
      "        -2.66404878e-02,  3.26102041e-02, -1.77922733e-02,\n",
      "        -1.05445869e-02,  3.43603231e-02],\n",
      "       [-4.16200981e-02, -3.68646979e-02,  3.43921892e-02,\n",
      "        -2.99186353e-02, -2.48629693e-02,  1.50619410e-02,\n",
      "        -5.01462072e-03, -3.79680283e-02],\n",
      "       [-2.11276114e-04,  3.96753810e-02, -2.43502017e-02,\n",
      "        -5.07425144e-03, -1.23413689e-02,  1.47156008e-02,\n",
      "         3.85177024e-02,  3.64763848e-02],\n",
      "       [ 3.82987000e-02,  4.79196385e-03,  1.36850961e-02,\n",
      "        -4.89234217e-02, -7.62814283e-03, -1.94343459e-02,\n",
      "         4.05356176e-02, -5.41444868e-03],\n",
      "       [-3.74498703e-02,  1.30236372e-01,  1.42154992e-01,\n",
      "        -8.84348229e-02, -1.31201312e-01, -1.31135836e-01,\n",
      "        -9.09620747e-02, -8.02542344e-02],\n",
      "       [ 1.31572738e-01,  5.44598848e-02,  1.03238679e-01,\n",
      "         1.23520985e-01, -8.13489929e-02,  1.31479695e-01,\n",
      "         1.21889763e-01, -1.03636011e-01],\n",
      "       [-3.44789028e-02,  1.26743205e-02,  2.67877318e-02,\n",
      "        -2.13473681e-02,  3.49123739e-02,  1.12790354e-02,\n",
      "        -5.41354343e-03, -3.55566964e-02],\n",
      "       [-1.27702817e-01,  7.30508566e-02,  1.19967408e-01,\n",
      "         8.56134444e-02, -1.35003090e-01,  1.33873299e-01,\n",
      "         1.63287789e-01, -1.33761331e-01],\n",
      "       [ 2.76777036e-02,  2.44518258e-02, -2.43045017e-03,\n",
      "        -2.81578302e-02, -6.67472929e-03,  3.40043940e-02,\n",
      "         3.74088436e-03,  6.93570822e-04],\n",
      "       [ 2.02073455e-02,  2.94452514e-02,  2.31720470e-02,\n",
      "        -5.40700057e-05, -2.37033851e-02,  4.77408692e-02,\n",
      "        -1.31957810e-02, -3.49997319e-02],\n",
      "       [ 1.20016925e-01, -8.53572562e-02, -7.25273043e-02,\n",
      "        -1.46297619e-01,  8.08605775e-02, -1.28363371e-01,\n",
      "        -1.61475629e-01,  1.27090588e-01],\n",
      "       [-6.61499426e-03, -1.78650841e-02, -3.78209576e-02,\n",
      "         3.72546650e-02, -2.23232042e-02,  7.34323263e-03,\n",
      "        -4.79101911e-02,  2.70697512e-02],\n",
      "       [-4.65823412e-02, -1.15211494e-02,  5.24830073e-04,\n",
      "        -7.66612589e-04, -4.07127291e-03, -8.25840235e-03,\n",
      "        -5.36084175e-04,  3.82142104e-02],\n",
      "       [ 1.03968188e-01,  1.09589085e-01,  7.74130523e-02,\n",
      "        -1.23851024e-01, -1.51139662e-01, -1.33563846e-01,\n",
      "        -1.07028246e-01, -1.46226272e-01],\n",
      "       [-2.18360312e-02, -2.08271872e-02,  3.91653441e-02,\n",
      "        -3.17618139e-02, -4.10533175e-02,  2.04103105e-02,\n",
      "        -3.96764502e-02,  1.35130547e-02],\n",
      "       [ 4.16355990e-02,  1.35681890e-02, -5.15161827e-03,\n",
      "         2.77458094e-02,  1.32545084e-03, -3.14503908e-02,\n",
      "        -4.09910679e-02, -4.04780880e-02],\n",
      "       [ 1.25920698e-01, -1.37650132e-01, -8.32531005e-02,\n",
      "        -3.63135971e-02, -1.17045261e-01,  1.74083021e-02,\n",
      "         8.27008337e-02,  6.52465820e-02],\n",
      "       [-2.16866136e-02,  4.13646810e-02, -1.66014433e-02,\n",
      "        -1.51780024e-02,  8.64658505e-03, -1.74358487e-02,\n",
      "        -3.72581109e-02, -2.32456215e-02],\n",
      "       [ 4.98321094e-02, -1.31017677e-02, -3.00016757e-02,\n",
      "         3.52717154e-02,  1.34622194e-02,  1.44667551e-03,\n",
      "        -2.80967951e-02, -1.46480910e-02],\n",
      "       [ 4.39108536e-03, -1.02404505e-03, -2.19083186e-02,\n",
      "         2.73661949e-02,  3.99474837e-02, -2.05892809e-02,\n",
      "         4.79017161e-02, -3.59805226e-02],\n",
      "       [ 1.22169480e-02,  3.45968492e-02,  7.71068037e-04,\n",
      "        -4.20971140e-02,  1.09663382e-02,  9.81803983e-03,\n",
      "         3.94362696e-02, -3.23885456e-02],\n",
      "       [-7.25854486e-02, -1.37808755e-01, -8.20303187e-02,\n",
      "         7.47192055e-02,  6.09146953e-02,  1.43233463e-01,\n",
      "         1.33977473e-01,  8.99255127e-02],\n",
      "       [-1.73668936e-03, -3.85993943e-02,  1.81568526e-02,\n",
      "        -4.07350548e-02, -7.20812008e-03,  2.33794376e-03,\n",
      "        -1.40483603e-02,  1.89022906e-02],\n",
      "       [ 4.92309816e-02,  1.25241280e-03, -4.49079983e-02,\n",
      "         4.90389727e-02,  7.10040331e-03, -3.68149504e-02,\n",
      "        -3.36388499e-02,  2.44540907e-02],\n",
      "       [ 7.13890120e-02, -5.96655272e-02, -7.41720200e-02,\n",
      "        -1.11592956e-01,  8.48936141e-02, -1.09633878e-01,\n",
      "        -1.10216260e-01,  1.42637342e-01],\n",
      "       [ 3.16339619e-02,  1.44764893e-02,  1.97224952e-02,\n",
      "         3.14760208e-03,  3.07879597e-03,  2.39167474e-02,\n",
      "         2.22613700e-02, -3.25145870e-02],\n",
      "       [ 1.47053301e-01, -1.49222419e-01, -1.16918348e-01,\n",
      "        -5.59649877e-02,  9.97040719e-02, -6.59333616e-02,\n",
      "        -7.78462514e-02,  6.46042824e-02],\n",
      "       [-1.05294377e-01,  1.42578408e-01,  1.20506063e-01,\n",
      "         5.92038780e-02, -1.12155899e-01,  1.24044374e-01,\n",
      "         1.20608859e-01, -5.86235970e-02],\n",
      "       [-9.03630257e-03,  4.35552709e-02, -2.45991107e-02,\n",
      "         2.86003388e-02,  2.22371332e-02,  3.29569913e-02,\n",
      "         1.47115476e-02,  3.59623544e-02],\n",
      "       [ 5.26278093e-03,  4.41319607e-02,  3.94199975e-02,\n",
      "         4.45728935e-02, -3.97939309e-02,  3.06106843e-02,\n",
      "        -2.24255808e-02,  4.53190468e-02],\n",
      "       [-1.12559646e-03, -8.12286139e-03, -3.52341048e-02,\n",
      "         3.24758552e-02, -1.00637786e-02,  3.22290510e-03,\n",
      "        -3.92202027e-02, -1.30762681e-02],\n",
      "       [-1.51422992e-02,  3.01110856e-02, -5.56067377e-03,\n",
      "        -1.77193396e-02, -1.55406110e-02,  1.35609768e-02,\n",
      "         1.46402828e-02,  7.69706815e-03],\n",
      "       [-2.81559117e-02, -2.10944545e-02,  1.70862414e-02,\n",
      "        -5.89495897e-03, -5.49186394e-03,  4.77211140e-02,\n",
      "        -7.19447061e-03,  2.82587074e-02],\n",
      "       [-1.72454938e-02, -9.73246992e-04, -1.19401924e-02,\n",
      "        -1.60130113e-03, -1.86638124e-02,  1.52602680e-02,\n",
      "        -7.94810057e-03, -4.15552482e-02],\n",
      "       [-9.93949175e-03,  3.40985097e-02,  1.84216537e-02,\n",
      "         3.07177044e-02, -5.30803204e-03, -1.77914612e-02,\n",
      "         2.53076293e-02, -4.43017595e-02],\n",
      "       [ 4.39007543e-02, -4.91391197e-02, -3.81500646e-03,\n",
      "         6.40837103e-03, -1.14386790e-02, -2.03427076e-02,\n",
      "         8.76239687e-03,  3.26521657e-02],\n",
      "       [-2.38515865e-02, -4.67003845e-02,  2.18842290e-02,\n",
      "         3.54081057e-02, -4.04736772e-02,  4.66918983e-02,\n",
      "         3.13981809e-02,  2.97445059e-03],\n",
      "       [ 2.33231299e-02, -2.66075861e-02,  2.67822854e-02,\n",
      "         4.76033799e-02, -1.80492997e-02, -9.64826345e-03,\n",
      "        -8.73423740e-03,  4.65954877e-02],\n",
      "       [-3.16178575e-02, -4.28569689e-02,  4.80644591e-02,\n",
      "        -9.59502533e-03,  3.28160785e-02,  2.81270780e-02,\n",
      "         7.52024725e-03, -4.61859480e-02]], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer.get_weights()[0].shape)\n",
    "print(embedding_layer.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate this neural network's accuracy, including both the embeddings and the learned dense layer.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = model.evaluate(padded_reviews, labels, verbose=0)\n",
    "print(f'Accuracy: {accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy is a perfect 1.0, indicating there is likely overfitting. For a more complex data set, it would be good to use early stopping to not overfit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log-loss: 0.4717821180820465\n"
     ]
    }
   ],
   "source": [
    "print(f'Log-loss: {loss}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the loss is not perfect, meaning that even though the predicted probabilities indicated a correct prediction in every case, the program did not achieve absolute confidence in each correct answer.  The lack of confidence was likely due to the small amount of noise (previously discussed) in the data set.  Additionally, the fact that some words appeared in both positive and negative reviews contributed to this lack of absolute certainty. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}