t81_558_class_11_01_spacy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_01_spacy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* **Part 11.1: Getting Started with Spacy in Python** [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* Part 11.2: Word2Vec and Text Classification [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* Part 11.3: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.1: Getting Started with Spacy in Python\n",
    "\n",
    "When we apply neural networks to Natural Language Processing (NLP), we must decide if you want to operate at the word or character level.  Up to this point, we've worked primarily at the character level, which was the case for the Treasure Island text pirate story generator that we previously saw.  Likewise, we used word-level NLP for the image caption generator.  In this module, the focus will be primarily on word-level NLP. Notably, we will examine some of the NLP tools that we can be used to process words before we send them to a neural network.  There are two prevalent NLP libraries for Python:\n",
    "\n",
    "* [NLTK](https://www.nltk.org/)\n",
    "* [Spacy](https://spacy.io/)\n",
    "\n",
    "In this course, we will focus on Spacy.  I prefer spacy because of the object abstraction of sentences that it provides.  However, both are widely used libraries.\n",
    "\n",
    "### Installing Spacy\n",
    "\n",
    "You can install Spacy with a simple PIP install. Spacy was included in the list of packages to install for this course.  You will need to ensure that you've installed a language with Spacy.  If you do not, you will get the following error:\n",
    "\n",
    "```\n",
    "OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem \n",
    "to be a shortcut link, a Python package or a valid path to a \n",
    "data directory.\n",
    "```\n",
    "\n",
    "To install English, use the following command:\n",
    "\n",
    "```\n",
    "python -m spacy download en\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "\n",
    "Tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation.  Consider how the program might break up the following sentences into words.\n",
    "\n",
    "* This is a test.\n",
    "* Ok, but what about this?\n",
    "* Is U.S.A. the same as USA.?\n",
    "* What is the best data-set to use?\n",
    "* I think I will do this-no wait; I will do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple\n",
      "is\n",
      "looking\n",
      "at\n",
      "buying\n",
      "a\n",
      "U.K.\n",
      "startup\n",
      "for\n",
      "$\n",
      "1\n",
      "billion\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(u\"Apple is looking at buying a U.K. startup for $1 billion\")\n",
    "for token in doc:\n",
    "    print(token.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also obtain the part of speech for each word.  Common parts of speech include nouns, verbs, pronouns, and adjectives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple PROPN\n",
      "is AUX\n",
      "looking VERB\n",
      "at ADP\n",
      "buying VERB\n",
      "a DET\n",
      "U.K. PROPN\n",
      "startup NOUN\n",
      "for ADP\n",
      "$ SYM\n",
      "1 NUM\n",
      "billion NUM\n"
     ]
    }
   ],
   "source": [
    "for word in doc:  \n",
    "    print(word.text,  word.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spacy includes functions to check if parts of a sentence appear to be numbers, acronyms, or other entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Apple is like number? False\n",
      "is is like number? False\n",
      "looking is like number? False\n",
      "at is like number? False\n",
      "buying is like number? False\n",
      "a is like number? False\n",
      "U.K. is like number? False\n",
      "startup is like number? False\n",
      "for is like number? False\n",
      "$ is like number? False\n",
      "1 is like number? True\n",
      "billion is like number? True\n"
     ]
    }
   ],
   "source": [
    "for word in doc:\n",
    "    print(f\"{word} is like number? {word.like_num}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence Diagramming\n",
    "\n",
    "For years grade school children have had to endure \"sentence diagramming.\"  Such diagrams can help students to understand sentence structure.  Spacy provides a means to produce similar sentence diagrams.  For example, the sentence \"My name is Jeff\" is diagrammed as follows by Spacy as Figure 11.SPACY.\n",
    "\n",
    "**Figure 11.SPACY: Spacy Sentence Diagram**\n",
    "![Spacy Diagram](./images/spacy-diagram.png)\n",
    "\n",
    "I provide the code needed to diagram this sentence below.  This code generates a scrollable, interactive display.  Because of this interactivity, you will need to stop this cell in Jupyter to continue.  Also, the code does not show its output until you run this cell.  You will not see it on GitHub or the printed book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jheaton/miniconda3/envs/tensorflow/lib/python3.7/site-packages/spacy/displacy/__init__.py:94: UserWarning: [W011] It looks like you're calling displacy.serve from within a Jupyter notebook or a similar environment. This likely means you're already running a local web server, so there's no need to make displaCy start another one. Instead, you should be able to replace displacy.serve with displacy.render to show the visualization.\n",
      "  warnings.warn(Warnings.W011)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "    <head>\n",
       "        <title>displaCy</title>\n",
       "    </head>\n",
       "\n",
       "    <body style=\"font-size: 16px; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; padding: 4rem 2rem; direction: ltr\">\n",
       "<figure style=\"margin-bottom: 6rem\">\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"7ea07da856bc482785eb2cf1aca13a80-0\" class=\"displacy\" width=\"750\" height=\"224.5\" direction=\"ltr\" style=\"max-width: none; height: 224.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">My</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">name</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">Jeff.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-7ea07da856bc482785eb2cf1aca13a80-0-0\" stroke-width=\"2px\" d=\"M70,89.5 C70,2.0 225.0,2.0 225.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-7ea07da856bc482785eb2cf1aca13a80-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">poss</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,91.5 L62,79.5 78,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-7ea07da856bc482785eb2cf1aca13a80-0-1\" stroke-width=\"2px\" d=\"M245,89.5 C245,2.0 400.0,2.0 400.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-7ea07da856bc482785eb2cf1aca13a80-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M245,91.5 L237,79.5 253,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-7ea07da856bc482785eb2cf1aca13a80-0-2\" stroke-width=\"2px\" d=\"M420,89.5 C420,2.0 575.0,2.0 575.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-7ea07da856bc482785eb2cf1aca13a80-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">attr</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M575.0,91.5 L583.0,79.5 567.0,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>\n",
       "</figure>\n",
       "</body>\n",
       "</html></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Using the 'dep' visualizer\n",
      "Serving on http://0.0.0.0:5000 ...\n",
      "\n",
      "Shutting down server on port 5000.\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(u\"My name is Jeff.\")\n",
    "displacy.serve(doc, style=\"dep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note, you will have to manually stop the above cell**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My name is Jeff.\n"
     ]
    }
   ],
   "source": [
    "print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows how to reduce words to their stems.  Here the sentence words are reduced to their most basic form.  For example, \"striped\" to \"stripe.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the stripe bat be hang on -PRON- foot for good'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Initialize spacy 'en' model, keeping only tagger \n",
    "# component needed for lemmatization\n",
    "nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])\n",
    "\n",
    "sentence = \"The striped bats are hanging on their feet for best\"\n",
    "\n",
    "# Parse the sentence using the loaded 'en' model object `nlp`\n",
    "doc = nlp(sentence)\n",
    "\n",
    "# Extract the lemma for each token and join\n",
    "\n",
    "\" \".join([token.lemma_ for token in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Words\n",
    "\n",
    "Stop words are words which are filtered out before or after processing of natural language text. Though \"stop words\" usually refers to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools. Spacy contains a list of stop words that you may wish to exclude or treat with less importance in your programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'toward', 'around', 'yours', 'moreover', 'for', 'and', 'many', 'have', 'noone', 'thereby', 'must', 'among', 'beside', 'make', 'might', \"'s\", 'please', 'three', 'indeed', 'seem', 'ever', 'he', 'ten', 'too', 'already', 'who', 'had', 'wherever', 'back', 'hence', 'when', 'done', 'now', '’m', 'therefore', 'using', 'elsewhere', 'will', 'them', 'being', 'anything', 'twelve', 'part', 'ca', 'nothing', 'get', 'nobody', 'we', 'somehow', 'alone', 'beforehand', 'ourselves', 'show', 'off', 'him', 'amongst', 'although', 'doing', 'a', 'am', 'anyone', 'as', 'something', 'else', 'those', 'if', 'whoever', 'has', 'does', 'this', 'only', 'becomes', 'twenty', 'by', 'became', 'except', 'well', 'any', 'across', 'fifty', 'neither', 'in', 'front', 'formerly', 'perhaps', 'through', 'whence', '‘re', 'again', 'to', 're', 'below', 'whose', 'hers', 'first', 'most', 'did', 'serious', 'may', 'they', 'herein', 'everyone', 'it', 'anyhow', 'whither', 'was', 'his', 'whenever', 'itself', 'call', 'becoming', 'fifteen', 'thence', 'her', 'throughout', 'whether', 'what', 'side', 'up', \"'d\", 'while', 'be', 'are', 'would', '‘ve', 'into', 'towards', 'along', 'next', 'say', 'herself', 'everywhere', 'since', 'anywhere', 'just', '‘s', 'your', 'almost', 'former', 'why', 'wherein', '’s', 'therein', 'whereafter', 'yourselves', 'i', 'whereas', 'various', 'latter', 'eleven', 'is', 'not', 'whom', 'because', 'could', 'with', 'move', 'thus', 'same', 'other', 'here', 'mine', 'still', 'above', 'mostly', 'where', \"'re\", 'whereupon', 'anyway', 'own', 'per', 'from', '‘m', 'least', 'afterwards', 'these', 'whatever', 'then', 'together', 'about', 'been', 'six', 'however', 'never', 'our', 'put', \"n't\", 'every', 'the', 'go', 'once', 'can', 'five', 'hundred', 'that', 'me', 'sixty', 'bottom', \"'ll\", 'during', '’d', 'enough', 'latterly', 'but', 'often', 'namely', '‘ll', 'see', 'via', 'everything', 'upon', 'than', 'all', 'due', 'such', 'there', 'its', 'give', \"'m\", 'us', 'after', 'hereby', 'somewhere', 'she', 'sometime', 'others', 'both', 'forty', 'nevertheless', 'keep', 'seeming', 'nor', 'full', 'four', 'ours', '’re', '‘d', 'also', 'thereupon', 'regarding', 'whole', 'on', 'beyond', 'rather', 'their', 'between', 'hereafter', 'none', 'themselves', 'nine', 'used', 'several', 'become', 'few', 'some', 'meanwhile', 'another', 'n‘t', 'more', 'yourself', 'until', 'amount', 'of', 'less', 'someone', 'at', 'hereupon', 'last', 'even', 'unless', 'take', 'thru', 'out', 'one', 'besides', 'cannot', 'though', \"'ve\", 'made', 'seemed', 'how', 'himself', 'within', 'top', 'empty', 'over', 'were', 'each', 'nowhere', 'under', 'do', 'seems', 'yet', 'further', 'name', 'my', 'against', 'quite', 'myself', 'behind', 'an', 'whereby', '’ll', 'two', '’ve', 'onto', 'much', 'really', 'n’t', 'no', 'sometimes', 'down', 'or', 'should', 'third', 'thereafter', 'you', 'eight', 'without', 'which', 'very', 'so', 'always', 'before', 'otherwise', 'either'}\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "print(STOP_WORDS)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}