33-text-classification-pretrained_word_embeddings.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "exlopWBS-PBa"
      },
      "source": [
        "# Using pre-trained word embeddings\n",
        "\n",
        "**Author:** [fchollet](https://twitter.com/fchollet)<br>\n",
        "**Date created:** 2020/05/05<br>\n",
        "**Last modified:** 2020/05/05<br>\n",
        "**Description:** Text classification on the Newsgroup20 dataset using pre-trained GloVe word embeddings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3apEl6o-PBb"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "15sR_QV_-PBc"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3b8MDZXE-PBd"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "In this example, we show how to train a text classification model that uses pre-trained\n",
        "word embeddings.\n",
        "\n",
        "We'll work with the Newsgroup20 dataset, a set of 20,000 message board messages\n",
        "belonging to 20 different topic categories.\n",
        "\n",
        "For the pre-trained word embeddings, we'll use\n",
        "[GloVe embeddings](http://nlp.stanford.edu/projects/glove/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiPZ5aDS-PBe"
      },
      "source": [
        "## Download the Newsgroup20 data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "z45r7ZKM-PBf"
      },
      "outputs": [],
      "source": [
        "data_path = keras.utils.get_file(\n",
        "    \"news20.tar.gz\",\n",
        "    \"http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/news20.tar.gz\",\n",
        "    untar=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVB62kLS-PBf"
      },
      "source": [
        "## Let's take a look at the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vosEBk5P-PBg"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pathlib\n",
        "\n",
        "data_dir = pathlib.Path(data_path).parent / \"20_newsgroup\"\n",
        "dirnames = os.listdir(data_dir)\n",
        "print(\"Number of directories:\", len(dirnames))\n",
        "print(\"Directory names:\", dirnames)\n",
        "\n",
        "fnames = os.listdir(data_dir / \"comp.graphics\")\n",
        "print(\"Number of files in comp.graphics:\", len(fnames))\n",
        "print(\"Some example filenames:\", fnames[:5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGFzQCSX-PBh"
      },
      "source": [
        "Here's a example of what one file contains:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hbba2zQM-PBk"
      },
      "outputs": [],
      "source": [
        "print(open(data_dir / \"comp.graphics\" / \"38987\").read())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOC1vgOO-PBl"
      },
      "source": [
        "As you can see, there are header lines that are leaking the file's category, either\n",
        "explicitly (the first line is literally the category name), or implicitly, e.g. via the\n",
        "`Organization` filed. Let's get rid of the headers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RWzj-iD7-PBm"
      },
      "outputs": [],
      "source": [
        "samples = []\n",
        "labels = []\n",
        "class_names = []\n",
        "class_index = 0\n",
        "for dirname in sorted(os.listdir(data_dir)):\n",
        "    class_names.append(dirname)\n",
        "    dirpath = data_dir / dirname\n",
        "    fnames = os.listdir(dirpath)\n",
        "    print(\"Processing %s, %d files found\" % (dirname, len(fnames)))\n",
        "    for fname in fnames:\n",
        "        fpath = dirpath / fname\n",
        "        f = open(fpath, encoding=\"latin-1\")\n",
        "        content = f.read()\n",
        "        lines = content.split(\"\\n\")\n",
        "        lines = lines[10:]\n",
        "        content = \"\\n\".join(lines)\n",
        "        samples.append(content)\n",
        "        labels.append(class_index)\n",
        "    class_index += 1\n",
        "\n",
        "print(\"Classes:\", class_names)\n",
        "print(\"Number of samples:\", len(samples))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXsf2P7W-PBm"
      },
      "source": [
        "There's actually one category that doesn't have the expected number of files, but the\n",
        "difference is small enough that the problem remains a balanced classification problem."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0EXNqEv9-PBn"
      },
      "source": [
        "## Shuffle and split the data into training & validation sets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-Dh_HuzS-PBn"
      },
      "outputs": [],
      "source": [
        "# Shuffle the data\n",
        "seed = 1337\n",
        "rng = np.random.RandomState(seed)\n",
        "rng.shuffle(samples)\n",
        "rng = np.random.RandomState(seed)\n",
        "rng.shuffle(labels)\n",
        "\n",
        "# Extract a training & validation split\n",
        "validation_split = 0.2\n",
        "num_validation_samples = int(validation_split * len(samples))\n",
        "train_samples = samples[:-num_validation_samples]\n",
        "val_samples = samples[-num_validation_samples:]\n",
        "train_labels = labels[:-num_validation_samples]\n",
        "val_labels = labels[-num_validation_samples:]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pYIBgFwL-PBo"
      },
      "source": [
        "## Create a vocabulary index\n",
        "\n",
        "Let's use the `TextVectorization` to index the vocabulary found in the dataset.\n",
        "Later, we'll use the same layer instance to vectorize the samples.\n",
        "\n",
        "Our layer will only consider the top 20,000 words, and will truncate or pad sequences to\n",
        "be actually 200 tokens long."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5tIyS2dY-PBo"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.layers import TextVectorization\n",
        "\n",
        "vectorizer = TextVectorization(max_tokens=20000, output_sequence_length=200)\n",
        "text_ds = tf.data.Dataset.from_tensor_slices(train_samples).batch(128)\n",
        "vectorizer.adapt(text_ds)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8nQrNomM-PBp"
      },
      "source": [
        "You can retrieve the computed vocabulary used via `vectorizer.get_vocabulary()`. Let's\n",
        "print the top 5 words:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ipefT37i-PBp"
      },
      "outputs": [],
      "source": [
        "vectorizer.get_vocabulary()[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YS_EMWeG-PBp"
      },
      "source": [
        "Let's vectorize a test sentence:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "84yeHeHs-PBq"
      },
      "outputs": [],
      "source": [
        "output = vectorizer([[\"the cat sat on the mat\"]])\n",
        "output.numpy()[0, :6]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xXvzGAa--PBq"
      },
      "source": [
        "As you can see, \"the\" gets represented as \"2\". Why not 0, given that \"the\" was the first\n",
        "word in the vocabulary? That's because index 0 is reserved for padding and index 1 is\n",
        "reserved for \"out of vocabulary\" tokens.\n",
        "\n",
        "Here's a dict mapping words to their indices:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rsoVjp6G-PBr"
      },
      "outputs": [],
      "source": [
        "voc = vectorizer.get_vocabulary()\n",
        "word_index = dict(zip(voc, range(len(voc))))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OtltTx3D-PBr"
      },
      "source": [
        "As you can see, we obtain the same encoding as above for our test sentence:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hafHL8S8-PBr"
      },
      "outputs": [],
      "source": [
        "test = [\"the\", \"cat\", \"sat\", \"on\", \"the\", \"mat\"]\n",
        "[word_index[w] for w in test]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LJbuXghC-PBs"
      },
      "source": [
        "## Load pre-trained word embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6garYN--PBt"
      },
      "source": [
        "Let's download pre-trained GloVe embeddings (a 822M zip file).\n",
        "\n",
        "You'll need to run the following commands:\n",
        "\n",
        "```\n",
        "!wget http://nlp.stanford.edu/data/glove.6B.zip\n",
        "!unzip -q glove.6B.zip\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zDF0lbr-PBt"
      },
      "source": [
        "The archive contains text-encoded vectors of various sizes: 50-dimensional,\n",
        "100-dimensional, 200-dimensional, 300-dimensional. We'll use the 100D ones.\n",
        "\n",
        "Let's make a dict mapping words (strings) to their NumPy vector representation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qcjIgsD7-PBt"
      },
      "outputs": [],
      "source": [
        "path_to_glove_file = os.path.join(\n",
        "    os.path.expanduser(\"~\"), \".keras/datasets/glove.6B.100d.txt\"\n",
        ")\n",
        "\n",
        "embeddings_index = {}\n",
        "with open(path_to_glove_file) as f:\n",
        "    for line in f:\n",
        "        word, coefs = line.split(maxsplit=1)\n",
        "        coefs = np.fromstring(coefs, \"f\", sep=\" \")\n",
        "        embeddings_index[word] = coefs\n",
        "\n",
        "print(\"Found %s word vectors.\" % len(embeddings_index))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWZ2KaqI-PBu"
      },
      "source": [
        "Now, let's prepare a corresponding embedding matrix that we can use in a Keras\n",
        "`Embedding` layer. It's a simple NumPy matrix where entry at index `i` is the pre-trained\n",
        "vector for the word of index `i` in our `vectorizer`'s vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vrKBmCu2-PBu"
      },
      "outputs": [],
      "source": [
        "num_tokens = len(voc) + 2\n",
        "embedding_dim = 100\n",
        "hits = 0\n",
        "misses = 0\n",
        "\n",
        "# Prepare embedding matrix\n",
        "embedding_matrix = np.zeros((num_tokens, embedding_dim))\n",
        "for word, i in word_index.items():\n",
        "    embedding_vector = embeddings_index.get(word)\n",
        "    if embedding_vector is not None:\n",
        "        # Words not found in embedding index will be all-zeros.\n",
        "        # This includes the representation for \"padding\" and \"OOV\"\n",
        "        embedding_matrix[i] = embedding_vector\n",
        "        hits += 1\n",
        "    else:\n",
        "        misses += 1\n",
        "print(\"Converted %d words (%d misses)\" % (hits, misses))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1fnUZ3QE-PBu"
      },
      "source": [
        "Next, we load the pre-trained word embeddings matrix into an `Embedding` layer.\n",
        "\n",
        "Note that we set `trainable=False` so as to keep the embeddings fixed (we don't want to\n",
        "update them during training)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xz7rJ3sp-PBv"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.layers import Embedding\n",
        "\n",
        "embedding_layer = Embedding(\n",
        "    num_tokens,\n",
        "    embedding_dim,\n",
        "    embeddings_initializer=keras.initializers.Constant(embedding_matrix),\n",
        "    trainable=False,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMumMiFL-PBv"
      },
      "source": [
        "## Build the model\n",
        "\n",
        "A simple 1D convnet with global max pooling and a classifier at the end."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6LjIjmk4-PBw"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "int_sequences_input = keras.Input(shape=(None,), dtype=\"int64\")\n",
        "embedded_sequences = embedding_layer(int_sequences_input)\n",
        "x = layers.Conv1D(128, 5, activation=\"relu\")(embedded_sequences)\n",
        "x = layers.MaxPooling1D(5)(x)\n",
        "x = layers.Conv1D(128, 5, activation=\"relu\")(x)\n",
        "x = layers.MaxPooling1D(5)(x)\n",
        "x = layers.Conv1D(128, 5, activation=\"relu\")(x)\n",
        "x = layers.GlobalMaxPooling1D()(x)\n",
        "x = layers.Dense(128, activation=\"relu\")(x)\n",
        "x = layers.Dropout(0.5)(x)\n",
        "preds = layers.Dense(len(class_names), activation=\"softmax\")(x)\n",
        "model = keras.Model(int_sequences_input, preds)\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5paCOAdA-PBw"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "First, convert our list-of-strings data to NumPy arrays of integer indices. The arrays\n",
        "are right-padded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oLfBCjQH-PBw"
      },
      "outputs": [],
      "source": [
        "x_train = vectorizer(np.array([[s] for s in train_samples])).numpy()\n",
        "x_val = vectorizer(np.array([[s] for s in val_samples])).numpy()\n",
        "\n",
        "y_train = np.array(train_labels)\n",
        "y_val = np.array(val_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2dyKMVvM-PBx"
      },
      "source": [
        "We use categorical crossentropy as our loss since we're doing softmax classification.\n",
        "Moreover, we use `sparse_categorical_crossentropy` since our labels are integers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NVDON_QH-PBx"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    loss=\"sparse_categorical_crossentropy\", optimizer=\"rmsprop\", metrics=[\"acc\"]\n",
        ")\n",
        "model.fit(x_train, y_train, batch_size=128, epochs=20, validation_data=(x_val, y_val))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q8r94p94-PBx"
      },
      "source": [
        "## Export an end-to-end model\n",
        "\n",
        "Now, we may want to export a `Model` object that takes as input a string of arbitrary\n",
        "length, rather than a sequence of indices. It would make the model much more portable,\n",
        "since you wouldn't have to worry about the input preprocessing pipeline.\n",
        "\n",
        "Our `vectorizer` is actually a Keras layer, so it's simple:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k0EBLl2K-PBy"
      },
      "outputs": [],
      "source": [
        "string_input = keras.Input(shape=(1,), dtype=\"string\")\n",
        "x = vectorizer(string_input)\n",
        "preds = model(x)\n",
        "end_to_end_model = keras.Model(string_input, preds)\n",
        "\n",
        "probabilities = end_to_end_model.predict(\n",
        "    [[\"this message is about computer graphics and 3D modeling\"]]\n",
        ")\n",
        "\n",
        "class_names[np.argmax(probabilities[0])]"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "pretrained_word_embeddings",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}