unsupervised_lexical_simplification_with_bert.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "unsupervised_lexical_simplification_with_bert.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyON8SEl9gy4ZSbGuDcmQmwA",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tkmaroon/use-cases-of-bert/blob/master/unsupervised_lexical_simplification_with_bert.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pWD5xb4L_JV",
        "colab_type": "text"
      },
      "source": [
        "# Unsupervised Lexical Simplification with BERT\n",
        "## Overview\n",
        "Lexical simplification aims to replace complex words in a given sentence with their simpler alternatives whlie preserve meaning.  This notebook performs lexical simplification with BERT following \"*A Simple BERT-Based Approach for Lexical Simplification*\" [1]. The proposed method is a very simple, but effective.   \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e1SFYlc3QsTw",
        "colab_type": "text"
      },
      "source": [
        "## Settings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vhYTBzUDQp8X",
        "colab_type": "code",
        "cellView": "both",
        "outputId": "5c155e7c-6cf5-4a79-96f0-18406147bdd6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "#@title Setup environment\n",
        "!pip install --quiet pytorch-transformers\n",
        "!pip install --quiet pytorch-nlp\n",
        "!pip install --quiet tqdm"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 184kB 2.8MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.0MB 8.2MB/s \n",
            "\u001b[K     |████████████████████████████████| 870kB 20.2MB/s \n",
            "\u001b[?25h  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[K     |████████████████████████████████| 92kB 2.9MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMCjFnONQvPR",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "#@title Setup common imports\n",
        "from collections import Counter\n",
        "from tqdm import tqdm\n",
        "import numpy as np\n",
        "import torch\n",
        "from pytorch_transformers import (\n",
        "    BertTokenizer,\n",
        "    BertForMaskedLM,\n",
        ")\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A90nF_ctPVy_",
        "colab_type": "text"
      },
      "source": [
        "## Substitution Generation\n",
        "We obtain substitution candidate...\n",
        "<!-- Figure: The model architechture -->\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ape_pX4ARUKT",
        "colab_type": "code",
        "outputId": "1fc06ad9-3c7f-43f8-b13a-a8229e428dfd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Build model\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "model = BertForMaskedLM.from_pretrained('bert-base-uncased')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 231508/231508 [00:00<00:00, 5965161.35B/s]\n",
            "100%|██████████| 361/361 [00:00<00:00, 76719.89B/s]\n",
            "100%|██████████| 440473133/440473133 [00:07<00:00, 59185022.20B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3P-4wiBxPX7S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Model's input\n",
        "text = \"[CLS] the cat perched on the mat [SEP] the cat perched on the mat [SEP]\"\n",
        "masked_idx = 10"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gWo9CFXIRzmw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Tokenize a text\n",
        "tokenized_text = tokenizer.tokenize(text)\n",
        "\n",
        "# Mask a complex token which should be substituted\n",
        "complex_word = tokenized_text[masked_idx]\n",
        "tokenized_text[masked_idx] = '[MASK]'\n",
        "\n",
        "# Convert inputs to PyTorch tensors\n",
        "tokens_ids = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
        "segments_ids = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\n",
        "tokens_tensor = torch.tensor([tokens_ids])\n",
        "segments_tensors = torch.tensor([segments_ids])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QsyoSBm3SEJi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Predict a masked token\n",
        "model.eval()\n",
        "if torch.cuda.is_available():\n",
        "    tokens_tensor = tokens_tensor.to('cuda')\n",
        "    segments_tensors = segments_tensors.to('cuda')\n",
        "    model.to('cuda')\n",
        "\n",
        "with torch.no_grad():\n",
        "    outputs = model(tokens_tensor, token_type_ids=segments_tensors)\n",
        "    predictions = outputs[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lSueDInYSSXO",
        "colab_type": "code",
        "outputId": "959cf97f-fc73-40b1-aa65-bd8a35f2b3c0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336
        }
      },
      "source": [
        "# Output top 10 of candidates\n",
        "topk_score, topk_index = torch.topk(predictions[0, masked_idx], 10)\n",
        "topk_tokens = tokenizer.convert_ids_to_tokens(topk_index.tolist())\n",
        "print(f'Input: {\" \".join(tokenized_text)}')\n",
        "print(f'Top10: {topk_tokens}')\n",
        "\n",
        "# Visualize output probabilities\n",
        "plt.bar(topk_tokens, torch.softmax(topk_score, 0).tolist())\n",
        "plt.xticks(rotation=70)\n",
        "plt.ylabel('Probability')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: [CLS] the cat perched on the mat [SEP] the cat [MASK] on the mat [SEP]\n",
            "Top10: ['perched', 'sat', 'landed', 'was', 'rested', 'stood', 'settled', 'hovered', 'sitting', 'crouched']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEdCAYAAAABymAfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZhcZZXH8e8viSEg+y4ECPvqKBAB\nB2RRZJUwDsgmIIvGBQREUWAGVBxHdhWEYYIgm2yKMhECCAi4sAYQkG0IESQMS5BVEALkzB/nlrk0\n3ekOqbeK9P19nqefrq3rfaur6p53PVcRgZmZNdeQblfAzMy6y4HAzKzhHAjMzBrOgcDMrOEcCMzM\nGm5YtyswqxZddNEYNWpUt6thZjZHuf3225+JiMV6u69YIJB0JvAJ4OmIWKuX+wX8ENgGeAXYKyLu\n6O95R40axcSJE9tdXTOzQU3So33dV3Jo6Cxgq5ncvzWwcvUzFvivgnUxM7M+FAsEEfFb4NmZPGR7\n4JxINwMLSnpfqfqYmVnvujlZvDTwWO36lOq2t5E0VtJESROnTp3akcqZmTXFHLFqKCLGRcToiBi9\n2GK9znWYmdk71M1A8DiwTO36yOo2MzProG4GgvHAnkobAC9ExBNdrI+ZWSOVXD56AbApsKikKcA3\ngfcARMRpwARy6egkcvno3qXqYmZmfSsWCCJi137uD2C/UuWbmdnAzBGTxWZmVs4cl2Jidow69PLi\nZTxy9LbFyzAzayf3CMzMGs6BwMys4RwIzMwazoHAzKzhHAjMzBrOgcDMrOEcCMzMGs6BwMys4RwI\nzMwazoHAzKzhHAjMzBrOgcDMrOEcCMzMGs6BwMys4RwIzMwazoHAzKzhHAjMzBrOgcDMrOEcCMzM\nGs6BwMys4RwIzMwazoHAzKzhHAjMzBrOgcDMrOEcCMzMGs6BwMys4RwIzMwazoHAzKzhHAjMzBrO\ngcDMrOEcCMzMGs6BwMys4YoGAklbSXpQ0iRJh/Zy/7KSrpN0p6S7JW1Tsj5mZvZ2xQKBpKHAKcDW\nwBrArpLW6PGwfwcujoi1gV2AU0vVx8zMeleyR7AeMCkiJkfENOBCYPsejwlg/uryAsD/FayPmZn1\nomQgWBp4rHZ9SnVb3beA3SVNASYAX+7tiSSNlTRR0sSpU6eWqKuZWWN1e7J4V+CsiBgJbAOcK+lt\ndYqIcRExOiJGL7bYYh2vpJnZYFYyEDwOLFO7PrK6rW5f4GKAiLgJGAEsWrBOZmbWQ8lAcBuwsqTl\nJQ0nJ4PH93jMX4CPAUhanQwEHvsxM+ugYoEgIt4A9geuAu4nVwfdK+koSWOqh30V+Jyku4ALgL0i\nIkrVyczM3m5YySePiAnkJHD9tiNrl+8DNixZBzMzm7luTxabmVmXORCYmTWcA4GZWcM5EJiZNZwD\ngZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZ\nWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnD\nORCYmTWcA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZWcMNKBBI+oWkbSU5cJiZ\nDTIDPbCfCuwGPCTpaEmrDuSPJG0l6UFJkyQd2sdjdpJ0n6R7JZ0/wPqYmVmbDBvIgyLiGuAaSQsA\nu1aXHwNOB86LiNd7/o2kocApwMeBKcBtksZHxH21x6wMHAZsGBHPSVp8tl+RmZnNkgEP9UhaBNgL\n+CxwJ/BDYB3g6j7+ZD1gUkRMjohpwIXA9j0e8znglIh4DiAinp6l2puZ2Wwb6BzBL4HfAfMA20XE\nmIi4KCK+DMzbx58tDTxWuz6luq1uFWAVSX+QdLOkrfoof6ykiZImTp06dSBVNjOzARrQ0BBwekRM\nqN8gaa6IeC0iRs9m+SsDmwIjgd9Ken9EPF9/UESMA8YBjB49OmajPDMz62GgQ0P/0cttN/XzN48D\ny9Suj6xuq5sCjI+I1yPiz8D/koHBzMw6ZKY9AklLksM5c0taG1B11/zkMNHM3AasLGl5MgDsQq48\nqruUnHz+iaRFyaGiybP0CszMbLb0NzS0JTlBPBI4sXb7S8DhM/vDiHhD0v7AVcBQ4MyIuFfSUcDE\niBhf3beFpPuAN4FDIuKv7+iVmJnZOzLTQBARZwNnS9ohIi6Z1Sev5hUm9LjtyNrlAA6ufszMrAv6\nGxraPSLOA0ZJetvBOiJO7OXPzMxsDtLf0NB7q999LRE1M7M5XH9DQ/9d/f52Z6pjZmad1t/Q0Ekz\nuz8iDmhvdczMrNP6Gxq6vSO1MDOzrhnIqiEzMxvE+hsa+kFEHCTpV8DbUjtExJhiNTMzs47ob2jo\n3Or38aUrYmZm3dHf0NDt1e8bJA0HViN7Bg9WqaXNzGwON6Dso5K2BU4DHibzDS0v6fMRcUXJypmZ\nWXkDTUN9ArBZREwCkLQicDngQGBmNocbaBrql1pBoDKZTDxnZmZzuP5WDf1rdXGipAnAxeQcwafI\nNNNmZjaH629oaLva5aeATarLU4G5i9TIzMw6qr9VQ3t3qiJmZtYdA101NALYF1gTGNG6PSL2KVQv\nMzPrkIFOFp8LLEmesewG8oxlniw2MxsEBhoIVoqII4CXq/xD2wLrl6uWmZl1ykADwevV7+clrQUs\nACxepkpmZtZJA91QNk7SQsARwHjyjGVHFKuVmZl1zIACQUT8uLp4A7BCueqYmVmnDWhoSNIikk6W\ndIek2yX9QNIipStnZmblDXSO4ELgaWAHYEfgGeCiUpUyM7POGegcwfsi4ju16/8haecSFTIzs84a\naI/g15J2kTSk+tkJuKpkxczMrDP6Szr3EplkTsBBwHnVXUOAvwFfK1o7MzMrrr9cQ/N1qiJmZtYd\nA50jQNIYYOPq6vURcVmZKpmZWScNdPno0cCBwH3Vz4GSvleyYmZm1hkD7RFsA3wwIqYDSDobuBM4\nrFTFzMysMwa6aghgwdrlBdpdETMz646B9gi+B9wp6TpyBdHGwKHFamVmZh3TbyCQJOD3wAbAh6qb\nvxERT5asmJmZdUa/Q0MREcCEiHgiIsZXPwMKApK2kvSgpEmS+uxBSNpBUkgaPQt1NzOzNhjoHMEd\nkj7U/8NmkDQUOAXYGlgD2FXSGr08bj5yRdIts/L8ZmbWHgMNBOsDN0t6WNLdku6RdHc/f7MeMCki\nJkfENDJx3fa9PO47wDHAqwOutZmZtc1AJ4u3fAfPvTTwWO36FHqc3lLSOsAyEXG5pEP6eiJJY4Gx\nAMsuu+w7qIqZmfWlv1xDI4AvACsB9wBnRMQb7ShY0hDgRGCv/h4bEeOAcQCjR4+OdpRvZmapv6Gh\ns4HRZBDYGjhhFp77cWCZ2vWR1W0t8wFrAddLeoRclTTeE8ZmZp3V39DQGhHxfgBJZwC3zsJz3was\nLGl5MgDsAuzWujMiXgAWbV2XdD3wtYiYOAtlmJnZbOqvR/B668KsDglVj9+fPG/B/cDFEXGvpKOq\nBHZmZvYu0F+P4AOSXqwuC5i7ui5yi8H8M/vjiJgATOhx25F9PHbTAdXYzMzaqr/zEQztVEXMzKw7\nZiXpnJmZDUIOBGZmDedAYGbWcA4EZmYN50BgZtZwDgRmZg3nQGBm1nAOBGZmDedAYGbWcA4EZmYN\n50BgZtZwDgRmZg3nQGBm1nAOBGZmDedAYGbWcA4EZmYN50BgZtZwDgRmZg3nQGBm1nAOBGZmDedA\nYGbWcA4EZmYN50BgZtZwDgRmZg3nQGBm1nAOBGZmDedAYGbWcA4EZmYN50BgZtZwDgRmZg3nQGBm\n1nAOBGZmDedAYGbWcEUDgaStJD0oaZKkQ3u5/2BJ90m6W9K1kpYrWR8zM3u7YoFA0lDgFGBrYA1g\nV0lr9HjYncDoiPgn4OfAsaXqY2ZmvSvZI1gPmBQRkyNiGnAhsH39ARFxXUS8Ul29GRhZsD5mZtaL\nkoFgaeCx2vUp1W192Re4orc7JI2VNFHSxKlTp7aximZm9q6YLJa0OzAaOK63+yNiXESMjojRiy22\nWGcrZ2Y2yA0r+NyPA8vUro+sbnsLSZsD/wZsEhGvFayPmZn1omSP4DZgZUnLSxoO7AKMrz9A0trA\nfwNjIuLpgnUxM7M+FAsEEfEGsD9wFXA/cHFE3CvpKEljqocdB8wL/EzSHyWN7+PpzMyskJJDQ0TE\nBGBCj9uOrF3evGT5ZmbWv3fFZLGZmXWPA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWc\nA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOB\nmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZwzkQmJk1nAOBmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZ\nwzkQmJk1nAOBmVnDORCYmTWcA4GZWcM5EJiZNZwDgZlZww0r+eSStgJ+CAwFfhwRR/e4fy7gHGBd\n4K/AzhHxSMk6dcuoQy8vXsYjR29bvAwzG3yKBQJJQ4FTgI8DU4DbJI2PiPtqD9sXeC4iVpK0C3AM\nsHOpOjWVg5CZzUzJHsF6wKSImAwg6UJge6AeCLYHvlVd/jnwI0mKiChYL+ugbgeh0uU7ANpgoFLH\nXEk7AltFxGer63sA60fE/rXH/Kl6zJTq+sPVY57p8VxjgbHV1VWBB4tUuneLAs/0+yiX7bJdtst+\nd5e9XEQs1tsdRecI2iUixgHjulG2pIkRMdplu2yX7bIHS9k9lVw19DiwTO36yOq2Xh8jaRiwADlp\nbGZmHVIyENwGrCxpeUnDgV2A8T0eMx74THV5R+A3nh8wM+usYkNDEfGGpP2Bq8jlo2dGxL2SjgIm\nRsR44AzgXEmTgGfJYPFu05UhKZftsl22y+6UYpPFZmY2Z/DOYjOzhnMgMDNrOAcCs4okVb+HtC6b\nNYEDQT/qB4QqbYYNXnNLGhER01ur1yQN7URQkLS6pIVLlzOnaFogljRfN8t3IOhHRISkDavLb8I/\nWoxF/ne1VulOkt7TrS+EpFUkbS5pZLXHo1Plzt/LbZ36nO4AvCLpV5K2gHzPa0GhZD32AY6V9BVJ\nH+zk/xxA0oclnSBpi97egw6Uv6qk0yWtC/m9q27vRBAe1s1GnqQPAUdK2qVqEHR8o69XDfVD0prA\n7cCTwLnkMtg/1+4fEhHT21TWMGApYDngjIhYpR3P+w7r8jVgQ+AO4C/AncATETG1cLnHA1dExLWS\nFoqI50qW10v5w4GzyTxYLwFXAt+PiD8WLvOfqp/NgSWBPwH3ANe28nWVJGkzYBNgcWAu8v2+Hbi1\n1QAqXP5ywBHAPwMvAFcDP6l/1wqUOSQipkvaGtiJXOr+x4h4oFSZfdTjI8BmwILA+4A/AvcDd7TS\n7xSvgwPBwEg6ADgEWIg8MJ5GflBfamMZS5NJ+LYik/PtA0yLiKmSPghsExH/2a7yBlCfDwDHAqOB\nG4GJ5Id0MvBgRExrc3mLA9cBHyC/EKdVZe8cEde3s6xeym4dFOYhU6dfSu5t2QP4NDAN2CAiHi5Y\nh28CS5Cfr6nAx4DFgKsj4thS5dbKXxw4DliJDEQALwMPAedHxAsdqMN3yHxiS1X1mEwG5nMi4u+F\nyjwa2JI8+L5E/u9vBG6MiOc7kQiz6m2OB6YDTwOvAPMBDwDnRsT/lSx/jsg11E21Fv8mwO7kh2Uj\n4HvADyTtEBG/bEdZEfG4pC+SB8B5yHM1PC7p8qrs+2b29+0iaVhEvAGMAp4g04BsQLaSzwbOB75c\noOiPAr+tNiN+jWyhnQp8Eri+QHl1rS/63sD7IqKVtvQmSXcCHywZBCo7ARtWB595gUfIxsftJQuV\nNLRq9W8OzBsRG0paAVgH+AbZ+PlxwfJVDcGuBWwZEetVt48A/hs4AbiBPCi2u8z1yO/zJuTnfQ3g\nq2RKnNWBE0oGgdr//lPAaxGxg6T3Ah8iG4VDycBQlANBP6pW4kbAKhFxQxUYfiFpCvnmXQszPliz\nU1b1HG9I+nJE/F3SgsA2wBbkgfCM2Xs1A1MFAcgDw/0R8QrwG+A3kp4GRhQaLvhfYH9JTwHHRcRJ\nkvYD5i5Q1lvU3rsbgfUkrVAbklmcbKUXUx30bgAOl3RCRDwFXCvpGODekmXX3svFyfeA6rVPlrQE\nsFBEvF6w/Nb/fjngeUmLAM9HxKuSjgWeafdwTa3M9wN3RcSLwN3A3dW8xNbAhpL+GhFntbPsHvVo\n/e+HA9MlLRgRzwPXSzqFbBi80fcztIcDwcD8kWwZ7h0RP6luWw0YWX2AaEMQaA1NbAeMkrQomW77\nhog4f3aeezaMBw6VNJn8kvyZnDc4uURhEXGHpC2BNSPi1moC70vAniXK66MOd0q6D7hV0h3AreRQ\n3RdKlVk1AF6tvvgHk8FgUeDvwFMR8WSpsnu4ErhI0kLAr8meyM7AiZ0oPCIul7QpeYKqcdVwyeGU\n7RH9DviKpJOAn0fEb8kgcAkZGFcvWHbd5WSv5Ehlev43yaHh73eicM8RDFC1cuiH5ETedcAi5BzB\nz2rdu9ktYxh54BlHTpxNILuok8gJy+KThr3U6dNk1/lNstv8ZETsVqCcFYHlybH4J4DngdfJFlHR\ns8vUgvBQYH0yR/xfyC/iEODS0pN2VdlLknMiw4H5AQG/qnoHHVENCW1JDk2MJucGjp75X7Wt7GHk\n//tgYAzwMNlDOTUiimUllrQsmedsVXLS9jpy6PNS4IiIuKVQuW8ZRajm5DYnJ41XJIdJTytR9tvq\n4kDQu9rBYV7yCzEyIs6TtDbZG7i81RtoY5k7kR+Eg8lVE/9Cjo++Bny+E13EWl0+QB74n69+3ktO\nGj7bzgnyWnmXk2OhW5Kv/XGyF3J1yYNAjzr8ghyGORxYKyLulzR3NUzX9gnDVgOiamQcSK7WeRh4\nlRyb7lhKdkm7kRP0I4GzyEnaV4Dp7X7dPcpt/Q+2AT5Bfte2i4inJL03Il4uUGbru70WORw1mRx+\nfI6cLH6RDMoHRsRX211+L/X5ItnzGA4cTw6FPV+63DrvI+hb68N/DrmK5UeS9oyIO8kWw0vVWGI7\nLVKV9ynguqoleDnwSieCQOv1SFod+C4ZiE6KiJuA35If0BJB4KPk6Nre5PkoLiFbhLuSPYRiquEH\nqiGJIWSv77YqCCwLfK86IJU4GLae8yjgV8C/AReRLcLDJQ0v8Bn7hx7v9+fIFStbRsRdZFBah+yV\nFFMFgSHk8NOxwNLA0pLmArarhqnaXeZ0SXOTE+BHkkOdnwE2JhdGTI+Iv5ANgiJq//s1gd2Aa4Bt\ngUeBNyRtpw7uJ3Ag6EO1omAkuZzvZHK10ITq7mOA5dt5cFAuW3yKXBl0C7C9pOOA/cgWcie0vvR7\nAheTX8yJ1W2fJFczlbAOGWg/TZ6TYjxwKLlvoe2Bpy5m7AHZEDiP/DLeVN22Bjlf0fZWaavs6n0f\nApwXEX+qhiEOIhsfKxVetth6v/cAziTHy++ublsb+Fa0aY9Mr4XPCHLbkw2N14E/R8QdwAjgK+Q8\nSTvLbB3zxgC3R8T65BDg/eTcwBHkMCgR8Vo7y+7DtmTjbzLw+2pS/sPAYZ0cAXAgmLm1yIP/PwOP\nRsQzVStxnXaN19c+mAcCH4mIZ8nu6V3AdsDp1YGxE1pfzKFk9/hb5MER8sN5f9sLzIPBB4G/kUND\na0v6JNlCnTizv22z08n/9zFAa0HAF8ilsiVNIwPPLZLGVC3g+YEVIqL0cuFWkLmHHBY6hhnLRLcD\n/lC08BlB7mHyRFaH18r/JPBYRLza5mJb37cPAMMlzRMRf4mI0yJiF+CAqhFY9NhYe+0PkPMBZ5G9\nQciVghN6+bNivGpoJiLiSkk7AF8nh0ogWwzj4S1rgGenjFaLawvgc1Uv5PvkiqF6L6S42msZR77m\n5YHHqrmLjcgvZ7vNT77OjwA/A04iV6o8S7aUiqmNT69Etjx/QR4QJ0i6h/ySFg0EkcuF/53sCa4L\nfAeYAhSfoK0djK4Evk2Oi79fudN1NPk+FKVcMy/ywLwXMLSal9uB/F+0VfX/HkEOg60NnCrpbjIY\n3xoRU6r5oGI9oR71GV99/qYDK0j6OTkf97lOlN/iyeJ+VMv49iI3Gk0Dfkmmf3i8XROIkhYgWwR3\nAZuSB+JLyA/nXhFxd59/3CaSfgQ8Rq6EelrS+sAB1d2vAb+OiAsLlT0cOIw8XekvyTQej5Qoq0e5\no8iezyXk5Oxl1e2LA8tFxG2Fym1tZpqH7A2tQI6NTyH3iwwjW8Mlh2XOIedjzoyIe6oD8h7AwsxY\nEfenmT3HbJbfmrDdj9yXcoIyz9AXyJ7hT9vdI5L0JWB8awWYpGXIVUKrk///RyPiiHaW2Uc9Wu//\ncPIY/JqkjwFrkqMBf4iI/y1djzr3CHpQtau2WlGwNDlWOZ5spc9Xn81v1/htRLxQtQp3AK6MiPOV\nOx5f71AQEDk+vD1wnaSbgQvJ3cxzFeiet8pdgOwO/4qcFH+DDD4flLRvFMxrVE0Wbkz2xFYDXq+W\nTj5TBcLtJD1SqA5DyHHo/cg0Ev9H9gjWA+6NgnmNan5MTpD+XLlJ8KfAL6NDS1VrQW5RcoKUiLid\nWku4wEqtvwJPSrqEXAl3WkScUw0DbUA1VKY25g/rTRUEhgLfBD5R/f9PjIiTSpXZH/cI+lANDUwi\nW8kihyoeIdd1P1OozHpLYV/gzYgofl7T2hDJaGAsmedlLjLPza+ByyI32rS73DXI3dIrkJPTl5GT\ntqMjYpt2l9dL+XMD+5MTdg+QPZ+7yAP1f0bE4oXLfwj4UGRKiVFkz3N1cpy62AG554FO0s7kAoHV\nyGG6CyPivL7+vo31GEk2Ap4jV+/cB7wUBXcxV6uRViMbXR+vbr4MuCgiJpUqt1b+PBHxSjXcuh/Z\nC94N+Cy5lPUaYIfCiwTeXi8HghkkrRoRD0pamZy136e6vDqwMrm07PDIlAul6zKC7BF0IvNjq5t+\nLZnb53/I1tG+5J6GCRFxcKGy5yX3DmwEXBURV5Yop5dyW0H3n8lMm4uT8xQfJtfP3x0R5xYsfyFy\nCPB71SqZ1u23kckFS/aGRkTuZB4PnBwRV9fqtCewYER8u1T5VVmtydjNyR21G5I7128ErolCWUeV\na/YfiIjrqs/eBuQy5XkjohNzIp8je0AbkiukzqrdtwKwUUQUnRvrjYeG3mo/ZYqHyVS5ZSLiIeCh\nqpW+dCeCQFVukeGYPsqaXgWe18gvSWvZ2jhJ65CJv9pK0mLk7s3h5Cauh4FDJK3fgYNQKwisRi7V\n3D0iHq1aa0Vbwq2yI+I55Sa605Ub2R4H5iX3jJQMAvMBX1euX1+LWjK/qk5/oHCSu6qs6dWQ5N/I\nTVTHk0s6P01O3LctECjX43+GnIQdS06EExF/A66pembPVI8tNixU/e8/QDY45iJzGT1D9oSeiCq/\nU4my+62bewQzVB/MNcksmAeRk8PjyYmru7pZt06QtDu5euRM4AJyiOinEbFcgbJGkq3BN8ilipBp\nd+eJiE3bXV6PsltDYSeTE4THSzoI+Bo5jrxtlE8pcXJEfLlaobMe+Zm7h0y3XHTZqDKZ4YnkROlz\n5AbJH5NDoNdExFIFy271Pj9MTgwPB5aIiI/2fEwbyxxGHoCPq35fS36vz6/qcinZGPhbu8rspz6L\nku/5x8mJ+SfIock/lVqg0G+dHAhS7QO6ABkAppPdxk+SK3n+GhEf62IVO6JaLbQvOVxzLZni4YIO\nlDuMbLG91qnekKRzybmJZcm13IeT6cVvjIifFShvlaqc5YGPRsSO1RDJXFEo134vdWh9zg8j904s\nQx6QNyeXLF8TEcWSzNXKP5Ocf1qRHIo6RLmhkIj4aZvLbPUADyNXZg0hd++vTAb+JyLTPxedJK4t\nRNmH3EA4rVqUsgl5jLksIs4uVf5MRYR/MhgOqX6fDXy4x32LAetWl4d2u65tfM2thsAi5KTVWHLi\ndMlu161Dr39LctPaRHIH+RByWGTZQuWNIpcJT6vK3B5YuLrvS8DxhV/v/GTQ25icA+l5/8Kd+HyT\nPY+rqsuXkRs0Ifdx7NPmsobVLv+cXKoKOTSzKrlqbFR1W7HXXvuurUbOg8zdy2PmKv2/7+vHcwT8\no8UwvRquWDcyt05rqEhky+FWeMumq8FA5KTw18lxy9vI17qFpGfJLe/XdrF+pf2BbI1F5EqOncmk\nekXOPRC5N2IvSY+S68UPAo6W9Euy91k63fPS5OT/VsADyjPivRkRTyqTDG4REceVrECtR3CupN8C\ni0emHx9J9g7a3ftcTXn606XJlN6vQqaPkPQY2eh5pLqtE9/tPcjTj/5d0lxVPTYh//f/1oHye+UU\nE7xlP8AGVCfmkDS8un15sqXWsbwfnRIzusFLkCtVDiTXk99EDtMUP3F4N0gaIum7ZAqNC4H3KE/Y\nfg25cbBEmUOr34eR+1GOB/6VDL4fo7ahrZSIuB/4InAFuXfhXDIQ7QT8B9nzLaYWBNaOnJQfDzyh\nzL9/Ajlc0tYhsshNcZ8mV4KtKekvkn5Qvd97kkt263mPiqgdY+4hzzeyZMzIZfRJcmNj13iOoKZa\nV34y2RI+q1opdAw5bn2o2nTegXeD2rjpumQr7LCIuKR2/4LAy1FwTXen1SaJdyNbxb8Ddo2Ijyoz\ncK5Y+mAs6Tpy09Sb5OTlQ+RqrVMjotgpCVvvd3V5OTIQzEUeJDcgD1BnRuH0x9V37HpyUvyU6oC8\nLLlaqq0rZqrlmItExG2SPh4RV0vagNy0uA25bPiQiJhYen6gVqeFyZ7fE+RKpXmrunyqVE90QPVq\neiCotVJWJb8QL5CtowWAm8nlbd+OiEfqX6bBoprA/C65fv4hspV2QTVcMOheL4Cki8mcUVuR68e/\nK+kr5DkI9i1YbtdSidSC4AHkXMHuwEHRoX0bPeqyKpld9imyJ1Rkuayk7cnJ4KHk//sa4KbWAb9q\nlRc/+1vtGLNiRDysTGGyE3mMWZhcvVR8ye5M6zgIv+ezpPYmfYccQ/xRdfsqwFIRcX1XK9ghVQ9g\nS3It9+ZkS/k33a1V+1Ut0k7ABsMAAAarSURBVN3IFNOfADaIXD//a/IscFcULn9NclfrtIg4WplK\n5OTIdMhFacYZ8D5DLhHeJzLP0BfJ9BKdOiVmq7X+JXIj31FRcFevpM+SS2WXJIPP1eTpZ+/uVA+/\nGnq6hRyiuoicLO/KnoHeND4QtEg6muyi/wB4cbAMAfWmFvwWJg/685MfzMeq+5cDpgym/0HtNR9M\n7hV5lAx6d5MbKxeKiO1m9hxtrEu3UomMIYchvk7uFt9ImfjuFjKtR5H8+7XXuyS5SucOco/KCPLz\ntxmwabTxjGy1HtC6wKcj4mDlTuITyPNfBDCmQz2C+rDcv5DDceuSQemHUSiZ46zwqiFAeb7czYD3\nkB+Qq5S7DZ8bTAfDXvyEPAgcCrwg6UFy8vTiwfa6a+O/y5Kb5H6j3EW7FPm+X9zBukT1e5qkn5An\nZOmEW8jhzwuZ8Xr3Bf5YKgjAWyZKVyEn43cgN1D9E7ko4e52BoEe9qnKgpwofy85R3F6p3pAVRBc\nPCKejohLgUuVmU9/yoz3o6sa3yOoTxJVY5djyd2H08jshJ06KUxHVeOUl0XEepJ+T05gHkyOHW8c\nXdrhWFL1mieSDaDDybmQTpyFqmtqrfF5ycbOUmSCs78x44Q0/9WpYcDWpqra9RHkHp4iqVskXU0O\ng61K7lw/h0x5fkFE/E+JMnupwzxkptHJZFK/ByIz3J5Kpvvu+nfNPQIYImlz8jR1D0TEV6s3bgw5\ns188LW0n1bqpmwLXS3o/mdzufklHAm+8Gz6YJVRfvg+T52LeGThQ0sPkGP0N3a1dMfWU10tUQyR3\nAp8nFwhsHYWy6cJbhuRWIXfzjpa0CHkynDOifNrr75DLRNcAtoqIF6u6FD37Wg8LkZPWK1f1+Luk\nJcj5qS91sB59amyPoDaGuDeZ5+URYL2I2FKZHvnJyNNGDlrVF3JJ8uTpNwDrk6k0DpjpHw4SyqRz\ne5LnAGhrWoN3m9qy1VfJ5YsPk8OgJ5UcIql9z8aRyeTOI/MLjSWHpL5fquyq/CHk6py/R2Zc3QnY\nKar0Hp1q4FUT9UPIhHdrkCMOf46I33Wi/P40ORC0uszXkK2lz5I5R06U9A1yaO/Y7tayvSRtROZ/\nPwb4caslWE1gbU5uIPthdPjsSFZWj2Wrm5HZZC8hUx3sXXLZalX+EHIYasPIcy8IWIlMef7V0uXX\n6jGUHApTRDxWMhDUji9LkfMie5CNzbPJ8z28qw68jd1ZXL1J72HGCdI/Qp6FDHJlxUR4S970OV5E\n/J4cK50G3CvpJkl7RMSlEbF/ROznIDD4RMQLwL+TiRSviIjzyXmwNzoQBFrn//0F1dnHqoPgo+RO\n5o593iLizYiY0lodV7g30DpufIWcl9mMXJyxP3Dhu+240tgeQYukbckUvM+QWRhXBPbrxLrubqvG\nSr9CbnG/D9g3Cp0QxLqvW8tWq7K3J3dSB5nY7xkyydrnO1F+t0i6gdy1f2PttouAH71bhoWg4YFA\n0l7kzr6NyZbyPORSrvMj4i4NopQSM1N1mTcC7hns8yKWVPgMeLWgsyjZEn6YTAG9EnlKxjOA/4lB\nlMKkN9Uc5GrkprmXqw2NN9OBc17MisYFgtoHdDS5lOw0cknX6uSk0okR8XI362g2WChPzbgZOR+w\nMrk4YS7y2PPNbtatlNpKqR3JE/98gcx+ejuwILmrvFgqk3eiiYGgtYphP/INOV3Se8klXicAd0TE\nMd2tpdmcTdKVZG6fZcjJ0VZq9yXIXbUvVnNWg1LVy/5DRGxQXd+IXDF0PTApOnQ2tIFq3D6CWld4\nH+BlSTdUE6QvS/or1S7PpgwLmbVbdRA8k1yJtgGwo6RvkTu6nwImdLF6RdVWIq0JPF0tRX+wCnrv\n2sDXuB4B/GNN7xhywmxV4ElyDPM9EbFbN+tmNphIeh+ZUmJLsncwiZwovb6b9SqlNvR8CHku7j+T\n6TweBR6PiOe6WsE+NDIQ1NU+qDuSydceAMYN1g+qWbcM9g18tSCwELkU/QJy9/aKwPPkPoKzolxe\npXes8YGgpdrksiqZovdPg/GDambl1OYfv0ae9/qA6vaFyBxea0TEF7tayT40bo6gL9UmlwfIhFRm\nZrOkNqe4EXA8/OOUt89Jmgbc27XK9eNdtbvNzGwQuAL4pqTWeQ8gU2B3MtHdLPHQkJlZG1Wbxr5B\nJtdbhty0+mxE7NHVis2EA4GZWZtVwWA98hwIrwG3RMSL3a1V3xwIzMwaznMEZmYN50BgZtZwDgRm\nZg3nQGBm1nAOBGZmDedAYGbWcP8PZlvKKyrqFIUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FVPajT4APaA1",
        "colab_type": "text"
      },
      "source": [
        "## Substitution Ranking\n",
        "Previous section, we obtained subsitution candidates of a complex word \"*perched*\".  \n",
        "In this section, we rank substitution candidates by following four features:\n",
        "\n",
        "- Probability of BERT prediction;\n",
        "- Probability of n-gram language model;\n",
        "- Similarity obtained by FastText;\n",
        "- Word frequency;\n",
        "\n",
        "We choose the top of candidate with the highest average rank over all features as the simplification replacement.\n",
        "\n",
        "In this notebook, we use only a **probabitity of BERT predicaiton** and a **word frequency** for ranking step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8BNPz8jrUysR",
        "colab_type": "code",
        "cellView": "both",
        "outputId": "29019cfb-10e0-4d7f-db10-1731ee7b73c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#@title Download the Children's Book Test (CBT)\n",
        "!wget http://www.thespermwhale.com/jaseweston/babi/CBTest.tgz\n",
        "!tar -xzf ./CBTest.tgz\n",
        "!DATA_PATH=./CBTest/data; cat ${DATA_PATH}/cbt_train.txt ${DATA_PATH}/cbt_valid.txt ${DATA_PATH}/cbt_test.txt > ./cbt_all.txt\n",
        "!wc ./cbt_all.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-02-06 05:24:26--  http://www.thespermwhale.com/jaseweston/babi/CBTest.tgz\n",
            "Resolving www.thespermwhale.com (www.thespermwhale.com)... 69.65.3.213\n",
            "Connecting to www.thespermwhale.com (www.thespermwhale.com)|69.65.3.213|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 120547669 (115M) [application/x-tar]\n",
            "Saving to: ‘CBTest.tgz’\n",
            "\n",
            "CBTest.tgz          100%[===================>] 114.96M  10.9MB/s    in 11s     \n",
            "\n",
            "2020-02-06 05:24:37 (10.5 MB/s) - ‘CBTest.tgz’ saved [120547669/120547669]\n",
            "\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.dev'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.ino'\n",
            "tar: Ignoring unknown extended header keyword 'SCHILY.nlink'\n",
            "  292912  6135083 28453805 ./cbt_all.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RHHukcMobWyz",
        "colab_type": "code",
        "cellView": "both",
        "outputId": "d60a095b-5c07-4fef-b71f-0a4cc5f8efe1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#@title Build a table of word frequency\n",
        "def count_lines(path):\n",
        "  with open(path, 'r') as f:\n",
        "    return sum([1 for _ in f])\n",
        "\n",
        "word_frequency = Counter()\n",
        "filepath = './cbt_all.txt'\n",
        "n_lines = count_lines(filepath)\n",
        "with open(filepath, 'r') as f:\n",
        "  for line in tqdm(f, total=n_lines):\n",
        "    if line.startswith(\"_BOOK_TITLE_\"):\n",
        "      continue\n",
        "    else:\n",
        "      tokens = tokenizer.tokenize(line.rstrip())\n",
        "      for token in tokens:\n",
        "        word_frequency[token] += 1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 292912/292912 [02:01<00:00, 2414.95it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R2zj_Ow9u4ch",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# substitution ranking\n",
        "bert_rank = np.array([i for i in range(len(topk_tokens))])\n",
        "frequency_rank = np.argsort([-word_frequency[token] for token in topk_tokens])\n",
        "avg_rank = np.argsort((bert_rank + frequency_rank) / 2)\n",
        "\n",
        "# sort candidates and except a complex word\n",
        "candidates = [topk_tokens[i] for i in avg_rank if topk_tokens[i] != complex_word]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sR1gcnLxwZeC",
        "colab_type": "code",
        "outputId": "633255c2-43dd-45c3-e44a-be50036f9e4a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "tokenized_text[masked_idx] = candidates[0]\n",
        "print(\" \".join(tokenized_text))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[CLS] the cat perched on the mat [SEP] the cat sat on the mat [SEP]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hzOHx9T-QgDV",
        "colab_type": "text"
      },
      "source": [
        "## References\n",
        "\n",
        "[1] [A Simple BERT-Based Approach for Lexical Simplification\n",
        "](https://arxiv.org/abs/1907.06226)  \n",
        "[2] [huggingface/transformers](https://github.com/huggingface/transformers)\n"
      ]
    }
  ]
}