nlp_second_order_attack_demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nlp-second-order-attack-demo.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPDlPbH7CCK4k2q7pSMh6Oy",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chong-z/nlp-second-order-attack/blob/main/nlp_second_order_attack_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GlfRvTFNAqj0"
      },
      "source": [
        "# Demo Notebook\n",
        "\n",
        "This notebook provides the demo for the following paper:\n",
        "\n",
        "> Chong Zhang, Jieyu Zhao, Huan Zhang, Kai-Wei Chang, and Cho-Jui Hsieh, \"*Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation*\", NAACL 2021\n",
        "\n",
        "The demo first setup the environment, and then perform the second-order attack on a pre-trained LSTM model from TextAttack. Please use the GPU runtime.\n",
        "\n",
        "Please refer to https://github.com/chong-z/nlp-second-order-attack for more details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-lCP4nEBpeV"
      },
      "source": [
        "# 1. Clone the repo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gMZAhn8cAgvR",
        "outputId": "7c80f1be-f563-4f73-c1c4-9178cf029950"
      },
      "source": [
        "!git clone --recurse-submodules https://github.com/chong-z/nlp-second-order-attack.git\n",
        "%cd nlp-second-order-attack"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'nlp-second-order-attack'...\n",
            "remote: Enumerating objects: 56, done.\u001b[K\n",
            "remote: Counting objects: 100% (56/56), done.\u001b[K\n",
            "remote: Compressing objects: 100% (50/50), done.\u001b[K\n",
            "remote: Total 56 (delta 11), reused 51 (delta 6), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (56/56), done.\n",
            "Submodule 'libs/TextAttack' (https://github.com/chong-z/TextAttack.git) registered for path 'libs/TextAttack'\n",
            "Submodule 'libs/jia_certified' (https://github.com/chong-z/certified-word-sub.git) registered for path 'libs/jia_certified'\n",
            "Submodule 'libs/xu_auto_LiRPA' (https://github.com/KaidiXu/auto_LiRPA.git) registered for path 'libs/xu_auto_LiRPA'\n",
            "Cloning into '/content/nlp-second-order-attack/libs/TextAttack'...\n",
            "remote: Enumerating objects: 22, done.        \n",
            "remote: Counting objects: 100% (22/22), done.        \n",
            "remote: Compressing objects: 100% (15/15), done.        \n",
            "remote: Total 13659 (delta 7), reused 14 (delta 7), pack-reused 13637        \n",
            "Receiving objects: 100% (13659/13659), 108.47 MiB | 34.39 MiB/s, done.\n",
            "Resolving deltas: 100% (10087/10087), done.\n",
            "Cloning into '/content/nlp-second-order-attack/libs/jia_certified'...\n",
            "remote: Enumerating objects: 55, done.        \n",
            "remote: Counting objects: 100% (55/55), done.        \n",
            "remote: Compressing objects: 100% (43/43), done.        \n",
            "remote: Total 55 (delta 15), reused 47 (delta 9), pack-reused 0        \n",
            "Cloning into '/content/nlp-second-order-attack/libs/xu_auto_LiRPA'...\n",
            "remote: Enumerating objects: 367, done.        \n",
            "remote: Counting objects: 100% (367/367), done.        \n",
            "remote: Compressing objects: 100% (230/230), done.        \n",
            "remote: Total 367 (delta 197), reused 292 (delta 122), pack-reused 0        \n",
            "Receiving objects: 100% (367/367), 3.97 MiB | 31.75 MiB/s, done.\n",
            "Resolving deltas: 100% (197/197), done.\n",
            "Submodule path 'libs/TextAttack': checked out '995f098aca785d9ac37ccbb743ad8d7d0b2ed3c6'\n",
            "Submodule path 'libs/jia_certified': checked out '54c602dcb29782a65aa5100ca9a1df1d32890c5d'\n",
            "Submodule path 'libs/xu_auto_LiRPA': checked out 'c8935c6d22cd76e137b1a9b1b3ea67f7d234601d'\n",
            "/content/nlp-second-order-attack\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OyfinsAuWj7g"
      },
      "source": [
        "# 2. Install required packages\n",
        "\n",
        "Note: Please run `setup.sh` instead of `quick_setup.sh` if you want to experiment on certified models such as Jia et al., 2019.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4JVSyYU8t3Sm",
        "outputId": "aaa625e8-b88b-45cd-e016-e0134b59a09b"
      },
      "source": [
        "!./quick_setup.sh"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
            "Collecting torch==1.7.1+cu110\n",
            "\u001b[?25l  Downloading https://download.pytorch.org/whl/cu110/torch-1.7.1%2Bcu110-cp37-cp37m-linux_x86_64.whl (1156.8MB)\n",
            "\u001b[K     |███████████████████████         | 834.1MB 1.2MB/s eta 0:04:22tcmalloc: large alloc 1147494400 bytes == 0x55ad17d8a000 @  0x7fa19d385615 0x55acde0c006c 0x55acde19feba 0x55acde0c2e8d 0x55acde1b499d 0x55acde136fe9 0x55acde131b0e 0x55acde0c477a 0x55acde136e50 0x55acde131b0e 0x55acde0c477a 0x55acde13386a 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde237431 0x55acde198049 0x55acde102c84 0x55acde0c38e9 0x55acde137ade 0x55acde0c469a 0x55acde132a45 0x55acde131e0d 0x55acde0c477a 0x55acde132a45 0x55acde0c469a 0x55acde132a45\n",
            "\u001b[K     |█████████████████████████████▏  | 1055.7MB 1.3MB/s eta 0:01:20tcmalloc: large alloc 1434370048 bytes == 0x55ad5c3e0000 @  0x7fa19d385615 0x55acde0c006c 0x55acde19feba 0x55acde0c2e8d 0x55acde1b499d 0x55acde136fe9 0x55acde131b0e 0x55acde0c477a 0x55acde136e50 0x55acde131b0e 0x55acde0c477a 0x55acde13386a 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde132ee2 0x55acde1b57c6 0x55acde237431 0x55acde198049 0x55acde102c84 0x55acde0c38e9 0x55acde137ade 0x55acde0c469a 0x55acde132a45 0x55acde131e0d 0x55acde0c477a 0x55acde132a45 0x55acde0c469a 0x55acde132a45\n",
            "\u001b[K     |████████████████████████████████| 1156.7MB 1.2MB/s eta 0:00:01tcmalloc: large alloc 1445945344 bytes == 0x55adb1bcc000 @  0x7fa19d385615 0x55acde0c006c 0x55acde19feba 0x55acde0c2e8d 0x55acde1b499d 0x55acde136fe9 0x55acde131b0e 0x55acde0c477a 0x55acde132c9e 0x55acde131b0e 0x55acde0c477a 0x55acde132c9e 0x55acde131b0e 0x55acde0c477a 0x55acde132c9e 0x55acde131b0e 0x55acde0c477a 0x55acde132c9e 0x55acde131b0e 0x55acde0c477a 0x55acde132c9e 0x55acde0c469a 0x55acde132c9e 0x55acde131b0e 0x55acde0c477a 0x55acde13386a 0x55acde131b0e 0x55acde0c477a 0x55acde13386a 0x55acde131b0e 0x55acde0c4e11\n",
            "\u001b[K     |████████████████████████████████| 1156.8MB 11kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1+cu110) (1.19.5)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.7.1+cu110) (3.7.4.3)\n",
            "\u001b[31mERROR: torchvision 0.9.1+cu101 has requirement torch==1.8.1, but you'll have torch 1.7.1+cu110 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: torchtext 0.9.1 has requirement torch==1.8.1, but you'll have torch 1.7.1+cu110 which is incompatible.\u001b[0m\n",
            "Installing collected packages: torch\n",
            "  Found existing installation: torch 1.8.1+cu101\n",
            "    Uninstalling torch-1.8.1+cu101:\n",
            "      Successfully uninstalled torch-1.8.1+cu101\n",
            "Successfully installed torch-1.7.1+cu110\n",
            "Collecting tensorflow==2.2.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4c/1a/0d79814736cfecc825ab8094b39648cc9c46af7af1bae839928acb73b4dd/tensorflow-2.2.0-cp37-cp37m-manylinux2010_x86_64.whl (516.2MB)\n",
            "\u001b[K     |████████████████████████████████| 516.2MB 33kB/s \n",
            "\u001b[?25hCollecting transformers==3.0.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/27/3c/91ed8f5c4e7ef3227b4119200fc0ed4b4fd965b1f0172021c25701087825/transformers-3.0.2-py3-none-any.whl (769kB)\n",
            "\u001b[K     |████████████████████████████████| 778kB 39.3MB/s \n",
            "\u001b[?25hCollecting sentence-transformers==0.3.4\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1d/09/36bcda3e1839fee5ba7bd64779ab3824b5f0bbf19ba32d985692c4141ec0/sentence-transformers-0.3.4.tar.gz (61kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 9.5MB/s \n",
            "\u001b[?25hCollecting sentencepiece==0.1.91\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f2/e2/813dff3d72df2f49554204e7e5f73a3dc0f0eb1e3958a4cad3ef3fb278b7/sentencepiece-0.1.91-cp37-cp37m-manylinux1_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 50.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 5)) (1.4.1)\n",
            "Collecting scikit-learn==0.24.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f3/74/eb899f41d55f957e2591cde5528e75871f817d9fb46d4732423ecaca736d/scikit_learn-0.24.1-cp37-cp37m-manylinux2010_x86_64.whl (22.3MB)\n",
            "\u001b[K     |████████████████████████████████| 22.3MB 1.4MB/s \n",
            "\u001b[?25hCollecting flair==0.6.1.post1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4a/49/a812ed93088ba9519cbb40eb9f52341694b31cfa126bfddcd9db3761f3ac/flair-0.6.1.post1-py3-none-any.whl (337kB)\n",
            "\u001b[K     |████████████████████████████████| 337kB 45.4MB/s \n",
            "\u001b[?25hCollecting pyarrow==0.17.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/78/dcd7f290cd018581b5c73f6c87e2b004f1161cdf6f55c7b2c87d78174592/pyarrow-0.17.1-cp37-cp37m-manylinux2014_x86_64.whl (63.8MB)\n",
            "\u001b[K     |████████████████████████████████| 63.8MB 44kB/s \n",
            "\u001b[?25hCollecting wandb\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/47/af/4cfe48fe55046181b992251933cff4ceb3bfd71a42838f5fe683683cd925/wandb-0.10.25-py2.py3-none-any.whl (2.1MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1MB 47.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 11)) (1.1.5)\n",
            "Collecting bert-score\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/27/ccf86d5dfc19f89bee4449e96ac6e0f7c312f1614de86609c5f6da5c40af/bert_score-0.3.8-py3-none-any.whl (58kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 8.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: datasets in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 13)) (1.5.0)\n",
            "Collecting visdom\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c9/75/e078f5a2e1df7e0d3044749089fc2823e62d029cc027ed8ae5d71fafcbdc/visdom-0.1.8.9.tar.gz (676kB)\n",
            "\u001b[K     |████████████████████████████████| 686kB 50.1MB/s \n",
            "\u001b[?25hCollecting tensorboardX\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/07/84/46421bd3e0e89a92682b1a38b40efc22dafb6d8e3d947e4ceefd4a5fabc7/tensorboardX-2.2-py2.py3-none-any.whl (120kB)\n",
            "\u001b[K     |████████████████████████████████| 122kB 52.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorflow_hub in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 16)) (0.11.0)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 17)) (3.2.5)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 18)) (1.19.5)\n",
            "Collecting pymemcache\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b8/94/16a3ae7ce435c8abb90439baa6ebd465f7c5d202bc6b84d8fd69f1534e3e/pymemcache-3.4.1-py2.py3-none-any.whl (49kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 8.0MB/s \n",
            "\u001b[?25hCollecting mezmorize\n",
            "  Downloading https://files.pythonhosted.org/packages/f1/73/c3153951bf8956c92e0a481daa804d57f13970457c32a6692ca6723a026f/mezmorize-0.28.2-py2.py3-none-any.whl\n",
            "Collecting cached_property\n",
            "  Downloading https://files.pythonhosted.org/packages/48/19/f2090f7dad41e225c7f2326e4cfe6fff49e57dedb5b53636c9551f86b069/cached_property-1.5.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 22)) (2.23.0)\n",
            "Collecting lru-dict\n",
            "  Downloading https://files.pythonhosted.org/packages/68/ea/997af58d4e6da019ad825a412f93081d9df67e9dda11cfb026a3d7cd0b6c/lru-dict-1.1.7.tar.gz\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/2a/dc/97f2b63ef0fa1fd78dcb7195aca577804f6b2b51e712516cc0e902a9a201/python-Levenshtein-0.12.2.tar.gz (50kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 7.7MB/s \n",
            "\u001b[?25hCollecting lemminflect\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4b/67/d04ca98b661d4ad52b9b965c9dabb1f1a2c85541d20f8decb9a9df4e4b32/lemminflect-0.2.2-py3-none-any.whl (769kB)\n",
            "\u001b[K     |████████████████████████████████| 778kB 47.0MB/s \n",
            "\u001b[?25hCollecting language_tool_python\n",
            "  Downloading https://files.pythonhosted.org/packages/37/26/48b22ad565fd372edec3577218fb817e0e6626bf4e658033197470ad92b3/language_tool_python-2.5.3-py3-none-any.whl\n",
            "Requirement already satisfied: editdistance in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 29)) (0.5.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 30)) (3.0.12)\n",
            "Collecting terminaltables\n",
            "  Downloading https://files.pythonhosted.org/packages/9b/c4/4a21174f32f8a7e1104798c445dacdc1d4df86f2f26722767034e4de4bff/terminaltables-3.1.0.tar.gz\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 32)) (4.41.1)\n",
            "Collecting word2number\n",
            "  Downloading https://files.pythonhosted.org/packages/4a/29/a31940c848521f0725f0df6b25dca8917f13a2025b0e8fcbe5d0457e45e6/word2number-1.1.zip\n",
            "Collecting num2words\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/eb/a2/ea800689730732e27711c41beed4b2a129b34974435bdc450377ec407738/num2words-0.5.10-py3-none-any.whl (101kB)\n",
            "\u001b[K     |████████████████████████████████| 102kB 12.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: appdirs>=1.4 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 37)) (1.4.4)\n",
            "Collecting oslo.concurrency>=4.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/30/2d/d9dd1b17bdbcd8f269c025052677b7bc3b54b6f91c3df6ba7732c4152327/oslo.concurrency-4.4.0-py3-none-any.whl (47kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 7.9MB/s \n",
            "\u001b[?25hCollecting pytest>=5.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/76/4d/9c00146923da9f1cabd1878209d71b1380d537ec331a1a613e8f4b9d7985/pytest-6.2.3-py3-none-any.whl (280kB)\n",
            "\u001b[K     |████████████████████████████████| 286kB 58.9MB/s \n",
            "\u001b[?25hCollecting pytorch_pretrained_bert\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 59.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.32.0)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (0.12.0)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (3.12.4)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.12.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.1.0)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (0.3.3)\n",
            "Collecting tensorboard<2.3.0,>=2.2.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1d/74/0a6fcb206dcc72a6da9a62dd81784bfdbff5fedb099982861dc2219014fb/tensorboard-2.2.2-py3-none-any.whl (3.0MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0MB 54.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (0.2.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.1.2)\n",
            "Collecting tensorflow-estimator<2.3.0,>=2.2.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a4/f5/926ae53d6a226ec0fda5208e0e581cffed895ccc89e36ba76a8e60895b78/tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454kB)\n",
            "\u001b[K     |████████████████████████████████| 460kB 43.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (0.36.2)\n",
            "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (2.10.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (3.3.0)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.15.0)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0->-r requirements.txt (line 1)) (1.6.3)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/08/cd/342e584ee544d044fb573ae697404ce22ede086c9e87ce5960772084cad0/sacremoses-0.0.44.tar.gz (862kB)\n",
            "\u001b[K     |████████████████████████████████| 870kB 50.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2->-r requirements.txt (line 2)) (20.9)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==3.0.2->-r requirements.txt (line 2)) (2019.12.20)\n",
            "Collecting tokenizers==0.8.1.rc1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/02/59/68c7e3833f535615fb97d33ffcb7b30bbf62bc7477a9c59cd19ad8535d72/tokenizers-0.8.1rc1-cp37-cp37m-manylinux1_x86_64.whl (3.0MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0MB 49.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from sentence-transformers==0.3.4->-r requirements.txt (line 3)) (1.7.1+cu110)\n",
            "Collecting threadpoolctl>=2.0.0\n",
            "  Downloading https://files.pythonhosted.org/packages/f7/12/ec3f2e203afa394a149911729357aa48affc59c20e2c1c8297a60f33f133/threadpoolctl-2.1.0-py3-none-any.whl\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.24.1->-r requirements.txt (line 6)) (1.0.1)\n",
            "Collecting ftfy\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/78/50/ba5ec9ff8b56e09c0aa8e13d2cc6e24b31bdd23e2bab8f510929bcc4ac48/ftfy-6.0.tar.gz (63kB)\n",
            "\u001b[K     |████████████████████████████████| 71kB 11.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (0.1.2)\n",
            "Collecting janome\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a8/63/98858cbead27df7536c7e300c169da0999e9704d02220dc6700b804eeff0/Janome-0.4.1-py2.py3-none-any.whl (19.7MB)\n",
            "\u001b[K     |████████████████████████████████| 19.7MB 9.1MB/s \n",
            "\u001b[?25hCollecting sqlitedict>=1.6.0\n",
            "  Downloading https://files.pythonhosted.org/packages/5c/2d/b1d99e9ad157dd7de9cd0d36a8a5876b13b55e4b75f7498bc96035fb4e96/sqlitedict-1.7.0.tar.gz\n",
            "Collecting konoha<5.0.0,>=4.0.0\n",
            "  Downloading https://files.pythonhosted.org/packages/02/be/4dd30d56a0a19619deb9bf41ba8202709fa83b1b301b876572cd6dc38117/konoha-4.6.4-py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (3.2.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (4.2.6)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (3.6.0)\n",
            "Collecting bpemb>=0.3.2\n",
            "  Downloading https://files.pythonhosted.org/packages/91/77/3f0f53856e86af32b1d3c86652815277f7b5f880002584eb30db115b6df5/bpemb-0.3.2-py3-none-any.whl\n",
            "Collecting mpld3==0.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/95/a52d3a83d0a29ba0d6898f6727e9858fe7a43f6c2ce81a5fe7e05f0f4912/mpld3-0.3.tar.gz (788kB)\n",
            "\u001b[K     |████████████████████████████████| 798kB 50.3MB/s \n",
            "\u001b[?25hCollecting segtok>=1.5.7\n",
            "  Downloading https://files.pythonhosted.org/packages/41/08/582dab5f4b1d5ca23bc6927b4bb977c8ff7f3a87a3b98844ef833e2f5623/segtok-1.5.10.tar.gz\n",
            "Collecting deprecated>=1.2.4\n",
            "  Downloading https://files.pythonhosted.org/packages/fb/73/994edfcba74443146c84b91921fcc269374354118d4f452fb0c54c1cbb12/Deprecated-1.2.12-py2.py3-none-any.whl\n",
            "Collecting langdetect\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/56/a3/8407c1e62d5980188b4acc45ef3d94b933d14a2ebc9ef3505f22cf772570/langdetect-1.0.8.tar.gz (981kB)\n",
            "\u001b[K     |████████████████████████████████| 983kB 30.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (0.8.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (2.8.1)\n",
            "Requirement already satisfied: gdown in /usr/local/lib/python3.7/dist-packages (from flair==0.6.1.post1->-r requirements.txt (line 7)) (3.6.4)\n",
            "Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb->-r requirements.txt (line 10)) (5.4.8)\n",
            "Collecting subprocess32>=3.5.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/32/c8/564be4d12629b912ea431f1a50eb8b3b9d00f1a0b1ceff17f266be190007/subprocess32-3.5.4.tar.gz (97kB)\n",
            "\u001b[K     |████████████████████████████████| 102kB 13.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: promise<3,>=2.0 in /usr/local/lib/python3.7/dist-packages (from wandb->-r requirements.txt (line 10)) (2.3)\n",
            "Collecting configparser>=3.8.1\n",
            "  Downloading https://files.pythonhosted.org/packages/fd/01/ff260a18caaf4457eb028c96eeb405c4a230ca06c8ec9c1379f813caa52e/configparser-5.0.2-py3-none-any.whl\n",
            "Collecting docker-pycreds>=0.4.0\n",
            "  Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: Click>=7.0 in /usr/local/lib/python3.7/dist-packages (from wandb->-r requirements.txt (line 10)) (7.1.2)\n",
            "Collecting pathtools\n",
            "  Downloading https://files.pythonhosted.org/packages/e7/7f/470d6fcdf23f9f3518f6b0b76be9df16dcc8630ad409947f8be2eb0ed13a/pathtools-0.1.2.tar.gz\n",
            "Collecting GitPython>=1.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/99/98019716955ba243657daedd1de8f3a88ca1f5b75057c38e959db22fb87b/GitPython-3.1.14-py3-none-any.whl (159kB)\n",
            "\u001b[K     |████████████████████████████████| 163kB 55.7MB/s \n",
            "\u001b[?25hCollecting shortuuid>=0.5.0\n",
            "  Downloading https://files.pythonhosted.org/packages/25/a6/2ecc1daa6a304e7f1b216f0896b26156b78e7c38e1211e9b798b4716c53d/shortuuid-1.0.1-py3-none-any.whl\n",
            "Collecting sentry-sdk>=0.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f3/92/5a33be64990ba815364a8f2dd9e6f51de60d23dfddafb4f1fc5577d4dc64/sentry_sdk-1.0.0-py2.py3-none-any.whl (131kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 49.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from wandb->-r requirements.txt (line 10)) (3.13)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->-r requirements.txt (line 11)) (2018.9)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (3.8.1)\n",
            "Requirement already satisfied: huggingface-hub<0.1.0 in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (0.0.8)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (0.3.3)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (2.0.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (0.70.11.1)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.7/dist-packages (from datasets->-r requirements.txt (line 13)) (0.9.0)\n",
            "Requirement already satisfied: tornado in /usr/local/lib/python3.7/dist-packages (from visdom->-r requirements.txt (line 14)) (5.1.1)\n",
            "Requirement already satisfied: pyzmq in /usr/local/lib/python3.7/dist-packages (from visdom->-r requirements.txt (line 14)) (22.0.3)\n",
            "Collecting jsonpatch\n",
            "  Downloading https://files.pythonhosted.org/packages/a3/55/f7c93bae36d869292aedfbcbae8b091386194874f16390d680136edd2b28/jsonpatch-1.32-py2.py3-none-any.whl\n",
            "Collecting torchfile\n",
            "  Downloading https://files.pythonhosted.org/packages/91/af/5b305f86f2d218091af657ddb53f984ecbd9518ca9fe8ef4103a007252c9/torchfile-0.1.0.tar.gz\n",
            "Collecting websocket-client\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/08/33/80e0d4f60e84a1ddd9a03f340be1065a2a363c47ce65c4bd3bae65ce9631/websocket_client-0.58.0-py2.py3-none-any.whl (61kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 9.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from visdom->-r requirements.txt (line 14)) (7.1.2)\n",
            "Requirement already satisfied: werkzeug<=2.0.0,>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from mezmorize->-r requirements.txt (line 20)) (1.0.1)\n",
            "Collecting cachelib<=0.2,>=0.1\n",
            "  Downloading https://files.pythonhosted.org/packages/e6/fc/9c5571cf72ac3ea64ad5cd9d704c1000452cb483a6a3233357d8f3da6991/cachelib-0.1.1-py3-none-any.whl\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->-r requirements.txt (line 22)) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->-r requirements.txt (line 22)) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->-r requirements.txt (line 22)) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->-r requirements.txt (line 22)) (2020.12.5)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from python-Levenshtein->-r requirements.txt (line 24)) (54.2.0)\n",
            "Requirement already satisfied: docopt>=0.6.2 in /usr/local/lib/python3.7/dist-packages (from num2words->-r requirements.txt (line 34)) (0.6.2)\n",
            "Collecting fasteners>=0.7.0\n",
            "  Downloading https://files.pythonhosted.org/packages/78/20/c862d765287e9e8b29f826749ebae8775bdca50b2cb2ca079346d5fbfd76/fasteners-0.16-py2.py3-none-any.whl\n",
            "Collecting pbr!=2.1.0,>=2.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fb/48/69046506f6ac61c1eaa9a0d42d22d54673b69e176d30ca98e3f61513e980/pbr-5.5.1-py2.py3-none-any.whl (106kB)\n",
            "\u001b[K     |████████████████████████████████| 112kB 42.5MB/s \n",
            "\u001b[?25hCollecting oslo.config>=5.2.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/91/4dd50389dea8b9c76812f6f89c20bc35b48818c68a7ce2174ab9fd78bdbe/oslo.config-8.5.0-py3-none-any.whl (127kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 57.2MB/s \n",
            "\u001b[?25hCollecting oslo.utils>=3.33.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/cc/ba/77f27f4b2fecbadbe40c3e367110b781afef85a3b5b576450040dfd1a1d1/oslo.utils-4.8.0-py3-none-any.whl (102kB)\n",
            "\u001b[K     |████████████████████████████████| 102kB 13.5MB/s \n",
            "\u001b[?25hCollecting oslo.i18n>=3.15.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/89/ac/b71a66e54c8fcf22c4205efe2b5f94dbf282c194f9f07dbf0a1ac52d4633/oslo.i18n-5.0.1-py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 8.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.7/dist-packages (from pytest>=5.0->-r requirements.txt (line 39)) (20.3.0)\n",
            "Collecting pluggy<1.0.0a1,>=0.12\n",
            "  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pytest>=5.0->-r requirements.txt (line 39)) (0.10.2)\n",
            "Requirement already satisfied: iniconfig in /usr/local/lib/python3.7/dist-packages (from pytest>=5.0->-r requirements.txt (line 39)) (1.1.1)\n",
            "Requirement already satisfied: py>=1.8.2 in /usr/local/lib/python3.7/dist-packages (from pytest>=5.0->-r requirements.txt (line 39)) (1.10.0)\n",
            "Collecting boto3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fc/79/64c0815cbe8c6abd7fe5525ec37a2689d3cf10e387629ba4a6e44daff6d0/boto3-1.17.49-py2.py3-none-any.whl (131kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 51.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (1.28.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (0.4.3)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (3.3.4)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (1.8.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers==3.0.2->-r requirements.txt (line 2)) (2.4.7)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.2.0->sentence-transformers==0.3.4->-r requirements.txt (line 3)) (3.7.4.3)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from ftfy->flair==0.6.1.post1->-r requirements.txt (line 7)) (0.2.5)\n",
            "Requirement already satisfied: pymongo in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair==0.6.1.post1->-r requirements.txt (line 7)) (3.11.3)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair==0.6.1.post1->-r requirements.txt (line 7)) (0.16.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from hyperopt>=0.1.1->flair==0.6.1.post1->-r requirements.txt (line 7)) (2.5)\n",
            "Collecting overrides<4.0.0,>=3.0.0\n",
            "  Downloading https://files.pythonhosted.org/packages/ff/b1/10f69c00947518e6676bbd43e739733048de64b8dd998e9c2d5a71f44c5d/overrides-3.1.0.tar.gz\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2.3->flair==0.6.1.post1->-r requirements.txt (line 7)) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2.3->flair==0.6.1.post1->-r requirements.txt (line 7)) (1.3.1)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.7/dist-packages (from gensim>=3.4.0->flair==0.6.1.post1->-r requirements.txt (line 7)) (4.2.0)\n",
            "Collecting gitdb<5,>=4.0.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ea/e8/f414d1a4f0bbc668ed441f74f44c116d9816833a48bf81d22b697090dba8/gitdb-4.0.7-py3-none-any.whl (63kB)\n",
            "\u001b[K     |████████████████████████████████| 71kB 10.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets->-r requirements.txt (line 13)) (3.4.1)\n",
            "Collecting jsonpointer>=1.9\n",
            "  Downloading https://files.pythonhosted.org/packages/23/52/05f67532aa922e494c351344e0d9624a01f74f5dd8402fe0d1b563a6e6fc/jsonpointer-2.1-py2.py3-none-any.whl\n",
            "Collecting netaddr>=0.7.18\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ff/cd/9cdfea8fc45c56680b798db6a55fa60a22e2d3d3ccf54fc729d083b50ce4/netaddr-0.8.0-py2.py3-none-any.whl (1.9MB)\n",
            "\u001b[K     |████████████████████████████████| 1.9MB 47.4MB/s \n",
            "\u001b[?25hCollecting debtcollector>=1.2.0\n",
            "  Downloading https://files.pythonhosted.org/packages/8e/50/07a7ccf4dbbe90b58e96f97b747ff98aef9d8c841d2616c48cc05b07db33/debtcollector-2.2.0-py3-none-any.whl\n",
            "Collecting rfc3986>=1.2.0\n",
            "  Downloading https://files.pythonhosted.org/packages/78/be/7b8b99fd74ff5684225f50dd0e865393d2265656ef3b4ba9eaaaffe622b8/rfc3986-1.4.0-py2.py3-none-any.whl\n",
            "Collecting stevedore>=1.20.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/49/b602307aeac3df3384ff1fcd05da9c0376c622a6c48bb5325f28ab165b57/stevedore-3.3.0-py3-none-any.whl (49kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 8.3MB/s \n",
            "\u001b[?25hCollecting iso8601>=0.1.11\n",
            "  Downloading https://files.pythonhosted.org/packages/c5/10/da48dc228b821a64407c2527e1e8ee98917b36e80a181f2ca06ea3cb676b/iso8601-0.1.14-py2.py3-none-any.whl\n",
            "Collecting netifaces>=0.10.4\n",
            "  Downloading https://files.pythonhosted.org/packages/0d/18/fd6e9c71a35b67a73160ec80a49da63d1eed2d2055054cc2995714949132/netifaces-0.10.9.tar.gz\n",
            "Collecting jmespath<1.0.0,>=0.7.1\n",
            "  Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl\n",
            "Collecting s3transfer<0.4.0,>=0.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/98/14/0b4be62b65c52d6d1c442f24e02d2a9889a73d3c352002e14c70f84a679f/s3transfer-0.3.6-py2.py3-none-any.whl (73kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 11.5MB/s \n",
            "\u001b[?25hCollecting botocore<1.21.0,>=1.20.49\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/68/59/6e28ce58206039ad2592992b75ee79a8f9dbc902a9704373ddacc4f96300/botocore-1.20.49-py2.py3-none-any.whl (7.4MB)\n",
            "\u001b[K     |████████████████████████████████| 7.4MB 48.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (4.7.2)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (4.2.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (1.3.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from networkx->hyperopt>=0.1.1->flair==0.6.1.post1->-r requirements.txt (line 7)) (4.4.2)\n",
            "Collecting smmap<5,>=3.0.1\n",
            "  Downloading https://files.pythonhosted.org/packages/68/ee/d540eb5e5996eb81c26ceffac6ee49041d473bc5125f2aa995cf51ec1cf1/smmap-4.0.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->-r requirements.txt (line 1)) (3.1.0)\n",
            "Building wheels for collected packages: sentence-transformers, visdom, lru-dict, python-Levenshtein, terminaltables, word2number, sacremoses, ftfy, sqlitedict, mpld3, segtok, langdetect, subprocess32, pathtools, torchfile, overrides, netifaces\n",
            "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sentence-transformers: filename=sentence_transformers-0.3.4-cp37-none-any.whl size=99829 sha256=facdbf6e5f8a1ad52ec2d883b95a1d5cb1471dc474bebddeaa361ddba7106a5a\n",
            "  Stored in directory: /root/.cache/pip/wheels/39/b3/0a/c25bcdeeb0858f691d377f06d4bbf5e735598fa3a54d01c04f\n",
            "  Building wheel for visdom (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for visdom: filename=visdom-0.1.8.9-cp37-none-any.whl size=655251 sha256=585e99f2de9e9b44d450b38f2839b15b1b46ff291573f0e809dd2918bd46db2a\n",
            "  Stored in directory: /root/.cache/pip/wheels/70/19/a7/6d589ed967f4dfefd33bc166d081257bd4ed0cb618dccfd62a\n",
            "  Building wheel for lru-dict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for lru-dict: filename=lru_dict-1.1.7-cp37-cp37m-linux_x86_64.whl size=28380 sha256=8512f2dbafd0f6931ce90bf7b36fd04be6da8d4f5604ebddc049d3cba2a4693c\n",
            "  Stored in directory: /root/.cache/pip/wheels/ae/51/23/0a416781dead9225c7d66d25b9f223c7e32304e99a0b01d566\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.2-cp37-cp37m-linux_x86_64.whl size=149807 sha256=164aea6d35c0695a3bba5d04d68d3bbcb61c29a83e4fd09438cdded677c9fad4\n",
            "  Stored in directory: /root/.cache/pip/wheels/b3/26/73/4b48503bac73f01cf18e52cd250947049a7f339e940c5df8fc\n",
            "  Building wheel for terminaltables (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for terminaltables: filename=terminaltables-3.1.0-cp37-none-any.whl size=15356 sha256=4a678f319d0248bcfe387d115f64a21050430334f2aa6133ea5189fe0c00b188\n",
            "  Stored in directory: /root/.cache/pip/wheels/30/6b/50/6c75775b681fb36cdfac7f19799888ef9d8813aff9e379663e\n",
            "  Building wheel for word2number (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for word2number: filename=word2number-1.1-cp37-none-any.whl size=5589 sha256=7ed7c059ac1095ab70ad1cf8907edf30a26ce38dd45e13a21ff4ce92e1c44852\n",
            "  Stored in directory: /root/.cache/pip/wheels/46/2f/53/5f5c1d275492f2fce1cdab9a9bb12d49286dead829a4078e0e\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.44-cp37-none-any.whl size=886084 sha256=478135448291701d88fecb8de4622d6d52c1b5fc9eac17f6c3ffdd93509040f9\n",
            "  Stored in directory: /root/.cache/pip/wheels/3e/fb/c0/13ab4d63d537658f448366744654323077c4d90069b6512f3c\n",
            "  Building wheel for ftfy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ftfy: filename=ftfy-6.0-cp37-none-any.whl size=41622 sha256=2e11870432e02026fdb563a776ba77b080cb41f4a7abb248e910ce25bb8db234\n",
            "  Stored in directory: /root/.cache/pip/wheels/22/8b/08/7d1c17849e10371206a262304973b5a9f45e8b9d0a2179f465\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sqlitedict: filename=sqlitedict-1.7.0-cp37-none-any.whl size=14376 sha256=88a082d860346dca4bf746ca08164a42e6dd23c064aab3e24ec5129e14979152\n",
            "  Stored in directory: /root/.cache/pip/wheels/cf/c6/4f/2c64a43f041415eb8b8740bd80e15e92f0d46c5e464d8e4b9b\n",
            "  Building wheel for mpld3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mpld3: filename=mpld3-0.3-cp37-none-any.whl size=116679 sha256=f177d1c2add7ab7fabf8b24c5484f92b10fe5aea129dd82aaf5d669b42aacb7f\n",
            "  Stored in directory: /root/.cache/pip/wheels/c0/47/fb/8a64f89aecfe0059830479308ad42d62e898a3e3cefdf6ba28\n",
            "  Building wheel for segtok (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for segtok: filename=segtok-1.5.10-cp37-none-any.whl size=25019 sha256=0133bef9be3f24091c665f810e39f9359cdcb2f1faae38400051f52bc3aa6420\n",
            "  Stored in directory: /root/.cache/pip/wheels/b4/39/f6/9ca1c5cabde964d728023b5751c3a206a5c8cc40252321fb6b\n",
            "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.8-cp37-none-any.whl size=993193 sha256=aa51288e257e9daac3744a1b9efed7f4433ca074823fde52ebfbfd674d5c2a3d\n",
            "  Stored in directory: /root/.cache/pip/wheels/8d/b3/aa/6d99de9f3841d7d3d40a60ea06e6d669e8e5012e6c8b947a57\n",
            "  Building wheel for subprocess32 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for subprocess32: filename=subprocess32-3.5.4-cp37-none-any.whl size=6489 sha256=3100d3ef379f47d15305d2d4ecf05102c0e2ce836eb86004eba5d03fb2cf67d2\n",
            "  Stored in directory: /root/.cache/pip/wheels/68/39/1a/5e402bdfdf004af1786c8b853fd92f8c4a04f22aad179654d1\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-cp37-none-any.whl size=8786 sha256=1393968b279b1cb78c753f8af6b2a00f334f921e920727708be986472399e356\n",
            "  Stored in directory: /root/.cache/pip/wheels/0b/04/79/c3b0c3a0266a3cb4376da31e5bfe8bba0c489246968a68e843\n",
            "  Building wheel for torchfile (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for torchfile: filename=torchfile-0.1.0-cp37-none-any.whl size=5713 sha256=2dcd3348c63322d7e7672fab118f9dc0c42cc886f532809feb985d86eb0ce51f\n",
            "  Stored in directory: /root/.cache/pip/wheels/b1/c3/d6/9a1cc8f3a99a0fc1124cae20153f36af59a6e683daca0a0814\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for overrides: filename=overrides-3.1.0-cp37-none-any.whl size=10174 sha256=c95d11fdd18c41888216ab91ce42eef877ad39e5b781210cfac880270738fd90\n",
            "  Stored in directory: /root/.cache/pip/wheels/5c/24/13/6ef8600e6f147c95e595f1289a86a3cc82ed65df57582c65a9\n",
            "  Building wheel for netifaces (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for netifaces: filename=netifaces-0.10.9-cp37-cp37m-linux_x86_64.whl size=37423 sha256=2d54ef03a1ce94cd672662832eb64051424645257818e8d964fd28e13a4791f1\n",
            "  Stored in directory: /root/.cache/pip/wheels/23/8f/f3/7054578f04c904f70757c5c85a6e2823baa69d42365526e93d\n",
            "Successfully built sentence-transformers visdom lru-dict python-Levenshtein terminaltables word2number sacremoses ftfy sqlitedict mpld3 segtok langdetect subprocess32 pathtools torchfile overrides netifaces\n",
            "\u001b[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: konoha 4.6.4 has requirement requests<3.0.0,>=2.25.1, but you'll have requests 2.23.0 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: oslo-config 8.5.0 has requirement PyYAML>=5.1, but you'll have pyyaml 3.13 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: botocore 1.20.49 has requirement urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.3 which is incompatible.\u001b[0m\n",
            "Installing collected packages: tensorboard, tensorflow-estimator, tensorflow, sacremoses, sentencepiece, tokenizers, transformers, threadpoolctl, scikit-learn, sentence-transformers, ftfy, janome, sqlitedict, overrides, konoha, bpemb, mpld3, segtok, deprecated, langdetect, flair, pyarrow, subprocess32, configparser, docker-pycreds, pathtools, smmap, gitdb, GitPython, shortuuid, sentry-sdk, wandb, bert-score, jsonpointer, jsonpatch, torchfile, websocket-client, visdom, tensorboardX, pymemcache, cachelib, mezmorize, cached-property, lru-dict, python-Levenshtein, lemminflect, language-tool-python, terminaltables, word2number, num2words, fasteners, pbr, netaddr, debtcollector, rfc3986, stevedore, oslo.i18n, oslo.config, iso8601, netifaces, oslo.utils, oslo.concurrency, pluggy, pytest, jmespath, botocore, s3transfer, boto3, pytorch-pretrained-bert\n",
            "  Found existing installation: tensorboard 2.4.1\n",
            "    Uninstalling tensorboard-2.4.1:\n",
            "      Successfully uninstalled tensorboard-2.4.1\n",
            "  Found existing installation: tensorflow-estimator 2.4.0\n",
            "    Uninstalling tensorflow-estimator-2.4.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.4.0\n",
            "  Found existing installation: tensorflow 2.4.1\n",
            "    Uninstalling tensorflow-2.4.1:\n",
            "      Successfully uninstalled tensorflow-2.4.1\n",
            "  Found existing installation: scikit-learn 0.22.2.post1\n",
            "    Uninstalling scikit-learn-0.22.2.post1:\n",
            "      Successfully uninstalled scikit-learn-0.22.2.post1\n",
            "  Found existing installation: pyarrow 3.0.0\n",
            "    Uninstalling pyarrow-3.0.0:\n",
            "      Successfully uninstalled pyarrow-3.0.0\n",
            "  Found existing installation: pluggy 0.7.1\n",
            "    Uninstalling pluggy-0.7.1:\n",
            "      Successfully uninstalled pluggy-0.7.1\n",
            "  Found existing installation: pytest 3.6.4\n",
            "    Uninstalling pytest-3.6.4:\n",
            "      Successfully uninstalled pytest-3.6.4\n",
            "Successfully installed GitPython-3.1.14 bert-score-0.3.8 boto3-1.17.49 botocore-1.20.49 bpemb-0.3.2 cached-property-1.5.2 cachelib-0.1.1 configparser-5.0.2 debtcollector-2.2.0 deprecated-1.2.12 docker-pycreds-0.4.0 fasteners-0.16 flair-0.6.1.post1 ftfy-6.0 gitdb-4.0.7 iso8601-0.1.14 janome-0.4.1 jmespath-0.10.0 jsonpatch-1.32 jsonpointer-2.1 konoha-4.6.4 langdetect-1.0.8 language-tool-python-2.5.3 lemminflect-0.2.2 lru-dict-1.1.7 mezmorize-0.28.2 mpld3-0.3 netaddr-0.8.0 netifaces-0.10.9 num2words-0.5.10 oslo.concurrency-4.4.0 oslo.config-8.5.0 oslo.i18n-5.0.1 oslo.utils-4.8.0 overrides-3.1.0 pathtools-0.1.2 pbr-5.5.1 pluggy-0.13.1 pyarrow-0.17.1 pymemcache-3.4.1 pytest-6.2.3 python-Levenshtein-0.12.2 pytorch-pretrained-bert-0.6.2 rfc3986-1.4.0 s3transfer-0.3.6 sacremoses-0.0.44 scikit-learn-0.24.1 segtok-1.5.10 sentence-transformers-0.3.4 sentencepiece-0.1.91 sentry-sdk-1.0.0 shortuuid-1.0.1 smmap-4.0.0 sqlitedict-1.7.0 stevedore-3.3.0 subprocess32-3.5.4 tensorboard-2.2.2 tensorboardX-2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0 terminaltables-3.1.0 threadpoolctl-2.1.0 tokenizers-0.8.1rc1 torchfile-0.1.0 transformers-3.0.2 visdom-0.1.8.9 wandb-0.10.25 websocket-client-0.58.0 word2number-1.1\n",
            "Collecting lm-scorer==0.4.2\n",
            "  Downloading https://files.pythonhosted.org/packages/c8/89/d86ee877bfa51104b338a67413c76b6fde50a76c7b7e0c55c546effe97e9/lm_scorer-0.4.2-py3-none-any.whl\n",
            "Installing collected packages: lm-scorer\n",
            "Successfully installed lm-scorer-0.4.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-giWq1YMXyyn"
      },
      "source": [
        "# 3. Perform the Second-Order Attack\n",
        "Attack a pre-trained model `lstm-sst2` in [TextAttack Model Zoo](https://github.com/chong-z/TextAttack/blob/d6ebeeb1afae215d7de5f04c3aac743bbeaf54db/textattack/models/README.md):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qsTn8Ke3W_Bz",
        "outputId": "094b2af5-eb20-4df1-e071-94b2cecef23a"
      },
      "source": [
        "!./patched_textattack attack --attack-from-file=biasattack.py:SOBeamAttack \\\n",
        "  --dataset-from-nlp=glue:sst2:validation --num-examples=10 --shuffle=False \\\n",
        "  --model=lstm-sst2"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\n",
            "2021-04-11 03:49:21; \u001b[34;1mtextattack\u001b[0m: Downloading https://textattack.s3.amazonaws.com/models/classification/lstm/sst2.\n",
            "100% 297M/297M [00:06<00:00, 49.2MB/s]\n",
            "2021-04-11 03:49:27; \u001b[34;1mtextattack\u001b[0m: Unzipping file /root/.cache/textattack/tmp1djwcqd_.zip to /root/.cache/textattack/models/classification/lstm/sst2.\n",
            "2021-04-11 03:49:30; \u001b[34;1mtextattack\u001b[0m: Successfully saved models/classification/lstm/sst2 to cache.\n",
            "2021-04-11 03:49:30; \u001b[34;1mtextattack\u001b[0m: Loading pre-trained TextAttack LSTM: \u001b[94mlstm-sst2\u001b[0m\n",
            "2021-04-11 03:49:30; \u001b[34;1mtextattack\u001b[0m: Downloading https://textattack.s3.amazonaws.com/word_embeddings/glove200.\n",
            "100% 389M/389M [00:06<00:00, 61.0MB/s]\n",
            "2021-04-11 03:49:37; \u001b[34;1mtextattack\u001b[0m: Unzipping file /root/.cache/textattack/tmp6fmon5_9.zip to /root/.cache/textattack/word_embeddings/glove200.\n",
            "2021-04-11 03:49:43; \u001b[34;1mtextattack\u001b[0m: Successfully saved word_embeddings/glove200 to cache.\n",
            "2021-04-11 03:50:01; \u001b[34;1mtextattack\u001b[0m: Loading module from `\u001b[94mbiasattack.py\u001b[0m`.\n",
            "Generating counterfitted anchor words to .cache/counterfitted_anchord_words_model_lstm-sst2.json\n",
            "100% 49966/49966 [00:00<00:00, 1230225.44it/s]\n",
            "100% 49966/49966 [00:00<00:00, 699314.25it/s]\n",
            "Loaded 109076 biaswords from utils.\n",
            " Samples: [('unconvincing', 'persuasive'), ('persuasive', 'unconvincing'), ('utmost', 'abject'), ('abject', 'utmost'), ('perseverance', 'stubbornness'), ('stubbornness', 'perseverance'), ('stubbornness', 'persevering'), ('persevering', 'stubbornness'), ('entertained', 'sidetracked'), ('sidetracked', 'entertained')]...[('ohhhhh', 'ahhhhhh'), ('sodding', 'frakkin'), ('becouse', 'actualy'), ('beatiful', 'wonderfull'), ('ahhhhhh', 'ohhhhh'), ('frakkin', 'sodding'), ('werent', 'actualy'), ('totaly', 'completly'), ('totaly', 'actualy'), ('totaly', 'becouse')]\n",
            "Downloading: 100% 442/442 [00:00<00:00, 629kB/s]\n",
            "Downloading: 100% 232k/232k [00:00<00:00, 23.0MB/s]\n",
            "Downloading: 100% 268M/268M [00:04<00:00, 63.0MB/s]\n",
            "Downloading: 1.75kB [00:00, 1.76MB/s]     \n",
            "Downloading: 1.90kB [00:00, 2.55MB/s]     \n",
            "Loaded 26045 blacklist_words from ['https://raw.githubusercontent.com/uclanlp/gn_glove/master/wordlist/male_word_file.txt', 'https://raw.githubusercontent.com/uclanlp/gn_glove/master/wordlist/female_word_file.txt'].\n",
            " Samples: ['insanely', 'lopped', 'rust', 'spew', 'cursing', 'bracelet', 'molester', 'offered', 'ridding', 'outsourced']...['grandmother', 'mare', 'maternity', 'hens', 'uterus', 'nuns', 'maidservants', \"seamstress'\", 'busgirl', 'heroines']\n",
            "gpu_memory_MB = 15109\n",
            "Attack(\n",
            "  (search_method): BeamSearchPlus(\n",
            "    (beam_width):  20\n",
            "    (beam_sampling_method):  max\n",
            "    (transformations_sampling_ratio):  1\n",
            "    (search_depth):  6\n",
            "  )\n",
            "  (goal_function):  BiasGoalFunction(\n",
            "    (biaswords_list_len):  109076\n",
            "    (active_biaswords_logit_threshold):  1.5\n",
            "    (biasthreshold):  0.5\n",
            "    (diffthreshold):  0.03\n",
            "    (stepweight):  0.1\n",
            "    (skipthreshold):  0.1\n",
            "    (score_mode):  max\n",
            "    (lm_scorer):  distilbert-base-uncased\n",
            "    (model_batch_size):  512\n",
            "    (query_budget):  50000\n",
            "  )\n",
            "  (transformation):  CompositeTransformation(\n",
            "    (0): InitialBiasWord(\n",
            "        (biaswords_flatten_len):  25601\n",
            "      )\n",
            "    (1): WordSwapMaskedLMPlus(\n",
            "        (max_masks):  1\n",
            "        (max_trials):  -1\n",
            "        (logit_threshold):  3\n",
            "        (force_rte_format):  False\n",
            "        (method):  bae_plus\n",
            "        (masked_lm_name):  distilbert-base-uncased\n",
            "        (max_length):  128\n",
            "        (max_candidates):  20\n",
            "      )\n",
            "    )\n",
            "  (constraints): \n",
            "    (0): ActiveAnchorWordsModification(\n",
            "        (compare_against_original):  False\n",
            "      )\n",
            "  (is_black_box):  True\n",
            ") \n",
            "\n",
            "2021-04-11 03:50:18; \u001b[34;1mtextattack\u001b[0m: Load time: 57.079678535461426s\n",
            "Reusing dataset glue (/root/.cache/huggingface/datasets/glue/sst2/1.0.0/7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4)\n",
            "2021-04-11 03:50:19; \u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94mglue\u001b[0m, subset \u001b[94msst2\u001b[0m, split \u001b[94mvalidation\u001b[0m.\n",
            "  0% 0/10 [00:00<?, ?it/s]best_score: 99.93251037597656 raw_output: tensor([0.9690, 0.0354]) text: it 's a charming and often frustrating journey . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: True score: 99.93251037597656 raw_output: tensor([0.9690, 0.0354]) text: 'it 's a charming and often frustrating journey . '\n",
            "  beam_succeed: True score: 99.85243225097656 raw_output: tensor([0.9963, 0.1340]) text: 'it 's a charming and often confusing journey . '\n",
            "  beam_succeed: True score: 99.78048706054688 raw_output: tensor([0.9946, 0.1927]) text: 'it 's a charming and often painful journey . '\n",
            "  beam_succeed: False score: -0.9885784983634949 raw_output: tensor([0.9998, 0.6278]) text: 'it 's a charming and often tragic journey . '\n",
            "  beam_succeed: False score: -1.4234060049057007 raw_output: tensor([1.0000, 0.7591]) text: 'it 's a charming and often affecting word . '\n",
            "\n",
            "--------------------------------------------- Result 1 ---------------------------------------------\n",
            "\n",
            "it 's a \u001b[92mcharming\u001b[0m/\u001b[35mlovable\u001b[0m and often affecting journey .  (diff: 0.002566, preds: \u001b[92m0.9999988\u001b[0m/\u001b[35m0.99743265\u001b[0m, lm_scores: -5.702733/-6.777777, truth:1)\n",
            " --> \n",
            "it 's a \u001b[92mcharming\u001b[0m/\u001b[35mlovable\u001b[0m and often frustrating journey .  (diff: 0.933645, preds: \u001b[92m0.9690206\u001b[0m/\u001b[35m0.035376053\u001b[0m, lm_scores: -4.889042/-5.596186)\n",
            "\n",
            "\n",
            "it 's a charming and often \u001b[91maffecting\u001b[0m journey . \n",
            "\n",
            "it 's a charming and often \u001b[94mfrustrating\u001b[0m journey . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 1 / 0 / 1:  10% 1/10 [00:00<00:07,  1.27it/s]\n",
            "--------------------------------------------- Result 2 ---------------------------------------------\n",
            "\n",
            "unflinchingly \u001b[35mbleak\u001b[0m/\u001b[92msomber\u001b[0m and desperate  (diff: 0.550431, preds: \u001b[35m0.34175178\u001b[0m/\u001b[92m0.89218235\u001b[0m, lm_scores: -5.570937/-6.847361, truth:0)\n",
            " --> \n",
            "unflinchingly \u001b[35mbleak\u001b[0m/\u001b[92msomber\u001b[0m and desperate  (diff: 0.550431, preds: \u001b[35m0.34175178\u001b[0m/\u001b[92m0.89218235\u001b[0m, lm_scores: -5.570937/-6.847361, truth:0)\n",
            "\n",
            "\n",
            "unflinchingly bleak and desperate \n",
            "\n",
            "unflinchingly bleak and desperate \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 2 / 0 / 2:  20% 2/10 [00:01<00:04,  1.84it/s]best_score: -2.1716201305389404 raw_output: tensor([0.8946, 0.8726]) text: allows us to hope that nolan is poised to embark a major career as a commercial yet unknown filmmaker . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: False score: -2.1716201305389404 raw_output: tensor([0.8946, 0.8726]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial yet unknown filmmaker . '\n",
            "  beam_succeed: False score: -2.1785998344421387 raw_output: tensor([0.8752, 0.9074]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial yet freelance filmmaker . '\n",
            "  beam_succeed: False score: -2.1963839530944824 raw_output: tensor([0.8809, 0.9338]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial yet amateur filmmaker . '\n",
            "  beam_succeed: False score: -2.3525290489196777 raw_output: tensor([0.9000, 0.9509]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial yet experimental filmmaker . '\n",
            "  beam_succeed: False score: -2.531729221343994 raw_output: tensor([0.9472, 0.9160]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial yet aspiring filmmaker . '\n",
            "best_score: 99.35150909423828 raw_output: tensor([0.3670, 0.8260]) text: allows us to hope that nolan is poised to start a major career as a commercial yet experimental filmmaker . \n",
            "  depth = 2, len(beam) = 20\n",
            "  beam_succeed: True score: 99.35150909423828 raw_output: tensor([0.3670, 0.8260]) text: 'allows us to hope that nolan is poised to start a major career as a commercial yet experimental filmmaker . '\n",
            "  beam_succeed: True score: 99.33177947998047 raw_output: tensor([0.2298, 0.6655]) text: 'allows us to hope that nolan is poised to start a major career as a commercial yet amateur filmmaker . '\n",
            "  beam_succeed: True score: 99.3236083984375 raw_output: tensor([0.2264, 0.6573]) text: 'allows us to hope that nolan is poised to start a major career as a commercial yet freelance filmmaker . '\n",
            "  beam_succeed: True score: 99.29287719726562 raw_output: tensor([0.8206, 0.3992]) text: 'allows us to hope that nolan is poised to embark a major career as a commercial or aspiring filmmaker . '\n",
            "  beam_succeed: True score: 99.2821273803711 raw_output: tensor([0.8450, 0.4227]) text: 'allows us to hope that nolan is destined to embark a major career as a commercial yet aspiring filmmaker . '\n",
            "\n",
            "--------------------------------------------- Result 3 ---------------------------------------------\n",
            "\n",
            "allows us to hope that nolan is poised to embark a major \u001b[92mcareer\u001b[0m/\u001b[35moccupations\u001b[0m as a commercial yet inventive filmmaker .  (diff: 0.000020, preds: \u001b[92m0.9995503\u001b[0m/\u001b[35m0.99952984\u001b[0m, lm_scores: -3.527787/-4.457861, truth:1)\n",
            " --> \n",
            "allows us to hope that nolan is poised to start a major \u001b[35mcareer\u001b[0m/\u001b[92moccupations\u001b[0m as a commercial yet experimental filmmaker .  (diff: 0.458972, preds: \u001b[35m0.36703235\u001b[0m/\u001b[92m0.8260042\u001b[0m, lm_scores: -2.868394/-3.992938)\n",
            "\n",
            "\n",
            "allows us to hope that nolan is poised to \u001b[91membark\u001b[0m a major career as a commercial yet \u001b[91minventive\u001b[0m filmmaker . \n",
            "\n",
            "allows us to hope that nolan is poised to \u001b[94mstart\u001b[0m a major career as a commercial yet \u001b[94mexperimental\u001b[0m filmmaker . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 3 / 0 / 3:  30% 3/10 [00:06<00:14,  2.03s/it]best_score: -1.2454756498336792 raw_output: tensor([0.8677, 0.6683]) text: the acting , costumes , music , cinematography and sound are all chosen given the production 's austere locales . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: False score: -1.2454756498336792 raw_output: tensor([0.8677, 0.6683]) text: 'the acting , costumes , music , cinematography and sound are all chosen given the production 's austere locales . '\n",
            "  beam_succeed: False score: -1.3685411214828491 raw_output: tensor([0.3251, 0.2172]) text: 'the acting , costumes , music , cinematography and sound are virtually astounding given the production 's austere locales . '\n",
            "  beam_succeed: False score: -1.502795934677124 raw_output: tensor([0.9425, 0.7639]) text: 'the acting , costumes , music , cinematography and sound are all sourced given the production 's austere locales . '\n",
            "  beam_succeed: False score: -1.6282755136489868 raw_output: tensor([0.9654, 0.7967]) text: 'the acting , costumes , music , cinematography and sound are all adapted given the production 's austere locales . '\n",
            "  beam_succeed: False score: -1.6327115297317505 raw_output: tensor([0.9653, 0.7976]) text: 'the acting , costumes , music , cinematography and sound are all noteworthy given the production 's austere locales . '\n",
            "best_score: 99.43141174316406 raw_output: tensor([0.6904, 0.1798]) text: the acting , costumes , music , cinematography and sound are loosely adapted given the production 's austere locales . \n",
            "  depth = 2, len(beam) = 20\n",
            "  beam_succeed: True score: 99.43141174316406 raw_output: tensor([0.6904, 0.1798]) text: 'the acting , costumes , music , cinematography and sound are loosely adapted given the production 's austere locales . '\n",
            "  beam_succeed: True score: 99.39448547363281 raw_output: tensor([0.7766, 0.2972]) text: 'the acting , costumes , music , cinematography and sound are largely adapted given the production 's austere locales . '\n",
            "  beam_succeed: True score: 99.367431640625 raw_output: tensor([0.6840, 0.2234]) text: 'the acting , costumes , music , cinematography and sound are all adapted given the production or austere locales . '\n",
            "  beam_succeed: True score: 99.36519622802734 raw_output: tensor([0.8146, 0.3493]) text: 'the acting , costumes , music , props and sound are all sourced given the production 's austere locales . '\n",
            "  beam_succeed: True score: 99.3533706665039 raw_output: tensor([0.7066, 0.2587]) text: 'the acting , costumes , music , cinematography and sound are largely unknown given the production 's austere locales . '\n",
            "\n",
            "--------------------------------------------- Result 4 ---------------------------------------------\n",
            "\n",
            "the \u001b[92macting\u001b[0m/\u001b[35mbehaving\u001b[0m , costumes , music , cinematography and sound are all astounding given the production 's austere locales .  (diff: 0.018201, preds: \u001b[92m0.98971903\u001b[0m/\u001b[35m0.9715176\u001b[0m, lm_scores: -5.667758/-6.759575, truth:1)\n",
            " --> \n",
            "the \u001b[92macting\u001b[0m/\u001b[35mbehaving\u001b[0m , costumes , music , cinematography and sound are loosely adapted given the production 's austere locales .  (diff: 0.510676, preds: \u001b[92m0.69044244\u001b[0m/\u001b[35m0.17976637\u001b[0m, lm_scores: -5.169281/-6.187751)\n",
            "\n",
            "\n",
            "the acting , costumes , music , cinematography and sound are \u001b[91mall\u001b[0m \u001b[91mastounding\u001b[0m given the production 's austere locales . \n",
            "\n",
            "the acting , costumes , music , cinematography and sound are \u001b[94mloosely\u001b[0m \u001b[94madapted\u001b[0m given the production 's austere locales . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 4 / 0 / 4:  40% 4/10 [00:11<00:16,  2.75s/it]\n",
            "--------------------------------------------- Result 5 ---------------------------------------------\n",
            "\n",
            "it 's slow -- very , very slow .  (diff: 0.000000, preds: \u001b[35m\u001b[92m0.020822672\u001b[0m\u001b[0m/0.020822672, lm_scores: -3.476258/-3.476258, truth:0)\n",
            " --> \u001b[37m[SKIPPED]\u001b[0m\n",
            "\n",
            "it 's slow -- very , very slow . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 4 / 0 / 5:  50% 5/10 [00:11<00:11,  2.27s/it]best_score: 99.42709350585938 raw_output: tensor([0.8839, 0.3621]) text: although laced with humor and a few fanciful touches , the film is a particularly serious look at young women . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: True score: 99.42709350585938 raw_output: tensor([0.8839, 0.3621]) text: 'although laced with humor and a few fanciful touches , the film is a particularly serious look at young women . '\n",
            "  beam_succeed: True score: 99.34227752685547 raw_output: tensor([0.8986, 0.4235]) text: 'although laced with humor and a few fanciful touches , the film is a mostly serious look at young women . '\n",
            "  beam_succeed: True score: 99.3410415649414 raw_output: tensor([0.8163, 0.3662]) text: 'although laced with humor and a few fanciful touches , the film is a decidedly serious look at young women . '\n",
            "  beam_succeed: True score: 99.24148559570312 raw_output: tensor([0.6596, 0.2899]) text: 'although laced with humor and a few fanciful touches , the film is a generally serious look at young women . '\n",
            "  beam_succeed: True score: 99.2240982055664 raw_output: tensor([0.7749, 0.4060]) text: 'although laced with humor and a few fanciful touches , the film is a less serious look at young women . '\n",
            "\n",
            "--------------------------------------------- Result 6 ---------------------------------------------\n",
            "\n",
            "although laced with humor and a few fanciful touches , the film is a refreshingly serious \u001b[92mlook\u001b[0m/\u001b[35mglance\u001b[0m at young women .  (diff: 0.001774, preds: \u001b[92m0.999814\u001b[0m/\u001b[35m0.9980404\u001b[0m, lm_scores: -3.671559/-3.991913, truth:1)\n",
            " --> \n",
            "although laced with humor and a few fanciful touches , the film is a particularly serious \u001b[92mlook\u001b[0m/\u001b[35mglance\u001b[0m at young women .  (diff: 0.521858, preds: \u001b[92m0.8839249\u001b[0m/\u001b[35m0.3620665\u001b[0m, lm_scores: -2.902028/-3.189692)\n",
            "\n",
            "\n",
            "although laced with humor and a few fanciful touches , the film is a \u001b[91mrefreshingly\u001b[0m serious look at young women . \n",
            "\n",
            "although laced with humor and a few fanciful touches , the film is a \u001b[94mparticularly\u001b[0m serious look at young women . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 5 / 0 / 6:  60% 6/10 [00:12<00:08,  2.15s/it]best_score: 99.82077026367188 raw_output: tensor([0.1171, 0.9468]) text: a sometimes edited film . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: True score: 99.82077026367188 raw_output: tensor([0.1171, 0.9468]) text: 'a sometimes edited film . '\n",
            "  beam_succeed: True score: 99.80204772949219 raw_output: tensor([0.1182, 0.9303]) text: 'a sometimes satirical film . '\n",
            "  beam_succeed: True score: 99.77606964111328 raw_output: tensor([0.1349, 0.9240]) text: 'a sometimes controversial film . '\n",
            "  beam_succeed: True score: 99.74348449707031 raw_output: tensor([0.0085, 0.7804]) text: 'a sometimes pornographic film . '\n",
            "  beam_succeed: True score: 99.56480407714844 raw_output: tensor([0.1243, 0.7390]) text: 'a sometimes disturbing film . '\n",
            "\n",
            "--------------------------------------------- Result 7 ---------------------------------------------\n",
            "\n",
            "a \u001b[35msometimes\u001b[0m/\u001b[92msometime\u001b[0m tedious film .  (diff: 0.001001, preds: \u001b[35m5.883628e-05\u001b[0m/\u001b[92m0.0010600894\u001b[0m, lm_scores: -7.006533/-8.209057, truth:0)\n",
            " --> \n",
            "a \u001b[35msometimes\u001b[0m/\u001b[92msometime\u001b[0m edited film .  (diff: 0.829683, preds: \u001b[35m0.117140725\u001b[0m/\u001b[92m0.94682413\u001b[0m, lm_scores: -6.373517/-8.426194)\n",
            "\n",
            "\n",
            "a sometimes \u001b[91mtedious\u001b[0m film . \n",
            "\n",
            "a sometimes \u001b[94medited\u001b[0m film . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 6 / 0 / 7:  70% 7/10 [00:13<00:05,  1.89s/it]best_score: 99.59835052490234 raw_output: tensor([0.8974, 0.2543]) text: or doing last year 's taxes with your folks . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: True score: 99.59835052490234 raw_output: tensor([0.8974, 0.2543]) text: 'or doing last year 's taxes with your folks . '\n",
            "  beam_succeed: True score: 99.45821380615234 raw_output: tensor([0.8780, 0.3374]) text: 'or doing last year 's taxes with your friends . '\n",
            "  beam_succeed: True score: 99.43672180175781 raw_output: tensor([0.7762, 0.2665]) text: 'or doing last year 's taxes with your earnings . '\n",
            "  beam_succeed: True score: 99.42059326171875 raw_output: tensor([0.6861, 0.1835]) text: 'or doing last year 's taxes with your parents . '\n",
            "  beam_succeed: True score: 99.41078186035156 raw_output: tensor([0.6794, 0.1834]) text: 'or doing last year 's taxes with your kids . '\n",
            "\n",
            "--------------------------------------------- Result 8 ---------------------------------------------\n",
            "\n",
            "or doing last year 's \u001b[92mfee\u001b[0m/\u001b[35mtaxes\u001b[0m with your ex-wife .  (diff: 0.107292, preds: \u001b[92m0.1331513\u001b[0m/\u001b[35m0.025859617\u001b[0m, lm_scores: -6.619829/-6.360718, truth:0)\n",
            " --> \n",
            "or doing last year 's \u001b[92mfee\u001b[0m/\u001b[35mtaxes\u001b[0m with your folks .  (diff: 0.643129, preds: \u001b[92m0.89740866\u001b[0m/\u001b[35m0.2542798\u001b[0m, lm_scores: -6.286301/-5.669292)\n",
            "\n",
            "\n",
            "or doing last year 's taxes with your \u001b[91mex-wife\u001b[0m . \n",
            "\n",
            "or doing last year 's taxes with your \u001b[94mfolks\u001b[0m . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 7 / 0 / 8:  80% 8/10 [00:13<00:03,  1.75s/it]best_score: -2.383617401123047 raw_output: tensor([0.9915, 0.9070]) text: you do n't have to know about music to appreciate the film 's easygoing mixture of comedy and romance . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: False score: -2.383617401123047 raw_output: tensor([0.9915, 0.9070]) text: 'you do n't have to know about music to appreciate the film 's easygoing mixture of comedy and romance . '\n",
            "  beam_succeed: False score: -2.5578012466430664 raw_output: tensor([0.9991, 0.9225]) text: 'you do n't have to know about music to appreciate the film 's peculiar blend of comedy and romance . '\n",
            "  beam_succeed: False score: -2.838573455810547 raw_output: tensor([0.9975, 0.9413]) text: 'you do n't have to know about music to appreciate the movie 's easygoing blend of comedy and romance . '\n",
            "  beam_succeed: False score: -2.9092302322387695 raw_output: tensor([0.9979, 0.9454]) text: 'you do n't have to know about music to justify the film 's easygoing blend of comedy and romance . '\n",
            "  beam_succeed: False score: -3.211038589477539 raw_output: tensor([0.9984, 0.9596]) text: 'you do n't have to know about music to appreciate the genre 's easygoing blend of comedy and romance . '\n",
            "best_score: 99.8505859375 raw_output: tensor([0.9306, 0.0746]) text: you do n't have to know about music to enhance the film 's usual blend of comedy and romance . \n",
            "  depth = 2, len(beam) = 20\n",
            "  beam_succeed: True score: 99.8505859375 raw_output: tensor([0.9306, 0.0746]) text: 'you do n't have to know about music to enhance the film 's usual blend of comedy and romance . '\n",
            "  beam_succeed: True score: 99.79385375976562 raw_output: tensor([0.9962, 0.1832]) text: 'you do n't have to know about music to enhance the film 's typical blend of comedy and romance . '\n",
            "  beam_succeed: True score: 99.66349029541016 raw_output: tensor([0.9047, 0.2105]) text: 'you do n't have to know about music to sustain the film 's usual blend of comedy and romance . '\n",
            "  beam_succeed: True score: 99.65692138671875 raw_output: tensor([0.8540, 0.1691]) text: 'you do n't have to know about music to appreciate the genre or easygoing blend of comedy and romance . '\n",
            "  beam_succeed: True score: 99.6502914428711 raw_output: tensor([0.9919, 0.2894]) text: 'you do n't have to know about music to sustain the film 's typical blend of comedy and romance . '\n",
            "\n",
            "--------------------------------------------- Result 9 ---------------------------------------------\n",
            "\n",
            "you do n't have to know about music to appreciate the film 's easygoing blend of \u001b[92mcomedy\u001b[0m/\u001b[35mtravesty\u001b[0m and romance .  (diff: 0.004412, preds: \u001b[92m0.9997795\u001b[0m/\u001b[35m0.9953673\u001b[0m, lm_scores: -4.406572/-5.327100, truth:1)\n",
            " --> \n",
            "you do n't have to know about music to enhance the film 's usual blend of \u001b[92mcomedy\u001b[0m/\u001b[35mtravesty\u001b[0m and romance .  (diff: 0.856042, preds: \u001b[92m0.9306187\u001b[0m/\u001b[35m0.07457624\u001b[0m, lm_scores: -3.960481/-4.826675)\n",
            "\n",
            "\n",
            "you do n't have to know about music to \u001b[91mappreciate\u001b[0m the film 's \u001b[91measygoing\u001b[0m blend of comedy and romance . \n",
            "\n",
            "you do n't have to know about music to \u001b[94menhance\u001b[0m the film 's \u001b[94musual\u001b[0m blend of comedy and romance . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 8 / 0 / 9:  90% 9/10 [00:19<00:02,  2.12s/it]best_score: -1.404537558555603 raw_output: tensor([0.2962, 0.3488]) text: in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to utter turkey . \n",
            "  depth = 1, len(beam) = 20\n",
            "  beam_succeed: False score: -1.404537558555603 raw_output: tensor([0.2962, 0.3488]) text: 'in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to utter turkey . '\n",
            "  beam_succeed: False score: -3.5870394706726074 raw_output: tensor([0.0279, 0.0078]) text: 'in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to crisp turkey . '\n",
            "  beam_succeed: False score: -3.637892961502075 raw_output: tensor([0.0245, 0.0270]) text: 'lasting exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to utter turkey . '\n",
            "  beam_succeed: False score: -3.683959484100342 raw_output: tensor([0.0258, 0.0253]) text: 'in exactly 89 minutes , most of which passed as slowly as if i 'd been photographed naked on an igloo , formula 51 sank from quirky to jerky to utter turkey . '\n",
            "  beam_succeed: False score: -3.748624563217163 raw_output: tensor([0.0239, 0.0132]) text: 'in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to perfect turkey . '\n",
            "best_score: 99.35423278808594 raw_output: tensor([0.6119, 0.1432]) text: in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to delicious to crisp turkey . \n",
            "  depth = 2, len(beam) = 20\n",
            "  beam_succeed: True score: 99.35423278808594 raw_output: tensor([0.6119, 0.1432]) text: 'in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to delicious to crisp turkey . '\n",
            "  beam_succeed: True score: 99.09068298339844 raw_output: tensor([0.3537, 0.6232]) text: 'in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an igloo , formula slowly sank from quirky to jerky to utter turkey . '\n",
            "  beam_succeed: True score: 99.06494903564453 raw_output: tensor([0.3104, 0.5692]) text: 'in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an elevator , formula 51 sank from quirky to jerky to utter turkey . '\n",
            "  beam_succeed: True score: 99.05068969726562 raw_output: tensor([0.6000, 0.3549]) text: 'in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to jerky to crisp turkey . '\n",
            "  beam_succeed: True score: 99.04188537597656 raw_output: tensor([0.2844, 0.5361]) text: 'in exactly 89 minutes , most of which passed remarkably slowly as if i 'd been sitting naked on an igloo , formula one sank from quirky to jerky to utter turkey . '\n",
            "\n",
            "--------------------------------------------- Result 10 ---------------------------------------------\n",
            "\n",
            "in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from \u001b[92mquirky\u001b[0m/\u001b[35mlunatic\u001b[0m to jerky to utter turkey .  (diff: 0.000365, preds: \u001b[92m0.0037071821\u001b[0m/\u001b[35m0.0033418552\u001b[0m, lm_scores: -5.123510/-5.007088, truth:0)\n",
            " --> \n",
            "in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from \u001b[92mquirky\u001b[0m/\u001b[35mlunatic\u001b[0m to delicious to crisp turkey .  (diff: 0.468668, preds: \u001b[92m0.6119155\u001b[0m/\u001b[35m0.14324778\u001b[0m, lm_scores: -4.804120/-4.804063)\n",
            "\n",
            "\n",
            "in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to \u001b[91mjerky\u001b[0m to \u001b[91mutter\u001b[0m turkey . \n",
            "\n",
            "in exactly 89 minutes , most of which passed as slowly as if i 'd been sitting naked on an igloo , formula 51 sank from quirky to \u001b[94mdelicious\u001b[0m to \u001b[94mcrisp\u001b[0m turkey . \n",
            "\n",
            "\n",
            "[Succeeded / Failed / Total] 9 / 0 / 10: 100% 10/10 [00:29<00:00,  2.90s/it]\n",
            "\n",
            "+-------------------+----------------------------------------------------+\n",
            "| Attack Details    |                                                    |\n",
            "+-------------------+----------------------------------------------------+\n",
            "| Attack algorithm: | Attack(                                            |\n",
            "|                   |   (search_method): BeamSearchPlus(                 |\n",
            "|                   |     (beam_width):  20                              |\n",
            "|                   |     (beam_sampling_method):  max                   |\n",
            "|                   |     (transformations_sampling_ratio):  1           |\n",
            "|                   |     (search_depth):  6                             |\n",
            "|                   |   )                                                |\n",
            "|                   |   (goal_function):  BiasGoalFunction(              |\n",
            "|                   |     (biaswords_list_len):  109076                  |\n",
            "|                   |     (active_biaswords_logit_threshold):  1.5       |\n",
            "|                   |     (biasthreshold):  0.5                          |\n",
            "|                   |     (diffthreshold):  0.03                         |\n",
            "|                   |     (stepweight):  0.1                             |\n",
            "|                   |     (skipthreshold):  0.1                          |\n",
            "|                   |     (score_mode):  max                             |\n",
            "|                   |     (lm_scorer):  distilbert-base-uncased          |\n",
            "|                   |     (model_batch_size):  512                       |\n",
            "|                   |     (query_budget):  50000                         |\n",
            "|                   |   )                                                |\n",
            "|                   |   (transformation):  CompositeTransformation(      |\n",
            "|                   |     (0): InitialBiasWord(                          |\n",
            "|                   |         (biaswords_flatten_len):  25601            |\n",
            "|                   |       )                                            |\n",
            "|                   |     (1): WordSwapMaskedLMPlus(                     |\n",
            "|                   |         (max_masks):  1                            |\n",
            "|                   |         (max_trials):  -1                          |\n",
            "|                   |         (logit_threshold):  3                      |\n",
            "|                   |         (force_rte_format):  False                 |\n",
            "|                   |         (method):  bae_plus                        |\n",
            "|                   |         (masked_lm_name):  distilbert-base-uncased |\n",
            "|                   |         (max_length):  128                         |\n",
            "|                   |         (max_candidates):  20                      |\n",
            "|                   |       )                                            |\n",
            "|                   |     )                                              |\n",
            "|                   |   (constraints):                                   |\n",
            "|                   |     (0): ActiveAnchorWordsModification(            |\n",
            "|                   |         (compare_against_original):  False         |\n",
            "|                   |       )                                            |\n",
            "|                   |   (is_black_box):  True                            |\n",
            "|                   | )                                                  |\n",
            "+-------------------+----------------------------------------------------++--------------------------+-------------------------+\n",
            "| Extra Stats              |                         |\n",
            "+--------------------------+-------------------------+\n",
            "| lm_scorer_type           | distilbert-base-uncased |\n",
            "| lm_score_original_stats  |               0         |\n",
            "|                          | count  9.000000         |\n",
            "|                          | mean  -5.226456         |\n",
            "|                          | std    1.171616         |\n",
            "|                          | min   -7.006533         |\n",
            "|                          | 25%   -5.702733         |\n",
            "|                          | 50%   -5.570937         |\n",
            "|                          | 75%   -4.406572         |\n",
            "|                          | max   -3.527787         |\n",
            "| lm_score_perturbed_stats |               0         |\n",
            "|                          | count  9.000000         |\n",
            "|                          | mean  -4.689677         |\n",
            "|                          | std    1.219577         |\n",
            "|                          | min   -6.373517         |\n",
            "|                          | 25%   -5.570937         |\n",
            "|                          | 50%   -4.889042         |\n",
            "|                          | 75%   -3.960481         |\n",
            "|                          | max   -2.868394         |\n",
            "+--------------------------+-------------------------++-------------------------------+---------+\n",
            "| Attack Results                |         |\n",
            "+-------------------------------+---------+\n",
            "| Number of successful attacks: | 9       |\n",
            "| Number of failed attacks:     | 0       |\n",
            "| Number of skipped attacks:    | 1       |\n",
            "| Number of total attacks:      | 10      |\n",
            "| Original accuracy:            | 90.0%   |\n",
            "| Accuracy under attack:        | 0.0%    |\n",
            "| Attack success rate:          | 100.0%  |\n",
            "| Average successful score:     | 99.5816 |\n",
            "| Average failed score:         | 0       |\n",
            "| Average perturbed word #:     | 1.33    |\n",
            "| Average perturbed word %:     | 11.76%  |\n",
            "| Average num. words per input: | 13.4    |\n",
            "| Avg num queries:              | 1561.67 |\n",
            "+-------------------------------+---------+\n",
            "2021-04-11 03:50:51; \u001b[34;1mtextattack\u001b[0m: Attack time: 32.519087076187134s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EhTHMGmXCxWg"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}