Fine_Tuning_BERT.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Fine-Tuning BERT.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/srulikbd/Deep-Learning/blob/master/Fine_Tuning_BERT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QWmtbdYHI9HN",
        "colab_type": "text"
      },
      "source": [
        "# Fine-Tuning GPT2 on Colab GPU… For Free!\n",
        "\n",
        "This is a colab notebook for the [associated Medium article](https://medium.com/p/340468c92ed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ov2wQbRIs8J",
        "colab_type": "text"
      },
      "source": [
        "## Installing Dependencies\n",
        "We would run pip3 install transformers normally in Bash, but because this is in Colab, we have to run it with !"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GVQ5Le5kF7CV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 655
        },
        "outputId": "de6cc4a3-3da9-41ef-b1f0-92551c04e3b9"
      },
      "source": [
        "!pip3 install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/48/35/ad2c5b1b8f99feaaf9d7cdadaeef261f098c6e1a6a2935d4d07662a6b780/transformers-2.11.0-py3-none-any.whl (674kB)\n",
            "\u001b[K     |████████████████████████████████| 675kB 8.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Collecting sentencepiece\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 23.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 53.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)\n",
            "Collecting tokenizers==0.7.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/e5/a26eb4716523808bb0a799fcfdceb6ebf77a18169d9591b2f46a9adb87d9/tokenizers-0.7.0-cp36-cp36m-manylinux1_x86_64.whl (3.8MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8MB 55.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.4.5.2)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.9)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (1.12.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.15.1)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893260 sha256=217e35c08413818bb7f62a3211946b7e6629e7a4299abe47e8002eb95868beec\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sentencepiece, sacremoses, tokenizers, transformers\n",
            "Successfully installed sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.7.0 transformers-2.11.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wdfc4qUVKYLj",
        "colab_type": "text"
      },
      "source": [
        "## Getting WikiText Data\n",
        "\n",
        "You can read more about WikiText data here. Overall, there's WikiText-2 and WikiText-103. We're going to use WikiText-2 because it's smaller, and we have limits in terms of how long we can run on GPU, and how much data we can load into memory in Colab. To download and run"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C4RMT_FQIrGi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "# wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip\n",
        "# unzip wikitext-2-raw-v1.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEJjg5wkLXgI",
        "colab_type": "text"
      },
      "source": [
        "## Fine-Tuning GPT2\n",
        "\n",
        "HuggingFace actually provides a script to help fine tune models here. We can just download the script by running"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E-L4LiHiKdr1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "outputId": "4fcf38e9-3608-4ebb-98a7-19752cd08b84"
      },
      "source": [
        "! wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_language_modeling.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-06-23 13:26:28--  https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_language_modeling.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10399 (10K) [text/plain]\n",
            "Saving to: ‘run_language_modeling.py’\n",
            "\n",
            "\rrun_language_modeli   0%[                    ]       0  --.-KB/s               \rrun_language_modeli 100%[===================>]  10.16K  --.-KB/s    in 0s      \n",
            "\n",
            "2020-06-23 13:26:28 (108 MB/s) - ‘run_language_modeling.py’ saved [10399/10399]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sklH4LNoMxRC",
        "colab_type": "text"
      },
      "source": [
        "Now we are ready to fine tune.\n",
        "\n",
        "There are many parameters to the script, and you can understand them by reading the manual. I'm just going to go over the important ones for basic training.\n",
        "\n",
        "- `output_dir` is where the model will be output\n",
        "- `model_type` is what model you want to use. In our case, it's gpt2 \n",
        "- `model_name_or_path` is the path to the model. If you want to train from scratch, you can leave this blank. In our case, it's also gpt2 \n",
        "- `do_train` tells it to train\n",
        "- `train_data_file` points to the training file\n",
        "- `do_eval` tells it to evaluate afterwards. Not always required, but good to have\n",
        "- `eval_data_file` points to the evaluation file\n",
        "\n",
        "Some extra ones you MAY care about, but you can also skip this.\n",
        "- `save_steps` is when to save checkpoints. If you have limited memory, you can set this to -1 so it'll skip saving until the end\n",
        "- `per_gpu_train_batch_size` is batch size for GPU. You can increase this if your GPU has enough memory. To be safe, you can start with 1 and ramp it up if you still have memory\n",
        "- `num_train_epochs` is the number of epochs to train. Since we're fine-tuning, I'm going to set this to 2\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ab20QwA7Di9s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N36iOaYoll3b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "af17b144-7cf6-4349-e054-099329c1eba0"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "data=pd.read_csv(\"/content/sample_data/DB_spanish_clean-train.csv\",encoding='latin1', header=None)\n",
        "print(data.head())\n",
        "clean_data = data[0]\n",
        "np.savetxt(r'/content/sample_data/LML-small-train.txt', data.values, fmt='%s')\n",
        "\n",
        "data=pd.read_csv(\"/content/sample_data/DB_spanish_clean-test.csv\",encoding='latin1', header=None)\n",
        "print(data.head())\n",
        "clean_data = data[0]\n",
        "np.savetxt(r'/content/sample_data/LML-small-test.txt', data.values, fmt='%s')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                   0\n",
            "0  ï»¿La sociedad civil pide la urgente liberaci?...\n",
            "1  Espa?a va a la ruina pero aumenta las  transfe...\n",
            "2  Israel mantiene prisioneros a 200 ni?os con la...\n",
            "3  Muere un preso palestino por negligencia m?dic...\n",
            "4  Israel confisca terrenos palestinos de la mezq...\n",
            "                                                   0  ...        5\n",
            "0  ï»¿\"Espero que se reanude la audiencia de dete...  ...   coâ¦\"\n",
            "1  Estado criminal de Israel ha asesinado un ni?o...  ...      NaN\n",
            "2  #IAI de #Israel utiliza inteligencia artificia...  ...      NaN\n",
            "3  Â¿C?mo afecta a #Israel el conflicto entre la ...  ...      NaN\n",
            "4  #LaCuarentenaMata Instituto cardiovascular anu...  ...      NaN\n",
            "\n",
            "[5 rows x 6 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y6fspISiMx5V",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "08a144cb-eae4-4ff9-b6ee-b56fb3152919"
      },
      "source": [
        "#ORIGINAL code\n",
        "# %%bash\n",
        "# export TRAIN_FILE=wikitext-2-raw/wiki.train.raw\n",
        "# export TEST_FILE=wikitext-2-raw/wiki.test.raw\n",
        "# export MODEL_NAME=bert-base-multilingual-cased\n",
        "# export OUTPUT_DIR=output\n",
        " \n",
        "# python run_language_modeling.py \\\n",
        "#     --output_dir=$OUTPUT_DIR \\\n",
        "#     --model_type=$MODEL_NAME \\\n",
        "#     --model_name_or_path=$MODEL_NAME \\\n",
        "#     --do_train \\\n",
        "#     --train_data_file=$TRAIN_FILE \\\n",
        "#     --do_eval \\\n",
        "#     --eval_data_file=$TEST_FILE \\\n",
        "#     --per_gpu_train_batch_size=1 \\\n",
        "#     --save_steps=-1 \\\n",
        "#     --num_train_epochs=2 \\\n",
        "#     --mlm \\\n",
        "#     --line_by_line\n",
        " \n",
        "# python run_language_modeling.py \\\n",
        "#     --help\n",
        " \n",
        " \n",
        " \n",
        "%%bash\n",
        "export TRAIN_FILE=/content/sample_data/LML-small-train.txt\n",
        "export TEST_FILE=/content/sample_data/LML-small-test.txt\n",
        "export MODEL_NAME=bert-base-multilingual-cased\n",
        "export OUTPUT_DIR=/content/output\n",
        " \n",
        " \n",
        " \n",
        "python run_language_modeling.py \\\n",
        "    --output_dir=$OUTPUT_DIR \\\n",
        "    --model_type=$MODEL_NAME \\\n",
        "    --model_name_or_path=$MODEL_NAME \\\n",
        "    --do_train \\\n",
        "    --train_data_file=$TRAIN_FILE \\\n",
        "    --do_eval \\\n",
        "    --eval_data_file=$TEST_FILE \\\n",
        "    --per_gpu_train_batch_size=1 \\\n",
        "    --save_steps=-1 \\\n",
        "    --num_train_epochs=2 \\\n",
        "    --mlm \\\n",
        "    --line_by_line \\\n",
        "    --overwrite_output_dir\n",
        " \n",
        "python run_language_modeling.py \\\n",
        "    --help"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\"loss\": 3.4809115029647946, \"learning_rate\": 4.990124042032077e-05, \"epoch\": 0.003950383187169155, \"step\": 500}\n",
            "{\"loss\": 3.370515186324716, \"learning_rate\": 4.980248084064155e-05, \"epoch\": 0.00790076637433831, \"step\": 1000}\n",
            "{\"loss\": 3.291939713080108, \"learning_rate\": 4.9703721260962314e-05, \"epoch\": 0.011851149561507466, \"step\": 1500}\n",
            "{\"loss\": 3.198993044391449, \"learning_rate\": 4.9604961681283084e-05, \"epoch\": 0.01580153274867662, \"step\": 2000}\n",
            "{\"loss\": 3.1517792300977745, \"learning_rate\": 4.950620210160386e-05, \"epoch\": 0.019751915935845778, \"step\": 2500}\n",
            "{\"loss\": 3.2199107205306645, \"learning_rate\": 4.940744252192463e-05, \"epoch\": 0.02370229912301493, \"step\": 3000}\n",
            "{\"loss\": 3.1380331875588743, \"learning_rate\": 4.93086829422454e-05, \"epoch\": 0.027652682310184088, \"step\": 3500}\n",
            "{\"loss\": 3.1134904662013434, \"learning_rate\": 4.920992336256617e-05, \"epoch\": 0.03160306549735324, \"step\": 4000}\n",
            "{\"loss\": 3.1257499504419974, \"learning_rate\": 4.911116378288694e-05, \"epoch\": 0.0355534486845224, \"step\": 4500}\n",
            "{\"loss\": 3.1288266300729375, \"learning_rate\": 4.901240420320771e-05, \"epoch\": 0.039503831871691555, \"step\": 5000}\n",
            "{\"loss\": 3.2566165954437456, \"learning_rate\": 4.891364462352849e-05, \"epoch\": 0.04345421505886071, \"step\": 5500}\n",
            "{\"loss\": 2.9890558199403747, \"learning_rate\": 4.8814885043849254e-05, \"epoch\": 0.04740459824602986, \"step\": 6000}\n",
            "{\"loss\": 3.083977729254868, \"learning_rate\": 4.8716125464170024e-05, \"epoch\": 0.05135498143319902, \"step\": 6500}\n",
            "{\"loss\": 3.045721950461477, \"learning_rate\": 4.8617365884490795e-05, \"epoch\": 0.055305364620368176, \"step\": 7000}\n",
            "{\"loss\": 3.042492155402826, \"learning_rate\": 4.851860630481157e-05, \"epoch\": 0.05925574780753733, \"step\": 7500}\n",
            "{\"loss\": 3.004332521767021, \"learning_rate\": 4.841984672513234e-05, \"epoch\": 0.06320613099470648, \"step\": 8000}\n",
            "{\"loss\": 3.0962674845457077, \"learning_rate\": 4.8321087145453106e-05, \"epoch\": 0.06715651418187564, \"step\": 8500}\n",
            "{\"loss\": 3.1685527888089418, \"learning_rate\": 4.822232756577388e-05, \"epoch\": 0.0711068973690448, \"step\": 9000}\n",
            "{\"loss\": 3.1302304696830396, \"learning_rate\": 4.8123567986094653e-05, \"epoch\": 0.07505728055621395, \"step\": 9500}\n",
            "{\"loss\": 2.970430748871528, \"learning_rate\": 4.8024808406415424e-05, \"epoch\": 0.07900766374338311, \"step\": 10000}\n",
            "{\"loss\": 3.052051359906087, \"learning_rate\": 4.7926048826736194e-05, \"epoch\": 0.08295804693055227, \"step\": 10500}\n",
            "{\"loss\": 3.038915867866657, \"learning_rate\": 4.7827289247056965e-05, \"epoch\": 0.08690843011772142, \"step\": 11000}\n",
            "{\"loss\": 3.1038201808128507, \"learning_rate\": 4.7728529667377735e-05, \"epoch\": 0.09085881330489057, \"step\": 11500}\n",
            "{\"loss\": 3.0548545603951207, \"learning_rate\": 4.762977008769851e-05, \"epoch\": 0.09480919649205972, \"step\": 12000}\n",
            "{\"loss\": 2.914063408655929, \"learning_rate\": 4.753101050801928e-05, \"epoch\": 0.09875957967922888, \"step\": 12500}\n",
            "{\"loss\": 3.2513942787169943, \"learning_rate\": 4.743225092834005e-05, \"epoch\": 0.10270996286639804, \"step\": 13000}\n",
            "{\"loss\": 3.0408398781552095, \"learning_rate\": 4.733349134866082e-05, \"epoch\": 0.1066603460535672, \"step\": 13500}\n",
            "{\"loss\": 3.1501659149282495, \"learning_rate\": 4.7234731768981594e-05, \"epoch\": 0.11061072924073635, \"step\": 14000}\n",
            "{\"loss\": 2.865317083110873, \"learning_rate\": 4.7135972189302364e-05, \"epoch\": 0.11456111242790551, \"step\": 14500}\n",
            "{\"loss\": 2.950732013755769, \"learning_rate\": 4.703721260962314e-05, \"epoch\": 0.11851149561507467, \"step\": 15000}\n",
            "{\"loss\": 2.9070220271501457, \"learning_rate\": 4.6938453029943905e-05, \"epoch\": 0.12246187880224382, \"step\": 15500}\n",
            "{\"loss\": 2.955156158924103, \"learning_rate\": 4.6839693450264675e-05, \"epoch\": 0.12641226198941297, \"step\": 16000}\n",
            "{\"loss\": 2.9541794698581216, \"learning_rate\": 4.674093387058545e-05, \"epoch\": 0.13036264517658214, \"step\": 16500}\n",
            "{\"loss\": 2.9404459119275272, \"learning_rate\": 4.664217429090622e-05, \"epoch\": 0.13431302836375128, \"step\": 17000}\n",
            "{\"loss\": 2.9214531363554705, \"learning_rate\": 4.654341471122699e-05, \"epoch\": 0.13826341155092045, \"step\": 17500}\n",
            "{\"loss\": 3.0703270422792994, \"learning_rate\": 4.644465513154776e-05, \"epoch\": 0.1422137947380896, \"step\": 18000}\n",
            "{\"loss\": 3.0444178624745692, \"learning_rate\": 4.6345895551868534e-05, \"epoch\": 0.14616417792525874, \"step\": 18500}\n",
            "{\"loss\": 2.904039360095121, \"learning_rate\": 4.6247135972189304e-05, \"epoch\": 0.1501145611124279, \"step\": 19000}\n",
            "{\"loss\": 2.9335333208628, \"learning_rate\": 4.6148376392510075e-05, \"epoch\": 0.15406494429959705, \"step\": 19500}\n",
            "{\"loss\": 3.084042174328119, \"learning_rate\": 4.6049616812830845e-05, \"epoch\": 0.15801532748676622, \"step\": 20000}\n",
            "{\"loss\": 2.881011160062626, \"learning_rate\": 4.5950857233151616e-05, \"epoch\": 0.16196571067393536, \"step\": 20500}\n",
            "{\"loss\": 2.904991966372734, \"learning_rate\": 4.5852097653472386e-05, \"epoch\": 0.16591609386110454, \"step\": 21000}\n",
            "{\"loss\": 2.747880072373897, \"learning_rate\": 4.575333807379316e-05, \"epoch\": 0.16986647704827368, \"step\": 21500}\n",
            "{\"loss\": 2.8960947299055113, \"learning_rate\": 4.5654578494113934e-05, \"epoch\": 0.17381686023544285, \"step\": 22000}\n",
            "{\"loss\": 2.878027852674946, \"learning_rate\": 4.55558189144347e-05, \"epoch\": 0.177767243422612, \"step\": 22500}\n",
            "{\"loss\": 3.0142162559656427, \"learning_rate\": 4.5457059334755474e-05, \"epoch\": 0.18171762660978114, \"step\": 23000}\n",
            "{\"loss\": 2.833651960285177, \"learning_rate\": 4.5358299755076245e-05, \"epoch\": 0.1856680097969503, \"step\": 23500}\n",
            "{\"loss\": 3.0386166405777915, \"learning_rate\": 4.5259540175397015e-05, \"epoch\": 0.18961839298411945, \"step\": 24000}\n",
            "{\"loss\": 2.8380062748383206, \"learning_rate\": 4.5160780595717786e-05, \"epoch\": 0.19356877617128862, \"step\": 24500}\n",
            "{\"loss\": 2.861651923158817, \"learning_rate\": 4.5062021016038556e-05, \"epoch\": 0.19751915935845776, \"step\": 25000}\n",
            "{\"loss\": 2.754984182869084, \"learning_rate\": 4.4963261436359326e-05, \"epoch\": 0.20146954254562693, \"step\": 25500}\n",
            "{\"loss\": 2.871991731901653, \"learning_rate\": 4.4864501856680104e-05, \"epoch\": 0.20541992573279608, \"step\": 26000}\n",
            "{\"loss\": 2.7862389608932427, \"learning_rate\": 4.4765742277000874e-05, \"epoch\": 0.20937030891996525, \"step\": 26500}\n",
            "{\"loss\": 2.78666270462246, \"learning_rate\": 4.4666982697321644e-05, \"epoch\": 0.2133206921071344, \"step\": 27000}\n",
            "{\"loss\": 2.8127000698954214, \"learning_rate\": 4.4568223117642415e-05, \"epoch\": 0.21727107529430353, \"step\": 27500}\n",
            "{\"loss\": 2.8932981126875674, \"learning_rate\": 4.4469463537963185e-05, \"epoch\": 0.2212214584814727, \"step\": 28000}\n",
            "{\"loss\": 2.8329817837417357, \"learning_rate\": 4.4370703958283955e-05, \"epoch\": 0.22517184166864185, \"step\": 28500}\n",
            "{\"loss\": 2.693137484416191, \"learning_rate\": 4.427194437860473e-05, \"epoch\": 0.22912222485581102, \"step\": 29000}\n",
            "{\"loss\": 2.944122396456776, \"learning_rate\": 4.4173184798925496e-05, \"epoch\": 0.23307260804298016, \"step\": 29500}\n",
            "{\"loss\": 2.86720035784645, \"learning_rate\": 4.407442521924627e-05, \"epoch\": 0.23702299123014933, \"step\": 30000}\n",
            "{\"loss\": 2.829259027672408, \"learning_rate\": 4.397566563956704e-05, \"epoch\": 0.24097337441731848, \"step\": 30500}\n",
            "{\"loss\": 2.7501019349695417, \"learning_rate\": 4.3876906059887814e-05, \"epoch\": 0.24492375760448765, \"step\": 31000}\n",
            "{\"loss\": 2.896386784043163, \"learning_rate\": 4.3778146480208585e-05, \"epoch\": 0.2488741407916568, \"step\": 31500}\n",
            "{\"loss\": 2.744155945962062, \"learning_rate\": 4.367938690052935e-05, \"epoch\": 0.25282452397882593, \"step\": 32000}\n",
            "{\"loss\": 2.913533203096129, \"learning_rate\": 4.3580627320850125e-05, \"epoch\": 0.2567749071659951, \"step\": 32500}\n",
            "{\"loss\": 2.751950399627327, \"learning_rate\": 4.3481867741170896e-05, \"epoch\": 0.2607252903531643, \"step\": 33000}\n",
            "{\"loss\": 2.7324166980540903, \"learning_rate\": 4.3383108161491666e-05, \"epoch\": 0.2646756735403334, \"step\": 33500}\n",
            "{\"loss\": 2.884825499239145, \"learning_rate\": 4.3284348581812437e-05, \"epoch\": 0.26862605672750256, \"step\": 34000}\n",
            "{\"loss\": 2.9024080105038594, \"learning_rate\": 4.318558900213321e-05, \"epoch\": 0.27257643991467173, \"step\": 34500}\n",
            "{\"loss\": 2.9297430072698334, \"learning_rate\": 4.308682942245398e-05, \"epoch\": 0.2765268231018409, \"step\": 35000}\n",
            "{\"loss\": 2.8529280159448973, \"learning_rate\": 4.2988069842774755e-05, \"epoch\": 0.28047720628901, \"step\": 35500}\n",
            "{\"loss\": 2.8206652539126806, \"learning_rate\": 4.2889310263095525e-05, \"epoch\": 0.2844275894761792, \"step\": 36000}\n",
            "{\"loss\": 2.837743317022687, \"learning_rate\": 4.279055068341629e-05, \"epoch\": 0.28837797266334836, \"step\": 36500}\n",
            "{\"loss\": 2.6654541892058914, \"learning_rate\": 4.2691791103737066e-05, \"epoch\": 0.2923283558505175, \"step\": 37000}\n",
            "{\"loss\": 2.9877890567888388, \"learning_rate\": 4.2593031524057836e-05, \"epoch\": 0.29627873903768664, \"step\": 37500}\n",
            "{\"loss\": 2.8390556419219357, \"learning_rate\": 4.2494271944378607e-05, \"epoch\": 0.3002291222248558, \"step\": 38000}\n",
            "{\"loss\": 2.7855924675818824, \"learning_rate\": 4.239551236469938e-05, \"epoch\": 0.304179505412025, \"step\": 38500}\n",
            "{\"loss\": 2.91712626393931, \"learning_rate\": 4.229675278502015e-05, \"epoch\": 0.3081298885991941, \"step\": 39000}\n",
            "{\"loss\": 2.927328155957308, \"learning_rate\": 4.219799320534092e-05, \"epoch\": 0.3120802717863633, \"step\": 39500}\n",
            "{\"loss\": 2.793938819427276, \"learning_rate\": 4.2099233625661695e-05, \"epoch\": 0.31603065497353244, \"step\": 40000}\n",
            "{\"loss\": 2.6523098194638735, \"learning_rate\": 4.2000474045982465e-05, \"epoch\": 0.3199810381607016, \"step\": 40500}\n",
            "{\"loss\": 2.6609509968223573, \"learning_rate\": 4.1901714466303236e-05, \"epoch\": 0.32393142134787073, \"step\": 41000}\n",
            "{\"loss\": 2.6647554636407293, \"learning_rate\": 4.1802954886624006e-05, \"epoch\": 0.3278818045350399, \"step\": 41500}\n",
            "{\"loss\": 2.85227825602144, \"learning_rate\": 4.1704195306944776e-05, \"epoch\": 0.33183218772220907, \"step\": 42000}\n",
            "{\"loss\": 2.720023458413547, \"learning_rate\": 4.160543572726555e-05, \"epoch\": 0.3357825709093782, \"step\": 42500}\n",
            "{\"loss\": 2.7650916148089744, \"learning_rate\": 4.150667614758632e-05, \"epoch\": 0.33973295409654736, \"step\": 43000}\n",
            "{\"loss\": 2.6105473475856416, \"learning_rate\": 4.140791656790709e-05, \"epoch\": 0.34368333728371653, \"step\": 43500}\n",
            "{\"loss\": 2.6992212831247597, \"learning_rate\": 4.130915698822786e-05, \"epoch\": 0.3476337204708857, \"step\": 44000}\n",
            "{\"loss\": 2.8231090952188533, \"learning_rate\": 4.121039740854863e-05, \"epoch\": 0.3515841036580548, \"step\": 44500}\n",
            "{\"loss\": 2.7184550144526294, \"learning_rate\": 4.1111637828869406e-05, \"epoch\": 0.355534486845224, \"step\": 45000}\n",
            "{\"loss\": 2.765630405401229, \"learning_rate\": 4.1012878249190176e-05, \"epoch\": 0.35948487003239316, \"step\": 45500}\n",
            "{\"loss\": 2.8014400599953952, \"learning_rate\": 4.091411866951094e-05, \"epoch\": 0.36343525321956227, \"step\": 46000}\n",
            "{\"loss\": 2.6924691200684174, \"learning_rate\": 4.081535908983172e-05, \"epoch\": 0.36738563640673144, \"step\": 46500}\n",
            "{\"loss\": 2.740469143120863, \"learning_rate\": 4.071659951015249e-05, \"epoch\": 0.3713360195939006, \"step\": 47000}\n",
            "{\"loss\": 2.6951552927980083, \"learning_rate\": 4.061783993047326e-05, \"epoch\": 0.3752864027810698, \"step\": 47500}\n",
            "{\"loss\": 2.7876171856054572, \"learning_rate\": 4.051908035079403e-05, \"epoch\": 0.3792367859682389, \"step\": 48000}\n",
            "{\"loss\": 2.7306606264467117, \"learning_rate\": 4.04203207711148e-05, \"epoch\": 0.38318716915540807, \"step\": 48500}\n",
            "{\"loss\": 2.758545202432666, \"learning_rate\": 4.032156119143557e-05, \"epoch\": 0.38713755234257724, \"step\": 49000}\n",
            "{\"loss\": 2.5415976999078413, \"learning_rate\": 4.0222801611756346e-05, \"epoch\": 0.3910879355297464, \"step\": 49500}\n",
            "{\"loss\": 2.9058702518215287, \"learning_rate\": 4.0124042032077116e-05, \"epoch\": 0.3950383187169155, \"step\": 50000}\n",
            "{\"loss\": 2.79018478433101, \"learning_rate\": 4.002528245239788e-05, \"epoch\": 0.3989887019040847, \"step\": 50500}\n",
            "{\"loss\": 2.8107297151358215, \"learning_rate\": 3.992652287271866e-05, \"epoch\": 0.40293908509125387, \"step\": 51000}\n",
            "{\"loss\": 2.714560506824404, \"learning_rate\": 3.982776329303943e-05, \"epoch\": 0.406889468278423, \"step\": 51500}\n",
            "{\"loss\": 2.820684183811478, \"learning_rate\": 3.97290037133602e-05, \"epoch\": 0.41083985146559215, \"step\": 52000}\n",
            "{\"loss\": 2.67965393685468, \"learning_rate\": 3.963024413368097e-05, \"epoch\": 0.4147902346527613, \"step\": 52500}\n",
            "{\"loss\": 2.670394297598803, \"learning_rate\": 3.953148455400174e-05, \"epoch\": 0.4187406178399305, \"step\": 53000}\n",
            "{\"loss\": 2.749626072260784, \"learning_rate\": 3.943272497432251e-05, \"epoch\": 0.4226910010270996, \"step\": 53500}\n",
            "{\"loss\": 2.7771817642910173, \"learning_rate\": 3.933396539464328e-05, \"epoch\": 0.4266413842142688, \"step\": 54000}\n",
            "{\"loss\": 2.9321938381741637, \"learning_rate\": 3.923520581496406e-05, \"epoch\": 0.43059176740143795, \"step\": 54500}\n",
            "{\"loss\": 2.606289192499942, \"learning_rate\": 3.913644623528483e-05, \"epoch\": 0.43454215058860707, \"step\": 55000}\n",
            "{\"loss\": 2.6468204576282877, \"learning_rate\": 3.903768665560559e-05, \"epoch\": 0.43849253377577624, \"step\": 55500}\n",
            "{\"loss\": 2.548060212453478, \"learning_rate\": 3.893892707592637e-05, \"epoch\": 0.4424429169629454, \"step\": 56000}\n",
            "{\"loss\": 2.8762970976986106, \"learning_rate\": 3.884016749624714e-05, \"epoch\": 0.4463933001501146, \"step\": 56500}\n",
            "{\"loss\": 2.8805275343582033, \"learning_rate\": 3.874140791656791e-05, \"epoch\": 0.4503436833372837, \"step\": 57000}\n",
            "{\"loss\": 2.5987377090905213, \"learning_rate\": 3.864264833688868e-05, \"epoch\": 0.45429406652445287, \"step\": 57500}\n",
            "{\"loss\": 2.8169290104839018, \"learning_rate\": 3.854388875720945e-05, \"epoch\": 0.45824444971162204, \"step\": 58000}\n",
            "{\"loss\": 2.8102052619154567, \"learning_rate\": 3.844512917753022e-05, \"epoch\": 0.4621948328987912, \"step\": 58500}\n",
            "{\"loss\": 2.5269063914472936, \"learning_rate\": 3.8346369597851e-05, \"epoch\": 0.4661452160859603, \"step\": 59000}\n",
            "{\"loss\": 2.7376451457332003, \"learning_rate\": 3.824761001817177e-05, \"epoch\": 0.4700955992731295, \"step\": 59500}\n",
            "{\"loss\": 2.6490981514486482, \"learning_rate\": 3.814885043849253e-05, \"epoch\": 0.47404598246029866, \"step\": 60000}\n",
            "{\"loss\": 2.706161173705419, \"learning_rate\": 3.805009085881331e-05, \"epoch\": 0.4779963656474678, \"step\": 60500}\n",
            "{\"loss\": 2.781353258790623, \"learning_rate\": 3.795133127913408e-05, \"epoch\": 0.48194674883463695, \"step\": 61000}\n",
            "{\"loss\": 2.7162894815226317, \"learning_rate\": 3.785257169945485e-05, \"epoch\": 0.4858971320218061, \"step\": 61500}\n",
            "{\"loss\": 2.691967201689258, \"learning_rate\": 3.775381211977562e-05, \"epoch\": 0.4898475152089753, \"step\": 62000}\n",
            "{\"loss\": 2.7715283238593837, \"learning_rate\": 3.765505254009639e-05, \"epoch\": 0.4937978983961444, \"step\": 62500}\n",
            "{\"loss\": 2.722436371983611, \"learning_rate\": 3.755629296041716e-05, \"epoch\": 0.4977482815833136, \"step\": 63000}\n",
            "{\"loss\": 2.7415676426860154, \"learning_rate\": 3.745753338073794e-05, \"epoch\": 0.5016986647704827, \"step\": 63500}\n",
            "{\"loss\": 2.6895663914327743, \"learning_rate\": 3.735877380105871e-05, \"epoch\": 0.5056490479576519, \"step\": 64000}\n",
            "{\"loss\": 2.6098531928823214, \"learning_rate\": 3.726001422137947e-05, \"epoch\": 0.5095994311448211, \"step\": 64500}\n",
            "{\"loss\": 2.6107595351486816, \"learning_rate\": 3.716125464170025e-05, \"epoch\": 0.5135498143319902, \"step\": 65000}\n",
            "{\"loss\": 2.7101302631536965, \"learning_rate\": 3.706249506202102e-05, \"epoch\": 0.5175001975191593, \"step\": 65500}\n",
            "{\"loss\": 2.8990770798582814, \"learning_rate\": 3.696373548234179e-05, \"epoch\": 0.5214505807063285, \"step\": 66000}\n",
            "{\"loss\": 2.774260461732629, \"learning_rate\": 3.686497590266256e-05, \"epoch\": 0.5254009638934977, \"step\": 66500}\n",
            "{\"loss\": 2.6953678200767026, \"learning_rate\": 3.676621632298333e-05, \"epoch\": 0.5293513470806668, \"step\": 67000}\n",
            "{\"loss\": 2.6803349055351573, \"learning_rate\": 3.66674567433041e-05, \"epoch\": 0.533301730267836, \"step\": 67500}\n",
            "{\"loss\": 2.7538293112132233, \"learning_rate\": 3.656869716362487e-05, \"epoch\": 0.5372521134550051, \"step\": 68000}\n",
            "{\"loss\": 2.676775964350032, \"learning_rate\": 3.646993758394565e-05, \"epoch\": 0.5412024966421742, \"step\": 68500}\n",
            "{\"loss\": 2.6277337674097505, \"learning_rate\": 3.637117800426642e-05, \"epoch\": 0.5451528798293435, \"step\": 69000}\n",
            "{\"loss\": 2.658511048705783, \"learning_rate\": 3.627241842458718e-05, \"epoch\": 0.5491032630165126, \"step\": 69500}\n",
            "{\"loss\": 2.7244190655740677, \"learning_rate\": 3.617365884490796e-05, \"epoch\": 0.5530536462036818, \"step\": 70000}\n",
            "{\"loss\": 2.658420501822955, \"learning_rate\": 3.607489926522873e-05, \"epoch\": 0.5570040293908509, \"step\": 70500}\n",
            "{\"loss\": 2.587608420547564, \"learning_rate\": 3.59761396855495e-05, \"epoch\": 0.56095441257802, \"step\": 71000}\n",
            "{\"loss\": 2.7746395696722903, \"learning_rate\": 3.587738010587027e-05, \"epoch\": 0.5649047957651893, \"step\": 71500}\n",
            "{\"loss\": 2.586447487058409, \"learning_rate\": 3.577862052619104e-05, \"epoch\": 0.5688551789523584, \"step\": 72000}\n",
            "{\"loss\": 2.6018865089270986, \"learning_rate\": 3.567986094651181e-05, \"epoch\": 0.5728055621395275, \"step\": 72500}\n",
            "{\"loss\": 2.6601321092781145, \"learning_rate\": 3.558110136683259e-05, \"epoch\": 0.5767559453266967, \"step\": 73000}\n",
            "{\"loss\": 2.723923187014472, \"learning_rate\": 3.548234178715336e-05, \"epoch\": 0.5807063285138658, \"step\": 73500}\n",
            "{\"loss\": 2.584230205563479, \"learning_rate\": 3.538358220747412e-05, \"epoch\": 0.584656711701035, \"step\": 74000}\n",
            "{\"loss\": 2.6909923675616736, \"learning_rate\": 3.52848226277949e-05, \"epoch\": 0.5886070948882042, \"step\": 74500}\n",
            "{\"loss\": 2.5960425300663337, \"learning_rate\": 3.518606304811567e-05, \"epoch\": 0.5925574780753733, \"step\": 75000}\n",
            "{\"loss\": 2.608063608747674, \"learning_rate\": 3.508730346843644e-05, \"epoch\": 0.5965078612625425, \"step\": 75500}\n",
            "{\"loss\": 2.685935434790328, \"learning_rate\": 3.498854388875721e-05, \"epoch\": 0.6004582444497116, \"step\": 76000}\n",
            "{\"loss\": 2.663209811239853, \"learning_rate\": 3.488978430907798e-05, \"epoch\": 0.6044086276368807, \"step\": 76500}\n",
            "{\"loss\": 2.8354935784297415, \"learning_rate\": 3.479102472939875e-05, \"epoch\": 0.60835901082405, \"step\": 77000}\n",
            "{\"loss\": 2.6377012752251465, \"learning_rate\": 3.469226514971952e-05, \"epoch\": 0.6123093940112191, \"step\": 77500}\n",
            "{\"loss\": 2.532906274185341, \"learning_rate\": 3.45935055700403e-05, \"epoch\": 0.6162597771983882, \"step\": 78000}\n",
            "{\"loss\": 2.715696574905829, \"learning_rate\": 3.449474599036106e-05, \"epoch\": 0.6202101603855574, \"step\": 78500}\n",
            "{\"loss\": 2.5651929642129105, \"learning_rate\": 3.439598641068183e-05, \"epoch\": 0.6241605435727265, \"step\": 79000}\n",
            "{\"loss\": 2.7038146265301037, \"learning_rate\": 3.429722683100261e-05, \"epoch\": 0.6281109267598957, \"step\": 79500}\n",
            "{\"loss\": 2.6863540927785796, \"learning_rate\": 3.419846725132338e-05, \"epoch\": 0.6320613099470649, \"step\": 80000}\n",
            "{\"loss\": 2.6626073985257537, \"learning_rate\": 3.409970767164415e-05, \"epoch\": 0.636011693134234, \"step\": 80500}\n",
            "{\"loss\": 2.642310571702721, \"learning_rate\": 3.400094809196492e-05, \"epoch\": 0.6399620763214032, \"step\": 81000}\n",
            "{\"loss\": 2.650377709068358, \"learning_rate\": 3.390218851228569e-05, \"epoch\": 0.6439124595085723, \"step\": 81500}\n",
            "{\"loss\": 2.646418583610561, \"learning_rate\": 3.380342893260646e-05, \"epoch\": 0.6478628426957415, \"step\": 82000}\n",
            "{\"loss\": 2.6090767648434268, \"learning_rate\": 3.370466935292724e-05, \"epoch\": 0.6518132258829107, \"step\": 82500}\n",
            "{\"loss\": 2.6745717001985176, \"learning_rate\": 3.360590977324801e-05, \"epoch\": 0.6557636090700798, \"step\": 83000}\n",
            "{\"loss\": 2.7004881300621784, \"learning_rate\": 3.350715019356877e-05, \"epoch\": 0.6597139922572489, \"step\": 83500}\n",
            "{\"loss\": 2.5269991350090133, \"learning_rate\": 3.340839061388955e-05, \"epoch\": 0.6636643754444181, \"step\": 84000}\n",
            "{\"loss\": 2.5438788774777206, \"learning_rate\": 3.330963103421032e-05, \"epoch\": 0.6676147586315873, \"step\": 84500}\n",
            "{\"loss\": 2.508322351780429, \"learning_rate\": 3.321087145453109e-05, \"epoch\": 0.6715651418187564, \"step\": 85000}\n",
            "{\"loss\": 2.576633948777162, \"learning_rate\": 3.311211187485186e-05, \"epoch\": 0.6755155250059256, \"step\": 85500}\n",
            "{\"loss\": 2.698074739413045, \"learning_rate\": 3.301335229517263e-05, \"epoch\": 0.6794659081930947, \"step\": 86000}\n",
            "{\"loss\": 2.6899262211501482, \"learning_rate\": 3.29145927154934e-05, \"epoch\": 0.6834162913802639, \"step\": 86500}\n",
            "{\"loss\": 2.469805496306857, \"learning_rate\": 3.281583313581418e-05, \"epoch\": 0.6873666745674331, \"step\": 87000}\n",
            "{\"loss\": 2.543441287716676, \"learning_rate\": 3.271707355613495e-05, \"epoch\": 0.6913170577546022, \"step\": 87500}\n",
            "{\"loss\": 2.6044299232594783, \"learning_rate\": 3.2618313976455714e-05, \"epoch\": 0.6952674409417714, \"step\": 88000}\n",
            "{\"loss\": 2.5654737147049165, \"learning_rate\": 3.251955439677649e-05, \"epoch\": 0.6992178241289405, \"step\": 88500}\n",
            "{\"loss\": 2.6121437421226292, \"learning_rate\": 3.242079481709726e-05, \"epoch\": 0.7031682073161096, \"step\": 89000}\n",
            "{\"loss\": 2.5724016154494604, \"learning_rate\": 3.232203523741803e-05, \"epoch\": 0.7071185905032789, \"step\": 89500}\n",
            "{\"loss\": 2.6969012985862792, \"learning_rate\": 3.22232756577388e-05, \"epoch\": 0.711068973690448, \"step\": 90000}\n",
            "{\"loss\": 2.678511506657582, \"learning_rate\": 3.212451607805957e-05, \"epoch\": 0.7150193568776171, \"step\": 90500}\n",
            "{\"loss\": 2.7633058671351174, \"learning_rate\": 3.202575649838034e-05, \"epoch\": 0.7189697400647863, \"step\": 91000}\n",
            "{\"loss\": 2.5965329246846958, \"learning_rate\": 3.192699691870111e-05, \"epoch\": 0.7229201232519554, \"step\": 91500}\n",
            "{\"loss\": 2.5941990685329075, \"learning_rate\": 3.182823733902189e-05, \"epoch\": 0.7268705064391245, \"step\": 92000}\n",
            "{\"loss\": 2.522083583692438, \"learning_rate\": 3.1729477759342654e-05, \"epoch\": 0.7308208896262938, \"step\": 92500}\n",
            "{\"loss\": 2.683781288566184, \"learning_rate\": 3.1630718179663424e-05, \"epoch\": 0.7347712728134629, \"step\": 93000}\n",
            "{\"loss\": 2.5698338679438457, \"learning_rate\": 3.15319585999842e-05, \"epoch\": 0.7387216560006321, \"step\": 93500}\n",
            "{\"loss\": 2.3989617017454585, \"learning_rate\": 3.143319902030497e-05, \"epoch\": 0.7426720391878012, \"step\": 94000}\n",
            "{\"loss\": 2.6507611893728606, \"learning_rate\": 3.133443944062574e-05, \"epoch\": 0.7466224223749703, \"step\": 94500}\n",
            "{\"loss\": 2.599648575364845, \"learning_rate\": 3.123567986094651e-05, \"epoch\": 0.7505728055621396, \"step\": 95000}\n",
            "{\"loss\": 2.5444050601528723, \"learning_rate\": 3.113692028126728e-05, \"epoch\": 0.7545231887493087, \"step\": 95500}\n",
            "{\"loss\": 2.483307492894586, \"learning_rate\": 3.1038160701588054e-05, \"epoch\": 0.7584735719364778, \"step\": 96000}\n",
            "{\"loss\": 2.43630068562564, \"learning_rate\": 3.093940112190883e-05, \"epoch\": 0.762423955123647, \"step\": 96500}\n",
            "{\"loss\": 2.524785908180871, \"learning_rate\": 3.08406415422296e-05, \"epoch\": 0.7663743383108161, \"step\": 97000}\n",
            "{\"loss\": 2.556490939619951, \"learning_rate\": 3.0741881962550365e-05, \"epoch\": 0.7703247214979853, \"step\": 97500}\n",
            "{\"loss\": 2.635647827567998, \"learning_rate\": 3.064312238287114e-05, \"epoch\": 0.7742751046851545, \"step\": 98000}\n",
            "{\"loss\": 2.641541033706162, \"learning_rate\": 3.054436280319191e-05, \"epoch\": 0.7782254878723236, \"step\": 98500}\n",
            "{\"loss\": 2.6145727896573954, \"learning_rate\": 3.044560322351268e-05, \"epoch\": 0.7821758710594928, \"step\": 99000}\n",
            "{\"loss\": 2.601786660960643, \"learning_rate\": 3.0346843643833456e-05, \"epoch\": 0.7861262542466619, \"step\": 99500}\n",
            "{\"loss\": 2.610073220228078, \"learning_rate\": 3.0248084064154223e-05, \"epoch\": 0.790076637433831, \"step\": 100000}\n",
            "{\"loss\": 2.641134526169044, \"learning_rate\": 3.0149324484474994e-05, \"epoch\": 0.7940270206210003, \"step\": 100500}\n",
            "{\"loss\": 2.5176630783367435, \"learning_rate\": 3.005056490479577e-05, \"epoch\": 0.7979774038081694, \"step\": 101000}\n",
            "{\"loss\": 2.508213760421262, \"learning_rate\": 2.9951805325116538e-05, \"epoch\": 0.8019277869953385, \"step\": 101500}\n",
            "{\"loss\": 2.5700921985968015, \"learning_rate\": 2.985304574543731e-05, \"epoch\": 0.8058781701825077, \"step\": 102000}\n",
            "{\"loss\": 2.4821639169112313, \"learning_rate\": 2.975428616575808e-05, \"epoch\": 0.8098285533696769, \"step\": 102500}\n",
            "{\"loss\": 2.7269332332692575, \"learning_rate\": 2.9655526586078853e-05, \"epoch\": 0.813778936556846, \"step\": 103000}\n",
            "{\"loss\": 2.5595454098747576, \"learning_rate\": 2.9556767006399623e-05, \"epoch\": 0.8177293197440152, \"step\": 103500}\n",
            "{\"loss\": 2.5901765798357084, \"learning_rate\": 2.945800742672039e-05, \"epoch\": 0.8216797029311843, \"step\": 104000}\n",
            "{\"loss\": 2.497085843809298, \"learning_rate\": 2.9359247847041167e-05, \"epoch\": 0.8256300861183535, \"step\": 104500}\n",
            "{\"loss\": 2.553537710214965, \"learning_rate\": 2.9260488267361934e-05, \"epoch\": 0.8295804693055226, \"step\": 105000}\n",
            "{\"loss\": 2.559274927394581, \"learning_rate\": 2.9161728687682705e-05, \"epoch\": 0.8335308524926918, \"step\": 105500}\n",
            "{\"loss\": 2.744837716000737, \"learning_rate\": 2.906296910800348e-05, \"epoch\": 0.837481235679861, \"step\": 106000}\n",
            "{\"loss\": 2.460783166519017, \"learning_rate\": 2.896420952832425e-05, \"epoch\": 0.8414316188670301, \"step\": 106500}\n",
            "{\"loss\": 2.6272504481654613, \"learning_rate\": 2.886544994864502e-05, \"epoch\": 0.8453820020541992, \"step\": 107000}\n",
            "{\"loss\": 2.6064573623198086, \"learning_rate\": 2.8766690368965793e-05, \"epoch\": 0.8493323852413684, \"step\": 107500}\n",
            "{\"loss\": 2.342772400524584, \"learning_rate\": 2.8667930789286563e-05, \"epoch\": 0.8532827684285376, \"step\": 108000}\n",
            "{\"loss\": 2.5984615663817383, \"learning_rate\": 2.856917120960733e-05, \"epoch\": 0.8572331516157067, \"step\": 108500}\n",
            "{\"loss\": 2.536856712748879, \"learning_rate\": 2.8470411629928107e-05, \"epoch\": 0.8611835348028759, \"step\": 109000}\n",
            "{\"loss\": 2.5435850747020448, \"learning_rate\": 2.8371652050248874e-05, \"epoch\": 0.865133917990045, \"step\": 109500}\n",
            "{\"loss\": 2.528700555966934, \"learning_rate\": 2.8272892470569645e-05, \"epoch\": 0.8690843011772141, \"step\": 110000}\n",
            "{\"loss\": 2.4081329553080724, \"learning_rate\": 2.817413289089042e-05, \"epoch\": 0.8730346843643834, \"step\": 110500}\n",
            "{\"loss\": 2.4176169568711194, \"learning_rate\": 2.807537331121119e-05, \"epoch\": 0.8769850675515525, \"step\": 111000}\n",
            "{\"loss\": 2.5518548266178693, \"learning_rate\": 2.797661373153196e-05, \"epoch\": 0.8809354507387217, \"step\": 111500}\n",
            "{\"loss\": 2.525234936630004, \"learning_rate\": 2.7877854151852733e-05, \"epoch\": 0.8848858339258908, \"step\": 112000}\n",
            "{\"loss\": 2.636305126886, \"learning_rate\": 2.7779094572173504e-05, \"epoch\": 0.8888362171130599, \"step\": 112500}\n",
            "{\"loss\": 2.515446968986071, \"learning_rate\": 2.768033499249427e-05, \"epoch\": 0.8927866003002292, \"step\": 113000}\n",
            "{\"loss\": 2.507912656565895, \"learning_rate\": 2.758157541281504e-05, \"epoch\": 0.8967369834873983, \"step\": 113500}\n",
            "{\"loss\": 2.450785272857058, \"learning_rate\": 2.7482815833135818e-05, \"epoch\": 0.9006873666745674, \"step\": 114000}\n",
            "{\"loss\": 2.549659381629666, \"learning_rate\": 2.7384056253456585e-05, \"epoch\": 0.9046377498617366, \"step\": 114500}\n",
            "{\"loss\": 2.4175343140813057, \"learning_rate\": 2.7285296673777356e-05, \"epoch\": 0.9085881330489057, \"step\": 115000}\n",
            "{\"loss\": 2.45594966934924, \"learning_rate\": 2.718653709409813e-05, \"epoch\": 0.9125385162360748, \"step\": 115500}\n",
            "{\"loss\": 2.520881127016852, \"learning_rate\": 2.70877775144189e-05, \"epoch\": 0.9164888994232441, \"step\": 116000}\n",
            "{\"loss\": 2.3610649109855295, \"learning_rate\": 2.698901793473967e-05, \"epoch\": 0.9204392826104132, \"step\": 116500}\n",
            "{\"loss\": 2.5327254416446667, \"learning_rate\": 2.6890258355060444e-05, \"epoch\": 0.9243896657975824, \"step\": 117000}\n",
            "{\"loss\": 2.5288216578457505, \"learning_rate\": 2.6791498775381214e-05, \"epoch\": 0.9283400489847515, \"step\": 117500}\n",
            "{\"loss\": 2.5762064391421156, \"learning_rate\": 2.669273919570198e-05, \"epoch\": 0.9322904321719206, \"step\": 118000}\n",
            "{\"loss\": 2.4219354780757567, \"learning_rate\": 2.659397961602276e-05, \"epoch\": 0.9362408153590899, \"step\": 118500}\n",
            "{\"loss\": 2.5160958767253905, \"learning_rate\": 2.6495220036343526e-05, \"epoch\": 0.940191198546259, \"step\": 119000}\n",
            "{\"loss\": 2.4968084314198933, \"learning_rate\": 2.6396460456664296e-05, \"epoch\": 0.9441415817334281, \"step\": 119500}\n",
            "{\"loss\": 2.4934675935923587, \"learning_rate\": 2.629770087698507e-05, \"epoch\": 0.9480919649205973, \"step\": 120000}\n",
            "{\"loss\": 2.5540963805824286, \"learning_rate\": 2.619894129730584e-05, \"epoch\": 0.9520423481077664, \"step\": 120500}\n",
            "{\"loss\": 2.6349887417531574, \"learning_rate\": 2.610018171762661e-05, \"epoch\": 0.9559927312949356, \"step\": 121000}\n",
            "{\"loss\": 2.6047573568496154, \"learning_rate\": 2.6001422137947384e-05, \"epoch\": 0.9599431144821048, \"step\": 121500}\n",
            "{\"loss\": 2.3650614000821952, \"learning_rate\": 2.5902662558268155e-05, \"epoch\": 0.9638934976692739, \"step\": 122000}\n",
            "{\"loss\": 2.592711076547275, \"learning_rate\": 2.580390297858892e-05, \"epoch\": 0.9678438808564431, \"step\": 122500}\n",
            "{\"loss\": 2.4959875136070186, \"learning_rate\": 2.57051433989097e-05, \"epoch\": 0.9717942640436122, \"step\": 123000}\n",
            "{\"loss\": 2.5743278265718836, \"learning_rate\": 2.5606383819230466e-05, \"epoch\": 0.9757446472307814, \"step\": 123500}\n",
            "{\"loss\": 2.4629038603793596, \"learning_rate\": 2.5507624239551236e-05, \"epoch\": 0.9796950304179506, \"step\": 124000}\n",
            "{\"loss\": 2.507595943594701, \"learning_rate\": 2.540886465987201e-05, \"epoch\": 0.9836454136051197, \"step\": 124500}\n",
            "{\"loss\": 2.4574152587489224, \"learning_rate\": 2.531010508019278e-05, \"epoch\": 0.9875957967922888, \"step\": 125000}\n",
            "{\"loss\": 2.365709158676444, \"learning_rate\": 2.521134550051355e-05, \"epoch\": 0.991546179979458, \"step\": 125500}\n",
            "{\"loss\": 2.4661180251082406, \"learning_rate\": 2.5112585920834318e-05, \"epoch\": 0.9954965631666272, \"step\": 126000}\n",
            "{\"loss\": 2.3851188067101177, \"learning_rate\": 2.5013826341155095e-05, \"epoch\": 0.9994469463537963, \"step\": 126500}\n",
            "{\"loss\": 2.366923461256083, \"learning_rate\": 2.4915066761475865e-05, \"epoch\": 1.0033973295409655, \"step\": 127000}\n",
            "{\"loss\": 2.291158866269281, \"learning_rate\": 2.4816307181796636e-05, \"epoch\": 1.0073477127281347, \"step\": 127500}\n",
            "{\"loss\": 2.404655659801676, \"learning_rate\": 2.471754760211741e-05, \"epoch\": 1.0112980959153037, \"step\": 128000}\n",
            "{\"loss\": 2.3699409196658525, \"learning_rate\": 2.4618788022438177e-05, \"epoch\": 1.015248479102473, \"step\": 128500}\n",
            "{\"loss\": 2.421523323475616, \"learning_rate\": 2.452002844275895e-05, \"epoch\": 1.0191988622896422, \"step\": 129000}\n",
            "{\"loss\": 2.2987037485921755, \"learning_rate\": 2.4421268863079717e-05, \"epoch\": 1.0231492454768112, \"step\": 129500}\n",
            "{\"loss\": 2.5106960814206394, \"learning_rate\": 2.432250928340049e-05, \"epoch\": 1.0270996286639804, \"step\": 130000}\n",
            "{\"loss\": 2.6093366662173065, \"learning_rate\": 2.422374970372126e-05, \"epoch\": 1.0310500118511496, \"step\": 130500}\n",
            "{\"loss\": 2.451072645873763, \"learning_rate\": 2.4124990124042032e-05, \"epoch\": 1.0350003950383186, \"step\": 131000}\n",
            "{\"loss\": 2.5329149361808088, \"learning_rate\": 2.4026230544362806e-05, \"epoch\": 1.0389507782254879, \"step\": 131500}\n",
            "{\"loss\": 2.5111382078161695, \"learning_rate\": 2.3927470964683576e-05, \"epoch\": 1.042901161412657, \"step\": 132000}\n",
            "{\"loss\": 2.5360070299040527, \"learning_rate\": 2.3828711385004346e-05, \"epoch\": 1.046851544599826, \"step\": 132500}\n",
            "{\"loss\": 2.3892052019443364, \"learning_rate\": 2.3729951805325117e-05, \"epoch\": 1.0508019277869953, \"step\": 133000}\n",
            "{\"loss\": 2.3961676307081943, \"learning_rate\": 2.363119222564589e-05, \"epoch\": 1.0547523109741646, \"step\": 133500}\n",
            "{\"loss\": 2.5243916774221, \"learning_rate\": 2.353243264596666e-05, \"epoch\": 1.0587026941613336, \"step\": 134000}\n",
            "{\"loss\": 2.522844349779887, \"learning_rate\": 2.343367306628743e-05, \"epoch\": 1.0626530773485028, \"step\": 134500}\n",
            "{\"loss\": 2.4277219994916814, \"learning_rate\": 2.3334913486608202e-05, \"epoch\": 1.066603460535672, \"step\": 135000}\n",
            "{\"loss\": 2.370716626425856, \"learning_rate\": 2.3236153906928972e-05, \"epoch\": 1.070553843722841, \"step\": 135500}\n",
            "{\"loss\": 2.504712550081429, \"learning_rate\": 2.3137394327249746e-05, \"epoch\": 1.0745042269100102, \"step\": 136000}\n",
            "{\"loss\": 2.2966428938169265, \"learning_rate\": 2.3038634747570513e-05, \"epoch\": 1.0784546100971795, \"step\": 136500}\n",
            "{\"loss\": 2.4949071309965802, \"learning_rate\": 2.2939875167891287e-05, \"epoch\": 1.0824049932843485, \"step\": 137000}\n",
            "{\"loss\": 2.3925731664118355, \"learning_rate\": 2.2841115588212057e-05, \"epoch\": 1.0863553764715177, \"step\": 137500}\n",
            "{\"loss\": 2.2869361601767597, \"learning_rate\": 2.2742356008532828e-05, \"epoch\": 1.090305759658687, \"step\": 138000}\n",
            "{\"loss\": 2.3418769765258767, \"learning_rate\": 2.26435964288536e-05, \"epoch\": 1.0942561428458562, \"step\": 138500}\n",
            "{\"loss\": 2.2983080207942983, \"learning_rate\": 2.2544836849174372e-05, \"epoch\": 1.0982065260330252, \"step\": 139000}\n",
            "{\"loss\": 2.5094925095206126, \"learning_rate\": 2.2446077269495142e-05, \"epoch\": 1.1021569092201944, \"step\": 139500}\n",
            "{\"loss\": 2.365111180587439, \"learning_rate\": 2.2347317689815913e-05, \"epoch\": 1.1061072924073636, \"step\": 140000}\n",
            "{\"loss\": 2.4125226708982375, \"learning_rate\": 2.2248558110136683e-05, \"epoch\": 1.1100576755945326, \"step\": 140500}\n",
            "{\"loss\": 2.3596947922367835, \"learning_rate\": 2.2149798530457457e-05, \"epoch\": 1.1140080587817018, \"step\": 141000}\n",
            "{\"loss\": 2.5209655638675903, \"learning_rate\": 2.2051038950778227e-05, \"epoch\": 1.117958441968871, \"step\": 141500}\n",
            "{\"loss\": 2.4491515667362838, \"learning_rate\": 2.1952279371098998e-05, \"epoch\": 1.12190882515604, \"step\": 142000}\n",
            "{\"loss\": 2.4704814074635504, \"learning_rate\": 2.1853519791419768e-05, \"epoch\": 1.1258592083432093, \"step\": 142500}\n",
            "{\"loss\": 2.382719478946412, \"learning_rate\": 2.175476021174054e-05, \"epoch\": 1.1298095915303785, \"step\": 143000}\n",
            "{\"loss\": 2.4052311274203237, \"learning_rate\": 2.165600063206131e-05, \"epoch\": 1.1337599747175475, \"step\": 143500}\n",
            "{\"loss\": 2.488868587736739, \"learning_rate\": 2.1557241052382082e-05, \"epoch\": 1.1377103579047168, \"step\": 144000}\n",
            "{\"loss\": 2.4140036366331623, \"learning_rate\": 2.1458481472702853e-05, \"epoch\": 1.141660741091886, \"step\": 144500}\n",
            "{\"loss\": 2.4101151229913813, \"learning_rate\": 2.1359721893023623e-05, \"epoch\": 1.145611124279055, \"step\": 145000}\n",
            "{\"loss\": 2.2498216859833335, \"learning_rate\": 2.1260962313344397e-05, \"epoch\": 1.1495615074662242, \"step\": 145500}\n",
            "{\"loss\": 2.290047091628192, \"learning_rate\": 2.1162202733665164e-05, \"epoch\": 1.1535118906533934, \"step\": 146000}\n",
            "{\"loss\": 2.4513703243797647, \"learning_rate\": 2.1063443153985938e-05, \"epoch\": 1.1574622738405624, \"step\": 146500}\n",
            "{\"loss\": 2.4134167881318134, \"learning_rate\": 2.0964683574306708e-05, \"epoch\": 1.1614126570277317, \"step\": 147000}\n",
            "{\"loss\": 2.3081428992764557, \"learning_rate\": 2.086592399462748e-05, \"epoch\": 1.165363040214901, \"step\": 147500}\n",
            "{\"loss\": 2.237201063715853, \"learning_rate\": 2.0767164414948252e-05, \"epoch\": 1.1693134234020701, \"step\": 148000}\n",
            "{\"loss\": 2.4649053666356484, \"learning_rate\": 2.0668404835269023e-05, \"epoch\": 1.1732638065892391, \"step\": 148500}\n",
            "{\"loss\": 2.3681834181244485, \"learning_rate\": 2.0569645255589793e-05, \"epoch\": 1.1772141897764083, \"step\": 149000}\n",
            "{\"loss\": 2.486252764441655, \"learning_rate\": 2.0470885675910564e-05, \"epoch\": 1.1811645729635774, \"step\": 149500}\n",
            "{\"loss\": 2.4420453052588273, \"learning_rate\": 2.0372126096231337e-05, \"epoch\": 1.1851149561507466, \"step\": 150000}\n",
            "{\"loss\": 2.2717950030072824, \"learning_rate\": 2.0273366516552104e-05, \"epoch\": 1.1890653393379158, \"step\": 150500}\n",
            "{\"loss\": 2.2358960803101073, \"learning_rate\": 2.0174606936872878e-05, \"epoch\": 1.193015722525085, \"step\": 151000}\n",
            "{\"loss\": 2.419551307009882, \"learning_rate\": 2.007584735719365e-05, \"epoch\": 1.196966105712254, \"step\": 151500}\n",
            "{\"loss\": 2.307371934231836, \"learning_rate\": 1.997708777751442e-05, \"epoch\": 1.2009164888994233, \"step\": 152000}\n",
            "{\"loss\": 2.319561101732659, \"learning_rate\": 1.9878328197835193e-05, \"epoch\": 1.2048668720865925, \"step\": 152500}\n",
            "{\"loss\": 2.3357058376247295, \"learning_rate\": 1.977956861815596e-05, \"epoch\": 1.2088172552737615, \"step\": 153000}\n",
            "{\"loss\": 2.4161182693068404, \"learning_rate\": 1.9680809038476734e-05, \"epoch\": 1.2127676384609307, \"step\": 153500}\n",
            "{\"loss\": 2.3374859005250035, \"learning_rate\": 1.9582049458797504e-05, \"epoch\": 1.2167180216481, \"step\": 154000}\n",
            "{\"loss\": 2.5291473101135344, \"learning_rate\": 1.9483289879118274e-05, \"epoch\": 1.220668404835269, \"step\": 154500}\n",
            "{\"loss\": 2.380177289009094, \"learning_rate\": 1.9384530299439048e-05, \"epoch\": 1.2246187880224382, \"step\": 155000}\n",
            "{\"loss\": 2.299456073255744, \"learning_rate\": 1.928577071975982e-05, \"epoch\": 1.2285691712096074, \"step\": 155500}\n",
            "{\"loss\": 2.271358358401107, \"learning_rate\": 1.918701114008059e-05, \"epoch\": 1.2325195543967764, \"step\": 156000}\n",
            "{\"loss\": 2.4366219257771737, \"learning_rate\": 1.908825156040136e-05, \"epoch\": 1.2364699375839456, \"step\": 156500}\n",
            "{\"loss\": 2.3575907371923095, \"learning_rate\": 1.8989491980722133e-05, \"epoch\": 1.2404203207711149, \"step\": 157000}\n",
            "{\"loss\": 2.430109200156992, \"learning_rate\": 1.88907324010429e-05, \"epoch\": 1.2443707039582839, \"step\": 157500}\n",
            "{\"loss\": 2.1938178988664876, \"learning_rate\": 1.8791972821363674e-05, \"epoch\": 1.248321087145453, \"step\": 158000}\n",
            "{\"loss\": 2.451648552888422, \"learning_rate\": 1.8693213241684444e-05, \"epoch\": 1.2522714703326223, \"step\": 158500}\n",
            "{\"loss\": 2.3671753072141435, \"learning_rate\": 1.8594453662005215e-05, \"epoch\": 1.2562218535197913, \"step\": 159000}\n",
            "{\"loss\": 2.356927789664012, \"learning_rate\": 1.849569408232599e-05, \"epoch\": 1.2601722367069605, \"step\": 159500}\n",
            "{\"loss\": 2.341630360955838, \"learning_rate\": 1.8396934502646755e-05, \"epoch\": 1.2641226198941298, \"step\": 160000}\n",
            "{\"loss\": 2.2360320592637875, \"learning_rate\": 1.829817492296753e-05, \"epoch\": 1.268073003081299, \"step\": 160500}\n",
            "{\"loss\": 2.258037007719744, \"learning_rate\": 1.81994153432883e-05, \"epoch\": 1.272023386268468, \"step\": 161000}\n",
            "{\"loss\": 2.235063086605165, \"learning_rate\": 1.810065576360907e-05, \"epoch\": 1.2759737694556372, \"step\": 161500}\n",
            "{\"loss\": 2.0950400355462917, \"learning_rate\": 1.8001896183929844e-05, \"epoch\": 1.2799241526428062, \"step\": 162000}\n",
            "{\"loss\": 2.2315762868183664, \"learning_rate\": 1.7903136604250614e-05, \"epoch\": 1.2838745358299755, \"step\": 162500}\n",
            "{\"loss\": 2.383962252751924, \"learning_rate\": 1.7804377024571385e-05, \"epoch\": 1.2878249190171447, \"step\": 163000}\n",
            "{\"loss\": 2.416592261115904, \"learning_rate\": 1.7705617444892155e-05, \"epoch\": 1.291775302204314, \"step\": 163500}\n",
            "{\"loss\": 2.383526426543016, \"learning_rate\": 1.7606857865212925e-05, \"epoch\": 1.295725685391483, \"step\": 164000}\n",
            "{\"loss\": 2.3402662961044114, \"learning_rate\": 1.7508098285533696e-05, \"epoch\": 1.2996760685786521, \"step\": 164500}\n",
            "{\"loss\": 2.205930517444969, \"learning_rate\": 1.740933870585447e-05, \"epoch\": 1.3036264517658211, \"step\": 165000}\n",
            "{\"loss\": 2.185459097011364, \"learning_rate\": 1.731057912617524e-05, \"epoch\": 1.3075768349529904, \"step\": 165500}\n",
            "{\"loss\": 2.1822560797139303, \"learning_rate\": 1.721181954649601e-05, \"epoch\": 1.3115272181401596, \"step\": 166000}\n",
            "{\"loss\": 2.2747855840779376, \"learning_rate\": 1.7113059966816784e-05, \"epoch\": 1.3154776013273288, \"step\": 166500}\n",
            "{\"loss\": 2.3166304792209993, \"learning_rate\": 1.701430038713755e-05, \"epoch\": 1.3194279845144978, \"step\": 167000}\n",
            "{\"loss\": 2.25883892254706, \"learning_rate\": 1.6915540807458325e-05, \"epoch\": 1.323378367701667, \"step\": 167500}\n",
            "{\"loss\": 2.316869961041957, \"learning_rate\": 1.6816781227779095e-05, \"epoch\": 1.3273287508888363, \"step\": 168000}\n",
            "{\"loss\": 2.32988773147366, \"learning_rate\": 1.6718021648099866e-05, \"epoch\": 1.3312791340760053, \"step\": 168500}\n",
            "{\"loss\": 2.3644229082659587, \"learning_rate\": 1.661926206842064e-05, \"epoch\": 1.3352295172631745, \"step\": 169000}\n",
            "{\"loss\": 2.3722959176797884, \"learning_rate\": 1.652050248874141e-05, \"epoch\": 1.3391799004503437, \"step\": 169500}\n",
            "{\"loss\": 2.2340956331618362, \"learning_rate\": 1.642174290906218e-05, \"epoch\": 1.343130283637513, \"step\": 170000}\n",
            "{\"loss\": 2.316941446033423, \"learning_rate\": 1.632298332938295e-05, \"epoch\": 1.347080666824682, \"step\": 170500}\n",
            "{\"loss\": 2.331887701037922, \"learning_rate\": 1.622422374970372e-05, \"epoch\": 1.3510310500118512, \"step\": 171000}\n",
            "{\"loss\": 2.4385871562694663, \"learning_rate\": 1.612546417002449e-05, \"epoch\": 1.3549814331990202, \"step\": 171500}\n",
            "{\"loss\": 2.3669860545358388, \"learning_rate\": 1.6026704590345265e-05, \"epoch\": 1.3589318163861894, \"step\": 172000}\n",
            "{\"loss\": 2.4379768844321372, \"learning_rate\": 1.5927945010666036e-05, \"epoch\": 1.3628821995733587, \"step\": 172500}\n",
            "{\"loss\": 2.2857468397228513, \"learning_rate\": 1.5829185430986806e-05, \"epoch\": 1.3668325827605279, \"step\": 173000}\n",
            "{\"loss\": 2.3768208482459885, \"learning_rate\": 1.573042585130758e-05, \"epoch\": 1.3707829659476969, \"step\": 173500}\n",
            "{\"loss\": 2.2133642779153306, \"learning_rate\": 1.5631666271628347e-05, \"epoch\": 1.3747333491348661, \"step\": 174000}\n",
            "{\"loss\": 2.298791284228908, \"learning_rate\": 1.553290669194912e-05, \"epoch\": 1.3786837323220351, \"step\": 174500}\n",
            "{\"loss\": 2.163070665454259, \"learning_rate\": 1.543414711226989e-05, \"epoch\": 1.3826341155092043, \"step\": 175000}\n",
            "{\"loss\": 2.211237366236979, \"learning_rate\": 1.533538753259066e-05, \"epoch\": 1.3865844986963736, \"step\": 175500}\n",
            "{\"loss\": 2.2118681682276073, \"learning_rate\": 1.5236627952911433e-05, \"epoch\": 1.3905348818835428, \"step\": 176000}\n",
            "{\"loss\": 2.1344814662701683, \"learning_rate\": 1.5137868373232204e-05, \"epoch\": 1.3944852650707118, \"step\": 176500}\n",
            "{\"loss\": 2.3646715195568975, \"learning_rate\": 1.5039108793552976e-05, \"epoch\": 1.398435648257881, \"step\": 177000}\n",
            "{\"loss\": 2.2693099171677606, \"learning_rate\": 1.4940349213873748e-05, \"epoch\": 1.4023860314450503, \"step\": 177500}\n",
            "{\"loss\": 2.1157009785057745, \"learning_rate\": 1.4841589634194517e-05, \"epoch\": 1.4063364146322193, \"step\": 178000}\n",
            "{\"loss\": 2.26128002939187, \"learning_rate\": 1.4742830054515289e-05, \"epoch\": 1.4102867978193885, \"step\": 178500}\n",
            "{\"loss\": 2.3280169006075013, \"learning_rate\": 1.4644070474836061e-05, \"epoch\": 1.4142371810065577, \"step\": 179000}\n",
            "{\"loss\": 2.3219789087403333, \"learning_rate\": 1.454531089515683e-05, \"epoch\": 1.4181875641937267, \"step\": 179500}\n",
            "{\"loss\": 2.128194275889662, \"learning_rate\": 1.4446551315477602e-05, \"epoch\": 1.422137947380896, \"step\": 180000}\n",
            "{\"loss\": 2.316859343964257, \"learning_rate\": 1.4347791735798374e-05, \"epoch\": 1.4260883305680652, \"step\": 180500}\n",
            "{\"loss\": 2.2587480404346714, \"learning_rate\": 1.4249032156119144e-05, \"epoch\": 1.4300387137552342, \"step\": 181000}\n",
            "{\"loss\": 2.4726848640937824, \"learning_rate\": 1.4150272576439916e-05, \"epoch\": 1.4339890969424034, \"step\": 181500}\n",
            "{\"loss\": 2.1661747673057254, \"learning_rate\": 1.4051512996760685e-05, \"epoch\": 1.4379394801295726, \"step\": 182000}\n",
            "{\"loss\": 2.387141925258911, \"learning_rate\": 1.3952753417081457e-05, \"epoch\": 1.4418898633167418, \"step\": 182500}\n",
            "{\"loss\": 2.2761098129021704, \"learning_rate\": 1.3853993837402229e-05, \"epoch\": 1.4458402465039109, \"step\": 183000}\n",
            "{\"loss\": 2.2914703285739524, \"learning_rate\": 1.3755234257723e-05, \"epoch\": 1.44979062969108, \"step\": 183500}\n",
            "{\"loss\": 2.425301471309154, \"learning_rate\": 1.3656474678043772e-05, \"epoch\": 1.453741012878249, \"step\": 184000}\n",
            "{\"loss\": 2.175819986001705, \"learning_rate\": 1.3557715098364544e-05, \"epoch\": 1.4576913960654183, \"step\": 184500}\n",
            "{\"loss\": 2.378743868199992, \"learning_rate\": 1.3458955518685312e-05, \"epoch\": 1.4616417792525875, \"step\": 185000}\n",
            "{\"loss\": 2.3569702279110207, \"learning_rate\": 1.3360195939006084e-05, \"epoch\": 1.4655921624397568, \"step\": 185500}\n",
            "{\"loss\": 2.2739528321366524, \"learning_rate\": 1.3261436359326857e-05, \"epoch\": 1.4695425456269258, \"step\": 186000}\n",
            "{\"loss\": 2.2202660497213946, \"learning_rate\": 1.3162676779647625e-05, \"epoch\": 1.473492928814095, \"step\": 186500}\n",
            "{\"loss\": 2.194291556180222, \"learning_rate\": 1.3063917199968397e-05, \"epoch\": 1.477443312001264, \"step\": 187000}\n",
            "{\"loss\": 2.327782355729956, \"learning_rate\": 1.296515762028917e-05, \"epoch\": 1.4813936951884332, \"step\": 187500}\n",
            "{\"loss\": 2.1383336560192983, \"learning_rate\": 1.286639804060994e-05, \"epoch\": 1.4853440783756025, \"step\": 188000}\n",
            "{\"loss\": 2.1067519095230383, \"learning_rate\": 1.2767638460930712e-05, \"epoch\": 1.4892944615627717, \"step\": 188500}\n",
            "{\"loss\": 2.248177884191624, \"learning_rate\": 1.266887888125148e-05, \"epoch\": 1.4932448447499407, \"step\": 189000}\n",
            "{\"loss\": 2.187677398086875, \"learning_rate\": 1.2570119301572253e-05, \"epoch\": 1.49719522793711, \"step\": 189500}\n",
            "{\"loss\": 2.230928117931355, \"learning_rate\": 1.2471359721893023e-05, \"epoch\": 1.501145611124279, \"step\": 190000}\n",
            "{\"loss\": 2.2939154928777135, \"learning_rate\": 1.2372600142213795e-05, \"epoch\": 1.5050959943114481, \"step\": 190500}\n",
            "{\"loss\": 2.300916045904858, \"learning_rate\": 1.2273840562534567e-05, \"epoch\": 1.5090463774986174, \"step\": 191000}\n",
            "{\"loss\": 2.1806494728797117, \"learning_rate\": 1.2175080982855338e-05, \"epoch\": 1.5129967606857866, \"step\": 191500}\n",
            "{\"loss\": 2.194707444301457, \"learning_rate\": 1.207632140317611e-05, \"epoch\": 1.5169471438729558, \"step\": 192000}\n",
            "{\"loss\": 2.2875531788264634, \"learning_rate\": 1.197756182349688e-05, \"epoch\": 1.5208975270601248, \"step\": 192500}\n",
            "{\"loss\": 2.1943395520321793, \"learning_rate\": 1.187880224381765e-05, \"epoch\": 1.5248479102472938, \"step\": 193000}\n",
            "{\"loss\": 2.1448214491013204, \"learning_rate\": 1.1780042664138421e-05, \"epoch\": 1.528798293434463, \"step\": 193500}\n",
            "{\"loss\": 2.1759470328646713, \"learning_rate\": 1.1681283084459193e-05, \"epoch\": 1.5327486766216323, \"step\": 194000}\n",
            "{\"loss\": 2.292906208711094, \"learning_rate\": 1.1582523504779965e-05, \"epoch\": 1.5366990598088015, \"step\": 194500}\n",
            "{\"loss\": 2.207729184205411, \"learning_rate\": 1.1483763925100735e-05, \"epoch\": 1.5406494429959707, \"step\": 195000}\n",
            "{\"loss\": 2.113204082692857, \"learning_rate\": 1.1385004345421506e-05, \"epoch\": 1.5445998261831397, \"step\": 195500}\n",
            "{\"loss\": 2.345252431985573, \"learning_rate\": 1.1286244765742278e-05, \"epoch\": 1.548550209370309, \"step\": 196000}\n",
            "{\"loss\": 2.050717592463712, \"learning_rate\": 1.1187485186063048e-05, \"epoch\": 1.552500592557478, \"step\": 196500}\n",
            "{\"loss\": 2.0622613542816834, \"learning_rate\": 1.1088725606383819e-05, \"epoch\": 1.5564509757446472, \"step\": 197000}\n",
            "{\"loss\": 2.1864498089695115, \"learning_rate\": 1.098996602670459e-05, \"epoch\": 1.5604013589318164, \"step\": 197500}\n",
            "{\"loss\": 2.2117381851000246, \"learning_rate\": 1.0891206447025363e-05, \"epoch\": 1.5643517421189856, \"step\": 198000}\n",
            "{\"loss\": 2.3156211227419554, \"learning_rate\": 1.0792446867346133e-05, \"epoch\": 1.5683021253061546, \"step\": 198500}\n",
            "{\"loss\": 2.344507934153662, \"learning_rate\": 1.0693687287666904e-05, \"epoch\": 1.5722525084933239, \"step\": 199000}\n",
            "{\"loss\": 2.0785894501500297, \"learning_rate\": 1.0594927707987676e-05, \"epoch\": 1.5762028916804929, \"step\": 199500}\n",
            "{\"loss\": 2.2354357034181014, \"learning_rate\": 1.0496168128308446e-05, \"epoch\": 1.580153274867662, \"step\": 200000}\n",
            "{\"loss\": 2.253999516487005, \"learning_rate\": 1.0397408548629217e-05, \"epoch\": 1.5841036580548313, \"step\": 200500}\n",
            "{\"loss\": 2.318305138313677, \"learning_rate\": 1.0298648968949989e-05, \"epoch\": 1.5880540412420006, \"step\": 201000}\n",
            "{\"loss\": 2.22658238423639, \"learning_rate\": 1.019988938927076e-05, \"epoch\": 1.5920044244291698, \"step\": 201500}\n",
            "{\"loss\": 2.0853195174989523, \"learning_rate\": 1.0101129809591531e-05, \"epoch\": 1.5959548076163388, \"step\": 202000}\n",
            "{\"loss\": 2.2763058036370203, \"learning_rate\": 1.0002370229912302e-05, \"epoch\": 1.5999051908035078, \"step\": 202500}\n",
            "{\"loss\": 2.1218612121997866, \"learning_rate\": 9.903610650233074e-06, \"epoch\": 1.603855573990677, \"step\": 203000}\n",
            "{\"loss\": 2.106968766754144, \"learning_rate\": 9.804851070553844e-06, \"epoch\": 1.6078059571778462, \"step\": 203500}\n",
            "{\"loss\": 2.207810270186048, \"learning_rate\": 9.706091490874614e-06, \"epoch\": 1.6117563403650155, \"step\": 204000}\n",
            "{\"loss\": 2.068290205943864, \"learning_rate\": 9.607331911195387e-06, \"epoch\": 1.6157067235521847, \"step\": 204500}\n",
            "{\"loss\": 2.302594452062156, \"learning_rate\": 9.508572331516159e-06, \"epoch\": 1.6196571067393537, \"step\": 205000}\n",
            "{\"loss\": 2.1526945471563375, \"learning_rate\": 9.409812751836929e-06, \"epoch\": 1.6236074899265227, \"step\": 205500}\n",
            "{\"loss\": 2.160462302379077, \"learning_rate\": 9.3110531721577e-06, \"epoch\": 1.627557873113692, \"step\": 206000}\n",
            "{\"loss\": 2.2973185492337214, \"learning_rate\": 9.212293592478471e-06, \"epoch\": 1.6315082563008612, \"step\": 206500}\n",
            "{\"loss\": 2.0825212610880843, \"learning_rate\": 9.113534012799242e-06, \"epoch\": 1.6354586394880304, \"step\": 207000}\n",
            "{\"loss\": 2.016119079251774, \"learning_rate\": 9.014774433120012e-06, \"epoch\": 1.6394090226751996, \"step\": 207500}\n",
            "{\"loss\": 2.2665825042030776, \"learning_rate\": 8.916014853440784e-06, \"epoch\": 1.6433594058623686, \"step\": 208000}\n",
            "{\"loss\": 2.1512622337632346, \"learning_rate\": 8.817255273761556e-06, \"epoch\": 1.6473097890495378, \"step\": 208500}\n",
            "{\"loss\": 2.2594429273952263, \"learning_rate\": 8.718495694082327e-06, \"epoch\": 1.6512601722367068, \"step\": 209000}\n",
            "{\"loss\": 2.2312996281778905, \"learning_rate\": 8.619736114403097e-06, \"epoch\": 1.655210555423876, \"step\": 209500}\n",
            "{\"loss\": 2.1343591839999427, \"learning_rate\": 8.52097653472387e-06, \"epoch\": 1.6591609386110453, \"step\": 210000}\n",
            "{\"loss\": 2.223950414756313, \"learning_rate\": 8.42221695504464e-06, \"epoch\": 1.6631113217982145, \"step\": 210500}\n",
            "{\"loss\": 2.2054204164277764, \"learning_rate\": 8.32345737536541e-06, \"epoch\": 1.6670617049853835, \"step\": 211000}\n",
            "{\"loss\": 2.0911421825992873, \"learning_rate\": 8.224697795686182e-06, \"epoch\": 1.6710120881725528, \"step\": 211500}\n",
            "{\"loss\": 2.0929686302188784, \"learning_rate\": 8.125938216006954e-06, \"epoch\": 1.6749624713597218, \"step\": 212000}\n",
            "{\"loss\": 2.145659804592142, \"learning_rate\": 8.027178636327725e-06, \"epoch\": 1.678912854546891, \"step\": 212500}\n",
            "{\"loss\": 2.201962039946811, \"learning_rate\": 7.928419056648495e-06, \"epoch\": 1.6828632377340602, \"step\": 213000}\n",
            "{\"loss\": 2.239692575152498, \"learning_rate\": 7.829659476969265e-06, \"epoch\": 1.6868136209212294, \"step\": 213500}\n",
            "{\"loss\": 2.1988512739385477, \"learning_rate\": 7.730899897290038e-06, \"epoch\": 1.6907640041083987, \"step\": 214000}\n",
            "{\"loss\": 2.090394895130536, \"learning_rate\": 7.632140317610808e-06, \"epoch\": 1.6947143872955677, \"step\": 214500}\n",
            "{\"loss\": 2.130400830048835, \"learning_rate\": 7.533380737931579e-06, \"epoch\": 1.6986647704827367, \"step\": 215000}\n",
            "{\"loss\": 2.1415296492120253, \"learning_rate\": 7.434621158252351e-06, \"epoch\": 1.702615153669906, \"step\": 215500}\n",
            "{\"loss\": 2.06122101310757, \"learning_rate\": 7.3358615785731225e-06, \"epoch\": 1.7065655368570751, \"step\": 216000}\n",
            "{\"loss\": 2.0195778457478153, \"learning_rate\": 7.237101998893893e-06, \"epoch\": 1.7105159200442444, \"step\": 216500}\n",
            "{\"loss\": 2.253486776552163, \"learning_rate\": 7.138342419214663e-06, \"epoch\": 1.7144663032314136, \"step\": 217000}\n",
            "{\"loss\": 2.158788146772189, \"learning_rate\": 7.039582839535435e-06, \"epoch\": 1.7184166864185826, \"step\": 217500}\n",
            "{\"loss\": 2.327574083995307, \"learning_rate\": 6.940823259856207e-06, \"epoch\": 1.7223670696057518, \"step\": 218000}\n",
            "{\"loss\": 2.1696503852410243, \"learning_rate\": 6.842063680176977e-06, \"epoch\": 1.7263174527929208, \"step\": 218500}\n",
            "{\"loss\": 1.9842761034828145, \"learning_rate\": 6.743304100497748e-06, \"epoch\": 1.73026783598009, \"step\": 219000}\n",
            "{\"loss\": 2.259596766509116, \"learning_rate\": 6.64454452081852e-06, \"epoch\": 1.7342182191672593, \"step\": 219500}\n",
            "{\"loss\": 2.2110490483790635, \"learning_rate\": 6.545784941139291e-06, \"epoch\": 1.7381686023544285, \"step\": 220000}\n",
            "{\"loss\": 2.174991591101978, \"learning_rate\": 6.447025361460061e-06, \"epoch\": 1.7421189855415975, \"step\": 220500}\n",
            "{\"loss\": 2.0388635169023184, \"learning_rate\": 6.348265781780833e-06, \"epoch\": 1.7460693687287667, \"step\": 221000}\n",
            "{\"loss\": 2.1823258927864955, \"learning_rate\": 6.2495062021016045e-06, \"epoch\": 1.7500197519159357, \"step\": 221500}\n",
            "{\"loss\": 2.268427816085052, \"learning_rate\": 6.150746622422375e-06, \"epoch\": 1.753970135103105, \"step\": 222000}\n",
            "{\"loss\": 2.024699495255947, \"learning_rate\": 6.051987042743146e-06, \"epoch\": 1.7579205182902742, \"step\": 222500}\n",
            "{\"loss\": 2.2481521329325624, \"learning_rate\": 5.953227463063917e-06, \"epoch\": 1.7618709014774434, \"step\": 223000}\n",
            "{\"loss\": 2.171037616644986, \"learning_rate\": 5.8544678833846886e-06, \"epoch\": 1.7658212846646126, \"step\": 223500}\n",
            "{\"loss\": 2.059644529737532, \"learning_rate\": 5.75570830370546e-06, \"epoch\": 1.7697716678517816, \"step\": 224000}\n",
            "{\"loss\": 2.025228266834747, \"learning_rate\": 5.656948724026231e-06, \"epoch\": 1.7737220510389506, \"step\": 224500}\n",
            "{\"loss\": 2.083240580908023, \"learning_rate\": 5.558189144347002e-06, \"epoch\": 1.7776724342261199, \"step\": 225000}\n",
            "{\"loss\": 2.2372176613528283, \"learning_rate\": 5.459429564667773e-06, \"epoch\": 1.781622817413289, \"step\": 225500}\n",
            "{\"loss\": 2.1545673003008123, \"learning_rate\": 5.360669984988544e-06, \"epoch\": 1.7855732006004583, \"step\": 226000}\n",
            "{\"loss\": 2.139720974325668, \"learning_rate\": 5.261910405309315e-06, \"epoch\": 1.7895235837876275, \"step\": 226500}\n",
            "{\"loss\": 2.167489742804086, \"learning_rate\": 5.163150825630086e-06, \"epoch\": 1.7934739669747966, \"step\": 227000}\n",
            "{\"loss\": 2.2429226984421256, \"learning_rate\": 5.064391245950857e-06, \"epoch\": 1.7974243501619656, \"step\": 227500}\n",
            "{\"loss\": 2.1674911846441685, \"learning_rate\": 4.965631666271629e-06, \"epoch\": 1.8013747333491348, \"step\": 228000}\n",
            "{\"loss\": 1.9835903220814652, \"learning_rate\": 4.8668720865924e-06, \"epoch\": 1.805325116536304, \"step\": 228500}\n",
            "{\"loss\": 2.086668462866452, \"learning_rate\": 4.7681125069131705e-06, \"epoch\": 1.8092754997234732, \"step\": 229000}\n",
            "{\"loss\": 2.10674355756538, \"learning_rate\": 4.669352927233942e-06, \"epoch\": 1.8132258829106425, \"step\": 229500}\n",
            "{\"loss\": 2.1246392310750672, \"learning_rate\": 4.570593347554713e-06, \"epoch\": 1.8171762660978115, \"step\": 230000}\n",
            "{\"loss\": 2.1395059757493438, \"learning_rate\": 4.471833767875484e-06, \"epoch\": 1.8211266492849807, \"step\": 230500}\n",
            "{\"loss\": 2.144453030739445, \"learning_rate\": 4.373074188196255e-06, \"epoch\": 1.8250770324721497, \"step\": 231000}\n",
            "{\"loss\": 2.0614322727508845, \"learning_rate\": 4.274314608517027e-06, \"epoch\": 1.829027415659319, \"step\": 231500}\n",
            "{\"loss\": 2.213131626553368, \"learning_rate\": 4.175555028837797e-06, \"epoch\": 1.8329777988464881, \"step\": 232000}\n",
            "{\"loss\": 1.9807002887472045, \"learning_rate\": 4.076795449158568e-06, \"epoch\": 1.8369281820336574, \"step\": 232500}\n",
            "{\"loss\": 2.192152676404454, \"learning_rate\": 3.97803586947934e-06, \"epoch\": 1.8408785652208264, \"step\": 233000}\n",
            "{\"loss\": 2.093590562196914, \"learning_rate\": 3.879276289800111e-06, \"epoch\": 1.8448289484079956, \"step\": 233500}\n",
            "{\"loss\": 2.1883398495027797, \"learning_rate\": 3.780516710120882e-06, \"epoch\": 1.8487793315951646, \"step\": 234000}\n",
            "{\"loss\": 2.0894771555704064, \"learning_rate\": 3.681757130441653e-06, \"epoch\": 1.8527297147823338, \"step\": 234500}\n",
            "{\"loss\": 2.24969777325145, \"learning_rate\": 3.582997550762424e-06, \"epoch\": 1.856680097969503, \"step\": 235000}\n",
            "{\"loss\": 2.0038924464308656, \"learning_rate\": 3.484237971083195e-06, \"epoch\": 1.8606304811566723, \"step\": 235500}\n",
            "{\"loss\": 2.124637153679505, \"learning_rate\": 3.3854783914039666e-06, \"epoch\": 1.8645808643438415, \"step\": 236000}\n",
            "{\"loss\": 2.127436412377516, \"learning_rate\": 3.286718811724737e-06, \"epoch\": 1.8685312475310105, \"step\": 236500}\n",
            "{\"loss\": 2.301473924522754, \"learning_rate\": 3.1879592320455087e-06, \"epoch\": 1.8724816307181795, \"step\": 237000}\n",
            "{\"loss\": 2.1127586819836868, \"learning_rate\": 3.0891996523662795e-06, \"epoch\": 1.8764320139053488, \"step\": 237500}\n",
            "{\"loss\": 2.0644409073654097, \"learning_rate\": 2.9904400726870507e-06, \"epoch\": 1.880382397092518, \"step\": 238000}\n",
            "{\"loss\": 2.025694973119069, \"learning_rate\": 2.891680493007822e-06, \"epoch\": 1.8843327802796872, \"step\": 238500}\n",
            "{\"loss\": 2.0631086525928697, \"learning_rate\": 2.792920913328593e-06, \"epoch\": 1.8882831634668564, \"step\": 239000}\n",
            "{\"loss\": 2.0373546670551876, \"learning_rate\": 2.694161333649364e-06, \"epoch\": 1.8922335466540254, \"step\": 239500}\n",
            "{\"loss\": 2.169262802575715, \"learning_rate\": 2.5954017539701353e-06, \"epoch\": 1.8961839298411944, \"step\": 240000}\n",
            "{\"loss\": 2.0147063178655227, \"learning_rate\": 2.4966421742909065e-06, \"epoch\": 1.9001343130283637, \"step\": 240500}\n",
            "{\"loss\": 2.136391996548511, \"learning_rate\": 2.3978825946116773e-06, \"epoch\": 1.904084696215533, \"step\": 241000}\n",
            "{\"loss\": 1.9616341807758435, \"learning_rate\": 2.2991230149324486e-06, \"epoch\": 1.9080350794027021, \"step\": 241500}\n",
            "{\"loss\": 2.0262828786643223, \"learning_rate\": 2.20036343525322e-06, \"epoch\": 1.9119854625898713, \"step\": 242000}\n",
            "{\"loss\": 2.124396244475385, \"learning_rate\": 2.1016038555739906e-06, \"epoch\": 1.9159358457770403, \"step\": 242500}\n",
            "{\"loss\": 2.0607473471406847, \"learning_rate\": 2.002844275894762e-06, \"epoch\": 1.9198862289642096, \"step\": 243000}\n",
            "{\"loss\": 2.0638399146748707, \"learning_rate\": 1.904084696215533e-06, \"epoch\": 1.9238366121513786, \"step\": 243500}\n",
            "{\"loss\": 2.038383716311306, \"learning_rate\": 1.805325116536304e-06, \"epoch\": 1.9277869953385478, \"step\": 244000}\n",
            "{\"loss\": 2.140926609135233, \"learning_rate\": 1.7065655368570754e-06, \"epoch\": 1.931737378525717, \"step\": 244500}\n",
            "{\"loss\": 2.0924636465145743, \"learning_rate\": 1.6078059571778464e-06, \"epoch\": 1.9356877617128863, \"step\": 245000}\n",
            "{\"loss\": 2.144259115424473, \"learning_rate\": 1.5090463774986174e-06, \"epoch\": 1.9396381449000553, \"step\": 245500}\n",
            "{\"loss\": 2.0421910843672233, \"learning_rate\": 1.4102867978193885e-06, \"epoch\": 1.9435885280872245, \"step\": 246000}\n",
            "{\"loss\": 2.060717963166069, \"learning_rate\": 1.3115272181401597e-06, \"epoch\": 1.9475389112743935, \"step\": 246500}\n",
            "{\"loss\": 2.074619374336442, \"learning_rate\": 1.2127676384609307e-06, \"epoch\": 1.9514892944615627, \"step\": 247000}\n",
            "{\"loss\": 2.2072574870050885, \"learning_rate\": 1.1140080587817018e-06, \"epoch\": 1.955439677648732, \"step\": 247500}\n",
            "{\"loss\": 1.9970858177319168, \"learning_rate\": 1.015248479102473e-06, \"epoch\": 1.9593900608359012, \"step\": 248000}\n",
            "{\"loss\": 2.1479498513182627, \"learning_rate\": 9.164888994232442e-07, \"epoch\": 1.9633404440230704, \"step\": 248500}\n",
            "{\"loss\": 2.0326881874585525, \"learning_rate\": 8.177293197440152e-07, \"epoch\": 1.9672908272102394, \"step\": 249000}\n",
            "{\"loss\": 2.041173490334302, \"learning_rate\": 7.189697400647863e-07, \"epoch\": 1.9712412103974084, \"step\": 249500}\n",
            "{\"loss\": 2.0727451135672164, \"learning_rate\": 6.202101603855575e-07, \"epoch\": 1.9751915935845776, \"step\": 250000}\n",
            "{\"loss\": 2.0372811126266606, \"learning_rate\": 5.214505807063286e-07, \"epoch\": 1.9791419767717469, \"step\": 250500}\n",
            "{\"loss\": 2.1711376363027375, \"learning_rate\": 4.2269100102709967e-07, \"epoch\": 1.983092359958916, \"step\": 251000}\n",
            "{\"loss\": 2.188680204695091, \"learning_rate\": 3.2393142134787075e-07, \"epoch\": 1.9870427431460853, \"step\": 251500}\n",
            "{\"loss\": 2.0330088819733354, \"learning_rate\": 2.2517184166864188e-07, \"epoch\": 1.9909931263332543, \"step\": 252000}\n",
            "{\"loss\": 2.013848945663776, \"learning_rate\": 1.2641226198941297e-07, \"epoch\": 1.9949435095204233, \"step\": 252500}\n",
            "{\"loss\": 2.086921760888072, \"learning_rate\": 2.765268231018409e-08, \"epoch\": 1.9988938927075925, \"step\": 253000}\n",
            "{\"eval_loss\": 2.140047362709411, \"epoch\": 2.0, \"step\": 253140}\n",
            "usage: run_language_modeling.py [-h] [--model_name_or_path MODEL_NAME_OR_PATH]\n",
            "                                [--model_type MODEL_TYPE]\n",
            "                                [--config_name CONFIG_NAME]\n",
            "                                [--tokenizer_name TOKENIZER_NAME]\n",
            "                                [--cache_dir CACHE_DIR]\n",
            "                                [--train_data_file TRAIN_DATA_FILE]\n",
            "                                [--eval_data_file EVAL_DATA_FILE]\n",
            "                                [--line_by_line] [--mlm]\n",
            "                                [--mlm_probability MLM_PROBABILITY]\n",
            "                                [--block_size BLOCK_SIZE] [--overwrite_cache]\n",
            "                                --output_dir OUTPUT_DIR\n",
            "                                [--overwrite_output_dir] [--do_train]\n",
            "                                [--do_eval] [--do_predict]\n",
            "                                [--evaluate_during_training]\n",
            "                                [--per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE]\n",
            "                                [--per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE]\n",
            "                                [--per_gpu_train_batch_size PER_GPU_TRAIN_BATCH_SIZE]\n",
            "                                [--per_gpu_eval_batch_size PER_GPU_EVAL_BATCH_SIZE]\n",
            "                                [--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS]\n",
            "                                [--learning_rate LEARNING_RATE]\n",
            "                                [--weight_decay WEIGHT_DECAY]\n",
            "                                [--adam_epsilon ADAM_EPSILON]\n",
            "                                [--max_grad_norm MAX_GRAD_NORM]\n",
            "                                [--num_train_epochs NUM_TRAIN_EPOCHS]\n",
            "                                [--max_steps MAX_STEPS]\n",
            "                                [--warmup_steps WARMUP_STEPS]\n",
            "                                [--logging_dir LOGGING_DIR]\n",
            "                                [--logging_first_step]\n",
            "                                [--logging_steps LOGGING_STEPS]\n",
            "                                [--save_steps SAVE_STEPS]\n",
            "                                [--save_total_limit SAVE_TOTAL_LIMIT]\n",
            "                                [--no_cuda] [--seed SEED] [--fp16]\n",
            "                                [--fp16_opt_level FP16_OPT_LEVEL]\n",
            "                                [--local_rank LOCAL_RANK]\n",
            "                                [--tpu_num_cores TPU_NUM_CORES]\n",
            "                                [--tpu_metrics_debug]\n",
            "\n",
            "optional arguments:\n",
            "  -h, --help            show this help message and exit\n",
            "  --model_name_or_path MODEL_NAME_OR_PATH\n",
            "                        The model checkpoint for weights initialization. Leave\n",
            "                        None if you want to train a model from scratch.\n",
            "  --model_type MODEL_TYPE\n",
            "                        If training from scratch, pass a model type from the\n",
            "                        list: t5, distilbert, albert, camembert, xlm-roberta,\n",
            "                        marian, bart, longformer, roberta, bert, openai-gpt,\n",
            "                        gpt2, transfo-xl, xlnet, flaubert, xlm, ctrl, electra,\n",
            "                        encoder_decoder, reformer\n",
            "  --config_name CONFIG_NAME\n",
            "                        Pretrained config name or path if not the same as\n",
            "                        model_name\n",
            "  --tokenizer_name TOKENIZER_NAME\n",
            "                        Pretrained tokenizer name or path if not the same as\n",
            "                        model_name\n",
            "  --cache_dir CACHE_DIR\n",
            "                        Where do you want to store the pretrained models\n",
            "                        downloaded from s3\n",
            "  --train_data_file TRAIN_DATA_FILE\n",
            "                        The input training data file (a text file).\n",
            "  --eval_data_file EVAL_DATA_FILE\n",
            "                        An optional input evaluation data file to evaluate the\n",
            "                        perplexity on (a text file).\n",
            "  --line_by_line        Whether distinct lines of text in the dataset are to\n",
            "                        be handled as distinct sequences.\n",
            "  --mlm                 Train with masked-language modeling loss instead of\n",
            "                        language modeling.\n",
            "  --mlm_probability MLM_PROBABILITY\n",
            "                        Ratio of tokens to mask for masked language modeling\n",
            "                        loss\n",
            "  --block_size BLOCK_SIZE\n",
            "                        Optional input sequence length after tokenization.The\n",
            "                        training dataset will be truncated in block of this\n",
            "                        size for training.Default to the model max input\n",
            "                        length for single sentence inputs (take into account\n",
            "                        special tokens).\n",
            "  --overwrite_cache     Overwrite the cached training and evaluation sets\n",
            "  --output_dir OUTPUT_DIR\n",
            "                        The output directory where the model predictions and\n",
            "                        checkpoints will be written.\n",
            "  --overwrite_output_dir\n",
            "                        Overwrite the content of the output directory.Use this\n",
            "                        to continue training if output_dir points to a\n",
            "                        checkpoint directory.\n",
            "  --do_train            Whether to run training.\n",
            "  --do_eval             Whether to run eval on the dev set.\n",
            "  --do_predict          Whether to run predictions on the test set.\n",
            "  --evaluate_during_training\n",
            "                        Run evaluation during training at each logging step.\n",
            "  --per_device_train_batch_size PER_DEVICE_TRAIN_BATCH_SIZE\n",
            "                        Batch size per GPU/TPU core/CPU for training.\n",
            "  --per_device_eval_batch_size PER_DEVICE_EVAL_BATCH_SIZE\n",
            "                        Batch size per GPU/TPU core/CPU for evaluation.\n",
            "  --per_gpu_train_batch_size PER_GPU_TRAIN_BATCH_SIZE\n",
            "                        Deprecated, the use of `--per_device_train_batch_size`\n",
            "                        is preferred. Batch size per GPU/TPU core/CPU for\n",
            "                        training.\n",
            "  --per_gpu_eval_batch_size PER_GPU_EVAL_BATCH_SIZE\n",
            "                        Deprecated, the use of `--per_device_eval_batch_size`\n",
            "                        is preferred.Batch size per GPU/TPU core/CPU for\n",
            "                        evaluation.\n",
            "  --gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS\n",
            "                        Number of updates steps to accumulate before\n",
            "                        performing a backward/update pass.\n",
            "  --learning_rate LEARNING_RATE\n",
            "                        The initial learning rate for Adam.\n",
            "  --weight_decay WEIGHT_DECAY\n",
            "                        Weight decay if we apply some.\n",
            "  --adam_epsilon ADAM_EPSILON\n",
            "                        Epsilon for Adam optimizer.\n",
            "  --max_grad_norm MAX_GRAD_NORM\n",
            "                        Max gradient norm.\n",
            "  --num_train_epochs NUM_TRAIN_EPOCHS\n",
            "                        Total number of training epochs to perform.\n",
            "  --max_steps MAX_STEPS\n",
            "                        If > 0: set total number of training steps to perform.\n",
            "                        Override num_train_epochs.\n",
            "  --warmup_steps WARMUP_STEPS\n",
            "                        Linear warmup over warmup_steps.\n",
            "  --logging_dir LOGGING_DIR\n",
            "                        Tensorboard log dir.\n",
            "  --logging_first_step  Log and eval the first global_step\n",
            "  --logging_steps LOGGING_STEPS\n",
            "                        Log every X updates steps.\n",
            "  --save_steps SAVE_STEPS\n",
            "                        Save checkpoint every X updates steps.\n",
            "  --save_total_limit SAVE_TOTAL_LIMIT\n",
            "                        Limit the total amount of checkpoints.Deletes the\n",
            "                        older checkpoints in the output_dir. Default is\n",
            "                        unlimited checkpoints\n",
            "  --no_cuda             Do not use CUDA even when it is available\n",
            "  --seed SEED           random seed for initialization\n",
            "  --fp16                Whether to use 16-bit (mixed) precision (through\n",
            "                        NVIDIA apex) instead of 32-bit\n",
            "  --fp16_opt_level FP16_OPT_LEVEL\n",
            "                        For fp16: Apex AMP optimization level selected in\n",
            "                        ['O0', 'O1', 'O2', and 'O3'].See details at\n",
            "                        https://nvidia.github.io/apex/amp.html\n",
            "  --local_rank LOCAL_RANK\n",
            "                        For distributed training: local_rank\n",
            "  --tpu_num_cores TPU_NUM_CORES\n",
            "                        TPU: Number of TPU cores (automatically passed by\n",
            "                        launcher script)\n",
            "  --tpu_metrics_debug   TPU: Whether to print debug metrics\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "IOPub data rate exceeded.\n",
            "The notebook server will temporarily stop sending output\n",
            "to the client in order to avoid crashing it.\n",
            "To change this limit, set the config variable\n",
            "`--NotebookApp.iopub_data_rate_limit`.\n",
            "\n",
            "Current values:\n",
            "NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)\n",
            "NotebookApp.rate_limit_window=3.0 (secs)\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PHyXLGcokOpZ",
        "colab_type": "text"
      },
      "source": [
        "# New Section"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s0Fwb6TEUgj1",
        "colab_type": "text"
      },
      "source": [
        "## Results\n",
        "\n",
        "To use it, you can run something like"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EGAlyaB3Sfiu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
        "# import torch\n",
        "# import numpy as np\n",
        "\n",
        "# OUTPUT_DIR = \"./output\"\n",
        "# device = 'cpu'\n",
        "# if torch.cuda.is_available():\n",
        "#     device = 'cuda'\n",
        "\n",
        "# tokenizer = GPT2Tokenizer.from_pretrained(OUTPUT_DIR)\n",
        "# model = GPT2LMHeadModel.from_pretrained(OUTPUT_DIR)\n",
        "# model = model.to(device)\n",
        "                                        \n",
        "# def generate(input_str, length=250, n=5):\n",
        "#   cur_ids = torch.tensor(tokenizer.encode(input_str)).unsqueeze(0).long().to(device)\n",
        "#   model.eval()\n",
        "#   with torch.no_grad():\n",
        "#     for i in range(length):\n",
        "#       outputs = model(cur_ids[:, -1024:], labels=cur_ids[:, -1024:])\n",
        "#       loss, logits = outputs[:2]\n",
        "#       softmax_logits = torch.softmax(logits[0,-1], dim=0)\n",
        "#       next_token_id = choose_from_top(softmax_logits.to('cpu').numpy(), n=n)\n",
        "#       cur_ids = torch.cat([cur_ids, torch.ones((1,1)).long().to(device) * next_token_id], dim=1)\n",
        "#     output_list = list(cur_ids.squeeze().to('cpu').numpy())\n",
        "#     output_text = tokenizer.decode(output_list)\n",
        "#     return output_text\n",
        "\n",
        "# def choose_from_top(probs, n=5):\n",
        "#     ind = np.argpartition(probs, -n)[-n:]\n",
        "#     top_prob = probs[ind]\n",
        "#     top_prob = top_prob / np.sum(top_prob) # Normalize\n",
        "#     choice = np.random.choice(n, 1, p = top_prob)\n",
        "#     token_id = ind[choice][0]\n",
        "#     return int(token_id)\n",
        "\n",
        "# generated_text = generate(\" = Toronto Raptors = \\n\")\n",
        "# print(generated_text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KvLfcwTOUnlt",
        "colab_type": "text"
      },
      "source": [
        "## Compressing/Zipping Model\n",
        "\n",
        "In order for us to preserve this model, we should compress it and save it somewhere. This can be done easily with"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W4zDMZZdUkW2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! tar -czf bert-base-multilingual-cased-Spanish.tar.gz output/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LT8ffxEsVL0-",
        "colab_type": "text"
      },
      "source": [
        "which creates a file called `gpt2-tuned.tar.gz`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8vGGDE3bVO6s",
        "colab_type": "text"
      },
      "source": [
        "## Saving it to Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X9J6NUCkVPpn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 128
        },
        "outputId": "33766046-7dbe-437c-841d-371c66c3a99a"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_MmLCVxSXDc-",
        "colab_type": "text"
      },
      "source": [
        "Now you can copy your output model to your Google Drive by running"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XyUwAAqJXMxS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp bert-base-multilingual-cased-Spanish.tar.gz /content/drive/My\\ Drive/"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}