model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d181f805",
   "metadata": {},
   "source": [
    "# Hugging Face on Amazon SageMaker\n",
    "\n",
    "Hugging Face Deep DLCs( Deep Learning Containers ) make it easier than ever to train Transformer models in SageMaker. Here is why you should consider using Hugging Face DLCs to train and deploy your next machine learning models:\n",
    "* One command is all you need\n",
    "* Accelerate machine learning from science to production\n",
    "* Built-in performance\n",
    "\n",
    "# Problem description\n",
    "\n",
    "**Diagnostics prediction** aims to automatically predict diagnostics needed for a patient with certain anamnesis.\n",
    "The anamnesis is represented by a raw text file with the doctor's notes about the patient, including his/her age, complaints described on a freeway, the patient's history and so on. It is unstructured - different sections of one patient's anamnesis may be absent in another's.\n",
    "\n",
    "The target labels are represented by the name of the needed diagnostics procedure.\n",
    "\n",
    "The value of the solution might be found in helping a doctor to find the optimal solution for diagnostics order. The patient can save time and money, and the doctor can serve a patient more efficiently by sparing time for unnecessary diagnostics. Moreover, in difficult cases, the algorithm may help a doctor to find a diagnosis faster, which in some cases may be extremely valuable, up to saving lives.\n",
    "\n",
    "Theoretically, some regularities found by the algorithm may help medical researchers to find the idea of treating some diseases, based on their unobvious interconnections with some symptoms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43bac952",
   "metadata": {},
   "source": [
    "# Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c525e7b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (2.86.2)\n",
      "Collecting sagemaker\n",
      "  Downloading sagemaker-2.91.1.tar.gz (534 kB)\n",
      "     |████████████████████████████████| 534 kB 23.1 MB/s            \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: attrs==20.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (20.3.0)\n",
      "Requirement already satisfied: boto3>=1.20.21 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (1.21.42)\n",
      "Requirement already satisfied: google-pasta in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: numpy>=1.9.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (1.19.5)\n",
      "Requirement already satisfied: protobuf>=3.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (3.17.2)\n",
      "Requirement already satisfied: protobuf3-to-dict>=0.1.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (0.1.5)\n",
      "Requirement already satisfied: smdebug_rulesconfig==1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata>=1.4.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (4.5.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (21.3)\n",
      "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (1.1.5)\n",
      "Requirement already satisfied: pathos in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sagemaker) (0.2.8)\n",
      "Requirement already satisfied: botocore<1.25.0,>=1.24.42 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from boto3>=1.20.21->sagemaker) (1.24.42)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from boto3>=1.20.21->sagemaker) (0.10.0)\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from boto3>=1.20.21->sagemaker) (0.5.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker) (3.4.1)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker) (3.10.0.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (2.4.7)\n",
      "Requirement already satisfied: six>=1.9 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from protobuf>=3.1->sagemaker) (1.16.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pandas->sagemaker) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pandas->sagemaker) (2021.1)\n",
      "Requirement already satisfied: pox>=0.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pathos->sagemaker) (0.3.0)\n",
      "Requirement already satisfied: multiprocess>=0.70.12 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pathos->sagemaker) (0.70.12.2)\n",
      "Requirement already satisfied: dill>=0.3.4 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pathos->sagemaker) (0.3.4)\n",
      "Requirement already satisfied: ppft>=1.6.6.4 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pathos->sagemaker) (1.6.6.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from botocore<1.25.0,>=1.24.42->boto3>=1.20.21->sagemaker) (1.26.8)\n",
      "Building wheels for collected packages: sagemaker\n",
      "  Building wheel for sagemaker (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sagemaker: filename=sagemaker-2.91.1-py2.py3-none-any.whl size=742450 sha256=ce79b96c744859cc6dd861dba821b99dfc1f62f96a2634ee4da973ee6da7b56f\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/49/c1/2e/5d7bcd98cc65e1db77f617e66ec2577082381ba5570282474f\n",
      "Successfully built sagemaker\n",
      "Installing collected packages: sagemaker\n",
      "  Attempting uninstall: sagemaker\n",
      "    Found existing installation: sagemaker 2.86.2\n",
      "    Uninstalling sagemaker-2.86.2:\n",
      "      Successfully uninstalled sagemaker-2.86.2\n",
      "Successfully installed sagemaker-2.91.1\n",
      "Collecting transformers\n",
      "  Downloading transformers-4.18.0-py3-none-any.whl (4.0 MB)\n",
      "     |████████████████████████████████| 4.0 MB 35.9 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (4.61.1)\n",
      "Requirement already satisfied: dataclasses in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (0.8)\n",
      "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (3.0.12)\n",
      "Requirement already satisfied: requests in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (2.25.1)\n",
      "Collecting huggingface-hub<1.0,>=0.1.0\n",
      "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
      "     |████████████████████████████████| 67 kB 9.1 MB/s             \n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (1.19.5)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (2021.4.4)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (5.4.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (21.3)\n",
      "Collecting tokenizers!=0.11.3,<0.13,>=0.11.1\n",
      "  Downloading tokenizers-0.12.1-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
      "     |████████████████████████████████| 6.6 MB 61.4 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: importlib-metadata in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from transformers) (4.5.0)\n",
      "Collecting sacremoses\n",
      "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
      "     |████████████████████████████████| 880 kB 56.4 MB/s            \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (3.10.0.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from packaging>=20.0->transformers) (2.4.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from importlib-metadata->transformers) (3.4.1)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests->transformers) (2.10)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests->transformers) (1.26.8)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests->transformers) (4.0.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests->transformers) (2021.5.30)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sacremoses->transformers) (1.16.0)\n",
      "Requirement already satisfied: click in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sacremoses->transformers) (8.0.1)\n",
      "Requirement already satisfied: joblib in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from sacremoses->transformers) (1.0.1)\n",
      "Building wheels for collected packages: sacremoses\n",
      "  Building wheel for sacremoses (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=909303 sha256=699a4ae5ec7986f23abb871558b68f1689f346ed51937f1d157de3bb251fa090\n",
      "  Stored in directory: /home/ec2-user/.cache/pip/wheels/4c/64/31/e9900a234b23fb3e9dc565d6114a9d6ff84a72dbdd356502b4\n",
      "Successfully built sacremoses\n",
      "Installing collected packages: tokenizers, sacremoses, huggingface-hub, transformers\n",
      "Successfully installed huggingface-hub-0.4.0 sacremoses-0.0.53 tokenizers-0.12.1 transformers-4.18.0\n",
      "Collecting datasets[s3]\n",
      "  Downloading datasets-2.2.2-py3-none-any.whl (346 kB)\n",
      "     |████████████████████████████████| 346 kB 23.2 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: importlib-metadata in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (4.5.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.1.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (0.4.0)\n",
      "Requirement already satisfied: dill<0.3.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (0.3.4)\n",
      "Requirement already satisfied: multiprocess in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (0.70.12.2)\n",
      "Collecting responses<0.19\n",
      "  Downloading responses-0.17.0-py2.py3-none-any.whl (38 kB)\n",
      "Collecting tqdm>=4.62.1\n",
      "  Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)\n",
      "     |████████████████████████████████| 78 kB 11.7 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: pyarrow>=6.0.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (6.0.1)\n",
      "Collecting xxhash\n",
      "  Downloading xxhash-3.0.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (211 kB)\n",
      "     |████████████████████████████████| 211 kB 69.7 MB/s            \n",
      "\u001b[?25hCollecting fsspec[http]>=2021.05.0\n",
      "  Downloading fsspec-2022.1.0-py3-none-any.whl (133 kB)\n",
      "     |████████████████████████████████| 133 kB 69.7 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: packaging in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (21.3)\n",
      "Requirement already satisfied: pandas in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (1.1.5)\n",
      "Requirement already satisfied: requests>=2.19.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (2.25.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (1.19.5)\n",
      "Requirement already satisfied: dataclasses in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (0.8)\n",
      "Requirement already satisfied: aiohttp in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (3.8.1)\n",
      "Requirement already satisfied: s3fs in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (2021.4.0)\n",
      "Requirement already satisfied: botocore in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (1.24.42)\n",
      "Requirement already satisfied: boto3 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (1.21.42)\n",
      "Requirement already satisfied: fsspec in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from datasets[s3]) (2021.4.0)\n",
      "Requirement already satisfied: pyyaml in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets[s3]) (5.4.1)\n",
      "Requirement already satisfied: filelock in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets[s3]) (3.0.12)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets[s3]) (3.10.0.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from packaging->datasets[s3]) (2.4.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.19.0->datasets[s3]) (1.26.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.19.0->datasets[s3]) (2021.5.30)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.19.0->datasets[s3]) (4.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.19.0->datasets[s3]) (2.10)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from responses<0.19->datasets[s3]) (1.16.0)\n",
      "Collecting importlib-resources\n",
      "  Downloading importlib_resources-5.4.0-py3-none-any.whl (28 kB)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (1.2.0)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (4.0.1)\n",
      "Requirement already satisfied: asynctest==0.13.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (0.13.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (20.3.0)\n",
      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (2.0.9)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (1.2.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (1.6.3)\n",
      "Requirement already satisfied: idna-ssl>=1.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (1.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiohttp->datasets[s3]) (5.1.0)\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from boto3->datasets[s3]) (0.5.0)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from boto3->datasets[s3]) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from botocore->datasets[s3]) (2.8.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from importlib-metadata->datasets[s3]) (3.4.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from pandas->datasets[s3]) (2021.1)\n",
      "Collecting s3fs\n",
      "  Downloading s3fs-2022.1.0-py3-none-any.whl (25 kB)\n",
      "Collecting aiobotocore~=2.1.0\n",
      "  Downloading aiobotocore-2.1.2.tar.gz (58 kB)\n",
      "     |████████████████████████████████| 58 kB 9.8 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.1.1.tar.gz (57 kB)\n",
      "     |████████████████████████████████| 57 kB 7.5 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.1.0.tar.gz (54 kB)\n",
      "     |████████████████████████████████| 54 kB 249 kB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hINFO: pip is looking at multiple versions of xxhash to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting xxhash\n",
      "  Downloading xxhash-2.0.2-cp36-cp36m-manylinux2010_x86_64.whl (243 kB)\n",
      "     |████████████████████████████████| 243 kB 61.2 MB/s            \n",
      "\u001b[?25hINFO: pip is looking at multiple versions of s3fs to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting s3fs\n",
      "  Downloading s3fs-2021.11.1-py3-none-any.whl (25 kB)\n",
      "  Downloading s3fs-2021.11.0-py3-none-any.whl (25 kB)\n",
      "  Downloading s3fs-2021.10.1-py3-none-any.whl (26 kB)\n",
      "Collecting aiobotocore~=1.4.1\n",
      "  Downloading aiobotocore-1.4.2.tar.gz (52 kB)\n",
      "     |████████████████████████████████| 52 kB 1.8 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting s3fs\n",
      "  Downloading s3fs-2021.10.0-py3-none-any.whl (26 kB)\n",
      "  Downloading s3fs-2021.9.0-py3-none-any.whl (26 kB)\n",
      "  Downloading s3fs-2021.8.1-py3-none-any.whl (26 kB)\n",
      "  Downloading s3fs-2021.8.0-py3-none-any.whl (26 kB)\n",
      "  Downloading s3fs-2021.7.0-py3-none-any.whl (25 kB)\n",
      "  Downloading s3fs-2021.6.1-py3-none-any.whl (25 kB)\n",
      "Requirement already satisfied: aiobotocore>=1.0.1 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from s3fs->datasets[s3]) (1.3.0)\n",
      "  Downloading s3fs-2021.6.0-py3-none-any.whl (24 kB)\n",
      "  Downloading s3fs-2021.5.0-py3-none-any.whl (24 kB)\n",
      "  Downloading s3fs-0.5.1-py3-none-any.whl (21 kB)\n",
      "Requirement already satisfied: wrapt>=1.10.10 in /home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from aiobotocore>=1.0.1->s3fs->datasets[s3]) (1.12.1)\n",
      "Collecting aiobotocore>=1.0.1\n",
      "  Downloading aiobotocore-2.3.2.tar.gz (104 kB)\n",
      "     |████████████████████████████████| 104 kB 74.1 MB/s            \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.3.1.tar.gz (65 kB)\n",
      "     |████████████████████████████████| 65 kB 944 kB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.3.0.tar.gz (65 kB)\n",
      "     |████████████████████████████████| 65 kB 3.6 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.2.0.tar.gz (59 kB)\n",
      "     |████████████████████████████████| 59 kB 10.2 MB/s            \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.0.1.tar.gz (54 kB)\n",
      "     |████████████████████████████████| 54 kB 481 kB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-2.0.0.tar.gz (52 kB)\n",
      "     |████████████████████████████████| 52 kB 239 kB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.4.1.tar.gz (52 kB)\n",
      "     |████████████████████████████████| 52 kB 1.4 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.4.0.tar.gz (51 kB)\n",
      "     |████████████████████████████████| 51 kB 498 kB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.3.3.tar.gz (50 kB)\n",
      "     |████████████████████████████████| 50 kB 2.0 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.3.2.tar.gz (49 kB)\n",
      "     |████████████████████████████████| 49 kB 8.2 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.3.1.tar.gz (48 kB)\n",
      "     |████████████████████████████████| 48 kB 7.9 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.2.2.tar.gz (48 kB)\n",
      "     |████████████████████████████████| 48 kB 1.2 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.2.1.tar.gz (48 kB)\n",
      "     |████████████████████████████████| 48 kB 7.2 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.2.0.tar.gz (47 kB)\n",
      "     |████████████████████████████████| 47 kB 6.7 MB/s             \n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Downloading aiobotocore-1.1.2-py3-none-any.whl (45 kB)\n",
      "     |████████████████████████████████| 45 kB 4.9 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.1.1-py3-none-any.whl (45 kB)\n",
      "     |████████████████████████████████| 45 kB 555 kB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.1.0-py3-none-any.whl (43 kB)\n",
      "     |████████████████████████████████| 43 kB 3.3 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.7-py3-none-any.whl (42 kB)\n",
      "     |████████████████████████████████| 42 kB 2.1 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.6-py3-none-any.whl (42 kB)\n",
      "     |████████████████████████████████| 42 kB 1.5 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.5-py3-none-any.whl (42 kB)\n",
      "     |████████████████████████████████| 42 kB 217 kB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.4-py3-none-any.whl (41 kB)\n",
      "     |████████████████████████████████| 41 kB 290 kB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.3-py3-none-any.whl (40 kB)\n",
      "     |████████████████████████████████| 40 kB 8.7 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.2-py3-none-any.whl (40 kB)\n",
      "     |████████████████████████████████| 40 kB 7.0 MB/s             \n",
      "\u001b[?25h  Downloading aiobotocore-1.0.1-py3-none-any.whl (40 kB)\n",
      "     |████████████████████████████████| 40 kB 7.6 MB/s             \n",
      "\u001b[?25hCollecting s3fs\n",
      "  Downloading s3fs-0.5.0-py3-none-any.whl (21 kB)\n",
      "  Downloading s3fs-0.4.2-py3-none-any.whl (19 kB)\n",
      "Installing collected packages: importlib-resources, tqdm, fsspec, xxhash, responses, s3fs, datasets\n",
      "  Attempting uninstall: tqdm\n",
      "    Found existing installation: tqdm 4.61.1\n",
      "    Uninstalling tqdm-4.61.1:\n",
      "      Successfully uninstalled tqdm-4.61.1\n",
      "  Attempting uninstall: fsspec\n",
      "    Found existing installation: fsspec 2021.4.0\n",
      "    Uninstalling fsspec-2021.4.0:\n",
      "      Successfully uninstalled fsspec-2021.4.0\n",
      "  Attempting uninstall: s3fs\n",
      "    Found existing installation: s3fs 2021.4.0\n",
      "    Uninstalling s3fs-2021.4.0:\n",
      "      Successfully uninstalled s3fs-2021.4.0\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "fastai 1.0.61 requires nvidia-ml-py3, which is not installed.\n",
      "spacy 3.0.6 requires pydantic<1.8.0,>=1.7.1, but you have pydantic 1.8.2 which is incompatible.\u001b[0m\n",
      "Successfully installed datasets-2.2.2 fsspec-2022.1.0 importlib-resources-5.4.0 responses-0.17.0 s3fs-0.4.2 tqdm-4.64.0 xxhash-3.0.0\n"
     ]
    }
   ],
   "source": [
    "# make sure the Amazon SageMaker SDK is updated\n",
    "!pip install \"sagemaker\" --upgrade\n",
    "!pip install transformers\n",
    "!pip install datasets[s3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6acbdf3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import a few libraries that will be needed\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "import torch\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker.huggingface import HuggingFace\n",
    "from datasets import load_dataset\n",
    "import boto3\n",
    "\n",
    "import pandas as pd\n",
    "import os, time, tarfile\n",
    "import io\n",
    "import re\n",
    "import json\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34296ce",
   "metadata": {},
   "source": [
    "# Permissions\n",
    "\n",
    "You need access to an IAM Role with the required permissions for Sagemaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9e6fac09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker bucket: medical-transcription-repo\n",
      "sagemaker session region: us-east-1\n"
     ]
    }
   ],
   "source": [
    "# gets role for executing training job and set a few variables\n",
    "sagemaker_session = sagemaker.Session()\n",
    "bucket = 'medical-transcription-repo'\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "print(f\"sagemaker bucket: {bucket}\")\n",
    "print(f\"sagemaker session region: {region}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf66fa4c",
   "metadata": {},
   "source": [
    "# Dataset\n",
    "This dataset contains sample medical transcriptions for various medical specialties. This dataset offers a solution by providing medical transcription samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a92e559",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>sample_name</th>\n",
       "      <th>transcription</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A 23-year-old white female presents with comp...</td>\n",
       "      <td>Allergy / Immunology</td>\n",
       "      <td>Allergic Rhinitis</td>\n",
       "      <td>SUBJECTIVE:,  This 23-year-old white female pr...</td>\n",
       "      <td>allergy / immunology, allergic rhinitis, aller...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Consult for laparoscopic gastric bypass.</td>\n",
       "      <td>Bariatrics</td>\n",
       "      <td>Laparoscopic Gastric Bypass Consult - 2</td>\n",
       "      <td>PAST MEDICAL HISTORY:, He has difficulty climb...</td>\n",
       "      <td>bariatrics, laparoscopic gastric bypass, weigh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Consult for laparoscopic gastric bypass.</td>\n",
       "      <td>Bariatrics</td>\n",
       "      <td>Laparoscopic Gastric Bypass Consult - 1</td>\n",
       "      <td>HISTORY OF PRESENT ILLNESS: , I have seen ABC ...</td>\n",
       "      <td>bariatrics, laparoscopic gastric bypass, heart...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2-D M-Mode. Doppler.</td>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>2-D Echocardiogram - 1</td>\n",
       "      <td>2-D M-MODE: , ,1.  Left atrial enlargement wit...</td>\n",
       "      <td>cardiovascular / pulmonary, 2-d m-mode, dopple...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2-D Echocardiogram</td>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>2-D Echocardiogram - 2</td>\n",
       "      <td>1.  The left ventricular cavity size and wall ...</td>\n",
       "      <td>cardiovascular / pulmonary, 2-d, doppler, echo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         description  \\\n",
       "0   A 23-year-old white female presents with comp...   \n",
       "1           Consult for laparoscopic gastric bypass.   \n",
       "2           Consult for laparoscopic gastric bypass.   \n",
       "3                             2-D M-Mode. Doppler.     \n",
       "4                                 2-D Echocardiogram   \n",
       "\n",
       "             medical_specialty                                sample_name  \\\n",
       "0         Allergy / Immunology                         Allergic Rhinitis    \n",
       "1                   Bariatrics   Laparoscopic Gastric Bypass Consult - 2    \n",
       "2                   Bariatrics   Laparoscopic Gastric Bypass Consult - 1    \n",
       "3   Cardiovascular / Pulmonary                    2-D Echocardiogram - 1    \n",
       "4   Cardiovascular / Pulmonary                    2-D Echocardiogram - 2    \n",
       "\n",
       "                                       transcription  \\\n",
       "0  SUBJECTIVE:,  This 23-year-old white female pr...   \n",
       "1  PAST MEDICAL HISTORY:, He has difficulty climb...   \n",
       "2  HISTORY OF PRESENT ILLNESS: , I have seen ABC ...   \n",
       "3  2-D M-MODE: , ,1.  Left atrial enlargement wit...   \n",
       "4  1.  The left ventricular cavity size and wall ...   \n",
       "\n",
       "                                            keywords  \n",
       "0  allergy / immunology, allergic rhinitis, aller...  \n",
       "1  bariatrics, laparoscopic gastric bypass, weigh...  \n",
       "2  bariatrics, laparoscopic gastric bypass, heart...  \n",
       "3  cardiovascular / pulmonary, 2-d m-mode, dopple...  \n",
       "4  cardiovascular / pulmonary, 2-d, doppler, echo...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket = 'medical-transcription-repo'\n",
    "key = 'dataset/mtsamples.csv'\n",
    "s3 = boto3.client('s3')\n",
    "obj = s3.get_object(Bucket=bucket, Key=key)\n",
    "df = pd.read_csv(io.BytesIO(obj['Body'].read()))\n",
    "df.drop('Unnamed: 0', axis=1, inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1ad90138",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "description             0\n",
       "medical_specialty       0\n",
       "sample_name             0\n",
       "transcription          33\n",
       "keywords             1068\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5611c304",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "After preprocessing, the dataset will be uploaded to our s3_bucket to be used within our training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "49d3e5cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['transcription'].fillna(df['description'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b52b6f64",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['medical_specialty'] = df['medical_specialty'].str.replace(r'\\A ', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b7af1e79",
   "metadata": {},
   "outputs": [],
   "source": [
    "feature = 'medical_specialty'\n",
    "df[feature] = df[feature].apply(lambda x:str.strip(x))\n",
    "# renaming specialties under Anesthesiology and Internal Medicine\n",
    "new_feature = 'medical_specialty_supergroup'\n",
    "new_class = 'Internal Medicine'\n",
    "df[new_feature] = df[feature].copy()\n",
    "# Grouping all anesthesiology specialties\n",
    "df[new_feature].mask(df[new_feature] == 'Hospice - Palliative Care', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Pain Management', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Sleep Medicine', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Endocrinology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Gastroenterology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Hematology - Oncology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Nephrology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Rheumatology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Cardiovascular / Pulmonary', new_class, inplace=True)\n",
    "# General medicine is also known as Internal Medicine\n",
    "df[new_feature].mask(df[new_feature] == 'General Medicine', new_class, inplace=True)\n",
    "\n",
    "new_class = 'Surgery'\n",
    "# Grouping all surgery specialties\n",
    "df[new_feature].mask(df[new_feature] == 'Surgery', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Cosmetic / Plastic Surgery', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Neurosurgery', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'ENT - Otolaryngology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Obstetrics / Gynecology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Urology', new_class, inplace=True)\n",
    "\n",
    "new_class = 'Medical Records'\n",
    "# Grouping all documents\n",
    "df[new_feature].mask(df[new_feature] == 'Consult - History and Phy.', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Discharge Summary', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Emergency Room Reports', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'IME-QME-Work Comp etc.', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Letters', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Office Notes', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'SOAP / Chart / Progress Notes', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Radiology', new_class, inplace=True)\n",
    "\n",
    "new_class = 'Other' \n",
    "# Grouping less popular specialties and specialties with the least data points\n",
    "df[new_feature].mask(df[new_feature] == 'Diets and Nutritions', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Bariatrics', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Dentistry', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Ophthalmology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Pediatrics - Neonatal', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Dermatology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Allergy / Immunology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Speech - Language', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Psychiatry / Psychology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Autopsy', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Lab Medicine - Pathology', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Physical Medicine - Rehab', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Orthopedic', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Chiropractic', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Podiatry', new_class, inplace=True)\n",
    "df[new_feature].mask(df[new_feature] == 'Neurology', new_class, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4e83b1a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>medical_specialty</th>\n",
       "      <th>sample_name</th>\n",
       "      <th>transcription</th>\n",
       "      <th>keywords</th>\n",
       "      <th>medical_specialty_supergroup</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A 23-year-old white female presents with comp...</td>\n",
       "      <td>Allergy / Immunology</td>\n",
       "      <td>Allergic Rhinitis</td>\n",
       "      <td>SUBJECTIVE:,  This 23-year-old white female pr...</td>\n",
       "      <td>allergy / immunology, allergic rhinitis, aller...</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Consult for laparoscopic gastric bypass.</td>\n",
       "      <td>Bariatrics</td>\n",
       "      <td>Laparoscopic Gastric Bypass Consult - 2</td>\n",
       "      <td>PAST MEDICAL HISTORY:, He has difficulty climb...</td>\n",
       "      <td>bariatrics, laparoscopic gastric bypass, weigh...</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Consult for laparoscopic gastric bypass.</td>\n",
       "      <td>Bariatrics</td>\n",
       "      <td>Laparoscopic Gastric Bypass Consult - 1</td>\n",
       "      <td>HISTORY OF PRESENT ILLNESS: , I have seen ABC ...</td>\n",
       "      <td>bariatrics, laparoscopic gastric bypass, heart...</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2-D M-Mode. Doppler.</td>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>2-D Echocardiogram - 1</td>\n",
       "      <td>2-D M-MODE: , ,1.  Left atrial enlargement wit...</td>\n",
       "      <td>cardiovascular / pulmonary, 2-d m-mode, dopple...</td>\n",
       "      <td>Internal Medicine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2-D Echocardiogram</td>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>2-D Echocardiogram - 2</td>\n",
       "      <td>1.  The left ventricular cavity size and wall ...</td>\n",
       "      <td>cardiovascular / pulmonary, 2-d, doppler, echo...</td>\n",
       "      <td>Internal Medicine</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         description  \\\n",
       "0   A 23-year-old white female presents with comp...   \n",
       "1           Consult for laparoscopic gastric bypass.   \n",
       "2           Consult for laparoscopic gastric bypass.   \n",
       "3                             2-D M-Mode. Doppler.     \n",
       "4                                 2-D Echocardiogram   \n",
       "\n",
       "            medical_specialty                                sample_name  \\\n",
       "0        Allergy / Immunology                         Allergic Rhinitis    \n",
       "1                  Bariatrics   Laparoscopic Gastric Bypass Consult - 2    \n",
       "2                  Bariatrics   Laparoscopic Gastric Bypass Consult - 1    \n",
       "3  Cardiovascular / Pulmonary                    2-D Echocardiogram - 1    \n",
       "4  Cardiovascular / Pulmonary                    2-D Echocardiogram - 2    \n",
       "\n",
       "                                       transcription  \\\n",
       "0  SUBJECTIVE:,  This 23-year-old white female pr...   \n",
       "1  PAST MEDICAL HISTORY:, He has difficulty climb...   \n",
       "2  HISTORY OF PRESENT ILLNESS: , I have seen ABC ...   \n",
       "3  2-D M-MODE: , ,1.  Left atrial enlargement wit...   \n",
       "4  1.  The left ventricular cavity size and wall ...   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  allergy / immunology, allergic rhinitis, aller...   \n",
       "1  bariatrics, laparoscopic gastric bypass, weigh...   \n",
       "2  bariatrics, laparoscopic gastric bypass, heart...   \n",
       "3  cardiovascular / pulmonary, 2-d m-mode, dopple...   \n",
       "4  cardiovascular / pulmonary, 2-d, doppler, echo...   \n",
       "\n",
       "  medical_specialty_supergroup  \n",
       "0                        Other  \n",
       "1                        Other  \n",
       "2                        Other  \n",
       "3            Internal Medicine  \n",
       "4            Internal Medicine  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9846a3cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "===========Original Categories =======================\n",
      "Cat:1 Allergy / Immunology : 7\n",
      "Cat:2 Autopsy : 8\n",
      "Cat:3 Bariatrics : 18\n",
      "Cat:4 Cardiovascular / Pulmonary : 372\n",
      "Cat:5 Chiropractic : 14\n",
      "Cat:6 Consult - History and Phy. : 516\n",
      "Cat:7 Cosmetic / Plastic Surgery : 27\n",
      "Cat:8 Dentistry : 27\n",
      "Cat:9 Dermatology : 29\n",
      "Cat:10 Diets and Nutritions : 10\n",
      "Cat:11 Discharge Summary : 108\n",
      "Cat:12 ENT - Otolaryngology : 98\n",
      "Cat:13 Emergency Room Reports : 75\n",
      "Cat:14 Endocrinology : 19\n",
      "Cat:15 Gastroenterology : 230\n",
      "Cat:16 General Medicine : 259\n",
      "Cat:17 Hematology - Oncology : 90\n",
      "Cat:18 Hospice - Palliative Care : 6\n",
      "Cat:19 IME-QME-Work Comp etc. : 16\n",
      "Cat:20 Lab Medicine - Pathology : 8\n",
      "Cat:21 Letters : 23\n",
      "Cat:22 Nephrology : 81\n",
      "Cat:23 Neurology : 223\n",
      "Cat:24 Neurosurgery : 94\n",
      "Cat:25 Obstetrics / Gynecology : 160\n",
      "Cat:26 Office Notes : 51\n",
      "Cat:27 Ophthalmology : 83\n",
      "Cat:28 Orthopedic : 355\n",
      "Cat:29 Pain Management : 62\n",
      "Cat:30 Pediatrics - Neonatal : 70\n",
      "Cat:31 Physical Medicine - Rehab : 21\n",
      "Cat:32 Podiatry : 47\n",
      "Cat:33 Psychiatry / Psychology : 53\n",
      "Cat:34 Radiology : 273\n",
      "Cat:35 Rheumatology : 10\n",
      "Cat:36 SOAP / Chart / Progress Notes : 166\n",
      "Cat:37 Sleep Medicine : 20\n",
      "Cat:38 Speech - Language : 9\n",
      "Cat:39 Surgery : 1103\n",
      "Cat:40 Urology : 158\n",
      "==================================\n"
     ]
    }
   ],
   "source": [
    "df['target'] = df['medical_specialty'].copy()\n",
    "data_categories  = df.groupby(df['target'])\n",
    "i = 1\n",
    "print('===========Original Categories =======================')\n",
    "for catName,dataCategory in data_categories:\n",
    "    print('Cat:'+str(i)+' '+catName + ' : '+ str(len(dataCategory)) )\n",
    "    i = i+1\n",
    "print('==================================')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dbb3747d",
   "metadata": {},
   "outputs": [],
   "source": [
    "counts = df['target'].value_counts()\n",
    "others = [k for k,v in counts.items() if v<50]\n",
    "for each_spec in others:\n",
    "    df.loc[df['target']==each_spec,'target']=' others' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ed4ba0fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============Reduced Categories ======================\n",
      "Cat:1  others : 319\n",
      "Cat:2 Cardiovascular / Pulmonary : 372\n",
      "Cat:3 Consult - History and Phy. : 516\n",
      "Cat:4 Discharge Summary : 108\n",
      "Cat:5 ENT - Otolaryngology : 98\n",
      "Cat:6 Emergency Room Reports : 75\n",
      "Cat:7 Gastroenterology : 230\n",
      "Cat:8 General Medicine : 259\n",
      "Cat:9 Hematology - Oncology : 90\n",
      "Cat:10 Nephrology : 81\n",
      "Cat:11 Neurology : 223\n",
      "Cat:12 Neurosurgery : 94\n",
      "Cat:13 Obstetrics / Gynecology : 160\n",
      "Cat:14 Office Notes : 51\n",
      "Cat:15 Ophthalmology : 83\n",
      "Cat:16 Orthopedic : 355\n",
      "Cat:17 Pain Management : 62\n",
      "Cat:18 Pediatrics - Neonatal : 70\n",
      "Cat:19 Psychiatry / Psychology : 53\n",
      "Cat:20 Radiology : 273\n",
      "Cat:21 SOAP / Chart / Progress Notes : 166\n",
      "Cat:22 Surgery : 1103\n",
      "Cat:23 Urology : 158\n",
      "============ Reduced Categories ======================\n"
     ]
    }
   ],
   "source": [
    "final_data_categories = df.groupby(df['target'])\n",
    "i=1\n",
    "print('============Reduced Categories ======================')\n",
    "for catName,dataCategory in final_data_categories:\n",
    "    print('Cat:'+str(i)+' '+catName + ' : '+ str(len(dataCategory)) )\n",
    "    i = i+1\n",
    "\n",
    "print('============ Reduced Categories ======================')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a8b96dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('pre-process.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a906a4bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource('s3')\n",
    "s3.meta.client.upload_file('pre-process.csv', bucket, 'preprocess/pre-processed/pre-process.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d203005a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>others</td>\n",
       "      <td>SUBJECTIVE:,  This 23-year-old white female pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>others</td>\n",
       "      <td>PAST MEDICAL HISTORY:, He has difficulty climb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>others</td>\n",
       "      <td>HISTORY OF PRESENT ILLNESS: , I have seen ABC ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>2-D M-MODE: , ,1.  Left atrial enlargement wit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Cardiovascular / Pulmonary</td>\n",
       "      <td>1.  The left ventricular cavity size and wall ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       target  \\\n",
       "0                      others   \n",
       "1                      others   \n",
       "2                      others   \n",
       "3  Cardiovascular / Pulmonary   \n",
       "4  Cardiovascular / Pulmonary   \n",
       "\n",
       "                                                text  \n",
       "0  SUBJECTIVE:,  This 23-year-old white female pr...  \n",
       "1  PAST MEDICAL HISTORY:, He has difficulty climb...  \n",
       "2  HISTORY OF PRESENT ILLNESS: , I have seen ABC ...  \n",
       "3  2-D M-MODE: , ,1.  Left atrial enlargement wit...  \n",
       "4  1.  The left ventricular cavity size and wall ...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feat = ['target', 'transcription']\n",
    "df = df[feat]\n",
    "df.columns = ['target', 'text']\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "295e6fe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process(df):\n",
    "    REPLACE_BY_SPACE_RE = re.compile('[/(){}\\[\\]\\|@,;]')\n",
    "    BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')\n",
    "    df['text'] = df['text'].apply(lambda x: x.lower())\n",
    "    df['text'] = df['text'].apply(lambda x: REPLACE_BY_SPACE_RE.sub('',  x)) # replace REPLACE_BY_SPACE_RE symbols by space in text\n",
    "    df['text'] = df['text'].apply(lambda x: BAD_SYMBOLS_RE.sub('',  x)) # delete symbols which are in BAD_SYMBOLS_RE from text\n",
    "    df['text'] = df['text'].apply(lambda x:str.strip(x))\n",
    "    \n",
    "    return df\n",
    "\n",
    "df = pre_process(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d9d82754",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['target'] = df['target'].astype('category').cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7712249a",
   "metadata": {},
   "source": [
    "## Estimating class weights\n",
    "For class imbalance, one aspect to consider is that each batch has enough signal to provide some coverage of all the classes, even the unbalanced ones. Otherwise, it may degenerate during training.\n",
    "\n",
    "We use class weights to handle an imbalanced dataset, in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bf2c52d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6813411476080141,\n",
       " 1: 0.5842683496961197,\n",
       " 2: 0.421216717222784,\n",
       " 3: 2.012479871175523,\n",
       " 4: 2.217834960070985,\n",
       " 5: 2.8979710144927537,\n",
       " 6: 0.9449905482041588,\n",
       " 7: 0.8391807957025348,\n",
       " 8: 2.414975845410628,\n",
       " 9: 2.6833064949006977,\n",
       " 10: 0.9746539286410606,\n",
       " 11: 2.3122109158186865,\n",
       " 12: 1.3584239130434783,\n",
       " 13: 4.261722080136402,\n",
       " 14: 2.6186485070717653,\n",
       " 15: 0.6122473974280466,\n",
       " 16: 3.505610098176718,\n",
       " 17: 3.104968944099379,\n",
       " 18: 4.10090237899918,\n",
       " 19: 0.7961458831024049,\n",
       " 20: 1.3093242535358827,\n",
       " 21: 0.19705151957113012,\n",
       " 22: 1.375619152449092}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils import class_weight\n",
    "import numpy as np\n",
    "class_weights = dict(enumerate(class_weight.compute_class_weight('balanced',\n",
    "                                                         classes=np.unique(df['target']),\n",
    "                                                         y=df['target'])))\n",
    "class_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7b8b2cf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = train_test_split(df, \n",
    "                               test_size=0.2,\n",
    "                               #stratify=df['target'],\n",
    "                               random_state=42)\n",
    "\n",
    "train.reset_index(inplace = True, drop = True)\n",
    "test.reset_index(inplace = True, drop = True)\n",
    "\n",
    "train.to_csv('med-train.csv', index=False)\n",
    "test.to_csv('med-test.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bc7686a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default-3807c6c770e47b96\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset csv/default to /home/ec2-user/.cache/huggingface/datasets/csv/default-3807c6c770e47b96/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e34da3db49b443deba008cf7fe7181a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0a652a5578fa485ab2d51aedaa444fca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset csv downloaded and prepared to /home/ec2-user/.cache/huggingface/datasets/csv/default-3807c6c770e47b96/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa1d1f19be6344e7af4ec403c0183aca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#train_dataset = load_dataset('csv', data_files='med-train.csv', delimiter=\",\", split=\"train\")\n",
    "#test_dataset = load_dataset('csv', data_files='med-test.csv',  delimiter=\",\", split=\"test\")\n",
    "dataset = load_dataset('csv', data_files={\n",
    "    \"train\": 'med-train.csv',\n",
    "    \"test\": 'med-test.csv'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d55cf673",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e0233eda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "409a21d22c74423694a385d96487258b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8899f62075374610a0346425bf412bf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c620588bf9504907bad8ab40f19ac5b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cb1faa2d9114589b4d97288fa71b9e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          | 0.00/455k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "847567c7db16451eab8239a87ba81e6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35c32871296b4a229d2c921efde5969b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "# tokenizer used in preprocessing\n",
    "tokenizer_name = 'distilbert-base-uncased'\n",
    "# download tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)\n",
    "\n",
    "# tokenizer helper function\n",
    "def tokenize(batch):\n",
    "    return tokenizer(batch['text'],padding=True, truncation=True)\n",
    "\n",
    "# tokenize dataset\n",
    "train_dataset = dataset['train'].map(tokenize, batched=True)\n",
    "test_dataset = dataset['test'].map(tokenize, batched=True)\n",
    "\n",
    "# set format for pytorch\n",
    "train_dataset =  train_dataset.rename_column(\"target\", \"labels\")\n",
    "train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])\n",
    "test_dataset = test_dataset.rename_column(\"target\", \"labels\")\n",
    "test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11794d64",
   "metadata": {},
   "source": [
    "## Uploading data to s3_bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2d9fe209",
   "metadata": {},
   "outputs": [],
   "source": [
    "import botocore\n",
    "from datasets.filesystems import S3FileSystem\n",
    "\n",
    "s3 = S3FileSystem()  \n",
    "prefix = 'preprocess'\n",
    "\n",
    "# save train_dataset to s3\n",
    "training_input_path = f's3://{bucket}/{prefix}/train'\n",
    "train_dataset.save_to_disk(training_input_path,fs=s3)\n",
    "\n",
    "# save test_dataset to s3\n",
    "test_input_path = f's3://{bucket}/{prefix}/test'\n",
    "test_dataset.save_to_disk(test_input_path,fs=s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44e2ecf6",
   "metadata": {},
   "source": [
    "# Fine-tuning & starting Sagemaker Training Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f433fa72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m AutoModelForSequenceClassification, Trainer, TrainingArguments, AutoTokenizer\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msklearn\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mmetrics\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m accuracy_score, precision_recall_fscore_support\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mdatasets\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m load_from_disk\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrandom\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mlogging\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msys\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "\n",
      "    parser = argparse.ArgumentParser()\n",
      "\n",
      "    \u001b[37m# hyperparameters sent by the client are passed as command-line arguments to the script.\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--epochs\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m3\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--train-batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m32\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--eval-batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m64\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--warmup_steps\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m500\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model_name\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m)\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--learning_rate\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[34m5e-5\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[37m# Data, model, and output directories\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--output-data-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_OUTPUT_DATA_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--module_dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_MODULE_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--n_gpus\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_NUM_GPUS\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--training_dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_TRAIN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--test_dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_TEST\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "\n",
      "    args, _ = parser.parse_known_args()\n",
      "\n",
      "    \u001b[37m# Set up logging\u001b[39;49;00m\n",
      "    logger = logging.getLogger(\u001b[31m__name__\u001b[39;49;00m)\n",
      "\n",
      "    logging.basicConfig(\n",
      "        level=logging.getLevelName(\u001b[33m\"\u001b[39;49;00m\u001b[33mINFO\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m),\n",
      "        handlers=[logging.StreamHandler(sys.stdout)],\n",
      "        \u001b[36mformat\u001b[39;49;00m=\u001b[33m\"\u001b[39;49;00m\u001b[33m%(asctime)s\u001b[39;49;00m\u001b[33m - \u001b[39;49;00m\u001b[33m%(name)s\u001b[39;49;00m\u001b[33m - \u001b[39;49;00m\u001b[33m%(levelname)s\u001b[39;49;00m\u001b[33m - \u001b[39;49;00m\u001b[33m%(message)s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "\n",
      "    \u001b[37m# load datasets\u001b[39;49;00m\n",
      "    train_dataset = load_from_disk(args.training_dir)\n",
      "    test_dataset = load_from_disk(args.test_dir)\n",
      "\n",
      "    logger.info(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m loaded train_dataset length is: \u001b[39;49;00m\u001b[33m{\u001b[39;49;00m\u001b[36mlen\u001b[39;49;00m(train_dataset)\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    logger.info(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m loaded test_dataset length is: \u001b[39;49;00m\u001b[33m{\u001b[39;49;00m\u001b[36mlen\u001b[39;49;00m(test_dataset)\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[37m# compute metrics function for binary classification\u001b[39;49;00m\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32mcompute_metrics\u001b[39;49;00m(pred):\n",
      "        labels = pred.label_ids\n",
      "        preds = pred.predictions.argmax(-\u001b[34m1\u001b[39;49;00m)\n",
      "        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average=\u001b[33m\"\u001b[39;49;00m\u001b[33mmacro\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "        acc = accuracy_score(labels, preds)\n",
      "        \u001b[34mreturn\u001b[39;49;00m {\u001b[33m\"\u001b[39;49;00m\u001b[33maccuracy\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: acc, \u001b[33m\"\u001b[39;49;00m\u001b[33mf1\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: f1, \u001b[33m\"\u001b[39;49;00m\u001b[33mprecision\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: precision, \u001b[33m\"\u001b[39;49;00m\u001b[33mrecall\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: recall}\n",
      "\n",
      "    \u001b[37m# download model from model hub\u001b[39;49;00m\n",
      "    model = AutoModelForSequenceClassification.from_pretrained(args.model_name, \n",
      "                                                               num_labels=\u001b[34m23\u001b[39;49;00m)\n",
      "    tokenizer = AutoTokenizer.from_pretrained(args.model_name)\n",
      "\n",
      "    \u001b[37m# define training args\u001b[39;49;00m\n",
      "    training_args = TrainingArguments(\n",
      "        output_dir=args.model_dir,\n",
      "        num_train_epochs=args.epochs,\n",
      "        per_device_train_batch_size=args.train_batch_size,\n",
      "        per_device_eval_batch_size=args.eval_batch_size,\n",
      "        warmup_steps=args.warmup_steps,\n",
      "        evaluation_strategy=\u001b[33m\"\u001b[39;49;00m\u001b[33mepoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        logging_dir=\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m{\u001b[39;49;00margs.output_data_dir\u001b[33m}\u001b[39;49;00m\u001b[33m/logs\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        learning_rate=\u001b[36mfloat\u001b[39;49;00m(args.learning_rate),\n",
      "    )\n",
      "    \u001b[37m# class weights\u001b[39;49;00m\n",
      "    class_weights = {\u001b[34m0\u001b[39;49;00m: \u001b[34m0.6813411476080141\u001b[39;49;00m,\n",
      "                     \u001b[34m1\u001b[39;49;00m: \u001b[34m0.5842683496961197\u001b[39;49;00m,\n",
      "                     \u001b[34m2\u001b[39;49;00m: \u001b[34m0.421216717222784\u001b[39;49;00m,\n",
      "                     \u001b[34m3\u001b[39;49;00m: \u001b[34m2.012479871175523\u001b[39;49;00m,\n",
      "                     \u001b[34m4\u001b[39;49;00m: \u001b[34m2.217834960070985\u001b[39;49;00m,\n",
      "                     \u001b[34m5\u001b[39;49;00m: \u001b[34m2.8979710144927537\u001b[39;49;00m,\n",
      "                     \u001b[34m6\u001b[39;49;00m: \u001b[34m0.9449905482041588\u001b[39;49;00m,\n",
      "                     \u001b[34m7\u001b[39;49;00m: \u001b[34m0.8391807957025348\u001b[39;49;00m,\n",
      "                     \u001b[34m8\u001b[39;49;00m: \u001b[34m2.414975845410628\u001b[39;49;00m,\n",
      "                     \u001b[34m9\u001b[39;49;00m: \u001b[34m2.6833064949006977\u001b[39;49;00m,\n",
      "                     \u001b[34m10\u001b[39;49;00m: \u001b[34m0.9746539286410606\u001b[39;49;00m,\n",
      "                     \u001b[34m11\u001b[39;49;00m: \u001b[34m2.3122109158186865\u001b[39;49;00m,\n",
      "                     \u001b[34m12\u001b[39;49;00m: \u001b[34m1.3584239130434783\u001b[39;49;00m,\n",
      "                     \u001b[34m13\u001b[39;49;00m: \u001b[34m4.261722080136402\u001b[39;49;00m,\n",
      "                     \u001b[34m14\u001b[39;49;00m: \u001b[34m2.6186485070717653\u001b[39;49;00m,\n",
      "                     \u001b[34m15\u001b[39;49;00m: \u001b[34m0.6122473974280466\u001b[39;49;00m,\n",
      "                     \u001b[34m16\u001b[39;49;00m: \u001b[34m3.505610098176718\u001b[39;49;00m,\n",
      "                     \u001b[34m17\u001b[39;49;00m: \u001b[34m3.104968944099379\u001b[39;49;00m,\n",
      "                     \u001b[34m18\u001b[39;49;00m: \u001b[34m4.10090237899918\u001b[39;49;00m,\n",
      "                     \u001b[34m19\u001b[39;49;00m: \u001b[34m0.7961458831024049\u001b[39;49;00m,\n",
      "                     \u001b[34m20\u001b[39;49;00m: \u001b[34m1.3093242535358827\u001b[39;49;00m,\n",
      "                     \u001b[34m21\u001b[39;49;00m: \u001b[34m0.19705151957113012\u001b[39;49;00m,\n",
      "                     \u001b[34m22\u001b[39;49;00m: \u001b[34m1.375619152449092\u001b[39;49;00m}\n",
      "    \u001b[37m# create Trainer instance\u001b[39;49;00m\n",
      "    \u001b[37m# Subclass Trainer and override the compute_loss method\u001b[39;49;00m\n",
      "    \u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mMedModelTrainer\u001b[39;49;00m(Trainer):\n",
      "        \u001b[34mdef\u001b[39;49;00m \u001b[32mcompute_loss\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, model, inputs, return_outputs=\u001b[34mFalse\u001b[39;49;00m):\n",
      "            labels = inputs.get(\u001b[33m\"\u001b[39;49;00m\u001b[33mlabels\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "            \u001b[37m# forward pass\u001b[39;49;00m\n",
      "            outputs = model(**inputs)\n",
      "            logits = outputs.get(\u001b[33m\"\u001b[39;49;00m\u001b[33mlogits\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "            \u001b[37m# compute custom loss\u001b[39;49;00m\n",
      "            loss_fct = torch.nn.CrossEntropyLoss(weight=torch.tensor(\u001b[36mlist\u001b[39;49;00m(class_weights.values())))\n",
      "            loss = loss_fct(logits.view(-\u001b[34m1\u001b[39;49;00m, \u001b[36mself\u001b[39;49;00m.model.config.num_labels), labels.view(-\u001b[34m1\u001b[39;49;00m))\n",
      "            \u001b[34mreturn\u001b[39;49;00m (loss, outputs) \u001b[34mif\u001b[39;49;00m return_outputs \u001b[34melse\u001b[39;49;00m loss\n",
      "    \n",
      "    trainer = MedModelTrainer(\n",
      "        model=model,\n",
      "        args=training_args,\n",
      "        compute_metrics=compute_metrics,\n",
      "        train_dataset=train_dataset,\n",
      "        eval_dataset=test_dataset,\n",
      "        tokenizer=tokenizer,\n",
      "    )\n",
      "\n",
      "    \u001b[37m# train model\u001b[39;49;00m\n",
      "    trainer.train()\n",
      "\n",
      "    \u001b[37m# evaluate model\u001b[39;49;00m\n",
      "    eval_result = trainer.evaluate(eval_dataset=test_dataset)\n",
      "\n",
      "    \u001b[37m# writes eval result to file which can be accessed later in s3 ouput\u001b[39;49;00m\n",
      "    \u001b[34mwith\u001b[39;49;00m \u001b[36mopen\u001b[39;49;00m(os.path.join(args.output_data_dir, \u001b[33m\"\u001b[39;49;00m\u001b[33meval_results.txt\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m), \u001b[33m\"\u001b[39;49;00m\u001b[33mw\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m) \u001b[34mas\u001b[39;49;00m writer:\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m***** Eval results *****\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "        \u001b[34mfor\u001b[39;49;00m key, value \u001b[35min\u001b[39;49;00m \u001b[36msorted\u001b[39;49;00m(eval_result.items()):\n",
      "            writer.write(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m{\u001b[39;49;00mkey\u001b[33m}\u001b[39;49;00m\u001b[33m = \u001b[39;49;00m\u001b[33m{\u001b[39;49;00mvalue\u001b[33m}\u001b[39;49;00m\u001b[33m\\n\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[37m# Saves the model to s3\u001b[39;49;00m\n",
      "    trainer.save_model(args.model_dir)\n"
     ]
    }
   ],
   "source": [
    "!pygmentize ./scripts/train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b52a074",
   "metadata": {},
   "source": [
    "## Creating an Estimator and starting a training job\n",
    "\n",
    "This estimator runs a Hugging Face training script in a SageMaker training environment.\n",
    "\n",
    "The estimator initiates the SageMaker-managed Hugging Face environment by using the pre-built Hugging Face Docker container and runs the Hugging Face training script that user provides through the entry_point argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "030ff892",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFace, TrainingCompilerConfig\n",
    "# initialize the Amazon Training Compiler\n",
    "compiler_config=TrainingCompilerConfig()\n",
    "\n",
    "# hyperparameters, which are passed into the training job\n",
    "hyperparameters={'epochs': 10,\n",
    "                 'train_batch_size': 64,\n",
    "                 'eval_batch_size': 32,\n",
    "                 'learning_rate': 3e-5, \n",
    "                 'model_name':'distilbert-base-uncased'\n",
    "                 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "757e4a46",
   "metadata": {},
   "outputs": [],
   "source": [
    "huggingface_estimator = HuggingFace(entry_point='train.py',\n",
    "                            source_dir='./scripts',\n",
    "                            instance_type='ml.p3.2xlarge',\n",
    "                            instance_count=1,\n",
    "                            role=role,\n",
    "                            transformers_version='4.11.0',\n",
    "                            pytorch_version='1.9.0',\n",
    "                            py_version='py38',\n",
    "                            output_path='s3://{}/models'.format(bucket),\n",
    "                            hyperparameters = hyperparameters,\n",
    "                            compiler_config = compiler_config\n",
    "                                   )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e73cea2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-05-23 13:59:34 Starting - Starting the training job...\n",
      "2022-05-23 14:00:00 Starting - Preparing the instances for trainingProfilerReport-1653314373: InProgress\n",
      ".........\n",
      "2022-05-23 14:01:32 Downloading - Downloading input data\n",
      "2022-05-23 14:01:32 Training - Downloading the training image.......................................\n",
      "2022-05-23 14:08:00 Training - Training image download completed. Training in progress..\u001b[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[34mbash: no job control in this shell\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:03,329 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:03,358 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:03,366 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:03,978 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {\n",
      "        \"sagemaker_training_compiler_debug_mode\": false,\n",
      "        \"sagemaker_training_compiler_enabled\": true\n",
      "    },\n",
      "    \"channel_input_dirs\": {\n",
      "        \"test\": \"/opt/ml/input/data/test\",\n",
      "        \"train\": \"/opt/ml/input/data/train\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"epochs\": 10,\n",
      "        \"eval_batch_size\": 32,\n",
      "        \"learning_rate\": 3e-05,\n",
      "        \"model_name\": \"distilbert-base-uncased\",\n",
      "        \"train_batch_size\": 64\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"test\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"train\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"huggingface-pytorch-trcomp-training-2022-05-23-13-59-33-558\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://medical-transcription-repo/huggingface-pytorch-trcomp-training-2022-05-23-13-59-33-558/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 8,\n",
      "    \"num_gpus\": 1,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"current_instance_type\": \"ml.p3.2xlarge\",\n",
      "        \"current_group_name\": \"homogeneousCluster\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"instance_groups\": [\n",
      "            {\n",
      "                \"instance_group_name\": \"homogeneousCluster\",\n",
      "                \"instance_type\": \"ml.p3.2xlarge\",\n",
      "                \"hosts\": [\n",
      "                    \"algo-1\"\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"epochs\":10,\"eval_batch_size\":32,\"learning_rate\":3e-05,\"model_name\":\"distilbert-base-uncased\",\"train_batch_size\":64}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={\"sagemaker_training_compiler_debug_mode\":false,\"sagemaker_training_compiler_enabled\":true}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.p3.2xlarge\",\"hosts\":[\"algo-1\"],\"instance_groups\":[{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.2xlarge\"}],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"test\",\"train\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=8\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=1\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://medical-transcription-repo/huggingface-pytorch-trcomp-training-2022-05-23-13-59-33-558/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{\"sagemaker_training_compiler_debug_mode\":false,\"sagemaker_training_compiler_enabled\":true},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"epochs\":10,\"eval_batch_size\":32,\"learning_rate\":3e-05,\"model_name\":\"distilbert-base-uncased\",\"train_batch_size\":64},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"huggingface-pytorch-trcomp-training-2022-05-23-13-59-33-558\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://medical-transcription-repo/huggingface-pytorch-trcomp-training-2022-05-23-13-59-33-558/source/sourcedir.tar.gz\",\"module_name\":\"train\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":1,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.p3.2xlarge\",\"hosts\":[\"algo-1\"],\"instance_groups\":[{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.2xlarge\"}],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--epochs\",\"10\",\"--eval_batch_size\",\"32\",\"--learning_rate\",\"3e-05\",\"--model_name\",\"distilbert-base-uncased\",\"--train_batch_size\",\"64\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TEST=/opt/ml/input/data/test\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAIN=/opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=10\u001b[0m\n",
      "\u001b[34mSM_HP_EVAL_BATCH_SIZE=32\u001b[0m\n",
      "\u001b[34mSM_HP_LEARNING_RATE=3e-05\u001b[0m\n",
      "\u001b[34mSM_HP_MODEL_NAME=distilbert-base-uncased\u001b[0m\n",
      "\u001b[34mSM_HP_TRAIN_BATCH_SIZE=64\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.8 train.py --epochs 10 --eval_batch_size 32 --learning_rate 3e-05 --model_name distilbert-base-uncased --train_batch_size 64\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:04.574 torch.__training_compiler__.TrainingCompilerConfig INFO] Found configuration for Training Compiler. Compiler will be configured during import of torch_xla.\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:06.719 torch_xla.__training_compiler__.TrainingCompilerConfig INFO] Configuring SM Training Compiler...\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:10,337 - __main__ - INFO -  loaded train_dataset length is: 3999\u001b[0m\n",
      "\u001b[34m2022-05-23 14:08:10,337 - __main__ - INFO -  loaded test_dataset length is: 1000\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/483 [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 483/483 [00:00<00:00, 437kB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/256M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading:   2%|▏         | 5.62M/256M [00:00<00:04, 58.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   5%|▍         | 12.0M/256M [00:00<00:03, 63.8MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   7%|▋         | 18.5M/256M [00:00<00:03, 65.4MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  10%|▉         | 24.7M/256M [00:00<00:03, 63.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  12%|█▏        | 31.0M/256M [00:00<00:03, 64.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  15%|█▍        | 37.2M/256M [00:00<00:03, 62.1MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  17%|█▋        | 43.1M/256M [00:00<00:03, 59.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  19%|█▉        | 49.2M/256M [00:00<00:03, 60.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  22%|██▏       | 55.7M/256M [00:00<00:03, 63.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  24%|██▍       | 62.3M/256M [00:01<00:03, 64.8MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  27%|██▋       | 68.9M/256M [00:01<00:02, 66.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  30%|██▉       | 75.5M/256M [00:01<00:02, 67.1MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  32%|███▏      | 82.1M/256M [00:01<00:02, 67.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  35%|███▍      | 88.6M/256M [00:01<00:02, 66.3MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  37%|███▋      | 95.3M/256M [00:01<00:02, 67.4MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  40%|███▉      | 102M/256M [00:01<00:02, 67.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  42%|████▏     | 109M/256M [00:01<00:02, 68.5MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  45%|████▌     | 115M/256M [00:01<00:02, 69.0MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  48%|████▊     | 122M/256M [00:01<00:01, 70.0MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  50%|█████     | 129M/256M [00:02<00:01, 70.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  53%|█████▎    | 136M/256M [00:02<00:01, 67.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  56%|█████▌    | 142M/256M [00:02<00:01, 66.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  58%|█████▊    | 149M/256M [00:02<00:01, 64.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  61%|██████    | 155M/256M [00:02<00:01, 63.8MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  63%|██████▎   | 161M/256M [00:02<00:01, 63.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  65%|██████▌   | 167M/256M [00:02<00:01, 62.3MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  68%|██████▊   | 173M/256M [00:02<00:01, 62.5MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  70%|███████   | 179M/256M [00:02<00:01, 64.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  73%|███████▎  | 186M/256M [00:02<00:01, 63.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  75%|███████▌  | 192M/256M [00:03<00:01, 63.0MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  77%|███████▋  | 198M/256M [00:03<00:00, 62.5MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  80%|███████▉  | 204M/256M [00:03<00:00, 63.0MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  82%|████████▏ | 210M/256M [00:03<00:00, 62.5MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  84%|████████▍ | 216M/256M [00:03<00:00, 61.4MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  87%|████████▋ | 222M/256M [00:03<00:00, 61.6MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  89%|████████▉ | 228M/256M [00:03<00:00, 61.9MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  92%|█████████▏| 234M/256M [00:03<00:00, 64.1MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  94%|█████████▍| 240M/256M [00:03<00:00, 64.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  96%|█████████▋| 247M/256M [00:04<00:00, 63.4MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  99%|█████████▉| 253M/256M [00:04<00:00, 64.8MB/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 256M/256M [00:04<00:00, 64.7MB/s]\u001b[0m\n",
      "\u001b[34mSome weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_transform.weight', 'vocab_projector.weight', 'vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias']\u001b[0m\n",
      "\u001b[34m- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\u001b[0m\n",
      "\u001b[34m- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\u001b[0m\n",
      "\u001b[34mSome weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\u001b[0m\n",
      "\u001b[34mYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\u001b[0m\n",
      "\u001b[34mSome weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_transform.weight', 'vocab_projector.weight', 'vocab_layer_norm.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias']\u001b[0m\n",
      "\u001b[34m- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\u001b[0m\n",
      "\u001b[34m- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\u001b[0m\n",
      "\u001b[34mSome weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\u001b[0m\n",
      "\u001b[34mYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 24.7kB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/226k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 226k/226k [00:00<00:00, 37.1MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/455k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 455k/455k [00:00<00:00, 48.0MB/s]\u001b[0m\n",
      "\u001b[34mUsing XLA device\u001b[0m\n",
      "\u001b[34mUsing XLA device\u001b[0m\n",
      "\u001b[34mThe following columns in the training set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the training set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running training *****\n",
      "  Num examples = 3999\u001b[0m\n",
      "\u001b[34mNum Epochs = 10\n",
      "  Instantaneous batch size per device = 32\u001b[0m\n",
      "\u001b[34m***** Running training *****\n",
      "  Num examples = 3999\n",
      "  Num Epochs = 10\n",
      "  Instantaneous batch size per device = 32\u001b[0m\n",
      "\u001b[34mTotal train batch size (w. parallel, distributed & accumulation) = 32\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 1250\u001b[0m\n",
      "\u001b[34mTotal train batch size (w. parallel, distributed & accumulation) = 32\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 1250\u001b[0m\n",
      "\u001b[34m0%|          | 0/1250 [00:00<?, ?it/s]\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.623 algo-1:26 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.746 algo-1:26 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.747 algo-1:26 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.748 algo-1:26 INFO hook.py:200] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.749 algo-1:26 INFO hook.py:255] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:22.749 algo-1:26 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:23.239 algo-1:26 INFO hook.py:591] name:weight count_params:589824\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:23.239 algo-1:26 INFO hook.py:591] name:bias count_params:768\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:23.239 algo-1:26 INFO hook.py:593] Total Trainable Params: 590592\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:23.239 algo-1:26 INFO hook.py:424] Monitoring the collections: losses\u001b[0m\n",
      "\u001b[34m[2022-05-23 14:08:23.242 algo-1:26 INFO hook.py:488] Hook is writing from the hook with pid: 26\u001b[0m\n",
      "\u001b[34m0%|          | 1/1250 [00:12<4:14:55, 12.25s/it]\u001b[0m\n",
      "\u001b[34m0%|          | 2/1250 [00:31<5:38:37, 16.28s/it]\u001b[0m\n",
      "\u001b[34m0%|          | 3/1250 [00:51<6:17:22, 18.16s/it]\u001b[0m\n",
      "\u001b[34m0%|          | 4/1250 [00:52<3:52:10, 11.18s/it]\u001b[0m\n",
      "\u001b[34m0%|          | 5/1250 [00:52<2:31:40,  7.31s/it]\u001b[0m\n",
      "\u001b[34m0%|          | 6/1250 [00:53<1:43:11,  4.98s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 7/1250 [00:53<1:12:31,  3.50s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 8/1250 [00:54<52:29,  2.54s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 9/1250 [00:54<38:58,  1.88s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 10/1250 [00:54<29:50,  1.44s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 11/1250 [00:55<23:36,  1.14s/it]\u001b[0m\n",
      "\u001b[34m1%|          | 12/1250 [00:55<19:19,  1.07it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 13/1250 [00:56<16:17,  1.27it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 14/1250 [00:56<14:09,  1.45it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 15/1250 [00:57<12:41,  1.62it/s]\u001b[0m\n",
      "\u001b[34m1%|▏         | 16/1250 [00:57<11:49,  1.74it/s]\u001b[0m\n",
      "\u001b[34m1%|▏         | 17/1250 [00:58<11:04,  1.86it/s]\u001b[0m\n",
      "\u001b[34m1%|▏         | 18/1250 [00:58<10:31,  1.95it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 19/1250 [00:59<10:09,  2.02it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 20/1250 [00:59<09:59,  2.05it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 21/1250 [01:00<09:51,  2.08it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 22/1250 [01:00<09:39,  2.12it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 23/1250 [01:00<09:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 24/1250 [01:01<09:34,  2.13it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 25/1250 [01:01<09:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 26/1250 [01:02<09:24,  2.17it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 27/1250 [01:02<09:21,  2.18it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 28/1250 [01:03<09:24,  2.17it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 29/1250 [01:03<09:19,  2.18it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 30/1250 [01:04<09:16,  2.19it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 31/1250 [01:04<09:14,  2.20it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 32/1250 [01:05<09:20,  2.17it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 33/1250 [01:05<09:18,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 34/1250 [01:05<09:15,  2.19it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 35/1250 [01:06<09:14,  2.19it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 36/1250 [01:06<09:16,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 37/1250 [01:07<09:15,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 38/1250 [01:07<09:12,  2.19it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 39/1250 [01:08<09:10,  2.20it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 40/1250 [01:08<09:15,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 41/1250 [01:09<09:15,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 42/1250 [01:09<09:13,  2.18it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 43/1250 [01:10<09:11,  2.19it/s]\u001b[0m\n",
      "\u001b[34m4%|▎         | 44/1250 [01:10<09:16,  2.17it/s]\u001b[0m\n",
      "\u001b[34m4%|▎         | 45/1250 [01:11<09:12,  2.18it/s]\u001b[0m\n",
      "\u001b[34m4%|▎         | 46/1250 [01:11<09:11,  2.18it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 47/1250 [01:11<09:10,  2.19it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 48/1250 [01:12<09:11,  2.18it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 49/1250 [01:12<09:07,  2.20it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 50/1250 [01:13<09:04,  2.21it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 51/1250 [01:13<09:05,  2.20it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 52/1250 [01:14<09:09,  2.18it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 53/1250 [01:14<09:06,  2.19it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 54/1250 [01:15<09:04,  2.20it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 55/1250 [01:15<09:03,  2.20it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 56/1250 [01:16<09:10,  2.17it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 57/1250 [01:16<09:08,  2.17it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 58/1250 [01:16<09:06,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 59/1250 [01:17<09:06,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 60/1250 [01:17<09:09,  2.16it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 61/1250 [01:18<09:05,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 62/1250 [01:18<09:03,  2.19it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 63/1250 [01:19<09:02,  2.19it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 64/1250 [01:19<09:04,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 65/1250 [01:20<09:02,  2.19it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 66/1250 [01:20<09:02,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 67/1250 [01:21<09:02,  2.18it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 68/1250 [01:21<09:01,  2.18it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 69/1250 [01:21<08:58,  2.19it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 70/1250 [01:22<08:57,  2.20it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 71/1250 [01:22<08:55,  2.20it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 72/1250 [01:23<09:00,  2.18it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 73/1250 [01:23<08:57,  2.19it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 74/1250 [01:24<08:57,  2.19it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 75/1250 [01:24<08:55,  2.20it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 76/1250 [01:25<09:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 77/1250 [01:25<08:55,  2.19it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 78/1250 [01:26<08:53,  2.20it/s]\u001b[0m\n",
      "\u001b[34m6%|▋         | 79/1250 [01:26<08:53,  2.19it/s]\u001b[0m\n",
      "\u001b[34m6%|▋         | 80/1250 [01:27<08:59,  2.17it/s]\u001b[0m\n",
      "\u001b[34m6%|▋         | 81/1250 [01:27<08:56,  2.18it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 82/1250 [01:27<08:54,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 83/1250 [01:28<08:51,  2.20it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 84/1250 [01:28<08:54,  2.18it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 85/1250 [01:29<08:51,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 86/1250 [01:29<08:50,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 87/1250 [01:30<08:47,  2.20it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 88/1250 [01:30<08:52,  2.18it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 89/1250 [01:31<08:50,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 90/1250 [01:31<08:50,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 91/1250 [01:32<08:49,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 92/1250 [01:32<08:52,  2.17it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 93/1250 [01:32<08:50,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 94/1250 [01:33<08:50,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 95/1250 [01:33<08:48,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 96/1250 [01:34<08:50,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 97/1250 [01:34<08:48,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 98/1250 [01:35<08:46,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 99/1250 [01:35<08:46,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 100/1250 [01:36<08:49,  2.17it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 101/1250 [01:36<08:45,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 102/1250 [01:37<08:44,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 103/1250 [01:37<08:45,  2.18it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 104/1250 [01:37<08:43,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 105/1250 [01:38<08:42,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 106/1250 [01:38<08:41,  2.19it/s]\u001b[0m\n",
      "\u001b[34m9%|▊         | 107/1250 [01:39<08:41,  2.19it/s]\u001b[0m\n",
      "\u001b[34m9%|▊         | 108/1250 [01:39<08:45,  2.17it/s]\u001b[0m\n",
      "\u001b[34m9%|▊         | 109/1250 [01:40<08:43,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 110/1250 [01:40<08:42,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 111/1250 [01:41<08:39,  2.19it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 112/1250 [01:41<08:41,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 113/1250 [01:42<08:41,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 114/1250 [01:42<08:41,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 115/1250 [01:43<08:40,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 116/1250 [01:43<08:42,  2.17it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 117/1250 [01:43<08:40,  2.18it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 118/1250 [01:44<08:38,  2.18it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 119/1250 [01:44<08:37,  2.19it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 120/1250 [01:45<08:35,  2.19it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 121/1250 [01:45<08:33,  2.20it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 122/1250 [01:46<08:34,  2.19it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 123/1250 [01:46<08:33,  2.19it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 124/1250 [01:47<08:35,  2.19it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 125/1250 [01:47<08:46,  2.14it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.88it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.87it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.21it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.90it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.74it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.64it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.55it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:01,  3.51it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.48it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.46it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.46it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:07<00:00,  1.09s/it]#033[A\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 3.0674848556518555, 'eval_accuracy': 0.127, 'eval_f1': 0.07203143223565314, 'eval_precision': 0.13279229074533422, 'eval_recall': 0.16457261284567257, 'eval_runtime': 11.1545, 'eval_samples_per_second': 89.65, 'eval_steps_per_second': 1.434, 'epoch': 1.0}\u001b[0m\n",
      "\u001b[34m10%|█         | 125/1250 [02:19<08:46,  2.14it/s]\u001b[0m\n",
      "\u001b[34m#015100%|██████████| 16/16 [00:07<00:00,  1.09s/it]\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m10%|█         | 126/1250 [02:20<3:08:29, 10.06s/it]\u001b[0m\n",
      "\u001b[34m10%|█         | 127/1250 [02:20<2:14:23,  7.18s/it]\u001b[0m\n",
      "\u001b[34m10%|█         | 128/1250 [02:21<1:36:33,  5.16s/it]\u001b[0m\n",
      "\u001b[34m10%|█         | 129/1250 [02:21<1:10:07,  3.75s/it]\u001b[0m\n",
      "\u001b[34m10%|█         | 130/1250 [02:21<51:36,  2.76s/it]\u001b[0m\n",
      "\u001b[34m10%|█         | 131/1250 [02:22<38:37,  2.07s/it]\u001b[0m\n",
      "\u001b[34m11%|█         | 132/1250 [02:22<29:35,  1.59s/it]\u001b[0m\n",
      "\u001b[34m11%|█         | 133/1250 [02:23<23:19,  1.25s/it]\u001b[0m\n",
      "\u001b[34m11%|█         | 134/1250 [02:23<18:51,  1.01s/it]\u001b[0m\n",
      "\u001b[34m11%|█         | 135/1250 [02:24<15:43,  1.18it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 136/1250 [02:24<13:34,  1.37it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 137/1250 [02:25<12:07,  1.53it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 138/1250 [02:25<11:00,  1.68it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 139/1250 [02:26<10:13,  1.81it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 140/1250 [02:26<09:40,  1.91it/s]\u001b[0m\n",
      "\u001b[34m11%|█▏        | 141/1250 [02:26<09:21,  1.97it/s]\u001b[0m\n",
      "\u001b[34m11%|█▏        | 142/1250 [02:27<09:03,  2.04it/s]\u001b[0m\n",
      "\u001b[34m11%|█▏        | 143/1250 [02:27<08:51,  2.08it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 144/1250 [02:28<08:45,  2.11it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 145/1250 [02:28<08:43,  2.11it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 146/1250 [02:29<08:37,  2.13it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 147/1250 [02:29<08:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 148/1250 [02:30<08:31,  2.16it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 149/1250 [02:30<08:28,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 150/1250 [02:31<08:26,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 151/1250 [02:31<08:26,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 152/1250 [02:32<08:25,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 153/1250 [02:32<08:28,  2.16it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 154/1250 [02:32<08:25,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 155/1250 [02:33<08:23,  2.18it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 156/1250 [02:33<08:22,  2.18it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 157/1250 [02:34<08:26,  2.16it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 158/1250 [02:34<08:24,  2.16it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 159/1250 [02:35<08:23,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 160/1250 [02:35<08:21,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 161/1250 [02:36<08:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 162/1250 [02:36<08:21,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 163/1250 [02:37<08:21,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 164/1250 [02:37<08:20,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 165/1250 [02:38<08:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 166/1250 [02:38<08:22,  2.16it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 167/1250 [02:38<08:20,  2.17it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 168/1250 [02:39<08:20,  2.16it/s]\u001b[0m\n",
      "\u001b[34m14%|█▎        | 169/1250 [02:39<08:24,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▎        | 170/1250 [02:40<08:20,  2.16it/s]\u001b[0m\n",
      "\u001b[34m14%|█▎        | 171/1250 [02:40<08:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 172/1250 [02:41<08:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 173/1250 [02:41<08:23,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 174/1250 [02:42<08:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 175/1250 [02:42<08:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 176/1250 [02:43<08:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 177/1250 [02:43<08:25,  2.12it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 178/1250 [02:44<08:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 179/1250 [02:44<08:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 180/1250 [02:45<08:15,  2.16it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 181/1250 [02:45<08:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 182/1250 [02:45<08:13,  2.16it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 183/1250 [02:46<08:10,  2.17it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 184/1250 [02:46<08:10,  2.17it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 185/1250 [02:47<08:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 186/1250 [02:47<08:12,  2.16it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 187/1250 [02:48<08:11,  2.16it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 188/1250 [02:48<08:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 189/1250 [02:49<08:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 190/1250 [02:49<08:15,  2.14it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 191/1250 [02:50<08:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 192/1250 [02:50<08:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 193/1250 [02:51<08:16,  2.13it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 194/1250 [02:51<08:12,  2.14it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 195/1250 [02:51<08:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 196/1250 [02:52<08:07,  2.16it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 197/1250 [02:52<08:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 198/1250 [02:53<08:07,  2.16it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 199/1250 [02:53<08:05,  2.16it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 200/1250 [02:54<08:03,  2.17it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 201/1250 [02:54<08:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 202/1250 [02:55<08:04,  2.16it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 203/1250 [02:55<08:03,  2.17it/s]\u001b[0m\n",
      "\u001b[34m16%|█▋        | 204/1250 [02:56<08:01,  2.17it/s]\u001b[0m\n",
      "\u001b[34m16%|█▋        | 205/1250 [02:56<08:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m16%|█▋        | 206/1250 [02:57<08:01,  2.17it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 207/1250 [02:57<08:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 208/1250 [02:57<08:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 209/1250 [02:58<08:04,  2.15it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 210/1250 [02:58<08:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 211/1250 [02:59<07:59,  2.17it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 212/1250 [02:59<08:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 213/1250 [03:00<08:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 214/1250 [03:00<08:02,  2.15it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 215/1250 [03:01<08:01,  2.15it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 216/1250 [03:01<07:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 217/1250 [03:02<08:03,  2.13it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 218/1250 [03:02<08:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 219/1250 [03:03<07:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 220/1250 [03:03<07:58,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 221/1250 [03:04<08:02,  2.13it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 222/1250 [03:04<08:01,  2.14it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 223/1250 [03:04<07:57,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 224/1250 [03:05<07:56,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 225/1250 [03:05<07:59,  2.14it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 226/1250 [03:06<07:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 227/1250 [03:06<07:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 228/1250 [03:07<07:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 229/1250 [03:07<07:57,  2.14it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 230/1250 [03:08<07:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 231/1250 [03:08<07:53,  2.15it/s]\u001b[0m\n",
      "\u001b[34m19%|█▊        | 232/1250 [03:09<07:51,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▊        | 233/1250 [03:09<07:57,  2.13it/s]\u001b[0m\n",
      "\u001b[34m19%|█▊        | 234/1250 [03:10<07:55,  2.14it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 235/1250 [03:10<07:54,  2.14it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 236/1250 [03:11<07:54,  2.14it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 237/1250 [03:11<07:50,  2.15it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 238/1250 [03:11<07:49,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 239/1250 [03:12<07:48,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 240/1250 [03:12<07:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 241/1250 [03:13<07:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 242/1250 [03:13<07:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 243/1250 [03:14<07:45,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 244/1250 [03:14<07:45,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 245/1250 [03:15<07:43,  2.17it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 246/1250 [03:15<07:44,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 247/1250 [03:16<07:44,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 248/1250 [03:16<07:43,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 249/1250 [03:17<07:43,  2.16it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 250/1250 [03:17<07:41,  2.17it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.84it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.16it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.86it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.70it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.61it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.52it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 2.7169127464294434, 'eval_accuracy': 0.235, 'eval_f1': 0.19931434229804842, 'eval_precision': 0.25445291391253894, 'eval_recall': 0.31939895453664735, 'eval_runtime': 5.1324, 'eval_samples_per_second': 194.841, 'eval_steps_per_second': 3.117, 'epoch': 2.0}\u001b[0m\n",
      "\u001b[34m20%|██        | 250/1250 [03:23<07:41,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m20%|██        | 251/1250 [03:23<35:00,  2.10s/it]\u001b[0m\n",
      "\u001b[34m20%|██        | 252/1250 [03:23<26:45,  1.61s/it]\u001b[0m\n",
      "\u001b[34m20%|██        | 253/1250 [03:24<21:00,  1.26s/it]\u001b[0m\n",
      "\u001b[34m20%|██        | 254/1250 [03:24<17:03,  1.03s/it]\u001b[0m\n",
      "\u001b[34m20%|██        | 255/1250 [03:25<14:13,  1.17it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 256/1250 [03:25<12:15,  1.35it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 257/1250 [03:26<10:51,  1.52it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 258/1250 [03:26<10:00,  1.65it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 259/1250 [03:27<09:17,  1.78it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 260/1250 [03:27<08:48,  1.87it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 261/1250 [03:28<08:26,  1.95it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 262/1250 [03:28<08:17,  1.99it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 263/1250 [03:29<08:03,  2.04it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 264/1250 [03:29<07:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 265/1250 [03:29<07:46,  2.11it/s]\u001b[0m\n",
      "\u001b[34m21%|██▏       | 266/1250 [03:30<07:42,  2.13it/s]\u001b[0m\n",
      "\u001b[34m21%|██▏       | 267/1250 [03:30<07:39,  2.14it/s]\u001b[0m\n",
      "\u001b[34m21%|██▏       | 268/1250 [03:31<07:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 269/1250 [03:31<07:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 270/1250 [03:32<07:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 271/1250 [03:32<07:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 272/1250 [03:33<07:33,  2.16it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 273/1250 [03:33<07:32,  2.16it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 274/1250 [03:34<07:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 275/1250 [03:34<07:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 276/1250 [03:35<07:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 277/1250 [03:35<07:31,  2.16it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 278/1250 [03:35<07:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 279/1250 [03:36<07:31,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 280/1250 [03:36<07:29,  2.16it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 281/1250 [03:37<07:28,  2.16it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 282/1250 [03:37<07:28,  2.16it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 283/1250 [03:38<07:27,  2.16it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 284/1250 [03:38<07:26,  2.16it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 285/1250 [03:39<07:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 286/1250 [03:39<07:32,  2.13it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 287/1250 [03:40<07:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 288/1250 [03:40<07:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 289/1250 [03:41<07:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 290/1250 [03:41<07:31,  2.12it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 291/1250 [03:42<07:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 292/1250 [03:42<07:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 293/1250 [03:42<07:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m24%|██▎       | 294/1250 [03:43<07:31,  2.12it/s]\u001b[0m\n",
      "\u001b[34m24%|██▎       | 295/1250 [03:43<07:28,  2.13it/s]\u001b[0m\n",
      "\u001b[34m24%|██▎       | 296/1250 [03:44<07:26,  2.14it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 297/1250 [03:44<07:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 298/1250 [03:45<07:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 299/1250 [03:45<07:24,  2.14it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 300/1250 [03:46<07:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 301/1250 [03:46<07:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 302/1250 [03:47<07:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 303/1250 [03:47<07:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 304/1250 [03:48<07:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 305/1250 [03:48<07:16,  2.16it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 306/1250 [03:49<07:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 307/1250 [03:49<07:19,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 308/1250 [03:49<07:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 309/1250 [03:50<07:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 310/1250 [03:50<07:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 311/1250 [03:51<07:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 312/1250 [03:51<07:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 313/1250 [03:52<07:15,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 314/1250 [03:52<07:15,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 315/1250 [03:53<07:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 316/1250 [03:53<07:13,  2.16it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 317/1250 [03:54<07:11,  2.16it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 318/1250 [03:54<07:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 319/1250 [03:55<07:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 320/1250 [03:55<07:09,  2.17it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 321/1250 [03:55<07:09,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 322/1250 [03:56<07:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 323/1250 [03:56<07:09,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 324/1250 [03:57<07:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 325/1250 [03:57<07:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 326/1250 [03:58<07:11,  2.14it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 327/1250 [03:58<07:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 328/1250 [03:59<07:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m26%|██▋       | 329/1250 [03:59<07:08,  2.15it/s]\u001b[0m\n",
      "\u001b[34m26%|██▋       | 330/1250 [04:00<07:11,  2.13it/s]\u001b[0m\n",
      "\u001b[34m26%|██▋       | 331/1250 [04:00<07:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 332/1250 [04:01<07:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 333/1250 [04:01<07:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 334/1250 [04:02<07:05,  2.15it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 335/1250 [04:02<07:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 336/1250 [04:02<07:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 337/1250 [04:03<07:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 338/1250 [04:03<07:10,  2.12it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 339/1250 [04:04<07:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 340/1250 [04:04<07:05,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 341/1250 [04:05<07:04,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 342/1250 [04:05<07:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 343/1250 [04:06<07:02,  2.15it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 344/1250 [04:06<07:01,  2.15it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 345/1250 [04:07<06:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 346/1250 [04:07<07:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 347/1250 [04:08<06:58,  2.16it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 348/1250 [04:08<06:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 349/1250 [04:09<06:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 350/1250 [04:09<07:01,  2.14it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 351/1250 [04:09<07:00,  2.14it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 352/1250 [04:10<06:58,  2.14it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 353/1250 [04:10<06:59,  2.14it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 354/1250 [04:11<06:56,  2.15it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 355/1250 [04:11<06:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 356/1250 [04:12<06:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▊       | 357/1250 [04:12<06:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▊       | 358/1250 [04:13<06:54,  2.15it/s]\u001b[0m\n",
      "\u001b[34m29%|██▊       | 359/1250 [04:13<06:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 360/1250 [04:14<06:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 361/1250 [04:14<06:54,  2.14it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 362/1250 [04:15<06:54,  2.14it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 363/1250 [04:15<06:52,  2.15it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 364/1250 [04:15<06:50,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 365/1250 [04:16<06:49,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 366/1250 [04:16<06:49,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 367/1250 [04:17<06:49,  2.16it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 368/1250 [04:17<06:48,  2.16it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 369/1250 [04:18<06:46,  2.17it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 370/1250 [04:18<06:47,  2.16it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 371/1250 [04:19<06:45,  2.17it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 372/1250 [04:19<06:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 373/1250 [04:20<06:45,  2.16it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 374/1250 [04:20<06:45,  2.16it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 375/1250 [04:21<06:44,  2.16it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.74it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.79it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.16it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.86it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.69it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.60it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.54it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.40it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 2.170170307159424, 'eval_accuracy': 0.276, 'eval_f1': 0.26939694373739204, 'eval_precision': 0.22967492378095833, 'eval_recall': 0.429993658876123, 'eval_runtime': 5.1339, 'eval_samples_per_second': 194.785, 'eval_steps_per_second': 3.117, 'epoch': 3.0}\u001b[0m\n",
      "\u001b[34m30%|███       | 375/1250 [04:26<06:44,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m30%|███       | 376/1250 [04:26<30:31,  2.10s/it]\u001b[0m\n",
      "\u001b[34m30%|███       | 377/1250 [04:27<23:26,  1.61s/it]\u001b[0m\n",
      "\u001b[34m30%|███       | 378/1250 [04:27<18:24,  1.27s/it]\u001b[0m\n",
      "\u001b[34m30%|███       | 379/1250 [04:28<14:55,  1.03s/it]\u001b[0m\n",
      "\u001b[34m30%|███       | 380/1250 [04:28<12:27,  1.16it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 381/1250 [04:29<10:43,  1.35it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 382/1250 [04:29<09:29,  1.52it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 383/1250 [04:30<08:42,  1.66it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 384/1250 [04:30<08:06,  1.78it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 385/1250 [04:31<07:41,  1.87it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 386/1250 [04:31<07:22,  1.95it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 387/1250 [04:32<07:12,  2.00it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 388/1250 [04:32<07:02,  2.04it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 389/1250 [04:33<06:54,  2.08it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 390/1250 [04:33<06:50,  2.10it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 391/1250 [04:34<06:49,  2.10it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 392/1250 [04:34<06:45,  2.12it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 393/1250 [04:34<06:43,  2.12it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 394/1250 [04:35<06:41,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 395/1250 [04:35<06:43,  2.12it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 396/1250 [04:36<06:40,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 397/1250 [04:36<06:37,  2.14it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 398/1250 [04:37<06:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 399/1250 [04:37<06:42,  2.11it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 400/1250 [04:38<06:39,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 401/1250 [04:38<06:38,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 402/1250 [04:39<06:37,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 403/1250 [04:39<06:41,  2.11it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 404/1250 [04:40<06:39,  2.12it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 405/1250 [04:40<06:37,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 406/1250 [04:41<06:36,  2.13it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 407/1250 [04:41<06:40,  2.10it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 408/1250 [04:42<06:38,  2.11it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 409/1250 [04:42<06:36,  2.12it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 410/1250 [04:42<06:37,  2.12it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 411/1250 [04:43<06:39,  2.10it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 412/1250 [04:43<06:36,  2.11it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 413/1250 [04:44<06:32,  2.13it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 414/1250 [04:44<06:31,  2.13it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 415/1250 [04:45<06:34,  2.12it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 416/1250 [04:45<06:31,  2.13it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 417/1250 [04:46<06:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 418/1250 [04:46<06:28,  2.14it/s]\u001b[0m\n",
      "\u001b[34m34%|███▎      | 419/1250 [04:47<06:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m34%|███▎      | 420/1250 [04:47<06:28,  2.14it/s]\u001b[0m\n",
      "\u001b[34m34%|███▎      | 421/1250 [04:48<06:27,  2.14it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 422/1250 [04:48<06:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 423/1250 [04:49<06:27,  2.13it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 424/1250 [04:49<06:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 425/1250 [04:49<06:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 426/1250 [04:50<06:22,  2.15it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 427/1250 [04:50<06:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 428/1250 [04:51<06:25,  2.13it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 429/1250 [04:51<06:22,  2.15it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 430/1250 [04:52<06:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 431/1250 [04:52<06:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 432/1250 [04:53<06:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 433/1250 [04:53<06:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 434/1250 [04:54<06:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 435/1250 [04:54<06:22,  2.13it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 436/1250 [04:55<06:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 437/1250 [04:55<06:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 438/1250 [04:56<06:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 439/1250 [04:56<06:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 440/1250 [04:56<06:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 441/1250 [04:57<06:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 442/1250 [04:57<06:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 443/1250 [04:58<06:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 444/1250 [04:58<06:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 445/1250 [04:59<06:13,  2.16it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 446/1250 [04:59<06:12,  2.16it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 447/1250 [05:00<06:11,  2.16it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 448/1250 [05:00<06:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 449/1250 [05:01<06:09,  2.17it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 450/1250 [05:01<06:09,  2.17it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 451/1250 [05:02<06:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 452/1250 [05:02<06:12,  2.14it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 453/1250 [05:03<07:31,  1.76it/s]\u001b[0m\n",
      "\u001b[34m36%|███▋      | 454/1250 [05:21<1:18:06,  5.89s/it]\u001b[0m\n",
      "\u001b[34m36%|███▋      | 455/1250 [05:22<56:30,  4.26s/it]\u001b[0m\n",
      "\u001b[34m36%|███▋      | 456/1250 [05:22<41:18,  3.12s/it]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 457/1250 [05:23<30:43,  2.32s/it]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 458/1250 [05:23<23:18,  1.77s/it]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 459/1250 [05:23<18:09,  1.38s/it]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 460/1250 [05:24<14:30,  1.10s/it]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 461/1250 [05:24<11:57,  1.10it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 462/1250 [05:25<10:09,  1.29it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 463/1250 [05:25<08:58,  1.46it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 464/1250 [05:26<08:05,  1.62it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 465/1250 [05:26<07:31,  1.74it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 466/1250 [05:27<07:04,  1.85it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 467/1250 [05:27<06:47,  1.92it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 468/1250 [05:28<06:33,  1.99it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 469/1250 [05:28<06:22,  2.04it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 470/1250 [05:29<06:14,  2.08it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 471/1250 [05:29<06:10,  2.11it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 472/1250 [05:29<06:07,  2.12it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 473/1250 [05:30<06:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 474/1250 [05:30<06:02,  2.14it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 475/1250 [05:31<06:04,  2.13it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 476/1250 [05:31<06:02,  2.14it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 477/1250 [05:32<06:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 478/1250 [05:32<05:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 479/1250 [05:33<06:00,  2.14it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 480/1250 [05:33<05:58,  2.15it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 481/1250 [05:34<05:56,  2.15it/s]\u001b[0m\n",
      "\u001b[34m39%|███▊      | 482/1250 [05:34<05:56,  2.15it/s]\u001b[0m\n",
      "\u001b[34m39%|███▊      | 483/1250 [05:35<05:59,  2.13it/s]\u001b[0m\n",
      "\u001b[34m39%|███▊      | 484/1250 [05:35<05:57,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 485/1250 [05:36<05:57,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 486/1250 [05:36<05:56,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 487/1250 [05:36<05:56,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 488/1250 [05:37<05:54,  2.15it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 489/1250 [05:37<05:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 490/1250 [05:38<05:51,  2.16it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 491/1250 [05:38<05:52,  2.15it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 492/1250 [05:39<05:53,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 493/1250 [05:39<05:51,  2.15it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 494/1250 [05:40<05:52,  2.14it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 495/1250 [05:40<05:53,  2.14it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 496/1250 [05:41<05:51,  2.15it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 497/1250 [05:41<05:50,  2.15it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 498/1250 [05:42<05:49,  2.15it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 499/1250 [05:42<05:48,  2.15it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 500/1250 [05:43<05:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m{'loss': 2.6115, 'learning_rate': 3e-05, 'epoch': 4.0}\u001b[0m\n",
      "\u001b[34m40%|████      | 500/1250 [05:43<05:46,  2.16it/s]Saving model checkpoint to /opt/ml/model/checkpoint-500\u001b[0m\n",
      "\u001b[34mSaving model checkpoint to /opt/ml/model/checkpoint-500\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/checkpoint-500/config.json\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/checkpoint-500/config.json\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/checkpoint-500/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/checkpoint-500/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/checkpoint-500/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/checkpoint-500/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/checkpoint-500/special_tokens_map.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/checkpoint-500/special_tokens_map.json\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.87it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.84it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.19it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.88it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.72it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.62it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.55it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:01,  3.51it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.48it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.46it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1248: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.7977099418640137, 'eval_accuracy': 0.311, 'eval_f1': 0.3150531729968047, 'eval_precision': 0.26571929604999306, 'eval_recall': 0.48492052031582783, 'eval_runtime': 5.1111, 'eval_samples_per_second': 195.651, 'eval_steps_per_second': 3.13, 'epoch': 4.0}\u001b[0m\n",
      "\u001b[34m40%|████      | 500/1250 [05:52<05:46,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m40%|████      | 501/1250 [05:52<39:53,  3.20s/it]\u001b[0m\n",
      "\u001b[34m40%|████      | 502/1250 [05:53<29:35,  2.37s/it]\u001b[0m\n",
      "\u001b[34m40%|████      | 503/1250 [05:53<22:24,  1.80s/it]\u001b[0m\n",
      "\u001b[34m40%|████      | 504/1250 [05:53<17:23,  1.40s/it]\u001b[0m\n",
      "\u001b[34m40%|████      | 505/1250 [05:54<13:52,  1.12s/it]\u001b[0m\n",
      "\u001b[34m40%|████      | 506/1250 [05:54<11:24,  1.09it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 507/1250 [05:55<09:40,  1.28it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 508/1250 [05:55<08:31,  1.45it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 509/1250 [05:56<07:39,  1.61it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 510/1250 [05:56<07:03,  1.75it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 511/1250 [05:57<06:39,  1.85it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 512/1250 [05:57<06:27,  1.90it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 513/1250 [05:58<06:13,  1.97it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 514/1250 [05:58<06:03,  2.02it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 515/1250 [05:59<05:55,  2.07it/s]\u001b[0m\n",
      "\u001b[34m41%|████▏     | 516/1250 [05:59<05:52,  2.08it/s]\u001b[0m\n",
      "\u001b[34m41%|████▏     | 517/1250 [06:00<05:48,  2.10it/s]\u001b[0m\n",
      "\u001b[34m41%|████▏     | 518/1250 [06:00<05:44,  2.12it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 519/1250 [06:00<05:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 520/1250 [06:01<05:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 521/1250 [06:01<05:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 522/1250 [06:02<05:37,  2.16it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 523/1250 [06:02<05:36,  2.16it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 524/1250 [06:03<05:39,  2.14it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 525/1250 [06:03<05:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 526/1250 [06:04<05:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 527/1250 [06:04<05:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 528/1250 [06:05<05:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 529/1250 [06:05<05:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 530/1250 [06:06<05:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 531/1250 [06:06<05:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 532/1250 [06:07<05:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 533/1250 [06:07<05:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 534/1250 [06:07<05:33,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 535/1250 [06:08<05:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 536/1250 [06:08<05:34,  2.13it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 537/1250 [06:09<05:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 538/1250 [06:09<05:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 539/1250 [06:10<05:29,  2.16it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 540/1250 [06:10<05:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 541/1250 [06:11<05:29,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 542/1250 [06:11<05:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 543/1250 [06:12<05:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▎     | 544/1250 [06:12<05:31,  2.13it/s]\u001b[0m\n",
      "\u001b[34m44%|████▎     | 545/1250 [06:13<05:30,  2.14it/s]\u001b[0m\n",
      "\u001b[34m44%|████▎     | 546/1250 [06:13<05:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 547/1250 [06:13<05:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 548/1250 [06:14<05:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 549/1250 [06:14<05:27,  2.14it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 550/1250 [06:15<05:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 551/1250 [06:15<05:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 552/1250 [06:16<05:27,  2.13it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 553/1250 [06:16<05:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 554/1250 [06:17<05:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 555/1250 [06:17<05:22,  2.15it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 556/1250 [06:18<05:24,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 557/1250 [06:18<05:22,  2.15it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 558/1250 [06:19<05:23,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 559/1250 [06:19<05:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 560/1250 [06:20<05:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 561/1250 [06:20<05:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 562/1250 [06:20<05:19,  2.15it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 563/1250 [06:21<05:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 564/1250 [06:21<05:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 565/1250 [06:22<05:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 566/1250 [06:22<05:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 567/1250 [06:23<05:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 568/1250 [06:23<05:19,  2.13it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 569/1250 [06:24<05:18,  2.14it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 570/1250 [06:24<05:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 571/1250 [06:25<05:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 572/1250 [06:25<05:18,  2.13it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 573/1250 [06:26<05:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 574/1250 [06:26<05:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 575/1250 [06:27<05:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 576/1250 [06:27<05:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 577/1250 [06:27<05:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 578/1250 [06:28<05:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▋     | 579/1250 [06:28<05:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▋     | 580/1250 [06:29<05:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m46%|████▋     | 581/1250 [06:29<05:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 582/1250 [06:30<05:09,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 583/1250 [06:30<05:08,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 584/1250 [06:31<05:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 585/1250 [06:31<05:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 586/1250 [06:32<05:07,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 587/1250 [06:32<05:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 588/1250 [06:33<05:08,  2.15it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 589/1250 [06:33<05:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 590/1250 [06:34<05:05,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 591/1250 [06:34<05:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 592/1250 [06:34<05:05,  2.16it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 593/1250 [06:35<05:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 594/1250 [06:35<05:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 595/1250 [06:36<05:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 596/1250 [06:36<05:05,  2.14it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 597/1250 [06:37<05:06,  2.13it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 598/1250 [06:37<05:04,  2.14it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 599/1250 [06:38<05:03,  2.15it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 600/1250 [06:38<05:05,  2.13it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 601/1250 [06:39<05:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 602/1250 [06:39<05:03,  2.13it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 603/1250 [06:40<05:02,  2.14it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 604/1250 [06:40<05:05,  2.12it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 605/1250 [06:41<05:05,  2.11it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 606/1250 [06:41<05:04,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▊     | 607/1250 [06:41<05:02,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▊     | 608/1250 [06:42<05:07,  2.09it/s]\u001b[0m\n",
      "\u001b[34m49%|████▊     | 609/1250 [06:42<05:05,  2.10it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 610/1250 [06:43<05:03,  2.11it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 611/1250 [06:43<05:00,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 612/1250 [06:44<05:01,  2.11it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 613/1250 [06:44<05:00,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 614/1250 [06:45<04:58,  2.13it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 615/1250 [06:45<04:56,  2.14it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 616/1250 [06:46<04:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 617/1250 [06:46<04:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 618/1250 [06:47<04:53,  2.15it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 619/1250 [06:47<04:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 620/1250 [06:48<04:53,  2.15it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 621/1250 [06:48<04:52,  2.15it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 622/1250 [06:48<04:52,  2.15it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 623/1250 [06:49<04:50,  2.16it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 624/1250 [06:49<04:49,  2.16it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 625/1250 [06:50<04:48,  2.17it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.83it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.16it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.87it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.69it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.60it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.54it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.6591743230819702, 'eval_accuracy': 0.328, 'eval_f1': 0.34143418773945083, 'eval_precision': 0.2932168595121422, 'eval_recall': 0.49666651173045023, 'eval_runtime': 5.1571, 'eval_samples_per_second': 193.907, 'eval_steps_per_second': 3.103, 'epoch': 5.0}\u001b[0m\n",
      "\u001b[34m50%|█████     | 625/1250 [06:55<04:48,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 626/1250 [06:56<21:47,  2.10s/it]\u001b[0m\n",
      "\u001b[34m50%|█████     | 627/1250 [06:56<16:39,  1.60s/it]\u001b[0m\n",
      "\u001b[34m50%|█████     | 628/1250 [06:57<13:04,  1.26s/it]\u001b[0m\n",
      "\u001b[34m50%|█████     | 629/1250 [06:57<10:36,  1.03s/it]\u001b[0m\n",
      "\u001b[34m50%|█████     | 630/1250 [06:58<08:51,  1.17it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 631/1250 [06:58<07:36,  1.36it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 632/1250 [06:59<06:44,  1.53it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 633/1250 [06:59<06:11,  1.66it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 634/1250 [07:00<05:45,  1.78it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 635/1250 [07:00<05:27,  1.88it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 636/1250 [07:00<05:13,  1.96it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 637/1250 [07:01<05:06,  2.00it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 638/1250 [07:01<05:00,  2.04it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 639/1250 [07:02<04:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 640/1250 [07:02<04:50,  2.10it/s]\u001b[0m\n",
      "\u001b[34m51%|█████▏    | 641/1250 [07:03<04:49,  2.10it/s]\u001b[0m\n",
      "\u001b[34m51%|█████▏    | 642/1250 [07:03<04:47,  2.12it/s]\u001b[0m\n",
      "\u001b[34m51%|█████▏    | 643/1250 [07:04<04:45,  2.13it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 644/1250 [07:04<04:43,  2.14it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 645/1250 [07:05<04:44,  2.13it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 646/1250 [07:05<04:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 647/1250 [07:06<04:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 648/1250 [07:06<04:39,  2.16it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 649/1250 [07:07<04:41,  2.13it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 650/1250 [07:07<04:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 651/1250 [07:07<04:38,  2.15it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 652/1250 [07:08<04:38,  2.15it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 653/1250 [07:08<04:39,  2.13it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 654/1250 [07:09<04:38,  2.14it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 655/1250 [07:09<04:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 656/1250 [07:10<04:35,  2.16it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 657/1250 [07:10<04:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 658/1250 [07:11<04:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 659/1250 [07:11<04:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 660/1250 [07:12<04:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 661/1250 [07:12<04:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 662/1250 [07:13<04:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 663/1250 [07:13<04:33,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 664/1250 [07:13<04:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 665/1250 [07:14<04:34,  2.13it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 666/1250 [07:14<04:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 667/1250 [07:15<04:31,  2.15it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 668/1250 [07:15<04:31,  2.14it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▎    | 669/1250 [07:16<04:33,  2.13it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▎    | 670/1250 [07:16<04:31,  2.14it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▎    | 671/1250 [07:17<04:29,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 672/1250 [07:17<04:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 673/1250 [07:18<04:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 674/1250 [07:18<04:27,  2.16it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 675/1250 [07:19<04:26,  2.16it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 676/1250 [07:19<04:25,  2.16it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 677/1250 [07:20<04:27,  2.14it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 678/1250 [07:20<04:26,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 679/1250 [07:20<04:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 680/1250 [07:21<04:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 681/1250 [07:21<04:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 682/1250 [07:22<04:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 683/1250 [07:22<04:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 684/1250 [07:23<04:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 685/1250 [07:23<04:25,  2.12it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 686/1250 [07:24<04:24,  2.13it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 687/1250 [07:24<04:23,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 688/1250 [07:25<04:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 689/1250 [07:25<04:23,  2.13it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 690/1250 [07:26<04:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 691/1250 [07:26<04:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 692/1250 [07:27<04:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 693/1250 [07:27<04:22,  2.12it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 694/1250 [07:27<04:20,  2.13it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 695/1250 [07:28<04:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 696/1250 [07:28<04:18,  2.14it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 697/1250 [07:29<04:19,  2.13it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 698/1250 [07:29<04:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 699/1250 [07:30<04:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 700/1250 [07:30<04:15,  2.16it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 701/1250 [07:31<04:17,  2.13it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 702/1250 [07:31<04:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 703/1250 [07:32<04:15,  2.14it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 704/1250 [07:32<04:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 705/1250 [07:33<04:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 706/1250 [07:33<04:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 707/1250 [07:34<04:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 708/1250 [07:34<04:11,  2.16it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 709/1250 [07:34<04:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 710/1250 [07:35<04:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 711/1250 [07:35<04:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 712/1250 [07:36<04:08,  2.16it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 713/1250 [07:36<04:11,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 714/1250 [07:37<04:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 715/1250 [07:37<04:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 716/1250 [07:38<04:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 717/1250 [07:38<04:10,  2.13it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 718/1250 [07:39<04:09,  2.13it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 719/1250 [07:39<04:08,  2.13it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 720/1250 [07:40<04:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 721/1250 [07:40<04:08,  2.13it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 722/1250 [07:41<04:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 723/1250 [07:41<04:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 724/1250 [07:41<04:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 725/1250 [07:42<04:10,  2.10it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 726/1250 [07:42<04:07,  2.11it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 727/1250 [07:43<04:05,  2.13it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 728/1250 [07:43<04:04,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 729/1250 [07:44<04:05,  2.12it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 730/1250 [07:44<04:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 731/1250 [07:45<04:01,  2.15it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▊    | 732/1250 [07:45<04:01,  2.14it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▊    | 733/1250 [07:46<04:02,  2.13it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▊    | 734/1250 [07:46<04:01,  2.13it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 735/1250 [07:47<04:00,  2.14it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 736/1250 [07:47<03:59,  2.14it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 737/1250 [07:48<03:58,  2.15it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 738/1250 [07:48<03:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 739/1250 [07:49<03:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 740/1250 [07:49<03:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 741/1250 [07:49<03:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 742/1250 [07:50<03:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 743/1250 [07:50<03:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 744/1250 [07:51<03:53,  2.17it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 745/1250 [07:51<03:53,  2.17it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 746/1250 [07:52<03:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 747/1250 [07:52<03:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 748/1250 [07:53<03:51,  2.17it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 749/1250 [07:53<03:51,  2.16it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 750/1250 [07:54<03:50,  2.16it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.84it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.83it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.15it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.86it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.70it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.59it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.53it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.6530165672302246, 'eval_accuracy': 0.331, 'eval_f1': 0.3559729472608992, 'eval_precision': 0.31019492178210284, 'eval_recall': 0.5211931016278841, 'eval_runtime': 5.1128, 'eval_samples_per_second': 195.588, 'eval_steps_per_second': 3.129, 'epoch': 6.0}\u001b[0m\n",
      "\u001b[34m60%|██████    | 750/1250 [07:59<03:50,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m60%|██████    | 751/1250 [07:59<17:21,  2.09s/it]\u001b[0m\n",
      "\u001b[34m60%|██████    | 752/1250 [08:00<13:16,  1.60s/it]\u001b[0m\n",
      "\u001b[34m60%|██████    | 753/1250 [08:00<10:25,  1.26s/it]\u001b[0m\n",
      "\u001b[34m60%|██████    | 754/1250 [08:01<08:28,  1.03s/it]\u001b[0m\n",
      "\u001b[34m60%|██████    | 755/1250 [08:01<07:04,  1.16it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 756/1250 [08:02<06:04,  1.35it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 757/1250 [08:02<05:23,  1.53it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 758/1250 [08:03<04:57,  1.65it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 759/1250 [08:03<04:36,  1.78it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 760/1250 [08:04<04:20,  1.88it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 761/1250 [08:04<04:09,  1.96it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 762/1250 [08:05<04:04,  2.00it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 763/1250 [08:05<03:57,  2.05it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 764/1250 [08:06<03:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 765/1250 [08:06<03:50,  2.11it/s]\u001b[0m\n",
      "\u001b[34m61%|██████▏   | 766/1250 [08:06<03:48,  2.12it/s]\u001b[0m\n",
      "\u001b[34m61%|██████▏   | 767/1250 [08:07<03:47,  2.12it/s]\u001b[0m\n",
      "\u001b[34m61%|██████▏   | 768/1250 [08:07<03:46,  2.13it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 769/1250 [08:08<03:44,  2.14it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 770/1250 [08:08<03:45,  2.13it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 771/1250 [08:09<03:43,  2.14it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 772/1250 [08:09<03:42,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 773/1250 [08:10<03:41,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 774/1250 [08:10<03:43,  2.13it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 775/1250 [08:11<03:41,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 776/1250 [08:11<03:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 777/1250 [08:12<03:39,  2.16it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 778/1250 [08:12<03:40,  2.14it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 779/1250 [08:13<03:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 780/1250 [08:13<03:38,  2.15it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 781/1250 [08:13<03:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 782/1250 [08:14<03:40,  2.12it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 783/1250 [08:14<03:38,  2.14it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 784/1250 [08:15<03:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 785/1250 [08:15<03:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 786/1250 [08:16<03:37,  2.13it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 787/1250 [08:16<03:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 788/1250 [08:17<03:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 789/1250 [08:17<03:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 790/1250 [08:18<03:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 791/1250 [08:18<03:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 792/1250 [08:19<03:33,  2.15it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 793/1250 [08:19<03:32,  2.16it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▎   | 794/1250 [08:20<03:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▎   | 795/1250 [08:20<03:31,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▎   | 796/1250 [08:20<03:30,  2.16it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 797/1250 [08:21<03:29,  2.16it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 798/1250 [08:21<03:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 799/1250 [08:22<03:29,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 800/1250 [08:22<03:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 801/1250 [08:23<03:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 802/1250 [08:23<03:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 803/1250 [08:24<03:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 804/1250 [08:24<03:28,  2.14it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 805/1250 [08:25<03:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 806/1250 [08:25<03:27,  2.14it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 807/1250 [08:26<03:26,  2.14it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 808/1250 [08:26<03:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 809/1250 [08:26<03:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 810/1250 [08:27<03:25,  2.14it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 811/1250 [08:27<03:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 812/1250 [08:28<03:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 813/1250 [08:28<03:22,  2.16it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 814/1250 [08:29<03:23,  2.14it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 815/1250 [08:29<03:22,  2.15it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 816/1250 [08:30<03:21,  2.16it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 817/1250 [08:30<03:20,  2.16it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 818/1250 [08:31<03:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 819/1250 [08:31<03:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 820/1250 [08:32<03:18,  2.17it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 821/1250 [08:32<03:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 822/1250 [08:33<03:19,  2.15it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 823/1250 [08:33<03:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 824/1250 [08:33<03:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 825/1250 [08:34<03:16,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 826/1250 [08:34<03:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 827/1250 [08:35<03:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 828/1250 [08:35<03:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▋   | 829/1250 [08:36<03:15,  2.16it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▋   | 830/1250 [08:36<03:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▋   | 831/1250 [08:37<03:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 832/1250 [08:37<03:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 833/1250 [08:38<03:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 834/1250 [08:38<03:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 835/1250 [08:39<03:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 836/1250 [08:39<03:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 837/1250 [08:40<03:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 838/1250 [08:40<03:13,  2.13it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 839/1250 [08:40<03:12,  2.13it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 840/1250 [08:41<03:12,  2.13it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 841/1250 [08:41<03:11,  2.13it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 842/1250 [08:42<03:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 843/1250 [08:42<03:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 844/1250 [08:43<03:08,  2.15it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 845/1250 [08:43<03:07,  2.15it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 846/1250 [08:44<03:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 847/1250 [08:44<03:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 848/1250 [08:45<03:07,  2.15it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 849/1250 [08:45<03:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 850/1250 [08:46<03:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 851/1250 [08:46<03:07,  2.13it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 852/1250 [08:47<03:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 853/1250 [08:47<03:05,  2.14it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 854/1250 [08:47<03:06,  2.12it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 855/1250 [08:48<03:05,  2.13it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 856/1250 [08:48<03:04,  2.13it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▊   | 857/1250 [08:49<03:03,  2.14it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▊   | 858/1250 [08:49<03:03,  2.13it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▊   | 859/1250 [08:50<03:02,  2.14it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 860/1250 [08:50<03:01,  2.15it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 861/1250 [08:51<03:00,  2.16it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 862/1250 [08:51<02:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 863/1250 [08:52<02:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 864/1250 [08:52<02:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 865/1250 [08:53<02:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 866/1250 [08:53<02:57,  2.17it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 867/1250 [08:53<02:56,  2.17it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 868/1250 [08:54<02:56,  2.17it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 869/1250 [08:54<02:55,  2.17it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 870/1250 [08:55<02:55,  2.17it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 871/1250 [08:55<02:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 872/1250 [08:56<02:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 873/1250 [08:56<02:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 874/1250 [08:57<02:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 875/1250 [08:57<02:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.76it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.78it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.14it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.86it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.69it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.59it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.53it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.6353627443313599, 'eval_accuracy': 0.318, 'eval_f1': 0.34338772711001303, 'eval_precision': 0.2897201580031375, 'eval_recall': 0.48540334900948706, 'eval_runtime': 5.1355, 'eval_samples_per_second': 194.722, 'eval_steps_per_second': 3.116, 'epoch': 7.0}\u001b[0m\n",
      "\u001b[34m70%|███████   | 875/1250 [09:03<02:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#015                                               #033[A\u001b[0m\n",
      "\u001b[34m70%|███████   | 876/1250 [09:03<13:03,  2.09s/it]\u001b[0m\n",
      "\u001b[34m70%|███████   | 877/1250 [09:04<09:58,  1.60s/it]\u001b[0m\n",
      "\u001b[34m70%|███████   | 878/1250 [09:04<07:49,  1.26s/it]\u001b[0m\n",
      "\u001b[34m70%|███████   | 879/1250 [09:04<06:20,  1.03s/it]\u001b[0m\n",
      "\u001b[34m70%|███████   | 880/1250 [09:05<05:16,  1.17it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 881/1250 [09:05<04:32,  1.35it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 882/1250 [09:06<04:01,  1.53it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 883/1250 [09:06<03:40,  1.66it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 884/1250 [09:07<03:24,  1.79it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 885/1250 [09:07<03:13,  1.89it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 886/1250 [09:08<03:05,  1.96it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 887/1250 [09:08<03:00,  2.01it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 888/1250 [09:09<02:56,  2.05it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 889/1250 [09:09<03:29,  1.72it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 890/1250 [09:10<03:16,  1.83it/s]\u001b[0m\n",
      "\u001b[34m71%|███████▏  | 891/1250 [09:10<03:08,  1.91it/s]\u001b[0m\n",
      "\u001b[34m71%|███████▏  | 892/1250 [09:11<03:01,  1.97it/s]\u001b[0m\n",
      "\u001b[34m71%|███████▏  | 893/1250 [09:11<02:56,  2.03it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 894/1250 [09:12<02:52,  2.06it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 895/1250 [09:12<02:51,  2.07it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 896/1250 [09:13<02:49,  2.09it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 897/1250 [09:13<02:46,  2.12it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 898/1250 [09:14<02:45,  2.13it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 899/1250 [09:14<02:45,  2.12it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 900/1250 [09:15<02:44,  2.12it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 901/1250 [09:15<02:43,  2.13it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 902/1250 [09:16<02:42,  2.14it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 903/1250 [09:16<02:42,  2.13it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 904/1250 [09:16<02:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 905/1250 [09:17<02:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 906/1250 [09:17<02:39,  2.16it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 907/1250 [09:18<02:40,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 908/1250 [09:18<02:38,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 909/1250 [09:19<02:38,  2.16it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 910/1250 [09:19<02:37,  2.16it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 911/1250 [09:20<02:38,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 912/1250 [09:20<02:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 913/1250 [09:21<02:35,  2.16it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 914/1250 [09:21<02:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 915/1250 [09:22<02:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 916/1250 [09:22<02:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 917/1250 [09:23<02:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 918/1250 [09:23<02:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▎  | 919/1250 [09:23<02:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▎  | 920/1250 [09:24<02:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▎  | 921/1250 [09:24<02:33,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 922/1250 [09:25<02:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 923/1250 [09:25<02:33,  2.13it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 924/1250 [09:26<02:32,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 925/1250 [09:26<02:31,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 926/1250 [09:27<02:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 927/1250 [09:27<02:31,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 928/1250 [09:28<02:30,  2.13it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 929/1250 [09:28<02:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 930/1250 [09:29<02:28,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 931/1250 [09:29<02:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 932/1250 [09:30<02:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 933/1250 [09:30<02:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 934/1250 [09:30<02:26,  2.16it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 935/1250 [09:31<02:27,  2.14it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 936/1250 [09:31<02:26,  2.14it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 937/1250 [09:32<02:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 938/1250 [09:32<02:24,  2.16it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 939/1250 [09:33<02:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 940/1250 [09:33<02:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 941/1250 [09:34<02:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 942/1250 [09:34<02:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 943/1250 [09:35<02:24,  2.13it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 944/1250 [09:35<02:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 945/1250 [09:36<02:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 946/1250 [09:36<02:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 947/1250 [09:37<02:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 948/1250 [09:37<02:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 949/1250 [09:37<02:19,  2.15it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 950/1250 [09:38<02:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 951/1250 [09:38<02:20,  2.12it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 952/1250 [09:39<02:19,  2.13it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 953/1250 [09:39<02:18,  2.14it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▋  | 954/1250 [09:40<02:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▋  | 955/1250 [09:40<02:18,  2.13it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▋  | 956/1250 [09:41<02:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 957/1250 [09:41<02:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 958/1250 [09:42<02:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 959/1250 [09:42<02:17,  2.12it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 960/1250 [09:43<02:15,  2.14it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 961/1250 [09:43<02:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 962/1250 [09:44<02:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 963/1250 [09:44<02:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 964/1250 [09:44<02:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 965/1250 [09:45<02:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 966/1250 [09:45<02:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 967/1250 [09:46<02:12,  2.14it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 968/1250 [09:46<02:11,  2.14it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 969/1250 [09:47<02:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 970/1250 [09:47<02:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 971/1250 [09:48<02:11,  2.13it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 972/1250 [09:48<02:10,  2.13it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 973/1250 [09:49<02:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 974/1250 [09:49<02:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 975/1250 [09:50<02:09,  2.13it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 976/1250 [09:50<02:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 977/1250 [09:51<02:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 978/1250 [09:51<02:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 979/1250 [09:51<02:06,  2.13it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 980/1250 [09:52<02:05,  2.15it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 981/1250 [09:52<02:05,  2.15it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▊  | 982/1250 [09:53<02:04,  2.15it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▊  | 983/1250 [09:53<02:05,  2.12it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▊  | 984/1250 [09:54<02:04,  2.14it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 985/1250 [09:54<02:03,  2.15it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 986/1250 [09:55<02:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 987/1250 [09:55<02:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 988/1250 [09:56<02:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 989/1250 [09:56<02:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 990/1250 [09:57<02:00,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 991/1250 [09:57<02:00,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 992/1250 [09:57<01:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 993/1250 [09:58<01:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 994/1250 [09:58<01:58,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 995/1250 [09:59<01:58,  2.16it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 996/1250 [09:59<01:57,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 997/1250 [10:00<01:57,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 998/1250 [10:00<01:57,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 999/1250 [10:01<01:56,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1000/1250 [10:01<01:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m{'loss': 1.3168, 'learning_rate': 9.999999999999999e-06, 'epoch': 8.0}\u001b[0m\n",
      "\u001b[34m80%|████████  | 1000/1250 [10:02<01:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34mSaving model checkpoint to /opt/ml/model/checkpoint-1000\u001b[0m\n",
      "\u001b[34mSaving model checkpoint to /opt/ml/model/checkpoint-1000\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/checkpoint-1000/config.json\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/checkpoint-1000/config.json\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/checkpoint-1000/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/checkpoint-1000/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/checkpoint-1000/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/checkpoint-1000/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/checkpoint-1000/special_tokens_map.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/checkpoint-1000/special_tokens_map.json\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.83it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.83it/s]\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.18it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.88it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.72it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.62it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.54it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.50it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.48it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.46it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.43it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.6620429754257202, 'eval_accuracy': 0.316, 'eval_f1': 0.34671449752585226, 'eval_precision': 0.304080117278383, 'eval_recall': 0.48566701861482536, 'eval_runtime': 5.1684, 'eval_samples_per_second': 193.485, 'eval_steps_per_second': 3.096, 'epoch': 8.0}\u001b[0m\n",
      "\u001b[34m80%|████████  | 1000/1250 [10:10<01:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m80%|████████  | 1001/1250 [10:11<13:01,  3.14s/it]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1002/1250 [10:11<09:39,  2.34s/it]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1003/1250 [10:12<07:17,  1.77s/it]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1004/1250 [10:12<05:40,  1.38s/it]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1005/1250 [10:12<04:31,  1.11s/it]\u001b[0m\n",
      "\u001b[34m80%|████████  | 1006/1250 [10:13<03:42,  1.10it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1007/1250 [10:13<03:09,  1.28it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1008/1250 [10:14<02:46,  1.46it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1009/1250 [10:14<02:29,  1.61it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1010/1250 [10:15<02:17,  1.75it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1011/1250 [10:15<02:08,  1.85it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1012/1250 [10:16<02:03,  1.93it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1013/1250 [10:16<01:58,  2.00it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1014/1250 [10:17<01:55,  2.05it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 1015/1250 [10:17<01:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 1016/1250 [10:18<01:52,  2.09it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 1017/1250 [10:18<01:50,  2.11it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 1018/1250 [10:18<01:49,  2.13it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1019/1250 [10:19<01:48,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1020/1250 [10:19<01:48,  2.13it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1021/1250 [10:20<01:46,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1022/1250 [10:20<01:46,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1023/1250 [10:21<01:45,  2.15it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1024/1250 [10:21<01:46,  2.13it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1025/1250 [10:22<01:44,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1026/1250 [10:22<01:44,  2.15it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1027/1250 [10:23<01:43,  2.15it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1028/1250 [10:23<01:43,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1029/1250 [10:24<01:43,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1030/1250 [10:24<01:42,  2.15it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 1031/1250 [10:25<01:41,  2.15it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1032/1250 [10:25<01:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1033/1250 [10:25<01:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1034/1250 [10:26<01:39,  2.16it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1035/1250 [10:26<01:39,  2.17it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1036/1250 [10:27<01:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1037/1250 [10:27<01:38,  2.16it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1038/1250 [10:28<01:38,  2.16it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1039/1250 [10:28<01:37,  2.16it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1040/1250 [10:29<01:37,  2.14it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1041/1250 [10:29<01:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1042/1250 [10:30<01:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 1043/1250 [10:30<01:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▎ | 1044/1250 [10:31<01:36,  2.13it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▎ | 1045/1250 [10:31<01:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▎ | 1046/1250 [10:32<01:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1047/1250 [10:32<01:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1048/1250 [10:32<01:34,  2.13it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1049/1250 [10:33<01:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1050/1250 [10:33<01:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1051/1250 [10:34<01:32,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1052/1250 [10:34<01:32,  2.14it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1053/1250 [10:35<01:31,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1054/1250 [10:35<01:31,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1055/1250 [10:36<01:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 1056/1250 [10:36<01:31,  2.13it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1057/1250 [10:37<01:30,  2.13it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1058/1250 [10:37<01:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1059/1250 [10:38<01:29,  2.14it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1060/1250 [10:38<01:29,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1061/1250 [10:39<01:29,  2.10it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 1062/1250 [10:39<01:29,  2.11it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1063/1250 [10:40<01:28,  2.11it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1064/1250 [10:40<01:27,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1065/1250 [10:40<01:27,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1066/1250 [10:41<01:26,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1067/1250 [10:41<01:26,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 1068/1250 [10:42<01:26,  2.10it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1069/1250 [10:42<01:26,  2.10it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1070/1250 [10:43<01:25,  2.11it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1071/1250 [10:43<01:24,  2.11it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1072/1250 [10:44<01:24,  2.11it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1073/1250 [10:44<01:23,  2.12it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1074/1250 [10:45<01:22,  2.13it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1075/1250 [10:45<01:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1076/1250 [10:46<01:21,  2.13it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1077/1250 [10:46<01:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 1078/1250 [10:47<01:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▋ | 1079/1250 [10:47<01:19,  2.15it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▋ | 1080/1250 [10:47<01:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▋ | 1081/1250 [10:48<01:18,  2.14it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1082/1250 [10:48<01:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1083/1250 [10:49<01:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1084/1250 [10:49<01:17,  2.13it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1085/1250 [10:50<01:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1086/1250 [10:50<01:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1087/1250 [10:51<01:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1088/1250 [10:51<01:16,  2.12it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1089/1250 [10:52<01:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1090/1250 [10:52<01:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1091/1250 [10:53<01:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1092/1250 [10:53<01:14,  2.13it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 1093/1250 [10:54<01:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1094/1250 [10:54<01:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1095/1250 [10:55<01:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1096/1250 [10:55<01:12,  2.12it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1097/1250 [10:55<01:11,  2.13it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1098/1250 [10:56<01:11,  2.13it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1099/1250 [10:56<01:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1100/1250 [10:57<01:10,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1101/1250 [10:57<01:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1102/1250 [10:58<01:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1103/1250 [10:58<01:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1104/1250 [10:59<01:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1105/1250 [10:59<01:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 1106/1250 [11:00<01:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▊ | 1107/1250 [11:00<01:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▊ | 1108/1250 [11:01<01:06,  2.12it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▊ | 1109/1250 [11:01<01:06,  2.13it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1110/1250 [11:02<01:05,  2.14it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1111/1250 [11:02<01:04,  2.15it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1112/1250 [11:02<01:04,  2.15it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1113/1250 [11:03<01:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1114/1250 [11:03<01:03,  2.15it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1115/1250 [11:04<01:02,  2.15it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1116/1250 [11:04<01:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1117/1250 [11:05<01:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 1118/1250 [11:05<01:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1119/1250 [11:06<01:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1120/1250 [11:06<01:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1121/1250 [11:07<01:00,  2.15it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1122/1250 [11:07<00:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1123/1250 [11:08<00:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 1124/1250 [11:08<00:58,  2.16it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1125/1250 [11:08<00:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.73it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.77it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.15it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.86it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.69it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.60it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.53it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.46it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.41it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.7185317277908325, 'eval_accuracy': 0.312, 'eval_f1': 0.3408747530246319, 'eval_precision': 0.2951857191785077, 'eval_recall': 0.47362621121431464, 'eval_runtime': 5.1409, 'eval_samples_per_second': 194.519, 'eval_steps_per_second': 3.112, 'epoch': 9.0}\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1125/1250 [11:14<00:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1126/1250 [11:14<04:20,  2.10s/it]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1127/1250 [11:15<03:17,  1.61s/it]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1128/1250 [11:15<02:34,  1.26s/it]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1129/1250 [11:16<02:04,  1.03s/it]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1130/1250 [11:16<01:43,  1.16it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 1131/1250 [11:17<01:28,  1.35it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1132/1250 [11:17<01:17,  1.52it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1133/1250 [11:18<01:10,  1.66it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1134/1250 [11:18<01:05,  1.78it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1135/1250 [11:19<01:01,  1.88it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1136/1250 [11:19<00:58,  1.96it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1137/1250 [11:20<00:56,  2.00it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1138/1250 [11:20<00:54,  2.04it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1139/1250 [11:20<00:53,  2.07it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 1140/1250 [11:21<00:52,  2.10it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████▏| 1141/1250 [11:21<00:51,  2.10it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████▏| 1142/1250 [11:22<00:50,  2.12it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████▏| 1143/1250 [11:22<00:50,  2.13it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1144/1250 [11:23<00:49,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1145/1250 [11:23<00:49,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1146/1250 [11:24<00:48,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1147/1250 [11:24<00:48,  2.13it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1148/1250 [11:25<00:47,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1149/1250 [11:25<00:47,  2.13it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1150/1250 [11:26<00:46,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1151/1250 [11:26<00:46,  2.15it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1152/1250 [11:27<00:45,  2.15it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1153/1250 [11:27<00:45,  2.13it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1154/1250 [11:27<00:44,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1155/1250 [11:28<00:44,  2.15it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 1156/1250 [11:28<00:43,  2.15it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1157/1250 [11:29<00:43,  2.13it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1158/1250 [11:29<00:42,  2.14it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1159/1250 [11:30<00:42,  2.14it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1160/1250 [11:30<00:41,  2.15it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1161/1250 [11:31<00:41,  2.13it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1162/1250 [11:31<00:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1163/1250 [11:32<00:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1164/1250 [11:32<00:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1165/1250 [11:33<00:39,  2.15it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1166/1250 [11:33<00:39,  2.14it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1167/1250 [11:34<00:38,  2.14it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 1168/1250 [11:34<00:38,  2.15it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▎| 1169/1250 [11:34<00:37,  2.14it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▎| 1170/1250 [11:35<00:37,  2.14it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▎| 1171/1250 [11:35<00:36,  2.14it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1172/1250 [11:36<00:36,  2.15it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1173/1250 [11:36<00:36,  2.13it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1174/1250 [11:37<00:35,  2.14it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1175/1250 [11:37<00:34,  2.15it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1176/1250 [11:38<00:34,  2.16it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1177/1250 [11:38<00:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1178/1250 [11:39<00:33,  2.13it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1179/1250 [11:39<00:33,  2.13it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1180/1250 [11:40<00:32,  2.13it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 1181/1250 [11:40<00:32,  2.12it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1182/1250 [11:41<00:32,  2.12it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1183/1250 [11:41<00:31,  2.14it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1184/1250 [11:41<00:30,  2.14it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1185/1250 [11:42<00:30,  2.12it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1186/1250 [11:42<00:30,  2.12it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 1187/1250 [11:43<00:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1188/1250 [11:43<00:28,  2.14it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1189/1250 [11:44<00:28,  2.13it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1190/1250 [11:44<00:28,  2.14it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1191/1250 [11:45<00:27,  2.15it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1192/1250 [11:45<00:26,  2.15it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 1193/1250 [11:46<00:26,  2.14it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1194/1250 [11:46<00:26,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1195/1250 [11:47<00:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1196/1250 [11:47<00:25,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1197/1250 [11:48<00:24,  2.13it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1198/1250 [11:48<00:24,  2.14it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1199/1250 [11:49<00:23,  2.14it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1200/1250 [11:49<00:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1201/1250 [11:49<00:22,  2.13it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1202/1250 [11:50<00:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 1203/1250 [11:50<00:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▋| 1204/1250 [11:51<00:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▋| 1205/1250 [11:51<00:21,  2.13it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▋| 1206/1250 [11:52<00:20,  2.14it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1207/1250 [11:52<00:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1208/1250 [11:53<00:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1209/1250 [11:53<00:19,  2.13it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1210/1250 [11:54<00:18,  2.13it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1211/1250 [11:54<00:18,  2.15it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1212/1250 [11:55<00:17,  2.15it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1213/1250 [11:55<00:17,  2.14it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1214/1250 [11:56<00:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1215/1250 [11:56<00:16,  2.15it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1216/1250 [11:56<00:15,  2.15it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1217/1250 [11:57<00:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 1218/1250 [11:57<00:14,  2.14it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1219/1250 [11:58<00:14,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1220/1250 [11:58<00:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1221/1250 [11:59<00:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1222/1250 [11:59<00:12,  2.16it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1223/1250 [12:00<00:12,  2.16it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1224/1250 [12:00<00:11,  2.17it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1225/1250 [12:01<00:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1226/1250 [12:01<00:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1227/1250 [12:02<00:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1228/1250 [12:02<00:10,  2.16it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1229/1250 [12:02<00:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1230/1250 [12:03<00:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 1231/1250 [12:03<00:08,  2.15it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▊| 1232/1250 [12:04<00:08,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▊| 1233/1250 [12:04<00:07,  2.15it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▊| 1234/1250 [12:05<00:07,  2.15it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1235/1250 [12:05<00:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1236/1250 [12:06<00:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1237/1250 [12:06<00:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1238/1250 [12:07<00:05,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1239/1250 [12:07<00:05,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1240/1250 [12:08<00:04,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1241/1250 [12:08<00:04,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1242/1250 [12:08<00:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 1243/1250 [12:09<00:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1244/1250 [12:09<00:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1245/1250 [12:10<00:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1246/1250 [12:10<00:01,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1247/1250 [12:11<00:01,  2.16it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1248/1250 [12:11<00:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 1249/1250 [12:12<00:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 1250/1250 [12:12<00:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34mBatch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.81it/s]#033[A\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.17it/s]#033[A\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.87it/s]#033[A\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.70it/s]#033[A\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.60it/s]#033[A\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.53it/s]#033[A\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]#033[A\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]#033[A\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]#033[A\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.44it/s]#033[A\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.42it/s]#033[A\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34m{'eval_loss': 1.9757916927337646, 'eval_accuracy': 0.271, 'eval_f1': 0.31387416133659707, 'eval_precision': 0.2855350351504376, 'eval_recall': 0.4095747381187562, 'eval_runtime': 5.1413, 'eval_samples_per_second': 194.504, 'eval_steps_per_second': 3.112, 'epoch': 10.0}\u001b[0m\n",
      "\u001b[34m100%|██████████| 1250/1250 [12:17<00:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m#015100%|██████████| 16/16 [00:04<00:00,  3.82it/s]#033[A\u001b[0m\n",
      "\u001b[34m#033[A\u001b[0m\n",
      "\u001b[34mTraining completed. Do not forget to share your model on huggingface.co/models =)\u001b[0m\n",
      "\u001b[34mTraining completed. Do not forget to share your model on huggingface.co/models =)\u001b[0m\n",
      "\u001b[34m{'train_runtime': 737.8363, 'train_samples_per_second': 54.199, 'train_steps_per_second': 1.694, 'train_loss': 1.7685637939453125, 'epoch': 10.0}\u001b[0m\n",
      "\u001b[34m100%|██████████| 1250/1250 [12:17<00:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 1250/1250 [12:17<00:00,  1.69it/s]\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34mThe following columns in the evaluation set  don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text.\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34mNum examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m***** Running Evaluation *****\n",
      "  Num examples = 1000\n",
      "  Batch size = 64\u001b[0m\n",
      "\u001b[34m0%|          | 0/16 [00:00<?, ?it/s]\u001b[0m\n",
      "\u001b[34m12%|█▎        | 2/16 [00:00<00:02,  6.85it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 3/16 [00:00<00:02,  4.81it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 4/16 [00:00<00:02,  4.17it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 5/16 [00:01<00:02,  3.85it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 6/16 [00:01<00:02,  3.68it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 7/16 [00:01<00:02,  3.59it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 8/16 [00:02<00:02,  3.52it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 9/16 [00:02<00:02,  3.49it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▎   | 10/16 [00:02<00:01,  3.47it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 11/16 [00:02<00:01,  3.45it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 12/16 [00:03<00:01,  3.43it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 13/16 [00:03<00:00,  3.41it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 14/16 [00:03<00:00,  3.40it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 15/16 [00:04<00:00,  3.40it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.80it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 16/16 [00:04<00:00,  3.70it/s]\u001b[0m\n",
      "\u001b[34m***** Eval results *****\u001b[0m\n",
      "\u001b[34mSaving model checkpoint to /opt/ml/model\u001b[0m\n",
      "\u001b[34mSaving model checkpoint to /opt/ml/model\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/config.json\u001b[0m\n",
      "\u001b[34mConfiguration saved in /opt/ml/model/config.json\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mModel weights saved in /opt/ml/model/pytorch_model.bin\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mtokenizer config file saved in /opt/ml/model/tokenizer_config.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/special_tokens_map.json\u001b[0m\n",
      "\u001b[34mSpecial tokens file saved in /opt/ml/model/special_tokens_map.json\u001b[0m\n",
      "\u001b[34m2022-05-23 14:20:47,683 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2022-05-23 14:21:02 Uploading - Uploading generated training model\n",
      "2022-05-23 14:24:43 Completed - Training job completed\n",
      "ProfilerReport-1653314373: IssuesFound\n",
      "Training seconds: 1403\n",
      "Billable seconds: 1403\n"
     ]
    }
   ],
   "source": [
    "# starting the train job with our uploaded datasets as input\n",
    "huggingface_estimator.fit({'train': training_input_path, 'test': test_input_path})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d72af83b",
   "metadata": {},
   "source": [
    "# Deploying the endpoint\n",
    "\n",
    "To deploy our endpoint, we call deploy() on our HuggingFace estimator object, passing in our desired number of instances and instance type.\n",
    "\n",
    "For inference, you can use your trained Hugging Face model or one of the pretrained Hugging Face models to deploy an inference job with SageMaker. With this collaboration, you only need one line of code to deploy both your trained models and pre-trained models with SageMaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "59a703d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------!"
     ]
    }
   ],
   "source": [
    "predictor = huggingface_estimator.deploy(1,\"ml.t2.medium\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fdb9ccd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'label': 'LABEL_8', 'score': 0.5507590770721436}]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentiment_input= {\"inputs\":'HISTORY OF PRESENT ILLNESS:,  The patient is well known to me for a history of iron-deficiency anemia due to chronic blood loss from colitis.  We corrected her hematocrit last year with intravenous (IV) iron.  Ultimately, she had a total proctocolectomy done on 03/14/2007 to treat her colitis.  Her course has been very complicated since then with needing multiple surgeries for removal of hematoma.  This is partly because she was on anticoagulation for a right arm deep venous thrombosis (DVT) she had early this year, complicated by septic phlebitis.,Chart was reviewed, and I will not reiterate her complex history.,I am asked to see the patient again because of concerns for coagulopathy.,She had surgery again last month to evacuate a pelvic hematoma, and was found to have vancomycin resistant enterococcus, for which she is on multiple antibiotics and followed by infectious disease now.,She is on total parenteral nutrition (TPN) as well.,LABORATORY DATA:,  Labs today showed a white blood count of 7.9, hemoglobin 11.0, hematocrit 32.8, and platelets 1,121,000.  MCV is 89.  Her platelets have been elevated for at least the past week, with counts initially at the 600,000 to 700,000 range and in the last couple of day rising above 1,000,000.  Her hematocrit has been essentially stable for the past month or so.  White blood count has improved.,PT has been markedly elevated and today is 44.9 with an INR of 5.0.  This is despite stopping Coumadin on 05/31/2007, and with administration of vitamin K via the TPN, as well as additional doses IV.'}\n",
    "predictor.predict(sentiment_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "659b83fe",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "Model prediction over evaluation set to be uploaded to s3 bucket for Quicksight Dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a95fff4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df = pd.read_csv('med-test.csv')\n",
    "#eval_df = pre_process(eval_df)\n",
    "eval_df['pred'] = 0\n",
    "eval_df['score'] = 0.0\n",
    "max_length = 512\n",
    "for i in range(len(eval_df)):\n",
    "    sentiment_input= {\"inputs\": eval_df['text'].iloc[i][:max_length]}\n",
    "    eval_df['pred'][i] = int(predictor.predict(sentiment_input)[0]['label'].split('_')[1])\n",
    "    eval_df['score'][i] = float(predictor.predict(sentiment_input)[0]['score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ce51631a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>text</th>\n",
       "      <th>pred</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22</td>\n",
       "      <td>operative note  the patient was taken to the o...</td>\n",
       "      <td>22</td>\n",
       "      <td>0.428610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13</td>\n",
       "      <td>vital signs  reveal a blood pressure of  tempe...</td>\n",
       "      <td>13</td>\n",
       "      <td>0.724876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>history  neurologic consultation was requested...</td>\n",
       "      <td>10</td>\n",
       "      <td>0.300070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>anatomical summary1 sharp force wound of neck ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0.629299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>indications for procedure the patient has pres...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.542188</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   target                                               text  pred     score\n",
       "0      22  operative note  the patient was taken to the o...    22  0.428610\n",
       "1      13  vital signs  reveal a blood pressure of  tempe...    13  0.724876\n",
       "2      10  history  neurologic consultation was requested...    10  0.300070\n",
       "3       0  anatomical summary1 sharp force wound of neck ...     0  0.629299\n",
       "4       1  indications for procedure the patient has pres...     1  0.542188"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "157ceda8",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df.to_csv(\"model-performance.csv\", index=False)\n",
    "s3 = boto3.resource('s3')\n",
    "s3.meta.client.upload_file('model-performance.csv', bucket, 'preprocess/pre-processed/eval/model-performance.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88e746d9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}