task9_bert_sentiment_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jNKaJz5j_ylj"
   },
   "source": [
    "# Определение эмоциональной окраски твитов с помощью BERT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Если Вы запускаете ноутбук на colab,\n",
    "# выполните следующие строчки, чтобы подгрузить библиотеку dlnlputils:\n",
    "\n",
    "# !git clone https://github.com/Samsung-IT-Academy/stepik-dl-nlp.git\n",
    "# import sys; sys.path.append('/content/stepik-dl-nlp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RX_ZDhicpHkV"
   },
   "source": [
    "## Установка библиотек"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 382
    },
    "colab_type": "code",
    "id": "0NmMdkZO8R6q",
    "outputId": "1cc59bfa-1dbb-4540-cb22-196f399f62af"
   },
   "outputs": [],
   "source": [
    "# !pip install pytorch-transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Ok002ceNB8E7",
    "outputId": "06ef90d2-7518-4209-da66-1dd45c357c78"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from sklearn.model_selection import train_test_split\n",
    "from pytorch_transformers import BertTokenizer, BertConfig\n",
    "from pytorch_transformers import AdamW, BertForSequenceClassification\n",
    "from tqdm import tqdm, trange\n",
    "import pandas as pd\n",
    "import io\n",
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "oYsV4H8fCpZ-",
    "outputId": "b8812c8e-3149-475f-b4c0-262160485c39"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "if device == 'cpu':\n",
    "    print('cpu')\n",
    "else:\n",
    "    n_gpu = torch.cuda.device_count()\n",
    "    print(torch.cuda.get_device_name(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "guw6ZNtaswKc"
   },
   "source": [
    "## Загрузка данных\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы выбрали необычный датасет с разметкой сентимента русскоязычных твитов (подробнее про него в [статье](http://www.swsys.ru/index.php?page=article&id=3962&lang=)). В корпусе, который мы использовали 114,911 положительных и 111,923 отрицательных записей. Загрузить его можно [тут](https://study.mokoron.com/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Если Вы запускаете ноутбук на colab, добавьте в начало пути /content/stepik-dl-nlp\n",
    "pos_texts = pd.read_csv('datasets/bert_sentiment_analysis/positive.csv', encoding='utf8', sep=';', header=None)\n",
    "neg_texts = pd.read_csv('datasets/bert_sentiment_analysis/negative.csv', encoding='utf8', sep=';', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pos_texts.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = np.concatenate([pos_texts[3].values, neg_texts[3].values])\n",
    "\n",
    "sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
    "labels = [[1] for _ in range(pos_texts.shape[0])] + [[0] for _ in range(neg_texts.shape[0])]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(sentences) == len(labels) == pos_texts.shape[0] + neg_texts.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sentences[1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_sentences, test_sentences, train_gt, test_gt = train_test_split(sentences, labels, test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(train_gt), len(test_gt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ex5O1eV-Pfct"
   },
   "source": [
    "## Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "Z474sSC6oe7A",
    "outputId": "fbaa8fd8-bccd-4feb-ce52-beba5d293cfa"
   },
   "outputs": [],
   "source": [
    "from pytorch_transformers import BertTokenizer, BertConfig\n",
    "\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
    "\n",
    "tokenized_texts = [tokenizer.tokenize(sent) for sent in train_sentences]\n",
    "print (tokenized_texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "87_kXUeT2-br"
   },
   "source": [
    "BERTу нужно предоставить специальный формат входных данных.\n",
    "\n",
    "\n",
    "- **input ids**: последовательность чисел, отождествляющих каждый токен с его номером в словаре.\n",
    "- **labels**: вектор из нулей и единиц. В нашем случае нули обозначают негативную эмоциональную окраску, единицы - положительную.\n",
    "- **segment mask**: (необязательно) последовательность нулей и единиц, которая показывает, состоит ли входной текст из одного или двух предложений. Для случая одного предложения получится вектор из одних нулей. Для двух: <length_of_sent_1> нулей и <length_of_sent_2> единиц.\n",
    "- **attention mask**: (необязательно) последовательность нулей и единиц, где единицы обозначают токены предложения, нули - паддинг."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cp9BPRd1tMIo"
   },
   "outputs": [],
   "source": [
    "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n",
    "input_ids = pad_sequences(\n",
    "    input_ids,\n",
    "    maxlen=100,\n",
    "    dtype=\"long\",\n",
    "    truncating=\"post\",\n",
    "    padding=\"post\"\n",
    ")\n",
    "attention_masks = [[float(i>0) for i in seq] for seq in input_ids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aFbE-UHvsb7-"
   },
   "outputs": [],
   "source": [
    "train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(\n",
    "    input_ids, train_gt, \n",
    "    random_state=42,\n",
    "    test_size=0.1\n",
    ")\n",
    "\n",
    "train_masks, validation_masks, _, _ = train_test_split(\n",
    "    attention_masks,\n",
    "    input_ids,\n",
    "    random_state=42,\n",
    "    test_size=0.1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jw5K2A5Ko1RF"
   },
   "outputs": [],
   "source": [
    "train_inputs = torch.tensor(train_inputs)\n",
    "train_labels = torch.tensor(train_labels)\n",
    "train_masks = torch.tensor(train_masks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_inputs = torch.tensor(validation_inputs)\n",
    "validation_labels = torch.tensor(validation_labels)\n",
    "validation_masks = torch.tensor(validation_masks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GEgLpFVlo1Z-"
   },
   "outputs": [],
   "source": [
    "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
    "train_dataloader = DataLoader(\n",
    "    train_data,\n",
    "    sampler=RandomSampler(train_data),\n",
    "    batch_size=32\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n",
    "validation_dataloader = DataLoader(\n",
    "    validation_data,\n",
    "    sampler=SequentialSampler(validation_data),\n",
    "    batch_size=32\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pNl8khAhPYju"
   },
   "source": [
    "## Обучение модели"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Загружаем [BertForSequenceClassification](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L1129):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_transformers import AdamW, BertForSequenceClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Аналогичные модели есть и для других задач:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_transformers import BertForQuestionAnswering, BertForTokenClassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "gFsCTp_mporB",
    "outputId": "dd067229-1925-4b37-f517-0c14e25420d1"
   },
   "outputs": [],
   "source": [
    "model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\n",
    "model.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QxSMw0FrptiL"
   },
   "outputs": [],
   "source": [
    "param_optimizer = list(model.named_parameters())\n",
    "no_decay = ['bias', 'gamma', 'beta']\n",
    "optimizer_grouped_parameters = [\n",
    "    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "     'weight_decay_rate': 0.01},\n",
    "    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "     'weight_decay_rate': 0.0}\n",
    "]\n",
    "\n",
    "optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 172
    },
    "colab_type": "code",
    "id": "6J-FYdx6nFE_",
    "outputId": "8e388ad1-f9db-4c7b-d080-6c0a0e964610"
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "# Будем сохранять loss во время обучения\n",
    "# и рисовать график в режиме реального времени\n",
    "train_loss_set = []\n",
    "train_loss = 0\n",
    "\n",
    "\n",
    "# Обучение\n",
    "# Переводим модель в training mode\n",
    "model.train()\n",
    "\n",
    "\n",
    "for step, batch in enumerate(train_dataloader):\n",
    "    # добавляем батч для вычисления на GPU\n",
    "    batch = tuple(t.to(device) for t in batch)\n",
    "    # Распаковываем данные из dataloader\n",
    "    b_input_ids, b_input_mask, b_labels = batch\n",
    "    \n",
    "    # если не сделать .zero_grad(), градиенты будут накапливаться\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n",
    "\n",
    "    train_loss_set.append(loss[0].item())  \n",
    "    \n",
    "    # Backward pass\n",
    "    loss[0].backward()\n",
    "    \n",
    "    # Обновляем параметры и делаем шаг используя посчитанные градиенты\n",
    "    optimizer.step()\n",
    "\n",
    "    # Обновляем loss\n",
    "    train_loss += loss[0].item()\n",
    "    \n",
    "    # Рисуем график\n",
    "    clear_output(True)\n",
    "    plt.plot(train_loss_set)\n",
    "    plt.title(\"Training loss\")\n",
    "    plt.xlabel(\"Batch\")\n",
    "    plt.ylabel(\"Loss\")\n",
    "    plt.show()\n",
    "    \n",
    "print(\"Loss на обучающей выборке: {0:.5f}\".format(train_loss / len(train_dataloader)))\n",
    "\n",
    "\n",
    "# Валидация\n",
    "# Переводим модель в evaluation mode\n",
    "model.eval()\n",
    "\n",
    "valid_preds, valid_labels = [], []\n",
    "\n",
    "for batch in validation_dataloader:   \n",
    "    # добавляем батч для вычисления на GPU\n",
    "    batch = tuple(t.to(device) for t in batch)\n",
    "    \n",
    "    # Распаковываем данные из dataloader\n",
    "    b_input_ids, b_input_mask, b_labels = batch\n",
    "    \n",
    "    # При использовании .no_grad() модель не будет считать и хранить градиенты.\n",
    "    # Это ускорит процесс предсказания меток для валидационных данных.\n",
    "    with torch.no_grad():\n",
    "        logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
    "\n",
    "    # Перемещаем logits и метки классов на CPU для дальнейшей работы\n",
    "    logits = logits[0].detach().cpu().numpy()\n",
    "    label_ids = b_labels.to('cpu').numpy()\n",
    "    \n",
    "    batch_preds = np.argmax(logits, axis=1)\n",
    "    batch_labels = np.concatenate(label_ids)     \n",
    "    valid_preds.extend(batch_preds)\n",
    "    valid_labels.extend(batch_labels)\n",
    "\n",
    "print(\"Процент правильных предсказаний на валидационной выборке: {0:.2f}%\".format(\n",
    "    accuracy_score(valid_labels, valid_preds) * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Процент правильных предсказаний на валидационной выборке: {0:.2f}%\".format(\n",
    "    accuracy_score(valid_labels, valid_preds) * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mkyubuJSOzg3"
   },
   "source": [
    "# Оценка качества на отложенной выборке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mAN0LZBOOPVh"
   },
   "outputs": [],
   "source": [
    "tokenized_texts = [tokenizer.tokenize(sent) for sent in test_sentences]\n",
    "input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n",
    "\n",
    "input_ids = pad_sequences(\n",
    "    input_ids,\n",
    "    maxlen=100,\n",
    "    dtype=\"long\",\n",
    "    truncating=\"post\",\n",
    "    padding=\"post\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "attention_masks = [[float(i>0) for i in seq] for seq in input_ids]\n",
    "\n",
    "prediction_inputs = torch.tensor(input_ids)\n",
    "prediction_masks = torch.tensor(attention_masks)\n",
    "prediction_labels = torch.tensor(test_gt)\n",
    "\n",
    "prediction_data = TensorDataset(\n",
    "    prediction_inputs,\n",
    "    prediction_masks,\n",
    "    prediction_labels\n",
    ")\n",
    "\n",
    "prediction_dataloader = DataLoader(\n",
    "    prediction_data, \n",
    "    sampler=SequentialSampler(prediction_data),\n",
    "    batch_size=32\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Hba10sXR7Xi6"
   },
   "outputs": [],
   "source": [
    "model.eval()\n",
    "test_preds, test_labels = [], []\n",
    "\n",
    "for batch in prediction_dataloader:\n",
    "    # добавляем батч для вычисления на GPU\n",
    "    batch = tuple(t.to(device) for t in batch)\n",
    "    \n",
    "    # Распаковываем данные из dataloader\n",
    "    b_input_ids, b_input_mask, b_labels = batch\n",
    "    \n",
    "    # При использовании .no_grad() модель не будет считать и хранить градиенты.\n",
    "    # Это ускорит процесс предсказания меток для тестовых данных.\n",
    "    with torch.no_grad():\n",
    "        logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
    "\n",
    "    # Перемещаем logits и метки классов на CPU для дальнейшей работы\n",
    "    logits = logits[0].detach().cpu().numpy()\n",
    "    label_ids = b_labels.to('cpu').numpy()\n",
    "\n",
    "    # Сохраняем предсказанные классы и ground truth\n",
    "    batch_preds = np.argmax(logits, axis=1)\n",
    "    batch_labels = np.concatenate(label_ids)  \n",
    "    test_preds.extend(batch_preds)\n",
    "    test_labels.extend(batch_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_score = accuracy_score(test_labels, test_preds)\n",
    "print('Процент правильных предсказаний на отложенной выборке составил: {0:.2f}%'.format(\n",
    "    acc_score*100\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Неправильных предсказаний: {0}/{1}'.format(\n",
    "    sum(test_labels != test_preds),\n",
    "    len(test_labels)\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Домашнее задание"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Скачайте датасет с отзывами на фильмы. Например, используйте датасет [IMDB Dataset of 50K Movie Reviews](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "dataset = pd.read_csv('datasets/bert_sentiment_analysis/homework/IMDB_Dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>One of the other reviewers has mentioned that ...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A wonderful little production. &lt;br /&gt;&lt;br /&gt;The...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I thought this was a wonderful way to spend ti...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Basically there's a family where a little boy ...</td>\n",
       "      <td>negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Petter Mattei's \"Love in the Time of Money\" is...</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review sentiment\n",
       "0  One of the other reviewers has mentioned that ...  positive\n",
       "1  A wonderful little production. <br /><br />The...  positive\n",
       "2  I thought this was a wonderful way to spend ti...  positive\n",
       "3  Basically there's a family where a little boy ...  negative\n",
       "4  Petter Mattei's \"Love in the Time of Money\" is...  positive"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Используйте для дообучения BERT датасет IMDB. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ответьте на вопросы:\n",
    "1. удалось ли достичь такого же accuracy (98\\%) при использовании IMDB датасета?\n",
    "2. удалось ли получить хорошее качество классификации всего за одну эпоху?\n",
    "3. подумайте, в чем может быть причина различий в дообучении одной и той же модели на разных датасетах\n",
    "    - Внимательно изучите датасет с русскими твитами. В чем его особенности? Нет ли явных паттернов или ключевых слов, которые однозначно определяют сентимент твита?\n",
    "    - Попробуйте удалить пунктуацию из датасета с русскими твитами и перезапустите дообучение модели. Изменилось ли итоговое качество работы модели? Почему?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "BERT Fine-Tuning Sentence Classification.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}