examples/1_bias_bios_no_hydra.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import yaml, json\n",
    "from types import SimpleNamespace\n",
    "import pytorch_lightning as pl\n",
    "from pytorch_lightning import seed_everything\n",
    "from pytorch_lightning.callbacks import ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Bias in Bios $-$ ProfTeacher\n",
    "***This tutorial is a minimal example of what needs to be defined,\n",
    "and how to intertwine it, so that you can train Weasel, without Hydra as Config manager.***\n",
    "<br>$\\rightarrow$ *For an extended, more feature-rich version of this notebook, check the [1_bias_bios_full script](1_bias_bios_full.py) out (including logging to wandb, callbacks, and how to retrieve your stand-alone end-model after training it with Weasel).*\n",
    "<br>$\\rightarrow$ *For the recommended config managing with Hydra, check [this notebook](./1_bias_bios.ipynb) out*\n",
    "\n",
    "First let's load some pre-defined dataset & model parameters and hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "namespace(seed=3,\n",
       "          datamodule=namespace(batch_size=64,\n",
       "                               val_test_split=[250, -1],\n",
       "                               seed=3),\n",
       "          trainer=namespace(gpus=0, max_epochs=100),\n",
       "          end_model=namespace(dropout=0.3,\n",
       "                              net_norm='none',\n",
       "                              activation_func='ReLU',\n",
       "                              input_dim=300,\n",
       "                              hidden_dims=[50, 50, 25],\n",
       "                              output_dim=2,\n",
       "                              adjust_thresh=True),\n",
       "          Weasel=namespace(num_LFs=99,\n",
       "                           n_classes=2,\n",
       "                           class_balance=[0.5, 0.5],\n",
       "                           loss_function='cross_entropy',\n",
       "                           temperature=2.0,\n",
       "                           accuracy_scaler='sqrt',\n",
       "                           use_aux_input_for_encoder=True,\n",
       "                           class_conditional_accuracies=True,\n",
       "                           encoder=namespace(dropout=0.3,\n",
       "                                             net_norm='batch_norm',\n",
       "                                             activation_func='ReLU',\n",
       "                                             hidden_dims=[70, 70]),\n",
       "                           optim_end_model=namespace(name='Adam',\n",
       "                                                     lr='1e-4',\n",
       "                                                     weight_decay='7e-7'),\n",
       "                           optim_encoder=namespace(name='Adam',\n",
       "                                                   lr='1e-4',\n",
       "                                                   weight_decay='7e-7'),\n",
       "                           scheduler=None))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(f\"configs/profTeacher_no_hydra.yaml\") as f:\n",
    "    config_dict = yaml.load(f)\n",
    "    config = json.loads(json.dumps(config_dict), object_hook=lambda d: SimpleNamespace(**d))\n",
    "    seed_everything(config.seed, workers=True)  # seed for reproducibility\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Data in WeaSEL\n",
    "\n",
    "In this example we have $C=2$ classes (teacher or professor biography within a text document).\n",
    "\n",
    "We can make use of data features, *X_train* and *X_test*, for all training and test data points.\n",
    "Eventually we hope to make predictions with our end-model based solely on these features.\n",
    "\n",
    "However, what if we don't have labels, Y, for our training examples, i.e. can't train our end-model\n",
    "in the traditional supervised way on (X, Y) examples?\n",
    "This is where multi-source weak supervision & Weasel come to the rescue! :D\n",
    "\n",
    "#### Multiple Labeling Heuristics replace ground truth training labels\n",
    "For this concrete problem, we created $m=99$ regex-based LFs, that we then applied on the $n=12.5k$ training examples, which gives us\n",
    "a label matrix $L$ of shape $n \\times m$ with values in $\\{-1, 0, .., C-1\\} = \\{-1, 0, 1\\}$.\n",
    "\n",
    "Here, $-1$ means that a LF abstained from labeling, while $0, 1$ indicate that the LF believes that the particular\n",
    "example is a teacher, professor biography, respectively.\n",
    "\n",
    "#### Optional, but recommended evaluation of the end-model\n",
    "Lastly, we will also want to evaluate the skill of our end-model on a small test set that contains\n",
    "ground truth labels *Y_test* (with the corresponding features *X_test*).\n",
    "\n",
    "### All in one place\n",
    "\n",
    "All this is conveniently encapsulated in an [abstract PyTorch Lightning DataModule](../weasel/datamodules/base_datamodule.py),\n",
    " that makes it simple for you to set up your own DataModule to easily train your end-model, provided that you already have all of the above at hand:\n",
    "\n",
    "- Training and Test features, *X_train, X_test*\n",
    "- Label matrix, *L*\n",
    "- Ground truth test labels, *Y_test*\n",
    "\n",
    "See the [code that defines ProfTeacher_DataModule](datamodules/ProfTeacher_datamodule.py)\n",
    "to see how simple it then is to get started.\n",
    "Alternatively to the approach of the ``ProfTeacher_DataModule`` that encapsulates the data loading within the DataModule itself,\n",
    " you may also create a ``base_datamodule.BasicWeaselDataModule`` by passing it in the constructor the four data components above,\n",
    " see [this notebook](0_full_pipeline.ipynb) for a simple synthetic example of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from examples.datamodules.ProfTeacher_datamodule import ProfTeacher_DataModule\n",
    "data_module = ProfTeacher_DataModule(**vars(config.datamodule))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## End-model\n",
    "\n",
    "Having set up the data part, you'll first have to choose your favorite neural net as the end-model <br>\n",
    " (the one that you want to use as ``predictions = end-model(X)`` eventually).\n",
    "\n",
    "Here, we'll use a simple 2-layer feed-forward net/MLP, but you\n",
    "can easily replace it with *any* neural net model, see the instructions in the Readme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from weasel.models.downstream_models.MLP import MLPNet\n",
    "endmodel = MLPNet(**vars(config.end_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Weasel: Marrying your end-model with the LFs\n",
    "We now pass this end-model to the wrapping Weasel model, which will take care of learning the end-model based\n",
    "on *X_train* and the LFs, i.e. the label matrix *L*.\n",
    "To do so, there is an encoder net (another MLP in this case) in Weasel's core, that will be predicting labels based on the\n",
    "LFs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from weasel.models import Weasel\n",
    "weasel = Weasel(**vars(config.Weasel), end_model=endmodel)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Training Weasel and end-model\n",
    "\n",
    "Before fitting Weasel and the end-model, we now just need to instantiate a pl.Trainer instance\n",
    "(we will checkpoint the best model w.r.t. AUC performance on a small validation set that is split off the test set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "\n",
      "  | Name          | Type       | Params\n",
      "---------------------------------------------\n",
      "0 | end_model     | MLPNet     | 18.9 K\n",
      "1 | encoder       | MLPEncoder | 47.3 K\n",
      "2 | accuracy_func | Softmax    | 0     \n",
      "---------------------------------------------\n",
      "66.2 K    Trainable params\n",
      "0         Non-trainable params\n",
      "66.2 K    Total params\n",
      "0.265     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                              "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:  98%|█████████▊| 193/197 [00:03<00:00, 48.32it/s, Val/accuracy=0.641, Val/recall=0.846, Val/precision=0.604, Val/f1=0.705, Val/auc=0.593, decision_thresh=0.501]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 0: 100%|██████████| 197/197 [00:04<00:00, 48.36it/s, Val/accuracy=0.680, Val/recall=0.951, Val/precision=0.611, Val/f1=0.744, Val/auc=0.784, decision_thresh=0.499]\n",
      "Epoch 1:  98%|█████████▊| 193/197 [00:04<00:00, 47.76it/s, Val/accuracy=0.680, Val/recall=0.951, Val/precision=0.611, Val/f1=0.744, Val/auc=0.784, decision_thresh=0.499]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 1: 100%|██████████| 197/197 [00:04<00:00, 47.82it/s, Val/accuracy=0.780, Val/recall=0.918, Val/precision=0.713, Val/f1=0.803, Val/auc=0.864, decision_thresh=0.501]\n",
      "Epoch 2:  98%|█████████▊| 193/197 [00:03<00:00, 49.66it/s, Val/accuracy=0.780, Val/recall=0.918, Val/precision=0.713, Val/f1=0.803, Val/auc=0.864, decision_thresh=0.501]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 2: 100%|██████████| 197/197 [00:03<00:00, 49.69it/s, Val/accuracy=0.872, Val/recall=0.877, Val/precision=0.863, Val/f1=0.870, Val/auc=0.913, decision_thresh=0.502]\n",
      "Epoch 3:  98%|█████████▊| 193/197 [00:03<00:00, 49.80it/s, Val/accuracy=0.872, Val/recall=0.877, Val/precision=0.863, Val/f1=0.870, Val/auc=0.913, decision_thresh=0.502]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 3: 100%|██████████| 197/197 [00:03<00:00, 49.83it/s, Val/accuracy=0.888, Val/recall=0.902, Val/precision=0.873, Val/f1=0.887, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 4:  98%|█████████▊| 193/197 [00:03<00:00, 49.15it/s, Val/accuracy=0.888, Val/recall=0.902, Val/precision=0.873, Val/f1=0.887, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 4: 100%|██████████| 197/197 [00:04<00:00, 49.19it/s, Val/accuracy=0.888, Val/recall=0.910, Val/precision=0.867, Val/f1=0.888, Val/auc=0.935, decision_thresh=0.504]\n",
      "Epoch 5:  98%|█████████▊| 193/197 [00:03<00:00, 49.15it/s, Val/accuracy=0.888, Val/recall=0.910, Val/precision=0.867, Val/f1=0.888, Val/auc=0.935, decision_thresh=0.504]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 5: 100%|██████████| 197/197 [00:04<00:00, 49.17it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 6:  98%|█████████▊| 193/197 [00:03<00:00, 49.74it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 6: 100%|██████████| 197/197 [00:03<00:00, 49.81it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 7:  98%|█████████▊| 193/197 [00:03<00:00, 49.24it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 7: 100%|██████████| 197/197 [00:03<00:00, 49.27it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.935, decision_thresh=0.499]\n",
      "Epoch 8:  98%|█████████▊| 193/197 [00:03<00:00, 49.48it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.935, decision_thresh=0.499]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 8: 100%|██████████| 197/197 [00:03<00:00, 49.45it/s, Val/accuracy=0.896, Val/recall=0.918, Val/precision=0.875, Val/f1=0.896, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 9:  98%|█████████▊| 193/197 [00:03<00:00, 49.20it/s, Val/accuracy=0.896, Val/recall=0.918, Val/precision=0.875, Val/f1=0.896, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 9: 100%|██████████| 197/197 [00:04<00:00, 49.19it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 10:  98%|█████████▊| 193/197 [00:03<00:00, 48.78it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 10: 100%|██████████| 197/197 [00:04<00:00, 48.82it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 11:  98%|█████████▊| 193/197 [00:03<00:00, 49.61it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 11: 100%|██████████| 197/197 [00:03<00:00, 49.63it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.934, decision_thresh=0.498]\n",
      "Epoch 12:  98%|█████████▊| 193/197 [00:03<00:00, 49.70it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.934, decision_thresh=0.498]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 12: 100%|██████████| 197/197 [00:03<00:00, 49.71it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 13:  98%|█████████▊| 193/197 [00:03<00:00, 48.81it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 13: 100%|██████████| 197/197 [00:04<00:00, 48.85it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 14:  98%|█████████▊| 193/197 [00:03<00:00, 49.82it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 14: 100%|██████████| 197/197 [00:03<00:00, 49.87it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.933, decision_thresh=0.502]\n",
      "Epoch 15:  98%|█████████▊| 193/197 [00:04<00:00, 48.18it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.933, decision_thresh=0.502]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 15: 100%|██████████| 197/197 [00:04<00:00, 48.20it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.501]\n",
      "Epoch 16:  98%|█████████▊| 193/197 [00:03<00:00, 48.97it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.501]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 16: 100%|██████████| 197/197 [00:04<00:00, 49.01it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 17:  98%|█████████▊| 193/197 [00:03<00:00, 49.50it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 17: 100%|██████████| 197/197 [00:03<00:00, 49.51it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 18:  98%|█████████▊| 193/197 [00:04<00:00, 48.24it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 18: 100%|██████████| 197/197 [00:04<00:00, 48.23it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.501]\n",
      "Epoch 19:  98%|█████████▊| 193/197 [00:03<00:00, 49.00it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.501]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 19: 100%|██████████| 197/197 [00:04<00:00, 49.04it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 20:  98%|█████████▊| 193/197 [00:03<00:00, 48.54it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 20: 100%|██████████| 197/197 [00:04<00:00, 48.60it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 21:  98%|█████████▊| 193/197 [00:03<00:00, 49.01it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 21: 100%|██████████| 197/197 [00:04<00:00, 49.07it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 22:  98%|█████████▊| 193/197 [00:03<00:00, 48.46it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 22: 100%|██████████| 197/197 [00:04<00:00, 48.46it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 23:  98%|█████████▊| 193/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 23: 100%|██████████| 197/197 [00:04<00:00, 48.02it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 24:  98%|█████████▊| 193/197 [00:03<00:00, 48.36it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 24: 100%|██████████| 197/197 [00:04<00:00, 48.39it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 25:  98%|█████████▊| 193/197 [00:03<00:00, 49.86it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 25: 100%|██████████| 197/197 [00:03<00:00, 49.91it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 26:  98%|█████████▊| 193/197 [00:03<00:00, 49.85it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 26: 100%|██████████| 197/197 [00:03<00:00, 49.87it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 27:  98%|█████████▊| 193/197 [00:03<00:00, 50.18it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 27: 100%|██████████| 197/197 [00:03<00:00, 50.15it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 28:  98%|█████████▊| 193/197 [00:03<00:00, 48.65it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 28: 100%|██████████| 197/197 [00:04<00:00, 48.68it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 29:  98%|█████████▊| 193/197 [00:03<00:00, 49.85it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 29: 100%|██████████| 197/197 [00:03<00:00, 49.86it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 30:  98%|█████████▊| 193/197 [00:03<00:00, 49.30it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 30: 100%|██████████| 197/197 [00:03<00:00, 49.25it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 31:  98%|█████████▊| 193/197 [00:03<00:00, 48.52it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 31: 100%|██████████| 197/197 [00:04<00:00, 48.57it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 32:  98%|█████████▊| 193/197 [00:03<00:00, 49.97it/s, Val/accuracy=0.908, Val/recall=0.910, Val/precision=0.902, Val/f1=0.906, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 32: 100%|██████████| 197/197 [00:03<00:00, 49.93it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 33:  98%|█████████▊| 193/197 [00:04<00:00, 47.73it/s, Val/accuracy=0.904, Val/recall=0.910, Val/precision=0.895, Val/f1=0.902, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 33: 100%|██████████| 197/197 [00:04<00:00, 47.77it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.930, decision_thresh=0.499]\n",
      "Epoch 34:  98%|█████████▊| 193/197 [00:04<00:00, 48.17it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.930, decision_thresh=0.499]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 34: 100%|██████████| 197/197 [00:04<00:00, 48.17it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 35:  98%|█████████▊| 193/197 [00:03<00:00, 49.08it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 35: 100%|██████████| 197/197 [00:04<00:00, 49.12it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 36:  98%|█████████▊| 193/197 [00:03<00:00, 49.83it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 36: 100%|██████████| 197/197 [00:03<00:00, 49.87it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 37:  98%|█████████▊| 193/197 [00:03<00:00, 50.17it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 37: 100%|██████████| 197/197 [00:03<00:00, 50.17it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 38:  98%|█████████▊| 193/197 [00:03<00:00, 48.45it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 38: 100%|██████████| 197/197 [00:04<00:00, 48.53it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Epoch 39:  98%|█████████▊| 193/197 [00:03<00:00, 49.55it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.930, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 39: 100%|██████████| 197/197 [00:03<00:00, 49.56it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 40:  98%|█████████▊| 193/197 [00:03<00:00, 50.12it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 40: 100%|██████████| 197/197 [00:03<00:00, 50.13it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 41:  98%|█████████▊| 193/197 [00:03<00:00, 50.10it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 41: 100%|██████████| 197/197 [00:03<00:00, 50.15it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 42:  98%|█████████▊| 193/197 [00:03<00:00, 49.47it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 42: 100%|██████████| 197/197 [00:03<00:00, 49.47it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.931, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 43:  98%|█████████▊| 193/197 [00:03<00:00, 49.64it/s, Val/accuracy=0.896, Val/recall=0.902, Val/precision=0.887, Val/f1=0.894, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 43: 100%|██████████| 197/197 [00:03<00:00, 49.66it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 44:  98%|█████████▊| 193/197 [00:03<00:00, 49.58it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 44: 100%|██████████| 197/197 [00:03<00:00, 49.63it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 45:  98%|█████████▊| 193/197 [00:03<00:00, 50.32it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 45: 100%|██████████| 197/197 [00:03<00:00, 50.32it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 46:  98%|█████████▊| 193/197 [00:03<00:00, 48.77it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 46: 100%|██████████| 197/197 [00:04<00:00, 48.80it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Epoch 47:  98%|█████████▊| 193/197 [00:03<00:00, 48.70it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.931, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 47: 100%|██████████| 197/197 [00:04<00:00, 48.72it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 48:  98%|█████████▊| 193/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.896, Val/recall=0.910, Val/precision=0.881, Val/f1=0.895, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 48: 100%|██████████| 197/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 49:  98%|█████████▊| 193/197 [00:03<00:00, 49.53it/s, Val/accuracy=0.900, Val/recall=0.893, Val/precision=0.901, Val/f1=0.897, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 49: 100%|██████████| 197/197 [00:03<00:00, 49.55it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.932, decision_thresh=0.500]\n",
      "Epoch 50:  98%|█████████▊| 193/197 [00:03<00:00, 49.39it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.932, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 50: 100%|██████████| 197/197 [00:03<00:00, 49.43it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Epoch 51:  98%|█████████▊| 193/197 [00:03<00:00, 49.41it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.933, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 51: 100%|██████████| 197/197 [00:03<00:00, 49.45it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 52:  98%|█████████▊| 193/197 [00:03<00:00, 49.86it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 52: 100%|██████████| 197/197 [00:03<00:00, 49.88it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 53:  98%|█████████▊| 193/197 [00:04<00:00, 48.04it/s, Val/accuracy=0.904, Val/recall=0.902, Val/precision=0.902, Val/f1=0.902, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 53: 100%|██████████| 197/197 [00:04<00:00, 48.08it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 54:  98%|█████████▊| 193/197 [00:03<00:00, 48.78it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 54: 100%|██████████| 197/197 [00:04<00:00, 48.87it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 55:  98%|█████████▊| 193/197 [00:03<00:00, 49.76it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 55: 100%|██████████| 197/197 [00:03<00:00, 49.76it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 56:  98%|█████████▊| 193/197 [00:03<00:00, 48.88it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 56: 100%|██████████| 197/197 [00:04<00:00, 48.88it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Epoch 57:  98%|█████████▊| 193/197 [00:03<00:00, 49.39it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.934, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 57: 100%|██████████| 197/197 [00:03<00:00, 49.43it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 58:  98%|█████████▊| 193/197 [00:03<00:00, 48.35it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 58: 100%|██████████| 197/197 [00:04<00:00, 48.38it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 59:  98%|█████████▊| 193/197 [00:03<00:00, 49.34it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 59: 100%|██████████| 197/197 [00:03<00:00, 49.39it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Epoch 60:  98%|█████████▊| 193/197 [00:03<00:00, 50.73it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.935, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 60: 100%|██████████| 197/197 [00:03<00:00, 50.78it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 61:  98%|█████████▊| 193/197 [00:03<00:00, 50.52it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 61: 100%|██████████| 197/197 [00:03<00:00, 50.49it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 62:  98%|█████████▊| 193/197 [00:03<00:00, 49.85it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 62: 100%|██████████| 197/197 [00:03<00:00, 49.87it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Epoch 63:  98%|█████████▊| 193/197 [00:03<00:00, 49.52it/s, Val/accuracy=0.900, Val/recall=0.926, Val/precision=0.876, Val/f1=0.900, Val/auc=0.936, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 63: 100%|██████████| 197/197 [00:03<00:00, 49.53it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 64:  98%|█████████▊| 193/197 [00:04<00:00, 48.02it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 64: 100%|██████████| 197/197 [00:04<00:00, 48.11it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 65:  98%|█████████▊| 193/197 [00:03<00:00, 50.04it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 65: 100%|██████████| 197/197 [00:03<00:00, 50.03it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 66:  98%|█████████▊| 193/197 [00:03<00:00, 48.29it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 66: 100%|██████████| 197/197 [00:04<00:00, 48.36it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Epoch 67:  98%|█████████▊| 193/197 [00:03<00:00, 48.52it/s, Val/accuracy=0.900, Val/recall=0.934, Val/precision=0.870, Val/f1=0.901, Val/auc=0.937, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 67: 100%|██████████| 197/197 [00:04<00:00, 48.57it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.938, decision_thresh=0.500]\n",
      "Epoch 68:  98%|█████████▊| 193/197 [00:03<00:00, 49.19it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.938, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 68: 100%|██████████| 197/197 [00:04<00:00, 49.21it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.939, decision_thresh=0.500]\n",
      "Epoch 69:  98%|█████████▊| 193/197 [00:03<00:00, 49.63it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.939, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 69: 100%|██████████| 197/197 [00:03<00:00, 49.66it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 70:  98%|█████████▊| 193/197 [00:04<00:00, 47.98it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 70: 100%|██████████| 197/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.940, decision_thresh=0.500]\n",
      "Epoch 71:  98%|█████████▊| 193/197 [00:04<00:00, 46.84it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.940, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 71: 100%|██████████| 197/197 [00:04<00:00, 46.87it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 72:  98%|█████████▊| 193/197 [00:04<00:00, 47.88it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 72: 100%|██████████| 197/197 [00:04<00:00, 47.97it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.942, decision_thresh=0.500]\n",
      "Epoch 73:  98%|█████████▊| 193/197 [00:04<00:00, 47.93it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.942, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 73: 100%|██████████| 197/197 [00:04<00:00, 47.98it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.943, decision_thresh=0.500]\n",
      "Epoch 74:  98%|█████████▊| 193/197 [00:04<00:00, 43.06it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.943, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 74: 100%|██████████| 197/197 [00:04<00:00, 43.01it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.943, decision_thresh=0.500]\n",
      "Epoch 75:  98%|█████████▊| 193/197 [00:04<00:00, 40.70it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.943, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 75: 100%|██████████| 197/197 [00:04<00:00, 40.70it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.945, decision_thresh=0.500]\n",
      "Epoch 76:  98%|█████████▊| 193/197 [00:04<00:00, 41.58it/s, Val/accuracy=0.908, Val/recall=0.934, Val/precision=0.884, Val/f1=0.908, Val/auc=0.945, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 76: 100%|██████████| 197/197 [00:04<00:00, 41.59it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.944, decision_thresh=0.500]\n",
      "Epoch 77:  98%|█████████▊| 193/197 [00:04<00:00, 41.57it/s, Val/accuracy=0.904, Val/recall=0.934, Val/precision=0.877, Val/f1=0.905, Val/auc=0.944, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 77: 100%|██████████| 197/197 [00:04<00:00, 41.71it/s, Val/accuracy=0.912, Val/recall=0.926, Val/precision=0.897, Val/f1=0.911, Val/auc=0.945, decision_thresh=0.500]\n",
      "Epoch 78:  98%|█████████▊| 193/197 [00:03<00:00, 49.27it/s, Val/accuracy=0.912, Val/recall=0.926, Val/precision=0.897, Val/f1=0.911, Val/auc=0.945, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 78: 100%|██████████| 197/197 [00:03<00:00, 49.30it/s, Val/accuracy=0.908, Val/recall=0.926, Val/precision=0.890, Val/f1=0.908, Val/auc=0.945, decision_thresh=0.500]\n",
      "Epoch 79:  98%|█████████▊| 193/197 [00:04<00:00, 48.06it/s, Val/accuracy=0.908, Val/recall=0.926, Val/precision=0.890, Val/f1=0.908, Val/auc=0.945, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 79: 100%|██████████| 197/197 [00:04<00:00, 48.09it/s, Val/accuracy=0.908, Val/recall=0.926, Val/precision=0.890, Val/f1=0.908, Val/auc=0.947, decision_thresh=0.500]\n",
      "Epoch 80:  98%|█████████▊| 193/197 [00:03<00:00, 48.42it/s, Val/accuracy=0.908, Val/recall=0.926, Val/precision=0.890, Val/f1=0.908, Val/auc=0.947, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 80: 100%|██████████| 197/197 [00:04<00:00, 48.47it/s, Val/accuracy=0.912, Val/recall=0.926, Val/precision=0.897, Val/f1=0.911, Val/auc=0.948, decision_thresh=0.500]\n",
      "Epoch 81:  98%|█████████▊| 193/197 [00:04<00:00, 48.03it/s, Val/accuracy=0.912, Val/recall=0.926, Val/precision=0.897, Val/f1=0.911, Val/auc=0.948, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 81: 100%|██████████| 197/197 [00:04<00:00, 48.09it/s, Val/accuracy=0.916, Val/recall=0.910, Val/precision=0.917, Val/f1=0.914, Val/auc=0.950, decision_thresh=0.500]\n",
      "Epoch 82:  98%|█████████▊| 193/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.916, Val/recall=0.910, Val/precision=0.917, Val/f1=0.914, Val/auc=0.950, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 82: 100%|██████████| 197/197 [00:04<00:00, 48.03it/s, Val/accuracy=0.920, Val/recall=0.910, Val/precision=0.925, Val/f1=0.917, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 83:  98%|█████████▊| 193/197 [00:04<00:00, 47.77it/s, Val/accuracy=0.920, Val/recall=0.910, Val/precision=0.925, Val/f1=0.917, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 83: 100%|██████████| 197/197 [00:04<00:00, 47.77it/s, Val/accuracy=0.916, Val/recall=0.902, Val/precision=0.924, Val/f1=0.913, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 84:  98%|█████████▊| 193/197 [00:03<00:00, 49.38it/s, Val/accuracy=0.916, Val/recall=0.902, Val/precision=0.924, Val/f1=0.913, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 84: 100%|██████████| 197/197 [00:03<00:00, 49.43it/s, Val/accuracy=0.912, Val/recall=0.918, Val/precision=0.903, Val/f1=0.911, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 85:  98%|█████████▊| 193/197 [00:04<00:00, 47.68it/s, Val/accuracy=0.912, Val/recall=0.918, Val/precision=0.903, Val/f1=0.911, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 85: 100%|██████████| 197/197 [00:04<00:00, 47.67it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.950, decision_thresh=0.500]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 86:  98%|█████████▊| 193/197 [00:04<00:00, 47.77it/s, Val/accuracy=0.908, Val/recall=0.918, Val/precision=0.896, Val/f1=0.907, Val/auc=0.950, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 86: 100%|██████████| 197/197 [00:04<00:00, 46.81it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 87:  98%|█████████▊| 193/197 [00:04<00:00, 47.22it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 87: 100%|██████████| 197/197 [00:04<00:00, 47.31it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 88:  98%|█████████▊| 193/197 [00:04<00:00, 47.99it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 88: 100%|██████████| 197/197 [00:04<00:00, 48.06it/s, Val/accuracy=0.904, Val/recall=0.877, Val/precision=0.922, Val/f1=0.899, Val/auc=0.950, decision_thresh=0.500]\n",
      "Epoch 89:  98%|█████████▊| 193/197 [00:04<00:00, 47.27it/s, Val/accuracy=0.904, Val/recall=0.877, Val/precision=0.922, Val/f1=0.899, Val/auc=0.950, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 89: 100%|██████████| 197/197 [00:04<00:00, 47.32it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.950, decision_thresh=0.500]\n",
      "Epoch 90:  98%|█████████▊| 193/197 [00:04<00:00, 46.98it/s, Val/accuracy=0.900, Val/recall=0.902, Val/precision=0.894, Val/f1=0.898, Val/auc=0.950, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 90: 100%|██████████| 197/197 [00:04<00:00, 47.03it/s, Val/accuracy=0.896, Val/recall=0.943, Val/precision=0.858, Val/f1=0.898, Val/auc=0.950, decision_thresh=0.500]\n",
      "Epoch 91:  98%|█████████▊| 193/197 [00:04<00:00, 47.66it/s, Val/accuracy=0.896, Val/recall=0.943, Val/precision=0.858, Val/f1=0.898, Val/auc=0.950, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 91: 100%|██████████| 197/197 [00:04<00:00, 47.65it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 92:  98%|█████████▊| 193/197 [00:04<00:00, 47.37it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 92: 100%|██████████| 197/197 [00:04<00:00, 47.39it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 93:  98%|█████████▊| 193/197 [00:04<00:00, 47.17it/s, Val/accuracy=0.904, Val/recall=0.918, Val/precision=0.889, Val/f1=0.903, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 93: 100%|██████████| 197/197 [00:04<00:00, 47.24it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 94:  98%|█████████▊| 193/197 [00:04<00:00, 46.96it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 94: 100%|██████████| 197/197 [00:04<00:00, 47.06it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 95:  98%|█████████▊| 193/197 [00:04<00:00, 47.50it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 95: 100%|██████████| 197/197 [00:04<00:00, 47.56it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Epoch 96:  98%|█████████▊| 193/197 [00:04<00:00, 47.67it/s, Val/accuracy=0.896, Val/recall=0.951, Val/precision=0.853, Val/f1=0.899, Val/auc=0.951, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 96: 100%|██████████| 197/197 [00:04<00:00, 47.74it/s, Val/accuracy=0.904, Val/recall=0.951, Val/precision=0.866, Val/f1=0.906, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 97:  98%|█████████▊| 193/197 [00:04<00:00, 46.00it/s, Val/accuracy=0.904, Val/recall=0.951, Val/precision=0.866, Val/f1=0.906, Val/auc=0.952, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 97: 100%|██████████| 197/197 [00:04<00:00, 46.08it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 98:  98%|█████████▊| 193/197 [00:03<00:00, 48.32it/s, Val/accuracy=0.900, Val/recall=0.910, Val/precision=0.888, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 98: 100%|██████████| 197/197 [00:04<00:00, 48.41it/s, Val/accuracy=0.904, Val/recall=0.877, Val/precision=0.922, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 99:  98%|█████████▊| 193/197 [00:04<00:00, 47.26it/s, Val/accuracy=0.904, Val/recall=0.877, Val/precision=0.922, Val/f1=0.899, Val/auc=0.952, decision_thresh=0.500]\n",
      "Validating: 0it [00:00, ?it/s]\u001B[A\n",
      "Epoch 99: 100%|██████████| 197/197 [00:04<00:00, 47.31it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.952, decision_thresh=0.500]\n",
      "Epoch 99: 100%|██████████| 197/197 [00:04<00:00, 47.27it/s, Val/accuracy=0.900, Val/recall=0.918, Val/precision=0.882, Val/f1=0.900, Val/auc=0.952, decision_thresh=0.500]\n"
     ]
    }
   ],
   "source": [
    "checkpoint_callback = ModelCheckpoint(monitor=\"Val/auc\", mode=\"max\")\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    **vars(config.trainer),\n",
    "    logger=None,\n",
    "    deterministic=True,\n",
    "    callbacks=[checkpoint_callback]\n",
    ")\n",
    "\n",
    "trainer.fit(weasel, datamodule=data_module)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Evaluation\n",
    "\n",
    "Now that Weasel has finished training, we can evaluate on the held-out test set to see how well Weasel did,\n",
    "i.e. how well the MLP end-model from above generalizes beyond the weak training signal given by the 99 LFs.\n",
    "\n",
    "That is, evaluate how good the ``predictions = end-model(X_test)`` are with respect to our gold test labels\n",
    "*Y_test*.  <br>\n",
    "$\\rightarrow$Note that the LFs, *L*, and Weasel are not needed anymore after training/for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# See the sister notebook, 1_bias_bios.ipynb for a Snorkel baseline\n",
    "test_stats = trainer.test(weasel, datamodule=data_module, ckpt_path='best')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python WE2ES",
   "language": "python",
   "name": "e2e"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}