lab1_ml-recap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Lab1 - Introduction + Neural Networks + PyTorch recap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Plan for today\n",
    "\n",
    "1. Get to know course rules, timetable, etc.\n",
    "2. Briefly recap our ML knowledge:\n",
    "    * implement basic logistic regression from scratch\n",
    "    * get (re)accustomed with PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 1. Course logistics\n",
    "\n",
    "Let's go over [the course page](https://github.com/gmum/wzum-24) on GitHub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 2. Logistic regression from scratch\n",
    "\n",
    "We will tackle the problem of **classification**, i.e. prediction of a discrete value (class):\n",
    "\n",
    "$$\n",
    "f(x) = y, y \\in \\{0...N\\}\n",
    "$$\n",
    "\n",
    "The most basic variant of this is **binary** classification: $y \\in \\{0, 1\\}$. We will focus on that for the time being.\n",
    "\n",
    "**Logistic regression** is a model which predicts the probability that a given example belongs to the class 1:\n",
    "\n",
    "$$\n",
    "g(x) = \\hat{p}(y = 1 | x )\n",
    "$$\n",
    "\n",
    "**Questions for you:**\n",
    "* what is the probability that $y=0$?\n",
    "* in the multi-class case, how many outputs will the model have?\n",
    "* what conditions must the model outputs satisfy?\n",
    "\n",
    "As an example, we will work with a breast cancer prediction dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "import torch\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import matplotlib.pyplot as plt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "print(load_breast_cancer().DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_val, y_train, y_val= train_test_split(X, y, train_size=0.9)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_val = scaler.transform(X_val)\n",
    "\n",
    "\n",
    "print(\"array shapes\", [t.shape for t in [X_train, X_val, y_train, y_val]])\n",
    "print(\"y values\", np.unique(y_train), np.unique(y_val) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Linear vs logistic regression\n",
    "\n",
    "We need to transform a vector of 30 features into a value $\\in (0,1)$. Can we use linear regression for that?\n",
    "\n",
    "![classification_regression](https://raw.githubusercontent.com/aghbit/BIT_AI/master/3_logistic_regression/img/clas_reg.png)\n",
    "\n",
    "Recall that in linear regression $f(x) \\in \\mathbb{R} $ is defined as:\n",
    "$$\n",
    "f(x) = w^T x + b\n",
    "$$\n",
    "\n",
    "Where $w, b$ are trainable parameters.\n",
    "\n",
    "In logistic regression, we will need to squash the output, so that $f(x) \\in [0,1]$. A convenient way to do this is the **sigmoid** function:\n",
    "\n",
    "$$\\sigma(x) = \\frac{1}{1+e^{-x}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "for x in [\n",
    "   -np.inf,\n",
    "    0,\n",
    "    1,\n",
    "    np.inf\n",
    "]:\n",
    "    print(f\"sigmoid({x}) = {sigmoid(x)}\")\n",
    "\n",
    "x = np.linspace(-10, 10)\n",
    "\n",
    "plt.plot(x, sigmoid(x))\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To sum it up, in case of binary classification:\n",
    "$$\n",
    "\\hat{p}(y=1 | x) = \\sigma(w^Tx + b)\n",
    "$$\n",
    "\n",
    "What about the loss function which can train such a model?\n",
    "\n",
    "We'll use a logarithmic loss function which quite nicely captures an intuition, that we want the predictions datapoints which should be predicted as $0$ as close to $0$ as possible, and, analogically, predictions which should be $1$, as close to $1$ as possible:\n",
    "\n",
    "$$ L = \\frac{-1}{n}\\Big(\\sum_{i=0}^n y^{(i)}\\log{f(x^{(i)})} + (1-y^{(i)})\\log{(1-f(x^{(i)}))} \\Big)$$\n",
    "\n",
    "This function is called **Binary Cross-Entropy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x = np.linspace(1e-6, 1 - (1e-6), 1000)\n",
    "\n",
    "plt.plot(x, -np.log(1 - x), label=\"loss when y = 0\")\n",
    "plt.plot(x, -np.log(x), label=\"loss when y = 1\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Task for you: implement the binary cross-entropy function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def binary_cross_entropy(x, y, w, b) -> float:\n",
    "    \"\"\"\n",
    "    All arguments are numpy arrays with shapes:\n",
    "        x: [N, F]\n",
    "        y: [N]\n",
    "        w: [F, 1]\n",
    "        b: [1]\n",
    "\n",
    "    Returns:\n",
    "        The value of binary cross-entropy (a single number).\n",
    "    \"\"\"\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In order to train the parameters $w, b$ of our model, we need to calculate the gradients of loss with regard to those parameters:\n",
    "\n",
    "$$ \\frac{\\partial L}{\\partial w} \\text{ and } \\frac{\\partial L}{\\partial b} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Task for you: implement the function which calculates $\\frac{\\partial L}{\\partial w}$ and $\\frac{\\partial L}{\\partial b}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def calculate_gradients(x, y, w, b) -> Tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"\n",
    "    All arguments are numpy arrays with shapes:\n",
    "        x: [N, F]\n",
    "        y: [N]\n",
    "        w: [F, 1]\n",
    "        b: [1]\n",
    "\n",
    "    Returns:\n",
    "        The gradients of loss L with respect to `w` and `b`. Their shapes should be identical to the shapes of `w` and `b`, respectively.\n",
    "    \"\"\"\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "With the gradient calculation implemented, we should be able to train our model with the **Gradient Descent** method.\n",
    "\n",
    "With a good choice of learning rate, you should be able to achieve around 90% validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "w = np.random.randn(30, 1) \n",
    "b = np.random.randn(1)\n",
    "# how should we initialize w and b?\n",
    "\n",
    "lr = 1e-1\n",
    "# how big should be the learning rate?\n",
    "\n",
    "history = []\n",
    "\n",
    "for epoch in range(100):\n",
    "    \n",
    "    y_pred_train = ... # calculate predictions based on x, w, and b (don't forget sigmoid!)\n",
    "    y_pred_train = (y_pred_train > 0.5).astype(int)\n",
    "    train_accuracy = (y_pred_train.reshape(-1) == y_train).mean()\n",
    "    \n",
    "    y_pred_val = ... # calculate predictions based on x, w, and b (don't forget sigmoid!)\n",
    "    y_pred_val = (y_pred_val > 0.5).astype(int)\n",
    "    val_accuracy = (y_pred_val.reshape(-1) == y_val).mean()\n",
    "    \n",
    "    l_train = binary_cross_entropy(X_train, y_train, w=w, b=b)\n",
    "    dw, db = calculate_gradients(X_train, y_train, w=w, b=b)\n",
    "\n",
    "    w = ... # assign new value of w based on gradient dw\n",
    "    b = ... # assign new value of b based on gradient db\n",
    "    \n",
    "    elem = dict(epoch=epoch, loss=l_train, train_accuracy=train_accuracy, val_accuracy=val_accuracy)\n",
    "    history.append(elem)\n",
    "\n",
    "\n",
    "\n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"loss\"] for h in history], label=\"loss\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"train_accuracy\"] for h in history], label=\"train acc\")\n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"val_accuracy\"] for h in history], label=\"val acc\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 3. Logistic regression with PyTorch automatic differentiation / loss / optimization tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.optim import SGD\n",
    "from torch import nn\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task for you: initialize a logistic regression PyTorch model and train it\n",
    "\n",
    "You should achieve around 90% validation accuracy.\n",
    "\n",
    "**Hint: if you think you did everything right, but your model can't learn, check whether the dimensions of the tensors throughout the training are correct.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(\n",
    "    # we need a sequence of two pytorch layers for a basic logistic regression model - which ones?\n",
    ")\n",
    "opt = SGD(model.parameters(), lr=1e-1)\n",
    "loss_fn = nn.BCELoss()\n",
    "\n",
    "print({\n",
    "    name: p.shape\n",
    "    for (name, p) in model.named_parameters()\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "history = []\n",
    "\n",
    "for epoch in range(100):\n",
    "    opt.zero_grad()\n",
    "    \n",
    "    ### YOUR CODE HERE ###\n",
    "    y_pred_train = ... # make predictions for train set\n",
    "    y_pred_val = ... # make_predictions for val set\n",
    "    \n",
    "    # calculate loss\n",
    "    l_train = ...\n",
    "    # calculate gradients with respect to l_train\n",
    "    \n",
    "    # perform the optimization step with the optimizer\n",
    "    \n",
    "    #######################\n",
    "    \n",
    "    train_accuracy = ((y_pred_train.detach().numpy() > 0.5) == y_train).mean()\n",
    "    val_accuracy = ((y_pred_val.detach().numpy() > 0.5) == y_val).mean()\n",
    "\n",
    "    elem = dict(epoch=epoch, loss=l_train.item(), train_accuracy=train_accuracy, val_accuracy=val_accuracy)\n",
    "    history.append(elem)\n",
    "    \n",
    "\n",
    "\n",
    "    \n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"loss\"] for h in history], label=\"loss\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"train_accuracy\"] for h in history], label=\"train acc\")\n",
    "plt.plot([h[\"epoch\"] for h in history], [h[\"val_accuracy\"] for h in history], label=\"val acc\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 4. FashionMNIST: a bigger task\n",
    "\n",
    "Let's now train a neural net on a more challenging, multi-label FashionMNIST task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision.datasets import FashionMNIST\n",
    "from torchvision import transforms as tv\n",
    "from torch.utils.data import DataLoader\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task: load the dataset and visualize some examples along with their classes\n",
    "* what is the shape of a single example from the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = FashionMNIST('./data', train=True, target_transform=None, download=True, transform=tv.ToTensor()) # transform the data from PIL image to a tensor\n",
    "ds_test = FashionMNIST('./data', train=False, target_transform=None, download=True, transform=tv.ToTensor()) # transform the data from PIL image to a tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize 10 examples from the dataset, along with their class numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=64\n",
    "\n",
    "train_dl = DataLoader(ds, batch_size, shuffle=True)\n",
    "valid_dl = DataLoader(ds_test, batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task: implement and train a neural network with two linear layers and ReLU activation between them. \n",
    "\n",
    "You should achieve at least 80% validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FashionNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        \n",
    "        # initialize the layers of the network\n",
    "        # what is the size of the input?\n",
    "        # what is the output size?\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # process x through:\n",
    "        # 1) first layer\n",
    "        # 2) relu activation\n",
    "        # 3) second layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = FashionNN() # actually initialize the net\n",
    "loss_fn = ... # what loss do we use for multilabel classification?\n",
    "opt = torch.optim.Adam(net.parameters()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_epochs = 10\n",
    "\n",
    "for i in range(number_of_epochs):\n",
    "    train_loss = 0\n",
    "    for iteration, (X_train, y_train) in enumerate(train_dl):\n",
    "        \n",
    "        # perform optimization on the train set and calculate the total train loss\n",
    "        ...\n",
    "        \n",
    "        \n",
    "    val_loss = 0\n",
    "    y_predicted = []\n",
    "    y_true = []\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for iteration, (X_val, y_val) in enumerate(valid_dl):\n",
    "            # perform predictions on the validation set and gather them to calculate accuracy\n",
    "            \n",
    "            y_pred = net(X_val)\n",
    "            loss = loss_fn(y_pred, y_val)\n",
    "            val_loss += loss.item()\n",
    "            y_pred = y_pred.argmax(dim=1)\n",
    "            y_true.extend(y_val.numpy())\n",
    "            y_predicted.extend(y_pred.numpy())\n",
    "    \n",
    "            \n",
    "    val_acc = accuracy_score(y_true, y_predicted)\n",
    "    print(f'#Epoch: {i}, train loss: {train_loss}, val loss: {val_loss}, val_acc: {val_acc}')\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "uj",
   "language": "python",
   "name": "uj"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}