4-supervised-pt2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised Learning Part II - PyTorch\n",
    "\n",
    "The library we've been using on our examples so far, **scikit-learn**, offers many options of \"canned\" Machine Learning models via an interface where different algorithms can be manipulated as simple objects. It is a great way to get started with Machine Learning and even solve real-life problems using simple to moderately complex models, with moderately large amounts of data.\n",
    "\n",
    "However, to tackle more data-intensive tasks you need more powerful tools and **scikit-learn's** canned models won't be enough.\n",
    "\n",
    "This is where **PyTorch** comes in. It has many performance advantages over **scikit-learn**, including but not limited to:\n",
    "\n",
    "1. It supports GPU acceleration whereas sklearn only supports CPU.\n",
    "\n",
    "2. It explicitly requires you to define and control the class of functions (we'll call it Models for now on) you'll use to approximate the Target, the loss functions and the optimization solvers as separate objects (some may argue this is an inconvenience). This allows more flexibility to pick the right combination of tools for each type of problem.\n",
    "\n",
    "3. Its interface for defining Models is more flexible and expressive than sklearn's. This enables you to easily create very complex models. \n",
    "\n",
    "4. It has user-friendly frameworks for parallel and distributed model training.\n",
    "\n",
    "5. It offers out-of-the-box class templates that make it easy to deal with massive datasets.\n",
    "\n",
    "**PyTorch** can be seen as a general purpose Machine Learning, or even Scientific Computing framework. However, here we focus on the most common usage of the library: building Neural Networks.\n",
    "\n",
    "# Neural Networks Primer:\n",
    "\n",
    "If you look beyond the cool name, you will see that Neural Networks are just another \"family\" of mathematical function $f(X;\\theta)$. But this family of functions is so useful that it sometimes seems almost magical. To add to their allure, Neural Networks can be difficult to write down mathematically depending on how complex their \"architeture\" is. It is easier to visualize how they work by drawing them as a graph:\n",
    "\n",
    "![](./images/nnet.png)\n",
    "\n",
    "Where each neuron represents a linear combination of its inputs with a non-linear function applied to it:\n",
    "\n",
    "![](./images/neuron.png)\n",
    "\n",
    "The **weights** $W$ of each neuron in each layer are the parameters $\\theta$ of this class of function.\n",
    "\n",
    "Common choices of non-linear **activation** functions for neurons are the *Sigmoid (or Logistic)* function:\n",
    "\n",
    "$f(x) = \\frac{1}{1+\\exp{-x}}$\n",
    "\n",
    "And the Rectified Linear Unit (a.k.a. ReLU):\n",
    "\n",
    "$f(x) = \\max{(0,x)}$\n",
    "\n",
    "The type of neural network represented above is known as a **Multi Layer Perceptron** and it is the most \"basic\" member of this family of functions. More complex Neural Networks include ones where neurons in one layer do not necessarily connect to all neurons in the next layer; where other mathematical operations, like *convolutions* take place inside a neuron; and where ouputs in one layer can become inputs of preceding layers.\n",
    "\n",
    "As we've seen, Machine Learning algorithms generally involve solving the optimization problem\n",
    "\n",
    "$\\hat{F(X)} = \\underset{\\theta}{\\operatorname{argmin}} L(Y,f(X;\\theta))$\n",
    "\n",
    "This generally requires finding critical points of a function, that is, points where the derivative of the function with respect to its parameters $\\theta$ is zero.\n",
    "\n",
    "Whether you attempt to do it analytically or numerically... How do you compute the derivatives of $L$ with respect to $\\theta$ when there's a crazy Neural Network nested in it?\n",
    "\n",
    "# Backpropagation\n",
    "\n",
    "For reasons we will not delve into in this workshop, an efficient way of solving the optimization problem above is by combining two numerical algorithms: **Backpropagation** and variants of **Gradient Descent**.\n",
    "\n",
    "Here is a summary of the method with vanilla Gradient Descent:\n",
    "\n",
    "1. Randomly initialize all Weights $W$ of the neural network.\n",
    "\n",
    "2. Forward Propagation: Run your inputs X through the neural neutwork and get an output $\\hat{Y}$.\n",
    "\n",
    "3. Take the output of the Neural Network and compute the Loss $L(\\hat{Y}, Y)$.\n",
    "\n",
    "4. Backpropagation: Run the computed loss value backwards through all layers of the network, but this time each layer will represent the derivatives of the loss function with respect to the **Weights** in that layer. At the end of this process, you will obtain an estimate of $\\nabla_W L$, the gradient of $L$ with respect to all **Weights** of the neural network.\n",
    "    \n",
    "5. Gradient Descent - Update the **Weights** $W$ using the gradient computed in step 4: $W = W - \\alpha\\nabla_W L$, where $\\alpha$ is a (usually small) constant called the **learning rate**. \n",
    "\n",
    "6. Repeat steps 1 through 5 until a stoppage criterion is reached. Common criteria include:\n",
    "\n",
    "    a. The Loss reaches zero, or a value smaller than a pre-defined threshold.\n",
    "    \n",
    "    b. Steps 1 through 5 have been repeated a (usually large) pre-defined number of times.\n",
    "    \n",
    "Variations of this method include, but are not limited to:\n",
    "\n",
    "* Running all examples X at once through the network, resulting in large matrix multiplications being performed; \n",
    "\n",
    "* Running smaller, *randomly selected batches* of examples X through the network instead; \n",
    "\n",
    "* Using an adaptive learning rate;\n",
    "\n",
    "* Using other types of weight update (also called \"a step\"). For example: Adam, SGD with momentum, RMSProp;\n",
    "\n",
    "* Randomly selecting neurons to be *dropped out* from the computations at each iteration - A practice known as... \"Drop Out\";\n",
    "\n",
    "# Loss Function\n",
    "\n",
    "You know the Loss function is a measure of the error we incur in when using a function $f(x;\\theta)$ to approximate a target $F^*$, but we haven't seen what it looks like. The choice of an appropriate Loss function will depend first on the type of task at hand, then on statistical properties of the data and, to a smaller degree, on the choice of algorithm. Below are some common choices of Loss function.\n",
    "\n",
    "### Task: Regression\n",
    "\n",
    "**Mean Squared Error (MSE) Loss:**\n",
    "\n",
    "$L(Y,\\hat{Y}) = \\frac{1}{N}\\sum{(Y-\\hat{Y})^2}$\n",
    "\n",
    "**Mean Absolute Error (MAE) Loss:**\n",
    "\n",
    "$L(Y,\\hat{Y}) =\\frac{1}{N}\\sum{|Y-\\hat{Y}|}$\n",
    "\n",
    "### Task: Classification\n",
    "\n",
    "**Cross Entropy Loss:**\n",
    "\n",
    "$L(Y,\\hat{Y}) = -\\sum{Y * log(\\hat{Y})}$\n",
    "\n",
    "This Loss is commonly used in Classification problems where there are more than two classes of outputs.\n",
    "\n",
    "**The Binary Cross Entropy Loss:**\n",
    "\n",
    "$L(Y,\\hat{Y}) = -(Y*log(\\hat{Y}) + (1-Y)*log(1-\\hat{Y}))$\n",
    "\n",
    "You may recognize this as the general Cross Entropy above with only two output classes, or the negative log-likelihood of a Bernoulli Distribution.\n",
    "\n",
    "____\n",
    "\n",
    "## Neural Networks With PyTorch Example 1 - Iris Dataset Revisited\n",
    "    \n",
    "To see all this in action, let's go back to the Iris Dataset and train a logistic regression model with **PyTorch**, this time representing it as a neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas import read_csv\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn.metrics import accuracy_score\n",
    "import numpy as np\n",
    "\n",
    "# LET'S CREATE OUR TRAINING AND TEST SETS\n",
    "\n",
    "header = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'species']\n",
    "\n",
    "iris_dataset = read_csv('./data/iris.csv',names = header) \n",
    "\n",
    "# ENCODE SPECIES AS CATEGORY NUMBERS \n",
    "iris_dataset.loc[iris_dataset.species=='Iris-setosa', 'species'] = 0\n",
    "iris_dataset.loc[iris_dataset.species=='Iris-versicolor', 'species'] = 1\n",
    "iris_dataset.loc[iris_dataset.species=='Iris-virginica', 'species'] = 2\n",
    "\n",
    "X = iris_dataset.values[:,0:4].astype('float32')\n",
    "Y = iris_dataset.values[:,4].astype('int32')\n",
    "\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "# CONVERT DATASETS TO PYTORCH TENSORS\n",
    "X_train = torch.Tensor(X_train).float()\n",
    "X_test = torch.Tensor(X_test).float()\n",
    "Y_train= torch.Tensor(Y_train).long()\n",
    "Y_test = torch.Tensor(Y_test).long()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DEFINE OUR LOGISTIC REGRESSION MODEL AS A NEURAL NETWORK, INITIALIZE AN OPTIMIZER AND PICK A LOSS FUNCTION\n",
    "\n",
    "\n",
    "# THERE IS A WAY TO CALL MODELS AS FUNCTIONS LIKE WE DID WITH SKLEARN, BUT CLASSES ARE PREFERABLE\n",
    "class LogisticRegression(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(LogisticRegression, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(4, 3)\n",
    "    def forward(self, X):\n",
    "        X = self.fc1(X)\n",
    "\n",
    "        return X  \n",
    "\n",
    "model = LogisticRegression()\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  \n",
    "\n",
    "print(optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_function = torch.nn.CrossEntropyLoss()  \n",
    "\n",
    "print(loss_function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOW WE TRAIN THE MODEL\n",
    "\n",
    "epochs = 1000\n",
    "\n",
    "for i in range(epochs):\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # FORWARD-PROPAGATION     \n",
    "    Y_hat = model(X_train)\n",
    "\n",
    "    loss = loss_function(Y_hat, Y_train)\n",
    "    print(\"Loss at step\", i, \"is:\" , loss.data.item())\n",
    "    \n",
    "    # BACKPROPAGATION\n",
    "    loss.backward()\n",
    "\n",
    "    # UPDATE WEIGHTS\n",
    "    optimizer.step()\n",
    "    \n",
    "# NOTICE HOW THE ERROR DECREASES WITH EACH STEP\n",
    "\n",
    "print('Finished Training!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# HOW DID IT DO ON THE TEST SET?\n",
    "\n",
    "Y_hat_test = model(X_test)\n",
    "Y_predicted = torch.max(Y_hat_test, 1).indices\n",
    "\n",
    "print(\"\\n This model got\", accuracy_score(Y_test, Y_predicted)*100, \"% right\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.max(Y_hat_test, 1).indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOW WHAT HAPPENS IF WE USE A MORE COMPLEX NEURAL NET INSTEAD OF LOGISTIC REGRESSION?\n",
    "\n",
    "class NeuralNet(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(NeuralNet, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(4, 100)\n",
    "        self.fc2 = torch.nn.Linear(100, 100) # NOTICE THE SIZE OF THE OUTPUT MATCHES THE SIZE OF THE INPUT OF THE NEXT LAYER\n",
    "        self.fc3 = torch.nn.Linear(100, 3)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        X = torch.nn.functional.sigmoid(self.fc1(X)) # NOTICE THE SIGMOID ACTIVATION APPLIED TO EACH LAYER\n",
    "        X = torch.nn.functional.sigmoid(self.fc2(X))\n",
    "        X = self.fc3(X)\n",
    "        \n",
    "        return X\n",
    "\n",
    "nnet_model = NeuralNet()\n",
    "\n",
    "print(nnet_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADDING A MOMENTUM TERM TO THE WEIGHT UPDATES \n",
    "nnet_optimizer = torch.optim.SGD(nnet_model.parameters(), lr=learning_rate, momentum = 0.9)\n",
    "\n",
    "print(nnet_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nnet_loss_function = torch.nn.CrossEntropyLoss()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(epochs):\n",
    "    \n",
    "    nnet_optimizer.zero_grad()\n",
    "    \n",
    "    # FORWARD-PROPAGATION     \n",
    "    Y_hat = nnet_model(X_train)\n",
    "\n",
    "    loss = nnet_loss_function(Y_hat, Y_train)\n",
    "    \n",
    "    # BACKPROPAGATION\n",
    "    loss.backward()\n",
    "    print(\"Loss at step\", i, \"is:\" , loss.data.item())\n",
    "    # UPDATE WEIGHTS\n",
    "    nnet_optimizer.step()\n",
    "    \n",
    "print('Finished Training!')\n",
    "    \n",
    "Y_hat_test = nnet_model(X_test)\n",
    "Y_predicted = torch.max(Y_hat_test, 1).indices\n",
    "\n",
    "print(\"\\n This model got\", accuracy_score(Y_test, Y_predicted)*100, \"% right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3 - Wine Classification with PyTorch\n",
    "\n",
    "Now you try. Let's go back to the Wine dataset and train a model with PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = ['wine_type','alcohol', 'malic_acid','ash','alcalinity_of_ash','magnesium',\n",
    "           'total_phenols','flavanoids','nonflavanoid_phenols','proanthocyanins','color_intensity','hue','OD280_OD315','proline']\n",
    "\n",
    "wine_dataset = read_csv(####)\n",
    "\n",
    "# ENCODE SPECIES AS CATEGORY NUMBERS \n",
    "wine_dataset.loc[wine_dataset.wine_type=='wine_1', 'wine_type'] = ###\n",
    "wine_dataset.loc[wine_dataset.wine_type=='wine_2', 'wine_type'] = ###\n",
    "wine_dataset.loc[wine_dataset.wine_type=='wine_3', 'wine_type'] = ###\n",
    "\n",
    "X = wine_dataset.values[###].astype('float32')\n",
    "Y = wine_dataset.values[###].astype('int32')\n",
    "\n",
    "X_train, X_test, Y_train, Y_test = ####\n",
    "\n",
    "# CONVERT DATASETS TO PYTORCH TENSORS\n",
    "X_train = torch.Tensor(X_train).float()\n",
    "X_test = torch.Tensor(X_test).float()\n",
    "Y_train= torch.Tensor(Y_train).long()\n",
    "Y_test = torch.Tensor(Y_test).long()\n",
    "    \n",
    "class Exercise3(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Exercise3, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(####)\n",
    "    def forward(self, X):\n",
    "        X = self.fc1(X)\n",
    "\n",
    "        return X  \n",
    "\n",
    "model = ####\n",
    "            \n",
    "learning_rate = ###\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  \n",
    "loss_function = ####\n",
    "            \n",
    "epochs = ####\n",
    "            \n",
    "for i in range(epochs):\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # FORWARD-PROPAGATION     \n",
    "    Y_hat = ####\n",
    "\n",
    "    loss = ###\n",
    "    \n",
    "    # BACKPROPAGATION\n",
    "    ###\n",
    "    print(\"Loss at step\", i, \"is:\" , loss.data.item())\n",
    "    # UPDATE WEIGHTS\n",
    "    ####\n",
    "    \n",
    "print('Finished Training!')\n",
    "    \n",
    "Y_hat_test = ####\n",
    "Y_predicted = torch.max(Y_hat_test, 1).indices\n",
    "\n",
    "print(\"\\n This model got\", accuracy_score(Y_test, Y_predicted)*100, \"% right\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "# High Performance Machine Learning\n",
    "\n",
    "So far we have trained models using **scikit-learn** and **PyTorch** on small datasets, where any modern Laptop can carry out the work in a matter of seconds. But what do you do when your problem involves massive amounts of data and training complex models many times over until you get results you are happy with?\n",
    "\n",
    "If you recall when we introduced **PyTorch** on the previous notebook, we listed GPU acceleration, tools to handle massive datasets and user-friendly parallel/distributed training as advantages over **scikit-learn**.\n",
    "\n",
    "We will now explore these advantages.\n",
    "\n",
    "## GPU Acceleration\n",
    "\n",
    "When people first learn about how computers work, the CPU is often referred to as the \"brain of the computer\". And in many ways it is. The CPU has to carry out many different types of instructions to enable everything from responding to keyboard/mouse events to playing YouTube videos while you type into a word processor at the same time. In other words, CPUs are built to be good at doing different types of things at the same time.\n",
    "\n",
    "The Graphical Processing Unit (a.k.a the GPU) on the other hand, only has to be good at one thing: rendering graphics. \n",
    "\n",
    "As it turns out, being good at rendering graphics really means being good at performing Vector/Matrix/Tensor operations - the same type of operations that happen under the hood in Neural Networks. It follows that GPUs are better than CPUs at training Neural Networks.\n",
    "\n",
    "Modern Machine Learning libraries, like **PyTorch** allow you to take advantage of this fact and use GPUs to train models faster.\n",
    "\n",
    "Let's see this in action:\n",
    "\n",
    "## Example - Training a Neural Network with GPU:\n",
    "\n",
    "In this example, we will take the Iris Dataset and train a model using the GPU instead of the CPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CREATE OUR IRIS TRAINING AND TEST SETS AGAIN\n",
    "\n",
    "X = iris_dataset.values[:,0:4].astype('float32')\n",
    "Y = iris_dataset.values[:,4].astype('int32')\n",
    "\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20)\n",
    "\n",
    "# CONVERT DATASETS TO PYTORCH TENSORS\n",
    "X_train = torch.Tensor(X_train).float()\n",
    "X_test = torch.Tensor(X_test).float()\n",
    "Y_train= torch.Tensor(Y_train).long()\n",
    "Y_test = torch.Tensor(Y_test).long()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# THIS LINE ENABLES GPU ONLY IF A GPU AND CUDA ARE AVAILABLE\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "print(\"The device Object is tied to:\", device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#INITIALIZE OUR NEURAL NET\n",
    "\n",
    "nnet_model = NeuralNet()\n",
    "\n",
    "# LOAD OUR MODEL ON THE GPU\n",
    "nnet_model.to(device)\n",
    "\n",
    "# LOAD TRAINING DATA ON THE GPU\n",
    "\n",
    "X_train.to(device)\n",
    "Y_train.to(device)\n",
    "\n",
    "for i in range(epochs):\n",
    "    \n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # FORWARD-PROPAGATION     \n",
    "    Y_hat = nnet_model(X_train)\n",
    "\n",
    "    loss = loss_function(Y_hat, Y_train)\n",
    "    #print(\"Loss at step\", i, \"is:\" , loss.data.item())\n",
    "    \n",
    "    # BACKPROPAGATION\n",
    "    loss.backward()\n",
    "\n",
    "    # UPDATE WEIGHTS\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling Massive Amounts of Data - Loaders, Datasets and Batch Training\n",
    "\n",
    "In the example above, our dataset was still very small - we can be pretty sure it will fit entirely inside any modern Graphics Card's memory when we load it on the GPU. But what if we had not a small and nicely structured csv file, but a large-ish dataset of, say... 200GB worth of video and image files? Unless you have the ultimate gaming station, you will probably have a hard time fitting all that in memory whether it's on your Graphics Card or not.\n",
    "\n",
    "With that in mind, **PyTorch** offers a convenient set of tools called *Datasets* and *Loaders* that, when used in conjunction with a Machine Learning technique called *Mini-Batch Training*, offer a good balance between performance, stability and scalability.\n",
    "\n",
    "Let's see how this works using the MNIST Handwritten Digit Dataset from the previous notebook (which is still a very small dataset).\n",
    "\n",
    "### Example - Using Torchvision and Built-in Dataset and Loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torchvision import datasets, transforms #TORCHVISION IS A SUBPACKAGE THAT CONTAINS MANY INTERESTING UTILITIES FOR HANDLING IMAGES\n",
    "from torchvision.utils import make_grid\n",
    "\n",
    "# USE TRANSFORMS TO RE-SCALE, NORMALIZE... TRANSFORM! HERE WE JUST GO FROM NUMPY ARRAY TO PYTORCH TENSOR\n",
    "transform = transforms.Compose([transforms.ToTensor()])\n",
    "\n",
    "mnist_trainingset = datasets.ImageFolder(root='./MNIST-Dataset/Training', transform=transform)\n",
    "\n",
    "# NOTICE BATCH_SIZE AND NUM_WORKERS PARAMETERS - LOAD SMALL A AMOUNT AT A TIME, IN PARALLEL!\n",
    "trainingset_loader = torch.utils.data.DataLoader(mnist_trainingset, batch_size=10,shuffle=True, num_workers=1)\n",
    "\n",
    "# LET'S LOAD A SINGLE BATCH AND DISPLAY IT\n",
    "def imshow(img):\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()\n",
    "    \n",
    "dataiter = iter(trainingset_loader)\n",
    "\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "imshow(make_grid(images))\n",
    "\n",
    "print(\"Labels:\",labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LET'S NOT FORGET TO LOAD THE TEST SET\n",
    "\n",
    "mnist_testset = datasets.ImageFolder(root='./MNIST-Dataset/Test', transform=transform)\n",
    "testset_loader = torch.utils.data.DataLoader(mnist_testset, batch_size=10,shuffle=True, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#WHAT SHOULD THE MODEL INPUT SIZE BE?\n",
    "images[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOW LET'S TRAIN A MODEL\n",
    "\n",
    "class ImageLogisticRegression(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(ImageLogisticRegression, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(3*28*28, 3)\n",
    "    def forward(self, X):\n",
    "        #FLATTEN THE TENSOR INTO A (3*28*28,1) VECTOR\n",
    "        X = X.reshape(-1,3*28*28)\n",
    "        X = self.fc1(X)\n",
    "\n",
    "        return X  \n",
    "\n",
    "model = ImageLogisticRegression()\n",
    "\n",
    "#model.to(device) # LOAD MODEL ON GPU\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  \n",
    "loss_function = torch.nn.CrossEntropyLoss()  \n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(2): \n",
    "\n",
    "    for i,data in enumerate(trainingset_loader):\n",
    "        X_batch, Y_batch = data\n",
    "        \n",
    "        X_batch.to(device) # LOAD DATA ON GPU ONE BATCH AT A TIME\n",
    "        Y_batch.to(device)\n",
    "    \n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        Y_hat = model(X_batch)\n",
    "        loss = loss_function(Y_hat, Y_batch)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "print(\"Final Loss:\",loss.data.item())\n",
    "print('Finished Training!')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we used built-in utilities and trained a model on image files. But what if you need to load a massive csv or other types of data for which there is no built-in option available? In those cases you can build your own Custom Datasets and Loaders.\n",
    "\n",
    "### Example - Loading the Iris Dataset CSV file using Custom Dataset and Loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "\n",
    "# CUSTOM DATASETS INHERIT FROM PYTORCH CLASS 'Dataset'\n",
    "\n",
    "class CSVDataset(Dataset):\n",
    "    \n",
    "    def __init__(self, path):\n",
    "        \n",
    "        self.iris_dataset = pd.read_csv(path, iterator=True, names=['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'species'])    \n",
    "\n",
    "    # THIS METHOD DRIVES WHAT HAPPENS WHEN A LOADER IS USED TO LOAD A BATCH\n",
    "    def __getitem__(self, index):\n",
    "        \n",
    "        row = self.iris_dataset.get_chunk(1)\n",
    "        \n",
    "        row.loc[row.species=='Iris-setosa', 'species'] = 0\n",
    "        row.loc[row.species=='Iris-versicolor', 'species'] = 1\n",
    "        row.loc[row.species=='Iris-virginica', 'species'] = 2\n",
    "        \n",
    "        X = row.values[:,0:4].astype('float32')\n",
    "        Y = row.values[:,4].astype('int32')\n",
    "       \n",
    "        X = torch.Tensor(X).float()\n",
    "        Y = torch.Tensor(Y).long()\n",
    "    \n",
    "        return X,Y\n",
    "\n",
    "    def __len__(self):\n",
    "        return 150 # Here we hardcoded the length of the iris dataset where we could have just loaded it and used len(). But imagine this is an actual huge csv (hundreds of GBs). \n",
    "    \n",
    "iris_dataset = CSVDataset('./data/iris.csv')\n",
    "iris_loader = torch.utils.data.DataLoader(iris_dataset, batch_size=10,shuffle=False, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression()\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  \n",
    "loss_function = torch.nn.CrossEntropyLoss()  \n",
    "\n",
    "for epoch in range(3): \n",
    "    for i, data in enumerate(iris_loader):\n",
    "        X_batch, Y_batch = data\n",
    "        \n",
    "        X_batch = X_batch.reshape(-1,4) #CHOPPING ONE DIMENSION OFF OF LOADER OUTPUT\n",
    "        Y_batch = Y_batch.reshape(-1)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        Y_hat = model(X_batch)\n",
    "        loss = loss_function(Y_hat, Y_batch)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "print(\"Final Loss:\",loss.data.item())        \n",
    "print('Finished Training!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4 - Datasets, Loaders, Mini-Batching and GPU\n",
    "\n",
    "Now you try. Load either the Fruit Images dataset or the Wine csv and train a model using mini-batching and GPU. If you're looking for a challenge, try implementing a Custom Dataset for the Fruit Images dataset instead of using the built-in utility from torchvision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on Multiple GPUs\n",
    "\n",
    "The last advantage of **PyTorch** over **sckit-learn** on our list of performance enhancements was user-friendly parallel/distributed training. You have seen how to enable training a entire model on a single GPU so far, but what if you have multiple GPUs at your disposal as it is the case on **Compute Canada**'s clusters? Can this accelerate training even more? \n",
    "\n",
    "The answer is yes. \n",
    "\n",
    "**But keep in mind Compute Canada Clusters are shared environments. Don't request GPUs for just any job - make sure your code is actually able to use a GPU effectivelly before considering using multiple of them.**\n",
    "\n",
    "As you will see in the next example, doing this in PyTorch is as simple as adding a couple extra lines of code to the single GPU case. Concretely, what this will do is what we call \"Data Parallelism\". Very literally, PyTorch will split your inputs into a number of parts equal to however many GPUs are available and run your training loop on them in parallel. This works very well in simple cases like the kinds of neural networks we've been training so far, but extra care need to be taken if your models are more complex than just a series of torch.nn modules.\n",
    "\n",
    "There is another type of Parallelism that is possible on PyTorch, but that we will not cover in this introductory workshop: multi-node distributed training. In this type of parallelism, you may have not only multiple GPUs, but multiple computers, each maybe with its own set of multiple GPUs.\n",
    "\n",
    "## Example - Training on Multiple GPUs\n",
    "\n",
    "In this example, we'll revisit the MNIST handwritten digit dataset and train a model using multiple GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "#LOAD THE SAME MODEL ARCHITECTURE FROM THE SINGLE GPU EXAMPLE\n",
    "model = ImageLogisticRegression()\n",
    "\n",
    "# USE nn.DataParallel TO WRAP THE MODEL\n",
    "if torch.cuda.device_count() > 1:\n",
    "  print(\"Let's use\", torch.cuda.device_count(), \"GPUs!\")\n",
    "\n",
    "  model = nn.DataParallel(model)\n",
    "\n",
    "# THIS TIME SEND MODEL TO MULTIPLE GPUs    \n",
    "model.to(device)\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  \n",
    "loss_function = torch.nn.CrossEntropyLoss()  \n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(2): \n",
    "\n",
    "    for i, data in enumerate(trainingset_loader):\n",
    "        X_batch, Y_batch = data\n",
    "        \n",
    "        X_batch.to(device) # LOAD DATA ON GPU ONE BATCH AT A TIME\n",
    "        Y_batch.to(device)\n",
    "    \n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        Y_hat = model(X_batch)\n",
    "        loss = loss_function(Y_hat, Y_batch)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "print(\"Final Loss:\",loss.data.item())\n",
    "print('Finished Training!')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}