Practical_6_[Lab_Sheet]_Evaluating_Models_through_Curves.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Practical 6 [Lab Sheet] - Evaluating Models through Curves.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMOMbQ9n/IXPEdpQt5cma3B",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DulanDias/HandsOnMachineLearning/blob/master/Practical_6_%5BLab_Sheet%5D_Evaluating_Models_through_Curves.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ijuiybnH4QEC",
        "colab_type": "text"
      },
      "source": [
        "# **CSC 319 1.5 Machine Learning 1**\n",
        "## **Practical 6 [Lab Sheet] - Evaluating Models through Curves**\n",
        "\n",
        "In this section, we will try to understand the trained models by plotting graphs.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4mStKfQn5bEN",
        "colab_type": "text"
      },
      "source": [
        "#### **From the Previous Section**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oyZ86EWg52zD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObbnN3-06v55",
        "colab_type": "text"
      },
      "source": [
        "The F1 score favors classifiers that have similar precision and recall. \n",
        "\n",
        "This is not always what you want: in some contexts you mostly care about precision, and in other contexts you really care about recall. For example, if you trained a classifier to detect videos that are safe for kids, you would probably prefer a classifier that rejects many good videos (low recall) but keeps only safe ones (high precision), rather than a classifier that has a much higher recall but lets a few really bad videos show up in your product (in such cases, you may even want to add a human pipeline to check the classifier’s video selection). \n",
        "\n",
        "On the other hand, suppose you train a classifier to detect shoplifters on surveillance images: it is probably fine if your classifier has only 30% precision as long as it has 99% recall (sure the security guards will get a few false alerts, but almost all shoplifters will get caught).\n",
        "\n",
        "This tradeoff is called, \"precision/recall tradeoff\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gRaK_zNh7qhS",
        "colab_type": "text"
      },
      "source": [
        "### **Precision/Recall Tradeoff**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bez9IT6O72hR",
        "colab_type": "text"
      },
      "source": [
        "To understand this tradeoff, let’s look at how the SGDClassifier makes its classification decisions. For each instance, it computes a score based on a decision function, and if that score is greater than a threshold, it assigns the instance to the positive class, or else it assigns it to the negative class. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NoZvud9p8ONA",
        "colab_type": "text"
      },
      "source": [
        "Scikit-Learn does not let you set the threshold directly, but it does give you access to the decision scores that it uses to make predictions. Instead of calling the classifier’s predict() method, you can call its decision_function() method, which returns a score for each instance, and then make predictions based on those scores using any threshold you want:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qI-Xw_5g8Pe8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AAFNUXCY8xgN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4kC3tK9682HI",
        "colab_type": "text"
      },
      "source": [
        "The SGDClassifier uses a threshold equal to 0, so the previous code returns the same result as the predict() method (i.e., True). Let’s raise the threshold:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iszotpRD83DG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "epgWxm1N8816",
        "colab_type": "text"
      },
      "source": [
        "This confirms that raising the threshold decreases recall. The image actually represents a 5, and the classifier detects it when the threshold is 0, but it misses it when the threshold is increased to 8,000."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "anCcggMb9MHW",
        "colab_type": "text"
      },
      "source": [
        "Now how do you decide which threshold to use? For this you will first need to get the scores of all instances in the training set using the cross_val_predict() function again, but this time specifying that you want it to return decision scores instead of predictions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8L687Q2b9NOZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8b_GRhCQ9SKJ",
        "colab_type": "text"
      },
      "source": [
        "Now with these scores you can compute precision and recall for all possible thresholds using the precision_recall_curve() function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "knKxvtml9Tin",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kDf-iFC79Xbr",
        "colab_type": "text"
      },
      "source": [
        "Finally, you can plot precision and recall as functions of the threshold value using Matplotlib:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XuECPxU39YO6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltrD-j3q_B0S",
        "colab_type": "text"
      },
      "source": [
        "Another way to select a good precision/recall tradeoff is to plot precision directly against recall."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0uFuIQMnA-HC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0VniOVBnCCRN",
        "colab_type": "text"
      },
      "source": [
        "You can see that precision really starts to fall sharply around 80% recall. You will probably want to select a precision/recall tradeoff just before that drop—for example, at around 60% recall. But of course the choice depends on your project."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lnZ7pfn8_RGZ",
        "colab_type": "text"
      },
      "source": [
        "So let’s suppose you decide to aim for 90% precision. You look up the first plot and find that you need to use a threshold of about 8,000. To be more precise you can search for the lowest threshold that gives you at least 90% precision (np.argmax() will give us the first index of the maximum value, which in this case means the first True value):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YsUD5k79_VfB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1kZfi4q_bkl",
        "colab_type": "text"
      },
      "source": [
        "To make predictions (on the training set for now), instead of calling the classifier’s\n",
        "predict() method, you can just run this code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MkkujaH0_aYI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z94zfdPw_d-w",
        "colab_type": "text"
      },
      "source": [
        "Let’s check these predictions’ precision and recall:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Wk8pWgE_fao",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VjXSkdZX_gt0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsavRqVB_puG",
        "colab_type": "text"
      },
      "source": [
        "Great, you have a 90% precision classifier ! As you can see, it is fairly easy to create a classifier with virtually any precision you want: just set a high enough threshold, and you’re done. Hmm, not so fast. A high-precision classifier is not very useful if its recall is too low!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIlL15_z_sA8",
        "colab_type": "text"
      },
      "source": [
        "### **The ROC Curve**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nyUG_U1N_2Ia",
        "colab_type": "text"
      },
      "source": [
        "The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. It is very similar to the precision/recall curve, but instead of plotting precision versus recall, the ROC curve plots the true positive rate (another name for recall) against the false positive rate. The FPR is the ratio of negative instances that are incorrectly classified as positive. It is equal to one minus the true negative rate, which is the ratio of negative instances that are correctly classified as negative. The TNR is also called specificity. Hence the ROC curve plots sensitivity (recall) versus 1 – specificity."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouY1b0kx_-0H",
        "colab_type": "text"
      },
      "source": [
        "To plot the ROC curve, you first need to compute the TPR and FPR for various thres‐ hold values, using the roc_curve() function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4RerYdv_8Hy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZbJRaxUAA74",
        "colab_type": "text"
      },
      "source": [
        "Then you can plot the FPR against the TPR using Matplotlib:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nQrskaHbADx5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HrHfA_s9AnGD",
        "colab_type": "text"
      },
      "source": [
        "Once again there is a tradeoff: the higher the recall (TPR), the more false positives (FPR) the classifier produces. The dotted line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_d9ypHTArPE",
        "colab_type": "text"
      },
      "source": [
        "One way to compare classifiers is to measure the area under the curve (AUC). A perfect classifier will have a ROC AUC equal to 1, whereas a purely random classifier will have a ROC AUC equal to 0.5. Scikit-Learn provides a function to compute the ROC AUC:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uInJ8LMOAtga",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OI5EbubSCPdD",
        "colab_type": "text"
      },
      "source": [
        "Note: Since the ROC curve is so similar to the precision/recall (or PR) curve, you may wonder how to decide which one to use. As a rule of thumb, you should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negatives, and the ROC curve otherwise. For example, looking at the previous ROC curve (and the ROC AUC score), you may think that the classifier is really good. But this is mostly because there are few positives (5s) compared to the negatives (non-5s). In contrast, the PR curve makes it clear that the classifier has room for improvement (the curve could be closer to the top- right corner)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cRxjD4F8CWJv",
        "colab_type": "text"
      },
      "source": [
        "Let’s train a RandomForestClassifier and compare its ROC curve and ROC AUC score to the SGDClassifier. First, you need to get scores for each instance in the training set. But due to the way it works, the RandomForestClassifier class does not have a decision_function() method. Instead it has a pre dict_proba() method. Scikit-Learn classifiers generally have one or the other. The predict_proba() method returns an array containing a row per instance and a column per class, each containing the probability that the given instance belongs to the given class (e.g., 70% chance that the image represents a 5):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ASlQuJGRCbZr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AqEirOVbCg7r",
        "colab_type": "text"
      },
      "source": [
        "But to plot a ROC curve, you need scores, not probabilities. A simple solution is to\n",
        "use the positive class’s probability as the score:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0M-IqncICh42",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KJmfZ5syCm9j",
        "colab_type": "text"
      },
      "source": [
        "Now you are ready to plot the ROC curve. It is useful to plot the first ROC curve as well to see how they compare:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EkPxVEl9Co2W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RklSeAp9CsUV",
        "colab_type": "text"
      },
      "source": [
        "As you can see, the RandomForestClassifier’s ROC curve looks much better than the SGDClassifier’s: it comes much closer to the top-left corner. As a result, its ROC AUC score is also significantly better:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kflXkBJECvBB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rbsyUS0BDBMP",
        "colab_type": "text"
      },
      "source": [
        "Try measuring the precision and recall scores: you should find 99.0% precision and 86.6% recall."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "59Ls5tdrDEmR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# YOUR CODE  COMES HERE"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}