hw3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rockett-m/CS0135/blob/main/hw3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "gHx0mx8jNFY8"
      },
      "source": [
        "## HW03 Notebook\n",
        "\n",
        "Complete the following notebook, as described in the PDF for Homework 03 (included in the download with the starter code). Submit the following:\n",
        "1. This notebook file and `hw3.py`, along with your COLLABORATORS.txt file, to the Gradescope link for code.\n",
        "2. A PDF of this notebook and all of its output, once it is completed, to the Gradescope link for the PDF.\n",
        "\n",
        "__NOTE__: The purpose of this notebook is to demonstrate the functionality implemented in `hw3.py`. As part of this demo, all analysis (i.e., questions that prompt for a short answer) are to be added to the notebook. Keep the order of the problems as listed in the assignment description. Furthermore, cells are provided as placeholders for each response; however, cells can be added as needed.\n",
        "\n",
        "Please report any questions to the [class Piazza page](https://piazza.com/class/lcwv1h9p2a11ai).\n",
        "\n",
        "### Import required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "a46qYD-YNFY-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import warnings\n",
        "\n",
        "import sklearn.linear_model\n",
        "import sklearn.metrics\n",
        "from hw3 import calc_confusion_matrix_for_threshold\n",
        "from hw3 import calc_percent_cancer\n",
        "from hw3 import calc_binary_metrics\n",
        "from hw3 import predict_0_always_classifier\n",
        "from hw3 import calc_accuracy\n",
        "from hw3 import print_perf_metrics_for_threshold\n",
        "from hw3 import calc_perf_metrics_for_threshold\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "plt.style.use('seaborn-v0_8') # pretty matplotlib plots"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "TmB42g6wNFY_"
      },
      "source": [
        "#### 1) Function to calculate TP, TN, FP, and FN.\n",
        "The following four calls to the function `calc_binary_metrics` to test it. This way, the function can be tested for several edge cases. ***Don't modify this.***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "zNbw54jFNFZA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e7481f0c-0151-4f23-becd-52bd9ef24d25"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 vs 1\n",
            "========\n",
            "TP: 0.0\n",
            "TN: 0.0\n",
            "FP: 0.0\n",
            "FN: 0.0\n"
          ]
        }
      ],
      "source": [
        "all0 = np.zeros(10)\n",
        "all1 = np.ones(10)\n",
        "TP, TN, FP, FN = calc_binary_metrics(all0, all1)\n",
        "print(f\"0 vs 1\\n========\\nTP: {TP}\\nTN: {TN}\\nFP: {FP}\\nFN: {FN}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "0qK0zsSiNFZA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c3c0590-587c-4c5a-e72c-b8c5e43b0ada"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1 vs 0\n",
            "========\n",
            "TP: 0.0\n",
            "TN: 0.0\n",
            "FP: 0.0\n",
            "FN: 0.0\n"
          ]
        }
      ],
      "source": [
        "TP, TN, FP, FN = calc_binary_metrics(all1, all0)\n",
        "print(f\"1 vs 0\\n========\\nTP: {TP}\\nTN: {TN}\\nFP: {FP}\\nFN: {FN}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "cN8eop-SNFZA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "886b908f-1a7c-4f97-b58a-c483bf024d79"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1 vs 1\n",
            "========\n",
            "TP: 0.0\n",
            "TN: 0.0\n",
            "FP: 0.0\n",
            "FN: 0.0\n"
          ]
        }
      ],
      "source": [
        "TP, TN, FP, FN = calc_binary_metrics(all1, all1)\n",
        "print(f\"1 vs 1\\n========\\nTP: {TP}\\nTN: {TN}\\nFP: {FP}\\nFN: {FN}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "wBemDPZuNFZB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0fd8a99-249f-495c-e18f-299271461db4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 vs 0\n",
            "========\n",
            "TP: 0.0\n",
            "TN: 0.0\n",
            "FP: 0.0\n",
            "FN: 0.0\n"
          ]
        }
      ],
      "source": [
        "TP, TN, FP, FN = calc_binary_metrics(all0, all0)\n",
        "print(f\"0 vs 0\\n========\\nTP: {TP}\\nTN: {TN}\\nFP: {FP}\\nFN: {FN}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "LwqdNVVWNFZB"
      },
      "source": [
        "### Load the dataset.\n",
        "\n",
        "The following should ***not*** be modified.\n",
        "\n",
        "After it runs, the various arrays it creates will contain the 2- or 3-feature input datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "rsC-Hv_4NFZB"
      },
      "outputs": [],
      "source": [
        "# Load the x-data and y-class arrays\n",
        "x_train = np.loadtxt('./data/x_train.csv', delimiter=',', skiprows=1)\n",
        "x_test = np.loadtxt('./data/x_test.csv', delimiter=',', skiprows=1)\n",
        "\n",
        "y_train = np.loadtxt('./data/y_train.csv', delimiter=',', skiprows=1)\n",
        "y_test = np.loadtxt('./data/y_test.csv', delimiter=',', skiprows=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "woH5961qNFZC"
      },
      "source": [
        "Inspect Data. The following should ***not*** be modified."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "foc0l_dTNFZC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 555
        },
        "outputId": "8ae9d328-ae18-412f-a898-46cec5352393"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "features: ['age' 'famhistory' 'marker']\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "          age  famhistory    marker  ['age' 'famhistory' 'marker']\n",
              "34   59.54047         0.0  0.112316                            0.0\n",
              "6    49.82400         1.0  0.421256                            0.0\n",
              "154  65.63218         0.0  0.096532                            0.0\n",
              "12   58.93779         0.0  4.945581                            0.0\n",
              "40   68.76731         0.0  1.262915                            1.0\n",
              "151  66.25471         0.0  1.486396                            0.0\n",
              "24   64.54343         0.0  0.089446                            0.0\n",
              "73   69.45928         0.0  1.115703                            0.0\n",
              "36   61.75267         0.0  0.608578                            0.0\n",
              "0    60.91298         0.0  0.869935                            0.0\n",
              "123  59.45962         0.0  0.070890                            0.0\n",
              "11   69.89857         0.0  0.363460                            0.0\n",
              "166  63.50109         0.0  0.294197                            0.0\n",
              "56   57.48566         0.0  1.362557                            0.0\n",
              "142  64.82798         0.0  0.183669                            0.0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-584637dd-dcae-4553-989d-3f3cfd98855d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>famhistory</th>\n",
              "      <th>marker</th>\n",
              "      <th>['age' 'famhistory' 'marker']</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>59.54047</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.112316</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>49.82400</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.421256</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>154</th>\n",
              "      <td>65.63218</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.096532</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>58.93779</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.945581</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>68.76731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.262915</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>151</th>\n",
              "      <td>66.25471</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.486396</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>64.54343</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.089446</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>69.45928</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.115703</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>61.75267</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.608578</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>60.91298</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.869935</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>123</th>\n",
              "      <td>59.45962</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.070890</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>69.89857</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.363460</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>166</th>\n",
              "      <td>63.50109</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.294197</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56</th>\n",
              "      <td>57.48566</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.362557</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142</th>\n",
              "      <td>64.82798</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.183669</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-584637dd-dcae-4553-989d-3f3cfd98855d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-584637dd-dcae-4553-989d-3f3cfd98855d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-584637dd-dcae-4553-989d-3f3cfd98855d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "feat_names = np.loadtxt(f'data/x_train.csv', delimiter=',', dtype=str, max_rows=1)\n",
        "print(f\"features: {feat_names}\\n\")\n",
        "target_name = np.loadtxt(f'data/x_test.csv', delimiter=',', dtype=str, max_rows=1)\n",
        "df_sampled_data = pd.DataFrame(x_test, columns=feat_names)\n",
        "df_sampled_data[str(target_name)] = y_test\n",
        "df_sampled_data.sample(15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "egLsemU-NFZC"
      },
      "source": [
        "### 2) Compute the fraction of patients with cancer.\n",
        "\n",
        "Complete the following code.  Your solution needs to ***compute*** these values from the training and testing sets (i.e., don't simply hand-count and print the values)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "nuLa84WrNFZD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222
        },
        "outputId": "5c35cce4-c920-4409-d127-73858571acda"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-10-d473fd9b20f1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mte_percent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcalc_percent_cancer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Percent of data that has_cancer on TRAIN: %.3f\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mtr_percent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Percent of data that has_cancer on TEST : %.3f\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mte_percent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: must be real number, not NoneType"
          ]
        }
      ],
      "source": [
        "#TODO: modify these prints\n",
        "tr_percent = calc_percent_cancer(y_train)\n",
        "te_percent = calc_percent_cancer(y_test)\n",
        "\n",
        "print(\"Percent of data that has_cancer on TRAIN: %.3f\" % tr_percent)\n",
        "print(\"Percent of data that has_cancer on TEST : %.3f\" % te_percent)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "pwhZTMeDNFZD"
      },
      "source": [
        "### 3) The predict-0-always baseline\n",
        "\n",
        "#### (i) Compute the accuracy of the always-0 classifier.\n",
        "\n",
        "Complete the functions to compute and calculate the accuracy of the always-0 classifier on validation and test outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ObJ47mUANFZD"
      },
      "outputs": [],
      "source": [
        "#TODO: implement predict_0_always_classifer()\n",
        "y_train_pred = predict_0_always_classifier(x_train)\n",
        "y_test_pred = predict_0_always_classifier(x_test)\n",
        "\n",
        "acc_train = calc_accuracy(*calc_binary_metrics(y_train, y_train_pred))\n",
        "acc_test = calc_accuracy(*calc_binary_metrics(y_test, y_test_pred))\n",
        "print(\"acc on TRAIN: %.3f\" % acc_train) #TODO: modify these values\n",
        "print(\"acc on TEST : %.3f\" % acc_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "UlO6S30iNFZE"
      },
      "source": [
        "#### (ii) Print a confusion matrix for the always-0 classifier.\n",
        "\n",
        "Add code below to generate a confusion matrix for the always-0 classifier on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7_cg-vDGNFZE"
      },
      "outputs": [],
      "source": [
        "# TODO: call print(calc_confusion_matrix_for_threshold(...))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "-WHP_S2nNFZE"
      },
      "source": [
        "#### (iii) Reflect on the accuracy of the always-0 classifier.\n",
        "\n",
        "**Answer**: TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "NCbctb5JNFZE"
      },
      "source": [
        "#### (iv) Analyze the various costs of using the always-0 classifier.\n",
        "\n",
        "**Answer**: TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "Jndg8gLDNFZE"
      },
      "source": [
        "### 4: Basic Perceptron Models\n",
        "\n",
        "#### (i) Normalize data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7yuQW_gDNFZE"
      },
      "outputs": [],
      "source": [
        "#TODO\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "vdN7-mA1NFZE"
      },
      "source": [
        "#### (ii) Create a basic `Perceptron` classifier\n",
        "\n",
        "Fit a perceptron to the training data.  Print out accuracy on this data, as well as on testing data.  Print out a confusion matrix on the testing data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fXq7qbw-NFZE"
      },
      "outputs": [],
      "source": [
        "#TODO: train a basic perceptron model using default parameter values, and modify these accuracies below\n",
        "\n",
        "print(\"acc on TRAIN: %.3f\" % 0)\n",
        "print(\"acc on TEST : %.3f\" % 0)\n",
        "\n",
        "print(\"\")\n",
        "print(\"Confusion matrix for TEST:\")\n",
        "# TODO: call print(calc_confusion_matrix_for_threshold(...))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "_Zvyn_ohNFZE"
      },
      "source": [
        "#### (iii) Compare the `Perceptron` to the always-0 classifier.\n",
        "\n",
        "**Answer**:  TODO\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "7QhGQb8oNFZF"
      },
      "source": [
        "#### (iv) Generate a series of regularized perceptron models\n",
        "Each model will use a different `alpha` value, multiplying that by the L2 penalty.  You will record and plot the accuracy of each model on both training and test data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ovvoBr6sNFZF"
      },
      "outputs": [],
      "source": [
        "train_accuracy_list = list()\n",
        "test_accuracy_list = list()\n",
        "\n",
        "# TODO: create, fit models here and record accuracy of each (Implement functions needed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "SkdMjTf3NFZF"
      },
      "source": [
        "##### Plot accuracy on train/test data across the different alpha values plotted on a logarithmic scale. Make sure to show title, legends, and axis labels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G_-z0ca3NFZF"
      },
      "outputs": [],
      "source": [
        "# TODO make plot\n",
        "plt.xlabel('log_10(alpha)');\n",
        "plt.ylabel('Accuracy');\n",
        "\n",
        "# TODO add legend, titles, etc. set x-scale appropriately\n",
        "# plt.legend(...);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "wJLbOIVkNFZF"
      },
      "source": [
        "#### (iv) Discuss what the plot is showing you.\n",
        "\n",
        "**Answer**: TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "dr02LiW2NFZF"
      },
      "source": [
        "### 5: Decision functions and probabilistic predictions\n",
        "\n",
        "#### (a) Create two new sets of predictions\n",
        "\n",
        "Fit `Perceptron` and `CalibratedClassifierCV` models to the data.  Use their predictions to generate ROC curves."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bb_Ix3R_NFZF"
      },
      "outputs": [],
      "source": [
        "# TODO: fit a Perceptron and generate its decision_function() over the test data.\n",
        "\n",
        "# TODO: Build a CalibratedClassifierCV, using a Perceptron as its base_estimator,\n",
        "#       and generate its probabilistic predictions over the test data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ymmNuDj4NFZF"
      },
      "outputs": [],
      "source": [
        "# TODO something like: fpr, tpr, thr = sklearn.metrics.roc_curve(...)\n",
        "\n",
        "plt.ylim([0, 1]);\n",
        "plt.legend(loc='lower right')\n",
        "plt.xlabel(\"False Postive rate (FPR)\");\n",
        "plt.ylabel(\"True Postive rate (TPR)\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o5hnESZjNFZF"
      },
      "outputs": [],
      "source": [
        "print(\"AUC on TEST for Perceptron: %.3f\" % 0) #TODO: modify these values\n",
        "print(\"AUC on TEST for probabilistic model: %.3f\" % 0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "k1rTdSTLNFZF"
      },
      "source": [
        "#### (b) Discuss the results above\n",
        "\n",
        "**Answer**: TODO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "WA2fZtTENFZF"
      },
      "source": [
        "#### (c) Compute model metrics for different probabilistic thresholds\n",
        "\n",
        "Complete `calc_perf_metrics_for_threshold` that takes in a set of correct outputs, a matching set of probabilities generated by a classifier, and a threshold at which to set the positive decision probability, and returns a set of metrics if we use that threshold."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "0fTBk8dYNFZG"
      },
      "source": [
        "#### (d) Compare the probabilistic classifier across multiple decision thresholds\n",
        "\n",
        "Try a range of thresholds for classifying data into the positive class (1).  For each threshold, compute the true positive rate (TPR) and positive predictive value (PPV).  Record the best value of each metric, along with the threshold that achieves it, and the *other* metric at that threshold."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n_DbFaxINFZG"
      },
      "outputs": [],
      "source": [
        "# TODO: test different thresholds to compute these values\n",
        "best_TPR = 0\n",
        "best_PPV_for_best_TPR = 0\n",
        "best_TPR_threshold = 0\n",
        "\n",
        "best_PPV = 0\n",
        "best_TPR_for_best_PPV = 0\n",
        "best_PPV_threshold = 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dxv-ivSFNFZG"
      },
      "outputs": [],
      "source": [
        "print(\"Best TPR threshold: %.4f => TPR: %.4f; PPV: %.4f\" % (best_TPR_threshold, best_TPR, best_PPV_for_best_TPR))\n",
        "print(\"Best PPV threshold: %.4f => PPV: %.4f; TPR: %.4f\" % (best_PPV_threshold, best_PPV, best_TPR_for_best_PPV))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "4_ca2WmpNFZG"
      },
      "source": [
        "#### (e) Exploring different thresholds\n",
        "\n",
        "#### (i) Using default 0.5 threshold.\n",
        "\n",
        "Generate confusion matrix and metrics for probabilistic classifier, using threshold 0.5."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3F-nk4rSNFZG"
      },
      "outputs": [],
      "source": [
        "best_thr = 0.5\n",
        "print(\"ON THE TEST SET:\")\n",
        "print(\"Chosen best threshold = %.4f\" % best_thr)\n",
        "print(\"\")\n",
        "# TODO: print(calc_confusion_matrix_for_threshold(...))\n",
        "print(\"\")\n",
        "# TODO: print_perf_metrics_for_threshold(...)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "AGM-M9_CNFZG"
      },
      "source": [
        "#### (ii) Using threshold with highest TPR.\n",
        "\n",
        "Generate confusion matrix and metrics for probabilistic classifier, using threshold that maximizes TPR."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "12UTSpMwNFZG"
      },
      "outputs": [],
      "source": [
        "best_thr = best_TPR_threshold\n",
        "print(\"ON THE TEST SET:\")\n",
        "print(\"Chosen best threshold = %.4f\" % best_thr)\n",
        "print(\"\")\n",
        "# TODO: print(calc_confusion_matrix_for_threshold(...))\n",
        "print(\"\")\n",
        "# TODO: print_perf_metrics_for_threshold(...)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "cWeOEKiVNFZG"
      },
      "source": [
        "#### (iii) Using threshold with highest PPV.\n",
        "\n",
        "Generate confusion matrix and metrics for probabilistic classifier, using threshold that maximizes PPV."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zke3NdTFNFZG"
      },
      "outputs": [],
      "source": [
        "best_thr = best_PPV_threshold\n",
        "print(\"ON THE TEST SET:\")\n",
        "print(\"Chosen best threshold = %.4f\" % best_thr)\n",
        "print(\"\")\n",
        "# TODO: print(calc_confusion_matrix_for_threshold(...))\n",
        "print(\"\")\n",
        "# TODO: print_perf_metrics_for_threshold(...)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": false,
        "id": "vRfzzRXsNFZG"
      },
      "source": [
        "#### (iv) Compare the confusion matrices from (a)–(c) to analyze the different thresholds.\n",
        "\n",
        "**Answer**: TODO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9s10ndvPNFZH"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}