notebooks/02_numerical_pipeline_hands_on.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with numerical data\n",
    "\n",
    "In the previous notebook, we trained a k-nearest neighbors model on some data.\n",
    "\n",
    "However, we oversimplified the procedure by loading a dataset that contained\n",
    "exclusively numerical data. Besides, we used datasets which were already split\n",
    "into train-test sets.\n",
    "\n",
    "In this notebook, we aim at:\n",
    "\n",
    "* identifying numerical data in a heterogeneous dataset;\n",
    "* selecting the subset of columns corresponding to numerical data;\n",
    "* using a scikit-learn helper to separate data into train-test sets;\n",
    "* training and evaluating a more complex scikit-learn model.\n",
    "\n",
    "We start by loading the adult census dataset used during the data exploration.\n",
    "\n",
    "## Loading the entire dataset\n",
    "\n",
    "As in the previous notebook, we rely on pandas to open the CSV file into a\n",
    "pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")\n",
    "# drop the duplicated column `\"education-num\"` as stated in the first notebook\n",
    "adult_census = adult_census.drop(columns=\"education-num\")\n",
    "adult_census"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step separates the target from the data. We performed the same\n",
    "procedure in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, target = adult_census.drop(columns=\"class\"), adult_census[\"class\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">Here and later, we use the name <tt class=\"docutils literal\">data</tt> and <tt class=\"docutils literal\">target</tt> to be explicit. In\n",
    "scikit-learn documentation, <tt class=\"docutils literal\">data</tt> is commonly named <tt class=\"docutils literal\">X</tt> and <tt class=\"docutils literal\">target</tt> is\n",
    "commonly called <tt class=\"docutils literal\">y</tt>.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we can focus on the data we want to use to train our predictive\n",
    "model.\n",
    "\n",
    "## Identify numerical data\n",
    "\n",
    "Numerical data are represented with numbers. They are linked to measurable\n",
    "(quantitative) data, such as age or the number of hours a person works a week.\n",
    "\n",
    "Predictive models are natively designed to work with numerical data. Moreover,\n",
    "numerical data usually requires very little work before getting started with\n",
    "training.\n",
    "\n",
    "The first task here is to identify numerical data in our dataset.\n",
    "\n",
    "<div class=\"admonition caution alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Caution!</p>\n",
    "<p class=\"last\">Numerical data are represented with numbers, but numbers do not always\n",
    "represent numerical data. Categories could already be encoded with\n",
    "numbers and you may need to identify these features.</p>\n",
    "</div>\n",
    "\n",
    "Thus, we can check the data type for each of the column in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We seem to have only two data types: `int64` and `object`. We can make sure by\n",
    "checking for unique data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dtypes.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the only two types in the dataset are integer `int64` and `object`. We\n",
    "can look at the first few lines of the dataframe to understand the meaning of\n",
    "the `object` data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the `object` data type corresponds to columns containing strings.\n",
    "As we saw in the exploration section, these columns contain categories and we\n",
    "will see later how to handle those. We can select the columns containing\n",
    "integers and check their content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_columns = [\"age\", \"capital-gain\", \"capital-loss\", \"hours-per-week\"]\n",
    "data[numerical_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we limited the dataset to numerical columns only, we can analyse\n",
    "these numbers to figure out what they represent. We can identify two types of\n",
    "usage.\n",
    "\n",
    "The first column, `\"age\"`, is self-explanatory. We can note that the values\n",
    "are continuous, meaning they can take up any number in a given range. Let's\n",
    "find out what this range is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"age\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the age varies between 17 and 90 years.\n",
    "\n",
    "We could extend our analysis and we would find that `\"capital-gain\"`,\n",
    "`\"capital-loss\"`, and `\"hours-per-week\"` are also representing quantitative\n",
    "data.\n",
    "\n",
    "Now, we store the subset of numerical columns in a new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_numeric = data[numerical_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train-test split the dataset\n",
    "\n",
    "In the previous notebook, we loaded two separate datasets: a training one and\n",
    "a testing one. However, having separate datasets in two distincts files is\n",
    "unusual: most of the time, we have a single file containing all the data that\n",
    "we need to split once loaded in the memory.\n",
    "\n",
    "Scikit-learn provides the helper function\n",
    "`sklearn.model_selection.train_test_split` which is used to automatically\n",
    "split the dataset into two subsets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data_numeric, target, random_state=42, test_size=0.25\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition tip alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Tip</p>\n",
    "<p class=\"last\">In scikit-learn setting the <tt class=\"docutils literal\">random_state</tt> parameter allows to get\n",
    "deterministic results when we use a random number generator. In the\n",
    "<tt class=\"docutils literal\">train_test_split</tt> case the randomness comes from shuffling the data, which\n",
    "decides how the dataset is split into a train and a test set).</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When calling the function `train_test_split`, we specified that we would like\n",
    "to have 25% of samples in the testing set while the remaining samples (75%)\n",
    "are assigned to the training set. We can check quickly if we got what we\n",
    "expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"Number of samples in testing: {data_test.shape[0]} => \"\n",
    "    f\"{data_test.shape[0] / data_numeric.shape[0] * 100:.1f}% of the\"\n",
    "    \" original set\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"Number of samples in training: {data_train.shape[0]} => \"\n",
    "    f\"{data_train.shape[0] / data_numeric.shape[0] * 100:.1f}% of the\"\n",
    "    \" original set\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook, we used a k-nearest neighbors model. While this\n",
    "model is intuitive to understand, it is not widely used in practice. Now, we\n",
    "use a more useful model, called a logistic regression, which belongs to the\n",
    "linear models family.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p>In short, linear models find a set of weights to combine features linearly\n",
    "and predict the target. For instance, the model can come up with a rule such\n",
    "as:</p>\n",
    "<ul class=\"simple\">\n",
    "<li>if <tt class=\"docutils literal\">0.1 * age + 3.3 * <span class=\"pre\">hours-per-week</span> - 15.1 &gt; 0</tt>, predict <tt class=\"docutils literal\"><span class=\"pre\">high-income</span></tt></li>\n",
    "<li>otherwise predict <tt class=\"docutils literal\"><span class=\"pre\">low-income</span></tt></li>\n",
    "</ul>\n",
    "<p class=\"last\">Linear models, and in particular the logistic regression, will be covered\n",
    "more in detail in the \"Linear models\" module later in this course. For now the\n",
    "focus is to use this logistic regression model in scikit-learn rather than\n",
    "understand how it works in details.</p>\n",
    "</div>\n",
    "\n",
    "To create a logistic regression model in scikit-learn you can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the model has been created, you can use it exactly the same way as we\n",
    "used the k-nearest neighbors model in the previous notebook. In particular, we\n",
    "can use the `fit` method to train the model using the training data and\n",
    "labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `score` method to check the model generalization\n",
    "performance on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model.score(data_test, target_test)\n",
    "print(f\"Accuracy of logistic regression: {accuracy:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook recap\n",
    "\n",
    "In scikit-learn, the `score` method of a classification model returns the\n",
    "accuracy, i.e. the fraction of correctly classified samples. In this case,\n",
    "around 8 / 10 of the times the logistic regression predicts the right income\n",
    "of a person. Now the real question is: is this generalization performance\n",
    "relevant of a good predictive model? Find out by solving the next exercise!\n",
    "\n",
    "In this notebook, we learned to:\n",
    "\n",
    "* identify numerical data in a heterogeneous dataset;\n",
    "* select the subset of columns corresponding to numerical data;\n",
    "* use the scikit-learn `train_test_split` function to separate data into a\n",
    "  train and a test set;\n",
    "* train and evaluate a logistic regression model."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}