code/tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Machine Learning with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### at MSU Data Science 2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "last updated: 2018-02-21 \n",
      "\n",
      "numpy 1.12.1\n",
      "scipy 1.0.0\n",
      "matplotlib 2.1.2\n",
      "sklearn 0.19.1\n",
      "pandas 0.22.0\n",
      "mlxtend 0.10.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a \"Sebastian Raschka\" -u -d -p numpy,scipy,matplotlib,sklearn,pandas,mlxtend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Links\n",
    "\n",
    "- GitHub repository: https://github.com/rasbt/msu-datascience-ml-tutorial-2018\n",
    "- Slides: https://speakerdeck.com/rasbt/machine-learning-with-python         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "* [1 Introduction to Machine Learning](#1-Introduction-to-Machine-Learning)\n",
    "* [2 Linear Regression](#2-Linear-Regression)\n",
    "    * [Loading the dataset](#Loading-the-dataset)\n",
    "    * [Preparing the dataset](#Preparing-the-dataset)\n",
    "    * [Fitting the model](#Fitting-the-model)\n",
    "    * [Evaluating the model](#Evaluating-the-model)\n",
    "* [3 Introduction to Classification](#3-Introduction-to-Classification)\n",
    "    * [The Iris dataset](#The-Iris-dataset)\n",
    "    * [Class label encoding](#Class-label-encoding)\n",
    "    * [Scikit-learn's in-build datasets](#Scikit-learn's-in-build-datasets)\n",
    "    * [Test/train splits](#Test/train-splits)\n",
    "    * [Logistic Regression](#Logistic-Regression)\n",
    "    * [K-Nearest Neighbors](#K-Nearest-Neighbors)\n",
    "    * [3 - Exercises](#3---Exercises)\n",
    "* [4 - Feature Preprocessing & scikit-learn Pipelines](#4---Feature-Preprocessing-&-scikit-learn-Pipelines)\n",
    "    * [Categorical features: nominal vs ordinal](#Categorical-features:-nominal-vs-ordinal)\n",
    "    * [Normalization](#Normalization)\n",
    "    * [Pipelines](#Pipelines)\n",
    "    * [4 - Exercises](#4---Exercises)\n",
    "* [5 - Dimensionality Reduction: Feature Selection & Extraction](#5---Dimensionality-Reduction:-Feature-Selection-&-Extraction)\n",
    "    * [Recursive Feature Elimination](#Recursive-Feature-Elimination)\n",
    "    * [Sequential Feature Selection](#Sequential-Feature-Selection)\n",
    "    * [Principal Component Analysis](#Principal-Component-Analysis)\n",
    "* [6 - Model Evaluation & Hyperparameter Tuning](#6---Model-Evaluation-&-Hyperparameter-Tuning)\n",
    "    * [Wine Dataset](#Wine-Dataset)\n",
    "    * [Stratified K-Fold](#Stratified-K-Fold)\n",
    "    * [Grid Search](#Grid-Search)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import general packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Introduction to Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source: R.J. Gladstone (1905). \"A Study of the Relations of the Brain to \n",
    "to the Size of the Head\", Biometrika, Vol. 4, pp105-123\n",
    "\n",
    "\n",
    "Description: Brain weight (grams) and head size (cubic cm) for 237\n",
    "adults classified by gender and age group.\n",
    "\n",
    "\n",
    "Variables/Columns\n",
    "- Gender (1=Male, 2=Female)\n",
    "- Age Range (1=20-46, 2=46+)\n",
    "- Head size (cm^3)\n",
    "- Brain weight (grams)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a screenshot showing the top rows of the dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/dataset_brain_screenshot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load the dataset conveniently using pandas' CSV (comma separated values) reader. Note that the columns in the data file are actually not comma-separated; thus, we use the `'\\s+'` string, which is a regex shortcut specifying and arbitrary number of whitespaces as column separator. Typically, we use `sep=','` (the default) for comma-separated columns or `sep='\\t'` for tab-separated columns. Further, the `comment='#'` argument specifies that the first few lines of the dataset (here, a description of the dataset) should be skipped. The `encoding='utf-8'` is typically not necessary, but based on my experience in the past, omitting it can cause issues for some Windows users.\n",
    "\n",
    "Also, for more information on specific functions, you can always use the `?` operator in pandas (which is sometimes more convenient than looking for documentation online); e.g., copy and paste the following command into a new notebook cell and execute it to get more information: `pd.read_csv?`.\n",
    "\n",
    "Now, let's get back to reading the `dataset_brain.txt` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('dataset_brain.txt',\n",
    "                 encoding='utf-8',\n",
    "                 comment='#',\n",
    "                 sep='\\s+')\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's visualize the dataset using matplotlib.pyplot, which we imported as `plt` earlier. Here, we will only be looking at two variables, `'head-size'` and `'brain-weight'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(df['head-size'], df['brain-weight'])\n",
    "plt.xlabel('Head size (cm^3)')\n",
    "plt.ylabel('Brain weight (grams)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming that `'brain-weight'` is the target variable (which we want to predict) and `'head-size'` is the feature we use for the prediction, let's prepare the dataset for scikit-learn. I.e., we assign the `'brain-weight'` to an array `y` and the feature to an array `X` (a common convention when using scikit-learn):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['brain-weight'].values\n",
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df['head-size'].values\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that scikit-learn expects 2D arrays as feature arrays; hence, we use `np.newaxis` to append another axis in this case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X[:, np.newaxis]\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both the targets and the features in a \"comfortable\" format, let's split the dataset into two separate, non-overlapping sets, a training and a test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, shuffle=True, test_size=0.3, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We used a fixed random seed to perform the split (here, `random_state=123`), which ensures that the data is always shuffled the same way prior to splitting -- this done is for reproducibility purposes. Note that you can choose any (positive) integer you like as a random seed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see how our splits look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X_train, y_train, marker='o', label='training data')\n",
    "plt.scatter(X_test, y_test, marker='s', label='test data')\n",
    "plt.xlabel('Head size (cm^3)')\n",
    "plt.ylabel('Brain weight (grams)')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, on to the more exciting parts, fitting a linear regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `fit` method, we train the model on the training dataset (the features and targets), so that the model \"learns\" the model coefficients of the linear regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.coef_ # slope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.intercept_ # y-axis intercept"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, to make predictions on new data (here: predicting the brain weight based on the head size), we use the `predict` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = lr.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A useful quantitative measure of a model's performance is the so-called Mean Squared Error (MSE), which is simply the average value of the SSE cost function that we minimize to fit the linear regression model. The MSE is useful to for comparing different regression models or for tuning their parameters via a grid search and cross-validation:\n",
    "\n",
    "$$MSE = \\frac{1}{n} \\sum_{i=1}^{n} \\big( y^{(i)} - \\widehat{y}^{(i)}  \\big)^2.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it may be more useful to report the coefficient of determination ($R^2$), which can be understood as a standardized version of the MSE, for better interpretability of the model performance. In other words, $R^2$ is the fraction of response variance that is captured by the model. The $R^2$ value is defined as follows:\n",
    "\n",
    "$$R^2 = 1 - \\frac{SSE}{SST}$$\n",
    "\n",
    "Here, SSE is the sum of squared errors and SST is the total sum of squares $SST = \\sum^{n}_{i=1} \\big( y^{(i)} - \\mu_y \\big)^2$, or in other words, it is simply the variance of the response. Let's quickly show that $R^2$ is indeed just the rescaled version of the MSE:\n",
    "\n",
    "$$R^2 = 1 - \\frac{SSE}{SST}$$\n",
    "\n",
    "$$= 1 - \\frac{\\frac{1}{n} \\sum^{n}_{i=1} \\big( y^{(i)} - \\hat{y}^{(i)}  \\big)^2 }{\\frac{1}{n} \\sum_{i=1}^{n} \\big( y^{(i)} - \\mu_y \\big)^2 }$$\n",
    " \n",
    "$$= 1 - \\frac{MSE}{Var(y)}$$\n",
    "\n",
    "For the training dataset, $R^2$ is bounded between 0 and 1, but it can become negative for the test set. If $R^2$ =1, the model  fits the data perfectly with a\n",
    "corresponding $MSE = 0$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do this calculation \"manually,\" first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sum_of_squares = ((y_test - y_pred)**2).sum()\n",
    "res_sum_of_squares = ((y_test - y_test.mean())**2).sum()\n",
    "r2_score = 1 - (sum_of_squares / res_sum_of_squares)\n",
    "print('R2 score: %.3f' % r2_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the $R^2$ can conveniently be computed via the regressor's `score` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('R2 score: %.3f' % lr.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's visualize how the regression fit looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_pred = X_train.min() * lr.coef_ + lr.intercept_\n",
    "max_pred = X_train.max() * lr.coef_ + lr.intercept_\n",
    "\n",
    "plt.scatter(X_train, y_train, marker='o')\n",
    "plt.plot([X_train.min(), X_train.max()],\n",
    "         [min_pred, max_pred],\n",
    "         color='darkorange',\n",
    "         linewidth=4)\n",
    "plt.xlabel('Head size (cm^3)')\n",
    "plt.ylabel('Brain weight (grams)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that other performance metrics are available from sckit-learn's `metrics` module, http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics. For instance, we can do the following to compute the MSE:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "mean_squared_error(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note that the MSE value looks very large; this is because we haven't normalized the data; we will talk about that later.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Introduction to Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Iris dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's start by taking a quick look at the structure of the dataset, which is shown in the following screenshot:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/dataset_iris_screenshot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the dataset is organized in 5 columns, which are separated by commas. The first four columns contain the features (several flower leaf dimensions in cm), and the last column contains the class labels. Using the convenient pandas `read_csv` function, we read the dataset into a DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('dataset_iris.txt',\n",
    "                 encoding='utf-8',\n",
    "                 comment='#',\n",
    "                 sep=',')\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we assign the features to an NumPy array `X` and the class labels to a NumPy array `y`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:, :4].values\n",
    "y = df['class'].values\n",
    "\n",
    "print('X shape:', X.shape)\n",
    "print('y shape:', y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unique class labels are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Class label encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While most scikit-learn classifier objects can take care of the label encoding internally, it's usually recommended/common practice in machine learning to work with numeric (integer) labels for computationally efficiency/compatibility reasons.\n",
    "\n",
    "So, this dataset offers a good opportunity for explaining how we can conveniently use some of scikit-learn's utilities to encode labels, starting with the `LabelEncoder`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "l_encoder = LabelEncoder()\n",
    "l_encoder.fit(y)\n",
    "l_encoder.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Via the `fit` method, the encoder learned a mapping between string and integer class labels. For instance, the first element in the `classes_` array corresponds to label 0, the second entry corresponds to label 1, and so forth. Thus, the mapping in this case would be the following one:\n",
    "\n",
    "- 'Iris-setosa' -> 0\n",
    "- 'Iris-versicolor' -> 1\n",
    "- 'Iris-virginica' -> 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To transform the class labels stored in array y, we can call the `transform` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_enc = l_encoder.transform(y)\n",
    "np.unique(y_enc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conveniently, there's also a `inverse_transform` method that allows us to transform integer class labels back into their string representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.unique(l_encoder.inverse_transform(y_enc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scikit-learn's in-built datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our convenience, scikit-learn comes with many different toy/benchmark datasets built-in, which are useful for learning and debugging purposes: http://scikit-learn.org/stable/datasets/index.html#toy-datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can load the same iris dataset from scikit-learn as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "print(iris['DESCR'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test/train splits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyways, after this little \"label encoding digression\" let's continue with the main objective, fitting a classifier. To fit a classifier, we need a training dataset (and an independent dataset for testing).\n",
    "\n",
    "Here, we are only focusing on 2 features, sepal length and sepal width, for illustrative purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = iris.data[:, :2], iris.target\n",
    "# ! We only use 2 features for visual purposes\n",
    "\n",
    "print('Class labels:', np.unique(y))\n",
    "print('Class proportions:', np.bincount(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to split the dataset into 70% training data and 30% test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123)\n",
    "\n",
    "print('Class labels:', np.unique(y_train))\n",
    "print('Class proportions (training set):', np.bincount(y_train))\n",
    "print('Class proportions (test set):', np.bincount(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the discussion about class imbalance and stratification from the lecture? Especially if the dataset is small, random subsampling can skew the class label distributions a lot! Fortunately, it's easy to perform a stratified split by providing the class labels to the `stratify` parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123, stratify=y)\n",
    "\n",
    "print('Class labels:', np.unique(y_train))\n",
    "print('Class proportions (training set):', np.bincount(y_train))\n",
    "print('Class proportions (test set):', np.bincount(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to fit a logistic regression model to predict whether an unseen flower is either a Iris setosa, versicolor, or virginica flower. Note that logistic regression is actually a binary classifier, meaning that it can only predict binary outcomes (0 or 1). There are multiple ways for applying logistic regression to multi-class problems. One of these is called OvR (one-vs-rest). \n",
    "\n",
    "in OvR, we fit 3 different logistic regression classifiers:\n",
    "1. Setosa vs versicolor & virginica\n",
    "2. Versicolor vs setosa & virginica\n",
    "3. Virginica vs setosa & versicolor\n",
    "\n",
    "Then, during prediction time, we let all of these classifiers make a prediction, and for the final prediction, we take the one classifier that outputs the highest probability for the *one* class it was trained on against the *rest*.\n",
    "\n",
    "Another method is OvO (one-vs-one). Here, we train $K (K − 1) / 2$ binary logistic regression classifiers, where K is the number of classes (here: $3 (3-1)/2 = 3$). Then, to make a prediction, all of the binary classifiers are used, and the class with the highest number of votes gets selected.\n",
    "\n",
    "The third strategy, and probably the most efficient and natural one when it comes to logistic regression, is the so-called multinomial regression model (or also known as softmax regression), which is a generalization of logistic regression for more than two classes. In a nutshell, the modification is depicted in the following figure:\n",
    "\n",
    "![](images/softmax.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't go over the mathematical details due to timing constraints, but if you are interested, I have a write-up at http://rasbt.github.io/mlxtend/user_guide/classifier/SoftmaxRegression/.\n",
    "\n",
    "In any case, this was just the long way of explaining where the term \"multinomial\" in the following code example comes from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lr = LogisticRegression(solver='newton-cg', \n",
    "                        multi_class='multinomial', \n",
    "                        random_state=1)\n",
    "\n",
    "lr.fit(X_train, y_train)\n",
    "print('Test accuracy %.2f' % lr.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code snippet above should look familiar to you. One of the nice things about scikit-learn is that the API is not only convenient but also consistent. Hence, the procedure for fitting and evaluating a classifier is exactly is equivalent to the approach we've used earlier to fit the linear regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to use a handy tool that I implemented in mlxtend to visualize the decision regions of classifiers in 2D:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.plotting import plot_decision_regions\n",
    "\n",
    "plot_decision_regions\n",
    "\n",
    "plot_decision_regions(X=X, y=y, clf=lr, X_highlight=X_test)\n",
    "plt.xlabel('sepal length [cm]')\n",
    "plt.xlabel('sepal width [cm]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that both training and test data points are shown, where the test data points are circled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Nearest Neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we are going to use a K-nearest neighbor classifier, which is a \"lazy\" learner as discussed in the lecture slides:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "kn = KNeighborsClassifier(n_neighbors=4)\n",
    "\n",
    "kn.fit(X_train, y_train)\n",
    "print('Test accuracy %.2f' % kn.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_decision_regions(X=X, y=y, clf=kn, X_highlight=X_test)\n",
    "plt.xlabel('sepal length [cm]')\n",
    "plt.xlabel('sepal width [cm]');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Which of the two models above would you prefer if you had to choose? Why?\n",
    "- What would be possible ways to resolve ties in KNN when `n_neighbors` is an even number?\n",
    "- Can you find the right spot in the scikit-learn documentation to read about how scikit-learn handles this?\n",
    "- Train & evaluate the Logistic Regression and KNN algorithms on the 4-dimensional iris datasets. \n",
    "  - What performance do you observe? \n",
    "  - Why is it different vs. using only 2 dimensions? \n",
    "  - Would adding more dimensions help?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Feature Preprocessing & scikit-learn Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical features: nominal vs ordinal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are working with categorical features, we need to be careful to encode them correctly. For instance if a feature is nominal (no order implied), we do not want to simply replace feature values by different integers as machine learning algorithms usually assume the following relationship: 0 < 1 < 2 < 3 ... etc. In the next sections, we will look over some approaches on how to accomplish that. But first, let's start with a simple toy data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame([['green', 'M', 10.0],\n",
    "                   ['red', 'L', 13.5],\n",
    "                   ['blue', 'XL', 15.3]])\n",
    "\n",
    "df.columns = ['color', 'size', 'prize']\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One out of many approaches includes using scikit-learn's `DictVectorizer`, which requires a Python dictionary as input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_as_dict = df.transpose().to_dict()\n",
    "data_as_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usage of transformer classes in scikit-learn is similar to how we use classifiers or regressors. First, we call a `fit` method on the dataset, and then we invoke a `transform` method (instead of `predict`) to transform a dataset. Note that `.fit_transform(X)` is a shortcut for `.fit(X).transform(X)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "\n",
    "dvec = DictVectorizer(sparse=False)\n",
    "\n",
    "X = dvec.fit_transform(data_as_dict.values())\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the `DictVectorizer` automatically converts string variables into a one-hot encoded format. However, if you are interested in just converting integers into a one-hot encoded format, also have a look at scikit-learn's `OneHotEncoder` class (http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html).\n",
    "\n",
    "However, in practice, we may want to encode ordinal variables in a different way as they have an internal order (the values needed to be decided by the researcher and varies from case to case). For instance, it's super easy to achieve such an encoding using our pandas DataFrame via the map function and a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size_mapping = {'XL': 3, 'L': 2, 'M': 1}\n",
    "\n",
    "df_new = df.copy()\n",
    "df_new['size'] = df_new['size'].map(size_mapping)\n",
    "df_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = dvec.fit_transform(df_new.transpose().to_dict().values())\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that NumPy arrays only have a single type, hence, the last column contains floats, not integers as in the DataFrame `df_new`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data normalization can be performed quite easily with scikit-learn as well, using the transformer interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame([1., 2., 3., 4., 5., 6.], columns=['feature'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "mmxsc = MinMaxScaler()\n",
    "stdsc = StandardScaler()\n",
    "\n",
    "X = df['feature'].values[:, np.newaxis]\n",
    "\n",
    "df['minmax'] = mmxsc.fit_transform(X)\n",
    "df['z-score'] = stdsc.fit_transform(X)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, it's usually recommended to normalize your input data. The only algorithms that is scale-invariant and comes to mind are tree-based algorithms (decision trees, random forests, etc.). \n",
    "\n",
    "\n",
    "Some examples of algorithms where feature scaling matters are:\n",
    "\n",
    "- k-nearest neighbors with an Euclidean distance measure if want all features to contribute equally\n",
    "- k-means (see k-nearest neighbors)\n",
    "- logistic regression, SVMs, perceptrons, neural networks etc. if you are using gradient descent/ascent-based optimization, otherwise some weights will update much faster than others\n",
    "- linear discriminant analysis, principal component analysis, kernel principal component analysis since you want to find directions of maximizing the variance (under the constraints that those directions/eigenvectors/principal components are orthogonal); you want to have features on the same scale since you'd emphasize variables on \"larger measurement scales\" more.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another important point to highlight is that we want to retrieve the normalization parameters from the training set and apply these to the test set (**don't refit a transformer on the test set!**), otherwise, funny things might happen.\n",
    "\n",
    "\n",
    "Let me give a hands-on example why this is important!\n",
    "\n",
    "Let's imagine we have a simple training set consisting of 3 samples with 1 feature column (let's call the feature column length in cm\"):\n",
    "\n",
    "- sample1: 10 cm -> class 2\n",
    "- sample2: 20 cm -> class 2\n",
    "- sample3: 30 cm -> class 1\n",
    "\n",
    "Given the data above, we compute the following parameters:\n",
    "\n",
    "- mean: 20\n",
    "- standard deviation: 8.2\n",
    "\n",
    "If we use these parameters to standardize the same dataset, we get the following values:\n",
    "\n",
    "- sample1: -1.21 -> class 2\n",
    "- sample2: 0 -> class 2\n",
    "- sample3: 1.21 -> class 1\n",
    "\n",
    "Now, let's say our model has learned the following hypotheses: It classifies samples with a standardized length value < 0.6 as class 2 (class 1 otherwise). So far so good. Now, let’s imagine we have 3 new unlabeled data points that you want to classify.\n",
    "\n",
    "- sample4: 5 cm -> class ?\n",
    "- sample5: 6 cm -> class ?\n",
    "- sample6: 7 cm -> class ?\n",
    "\n",
    "If we look at the \"unstandardized \"length in cm\" values in our training dataset, it is intuitive to say that all of these samples are likely belonging to class 2. However, if we standardize these by re-computing the *standard deviation* and and *mean* from the new data, we would get similar values as before (i.e., properties of a standard normal distribution) in the training set and our classifier would (probably incorrectly) assign the \"class 2\" label to the samples 4 and 5.\n",
    "\n",
    "- sample5: -1.21 -> class 2\n",
    "- sample6: 0 -> class 2\n",
    "- sample7: 1.21 -> class 1\n",
    "\n",
    "However, if we use the parameters from your \"training set standardization, we will get the following standardized values\n",
    "\n",
    "- sample5: -18.37\n",
    "- sample6: -17.15 \n",
    "- sample7: -15.92\n",
    "\n",
    "Note that these values are more negative than the value of `sample1` in the original training set, which makes much more sense now!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn's pipelines are a super convenient tools that allow us to group multiple, different preprocessing steps together. For instance, in this simple example, we will be creating a pipelines that bundles a `StandardScaler` with a `LogisticRegression` classifier.\n",
    "\n",
    "Note that the a `Pipeline` has a API that is similar to the classifier's API. If we call `fit` using a training dataset, it will fit the `StandardScaler` within the pipeline, transform the training dataset (here: standardizing it), and provide it to the classifier's fit method. Then, upon calling `predict` on the pipeline (or `score`), the test data will automatically be transformed via the `StandardScaler` and passed on to the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "X, y = iris.data, iris.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123, stratify=y)\n",
    "\n",
    "lr = LogisticRegression(solver='newton-cg',\n",
    "                        multi_class='multinomial',\n",
    "                        random_state=1)\n",
    "\n",
    "lr_pipe = make_pipeline(StandardScaler(), lr)\n",
    "\n",
    "lr_pipe.fit(X_train, y_train)\n",
    "lr_pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this is just a simple example, pipelines are a super important concept in scikit-learn, because it not only allows us to perform operations more conveniently by grouping them, but also enforces doing them the \"right\" way, i.e., using training set parameters (e.g., the mean and standard deviation from standardization) on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in case we want to access the individual elements within the pipeline, we can do so via the `named_steps` dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_pipe.named_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_pipe.named_steps['standardscaler'].transform(X[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 - Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Why is it important that we scale test and training sets separately?\n",
    "- Fit a KNN classifier to the standardized Iris dataset. Do you notice difference in the predictive performance of the model compared to the non-standardized one? Why or why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Dimensionality Reduction: Feature Selection & Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will introduce some basic feature selection and feature extraction techniques that can be used for dimensionality reduction. As always, we start by loading a dataset and splitting it into a training and a test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "iris = load_iris()\n",
    "X, y = iris.data, iris.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123, stratify=y)\n",
    "\n",
    "sc = StandardScaler()\n",
    "X_train = sc.fit_transform(X_train)\n",
    "X_test = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recursive Feature Elimination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature elimination fits an a regression model iteratively. It starts with the full feature subset (the 4 Iris features) and removes the feature with the smallest regression coefficient in each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_selection import RFE\n",
    "\n",
    "lr = LogisticRegression()\n",
    "rfe = RFE(lr, step=1)\n",
    "\n",
    "rfe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When it's finished, we can access the number of features that resulted in the best model via the `n_features_` attribute. Also, if we want to know which features were selected, we can look at the array stored under the `ranking_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Number of features:', rfe.n_features_)\n",
    "print('Feature ranking', rfe.ranking_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lower the ranking number the better; ranking values of 1 correspond to the selected features. In this case, this would be the petal length and petal width of the iris flowers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequential Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequential feature selection algorithms are a family of greedy search algorithms that are used to reduce an initial d-dimensional feature space to a k-dimensional feature subspace where k < d. The motivation behind feature selection algorithms is to automatically select a subset of features that is most relevant to the problem. The goal of feature selection is two-fold: We want to improve the computational efficiency and reduce the generalization error of the model by removing irrelevant features or noise. A wrapper approach such as sequential feature selection is especially useful if embedded feature selection -- for example, a regularization penalty like LASSO -- is not applicable.\n",
    "\n",
    "In a nutshell, SFAs remove or add one feature at the time based on the classifier performance until a feature subset of the desired size k is reached.\n",
    "\n",
    "\n",
    "How is this different from Recursive Feature Elimination (RFE) -- e.g., as implemented in `sklearn.feature_selection.RFECV`? `RFECV` is computationally less complex using the feature weight coefficients (e.g., linear models) or feature importance (tree-based algorithms) to eliminate features recursively, whereas SFSs eliminate (or add) features based on a user-defined classifier/regression performance metric.\n",
    "\n",
    "(Please see http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/ for more info).\n",
    "\n",
    "Note that sequential feature selection algorithms are currently not implemented in scikit-learn; however, we are in the process of porting my mlxtend implementation over to scikit-learn.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following code, we will let the SFS run over all 4 features to suggest a feature subset for each feature size via forward selection. Once it's completed, we can access the scores for each subset via the `subsets_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.feature_selection import SequentialFeatureSelector as SFS\n",
    "\n",
    "sfs = SFS(lr, \n",
    "          k_features=4, \n",
    "          forward=True, \n",
    "          scoring='accuracy',\n",
    "          cv=5)\n",
    "\n",
    "sfs.fit(X_train, y_train)\n",
    "sfs.subsets_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that both RFECV and SFS are performing cross-validation internally to estimate the performance during the selection. We haven't talked about cross-validation, yet, but will do so in section 5. So, for now, only the \"avg_score\" is of interest. In this case, the results indicate the using more features improves the performance. \n",
    "\n",
    "In addition, we can use a helper utility to visualize the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.plotting import plot_sequential_feature_selection as plot_sfs\n",
    "\n",
    "\n",
    "fig1 = plot_sfs(sfs.get_metric_dict(), kind='std_err')\n",
    "\n",
    "plt.ylim([0.8, 1])\n",
    "plt.title('Sequential Forward Selection (w. StdErr)')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A good rule of thumb for selecting feature subsets is to select the smallest feature subsets that is within 1 standard error of the best performing one, which would be the feature subset with `'feature_idx': (2, 3)`, i.e., petal length and petal width."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Principal Component Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Principal component analysis is a technique that compresses data onto a smaller-dimensional subspace while retaining most of its variance or spread (aka information). We won't go into the mathematical details of PCA here,  but I have written a post about it if you are interested: https://sebastianraschka.com/Articles/2015_pca_in_3_steps.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we specify n_components=4 and run this algorithm on our 4-feature iris dataset, it actually wouldn't reduce the dimensionality. However, this is still a useful approach since we can look at the variance-explained ratios of the individual components, which give us a clue in terms of how many components we should keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "\n",
    "pca = PCA(n_components=4)\n",
    "\n",
    "pca.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_exp = pca.explained_variance_ratio_\n",
    "cum_var_exp = np.cumsum(var_exp)\n",
    "\n",
    "idx = [i for i in range(len(var_exp))]\n",
    "labels = [str(i + 1) for i in idx]\n",
    "\n",
    "plt.bar(range(4),\n",
    "        var_exp,\n",
    "        alpha=0.5,\n",
    "        align='center',\n",
    "        label='individual explained variance')\n",
    "\n",
    "plt.step(range(4),\n",
    "         cum_var_exp,\n",
    "         where='mid', \n",
    "         label='cumulative explained variance')\n",
    "\n",
    "plt.ylabel('Explained variance ratio')\n",
    "plt.xlabel('Principal components')\n",
    "plt.xticks(idx, labels)\n",
    "plt.legend(loc='center right')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cumulative distribution plot above indicates that approx. 95% of the variance is explained by the first two components. Thus, a dimensionality reduction down to 2 dimensions would make sense in this case. In the following plot, the transformed iris data is visualized on the two new component axes. Note that PCA is a unsupervised technique and the class labels are only shown for illustrative purposes (or in other words, PCA does not use class labels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_pca = pca.transform(X_train)\n",
    "\n",
    "for lab, col, mar in zip((0, 1, 2),\n",
    "                         ('blue', 'red', 'green'),\n",
    "                         ('o', 's', '^')):\n",
    "    plt.scatter(X_train_pca[y_train == lab, 0],\n",
    "                X_train_pca[y_train == lab, 1],\n",
    "                label=lab,\n",
    "                marker=mar,\n",
    "                c=col)\n",
    "plt.xlabel('Principal Component 1')\n",
    "plt.ylabel('Principal Component 2')\n",
    "plt.legend(loc='lower right')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='height:100px;'></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 - Model Evaluation & Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.data import wine_data\n",
    "\n",
    "X, y = wine_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wine dataset.\n",
    "\n",
    "Source : https://archive.ics.uci.edu/ml/datasets/Wine\n",
    "\n",
    "Number of samples : 178\n",
    "\n",
    "Class labels : {0, 1, 2}, distribution: [59, 71, 48]\n",
    "\n",
    "Dataset Attributes:\n",
    "\n",
    "1. Alcohol\n",
    "2. Malic acid\n",
    "3. Ash\n",
    "4. Alcalinity of ash\n",
    "5. Magnesium\n",
    "6. Total phenols\n",
    "7. Flavanoids\n",
    "8. Nonflavanoid phenols\n",
    "9. Proanthocyanins\n",
    "10. Color intensity\n",
    "11. Hue\n",
    "12. OD280/OD315 of diluted wines\n",
    "13. Proline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stratified K-Fold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, random_state=123, stratify=y)\n",
    "\n",
    "pipe_kn = make_pipeline(StandardScaler(), \n",
    "                        PCA(n_components=1),\n",
    "                        KNN(n_neighbors=3))\n",
    "\n",
    "kfold = StratifiedKFold(y=y_train, \n",
    "                        n_folds=10,\n",
    "                        random_state=1)\n",
    "\n",
    "scores = []\n",
    "for k, (train, test) in enumerate(kfold):\n",
    "    pipe_kn.fit(X_train[train], y_train[train])\n",
    "    score = pipe_kn.score(X_train[test], y_train[test])\n",
    "    scores.append(score)\n",
    "    print('Fold: %s, Class dist.: %s, Acc: %.3f' % (k+1,\n",
    "          np.bincount(y_train[train]), score))\n",
    "    \n",
    "print('\\nCV accuracy: %.3f +/- %.3f' % (np.mean(scores), np.std(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "scores = cross_val_score(estimator=pipe_kn,\n",
    "                         X=X_train,\n",
    "                         y=y_train,\n",
    "                         cv=10,\n",
    "                         n_jobs=2)\n",
    "\n",
    "print('CV accuracy scores: %s' % scores)\n",
    "print('CV accuracy: %.3f +/- %.3f' % (np.mean(scores), np.std(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_kn.named_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "param_grid = {'pca__n_components': [1, 2, 3, 4, 5, 6, None],\n",
    "              'kneighborsclassifier__n_neighbors': [1, 3, 5, 7, 9, 11]}\n",
    "\n",
    "gs = GridSearchCV(estimator=pipe_kn, \n",
    "                  param_grid=param_grid, \n",
    "                  scoring='accuracy', \n",
    "                  cv=10,\n",
    "                  n_jobs=2,\n",
    "                  refit=True)\n",
    "gs = gs.fit(X_train, y_train)\n",
    "print(gs.best_score_)\n",
    "print(gs.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs.score(X_test, y_test)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}