t81_558_class_08_2_keras_ensembles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_2_keras_ensembles.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 8: Kaggle Data Sets**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=v4lJBhdCuCU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)\n",
    "* **Part 8.2: Building Ensembles with Scikit-Learn and Keras** [[Video]](https://www.youtube.com/watch?v=LQ-9ZRBLasw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_keras_ensembles.ipynb)\n",
    "* Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters [[Video]](https://www.youtube.com/watch?v=1q9klwSoUQw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_3_keras_hyperparameters.ipynb)\n",
    "* Part 8.4: Bayesian Hyperparameter Optimization for Keras [[Video]](https://www.youtube.com/watch?v=sXdxyUCCm8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]](https://www.youtube.com/watch?v=PHQt0aUasRg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow.\n",
    "  Running the following code will map your GDrive to ```/content/drive```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 8.2: Building Ensembles with Scikit-Learn and Keras\n",
    "\n",
    "### Evaluating Feature Importance\n",
    "\n",
    "Feature importance tells us how important each of the features (from the feature/import vector is to the prediction of a neural network or another model.  There are many different ways to evaluate the feature importance for neural networks.  The following paper presents an excellent (and readable) overview of the various means of assessing the significance of neural network inputs/features.\n",
    "\n",
    "* An accurate comparison of methods for quantifying variable importance in artificial neural networks using simulated data [[Cite:olden2004accurate]](http://depts.washington.edu/oldenlab/wordpress/wp-content/uploads/2013/03/EcologicalModelling_2004.pdf). *Ecological Modelling*, 178(3), 389-397.\n",
    "\n",
    "In summary, the following methods are available to neural networks:\n",
    "\n",
    "* Connection Weights Algorithm\n",
    "* Partial Derivatives\n",
    "* Input Perturbation\n",
    "* Sensitivity Analysis\n",
    "* Forward Stepwise Addition \n",
    "* Improved Stepwise Selection 1\n",
    "* Backward Stepwise Elimination\n",
    "* Improved Stepwise Selection\n",
    "\n",
    "For this class, we will use the **Input Perturbation** feature ranking algorithm.  This algorithm will work with any regression or classification network. I provide an implementation of the input perturbation algorithm for scikit-learn in the next section. This code implements a function below that will work with any scikit-learn model.\n",
    "\n",
    "[Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman) provided this algorithm in his seminal paper on random forests. [[Citebreiman2001random:]](https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf)  Although he presented this algorithm in conjunction with random forests, it is model-independent and appropriate for any supervised learning model.  This algorithm, known as the input perturbation algorithm, works by evaluating a trained model’s accuracy with each of the inputs individually shuffled from a data set.  Shuffling an input causes it to become useless—effectively removing it from the model. More important inputs will produce a less accurate score when they are removed by shuffling them. This process makes sense because important features will contribute to the accuracy of the model.  I first presented the TensorFlow implementation of this algorithm in the following paper.\n",
    "\n",
    "* Early stabilizing feature importance for TensorFlow deep neural networks[[Cite:heaton2017early]](https://www.heatonresearch.com/dload/phd/IJCNN%202017-v2-final.pdf)\n",
    "\n",
    "This algorithm will use log loss to evaluate a classification problem and RMSE for regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "import scipy as sp\n",
    "import numpy as np\n",
    "import math\n",
    "from sklearn import metrics\n",
    "\n",
    "def perturbation_rank(model, x, y, names, regression):\n",
    "    errors = []\n",
    "\n",
    "    for i in range(x.shape[1]):\n",
    "        hold = np.array(x[:, i])\n",
    "        np.random.shuffle(x[:, i])\n",
    "        \n",
    "        if regression:\n",
    "            pred = model.predict(x)\n",
    "            error = metrics.mean_squared_error(y, pred)\n",
    "        else:\n",
    "            pred = model.predict_proba(x)\n",
    "            error = metrics.log_loss(y, pred)\n",
    "            \n",
    "        errors.append(error)\n",
    "        x[:, i] = hold\n",
    "        \n",
    "    max_error = np.max(errors)\n",
    "    importance = [e/max_error for e in errors]\n",
    "\n",
    "    data = {'name':names,'error':errors,'importance':importance}\n",
    "    result = pd.DataFrame(data, columns = ['name','error','importance'])\n",
    "    result.sort_values(by=['importance'], ascending=[0], inplace=True)\n",
    "    result.reset_index(inplace=True, drop=True)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification and Input Perturbation Ranking\n",
    "\n",
    "We now look at the code to perform perturbation ranking for a classification neural network.  The implementation technique is slightly different for classification vs regression, so I must provide two different implementations.  The primary difference between classification and regression is how we evaluate the accuracy of the neural network in each of these two network types.  For regression neural networks, we will use the Root Mean Square (RMSE) error calculation; whereas, we will use log loss for classification.\n",
    "\n",
    "The code presented below creates a classification neural network that will predict for the classic iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 112 samples\n",
      "Epoch 1/100\n",
      "112/112 - 0s - loss: 1.1435\n",
      "Epoch 2/100\n",
      "112/112 - 0s - loss: 1.0499\n",
      "Epoch 3/100\n",
      "112/112 - 0s - loss: 0.9736\n",
      "Epoch 4/100\n",
      "112/112 - 0s - loss: 0.9220\n",
      "Epoch 5/100\n",
      "112/112 - 0s - loss: 0.8958\n",
      "Epoch 6/100\n",
      "112/112 - 0s - loss: 0.8628\n",
      "Epoch 7/100\n",
      "112/112 - 0s - loss: 0.8333\n",
      "Epoch 8/100\n",
      "112/112 - 0s - loss: 0.8039\n",
      "Epoch 9/100\n",
      "112/112 - 0s - loss: 0.7735\n",
      "Epoch 10/100\n",
      "112/112 - 0s - loss: 0.7453\n",
      "Epoch 11/100\n",
      "112/112 - 0s - loss: 0.7221\n",
      "Epoch 12/100\n",
      "112/112 - 0s - loss: 0.7009\n",
      "Epoch 13/100\n",
      "112/112 - 0s - loss: 0.6795\n",
      "Epoch 14/100\n",
      "112/112 - 0s - loss: 0.6627\n",
      "Epoch 15/100\n",
      "112/112 - 0s - loss: 0.6416\n",
      "Epoch 16/100\n",
      "112/112 - 0s - loss: 0.6239\n",
      "Epoch 17/100\n",
      "112/112 - 0s - loss: 0.6096\n",
      "Epoch 18/100\n",
      "112/112 - 0s - loss: 0.5938\n",
      "Epoch 19/100\n",
      "112/112 - 0s - loss: 0.5772\n",
      "Epoch 20/100\n",
      "112/112 - 0s - loss: 0.5671\n",
      "Epoch 21/100\n",
      "112/112 - 0s - loss: 0.5529\n",
      "Epoch 22/100\n",
      "112/112 - 0s - loss: 0.5442\n",
      "Epoch 23/100\n",
      "112/112 - 0s - loss: 0.5314\n",
      "Epoch 24/100\n",
      "112/112 - 0s - loss: 0.5210\n",
      "Epoch 25/100\n",
      "112/112 - 0s - loss: 0.5104\n",
      "Epoch 26/100\n",
      "112/112 - 0s - loss: 0.5009\n",
      "Epoch 27/100\n",
      "112/112 - 0s - loss: 0.4912\n",
      "Epoch 28/100\n",
      "112/112 - 0s - loss: 0.4780\n",
      "Epoch 29/100\n",
      "112/112 - 0s - loss: 0.4707\n",
      "Epoch 30/100\n",
      "112/112 - 0s - loss: 0.4515\n",
      "Epoch 31/100\n",
      "112/112 - 0s - loss: 0.4405\n",
      "Epoch 32/100\n",
      "112/112 - 0s - loss: 0.4297\n",
      "Epoch 33/100\n",
      "112/112 - 0s - loss: 0.4191\n",
      "Epoch 34/100\n",
      "112/112 - 0s - loss: 0.4111\n",
      "Epoch 35/100\n",
      "112/112 - 0s - loss: 0.4015\n",
      "Epoch 36/100\n",
      "112/112 - 0s - loss: 0.3927\n",
      "Epoch 37/100\n",
      "112/112 - 0s - loss: 0.3848\n",
      "Epoch 38/100\n",
      "112/112 - 0s - loss: 0.3739\n",
      "Epoch 39/100\n",
      "112/112 - 0s - loss: 0.3653\n",
      "Epoch 40/100\n",
      "112/112 - 0s - loss: 0.3573\n",
      "Epoch 41/100\n",
      "112/112 - 0s - loss: 0.3484\n",
      "Epoch 42/100\n",
      "112/112 - 0s - loss: 0.3407\n",
      "Epoch 43/100\n",
      "112/112 - 0s - loss: 0.3332\n",
      "Epoch 44/100\n",
      "112/112 - 0s - loss: 0.3265\n",
      "Epoch 45/100\n",
      "112/112 - 0s - loss: 0.3159\n",
      "Epoch 46/100\n",
      "112/112 - 0s - loss: 0.3089\n",
      "Epoch 47/100\n",
      "112/112 - 0s - loss: 0.3010\n",
      "Epoch 48/100\n",
      "112/112 - 0s - loss: 0.2926\n",
      "Epoch 49/100\n",
      "112/112 - 0s - loss: 0.2856\n",
      "Epoch 50/100\n",
      "112/112 - 0s - loss: 0.2824\n",
      "Epoch 51/100\n",
      "112/112 - 0s - loss: 0.2709\n",
      "Epoch 52/100\n",
      "112/112 - 0s - loss: 0.2661\n",
      "Epoch 53/100\n",
      "112/112 - 0s - loss: 0.2609\n",
      "Epoch 54/100\n",
      "112/112 - 0s - loss: 0.2519\n",
      "Epoch 55/100\n",
      "112/112 - 0s - loss: 0.2458\n",
      "Epoch 56/100\n",
      "112/112 - 0s - loss: 0.2443\n",
      "Epoch 57/100\n",
      "112/112 - 0s - loss: 0.2342\n",
      "Epoch 58/100\n",
      "112/112 - 0s - loss: 0.2313\n",
      "Epoch 59/100\n",
      "112/112 - 0s - loss: 0.2243\n",
      "Epoch 60/100\n",
      "112/112 - 0s - loss: 0.2211\n",
      "Epoch 61/100\n",
      "112/112 - 0s - loss: 0.2137\n",
      "Epoch 62/100\n",
      "112/112 - 0s - loss: 0.2083\n",
      "Epoch 63/100\n",
      "112/112 - 0s - loss: 0.2094\n",
      "Epoch 64/100\n",
      "112/112 - 0s - loss: 0.2042\n",
      "Epoch 65/100\n",
      "112/112 - 0s - loss: 0.1972\n",
      "Epoch 66/100\n",
      "112/112 - 0s - loss: 0.1936\n",
      "Epoch 67/100\n",
      "112/112 - 0s - loss: 0.1874\n",
      "Epoch 68/100\n",
      "112/112 - 0s - loss: 0.1829\n",
      "Epoch 69/100\n",
      "112/112 - 0s - loss: 0.1810\n",
      "Epoch 70/100\n",
      "112/112 - 0s - loss: 0.1769\n",
      "Epoch 71/100\n",
      "112/112 - 0s - loss: 0.1737\n",
      "Epoch 72/100\n",
      "112/112 - 0s - loss: 0.1704\n",
      "Epoch 73/100\n",
      "112/112 - 0s - loss: 0.1671\n",
      "Epoch 74/100\n",
      "112/112 - 0s - loss: 0.1634\n",
      "Epoch 75/100\n",
      "112/112 - 0s - loss: 0.1622\n",
      "Epoch 76/100\n",
      "112/112 - 0s - loss: 0.1568\n",
      "Epoch 77/100\n",
      "112/112 - 0s - loss: 0.1563\n",
      "Epoch 78/100\n",
      "112/112 - 0s - loss: 0.1534\n",
      "Epoch 79/100\n",
      "112/112 - 0s - loss: 0.1506\n",
      "Epoch 80/100\n",
      "112/112 - 0s - loss: 0.1487\n",
      "Epoch 81/100\n",
      "112/112 - 0s - loss: 0.1455\n",
      "Epoch 82/100\n",
      "112/112 - 0s - loss: 0.1435\n",
      "Epoch 83/100\n",
      "112/112 - 0s - loss: 0.1411\n",
      "Epoch 84/100\n",
      "112/112 - 0s - loss: 0.1384\n",
      "Epoch 85/100\n",
      "112/112 - 0s - loss: 0.1374\n",
      "Epoch 86/100\n",
      "112/112 - 0s - loss: 0.1351\n",
      "Epoch 87/100\n",
      "112/112 - 0s - loss: 0.1381\n",
      "Epoch 88/100\n",
      "112/112 - 0s - loss: 0.1326\n",
      "Epoch 89/100\n",
      "112/112 - 0s - loss: 0.1333\n",
      "Epoch 90/100\n",
      "112/112 - 0s - loss: 0.1304\n",
      "Epoch 91/100\n",
      "112/112 - 0s - loss: 0.1274\n",
      "Epoch 92/100\n",
      "112/112 - 0s - loss: 0.1248\n",
      "Epoch 93/100\n",
      "112/112 - 0s - loss: 0.1310\n",
      "Epoch 94/100\n",
      "112/112 - 0s - loss: 0.1227\n",
      "Epoch 95/100\n",
      "112/112 - 0s - loss: 0.1234\n",
      "Epoch 96/100\n",
      "112/112 - 0s - loss: 0.1220\n",
      "Epoch 97/100\n",
      "112/112 - 0s - loss: 0.1180\n",
      "Epoch 98/100\n",
      "112/112 - 0s - loss: 0.1176\n",
      "Epoch 99/100\n",
      "112/112 - 0s - loss: 0.1187\n",
      "Epoch 100/100\n",
      "112/112 - 0s - loss: 0.1148\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x249cb0fba08>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x_train,y_train,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we evaluate the accuracy of the trained model.  Here we see that the neural network is performing great, with an accuracy of 1.0.  For a more complex dataset, we might fear overfitting with such high accuracy.  However, for this example, we are more interested in determining the importance of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "pred = model.predict(x_test)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y_test,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to call the input perturbation algorithm.  First, we extract the column names and remove the target column.  The target column does not have importance, as it is the objective, not one of the inputs.  In supervised learning, the target is of the utmost importance.\n",
    "\n",
    "We can see the importance displayed in the following table.  The most important column is always 1.0, and lessor columns will continue in a downward trend.  The least important column will have the lowest rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>petal_l</td>\n",
       "      <td>2.457038</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>petal_w</td>\n",
       "      <td>0.564843</td>\n",
       "      <td>0.229888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sepal_l</td>\n",
       "      <td>0.206589</td>\n",
       "      <td>0.084081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sepal_w</td>\n",
       "      <td>0.186969</td>\n",
       "      <td>0.076095</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      name     error  importance\n",
       "0  petal_l  2.457038    1.000000\n",
       "1  petal_w  0.564843    0.229888\n",
       "2  sepal_l  0.206589    0.084081\n",
       "3  sepal_w  0.186969    0.076095"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df.columns) # x+y column names\n",
    "names.remove(\"species\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, False)\n",
    "display(rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression and Input Perturbation Ranking\n",
    "\n",
    "We now see how to use input perturbation ranking for a regression neural network.  We will use the MPG dataset as a demonstration.  The code below loads the MPG dataset and creates a regression neural network for this dataset.  The code trains the neural network and calculates an RMSE evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 298 samples\n",
      "Epoch 1/100\n",
      "298/298 - 0s - loss: 40307.5655\n",
      "Epoch 2/100\n",
      "298/298 - 0s - loss: 12333.8158\n",
      "Epoch 3/100\n",
      "298/298 - 0s - loss: 1765.7875\n",
      "Epoch 4/100\n",
      "298/298 - 0s - loss: 1373.6327\n",
      "Epoch 5/100\n",
      "298/298 - 0s - loss: 726.2777\n",
      "Epoch 6/100\n",
      "298/298 - 0s - loss: 245.1450\n",
      "Epoch 7/100\n",
      "298/298 - 0s - loss: 135.2508\n",
      "Epoch 8/100\n",
      "298/298 - 0s - loss: 125.4094\n",
      "Epoch 9/100\n",
      "298/298 - 0s - loss: 112.5616\n",
      "Epoch 10/100\n",
      "298/298 - 0s - loss: 98.0328\n",
      "Epoch 11/100\n",
      "298/298 - 0s - loss: 96.0137\n",
      "Epoch 12/100\n",
      "298/298 - 0s - loss: 99.4858\n",
      "Epoch 13/100\n",
      "298/298 - 0s - loss: 95.2996\n",
      "Epoch 14/100\n",
      "298/298 - 0s - loss: 93.6376\n",
      "Epoch 15/100\n",
      "298/298 - 0s - loss: 91.2554\n",
      "Epoch 16/100\n",
      "298/298 - 0s - loss: 89.7455\n",
      "Epoch 17/100\n",
      "298/298 - 0s - loss: 90.1669\n",
      "Epoch 18/100\n",
      "298/298 - 0s - loss: 87.6308\n",
      "Epoch 19/100\n",
      "298/298 - 0s - loss: 86.2253\n",
      "Epoch 20/100\n",
      "298/298 - 0s - loss: 88.2544\n",
      "Epoch 21/100\n",
      "298/298 - 0s - loss: 84.9131\n",
      "Epoch 22/100\n",
      "298/298 - 0s - loss: 82.8037\n",
      "Epoch 23/100\n",
      "298/298 - 0s - loss: 83.0567\n",
      "Epoch 24/100\n",
      "298/298 - 0s - loss: 80.5955\n",
      "Epoch 25/100\n",
      "298/298 - 0s - loss: 81.1876\n",
      "Epoch 26/100\n",
      "298/298 - 0s - loss: 81.5474\n",
      "Epoch 27/100\n",
      "298/298 - 0s - loss: 82.0906\n",
      "Epoch 28/100\n",
      "298/298 - 0s - loss: 78.8021\n",
      "Epoch 29/100\n",
      "298/298 - 0s - loss: 75.8490\n",
      "Epoch 30/100\n",
      "298/298 - 0s - loss: 77.6923\n",
      "Epoch 31/100\n",
      "298/298 - 0s - loss: 78.4401\n",
      "Epoch 32/100\n",
      "298/298 - 0s - loss: 76.0245\n",
      "Epoch 33/100\n",
      "298/298 - 0s - loss: 73.3758\n",
      "Epoch 34/100\n",
      "298/298 - 0s - loss: 71.2994\n",
      "Epoch 35/100\n",
      "298/298 - 0s - loss: 71.8301\n",
      "Epoch 36/100\n",
      "298/298 - 0s - loss: 71.0497\n",
      "Epoch 37/100\n",
      "298/298 - 0s - loss: 67.7307\n",
      "Epoch 38/100\n",
      "298/298 - 0s - loss: 66.5521\n",
      "Epoch 39/100\n",
      "298/298 - 0s - loss: 67.5412\n",
      "Epoch 40/100\n",
      "298/298 - 0s - loss: 65.9181\n",
      "Epoch 41/100\n",
      "298/298 - 0s - loss: 67.0536\n",
      "Epoch 42/100\n",
      "298/298 - 0s - loss: 72.0487\n",
      "Epoch 43/100\n",
      "298/298 - 0s - loss: 64.2251\n",
      "Epoch 44/100\n",
      "298/298 - 0s - loss: 65.1870\n",
      "Epoch 45/100\n",
      "298/298 - 0s - loss: 63.3230\n",
      "Epoch 46/100\n",
      "298/298 - 0s - loss: 60.1012\n",
      "Epoch 47/100\n",
      "298/298 - 0s - loss: 59.9516\n",
      "Epoch 48/100\n",
      "298/298 - 0s - loss: 59.9162\n",
      "Epoch 49/100\n",
      "298/298 - 0s - loss: 58.6367\n",
      "Epoch 50/100\n",
      "298/298 - 0s - loss: 59.2871\n",
      "Epoch 51/100\n",
      "298/298 - 0s - loss: 59.2135\n",
      "Epoch 52/100\n",
      "298/298 - 0s - loss: 56.7722\n",
      "Epoch 53/100\n",
      "298/298 - 0s - loss: 56.7388\n",
      "Epoch 54/100\n",
      "298/298 - 0s - loss: 56.2866\n",
      "Epoch 55/100\n",
      "298/298 - 0s - loss: 55.9870\n",
      "Epoch 56/100\n",
      "298/298 - 0s - loss: 53.7034\n",
      "Epoch 57/100\n",
      "298/298 - 0s - loss: 53.8814\n",
      "Epoch 58/100\n",
      "298/298 - 0s - loss: 55.2509\n",
      "Epoch 59/100\n",
      "298/298 - 0s - loss: 57.0467\n",
      "Epoch 60/100\n",
      "298/298 - 0s - loss: 52.1042\n",
      "Epoch 61/100\n",
      "298/298 - 0s - loss: 50.5720\n",
      "Epoch 62/100\n",
      "298/298 - 0s - loss: 50.7421\n",
      "Epoch 63/100\n",
      "298/298 - 0s - loss: 49.3410\n",
      "Epoch 64/100\n",
      "298/298 - 0s - loss: 49.2148\n",
      "Epoch 65/100\n",
      "298/298 - 0s - loss: 48.1832\n",
      "Epoch 66/100\n",
      "298/298 - 0s - loss: 48.0912\n",
      "Epoch 67/100\n",
      "298/298 - 0s - loss: 47.7567\n",
      "Epoch 68/100\n",
      "298/298 - 0s - loss: 49.4456\n",
      "Epoch 69/100\n",
      "298/298 - 0s - loss: 46.2840\n",
      "Epoch 70/100\n",
      "298/298 - 0s - loss: 47.5307\n",
      "Epoch 71/100\n",
      "298/298 - 0s - loss: 45.8420\n",
      "Epoch 72/100\n",
      "298/298 - 0s - loss: 45.2659\n",
      "Epoch 73/100\n",
      "298/298 - 0s - loss: 45.1073\n",
      "Epoch 74/100\n",
      "298/298 - 0s - loss: 48.3373\n",
      "Epoch 75/100\n",
      "298/298 - 0s - loss: 45.0688\n",
      "Epoch 76/100\n",
      "298/298 - 0s - loss: 43.6362\n",
      "Epoch 77/100\n",
      "298/298 - 0s - loss: 46.0108\n",
      "Epoch 78/100\n",
      "298/298 - 0s - loss: 41.8529\n",
      "Epoch 79/100\n",
      "298/298 - 0s - loss: 41.2489\n",
      "Epoch 80/100\n",
      "298/298 - 0s - loss: 41.6264\n",
      "Epoch 81/100\n",
      "298/298 - 0s - loss: 41.0539\n",
      "Epoch 82/100\n",
      "298/298 - 0s - loss: 39.5903\n",
      "Epoch 83/100\n",
      "298/298 - 0s - loss: 39.6608\n",
      "Epoch 84/100\n",
      "298/298 - 0s - loss: 40.8665\n",
      "Epoch 85/100\n",
      "298/298 - 0s - loss: 40.0233\n",
      "Epoch 86/100\n",
      "298/298 - 0s - loss: 38.3439\n",
      "Epoch 87/100\n",
      "298/298 - 0s - loss: 36.8931\n",
      "Epoch 88/100\n",
      "298/298 - 0s - loss: 39.9279\n",
      "Epoch 89/100\n",
      "298/298 - 0s - loss: 36.5694\n",
      "Epoch 90/100\n",
      "298/298 - 0s - loss: 36.7673\n",
      "Epoch 91/100\n",
      "298/298 - 0s - loss: 35.6763\n",
      "Epoch 92/100\n",
      "298/298 - 0s - loss: 35.3738\n",
      "Epoch 93/100\n",
      "298/298 - 0s - loss: 36.4481\n",
      "Epoch 94/100\n",
      "298/298 - 0s - loss: 36.7314\n",
      "Epoch 95/100\n",
      "298/298 - 0s - loss: 36.0925\n",
      "Epoch 96/100\n",
      "298/298 - 0s - loss: 34.3974\n",
      "Epoch 97/100\n",
      "298/298 - 0s - loss: 38.1142\n",
      "Epoch 98/100\n",
      "298/298 - 0s - loss: 37.1556\n",
      "Epoch 99/100\n",
      "298/298 - 0s - loss: 35.1785\n",
      "Epoch 100/100\n",
      "298/298 - 0s - loss: 33.2472\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "save_path = \".\"\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x_train,y_train,verbose=2,epochs=100)\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as before, we extract the column names and discard the target.  We can now create a ranking of the importance of each of the input features.  The feature with a ranking of 1.0 is the most important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>displacement</td>\n",
       "      <td>287.936080</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>weight</td>\n",
       "      <td>198.281957</td>\n",
       "      <td>0.688632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>horsepower</td>\n",
       "      <td>32.233535</td>\n",
       "      <td>0.111947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>year</td>\n",
       "      <td>30.184242</td>\n",
       "      <td>0.104830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cylinders</td>\n",
       "      <td>26.632769</td>\n",
       "      <td>0.092495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>origin</td>\n",
       "      <td>26.346470</td>\n",
       "      <td>0.091501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>acceleration</td>\n",
       "      <td>26.298675</td>\n",
       "      <td>0.091335</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           name       error  importance\n",
       "0  displacement  287.936080    1.000000\n",
       "1        weight  198.281957    0.688632\n",
       "2    horsepower   32.233535    0.111947\n",
       "3          year   30.184242    0.104830\n",
       "4     cylinders   26.632769    0.092495\n",
       "5        origin   26.346470    0.091501\n",
       "6  acceleration   26.298675    0.091335"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df.columns) # x+y column names\n",
    "names.remove(\"name\")\n",
    "names.remove(\"mpg\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, True)\n",
    "display(rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Biological Response with Neural Network\n",
    "\n",
    "The following sections will demonstrate how to use feature importance ranking and ensembling with a more complex dataset.  Ensembling is the process where you combine multiple models for greater accuracy. Kaggle competition winners frequently make use of ensembling for high ranking solutions.\n",
    "\n",
    "We will use the biological response dataset, a Kaggle dataset, where there is an unusually high number of columns.  Because of the large number of columns, it is essential to use feature ranking to determine the importance of these columns.  We begin by loading the dataset and preprocessing.  This Kaggle dataset is a binary classification problem. You must predict if certain conditions will cause a biological response.\n",
    "\n",
    "* [Predicting a Biological Response](https://www.kaggle.com/c/bioresponse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import KFold\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "if COLAB:\n",
    "    path = \"/content/drive/My Drive/data/\"\n",
    "else:\n",
    "    path = \"./data/\"\n",
    "    \n",
    "filename_train = os.path.join(path,\"bio_train.csv\")\n",
    "filename_test = os.path.join(path,\"bio_test.csv\")\n",
    "filename_submit = os.path.join(path,\"bio_submit.csv\")\n",
    "\n",
    "df_train = pd.read_csv(filename_train,na_values=['NA','?'])\n",
    "df_test = pd.read_csv(filename_test,na_values=['NA','?'])\n",
    "\n",
    "activity_classes = df_train['Activity']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A large number of columns is evident when we display the shape of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3751, 1777)\n"
     ]
    }
   ],
   "source": [
    "print(df_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code constructs a classification neural network and trains it for the biological response dataset.  Once trained, the accuracy is measured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting/Training...\n",
      "Epoch 00008: early stopping\n",
      "Fitting done...\n",
      "Validation logloss: 0.5671799306256637\n",
      "Validation accuracy score: 0.7643923240938166\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import numpy as np\n",
    "import sklearn\n",
    "\n",
    "# Encode feature vector\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df_train.columns.drop('Activity')\n",
    "x = df_train[x_columns].values\n",
    "y = df_train['Activity'].values # Classification\n",
    "x_submit = df_test[x_columns].values.astype(np.float32)\n",
    "\n",
    "\n",
    "# Split into train/test\n",
    "x_train, x_test, y_train, y_test = train_test_split(    \n",
    "    x, y, test_size=0.25, random_state=42) \n",
    "\n",
    "print(\"Fitting/Training...\")\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu'))\n",
    "model.add(Dense(10))\n",
    "model.add(Dense(1,activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam')\n",
    "monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "                        patience=5, verbose=1, mode='auto')\n",
    "model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "          callbacks=[monitor],verbose=0,epochs=1000)\n",
    "print(\"Fitting done...\")\n",
    "\n",
    "# Predict\n",
    "pred = model.predict(x_test).flatten()\n",
    "\n",
    "\n",
    "# Clip so that min is never exactly 0, max never 1\n",
    "pred = np.clip(pred,a_min=1e-6,a_max=(1-1e-6)) \n",
    "print(\"Validation logloss: {}\".format(\n",
    "    sklearn.metrics.log_loss(y_test,pred)))\n",
    "\n",
    "# Evaluate success using accuracy\n",
    "pred = pred>0.5 # If greater than 0.5 probability, then true\n",
    "score = metrics.accuracy_score(y_test, pred)\n",
    "print(\"Validation accuracy score: {}\".format(score))\n",
    "\n",
    "# Build real submit file\n",
    "pred_submit = model.predict(x_submit)\n",
    "\n",
    "# Clip so that min is never exactly 0, max never 1 (would be a NaN score)\n",
    "pred = np.clip(pred,a_min=1e-6,a_max=(1-1e-6)) \n",
    "submit_df = pd.DataFrame({'MoleculeId':[x+1 for x \\\n",
    "        in range(len(pred_submit))],'PredictedProbability':\\\n",
    "                          pred_submit.flatten()})\n",
    "submit_df.to_csv(filename_submit, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What Features/Columns are Important\n",
    "The following uses perturbation ranking to evaluate the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>error</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>D27</td>\n",
       "      <td>0.637687</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>D1049</td>\n",
       "      <td>0.576325</td>\n",
       "      <td>0.903773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>D1071</td>\n",
       "      <td>0.574874</td>\n",
       "      <td>0.901498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>D961</td>\n",
       "      <td>0.574213</td>\n",
       "      <td>0.900462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>D1407</td>\n",
       "      <td>0.573172</td>\n",
       "      <td>0.898829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>D1333</td>\n",
       "      <td>0.572745</td>\n",
       "      <td>0.898159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>D958</td>\n",
       "      <td>0.572173</td>\n",
       "      <td>0.897263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>D1271</td>\n",
       "      <td>0.571364</td>\n",
       "      <td>0.895994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>D1167</td>\n",
       "      <td>0.571352</td>\n",
       "      <td>0.895975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>D51</td>\n",
       "      <td>0.571237</td>\n",
       "      <td>0.895795</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name     error  importance\n",
       "0    D27  0.637687    1.000000\n",
       "1  D1049  0.576325    0.903773\n",
       "2  D1071  0.574874    0.901498\n",
       "3   D961  0.574213    0.900462\n",
       "4  D1407  0.573172    0.898829\n",
       "5  D1333  0.572745    0.898159\n",
       "6   D958  0.572173    0.897263\n",
       "7  D1271  0.571364    0.895994\n",
       "8  D1167  0.571352    0.895975\n",
       "9    D51  0.571237    0.895795"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Rank the features\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "names = list(df_train.columns) # x+y column names\n",
    "names.remove(\"Activity\") # remove the target(y)\n",
    "rank = perturbation_rank(model, x_test, y_test, names, False)\n",
    "display(rank[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network Ensemble\n",
    "\n",
    "A neural network ensemble combines neural network predictions with other models. The program determines the exact blend of all of these models by logistic regression. The following code performs this blend for a classification.  If you present the final predictions from the ensemble to Kaggle, you will see that the result is very accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "Model: 0 : <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object at 0x0000024F619B5E48>\n",
      "Train on 3375 samples\n",
      "3375/3375 [==============================] - 0s 110us/sample - loss: 0.6152\n",
      "Fold #0: loss=0.5670192397249542\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 110us/sample - loss: 0.5876\n",
      "Fold #1: loss=0.5181313306906605\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 108us/sample - loss: 0.5864\n",
      "Fold #2: loss=0.5357788761797414\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 110us/sample - loss: 0.6031\n",
      "Fold #3: loss=0.5086871196665781\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 109us/sample - loss: 0.5959\n",
      "Fold #4: loss=0.5346533003082558\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 112us/sample - loss: 0.6080\n",
      "Fold #5: loss=0.5826812768114207\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 109us/sample - loss: 0.6062\n",
      "Fold #6: loss=0.5433827307490822\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 1s 150us/sample - loss: 0.6046\n",
      "Fold #7: loss=0.5441332130779082\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 111us/sample - loss: 0.6014\n",
      "Fold #8: loss=0.5631595507242343\n",
      "Train on 3376 samples\n",
      "3376/3376 [==============================] - 0s 109us/sample - loss: 0.5859\n",
      "Fold #9: loss=0.5179122002838316\n",
      "KerasClassifier: Mean loss=0.5415538838216667\n",
      "Model: 1 : KNeighborsClassifier(n_neighbors=3)\n",
      "Fold #0: loss=3.606678388314123\n",
      "Fold #1: loss=2.2256421551487593\n",
      "Fold #2: loss=3.6815437059542186\n",
      "Fold #3: loss=2.416161292225968\n",
      "Fold #4: loss=4.442472310149748\n",
      "Fold #5: loss=4.321350530738247\n",
      "Fold #6: loss=3.400455469543658\n",
      "Fold #7: loss=3.1724147110842513\n",
      "Fold #8: loss=2.117356283193681\n",
      "Fold #9: loss=3.0532135963322586\n",
      "KNeighborsClassifier: Mean loss=3.243728844268491\n",
      "Model: 2 : RandomForestClassifier(n_jobs=-1)\n",
      "Fold #0: loss=0.4657177982691548\n",
      "Fold #1: loss=0.4346825805694879\n",
      "Fold #2: loss=0.4593868993445528\n",
      "Fold #3: loss=0.41674899522216713\n",
      "Fold #4: loss=0.4851849131056564\n",
      "Fold #5: loss=0.48473291073937\n",
      "Fold #6: loss=0.41274608628217674\n",
      "Fold #7: loss=0.47405291219252377\n",
      "Fold #8: loss=0.44974230059938286\n",
      "Fold #9: loss=0.46340159258241087\n",
      "RandomForestClassifier: Mean loss=0.45463969889068834\n",
      "Model: 3 : RandomForestClassifier(criterion='entropy', n_jobs=-1)\n",
      "Fold #0: loss=0.4511847247326708\n",
      "Fold #1: loss=0.42707704254926593\n",
      "Fold #2: loss=0.5550335199035183\n",
      "Fold #3: loss=0.42186970733328516\n",
      "Fold #4: loss=0.4794331756190797\n",
      "Fold #5: loss=0.4730559509802762\n",
      "Fold #6: loss=0.41116235817215196\n",
      "Fold #7: loss=0.46835919493314265\n",
      "Fold #8: loss=0.4496144890690015\n",
      "Fold #9: loss=0.4625902934553457\n",
      "RandomForestClassifier: Mean loss=0.4599380456747738\n",
      "Model: 4 : ExtraTreesClassifier(n_jobs=-1)\n",
      "Fold #0: loss=0.45496751079363495\n",
      "Fold #1: loss=0.5013051157905043\n",
      "Fold #2: loss=0.5886179891724027\n",
      "Fold #3: loss=0.41646902160044674\n",
      "Fold #4: loss=0.4957910697444236\n",
      "Fold #5: loss=0.4773401028797005\n",
      "Fold #6: loss=0.41935061504547827\n",
      "Fold #7: loss=0.5757908399174205\n",
      "Fold #8: loss=0.4585195863412778\n",
      "Fold #9: loss=0.6210675972963805\n",
      "ExtraTreesClassifier: Mean loss=0.500921944858167\n",
      "Model: 5 : ExtraTreesClassifier(criterion='entropy', n_jobs=-1)\n",
      "Fold #0: loss=0.44825346440152214\n",
      "Fold #1: loss=0.40764412171784686\n",
      "Fold #2: loss=0.5819367378417363\n",
      "Fold #3: loss=0.4140589874942631\n",
      "Fold #4: loss=0.4923489720481471\n",
      "Fold #5: loss=0.5744429921555051\n",
      "Fold #6: loss=0.42334390524742155\n",
      "Fold #7: loss=0.6409291880353659\n",
      "Fold #8: loss=0.45627884947155956\n",
      "Fold #9: loss=0.466653395317917\n",
      "ExtraTreesClassifier: Mean loss=0.49058906137312847\n",
      "Model: 6 : GradientBoostingClassifier(learning_rate=0.05, max_depth=6, n_estimators=50,\n",
      "                           subsample=0.5)\n",
      "Fold #0: loss=0.4820591428786016\n",
      "Fold #1: loss=0.45248898831497514\n",
      "Fold #2: loss=0.46929703269387824\n",
      "Fold #3: loss=0.44565147942743144\n",
      "Fold #4: loss=0.48935648416855027\n",
      "Fold #5: loss=0.48923863501396875\n",
      "Fold #6: loss=0.4478700543685477\n",
      "Fold #7: loss=0.4594068991947806\n",
      "Fold #8: loss=0.463832905944738\n",
      "Fold #9: loss=0.46507605175072847\n",
      "GradientBoostingClassifier: Mean loss=0.46642776737562003\n",
      "\n",
      "Blending models.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import math\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.ensemble import RandomForestClassifier \n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "PATH = \"./data/\"\n",
    "SHUFFLE = False\n",
    "FOLDS = 10\n",
    "\n",
    "def build_ann(input_size,classes,neurons):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_dim=input_size, activation='relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Dense(classes,activation='softmax'))\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "    return model\n",
    "\n",
    "def mlogloss(y_test, preds):\n",
    "    epsilon = 1e-15\n",
    "    sum = 0\n",
    "    for row in zip(preds,y_test):\n",
    "        x = row[0][row[1]]\n",
    "        x = max(epsilon,x)\n",
    "        x = min(1-epsilon,x)\n",
    "        sum+=math.log(x)\n",
    "    return( (-1/len(preds))*sum)\n",
    "\n",
    "def stretch(y):\n",
    "    return (y - y.min()) / (y.max() - y.min())\n",
    "\n",
    "\n",
    "def blend_ensemble(x, y, x_submit):\n",
    "    kf = StratifiedKFold(FOLDS)\n",
    "    folds = list(kf.split(x,y))\n",
    "\n",
    "    models = [\n",
    "        KerasClassifier(build_fn=build_ann,neurons=20,\n",
    "                    input_size=x.shape[1],classes=2),\n",
    "        KNeighborsClassifier(n_neighbors=3),\n",
    "        RandomForestClassifier(n_estimators=100, n_jobs=-1, \n",
    "                               criterion='gini'),\n",
    "        RandomForestClassifier(n_estimators=100, n_jobs=-1, \n",
    "                               criterion='entropy'),\n",
    "        ExtraTreesClassifier(n_estimators=100, n_jobs=-1, \n",
    "                             criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=100, n_jobs=-1, \n",
    "                             criterion='entropy'),\n",
    "        GradientBoostingClassifier(learning_rate=0.05, \n",
    "                subsample=0.5, max_depth=6, n_estimators=50)]\n",
    "\n",
    "    dataset_blend_train = np.zeros((x.shape[0], len(models)))\n",
    "    dataset_blend_test = np.zeros((x_submit.shape[0], len(models)))\n",
    "\n",
    "    for j, model in enumerate(models):\n",
    "        print(\"Model: {} : {}\".format(j, model) )\n",
    "        fold_sums = np.zeros((x_submit.shape[0], len(folds)))\n",
    "        total_loss = 0\n",
    "        for i, (train, test) in enumerate(folds):\n",
    "            x_train = x[train]\n",
    "            y_train = y[train]\n",
    "            x_test = x[test]\n",
    "            y_test = y[test]\n",
    "            model.fit(x_train, y_train)\n",
    "            pred = np.array(model.predict_proba(x_test))\n",
    "            # pred = model.predict_proba(x_test)\n",
    "            dataset_blend_train[test, j] = pred[:, 1]\n",
    "            pred2 = np.array(model.predict_proba(x_submit))\n",
    "            #fold_sums[:, i] = model.predict_proba(x_submit)[:, 1]\n",
    "            fold_sums[:, i] = pred2[:, 1]\n",
    "            loss = mlogloss(y_test, pred)\n",
    "            total_loss+=loss\n",
    "            print(\"Fold #{}: loss={}\".format(i,loss))\n",
    "        print(\"{}: Mean loss={}\".format(model.__class__.__name__,\n",
    "                                        total_loss/len(folds)))\n",
    "        dataset_blend_test[:, j] = fold_sums.mean(1)\n",
    "\n",
    "    print()\n",
    "    print(\"Blending models.\")\n",
    "    blend = LogisticRegression(solver='lbfgs')\n",
    "    blend.fit(dataset_blend_train, y)\n",
    "    return blend.predict_proba(dataset_blend_test)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    np.random.seed(42)  # seed to shuffle the train set\n",
    "\n",
    "    print(\"Loading data...\")\n",
    "    filename_train = os.path.join(PATH, \"bio_train.csv\")\n",
    "    df_train = pd.read_csv(filename_train, na_values=['NA', '?'])\n",
    "\n",
    "    filename_submit = os.path.join(PATH, \"bio_test.csv\")\n",
    "    df_submit = pd.read_csv(filename_submit, na_values=['NA', '?'])\n",
    "\n",
    "    predictors = list(df_train.columns.values)\n",
    "    predictors.remove('Activity')\n",
    "    x = df_train[predictors].values\n",
    "    y = df_train['Activity']\n",
    "    x_submit = df_submit.values\n",
    "\n",
    "    if SHUFFLE:\n",
    "        idx = np.random.permutation(y.size)\n",
    "        x = x[idx]\n",
    "        y = y[idx]\n",
    "\n",
    "    submit_data = blend_ensemble(x, y, x_submit)\n",
    "    submit_data = stretch(submit_data)\n",
    "\n",
    "    ####################\n",
    "    # Build submit file\n",
    "    ####################\n",
    "    ids = [id+1 for id in range(submit_data.shape[0])]\n",
    "    submit_filename = os.path.join(PATH, \"bio_submit.csv\")\n",
    "    submit_df = pd.DataFrame({'MoleculeId': ids, \n",
    "                              'PredictedProbability': \n",
    "                              submit_data[:, 1]},\n",
    "                             columns=['MoleculeId',\n",
    "                            'PredictedProbability'])\n",
    "    submit_df.to_csv(submit_filename, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}