Hyper_parameter_tuning_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6LIXVJeIAmVa"
   },
   "source": [
    "# Hyper-parameter Tunning of Machine Learning (ML) Models\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4XhQDOADAmVe"
   },
   "source": [
    "### Code for Regression Problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zMYHvDbUAmVf"
   },
   "source": [
    "#### `Dataset Used:`\n",
    "Boston housing dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GI497R4ZAmVh"
   },
   "source": [
    "#### `Machine Learning Algorithm Used:`\n",
    "* Random Forest (RF) \n",
    "* Support Vector Machine (SVM) \n",
    "* K-Nearest Neighbor (KNN) \n",
    "* Artificial Neural Network (ANN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yLOZ_qJ1AmVj"
   },
   "source": [
    "#### `Hyper-parameter Tuning Algorithms Used:`\n",
    "* Grid Search \n",
    "* Random Search\n",
    "* Bayesian Optimization with Gaussian Processes (BO-GP)\n",
    "* Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A2MwCNNhAmVk"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "sbrOyFciAmVl"
   },
   "outputs": [],
   "source": [
    "# Importing required libraries\n",
    "import numpy as np\n",
    "import pandas as pd \n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import scipy.stats as stats\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.svm import SVR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RqEAcbr_AmVs"
   },
   "source": [
    "#### Loading Boston Housing Dataset\n",
    "Boston Housing dataset contains information about different houses in Boston. It contains 506 records with 13 columns. The main goal is to predict the value of prices of the house using the given features.\n",
    "\n",
    "For more details about the dataset click here: \n",
    "[Details-1](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) ,\n",
    "[Details-2](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "AxwfEuWEAmVt",
    "outputId": "21db46e9-5f8d-4630-e680-b3ddf6768950"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'DESCR': \".. _boston_dataset:\\n\\nBoston house prices dataset\\n---------------------------\\n\\n**Data Set Characteristics:**  \\n\\n    :Number of Instances: 506 \\n\\n    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\\n\\n    :Attribute Information (in order):\\n        - CRIM     per capita crime rate by town\\n        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\\n        - INDUS    proportion of non-retail business acres per town\\n        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\\n        - NOX      nitric oxides concentration (parts per 10 million)\\n        - RM       average number of rooms per dwelling\\n        - AGE      proportion of owner-occupied units built prior to 1940\\n        - DIS      weighted distances to five Boston employment centres\\n        - RAD      index of accessibility to radial highways\\n        - TAX      full-value property-tax rate per $10,000\\n        - PTRATIO  pupil-teacher ratio by town\\n        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\\n        - LSTAT    % lower status of the population\\n        - MEDV     Median value of owner-occupied homes in $1000's\\n\\n    :Missing Attribute Values: None\\n\\n    :Creator: Harrison, D. and Rubinfeld, D.L.\\n\\nThis is a copy of UCI ML housing dataset.\\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/housing/\\n\\n\\nThis dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\\n\\nThe Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\\nprices and the demand for clean air', J. Environ. Economics & Management,\\nvol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\\n...', Wiley, 1980.   N.B. Various transformations are used in the table on\\npages 244-261 of the latter.\\n\\nThe Boston house-price data has been used in many machine learning papers that address regression\\nproblems.   \\n     \\n.. topic:: References\\n\\n   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\\n   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\\n\",\n",
       " 'data': array([[6.3200e-03, 1.8000e+01, 2.3100e+00, ..., 1.5300e+01, 3.9690e+02,\n",
       "         4.9800e+00],\n",
       "        [2.7310e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9690e+02,\n",
       "         9.1400e+00],\n",
       "        [2.7290e-02, 0.0000e+00, 7.0700e+00, ..., 1.7800e+01, 3.9283e+02,\n",
       "         4.0300e+00],\n",
       "        ...,\n",
       "        [6.0760e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
       "         5.6400e+00],\n",
       "        [1.0959e-01, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9345e+02,\n",
       "         6.4800e+00],\n",
       "        [4.7410e-02, 0.0000e+00, 1.1930e+01, ..., 2.1000e+01, 3.9690e+02,\n",
       "         7.8800e+00]]),\n",
       " 'feature_names': array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "        'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7'),\n",
       " 'filename': '/usr/local/lib/python3.6/dist-packages/sklearn/datasets/data/boston_house_prices.csv',\n",
       " 'target': array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 27.1, 16.5, 18.9, 15. ,\n",
       "        18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2, 13.6, 19.6,\n",
       "        15.2, 14.5, 15.6, 13.9, 16.6, 14.8, 18.4, 21. , 12.7, 14.5, 13.2,\n",
       "        13.1, 13.5, 18.9, 20. , 21. , 24.7, 30.8, 34.9, 26.6, 25.3, 24.7,\n",
       "        21.2, 19.3, 20. , 16.6, 14.4, 19.4, 19.7, 20.5, 25. , 23.4, 18.9,\n",
       "        35.4, 24.7, 31.6, 23.3, 19.6, 18.7, 16. , 22.2, 25. , 33. , 23.5,\n",
       "        19.4, 22. , 17.4, 20.9, 24.2, 21.7, 22.8, 23.4, 24.1, 21.4, 20. ,\n",
       "        20.8, 21.2, 20.3, 28. , 23.9, 24.8, 22.9, 23.9, 26.6, 22.5, 22.2,\n",
       "        23.6, 28.7, 22.6, 22. , 22.9, 25. , 20.6, 28.4, 21.4, 38.7, 43.8,\n",
       "        33.2, 27.5, 26.5, 18.6, 19.3, 20.1, 19.5, 19.5, 20.4, 19.8, 19.4,\n",
       "        21.7, 22.8, 18.8, 18.7, 18.5, 18.3, 21.2, 19.2, 20.4, 19.3, 22. ,\n",
       "        20.3, 20.5, 17.3, 18.8, 21.4, 15.7, 16.2, 18. , 14.3, 19.2, 19.6,\n",
       "        23. , 18.4, 15.6, 18.1, 17.4, 17.1, 13.3, 17.8, 14. , 14.4, 13.4,\n",
       "        15.6, 11.8, 13.8, 15.6, 14.6, 17.8, 15.4, 21.5, 19.6, 15.3, 19.4,\n",
       "        17. , 15.6, 13.1, 41.3, 24.3, 23.3, 27. , 50. , 50. , 50. , 22.7,\n",
       "        25. , 50. , 23.8, 23.8, 22.3, 17.4, 19.1, 23.1, 23.6, 22.6, 29.4,\n",
       "        23.2, 24.6, 29.9, 37.2, 39.8, 36.2, 37.9, 32.5, 26.4, 29.6, 50. ,\n",
       "        32. , 29.8, 34.9, 37. , 30.5, 36.4, 31.1, 29.1, 50. , 33.3, 30.3,\n",
       "        34.6, 34.9, 32.9, 24.1, 42.3, 48.5, 50. , 22.6, 24.4, 22.5, 24.4,\n",
       "        20. , 21.7, 19.3, 22.4, 28.1, 23.7, 25. , 23.3, 28.7, 21.5, 23. ,\n",
       "        26.7, 21.7, 27.5, 30.1, 44.8, 50. , 37.6, 31.6, 46.7, 31.5, 24.3,\n",
       "        31.7, 41.7, 48.3, 29. , 24. , 25.1, 31.5, 23.7, 23.3, 22. , 20.1,\n",
       "        22.2, 23.7, 17.6, 18.5, 24.3, 20.5, 24.5, 26.2, 24.4, 24.8, 29.6,\n",
       "        42.8, 21.9, 20.9, 44. , 50. , 36. , 30.1, 33.8, 43.1, 48.8, 31. ,\n",
       "        36.5, 22.8, 30.7, 50. , 43.5, 20.7, 21.1, 25.2, 24.4, 35.2, 32.4,\n",
       "        32. , 33.2, 33.1, 29.1, 35.1, 45.4, 35.4, 46. , 50. , 32.2, 22. ,\n",
       "        20.1, 23.2, 22.3, 24.8, 28.5, 37.3, 27.9, 23.9, 21.7, 28.6, 27.1,\n",
       "        20.3, 22.5, 29. , 24.8, 22. , 26.4, 33.1, 36.1, 28.4, 33.4, 28.2,\n",
       "        22.8, 20.3, 16.1, 22.1, 19.4, 21.6, 23.8, 16.2, 17.8, 19.8, 23.1,\n",
       "        21. , 23.8, 23.1, 20.4, 18.5, 25. , 24.6, 23. , 22.2, 19.3, 22.6,\n",
       "        19.8, 17.1, 19.4, 22.2, 20.7, 21.1, 19.5, 18.5, 20.6, 19. , 18.7,\n",
       "        32.7, 16.5, 23.9, 31.2, 17.5, 17.2, 23.1, 24.5, 26.6, 22.9, 24.1,\n",
       "        18.6, 30.1, 18.2, 20.6, 17.8, 21.7, 22.7, 22.6, 25. , 19.9, 20.8,\n",
       "        16.8, 21.9, 27.5, 21.9, 23.1, 50. , 50. , 50. , 50. , 50. , 13.8,\n",
       "        13.8, 15. , 13.9, 13.3, 13.1, 10.2, 10.4, 10.9, 11.3, 12.3,  8.8,\n",
       "         7.2, 10.5,  7.4, 10.2, 11.5, 15.1, 23.2,  9.7, 13.8, 12.7, 13.1,\n",
       "        12.5,  8.5,  5. ,  6.3,  5.6,  7.2, 12.1,  8.3,  8.5,  5. , 11.9,\n",
       "        27.9, 17.2, 27.5, 15. , 17.2, 17.9, 16.3,  7. ,  7.2,  7.5, 10.4,\n",
       "         8.8,  8.4, 16.7, 14.2, 20.8, 13.4, 11.7,  8.3, 10.2, 10.9, 11. ,\n",
       "         9.5, 14.5, 14.1, 16.1, 14.3, 11.7, 13.4,  9.6,  8.7,  8.4, 12.8,\n",
       "        10.5, 17.1, 18.4, 15.4, 10.8, 11.8, 14.9, 12.6, 14.1, 13. , 13.4,\n",
       "        15.2, 16.1, 17.8, 14.9, 14.1, 12.7, 13.5, 14.9, 20. , 16.4, 17.7,\n",
       "        19.5, 20.2, 21.4, 19.9, 19. , 19.1, 19.1, 20.1, 19.9, 19.6, 23.2,\n",
       "        29.8, 13.8, 13.3, 16.7, 12. , 14.6, 21.4, 23. , 23.7, 25. , 21.8,\n",
       "        20.6, 21.2, 19.1, 20.6, 15.2,  7. ,  8.1, 13.6, 20.1, 21.8, 24.5,\n",
       "        23.1, 19.7, 18.3, 21.2, 17.5, 16.8, 22.4, 20.6, 23.9, 22. , 11.9])}"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loading dataset\n",
    "X, y = datasets.load_boston(return_X_y=True)\n",
    "datasets.load_boston()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "nrQdYgKfAmV4",
    "outputId": "56c355f2-266e-4170-a5ff-161fa643967a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506, 13)\n"
     ]
    }
   ],
   "source": [
    "print(X.shape) #The data matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "oWxYxskcAmV-",
    "outputId": "a6c517d8-ebc9-42a1-c481-9ab348eba0ce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(506,)\n"
     ]
    }
   ],
   "source": [
    "print(y.shape) #The regression target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WSCCggf_AmWG"
   },
   "source": [
    "### Baseline Machine Learning Models: Regressor with default Hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "__AfrricAmWH"
   },
   "source": [
    "### `Random Forest` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "5dnEpJ_9AmWI",
    "outputId": "4a4b7d41-25fd-478f-bd3c-447269e878b5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Square Error (RF) :29.748647641812237\n"
     ]
    }
   ],
   "source": [
    "# Random Forest (RF) with 3-fold cross validation\n",
    "RF_clf = RandomForestRegressor()\n",
    "RF_scores = cross_val_score(RF_clf, X, y, cv = 3, scoring = 'neg_mean_squared_error') \n",
    "print(\"Mean Square Error (RF) :\" + str(-RF_scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KlCw8FxnAmWQ"
   },
   "source": [
    "### `Support Vector Machine`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "czTHVWzLAmWT",
    "outputId": "3c4a279e-d0ed-45bd-eec4-fffec19bc25e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Square Error (SVM) :77.42951812579332\n"
     ]
    }
   ],
   "source": [
    "# Support Vector Machine (SVM)\n",
    "SVM_clf = SVR(gamma ='scale')\n",
    "SVM_scores = cross_val_score(SVM_clf, X, y, cv = 3, scoring = 'neg_mean_squared_error')\n",
    "print(\"Mean Square Error (SVM) :\" + str(-SVM_scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MaB4c8p9AmWc"
   },
   "source": [
    "### `K-Nearest Neighbor` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "Fm0bdf8DAmWd",
    "outputId": "d76d8e8e-e08c-4554-a6b2-feff342e2bbf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Square Error (KNN) :81.48773186343571\n"
     ]
    }
   ],
   "source": [
    "# K-Nearest Neighbor (KNN)\n",
    "KN_clf = KNeighborsRegressor()\n",
    "KN_scores = cross_val_score(KN_clf, X, y, cv = 3,scoring = 'neg_mean_squared_error')\n",
    "print(\"Mean Square Error (KNN) :\" + str(-KN_scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6B1BhwhxAmWl"
   },
   "source": [
    "### `Artificial Neural Network`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "U9xTUiVpAmWn",
    "outputId": "1248b84d-b24a-4112-c68c-14c554d10433"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Square Error (ANN):44.56844398014558\n"
     ]
    }
   ],
   "source": [
    "# Artificial Neural Network (ANN)\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Input\n",
    "from keras.wrappers.scikit_learn import KerasRegressor\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "def ann_model(optimizer = 'adam', neurons = 32,batch_size = 32, epochs = 50 ,activation = 'relu',patience = 5,loss = 'mse'):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(neurons, input_shape = (X.shape[1],), activation = activation))\n",
    "    model.add(Dense(neurons, activation = activation))\n",
    "    model.add(Dense(1))\n",
    "    model.compile(optimizer = optimizer ,loss = loss) \n",
    "    early_stopping = EarlyStopping(monitor = \"loss\", patience = patience)\n",
    "    history = model.fit(X, y,batch_size = batch_size,epochs = epochs,callbacks = [early_stopping],verbose=0)\n",
    "    return model\n",
    "\n",
    "ANN_clf = KerasRegressor(build_fn = ann_model, verbose = 0)\n",
    "ANN_scores = cross_val_score(ANN_clf, X, y, cv = 3,scoring = 'neg_mean_squared_error')\n",
    "print(\"Mean Square Error (ANN):\"+ str(-ANN_scores.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dpD-x-QcAmWu"
   },
   "source": [
    "### Hyper-parameter Tuning Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "riW63iO6AmWw"
   },
   "source": [
    "### ` 1] Grid Search`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "AbdwwVzkAmWz"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MVFXL1vhAmXA"
   },
   "source": [
    "#### `Random Forest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "UolMvFxqAmXC",
    "outputId": "335715e4-d2d6-4326-a2c6-da4f68771d9c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 15, 'n_estimators': 20}\n",
      "Mean Square Error (RF) : 29.059884504204902\n"
     ]
    }
   ],
   "source": [
    "# Random Forest (RF)\n",
    "RF_params = {\n",
    "    'n_estimators': [10, 20, 30],\n",
    "    'max_depth': [15,20,25,30,50],\n",
    "}\n",
    "RF_clf = RandomForestRegressor(random_state = 0)\n",
    "RF_grid = GridSearchCV(RF_clf, RF_params, cv = 3, scoring = 'neg_mean_squared_error')\n",
    "RF_grid.fit(X, y)\n",
    "print(RF_grid.best_params_)\n",
    "print(\"Mean Square Error (RF) : \"+ str(-RF_grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l7H5vmJYAmXI"
   },
   "source": [
    "#### `Support Vector Machine`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "tIZAyy3QAmXJ",
    "outputId": "85047353-f1dd-4c29-ada8-aa9a938d769f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 1000, 'epsilon': 1, 'kernel': 'rbf'}\n",
      "Mean Square Error (SVM) :49.71699970931197\n"
     ]
    }
   ],
   "source": [
    "# Support Vector Machine (SVM)\n",
    "SVM_params = {\n",
    "    'C': [1,10, 100,1000],\n",
    "    'kernel' :['poly','rbf','sigmoid'],\n",
    "    'epsilon':[0.001, 0.01,0.1,1]\n",
    "}\n",
    "SVM_clf = SVR(gamma = 'scale')\n",
    "SVM_grid = GridSearchCV(SVM_clf, SVM_params, cv = 3, scoring = 'neg_mean_squared_error')\n",
    "SVM_grid.fit(X, y)\n",
    "print(SVM_grid.best_params_)\n",
    "print(\"Mean Square Error (SVM) :\"+ str(-SVM_grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJn0VVPzAmXP"
   },
   "source": [
    "#### `K-Nearest Neighbor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "CTGQZqywAmXQ",
    "outputId": "896a8e19-8284-4405-ac8d-aaaa8b332d30"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 6}\n",
      "Mean Square Error (KNN) :80.83005201647829\n"
     ]
    }
   ],
   "source": [
    "# K-nearest Neighnor (KNN)\n",
    "KNN_params = {\n",
    "    'n_neighbors': [2,4,6,8]\n",
    "}\n",
    "KNN_clf = KNeighborsRegressor()\n",
    "KNN_grid = GridSearchCV(KNN_clf, KNN_params, cv=3, scoring='neg_mean_squared_error')\n",
    "KNN_grid.fit(X, y)\n",
    "print(KNN_grid.best_params_)\n",
    "print(\"Mean Square Error (KNN) :\"+ str(-KNN_grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ROt4p1SWAmXY"
   },
   "source": [
    "#### `Artificial Neural Network`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "FV6DimwOAmXZ",
    "outputId": "c66fd566-9e04-4b18-dd9f-f84be1abbc5f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'tanh', 'batch_size': 16, 'epochs': 50, 'loss': 'mse', 'neurons': 32, 'optimizer': 'rmsprop', 'patience': 3}\n",
      "MSE:37.868857592056166\n"
     ]
    }
   ],
   "source": [
    "# Artificial Neural Network (ANN)\n",
    "RF_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32],\n",
    "    'neurons':[16,32],\n",
    "    'epochs':[20,50],\n",
    "    'patience':[3,5]\n",
    "}\n",
    "RF_clf = KerasRegressor(build_fn = ann_model, verbose = 0)\n",
    "RF_grid = GridSearchCV(RF_clf, RF_params, cv=3,scoring = 'neg_mean_squared_error')\n",
    "RF_grid.fit(X, y)\n",
    "print(RF_grid.best_params_)\n",
    "print(\"MSE:\"+ str(-RF_grid.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nyvvnbT4AmXe"
   },
   "source": [
    "###  `2] Random Search`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "ISy1hFRsAmXf"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import randint as sp_randint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1jIHAV8XAmXl"
   },
   "source": [
    "#### `Random Forest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "lsNcD38LAmXm",
    "outputId": "80f1513e-6ffc-425e-f133-2bffa3e216d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': 'mse', 'max_depth': 15, 'n_estimators': 93}\n",
      "Mean Square Error (RF):28.819638693418824\n"
     ]
    }
   ],
   "source": [
    "# Random Forest (RF)\n",
    "RF_params = {\n",
    "    'n_estimators': sp_randint(10,100),\n",
    "    'max_depth': sp_randint(5,50),\n",
    "    \"criterion\":['mse','mae']\n",
    "} \n",
    "RF_clf = RandomForestRegressor(random_state = 0)\n",
    "RF_Random = RandomizedSearchCV(RF_clf, param_distributions = RF_params,\n",
    "                               n_iter = 20 ,cv = 3,scoring = 'neg_mean_squared_error')\n",
    "RF_Random.fit(X, y)\n",
    "print(RF_Random.best_params_)\n",
    "print(\"Mean Square Error (RF):\"+ str(-RF_Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aeoZDfEiAmXq"
   },
   "source": [
    "#### `Support Vector Machine`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "BKAwYL1LAmXr",
    "outputId": "665a4b31-e605-4f86-e46c-630c09709a4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 27.21076956480878, 'epsilon': 0.46048652191242667, 'kernel': 'poly'}\n",
      "Mean Square Error (SVM) :59.99237088014893\n"
     ]
    }
   ],
   "source": [
    "# Support Vector Machine (SVM)\n",
    "SVM_params = {\n",
    "    'C': stats.uniform(0,50),\n",
    "    \"kernel\":['poly','rbf'],\n",
    "    \"epsilon\":stats.uniform(0,1)\n",
    "}\n",
    "SVM_clf = SVR(gamma = 'scale')\n",
    "SVM_Random = RandomizedSearchCV(SVM_clf, param_distributions = SVM_params,\n",
    "                            n_iter = 20,cv = 3,scoring = 'neg_mean_squared_error')\n",
    "SVM_Random.fit(X, y)\n",
    "print(SVM_Random.best_params_)\n",
    "print(\"Mean Square Error (SVM) :\"+ str(-SVM_Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CMb8P2maAmXw"
   },
   "source": [
    "#### `K-Nearest Neighbor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "sQeUuCtyAmXx",
    "outputId": "37e1e6b8-b97e-4cef-8a01-b5a30f65adad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_neighbors': 13}\n",
      "Mean Square Error (KNN) :80.74121499347262\n"
     ]
    }
   ],
   "source": [
    "# K-Nearest Neighbor (KNN)\n",
    "KNN_params = {\n",
    "    'n_neighbors': sp_randint(1,20),\n",
    "}\n",
    "KNN_clf = KNeighborsRegressor()\n",
    "KNN_Random = RandomizedSearchCV(KNN_clf, param_distributions = KNN_params,\n",
    "                            n_iter = 10,cv = 3,scoring = 'neg_mean_squared_error')\n",
    "KNN_Random.fit(X, y)\n",
    "print(KNN_Random.best_params_)\n",
    "print(\"Mean Square Error (KNN) :\"+ str(-KNN_Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uF_fMtsPAmX0"
   },
   "source": [
    "#### `Artificial Neural Network`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "Ck57nEMUAmX0",
    "outputId": "d142a86c-9fbe-4e99-b797-0d3740ca3472"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'activation': 'relu', 'batch_size': 16, 'epochs': 50, 'loss': 'mse', 'neurons': 71, 'optimizer': 'adam', 'patience': 18}\n",
      "Mean Square Error (ANN):45.48700310136937\n"
     ]
    }
   ],
   "source": [
    "# Artificial Neural Network (ANN)\n",
    "ANN_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32],\n",
    "    'neurons':sp_randint(10,100),\n",
    "    'epochs':[20,50],\n",
    "    'patience':sp_randint(5,20)\n",
    "}\n",
    "ANN_clf = KerasRegressor(build_fn = ann_model, verbose = 0)\n",
    "ANN_Random = RandomizedSearchCV(ANN_clf, param_distributions = ANN_params,\n",
    "                                n_iter = 10,cv = 3,scoring = 'neg_mean_squared_error')\n",
    "ANN_Random.fit(X, y)\n",
    "print(ANN_Random.best_params_)\n",
    "print(\"Mean Square Error (ANN):\"+ str(-ANN_Random.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QBA34Ri-Gshj"
   },
   "source": [
    "### `3] Bayesian Optimization with Gaussian Processes (BO-GP)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "pC_Rp7jTAmX3"
   },
   "outputs": [],
   "source": [
    "from skopt import Optimizer\n",
    "from skopt import BayesSearchCV \n",
    "from skopt.space import Real, Categorical, Integer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LySXAx7VJBmb"
   },
   "source": [
    "#### `Random Forest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "J_xhnDkLIn9v",
    "outputId": "94d088e6-ef7e-4839-e0a1-1c3d7c05005e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('criterion', 'mse'), ('max_depth', 31), ('n_estimators', 52)])\n",
      "Mean Square Error (RF):28.846466514254967\n"
     ]
    }
   ],
   "source": [
    "# Random Forest (RF)\n",
    "RF_params = {\n",
    "    'n_estimators': Integer(10,100),\n",
    "    'max_depth': Integer(5,50),\n",
    "    \"criterion\":['mse','mae']\n",
    "}\n",
    "RF_clf = RandomForestRegressor(random_state = 0)\n",
    "RF_Bayes = BayesSearchCV(RF_clf, RF_params,cv = 3,n_iter = 20, scoring = 'neg_mean_squared_error') \n",
    "RF_Bayes.fit(X, y)\n",
    "print(RF_Bayes.best_params_)\n",
    "print(\"Mean Square Error (RF):\"+ str(-RF_Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gJ9-TVAEJy2W"
   },
   "source": [
    "### `Support Vector Machine`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "Oa61n1SAJ2N0",
    "outputId": "31247e85-e408-4836-9934-91761cc9dbf1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('C', 43.7998974955581), ('epsilon', 0.14977167499173435), ('kernel', 'poly')])\n",
      "Mean Square Error (SVM):59.450729055644864\n"
     ]
    }
   ],
   "source": [
    "# Support Vector Machine (SVM)\n",
    "SVM_params = {\n",
    "    \"kernel\":['poly','rbf'],\n",
    "    'C': Real(1,50),\n",
    "    'epsilon': Real(0,1)\n",
    "}\n",
    "SVM_clf = SVR(gamma='scale')\n",
    "SVM_Bayes = BayesSearchCV(SVM_clf, SVM_params,cv = 3,n_iter = 20, scoring = 'neg_mean_squared_error')\n",
    "SVM_Bayes.fit(X, y)\n",
    "print(SVM_Bayes.best_params_)\n",
    "print(\"Mean Square Error (SVM):\"+ str(-SVM_Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6rewYf76KmQG"
   },
   "source": [
    "#### `K-Nearest Neighbor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "KQCocUNzKlXU",
    "outputId": "a1ff1aea-e576-4257-bb92-3d4154d568a6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('n_neighbors', 12)])\n",
      "Mean Square Error (KNN):81.38111289525693\n"
     ]
    }
   ],
   "source": [
    "# K-Nearest Neighbor (KNN)\n",
    "KNN_params = {\n",
    "    'n_neighbors': Integer(1,20),\n",
    "}\n",
    "KNN_clf = KNeighborsRegressor()\n",
    "KNN_Bayes = BayesSearchCV(KNN_clf, KNN_params,cv = 3,n_iter = 10, scoring = 'neg_mean_squared_error')\n",
    "KNN_Bayes.fit(X, y)\n",
    "print(KNN_Bayes.best_params_)\n",
    "print(\"Mean Square Error (KNN):\"+ str(-KNN_Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Q4IDrnPL2tM"
   },
   "source": [
    "#### `Artificial Neural Network (ANN)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "y8brG7bOL3CZ",
    "outputId": "b84063f1-9d97-45a8-cb0b-b231359ca1ac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('activation', 'tanh'), ('batch_size', 28), ('epochs', 44), ('loss', 'mse'), ('neurons', 81), ('optimizer', 'adam'), ('patience', 9)])\n",
      "Mean Square Error (ANN):43.640675892710725\n"
     ]
    }
   ],
   "source": [
    "# Artificial Neural Network (ANN)\n",
    "ANN_params = {\n",
    "    'optimizer': ['adam','rmsprop'],\n",
    "    'activation': ['relu','tanh'],\n",
    "    'loss': ['mse','mae'],\n",
    "    'batch_size': [16,32],\n",
    "    'neurons':Integer(10,100),\n",
    "    'epochs':[20,50],\n",
    "    'patience':Integer(5,20)\n",
    "}\n",
    "ANN_clf = KerasRegressor(build_fn = ann_model, verbose = 0)\n",
    "ANN_Bayes = BayesSearchCV(ANN_clf, ANN_params,cv = 3,n_iter = 10, scoring = 'neg_mean_squared_error')\n",
    "ANN_Bayes.fit(X, y)\n",
    "print(ANN_Bayes.best_params_)\n",
    "print(\"Mean Square Error (ANN):\"+ str(-ANN_Bayes.best_score_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N1l86GDIMk4H"
   },
   "source": [
    "### `4] Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "B3Vcjoo1NIbX"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from hyperopt import hp, fmin, tpe, STATUS_OK, Trials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5KQ4_b74NE2c"
   },
   "source": [
    "#### `Random Forest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "jqTyukq-Nd0N",
    "outputId": "6e72b18e-edcb-44f2-b362-4ec34318d434"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20/20 [00:20<00:00,  1.01s/it, best loss: 28.50435224273346]\n",
      "Estimated optimum (RF):{'criterion': 0, 'max_depth': 38.0, 'n_estimators': 44.0}\n"
     ]
    }
   ],
   "source": [
    "# Random Forest (RF)\n",
    "def RF_fun(params):\n",
    "    params = {\n",
    "        'n_estimators': int(params['n_estimators']), \n",
    "        'max_depth': int(params['max_depth']),\n",
    "        \"criterion\":str(params['criterion'])\n",
    "    }\n",
    "    RF_clf = RandomForestRegressor(**params)\n",
    "    RF_score = -np.mean(cross_val_score(RF_clf, X, y, cv = 3, n_jobs = -1,scoring = \"neg_mean_squared_error\"))\n",
    "    return {'loss':RF_score, 'status': STATUS_OK }\n",
    "\n",
    "RF_space = {\n",
    "    'n_estimators': hp.quniform('n_estimators', 10, 100, 1),\n",
    "    'max_depth': hp.quniform('max_depth', 5, 50, 1),\n",
    "    \"criterion\":hp.choice('criterion',['mse','mae'])\n",
    "}\n",
    "\n",
    "RF_best = fmin(fn = RF_fun, space = RF_space, algo = tpe.suggest, max_evals = 20)\n",
    "print(\"Estimated optimum (RF):\" +str (RF_best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "owln7VETNYkF"
   },
   "source": [
    "#### `Support Vector Machine`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "xIEZqHnyMt7H",
    "outputId": "54c72df3-c86e-45bc-a5ef-14fa491d39d2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20/20 [00:00<00:00, 22.73it/s, best loss: 60.248025652042806]\n",
      "Estimated optimum (SVM):{'C': 31.83399735036318, 'epsilon': -0.10130152267175614, 'kernel': 0}\n"
     ]
    }
   ],
   "source": [
    "# Support Vector Machine (SVM)\n",
    "def SVM_fun(params):\n",
    "    params = {\n",
    "        \"kernel\":str(params['kernel']),\n",
    "        'C': abs(float(params['C'])), \n",
    "        'epsilon': abs(float(params['epsilon'])),\n",
    "    }\n",
    "    SVM_clf = SVR(gamma='scale', **params)\n",
    "    SVM_score = -np.mean(cross_val_score(SVM_clf, X, y, cv = 3, n_jobs = -1, scoring=\"neg_mean_squared_error\"))\n",
    "    return {'loss':SVM_score, 'status': STATUS_OK }\n",
    "\n",
    "SVM_space = {\n",
    "    \"kernel\":hp.choice('kernel',['poly','rbf']),\n",
    "    'C': hp.normal('C', 0, 50),\n",
    "    'epsilon': hp.normal('epsilon', 0, 1),\n",
    "}\n",
    "\n",
    "SVM_best = fmin(fn = SVM_fun ,space = SVM_space, algo=tpe.suggest, max_evals = 20)\n",
    "print(\"Estimated optimum (SVM):\" +str(SVM_best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bkN5jDKgNezb"
   },
   "source": [
    "#### `K-Nearest Neighbor`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "Nr8oCSEENieY",
    "outputId": "a2d748c9-ac54-4343-8935-9c4bc3f65fb3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:00<00:00, 64.76it/s, best loss: 80.83005201647829]\n",
      "Estimated optimum (KNN):{'n_neighbors': 6.0}\n"
     ]
    }
   ],
   "source": [
    "#K-Nearest Neighbor (KNN)\n",
    "def KNN_fun(params):\n",
    "    params = {'n_neighbors': abs(int(params['n_neighbors']))}\n",
    "    KNN_clf = KNeighborsRegressor(**params)\n",
    "    KNN_score = -np.mean(cross_val_score(KNN_clf, X, y, cv = 3, n_jobs = -1, scoring = \"neg_mean_squared_error\"))\n",
    "    return {'loss':KNN_score, 'status': STATUS_OK }\n",
    "\n",
    "KNN_space = {'n_neighbors': hp.quniform('n_neighbors', 1, 20, 1),}\n",
    "\n",
    "KNN_best = fmin(fn = KNN_fun, space = KNN_space,algo = tpe.suggest, max_evals = 10)\n",
    "print(\"Estimated optimum (KNN):\"+str(KNN_best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qlxhyxvxNkRO"
   },
   "source": [
    "#### `Artificial Neural Network`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "1w0TTrOyNjFD",
    "outputId": "ad7b9c23-b37e-4e26-98ed-f3f9460d7325"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:55<00:00,  5.56s/it, best loss: 36.59635901614879]\n",
      "Estimated optimum (ANN): {'activation': 1, 'batch_size': 16.0, 'epochs': 40.0, 'loss': 0, 'neurons': 100.0, 'optimizer': 1, 'patience': 10.0}\n"
     ]
    }
   ],
   "source": [
    "#Artificial Neural Network (ANN)\n",
    "def ANN_fun(params):\n",
    "    params = {\n",
    "        \"optimizer\":str(params['optimizer']),\n",
    "        \"activation\":str(params['activation']),\n",
    "        \"loss\":str(params['loss']),\n",
    "        'batch_size': abs(int(params['batch_size'])),\n",
    "        'neurons': abs(int(params['neurons'])),\n",
    "        'epochs': abs(int(params['epochs'])),\n",
    "        'patience': abs(int(params['patience']))\n",
    "    }\n",
    "    ANN_clf = KerasRegressor(build_fn = ann_model,**params, verbose = 0)\n",
    "    ANN_score = -np.mean(cross_val_score(ANN_clf, X, y, cv = 3, scoring = \"neg_mean_squared_error\"))\n",
    "    return {'loss':ANN_score, 'status': STATUS_OK }\n",
    "\n",
    "ANN_space = {\n",
    "    \"optimizer\":hp.choice('optimizer',['adam','rmsprop']),\n",
    "    \"activation\":hp.choice('activation',['relu','tanh']),\n",
    "    \"loss\":hp.choice('loss',['mse','mae']),\n",
    "    'batch_size': hp.quniform('batch_size', 16, 32,16),\n",
    "    'neurons': hp.quniform('neurons', 10, 100,10),\n",
    "    'epochs': hp.quniform('epochs', 20, 50,20),\n",
    "    'patience': hp.quniform('patience', 5, 20,5),\n",
    "}\n",
    "\n",
    "ANN_best = fmin(fn = ANN_fun, space = ANN_space, algo = tpe.suggest, max_evals = 10)\n",
    "print(\"Estimated optimum (ANN): \" + str(ANN_best))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Gpvu3XbrTo1f"
   },
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Hyper_parameter_tuning_Regression.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}