t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 8: Kaggle Data Sets**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 8 Material\n",
    "\n",
    "* Part 8.1: Introduction to Kaggle [[Video]](https://www.youtube.com/watch?v=v4lJBhdCuCU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_1_kaggle_intro.ipynb)\n",
    "* Part 8.2: Building Ensembles with Scikit-Learn and Keras [[Video]](https://www.youtube.com/watch?v=LQ-9ZRBLasw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_2_keras_ensembles.ipynb)\n",
    "* Part 8.3: How Should you Architect Your Keras Neural Network: Hyperparameters [[Video]](https://www.youtube.com/watch?v=1q9klwSoUQw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_3_keras_hyperparameters.ipynb)\n",
    "* **Part 8.4: Bayesian Hyperparameter Optimization for Keras** [[Video]](https://www.youtube.com/watch?v=sXdxyUCCm8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_4_bayesian_hyperparameter_opt.ipynb)\n",
    "* Part 8.5: Current Semester's Kaggle [[Video]](https://www.youtube.com/watch?v=PHQt0aUasRg&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_08_5_kaggle_project.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "# Startup Google CoLab\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 8.4: Bayesian Hyperparameter Optimization for Keras\n",
    "\n",
    "Bayesian Hyperparameter Optimization is a method of finding hyperparameters in a more efficient way than a grid search.  Because each candidate set of hyperparameters requires a retraining of the neural network, it is best to keep the number of candidate sets to a minimum. Bayesian Hyperparameter Optimization achieves this by training a model to predict good candidate sets of hyperparameters. [[Cite:snoek2012practical]](https://arxiv.org/pdf/1206.2944.pdf)\n",
    "\n",
    "* [bayesian-optimization](https://github.com/fmfn/BayesianOptimization)\n",
    "* [hyperopt](https://github.com/hyperopt/hyperopt)\n",
    "* [spearmint](https://github.com/JasperSnoek/spearmint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ignore useless W0819 warnings generated by TensorFlow 2.0.  \n",
    "# Hopefully can remove this ignore in the future.\n",
    "# See https://github.com/tensorflow/tensorflow/issues/31308\n",
    "import logging, os\n",
    "logging.disable(logging.WARNING)\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"\n",
    "\n",
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\")],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\")],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['age'] = zscore(df['age'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop('product').drop('id')\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df['product']) # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've preprocessed the data, we can begin the hyperparameter optimization.  We start by creating a function that generates the model based on just three parameters.  Bayesian optimization works on a vector of numbers, not on a problematic notion like how many layers and neurons are on each layer.  To represent this complex neuron structure as a vector, we use several numbers to describe this structure.   \n",
    "\n",
    "* **dropout** - The dropout percent for each layer.\n",
    "* **neuronPct** - What percent of our fixed 5,000 maximum number of neurons do we wish to use?  This parameter specifies the total count of neurons in the entire network.\n",
    "* **neuronShrink** - Neural networks usually start with more neurons on the first hidden layer and then decrease this count for additional layers.  This percent specifies how much to shrink subsequent layers based on the previous layer.  Once we run out of neurons (with the count specified by neuronPft), we stop adding more layers.\n",
    "\n",
    "These three numbers define the structure of the neural network.  The commends in the below code show exactly how the program constructs the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np\n",
    "import time\n",
    "import tensorflow.keras.initializers\n",
    "import statistics\n",
    "import tensorflow.keras\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation, Dropout, InputLayer\n",
    "from tensorflow.keras import regularizers\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from tensorflow.keras.layers import LeakyReLU,PReLU\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "def generate_model(dropout, neuronPct, neuronShrink):\n",
    "    # We start with some percent of 5000 starting neurons on \n",
    "    # the first hidden layer.\n",
    "    neuronCount = int(neuronPct * 5000)\n",
    "    \n",
    "    # Construct neural network\n",
    "    model = Sequential()\n",
    "\n",
    "    # So long as there would have been at least 25 neurons and \n",
    "    # fewer than 10\n",
    "    # layers, create a new layer.\n",
    "    layer = 0\n",
    "    while neuronCount>25 and layer<10:\n",
    "        # The first (0th) layer needs an input input_dim(neuronCount)\n",
    "        if layer==0:\n",
    "            model.add(Dense(neuronCount, \n",
    "                input_dim=x.shape[1], \n",
    "                activation=PReLU()))\n",
    "        else:\n",
    "            model.add(Dense(neuronCount, activation=PReLU())) \n",
    "        layer += 1\n",
    "\n",
    "        # Add dropout after each hidden layer\n",
    "        model.add(Dropout(dropout))\n",
    "\n",
    "        # Shrink neuron count for each layer\n",
    "        neuronCount = neuronCount * neuronShrink\n",
    "\n",
    "    model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test this code to see how it creates a neural network based on three such parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 500)               24500     \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 125)               62750     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 125)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 31)                3937      \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 31)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 7)                 224       \n",
      "=================================================================\n",
      "Total params: 91,411\n",
      "Trainable params: 91,411\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Generate a model and see what the resulting structure looks like.\n",
    "model = generate_model(dropout=0.2, neuronPct=0.1, neuronShrink=0.25)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a function to evaluate the neural network, using three such parameters.  We use bootstrapping because one single training run might simply have \"bad luck\" with the random weights assigned.  We use this function to train and then evaluate the neural network.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.6720430161559489\n"
     ]
    }
   ],
   "source": [
    "def evaluate_network(dropout,lr,neuronPct,neuronShrink):\n",
    "    SPLITS = 2\n",
    "\n",
    "    # Bootstrap\n",
    "    boot = StratifiedShuffleSplit(n_splits=SPLITS, test_size=0.1)\n",
    "\n",
    "    # Track progress\n",
    "    mean_benchmark = []\n",
    "    epochs_needed = []\n",
    "    num = 0\n",
    "    \n",
    "\n",
    "    # Loop through samples\n",
    "    for train, test in boot.split(x,df['product']):\n",
    "        start_time = time.time()\n",
    "        num+=1\n",
    "\n",
    "        # Split train and test\n",
    "        x_train = x[train]\n",
    "        y_train = y[train]\n",
    "        x_test = x[test]\n",
    "        y_test = y[test]\n",
    "\n",
    "        model = generate_model(dropout, neuronPct, neuronShrink)\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=Adam(lr=lr))\n",
    "        monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, \n",
    "        patience=100, verbose=0, mode='auto', restore_best_weights=True)\n",
    "\n",
    "        # Train on the bootstrap sample\n",
    "        model.fit(x_train,y_train,validation_data=(x_test,y_test),\n",
    "                  callbacks=[monitor],verbose=0,epochs=1000)\n",
    "        epochs = monitor.stopped_epoch\n",
    "        epochs_needed.append(epochs)\n",
    "\n",
    "        # Predict on the out of boot (validation)\n",
    "        pred = model.predict(x_test)\n",
    "\n",
    "        # Measure this bootstrap's log loss\n",
    "        y_compare = np.argmax(y_test,axis=1) # For log loss calculation\n",
    "        score = metrics.log_loss(y_compare, pred)\n",
    "        mean_benchmark.append(score)\n",
    "        m1 = statistics.mean(mean_benchmark)\n",
    "        m2 = statistics.mean(epochs_needed)\n",
    "        mdev = statistics.pstdev(mean_benchmark)\n",
    "\n",
    "        # Record this iteration\n",
    "        time_took = time.time() - start_time\n",
    "        \n",
    "    tensorflow.keras.backend.clear_session()\n",
    "    return (-m1)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can try any combination of our three hyperparameters, plus the learning rate, to see how effective these four numbers are.  Of course, our goal is not to manually choose different combinations of these four hyperparameters; we seek to automate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(evaluate_network(\n",
    "    dropout=0.2,\n",
    "    lr=1e-3,\n",
    "    neuronPct=0.2,\n",
    "    neuronShrink=0.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now automat this process.  We define the bounds for each of these four hyperparameters and begin the Bayesian optimization.  Once the program completes, the best combination of hyperparameters found is displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|   iter    |  target   |  dropout  |    lr     | neuronPct | neuron... |\n",
      "-------------------------------------------------------------------------\n",
      "| \u001b[0m 1       \u001b[0m | \u001b[0m-0.7124  \u001b[0m | \u001b[0m 0.2081  \u001b[0m | \u001b[0m 0.07203 \u001b[0m | \u001b[0m 0.01011 \u001b[0m | \u001b[0m 0.3093  \u001b[0m |\n",
      "| \u001b[0m 2       \u001b[0m | \u001b[0m-0.7159  \u001b[0m | \u001b[0m 0.07323 \u001b[0m | \u001b[0m 0.009234\u001b[0m | \u001b[0m 0.1944  \u001b[0m | \u001b[0m 0.3521  \u001b[0m |\n",
      "| \u001b[0m 3       \u001b[0m | \u001b[0m-13.85   \u001b[0m | \u001b[0m 0.198   \u001b[0m | \u001b[0m 0.05388 \u001b[0m | \u001b[0m 0.425   \u001b[0m | \u001b[0m 0.6884  \u001b[0m |\n",
      "| \u001b[0m 4       \u001b[0m | \u001b[0m-0.8022  \u001b[0m | \u001b[0m 0.102   \u001b[0m | \u001b[0m 0.08781 \u001b[0m | \u001b[0m 0.03711 \u001b[0m | \u001b[0m 0.6738  \u001b[0m |\n",
      "| \u001b[0m 5       \u001b[0m | \u001b[0m-0.8295  \u001b[0m | \u001b[0m 0.2082  \u001b[0m | \u001b[0m 0.05587 \u001b[0m | \u001b[0m 0.149   \u001b[0m | \u001b[0m 0.2061  \u001b[0m |\n",
      "| \u001b[0m 6       \u001b[0m | \u001b[0m-16.32   \u001b[0m | \u001b[0m 0.3996  \u001b[0m | \u001b[0m 0.09683 \u001b[0m | \u001b[0m 0.3203  \u001b[0m | \u001b[0m 0.6954  \u001b[0m |\n",
      "| \u001b[0m 7       \u001b[0m | \u001b[0m-6.339   \u001b[0m | \u001b[0m 0.4373  \u001b[0m | \u001b[0m 0.08946 \u001b[0m | \u001b[0m 0.09419 \u001b[0m | \u001b[0m 0.04866 \u001b[0m |\n",
      "| \u001b[0m 8       \u001b[0m | \u001b[0m-0.8378  \u001b[0m | \u001b[0m 0.08475 \u001b[0m | \u001b[0m 0.08781 \u001b[0m | \u001b[0m 0.1074  \u001b[0m | \u001b[0m 0.4269  \u001b[0m |\n",
      "| \u001b[0m 9       \u001b[0m | \u001b[0m-3.887   \u001b[0m | \u001b[0m 0.478   \u001b[0m | \u001b[0m 0.05332 \u001b[0m | \u001b[0m 0.695   \u001b[0m | \u001b[0m 0.3224  \u001b[0m |\n",
      "| \u001b[0m 10      \u001b[0m | \u001b[0m-1.326   \u001b[0m | \u001b[0m 0.3426  \u001b[0m | \u001b[0m 0.08346 \u001b[0m | \u001b[0m 0.02811 \u001b[0m | \u001b[0m 0.7526  \u001b[0m |\n",
      "| \u001b[95m 11      \u001b[0m | \u001b[95m-0.6959  \u001b[0m | \u001b[95m 0.1134  \u001b[0m | \u001b[95m 0.03688 \u001b[0m | \u001b[95m 0.09571 \u001b[0m | \u001b[95m 0.2889  \u001b[0m |\n",
      "| \u001b[0m 12      \u001b[0m | \u001b[0m-0.7513  \u001b[0m | \u001b[0m 0.1663  \u001b[0m | \u001b[0m 0.05954 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5531  \u001b[0m |\n",
      "| \u001b[0m 13      \u001b[0m | \u001b[0m-0.7432  \u001b[0m | \u001b[0m 0.2136  \u001b[0m | \u001b[0m 0.06988 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7171  \u001b[0m |\n",
      "| \u001b[95m 14      \u001b[0m | \u001b[95m-0.6135  \u001b[0m | \u001b[95m 0.0725  \u001b[0m | \u001b[95m 0.01088 \u001b[0m | \u001b[95m 0.1955  \u001b[0m | \u001b[95m 0.348   \u001b[0m |\n",
      "| \u001b[0m 15      \u001b[0m | \u001b[0m-2.54    \u001b[0m | \u001b[0m 0.06103 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.2221  \u001b[0m | \u001b[0m 0.2524  \u001b[0m |\n",
      "| \u001b[0m 16      \u001b[0m | \u001b[0m-2.147   \u001b[0m | \u001b[0m 0.146   \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.196   \u001b[0m | \u001b[0m 0.2765  \u001b[0m |\n",
      "| \u001b[0m 17      \u001b[0m | \u001b[0m-0.7381  \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.02252 \u001b[0m | \u001b[0m 0.1543  \u001b[0m | \u001b[0m 0.3223  \u001b[0m |\n",
      "| \u001b[0m 18      \u001b[0m | \u001b[0m-0.7914  \u001b[0m | \u001b[0m 0.1091  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.3841  \u001b[0m |\n",
      "| \u001b[0m 19      \u001b[0m | \u001b[0m-0.8339  \u001b[0m | \u001b[0m 0.1761  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.05201 \u001b[0m | \u001b[0m 0.2143  \u001b[0m |\n",
      "| \u001b[0m 20      \u001b[0m | \u001b[0m-0.7171  \u001b[0m | \u001b[0m 0.04743 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5462  \u001b[0m |\n",
      "| \u001b[0m 21      \u001b[0m | \u001b[0m-0.8135  \u001b[0m | \u001b[0m 0.1964  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.1017  \u001b[0m | \u001b[0m 0.3088  \u001b[0m |\n",
      "| \u001b[0m 22      \u001b[0m | \u001b[0m-2.062   \u001b[0m | \u001b[0m 0.08221 \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.625   \u001b[0m |\n",
      "| \u001b[0m 23      \u001b[0m | \u001b[0m-0.7804  \u001b[0m | \u001b[0m 0.2147  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.6256  \u001b[0m |\n",
      "| \u001b[0m 24      \u001b[0m | \u001b[0m-0.7681  \u001b[0m | \u001b[0m 0.2156  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.4521  \u001b[0m |\n",
      "| \u001b[0m 25      \u001b[0m | \u001b[0m-1.104   \u001b[0m | \u001b[0m 0.198   \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.8475  \u001b[0m |\n",
      "| \u001b[0m 26      \u001b[0m | \u001b[0m-1.965   \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.2578  \u001b[0m | \u001b[0m 0.3265  \u001b[0m |\n",
      "| \u001b[0m 27      \u001b[0m | \u001b[0m-0.7061  \u001b[0m | \u001b[0m 0.06073 \u001b[0m | \u001b[0m 0.07746 \u001b[0m | \u001b[0m 0.1525  \u001b[0m | \u001b[0m 0.3411  \u001b[0m |\n",
      "| \u001b[0m 28      \u001b[0m | \u001b[0m-0.9526  \u001b[0m | \u001b[0m 0.03289 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7959  \u001b[0m |\n",
      "| \u001b[0m 29      \u001b[0m | \u001b[0m-4.137   \u001b[0m | \u001b[0m 0.09628 \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.1711  \u001b[0m |\n",
      "| \u001b[0m 30      \u001b[0m | \u001b[0m-0.7753  \u001b[0m | \u001b[0m 0.123   \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.4913  \u001b[0m |\n",
      "| \u001b[0m 31      \u001b[0m | \u001b[0m-0.8205  \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.03434 \u001b[0m | \u001b[0m 0.436   \u001b[0m |\n",
      "| \u001b[0m 32      \u001b[0m | \u001b[0m-9.352   \u001b[0m | \u001b[0m 0.02033 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.9772  \u001b[0m |\n",
      "| \u001b[0m 33      \u001b[0m | \u001b[0m-0.7162  \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7018  \u001b[0m |\n",
      "| \u001b[0m 34      \u001b[0m | \u001b[0m-0.8192  \u001b[0m | \u001b[0m 0.1169  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7553  \u001b[0m |\n",
      "| \u001b[0m 35      \u001b[0m | \u001b[0m-0.7325  \u001b[0m | \u001b[0m 0.2736  \u001b[0m | \u001b[0m 0.008006\u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.8185  \u001b[0m |\n",
      "| \u001b[0m 36      \u001b[0m | \u001b[0m-0.9268  \u001b[0m | \u001b[0m 0.3194  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.8683  \u001b[0m |\n",
      "| \u001b[0m 37      \u001b[0m | \u001b[0m-3.931   \u001b[0m | \u001b[0m 0.1837  \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.4149  \u001b[0m |\n",
      "| \u001b[0m 38      \u001b[0m | \u001b[0m-0.8401  \u001b[0m | \u001b[0m 0.2746  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.04136 \u001b[0m | \u001b[0m 0.2618  \u001b[0m |\n",
      "| \u001b[0m 39      \u001b[0m | \u001b[0m-12.39   \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.1077  \u001b[0m | \u001b[0m 0.548   \u001b[0m |\n",
      "| \u001b[0m 40      \u001b[0m | \u001b[0m-0.8017  \u001b[0m | \u001b[0m 0.05593 \u001b[0m | \u001b[0m 0.0818  \u001b[0m | \u001b[0m 0.06634 \u001b[0m | \u001b[0m 0.3743  \u001b[0m |\n",
      "| \u001b[0m 41      \u001b[0m | \u001b[0m-0.8975  \u001b[0m | \u001b[0m 0.2093  \u001b[0m | \u001b[0m 0.06154 \u001b[0m | \u001b[0m 0.07462 \u001b[0m | \u001b[0m 0.2575  \u001b[0m |\n",
      "| \u001b[0m 42      \u001b[0m | \u001b[0m-0.7133  \u001b[0m | \u001b[0m 0.1123  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5779  \u001b[0m |\n",
      "| \u001b[0m 43      \u001b[0m | \u001b[0m-0.9182  \u001b[0m | \u001b[0m 0.1271  \u001b[0m | \u001b[0m 0.09294 \u001b[0m | \u001b[0m 0.0901  \u001b[0m | \u001b[0m 0.3582  \u001b[0m |\n",
      "| \u001b[0m 44      \u001b[0m | \u001b[0m-0.7679  \u001b[0m | \u001b[0m 0.05823 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.4416  \u001b[0m |\n",
      "| \u001b[0m 45      \u001b[0m | \u001b[0m-0.6978  \u001b[0m | \u001b[0m 0.06482 \u001b[0m | \u001b[0m 0.01815 \u001b[0m | \u001b[0m 0.1409  \u001b[0m | \u001b[0m 0.3176  \u001b[0m |\n",
      "| \u001b[0m 46      \u001b[0m | \u001b[0m-0.7786  \u001b[0m | \u001b[0m 0.2628  \u001b[0m | \u001b[0m 0.07725 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7986  \u001b[0m |\n",
      "| \u001b[0m 47      \u001b[0m | \u001b[0m-0.8383  \u001b[0m | \u001b[0m 0.1519  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.04377 \u001b[0m | \u001b[0m 0.2901  \u001b[0m |\n",
      "| \u001b[0m 48      \u001b[0m | \u001b[0m-0.7278  \u001b[0m | \u001b[0m 0.05433 \u001b[0m | \u001b[0m 0.07369 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7285  \u001b[0m |\n",
      "| \u001b[0m 49      \u001b[0m | \u001b[0m-0.7894  \u001b[0m | \u001b[0m 0.1605  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.6752  \u001b[0m |\n",
      "| \u001b[0m 50      \u001b[0m | \u001b[0m-0.9531  \u001b[0m | \u001b[0m 0.153   \u001b[0m | \u001b[0m 0.09545 \u001b[0m | \u001b[0m 0.1186  \u001b[0m | \u001b[0m 0.2479  \u001b[0m |\n",
      "| \u001b[0m 51      \u001b[0m | \u001b[0m-0.7539  \u001b[0m | \u001b[0m 0.2158  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.531   \u001b[0m |\n",
      "| \u001b[0m 52      \u001b[0m | \u001b[0m-0.7346  \u001b[0m | \u001b[0m 0.1948  \u001b[0m | \u001b[0m 0.03219 \u001b[0m | \u001b[0m 0.02229 \u001b[0m | \u001b[0m 0.7862  \u001b[0m |\n",
      "| \u001b[0m 53      \u001b[0m | \u001b[0m-0.8059  \u001b[0m | \u001b[0m 0.00137 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.3796  \u001b[0m |\n",
      "| \u001b[0m 54      \u001b[0m | \u001b[0m-0.7489  \u001b[0m | \u001b[0m 0.1714  \u001b[0m | \u001b[0m 0.08135 \u001b[0m | \u001b[0m 0.07052 \u001b[0m | \u001b[0m 0.7393  \u001b[0m |\n",
      "| \u001b[0m 55      \u001b[0m | \u001b[0m-0.74    \u001b[0m | \u001b[0m 0.2507  \u001b[0m | \u001b[0m 0.05027 \u001b[0m | \u001b[0m 0.07352 \u001b[0m | \u001b[0m 0.8238  \u001b[0m |\n",
      "| \u001b[0m 56      \u001b[0m | \u001b[0m-0.8957  \u001b[0m | \u001b[0m 0.2548  \u001b[0m | \u001b[0m 0.02342 \u001b[0m | \u001b[0m 0.05878 \u001b[0m | \u001b[0m 0.7483  \u001b[0m |\n",
      "| \u001b[0m 57      \u001b[0m | \u001b[0m-0.9771  \u001b[0m | \u001b[0m 0.2542  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.04361 \u001b[0m | \u001b[0m 0.3478  \u001b[0m |\n",
      "| \u001b[0m 58      \u001b[0m | \u001b[0m-1.054   \u001b[0m | \u001b[0m 0.1929  \u001b[0m | \u001b[0m 0.04328 \u001b[0m | \u001b[0m 0.05706 \u001b[0m | \u001b[0m 0.6573  \u001b[0m |\n",
      "| \u001b[0m 59      \u001b[0m | \u001b[0m-0.8613  \u001b[0m | \u001b[0m 0.2447  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.1035  \u001b[0m | \u001b[0m 0.2016  \u001b[0m |\n",
      "| \u001b[0m 60      \u001b[0m | \u001b[0m-0.7296  \u001b[0m | \u001b[0m 0.1765  \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.08366 \u001b[0m | \u001b[0m 0.4508  \u001b[0m |\n",
      "| \u001b[0m 61      \u001b[0m | \u001b[0m-19.72   \u001b[0m | \u001b[0m 0.06574 \u001b[0m | \u001b[0m 0.1     \u001b[0m | \u001b[0m 0.07542 \u001b[0m | \u001b[0m 0.7539  \u001b[0m |\n",
      "| \u001b[0m 62      \u001b[0m | \u001b[0m-1.15    \u001b[0m | \u001b[0m 0.244   \u001b[0m | \u001b[0m 0.04002 \u001b[0m | \u001b[0m 0.03453 \u001b[0m | \u001b[0m 0.7953  \u001b[0m |\n",
      "| \u001b[0m 63      \u001b[0m | \u001b[0m-0.8319  \u001b[0m | \u001b[0m 0.04624 \u001b[0m | \u001b[0m 0.03423 \u001b[0m | \u001b[0m 0.1687  \u001b[0m | \u001b[0m 0.3371  \u001b[0m |\n",
      "| \u001b[0m 64      \u001b[0m | \u001b[0m-0.9172  \u001b[0m | \u001b[0m 0.17    \u001b[0m | \u001b[0m 0.0789  \u001b[0m | \u001b[0m 0.02461 \u001b[0m | \u001b[0m 0.7283  \u001b[0m |\n",
      "| \u001b[0m 65      \u001b[0m | \u001b[0m-0.8027  \u001b[0m | \u001b[0m 0.2025  \u001b[0m | \u001b[0m 0.0905  \u001b[0m | \u001b[0m 0.05269 \u001b[0m | \u001b[0m 0.2927  \u001b[0m |\n",
      "| \u001b[0m 66      \u001b[0m | \u001b[0m-0.8631  \u001b[0m | \u001b[0m 0.1643  \u001b[0m | \u001b[0m 0.07876 \u001b[0m | \u001b[0m 0.08391 \u001b[0m | \u001b[0m 0.2797  \u001b[0m |\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| \u001b[0m 67      \u001b[0m | \u001b[0m-0.8868  \u001b[0m | \u001b[0m 0.2057  \u001b[0m | \u001b[0m 0.05789 \u001b[0m | \u001b[0m 0.04541 \u001b[0m | \u001b[0m 0.7491  \u001b[0m |\n",
      "| \u001b[0m 68      \u001b[0m | \u001b[0m-0.9341  \u001b[0m | \u001b[0m 0.07491 \u001b[0m | \u001b[0m 0.09614 \u001b[0m | \u001b[0m 0.04223 \u001b[0m | \u001b[0m 0.4075  \u001b[0m |\n",
      "| \u001b[0m 69      \u001b[0m | \u001b[0m-0.8228  \u001b[0m | \u001b[0m 0.1993  \u001b[0m | \u001b[0m 0.09549 \u001b[0m | \u001b[0m 0.09386 \u001b[0m | \u001b[0m 0.2339  \u001b[0m |\n",
      "| \u001b[0m 70      \u001b[0m | \u001b[0m-0.846   \u001b[0m | \u001b[0m 0.08722 \u001b[0m | \u001b[0m 0.05639 \u001b[0m | \u001b[0m 0.1184  \u001b[0m | \u001b[0m 0.3288  \u001b[0m |\n",
      "| \u001b[0m 71      \u001b[0m | \u001b[0m-0.7841  \u001b[0m | \u001b[0m 0.03058 \u001b[0m | \u001b[0m 0.09174 \u001b[0m | \u001b[0m 0.02614 \u001b[0m | \u001b[0m 0.406   \u001b[0m |\n",
      "| \u001b[0m 72      \u001b[0m | \u001b[0m-0.7823  \u001b[0m | \u001b[0m 0.1474  \u001b[0m | \u001b[0m 0.09688 \u001b[0m | \u001b[0m 0.01154 \u001b[0m | \u001b[0m 0.5372  \u001b[0m |\n",
      "| \u001b[0m 73      \u001b[0m | \u001b[0m-0.8549  \u001b[0m | \u001b[0m 0.182   \u001b[0m | \u001b[0m 0.09591 \u001b[0m | \u001b[0m 0.02286 \u001b[0m | \u001b[0m 0.4926  \u001b[0m |\n",
      "| \u001b[0m 74      \u001b[0m | \u001b[0m-0.7629  \u001b[0m | \u001b[0m 0.09285 \u001b[0m | \u001b[0m 0.09487 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5355  \u001b[0m |\n",
      "| \u001b[0m 75      \u001b[0m | \u001b[0m-1.005   \u001b[0m | \u001b[0m 0.1929  \u001b[0m | \u001b[0m 0.08071 \u001b[0m | \u001b[0m 0.04537 \u001b[0m | \u001b[0m 0.6924  \u001b[0m |\n",
      "| \u001b[0m 76      \u001b[0m | \u001b[0m-0.7259  \u001b[0m | \u001b[0m 0.1224  \u001b[0m | \u001b[0m 0.07941 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.6986  \u001b[0m |\n",
      "| \u001b[0m 77      \u001b[0m | \u001b[0m-0.768   \u001b[0m | \u001b[0m 0.04431 \u001b[0m | \u001b[0m 0.06843 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.6921  \u001b[0m |\n",
      "| \u001b[0m 78      \u001b[0m | \u001b[0m-0.8612  \u001b[0m | \u001b[0m 0.09061 \u001b[0m | \u001b[0m 0.03981 \u001b[0m | \u001b[0m 0.1681  \u001b[0m | \u001b[0m 0.3335  \u001b[0m |\n",
      "| \u001b[0m 79      \u001b[0m | \u001b[0m-0.8015  \u001b[0m | \u001b[0m 0.131   \u001b[0m | \u001b[0m 0.07947 \u001b[0m | \u001b[0m 0.02107 \u001b[0m | \u001b[0m 0.6457  \u001b[0m |\n",
      "| \u001b[0m 80      \u001b[0m | \u001b[0m-0.7253  \u001b[0m | \u001b[0m 0.172   \u001b[0m | \u001b[0m 0.09186 \u001b[0m | \u001b[0m 0.01376 \u001b[0m | \u001b[0m 0.5908  \u001b[0m |\n",
      "| \u001b[0m 81      \u001b[0m | \u001b[0m-0.7177  \u001b[0m | \u001b[0m 0.0787  \u001b[0m | \u001b[0m 0.08013 \u001b[0m | \u001b[0m 0.1138  \u001b[0m | \u001b[0m 0.3766  \u001b[0m |\n",
      "| \u001b[0m 82      \u001b[0m | \u001b[0m-8.265   \u001b[0m | \u001b[0m 0.2801  \u001b[0m | \u001b[0m 0.05692 \u001b[0m | \u001b[0m 0.0281  \u001b[0m | \u001b[0m 0.8365  \u001b[0m |\n",
      "| \u001b[0m 83      \u001b[0m | \u001b[0m-0.8026  \u001b[0m | \u001b[0m 0.2406  \u001b[0m | \u001b[0m 0.07305 \u001b[0m | \u001b[0m 0.01112 \u001b[0m | \u001b[0m 0.7742  \u001b[0m |\n",
      "| \u001b[0m 84      \u001b[0m | \u001b[0m-0.8939  \u001b[0m | \u001b[0m 0.1819  \u001b[0m | \u001b[0m 0.09592 \u001b[0m | \u001b[0m 0.06157 \u001b[0m | \u001b[0m 0.2557  \u001b[0m |\n",
      "| \u001b[0m 85      \u001b[0m | \u001b[0m-0.7485  \u001b[0m | \u001b[0m 0.2304  \u001b[0m | \u001b[0m 0.04564 \u001b[0m | \u001b[0m 0.07614 \u001b[0m | \u001b[0m 0.7973  \u001b[0m |\n",
      "| \u001b[0m 86      \u001b[0m | \u001b[0m-0.8307  \u001b[0m | \u001b[0m 0.1458  \u001b[0m | \u001b[0m 0.07532 \u001b[0m | \u001b[0m 0.03589 \u001b[0m | \u001b[0m 0.6804  \u001b[0m |\n",
      "| \u001b[0m 87      \u001b[0m | \u001b[0m-0.8133  \u001b[0m | \u001b[0m 0.06522 \u001b[0m | \u001b[0m 0.04467 \u001b[0m | \u001b[0m 0.1369  \u001b[0m | \u001b[0m 0.3532  \u001b[0m |\n",
      "| \u001b[0m 88      \u001b[0m | \u001b[0m-0.7696  \u001b[0m | \u001b[0m 0.1853  \u001b[0m | \u001b[0m 0.09132 \u001b[0m | \u001b[0m 0.01964 \u001b[0m | \u001b[0m 0.5512  \u001b[0m |\n",
      "| \u001b[0m 89      \u001b[0m | \u001b[0m-0.9168  \u001b[0m | \u001b[0m 0.1793  \u001b[0m | \u001b[0m 0.08204 \u001b[0m | \u001b[0m 0.02261 \u001b[0m | \u001b[0m 0.6398  \u001b[0m |\n",
      "| \u001b[0m 90      \u001b[0m | \u001b[0m-2.326   \u001b[0m | \u001b[0m 0.2554  \u001b[0m | \u001b[0m 0.0     \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7926  \u001b[0m |\n",
      "| \u001b[0m 91      \u001b[0m | \u001b[0m-0.7412  \u001b[0m | \u001b[0m 0.09333 \u001b[0m | \u001b[0m 0.08684 \u001b[0m | \u001b[0m 0.08057 \u001b[0m | \u001b[0m 0.3895  \u001b[0m |\n",
      "| \u001b[0m 92      \u001b[0m | \u001b[0m-0.8652  \u001b[0m | \u001b[0m 0.1411  \u001b[0m | \u001b[0m 0.07956 \u001b[0m | \u001b[0m 0.02371 \u001b[0m | \u001b[0m 0.5707  \u001b[0m |\n",
      "| \u001b[0m 93      \u001b[0m | \u001b[0m-9.351   \u001b[0m | \u001b[0m 0.2104  \u001b[0m | \u001b[0m 0.0653  \u001b[0m | \u001b[0m 0.03944 \u001b[0m | \u001b[0m 0.7927  \u001b[0m |\n",
      "| \u001b[0m 94      \u001b[0m | \u001b[0m-0.7227  \u001b[0m | \u001b[0m 0.06776 \u001b[0m | \u001b[0m 0.04739 \u001b[0m | \u001b[0m 0.1483  \u001b[0m | \u001b[0m 0.3276  \u001b[0m |\n",
      "| \u001b[0m 95      \u001b[0m | \u001b[0m-1.011   \u001b[0m | \u001b[0m 0.1971  \u001b[0m | \u001b[0m 0.06592 \u001b[0m | \u001b[0m 0.03898 \u001b[0m | \u001b[0m 0.7251  \u001b[0m |\n",
      "| \u001b[0m 96      \u001b[0m | \u001b[0m-0.6726  \u001b[0m | \u001b[0m 0.07044 \u001b[0m | \u001b[0m 0.01908 \u001b[0m | \u001b[0m 0.1653  \u001b[0m | \u001b[0m 0.3379  \u001b[0m |\n",
      "| \u001b[0m 97      \u001b[0m | \u001b[0m-0.8342  \u001b[0m | \u001b[0m 0.1291  \u001b[0m | \u001b[0m 0.09084 \u001b[0m | \u001b[0m 0.01889 \u001b[0m | \u001b[0m 0.6739  \u001b[0m |\n",
      "| \u001b[0m 98      \u001b[0m | \u001b[0m-0.727   \u001b[0m | \u001b[0m 0.1621  \u001b[0m | \u001b[0m 0.08758 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5628  \u001b[0m |\n",
      "| \u001b[0m 99      \u001b[0m | \u001b[0m-0.973   \u001b[0m | \u001b[0m 0.2498  \u001b[0m | \u001b[0m 0.03354 \u001b[0m | \u001b[0m 0.07747 \u001b[0m | \u001b[0m 0.802   \u001b[0m |\n",
      "| \u001b[0m 100     \u001b[0m | \u001b[0m-0.862   \u001b[0m | \u001b[0m 0.1952  \u001b[0m | \u001b[0m 0.08825 \u001b[0m | \u001b[0m 0.08235 \u001b[0m | \u001b[0m 0.2771  \u001b[0m |\n",
      "| \u001b[0m 101     \u001b[0m | \u001b[0m-0.7882  \u001b[0m | \u001b[0m 0.2641  \u001b[0m | \u001b[0m 0.06937 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.7737  \u001b[0m |\n",
      "| \u001b[0m 102     \u001b[0m | \u001b[0m-0.686   \u001b[0m | \u001b[0m 0.1895  \u001b[0m | \u001b[0m 0.01591 \u001b[0m | \u001b[0m 0.01557 \u001b[0m | \u001b[0m 0.7756  \u001b[0m |\n",
      "| \u001b[0m 103     \u001b[0m | \u001b[0m-0.7315  \u001b[0m | \u001b[0m 0.1417  \u001b[0m | \u001b[0m 0.08417 \u001b[0m | \u001b[0m 0.0213  \u001b[0m | \u001b[0m 0.5701  \u001b[0m |\n",
      "| \u001b[0m 104     \u001b[0m | \u001b[0m-0.7239  \u001b[0m | \u001b[0m 0.06779 \u001b[0m | \u001b[0m 0.08427 \u001b[0m | \u001b[0m 0.09106 \u001b[0m | \u001b[0m 0.3971  \u001b[0m |\n",
      "| \u001b[0m 105     \u001b[0m | \u001b[0m-0.7397  \u001b[0m | \u001b[0m 0.1199  \u001b[0m | \u001b[0m 0.0897  \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5509  \u001b[0m |\n",
      "| \u001b[0m 106     \u001b[0m | \u001b[0m-0.7961  \u001b[0m | \u001b[0m 0.08417 \u001b[0m | \u001b[0m 0.06903 \u001b[0m | \u001b[0m 0.1387  \u001b[0m | \u001b[0m 0.3502  \u001b[0m |\n",
      "| \u001b[0m 107     \u001b[0m | \u001b[0m-0.7934  \u001b[0m | \u001b[0m 0.1753  \u001b[0m | \u001b[0m 0.09946 \u001b[0m | \u001b[0m 0.06797 \u001b[0m | \u001b[0m 0.2961  \u001b[0m |\n",
      "| \u001b[0m 108     \u001b[0m | \u001b[0m-0.7523  \u001b[0m | \u001b[0m 0.1808  \u001b[0m | \u001b[0m 0.08483 \u001b[0m | \u001b[0m 0.01    \u001b[0m | \u001b[0m 0.5235  \u001b[0m |\n",
      "| \u001b[0m 109     \u001b[0m | \u001b[0m-0.701   \u001b[0m | \u001b[0m 0.08097 \u001b[0m | \u001b[0m 0.07741 \u001b[0m | \u001b[0m 0.08763 \u001b[0m | \u001b[0m 0.3628  \u001b[0m |\n",
      "| \u001b[0m 110     \u001b[0m | \u001b[0m-0.7594  \u001b[0m | \u001b[0m 0.2368  \u001b[0m | \u001b[0m 0.04896 \u001b[0m | \u001b[0m 0.0952  \u001b[0m | \u001b[0m 0.8146  \u001b[0m |\n",
      "=========================================================================\n",
      "Total runtime: 3:31:16.15\n",
      "{'target': -0.6134732591234944, 'params': {'dropout': 0.07250303494641018, 'lr': 0.010879552981584925, 'neuronPct': 0.19549892758598567, 'neuronShrink': 0.3480265521049257}}\n"
     ]
    }
   ],
   "source": [
    "from bayes_opt import BayesianOptimization\n",
    "import time\n",
    "\n",
    "# Supress NaN warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\",category =RuntimeWarning)\n",
    "\n",
    "# Bounded region of parameter space\n",
    "pbounds = {'dropout': (0.0, 0.499),\n",
    "           'lr': (0.0, 0.1),\n",
    "           'neuronPct': (0.01, 1),\n",
    "           'neuronShrink': (0.01, 1)\n",
    "          }\n",
    "\n",
    "optimizer = BayesianOptimization(\n",
    "    f=evaluate_network,\n",
    "    pbounds=pbounds,\n",
    "    verbose=2,  # verbose = 1 prints only when a maximum \n",
    "    # is observed, verbose = 0 is silent\n",
    "    random_state=1,\n",
    ")\n",
    "\n",
    "start_time = time.time()\n",
    "optimizer.maximize(init_points=10, n_iter=100,)\n",
    "time_took = time.time() - start_time\n",
    "\n",
    "print(f\"Total runtime: {hms_string(time_took)}\")\n",
    "print(optimizer.max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "{'target': -0.6500334282952827, 'params': {'dropout': 0.12771198428037775, 'lr': 0.0074010841641111965, 'neuronPct': 0.10774655638231533, 'neuronShrink': 0.2784788676498257}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}