notebooks/New_fitness_functions_and_acquisition_functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using a new function to evaluate or evaluating a new acquisition function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we describe how to integrate a new fitness function to the testing framework as well as how to integrate a new acquisition function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# add the egreedy module to the path (one directory up from this)\n",
    "import sys, os\n",
    "sys.path.append(os.path.realpath(os.path.pardir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New fitness function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `perform_experiment` function in the `optimizer` class, used to carry out the optimisation runs (see its docstring and `run_all_experiments.py` for usage examples), imports a fitness **class**. This class, when instantiated is also callable. The class is imported from the `test_problems` module. Therefore, the easiest way to incorporate your own fitness function is to add it to the `test_problems` module by creating a python file in the `egreedy/test_problems/` directory and adding a line importing it into the namespace (see `egreedy/test_problems/__init__.py` for examples) so that it can be directly imported from `test_problems`.\n",
    "\n",
    "If, for example, your fitness class is called `xSquared` and is located in the file `xs.py`, you would place the python file in the directory `egreedy/test_problems` and add the line:\n",
    "```\n",
    "from .xs import xSquared\n",
    "```\n",
    "to the `egreedy/test_problems/__init__.py` file.\n",
    "\n",
    "We will now detail how to structure your fitness class and show the required class methods by creating a new fitness class for the function\n",
    "\\begin{equation}\n",
    "f( \\mathbf{x} ) = \\sum_{i=1}^2 x_i^2,\n",
    "\\end{equation}\n",
    "where $\\mathbf{x} \\in [-5, 5]^2.$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class xSquared:\n",
    "    \"\"\"Example fitness class.\n",
    "    \n",
    "    .. math::\n",
    "        f(x) = \\sum_{i=1}^2 x_i^2\n",
    "        \n",
    "    This demonstration class shows all the required attributes and \n",
    "    functionality of the fitness function class.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        \"\"\"Initialisation function.\n",
    "        \n",
    "        This is called when the class is instantiated and sets up its\n",
    "        attributes as well as any other internal variables that may\n",
    "        be needed.\n",
    "        \"\"\"\n",
    "        # problem dimensionality\n",
    "        self.dim = 2\n",
    "        \n",
    "        # lower and upper bounds for each dimension (must be numpy.ndarray)\n",
    "        self.lb = np.array([-5., -5.])\n",
    "        self.ub = np.array([5., 5.])\n",
    "        \n",
    "        # location(s) of the optima (optional - not always known)\n",
    "        self.xopt = np.array([0.])\n",
    "        \n",
    "        # its/thier fitness value(s)\n",
    "        self.yopt = np.array([0.])\n",
    "        \n",
    "        # callable constraint function for the problem - should return\n",
    "        # True if the argument value is **valid** - if no constraint function\n",
    "        # is required then this can take the value of None\n",
    "        self.cf = None\n",
    "        \n",
    "    def __call__(self, x):\n",
    "        \"\"\"Main callable function.\n",
    "        \n",
    "        This is called after the class is instantiated, e.g.\n",
    "        >>> f = xSquared()\n",
    "        >>> f(np.array([2., 2.]))\n",
    "        array([8.])\n",
    "        \n",
    "        Note that it is useful to have a function that is able deal with\n",
    "        multiple inputs, which should a numpy.ndarray of shape (N, dim)\n",
    "        \"\"\"\n",
    "        # ensure the input is at least 2d, this will cause one-dimensional\n",
    "        # vectors to be reshaped to shape (1, dim)\n",
    "        x = np.atleast_2d(x)\n",
    "        \n",
    "        # evaluate the function\n",
    "        val = np.sum(np.square(x), axis=1)\n",
    "        \n",
    "        # return the evaluations\n",
    "        return val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This class can then either be placed in the directories discussed above and used for evaluating multiple techniques on it or used for testing purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimising the new test function with an acquistion function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code outlines how to optimise your newly created test function with the $\\epsilon$-greedy with Pareto front selection ($\\epsilon$-PF) algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training a GP model with 4 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [2.74739209 1.85140059]\n",
      "Function value: 10.9758\n",
      "Best function value so far: 2.34029\n",
      "\n",
      "Training a GP model with 5 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.8734869  0.23262576]\n",
      "Function value: 0.817094\n",
      "Best function value so far: 0.817094\n",
      "\n",
      "Training a GP model with 6 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.48106594 0.1616895 ]\n",
      "Function value: 0.257568\n",
      "Best function value so far: 0.257568\n",
      "\n",
      "Training a GP model with 7 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.18353004 0.31916069]\n",
      "Function value: 0.135547\n",
      "Best function value so far: 0.135547\n",
      "\n",
      "Training a GP model with 8 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.13640659 0.16686483]\n",
      "Function value: 0.0464506\n",
      "Best function value so far: 0.0464506\n",
      "\n",
      "Training a GP model with 9 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.01887471 0.03552285]\n",
      "Function value: 0.00161813\n",
      "Best function value so far: 0.00161813\n",
      "\n",
      "Training a GP model with 10 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.00498028  0.00452991]\n",
      "Function value: 4.53233e-05\n",
      "Best function value so far: 4.53233e-05\n",
      "\n",
      "Training a GP model with 11 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.00296791  0.01886975]\n",
      "Function value: 0.000364876\n",
      "Best function value so far: 4.53233e-05\n",
      "\n",
      "Training a GP model with 12 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.00228287 -0.0035852 ]\n",
      "Function value: 1.80651e-05\n",
      "Best function value so far: 1.80651e-05\n",
      "\n",
      "Training a GP model with 13 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.00136503  0.01035887]\n",
      "Function value: 0.00010917\n",
      "Best function value so far: 1.80651e-05\n",
      "\n",
      "Training a GP model with 14 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.00031969 -0.00034095]\n",
      "Function value: 2.18449e-07\n",
      "Best function value so far: 2.18449e-07\n",
      "\n",
      "Training a GP model with 15 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.0022843  0.00156171]\n",
      "Function value: 7.65699e-06\n",
      "Best function value so far: 2.18449e-07\n",
      "\n",
      "Training a GP model with 16 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.00435965 0.00438578]\n",
      "Function value: 3.82417e-05\n",
      "Best function value so far: 2.18449e-07\n",
      "\n",
      "Training a GP model with 17 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [-0.0038385  -0.01013353]\n",
      "Function value: 0.000117422\n",
      "Best function value so far: 2.18449e-07\n",
      "\n",
      "Training a GP model with 18 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.01889642 0.02033263]\n",
      "Function value: 0.000770491\n",
      "Best function value so far: 2.18449e-07\n",
      "\n",
      "Training a GP model with 19 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [ 0.00050108 -0.02413065]\n",
      "Function value: 0.000582539\n",
      "Best function value so far: 2.18449e-07\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyDOE2 import lhs\n",
    "from egreedy.optimizer import perform_BO_iteration\n",
    "\n",
    "# ---- instantiate the test problem\n",
    "f = xSquared()\n",
    "\n",
    "# ---- Generate testing data by Latin hypercube sampling across the domain\n",
    "n_training = 2 * f.dim\n",
    "\n",
    "# LHS sample in [0, 1]^2 and rescale to problem domain\n",
    "Xtr = lhs(f.dim, n_training, criterion='maximin')\n",
    "Xtr = (f.ub - f.lb) * Xtr + f.lb\n",
    "\n",
    "# expensively evaluate and ensure shape is (n_training, 1)\n",
    "Ytr = np.reshape(f(Xtr), (n_training, 1))\n",
    "\n",
    "# ---- Select an acquistion function with optimiser.\n",
    "#      In this case we select e-greedy with Pareto front selection (e-PF)\n",
    "#      known as eFront.\n",
    "#\n",
    "#      All the acqusition functions have the same parameters:\n",
    "#      lb : lower-bound constraints (numpy.ndarray)\n",
    "#      ub : upper-bound constraints (numpy.ndarray)\n",
    "#      acq_budget : max number of calls to the GP model\n",
    "#      cf : callable function constraint function that returns True if\n",
    "#           the argument vector is VALID. Optional, has a value of None\n",
    "#           if not used\n",
    "#      acquisition_args : optional dictionary containing key:value pairs\n",
    "#                         of arguments to a specific acqutision function.\n",
    "#                         e.g. for an e-greedy method then the dict\n",
    "#                        {'epsilon': 0.1} would dictate the epsilon value.\n",
    "\n",
    "#      e-greedy with Pareto front selection (e-PF), known as eFront\n",
    "from egreedy.acquisition_functions import eFront\n",
    "\n",
    "# instantiate the optimiser with a budget of 5000d and epsilon = 0.1\n",
    "acq_budget = 5000 * f.dim\n",
    "acquisition_args = {'epsilon': 0.1}\n",
    "\n",
    "acq_func = eFront(lb=f.lb, ub=f.ub, cf=None, acq_budget=acq_budget,\n",
    "                  acquisition_args=acquisition_args)\n",
    "\n",
    "# ---- Perform the bayesian optimisation loop for a total budget of 20\n",
    "#      function evaluations (including those used for LHS sampling)\n",
    "total_budget = 20\n",
    "\n",
    "while Xtr.shape[0] < total_budget:\n",
    "    # perform one iteration of BO:\n",
    "    # this returns the new location and function value (Xtr, Ynew) as well\n",
    "    # as the trained model used to select the new location\n",
    "    Xnew, Ynew, model = perform_BO_iteration(Xtr, Ytr,f, acq_func, verbose=True)\n",
    "    \n",
    "    # augment the training data and repeat\n",
    "    Xtr = np.concatenate((Xtr, np.atleast_2d(Xnew)))\n",
    "    Ytr = np.concatenate((Ytr, np.atleast_2d(Ynew)))\n",
    "    \n",
    "    print('Best function value so far: {:g}'.format(np.min(Ytr)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot below shows the difference between the best seen function value and the true minimum, i.e. $|f^\\star - f_{min}|$, over each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAELCAYAAACmvJ7eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnG4QdZA9rEJBFFBoQlU1BBCpSrVXR/twQtLVerb320nqvtlq7aG179YfXolLF61Vra10RcAVEVIICEtn3NSxhkwAh4fP7Y4be/GKGLExyZnk/H488MnPmzMn7MEl458z3fI+5OyIiIpLcUoIOICIiIsFTIRAREREVAhEREVEhEBEREVQIREREBEgLOkCQmjdv7p06dQo6hoiISK1YtGjRbndvUd5jSV0IOnXqRG5ubtAxREREaoWZbYz0mN4yEBERERUCERERUSEQERERVAhEREQEFQIRERFBhUBERERQIRARERGSfB6CaPpw5U4+37g36BiVNqJnK/q0axJ0DBERiREqBFHy8do9PDlvXdAxKsUdnv5oPS/feh492zYKOo6IiMQAc/egMwQmJyfHk3Gmwh37j3DZ4/Nxh3/cdh5tGmcGHUlERGqBmS1y95zyHtMYgiTUunFdpt3Qn6+PFnPTM7l8fbQ46EgiIhIwFYIk1aNNI6Zc249V+Qe57fnPKS45HnQkEREJkApBEhvarQUPjOvNnFW7uPf1PJL57SMRkWSnQYVJ7ppzOrCpoJAn5qylY7N63DK0S9CRREQkACoEwk8v7s7mvYX85u0VtGtaj2/3aRN0JBERqWUqBEJKivHI985ix/4j/Pivi2nduC7f6tg06FgiIlKLNIZAAKibnsqT1+XQtnFdJk7PZeOeQ0FHEhGRWqRCIP/UrH4Gf7lxAMfdufEvC9l7qCjoSCIiUktUCOT/07l5fZ68Loctew9zy3OLOFpcEnQkERGpBSoE8g39OzXj4e/14bMNBfz0b0t1OqKISBLQoEIp17izs9iy9zAPz1pJh2b1+MnI7kFHEhGRGqRCIBH9cFgXNu0p5LH319C+WT2uzGkfdCQREakhKgQSkZnxq8t6s23/YX7+ype0bZzJoK7Ng44lIiI1QGMI5KTSU1OYcm0/urRowA/+exErdxwMOpKIiNQAFQKpUKO66Uy7sT91M1K56ZmF7DxwJOhIIiISZSoEUilZTTKZdn1/Cg4VMeHZXAqLdMlkEZFEokIglXZmu8Y8Nr4vedv28y8vLKbkuE5HFBFJFCoEUiUjerbivrG9eHd5Pg+8+VXQcUREJEp0loFU2fXndWJTQSFPf7SeDs3qcdOgzkFHEhGRU6RCINXy8zE92LK3kAfe+op2TTMZ2at10JFEROQU6C0DqZbUFONPV/WlT1Zj/uXFL1iyeV/QkURE5BSoEEi1ZWak8tT1/WneoA4Tns1lc0Fh0JFERKSaLJkvXJOTk+O5ublBx4h7q/MPcvl/fUxGagotG9WN6rbTUoxL+rThunM7kZmRGtVti4gkGzNb5O455T6mQqBCEA2LNhbw1Lz1FEf5VMSCQ0Us2riXVo3qcPuFXbmqf3vSU3VgS0SkOlQIIlAhiA+frtvDw7NWkrtxLx1Pq8ddF3VjbJ+2pKRY0NFEROLKyQqB/tSSmHdO9mm8fOu5TLshh3oZadzx4mLGPDqP95bnk8yFVkQkmlQIJC6YGRee0Yq3bh/Eo+P7cuRYCROezeWKJxbwybo9QccTEYl7KgQSV1JSjEvPass7dw3l15edyZa9hVw99ROum/YZy7buDzqeiEjc0hgCjSGIa0eOlTB9wQYe/3At+wqPMebM1tx1UXdOb9kg6GgiIjEnKQYVmll94HGgCPjQ3Z+v6DkqBInjwJFjPDV3HU99tJ4jx0q44lvtuGNEN7KaZAYdTUQkZsTtoEIzm2ZmO81sWZnlo8xspZmtMbPJ4cWXA39z94nApbUeVgLVqG46d43sztyfXsAN53Xm1S+2ccHDH3L/G1+x++ujQccTEYl5MV0IgGeAUaUXmFkqMAUYDfQExptZT6AdsDm8WkktZpQY0rxBHe4d25MP7h7GZX2zeObj9Qx96AP+MHslB44cCzqeiEjMiulC4O5zgYIyiwcAa9x9nbsXAS8C44AthEoBnGS/zGySmeWaWe6uXbtqIrbEgKwmmfzuij68c9dQhnVvyaPvr2HIQx8wde5ajhxTXxQRKSumC0EEWfzvkQAIFYEs4BXgu2b2X8AbkZ7s7lPdPcfdc1q0aFGzSSVwXVo0YMq1/XjjR4Po064Jv56xgqEPf8CLn22iJMqzKoqIxLN4LATlTU/n7n7I3W909x9UZkChJJcz2zVm+k0DeHHSQLKaZDL5lS8Z+9hHfKo5DEREgPgsBFuA9qXutwO2BZRF4szA7NP4+w/O47HxfdlXWMRVUz/htv/5nC17daVGEUlu8VgIFgJdzayzmWUAVwOvB5xJ4oiZMfastrz3k2HcOaIr7y3PZ/gjc/jDO6soLCoOOp6ISCBiuhCY2QvAAqC7mW0xswnuXgz8CJgFLAf+6u55QeaU+JSZkcqdI7rx3k+GMbJXax59bzXDH5nDa4u36hoJIpJ0EmZiourQxERS2sINBfzyjTyWbT3Atzo25b6xPenTrknQsUREoiZuJyYSqU39OzXj9dsG8dB3+7BxzyHGTZnP3S8vYefBI0FHExGpcSoEIqWkpBhX9m/PB/86jEmDs3l18VYu/P0cnpizlqPFmr9ARBKXCoFIORrWTednY3ow+8dDGZh9Gr99ewUj/ziXd77K1/gCEUlIKgQiJ9G5eX2euj6H6TcNICM1hYnTc7lu2mesyj8YdDQRkahSIRCphCHdWjDjjsH8YmxPlmzex+j/nMd9ry1jX2FR0NFERKJChUCkktJTU7jh/M58ePcFXDOgA899spFhv/+Q6Qs2UFxyPOh4IiKnRIVApIqa1c/gge/0ZsYdg+nZphH3vpbHmEfnMWfVLo7r+ggiEqc0D4HmIZBT4O7M/iqfB99azqaCQlo1qsPFvVozqldrBnRuRlqqOreIxI6TzUOgQqBCIFFw5FgJM5ftYOayHXy4aidHjh2nab10LurZilG9W3P+6c2pk5YadEwRSXIqBBGoEEhNOFxUwpxVu5iVt4N3l+dz8EgxDeqkccEZLRnduzVDu7Wgfp20oGOKSBI6WSHQbyWRKMvMSGVU79aM6t2aouLjfLx2N7PydjA7L583lmyjTloKQ7q1YFSv1ozo0YrG9dKDjiwioiMEOkIgtaXkuLNwQwEzl+1gVt4Otu8/QlqKcW6X0xjVuzUX9WxFy4Z1g44pIglMbxlEoEIgQXF3lm7Zz9vLdjBz2XY27CnEDHI6NmVU7zZc3KsV7ZrWCzqmiCQYFYIIVAgkFrg7q/K/ZuayHby9bDsrdoRmQTwzqzGjerfmezntdORARKJChSACFQKJRRt2H2JW3g5m5u3gi037yExPZcKgzkwamk2juhpvICLVp0IQgQqBxLr1uw/xh3dW8caSbTStl85tF5zO9wd2pG66TmEUkapTIYhAhUDixZdb9vPQrBXMW72brCaZ/PiiblzWN4vUFAs6mojEkZMVAk2jJhIHzmzXmOcmnMPzN5/DaQ0y+NeXlzDmP+fxri7HLCJRokIgEkfOP705r912PlOu6UdRyXFunp7L955YQO6GgqCjiUicUyEQiTNmxrf7tGH2j4fw4GW92VhQyBVPLODmZ3NZlX8w6HgiEqc0hkBjCCTOFRYV85f5G3jiw7UcKirm8n7t+PFF3chqkhl0NBGJMRpUGIEKgSSSvYeKePzDNTy7YCMA1w3syG0XnE7T+hkBJxORWKFCEIEKgSSirfsO86d3VvH3z7dQPyONW4Zmc9OgztTL0KVLRJKdCkEEKgSSyFblH+ShmSt5d3k+LRrW4Y7hXbmqf3vSUzV0SCRZ6bRDkSTUrVVDnro+h7/dei6dTqvHv7+6jJF/nMubS7fpVEUR+YYKjxCYWYdT+QLuvulUnl+TdIRAkoW78/6KnTw0cyUr8w8yokdLnrwuBzNNbCSSTE52hKAybypuAKrz54SFn6c5VkUCZmYM79GKYd1b8vgHa3jknVW8sXQ7l57VNuhoIhIjKlMIOtd4ChGpFakpxg8vOJ3ZX+Xz4FtfceEZLWlQR4MNRaQShcDdN9ZGEBGpHakpxi/H9eLyxz/msfdX87PRPYKOJCIxQIMKRZJQvw5NuTKnHdM+Ws+anV8HHUdEYsApFwIz+7mZbTOzpWY23czuMrMLoxFORGrOT0edQWZ6Kr98I09nHYhIVI4Q3A6cDVwMvADUBW6NwnZFpAY1b1CHn4zszrzVu5mVtyPoOCISsGiMJlrm7jvDt7cDb0dhmyJSC649pwMvfLaJB95cztBuLcnM0ElBIskqGkcIvjCzP5iZrqQiEmfSUlO4f1xvtu47zOMfrgk6jogEKBqFoCkwCNhiZgvM7HEzmxSF7YpILRjQuRmX9c3iz3PWsWH3oaDjiEhATrkQuPtEdx8AtABuAuaiuQtE4srPRp9BRloK97/5VdBRRCQg1S4EZvbn8OfvmVlXdz/u7svd/UV3/1n0IopITWvZqC53jujK+yt28t7y/KDjiEgATuUIwR/Dn4cDz5lZvpnNN7MpZjYxCtlEpBZdf14nTm/ZgF++8RVHjpUEHUdEalm1C4G7rwh/vtXdBwKtgQnAPCA7OvFEpLakp6Zw/6W92FRQyNS564KOIyK1LBoTE3Uzs8nARKAJ8HpQbxmY2XfM7Ekze83MRgaRQSSenXd6c77dpw1TPljD5oLCoOOISC2KxlkGbwGNgS7AL4DVZrayqhsxs2lmttPMlpVZPsrMVprZmnDxiMjdX3X3icANwFVVzSAicM+YHqSY8au3NMBQJJlEY2Ki3WWPCJhZq2ps5xng/wLTS20nFZgCXARsARaa2euELqn8mzLPv6nUBEn/Hn6eiFRR2yaZ3D78dB6auZI5q3YxtFuLoCOJSC2IxhGCV8oennf3Kg9Tdve5QEGZxQOANe6+zt2LgBeBce7+pbtfUuZjp4X8Dnjb3T8v7+uY2SQzyzWz3F27dlU1pkhSmDCoM52b1+cXr+dxtFgDDEWSQTQKwU7gb2Y2y8zuM7NLzaxDFLYLkAVsLnV/S3hZJLcDI4ArzKzc6ym4+1R3z3H3nBYt9JePSHnqpKVy39ierN99iKc/Wh90HBGpBdEoBA8CY8OfC4DvAK9GYbsAVs6yiJdlc/dH3f1b4TMfnohSBpGkNKx7S0b2bMVj761h+/7DQccRkRoWjUKw0t3nuPtcd3/M3W9y935R2C6Ejgi0L3W/HbAtStsWkQr8xyU9Oe7Og28tDzqKiNSwCguBmTWrYJXVZvZIDV3caCHQ1cw6m1kGcDXweg18HREpR/tm9fjhsNN5c+l2Pl67O+g4IlKDKnOEYJeZ5QCYWesI65wPbD6VixuZ2QvAAqC7mW0xswnuXgz8CJgFLAf+6u55Vd22iFTfLUOzad8sk/tey+NYyfGg44hIDanMaYeHgYzw7a1mNtDdF5540N1vBTAzA7oDZwNnVTWIu4+PsHwGMKOq2xOR6Kibnsp9l/Ti5um5PPvxBm4erIlIRRJRZY4Q5AG3mFkLyh/kB4CHrNDFjUQSz/AeLbmgewv+9O5qdh44EnQcEakBlSkEdwOjgR2ERvj/3sx+Y2bjzaxXePIgEUlgZsZ9Y3tRVHyc37y9Iug4IlIDKiwE4QmDsgiVAgOOAKOAvwBLgUNmttjMppvZT2oyrIgEp1Pz+kwaks0/vtjKZ+vLziEmIvGuUqcduvsxd58NzAfudve+QH2gD3AjMBNoDtxZU0FFJHg/vKALbRvX5d7XllGsAYYiCaVK8xC4+2B3Xxq+XeLuee7+grtPdvcx7t6+om2ISPyql5HGf1zSkxU7DvL8p5uCjiMiURSNiYlEJImM6t2aQac355HZK9n99dGg44hIlKgQiEiVmBm/uLQnhUUlPDRTAwxFEoUKgYhU2ektGzJhUGf+mruFLzbtDTqOiESBCoGIVMvtw7vSqlEd7n0tj5LjEa85JiJxQoVARKqlQZ00fj6mB19u3c9LCzdX/AQRiWnVLgRm1t3MSqIZRkTiy6VntWVA52Y8NGsFew8VBR1HRE7BqR4hiDiVsYgkPjPj/nG9OHikmN/PXhl0HBE5BadaCPTGoUiSO6N1I647tyP/89kmlm3dH3QcEakmjSEQkVN254huNMlM59czluOuvxNE4pEKgYicssaZ6dwxvCsfr93D+yt2Bh1HRKpBhUBEouLagR3Jbl6fX89YzjFd50Ak7qgQiEhUpKemMHn0GazddYgXPtN1DkTijQqBiETNRT1bMTC7GX96dzUHjhwLOo6IVIFOOxSRqDEz/v3bPdlbWMSUD9YEHUdEquBUCsF2YGK0gohIYuid1ZjL+mbxl482sLmgMOg4IlJJ1S4E7n7A3Z+OZhgRSQx3X9ydlBR4aJYmKxKJFxpDICJR16ZxJhMHZ/PGkm26GqJInFAhEJEaccvQLjRvUIdfvaXJikTigQqBiNSIBnXS+MnIbizauJe3l+0IOo6IVECFQERqzJU57eneqiG/fXsFR4t1cVSRWKZCICI1JjXFuOfbPdhUUMhzCzYGHUdETkKFQERq1JBuLRjarQWPvreavYeKgo4jIhFUqhCYWUZNBxGRxHXPt3vw9dFi/vO91UFHEZEIKnuE4BEzG2hmrWo0jYgkpG6tGnJV/w789ycbWbfr66DjiEg5TloIzKyemf0auAV40N3zayeWiCSauy7qRp20FH779oqgo4hIOU5aCNy9EHgAWAms1hECEamuFg3r8INhXZj9VT6frNsTdBwRKaPCtwzc/TDwfXe/VUcIRORUTBiUTZvGdXnwreUcP67JikRiSaXGELj7kpoOIiKJLzMjlZ+O6s6XW/fz2pKtQccRkVIqLARm1qw2gohIchh3VhZ92jXm4ZkrOXJMkxWJxIrKHCHYZWY5AGbWuobziEiCS0kx7hnTg237j/D0R+uDjiMiYZUpBIeBE/MQbDWz/jWYR0SSwDnZpzGyZyse/2ANOw8eCTqOiFC5QpAH3GJmLQCr4TwikiQmjz6Do8XH+eM7mqxIJBZUphDcDYwGdgAO/N7MfmNm482sl5ml1mhCEUlI2S0a8P2BHXlp4SZW7jgYdByRpFeZ0w7nAlmESoEBR4BRwF+ApcAhM1tsZtPN7Cc1GVZEEssdw7vSoE4av56xPOgoIkmvsqcdHnP32cB84G537wvUB/oANwIzgebAnTUVtDLMrL6ZLTKzS4LMISKV07R+Brdf2JU5q3Yxd9WuoOOIJLUqXe3Q3Qe7+9Lw7RJ3z3P3F9x9sruPcff21QlhZtPMbKeZLSuzfJSZrTSzNWY2uRKb+jfgr9XJICLBuO68jnRoVo9fz1hOiSYrEglMrFz++BlCb0P8U3hswhRCb1X0BMabWU8zO9PM3izz0dLMRgBfAZpNUSSO1ElLZfLoM1ix4yAv524OOo5I0kqraAUz63AqX8DdN1Vinblm1qnM4gHAGndfF87xIjDO3X8DfOMtATO7gNDbGD2Bw2Y2w92Pl7PeJGASQIcOp7RrIhIlo3u3JqdjUx55ZxVjz2pL/ToV/moSkSirzE/dBkJnF1SVhZ9X3bMQsoDSfy5sAc6JtLK73wNgZjcAu8srA+H1pgJTAXJycnR8UiQGmBn3fLsHlz3+MX+es5a7RnYPOpJI0qlMIehc4ynKV96cBxX+B+7uz0Q/iojUtL4dmjL2rLZMnbeO8ed0oE3jzKAjiSSVCguBu2+sjSDl2AKUHqTYDtgWUBYRqQU/vbg7s/J28PCslfzhyrODjiOSVGJlUGF5FgJdzayzmWUAVwOvB5xJRGpQ+2b1uPH8Trzy+VaWbd0fdByRpBIThcDMXgAWAN3NbIuZTXD3YuBHwCxgOfBXd88LMqeI1LzbLjidZvUz+NVbX+GuYT4itSUmhvK6+/gIy2cAM2o5jogEqFHddH48oiv/8Voe7y7fyUU9WwUdSSQpxMQRAhGR0sYP6ECXFvX5zYzlHCsp94QhEYmymDhCICJSWlpqCj8f04MJz+Yy7OEPqZMe3b9d2jSuy43ndebCM1qSkqKLuIqACoGIxKgLz2jJ5NFnRH1woQOLN+3j5um5nN6yAZOGZDPu7LbUSdOFWyW5WTIP2snJyfHc3NygY4hILTtWcpwZX27niTnrWL79AK0a1eHG8ztzzTkdaFQ3Peh4IjXGzBa5e065j6kQqBCIJCt3Z97q3fx57lrmr9lDgzppXHtOB248vzOtG9cNOp5I1KkQRKBCICInLNu6nz/PXcdbS7eRmmJ85+wsJg3JpmurhkFHE4kaFYIIVAhEpKzNBYU8NW8dL+Vu5six4ww/oyW3DO1C/05NMdMARIlvKgQRqBCISCQFh4p4bsFGnl2wgYJDRfTt0IRbhnThop6tSNWZCRKnVAgiUCEQkYocLirhb4s28+S89WwqKCS7eX1uHpzN5f2yqJuuMxMkvqgQRKBCICKVVXLcmblsB0/MWcuXW/fTvEEdbjy/E98/pyON6+nMBIkPKgQRqBCISFW5OwvW7eHPc9YxZ9Uu6mWkcnX/Dtw0qBOtG0X/zIS0VE0oK9GjQhCBCoGInIrl2w/w5Nx1vL5kG8XHa+Z36S1Ds/nZ6B41sm1JPioEEagQiEg0bN13mLeWbuPosehed2H+2t0s3ryPjycPp1n9jKhuW5LTyQqBpi4WETlFWU0ymTSkS9S3e3Hv1oz841z++5ON/MvwrlHfvkhpenNKRCRGdWvVkAu6t+DZjzdw5FhJ0HEkwakQiIjEsIlDstlzqIh/fLE16CiS4FQIRERi2LnZp9E7qxFPzlvH8RoauCgCKgQiIjHNzJg4OJt1uw7x/oqdQceRBKZCICIS48ac2YasJpk8OW9d0FEkgakQiIjEuPTUFG48vxOfri9gyeZ9QceRBKVCICISB67q356GddJ0lEBqjAqBiEgcaFg3nWvO6cCML7ezuaAw6DiSgFQIRETixA3ndyLFjGnz1wcdRRKQCoGISJxo0ziTS89qy0sLN7O/8FjQcSTBqBCIiMSRmwdnU1hUwvOfbQw6iiQYFQIRkTjSs20jBndtzjPzN3C0WNMZS/SoEIiIxJmJg7PZefAory/eFnQUSSAqBCIicWZw1+ac0bohT81bTzJfwl6iS4VARCTOnJjOeGX+Qeau3h10HEkQKgQiInFo7FltadWoDk/O1URFEh0qBCIicSgjLYUbzuvMR2t2k7dtf9BxJAGoEIiIxKlrzulA/YxUnpqniYrk1KkQiIjEqcaZ6VzVvwNvLNnGtn2Hg44jcU6FQEQkjt14ficceObjDUFHkTinQiAiEsfaN6vHmDPb8D+fbuLAEU1nLNWnQiAiEucmDu7M10eLeemzzUFHkTimQiAiEuf6tGvCwOxm/GX+eo6VHA86jsQpFQIRkQQwaUg22/YfYcaX24OOInFKhUBEJAEM69aSLi3qM3XuOk1nLNWSMIXAzFLM7EEze8zMrg86j4hIbUpJCU1nnLftAAvW7gk6jsShmCgEZjbNzHaa2bIyy0eZ2UozW2NmkyvYzDggCzgGbKmprCIiseo7fbNo3iCDqfM0nbFUXUwUAuAZYFTpBWaWCkwBRgM9gfFm1tPMzjSzN8t8tAS6Awvc/S7gB7WcX0QkcHXTU7n+3E58uHIXK3ccDDqOxJmYKATuPhcoKLN4ALDG3de5exHwIjDO3b9090vKfOwkdFRgb/i5JbWXXkQkdnx/YEfqpqfwlI4SSBXFRCGIIAsofVLtlvCySF4BLjazx4C5kVYys0lmlmtmubt27YpOUhGRGNG0fgZX5rTn1cVb2XngSNBxJI7EciGwcpZFHDrr7oXuPsHdb3f3KSdZb6q757h7TosWLaISVEQklkwY1Jni467pjKVKYrkQbAHal7rfDtgWUBYRkbjR8bT6jOrVmuc/3cSho8VBx5E4EcuFYCHQ1cw6m1kGcDXwesCZRETiwsQh2ew/fIyXczWdsVROTBQCM3sBWAB0N7MtZjbB3YuBHwGzgOXAX909L8icIiLxol+HpnyrY1Oenr+eYk1nLJUQE4XA3ce7ext3T3f3du7+dHj5DHfv5u5d3P3BoHOKiMSTiYOz2VxwmFl5+UFHkTgQE4VARESi76Kereh0Wj2mzl2r6YylQioEIiIJKjXFmDA4myVb9rNww96KnyBJTYVARCSBXdGvHU3rpTN1riYqkpNTIRARSWCZGan8n3M78e7yfNbu+jroOBLDVAhERBLcded2JCMthafmrQ86isQwFQIRkQTXvEEdvtuvHa98voXdXx8NOo7EKBUCEZEkcPPgzhwtPs5zCzYGHUVilAqBiEgS6NKiASN6tOS5TzZyuEgXhJVvUiEQEUkSEwdnU3CoiL9/viXoKBKD0oIOICIitWNA52ac1a4xv317Bc/qSohxoVfbRvzp6r618rVUCEREkoSZce/YXkybv14zF8aJdk3r1drXUiEQEUki3+oYuuiRSFkaQyAiIiIqBCIiIqJCICIiIqgQiIiICCoEIiIiggqBiIiIoEIgIiIiqBCIiIgIYMk8W5WZ7QKieemv5sDuKG4vViTifmmf4kci7lci7hMk5n4l2j51dPcW5T2Q1IUg2sws191zgs4RbYm4X9qn+JGI+5WI+wSJuV+JuE+R6C0DERERUSEQERERFYJomxp0gBqSiPulfYofibhfibhPkJj7lYj7VC6NIRAREREdIRAREREVAhEREUGFoFrMbJSZrTSzNWY2uZzH65jZS+HHPzWzTrWfsvLMrL2ZfWBmy80sz8zuKGedYWa238wWhz/uDSJrVZnZBjP7Mpw5t5zHzcweDb9WS82sXxA5K8vMupd6DRab2QEzu7PMOnHxWpnZNDPbaWbLSi1rZmbvmNnq8OemEZ57fXid1WZ2fe2lPrkI+/Swma0If3/9w8yaRHjuSb9XgxRhv35hZltLfZ+NifDck/6+DEqEfXqp1P5sMLPFEZ4bs6/VKXF3fVThA0gF1gLZQAawBOhZZp0fAk+Eb4NOeSUAAAexSURBVF8NvBR07gr2qQ3QL3y7IbCqnH0aBrwZdNZq7NsGoPlJHh8DvA0YMBD4NOjMVdi3VGAHoYlG4u61AoYA/YBlpZY9BEwO354M/K6c5zUD1oU/Nw3fbhr0/pxkn0YCaeHbvytvn8KPnfR7NQb36xfAv1bwvAp/X8bSPpV5/BHg3nh7rU7lQ0cIqm4AsMbd17l7EfAiMK7MOuOAZ8O3/wYMNzOrxYxV4u7b3f3z8O2DwHIgK9hUtWYcMN1DPgGamFmboENV0nBgrbtHc7bNWuPuc4GCMotL/+w8C3ynnKdeDLzj7gXuvhd4BxhVY0GroLx9cvfZ7l4cvvsJ0K7Wg52iCK9VZVTm92UgTrZP4d/XVwIv1GqogKkQVF0WsLnU/S188z/Pf64T/kWwHzitVtKdovDbG32BT8t5+FwzW2Jmb5tZr1oNVn0OzDazRWY2qZzHK/N6xqqrifwLKx5fK4BW7r4dQkUVaFnOOvH8mt1E6IhUeSr6Xo1FPwq/FTItwts78fpaDQby3X11hMfj8bWqkApB1ZX3l37Zczcrs07MMbMGwN+BO939QJmHPyd0aPos4DHg1drOV03nu3s/YDRwm5kNKfN4vL5WGcClwMvlPByvr1Vlxetrdg9QDDwfYZWKvldjzX8BXYCzge2EDrGXFZevFTCekx8diLfXqlJUCKpuC9C+1P12wLZI65hZGtCY6h1uqzVmlk6oDDzv7q+UfdzdD7j71+HbM4B0M2teyzGrzN23hT/vBP5B6BBmaZV5PWPRaOBzd88v+0C8vlZh+Sfesgl/3lnOOnH3moUHPl4CXOvhN6HLqsT3akxx93x3L3H348CTlJ83Hl+rNOBy4KVI68Tba1VZKgRVtxDoamadw3+lXQ28Xmad14ETI5+vAN6P9EsgFoTfL3saWO7uf4iwTusT4yDMbACh7509tZey6sysvpk1PHGb0OCuZWVWex24Lny2wUBg/4lD1jEu4l8w8fhalVL6Z+d64LVy1pkFjDSzpuHD1CPDy2KSmY0C/g241N0LI6xTme/VmFJmrM1llJ+3Mr8vY80IYIW7bynvwXh8rSot6FGN8fhBaGT6KkKjZ+8JL7uf0A88QF1Ch3LXAJ8B2UFnrmB/BhE6jLcUWBz+GAPcCtwaXudHQB6hUcKfAOcFnbsS+5UdzrsknP3Ea1V6vwyYEn4tvwRygs5dif2qR+g/+MallsXda0Wo0GwHjhH6S3ICobE27wGrw5+bhdfNAZ4q9dybwj9fa4Abg96XCvZpDaH30U/8bJ04A6ktMONk36ux8hFhv54L/8wsJfSffJuy+xW+/43fl7HwUd4+hZc/c+JnqdS6cfNancqHpi4WERERvWUgIiIiKgQiIiKCCoGIiIigQiAiIiKoEIiIiAgqBCJJLXzFut3h293C98u9Gl8N57jSzG4oZ/mHZva32s4jkoxUCETkhG7AfUCtFwJCF5K5oZzlPwR+VrtRRJJTWtABRCQxmVmmux8+lW24+1fRyiMiJ6cjBCKCmQ0D3gjfXW9mbmYbSj3ewcxeNLMCMys0s1lm1r3U453Cz7nWzKab2b4T2zOz68zso/Bz95rZB2aWU+q5zwDfBYaGt+Fm9ovwY994y8DMLjSzT83siJnlm9nj4Qtz/XNfwtsYZmYvm9nXZrbOzH4Y5X82kYSiIwQiAqErJP4r8HtCF3bZDhwFMLNmwEeEpku+FSgEJgPvmlm3MkcBfg+8AnwPKAkv6wRMJzR1bQZwDTDXzHq7+zrgAaADobcqTvynHWke+Z7ATOAdQiWiPfBbQtPJjiqz+pPAs8BUQtd+mGJmue7+WRX+XUSShgqBiODuB8xsZfjuF+6+odTDPwbqA2e7ewGAmc0HNhC6psCUUut+4u63ldn2/Sdum1kKof/M+wPfB+5397VmVgCkuPsnFUS9F9hI6LohJeFtFgAvmdm57r6g1LovuPuvwut8CIwlVHZUCETKobcMRKQiIwj9J37AzNLCl4c9CCwidNGh0t4q+2Qz62Fm/zCzfEJHDY4B3QkNYqyqAcA/TpSBsL8DxYQu0lXa7BM33P0YoQsmtavG1xRJCjpCICIVaQ4MBK4q57H3ytzPL30nfJnY2eHldxH66/4I8BShq4JWVZuyX8PdS8xsD9CszLr7ytwvqubXFEkKKgQiUpECQpe3faCcxw6WuV/28qnnEvqr/CJ3X3FioZk1rmaW7UDL0gvMLJXQZZMLqrlNEUGFQET+V1H4c9m/ot8jNE9AXjVOI8wMfz56YoGZnUdooOGiMl+7Mn+9fwpcZmY/L/W2weWEfpd9VMVsIlKKxhCIyAknBhXeYmbnmNmZ4ft/IHR2wPtmdo2ZDQ3PLDjFzMZXsM1PgK+BJ81spJndBLwIbC2z3grgTDP7jpnlmFnbCNv7FaEy8aqZjTGzSYTOIphVZkChiFSRCoGIAODuGwmdeng5MJ/wPALuvpvQGIIVwB8JjQl4CGgMLK1gm/mETkFsDbwG3Eno1MU1ZVZ9PLzdacBCYFKE7eUBowm9bfAKoYLwAnBFVfZVRL7J3Mu+5SciIiLJRkcIRERERIVAREREVAhEREQEFQIRERFBhUBERERQIRARERFUCERERAQVAhEREQH+H+HNIYd4btODAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(8, 4))\n",
    "ax.semilogy(np.minimum.accumulate(np.abs(Ytr - f.yopt)))\n",
    "ax.set_xlabel('Iteration', fontsize=15)\n",
    "ax.set_ylabel('$|f^\\star - f_{min}|$', fontsize=15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New acquisition function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now detail how to create your own acquisition function class and integrate it into the testing suite.\n",
    "\n",
    "In a similar manner to the fitness function classes, the acquisition function classes are imported from the `egreedy.acquisition_functions` module, with the specific classes available determined by the `__ini__.py` file in the same module. \n",
    "\n",
    "If, for example, your acquisition function class is called `greed` and is located in the file `gr.py`, you would place the python file in the directory `egreedy/acquisition_functions` and add the line:\n",
    "```\n",
    "from .gr import greed\n",
    "```\n",
    "to the `egreedy/acquisition_functions/__init__.py` file.\n",
    "\n",
    "The python file `egreedy/acquisition_functions/acq_func_optimisers.py` contains base classes for the acquisition function classes. We will now demonstrate how to implement two simple acquisition functions and then show how to optimise one of the test functions included in the suite."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BaseOptimiser` class is the base acquisition function class that implements the standard interface for acquisition function optimizers. It only contains an initialisation function with several arguments:\n",
    "\n",
    "- lb: lower-bound constraint\n",
    "- ub: upper-bound constraint\n",
    "- acq_budget : maximum number of calls to the Gaussian Process\n",
    "- cf : callable constraint function that returns True if the argument decision vector is VALID (optional, default value: None)\n",
    "- acquisition_args : Optional dictionary containing additional arguments that are unpacked into key=value arguments for an internal acquisition function; e.g. {'epsilon':0.1}.\n",
    "\n",
    "The `ParetoFrontOptimiser` class implements the base class as well as an additional function named `get_front(model)` that takes in a GPRegression model from GPy and approximates its Pareto front of model prediction and uncertainty. It returns the decision vectors belonging to the members of the front, an array containing corresponding their predicted value, and an array containing the prediction uncertainty.\n",
    "\n",
    "We first create a simple acquisition function, extending the base class, that generates uniform samples in space and uses the Gaussian Process's mean prediction to select the best (lowest value) predicted location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from egreedy.acquisition_functions.acq_func_optimisers import BaseOptimiser\n",
    "\n",
    "class greedy_sample(BaseOptimiser):\n",
    "    \"\"\"Greedy function that uniformly samples the GP posterior\n",
    "    and returns the location with the best (lowest) mean predicted value.\n",
    "    \"\"\"\n",
    "    \n",
    "    # note we do not need to implement an __init__ method because the\n",
    "    # base class already does this. Here we will include a commented\n",
    "    # version for clarity.\n",
    "#     def __init__(self, lb, ub, acq_budget, cf=None, acquisition_args={}):\n",
    "#         self.lb = lb\n",
    "#         self.ub = ub\n",
    "#         self.cf = cf\n",
    "#         self.acquisition_args = acquisition_args\n",
    "#         self.acq_budget = acq_budget\n",
    "\n",
    "    def __call__(self, model):\n",
    "        \"\"\"Returns the location with the best (lowest) predicted value\n",
    "        after uniformly sampling decision space.\n",
    "        \"\"\"\n",
    "        \n",
    "        # generate samples\n",
    "        X = np.random.uniform(self.lb, self.ub,\n",
    "                              size=(acq_budget, self.lb.size))\n",
    "        \n",
    "        # evaluate them with the gp\n",
    "        mu, sigmasqr = model.predict(X, full_cov=False)\n",
    "        \n",
    "        # find the index of the best value\n",
    "        argmin = np.argmin(mu.flat)\n",
    "        \n",
    "        # return the best found value\n",
    "        return X[argmin, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from egreedy.acquisition_functions.acq_func_optimisers import ParetoFrontOptimiser\n",
    "\n",
    "\n",
    "class greedy_pfront(ParetoFrontOptimiser):\n",
    "    \"\"\"Exploitative method that calculates the approximate Pareto front\n",
    "    of a GP model and returns the Pareto set member that has the best\n",
    "    (lowest) predicted value.\n",
    "    \"\"\"\n",
    "    \n",
    "    # again here we do not need to implement an __init__ method.\n",
    "    \n",
    "    def __call__(self, model):\n",
    "        \"\"\"Returns the location with the best (lowest) predicted value\n",
    "        from the approximate Pareto set of the GP's predicted value and\n",
    "        its corresponding uncertainty.\n",
    "        \"\"\"\n",
    "        \n",
    "        # approximate the pareto set; here X are the locations of the\n",
    "        # members of the set and mu and sigma are their predicted values\n",
    "        # and uncertainty\n",
    "        X, mu, sigma = self.get_front(model)\n",
    "        \n",
    "        # find the index of the best value\n",
    "        argmin = np.argmin(mu.flat)\n",
    "        \n",
    "        # return the best found value\n",
    "        return X[argmin, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create a similar script to the one used above in the function example. This time we will optimise the `push4` function included in the test suite and load the training data associated with the first run all techniques evaluated in the paper carried out. \n",
    "\n",
    "Note that in this case the training data contains arguments to be passed into the function during instantiation. This is because the `push4` runs are evaluated on a *problem instance* basis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training a GP model with 8 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.99363897 0.49599772 0.99868045 0.97496078]\n",
      "Function value: 8.38627\n",
      "Best function value so far: 2.01458\n",
      "\n",
      "Training a GP model with 9 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.99735174 0.05688416 0.95975881 0.09915249]\n",
      "Function value: 6.39738\n",
      "Best function value so far: 2.01458\n",
      "\n",
      "Training a GP model with 10 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.99426994 0.48176988 0.04539143 0.52413627]\n",
      "Function value: 4.30593\n",
      "Best function value so far: 2.01458\n",
      "\n",
      "Training a GP model with 11 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.97676317 0.20990921 0.03083971 0.94382162]\n",
      "Function value: 4.32446\n",
      "Best function value so far: 2.01458\n",
      "\n",
      "Training a GP model with 12 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.98233984 0.45681325 0.32258875 0.65155143]\n",
      "Function value: 1.9939\n",
      "Best function value so far: 1.9939\n",
      "\n",
      "Training a GP model with 13 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.98055465 0.40088946 0.31056256 0.72057199]\n",
      "Function value: 2.33662\n",
      "Best function value so far: 1.9939\n",
      "\n",
      "Training a GP model with 14 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.87735007 0.4604941  0.26450847 0.67341351]\n",
      "Function value: 0.679518\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 15 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.77766143 0.49442689 0.30924655 0.68061374]\n",
      "Function value: 1.68864\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 16 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.86687143 0.5127836  0.19990311 0.69685941]\n",
      "Function value: 0.951315\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 17 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.85716692 0.48996299 0.19234286 0.6498631 ]\n",
      "Function value: 2.24849\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 18 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.87333699 0.56790019 0.27448691 0.80146525]\n",
      "Function value: 1.56872\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 19 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.88507765 0.4508956  0.22725763 0.75254124]\n",
      "Function value: 1.00578\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 20 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.90001785 0.5103195  0.25189873 0.67868624]\n",
      "Function value: 2.1292\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 21 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.82927784 0.53929787 0.12721344 0.76115955]\n",
      "Function value: 1.65907\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 22 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.88407381 0.3162302  0.39928955 0.58348638]\n",
      "Function value: 1.58757\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 23 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.02661341 0.95209027 0.97135965 0.98684695]\n",
      "Function value: 7.67696\n",
      "Best function value so far: 0.679518\n",
      "\n",
      "Training a GP model with 24 data points.\n",
      "Optimising the acquisition function.\n",
      "Expensively evaluating the new selected location.\n",
      "New location  : [0.89053271 0.37854249 0.30893649 0.67180526]\n",
      "Function value: 3.55493\n",
      "Best function value so far: 0.679518\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from egreedy.optimizer import perform_BO_iteration\n",
    "from egreedy import test_problems\n",
    "\n",
    "# ---- optimisation run details\n",
    "problem_name = 'push4'\n",
    "run_no = 1\n",
    "acq_budget = 5000 * 4  # because the problem dimensionality is 4\n",
    "\n",
    "total_budget = 25\n",
    "\n",
    "# ---- load the training data\n",
    "data_file = f'../training_data/{problem_name:}_{run_no:}.npz'\n",
    "\n",
    "with np.load(data_file, allow_pickle=True) as data:\n",
    "    Xtr = data['arr_0']\n",
    "    Ytr = data['arr_1']\n",
    "    \n",
    "    if 'arr_2' in data:\n",
    "        f_optional_arguments = data['arr_2'].item()\n",
    "    else:\n",
    "        f_optional_arguments = {}\n",
    "        \n",
    "# ---- instantiate the test problem\n",
    "f_class = getattr(test_problems, problem_name)\n",
    "f = f_class(**f_optional_arguments)\n",
    "\n",
    "\n",
    "# ---- instantiate the acquistion function we created earlier\n",
    "acq_func = greedy_sample(lb=f.lb, ub=f.ub, cf=None, acq_budget=acq_budget,\n",
    "                          acquisition_args=acquisition_args)\n",
    "\n",
    "while Xtr.shape[0] < total_budget:\n",
    "    # perform one iteration of BO:\n",
    "    # this returns the new location and function value (Xtr, Ynew) as well\n",
    "    # as the trained model used to select the new location\n",
    "    Xnew, Ynew, model = perform_BO_iteration(Xtr, Ytr, f, acq_func, verbose=True)\n",
    "\n",
    "    # augment the training data and repeat\n",
    "    Xtr = np.concatenate((Xtr, np.atleast_2d(Xnew)))\n",
    "    Ytr = np.concatenate((Ytr, np.atleast_2d(Ynew)))\n",
    "    \n",
    "    print('Best function value so far: {:g}'.format(np.min(Ytr)))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAELCAYAAADA0B94AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdCklEQVR4nO3de5hddX3v8c9nbrlnkrkkE0JCMgwIDPeOyZQE9XBJQI+KtHpK6zlaq0irrbZqqz49R+vl2Poo9jyn0qdYb7QWWhWo1R6IIhATJTDhEgkJJoRwScJkQsj9Npfv+WPvsemYZC57zV57r/1+PU+e2bP32ms+WWzymbV+a62fI0IAAKD8VaUdAAAAJINSBwAgIyh1AAAyglIHACAjKHUAADKiJu0AY9HU1BQLFixIOwYAAEWxdu3aXRHRPNxyZVnqCxYsUFdXV9oxAAAoCtvPjWQ5Dr8DAJARlDoAABlBqQMAkBGUOgAAGUGpAwCQEZQ6AAAZQakDAJARZXmdepK+snKL9h/pLWgdtvUbl56u+Y2TE0oFAMDoVXypf+OnW7V97+GC1hEh7dh7WJ//zYsSSgUAwOhVfKmv/ugVBa/j3d/s0ppndyeQBgCAsWNMPQGdrQ167uVDemnvkbSjAAAqGKWegM7WRknSmmdfTjkJAKCSUeoJOHfOdE2bWKOHtnAIHgCQHko9AdVV1qsXNGjNFvbUAQDpodQTsnhhg7bsOqid+xhXBwCkg1JPyH+Mq3MIHgCQDko9Ie2nTdfUCTWcLAcASA2lnpCa6ir92hkzOVkOAJAaSj1Bi1sbtHnnAe06cDTtKACACkSpJ2hwXP1hxtUBACmg1BN0wdx6Ta6r5tI2AEAqKPUE1TKuDgBIEaWesMULG/R0937tPngs7SgAgApDqSeMcXUAQFoo9YRdePoMTayt4np1AEDRUeoJq6up0qXzGVcHABRfSZS67a22f277cdtdaecp1OKFjdr40j7tPdSbdhQAQAUpiVLP+y8RcXFEdKQdpFCdrQ2KkB7eyt46AKB4SqnUM+OieTNUV1PF9eoAgKIqlVIPSStsr7V9Y9phCjWxtlqXzJvBjG0AgKIqlVJfEhGXSrpW0vtsv2boArZvtN1lu6unp6f4CUdpcWuj1m/fq31HGFcHABRHSZR6RGzPf90p6S5Ji06wzK0R0RERHc3NzcWOOGqdrQ0aCKmLcXUAQJGkXuq2p9ieNvhY0jJJT6abqnCXzp+puuoqreHSNgBAkdSkHUDSbEl32ZZyef4pIu5JN1LhJtZW66J59XqIcXUAQJGkXuoRsUXSRWnnGA+LFzbqbx98RgeO9mnqhNQ3NQAg41I//J5lna2N6h8IxtUBAEVBqY+jS8+YoZoqc2kbAKAoKPVxNLmuRheeXs9NaAAARUGpj7PFrY1a9+JeHTrWl3YUAEDGUerjrLO1UX0DobXPvZJ2FABAxlHq4+zXzpip6ipzvToAYNxR6uNs6oQanT+3XmueZVwdADC+KPUi6FzYoMdf2KPDx/rTjgIAyDBKvQg6WxvV2x967HnG1QEA44dSL4KOBTNVZXHLWADAuKLUi2DaxFq1n8b16gCA8UWpF8nihQ167IU9OtLLuDoAYHxQ6kXS2dqoY30DevyFPWlHAQBkFKVeJK9e2CBbXK8OABg3lHqR1E+q1bkt07leHQAwbij1Ilrc2qBHn39FR/sYVwcAJI9SL6LO1kYd6R3Quhf3ph0FAJBBlHoRLVrQIElc2gYAGBeUehHNnFKnc1qmaQ03oQEAjANKvcgWL2xQ19ZX1Ns/kHYUAEDGUOpF1tnaqMO9/YyrAwASR6kX2aKF+XF1Lm0DACSMUi+yxqkTdNasqdyEBgCQOEo9BYtbG9S1dbf6GFcHACSIUk9BZ2ujDh7r1/rt+9KOAgDIEEo9BYPj6g9xvToAIEGUegpmTZuo1uYpXK8OAEgUpZ6SxQsb9cizu9U/EGlHAQBkBKWeks7WBu0/2qcNOxhXBwAkg1JPSWdroyTG1QEAyaHUUzJ7+kQtaJysh7heHQCQEEo9RYsXNuqRrbs1wLg6ACABJVPqtqttP2b7+2lnKZbOMxu093CvNr60P+0oAIAMKJlSl/QBSRvSDlFMixcyrg4ASE5JlLrt0yW9QdLfp52lmE6bMUnzGiYxuQsAIBElUeqS/lrSn0o66c3Qbd9ou8t2V09PT/GSjbPFCxv18LOMqwMACpd6qdv+r5J2RsTaUy0XEbdGREdEdDQ3Nxcp3fjrbG3UK4d6tWnngbSjAADKXOqlLmmJpDfZ3irpDklX2P7HdCMVz2LuAw8ASEjqpR4RH4uI0yNigaTfkvTjiHh7yrGKZl7DZM2dwbg6AKBwNcMtYHt+IT8gIp4v5P2VYPHCBq3c1KOIkO204wAAytSwpS5pq6SxnMXl/PuqR/qGiHhA0gNj+FllrbO1UXc+tk3rXtyrBU1Txrye2mprct1I/pMCALJoJA2wcNxTVLjB+8C/+curC1qPLd32rkW6/KzsnEgIABi5YUs9Ip4rRpBKNr9xsm75nUu1Y++RgtbzxRVPa8X6bkodACoUx2pLxOsvmFPwOlZv3qXVm3clkAYAUI4KPvvd9sdtb7e9zvZttv/E9hVJhMPoLGlr0pZdB7Vtz+G0owAAUpDEJW1/KOliScsl3S5poqSbElgvRmlpW5MksbcOABUqicPvT0bEzvzjHZL+XwLrxBicPXuqmqZO0OrNu/S2jnlpxwEAFFkSe+qP2b7Z9qQE1oUC2NbStkat3rxLEdxLHgAqTRKlPlPSUkkv2v6Z7Vts35jAejEGS9qatOvAMeZoB4AKVPDh94h4jyTZrpL0KkkX5f8gBUuOG1c/d870lNMAAIppzHvqtv8u//Wtts+KiIGI2BARd0TEx5KLiNE4bcYktTZP0SpOlgOAilPI4fcv5b9eKekfbHfbXm37y7bfk0A2jNHStiat2bJbx/pOOj09ACCDxlzqEbEx//WmiOiU1CLp9yT9RFJrMvEwFkvamnS4t1+PPf9K2lEAAEVU8Ji67bMlXS9pt6R1kr4XEXcUul6MXWdro6qcG1dfnL+vPAAg+5I4+/0HkuolnSnpk5I22X46gfVijOon1eqieTMYVweACpPEzWd2DT0xzvbsBNaLAixta9ItDzyjfUd6NX1ibdpxAABFkMSe+p22lx3/RER0J7BeFGBJW5P6B0JrtuxOOwoAoEiSKPWdkr5j+17bn7D9JtvzE1gvCnDJ/BmaVFvNfeABoIIkUeqflfTG/Nfdkq6TdHcC60UBJtRUa9HCBsbVAaCCJDGm/nREPJh/vDKB9SEhS9ua9Nl/36CX9h5RS/3EtOMAAMbZsHvqthuGWWST7S8yoUvpWcJUrABQUUZy+L3Hdock2W45yTJLJL3AhC6l5ZyWaWqcUkepA0CFGMnh98OS6vKPt9nujIhHBl+MiJskybaVm9DlYjGhS0moqrIua2vSqvxUrLn/RACArBrJnvp6Se+13SzppK0QORuZ0KW0LG1r1M79R7Vp54G0owAAxtlISv0jkq6V9JKkkPQF25+zfYPtdtvV45oQBRkcV1+1iUPwAJB1w5Z6RKyUNFe5YrekI5KukfR15e71ftD247Zvs/2h8QyL0Tt95mQtaJzMuDoAVIARXaceEb0RsULSakkfiYhLJE2RdKGk35V0j6QmSR8cr6AYuyVtTXpoy8vq7WcqVgDIslHdfCYiLo+IdfnH/RGxPiJuj4iPRsTrI2Le+MREIZa2NengsX498cKetKMAAMZREneUQ4m77Mwm2eLucgCQcZR6BaifXKsL59Yzrg4AGUepV4glbU167Pk9OnC0L+0oAIBxQqlXiKVtTeobCD387MtpRwEAjBNKvUJcesZMTaip0qpNlDoAZNWYS932q2z3FxrA9kTbD9t+wvZ6239R6DrxqybW5qZiZVwdALKr0D31JG4mflTSFRFxkXL3jb/GdmcC68UQS9qa9HT3fu3cfyTtKACAcVBoqUehAfL3jB+8MXlt/k/B68WvWpq/ZexPN3MIHgCyqCTG1G1X235c0k5JP4yINSdY5kbbXba7enp6ih8yA86bM10zJtdyvToAZFRJlHr+7nQXSzpd0iLb559gmVsjoiMiOpqbm4sfMgOqqqwlZzZpdX4qVgBAtpREqQ+KiD2SHlBuwhiMgyVtTdqx94ie6TmYdhQAQMJSL3XbzbZn5B9PknSVpI3ppsquwXF1zoIHgOxJvdQlzZF0v+11kh5Rbkz9+ylnyqz5jZM1r2ES4+oAkEE1Bb6/4Eva8rO+XVLoejByS9ua9P0ndqivf0A11aXwex0AIAmF/Iu+Q9J7kgqC4lnS1qT9R/u0btvetKMAABI05lKPiH0R8dUkw6A4BqdiXb2JQ/AAkCUce61ADVPq1H7adMbVASBjKPUKtaStSY8+/4oOHWMqVgDICkq9Qi1ta1Jvf+jhZ3enHQUAkBBKvUK9ekGD6mqquF4dADKEUq9QE2ur1XHGTK1ichcAyAxKvYItaWvShh37tOvA0bSjAAASMKJSt1033kFQfL+civUZ9tYBIAtGuqf+RdudtmePaxoU1flz6zV9Yg3XqwNARpzyNrG2J0v6c0nvlXReRFxZlFQoiuoq67Izm7QqPxWrXfBdfwEAKTrlnnpEHJL0aUlPS9rEnnr2LDmrSdv2HNbWlw+lHQUAUKBhJ3SJiMO23x4RTxQjEIprcFx91eZdWtg0JeU0AIBCjGhMnULPrgWNkzV3xiTG1QEgA4YtddsNxQiCdNjWkrZG/fSZXeofiLTjAAAKMJI99R7bHZJku2Wc8yAFS89q1r4jfXqSqVgBoKyNpNQPSxq8Tn2b7VePYx6k4LIzGyWJWdsAoMyNpNTXS3qv7WZJXPOUQU1TJ+jcOdO5DzwAlLmRlPpHJF0r6SVJIekLtj9n+wbb7barxzUhimJpW6O6tr6iw8f6044CABijYUs9IlZKmqtcsVvSEUnXSPq6pHWSDtp+3PZttj80nmExfpa0NelY/4C6nmMqVgAoVyO9pK03IlZIWi3pIxFxiaQpki6U9LuS7pHUJOmD4xUU42vRwgbVVptxdQAoY8PefOZ4EXH5cY/7lRtvXy/p9oRzocgm19Xo0vkzGVcHgDLG1Kv4paVtTVq/fZ+69x1JOwoAYAyG3VO3Pb+QHxARzxfyfhTPsvYW/Z/7NumN/3eV/vdbLtBV53GrfwAoJ4449V3EbA8od9b7qNctKSIi8bPjOzo6oqurK+nVQtKT2/bqw99+Qhtf2q/rLj5Nn3hju2ZOqRv+jQCAcWN7bUR0DLfcSMbUFyaQB2Xi/Ln1+t77l+qWBzbrb368Was2v6zPXNeua86fk3Y0AMAwht1TL0XsqRfHU9v36SPfeULrt+/TGy6co0+9qV2NUyekHQsAKs5I99Q5UQ4ndd5p03X3+5bow8vO1g/Xd+vqL63Uvz2xXeX4iyAAVAJKHadUW12l919xlr7/R0s1b+Yk/eHtj+mmf1yrnfs5Qx4ASg2ljhE5e/Y0fff3L9NHrz1H9z/do2VfWqm7HnuRvXYAKCGUOkasprpKN732TP37H12u1qYp+uN/fkLv/mYX17UDQImg1DFqbbOm6ts3XaY/f8O5Wv3MLl1184P6l64X2GsHgJSlXuq259m+3/YG2+ttfyDtTBhedZX17stbdc8HXqNz50zXn35nnd759Ue0fc/htKMBQMVK/ZI223MkzYmIR21Pk7RW0nUR8dTJ3sMlbaVlYCD0Dw89p7+6Z6OqbH3wqrN0RuOUtGNhiLqaKk2dUK0pE2o0pa4m93VCtSbUMHsyUOqSvPnMuIqIHZJ25B/vt71BualeT1rqKC1VVdY7LlugK86ZpT/77jp95gcb0o6EUaittibX1WhqvuSnTMg9nlz3H49zvwhUq6a6sIN7NVXWWy6Zy/0OgHGSeqkfz/YCSZdIWnOC126UdKMkzZ9f0O3oMU7mNUzWt969WL/oPqDe/oG04+A4EdKx/n4dONqvg0f7dOBonw4d7dPBY/06cLRPB4/26WD+tYPHcq937zuSe+5Y7vXe/mSO6vUcOKqPXXtuIusC8J+lfvh9kO2pkh6U9NmIuPNUy3L4HSi+o339Gijwd7X33NalbXsO68cfeq1sJxMMqABlc/hdkmzXSvqupG8NV+gA0pHE2Pvy81v0P+9+Upt2HtDZs6clkArA8Urh7HdL+qqkDRFxc9p5AIyfZfnpfFesfynlJEA2pV7qkpZI+u+SrrD9eP7P69MOBSB5s6dP1CXzZ+je9d1pRwEyKfXD7xGxSrm51wFUgGXnteiv7tmobXsOa+6MSWnHATKlFPbUAVSQ5e0cggfGC6UOoKham6fqrFlTtYJD8EDiKHUARbesfbYe3rpbrxw8lnYUIFModQBFt7y9Rf0DoR9tYG8dSBKlDqDoLphbr9PqJ2rFU5Q6kCRKHUDR2day9hat/EWPDh3rSzsOkBmUOoBULDtvto72DWjlL3rSjgJkBqUOIBWLFjZoxuRabkQDJIhSB5CKmuoqXXnObN23oZtZ/YCEUOoAUrO8fbb2HenTmi27044CZAKlDiA1l5/VrIm1VbqXu8sBiaDUAaRmUl21Xnt2s1Y89ZIGBiLtOEDZo9QBpGp5e4u69x3Vum17044ClD1KHUCqrjxntqqrzCF4IAGUOoBU1U+uVWdrA6UOJIBSB5C65e0t2tJzUJt3Hkg7ClDWKHUAqbv6vNwc6+ytA4Wh1AGkbk79JF10er1WUOpAQSh1ACVhWXuLnnhxr3bsPZx2FKBsUeoASsLy9hZJ0g+ZjhUYM0odQElomzVVZzZPYVwdKAClDqBkLGtv0UNbdmvPoWNpRwHKEqUOoGQsb29R/0Dovg07044ClCVKHUDJuHBuvVqmT9SKpzgED4wFpQ6gZFRVWcvaZ+vBX/To8LH+tOMAZYdSB1BSlp3XoiO9A1q5qSftKEDZodQBlJTFrQ2qn1SrFeu5tA0YLUodQEmpra7SlefM0n0bu9XXP5B2HKCsUOoASs6y9hbtOdSrh5/dnXYUoKxQ6gBKzmvObtKEmipuRAOMEqUOoORMrqvRa85u1oqnuhURaccBygalDqAkLW9v0Y69R/TzbXvTjgKUjdRL3fbXbO+0/WTaWQCUjivPmaXqKnMIHhiF1Etd0jckXZN2CAClZeaUOi1a0KB7ubQNGLHUSz0iVkriFFcAv2J5+2xt3nlAz/QcSDsKUBZSL/WRsn2j7S7bXT093GkKqATL8nOscyMaYGTKptQj4taI6IiIjubm5rTjACiC02ZM0gVz6xlXB0aobEodQGVa3j5bj7+wR937jqQdBSh5lDqAkrZ88BD8UxyCB4aTeqnbvl3SzyS9yvaLtn8v7UwASkfbrKlqbZqiFRyCB4ZVk3aAiLgh7QwASpdtXd0+W1/9ybPae6hX9ZNr044ElKzU99QBYDjL21vUNxC6/+mdaUcBShqlDqDkXXz6DM2aNoGz4IFhUOoASl5VlXX1ebP1wNM9OtLbn3YcoGRR6gDKwvL2Fh3u7deqTbvSjgKULEodQFnobG3UtIk1HIIHToFSB1AW6mqqdOU5s/SjDd3q6x9IOw5QklK/pA0ARmpZe4vufny7rrr5QdVWs0+C0vX7rztT1196etF/LqUOoGxccc4s3bBovvYePpZ2FOCUZqR0PwVKHUDZmFhbrc9df0HaMYCSxfErAAAyglIHACAjKHUAADKCUgcAICModQAAMoJSBwAgIyh1AAAyglIHACAjHBFpZxg12z2SnktwlU2SmPopOWzP5LFNk8c2TRbbM3nHb9MzIqJ5uDeUZaknzXZXRHSknSMr2J7JY5smj22aLLZn8sayTTn8DgBARlDqAABkBKWec2vaATKG7Zk8tmny2KbJYnsmb9TblDF1AAAygj11AAAyglIHACAjKrrUbV9j+2nbm21/NO08WWB7q+2f237cdlfaecqR7a/Z3mn7yeOea7D9Q9ub8l9nppmxnJxke37S9rb85/Rx269PM2M5sT3P9v22N9heb/sD+ef5jI7RKbbpqD+nFTumbrta0i8kXS3pRUmPSLohIp5KNViZs71VUkdEcBOKMbL9GkkHJN0WEefnn/u8pN0R8Zf5X0BnRsSfpZmzXJxke35S0oGI+EKa2cqR7TmS5kTEo7anSVor6TpJ7xSf0TE5xTZ9m0b5Oa3kPfVFkjZHxJaIOCbpDklvTjkToIhYKWn3kKffLOmb+cffVO5/eIzASbYnxigidkTEo/nH+yVtkDRXfEbH7BTbdNQqudTnSnrhuO9f1Bg3Iv6TkLTC9lrbN6YdJkNmR8QOKfcPgKRZKefJgvfbXpc/PM+h4jGwvUDSJZLWiM9oIoZsU2mUn9NKLnWf4LnKHItI1pKIuFTStZLelz/0CZSav5V0pqSLJe2Q9MV045Qf21MlfVfSByNiX9p5suAE23TUn9NKLvUXJc077vvTJW1PKUtmRMT2/Nedku5SbpgDhevOj7sNjr/tTDlPWYuI7ojoj4gBSV8Rn9NRsV2rXPl8KyLuzD/NZ7QAJ9qmY/mcVnKpPyLpLNsLbddJ+i1J30s5U1mzPSV/kodsT5G0TNKTp34XRuh7kt6Rf/wOSf+aYpayN1g+eW8Rn9MRs21JX5W0ISJuPu4lPqNjdLJtOpbPacWe/S5J+csD/lpStaSvRcRnU45U1my3Krd3Lkk1kv6JbTp6tm+X9Drlpl3slvQJSXdL+hdJ8yU9L+mtEcHJXyNwku35OuUOaYakrZLeOzgejFOzvVTSTyT9XNJA/umPKzcGzGd0DE6xTW/QKD+nFV3qAABkSSUffgcAIFModQAAMoJSBwAgIyh1AAAyglIHACAjKHWgzORnbtqVf3x2/vsZKeR4m+13nuD5B2x/p9h5AFDqQLk7W7nrrote6srNIPXOEzz/B5I+VtwoAKTcDUIAQJJke1JEHC5kHUxfDKSHPXWgTNl+naR/y3/7rO3Iz2c/+Pp823fY3m37kO17bb/quNcX5N/zO7Zvs71ncH22/4ftVfn3vmL7ftsdx733G5J+Q9Jr8+uI/BzlJzz8bvsK22tsH7HdbfuW/OQVv/y75NfxOtvftn3A9hbbf5DwZgMyjT11oHw9KunDkr4g6XrlZnE6Kkm2GyStkvSypJskHZL0UUk/sn32kL3xL0i6U9JbJfXnn1sg6TZJz0iqk/TbklbaPj8itkj6tHK3A52h3OF2KTdJ0q+wfZ6keyT9ULlfBOZJ+ktJrZKuGbL4V5Sbi/tW5W6R+WXbXRHx8Ci2C1CxKHWgTEXEPttP5799LCK2HvfyH0uaIuniwftv216t3P2j3yXpy8ct+1BEvG/Iuj81+Nh2lXKF/GpJb5f0qYh4xvZuSVUR8dAwUf+XpOckvSki+vPr3C3pn23/ekT87Lhlb4+Iz+SXeUDSG5X7hYVSB0aAw+9ANl2lXBHvs11ju0bSfklrJXUMWfYHQ99s+1zbd9nuVm7vvVfSq5Q7MW+0Fkm6a7DQ874rqU/S0iHLrhh8EBG9kjYpNy0ygBFgTx3IpiZJnZL+2wleu2/I993Hf5OfPndF/vk/UW4v+4ikv5c0cQxZ5gz9GRHRb/tlSQ1Dlt0z5PtjY/yZQEWi1IFs2q3c/NafPsFr+4d8P3Sqxl9Xbu/46ojYOPik7foxZtkhadbxT9iultSYzwkgIZQ6UN6O5b8O3Zu9T7nryNeP4RK1SfmvRwefsH2ZcifPrR3ys0eyF71G0ltsf/y4Q/DXK/fvz6pRZgNwCoypA+Vt8ES599pebPuC/Pc3K3fW+o9t/7bt1+bvAPdl2zcMs86HJB2Q9BXby2y/S9IdkrYNWW6jpAtsX2e7w/ZpJ1nfZ5T7heBu26+3faNyZ7ffO+QkOQAFotSBMhYRzyl3Wdv1klYrf515ROxSbkx9o6QvKTdG/nlJ9ZLWDbPObuUub2uR9K+SPqjcZXGbhyx6S369X5P0iKQbT7K+9ZKuVe4Q/J3Klfztkn5zNH9XAMNzxNDhNAAAUI7YUwcAICModQAAMoJSBwAgIyh1AAAyglIHACAjKHUAADKCUgcAICModQAAMuL/A+8YlyMVCioaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(8, 4))\n",
    "ax.plot(np.minimum.accumulate(np.abs(Ytr - f.yopt)))\n",
    "ax.set_xlabel('Iteration', fontsize=15)\n",
    "ax.set_ylabel('$|f^\\star - f_{min}|$', fontsize=15)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}