Interactive_Bootstrap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "## Interactive Bootstrap Demonstration\n",
    "\n",
    "### Boostrap for Uncertainty in Sample Statistics Tutorial\n",
    "\n",
    "* interactive plot demonstration with ipywidget package\n",
    "\n",
    "#### Michael Pyrcz, Associate Professor, The University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)\n",
    "\n",
    "#### Bootstrap\n",
    "\n",
    "Uncertainty in the sample statistics\n",
    "* one source of uncertainty is the paucity of data.\n",
    "* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?\n",
    "\n",
    "Would it be useful to know the uncertainty in these statistics due to limited sampling?\n",
    "* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?\n",
    "\n",
    "**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.\n",
    "\n",
    "Assumptions\n",
    "* sufficient, representative sampling, identical, idependent samples\n",
    "\n",
    "Limitations\n",
    "1. assumes the samples are representative \n",
    "2. assumes stationarity\n",
    "3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data\n",
    "4. does not account for boundary of area of interest \n",
    "5. assumes the samples are independent\n",
    "6. does not account for other local information sources\n",
    "\n",
    "The Bootstrap Approach (Efron, 1982)\n",
    "\n",
    "Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.\n",
    "* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error: \n",
    "\n",
    "\\begin{equation}\n",
    "\\sigma^2_\\overline{x} = \\frac{\\sigma^2_s}{n}\n",
    "\\end{equation}\n",
    "\n",
    "Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.\n",
    "* Would not be possible access general uncertainty in any statistic without bootstrap.\n",
    "* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).\n",
    "\n",
    "Steps: \n",
    "\n",
    "1. assemble a sample set, must be representative, reasonable to assume independence between samples\n",
    "\n",
    "2. optional: build a cumulative distribution function (CDF)\n",
    "    * may account for declustering weights, tail extrapolation\n",
    "    * could use analogous data to support\n",
    "\n",
    "3. For $\\ell = 1, \\ldots, L$ realizations, do the following:\n",
    "\n",
    "    * For $i = \\alpha, \\ldots, n$ data, do the following:\n",
    "\n",
    "        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available). \n",
    "\n",
    "6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\\ell$, $\\sigma^2_{\\ell}$. Return to 3 for another realization.\n",
    "\n",
    "7. Compile and summarize the $L$ realizations of the statistic of interest.\n",
    "\n",
    "This is a very powerful method.  Let's try it out.\n",
    "\n",
    "\n",
    "#### Objective \n",
    "\n",
    "Provide an example and demonstration for:\n",
    "\n",
    "1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets\n",
    "2. provide an intuitive hands-on example of statistical boostrap   \n",
    "\n",
    "#### Getting Started\n",
    "\n",
    "Here's the steps to get setup in Python with the GeostatsPy package:\n",
    "\n",
    "1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/). \n",
    "2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality. \n",
    "\n",
    "#### Load the Required Libraries\n",
    "\n",
    "The following code loads the required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from ipywidgets import interactive                        # widgets and interactivity\n",
    "from ipywidgets import widgets                            \n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import Label\n",
    "from ipywidgets import VBox, HBox\n",
    "import matplotlib.pyplot as plt                           # plotting\n",
    "import numpy as np                                        # working with arrays\n",
    "import pandas as pd                                       # working with DataFrames\n",
    "from scipy.stats import triang                            # parametric distributions\n",
    "from scipy.stats import binom\n",
    "from scipy.stats import norm\n",
    "from scipy.stats import uniform\n",
    "from scipy.stats import triang\n",
    "from scipy import stats                                   # statistical calculations\n",
    "import random                                             # random drawing / bootstrap realizations of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a Synthetic Dataset\n",
    "\n",
    "This is an interactive method to:\n",
    "\n",
    "* select a parametric distribution\n",
    "\n",
    "* select the distribution parameters\n",
    "\n",
    "* select the number of samples and visualize the synthetic dataset distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters for the synthetic dataset\n",
    "bins = np.linspace(0,1000,1000)\n",
    "\n",
    "# interactive calculation of the sample set (control of source parametric distribution and number of samples)\n",
    "l = widgets.Text(value='                                      Boostrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))\n",
    "dist = widgets.Dropdown(\n",
    "    options=['Triangular', 'Uniform', 'Gaussian'],\n",
    "    value='Gaussian',\n",
    "    description='Dataset Distribution:',\n",
    "    disabled=False,\n",
    "    layout=Layout(width='200px', height='30px')\n",
    ")\n",
    "a = widgets.FloatSlider(min=0.0, max = 100.0, value = 50.0, description = 'Dataset: Mean / Mode',orientation='vertical',layout=Layout(width='170px', height='200px'),continuous_update=False)\n",
    "a.style.handle_color = 'blue'\n",
    "d = widgets.FloatSlider(min=0.01, max = 30.0, value = 10.0, step = 1.0, description = 'Dataset: St. Deviation',orientation='vertical',layout=Layout(width='130px', height='200px'),continuous_update=False)\n",
    "d.style.handle_color = 'green'\n",
    "b = widgets.FloatSlider(min = 0, max = 100.0, value = 0.0, description = 'Dataset: Minimum',orientation='vertical',layout=Layout(width='130px', height='200px'),continuous_update=False)\n",
    "b.style.handle_color = 'red'\n",
    "c = widgets.IntSlider(min = 0, max = 100, value = 100, description = 'Dataset: Maximum',orientation='vertical',layout=Layout(width='130px', height='200px'),continuous_update=False)\n",
    "c.style.handle_color = 'orange'\n",
    "n = widgets.IntSlider(min = 2, max = 1000, value = 100, description = 'Dataset: Number Samples',orientation='vertical',layout=Layout(width='180px', height='200px'),continuous_update=False)\n",
    "n.style.handle_color = 'gray'\n",
    "\n",
    "ui = widgets.HBox([dist,a,d,b,c,n],)                      # basic widget formatting           \n",
    "ui2 = widgets.VBox([l,ui],)\n",
    "\n",
    "def f_make(dist,a, b, c, d, n):                           # function to take parameters, make sample and plot\n",
    "    global df\n",
    "    dataset = make_data(dist,a, b, c, d, n)\n",
    "    df = pd.DataFrame({'DataSet':dataset})\n",
    "    plt.subplot(111) \n",
    "    plt.hist(\n",
    "        dataset,\n",
    "        alpha=0.8,\n",
    "        color=\"darkorange\",\n",
    "        edgecolor=\"black\",\n",
    "        bins=bins) \n",
    "    plt.xlim(0.0,100.0); plt.title('Synthetic Dataset'); plt.ylabel('Frequency'); plt.xlabel('Data Values')\n",
    "            \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.6, wspace=0.2, hspace=0.3)\n",
    "    plt.show()\n",
    "    return df\n",
    "\n",
    "def make_data(dist,a, b, c, d, n):                        # function to check parameters and make sample   \n",
    "    if dist == 'Uniform':\n",
    "        if b >= c:\n",
    "            print('Invalid uniform distribution parameters')\n",
    "            return None\n",
    "        dataset = uniform.rvs(size=n, loc = b, scale = c, random_state = 73073).tolist()\n",
    "        return dataset\n",
    "    elif dist == 'Triangular':\n",
    "        interval = c - b\n",
    "        if b >= a or a >= c or interval <= 0:\n",
    "            print('Invalid triangular distribution parameters')\n",
    "            return None        \n",
    "        dataset = triang.rvs(size=n, loc = b, c = (a-b)/interval, scale = interval, random_state = 73073).tolist()\n",
    "        return dataset\n",
    "    elif dist == 'Gaussian':\n",
    "        dataset = norm.rvs(size=n, loc = a, scale = d, random_state = 73073).tolist()\n",
    "        return dataset\n",
    "\n",
    "# connect the function to make the samples and plot to the widgets    \n",
    "interactive_plot = widgets.interactive_output(f_make, {'dist': dist,'a': a, 'd': d, 'b': b, 'c': c, 'n': n})\n",
    "interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the GUI for Building the Synthetic Dataset\n",
    "\n",
    "We display the GUI now.  Select the desired parametric distribution and associated parameters.\n",
    "\n",
    "* if the parameters are invalid (e.g. traingular mode > max) an error message should display."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dffaa1ae75074199b7219cf5e1868252",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='                                      Boostrap Demonstration, Michael Pyrcz, Associ…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5aa317dca5ed47828b75b10e7a6c6e7e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui2, interactive_plot)                            # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a synthetic dataset to work with. Now we can:\n",
    "\n",
    "* assign the samples to a 1D ndarray \n",
    "\n",
    "* make a DataFrame with the samples\n",
    "\n",
    "* check the summary statistics\n",
    "\n",
    "This is our sample set that we will apply ot bootstrap.\n",
    "\n",
    "#### Perform Bootstrap \n",
    "\n",
    "Now we take our synthetic dataset, sampled from the parametric distributioin above, and apply it to statistical bootstrap.\n",
    "\n",
    "* we calculate the sampling distributions / uncertainty in the mean and standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5wAAAGWCAYAAAAUtoyeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAy2ElEQVR4nO3de7hddX3v+/fHRLkoCpRAQ8JNm1rBrWgj2upxW7GCV+hzNjbuY3dULLalKt12K7g9artPqj1Pq7h3Sy1aS7xUTL0RbatiLPVoq2lQFAHZpCAQCElEEbwUG/yeP8ZInQlrZc2srN9ac671fj3PeuYYv/EbY37Hb8w1f/M7rqkqJEmSJEmaaQ+Y6wAkSZIkSfOTCackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU2YcEqSJEmSmjDhlBaIJNckedosvM+xSb6XZNEMLe9NSd43E8uSJI0H+6y5l6SS/MwMLev4fnmLZ2J5+/je+7WNk7wuybtmOq6FxIRT+yzJN5P8sP/n/U6Sv0lyzAwsd5++jGbyi3DI93txks9PUP7NJM+YrTj699znL+6qOqmqrhhy+dNep6q6paoeUlX37eu8SZ6WZMt03nfI5V/St9vz9yi/sC9/cav3ljQ37LPuV26ftfv7jHKftTzJh5N8K8l3k1y9q5+aywSuhYH/03uS3JXkH5P8RpL9zlX2ZRtPtE2r6g+q6mX7G8dCZsKp6XpeVT0EWApsA/7XHMdzP/PlS3hP83W9ZtH/BlbvGunb8yzgX+YsIkmt2WfNkfm6XrPkvcCtwHHATwH/he7zO9L2Y5s/r6oOoVvftwCvBf5ixgLTnDHh1H6pqn8FPgScuKssycOSvCfJjiQ3J3n9rj1USR7Qj9+cZHtf72H9rJ/rX+/q90T/QpKfSfIP/Z69byX5YL+cXXW/2tf91V17pZK8NskdwF8mOSzJJ/pYvtMPLx+I9Yokb06ysX+Py5IcPt326I+g/Wm/B/2eJF9K8oiB6ScluTzJt5NsS/K6gXY5P8m/JLkzybpdcQzsxTw7yS3AZydpq0ck+Ww//7eSvD/JoQPv/e97gNOd8rOub/970p26tLKf9l7gWODj/bJf06/PK/ZY168lOXOCNthtr2vfxv8jyRf69/p0kiMmmO/BwN8BR/fv+70kR/eTHzRRrP18R6fbA7wjyU1JXjnFZvo48OQkh/XjpwNfA+7YI56XJrmu/9x8KslxA9PenuTWJHcnuTLJ/zEwbdK2lTS37LN2F/usUe+zngBcUlXfr6qdVfWVqvq7ftp02/R3+7b4bpIPJjlwYPp/S7I1ye1JXrrH+j4nyVfS9Xu3JnnTBG3479s8yaIkf9THcSPwnL2s526q6rtVtR74VWB1kkf373NAv8xb+s/jO5Ic1E+7LslzB2Ja3L/34yfYxi/p69+T5MYkL+/LJ9ym2eM06STP77frXf3n5VHDtvGCVVX++bdPf8A3gWf0wwcDa4H3DEx/D3AZcAhwPN0RpbP7aS8FNgMPBx4CfAR4bz/teKCAxQPL+gDw3+l2jhwIPGVgWgE/MzD+NGAn8IfAAcBBdHsE/88+zkOAvwY+NjDPFcBtwKOBBwMfBt43yXq/GPj8FO1xCfBt4BRgMfB+4NJ+2iHAVuDV/bocAjyxn3Ye8EVgeR/7nwMf2KNd3tPHeNAkbfUzwC/38y+h64wunCTONwH/CjwbWAS8GfjiRHX78RcAXxoYfyxwJ/CgCdpjt9j6Nv4X4Gf72K8A3jJJGz8N2LJH2aSx9p+LK4E3AA+i+1zdCJw2yfIvAf4f4GLgN/uydcALgc8DL+7LzqT7nD6q346vB/5xYDkvovtsLe635x3AgcO0rX/++Te7f9hn7a09LsE+a7fYGK0+6zPAF4BVwLF7i3sf2nQjcDRwOHAd8Bv9tNPpjp7u+mz9FQOf2X5d/0O/Do/p6565l23+G8A3gGP69/r7PeOd7HO5R/kt/KS/vhBY3y/vELodyG/up70BeP/AfM8BvjHJNn4O8AggwH8EfgA8fopt+r5++GeB7/ft/EDgNXTfEQ+aqo0X8t+cB+Df+P31/0zfA+6i6yxvB/5DP20RcC9w4kD9lwNX9MMbgN8amPZI4N/oOrrdvhD66e+hSw6WTxDHRJ33j+h/+E8S+8nAdwbGr2CgI6Hb6/0jYNEE876Y4Trvdw1Me/bAF94Lga9MEtd1wKkD40snaJeHD0y/X1tNsMwzB9+P+3fen9ljvX84Ud1+/AC6HyUr+vE/Ai6a5H13i61v49cPTP8t4JOTzPs0Jv6inzBW4InALXvUvwD4y0mWfwldwvkU4J+Ah9F1mgexe8L5d/Q/OPvxB9B1SMdNstzvAI8dpm3988+/2f3DPmui9rDPmiQ2RqvPOozu1NJrgPuAq4An7Gebvmhg/P8F3tEPv3uPz9bPssdndo9lXwi8bY9YBrf5ZxlItIBn7i3ePbfhQPkX6XbihC7Re8TAtF8AbuqHfwa4Bzi4H38/8IZh2gr4GPCqKbbproTz/wbWDUx7AN1OoKdN1cYL+c9TajVdZ1bVoXRf6r8N/EOSnwaOoNtrd/NA3ZuBZf3w0RNMWwwcNcn7vIbuS2Zjf/rCSyept8uO6k6ZAiDJwUn+PN3pUHfT7e07NLvfqezWPeJ5YL8ee9rZT9vTA+k62l0GT838Ad1ecej28k12neBxwEf70zPuouvM72P3drl1ohl3SXJkkkuT3Nav6/smWY/J4jwwk1x3UVX30h0JfFG6U81eSHdtybAma5Ppzr8r1uPoTn25a6DtXsfknycAqurzdHt/Xw98oqp+uEeV44C3Dyzz23Sfw2UASV7dn47z3X76w9i9rYduW0mzwj7rJ+yzpjYSfVZVfaeqzq+qk/o6VwEfS5KJ6g/ZppOt29Hc/7M1uOwnJvn7dKcCf5fuCOaeyx6cf6/L2wfL6PrgJXRH/q8caLtP9uVU1Wa6z+HzkhwMPJ/uKO39JHlWki+mO1X8LrodLXv77A3a7Tuhqn5Mt57LBurs7+dn3jHh1H6pqvuq6iN0Hc1TgG/RdWTHDVQ7lm7vD3R7lvectpPuKFNNsPw7qurXq+pour3OF2Xvd/nbcxmvptsj/cSqeijw1L588Mt68G6Fx/bxf2uCZd8CHDv4Rd9/qR3JcF+kt9KdwjHZtGdV1aEDfwdW1W0DdWqS4V3e3Jc/pl/XF7H7eu6LiZa/Fvi/gFOBH1TVP01z2fv6vntzK93ezcF2O6Sqnj3EvO+j+3y8Z5LlvnyP5R5UVf+Y7nrN19KdsnVY/yP2u0y/rSXNEvss+6wZNmt9VlV9i+5I7a5TNWe6Tbdy/8/WoL+iO531mKp6GPCOCZY9GNNUy5tSkifQJXKfp/uM/xA4aaDtHlbdzcB2+QDdzoUzgGv7JHTPZR5Adyr6HwFH9X343w6sy1TbdLfvhP7/6xh+8p2hCZhwar+kcwbdaR/XVXfL6XXAmiSHpLvRyn+l+3EP3ZfB7yQ5IclDgD8APlhVO4EdwI/prmnYtfyz8pMbJnyH7otg122ttw3WncQhdF9Qd6W7ocEbJ6jzoiQn9h3x7wMfqolvnf0luusyzk9yYH9x+VuATQzXeX8C+Okk56W78P2QJE/sp72Drs2O69d7Sd+uk7lfW/Xr+r1+XZcB/22ImCZzv7btO+sfA3/Mvu0p3tf3/an85KYcU9kI3J3uphsHpbtJwaP7Tmoq/5PuGozPTTDtHcAFSU6Cf7+pyFn9tEPofnDuABYneQPw0CHjlTSH7LPss2ZY0z4ryR/20xcnOQT4TWBzVd3JzLfpOuDFA5+tPT97hwDfrqp/TXIK8J+HWN4r0z3a5TDg/GEDSfLQdDcAupTuVNar+yOJ7wTeluTIvt6yJKcNzHop3am7v8kkRzfpzmg4gK79diZ5Vj/PLlNt03XAc5KcmuSBdDuJ7gX+cdj1W4hMODVdH0/yPeBuYA2wuqqu6ae9gu48+xvp9kr9Fd21AfSv76X7kX8TXWf4CoCq+kG/rC/0p0s8ie4ObV/q32s93Tn2N/XLehOwtq/7gknivJDu+rxv0V0H8MkJ6ryX7jqWO+hujDDhHeP6U3SeQ39+f79+RwMvqKop93JW1T10Cc7z+ve6AfilfvLb+/X7dJJ7+lifONFy+mVN1Fa/Bzye7mjb39Dd3GK63gy8vl/27w6Uv4fupgFNHmpdVd+g+4F3Y//eR09R/z669jyZ7vP0LeBddKe4TvVe366qDRNtu6r6KN2NPC7tT0v6OvCsfvKn6K7x/N90P9r+lSlOHZM05+yz7LNm3Cz0WQcDH6W7/vhGuiNrz++XNaNtWt3dby+ku/Zyc/866LeA3++39xvoEq+9eSddf/lV4MtDxvLxfvm30l23+VbgJQPTX9vH9sW+b/4M3RkBu9ZhK939GX4R+OBEb9B/rl/Zx/8dusR5/cD0vW7Tqrqe7sjx/6Lbfs+je5zLj4ZYvwUrQ3znSPNWkivo9p69a65jGQdJ/gtwTlU9Za5jkaSFxj5r39hnSaPBI5yShtKfYvNbdHdglCRpZNlnSaPDhFPSlPprJHbQXdsw2XURkiTNOfssabR4Sq0kSZIkqQmPcEqSJEmSmhjrB5EfccQRdfzxx891GJKkEXTllVd+q6qWzHUc48j+VZI0ken0rWOdcB5//PFs2rRprsOQJI2gJMM8a1ATsH+VJE1kOn2rp9RKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCYWz3UAkqbnec98Kltvu3nKekuXHcfHP/25WYhIkgR+P0vSIBNOaUxtve1mNl2wZMp6K9889Y8eSdLM8ftZkn7CU2olSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITPodTkiRpDtxxxzZWnnTclPWWLjuOj3/6c7MQkSTNvKYJZ5LfAV4GFHA18BLgYOCDwPHAN4EXVNV3+voXAGcD9wGvrKpPtYxPkiRprtSPd7LpgiVT1lv55ptnIRpJaqPZKbVJlgGvBFZW1aOBRcAq4HxgQ1WtADb04yQ5sZ9+EnA6cFGSRa3ikyRJkiS11foazsXAQUkW0x3ZvB04A1jbT18LnNkPnwFcWlX3VtVNwGbglMbxSZIkSZIaaZZwVtVtwB8BtwBbge9W1aeBo6pqa19nK3BkP8sy4NaBRWzpy3aT5Jwkm5Js2rFjR6vwJUmSJEn7qeUptYfRHbU8ATgaeHCSF+1tlgnK6n4FVRdX1cqqWrlkydTXPUiSJEmS5kbLU2qfAdxUVTuq6t+AjwC/CGxLshSgf93e198CHDMw/3K6U3AlSZIkSWOoZcJ5C/CkJAcnCXAqcB2wHljd11kNXNYPrwdWJTkgyQnACmBjw/gkSZIkSQ01eyxKVX0pyYeALwM7ga8AFwMPAdYlOZsuKT2rr39NknXAtX39c6vqvlbxSZIkSZLaavoczqp6I/DGPYrvpTvaOVH9NcCaljFJkiRJkmZH68eiSJIkSZIWKBNOSZIkSVITJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5KkMZNkUZKvJPlEP354ksuT3NC/HjZQ94Ikm5Ncn+S0uYtakrQQmXBKkjR+XgVcNzB+PrChqlYAG/pxkpwIrAJOAk4HLkqyaJZjlSQtYCackiSNkSTLgecA7xooPgNY2w+vBc4cKL+0qu6tqpuAzcApsxSqJEkmnJIkjZkLgdcAPx4oO6qqtgL0r0f25cuAWwfqbenL7ifJOUk2Jdm0Y8eOGQ9akrQwmXBKkjQmkjwX2F5VVw47ywRlNVHFqrq4qlZW1colS5ZMO0ZJkgYtnusAJEnS0J4MPD/Js4EDgYcmeR+wLcnSqtqaZCmwva+/BThmYP7lwO2zGrEkaUHzCKckSWOiqi6oquVVdTzdzYA+W1UvAtYDq/tqq4HL+uH1wKokByQ5AVgBbJzlsCVJC5hHOCVJGn9vAdYlORu4BTgLoKquSbIOuBbYCZxbVffNXZiSpIXGhFOSpDFUVVcAV/TDdwKnTlJvDbBm1gKTJGmAp9RKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITzRLOJI9MctXA391JzktyeJLLk9zQvx42MM8FSTYnuT7Jaa1ikyRJkiS11yzhrKrrq+rkqjoZ+HngB8BHgfOBDVW1AtjQj5PkRGAVcBJwOnBRkkWt4pMkSZIktTVbp9SeCvxLVd0MnAGs7cvXAmf2w2cAl1bVvVV1E7AZOGWW4pMkSZIkzbDZSjhXAR/oh4+qqq0A/euRffky4NaBebb0ZbtJck6STUk27dixo2HIkiRJkqT90TzhTPIg4PnAX09VdYKyul9B1cVVtbKqVi5ZsmQmQpQkSZIkNTAbRzifBXy5qrb149uSLAXoX7f35VuAYwbmWw7cPgvxSZIkSZIamI2E84X85HRagPXA6n54NXDZQPmqJAckOQFYAWychfgkSZIkSQ0sbrnwJAcDvwy8fKD4LcC6JGcDtwBnAVTVNUnWAdcCO4Fzq+q+lvFJkiRJktppmnBW1Q+An9qj7E66u9ZOVH8NsKZlTJIkSZKk2TFbd6mVJEmSJC0wJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLURNOEM8mhST6U5BtJrkvyC0kOT3J5khv618MG6l+QZHOS65Oc1jI2SZIkSVJbrY9wvh34ZFX9HPBY4DrgfGBDVa0ANvTjJDkRWAWcBJwOXJRkUeP4JEmSJEmNNEs4kzwUeCrwFwBV9aOqugs4A1jbV1sLnNkPnwFcWlX3VtVNwGbglFbxSZIkSZLaanmE8+HADuAvk3wlybuSPBg4qqq2AvSvR/b1lwG3Dsy/pS/bTZJzkmxKsmnHjh0Nw5ckSZIk7Y+WCedi4PHAn1XV44Dv058+O4lMUFb3K6i6uKpWVtXKJUuWzEykkiRJkqQZ1zLh3AJsqaov9eMfoktAtyVZCtC/bh+of8zA/MuB2xvGJ0mSJElqqFnCWVV3ALcmeWRfdCpwLbAeWN2XrQYu64fXA6uSHJDkBGAFsLFVfJIkSZKkthY3Xv4rgPcneRBwI/ASuiR3XZKzgVuAswCq6pok6+iS0p3AuVV1X+P4JEmSJEmNNE04q+oqYOUEk06dpP4aYE3LmCRJkiRJs6P1czglSZIkSQuUCackSZIkqQkTTkmSJElSEyackiSNiSQHJtmY5KtJrknye3354UkuT3JD/3rYwDwXJNmc5Pokp81d9JKkhciEU5Kk8XEv8PSqeixwMnB6kicB5wMbqmoFsKEfJ8mJwCrgJOB04KIki+YicEnSwmTCKUnSmKjO9/rRB/Z/BZwBrO3L1wJn9sNnAJdW1b1VdROwGThl9iKWJC10JpySJI2RJIuSXAVsBy6vqi8BR1XVVoD+9ci++jLg1oHZt/RlEy33nCSbkmzasWNHs/glSQuLCackSWOkqu6rqpOB5cApSR69l+qZaBGTLPfiqlpZVSuXLFkyA5FKkmTCKUnSWKqqu4Ar6K7N3JZkKUD/ur2vtgU4ZmC25cDtsxelJGmhM+GUJGlMJFmS5NB++CDgGcA3gPXA6r7aauCyfng9sCrJAUlOAFYAG2c1aEnSgrZ4rgOQJElDWwqs7e80+wBgXVV9Isk/AeuSnA3cApwFUFXXJFkHXAvsBM6tqvvmKHZJ0gJkwilJ0pioqq8Bj5ug/E7g1EnmWQOsaRyaJEkT8pRaSZIkSVITJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmmiacSb6Z5OokVyXZ1JcdnuTyJDf0r4cN1L8gyeYk1yc5rWVskiRJkqS2ZuMI5y9V1clVtbIfPx/YUFUrgA39OElOBFYBJwGnAxclWTQL8UmSJEmSGpiLU2rPANb2w2uBMwfKL62qe6vqJmAzcMrshydJkiRJmgmtE84CPp3kyiTn9GVHVdVWgP71yL58GXDrwLxb+rLdJDknyaYkm3bs2NEwdEmSJEnS/ljcePlPrqrbkxwJXJ7kG3upmwnK6n4FVRcDFwOsXLnyftMlSZIkSaNhqCOcSR49nYVX1e3963bgo3SnyG5LsrRf7lJge199C3DMwOzLgdun876SJI266fatkiSNk2FPqX1Hko1JfivJocPMkOTBSQ7ZNQw8E/g6sB5Y3VdbDVzWD68HViU5IMkJwApg45DxSZI0bva5b5UkadwMdUptVT0lyQrgpcCmJBuBv6yqy/cy21HAR5Psep+/qqpPJvlnYF2Ss4FbgLP697gmyTrgWmAncG5V3TfdFZMkaZRNs2+VJGmsDH0NZ1XdkOT1wCbgfwKPS5dNvq6qPjJB/RuBx05Qfidw6iTvsQZYM2xMkiSNs33tWyVJGjfDXsP5mCRvA64Dng48r6oe1Q+/rWF8kiTNS/atkqSFYNgjnH8CvJNuj+sPdxX2d6B9fZPIJEma3+xbJUnz3rAJ57OBH+66pjLJA4ADq+oHVfXeZtFJkjR/2bdKkua9Ye9S+xngoIHxg/sySZI0PfatkqR5b9iE88Cq+t6ukX744DYhSZK0INi3SpLmvWETzu8nefyukSQ/D/xwL/UlSdLe2bdKkua9Ya/hPA/46yS39+NLgV9tEpEkSQvDedi3SpLmuaESzqr65yQ/BzwSCPCNqvq3ppFJkjSP2bdKkhaCYY9wAjwBOL6f53FJqKr3NIlKkqSFwb5VkjSvDZVwJnkv8AjgKuC+vrgAO0VJkqbBvlWStBAMe4RzJXBiVVXLYCRJWkDsWyVJ896wd6n9OvDTLQORJGmBsW+VJM17wx7hPAK4NslG4N5dhVX1/CZRSZI0/9m3SpLmvWETzje1DEKSpAXoTXMdgCRJrQ37WJR/SHIcsKKqPpPkYGBR29AkSZq/7FslSQvBUNdwJvl14EPAn/dFy4CPNYpJkqR5z75VkrQQDHvToHOBJwN3A1TVDcCRrYKSJGkBsG+VJM17wyac91bVj3aNJFlM96wwSZI0PfatkqR5b9iE8x+SvA44KMkvA38NfLxdWJIkzXv2rZKkeW/YhPN8YAdwNfBy4G+B17cKSpKkBcC+VZI07w17l9ofA+/s/yRJ0n6yb5UkLQRDJZxJbmKC60qq6uEzHpEkSQuAfaskaSEYKuEEVg4MHwicBRw+8+FIkrRg2LdKkua9oa7hrKo7B/5uq6oLgae3DU2SpPnLvlWStBAMe0rt4wdGH0C3V/aQJhFJkrQA2LdKkhaCYU+p/eOB4Z3AN4EXzHg0kiQtHPatkqR5b9i71P5S60AkSVpI7FslSQvBsKfU/te9Ta+qt85MOJIkLQz2rZKkhWBf7lL7BGB9P/484HPArS2CkiRpAbBvlSTNe8MmnEcAj6+qewCSvAn466p6WavAJEma5+xbJUnz3lCPRQGOBX40MP4j4PgZj0aSpIXDvlWSNO8Ne4TzvcDGJB8FCvgV4D3DzJhkEbAJuK2qnpvkcOCDdJ3qN4EXVNV3+roXAGcD9wGvrKpPDb8qkiSNlWn3rZIkjYuhjnBW1RrgJcB3gLuAl1TVHwz5Hq8CrhsYPx/YUFUrgA39OElOBFYBJwGnAxf1yaokSfPOfvatkiSNhWFPqQU4GLi7qt4ObElywlQzJFkOPAd410DxGcDafngtcOZA+aVVdW9V3QRsBk7Zh/gkSRo3+9y3SpI0ToZKOJO8EXgtcEFf9EDgfUPMeiHwGuDHA2VHVdVWgP71yL58GbvfmW9LX7ZnLOck2ZRk044dO4YJX5KkkbMffaskSWNj2COcvwI8H/g+QFXdDhyytxmSPBfYXlVXDvkemaCs7ldQdXFVrayqlUuWLBly0ZIkjZx97lslSRo3w9406EdVVUkKIMmDh5jnycDzkzwbOBB4aJL3AduSLK2qrUmWAtv7+luAYwbmXw7cPmR8kiSNm+n0rZIkjZVhj3CuS/LnwKFJfh34DPDOvc1QVRdU1fKqOp7uZkCfraoX0T3genVfbTVwWT+8HliV5ID+GpYVwMZ9WhtJksbHPvetkiSNmymPcCYJ3WNMfg64G3gk8Iaqunya7/kWuk72bOAW4CyAqromyTrgWmAncG5V3TfN95AkaWQ16FslSRpJUyac/ek+H6uqnwem1RFW1RXAFf3wncCpk9RbA6yZzntIkjQuptu3JjmG7lmdP013Q76Lq+rtPuNakjSqhj2l9otJntA0EkmSFpbp9K07gVdX1aOAJwHn9s+x9hnXkqSRNGzC+Ut0HeO/JPlakquTfK1lYJIkzXP73LdW1daq+nI/fA9wHd0jxHzGtSRpJO31lNokx1bVLcCzZikeSZLmtZnqW5McDzwO+BJ7POM6yeAzrr84MNuEz7jul3cOcA7Ascceuz+hSZL076Y6wvkxgKq6GXhrVd08+Nc8OkmS5p+Pwf71rUkeAnwYOK+q7t5b1QnK7veM6z4en3MtSZpxUyWcgx3Vw1sGIknSArFffWuSB9Ilm++vqo/0xdv6Z1vjM64lSaNkqoSzJhmWJEnTM+2+tX+cyl8A11XVWwcm+YxrSdJImuqxKI9Ncjfd3tiD+mH68aqqhzaNTpKk+Wd/+tYnA78GXJ3kqr7sdfiMa0nSiNprwllV3jpdkqQZtD99a1V9nomvywSfcS1JGkHDPhZFkiRJkqR9YsIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmmiWcSQ5MsjHJV5Nck+T3+vLDk1ye5Ib+9bCBeS5IsjnJ9UlOaxWbJEmSJKm9lkc47wWeXlWPBU4GTk/yJOB8YENVrQA29OMkORFYBZwEnA5clGRRw/gkSZIkSQ01Szir871+9IH9XwFnAGv78rXAmf3wGcClVXVvVd0EbAZOaRWfJEmSJKmtptdwJlmU5CpgO3B5VX0JOKqqtgL0r0f21ZcBtw7MvqUv23OZ5yTZlGTTjh07WoYvSZIkSdoPTRPOqrqvqk4GlgOnJHn0XqpnokVMsMyLq2plVa1csmTJDEUqSZIkSZpps3KX2qq6C7iC7trMbUmWAvSv2/tqW4BjBmZbDtw+G/FJkiRJkmZey7vULklyaD98EPAM4BvAemB1X201cFk/vB5YleSAJCcAK4CNreKTJEmSJLW1uOGylwJr+zvNPgBYV1WfSPJPwLokZwO3AGcBVNU1SdYB1wI7gXOr6r6G8UmSJEmSGmqWcFbV14DHTVB+J3DqJPOsAda0ikmSJEmSNHtm5RpOSZIkSdLCY8IpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCZMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCZMOCVJkiRJTTRLOJMck+Tvk1yX5Jokr+rLD09yeZIb+tfDBua5IMnmJNcnOa1VbJIkSZKk9loe4dwJvLqqHgU8CTg3yYnA+cCGqloBbOjH6aetAk4CTgcuSrKoYXySJEmSpIaaJZxVtbWqvtwP3wNcBywDzgDW9tXWAmf2w2cAl1bVvVV1E7AZOKVVfJIkSZKktmblGs4kxwOPA74EHFVVW6FLSoEj+2rLgFsHZtvSl+25rHOSbEqyaceOHU3jliRJkiRNX/OEM8lDgA8D51XV3XurOkFZ3a+g6uKqWllVK5csWTJTYUqSNBaSvDvJ9iRfHyjz/giSpJHUNOFM8kC6ZPP9VfWRvnhbkqX99KXA9r58C3DMwOzLgdtbxidJ0hi6hO5eB4O8P4IkaSS1vEttgL8Arquqtw5MWg+s7odXA5cNlK9KckCSE4AVwMZW8UmSNI6q6nPAt/co9v4IkqSRtLjhsp8M/BpwdZKr+rLXAW8B1iU5G7gFOAugqq5Jsg64lu4Ot+dW1X0N45Mkab7Y7f4ISQbvj/DFgXoT3h8BunskAOcAHHvssQ1DlSQtJM0Szqr6PBNflwlw6iTzrAHWtIpJkqQFZqj7I0B3jwTgYoCVK1dOWEeSpH01K3eplSRJTXl/BEnSSDLhlCRp/Hl/BEnSSGp5DackSZphST4APA04IskW4I14fwRJ0ogy4ZQkaYxU1QsnmeT9ESRJI8dTaiVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqYnFcx2AJEmSJnfHHdtYedJxU9Zbuuw4Pv7pz81CRJI0PBNOSZKkEVY/3smmC5ZMWW/lm2+ehWgkad94Sq0kSZIkqQkTTkmSJElSEyackiRJkqQmvIZTmue82YQkSZLmigmnNM95swlJkiTNFU+plSRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITzRLOJO9Osj3J1wfKDk9yeZIb+tfDBqZdkGRzkuuTnNYqLkmSJEnS7Gj5WJRLgD8B3jNQdj6woarekuT8fvy1SU4EVgEnAUcDn0nys1V1X8P4JEmShva8Zz6VrbdN/Qip7du2AVM/jkqSFoJmCWdVfS7J8XsUnwE8rR9eC1wBvLYvv7Sq7gVuSrIZOAX4p1bxSZIk7Yutt9081HONl73itlmIRpLGw2xfw3lUVW0F6F+P7MuXAbcO1NvSl91PknOSbEqyaceOHU2DlSRJkiRN36jcNCgTlNVEFavq4qpaWVUrlyzxdBVJkiRJGlWznXBuS7IUoH/d3pdvAY4ZqLccuH2WY5MkSZIkzaDZTjjXA6v74dXAZQPlq5IckOQEYAWwcZZjkyRJkiTNoGY3DUryAbobBB2RZAvwRuAtwLokZwO3AGcBVNU1SdYB1wI7gXO9Q60kSZIkjbeWd6l94SSTTp2k/hpgTat4JEmSJEmza1RuGiRJkiRJmmdMOCVJkiRJTZhwSpIkSZKaMOGUJEmSJDVhwilJkiRJasKEU5IkSZLUhAmnJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSEyackiRJkqQmTDglSZIkSU0snusAJO3uec98Kltvu3nKetu3bQOWtA9IkiRJmiYTTmnEbL3tZjZdMHUiuewVt81CNJIkSdL0eUqtJEmSJKkJE05JkiRJUhMmnJIkSZKkJkw4JUmSJElNmHBKkiRJkpow4ZQkSZIkNWHCKUmSJElqwoRTkiRJktSECackSZIkqQkTTkmSJElSE4vnOgBJkiTtvzvu2MbKk46bst7SZcfx8U9/bhYikqQRTDiTnA68HVgEvKuq3jLHIUmSNNbsWxeG+vFONl2wZMp6K9988yxEI0mdkTqlNski4E+BZwEnAi9McuLcRiVJ0viyb5UkzaVRO8J5CrC5qm4ESHIpcAZw7ZxGJUnS+LJvncLznvlUtt429VG/7du2AVMfQZQk/USqaq5j+HdJ/hNwelW9rB//NeCJVfXbA3XOAc7pRx8NfH3WA50fjgC+NddBjCHbbfpsu+mz7abnkVV1yFwHMdeG6Vv78vnWv86H/xvXYTS4DqPBdRgN+9y3jtoRzkxQtltGXFUXAxcDJNlUVStnI7D5xrabHttt+my76bPtpifJprmOYURM2bfC/OtfXYfR4DqMBtdhNMyXddjXeUbqGk5gC3DMwPhy4PY5ikWSpPnAvlWSNGdGLeH8Z2BFkhOSPAhYBayf45gkSRpn9q2SpDkzUqfUVtXOJL8NfIru1u3vrqpr9jLLxbMT2bxk202P7TZ9tt302XbTY7sxrb4V5kfbuQ6jwXUYDa7DaFiQ6zBSNw2SJEmSJM0fo3ZKrSRJkiRpnjDhlCRJkiQ1MbYJZ5LTk1yfZHOS8+c6nlGV5Jgkf5/kuiTXJHlVX354ksuT3NC/HjbXsY6iJIuSfCXJJ/px220ISQ5N8qEk3+g/e79g2w0nye/0/6tfT/KBJAfadhNL8u4k25N8faBs0rZKckHfZ1yf5LS5iXq0zYe+Nck3k1yd5KpxejTOvn6eR9Ek6/CmJLf12+OqJM+eyxj3Zj78ZtrLOozNdgDo+76NSb7ar8fv9eXjtC0mW4dx2xb7/Vt4LBPOJIuAPwWeBZwIvDDJiXMb1cjaCby6qh4FPAk4t2+r84ENVbUC2NCP6/5eBVw3MG67DeftwCer6ueAx9K1oW03hSTLgFcCK6vq0XQ3eFmFbTeZS4DT9yibsK36771VwEn9PBf1fYl686xv/aWqOnnMnnd3CUN+nkfYJdx/HQDe1m+Pk6vqb2c5pn0xH34zTbYOMD7bAeBe4OlV9VjgZOD0JE9ivLbFZOsA47Ut9vu38FgmnMApwOaqurGqfgRcCpwxxzGNpKraWlVf7ofvofvALKNrr7V9tbXAmXMS4AhLshx4DvCugWLbbQpJHgo8FfgLgKr6UVXdhW03rMXAQUkWAwfTPS/RtptAVX0O+PYexZO11RnApVV1b1XdBGym60v0E/atc2gfP88jaZJ1GBvz4TfTXtZhrFTne/3oA/u/Yry2xWTrMDZm6rfwuCacy4BbB8a3MIb/TLMtyfHA44AvAUdV1VbovpyAI+cwtFF1IfAa4McDZbbb1B4O7AD+sj8F411JHoxtN6Wqug34I+AWYCvw3ar6NLbdvpisrew3pjZf2qiATye5Msk5cx3Mfpov//u/neRr/Sm3I3sK5KD58Jtpj3WAMdsO/amcVwHbgcurauy2xSTrAOOzLS5kBn4Lj2vCmQnKxmqPwWxL8hDgw8B5VXX3XMcz6pI8F9heVVfOdSxjaDHweODPqupxwPcZ7VNeRkbf6ZwBnAAcDTw4yYvmNqp5w35javOljZ5cVY+nOzX43CRPneuAFrg/Ax5Bd0rhVuCP5zSaIcyH30wTrMPYbYequq+qTgaWA6ckefQch7TPJlmHsdgWM/lbeFwTzi3AMQPjy+lOO9MEkjyQ7kvn/VX1kb54W5Kl/fSldHte9BNPBp6f5Jt0p5U9Pcn7sN2GsQXYMrAX70N0CahtN7VnADdV1Y6q+jfgI8AvYtvti8nayn5javOijarq9v51O/BRxvvU6bH/36+qbf2P7h8D72TEt8d8+M000TqM23YY1F+WcwXd9cFjtS12GVyHMdoWM/ZbeFwTzn8GViQ5IcmD6G4EsX6OYxpJSUJ3Ld11VfXWgUnrgdX98GrgstmObZRV1QVVtbyqjqf7fH22ql6E7TalqroDuDXJI/uiU4Frse2GcQvwpCQH9/+7p9Jdf2PbDW+ytloPrEpyQJITgBXAxjmIb5SNfd+a5MFJDtk1DDwT+Pre5xppY/+/v+uHae9XGOHtMR9+M022DuO0HQCSLElyaD98EN0O2W8wXttiwnUYl20xk7+FUzWOZ8tAfwvhC+nu4vjuqloztxGNpiRPAf4/4Gp+cv716+jO518HHEv3I/esqhrbC/1bSvI04Her6rlJfgrbbUpJTqa7wPxBwI3AS+h2cNl2U+hvm/6rdHca/ArwMuAh2Hb3k+QDwNOAI4BtwBuBjzFJWyX578BL6dr2vKr6u9mPerSNe9+a5OF0RzWhO73/r8ZlHfb18zyKJlmHp9GdOljAN4GX77r+a9TMh99Me1mHFzIm2wEgyWPobkiziP73Q1X9/jj9DtvLOryXMdoWsP+/hcc24ZQkSZIkjbZxPaVWkiRJkjTiTDglSZIkSU2YcEqSJEmSmjDhlCRJkiQ1YcIpSZIkSWrChFMaEUmuSHLaHmXnJbloL/VXzk50kiSNJ/tXaW6ZcEqj4wN0D9YdtKovlyRJ02P/Ks0hE05pdHwIeG6SAwCSHA8cDfznJJuSXJPk9yaaMcn3Bob/U5JL+uElST6c5J/7vyf35f8xyVX931eSHNJ43SRJmiv2r9IcWjzXAUjqVNWdSTYCpwOX0e19/SDw5qr6dpJFwIYkj6mqrw252LcDb6uqzyc5FvgU8Cjgd4Fzq+oLSR4C/OuMr5AkSSPA/lWaWx7hlEbL4Gk/u073eUGSLwNfAU4CTtyH5T0D+JMkVwHrgYf2e1u/ALw1ySuBQ6tq5wzFL0nSKLJ/leaICac0Wj4GnJrk8cBBwHfo9paeWlWPAf4GOHCC+WpgeHD6A4BfqKqT+79lVXVPVb0FeFn/Hl9M8nMN1kWSpFHxMexfpTlhwimNkKr6HnAF8G66va8PBb4PfDfJUcCzJpl1W5JHJXkA8CsD5Z8GfnvXSJKT+9dHVNXVVfWHwCbADlGSNG/Zv0pzx4RTGj0fAB4LXFpVX6U71ecauk7yC5PMcz7wCeCzwNaB8lcCK5N8Lcm1wG/05ecl+XqSrwI/BP5u5ldDkqSRYv8qzYFU1dS1JEmSJEnaRx7hlCRJkiQ1YcIpSZIkSWrChFOSJEmS1IQJpyRJkiSpCRNOSZIkSVITJpySJEmSpCZMOCVJkiRJTfz/6PRhFoG0gHQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary Statistics for Bootstrap for Uncertainty in the Mean:\n",
      "DescribeResult(nobs=1000, minmax=(46.4911862844161, 53.39117667902819), mean=50.268604394491426, variance=0.8519541651807531, skewness=0.08312739639977067, kurtosis=0.1175301689146524)\n",
      "P10: 49.064, P50: 50.261, P90: 51.421\n",
      "\n",
      "Summary Statistics for Bootstrap for Uncertainty in the Standard Deviation:\n",
      "DescribeResult(nobs=1000, minmax=(7.158715123778451, 11.71402702898678), mean=9.276719117116285, variance=0.4816916908140478, skewness=0.23332808535545702, kurtosis=0.08924833308776536)\n",
      "P10: 8.434, P50: 9.241, P90: 10.152\n"
     ]
    }
   ],
   "source": [
    "L = 1000                                                  # set the number of realizations\n",
    "mean = np.zeros(L); stdev = np.zeros(L)                   # declare arrays to hold the realizations of the statistics\n",
    "for l in range(0, L):                                     # loop over realizations\n",
    "    samples = random.choices(df['DataSet'].values, weights=None, cum_weights=None, k=len(df))\n",
    "    mean[l] = np.average(samples)\n",
    "    stdev[l] = np.std(samples)\n",
    "    \n",
    "plt.subplot(121)                                          # plot the distribution for uncertainty in the mean     \n",
    "plt.hist(mean,alpha=0.8,color=\"darkorange\",edgecolor=\"black\",bins=np.linspace(0,100,40)); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')\n",
    "\n",
    "plt.subplot(122)                                          # plot the distribution for uncertainty in the standard deviation\n",
    "plt.hist(stdev,alpha=0.8,color=\"darkorange\",edgecolor=\"black\",bins=np.linspace(0,40,40)); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')\n",
    "\n",
    "plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)\n",
    "plt.show()   \n",
    "\n",
    "# provide summary statistics, P10, P50 and P90\n",
    "print('Summary Statistics for Bootstrap for Uncertainty in the Mean:') \n",
    "print(stats.describe(mean))\n",
    "print('P10: ' + str(round(np.percentile(mean,10),3)) + ', P50: ' + str(round(np.percentile(mean,50),3)) + ', P90: ' + str(round(np.percentile(mean,90),3))) \n",
    "\n",
    "print('\\nSummary Statistics for Bootstrap for Uncertainty in the Standard Deviation:')\n",
    "print(stats.describe(stdev))\n",
    "print('P10: ' + str(round(np.percentile(stdev,10),3)) + ', P50: ' + str(round(np.percentile(stdev,50),3)) + ', P90: ' + str(round(np.percentile(stdev,90),3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Change the Number of Data\n",
    "\n",
    "Let's change the number of data drawn to observe the change in uncertainty\n",
    "\n",
    "* we will assume the same dataset and not recalculate it each time\n",
    "\n",
    "* we will just sample with replacement with the new number of samples for each bootstrap realization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters for the synthetic dataset\n",
    "bins = np.linspace(0,1000,1000)\n",
    "\n",
    "l = widgets.Text(value='                           Boostrap Demonstration with Modified Number of Data, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))\n",
    "\n",
    "n = widgets.IntSlider(min = 2, max = 1000, value = 100, description = 'New Number Samples',orientation='horizontal',layout=Layout(width='800px', height='20px'),continuous_update=False)\n",
    "n.style.handle_color = 'gray'\n",
    "\n",
    "ui3 = widgets.VBox([l,n],)\n",
    "\n",
    "def f_rerun(n):\n",
    "    L = 1000                                 # set the number of realizations\n",
    "    mean2 = np.zeros(L); stdev2 = np.zeros(L)    # declare arrays to hold the realizations of the statistics\n",
    "    for l in range(0, L):                      # loop over realizations\n",
    "        samples = random.choices(df['DataSet'].values, weights=None, cum_weights=None, k=n)\n",
    "        mean2[l] = np.average(samples)\n",
    "        stdev2[l] = np.std(samples)\n",
    "    \n",
    "    plt.subplot(121)\n",
    "    plt.hist(mean,alpha=0.5,color=\"red\",edgecolor=\"black\",bins=np.linspace(0,100,40),label='Original'); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')\n",
    "    plt.hist(mean2,alpha=0.5,color=\"blue\",edgecolor=\"black\",bins=np.linspace(0,100,40),label='New'); plt.xlim(0,100); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Mean')\n",
    "    plt.legend()\n",
    "\n",
    "    plt.subplot(122)\n",
    "    plt.hist(stdev,alpha=0.5,color=\"red\",edgecolor=\"black\",bins=np.linspace(0,40,40),label='Original'); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')\n",
    "    plt.hist(stdev2,alpha=0.5,color=\"blue\",edgecolor=\"black\",bins=np.linspace(0,40,40),label='New'); plt.xlim(0,40); plt.ylabel('Frequency'); plt.xlabel('Values'); plt.title('Bootstrap Uncertainty in the Standard Deviation')\n",
    "    plt.legend()\n",
    "    \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)  \n",
    "    plt.show()  \n",
    "\n",
    "interactive_plot3 = widgets.interactive_output(f_rerun, {'n': n})\n",
    "interactive_plot3.clear_output(wait = True)                # reduce flickering by delaying plot updating"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the GUI for Modifying the Number of Data\n",
    "\n",
    "We display the GUI now. Select the desired number of data\n",
    "\n",
    "* observed the change (new) vs. the original (original) bootstrap uncertainty as the number of data ranges from less than to greater than the original number of data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43dec4b70b99457bac577788df647617",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='                           Boostrap Demonstration with Modified Number of Data, Mic…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dfa183b6c384f8db5617708d2ec39ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '<Figure size 432x288 with 2 Axes>', 'i…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui3, interactive_plot3)                            # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observations\n",
    "\n",
    "Some observations form changing the number of data:\n",
    "\n",
    "* with less data the uncertainty increases\n",
    "\n",
    "* with more data the uncertianty decreases \n",
    "\n",
    "* there is a bias low in standard deviation with too few samples, as we fail to observe the dispersion well\n",
    "\n",
    "\n",
    "#### Comments\n",
    "\n",
    "This was a simple demonstration of interactive plots in Jupyter Notebook Python with the ipywidgets and matplotlib packages. \n",
    "\n",
    "I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy. \n",
    "  \n",
    "I hope this was helpful,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "#### The Author:\n",
    "\n",
    "### Michael Pyrcz, Associate Professor, University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}