Lecture14_Binary_Classification_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classfication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input vector $\\mathbb{x}=(x_1,...,x_m)$와, input vertor가 속할 수 있는 category list $\\mathbb{y}=\\{1,...,k\\}$가 주어졌다고 가정하자. Classification은 input data $\\mathbb{x}$에 대해서, $\\mathbb{x}$가 $k$개의 카테고리 중에서 어디에 속하는지 결정하고 대응시켜주는 task이다. Leaning 알고리즘은 $\\mathbb{x}$와 $\\mathbb{x}$가 대응되는 카테고리를 mapping 하는 함수를 학습한다. 함수는 다음과 같이 표현된다. \n",
    "$$f:\\mathbb{R}^n \\to \\{1,...,k\\}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifier의 종류\n",
    "- Naive Bayesian Classifier\n",
    "- Logistic Regression\n",
    "- Decsion Tree\n",
    "- Ensemble model\n",
    " - Bagging\n",
    " - Random forest \n",
    " - Gradient Boosting\n",
    " - Adaboost\n",
    " - XGBoost\n",
    "- SVM\n",
    "- Perceptron\n",
    "- Deep Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST와 scikit-learn을 활용한 binary classification tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 70,000개의 숫자 이미지\n",
    "- 각 이미지에 어떤 숫자가 대응되는지 labeling (0, ... ,9 까지의 정수)\n",
    "- 새로운 분류 알고리즘이 나올 때마다 MNIST 데이터 셋에서 얼마나 잘 작동하는지를 테스트\n",
    "- 머신러닝 알고리즘 영역의 hello world 데이터 셋"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/mnist.png\" style=\"height:400px width:400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Keras를 이용하여 데이터 셋 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이미지는 총 70,000개 (train : 60,000개 / test : 10,000개)\n",
    "- Feature의 갯수(pixel의 갯수)는 784개 ($28 \\times 28$)이며, 각 entry는 0~255까지의 픽셀 강도를 나타냄"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAABnhJREFUeJzt3T1sjX8fx3Ft6mmSiBaJjVgkRrFYlIRqwiBGsYiISCQWRDqS6CYisVikgyCSLo1Et0bCJDRNrEwWD00Qiv4X93h9T+9znLb6eb3Wj+tcZ+jbNfzac3rm5+dXAXl6l/oNAEtD/BBK/BBK/BBK/BBK/BBK/BBK/BBK/BCqb5Hv59cJoft6FvKPPPkhlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghlPghVN9SvwFWrXr37l25X79+vdxfv37duE1NTbX1nhbq6NGj5f7x48fGbdeuXeW1e/bsKfdTp06VOzVPfgglfgglfgglfgglfgglfgglfgjVMz8/v5j3W9SbLZaXL1+W+40bN8r92bNn5f727dv/+z39z6ZNm8p9586d5d7qvXVTf39/ub9//36R3sk/p2ch/8iTH0KJH0KJH0KJH0KJH0KJH0L5k94/7t27V+5nz55t3Obm5sprW+2Dg4PlPj4+Xu47duxo3Hp76//f+/rqH4EfP36U+6FDh8q9239STPs8+SGU+CGU+CGU+CGU+CGU+CGU+CGUc/4/Pn/+XO5fv35t+7U3b95c7qOjo+W+e/futu/dqVa/B9Dq9wg6MTw83LXXxpMfYokfQokfQokfQokfQokfQokfQvno7j9+/fpV7tVXTbeyevXqct+wYUPbr91t09PT5d7qLL762PF169aV1z58+LDch4aGyj2Yj+4GmokfQokfQokfQokfQokfQokfQjnnp7R27dpyb/WdBNVZ/qVLl8prR0ZGyp1GzvmBZuKHUOKHUOKHUOKHUOKHUOKHUD63fwWYnZ1t3O7fv19ee+3atXJvdY6/Zs2acr9y5UrjdvXq1fJausuTH0KJH0KJH0KJH0KJH0KJH0I56lsGvnz5Uu6nT58u94mJicat1VePd2rfvn3lfvLkya7en/Z58kMo8UMo8UMo8UMo8UMo8UMo8UMoH929DHz69Knct2zZUu6/f/9u3H7+/NnWe/pbBgYGGreNGzeW1545c6bcz58/X+69vbHPNh/dDTQTP4QSP4QSP4QSP4QSP4QSP4Ryzr8CTE9PN24vXrzo6LVv3rxZ7q9evero9TsxODhY7mNjY41b9fsHK4BzfqCZ+CGU+CGU+CGU+CGU+CGU+CGUc35K3759K/eZmZlyf/r0aeN2+fLltt7TQo2Pjzduw8PDXb33EnPODzQTP4QSP4QSP4QSP4QSP4QSP4Ryzk9XVT9fQ0ND5bVPnjzp6N4XL15s3EZHRzt67WXOOT/QTPwQSvwQSvwQSvwQSvwQqm+p3wArW09P86lTtf0N27dv7+rr/+s8+SGU+CGU+CGU+CGU+CGU+CGU+CGUc3666sGDB43b5ORkV+994MCBrr7+v86TH0KJH0KJH0KJH0KJH0KJH0KJH0I556cjU1NT5T4yMtK4zc3NdXTvY8eOlfvWrVs7ev2VzpMfQokfQokfQokfQokfQokfQokfQvmKbkp3794t93PnzpX79+/f2773tm3byv3Nmzflvn79+rbv/Y/zFd1AM/FDKPFDKPFDKPFDKPFDKH/Su8LNzMyU+61bt8r9zp075d7JUXF/f3+5P3r0qNyDj/L+Ck9+CCV+CCV+CCV+CCV+CCV+CCV+COWcf4Gq8/KJiYny2sOHD5f7hw8fyv358+flPj093bg9fvy4vHZ2drbcW+nrq3+Ejhw50rjdvn27vNZHb3eXJz+EEj+EEj+EEj+EEj+EEj+EEj+E8tHdC3Tw4MHGbXJychHfyeLau3dvuV+4cKHcT5w48TffDgvjo7uBZuKHUOKHUOKHUOKHUOKHUOKHUP6ef4GOHz/euC3nc/6BgYFyHxsbK/f9+/eXe0/Pgo6UWYY8+SGU+CGU+CGU+CGU+CGU+CGU+CGUv+eHlcff8wPNxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hxA+hFvsrun2fMywTnvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQSvwQ6j+LPQOfzQcAwQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 이미지 데이터를 plotting\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "digit = x_train[10000]\n",
    "plt.imshow(digit, cmap=matplotlib.cm.binary, interpolation='nearest')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 그 이미지의 label\n",
    "y_train[10000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tip : **Training set을 섞어서 모든 교차 검증 폴드가 비슷하게 만들어야 한다. (즉, 특정 폴드에서, 특정 숫자가 누락되면 안된다.)**\n",
    "     - 데이터셋을 랜덤하게 섞는다.\n",
    "     - 특정 카테고리에 속하는 데이터가 연속적으로 나오지 않도록 한다. (모델이 순서를 학습할 수 있다.)\n",
    "     - 단, 시계열 데이터는 섞으면 안된다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Binary classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "문제를 단순하게 하기 위해서,\n",
    " - 숫자 5이다.\n",
    " - 숫자 5가 아니다.\n",
    " \n",
    "로 분류"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_5 = (y_train == 5)  # 5는 True, 그 외는 False\n",
    "y_test_5 = (y_test == 5)  # 5는 True, 그 외는 False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True, False, False, False, False])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_5[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n",
      "(60000,)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train_5.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 SGDClassifier (Stochastic Gradient Descent Classifier)\n",
    "확률적 경사 하강법 분류기 : 하나의 독립된 훈련 샘플을 처리하고 모델을 업데이트"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 확률적 경사 하강법 분류기를 사용하기 위해서는 input shape이 2 dim 이어야 한다.\n",
    "import numpy as np\n",
    "\n",
    "x_train_2d = np.reshape(x_train, (-1, 784))\n",
    "x_test_2d = np.reshape(x_test, (-1, 784))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
       "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
       "       learning_rate='optimal', loss='hinge', max_iter=5, n_iter=None,\n",
       "       n_jobs=1, penalty='l2', power_t=0.5, random_state=42, shuffle=True,\n",
       "       tol=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "sgd = SGDClassifier(max_iter=5, random_state=42)\n",
    "# 훈련\n",
    "sgd.fit(x_train_2d, y_train_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction:  [False]\n",
      "Actual:  False\n"
     ]
    }
   ],
   "source": [
    "# 결과 check\n",
    "pred = sgd.predict([x_train_2d[10]])\n",
    "print('Prediction: ', pred)\n",
    "print('Actual: ', y_train_5[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Performance 측정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "mpl.rcParams['figure.figsize'] = (12, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-1. 교차 검증을 사용한 정확도 측정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모델이 학습되는 동안 overfitting을 확인 및 성능을 평가하기 위하여, 즉 모델의 학습 정도를 확인하기 위하여 validation set이 필요하다.\n",
    "\n",
    "Validation data를 생성하기 위해서, training data의 부족 현상이 발생할 수 있다. 이런 경우는 학습이 충분히 제대로 이루어지지 않는다. Training data가 부족하다면, validation set을 어떻게 확보할 수 있을까?\n",
    "- Multi($K$)-fold cross validation : Dataset을 $K$개의 부분 집합으로 나눈다. 하나의 부분집합을 validation set으로 사용하고, 나머지 $K-1$개의 부분집합을 training set으로 활용한다. 이 과정을 $K$번 반복한다. $K$번의 validation check가 가능하고 그 중 가장 작은 validation error를 보이는 모델을 선택한다. 물론 각 과정에서 training set과 validation set은 mutually exclusive하다. 학습 과정중에서 $K$개의 다양한 모델을 만들면서 학습하기 때문에 시간의 trade off가 발생한다.\n",
    "- Leave-one-out cross-validation : Dataset의 크기가 $n$ 이라면 1개의 data sample을 validation data로 사용하고 $n-1$개의 데이터를 training set으로 사용한다. $n$번 validation을 수행한 후, error를 계산한다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Customizing이 가능한 코드 (sklearn의 cross val score와 동일한 일을 수행한다.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.base import clone\n",
    "\n",
    "# 데이터의 각 class 별 비율을, fold에서 맞추어준다.\n",
    "skfolds = StratifiedKFold(n_splits=3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 총 3개의 \n",
    "(cv_train_1, cv_test_1), (cv_train_2, cv_test_2), (cv_train_3, cv_test_3) \\\n",
    "    = list(skfolds.split(x_train_2d, y_train_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 첫 번째 cross validation set을 위한 index 들"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "첫 번째 train set : 40000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([19964, 19965, 19966, ..., 59997, 59998, 59999])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train set\n",
    "print('첫 번째 train set :', len(cv_train_1))\n",
    "cv_train_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "첫 번째 test set : 20000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([    0,     1,     2, ..., 20331, 20342, 20359])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test set\n",
    "print('첫 번째 test set :', len(cv_test_1))\n",
    "cv_test_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([2036., 2036., 2036., 2036., 2036., 2036., 2036., 2036., 2036.,\n",
       "        1676.]),\n",
       " array([    0. ,  2035.9,  4071.8,  6107.7,  8143.6, 10179.5, 12215.4,\n",
       "        14251.3, 16287.2, 18323.1, 20359. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAEyCAYAAAAFlj3tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3XuwZnV95/v353QDSXkDpIsiDZxGB02BNWmxB5kysRiJ0GCOmDkZD9SUEsNMxyNU6UlmIsSpgZjDKUyiZKhjSGHoADMKEi+RckiwoyTGmuHSYKe5hbC5WHRXS/cIgo4ZMuB3/nh+G57e7Mtv79772U/3fr+qntprfddvrfVda7MW3/49v7V2qgpJkiRJc/vfljsBSZIkaX9h8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqtHq5E5jNEUccUevWrVvuNCRpQe6+++7/VlVrljuPUfK+LWl/1XvPHuvied26dWzdunW505CkBUnyneXOYdS8b0vaX/Xesx22IUmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSerUXTwnWZXk20m+2uaPS3JHkokkn09ycIsf0uYn2vJ1Q9u4uMUfSnLGYh+MJEmStJTm0/P8YeDBoflPAFdU1T8CngbOb/Hzgadb/IrWjiQnAOcAJwIbgT9Ismrf0pckSZJGp6t4TnI08C7gj9p8gHcAX2hNrgPe06bPbvO05ae19mcDN1bVc1X1GDABnLwYByFJkiSNwurOdr8P/Abwqjb/WuD7VfV8m98BrG3Ta4EnAKrq+STPtPZrgduHtjm8zouSbAI2ARx77LHdB7Kc1l30n5c7BUlL6PHL37XcKRzwvI9KB7YD6T46Z89zkl8AdlfV3SPIh6q6uqo2VNWGNWvWjGKXknTASHJMktuSPJDk/iQfbvHDk2xJ8nD7eViLJ8mV7XmU7UlOGtrWea39w0nOW65jkqRx0tPz/Dbg3UnOAn4CeDXwH4BDk6xuvc9HAztb+53AMcCOJKuB1wDfG4pPGl5nUdmDIWkFex749aq6J8mrgLuTbAF+Gfh6VV2e5CLgIuCjwJnA8e3zVuAq4K1JDgcuATYA1bZzc1U9PfIjkqQxMmfPc1VdXFVHV9U6Bg/8faOq/iVwG/BLrdl5wFfa9M1tnrb8G1VVLX5OexvHcQxu1Hcu2pFIkqiqXVV1T5v+AYMHvdey9/MoU59Tub4GbmfQMXIUcAawpaqeagXzFgYPe0vSitY75nk6HwVuTPL/At8Grmnxa4D/mGQCeIpBwU1V3Z/kJuABBj0jF1TVC/uwf0nSLNqrQt8M3AEcWVW72qLvAke26RefU2kmn0eZKS5JK9q8iueq+kvgL9v0o0zztoyq+h/Av5hh/cuAy+abpCRpfpK8Evgi8JGqenbw0qOBqqoktYj72u8e9JakhfIvDErSASbJQQwK589W1Zda+Mk2HIP2c3eLz/Q8SvdzKj7oLWklsXiWpANIe6/+NcCDVfWpoUXDz6NMfU7l/e2tG6cAz7ThHbcCpyc5rL2Z4/QWk6QVbV/GPEuSxs/bgPcB9ybZ1mK/CVwO3JTkfOA7wHvbsluAsxj84aofAR8AqKqnkvw2cFdr9/Gqemo0hyBJ48viWZIOIFX1LSAzLD5tmvYFXDDDtjYDmxcvO0na/zlsQ5IkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6zVk8J/mJJHcm+Zsk9yf5rRa/NsljSba1z/oWT5Irk0wk2Z7kpKFtnZfk4fY5b+kOS5IkSVp8qzvaPAe8o6p+mOQg4FtJ/qwt+7dV9YUp7c8Ejm+ftwJXAW9NcjhwCbABKODuJDdX1dOLcSCSJEnSUpuz57kGfthmD2qfmmWVs4Hr23q3A4cmOQo4A9hSVU+1gnkLsHHf0pckSZJGp2vMc5JVSbYBuxkUwHe0RZe1oRlXJDmkxdYCTwytvqPFZopLkiRJ+4Wu4rmqXqiq9cDRwMlJ3gRcDPw08E+Aw4GPLkZCSTYl2Zpk6549exZjk5K0YiTZnGR3kvuGYp8fej7l8dYZQpJ1Sf5+aNkfDq3zliT3tudXrkyS5TgeSRo383rbRlV9H7gN2FhVu9rQjOeAPwZObs12AscMrXZ0i80Un7qPq6tqQ1VtWLNmzXzSkyTBtUwZEldV/1dVrW+dIF8EvjS0+JHJZVX1waH4VcC/5qVnWBxmJ0n0vW1jTZJD2/RPAu8E/raNY6b1RrwHmOzluBl4f3vrxinAM1W1C7gVOD3JYUkOA05vMUnSIqmqbwJPTbes3a/fC9ww2zba/f3VVXV7VRVwPYP7vCSteD1v2zgKuC7JKgbF9k1V9dUk30iyBgiwDZjssbgFOAuYAH4EfACgqp5K8tvAXa3dx6tq2hu8JGlJ/BzwZFU9PBQ7Lsm3gWeBf1dVf83geZQdQ21mfUYlySZgE8Cxxx676ElL0jiZs3iuqu3Am6eJv2OG9gVcMMOyzcDmeeYoSVoc57J3r/Mu4Niq+l6StwB/muTE+W60qq4GrgbYsGHDbG9jkqT9Xk/PsyRpP5dkNfDPgbdMxtozK8+16buTPAK8gcHzKEcPrT7tMyqStBL557klaWX4eeBvq+rF4RjtmZZVbfp1DB4MfLQ9p/JsklPaOOn3A19ZjqQladxYPEvSASTJDcB/Bd6YZEeS89uic3j5g4JvB7a3V9d9Afjg0LMoHwL+iMHzK48Af4YkyWEbknQgqapzZ4j/8jSxLzJ4dd107bcCb1rU5CTpAGDPsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjr5wKAkSZKW1LqL/vPI9vX45e9a0u3b8yxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUac7iOclPJLkzyd8kuT/Jb7X4cUnuSDKR5PNJDm7xQ9r8RFu+bmhbF7f4Q0nOWKqDkiRJkpZCT8/zc8A7qupngPXAxiSnAJ8ArqiqfwQ8DZzf2p8PPN3iV7R2JDkBOAc4EdgI/EGSVYt5MJIkSdJSmrN4roEfttmD2qeAdwBfaPHrgPe06bPbPG35aUnS4jdW1XNV9RgwAZy8KEchSZIkjUDXmOckq5JsA3YDW4BHgO9X1fOtyQ5gbZteCzwB0JY/A7x2OD7NOsP72pRka5Kte/bsmf8RSdIKl2Rzkt1J7huKXZpkZ5Jt7XPW0LJph9Ql2dhiE0kuGvVxSNI46iqeq+qFqloPHM2gt/inlyqhqrq6qjZU1YY1a9Ys1W4k6UB2LYPhcVNdUVXr2+cWmHlIXRtW92ngTOAE4NzWVpJWtHm9baOqvg/cBvxT4NAkq9uio4GdbXoncAxAW/4a4HvD8WnWkSQtkqr6JvBUZ/OZhtSdDExU1aNV9Q/Aja2tJK1oPW/bWJPk0Db9k8A7gQcZFNG/1JqdB3ylTd/c5mnLv1FV1eLntLdxHAccD9y5WAciSZrThUm2t2Edh7XYTEPquobagcPtJK0sPT3PRwG3JdkO3AVsqaqvAh8Ffi3JBIMxzde09tcAr23xXwMuAqiq+4GbgAeAPwcuqKoXFvNgJEkzugp4PYO3Ju0CPrlYG3a4naSVZPVcDapqO/DmaeKPMs3bMqrqfwD/YoZtXQZcNv80JUn7oqqenJxO8hngq212tiF1DrWTpCn8C4OStAIkOWpo9heByTdxzDSk7i7g+PYHsQ5m8FDhzaPMWZLG0Zw9z5Kk/UuSG4BTgSOS7AAuAU5Nsp7Be/ofB34VBkPqkkwOqXueoSF1SS4EbgVWAZvb8DtJWtEsniXpAFNV504Tvmaa2GT7aYfUtdfZ3bKIqUnSfs9hG5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6jRn8ZzkmCS3JXkgyf1JPtzilybZmWRb+5w1tM7FSSaSPJTkjKH4xhabSHLR0hySJEmStDRWd7R5Hvj1qronyauAu5NsacuuqKrfG26c5ATgHOBE4KeAv0jyhrb408A7gR3AXUlurqoHFuNAJEmSpKU2Z/FcVbuAXW36B0keBNbOssrZwI1V9RzwWJIJ4OS2bKKqHgVIcmNra/EsSZKk/cK8xjwnWQe8GbijhS5Msj3J5iSHtdha4Imh1Xa02EzxqfvYlGRrkq179uyZT3qStOK1+/HuJPcNxX43yd+2+/WXkxza4uuS/P3Q8Ls/HFrnLUnubcPsrkyS5TgeSRo33cVzklcCXwQ+UlXPAlcBrwfWM+iZ/uRiJFRVV1fVhqrasGbNmsXYpCStJNcCG6fEtgBvqqp/DPwdcPHQskeqan37fHAofhXwr4Hj22fqNiVpReoqnpMcxKBw/mxVfQmgqp6sqheq6sfAZ3hpaMZO4Jih1Y9usZnikqRFUlXfBJ6aEvtaVT3fZm9ncP+dUZKjgFdX1e1VVcD1wHuWIl9J2t/0vG0jwDXAg1X1qaH4UUPNfhGY/IrwZuCcJIckOY5Bj8WdwF3A8UmOS3Iwg4cKb16cw5AkdfoV4M+G5o9L8u0kf5Xk51psLYOhdZOmHWY3yeF2klaSnrdtvA14H3Bvkm0t9pvAuUnWAwU8DvwqQFXdn+QmBg8CPg9cUFUvACS5ELgVWAVsrqr7F/FYJEmzSPIxBvflz7bQLuDYqvpekrcAf5rkxPlut6quBq4G2LBhQy1WvpI0jnretvEtYLoHRW6ZZZ3LgMumid8y23qSpKWR5JeBXwBOa0MxaG9Feq5N353kEeANDIbUDQ/tcJidJDX+hUFJOsAl2Qj8BvDuqvrRUHxNklVt+nUMhtk92l5R+mySU9rQvfcDX1mG1CVp7PQM25Ak7SeS3ACcChyRZAdwCYO3axwCbGlvnLu9vVnj7cDHk/xP4MfAB6tq8mHDDzF4c8dPMhgjPTxOWpJWLItnSTqAVNW504SvmaHtFxm8SWm6ZVuBNy1iapJ0QHDYhiRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOs1ZPCc5JsltSR5Icn+SD7f44Um2JHm4/TysxZPkyiQTSbYnOWloW+e19g8nOW/pDkuSJElafD09z88Dv15VJwCnABckOQG4CPh6VR0PfL3NA5wJHN8+m4CrYFBsA5cAbwVOBi6ZLLglSZKk/cGcxXNV7aqqe9r0D4AHgbXA2cB1rdl1wHva9NnA9TVwO3BokqOAM4AtVfVUVT0NbAE2LurRSJJIsjnJ7iT3DcX8tlCSFsG8xjwnWQe8GbgDOLKqdrVF3wWObNNrgSeGVtvRYjPFJUmL61pe3jnht4WStAi6i+ckrwS+CHykqp4dXlZVBdRiJJRkU5KtSbbu2bNnMTYpSStKVX0TeGpK2G8LJWkRdBXPSQ5iUDh/tqq+1MJPthss7efuFt8JHDO0+tEtNlN8L1V1dVVtqKoNa9asmc+xSJJm5reFkrQIet62EeAa4MGq+tTQopuByTFw5wFfGYq/v42jOwV4pt2wbwVOT3JY++rv9BaTJI3QYn5bCH5jKGll6el5fhvwPuAdSba1z1nA5cA7kzwM/HybB7gFeBSYAD4DfAigqp4Cfhu4q30+3mKSpKW3JN8Wgt8YSlpZVs/VoKq+BWSGxadN076AC2bY1mZg83wSlCQtislvCy/n5d8WXpjkRgYPBz5TVbuS3Ar8f0MPCZ4OXDzinCVp7MxZPEuS9i9JbgBOBY5IsoPBWzMuB25Kcj7wHeC9rfktwFkMvi38EfABGHxbmGTy20Lw20JJAiyeJemAU1XnzrDIbwslaR/N6z3PkiRJ0kpm8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVKnOYvnJJuT7E5y31Ds0iQ7k2xrn7OGll2cZCLJQ0nOGIpvbLGJJBct/qFIkiRJS6un5/laYOM08Suqan373AKQ5ATgHODEts4fJFmVZBXwaeBM4ATg3NZWkjQCSd441OGxLcmzST6ykM4QSVrJVs/VoKq+mWRd5/bOBm6squeAx5JMACe3ZRNV9ShAkhtb2wfmnbEkad6q6iFgPUDr0NgJfBn4AIPOkN8bbj+lM+SngL9I8oaqemGkiUvSmNmXMc8XJtnehnUc1mJrgSeG2uxosZniL5NkU5KtSbbu2bNnH9KTJM3gNOCRqvrOLG1e7AypqseA4c4QSVqxFlo8XwW8nkEvxi7gk4uVUFVdXVUbqmrDmjVrFmuzkqSXnAPcMDQ/n86Ql7HTQ9JKsqDiuaqerKoXqurHwGd4qTdiJ3DMUNOjW2ymuCRphJIcDLwb+JMW2ufOEDs9JK0kCyqekxw1NPuLwOSbOG4GzklySJLjgOOBO4G7gOOTHNdu3Oe0tpKk0ToTuKeqnoQFdYZI0oo25wODSW4ATgWOSLIDuAQ4Ncl6oIDHgV8FqKr7k9zE4EHA54ELJh8uSXIhcCuwCthcVfcv+tFIkuZyLkNDNpIcVVW72uzUzpDPJfkUgwcGJztDJGlF63nbxrnThK+Zpf1lwGXTxG8BbplXdpKkRZPkFcA7aR0eze/MtzNEklayOYtnSdKBoar+O/DaKbH3zdJ+2s4QSVrJ/PPckiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjrNWTwn2Zxkd5L7hmKHJ9mS5OH287AWT5Irk0wk2Z7kpKF1zmvtH05y3tIcjiRJkrR0enqerwU2ToldBHy9qo4Hvt7mAc4Ejm+fTcBVMCi2gUuAtwInA5dMFtySpNFJ8niSe5NsS7K1xebdISJJK9WcxXNVfRN4akr4bOC6Nn0d8J6h+PU1cDtwaJKjgDOALVX1VFU9DWzh5QW5JGk0/llVra+qDW1+Xh0ikrSSLXTM85FVtatNfxc4sk2vBZ4YarejxWaKv0ySTUm2Jtm6Z8+eBaYnSZqH+XaISNKKtc8PDFZVAbUIuUxu7+qq2lBVG9asWbNYm5UkDRTwtSR3J9nUYvPtENmLnR6SVpKFFs9PTvY+tJ+7W3wncMxQu6NbbKa4JGm0fraqTmIwJOOCJG8fXriQDhE7PSStJAstnm8GJt+YcR7wlaH4+9tDJqcAz7TejFuB05Mc1h5EOb3FJEkjVFU728/dwJcZPMQ93w4RSVqxel5VdwPwX4E3JtmR5HzgcuCdSR4Gfr7NA9wCPApMAJ8BPgRQVU8Bvw3c1T4fbzFJ0ogkeUWSV01OM+jIuI/5d4hI0oq1eq4GVXXuDItOm6ZtARfMsJ3NwOZ5ZSdJWkxHAl9OAoP7/+eq6s+T3AXc1DpHvgO8t7W/BTiLQYfIj4APjD5lSRovcxbPkqQDQ1U9CvzMNPHvMc8OEUlaqfzz3JIkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI67VPxnOTxJPcm2ZZka4sdnmRLkofbz8NaPEmuTDKRZHuSkxbjACRJkqRRWYye539WVeurakObvwj4elUdD3y9zQOcCRzfPpuAqxZh35KkDkmOSXJbkgeS3J/kwy1+aZKdrRNkW5Kzhta5uHV4PJTkjOXLXpLGx+ol2ObZwKlt+jrgL4GPtvj1VVXA7UkOTXJUVe1aghwkSXt7Hvj1qronyauAu5NsacuuqKrfG26c5ATgHOBE4KeAv0jyhqp6YaRZS9KY2dee5wK+luTuJJta7Mihgvi7wJFtei3wxNC6O1psL0k2JdmaZOuePXv2MT1JEkBV7aqqe9r0D4AHmeYePORs4Maqeq6qHgMmgJOXPlNJGm/7Wjz/bFWdxGBIxgVJ3j68sPUy13w2WFVXV9WGqtqwZs2afUxPkjRVknXAm4E7WujC9izK5snnVOjs8Gjbs9ND0oqxT8VzVe1sP3cDX2bQK/FkkqMA2s/drflO4Jih1Y9uMUnSiCR5JfBF4CNV9SyD509eD6wHdgGfnO827fSQtJIsuHhO8oo2bo4krwBOB+4DbgbOa83OA77Spm8G3t/eunEK8IzjnSVpdJIcxKBw/mxVfQmgqp6sqheq6sfAZ3hpaIYdHpI0jX15YPBI4MtJJrfzuar68yR3ATclOR/4DvDe1v4W4CwG4+Z+BHxgH/YtSZqHDG7W1wAPVtWnhuLDD27/IoNOEBh0eHwuyacYPDB4PHDnCFOWpLG04OK5qh4Ffmaa+PeA06aJF3DBQvcnSdonbwPeB9ybZFuL/SZwbpL1DJ5PeRz4VYCquj/JTcADDN7UcYFv2pCkpXlVnSRpzFTVt4BMs+iWWda5DLhsyZKSpP2Qf55bkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUqeRF89JNiZ5KMlEkotGvX9JUj/v2ZK0t5EWz0lWAZ8GzgROAM5NcsIoc5Ak9fGeLUkvN+qe55OBiap6tKr+AbgROHvEOUiS+njPlqQpRl08rwWeGJrf0WKSpPHjPVuSpli93AlMlWQTsKnN/jDJQwvc1BHAf1ucrPbJuOQB45PLuOQB45PLuOQB45PLuORBPrHgXP73xc5lHC3SfXscft/jkAOYx1TmsTfz2NvL8sgnFrytrnv2qIvnncAxQ/NHt9iLqupq4Op93VGSrVW1YV+3c6DkAeOTy7jkAeOTy7jkAeOTy7jkAeOVy4jNec+Gxblvj8M5HocczMM8zGP88xj1sI27gOOTHJfkYOAc4OYR5yBJ6uM9W5KmGGnPc1U9n+RC4FZgFbC5qu4fZQ6SpD7esyXp5UY+5rmqbgFuGcGu9nnoxyIZlzxgfHIZlzxgfHIZlzxgfHIZlzxgvHIZqRV2zx6HHMA8pjKPvZnH3kaeR6pq1PuUJEmS9kv+eW5JkiSpk8WzJEmS1KuqxvbD4BVJtwEPAPcDH27xw4EtwMPt52EtHuBKYALYDpw0tK3zWvuHgfOG4m8B7m3rXEkbytKZx6UMXtu0rX3OGlrn4rbNh4AzhuIbW2wCuGgofhxwR4t/Hjh4hnPyE8CdwN+0XH5rtvWBQ9r8RFu+bqE5duZxLfDY0DlZv5S/myk5rQK+DXx1Oc7JLHksyzkBHm9ttwFbl+PamSWPSxnxtdPaHgp8Afhb4EHgny7HOfGz1++k+9pa4PaX/P49j1yW9JrszOGNQ8e8DXgW+MgozgewGdgN3DcUW4570nR5/C6D+8J24MvAoS2+Dvj7ofPyh3Ptb6Zj6sxj0X4P9NcV0+Xx+aEcHge2LeX5YEzqve7raF9uSkv9AY6aPCHAq4C/A04AfmfyPxDgIuATbfos4M/aST0FuGPo5D/afh7Wpid/AXe2tmnrnjmPPC4F/s007U9gUFQe0v7jfYRBQbWqTb8OOLi1OaGtcxNwTpv+Q+D/nuGcBHhlmz6oXRinzLQ+8KHJ/7gZvGbq8wvNsTOPa4Ffmqb9kvxupuzj14DP8VLROtJzMksey3JOGNzwjpgSG+m1M0selzLia6ctvw74V236YAbF9MjPiZ8Xfx/zurYWuI8lv3/PI5clvSYXeP6/y+APQyz5+QDeDpzE3kXactyTpsvjdGB1m/7EUB7rhttN2c60+5vpmDrzWLTfA/11xcvymLL8k8C/X8rzwZjUe72fsR62UVW7quqeNv0DBj1Fa4GzGfxPkPbzPW36bOD6GrgdODTJUcAZwJaqeqqqnmbwr5eNbdmrq+r2GpzZ64e21ZPHTM4Gbqyq56rqMQb/yjm5fSaq6tGq+gfgRuDsJAHewaBHbOoxTc2lquqHbfag9qlZ1h8+V18ATmv7m1eO88hjtnOy6L+bSUmOBt4F/FGbn+2cLsk5mS6POSzpOZllnyO7dhZgya6dJK9h8D+JawCq6h+q6vv7wTk5kHVfWwu11PfvRUhxUf77W+C+TwMeqarvzJHfopyPqvom8NQ02x/p9TddHlX1tap6vs3ezuAPAs1ojv3NdExz5jGLpawrZsyjbee9wA2zJbev52OW63Qs789jXTwPS7IOeDODHs4jq2pXW/Rd4Mg2vRZ4Ymi1HS02W3zHNPHePAAuTLI9yeZA71PUAAAEyklEQVQkhy0wj9cC3x+6cGfNI8mqJNsYfM2yhcG/Omda/8V9tuXPtP3NN8c586iqyXNyWTsnVyQ5ZIHnZL6/m98HfgP4cZuf7Zwu2TmZJo9Jy3FOCvhakrvbn0+G5bl2pssDRn/tHAfsAf44ybeT/FGSV7CM9xPN69raZ0t0/56PpbwmF+Ic9i6KRn0+YDyvv19h0DM56bh2z/irJD83lN9M+5vpmHqNvK6Yxc8BT1bVw0OxJT0f41LvzWa/KJ6TvBL4IvCRqnp2eFn7F8RsPZ5LmcdVwOuB9cAuBl9tLLmqeqGq1jP4l/HJwE+PYr9z5ZHkTQzGZP008E8YfG3y0aXOI8kvALur6u6l3tcC8xj5OWl+tqpOAs4ELkjy9uGFI7x2pstjOa6d1Qy+mryqqt4M/HcGXwO+aJT3E43WmNy/x+WapP3FyHcDf9JCy/L/s2HjcP0l+RjwPPDZFtoFHNvuGb8GfC7Jq3u3t4BjWvbfwxTnsvc/sJb0fIxLvTeXsS+ekxzE4ER+tqq+1MJPti74ya8Kdrf4TgaDzicd3WKzxY+eJt6VR1U92QrIHwOfYVDILiSP7zH4ymH1lPis2lfOtzF46Gmm9V/cZ1v+mra/+ebYk8fG9tVLVdVzwB+z8HPS/bsB3ga8O8njDL6yegfwHxj9OXlZHkn+0zKdE6pqZ/u5m8EDMCezDNfOdHks07WzA9gx9A3JFxgU0yM/J3rRvO83C7HE9+9uS3xNzteZwD1V9WTLaeTnoxmb6y/JLwO/APzLVqjRhkl8r03fzeCb3jfMsb+ZjmlOy11XDGvr/nMGDw9O5rdk52Nc6r0utcDB0qP4MBjUfT3w+1Piv8veA8h/p02/i70HkN9ZLw0gf4zB4PHD2vThNf0A8rPmkcdRQ9P/D4PxSAAnsvfA/kcZDOpf3aaP46WB/Se2df6EvQf2f2iGc7KGl54C/kngrxlc7NOuD1zA3g/H3bTQHDvzOGronP0+cPlS/m6myetUXnpQb6TnZJY8Rn5OgFcArxqa/i8MxkWO+tqZKY+RXztt+V8Db2zTl7bzMdJz4mev38e8r60F7GPJ79+deSz5NTnP83Ij8IFRnw+mPHC2XNffNHlsZPCmhzVT2q0BVrXp1zEouGbd30zH1JnHct0b98pj6Jz81SjOB2NS73VfPwtdcRQf4GcZdNFvZ+i1LQzG8nydwWtI/mLoxAT4NIN/Cd0LbBja1q8wGGA/wd43jA3AfW2d/5/pX1U3Ux7/se1nO3DzlP/oP9a2+RBDT3S29f6uLfvYUPx17Rc70f6DP2SGc/KPGbwGbXvL+9/Ptj6DV8r9SYvfCbxuoTl25vGNdk7uA/4TL72RY0l+N9PkdSovFa0jPSez5DHyc9KO/W946VWCH2vxUV87M+Ux8muntV0PbG37/VMGN9eRnhM/L/udzOvaWsD2l/z+3ZnHkl+T88jlFQx6Jl8zFFvy88Hg6/9dwP9k8E3Q+ctx/c2QxwSDsbJ7vYIN+D/b72sbcA/wf8y1v5mOqTOP5agrXpZHi18LfHBK2yU5H4xJvdf78c9zS5IkSZ3GfsyzJEmSNC4sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdfpfKIyoYPPUd44AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(1,2,1)\n",
    "plt.hist(cv_train_1)\n",
    "plt.subplot(1,2,2)\n",
    "plt.hist(cv_test_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 두 번째 cross validation set을 위한 index 들"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "두 번째 train set : 40000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([    0,     1,     2, ..., 59997, 59998, 59999])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train set\n",
    "print('두 번째 train set :', len(cv_train_2))\n",
    "cv_train_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "두 번째 train set : 20000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([19964, 19965, 19966, ..., 40088, 40125, 40127])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test set\n",
    "print('두 번째 train set :', len(cv_test_2))\n",
    "cv_test_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1981., 2016., 2016., 2017., 2016., 2016., 2017., 2016., 2016.,\n",
       "        1889.]),\n",
       " array([19964. , 21980.3, 23996.6, 26012.9, 28029.2, 30045.5, 32061.8,\n",
       "        34078.1, 36094.4, 38110.7, 40127. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAEyCAYAAAAFlj3tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3X20XXV95/H3Zwii4xNBbpmUQIM22oWuGjADOFYXI1OedAQ7rQOrSyKlTR1xltbOqlDXKlaHWdjW0jKtWCwp0EGQipQsRTHFx64ZHgLG8CTNlYchmUBSg9DWlhb8zh/nd3Hncm+yL7kP5+a+X2uddX77u39n7+/eOWffb/b57X1SVUiSJEnavX811wlIkiRJ84XFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLU06K5TmBXDjzwwFq2bNlcpyFJz8ntt9/+t1U1Mtd5zCaP25Lmq77H7KEunpctW8b69evnOg1Jek6SPDTXOcw2j9uS5qu+x2yHbUiSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPfUqnpPsn+SzSb6T5N4kr09yQJJ1STa158Wtb5JclGQ0ycYkR3aWs6r135Rk1UxtlCRJkjQT+p55/kPgS1X1U8BrgXuBc4Cbqmo5cFObBjgJWN4eq4GLAZIcAJwHHA0cBZw3VnBLkiRJ88Fui+ckLwXeBFwKUFX/XFXfB04BLm/dLgdObe1TgCtq4GZg/yRLgBOAdVW1o6oeA9YBJ07r1kjSApfkkCRfTXJPkruTvK/F/bZQkqbBoh59DgO2A3+W5LXA7cD7gIOqamvr8whwUGsfDDzcef3mFpssvpMkqxmcsebQQw/tvSFdy875wnN6nZ7twQveMtcp6DnyczB95tnn4Cng16vqjiQvBm5Psg54F4NvCy9Icg6Dbws/yM7fFh7N4NvCozvfFq4Eqi1nbTv5Ma/52ZBm1zw7hu5Wn2Ebi4AjgYur6gjgH/jREA0AqqoYHFz3WFVdUlUrq2rlyMjIdCxSkhaMqtpaVXe09t8xGGZ3MH5bKEnTok/xvBnYXFW3tOnPMiimH20HWNrztjZ/C3BI5/VLW2yyuCRpBiRZBhwB3MIMfVvY1rM6yfok67dv3z5t+UvSMNpt8VxVjwAPJ3lVCx0H3AOsBcbGwK0Crm/ttcAZbRzdMcDj7YB9I3B8ksVtrN3xLSZJmmZJXgRcC7y/qp7ozpvObwvb8vzGUNKC0WfMM8B/Ba5M8jzgfuBMBoX3NUnOAh4C3tH63gCcDIwCP2h9qaodST4K3Nb6faSqdkzLVkiSnpFkXwaF85VV9bkWfjTJkqraOoVvC48dF//aTOYtSfNBr+K5qjYwuGhkvOMm6FvA2ZMsZw2wZioJSpL6SxIGd0e6t6p+vzNr7NvCC3j2t4XvTXI1gwsGH28F9o3A/+jcUvR44NyZyNkL+CTNJ33PPEuS5oc3AO8E7kyyocV+k0HR7LeFkrSHLJ4laS9SVX8NZJLZflsoSXuo7y8MSpIkSQuexbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUkz+SIkmSpBmz7JwvzOr6HrzgLTO6fM88S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxL0l4kyZok25Lc1Yl9JsmG9ngwyYYWX5bkHzvzPtl5zeuS3JlkNMlFSTIX2yNJw2bRXCcgSZpWlwF/BFwxFqiq/zzWTvJx4PFO/+9W1YoJlnMx8CvALcANwInAF2cgX0maVzzzLEl7kar6BrBjonnt7PE7gKt2tYwkS4CXVNXNVVUMCvFTpztXSZqPLJ4laeF4I/BoVW3qxA5L8q0kX0/yxhY7GNjc6bO5xSaUZHWS9UnWb9++ffqzlqQhYvEsSQvH6ex81nkrcGhVHQF8APh0kpdMdaFVdUlVrayqlSMjI9OUqiQNJ8c8S9ICkGQR8HPA68ZiVfUk8GRr357ku8ArgS3A0s7Ll7aYJC14nnmWpIXhPwDfqapnhmMkGUmyT2u/HFgO3F9VW4EnkhzTxkmfAVw/F0lL0rCxeJakvUiSq4D/A7wqyeYkZ7VZp/HsCwXfBGxst677LPDuqhq72PA9wJ8Co8B38U4bkgQ4bEOS9ipVdfok8XdNELsWuHaS/uuB10xrcpK0F+h15rndVP/OdhP99S12QJJ1STa158UtnnZD/dEkG5Mc2VnOqtZ/U5JVM7NJkiRJ0syYyrCNf19VK6pqZZs+B7ipqpYDN7VpgJMYjJtbDqxmcKN9khwAnAccDRwFnDdWcEuSJEnzwZ6MeT4FuLy1L+dHN9A/BbiiBm4G9m833D8BWFdVO6rqMWAdg1+skiRJkuaFvsVzAV9OcnuS1S12ULsiG+AR4KDWPhh4uPPasZvrTxbfiTfblyRJ0rDqe8Hgz1TVliQ/BqxL8p3uzKqqJDUdCVXVJcAlACtXrpyWZUqSJEnTodeZ56ra0p63AdcxGLP8aBuOQXve1rpvAQ7pvHzs5vqTxSVJkqR5YbfFc5IXJnnxWBs4HrgLWAuM3TFjFT+6gf5a4Ix2141jgMfb8I4bgeOTLG4XCh7fYpIkSdK80GfYxkHAdYMfmWIR8Omq+lKS24Br2g34HwLe0frfAJzM4Mb6PwDOBKiqHUk+CtzW+n2kczN+SZIkaejttniuqvuB104Q/x5w3ATxAs6eZFlrgDVTT1OSJEmae/48tyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJO1FkqxJsi3JXZ3Yh5NsSbKhPU7uzDs3yWiS+5Kc0Imf2GKjSc6Z7e2QpGFl8SxJe5fLgBMniF9YVSva4waAJIcDpwGvbq/5RJJ9kuwD/DFwEnA4cHrrK0kL3qK5TkCSNH2q6htJlvXsfgpwdVU9CTyQZBQ4qs0brar7AZJc3freM83pStK845lnSVoY3ptkYxvWsbjFDgYe7vTZ3GKTxSeUZHWS9UnWb9++fbrzlqShYvEsSXu/i4FXACuArcDHp3PhVXVJVa2sqpUjIyPTuWhJGjoO25CkvVxVPTrWTvIp4PNtcgtwSKfr0hZjF3FJWtA88yxJe7kkSzqTbwfG7sSxFjgtyX5JDgOWA7cCtwHLkxyW5HkMLipcO5s5S9Kw8syzJO1FklwFHAscmGQzcB5wbJIVQAEPAr8KUFV3J7mGwYWATwFnV9XTbTnvBW4E9gHWVNXds7wpkjSULJ4laS9SVadPEL50F/3PB86fIH4DcMM0piZJewWHbUiSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPVk8S5IkST1ZPEuSJEk9WTxLkiRJPfUunpPsk+RbST7fpg9LckuS0SSfSfK8Ft+vTY+2+cs6yzi3xe9LcsJ0b4wkSZI0k6Zy5vl9wL2d6Y8BF1bVTwKPAWe1+FnAYy1+YetHksOB04BXAycCn0iyz56lL0mSJM2eXsVzkqXAW4A/bdMB3gx8tnW5HDi1tU9p07T5x7X+pwBXV9WTVfUAMAocNR0bIUmSJM2Gvmee/wD4DeCHbfplwPer6qk2vRk4uLUPBh4GaPMfb/2fiU/wmmckWZ1kfZL127dvn8KmSJIkSTNrt8VzkrcC26rq9lnIh6q6pKpWVtXKkZGR2VilJEmS1MuiHn3eALwtycnA84GXAH8I7J9kUTu7vBTY0vpvAQ4BNidZBLwU+F4nPqb7GkmSJGno7fbMc1WdW1VLq2oZgwv+vlJVvwh8Ffj51m0VcH1rr23TtPlfqapq8dPa3TgOA5YDt07blkiSJEkzrM+Z58l8ELg6yX8HvgVc2uKXAn+eZBTYwaDgpqruTnINcA/wFHB2VT29B+uXJEmSZtWUiueq+hrwtda+nwnullFV/wT8wiSvPx84f6pJSpIkScPAXxiUJEmSerJ4liRJknqyeJYkSZJ6sniWpL1IkjVJtiW5qxP73STfSbIxyXVJ9m/xZUn+McmG9vhk5zWvS3JnktEkF7VfipWkBc/iWZL2LpcBJ46LrQNeU1U/DfwNcG5n3nerakV7vLsTvxj4FQa3FV0+wTIlaUGyeJakvUhVfYPBbUK7sS+3H7QCuJnBj1RNKskS4CVVdXO7T/8VwKkzka8kzTcWz5K0sPwS8MXO9GFJvpXk60ne2GIHA5s7fTa32ISSrE6yPsn67du3T3/GkjRELJ4laYFI8iEGP1J1ZQttBQ6tqiOADwCfTvKSqS63qi6pqpVVtXJkZGT6EpakIbQnvzAoSZonkrwLeCtwXBuKQVU9CTzZ2rcn+S7wSmALOw/tWNpikrTgeeZZkvZySU4EfgN4W1X9oBMfSbJPa7+cwYWB91fVVuCJJMe0u2ycAVw/B6lL0tDxzLMk7UWSXAUcCxyYZDNwHoO7a+wHrGt3nLu53VnjTcBHkvwL8EPg3VU1drHhexjcueMFDMZId8dJS9KCZfEsSXuRqjp9gvClk/S9Frh2knnrgddMY2qStFdw2IYkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLU026L5yTPT3Jrkm8nuTvJb7f4YUluSTKa5DNJntfi+7Xp0TZ/WWdZ57b4fUlOmKmNkiRJkmZCnzPPTwJvrqrXAiuAE5McA3wMuLCqfhJ4DDir9T8LeKzFL2z9SHI4cBrwauBE4BNJ9pnOjZEkQZI1SbYluasTOyDJuiSb2vPiFk+Si9qJjY1Jjuy8ZlXrvynJqrnYFkkaNrstnmvg79vkvu1RwJuBz7b45cCprX1Km6bNPy5JWvzqqnqyqh4ARoGjpmUrJEldlzE4SdF1DnBTVS0HbmrTACcBy9tjNXAxDIpt4DzgaAbH6vPGCm5JWsh6jXlOsk+SDcA2YB3wXeD7VfVU67IZOLi1DwYeBmjzHwde1o1P8JruulYnWZ9k/fbt26e+RZK0wFXVN4Ad48LdExvjT3hc0U6U3Azsn2QJcAKwrqp2VNVjDI794wtySVpwehXPVfV0Va0AljI4A/FTM5VQVV1SVSurauXIyMhMrUaSFpqDqmpraz8CHNTak53Y6HXCAzzpIWlhmdLdNqrq+8BXgdczODuxqM1aCmxp7S3AIQBt/kuB73XjE7xGkjRLqqoYDL+bruV50kPSgtHnbhsjSfZv7RcAPwvcy6CI/vnWbRVwfWuvbdO0+V9pB+q1wGntbhyHMRhfd+t0bYgkaZcebcMxaM/bWnyyExue8JCkCfQ587wE+GqSjcBtDMbAfR74IPCBJKMMxjRf2vpfCrysxT9Auyilqu4GrgHuAb4EnF1VT0/nxkiSJtU9sTH+hMcZ7a4bxwCPt+EdNwLHJ1ncLhQ8vsUkaUFbtLsOVbUROGKC+P1McLeMqvon4BcmWdb5wPlTT1OS1FeSq4BjgQOTbGZw14wLgGuSnAU8BLyjdb8BOJnBHZB+AJwJUFU7knyUwUkTgI9U1fiLECVpwdlt8SxJml+q6vRJZh03Qd8Czp5kOWuANdOYmiTNe/48tyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJEmS1JPFsyRJktSTxbMkSZLUk8WzJC0ASV6VZEPn8USS9yf5cJItnfjJndecm2Q0yX1JTpjL/CVpWCya6wQkSTOvqu4DVgAk2QfYAlwHnAlcWFW/1+2f5HDgNODVwI8Df5XklVX19KwmLklDxjPPkrTwHAd8t6oe2kWfU4Crq+rJqnoAGAWOmpXsJGmIWTxL0sJzGnBVZ/q9STYmWZNkcYsdDDzc6bO5xSRpQbN4lqQFJMnzgLcBf9FCFwOvYDCkYyvw8eewzNVJ1idZv3379mnLVZKGkcWzJC0sJwF3VNWjAFX1aFU9XVU/BD7Fj4ZmbAEO6bxuaYs9S1VdUlUrq2rlyMjIDKYuSXPP4lmSFpbT6QzZSLKkM+/twF2tvRY4Lcl+SQ4DlgO3zlqWkjSkvNuGJC0QSV4I/Czwq53w7yRZARTw4Ni8qro7yTXAPcBTwNneaUOSLJ4lacGoqn8AXjYu9s5d9D8fOH+m85Kk+cRhG5IkSVJPuy2ekxyS5KtJ7klyd5L3tfgBSdYl2dSeF7d4klzUfpVqY5IjO8ta1fpvSrJq5jZLkiRJmn59zjw/Bfx6VR0OHAOc3X556hzgpqpaDtzUpmFwJffy9ljN4DZIJDkAOA84msHV3Od17icqSZIkDb3dFs9VtbWq7mjtvwPuZXCj/FOAy1u3y4FTW/sU4IoauBnYv13NfQKwrqp2VNVjwDrgxGndGkmSJGkGTWnMc5JlwBHALcBBVbW1zXoEOKi1J/tVKn+tSpIkSfNa77ttJHkRcC3w/qp6Iskz86qqktR0JJRkNYPhHhx66KHTsUjtgWXnfGFW1/fgBW+Z1fXNptnel5Ikafr1OvOcZF8GhfOVVfW5Fn507Ob67Xlbi0/2q1S9fq3KX6qSJEnSsOpzt40AlwL3VtXvd2atBcbumLEKuL4TP6PddeMY4PE2vONG4Pgki9uFgse3mCRJkjQv9Bm28QbgncCdSTa02G8CFwDXJDkLeAh4R5t3A3AyMAr8ADgToKp2JPkocFvr95Gq2jEtWyFJkiTNgt0Wz1X110AmmX3cBP0LOHuSZa0B1kwlQUmSJGlY+AuDkiRJUk8Wz5IkSVJPFs+SJElSTxbPkiRJUk8Wz5IkSVJPFs+SJElSTxbPkiRJUk8Wz5IkSVJPFs+SJElSTxbPkiRJUk8Wz5IkSVJPFs+SJElSTxbPkiRJUk8Wz5K0QCR5MMmdSTYkWd9iByRZl2RTe17c4klyUZLRJBuTHDm32UvScLB4lqSF5d9X1YqqWtmmzwFuqqrlwE1tGuAkYHl7rAYunvVMJWkIWTxL0sJ2CnB5a18OnNqJX1EDNwP7J1kyFwlK0jCxeJakhaOALye5PcnqFjuoqra29iPAQa19MPBw57WbW0ySFrRFc52AJGnW/ExVbUnyY8C6JN/pzqyqSlJTXWgrxFcDHHroodOTqSQNKc88S9ICUVVb2vM24DrgKODRseEY7Xlb674FOKTz8qUtNtFyL6mqlVW1cmRkZKbSl6ShYPEsSQtAkhcmefFYGzgeuAtYC6xq3VYB17f2WuCMdteNY4DHO8M7JGnBctiGJC0MBwHXJYHBsf/TVfWlJLcB1yQ5C3gIeEfrfwNwMjAK/AA4c/ZTlqThY/EsSQtAVd0PvHaC+PeA4yaIF3D2LKQmSfOKwzYkSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSedls8J1mTZFuSuzqxA5KsS7KpPS9u8SS5KMloko1Jjuy8ZlXrvynJqpnZHEmSJGnm9DnzfBlw4rjYOcBNVbUcuKlNA5wELG+P1cDFMCi2gfOAo4GjgPPGCm5JkiRpvtht8VxV3wB2jAufAlze2pcDp3biV9TAzcD+SZYAJwDrqmpHVT0GrOPZBbkkSZI01J7rmOeDqmpraz8CHNTaBwMPd/ptbrHJ4pIkSdK8sccXDFZVATUNuQCQZHWS9UnWb9++fboWK0mSJO2x51o8P9qGY9Cet7X4FuCQTr+lLTZZ/Fmq6pKqWllVK0dGRp5jepIkSdL0e67F81pg7I4Zq4DrO/Ez2l03jgEeb8M7bgSOT7K4XSh4fItJkiRJ88ai3XVIchVwLHBgks0M7ppxAXBNkrOAh4B3tO43ACcDo8APgDMBqmpHko8Ct7V+H6mq8RchSpIkSUNtt8VzVZ0+yazjJuhbwNmTLGcNsGZK2UmSJElDxF8YlCRJknqyeJYkSZJ6sniWpAUgySFJvprkniR3J3lfi384yZYkG9rj5M5rzk0ymuS+JCfMXfaSNDx2O+ZZkrRXeAr49aq6I8mLgduTrGvzLqyq3+t2TnI4cBrwauDHgb9K8sqqenpWs5akIeOZZ0laAKpqa1Xd0dp/B9zLrn/p9RTg6qp6sqoeYHAXpaNmPlNJGm4Wz5K0wCRZBhwB3NJC702yMcmadi9+GBTWD3detpldF9uStCBYPEvSApLkRcC1wPur6gngYuAVwApgK/Dx57DM1UnWJ1m/ffv2ac1XkoaNxbMkLRBJ9mVQOF9ZVZ8DqKpHq+rpqvoh8Cl+NDRjC3BI5+VLW+xZquqSqlpZVStHRkZmbgMkaQhYPEvSApAkwKXAvVX1+534kk63twN3tfZa4LQk+yU5DFgO3Dpb+UrSsPJuG5K0MLwBeCdwZ5INLfabwOlJVgAFPAj8KkBV3Z3kGuAeBnfqONs7bUiSxbMkLQhV9ddAJph1wy5ecz5w/owlJUnzkMM2JEmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeLJ4lSZKkniyeJUmSpJ4sniVJkqSeZr14TnJikvuSjCY5Z7bXL0nqz2O2JO1sVovnJPsAfwycBBwOnJ7k8NnMQZLUj8dsSXq22T7zfBQwWlX3V9U/A1cDp8xyDpKkfjxmS9I4s108Hww83Jne3GKSpOHjMVuSxlk01wmMl2Q1sLpN/n2S+57DYg4E/nb6spo1Cz7vfGw6ljIlC36fz7J5mXc+9pzz/onpzmUY7WXH7WHJA4YnF/PY2bDkAcOTy1DlsQe1RK9j9mwXz1uAQzrTS1vsGVV1CXDJnqwkyfqqWrkny5gL5j375mvu5j275mve02C3x2zYu47bw5IHDE8u5jGcecDw5LLQ8pjtYRu3AcuTHJbkecBpwNpZzkGS1I/HbEkaZ1bPPFfVU0neC9wI7AOsqaq7ZzMHSVI/HrMl6dlmfcxzVd0A3DDDq9mjrw/nkHnPvvmau3nPrvma9x6bpWM2DM8+HpY8YHhyMY+dDUseMDy5LKg8UlWzsR5JkiRp3vPnuSVJkqSeLJ4lSZKknva64jnJiUnuSzKa5Jw5ymFNkm1J7urEDkiyLsmm9ry4xZPkopbvxiRHdl6zqvXflGRVJ/66JHe211yUJNOQ8yFJvprkniR3J3nffMi7Lff5SW5N8u2W+2+3+GFJbmnr+0y7WwBJ9mvTo23+ss6yzm3x+5Kc0InPyPsqyT5JvpXk8/Ml57bsB9u/5YYk61tsPrxX9k/y2STfSXJvktfPh7zniwzJcWQXefxu+7ffmOS6JPu3+LIk/9jezxuSfHJ365tsm6aQy4eTbOms8+TOa6b0mc4kx42eeXymk8ODSTbM5D7JkByvd5HHle21d2Xwt3zfFj82yeOd/fFb0/HvsptcLkvyQGedK1o8mZnPzWR5fLOTw/9L8pczvU9a3+H9+1hVe82DwdXg3wVeDjwP+DZw+Bzk8SbgSOCuTux3gHNa+xzgY619MvBFIMAxwC0tfgBwf3te3NqL27xbW9+01540DTkvAY5s7RcDfwMcPux5t+UGeFFr7wvc0tZzDXBai38S+C+t/R7gk619GvCZ1j68vWf2Aw5r76V9ZvJ9BXwA+DTw+TY99Dm39T4IHDguNh/eK5cDv9zazwP2nw95z5cHQ3Ic2UUexwOLWvxjnTyW0Tlej1vWhOubbJumkMuHgf82Qf8pf6aZ5LjRJ49xfT4O/NZM7hOG5Hi9izxObvMCXNXJ41jacXrccvbo32U3uVwG/PwE/WfqczNhHuP6XAucMdP7pM0f2r+Pc36wnc4H8Hrgxs70ucC5c5TLMnYunu8DlrT2EuC+1v4T4PTx/YDTgT/pxP+kxZYA3+nEd+o3jflfD/zsPMz7XwN3AEcz+LWjsT+Uz7w3GNx26/Wtvaj1y/j3y1i/mXpfMfjBiZuANwOfbzkMdc6d5T3Is4vnoX6vAC8FHqBdKD1f8p7PD4bkODKWx7jY24ErW3sZExSKu1rfZNs0hX3yYSYunqf0mWYXx42p7JO2nIeB5bO1TxiS43U3j3HxXwPOb+1jmbhQnO5/l+4+uYyJi+cZ/9xMtE+AlwCPAS+Z6X3CkP993NuGbRzM4MM/ZnOLDYODqmpraz8CHNTak+W8q/jmCeLTpn3lcQSD/3XOi7zb1zsbgG3AOgb/q/x+VT01wfqeybHNfxx42XPYpj31B8BvAD9s0y+bBzmPKeDLSW7P4KeZYfjfK4cB24E/a18F/mmSF86DvOelYTmOjMuj65cYnIEbc1h7X3w9yRs7+U22vsm2aSq5vLd97b4mPxriMNV9sqvjRt88AN4IPFpVmzqxGdknw3K8Hp9HVd3Smbcv8E7gS52XvL4NafhiklePz2/c+qb077KLXM5v75ELk+y3m3Xu8edmV/sEOBW4qaqe6MRmap8M9d/Hva14nhdq8F+dmus8JpLkRQy+lnn/uA/IUOddVU9X1QoG/1s9CvipOU5pl5K8FdhWVbfPdS7P0c9U1ZF2yhzgAAAEAElEQVTAScDZSd7UnTmk75VFDIZTXVxVRwD/wOCr5WcMad7zzrAcRybLI8mHgKeAK1toK3Boe198APh0kpf0XU+fbZogl4uBVwAr2vo/3nd9e2IX/zanMximMGbG9smwHK/H55HkNZ3ZnwC+UVXfbNN3AD9RVa8F/ifwl7OQy7kM9s2/ZTAU44PTuc4p5DFm/HtkRvbJfPj7uLcVz1uAQzrTS1tsGDyaZAlAe97W4pPlvKv40gnie6z9T/taBl9lfm6+5N1VVd8Hvsrga5n9k4z9EFB3fc/k2Oa/FPjebnKf7vfVG4C3JXkQuJrBV1N/OOQ5P6OqtrTnbcB1DP4ADvt7ZTOwuXMm5bMMiulhz3teGZbjyCR5kORdwFuBX2wFHlX1ZFV9r7VvZ3Am9JW7Wd9k29Qrl6p6tBUqPwQ+xeAz9Fz2yfeY/LjRd58sAn4O+MxYbKb3SVvuUByvO3mc2NZzHjDC4D8NY32eqKq/b+0bgH2THLiL9fX+d5ksl6raWgNPAn/Gc3+PTPnYNME+ObCt/wudPjO1T4b/7+NUxngM+4PBmaX7GXw9OzYI/NVzlMsydh7z/LvsfCHF77T2W9h54P+tLX4Ag/GZi9vjAeCANm/8wP+TpyHfAFcAfzAuPtR5t+WOAPu39guAbzL44/gX7HxxwXta+2x2vrjgmtZ+NTtfXHA/gwsLZvR9RWfc2HzIGXgh8OJO+38zOMDOh/fKN4FXtfaHW85Dn/d8eTAkx5Fd5HEicA8wMi4+AuzT2i9n8Id0l+ubbJumkMuSTvvXgKtbe8qfaSY5bvTJo7Nfvj4b+4QhOV7vIo9fZnBMe8G4/v8GnvlhuaOA/9u2f4/+XXaTy9j48TAYxnDBDH9uJsyjTb8buHy29klnHccyhH8f5/xgO90PBleh/g2D/yV/aI5yuIrBV17/wuBs11kMxt/cBGwC/qrzhg7wxy3fO4GVneX8EjDaHmd24iuBu9pr/mjszbuHOf8Mg6/YNgIb2uPkYc+7LfengW+13O/iR1eLv5zBAWO0fej2a/Hnt+nRNv/lnWV9qOV3H52rkWfyfTXu4DD0Obccv90ed48te568V1YA69t75S8Z/IEZ+rzny4MhOY7sIo9RBmMdx2Jjf3D/U3svb2DwVfR/3N36JtumKeTy522bNwJr2bmYntJnmkmOG33yaPMuA949rv+M7BOG5Hi9izyeaq8b20dj8fe2/fFt4Gbg303Hv8tucvlKe4/cBfwvfnQnjJn63EyYR5v3NQZnw7v9Z2yfdPofyxD+ffTnuSVJkqSe9rYxz5IkSdKMsXiWJEmSerJ4liRJknqyeJYkSZJ6sniWJEmSerJ4liRJknqyeJYkSZJ6+v/5kuWNE0SeiQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(1,2,1)\n",
    "plt.hist(cv_train_2)\n",
    "plt.subplot(1,2,2)\n",
    "plt.hist(cv_test_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 세 번째 cross validation set을 위한 index 들"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "세 번째 train set : 40000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([    0,     1,     2, ..., 40088, 40125, 40127])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train set\n",
    "print('세 번째 train set :', len(cv_train_3))\n",
    "cv_train_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "세 번째 test set : 20000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([39988, 39989, 39990, ..., 59997, 59998, 59999])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test set\n",
    "print('세 번째 test set :', len(cv_test_3))\n",
    "cv_test_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1990., 2001., 2001., 2001., 2001., 2001., 2001., 2001., 2001.,\n",
       "        2002.]),\n",
       " array([39988. , 41989.1, 43990.2, 45991.3, 47992.4, 49993.5, 51994.6,\n",
       "        53995.7, 55996.8, 57997.9, 59999. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs8AAAEyCAYAAAAFlj3tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3X2w5nV93//nK9zF8Sag7FC6YBctSQadZiFbJD+jQ7WBhdhA2tTCZJQYWrRZZrTJ75dAnCnElBlNakhtlQyGDZAqSEQjk2J0iybGablZdOU2hMONZbcru3URtKZY8P374/s5cu3hXOd8zu65udjzfMxcc77f9/fu/f1c1/lc73Ndn+/3pKqQJEmSNL8fWukEJEmSpBcKi2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSp4NXOoG5HHnkkbVu3bqVTkOS9smdd975v6pqzUrnsZzstyW9UPX22RNdPK9bt46tW7eudBqStE+SfH2lc1hu9tuSXqh6+2yHbUiSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6dRfPSQ5K8tUkf9bmj0tyW5KpJJ9IcmiLH9bmp9rydSP7uLjFH0hy+mKfjCStdkmOTfLFJPcluTfJu1v85Um2JHmw/TyixZPkQ61vvivJSSP7Oq+t/2CS81bqnCRpkizkk+d3A/ePzH8AuLyq/j7wBHB+i58PPNHil7f1SHICcA7wGmAj8JEkB+1f+pKkGZ4Bfq2qTgBOATa1/vci4JaqOh64pc0DnAEc3x4XAFfAUGwDlwCvA04GLpkuuCVpNesqnpMcA/ws8IdtPsCbgE+2Va4Bzm7TZ7V52vI3t/XPAq6vqqer6hFgiqFDliQtkqraWVVfadPfZvjQYy17980z++xra3ArcHiSo4HTgS1VtaeqngC2MHzwIUmr2sGd6/0+8OvAS9v8K4BvVdUzbX47Q+dM+/kYQFU9k+TJtv5a4NaRfY5us6jWXfRflmK3kibEo+//2ZVO4QWhDZs7EbgNOKqqdrZF3wCOatM/6LOb6b55XHy241zA8Kk1r3zlKxcn+SXke4R0YFvq94h5i+ckbwF2VdWdSU5d0mx44XXCkjSJkrwEuBF4T1U9NXwBOKiqSlKLdayquhK4EmDDhg37tF8LWkkvFD3DNl4P/FySR4HrGYZr/AeGr/ami+9jgB1tegdwLEBb/iPAN0fjs2zzA1V1ZVVtqKoNa9asWfAJSdJql+QQhsL5Y1X1qRZ+vA3HoP3c1eLj+uauPluSVpt5i+equriqjqmqdQwX/H2hqn4R+CLwC22184DPtOmb2jxt+Reqqlr8nHY3juMYLk65fdHORJI0fU3KVcD9VfV7I4tG++aZffbb2103TgGebMM7PgecluSIdqHgaS0mSata75jn2fwGcH2Sfwd8laGzpv384yRTwB6GgpuqujfJDcB9DFeDb6qqZ/fj+JKk53s98Dbg7iTbWuw3gfcDNyQ5H/g68Na27GbgTIaLuL8LvAOgqvYk+W3gjrbe+6pqz/KcgiRNrgUVz1X1F8BftOmHmeVuGVX1f4B/Pmb7y4DLFpqkJKlPVX0ZyJjFb55l/QI2jdnXZmDz4mUnSS98/odBSZIkqZPFsyRJktRpf8Y8S9KKWO7bmnlfaUnSND95liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSTqAJNmcZFeSe0Zin0iyrT0eTbKtxdcl+duRZX8wss1PJrk7yVSSDyXJSpyPJE2aeYvnJD+c5PYkX0tyb5LfavGrkzwy0umub/G0jnYqyV1JThrZ13lJHmyP85butCRp1boa2DgaqKp/UVXrq2o9cCPwqZHFD00vq6p3jcSvAP4VcHx77LVPSVqtDu5Y52ngTVX1nSSHAF9O8tm27P+rqk/OWP8MnutsX8fQAb8uycuBS4ANQAF3Jrmpqp5YjBORJEFVfSnJutmWtU+P3wq8aa59JDkaeFlV3drmrwXOBj4713aStBrM+8lzDb7TZg9pj5pjk7OAa9t2twKHt474dGBLVe1pBfMW/CRDkpbTG4DHq+rBkdhxSb6a5C+TvKHF1gLbR9bZ3mKStOp1jXlOclAbI7eLoQC+rS26rA3NuDzJYS22FnhsZPPpTndcfOaxLkiyNcnW3bt3L/B0JElzOBe4bmR+J/DKqjoR+FXg40lettCd2m9LWk26iueqeraNlTsGODnJa4GLgR8H/iHwcuA3FiOhqrqyqjZU1YY1a9Ysxi4ladVLcjDwT4FPTMeq6umq+mabvhN4CPhRYAdDfz/tmBablf22pNVkQXfbqKpvAV8ENlbVzjY042ngj4CT22o7gGNHNpvudMfFJUlL7x8Df11VPxiOkWRNkoPa9KsYrlV5uKp2Ak8lOaWNk3478JmVSFqSJk3P3TbWJDm8Tb8I+Bngr9s45ukLUM4Gpm+LdBPw9nbXjVOAJ1tH/DngtCRHJDkCOK3FJEmLJMl1wH8HfizJ9iTnt0XnsPeQDYA3Ane1YXmfBN5VVXvasl8B/hCYYvhE2osFJYm+u20cDVzTPp34IeCGqvqzJF9IsgYIsA2YvsXRzcCZDB3ud4F3AFTVniS/DdzR1nvfSCctSVoEVXXumPgvzRK7keHWdbOtvxV47aImJ0kHgHmL56q6CzhxlvistzqqqgI2jVm2Gdi8wBwlSZKkieB/GJQkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6jRv8Zzkh5PcnuRrSe5N8lstflyS25JMJflEkkNb/LA2P9WWrxvZ18Ut/kCS05fqpCRpNUuyOcmuJPeMxC5NsiPJtvY4c2TZrH1zko0tNpXkouU+D0maRD2fPD8NvKmqfgJYD2xMcgrwAeDyqvr7wBPA+W3984EnWvzyth5JTgDOAV4DbAQ+kuSgxTwZSRIAVzP0szNdXlXr2+NmGN83t/75w8AZwAnAuW1dSVrV5i2ea/CdNntIexTwJuCTLX4NcHabPqvN05a/OUla/PqqerqqHgGmgJMX5SwkST9QVV8C9nSuPq5vPhmYqqqHq+p7wPVtXUla1brGPLdPIbYBu4AtwEPAt6rqmbbKdmBtm14LPAbQlj8JvGI0Pss2o8e6IMnWJFt379698DOSJI1zYZK72rCOI1psXN/c1WeD/bak1aWreK6qZ6tqPXAMw6cRP75UCVXVlVW1oao2rFmzZqkOI0mrzRXAqxmG3+0EPrhYO7bflrSaLOhuG1X1LeCLwE8Bhyc5uC06BtjRpncAxwK05T8CfHM0Pss2kqQlVFWPtw9Cvg98lOeGzY3rm+2zJWkWPXfbWJPk8Db9IuBngPsZiuhfaKudB3ymTd/U5mnLv1BV1eLntLtxHAccD9y+WCciSRovydEjsz8PTN+JY1zffAdwfLuz0qEMFxXetJw5S9IkOnj+VTgauKZdef1DwA1V9WdJ7gOuT/LvgK8CV7X1rwL+OMkUwwUr5wBU1b1JbgDuA54BNlXVs4t7OpKkJNcBpwJHJtkOXAKcmmQ9wwXfjwLvhLn75iQXAp8DDgI2V9W9y3wqkjRx5i2eq+ou4MRZ4g8zy90yqur/AP98zL4uAy5beJqSpF5Vde4s4atmiU2vP2vf3G5nd/MipiZJL3j+h0FJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE7zFs9Jjk3yxST3Jbk3ybtb/NIkO5Jsa48zR7a5OMlUkgeSnD4S39hiU0kuWppTkqTVK8nmJLuS3DMS+90kf53kriSfTnJ4i69L8rcj/fgfjGzzk0nubv31h5JkJc5HkiZNzyfPzwC/VlUnAKcAm5Kc0JZdXlXr2+NmgLbsHOA1wEbgI0kOSnIQ8GHgDOAE4NyR/UiSFsfVDH3vqC3Aa6vqHwB/A1w8suyhkX78XSPxK4B/BRzfHjP3KUmr0rzFc1XtrKqvtOlvA/cDa+fY5Czg+qp6uqoeAaaAk9tjqqoerqrvAde3dSVJi6SqvgTsmRH7fFU902ZvBY6Zax9JjgZeVlW3VlUB1wJnL0W+kvRCs6Axz0nWAScCt7XQhe1rwM1JjmixtcBjI5ttb7FxcUnS8vll4LMj88cl+WqSv0zyhhZby9BHT5uzv05yQZKtSbbu3r178TOWpAnSXTwneQlwI/CeqnqK4Su9VwPrgZ3ABxcjITthSVoaSd7LMBTvYy20E3hlVZ0I/Crw8SQvW+h+q+rKqtpQVRvWrFmzeAlL0gTqKp6THMJQOH+sqj4FUFWPV9WzVfV94KMMwzIAdgDHjmx+TIuNi+/FTliSFl+SXwLeAvxiG4pBG173zTZ9J/AQ8KMMffPo0I5Z+2tJWo167rYR4Crg/qr6vZH40SOr/TwwfWX3TcA5SQ5LchzDhSa3A3cAxyc5LsmhDBcV3rQ4pyFJGifJRuDXgZ+rqu+OxNe0i7lJ8iqG/vrhqtoJPJXklPYe8HbgMyuQuiRNnIM71nk98Dbg7iTbWuw3Ge6WsR4o4FHgnQBVdW+SG4D7GL4e3FRVzwIkuRD4HHAQsLmq7l3Ec5GkVS/JdcCpwJFJtgOXMNxd4zBgS7vj3K3tzhpvBN6X5P8C3wfeVVXTFxv+CsOdO17EMEZ6dJy0JK1a8xbPVfVlYLb7e948xzaXAZfNEr95ru0kSfunqs6dJXzVmHVvZBiSN9uyrcBrFzE1STog+B8GJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6zVs8Jzk2yReT3Jfk3iTvbvGXJ9mS5MH284gWT5IPJZlKcleSk0b2dV5b/8Ek5y3daUnS6pVkc5JdSe4ZidlnS9Ii6Pnk+Rng16rqBOAUYFOSE4CLgFuq6njgljYPcAZwfHtcAFwBQ8cNXAK8DjgZuGS685YkLaqrgY0zYvbZkrQI5i2eq2pnVX2lTX8buB9YC5wFXNNWuwY4u02fBVxbg1uBw5McDZwObKmqPVX1BLCF53fukqT9VFVfAvbMCNtnS9IiWNCY5yTrgBOB24CjqmpnW/QN4Kg2vRZ4bGSz7S02Li5JWnr22ZK0CLqL5yQvAW4E3lNVT40uq6oCajESSnJBkq1Jtu7evXsxdilJGrGYfTbYb0taXbqK5ySHMBTOH6uqT7Xw4+2rPdrPXS2+Azh2ZPNjWmxcfC9VdWVVbaiqDWvWrFnIuUiSxluSPhvstyWtLj132whwFXB/Vf3eyKKbgOmrr88DPjMSf3u7gvsU4Mn2VeHngNOSHNEuOjmtxSRJS88+W5IWwcEd67weeBtwd5JtLfabwPuBG5KcD3wdeGtbdjNwJjAFfBd4B0BV7Uny28Adbb33VdXMC1okSfspyXXAqcCRSbYz3DXDPluSFsG8xXNVfRnImMVvnmX9AjaN2ddmYPNCEpQkLUxVnTtmkX22JO0n/8OgJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSp07zFc5LNSXYluWckdmmSHUm2tceZI8suTjKV5IEkp4/EN7bYVJKLFv9UJEnjJPmxkT57W5KnkrxnX/pzSVrNDu5Y52rgPwHXzohfXlX/fjSQ5ATgHOA1wN8F/muSH22LPwz8DLAduCPJTVV1337kLknqVFUPAOsBkhwE7AA+DbyDBfTnVfXssiYuSRNm3k+eq+pLwJ7O/Z0FXF9VT1fVI8AUcHJ7TFXVw1X1PeD6tq4kafm9GXioqr4+xzrj+nNJWtX2Z8zzhUnuasM6jmixtcBjI+tsb7Fx8edJckGSrUm27t69ez/SkySNcQ5w3cj8Qvrz57HflrSa7GvxfAXwaoavAHcCH1yshKrqyqraUFUb1qxZs1i7lSQBSQ4Ffg74kxba7/7cflvSarJPxXNVPV5Vz1bV94GP8txXeTuAY0dWPabFxsUlScvrDOArVfU47FN/Lkmr2j4Vz0mOHpn9eWD6Thw3AeckOSzJccDxwO3AHcDxSY5rn3qc09aVJC2vcxkZsrEP/bkkrWrz3m0jyXXAqcCRSbYDlwCnJlkPFPAo8E6Aqro3yQ3AfcAzwKbpK7OTXAh8DjgI2FxV9y762UiSxkryYoa7Hr1zJPw7C+3PJWk1m7d4rqpzZwlfNcf6lwGXzRK/Gbh5QdlJkhZNVf1v4BUzYm+bY/1Z+3NJWs38D4OSJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ3mLZ6TbE6yK8k9I7GXJ9mS5MH284gWT5IPJZlKcleSk0a2Oa+t/2CS85bmdCRJc0nyaJK7k2xLsrXFFtynS9Jq1fPJ89XAxhmxi4Bbqup44JY2D3AGcHx7XABcAUPHDFwCvA44GbhkunOWJC27f1RV66tqQ5tfUJ8uSavZvMVzVX0J2DMjfBZwTZu+Bjh7JH5tDW4FDk9yNHA6sKWq9lTVE8AWnl+QS5JWxkL7dElatfZ1zPNRVbWzTX8DOKpNrwUeG1lve4uNiz9PkguSbE2ydffu3fuYniRpjAI+n+TOJBe02EL79L3Yb0taTfb7gsGqKobOeFFU1ZVVtaGqNqxZs2axditJGvx0VZ3EMCRjU5I3ji7clz7dflvSarKvxfPj01/dtZ+7WnwHcOzIese02Li4JGkZVdWO9nMX8GmG61AW2qdL0qq1r8XzTcD0HTPOAz4zEn97u0L7FODJ9lXg54DTkhzRLhQ8rcUkScskyYuTvHR6mqEvvoeF9+mStGodPN8KSa4DTgWOTLKd4a4Z7wduSHI+8HXgrW31m4EzgSngu8A7AKpqT5LfBu5o672vqmZehChJWlpHAZ9OAkP///Gq+vMkd7CAPl2SVrN5i+eqOnfMojfPsm4Bm8bsZzOweUHZSZIWTVU9DPzELPFvssA+XZJWK//DoCRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnSyeJUmSpE4Wz5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSp/0qnpM8muTuJNuSbG2xlyfZkuTB9vOIFk+SDyWZSnJXkpMW4wQkSfNLcmySLya5L8m9Sd7d4pcm2dH68W1JzhzZ5uLWZz+Q5PSVy16SJsdifPL8j6pqfVVtaPMXAbdU1fHALW0e4Azg+Pa4ALhiEY4tSerzDPBrVXUCcAqwKckJbdnlrR9fX1U3A7Rl5wCvATYCH0ly0EokLkmTZCmGbZwFXNOmrwHOHolfW4NbgcOTHL0Ex5ckzVBVO6vqK23628D9wNo5NjkLuL6qnq6qR4Ap4OSlz1SSJtv+Fs8FfD7JnUkuaLGjqmpnm/4GcFSbXgs8NrLtdubuuCVJSyDJOuBE4LYWurANp9s8PdSOBfTZSS5IsjXJ1t27dy9R1pI0Gfa3eP7pqjqJYUjGpiRvHF1YVcVQYHezE5akpZPkJcCNwHuq6imGIXSvBtYDO4EPLnSfVXVlVW2oqg1r1qxZ1HwladLsV/FcVTvaz13Apxm+0nt8ejhG+7mrrb4DOHZk82NabOY+7YQlaQkkOYShcP5YVX0KoKoer6pnq+r7wEd5bmhGV58tSavNPhfPSV6c5KXT08BpwD3ATcB5bbXzgM+06ZuAt7e7bpwCPDkyvEOStISSBLgKuL+qfm8kPnrtyc8z9OMw9NnnJDksyXEMF3vfvlz5StKkOng/tj0K+PTQH3Mw8PGq+vMkdwA3JDkf+Drw1rb+zcCZDBedfBd4x34cW5K0MK8H3gbcnWRbi/0mcG6S9QxD7B4F3glQVfcmuQG4j+FOHZuq6tllz1qSJsw+F89V9TDwE7PEvwm8eZZ4AZv29XiSpH1XVV8GMsuim+fY5jLgsiVLSpJegPwPg5IkSVIni2dJkiSpk8WzJEmS1MniWZIkSepk8SxJkiR1sniWJEmSOlk8S5IkSZ0sniVJkqROFs+SJElSJ4tnSZIkqZPFsyRJktTJ4lmSJEnqZPEsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKmTxbMkSZLUyeJZkiRJ6mTxLEmSJHWyeJYkSZI6WTxLkiRJnZa9eE6yMckDSaaSXLTcx5ck9bPPlqS9LWvxnOQg4MPAGcAJwLlJTljOHCRJfeyzJen5lvuT55OBqap6uKq+B1wPnLXMOUiS+thnS9IMy108rwUeG5nf3mKSpMljny1JMxy80gnMlOQC4II2+50kD+zDbo4E/tfiZbVfJiWXSckDJieXSckDJieXSckDJieXI/OBfc7j7y1qJhPqAOu3JyUPmJxcJiUPMJfZTEoeMCG55AP7nEdXn73cxfMO4NiR+WNa7Aeq6krgyv05SJKtVbVhf/axWCYll0nJAyYnl0nJAyYnl0nJAyYnl0nJY4XM22fDgdVvT0oeMDm5TEoeYC6TnAdMTi5LncdyD9u4Azg+yXFJDgXOAW5a5hwkSX3ssyVphmX95LmqnklyIfA54CBgc1Xdu5w5SJL62GdL0vMt+5jnqroZuHmJD7NfXx8usknJZVLygMnJZVLygMnJZVLygMnJZVLyWBHL1GfD5LTzpOQBk5PLpOQB5jKbSckDJieXJc0jVbWU+5ckSZIOGP57bkmSJKmTxbMkSZLUq6oOqAewEXgAmAIuWqJjPArcDWwDtrbYy4EtwIPt5xEtHuBDLZ+7gJNG9nNeW/9B4LzOY28GdgH3jMQW7djAT7Zzm2rbZgF5XMpwG6tt7XHmyLKL2z4fAE6f7/kCjgNua/FPAIeOyeNY4IvAfcC9wLtXsE3G5bKs7QL8MHA78LWWx2/NtS1wWJufasvX7Wt+C8jlauCRkTZZv9TPT1v3IOCrwJ+tVJsc6I9JauNZcvlY2/4ehj7skBY/FXhy5PX4bxfjd3GeXCbld+CvRnL4n8CfLkebsMTvowtsk9ly+V3gr9vxPg0c3uLrgL8daZc/mO+Y486rM49LWeb31Tly+cRIHo8C25ahTQ4HPtmei/uBn1qp18leee1PJzlpD4ZO4SHgVcChDG/YJyzBcR4FjpwR+53pFylwEfCBNn0m8Nn2pJ4C3Dbywnm4/TyiTc/64plxnDcCJ7F30bpox2YodE5p23wWOGMBeVwK/L+zrHtCey4OY/jlfag9V2OfL+AG4Jw2/QfAvx6Tx9HTvyDAS4G/acdbiTYZl8uytkvL8yVt+hCGzvKUcdsCv0Lr7BhuRfaJfc1vAblcDfzCLOsv2fPT1v1V4OM8Vzgse5sc6I9JauNZcjmzvU4CXDeSy6nT68zYfr/7qDlymYjfgRnLbgTevhxtwhK/jy6wTWbL5TTg4Db9gZFc1jHyvjdjm1mPOe68OvO4lGV+Xx2Xy4zlH6T9QbXEbXIN8C/b9KEMxfSKvE5GHwfasI2TgamqeriqvgdcD5y1TMc+i+FJpv08eyR+bQ1uBQ5PcjRwOrClqvZU1RMMfz1tnO8gVfUlYM9SHLste1lV3VrDq+rakX315DHOWcD1VfV0VT3C8BfeyYx5vpIEeBPDX5szz2lmHjur6itt+tsMf5muXaE2GZfLsrZLO7fvtNlD2qPm2Ha0rT4JvLkda0H5jWmTcbnM1SZL8vwkOQb4WeAP2/xc7blkbXIgm6Q2npkLDHcOaa+tYnjTPGaeU9rvPmpcLnNYtt+BGcte1s7pT+fJb1HaZIxl77fHqarPV9UzbfZW5nmtzHPMcee1P5bsfXU+bV9vZfgDdK719qtNkvwIwwd1VwFU1feq6ltzbLtsr5MDrXheCzw2Mr+duYuXfVXA55Pc2f4tLcBRVbWzTX8DOGqenBYz18U69to2vT85XZjkriSbkxyxj3m8AvjWSMfVlUeSdcCJDJ9urmibzMgFlrldkhyUZBvD0JotDJ9EjNv2B8dry59sx1qU1+7MXKpquk0ua21yeZLD9rFNFvL8/D7w68D32/xc7bmkbXIAm6Q2npnLDyQ5BHgb8Ocj4Z9K8rUkn03ympk5zjjmQvuocbms9O/AqLOBW6rqqZHYUrYf3Bq9AAAErklEQVTJUr6PLrTfni2XUb/M8KnktOOSfDXJXyZ5w0iO44457rx681iJ99W52uQNwONV9eBIbCna5DhgN/BHbd9/mOTFc2y7bDXPgVY8L5efrqqTgDOATUneOLqw/QUz16drS2Yljw1cAbwaWA/sZPhaZ1kkeQnDV47vmdH5L3ubzJLLsrdLVT1bVesZPi05GfjxpT5mby5JXsswVu/HgX/I8FXabyxlDkneAuyqqjuX8jir2SS1cUcuHwG+VFV/1ea/Avy9qvoJ4D8y/6evi5HLpP0OnMvenyQuWZs0k/Q+OjaXJO8FnmEYLw9DH/7KqjqRNgSmfWrfZZ7zmi2PlXpfnev5mflaWao2OZhheOgVbd//m2GYRs+2S+pAK553MFy0Ne2YFltUVbWj/dzFcCHBycDj7SuA6a8qds2T02LmuljH3sHeX00tKKeqerwVSt8HPsrQLvuSxzcZvm45eEZ8Vu1TpBuBj1XVp1p4RdpktlxWql3asb/FcBHjT82x7Q+O15b/SDvWor52R3LZWMMQl6qqp4E/Yt/bpPf5eT3wc0keZfga803Af2CF2+QAM0lt/LxckvzndqxLgDUMb/IAVNVT1YYX1fBPYQ5JcuQcx1zI7+KsuUzC78BImxzZjv9flqlNlvp9dEH99phcSPJLwFuAX2xFGm2YxDfb9J0M3+r96DzHHHde8+axUu8fc7TJwcA/Zbh4cHrdpWqT7cD2kW8rP8lQTK98zVMdA6NfKA+Gv1IeZviof3qg/GsW+RgvBl46Mv3fGMYq/y57D2D/nTb9s+w9gP32em4A+yMMg9ePaNMv78xhHXtfqLdox+b5g+fPXEAeR49M/xuG8VgAr2HvCxseZrioYezzBfwJe1/Y8CtjcgjDOKXfnxFf9jaZI5dlbReGwmD6yvAXMVxN/5Zx2wKb2PvCrRv2Nb8F5HL0SJv9PvD+5XjNtvVP5bmLtpa9TVbDY5LaeEYu/5Khz37RjHX+Ds/dBeBk4H+019N+91Fz5DIRvwNt/l3ANcvVJizD+2hvm8yRy0aGOyetmbH+GuCgNv0qhmJrzmOOO6/OPFbifXXWXNr8RuAvl6NN2rK/An6sTV/atluRmmevvPa1c5zUB8PVln/D8JfPe5dg/69qL8bpW2+9t8VfAdzCcBuU/zryxAT4cMvnbmDDyL5+mWGQ/xTwjs7jX8fwFcn/Zfir7PzFPDawgeE2Tg8B/4nxt2WbLY8/bse5C7iJvX/p39v2+QAjV7OOe75aO9/e8vsT4LAxefw0w1c2dzFyK58VapNxuSxruwD/gOFWVHe1vP/tXNsy3E7uT1r8duBV+5rfAnL5QmuTe4D/zHN35Fiy52dk/VN5roBZ9jZZDY9JauMZuTzTtt3r9mvAhQz9+dcYLg77fxbjd3GeXCbid6DN/wWtOBqJLVmbsAzvo71tMkcuUwzjZPe6/Rrwz9p62xiGtvyT+Y457rw681iJ99VZc2nLrgbeNWP9JWmTtt56YGs7/z9lKH6X/XUy8+G/55YkSZI6HWhjniVJkqQlY/EsSZIkdbJ4liRJkjpZPEuSJEmdLJ4lSZKkThbPkiRJUieLZ0mSJKnT/w81SqBWMzBt+QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.subplot(1,2,1)\n",
    "plt.hist(cv_train_3)\n",
    "plt.subplot(1,2,2)\n",
    "plt.hist(cv_test_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9532\n",
      "0.95125\n",
      "0.9625\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.base import clone\n",
    "\n",
    "# 데이터의 각 class 별 비율을, fold에서 맞추어준다.\n",
    "skfolds = StratifiedKFold(n_splits=3, random_state=42)\n",
    "\n",
    "for train_index, test_index in skfolds.split(x_train_2d, y_train_5):\n",
    "    clone_sgd = clone(sgd)\n",
    "    x_train_folds = x_train_2d[train_index]\n",
    "    y_train_folds = y_train_5[train_index]\n",
    "    x_test_folds = x_train_2d[test_index]\n",
    "    y_test_folds = y_train_5[test_index]\n",
    "    \n",
    "    clone_sgd.fit(x_train_folds, y_train_folds)\n",
    "    y_pred = clone_sgd.predict(x_test_folds)\n",
    "    n_correct = sum(y_pred == y_test_folds)\n",
    "    print(n_correct / len(y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `cross_val_score`를 이용한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.9532 , 0.95125, 0.9625 ])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(sgd, x_train_2d, y_train_5, cv=3, scoring='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "왜 이렇게 성능이 높을까?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set: Counter({False: 54579, True: 5421})\n",
      "Test set: Counter({False: 9108, True: 892})\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "print('Training set:', Counter(y_train_5))\n",
    "print('Test set:', Counter(y_test_5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이렇게 **불균형한 데이터를 다룰때**는 분류기의 성능 지표로 accuracy가 과연 좋을까?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-2 오차 행렬 (Confusion matrix)\n",
    " - 클래스 A의 샘플이 클래스 B로 분류된 횟수를 세는 것"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "# SGDClassifier 내부적으로 decision_function을 사용하여, \n",
    "# 각 샘플에 대한 결정함수 결과값이 나오고 그것을 이용하여 True, False인지 구분합니다. (hinge loss이므로, predict_prob를 사용하지 않습니다.)\n",
    "y_train_pred = cross_val_predict(sgd, x_train_2d, y_train_5, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 784)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_2d.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cross_val_predict`: K-fold 교차 검증을 수행하지만, 평가점수를 반환하지 않고, 예측값을 반환한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[52992,  1587],\n",
       "       [ 1074,  4347]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- confusion matrix의 row : 실제 클래스\n",
    "- confusion matrix의 column : 예측한 클래스"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Confusion mtx | Prediction(0, $H_0$)| Prediction(1, $H_1$) |\n",
    "| :----------------------: |:-------------------:| :--------------------: |\n",
    "| Actual class(0) | True Negative | False Positive |\n",
    "| Actual class(1) | False Negative | True Positive |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "우리의 예제를 살펴보면,\n",
    "\n",
    "| Confusion mtx | Prediction(0, $H_0$)| Prediction(1, $H_1$) |\n",
    "| :----------------------: |:-------------------:| :--------------------: |\n",
    "| Actual class(0) | 52992 | 1587 |\n",
    "| Actual class(1) | 1074 | 4347 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-3 F1 score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Precision, 정밀도\n",
    "\n",
    "- $\\text{Precision}=\\frac{TP}{TP+FP}$\n",
    "\n",
    "- 분류기가 \"1\"이라고 예측한 샘플 중에서, 실제 값이 \"1\"인 샘플의 비율"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Recall, 재현율 (또는 Sensitivity, 민감도, True Positive Rate, 진짜 양성 비율)\n",
    " - $\\text{Recall}=\\frac{TP}{TP+FN}$\n",
    " - 실제 값이 \"1\"인 샘플 중에서, 분류기가 \"1\"이라고 예측한 비율"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7325581395348837"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "precision_score(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5로 판별된 샘플중에서 73%로만 정확"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8018815716657444"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "전체 5에서 80%만 감지"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### $F_1$ socre\n",
    "- Precision과 Recall의 조화 평균(harmonic mean)\n",
    "- $F_1=\\frac{2}{\\frac{1}{\\text{Precision}}+\\frac{1}{\\text{Recall}}}=2\\times \\frac{\\text{Precision} \\times \\text{Recall}}{\\text{Precision} + \\text{Recall}}=\\frac{TP}{TP+\\frac{FN+FP}{2}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.765653896961691"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "f1_score(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "과연 $F_1$ score가 높은 것이 항상 좋을까?\n",
    " - 어린아이에게 안전한 동영상을 걸러내는 분류기를 훈련\n",
    "  - 좋은 동영상의 Recall은 낮아도, 좋은 동영상의 Precision이 높도록\n",
    " - 카메라를 통해 좀도둑을 잡아내는 분류기\n",
    "  - 좀도둑의 Precision은 낮아도, 좀도둑의 recall은 높도록"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4 Precision/Recall trade off"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sklearn의 모델들에서 결과를 얻는 방법\n",
    " - predict : prediction의 값을 얻습니다. 회귀이면 실수 결과값을 얻습니다. 분류이면 카테고리 결과값을 얻습니다.\n",
    " - predict_prob : 예측 할 때 사용하는 값이 0과 1사이의 확률값을 따르는 경우 사용할 수 있습니다.\n",
    " - decision_function : 예측 할 때 사용하는 값이 실수일 때(예를 들어, 사용하는 loss function이 hinge loss 인 경우) 사용할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SGDClassifier가 분류를 결정하는 방법 : `decision_function` (결정 함수)를 사용하여 각 샘플의 점수를 계산\n",
    " - 샘플의 점수 $\\geq$ 임계값(threshold) : 양성 클래스에 할당\n",
    " - 샘플의 점수 < 임계값(threshold) : 음성 클래스에 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   3  18  18  18 126 136 175  26 166 255\n",
      " 247 127   0   0   0   0   0   0   0   0   0   0   0   0  30  36  94 154\n",
      " 170 253 253 253 253 253 225 172 253 242 195  64   0   0   0   0   0   0\n",
      "   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251  93  82\n",
      "  82  56  39   0   0   0   0   0   0   0   0   0   0   0   0  18 219 253\n",
      " 253 253 253 253 198 182 247 241   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0  14   1 154 253  90   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0  11 190 253  70   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  35 241\n",
      " 225 160 108   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0  81 240 253 253 119  25   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0  45 186 253 253 150  27   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252 253 187\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0 249 253 249  64   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253\n",
      " 253 207   2   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0  39 148 229 253 253 253 250 182   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253\n",
      " 253 201  78   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0  23  66 213 253 253 253 253 198  81   2   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0  18 171 219 253 253 253 253 195\n",
      "  80   9   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  55 172 226 253 253 253 253 244 133  11   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0 136 253 253 253 212 135 132  16\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "   0   0   0   0   0   0   0   0   0   0]\n"
     ]
    }
   ],
   "source": [
    "# Sample을 하나 추출한다.\n",
    "print(x_train_2d[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-45148.19955869])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 결정 함수에 사용된 점수를 확인\n",
    "y_scores = sgd.decision_function([x_train_2d[0]])\n",
    "y_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사이킷런의 `cross_val_predict`는 이미 내부에서 임계값을 사용하여, 임계값보다 크면 True, 작으면 False를 반환하게 합니다(binary의 경우). 그리고 사이킷런에서는 임계값을 직접 설정할 수 없습니다. 즉 y_score를 `decision_function`을 통해 구했다면, 우리가 직접 임계값을 설정해주고, 임계값에 따른 분류결과를 리턴하는 코드를 구현해야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "현재 분류기의 결과값 : True\n",
      "실제 라벨의 값 : False\n"
     ]
    }
   ],
   "source": [
    "print('현재 분류기의 결과값 :', y_train_pred[0])\n",
    "print('실제 라벨의 값 :', y_test_5[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "결과는 보는바와 같이 다릅니다. 우리는 이 경우 *y_train_pred*를 구하는 식을 수정할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def customized_pred_results(result_of_decision_function, threshold):\n",
    "    return result_of_decision_function > threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customized_pred_results(y_scores[0], 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이렇게 임계값을 조절하면 예측의 결과를 조절할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "하지만 만약에 임계값을 변경하게 되면 예측값이 변화하게 되어, precision / recall이 변하게 됩니다. 따라서 적절한 임계값을 정하는것이 중요합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- <img src='./images/precision_recall.png', width=\"1000\" height=\"600\"> -->\n",
    "<img src=\"./images/precision_recall.png\" style=\"height:600px width:1000px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cross_val_predict`를 활용하여 precision recall을 조정해보겠습니다.\n",
    "\n",
    "`cross_val_predict`은 cross validation을 이용하여 결과를 반환해주는 함수입니다. 단, 그 결과는 *method*에 따라 달라집니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores = cross_val_predict(sgd, x_train_2d, y_train_5, cv=3, method='decision_function') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 130454.39322866, -292310.89416081, -851504.62149277,\n",
       "       -463884.18557022, -419318.98872844, -362596.81598131,\n",
       "       -376116.98274832, -524189.31146637,  -39326.60070662,\n",
       "       -219343.91422559])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_scores[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Threshold의 길이 : 59664\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-1689990.81265398, -1689514.10236993, -1689310.7182735 ,\n",
       "       -1688408.63594058, -1688403.31253102, -1687979.50911516,\n",
       "       -1687943.53878695, -1687314.58893952, -1686755.23292283,\n",
       "       -1686271.09265537])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Threshold 리스트를 반환합니다.\n",
    "print('Threshold의 길이 :', len(thresholds))\n",
    "thresholds[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 낮은 임계치를 사용하여 5와 5가 아닌것의 분류를 예측하겠습니다. 임계치를 이렇게 사용하여 분류기를 만들게 되면 우리의 분류기는, 599663개를 5로 예측한 것입니다. 따라서 recall은 거의 1과 동일하게 되고 precision은 매우 작게 됩니다. 왜냐하면 우리의 분류기가 대부분의 숫자를 5라고 분류했기 때문입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000\n",
      "59663\n"
     ]
    }
   ],
   "source": [
    "print(len(y_scores))\n",
    "print(sum(y_scores > thresholds[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision : 0.09085880933226066\n",
      "recall : 1.0\n"
     ]
    }
   ],
   "source": [
    "print('precision :', precisions[0])\n",
    "print('recall :', recalls[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAFBCAYAAAB0NWlIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzs3XmcjvXi//HXZ3bLGDsxtmQ3DMaeoyQktJFIWrRLi9TR9zipU51fSpsiUdKqUimibNWRFMa+72QsYRhmzIzZrt8fH2PGPrhnrvu+5/08j/tx33Nf131d76HD22c+1+cyjuMgIiIiIlLYBbgdQERERETEG6gYi4iIiIigYiwiIiIiAqgYi4iIiIgAKsYiIiIiIoCKsYiIiIgIkIdibIyZYIzZZ4xZfZbtxhgzyhiz2Riz0hjT1PMxRURERETyV15GjCcCXc6x/Tqg1vHH/cC7lx5LRERERKRgnbcYO44zDzh4jl1uAD52rD+BksaYyzwVUERERESkIHhijnFlYGeur+OOvyciIiIi4jOCCvJkxpj7sdMtKFasWLO6desW5OlJTk9m55GdZGVlnXjP4eJviZ3nz55lt0s5d54/n0/nznKycBznpOOceO3ndxk3xtgHJue97NfmlK+zX5/yfqAJxL5tThwz9/bcXwcGBJ71mCf2z3WOABNAgAnAZP/P5Bwz0ASeyJ57W+7PnOm4ub82xmbKnUdERC7M6tWQmQlhYRASAjVq2Pd37oTk5JP3DQuDatXs6x07IDX15O1Fi0KVKvb1tm2Qlnby9uLFofLxIcstWyAj4+TtJUrAZcd/1r9pE+SqSQCULAkVKtjXGzac/r2ULg3lytnPbdp0+vayZaFMmdPfL0hLliw54DhOufPt54livAuokuvryOPvncZxnHHAOICYmBgnNjbWA6cXb+Q4DllO1mmPo+lHSc9Mx8E5sU/260t5LyktiUwn86zndTjz+ynpKRzLPHbi68yszDPul/3IyMogKS3ppGPmPudp75F1UtYsJ4u0zDSS05NPnCvTyTzt/NnvnZrvfDkznUyS0pIK5Pc4KCCIEqElKB5SnKCAIIICgggOCCYoIIjQoFAiQiMIDQolJDCEYsHFCA8JJzgwmOCA4BPPZYqWISI0gjJFyxAcEExgQCBBAUEEmuPPAYEEBwRTsXhFyhcrf+IfCSLim9avhz17bIkqV84WquBgd7Js3AhHj0LFijZLUIEOFUJ4ODz4ILz5ZsGet7AyxuzIy36e+M9gKvCIMeYLoCVw2HGcPR44rviw7NHJQE4uMsVCirmUqPDILs4ZWRknPY5lHiPxWCKZzunbjqYdJSUj5awFPfd7yenJJKYlkpyezMGUgxzLPEZGVgbpmeknjpeUlkRiWiIHkg+QlpnG4WOHSUlPIT0rnfTMdNKz7L4XIiI0glplahEaGHpSwS4RWoIiQUVOKtzhoeGEh4QTFhRGeGg4lcMrU6F4BYqHFCciNILSRUqfGC0XEc9KSIAZM6BUKfj5Z4iMhMceg8OHoV69k/dt1Qr++MO+vuceOxJ6xRU5j9q17Wjjhdi/H4oUsaOkZ3P0KDRqBMeO2a+NseX4wQfh+eftiOucOdCgQc5IrSdlZsKtt0Lbtp4/tlya8xZjY8wk4CqgrDEmDhgOBAM4jjMWmAF0BTYDycDd+RVWRM4vwAQQEBhAcKBLwzB5lJmVycGUgySkJnAw5SCZTuaJQp/7dUpGCrsTd7Ni7wp2Je46Ua5TM1JJy0xj/YH1pGWmnSjc6ZnpJKYlnvPcwQHBXFH6CiJLRFKxeEVql6lNRGgEpYqUomzRspQvVp7IEpGUK1pOBVoKtW3bICLCjuzmVc+eMHduzteNGtlivGePHZV98klo0sQW2NzHTUqC2Fj48sucH+XfdBN8+619/fjj9sf9NWvmPEqUOPncO3bA5ZdDnTqwdq1974kn4MAB+37jxvbc5cvD99/DmjW2RO/dax9XX20/s2kTXH+9fV2xIlx7LTRtavN4oigHBsIHH1z6ccTzjOO4MyFUUylEJL9kZGWQmpFKakYqB1MOsuvILg4kHyA5PZkDyQfYcXgH6w+sJzEtke0J29mbtPeMxwkNDKV8sfIUDS5KWFAYEWERRJaIpEKxChQLLnaiWFeJqEK1iGoaiRav4Th2DmupUnZuqTG27EVE2JKYmmrnjFaqBHFx8NZbULUqjB8Pu3fDmDHQuzc0b27L6mWXQcOGEBUF7drBjTfa48XG2sJYtKgtjIcOwddfw733wu+/27JZu7b9bF6lpcH27bB5sy2+V15pR5sbNrRZc3v2WTvCe+SILcBpafDpp/D//h8MHWr36dULFi+2c3ezC3enTjBz5tkzpKTAwoWwYAEsXQrz5tki//PPtjx//z0MGQL160OXLtCsmZ3jW6ECBORhWYPMTLuf/rgoOMaYJY7jxJx3PxVjESnsUjNSSUpLYt/RfSSkJrAncQ87j+wk7kgc+5P3k5qRyrGMY+xP3k/ckTgOphw841zukmElqV6yOvXK1uPWBrcSVT6K4iHFKRJchPCQcJVmKRDTp0O3bjlf//GHnbJQpgwcPGXx1cOH4ZdfbNHNbdgweOEF+OknWLXKluzVq+0obLduMHkyjBplR4JzmznTTj/Iysq5GMyTjhyxF49t2WIvAmvdGjp0sEW6TRv4+287Grt9uy39uaWkwLJl9nOVK9tynFeOY/9BER5uL4SbNw/eecceb/PmnP127bLnnTXLvq5Tx45SFztlFuHUqXZkfdEiiI6+2F8NuRAqxiIi+SgzK5PtCduJT4kn7kgcWw9tZcvBLWxN2MqiXYtISE04af/suc8lw0rSKrIVjSs0JrpiNE0qNqFcsfNeKC1ykq1b7ZSE4GBbUq+80k5BSEqyZQzsnNnhw6F/fzvfdvJke8HZb7/ZaQgtW8Kdd9pR3lWr7FzgXbvssc72b7jMTFtOS5Wyo7Nr18KKFfaRnAzvvuvuKGh6uh0NDw8vmPM5jp2OsW0b/PWXnaMcGAi33WZ/P8CO0i9cmPP7AnZk/v777dSPqlULJmthp2IsIuKSYxnHWLBzAdsTtpOSkcLRtKMnpnL8deQvlu5ZStyRnJ8JVwqvRGSJSC4vdTlNKjahXtl61ChVgytKX0FYUJiL34l4k5QUmDLFlq5hw+x0AbA/kq9dG/7v/+wo6KBBdvpAr17u5i3MMjLsqPXbb9uR9eLFbXEuVQpWrrT/qJkzx/6ehun/4gVCxVhExIvFHYljY/xGftvxG1sTtrIncQ9r9q9hd+LuE/sEmABqla5FTKUY2lRpQ1T5KFpFtvL6CyvF83bvhvvus6s9TJ1q5/z+/DMMGOD++rBybgsW2H+0zJljLz586CEYO9Zuc6mCFUoqxiIiPmjf0X12Ssahraw7sI4le5awMG4hh1IPAbYst6zcksYVGjOg6QCaXdZMc5f9wJYtMGGCnbeamGgL01NPwcsvQ+fOtlRl27PHrpQgvmnPHjvvOywMunZ1O03hkddiXMDLWYuIyLmUL1ae8sXK07pK6xPvZTlZbE/YzoKdC5j/13xid8cybuk4xi4Zy2XFL6NO2To0KNeAHnV6cFX1qwgJDHHxO5ALMW+eXUbs6FH4739P3paZaZ+vvdYW45tugltuUSn2dZddBjff7HYKORuNGIuI+KCE1AQ+XPYhsXti2Ri/kdjd9s/TUmGl6FSzE0+2fpLmlZu7nFJONWaMXeM3LMyuz5uaCh9+CH362Dmp1appzqlIftCIsYiIHysZVpInWj9x4uuktCS+WfsNs7fOZtrGaXy55kvqlq1LlRJV6FGnB7c1vI2yRS/wFmJySRzHzglet86u+HDbbfa9b7+1F2GFhNg5p+3aQWjoyasWiIg7NGIsIuJn4pPj+XD5h/z2128s27OMnUd2AtCyckvuaHQHHS/vSO0ytTU3OR/Nnw9XXZUzHQLserfly9s7rQVpWEqkQOniOxERwXEcYnfHMmX9FKZtnMbqfasBqFCsAj3r96Rrra50uaILASYPt+uS8zpwAEqWhFtvtUurAXz3nb1b2qm3LxaRgqNiLCIiJ3Ech2V7l/HT5p9YsHMBs7fOJi0zjSYVm/Blzy+pVaaW2xF92jPPwJtv2rvLLVxobzLRrJnbqUQEVIxFROQ8Eo8l8uHyD/nnnH8SHBDMkDZDGNRiEKWKlHI7mk9Zt86OCP/9t/06+yI6EfEeeS3G+tmZiEghFR4azqMtH2XlgytpV60dw38dTvmR5RkbO9btaD4jIQFat84pxUePqhSL+DIVYxGRQq5WmVpM7zudn/v/TJOKTXho+kN0n9Sd9QfWux3N6x07BtWrw+jRkJUFRYu6nUhELoWKsYiIAHB1jatZMGAB/2r3L6ZvnE6bD9rw46YfcWvKnbfatw+aNoUbb7RlePlyePhh0CIfIr5PxVhERE4ICgjixQ4vsvSBpZQILUHXz7sS9W4U83bMczuaV5g7FypUgGXLYOZMKFbM7UQi4kkqxiIicproitGsHbiWkdeOZG/SXtpPbM8NX9zA1kNb3Y7mmvHjoWNH+7p7d0hO1hJsIv5GxVhERM6oaHBRnmzzJBsHbeSfbf/JnK1zaPRuIz5Z8Ynb0QrU0qWwc6cdKQ4JsaPGU6dq6oSIP1IxFhGRcypdpDQvd3yZlQ+upE7ZOvT/rj/dJ3UnPjne7Wj57pdf7FrE77wDPXrYi+06dHA7lYjkFxVjERHJk5qla7Lw3oU8+49n+WHjD7Sd0JYtB7e4HSvfTJmSU4Lbt3c3i4gUDBVjERHJs6CAIJ6/+nmm953O7sTdNBjTgInLJ7ody+MaNoSbb7avv/sOunZ1N4+IFAwVYxERuWBda3Vl5UMraRnZkru/v5tRC0e5Hcmjhg+3z7Nnww03uJtFRApOkNsBRETEN1UvWZ25/efS+dPOPDnrSTKyMrg7+m6fvqV0jx5w1VUweDBo+WaRwkcjxiIictGCAoL44pYvaFyhMU/OepLWH7TmYMpBt2NdlO+/h2nT4Mkn3U4iIm5RMRYRkUtSrlg5Ft23iB/6/MCG+A08NP0hjmUcczvWBZk2zd7JDmDVKneziIh7VIxFROSSBZgArq99PcPaDeOrNV/R55s+pGWmuR0rT/buhb59ISwMVq+2F96JSOGkYiwiIh7zQocXeKXjK0xZP4V/fPgPn1jObf9+KF4cfv4ZGjRwO42IuEnFWEREPOqptk/xQY8PWLN/DdHvRbN632q3I51TVBTs3g2tW7udRETcpmIsIiIed0+Te1j90GoCTSB3fXcXm+I3uR3pNKtX23nFCQm6vbOIWCrGIiKSL6qVrMZLHV5iyZ4lRL0bxccrPnY70gkpKXak+Pvv4cABt9OIiLdQMRYRkXwzsMVAVj20iioRVbjzuzvpP6U/6Znpbsfiscfsc1QUXHGFu1lExHuoGIuISL5qWL4hqx5axZDWQ/hk5Sc8+uOjOC7ePSM2FsaPt69XrHAthoh4Id35TkRE8l1YUBivdnqVtMw0Ri0aRWhQKK93fp0AU/DjM6uPXwu4bJnmFovIyVSMRUSkwLzZ5U0OpBzgrYVvsS1hG9/c+g1BAQXzV1FWlp1b3KcP3HADlPLdO1eLSD7RVAoRESkwxhg+velTXun4ClM3TOWpWU8V2LljYqBFC/tapVhEzkTFWERECpQxhqfaPsUDzR7gzYVvMnrR6Hw/58aNdupE1aoQEpLvpxMRH6ViLCIirnin6zt0qNGBx2c+zsK4hfl2ntRUqFPHvh45UvOKReTsVIxFRMQVQQFBfHLTJ5QKK8U1H1/DhgMb8uU8d9xhn195Rbd8FpFzUzEWERHXVAqvxG93/0aACaDHFz3YkbDDo8fPyoKMDGjfHp4quOnMIuKjVIxFRMRVdcrWYUrvKew6sourP7qanYd3euS4WVkQEABffQW//uqRQ4qIn1MxFhER111z+TXM7T+X/cn7afl+S+KOxF3yMVu2hHvvhSAtTCoieaRiLCIiXqFlZEvm3TWP+JR4+n3bj7TMtIs+1h9/2DvcZWbqYjsRyTsVYxER8RpNLmvCa51e4387/nfRy7gdO2ZHiqtUgVGjPBxQRPyairGIiHiVgc0H8o9q/+DJWU8ya8usC/78M8/A2rXw7rsQHp4PAUXEbxnHcVw5cUxMjBMbG+vKuUVExLslpSXRbFwzjhw7wvbHthMaFJqnz6WlQdGiUL8+rFyZzyFFxGcYY5Y4jhNzvv00YiwiIl6neEhxXurwEnuT9vLekvfy/LmQEEhIgDlz8jGciPitPBVjY0wXY8wGY8xmY8zQM2yvaoz5xRizzBiz0hjT1fNRRUSkMLml3i10qNGBJ2c9yep9q8+7f0oKJCZC8eJQvnwBBBQRv3PeYmyMCQRGA9cB9YE+xpj6p+w2DPjKcZwmwG3AGE8HFRGRwsUYw6RbJlE8pDh3f383GVkZ59x/0CC4/HL4++8CCigificvI8YtgM2O42x1HCcN+AK44ZR9HKDE8dcRwG7PRRQRkcKqfLHyvNbpNWJ3x/Lxio/Put/atfDBB1CqFFSoUIABRcSv5KUYVwZy34Yo7vh7uT0H9DPGxAEzgEFnOpAx5n5jTKwxJnb//v0XEVdERAqbu6LvokXlFgyeOZj45Pgz7vPEE/Z54sSCyyUi/sdTF9/1ASY6jhMJdAU+McacdmzHccY5jhPjOE5MuXLlPHRqERHxZwEmgPe6vcfhY4d5/n/Pn7b94EGYNQuqVoU2bVwIKCJ+Iy/FeBdQJdfXkcffy20A8BWA4zh/AGFAWU8EFBERia4YzT3R9zBm8RgOphw8adu2bXY1ii++cCmciPiNvBTjxUAtY0wNY0wI9uK6qafs8xdwDYAxph62GGuuhIiIeMz9ze4n08lk3JJxJ73fqBEsWACtW7sUTET8xnmLseM4GcAjwExgHXb1iTXGmP8YY3oc3+1J4D5jzApgEnCX49adQ0RExC+1jGzJTXVv4j//+w8Hkg8AMHkyDBsGTZu6HE5E/ILufCciIj5jxd4VRL8XzRud3+Cxlo8TcHx4JysLjHE3m4h4L935TkRE/E7jio1pHdmakQtG8uATdq7xsGEqxSLiGSrGIiLiU17r9Bp7k/YybtdDQM5SbSIil0rFWEREfErrKq35Z9t/QsOvuPHJ2ZQu7XYiEfEXKsYiIuJzhv1jGGWKlCG45Xi3o4iIH1ExFhERn5KUBJ06FKFlsT5M2zjtxAoVIiKXSsVYRER8yvXXw/z5cFut+0jPTGf4L8PdjiQifkLFWEREfMb+/TBvnn19R6dG3NPkHsYvHc+exD3uBhMRv6BiLCIiPmP88SnFn39un59u+zRZThZPzX7KvVAi4jdUjEVExCekpcG4cdC+PfTpY9+7ovQVDL1yKJ+t+oyFcQvdDSgiPk/FWEREfEJICPzyC4wZc/L7T7d9mmLBxXjxtxfdCSYifkPFWEREvF5GBhw8CDVqQP36J28rEVqCR1o8wg8bf2Droa3uBBQRv6BiLCIiXu/mm6FMGUhNPfP2gc0HEmACeH/p+wUbTET8ioqxiIh4teRkmDbNvg4LO/M+VSKqcH2t6/lg2QccyzhWcOFExK+oGIuIiFebOtU+f/zxufcb2Hwg+47uY9LqSfkfSkT8koqxiIh4tZdess/ZK1GcTaeanWhQrgEjfh9BRlZG/gcTEb+jYiwiIl7r6FFYvRqioyEo6Nz7GmMYeuVQ1h9Yr7nGInJRVIxFRMRrhYXBoEEwalTe9r896naiK0YzevFospys/A0nIn5HxVhERLyS48CxY7YUt2uXt88YY3i85eOs3rea6Run529AEfE7KsYiIuKVxo6F666zq1JciL5RfakWUY0X5r2gUWMRuSAqxiIi4pUefhjmzTv7Em1nExwYzPD2w1m8ezFTN0zNn3Ai4pdUjEVExOskJOS8DriIv6n6NepHZIlIhswaQmrGWe4KIiJyChVjERHxOq+8Yp/nz7+4zwcHBvNet/fYcmgL45eM91wwEfFrxnEcV04cExPjxMbGunJuERHxbsbY56ysnNcXynEcOn7SkaV7lrLxkY2UK1bOcwFFxKcYY5Y4jhNzvv00YiwiIl7niSfg7rsvvhSDXaHi7eve5nDqYd5a+JbnwomI31IxFhERr/P66zBhwqUfp365+nSr3Y3xS8eTnpl+6QcUEb+mYiwiIl7DcWDkSNi82XPHfKDZA+w7uo8fNv7guYOKiF9SMRYREa+xbBk89RT8/LPnjtn5is5UDq/MsF+GkZaZ5rkDi4jfUTEWERGv8emnEBwMPXt67phBAUG80fkN1u5fy+hFoz13YBHxOyrGIiLiFTIzYdIkuP56KF3as8fuWb8n111xHc/++ixxR+I8e3AR8RsqxiIi4hV+/hn27oXbb/f8sY0xjLpuFMcyjvHYT495/gQi4hdUjEVExCts2gQVKkC3bvlz/CtKX8EjLR7hu/XfsfPwzvw5iYj4NBVjERHxCg8/DDt3QlhY/p1jUItBOI7DhGUeWAtORPyOirGIiLguNdU+Bwfn73lqlKrBVdWvYvzS8SSnJ+fvyUTE56gYi4iI63r1ghtvLJhz/avdv9iVuIvnfn2uYE4oIj5DxVhERFx14AD89BPUqlUw57vm8mvockUXPln5CVlOVsGcVER8goqxiIi46quvICMD+vUruHPe2fhO9ibtZd6OeQV3UhHxeirGIiLiqs8+gwYNoFGjgjtnjzo9KFu0LC/Oe7HgTioiXk/FWEREXLN1KyxYYEeLjSm48xYNLsozVz7D3G1z+WPnHwV3YhHxairGIiLimnLl4P33C3YaRbb7m91PybCSvPHnGwV/chHxSirGIiLimvBwGDAAIiML/tzFQ4pzX9P7+GbdN+xI2FHwAUTE66gYi4iIKxYuhJdegsRE9zIMajEIg+GthW+5F0JEvIaKsYiIuGLUKHj1VQgMdC9DlYgq9Inqw3tL3mPf0X3uBRERr6BiLCIiBW7/fvj8c7jjDiha1N0sw9oNIzUjlRHzR7gbRERcp2IsIiIF7tVX7fODD7qbA6BO2Trc2fhORi8ezV+H/3I7joi4SMVYREQKVFYWTJoEl19u1y/2Bs9d9RyZTiZv/vmm21FExEV5KsbGmC7GmA3GmM3GmKFn2edWY8xaY8waY8znno0pIiL+4tAhaNMmZ9TYG1SNqErvBr0ZGzuWnYd3uh1HRFxy3mJsjAkERgPXAfWBPsaY+qfsUwt4BmjrOE4D4PF8yCoiIn6gTBn48ku4+Wa3k5zsxQ4v4uDwfz//n9tRRMQleRkxbgFsdhxnq+M4acAXwA2n7HMfMNpxnEMAjuPo0l4RETlNVhasWuV2ijOrXrI6A5oM4Ks1X2mFCpFCKi/FuDKQ++dKccffy602UNsY87sx5k9jTBdPBRQREf/x6afQqBHMnet2kjN7pMUjpGWm8dafWtdYpDDy1MV3QUAt4CqgDzDeGFPy1J2MMfcbY2KNMbH79+/30KlFRMRXfPutfW7f3t0cZ1O3bF261+7Oh8s/JDMr0+04IlLA8lKMdwFVcn0defy93OKAqY7jpDuOsw3YiC3KJ3EcZ5zjODGO48SUK1fuYjOLiIgP+vtvmD4dBg+GoCC305xd/8b92ZO0h+mbprsdRUQKWF6K8WKgljGmhjEmBLgNmHrKPt9hR4sxxpTFTq3Y6sGcIiLi40aOhIwMuOcet5Oc2411b6RKiSqMWjjK7SgiUsDOW4wdx8kAHgFmAuuArxzHWWOM+Y8xpsfx3WYC8caYtcAvwFOO48TnV2gREfE933xjn71l7eKzCQoI4oFmDzB321x2JOxwO46IFKA8zTF2HGeG4zi1Hcep6TjOS8ffe9ZxnKnHXzuO4wx2HKe+4zhRjuN8kZ+hRUTE96xbB1u2uJ0ib7rX6Q7AjE0zXE4iIgVJd74TEZECERpq73bnC6LKR1GzVE0mr53sdhQRKUAqxiIiku9uvRXGj3c7Rd4ZY+jfuD+/bP+FTfGb3I4jIgVExVhERPLVpk0weTIkJrqd5MLc2/ReAkwAL89/2e0oIlJAVIxFRCRfDR9un6+7zt0cF6pSeCUeinmICcsnsP7AerfjiEgBUDEWEZF84ziwcqV9Xa+eu1kuxtArhxJoAnl38btuRxGRAuBVS6ynp6cTFxdHamqq21F8UlhYGJGRkQQHB7sdRUQEgAULYM0aeO89t5NcnMgSkdxY90YmrZ7EyE4jCQ7Un68i/syrinFcXBzh4eFUr14dY4zbcXyK4zjEx8cTFxdHjRo13I4jIgJAqVLwwAPQp4/bSS7eHY3u4Jt13zBryyyur32923FEJB951VSK1NRUypQpo1J8EYwxlClTRqPtIuJV6teHsWMhPNztJBfvulrXEREawdfrvnY7iojkM68qxoBK8SXQr52IeJPFi2HOHMjKcjvJpQkJDOGWerfw5eov2Xd0n9txRCQfeV0xFhER//DCC3DHHfYCPF/3dNunOZZ5jJELRrodRUTykYrxKQIDA4mOjqZhw4b06tWL5OTkSz5mbGwsjz766Fm37969m549e17yeUREvMWhQ/DTT3D77RAY6HaaS1enbB1uqnsTHy7/kGMZx9yOIyL5RMX4FEWKFGH58uWsXr2akJAQxo4de9J2x3HIusCfC8bExDBq1Kizbq9UqRJff625ayLiP77+GtLTffuiu1Pd2/ReDiQf4LNVn7kdRUTyiYrxObRr147Nmzezfft26tSpQ//+/WnYsCE7d+5k1qxZtG7dmqZNm9KrVy+SkpIAWLx4MW3atKFx48a0aNGCxMREfv31V7p16wbA//73P6Kjo4mOjqZJkyYkJiayfft2GjZsCNgLEO+++26ioqJo0qQJv/zyCwATJ07k5ptvpkuXLtSqVYunn37anV8UEZE8+OQTqFsXmjZ1O4nndKrZiXpl6zFq4Sgcf5gfIiKn8arl2k511VWnv3frrfDww5CcDF27nr79rrvs48ABOHV2wq+/5v3cGRkZ/Pjjj3Tp0gWATZs28dFHH9GqVSsOHDjAiy++yJw5cyhWrBgjRozg9ddfZ+jQofTu3Zsvv/yS5s2bc+TIEYoUKXLScUeOHMno0aNp27YtSUlJhIWFnbR99Oju5HzTAAAgAElEQVTRGGNYtWoV69evp1OnTmzcuBGA5cuXs2zZMkJDQ6lTpw6DBg2iSpUqef+mREQKwOHDsGULDBoE/nRNcIAJ4Kk2T3HP1HuYt2Me7au3dzuSiHiYRoxPkZKSQnR0NDExMVStWpUBAwYAUK1aNVq1agXAn3/+ydq1a2nbti3R0dF89NFH7Nixgw0bNnDZZZfRvHlzAEqUKEFQ0Mn/9mjbti2DBw9m1KhRJCQknLZ9/vz59OvXD4C6detSrVq1E8X4mmuuISIigrCwMOrXr8+OHTvy9ddCRORiRETAjh1wjksrfNatDW6lWHAxXlnwittRRCQfePWI8blGeIsWPff2smUvbIQ4W/Yc41MVK1bsxGvHcbj22muZNGnSSfusWrXqvMcfOnQo119/PTNmzKBt27bMnDnztFHjswkNDT3xOjAwkIyMjDx9TkSkoGRmwsSJ0KsXlCjhdhrPKxZSjH//498MnTuUhXELaRnZ0u1IIuJBGjG+CK1ateL3339n8+bNABw9epSNGzdSp04d9uzZw+LFiwFITEw8rbxu2bKFqKgo/vnPf9K8eXPWr19/0vZ27drx2Wf2wo6NGzfy119/UadOnQL4rkRELt3XX8O998LSpW4nyT8DWwykVFgphv0yTHONRfyMivFFKFeuHBMnTqRPnz40atSI1q1bs379ekJCQvjyyy8ZNGgQjRs35tprrz3tTnRvvvkmDRs2pFGjRgQHB3PdddedtP3hhx8mKyuLqKgoevfuzcSJE08aKRYR8WZvvmmfW7d2N0d+Kh5SnGH/GMacrXP4ZOUnbscREQ8ybv1rNyYmxomNjT3pvXXr1lGvXj1X8vgL/RqKiFv27IFKlezaxZ9+6naa/JWZlUmbCW3YdWQX2x7bRnBgsNuRROQcjDFLHMeJOd9+GjEWERGPyL7s4rHH3M1REAIDAnn2H8+yK3GXRo1F/IiKsYiIXDLHgffes1Moji/M4/e61upKVPko3vzzTbKcC7vxk4h4JxVjERG5ZMbA99/D6NFuJyk4xhieavMUq/at4tt137odR0Q8QMVYREQ8om5daNLE7RQFq29UX6qUqMIbf77hdhQR8QAVYxERuSRZWfDQQ/DHH24nKXiBAYHc1/Q+/tj5B1sPbXU7johcIhVjERG5JLNmwdixsH2720ncMaDpAIICgnhn0TtuRxGRS6RifIrAwECio6Np2LAh3bt3JyEhwaPHnzhxIo888ggAzz33HCNHjvTo8UVECtoNN9jnW25xN4dbKoVXolvtbkxaPYnMrEy344jIJVAxPkX2LaFXr15N6dKlGV2YriQREblAcXGQlgY9ekBIiNtp3HN71O3sTdrL3G1z3Y4iIpdAxfgcWrduza5du058/eqrr9K8eXMaNWrE8OHDT7z/8ccf06hRIxo3bswdd9wBwLRp02jZsiVNmjShY8eO/P333wWeX0Qkv2XfyOP1193N4bbra19PqbBSjFk8xu0oInIJgtwOcDaP//Q4y/cu9+gxoytG82aXN/O0b2ZmJnPnzmXAgAEAzJo1i02bNrFo0SIcx6FHjx7MmzePMmXK8OKLL7JgwQLKli3LwYMHAbjyyiv5888/Mcbw/vvv88orr/Daa6959PsREXFbuXLQvz/UrOl2EneFBYXxQLMHeHXBq8Qnx1OmaBm3I4nIRfDaYuyWlJQUoqOj2bVrF/Xq1ePaa68FbDGeNWsWTY6vRZSUlMSmTZtYsWIFvXr1omzZsgCULl0agLi4OHr37s2ePXtIS0ujRo0a7nxDIiL5aMAA+xDoWb8nL//+MtM2TuOu6LvcjiMiF8Fri3FeR3Y9LXuOcXJyMp07d2b06NE8+uijOI7DM888wwMPPHDS/m+//fYZjzNo0CAGDx5Mjx49+PXXX3nuuecKIL2ISMGZNQuuuqpwzy3OrellTakWUY1JqyepGIv4KM0xPouiRYsyatQoXnvtNTIyMujcuTMTJkwgKSkJgF27drFv3z46dOjA5MmTiY+PBzgxleLw4cNUrlwZgI8++sidb0JEJJ/s3g3dusFLL7mdxHsYY+jXqB9zts5hT+Iet+OIyEVQMT6HJk2a0KhRIyZNmkSnTp3o27cvrVu3Jioqip49e5KYmEiDBg3417/+Rfv27WncuDGDBw8G7FJsvXr1olmzZiemWYiI+It334WMDDu/WHL0jepLlpPF6MVa0UjEFxnHcVw5cUxMjBMbG3vSe+vWraNevXqu5PEX+jUUkfx27BhUrQotWsC0aW6n8T49v+rJ7K2z2fH4DkqGlXQ7jogAxpgljuPEnG8/jRiLiMgF+eYb2LcPjt+rSE7xr3b/4sixI4xepFFjEV+jYiwiIhdk9my44go4vmiPnKLJZU3oWqsrb/z5BkfTjrodR0QugNcVY7emdvgD/dqJSEGYMAHmz4cAr/sbxHv8q92/iE+J54NlH7gdRUQugFf9sRYWFkZ8fLwK3kVwHIf4+HjCwsLcjiIifiw9HYyBChXcTuLd2lRpQ9sqbXl70dv6O03Eh3jVOsaRkZHExcWxf/9+t6P4pLCwMCIjI92OISJ+6tAhqFsX3ngD+vZ1O433uyv6Lu6bdh+xu2NpXrm523FEJA+8qhgHBwfrDnEiIl5qwgR70V2DBm4n8Q096/fkkRmP8PGKj1WMRXyEV02lEBER75SZCaNHw5VXQuPGbqfxDSXDStK9TnfeX/Y+cUfi3I4jInmgYiwiIuf1ww+wbRsMGuR2Et8youMI0jLTeOOPN9yOIiJ5oGIsIiLndeON9vnmm93N4WsuL3U53Wp3Y+KKiaRlprkdR0TOQ8VYRETOa8QIeOcdCPKqK1N8w52N7+RgykEW7VrkdhQROQ/9ESciIufkOPD0026n8F1XV7+aQBPIF6u/4MqqV7odR0TOIU8jxsaYLsaYDcaYzcaYoefY7xZjjGOMOe+9qEVExPulpdlVKObNczuJ7ypVpBS3NbyN95e+z8GUg27HEZFzOG8xNsYEAqOB64D6QB9jTP0z7BcOPAYs9HRIERFxx2efwbp1EKdFFS7JkDZDOJZ5jC9Wf+F2FBE5h7yMGLcANjuOs9VxnDTgC+CGM+z3AjACSPVgPhERcYnj2Jt5NG4Mffq4nca3Na7QmIblGzJuyTjdCU/Ei+WlGFcGdub6Ou74eycYY5oCVRzHme7BbCIi4qLffoNVq+Dhh+1toOXiGWMY2HwgK/5ewZI9S9yOIyJnccmrUhhjAoDXgSfzsO/9xphYY0ysbvssIuLdXnkFypWDfv3cTuIfbmt4G8VDivPWwrfcjiIiZ5GXYrwLqJLr68jj72ULBxoCvxpjtgOtgKlnugDPcZxxjuPEOI4TU65cuYtPLSIi+cpxoH17+Pe/oWhRt9P4h5JhJekX1Y9v1n5DaoZmHYp4o7wU48VALWNMDWNMCHAbMDV7o+M4hx3HKes4TnXHcaoDfwI9HMeJzZfEIiKS74yBp57Sne487baGt5GSkcKI+SPcjiIiZ3DeYuw4TgbwCDATWAd85TjOGmPMf4wxPfI7oIiIFKytW+HTT+1SbeJZ7au3p3eD3rz8+8skpCa4HUdETpGnOcaO48xwHKe24zg1Hcd56fh7zzqOM/UM+16l0WIREd/15ptwzz1wUEvu5oshbYaQmpHK6EWj3Y4iIqfQLaFFROSEgwfh/fehb1+oWNHtNP4pplIMnWp2YtzScWRmZbodR0RyUTEWEZET3nsPUlLgyfOuMySX4v6m9/PX4b/4aMVHbkcRkVxUjEVEBIDUVHjrLejUCaKi3E7j326udzPNLmvGS7+9RHpmuttxROQ4FWMREQFg/34oVQqeftrtJP7PGMO///Fvth7ayg8bf3A7jogcp2IsIiIAVKkCy5dDhw5uJykcutbqymXFL+Pd2HfdjiIix6kYi4gI330H8fEQGqrbPxeU4MBgBrUYxOyts5m9ZbbbcUQEFWMRkUIvPh5uugmeecbtJIXP460eJyI0gjGxY9yOIiKoGIuIFHrPPmuf77vP3RyFUZHgItzf7H6mbZjG3qS9bscRKfRUjEVECrHkZPjwQ6hZE5o3dztN4XRX9F1kOpl8suITt6OIFHoqxiIihdiYMXbd4o+0nK5r6perT4caHRjx+wiOHDvidhyRQk3FWESkENu5E7p2hbZt3U5SuL18zcvEp8Tz0XL9C0XETcZxHFdOHBMT48TGxrpybhERyZGZCYGBbqeQVu+34lDqIdYNXEeA0biViCcZY5Y4jhNzvv30/zwRkUJo2zZYuNC+Vin2Do+0eISN8RuZu3Wu21FECi0VYxGRQmjQIOjcGRIT3U4i2XrV70W5ouV4e9HbbkcRKbRUjEVECpnp0+1j2DAID3c7jWQLDQrlwZgHmbZxGtsObXM7jkihpGIsIlKIpKdD9+5QtSo8+qjbaeRUA5oMAOCzVZ+5nESkcFIxFhEpRN57DxwHhg+HkBC308ipqpWsxjU1rmH80vFkOVluxxEpdFSMRUQKEWPghhvg7rvdTiJnc1f0Xfx1+C8Wxi10O4pIoaNiLCJSiAwcCFOm2IIs3ql77e4EBwTrIjwRF6gYi4gUAqtXwxdf2GkUKsXeLSIsgsdaPsak1ZNY9fcqt+OIFCoqxiIifs5x7IV2AwfC4cNup5G8GHrlUMoWLcsDPzygucYiBUjFWETEz337LfzyC7zwApQs6XYayYsyRcsw8tqR/BH3B5+v+tztOCKFhm4JLSLix1JSoH59u17x0qUQFOR2Iskrx3FoNLYRACsfXInRHBiRi6ZbQouICG+9Bdu322eVYt9ijGFI6yGs3reamVtmuh1HpFBQMRYR8WN16sDDD8PVV7udRC5Gn6g+RJaIZPDMwaRmpLodR8TvqRiLiPixm26C0aPdTiEXKyQwhPHdx7PuwDpGL9JvpEh+UzEWEfFDL74I//2vXZFCfFuXK7rQuWZnXvrtJY4cO+J2HBG/pmIsIuJn5s+3t3zesEFrFvuLZ9s/y6HUQ3yy4hO3o4j4NRVjERE/kpAA/fpB9erwzjtupxFPaR3ZmuaVmjMmdgxurSYlUhioGIuI+AnHsRfaxcXB55/bJdrEPxhjeCjmIdbuX8uU9VPcjiPit1SMRUT8xJo18NVX8Pzz0LKl22nE025reBvVIqoxZNYQ0jLT3I4j4pdUjEVE/ETDhhAbC0OHup1E8kOR4CK80/UdtiVsY9yScW7HEfFLKsYiIj4uKQlmzLCvo6MhMNDdPJJ/rq91Pe2rtWf4r8M5lHLI7TgifkfFWETEh2Vlwd13Q48esGmT22kkvxljeP6q5zmYcpD3l77vdhwRv6NiLCLiw4YPh6+/hhEjoFYtt9NIQWhfvT1XVr2S8UvHa4UKEQ9TMRYR8VGffmpv5HHvvTB4sNtppCDd2+ReNh3cxO87f3c7iohfUTEWEfFBO3bAgAFw9dX2ls+6kUfhckv9WygeUlzTKUQ8TMVYRMQHVasGEyfaaRQhIW6nkYJWPKQ4/Rv159OVn7I7cbfbcUT8hoqxiIgPOXwYli2zr/v0gdKl3c0j7hncejAODv/++d9uRxHxGyrGIiI+Ii0Nbr3VTp84pJW6Cr2apWvyeMvHmbB8Ar/t+M3tOCJ+QcVYRMQHZGRAv34waxa8/jqUKuV2IvEGL3R4gcrhlXl6ztNaoULEA1SMRUS83LFj9q52kyfDyJFwzz1uJxJvUTS4KM+2f5Y/4/5kztY5bscR8XkqxiIiXm7MGNiwAV59FZ580u004m3ubHwnlcMr89hPj5Gcnux2HBGfpmIsIuLlHngAxo6FIUPcTiLeKDQolLHdxrLuwDrGLB7jdhwRn6ZiLCLihQ4dgr59YedOKFrUlmORs+lWuxudanbi5fkvczDloNtxRHyWirGIiJfZvRvat7drFK9Y4XYa8RWvdHyFgykHeXn+y25HEfFZeSrGxpguxpgNxpjNxpihZ9g+2Biz1hiz0hgz1xhTzfNRRUT835Il0KIFbNkCM2ZAt25uJxJf0bhiY/o16seohaPYnrDd7TgiPum8xdgYEwiMBq4D6gN9jDH1T9ltGRDjOE4j4GvgFU8HFRHxd7NmQZs2EBAACxZAx45uJxJf82KHFwF4/KfHtXybyEXIy4hxC2Cz4zhbHcdJA74Absi9g+M4vziOk30p7J9ApGdjioj4v1at4N577Z3tGjd2O434oqoRVXmxw4t8v+F7vt/wvdtxRHxOXopxZWBnrq/jjr93NgOAH8+0wRhzvzEm1hgTu3///rynFBHxU3v2wMCBkJwMJUrA6NFQpozbqcSXPd7qcSJLRPL07Kc5mnbU7TgiPsWjF98ZY/oBMcCrZ9ruOM44x3FiHMeJKVeunCdPLSLic955BypVsusUL17sdhrxF0EBQbzR+Q02HdzEnd/dqSkVIhcgL8V4F1Al19eRx987iTGmI/AvoIfjOMc8E09ExP/8/be9e92gQfbrRYvsKhQintKzfk+ea/8c36z7humbprsdR8Rn5KUYLwZqGWNqGGNCgNuAqbl3MMY0Ad7DluJ9no8pIuI/Pv4YPvkEnnnG3u65eXO3E4k/eqbdM9QsVZO7vruLPYl73I4j4hPOW4wdx8kAHgFmAuuArxzHWWOM+Y8xpsfx3V4FigOTjTHLjTFTz3I4EZFCacsW+PVX+/rRR2HNGvjvfyEkxNVY4sdCAkOY1mcaR9OP8tTsp9yOI+ITjFtzj2JiYpzY2FhXzi0iUlCOHoXXX4eXX4Zq1WD1arscm0hBeWbOM7z8+8tM7zudrrW6uh1HxBXGmCWO48Scbz/98Swikg8yM2HCBKhdG559Fq67DmbOVCmWgjf0yqE0LN+Q3l/3Zvne5W7HEfFq+iNaRCQfzJ4NAwZA1arw++/29s5Vqpz/cyKeFhEWwYy+MygSVIRrPr6Gvw7/5XYkEa+lYiwi4iGrVsGkSfZ15852hHjBAns3OxE3VYmowvx75pOWmaYl3ETOQcVYROQSrVkD/ftDdDQ8/TSkpYEx0KmTfRbxBrXL1GZExxH8uv1Xpm2c5nYcEa+kYiwicpE2bYK+faFhQ7v82mOPwYoVWmlCvNd9Te+jXtl6DPpxkJZwEzkDFWMRkQuQmQmHDtnXqakwfTr07g1//WVXnyhd2t18IucSHBjMe93eIz45ni6fdSEhNcHtSCJeRcVYRCQPduyA//wHrrgCBg6070VFwZ498MUXurBOfEe7au2Y0nsKa/ev5fGfHnc7johXUTEWETmH776Dq6+G6tVh+HCoUQN69crZXrSoa9FELtq1Na9lSOshfLTiI6asm+J2HBGvoWIsIpJLejr89BNkZNiv//gDdu6E55+H7dvh55/hpptcjSjiEc9d9RzNKzXn9m9vZ9meZW7HEfEKuvOdiBR6hw/bMjxtGvzwQ87XnTtDSgqEhWl1CfFPe5P20nx8czKyMvj9nt+5vNTlbkcSyRe6852IyFk4Dhw7Zl+vWgVly8Jtt9kyfPPNMHWqnT4BUKSISrH4r4rFKzK973RSM1Lp+HFHktKS3I4k4ioVYxEpFBIS7N3n7r8fLr/crjcMUL8+PPMMzJ8Pf/9tb+PcvbuWXJPCo1GFRnzX+zu2J2xn0I+D3I4j4qogtwOIiOSHtLScctujh11WLSsLSpSADh2gbVu7LTDQrjYhUpi1r96eIW2G8OqCV6lSogrPX/U8Rj8qkUJIxVhEfJ7jwObNsGiRvVju99/tPOGtW+32Fi3sXek6dYKWLSE42N28It7o+aue5++jf/PCvBc4mnaUEdeOIChANUEKF/0XLyI+xXHszTSWLYPrr7cld8gQe3MNgGLFbPm94Qa7wkRwMAwb5m5mEV9QJLgIH97wIWGBYbz+5+usPbCWH/r8QGBAoNvRRAqMirGIeKXsBXOMsSX4/fdh7Vp7sVx8vN22YgU0agQ9e0LdurYQ168PQfqTTeSiBJgAxnYbS4PyDXjsp8d47KfHeK3Ta4QGhbodTaRA6K8PEXFdUhIsXAhr1tjym/349FPo0gX27rWvGzSwawg3bQpNmkCdOvbzrVvbh4hcOmMMj7Z8lA0HNjB68Wi2JWxjSu8phATqilTxfyrGIpLv0tNh+XJ7W+Xt2+3z1q1w1132LnJbt0LHjnbfUqVsAb7lFihf3r7XubNdVULXAokUnHe6vkPViKoMnTuUgdMHMrbbWE2rEL+nYiwiFyU9HVJTITzcTnv4/HPYt8+O7u7ebe8W16MHDB4MR4/aC+CylSgBNWvaz4Md+Z0zxxbiChVOL8ABWlhSpMAZY/jnlf/kUOohRvw+giLBRXiry1tarUL8moqxiAD2FsgHD9rHvn32ER5uR2sBBgywKz9kbzt4EPr2hc8+s0X2wQftlIiQELjsMoiMtHeMA4iIsHeVi4yEGjXs17mFhsI11xTs9ysiefNyx5c5cuwIby96m4TUBCbcMEGrVYjf0n/ZIn4iIQEOHYLExJwH2Dm6AB98AKtX22XMEhLso0oV+Ogju71ZM1i58uRjtm+fU4z37rUFOCrKTnEoX97O9c22fDmULg0lS54+4msMdOvm+e9ZRArG29e9TYViFXjuf89xNP0oE3pMICIs4vwfFPExKsYiHpR9q+HU1JxHlSr2JhLZ82pzb0tNhf797faZM+36u6mpkJJinzMy4MMP7bFfecWOuiYn25HZo0ftiOzmzXb73XfDd9+dnKdaNTunF+xd3+bPt8U1+1G0aM6+gwfbMl26dE7xrVgxZ/v06ef+3mvWvKRfOhHxYoEBgQy/ajglQkvw1Oyn6J3Wmxm3zyDAaJ6T+BcVY/Eox7GPrKyc54AAu5ZsVpYtdNnvZ+9TtKh9ZGTYW/JmZJz8qFgRypSxn1258uRtmZl2ua7Kle2P93/++fTPd+kC1avbAvn11/Yzubffe68tdbGx9nbA2cfN3v7CC/YWwj/8YNfKPbXYzp1rpwe89ppdT/dUe/bY7+GDD+yxTtWrFxQvDrNmwRtv2LKb++E4OSOwISF2GkLx4na93lKlco7z8MNw4412W3i4feTePmPGuS9eu/POi/otF5FC5InWT1A0uCgPTn+Q27+93a57HBTmdiwRjyl0xXjKFJg8OWeN1OznCRNsOfv0U/j++9O3T55sC97YsfDjjznbHMeWlW++se+9+qotZ7m3lywJX35p33v2WViw4OTjV6pkzwvwxBOwZMnJn69VK2fUcMAAu6RV7s9HR8O4cfbrm26yo5K5y2m7djY3wJVX2qKWu7h26wZjxtjttWvbH7Xn3t6vH4waZbeXKGHLYu7tjz5qS+HRo7aUnerf/7a33N23z849PdWIEfD007Btmz3/qd59185f3bgx5za+uX3yic24fj306XP69u++s8V4wwZ45pmc9wMC7Ehtp062GO/caYtzYKBdBzf7kZSU82udkWG/x7Jlc4pr9m2HW7Wyv7+nFtvwcLv9zjvtrYiLFDl5e/ao7SuvwMiRZy+vTz9tH2dz7bVn3wZa0UFEPOP+ZveTkJrA0LlDWbd/HXP6z6Fs0bJuxxLxiEJXjHfvtiODkFMUjLGFB+yI5dq1p2/PLqLx8fZH4rm3heX6x3Jiot0n9/bAXKvbpKXlXImfvU/2uXPvn/vzIbmWjixa1I4YGpOzT+4LmSpVslkDAnL2qVo1Z3uTJnZ005icfaKicrZ37WqnAuTe3qpVzvYHHsg5bkCAfbRpY7eFhMDw4Tmfy96eXWbDw23xy34/e5/sz5cvbwt+7lIaFJQzj/WKK+x0g+z3swts9o/wY2Jg3bqTPxsYmDNq2rmznYaQ/f6pKx3cdJN9nE337vZxNm3bnrm4Z6tZ89zTDQK1CpKI+IDs1SouL3U5fb7pQ5136vBlzy/peHlHt6OJXDLjZDe+AhYTE+PEZjdUERER8TnL9y6nx6Qe7ErcxchrR/J4q8e1nJt4JWPMEsdxYs63n2bNi4iIyEWJrhjN6odX06lmJwbPGkzHTzqy8/BOt2OJXDQVYxEREbloJUJLMKPvDN69/l3m/zWfqz+6mvUH1rsdS+SiqBiLiIjIJTHG8GDMg8zsN5P4lHiajWvGuCXjcGu6psjFUjEWERERj7iq+lX8OeBP6patywM/PECbCW34eu3XpGemux1NJE9UjEVERMRj6pStQ+x9sYzvPp64I3H0mtyL+mPqM3nNZLKcLLfjiZyTirGIiIh4lDGGe5vey/bHtjO512RCA0O59etbaTG+BTM2zSAjK+P8BxFxgYqxiIiI5IvAgEB61u/J8geXM67bODYd3MT1n19P3XfqMnPzTM1BFq+jYiwiIiL5KiggiPua3ce+Ift4v/v7JKYl0uWzLlR9syoPT3+YLQe3uB1RBNANPkRERKSAHUw5yJR1U/hx849M3TCV9Kx0rqlxDe90fYe6Zeu6HU/8UF5v8KFiLCIiIq756/BffLbyM/47/7+kZqQysPlA/nP1fygRWsLtaOJHdOc7ERER8XpVI6ryTLtn2PLoFu5qfBdvLXyLqHejeC/2PZLSktyOJ4WMirGIiIi4rnyx8ozvMZ7f7/md8JBwHpz+IJGvRzJw+kDm7ZintZClQGgqhYiIiHgVx3H4M+5PRvw+gu83fA9AZIlIXrj6BTrU6EDViKouJxRfoznGIiIi4vP2H93PN+u+4dUFr7L10FYA6perz5VVruSOxnfQvFJzQoNCXU4p3k7FWERERPxGRlYGa/atYfbW2czZOofZW2eT5WQRFBBEnTJ1qBJRhaolqhJVIYrmlZoTXTFahVlOUDEWERERv7X/6H5+++s3lu5Zyqp9q9h1ZBfbE7YTnxIPQHBAMI0qNKJB+QY0u6wZrSJbEV0xmpDAEJeTixtUjEVERKRQcRyHuCNxLN69mEW7FhG7O5a1+9eyJ2kPACGBIdQuU5smFZvQrmo7GldsTN2ydbU0XCGgYiwiIiICxB2JY2HcQr5Y8wX7j+5nzf41HEg+cBId3FgAAArFSURBVGJ75fDK1CtXj441OtKsUjPql6vPZcUvwxjjYmrxJBVjERERkTNwHId1B9axMX4j6w+sZ+mepSzatYgdh3ec2KdUWClaVG5Bx8s70q5qO+qUrUNEaITKso/KazEOyuPBugBvAYHA+47jvHzK9lDgY6AZEA/0dhxn+4WGFhEREfn/7d1bjF1VHcfx72+m7QydyplehkJotS1WtKBWqNKEawQBMVqJNtQHqWAkgIg88IBpglVeRIO3lNigEi4xgmBUgpC2iIUEpVyk91KYQolteq8zpbfp7e/DXlP2nM6Zmc6cmTOn/X2SlVln7bX32uf8Z89es8/ae/U3SUxpmsKUpikdynfs3cHSzUtZs30Ni9cv5vl3n2fBugVHl9fV1nFaw2mMO3Uc4wvjGdswlgmNExjbMJZCfYFCXYFCfYHG+kZGnzKahmENA/3WrI+6vWIsqRZ4C/gCsAF4FfhGRKzO1bkV+FRE3CxpFnBtRFzX1XZ9xdjMzMwGs/yY5fUt69myewub92xmfct6Nu/ezJbdW2htay25fqGuwOTRkzn3tHMp1BUYPnR4h9QwtIHhQ4fT1NDEqFNG0VjfSKGuQMOwBmrkOdjKqZxXjD8HNEfEO2nDjwEzgNW5OjOAuSn/JDBPkqJS4zTMzMzM+kgS4wvjGV8YX7LOtj3b2LlvJ61trbTub6W1rZWW/S1s37ud91reY/X21Sxct5DdB3az58AeDsfhHrU9YtgICnUFxgwfw/Chw6kfUk/9kHrqhtQdzdfX1h9TXqgrcPqI02msb6S2phYhalRDjWqQPsjXqKbDsuLl/bVssA9F6UnH+Ezgv7nXG4ALStWJiEOSWoHRwHbMzMzMTlBNDU00NTT1uP7BwwfZe3Dv0bSrbRdb92zt0LF+v+19drXtoqWthZ37drLv4D72H9rP9r3baTvcxv5D+49JBw4f6Md3WR5CHPnhkUrvRpd6NMa4XCTdBNyUXu6WtHYg268yY/A/FtXE8ao+jln1ccyqi+NVffo1ZkGguRW7YvyRnlTqScd4I5D/DmFcKuuszgZJQ4AC2U14HUTEA8ADPdmxk52k13oyFsYGB8er+jhm1ccxqy6OV/VxzKAnI7tfBSZLmihpGDALeKqozlPA7JT/OvC8xxebmZmZWTXp9opxGjN8G7CA7HFtD0bEKkk/Bl6LiKeA3wOPSmoGdpJ1ns3MzMzMqkaPxhhHxDPAM0Vld+fy+4GZ5d21k56HnFQXx6v6OGbVxzGrLo5X9TnpY1axme/MzMzMzAYTPz3azMzMzAx3jMtO0kxJqyQdkTQtVz5B0j5JS1Oan1t2vqQVkpol/Vrp6deSRklaJOnt9HNkKleq1yxpuaTzctuaneq/LWl2d22c7ErFKy37Qfq81kq6Kld+dSprlnRXrnyipCWp/PF0syqS6tLr5rR8Qm/bsI4kzZW0MXdcXZNbVrH4WXn5eBh4ktanc8ZSSa+lsn4/J5Vqw44l6UFJWyWtzJVVLEZdtVFVIsKpjAn4BHA2sBiYliufAKwssc4rwHRAwLPAF1P5T4G7Uv4u4N6UvybVU1pvSSofBbyTfo5M+ZFdtXGypy7iNQVYBtQBE4F1ZDef1qb8JGBYqjMlrfMnYFbKzwduSflbgfkpPwt4vLdtOB0Tv7nAnZ2UVyx+lf5MTrTk46Fin/t6YExRWb+fk0q14dRpjC4BziPXt6hkjEq1UW3JV4zLLCLWRESPJy6RdAZwakS8HNlv1iPAV9PiGcDDKf9wUfkjkXkZaEzbuQpYFBE7I+J/wCLg6m7aOKl1Ea8ZwGMR0RYR7wLNZNOjH50iPSIOAI8BM9J/0p8nmxIdjo1XexyfBC5P9Y+rjfK+8xNeJeNn5eXjYfAYiHNSqTasSES8SPYksLxKxqhUG1XFHeOBNVHSG5JekHRxKjuTbJrtdhtSGcDYiNiU8puBsbl1iqfpPrOb8lJtWOeO9zMeDbRExKGi8g7bSsvbp0w/3jasc7elr+0ezH3tWsn4WXn5c66MABZKel3ZrLUwMOekUm1Yz1QyRifEsTqgU0KfKCQ9B5zeyaI5EfG3EqttAj4cETsknQ/8VdI5PW0zIkKSHyHSC72Mlw0SXcUP+A1wD9lJ/B7gPuDGgds7sxPWRRGxUdJpwCJJb+YXDsQ5yee9vnGMescd416IiCt6sU4b0Jbyr0taB3yMbDrtcbmq+Sm3t0g6IyI2pa8jtqbyUtN0bwQuKypf3E0bJ7zexIuup0LvrHwH2ddGQ9JVxXz9UlOmH28bJ6Wexk/Sb4Gn08tKx8/Kx59zBUTExvRzq6S/kA1pGYhzUqk2rGcqGaMT4lj1UIoBIqlJUm3KTwImA++kryN2SZqexi1eD7RfxcxPtT27qPz6dAfodKA1bWcBcKWkkekr5SuBBd20YZ17Cpil7IkEE8ni9QolpkhPY7D+STYlOhwbr86mTD+uNvr5/ValovFr1wLtd2dXMn5WXj4eBpikBkkfas+TnUtWMjDnpFJtWM9UMkal2qgufb17z+mYu0SvJRtX0wZsIfsFA/gasApYCvwH+HJunWlkf3TWAfP4YOKV0cA/gLeB54BRqVzA/an+Cjo+TeFGspuAmoEbumvjZE+l4pWWzUmf11pyT/Egu/P2rbRsTq58ElnHqBl4AqhL5fXpdXNaPqm3bTgdE79H0zGwnOyP8hmDIX5OZY+zj4eB/bwnkT39Y1k6b81J5f1+TirVhlOncfoj2TDNg+k89u1KxqirNqopeeY7MzMzMzM8lMLMzMzMDHDH2MzMzMwMcMfYzMzMzAxwx9jMzMzMDHDH2MzMzMwMcMfYzKxPJI2WtDSlzZI2pnyLpNX90N5lkp7uvmaHdRZLmtZJ+bckzSvf3pmZVTd3jM3M+iAidkTE1IiYCswHfpHyU4Ej3a2fZtMzM7NBwB1jM7P+Uyvpt5JWSVoo6RQ4egX3l5JeA76fZsb8s6RXU7ow1bs0dzX6jfbZyIARkp6U9KakP6SZqZB0eaq3QtKDkuqKd0jSDZLekvQKcGGufKaklZKWSXqx3z8ZM7NByB1jM7P+Mxm4PyLOAVrIZsBsNywipkXEfcCvyK40fzbV+V2qcyfw3XQF+mJgXyr/DHAHMIVslrILJdUDDwHXRcQngSHALfmdSVNo/4isQ3xRWr/d3cBVEfFp4CtleO9mZlXHHWMzs/7zbkQsTfnXgQm5ZY/n8lcA8yQtJZva+lRJI4CXgJ9Luh1ojIhDqf4rEbEhIo6QTTM/ATg7tfdWqvMwcEnR/lwALI6IbRFxoGgfXgIekvQdoLbX79jMrIp5bJuZWf9py+UPA6fkXu/J5WuA6RGxv2j9n0j6O3AN8JKkq0pst89/yyPiZkkXAF8CXpd0fkTs6Ot2zcyqia8Ym5lV3kLge+0vJE1NP8+KiBURcS/wKvDxLraxFpgg6aPp9TeBF4rqLAEuTU/SGArMzLV5VkQsiYi7gW3A+L6+KTOzauOOsZlZ5d0OTJO0PD3i7eZUfke6IW45cBB4ttQG0tXmG4AnJK0geyLG/KI6m4C5wL/Jhk6syS3+WbppbyXwL2BZWd6ZmVkVUURUeh/MzMzMzCrOV4zNzMzMzHDH2MzMzMwMcMfYzMzMzAxwx9jMzMzMDHDH2MzMzMwMcMfYzMzMzAxwx9jMzMzMDHDH2MzMzMwMgP8DAiO436XmfusAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
    "    plt.plot(thresholds, precisions[:-1], 'b--', label='Precision')\n",
    "    plt.plot(thresholds, recalls[:-1], 'g-', label='Recall')\n",
    "    plt.xlabel('Thresholds')\n",
    "    plt.legend(loc='center left')\n",
    "    plt.ylim([0, 1])\n",
    "\n",
    "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision의 오른쪽 상단의 경우를 살펴보면, 울통불퉁한 것을 볼 수 있다. 임계치가 올라가도 때에 따라서 precision이 낮아질 수 있다. 왜냐면 분모도 함께 줄어들기 때문이다. 하지만, recall은 계속 평탄하게 간다. 그 이유는 분모는 고정되어 있고 분자만 줄어들기 때문이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpl.rcParams['figure.figsize'] = (6, 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Precision')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAF3CAYAAABKeVdaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xl8lOW9/vHPNxsJIQmBhLAk7JvsYMClKm5VlgrW1u3U9thT29pT29p67E9bW621VdvanrZHRVvtYqt1aWtpAakLihtoENkXIWwBhLCFJYRs9++PGcaIQAaSZ+6ZyfV+vfJy5plnZq5HNBfPdt/mnENERAQgxXcAERGJHyoFERGJUCmIiEiESkFERCJUCiIiEqFSEBGRCJWCiIhEqBRERCRCpSAiIhEqBRERiUjzHeBEFRQUuN69e/uOISKSUBYsWLDDOVfY3HoJVwq9e/emrKzMdwwRkYRiZhuiWU+Hj0REJEKlICIiESoFERGJUCmIiEiESkFERCJUCiIiEqFSEBGRCJWCiIhEqBRERCQisFIws0fNbLuZLT3G62ZmvzKzNWa22MzGBJVFRESiE+Sewu+BCcd5fSIwIPzzJeDBALOIiEgUAhv7yDk318x6H2eVqcAfnXMOmGdmHc2sm3NuaxB5Nu6sZu2O/QCMKu5IfnZGEF8jIpLQfA6I1wPY1OR5RXjZR0rBzL5EaG+Cnj17ntSXzVq6lbtnrQTgsjE9+PkVo07qc0REkllCnGh2zj3snCt1zpUWFjY78utRXTq6B3//7zMBOFjb0JrxRESShs9S2AyUNHleHF4WiKLcTEb3zGdgUYcPLd9/qJ7yyv1RfYZzjord1cxZtZ2qg3VBxBQR8crn4aPpwA1m9hfgNKAqqPMJx/Lyqu1c+7u3AVj5wwlkpqd+6PXGRsfSLVXMXV3Jm+U7Wbp574fK4JyBhYztlc/4QYUMLMr5yPtFRBJNYKVgZk8A5wIFZlYB3A6kAzjnpgEzgUnAGqAa+HxQWY7muaVb+erjCyPPH3h5LX9fWMGDnzmV/OwMnnxrI0+VVfD+3hoAhnTLZdLwrgzpnsf3ng1dZTt3dSVzV1dy3/OrOat/AX+67rQTzuGcY9veQ+RlpZOVoVIREb8sdPFP4igtLXUtmXntol+8QlZGGmu376d/lw58rH9n7p+zNvJ6QYcM9h6sp66xkfEDC5k6qjtnDyikoEO7yDo1dQ2kmPHOxt2Urd/Fz/69GoA7LhlCSaf2XHBKERDa09hx4BCFHdrhHJTvOMCyLVWsrTzAuh0HKK/cz7odB6gOn+P461fO5NRe+ZH3pqTYSW+niEhTZrbAOVfa3HoJNx1na1i0aQ8ZqSn8+urRzF+3C4DrzurDb19bx479tVxZWsLXLuhPcX77o77/8GGi0/t25vS+nXl/bw1/mreRO/65HIDrx/djzfZ9vLK6krqGUOlmZ6RyIPzLP8WgOL89fQqyGdenE797fT0An3rwDcwgPTWF2vpGUgyuGteTs/sXsGl3Net2VPPOht2cf0oX3tu2n5JOWaSasWl3NRt2VvPNjw9k14Fatu45yOY9NSzZvIdfXT2a3p2zdWhLRKLS5vYUzv7JS2zadZArS0u499MjcM6x/1A9Wemp3PvcSiYN78bonvkn9JmH6ht4Yfl2nlv2Pv9ctAWAvgXZnNI9lxmLtzKyOI/RPfMZ2j2XYT3y6FuYTbu0D35J76upY8birdzytyWU9sonJcV4K1xWzclMT6GmrvFDy1IMGo/4Y504rCtTRnantqExtJdUlMOe6lo+OboH2/YeonvHTNpntMm/I4i0CdHuKbS5Uuh9ywwA/nzdaXysf0FrxQKgodGxYMNu+hRkU5jTrvk3NGPXgVoWVeyhc3YGvTpnk5eVTuW+Q3Rsn86e6jrMoHN2Bs7BnFXbyc1Kp3vHLIpy2pGWmsIP/7WcjbuqeX75tqi/s2en9owozmP9zgNcWVrClqoarh7bk56dj77XJCKJQaVwDIdLYdVdEz70t/Vktn1fDe9s2E1meirdO2bxflUNeVnp/OGN9fQv6sCGHdW8tmYHm/ccPO7nHN4rOat/AT+7fCQFHTJIS02IW11E2jyVwjEcLoX190xurUhJo76hkRQz6hsdq7fto0tOO554axNPvLUxchXWkdbdPQkznRAXiXcqhWN47M31FOa0Y8Kwbq0Xqg3ZtKuaaa+s5c/zN0aWfWpMMZt2V7NxZzXXj+/Lpt0HWbq5ivTUFC4d3YOVW/eSl5VOUV4mayv3M7K4I43O0dDoyMtK58x+BWSkaY9DJEgqBQlU1cE6Rv7g3wAU5bZj295DkdfSUkJ7Gy0xqCiHVdv28e9vnsPAopwWfZaIqBQkRhoaHakpRkOjY8XWvXTNy6Rzdga7DtSydMteenVqT019A5X7DlGUm8nK9/fhnCMzPZXdB2r53evr6ZLbjp37a2lodKzato++hdmUVx740PcM6ZbLuD6d+Op5/VvlJL5IW6NSkITmnOOXL77H/77w3lFfH1Gcx4ad1VQdrKNj+3QyUlPYeSBULFNHdSc3M53rz+1Hj45ZMU4uEp9UCpJUNu2q5k/zNvDQ3HIgdOnsxl3VZKSFbvRrzncmDY7cuzG4aw4pZgzuGjos1SlbV1FJ8lMpSJtUU9fAgUP1PDS3nL+9U8GO/bVRv/fnV4zkjH6d6ZanvQtJPioFkbB55TtJSzEO1Teyets+GhpdZMypvy88+mjtmekpDOqay3+e0YvLxhTHOLFI61MpiJyAeeU7eeKtjfzj3S1Hfb1dWgqPXjuWkSUd6dBOw4FI4lEpiLSAc47Zy97n+j+9c9TX75w6lCvHlrSZu+Il8akURFrRc0vfZ9X7+/jz/A1s33foQ6+d2iufL57dhz4FHSjplKWBBSUuqRREArJtbw2Pz9/IL188+uWynxpTzA8vHapykLiiUhCJgYO1DSzdUsXqbft45NV1lO/44Ka7KSO7879XjtJkSRIXVAoiHuzYf4hv/GUhr6/ZCYAZ3P3J4XxyTA+dfxCvVAoiHtXWNzLiB7M/NAHShKFduXhYEZ8crUtcJfZUCiJxYElFFbc9u4RFFVUfWn7pqO588+MD6dU521MyaWtUCiJxZtOuas7+yZyPLL+ytIRvXTSQotxMD6mkrVApiMSxx+Zt4HvPLv3QsouHFnFmvwLSUo1Te+UzuGuup3SSjFQKIgmgodHx5ccW8MKKo8+jPXl4N+7/zJgYp5JkpFIQSSB1DY2UVx7gUH0DK7fu4/bpyzhY1xB5/d5PDefKsT09JpREp1IQSXB7qmsZdefzkee9OrenU3YGFw/tysHaBr48vq9ukJOoqRREksRTZZu4Y/oyqmsbPvLa/1w0kK+c259U3SAnzVApiCShrVUHqalr5Ja/Lmb+ul2R5YO75vDItWM105wck0pBJMlt2XOQzz4yn7VN5rPulpfJ6//vfA2tIR8RbSloDkKRBNW9YxYv3nQu6++ZzM0XDwJga1UN4378AnUNzU9RKnI0KgWRJPDV8/oz9+bzANixv5YB353FFdPepKbuo+chRI5HpSCSJHp2bs+6uydR0il0XuGt9bsY/L3neGPNDs/JJJGoFESSiJnx6rfPZ+H3Pk5++3QA/uO383n1vUrPySRRqBREklB+dgYLv38R157ZG4DPPvIWvW+ZwYFD9X6DSdxTKYgksTumDGXaNadGng+9fTbTF20h0a46lNhRKYgkuQnDurL2x5Miz7/+xEL63DqTxkYVg3yUSkGkDUhNMdbfM5m/fuXMyLK+35nJc0vf95hK4pFKQaQNObVXPmt+NJH01NDNbdf/aQG3PbvEcyqJJyoFkTYmLTWF9340iWtOD426+qd5G+l9ywzmle/0nEzigUpBpI2669LhzP/OBZHnVz08j+/8XXsNbZ1KQaQNK8rNZP09k5k6qjsAj8/fyK1/W6KT0G2YSkFE+OVVo/ndtWMBeOKtjfT9zky++ud3qNcYSm2OSkFEADhvcBeW/uDiyPMZS7bS/7uzuOKhNz2mklhTKYhIRId2aay/ZzLLfnAx/bt0AOCtdbvYsf+Q52QSKyoFEfmI7HZpvPCt8dx16TAASu96gepaDZHRFqgUROSYrhpbEnk85PuzufT+1zlUr+G4k5lKQUSOKS01hfIfT2JYj1wA3t20h0G3PcevXnzPczIJikpBRI4rJcX419fOZtHtF0WW/fz51Vzy69c8ppKgqBREJCp5Wemsv2cyz914NgBLNlfxsXte8pxKWptKQUROyOCuuTz+xdMA2LznIL1vmeE5kbQmlYKInLAz+xWw4s4Jkee9b5lBxe5qj4mktagUROSkZGWkMu/WD8ZOOuveOezU/QwJT6UgIieta15o7KRxfToBcOpdL7Bpl/YYEplKQURa7Kkvn0FGaujXydk/mcNTb2/ynEhOlkpBRFrF6h9N5OKhRQB8+6+L+fYzizwnkpOhUhCRVvPQZ0uZfeM5ADxVVsHSzVWeE8mJUimISKsa1DWHBz8zBoBP/Po1aus1/HYiCbQUzGyCma0yszVmdstRXu9pZnPMbKGZLTazSUHmEZHYmDi8W+TxwNtmsX1fjcc0ciICKwUzSwXuByYCQ4CrzWzIEavdBjzlnBsNXAU8EFQeEYmtdXdPoii3HQDjfvQib6/f5TmRRCPIPYVxwBrnXLlzrhb4CzD1iHUckBt+nAdsCTCPiMSQmTH/Oxdy/fh+AFw+7U3Ov+9lnNNUn/EsyFLoATS9Lq0ivKypO4BrzKwCmAl8LcA8IuLBLRMHc9vkUwAorzxAn1tn0qA5oOOW7xPNVwO/d84VA5OAx8zsI5nM7EtmVmZmZZWVlTEPKSItc93ZfVlyxwejrF724Bse08jxBFkKm4GSJs+Lw8ua+gLwFIBz7k0gEyg48oOccw8750qdc6WFhYUBxRWRIOVkplP+49C1JIs27WHiL1/1nEiOJshSeBsYYGZ9zCyD0Ink6UessxG4AMDMTiFUCtoVEElSKSkWGS9pxda9PPTKWs+J5EiBlYJzrh64AZgNrCB0ldEyM7vTzKaEV7sJ+KKZLQKeAK51OgslktS65mXy4k3jAbh71kpq6jS9ZzyxRPsdXFpa6srKynzHEJEWunzaG7y9fjcA6++Z7DlN8jOzBc650ubW832iWUTaqKevPzPy+Lt/X+IxiTSlUhARb1bdFZqo58/zNzJ72fue0wioFETEo3Zpqfz+82MB+PJjC3jwZZ149k2lICJenTuoC9+8cCAA9z63kr63zqBRN7d5o1IQEe++ceEAHr/uNAAaXagcxA+VgojEhTP7F/DWd0P3MDw0t9xzmrZLpSAicaNLTibnD+4CwG9fVTH4oFIQkbjyq6tHA3DXjBWaoMcDlYKIxJUO7dJIsdDjgbfN0h3PMaZSEJG4s/quiZHHg7/3HM8v3+YxTduiUhCRuJOWmsL6eyYzuGsOAF/8Y5km54kRlYKIxK3nbjyHLjmhKT0/+cAbKoYYUCmISFyb++3zAHh30x763DqTe2bpHoYgqRREJK5lpqey4s4JXDy0CIBpr6xly56DnlMlL5WCiMS9rIxUHvpsKXdOHQrAmfe8pKuSAqJSEJGE8dnTe0Uen/+zl/0FSWIqBRFJGGbGmh+FLlfdUlXDok17PCdKPioFEUkoaakp/OLKkQBMvf91z2mSj0pBRBLOJ0cXRx6/uXanxyTJR6UgIgnpX187C4CrfzPPc5LkolIQkYQ0rEde5PHayv0ekyQXlYKIJKxp15wKwAX3veI5SfJQKYhIwpowrCu5mWkALNtS5TlNclApiEhC+94nhgAw+VevcfesFZ7TJD6VgogktMtLS7h14mAAHn1tnec0iU+lICIJ78vj+3Fqr3zqGhyvr9nhO05CUymISFK4/ZLQYaTP/Ha+5ySJTaUgIklhRHHHyONyXaJ60lQKIpI0fvrpEQDc9+/VnpMkLpWCiCSNySO6ATBjyVamL9riOU1iUimISNJon5HGbz9XCsDXn1joOU1iUimISFK5cEhR5LEm4jlxKgURSToThnYFYNIvX/WcJPGoFEQk6Tx4zRgAyncc0N7CCVIpiEjSMTMmDgvtLTz59ibPaRKLSkFEktIdU4YC8OLK7Z6TJBaVgogkpcIO7QCYu7oS55znNIlDpSAiSSklxRjbOx+A+et2eU6TOFQKIpK0Dg+rrfGQoqdSEJGkNbzJlJ0SHZWCiCQtM2P8wEIaGh27DtT6jpMQVAoiktQOX5p614zlnpMkBpWCiCS1y0tLAPjbO5t1I1sUVAoiktRSU4yLwuMh3T1Tczg3R6UgIknvgc+Ehr14skx3NzdHpSAiSS8tNfSrrqaukbqGRs9p4ptKQUTahM+d0QuAZxdu9pwkvqkURKRNuO6svgDc/Mxi7S0ch0pBRNqEnp3bR+ZZmKNB8o5JpSAibcZ/n9cPgFv+tsRzkvilUhCRNmNEcUf6FmSz60AtVQfrfMeJSyoFEWlTPn9WHwDOuPtFz0nik0pBRNqU/xjXE4Dq2gbKK/d7ThN/VAoi0qakphi/vno0ADc8vtBzmvijUhCRNueSkd3Jy0pn+da9mpXtCCoFEWmTpozsDmhWtiMFWgpmNsHMVpnZGjO75RjrXGFmy81smZk9HmQeEZHDrg6fW5j2ylrPSeJLWrQrmlkPoFfT9zjn5h5n/VTgfuDjQAXwtplNd84tb7LOAOBW4GPOud1m1uXEN0FE5MQN6Z4LwLzynZ6TxJeoSsHM7gWuBJYDhwckd8AxSwEYB6xxzpWHP+MvwNTwZxz2ReB+59xuAOecbjMUkZiqqWtk294ainIzfUeJC9EeProUGOScm+ScuyT8M6WZ9/QAmo5TWxFe1tRAYKCZvW5m88xsQpR5RERa7IrSYgCeWVDhOUn8iLYUyoH0AL4/DRgAnAtcDfzGzDoeuZKZfcnMysysrLKyMoAYItIW3fupEeRkpvHT2at8R4kb0Z5TqAbeNbMXgUOHFzrnvn6c92wGSpo8Lw4va6oCmO+cqwPWmdlqQiXxdtOVnHMPAw8DlJaW6voxEWkVZkZhTjv21dTz5tqdnNGvs+9I3kW7pzAd+CHwBrCgyc/xvA0MMLM+ZpYBXBX+nKaeJbSXgJkVEDqcVB5lJhGRFvv5FaMAuOaR+Z6TxIeo9hScc38I/2IfGF60Kvy3++O9p97MbgBmA6nAo865ZWZ2J1DmnJsefu0iMzt8Avtm55wuBRCRmBlVEjpi3dDocM5hZp4T+RXt1UfnAn8A1gMGlJjZfx7vklQA59xMYOYRy77f5LEDvhX+ERHxYvLwbsxYspWnF1RwRWlJ829IYtEeProPuMg5N945dw5wMfCL4GKJiMTOTReFDoIs37LXcxL/oi2FdOdc5PS8c241wVyNJCISc706ZwMa8gKiv/qozMx+C/wp/PwzQFkwkUREYis1xeiam8me6lrfUbyLdk/hK4TuRP56+Gd5eJmISFLoU5DN1qoaDtU3NL9yEouqFJxzh5xzP3fOXRb++YVz7lDz7xQRSQznDw4Nvfbq6h2ek/h13FIws6fC/1xiZouP/IlNRBGR4F1wSqgU/jhvg+ckfjV3TuEb4X9+IuggIiI+9S3swKCiHN7duNt3FK+Ou6fgnNsafrgD2OSc2wC0A0YCWwLOJiISUx0y09hbU09jY9sdTSfaE81zgczwnAr/Bj4L/D6oUCIiPnysfwEAW6oOek7iT7SlYM65auAy4AHn3OXA0OBiiYjEXr/C0P0KZ907p83O3Rx1KZjZGYTuT5gRXpYaTCQRET8Oz9sMcNNTizwm8SfaUriR0LSZfw8PatcXmBNcLBGR2DMz1v54EgDPL9/mOY0f0Y6S+grwSpPn5YRuYhMRSSqpKUb/Lh1Ys30/DY2O1JS2NWrqcUvBzP7XOXejmf2T0JzMHxLFlJwiIglnVElH1mzfz4qtexnWI893nJhqbk/hsfA/fxZ0EBGReDFhaFeeWVDBSyu3qxSacs4dnl2tDDjonGsEMLNUQvcriIgkndPD03JOe2UtX79ggOc0sRXtieYXgfZNnmcBL7R+HBER/zq0SyPFoLq27Q2OF20pZDrn9h9+En7c/jjri4gktEvCl6e+u2mP5ySxFW0pHDCzMYefmNmpQNu95U9Ekt6Xz+kHwKX3v+45SWxFO8nOjcDTZraF0BzNXYErA0slIuLZkO65kcd1DY2kp0b7d+jEFu18Cm8DgwlNrHM9cEqTk9AiIknphvP6AzBzydZm1kweUZWCmbUH/h/wDefcUqC3mWk4bRFJaleNKwHgN6+We04SO9HuD/0OqAXOCD/fDNwVSCIRkThRnB+6nmbp5r1U19Z7ThMb0ZZCP+fcT4A6gPCIqW3r3m8RaZP+62N9ABh2+2zPSWIj2lKoNbMswkNdmFk/QHM0i0jS+86kwQA0OmhoA5PvRFsKtwPPASVm9mdCN7N9O7BUIiJxIi01hS+cFdpbuO3ZpZ7TBK/ZUjAzA1YSmmDnWuAJoNQ593KgyURE4sTNFw8CYF9NneckwWv2PgXnnDOzmc654XwwwY6ISJuRmZ5KWorx3rb9za+c4KI9fPSOmY0NNImISBzLyUwjpQ3MrRDtHc2nAdeY2XrgAKErj5xzbkRQwURE4knvgmwWbkz+cZCiLYWLA00hIhLnenTMYuHGPTxdtonLS0t8xwnMcQ8fmVmmmd0I3AxMADY75zYc/olJQhGROHDjhQMBuPmZxZ6TBKu5cwp/AEqBJcBE4L7AE4mIxKH+XTowsqQjAM8u3Ow5TXCaK4UhzrlrnHMPAZ8Gzo5BJhGRuPT4dacBcOOT73pOEpzmSiFyUa5zrm0M/CEicgzZ7aI9DZu4miuFkWa2N/yzDxhx+LGZ7Y1FQBGReHLhKUUAvLNxt+ckwThuKTjnUp1zueGfHOdcWpPHucd7r4hIMrru7NCQF5c98IbnJMFoG1MJiYi0ktP7dgagtFe+5yTBUCmIiJygjw8pomzDbhqTcNRUlYKIyAkaVJQDwIOvrPWcpPWpFERETtDnzugFwE9nr/KcpPWpFERETlCX3MzI4+37ajwmaX0qBRGRk/Ctj4eGvXh5VaXnJK1LpSAichI+OboHANOS7LyCSkFE5CQU52cBkJ6SXL9Gk2trRERixMyYPLwbq7bt42Btg+84rUalICJykkaFR02dtXSr5yStR6UgInKSpo7qDsC3nlrkOUnrUSmIiJykppemJsvdzSoFEZEWuGxM6CqkZxZUeE7SOlQKIiItcMN5/QGYNjc5Lk1VKYiItEDfwg4AlFce8JykdagURERaKDsj1XeEVqNSEBFpoavG9QSgIQlONqsURERaKC8rHYDnl2/znKTlVAoiIi101dgSAJ5ZsMlzkpYLtBTMbIKZrTKzNWZ2y3HW+5SZOTMrDTKPiEgQuuRm0rF9Oq++t8N3lBYLrBTMLBW4H5gIDAGuNrMhR1kvB/gGMD+oLCIiQRtR3JFD9Y3U1CX2OEhB7imMA9Y458qdc7XAX4CpR1nvh8C9QHLNVCEibcrH+nUGoGz9bs9JWibIUugBND3AVhFeFmFmY4AS59yMAHOIiARueHEeAEs2V3lO0jLeTjSbWQrwc+CmKNb9kpmVmVlZZWVyzXIkIslhRHFoxNR7n1vpOUnLBFkKm4GSJs+Lw8sOywGGAS+b2XrgdGD60U42O+ceds6VOudKCwsLA4wsInJyOrRL8x2hVQRZCm8DA8ysj5llAFcB0w+/6Jyrcs4VOOd6O+d6A/OAKc65sgAziYgEpmen9gDMWbndc5KTF1gpOOfqgRuA2cAK4Cnn3DIzu9PMpgT1vSIivvzhv8YBcOOT73pOcvIC3d9xzs0EZh6x7PvHWPfcILOIiAStT0E2AFUH6zwnOXm6o1lEpBUdvrt5b01iFoNKQUSkFU0c3g2AB+Yk5vwKKgURkVZ0Wp9OAEx7RaUgItLmZaZ/MLfC+1WJN1CDSkFEpJU98JkxADy/IvGG0lYpiIi0snHhQ0gPJ+C8zSoFEZFWVtChHQCbdh30nOTEqRRERAIwIjxAXqJdmqpSEBEJQGmv0CGkx97c4DnJiVEpiIgE4Gvn9wfgvW37PCc5MSoFEZEA5GdnkN8+nWff3eI7yglRKYiIBOTwqKlvrE2cuZtVCiIiAfn6BQMAuOtfKzwniZ5KQUQkIBecUgTA8q17PSeJnkpBRCRA5w4KzRa5fW9iDHmhUhARCdDhvYVNu6s9J4mOSkFEJED9whPvvLAiMaboVCmIiARoVM+OADz62jrPSaKjUhARCVD7jNCsx4O75nhOEh2VgohIwM4eUMCiiiqcc76jNEulICISsPqGUBkcqm/0nKR5KgURkYCNDc+vULnvkOckzVMpiIgEbFj3XADWVO73nKR5KgURkYCd0i1UCrf/Y5nnJM1TKYiIBKwkPDDeofoGz0map1IQEYmBotx2bNurcwoiIgKU5If2Fmrj/AoklYKISAyc3rczAOt3HvCc5PhUCiIiMTCsRx4As5a87znJ8akURERi4Ix+oT2FfTV1npMcn0pBRCQG8rLS6dg+nZdWxfdoqSoFEZEYGdItF/MdohkqBRGRGMlvn8HaSp1oFhERYG/4fML+Q/WekxybSkFEJEbOHlAAwMOvrPWc5NhUCiIiMXLVuJ4A/OqlNZ6THJtKQUQkRnIz0+nQLjQTW7xOuKNSEBGJoUtGdgegYvdBz0mOTqUgIhJD4/rkA/Dc0vi8s1mlICISQ2cPKATgnY27PSc5OpWCiEgMFXRoB8As7SmIiAjA2N6hQ0g1dfE36Y5KQUQkxob36AhA5b74m3RHpSAiEmMjS0LDaO88UOs5yUepFEREYiwrPRWAxRV7PCf5KJWCiEiMjSoJHT76vzi8s1mlICISY11yM+lbkM0uHT4SERGAySO6Ud/o4m4mNpWCiIgHmeHzCi+tjK+Z2FQKIiIeXDy0KwDPLKjwnOTDVAoiIh70K8wGYMue+BoYT6UgIuKBmTGqpGPcTc+pUhAR8SS/fToAW6viZ29BpSAi4smEYaHzCi+vqvSc5AMqBRERT8YP7ALAyq17PSf5gEpBRMSTrnmZZGek8o9FW3xHiQi0FMxsgpmtMrM1ZnbLUV7/lpktN7PS35fCAAAJU0lEQVTFZvaimfUKMo+ISLypbWjkYG38DKEdWCmYWSpwPzARGAJcbWZDjlhtIVDqnBsBPAP8JKg8IiLxqLRXJw7VN9LY6HxHAYLdUxgHrHHOlTvnaoG/AFObruCcm+Ocqw4/nQcUB5hHRCTuDOqaA8DCTfExYmqQpdAD2NTkeUV42bF8AZgVYB4RkbgzZVR3AGYvi4/pOdN8BwAws2uAUmD8MV7/EvAlgJ49e8YwmYhIsIZ0ywWgbP0uz0lCgtxT2AyUNHleHF72IWZ2IfBdYIpz7qhz0znnHnbOlTrnSgsLCwMJKyLiw+GB8Va+v89zkpAgS+FtYICZ9TGzDOAqYHrTFcxsNPAQoUKIr6ECRURipEfHLKprG3DO/8nmwErBOVcP3ADMBlYATznnlpnZnWY2JbzaT4EOwNNm9q6ZTT/Gx4mIJK1zBoaOgPx7+TbPSQI+p+CcmwnMPGLZ95s8vjDI7xcRSQTXntmbJ97ayNNlmyJDavuiO5pFRDwb1DWHtBRj7uodvqOoFERE4kG/wg7UNjT6jqFSEBGJBwU5Gb4jACoFEZG4MLxHRwDvVyCpFERE4oAjVAaPvr7eaw6VgohIHLj2zN4APPbmep8xVAoiIvGgW14WI0s6sn5ntdcRU1UKIiJx4sx+nQF4bY2/S1NVCiIiceLCU0LTc7600t+oPyoFEZE4MbI4dAXSks1V3jKoFERE4kRaauhXskpBREQA6FeYTW29vzubVQoiInGkR357ANZW7vfy/SoFEZE48tnTewHwyGvrvHy/SkFEJI6MPzy3gqc5m1UKIiJxJCMt9Gt5x/5aLzexqRREROLMRUOKAHj09dgfQlIpiIjEmTunDgPgn4u3xvy7VQoiInGma14mAPUeJt1RKYiIxKHzBhWybMvemH+vSkFEJA7ltw/NxFZdWx/T71UpiIjEofGDQpemfv8fy2L6vSoFEZE4dMmI7gBsrToY0+9VKYiIxKGUFCMjLYXX1+yM7ffG9NtERCRqnxjeDYCNO6tj9p0qBRGROHXZmGIA/vHu5ph9p0pBRCROnda3EwBPL6iI2XeqFERE4lR6agr57dPZuKs6ZuMgqRREROLY184fAMDsGI2aqlIQEYljV4wtAeAns1fF5PtUCiIicaxDuzQuPKWIdTsOcM+slYF/n0pBRCTO3XfFSM4bVMjY3vmBf1da4N8gIiItkpeVzu8+Py4m36U9BRERiVApiIhIhEpBREQiVAoiIhKhUhARkQiVgoiIRKgUREQkQqUgIiIRKgUREYlQKYiISIRKQUREIlQKIiISoVIQEZEIcy42U7y1FjOrBDac5NsLgB2tGCcRaJvbBm1z29CSbe7lnCtsbqWEK4WWMLMy51yp7xyxpG1uG7TNbUMstlmHj0REJEKlICIiEW2tFB72HcADbXPboG1uGwLf5jZ1TkFERI6vre0piIjIcSRlKZjZBDNbZWZrzOyWo7zezsyeDL8+38x6xz5l64pim79lZsvNbLGZvWhmvXzkbE3NbXOT9T5lZs7MEv5KlWi22cyuCP9ZLzOzx2OdsbVF8d92TzObY2YLw/99T/KRs7WY2aNmtt3Mlh7jdTOzX4X/fSw2szGtGsA5l1Q/QCqwFugLZACLgCFHrPPfwLTw46uAJ33njsE2nwe0Dz/+SlvY5vB6OcBcYB5Q6jt3DP6cBwALgfzw8y6+c8dgmx8GvhJ+PARY7zt3C7f5HGAMsPQYr08CZgEGnA7Mb83vT8Y9hXHAGudcuXOuFvgLMPWIdaYCfwg/fga4wMwshhlbW7Pb7Jyb45yrDj+dBxTHOGNri+bPGeCHwL1ATSzDBSSabf4icL9zbjeAc257jDO2tmi22QG54cd5wJYY5mt1zrm5wK7jrDIV+KMLmQd0NLNurfX9yVgKPYBNTZ5XhJcddR3nXD1QBXSOSbpgRLPNTX2B0N80Elmz2xzerS5xzs2IZbAARfPnPBAYaGavm9k8M5sQs3TBiGab7wCuMbMKYCbwtdhE8+ZE/38/IWmt9UGSGMzsGqAUGO87S5DMLAX4OXCt5yixlkboENK5hPYG55rZcOfcHq+pgnU18Hvn3H1mdgbwmJkNc841+g6WiJJxT2EzUNLkeXF42VHXMbM0QrucO2OSLhjRbDNmdiHwXWCKc+5QjLIFpbltzgGGAS+b2XpCx16nJ/jJ5mj+nCuA6c65OufcOmA1oZJIVNFs8xeApwCcc28CmYTGCEpWUf3/frKSsRTeBgaYWR8zyyB0Inn6EetMB/4z/PjTwEsufAYnQTW7zWY2GniIUCEk+nFmaGabnXNVzrkC51xv51xvQudRpjjnyvzEbRXR/Lf9LKG9BMysgNDhpPJYhmxl0WzzRuACADM7hVApVMY0ZWxNBz4XvgrpdKDKObe1tT486Q4fOefqzewGYDahKxcedc4tM7M7gTLn3HTgEUK7mGsIndC5yl/ilotym38KdACeDp9T3+icm+ItdAtFuc1JJcptng1cZGbLgQbgZudcwu4FR7nNNwG/MbNvEjrpfG0i/yXPzJ4gVOwF4fMktwPpAM65aYTOm0wC1gDVwOdb9fsT+N+diIi0smQ8fCQiIidJpSAiIhEqBRERiVApiIhIhEpBREQiVAoiRzCzBjN718yWmtk/zaxjK3/+tWb2f+HHd5jZ/7Tm54u0hEpB5KMOOudGOeeGEbqP5au+A4nEikpB5PjepMlgY2Z2s5m9HR7H/gdNln8uvGyRmT0WXnZJeL6OhWb2gpkVecgvckKS7o5mkdZiZqmEhk94JPz8IkLjCI0jNJb9dDM7h9C4WbcBZzrndphZp/BHvAac7pxzZnYd8G1Cd9+KxC2VgshHZZnZu4T2EFYAz4eXXxT+WRh+3oFQSYwEnnbO7QBwzh0eC78YeDI81n0GsC428UVOng4fiXzUQefcKKAXoT2Cw+cUDLg7fL5hlHOuv3PukeN8zq+B/3PODQe+TGigNpG4plIQOYbwTHVfB24KD7E+G/gvM+sAYGY9zKwL8BJwuZl1Di8/fPgojw+GNP5PRBKADh+JHIdzbqGZLQauds49Fh6a+c3wSLP7gWvCo3b+CHjFzBoIHV66ltCMYE+b2W5CxdHHxzaInAiNkioiIhE6fCQiIhEqBRERiVApiIhIhEpBREQiVAoiIhKhUhARkQiVgoiIRKgUREQk4v8Dp/33SPu1qucAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Precision, recall graph를 그려보자.\n",
    "plt.plot(recalls[:-1], precisions[:-1])\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='green'>Recall이 0.8일때 precision이 급격하게 줄어들기 때문에, precision을 중요시하는 모델을 생성하기 위해서는, 0.6~0.7 정도의 recall을 달성하는 theshold를 선택하는 것이 좋다. 이렇게 threshold를 customizing하기 위해서는 `predict()` 메소드를 호출하지 말고, `y_train_pred = (y_scores > new threshold)`처럼 한다.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred_new_thre = (y_scores > 70000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8415374241402562"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_5, y_train_pred_new_thre)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6906474820143885"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_5, y_train_pred_new_thre)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-5 ROC 곡선"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Receiver Operating Characteristic(ROC)** 곡선은 binary classification에 널리 사용되는 도구\n",
    "- **FPR에 대한 TPR의 곡선**\n",
    " - FPR : False Positive Rate, 거짓 양성 비율, 양성(1)로 잘못 분류된 음성(0)의 비율, $\\frac{FP}{FP+TN}$, 1-specificity(TNR, 특이도). **즉 귀무가설이 참인데 귀무가설을 기각한 비율이다**.\n",
    " - TPR : True Positive Rate, Recall, Sensitivity, 진짜 양성 비율. **대립가설이 참이고, 귀무가설을 기각한 비율이다**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "fpr, tpr, _ = roc_curve(y_train_5, y_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAF3CAYAAAC7cgzXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xd8VFXCxvHfyaRXSgDpRUQFFxcEdGWlhK40BVRsK4ui7KqLbVkL6mJ5dVFZsQGissqqiIIigqiUYEMFUVBYEWnSBAIJ6cnMnPePDIgIIUAmd+7M8/18WDIlN8+OZJ455957rrHWIiIiciRRTgcQEZHQpqIQEZFyqShERKRcKgoRESmXikJERMqlohARkXIFrSiMMS8YY3YaY749wuPGGDPBGLPOGLPSGNM2WFlEROT4BXNEMRXoXc7jfYBTAn9GAM8GMYuIiBynoBWFtXYJsKecpwwAXrJllgLVjDF1g5VHRESOj5P7KOoDPx10e0vgPhERCSHRTgeoCGPMCMqmp0hKSjrrtNNOcziRiLiR31r8Fqy1+PyB236L11oMYCn7n2KvD0+UOXDbAkWlPqKjDAClPovX7yc6KgprLQUlPmI8UZT4/PitxWPKvtfvwBJJcdFRFHv9JMdF4ystZt+OzcQnp1K4b+9ua22t49mmk0WxFWh40O0Ggft+w1o7GZgM0K5dO7ts2bLgpxORoCjx+vluWw4lXj8+a/H7wWctu3OLATAG/LbsTXb/G/qe/GJyi73syi3GWojxGLx+yw8/55EQ4yEm2lBU6mf5pr0AxMdElW3Dbw+Uw7HwBP42B/2deNDjsYc8P7kC20yJj8YTZfAYQ1Z+CWkJMdSvlkC0x7Anv4SaSbHEeKI4rW4KUcYQZQzGgCfwd26Rl6bpSTSskXhgO7VT44iP8VArOY5qiTEYYw78vBUrVtCjRw/q1T2JhQsXcuqpp246tlfhF04WxWzgBmPMa8DZQI61druDeUQijt9vKfL6KPVaCkq95BZ52ZFTxIbd+UR7DH6/xee3+Cys25lLjaTYA2/APr/F67d8syWbRjUS8fosq7fvIzUhhliPOfBm/8sneMt32/ZVyf+volJ/uY+flBqPJ8qwNbuQdo2r83NuEQ2rJ1I9KZbowJvw5j0FtGlUjajAbU+UYXdeMafXTcUTZSj1+klLjCEtIQZPVBQGqJMaT2x0FDUSY/F4DNFRgT+eqp3l/+KLL+jVqxepqaksXLiQk08++YS2F7SiMMa8CnQB0o0xW4B7gRgAa+1EYC5wPrAOKACGBSuLSDjy+S2lPj+5RV7yir2UeP3szC0CIK/Iy6Y9BWzKyicu2kN2QQlzV+2gWa0k/rcjl+qJMeSX+Cjxlv+GWlErNmcf8/ckxXo4o35a2afjqLJP0D/uyuOsxtUP3I4ylP0dZcguKKFOajxRxtC4ZiIJMR6iPYaCEh8NqicS64ki2mOolhBDvWoJB7ZR9je/+rQdzoqKirjwwgupXr06ixYtonHjxie8zaAVhbV26FEet8Bfg/XzRUKJ1+dnT0EJJV4/Xp8lv8RLXpGX7TlFgamWwJx5YKokK7+ENdv3UVjiIy4miq82ZZOWEMP3P+dSKyWOXYFpmmP1vx25AOwtKP3V/Slx0UR7DHsLSmlZN5XNewpoVS+V5rWTD3rTNmzPKeSM+mmBN+CyN3Frweu3NKieQHSUocTnP/CJ3Rz8Zh+YQklPjuOktPgTfk3l8OLj45kxYwaNGjWiQYMGlbJNV+zMFnGatZbCUh879xWzLbuQDVn5RBlDXpGXDVn5bM4qYNOefOqkxLNs017qV0vA57fs2FdUaRn2b+vQkkiM9VBQ4uOU2snkFJaSFBdNoxqJWKBuajw+a2lVL5XEWA+p8TGcXDuZaokxxMd4iPVEER/jOcxPE7dZsGAB33//PX/5y18499xzK3XbKgqJCPnFXrLySsgtLsXvB6/fj89v2Z5TxLbsQhJif/1mmZVXwrbsQqI9Uby1YiuFpb4K/Zyf9hQCsDW78IjPaVQjkegow/rd+ZzdtAab9xRwdtMaB6ZYPMYQFQVx0R7Sk2Opm5ZAcnw01kL9agmkp8QSH+0hMc5DXLTe5AXee+89LrzwQlq0aMHw4cOJi4ur1O2rKMT1cgpK+W57Dht25+P3W7ILSpn9zTay8ktIiPGU+6Z9POJjokiMjaZ2Shztm9TAZy0xUYZaKXGcXCuZk9LiSYmPISnOgyfKkBDjISU+plIziOz3zjvvMHjwYFq1asUHH3xQ6SUBKgoJIcVeHzmFpRSXlu2U/ezHLGKjo/hu2z5+2lNAdmHZvHqsJ4r/7cglLSGGnMLSo2z1t1rVSyU6quzTu9dnKfb6iI2Ook3D6r963r6iUhrXSKRxzSTOqJ/GKbWTiYqKjB2i4g4zZ87kkksuoU2bNsyfP5/q1asf/ZuOg4pCgi63qJTNewr4YPXPREcZvtqcjddvWbJ2F83Sk1i/O/+4tntoSdRIiuWk1Hg6NK1BjaRYaqfE0bpBNVLio6mbFl/lhyiKBNuWLVvo0KEDc+fOJS0tLWg/x1gHzhw8ETrhLnQUe33szS8lr9jLt1tzWLU1h617C/n+51zioqPYll3IviLvMW+3blo823OKaFk3ldziUvq1rkdukZdmtZJoVCORumkJxEZHkRwXTXJ8NPHRUSoBiSi7d+8mPT0dgNLSUmJijj61aYxZbq1tdzw/TyMKOSKvz8+Pu/IpKvVR4vOzLbuQbdlFfLsthwVrfqbY6+dYPmfsPwt1UNsG5Jd4OaV2Cg2qJ1C/WgI1k2NJiPFEzLHuIsfr+eef55ZbbmHJkiWceeaZFSqJE6WiEApLfPywM5flm/byweqf2ZRVcEw7gBtUT+DnfUV0aFqD6omxNK+dTLNaybRrXJ0aSbHEeqI0ty9SCZ599ln+8pe/0Lt3b1q0aFFlP1dFEWGstazbmcekJev5avNe1u86+v6BlLhomtdJprCkbKdvYqyHs5vW5JL2DalXLaEKUovIE088wahRo+jXrx8zZswIytFNR6KiCEM+v+Xz9Vl8tG43m7MKyCksZf2uPLblHP3kr7MaV+eU2skMadeQhjUSqJ2iM2hFnDZnzhxGjRrFoEGDeOWVV4iNPXRZwuBSUbjcDz/nMmP5Fr7+KZsvNuyp8PIOddPiSYz18OCFv6Nd4+raGSwSwnr37s2ECRMYOXIk0dFV/7atonCJLXsLeP7jDWzPLmLphiyyC0qJ8RhKfb/em3xwScR6oji9XioDzqxHy3plK142qZlEenKsdhqLhDhrLRMmTODiiy+mbt263HjjjY5lUVGEqJzCUv75zncs/n4Xe/JLDvucUp/FGKiZFEuD6ol0alGLLqfWonGNRGokqQxE3Mpay+jRoxk3bhzZ2dnce++9juZRUYSYpeuzuHTy0iM+3qJOMlf+oQlNaibSok4K6clxeHREkUjYsNZy880388QTTzBy5EjGjBnjdCQVRSj49MfdjHhpOXnFvz05rVerOtze61Qa1UgiNlr7EUTCmd/v54YbbuDZZ5/lb3/7G+PHjw+JmQEVhUOKvT5GvfY1877dcdjH7+nbkmEdm4TEPxIRqRq5ubksWbKEv//97zz88MMh8/uvoqhiX/+Uzaufb2b6sp9+89gl7RrywIVnEKMjkEQiis/nw+fzkZaWxmeffUZycnLIlASoKKrM3vwS/vDwgt9cyzfjtNo8Mqg1tVKq7uQZEQkdpaWlXHXVVZSUlDBjxgxSUlKcjvQbKoog8vr8TFu6iacWrWN33q+PXLp/QCuGtGuoq4uJRLCSkhKGDh3KzJkzeeSRR4iKCs3ZBBVFEGzLLuTm6V/z+YY9v3nspozm3NLzVAdSiUgoKS4uZsiQIbzzzjuMHz+eUaNGOR3piFQUlWjR/3YybOqXv7m/emIM/xp8Jt1Prx1S844i4pw//elPvPPOOzzzzDOMHDnS6TjlUlFUgpzCUm6Z/jUL/rfzV/f/6Q+NubdfK62cKiK/MWrUKHr16sWwYcOcjnJUKooTUFTqY/wHa5m0ZP2B+9o0qsbDF7Xm1JNCb4eUiDgrNzeXd955h8suu4xzzjmHc845x+lIFaKiOE4bdufT9dHFv7pvTN+WDP9jU2cCiUhIy8nJoU+fPnzxxRe0a9euSq8ncaJUFMdh9Bsrf3UexIhOzfh7r1O1AquIHNbevXvp1asXK1asYPr06a4qCVBRHJN1O/Po88SSX63Y+tqIczinWU0HU4lIKNu9ezc9evRg9erVzJw5k379+jkd6ZipKCropz0FdH8888DtPzZPZ+qw9hpFiEi5Fi5cyPfff8/bb79N7969nY5zXFQUR+HzW/7z6UbGzll94L43R57LWY2rO5hKREKd3+8nKiqKiy++mD/+8Y/Uq1fP6UjHTR+Hy1FY4uP3/3z/VyXx3FXtVBIiUq4tW7bQtm1bFi9eDODqkgCNKMp10bOfkhtY+rtWShzPXN6W9k1qOJxKRELZxo0bycjIICsri7i48FjDTUVxGNZazn5oATsDlxW96/zTubZTM4dTiUio+/HHH8nIyGDfvn18+OGHtG/f3ulIlUJFcRhj3v72QEmc3bSGSkJEjmrr1q106tSJ4uJiFi5cSJs2bZyOVGlUFAcpKvVx3r8WsStQEh2a1mD6dX9wOJWIuMFJJ53EkCFDGD58OL/73e+cjlOpVBQBxV4ffZ/8+EBJdGxek/9e447T60XEOatWraJatWo0bNiQf//7307HCQoVRcBD765h3c48oOzIph4t6zicSERC3VdffUWPHj0488wzWbhwodNxgkZFAfzphS/IXLsLgGEdm6gkROSovvjiC3r16kVaWhpTpkxxOk5QRXRRFHt99Hh8CZv3FBy4b3Tv0xxMJCJu8Mknn9CnTx9q1arFwoULady4sdORgiqiT7i7YsrnB0rigtZ12fjwBbo0qYiUy1rLHXfcQd26dcnMzAz7koAIHlG8+sVmvty4F4Bxg1szpF1DhxOJiBsYY5g5cyalpaXUrVvX6ThVIiJHFFuzC7lj5ioAbujaXCUhIkc1d+5cBg8eTElJCenp6RFTEhCBRVFY4qPjw78cnXBLD3etCy8iVe/tt99m4MCBbNiwgfz8fKfjVLmIK4pnM3888PW7N/1R17MWkXK98cYbDB48mDZt2rBgwQKqV4+8RUEjqiiKSn3859ONAEwY2oZW9dKcDSQiIe3111/n0ksvpUOHDnzwwQdUq1bN6UiOiKiiuGPmKnIKS2mWnkS/1pEzvygix6d58+b069eP+fPnk5qa6nQcx0RMUeQXe5m1YisAw/7YFGM05SQih/fVV18B0LZtW2bNmkVycrLDiZwVMUVx3cvLAahfLYErzm7kcBoRCVVPP/00Z511Fq+99prTUUJGRBTF6m37+HjdbgDu699KowkROazx48dzww030L9/fy688EKn44SMsC8Kay3nT/gIgLSEGK3jJCKH9fDDD3PLLbcwaNAgZsyYETZXp6sMYV8Uzyz+5XDYV6/VsuEi8lsrV67kzjvvZOjQobz22mvExsY6HSmkhPUSHsVeH+Pmfw/AFec0omW9yD1qQUSOrHXr1ixcuJDzzjsPj0frvR0qrEcU+8+ZALinbyvngohIyLHWcueddzJ//nwAunTpopI4grAeUXy4eicAGafVJjY6rDtRRI6BtZZRo0YxYcIEioqK6NWrl9ORQlrYFsXmrAK+2LgHgLEDNJoQkTJ+v5+//OUvTJo0iZtvvpnHHnvM6UghL2w/Zj+zeB0AjWok0qB6osNpRCQU+Hw+rrnmGiZNmsQ//vEPHnvsMR0uXwFhWxRvfrUFKFtGXEQEyq4l4fF4uPfee3nooYdUEhUUllNP+cVeSn0WgP6/r+dwGhFxWmlpKbt27aJevXpMnjxZBXGMgjqiMMb0NsZ8b4xZZ4z5x2Eeb2SMWWSMWWGMWWmMOb8yfu67K7cf+FqXNhWJbCUlJVxyySV07NiRvLw8lcRxCFpRGGM8wNNAH6AlMNQY0/KQp90NvG6tbQNcCjxTGT978kfrAbip2ymVsTkRcamioiIuuugiZs2axc033xzxi/sdr2COKDoA66y16621JcBrwIBDnmOB/WfBpQHbTvSHFpX6WL8rD4DerU460c2JiEsVFBQwYMAA3n33XSZOnMhNN93kdCTXCuY+ivrATwfd3gKcfchz7gPeN8bcCCQB3U/0h36ybjd+C81rJ+tMbJEIdtddd/HBBx/wwgsvMGzYMKfjuJrTRz0NBaZaaxsA5wMvG2N+k8kYM8IYs8wYs2zXrl3lbvCtr8sGJeedkh6EuCLiFvfeey9vv/22SqISBLMotgIND7rdIHDfwYYDrwNYaz8D4oHfvMNbaydba9tZa9vVqlWr3B+6LHCSXb20hOMOLiLulJ2dzW233UZRURHVqlWjX79+TkcKC8Esii+BU4wxTY0xsZTtrJ59yHM2A90AjDGnU1YU5Q8ZjqKo1AdA19PKLxQRCS979uyhe/fuTJgwgS+//NLpOGElaEVhrfUCNwDzgTWUHd30nTFmrDGmf+BptwLXGmO+AV4FrrbW2uP9mV6fn31FXgDqV9PZ2CKRYvfu3WRkZLBq1SpmzpzJeeed53SksBLUE+6stXOBuYfcd89BX68GOlbWz/vfjlx8fku9tHgSYnX+hEgk+Pnnn+nWrRs//vgj77zzDj179nQ6UtgJqzOz1/6cC8AZ9dMcTiIiVSUrK4u8vDzeffddMjIynI4TlsKqKFZuyQGgRZ0Uh5OISLDt3buXatWq0bJlS9auXaur0gWR04fHVqovNpQd8dSohvZPiISzjRs3ctZZZ/HQQw8BqCSCLGyKwue3bM0uBKDFSRpRiISrdevW0alTJ7Kzs3XBoSoSNlNPq7ftI6ewlLpp8ZzZQPsoRMLR999/T0ZGBiUlJSxcuJDf//73TkeKCGFTFN8HdmSf1bi6VocUCUMFBQV069YNr9fLokWLOOOMM5yOFDHCpih+2lMAQL1qOiNbJBwlJiYyfvx4fve733Haaac5HSeihE1RLF5bdkJ3Q+3IFgkry5cvZ9u2bfTr148hQ4Y4HScihU1RFAeW7migEYVI2Fi6dCm9e/fmpJNOonfv3sTExDgdKSKFxVFPpT4//9tRto+ifdMaDqcRkcrw8ccf06NHD9LT03n//fdVEg4Ki6L45qfsA18nx4XNIEkkYi1evJhevXpRv359MjMzadSokdORIlpYFMX6XfkAOixWJEy8++67NGnShMWLF1O/fn2n40S8sCiKzYEjns5sWM3hJCJyIoqLiwH417/+xaeffspJJ+lyxqEgLIpiw+6yEcXvtBigiGu99dZbnH766axfvx5jDGlp+n0OFWFRFPuX7tAaTyLuNGPGDIYMGULt2rWpUUMHpISasCiKn/cVAVBXlz8VcZ3//ve/XHrppZxzzjm8//77VKumKeRQ4/qiKPb62J5ThDFQJy3O6TgicgzmzJnDlVdeSefOnZk3bx6pqalOR5LDcH1RbM8uG03USo4jLlpXtRNxk86dO3PrrbcyZ84ckpOTnY4jR+D6otiQVbYjOzdwrWwRCX1vvPEG+fn5pKSkMG7cOBITtX8xlLm+KFZv2wfAybWTHE4iIhXx2GOPMWTIEB577DGno0gFub4oVgUuf9qusY6UEAl1Dz30ELfddhtDhgzhjjvucDqOVJDri8JvLQANquuIJ5FQZa3lvvvu46677uLyyy/nlVde0dpNLuL6osguKAXgVF3+VCRk7d69m4kTJ3L11Vfzn//8h+horcnmJq7/r7V9X9nJdjqHQiT02MCIv1atWixbtox69eoRFeX6z6cRx9X/xUp9frYFDo/V1JNIaPH7/dx0003cfvvtWGtp0KCBSsKlXP1fbXdeMT6/JT05lvgYnUMhEir8fj8jR47kqaeecjqKVAJXF8W2wBpPfutwEBE5wOfzMXz4cCZPnsydd97JuHHjMMY4HUtOgKuLYlduCQDN0nUOhUiouOaaa5g6dSr//Oc/eeCBB1QSYcDVO7P3LwbYrJaKQiRU9O3bl9NOO43Ro0c7HUUqiauLYtmmvQDUq6Yd2SJOKi4u5vPPP6dTp04MGjTI6ThSyVw99bSvsOwcCu2jEHFOUVERF110Ed27d2fDhg1Ox5EgcPWIosTrB6CBRhQijigoKGDgwIF88MEHTJo0iaZNmzodSYLA1UWxX520eKcjiEScvLw8+vXrR2ZmJi+88ALDhg1zOpIEiauLYl9R2dRTtQStGSNS1aZNm8aSJUt4+eWXufzyy52OI0Hk2qKw1vJdYInxWim6sp1IVbvuuus4++yzadOmjdNRJMhcuzN7b2AxQIC6mnoSqRJ79uzhggsuYM2aNRhjVBIRwrVFsf+IJ0An9IhUgV27dtG1a1cWLFjApk2bnI4jVci1U09Z+WVnZTfVWdkiQbdjxw66devG+vXrmT17Nj179nQ6klQh1xZFsdcHQGGJz+EkIuFtx44ddOnShZ9++om5c+fStWtXpyNJFXPx1JMXgDPqpzqcRCS8paam0qJFC+bPn6+SiFCuHVHsyisGID1ZRzyJBMOmTZuoXr06qampzJ492+k44iDXjii27CkAdGisSDD88MMP/PGPf+SKK65wOoqEAPcWxd6ya1HogkUilWvNmjV07tyZoqIi7r//fqfjSAhw7dTT/rOy46Jd23UiIefbb7+lW7duGGNYvHgxrVq1cjqShADXFkV+cdnObO2jEKkc1lquvPJKoqOjWbhwIaeeeqrTkSREuLYo1mzPBbSPQqSyGGN4/fXXMcbQvHlzp+NICHHtvE3t1LKCqJEU63ASEXf77LPPGD16NNZaTjnlFJWE/IZri2JP4MxsrfMkcvyWLFlCz549mTlzJnv37nU6joQoVxaF32/JLSrbR5ESryXGRY7HwoUL6dOnDw0aNCAzM5MaNWo4HUlClCuLYn9JAHiitCCgyLGaP38+F1xwAc2aNWPx4sXUq1fP6UgSwlxZFPsPjY3VobEix6W0tJTWrVuzaNEi6tSp43QcCXGufKfNDlyLonmtZIeTiLjL1q1bAejbty+fffYZ6enpDicSN3BlUWzZW7Z8R0q8a4/uFaly06dP5+STT+a9994DICrKlb/+4gBX/kvx+i0AuwMLA4pI+aZNm8Zll11Ghw4d6Nixo9NxxGVcWRT7z8pu26i6w0lEQt8LL7zAVVddRZcuXZg3bx4pKSlORxKXcWVR7N+ZnZqgQ2NFyrN8+XKGDx9Ojx49mDNnDklJuiKkHDtXFsX3O/IASIrTPgqR8rRt25aXXnqJt99+m4SEBKfjiEsFtSiMMb2NMd8bY9YZY/5xhOdcbIxZbYz5zhjzSkW2GxdTFrug2HuUZ4pEpqeeeoqVK1dijOHKK68kPl4rGMjxC1pRGGM8wNNAH6AlMNQY0/KQ55wC3AF0tNa2AkZVZNtFpWXXyT6ljg6PFTnUgw8+yI033sizzz7rdBQJE8EcUXQA1llr11trS4DXgAGHPOda4Glr7V4Aa+3Oimw4L3Bmdpr2UYgcYK3l3nvv5e677+aKK67gySefdDqShIlgFkV94KeDbm8J3HewFkALY8wnxpilxpjeh9uQMWaEMWaZMWbZrl27KAyMKBJitY9CBMpK4o477mDs2LEMGzaMqVOnEh2t3w+pHE7vzI4GTgG6AEOB54wx1Q59krV2srW2nbW2Xa1atfh6czYA8VrCQwQAr9fLihUruP7665kyZQoejy4RLJUnmB85tgIND7rdIHDfwbYAn1trS4ENxpi1lBXHl+VtuGGNRFZv30eUFgSUCOf3+8nPzyclJYXZs2cTGxuLMfq9kMoVzI/kXwKnGGOaGmNigUuB2Yc85y3KRhMYY9Ipm4paf7QNl/r8AKRqiXGJYH6/n+uuu46MjAwKCwuJi4tTSUhQBK0orLVe4AZgPrAGeN1a+50xZqwxpn/gafOBLGPMamARcLu1Nuto2y72lhVFnKaeJEL5fD7+/Oc/M2XKFHr16qXDXyWogrq3y1o7F5h7yH33HPS1BW4J/KmwYm/ZzmwtMy6RyOv1ctVVV/Hqq68yduxYxowZ43QkCXOuPCyisKSsKBJjtcNOIs9tt93Gq6++ysMPP8zo0aOdjiMRwJVF8cvUk4pCIs+oUaNo2bIlI0aMcDqKRAhXzt3sLwpNPUmkKCwsZMKECfj9fpo0aaKSkCrlundaa3/5WtfLlkhQUFBA//79GTVqFJ988onTcSQCuW7qyXdwU4iEuby8PPr27ctHH33E1KlTOe+885yOJBHIdUXhD1zdrl6aDgeU8LZv3z7OP/98li5dyrRp0xg6dKjTkSRCua8oAgOKbTlFzgYRCbJVq1bxzTffMH36dAYNGuR0HIlgrisKS1lTnHaSLuco4am0tJSYmBg6duzIxo0bqVmzptORJMK5bmd2oCeI8bgvusjR7Ny5k/bt2zN16lQAlYSEBNeOKGI8OuJJwsv27dvp1q0bGzdupGHDhkf/BpEq4r6iCIwoojWikDCydetWMjIy2Lp1K/PmzaNz585ORxI5wHVFsX9ndnyMzsqW8JCbm0unTp3YtWsX8+fPp2PHjk5HEvkV1xWFDQwpdNEiCRcpKSlcf/31dO7cmQ4dOjgdR+Q3XFcU+0cUcRpRiMv98MMP5Obm0rZtW26//Xan44gckeuKosTnJwrtzBZ3W7NmDRkZGaSlpfHdd9/p0qUS0lw3f7N/faetewsdTiJyfFatWnVgZ/XMmTNVEhLyXFcU+/dRnF431eEkIsduxYoVdO3aldjYWDIzM2nZsqXTkUSOyoVFUfa3lhgXNxo/fjxJSUlkZmbSokULp+OIVIjr9lGU+vzEArE6j0JcxFqLMYbnnnuOXbt20aBBA6cjiVTYMb/bGmOijDGXByNMBX8+AHsKSpyKIHJMlixZwnnnnUdWVhZxcXEqCXGdIxaFMSbVGHOHMeYpY0xPU+ZGYD1wcdVFPLz61RKcjiByVAsWLKB3795kZWVRUqIPN+JO5U09vQzsBT4DrgHuBAww0Fr7dRVkOyx/YCdFSrzrZs0kwrz33ntceOGFNG/enA8//JA6deo4HUnkuJT3btvMWvssS0faAAAabUlEQVQ7AGPMFGA70Mha6+iFIPZfuChBJ9xJCHv//fcZMGAALVu25IMPPiA9Pd3pSCLHrbx9FKX7v7DW+oAtTpcE/HJmdnKcRhQSulq2bMnAgQNZuHChSkJcr7yiONMYs88Yk2uMyQVaH3R7X1UFPNT+8yjiYnTUk4SeTz75BJ/PR4MGDZg+fTrVq1d3OpLICTviu6211mOtTbXWpgT+RB9027Gz3QpKfQDERWvqSULLSy+9RKdOnXjsscecjiJSqY44f2OMiQeuB5oDK4EXrLXeqgp2JHGBE+28++egRELA888/z7XXXkvXrl3561//6nQckUpV3vzNf4B2wCrgfCAkPibtPzM7VUc9SYh45plnuOaaa+jVqxdz5swhKSnJ6Ugilaq8d9uWBx319DzwRdVEKt/+cYSumS2hYPv27dx+++3069ePGTNmEBcX53QkkUpXXlEcfNSTd/8Z0Y4LNMX+VWRFnFS3bl0+/vhjWrVqRWxsrNNxRIKivKL4/UFHNxkgIXDbANapHdo20BS6HoU46YEHHiA9PZ3rr7+eNm3aOB1HJKjKm7/5JnCUU2ooHfVU7PUDEB2lqSepetZaxowZw5gxY1i6dOmBw7VFwll5I4qQ/A2ICRREVKhMhUnEsNYyevRoxo0bx/Dhw5k0aRIhMyUrEkTlFUVtY8wtR3rQWvt4EPIc1f6pp/hYjSik6lhrueWWW/j3v//NyJEjeeqpp4jSqFYiRHlF4QGSKdsnETL2j/Rj9EsqVcgYQ/369Rk1ahSPP/64RhISUcoriu3W2rFVlqSCDhweqyvcSRXw+Xz8+OOPtGjRgttuu+3ABYhEIkl577Yh+duwf5nxaB0eK0Hm8/kYNmwY7du3Z+vWrQAqCYlI5RVFtypLcRx0KVQJptLSUq644gpefvllbr/9durXr+90JBHHHHHqyVq7pyqDHKsojSgkSEpKSrjssst48803eeSRR/j73//udCQRR7lywSSNJiSYnnzySd58803Gjx/PqFGjnI4j4jhXFoWW75Bguummmzj99NM5//zznY4iEhJc+dFcO7KlsuXn53Pdddexc+dOYmJiVBIiB3FlUXi0zpNUotzcXPr06cOUKVP49NNPnY4jEnJcOfWkEYVUlpycHPr06cMXX3zBK6+8wsCBA52OJBJyXFkUe/JLnI4gYWDv3r306tWLFStW8Prrr3PRRRc5HUkkJLmyKEJytUJxnZKSEkpKSpg5cyb9+vVzOo5IyHJlUTSonuB0BHGxrKwsUlNTqVOnDsuXL8fj8TgdSSSkuXNntpZRkOO0fft2zjvvPEaMGAGgkhCpAFcWha5FIcdjy5YtdO7cmc2bN3P11Vc7HUfENVw59aTlO+RYbdy4kYyMDLKysnj//fc599xznY4k4hquLApNPcmx8Pv99OvXj7179/Lhhx/Svn17pyOJuIori0IjCjkWUVFRTJw4kcTERNq0aeN0HBHXcek+CqcTiBusXr2aSZMmAdCxY0eVhMhxcuWIQosCytGsXLmS7t27Ex0dzSWXXEK1atWcjiTiWq4cUeQVe52OICHsq6++omvXrsTGxrJ48WKVhMgJcmdRFKko5PA+//xzMjIySElJYcmSJbRo0cLpSCKuF9SiMMb0NsZ8b4xZZ4z5RznPG2SMscaYdhXZbtP0pMoLKWHl66+/Jj09nczMTJo1a+Z0HJGwELSiMMZ4gKeBPkBLYKgxpuVhnpcC/A34vKLb1gl3cqj8/HwArrvuOlauXEnjxo0dTiQSPoI5ougArLPWrrfWlgCvAQMO87z7gUeAoopuOMqVE2YSLB9++CFNmzZl6dKlACQmJjqcSCS8BPMttz7w00G3twTuO8AY0xZoaK19t7wNGWNGGGOWGWOWgUYU8ou5c+fSt29fTjrpJE01iQSJY5/NjTFRwOPArUd7rrV2srW2nbW2HagopMzbb7/NwIEDadWqFYsWLaJ27dpORxIJS8Esiq1Aw4NuNwjct18KcAaw2BizETgHmF2RHdo6jUKWLl3K4MGDadOmDQsWLKBmzZpORxIJW8Esii+BU4wxTY0xscClwOz9D1prc6y16dbaJtbaJsBSoL+1dtnRNqwRhbRr14577rmHDz74QOdJiARZ0IrCWusFbgDmA2uA16213xljxhpj+p/Ito2KImLNmDGDbdu2ER0dzZgxY0hNTXU6kkjYC+o+CmvtXGttC2vtydbaBwP33WOtnX2Y53apyGgCNPUUqZ577jkuueQSxo4d63QUkYjiygNNtdZT5Hn66acZMWIEvXv35t///rfTcUQiiiuLQvsoIsv48eO54YYb6N+/P7NmzSI+Pt7pSCIRxZVFUeLzOx1BqkhRUREvvvgigwYNYsaMGcTFxTkdSSTiuHKZ8V25xU5HkCrg9XqJj49n0aJFpKWlER3tyn+uIq7nyhFFs1paFDCcWWu5++67GTx4MKWlpdSsWVMlIeIgVxaFrpkdvqy1/P3vf+fBBx+kdu3aeDwepyOJRDxXFkW0R0URjqy1jBo1ikcffZS//vWvTJw4kSitACniOFf+Fuqop/A0evRoJkyYwM0338yTTz6pkhAJEa6c+NV5FOFpyJAhJCUlcc899+jse5EQ4sqi0IgifHi9XubOnUv//v1p37497du3dzqSiBzClWP7aI0owkJpaSmXX345AwYMOHDRIREJPa4cUfy0t8DpCHKCSkpKuPTSS5k1axbjxo3jnHPOcTqSiByBK4uiSbrOo3CzoqIiBg8ezLvvvssTTzzBTTfd5HQkESmHK4tC51G426JFi5g3bx4TJ07kuuuuczqOiByFK4tCPeFO1lqMMfTp04fVq1dz6qmnOh1JRCrAlTuzDWoKt8nNzaV3794sXLgQQCUh4iKuLAod9OQu2dnZ9OzZkwULFrB7926n44jIMXLl1JPmntxjz5499OzZk5UrVzJjxgwuvPBCpyOJyDFyZVGoJtwhJyeHjIwM1qxZw8yZM+nbt6/TkUTkOLhy6kkDCndISUnhD3/4A7Nnz1ZJiLiYK0cUWsIjtG3btg2v10ujRo149tlnnY4jIifIlUWhmghdP/30ExkZGSQnJ7N8+XKtACsSBtxZFGqKkLRhwwYyMjLYs2cPL730kkpCJEy4tCjUFKFm3bp1ZGRkkJeXx4IFC2jXrp3TkUSkkri0KJxOIIe65ZZbKCwsZOHChfz+9793Oo6IVCJ3FoX2UoScqVOnsmPHDlq2bOl0FBGpZK6cRNaIIjR88803XHnllRQXF1OjRg2VhEiYcumIQpy2fPlyevToQVJSEj///DONGjVyOpKIBIkrRxQ6j8JZS5cupVu3bqSlpbFkyRKVhEiYc2VRqCec88knn9CjRw/S09PJzMykadOmTkcSkSBzZVGIc5KTk2ndujWZmZkaSYhECFcWhc6jqHrr1q0D4Mwzz+Tjjz+mfv36DicSkariyqLQ9Siq1ty5cznjjDOYNGkSoKIWiTSuLAqvzzodIWK8/fbbDBw4kFatWjF48GCn44iIA1xZFPklXqcjRIQZM2YwePBg2rZty4IFC6hZs6bTkUTEAa4sitop8U5HCHubN2/m8ssv55xzzuH999+nWrVqTkcSEYe48oQ77aMIvkaNGjFr1iw6d+5McnKy03FExEGuHFFEqSmC5rnnnmPu3LkAXHDBBSoJEXFnUXh01E1QPPXUU4wYMYIXXnjB6SgiEkJcWRS6Hk7le/zxx7nxxhsZOHAgr7zyitNxRCSEuPItV2s9Va7/+7//49Zbb2XIkCG8/vrrxMbGOh1JREKIK4vCo30UlcZay6ZNm7j88st55ZVXiImJcTqSiIQYlx71pKI4UdZasrKySE9P55lnnsFai8fjcTqWiIQgV44oVBQnxlrLbbfdRtu2bdm5cydRUVEqCRE5IlcWRV6xzsw+Xn6/n5tuuonHH3+cgQMHUqtWLacjiUiIc2VRVEvQPPrx8Pv9jBw5kqeeeopbb72VJ554Qgv8ichRubIodHjs8Rk3bhyTJ0/mzjvvZNy4cSoJEakQV+7MNrpq9nG5/vrrqVGjBtdcc41KQkQqzJ2fzfUeV2GlpaU89NBDFBQUkJaWxrXXXquSEJFj4sqi0NtcxRQXF3PxxRdz1113HVi/SUTkWLlz6kmfiI+qqKiIQYMGMXfuXJ588klddEhEjps7i8LpACGuoKCAgQMH8uGHHzJp0iRGjBjhdCQRcTF3FoWaolzbtm1j1apVvPDCC1x99dVOxxERl3NnUWhMcViFhYXEx8fTvHlz1q5dS0pKitORRCQMuHNntnriN7Kzs+natSv33HMPgEpCRCqNO4vC6QAhZs+ePXTv3p2vvvqKdu3aOR1HRMJMUIvCGNPbGPO9MWadMeYfh3n8FmPMamPMSmPMAmNM44ptuNKjutauXbvo2rUr3377LW+99RYDBgxwOpKIhJmgFYUxxgM8DfQBWgJDjTEtD3naCqCdtbY18AbwrwptW00BgNfrpUePHqxdu5Z33nmH888/3+lIIhKGgrkzuwOwzlq7HsAY8xowAFi9/wnW2kUHPX8pcEVFNqx9FGWio6O58847qVWrFl27dnU6joiEqWAWRX3gp4NubwHOLuf5w4F5FdlwpPfE5s2b+fbbbzn//PO5+OKLnY4jImEuJA6PNcZcAbQDOh/h8RHACIDYk5pH9JnZGzZsICMjg/z8fNavX09ycrLTkUQkzAVzZ/ZWoOFBtxsE7vsVY0x34C6gv7W2+HAbstZOtta2s9a2K/ueIKR1gR9++IFOnTqRk5PDvHnzVBIiUiWCWRRfAqcYY5oaY2KBS4HZBz/BGNMGmERZSeys6IYjsSfWrFlD586dKSoqYtGiRZx11llORxKRCBG0qSdrrdcYcwMwH/AAL1hrvzPGjAWWWWtnA+OAZGBGYDpps7W2/9G2HYkjiunTp+P3+1m8eDGtWrVyOo6IRBBjrXU6wzGJq3uK/XTpF5zVuLrTUaqE3+8nKioKay07duygbt26TkcSERcyxizfP31/rNx5ZnaEjCiWLVtG69atWbt2LcYYlYSIOMKdReF0gCrw2Wef0a1bN/Lz84mNjXU6johEMHcWRZgPKT766CN69uxJ7dq1WbJkCU2aNHE6kohEMHcWhdMBguiLL76gd+/eNGjQgMzMTBo2bHj0bxIRCSJ3FkUYN0XLli0ZOnQoixcvpl69ek7HERFxaVGE4ZgiMzOT3NxckpOTmTJlCnXq1HE6kogI4NaiCLOemDVrFj169ODOO+90OoqIyG+4sii8fned+1Ge6dOnM2TIENq1a8cDDzzgdBwRkd9wZVHERbsy9m9MmzaNyy67jHPPPZf58+eTlpbmdCQRkd8Ij3dcF8rPz+cf//gHXbp0Yd68ebrGtYiErJBYZvxYhcM+iqSkJDIzM6lXrx4JCQlOxxEROSJXjijcfNTThAkTuPnmm7HWcvLJJ6skRCTkubIo3OrRRx/lb3/7G5s2bcLn8zkdR0SkQlxZFG6cenrwwQe5/fbbufjii5k+fTrR0a6c9RORCOTKonCb+++/n7vvvpsrrriC//73v8TExDgdSUSkwlxZFG4bULRu3ZoRI0YwdepUjSRExHVceeGib7/+ilPqhPbhpNZaVqxYQdu2bZ2OIiKiCxeFGr/fz4033kiHDh34+uuvnY4jInJCNA9Syfx+P9dddx1Tpkzh9ttv58wzz3Q6kojICXHliCJU91L4fD7+/Oc/M2XKFO666y4eeeSRsL/IkoiEP5cWRWh68803+c9//sPYsWN54IEHVBIiEhZcOfUUqu+/Q4YMoVatWnTt2tXpKCIilUYjihNUXFzMn//8Z7777juMMSoJEQk7riyKUBlQFBYWcuGFF/Liiy+ydOlSp+OIiASFK6eeQkFBQQEDBgxgwYIFTJ48meHDhzsdSUQkKFxZFE7vJM7Ly6Nv37589NFHvPjii/zpT39yNI+ISDC5siic5vF4iIuLY9q0aQwdOtTpOCIiQeXKonBqPJGdnY0xhrS0NN577z3HRzYiIlXBnTuzHXh/zsrKIiMjgwEDBmCtVUmISMRw5Yiiqu3cuZPu3buzdu1aZs2apZIQkYjiyqKoykuhbt++nW7durFx40bmzJlD9+7dq+xni4iEAlcWRVW66qqr2Lx5M/PmzaNz585OxxERqXKuLIqqnPmZOHEiP//8M+eee27V/VARkRDiyp3ZwbZ+/XruuecerLWcfPLJKgkRiWgqikOsXbuWTp068fTTT7N582an44iIOE5FcZA1a9bQuXNnSkpKWLRoEY0bN3Y6koiI41xZFMHYR7Fq1aoDO6sXL15M69atK/+HiIi4kCuLIhi2bt1KSkoKmZmZtGzZ0uk4IiIhw6VHPVXekGLv3r1Ur16d3r17s2bNGmJjYytt2yIi4SCiRxSffvopzZo148033wRQSYiIHIYri6IyxhNLliyhZ8+e1KpVi7PPPrsStigiEp5cWRQnasGCBfTu3ZtGjRqRmZlJgwYNnI4kIhKyXFkUJ7KLYv369fTt25fmzZuzePFi6tatW3nBRETCkCuL4kQ0a9aMxx9/nIULF1K7dm2n44iIhDxXFsXxrB771ltvsWLFCgBGjhxJenp6ZccSEQlL7iyKY+yJ1157jcGDB3PfffcFJY+ISDhzZVEci5deeonLL7+cjh07Mm3aNKfjiIi4jiuLoqIDiueff56rr76aLl26MHfuXFJSUoKaS0QkHLmyKCrCWsvMmTPp1asXc+bMISkpyelIIiKu5MolPI42pCguLiYuLo433niDqKgo4uLiqiaXiEgYCrsRxb/+9S/OPfdccnJySEhIUEmIiJwgVxbFkQ6Pvf/++xk9ejQtWrQgMTGxilOJiIQnVxbFoay1jBkzhnvuuYcrr7ySadOmERMT43QsEZGw4MqiOPQ8ikcffZQHHniA4cOH8+KLL+LxeJwJJiIShty5M/sQl1xyCXl5edx7771ERbmy+0REQpYr31UN4Pf7efnll/H7/TRq1Ih//vOfKgkRkSAI6jurMaa3MeZ7Y8w6Y8w/DvN4nDFmeuDxz40xTSqyXZ/Px4gRI7jqqquYNWtWZccWEZGDBK0ojDEe4GmgD9ASGGqMOfRi1MOBvdba5sB44JGjbthabhw5gueff54xY8Zw0UUXVXJyERE5WDBHFB2Addba9dbaEuA1YMAhzxkA/Cfw9RtAN3OUC2J7c37m9Vf/y/3338/YsWMr9frZIiLyW8EsivrATwfd3hK477DPsdZ6gRygZnkb9Rflc9/9D3L33XdXYlQRETkSVxz1ZIwZAYwI3Cy+b8xd39435i4nI4WKdGC30yFChF6LX+i1+IVei1+cerzfGMyi2Ao0POh2g8B9h3vOFmNMNJAGZB26IWvtZGAygDFmmbW2XVASu4xei1/otfiFXotf6LX4hTFm2fF+bzCnnr4ETjHGNDXGxAKXArMPec5s4E+BrwcDC621NoiZRETkGAVtRGGt9RpjbgDmAx7gBWvtd8aYscAya+1s4HngZWPMOmAPZWUiIiIhJKj7KKy1c4G5h9x3z0FfFwFDjnGzkyshWrjQa/ELvRa/0GvxC70Wvzju18JopkdERMqjNS9ERKRcIVsUwVr+w40q8FrcYoxZbYxZaYxZYIxp7ETOqnC01+Kg5w0yxlhjTNge8VKR18IYc3Hg38Z3xphXqjpjVanA70gjY8wiY8yKwO/J+U7kDDZjzAvGmJ3GmG+P8LgxxkwIvE4rjTFtK7Rha23I/aFs5/ePQDMgFvgGaHnIc/4CTAx8fSkw3encDr4WXYHEwNcjI/m1CDwvBVgCLAXaOZ3bwX8XpwArgOqB27Wdzu3gazEZGBn4uiWw0encQXotOgFtgW+P8Pj5wDzK1lY9B/i8ItsN1RFFUJb/cKmjvhbW2kXW2oLAzaWUnbMSjiry7wLgfsrWDSuqynBVrCKvxbXA09bavQDW2p1VnLGqVOS1sEBq4Os0YFsV5qsy1tollB1BeiQDgJdsmaVANWNM3aNtN1SLIijLf7hURV6Lgw2n7BNDODrqaxEYSje01r5blcEcUJF/Fy2AFsaYT4wxS40xvassXdWqyGtxH3CFMWYLZUdi3lg10ULOsb6fAC5ZwkMqxhhzBdAO6Ox0FicYY6KAx4GrHY4SKqIpm37qQtkoc4kx5nfW2mxHUzljKDDVWvuYMeYPlJ2/dYa11u90MDcI1RHFsSz/QXnLf4SBirwWGGO6A3cB/a21xVWUraod7bVIAc4AFhtjNlI2Bzs7THdoV+TfxRZgtrW21Fq7AVhLWXGEm4q8FsOB1wGstZ8B8ZStAxVpKvR+cqhQLQot//GLo74Wxpg2wCTKSiJc56HhKK+FtTbHWpturW1irW1C2f6a/tba417jJoRV5HfkLcpGExhj0imbilpflSGrSEVei81ANwBjzOmUFcWuKk0ZGmYDVwWOfjoHyLHWbj/aN4Xk1JPV8h8HVPC1GAckAzMC+/M3W2v7OxY6SCr4WkSECr4W84GexpjVgA+43VobdqPuCr4WtwLPGWNupmzH9tXh+MHSGPMqZR8O0gP7Y+4FYgCstRMp2z9zPrAOKACGVWi7YfhaiYhIJQrVqScREQkRKgoRESmXikJERMqlohARkXKpKEREpFwqCpEKMsb4jDFfH/SniTGmizEmJ3B7jTHm3sBzD77/f8aYR53OL3K8QvI8CpEQVWit/f3BdwSWt//IWtvXGJMEfG2MeSfw8P77E4AVxphZ1tpPqjayyInTiEKkklhr84HlQPND7i8EvqYCi6+JhCIVhUjFJRw07TTr0AeNMTUpW1/qu0Pur07ZGktLqiamSOXS1JNIxf1m6ingPGPMCsAPPBxYPqJL4P5vKCuJf1trd1RhVpFKo6IQOXEfWWv7Hul+Y0xTYKkx5nVr7ddVHU7kRGnqSSTIAkt8PwyMdjqLyPFQUYhUjYlAp8BRUiKuotVjRUSkXBpRiIhIuVQUIiJSLhWFiIiUS0UhIiLlUlGIiEi5VBQiIlIuFYWIiJRLRSEiIuX6f89pPT9m0ZCSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_roc_curve(fpr, tpr, label=None):\n",
    "    plt.plot(fpr, tpr, linewidth=2, label=label)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.xlabel('FPR')\n",
    "    plt.ylabel('TPR')\n",
    "\n",
    "plot_roc_curve(fpr, tpr, 'SGD')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall이 증가하면, 분류기가 양성으로 분류하는 경향이 커지기 때문에, FPR도 증가한다. 즉, Recall이 높을수록 분류기가 만드는 FPR이 증가한다. 좋은 분류기는 왼쪽 오른쪽 모서리에 가까워지며, 대각선의 점선과는 멀어져야한다.\n",
    "\n",
    "점선은 랜덤 분류기의 ROC 곡선을 의미한다.\n",
    "\n",
    "곡선 아래의 면적(AUC - area under curve)를 측정하면 분류기들을 비교할 수 있다. 완벽한 분류기는 ROC의 AUC가 1이고, 완전 랜덤 분류기는 0.5가 된다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9584692857783504"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "roc_auc_score(y_train_5, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "양성 클래스 sample이 드물거나, 거짓 음성보다 거짓 양성이 더 중요할 때는 Precision-Recall graph를 사용하고, 그렇지 않을 때는 ROC 곡선을 사용한다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3-4. 다른 모델과의 성능 비교\n",
    "- RandomforestClassifier vs SGDClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kakao/.pyenv/versions/3.6.3/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(random_state=42)\n",
    "# RandomForestClassifier에서는 decision_function을 사용하지 않고, predict_proba를 사용합니다.\n",
    "y_prob_rf = cross_val_predict(rf, x_train_2d, y_train_5, cv=3, method='predict_proba')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 확률값으로 나오기 때문에, 양성 클래스에 대한 확률을 점수로 사용함\n",
    "y_scores_rf = y_prob_rf[:, 1]\n",
    "fpr_rf, tpr_rf, _ = roc_curve(y_train_5, y_scores_rf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAF3CAYAAAC7cgzXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzs3Xd4FNXbxvHvSUISCEkQAtJ7k95Fkd4C0qQICKiIIKAgoIiCWLC8KiKKoHT5IVJEQRGDiIGAiKA0AUEwFOm9Q0LKnvePSSBiSELYzdnZfT7XFc3szs7emYR59pwzc0ZprRFCCCFuxcd0ACGEEO5NCoUQQog0SaEQQgiRJikUQggh0iSFQgghRJqkUAghhEiTywqFUmqmUuqkUmrHLZ5XSqkJSqlopdQ2pVQNV2URQgiRea5sUcwCwtN4vhVQJumrH/CpC7MIIYTIJJcVCq31GuBsGqu0B2Zry3ogl1KqgKvyCCGEyByTYxSFgEMplg8nPSaEEMKN+JkOkBFKqX5Y3VMEBQXVLF++vOFEIiWd9B+dtPTv5eTvk57VKV5zfTmV16BvsW7S2mlt5+b30SmfsZ5LNdtNGW79PjeeSzkBzq2e0yneTP9r+b/5/pMhxcq3ek4m4XFzGpQCpRRocGjw8QEfZX2fmADZsimUgsREazkgwHpNQoIiIR6y5wAFXLsGDgfkyAEKRUyM9fcUFGQ9HxMLfr7g768AuHr5GjFn/yEwZwgxF8+d1lrnzcyPYLJQHAGKpFgunPTYf2itpwJTAWrVqqU3btzo+nQukujQnLp0jaMXYjh2Ppaj52M4cyWORIfD+qNxaBza+kp0WAcD6zGSHrvxvMMBiVpfXydRp1zfet6hNYla40jaxr9er8HhSHo+5fqpvV/y61O8n8OmRyiV9OWJlAJfpfDxUfgkf59y2SdpWSl8fayD043H/v28j4/1epW0rvV9inVSvkeK7Sulkt4XfHxUOhnARym0w/o+m5/127l8SZEju7p+QDx7WpEr1FqOuar4a5eifDlFntxw8qTi9w2KRo0gT25F5E+KzZsUw59XZA+EtT8rlv+gePstRYC/Yv58iFqpmDnTes9BzyiOHlV8s1iRzQ8G9ldcuaKYP8/K26G9olBBxfTp1s9SsyY0baz48ENrOV8+GDxI8cYYxcULUKqk4v2xiqeeUhw6BA0bwqxZ0KAB/PUXvPIKjB4NlSvD9u0wezYMHQoFC8KuXdZjHTuCnx/s3m19tW1r/W7Pn7cKRe7c6f8tbNmyhebNm3NXgfysXLmScuXK/ZPpvytXTgqolCoOLNVaV0rluQeBZ4DWwL3ABK11nfS26c6FQmvN+avxN4rAhRiOJhWDY0nfn7gYS4Jdj7C34Jt0AEg++PjcdJBQSQeElAed5Od9kg8q/znQcP3g4pt00Ep5gPvP+6U4aF1/v1QOWjdnTPXAlmL7PjflTe2gl/x6Xx9ubOs/eW9aP5Wf4V+vTz5Qpzxw3/T6/xy4kz+1OsGlS3D0KJQoAf7+cPYs/PMPVK1qfRreu9c6qIWHWwe09ethxw548knr9V9/Dfv3w/PPW8tffAGHDsGLL0J8POTKBY89Bp98Yj2fMyf07w/vvw9Xr1qfkN95B0aMsA6OhQvDa6/BgAGwcyfUqQNLl8KDD0JEhPX/DRusx7/+Gp54Ag4fhuBg672nToXlyyEw0FqOjISZM633XrQItm2ztg/www9WxrZtreV166zX1Ug6L3PPHmu7BZJGVK9csfZRtmxO2fVO8dtvv9GyZUtCQkJYuXIlpUqVQim1SWtdKzPbc1mhUErNAxoBYcAJ4FUgG4DWerKy/qInYp0ZdRXorbVOtwKYLBRX4xI4ej426aAfk+L72OvFISY+Md3t5Anyp0CuQAqGZqdgruzkDQ7Az0f9+yCZyqeu1D8F3uLAm8ZBL/n1/zlwp3JgTnnQutVBT7gPndTNAfD339anz3LlrOXhw6FxY2jd2joYd+sGvXvDQw/BwYNQvLh10A0Ph/nzoXt3qxiULw//+x+MHw/DhsGjj8LYsfDCC1ZByZnT+v7jjyEmxnqvJ56AqCjYt+/G8qZN8Mcf1vJLL0HZstb7A4wbB9WqQdOmkJAAc+ZAzZrWp+64OOtgnjs3NG9udb8cPmwdqIOCrJ/R4QBf3xs/uzeLjY2lVKlSBAQEsGrVKooVKwbgnoXCVVxVKOITHRy/EMuxC1YLIGXX0NELVkE4fzU+3e3kDPCjYK5ACoRmp2BSMSiQKzsFQwMpmCs7+UMDCczm6/T8wp7i462vHDmsA+ShQ5AnD4SEwMmT1gHzkUcgf37rQDtqFHz5pfX89OnW16+/WgfI3r2tT8P/JHUwtGtnfdr9+Wdr24UKQenS8Msv1gH9/vutLo9HH7Xea+BAePhh6+uff6z1HnwQQkOtQrJlC1SqBKVKwbFjcOQIVK9uHaAvXrS2effdZvensKxbt46iRYtSuHDh64/dSaGwxWC2M1yNS+DvE5dvHPiTikFyq+DkpWukVzP9fX0okCuQAkkHfasI3GgZFMgVSEigG7U/hUsk/50oBSdOWJ+6GzWyHps1yzo4P/20tTxwoNVN8e671nLlytCli9VPHRNjFYj33rM+7R87BiVLwrRpVhfOzp3w3HNWi+DBB62unDVrrC6e+++3ukPuuuvGp+nWra3t+fhY3UH9+1vPg7W8ebPVDw6QPbt14E+WLx989dWN5WLFrK9kRYtaX8kKFLjR9QJW4QoJudM9K+5EZGQku3fvZuDAgdx///1O3bZXtCjW/n2awfO3cPZK3C3X8VGQLzjQag0ktQAKJBWA5BZCniB/fHykbetpjh2zzjZJ/vC1Z4/VpVGokNXt8fTTMHiwdZBfuND6xH3kiHXQfeklqy/99GmrJdC+PZw6ZfVrg1UUsme3umVCQ60+9vr1rVaCwwFvv211ubRpYxWYr76C++6zumViYqziULq01QcuxK388MMPPPTQQ5QtW5bffvuNgICA/6wjXU+3oLXm09V7eX/5bhwaSoYFUTJvTqtLKFf2Gy2DXNnJFxxANl+Z+squtL7xyfrUKavL5umnrQPss8/Cn3/CihVWK6BTJ6ur5eefrdfmyQP33mv1z4P1SbpJE/jsM6tQBATAm29a3T6bNlmthv79oWJFqzWxYwd06OBeg5nCe3z33Xd07tyZihUrsmLFCvLkyZPqetL1lIpLsfE8v/APlv95AoDBTUrzbLOy+EqLwDbi4m58kp492+oeCU+aFKZtW+jZE7p2tfrSH3jAOtulY0f47Tdr0LVmTeuUxBIlYMGCG109rVvD5cs33mf6dKvIJPvkE+u9wDr4Hz5stS7A2mbNmjfWvece60sIExYtWkTXrl2pXr06y5cv567kvkYn88gWxd8nLvHUnE3sO3WF4EA/xj9cjWYVZJTNtBMnrP7zvHmt7p6hQ+HDD62B2sWLoW9fOH7c6k9/8kmre+ell6yDfeXK1uPJ/erly1vnp0+ZYhWUDh2slkPLllaLYu9eq0snMNDszyyEK02YMIEFCxYQERFBaGhomuveSYvC4/pavt92jPaTfmHfqSuUzx/Md888IEUii+zdaw22glUIOna0umzAGgPInx9mzLix7oIF8M031rJS1ifz6GhruWNH65N78if5776Dn3668V5//WUVCbBaHRERVpEAqxDVrStFQniu06dPAzB48GCioqLSLRJ3ymMKRUKig7cjdvH03M1cjUukfbWCLBp4P8XDgkxH8xhaW6dJJv2N8tVXUKaM1QoAePll6zx4sM6IeeABOHfOWvb1hVdfvdFtU6eONU7Qv7+13KGDNWaQPDtL69ZWkSlVylouXtwaSxDC282YMYNSpUrxR9JFKdmyYHDMIwrF6cvX6DljA1PX7MPPR/Fq2wp82LUaOfw9dgjGZXbssM7iAesgX6IEDBpkLWttDfR+/LG1XLiw1SXkm3RZyKhRVhdSsmHDrAupkr32mnXBFFitgLyZmnVGCO/16aef8uSTT3L//fdTtmzZLHtf2xeKfacu02bCWtbvO0ve4ADm9atL73ol5IrhdCQkWP//9VeoUMEaAAaYONEaF/jjD+sc/FatoHZt6zkfH6u7J/magbp1rStmkw/4lSpZp3kKIZzvo48+YuDAgbRt25ZvvvmG7NmzZ9l72/4j96dRezl+MZaaxe7i0x41yBciHdOQNENq0iyVq1fDypXWJ/q4OOtTva+v1a9fsKD1WGLSzCMjRljTLVStai0nz8WTrF+/LP0xhBDA0qVLGTJkCJ06dWLu3Ln4Z/GFNbZuUWitWb3nFABvdqjktUXC4bAmSEu+LqBnT6tAJA/+rlkDY8ZYxSAgwLrKt2VL63XFilkDyPfdZ61booQ1fiCEcB/h4eFMmDCB+fPnZ3mRAJsXir+OX+LkpWvkCw6gfP5g03Fc7ujRG+f7T5tmnRmktTWzZ79+MHKkNW/QrFnW1A/J1wKMGmW9zi+p/ThihDWLZ5CM8wvhtrTWfPTRRxw7dgw/Pz8GDRqEn5+ZTiBbF4rk1kTDsnk9ckxi0ybrWgKwxhAKFbKKQXKX0tKlVrdRWJh1quns2dYFYn5+1nTN1apZr/XxkVk1hbATrTUjRoxgyJAhTJ061XQcmxeK3UmFopznnD6jNcTGWi2DgQOt009PnrROKx00yDptVCno0+fG9BJgzR9UooTZ7EKIO6e1ZujQoYwdO5YBAwYwevRo05HsWyguX0tg4z9n8VHwQOkw03HuSHzS7OXJVy5PmWK1CsaPt+YnypfPGnyeMMG6SlkI4ZkcDgdPP/00H330Ec8++yyTJk3Cx8f8Ydq2Zz39uvcM8YmaGkVzkSuHfafW3LXLOj317FlrLv9u3azCoZQ1lbQQwntcunSJNWvW8MILL/DOO++4TZe6bQvF6j0nAWhYNp/hJLfvr7+sqSzGjrUuPCtVyroTWM2aMG+e6XRCiKyWmJhIYmIioaGh/Prrr+TMmdNtigTYtOtJa02UDccnrl61/h8VZQ02Hz5sFYno6H/PSCqE8B7x8fH07NmT7t2743A4CA4OdqsiATYtFPtPX+HwuRjuypGNyoVcOxmWM2htFYL69a1i0batdcvLFHcpFEJ4obi4OLp168b8+fO599573WI8IjXumSodyafF1i+T123vL3HunFUcJk+2rmF48UVr2gxfX+s0VykSQni3a9eu0blzZxYtWsT48eN54YUXTEe6JVsWivX7zgBQv4x7ne0UE2ONOxw7Zt0/OEcO2LbNGpzu0sWaPymVOxQKIbzQY489xnfffccnn3zCkCFDTMdJky0Hs4+ejwWgdL6chpP827x58Omn1rUPydNmCyFEaoYMGULLli3p3bu36SjpsmWL4uQlq1C4w9xOJ09Cu3Zw4YI1md6yZdC+velUQgh3dOnSJebOnQtA3bp1bVEkwKaF4vTlOADCcpq7fiJ5dtbs2a27ryXforNcOZkuQwjxXxcuXKBly5Y8+uij7Nmzx3Sc22K7QpHg0CQ6NLlyZCPAz9dIhgsXrGm4o6MhONg6gyn5Hg1CCHGzc+fO0bx5c37//XcWLFiQpTcdcgb7FYpEa/rUfMFZPyqc3IoICoIDB+DPP63H5QwmIcStnD59miZNmvDHH3+waNEiOnXqZDrSbbNdoYhP1ADkC8768YmePa3i4OdntSI6dMjyCEIIm1m5ciW7d+/m22+/pW3btqbjZIrtCkWCw0yLYudOWLcO5syxlkPd/zo/IYRBjqRj1cMPP0x0dDTh4eGGE2We7QpFcosib0jWFIo33rCujShdGn75Bd58M0veVghhY4cPH6ZGjRpERUUBULBgQbOB7pDtCkXyGEXenK4vFFrDmTM3bilasOCNu8QJIURqDhw4QIMGDdi/fz8BHnKFre0OewkOjT+uv4Zi717rRkDPPw+5c1uzvAohRFr27t1LkyZNuHjxIj/99BO1a9c2HckpbNeiiM+Cs54uXYKGDa0iUbiwNRWHEEKk5ciRIzRo0IArV66wcuVKjykSYMNCkXD9rCfXFAqtrWsjOne2rrgWQoiMyJ8/P126dGHVqlVUr17ddBynsmXXE0BeFxSKt9+2isSAAfDhh07fvBDCA23fvp1cuXJRpEgRPvTQA4ftWhQObRWKnAHOr3EJCdb9qpPOahNCiDRt3ryZRo0a8dhjj5mO4lK2KxQAAX4+Tr0D1Blr1nJeegl+/VUGroUQ6fvtt99o2rQpwcHBTJ8+3XQcl7JlofD3c17ss2ehaVN48EHIls3qehJCiLT88ssvNGvWjNy5c7N69WpKlixpOpJL2W6MAnDqZIC5c1s3FapTx2mbFEJ4MK01L730EgUKFCAyMpLCXjDZm00LxZ23KI4fh6lToU8fGDXKCaGEEF5BKcWiRYuIj4+nQIECpuNkCa/tepo/37qf9enTTggkhPB4ERERdO7cmbi4OMLCwrymSIBNC4UzWhRDhsBff1n3lRBCiLR8++23dOjQgf3793PlyhXTcbKcLQvFnbQoNm+GF1+Ew4chJMSJoYQQHumrr76ic+fOVK9encjISO666y7TkbKcLQvFnbQo1q6FWbPAQ+bqEkK40Jdffkm3bt2oU6cOK1asIFeuXKYjGWHLQnEnLYrBg+HoUcib14mBhBAeqXTp0rRt25bly5cT4sVdELYsFJk9PXbdOoiJAR9b/tRCiKyyefNmAGrUqMHixYvJmTOn4URm2fKQ6e97+7G3bYN69WDECBcEEkJ4jEmTJlGzZk3mz59vOorbsOV1FJnpeqpSBX7+GSpUcEEgIYRHGD9+PMOGDaNdu3Y89NBDpuO4DVu2KG53MHvLFuseEw88YF2JLYQQN3vnnXcYNmwYnTp1YuHChR5zdzpnsGWhuJ0WxfnzUL8+dOjgwkBCCFvbtm0bI0eOpHv37syfPx9/mRn0X2zZ9XQ7g9mhofDLLzLZnxDi1qpUqcLKlSupX78+vr7Om0vOU3h0i+LiRfjzT+vqaw+f3FEIcZu01owcOZLly5cD0KhRIykSt+DRhWLGDHj2WYiNdXEgIYStaK0ZMmQI//d//3e9UIhbs2nXU8YKRfnycOwYBAa6OJAQwjYcDgcDBw5kypQpDB06lHHjxpmO5PZsWSiy+Wbs7natWllfQggBkJiYSN++ffnss8948cUXefvtt516t0xPZcuuJ58M/GL/9z/49tssCCOEsA2lFL6+vrz66qtSJG6DLVsUGfnlTp1q3WuiffssCCSEcGvx8fGcOnWKggULMnXqVCkQt8mlLQqlVLhSardSKlop9WIqzxdVSq1SSm1RSm1TSrXOyHZ9MvA7XrkSVqy4/cxCCM8SFxdH165dqVevHpcvX5YikQkuKxRKKV9gEtAKqAB0V0rdPIHGy8CXWuvqQDfgk4xs2zedSnHwoDWNeNGitx1bCOFBYmNj6dixI4sXL2bo0KFeP7lfZrmyRVEHiNZa79NaxwHzgZs7gjSQPHdvKHA0IxtO7xPBo49atzoVQnivq1ev0r59e77//nsmT57M4MGDTUeyLVeOURQCDqVYPgzce9M6rwE/KqUGAUFAs4xsOL2up3Ll4NChtNcRQni2UaNGsWLFCmbOnEnv3r1Nx7E104PZ3YFZWutxSqn7gM+VUpW01o6UKyml+gH9APzzl8Y3nRbFlCmuiiuEsItXX32VJk2a0LZtW9NRbM+VXU9HgCIplgsnPZZSH+BLAK31r0AgEHbzhrTWU7XWtbTWtSDt02OPHYOEhDsLLoSwp/Pnz/P8888TGxtLrly5pEg4iSsLxe9AGaVUCaWUP9Zg9ZKb1jkINAVQSt2DVShOpbfhtBoUTz9tTQQohPAuZ8+epVmzZkyYMIHff//ddByP4rKuJ611glLqGWA54AvM1Fr/qZQaA2zUWi8BngOmKaWGYg1sP6611ultO60WxZNPQtOmTvkRhBA2cfr0aZo1a8auXbtYtGgR9evXNx3Jo7h0jEJrHQFE3PTYKym+3wnUu93tpnV6bOsMXYkhhPAUJ06coGnTpuzdu5fvvvuOFi1amI7kcWw5hcetGhTnz1t3sxNCeI8zZ85w+fJlvv/+eykSLmLLQnGrrqeNG6FGDeve2EIIz3bu3Dm01lSoUIE9e/bQpEkT05E8li0Lxa26nipWhHnzoFKlLA4khMhSBw4coGbNmrz99tsAcutSFzN9HUWm3GqIokAB6NYta7MIIbJWdHQ0TZo04fLly7Rs2dJ0HK9gyxZFalN4OBwwebJckS2EJ9u9ezcNGzYkJiaGlStXUqtWLdORvIItC0VqYxSHD8OAARARkcoLhBC2d/XqVZo2bUpCQgKrVq2iWrVqpiN5DVt2PfmmUt4KFoS9eyF37qzPI4RwvRw5cjB+/HgqV65M+fLlTcfxKrYsFKl1Pfn5QcmSBsIIIVxq06ZNHD16lLZt29KlSxfTcbySLQtFal1PX38NFy+CTBIphOdYv3494eHh5M+fn/DwcLJly2Y6kley5RhFarPHTp8On35qIIwQwiXWrl1L8+bNCQsL48cff5QiYZBNWxT/fSwiAi5fzvosQgjni4qK4sEHH6RIkSJERkZSqFAh05G8mi1bFKmNUSgFwcEGwgghnO7777+nePHiREVFSZFwA7YsFDe3KDZvhieegAMHjMQRQjjJtWvXAHjvvfdYt24d+fPnN5xIgE0Lxc0tiv37YcUK8LHlTyOEAPjmm2+455572LdvH0opQuXGMm7DIw6tnTpZV2QXLWo6iRAiMxYuXEiXLl3Ily8fueViKLdjy0KRzi2zhRA28sUXX9CtWzfq1q3Ljz/+SK5cuUxHEjexZaG42eDB8OabplMIIW7X0qVL6dWrFw0bNmTZsmWEhISYjiRSYctCcXODYutWuHDBSBQhxB1o2LAhzz33HEuXLiVnzpym44hbsOV1FDeLioKTJ02nEEJk1FdffUWrVq0IDg5m7NixpuOIdNizRZGiSREfD5cugZxFJ4Q9jBs3ji5dujBu3DjTUUQG2bJQpLRkiXVHuz//NJ1ECJGet99+m+eff54uXbrw0ksvmY4jMsj2hSIoCPLlk+nFhXBnWmtee+01Ro0aRY8ePZg7d67M3WQjSmttOsNtCShQRq9b/xs1i911/bH4eJC/OSHc16lTp6hcuTKtWrVi+vTp+Pr6mo7kdZRSm7TWmboloEcMZkuREMI9JX8QzZs3Lxs3bqRgwYL4yBQKtmPL31jKweyHHpJ7UAjhjhwOB4MHD2b48OForSlcuLAUCZuy/W+tcmWoUsV0CiFESg6HgwEDBjBx4kTTUYQT2LLrKeUFd2PGGIshhEhFYmIiTz75JLNmzWLkyJG8+eabqd4aQNiH7VsUQgj3klwkXn/9dSkSHsKWhSL5D2/XLggJgUWLDAcSQlzXpk0b3nnnHV555RUpEh7Cll1PybJnh759oXx500mE8G7Xrl1jw4YNNGjQgE6dOpmOI5zMni2KpP8XLw7jxkGFCibTCOHdYmNj6dixI82aNWP//v2m4wgXsHWLIiEBfH3l/hRCmHL16lU6dOjAihUrmDJlCiVKlDAdSbiAPVsUSYXh7bfBz88qGEKIrHX58mUefPBBfvrpJ2bOnEm/fv1MRxIuYusWRYMGMHq0VSyEEFlrzpw5rFmzhs8//5wePXqYjiNcyJZzPW38fSOVC8uN14UwSWvN1q1bqV69uukoIgPuZK4nW3Y9JTt2DC5fNp1CCO9x9uxZHnzwQXbt2oVSSoqEl7B1oahdG5591nQKIbzDqVOnaNy4MZGRkfzzzz+m44gsZMve/eTB7P/7PyhSxGwWIbzB8ePHadq0Kfv27WPJkiW0aNHCdCSRhWxZKJL16mU6gRCe7/jx4zRq1IhDhw4RERFB48aNTUcSWcy2XU/x8bBnD8TEmE4ihGcLCQmhbNmyLF++XIqEl7JtoYiOhnLlZJ4nIVzln3/+4eLFi+TIkYMlS5bwwAMPmI4kDLFloVDKuk/2559b11IIIZzr77//5oEHHqBnz56mowg3YNsxijx5QP6GhXC+Xbt20bRpU+Lj43njjTdMxxFuwJ4tChTHj8PmzTJ9hxDOtGPHDho1aoTD4SAqKoqqVauajiTcgC0LBcCECVCzJly6ZDqJEJ5Ba02vXr3w8/Nj9erVVKxY0XQk4SZs2fWkFPTpA6VKQajM5CGEUyil+PLLL1FKUbp0adNxhBuxbYuiVCmrWPjY9icQwj38+uuvjBgxAq01ZcqUkSIh/sOWh1mlYMsW2LbNdBIh7G3NmjW0aNGCRYsWce7cOdNxhJuyZdcTwLBhkJgIa9aYTiKEPa1cuZK2bdtStGhRIiMjyZ07t+lIwk3ZtlB8+KGc8SREZi1fvpwOHTpQunRpfvrpJ+6++27TkYQbs2WhUCjkrD0hMi8+Pp4qVarw/fffExYWZjqOcHO2vHHRti2b2fN7MPfcAzLuJkTGHTlyhEKFCgHgcDjwkbNBvIbX3bgoIQHatYM5c0wnEcI+FixYQKlSpfjhhx8ApEiIDLNl15OPD2zYAIGBppMIYQ9z5szhscceo169etSrV890HGEztiwUvr5Qp47pFELYw8yZM3nyySdp3LgxS5YsISgoyHQkYTO2bHueOwfz58OJE6aTCOHeNm3aRJ8+fWjevDlLly6VIiEyxZaFIjoauneHnTtNJxHCvdWoUYPZs2fz7bffkj17dtNxhE25tFAopcKVUruVUtFKqRdvsc7DSqmdSqk/lVJzM7LdKpWtIiHdT0KkbuLEiWzbtg2lFL169SJQBvTEHXBZoVBK+QKTgFZABaC7UqrCTeuUAV4C6mmtKwJDMrLt7NnhnntAWtFC/Ndbb73FoEGD+PTTT01HER7ClS2KOkC01nqf1joOmA+0v2mdvsAkrfU5AK31yYxs+K+/FNOnQ1ycU/MKYWtaa1599VVefvllevbsyccff2w6kvAQriwUhYBDKZYPJz2WUlmgrFLqF6XUeqVUeGobUkr1U0ptVEptBGt+p759wWbXCgrhMlprXnrpJcaMGUPv3r2ZNWsWfn62PKlRuCHTg9l+QBmgEdAdmKaUynXzSlrrqVrrWslXFfbqBYcOgb9/lmYVwm0lJCSwZcsW+vfvz/Tp0/Ek40zNAAAgAElEQVT19TUdSXgQV37kOAIUSbFcOOmxlA4DG7TW8cB+pdQerMLxe1obzpEDCud1ZlQh7MnhcHDlyhWCg4NZsmQJ/v7+KKVMxxIexpUtit+BMkqpEkopf6AbsOSmdb7Bak2glArD6oral96G1/8KM2Y4N6wQduNwOHjqqado0qQJMTExBAQESJEQLuGyQqG1TgCeAZYDu4AvtdZ/KqXGKKXaJa22HDijlNoJrAKGa63PpLftpUvh1VddlVwI95eYmMgTTzzB9OnTadmypZz+KlzKlrPHbvltC/mDciL3WRHeKCEhgUcffZR58+YxZswYRo8ebTqSsIE7mT3WlqdFBAYiRUJ4reeff5558+bxzjvvMGLECNNxhBewZaFYFqG4Kxs88ojpJEJkvSFDhlChQgX69etnOorwErbseqpTbSvqSpDcL1t4jZiYGKZNm8Yzzzwj95EQmeJ1XU8zZkChUNMphMgaV69epX379kRGRlK9enXq169vOpLwMrYsFP7+Ms+T8A6XL1+mTZs2/Pzzz8yaNUuKhDDClm3YN960TpEVwpNdvHiR8PBw1q5dy5w5c3j00UdNRxJeypYtiu+XwvG/oU0b00mEcJ3t27fzxx9/sGDBAjp16mQ6jvBithzM3rNjK0VzByEXoQpPFB8fT7Zs2QA4c+YMefLkMZxIeII7Gcy2ZdeTQkmREB7p5MmT1K5dm1mzZgFIkRBuwZaF4t334McfTacQwrmOHTtGo0aN2LNnD0WKFEn/BUJkEVsWitmzYf160ymEcJ4jR47QqFEjDh48yLJly2jatKnpSEJcZ8vB7J1/QhGZwkN4iEuXLtGgQQNOnTrF8uXLqVevnulIQvyLLQuFEJ4kODiY/v3707BhQ+rUqWM6jhD/Ycuznno9/AdPds9B3bqm0wiReX///TeXLl2iRo0apqMIL+B1Zz3NmgXR0aZTCJF5u3btokGDBjzyyCMkJiaajiNEmmzZ9bTzTwjLYTqFEJmzfft2mjZtiq+vL4sWLZL7Wwu3Z8sWRY4guR+FsKctW7bQuHFj/P39Wb16NRUqVDAdSYh02bJQjBwJO3aYTiHE7Rs/fjxBQUGsXr2asmXLmo4jRIbYcjA7d9A25k3PTqNGptMIkTFaa5RSXLt2jVOnTlG4cGHTkYSXydLBbKWUj1KqR2bezFk2bkSKhLCNNWvWUL9+fc6cOUNAQIAUCWE7tywUSqkQpdRLSqmJSqkWyjII2Ac8nHURU8lm8s2FuA2RkZGEh4dz5swZ4uLiTMcRIlPSalF8DpQDtgNPAquAzkAHrXX7LMh2S0OHwrFjJhMIkb4ffviBNm3aUKpUKaKioihQoIDpSEJkSlqnx5bUWlcGUEpNB44BRbXWsVmSLA2//QYJCaZTCHFrP/74I+3bt6dChQqsWLGCsLAw05GEyLS0WhTxyd9orROBw+5QJADWrQOZXFO4swoVKtChQwdWrlwpRULY3i3PelJKJQJXuDEkkB24mrSstdYhWZLwJgEFyugDf22jQGh2E28vRJp++eUX6tatKxfRCbfjkrOetNa+WusQrXVw0pdfimUjRSJZv74Kh8NkAiH+a/bs2TRo0IBx48aZjiKEU91yjEIpFQj0B0oD24CZWmu3GBn4eS342PJSQeGpZsyYQd++fWncuDFPP/206ThCOFVah9v/AbWwznpqDbjNx6Tdf5lOIMQNn3zyCU8++SQtW7Zk6dKlBAUFmY4khFOlddZThRRnPc0AfsuaSELYx7Fjxxg+fDht27Zl4cKFBAQEmI4khNNl9Kwnt+hySvbMM6YTCGEpUKAAa9eu5auvvpIiITxWWi2Kakqpi0nfKyB70rLRs54ATpww9c5CWN58803CwsLo378/1atXNx1HCJdKq0XxR9JZTiHudtbTwoUm3114M601o0ePZvTo0axfvx67TaopRGak1aKQfwFCpKC1ZsSIEYwdO5Y+ffowZcoUlJKZx4TnS6tQ5FNKDbvVk1rrD1yQJ0Oefx4+n2rq3YU30lozbNgwPvzwQwYMGMDEiRPxkXO0hZdIq1D4Ajlxw8laY2JMJxDeRilFoUKFGDJkCB988IG0JIRXSWsKj81a6xpZnCddAQXK6IO7t3N3SKDpKMILJCYmsnfv3ut3o0u+AZEQduOqGxe57b8GGT8UWSExMZHevXtTu3Ztjhw5AiBFQniltApF0yxLcZv69jWdQHi6+Ph4evbsyeeff87w4cMpVKiQ6UhCGHPLMQqt9dmsDHI7ChY0nUB4sri4OB555BG+/vpr3n33XV544QXTkYQwKq3BbLc1ZozpBMKTffzxx3z99deMHz+eIUOGmI4jhHG2LBRCuNLgwYO55557aN26tekoQrgFW54I/tZbphMIT3PlyhWeeuopTp48SbZs2aRICJGCLQtFDrm5nXCiS5cu0apVK6ZPn866detMxxHC7diy62noLa8XF+L2XLhwgVatWvHbb78xd+5cOnToYDqSEG7HloVCCGc4d+4cLVu2ZMuWLXz55Zd07NjRdCQh3JItu57ee890AuEJ4uLiiIuLY9GiRVIkhEiDLVsUefKYTiDs7MyZM4SEhHD33XezadMmfH19TUcSwq3ZskXRp4/pBMKujh07Rv369enXrx+AFAkhMsCWhUKIzDh8+DANGzbk4MGDPP7446bjCGEbtiwU48ebTiDs5sCBAzRo0IATJ07w448/0rBhQ9ORhLANW45RFChgOoGwE4fDQdu2bTl37hw//fQTtWvXNh1JCFuxZaHo3s10AmEnPj4+TJ48mRw5clC9enXTcYSwHVt2PQmRETt37mTKlCkA1KtXT4qEEJlky0Lx8UTTCYS727ZtG40aNeL111/n/PnzpuMIYWu2LBR35TKdQLizzZs307hxY/z9/YmKiiJXLvmDEeJO2LJQyEW04lY2bNhAkyZNCA4OZs2aNdfvdS2EyDyXFgqlVLhSardSKlop9WIa63VSSmmlVIZu/J0jh/MyCs+ydetWwsLCWL16NSVLljQdRwiP4LJCoZTyBSYBrYAKQHelVIVU1gsGngU2ZHTbSeOTQlx35coVAJ566im2bdtGsWLFDCcSwnO4skVRB4jWWu/TWscB84H2qaz3BvAuEJvRDZcq7ZyAwjP89NNPlChRgvXr1wOQQ5qcQjiVKwtFIeBQiuXDSY9dp5SqARTRWn+f1oaUUv2UUhuVUhsBmjV1dlRhVxEREbRp04b8+fNLV5MQLmJsMFsp5QN8ADyX3rpa66la61pa6wyNYQjv8O2339KhQwcqVqzIqlWryJcvn+lIQngkVxaKI0CRFMuFkx5LFgxUAqKUUgeAusCSjAxoz5jpxJTCltavX0/nzp2pXr06kZGR5JG554VwGVcWit+BMkqpEkopf6AbsCT5Sa31Ba11mNa6uNa6OLAeaKe13pjehuWMR1GrVi1eeeUVVqxYIddJCOFiLisUWusE4BlgObAL+FJr/adSaoxSqt2dbLv+A85IKOxo4cKFHD16FD8/P0aPHk1ISIjpSEJ4PJeOUWitI7TWZbXWpbTWbyU99orWekkq6zbKSGtCeK9p06bRtWtXxowZYzqKEF7Flldmz55tOoHIapMmTaJfv36Eh4fz4Ycfmo4jhFexZaEoV950ApGVxo8fzzPPPEO7du1YvHgxgYGBpiMJ4VWU1tp0htsSUKCMPrR7O/lC5GDhDWJjY6lTpw5ly5Zl7ty5+Pv7m44khC0ppTZl9hIDW964SHiHhIQEAgMDWbVqFaGhofj5yZ+rECbYsutp4ULTCYQraa15+eWX6dy5M/Hx8eTJk0eKhBAG2bJQlL/HdALhKlprXnjhBd566y3y5cuHr6+v6UhCeD1bForKlUwnEK6gtWbIkCG8//77PP3000yePBkfH1v+iQrhUWz5r9Bm4+8ig0aMGMGECRMYOnQoH3/8sRQJIdyELTt+IyKgd3fTKYSzdenShaCgIF555RWUUqbjCCGS2LJQlC1nOoFwloSEBCIiImjXrh21a9emdu3apiMJIW5iy7Z9mTKmEwhniI+Pp0ePHrRv3/76TYeEEO7Hli0KR6LpBOJOxcXF0a1bNxYvXszYsWOpW7eu6UhCiFuwZYtizRrTCcSdiI2NpWPHjixevJiPPvqI559/3nQkIUQabNmiKFXKdAJxJ1atWsWyZcuYPHkyTz31lOk4Qoh02LJQFClqOoHIDK01SilatWrFzp07KVdOzkoQwg5s2fWUEG86gbhdly5dIjw8nJUrVwJIkRDCRmxZKDZsMJ1A3I7z58/TokULIiMjOX36tOk4QojbZMuupxIlTCcQGXX27FlatGjBtm3bWLhwIQ899JDpSEKI22TLQlGwkOkEIiMuXLhAkyZN2LVrF4sWLaJNmzamIwkhMsGWXU/xcaYTiIwIDg7mvvvuY8mSJVIkhLAxW7YoNm+GQs1NpxC3cvToURISEihatCiffvqp6ThCiDtky0IhYxTu69ChQzRp0oScOXOyadMmmQFWCA9gy0KR727TCURq9u/fT5MmTTh79iyzZ8+WIiGEh7BloYi7BgSbTiFSio6OpkmTJly+fJnIyEhq1crUPdyFEG7Ilh/5duwwnUDcbNiwYcTExLBy5UopEkJ4GFu2KIoWM51A3GzWrFkcP36cChUqmI4ihHAyW7YowsJMJxAAf/zxB7169eLatWvkzp1bioQQHsqWLYprscgYhWGbNm2iefPmBAUFceLECYoWlZkahfBUtmxR7N5jOoF3W79+PU2bNiU0NJQ1a9ZIkRDCw9myUBQpbDqB9/rll19o3rw5YWFhrF69mhJyUYsQHs+WheKu3KYTeK+cOXNSpUoVVq9eLS0JIbyELQvFtVjTCbxPdHQ0AFWrVmXt2rUUKiQzMwrhLWxZKPbtM53Au0RERFCpUiWmTJkCgFLKcCIhRFayZaEoXFgOVFnl22+/pUOHDlSsWJHOnTubjiOEMMCWhSIkxHQC77Bw4UI6d+5MjRo1iIyMJE+ePKYjCSEMsGWhuHbNdALPd/DgQXr06EHdunX58ccfyZUrl+lIQghDbFkojh01ncDzFS1alMWLF7Ns2TJCpAknhFezZaGQacZdZ9q0aURERADw4IMPkjNnTsOJhBCm2bJQ5MhhOoFnmjhxIv369WPmzJmmowgh3IgtC0V8vOkEnueDDz5g0KBBdOjQgblz55qOI4RwI7YsFOfOmk7gWf7v//6P5557ji5duvDll1/i7+9vOpIQwo3YslDkust0As+hteaff/6hR48ezJ07l2zZspmOJIRwM7acZlw+8N45rTVnzpwhLCyMTz75BK01vr6+pmMJIdyQLVsUiYmmE9ib1prnn3+eGjVqcPLkSXx8fKRICCFuyZaF4upV0wnsy+FwMHjwYD744AM6dOhA3rx5TUcSQrg5WxaKoCDTCezJ4XAwYMAAJk6cyHPPPcdHH30kE/wJIdJly0LhY8vU5o0dO5apU6cycuRIxo4dK0VCCJEhthzMdjhMJ7Cn/v37kzt3bp588kkpEkKIDLPlZ/O4ONMJ7CM+Pp63336bq1evEhoaSt++faVICCFuiy0LhZwemzHXrl3j4YcfZtSoUdfnbxJCiNtly64nGaNIX2xsLJ06dSIiIoKPP/5YbjokhMg0WxYKGaNI29WrV+nQoQM//fQTU6ZMoV+/fqYjCSFszJaFQkuhSNPRo0fZvn07M2fO5PHHHzcdRwhhc7YsFL62TO16MTExBAYGUrp0afbs2UNwcLDpSEIID2DL3n45aee/zp8/T+PGjXnllVcApEgIIZzGloVCa9MJ3MvZs2dp1qwZmzdvplatWqbjCCE8jEsLhVIqXCm1WykVrZR6MZXnhymldiqltimlIpVSxVyZxxOdOnWKxo0bs2PHDr755hvat29vOpIQwsO4rFAopXyBSUAroALQXSlV4abVtgC1tNZVgK+A9zK2bWcmta+EhASaN2/Onj17+O6772jdurXpSEIID+TKYeE6QLTWeh+AUmo+0B7YmbyC1npVivXXAz1dmMfj+Pn5MXLkSPLmzUvjxo1NxxFCeChXFopCwKEUy4eBe9NYvw+wzIV5PMbBgwfZsWMHrVu35uGHHzYdRwjh4dziRFOlVE+gFtDwFs/3A/oB+OcvnYXJ3M/+/ftp0qQJV65cYd++feTMmdN0JCGEh3PlYPYRoEiK5cJJj/2LUqoZMApop7W+ltqGtNZTtda1tNZefUrP33//TYMGDbhw4QLLli2TIiGEyBKuLBS/A2WUUiWUUv5AN2BJyhWUUtWBKVhF4qQLs9jerl27aNiwIbGxsaxatYqaNWuajiSE8BIu63rSWicopZ4BlgO+wEyt9Z9KqTHARq31EmAskBNYmDT19UGtdTtXZbKzBQsW4HA4iIqKomLFiqbjCCG8iNI2u3otoEAZfeTvHYTlDDAdJUs4HA58fHzQWnP8+HEKFChgOpIQwoaUUpsy231vyyuzvcXGjRupUqUKe/bsQSklRUIIYYQUCjf166+/0rRpU65cuYK/3KlJCGGQFAo39PPPP9OiRQvy5cvHmjVrKF68uOlIQggvJoXCzfz222+Eh4dTuHBhVq9eTZEiRdJ/kRBCuJAUCjdToUIFunfvTlRUFAULFjQdRwgh7FkoPHFOwNWrV3Pp0iVy5szJ9OnTufvuu01HEkIIwKaFwtMsXryY5s2bM3LkSNNRhBDiP6RQGLZgwQK6dOlCrVq1ePPNN03HEUKI/5BCYdCcOXN45JFHuP/++1m+fDmhoaGmIwkhxH9IoTDkypUrvPjiizRq1Ihly5bJPa6FEG7LLaYZ90ZBQUGsXr2aggULkj17dtNxhBDilqRFkcUmTJjA0KFD0VpTqlQpKRJCCLcnhSILvf/++zz77LP8888/JCYmmo4jhBAZIoUii7z11lsMHz6chx9+mAULFuDnJ71+Qgh7kEKRBd544w1efvllevbsyRdffEG2bNlMRxJCiAyTQpEFqlSpQr9+/Zg1a5a0JIQQtmPLGxcd/XsHedz8xkVaa7Zs2UKNGjVMRxFCCLlxkbtxOBwMGjSIOnXqsHXrVtNxhBDijkg/iJM5HA6eeuoppk+fzvDhw6latarpSEIIcUekReFEiYmJPPHEE0yfPp1Ro0bx7rvvopQnznUrhPAmUiic6Ouvv+Z///sfY8aM4c0335QiIYTwCNL15ERdunQhb968NG7c2HQUIYRwGikUd+jatWsMGDCA5557jooVK0qREB4hPj6ew4cPExsbazqKuE2BgYEULlzYqddrSaG4AzExMXTq1Illy5ZRr149KlasaDqSEE5x+PBhgoODKV68uHSh2ojWmjNnznD48GFKlCjhtO3acozCHf5wr169Srt27fjhhx+YOnUqffr0MR1JCKeJjY0lT548bvFvTWScUoo8efI4vSUoLYpMuHz5Mm3atOHnn3/ms88+47HHHjMdSQinkyJhT674vdmyRWGar68vAQEBzJkzR4qEEC701ltvUbFiRapUqUK1atXYsGEDCQkJjBw5kjJlylCtWjWqVavGW2+9df01vr6+VKtWjYoVK1K1alXGjRuHw+Ew+FPYn7QobsP58+dRShEaGsoPP/wgn7iEcKFff/2VpUuXsnnzZgICAjh9+jRxcXG8/PLLHD9+nO3btxMYGMilS5cYN27c9ddlz579+owIJ0+e5JFHHuHixYu8/vrrpn4U25NCkUFnzpyhefPmhISEsGrVKikSQrjYsWPHCAsLIyDAmtctLCyMq1evMm3aNA4cOEBgYCAAwcHBvPbaa6luI1++fEydOpXatWvz2muvyb/bTJKupww4efIkjRs3ZufOnYwYMUL+2ITXadQIZs2yvo+Pt5bnzLGWr161lhcssJYvXLCWFy2ylk+ftpa/+85aPn48Y+/ZokULDh06RNmyZRk4cCCrV68mOjqaokWL3tY95kuWLEliYiInT57M8GvEv0mhSMexY8do1KgR0dHRLF26lFatWpmOJIRXyJkzJ5s2bWLq1KnkzZuXrl27EhUV9a91PvvsM6pVq0aRIkU4dOiQmaDeQGttqy///KX1mcvXdFZp1qyZDgoK0lFRUVn2nkKYtnPnTtMR/mPhwoW6WbNmOnfu3PrixYv/eq5ixYp6//79Wmutg4KC/vXc3r17de7cubXD4ciqqMal9vsDNupMHnelRZGOyZMn8+OPP9KwYUPTUYTwKrt37+bvv/++vrx161bKlStHnz59eOaZZ65fK5CYmEhcXFyq2zh16hT9+/fnmWeekS7jOyCD2anYt28fs2bN4vXXX6dUqVKUKlXKdCQhvM7ly5cZNGgQ58+fx8/Pj9KlSzN16lRCQ0MZPXo0lSpVIjg4mOzZs/PYY49RsGBBwJoxoVq1asTHx+Pn50evXr0YNmyY4Z/G3mx5h7tj0X+SO8jfJdvfs2cPTZo0ISYmhs2bN1OsWDGXvI8Q7mzXrl3cc889pmOITErt9yd3uHOSXbt20bBhQ+Li4li1apUUCSGEQArFddu3b78+DhEVFUWVKlUMJxJCCPcghSLJkSNHCA4OZvXq1VSoUMF0HCGEcBteP5h97tw57rrrLsLDw9m1axf+/q4Z+xBCCLvy6hbFunXrKFmyJF9//TWAFAkhhEiF1xaKNWvW0KJFC/Lmzcu9995rOo4QQrgtrywUkZGRhIeHU7RoUVavXk3hwoVNRxJC3CR5uvBKlSrRtm1bzp8/75TtHjhwgEqVKjllWym99tprFCpU6PrU5y+++KLT3yPZ1q1biYiIcNn2b2bLQnEn11fu27ePNm3aULp0aaKioihQoIDTcgkhnCd5uvAdO3aQO3duJk2aZDpSuoYOHcrWrVvZunUr77zzToZfl5iYeFvvI4XCxUqWLMkHH3zAypUryZcvn+k4QogMuO+++zhy5AhgXbHdtGlTatSoQeXKlfn2228Bq6Vwzz330LdvXypWrEiLFi2IiYkBYNOmTVStWpWqVav+q+DExsbSu3dvKleuTPXq1Vm1ahUAs2bNokOHDjRv3pzixYszceJEPvjgA6pXr07dunU5e/ZshrNHRkZSvXp1KleuzBNPPMG1a9cAKF68OCNGjKBGjRosXLiQvXv3Eh4eTs2aNalfvz5//fUXAAsXLqRSpUpUrVqVBg0aEBcXxyuvvMKCBQuoVq0aC5Kn7XUhrznr6ZtvvqFYsWJUr16dAQMGmI4jhG0Uf/F7l2z3wDsPZmi9xMREIiMjr9+XPjAwkMWLFxMSEsLp06epW7cu7dq1A+Dvv/9m3rx5TJs2jYcffpivv/6anj170rt3byZOnEiDBg0YPnz49W1PmjQJpRTbt2/nr7/+okWLFuzZsweAHTt2sGXLFmJjYyldujTvvvsuW7ZsYejQocyePZshQ4b8J+v48eOZkzT/+rvvvkvDhg15/PHHiYyMpGzZsjz66KN8+umn11+bJ08eNm/eDEDTpk2ZPHkyZcqUYcOGDQwcOJCVK1cyZswYli9fTqFChTh//jz+/v6MGTOGjRs3MnHixEzu/dvjFS2K+fPn07lz51ve3EQI4X6S52zKnz8/J06coHnz5oA14/XIkSOpUqUKzZo148iRI5w4cQKAEiVKUK1aNQBq1qzJgQMHOH/+POfPn6dBgwYA9OrV6/p7rF27lp49ewJQvnx5ihUrdr1QNG7cmODgYPLmzUtoaCht27YFoHLlyhw4cCDVzCm7nlq2bMnu3bspUaIEZcuWBeCxxx5jzZo119fv2rUrYLWS1q1bR5cuXahWrRpPPfUUx44dA6BevXo8/vjjTJs27ba7qJzF41sUs2fPpnfv3jzwwAPXK70QIuMy+snf2ZLHKK5evUrLli2ZNGkSgwcP5osvvuDUqVNs2rSJbNmyUbx48eszySbfDQ+swfDkrqfMSLktHx+f68s+Pj4kJCRkerspBQUFAeBwOMiVK9f1W7imNHnyZDZs2MD3339PzZo12bRpk1Pe+3Z4dItixowZPP744zRq1IiIiIjbuiuWEMI95MiRgwkTJjBu3DgSEhK4cOEC+fLlI1u2bKxatYp//vknzdfnypWLXLlysXbtWgC++OKL68/Vr1//+vKePXs4ePAg5cqVc1r2cuXKceDAAaKjowH4/PPPU71lQUhICCVKlGDhwoWA1Wr6448/ANi7dy/33nsvY8aMIW/evBw6dIjg4GAuXbrktJzp8dhCobVm0aJFtGzZkqVLl16v3EII+6levTpVqlRh3rx59OjRg40bN1K5cmVmz55N+fLl0339Z599xtNPP021atVIOWP2wIEDcTgcVK5cma5duzJr1qx/tSTuVGBgIJ999hldunShcuXK+Pj40L9//1TX/eKLL5gxYwZVq1alYsWK1wfphw8fTuXKlalUqRL3338/VatWvX5r5qwazLblNOPHo//krjSmGb927RoBAQHExMT8q8kohMgYmWbc3mSa8XS899573H///Vy4cIHs2bNLkRBCiDvkUYXijTfeYMSIEZQtW5YcOXKYjiOEEB7BIwqF1prRo0fzyiuv0KtXL+bMmUO2bNlMxxJCCI/gEYXi/fff580336RPnz589tln+Pr6mo4khO3ZbfxSWFzxe/OI6yi6du3K5cuXefXVV/Hx8YjaJ4RRgYGBnDlzhjx58qDUncyuJrKS1pozZ84QGBjo1O3a9qyn0Ox+fPHFF/To0UOKgxBOFh8fz+HDh69fyCbsIzAwkMKFC/+n+/1OznpyaYtCKRUOfAT4AtO11u/c9HwAMBuoCZwBumqtD6S33cTERPr1G8iMGTPIkSMHnTp1cn54IbxYtmzZKFGihOkYwk247KO4UsoXmAS0AioA3ZVSN9+Mug9wTmtdGhgPvJvuhrXmmf59mTFjBqNHj6Zjx45OTi6EECIlV/bZ1AGitdb7tNZxwHyg/U3rtAf+l/T9V0BTlU6HaMKFEyyY9wVvvBcHpNAAAAViSURBVPEGY8aMkf5TIYRwMVcWikLAoRTLh5MeS3UdrXUCcAHIk9ZGHbFXeO2Nt3j55ZedGFUIIcSt2OKsJ6VUP6Bf0uK110aP2vHa6FEmI7mLMOC06RBuQvbFDbIvbpB9cUOmZzt0ZaE4AhRJsVw46bHU1jmslPIDQrEGtf9Faz0VmAqglNqY2ZF7TyP74gbZFzfIvrhB9sUNSqmNmX2tK7uefgfKKKVKKKX8gW7AkpvWWQI8lvR9Z2Clttv5ukII4eFc1qLQWicopZ4BlmOdHjtTa/2nUmoMsFFrvQSYAXyulIoGzmIVEyGEEG7EpWMUWusIIOKmx15J8X0s0OU2NzvVCdE8heyLG2Rf3CD74gbZFzdkel/Y7spsIYQQWUvmvhBCCJEmty0USqlwpdRupVS0UurFVJ4PUEotSHp+g1KqeNanzBoZ2BfDlFI7lVLblFKRSqliJnJmhfT2RYr1OimltFLKY894yci+UEo9nPS38adSam5WZ8wqGfg3UlQptUoptSXp30lrEzldTSk1Uyl1Uim14xbPK6XUhKT9tE0pVSNDG9Zau90X1uD3XqAk4A/8AVS4aZ2BwOSk77sBC0znNrgvGgM5kr4f4M37Imm9YGANsB6oZTq3wb+LMsAW4K6k5XymcxvcF1OBAUnfVwAOmM7ton3RAKgB7LjF862BZYAC6gIbMrJdd21RuGT6D5tKd19orVdpra8mLa7HumbFE2Xk7wLgDax5wzx56tOM7Iu+wCSt9TkArfXJLM6YVTKyLzQQkvR9KHA0C/NlGa31GqwzSG+lPTBbW9YDuZRSBdLbrrsWCpdM/2FTGdkXKfXB+sTgidLdF0lN6SJa6++zMpgBGfm7KAuUVUr9opRanzSbsyfKyL54DeiplDqMdSbmoKyJ5nZu93gC2GQKD5ExSqmeQC2goeksJiilfIAPgMcNR3EXfljdT42wWplrlFKVtdbnjaYyozswS2s9Til1H9b1W5W01g7TwezAXVsUtzP9B2lN/+EBMrIvUEo1A0YB7bTW17IoW1ZLb18EA5WAKKXUAaw+2CUeOqCdkb+Lw8ASrXW81no/sAercHiajOyLPsCXAFrrX4FArHmgvE2Gjic3c9dCIdN/3JDuvlBKVQemYBUJT+2HhnT2hdb6gtY6TGtdXGtdHGu8pp3WOtNz3LixjPwb+QarNYFSKgyrK2pfVobMIhnZFweBpgBKqXuwCsWpLE3pHpYAjyad/VQXuKC1Ppbei9yy60nL9B/XZXBfjAVyAguTxvMPaq3bGQvtIhncF14hg/tiOdBCKbUTSASGa609rtWdwX3xHDBNKTUUa2D7cU/8YKmUmof14SAsaTzmVSAbgNZ6Mtb4TGsgGrgK9M7Qdj1wXwkhhHAid+16EkII4SakUAghhEiTFAohhBBpkkIhhBAiTVIohBBCpEkKhRAZpJRKVEptTfFVXCnVSCl1IWl5l1Lq1aR1Uz7+l1LqfdP5hcgst7yOQgg3FaO1rpbygaTp7X/WWrdRSgUBW5VS3yU9nfx4dmCLUmqx1vqXrI0sxJ2TFoUQTqK1vgJsAkrf9HgMsJUMTL4mhDuSQiFExmVP0e20+OYnlVJ5sOaX+vOmx+/CmmNpTdbEFMK5pOtJiIz7T9dTkvpKqS2AA3gnafqIRkmP/4FVJD7UWh/PwqxCOI0UCiHu3M9a6za3elwpVQJYr5T6Umu9NavDCXGnpOtJCBdLmuL7HWCE6SxCZIYUCiGyxmSgQdJZUkLYisweK4QQIk3SohBCCJEmKRRCCCHSJIVCCCH+v706EAAAAAAQ5G+9wQQlEUsUACxRALBEAcASBQBLFACsAIibUdQIqaN4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(fpr, tpr, 'b:',label='SGD')\n",
    "plot_roc_curve(fpr_rf, tpr_rf, 'Random Forest')\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ROC_AUC socre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9920527492698306"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train_5, y_scores_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Precision, Recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.986046511627907"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision_score(y_train_5, (y_scores_rf > 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8212506917542889"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_5, (y_scores_rf > 0.5))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}