ToBigs_2wk_optimization.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DKIX8PqcLMaw"
      },
      "source": [
        "# Gradient Descent 구현하기\n",
        "\n",
        "### 1)\"...\"표시되어 있는 빈 칸을 채워주세요\n",
        "### 2)강의내용과 코드에 대해 공부한 내용을 마크마운 또는 주석으로 설명해주세요"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2StPehwLMat"
      },
      "source": [
        "# Tobig's 19기 2주차 Optimization 과제"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6DNHHXfLMax"
      },
      "source": [
        "## 데이터"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "id": "EP3O4xptLMay"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import random"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "id": "oByQ9wXHLMay"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Label</th>\n",
              "      <th>bias</th>\n",
              "      <th>experience</th>\n",
              "      <th>salary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.7</td>\n",
              "      <td>48000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1.9</td>\n",
              "      <td>48000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2.5</td>\n",
              "      <td>60000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>4.2</td>\n",
              "      <td>63000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>6.0</td>\n",
              "      <td>76000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Label  bias  experience  salary\n",
              "0      1     1         0.7   48000\n",
              "1      0     1         1.9   48000\n",
              "2      1     1         2.5   60000\n",
              "3      0     1         4.2   63000\n",
              "4      0     1         6.0   76000"
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data = pd.read_csv('2wk_Optimization_assignment.csv')\n",
        "data.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ubOR3hWGLMaz"
      },
      "source": [
        "## Train Test 데이터 나누기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {
        "id": "IySSjlizLMaz"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {
        "id": "075EQI1bLMa0"
      },
      "outputs": [],
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(data.iloc[:, 1:], data.iloc[:, 0], test_size = 0.25, random_state = 0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "위 코드는 train 데이터와 test 데이터를 나누는 코드이다. 하나씩 살펴보면\n",
        "- data.iloc[:, 1:] : 첫 번째 열을 제외한 나머지 열 선택 //독립변수의 데이터 의미\n",
        "- data.iloc[:, 0] : 첫 번째 열만 선택 // 종속변수의 모든 데이터 의미\n",
        "- test_size = 0.25 : train 75% // test 25% 로 나눈 것을 의미\n",
        "- random_state = 0 : 나누는 과정에서 동일한 무작위 결과 얻을 수 있도록 고정하는 것\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {
        "id": "O8Ht5u8kLMa1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "((150, 3), (50, 3), (150,), (50,))"
            ]
          },
          "execution_count": 98,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "위의 결과를 확인했을 때, 75% // 25%로 train과 test가 잘 나뉜 것을 확인할 수 있다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hYmxND_xLMa2"
      },
      "source": [
        "## Scaling\n",
        "\n",
        "experience와 salary의 단위, 평균, 분산이 크게 차이나므로 scaler를 사용해 단위를 맞춰줍니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {
        "id": "UI0Xy0gHLMa3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bias</th>\n",
              "      <th>experience</th>\n",
              "      <th>salary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0.187893</td>\n",
              "      <td>-1.143335</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1.185555</td>\n",
              "      <td>0.043974</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>-0.310938</td>\n",
              "      <td>-0.351795</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>-1.629277</td>\n",
              "      <td>-1.341220</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>-1.308600</td>\n",
              "      <td>0.043974</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   bias  experience    salary\n",
              "0     1    0.187893 -1.143335\n",
              "1     1    1.185555  0.043974\n",
              "2     1   -0.310938 -0.351795\n",
              "3     1   -1.629277 -1.341220\n",
              "4     1   -1.308600  0.043974"
            ]
          },
          "execution_count": 99,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "scaler = StandardScaler()\n",
        "bias_train = X_train[\"bias\"]\n",
        "bias_train = bias_train.reset_index()[\"bias\"]\n",
        "X_train = pd.DataFrame(scaler.fit_transform(X_train), columns = X_train.columns)\n",
        "X_train[\"bias\"] = bias_train\n",
        "X_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xD7L7RwZLMa3"
      },
      "source": [
        "이때 scaler는 X_train에 fit 해주시고, fit한 scaler를 X_test에 적용시켜줍니다.  \n",
        "똑같이 X_test에다 fit하면 안돼요!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "id": "xBsUSCGGLMa3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bias</th>\n",
              "      <th>experience</th>\n",
              "      <th>salary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>-1.344231</td>\n",
              "      <td>-0.615642</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0.508570</td>\n",
              "      <td>0.307821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>-0.310938</td>\n",
              "      <td>0.571667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>1.363709</td>\n",
              "      <td>1.956862</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>-0.987923</td>\n",
              "      <td>-0.747565</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   bias  experience    salary\n",
              "0     1   -1.344231 -0.615642\n",
              "1     1    0.508570  0.307821\n",
              "2     1   -0.310938  0.571667\n",
              "3     1    1.363709  1.956862\n",
              "4     1   -0.987923 -0.747565"
            ]
          },
          "execution_count": 100,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "bias_test = X_test[\"bias\"]\n",
        "bias_test = bias_test.reset_index()[\"bias\"]\n",
        "X_test = pd.DataFrame(scaler.transform(X_test), columns = X_test.columns)\n",
        "X_test[\"bias\"] = bias_test\n",
        "X_test.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "* scaling 과정에서 보통 MinmaxScaler 혹은 StandardScaler를 많이 사용한다. 그 중 위에서는 StandardScaler를 사용하여 진행하였고, 이는 평균이 0 분산을 1로 만들어주는 표준화 과정이다.\n",
        "* 위에서 강조했듯이 X_train을 기준으로 표준화 fit을 진행하고, 이를 기준으로 X_test에다가도 적용해줘야 한다.\n",
        "* bias는 표준화 없이 그대로 사용하였다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "id": "m9sP3nzlLMa4"
      },
      "outputs": [],
      "source": [
        "# parameter 개수\n",
        "N = len(X_train.loc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "id": "qz7xz9dbLMa4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.88310631, 0.63516729, 0.22048042])"
            ]
          },
          "execution_count": 102,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 초기 parameter들을 임의로 설정해줍니다.\n",
        "parameters = np.array([random.random() for i in range(N)])\n",
        "random_parameters = parameters.copy()\n",
        "parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QINz-EAKLMa4"
      },
      "source": [
        "### * LaTeX   \n",
        "\n",
        "Jupyter Notebook은 LaTeX 문법으로 수식 입력을 지원하고 있습니다.  \n",
        "LaTeX문법으로 아래의 수식을 완성해주세요  \n",
        "http://triki.net/apps/3466  \n",
        "https://jjycjnmath.tistory.com/117"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D2DsTfXuLMa5"
      },
      "source": [
        "## Dot product\n",
        "## $z = X_i \\theta$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "$z = X_i \\theta$ 는 로지스틱 함수의 자연상수 e의 지수에 -를 제외한 값이다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "id": "2y05lS6xLMa5"
      },
      "outputs": [],
      "source": [
        "def dot_product(X, parameters):\n",
        "    z = 0\n",
        "    for i in range(len(parameters)):\n",
        "        z += X[i]*parameters[i]\n",
        "    return z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fOGPEhtOLMa5"
      },
      "source": [
        "## Logistic Function\n",
        "\n",
        "## $p = \\frac{1}{1-e^{-x_i \\theta}}$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "id": "2awM57u5LMa5"
      },
      "outputs": [],
      "source": [
        "def logistic(X, parameters):\n",
        "    z = dot_product(X, parameters)\n",
        "    p = 1 / (1 + np.exp(-z))\n",
        "    return p"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "id": "WVaZEwrdLMa5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.8383263535881001"
            ]
          },
          "execution_count": 105,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "logistic(X_train.iloc[1], parameters)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "위 코드를 통해 Logistic Function을 구현했다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6cXHl8bLMa6"
      },
      "source": [
        "## Object function\n",
        "\n",
        "Object Function : 목적함수는 Gradient Descent를 통해 최적화 하고자 하는 함수입니다.  \n",
        "<br>\n",
        "선형 회귀의 목적함수\n",
        "## $l(\\theta) = \\frac{1}{2}\\Sigma(y_i - \\theta^{T}X_i)^2$  \n",
        "참고) $\\hat{y_i} = \\theta^{T}X_i$\n",
        "  \n",
        "로지스틱 회귀의 목적함수를 작성해주세요  \n",
        "(선형 회귀의 목적함수처럼 강의에 나온대로 작성해주세요. 평균을 고려하는 것은 뒤에 코드에서 수행합니다)\n",
        "## $l(p) = - \\Sigma(y_{i}log p(X_{i}) + (1 - y_{i}) log(1 - p(X_{i}))) $"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {
        "id": "FnGRAur3LMa6"
      },
      "outputs": [],
      "source": [
        "def minus_log_cross_entropy_i(X, y, parameters):\n",
        "    p = logistic(X, parameters)\n",
        "    loss = -(y * np.log(p) + (1-y) * np.log(1-p))\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {
        "id": "C922eXYyLMa6"
      },
      "outputs": [],
      "source": [
        "def mse_i(X, y, parameters):\n",
        "    y_hat = np.dot(X, parameters.T)\n",
        "    loss = ((y - y_hat)**2) / 2\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "id": "0j-MhGkyLMa6"
      },
      "outputs": [],
      "source": [
        "def batch_loss(X_set, y_set, parameters, loss_function, n): #n:현재 배치의 데이터 수\n",
        "    loss = 0\n",
        "    for i in range(X_set.shape[0]):\n",
        "        X = X_set.iloc[i,:]\n",
        "        y = y_set.iloc[i]\n",
        "        loss += loss_function(X, y, parameters)\n",
        "    loss = loss / n #loss 평균값으로 계산\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "id": "uSkPS5olLMa7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1.226364931368473"
            ]
          },
          "execution_count": 109,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "batch_loss(X_test, y_test, parameters, minus_log_cross_entropy_i, len(X_test))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.8605071023014405"
            ]
          },
          "execution_count": 110,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "batch_loss(X_test, y_test, parameters, mse_i, len(X_test))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "minus_log_cross_entropy_i와 mse_i는 각각의 데이터 셋에 대한 목적 함수로, 시그마의 안쪽 부분을 의미하고 batch_loss를 통해서 시그마를 실행함으로써 전체의 목적 함수의 평균을 구하는 것이다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACLi9vCyLMa7"
      },
      "source": [
        "## Gradient\n",
        "위의 선형회귀의 목적함수 $l(\\theta)$와 로지스틱회귀의 목적함수 $l(p)$의 gradient를 작성해주세요  \n",
        "(위의 목적함수를 참고해서 작성해주세요 = 평균을 고려하는 것은 뒤에 코드에서 수행합니다)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "caMA-f00LMa7"
      },
      "source": [
        "## ${\\partial\\over{\\partial \\theta_j}}l(\\theta)=$ $  -\\Sigma(y_{i} - \\theta^{T}X_{i})X_{ij} $\n",
        "## ${\\partial\\over{\\partial \\theta_j}}l(p)=$ $  -\\Sigma(y_{i} - p_{i})X_{ij} $"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {
        "id": "apZ0Miz5LMa7"
      },
      "outputs": [],
      "source": [
        "def get_gradient_ij(X, y, parameters, j, model):\n",
        "    if model == 'linear':\n",
        "        y_hat = np.dot(X, parameters.T)\n",
        "        gradient = (y - y_hat) * X[j]\n",
        "    else:\n",
        "        p = logistic(X, parameters)\n",
        "        gradient = (y - p) * X[j]\n",
        "    return - gradient"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {
        "id": "XXBe6q8gLMa7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "-0.06026507167861816"
            ]
          },
          "execution_count": 112,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "get_gradient_ij(X_train.iloc[0,:], y_train.iloc[0], parameters, 1, 'logistic')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "-0.04690433211376088"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "get_gradient_ij(X_train.iloc[0,:], y_train.iloc[0], parameters, 1, 'linear')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "id": "BB9rbOHkdFgm",
        "outputId": "d7faa6e7-7713-41a0-ffd9-32dbb5a45eb7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "' from IPython.display import Image\\n\\nImage(\"C:/Users/rhskr/Desktop/배치알고리즘_구현.png\") '"
            ]
          },
          "execution_count": 114,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "\"\"\" from IPython.display import Image\n",
        "\n",
        "Image(\"C:/Users/rhskr/Desktop/배치알고리즘_구현.png\") \"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yBUnuDMOdFgm"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTfzKh_nLMa7"
      },
      "source": [
        "## Batch Gradient\n",
        "하나의 배치 (X_set, y_set)에 대해 기울기를 구하는 코드를 작성해주세요"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "metadata": {
        "id": "Qby2_X1vLMa7"
      },
      "outputs": [],
      "source": [
        "def batch_gradient(X_set, y_set, parameters, model):\n",
        "    gradients = [0 for _ in range(len(parameters))]\n",
        "\n",
        "    for i in range(len(X_set)):\n",
        "        X = X_set.iloc[i,:]\n",
        "        y = y_set.iloc[i]\n",
        "        for j in range(len(parameters)):\n",
        "            gradients[j] += get_gradient_ij(X, y, parameters, j, model)\n",
        "\n",
        "    return gradients"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 116,
      "metadata": {
        "id": "rHxBS5RnLMa8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[60.34210563679011, 4.953948092549627, 33.731021348435895]"
            ]
          },
          "execution_count": 116,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "gradients1 = batch_gradient(X_train, y_train, parameters, 'logistic')\n",
        "gradients1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[90.46594619310353, 101.74369190908004, 117.84781916016405]"
            ]
          },
          "execution_count": 117,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "gradients1 = batch_gradient(X_train, y_train, parameters, 'linear')\n",
        "gradients1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "하나의 배치 데이터에 대한 전체 Gradient를 구하는 함수로써 나중에 이를 통해 구해진 Gradient를 바탕으로 parameter를 수정하고자 한다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQnlDboALMa8"
      },
      "source": [
        "## mini-batch\n",
        "인덱스로 미니 배치 나누기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {
        "id": "LgnfT6eHLMa8"
      },
      "outputs": [],
      "source": [
        "def batch_idx(X_train, batch_size):\n",
        "    N = len(X_train)\n",
        "    nb = (N // batch_size)+1 #number of batch\n",
        "    idx = np.array([i for i in range(N)])\n",
        "    idx_list = [idx[i*batch_size:(i+1)*batch_size] for i in range(nb) if len(idx[i*batch_size:(i+1)*batch_size]) != 0]\n",
        "    return idx_list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9S9fk1UTLMa8"
      },
      "source": [
        "batch_idx 함수에 대한 설명을 batch_size와 함께 간략하게 작성해주세요  \n",
        "#### data 크기 : N // batch_size : b 라고 하면 <br> 한 번의 epoch를 계산할 때, (N//b)+1의 파라미터에 대한 업데이트가 이뤄진다. <br>따라서 mini-batch는 파라미터를 한 번 업데이트 할 때 사용되는 data 양이라고 볼 수 있다. <br> idx_list는 하나의 mini-batch를 의미한다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4pMuZbkQLMa8"
      },
      "source": [
        "## Update Parameters\n",
        "기울기를 갱신하는 코드를 작성해주세요  \n",
        "(loss와 마찬가지로 기울기를 갱신할 때 배치 사이즈를 고려해 평균으로 갱신해주세요)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 119,
      "metadata": {
        "id": "loeL51rPLMa8"
      },
      "outputs": [],
      "source": [
        "def step(parameters, gradients, learning_rate, n): #n:현재 배치의 데이터 수\n",
        "    for i in range(len(parameters)):\n",
        "        gradients[i] *= learning_rate / n\n",
        "\n",
        "    parameters -= gradients\n",
        "    return parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 120,
      "metadata": {
        "id": "NLB2dUVTLMa8"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.87707524, 0.62838438, 0.2126239 ])"
            ]
          },
          "execution_count": 120,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "step(parameters, gradients1, 0.01, len(X_train))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX8RJFd_LMa9"
      },
      "source": [
        "## Gradient Descent\n",
        "위에서 작성한 함수들을 조합해서 경사하강법 함수를 완성해주세요\n",
        "\n",
        "- learning_rate: 학습률  \n",
        "- tolerance: Step이 너무 작아서 더 이상의 학습이 무의미할 때 학습을 멈추는 조건  \n",
        "- batch: 기울기를 1번 갱신할 때 사용하는 데이터셋  \n",
        "- epoch: 현재 반복 횟수   \n",
        "- num_epoch: 총 학습 횟수\n",
        "<br>\n",
        "\n",
        "BGD: 전체 학습 데이터를 한 번에 사용하여 모델의 파라미터를 업데이트하는 방법\n",
        "<br>\n",
        "SGD:  각 학습 데이터를 개별적으로 사용하여 모델의 파라미터를 업데이트하는 방법 (BGD 방법 개선)  \n",
        "<br>\n",
        "MGD: 미니 배치(mini-batch)라고 불리는 작은 데이터 묶음을 사용하여 모델의 파라미터를 업데이트하는 방법 (BGD, SGD 장점 혼합)\n",
        "\n",
        "<br>\n",
        "batch_size에 따른 경사하강법의 종류를 적어주세요  <br>\n",
        "batch_size=1 -> SGD  <br>\n",
        "batch_size=k -> MGD <br>\n",
        "batch_size=whole -> BGD <br>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 121,
      "metadata": {
        "id": "ZGbnVHbbLMa9"
      },
      "outputs": [],
      "source": [
        "def gradient_descent(X_train, y_train, learning_rate = 0.1, num_epoch = 1000, tolerance = 0.00001, model = 'logistic', batch_size = 16):\n",
        "    stopper = False\n",
        "\n",
        "    N = len(X_train.iloc[0])\n",
        "    parameters = np.random.rand(N)\n",
        "    loss_function = minus_log_cross_entropy_i if model == 'logistic' else mse_i\n",
        "    loss = 999\n",
        "    batch_idx_list = batch_idx(X_train, batch_size)\n",
        "    \n",
        "    # 에포크 수만큼 학습 진행\n",
        "    for epoch in range(num_epoch):\n",
        "        if stopper:\n",
        "            break\n",
        "        for idx in batch_idx_list:\n",
        "            # 한 번 학습할 때에 사용할 x와 y 데이터 \n",
        "            X_batch = X_train.iloc[idx,]\n",
        "            y_batch = y_train.iloc[idx]\n",
        "            # gradient 구하기\n",
        "            gradients = batch_gradient(X_batch, y_batch, parameters, model)\n",
        "            # 위에서 구한 gradient를 이용하여 parameter 수정\n",
        "            parameters = step(parameters, gradients, learning_rate, len(X_batch))\n",
        "            # 수정된 parameter를 이용하여 새로운 손실함수 구함\n",
        "            new_loss = batch_loss(X_batch, y_batch, parameters, loss_function, len(X_batch))\n",
        "\n",
        "            #중단 조건\n",
        "            if abs(new_loss - loss) < tolerance:\n",
        "                stopper = True\n",
        "                break\n",
        "            loss = new_loss\n",
        "\n",
        "        #100epoch마다 학습 상태 출력\n",
        "        if epoch%100 == 0: #출력이 길게 나오면 check point를 수정해도 됩니다.\n",
        "            print(f\"epoch: {epoch}  loss: {new_loss}  params: {parameters}  gradients: {gradients}\")\n",
        "\n",
        "    return parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3CTtc3eiLMa9"
      },
      "source": [
        "## Implement\n",
        "경사하강법 함수를 이용해 최적의 모수 찾아보세요. 학습을 진행할 때, Hyper Parameter를 바꿔가면서 학습시켜보세요."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KnUpYC7_LMa9"
      },
      "source": [
        "## Logistic Regression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 122,
      "metadata": {
        "id": "-LS6o3aeLMa-"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epoch: 0  loss: 1.0186683697413745  params: [0.43091983 0.49554429 0.91249408]  gradients: [0.029728496006239963, 0.010564953789861385, 0.03336615263972843]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epoch: 100  loss: 0.4713687198929431  params: [-0.81401913  0.74624696 -0.64980386]  gradients: [0.003670721542351625, -0.006568810375337964, 0.007404152383448334]\n",
            "epoch: 200  loss: 0.40033228132916066  params: [-1.02010588  1.31240779 -1.22964326]  gradients: [0.0012837555989009743, -0.004775812220477308, 0.004701366107822872]\n",
            "epoch: 300  loss: 0.36613707442449744  params: [-1.12513798  1.72244805 -1.63044954]  gradients: [0.0008996903291317559, -0.003540584988531893, 0.0034423747929487464]\n",
            "epoch: 400  loss: 0.34649053481014797  params: [-1.2062689   2.03478866 -1.93305702]  gradients: [0.0007375636432954181, -0.002769085846776855, 0.002673579470026575]\n",
            "epoch: 500  loss: 0.3340409950265245  params: [-1.27404183  2.2838991  -2.17289983]  gradients: [0.0006240937534479165, -0.002249403763264273, 0.0021597262313430847]\n",
            "epoch: 600  loss: 0.32561935841248124  params: [-1.33179424  2.48907768 -2.36943803]  gradients: [0.0005351995049643916, -0.0018768915397699333, 0.0017937229363102734]\n",
            "epoch: 700  loss: 0.3196514955182844  params: [-1.38158151  2.66201876 -2.53438878]  gradients: [0.0004637745053929422, -0.0015971393225352696, 0.0015203902165226264]\n",
            "epoch: 800  loss: 0.31527364411845166  params: [-1.42492457  2.81031464 -2.67531889]  gradients: [0.0004055919657346147, -0.0013795011181543615, 0.0013088020886152102]\n",
            "epoch: 900  loss: 0.31197542581520293  params: [-1.46298524  2.9391703  -2.79739098]  gradients: [0.0003575876343482314, -0.0012054883708724335, 0.0011403812402170624]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "array([-1.49634344,  3.0512477 , -2.90328021])"
            ]
          },
          "execution_count": 122,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#BGD\n",
        "new_param_bgd = gradient_descent(X_train, y_train, batch_size = X_train.shape[0])\n",
        "new_param_bgd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 123,
      "metadata": {
        "id": "x0H5tnauLMa-"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epoch: 0  loss: 0.27297244495481  params: [-0.87938437  1.21745366 -1.33877609]  gradients: [0.024702386234399343, 0.01344306221580425, 0.017380370254633702]\n",
            "epoch: 100  loss: 0.07736665391069168  params: [-1.93032631  4.17502134 -4.06769281]  gradients: [0.007538537650980447, 0.004102479400034251, 0.0053040452978619975]\n",
            "epoch: 200  loss: 0.07736266551655074  params: [-1.9303681   4.1751431  -4.06780374]  gradients: [0.007538159521331779, 0.004102273621517777, 0.005303779249341999]\n",
            "epoch: 300  loss: 0.0773626651836168  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489767101, 0.00410227360434025, 0.005303779227133383]\n",
            "epoch: 400  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n",
            "epoch: 500  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n",
            "epoch: 600  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n",
            "epoch: 700  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n",
            "epoch: 800  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n",
            "epoch: 900  loss: 0.07736266518359017  params: [-1.9303681   4.17514311 -4.06780375]  gradients: [0.007538159489764571, 0.004102273604338872, 0.005303779227131603]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "array([-1.9303681 ,  4.17514311, -4.06780375])"
            ]
          },
          "execution_count": 123,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#SGD\n",
        "new_param_sgd = gradient_descent(X_train, y_train, batch_size=1)\n",
        "new_param_sgd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 124,
      "metadata": {
        "id": "iGfXGoJaLMa-"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epoch: 0  loss: 0.9923705977394728  params: [0.46074355 0.213963   0.0560158 ]  gradients: [0.052663903541010686, 0.0469343687368357, 0.06133567111135318]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "array([-1.02347429,  1.2375899 , -1.21846971])"
            ]
          },
          "execution_count": 124,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "new_param_mgd = gradient_descent(X_train, y_train)\n",
        "new_param_mgd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "MGD loss가 제일 적은 것을 확인할 수 있다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0oCaZ0tLMa-"
      },
      "source": [
        "### Predict Label"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 125,
      "metadata": {
        "id": "syJE3oiNLMa-"
      },
      "outputs": [],
      "source": [
        "y_predict = []\n",
        "for i in range(len(y_test)):\n",
        "    p = logistic(X_test.iloc[i,:], new_param_bgd)\n",
        "    if p> 0.5 :\n",
        "        y_predict.append(1)\n",
        "    else :\n",
        "        y_predict.append(0)\n",
        "y_predict_random = []\n",
        "for i in range(len(y_test)):\n",
        "    p = logistic(X_test.iloc[i,:], random_parameters)\n",
        "    if p> 0.5 :\n",
        "        y_predict_random.append(1)\n",
        "    else :\n",
        "        y_predict_random.append(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pZKpFItfLMa-"
      },
      "source": [
        "### Confusion Matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 126,
      "metadata": {
        "id": "W4E1PgX5LMa-"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import *"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 127,
      "metadata": {
        "id": "-veTwxu4LMa-"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[38,  2],\n",
              "       [ 4,  6]], dtype=int64)"
            ]
          },
          "execution_count": 127,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tn, fp, fn, tp = confusion_matrix(y_test, y_predict).ravel()\n",
        "confusion_matrix(y_test, y_predict)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 128,
      "metadata": {
        "id": "h4_dW9rDLMa_"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "accuracy: 0.88\n"
          ]
        }
      ],
      "source": [
        "accuracy = (tp+tn) / (tp+fn+fp+tn)\n",
        "print(\"accuracy:\",accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XIgqa85aLMa_"
      },
      "source": [
        "## Linear regression\n",
        "### $y = 0.5 + 2.7x$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qYeIg9QNLMa_"
      },
      "source": [
        "### Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 129,
      "metadata": {
        "id": "nv8-yhszLMa_"
      },
      "outputs": [],
      "source": [
        "raw_X = np.random.rand(150)\n",
        "y = 2.7*raw_X + 0.5 + np.random.randn(150)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 130,
      "metadata": {
        "id": "07XtxLGWLMa_"
      },
      "outputs": [],
      "source": [
        "tmp = np.array([1 for _ in range(150)])\n",
        "X = np.vstack((tmp, raw_X)).T\n",
        "X = pd.DataFrame(X)\n",
        "y = pd.Series(y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6oENC02TLMa_"
      },
      "source": [
        "### Estimation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 131,
      "metadata": {
        "id": "fu578YrKLMa_"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.58225479, 2.7178229 ])"
            ]
          },
          "execution_count": 131,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#정규방정식\n",
        "theta = np.linalg.inv(np.dot(X.T,X)).dot(X.T).dot(y)\n",
        "theta"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 132,
      "metadata": {
        "id": "M74iqj4WLMa_"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epoch: 0  loss: 0.3956780913850176  params: [0.92968789 1.38930257]  gradients: [-0.05128588625262459, -0.05104829267694299]\n",
            "epoch: 100  loss: 0.12938433936786695  params: [0.55776501 2.75803927]  gradients: [-0.0019483573999728868, -0.005630539556421216]\n",
            "epoch: 200  loss: 0.12927754028655367  params: [0.55646813 2.76042703]  gradients: [-0.0019370965953091326, -0.0055949961551300005]\n",
            "epoch: 300  loss: 0.12927733069798356  params: [0.55646557 2.76043173]  gradients: [-0.0019370744247171334, -0.005594926176272239]\n",
            "epoch: 400  loss: 0.1292773302853423  params: [0.55646557 2.76043174]  gradients: [-0.0019370743810670536, -0.005594926038495908]\n",
            "epoch: 500  loss: 0.12927733028453003  params: [0.55646557 2.76043174]  gradients: [-0.001937074380981137, -0.005594926038224671]\n",
            "epoch: 600  loss: 0.12927733028452795  params: [0.55646557 2.76043174]  gradients: [-0.0019370743809808725, -0.005594926038223976]\n",
            "epoch: 700  loss: 0.12927733028452795  params: [0.55646557 2.76043174]  gradients: [-0.0019370743809808725, -0.005594926038223976]\n",
            "epoch: 800  loss: 0.12927733028452795  params: [0.55646557 2.76043174]  gradients: [-0.0019370743809808725, -0.005594926038223976]\n",
            "epoch: 900  loss: 0.12927733028452795  params: [0.55646557 2.76043174]  gradients: [-0.0019370743809808725, -0.005594926038223976]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "array([0.55646557, 2.76043174])"
            ]
          },
          "execution_count": 132,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "#경사하강법\n",
        "new_param = gradient_descent(X, y, model = 'linear')\n",
        "new_param"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 133,
      "metadata": {
        "id": "Ii3zBOwSLMa_"
      },
      "outputs": [],
      "source": [
        "y_hat_NE = theta.dot(X.T)\n",
        "y_hat_GD = new_param.dot(X.T)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oCVynFSPLMbA"
      },
      "source": [
        "### Visualization\n",
        "시각화를 통해 정규방정식과 경사하강법을 통한 선형회귀를 비교해보세요  \n",
        "(밑의 코드를 실행만 시키면 됩니다. 추가 코드 x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 134,
      "metadata": {
        "id": "UoEACrbYLMbA"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAABEaUlEQVR4nO3deXwU9fkH8M/shiQgSUC5QjcEIRGVwwMrIlVjik1FW7WtWKQKbRQEFASpSlVQVA6v4olEVqH1ADyw/hQRhaBVsVIgLYpiIglklUvEJIDk2H1+f2yz5NjsOfd+3q/X/pHN7M53Z2dnnvl+n+8ziogIiIiIiAzgMLoBRERElLgYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhkoxuQCg+nw/ffvst0tLSoCiK0c0hIiKiCIgIampq0LNnTzgcofs8TB2IfPvtt8jKyjK6GURERBSDyspKuFyukMuYOhBJS0sD4P8g6enpBreGiIiIIlFdXY2srKzAeTwUUwcijcMx6enpDESIiIgsJpK0CiarEhERkWEYiBAREZFhGIgQERGRYUydIxIJEUFDQwO8Xq/RTTENp9OJpKQkTnkmIiLTs3QgUldXh927d+PIkSNGN8V0OnTogMzMTCQnJxvdFCIiojZZNhDx+XwoLy+H0+lEz549kZyczB4A+HuI6urqsH//fpSXlyM3NzdsMRkiIiKjWDYQqaurg8/nQ1ZWFjp06GB0c0ylffv2aNeuHXbu3Im6ujqkpqYa3SQiIqKgLH+pzKv94LhdiIjICni2IiIiIsMwECEiIiLDMBAhIiKyII/Hg+LiYng8HqObEhcGIgYYO3YsFEXBvHnzmj3/+uuvB2b+rF+/HoqiBH3s2bPHiGYTEZFJuN1uZGdnIz8/H9nZ2XC73UY3KWYMRAySmpqK+fPn4+DBgyGX2759O3bv3t3s0a1bN51aSUREZuPxeDBu3Dj4fD4A/nIW48ePt2zPiGWn7wYjAhhV26xDByCaMibDhw9HWVkZ5s6diwceeKDN5bp164ZOnTrF30AiIrKF0tLSQBDSyOv1oqysDC6Xy6BWxc5WgciRI0DHjsas+9Ah4LjjIl/e6XRizpw5uPrqqzF58mRL7jxERKS/xkKVTYMRp9OJnJwcA1sVOw7NGOiKK67A6aefjlmzZrW5jMvlQseOHQOP/v3769hCIiIyG5fLhaKiIjidTgD+IGTRokWWvaC1VY9Ihw7+ngmj1h2L+fPnIz8/H9OnTw/6/3/+859IS0sL/N2uXbvYVkRERLZRWFiIgoIClJWVIScnx7JBCGCzQERRohseMYPzzz8fBQUFmDFjBsaOHdvq/yeeeCJzRIiIqBWXy2XpAKSRrQIRq5o3bx5OP/109OvXz+imEBER6UrzHJFvvvkGf/jDH3DCCSegffv2GDhwIP79739rvVpLGThwIEaPHo3HHnus1f/27duHPXv2NHvU19cb0EoiIiL1aRqIHDx4EMOGDUO7du3w9ttvY9u2bXj44YfRuXNnLVdrSbNnz241HQsA+vXrh8zMzGaPTZs2GdBCIiIi9Wk6NDN//nxkZWXhueeeCzx34oknarlKS1iyZEmr53r37o3a2trA33l5eRARHVtFRESkP017RN544w2cddZZuPLKK9GtWzecccYZeOaZZ9pcvra2FtXV1c0eREREZF+aBiI7duzAwoULkZubi3feeQcTJkzA5MmTsXTp0qDLz507FxkZGYFHVlaWls0jIiIigymiYf9/cnIyzjrrLHz88ceB5yZPnoyNGzdiw4YNrZavra1tNjxRXV2NrKwsVFVVIT09vdmyR48eRXl5OU488USkpqZq9REsi9uHiIiMUl1djYyMjKDn75Y07RHJzMzEqaee2uy5U045Bbt27Qq6fEpKCtLT05s9iIiIyL40DUSGDRuG7du3N3vuq6++QnZ2tparJSIiIovQNBCZOnUqPvnkE8yZMwdlZWV48cUXUVRUhEmTJmm5WiIiIrIITQORn/70p1i5ciVeeuklDBgwAPfeey8WLFiA0aNHa7laIiIisgjNS7xfeumluPTSS7VeDREREVmQ5iXeiYiIiNrCQMQge/bswZQpU5CTk4PU1FR0794dw4YNw8KFC3HkyBEA/mqriqJAURS0b98evXv3xsiRI7Fu3TqDW09ERKQOBiIG2LFjB8444wysWbMGc+bMwZYtW7BhwwbceuutePPNN/Hee+8Flp09ezZ2796N7du3429/+xs6deqE4cOH4/777zfwExAREalD8xwRam3ixIlISkrCv//9bxx33HGB5/v06YPLLrus2T1m0tLS0KNHDwBAr169cP755yMzMxMzZ87E7373O/Tr10/39hMREanFXj0iIsDhw8Y8IixQe+DAAaxZswaTJk1qFoQ0pShKyPeYMmUKRAT/+Mc/ot5EREREZmKvHpEjR4COHY1Z96FDQBuBRVNlZWUQkVY9GV26dMHRo0cBAJMmTcL8+fPbfI/jjz8e3bp1Q0VFRVxNJiIiMpq9ekQs7NNPP0VJSQn69+/f7H47bRGRsD0nREREZmevHpEOHfw9E0atOwI5OTlQFKVV6fs+ffoAANq3bx/2PQ4cOID9+/fjxBNPjL6dREREJmKvQERRIhoeMdIJJ5yAiy66CE888QRuuummNvNEQnn00UfhcDhw+eWXq99AIiIiHdkrELGIp556CsOGDcNZZ52Fu+++G4MGDYLD4cDGjRvx5ZdfYvDgwYFla2pqsGfPHtTX16O8vBzPP/88Fi9ejLlz5yInJ8fAT0FERBQ/BiIG6Nu3L7Zs2YI5c+ZgxowZ8Hg8SElJwamnnorp06dj4sSJgWVnzpyJmTNnIjk5GT169MA555yDtWvX4sILLzTwExAREamDgYhBMjMz8fjjj+Pxxx9vcxnOiiEiIrvjrBkiIiIyDAMRIiIiMgwDESIiIjIMAxEiIiIyDAMRIiIiMozlAxGJ8GZziYbbhYiIrMCygUi7du0AAEeOHDG4JebUuF0atxMREZEZWbaOiNPpRKdOnbBv3z4AQIcOHXgTOPh7Qo4cOYJ9+/ahU6dOcDqdRjeJiIioTZYNRACgR48eABAIRuiYTp06BbYPERGRWVk6EFEUBZmZmejWrRvq6+uNbo5ptGvXjj0hRERkCZYORBo5nU6eeImIiCzIssmqREREZH0MRIiIiMgwDESIiEzA4/GguLgYHo/H6KYQ6YqBCBGRwdxuN7Kzs5Gfn4/s7Gy43W6jm0SkG0VMXIKzuroaGRkZqKqqQnp6utHNISJSncfjQXZ2Nnw+X+A5p9OJiooKuFwuA1tGFLtozt/sESEiMlBpaWmzIAQAvF4vysrKDGoRkb4YiBARGSg3NxcOR/NDsdPpRE5OjkEtItIXAxEiIgO5XC4UFRUFaiE5nU4sWrSIwzKUMJgjQkRkAh6PB2VlZcjJyWEQQpYXzfnbFpVViYiszuVyMQChhMShGSIiIjIMAxEiIiIyDAMRIiIiMoymgcjdd98NRVGaPU4++WQtV0lEREQWonmyav/+/fHee+8dW2ES82OJiIjIT/OoICkpCT169NB6NURERGRBmueIlJaWomfPnujTpw9Gjx6NXbt2tblsbW0tqqurmz2IiIjIvjQNRIYMGYIlS5Zg9erVWLhwIcrLy3HeeeehpqYm6PJz585FRkZG4JGVlaVl84iIiMhgulZW/eGHH5CdnY1HHnkEhYWFrf5fW1uL2trawN/V1dXIyspiZVUiIiILMW1l1U6dOuGkk05q866SKSkpSElJ0bNJREREZCBd64gcOnQIX3/9NTIzM/VcLREREZmUpoHI9OnT8f7776OiogIff/wxrrjiCjidTowaNUrL1RIREZFFaDo04/F4MGrUKBw4cABdu3bFz372M3zyySfo2rWrlqslIiIii9A0EFm2bJmWb09EREQWx3vNEBERkWEYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhGIgQERGRYRiIEBERkWEYiBAREZFhGIgQERGRYRiIEBERWZDH40FxcTE8Ho/RTYkLAxEiIiKLcbvdyM7ORn5+PrKzs+F2u41uUswUERGjG9GW6upqZGRkoKqqCunp6UY3h4iIyHAejwfZ2dnw+XyB55xOJyoqKuByuQxs2THRnL/ZI0JERGQhpaWlzYIQAPB6vSgrKzOoRfFhIEJERGQhubm5cDian76dTidycnIMalF8GIgQERFZiMvlQlFREZxOJwB/ELJo0SLTDMtEizkiREREFuTxeFBWVoacnBzTBSHRnL+TdGoTERERqcjlcpkuAIkFh2aIiIhsyCp1RhiIEBER2YyV6owwR4SISCcejwelpaXIzc21RZc6mZMZ6oywjggRkclY6QqVrM1qdUYYiBARaczj8WDcuHGBk4PP58P48eNNP3ZP1mS1OiMMRIiINGa1K1SyNqvVGeH0XSIijTVeobYcszfrFSpZX2FhIQoKCkxbZ6Qp9ogQEWnMaleoZA8ulwt5eXmm3884a4aISCfRVMLkDBuyMs6aISIyoUivUDnDhhIJe0SIiEzEDDUgrIA9RubGHhEiIoviDJvw2GNkLwxEiIhMxGo1IPTGmiz2o1sgMm/ePCiKgptvvlmvVRIRWQ5n2ITGHiP70aWOyMaNG7Fo0SIMGjRIj9UREVmalWpA6I01WexH8x6RQ4cOYfTo0XjmmWfQuXNnrVdHRGQLVqkBoTf2GNmP5oHIpEmTcMkll2D48OFhl62trUV1dXWzBxERUVOFhYWoqKhAcXExKioqUFhYaHSTKA6aDs0sW7YMmzdvxsaNGyNafu7cubjnnnu0bBIREdmAy+ViL4hNaNYjUllZiSlTpuCFF15AampqRK+ZMWMGqqqqAo/KykqtmkdEREQmoFlBs9dffx1XXHFFYBwP8Gc2K4oCh8OB2traZv8LhgXNiIiIrCea87dmQzM///nPsXXr1mbP/fGPf8TJJ5+M2267LWwQQkRERPanWSCSlpaGAQMGNHvuuOOOwwknnNDqeSIiIithiXn1sLIqERFRFFhiXl286R0REVGEeFPCyPCmd0RERBpgiXn1MRAhIiKKEG9KqD4GIkRERBFiiXn1MUeEiIgoSh6PhzclDMEUdUSIiIjsyuwl5q00vZhDM0RERDZitenFHJohIiKyCbNML+b0XSIiogRkxenFDESIiIhsworTixmIEBERBeHxeFBcXAyPx2OZ97bi9GIGIkRERC1omfCpdTJpYWEhKioqUFxcjIqKChQWFra5rJbBVqSYrEpERNSElgmfZkkmBfwB0bhx4+Dz+eBwOFBUVBQyaIkGk1WJiKJkhitDMgctEz7Nkkzq8XgCQQgA+Hw+jB8/3pD9n4EIEdlWpMGF1eoukLa0TPg0SzKpWQIigIEIEdlUpMGFma4MyRy0TPg0SzKpWQIigDkiRGRD0YzDFxcXIz8/v9V7FBcXIy8vT+umkolpeT8ZM9yrxu12Y/z48fB6vYGAyIgcEd5rhohsJ1S3c8uDfuOVYcugxcx1F0gfWt5Pxgz3qiksLERBQYHhARGHZojIdqLpdjZLVzmREVwuF/Ly8gzd3xmIEJHtRBtcRFN3gYjUxRwRIrItM4zDEyUi5ogQEcEc4/CkLo/Hg9LSUuTm5vK7tQkOzRARkSXEU++FBevMi4EIESUcnpSsJ556LyxYZ24MRIgoofCkZC2NQePHH38cUyVQFqwzPwYiRJQweFKylqZB46hRo6AoSrP/R1LvxUylzE2nuhpwu4EDBwxtBgMRohixe996eFKyjmBBo6IogfowkdZ7MVMpc1NYuxbo3x9QFCAjA7juOqBLF0ObxECEKAbs3rcmnpSsI1jQ6PP5sGzZsqjqvSR8wbqDB4Fp0/yBh6IAw4cD27Y1W6QI1xvUOD/WESGKUjT3MSHz0fL+GqQetX9nCVVTZvVqYMoU4Kuv2lzkIDphMh7DCxiNF150YNQodZsQzfmbPSJEUWL3vrXZoYpqIgwLqt2TYYZS5po5cAC46aZjvR4XXxw0CHkeo9ELO6FA8MTsg3j60DXwifpBSLTYI0IUJfaIkJHcbncgd8LhcKCoqMiSwVSkEqonI1IiwJtvApMnAxUVbS62D10xBY9iGX6Piy5S8NBDwKBB+jSRPSJEGtJizDkRrnApfok468fWPRnR2L8fmDDB3+PhcAC//nXQIOQ5jEVPfIPUFMFrC/fh7/WjIKJgzRr9gpBoMRAhioGa3ftMfNWOXgGeXusx87Agg2mViQArVwJZWf7go1s34OmnWy32DXrit3gFCny4epTgwvLn8K30xNGjwA03AElWuJGLmFhVVZUAkKqqKqObQqSJyspKcTgcAiDwcDqdUllZaXTTLG/x4sWBbetwOGTx4sWar0dRFJk+fbpm359Z9xe9trXt7d4t8qc/ifjDkDYfT2OcdMMecblEXn5ZxOczuuGtRXP+ZiBCFIfKykpZt25dzCeCdevWNTupND6Ki4vVbWiC0euEHWw9egQ+Tqcz8JmiXU+8+2yw9zNjcGQJPp/IsmUi3buHDDwq0Et+jdcF8MnkySL79xvd8PCiOX9zaIYoRmoMqbCuhTb0GsIIth5A29yNeIYFtRgGNPNwkSl5PMC11x7L9fj974G9e1st9iQmoiv24cwzBOXFO/EPuQwiCh591PD6Y+rTITCKGXtEyKzUvAqM9wqXWjO6R6TxYaaeLa22CXtEwvB6RZ5/XuT440P2enyFHPklVgkgcu+9IocPG93w+JimR2ThwoUYNGgQ0tPTkZ6ejqFDh+Ltt9/WcpVEulDzKtAOdS30EE0ypF7VNBvX07JXq3GdLXu2jEzo1KrnIuErlwazaxcwapS/18PpBP7wB+D771sttgBTcDwO4BcXCX78TyneloshAtx5J9ChgwHtNoqWEdEbb7whb731lnz11Veyfft2+ctf/iLt2rWTzz77LKLXs0eEzIpXgfqKNRmysrJSiouLNf9eKisrZfr06SF7toxO6NR6n9VrW5uS1yvy7LPiS0sL2euxDSfLRXhHUlNFFi4Uqa83uuHaMXWyaufOnSP+ATIQITPjkIo+rBT0tXUyjuUzqJ1UKsJ9VlU7doj87ndhZ7g8gOmSgYNy9dUi5eVGN1o/pgxEGhoa5KWXXpLk5GT5/PPPI3oNAxEyu4S+CtSJHWYWRfsZtOw94T4bo4YGkaIi8aWmhgw8SjBI8rBOsrJEXnnFnFNr9RDN+VvzEu9bt27F0KFDcfToUXTs2BEvvvgiRowYEXTZ2tpa1NbWBv6urq5GVlYWS7wTJTA7lNSP5jPY4fPaRmkpMH068MYbIRebgxmYh9vxpynpuOsu4IQTdGqfiZmqxHu/fv1QUlKCf/3rX5gwYQLGjBmDbS1uQdxo7ty5yMjICDyysrK0bh4RmZwdSupH8xk4HTY2qnynDQ3Ak09CkpL8iaYnnRQ0CNmEM3EePkBuTjVeXrEPf5E5qJZ0LFjAICQmmvfPtPDzn/9cxo0bF/R/R48elaqqqsCjsrKSQzNEUdIit8AM1BpSMDJpNJLPoHZOjF33h6bi+k6/+EJkxIiwuR53Y6Ychxq5916RJ598jpVkwzBljkijCy+8UMaMGRPRsswRIYqO0TMzzM4qia9qJZUmwv4Q9XdaVyeyYEHYwONjnCNDsEF+8QuR//wnjvUlKNMEIrfffru8//77Ul5eLv/973/l9ttvF0VRZM2aNRG9noEIUeR4gAzPSomv8fYAJcr+ENF3+t//im/48LDBxx24V45PPSxPP9321Fqr7ENG94RFc/7W9L58+/btw7XXXovdu3cjIyMDgwYNwjvvvIOLLrpIy9USJaRQuQVMcvRrLKnfMhHUjCX1XS5XXN9bouwPwb7T9g4HTn/vPeDCCwPPKUFe+0/8DFPxV/S7+izMmQPclw3cF8P6zLYPud1ujBs3Dj6fDw6HA0VFReYulKhDYBQz9ogQRS5RroDjlSi1NBJpf1i8eLGc6XDIujA9HgLIrZgnOa4f5dVXY59aa+Z9yCzfu2mGZuLFQIQoOmY+QJpJotTSsPX+8OOPInPmhA081uJCOQ1bZMoUke++U2/1Zt2HzDJ0ZKo6IvGIZh4yEfl5PB6UlZUhJyfHVl3wFBtb7Q8bN8I7ZSqcGz4KudgteAgfnX4j5i9IwQUX6NQ2+Ld1aWkpcnNzDdvWZqlDY6o6IkSkL5fLhby8POufdEgVlt4fjhwBZs/21/RQFODss4MGIe/gFxiArbj/PsGRw4KH5RZ8skXfIMTtdiM7Oxv5+fnIzs6G2+3Wb+VNWPEmhOwRISIi8/jkEzRMmoKkzZ+GXGwyHsXXF03AvIfbYeBAndrWBrP0QrRsk5E9YewRISIiazh0CHLnXcd6PYYODRqEvIlLcEbKNhQtEjTUCx6TyXhrjfFBCBB5NVw9K/qG6gnTu7JwOAxEiIhMymwnDNX885+oG3CmP/BIS4Nyf+tJs/VIwgQ8hWtH1WPXTsGl8ia2HD0F48YBSZoWnohe45TeplpO6TXL0I1Z2tGMxomzceGsmcRidAEeIjOxVVXUqirx/vm2sDNcXsPlcl6Pr+KaWmuUUDOUzDKlVs92RHP+Zo8ImYIpo3SyJDv0Ing8nkBBKgDw+XwYP368tT5TcTGO5g7093pkZMDx4PxWixxGBxRiMaZNbsCB7wRXyEp8sDsXv/mN/2VWUlhYiIqKChQXF6OioqJZATGz3MjQLO1oiYEIGc4WB12KSSRBQzSBhRYBrRGBjVlPGCH98APqJ99yLNcjPx+pZZ+1Wuxl/A6XDfgaH7wvOE4Owy2FeORRpy3uWttWXkYkQzd6MEs7WlG9P0ZFHJpJDGYpwEP6imToIZrhCS26nY0aHjFLV35Yq1fL4ax+IYdbfkC6XIslct9srxw+bHSDjWOW4nJ6tYOVVclSLHPQJdVE8p1Hu1+oHdAavV+a5cTVzIED8uO4yWFzPZ7H1fKH83fKf/9rdIPNxSzVWPVoh2luekcUicYCPOPHj4fX67VEAR6KTyQ3ZIv2pm1q34zM6JvGFRYWoqCgwNiqqCKQt1bh8HWT0XHvDgBAapDF9qML/tzuUZz72O/xp+scGJ0EjNa3pZYQ740M7daORgxEyBRMcdAl3UQSNEQbWKgd0JrhLquGnDD270fN1JlIe+FpAP671nYMsthSXIt/XzEHf17wE/TqBSzRs41kK0xWJdOwdClqikokZahjKVUdauaCFm20BRF4X30dNSdk+5NMu3ULBCFN7UYPTOiyAq+94oP4BGNkKR5/zR+EaMUOM6AoPJZ4JyLDRFKG2uhS1UavXxN79uD7SXfh+NcWh1zsGVyHyuvvxZS5PXSf1eJ2uwOz6RwOB4qKiuIKLElf0Zy/GYgQkW2Z4W6opiCC2hdeQd2EyUg7tKfNxXaiFx7r+yguc1+G8y8wrpCHGe/dQtHhvWaIKOElfJG8b77Bnl+O8Q+3OBxIuWZk0CBkIW7Aghl78eMRQbbsxMNllxsahAAWraNCMWOPCFGceNVtPma9otZ0XxFBTdFLUG6ego5Hv2tzsa/RB0vPfAwjl4zAgIHmLF+q1vfH36Zx2CNCpJOWV90PPvggk+tMwIxX1Fr00MiuSuw6b3Sg1yPthtFBg5CnkiZj6cPfoaFe0Fe+xuxNl4QNQoxMFFUjUfihhx5K7B4xK9GsmokKzFTQjDdko5aCFbxqfFj+JmUWZ3QxMs3a4/XK/oeWyOF26SELim3DyTI/f7Xs3Blbe81yw71YC2898MADrX6TLJKoL1ZWVZlZfpRkLm1V8uSBzxzMVJk0nqqv9V/tkLIzfxe2mmlR2jT5v799H/dda80WxEWrsrJSFEXhbSMMxrvvqog3ZKO2BLuBVFNqDgWwnkL01KwpEq+objbm9cJz92IcdR4HKAqSTuqDvptfabXYVgzAU79di+8P+EOR66sfxqXXdI77rrVmHNaKRmlpKSRI6qPD4QhbjI6/M2MwEAnD6j9K0k7LceyW1KrCmfCzP+JgliJ54XIeftxahi9PvcKf65GUBNc91yPVd6TV+zzX43Z8tKoKEMFA2YqJr+Tj+OPVbatp79D6P+GChbYuEObPnx9yP+DvzEDad9DEzgxDM1bvpiTtNY5jP/jgg6oPBXD/s5dAzkN5uWy/+SmpQ1LI4ZZNOEP+dt37cuSIvu0007BWU5EOkzdtv8PhkAceeCDk+/J3pj7miKjMrD9KMh+172qp9h1lE4nZEsy/3/CFfJY9Imyuxwt975TP/1VjdHNNc6fYRtEGC9G0n78z9TEQ0YDZfpRGMdvB3e54pRYbMySY+2rrZPOfHgsbeHzqOFtev+1jqa/XvYm6ivfYoWWwwN+Z+hiIkCbMcHBPROyRi46RJ5Vv1nwm/+nxi7DBxyun3SOV2w9r3h6zUOPYofX3yt+ZuhiIkOp4xWCsROqRM/OVc0v1h2vlo988FL7XI2WYFD/wadxTa61IzWOH1sFCIv3OtBbN+TspjjxXSiChZg8ZPSMhEbhcroTYzmrccbVx1kTL8uBqzfrYvrwERyZMwxkHi5EE4Nw2lntr2BwMXX4zjv9Je/xUlTVbk5rHjsLCQhQUFGh2N+RE+Z2ZDafvUkTMPqWPrE+tmj1qlAdv6sj3R7H2onn+qbWKgn6/PwNnHCxutdzG4y7A5sWbA/0gl3w4A8f/pH1M67QTtY8dZpmSTephIEIRUfvgTtSSmjV74i1mtnHRJmzueB6gKOhwQnv8/L0ZQZdbOmAyUtEeCoBzfvwntmBz1G2NltWKbml17LDadqAQdBgqihlzRMyHY6ikFSPzkL7bdVjeHDI7bK7HphMuktLX/mtYe62cMK7mscPK2yFRMFmViAwVa8KpXjMXfD6R9+7/RDa1GxI2+Pj4qgXScKS21Xu0lRQ7ffp0TdqsVuBj9Sn4TJy3Bt5rhogME0+pbC3vD7Nz22G8OmAWoChQHAp+fsc5OLP+X62WK/nJCOxeuy0QigxdNgXO9smtluvYsWPQ9Tz88MOaDBeoMXRlhzLmvO2G/SgiQe4OZBLV1dXIyMhAVVUV0tPTjW4OEYXh8XiQnZ3dasZKRUWF7vlEDQ3AG7d+iN4LpuBMaTt3owFOfH79oxj0xDgoye0ifv/i4mLk5+e3+b+8vLxomxxSvNvWTN9NPOzyOewumvM3e0SISDVGX61u/bgGL/We4b9rbTsFv/nreUGDkM/6/ho//Gs7IIIkacBpRZOiCkKAtm+uFsldXmMRb9Kn0d+NWpg4bz+aBiJz587FT3/6U6SlpaFbt264/PLLsX37di1XSUQG0nua9+HDwNI/rsd/lNMARcHAYekYtXNeq+WOKqko/XORv5tEBAPK/oFOZ58U17obT4hNP6+iKCgqKtLspBjP0JWdpuBrOYRHBtAyWaWgoECee+45+eyzz6SkpERGjBghvXr1kkOHDkX0eiarxs/qiWnh2P3zWZHWCafvv/GDLOk6PWyS6cvoJyciWfNZFZWVlbJixQpZsWJFTPuhnvswy5iTXkw7a2bfvn0CQN5///2IlmcgEh+7T3Gz++ezMjWnau7fL/LEZWtkG04OGXgcdqaJ577npHLnTsvMqjBiH+YUfNJDNOdvXZNVy8rKkJubi61bt2LAgAFhl2eyauzsntAV6efzeDwoLS1Fbm6uJp9b6/dPRCLAP577Ht9NvgfXHX4s5LJf//T36L1sHpx9sgPPtZVEqkUCaTzs/hulxGbKZFWfz4ebb74Zw4YNazMIqa2tRXV1dbMHxcYuiWltieTzaT1V0Q5TIc2iogKYd8HbKFVyoTgUXF54QtAg5FDy8Tjw6POA1wuIoO+nLzULQgDr5ELY/TdKFDHN+2f+54YbbpDs7OyQ3YGzZs0KWiCIQzPRs3vRn3CfT+vPH+v7M6fFr75exD1/vzzlmBg212PnhdeKb5c5C6PFI9Q+FOl+wv2JzMp0OSKTJk0Sl8slO3bsCLnc0aNHpaqqKvCorKxkIBIHKxyM4xHq82l9K/hY3j/Rc1pKtvjkrkGvSwV6hQw8qjp0l5rFy/zlT+NghVyIYPtwpPtJou9PZG6myREREdx0001YuXIl1q9fj9zc3KhezxyR+Hk8Hs1umW0GbX0+rcffo33/RMwHOHwYePqevUh78C6MwzMhl91zyZ/Q45n7gMxMnVpnHk33YQAR5z4l0v7EXCzrier8rWVENGHCBMnIyJD169fL7t27A48jR45E9HrOmqF4aN0jFM37a91DYxbr1vrkluyXxYOeoXs9Mlxy9KXX4u71sJtI95NY9ycrDuWw58eaTDM0E+yHAkCee+65iF7PQITipXX3fKTvb9ecnf37Rf4y9ht5FmPD5nocuHK8yN69RjfZ1CLdT2LZn6x4Qrfr7yYRmCYQiZcZAxErXlGQOdghZ8fnE1m+zCcTOr0o+9AlZOBR3aW3NKx8g70eUYp0P4lmf7LqCT1RehLtyDQ5IvEyW46I2+3GuHHj4PP54HA4UFRUpFppYY6BJgYr5uxUVAAPTq7E0P+bgT/ghZDLHhp7Izo+OAvo0kWfxkXBCr+xxjZ27NgRhw8fDrufRLo/WaW2SkuJlgtjJ6bJEYmXmXpEtLyisGKXKdlDsB6+ujqRJx7zyljn3+R7dArZ61HT8yTxrXrbwE8QGSv8xrRso1V7RETs0ZPYVKL0qnNoRgNadRFa+QChtkT5gZpF0xOfopwuP++7QZZhZNhcjx8nTBX5/nujmx8xK/zG9GijlU/oVpiKHQkrBMRqYSCiAa0OFBwD9UukH2gs1A7Stm/3iBN3yZ+wWGpwXMjA49CJ/UXee0+V9Rph+fLlpv+N6XUcsMsJ3YqsEBCriYGIRrS4oki0nTMYboPQ1ArS1q4VGdGvTF7FFWF7Peqm3Sryww8qfxL9Nd12Zt6/rPQbYM9lbBLtopOBiIa0uKKwcpepGhLtB9pSqAN7PCeo/ftFbppQL+PwtBxFcsjAYzNS5EKHw1Ynl2Dbzsw9blY4DrDnMnZWCjbVwEDEgozsMjX6CifRfqBNhTuwRxOk+Xwiy5eL/Kzrl/IGLg3b67F5xCWS8b91m/XEF4+2tt2KFSuMblqbzDx0ksi/U7VYIdhUCwMRiphZrnCs/gONJZiL5MAebpkdO0SuvrJObsRj4oUSMvCoHXSWyIcftmqDWU988eKJU12J3nOpFjv/5ppiIEIRMduB2qo/0FiDuUgP7E2DNIcjRa6++iMZlPS5rMIvw/Z6+GbdLXLokAaf2hqsHuCaidmOF2RuDEQoIrzCiV88B+dIX/v22yKpOCLLcWXYwKP+7HNF/vUvrT6uJVk1wDUjBnYUqWjO30mghJWbmwuHw9GqamHjXUApvNLS0mbbDwC8Xi/KysoC/2+rkqfL5UJRURHGjx8Pr9cLp9OJRYsW4bjjXMjLA9q9/y7exS/wSwA/hmrEffcB06YB7duDP+jWXC4Xq3CqpLCwEAUFBZarDkzm5jC6AWScxhOh0+kEgMCJkAeXyDUGc005nU5s3LgR2dnZyM/PR3Z2Ntxud9DXFxYWoqKiAtOnf4EUbxW6XPcGOh+vYP37Ct7FL4K+xjvsPGDTpmP9IHfcAbRvr/pnIwrG5XIhLy+PxwlSDe81Q5a8/4mZuN3uZr0a8+bNw2233Rby/hg7dwIXXgicXL4Kq3BJ+JVcdhnwwgvAccdp9TGIiFQTzfmbgQiRCpoGc6WlpUFuMKbgmmt2YOXfT8Ar+B0KsCb8m65ZA1x0kSbtJSLSUjTn74QdUrbCnTjJOlrmIfhzb04D8G9cjn9gJX4D/P3E0G8yciSwZAmHWShuPL6RlSRkjojb7Y5o/N6qPB4PiouL4fF4jG5KQqmtBa69FhiQ1RFrfedBsAUCpz8Iacv69cdyPZYvZxBCcbPL8Y3HsQSi8QyeuGgxfdfuc+HNUqAsUaxa5Y8iRmJZ2Km1Aohce63I0aNGN5sMplU1Y7sc33gcs75ozt8J1yMSbrqllXk8HowbNy7w+Xw+H8aPH88rChUdPAjk5QEnKAfwiXIOLh6hQKBgOX7f9os++uhYKLJ0KZCSolt7Y8ErUW1p2WNhh+Mbj2OJJ+ECkbamW9qhdoYdDkJmtHgxoCjAGGVpYGrtAXTBOfhX8Bdcfz1QV3cs+Dj3XH0bHAe7dOubldYnWTsc3/Q8jjHoNoeEC0TsXDvDDgchM6ioAHr3Brop+7BZORPXXe/v9ViKsW2/6NNPjwUeRUVAu3Y6tVY94U6SiXbQ1uLzan2StcPxTa/jGINuE9FhqChmWpZ4t2vZZ5Zgjp7XK3LHHf4o4joURZbrceONIvX1RjddVaFK/ifamL1Wn1evHA6rH9+0Po7ZJZfGzHivmQRn9YOQHjZt8scTPfCtbMPJ4QMPp1Nkyxajm62ptg7On376aUIdtLU+SfFiITJaHsd4ny3tMVk1wZmpBLNZuvMbp9YqCnCj8gTOHOwfbtmNnjgFXwZ/0S23AA0N/lCkoQE4/XRd26y3trr1Dx06lFC5R1oPnzSW9S8uLkZFRQUKCwtVeV+7aes4psYxhcPYJqNDYBQzM/eIaDX9zk6M7s5vnFrrwi4pQ5/wvR4dOoh89pmubTSjlleiidaNnWif10rUPKawZ0pbHJrRmNEnWCsw4mD+/fci558vAvjkFjwYWa7HnXf6k0QopEQ7aCfa57WCaI8pkVwschhbOwxENMSrpcjoNQb77rsiAwaI9MYO+QaZ4QOPzp1Ftm9XtQ2JItEO2on2ec0ummNKIl0smrV3njkiGmKtjshoNQa7fz8wcSKgKILfKq/i5Itc2PqZgnL0QU/sDv6i2bMBn88finz/PXDSSXG1IVGZKfdID4n2ec0u0mNKIhVEs8sUZAYiUWKSU2TUqmcgAixbBvToAWQqu/FGt0I8tVCBwIFX8Tu48E2r13wLYPeHHx7rB7nrLn+WKhFZVqTHlES5WLRTwMVAJEp2KBgUjlozXWKdHbBjh/9GtIoi+L1jOS4c1R179vpnuBTi2VbLlwO4DIDyv8dPAGyvr4+r7YnELDObiMKJ5JiSKBeLtgq4dBgqipkZc0Qa2XX82Iix1bo6kccf95fq+Akq5W/4Q/hcj4kTRfbvD5qz43A4bPe9aCWRxtIpcSRCsrHZ8xWZrEox0XPH3rRJJC9PRIFXRuPv8h2ODxl4fAXI6ilTgr7X4sWLRVGUQJsVRbHlgUdtZj+QEcXDrheLTZk54GKyaoJRq2tdy66+Q4eAWbP8qRrZyk5sHzwKxesV+ODE87gGJ+D7Vq9ZAOB4+IdbTgJwyRNPBP2MBQUFUJrkgIiIqcZKzTr0YauuXaIWEiHZ2C7F8RiIWFDTE5uaWdPBxlYBYOPGjTG933vvAQMGAA7Fh5vSnsP02WkQKNiJ3hiFZa1fcMopwJo1gAiK163DVAAHm/y7rZOkmU+oZs5qT5SxdCI7s0XApX0HTew4NNNayzH9pkMSUKFr/cEHH2w1Tz/S99y7V2TCBP9oyon4Wl7Bb8Lnevz5zyI//NDqvaIZNjDrEINZ29WUmbt2ici6ODRjU8Gma4lIs2Xi7QkYPHhwq+faek+fD3jpJf/UWqfixZ3di/DwwvYQKNiBvvgtXmu9gkGDgOLiY6HIAw8AGRmtFotmdpJZZzKZuaemkV26donIuhRpeSYzkerqamRkZKCqqgrp6elGN8dwxcXFyM/PD7mM0+lERUVFzCdhj8eD7OzsZifQpu/59dfA7bcDr7wC5KAUD2E6LsMbId9zDgDXE0/g2kmTYmpPWVkZcnJywn6maJbVQ7htSURkV9Gcv9kjYiHBxvQdDkfgOTV6Alr2LjgcKRg58p/o3duFJKUBj+Q8iWWvOCFQUIqTggYhdYMG4TxFCdT1uAPAn6ZMiSlZM5rxT7ONlZq1p6YtZk2qJSJ70zQQ+eCDD/CrX/0KPXv2hKIoeP3117Vcne0FO7EVFRVh586dqnatn3FGIc4++xAAwUm+Lbjmpdlo8CpoQDs8iRvhhK/1i2bO9E+NEcFHCxbgQ5WHjKzKKkMfZk6qJSJ703Ro5u2338ZHH32EwYMH4ze/+Q1WrlyJyy+/POLXc2gmOLWHIA4d8qdq3Hsv0A51mIinsABTQ7/onHOABQuAIUOCto9DEtbB74uI1BbN+TtJy4ZcfPHFuPjii7VcRUJyuVxxnyDefReYOhX4/HNgALbiEUyD4L3QL7rvPmDaNKB9+7DtKyoqwvjx4+H1ek0/JJHoQiXVWuE783g8KC0tRW5uriXaS0TNaRqIRKu2tha1tbWBv6urqw1sjb3s2wfcfTewcCGQjFpMxmP4DLeGftF55wF//SsQZCZNOIWFhSgoKDBV8igF15h71LJHxAr1RNxud2AmmcPhQFFRkWmHv4goOFMlq86dOxcZGRmBR1ZWltFNsiyfD3jxRaB7d38104LuWzByYR4ECmqRigfbCkLmzQN+/NE/tfaDD2IKQhqZLXmUgrNaUm0jO919lCiRmSoQmTFjBqqqqgKPyspKo5tkKV9/DVx5pT/w6OA8iq2j52LvPgUCBVtwJvLwfusX5ecDJSXH6nrcdhuQmqp728lYBQUFePzxxzFz5kxs2LDBEr0KVqjTEgxnJxE1Z6pAJCUlBenp6c0e1Lb6euCxxwCn0x98/D5nI25+ZRgECo6iPebiL8Ff+PDDQG2tP/BYuxY47TR9G06qi+fk5na70atXL0ycOBGzZ8/GkCFDLDFrxool6jk7iag1UwUiFN7mzUBenj/wyEg+ggNT7oHX5+/12IizMQwft35RQQHw2WfHej2mTQOSk3VvO8WnrWAjnpObx+PB9ddf36xCr5jspoFtsdqQEoeSiILTNBA5dOgQSkpKUFJSAgAoLy9HSUkJdu3apeVqbeXQIX+JDkXxPyYN3oD57w+BQMERHId7cHfrFymKv6ukrs4feKxeDfTvr3vbST1tBRvxntxKS0tb3SYAsMYQB2CdOi2AdYeSiDSn4T1vpLi4uNUN1ADImDFjInp9ot70bs0akf79/d0Xx6FGZuPO8DePu/RSkS+/NLrppIFQN89bt25d0N9YcXFxxO/d8saJTd+f1GOFmyASqcU0N73Ly8uDiLR6LFmyRMvVWs6+fcCECcd6Pe79xQf4++dnQKDgENJwF+5r/aLkZODpp/2JIiLA//0f0K+f/o1XGRP5Wgt1JR1vnoTL5cIzzzwDRVECzzkcDlMPcViV1YaSiHSjbUwUH7v2iHi9Ii+8INKtm78zIw1VMg+3hu/1uOIKkdJSo5uvmcWLFweuGB0OR8y3pG/sKbDLlWa4K+nFixeL0+kMPB/LdqusrJQVK1bIihUrDN1udvvugqmsrJTi4mJbf0aiaM7fCR2I6HnQKysT+e1vj8UUF2KtbEX/0IFHhw4iixeLNDRo3j6jqdVtrVYwYzbhgg07nNzs+t0RJSIGIhHQ+qBXVyfy6KMiiuKPKTJwUB7CtPC9HiNHiuzYoWpb9BZLgBdvrkPjeu08Bm+HYKMtdv/uiBKNaXJEzEqraXSbNh2bWpucDLw15R18If0gUPADOuMWPNL6RZ06AUuXAl6vPxRZvhw48cS42mGkWKeSqlETwu6zEuxcqdbu3x0RtS0hAxG1Dno1NcBddx1LMv3FWQfw2/dvgsBf1+Md/BL98FXrF44eDezc6Q88Dh4Err0WcMT+VZglwTOeAE+NRD4rFrgiP353RIkrIQOReA56a9b4S3IoCpCeLthy35soQ18IFBxAF9yEJ1q/qGtX/41ffD5/8PH880CvXqp8FjNVaow3wIu3JgRnJVgXvzuixKWIBKlmZBLV1dXIyMhAVVWV6uXe3W53q9vUBzvx7d0LzJoFLFrk/7sL9mM2ZmICng69grFjgfvvB3r2VLXdTXk8HmRnZ7e6a2pFRYXqB/BIbrWuZ3tC8Xg8vOuvRfG7I7KHqM7fmmesxEGPWTMtk/8ap9Z27dqYP+qTy/Ga7IIrdJJpz54ir7wi4vNp0tZg1EjwjEQ0ib1qTCUlIiJri+b8nbA9Ik19/bX/prOvvur/uzv24H7cgUI8G/qF118PzJ4N9OihWdtC0aMHIpZ18KqWiCixRXP+TsgcEQB4/XV/fqiiADk5AuXVl/EtMiFQsAeZwYOQXr38L2zM9SgqMiwIAfQZV48l78POszuIiEhdCdkjUlYGnJ/7LeZiBsbgb6EXnjgRuPtuf8KpSWnZA2GWvA8iIrKOaM7fSTq1yVR6fLMJ3+Ks4P/MyQEefRS4+GJ/d4kFuFwuzYKCxl6Xlom9DEKI7CGSRHQiLSXk0EzHDGfzJ6ZMAQ4c8A+3lJYCI0bEHISYpaaHmqx0q3UiipyZpv9T4krIoRmtuN3uQEEvh8OBoqIinrSJyJQ47EpaYrKqAbQqG09EpAWW1SezYCCiEv6oichKWFafzIKBSBRC5X/wR01EVsKy+mQWDEQiFC6piz9qIrIaJqKTGTBZNQLRJHWZoaoop+MREZGRmKyqsmjyP4yuKhqu58aO04uJiMi6GIhEwCr5H+Fm7rBmABERmQ0DkQhYJf8jVM8NpxfbH3u7iMiKGIhEyApJXaF6bji92N7Y20VEVsVAJApG53+EE6rnxirDSxQ99nYRkZUxELGZtnpurDK8ZCSrDm2wt4uIrIzTdxOMGaYXm5GV7xPEe4YQkdlEc/5mIEIJzw4ncrfbjfHjx8Pr9QZ6u6wSSBGR/URz/k7SqU1EphVqaMMqgUhhYSEKCgrY20VElsNAhBJeYyJvyx4RqyXyulwuBiBEZDlMVqWEx0ReIiLjMEeE6H+YyEtEpA7miBDFgEMbRET649BMEFatJ0FERGQ1DERaYKlsIiIi/TBHpAk71JMgIiIyWjTnb/aINMFS2erg0BYREUVKl0DkySefRO/evZGamoohQ4bg008/1WO1UeON4eLHoS0iIoqG5oHI8uXLMW3aNMyaNQubN2/GaaedhoKCAuzbt0/rVUeN9STiw7vAEhFRtDQPRB555BFcf/31+OMf/4hTTz0VTz/9NDp06IBnn31W61XHpK2711J4HNoiIqJoaVpHpK6uDps2bcKMGTMCzzkcDgwfPhwbNmxotXxtbS1qa2sDf1dXV2vZvDaxnkRs7FIqnYiI9KNpj8h3330Hr9eL7t27N3u+e/fu2LNnT6vl586di4yMjMAjKytLy+aphsmZfnYY2uJ3SUSkL1PNmpkxYwaqqqoCj8rKSqObFBaTM5uz8tAWv0siIv1pGoh06dIFTqcTe/fubfb83r170aNHj1bLp6SkID09vdnDzJic2TYTl6cJ2uvB75KIyBiaBiLJyckYPHgw1q5dG3jO5/Nh7dq1GDp0qJar1gWTM1sze69CW+2z+3fJISciMivNK6suX74cY8aMwaJFi3D22WdjwYIFWLFiBb788stWuSMtmf3uu6zE2pzZt0eo9gEwddvj4Xa7A709DocDRUVFlhoyIyLrMVVl1auuugoPPfQQZs6cidNPPx0lJSVYvXp12CDECuyQnKkms/cqhGqfXb9LDjkRkdnxXjMq8Hg8KCsrQ05OTswnLo/Hg9LSUuTm5lr25GflHpHG9qnxXWoh1v2juLgY+fn5QZ/Py8tTsYVERMeYqkckEbhcLuTl5cV84jJDXoUaOQRm71WIpH3xfpdaiGf/4G0LiMjs2CNiMDP0IqidQ2DWXoVGZm9fU2rsH263G+PHj4fX6w0EX8wRISItRXP+ZiBiMKO7zs0QCFHb1No/rBR8EZH1RXP+1rTEO4VndFn0cAmcZCy19g/etoCIzIo5IgYzOq+COQTmZvT+QUSkNQ7NmISRXefMITA/Dq0QkZUwR4SixhMdERGphTkiFDXmEBARkRGYI0JERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSBCREREhmEgQkRERIZhIEJERESGYSAShsfjQXFxMTweT0Kun4iISEsMREJwu93Izs5Gfn4+srOz4Xa7E2r9REREWlNERIxuRFuqq6uRkZGBqqoqpKen67puj8eD7Oxs+Hy+wHNOpxMVFRVwuVy2Xz8REVGsojl/a9Yjcv/99+Pcc89Fhw4d0KlTJ61Wo5nS0tJmQQAAeL1elJWVJcT6iYiI9KBZIFJXV4crr7wSEyZM0GoVmsrNzYXD0XzzOJ1O5OTkJMT6iYiI9KBZIHLPPfdg6tSpGDhwoFar0JTL5UJRURGcTicAfxCwaNEi3YZFjF4/ERGRHpKMbkBTtbW1qK2tDfxdXV1tYGuAwsJCFBQUoKysDDk5OboHAUavn4iISGumCkTmzp2Le+65x+hmNONyuQwNAIxePxERkZaiGpq5/fbboShKyMeXX34Zc2NmzJiBqqqqwKOysjLm9yIiIiLzi6pH5JZbbsHYsWNDLtOnT5+YG5OSkoKUlJSYX09ERETWElUg0rVrV3Tt2lWrthAREVGC0SxHZNeuXfj++++xa9cueL1elJSUAABycnLQsWNHrVZLREREFqJZIDJz5kwsXbo08PcZZ5wBACguLkZeXp5WqyUiIiILYYl3IiIiUpUpSrwTERERhcNAhIiIiAzDQISIiIgMw0CEiIiIDMNAhIiIiAxjqnvNtNQ4ocfom98RERFR5BrP25FMzDV1IFJTUwMAyMrKMrglREREFK2amhpkZGSEXMbUdUR8Ph++/fZbpKWlQVGUmN6juroaWVlZqKysZC0SHXB764fbWj/c1vri9taPVttaRFBTU4OePXvC4QidBWLqHhGHwwGXy6XKe6Wnp3OH1hG3t364rfXDba0vbm/9aLGtw/WENGKyKhERERmGgQgREREZxvaBSEpKCmbNmoWUlBSjm5IQuL31w22tH25rfXF768cM29rUyapERERkb7bvESEiIiLzYiBCREREhmEgQkRERIZhIEJERESGsUUg8uSTT6J3795ITU3FkCFD8Omnn4Zc/uWXX8bJJ5+M1NRUDBw4EKtWrdKppdYXzbZ+5plncN5556Fz587o3Lkzhg8fHva7oeai3bcbLVu2DIqi4PLLL9e2gTYS7bb+4YcfMGnSJGRmZiIlJQUnnXQSjyVRiHZ7L1iwAP369UP79u2RlZWFqVOn4ujRozq11ro++OAD/OpXv0LPnj2hKApef/31sK9Zv349zjzzTKSkpCAnJwdLlizRtpFiccuWLZPk5GR59tln5fPPP5frr79eOnXqJHv37g26/EcffSROp1MeeOAB2bZtm9x5553Srl072bp1q84tt55ot/XVV18tTz75pGzZskW++OILGTt2rGRkZIjH49G55dYU7fZuVF5eLj/5yU/kvPPOk8suu0yfxlpctNu6trZWzjrrLBkxYoR8+OGHUl5eLuvXr5eSkhKdW25N0W7vF154QVJSUuSFF16Q8vJyeeeddyQzM1OmTp2qc8utZ9WqVXLHHXfIa6+9JgBk5cqVIZffsWOHdOjQQaZNmybbtm2Txx9/XJxOp6xevVqzNlo+EDn77LNl0qRJgb+9Xq/07NlT5s6dG3T5kSNHyiWXXNLsuSFDhsj48eM1bacdRLutW2poaJC0tDRZunSpVk20lVi2d0NDg5x77rmyePFiGTNmDAORCEW7rRcuXCh9+vSRuro6vZpoK9Fu70mTJkl+fn6z56ZNmybDhg3TtJ12E0kgcuutt0r//v2bPXfVVVdJQUGBZu2y9NBMXV0dNm3ahOHDhweeczgcGD58ODZs2BD0NRs2bGi2PAAUFBS0uTz5xbKtWzpy5Ajq6+tx/PHHa9VM24h1e8+ePRvdunVDYWGhHs20hVi29RtvvIGhQ4di0qRJ6N69OwYMGIA5c+bA6/Xq1WzLimV7n3vuudi0aVNg+GbHjh1YtWoVRowYoUubE4kR50hT3/QunO+++w5erxfdu3dv9nz37t3x5ZdfBn3Nnj17gi6/Z88ezdppB7Fs65Zuu+029OzZs9VOTq3Fsr0//PBDuN1ulJSU6NBC+4hlW+/YsQPr1q3D6NGjsWrVKpSVlWHixImor6/HrFmz9Gi2ZcWyva+++mp89913+NnPfgYRQUNDA2644Qb85S9/0aPJCaWtc2R1dTV+/PFHtG/fXvV1WrpHhKxj3rx5WLZsGVauXInU1FSjm2M7NTU1uOaaa/DMM8+gS5cuRjfH9nw+H7p164aioiIMHjwYV111Fe644w48/fTTRjfNltavX485c+bgqaeewubNm/Haa6/hrbfewr333mt000gFlu4R6dKlC5xOJ/bu3dvs+b1796JHjx5BX9OjR4+olie/WLZ1o4ceegjz5s3De++9h0GDBmnZTNuIdnt//fXXqKiowK9+9avAcz6fDwCQlJSE7du3o2/fvto22qJi2bczMzPRrl07OJ3OwHOnnHIK9uzZg7q6OiQnJ2vaZiuLZXvfdddduOaaa3DdddcBAAYOHIjDhw9j3LhxuOOOO+Bw8JpaLW2dI9PT0zXpDQEs3iOSnJyMwYMHY+3atYHnfD4f1q5di6FDhwZ9zdChQ5stDwDvvvtum8uTXyzbGgAeeOAB3HvvvVi9ejXOOussPZpqC9Fu75NPPhlbt25FSUlJ4PHrX/8aF154IUpKSpCVlaVn8y0lln172LBhKCsrCwR7APDVV18hMzOTQUgYsWzvI0eOtAo2GoNA4e3SVGXIOVKzNFidLFu2TFJSUmTJkiWybds2GTdunHTq1En27NkjIiLXXHON3H777YHlP/roI0lKSpKHHnpIvvjiC5k1axan70Yo2m09b948SU5OlldeeUV2794deNTU1Bj1ESwl2u3dEmfNRC7abb1r1y5JS0uTG2+8UbZv3y5vvvmmdOvWTe677z6jPoKlRLu9Z82aJWlpafLSSy/Jjh07ZM2aNdK3b18ZOXKkUR/BMmpqamTLli2yZcsWASCPPPKIbNmyRXbu3CkiIrfffrtcc801geUbp+/++c9/li+++EKefPJJTt+NxOOPPy69evWS5ORkOfvss+WTTz4J/O+CCy6QMWPGNFt+xYoVctJJJ0lycrL0799f3nrrLZ1bbF3RbOvs7GwB0Ooxa9Ys/RtuUdHu200xEIlOtNv6448/liFDhkhKSor06dNH7r//fmloaNC51dYVzfaur6+Xu+++W/r27SupqamSlZUlEydOlIMHD+rfcIspLi4Oehxu3L5jxoyRCy64oNVrTj/9dElOTpY+ffrIc889p2kbFRH2axEREZExLJ0jQkRERNbGQISIiIgMw0CEiIiIDMNAhIiIiAzDQISIiIgMw0CEiIiIDMNAhIiIiAzDQISIiIgMw0CEiIiIDMNAhIiIiAzDQISIiIgMw0CEiIiIDPP/nIxHquWkkMcAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.plot(X.iloc[:,1], y, '.k') #산점도\n",
        "plt.plot(X.iloc[:,1], y_hat_NE, '-b', label = 'NE') #정규방정식\n",
        "plt.plot(X.iloc[:,1], y_hat_GD, '-r', label = 'GD') #경사하강법\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}