Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression - Gradient Descent\n",
    "\n",
    "This notebook shows an implementation of how to solve a linear regression model using the *gradient descent* algorithm.\n",
    "\n",
    "### Data Set\n",
    "\n",
    "The dataset that we will be working with is a 2 variable dataset showing the relationship between the number of years experience that each participant has at their job, and their current salary (one would indeed hope that there was a positive linear relationship there)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   YearsExperience   Salary\n",
      "0              1.1  39343.0\n",
      "1              1.3  46205.0\n",
      "2              1.5  37731.0\n",
      "3              2.0  43525.0\n",
      "4              2.2  39891.0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dataset = pd.read_csv(\"./data/salary-data.csv\")\n",
    "\n",
    "print(dataset.iloc[:5, :]) # first 5 rows\n",
    "\n",
    "X = dataset.iloc[:, :-1].values # get the data as a numpy array\n",
    "y = dataset.iloc[:, -1:].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Hypothesis\n",
    "\n",
    "For a linear regression model, we are assuming that the hypothesis (the shape of our model), will be of the form $h_\\theta(x) = \\theta_0 + \\theta_1 x$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_hypothesis(theta):\n",
    "    \n",
    "    def hypothesis(X):\n",
    "        theta_0, theta_1 = theta\n",
    "        return theta_0 + (theta_1 * X)\n",
    "    \n",
    "    return hypothesis\n",
    "        \n",
    "# sanity check tests\n",
    "test_h = get_hypothesis((1,1))\n",
    "assert test_h(1) == 2\n",
    "test_h = get_hypothesis((1,2))\n",
    "assert test_h(2) == 5\n",
    "test_h = get_hypothesis((10,12))\n",
    "assert test_h(3) == 46"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Cost Function\n",
    "\n",
    "The basic cost function used for linear regression is the *mean squared error* function, $J(\\theta) = \\frac{1}{2m} \\sum_{i=1}^m ( h_\\theta(x^{(i)}) - y^{(i)} )^2$. i.e. the average of the squared distance from the predicted value to the actual value. (The notation $x^{(i)}$ is used to denote the $i^{th}$ data point.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mean_squared_error(theta):\n",
    "    \n",
    "    def mean_squared_error(X, y):\n",
    "        h = get_hypothesis(theta)\n",
    "        return (1 / (2 * X.size)) * np.sum((h(X) - y)**2)\n",
    "    \n",
    "    return mean_squared_error\n",
    "    \n",
    "mse = get_mean_squared_error((0,1))\n",
    "assert mse(np.array([1,2,3]), np.array([2,3,4])) == 0.5\n",
    "mse = get_mean_squared_error((1,4))\n",
    "assert mse(np.array([1,2,3]), np.array([2,3,4])) == 21\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent\n",
    "\n",
    "To implement the gradient descent algorithm we need to calculate the partial differentials of the *cost function*, with respect to each of the $\\theta_i$. For the *mean squared error* function,\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\frac{1}{2m} \\sum_{i=1}^m ( \\theta_0 + \\theta_1 x^{(i)} - y^{(i)} )^2\n",
    "$$\n",
    "\n",
    "this gives us:\n",
    "\n",
    "$$\\begin{align}\n",
    "\\frac{\\partial J}{\\partial \\theta_0} &= \\frac{1}{m} \\sum_{i=1}^m ( \\theta_0 + \\theta_1 x^{(i)} - y^{(i)} ) \\\\\n",
    "\\frac{\\partial J}{\\partial \\theta_1} &= \\frac{1}{m} \\sum_{i=1}^m x^{(i)} ( \\theta_0 + \\theta_1 x^{(i)} - y^{(i)} ) \\\\.\n",
    "\\end{align}$$\n",
    "\n",
    "Using these results we can then perform the incremental updates of\n",
    "\n",
    "$$\n",
    "\\theta_{j, t+1} = \\theta_{j, t} - \\alpha \\frac{\\partial J}{\\partial \\theta_j}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_theta_0_partial(X, y):\n",
    "    \n",
    "    def partial(theta):\n",
    "        theta_0, theta_1 = theta\n",
    "        return (1 / len(X)) * np.sum(theta_0 + theta_1 * X - y)\n",
    "    \n",
    "    return partial\n",
    "\n",
    "def get_theta_1_partial(X, y):\n",
    "    \n",
    "    def partial(theta):\n",
    "        theta_0, theta_1 = theta\n",
    "        return (1 / len(X)) * np.sum(X * (theta_0 + theta_1 * X - y))\n",
    "    \n",
    "    return partial\n",
    "\n",
    "def get_step_gradient_descent(X, y, alpha):\n",
    "    \n",
    "    partial_theta_0 = get_theta_0_partial(X, y)\n",
    "    partial_theta_1 = get_theta_1_partial(X, y)\n",
    "    \n",
    "    def step_gradient_descent(theta):\n",
    "        theta_0, theta_1 = theta\n",
    "        theta_0_next = theta_0 - alpha * partial_theta_0(theta)\n",
    "        theta_1_next = theta_1 - alpha * partial_theta_1(theta)\n",
    "        return (theta_0_next, theta_1_next)\n",
    "    \n",
    "    return step_gradient_descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_linear_regression_model(X, y, theta=(0,1), alpha=0.01):\n",
    "    step_gradient_descent = get_step_gradient_descent(X, y, alpha)\n",
    "    step = 0\n",
    "    error_change_threshold = 0.1\n",
    "    mse = get_mean_squared_error(theta)\n",
    "    error = mse(X, y)\n",
    "    error_change = error\n",
    "    #print(f\"step {step}: theta = {theta}, mse = {error}, error change = {error_change}\")\n",
    "    while abs(error_change) > error_change_threshold:\n",
    "        step += 1\n",
    "        theta = step_gradient_descent(theta)\n",
    "        mse = get_mean_squared_error(theta)\n",
    "        new_error = mse(X, y)\n",
    "        error_change = error - new_error\n",
    "        error = new_error\n",
    "        #print(f\"step {step}: theta = {theta}, mse = {round(error)}, error change = {round(error_change)}\")\n",
    "    print(f\"returning after {step} steps...\")\n",
    "    print(f\"theta = {theta}\")\n",
    "    return theta\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "returning after 3490 steps...\n",
      "theta = (25777.466234939147, 9452.148779987654)\n"
     ]
    }
   ],
   "source": [
    "theta = fit_linear_regression_model(X, y, theta=(0,1), alpha=0.01)\n",
    "\n",
    "h = get_hypothesis(theta)\n",
    "\n",
    "y_prime = h(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "returning after 1 steps...\n",
      "theta = (0.0, 1.0)\n",
      "returning after 12068 steps...\n",
      "theta = (25763.389901736366, 9454.237648903281)\n",
      "returning after 6344 steps...\n",
      "theta = (25771.84062043273, 9452.983597628332)\n",
      "returning after 4350 steps...\n",
      "theta = (25775.59178085561, 9452.426941126103)\n",
      "returning after 3326 steps...\n",
      "theta = (25777.813009286136, 9452.097320126359)\n",
      "returning after 2701 steps...\n",
      "theta = (25779.364511705426, 9451.86708366282)\n",
      "returning after 2277 steps...\n",
      "theta = (25780.46503697841, 9451.703770319746)\n",
      "returning after 1971 steps...\n",
      "theta = (25781.33929571791, 9451.574033982957)\n",
      "returning after 1740 steps...\n",
      "theta = (25782.081346206574, 9451.46391679709)\n",
      "returning after 1558 steps...\n",
      "theta = (25782.66194936734, 9451.377757718774)\n",
      "returning after 1411 steps...\n",
      "theta = (25783.139004240023, 9451.306964772893)\n",
      "returning after 1290 steps...\n",
      "theta = (25783.55391460853, 9451.245393811263)\n",
      "returning after 1189 steps...\n",
      "theta = (25783.940825438895, 9451.187977863869)\n",
      "returning after 1103 steps...\n",
      "theta = (25784.279714220906, 9451.137688187402)\n",
      "returning after 1029 steps...\n",
      "theta = (25784.587882876865, 9451.091957249031)\n",
      "returning after 964 steps...\n",
      "theta = (25784.830870483416, 9451.055898905506)\n",
      "returning after 907 steps...\n",
      "theta = (25785.058252485986, 9451.022156368275)\n",
      "returning after 857 steps...\n",
      "theta = (25785.2980717327, 9450.986568196085)\n",
      "returning after 812 steps...\n",
      "theta = (25785.492547746428, 9450.957708769836)\n",
      "returning after 772 steps...\n",
      "theta = (25785.69986515565, 9450.92694373426)\n"
     ]
    }
   ],
   "source": [
    "for a in np.linspace(0, 0.05, 20):\n",
    "    fit_linear_regression_model(X, y, theta=(0,1), alpha=a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD4CAYAAAAZ1BptAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU1f3H8feXLBDWsAthlVUgKBJBxaWKCq4gYrW27tW22mqtPza1da1gsaitFmuLitaKNiBQFxDEXVE2DYQ17An7EtaEbOf3Ry42QCbrTO5k8nk9T55Mztx755uI85l7zrnnmnMOERGR4tTyuwAREQlfCgkREQlIISEiIgEpJEREJCCFhIiIBBTtdwHB1qxZM9ehQwe/yxARqVYWLVq0yznX/Pj2iAuJDh06sHDhQr/LEBGpVsxsY3Ht6m4SEZGAFBIiIhKQQkJERAJSSIiISEAKCRERCUghISIiASkkREQkIIWEiEg1tyxjHxPmrA7JsSPuYjoRkZoiOzefv85bw4ufrqNx3VhuOqs9zerXDuprKCRERKqhRRv3MDI5hbU7DzG8bxseuvwU4uvGBv11FBIiItXIoSN5jJ+9islfb6B1ozgm39aP87uesORS0CgkRESqgelLMnj83eXsPpQDwLmdmzHxxr7Urx3at3GFhIhImPv3Nxv5/fRU8p37oW3hxr3MXb4dgPGzV7ElM4vW8XGMGNSNoX0SgvbaCgkRkTA2a9k2Hpq+jAJ3bHtWbj6PzEzlSF4BWbn5AGRkZjFm2lKAoAWFpsCKiIShnQeOcPcbi/nlvxadEBBHZWbl/hAQR2Xl5jN+9qqg1aGQEBEJI845pi1O5+JnPmXO8u2MGNSN1o3qlOsYWzKzglZPqSFhZi+b2Q4zW1akbbyZrTSzFDN7x8ziizw3xszSzGyVmQ0q0j7Ya0szs9FF2jua2Tde+1tmFuu11/Z+TvOe7xCsX1pEJBxlZGZx66sL+N3b39OpeX3ev/dc7r6gMyMHdycuJuqYbeNiomhcN6bY47SOjwtaTWU5k3gVGHxc2xygl3OuN7AaGANgZj2A64Ge3j5/M7MoM4sCXgAuBXoAP/G2BXgKeMY51xnYC9zutd8O7PXan/G2ExGJOAUFjtfnb+SSCZ/y7fo9PHJlD97+xVl0blEfKBxfGDsskYT4OAxIiI9j7LBEHr6yZ7HhMWJQt6DVVurAtXPus+M/xTvnPizy43xguPd4CDDFOXcEWG9maUA/77k059w6ADObAgwxsxXAhcAN3jaTgUeAid6xHvHak4HnzcyccwF650REqp/1uw4xamoK367fwzmdmzF2WCJtm9Q9YbuhfRICDkaH++ym24C3vMcJFIbGUeleG8Dm49r7A02BTOdcXjHbJxzdxzmXZ2b7vO13HV+Amd0J3AnQrl27Sv46IiKhl5dfwKQv1jNhzmpqR9fiT8N7c23fNphZuY5TUngEQ6VCwsweBPKAN4JTTsU4514CXgJISkrSmYaIhLUVW/czamoKKen7uKRHS54Y2osWDcs3OF1VKhwSZnYLcAUwsEgXUAbQtshmbbw2ArTvBuLNLNo7myi6/dFjpZtZNNDI215EpFo6kpfPC/PS+Nsna4mvG8MLN5zOZYknlfvsoSpVKCTMbDAwEjjfOXe4yFMzgX+b2QSgNdAF+BYwoIuZdaTwzf964AbnnDOzjykc05gC3AzMKHKsm4GvvefnaTxCRKqrxZv2Mio5hTU7DjKsTwK/v6IHjesFf0G+YCs1JMzsTeBHQDMzSwcepnA2U21gjpeA851zv3TOpZrZ28ByCruh7nbO5XvH+TUwG4gCXnbOpXovMQqYYmZPAEuASV77JOB1b/B7D4XBIiISVqYvyShx4PhwTh5//nA1L3+5nlYN6/DKrWdwQbcWPlZcPhZpH86TkpLcwoUL/S5DRGqA6UsyGDNt6TFXPcfFRDF2WCJD+yTwVdouRk9byqY9h/nZme0YNbg7DeoUf22D38xskXMu6fh2rd0kIlJB42evKnZZjKdmreSb9bt589vNdGxWj7fuPJP+Jzf1qcrKUUiIiFRQoOUvtu7L5q0Fm/nF+Sdz30VdqXPcBW/ViUJCRKSCWsfHkVFMUETXMqbddTa928QXs1f1ogX+REQqaMSgbtSJPvZtNLqWMW5YYkQEBCgkREQqrP/JTejkra8E0KJBbZ6+9lSGJ7UtYa/qRd1NIhKWSpta6ifnHG9+u5mx768gt6CAhy4/hVsHdCSqVvheFFdRCgkRCTvHTy0NxR3XKmrj7kOMnrqUr9ft5qyTmzLumkTaN63na02hpJAQkbATaGrp+NmrfAuJ/ALHK1+u5+kPVxFTqxZjhyVy/Rltw3pJjWBQSIhI2Ak0tTSYd1wrj9XbDzAiOYXvN2dy0SkteGJoIicVc7e4cO4iqyiFhIiEnUBTS4N5x7WyyMkrYOIna3n+4zU0qBPDX37Shyt7tyr27CGcu8gqQ7ObRCTsjBjULeR3XCvN95szuer5L3hm7mouS2zFnPvO46pTWwfsXiqpi6w605mEiISdo5+8/ei6ycrJ59m5q/nH5+to0aAO/7wpiYt6tCx1v3DrIgsWhYSIhKVQ33GtOPPX7Wb01BQ27D7MT/q1Y8xl3WlYxgX5wqWLLNjU3SQiNd6B7FwefGcp1780nwIH/76jP2OHJZY5ICA8ushCQWcSIlKjfbxyBw++s5Rt+7P5+Tkduf+SbsTFln9BPj+7yEJJISEiEam06ah7D+Xw2LvLeWdJBl1a1Gfqr86mT7vGlXpNP7rIQk0hISIRp6TpqENOa817S7fy8IxU9mXlcs/ALtx9QSdqR1ff5bxDSSEhIhEn0HTUcR+s5P2lW/lw+XZ6t2nEv37en1NaNfSpyupBISEiESfQtNNt+7PZuzqHBy7rzm0DOhIdpbk7pVFIiEjECTQdNTaqFrN+ex4dm0XugnzBphgVkYhT3M2AYmoZ465JVECUk0JCRCJOr4SGxyzA17JhbcZfeyrDTm/jY1XVk7qbRCRi5OYX8PdP1/KXj9KoWzuKZ647laGnJUT8ct6hpJAQkYiwLGMfI5NTWL51P5f3bsWjV/WkWf3afpdV7SkkRKRay87N57mP1vDSZ+toUi+Wv9/Yl0E9Tyrz/pF4D4hgUkiISLW1YMMeRiWnsG7XIX6c1IYHL+tBo7plX28pUu8BEUwKCRGpdg4eyWP8rJW8Nn8jCfFx/Ov2/pzTpVm5jxOOt0kNNwoJEalWPl29kwemLWXLvixuObsD/3dJN+rVrthbWaTeAyKYFBIiUi1kHs7h8XdXMHVxOp2a1yP5l2fRt32TSh0zUu8BEUy6TkJEwt4HS7dy0YTPmP5dBr++oDPv3XNupQMCIvceEMGkMwkRCVs7DmTz8IxUPli2jZ6tGzL5tjPo2bpR0I4fqfeACCaFhIiEHeccUxdn8Pi7y8nKzWfk4G7cce7JxIRgQb5IvAdEMCkkRCSspO89zAPvLOOz1TtJat+Ycdf0pnOL+n6XVWMpJESkSgW6eK2gwPGvbzby1AcrccCjV/XkxjPbU6uWltTwk0JCRKpMoIvXtu/PZu6K7SzYsJdzuzRj7LBE2jSu63O1AgoJEalCgS5eG/vBShrWiWb88N4M79tGC/KFkVJHgczsZTPbYWbLirQ1MbM5ZrbG+97Yazcz+4uZpZlZipmdXmSfm73t15jZzUXa+5rZUm+fv5j3ryPQa4hI9VXSRWpz7z+fa5PaKiDCTFmmCrwKDD6ubTTwkXOuC/CR9zPApUAX7+tOYCIUvuEDDwP9gX7Aw0Xe9CcCdxTZb3ApryEi1VSgi9QS4uNo0aBOsc+Jv0oNCefcZ8Ce45qHAJO9x5OBoUXaX3OF5gPxZtYKGATMcc7tcc7tBeYAg73nGjrn5jvnHPDacccq7jVEpJoa3rcNx58n6OK18FbRMYmWzrmt3uNtQEvvcQKwuch26V5bSe3pxbSX9BonMLM7KTxzoV27duX9XUQkxA7n5DF+9ipe/WoD8XVjqGXGnkM5unitGqj0wLVzzpmZC0YxFX0N59xLwEsASUlJIa1FJFKF6r4KX6zZxehpKaTvzeKms9ozcnB36ldwQT6pehX9L7XdzFo557Z6XUY7vPYMoG2R7dp4bRnAj45r/8Rrb1PM9iW9hogEWSjuq7AvK5cn31vBWws307FZPd7+xVn061j59ZakalX0GveZwNEZSjcDM4q03+TNcjoT2Od1Gc0GLjGzxt6A9SXAbO+5/WZ2pjer6abjjlXca4hIkJV0X4WK+DB1GxdP+JTkxen88vxOfHDvuQqIaqrUMwkze5PCs4BmZpZO4SylccDbZnY7sBH4sbf5+8BlQBpwGLgVwDm3x8weBxZ42z3mnDs6GH4XhTOo4oAPvC9KeA0RCbJg3Vdh18EjPDIzlXdTtnJKq4ZMuvkMEtsEb0E+qXqlhoRz7icBnhpYzLYOuDvAcV4GXi6mfSHQq5j23cW9hogEX2Xvq+CcY8Z3W3j0v6kcOpLP/Rd35Zc/6hSSBfmkaum/oIhU6r4KWzKzuH3yQn771nd0aFaP9+45h98M7KKAiBCaYiAiFbqvQkGB480Fmxj7/kryCxx/uKIHN5/dgSgtyBdRFBIiApTvvgobdh1i1NQUvlm/hwGdmzL26t60a6oF+SKRQkJEyiwvv4CXv1zPnz9cTWx0LZ66JpEfa72liKaQEPFRqC5gC4WV2/YzKjmF79P3cdEpLfnj1b1o2VDrLUU6hYSIT0JxAVso5OQV8MLHafztkzQa1onhrz/pwxW9W+nsoYZQSIj4pKQL2MIlJL7bnMmo5BRWbT/A0NNa84cre9KkXqzfZUkVUkiI+CRYF7CFQlZOPhPmrGLSF+tp0aAOL9+SxIXdA66xKRFMISHik8pewBYqX6/dzehpKWzcfZgb+rdj9KXdaVgnxteaxD8KCRGfjBjU7ZgxCQjdvRXKMkC+PzuXse+v5M1vN9G+aV3evONMzurUNOi1SPWikBDxSUUuYKuIsgyQz1u5nQemLWPHgWzuOLcjv7u4G3GxUQGPKTWHQkLER+W5gK2iShogP69rcx79byozvttCt5YNePHGvpzWNj6k9Uj1opAQiXCBBsIzMrO4aMKnHMjO5d6BXbj7gs7ERmu9JTmWQkIkwgUaIAdo2ziOp4b3p/tJDau4Kqku9LFBJMIVt8IrwJBTWzPtrgEKCCmRziREItzQPgnsOniE8bNXcSSvgNioWowY1I07zjvZ79KkGlBIiESw/ALHq19t4M8friYmqha/v6IHN/RrRy0t5y1lpJAQiVBrth9g5NQUlmzK5IJuzfnj1Ym+X6gn1Y9CQiTC5OYXMPGTtTw/L416taN49rrTGHJaay3IJxWikBCJIEvT9zEi+XtWbjvAFb1b8chVPWlWv7bfZUk1ppAQiQDZufk8O3cN//h8HU3rxfLSjX25pOdJfpclEUAhIVLNfbt+D6OmprB+1yGuS2rLA5efQqM4LcgnwaGQEKmmDh7J46kPVvL6/I20bRLHGz/vz4DOzfwuSyKMQkKkGvpk1Q4efGcZW/ZlcduAjvzfoK7UjdX/zhJ8+lclUo1kHs7hsXeXM21xBp1b1Cf5l2fTt31jv8uSCKaQEKkm3l+6lT/MWEbm4Vx+c2Fnfn1hZ2pHazlvCS2FhEiY27E/mz/MSGVW6jZ6JTTktdv606O11luSqqGQEAlTzjmSF6Xz+LvLyc4rYNTg7txxbkeio7Qup1QdhYRIGNq85zAPvLOUz9fs4owOjXnqmt6c3Ly+32VJDaSQEClFWe4PHSwFBY7X52/kqVkrMeDxIT35af/2WpBPfKOQEClBWe4PHSxrdx5kVHIKCzfu5byuzXny6l60aVw3qK8hUl4KCZESlHR/6GCFRG5+Af/4fB3Pzl1DXEwUf772VIadnqAF+SQsKCREShDo/tCB2suiaPdVs/q1qR1Ti/S9WVyWeBKPXtWL5g20IJ+ED4WESAkC3R+6ovdlOL77aufBIwDcOqADD1/Zs+KFioSI5tKJlKC4+0PHxUQxYlC3Ch2vuO4rgA9Tt1foeCKhVqmQMLP7zCzVzJaZ2ZtmVsfMOprZN2aWZmZvmVmst21t7+c07/kORY4zxmtfZWaDirQP9trSzGx0ZWoVqYihfRIYOyyRhPg4DEiIj2PssMQKjUccOpJX7FkJVK77SiSUKtzdZGYJwD1AD+dclpm9DVwPXAY845ybYmYvArcDE73ve51znc3seuAp4Doz6+Ht1xNoDcw1s67ey7wAXAykAwvMbKZzbnlFaxapiKF9Eio9SP35mp0/zIoqjm4rKuGqst1N0UCcmUUDdYGtwIVAsvf8ZGCo93iI9zPe8wOtcPrGEGCKc+6Ic249kAb0877SnHPrnHM5wBRvW5FqY9/hXEb853tunPQtsdG1uOfCzkHtvhIJtQqfSTjnMszsaWATkAV8CCwCMp1zed5m6cDRj2AJwGZv3zwz2wc09drnFzl00X02H9fev7hazOxO4E6Adu3aVfRXEgmqWcu28fsZy9hzKIdf/agT9w7sQp2YKE5uXr/KLs4TqazKdDc1pvCTfUcgE/gPMDhIdZWLc+4l4CWApKQk50cNIkftPHCER2am8t7SrZzSqiGv3HIGvRIa/fB8MLqvRKpKZabAXgSsd87tBDCzacAAIN7Mor2ziTZAhrd9BtAWSPe6pxoBu4u0H1V0n0DtImHHOcc7SzJ47N3lHD6Sz4hB3bjzvJOJ0YJ8Uo1V5l/vJuBMM6vrjS0MBJYDHwPDvW1uBmZ4j2d6P+M9P88557z2673ZTx2BLsC3wAKgizdbKpbCwe2ZlahXJGQyMrO49dUF/O7t7zm5WT3ev/cc7r6gswJCqr3KjEl8Y2bJwGIgD1hCYZfPe8AUM3vCa5vk7TIJeN3M0oA9FL7p45xL9WZGLfeOc7dzLh/AzH4NzAaigJedc6kVrVckFAoKHG98u4lx76+gwMHDV/bgprM6EKUF+SRCWOGH+ciRlJTkFi5c6HcZUgOs33WIUVNT+Hb9Hs7p3IyxwxJp20QL8kn1ZGaLnHNJx7drWQ6RcsrLL2DSF+uZMGc1sdG1+NM1vbk2qY0W5JOIpJAQKYcVW/czamoKKen7uLhHS54Y2ouWDev4XZZIyCgkRMrgSF4+L3y8lr99nEajuBiev6EPlye20tmDRDyFhEgplmzay6ipKazefpCr+yTwhyt60LherN9liVQJhYRIAFk5+Tz94Spe/nI9JzWswyu3nMEF3Vv4XZZIlVJIiBTjq7W7GD11KZv2HOan/dsx+tLuNKgT43dZIlVOISFSxP7sXMa+v4I3v91Mh6Z1efOOMzmrU1O/yxLxjUJCxPPRiu08+M4ydhzI5s7zTua+i7oSFxtV+o4iEUwhITXe7oNHePS/y5n5/Ra6n9SAv9/Yl1PbxvtdlkhYUEhIjeWcY+b3W3j0v8s5kJ3LfRd15Vc/6kRstNZbEjlKISE10rZ92Tw0fSlzV+zgtLbx/Gl4b7q2bOB3WSJhRyEhNYpzjikLNvPkeyvILSjgoctP4dYBHbUgn0gACgmpMTbuPsToqUv5et1uzjq5KeOuSaR903p+lyUS1hQSEvHyCxyvfLmepz9cRUytWowdlsj1Z7TVkhoiZaCQkIi2evsBRian8N3mTAZ2b8ETV/eiVaM4v8sSqTYUEhKRcvIKePHTtfx13hoa1InhuetP46pTW+vsQaScFBIScVLSMxmZnMLKbQe46tTWPHxlD5rWr+13WSLVkkJCIkZ2bj7PzFnNPz5fR/MGtfnnTUlc1KOl32WJVGsKCYkI36zbzaipKWzYfZif9GvLmMtOoWExC/JNX5LB+Nmr2JKZRev4OEYM6sbQPgk+VCxSPSgkpFo7kJ3LU7NW8q/5m2jXpC7//nl/zu7crNhtpy/JYMy0pWTl5gOQkZnFmGlLARQUIgEoJKTa+njVDh6ctpSt+7O5/ZyO3H9JV+rGBv4nPX72qh8C4qis3HzGz16lkBAJQCEh1c7eQzk8/u5ypi3JoEuL+kz91dmc3q5xqfttycwqV7uIKCSkGnHO8d7SrTw8I5V9Wbncc2Fn7r6wM7Wjy7acd+v4ODKKCYTW8YXXTWi8QuRECgmpFnbsz+ah6cv4cPl2EhMa8a+f9+eUVg3LdYwRg7odMyYBEBcTxYhB3TReIRKAQkLCmnOO/yxM5/H3lpOTV8CYS7tz+zkdiY4q/3LeR9/siztbGDBunsYrRIqhkJCwtXnPYcZMW8oXabvo16EJ465J5OTm9St1zKF9Eop909d4hUjxFBJyAr/75vMLHK99vYE/zVpFLYPHh/bip/3aUSuEy3mXNl4hUlMpJOQYfvfNp+0oXJBv8aZMzu/anCeHJZJQBW/UJY1XiNRkCgk5hl/XEuTmF/DSZ+t4bu4a6taOYsKPT+XqPglVtiBfSeMVIjWZQkKOEeq++eK6sjq3qM/I5BSWb93P5YmteOSqnjRvUPUL8gUarxCpyRQScoxQ9s0X15X1f//5ngLnaFq/Ni/+rC+De51U6dcRkeAp/zxCiWgjBnUjLubYi9OC1TdfXFdWXoGjTkwUc+87XwEhEoZ0JiHHCGXffKAuq6ycfBrVPXHFVhHxn0JCThCqvvkm9WLZfSjnhPaKdmX5PVVXpCZQSEjI7Tucy+PvLWf3oRwMcEWeq2hXlt9TdUVqCo1JSEjNWraVi575lHeWZHD3BZ340/DeJMTHYUBCfBxjhyVW6E29pKm6IhI8lTqTMLN44J9ALwo/IN4GrALeAjoAG4AfO+f2WuGE9+eAy4DDwC3OucXecW4GHvIO+4RzbrLX3hd4FYgD3gfudc4V/SAqlRDK7pqdB47w8MxlvL90Gz1aNeSVW86gV0IjAK5Nalvp42sZDZGqUdkzieeAWc657sCpwApgNPCRc64L8JH3M8ClQBfv605gIoCZNQEeBvoD/YCHzezozQEmAncU2W9wJesVz9HumozMLBz/666ZviSjUsd1zjF1UToXTfiUuct3MGJQN2b8esAPAREsgcYxtIyGSHBVOCTMrBFwHjAJwDmX45zLBIYAk73NJgNDvcdDgNdcoflAvJm1AgYBc5xze5xze4E5wGDvuYbOufne2cNrRY4llRSK7pqMzCxueWUB9//nezq3qM/7957L3Rd0JqYCK7aWJpRTdUXkfyrT3dQR2Am8YmanAouAe4GWzrmt3jbbgJbe4wRgc5H90722ktrTi2k/gZndSeHZCe3atav4b1SDBLO7pqDA8cY3Gxn3wUoc8MiVPbjprA4hXZBPy2iIVI3KhEQ0cDrwG+fcN2b2HP/rWgLAOefMLORjCM65l4CXAJKSkjRmUQbBurJ63c6DjJ66lG837OHcLs148upE2japG6wyS6RlNERCrzL9AOlAunPuG+/nZApDY7vXVYT3fYf3fAZQdMSyjddWUnubYtolCCrbXZOXX8DET9Yy+LnPWbltP+OH9+a12/pVWUCISNWo8JmEc26bmW02s27OuVXAQGC593UzMM77PsPbZSbwazObQuEg9T7n3FYzmw08WWSw+hJgjHNuj5ntN7MzgW+Am4C/VrReOVZlumuWb9nPyKnfsyxjP4N6tuTxIb1o0bDOD89XZNaULowTCU+VvZjuN8AbZhYLrANupfDs5G0zux3YCPzY2/Z9Cqe/plE4BfZWAC8MHgcWeNs95pzb4z2+i/9Ngf3A+5JKqMyb8ZG8fJ6fl8bET9YSXzeGv/30dC5LbHXC8ct6kdvRWjIys465yE4XxomED4u0yw6SkpLcwoUL/S4jLB3/Bg6FXUxluaBt0ca9jJqaQtqOgww7PYHfX96DxvViT9huwLh5xY51JMTH8eXoC0uspbR9RCR0zGyRcy7p+HYty1GDVOSGQodz8hg/exWvfrWBVg3r8MqtZ3BBtxYBX6Oss6aKq6WsxxKRqqOQqEHKO+31izW7GD0thfS9Wdx4ZntGXdqd+rVL/idT1llTxW1T2j4iUvW0dlMNUtarlPdl5TIqOYWfTfqGmKhavHXnmTw+tFepAQFlnzUVVcptSXVhnEh4UEjUIGV5A/8wdRvnPjWPtxYWXt94JDefrfuyy/waQ/skMHZYYqmL+OWXMBZWmYX/RCS41N1Ug5Q07XXXwSM8MjOVd1O2UvQz/pZ92eWeaVSWi9wSAnRLabBaJLwoJGqY49/AnXNMX5LBo/9N5dCRfBrUieZAdt4x+5Q2uF0RIwZ1K3amlbqYRMKLuptqsK37svj55IX89q3vaN+0Hu/dcw4HjwuIo4I906is3VIi4i+dSdRABQWONxdsYuz7K8kvcDx0+SncOqAjUbUsaGs6lYXWXhIJfwqJGmbDrkOMnpbC/HV7OLtTU8YN6027pv9bb0ndQCJSlEKihsjLL+DlL9fz5w9XExtVi3HDErnujLbYcVNRtQS3iBSlkKgBVm7bz6jkFL5P38dFp7TkiaG9OKlRnYDbqxtIRI5SSESwnLwCXvg4jb99kkbDOjH89Sd9uKJ3qxPOHkREAlFIRKjvNmcyKjmFVdsPMOS01jx8ZU+aFLMgn4hISRQSESYrJ58Jc1Yx6Yv1tGhQh0k3JzHwlJal7ygiUgyFRAT5eu1uRk9LYePuw9zQvx2jL+1OwzoxfpclItWYQiIC7M/OZdwHK/n3N5to37Qu/76jP2d3auZ3WSISARQS1dy8ldt5YNoydhzI5ufndOT+S7oRFxtV+o4iImWgkKim9hzK4bH/pjL9uy10bVmfiT87mz7tGpe+o4hIOSgkqhnnHO+mbOWRmansy8rl3oFduPuCzsRGaxkuEQk+hUQ1sn1/Ng++s4y5K7bTu00j3rijP91Pauh3WSISwRQS1YBzjrcXbuaJ91aQk1fAA5d157YBHYmO0tmDiISWQiLMbdp9mDHvpPBl2m76dWzCU9f0pmOzen6XJSI1hEKiFNOXZPiy2F1+gePVrzbw9OxVRNUynhjaixv6taNWLS2pISJVRyFRgulLMo5ZNjsjM6vct/KsiLQdBxiZnMLiTZn8qFtznrw6MST3cxARKY1CogTjZ6865r4KEJpbeR6Vm1/A3z9dy18+SqNu7Sieue5Uhp6WoMzKnpoAAAhnSURBVAX5RMQ3CokSBLplZ7Bv5QmwLGMfI5JTWLF1P5f3bsWjV/WkWf3aQX8dEZHyUEiUoCpu5Zmdm8+zc9fwj8/X0bReLH+/sS+Dep4UtOOLiFSG5lCWYMSgbsTFHLvERTBv5blgwx4ue+5zXvx0LcNPb8Oc352vgBCRsKIziRKE6laeB4/k8adZK3nt641EebOVvkjbxccrd+iOcCISVhQSpQj2rTw/Xb2TB6YtZUtmFlG1jPwCB1TdzCkRkfJQd1MVyTycw/1vf8/NL39LXGwUTevH/hAQRx2dOSUiEi4UElXgg6VbuWjCZ8z4LoPfXNiZ9+45h90Hc4rdNhQzp0REKkrdTSG040A2f5ieyqzUbfRKaMjk286gZ+tGQNXMnBIRqSydSYSAc47/LNzMxRM+Y96qHYwa3J3pdw34ISAg9DOnRESCQWcSQZa+9zAPvLOMz1bv5IwOjRl3TW86Na9/wnahmjklIhJMlQ4JM4sCFgIZzrkrzKwjMAVoCiwCbnTO5ZhZbeA1oC+wG7jOObfBO8YY4HYgH7jHOTfbax8MPAdEAf90zo2rbL2hUlDgeH3+Rp6atRIDHhvSk5/1b1/ignzBnjklIhJswTiTuBdYARy9+81TwDPOuSlm9iKFb/4Tve97nXOdzex6b7vrzKwHcD3QE2gNzDWzrt6xXgAuBtKBBWY20zm3PAg1B9XanQcZlZzCwo17Oa9rc568uhdtGtet8PH8WnlWROR4lQoJM2sDXA78EfidFa5EdyFwg7fJZOARCkNiiPcYIBl43tt+CDDFOXcEWG9maUA/b7s059w677WmeNsGPSQq+qacl1/AS5+v49m5a4iLieLpa0/lmtMrtyCfXyvPiogUp7JnEs8CI4EG3s9NgUznXJ73czpw9J0tAdgM4JzLM7N93vYJwPwixyy6z+bj2vsXV4SZ3QncCdCuXbty/QIVfVNO3bKPkckppG7Zz6W9TuLRIT1p0aBOuV67OFW98qyISEkqPLvJzK4AdjjnFgWxngpxzr3knEtyziU1b968XPuW9KZcnOzcfJ6evYohz3/J9v1HmPjT05n4s75BCQio2pVnRURKU5kziQHAVWZ2GVCHwjGJ54B4M4v2zibaABne9hlAWyDdzKKBRhQOYB9tP6roPoHag6Y8b8qLNu5hZHIKa3ce4owOjUnfk8VdbywO6riBrp8QkXBS4TMJ59wY51wb51wHCgee5znnfgp8DAz3NrsZmOE9nun9jPf8POec89qvN7Pa3syoLsC3wAKgi5l1NLNY7zVmVrTeQAK9+RZtP3Qkj0dmpjL8xa/Jzi3gF+edzLKM/Wzdn43jf11U05dUPsN0/YSIhJNQXEw3isJB7DQKxxwmee2TgKZe+++A0QDOuVTgbQoHpGcBdzvn8r0zkV8DsymcPfW2t21Qlfam/MWaXQx69jNe/WoDN57Zntn3nce7KVvL1UVVHkP7JDB2WCIJ8XEYkBAfx9hhiRqPEBFfWOGH+ciRlJTkFi5cWK59ipvddEH3FvzxveW8vTCdjs3q8dQ1venXsQkAHUe/R3F/NQPWj7u88r+EiEgVM7NFzrmk49t1xTUnXtQ2O3UbF0/4lN2HcvjVjzpx78Au1ClytqFxAxGpKbR2UxE7Dxzh7jcW84vXF9G0fm2m3zWAUYO7HxMQoHEDEak5dCbh+SptF3f9ezGHj+Tzf5d05RfndyImqvgM1bpLIlJTKCQ8HZrVIzGhEX+4ogddWjYodXutuyQiNYFCwtM6Po7Xby/2gm4RkRpLYxIiIhKQQkJERAJSSIiISEAKCRERCUghISIiASkkREQkIIWEiIgEpJAQEZGAIm4VWDPbCWz0u45yagbs8rsIH9X03x/0N6jpvz/4/zdo75w74daeERcS1ZGZLSxuid6aoqb//qC/QU3//SF8/wbqbhIRkYAUEiIiEpBCIjy85HcBPqvpvz/ob1DTf38I07+BxiRERCQgnUmIiEhACgkREQlIIeETM2trZh+b2XIzSzWze/2uyQ9mFmVmS8zsXb9r8YOZxZtZspmtNLMVZnaW3zVVNTO7z/t/YJmZvWlmdfyuKdTM7GUz22Fmy4q0NTGzOWa2xvve2M8aj1JI+CcPuN851wM4E7jbzHr4XJMf7gVW+F2Ej54DZjnnugOnUsP+FmaWANwDJDnnegFRwPX+VlUlXgUGH9c2GvjIOdcF+Mj72XcKCZ8457Y65xZ7jw9Q+OZQo26abWZtgMuBf/pdix/MrBFwHjAJwDmX45zL9LcqX0QDcWYWDdQFtvhcT8g55z4D9hzXPASY7D2eDAyt0qICUEiEATPrAPQBvvG3kir3LDASKPC7EJ90BHYCr3hdbv80s3p+F1WVnHMZwNPAJmArsM8596G/VfmmpXNuq/d4G9DSz2KOUkj4zMzqA1OB3zrn9vtdT1UxsyuAHc65RX7X4qNo4HRgonOuD3CIMOliqCpev/sQCgOzNVDPzH7mb1X+c4XXJoTF9QkKCR+ZWQyFAfGGc26a3/VUsQHAVWa2AZgCXGhm//K3pCqXDqQ7546eQSZTGBo1yUXAeufcTudcLjANONvnmvyy3cxaAXjfd/hcD6CQ8I2ZGYV90SuccxP8rqeqOefGOOfaOOc6UDhQOc85V6M+QTrntgGbzayb1zQQWO5jSX7YBJxpZnW9/ycGUsMG74uYCdzsPb4ZmOFjLT9QSPhnAHAjhZ+gv/O+LvO7KKlyvwHeMLMU4DTgSZ/rqVLeWVQysBhYSuF7UlguTxFMZvYm8DXQzczSzex2YBxwsZmtofAMa5yfNR6lZTlERCQgnUmIiEhACgkREQlIISEiIgEpJEREJCCFhIiIBKSQEBGRgBQSIiIS0P8DNSES5+cw3ooAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.subplots()\n",
    "ax.scatter(X, y)\n",
    "ax.plot(X, y_prime)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}