Multivariate Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multivariate Linear Regression - Gradient Descent\n",
    "\n",
    "This notebook shows an implementation of how to solve a multivariable linear regression model using the *gradient descent* algorithm.\n",
    "\n",
    "### Dataset\n",
    "\n",
    "The dataset that we will be working with is a dataset of cars. The features are:\n",
    "\n",
    "- the miles per gallon (MPG) that the car can do;\n",
    "- the number of cylinders that the car's engine has;\n",
    "- the displacement of the car's engine;\n",
    "- the horsepower of the car's engine;\n",
    "- the weight of the car;\n",
    "- the acceleration of the car;\n",
    "- the year that the car was made;\n",
    "- the country of origin of the car;\n",
    "- the name of the car."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>model year</th>\n",
       "      <th>origin</th>\n",
       "      <th>car name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet,chevelle,malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick,skylark,320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth,satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc,rebel,sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford,torino</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  \\\n",
       "0  18.0          8         307.0       130.0  3504.0          12.0   \n",
       "1  15.0          8         350.0       165.0  3693.0          11.5   \n",
       "2  18.0          8         318.0       150.0  3436.0          11.0   \n",
       "3  16.0          8         304.0       150.0  3433.0          12.0   \n",
       "4  17.0          8         302.0       140.0  3449.0          10.5   \n",
       "\n",
       "   model year  origin                   car name  \n",
       "0          70       1  chevrolet,chevelle,malibu  \n",
       "1          70       1          buick,skylark,320  \n",
       "2          70       1         plymouth,satellite  \n",
       "3          70       1              amc,rebel,sst  \n",
       "4          70       1                ford,torino  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dataset = pd.read_csv(\"./data/auto-mpg-data.csv\")\n",
    "\n",
    "#print(dataset.iloc[:5, :]) # first 5 rows\n",
    "\n",
    "X = dataset.iloc[:, 1:-1].values # get the data as a numpy array\n",
    "y = dataset.iloc[:, :1].values\n",
    "\n",
    "dataset.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Hypothesis\n",
    "\n",
    "For a multivariate linear regression model, we take the hypothesis (the shape of our model), to be of the form $h_\\theta(x) = \\vec{\\theta} \\cdot \\mathbf{x}$. Where\n",
    "\n",
    "$$\\begin{align}\n",
    "\\vec{\\theta} &= \\theta_0 + \\theta_1 + \\dots + \\theta_n \\text{ and,} \\\\\n",
    "\\mathbf{x} &= x_0 + x_1 \\dots + x_n,\n",
    "\\end{align}$$\n",
    "\n",
    "and by convention we let $x_0 = 1$, making the first term of the dot product $\\theta_0$, (i.e. a contant term).\n",
    "\n",
    "$$\n",
    "h_\\theta(\\mathbf{x}) = \\vec{\\theta} \\cdot \\mathbf{x} = \\theta_0 + \\theta_1 x_1 + \\dots + \\theta_n x_n.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hypothesis(theta):\n",
    "    \"\"\"Curried hypothesis function - this function returns a function to be called on a feature vector\n",
    "    \n",
    "    Args:\n",
    "        theta (numpy array): vector of parameters of the linear regression\n",
    "    \"\"\"\n",
    "    N = len(theta)\n",
    "    \n",
    "    def hypothesis_of_X(X):\n",
    "        \"\"\"Calculates the predicted value of a feature vector\n",
    "        \n",
    "        Args:\n",
    "            X (numpy array): feature vector\n",
    "        \"\"\"\n",
    "        X = np.insert(X, 0, [1], 0)\n",
    "        \n",
    "        if len(X) != N:\n",
    "            raise ValueError(\"X is the wrong shape for this hypothesis\")\n",
    "            \n",
    "        return np.sum(theta * X)\n",
    "    \n",
    "    return hypothesis_of_X\n",
    "        \n",
    "# sanity check tests\n",
    "assert hypothesis(np.array([1,1]))(np.array([1])) == 2\n",
    "assert hypothesis(np.array([1,2]))(np.array([2])) == 5\n",
    "assert hypothesis(np.array([10,12]))(np.array([3])) == 46\n",
    "assert hypothesis(np.array([1,-2,3]))(np.array([3,3])) == 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Cost Function\n",
    "\n",
    "We will implement the *mean squared error* cost function, $J(\\theta) = \\frac{1}{m} \\sum_{i=1}^m ( h_\\theta(\\mathbf{x}^{(i)}) - y^{(i)} )^2$. i.e. the average of the squared distance from the predicted value to the actual value. (The notation $\\mathbf{x}^{(i)}$ is used to denote the $i^{th}$ feature vector.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_squared_error(X, y):\n",
    "    \"\"\"Curried mean squared error (MSE) function\n",
    "    \n",
    "    This function returns a function to be called on a feature vector and target value to calculate\n",
    "    the MSE of the hypothesis with the given parameters theta.\n",
    "    \n",
    "    Args:\n",
    "        theta (numpy array): vector of parameters of the linear regression\n",
    "    \"\"\"\n",
    "    \n",
    "    def mean_squared_error_of_X_y(theta):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            X (numpy array): feature vector\n",
    "            y (numpy array): target value\n",
    "        \"\"\"\n",
    "        h = hypothesis(theta)\n",
    "        predictions = np.apply_along_axis(h, 1, X)\n",
    "        return (1 / X.size) * np.sum((predictions - y.flatten())**2)\n",
    "    \n",
    "    return mean_squared_error_of_X_y\n",
    "    \n",
    "assert mean_squared_error(np.array([[1],[2],[3]]), np.array([[2],[3],[4]]))(np.array([0,1])) == 1\n",
    "assert mean_squared_error(np.array([[1],[2],[3]]), np.array([[2],[3],[4]]))(np.array([1,4])) == 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent\n",
    "\n",
    "To implement the gradient descent algorithm we need to calculate the partial differentials of the *cost function*, with respect to each of the $\\theta_i$. For the *mean squared error* function,\n",
    "\n",
    "$$\n",
    "J(\\theta) = \\frac{1}{m} \\sum_{i=1}^m ( h_\\theta(\\mathbf{x}^{(i)}) - y^{(i)} )^2\n",
    "$$\n",
    "\n",
    "this gives us:\n",
    "\n",
    "$$\\begin{align}\n",
    "\\frac{\\partial J}{\\partial \\theta_j} &= \\frac{2}{m} \\sum_{i=1}^m x_j^{(i)} ( h_\\theta(\\mathbf{x}^{(i)}) - y^{(i)} ). \\\\\n",
    "\\end{align}$$\n",
    "\n",
    "Using these results we can then perform the incremental updates of\n",
    "\n",
    "$$\n",
    "\\theta_{j, t+1} = \\theta_{j, t} - \\alpha \\frac{\\partial J}{\\partial \\theta_j}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def theta_j_partial(X, y):\n",
    "    \"\"\"Curried function to calculate the vector of dJ/d(theta_j)\n",
    "    \n",
    "    This function returns a function to be called on a theta vector to calculate the gradient vector\n",
    "    for the cost function with respect to each of the linear regression parameters.\n",
    "    \n",
    "    Args:\n",
    "        X (numpy array): feature vector\n",
    "        y (numpy array): target value\n",
    "    \"\"\"\n",
    "    \n",
    "    def J_theta_j_partials(theta):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            theta (numpy array): vector of parameters of the linear regression\n",
    "        \"\"\"\n",
    "        rows, cols = X.shape\n",
    "        X_pad_ones = np.ones((rows, cols + 1))\n",
    "        X_pad_ones[:, 1:] = X\n",
    "        h = hypothesis(theta)\n",
    "        predictions = np.apply_along_axis(h, 1, X)\n",
    "        errors = predictions - y.T\n",
    "        return (2 / X.size) * np.sum((X_pad_ones.T * errors), axis=1)\n",
    "    \n",
    "    return J_theta_j_partials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_linear_regression_model(X, y, theta=(0,1), alpha=0.01):\n",
    "    \"\"\"Function to train a linear regression model\n",
    "    \n",
    "    A function to train a linear regression model on feature vectors X and output vector y.\n",
    "    The function displays the steps of the training, and returns the learned value of the model's parameters, theta.\n",
    "    \n",
    "    Args:\n",
    "        X (numpy array): feature vector\n",
    "        y (numpy array): target value\n",
    "        theta (numpy array): initial values of the parameters for the model\n",
    "        alpha (float): the learning rate\n",
    "    \"\"\"\n",
    "    \n",
    "    step_gradient_descent = get_step_gradient_descent(X, y, alpha)\n",
    "    step = 0\n",
    "    error_change_threshold = 0.1\n",
    "    mse = mean_squared_error(X, y)\n",
    "    error = mse(theta)\n",
    "    error_change = error\n",
    "    print(f\"step {step}: theta = {theta}, mse = {error}, error change = {error_change}\")\n",
    "    while abs(error_change) > error_change_threshold:\n",
    "        step += 1\n",
    "        theta = step_gradient_descent(theta)\n",
    "        new_error = mse(theta)\n",
    "        error_change = error - new_error\n",
    "        error = new_error\n",
    "        print(f\"step {step}: theta = {np.round(theta, 2)}, mse = {round(error)}, error change = {round(error_change)}\")\n",
    "    print(f\"returning after {step} steps...\")\n",
    "    print(f\"theta = {theta}\")\n",
    "    return theta\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0: theta = (0, 1, 1, 1, 1, 1, 1, 1), mse = 1743489.8189176386, error change = 1743489.8189176386\n",
      "step 1: theta = [-0.    1.    1.    1.    0.97  1.    1.    1.  ], mse = 1648706.0, error change = 94784.0\n",
      "step 2: theta = [-0.    1.    1.    1.    0.94  1.    1.    1.  ], mse = 1559077.0, error change = 89630.0\n",
      "step 3: theta = [-0.    1.    0.99  1.    0.91  1.    1.    1.  ], mse = 1474321.0, error change = 84756.0\n",
      "step 4: theta = [-0.    1.    0.99  1.    0.88  1.    1.    1.  ], mse = 1394174.0, error change = 80147.0\n",
      "step 5: theta = [-0.    1.    0.99  0.99  0.85  1.    1.    1.  ], mse = 1318385.0, error change = 75789.0\n",
      "step 6: theta = [-0.    1.    0.99  0.99  0.83  1.    1.    1.  ], mse = 1246717.0, error change = 71668.0\n",
      "step 7: theta = [-0.    1.    0.99  0.99  0.8   1.    1.    1.  ], mse = 1178946.0, error change = 67771.0\n",
      "step 8: theta = [-0.    1.    0.98  0.99  0.78  1.    0.99  1.  ], mse = 1114861.0, error change = 64086.0\n",
      "step 9: theta = [-0.    1.    0.98  0.99  0.75  1.    0.99  1.  ], mse = 1054260.0, error change = 60601.0\n",
      "step 10: theta = [-0.    1.    0.98  0.99  0.73  1.    0.99  1.  ], mse = 996954.0, error change = 57306.0\n",
      "step 11: theta = [-0.    1.    0.98  0.99  0.7   1.    0.99  1.  ], mse = 942765.0, error change = 54190.0\n",
      "step 12: theta = [-0.    1.    0.98  0.99  0.68  1.    0.99  1.  ], mse = 891522.0, error change = 51243.0\n",
      "step 13: theta = [-0.    1.    0.98  0.99  0.66  1.    0.99  1.  ], mse = 843065.0, error change = 48457.0\n",
      "step 14: theta = [-0.    1.    0.97  0.99  0.64  1.    0.99  1.  ], mse = 797244.0, error change = 45822.0\n",
      "step 15: theta = [-0.    1.    0.97  0.99  0.62  1.    0.99  1.  ], mse = 753914.0, error change = 43330.0\n",
      "step 16: theta = [-0.    1.    0.97  0.99  0.6   1.    0.99  1.  ], mse = 712940.0, error change = 40974.0\n",
      "step 17: theta = [-0.    1.    0.97  0.98  0.58  1.    0.99  1.  ], mse = 674194.0, error change = 38746.0\n",
      "step 18: theta = [-0.    1.    0.97  0.98  0.56  1.    0.99  1.  ], mse = 637555.0, error change = 36639.0\n",
      "step 19: theta = [-0.    1.    0.97  0.98  0.54  1.    0.99  1.  ], mse = 602908.0, error change = 34647.0\n",
      "step 20: theta = [-0.    1.    0.97  0.98  0.52  1.    0.99  1.  ], mse = 570145.0, error change = 32763.0\n",
      "step 21: theta = [-0.    1.    0.97  0.98  0.5   1.    0.99  1.  ], mse = 539164.0, error change = 30981.0\n",
      "step 22: theta = [-0.    1.    0.96  0.98  0.49  1.    0.99  1.  ], mse = 509868.0, error change = 29297.0\n",
      "step 23: theta = [-0.    1.    0.96  0.98  0.47  1.    0.99  1.  ], mse = 482164.0, error change = 27704.0\n",
      "step 24: theta = [-0.    1.    0.96  0.98  0.45  1.    0.99  1.  ], mse = 455967.0, error change = 26197.0\n",
      "step 25: theta = [-0.    1.    0.96  0.98  0.44  1.    0.99  1.  ], mse = 431194.0, error change = 24773.0\n",
      "step 26: theta = [-0.    1.    0.96  0.98  0.42  1.    0.99  1.  ], mse = 407769.0, error change = 23426.0\n",
      "step 27: theta = [-0.    1.    0.96  0.98  0.41  1.    0.99  1.  ], mse = 385617.0, error change = 22152.0\n",
      "step 28: theta = [-0.    1.    0.96  0.98  0.39  1.    0.99  1.  ], mse = 364670.0, error change = 20947.0\n",
      "step 29: theta = [-0.    1.    0.96  0.98  0.38  1.    0.99  1.  ], mse = 344862.0, error change = 19808.0\n",
      "step 30: theta = [-0.    1.    0.96  0.98  0.36  1.    0.99  1.  ], mse = 326131.0, error change = 18731.0\n",
      "step 31: theta = [-0.    1.    0.95  0.98  0.35  1.    0.98  1.  ], mse = 308418.0, error change = 17713.0\n",
      "step 32: theta = [-0.    1.    0.95  0.98  0.34  1.    0.98  1.  ], mse = 291669.0, error change = 16749.0\n",
      "step 33: theta = [-0.    1.    0.95  0.98  0.32  1.    0.98  1.  ], mse = 275830.0, error change = 15839.0\n",
      "step 34: theta = [-0.    1.    0.95  0.98  0.31  1.    0.98  1.  ], mse = 260853.0, error change = 14977.0\n",
      "step 35: theta = [-0.    1.    0.95  0.98  0.3   1.    0.98  1.  ], mse = 246690.0, error change = 14163.0\n",
      "step 36: theta = [-0.    1.    0.95  0.97  0.29  1.    0.98  1.  ], mse = 233297.0, error change = 13393.0\n",
      "step 37: theta = [-0.    1.    0.95  0.97  0.28  1.    0.98  1.  ], mse = 220632.0, error change = 12665.0\n",
      "step 38: theta = [-0.    1.    0.95  0.97  0.27  1.    0.98  1.  ], mse = 208656.0, error change = 11976.0\n",
      "step 39: theta = [-0.    1.    0.95  0.97  0.26  1.    0.98  1.  ], mse = 197332.0, error change = 11325.0\n",
      "step 40: theta = [-0.    1.    0.95  0.97  0.25  1.    0.98  1.  ], mse = 186623.0, error change = 10709.0\n",
      "step 41: theta = [-0.    1.    0.95  0.97  0.24  1.    0.98  1.  ], mse = 176496.0, error change = 10127.0\n",
      "step 42: theta = [-0.    1.    0.95  0.97  0.23  1.    0.98  1.  ], mse = 166920.0, error change = 9576.0\n",
      "step 43: theta = [-0.    1.    0.95  0.97  0.22  1.    0.98  1.  ], mse = 157865.0, error change = 9055.0\n",
      "step 44: theta = [-0.    1.    0.94  0.97  0.21  1.    0.98  1.  ], mse = 149302.0, error change = 8563.0\n",
      "step 45: theta = [-0.    1.    0.94  0.97  0.2   1.    0.98  1.  ], mse = 141205.0, error change = 8097.0\n",
      "step 46: theta = [-0.    1.    0.94  0.97  0.19  1.    0.98  1.  ], mse = 133548.0, error change = 7657.0\n",
      "step 47: theta = [-0.    1.    0.94  0.97  0.18  1.    0.98  1.  ], mse = 126307.0, error change = 7241.0\n",
      "step 48: theta = [-0.    1.    0.94  0.97  0.17  1.    0.98  1.  ], mse = 119460.0, error change = 6847.0\n",
      "step 49: theta = [-0.    1.    0.94  0.97  0.16  1.    0.98  1.  ], mse = 112986.0, error change = 6475.0\n",
      "step 50: theta = [-0.    1.    0.94  0.97  0.16  1.    0.98  1.  ], mse = 106863.0, error change = 6123.0\n",
      "step 51: theta = [-0.    1.    0.94  0.97  0.15  1.    0.98  1.  ], mse = 101074.0, error change = 5790.0\n",
      "step 52: theta = [-0.    1.    0.94  0.97  0.14  1.    0.98  1.  ], mse = 95599.0, error change = 5475.0\n",
      "step 53: theta = [-0.    1.    0.94  0.97  0.13  1.    0.98  1.  ], mse = 90422.0, error change = 5177.0\n",
      "step 54: theta = [-0.    1.    0.94  0.97  0.13  1.    0.98  1.  ], mse = 85526.0, error change = 4896.0\n",
      "step 55: theta = [-0.    1.    0.94  0.97  0.12  1.    0.98  1.  ], mse = 80897.0, error change = 4629.0\n",
      "step 56: theta = [-0.    1.    0.94  0.97  0.11  1.    0.98  1.  ], mse = 76519.0, error change = 4378.0\n",
      "step 57: theta = [-0.    1.    0.94  0.97  0.11  1.    0.98  1.  ], mse = 72380.0, error change = 4140.0\n",
      "step 58: theta = [-0.    1.    0.94  0.97  0.1   1.    0.98  1.  ], mse = 68465.0, error change = 3914.0\n",
      "step 59: theta = [-0.    1.    0.94  0.97  0.09  1.    0.98  1.  ], mse = 64764.0, error change = 3702.0\n",
      "step 60: theta = [-0.    1.    0.94  0.97  0.09  1.    0.98  1.  ], mse = 61263.0, error change = 3500.0\n",
      "step 61: theta = [-0.    1.    0.94  0.97  0.08  1.    0.98  1.  ], mse = 57953.0, error change = 3310.0\n",
      "step 62: theta = [-0.    1.    0.94  0.97  0.08  1.    0.98  1.  ], mse = 54823.0, error change = 3130.0\n",
      "step 63: theta = [-0.    1.    0.94  0.97  0.07  1.    0.98  1.  ], mse = 51863.0, error change = 2960.0\n",
      "step 64: theta = [-0.    1.    0.94  0.97  0.07  1.    0.98  1.  ], mse = 49064.0, error change = 2799.0\n",
      "step 65: theta = [-0.    1.    0.93  0.97  0.06  1.    0.98  1.  ], mse = 46418.0, error change = 2647.0\n",
      "step 66: theta = [-0.    1.    0.93  0.97  0.06  1.    0.98  1.  ], mse = 43915.0, error change = 2503.0\n",
      "step 67: theta = [-0.    1.    0.93  0.97  0.05  1.    0.98  1.  ], mse = 41548.0, error change = 2367.0\n",
      "step 68: theta = [-0.    1.    0.93  0.97  0.05  1.    0.98  1.  ], mse = 39310.0, error change = 2238.0\n",
      "step 69: theta = [-0.    1.    0.93  0.97  0.04  1.    0.98  1.  ], mse = 37194.0, error change = 2116.0\n",
      "step 70: theta = [-0.    1.    0.93  0.97  0.04  1.    0.98  1.  ], mse = 35193.0, error change = 2001.0\n",
      "step 71: theta = [-0.    1.    0.93  0.97  0.03  1.    0.98  1.  ], mse = 33300.0, error change = 1892.0\n",
      "step 72: theta = [-0.    1.    0.93  0.97  0.03  1.    0.98  1.  ], mse = 31511.0, error change = 1790.0\n",
      "step 73: theta = [-0.    1.    0.93  0.97  0.02  1.    0.98  1.  ], mse = 29819.0, error change = 1692.0\n",
      "step 74: theta = [-0.    1.    0.93  0.97  0.02  1.    0.98  1.  ], mse = 28218.0, error change = 1600.0\n",
      "step 75: theta = [-0.    1.    0.93  0.97  0.02  1.    0.98  1.  ], mse = 26705.0, error change = 1513.0\n",
      "step 76: theta = [-0.    1.    0.93  0.96  0.01  1.    0.98  1.  ], mse = 25274.0, error change = 1431.0\n",
      "step 77: theta = [-0.    1.    0.93  0.96  0.01  1.    0.98  1.  ], mse = 23921.0, error change = 1353.0\n",
      "step 78: theta = [-0.    1.    0.93  0.96  0.01  1.    0.98  1.  ], mse = 22642.0, error change = 1280.0\n",
      "step 79: theta = [-0.    1.    0.93  0.96  0.    1.    0.98  1.  ], mse = 21432.0, error change = 1210.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 80: theta = [-0.    1.    0.93  0.96 -0.    1.    0.98  1.  ], mse = 20288.0, error change = 1144.0\n",
      "step 81: theta = [-0.    1.    0.93  0.96 -0.    1.    0.98  1.  ], mse = 19206.0, error change = 1082.0\n",
      "step 82: theta = [-0.    1.    0.93  0.96 -0.01  1.    0.98  1.  ], mse = 18183.0, error change = 1023.0\n",
      "step 83: theta = [-0.    1.    0.93  0.96 -0.01  1.    0.98  1.  ], mse = 17215.0, error change = 967.0\n",
      "step 84: theta = [-0.    1.    0.93  0.96 -0.01  1.    0.98  1.  ], mse = 16300.0, error change = 915.0\n",
      "step 85: theta = [-0.    1.    0.93  0.96 -0.02  1.    0.98  1.  ], mse = 15435.0, error change = 865.0\n",
      "step 86: theta = [-0.    1.    0.93  0.96 -0.02  1.    0.98  1.  ], mse = 14617.0, error change = 818.0\n",
      "step 87: theta = [-0.    1.    0.93  0.96 -0.02  1.    0.98  1.  ], mse = 13844.0, error change = 774.0\n",
      "step 88: theta = [-0.    1.    0.93  0.96 -0.03  1.    0.98  1.  ], mse = 13112.0, error change = 732.0\n",
      "step 89: theta = [-0.    1.    0.93  0.96 -0.03  1.    0.98  1.  ], mse = 12420.0, error change = 692.0\n",
      "step 90: theta = [-0.    1.    0.93  0.96 -0.03  1.    0.98  1.  ], mse = 11766.0, error change = 654.0\n",
      "step 91: theta = [-0.    1.    0.93  0.96 -0.03  1.    0.98  1.  ], mse = 11148.0, error change = 619.0\n",
      "step 92: theta = [-0.    1.    0.93  0.96 -0.04  1.    0.98  1.  ], mse = 10563.0, error change = 585.0\n",
      "step 93: theta = [-0.    1.    0.93  0.96 -0.04  1.    0.98  1.  ], mse = 10010.0, error change = 553.0\n",
      "step 94: theta = [-0.    1.    0.93  0.96 -0.04  1.    0.98  1.  ], mse = 9487.0, error change = 523.0\n",
      "step 95: theta = [-0.    1.    0.93  0.96 -0.04  1.    0.98  1.  ], mse = 8992.0, error change = 495.0\n",
      "step 96: theta = [-0.    1.    0.93  0.96 -0.04  1.    0.98  1.  ], mse = 8524.0, error change = 468.0\n",
      "step 97: theta = [-0.    1.    0.93  0.96 -0.05  1.    0.98  1.  ], mse = 8082.0, error change = 442.0\n",
      "step 98: theta = [-0.    1.    0.93  0.96 -0.05  1.    0.98  1.  ], mse = 7664.0, error change = 418.0\n",
      "step 99: theta = [-0.    1.    0.93  0.96 -0.05  1.    0.98  1.  ], mse = 7268.0, error change = 395.0\n",
      "step 100: theta = [-0.    1.    0.93  0.96 -0.05  1.    0.98  1.  ], mse = 6894.0, error change = 374.0\n",
      "step 101: theta = [-0.    1.    0.93  0.96 -0.05  1.    0.98  1.  ], mse = 6541.0, error change = 354.0\n",
      "step 102: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 6206.0, error change = 334.0\n",
      "step 103: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 5890.0, error change = 316.0\n",
      "step 104: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 5591.0, error change = 299.0\n",
      "step 105: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 5308.0, error change = 283.0\n",
      "step 106: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 5041.0, error change = 267.0\n",
      "step 107: theta = [-0.    1.    0.93  0.96 -0.06  1.    0.98  1.  ], mse = 4788.0, error change = 253.0\n",
      "step 108: theta = [-0.    1.    0.93  0.96 -0.07  1.    0.98  1.  ], mse = 4549.0, error change = 239.0\n",
      "step 109: theta = [-0.    1.    0.93  0.96 -0.07  1.    0.98  1.  ], mse = 4323.0, error change = 226.0\n",
      "step 110: theta = [-0.    1.    0.93  0.96 -0.07  1.    0.98  1.  ], mse = 4109.0, error change = 214.0\n",
      "step 111: theta = [-0.    1.    0.93  0.96 -0.07  1.    0.98  1.  ], mse = 3907.0, error change = 202.0\n",
      "step 112: theta = [-0.    1.    0.93  0.96 -0.07  1.    0.98  1.  ], mse = 3716.0, error change = 191.0\n",
      "step 113: theta = [-0.    1.    0.93  0.96 -0.07  0.99  0.97  1.  ], mse = 3535.0, error change = 181.0\n",
      "step 114: theta = [-0.    1.    0.92  0.96 -0.07  0.99  0.97  1.  ], mse = 3364.0, error change = 171.0\n",
      "step 115: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 3202.0, error change = 162.0\n",
      "step 116: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 3049.0, error change = 153.0\n",
      "step 117: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2905.0, error change = 145.0\n",
      "step 118: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2768.0, error change = 137.0\n",
      "step 119: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2639.0, error change = 129.0\n",
      "step 120: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2516.0, error change = 122.0\n",
      "step 121: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2401.0, error change = 116.0\n",
      "step 122: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2291.0, error change = 109.0\n",
      "step 123: theta = [-0.    1.    0.92  0.96 -0.08  0.99  0.97  1.  ], mse = 2188.0, error change = 103.0\n",
      "step 124: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 2090.0, error change = 98.0\n",
      "step 125: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1998.0, error change = 92.0\n",
      "step 126: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1911.0, error change = 87.0\n",
      "step 127: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1828.0, error change = 83.0\n",
      "step 128: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1750.0, error change = 78.0\n",
      "step 129: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1676.0, error change = 74.0\n",
      "step 130: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1606.0, error change = 70.0\n",
      "step 131: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1540.0, error change = 66.0\n",
      "step 132: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1477.0, error change = 62.0\n",
      "step 133: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1418.0, error change = 59.0\n",
      "step 134: theta = [-0.    1.    0.92  0.96 -0.09  0.99  0.97  1.  ], mse = 1362.0, error change = 56.0\n",
      "step 135: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1309.0, error change = 53.0\n",
      "step 136: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1260.0, error change = 50.0\n",
      "step 137: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1212.0, error change = 47.0\n",
      "step 138: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1168.0, error change = 45.0\n",
      "step 139: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1125.0, error change = 42.0\n",
      "step 140: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1085.0, error change = 40.0\n",
      "step 141: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1048.0, error change = 38.0\n",
      "step 142: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 1012.0, error change = 36.0\n",
      "step 143: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 978.0, error change = 34.0\n",
      "step 144: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 946.0, error change = 32.0\n",
      "step 145: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 916.0, error change = 30.0\n",
      "step 146: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 887.0, error change = 29.0\n",
      "step 147: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 860.0, error change = 27.0\n",
      "step 148: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 835.0, error change = 26.0\n",
      "step 149: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 811.0, error change = 24.0\n",
      "step 150: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 788.0, error change = 23.0\n",
      "step 151: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 766.0, error change = 22.0\n",
      "step 152: theta = [-0.    1.    0.92  0.96 -0.1   0.99  0.97  1.  ], mse = 746.0, error change = 20.0\n",
      "step 153: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 726.0, error change = 19.0\n",
      "step 154: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 708.0, error change = 18.0\n",
      "step 155: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 691.0, error change = 17.0\n",
      "step 156: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 674.0, error change = 16.0\n",
      "step 157: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 659.0, error change = 15.0\n",
      "step 158: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 644.0, error change = 15.0\n",
      "step 159: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 631.0, error change = 14.0\n",
      "step 160: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 617.0, error change = 13.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 161: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 605.0, error change = 12.0\n",
      "step 162: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 593.0, error change = 12.0\n",
      "step 163: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 582.0, error change = 11.0\n",
      "step 164: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 572.0, error change = 10.0\n",
      "step 165: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 562.0, error change = 10.0\n",
      "step 166: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 553.0, error change = 9.0\n",
      "step 167: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 544.0, error change = 9.0\n",
      "step 168: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 536.0, error change = 8.0\n",
      "step 169: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 528.0, error change = 8.0\n",
      "step 170: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 520.0, error change = 7.0\n",
      "step 171: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 513.0, error change = 7.0\n",
      "step 172: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 506.0, error change = 7.0\n",
      "step 173: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 500.0, error change = 6.0\n",
      "step 174: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 494.0, error change = 6.0\n",
      "step 175: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 488.0, error change = 6.0\n",
      "step 176: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 483.0, error change = 5.0\n",
      "step 177: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 478.0, error change = 5.0\n",
      "step 178: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 473.0, error change = 5.0\n",
      "step 179: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 469.0, error change = 5.0\n",
      "step 180: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 464.0, error change = 4.0\n",
      "step 181: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 460.0, error change = 4.0\n",
      "step 182: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 457.0, error change = 4.0\n",
      "step 183: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 453.0, error change = 4.0\n",
      "step 184: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 450.0, error change = 3.0\n",
      "step 185: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 446.0, error change = 3.0\n",
      "step 186: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 443.0, error change = 3.0\n",
      "step 187: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 440.0, error change = 3.0\n",
      "step 188: theta = [-0.    1.    0.92  0.96 -0.11  0.99  0.97  1.  ], mse = 438.0, error change = 3.0\n",
      "step 189: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 435.0, error change = 3.0\n",
      "step 190: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 433.0, error change = 2.0\n",
      "step 191: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 430.0, error change = 2.0\n",
      "step 192: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 428.0, error change = 2.0\n",
      "step 193: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 426.0, error change = 2.0\n",
      "step 194: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 424.0, error change = 2.0\n",
      "step 195: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 422.0, error change = 2.0\n",
      "step 196: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 420.0, error change = 2.0\n",
      "step 197: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 419.0, error change = 2.0\n",
      "step 198: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 417.0, error change = 2.0\n",
      "step 199: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 416.0, error change = 1.0\n",
      "step 200: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 414.0, error change = 1.0\n",
      "step 201: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 413.0, error change = 1.0\n",
      "step 202: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 412.0, error change = 1.0\n",
      "step 203: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 411.0, error change = 1.0\n",
      "step 204: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 409.0, error change = 1.0\n",
      "step 205: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 408.0, error change = 1.0\n",
      "step 206: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 407.0, error change = 1.0\n",
      "step 207: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 406.0, error change = 1.0\n",
      "step 208: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 406.0, error change = 1.0\n",
      "step 209: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 405.0, error change = 1.0\n",
      "step 210: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 404.0, error change = 1.0\n",
      "step 211: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 403.0, error change = 1.0\n",
      "step 212: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 402.0, error change = 1.0\n",
      "step 213: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 402.0, error change = 1.0\n",
      "step 214: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 401.0, error change = 1.0\n",
      "step 215: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 401.0, error change = 1.0\n",
      "step 216: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 400.0, error change = 1.0\n",
      "step 217: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 399.0, error change = 1.0\n",
      "step 218: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 399.0, error change = 1.0\n",
      "step 219: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 398.0, error change = 0.0\n",
      "step 220: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 398.0, error change = 0.0\n",
      "step 221: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 397.0, error change = 0.0\n",
      "step 222: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 397.0, error change = 0.0\n",
      "step 223: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 397.0, error change = 0.0\n",
      "step 224: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 396.0, error change = 0.0\n",
      "step 225: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 396.0, error change = 0.0\n",
      "step 226: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 396.0, error change = 0.0\n",
      "step 227: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 395.0, error change = 0.0\n",
      "step 228: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 395.0, error change = 0.0\n",
      "step 229: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 395.0, error change = 0.0\n",
      "step 230: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 394.0, error change = 0.0\n",
      "step 231: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 394.0, error change = 0.0\n",
      "step 232: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 394.0, error change = 0.0\n",
      "step 233: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 394.0, error change = 0.0\n",
      "step 234: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 394.0, error change = 0.0\n",
      "step 235: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 393.0, error change = 0.0\n",
      "step 236: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 393.0, error change = 0.0\n",
      "step 237: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 393.0, error change = 0.0\n",
      "step 238: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 393.0, error change = 0.0\n",
      "step 239: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 393.0, error change = 0.0\n",
      "step 240: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 241: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 242: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 243: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 244: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 245: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 246: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 247: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 392.0, error change = 0.0\n",
      "step 248: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 391.0, error change = 0.0\n",
      "step 249: theta = [-0.    1.    0.92  0.96 -0.12  0.99  0.97  1.  ], mse = 391.0, error change = 0.0\n",
      "returning after 249 steps...\n",
      "theta = [-3.42724073e-04  9.97929587e-01  9.20811158e-01  9.59835788e-01\n",
      " -1.19639597e-01  9.94857458e-01  9.74125386e-01  9.99517589e-01]\n"
     ]
    }
   ],
   "source": [
    "theta = fit_linear_regression_model(X, y, theta=(0,1,1,1,1,1,1,1), alpha=0.00000001)\n",
    "\n",
    "h = hypothesis(theta)\n",
    "\n",
    "#y_prime = h(X)\n",
    "y_prime = np.apply_along_axis(h, 1, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for a in np.linspace(0, 0.05, 20):\n",
    "#    fit_linear_regression_model(X, y, theta=(0,1), alpha=a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO2df5Ac5Znfv8+MWmIk+1gJtohYJEvGLinGIMleAy6lEksXWz6w8RrLyJRxSJXrdLnkqoxN7Xl1doy44KDLHj+SypVdOPjMFRQWIHuRAZeOQ3JdGUfiJK9kIYNiZH55kEF3YmWDBmm0++SP6V56e/rH2zPdM90930/V1s680/320+/sPvPO81NUFYQQQvJHqdsCEEIIaQ0qcEIIySlU4IQQklOowAkhJKdQgRNCSE6Z1cmLnXvuubpkyZJOXpIQQnLPvn37/llV+73jHVXgS5Yswd69ezt5SUIIyT0i8qLfOE0ohBCSU6jACSEkp1CBE0JITqECJ4SQnEIFTgghOaWjUSgknLHxKkZ3HMYrEzWc31fB8LplGFo10G2xCCEZhQo8I4yNV7HpBwdRq08CAKoTNWz6wUEAoBInhPhCE0pGGN1xeFp5O9TqkxjdcbhLEhFCsg4VeEZ4ZaIWa5wQQqjAM8L5fZVY44QQQgWeEYbXLUPFKs8Yq1hlDK9b1iWJCCFZh07MjOA4KhmFQggxhQo8QwytGqDCJoQYQxMKIYTkFGMFLiJlERkXkUfs598TkedFZL/9szI9MQkhhHiJY0L5EoBnAPyBa2xYVR9KViRCCCEmGClwEbkAwJUAvgngK6lKRFKBafqEFA/THfidAP4cwDs9498UkW8AeALAiKqe8p4oIhsBbASAxYsXtyFqseikQmWaPiHFJNIGLiKfAPCaqu7zvLQJwHIAHwKwAMBX/c5X1btUdVBVB/v7m1q69SSOQq1O1KB4W6GOjVdTuR7T9AkpJiZOzNUArhKRFwB8H8BaEblXVY9qg1MA/hbApSnKWSiiFOrYeBWrt+zE0pFHsXrLzrYVO9P0CSkmkQpcVTep6gWqugTA5wDsVNXrRGQhAIiIABgC8HSqkhaIMIWaxu6cafqEFJN24sDvE5GDAA4COBfALcmIVHzCFGoa5g6m6RNSTGIpcFX9iap+wn68VlUvVtX3q+p1qvpGOiJmh6RMG2EKNQ1zx9CqAdx69cUY6KtAAAz0VXDr1RfTgUlIzmEqvSFJRnKE1T0Z3XEYVR9l3a65g2n6hBQPKnBDwkwbrSjGIIU6vG4Zhh86gPqkTo9ZZaG5gxDSBGuhGNLRSA6NeE4IIaACN6ZTkRyjOw6jPjVTY9enlDHbhJAmqMAN6VQkB2O2CSGmUIEb0qlIDsZsE0JMoRMzBp2I5Bhet2xGtAvAmG1CiD9U4BmDrdUIIaZQgWcQxmwTQkygAs8ZrOtNCHGgAs8RrOtNCHHDKJQcwbrehBA3VOA5gjHihBA3NKFkiCj79vl9lVQKXRFC8gl34BnBpJED63oTQtxQgWcEE/s263oTQtzQhJIRTO3bjBEnhDgY78BFpCwi4yLyiP18qYjsEZHnRGSriMxOT8ziwxoohJC4xDGhfAnAM67nfwXgDlV9D4DXAXwxScF6Ddq3CSFxMVLgInIBgCsB/B/7uQBYC+Ah+5B70OhMT1qE9m1CSFxMbeB3AvhzAO+0n58DYEJVz9jPfwPAV9OIyEYAGwFg8eLFrUvaA/jZt5k6TwgJInIHLiKfAPCaqu5r5QKqepeqDqrqYH9/fytT9CwmoYWEkN7FZAe+GsBVInIFgLMA/AGA/wmgT0Rm2bvwCwBQqyRMK42U3Tv2vrkWVIETtTp374QUkMgduKpuUtULVHUJgM8B2KmqnwewC8B6+7DrATycmpQ9StzUee+O/fWTdUzU6ty9E1JQ2knk+SqAr4jIc2jYxO9ORiTiEDe00G/H7oaFrwgpFrEUuKr+RFU/YT/+tapeqqrvUdXPquqpdETsXeKGFpoUtWLhK0KKA1PpM0zc0EKTpB8mBhFSHERVO3axwcFB3bt3b8euVyS84YRrlvdj17PH8MpEDWdXLIg0bN5hVKxyR2LLGfpISLKIyD5VHfSOsxZKDvDrxHPv7pemX5+oBSvuilXCW/WpjilSdg0ipHNQgWecsfEqbnzgACZb/Ka0YN4cPDmyNmGpgmkl9JEQ0hq0gWcYZzfbqvIGOu+0ZNcgQjoHd+AZw20/Lom0pbyBzjst2TWIkM7BHXiG8CbitKu8u1HNkFUVCekc3IFniKhEHDezy4LTk80KviTAlDZCDrsR/eFcj1EohKQPFXiGiGMn9lPeAFAuCW5fv6KrCpNdgwjpDDShZIgk7MT1SWW6PCE9QqEV+Nh4Fau37MTSkUexesvOzBdy8rMfSwvzMOKDkN6gsCaUPCaU+NmP1yzvx7Z9VWPbOOC/k/fLjvRei7ZqQvJFYVPpV2/Z6RvONtBX6WhiSxK4le9ZVgm1+lTgsVZZsOFDi6bT7IM+BBxnpxsBoOieA5QQ4k9QKn1hTShFSigZWjWAJ0fW4o4NKxFlVLl0yXxs21ed0cXnvt0vNe3gvcobaChvgLXDCckLhVXgcWtp5wGTMMPdv3696ZhWvmOxdjgh2aewCryICSUm3x7aTf6Jez1CSPcorAKPW0s7D5h8eyiLv4mllWiWPH9bIaQXiIxCEZGzAPwjgDn28Q+p6k0i8j0A/w7ACfvQ/6iq+9MStBXynFDiFzUyvG7ZjMgaL1ZJsOHSRU0OS8c5WbZrq/RVLLx5+gzqAclAQP6/rRDSC5jswE8BWKuqKwCsBPBxEbncfm1YVVfaP5lS3nnGWxPFHQLpfKsAGpEkDn0VC6OfXYFbhi6ecYyjvIGGeaVilbH5qouw4UOLAq9fFsn9txVCeoHIHbg24gzfsJ9a9k/nYg97hKgqhI5T8cmRtZGK1fnm4RdKGeWcFAC3XdPdVHxCiBlGiTwiUgawD8B7APyNqu4RkT8F8E0R+QaAJwCM+DU2FpGNADYCwOLFixMTvEh4k46CHJHViRpWb9k5o43axMl6YBJOK6GUiuwmOhFCZmLkxFTVSVVdCeACAJeKyPsBbAKwHMCHACwA8NWAc+9S1UFVHezv709I7GIRpwqhY1aZqNXx+sn6DBOLN247LJQy6LUBOi4JyQ2xolBUdQLALgAfV9Wj2uAUgL8FcGkaAvYCSYTr1eqTuPGBAzPqvoSFUhYxzJKQXiNSgYtIv4j02Y8rAD4K4FkRWWiPCYAhAE+nKWiRSSpcb1I10OnpDaUsYpglIb1GZC0UEbkEwD0Aymgo/AdU9S9FZCeAfjT8XvsB/CdVfSN4ps7WQskTXht4UuSx7gshpJmgWigmUSi/ALDKZ5yaISG8VQiT6IUJNHbiY+NV7qoJKSiFLSebN9xJR0nuyLNeQpcQ0jqFTaXPM459OglYlIqQ4sIdeMZwJ/SUEzSlEEKKBxV4F/HWO/E2XkiysqDXFu5Xa4VmFkLyBU0oXcKv3olf4wWgUZtEEFxp0AS3GSWo1gobOBCSL6jAu4Rf9mXQfntSFc9vuRJTbezI3clCftemrZyQ/EEF3iXiZl++77/+OLKCWNj+3J0sVKR2c4T0MlTgXSJu9uXJkEbGDkEK3ioLhtctw9h4Fau37Aw8jg0cCMkXVOBdwq8WSRpYJWB0/QoAmLZ7+8E6KITkD0ahJIxpdIc3+/L8vgpeOVFDgoEnAIAz9sY9rOLhAKNQCMklkbVQkqTotVD8MigrVtm4SNSSkUdTkWugr4JX7IgTLwLg+S1XpnJdQkgyBNVCoQklQaKiOxwbtLvkq5u0anE7O3w/aPcmJL9QgSfE2Hg10L5cnahh1V/+PW7Yuj809nrN8v6mSJKKVcZcq723qWKVWP+bkAJCBZ4AjukkCAHw+sl607h3d75tX7XJzFESswiUME7Wp7D3xeOs/01Iweg5J2YaKeRhDkJ3V3g/nNjroDnePJ1MjfD797yMW4aosAkpEj2lwL1ORnfnmnYUW1ST4DAcG3TaSTRJ1lUhhGSDnjKhpJVCHtYgOMwx6bZBp+1MbKeOCiEkm5j0xDxLRJ4SkQMickhEbrbHl4rIHhF5TkS2isjs9MVtj7RSyOM2DwaAvoo1wwbdSmLPvNnlpnOC3tBrL1sUa25CSPYxMaGcArBWVd8QEQvAT0XkxwC+AuAOVf2+iHwbwBcBfCtFWZuIa88+v6/iGynS7u7XLylnzfL+6ednVyycZZUwcbI+Q04nrNA55zMfHMCuZ48Z1+8+eXoSd2xYidEdh1GdqEEEmPJYSkSAz1+2GLcMJdMgghCSHSJ34NrAaVZs2T8KYC2Ah+zxe9DoTN8xWimJmmYo3dCqATw5shbPb7kSw+uWYdu+6rRsE7U63qpP4Y4NK/HkyNpp5T384IEZ8m996mWsWd4Pq2xm7ji/r4KhVQMYXrcMVll8szhVgUcOHGWpWEIKiJENXETKIrIfwGsAHgdwBMCEqp6xD/kNgI6GN7Riz3ZalaUdSmci2+bth1D3bJfrU4r7dr+E+mS0w9H9wTO643DoORO1etOHW1RSESEk+xhFoajqJICVItIH4IcAlpteQEQ2AtgIAIsXL25FRl9atWe7mwenhYlsE7XmuHAgOmoFaNjPN1910fR9mNjwnQ8QZ/efRjQOIaSzxIpCUdUJALsAfBhAn4g4HwAXAPDdwqnqXao6qKqD/f39bQnrJsup4WnLNm/OrBmK1nTesJhzNnQgJH+YRKH02ztviEgFwEcBPIOGIl9vH3Y9gIfTEtKPLKeGR8k2Nl5FKcDMPW92OdIGXp2oTZs9xsarOP7mKSO5+uZaWL1lZ6CTlA0dCMkXJiaUhQDuEZEyGgr/AVV9RER+CeD7InILgHEAd6coZxN+kR9ZKYkaJptjvvBGizh8+gMDGHzXAnzthwdDszCrEzUMP3QAk5MKk0R7qyx4460zvin9DqY7eTZEJiQbsJxshwnbAQMNx+qTI2sjj4uir2Jh3pxZ00r2zVNnAu3ugHnZ23ZL5hJC4hNUTranUumzQJSZwnm9XXPGiVod+2/62PTzpSG1xuM0dAizn1OBE9JZeiqVPgtEmSmc19t1eHrPD0v3d2LTTWBDZEKyQ2F34Fmx07rl6Jtr4c1TZwKPtcqCN0+dwdKRR3F2xYqsZBg2j9eZO7xuWZPpQwAsOacyIxu0W9mshJD4FHIH3kqWZifkeP1kHacDEm7mz7UAbcSHO9mbpsq74mr4MH+uhdH1K5qU8NCqAXzmgwMzGkYogCePHM9MNishJB6F3IFnxU4bVifcjVOxMCxCJBzBnRtWRt7brmePRX4oRK1TlqN/COk1CqnAO22nDTLXmF6vXblMP5ySkqcT2ayEkGgKqcD75lq+u9k07LRhaelB9uIgudoJG4xSuo3kITFq7EB7NiH5oHAKfGy8ijfeanYU+jn2kiDMXLNmeT/u3f1S6PlW6W25vE7GOCiAr48dxK5nj80oaeuUpzV1iNKeTUh+KJwCH91xuKnKHwDMmz0rla/9QTvf6kQNW//p5dBzK1YJt159yQy53KaYJedU8LMjx42dme4Pi+pEbcZzvznKIrj2skUzlL47Y5R2bkKyTeEUeJBCPRGShdgOQWaSskhkWdgF8+bMUIp+tuV2MzLDmFL1bfTQjWqF/MAgJD6FCyPsdJVCv7A6q2Rma3YXpQoizQSZoDXpdLXCrIR9EpI3CqfAOx2n7G0S0VexgBj9g6OUVVofPGFr0ukoHpa3JaQ1CqfAO9V1x3tNp53avDmzjDrquAlTVq00Ow7C+VyJWpNOf4thej4hrVE4GzjQ3TjlMKUjAt++lUBjJ77ELjhV8mlObMJcq4Q5Vnm6efKScyrY/evXMak67bAMa27s2KH9olbS/BbD9HxCWqNwO/BuE1Y06o5rVhpZV1pR3gBwsj6FN946g765FqoTNfzsyPFpW/ykKrbtqwaaatx2aKChvE137O3C9HxCWoMKPGHClNHm7YdaKk4Vh/qUTicxea9Vq0/ixgcO+CpxPzu0In61wlbohtmLkCJQSBNKN0PSgmqFAMGNjDvJpKpvSGC37dBMzyckPoVT4FnouB4Uz50V/Gqn0A5NSP4waWq8SER2icgvReSQiHzJHt8sIlUR2W//XJG+uNGkGZI2Nl7F6i07sXTk0cj4bS9Zi6jwykM7NCH5w2QHfgbAjar6cxF5J4B9IvK4/dodqvrX6YkXn7RMAe3u7IMKbCVB2bBIlRvvzpplYgnJH5EKXFWPAjhqP/69iDwDILP/1WmZAtqtMZ5W7+iyCI7cekWslHsBfHfWtEMTki9iRaGIyBIAqwDssYf+TER+ISLfFZH5AedsFJG9IrL32LFjbQlrQlqmAJOdfZiJJa1aLM7OO843DEXn/AGEkPQwVuAi8g4A2wDcoKq/A/AtABcCWInGDv02v/NU9S5VHVTVwf7+/gREDietkLSo7MSoeh5pOQPLIrHnH6BjkpBCYBSFIiIWGsr7PlX9AQCo6quu178D4JFUJGyBNEwBfk2B3Tv7IBPL5u2HArMbk+DayxZhbLwa2izZDR2ThBSHSAUuIgLgbgDPqOrtrvGFtn0cAD4N4Ol0RMwGUU6+IBPGRK0+Hf+dtPK+7vLFGHzXAgw/eMC3BjrQqDleEsGbpxsfLnNmMXeLkKJgsgNfDeALAA6KyH577C8AXCsiK9HQSy8A+JNUJMwQYTv7sytWxxN1bhm6GCtv/vtA5Q0Ac2aVcerM1PTziVq943HxhJB0MIlC+Sn8C6Q+lrw4+WRsvIo3T5uZMJIm6kPD73W/6Bk2VCAkfxQuE7MbjO447FtCttWqgib0Vay2zvdGz3Q7e5UQEp+eVeBJ7jiD7N9pKW8A2HzVRQCA+S0mCLmjVtqNcW8F7vgJaZ+e9Ggl3cIrboigVZaWd9AVq4Q7N6ycVnY3ffKiFuaYGYnS6UJWbKFGSDL0pAJPul7K8LplkXW+yyLTcemj61dg/00fixWPPdBXwQtbrsQz/+2Pmhohx8EvLr7THXjYQo2QZOhJE0rSO86hVQO4Yev+0GOmVPH8litbvl7QsWPjVeNaKE4K/eiOw/jy1v3TpouoGPek6XbpWkKKQk/uwNPYcUaZRPzmjnM9v2MdU4RpIauzK5av6QJARxsqdHrHT0hR6UkFnnS9lKgwwqC5TRsWB53vZ4oIm0MEoc5KpzFz2h14WLqWkGQQTatMng+Dg4O6d+/ejl0vjKAoiFaiI8IqAQ70VbBmeT92PXvMd86x8So2bz80I157rlXC7FllnKjVUbFKqJ2ZmlHNcMCe48tb94dmdzqmlQG7wfGTR477HidAk3knbdzrfHbFggimmzEzIoWQmYjIPlUd9I73pA0c8M+qbDUeOsh269ico+Z0Z0oCgEKw+aqLsPfF47h390tN8zpzBNUYFwB3uCJVvj520Hceh26YLpz1Zww6Ia3TsztwP4J20n0VC/tv+ljs85wok6A5582ZFbpz/+2Jt0Lt230VCydqdd9duNOMGAAu3PRY4DxWSfCOs2Z1bfcbtnaO/IT0OkE78J60gQcRVpAqLEY5zKYbNmdYA4ZXJmqRzskg5e2c7xA6jwCvn6x3LR6bESmEtA4VuIswU0JYjHJYDfJWzRMm553fVwmMfnGf79QM98NbAqDT8diMSCGkdXrKhBLloBwbrwbGc7fq6PPaeE2oWGXcevXFobHlYbXFrZJgw6WLph2nFauEk/WpgKP9cafoz7VKUAA1e475cy3c9MmLEjG1+K2Pc/+0gRPSoOdNKCbp20OrBjB/bvSONg5+u/OgawAzd+9hseVhH7uzZ5WwbV91+l5P1qdQEsDZiJdFcN3li0MzQd3O0ZP1qWnl7bw2/NCBREwtaXVQIqQX6JkduKmzrBM7wrBrAJju4JM0Jvcap2sQHY2EdIaeDyM0dZZFdd5pB2/s81lWaUb0B4DY5pY4mNxrnA8OOhoJ6S49o8CDlJOfaSSNnpre3e5ErY6KVZ4Rr716y85ElLcI4PfFyuRew5KSTOaLC8vKEtI6kTZwEVkkIrtE5JcickhEvmSPLxCRx0XkV/bv+emL2zpBaesnT59p25Y7Nl7F6i07sXTkUazesnN6Pvf4jQ8ciKzAl9SOVrW5hZIAWLO8P/Lc4XXLYJWjais2SuIuOaeCCzc9hiUjj+LCTY/h62MHY8nJsrKEtIeJE/MMgBtV9X0ALgfwX0TkfQBGADyhqu8F8IT9PLM4zjKvY/D1k/W2lEaQEvr62MEZ40Gx2G6lnWTonPdqCmDbvmrkfQ6tGsDo+hWYa739pyEAXE8xf66FS5fMx5NHjk/f16Qq7t39UiwlzrKyhLRHpAJX1aOq+nP78e8BPANgAMCnANxjH3YPgKG0hEyKoVUDmDen2WrUjtIIUkL373nZyBziVtqmxa1aJc59qmsPrwBmlcu4c8NKvLDlSox/42PY/evXfc+7f8/LxvIwiYeQ9ogVRigiSwCsArAHwHmqetR+6bcAzgs4Z6OI7BWRvceOHWtD1GRIWmkEnWdS4tVbgc8dUpcWJvdpsjMOuj/T0rZj41WUAhKMmMRDiBnGTkwReQeAbQBuUNXfieufT1VVRHz/c1X1LgB3AY0wwvbEbZ84zsx25gtqslAWwZRqYCKR49BLi6C64iaRKG65wu4virA65q2WlaUzlPQiRjtwEbHQUN73qeoP7OFXRWSh/fpCAK+lIWCQg7BVkq5FHWT2CFJOt12zwrfm9th4FcMPHpi2mQcx28DBGMabp2Y6bf1s+EFXcCv/ay9b5HvM5e+O9mWH1TGfMyt+bhmdoaRXMYlCEQB3A3hGVW93vbQdwPX24+sBPJy0cGn8Yyad+efMF6RYrRKMrrN5+yHUI9rYl0uC05PtfYmZqM102vopU0VzFIv3Q+6WoYux+sIFTfP//KUTke9P2DcMr3wm0BlKehUTE8pqAF8AcFBEnOIcfwFgC4AHROSLAF4EcE3SwoX9Y7bz9TjpOO+wnpj1KeAFVw2VoK/67oYOfpRFMBmh4APP83wbcK9hkDJVND50wkwSL/xL87km709UwlDc95jOUNKrRCpwVf0pmjdkDn+YrDgzKdo/ZljzgihMnYNuKlY50FThrGGQMjVJk2/1/fFrohx3DvcHYSnAHk9nKCk6mS5mVbRSo2HfKMIKXLVKWESLs4bt+ARafX9Mom3C5vCa1pJ0hhKSJzKtwPPU/NbPHuwdD9ux3vTJi4wyIE257vLFGFo1ELmGQT4BAJHO46C51yzvjzzXaaJ854aVsd/jICdoWcTI35C0Y5yQbpH5aoSdDA8zuVbYMZ//zv9tahzsNCAGgBsfOBBqChEAc2eX8ebp9uqhrL5wAe774w/7ynx2xUJ9cmr6Gu4GymFFtZwqhe5Gyc6x7vVYs7wf2/ZVmxRsX8XC5qsumu6D6W7kPG92GVa5NEOGsPd46cijvpE67prtYU2rWX+c5I2gaoSZV+CdwuQfu9VjrJIA0tz9Joh5bSrxIIU0Nl7F8EMHQuWoWGWcZZV8myV7cZT6gEtBhhXDqlhlfOaDA9j61MtNETdWWTC6foWREo0qDRz2PgWV6mVpXJJler6hQxQmoWitHlOfUmPlDQAnT082mRWskhibWIJC6EZ3HI6Uo1afNFLewNv1VqoTNQw/2GjwEOZ8dEoM+IVL1ifVOOwvyiwU9j4VzTFOehsqcBuTf+x2jomDAk126dHPrsDo+hXGzk6/XWaaSqo+pdi8/VCkAzPMhGQqX1Qsf9j7VDTHOOlteqYeeBQmKfbtHBOHskhgrProjsNGO2S/lPYkZAtjolbH5qsuCg0RDErBd+QzJSyWP+x98gthzKpjnJAouAO3MYmoOHn6TMOe7ULQ2O060QxxKgoGRa5MqgZGR5juUp053JEWprW+28HZHft9U6hY5cBUe6ssGF63LJEIkTATC3twkiJBJ6YLb+SCX0SFVRbMmz0LE7V6U/9Ib1/LsCST+XMtjH/jY/j62EHcv+flwFhmr3Ix7ZgTJtvNPzo0o+N87cyUbwefuDj35GCyno4M//3qSwA0R7+0GiHC4lakSBQmCqWT/5hh0Q6Av5251SbJ7TRd9hLUmDgo0sI3cqYsgCKyPov7eCeKJOg9CrpHp0Jj0IcdI0RIr1OIpsZhqehpKPFWIhZabZLcTtPlNcv7sevZY7HKwZrICCCwxosbdxhh2HsUVTvdpGsRIeRtcqXA0ypuFUSU09LvtbMrFlZv2dmkrKPkS7LpctBON8xJGOY0DTPZeHfHYe9Rq05URogQ4k+uFHinY3ijIha8STElAX5/6sx0hmF1oobhhw4ACP6G4JgbnDrc3j2oU7/bnSjkt1OOsjW75fZmQs6fa+GmT17kK+Oa5f24b/dLviYZv+iNoPeiOlHDnRtWRpp/TK7h3EOSprS0TXO0yZM0yJUCT7qbThRh5o+x8WqTtp1SwOsNrE8qbv7RId9/Vq+5wU9JOvWxHbzmieEHD8zI8qxO1LBtXxWf+eDADLOKW+7hBw/MsG2/frLu+0EzNl7Ftn3VQHu6nxIKeo+c2BcnGzLMwRvWtchv3do1paVtmuu06Y/0DrlyYmapjoVpNIiDuyZ4K3OEOU6Djvdz/IVd03uOqWPVzdh4FV/eut/Iidrq+9mKXGEkPV+n5yfFpxCp9FmK4U3CbBNnjlcmarGOr07UfGOpw+aoTtSMjg2bY2jVQGBLOCde3pELQFPMuElLtSRNaWPj1dhO37gwfZ+kRa5MKEDy3XRaJY5Drq/in/4eZ44wx2kQ7jZ0QGPtoq5pcmyUyWogxIzijDtyfeaDA3irPjV9jNtk1EqmZRycbwBBJGWa67Tpj/QOJj0xvysir4nI066xzSJSFZH99s8V6YqZPfyy/ayyNC2oVRJsvuqiwDm8mZ1Ao/elG0HDmeh7TYMiV7X6JG7Yuh9LNz0a+QHgLoQVlFXqbYzsZc3y/qYWTn4O2lp9Evfufil2P8tW68R7szy/9sNgh2rUfM5cS0YexYWbHsOSiLrp3vfIyTz1k4v1yYkpJiaU7wH4uM/4Haq60v55LFmxso+fOWd0/QrcvmFlUxGqsG8MUz5j3t6XCmDbvqdORLkAAAuQSURBVMY/dVCRK2csDFN3h6Pkg9LiwxoP+zk+gxKLwogy08Q1pfk1yA4r2RvVEMKZC3g7fj206bZ3ATRYrnYbd5PewciJKSJLADyiqu+3n28G8Iaq/nWci2U9lb7TxHWEmji94s7pR1kER259+0tVHCdcWLZlnL6eSTv44jqMw64dNVccZzBgltFLeps0nJh/JiK/sE0s/hWKGhfeKCJ7RWTvsWPH2rhc8YjrxDI5Pk4xrSC8itbECeeYAYIU26SqsVxpVAeMs9ZR146ay/t62PrRwUnaoVUF/i0AFwJYCeAogNuCDlTVu1R1UFUH+/v7W7xcMYnrxDI53jEv+JWT9RJ0jLfZcFQNba9JIWjOMLlM+1m2iula91Uso6zZOK+HrR/rk5N2aEmBq+qrqjqpqlMAvgPg0mTF6g2CyruWgKbxOLvSoVUDuO2aFaHOzYpVxrWXLTJyBrbSAcfvWEcuv7luu2YFnt9yJZ4cWZtKlJHJN5OKVQ50OJvOFXf98tS4m2SPlsIIRWShqh61n34awNNhxxeBNFKhnfPd5V2d5r/e8TmzStj74vHINHq3XE7ZWy/u1PnBdy2YTuUvi6BWn8Tm7Ydw848OYeLk202G3RmU3utEfd2v1Sdx848OTT+fM6s0rfDD0vjD1ryV98N73SsvWeibrRqFO0PXWTd3o2fvHCYFzZzX+uZaUAW+vHU/RnccbjpubLzq+/eShdBa4k+aZRQinZgicj+AjwA4F8CrAG6yn69Ew5f+AoA/cSn0QPLqxOxGBqhJ2Vi/kq9O42C/utvuY6IaMYcd78XUQViSRoiku35MWAPmoDUH4tUNz1IGbxhRcgY1pbZKEhntRLpDUn97LTsxVfVaVV2oqpaqXqCqd6vqF1T1YlW9RFWvMlHeecakmXEnrumlPqlN9bqdxsFh55o0Yg473oup43RK0aR8whowB6153PejG+9fK0TJGdSUuj5l3hCadJa0//Zyl4nZDboRKdDO3CbhenEbMUfFZQNmtcNN506iFns7c3WDKDlbuXfSXdL+28tVLZRu0Y1IgXbmNolA8TZibleeoVUDTdErpvjNnWTkRl4iPaLkDJM3a/dCGqT9t8cduAGd7GQeVR/cTbkkTVmbVlmw4UOLQm3gJQFOnj6DpSOPTtcP3/rUy4Ht09yNm4Mcp3tfPI6jJ+LvKpwyASb9M51jB9+1IPT9cM91dsVCfbI539Xv/XOvfZRjMg0i68+vWxZoA2fUSjZJW3fkqpxsN+lEQf44/S7nz7VwolaHV+eWANy+YSWA6G46DlZJMIXmFH739aaP9XGcluBfEsAUqyQzapoDjT/yDyw+Gz87cjyycbS33nnUGvpFboSd10mHZ9TfGaNQ8kcSuqMwTY2LjGk0R1RtcHcadjup9XHT35Mm6PphaeYm9xunBIDJNQlJm0I0NS46po6NOKncaTtD06SVJsetOmTjpscTkgXoxMwQpo6NMEeed560naFpEnT9dp15cZymceYlpNNQgWcI03RvJwXbL1Xe69Dyq81tQlCqvR9Bf0SrL1zgW+/ci1WSphroVlmMU/3dRK1h0PlB62lyTUK6BRV4hvCrc33d5Yt9614PrRrA6PoVM2p191WsGRl5QbW5V1+4IDTkb97shtPulqGLcevVF/t2FHJU3UBfBbdvWInrLl88vWMui+C6yxfjvj/+MEY/u2LG+fPnWk33tOHSRc1/iAoMvmtB7Lrf3jXsq1iYP9cyO9/HYtNXsTKXsUmIA52YBSaqjveFmx4L7Arfaj3wNOTsBFmQgZAgCtHUmMQjKgssyEnYSj3wdshCpmQWZCAkLlTgBSYqCyysNnecedolC5mSWZCBkLhQgecUk0a4UbWmr71ske/c3nGTmtXtNObtdk3ssfEq3jx1pmk8SAY2ISZZgXHgOcSbNeg0wgUww9kWVYf6lqFGRuP9e17GpCrK0oj8cMZN5zGVJwiTetlpEZSBGVSnvN17JSRJ6MTMIVlzuGVNnjjElT3P90ryC52YBSJrDresyROHpErQ5uFeSfGgAs8hWXO4ZU2eOBS1NC3pDSIVuIh8V0ReE5GnXWMLRORxEfmV/Xt+umISN912+mVdnjjElT3P90qKh4kT83sA/jeAv3ONjQB4QlW3iMiI/fyryYtH/OiE0y9OCcy48nSiNK8pcWXvpsOVEC9GTkwRWQLgEVV9v/38MICPqOpREVkI4CeqGrkFoRMzH6TZBDgvDYYJyRJJOzHPczUy/i2A81qWjGSONBux5qXBMCF5oG0npja28IHbeBHZKCJ7RWTvsWPH2r0c6QBpRlowioOQ5GhVgb9qm05g/34t6EBVvUtVB1V1sL+/v8XLkU6SZqQFozgISY5WFfh2ANfbj68H8HAy4hCg+6naaUZaMIqDkOSIjEIRkfsBfATAuSLyGwA3AdgC4AER+SKAFwFck6aQvUQWUrXTjLRgFAchycFU+ozBVG1CiBem0ucEOvkIIaZQgWcMOvkIIaZQgWcMOvkIIaawHnjGoJOPEGIKFXgGcbrOE0JIGDShEEJITqECJ4SQnEIFTgghOYUKnBBCcgoVOCGE5JSOptKLyDE0aqeYcC6Af05RnLzB9ZgJ16MZrslMirQe71LVpnKuHVXgcRCRvX65/70K12MmXI9muCYz6YX1oAmFEEJyChU4IYTklCwr8Lu6LUDG4HrMhOvRDNdkJoVfj8zawAkhhIST5R04IYSQEKjACSEkp2RCgYvIZ0XkkIhMicig57VNIvKciBwWkXWu8Y/bY8+JyEjnpe4cvXSvDiLyXRF5TUSedo0tEJHHReRX9u/59riIyP+y1+cXIvKB7kmeDiKySER2icgv7f+VL9njPbkmInKWiDwlIgfs9bjZHl8qInvs+94qIrPt8Tn28+fs15d0U/7EUNWu/wD41wCWAfgJgEHX+PsAHAAwB8BSAEcAlO2fIwDeDWC2fcz7un0fKa1Nz9yr577/LYAPAHjaNfY/AIzYj0cA/JX9+AoAPwYgAC4HsKfb8qewHgsBfMB+/E4A/8/+/+jJNbHv6x32YwvAHvs+HwDwOXv82wD+1H78nwF82378OQBbu30PSfxkYgeuqs+o6mGflz4F4PuqekpVnwfwHIBL7Z/nVPXXqnoawPftY4tIL93rNKr6jwCOe4Y/BeAe+/E9AIZc43+nDXYD6BORhZ2RtDOo6lFV/bn9+PcAngEwgB5dE/u+3rCfWvaPAlgL4CF73Lsezjo9BOAPRUQ6JG5qZEKBhzAA4GXX89/YY0HjRaSX7jWK81T1qP34twDOsx/31BrZX/9XobHr7Nk1EZGyiOwH8BqAx9H4pjqhqmfsQ9z3PL0e9usnAJzTWYmTp2MdeUTkHwD8K5+XvqaqD3dKDlIMVFVFpOdiYEXkHQC2AbhBVX/n3kT22pqo6iSAlSLSB+CHAJZ3WaSO0zEFrqr/voXTqgAWuZ5fYI8hZLxohK1Br/GqiCxU1aO2OeA1e7wn1khELDSU932q+gN7uKfXBABUdUJEdgH4MBqmoln2Ltt9z856/EZEZgE4G8C/dEXgBMm6CWU7gM/ZHuSlAN4L4CkA/wTgvbbHeTYaTontXZQzTXrpXqPYDuB6+/H1AB52jf8HO/LicgAnXGaFQmDba+8G8Iyq3u56qSfXRET67Z03RKQC4KNo+AV2AVhvH+ZdD2ed1gPYqbZHM9d024tqr+Gn0bBXnQLwKoAdrte+hoZt6zCAP3KNX4GGJ/4IGmaYrt9HiuvTM/fquuf7ARwFULf/Nr6Ihs3yCQC/AvAPABbYxwqAv7HX5yBckUxF+QHwb9Bw0v0CwH7754peXRMAlwAYt9fjaQDfsMffjcYm7zkADwKYY4+fZT9/zn793d2+hyR+mEpPCCE5JesmFEIIIQFQgRNCSE6hAieEkJxCBU4IITmFCpwQQnIKFTghhOQUKnBCCMkp/x+5uWEOiGavgQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.subplots()\n",
    "ax.scatter(y_prime, y)\n",
    "#ax.plot(X, y_prime)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}