03_GradientTape.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.2.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(f\"TensorFlow version: {tf.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GradientTape\n",
    "GradientTape records operation for Automatic Differentation. With this, tensorflow can compute the derivative of a function with respect to any parameters.\n",
    "\n",
    "\n",
    "\n",
    "### 1 Example\n",
    "Given a dataset $S = [(xi\n",
    ", yi), i = 1, . . . , m]$. with $xi ∈ R$ and $ yi ∈ R $ and the mapping  $f:x \\to y$ described as: $ y = w_0 + w_1x $, where $w_0, w_1 ∈ R $ are two parameters to determine the best line to fit the data. Using a open algorithm to find the solution we defined our loss ass:\n",
    "\n",
    "$$J = \\frac{1}{m} \\sum_{i=1}^m(\\bar{y}-y_i)^2 $$\n",
    "\n",
    "When $J(w0, w1)$ is near zero, it means the proposed line can fit the dataset and model an accurate relation between $xi$ and $yi$. The best line with parameter $(w_0^*,w_1^*)$ can reach the minimum value of the error fucntion $J(w0, w1)$:\n",
    "\n",
    "$$(w_0^*,w_1^*) = argmin_{w_0,w_1} \\;\\; J(w_0,w_1)$$\n",
    "\n",
    "From here we know the solution is\n",
    "\n",
    "$$ \\bigtriangledown J(w_0,w_1) = 0 $$\n",
    "\n",
    "Since:\n",
    "$$J = \\frac{1}{m} \\sum_{i=1}^m( w_0 + w_1x-y_i)^2  $$\n",
    "\n",
    "$ \\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$\n",
    "$\\frac{dJ}{dw_1} = \\frac{2}{m} \\sum_{i=1}^m( w_0 + w_1x-y_i)x $ \n",
    "$ \\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$\n",
    "$\\frac{dJ}{dw_0} = \\frac{2}{m} \\sum_{i=1}^m( w_0 + w_1x-y_i) $\n",
    "\n",
    "In order to solve this simple linear regression with TensorFlow, we will only compute forward pass, since tensorflow can handle backpropagation by itself we will let it do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TRAIN SET:\n",
      "X:[0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]\n",
      "Y:[0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]\n",
      "\n",
      "TEST SET:\n",
      "X:[10. 11. 12. 13. 14. 15. 16. 17. 18. 19.]\n",
      "Y:[30. 32. 34. 36. 38. 40. 42. 44. 46. 48.]\n"
     ]
    }
   ],
   "source": [
    "### Set Training and Test set ###\n",
    "X_train = tf.constant(range(10), dtype=tf.float32)     # Create a R=1 Tensor from 0-9\n",
    "Y_train = 2 * X_train + 10                             # Create a R=1  maped by the function 2x + 10\n",
    "\n",
    "X_test = tf.constant(range(10, 20), dtype=tf.float32)  # Create a R=1 Tensor from 10-19\n",
    "Y_test = 2 * X_test + 10                               # Create a R=1  maped by the function 2x + 10\n",
    "\n",
    "### GET TO KNOW DATA ###\n",
    "print(\"TRAIN SET:\")\n",
    "print(f\"X:{X_train}\")\n",
    "print(f\"Y:{X_train}\\n\")\n",
    "\n",
    "print(\"TEST SET:\")\n",
    "print(f\"X:{X_test}\")\n",
    "print(f\"Y:{Y_test}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dw0: -38.0, Validation: -38.0\n",
      "dw1 -204.0, Validation: -204.0\n"
     ]
    }
   ],
   "source": [
    "def loss_mse(X, Y, w0, w1):\n",
    "    \"\"\" 1 Layer Forward pass\"\"\"\n",
    "    Y_hat = w1* X + w0             # Forward Propagation\n",
    "    errors = (Y_hat - Y)**2        # Compute  Squared error\n",
    "    error = tf.reduce_mean(errors) # Sum over erros to obtain a constant (r=0 Tensor) and divide by mean\n",
    "    return error\n",
    "\n",
    "def compute_gradients(X, Y, w0, w1): \n",
    "    \"\"\"Saves loss_mse in a tape\"\"\"\n",
    "    ### Use a contex Manager to obtain tape ###\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss = loss_mse(X, Y, w0, w1)         # Perfrom operations\n",
    "        \n",
    "    return  tape.gradient(loss, [w0, w1])  # Save Forward pass and variables you wish to obtain gradients\n",
    "\n",
    "def gradient_validation(X, Y, w0, w1):\n",
    "    \" THIS IS JUST A VALIDATION GRADIENT\"\n",
    "    x = X.numpy()\n",
    "    y = Y.numpy()\n",
    "    w_0 = w0.numpy()\n",
    "    w_1 = w1.numpy()\n",
    "    suma_w0=0\n",
    "    suma_w1=0\n",
    "    for xi,yi in zip(x,y):\n",
    "        suma_w0 += (w_0 + w_1*xi - yi)\n",
    "        suma_w1 += (w_0 + w_1*xi - yi)*xi\n",
    "    \n",
    "    suma_w0 = (2*suma_w0)/len(x)\n",
    "    suma_w1 = (2*suma_w1)/len(x)\n",
    "    \n",
    "    return suma_w0,suma_w1\n",
    "\n",
    "\n",
    "### QUICK TEST ###\n",
    "\n",
    "### Init Weights ###\n",
    "w0 = tf.Variable(0.0)\n",
    "w1 = tf.Variable(0.0)\n",
    "\n",
    "### Obtain  Gradient ###\n",
    "dw0, dw1 = compute_gradients(X_train, Y_train, w0, w1)\n",
    "dw_0,dw_1 = gradient_validation(X_train, Y_train, w0, w1)\n",
    "\n",
    "print(f\"dw0: {dw0.numpy()}, Validation: {dw_0}\")\n",
    "print(f\"dw1 {dw1.numpy()}, Validation: {dw_1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we now how to compute backproagation with Tensorflow, our life is way easier than it was before, Lets apply this to a open linear regression: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STEP 0 - loss: 35.70719528198242\n",
      "STEP 200 - loss: 0.26831889152526855\n",
      "STEP 400 - loss: 0.0028539239428937435\n",
      "STEP 600 - loss: 3.0356444767676294e-05\n",
      "STEP 800 - loss: 3.2238213520940917e-07\n",
      "STEP 1000 - loss: 3.6101481803996194e-09\n",
      "\n",
      "w0: 9.99988842010498, w1: 2.0000178813934326\n",
      "Train set loss: 3.6101481803996194e-09\n"
     ]
    }
   ],
   "source": [
    "### Hyper Parameters ###\n",
    "STEPS = 1000\n",
    "LEARNING_RATE = .02\n",
    "\n",
    "### Initialize Weights ###\n",
    "w0 = tf.Variable(0.0)\n",
    "w1 = tf.Variable(0.0)\n",
    "\n",
    "### Linear Regression ###\n",
    "\n",
    "for step in range(0, STEPS + 1):\n",
    "    \n",
    "    ### Forward PASS with gradient Tape, it obtains gradients ###\n",
    "    dw0, dw1 = compute_gradients(X_train, Y_train, w0, w1)\n",
    "    \n",
    "    ### GRADIENT DESCENT ###\n",
    "    w0.assign_sub(dw0 * LEARNING_RATE) # w0 =  w0 - (dw0)w0\n",
    "    w1.assign_sub(dw1 * LEARNING_RATE) # w1 =  w1 - (dw1)w1\n",
    "\n",
    "    if step % 200 == 0:\n",
    "        loss = loss_mse(X_train, Y_train, w0, w1)\n",
    "        print(f\"STEP {step} - loss: {loss}\")\n",
    "        \n",
    "### USER INTERACTION ###\n",
    "\n",
    "### PRINT MESSAGE ###\n",
    "print(f\"\\nw0: {w0.numpy()}, w1: {w1.numpy()}\")\n",
    "\n",
    "### LOSS ON TEST SET ###\n",
    "loss = loss_mse(X_train, Y_train, w0, w1)\n",
    "print(f\"Train set loss: {loss.numpy()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "machine_learning_env",
   "language": "python",
   "name": "machine_learning_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}