MIT_TF_From_the_Ground_Up_Exercises_3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MIT: TF From the Ground Up Exercises 3.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "au5kdSKU9ddV"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make a basic layer.  Note that the base class is tf.Module, which allows you to keep track of variables and submodules."
      ],
      "metadata": {
        "id": "MYw7JHQ1JqgT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Make some \n",
        "class Dense(tf.Module):\n",
        " def __init__(self, in_features, out_features, name=None):\n",
        "   super().__init__(name=name)\n",
        "\n",
        "   self.w = tf.Variable(\n",
        "     tf.random.normal([in_features, out_features]), name='w')\n",
        "   self.b = tf.Variable(tf.zeros([out_features]), name='b')\n",
        "\n",
        " def __call__(self, x):\n",
        "   y = tf.matmul(x, self.w) + self.b\n",
        "   return tf.nn.relu(y)"
      ],
      "metadata": {
        "id": "DWkNJmNvJt8I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, make a model with two layers.  It also depends on tf.Module. \n",
        "\n",
        "You can (and should!) nest tf.Modules (or, by extension, Keras layers) inside each other.  "
      ],
      "metadata": {
        "id": "bC3148NVRKRe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class SequentialModule(tf.Module):\n",
        " def __init__(self, name=None):\n",
        "   super().__init__(name=name)\n",
        "\n",
        "   self.dense_1 = Dense(in_features=3, out_features=3)\n",
        "   self.dense_2 = Dense(in_features=3, out_features=2)\n",
        "\n",
        " def __call__(self, x):\n",
        "   r = self.dense_1(x)\n",
        "   return self.dense_2(r)\n",
        "\n",
        "\n",
        "# Instantiate the model.\n",
        "myModel = SequentialModule()\n"
      ],
      "metadata": {
        "id": "I0saV5oTJxk0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Modules know about other modules, so you can look inside them."
      ],
      "metadata": {
        "id": "4vVkd2vaJ6bx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Variables:\")\n",
        "for variable in myModel.variables:\n",
        "  print(variable)\n",
        "\n",
        "print(\"\\nModules:\")\n",
        "for module in myModel.submodules:\n",
        "  print(module)"
      ],
      "metadata": {
        "id": "nqRbOCv_J-Eo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can write all the variable values out into a file."
      ],
      "metadata": {
        "id": "mALIj8cr9p1K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save it\n",
        "checkpoint = tf.train.Checkpoint(model=myModel)\n",
        "checkpoint.write(\"apath.cpt\")\n"
      ],
      "metadata": {
        "id": "e6mA7fmr9luu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can look at what got written out:"
      ],
      "metadata": {
        "id": "Bdkk7WTsVF0y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls apath*"
      ],
      "metadata": {
        "id": "Z05IcX5GUtgx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The single shard are the variable values in a protobuf, and the index is metadata. "
      ],
      "metadata": {
        "id": "uA6vNWCSU59N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training\n",
        "\n",
        "Once you have a model, you need to train it.\n",
        "\n",
        "Remember that training is:\n",
        "\n",
        "* Build a model\n",
        "* Choose an optimizer\n",
        "* Do a forward pass\n",
        "* Compare the output of the forward pass to the actual labels to generate a **loss*\n",
        "* Using the gardient tape, use the loss to calculate your gradients\n",
        "* Apply the gradient to the variables to improve your results (hopefully!)\n",
        "\n",
        "Here is a tiny model with nonsense inputs and outputs that we can train for a single step to see it in action."
      ],
      "metadata": {
        "id": "wBkw-brIVCJZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = SequentialModule()\n",
        "x = tf.constant([[1., 2., 3.]])  # some input data\n",
        "y = tf.constant([[2., 2.3]])  # some classification labels for this input data\n",
        "optimizer = tf.keras.optimizers.SGD()  # Gradient descent optimizer\n",
        "\n",
        "with tf.GradientTape() as tape:\n",
        "  # Forward pass and calculate loss\n",
        "  y_p = model(x)\n",
        "  loss = tf.keras.losses.MSE(y_p, y)\n",
        "\n",
        "# Calculate and apply gradients to minimize loss\n",
        "grads = tape.gradient(loss, model.trainable_variables)\n",
        "optimizer.apply_gradients(zip(grads, model.trainable_variables))\n"
      ],
      "metadata": {
        "id": "cCndTNKzLJPw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.variables"
      ],
      "metadata": {
        "id": "bH4xFSXdeMXT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Models in Keras\n",
        "\n",
        "The same model can work in Keras; just change the parent. Remember, Keras models are tf.Modules with extra features.\n",
        "\n",
        "Keras also comes with a built-in dense layer, which will save you some time."
      ],
      "metadata": {
        "id": "FnpPlqx7937X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class SequentialModel(tf.keras.Model):\n",
        " def __init__(self, name=None):\n",
        "   super().__init__(name=name)\n",
        "\n",
        "   self.dense_1 = tf.keras.layers.Dense(3, use_bias=False)\n",
        "   self.dense_2 = tf.keras.layers.Dense(2, use_bias=False)\n",
        "\n",
        " def call(self, x):\n",
        "   r = self.dense_1(x)\n",
        "   return self.dense_2(r)\n",
        "\n",
        "\n",
        "# Instantiate the model.\n",
        "myModel = SequentialModel()\n",
        "myModel.build(x.shape) # This allocates all the variables\n",
        "myModel.summary()"
      ],
      "metadata": {
        "id": "ILZjkO0or89j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Essentially the same model with Keras.  Keras has a lot more features, including helpful summaries, built-in bias and activation, and so on.  However, it's all built on top of `tf.Module`!\n",
        "\n",
        "In real life, you might move your layer allocations into a build step for\n",
        "Keras's convenience.\n",
        "\n",
        "### Functional API\n",
        "\n",
        "Keras also comes with a functional API for defining models, which can be elegant; it's optional."
      ],
      "metadata": {
        "id": "xzQrtTCj90ug"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "inputs = tf.keras.Input(shape=[3,])\n",
        "\n",
        "x = tf.keras.layers.Dense(3)(inputs)\n",
        "x = tf.keras.layers.Dense(2)(x)\n",
        "\n",
        "my_functional_model = tf.keras.Model(inputs=inputs,\n",
        "   outputs=x)\n",
        "\n",
        "my_functional_model.build(x.shape)\n",
        "\n",
        "my_functional_model.summary()"
      ],
      "metadata": {
        "id": "DZk09oonCpM8"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}