02_More_on_Intialization.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.2.0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "print(f\"TensorFlow version: {tf.__version__}\")\n",
    "\n",
    "tf.config.list_physical_devices('GPU')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More on Variables and constants\n",
    "Few people make this comparison, but TensorFlow and\n",
    "Numpy are quite similar. (Both are N-d array libraries!):\n",
    "\n",
    "* Numpy has Ndarray support, but doesn’t offer methods to\n",
    "create tensor functions and automatically compute\n",
    "derivatives (+ no GPU support)\n",
    "\n",
    "Asaides from that, you can view tensorflow (low-level TF API) as a graph caclualtor, with the advantage that it is only needed to calualte the forward pass, since TF is capable of computing the derrivatives by itself.\n",
    "\n",
    "Now a little bit of history.  When tensorflow 1.0.0 was relased, you had to create sessions, fill those sessions, and execute them at \n",
    "the end. With the introduction of tensorflow 2.0.0, sessions are no longer a thing.  Instead, graps are executed eagerly.\n",
    "\n",
    "$\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$\n",
    "<font color=\"green\"> TF1: How Graphs were built </font> $\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$ <font color=\"blue\">TF2: How Graphs are built now </font>\n",
    "\n",
    "```import tensorflow as tf```\n",
    "$\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$ \n",
    "```import tensorflow as tf```\n",
    "\n",
    "```x = tf.constant(1)```\n",
    "$\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$ \n",
    "```x = tf.constant(1)```\n",
    "\n",
    "```y = tf.constant(2)```\n",
    "$\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$ \n",
    "```y = tf.constant(2)```\n",
    "\n",
    "```z = tf.add(x,y)```\n",
    "$\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;$ \n",
    "```z = tf.add(x,y) # The graph runs here```\n",
    "\n",
    "```with tf.Session() as sess:```\n",
    "\n",
    "```sess.run(z)```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.executing_eagerly() # verify eagerly status "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tpye: <class 'tensorflow.python.framework.ops.EagerTensor'> \n",
      "z:22\n"
     ]
    }
   ],
   "source": [
    "## TF2 does not need a session to run! ##\n",
    "a = tf.constant(12)  # Define a tensor of rank 0\n",
    "b = tf.constant(10) # Define a tensor of rank 0\n",
    "z = tf.add(a,b)     # It runs the \"session\" eagerly \n",
    "\n",
    "print(f\"Tpye: {type(z)} \\nz:{z}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to Constants, Tensors, and Variables\n",
    "\n",
    "Since eager executation is now default, There is no need to create placeholders, as it was needed in TF1, now lets dig deep on TF2:\n",
    "\n",
    "Tensors in TensorFlow are either contant (tf.constant) or variables (tf.Variable). Constant values can not be changed, while variables values can be.\n",
    "\n",
    "Both Variables and Constants create a **Tensor** (which can also created directly )\n",
    "\n",
    "Basic Datatypes:\n",
    "\n",
    "* tf.float(16,32,64)\n",
    "* tf.int(8,16,32,64)\n",
    "* tf.bool\n",
    "\n",
    "**Initalization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(4, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# We can initalize a value  without specifiyng shape and type\n",
    "# This initialization applies for Constants and Variables\n",
    "x = tf.constant(4) # Init a constant\n",
    "y = tf.Variable(4) # Init a Varibale \n",
    "if x == y:         # IF both are the same value/dtype print the value\n",
    "    print(x) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=float64, numpy=\n",
       "array([[4., 4., 4., 4.],\n",
       "       [4., 4., 4., 4.],\n",
       "       [4., 4., 4., 4.],\n",
       "       [4., 4., 4., 4.]])>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Or we can explicitly tell Tensorflow all the information\n",
    "x = tf.constant(4, shape=(4,4),dtype=tf.float64) # Explcitly tell Tensorflow the initialization\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[4. 4. 4. 4.]\n",
      " [4. 4. 4. 4.]\n",
      " [4. 4. 4. 4.]\n",
      " [4. 4. 4. 4.]], shape=(4, 4), dtype=float16)\n"
     ]
    }
   ],
   "source": [
    "x = tf.cast(x,dtype=tf.float16) # You can use Cast to change dtype\n",
    "print(x) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=float16, numpy=\n",
       "array([[5., 5., 5., 5.],\n",
       "       [5., 5., 5., 5.],\n",
       "       [5., 5., 5., 5.],\n",
       "       [5., 5., 5., 5.]], dtype=float16)>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Broadcasting is also available (like in numpy)\n",
    "x+1 # adds 1 to all elements of tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 1. 1.]\n",
      " [1. 1. 1.]\n",
      " [1. 1. 1.]]\n",
      "\n",
      "[[0. 0.]\n",
      " [0. 0.]]\n",
      "\n",
      "[[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]]\n",
      "\n",
      "[[-0.9726208  -0.01479066 -1.3624561 ]\n",
      " [-0.04635035  0.11913349  3.050895  ]\n",
      " [-1.93892    -0.26182556  2.2397456 ]]\n"
     ]
    }
   ],
   "source": [
    "# Other Important Constant INitializations:\n",
    "o = tf.ones((3,3))  # Tensor of ones\n",
    "z = tf.zeros((2,2)) # Tenosr of zeros\n",
    "i = tf.eye(3)       # Identity Tensor\n",
    "r = tf.random.normal((3,3),mean=0, stddev=1) # Random Tensor from distribuiton\n",
    "\n",
    "print(f\"{o}\\n\\n{z}\\n\\n{i}\\n\\n{r}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Methods for Variables\n",
    "As we said before Tensors in TensorFlow are either contant (tf.constant) or variables (tf.Variable). Constant values can not be changed, while variables values can be.\n",
    "\n",
    "The main difference is that instances of tf.Variable have methods allowing us to change their values while tensors constructed with tf.constant don't have these methods, and therefore their values can not be changed. When you want to change the value of a tf.Variable x use one of the following method:\n",
    "\n",
    "* x.assign(new_value)\n",
    "* x.assign_add(value_to_be_added)\n",
    "* x.assign_sub(value_to_be_subtracted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable(2.0, dtype=tf.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=45.8>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign(45.8) # Asign a different value of variable\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=49.8>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign_add(4) # Add 4 to current variable\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=46.8>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign_sub(3) # Substract 3 to current value of variable\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Math operations\n",
    "#### Element  Wise operations\n",
    "\n",
    "Tensorflow offers similar point-wise tensor operations as numpy does:\n",
    "\n",
    "* tf.add allows to add the components of a tensor\n",
    "* tf.multiply allows us to multiply the components of a tensor\n",
    "* tf.subtract allow us to substract the components of a tensor\n",
    "* tf.math.* contains the usual math operations to be applied on the components of a tensor\n",
    "and many more...\n",
    "\n",
    "Most of the standard aritmetic operations (tf.add, tf.substrac, etc.) are overloaded by the usual corresponding arithmetic symbols (+, -, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Define Tensors ###\n",
    "A = tf.constant([[1,2,],[1,2,]],shape=(2,2))\n",
    "B = tf.constant([[1, 2],[3, 4]],shape=(2,2))\n",
    "### OPERATIONS ####\n",
    "\n",
    "z = tf.add(A,B)       # Adds two tensors element-wise\n",
    "z = A + B             # Overloaded form\n",
    "\n",
    "z = tf.subtract(A,B)  # Substracts two tensors element-wise\n",
    "z = A - B             # Overloaded form\n",
    "\n",
    "z = tf.divide(A,B)    # Divides two tensors element-wise\n",
    "z = A/B               # Overloaded form\n",
    "\n",
    "z = tf.multiply(A,B)  # Multiplies two tensors element-wise\n",
    "z = A*B               # Overloaded form\n",
    "\n",
    "z = A**2              # Raise a tensor element-wise (Overloaded form)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10 16 20]\n",
      " [10 16 20]]\n"
     ]
    }
   ],
   "source": [
    "### Define Tensors ###\n",
    "A = tf.constant([[1,2,3],[1,2,3]],shape=(2,3))\n",
    "B = tf.constant([[1, 2,3],[3, 4,4],[1,2,3]],shape=(3,3))\n",
    "### OPERATIONS ####\n",
    "\n",
    "Z = tf.matmul(A,B) # Tensor multiplciation\n",
    "Z = A@B            # Overloaded form\n",
    "print(f'{Z}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Miscellaneous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[1 2 3]\n",
      " [1 2 3]], shape=(2, 3), dtype=int32)\n",
      "\n",
      "tf.Tensor(\n",
      "[[1 1]\n",
      " [2 2]\n",
      " [3 3]], shape=(3, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "### Transpose a Tensor ###\n",
    "print(A)\n",
    "AT = tf.transpose(A)\n",
    "print()\n",
    "print(AT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=12>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Sum Tensor ###\n",
    "tf.math.reduce_sum(A) # Computes the sum of elements across dimensions of a tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[14, 14],\n",
       "       [14, 14]])>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Dot product ###\n",
    "tf.tensordot(A,AT,axes=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor Interoperability\n",
    "\n",
    "In addition to native TF tensors, tensorflow operations can take native python types and NumPy arrays as operands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([4, 6])>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# native python list\n",
    "a_py = [1, 2] \n",
    "b_py = [3, 4]\n",
    "tf.add(a_py, b_py) # TODO 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=int32, numpy=array([4, 6])>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numpy arrays\n",
    "a_np = np.array([1, 2])\n",
    "b_np = np.array([3, 4])\n",
    "tf.add(a_np, b_np) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bibliography**\n",
    "<br>\n",
    "[MLfromscratch](https://mlfromscratch.com/tensorflow-2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "machine_learning_env",
   "language": "python",
   "name": "machine_learning_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}