t81_558_class_03_1_neural_net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* **Part 3.1: Deep Learning and Neural Network Introduction** [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "* Part 3.2: Introduction to Tensorflow and Keras [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.1: Deep Learning and Neural Network Introduction\n",
    "\n",
    "Neural networks were one of the first machine learning models.  Their popularity has fallen twice and is now on its third rise.  Deep learning implies the use of neural networks.  The \"deep\" in deep learning refers to a neural network with many hidden layers.  Because neural networks have been around for so long, they have quite a bit of baggage.  Researchers have created many different training algorithms, activation/transfer functions, and structures over the years.  This course is only concerned with the latest, most current state of the art techniques for deep neural networks.  I am not going to spend any time discussing the history of neural networks.  If you would like to learn about some of the more classic structures of neural networks, there are several modules dedicated to this in your coursebook.  For the latest technology, I wrote an article for the Society of Actuaries on deep learning as the [third generation of neural networks](https://www.soa.org/Library/Newsletters/Predictive-Analytics-and-Futurism/2015/december/paf-iss12.pdf).\n",
    "\n",
    "Neural networks accept input and produce output.  The input to a neural network is called the feature vector.  The size of this vector is always a fixed length.  Changing the size of the feature vector means recreating the entire neural network.  Though the feature vector is called a \"vector,\" this is not always the case.  A vector implies a 1D array.  Historically the input to a neural network was always 1D.  However, with modern neural networks, you might see input data, such as:\n",
    "\n",
    "* **1D vector** - Classic input to a neural network, similar to rows in a spreadsheet.  Common in predictive modeling.\n",
    "* **2D Matrix** - Grayscale image input to a convolutional neural network (CNN).\n",
    "* **3D Matrix** - Color image input to a convolutional neural network (CNN).\n",
    "* **nD Matrix** - Higher-order input to a CNN.\n",
    "\n",
    "Before CNN's, programs sent the image input to a neural network by merely squashing the image matrix into a long array by placing the image's rows side-by-side.  CNNs are different, as the nD matrix passes through the neural network layers.\n",
    "\n",
    "Initially, this course will focus upon 1D input to neural networks.  However, later sessions will focus more heavily upon higher dimension input.\n",
    "\n",
    "**Dimensions** The term dimension can be confusing in neural networks.  In the sense of a 1D input vector, dimension refers to how many elements are in that 1D array.  For example, a neural network with ten input neurons has ten dimensions.  However, now that we have CNN's, the input has dimensions too.  The input to the neural network will *usually* have 1, 2, or 3 dimensions.  Four or more dimensions are unusual.  You might have a 2D input to a neural network that has 64x64 pixels.  This configuration would result in 4,096 input neurons.  This network is either 2D or 4,096D, depending on which set of dimensions you are referencing.\n",
    "\n",
    "## Classification or Regression\n",
    "\n",
    "Like many models, neural networks can function in classification or regression:\n",
    "\n",
    "* **Regression** - You expect a number as your neural network's prediction.\n",
    "* **Classification** - You expect a class/category as your neural network's prediction.\n",
    "\n",
    "A classification and regression neural network is shown by Figure 3.CLS-REG.\n",
    "\n",
    "**Figure 3.CLS-REG: Neural Network Classification and Regression**\n",
    "![Neural Network Classification and Regression](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann_class_reg.png \"Neural Network Classification and Regression\")\n",
    "\n",
    "Notice that the output of the regression neural network is numeric, and the output of the classification is a class.  Regression, or two-class classification, networks always have a single output.  Classification neural networks have an output neuron for each category. \n",
    "\n",
    "## Neurons and Layers\n",
    "\n",
    "Most neural network structures use some type of neuron. Many different kinds of neural networks exist, and programmers introduce experimental neural network structures all the time.  Consequently, it is not possible to cover every neural network architecture. However, there are some commonalities among neural network implementations.  An algorithm that is called a neural network will typically be composed of individual, interconnected units even though these units may or may not be called neurons. The name for a neural network processing unit varies among the literature sources. It could be called a node, neuron, or unit.\n",
    "\n",
    "A diagram shows the abstract structure of a single artificial neuron in Figure 3.ANN.\n",
    "\n",
    "**Figure 3.ANN: An Artificial Neuron**\n",
    "![An Artificial Neuron](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_abstract_nn.png \"An Artificial Neuron\")\n",
    " \n",
    "The artificial neuron receives input from one or more sources that may be other neurons or data fed into the network from a computer program.  This input is usually floating-point or binary.  Often binary input is encoded to floating-point by representing true or false as 1 or 0.  Sometimes the program also depicts the binary input as using a bipolar system with true as one and false as -1.\n",
    "\n",
    "An artificial neuron multiplies each of these inputs by a weight. Then it adds these multiplications and passes this sum to an activation function.  Some neural networks do not use an activation function.  The following equation summarizes the calculated output of a neuron:\n",
    "\n",
    "$f(x,w) = \\phi(\\sum_i(\\theta_i \\cdot x_i))$\n",
    "\n",
    "In the above equation, the variables $x$ and $\\theta$ represent the input and weights of the neuron.  The variable $i$ corresponds to the number of weights and inputs.  You must always have the same number of weights as inputs.  The neural network multiplies each weight by its respective input and feeds the products of these multiplications into an activation function that is denoted by the Greek letter $\\phi$ (phi). This process results in a single output from the neuron.  \n",
    "\n",
    "The above neuron has two inputs plus the bias as a third.  This neuron might accept the following input feature vector:\n",
    "\n",
    "$[1,2]$\n",
    "\n",
    "Because a bias neuron is present, the program should append the value of one, as follows:\n",
    "\n",
    "$[1,2,1]$\n",
    "\n",
    "The weights for a 3-input layer (2 real inputs + bias) will always have additional weight, for the bias.  A weight vector might be:\n",
    "\n",
    "$[ 0.1, 0.2, 0.3]$\n",
    "\n",
    "To calculate the summation, perform the following:\n",
    "\n",
    "$0.1*1 + 0.2*2 + 0.3*1 = 0.8$\n",
    "\n",
    "The program passes a value of 0.8 to the $\\phi$ (phi) function, which represents the activation function.\n",
    "\n",
    "The above figure shows the structure with just one building block.  You can chain together many artificial neurons to build an artificial neural network (ANN).  Think of the artificial neurons as building blocks for which the input and output circles are the connectors.  Figure 3.ANN-3 shows an artificial neural network composed of three neurons:\n",
    "\n",
    "**Figure 3.ANN-3: Three Neuron Neural Network**\n",
    "![Three Neuron Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ann-simple.png \"Three Neuron Neural Network\")\n",
    "\n",
    "The above diagram shows three interconnected neurons.  This representation is essentially this figure, minus a few inputs, repeated three times, and then connected.  It also has a total of four inputs and a single output.  The output of neurons **N1** and **N2** feed **N3** to produce the output O.  To calculate the output for this network; we perform the previous equation three times.  The first two times calculate **N1** and **N2**, and the third calculation uses the output of **N1** and **N2** to calculate **N3**.\n",
    "   \n",
    "Neural network diagrams do not typically show the level of detail seen in the previous figure.  To simplify the chart, we can omit the activation functions and intermediate outputs, and this process results in Figure 3.SANN-3. \n",
    "\n",
    "**Figure 3.SANN-3: Three Neuron Neural Network**\n",
    "![Three Neuron Neural Network](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/typical-ann.png \"Three Neuron Neural Network\")\n",
    " \n",
    " Looking at the previous figure, you can see two additional components of neural networks.  First, consider the graph represents the inputs and outputs as abstract dotted line circles.  The input and output could be parts of a more extensive neural network.  However, the input and output are often a particular type of neuron that accepts data from the computer program using the neural network, and the output neurons return a result to the program.  This type of neuron is called an input neuron.  We will discuss these neurons in the next section. This figure shows the neurons arranged in layers.  The input neurons are the first layer, the **N1** and **N2** neurons create the second layer, the third layer contains **N3**, and the fourth layer has O.  Most neural networks arrange neurons into layers. \n",
    "\n",
    "The neurons that form a layer share several characteristics.  First, every neuron in a layer has the same activation function.  However, the activation functions employed be each layer may be different.  Each of the layers fully connects to the next layer.  In other words, every neuron in one layer has a connection to neurons in the previous layer.  The former figure is not fully connected. Several layers are missing connections.  For example, **I1** and **N2** do not connect. The next neural network in Figure 3.F-ANN is fully connected and has an additional layer.\n",
    "\n",
    "**Figure 3.F-ANN: Fully Connected Neural Network Diagram**\n",
    "![Fully Connected Neural Network Diagram](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ann-dense.png \"Fully Connected Neural Network Diagram\")\n",
    "\n",
    "In this figure, you see a fully connected, multilayered neural network.  Networks, such as this one, will always have an input and output layer.  The hidden layer structure determines the name of the network architecture.  The network in this figure is a two-hidden-layer network.  Most networks will have between zero and two hidden layers.  Unless you have implemented deep learning strategies, networks with more than two hidden layers are rare.    \n",
    "\n",
    "You might also notice that the arrows always point downward or forward from the input to the output.  This type of neural network is called a feedforward neural network.  Later in this course, we will see recurrent neural networks that form inverted loops among the neurons.\n",
    "\n",
    "## Types of Neurons\n",
    "\n",
    "In the last section, we briefly introduced the idea that different types of neurons exist. Now we will explain all the neuron types described in the course. Not every neural network will use every kind of neuron.  It is also possible for a single neuron to fill the role of several different neuron types.\n",
    "\n",
    "There are usually four types of neurons in a neural network:\n",
    "\n",
    "* **Input Neurons** - Each input neuron is mapped to one element in the feature vector.\n",
    "* **Hidden Neurons** - Hidden neurons allow the neural network to be abstract and process the input into the output.\n",
    "* **Output Neurons** - Each output neuron calculates one part of the output.\n",
    "* **Context Neurons** - Holds state between calls to the neural network to predict.\n",
    "* **Bias Neurons** - Work similar to the y-intercept of a linear equation.  \n",
    "\n",
    "These neurons are grouped into layers:\n",
    "\n",
    "* **Input Layer** - The input layer accepts feature vectors from the dataset.  Input layers usually have a bias neuron.\n",
    "* **Output Layer** - The output from the neural network.  The output layer does not have a bias neuron.\n",
    "* **Hidden Layers** - Layers that occur between the input and output layers.  Each hidden layer will usually have a bias neuron.\n",
    "\n",
    "\n",
    "### Input and Output Neurons\n",
    "\n",
    "Nearly every neural network has input and output neurons.  The input neurons accept data from the program for the network. The output neuron provides processed data from the network back to the program. The program will group these input and output neurons into separate layers called the input and output layer.  The program normally represents the input to a neural network as an array or vector.  The number of elements contained in the vector must be equal to the number of input neurons.  For example, a neural network with three input neurons might accept the following input vector:\n",
    "\n",
    "$[0.5, 0.75, 0.2]$\n",
    "\n",
    "Neural networks typically accept floating-point vectors as their input.  Likewise, neural networks will output a vector with length equal to the number of output neurons.  The output will often be a single value from a single output neuron.  To be consistent, we will represent the output of a single output neuron network as a single-element vector.\n",
    "\n",
    "Notice that input neurons do not have activation functions.  As demonstrated earlier, input neurons are little more than placeholders.  The input is simply weighted and summed. Furthermore, the size of the input and output vectors for the neural network will be the same if the neural network has neurons that are both input and output.\n",
    "\n",
    "### Hidden Neurons\n",
    "\n",
    "Hidden neurons have two essential characteristics.  First, hidden neurons only receive input from other neurons, such as input or other hidden neurons.  Second, hidden neurons only output to other neurons, such as output or other hidden neurons.  Hidden neurons help the neural network understand the input, and they form the output. However, these hidden layers do not directly process to the incoming data or the eventual output.  Programmers often group hidden neurons into fully connected hidden layers.\n",
    "\n",
    "A common question for programmers concerns the number of hidden neurons in a network. Since the answer to this question is complex, more than one section of the course will include a relevant discussion of the number of hidden neurons. Before deep learning, researchers generally suggested that anything more than a single hidden layer is excessive (Hornik, 1991). Researchers have proven that a single-hidden-layer neural network can function as a universal approximator. In other words, this network should be able to learn to produce (or approximate) any output from any input as long as it has enough hidden neurons in a single layer. \n",
    "\n",
    "Another reason why researchers used to scoff at the idea of additional hidden layers is that these layers would impede the training of the neural network.  Training refers to the process that determines good weight values. Before researchers introduced deep learning techniques, we simply did not have an efficient way to train a deep network, which are neural networks with a large number of hidden layers.  Although a single-hidden-layer neural network can theoretically learn anything, deep learning facilitates a more complex representation of patterns in the data.  \n",
    "\n",
    "### Bias Neurons\n",
    "Programmers add bias neurons to neural networks to help them learn patterns. Bias neurons function like an input neuron that always produces a value of 1.  Because the bias neurons have a constant output of 1, they are not connected to the previous layer. The value of 1, which is called the bias activation, can be set to values other than 1. However, 1 is the most common bias activation.  Not all neural networks have bias neurons.  Figure 3.BIAS shows a single-hidden-layer neural network with bias neurons:\n",
    "\n",
    "**Figure 3.BIAS: Neural Network with Bias Neurons**\n",
    "![Neural Network with Bias Neurons](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_ann.png \"Neural Network with Bias Neurons\")\n",
    " \n",
    "The above network contains three bias neurons.  Every level, except for the output layer, includes a single bias neuron. Bias neurons allow the program to shift the output of an activation function.  We will see precisely how this shifting occurs later in the module when we discuss activation functions.  \n",
    "\n",
    "### Context Neurons\n",
    "\n",
    "Recurrent neural networks make use of context neurons to hold state.  This type of neuron allows the neural network to maintain state.  As a result, a given input may not always produce the same output.  This inconsistency is similar to the workings of biological brains. Consider how context factors in your response when you hear a loud horn.  If you hear the noise while you are crossing the street, you might startle, stop walking, and look in the direction of the horn. If you hear the horn while you are watching an action-adventure film in a movie theatre, you don't respond in the same way. Therefore, prior inputs give you the context for processing the audio input of a horn.\n",
    "\n",
    "Time series is one application of context neurons.  You might need to train a neural network to learn input signals to perform speech recognition or to predict trends in security prices.  Context neurons are one way for neural networks to deal with time-series data.  Figure 3.CTX shows how a neural network might arrange context neurons:\n",
    "\n",
    "**Figure 3.CTX: Neural Network with Context Neurons**\n",
    "![Neural Network with Context Neurons](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ann-context.png \"Neural Network with Context Neurons\")\n",
    " \n",
    "This neural network has a single input and output neuron.  Between the input and output layers are two hidden neurons and two context neurons.  Other than the two context neurons, this network is the same as networks demonstrated previously. \n",
    "\n",
    "Each context neuron holds a value that starts at 0 and always receives a copy of either hidden one or hidden two from the previous use of the network.   The two dashed lines in this figure mean that the context neuron is a direct copy with no other weighting.  The other lines indicate that the output is weighted by one of the six weight values listed above.  The calculation of the output still occurs in the same way.  The value of the output neuron would be the sum of all four inputs, multiplied by their weights, and applied to the activation function.\n",
    "\n",
    "### Other Neuron Types\n",
    "\n",
    "The individual units that comprise a neural network are not always called neurons.  Researchers will sometimes refer to these neurons as nodes, units, or summations.  In later modules of this course, we will explore deep learning that utilizes Boltzmann machines to fill the role of neurons. You will almost always construct neural networks of weighted connections between these units.\n",
    "\n",
    "## Why are Bias Neurons Needed?\n",
    "\n",
    "The activation functions seen in the previous section specifies the output of a single neuron.  Together, the weight and bias of a neuron shape the output of the activation to produce the desired output.  To see how this process occurs, consider the following equation. It represents a single-input sigmoid activation neural network.\n",
    "\n",
    "$ f(x,w,b) = \\frac{1}{1 + e^{-(wx+b)}} $ \n",
    "\n",
    "The $x$ variable represents the single input to the neural network.  The $w$ and $b$ variables specify the weight and bias of the neural network.  The above equation is a combination of the weighted sum of the inputs and the sigmoid activation function.  For this section, we will consider the sigmoid function because it demonstrates the effect that a bias neuron has.\n",
    "\n",
    "The weights of the neuron allow you to adjust the slope or shape of the activation function.  Figure 3.A-WEIGHT shows the effect on the output of the sigmoid activation function if the weight is varied:\n",
    "\n",
    "**Figure 3.A-WEIGHT: Neuron Weight Shifting**\n",
    "![Adjusting Weight](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_weight.png \"Neuron Weight Shifting\")\n",
    "\n",
    "The above diagram shows several sigmoid curves using the following parameters:\n",
    "\n",
    "$ \\\\\n",
    "f(x,0.5,0.0) \\\\\n",
    "f(x,1.0,0.0) \\\\\n",
    "f(x,1.5,0.0) \\\\\n",
    "f(x,2.0,0.0)\n",
    "$\n",
    "\n",
    "To produce the curves, we did not use bias, which is evident in the third parameter of 0 in each case. Using four weight values yields four different sigmoid curves in the above figure. No matter the weight, we always get the same value of 0.5 when x is 0 because all of the curves hit the same point when x is 0.  We might need the neural network to produce other values when the input is near 0.5.  \n",
    "\n",
    "Bias does shift the sigmoid curve, which allows values other than 0.5 when x is near 0. Figure 3.A-BIAS shows the effect of using a weight of 1.0 with several different biases:\n",
    "\n",
    "**Figure 3.A-BIAS: Neuron Bias Shifting**\n",
    "![Adjusting Bias](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_bias_value.png \"Neuron Bias Shifting\")\n",
    "\n",
    "The above diagram shows several sigmoid curves with the following parameters:\n",
    "\n",
    "$ \\\\\n",
    "f(x,1.0,1.0) \\\\\n",
    "f(x,1.0,0.5) \\\\\n",
    "f(x,1.0,1.5) \\\\\n",
    "f(x,1.0,2.0)\n",
    "$\n",
    "\n",
    "We used a weight of 1.0 for these curves in all cases.  When we utilized several different biases, sigmoid curves shifted to the left or right.  Because all the curves merge at the top right or bottom left, it is not a complete shift.\n",
    "\n",
    "When we put bias and weights together, they produced a curve that created the necessary output from a neuron.  The above curves are the output from only one neuron.  In a complete network, the output from many different neurons will combine to produce intricate output patterns.\n",
    "\n",
    "\n",
    "## Modern Activation Functions\n",
    "\n",
    "Activation functions, also known as transfer functions, are used to calculate the output of each layer of a neural network.  Historically neural networks have used a hyperbolic tangent, sigmoid/logistic, or linear activation function.  However, modern deep neural networks primarily make use of the following activation functions:\n",
    "\n",
    "* **Rectified Linear Unit (ReLU)** - Used for the output of hidden layers. [[Cite:glorot2011deep]](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf)\n",
    "* **Softmax** - Used for the output of classification neural networks. [Softmax Example](http://www.heatonresearch.com/aifh/vol3/softmax.html)\n",
    "* **Linear** - Used for the output of regression neural networks (or 2-class classification).\n",
    "\n",
    "### Linear Activation Function\n",
    "The most basic activation function is the linear function because it does not change the neuron output at all.  The following equation 1.2 shows how the program typically implements a linear activation function:\n",
    "\n",
    "$\\phi(x) = x$\n",
    "\n",
    "As you can observe, this activation function simply returns the value that the neuron inputs passed to it.  Figure 3.LIN shows the graph for a linear activation function:\n",
    "\n",
    "**Figure 3.LIN: Linear Activation Function**\n",
    "![Linear Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-linear.png \"Linear Activation Function\")\n",
    "\n",
    " \n",
    "Regression neural networks, those that learn to provide numeric values, will usually use a linear activation function on their output layer.  Classification neural networks, those that determine an appropriate class for their input, will often utilize a softmax activation function for their output layer.\n",
    "\n",
    "### Rectified Linear Units (ReLU)\n",
    "\n",
    "Introduced in 2000 by Teh & Hinton, the rectified linear unit (ReLU) has seen very rapid adoption over the past few years.  Before the ReLU activation function, the programmers generally regarded the hyperbolic tangent as the activation function of choice.  Most current research now recommends the ReLU due to superior training results.  As a result, most neural networks should utilize the ReLU on hidden layers and either softmax or linear on the output layer. The following equation shows the straightforward ReLU function:\n",
    "\n",
    "$ \\phi(x) = \\max(0, x) $\n",
    "\n",
    "We will now examine why ReLU typically performs better than other activation functions for hidden layers. Part of the increased performance is because the ReLU activation function is a linear, non-saturating function.  Unlike the sigmoid/logistic or the hyperbolic tangent activation functions, the ReLU does not saturate to -1, 0, or 1.  A saturating activation function moves towards and eventually attains a value.  The hyperbolic tangent function, for example, saturates to -1 as x decreases and one as x increases.  Figure 3.RELU shows the graph of the ReLU activation function:\n",
    "\n",
    "**Figure 3.RELU: Rectified Linear Units (ReLU)**\n",
    "![Rectified Linear Units (ReLU)](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-relu.png \"Rectified Linear Units (ReLU)\")\n",
    " \n",
    "Most current research states that the hidden layers of your neural network should use the ReLU activation.\n",
    "\n",
    "### Softmax Activation Function\n",
    "\n",
    "The final activation function that we will examine is the softmax activation function.  Along with the linear activation function, you can usually find the softmax function in the output layer of a neural network.  Classification neural networks typically employ the softmax function. The neuron that has the highest value claims the input as a member of its class.  Because it is a preferable method, the softmax activation function forces the output of the neural network to represent the probability that the input falls into each of the classes.  Without the softmax, the neuron's outputs are simply numeric values, with the highest indicating the winning class.\n",
    "\n",
    "To see how the program uses the softmax activation function, we will look at a typical neural network classification problem.  The iris data set contains four measurements for 150 different iris flowers.  Each of these flowers belongs to one of three species of iris.  When you provide the measurements of a flower, the softmax function allows the neural network to give you the probability that these measurements belong to each of the three species.  For example, the neural network might tell you that there is an 80% chance that the iris is setosa, a 15% probability that it is virginica, and only a 5% probability of versicolor.  Because these are probabilities, they must add up to 100%.  There could not be an 80% probability of setosa, a 75% probability of virginica, and a 20% probability of versicolor—this type of result would be nonsensical.\n",
    "\n",
    "To classify input data into one of three iris species, you will need one output neuron for each of the three species. The output neurons do not inherently specify the probability of each of the three species.  Therefore, it is desirable to provide probabilities that sum to 100%.  The neural network will tell you the probability of a flower being each of the three species. To get the probability, use the softmax function in the following equation: \n",
    "\n",
    "$ \\phi_i(x) = \\frac{e^{z_i}}{\\sum\\limits_{j \\in group}e^{z_j}} $\n",
    "\n",
    "In the above equation, $i$ represents the index of the output neuron ($o$) that the program is calculating, and $j$ represents the indexes of all neurons in the group/level.  The variable z designates the array of output neurons. It's important to note that the program calculates the softmax activation differently than the other activation functions in this module. When softmax is the activation function, the output of a single neuron is dependent on the other output neurons. \n",
    " \n",
    "To see the softmax function in operation, refer to this [Softmax example web site](http://www.heatonresearch.com/aifh/vol3/softmax.html).\n",
    "\n",
    "Consider a trained neural network that classifies data into three categories, such as the three iris species.  In this case, you would use one output neuron for each of the target classes.  Consider if the neural network were to output the following:   \n",
    "\n",
    "* **Neuron 1**: setosa: 0.9\n",
    "* **Neuron 2**: versicolour: 0.2\n",
    "* **Neuron 3**: virginica: 0.4\n",
    "\n",
    "From the above output, we can see that the neural network considers the data to represent a setosa iris.  However, these numbers are not probabilities.  The 0.9 value does not represent a 90% likelihood of the data representing a setosa.  These values sum to 1.5. For the program to treat them as probabilities, they must sum to 1.0.  The output vector for this neural network is the following:\n",
    "\n",
    "$ [0.9,0.2,0.4] $\n",
    "\n",
    "If you provide this vector to the softmax function it will return the following vector:\n",
    "\n",
    "$ [0.47548495534876745 , 0.2361188410001125 , 0.28839620365112] $\n",
    "\n",
    "The above three values do sum to 1.0 and can be treated as probabilities.  The likelihood of the data representing a setosa iris is 48% because the first value in the vector rounds to 0.48 (48%).  You can calculate this value in the following manner:\n",
    "\n",
    "$ sum=\\exp(0.9)+\\exp(0.2)+\\exp(0.4)=5.17283056695839 \\\\\n",
    "j_0= \\exp(0.9)/sum = 0.47548495534876745 \\\\\n",
    "j_1= \\exp(0.2)/sum = 0.2361188410001125 \\\\\n",
    "j_2= \\exp(0.4)/sum = 0.28839620365112\n",
    "$\n",
    "\n",
    "## Classic Activation Functions\n",
    "\n",
    "### Step Activation Function\n",
    "\n",
    "The step or threshold activation function is another simple activation function.  Neural networks were originally called perceptrons.  McCulloch & Pitts (1943) introduced the original perceptron and used a step activation function like the following equation:[[Cite:mcculloch1943logical]](https://link.springer.com/article/10.1007/BF02478259)\n",
    "\n",
    "$ \\phi(x)=\\begin{cases}\n",
    "    1, \\text{if x >= 0.5}.\\\\\n",
    "    0, \\text{otherwise}.\n",
    "  \\end{cases}$\n",
    "  \n",
    "This equation outputs a value of 1.0 for incoming values of 0.5 or higher and 0 for all other values.  Step functions, also known as threshold functions, only return 1 (true) for values that are above the specified threshold, as seen in Figure 3.STEP.\n",
    "\n",
    "**Figure 3.STEP: Step Activation Function**\n",
    "![Step Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-step.png \"Step Activation Function\")\n",
    " \n",
    "### Sigmoid Activation Function\n",
    "\n",
    "The sigmoid or logistic activation function is a very common choice for feedforward neural networks that need to output only positive numbers. Despite its widespread use, the hyperbolic tangent or the rectified linear unit (ReLU) activation function is usually a more suitable choice.  We introduce the ReLU activation function later in this module. The following equation shows the sigmoid activation function:\n",
    "\n",
    "$ \\phi(x) = \\frac{1}{1 + e^{-x}} $\n",
    "\n",
    "Use the sigmoid function to ensure that values stay within a relatively small range, as seen in Figure 3.SIGMOID:\n",
    "\n",
    "**Figure 3.SIGMOID: Sigmoid Activation Function**\n",
    "![Sigmoid Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-sigmoid.png \"Sigmoid Activation Function\")\n",
    "\n",
    "As you can see from the above graph, values can be forced to a range. Here, the function compressed values above or below 0 to the approximate range between 0 and 1.\n",
    "\n",
    "### Hyperbolic Tangent Activation Function\n",
    "\n",
    "The hyperbolic tangent function is also a prevalent activation function for neural networks that must output values in the range between -1 and 1.  This activation function is simply the hyperbolic tangent (tanh) function, as shown in the following equation:\n",
    "\n",
    "$ \\phi(x) = \\tanh(x) $\n",
    "\n",
    "The graph of the hyperbolic tangent function has a similar shape to the sigmoid activation function, as seen in Figure 3.HTAN.\n",
    "\n",
    "**Figure 3.HTAN: Hyperbolic Tangent Activation Function**\n",
    "![Hyperbolic Tangent Activation Function](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/graphs-tanh.png \"Hyperbolic Tangent Activation Function\")\n",
    " \n",
    "The hyperbolic tangent function has several advantages over the sigmoid activation function.  \n",
    "\n",
    "## Why ReLU?\n",
    "\n",
    "Why is the ReLU activation function so popular?  It was one of the critical improvements to neural networks that makes deep learning work. [[Cite:nair2010rectified]](https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf) Before deep learning, the sigmoid activation function was prevalent.  We covered the sigmoid activation function earlier in this module.  Frameworks, like Keras often train neural networks with gradient descent.  For the neural network to make use of gradient descent, it is necessary to take the derivative of the activation function.  The program must derive partial derivatives of each of the weights with respect to the error function.  Figure 3.DERV shows a derivative, the instantaneous rate of change.\n",
    "\n",
    "**Figure 3.DERV: Derivative**\n",
    "![Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv.png \"Derivative\")\n",
    "\n",
    "The derivative of the sigmoid function is given here:\n",
    "\n",
    "$ \\phi'(x)=\\phi(x)(1-\\phi(x)) $\n",
    "\n",
    "Textbooks often give this derivative in other forms.  We use the above form for computational efficiency. To see how we determined this derivative, [refer to the following article](http://www.heatonresearch.com/aifh/vol3/deriv_sigmoid.html).\n",
    "\n",
    "We present the graph of the sigmoid derivative in Figure 3.SDERV.\n",
    "\n",
    "**Figure 3.SDERV: Sigmoid Derivative**\n",
    "![Sigmoid Derivative](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_deriv_sigmoid.png \"Sigmoid Derivative\")\n",
    "\n",
    "The derivative quickly saturates to zero as $x$ moves from zero.  This is not a problem for the derivative of the ReLU, which is given here:\n",
    "\n",
    "$ \\phi'(x) = \\begin{cases} 1 & x > 0 \\\\ 0 & x \\leq 0 \\end{cases} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 3](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}