Neural Networks From Scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "# Neural Networks from Scratch\n",
    "\n",
    "## Purpose\n",
    "Make a basic multilayer perceptron artificial neural network (ANN) [\\[1\\]][1] from scratch.  And we'll use it to classify a dataset of handwritten digits.\n",
    "\n",
    "This will be accomplished ONLY using NumPy [\\[2\\]][2][\\[3\\]][3] as a dependency.  \n",
    "This code WILL NOT be optimized for efficiency or much of anything else.  The code will be written in a way to make it as readable and understandable as possible.  The is for educational purposes ONLY.  \n",
    "Any real implementation of neural networks should consider the existing variations such as convolutional neural networks (CNN) [\\[4\\]][4], recurrent neural networks (RNN) [\\[5\\]][5][\\[6\\]][6], etc. [\\[7\\]][7][\\[8\\]][8] as well as existing frameworks/libraries that implement these algorithms such as Keras/Tensorflow [\\[9\\]][9][\\[10\\]][10], Theano [\\[11\\]][11] (note: project is considered dead), PyTorch/Torch/Pyro [\\[12\\]][12][\\[13\\]][13][\\[14\\]][14], etc. [\\[15\\]][15][\\[16\\]][16][\\[17\\]][17][\\[18\\]][18][\\[19\\]][19][\\[28\\]][28], and performance such as GPU processing and parallelization across clusters.\n",
    "\n",
    "For further learn about deep learning, here is a good place to start learning: [Deep Learning for Coders Part 1][20] and [Deep Learning for Coders Part 2][21].\n",
    "\n",
    "  \n",
    "## Data\n",
    "The dataset that will be used can downloaded [here \\[24\\]][24].  \n",
    "In this case the dataset has been downloaded to the folder *MNIST_digits*.\n",
    "\n",
    "The data is a collection of handwritten digits from 0 to 9 in the format of a 28 by 28 uint8 grayscale bitmaps from the MNIST database.  \n",
    "More details on the MNIST database can be found [here \\[22\\]][22].  \n",
    "\n",
    "For further research, a list of databases for machine learning research can be found [here \\[23\\]][23].\n",
    "\n",
    "## Project\n",
    "This neural network, which shall be built from scratch uses the equations and a modified apporach found in this [online book \\[25\\]][25], [chapter 2 \\[26\\]][26] in particular.  \n",
    "\n",
    "*Note: Though we are using [chapter 2 \\[26\\]][26] of the online book as a guide, the notational convensions will be changed to make the math easier to understand.*  \n",
    "\n",
    "It's also important to have a basic understanding of [Matrix Calculus \\[27\\]][27], especially when involving derivatives.  If your not familiar with [Matrix Calculus \\[27\\]][27], it's very important that you aquire a basic understanding in order to understand the equations, rather than blindly copying and pasting the example code. \n",
    "\n",
    "## References\n",
    "- [\\[1\\]  Multilayer Preceptron (Wikipedia)][1]\n",
    "- [\\[2\\]  NumPy (Official Website)][2]\n",
    "- [\\[3\\]  NumPy (Wikipedia)][3]\n",
    "- [\\[4\\]  Convolutional Neural Networks (CNN) (Wikipedia)][4]\n",
    "- [\\[5\\]  Recurrent Neural Networks (RNN) (Wikipedia)][5]\n",
    "- [\\[6\\]  Long Short-Term Memory (LSTM) (Wikipedia)][6]\n",
    "- [\\[7\\]  The mostly complete chart of Neural Networks, explained (Towards Data Scienct)][7]\n",
    "- [\\[8\\]  Types of artificial neural networks (Wikipedia)][8]\n",
    "- [\\[9\\]  Keras (Official Website)][9]\n",
    "- [\\[10\\] Tensorflow (Official Website)][10]\n",
    "- [\\[11\\] Theano (Official Documentation)][11]\n",
    "- [\\[12\\] PyTorch (Official Website)][12]\n",
    "- [\\[13\\] Torch (Official Website)][13]\n",
    "- [\\[14\\] Pyro (Official Website)][14]\n",
    "- [\\[15\\] Microsoft Cognitive Toolkit (Official Website)][15]\n",
    "- [\\[16\\] BigDL (Official Source Code)][16]\n",
    "- [\\[17\\] Apache MXNet (Official Website)][17]\n",
    "- [\\[18\\] Deep Learning for Java (Deeplearning4j) (Official Website)][18]\n",
    "- [\\[19\\] deeplearn.js (Official Website) *(Yes this actually runs in the browser)*][19]\n",
    "- [\\[20\\] Deep Learning for Coders Part 1 (fast.ai)][20]\n",
    "- [\\[21\\] Deep Learning for Coders Part 2 (fast.ai)][21]\n",
    "- [\\[22\\] MNIST Database (Wikipedia)][22]\n",
    "- [\\[23\\] List of Datasets for Machine Learning Research (Wikipedia)][23]\n",
    "- [\\[24\\] THE MNIST DATABASE of handwritten digits (Official Website)][24]\n",
    "- [\\[25\\] Neural Networks and Deep Learning (Online Book)][25]\n",
    "- [\\[26\\] Neural Networks and Deep Learning (Online Book, Chapter 2)][26]\n",
    "- [\\[27\\] Matrix Calculus (Wikipedia)][27]\n",
    "- [\\[28\\] ConvnetJS *(Yes another one that actually runs in the browser)*][28]\n",
    "- [\\[29\\] Chainer][29]\n",
    "- [\\[30\\] TensorFlow.js *(Actually it's deeplearn.js, but taken over by the TensorFlow project)*][30]\n",
    "- [\\[31\\] DMLC for Scalable and Reliable Machine Learning][31]\n",
    "[1]:https://en.wikipedia.org/wiki/Multilayer_perceptron\n",
    "[2]:http://www.numpy.org/\n",
    "[3]:https://en.wikipedia.org/wiki/NumPy\n",
    "[4]:https://en.wikipedia.org/wiki/Convolutional_neural_network\n",
    "[5]:https://en.wikipedia.org/wiki/Recurrent_neural_network\n",
    "[6]:https://en.wikipedia.org/wiki/Long_short-term_memory\n",
    "[7]:https://towardsdatascience.com/the-mostly-complete-chart-of-neural-networks-explained-3fb6f2367464\n",
    "[8]:https://en.wikipedia.org/wiki/Types_of_artificial_neural_networks\n",
    "[9]:https://keras.io/\n",
    "[10]:https://www.tensorflow.org/\n",
    "[11]:http://deeplearning.net/software/theano/\n",
    "[12]:http://pytorch.org/\n",
    "[13]:http://torch.ch/\n",
    "[14]:http://pyro.ai/\n",
    "[15]:https://www.microsoft.com/en-us/cognitive-toolkit/\n",
    "[16]:https://github.com/intel-analytics/BigDL\n",
    "[17]:https://mxnet.apache.org/\n",
    "[18]:https://deeplearning4j.org/\n",
    "[19]:https://deeplearnjs.org/\n",
    "[20]:http://course.fast.ai/lessons/lessons.html\n",
    "[21]:http://course.fast.ai/part2.html\n",
    "[22]:http://yann.lecun.com/exdb/mnist/\n",
    "[23]:https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research\n",
    "[24]:http://yann.lecun.com/exdb/mnist/\n",
    "[25]:http://neuralnetworksanddeeplearning.com/about.html\n",
    "[26]:http://neuralnetworksanddeeplearning.com/chap2.html\n",
    "[27]:https://en.wikipedia.org/wiki/Matrix_calculus\n",
    "[28]:https://github.com/karpathy/convnetjs\n",
    "[29]:https://chainer.org/\n",
    "[30]:https://js.tensorflow.org/\n",
    "[31]:http://dmlc.ml/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "# Explanation of the Mathmatics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "## Notation\n",
    "\n",
    "The mathmatics required to do this isn't trivial, but what makes it complicated are the number of variables to track.  When going through the math you find that you'll quickly run out of places to denote indicies, you quickly lose track of where you are and whether your dealing with a vector, scalar, matrix, etc.  \n",
    "\n",
    "Here we'll define a new convention which will allow us better understand the mathmatics and communicate the concepts via the language of mathmatics.  The theories and math behind this is complicated enough, why complicate it further with confusing notation.  \n",
    "\n",
    "### Angle Bracket Convension\n",
    "\n",
    "$$\n",
    "% Macro to define the angle bracket convension\n",
    "% Usage:\n",
    "%    \\agl{subject}{epoch}{training datapoint}{layer}\n",
    "% Example:\n",
    "%    \\agl{x}{e}{t}{l}\n",
    "\\newcommand{\\agl}[4]{\\langle_{#2}{#1}_{#4}^{#3}\\rangle}\n",
    "$$\n",
    "**Indexing**  \n",
    "\n",
    "Because of the large amount of items we'll be tracking, we will be running out of spaces to put indexes.  Also, since we don't want to confuse with existing convensions, such as superscripts for exponentiation, we'll be using angle brackets to indicate a custom convension.  \n",
    "\n",
    "For example the following: $x^t$, would indicate $x$ raised to the power of $t$. \n",
    "\n",
    "But we'll be using the following: $\\agl{x}{}{t}{}$ to indicate $x$ at index $t$ to help clarify that we mean to index something rather than raise to a power.  \n",
    "\n",
    "Therefore, $\\agl{x}{}{t}{}^t$ would mean $x$ at index $t$ raised to the power of $t$.  \n",
    "\n",
    "Going further, we'll also define our convention to indicate that the position of the superscripts and subscripts to indicate which index we mean.  \n",
    "Here, we'll use the subscript that appears before the letter to indicate the training epoch, the superscript will be used to indicate the training datapoint, and the subscript which appears after the letter to indicate the layer in the network.\n",
    "\n",
    "For example: $\\agl{x}{e}{t}{l}$ means $x$ in training epoch $e$, for training datapoint $t$, at layer $l$.\n",
    "\n",
    "**Scalars, Vectors, and Matricies**  \n",
    "\n",
    "In addition to the indicies, we'll also use certain letters to help us destinguish between scalars, vectors, and matricies.  This will help keep our equations easy to understand.\n",
    "- ALL matricies will be denoted with a capital letter with a bold fontface.  For example: $\\bf M$  \n",
    "\n",
    "- ALL vectors will be denoted with a lowercase letter with an arrow or hat accent mark.  For example: $\\vec v$, and $\\hat v$\n",
    "\n",
    "- If it's not indicated as a matrix or vector, then it can safely be assumed to be a scalar.  For example:\n",
    " - $M$ is not a matrix, but a scalar, because though it's a capital letter, it's NOT bold. \n",
    " - $v$ and $\\bar v$ are not a vectors, but a scalars, because the lack of an accent mark or the accent mark is NOT an arrow or hat, but a bar.\n",
    " - and of course $x$ would be a scalar.\n",
    " \n",
    "**Putting it all together**  \n",
    " \n",
    "Combining the notations we'll end up with something like $\\agl{\\bf M}{e}{t}{l}$, means matix $\\bf M$ in training epoch $e$, for training datapoint $t$, at layer $l$.\n",
    "\n",
    "Outside this angle bracket convenstion, regular mathmatical convension applies.\n",
    "\n",
    "### Matrix and Vector indexing Convension\n",
    "\n",
    "In addition to the angle bracket convension, we'll use another convension to indicate if we are refering to an index of a matrix or vector.  It'll be denoted by a letter or an angle bracket expression wrapped by square brackets, followed by a subscript.  \n",
    "\n",
    "A single subscript indicates a particular element of a vector and two subscripts seperated by a comma indicates a particular element of a matrix.  \n",
    "\n",
    "For example:\n",
    "\n",
    "- $[{\\bf M}]_{2,3}$ means element on row 2, column 3 in matrix $\\bf M$\n",
    "- $[{\\vec v}]_{2}$ means element on row 2 in vector $\\vec v$\n",
    "- However, something like $[{\\vec v}]_{2,3}$ is invalid because $\\vec v$ is a vector and doesn't have another dimension for $3$ to refer to.\n",
    "- And $[{\\bf M}]_{2}$ is also invalid because we need to specify a column on the matrix.\n",
    "- Also, it goes without saying that, $[x]_{2}$ and $[x]_{2,3}$ would be invalid because $x$ is a scalar.\n",
    "\n",
    "Don't forget, this convension also appies to bracket convension too: \n",
    "\n",
    "- $[\\agl{\\bf M}{}{}{}]_{2,3}$ means element on row 2, column 3 in matrix $\\agl{\\bf M}{}{}{}$\n",
    "- $[\\agl{\\vec v}{}{}{}]_{2}$ means element on row 2 in vector $\\agl{\\vec v}{}{}{}$\n",
    "- However, something like $[\\agl{\\vec v}{}{}{}]_{2,3}$ is invalid because $\\agl{\\vec v}{}{}{}$ is a vector and doesn't have another dimension for $3$ to refer to.\n",
    "- And $[\\agl{\\bf M}{}{}{}]_{2}$ is also invalid because we need to specify a column on the matrix.\n",
    "- Also, it goes without saying that, $[\\agl{x}{}{}{}]_{2}$ and $[\\agl{x}{}{}{}]_{2,3}$ would be invalid because $\\agl{x}{}{}{}$ is a scalar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "## Definitions\n",
    "In order to track the increadible amount of items we'll define how we'll denote those items to help of keep track.  \n",
    "Yes, there is a total of 25 items to track at any given point, but with a properly organized system this will become a simple task.  \n",
    "\n",
    "**Indecies**\n",
    "- A particular training epoch will be indexed by $e$\n",
    "- A particular training datapoint will be indexed by $t$\n",
    "- A particular layer in the network will be indexed by $l$\n",
    "\n",
    "**Constants**\n",
    "- The total number of training epochs will be denoted by $E$\n",
    "- The total number of training datapoints will be denoted by $T$\n",
    "- The total number of layers in the network will be denoted by $L$\n",
    "$\\newcommand{\\I}[2]{\\agl{I}{#1}{}{l}}$\n",
    "- The total number of rows at layer $l$ in training epoch $e$ will be denoted by $\\I{e}{l}$\n",
    "$\\newcommand{\\J}[2]{\\agl{J}{#1}{}{l}}$\n",
    "- The total number of columns at layer $l$ in training epoch $e$ will be denoted by $\\J{e}{l}$\n",
    "\n",
    "**Features and Labels**\n",
    "$\\newcommand{\\x}[1]{\\agl{\\vec{x}}{}{\\phantom{|}#1}{}}$\n",
    "- Feature vector of a given training datapoint will be denoted by $\\x{t}$\n",
    "$\\newcommand{\\y}[1]{\\agl{\\vec{y}}{}{\\phantom{|}#1}{}}$\n",
    "- Label vector corresponding to the feature vector of a given training datapoint will be denoted by $\\y{t}$\n",
    "\n",
    "**Activations, Weights, Biases, etc.**\n",
    "- The activation function will be denoted by $f(z)$\n",
    "There are many activation functions that can be used such as $tanh$ (hyperbolic tangent), $ReLU$ (rectified linear unit), etc., however in this case we'll be using the sigmoid (a.k.a the logistic function).  \n",
    "The sigmoid is defined to be:\n",
    "$$f(z) = \\frac{1}{1+exp(-z)}$$\n",
    "- The derivative of the activation function will be denoted by $f^\\prime(z)$\n",
    "The derivative of the sigmoid is defined to be:\n",
    "$$f^\\prime(z) = \\frac{\\partial{f(z)}}{\\partial{z}} = f(z)\\cdot(1-f(z))$$\n",
    "*Note: We'll be using this for backpropagation.*\n",
    "$\\newcommand{\\W}[2]{\\agl{\\bf{W}}{#1}{}{#2}}$\n",
    "- The weight matrix at layer $l$ in training epoch $e$ will be denoted by $\\W{e}{l}$\n",
    "$\\newcommand{\\b}[2]{\\agl{\\vec{b}}{#1}{}{#2}}$\n",
    "- The bias vector at layer $l$ in training epoch $e$ will be denoted by $\\b{e}{l}$\n",
    "$\\newcommand{\\z}[3]{\\agl{\\vec{z}}{#1}{\\phantom{|}#2}{#3}}$\n",
    "- Pre-activation vector at layer $l$, for training datapoint $t$, in epoch $e$ will be denoted by $\\z{e}{t}{l}$\n",
    "$\\newcommand{\\a}[3]{\\agl{\\vec{a}}{#1}{\\phantom{|}#2}{#3}}$\n",
    "- Activation vector at layer $l$, for training datapoint $t$, in epoch $e$ will be denoted by $\\a{e}{t}{l}$\n",
    "\n",
    "**Predictions, evaluations, and adjustments**\n",
    "$\\newcommand{\\pred}[2]{\\agl{\\hat{y}}{#1}{\\phantom{|}#2}{}}$\n",
    "- Predicted label vector for datapoint $t$ in epoch $e$ will be denoted by $\\pred{e}{t}$\n",
    "$\\newcommand{\\c}[2]{\\agl{C}{#1}{#2}{}}$\n",
    "- The cost for training datapoint $t$ in epoch $e$ will be denoted by $\\c{e}{t}$\n",
    "$\\newcommand{\\d}[3]{\\agl{\\vec{\\delta}}{#1}{\\phantom{|}#2}{#3}}$\n",
    "- The error vector at layer $l$, for training datapoint $t$, in epoch $e$ will be denoted by $\\d{e}{t}{l}$\n",
    "$\\newcommand{\\C}[1]{\\agl{\\bar{C}}{#1}{}{}}$\n",
    "- The average cost for all training datapoints in epoch $e$ will be denoted by $\\C{e}$\n",
    "$\\newcommand{\\dcdW}[3]{\\frac{\\partial\\c{#1}{#2}}{\\partial\\W{#1}{#3}}}$\n",
    "- The adjustment to the weight matrix needed for the next epoch to lower the cost for the particular training datapoint $t$ will be denoted by $\\dcdW{e}{t}{l}$\n",
    "$\\newcommand{\\dcdb}[3]{\\frac{\\partial\\c{#1}{#2}}{\\partial\\b{#1}{#3}}}$\n",
    "- The adjustment to the bias vector needed for the next epoch to lower the cost for the particular training datapoint $t$ will be denoted by $\\dcdb{e}{t}{l}$\n",
    "$\\newcommand{\\dCdW}[2]{\\frac{\\partial\\C{#1}}{\\partial\\W{#1}{#2}}}$\n",
    "- The adjustment to the weight matrix needed for the next epoch to lower the average cost across all training datapoints will be denoted by $\\dCdW{e}{l}$\n",
    "$\\newcommand{\\dCdb}[2]{\\frac{\\partial\\C{#1}}{\\partial\\b{#1}{#2}}}$\n",
    "- The adjustment to the bias vector needed for the next epoch to lower the average cost across all training datapoints will be denoted by $\\dCdb{e}{l}$\n",
    "- The learning rate hyperparameter will be denoted by $\\eta$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "## Overview of the algorithm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "# Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "## Initialzations\n",
    "The following codeblock imports the packages and defines a few helper functions to make it easier do this.  While it's true that NumPy is our only dependency for the actual neural network, IO, presentation, and usability if often not simple and tends to be verbose.  To ensure simplicity and cut down on verbosity on our code moving forward we'll need quite a few imports and functions that do things beyond implementing the actual neural network.  \n",
    "\n",
    "Here we create a class called `MNIST_Digits` which wraps our helper functions that loads the MNIST Handwritten Digits dataset from the filesystem and loads and transforms it in memory as proper NumPy objects for easier manipulation.  It also comes with a few visualization methods to allow us to see our data.  Since we're working on images it's quite helpful to actually see our images that we'll be catagorizing.  \n",
    "\n",
    "We also create a class called `Timer`.  This will be useful when we evaluate our model.  It'll be useful to know how long it takes to train our model.  Accuracy of predictions is important, but we cannot forget the other evaluation metrics such as performance.  \n",
    "\n",
    "Included is a some JavaScript code that's injected into the DOM.  This add buttons to the cells to allow it to toggle.  This will be convenient because we have alot of cells, and enhance usibility to be able to temporarily hide the cells we're not concerned with.  It's quick and dirty, but it's good enough for our purpose.  Perhaps one day there'll be an extension that will do this in a more aesthetically pleasing way.  \n",
    "\n",
    "Also, included is a custom kernel extension that allow for skipping the execution of code some cells.  \n",
    "This depends on `skip_kernel_extension.py` so aside form the pip installed dependencies, and the downloaded MNIST dataset, this notebook also requires that python script inorder to work.  \n",
    "\n",
    "Finally, we also set the NumPy random seed to zero.  This ensures consistent replicable results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cell_style": "center",
    "hidden": false,
    "hide_input": false,
    "hide_output": false,
    "run_control": {
     "marked": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "from IPython.display import display, HTML, Javascript\n",
    "from __future__ import print_function, with_statement\n",
    "\n",
    "# Helper dependencies\n",
    "import struct, gc, os\n",
    "import time, math\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "# loads skip_kernel_extension.py, useful for skiping execution of cells\n",
    "# %load_ext skip_kernel_extension\n",
    "# add a toggle button to the cells, but only if the output exists\n",
    "# display(Javascript(\"\"\"\n",
    "#     setTimeout(function()\n",
    "#     {\n",
    "#         _.each(Jupyter.notebook.get_cells(), function(currentCell)\n",
    "#         {\n",
    "#             return\n",
    "#             if(currentCell.element.find(\".toggle-btn\").length === 0)\n",
    "#             {\n",
    "#                 currentCell.metadata.hidden = !!currentCell.metadata.hidden;\n",
    "#                 var $button = $(document.createElement(\"button\"))\n",
    "#                 $button.addClass(\"toggle-btn\")\n",
    "#                 $button.click(function()\n",
    "#                 {\n",
    "#                     var allCells = Jupyter.notebook.get_cells();\n",
    "#                     var currentElem = $(this).parent()[0], currentCell;\n",
    "#                     for(var inc=0,len=allCells.length; inc<len; inc++)\n",
    "#                     {\n",
    "#                         if(currentElem === allCells[inc].element[0])\n",
    "#                         {\n",
    "#                             currentCell = allCells[inc];\n",
    "#                             break;\n",
    "#                         }\n",
    "#                     }\n",
    "#                     $(currentElem).find(\".input, .output_wrapper, .inner_cell\").toggleClass(\"hide\")\n",
    "#                     currentCell.metadata.hidden = !currentCell.metadata.hidden\n",
    "#                 })\n",
    "#                 if(currentCell instanceof Jupyter.MarkdownCell)\n",
    "#                 {\n",
    "#                     $button.html(\"toggle Markdown Cell\")\n",
    "#                     currentCell.element.append($button)\n",
    "#                 }\n",
    "#                 else\n",
    "#                 {\n",
    "#                     $button.html(\"toggle Code Cell\")\n",
    "#                     currentCell.element.prepend($button)\n",
    "#                 }\n",
    "#             }\n",
    "#             if(currentCell.metadata.hidden)\n",
    "#             {\n",
    "#                 $(currentCell.element).find(\".input, .output_wrapper, .inner_cell\").addClass(\"hide\")\n",
    "#             }\n",
    "#             else\n",
    "#             {\n",
    "#                 $(currentCell.element).find(\".input, .output_wrapper, .inner_cell\").removeClass(\"hide\")\n",
    "#             }\n",
    "#         })\n",
    "#     }, 300);\n",
    "# \"\"\"))\n",
    "\n",
    "# The ONLY dependency for the actual neural network in this project\n",
    "import numpy as np # also used by MNIST_Digits helper class\n",
    "np.random.seed(0) # Setting seed to force replicable results\n",
    "\n",
    "# helper classes/functions\n",
    "class Timer:\n",
    "    \"\"\"Timer\n",
    "    \n",
    "        A timer for performance measurement.\n",
    "        It's a wrapper class that contains the following static methods:\n",
    "        - start, which sets the start time, internally stored in the class.\n",
    "        - stop, which sets the end time, internally stored in the class.\n",
    "        - get_lapse, returns the elapsed time in milliseconds.\n",
    "\n",
    "        Example:\n",
    "            # BEGIN performance measurement\n",
    "            Timer.start()\n",
    "            ...\n",
    "            Timer.stop()\n",
    "            print(\"Code executed in\", Timer.get_lapse(), \"miliseconds\")\n",
    "            # END performance measurement\n",
    "    \"\"\"\n",
    "    class TimerException(Exception):\n",
    "        \"\"\"TimerException\n",
    "        \n",
    "            A timer exception class.  Thorwn when start and stop is called out of order.\n",
    "        \"\"\"\n",
    "        pass\n",
    "    __is_timer_started = None\n",
    "    __startTime = float(\"nan\")\n",
    "    __endTime = float(\"nan\")\n",
    "    def start():\n",
    "        \"\"\"start\n",
    "        \n",
    "            Starts the Timer.\n",
    "            Will throw a TimerException if called after the timer has been started.\n",
    "\n",
    "            raises: TimerException\n",
    "        \"\"\"\n",
    "        if Timer.__is_timer_started: raise Timer.TimerException(\"Timer has already started.  You must call Timer.stop() to stop the timer before you can restart it.\")\n",
    "        Timer.__is_timer_started = True\n",
    "        gc.disable() # disable garbage collector to perevent if from interfaring with the performance measurement\n",
    "        Timer.__startTime = time.time()\n",
    "    def stop():\n",
    "        \"\"\"stop\n",
    "        \n",
    "            Stops the Timer.\n",
    "            Will throw a TimerException if called before the timer has been started.\n",
    "\n",
    "            raises: TimerException\n",
    "        \"\"\"\n",
    "        if not Timer.__is_timer_started: raise Timer.TimerException(\"Timer has not been started.  You must call Timer.start() to start the timer before you can stop it.\")\n",
    "        Timer.__endTime = time.time()\n",
    "        Timer.__is_timer_started = False\n",
    "        gc.enable() # re-enable garbage collector\n",
    "    def get_lapse():\n",
    "        \"\"\"get_lapse\n",
    "        \n",
    "            Returns the elapsed time.  The difference between the end time and the start time.\n",
    "            \n",
    "            raises: TimerException\n",
    "            returns: float\n",
    "        \"\"\"\n",
    "        if Timer.__is_timer_started: raise Timer.TimerException(\"Timer has yet to be stopped.  You must call Timer.stop() to stop the timer before you can get the elapsed time.\")\n",
    "        if math.isnan(Timer.__endTime) or math.isnan(Timer.__startTime): raise Timer.TimerException(\"No measurements has yet to be made, therefore no elapse time measurements to return.  You must run Timer.start() then Timer.stop() before getting the elapsed time.\")\n",
    "        return (Timer.__endTime-Timer.__startTime) * 1000\n",
    "\n",
    "class MNIST_Digits:\n",
    "    \"\"\"MNIST_Digits\n",
    "    \n",
    "        A helper class which contains helper functions to read the MNIST handwritten digits dataset.\n",
    "        It's a wrapper class that contains the following static methods:\n",
    "        - load,\n",
    "        - make_figure\n",
    "    \"\"\"\n",
    "    __training_labels = None\n",
    "    __training_images = None\n",
    "    __testing_labels = None\n",
    "    __testing_images = None\n",
    "    def load(training_labels_path=None,\n",
    "             training_images_path=None,\n",
    "             testing_labels_path=None,\n",
    "             testing_images_path=None):\n",
    "        \"\"\"load\n",
    "            \n",
    "            Loads the MNIST Handwriten Digits dataset from the filesystem.\n",
    "            Must specify the paths to the MNIST data.\n",
    "            Will cache the data, so it will only load once.\n",
    "            Returns a tuple of the datasets.\n",
    "            \n",
    "            \n",
    "            Expects the label datafile to be in the following format:\n",
    "            [offset] [type]          [value]          [description] \n",
    "            0000     32 bit integer  0x00000801(2049) magic number (MSB first) \n",
    "            0004     32 bit integer  ??               number of items \n",
    "            0008     unsigned byte   ??               label \n",
    "            0009     unsigned byte   ??               label \n",
    "            ........ \n",
    "            xxxx     unsigned byte   ??               label\n",
    "            \n",
    "            Expects the image datafile to be in the following format:\n",
    "            [offset] [type]          [value]          [description] \n",
    "            0000     32 bit integer  0x00000803(2051) magic number \n",
    "            0004     32 bit integer  ??               number of images \n",
    "            0008     32 bit integer  28               number of rows \n",
    "            0012     32 bit integer  28               number of columns \n",
    "            0016     unsigned byte   ??               pixel \n",
    "            0017     unsigned byte   ??               pixel \n",
    "            ........ \n",
    "            xxxx     unsigned byte   ??               pixel\n",
    "            \n",
    "            \n",
    "            Example:\n",
    "                (training_labels,\n",
    "                 training_images,\n",
    "                 testing_labels,\n",
    "                 testing_images) = MNIST_Digits.load(training_labels_path=os.getcwd()+\"/MNIST_digits/train-labels-idx1-ubyte\",\n",
    "                                                   training_images_path=os.getcwd()+\"/MNIST_digits/train-images-idx3-ubyte\",\n",
    "                                                   testing_labels_path=os.getcwd()+\"/MNIST_digits/t10k-labels-idx1-ubyte\",\n",
    "                                                   testing_images_path=os.getcwd()+\"/MNIST_digits/t10k-images-idx3-ubyte\")\n",
    "            parameters:\n",
    "            \n",
    "            returns:  \n",
    "        \"\"\"\n",
    "        # If the data is already loaded, just skip oper\n",
    "        if MNIST_Digits.__training_labels == None:\n",
    "            with open(training_labels_path, 'rb') as training_labels_file:\n",
    "                # 1 - Read the entire file as a byte string\n",
    "                # 2 - Slice the byte string from the 8th index (thats where the uint8 label data begins) to the end\n",
    "                # 3 - Convert it to a list\n",
    "                # 4 - Store the resut to MNIST_Digits.__training_labels\n",
    "                MNIST_Digits.__training_labels = np.frombuffer(training_labels_file.read(os.path.getsize(training_labels_path))[8:], dtype=np.uint8)\n",
    "        if MNIST_Digits.__training_images == None:\n",
    "            with open(training_images_path, 'rb') as training_images_file:\n",
    "                # 1 - Read the entire file as a byte string\n",
    "                # 2 - Slice the byte string from the 16th index (thats where the uint8 pixel data begins) to the end\n",
    "                # 3 - Convert it to a NumPy array and reshape\n",
    "                # 4 - Store the resut to MNIST_Digits.__training_images\n",
    "                MNIST_Digits.__training_images = np.frombuffer(training_images_file.read(os.path.getsize(training_images_path))[16:], dtype=np.uint8).reshape(60000,28,28)\n",
    "        if MNIST_Digits.__testing_labels == None:\n",
    "            with open(testing_labels_path, 'rb') as testing_labels_file:\n",
    "                # 1 - Read the entire file as a byte string\n",
    "                # 2 - Slice the byte string from the 8th index (thats where the uint8 label data begins) to the end\n",
    "                # 3 - Convert it to a list\n",
    "                # 4 - Store the resut to MNIST_Digits.__testing_labels\n",
    "                MNIST_Digits.__testing_labels = np.frombuffer(testing_labels_file.read(os.path.getsize(testing_labels_path))[8:], dtype=np.uint8)\n",
    "        if MNIST_Digits.__testing_images == None:\n",
    "            with open(testing_images_path, 'rb') as testing_images_file:\n",
    "                # 1 - Read the entire file as a byte string\n",
    "                # 2 - Slice the byte string from the 16th index (thats where the uint8 pixel data begins) to the end\n",
    "                # 3 - Convert it to an NumPy array and reshape\n",
    "                # 4 - Store the resut to MNIST_Digits.__testing_images\n",
    "                MNIST_Digits.__testing_images = np.frombuffer(testing_images_file.read(os.path.getsize(testing_images_path))[16:], dtype=np.uint8).reshape(10000,28,28)\n",
    "        return MNIST_Digits.__training_labels, MNIST_Digits.__training_images, MNIST_Digits.__testing_labels, MNIST_Digits.__testing_images\n",
    "    def make_subplot(image, ax, title=None):\n",
    "        \"\"\"make_subplot\n",
    "        \n",
    "            Takes a matplotlib.axes.Axes object and modifies it to create the subplot so the image can be visualized.\n",
    "            \n",
    "            Example:\n",
    "                fig = plt.figure(figsize=(15,12))\n",
    "                for index in range(0,5):\n",
    "                    ax = fig.add_subplot(1,5,1+index)\n",
    "                    MNIST_Digits.make_subplot(training_images[index], ax, \"Training Image [\"+str(index)+\"]\\nlabel: \"+str(training_labels[index]))\n",
    "                plt.show()\n",
    "                fig = plt.figure(figsize=(15,12))\n",
    "                for index in range(0,5):\n",
    "                    ax = fig.add_subplot(1,5,1+index)\n",
    "                    MNIST_Digits.make_subplot(training_images[index+5], ax, \"Training Image [\"+str(index+5)+\"]\\nlabel: \"+str(training_labels[index+5]))\n",
    "                plt.show()\n",
    "        \"\"\"\n",
    "        ax.tick_params(axis=\"both\",       which=\"both\",\n",
    "                       left=\"off\",        right=\"off\",\n",
    "                       bottom=\"off\",      top=\"off\",\n",
    "                       labelleft=\"off\",   labelright=\"off\",\n",
    "                       labelbottom=\"off\", labeltop=\"off\")\n",
    "        if title != None: ax.set_title(title)\n",
    "        imgplot = ax.imshow(image, cmap=mpl.cm.Greys)\n",
    "        imgplot.set_interpolation(\"nearest\")\n",
    "    def view_image(image, title=None):\n",
    "        \"\"\"view_image\n",
    "        \n",
    "            A simple method to view the image.            \n",
    "        \"\"\"\n",
    "        fig = plt.figure()\n",
    "        ax = fig.add_subplot(1,1,1)\n",
    "        MNIST_Digits.make_subplot(image, ax, title=title)\n",
    "        plt.show()\n",
    "    def view_data(image, title=None):\n",
    "        \"\"\"view_data\n",
    "        \n",
    "            A simple method to view the image as a grid of pixels.            \n",
    "        \"\"\"\n",
    "        if title != None: print(title)\n",
    "        for image_row in image: print( \" \".join([ str(image_row_item).rjust(3, \" \") for image_row_item in image_row ]) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "## Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "cell_style": "center",
    "hidden": true,
    "hide_input": false,
    "hide_output": false,
    "run_control": {
     "marked": false
    }
   },
   "outputs": [],
   "source": [
    "(training_labels,\n",
    " training_images,\n",
    " testing_labels,\n",
    " testing_images) = MNIST_Digits.load(training_labels_path=os.getcwd() + \"/MNIST_digits/train-labels-idx1-ubyte\",\n",
    "                                     training_images_path=os.getcwd() + \"/MNIST_digits/train-images-idx3-ubyte\",\n",
    "                                     testing_labels_path=os.getcwd()  + \"/MNIST_digits/t10k-labels-idx1-ubyte\",\n",
    "                                     testing_images_path=os.getcwd()  + \"/MNIST_digits/t10k-images-idx3-ubyte\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "**Visualize the data**  \n",
    "Now let's see the actual image.  In this case we'll view the 0th image of the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cell_style": "center",
    "hidden": true,
    "hide_input": false,
    "hide_output": false,
    "run_control": {
     "marked": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ8AAAEoCAYAAABGhnSwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAZA0lEQVR4nO3dfYxc9X3v8fcnxo4BO4kNi+sQyCaIpOIi1ZCBFmG4LlAX0K0MIteKb8l1FFpHJKh1RNPkWgo4pEEIhQQiaNSlNjgJoQRsnhra8iAigi4JDM8GgxOCt7bjh6E8mgC5tr/3j3M2GS87Z2Z/OzvneP15SaOdne+cOd859u+zZ86cB0UEZmaj9Z6yGzCzvZPDw8ySODzMLInDw8ySODzMLInDw8ySODwqRtK/SVrc7edORJJ+IultSQ90+PzTJO2QtFvSaePd30Tn8OiC/D/kjqb/mG81/f6Xo3mtiDgjIlZ1+7mjIWmepE3dft1xckFEnDz0i6SZkm6V9KakQUn/a6gWEfdGxDTgP0vpdILZr+wGJoL8PyQAkjYAfxUR9w5/nqT9ImJnL3vbB10D/BaYBcwBfizpyYh4pty2Jh6veYyjob/gkr4saStwnaQZkv5VUkPSK/n9DzVN8xNJf5Xf/4ykByV9M3/ui5LOSHzuRyQ9IOkNSfdKukbSDzp8Hz+R9A+S/m++NnWnpIMk3SDpdUmPSOpvev5VkjbmtUclndRU21/SqrzHdZL+vnktR9IHJa3Ol8+Lkv5mFMv7QOAc4KsRsSMiHgTuAD7d6WtY5xwe4+8PgJnAh4ElZMv8uvz3w4G3gKsLpv9j4HngYOByYIUkJTz3h8DDwEHAckY/oD6VT3MocATwUP4+ZgLrgIubnvsI2V/9mfl8b5Y0Na9dDPQDHwX+DDh3aCJJ7wHuBJ7M53MqsFTSn3fY48eAnRGxvumxJ4H/1umbtM45PMbfbuDiiHgnIt6KiP+KiNUR8ZuIeAP4BvDfC6YfjIhrI2IXsAqYTbZK3vFzJR0OHAdcFBG/bfqLPBrXRcQLEfEa8G/AC/k2hJ3AzcAxQ0+MiB/k73NnRFwBvBf4eF5eCFwaEa9ExCbgO03zOA7oi4hL8j5/BVxLFlydmAa8Puyx14Dpo3yv1gFv8xh/jYh4e+gXSQcA3wZOB2bkD0+XNCkf9MNtHboTEb/JVySmjfC8ouceDLwcEb9peu5G4LBRvI9tTfffGuH35u0+fwecB3wQCOB9eQ/kj20c1seQDwMflPRq02OTgJ922OOOfF7N3ge80eH0Ngpe8xh/ww9bvpDsr/AfR8T7gKFvClp9FOmGLcDMPLiGjCY4OpZv3/h7sjWMGRHxAbK//kPvbwvwoaZJmvvYCLwYER9ouk2PiDM7nP16YD9JRzY99keAN5aOA4dH700n+0v9qqSZ7LmtYFxExCBQB5ZLmiLpBOAvxml204GdQINsIF/EnmsDPwL+T77h+FDggqbaw8Ab+Qbm/SVNknS0pOM6mXFEvAmsAS6RdKCkE4EFwPe78L5sGIdH710J7A+8BPwM+PcezfcvgROA/wL+AbgJeGcc5vMfZO9pPTAIvM2eH00uATYBLwL3ArcM9ZF/bPsfZBtbXyRbRv8MvH8U8/882fLdDtwInO+vaceHfDKgfZOkm4DnImLc13za9HE+8KmIKNpo3Grau8kCsR4Rf9rB808FVpNtwD0zIu4f7Tzt9xwe+4h81f9lsr/o84HbgBMi4vEe9zGb7Gvah4AjgR8DV0fElb3sw8bO37bsO/6AbHvAQWQfG87vdXDkpgD/BHwEeBX4F+AfS+jDxshrHmaWxBtMzSyJw2Mf1nxsTC+ntYnB4TEBSNpQ5fNT5Aft7Rp26oJ5ZfdlY+MNptYrD0XE3LKbsO7xmscE1u7w/9wRkh7OD5+/Pd/rdWj6P8kPw39V0pNeW7BmDo+JrZPD//838FmyI3B3kh/lmu86/mOyvVFnAn8HrJbUN3wmkg7PA+bwgl6OkfSSpPWSvirJa717OYfHBNbh4f/fj4i1+XEhXwUWSppEdp6NuyLirojYHRH3kB0f866D1CLiP/OD2Fqd3u8B4GjgELKT9SwCvtSVN2mlcXhMYJIOkPRPys7l+TrZIP5AHg5Dmo87GQQmkx0+/2Hgf+ZrFK/mh8nPJVtDGZWI+FVEvJiH0NNkx7d8MvV9WTV41XFiaz78f6ukOcDj7Hn4f/Mh8YcD/4/sgLSNZGslfz0OfQXjewoC6wGveUwckyVNbbrtR2eH/58r6aj8XB+XALfkR7f+APgLSX+eHxo/Vdk5WYdvcG1L0hmSZuX3/5Ds49Htie/TKsLhMXHcRRYUQ7fldHb4//eB68nOQjYV+BuAiNhIdi6MZWTn5thItp3iXf9n8g2mOwo2mJ4KPCXpzbzPNcClCe/RKsTHtphZEq95mFkSh4eZJXF4mFkSh4eZJXF4mFmSUsJD0umSnpf0S0lfKaOHVvLD25+W9ISkesm9rJS0XdLapsdmSrpH0i/ynzOKXqPHvS2XtDlfdk9I6vR6K93u7TBJ90t6VtIzkv42f7zUZVfQVyWW22j1/KvafNfo9WTXKd1Edl3TRRHxbE8baUHZVe5rEfFSBXo5mewqaN+LiKPzxy4nu/rbZXnwzoiIL1ekt+XAjoj4Zq/7GdbbbGB2RDwmaTrwKHAW8BlKXHYFfS2kAstttMpY8zge+GV+vMNvyU6Au6CEPiovIh4gO+N5swVk16El/3lWT5vKteitEiJiS0Q8lt9/g+xC3IdS8rIr6GuvVEZ4HMqeB2NtoloLMIC7JT0qaUnZzYxgVkRsye9vpfVFr8tygaSn8o81pXykaiapn+wi3D+nQstuWF9QseXWCW8wfbe5EXEscAbwhXz1vJIi+8xZpV2EvwscQXbFty3AFWU2I2ka2UWelkbE6821MpfdCH1Varl1qozw2MyeR3J+KH+sEiJic/5zO3Ar2cesKtmWf3Ye+gy9veR+ficitkXErojYDVxLictO0mSyAXpDRKzJHy592Y3UV5WW22iUER6PAEdK+oikKcCngDtK6ONdlF0cefrQfbIrq60tnqrn7gAW5/cXU6GjU4cGZu5sSlp2kgSsANZFxLeaSqUuu1Z9VWW5jVYpB8blX0VdCUwCVkbEN3rexAgkfZRsbQOyc538sMzeJN0IzCM7Oc82skPqbyO70vzhZCfvWRgRPd9w2aK3eWSr3gFsAD7XtI2hl73NBX4KPA3szh9eRrZ9obRlV9DXIiqw3EbLR9WaWRJvMDWzJA4PM0vi8DCzJA4PM0tSanhUdA9OoLq9VbUvcG+pqtxbkbLXPKq80KraW1X7AveWqsq9tVR2eJjZXqqn+3kcfPDB0d/f/7vfG40GfX3vuvRpJVS1t6r2Be4tVTd727BhAy+99FJPLqg1pivGSToduIpsT9F/jojLip7f399PvV7q+XXMJrRardb2OaMdt60kf2zJT+pzDdnRp0cBiyQdlfp6Zjb+ujlux7LNwyf1Mdv7dG3cjiU8Ojqpj6QlkuqS6o1GYwyzM7N2Go0GQ+Mtvw3/JqdrJ+Ma0zaPTkTEADAAUKvVfBSe2Tjq6+tjcHCw/YaPLhjLmkelT+pjZiPq2rgdS3hU9qQ+ZtZS18Zt8seWiNgp6QLgP/j9SX2eSX09Mxt/3Ry3Y9rmERF3AXeN5TXMrLe6NW69e7qZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJdmv7AZsfO3evbuw/s4774zr/FetWtWy9uabbxZO++yzzxbWr7zyysL6smXLWtauvvrqwmn333//wvoVV1xRWD///PML6xPBmMJD0gbgDWAXsDMiat1oyszGT7fGbTfWPP40Il7qwuuYWe+Medx6m4eZJRlreARwt6RHJS0Z6QmSlkiqS6o3Go0xzs7MijQaDYbGW34baVy2HbedGOvHlrkRsVnSIcA9kp6LiAf26DJiABgAqNVqMcb5mVmBvr4+BgcH223DaDtuOzGmNY+I2Jz/3A7cChw/ltczs/HXrXGbHB6SDpQ0feg+MB9Ym/p6Zjb+ujlux/KxZRZwq6Sh1/lhRPz7GF5vwnrttdcK67t27SqsP/nkk4X1u+++u2Xt1VdfLZx2YGCgsF6m/v7+wvqFF15YWF+xYkXL2vvf//7CaU866aTC+imnnFJYr7Cujdvk8IiIXwF/lDq9mfVeN8etv6o1syQODzNL4vAwsyQODzNL4vAwsyQ+JL8LNm3aVFifM2dOYf2VV17pZjt7jfe8p/hvV9FXrdD+sPnzzjuvZe2QQw4pnHbatGmF9b6+vsL6vsBrHmaWxOFhZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWxPt5dMFBBx1UWJ81a1Zhvcr7ecyfP7+w3u69r1mzpmXtve99b+G08+bNK6xbubzmYWZJHB5mlsThYWZJHB5mlsThYWZJHB5mlsThYWZJvJ9HF7Q7r8T1119fWL/lllsK6yeccEJh/ZxzzimsF5k7d25h/fbbby+sT5kypbC+devWlrWrrrqqcFqrNq95mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSRUTPZlar1aJer/dsfnuLd955p7Debl+KZcuWtaxdfvnlhdPef//9hfWTTz65sG7VUqvVqNfr6sW82q55SFopabuktU2PzZR0j6Rf5D9njG+bZtYt3RrTnXxsuR44fdhjXwHui4gjgfvy381s73A9XRjTbcMjIh4AXh728AJgVX5/FXBWu9cxs2ro1phO3WA6KyK25Pe3Ai1P0ilpiaS6pHqj0UicnZl1otFoMDTe8tuSDifteEwPGfOBcRERklpudY2IAWAAsg2mY52fmbXW19fH4OBgbSyv0W5MD0ld89gmaTZA/nN74uuYWTWMekynhscdwOL8/mKg+LhtM6u6UY/pth9bJN0IzAMOlrQJuBi4DPiRpPOAQWBhYsNG++uXtDNjRvo35d/5zncK6yeddFJhXerJLgXWRd0a023DIyIWtSid2nG3ZlYZ3RrT3j3dzJI4PMwsicPDzJI4PMwsicPDzJL40gsTwNKlS1vWHn744cJpb7311sL6M888U1g/+uijC+s2cXnNw8ySODzMLInDw8ySODzMLInDw8ySODzMLInDw8ySeD+PCaDo0gwDAwOF0953332F9QULFhTWzzqr+FSXJ554Ysva2WefXTitD/evNq95mFkSh4eZJXF4mFkSh4eZJXF4mFkSh4eZJXF4mFkSRfTuIm61Wi3q9XrP5mfttTvfx+mnD78e8p5ee+215HmvXLmysH7OOecU1qdNm5Y874mqVqtRr9d7soOM1zzMLInDw8ySODzMLInDw8ySODzMLInDw8ySODzMLInP57GPO/744wvr7a7b8sUvfrGwfvPNN7esffazny2c9oUXXiisf+lLXyqsT58+vbBuY9N2zUPSSknbJa1temy5pM2SnshvZ45vm2bWLd0a0518bLkeGGk3w29HxJz8dlfnrZtZya6nC2O6bXhExAPAy6Pvz8yqqFtjeiwbTC+Q9FS+CjSj1ZMkLZFUl1RvNBpjmJ2ZtdNoNBgab/ltySgm72hMD0kNj+8CRwBzgC3AFa2eGBEDEVGLiFpfX1/i7MysE319fQyNt/xWfAbs3+t4TA9JCo+I2BYRuyJiN3AtULzJ3swqLWVMJ4WHpNlNv54NrG31XDOrvpQx3fZ8HpJuBOYBBwPbgIvz3+cAAWwAPhcRW9rNzOfzmHjefvvtwvrPfvazlrXTTjutcNp2/zc/+clPFtZvuummwvpE1Mn5PLo1ptvuJBYRi0Z4eEW76cysmro1pr17upklcXiYWRKHh5klcXiYWRKHh5kl8SH5NiZTp04trM+bN69lbdKkSYXT7ty5s7B+2223Fdaff/75lrWPf/zjhdNae17zMLMkDg8zS+LwMLMkDg8zS+LwMLMkDg8zS+LwMLMk3s/DCv36178urK9Zs6aw/tBDD7WstduPo53jjjuusP6xj31sTK9vxbzmYWZJHB5mlsThYWZJHB5mlsThYWZJHB5mlsThYWZJvJ/HBNfuEp/XXHNNYf26664rrG/atGnUPXWq3fk++vv7C+tS4RUIbIy85mFmSRweZpbE4WFmSRweZpbE4WFmSRweZpbE4WFmSdru5yHpMOB7wCwggIGIuErSTOAmoB/YACyMiFfGr9V9144dOwrrd955Z8vaJZdcUjjt+vXrk3rqhlNOOaWwftlllxXWP/GJT3SznX1CN8dzJ2seO4ELI+Io4E+AL0g6CvgKcF9EHAncl/9uZtXWtfHcNjwiYktEPJbffwNYBxwKLABW5U9bBZyV8EbMrIe6OZ5Htc1DUj9wDPBzYFZEbMlLW8lWg8xsLzHW8dxxeEiaBqwGlkbE6821iAiyz08jTbdEUl1Svd1xFmY2No1Gg6Hxlt+WjPS81PHcrKPwkDQ5n9ENETF0xtttkmbn9dnA9pGmjYiBiKhFRK2vr6+T2ZlZor6+PobGW34bGP6csYznZm3DQ9mhiSuAdRHxrabSHcDi/P5i4PZ2r2Vm5ermeO7kkPwTgU8DT0t6In9sGXAZ8CNJ5wGDwMLO2t/3vPnmm4X1jRs3FtbPPffcwvrjjz8+6p66Zf78+YX1r33tay1r7S6d4EPqx0XXxnPb8IiIB4FW/4qndtSumVVCN8ez9zA1syQODzNL4vAwsyQODzNL4vAwsyQODzNL4ksvdOitt95qWVu6dGnhtA8++GBh/bnnnkvqqRvOPPPMwvpFF11UWJ8zZ05hffLkyaPuyfYOXvMwsyQODzNL4vAwsyQODzNL4vAwsyQODzNL4vAwsyT7zH4eGzZsKKxfeumlhfV77723ZW1wcDClpa454IADWta+/vWvF077+c9/vrA+ZcqUpJ5s4vOah5klcXiYWRKHh5klcXiYWRKHh5klcXiYWRKHh5kl2Wf281i9enVhfcWKFeM272OPPbawvmjRosL6fvsV/zMtWTLiFQUBmDp1auG0Zqm85mFmSRweZpbE4WFmSRweZpbE4WFmSRweZpbE4WFmSRQRxU+QDgO+B8wCAhiIiKskLQf+GmjkT10WEXcVvVatVot6vT7mps1sZLVajXq9rlb1bo7nTnYS2wlcGBGPSZoOPCrpnrz27Yj4ZgevYWbV0LXx3DY8ImILsCW//4akdcChCU2bWcm6OZ5Htc1DUj9wDPDz/KELJD0laaWkGS2mWSKpLqneaDRGeoqZdUmj0WBovOW3lscupIznZh2Hh6RpwGpgaUS8DnwXOAKYQ5ZkV4w0XUQMREQtImp9fX2dzs7MEvT19TE03vLbwEjPSx3PzToKD0mT8xndEBFrACJiW0TsiojdwLXA8Z28lpmVq1vjuW14SBKwAlgXEd9qenx209POBtaO7i2YWa91czx38m3LicCngaclPZE/tgxYJGkO2dc9G4DPddS9mZWpa+O5k29bHgRG+t648DtgM6uebo5n72FqZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWxOFhZkkcHmaWpO2lF7o6M6kBDDY9dDDwUs8aGJ2q9lbVvsC9pepmbx+OiJ6c77On4fGumUv1iKiV1kCBqvZW1b7AvaWqcm9F/LHFzJI4PMwsSdnhMeJp4Suiqr1VtS9wb6mq3FtLpW7zMLO9V9lrHma2l3J4mFmSUsJD0umSnpf0S0lfKaOHViRtkPS0pCck1UvuZaWk7ZLWNj02U9I9kn6R/2x7TdEe9rZc0uZ82T0h6cySejtM0v2SnpX0jKS/zR8vddkV9FWJ5TZaPd/mIWkSsB74M2AT8AiwKCKe7WkjLUjaANQiovQdiiSdDOwAvhcRR+ePXQ68HBGX5cE7IyK+XJHelgM7IuKbve5nWG+zgdkR8Zik6cCjwFnAZyhx2RX0tZAKLLfRKmPN43jglxHxq4j4LfAvwIIS+qi8iHgAeHnYwwuAVfn9VWT/+XquRW+VEBFbIuKx/P4bwDrgUEpedgV97ZXKCI9DgY1Nv2+iWgswgLslPSppSdnNjGBWRGzJ728FZpXZzAgukPRU/rGmlI9UzST1A8cAP6dCy25YX1Cx5dYJbzB9t7kRcSxwBvCFfPW8kiL7zFml79q/CxwBzAG2AFeU2YykaWRXg18aEa8318pcdiP0Vanl1qkywmMzcFjT7x/KH6uEiNic/9wO3Er2MatKtg1d0Tz/ub3kfn4nIrZFxK6I2A1cS4nLTtJksgF6Q0SsyR8ufdmN1FeVlttolBEejwBHSvqIpCnAp4A7SujjXSQdmG/IQtKBwHxgbfFUPXcHsDi/vxi4vcRe9jA0MHNnU9KykyRgBbAuIr7VVCp12bXqqyrLbbRK2cM0/yrqSmASsDIivtHzJkYg6aNkaxsA+wE/LLM3STcC88gO2d4GXAzcBvwIOJzs9AYLI6LnGy5b9DaPbNU7gA3A55q2MfSyt7nAT4Gngd35w8vIti+UtuwK+lpEBZbbaHn3dDNL4g2mZpbE4WFmSRweZpbE4WFmSRweZpbE4WFmSRweZpbk/wPdET8TJPiC1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "MNIST_Digits.view_image(training_images[0], title=\"Training Image [0]\\nLabel: \"+str(training_labels[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": false
   },
   "source": [
    "**What the computer \"sees\"**  \n",
    "However, it's important to recognize that there is a distiction between what we see and what the computer \"sees\".  \n",
    "To the computer the same image looks like this.  \n",
    "Though it's a grid of numbers, you can still kind of see the digit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Image [0]\n",
      "Label: 5\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   3  18  18  18 126 136 175  26 166 255 247 127   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0  30  36  94 154 170 253 253 253 253 253 225 172 253 242 195  64   0   0   0   0\n",
      "  0   0   0   0   0   0   0  49 238 253 253 253 253 253 253 253 253 251  93  82  82  56  39   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0  18 219 253 253 253 253 253 198 182 247 241   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0  80 156 107 253 253 205  11   0  43 154   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0  14   1 154 253  90   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0 139 253 190   2   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0  11 190 253  70   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0  35 241 225 160 108   1   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0  81 240 253 253 119  25   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0  45 186 253 253 150  27   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  16  93 252 253 187   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 249 253 249  64   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0  46 130 183 253 253 207   2   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0  39 148 229 253 253 253 250 182   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0  24 114 221 253 253 253 253 201  78   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0  23  66 213 253 253 253 253 198  81   2   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0  18 171 219 253 253 253 253 195  80   9   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0  55 172 226 253 253 253 253 244 133  11   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0 136 253 253 253 212 135 132  16   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n"
     ]
    }
   ],
   "source": [
    "MNIST_Digits.view_data(training_images[0], title=\"Training Image [0]\\nLabel: \"+str(training_labels[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": false,
    "hide_input": false,
    "hide_output": false,
    "run_control": {
     "marked": false
    }
   },
   "outputs": [],
   "source": [
    "# Just to recap out dataset is currently stored in the following variables:\n",
    "# - training_labels\n",
    "# - training_images\n",
    "# - testing_labels\n",
    "# - testing_images\n",
    "\n",
    "# input data\n",
    "# Takes the first training image and reshapes it into a vector of pixel data\n",
    "# Yes the class is NumPy Matrix, there is no NumPy Vector class, but\n",
    "#     using this shape will have the math work the same as a vector.\n",
    "# And yes, we are differing from most example that use NumPy arrays.\n",
    "# We are using NumPy Matix class for additional clarity.\n",
    "X = np.matrix(training_images[0].reshape(784,1)) # vector of pixel data (28-row by 28-column matrix becomes a 784-rows by 1-column matix or 784-element vector)\n",
    "\n",
    "# And to prove, that it's still the same piece of data we'll display the image.\n",
    "# Notice that we still get back the same image, Training Image [0] as seen previously.\n",
    "# Keep in mind that X is the vectorized representation of the image pixel grid as a NumPy Matrix class,\n",
    "#     therefore, in order to display, we need to convert it to a NumPy Array,\n",
    "#     then reshape it back to 28x28 grid of image pixels.  \n",
    "print(\"Training Image [0] (image)\")\n",
    "MNIST_Digits.view_image(np.array(X).reshape(28,28))\n",
    "# And once again, this is what the computer \"sees\"\n",
    "print(\"Training Image [0] (pixel data)\")\n",
    "MNIST_Digits.view_data(np.array(X).reshape(28,28))\n",
    "\n",
    "# Before we move forward, we must make one final adjustment.\n",
    "# We need to convert the data type from a NumPy unsigned 8-bit interger (uint8) to a 64-bit NumPy float (float64),\n",
    "# this will (hopefully, but not guarantee) prevent a number overflow.\n",
    "X = X.astype(np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make Input Vector and Expectation Vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now lets make our expected label vector.\n",
    "# This is not quite as strait forward.\n",
    "# Unlike the input vector, were it's taken directly from the dataset, we need to create our expected.\n",
    "# Since we only have 10 catagories we need a 10-element vector to repesent each of the possible catagories.\n",
    "# To keep things simple we'll have:\n",
    "# - the 0th element in the vector correspont to the handwritten digit for zero\n",
    "# - the 1st element in the vector correspont to the handwritten digit for one\n",
    "# - the 2nd element in the vector correspont to the handwritten digit for two\n",
    "# - the 3rd element in the vector correspont to the handwritten digit for three\n",
    "# - the 4th element in the vector correspont to the handwritten digit for four\n",
    "# - the 5th element in the vector correspont to the handwritten digit for five\n",
    "# - the 6th element in the vector correspont to the handwritten digit for six\n",
    "# - the 7th element in the vector correspont to the handwritten digit for seven\n",
    "# - the 8th element in the vector correspont to the handwritten digit for eight\n",
    "# - the 9th element in the vector correspont to the handwritten digit for nine\n",
    "# And yes if we really want to we can make any element correspond to the any digit,\n",
    "#    and the math will still work as long as it corresponds to a unique catagory.\n",
    "\n",
    "# In the case of Training Image [0], it's an image of a handwritten digit for five\n",
    "# To prove this let's print Training Label [0], which corresponds to it.\n",
    "print(\"Training Label [0]\")\n",
    "print(training_labels[0])\n",
    "\n",
    "# Now based on this, let's construct a vector that represents out expectation.\n",
    "# Notice that we set the element corresponding to the handwritten digit for five to 1.\n",
    "# What we're saying is that for this Image, there is a 100% chance that it's a five and \n",
    "#    a 0% chance that it's anything else.\n",
    "y = np.matrix(\"0;0;0;0;0;1;0;0;0;0\") # We use commas to diliminate columns and semicolons to diliminate rows\n",
    "print(\"Our expectation vector for Training Image [0]\")\n",
    "print(y)\n",
    "\n",
    "# For our convenience, let's define a function that creates our expectation vector based on the given label\n",
    "# Rather than manually making our expectation vector, we'll be using this to generate it for us.\n",
    "def create_expectation_vector(label):\n",
    "    return np.matrix(np.array([ int(item == label) for item in range(0,10)]).reshape(10,1))\n",
    "# And to prove that it works\n",
    "print(\"Testing create_expectation_vector.\")\n",
    "print(create_expectation_vector(training_labels[0]))\n",
    "print(\"Is identical to the vector we manually created?\")\n",
    "print(np.array_equal(create_expectation_vector(training_labels[0]), y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedforward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weights and Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we define our weights, biases, and activations for our layers.\n",
    "# In this case we'll create a simple 4-layer netowrk.\n",
    "# For symplicity we'll manually code the layer rather than use indicies to accomodate a variable number of layers.\n",
    "# The size of our layers will be:\n",
    "# - 784 (0th layer, input)\n",
    "# - 16 (1st layer, hidden)\n",
    "# - 16 (2nd layer, hidden)\n",
    "# - 10 (3rd layer, output)\n",
    "# It must be noted that we have no choice over the size of the input or output layer,\n",
    "#     but we do have choice over the size of each hidden and number of hidden layers.\n",
    "\n",
    "# Weights and biases\n",
    "# The number of weights and biases is the number of layers minus 1, 3 in this case.\n",
    "# We will begin our indexing at one rather that zeros because it makes the math look cleaner.\n",
    "# Since we don't know which weights to use to initialize the model, we'll simple fill it with random numbers.\n",
    "# If you happen to know which numbers to use in advance, then by all means initalize with those.\n",
    "# The closer you are to the optimal weights the quicker the model will converge.\n",
    "W1 = np.matrix(np.random.randn(16,784)) # Because the input layer gives us a 784-element vector we need this matrix to have 784 columns\n",
    "W2 = np.matrix(np.random.randn(16,16))  # Because the result of the previous layer gave us a 16-element vector, we need a matrix with 16 columns\n",
    "W3 = np.matrix(np.random.randn(10,16))  # Since we need 10-element vector for our output layer, we give this matrix 10 rows\n",
    "\n",
    "b1 = np.matrix(np.random.randn(16,1))   # Because the input layer gives us a 784-element vector, we need a 784-element bias vector\n",
    "b2 = np.matrix(np.random.randn(16,1))   # Because the result of the previous layer gave us a 16-element vector, we need a 16-element bias vector\n",
    "b3 = np.matrix(np.random.randn(10,1))   # Since we need 10-element vector for our output layer, we need a 10-element bias vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation Function\n",
    "In this case we'll be using he sigmoid (aka logistic) function as our activation function.\n",
    "$$\n",
    "f(z) = \\frac{1}{1+exp(-z)}\n",
    "$$\n",
    "Where $z$ is the input and $exp$ is the exponential function (or euler's constant raised to the power of it's input).  \n",
    "\n",
    "*NOTE: There are many other activation functions that we can use (some that are actually beter such as Rectified Linear Unit $ReLU$, hyperbolic tangent $tanh$, etc.), but for now we'll stick to this function.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we define our activation function\n",
    "# NOTE: Eventually, we'll need the derivative, but we'll define that later\n",
    "def f(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation\n",
    "Now we'll use our activation function with our weight, biases, and input vector to calculate our activation vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Now that we have our Input vector, our Weight maticies\n",
    "# Each layer yields a activation\n",
    "# And each activation in one layer is needed for the activation in the next layer\n",
    "\n",
    "# 0th (input) layer\n",
    "# The activation for the 0th (input) layer is special.\n",
    "# For this we simply pass our input vector through our activation function\n",
    "A0 = f(X)\n",
    "\n",
    "# 1st (hidden) layer\n",
    "# However, subsequent activations (including the output) will be created from the weights and biases, as well as the previous activation.\n",
    "z1 = W1*A0+b1 # To remain consistent with the equations, we will be creating another variable to store the value of z\n",
    "A1 = f(z1)\n",
    "\n",
    "# 2nd (hidden) layer\n",
    "z2 = W2*A1+b2 # To remain consistent with the equations, we will be creating another variable to store the value of z\n",
    "A2 = f(z2)\n",
    "\n",
    "# 3rd (output) layer\n",
    "z3 = W3*A2+b3 # To remain consistent with the equations, we will be creating another variable to store the value of z\n",
    "A3 = f(z3)\n",
    "y_hat = A3 # To ramain consistent with the equation, we'll be using a variable for y hat our prediction\n",
    "\n",
    "# The last activation\n",
    "print(\"Based on the first feedforward our prediction for the handwritten digits are the following:\")\n",
    "print(\"For the handwriten digit being zero: \", float(y_hat[0][0]))\n",
    "print(\"For the handwriten digit being one:  \", float(y_hat[1][0]))\n",
    "print(\"For the handwriten digit being two:  \", float(y_hat[2][0]))\n",
    "print(\"For the handwriten digit being three:\", float(y_hat[3][0]))\n",
    "print(\"For the handwriten digit being four: \", float(y_hat[4][0]))\n",
    "print(\"For the handwriten digit being five: \", float(y_hat[5][0]))\n",
    "print(\"For the handwriten digit being six:  \", float(y_hat[6][0]))\n",
    "print(\"For the handwriten digit being seven:\", float(y_hat[7][0]))\n",
    "print(\"For the handwriten digit being eight:\", float(y_hat[8][0]))\n",
    "print(\"For the handwriten digit being nine: \", float(y_hat[9][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Derivative of the Activation Function\n",
    "The first step thing we must to for backpropagation is to calculate the error vector.  \n",
    "But before we can do that, we must first get the derivative of our activation function.  \n",
    "This will be used to calculate our error for that layer.  \n",
    "$$\n",
    "f^\\prime(z) = \\frac{\\partial{f(z)}}{\\partial{z}} = f(z)\\cdot(1-f(z))\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f_prime(z):\n",
    "    return np.multiply(f(z), (1 - f(z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Error vectors\n",
    "The error vectors for each layer is calculated backwards. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The error for the last layer is special\n",
    "# For this one we calculate it using the differnce in the expected and predicted vectors\n",
    "delta3 = np.multiply(y_hat-y, f_prime(z3))\n",
    "# For subsequent layers, we iterate backwards using the error from the previous layer and\n",
    "#     the derivative of the activation function\n",
    "delta2 = np.multiply(W3.T*delta3, f_prime(z2))\n",
    "delta1 = np.multiply(W2.T*delta2, f_prime(z1))\n",
    "\n",
    "# It should be noted that when we perform gradient decent we actually update our weights and biases with\n",
    "#    the average of the error vectors for each training example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "In this step we update our weights and biases using our error vectors.  \n",
    "Normally we use the average error vectors for each training example, but for now we're only working with one.  \n",
    "We will address this later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before we continue, first let's calculate our cost.\n",
    "# We will need this because this is what we need to minimize.\n",
    "# Actually, we'll need to minimize the average cost function across the training dataset.\n",
    "C = np.mean(np.power(y - y_hat, 2))\n",
    "print(\"The current cost is\", C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update Weights and Biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we'll use the error vectors we calculated from backpropagation to update the weights and biases\n",
    "\n",
    "# Updating the biases is easy, it's subtracting the errors from it\n",
    "# yes delta is the partial derivative of the cost with respect to the bias\n",
    "b1 -= delta1\n",
    "b2 -= delta2\n",
    "b3 -= delta3\n",
    "\n",
    "# However, updating the weights is a little more complicated\n",
    "# yes delta times the transpose of the previous activation vector is the partial derivative of the cost with respect to the weight\n",
    "W1 -= delta1*A0.T\n",
    "W2 -= delta2*A1.T\n",
    "W3 -= delta3*A2.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's run through feedforward again and evaluate the new cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 0th (input) layer\n",
    "A0 = f(X)\n",
    "\n",
    "# 1st (hidden) layer\n",
    "z1 = W1*A0+b1\n",
    "A1 = f(z1)\n",
    "\n",
    "# 2nd (hidden) layer\n",
    "z2 = W2*A1+b2\n",
    "A2 = f(z2)\n",
    "\n",
    "# 3rd (output) layer\n",
    "z3 = W3*A2+b3\n",
    "A3 = f(z3)\n",
    "y_hat = A3\n",
    "\n",
    "# Calculate the new cost\n",
    "# Notice how it's now lower that it was before.\n",
    "# This isn't always the case, but here it is.\n",
    "# So how low is low enough?  0 is ideal, but in practice we just whichever comes first of the following:\n",
    "# - if the cost is below a certain threshold that we know gives us a satisfiable accuracy on the testing data\n",
    "# - if we hit a training epoch limit\n",
    "# - if enough time has elapsed\n",
    "C = np.mean(np.power(y - y_hat, 2))\n",
    "print(\"The new cost is now\", C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And don't forget, if we want to do another run we'll need to calculaer our new the error vectors\n",
    "delta3 = np.multiply(y_hat-y, f_prime(z3))\n",
    "delta2 = np.multiply(W3.T*delta3, f_prime(z2))\n",
    "delta1 = np.multiply(W2.T*delta2, f_prime(z1))\n",
    "# And update the weights and biases\n",
    "b1 -= delta1\n",
    "b2 -= delta2\n",
    "b3 -= delta3\n",
    "W1 -= delta1*A0.T\n",
    "W2 -= delta2*A1.T\n",
    "W3 -= delta3*A2.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's run it many more time and check the new cost\n",
    "Notice how at each successive run the cost continues to decline.  \n",
    "\n",
    "*Note: We are zero indexing our training epochs.  So epoch 0 corresponds to the first epoch, epoch 1 to the second, epoch 2 to the third, and so forth.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we already ran it twice, we already completed 2 training epochs.\n",
    "# Therefore we're starting the loop at epoch 2 or the 3rd epoch\n",
    "for e in range(2,20):\n",
    "    # Feedforward\n",
    "    A0 = f(X)\n",
    "    z1 = W1*A0+b1\n",
    "    A1 = f(z1)\n",
    "    z2 = W2*A1+b2\n",
    "    A2 = f(z2)\n",
    "    z3 = W3*A2+b3\n",
    "    A3 = f(z3)\n",
    "    y_hat = A3\n",
    "    # Back Propagation\n",
    "    delta3 = np.multiply(y_hat-y, f_prime(z3))\n",
    "    delta2 = np.multiply(W3.T*delta3, f_prime(z2))\n",
    "    delta1 = np.multiply(W2.T*delta2, f_prime(z1))\n",
    "    # Gradient Decent\n",
    "    b1 -= delta1\n",
    "    b2 -= delta2\n",
    "    b3 -= delta3\n",
    "    W1 -= delta1*A0.T\n",
    "    W2 -= delta2*A1.T\n",
    "    W3 -= delta3*A2.T\n",
    "\n",
    "    C = np.mean(np.power(y - y_hat, 2))\n",
    "    print(\"At training epoch\", e, \"the cost is now\", C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training across the training datapoints and training epochs\n",
    "Now that we've gone through the basics of feedforward, backpropagation, and gradient descent, let's start to put together a real program.  \n",
    "\n",
    "So far all we've done was code a neural network that looks at only one datapoint and trains itself only against that datapoint.  But what we want to do is to distinguish between different handwritten digits.  So we'll need to train it across all out training datapoints and epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining functions\n",
    "Before we can continue, it's important to note that we have quite a bit of code.  Before this this becomes more complicated than it should be, it'll be prudent to define some functions to package what we've just done thus far for convenience.  \n",
    "\n",
    "For the time being, we'll define some very simple functions to hande our simple 4 layer case.  In practice you'll need to make this flexible, but for now we'll keep it simple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feedforward function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feedforward():\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Backpropagation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backpropagation():\n",
    "    pass"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "oldHeight": 223.181818,
   "position": {
    "height": "40px",
    "left": "766.764px",
    "right": "20px",
    "top": "4.94602px",
    "width": "355px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "varInspector_section_display": "none",
   "window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}