t81_558_class_09_1_keras_transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_5_kaggle_project.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Transfer Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Material\n",
    "\n",
    "* **Part 9.1: Introduction to Keras Transfer Learning** [[Video]](https://www.youtube.com/watch?v=WLlP6S-Z8Xs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_1_keras_transfer.ipynb)\n",
    "* Part 9.2: Popular Pretrained Neural Networks for Keras [[Video]](https://www.youtube.com/watch?v=ctVA1_46YEE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_2_popular_transfer.ipynb)\n",
    "* Part 9.3: Transfer Learning for Computer Vision and Keras [[Video]](https://www.youtube.com/watch?v=61vMUm_XBMI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_3_transfer_cv.ipynb)\n",
    "* Part 9.4: Transfer Learning for Languages and Keras [[Video]](https://www.youtube.com/watch?v=ajmAAg9FxXA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_4_transfer_nlp.ipynb)\n",
    "* Part 9.5: Transfer Learning for Keras Feature Engineering [[Video]](https://www.youtube.com/watch?v=Dttxsm8zpL8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_5_transfer_feature_eng.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "# Start CoLab\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9.1: Introduction to Keras Transfer Learning\n",
    "\n",
    "Human beings learn new skills throughout their entire lives.  However, this learning is rarely from scratch.  No matter what task a human is learning, they are most likely drawing on experiences early in life to learn this new skill.  In this way, humans learn much differently than most deep learning projects. \n",
    "\n",
    "At some point, a human being learns to tell the difference between a cat and a dog.  To teach a neural network, the difference you would obtain a large number of cat pictures and dog pictures.  The neural network would iterate over all of these pictures and train on the differences.  The human child that learned to distinguish between the two animals would probably need to see a few examples when parents told them the name of each type of animal.  The human child would use previous knowledge of looking at different living and non-living objects to help make this classification.  The child would already know the physical appearance of sub-objects, such as fur, eyes, ears, noses, tails, and teeth.\n",
    "\n",
    "Transfer learning attempts to teach a neural network by similar means.  Rather than training your neural network from scratch, you begin training with a preloaded set of weights. Usually, you will remove the topmost layers of the pretrained neural network and retrain it with new uppermost layers.  The layers remaining from the previous neural network will be locked so that training does not change these weights.  Only the newly added layers will be trained.  Figure 9.XFER-LEARN summarizes this process.\n",
    "\n",
    "**Figure 9.XFER-LEARN: Transfer Learning**\n",
    "![Transfer Learning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/transfer.png \"Transfer Learning\")\n",
    "\n",
    "It can take a great deal of computing power to train a neural network for a large image dataset.  Google, Facebook, Microsoft, and other tech companies have utilized GPU arrays for training high-quality neural networks for a variety of applications.  Transferring these weights into your neural network can save you considerable effort and compute time.  It is unlikely that a pretrained model will exactly fit the application that you seek to implement.  Finding the closest pretrained model and using transfer learning is an essential skill for a deep learning engineer.\n",
    "\n",
    "The imagenet dataset has several high-quality neural networks that fit it. [[Cite:deng2009imagenet]](https://ieeexplore.ieee.org/abstract/document/5206848)  In the next parts of this module, we will take a much closer look at this data set.  For many image recognition tasks, an imagenet trained neural network can be a great starting point for your neural networks.\n",
    "\n",
    "### Transfer Learning Example\n",
    "\n",
    "Let's look at an example of where we could use transfer learning to build upon an imagenet neural network.  Microsoft released a website that accepts a picture of a dog and attempts to classify them by breed. We provide the Microsoft dog breed website here: \n",
    "\n",
    "[What breed is that dog?](https://www.bing.com/visualsearch/Microsoft/WhatDog)\n",
    "\n",
    "Let's see how we can classify a specific dog breed.  For this task, it is necessary to obtain pictures of dogs labeled according to breed. Such a network could be trained entirely from scratch.  However, it would require a large quantity of breed-labeled images.  Transfer learning with imagenet could be very beneficial for a neural network project such as this.  A neural network pre-trained on imagenet would already contain neurons that can recognize many subcomponents of the various dog breeds that the neural network had previously seen on the other animal images in imagenet.\n",
    "\n",
    "Finally, we evaluate the accuracy of the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 150 samples\n",
      "Epoch 1/100\n",
      "150/150 - 1s - loss: 1.0437\n",
      "Epoch 2/100\n",
      "150/150 - 0s - loss: 0.9085\n",
      "Epoch 3/100\n",
      "150/150 - 0s - loss: 0.8192\n",
      "Epoch 4/100\n",
      "150/150 - 0s - loss: 0.7486\n",
      "Epoch 5/100\n",
      "150/150 - 0s - loss: 0.6859\n",
      "Epoch 6/100\n",
      "150/150 - 0s - loss: 0.6365\n",
      "Epoch 7/100\n",
      "150/150 - 0s - loss: 0.5906\n",
      "Epoch 8/100\n",
      "150/150 - 0s - loss: 0.5511\n",
      "Epoch 9/100\n",
      "150/150 - 0s - loss: 0.5183\n",
      "Epoch 10/100\n",
      "150/150 - 0s - loss: 0.4888\n",
      "Epoch 11/100\n",
      "150/150 - 0s - loss: 0.4636\n",
      "Epoch 12/100\n",
      "150/150 - 0s - loss: 0.4415\n",
      "Epoch 13/100\n",
      "150/150 - 0s - loss: 0.4230\n",
      "Epoch 14/100\n",
      "150/150 - 0s - loss: 0.4047\n",
      "Epoch 15/100\n",
      "150/150 - 0s - loss: 0.3886\n",
      "Epoch 16/100\n",
      "150/150 - 0s - loss: 0.3741\n",
      "Epoch 17/100\n",
      "150/150 - 0s - loss: 0.3611\n",
      "Epoch 18/100\n",
      "150/150 - 0s - loss: 0.3486\n",
      "Epoch 19/100\n",
      "150/150 - 0s - loss: 0.3357\n",
      "Epoch 20/100\n",
      "150/150 - 0s - loss: 0.3284\n",
      "Epoch 21/100\n",
      "150/150 - 0s - loss: 0.3112\n",
      "Epoch 22/100\n",
      "150/150 - 0s - loss: 0.3033\n",
      "Epoch 23/100\n",
      "150/150 - 0s - loss: 0.2909\n",
      "Epoch 24/100\n",
      "150/150 - 0s - loss: 0.2814\n",
      "Epoch 25/100\n",
      "150/150 - 0s - loss: 0.2713\n",
      "Epoch 26/100\n",
      "150/150 - 0s - loss: 0.2615\n",
      "Epoch 27/100\n",
      "150/150 - 0s - loss: 0.2531\n",
      "Epoch 28/100\n",
      "150/150 - 0s - loss: 0.2465\n",
      "Epoch 29/100\n",
      "150/150 - 0s - loss: 0.2382\n",
      "Epoch 30/100\n",
      "150/150 - 0s - loss: 0.2297\n",
      "Epoch 31/100\n",
      "150/150 - 0s - loss: 0.2248\n",
      "Epoch 32/100\n",
      "150/150 - 0s - loss: 0.2190\n",
      "Epoch 33/100\n",
      "150/150 - 0s - loss: 0.2111\n",
      "Epoch 34/100\n",
      "150/150 - 0s - loss: 0.2037\n",
      "Epoch 35/100\n",
      "150/150 - 0s - loss: 0.1987\n",
      "Epoch 36/100\n",
      "150/150 - 0s - loss: 0.1918\n",
      "Epoch 37/100\n",
      "150/150 - 0s - loss: 0.1876\n",
      "Epoch 38/100\n",
      "150/150 - 0s - loss: 0.1810\n",
      "Epoch 39/100\n",
      "150/150 - 0s - loss: 0.1779\n",
      "Epoch 40/100\n",
      "150/150 - 0s - loss: 0.1710\n",
      "Epoch 41/100\n",
      "150/150 - 0s - loss: 0.1688\n",
      "Epoch 42/100\n",
      "150/150 - 0s - loss: 0.1630\n",
      "Epoch 43/100\n",
      "150/150 - 0s - loss: 0.1596\n",
      "Epoch 44/100\n",
      "150/150 - 0s - loss: 0.1553\n",
      "Epoch 45/100\n",
      "150/150 - 0s - loss: 0.1522\n",
      "Epoch 46/100\n",
      "150/150 - 0s - loss: 0.1507\n",
      "Epoch 47/100\n",
      "150/150 - 0s - loss: 0.1452\n",
      "Epoch 48/100\n",
      "150/150 - 0s - loss: 0.1413\n",
      "Epoch 49/100\n",
      "150/150 - 0s - loss: 0.1389\n",
      "Epoch 50/100\n",
      "150/150 - 0s - loss: 0.1361\n",
      "Epoch 51/100\n",
      "150/150 - 0s - loss: 0.1367\n",
      "Epoch 52/100\n",
      "150/150 - 0s - loss: 0.1316\n",
      "Epoch 53/100\n",
      "150/150 - 0s - loss: 0.1316\n",
      "Epoch 54/100\n",
      "150/150 - 0s - loss: 0.1247\n",
      "Epoch 55/100\n",
      "150/150 - 0s - loss: 0.1233\n",
      "Epoch 56/100\n",
      "150/150 - 0s - loss: 0.1213\n",
      "Epoch 57/100\n",
      "150/150 - 0s - loss: 0.1199\n",
      "Epoch 58/100\n",
      "150/150 - 0s - loss: 0.1222\n",
      "Epoch 59/100\n",
      "150/150 - 0s - loss: 0.1143\n",
      "Epoch 60/100\n",
      "150/150 - 0s - loss: 0.1248\n",
      "Epoch 61/100\n",
      "150/150 - 0s - loss: 0.1122\n",
      "Epoch 62/100\n",
      "150/150 - 0s - loss: 0.1165\n",
      "Epoch 63/100\n",
      "150/150 - 0s - loss: 0.1228\n",
      "Epoch 64/100\n",
      "150/150 - 0s - loss: 0.1091\n",
      "Epoch 65/100\n",
      "150/150 - 0s - loss: 0.1081\n",
      "Epoch 66/100\n",
      "150/150 - 0s - loss: 0.1070\n",
      "Epoch 67/100\n",
      "150/150 - 0s - loss: 0.1034\n",
      "Epoch 68/100\n",
      "150/150 - 0s - loss: 0.1034\n",
      "Epoch 69/100\n",
      "150/150 - 0s - loss: 0.1058\n",
      "Epoch 70/100\n",
      "150/150 - 0s - loss: 0.1004\n",
      "Epoch 71/100\n",
      "150/150 - 0s - loss: 0.1021\n",
      "Epoch 72/100\n",
      "150/150 - 0s - loss: 0.1022\n",
      "Epoch 73/100\n",
      "150/150 - 0s - loss: 0.0984\n",
      "Epoch 74/100\n",
      "150/150 - 0s - loss: 0.0960\n",
      "Epoch 75/100\n",
      "150/150 - 0s - loss: 0.0954\n",
      "Epoch 76/100\n",
      "150/150 - 0s - loss: 0.0980\n",
      "Epoch 77/100\n",
      "150/150 - 0s - loss: 0.0925\n",
      "Epoch 78/100\n",
      "150/150 - 0s - loss: 0.0944\n",
      "Epoch 79/100\n",
      "150/150 - 0s - loss: 0.0911\n",
      "Epoch 80/100\n",
      "150/150 - 0s - loss: 0.0947\n",
      "Epoch 81/100\n",
      "150/150 - 0s - loss: 0.0896\n",
      "Epoch 82/100\n",
      "150/150 - 0s - loss: 0.0914\n",
      "Epoch 83/100\n",
      "150/150 - 0s - loss: 0.0898\n",
      "Epoch 84/100\n",
      "150/150 - 0s - loss: 0.0905\n",
      "Epoch 85/100\n",
      "150/150 - 0s - loss: 0.0959\n",
      "Epoch 86/100\n",
      "150/150 - 0s - loss: 0.0906\n",
      "Epoch 87/100\n",
      "150/150 - 0s - loss: 0.0898\n",
      "Epoch 88/100\n",
      "150/150 - 0s - loss: 0.0876\n",
      "Epoch 89/100\n",
      "150/150 - 0s - loss: 0.0886\n",
      "Epoch 90/100\n",
      "150/150 - 0s - loss: 0.0845\n",
      "Epoch 91/100\n",
      "150/150 - 0s - loss: 0.0861\n",
      "Epoch 92/100\n",
      "150/150 - 0s - loss: 0.0843\n",
      "Epoch 93/100\n",
      "150/150 - 0s - loss: 0.0848\n",
      "Epoch 94/100\n",
      "150/150 - 0s - loss: 0.0811\n",
      "Epoch 95/100\n",
      "150/150 - 0s - loss: 0.0836\n",
      "Epoch 96/100\n",
      "150/150 - 0s - loss: 0.0832\n",
      "Epoch 97/100\n",
      "150/150 - 0s - loss: 0.0820\n",
      "Epoch 98/100\n",
      "150/150 - 0s - loss: 0.0827\n",
      "Epoch 99/100\n",
      "150/150 - 0s - loss: 0.0794\n",
      "Epoch 100/100\n",
      "150/150 - 0s - loss: 0.0848\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1d1fb8db988>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep this example simple, we are not setting aside a validation set.  The goal of this example is to show how to create a multi-layer neural network, where we transfer the weights to another network.  We begin by evaluating the accuracy of the network on the training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the model summary is as expected; we can see the three layers previously defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 3)                 78        \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've trained a neural network on the iris dataset, we can transfer the knowledge of this neural network to other neural networks.  It is possible to create a new neural network from some or all of the layers of this neural network.  To demonstrate the technique, we will create a new neural network that is essentially a clone of the first neural network. We now transfer all of the layers from the original neural network into the new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 3)                 78        \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2 = Sequential()\n",
    "for layer in model.layers:\n",
    "    model2.add(layer)\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check, we would like to calculate the accuracy of the newly created model.  The in-sample accuracy should be the same as the previous model that the new model transferred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model2.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The in-sample accuracy of the newly created neural network is the same as the first neural network.  We've successfully transferred all of the layers from the original neural network.\n",
    "\n",
    "For this example, we will train a neural network to classify three new hypothetical flowers that are uncreatively named:\n",
    "\n",
    "* Fake Flower 1\n",
    "* Fake Flower 2\n",
    "* Fake Flower 3\n",
    "* Fake Flower 4\n",
    "\n",
    "We have measurements for samples of these flowers that conform to the predictors contained in the original iris dataset: sepal width, sepal length, petal width, and petal length. For transfer learning to be effective, the input for the newly trained neural network most closely conforms to the first neural network we transfer. \n",
    "\n",
    "We will strip away the last output layer that contains the softmax activation function that performs this final classification.  We will create a new output layer that will classify the four new flowers.  We will only train the weights in this new layer.  We will mark the first two layers as non-trainable.  The hope is that the first few layers have learned to abstract the raw input data in a way that is also helpful to the new neural network.\n",
    "\n",
    "This process is accomplished by looping over the first few layers and copying them to the new neural network. We output a summary of the new neural network to verify that Keras stripped the previous output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "=================================================================\n",
      "Total params: 1,525\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3 = Sequential()\n",
    "for i in range(2):\n",
    "    layer = model.layers[i]\n",
    "    layer.trainable = False\n",
    "    model3.add(layer)\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the new neural network, we add a 4-neuron classification layer and compile for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 4)                 104       \n",
      "=================================================================\n",
      "Total params: 1,629\n",
      "Trainable params: 104\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3.add(Dense(4,activation='softmax')) # Output\n",
    "\n",
    "model3.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we generate some training data for the 4 fake flowers, and train the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1d1ff923e08>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([\n",
    "    [2.1,0.9,0.8,1.1], # 1\n",
    "    [2.5,1.2,0.8,1.2],\n",
    "    [1.1,3.1,1.1,1.1], # 2\n",
    "    [0.8,2.2,0.7,1.2],\n",
    "    [1.2,0.7,3.1,1.1], # 3\n",
    "    [1.0,1.1,2.4,0.9],\n",
    "    [0.1,1.1,4.1,1.2], # 4\n",
    "    [1.2,0.8,3.1,0.1],\n",
    "])\n",
    "\n",
    "y = np.array([\n",
    "    [0,0,0,1],\n",
    "    [0,0,0,1],\n",
    "    [0,0,1,0],\n",
    "    [0,0,1,0],\n",
    "    [0,1,0,0],\n",
    "    [0,1,0,0],\n",
    "    [1,0,0,0],\n",
    "    [1,0,0,0],\n",
    "])\n",
    "\n",
    "model3.fit(x,y,verbose=0,epochs=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the in-sample accuracy for the new model containing transferred layers from the previous model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model3.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 9](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class9.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}