Building and serialising Keras models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras exercise\n",
    "\n",
    "In this exercise you will be creating a Keras model by loading a data set, preprocessing input data, building a Sequential Keras model and compiling the model with a training configuration. Afterwards, you train your model on the training data and evaluate it on the test set. To finish this exercise, you will then save your model in HDF5 format.\n",
    "\n",
    "QUICK NOTE: ASSIGNMENT DOESN'T SEEM TO WORK IN PYTHON 3.6, PLEASE USE 3.5 FOR NOW - CURRENTLY WORKING ON IT https://www.coursera.org/learn/ai/discussions/all/threads/7LdXcpbGEem16w7sr-jIxA\n",
    "\n",
    "## Data\n",
    "\n",
    "For this exercise we will use the Reuters newswire dataset. This dataset consists of 11,228 newswires from the Reuters news agency. Each wire is encoded as a sequence of word indexes, just as in the IMDB data we encountered in lecture 5 of this series. Moreover, each wire is categorised into one of 46 topics, which will serve as our label. This dataset is available through the Keras API.\n",
    "\n",
    "## Goal\n",
    "\n",
    "We want to create a Multi-layer perceptron (MLP) using Keras which we can train to classify news items into the specified 46 topics.\n",
    "\n",
    "## Instructions\n",
    "\n",
    "We start by importing everything we need for this exercise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pip\n",
    "\n",
    "try:\n",
    "    __import__('keras')\n",
    "except ImportError:\n",
    "    pip.main(['install', 'keras']) \n",
    "    \n",
    "try:\n",
    "    __import__('h5py')\n",
    "except ImportError:\n",
    "    pip.main(['install', 'h5py']) \n",
    "\n",
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "seed = 1337\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT! => In case h5py has been installed, please restart the kernel by clicking on \"Kernel\"->\"Restart and Clear Outout\" and wait until all output disapears. Then your changes are beeing picked up\n",
    "\n",
    "As you can see, we use Keras' Sequential model with only two types of layers: Dense and Dropout. We also specify a random seed to make our results reproducible. Next, we load the Reuters data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/reuters.npz\n",
      "2113536/2110848 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import reuters\n",
    "\n",
    "max_words = 1000\n",
    "(x_train, y_train), (x_test, y_test) = reuters.load_data(num_words=max_words,\n",
    "                                                         test_split=0.2,\n",
    "                                                         seed=seed)\n",
    "num_classes = np.max(y_train) + 1  # 46 topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we cap the maximum number of words in a news item to 1000 by specifying the *num_words* key word. Also, 20% of the data will be test data and we ensure reproducibility by setting our random seed.\n",
    "\n",
    "Our training features are still simply sequences of indexes and we need to further preprocess them, so that we can plug them into a *Dense* layer. For this we use a *Tokenizer* from Keras' text preprocessing module. This tokenizer will take an index sequence and map it to a vector of length *max_words=1000*. Each of the 1000 vector positions corresponds to one of the words in our newswire corpus. The output of the tokenizer has a 1 at the i-th position of the vector, if the word corresponding to i is in the description of the newswire, and 0 otherwise. Even if this word appears multiple times, we still just put a 1 into our vector, i.e. our tokenizer is binary. We use this tokenizer to transform both train and test features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')\n",
    "x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exercise part: label encoding\n",
    "\n",
    "Use to_categorical, as we did in the lectures, to transform both *y_train* and *y_test* into one-hot encoded vectors of length *num_classes*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = to_categorical(y_train,num_classes)\n",
    "y_test = to_categorical(y_test,num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exercise part: model definition\n",
    "\n",
    "Next, initialise a Keras *Sequential* model and add three layers to it:\n",
    "\n",
    "    Layer: Add a *Dense* layer with in input_shape=(max_words,), 512 output units and \"relu\" activation.\n",
    "    Layer: Add a *Dropout* layer with dropout rate of 50%.\n",
    "    Layer: Add a *Dense* layer with num_classes output units and \"softmax\" activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/envs/Python36/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /opt/conda/envs/Python36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()  # Instantiate sequential model\n",
    "model.add(Dense(units = 512,input_shape = (max_words,), activation = 'relu')) # Add first layer. Make sure to specify input shape\n",
    "model.add(Dropout(0.5)) # Add second layer\n",
    "model.add(Dense(units = num_classes, activation = 'softmax')) # Add third layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Exercise part: model compilation\n",
    "\n",
    "As the next step, we need to compile our Keras model with a training configuration. Compile your model with \"categorical_crossentropy\" as loss function, \"adam\" as optimizer and specify \"accuracy\" as evaluation metric. NOTE: In case you get an error regarding h5py, just restart the kernel and start from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer = 'adam', loss='categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#some learners constantly reported 502 errors in Watson Studio. \n",
    "#This is due to the limited resources in the free tier and the heavy resource consumption of Keras.\n",
    "#This is a workaround to limit resource consumption\n",
    "\n",
    "from keras import backend as K\n",
    "\n",
    "K.set_session(K.tf.Session(config=K.tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Exercise part: model training and evaluation\n",
    "\n",
    "Next, define the batch_size for training as 32 and train the model for 5 epochs on *x_train* and *y_train* by using the *fit* method of your model. Then calculate the score for your trained model by running *evaluate* on *x_test* and *y_test* with the same batch size as used in *fit*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/conda/envs/Python36/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/5\n",
      "8982/8982 [==============================] - 10s 1ms/step - loss: 1.3969 - acc: 0.6885\n",
      "Epoch 2/5\n",
      "8982/8982 [==============================] - 11s 1ms/step - loss: 0.7714 - acc: 0.8192\n",
      "Epoch 3/5\n",
      "8982/8982 [==============================] - 10s 1ms/step - loss: 0.5568 - acc: 0.8652\n",
      "Epoch 4/5\n",
      "8982/8982 [==============================] - 7s 785us/step - loss: 0.4190 - acc: 0.8926\n",
      "Epoch 5/5\n",
      "8982/8982 [==============================] - 10s 1ms/step - loss: 0.3466 - acc: 0.9107\n",
      "2246/2246 [==============================] - 1s 305us/step\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "model.fit(x=x_train, y=y_train, batch_size=batch_size, epochs=5)\n",
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have done everything as specified, in particular set the random seed as we did above, your test accuracy should be around 80% "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.798753339269813"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## 5. Exercise part: model serialisation\n",
    "\n",
    "As the final step in this exercise, save your model as \"model.h5\" and upload this file for evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"model.h5\")  # upload this file to the grader in the next code block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To upload the exported model to the grader we first need to encode it based64, we are doing this using a shell command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "!base64 model.h5 > model.h5.base64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we have to install a little library in order to submit to coursera"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-02 09:58:28--  https://raw.githubusercontent.com/IBM/coursera/master/rklib.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2540 (2.5K) [text/plain]\n",
      "Saving to: ‘rklib.py’\n",
      "\n",
      "100%[======================================>] 2,540       --.-K/s   in 0s      \n",
      "\n",
      "2019-09-02 09:58:28 (31.6 MB/s) - ‘rklib.py’ saved [2540/2540]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -f rklib.py\n",
    "!wget https://raw.githubusercontent.com/IBM/coursera/master/rklib.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please provide your email address and obtain a submission token (secret) on the grader’s submission page in coursera, then execute the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submission successful, please check on the coursera grader page for the status\n",
      "-------------------------\n",
      "{\"elements\":[{\"itemId\":\"ozVf2\",\"id\":\"tE4j0qhMEeecqgpT6QjMdA~ozVf2~WKzA5s1oEemSaApDSRskKA\",\"courseId\":\"tE4j0qhMEeecqgpT6QjMdA\"}],\"paging\":{},\"linked\":{}}\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "from rklib import submit\n",
    "key = \"XbAMqtjdEeepUgo7OOVwng\"\n",
    "part = \"LqPRQ\"\n",
    "email = \"dennislamcv@gmail.com\"\n",
    "secret = \"Cxs3fcXN6D76hcGv\"\n",
    "\n",
    "with open('model.h5.base64', 'r') as myfile:\n",
    "    data=myfile.read()\n",
    "submit(email, secret, key, part, [part], data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}