Functional_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference: Deep Learning with Python, 1st ed, by F. Challot "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from keras import models\n",
    "from keras import layers\n",
    "from keras import optimizers\n",
    "from keras import Input, layers  # for 7.1.1\n",
    "from keras.models import Sequential, Model  # for 7.1.1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 7 cover**:\n",
    "* The Keras functional API\n",
    "* Using Keras callbacks\n",
    "* Working with the TensorBoard visualization tool\n",
    "\n",
    "**Functional API** \n",
    "\n",
    "The Sequential model makes the assumption that the network has exactly one input and exactly one output, and that it consists of a linear stack of layers. But this assumption can be quite in-flexible in a number of cases. Scenarios where sequential models are not enough:\n",
    "1) multi-input models (Eg. multi-modal input data)\n",
    "2) multi-output models \n",
    "3) neural networks with non-linear network topologies (Eg. Residual connections)\n",
    "\n",
    "Functional API is more general and flexible way of working with different models\n",
    "\n",
    "![Getting Started](resid.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.1.1 : Intro to functional API\n",
    "\n",
    "In the functional API, you directly manipulate tensors, and you use layers as functions that take tensors and return tensors (hence, the name functional API)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " dense_15 (Dense)            (None, 32)                2080      \n",
      "                                                                 \n",
      " dense_16 (Dense)            (None, 32)                1056      \n",
      "                                                                 \n",
      " dense_17 (Dense)            (None, 10)                330       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,466\n",
      "Trainable params: 3,466\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# sequential model: .add adds layers to the current model of type \"sequential\"\n",
    "# each feature vector is 1000 dimensional but number of such feature vectors is left arbitrary\n",
    "\n",
    "seq_model = models.Sequential()\n",
    "seq_model.add(layers.Dense(32, activation='relu', input_shape=(64,)))\n",
    "seq_model.add(layers.Dense(32, activation='relu'))\n",
    "seq_model.add(layers.Dense(10, activation='softmax'))\n",
    "\n",
    "seq_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiating a \"Model\" object:\n",
    "* The Model class turns an input tensor and output tensor into a model.\n",
    "* Keras retrieves every layer involved in going from input_tensor to output_tensor, bringing them together into a graph-like data structure—a Model. Wouldn't work for unrelated input output tensor pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_2 (InputLayer)        [(None, 64)]              0         \n",
      "                                                                 \n",
      " dense_12 (Dense)            (None, 32)                2080      \n",
      "                                                                 \n",
      " dense_13 (Dense)            (None, 32)                1056      \n",
      "                                                                 \n",
      " dense_14 (Dense)            (None, 10)                330       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,466\n",
      "Trainable params: 3,466\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# corresponding functional API code\n",
    "# model inputs and output tensors of all layers\n",
    "\n",
    "input_tensor = Input(shape=(64,)) \n",
    "x = layers.Dense(32, activation='relu')(input_tensor)                  \n",
    "x = layers.Dense(32, activation='relu')(x)                             \n",
    "output_tensor = layers.Dense(10, activation='softmax')(x)      \n",
    "\n",
    "model = Model(input_tensor, output_tensor)   \n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compiling, training, or evaluating such an instance of Model, the API is the same as that of Sequential."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "8/8 [==============================] - 1s 2ms/step - loss: 12.9514\n",
      "Epoch 2/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 14.8559\n",
      "Epoch 3/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 17.9237\n",
      "Epoch 4/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 22.5024\n",
      "Epoch 5/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 28.4680\n",
      "Epoch 6/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 35.7465\n",
      "Epoch 7/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 43.5576\n",
      "Epoch 8/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 51.8105\n",
      "Epoch 9/10\n",
      "8/8 [==============================] - 0s 2ms/step - loss: 60.8260\n",
      "Epoch 10/10\n",
      "8/8 [==============================] - 0s 1ms/step - loss: 70.0321\n",
      "32/32 [==============================] - 0s 990us/step - loss: 75.3479\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy') # compiles the model\n",
    "\n",
    "# generate x_train, y_train data\n",
    "x_train = np.random.random((1000, 64)) # 1000 feature vectors of length 64 (one row = one feature vector)\n",
    "y_train = np.random.random((1000, 10)) # 1000 output vectors of length 10 (each row doesn't add up to 1)\n",
    "\n",
    "#  Trains the model for 10 epochs using SGD with minibatches of size 128\n",
    "model.fit(x_train, y_train, epochs=10, batch_size=128)    \n",
    "\n",
    "# Evaluates the model\n",
    "score = model.evaluate(x_train, y_train)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "75.34786224365234"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.1.2: Multi-input models\n",
    "\n",
    "* The functional API can be used to build models that have multiple inputs.\n",
    "* Typically, such models at some point merge their different input branches using a layer that can combine several tensors: by adding (keras.layers.add) them , concatenating (keras.layers.concatenate) them, and so on.\n",
    "\n",
    "We try to build following question and answering model using functional API. It has two inputs - question (text) and news article text info (used to answer question).\n",
    "\n",
    "\n",
    "![Getting Started](qand1.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorizing text\n",
    "Vectorizing text is the process of transforming text into numeric tensors. This can be done in multiple ways:\n",
    "- Segment text into words, and transform each word into a vector.\n",
    "- Segment text into characters, and transform each character into a vector.\n",
    "- Extract n-grams of words or characters, and transform each n-gram into a vector. N-grams are overlapping groups of multiple consecutive words or characters.\n",
    "\n",
    "The units to which text is broken down into are called **tokens** and breaking texts to tokens is called **tokenization**. One such tokenization is one-hot encoding.\n",
    "\n",
    "\n",
    "Problem: One-hot encoding vectors aresparce and very high-dimensional. Therefore, we embed word vectors into low-dimensional spaces.\n",
    "\n",
    "These word-representations are called **word-embeddings**. (Eg. GloVe) There are two ways to obtain word-embeddings:\n",
    "- Use pre-trained word-embeddings\n",
    "- Learn word-embeddings from data using an Embedding layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding layer vs Dense layer\n",
    "Reference: https://stackoverflow.com/questions/47868265/what-is-the-difference-between-an-embedding-layer-and-a-dense-layer\n",
    "\n",
    "Summary: Both do the same multiplication. However, embedding layer works in such a way that the computation is faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details about the code below:\n",
    "\n",
    "* In the Input() function:  \"shape = (32,)\" indicates that the expected input will be batches of 32-dimensional vectors. \"shape=(None,)\" represents dimensions where the shape is not known"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " text (InputLayer)              [(None, None)]       0           []                               \n",
      "                                                                                                  \n",
      " question (InputLayer)          [(None, None)]       0           []                               \n",
      "                                                                                                  \n",
      " embedding_4 (Embedding)        (None, None, 10000)  640000      ['text[0][0]']                   \n",
      "                                                                                                  \n",
      " embedding_5 (Embedding)        (None, None, 10000)  320000      ['question[0][0]']               \n",
      "                                                                                                  \n",
      " lstm_4 (LSTM)                  (None, 32)           1284224     ['embedding_4[0][0]']            \n",
      "                                                                                                  \n",
      " lstm_5 (LSTM)                  (None, 16)           641088      ['embedding_5[0][0]']            \n",
      "                                                                                                  \n",
      " concatenate_1 (Concatenate)    (None, 48)           0           ['lstm_4[0][0]',                 \n",
      "                                                                  'lstm_5[0][0]']                 \n",
      "                                                                                                  \n",
      " dense_19 (Dense)               (None, 500)          24500       ['concatenate_1[0][0]']          \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 2,909,812\n",
      "Trainable params: 2,909,812\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# dictionary lengths for reference text, question and final answer (i.e. output)\n",
    "text_vocabulary_size = 10000\n",
    "question_vocabulary_size = 10000\n",
    "answer_vocabulary_size = 500\n",
    "\n",
    "# text_input represents a tensor-like object that deals with batches of feature vectors of unknown size. It will be of data type int \n",
    "# layer named \"text\" \n",
    "text_input = Input(shape=(None,), dtype='int32', name='text') \n",
    "\n",
    "# takes in \"text_input\" and outputs a word embedding of size 64\n",
    "embedded_text = layers.Embedding(64, text_vocabulary_size)(text_input)  \n",
    "\n",
    "# pass the word-embeddings into an LSTM\n",
    "encoded_text = layers.LSTM(32)(embedded_text)\n",
    "\n",
    "question_input = Input(shape=(None,), dtype='int32', name='question')  \n",
    "embedded_question = layers.Embedding(32, question_vocabulary_size)(question_input)\n",
    "encoded_question = layers.LSTM(16)(embedded_question)\n",
    "\n",
    "\n",
    "concatenated = layers.concatenate([encoded_text, encoded_question], axis=-1) \n",
    "\n",
    "answer = layers.Dense(answer_vocabulary_size, activation='softmax')(concatenated)  \n",
    "\n",
    "\n",
    "model_QA = Model([text_input, question_input], answer)  \n",
    "\n",
    "\n",
    "model_QA.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['acc'])\n",
    "\n",
    "\n",
    "model_QA.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conv1D , Conv2D and Conv3D in keras\n",
    "References:\n",
    "1) https://stackoverflow.com/questions/42883547/intuitive-understanding-of-1d-2d-and-3d-convolutions-in-convolutional-neural-n?noredirect=1&lq=1\n",
    "2) https://datascience.stackexchange.com/questions/51470/what-are-the-differences-between-convolutional1d-convolutional2d-and-convoluti\n",
    "\n",
    "\n",
    "(batch_size, input_dim, channels):\n",
    "input_dim can be 1D, 2D or 3D . Accordingly use Conv1D, conv2D, conv3D respectively.\n",
    "\n",
    "Examples:\n",
    "- 1D input (used for voice signals): 1 second stereo voice signal sampled at 44100 Hz. Here, input_dim = 44100 (44100 points sampled) and channels = 2 (amplitude and frequency stored for each sample in 1D array)\n",
    "- 2D input (used for images): 32x32 RGB image. Here, input_dim = 32x32 (number of points/positions/pixels) and channels = 3 (RBG intensities calculated for each pixel in 2D array) \n",
    "- 3D input (used for videos): 1 second video of 32x32 RGB images at 24 frames per second. So input_dim = 32x32x3 and channels = 24 (24 measurements for each pixel in the 3D array)\n",
    "\n",
    "**Channels**:\n",
    "In case of 1D, 2D, 3D inputs te kernels are of 1D (vector), 2D (matrix) and 3D (array) resp.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.15 ('myenv')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15 (main, Nov 24 2022, 14:39:17) [MSC v.1916 64 bit (AMD64)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "50d8fb942e058ba59e44f174bc1742a5a674676760679353479a427e4b1522e2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}