16_loading_models_EuroSat.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.image import load_img, img_to_array\n",
    "from tensorflow.keras.models import Sequential, load_model\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and loading models, with application to the EuroSat dataset\n",
    "\n",
    "#### The EuroSAT dataset\n",
    "\n",
    "![euro](Datasets/EuroSAT/eurosat_overview.jpg)\n",
    "\n",
    "In this assignment, you will use the [EuroSAT dataset](https://github.com/phelber/EuroSAT). It consists of 27000 labelled Sentinel-2 satellite images of different land uses: residential, industrial, highway, river, forest, pasture, herbaceous vegetation, annual crop, permanent crop and sea/lake. For a reference, see the following papers:\n",
    "- Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.\n",
    "- Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. Patrick Helber, Benjamin Bischke, Andreas Dengel. 2018 IEEE International Geoscience and Remote Sensing Symposium, 2018.\n",
    "\n",
    "Your goal is to construct a neural network that classifies a satellite image into one of these 10 classes, as well as applying some of the saving and loading techniques you have learned in the previous sessions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 21600 images belonging to 10 classes.\n",
      "Found 5400 images belonging to 10 classes.\n"
     ]
    }
   ],
   "source": [
    "### Initiate an instnace of ImagaDataGenerator ###\n",
    "image_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,validation_split=0.2)\n",
    "\n",
    "path = \"Datasets\\EuroSAT\\EuroSAT_data\"\n",
    "\n",
    "train_dataset = image_generator.flow_from_directory(batch_size=64,\n",
    "                                                 directory=path,\n",
    "                                                 shuffle=True,\n",
    "                                                 target_size=(64, 64), \n",
    "                                                 subset=\"training\",\n",
    "                                                 class_mode='categorical')\n",
    "\n",
    "validation_dataset = image_generator.flow_from_directory(batch_size=64,\n",
    "                                                 directory=path,\n",
    "                                                 shuffle=True,\n",
    "                                                 target_size=(64, 64), \n",
    "                                                 subset=\"validation\",\n",
    "                                                 class_mode='categorical')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the neural network model\n",
    "You can now construct a model to fit to the data. Using the Sequential API, build your model according to the following specifications:\n",
    "\n",
    "* The model should use the input_shape in the function argument to set the input size in the first layer.\n",
    "* The first layer should be a Conv2D layer with 16 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_1'.\n",
    "* The second layer should also be a Conv2D layer with 8 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_2'.\n",
    "* The third layer should be a MaxPooling2D layer with a pooling window size of 8x8. Name this layer 'pool_1'.\n",
    "* The fourth layer should be a Flatten layer, named 'flatten'.\n",
    "* The fifth layer should be a Dense layer with 32 units, a ReLU activation. Name this layer 'dense_1'.\n",
    "* The sixth and final layer should be a Dense layer with 10 units and softmax activation. Name this layer 'dense_2'.\n",
    "\n",
    "In total, the network should have 6 layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train steps per epoch: 337\n",
      "Validation steps per epoch: 84\n"
     ]
    }
   ],
   "source": [
    "val_steps = validation_dataset.n // validation_dataset.batch_size\n",
    "train_steps = train_dataset.n // train_dataset.batch_size\n",
    "print(f\"Train steps per epoch: {train_steps}\")\n",
    "print(f\"Validation steps per epoch: {val_steps}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does step per epoch mean? It means the number of batches that makes up an epoch. For instant \n",
    "for the validation set there are 83 batches of 64 photos  and 1 batch of 88 photos that is 84 batches in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_new_model():\n",
    "    \"\"\"\n",
    "    This function should build a Sequential model according to the above specification. Ensure the \n",
    "    weights are initialised by providing the input_shape argument in the first layer, given by the\n",
    "    function argument.\n",
    "    Your function should also compile the model with the Adam optimiser, sparse categorical cross\n",
    "    entropy loss function, and a single accuracy metric.\n",
    "    \"\"\"\n",
    "    model = tf.keras.Sequential()\n",
    "    model.add(tf.keras.Input(shape=((64, 64, 3))))\n",
    "    model.add(tf.keras.layers.Conv2D(filters=16,kernel_size=(3,3),activation=\"relu\",padding=\"same\",name=\"conv_1\"))\n",
    "    model.add(tf.keras.layers.Conv2D(filters=8,kernel_size=(3,3),activation=\"relu\",padding=\"same\",name=\"conv_2\"))\n",
    "    model.add(tf.keras.layers.MaxPooling2D(pool_size=(8,8),name=\"pool_1\"))\n",
    "    \n",
    "    model.add(tf.keras.layers.Flatten(name=\"flatten\"))\n",
    "    model.add(tf.keras.layers.Dense(units=32,activation=\"relu\",name=\"dense_1\"))\n",
    "    model.add(tf.keras.layers.Dense(units=10,activation=\"softmax\",name=\"dense_2\"))\n",
    "    \n",
    "    model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "    return model \n",
    "\n",
    "def get_test_accuracy(model,data,steps,**kwargs):\n",
    "    \"\"\"Test model classification accuracy\"\"\"\n",
    "    test_loss, test_acc = model.evaluate_generator(data,steps,**kwargs)\n",
    "    print('accuracy: {acc:0.3f}'.format(acc=test_acc))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_13\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv_1 (Conv2D)              (None, 64, 64, 16)        448       \n",
      "_________________________________________________________________\n",
      "conv_2 (Conv2D)              (None, 64, 64, 8)         1160      \n",
      "_________________________________________________________________\n",
      "pool_1 (MaxPooling2D)        (None, 8, 8, 8)           0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 32)                16416     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 18,354\n",
      "Trainable params: 18,354\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "accuracy: 0.129\n"
     ]
    }
   ],
   "source": [
    "model = get_new_model()\n",
    "model.summary()\n",
    "get_test_accuracy(model,validation_dataset,val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create checkpoints to save model during training, with a criterion\n",
    "\n",
    "You will now create three callbacks:\n",
    "- `checkpoint_every_epoch`: checkpoint that saves the model weights every epoch during training\n",
    "- `checkpoint_best_only`: checkpoint that saves only the weights with the highest validation accuracy. Use the testing data as the validation data.\n",
    "- `early_stopping`: early stopping object that ends training if the validation accuracy has not improved in 3 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_checkpoint_every_epoch():\n",
    "    \"\"\"\n",
    "    This function should return a ModelCheckpoint object that:\n",
    "    - saves the weights only at the end of every epoch\n",
    "    - saves into a directory called 'checkpoints_every_epoch' inside the current working directory\n",
    "    - generates filenames in that directory like 'checkpoint_XXX' where\n",
    "      XXX is the epoch number formatted to have three digits, e.g. 001, 002, 003, etc.\n",
    "    \"\"\"\n",
    "    path = 'Datasets/EuroSAT/checkpoints_every_epoch/checkpoint_{epoch:02d}' # path to save model\n",
    "    checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=path, frequency='epoch',save_weights_only=True,verbose=2)\n",
    "    return checkpoint\n",
    "\n",
    "    \n",
    "\n",
    "def get_checkpoint_best_only():\n",
    "    \"\"\"\n",
    "    This function should return a ModelCheckpoint object that:\n",
    "    - saves only the weights that generate the highest validation (testing) accuracy\n",
    "    - saves into a directory called 'checkpoints_best_only' inside the current working directory\n",
    "    - generates a file called 'checkpoints_best_only/checkpoint'\n",
    "    \"\"\"\n",
    "    path = 'Datasets/EuroSAT/checkpoints_best_only/checkpoint'# path to save model\n",
    "    checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=path, save_best_only=True,save_weights_only=True,verbose=2)\n",
    "    return checkpoint\n",
    "\n",
    "def get_early_stopping():\n",
    "    \"\"\"\n",
    "    This function should return an EarlyStopping callback that stops training when\n",
    "    the validation (testing) accuracy has not improved in the last 3 epochs.\n",
    "    HINT: use the EarlyStopping callback with the correct 'monitor' and 'patience'\n",
    "    \"\"\"\n",
    "    return tf.keras.callbacks.EarlyStopping(monitor='accuracy',min_delta=0.01,patience=3,mode=\"max\")\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train model using the callbacks\n",
    "\n",
    "Now, you will train the model using the three callbacks you created. If you created the callbacks correctly, three things should happen:\n",
    "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_every_epoch`\n",
    "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_best_only` **only** if those weights lead to the highest test accuracy\n",
    "- Training stops when the testing accuracy has not improved in three epochs.\n",
    "\n",
    "You should then have two directories:\n",
    "- A directory called `checkpoints_every_epoch` containing filenames that include `checkpoint_001`, `checkpoint_002`, etc with the `001`, `002` corresponding to the epoch\n",
    "- A directory called `checkpoints_best_only` containing filenames that include `checkpoint`, which contain only the weights leading to the highest testing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "337/337 [==============================] - ETA: 0s - loss: 0.6774 - accuracy: 0.7552\n",
      "Epoch 00001: saving model to Datasets/EuroSAT/checkpoints_every_epoch/checkpoint_01\n",
      "WARNING:tensorflow:Can save best model only with val_loss available, skipping.\n",
      "337/337 [==============================] - 71s 210ms/step - loss: 0.6774 - accuracy: 0.7552\n",
      "Epoch 2/2\n",
      "337/337 [==============================] - ETA: 0s - loss: 0.6518 - accuracy: 0.7671\n",
      "Epoch 00002: saving model to Datasets/EuroSAT/checkpoints_every_epoch/checkpoint_02\n",
      "WARNING:tensorflow:Can save best model only with val_loss available, skipping.\n",
      "337/337 [==============================] - 65s 192ms/step - loss: 0.6518 - accuracy: 0.7671\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to create the callbacks\n",
    "\n",
    "checkpoint_every_epoch = get_checkpoint_every_epoch()\n",
    "checkpoint_best_only = get_checkpoint_best_only()\n",
    "early_stopping = get_early_stopping()\n",
    "\n",
    "\n",
    "# Train model using the callbacks you just created\n",
    "\n",
    "callbacks = [checkpoint_every_epoch, checkpoint_best_only, early_stopping]\n",
    "history = model.fit(train_dataset,steps_per_epoch=train_steps,epochs=2,callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create new instance of model and load on both sets of weights\n",
    "\n",
    "Now you will use the weights you just saved in a fresh model. You should create two functions, both of which take a freshly instantiated model instance:\n",
    "- `model_last_epoch` should contain the weights from the latest saved epoch\n",
    "- `model_best_epoch` should contain the weights from the saved epoch with the highest testing accuracy\n",
    "\n",
    "_Hint: use the_ `tf.train.latest_checkpoint` _function to get the filename of the latest saved checkpoint file. Check the docs_ [_here_](https://www.tensorflow.org/api_docs/python/tf/train/latest_checkpoint)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_last_epoch(model):\n",
    "    \"\"\"\n",
    "    This function should create a new instance of the CNN you created earlier,\n",
    "    load on the weights from the last training epoch, and return this model.\n",
    "    \"\"\"\n",
    "   \n",
    "    path = tf.train.latest_checkpoint(checkpoint_dir=\"Datasets/EuroSAT/checkpoints_every_epoch\")\n",
    "    model.load_weights(filepath= path)\n",
    "    return model\n",
    "    \n",
    "    \n",
    "    \n",
    "### THIS CALLNACK IS ONLy CALLED WHEN WE HAVE A VALIDATION SPLIT ###\n",
    "def get_model_best_epoch(model):\n",
    "    \"\"\"\n",
    "    This function should create a new instance of the CNN you created earlier, load \n",
    "    on the weights leading to the highest validation accuracy, and return this model.\n",
    "    \"\"\"\n",
    "    path = \"Datasets/EuroSAT/checkpoints_best_only/checkpoint\" # path to save model\n",
    "    model.load_weights(filepath= path)\n",
    "    return model \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model with last epoch weights:\n",
      "accuracy: 0.771\n"
     ]
    }
   ],
   "source": [
    "model_last_epoch = get_model_last_epoch(get_new_model())\n",
    "print('Model with last epoch weights:')\n",
    "get_test_accuracy(model_last_epoch,validation_dataset,val_steps)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load, from scratch, a model trained on the EuroSat dataset.\n",
    "\n",
    "In your workspace, you will find another model trained on the `EuroSAT` dataset in `.h5` format. This model is trained on a larger subset of the EuroSAT dataset and has a more complex architecture. The path to the model is `models/EuroSatNet.h5`. See how its testing accuracy compares to your model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_eurosatnet():\n",
    "    \"\"\"\n",
    "    This function should return the pretrained EuroSatNet.h5 model.\n",
    "    \"\"\"\n",
    "\n",
    "    path = \"Datasets\\EuroSAT\\EuroSatNet.h5\" # path to save model\n",
    "    model = tf.keras.models.load_model(filepath=path)\n",
    "    return model\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_21\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv_1 (Conv2D)              (None, 64, 64, 16)        448       \n",
      "_________________________________________________________________\n",
      "conv_2 (Conv2D)              (None, 64, 64, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_1 (MaxPooling2D)        (None, 32, 32, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv_3 (Conv2D)              (None, 32, 32, 16)        2320      \n",
      "_________________________________________________________________\n",
      "conv_4 (Conv2D)              (None, 32, 32, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_2 (MaxPooling2D)        (None, 16, 16, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv_5 (Conv2D)              (None, 16, 16, 16)        2320      \n",
      "_________________________________________________________________\n",
      "conv_6 (Conv2D)              (None, 16, 16, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_3 (MaxPooling2D)        (None, 8, 8, 16)          0         \n",
      "_________________________________________________________________\n",
      "conv_7 (Conv2D)              (None, 8, 8, 16)          2320      \n",
      "_________________________________________________________________\n",
      "conv_8 (Conv2D)              (None, 8, 8, 16)          6416      \n",
      "_________________________________________________________________\n",
      "pool_4 (MaxPooling2D)        (None, 4, 4, 16)          0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 32)                8224      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 41,626\n",
      "Trainable params: 41,626\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "Euro = get_model_eurosatnet()\n",
    "Euro.summary()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "machine_learning_env",
   "language": "python",
   "name": "machine_learning_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}