potential_dishes.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Identify creative dishes: Sushi & Sandwiches  <a class=\"tocSkip\">\n",
        "    \n",
        "**Deep Learning assignment with public available data** \n",
        "\n",
        "Angel Martinez-Tenor "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "toc": true
      },
      "source": [
        "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
        "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Description\" data-toc-modified-id=\"Description-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Description</a></span></li><li><span><a href=\"#Setup\" data-toc-modified-id=\"Setup-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Setup</a></span></li><li><span><a href=\"#Explore-the-data\" data-toc-modified-id=\"Explore-the-data-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Explore the data</a></span></li><li><span><a href=\"#ML-Modelling\" data-toc-modified-id=\"ML-Modelling-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>ML Modelling</a></span><ul class=\"toc-item\"><li><span><a href=\"#Create-image-generators-with-data-augmentation\" data-toc-modified-id=\"Create-image-generators-with-data-augmentation-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Create image generators with data augmentation</a></span></li><li><span><a href=\"#Use-a-pre-trained-convolutional-model-to-extract-the-bottleneck-features\" data-toc-modified-id=\"Use-a-pre-trained-convolutional-model-to-extract-the-bottleneck-features-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Use a pre-trained convolutional model to extract the bottleneck features</a></span></li><li><span><a href=\"#Build-and-train-the-top-classifier\" data-toc-modified-id=\"Build-and-train-the-top-classifier-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Build and train the top classifier</a></span></li><li><span><a href=\"#Build-the-complete-trained-model\" data-toc-modified-id=\"Build-the-complete-trained-model-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>Build the complete trained model</a></span></li></ul></li><li><span><a href=\"#Make-predictions,-and-the-identified-potential-dishes\" data-toc-modified-id=\"Make-predictions,-and-the-identified-potential-dishes-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Make predictions, and the identified potential dishes</a></span></li><li><span><a href=\"#Analysis-of-results-and-&amp;-Future-work\" data-toc-modified-id=\"Analysis-of-results-and-&amp;-Future-work-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Analysis of results and &amp; Future work</a></span></li></ul></div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Description\n",
        "\n",
        "<b>Goal:</b> Identify samples that could potentially be considered as a combination of two dishes given their pictures\n",
        "\n",
        "<b> Input: </b> Two separated folders with pictures of each class. The example provided here uses a dataset with 402 pictures of sandwiches and 402 pictures of sushi. [Link](http://research.us-east-1.s3.amazonaws.com/public/sushi_or_sandwich_photos.zip)\n",
        "\n",
        "Only the best model obtained is shown here: MobileNet with input size (224,224) pre-trained with Imagenet with a small fully connected classified trained and tuned with this data.\n",
        "\n",
        "This implementation is largely influenced and reuses code from the following sources:\n",
        "\n",
        "- [Francois Chollet: 'Building powerful image classification models using very little data'](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)  (main guide)\n",
        "\n",
        "- [Bharat Kunwar: 'Sushi or Sandwich classifier'](https://github.com/brtknr/SushiSandwichClassifier/blob/master/sushi-or-sandwich-keras.ipynb) (base classifier)\n",
        "\n",
        "- [Angel Martinez-Tenor: 'Data science projects with Keras'](https://github.com/angelmtenor/data-science-keras) (setup, structure, and helper functions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Setup\n",
        "\n",
        "Download, extract & split the pictures (train, validation)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-30T17:28:59.820427Z",
          "start_time": "2021-05-30T17:28:57.606680Z"
        }
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import potential_dishes as di\n",
        "\n",
        "# Download, extract & split the pictures (train, validation)\n",
        "di.setup()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Explore the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-30T17:29:03.144854Z",
          "start_time": "2021-05-30T17:28:59.822099Z"
        }
      },
      "outputs": [],
      "source": [
        "di.plot_samples()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# ML Modelling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create image generators with data augmentation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-05-30T17:29:03.148806Z",
          "start_time": "2021-05-30T17:29:03.146464Z"
        }
      },
      "outputs": [],
      "source": [
        "train_datagen, val_datagen = di.create_image_generators()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Use a pre-trained convolutional model to extract the bottleneck features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "start_time": "2021-05-30T17:28:57.614Z"
        }
      },
      "outputs": [],
      "source": [
        "(\n",
        "    model_bottleneck,\n",
        "    train_bottleneck,\n",
        "    val_bottleneck,\n",
        "    train_labels,\n",
        "    val_labels,\n",
        ") = di.get_bottleneck(train_datagen, val_datagen)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##  Build and train the top classifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "start_time": "2021-05-30T17:28:57.618Z"
        }
      },
      "outputs": [],
      "source": [
        "model_top = di.build_top_nn(input_shape=train_bottleneck.shape[1:], summary=True)\n",
        "model_top = di.train_nn(\n",
        "    model_top,\n",
        "    train_bottleneck,\n",
        "    val_bottleneck,\n",
        "    train_labels,\n",
        "    val_labels,\n",
        "    show_plots=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "##  Build the complete trained model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "start_time": "2021-05-30T17:28:57.623Z"
        }
      },
      "outputs": [],
      "source": [
        "full_model = di.build_full_model(model_bottleneck, model_top)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Make predictions, and the identified potential dishes\n",
        "\n",
        "Potential Dishes = pictures misclassified or with output (sigmoid) $\\in$ (0.45, 0.55). Only the validation set is used here to avoid trained samples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecuteTime": {
          "start_time": "2021-05-30T17:28:57.628Z"
        }
      },
      "outputs": [],
      "source": [
        "di.predict_and_save_potential_dishes(full_model, val_datagen)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Analysis of results and & Future work\n",
        "\n",
        "The best model obtained, based on transfer learning with a pre-trained MobileNet, achieved accuracies between 89-92% on the validation set. Less than 80% of accuracy was obtained with smaller custom convolutional models without transfer learning.\n",
        "\n",
        "The generator of the augmented images used to train the classifier is based on the fact that the dishes are usually centered and photographed from different angles.\n",
        "\n",
        "The identified potential dishes contain both actual potential combination and no combination at all. New potential dishes can be obtained by changing the 'SEED' parameter in the main script (different validation set).\n",
        "\n",
        "Better accuracies of the classifier can be obtained by training with a large dataset or by fine-tuning the top layers of the pre-trained MobileNet network. However, it is likely that the identification of potential dishes does not improve. \n",
        "\n",
        "Alternate advanced methods could include Style Transfer or using Generative Adversarial Networks for combining data, as [RemixNet](https://ieeexplore.ieee.org/document/7889574)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3.10.4 ('potential-dishes')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": true,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    },
    "vscode": {
      "interpreter": {
        "hash": "fa91da45ab33ea675abe892dad173c89d8b8a8dbfc839ad50358d8e0754634fa"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}