module_3-lec3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OpenCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, we're just about at the project for this course. If you reflect on the specialization \n",
    "# as a whole you'll realize that you started with probably little or no understanding of python,\n",
    "# progressed through the basic control structures and libraries included with the language\n",
    "# with the help of a digital textbook, moved on to more high level representations of data\n",
    "# and functions with objects, and now started to explore third party libraries that exist for\n",
    "# python which allow you to manipulate and display images. This is quite an achievement!\n",
    "#\n",
    "# You have also no doubt found that as you have progressed the demands on you to engage in self-\n",
    "# discovery have also increased. Where the first assignments were maybe straight forward, the\n",
    "# ones in this week require you to struggle a bit more with planning and debugging code as\n",
    "# you develop.\n",
    "#\n",
    "# But, you've persisted, and I'd like to share with you just one more set of features before\n",
    "# we head over to a project. The OpenCV library contains mechanisms to do face detection on\n",
    "# images. The technique used is based on Haar cascades, which is a machine learning approach.\n",
    "# Now, we're not going to go into the machine learning bits, we have another specialization on\n",
    "# Applied Data Science with Python which you can take after this if you're interested in that topic.\n",
    "# But here we'll treat OpenCV like a black box.\n",
    "#\n",
    "# OpenCV comes with trained models for detecting faces, eyes, and smiles which we'll be using.\n",
    "# You can train models for detecting other things - like hot dogs or flutes - and if you're\n",
    "# interested in that I'd recommend you check out the Open CV docs on how to train a cascade\n",
    "# classifier: https://docs.opencv.org/3.4/dc/d88/tutorial_traincascade.html\n",
    "# However, in this lecture we just want to use the current classifiers and see if we can detect\n",
    "# portions of an image which are interesting.\n",
    "#\n",
    "# First step is to load opencv and the XML-based classifiers\n",
    "import cv2 as cv\n",
    "face_cascade = cv.CascadeClassifier('readonly/haarcascade_frontalface_default.xml')\n",
    "eye_cascade = cv.CascadeClassifier('readonly/haarcascade_eye.xml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, with the classifiers loaded, we now want to try and detect a face. Lets pull in the\n",
    "# picture we played with last time\n",
    "img = cv.imread('readonly/floyd.jpg')\n",
    "# And we'll convert it to grayscale using the cvtColor image\n",
    "gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)\n",
    "# The next step is to use the face_cascade classifier. I'll let you go explore the docs if you\n",
    "# would like to, but the norm is to use the detectMultiScale() function. This function returns\n",
    "# a list of objects as rectangles. The first parameter is an ndarray of the image.\n",
    "faces = face_cascade.detectMultiScale(gray)\n",
    "# And lets just print those faces out to the screen\n",
    "faces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "faces.tolist()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The resulting rectangles are in the format of (x,y,w,h) where x and y denote the upper\n",
    "# left hand point for the image and the width and height represent the bounding box. We know\n",
    "# how to handle this in PIL\n",
    "from PIL import Image\n",
    "\n",
    "# Lets create a PIL image object\n",
    "pil_img=Image.fromarray(gray,mode=\"L\")\n",
    "\n",
    "# Now lets bring in our drawing object\n",
    "from PIL import ImageDraw\n",
    "# And lets create our drawing context\n",
    "drawing=ImageDraw.Draw(pil_img)\n",
    "\n",
    "# Now lets pull the rectangle out of the faces object\n",
    "rec=faces.tolist()[0]\n",
    "\n",
    "# Now we just draw a rectangle around the bounds\n",
    "drawing.rectangle(rec, outline=\"white\")\n",
    "\n",
    "# And display\n",
    "display(pil_img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# So, not quite what we were looking for. What do you think went wrong?\n",
    "# Well, a quick double check of the docs and it is apparent that OpenCV is return the coordinates\n",
    "# as (x,y,w,h), while PIL.ImageDraw is looking for (x1,y1,x2,y2). Looks like an easy fix\n",
    "# Wipe our old image\n",
    "pil_img=Image.fromarray(gray,mode=\"L\")\n",
    "# Setup our drawing context\n",
    "drawing=ImageDraw.Draw(pil_img)\n",
    "# And draw the new box\n",
    "drawing.rectangle((rec[0],rec[1],rec[0]+rec[2],rec[1]+rec[3]), outline=\"white\")\n",
    "# And display\n",
    "display(pil_img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We see the face detection works pretty good on this image! Note that it's apparent that this is\n",
    "# not head detection, but that the haarcascades file we used is looking for eyes and a mouth.\n",
    "# Lets try this on something a bit more complex, lets read in our MSI recruitment image\n",
    "img = cv.imread('readonly/msi_recruitment.gif')\n",
    "# And lets take a look at that image\n",
    "display(Image.fromarray(img))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Whoa, what's that error about? It looks like there is an error on a line deep within the PIL\n",
    "# Image.py file, and it is trying to call an internal private member called __array_interface__\n",
    "# on the img object, but this object is None\n",
    "#\n",
    "# It turns out that the root of this error is that OpenCV can't work with Gif images. This is\n",
    "# kind of a pain and unfortunate. But we know how to fix that right? One was is that we could\n",
    "# just open this in PIL and then save it as a png, then open that in open cv.\n",
    "#\n",
    "# Lets use PIL to open our image\n",
    "pil_img=Image.open('readonly/msi_recruitment.gif')\n",
    "# now lets convert it to greyscale for opencv, and get the bytestream\n",
    "open_cv_version=pil_img.convert(\"L\")\n",
    "# now lets just write that to a file\n",
    "open_cv_version.save(\"msi_recruitment.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, now that the conversion of format is done, lets try reading this back into opencv\n",
    "cv_img=cv.imread('msi_recruitment.png')\n",
    "# We don't need to color convert this, because we saved it as grayscale\n",
    "# lets try and detect faces in that image\n",
    "faces = face_cascade.detectMultiScale(cv_img)\n",
    "\n",
    "# Now, we still have our PIL color version in a gif\n",
    "pil_img=Image.open('readonly/msi_recruitment.gif')\n",
    "# Set our drawing context\n",
    "drawing=ImageDraw.Draw(pil_img)\n",
    "\n",
    "# For each item in faces, lets surround it with a red box\n",
    "for x,y,w,h in faces:\n",
    "    # That might be new syntax for you! Recall that faces is a list of rectangles in (x,y,w,h)\n",
    "    # format, that is, a list of lists. Instead of having to do an iteration and then manually\n",
    "    # pull out each item, we can use tuple unpacking to pull out individual items in the sublist\n",
    "    # directly to variables. A really nice python feature\n",
    "    #\n",
    "    # Now we just need to draw our box\n",
    "    drawing.rectangle((x,y,x+w,y+h), outline=\"white\")\n",
    "display(pil_img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# What happened here!? We see that we have detected faces, and that we have drawn boxes\n",
    "# around those faces on the image, but that the colors have gone all weird! This, it turns\n",
    "# out, has to do with color limitations for gif images. In short, a gif image has a very\n",
    "# limited number of colors. This is called a color pallette after the pallette artists\n",
    "# use to mix paints. For gifs the pallette can only be 256 colors -- but they can be *any*\n",
    "# 256 colors. When a new color is introduced, is has to take the space of an old color.\n",
    "# In this case, PIL adds white to the pallette but doesn't know which color to replace and\n",
    "# thus messes up the image.\n",
    "#\n",
    "# Who knew there was so much to learn about image formats? We can see what mode the image\n",
    "# is in with the .mode attribute\n",
    "pil_img.mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can see a list of modes in the PILLOW documentation, and they correspond with the\n",
    "# color spaces we have been using. For the moment though, lets change back to RGB, which\n",
    "# represents color as a three byte tuple instead of in a pallette.\n",
    "# Lets read in the image\n",
    "pil_img=Image.open('readonly/msi_recruitment.gif')\n",
    "# Lets convert it to RGB mode\n",
    "pil_img = pil_img.convert(\"RGB\")\n",
    "# And lets print out the mode\n",
    "pil_img.mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, now lets go back to drawing rectangles. Lets get our drawing object\n",
    "drawing=ImageDraw.Draw(pil_img)\n",
    "# And iterate through the faces sequence, tuple unpacking as we go\n",
    "for x,y,w,h in faces:\n",
    "    # And remember this is width and height so we have to add those appropriately.\n",
    "    drawing.rectangle((x,y,x+w,y+h), outline=\"white\")\n",
    "display(pil_img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Awesome! We managed to detect a bunch of faces in that image. Looks like we have missed \n",
    "# four faces. In the machine learning world we would call these false negatives - something\n",
    "# which the machine thought was not a face (so a negative), but that it was incorrect on.\n",
    "# Consequently, we would call the actual faces that were detected as true positives -\n",
    "# something that the machine thought was a face and it was correct on. This leaves us with\n",
    "# false positives - something the machine thought was a face but it wasn't. We see there are\n",
    "# two of these in the image, picking up shadow patterns or textures in shirts and matching\n",
    "# them with the haarcascades. Finally, we have true negatives, or the set of all possible\n",
    "# rectangles the machine learning classifier could consider where it correctly indicated that\n",
    "# the result was not a face. In this case there are many many true negatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# There are a few ways we could try and improve this, and really, it requires a lot of \n",
    "# experimentation to find good values for a given image. First, lets create a function\n",
    "# which will plot rectanges for us over the image\n",
    "def show_rects(faces):\n",
    "    #Lets read in our gif and convert it\n",
    "    pil_img=Image.open('readonly/msi_recruitment.gif').convert(\"RGB\")\n",
    "    # Set our drawing context\n",
    "    drawing=ImageDraw.Draw(pil_img)\n",
    "    # And plot all of the rectangles in faces\n",
    "    for x,y,w,h in faces:\n",
    "        drawing.rectangle((x,y,x+w,y+h), outline=\"white\")\n",
    "    #Finally lets display this\n",
    "    display(pil_img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, first up, we could try and binarize this image. It turns out that opencv has a built in\n",
    "# binarization function called threshold(). You simply pass in the image, the midpoint, and\n",
    "# the maximum value, as well as a flag which indicates whether the threshold should be\n",
    "# binary or something else. Lets try this.\n",
    "cv_img_bin=cv.threshold(img,120,255,cv.THRESH_BINARY)[1] # returns a list, we want the second value\n",
    "# Now do the actual face detection\n",
    "faces = face_cascade.detectMultiScale(cv_img_bin)\n",
    "# Now lets see the results\n",
    "show_rects(faces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# That's kind of interesting. Not better, but we do see that there is one false positive\n",
    "# towards the bottom, where the classifier detected the sunglasses as eyes and the dark shadow\n",
    "# line below as a mouth.\n",
    "#\n",
    "# If you're following in the notebook with this video, why don't you pause things and try a\n",
    "# few different parameters for the thresholding value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The detectMultiScale() function from OpenCV also has a couple of parameters. The first of\n",
    "# these is the scale factor. The scale factor changes the size of rectangles which are\n",
    "# considered against the model, that is, the haarcascades XML file. You can think of it as if\n",
    "# it were changing the size of the rectangles which are on the screen.\n",
    "#\n",
    "# Lets experiment with the scale factor. Usually it's a small value, lets try 1.05\n",
    "faces = face_cascade.detectMultiScale(cv_img,1.05)\n",
    "# Show those results\n",
    "show_rects(faces)\n",
    "# Now lets also try 1.15\n",
    "faces = face_cascade.detectMultiScale(cv_img,1.15)\n",
    "# Show those results\n",
    "show_rects(faces)\n",
    "# Finally lets also try 1.25\n",
    "faces = face_cascade.detectMultiScale(cv_img,1.25)\n",
    "# Show those results\n",
    "show_rects(faces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We can see that as we change the scale factor we change the number of true and \n",
    "# false positives and negatives. With the scale set to 1.05, we have 7 true positives,\n",
    "# which are correctly identified faces, and 3 false negatives, which are faces which\n",
    "# are there but not detected, and 3 false positives, where are non-faces which\n",
    "# opencv thinks are faces. When we change this to 1.15 we lose the false positives but\n",
    "# also lose one of the true positives, the person to the right wearing a hat. And\n",
    "# when we change this to 1.25 we lost more true positives as well.\n",
    "#\n",
    "# This is actually a really interesting phenomena in machine learning and artificial\n",
    "# intelligence. There is a trade off between not only how accurate a model is, but how\n",
    "# the inaccuracy actually happens. Which of these three models do you think is best?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Well, the answer to that question is really, \"it depends\". It depends why you are trying\n",
    "# to detect faces, and what you are going to do with them. If you think these issues\n",
    "# are interesting, you might want to check out the Applied Data Science with Python\n",
    "# specialization Michigan offers on Coursera.\n",
    "#\n",
    "# Ok, beyond an opportunity to advertise, did you notice anything else that happened when\n",
    "# we changed the scale factor? It's subtle, but the speed at which the processing ran\n",
    "# took longer at smaller scale factors. This is because more subimages are being considered\n",
    "# for these scales. This could also affect which method we might use.\n",
    "#\n",
    "# Jupyter has nice support for timing commands. You might have seen this before, a line\n",
    "# that starts with a percentage sign in jupyter is called a \"magic function\". This isn't\n",
    "# normal python - it's actually a shorthand way of writing a function which Jupyter\n",
    "# has predefined. It looks a lot like the decorators we talked about in a previous\n",
    "# lecture, but the magic functions were around long before decorators were part of the\n",
    "# python language. One of the built-in magic functions in juptyer is called timeit, and this\n",
    "# repeats a piece of python ten times (by default) and tells you the average speed it\n",
    "# took to complete.\n",
    "#\n",
    "# Lets time the speed of detectmultiscale when using a scale of 1.05\n",
    "%timeit face_cascade.detectMultiScale(cv_img,1.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ok, now lets compare that to the speed at scale = 1.15\n",
    "%timeit face_cascade.detectMultiScale(cv_img,1.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You can see that this is a dramatic difference, roughly two and a half times slower\n",
    "# when using the smaller scale!\n",
    "#\n",
    "# This wraps up our discussion of detecting faces in opencv. You'll see that, like OCR, this\n",
    "# is not a foolproof process. But we can build on the work others have done in machine learning\n",
    "# and leverage powerful libraries to bring us closer to building a turn key python-based\n",
    "# solution. Remember that the detection mechanism isn't specific to faces, that's just the\n",
    "# haarcascades training data we used. On the web you'll be able to find other training data\n",
    "# to detect other objects, including eyes, animals, and so forth."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}