module_3-lec2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing Image Data Structures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OpenCV supports reading of images in most file formats, such as JPEG, PNG, and TIFF. Most image and \n",
    "# video analysis requires converting images into grayscale first. This simplifies the image and reduces \n",
    "# noise allowing for improved analysis. Let's write some code that reads an image of as person, Floyd \n",
    "# Mayweather and converts it into greyscale.\n",
    "\n",
    "# First we will import the open cv package cv2 \n",
    "import cv2 as cv\n",
    "# We'll load the floyd.jpg image \n",
    "img = cv.imread('readonly/floyd.jpg')\n",
    "# And we'll convert it to grayscale using the cvtColor image\n",
    "gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)\n",
    "\n",
    "# Now, before we get to the result, lets talk about docs. Just like tesseract, opencv is an external\n",
    "# package written in C++, and the docs for python are really poor. This is unfortunatly quite common\n",
    "# when python is being used as a wrapper. Thankfully, the web docs for opencv are actually pretty good,\n",
    "# so hit the website docs.opencv.org when you want to learn more about a particular function. In this\n",
    "# case cvtColor converts from one color space to another, and we are convering our image to grayscale.\n",
    "# Of course, we already know at least two different ways of doing this, using binarization and PIL\n",
    "# color spaces conversions\n",
    "\n",
    "# Lets instpec this object that has been returned.\n",
    "import inspect\n",
    "inspect.getmro(type(gray))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We see that it is of type ndarray, which is a fundamental list type coming from the numerical\n",
    "# python project. That's a bit surprising - up until this point we have been used to working with\n",
    "# PIL.Image objects. OpenCV, however, wants to represent an image as a two dimensional sequence \n",
    "# of bytes, and the ndarray, which stands for n dimensional array, is the ideal way to do this.\n",
    "# Lets look at the array contents.\n",
    "gray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The array is shown here as a list of lists, where the inner lists are filled with integers.\n",
    "# The dtype=uint8 definition indicates that each of the items in an array is an 8 bit unsigned\n",
    "# integer, which is very common for black and white images. So this is a pixel by pixel definition\n",
    "# of the image.\n",
    "#\n",
    "# The display package, however, doesn't know what to do with this image. So lets convert it\n",
    "# into a PIL object to render it in the browser.\n",
    "from PIL import Image\n",
    "\n",
    "# PIL can take an array of data with a given color format and convert this into a PIL object.\n",
    "# This is perfect for our situation, as the PIL color mode, \"L\" is just an array of luminance\n",
    "# values in unsigned integers\n",
    "image = Image.fromarray(gray, \"L\")\n",
    "display(image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets talk a bit more about images for a moment. Numpy arrays are multidimensional. For \n",
    "# instance, we can define an array in a single dimension:\n",
    "import numpy as np\n",
    "single_dim = np.array([25, 50 , 25, 10, 10])\n",
    "\n",
    "# In an image, this is analagous to a single row of 5 pixels each in grayscale. But actually,\n",
    "# all imaging libraries tend to expect at least two dimensions, a width and a height, and to\n",
    "# show a matrix. So if we put the single_dim inside of another array, this would be a two\n",
    "# dimensional array with element in the height direction, and five in the width direction\n",
    "double_dim = np.array([single_dim])\n",
    "\n",
    "double_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This should look pretty familiar, it's a lot like a list of lists! Lets see what this new\n",
    "# two dimensional array looks like if we display it\n",
    "display(Image.fromarray(double_dim, \"L\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pretty unexciting - it's just a little line. Five pixels in a row to be exact, of different\n",
    "# levels of black. The numpy library has a nice attribute called shape that allows us to see how\n",
    "# many dimensions big an array is. The shape attribute returns a tuple that shows the height of\n",
    "# the image, by the width of the image\n",
    "double_dim.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets take a look at the shape of our initial image which we loaded into the img variable\n",
    "img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This image has three dimensions! That's because it has a width, a height, and what's called\n",
    "# a color depth. In this case, the color is represented as an array of three values. Lets take a \n",
    "# look at the color of the first pixel\n",
    "first_pixel=img[0][0]\n",
    "first_pixel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Here we see that the color value is provided in full RGB using an unsigned integer. This\n",
    "# means that each color can have one of 256 values, and the total number of unique colors\n",
    "# that can be represented by this data is 256 * 256 *256 which is roughly 16 million colors.\n",
    "# We call this 24 bit color, which is 8+8+8.\n",
    "#\n",
    "# If you find yourself shopping for a television, you might notice that some expensive models\n",
    "# are advertised as having 10 bit or even 12 bit panels. These are televisions where each of\n",
    "# the red, green, and blue color channels are represented by 10 or 12 bits instead of 8. For\n",
    "# ten bit panels this means that there are 1 billion colors capable, and 12 bit panels are\n",
    "# capable of over 68 billion colors!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're not going to talk much more about color in this course, but it's a fun subject. Instead,\n",
    "# lets go back to this array representation of images, because we can do some interesting things\n",
    "# with this.\n",
    "#\n",
    "# One of the most common things to do with an ndarray is to reshape it -- to change the number\n",
    "# of rows and columns that are represented so that we can do different kinds of operations.\n",
    "# Here is our original two dimensional image\n",
    "print(\"Original image\")\n",
    "print(gray)\n",
    "# If we wanted to represent that as a one dimensional image, we just call reshape\n",
    "print(\"New image\")\n",
    "# And reshape takes the image as the first parameter, and a new shape as the second\n",
    "image1d=np.reshape(gray,(1,gray.shape[0]*gray.shape[1]))\n",
    "print(image1d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# So, why are we talking about these nested arrays of bytes, we were supposed to be talking\n",
    "# about OpenCV as a library. Well, I wanted to show you that often libraries working on the\n",
    "# same kind of principles, in this case images stored as arrays of bytes, are not representing\n",
    "# data in the same way in their APIs. But, by exploring a bit you can learn how the internal\n",
    "# representation of data is stored, and build routines to convert between formats.\n",
    "#\n",
    "# For instance, remember in the last lecture when we wanted to look for gaps in an image so\n",
    "# that we could draw lines to feed into kraken? Well, we use PIL to do this, using getpixel()\n",
    "# to look at individual pixels and see what the luminosity was, then ImageDraw.rectangle to\n",
    "# actually fill in a black bar separator. This was a nice high level API, and let us write\n",
    "# routines to do the work we wanted without having to understand too much about how the images\n",
    "# were being stored. But it was computationally very slow.\n",
    "#\n",
    "# Instead, we could write the code to do this using matrix features within numpy. Lets take\n",
    "# a look.\n",
    "import cv2 as cv\n",
    "# We'll load the 2 column image\n",
    "img = cv.imread('readonly/two_col.png')\n",
    "# And we'll convert it to grayscale using the cvtColor image\n",
    "gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, remember how slicing on a list works, if you have a list of number such as \n",
    "# a=[0,1,2,3,4,5] then a[2:4] will return the sublist of numbers at position 2 through 4 \n",
    "# inclusive - don't forget that lists start indexing at 0!\n",
    "# If we have a two dimensional array, we can slice out a smaller piece of that using the\n",
    "# format a[2:4,1:3]. You can think of this as first slicing along the rows dimension, then\n",
    "# in the columns dimension. So in this example, that would be a matrix of rows 2, and 3,\n",
    "# and columns 1, and 2. Here's a look at our image.\n",
    "gray[2:4,1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# So we see that it is all white. We can use this as a \"window\" and move it around our\n",
    "# our big image.\n",
    "#\n",
    "# Finally, the ndarray library has lots of matrix functions which are generally very fast\n",
    "# to run. One that we want to consider in this case is count_nonzero(), which just returns\n",
    "# the number of entries in the matrix which are not zero.\n",
    "np.count_nonzero(gray[2:4,1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ok, the last benefit of going to this low level approach to images is that we can change\n",
    "# pixels very fast as well. Previously we were drawing rectangles and setting a fill and line\n",
    "# width. This is nice if you want to do something like change the color of the fill from the\n",
    "# line, or draw complex shapes. But we really just want a line here. That's really easy to\n",
    "# do - we just want to change a number of luminosity values from 255 to 0.\n",
    "#\n",
    "# As an example, lets create a big white matrix\n",
    "white_matrix=np.full((12,12),255,dtype=np.uint8)\n",
    "display(Image.fromarray(white_matrix,\"L\"))\n",
    "white_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# looks pretty boring, it's just a giant white square we can't see. But if we want, we can\n",
    "# easily color a column to be black\n",
    "white_matrix[:,6]=np.full((1,12),0,dtype=np.uint8)\n",
    "display(Image.fromarray(white_matrix,\"L\"))\n",
    "white_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# And that's exactly what we wanted to do. So, why do it this way, when it seems so much\n",
    "# more low level? Really, the answer is speed. This paradigm of using matricies to store\n",
    "# and manipulate bytes of data for images is much closer to how low level API and hardware\n",
    "# developers think about storing files and bytes in memory.\n",
    "#\n",
    "# How much faster is it? Well, that's up to you to discover; there's an optional assignment\n",
    "# for this week to convert our old code over into this new format, to compare both the\n",
    "# readability and speed of the two different approaches."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}