03-python2-defense-instructors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defensive Programming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<section class=\"objectives panel panel-warning\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-certificate\"></span>Learning Objectives</h2>\n",
    "</div>\n",
    "<div class=\"panel-body\">\n",
    "<ul>\n",
    "<li>Explain what an assertion is.</li>\n",
    "<li>Add assertions that check the program’s state is correct.</li>\n",
    "<li>Correctly add precondition and postcondition assertions to functions.</li>\n",
    "<li>Explain what test-driven development is, and use it when creating new functions.</li>\n",
    "<li>Explain why variables should be initialized using actual data values rather than arbitrary constants.</li>\n",
    "</ul>\n",
    "</div>\n",
    "</section>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our previous lessons have introduced the basic tools of programming: variables and lists, file I/O, loops, conditionals, and functions. What they haven’t done is show us how to tell whether a program is getting the right answer, and how to tell if it’s still getting the right answer as we make changes to it.\n",
    "\n",
    "To achieve that, we need to:\n",
    "\n",
    "- Write programs that check their own operation.\n",
    "- Write and run tests for widely-used functions.\n",
    "- Make sure we know what “correct” actually means.\n",
    "\n",
    "The good news is, doing these things will speed up our programming, not slow it down. As in real carpentry — the kind done with lumber — the time saved by measuring carefully before cutting a piece of wood is much greater than the time that measuring takes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assertions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step toward getting the right answers from our programs is to assume that mistakes _will_ happen and to guard against them. This is called [defensive programming](http://swcarpentry.github.io/python-novice-inflammation/reference.html#defensive-programming), and the most common way to do it is to add [assertions](http://swcarpentry.github.io/python-novice-inflammation/reference.html#assertion) to our code so that it checks itself as it runs. An assertion is simply a statement that something must be true at a certain point in a program. When Python sees one, it evaluates the assertion’s condition. If it’s true, Python does nothing, but if it’s false, Python halts the program immediately and prints the error message if one is provided. For example, this piece of code halts as soon as the loop encounters a value that isn’t positive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Usual import first\n",
    "from __future__ import print_function, division\n",
    "\n",
    "numbers = [1.5, 2.3, 0.7, -0.001, 4.4]\n",
    "total = 0.0\n",
    "for n in numbers:\n",
    "    assert n > 0.0, 'Data should only contain positive values'\n",
    "    total += n\n",
    "print('total is:', total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programs like the Firefox browser are full of assertions: 10-20% of the code they contain are there to check that the other 80-90% are working correctly. Broadly speaking, assertions fall into three categories:\n",
    "\n",
    "- A [precondition](http://swcarpentry.github.io/python-novice-inflammation/reference.html#precondition) is something that must be true at the start of a function in order for it to work correctly.\n",
    "- A [postcondition](http://swcarpentry.github.io/python-novice-inflammation/reference.html#postcondition) is something that the function guarantees is true when it finishes.\n",
    "- An [invariant](http://swcarpentry.github.io/python-novice-inflammation/reference.html#invariant) is something that is always true at a particular point inside a piece of code.\n",
    "\n",
    "For example, suppose we are representing rectangles using a [tuple](http://swcarpentry.github.io/python-novice-inflammation/reference.html#tuple) of four coordinates `(x0, y0, x1, y1)`, representing the lower left and upper right corners of the rectangle. In order to do some calculations, we need to normalize the rectangle so that the lower left corner is at the origin and the longest side is 1.0 units long. This function does that, but checks that its input is correctly formatted and that its result makes sense:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def normalize_rectangle(rect):\n",
    "    '''Normalizes a rectangle so that it is at the origin and 1.0 units long on its longest axis.'''\n",
    "    # Some preconditions to catch invalid inputs\n",
    "    assert len(rect) == 4, 'Rectangles must contain 4 coordinates'\n",
    "    x0, y0, x1, y1 = rect\n",
    "    assert x0 < x1, 'Invalid X coordinates'\n",
    "    assert y0 < y1, 'Invalid Y coordinates'\n",
    "\n",
    "    # Define width of rectangle in either direction\n",
    "    dx = x1 - x0\n",
    "    dy = y1 - y0\n",
    "    # If x is longer, scale to x\n",
    "    if dx > dy:\n",
    "        scaled = dx / dy\n",
    "        upper_x, upper_y = 1.0, scaled\n",
    "    # Otherwise scale to y\n",
    "    else:\n",
    "        scaled = dx / dy\n",
    "        upper_x, upper_y = scaled, 1.0\n",
    "\n",
    "    # Reality check on results\n",
    "    assert 0 < upper_x <= 1.0, 'Calculated upper X coordinate invalid'\n",
    "    assert 0 < upper_y <= 1.0, 'Calculated upper Y coordinate invalid'\n",
    "\n",
    "    return (0, 0, upper_x, upper_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preconditions on lines 2, 4, and 5 catch invalid inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(normalize_rectangle( (0.0, 1.0, 2.0) )) # missing the fourth coordinate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(normalize_rectangle( (4.0, 2.0, 1.0, 5.0) )) # X axis inverted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The post-conditions help us catch bugs by telling us when our calculations cannot have been correct. For example, if we normalize a rectangle that is taller than it is wide everything seems OK:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(normalize_rectangle( (0.0, 0.0, 1.0, 5.0) ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but if we normalize one that’s wider than it is tall, the assertion is triggered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(normalize_rectangle( (0.0, 0.0, 5.0, 1.0) ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-reading our function, we realize that line 10 should divide `dy` by `dx` rather than `dx` by `dy`. (You can display line numbers by typing Ctrl-M, then L.) If we had left out the assertion at the end of the function, we would have created and returned something that had the right shape as a valid answer, but wasn’t. Detecting and debugging that would almost certainly have taken more time in the long run than writing the assertion.\n",
    "\n",
    "But assertions aren’t just about catching errors: they also help people understand programs. Each assertion gives the person reading the program a chance to check (consciously or otherwise) that their understanding matches what the code is doing.\n",
    "\n",
    "Most good programmers follow two rules when adding assertions to their code. The first is, _fail early, fail often_. The greater the distance between when and where an error occurs and when it’s noticed, the harder the error will be to debug, so good code catches mistakes as early as possible.\n",
    "\n",
    "The second rule is, _turn bugs into assertions or tests_. Whenever you fix a bug, write an assertion that catches the mistake should you make it again. If you made a mistake in a piece of code, the odds are good that you have made other mistakes nearby, or will make the same mistake (or a related one) the next time you change it. Writing assertions to check that you haven’t [regressed](http://swcarpentry.github.io/python-novice-inflammation/reference.html#regression) (i.e., haven’t re-introduced an old problem) can save a lot of time in the long run, and helps to warn people who are reading the code (including your future self) that this bit is tricky."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test-Driven Development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An assertion checks that something is true at a particular point in the program. The next step is to check the overall behavior of a piece of code, i.e., to make sure that it produces the right output when it’s given a particular input. For example, suppose we need to find where two or more time series overlap. The range of each time series is represented as a pair of numbers, which are the time the interval started and ended. The output is the largest range that they all include:\n",
    "\n",
    "![Overlapping Ranges](fig/python-overlapping-ranges.svg)\n",
    "\n",
    "Most novice programmers would solve this problem like this:\n",
    "\n",
    "1. Write a function `range_overlap`.\n",
    "2. Call it interactively on two or three different inputs.\n",
    "3. If it produces the wrong answer, fix the function and re-run that test.\n",
    "\n",
    "This clearly works — after all, thousands of scientists are doing it right now — but there’s a better way:\n",
    "\n",
    "1. Write a short function for each test.\n",
    "2. Write a `range_overlap` function that should pass those tests.\n",
    "3. If `range_overlap` produces any wrong answers, fix it and re-run the test functions.\n",
    "\n",
    "Writing the tests before writing the function they exercise is called [test-driven development](http://swcarpentry.github.io/python-novice-inflammation/reference.html#test-driven-development) (TDD). Its advocates believe it produces better code faster because:\n",
    "\n",
    "1. If people write tests after writing the thing to be tested, they are subject to confirmation bias, i.e., they subconsciously write tests to show that their code is correct, rather than to find errors.\n",
    "2. Writing tests helps programmers figure out what the function is actually supposed to do.\n",
    "\n",
    "Here are three test functions for `range_overlap`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "assert range_overlap([ (0.0, 1.0) ]) == (0.0, 1.0)\n",
    "assert range_overlap([ (2.0, 3.0), (2.0, 4.0) ]) == (2.0, 3.0)\n",
    "assert range_overlap([ (0.0, 1.0), (0.0, 2.0), (-1.0, 1.0) ]) == (0.0, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error is actually reassuring: we haven’t written `range_overlap` yet, so if the tests passed, it would be a sign that someone else had and that we were accidentally using their function.\n",
    "\n",
    "And as a bonus of writing these tests, we’ve implicitly defined what our input and output look like: we expect a list of pairs as input, and produce a single pair as output.\n",
    "\n",
    "Something important is missing, though. We don’t have any tests for the case where the ranges don’t overlap at all:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What should `range_overlap` do in this case: fail with an error message, produce a special value like `(0.0, 0.0)` to signal that there’s no overlap, or something else? Any actual implementation of the function will do one of these things; writing the tests first helps us figure out which is best _before_ we’re emotionally invested in whatever we happened to write before we realized there was an issue.\n",
    "\n",
    "And what about this case?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do two segments that touch at their endpoints overlap or not? Mathematicians usually say \"yes\", but engineers usually say \"no\". The best answer is \"whatever is most useful in the rest of our program\", but again, any actual implementation of `range_overlap` is going to do _something_, and whatever it is ought to be consistent with what it does when there’s no overlap at all.\n",
    "\n",
    "Since we’re planning to use the range this function returns as the X axis in a time series chart, we decide that:\n",
    "\n",
    "1. every overlap has to have non-zero width, and\n",
    "2. we will return the special value None when there’s no overlap.\n",
    "\n",
    "`None` is built into Python, and means “nothing here”. (Other languages often call the equivalent value `null` or `nil`). With that decision made, we can finish writing our last two tests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == None\n",
    "assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we get an error because we haven’t written our function, but we’re now ready to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def range_overlap(ranges):\n",
    "    '''Return common overlap among a set of [low, high] ranges.'''\n",
    "    lowest = 0.0\n",
    "    highest = 1.0\n",
    "    for (low, high) in ranges:\n",
    "        lowest = max(lowest, low)\n",
    "        highest = min(highest, high)\n",
    "    return (lowest, highest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Take a moment to think about why we use `max` to raise `lowest` and `min` to lower `highest`). We’d now like to re-run our tests, but they’re scattered across three different cells. To make running them easier, let’s put them all in a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_range_overlap():\n",
    "    assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == None\n",
    "    assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == None\n",
    "    assert range_overlap([ (0.0, 1.0) ]) == (0.0, 1.0)\n",
    "    assert range_overlap([ (2.0, 3.0), (2.0, 4.0) ]) == (2.0, 3.0)\n",
    "    assert range_overlap([ (0.0, 1.0), (0.0, 2.0), (-1.0, 1.0) ]) == (0.0, 1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now test `range_overlap` with a single function call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_range_overlap()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first test that was supposed to produce `None` fails, so we know something is wrong with our function. We _don’t_ know whether the other tests passed or failed because Python halted the program as soon as it spotted the first error. Still, some information is better than none, and if we trace the behavior of the function with that input, we realize that we’re initializing `lowest` and `highest` to 0.0 and 1.0 respectively, regardless of the input values. This violates another important rule of programming: _always initialize from data_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<section class=\"challenge panel panel-success\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Pre- and post-conditions</h2>\n",
    "</div>\n",
    "<div class=\"panel-body\">\n",
    "<p>Suppose you are writing a function called <code>average</code> that calculates the average of the numbers in a list. What pre-conditions and post-conditions would you write for it? Compare your answer to your neighbor’s: can you think of a function that will pass your tests but not hers or vice versa?</p>\n",
    "</div>\n",
    "</section>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<section class=\"challenge panel panel-success\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Testing assertions</h2>\n",
    "</div>\n",
    "<div class=\"panel-body\">\n",
    "<p>Given a sequence of values, the function <code>running</code> returns a list containing the running totals at each index.</p>\n",
    "<pre class=\"sourceCode python\"><code class=\"sourceCode python\">running([<span class=\"dv\">1</span>, <span class=\"dv\">2</span>, <span class=\"dv\">3</span>, <span class=\"dv\">4</span>])</code></pre>\n",
    "<pre class=\"output\"><code>[1, 3, 6, 10]</code></pre>\n",
    "<pre class=\"sourceCode python\"><code class=\"sourceCode python\">running(<span class=\"st\">'abc'</span>)</code></pre>\n",
    "<pre class=\"output\"><code>['a', 'ab', 'abc']</code></pre>\n",
    "<p>Explain in words what the assertions in this function check, and for each one, give an example of input that will make that assertion fail.</p>\n",
    "<pre class=\"sourceCode python\"><code class=\"sourceCode python\"><span class=\"kw\">def</span> running(values):\n",
    "    <span class=\"kw\">assert</span> <span class=\"dt\">len</span>(values) &gt; <span class=\"dv\">0</span>\n",
    "    result = [values[<span class=\"dv\">0</span>]]\n",
    "    <span class=\"kw\">for</span> v in values[<span class=\"dv\">1</span>:]:\n",
    "        <span class=\"kw\">assert</span> result[-<span class=\"dv\">1</span>] &gt;= <span class=\"dv\">0</span>\n",
    "        result.append(result[-<span class=\"dv\">1</span>] + v)\n",
    "        <span class=\"kw\">assert</span> result[-<span class=\"dv\">1</span>] &gt;= result[<span class=\"dv\">0</span>]\n",
    "    <span class=\"kw\">return</span> result</code></pre>\n",
    "</div>\n",
    "</section>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# assert len(values) > 0 tests that you haven't passed an empty sequence in\n",
    "running([])\n",
    "running ('')\n",
    "\n",
    "# assert result[-1] >= 0 checks that the last item in the results list is greater than or equal to 0\n",
    "running([-1, 0, 1])\n",
    "\n",
    "# assert result[-1] >= result[0] checks that the last item in the results list is greater than or equal to the first item\n",
    "running([1, -1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<section class=\"challenge panel panel-success\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-pencil\"></span>Fixing and testing</h2>\n",
    "</div>\n",
    "<div class=\"panel-body\">\n",
    "<p>Fix <code>range_overlap</code>. Re-run <code>test_range_overlap</code> after each change you make.</p>\n",
    "</div>\n",
    "</section>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}