PyTest101.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:b33cc10e21c59cd2a7c0416c042b8228fa4a125bb288fa0dc8ccd12c92b5b4fe"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Code Design and Testing - Unit Testing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Learning Goals\n",
      "- Create Python functions that use `assert` statements to test our Python code.\n",
      "- Use the [pytest] tool to discover, run and report on the results of unit test code.\n",
      "- Explain the concepts of test code coverage and baseline testing.\n",
      "\n",
      "[pytest]: http://pytest.org/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Some of the material in this notebook is courtesy of Katy Huff, Rachel Slaybaugh, and Anthony Scopatz."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Unit Testing Concepts\n",
      "\n",
      "As practicing scientists, \n",
      "we would never trust a lab measurement that we made with uncalibrated instruments. \n",
      "Similarly, as computational scientists, \n",
      "we shouldn't trust the results that our code gives us until we have tested it. \n",
      "Without calibration/testing, \n",
      "how do we know that our code is giving us the right answers?\n",
      "\n",
      "In this lesson, \n",
      "we'll focus on unit tests, \n",
      "perhaps the most basic type of testing that we can run. \n",
      "Unit tests focus on a single \"unit\" of code, \n",
      "which in our case will be functions that we've written. \n",
      "We'll write tests to ensure that when our function is given a certain set of arguments as input, \n",
      "it generates output that we know to be correct. \n",
      "Once we have a complete test suite for a function, \n",
      "we can run the entire suite to make sure that all the tests pass \n",
      "(i.e. that our function gives the correct output for all the combinations of input \n",
      "that we have decided to test).\n",
      "\n",
      "For example, \n",
      "let's say that we have a function that reads in a data file, \n",
      "does some processing, \n",
      "and returns a result. \n",
      "We can test the function by giving it a small data file, \n",
      "for which we can calculate the correct result by hand, \n",
      "and making sure that the function gives the correct answer for this small file. \n",
      "This gives us more confidence that if we run the function on a different data set, \n",
      "perhaps a huge one for which we can't verify the results by hand, \n",
      "that we'll get an accurate result.\n",
      "\n",
      "Even better, \n",
      "if we make changes to the internals of our function, \n",
      "we can run our tests again to make sure that we haven't accidentally broken anything \n",
      "(this is known as a \"regression\"). \n",
      "This makes us more free to continue to improve the performance of our code over time, \n",
      "and avoids the dreaded \"it's working, don't touch it\" phenomena."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##A Unit Testing Example\n",
      "\n",
      "We're going to use a simple function that calculates the mean of a\n",
      "list of numbers to introduce unit testing.\n",
      "Of course,\n",
      "we would never actually write a function like this because we now know that\n",
      "NumPy and SciPy provide descriptive statistics functions that are far faster,\n",
      "more versatile,\n",
      "and more robust than anything we're likely to write."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def mean(numbers):\n",
      "    \"\"\"Return the mean of numbers.\n",
      "    \"\"\"\n",
      "    return sum(numbers) / len(numbers)\n",
      "\n",
      "result = mean([1, 2, 3])\n",
      "assert result == 2"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We've used this pattern before,\n",
      "haven't we?\n",
      "\n",
      "We're developing a function,\n",
      "and we have a little bit of code right below in it the cell that test the function.\n",
      "When we used this pattern before we had to look at the output from that bit of test code \n",
      "to see if thingswere working,\n",
      "but now we're using an `assert` statement to let the computer do the checking.\n",
      "\n",
      "This is such a useful pattern that programmers actually put their tests in functions:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def mean(numbers):\n",
      "    \"\"\"Return the mean of numbers.\n",
      "    \"\"\"\n",
      "    return sum(numbers) / len(numbers)\n",
      "\n",
      "def test_mean():\n",
      "    \"\"\"Return the expected mean value\n",
      "    \"\"\"\n",
      "    result = mean([1, 2, 3])\n",
      "    assert result == 2\n",
      "    \n",
      "test_mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There's 2 problems with the cell above:\n",
      "\n",
      "1. If we add more tests,\n",
      "we have to remember to add a line at the bottom to call the new test functions.\n",
      "2. We don't get any feedback when the test passes.\n",
      "How do we know it even ran?\n",
      "\n",
      "Fortunately,\n",
      "there are tools to help with both of those issues.\n",
      "\n",
      "We're going to leave IPython Notebook now and work in an editor and the shell.\n",
      "Use the `%%writefile mean.py` magic at the top of the cell above to export the\n",
      "code to a file.\n",
      "\n",
      "If we use `python` to run our `mean.py` file we see the same \"I guess it worked...\"\n",
      "result."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Exercise:**\n",
      "Change the test input in your editor from `[1, 2, 3]` to `[1, 2, 3, 4]` \n",
      "and the expected result from `2` to `2.5`.\n",
      "While you're there,\n",
      "delete the call to `test_mean()` from the end of the file.\n",
      "Save the file and run it again with `python` at the command line."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Introducing `py.test`\n",
      "\n",
      "Now run your `mean.py` file again but use the command:\n",
      "\n",
      "```bash\n",
      "$ py.test mean.py\n",
      "    \n",
      "================================= test session starts ==================================\n",
      "platform darwin -- Python 2.7.5 -- pytest-2.3.5\n",
      "collected 1 items\n",
      "\n",
      "mean.py F\n",
      "\n",
      "======================================= FAILURES =======================================\n",
      "______________________________________ test_mean _______________________________________\n",
      "\n",
      "    def test_mean():\n",
      "        \"\"\"Return the expected mean value\n",
      "        \"\"\"\n",
      "        result = mean([1, 2, 3, 4])\n",
      ">       assert result == 2.5\n",
      "E       assert 2 == 2.5\n",
      "\n",
      "mean.py:11: AssertionError\n",
      "=============================== 1 failed in 0.04 seconds ===============================\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Look at all that delicious output!\n",
      "\n",
      "`py.test` tells us:\n",
      "\n",
      "- How many tests it found\n",
      "(we'll come back to how it found them in a moment)\n",
      "- That the test in the `mean.py` failed\n",
      "- How the test failed\n",
      "- What line the test failed on\n",
      "- How many tests in total failed\n",
      "- How long the tests took"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The test failed because we forgot about how Python handles integers,\n",
      "floats,\n",
      "and division,\n",
      "didn't we?\n",
      "\n",
      "So,\n",
      "fix the code in `mean()` and run the file again with `py.test`:\n",
      "\n",
      "```bash\n",
      "================================= test session starts ==================================\n",
      "platform darwin -- Python 2.7.5 -- pytest-2.3.5\n",
      "collected 1 items\n",
      "\n",
      "mean.py .\n",
      "\n",
      "=============================== 1 passed in 0.02 seconds ===============================\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This time we got feedback that our test ran,\n",
      "and that it passed\n",
      "(that's what the dot beside `mean.py` means)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Testing `climate_data.py`\n",
      "\n",
      "Now, let's write some tests for the functions in our `climate_data.py` module.\n",
      "\n",
      "By convention,\n",
      "unit test functions have names that start with `test_`\n",
      "and they are kept in separate modules from the code they test.\n",
      "Those modules have file names that start with `test_`.\n",
      "So,\n",
      "let's create a `test_climate_data.py` file and put the following code in it:\n",
      "\n",
      "```python\n",
      "\"\"\"Unit tests for the climate_data module.\n",
      "\"\"\"\n",
      "from datetime import date\n",
      "import climate_data\n",
      "\n",
      "def test_calc_params_hourly():\n",
      "    \"\"\"calc_params sets timeframe correctly for hourly data frequency\n",
      "    \"\"\"\n",
      "    start_date = date(2013, 9, 1)\n",
      "    params = climate_data.calc_params(start_date, 'hourly')\n",
      "    assert params['timeframe'] == 1\n",
      "\n",
      "def test_calc_params_daily():\n",
      "    \"\"\"calc_params sets timeframe correctly for daily data frequency\n",
      "    \"\"\"\n",
      "    start_date = date(2013, 9, 1)\n",
      "    params = climate_data.calc_params(start_date, 'daily')\n",
      "    assert params['timeframe'] == 2\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`py.test`\n",
      "(and other Python unit testing tools)\n",
      "are set up to treat functions whose names start with `test_` as test functions that they should run.\n",
      "They also recognize that modules whose file names start with `test_` are places that they should look\n",
      "to find unit test functions.\n",
      "That mean that we don't even have to tell `py.test` what module to run.\n",
      "It just searches through all of our Python code,\n",
      "find the test functions,\n",
      "runs them,\n",
      "and reports the results:\n",
      "\n",
      "```bash\n",
      "$ py.test\n",
      "\n",
      "================================= test session starts ==================================\n",
      "platform darwin -- Python 2.7.5 -- pytest-2.3.5\n",
      "collected 2 items\n",
      "\n",
      "test_climate_data.py ..\n",
      "\n",
      "=============================== 2 passed in 0.31 seconds ===============================\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Exercise:**\n",
      "Add a few more unit tests for 1 or more of the functions in `climate_data.py`\n",
      "and use `py.test` to run your growing test suite."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr>\n",
      "##Test Driven Development\n",
      "\n",
      "You may be noticing that our units tests are statements about the design intent\n",
      "of our functions.\n",
      "\n",
      "Test Driven Development (TDD) is a methodology wherein you write a collection\n",
      "of unit tests that describe the design intent \n",
      "(all of the outputs of a function for the various combinations of inputs)\n",
      "*before* you write any of the code for the actual function.\n",
      "Then you go into a cycle of writing code and running the test suite,\n",
      "working toward the goal of having all of the tests pass,\n",
      "which means that your implementation is complete\n",
      "(assuming,\n",
      "of course,\n",
      "that your test suite really did described everything that you function\n",
      "needs to do).\n",
      "\n",
      "TDD may not be a practical methodology for a working scientist,\n",
      "but a variation on it is very useful when you find a bug in existing code.\n",
      "That is,\n",
      "start the process of fixing the bug by writing 1 or more tests that demonstrate\n",
      "the bug.\n",
      "Then run the test suite to confirm that the bug causes those tests to fail.\n",
      "As you write code to fix the bug run the test suite to measure your success\n",
      "and to confirm that your bug fix doesn't break some other aspect of the code\n",
      "as a side-effect.\n",
      "<hr>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Testing `assert` Statements\n",
      "\n",
      "Remember how we had to run `climate_data.py` from the command line 4 times\n",
      "with different combinations of incorrect arguments to test all of the `assert`\n",
      "statement error handling in the `parse_args()` function?\n",
      "That was drudgery that should be automated!\n",
      "\n",
      "`py.test` provides a way of testing that our code raises an `AssertionError` exception\n",
      "when we expect it to:\n",
      "\n",
      "```python\n",
      "import pytest\n",
      "\n",
      "def test_parse_args_wrong_number_of_args():\n",
      "    \"\"\"parse_args raises AssertionError for wrong number of args\n",
      "    \"\"\"\n",
      "    with pytest.raises(AssertionError):\n",
      "        climate_data.parse_args(['foo'])\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Exercise:**\n",
      "Write 3 more tests that exercise the other error trapping assertions in `parse_args()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Notice how the fact that we implemented our code as a collection of functions\n",
      "that do specific parts of the overall objective of the code allows us to test\n",
      "each aspect of the code in isolation with a minimum of setup for each test.\n",
      "\n",
      "Not only does breaking our code into small,\n",
      "limited scope functions make it easier to understand\n",
      "(remember \"Seven Plus or Minus Two\"),\n",
      "it make it easier to test."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Test Assertions and Floats\n",
      "\n",
      "Sooner or later you will probably have to deal with the precision issues\n",
      "of floating point numbers in a test function.\n",
      "\n",
      "In Python itself the built-in `round()` function can be used:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import math\n",
      "\n",
      "result = math.cos(math.pi/2)\n",
      "assert round(result, 5) == 0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "while NumPy provides a collection of special assertion functions\n",
      "in the `numpy.testing` namespace that operate on `numpy.ndarray` objects,\n",
      "such as `numpy.testing.assert_allclose()`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "\n",
      "a = np.array([i * np.pi/2 for i in (1, 3, 5, 7)])\n",
      "result = np.cos(a)\n",
      "np.testing.assert_allclose(result, np.zeros_like(a), atol=1e-5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Measuring Test Suite Coverage\n",
      "\n",
      "[coverage.py] is a Python tool that monitors the execution of a test suite\n",
      "and reports metrics about the extent to which the tests touch the code.\n",
      "It is particularly useful if you are adding tests to an existing codebase that has\n",
      "some (or no) unit tests.\n",
      "Test coverage reports help you decide where to focus your test writing efforts.\n",
      "\n",
      "[coverage.py]: http://nedbatchelder.com/code/coverage/\n",
      "\n",
      "[coverage.py] is not included in the Anaconda distribution at this time\n",
      "so we have to install it from the [Python Package Index][pypi] (PyPI)\n",
      "using the `pip` tool:\n",
      "\n",
      "```bash\n",
      "$ pip install coverage\n",
      "```\n",
      "\n",
      "[pypi]: https://pypi.python.org/\n",
      "\n",
      "With `coverage` installed we can use it to generate data about our test suite\n",
      "with the command:\n",
      "\n",
      "```bash\n",
      "$ coverage run --source=climate_data.py --branch -m py.test\n",
      "```\n",
      "The data that `coverage` generates is stored in a hidden file named `.coverage`.\n",
      "\n",
      "The command:\n",
      "\n",
      "```bash\n",
      "$ coverage report -m\n",
      "```\n",
      "\n",
      "produces a coverage report in the terminal window \n",
      "that includes a list of the line ranges that the test suite does not touch:\n",
      "\n",
      "    Name           Stmts   Miss Branch BrMiss  Cover   Missing\n",
      "    ----------------------------------------------------------\n",
      "    climate_data      71     50     20     17    26%   21-28, 35-50, 73-77, 83-99, 105-110, 117-119, 123\n",
      "\n",
      "For additional insight we can use:\n",
      "\n",
      "```bash\n",
      "$ coverage html -d coverage_report\n",
      "```\n",
      "\n",
      "to generate an HTML and Javascript coverage report that is viewable in a web browser\n",
      "and includes highlighted source code to indicate its test coverage status:\n",
      "![HTML coverage report](files/climate_data_coverage.png)\n",
      "![Coverage details](files/climate_data_coverage_details.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Baseline Testing\n",
      "\n",
      "What if you can't create unit tests for a piece of code for some reason?\n",
      "You can (and should) come up with a means of testing the results of the\n",
      "code so that you can gauge the effects of changes that you make to the code.\n",
      "One technique is to store the results of a run as a baseline\n",
      "(other names: reference results, refs, gold standard, golden results)\n",
      "and routinely compare results from subsequent runs as development proceeds.\n",
      "If the goal of your development is to improve the code\n",
      "(reorganize for readability, apply DRY, add unit tests, etc.),\n",
      "then you would expect to see no differences between development run results and\n",
      "the baseline results.\n",
      "If you are adding a feature,\n",
      "the differences between development run results and baseline results can be used\n",
      "to gauge whether or not the implementation of the feature is producing the intended\n",
      "change in the results.\n",
      "\n",
      "In any case,\n",
      "the comparison of development run results to the baseline results should be automated\n",
      "so that it is easy to do often and it is repeatable.\n",
      "Shell scripts that use `diff` or other tools that calculate the differences between\n",
      "results files\n",
      "(or custom tools that you write)\n",
      "are one way to do that.\n",
      "\n",
      "```bash\n",
      "#!/bin/bash\n",
      "CHECK_DIFFS=../../SOG-code/tests/tools/check_diff.py\n",
      "REFS=../SOG-dev-refs-2013-03-01-r1682\n",
      "OUTFILES=\"\n",
      "infile.short.out\n",
      "total_check\n",
      "salinity_check\n",
      "timeseries/std_bio_SOG-short.out\n",
      "timeseries/std_phys_SOG-short.out\n",
      "timeseries/user_bio_SOG-short.out\n",
      "timeseries/user_phys_SOG-short.out\n",
      "timeseries/std_chem_SOG-short.out\n",
      "profiles/halo-SOG-short.out\n",
      "profiles/hoff-SOG-short.dat\n",
      "profiles/SOG-short-2004-10-29T120700\n",
      "\"\n",
      "\n",
      "for OUTFILE in $OUTFILES\n",
      "do\n",
      "    python $CHECK_DIFFS $OUTFILE $REFS/$OUTFILE\n",
      "done\n",
      "```\n",
      "\n",
      "Scripts that generate graphs in which the development run results and the baseline results\n",
      "are overlaid for visual comparison are another possibility.\n",
      "![Development vs. baseline results for SOG](files/results_vs_refs.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Key Points\n",
      "- Testing code is analogous to calibrating instruments\n",
      "- Use `assert` statements to let the computer compare the actual result of a function\n",
      "to the expected result\n",
      "- Write test code as functions\n",
      "- [pytest] is a Python tool that finds, runs, and reports the results of Python unit test code\n",
      "- Test Driven Developement (TDD) is particularly useful when fixing bugs\n",
      "- Use the built-in `round()` function or `numpy.testing.assert_allclose()` to deal with\n",
      "floating point precision issues in tests\n",
      "- [coverage] is a Python tool that measures and reports on how well a test suite exercises the\n",
      "code it tests\n",
      "- Baseline testing is a technique for monitoring changes in a program's results during development\n",
      "\n",
      "[pytest]: http://pytest.org/\n",
      "[coverage]: http://nedbatchelder.com/code/coverage/"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}