05_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Medical Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook is part of [Bite Size Bayes](https://allendowney.github.io/BiteSizeBayes/), an introduction to probability and Bayesian statistics using Python.\n",
    "\n",
    "Copyright 2020 Allen B. Downey\n",
    "\n",
    "License: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "The following cell downloads `utils.py`, which contains some utility function we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print('Downloaded ' + local)\n",
    "\n",
    "download('https://github.com/AllenDowney/BiteSizeBayes/raw/master/utils.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "If everything we need is installed, the following cell should run with no error messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from utils import values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "\n",
    "[In the previous notebook](https://colab.research.google.com/github/AllenDowney/BiteSizeBayes/blob/master/04_dice.ipynb) I presented Theorem 4, which is a way to compute the probability of a disjunction (`or` operation) using the probability of a conjunction (`and` operation).\n",
    "\n",
    "$P(A ~or~ B) = P(A) + P(B) - P(A ~and~ B)$\n",
    "\n",
    "Then we used it to show that the sum of the unnormalized posteriors is the total probability of the data, which is why the Bayes table works.\n",
    "\n",
    "We saw several examples involving dice, and I used them to show how prediction and inference are related: the Bayes table actually solves the prediction problem on the way to solving the inference problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayesville\n",
    "\n",
    "In this notebook we'll consider a famous example where Bayes's Theorem brings clarity to a confusing topic: medical testing.\n",
    "\n",
    "Joe Blitzstein explains the scenario in this video from Stat110x at HarvardX.\n",
    "\n",
    "(You might have to run the following cell to see the video.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('otdaJPVQIgg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll paraphase the problem posed in the video: \n",
    "\n",
    "> In Bayesville, 1% of the population has an undiagnosed medical condition.  Jimmy gets tested for the condition and the test comes back positive; that is, the test says Jimmy has the condition.  \n",
    ">\n",
    "> The test is 95% accurate, which means\n",
    ">\n",
    "> * If you give the test to someone with the condition, the probability is 95% that the test will be positive, and \n",
    ">\n",
    "> * If you give it to someone who does not have the condition, the probability is 95% that the test will be negative.\n",
    ">\n",
    "> What is the probability that Jimmy actually has the condition?  \n",
    "\n",
    "Because the test is 95% accurate, it is tempting to say that the probability is 95% that the test is correct and Jimmy has the condition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But that is wrong.  Or maybe I should say it's the right answer to a different question.  95% is the probability of a positive test, given a patient with the condition.  But that's not what the question asked, or what Jimmy wants to know.\n",
    "\n",
    "To Jimmy, the important question is the probability he has the condition, given a positive test.  As we have seen, and as Joe explains in the video:\n",
    "\n",
    "$P(A|B) ≠ P(B|A)$\n",
    "\n",
    "We can use a Bayes table to answer Jimmy's question correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes table\n",
    "\n",
    "I'll use two strings to represent the hypotheses: `condition` and `no condition`.\n",
    "\n",
    "The prior for `condition` is the probability a random citizen of Bayesville has the condition, which is 1%.\n",
    "\n",
    "The prior for `no condition` is the probability that a random citizen does not have the disease, which is 99%.\n",
    "\n",
    "Let's put those values into a Bayes table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "table = pd.DataFrame(index=['condition', 'no condition'])\n",
    "table['prior'] = 0.01, 0.99\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is the positive test, so the likelihoods are:\n",
    "\n",
    "* The probability of a correct positive test, given the condition, which is 95%.\n",
    "\n",
    "* The probability of an incorrect positive test, given no condition, which is 5%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "table['likelihood'] = 0.95, 0.05\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have priors and likelihoods, the remaining steps are always the same.  We compute the unnormalized posteriors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "table['unnorm'] = table['prior'] * table['likelihood']\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the total probability of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob_data = table['unnorm'].sum()\n",
    "prob_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then divide through to get the normalized posteriors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "table['posterior'] = table['unnorm'] / prob_data\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The posterior for `condition` is substantially higher than the prior, so the positive test is evidence in favor of `condition`.\n",
    "\n",
    "But the prior is small and the evidence is not strong enough to overcome it; despite the positive test, the probability that Jimmy has the condition is only about 16%.\n",
    "\n",
    "Many people find this result surprising and some insist that the probability is 95% that Jimmy has the condition.\n",
    "\n",
    "The mistake they are making is called the [base rate fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy) because it ignores the \"base rate\" of the condition, which is the prior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Put a function on it\n",
    "\n",
    "At this point you might be sick of seeing the same six lines of code over and over, so let's put them in a function where you will never have to see them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_bayes_table(hypos, prior, likelihood):\n",
    "    \"\"\"Make a Bayes table.\n",
    "    \n",
    "    hypos: sequence of hypotheses\n",
    "    prior: prior probabilities\n",
    "    likelihood: sequence of likelihoods\n",
    "    \n",
    "    returns: DataFrame\n",
    "    \"\"\"\n",
    "    table = pd.DataFrame(index=hypos)\n",
    "    table['prior'] = prior\n",
    "    table['likelihood'] = likelihood\n",
    "    table['unnorm'] = table['prior'] * table['likelihood']\n",
    "    prob_data = table['unnorm'].sum()\n",
    "    table['posterior'] = table['unnorm'] / prob_data\n",
    "    return table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function takes three parameters:\n",
    "\n",
    "* `hypos`, which should be a sequence of hypotheses.  You can use almost any type to represent the hypotheses, including string, `int`, and `float`.\n",
    "\n",
    "* `prior`, which is sequence of prior probabilities, $P(H)$  for each $H$.\n",
    "\n",
    "* `likelihood`, which is a sequence of likelihoods, $P(D|H)$  for each $H$.\n",
    "\n",
    "All three sequences should be the same length.\n",
    "\n",
    "Here's a solution to the previous problem using `make_bayes_table`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypos = ['condition', 'no condition']\n",
    "prior = 0.01, 0.99\n",
    "likelihood = 0.95, 0.05\n",
    "\n",
    "make_bayes_table(hypos, prior, likelihood)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Suppose we take the same test to another town, called Sickville, where the base rate of the disease is 10%, substantially higher than in Bayesville.  If a citizen of Sickville tests positive, what is the probability that they have the condition?\n",
    "\n",
    "Use `make_bayes_table` to compute the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a higher base rate, the posterior probability is substantially higher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Suppose we go back to Bayesville, where the base rate is 1%, with a new test that is 99.5% accurate.\n",
    "\n",
    "If a citizen of Bayesville tests positive with the new test, what is the probability they have the condition?\n",
    "\n",
    "Use `make_bayes_table` to compute the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With an accuracy of 99.5%, the positive test provides stronger evidence, so it is able to overcome the small prior."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Elvis problem\n",
    "\n",
    "Here's a problem from [*Bayesian Data Analysis*](http://www.stat.columbia.edu/~gelman/book/):\n",
    "\n",
    "> Elvis Presley had a twin brother (who died at birth).  What is the probability that Elvis was an identical twin?\n",
    "\n",
    "For background information, I used data from the\n",
    "U.S. Census Bureau, [Birth, Stillbirth, and Infant Mortality Statistics for the Continental United States, the Territory of Hawaii, the Virgin Islands 1935](https://www.cdc.gov/nchs/data/vsushistorical/birthstat_1935.pdf) to estimate that in 1935, about 1/3 of twins were identical.\n",
    "\n",
    "**Exercise:** Use this base rate and a Bayes table to compute the probability that Elvis was an identical twin.\n",
    "\n",
    "Hint: Because identical twins have the same genes, they are almost always the same sex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook, we used Bayes's Theorem, in the form of a Bayes table, to solve an important problem: interpreting the result of a medical test correctly.\n",
    "\n",
    "Many people, including many doctors, get this problem wrong, with bad consequences for patients.  Now that you know about the \"base rate fallacy\", you will see that it appears in many other domains, not just medicine.\n",
    "\n",
    "Finally, I presented the Elvis problem, which I hope is a fun way to apply what you have learned so far.  If you like the Elvis problem, you might enjoy [this notebook](https://colab.research.google.com/github/AllenDowney/BiteSizeBayes/blob/master/elvis.ipynb) where I dig into it a little deeper.\n",
    "\n",
    "[In the next notebook](https://colab.research.google.com/github/AllenDowney/BiteSizeBayes/blob/master/06_pmf.ipynb) I'll introduce the probability mass function (PMF) and we'll use it to solve new versions of the cookie problem and the dice problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}