02_bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayes's Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook is part of [Bite Size Bayes](https://allendowney.github.io/BiteSizeBayes/), an introduction to probability and Bayesian statistics using Python.\n",
    "\n",
    "Copyright 2020 Allen B. Downey\n",
    "\n",
    "License: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "The following cell downloads `utils.py`, which contains some utility function we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print('Downloaded ' + local)\n",
    "\n",
    "download('https://github.com/AllenDowney/BiteSizeBayes/raw/master/utils.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "The following cell downloads the data file we'll use in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "download('https://github.com/AllenDowney/BiteSizeBayes/raw/master/gss_bayes.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "If everything we need is installed, the following cell should run with no error messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from utils import values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "\n",
    "[In the previous notebook](https://colab.research.google.com/github/AllenDowney/BiteSizeBayes/blob/master/01_linda.ipynb) I defined probability, conjunction, and conditional probability, and used data from the General Social Survey (GSS) to compute the probability of various logical propositions.\n",
    "\n",
    "To review, here's how we loaded the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "gss = pd.read_csv('gss_bayes.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are the logical propositions we defined, represented using Boolean series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "banker = (gss['indus10'] == 6870)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "female = (gss['sex'] == 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "liberal = (gss['polviews'] < 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "democrat = (gss['partyid'] <= 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I defined the following function, which uses `mean` to compute the fraction of `True` values in a Boolean series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prob(A):\n",
    "    \"\"\"Computes the probability of a proposition, A.\n",
    "    \n",
    "    A: Boolean series\n",
    "    \n",
    "    returns: probability\n",
    "    \"\"\"\n",
    "    assert isinstance(A, pd.Series)\n",
    "    assert A.dtype == 'bool'\n",
    "    \n",
    "    return A.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can compute the probability of a proposition like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(female)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we used the `&` operator to compute the probability of a conjunction, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(female & banker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I defined the following function, which uses the bracket operator to compute conditional probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conditional(A, B):\n",
    "    \"\"\"Conditional probability of A given B.\n",
    "    \n",
    "    A: Boolean series\n",
    "    B: Boolean series\n",
    "    \n",
    "    returns: probability\n",
    "    \"\"\"\n",
    "    return prob(A[B])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We showed that conjunction is commutative, so `prob(A & B)` equals `prob(B & A)`, for any logical propositions `A` and `B`.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(liberal & democrat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(democrat & liberal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But conditional probability is NOT commutative, so `conditional(A, B)` is generally not the same as `conditional(B, A)`.\n",
    "\n",
    "For example, here's the probability that a respondent is female, given that they are a banker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional(female, banker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the probability that a respondent is a banker, given that they are female."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional(banker, female)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not even close."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More propositions\n",
    "\n",
    "For the sake of variety in our examples, let's define some new propositions.\n",
    "\n",
    "Here's the probability that a random respondent is male."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "male = (gss['sex']==1)\n",
    "prob(male)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The industry code for \"Construction\" is `770`.  Let's call someone in this field a \"builder\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = (gss['indus10'] == 770)\n",
    "prob(builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's define propositions for conservatives and Republicans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "conservative = (gss['polviews'] > 4)\n",
    "prob(conservative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "republican = (gss['partyid'].isin([5,6]))\n",
    "prob(republican)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `isin` function checks whether values are in a given sequence.  In this example, the values `5` and `6` represent the responses \"Strong Republican\" and \"Not Strong Republican\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, I'll use `age` to define propositions for `young` and `old`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "young = (gss['age'] < 30)\n",
    "prob(young)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "old = (gss['age'] >= 65)\n",
    "prob(old)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these thresholds, I chose round numbers near the 20th and 80th percentiles.  Depending on your age, you may or may not agree with these definitions of \"young\" and \"old\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** There's a [famous quote](https://quoteinvestigator.com/2014/02/24/heart-head/) about young people, old people, liberals, and conservatives that goes something like:\n",
    "\n",
    "> If you are not a liberal at 25, you have no heart. If you are not a conservative at 35, you have no brain.\n",
    "\n",
    "Whether you agree with this proposition or not, it suggests some probabilities we can compute as a review exercise.  \n",
    "Use `prob` and `conditional` to compute these probabilities.\n",
    "\n",
    "* What is the probability that a randomly chosen respondent is a young liberal?\n",
    "\n",
    "* What is the probability that a young person is liberal?\n",
    "\n",
    "* What fraction of respondents are old conservatives?\n",
    "\n",
    "* What fraction of conservatives are old?\n",
    "\n",
    "For each statement, think about whether it is expressing a conjunction, or a conditional probability, or both.\n",
    "\n",
    "And for the conditional probabilities, be careful about the order!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your last answer is greater than 30%, you have it backwards!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Onward!\n",
    "\n",
    "In this notebook, we'll derive three relationships between conjunction and conditional probability:\n",
    "\n",
    "* Theorem 1: Using conjunction to compute a conditional probability.\n",
    "\n",
    "* Theorem 2: Using a conditional probability to compute a conjunction.\n",
    "\n",
    "* Theorem 3: Using `conditional(A, B)` to compute `conditional(B, A)`.\n",
    "\n",
    "Theorem 3 is also known as Bayes's Theorem, which is the foundation of Bayesian statistics.\n",
    "\n",
    "For parts of this notebook it will be useful to use mathematical notation for probability, so I'll introduce that now.\n",
    "\n",
    "* $P(A)$ is the probability of proposition $A$.\n",
    "\n",
    "* $P(A~\\mathrm{and}~B)$ is the probability of the conjunction of $A$ and $B$, that is, the probability that both are true.\n",
    "\n",
    "* $P(A | B)$ is the conditional probability of $A$ given that $B$ is true.  The vertical line between $A$ and $B$ is pronounced \"given\". \n",
    "\n",
    "With that, we are ready for Theorem 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theorem 1\n",
    "\n",
    "What fraction of builders are male?  We have already seen one way to compute the answer:\n",
    "\n",
    "1. Use the bracket operator to select the builders, then\n",
    "\n",
    "2. Use `mean` to compute the fraction of builders who are male.\n",
    "\n",
    "We can write these steps like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "male[builder].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use the `conditional` function, which does the same thing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional(male, builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But there is another way: to compute the fraction of builders who are male, we can compute the ratio of two probabilities:\n",
    "\n",
    "1. The fraction of respondents who are male builders, and\n",
    "\n",
    "2. The fraction of respondents who are builders.\n",
    "\n",
    "Here's what that looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(male & builder) / prob(builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is the same.\n",
    "\n",
    "This example demonstrates a general rule that relates conditional probability and conjunction.  Here's what it looks like in math notation:\n",
    "\n",
    "$P(A|B) = \\frac{P(A~\\mathrm{and}~B)}{P(B)}$\n",
    "\n",
    "And that's Theorem 1.\n",
    "\n",
    "In this example:\n",
    "\n",
    "`conditional(male, builder) = prob(male & builder) / prob(builder)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:**  What fraction of conservatives are Republican?  Compute the answer two ways:\n",
    "\n",
    "* Use `conditional` (which uses the bracket operator), and\n",
    "\n",
    "* Use Theorem 1.\n",
    "\n",
    "Confirm that you get the same answer.\n",
    "\n",
    "Note: Due to floating-point arithmetic, the results might not be exactly the same, but almost all of the digits should be the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proof?\n",
    "\n",
    "I didn't really prove Theorem 1; mostly, it is a statement of what conditional probability means.\n",
    "\n",
    "For example, consider this Venn diagram:\n",
    "\n",
    "<img width=\"200\" src=\"https://github.com/AllenDowney/BiteSizeBayes/raw/master/theorem1_venn_diagram.png\">\n",
    "\n",
    "\n",
    "The blue circle represents male respondents.  The red circle represents builders.  The intersection represents male builders.\n",
    "\n",
    "To compute the fraction of builders who are male, we can compute the ratio of the intersection, which is `prob(male & builder)` to the red circle, which is `prob(builder)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** For practice, compute fraction of bankers who are old both ways: using `conditional` and using Theorem 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theorem 2\n",
    "\n",
    "Here's Theorem 1 again:\n",
    "\n",
    "$P(A|B) = \\frac{P(A~\\mathrm{and}~B)}{P(B)}$ \n",
    "\n",
    "If we multiply both sides by $P(B)$, we get Theorem 2.\n",
    "\n",
    "$P(A~\\mathrm{and}~B) = P(B) P(A|B)$\n",
    "\n",
    "This formula suggests a second way to compute a conjunction: instead of using the `&` operator, we can compute the product of two probabilities.\n",
    "\n",
    "Let's see if it works for `conservative` and `republican`.  Here's the result using `&`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(conservative & republican)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the result using Theorem 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(republican) * conditional(conservative, republican)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because of floating-point errors, they might not be identical, but almost all of the digits are the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Check Theorem 2 one more time by computing the fraction of respondents who are old liberals both ways:\n",
    "\n",
    "* Using the `&` operator, and\n",
    "\n",
    "* Using Theorem 2.\n",
    "\n",
    "The results should be the same, or at least very close."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conjunction is commutative\n",
    "\n",
    "We have already established that conjunction is commutative.  In math notation, that means:\n",
    "\n",
    "$P(A~\\mathrm{and}~B) = P(B~\\mathrm{and}~A)$\n",
    "\n",
    "If we apply Theorem 2 to both sides, we have\n",
    "\n",
    "$P(B) P(A|B) = P(A) P(B|A)$\n",
    "\n",
    "Here's one way to interpret that: if you want to check $A$ and $B$, you can do it in either order:\n",
    "\n",
    "1. You can check $B$ first, then $A$ conditioned on $B$, or\n",
    "\n",
    "2. You can check $A$ first, then $B$ conditioned on $A$.\n",
    "\n",
    "To try it out, I'll compute the fraction of young builders both ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(young) * conditional(builder, young)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(builder) * conditional(young, builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same thing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Compute the probability of being a male banker both ways and see if you get the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Solution goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theorem 3\n",
    "\n",
    "In the previous section we established that \n",
    "\n",
    "$P(B) P(A|B) = P(A) P(B|A)$\n",
    "\n",
    "If we divide through by $P(B)$, we get Theorem 3:\n",
    "\n",
    "$P(A|B) = \\frac{P(A) P(B|A)}{P(B)}$\n",
    "\n",
    "And that, my friends, is Bayes's Theorem.\n",
    "\n",
    "To see how it works, let's try one more combination of our propositions.  Let's compute the fraction of builders who are liberal, first using `conditional`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional(liberal, builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using Bayes's Theorem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(liberal) * conditional(builder, liberal) / prob(builder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same thing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Try it yourself!  Compute the fraction of young people who are Republican both ways: using `conditional` and using Bayes's Theorem.  See if you get the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "conditional(republican, young)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob(republican) * conditional(young, republican) / prob(young)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "Here's what we have so far:\n",
    "\n",
    "**Theorem 1** gives us a new way to compute a conditional probability using a conjunction:\n",
    "\n",
    "$P(A|B) = \\frac{P(A~\\mathrm{and}~B)}{P(B)}$ \n",
    "\n",
    "**Theorem 2** gives us a new way to compute a conjunction using a conditional probability:\n",
    "\n",
    "$P(A~\\mathrm{and}~B) = P(B) P(A|B)$\n",
    "\n",
    "**Theorem 3**, also known as Bayes's Theorem, gives us a way to get from $P(A|B)$ to $P(B|A)$, or the other way around:\n",
    "\n",
    "$P(A|B) = \\frac{P(A) P(B|A)}{P(B)}$\n",
    "\n",
    "But at this point you might ask, \"So what?\"  If we have all of the data, we can compute any probability we want, any conjunction, or any conditional probability, just by counting.  Why do we need these formulas?\n",
    "\n",
    "And you are right, *if* we have all of the data.  But often we don't, and in that case, these formulas can be pretty useful -- especially Bayes's Theorem.\n",
    "\n",
    "In the next notebook, we'll see how."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}