Statistics - Hypothesis Testing - With solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fe2c7128-04a5-4a57-bba9-73a7d3aaec9a",
   "metadata": {},
   "source": [
    "# Statistics - Hypothsis testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cec0f0a1-f8da-4886-92ee-15fa1ecc28d6",
   "metadata": {},
   "source": [
    "## The problem "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60467c81-9619-42d8-a22a-53301d34f0bc",
   "metadata": {},
   "source": [
    "You are a data scientist in [booking.com](https://www.booking.com/). You are in charge on developing models to increase the conversion rates of users. The competition is fierce, but your company is the market leader and you are proud to be a part of the essential data science team which is involved in the core of the buisness.  One day, Your CEO is informing you that a new, start-up company, achieves siginifacntly higher conversion rates, and that the company is contemplating buying it. He ask you to plan an accurate experiment to evalute the lift in conversion rate if the start-up would be bought and integrated in the company.  \n",
    "\n",
    "\n",
    "Design such plan. Consider:\n",
    "* What inputs you need or missing\n",
    "* What is the correct methodoligy to apply to fair test the two systems? \n",
    "* What are the needed confidance parameters? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c6f39ae-a8cd-4ba5-af2a-bfcda71d2553",
   "metadata": {},
   "source": [
    "In the notebook about  <a href=\"../Lesson 2/Statistics - Confidence Intervals.ipynb\"> condifdence intervals </a> we've reached the following condfidance interval of the difference between two independent samples (denoted as sample $x$ and sample $y$):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6bdc507-cd57-4f77-beea-caf3001cdbc9",
   "metadata": {},
   "source": [
    "$$\\large CI = (\\bar{x}-\\bar{y}) \\pm{1.96 * \\sqrt{\\dfrac{\\bar{s_{x}^{2}}}{n} + \\dfrac{\\bar{s_{y}^{2}}}{n}}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7101072-b995-4ed0-9247-c2a60067ed8a",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: Is the above CI forumla enough to design proper AB test as described above? Try it out first.</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65215adc-93df-4fca-aaf6-d397d697e6ef",
   "metadata": {},
   "source": [
    "## Hypotheses testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1320ced7-5f68-4c00-b40b-8fae9ce00010",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: In the probelm above, what is the status quo and what's the innovation? On which side are we? </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f8c94d-55f5-4384-9b99-3a979d699b36",
   "metadata": {},
   "source": [
    "### Single known mean Hypotheses test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f275e9e-f2c2-4fd8-824d-f942f37c4035",
   "metadata": {},
   "source": [
    "* Two sided test, comparing against a known single mean $\\mu_{0}$:\n",
    "\n",
    "$$ \\large H_{0}: \\mu = \\mu_0$$\n",
    "<br>\n",
    "$$ \\large H_{1}: \\mu \\neq{\\mu_0}$$\n",
    "\n",
    "* One sided (left), comparing against a known single mean $\\mu_{0}$:\n",
    "$$ \\large H_{0}: \\mu > \\mu_0$$\n",
    "<br>\n",
    "$$ \\large H_{1}: \\mu \\leq{\\mu_0}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bf214ed-2272-4d81-9691-9c66be504e65",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: How to we know if a test is left or right sided? </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cddfaf6f-bf02-4cb4-b5bb-12c2780dbe69",
   "metadata": {},
   "source": [
    "In both these cases, the statistics (Z or T, depends on the number of samples) we want to calculate is:\n",
    "\n",
    "$$ Z = \\frac{\\bar{x} - \\mu_0}{SE}$$\n",
    "\n",
    "where SE is the standard error, $\\frac{s}{\\sqrt{n}}$. The p-value will be calculated according to this z-score. The p-value for the one-sided test is half of that of the two sided test. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5351443a-ea2a-40ba-a88e-ac28707f467e",
   "metadata": {},
   "source": [
    "### Terminoligy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3f55432-f64d-4dce-a16f-1fb7e73de3d8",
   "metadata": {},
   "source": [
    "Usually, we are testing two hypotheses, one against the other. The null hypothesis vs. the alternative hypothesis:\n",
    "\n",
    "* The null hypothesis - Usually denoted $H_0$ - The status quo, the hypothsis which state the current state of affairs.\n",
    "* The alternative hypothesis - Usually denoted $H_a$ or $H_1$ - Represent the change or innovation we are considering introducing. \n",
    "* p-value - The probability of getting a result that is as extreme or more extreme when the null hypothesis is true\n",
    "\n",
    "\n",
    "Errors:\n",
    "\n",
    "* Type 1 error - Rejecting the null hypothesis while in fact, it is true - Also called a false postive error:\n",
    "  * $\\alpha$ - The probabliity that will will falsly reject the null hypothesis - i.e. make a type 1 error - Typical values: 0.01, 0.05, 0.1\n",
    "  * Significance level - $1-\\alpha$ - The probablity of **not** making a type 1 error\n",
    "  \n",
    "  \n",
    "* Type 2 error - Rejecting the alternative hypothesis while in fact, it is true - Also called a False negetive:\n",
    "  * $\\beta$ - The probabliity of falsy accept the alternative hypothesis - i.e. make a type 2 error - Typical values: 0.2, 0.05\n",
    "  * Power - $1 - \\beta$ - The probablity of correctly accepting the alternative hypothesis - i.e. the probability of correctly detecting an effect difference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99614340-1807-46d0-b678-e58e2de953aa",
   "metadata": {},
   "source": [
    "**Usually, as the innovators, our personal goal would be to reject the null hypothesis and accept the alternative hypothesis - So we could adpot our innovative solution as the new standard**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1972790e-b6bb-4178-9c7f-ba0e1e3698ec",
   "metadata": {},
   "source": [
    "![alt text](https://cdn.scribbr.com/wp-content/uploads/2021/01/type-i-and-ii-error-2-768x488.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a485d087-fd47-46d6-b7b5-da6f656454a3",
   "metadata": {},
   "source": [
    "### Visualization: The null hypothesis reject area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b72a6276-7cd1-4daa-9bb2-584615ce9262",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys; sys.path.append('../')\n",
    "from src.statistics import calculate_normal_density_distribution\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "30df2052-8459-4ce1-9f00-add2a50541c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interact\n",
    "import plotly_express as px"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e2427cf-c0e6-4b97-bad3-b8ae2f71cf20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9919bda4207456f9d5a800ca44f4e32",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='alpha', options=(0.2, 0.1, 0.05, 0.01), value=0.2), Dropdown(descr…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import scipy.stats as st\n",
    "test_types = ['Two sided', 'One sided - left', 'One sided - right']\n",
    "def infer_area(x, critical, test_type):\n",
    "    return 'Rejected Area' if (test_type =='Two sided' and abs(x) > critical or\n",
    "                               test_type == 'One sided - right' and x > critical or \n",
    "                               test_type == 'One sided - left' and x < -1*critical) else 'Acceptance area'\n",
    "\n",
    "@interact\n",
    "def illustrate_rejection_and_acceptance_regions(alpha=[0.2, 0.1, 0.05, 0.01], test_type=test_types):\n",
    "    data = calculate_normal_density_distribution(0,1, 1000)\n",
    "    critical = st.norm.ppf(1- (alpha if test_type != 'Two sided' else alpha/2))\n",
    "    print(f'\\nThe critical value for a {1-alpha} {test_type} Test is {critical:.4}')\n",
    "    data['area'] = data.x.apply(lambda x: infer_area(x, critical, test_type))\n",
    "    y_highet = data[data.x.apply(abs) < critical].y.min()\n",
    "    fig = px.area(data.sort_values(by='area'), x='x',y='y', color='area', color_discrete_sequence=[\"blue\", \"red\"])\n",
    "    return fig.update_layout(title_text=f\"Acceptance (Blue) and Rejection (Red) areas for a {test_type} Hypotheses test (Confidance: {1-alpha})\", title_x=0.5)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763d1890-d640-4c30-90ec-438a5978db9f",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: According to wikipedia, the average height of the isreali man is 176.5. We've sampled 50 man with age over 25 in the file isreali_man_100_samples. Conduct an hypothesis test to validate or reject wikipedia's claims in confidance levels of 95% and 99% </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95a67528-4221-46f4-a669-f8d6ac4ea886",
   "metadata": {},
   "source": [
    "We are coducting a two sided hypothesis test with signle known mean:\n",
    "\n",
    "$$ \\large H_{0}: \\mu = 176.5$$\n",
    "<br>\n",
    "$$ \\large H_{0}: \\mu \\neq{176.5}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ce4b6092-c068-45bd-a922-e42aacec5f4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>185.142460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>184.390968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>165.045359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>173.479874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>172.906312</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            h\n",
       "0  185.142460\n",
       "1  184.390968\n",
       "2  165.045359\n",
       "3  173.479874\n",
       "4  172.906312"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "height_df_1 = pd.read_csv('data/isreali_man_50_samples.csv')\n",
    "height_df_1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "50ee3d0e-01cb-4c7b-b005-f4c8d464e695",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178.65902383067183"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "height_df_1.h.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "310ad0be-9b69-48a4-9e2b-37dbc55dd1ae",
   "metadata": {},
   "source": [
    "By CLT, we know that the sample mean is normally (n>30) distributed with Let's calculate the sample mean and standard error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "20b4f630-d875-465f-b70a-0f790821f3d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Open the hypothesis_testings.py file under the src folder to view the code:\n",
    "from src.hypothesis_testings import calc_z_score_and_p_value_given_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6f9939e3-afa8-4843-a2ab-f8afcb25ad1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.705825053813384, 0.04402032848789611)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_score, p_value = calc_z_score_and_p_value_given_samples(height_df_1, 176.5, is_one_sided=True)\n",
    "z_score, p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c286aaa0-47da-4b4f-8b0a-153c1fb9bfae",
   "metadata": {},
   "source": [
    "The resulted z-score is 1.7. The critical value for a 95% confindance interval is 1.96 for a two sided test, thus, we fail to rejcet the null hypothesis and we accept that the average height is indeed 176.5 cm. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce140b9b-c194-4820-8b58-62b5c53a381e",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: Could we reject the results above if we conduct a left sided hypotheses test?  </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356669fb-c9fb-434e-b190-e1de5eb74f0f",
   "metadata": {},
   "source": [
    "\n",
    "$$ \\large H_{0}: \\mu \\geq{176.5}$$\n",
    "<br>\n",
    "$$ \\large H_{0}: \\mu < 176.5$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7babde5b-ad67-443b-b00f-a83a26b1e9af",
   "metadata": {},
   "source": [
    "Answer - The calculations are the same. But since the Z-score is positive, than the anomaly is on the right while the rejection zone is on the left. Conculsion - do not look only at the p-value!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e52319-80b4-4c16-aacb-0cd2496f8563",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: Could we reject the results above if we conduct a right sided hypotheses test?  </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6053d1e4-8edd-4891-bcb0-0c73b213f2c6",
   "metadata": {},
   "source": [
    "\n",
    "$$ \\large H_{0}: \\mu \\leq{176.5}$$\n",
    "<br>\n",
    "$$ \\large H_{0}: \\mu > 176.5$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "81522a9f-5577-4ec1-bbab-14f5eac40dcb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.705825053813384, 0.04402032848789611)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_score, p_value = calc_z_score_and_p_value_given_samples(height_df_1, 176.5, is_one_sided=True)\n",
    "z_score, p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74b9d5c-236a-4e5c-ad1f-81c2403c754d",
   "metadata": {},
   "source": [
    "Answer: We can reject the null hypothesis in the 95% confidance level if the test was a one sided test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9841717-4d50-4b6c-82f4-7e7098737f1e",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: Should we switch to a one-sided hypothesis test to reject the null hypothesis? </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a94af96-5e7e-4a35-883e-5e61ce4f7ec0",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: We've increased the samples to 1000, in the file isreali_man_1000_samples. Do we still fail to reject the null hypothesis in 95% and 99% confidance level? </b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3f26a5b7-9481-4391-b411-117d87d925ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_df_1 = pd.read_csv('data/isreali_man_100_samples.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7b5d057e-e059-4451-8e9e-06ca4819f58e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 1)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "height_df_1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "31f0e079-afc2-4b13-a5b1-7b2c5359e35c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2.4159077188743225, 0.01569603876048797)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z_score, p_value = calc_z_score_and_p_value_given_samples(height_df_1, 176.5, is_one_sided=False)\n",
    "z_score, p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2889650c-155f-4427-b0ab-69bbe49a3605",
   "metadata": {},
   "source": [
    "We can now reject the two sided null hypothesis in the 95% confidance level. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb242f77-4d72-45af-b9fc-9a1551fd6278",
   "metadata": {},
   "source": [
    "### Comparing two unkonwn means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f0895bf-5b29-4407-90b1-ba739dfbaa67",
   "metadata": {},
   "source": [
    "* Two sided test, comparing two means:\n",
    "\n",
    "$$ \\large H_{0}: \\mu_{0} = \\mu_1$$\n",
    "<br>\n",
    "$$ \\large H_{1}: \\mu_0 \\neq{\\mu_1}$$\n",
    "\n",
    "* One sided (left), comparing two means:\n",
    "\n",
    "\n",
    "$$ \\large H_{0}: \\mu_0 \\geq{\\mu_1}$$\n",
    "<br>\n",
    "$$ \\large H_{1}: \\mu_0 < \\mu_1$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c11b565e-d636-44ce-977b-155515405caf",
   "metadata": {},
   "source": [
    "The mean of each sample is normally distributed (by CLT), and difference between them as well:\n",
    "\n",
    "$$\n",
    "\\Large {\\text{diff} \\sim \\mathcal{N}(\\mu_0-\\mu_1,\\,\\dfrac{\\sigma_0^{2}}{n} + \\dfrac{\\sigma_1^{2}}{n})\\,}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "392246ef-5e9f-4c2f-bbe1-8044c829ad94",
   "metadata": {},
   "source": [
    "Since the null hypothesis assumes the difference between the populations is 0 (or bigger), the Z statistic we will use is:\n",
    "\n",
    "$$\n",
    "\\Large \\frac{(\\bar{x}-\\bar{y}) - 0}{\\sqrt{\\dfrac{\\bar{s_{x}^{2}}}{n} + \\dfrac{\\bar{s_{y}^{2}}}{n}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c8181c-6fd3-4846-9c6a-39646120a0ea",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: What is the literal meaning of the above Z statistic?</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52567d77-eb64-4941-b378-a33c2a2b98ba",
   "metadata": {},
   "source": [
    "## Power"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336bb6d1-c830-4db4-92a6-4e48fa0da1b9",
   "metadata": {},
   "source": [
    "Power is the probablity of **correctly** rejecting the null hypothesis. Power, and power analysis, are the most important tools for pre-experimentation design and prevention of p-value hacking."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469a8e04-3231-4776-b493-598ebc28664d",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: What is P-value hacking?</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06c0688a-8d39-440d-85ef-7ffd463f16ff",
   "metadata": {},
   "source": [
    "P-value hacking is the misuse of post-experiment data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. Some examples:\n",
    "\n",
    "* Post experiment removal of samples \n",
    "* Post experiment addition of samples\n",
    "* Post experiment Switching from two sided to one sided test \n",
    "* Repeadtly running tests until one present the results of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e72e3393-07c8-479d-863a-60c229735b88",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "proprotion_1 = widgets.FloatSlider(\n",
    "    value=0.5,\n",
    "    min=0.4,\n",
    "    max=0.6,\n",
    "    step=.02,\n",
    "    description='Coin A:'\n",
    ")\n",
    "\n",
    "proprotion_2 = widgets.FloatSlider(\n",
    "    value=.55,\n",
    "    min=.4,\n",
    "    max=.6,\n",
    "    step=.02,\n",
    "    description='Coin B:'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4fb8ad75-f512-4632-ae3b-799b1dd5ab90",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f35cc1af-8b55-41e7-b1f2-a182e6b76b29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "751298c1f87449a394d332270c63aa62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(FloatSlider(value=0.5, description='Coin A:', max=0.6, min=0.4, step=0.02), FloatSlider(…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact\n",
    "def illusrate_power(p_1=proprotion_1, p_2=proprotion_2, samples=[50, 100, 500, 1000], show_power=True):\n",
    "    from statsmodels.stats.power import TTestIndPower\n",
    "    from src.hypothesis_testings import calc_power\n",
    "    analysis = TTestIndPower()\n",
    "    prop_1_distribution = calculate_normal_density_distribution(p_1,math.sqrt(p_1*(1-p_1)/samples), 1000)\n",
    "    prop_1_distribution['coin_id'] = 'A'\n",
    "    prop_2_distribution = calculate_normal_density_distribution(p_2,math.sqrt(p_2*(1-p_2)/samples), 1000)\n",
    "    prop_2_distribution['coin_id'] = 'B'\n",
    "    if show_power:\n",
    "        print(f'The power is: {calc_power(p_1, p_2, samples, 0.05)}')\n",
    "    data = prop_1_distribution.append(prop_2_distribution).sort_values(by='x')\n",
    "    fig = px.line(data, x='x',y='y', color='coin_id', range_x=(0.2,0.8))\n",
    "    return fig.update_layout(title_text=f\"Distribution of mean A ({p_1}) and mean B ({p_2})\", title_x=0.5)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a40ee0-87ce-4b10-b82f-deea2444366f",
   "metadata": {},
   "source": [
    "### Power analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ef43ba-29fc-4d64-ba66-80effbda3982",
   "metadata": {},
   "source": [
    "The test power is determend four fatcors - The two main ones are:\n",
    "* The true difference between the means, AKA effect size\n",
    "* The number of samples\n",
    "\n",
    "But is also effected by:\n",
    "* The desired type 1 error level $\\alpha$\n",
    "* The desired type 2 error level $\\beta$\n",
    "\n",
    "Power analysis involves estimating one of these four parameters given values for three other parameters. This is a powerful tool in both the design and in the analysis of experiments that we wish to interpret using statistical hypothesis tests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a432ea-9dd6-4760-9256-d943dcad22fb",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: Use <a href=\"https://www.statsmodels.org/stable/generated/statsmodels.stats.power.TTestIndPower.power.html#statsmodels.stats.power.TTestIndPower.power\"> statsmodel TTestIndPower class </a> to calculate the power and print the power in each of the illustrations above. Print the power if the show_power flag in on</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "204aa725-ec06-47ff-9747-6521399e64fc",
   "metadata": {},
   "source": [
    "Answer: The code was added to the illustration, See the implementation in the hypothesis_testings.py file under the src folder. In the following you'll see an example of the usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b406cf1b-4f6d-48a9-b353-456dfaf4456e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.hypothesis_testings import calc_power"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6b2353ee-5f10-4e23-a36f-9f1851a2581d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m \u001b[0mcalc_power\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp_1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp_2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_samples\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0.05\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m <no docstring>\n",
       "\u001b[0;31mFile:\u001b[0m      ~/dev/Python-ML-2022/Python-ML---Lesson-2/src/hypothesis_testings.py\n",
       "\u001b[0;31mType:\u001b[0m      function\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "calc_power?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "40a630ec-6e15-4196-b60d-c31b579eec28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07852398768985813"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_power(p_1 = 0.5, p_2=0.55, n_samples=50, alpha=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfd847b7-a408-48c1-b534-4a19947dcf59",
   "metadata": {},
   "source": [
    "So although coin B has a 10% (0.5 -> 0.55) higher effect size, by using only 50 samples we will **fail to reject the null hypothesis in 92% of the cases**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d42de52-bb90-4fcd-a6e7-8fa557e8a2ec",
   "metadata": {},
   "source": [
    "## AB tests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7052c9b-9c2c-4c95-b3b8-fe05d4468026",
   "metadata": {},
   "source": [
    "An AB test is a broader scope experimint in which hypotheses testing is the main statistical tool, and is consists of two parts:\n",
    "\n",
    "* Experiment design - In this step we define:\n",
    "  * The group assingment logic\n",
    "  * The proportion of traffic/population to be used\n",
    "  * Sub-populations, if any, on which the test will be run\n",
    "  * The needed number of samples to reach the proper statistical signifance in both power and confidance level\n",
    "  * The test duration in days/weeks, as derived from the needed number of samples.\n",
    "  * Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4da8711-e906-480e-a698-13ae8757d3de",
   "metadata": {},
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "831061b4-3e03-458d-90a7-a08f29ba250e",
   "metadata": {},
   "source": [
    "The new company is promising a better conversion rate (proportion). Our goal is to understand the buisness needs and translate them into accurate test. To do so, there are few questions we should ask our CEO:\n",
    "\n",
    "* What is the buisness impact of each 1% relative improvement in conversions? (Or in other words, how much each relative improvement worth to the company)\n",
    "* What is the buying price of the new company? what is the minimal conversion rate increase to justify the asked price?\n",
    "* Is the new company supposed to be better on all traffic or a subsegment of the traffic (for example, they might be working only in the US marketplace)\n",
    "* Do we want to compare the performances by segment or overall?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca46bd3-c49c-41e6-9f8f-c0051cb55bfc",
   "metadata": {},
   "source": [
    "Let's assume that the input from the CEO that the minimal increase for purchasing the company is 5% higher from the current state, 7% is a good deal and 10% is a great deal and that the new company works on all marketplaces.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0abec8dd-a504-40d4-9b83-7a9c49be18de",
   "metadata": {},
   "source": [
    "#### Proposed design"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "455e31e4-bb5e-41b6-b55d-e514185ebe64",
   "metadata": {},
   "source": [
    "A proper AB test in that case:\n",
    "\n",
    "* Will trigger the test on all traffic (no sub-categories)\n",
    "* Will run the test on whole number of weeks (1, 2, 3 weeks) to avoid biases of day-of-week \n",
    "* Will design a two sided hypothesis test with a null hypothesis stating that the new company's conversion rate is identical to ours \n",
    "* Will design the test with enough power to detect a 5% relative effect increase with 95% and 99% probabilites (will offer both numbers). (Note that this means that higher effect sizes will also be decteted in even higher probabilities)\n",
    "* Will design the test with an alpha of 0.05 and 0.01 - to prevent falsy rejecting the null hypothesis\n",
    "* **Will supply a table holding all possible durations and sample sizes - And allow the stake holders to make a data-driven decion**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ad76c66-a560-4031-8eb9-1dd8233b03da",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: Use the solve_power method in TTestIndPower to prepare a table with all possible sample needed. Assume the current conversion rate is 15%. </b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "88435761-08a0-45d1-9947-8c206536bd8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "alphes = [0.05, 0.01]\n",
    "powers = [0.95, 0.99]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "555e095c-eec1-4b6c-92da-701185d96692",
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.hypothesis_testings import calc_needed_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "973a0205-ace7-4819-8e8f-518271fe510f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "58910.314791263045"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_needed_samples(0.15, 0.05, 0.05, 0.95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "985938b8-56b3-4b43-8076-52ef1442a1dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples_needed = []\n",
    "curr_conv_rate = 0.15\n",
    "for relative_inc in [0.05, 0.07, 0.1]:\n",
    "    for alpha in alphes:\n",
    "        for power in powers:\n",
    "            needed_samples = calc_needed_samples(curr_conv_rate, relative_inc, alpha, power)\n",
    "            samples_needed.append((100 * curr_conv_rate, \n",
    "                                   100 * curr_conv_rate*(1 + relative_inc),\n",
    "                                   int(100 * relative_inc), \n",
    "                                   1-alpha, power, int(needed_samples)))           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5cb8da11-5d99-439b-971a-1443a2c91b85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Current rate</th>\n",
       "      <th>New rate</th>\n",
       "      <th>Relative difference (%)</th>\n",
       "      <th>Confidance Level</th>\n",
       "      <th>Power</th>\n",
       "      <th># Samples</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15.0</td>\n",
       "      <td>15.75</td>\n",
       "      <td>5</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.99</td>\n",
       "      <td>108943</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>15.75</td>\n",
       "      <td>5</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.99</td>\n",
       "      <td>83289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15.0</td>\n",
       "      <td>15.75</td>\n",
       "      <td>5</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.95</td>\n",
       "      <td>80759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15.0</td>\n",
       "      <td>15.75</td>\n",
       "      <td>5</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.95</td>\n",
       "      <td>58910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.05</td>\n",
       "      <td>7</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.99</td>\n",
       "      <td>55584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.05</td>\n",
       "      <td>7</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.99</td>\n",
       "      <td>42495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.05</td>\n",
       "      <td>7</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.95</td>\n",
       "      <td>41204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.05</td>\n",
       "      <td>7</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.95</td>\n",
       "      <td>30056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.50</td>\n",
       "      <td>10</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.99</td>\n",
       "      <td>27237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.50</td>\n",
       "      <td>10</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.99</td>\n",
       "      <td>20823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.50</td>\n",
       "      <td>10</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.95</td>\n",
       "      <td>20191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>15.0</td>\n",
       "      <td>16.50</td>\n",
       "      <td>10</td>\n",
       "      <td>0.95</td>\n",
       "      <td>0.95</td>\n",
       "      <td>14728</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Current rate  New rate  Relative difference (%)  Confidance Level  Power  \\\n",
       "3           15.0     15.75                        5              0.99   0.99   \n",
       "1           15.0     15.75                        5              0.95   0.99   \n",
       "2           15.0     15.75                        5              0.99   0.95   \n",
       "0           15.0     15.75                        5              0.95   0.95   \n",
       "7           15.0     16.05                        7              0.99   0.99   \n",
       "5           15.0     16.05                        7              0.95   0.99   \n",
       "6           15.0     16.05                        7              0.99   0.95   \n",
       "4           15.0     16.05                        7              0.95   0.95   \n",
       "11          15.0     16.50                       10              0.99   0.99   \n",
       "9           15.0     16.50                       10              0.95   0.99   \n",
       "10          15.0     16.50                       10              0.99   0.95   \n",
       "8           15.0     16.50                       10              0.95   0.95   \n",
       "\n",
       "    # Samples  \n",
       "3      108943  \n",
       "1       83289  \n",
       "2       80759  \n",
       "0       58910  \n",
       "7       55584  \n",
       "5       42495  \n",
       "6       41204  \n",
       "4       30056  \n",
       "11      27237  \n",
       "9       20823  \n",
       "10      20191  \n",
       "8       14728  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(samples_needed, columns=['Current rate','New rate','Relative difference (%)', 'Confidance Level', 'Power','# Samples'])\n",
    "results.sort_values(by='# Samples', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff644231-f097-4123-aac0-a01d6e6b061c",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: What are the trade offs in each row of the results table? </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a2d948e-55ab-4908-b5d0-bd71bc6b7226",
   "metadata": {},
   "source": [
    "### Post experimet analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a387ba5c-5f93-43a7-bb90-6ca92cf929bb",
   "metadata": {},
   "source": [
    "Analysis of the results should only be started after the AB test terimenated (according to the pre-defined duration). Avoid doing pre-termination analysis as those can be misleading. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab353fd8-8f94-41db-aa27-a08e3e9cf86a",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Exercise: The AB test was terminated after 1 week on-air, you've gathered 50,000 samples of each group. The mean of the control group was 0.151 while the mean of the treatment (new company) was 0.169. compute a 99% CI for the difference between the treatment and control conversion rate </b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75a29cb4-8778-4029-b045-12541bb9c41c",
   "metadata": {},
   "source": [
    "As the expermint owner, we now need to evaluate two things:\n",
    "1. What is the p-value, and as such, Which hypothesis is to be rejected and which to be excepted?\n",
    "2. If the null hypothesis is to be rejected, what is the most likely diff and what is a 95% and 99% confidance interval for the diff between the methods? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80626e78-4a02-4d59-aed7-b8593a0261c4",
   "metadata": {},
   "source": [
    "Let's start with question 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "bf298fc9-df91-4c1a-87d5-17c82473cabd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will name the A and B group as control and treatment:\n",
    "control_m, control_s = 0.151, math.sqrt((1-0.151) * 0.151)\n",
    "treatment_m, treatment_s = 0.169, math.sqrt((1-0.169) * 0.169)\n",
    "n_samples = 50000\n",
    "z_score = (control_m - treatment_s) / math.sqrt((control_s ** 2 + treatment_s ** 2) / n_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bdec5a8f-2abd-4ba1-a22b-4a1499ac4d50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from src.hypothesis_testings import calc_p_value_from_z_score\n",
    "calc_p_value_from_z_score(z_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f809888-6c26-4491-9d1a-d34fa0b236b4",
   "metadata": {},
   "source": [
    "There is 0% Chance to get such extreme results if the null hypothesis is correct, the null hypothesis is to be rejected with 100% certainty. So we now know that there is a difference in the conversion rates, let's calculate a 99% confidance interval that contains the true difference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f6b1f22f-95a0-4080-8245-d8a77a94c2bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.01202942992151482, 0.018000000000000016, 0.02397057007848521)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "estimated_diff = treatment_m - control_m\n",
    "se = math.sqrt(treatment_s**2 / n_samples + control_s**2 / n_samples)\n",
    "z_half_alpha = stats.norm.ppf(1 - 0.01/2)\n",
    "diff_ci = (estimated_diff - z_half_alpha * se, estimated_diff, estimated_diff + z_half_alpha * se)\n",
    "diff_ci"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8698b39-73c0-41cb-bab8-9c99d9da295c",
   "metadata": {},
   "source": [
    "The most likely difference between the control and the test is 0.018 difference points, which is 1.8% increase to the current conversion rate. The true difference is in 99% certainty somewhere between 1.2% (0.012) and 2.3% (0.023). We should also calculate the relative difference: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c6877412-605c-47cb-b736-1efa35d7ec24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.07966509881797895, 0.11920529801324514, 0.15874549720851133]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "relative_ci = [bound / control_m for bound in diff_ci]\n",
    "relative_ci"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d2af9e-13f3-4dd7-ae7f-14b352641e72",
   "metadata": {},
   "source": [
    "The true relative difference is in 99% certainty somewhere between 8% and 15%."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17259528-252a-458b-9da9-9eb73cad9d3c",
   "metadata": {},
   "source": [
    "<font size=3 color='red'><b>Question: Given the AB test results, what will you recommend the CEO to do?  </b></font>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}