Bayesian_Inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our Mission ##\n",
    "\n",
    "Spam detection is one of the major applications of Machine Learning in the interwebs today. Pretty much all of the major email service providers have spam detection systems built in and automatically classify such mail as 'Junk Mail'. \n",
    "\n",
    "In this mission we will be using the Naive Bayes algorithm to create a model that can classify SMS messages as spam or not spam, based on the training we give to the model. It is important to have some level of intuition as to what a spammy text message might look like. Often they have words like 'free', 'win', 'winner', 'cash', 'prize' and the like in them as these texts are designed to catch your eye and in some sense tempt you to open them. Also, spam messages tend to have words written in all capitals and also tend to use a lot of exclamation marks. To the human recipient, it is usually pretty straightforward to identify a spam text and our objective here is to train a model to do that for us!\n",
    "\n",
    "Being able to identify spam messages is a binary classification problem as messages are classified as either 'Spam' or 'Not Spam' and nothing else. Also, this is a supervised learning problem, as we will be feeding a labelled dataset into the model, that it can learn from, to make future predictions. \n",
    "\n",
    "# Overview\n",
    "\n",
    "This project has been broken down in to the following steps: \n",
    "\n",
    "- Step 0: Introduction to the Naive Bayes Theorem\n",
    "- Step 1.1: Understanding our dataset\n",
    "- Step 1.2: Data Preprocessing\n",
    "- Step 2.1: Bag of Words (BoW)\n",
    "- Step 2.2: Implementing BoW from scratch\n",
    "- Step 2.3: Implementing Bag of Words in scikit-learn\n",
    "- Step 3.1: Training and testing sets\n",
    "- Step 3.2: Applying Bag of Words processing to our dataset.\n",
    "- Step 4.1: Bayes Theorem implementation from scratch\n",
    "- Step 4.2: Naive Bayes implementation from scratch\n",
    "- Step 5: Naive Bayes implementation using scikit-learn\n",
    "- Step 6: Evaluating our model\n",
    "- Step 7: Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 0: Introduction to the Naive Bayes Theorem ###\n",
    "\n",
    "Bayes Theorem is one of the earliest probabilistic inference algorithms. It was developed by Reverend Bayes (which he used to try and infer the existence of God no less), and still performs extremely well for certain use cases. \n",
    "\n",
    "It's best to understand this theorem using an example. Let's say you are a member of the Secret Service and you have been deployed to protect the Democratic presidential nominee during one of his/her campaign speeches. Being a public event that is open to all, your job is not easy and you have to be on the constant lookout for threats. So one place to start is to put a certain threat-factor for each person. So based on the features of an individual, like age, whether the person is carrying a bag, looks nervous, etc., you can make a judgment call as to whether that person is a viable threat. \n",
    "\n",
    "If an individual ticks all the boxes up to a level where it crosses a threshold of doubt in your mind, you can take action and remove that person from the vicinity. Bayes Theorem works in the same way, as we are computing the probability of an event (a person being a threat) based on the probabilities of certain related events (age, presence of bag or not, nervousness of the person, etc.). \n",
    "\n",
    "One thing to consider is the independence of these features amongst each other. For example if a child looks nervous at the event then the likelihood of that person being a threat is not as much as say if it was a grown man who was nervous. To break this down a bit further, here there are two features we are considering, age AND nervousness. Say we look at these features individually, we could design a model that flags ALL persons that are nervous as potential threats. However, it is likely that we will have a lot of false positives as there is a strong chance that minors present at the event will be nervous. Hence by considering the age of a person along with the 'nervousness' feature we would definitely get a more accurate result as to who are potential threats and who aren't. \n",
    "\n",
    "This is the 'Naive' bit of the theorem where it considers each feature to be independent of each other which may not always be the case and hence that can affect the final judgement.\n",
    "\n",
    "In short, Bayes Theorem calculates the probability of a certain event happening (in our case, a message being spam) based on the joint probabilistic distributions of certain other events (in our case, the appearance of certain words in a message). We will dive into the workings of Bayes Theorem later in the mission, but first, let us understand the data we are going to work with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.1: Understanding our dataset ### \n",
    "\n",
    "\n",
    "We will be using a dataset originally compiled and posted on the UCI Machine Learning repository which has a very good collection of datasets for experimental research purposes. If you're interested, you can review the [abstract](https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection) and the original [compressed data file](https://archive.ics.uci.edu/ml/machine-learning-databases/00228/) on the UCI site. For this exercise, however, we've gone ahead and downloaded the data for you.\n",
    "\n",
    "\n",
    " **Here's a preview of the data:** \n",
    "\n",
    "<img src=\"images/dqnb.png\" height=\"1242\" width=\"1242\">\n",
    "\n",
    "The columns in the data set are currently not named and as you can see, there are 2 columns. \n",
    "\n",
    "The first column takes two values, 'ham' which signifies that the message is not spam, and 'spam' which signifies that the message is spam. \n",
    "\n",
    "The second column is the text content of the SMS message that is being classified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">**Instructions:**\n",
    "* Import the dataset into a pandas dataframe using the **read_table** method. The file has already been downloaded, and you can access it using the filepath 'smsspamcollection/SMSSpamCollection'. Because this is a tab separated dataset we will be using '\\\\t' as the value for the 'sep' argument which specifies this format. \n",
    "* Also, rename the column names by specifying a list ['label', 'sms_message'] to the 'names' argument of read_table().\n",
    "* Print the first five values of the dataframe with the new column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List all the files in the current directory\n",
      "\n",
      "Bayesian_Inference.ipynb\t   images\n",
      "Bayesian_Inference_solution.ipynb  smsspamcollection\n",
      "\n",
      " List all the files inside the smsspamcollection directory\n",
      "\n",
      "readme\tSMSSpamCollection\n"
     ]
    }
   ],
   "source": [
    "# '!' allows you to run bash commands from jupyter notebook.\n",
    "print(\"List all the files in the current directory\\n\")\n",
    "!ls\n",
    "# The required data table can be found under smsspamcollection/SMSSpamCollection\n",
    "print(\"\\n List all the files inside the smsspamcollection directory\\n\")\n",
    "!ls smsspamcollection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>sms_message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                        sms_message\n",
       "0   ham  Go until jurong point, crazy.. Available only ...\n",
       "1   ham                      Ok lar... Joking wif u oni...\n",
       "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3   ham  U dun say so early hor... U c already then say...\n",
       "4   ham  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "# Dataset available using filepath 'smsspamcollection/SMSSpamCollection'\n",
    "df = pd.read_table('data/SMSSpamCollection', sep = '\\t', names = ['label', 'sms_message'])\n",
    "\n",
    "# Output printing out first 5 rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2: Data Preprocessing ###\n",
    "\n",
    "Now that we have a basic understanding of what our dataset looks like, let's convert our labels to binary variables, 0 to represent 'ham'(i.e. not spam) and 1 to represent 'spam' for ease of computation. \n",
    "\n",
    "You might be wondering why do we need to do this step? The answer to this lies in how scikit-learn handles inputs. Scikit-learn only deals with numerical values and hence if we were to leave our label values as strings, scikit-learn would do the conversion internally (more specifically, the string labels will be cast to unknown float values). \n",
    "\n",
    "Our model would still be able to make predictions if we left our labels as strings but we could have issues later when calculating performance metrics, for example when calculating our precision and recall scores. Hence, to avoid unexpected 'gotchas' later, it is good practice to have our categorical values be fed into our model as integers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">**Instructions:**\n",
    "* Convert the values in the 'label' column to numerical values using map method as follows:\n",
    "{'ham':0, 'spam':1} This maps the 'ham' value to 0 and the 'spam' value to 1.\n",
    "* Also, to get an idea of the size of the dataset we are dealing with, print out number of rows and columns using \n",
    "'shape'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>sms_message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label                                        sms_message\n",
       "0      0  Go until jurong point, crazy.. Available only ...\n",
       "1      0                      Ok lar... Joking wif u oni...\n",
       "2      1  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      0  U dun say so early hor... U c already then say...\n",
       "4      0  Nah I don't think he goes to usf, he lives aro..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['label'] = df.label.map({'ham':0, 'spam':1})\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5572, 2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.1: Bag of Words ###\n",
    "\n",
    "What we have here in our data set is a large collection of text data (5,572 rows of data). Most ML algorithms rely on numerical data to be fed into them as input, and email/sms messages are usually text heavy. \n",
    "\n",
    "Here we'd like to introduce the Bag of Words (BoW) concept which is a term used to specify the problems that have a 'bag of words' or a collection of text data that needs to be worked with. The basic idea of BoW is to take a piece of text and count the frequency of the words in that text. It is important to note that the BoW concept treats each word individually and the order in which the words occur does not matter. \n",
    "\n",
    "Using a process which we will go through now, we can convert a collection of documents to a matrix, with each document being a row and each word (token) being the column, and the corresponding (row, column) values being the frequency of occurrence of each word or token in that document.\n",
    "\n",
    "For example: \n",
    "\n",
    "Let's say we have 4 documents, which are text messages\n",
    "in our case, as follows:\n",
    "\n",
    "`['Hello, how are you!',\n",
    "'Win money, win from home.',\n",
    "'Call me now',\n",
    "'Hello, Call you tomorrow?']`\n",
    "\n",
    "Our objective here is to convert this set of texts to a frequency distribution matrix, as follows:\n",
    "\n",
    "<img src=\"images/countvectorizer.png\" height=\"542\" width=\"542\">\n",
    "\n",
    "Here as we can see, the documents are numbered in the rows, and each word is a column name, with the corresponding value being the frequency of that word in the document.\n",
    "\n",
    "Let's break this down and see how we can do this conversion using a small set of documents.\n",
    "\n",
    "To handle this, we will be using sklearn's \n",
    "[count vectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer) method which does the following:\n",
    "\n",
    "* It tokenizes the string (separates the string into individual words) and gives an integer ID to each token.\n",
    "* It counts the occurrence of each of those tokens.\n",
    "\n",
    "**Please Note:** \n",
    "\n",
    "* The CountVectorizer method automatically converts all tokenized words to their lower case form so that it does not treat words like 'He' and 'he' differently. It does this using the `lowercase` parameter which is by default set to `True`.\n",
    "\n",
    "* It also ignores all punctuation so that words followed by a punctuation mark (for example: 'hello!') are not treated differently than the same words not prefixed or suffixed by a punctuation mark (for example: 'hello'). It does this using the `token_pattern` parameter which has a default regular expression which selects tokens of 2 or more alphanumeric characters.\n",
    "\n",
    "* The third parameter to take note of is the `stop_words` parameter. Stop words refer to the most commonly used words in a language. They include words like 'am', 'an', 'and', 'the', etc. By setting this parameter value to `english`, CountVectorizer will automatically ignore all words (from our input text) that are found in the built in list of English stop words in scikit-learn. This is extremely helpful as stop words can skew our calculations when we are trying to find certain key words that are indicative of spam.\n",
    "\n",
    "We will dive into the application of each of these into our model in a later step, but for now it is important to be aware of such preprocessing techniques available to us when dealing with textual data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.2: Implementing Bag of Words from scratch ###\n",
    "\n",
    "Before we dive into scikit-learn's Bag of Words (BoW) library to do the dirty work for us, let's implement it ourselves first so that we can understand what's happening behind the scenes. \n",
    "\n",
    "**Step 1: Convert all strings to their lower case form.**\n",
    "\n",
    "Let's say we have a document set:\n",
    "\n",
    "```\n",
    "documents = ['Hello, how are you!',\n",
    "             'Win money, win from home.',\n",
    "             'Call me now.',\n",
    "             'Hello, Call hello you tomorrow?']\n",
    "```\n",
    ">>**Instructions:**\n",
    "* Convert all the strings in the documents set to their lower case. Save them into a list called 'lower_case_documents'. You can convert strings to their lower case in python by using the lower() method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello, how are you!', 'win money, win from home.', 'call me now.', 'hello, call hello you tomorrow?']\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution:\n",
    "'''\n",
    "documents = ['Hello, how are you!',\n",
    "             'Win money, win from home.',\n",
    "             'Call me now.',\n",
    "             'Hello, Call hello you tomorrow?']\n",
    "\n",
    "lower_case_documents = []\n",
    "for i in documents:\n",
    "    lower_case_documents.append(i.lower())\n",
    "print(lower_case_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2: Removing all punctuation**\n",
    "\n",
    ">>**Instructions:**\n",
    "Remove all punctuation from the strings in the document set. Save the strings into a list called \n",
    "'sans_punctuation_documents'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello how are you', 'win money win from home', 'call me now', 'hello call hello you tomorrow']\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution:\n",
    "'''\n",
    "sans_punctuation_documents = []\n",
    "import re, string\n",
    "\n",
    "regex = re.compile('[%s]' % re.escape(string.punctuation))\n",
    "\n",
    "for i in lower_case_documents:\n",
    "    sans_punctuation_documents.append(regex.sub('', i))\n",
    "    \n",
    "print(sans_punctuation_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3: Tokenization**\n",
    "\n",
    "Tokenizing a sentence in a document set means splitting up the sentence into individual words using a delimiter. The delimiter specifies what character we will use to identify the beginning and  end of a word. Most commonly, we use a single space as the delimiter character for identifying words, and this is true in our documents in this case also."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">>**Instructions:**\n",
    "Tokenize the strings stored in 'sans_punctuation_documents' using the split() method. Store the final document set \n",
    "in a list called 'preprocessed_documents'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['hello', 'how', 'are', 'you'], ['win', 'money', 'win', 'from', 'home'], ['call', 'me', 'now'], ['hello', 'call', 'hello', 'you', 'tomorrow']]\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution:\n",
    "'''\n",
    "preprocessed_documents = []\n",
    "for i in sans_punctuation_documents:\n",
    "    preprocessed_documents.append(i.split(' '))\n",
    "print(preprocessed_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 4: Count frequencies**\n",
    "\n",
    "Now that we have our document set in the required format, we can proceed to counting the occurrence of each word in each document of the document set. We will use the `Counter` method from the Python `collections` library for this purpose. \n",
    "\n",
    "`Counter` counts the occurrence of each item in the list and returns a dictionary with the key as the item being counted and the corresponding value being the count of that item in the list. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">>**Instructions:**\n",
    "Using the Counter() method and preprocessed_documents as the input, create a dictionary with the keys being each word in each document and the corresponding values being the frequency of occurrence of that word. Save each Counter dictionary as an item in a list called 'frequency_list'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Counter({'hello': 1, 'how': 1, 'are': 1, 'you': 1}),\n",
      " Counter({'win': 2, 'money': 1, 'from': 1, 'home': 1}),\n",
      " Counter({'call': 1, 'me': 1, 'now': 1}),\n",
      " Counter({'hello': 2, 'call': 1, 'you': 1, 'tomorrow': 1})]\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "frequency_list = []\n",
    "import pprint\n",
    "from collections import Counter\n",
    "\n",
    "for i in preprocessed_documents:\n",
    "    frequency_list.append(Counter(i))\n",
    "    \n",
    "pprint.pprint(frequency_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have implemented the Bag of Words process from scratch! As we can see in our previous output, we have a frequency distribution dictionary which gives a clear view of the text that we are dealing with.\n",
    "\n",
    "We should now have a solid understanding of what is happening behind the scenes in the `sklearn.feature_extraction.text.CountVectorizer` method of scikit-learn. \n",
    "\n",
    "We will now implement `sklearn.feature_extraction.text.CountVectorizer` method in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.3: Implementing Bag of Words in scikit-learn ###\n",
    "\n",
    "Now that we have implemented the BoW concept from scratch, let's go ahead and use scikit-learn to do this process in a clean and succinct way. We will use the same document set as we used in the previous step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Here we will look to create a frequency matrix on a smaller document set to make sure we understand how the \n",
    "document-term matrix generation happens. We have created a sample document set 'documents'.\n",
    "'''\n",
    "documents = ['Hello, how are you!',\n",
    "                'Win money, win from home.',\n",
    "                'Call me now.',\n",
    "                'Hello, Call hello you tomorrow?']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">>**Instructions:**\n",
    "Import the sklearn.feature_extraction.text.CountVectorizer method and create an instance of it called 'count_vector'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vector = CountVectorizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data preprocessing with CountVectorizer()**\n",
    "\n",
    "In Step 2.2, we implemented a version of the CountVectorizer() method from scratch that entailed cleaning our data first. This cleaning involved converting all of our data to lower case and removing all punctuation marks. CountVectorizer() has certain parameters which take care of these steps for us. They are:\n",
    "\n",
    "* `lowercase = True`\n",
    "    \n",
    "    The `lowercase` parameter has a default value of `True` which converts all of our text to its lower case form.\n",
    "\n",
    "\n",
    "* `token_pattern = (?u)\\\\b\\\\w\\\\w+\\\\b`\n",
    "    \n",
    "    The `token_pattern` parameter has a default regular expression value of `(?u)\\\\b\\\\w\\\\w+\\\\b` which ignores all punctuation marks and treats them as delimiters, while accepting alphanumeric strings of length greater than or equal to 2, as individual tokens or words.\n",
    "\n",
    "\n",
    "* `stop_words`\n",
    "\n",
    "    The `stop_words` parameter, if set to `english` will remove all words from our document set that match a list of English stop words defined in scikit-learn. Considering the small size of our dataset and the fact that we are dealing with SMS messages and not larger text sources like e-mail, we will not use stop words, and we won't be setting this parameter value.\n",
    "\n",
    "You can take a look at all the parameter values of your `count_vector` object by simply printing out the object as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
      "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
      "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
      "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
      "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
      "        tokenizer=None, vocabulary=None)\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Practice node:\n",
    "Print the 'count_vector' object which is an instance of 'CountVectorizer()'\n",
    "'''\n",
    "\n",
    "print(count_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">>**Instructions:**\n",
    "Fit your document dataset to the CountVectorizer object you have created using fit(), and get the list of words \n",
    "which have been categorized as features using the get_feature_names() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['are',\n",
       " 'call',\n",
       " 'from',\n",
       " 'hello',\n",
       " 'home',\n",
       " 'how',\n",
       " 'me',\n",
       " 'money',\n",
       " 'now',\n",
       " 'tomorrow',\n",
       " 'win',\n",
       " 'you']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Solution:\n",
    "'''\n",
    "\n",
    "count_vector.fit(documents)\n",
    "count_vector.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `get_feature_names()` method returns our feature names for this dataset, which is the set of words that make up our vocabulary for 'documents'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">>**Instructions:**\n",
    "Create a matrix with each row representing one of the 4 documents, and each column representing a word (feature name). \n",
    "Each value in the matrix will represent the frequency of the word in that column occurring in the particular document in that row. \n",
    "You can do this using the transform() method of CountVectorizer, passing in the document data set as the argument. The transform() method returns a matrix of NumPy integers, which you can convert to an array using\n",
    "toarray(). Call the array 'doc_array'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1],\n",
       "       [0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 2, 0],\n",
       "       [0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0],\n",
       "       [0, 1, 0, 2, 0, 0, 0, 0, 0, 1, 0, 1]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "doc_array = count_vector.transform(documents).toarray()\n",
    "doc_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a clean representation of the documents in terms of the frequency distribution of the words in them. To make it easier to understand our next step is to convert this array into a dataframe and name the columns appropriately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">>**Instructions:**\n",
    "Convert the 'doc_array' we created into a dataframe, with the column names as the words (feature names). Call the dataframe 'frequency_matrix'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>are</th>\n",
       "      <th>call</th>\n",
       "      <th>from</th>\n",
       "      <th>hello</th>\n",
       "      <th>home</th>\n",
       "      <th>how</th>\n",
       "      <th>me</th>\n",
       "      <th>money</th>\n",
       "      <th>now</th>\n",
       "      <th>tomorrow</th>\n",
       "      <th>win</th>\n",
       "      <th>you</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   are  call  from  hello  home  how  me  money  now  tomorrow  win  you\n",
       "0    1     0     0      1     0    1   0      0    0         0    0    1\n",
       "1    0     0     1      0     1    0   0      1    0         0    2    0\n",
       "2    0     1     0      0     0    0   1      0    1         0    0    0\n",
       "3    0     1     0      2     0    0   0      0    0         1    0    1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "frequency_matrix = pd.DataFrame(doc_array, columns=count_vector.get_feature_names())\n",
    "frequency_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully implemented a Bag of Words problem for a document dataset that we created. \n",
    "\n",
    "One potential issue that can arise from using this method is that if our dataset of text is extremely large (say if we have a large collection of news articles or email data), there will be certain values that are more common than others simply due to the structure of the language itself. For example, words like 'is', 'the', 'an', pronouns, grammatical constructs, etc., could skew our matrix and affect our analyis. \n",
    "\n",
    "There are a couple of ways to mitigate this. One way is to use the `stop_words` parameter and set its value to `english`. This will automatically ignore all the words in our input text that are found in a built-in list of English stop words in scikit-learn.\n",
    "\n",
    "Another way of mitigating this is by using the [tfidf](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer) method. This method is out of scope for the context of this lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.1: Training and testing sets ###\n",
    "\n",
    "Now that we understand how to use the Bag of Words approach, we can return to our original, larger UCI dataset and proceed with our analysis. Our first step is to split our dataset into a training set and a testing set so we can first train, and then test our model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    ">>**Instructions:**\n",
    "Split the dataset into a training and testing set using the train_test_split method in sklearn, and print out the number of rows we have in each of our training and testing data. Split the data\n",
    "using the following variables:\n",
    "* `X_train` is our training data for the 'sms_message' column.\n",
    "* `y_train` is our training data for the 'label' column\n",
    "* `X_test` is our testing data for the 'sms_message' column.\n",
    "* `y_test` is our testing data for the 'label' column. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows in the total set: 5572\n",
      "Number of rows in the training set: 4179\n",
      "Number of rows in the test set: 1393\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution \n",
    "'''\n",
    "# split into training and testing sets\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['sms_message'], \n",
    "                                                    df['label'], \n",
    "                                                    random_state=1)\n",
    "\n",
    "print('Number of rows in the total set: {}'.format(df.shape[0]))\n",
    "print('Number of rows in the training set: {}'.format(X_train.shape[0]))\n",
    "print('Number of rows in the test set: {}'.format(X_test.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.2: Applying Bag of Words processing to our dataset. ###\n",
    "\n",
    "Now that we have split the data, our next objective is to follow the steps from \"Step 2: Bag of Words,\" and convert our data into the desired matrix format. To do this we will be using CountVectorizer() as we did before. There are two  steps to consider here:\n",
    "\n",
    "* First, we have to fit our training data (`X_train`) into `CountVectorizer()` and return the matrix.\n",
    "* Secondly, we have to transform our testing data (`X_test`) to return the matrix. \n",
    "\n",
    "Note that `X_train` is our training data for the 'sms_message' column in our dataset and we will be using this to train our model. \n",
    "\n",
    "`X_test` is our testing data for the 'sms_message' column and this is the data we will be using (after transformation to a matrix) to make predictions on. We will then compare those predictions with `y_test` in a later step. \n",
    "\n",
    "For now, we have provided the code that does the matrix transformations for you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n[Practice Node]\\n\\nThe code for this segment is in 2 parts. First, we are learning a vocabulary dictionary for the training data \\nand then transforming the data into a document-term matrix; secondly, for the testing data we are only \\ntransforming the data into a document-term matrix.\\n\\nThis is similar to the process we followed in Step 2.3.\\n\\nWe will provide the transformed data to students in the variables 'training_data' and 'testing_data'.\\n\""
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "[Practice Node]\n",
    "\n",
    "The code for this segment is in 2 parts. First, we are learning a vocabulary dictionary for the training data \n",
    "and then transforming the data into a document-term matrix; secondly, for the testing data we are only \n",
    "transforming the data into a document-term matrix.\n",
    "\n",
    "This is similar to the process we followed in Step 2.3.\n",
    "\n",
    "We will provide the transformed data to students in the variables 'training_data' and 'testing_data'.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "# Instantiate the CountVectorizer method\n",
    "count_vector = CountVectorizer()\n",
    "\n",
    "# Fit the training data and then return the matrix\n",
    "training_data = count_vector.fit_transform(X_train)\n",
    "\n",
    "# Transform testing data and return the matrix. Note we are not fitting the testing data into the CountVectorizer()\n",
    "testing_data = count_vector.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4.1: Bayes Theorem implementation from scratch ###\n",
    "\n",
    "Now that we have our dataset in the format that we need, we can move onto the next portion of our mission which is the  algorithm we will use to make our predictions to classify a message as spam or not spam. Remember that at the start of the mission we briefly discussed the Bayes theorem but now we shall go into a little more detail. In layman's terms, the Bayes theorem calculates the probability of an event occurring, based on certain other probabilities that are related to the event in question. It is composed of \"prior probabilities\" - or just \"priors.\" These \"priors\" are the probabilities that we are aware of, or that are given to us. And Bayes theorem is also composed of the \"posterior probabilities,\" or just \"posteriors,\" which are the probabilities we are looking to compute using the \"priors\". \n",
    "\n",
    "Let us implement the Bayes Theorem from scratch using a simple example. Let's say we are trying to find the odds of an individual having diabetes, given that he or she was tested for it and got a positive result. \n",
    "In the medical field, such probabilities play a very important role as they often deal with life and death situations. \n",
    "\n",
    "We assume the following:\n",
    "\n",
    "`P(D)` is the probability of a person having Diabetes. Its value is `0.01`, or in other words, 1% of the general population has diabetes (disclaimer: these values are assumptions and are not reflective of any actual medical study).\n",
    "\n",
    "`P(Pos)` is the probability of getting a positive test result.\n",
    "\n",
    "`P(Neg)` is the probability of getting a negative test result.\n",
    "\n",
    "`P(Pos|D)` is the probability of getting a positive result on a test done for detecting diabetes, given that you have diabetes. This has a value `0.9`. In other words the test is correct 90% of the time. This is also called the Sensitivity or True Positive Rate.\n",
    "\n",
    "`P(Neg|~D)` is the probability of getting a negative result on a test done for detecting diabetes, given that you do not have diabetes. This also has a value of `0.9` and is therefore correct, 90% of the time. This is also called the Specificity or True Negative Rate.\n",
    "\n",
    "The Bayes formula is as follows:\n",
    "\n",
    "<img src=\"images/bayes_formula.png\" height=\"242\" width=\"242\">\n",
    "\n",
    "* `P(A)` is the prior probability of A occurring independently. In our example this is `P(D)`. This value is given to us.\n",
    "\n",
    "* `P(B)` is the prior probability of B occurring independently. In our example this is `P(Pos)`.\n",
    "\n",
    "* `P(A|B)` is the posterior probability that A occurs given B. In our example this is `P(D|Pos)`. That is, **the probability of an individual having diabetes, given that this individual got a positive test result. This is the value that we are looking to calculate.**\n",
    "\n",
    "* `P(B|A)` is the prior probability of B occurring, given A. In our example this is `P(Pos|D)`. This value is given to us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting our values into the formula for Bayes theorem we get:\n",
    "\n",
    "`P(D|Pos) = P(D) * P(Pos|D) / P(Pos)`\n",
    "\n",
    "The probability of getting a positive test result `P(Pos)` can be calculated using the Sensitivity and Specificity as follows:\n",
    "\n",
    "`P(Pos) = [P(D) * Sensitivity] + [P(~D) * (1-Specificity))]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nInstructions:\\nCalculate probability of getting a positive test result, P(Pos)\\n'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Calculate probability of getting a positive test result, P(Pos)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of getting a positive test result P(Pos) is: 0.10799999999999998\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution (skeleton code will be provided)\n",
    "'''\n",
    "# P(D)\n",
    "p_diabetes = 0.01\n",
    "\n",
    "# P(~D)\n",
    "p_no_diabetes = 0.99\n",
    "\n",
    "# Sensitivity or P(Pos|D)\n",
    "p_pos_diabetes = 0.9\n",
    "\n",
    "# Specificity or P(Neg|~D)\n",
    "p_neg_no_diabetes = 0.9\n",
    "\n",
    "# P(Pos)\n",
    "p_pos = p_diabetes * p_pos_diabetes + p_no_diabetes * (1 - p_neg_no_diabetes)\n",
    "print('The probability of getting a positive test result P(Pos) is:',format(p_pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Using all of this information we can calculate our posteriors as follows:**\n",
    "    \n",
    "The probability of an individual having diabetes, given that, that individual got a positive test result:\n",
    "\n",
    "`P(D|Pos) = (P(D) * Sensitivity)) / P(Pos)`\n",
    "\n",
    "The probability of an individual not having diabetes, given that, that individual got a positive test result:\n",
    "\n",
    "`P(~D|Pos) = (P(~D) * (1-Specificity)) / P(Pos)`\n",
    "\n",
    "The sum of our posteriors will always equal `1`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nInstructions:\\nCompute the probability of an individual having diabetes, given that, that individual got a positive test result.\\nIn other words, compute P(D|Pos).\\n\\nThe formula is: P(D|Pos) = (P(D) * P(Pos|D)) / P(Pos)\\n'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Compute the probability of an individual having diabetes, given that, that individual got a positive test result.\n",
    "In other words, compute P(D|Pos).\n",
    "\n",
    "The formula is: P(D|Pos) = (P(D) * P(Pos|D)) / P(Pos)\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of an individual having diabetes, given that that individual got a positive test result is: 0.08333333333333336\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "# P(D|Pos)\n",
    "p_diabetes_pos = (p_diabetes * p_pos_diabetes) / p_pos\n",
    "print('Probability of an individual having diabetes, given that that individual got a positive test result is:\\\n",
    "',format(p_diabetes_pos)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nInstructions:\\nCompute the probability of an individual not having diabetes, given that, that individual got a positive test result.\\nIn other words, compute P(~D|Pos).\\n\\nThe formula is: P(~D|Pos) = P(~D) * P(Pos|~D) / P(Pos)\\n\\nNote that P(Pos|~D) can be computed as 1 - P(Neg|~D). \\n\\nTherefore:\\nP(Pos|~D) = p_pos_no_diabetes = 1 - 0.9 = 0.1\\n'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Compute the probability of an individual not having diabetes, given that, that individual got a positive test result.\n",
    "In other words, compute P(~D|Pos).\n",
    "\n",
    "The formula is: P(~D|Pos) = P(~D) * P(Pos|~D) / P(Pos)\n",
    "\n",
    "Note that P(Pos|~D) can be computed as 1 - P(Neg|~D). \n",
    "\n",
    "Therefore:\n",
    "P(Pos|~D) = p_pos_no_diabetes = 1 - 0.9 = 0.1\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of an individual not having diabetes, given that that individual got a positive test result is: 0.9166666666666669\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "# P(Pos|~D)\n",
    "p_pos_no_diabetes = 0.1\n",
    "\n",
    "# P(~D|Pos)\n",
    "p_no_diabetes_pos = p_no_diabetes * p_pos_no_diabetes / p_pos\n",
    "print('Probability of an individual not having diabetes, given that that individual got a positive test result is:'\\\n",
    ",p_no_diabetes_pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have implemented Bayes Theorem from scratch. Your analysis shows that even if you get a positive test result, there is only an 8.3% chance that you actually have diabetes and a 91.67% chance that you do not have diabetes. This is of course assuming that only 1% of the entire population has diabetes which is only an assumption."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What does the term 'Naive' in 'Naive Bayes' mean ?** \n",
    "\n",
    "The term 'Naive' in Naive Bayes comes from the fact that the algorithm considers the features that it is using to make the predictions to be independent of each other, which may not always be the case. So in our Diabetes example, we are considering only one feature, that is the test result. Say we added another feature, 'exercise'. Let's say this feature has a binary value of `0` and `1`, where the former signifies that the individual exercises less than or equal to 2 days a week and the latter signifies that the individual exercises greater than or equal to 3 days a week. If we had to use both of these features, namely the test result and the value of the 'exercise' feature, to compute our final probabilities, Bayes' theorem would fail. Naive Bayes' is an extension of Bayes' theorem that assumes that all the features are independent of each other. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4.2: Naive Bayes implementation from scratch ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have understood the ins and outs of Bayes Theorem, we will extend it to consider cases where we have more than one feature. \n",
    "\n",
    "Let's say that we have two political parties' candidates, 'Jill Stein' of the Green Party and 'Gary Johnson' of the Libertarian Party and we have the probabilities of each of these candidates saying the words 'freedom', 'immigration' and 'environment' when they give a speech:\n",
    "\n",
    "* Probability that Jill Stein says 'freedom': 0.1 ---------> `P(F|J)`\n",
    "* Probability that Jill Stein says 'immigration': 0.1 -----> `P(I|J)`\n",
    "* Probability that Jill Stein says 'environment': 0.8 -----> `P(E|J)`\n",
    "\n",
    "\n",
    "* Probability that Gary Johnson says 'freedom': 0.7 -------> `P(F|G)`\n",
    "* Probability that Gary Johnson says 'immigration': 0.2 ---> `P(I|G)`\n",
    "* Probability that Gary Johnson says 'environment': 0.1 ---> `P(E|G)`\n",
    "\n",
    "\n",
    "And let us also assume that the probability of Jill Stein giving a speech, `P(J)` is `0.5` and the same for Gary Johnson, `P(G) = 0.5`. \n",
    "\n",
    "\n",
    "Given this, what if we had to find the probabilities of Jill Stein saying the words 'freedom' and 'immigration'? This is where the Naive Bayes' theorem comes into play as we are considering two features, 'freedom' and 'immigration'.\n",
    "\n",
    "Now we are at a place where we can define the formula for the Naive Bayes' theorem:\n",
    "\n",
    "<img src=\"images/naivebayes.png\" height=\"342\" width=\"342\">\n",
    "\n",
    "Here, `y` is the class variable (in our case the name of the candidate) and `x1` through `xn` are the feature vectors (in our case the individual words). The theorem makes the assumption that each of the feature vectors or words (`xi`) are independent of each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To break this down, we have to compute the following posterior probabilities:\n",
    "\n",
    "* `P(J|F,I)`: Given the words 'freedom' and 'immigration' were said, what's the probability they were said by Jill?\n",
    "\n",
    "    Using the formula and our knowledge of Bayes' theorem, we can compute this as follows: `P(J|F,I)` = `(P(J) * P(F|J) * P(I|J)) / P(F,I)`. Here `P(F,I)` is the probability of the words 'freedom' and 'immigration' being said in a speech.\n",
    "    \n",
    "\n",
    "* `P(G|F,I)`: Given the words 'freedom' and 'immigration' were said, what's the probability they were said by Gary?\n",
    "    \n",
    "    Using the formula, we can compute this as follows: `P(G|F,I)` = `(P(G) * P(F|G) * P(I|G)) / P(F,I)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nInstructions: Compute the probability of the words 'freedom' and 'immigration' being said in a speech, or\\nP(F,I).\\n\\nThe first step is multiplying the probabilities of Jill Stein giving a speech with her individual \\nprobabilities of saying the words 'freedom' and 'immigration'. Store this in a variable called p_j_text.\\n\\nThe second step is multiplying the probabilities of Gary Johnson giving a speech with his individual \\nprobabilities of saying the words 'freedom' and 'immigration'. Store this in a variable called p_g_text.\\n\\nThe third step is to add both of these probabilities and you will get P(F,I).\\n\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions: Compute the probability of the words 'freedom' and 'immigration' being said in a speech, or\n",
    "P(F,I).\n",
    "\n",
    "The first step is multiplying the probabilities of Jill Stein giving a speech with her individual \n",
    "probabilities of saying the words 'freedom' and 'immigration'. Store this in a variable called p_j_text.\n",
    "\n",
    "The second step is multiplying the probabilities of Gary Johnson giving a speech with his individual \n",
    "probabilities of saying the words 'freedom' and 'immigration'. Store this in a variable called p_g_text.\n",
    "\n",
    "The third step is to add both of these probabilities and you will get P(F,I).\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.005000000000000001\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution: Step 1\n",
    "'''\n",
    "# P(J)\n",
    "p_j = 0.5\n",
    "\n",
    "# P(F/J)\n",
    "p_j_f = 0.1\n",
    "\n",
    "# P(I/J)\n",
    "p_j_i = 0.1\n",
    "\n",
    "p_j_text = p_j *  p_j_f * p_j_i\n",
    "print(p_j_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.06999999999999999\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution: Step 2\n",
    "'''\n",
    "# P(G)\n",
    "p_g = 0.5\n",
    "\n",
    "# P(F/G)\n",
    "p_g_f = 0.7\n",
    "\n",
    "# P(I/G)\n",
    "p_g_i = 0.2\n",
    "\n",
    "p_g_text = p_g * p_g_f * p_g_i\n",
    "print(p_g_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability of words freedom and immigration being said are:  0.075\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution: Step 3: Compute P(F,I) and store in p_f_i\n",
    "'''\n",
    "p_f_i = p_j_text + p_g_text\n",
    "print('Probability of words freedom and immigration being said are: ', format(p_f_i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compute the probability of `P(J|F,I)`, the probability of Jill Stein saying the words 'freedom' and 'immigration' and `P(G|F,I)`, the probability of Gary Johnson saying the words 'freedom' and 'immigration'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nInstructions:\\nCompute P(J|F,I) using the formula P(J|F,I) = (P(J) * P(F|J) * P(I|J)) / P(F,I) and store it in a variable p_j_fi\\n'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Compute P(J|F,I) using the formula P(J|F,I) = (P(J) * P(F|J) * P(I|J)) / P(F,I) and store it in a variable p_j_fi\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of Jill Stein saying the words Freedom and Immigration:  0.06666666666666668\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "p_j_fi = (p_j * p_j_f * p_j_i) / p_f_i\n",
    "print('The probability of Jill Stein saying the words Freedom and Immigration: ', format(p_j_fi))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nInstructions:\\nCompute P(G|F,I) using the formula P(G|F,I) = (P(G) * P(F|G) * P(I|G)) / P(F,I) and store it in a variable p_g_fi\\n'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Compute P(G|F,I) using the formula P(G|F,I) = (P(G) * P(F|G) * P(I|G)) / P(F,I) and store it in a variable p_g_fi\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of Gary Johnson saying the words Freedom and Immigration:  0.9333333333333332\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "p_g_fi = (p_g * p_g_f * p_g_i) / p_f_i\n",
    "print('The probability of Gary Johnson saying the words Freedom and Immigration: ', format(p_g_fi))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And as we can see, just like in the Bayes' theorem case, the sum of our posteriors is equal to 1. \n",
    "\n",
    "Congratulations! You have implemented the Naive Bayes' theorem from scratch. Our analysis shows that there is only a 6.6% chance that Jill Stein of the Green Party uses the words 'freedom' and 'immigration' in her speech as compared with the 93.3% chance for Gary Johnson of the Libertarian party."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For another example of Naive Bayes, let's consider searching for images using the term 'Sacramento Kings' in a search engine. In order for us to get the results pertaining to the Scramento Kings NBA basketball team, the search engine needs to be able to associate the two words together and not treat them individually. If the search engine only searched for the words individually, we would get results of images tagged with 'Sacramento,' like pictures of city landscapes, and images of 'Kings,' which might be pictures of crowns or kings from history. But associating the two terms together would produce images of the basketball team. In the first approach we would treat the words as independent entities, so it would be considered 'naive.' We don't usually want this approach from a search engine, but it can be extremely useful in other cases. \n",
    "\n",
    "\n",
    "Applying this to our problem of classifying messages as spam, the Naive Bayes algorithm *looks at each word individually and not as associated entities* with any kind of link between them. In the case of spam detectors, this usually works, as there are certain red flag words in an email which are highly reliable in classifying it as spam. For example, emails with words like 'viagra' are usually classified as spam."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Naive Bayes implementation using scikit-learn ###\n",
    "\n",
    "Now let's return to our spam classification context. Thankfully, sklearn has several Naive Bayes implementations that we can use, so we do not have to do the math from scratch. We will be using sklearn's `sklearn.naive_bayes` method to make predictions on our SMS messages dataset. \n",
    "\n",
    "Specifically, we will be using the multinomial Naive Bayes algorithm. This particular classifier is suitable for classification with discrete features (such as in our case, word counts for text classification). It takes in integer word counts as its input. On the other hand, Gaussian Naive Bayes is better suited for continuous data as it assumes that the input data has a Gaussian (normal) distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nInstructions:\\n\\nWe have loaded the training data into the variable 'training_data' and the testing data into the \\nvariable 'testing_data'.\\n\\nImport the MultinomialNB classifier and fit the training data into the classifier using fit(). Name your classifier\\n'naive_bayes'. You will be training the classifier using 'training_data' and 'y_train' from our split earlier. \\n\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "\n",
    "We have loaded the training data into the variable 'training_data' and the testing data into the \n",
    "variable 'testing_data'.\n",
    "\n",
    "Import the MultinomialNB classifier and fit the training data into the classifier using fit(). Name your classifier\n",
    "'naive_bayes'. You will be training the classifier using 'training_data' and 'y_train' from our split earlier. \n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "naive_bayes = MultinomialNB()\n",
    "\n",
    "naive_bayes.fit(training_data, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nInstructions:\\nNow that our algorithm has been trained using the training data set we can now make some predictions on the test data\\nstored in 'testing_data' using predict(). Save your predictions into the 'predictions' variable.\\n\""
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Now that our algorithm has been trained using the training data set we can now make some predictions on the test data\n",
    "stored in 'testing_data' using predict(). Save your predictions into the 'predictions' variable.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "predictions = naive_bayes.predict(testing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that predictions have been made on our test set, we need to check the accuracy of our predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Evaluating our model ###\n",
    "\n",
    "Now that we have made predictions on our test set, our next goal is to evaluate how well our model is doing. There are various mechanisms for doing so, so first let's review them.\n",
    "\n",
    "**Accuracy** measures how often the classifier makes the correct prediction. It’s the ratio of the number of correct predictions to the total number of predictions (the number of test data points).\n",
    "\n",
    "**Precision** tells us what proportion of messages we classified as spam, actually were spam.\n",
    "It is a ratio of true positives (words classified as spam, and which actually are spam) to all positives (all words classified as spam, regardless of whether that was the correct classification). In other words, precision is the ratio of\n",
    "\n",
    "`[True Positives/(True Positives + False Positives)]`\n",
    "\n",
    "**Recall (sensitivity)** tells us what proportion of messages that actually were spam were classified by us as spam.\n",
    "It is a ratio of true positives (words classified as spam, and which actually are spam) to all the words that were actually spam. In other words, recall is the ratio of\n",
    "\n",
    "`[True Positives/(True Positives + False Negatives)]`\n",
    "\n",
    "For classification problems that are skewed in their classification distributions like in our case - for example if we had 100 text messages and only 2 were spam and the other 98 weren't - accuracy by itself is not a very good metric. We could classify 90 messages as not spam (including the 2 that were spam but we classify them as not spam, hence they would be false negatives) and 10 as spam (all 10 false positives) and still get a reasonably good accuracy score. For such cases, precision and recall come in very handy. These two metrics can be combined to get the **F1 score**, which is the weighted average of the precision and recall scores. This score can range from 0 to 1, with 1 being the best possible F1 score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using all 4 of these metrics to make sure our model does well. For all 4 metrics whose values can range from 0 to 1, having a score as close to 1 as possible is a good indicator of how well our model is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Instructions:\n",
    "Compute the accuracy, precision, recall and F1 scores of your model using your test data 'y_test' and the predictions\n",
    "you made earlier stored in the 'predictions' variable.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score:  0.9885139985642498\n",
      "Precision score:  0.9720670391061452\n",
      "Recall score:  0.9405405405405406\n",
      "F1 score:  0.9560439560439562\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Solution\n",
    "'''\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "print('Accuracy score: ', format(accuracy_score(y_test, predictions)))\n",
    "print('Precision score: ', format(precision_score(y_test, predictions)))\n",
    "print('Recall score: ', format(recall_score(y_test, predictions)))\n",
    "print('F1 score: ', format(f1_score(y_test, predictions)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Step 7: Conclusion ###\n",
    "\n",
    "One of the major advantages that Naive Bayes has over other classification algorithms is its ability to handle an extremely large number of features. In our case, each word is treated as a feature and there are thousands of different words. Also, it performs well even with the presence of irrelevant features and is relatively unaffected by them. The other major advantage it has is its relative simplicity. Naive Bayes' works well right out of the box and tuning its parameters is rarely ever necessary, except usually in cases where the distribution of the data is known. \n",
    "It rarely ever overfits the data. Another important advantage is that its model training and prediction times are very fast for the amount of data it can handle. All in all, Naive Bayes' really is a gem of an algorithm!\n",
    "\n",
    "Congratulations! You have successfully designed a model that can efficiently predict if an SMS message is spam or not!\n",
    "\n",
    "Thank you for learning with us!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}