FTE_Week4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 4 Data Munging - Data Types, Strings and Text Data \n",
    "\n",
    "Week 4 reading: **Pandas for Everyone** chapters 7,8 (pages 146 - 170)\n",
    "\n",
    "Outline:\n",
    "\n",
    "* Chapter 7 - Data Types\n",
    "    1. Data types\n",
    "        * How computers use binary and what does 64-bit mean?\n",
    "        * 16, 32, 64 bit? Why?\n",
    "    2. Numbers vs. characters\n",
    "* Chapter 8 - Strings and Text Data\n",
    "    1. Regular expressions\n",
    "    \n",
    "## Overview\n",
    "\n",
    "Python, and particularly the NumPy library, recognizes and can work with a wide variety of data types, but in the end, it all really comes down to 3 types: **integer, floating point, and string**.\n",
    "\n",
    "Pandas, utilizing NumPy arrays for storage, sees data as 64-bit integers (int64), 64-bit floating point (float64), or string (object). **Pandas for Everyone** does a great job of discussing how to convert from one type to another (chapter 7) as well as a more in-depth look at string slicing, functions that operate on strings, formatting, and pattern matching using \"regular expressions.\" \n",
    "\n",
    "As with last week, this week's From The Expert will mainly serve to highlight important concepts from the book. Students are encouraged to read the book and follow along with the examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data Types\n",
    "\n",
    "### How computers use binary and what does 64-bit mean?\n",
    "\n",
    "Computers use binary (`0 and 1` or `off and on`) at the lowest level to store everything -- integers, floating point numbers, strings, etc. This is done primarily because transistors (the building blocks of electronic circuits) function as on/off switches. RAM is essentially just large square matrices of transistors, each one in an *off* or *on* state, plus some logic circuits that can read those states and report on it. \n",
    "\n",
    "Data moving from RAM to CPU or hard drive to RAM is represented as electrical voltages on a wire. 0 volts means \"off\" and 5 volts (for example) represents \"on\". The voltage on the wire (data line) will stay high or low for a prescribed amount of time to distinguish two 0s or 1s from just a very long pulse.\n",
    "\n",
    "Now, let's say we have a sequence of 4 bits -- 0110 -- that we need to transmit from memory to the CPU. Given that each high or low pulse must be maintained for some length of time (to make the math easy we will use 1 second), the data transmission would look like this:\n",
    "\n",
    "0 (1 sec.)<br>\n",
    "-<br>\n",
    "1 (1 sec.)<br>\n",
    "-<br>\n",
    "1 (1 sec.)<br>\n",
    "-<br>\n",
    "0 (1 sec.)<br>\n",
    "\n",
    "This is called \"serial transmission\" and it should be fairly apparent that 4 bits would take 4 seconds. But, what would happen if we could have four of those wires, each one carrying a bit?\n",
    "\n",
    "0 | 1 | 1 | 0 (1 sec.)\n",
    "\n",
    "Four bits would only take 1 second, and then another four bits could be sent the next second, thus quadrupling the amount of data that can be moved in one time unit. If this represented 4 data lines from RAM to the CPU, it would be a **4 bit computer**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16, 32, 64 bit? Why?\n",
    "\n",
    "The computer **does not** know what an 'A' is. Similarly, it does not know what 'か' is or '食' or any other human language. It knows '0' and '1' and somehow humans must find a clever way to make a bunch of 0s and 1s into an 'A'. \n",
    "\n",
    "Originally, this clever way was called ASCII (American Standard Code for Information Interchange) which used 7 bits to encode the English alphabet (upper and lower case) plus numbers and some symbols. \n",
    "\n",
    "**CAUTION: MATH AHEAD**\n",
    "\n",
    "Now, since $2^7=128$ we could represent 128 characters with ASCII on our 8-bit computers. \n",
    "\n",
    "Letter|ASCII|Binary\n",
    "------|-----|------\n",
    "A | 65 | 01000001\n",
    "B | 66 | 01000010\n",
    "C | 67 | 01000011\n",
    "D | 68 | 01000100\n",
    "\n",
    "Going back to our hypothetical computer that takes 1 second per transmission, sending 'ABCD' would look like this:\n",
    "\n",
    "01000001 (1 sec.)<br>\n",
    "-<br>\n",
    "01000010 (1 sec.)<br>\n",
    "-<br>\n",
    "01000011 (1 sec.)<br>\n",
    "-<br>\n",
    "01000100 (1 sec.)<br>\n",
    "\n",
    "But, what if we had a 16-bit computer?\n",
    "\n",
    "01000001 | 01000010 (1 sec.)<br>\n",
    "-<br>\n",
    "01000011 | 01000100 (1 sec.)<br>\n",
    "\n",
    "Or 32-bits?\n",
    "\n",
    "01000001 | 01000010 | 01000011 | 01000100 (1 sec.)<br>\n",
    "\n",
    "Another benefit is the size of number that can be represented. \n",
    "\n",
    "We calculate the largest number by taking 2 possible states (on or off) and raise it to the power of data bits (8 bits):\n",
    "\n",
    "$$2^8=256$$\n",
    "\n",
    "But if we increase our data lines by the next power of 2 to 16, the largest integer we can represent is:\n",
    "\n",
    "$$2^{16}=65536$$\n",
    "\n",
    "Until only recently, Microsoft Windows was a 32-bit operating system, meaning the largest integer it could recognize as a chunk was:\n",
    "\n",
    "$$2^{32}=4294967296$$\n",
    "\n",
    "Now, Windows is 64-bit, and Pandas natively uses 64 bits for integers, so the largest integer value we can put in a Pandas int64 column is:\n",
    "\n",
    "$$2^{64}=18446744073709551616$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Numbers vs. characters\n",
    "\n",
    "A point of confusion for many students is the difference between a number and a text character of a number. To illustrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "num = 4\n",
    "char = '4'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the single quotes around the value in char. Let's see what happens if we try to do some math:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num + num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'44'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char + char"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'int' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-3afa5b8d070e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnum\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mchar\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'str'"
     ]
    }
   ],
   "source": [
    "num + char"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why did char + char print out 44? That is obviously the wrong answer.\n",
    "\n",
    "It happened because '4' is a string, and + for strings joins them together.\n",
    "\n",
    "<hr> \n",
    "\n",
    "Quick quiz:\n",
    "\n",
    "Which one of these is really a number?\n",
    "\n",
    "    A. Social Security Number\n",
    "    B. Phone number\n",
    "    C. Employee ID\n",
    "    D. Yearly salary\n",
    "    E. All of the above\n",
    "\n",
    "<hr>\n",
    "\n",
    "The ONLY right answer is D. \n",
    "\n",
    "D? Why D? \n",
    "\n",
    "Because D is the only one that you might want to do math with. Think about it a second -- does your SSN + my SSN have any meaning? Does your phone number have a square root? What would you use it for? And does 2 * employee_id mean s/he does twice the work or gets paid twice as much?\n",
    "\n",
    "Let's revisit our employee dataframe from Week 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "emp_df = pd.DataFrame({'emp_id':['1','2','3','4'], 'emp_name':['Tom', 'Mary','John', 'Tim'], 'dept_id':['1','2', '3', '1'] })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emp_id</th>\n",
       "      <th>emp_name</th>\n",
       "      <th>dept_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Tom</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Mary</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>John</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Tim</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  emp_id emp_name dept_id\n",
       "0      1      Tom       1\n",
       "1      2     Mary       2\n",
       "2      3     John       3\n",
       "3      4      Tim       1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emp_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Emp_id and dept_id both look like numbers, correct?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4 entries, 0 to 3\n",
      "Data columns (total 3 columns):\n",
      "emp_id      4 non-null object\n",
      "emp_name    4 non-null object\n",
      "dept_id     4 non-null object\n",
      "dtypes: object(3)\n",
      "memory usage: 176.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "emp_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas thinks they are strings. Does it matter? Not in the short term but many machine learning algorithms do not like strings.\n",
    "\n",
    "One final word on the topic. Dept_id looks like an integer, is classified as a string, but in reality it isn't either one. What is it? A category. You wouldn't do math with the number itself, but you may want to use it to group employees.\n",
    "\n",
    "**LOOK** at your data. **THINK** about what each variable *really means*. Does a dept_id of 3 mean there are three of them? Or maybe the employee has worked up from 1? Of course not. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8 - Strings and Text Data\n",
    "\n",
    "Hopefully, most of chapter 8 is review and easily understood. Once again, students are encouraged to work through the examples in the book on their own to make sure they understand string manipulation. \n",
    "\n",
    "The section that may cause confusion is regular expressions. Subsequently, that is where we will focus our attention next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Regular expressions\n",
    "\n",
    "There is a programmer's joke that goes\n",
    "> A programmer is trying to solve a problem and thinks to himself, \"I'll use regular expressions!\" <br>\n",
    "> Now he has **two** problems.\n",
    "\n",
    "Regular expressions come from the study of *formal languages* in theoretical computer science in the 1950s. Regex was built into the Unix operting system in the late 1960s as part of a text editor and has since come into common use by Unix/Linux system administrators as well as many programmers. Most, if not all, programming languages have some form of regular expression pattern matching built in or available as an add-on library. The Perl programming language makes particularly heavy use of regex.\n",
    "\n",
    "Regex is a particularly fast pattern matching engine, often used to find particular words or phrases in very large log files. It is also used to verify input patterns. For example, insuring telephone numbers or Social Security Numbers are entered in the proper format or that email addresses have an '@' symbol and a '.' in the latter section. \n",
    "\n",
    "*reference: https://en.wikipedia.org/wiki/Regular_expression*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions work on strings. The simplest form will match the pattern you specify exactly. For example if your regular expression is `Python`, it will **only** match `Python`. It will skip both `python` and `PYTHON`. Regex can be told to ignore case. This will be demonstrated later.\n",
    "\n",
    "Table 8.5 on Page 165 of the book shows **Basic RegEx Syntax**. Many sources will call these **metacharacters**, meaning characters used for control (meta - analyze at a higher level, thus meta-characters are characters used to analyze characters).\n",
    "\n",
    "When working with regex we need to import the `re` library and the function we often use is `search()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use a regex, we need two things:\n",
    "\n",
    "1. A pattern to match.\n",
    "2. A string to search for the pattern in.\n",
    "\n",
    "Let's start with the most basic case: finding a literal string inside another string. Note the use of a **raw string** for our pattern -- this tells regex to take the string as it is, don't try to interpret special characters like `\\`. We will also use the `group()` function to show the string matched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Python'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern = r'Python'\n",
    "strng = 'I love Python programming!'\n",
    "\n",
    "re.search(pattern, strng).group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not very exciting, I know. What would happen if the search didn't find the pattern?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'group'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-20-17dbbcaec782>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mre\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Functional programming rocks, too!'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'group'"
     ]
    }
   ],
   "source": [
    "re.search(pattern, 'Functional programming rocks, too!').group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.` is used to match any single character, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Python'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern = r'P.th.n'\n",
    "\n",
    "re.search(pattern, strng).group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The book demonstrates the `match()` function whereas we used `search()` above. What is the difference? `match()` only checks the beginning of the string but `search()` checks the entire string. For contrast:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'group'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-22-eca5fb3b8f97>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mre\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstrng\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'group'"
     ]
    }
   ],
   "source": [
    "re.match(pattern, strng).group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's take a look at a common usage for regex -- checking if user input matches a particular pattern.\n",
    "\n",
    "Imagine for a moment that we have a web page that is asking users to input their email address and we want to verify that the format of their response is correct. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Email address is properly formatted.\n"
     ]
    }
   ],
   "source": [
    "good_user_email = 'noname@example.com' # This is what we got from the user\n",
    "email_pattern = r'[\\w\\.-]+@[\\w\\.-]+'   # Check regex syntax in book for info\n",
    "\n",
    "valid_input = re.search(email_pattern, good_user_email)\n",
    "\n",
    "if (bool(valid_input)):    # bool() turns the search return into either a True or False\n",
    "    print('Email address is properly formatted.')\n",
    "else:\n",
    "    print('Email address is improperly formatted.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously we could have done more significant processing than just printing a message.\n",
    "\n",
    "Let's try a bad one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Email address is improperly formatted.\n"
     ]
    }
   ],
   "source": [
    "good_user_email = 'noname#example.com'    # Maybe they just fat-fingered the @\n",
    "email_pattern = r'[\\w\\.-]+@[\\w\\.-]+'  \n",
    "\n",
    "valid_input = re.search(email_pattern, good_user_email)\n",
    "\n",
    "if (bool(valid_input)):    \n",
    "    print('Email address is properly formatted.')\n",
    "else:\n",
    "    print('Email address is improperly formatted.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pattern above just does the most basic check and is usually sufficient. \n",
    "\n",
    "Just for fun, here is the General Email Regex as discussed in the RFC 5322 Official Standard.\n",
    "\n",
    "<code>\n",
    "(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?: (?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])\n",
    "</code><br>\n",
    "<br>\n",
    "BTW - you may remember that these \"From The Expert\" pages were generated from Jupyter notebooks, which uses regular expressions to parse the markdown (text) cells. Meaning it is incredibly difficult to put complex regex like above in a cell and have it formatted correctly!\n",
    "<br><br>\n",
    "Reference: https://emailregex.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}