FTE_Week6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 6 Munging Continued - Date Time Datatype and Integrative Lab "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Week 6 reading: **Pandas for Everyone** chapter 11 (pages 213 - 240)\n",
    "\n",
    "Outline: \n",
    "\n",
    "* Chapter 11 - Datetime\n",
    "    1. Python datetime calculations\n",
    "\n",
    "    \n",
    "* Lab\n",
    "\n",
    "\n",
    "\n",
    "## Overview\n",
    "\n",
    "Working with dates and times is difficult. \n",
    "\n",
    "**Quick quiz:** How many time zones does the United States have?\n",
    "1. 4 (Pacific, Mountain, etc.)\n",
    "2. 6 \n",
    "3. 8\n",
    "4. 11\n",
    "\n",
    "The reflex answer for most of us is #1 - four time zones (Pacific, Mountain, Central, Eastern), but that answer is wrong!\n",
    "\n",
    "From [timeanddate.com](https://www.timeanddate.com/time/zone/usa):\n",
    "\n",
    ">**How Many Time Zones Are There in the US?**\n",
    "\n",
    ">There are 9 time zones by law in the USA and its dependencies, however, adding the time zones of 2 uninhabited US territories gives **11 time zones in total** \\[emphasis mine\\]. The contiguous US has 4 standard time zones. In addition Alaska, Hawaii, and 5 US dependencies all have their own time zones. As neither Hawaii nor the 5 dependencies use Daylight Saving Time (DST), there are only 6 corresponding DST time zones.\n",
    "\n",
    "In general, times and time zones are a **nightmare** for programers. For example, a few years ago I worked for a company that created and sold software to hospitals for patient tracking and charting. With that thought in mind, consider this:\n",
    "\n",
    "1. Many, if not all, hospital orders (medications, therapy, lab tests & results) are **EXTREMELY** time sensitive. In many cases, **_life or death_**-type time sensitive.\n",
    "2. During the change to Daylight Savings time in the spring, **almost an entire hour doesn't exist** (from 2:01 a.m. to 2:59 a.m.). What happens to orders that were written for \"every four hours\" and it falls on 2:28 a.m.?\n",
    "3. During the change from DST to Standard Time, **almost an entire hour exists twice**.\n",
    "\n",
    "The answer at that time (more than a decade ago), was to literally _turn off the application_ for the time-change hour, go back to charting everything by hand, then have the clinicians enter all the charting later. \n",
    "\n",
    "Since **Pandas for Everyone** does a good job of highlighting Pandas datetime functionality, the remainder of this *From the Expert* will discuss datetime calculations using Python's native datetime library and then finish with topics pertinent to this week's lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Python datetime calculations\n",
    "\n",
    "The Python datetime library consists of five parts:\n",
    "\n",
    "* date\n",
    "* time\n",
    "* datetime\n",
    "* timedelta\n",
    "* tzinfo\n",
    "\n",
    "The first three are reasonably self-explanatory, but what about `timedelta` and `tzinfo`? The `timedelta` class is used for dealing with time duration between events and `tzinfo` is the basis for working with time zones.\n",
    "\n",
    "First thing we will do is import the classes we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date\n",
    "from datetime import time\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will read the Ebola data file into a list so that we can manipulate the date and day columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Date,Day,Cases_Guinea,Cases_Liberia,Cases_SierraLeone,Cases_Nigeria,Cases_Senegal,Cases_UnitedStates,Cases_Spain,Cases_Mali,Deaths_Guinea,Deaths_Liberia,Deaths_SierraLeone,Deaths_Nigeria,Deaths_Senegal,Deaths_UnitedStates,Deaths_Spain,Deaths_Mali\n",
      "\n",
      "1/5/2015,289,2776,,10030,,,,,,1786,,2977,,,,,\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ebola = []\n",
    "with open ('../pandas_for_everyone/data/country_timeseries.csv') as infile:\n",
    "    for line in infile:\n",
    "        ebola.append(line)\n",
    "\n",
    "# The first row is header information\n",
    "print(ebola[0])\n",
    "\n",
    "# The next line is actual data\n",
    "print(ebola[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, data loaded like this is just one big string. We have to use something like `split()` to get the element we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1/5/2015',\n",
       " '289',\n",
       " '2776',\n",
       " '',\n",
       " '10030',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '1786',\n",
       " '',\n",
       " '2977',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '',\n",
       " '\\n']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ebola[1].split(',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, from that split, we want the first (zeroth) element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1/5/2015'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remember, this line just explores \"what if.\" \n",
    "# If I want to save the value, I have to set it = to something.\n",
    "\n",
    "ebola[1].split(',')[0] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function that will convert this to a date is part of the `datetime` class called `strptime()`. This function takes as parameters a string and a **format**. These formatting strings are explained on page 216 of **Pandas for Everyone** and are a 'semi-standard' way to represent time and date string formats.\n",
    "\n",
    "Let's convert that string date above to a date object. Since we only have a date string to work with, we will chain a `date()` function on the end to give us what we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_string = ebola[1].split(',')[0] \n",
    "\n",
    "the_date = datetime.strptime(date_string, '%m/%d/%Y').date()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2015-01-05\n"
     ]
    }
   ],
   "source": [
    "print(the_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.date"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(the_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worth noting that on page 218, the book prints the type of an extracted date and gets a `Timestamp` object. In most cases, that `Timestamp` object is equivalent to Python's native `Datetime` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2015"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "the_date.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "the_date.month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "the_date.day"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even get the day number (0 = Monday, 6 = Sunday) and use it to convert to a text day name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Monday\n"
     ]
    }
   ],
   "source": [
    "day_names = {0 : 'Monday',\n",
    "            1: 'Tuesday',\n",
    "            2: 'Wednesday',\n",
    "            3: 'Thursday',\n",
    "            4: 'Friday',\n",
    "            5: 'Saturday',\n",
    "            6: 'Sunday'}\n",
    "\n",
    "\n",
    "day_number = the_date.weekday()\n",
    "print(day_number)\n",
    "print(day_names[day_number])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab Info\n",
    "\n",
    "TBD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}