FTE_Week3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 3 - More on Tidy Data and Related Python Topics\n",
    "\n",
    "Week 3 reading: **Pandas for Everyone** chapter 6 (pages 124 - 141)\n",
    "\n",
    "Outline:\n",
    "1. Review of Tidy Data criteria\n",
    "2. Python string tutorial (supporting ch. 6.3.1)\n",
    "3. Python iterator tutorial (supporting ch. 6.3.3)\n",
    "4. Python zip tutorial (supporting ch. 6.3.3)\n",
    "5. Python tuple tutorial (supporting ch. 6.3.3)\n",
    "6. Using the requests module to gather multi-file data (alternative to ch. 6.6 method)\n",
    "    1. Wildcard and glob tutorial (supporting ch. 6.6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Review of Tidy Data criteria\n",
    "\n",
    "Last week we briefly discussed Hadley Wickham's 2014 article in the *Journal of Statistical Software* and what he termed \"tidy data\" as a framework for organizing data for analysis. The paper defines *tidy data* as fitting the following criteria:\n",
    "\n",
    "* Each row is an observation\n",
    "* Each column is a variable\n",
    "* Each type of observational unit forms a table\n",
    "\n",
    "Chapter 6 of **Pandas for Everyone** describes a variety of ways data can be grouped based on how it is collected or displayed that do **not** conform to tidy criteria, such as \"wide data format\" shown in section 6.2.1 and how to \"melt\" the data set into a more analysis-friendly format.\n",
    "\n",
    "Another item of note shown in 6.2.1 is to name the melted columns to help with identifying data and to make later analysis easier.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "<hr>\n",
    "As it is relatively easy to load the data and follow along with the book, the rest of this week's material will provide tutorials that further explain or enhance techniques shown in the book, or in some cases show alternate techniques for reasons described in the section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Python string tutorial (supporting ch. 6.3.1)\n",
    "\n",
    "**Section 6.3.1** describes splitting columns based on some delimiter, such as an underscore (\\_) or a comma (,). Let's take a look at how Python sees and manipulates strings to get a better understanding of what is going on. \n",
    "\n",
    "We saw in **week 1** how a Python list is a container for objects, each object individually addressable with an index. You may recall we had a list of names and could display any one of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Cheryl'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = [\"Bob\",\"Cheryl\",\"Dave\",\"Jenny\", \"Waldo\"]\n",
    "names[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python looks at strings as if they are lists of characters. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'v'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_string = 'I love Python'\n",
    "py_string[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Strings can be *concatenated*, or added to, with a simple \"+\" operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I love Python programming!'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_string + ' programming!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do some nifty functions like counting characters, slicing, reversing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "love\n",
      "nohtyP evol I\n"
     ]
    }
   ],
   "source": [
    "print( py_string.count('o') )  # Count the number of 'm' characters\n",
    "\n",
    "print( py_string[2:6] ) # Show characters 2 through 5\n",
    "\n",
    "print( py_string[::-1] ) # Display string backwards (with a slicing trick)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even split the string on some character of choice. This will give us back a **list**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', 'love', 'Python']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split string at each space character. \n",
    "#Notice the space is NOT included in the list.\n",
    "\n",
    "py_string.split(' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists of strings can also be re-joined with your choice of separating character. This is one method for making a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I,love,Python'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_list = py_string.split(' ')\n",
    "\n",
    "','.join(py_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The join syntax deserves a little explanation. In Python, everything is an object, and objects have functions (methods) that can operate on the object. Even an explicit string like\n",
    "\n",
    "`','` is an object and the `join()` method says to use its string (the ',' in this case) to separate the strings in the list. We could have put the list back together with spaces like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I love Python'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(py_list) # Notice there is a <space> character between the single quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I love Python'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double quotes work, too\n",
    "\n",
    "\" \".join(py_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This demonstration only scratches the surface of what can be done with strings. We will discuss more properties of strings in the sections on iterators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Python iterator tutorial (supporting ch. 6.3.3)\n",
    "\n",
    "Another great feature of lists is that they function as containers that we can step through (called *iteration*). \n",
    "\n",
    "In Python, the `for` loop was made for iterating containers like lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bob\n",
      "Cheryl\n",
      "Dave\n",
      "Jenny\n",
      "Waldo\n"
     ]
    }
   ],
   "source": [
    "for name in names:\n",
    "    print(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general syntax to *iterate* a for loop is:\n",
    "\n",
    "```\n",
    "for <item variable> in <container variable>:\n",
    "    use item variable\n",
    "```\n",
    "Really, the Python `for` loop functions like a *for_each* loop in other languages. In fact, thinking about for loops is easier if you concieve of it like this:\n",
    "\n",
    "```\n",
    "for-each <item> in <container> do:\n",
    "    operation using item\n",
    "```\n",
    "\n",
    "Let's test that out. We can make each name lower case. **Conceptually**:\n",
    "\n",
    "```\n",
    "for-each name in names do:\n",
    "    print(name.lower())\n",
    "```\n",
    "\n",
    "Actual Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bob\n",
      "cheryl\n",
      "dave\n",
      "jenny\n",
      "waldo\n"
     ]
    }
   ],
   "source": [
    "# name is the item and names is the container\n",
    "for name in names:       \n",
    "    print(name.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formally:\n",
    "\n",
    "* __iteration__ : The action of using a loop to apply some code to every item in a container.\n",
    "* __iterable__ : The container holding the items of interest.\n",
    "* __iterator__ : The variable containing each item of interest from the container.\n",
    "\n",
    "Not coincidentally, Python views strings as iterable containers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I\n",
      " \n",
      "l\n",
      "o\n",
      "v\n",
      "e\n",
      " \n",
      "P\n",
      "y\n",
      "t\n",
      "h\n",
      "o\n",
      "n\n"
     ]
    }
   ],
   "source": [
    "# ch is just a variable that I think makes a good\n",
    "# abbreviation for \"character.\"\n",
    "\n",
    "for ch in py_string:  \n",
    "    print(ch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That fact probably isn't useful at this particular time, but it is handy to know and could come up later, during some text processing task.\n",
    "\n",
    "A trick that is particularly useful is to use the `enumerate()` function to keep a count of loop iterations. We have to have an extra variable along with our item variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 I\n",
      "1  \n",
      "2 l\n",
      "3 o\n",
      "4 v\n",
      "5 e\n",
      "6  \n",
      "7 P\n",
      "8 y\n",
      "9 t\n",
      "10 h\n",
      "11 o\n",
      "12 n\n"
     ]
    }
   ],
   "source": [
    "for indx, ch in enumerate(py_string):\n",
    "    print(indx, ch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see that trick again below when we only want to read the first 5 lines of a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Python zip tutorial (supporting ch. 6.3.3)\n",
    "\n",
    "Section 6.3.3 on page 132 mentions the `zip()` function and gives a minimal demonstration of its functionality. `zip()` is a built-in function in Python. If you were to consult the documentation, it would say something like:\n",
    "\n",
    ">Zip takes zero or more iterables as arguments and returns an iterator to tuples matched from the indexes of the inputs.\n",
    "\n",
    "Since it is somewhat silly (though possible) to pass an empty list or a single list to zip, we will pass over those trivial cases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'zip'>\n",
      "<zip object at 0x10f17aa88>\n"
     ]
    }
   ],
   "source": [
    "list1 = ['red','green','blue']\n",
    "list2 = ['apple', 'grape', 'berry']\n",
    "\n",
    "z = zip(list1,list2)\n",
    "\n",
    "print(type(z))\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, you can see that `z` is a zip object, but what good does that do us? Well, first of all, we can iterate on `z`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('red', 'apple')\n",
      "('green', 'grape')\n",
      "('blue', 'berry')\n"
     ]
    }
   ],
   "source": [
    "for i in z:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, after we have iterated on our zip object, it is empty. Let's try to turn `z` into a list and print it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "list3 = list(z)\n",
    "print(list3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('red', 'apple'), ('green', 'grape'), ('blue', 'berry')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = zip(list1, list2)\n",
    "list(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and assign that list to a variable so we can work with it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zlist = list(z)\n",
    "zlist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoops! That \"drained\" our iterator too! \n",
    "\n",
    "Let's try again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('red', 'apple'), ('green', 'grape'), ('blue', 'berry')]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = zip(list1, list2)\n",
    "zlist = list(z)\n",
    "zlist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, each list item is a **tuple** (from a generalization of single, double, triple, quadruple, quintuple, sextuple, ..., etc). We will look at tuples in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Python tuple tutorial (supporting ch. 6.3.3)\n",
    "\n",
    "I have bad news and I have good news. Which do you want to hear first?\n",
    "\n",
    "Let's start with the bad news: **tuples** are *one more* Python data structure (container) you need to learn.\n",
    "\n",
    "Now the good news: **Tuples** are identical to lists, except for one detail -- **tuples are immutable**, which is a computer science-y way of saying **_they can't be changed after they are created_**.\n",
    "\n",
    "Let's look at a couple of examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('red', 'green', 'blue', 'cyan', 'magenta')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tple_1 = ('red', 'green', 'blue', 'cyan', 'magenta')\n",
    "tple_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'red'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tple_1[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tple_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('magenta', 'cyan', 'blue', 'green', 'red')"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tple_1[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'tuple' object does not support item assignment",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-28-de2ba51eefcf>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtple_1\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'pink'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'tuple' object does not support item assignment"
     ]
    }
   ],
   "source": [
    "tple_1[0] = 'pink'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoops! There is the immutability I mentioned.\n",
    "\n",
    "What about our `zlist` object above, that is a list composed of tuples? How do we get to the individual elements of each tuple?\n",
    "\n",
    "The key lies in understanding that each tuple has indexed elements, and in turn, each tuple is an indexed element of the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('red', 'apple')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zlist[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so a subscript returns a tuple. And we know that each item of a tuple has a subscript, so, let's just keep piling them on! \n",
    "\n",
    "Suppose I wanted the second item in the first tuple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'apple'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zlist[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('yellow', 'banana'), ('green', 'grape'), ('blue', 'berry')]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zlist[0] = ('yellow', 'banana')\n",
    "zlist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Using the requests module to gather multi-file data (alternative to ch. 6.6 method)\n",
    "\n",
    "Section 6 discusses the fairly common occurrence of data being spread across multiple files. The book uses the older **urllib** library to retrieve a list of URLs. It is much easier and more standard to use the **requests** library instead. we will look at requests-based retrieval as an alternative.\n",
    "\n",
    "First, though, I think we should look at the first few lines of the URL data file the book uses, to verify location and content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-01.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-02.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-03.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-04.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-05.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-06.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-07.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-08.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-09.csv\n",
      "https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2015-10.csv\n"
     ]
    }
   ],
   "source": [
    "!head ../pandas_for_everyone/data/raw_data_urls.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm also reversing the logic a bit. It isn't really common to use the `break` command to stop a loop. Better to say there are a group of steps to be performed if the count is < 5 (remember, 0 to 4 is five things)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading fhv_tripdata_2015-01.csv...\n",
      "Writing data/fhv_tripdata_2015-01.csv...\n",
      "\n",
      "\n",
      "Reading fhv_tripdata_2015-02.csv...\n",
      "Writing data/fhv_tripdata_2015-02.csv...\n",
      "\n",
      "\n",
      "Reading fhv_tripdata_2015-03.csv...\n",
      "Writing data/fhv_tripdata_2015-03.csv...\n",
      "\n",
      "\n",
      "Reading fhv_tripdata_2015-04.csv...\n",
      "Writing data/fhv_tripdata_2015-04.csv...\n",
      "\n",
      "\n",
      "Reading fhv_tripdata_2015-05.csv...\n",
      "Writing data/fhv_tripdata_2015-05.csv...\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "url_list = []  # new, empty list to hold URLs we get from the data file.\n",
    "data_dir = 'data/'\n",
    "\n",
    "with open('../pandas_for_everyone/data/raw_data_urls.txt', 'r') as infile:\n",
    "    for indx, url in enumerate(infile):\n",
    "        if indx < 5:\n",
    "            url = url.strip()\n",
    "            filename = url.split('/')[-1]\n",
    "            print(f'Reading {filename}...')\n",
    "            response = requests.get(url)\n",
    "            new_file = data_dir + filename\n",
    "            print(f'Writing {new_file}...\\n\\n')\n",
    "            with open(new_file, 'w') as outfile:\n",
    "                outfile.write(response.text)\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those are pretty big files and a normal text editor may have some trouble opening them to verify. Let's fall back on the Linux head command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dispatching_base_num,Pickup_date,locationID\n",
      "B00013,2015-01-01 00:30:00,\n",
      "B00013,2015-01-01 01:22:00,\n",
      "B00013,2015-01-01 01:23:00,\n",
      "B00013,2015-01-01 01:44:00,\n",
      "B00013,2015-01-01 02:00:00,\n",
      "B00013,2015-01-01 02:00:00,\n",
      "B00013,2015-01-01 02:00:00,\n",
      "B00013,2015-01-01 02:50:00,\n",
      "B00013,2015-01-01 04:45:00,\n"
     ]
    }
   ],
   "source": [
    "!head data/fhv_tripdata_2015-01.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dispatching_base_num,Pickup_date,locationID\n",
      "B00013,2015-02-01 00:00:00,\n",
      "B00013,2015-02-01 00:01:00,\n",
      "B00013,2015-02-01 00:21:00,\n",
      "B00013,2015-02-01 01:00:00,\n",
      "B00013,2015-02-01 02:10:00,\n",
      "B00013,2015-02-01 03:34:00,\n",
      "B00013,2015-02-01 03:37:00,\n",
      "B00013,2015-02-01 03:39:00,\n",
      "B00013,2015-02-01 03:42:00,\n"
     ]
    }
   ],
   "source": [
    "!head data/fhv_tripdata_2015-02.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1. Wildcard and glob tutorial (supporting ch. 6.6)\n",
    "\n",
    "Page 138 uses the oddly-named `glob` library to take advantage of the sequential naming of the data files. The `glob` library uses the asterisk ('__\\*__', also known as a star or a splat) to return a list of all files whose names match a pattern.\n",
    "\n",
    "In this case, the book shows a filename of `fhv_*`. The splat means 'match anything' so giving `fhv_*` to `glob` returns a list of all files whose names start with the pattern fhv_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Alternative merge method (supporting ch. 6.6.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}