Data_Exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eda_csv(frames, date_set, category_set):\n",
    "    '''\n",
    "    Explores data homogeneity across the frames provided. Data is explored across numeric data, datetime data, and categorical data. \n",
    "    This function displays the output in standard out, and is designed to be used in a python notebook\n",
    "\n",
    "    Args:\n",
    "        frames (iter)      : An iterator of pandas dataframes containing chunks of the data-set being analyzed\n",
    "        date_set (set)     : Contains strings of the column names of any datetime data\n",
    "        category_set (set) : Contains strings of the column names of any categorical data\n",
    "\n",
    "    Returns: None\n",
    "    '''\n",
    "    print('Taking a look at the data')\n",
    "    display(frames[0].head(15))\n",
    "    \n",
    "    print('\\n\\n')\n",
    "    print('Looking at data types')\n",
    "    display(pd.DataFrame(frames[0].dtypes).transpose().style.hide_index())\n",
    "\n",
    "    print('\\n\\n')\n",
    "    numeric_frames = [frame.drop(date_set.union(category_set), axis=1, inplace=False) for frame in frames]\n",
    "    date_frames = [frame[date_set].astype('datetime64') for frame in frames]\n",
    "    category_frames = [frame[category_set] for frame in frames]\n",
    "\n",
    "    print('Numeric Data Distribution for the First Frame')\n",
    "    display(numeric_frames[0].describe(percentiles=[0.5]))\n",
    "\n",
    "    print('\\n\\n')\n",
    "    print('Variance Percent in Numeric Data Distributions')\n",
    "\n",
    "    num_one = numeric_frames[0].describe(percentiles=[0.5])\n",
    "    num_two = numeric_frames[1].describe(percentiles=[0.5])\n",
    "\n",
    "    numeric_diff = abs(num_one-num_two)/abs(num_one)\n",
    "    display(numeric_diff.style.format('{:,.2%}', na_rep='Div_0'))\n",
    "\n",
    "    print('\\n\\n')\n",
    "    print('Date Data Distributions for the First Frame')\n",
    "    display(date_frames[0].astype('datetime64').describe(percentiles=[0.5], datetime_is_numeric=True))\n",
    "\n",
    "    print('\\n\\n')\n",
    "    print('Variance in Date Data Distributions')\n",
    "\n",
    "    date_one = date_frames[0].describe(percentiles=[0.5], datetime_is_numeric=True)\n",
    "    date_two = date_frames[1].describe(percentiles=[0.5], datetime_is_numeric=True)\n",
    "\n",
    "    date_diff = abs(date_one-date_two)\n",
    "    display(date_diff)\n",
    "\n",
    "    print('\\n\\n')\n",
    "    print('Categorical Data Value Counts for the First Frame')\n",
    "    for cat in category_set:\n",
    "        display(category_frames[0][cat].value_counts(dropna=False))\n",
    "        print('\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Homogeneous is the Data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Yellow Taxi Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "ytaxi_iter = pd.read_csv('D://Capstone Data//Yellow Taxi//yellowtaxi2014.csv', chunksize=250000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First Chunk Data\n",
    "first_c = next(ytaxi_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Second Chunk Data\n",
    "second_c = next(ytaxi_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taking a look at the data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vendor_id</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>imp_surcharge</th>\n",
       "      <th>rate_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-11-23T20:31:29.000</td>\n",
       "      <td>2014-11-23T20:31:29.000</td>\n",
       "      <td>3</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>N</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>CSH</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.00</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-04-16T13:45:06.000</td>\n",
       "      <td>2014-04-16T14:00:21.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.40</td>\n",
       "      <td>-73.990027</td>\n",
       "      <td>40.757197</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.958484</td>\n",
       "      <td>40.765540</td>\n",
       "      <td>CSH</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.50</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-02-09T22:43:54.000</td>\n",
       "      <td>2014-02-09T23:06:21.000</td>\n",
       "      <td>2</td>\n",
       "      <td>9.30</td>\n",
       "      <td>-73.870607</td>\n",
       "      <td>40.773682</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.971970</td>\n",
       "      <td>40.784683</td>\n",
       "      <td>CRD</td>\n",
       "      <td>27.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>6.76</td>\n",
       "      <td>5.33</td>\n",
       "      <td>40.59</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-10-27T23:15:23.000</td>\n",
       "      <td>2014-10-27T23:23:54.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.10</td>\n",
       "      <td>-73.983611</td>\n",
       "      <td>40.769135</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.974267</td>\n",
       "      <td>40.748188</td>\n",
       "      <td>CRD</td>\n",
       "      <td>8.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.37</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11.87</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>VTS</td>\n",
       "      <td>2014-09-14T01:32:00.000</td>\n",
       "      <td>2014-09-14T01:44:00.000</td>\n",
       "      <td>1</td>\n",
       "      <td>3.36</td>\n",
       "      <td>-73.949010</td>\n",
       "      <td>40.754087</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-73.960682</td>\n",
       "      <td>40.712792</td>\n",
       "      <td>CRD</td>\n",
       "      <td>12.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.60</td>\n",
       "      <td>0.00</td>\n",
       "      <td>16.10</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>VTS</td>\n",
       "      <td>2014-02-21T22:52:00.000</td>\n",
       "      <td>2014-02-21T23:01:00.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.41</td>\n",
       "      <td>-73.991533</td>\n",
       "      <td>40.715557</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-73.960765</td>\n",
       "      <td>40.714267</td>\n",
       "      <td>CSH</td>\n",
       "      <td>9.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.50</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-12-29T06:45:07.000</td>\n",
       "      <td>2014-12-29T06:57:42.000</td>\n",
       "      <td>1</td>\n",
       "      <td>3.00</td>\n",
       "      <td>-73.983400</td>\n",
       "      <td>40.730120</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.986400</td>\n",
       "      <td>40.762110</td>\n",
       "      <td>CRD</td>\n",
       "      <td>11.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>13.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-02-06T20:34:19.000</td>\n",
       "      <td>2014-02-06T20:42:27.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.30</td>\n",
       "      <td>-73.997655</td>\n",
       "      <td>40.721188</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.959682</td>\n",
       "      <td>40.710732</td>\n",
       "      <td>CRD</td>\n",
       "      <td>9.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.60</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-12-08T14:06:50.000</td>\n",
       "      <td>2014-12-08T14:27:54.000</td>\n",
       "      <td>1</td>\n",
       "      <td>8.60</td>\n",
       "      <td>-73.959190</td>\n",
       "      <td>40.771055</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.872402</td>\n",
       "      <td>40.774387</td>\n",
       "      <td>CRD</td>\n",
       "      <td>26.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>8.05</td>\n",
       "      <td>5.33</td>\n",
       "      <td>40.38</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>VTS</td>\n",
       "      <td>2014-01-05T13:30:00.000</td>\n",
       "      <td>2014-01-05T13:42:00.000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.67</td>\n",
       "      <td>-73.981507</td>\n",
       "      <td>40.765767</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-73.987467</td>\n",
       "      <td>40.758612</td>\n",
       "      <td>CSH</td>\n",
       "      <td>8.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>VTS</td>\n",
       "      <td>2014-12-11T11:05:00.000</td>\n",
       "      <td>2014-12-11T11:20:00.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.14</td>\n",
       "      <td>-73.971572</td>\n",
       "      <td>40.744195</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-73.953260</td>\n",
       "      <td>40.766670</td>\n",
       "      <td>CSH</td>\n",
       "      <td>11.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-11-02T22:27:54.000</td>\n",
       "      <td>2014-11-02T22:32:58.000</td>\n",
       "      <td>2</td>\n",
       "      <td>0.60</td>\n",
       "      <td>-73.988338</td>\n",
       "      <td>40.737827</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.985209</td>\n",
       "      <td>40.732171</td>\n",
       "      <td>CRD</td>\n",
       "      <td>5.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-03-04T19:07:12.000</td>\n",
       "      <td>2014-03-04T19:29:12.000</td>\n",
       "      <td>1</td>\n",
       "      <td>5.00</td>\n",
       "      <td>-73.960530</td>\n",
       "      <td>40.766422</td>\n",
       "      <td>N</td>\n",
       "      <td>-73.961782</td>\n",
       "      <td>40.719436</td>\n",
       "      <td>CRD</td>\n",
       "      <td>19.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>4.20</td>\n",
       "      <td>0.00</td>\n",
       "      <td>25.20</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>CMT</td>\n",
       "      <td>2014-10-23T05:55:49.000</td>\n",
       "      <td>2014-10-23T06:10:55.000</td>\n",
       "      <td>1</td>\n",
       "      <td>3.60</td>\n",
       "      <td>-73.993942</td>\n",
       "      <td>40.751130</td>\n",
       "      <td>N</td>\n",
       "      <td>-74.015278</td>\n",
       "      <td>40.711380</td>\n",
       "      <td>CRD</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>18.00</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>VTS</td>\n",
       "      <td>2014-03-22T20:23:00.000</td>\n",
       "      <td>2014-03-22T20:31:00.000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.08</td>\n",
       "      <td>-73.973345</td>\n",
       "      <td>40.759012</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-73.978367</td>\n",
       "      <td>40.778037</td>\n",
       "      <td>CRD</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>11.90</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   vendor_id          pickup_datetime         dropoff_datetime  \\\n",
       "0        CMT  2014-11-23T20:31:29.000  2014-11-23T20:31:29.000   \n",
       "1        CMT  2014-04-16T13:45:06.000  2014-04-16T14:00:21.000   \n",
       "2        CMT  2014-02-09T22:43:54.000  2014-02-09T23:06:21.000   \n",
       "3        CMT  2014-10-27T23:15:23.000  2014-10-27T23:23:54.000   \n",
       "4        VTS  2014-09-14T01:32:00.000  2014-09-14T01:44:00.000   \n",
       "5        VTS  2014-02-21T22:52:00.000  2014-02-21T23:01:00.000   \n",
       "6        CMT  2014-12-29T06:45:07.000  2014-12-29T06:57:42.000   \n",
       "7        CMT  2014-02-06T20:34:19.000  2014-02-06T20:42:27.000   \n",
       "8        CMT  2014-12-08T14:06:50.000  2014-12-08T14:27:54.000   \n",
       "9        VTS  2014-01-05T13:30:00.000  2014-01-05T13:42:00.000   \n",
       "10       VTS  2014-12-11T11:05:00.000  2014-12-11T11:20:00.000   \n",
       "11       CMT  2014-11-02T22:27:54.000  2014-11-02T22:32:58.000   \n",
       "12       CMT  2014-03-04T19:07:12.000  2014-03-04T19:29:12.000   \n",
       "13       CMT  2014-10-23T05:55:49.000  2014-10-23T06:10:55.000   \n",
       "14       VTS  2014-03-22T20:23:00.000  2014-03-22T20:31:00.000   \n",
       "\n",
       "    passenger_count  trip_distance  pickup_longitude  pickup_latitude  \\\n",
       "0                 3           0.00          0.000000         0.000000   \n",
       "1                 1           2.40        -73.990027        40.757197   \n",
       "2                 2           9.30        -73.870607        40.773682   \n",
       "3                 1           2.10        -73.983611        40.769135   \n",
       "4                 1           3.36        -73.949010        40.754087   \n",
       "5                 1           2.41        -73.991533        40.715557   \n",
       "6                 1           3.00        -73.983400        40.730120   \n",
       "7                 1           2.30        -73.997655        40.721188   \n",
       "8                 1           8.60        -73.959190        40.771055   \n",
       "9                 1           0.67        -73.981507        40.765767   \n",
       "10                1           2.14        -73.971572        40.744195   \n",
       "11                2           0.60        -73.988338        40.737827   \n",
       "12                1           5.00        -73.960530        40.766422   \n",
       "13                1           3.60        -73.993942        40.751130   \n",
       "14                1           2.08        -73.973345        40.759012   \n",
       "\n",
       "   store_and_fwd_flag  dropoff_longitude  dropoff_latitude payment_type  \\\n",
       "0                   N           0.000000          0.000000          CSH   \n",
       "1                   N         -73.958484         40.765540          CSH   \n",
       "2                   N         -73.971970         40.784683          CRD   \n",
       "3                   N         -73.974267         40.748188          CRD   \n",
       "4                 NaN         -73.960682         40.712792          CRD   \n",
       "5                 NaN         -73.960765         40.714267          CSH   \n",
       "6                   N         -73.986400         40.762110          CRD   \n",
       "7                   N         -73.959682         40.710732          CRD   \n",
       "8                   N         -73.872402         40.774387          CRD   \n",
       "9                 NaN         -73.987467         40.758612          CSH   \n",
       "10                NaN         -73.953260         40.766670          CSH   \n",
       "11                  N         -73.985209         40.732171          CRD   \n",
       "12                  N         -73.961782         40.719436          CRD   \n",
       "13                  N         -74.015278         40.711380          CRD   \n",
       "14                NaN         -73.978367         40.778037          CRD   \n",
       "\n",
       "    fare_amount  mta_tax  tip_amount  tolls_amount  total_amount  \\\n",
       "0           3.0      0.5        0.00          0.00          4.00   \n",
       "1          12.0      0.5        0.00          0.00         12.50   \n",
       "2          27.5      0.5        6.76          5.33         40.59   \n",
       "3           8.5      0.5        2.37          0.00         11.87   \n",
       "4          12.5      0.5        2.60          0.00         16.10   \n",
       "5           9.5      0.5        0.00          0.00         10.50   \n",
       "6          11.5      0.5        1.00          0.00         13.00   \n",
       "7           9.5      0.5        2.10          0.00         12.60   \n",
       "8          26.5      0.5        8.05          5.33         40.38   \n",
       "9           8.5      0.5        0.00          0.00          9.00   \n",
       "10         11.5      0.5        0.00          0.00         12.00   \n",
       "11          5.5      0.5        1.30          0.00          7.80   \n",
       "12         19.5      0.5        4.20          0.00         25.20   \n",
       "13         14.0      0.5        3.00          0.00         18.00   \n",
       "14          9.0      0.5        1.90          0.00         11.90   \n",
       "\n",
       "    imp_surcharge  rate_code  \n",
       "0             0.5          1  \n",
       "1             0.0          1  \n",
       "2             0.5          1  \n",
       "3             0.5          1  \n",
       "4             0.5          1  \n",
       "5             0.5          1  \n",
       "6             0.0          1  \n",
       "7             0.5          1  \n",
       "8             0.0          1  \n",
       "9             0.0          1  \n",
       "10            0.0          1  \n",
       "11            0.5          1  \n",
       "12            1.0          1  \n",
       "13            0.5          1  \n",
       "14            0.5          1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Looking at data types\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_dc10d_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"col_heading level0 col0\" >vendor_id</th>\n",
       "      <th class=\"col_heading level0 col1\" >pickup_datetime</th>\n",
       "      <th class=\"col_heading level0 col2\" >dropoff_datetime</th>\n",
       "      <th class=\"col_heading level0 col3\" >passenger_count</th>\n",
       "      <th class=\"col_heading level0 col4\" >trip_distance</th>\n",
       "      <th class=\"col_heading level0 col5\" >pickup_longitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >pickup_latitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >store_and_fwd_flag</th>\n",
       "      <th class=\"col_heading level0 col8\" >dropoff_longitude</th>\n",
       "      <th class=\"col_heading level0 col9\" >dropoff_latitude</th>\n",
       "      <th class=\"col_heading level0 col10\" >payment_type</th>\n",
       "      <th class=\"col_heading level0 col11\" >fare_amount</th>\n",
       "      <th class=\"col_heading level0 col12\" >mta_tax</th>\n",
       "      <th class=\"col_heading level0 col13\" >tip_amount</th>\n",
       "      <th class=\"col_heading level0 col14\" >tolls_amount</th>\n",
       "      <th class=\"col_heading level0 col15\" >total_amount</th>\n",
       "      <th class=\"col_heading level0 col16\" >imp_surcharge</th>\n",
       "      <th class=\"col_heading level0 col17\" >rate_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_dc10d_row0_col0\" class=\"data row0 col0\" >object</td>\n",
       "      <td id=\"T_dc10d_row0_col1\" class=\"data row0 col1\" >object</td>\n",
       "      <td id=\"T_dc10d_row0_col2\" class=\"data row0 col2\" >object</td>\n",
       "      <td id=\"T_dc10d_row0_col3\" class=\"data row0 col3\" >int64</td>\n",
       "      <td id=\"T_dc10d_row0_col4\" class=\"data row0 col4\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col5\" class=\"data row0 col5\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col6\" class=\"data row0 col6\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col7\" class=\"data row0 col7\" >object</td>\n",
       "      <td id=\"T_dc10d_row0_col8\" class=\"data row0 col8\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col9\" class=\"data row0 col9\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col10\" class=\"data row0 col10\" >object</td>\n",
       "      <td id=\"T_dc10d_row0_col11\" class=\"data row0 col11\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col12\" class=\"data row0 col12\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col13\" class=\"data row0 col13\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col14\" class=\"data row0 col14\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col15\" class=\"data row0 col15\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col16\" class=\"data row0 col16\" >float64</td>\n",
       "      <td id=\"T_dc10d_row0_col17\" class=\"data row0 col17\" >int64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1b7879a2c08>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Numeric Data Distribution for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>imp_surcharge</th>\n",
       "      <th>rate_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.696380</td>\n",
       "      <td>2.954888</td>\n",
       "      <td>-72.482509</td>\n",
       "      <td>39.928659</td>\n",
       "      <td>-72.483211</td>\n",
       "      <td>39.930245</td>\n",
       "      <td>12.699273</td>\n",
       "      <td>0.498078</td>\n",
       "      <td>1.518990</td>\n",
       "      <td>0.277575</td>\n",
       "      <td>15.312457</td>\n",
       "      <td>0.318139</td>\n",
       "      <td>1.036728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.355228</td>\n",
       "      <td>3.508526</td>\n",
       "      <td>10.408435</td>\n",
       "      <td>5.728149</td>\n",
       "      <td>10.402333</td>\n",
       "      <td>5.724365</td>\n",
       "      <td>10.502327</td>\n",
       "      <td>0.030940</td>\n",
       "      <td>2.347381</td>\n",
       "      <td>1.276485</td>\n",
       "      <td>12.716466</td>\n",
       "      <td>0.360349</td>\n",
       "      <td>0.500763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-75.922994</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-78.822854</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.800000</td>\n",
       "      <td>-73.981902</td>\n",
       "      <td>40.752336</td>\n",
       "      <td>-73.980025</td>\n",
       "      <td>40.752702</td>\n",
       "      <td>9.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>97.900000</td>\n",
       "      <td>159.403213</td>\n",
       "      <td>55.558723</td>\n",
       "      <td>159.403213</td>\n",
       "      <td>56.103127</td>\n",
       "      <td>450.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>185.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>450.500000</td>\n",
       "      <td>2.130000</td>\n",
       "      <td>210.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       passenger_count  trip_distance  pickup_longitude  pickup_latitude  \\\n",
       "count    250000.000000  250000.000000     250000.000000    250000.000000   \n",
       "mean          1.696380       2.954888        -72.482509        39.928659   \n",
       "std           1.355228       3.508526         10.408435         5.728149   \n",
       "min           0.000000       0.000000        -75.922994         0.000000   \n",
       "50%           1.000000       1.800000        -73.981902        40.752336   \n",
       "max           6.000000      97.900000        159.403213        55.558723   \n",
       "\n",
       "       dropoff_longitude  dropoff_latitude    fare_amount        mta_tax  \\\n",
       "count      250000.000000     250000.000000  250000.000000  250000.000000   \n",
       "mean          -72.483211         39.930245      12.699273       0.498078   \n",
       "std            10.402333          5.724365      10.502327       0.030940   \n",
       "min           -78.822854          0.000000       2.500000       0.000000   \n",
       "50%           -73.980025         40.752702       9.500000       0.500000   \n",
       "max           159.403213         56.103127     450.000000       0.500000   \n",
       "\n",
       "          tip_amount   tolls_amount   total_amount  imp_surcharge  \\\n",
       "count  250000.000000  250000.000000  250000.000000  250000.000000   \n",
       "mean        1.518990       0.277575      15.312457       0.318139   \n",
       "std         2.347381       1.276485      12.716466       0.360349   \n",
       "min         0.000000       0.000000       2.500000       0.000000   \n",
       "50%         1.000000       0.000000      11.500000       0.000000   \n",
       "max       185.000000      20.000000     450.500000       2.130000   \n",
       "\n",
       "           rate_code  \n",
       "count  250000.000000  \n",
       "mean        1.036728  \n",
       "std         0.500763  \n",
       "min         0.000000  \n",
       "50%         1.000000  \n",
       "max       210.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance Percent in Numeric Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_d5207_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >passenger_count</th>\n",
       "      <th class=\"col_heading level0 col1\" >trip_distance</th>\n",
       "      <th class=\"col_heading level0 col2\" >pickup_longitude</th>\n",
       "      <th class=\"col_heading level0 col3\" >pickup_latitude</th>\n",
       "      <th class=\"col_heading level0 col4\" >dropoff_longitude</th>\n",
       "      <th class=\"col_heading level0 col5\" >dropoff_latitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >fare_amount</th>\n",
       "      <th class=\"col_heading level0 col7\" >mta_tax</th>\n",
       "      <th class=\"col_heading level0 col8\" >tip_amount</th>\n",
       "      <th class=\"col_heading level0 col9\" >tolls_amount</th>\n",
       "      <th class=\"col_heading level0 col10\" >total_amount</th>\n",
       "      <th class=\"col_heading level0 col11\" >imp_surcharge</th>\n",
       "      <th class=\"col_heading level0 col12\" >rate_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row0\" class=\"row_heading level0 row0\" >count</th>\n",
       "      <td id=\"T_d5207_row0_col0\" class=\"data row0 col0\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col1\" class=\"data row0 col1\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col2\" class=\"data row0 col2\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col3\" class=\"data row0 col3\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col4\" class=\"data row0 col4\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col5\" class=\"data row0 col5\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col6\" class=\"data row0 col6\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col7\" class=\"data row0 col7\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col8\" class=\"data row0 col8\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col9\" class=\"data row0 col9\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col10\" class=\"data row0 col10\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col11\" class=\"data row0 col11\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row0_col12\" class=\"data row0 col12\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row1\" class=\"row_heading level0 row1\" >mean</th>\n",
       "      <td id=\"T_d5207_row1_col0\" class=\"data row1 col0\" >0.08%</td>\n",
       "      <td id=\"T_d5207_row1_col1\" class=\"data row1 col1\" >0.03%</td>\n",
       "      <td id=\"T_d5207_row1_col2\" class=\"data row1 col2\" >0.08%</td>\n",
       "      <td id=\"T_d5207_row1_col3\" class=\"data row1 col3\" >0.08%</td>\n",
       "      <td id=\"T_d5207_row1_col4\" class=\"data row1 col4\" >0.08%</td>\n",
       "      <td id=\"T_d5207_row1_col5\" class=\"data row1 col5\" >0.07%</td>\n",
       "      <td id=\"T_d5207_row1_col6\" class=\"data row1 col6\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row1_col7\" class=\"data row1 col7\" >0.01%</td>\n",
       "      <td id=\"T_d5207_row1_col8\" class=\"data row1 col8\" >0.06%</td>\n",
       "      <td id=\"T_d5207_row1_col9\" class=\"data row1 col9\" >0.13%</td>\n",
       "      <td id=\"T_d5207_row1_col10\" class=\"data row1 col10\" >0.01%</td>\n",
       "      <td id=\"T_d5207_row1_col11\" class=\"data row1 col11\" >0.62%</td>\n",
       "      <td id=\"T_d5207_row1_col12\" class=\"data row1 col12\" >0.14%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row2\" class=\"row_heading level0 row2\" >std</th>\n",
       "      <td id=\"T_d5207_row2_col0\" class=\"data row2 col0\" >0.12%</td>\n",
       "      <td id=\"T_d5207_row2_col1\" class=\"data row2 col1\" >0.04%</td>\n",
       "      <td id=\"T_d5207_row2_col2\" class=\"data row2 col2\" >2.06%</td>\n",
       "      <td id=\"T_d5207_row2_col3\" class=\"data row2 col3\" >1.97%</td>\n",
       "      <td id=\"T_d5207_row2_col4\" class=\"data row2 col4\" >1.92%</td>\n",
       "      <td id=\"T_d5207_row2_col5\" class=\"data row2 col5\" >1.80%</td>\n",
       "      <td id=\"T_d5207_row2_col6\" class=\"data row2 col6\" >2.19%</td>\n",
       "      <td id=\"T_d5207_row2_col7\" class=\"data row2 col7\" >0.73%</td>\n",
       "      <td id=\"T_d5207_row2_col8\" class=\"data row2 col8\" >0.64%</td>\n",
       "      <td id=\"T_d5207_row2_col9\" class=\"data row2 col9\" >0.14%</td>\n",
       "      <td id=\"T_d5207_row2_col10\" class=\"data row2 col10\" >1.70%</td>\n",
       "      <td id=\"T_d5207_row2_col11\" class=\"data row2 col11\" >0.12%</td>\n",
       "      <td id=\"T_d5207_row2_col12\" class=\"data row2 col12\" >45.49%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row3\" class=\"row_heading level0 row3\" >min</th>\n",
       "      <td id=\"T_d5207_row3_col0\" class=\"data row3 col0\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col1\" class=\"data row3 col1\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col2\" class=\"data row3 col2\" >22.18%</td>\n",
       "      <td id=\"T_d5207_row3_col3\" class=\"data row3 col3\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col4\" class=\"data row3 col4\" >2.53%</td>\n",
       "      <td id=\"T_d5207_row3_col5\" class=\"data row3 col5\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col6\" class=\"data row3 col6\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row3_col7\" class=\"data row3 col7\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col8\" class=\"data row3 col8\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col9\" class=\"data row3 col9\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col10\" class=\"data row3 col10\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row3_col11\" class=\"data row3 col11\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row3_col12\" class=\"data row3 col12\" >Div_0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row4\" class=\"row_heading level0 row4\" >50%</th>\n",
       "      <td id=\"T_d5207_row4_col0\" class=\"data row4 col0\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col1\" class=\"data row4 col1\" >0.56%</td>\n",
       "      <td id=\"T_d5207_row4_col2\" class=\"data row4 col2\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col3\" class=\"data row4 col3\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col4\" class=\"data row4 col4\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col5\" class=\"data row4 col5\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col6\" class=\"data row4 col6\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col7\" class=\"data row4 col7\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col8\" class=\"data row4 col8\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col9\" class=\"data row4 col9\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row4_col10\" class=\"data row4 col10\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row4_col11\" class=\"data row4 col11\" >Div_0</td>\n",
       "      <td id=\"T_d5207_row4_col12\" class=\"data row4 col12\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d5207_level0_row5\" class=\"row_heading level0 row5\" >max</th>\n",
       "      <td id=\"T_d5207_row5_col0\" class=\"data row5 col0\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row5_col1\" class=\"data row5 col1\" >16.85%</td>\n",
       "      <td id=\"T_d5207_row5_col2\" class=\"data row5 col2\" >100.00%</td>\n",
       "      <td id=\"T_d5207_row5_col3\" class=\"data row5 col3\" >18.39%</td>\n",
       "      <td id=\"T_d5207_row5_col4\" class=\"data row5 col4\" >100.00%</td>\n",
       "      <td id=\"T_d5207_row5_col5\" class=\"data row5 col5\" >19.18%</td>\n",
       "      <td id=\"T_d5207_row5_col6\" class=\"data row5 col6\" >11.11%</td>\n",
       "      <td id=\"T_d5207_row5_col7\" class=\"data row5 col7\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row5_col8\" class=\"data row5 col8\" >35.14%</td>\n",
       "      <td id=\"T_d5207_row5_col9\" class=\"data row5 col9\" >0.00%</td>\n",
       "      <td id=\"T_d5207_row5_col10\" class=\"data row5 col10\" >13.98%</td>\n",
       "      <td id=\"T_d5207_row5_col11\" class=\"data row5 col11\" >40.85%</td>\n",
       "      <td id=\"T_d5207_row5_col12\" class=\"data row5 col12\" >97.14%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1b7bf17b1c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Date Data Distributions for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dropoff_datetime</th>\n",
       "      <th>pickup_datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>250000</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2014-06-29 05:57:39.230044160</td>\n",
       "      <td>2014-06-29 05:44:17.661416192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2014-01-01 00:17:00</td>\n",
       "      <td>2014-01-01 00:05:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2014-06-24 18:34:56</td>\n",
       "      <td>2014-06-24 18:21:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2015-01-01 00:17:00</td>\n",
       "      <td>2014-12-31 23:51:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    dropoff_datetime                pickup_datetime\n",
       "count                         250000                         250000\n",
       "mean   2014-06-29 05:57:39.230044160  2014-06-29 05:44:17.661416192\n",
       "min              2014-01-01 00:17:00            2014-01-01 00:05:00\n",
       "50%              2014-06-24 18:34:56            2014-06-24 18:21:00\n",
       "max              2015-01-01 00:17:00            2014-12-31 23:51:02"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance in Date Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dropoff_datetime</th>\n",
       "      <th>pickup_datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0 days 04:55:47.510567936</td>\n",
       "      <td>0 days 04:55:46.512060160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0 days 00:08:00</td>\n",
       "      <td>0 days 00:03:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0 days 00:27:56</td>\n",
       "      <td>0 days 00:27:13.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0 days 00:19:21</td>\n",
       "      <td>0 days 00:01:30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                dropoff_datetime            pickup_datetime\n",
       "count                          0                          0\n",
       "mean   0 days 04:55:47.510567936  0 days 04:55:46.512060160\n",
       "min              0 days 00:08:00            0 days 00:03:58\n",
       "50%              0 days 00:27:56     0 days 00:27:13.500000\n",
       "max              0 days 00:19:21            0 days 00:01:30"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Categorical Data Value Counts for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "NaN    127444\n",
       "N      119730\n",
       "Y        2826\n",
       "Name: store_and_fwd_flag, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CRD    144894\n",
       "CSH    103035\n",
       "UNK      1170\n",
       "NOC       689\n",
       "DIS       212\n",
       "Name: payment_type, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "VTS    127444\n",
       "CMT    122556\n",
       "Name: vendor_id, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eda_csv([first_c,second_c], {'pickup_datetime', 'dropoff_datetime'}, {'vendor_id', 'store_and_fwd_flag','payment_type'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Green Taxi Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "gtaxi_iter = pd.read_csv('D://Capstone Data//Green Taxi//greentaxi2014.csv', chunksize=250000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_c = next(gtaxi_iter)\n",
    "second_c = next(gtaxi_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['vendorid', 'lpep_pickup_datetime', 'lpep_dropoff_datetime',\n",
       "       'store_and_fwd_flag', 'ratecodeid', 'pickup_longitude',\n",
       "       'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude',\n",
       "       'passenger_count', 'trip_distance', 'fare_amount', 'extra', 'mta_tax',\n",
       "       'tip_amount', 'tolls_amount', 'total_amount', 'payment_type',\n",
       "       'trip_type'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_c.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "vendorid                   int64\n",
       "lpep_pickup_datetime      object\n",
       "lpep_dropoff_datetime     object\n",
       "store_and_fwd_flag        object\n",
       "ratecodeid                 int64\n",
       "pickup_longitude         float64\n",
       "pickup_latitude          float64\n",
       "dropoff_longitude        float64\n",
       "dropoff_latitude         float64\n",
       "passenger_count            int64\n",
       "trip_distance            float64\n",
       "fare_amount              float64\n",
       "extra                    float64\n",
       "mta_tax                  float64\n",
       "tip_amount               float64\n",
       "tolls_amount             float64\n",
       "total_amount             float64\n",
       "payment_type               int64\n",
       "trip_type                float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_c.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taking a look at the data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vendorid</th>\n",
       "      <th>lpep_pickup_datetime</th>\n",
       "      <th>lpep_dropoff_datetime</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>ratecodeid</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>trip_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-13T16:47:42.000</td>\n",
       "      <td>2014-10-13T16:48:43.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.20</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.50</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-11-22T01:09:08.000</td>\n",
       "      <td>2014-11-22T01:18:02.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>4.00</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>14.00</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-08-04T22:46:53.000</td>\n",
       "      <td>2014-08-04T23:05:24.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>4.13</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.20</td>\n",
       "      <td>0.00</td>\n",
       "      <td>20.20</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-11-25T09:41:35.000</td>\n",
       "      <td>2014-11-25T09:54:44.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.957542</td>\n",
       "      <td>40.728981</td>\n",
       "      <td>-73.953972</td>\n",
       "      <td>40.742592</td>\n",
       "      <td>1</td>\n",
       "      <td>1.40</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.60</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-09-30T09:14:56.000</td>\n",
       "      <td>2014-09-30T09:22:07.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.934090</td>\n",
       "      <td>40.853523</td>\n",
       "      <td>-73.939621</td>\n",
       "      <td>40.841522</td>\n",
       "      <td>1</td>\n",
       "      <td>0.89</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.00</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-06-27T19:27:14.000</td>\n",
       "      <td>2014-06-27T19:34:06.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.930634</td>\n",
       "      <td>40.853634</td>\n",
       "      <td>-73.939034</td>\n",
       "      <td>40.842319</td>\n",
       "      <td>1</td>\n",
       "      <td>0.89</td>\n",
       "      <td>6.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9.00</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-22T23:11:46.000</td>\n",
       "      <td>2014-10-22T23:17:37.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.969048</td>\n",
       "      <td>40.693134</td>\n",
       "      <td>-73.971199</td>\n",
       "      <td>40.680859</td>\n",
       "      <td>1</td>\n",
       "      <td>1.51</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.50</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-09-18T15:50:30.000</td>\n",
       "      <td>2014-09-18T15:59:47.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.901405</td>\n",
       "      <td>40.862232</td>\n",
       "      <td>-73.898750</td>\n",
       "      <td>40.873913</td>\n",
       "      <td>1</td>\n",
       "      <td>1.04</td>\n",
       "      <td>7.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.00</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-08-30T20:26:57.000</td>\n",
       "      <td>2014-08-30T20:40:24.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.939323</td>\n",
       "      <td>40.805096</td>\n",
       "      <td>-73.922997</td>\n",
       "      <td>40.793377</td>\n",
       "      <td>1</td>\n",
       "      <td>3.27</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>5.65</td>\n",
       "      <td>5.33</td>\n",
       "      <td>24.98</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-30T11:41:48.000</td>\n",
       "      <td>2014-10-30T11:46:32.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.952835</td>\n",
       "      <td>40.810822</td>\n",
       "      <td>-73.949638</td>\n",
       "      <td>40.802246</td>\n",
       "      <td>2</td>\n",
       "      <td>0.77</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.00</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-09-26T20:40:47.000</td>\n",
       "      <td>2014-09-26T20:43:09.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.808022</td>\n",
       "      <td>40.700661</td>\n",
       "      <td>-73.808571</td>\n",
       "      <td>40.703892</td>\n",
       "      <td>6</td>\n",
       "      <td>0.30</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.50</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-03-26T14:40:21.000</td>\n",
       "      <td>2014-03-26T14:57:08.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.866760</td>\n",
       "      <td>40.757507</td>\n",
       "      <td>-73.913010</td>\n",
       "      <td>40.765049</td>\n",
       "      <td>1</td>\n",
       "      <td>3.30</td>\n",
       "      <td>14.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>15.00</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-05-08T14:42:38.000</td>\n",
       "      <td>2014-05-08T15:00:49.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.937538</td>\n",
       "      <td>40.804192</td>\n",
       "      <td>-73.865433</td>\n",
       "      <td>40.770844</td>\n",
       "      <td>1</td>\n",
       "      <td>6.70</td>\n",
       "      <td>22.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3.00</td>\n",
       "      <td>5.33</td>\n",
       "      <td>31.33</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-09-10T19:39:54.000</td>\n",
       "      <td>2014-09-10T19:50:21.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.946274</td>\n",
       "      <td>40.811600</td>\n",
       "      <td>-73.907654</td>\n",
       "      <td>40.816624</td>\n",
       "      <td>1</td>\n",
       "      <td>2.48</td>\n",
       "      <td>10.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>12.00</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-06-27T22:17:15.000</td>\n",
       "      <td>2014-06-27T22:30:36.000</td>\n",
       "      <td>N</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.988861</td>\n",
       "      <td>40.691723</td>\n",
       "      <td>-73.973717</td>\n",
       "      <td>40.743927</td>\n",
       "      <td>1</td>\n",
       "      <td>5.88</td>\n",
       "      <td>18.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>21.50</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    vendorid     lpep_pickup_datetime    lpep_dropoff_datetime  \\\n",
       "0          2  2014-10-13T16:47:42.000  2014-10-13T16:48:43.000   \n",
       "1          1  2014-11-22T01:09:08.000  2014-11-22T01:18:02.000   \n",
       "2          2  2014-08-04T22:46:53.000  2014-08-04T23:05:24.000   \n",
       "3          1  2014-11-25T09:41:35.000  2014-11-25T09:54:44.000   \n",
       "4          2  2014-09-30T09:14:56.000  2014-09-30T09:22:07.000   \n",
       "5          2  2014-06-27T19:27:14.000  2014-06-27T19:34:06.000   \n",
       "6          2  2014-10-22T23:11:46.000  2014-10-22T23:17:37.000   \n",
       "7          2  2014-09-18T15:50:30.000  2014-09-18T15:59:47.000   \n",
       "8          2  2014-08-30T20:26:57.000  2014-08-30T20:40:24.000   \n",
       "9          2  2014-10-30T11:41:48.000  2014-10-30T11:46:32.000   \n",
       "10         2  2014-09-26T20:40:47.000  2014-09-26T20:43:09.000   \n",
       "11         1  2014-03-26T14:40:21.000  2014-03-26T14:57:08.000   \n",
       "12         2  2014-05-08T14:42:38.000  2014-05-08T15:00:49.000   \n",
       "13         2  2014-09-10T19:39:54.000  2014-09-10T19:50:21.000   \n",
       "14         2  2014-06-27T22:17:15.000  2014-06-27T22:30:36.000   \n",
       "\n",
       "   store_and_fwd_flag  ratecodeid  pickup_longitude  pickup_latitude  \\\n",
       "0                   N           1          0.000000         0.000000   \n",
       "1                   N           1          0.000000         0.000000   \n",
       "2                   N           1          0.000000         0.000000   \n",
       "3                   N           1        -73.957542        40.728981   \n",
       "4                   N           1        -73.934090        40.853523   \n",
       "5                   N           1        -73.930634        40.853634   \n",
       "6                   N           1        -73.969048        40.693134   \n",
       "7                   N           1        -73.901405        40.862232   \n",
       "8                   N           1        -73.939323        40.805096   \n",
       "9                   N           1        -73.952835        40.810822   \n",
       "10                  N           1        -73.808022        40.700661   \n",
       "11                  N           1        -73.866760        40.757507   \n",
       "12                  N           1        -73.937538        40.804192   \n",
       "13                  N           1        -73.946274        40.811600   \n",
       "14                  N           1        -73.988861        40.691723   \n",
       "\n",
       "    dropoff_longitude  dropoff_latitude  passenger_count  trip_distance  \\\n",
       "0            0.000000          0.000000                1           0.20   \n",
       "1            0.000000          0.000000                1           4.00   \n",
       "2            0.000000          0.000000                6           4.13   \n",
       "3          -73.953972         40.742592                1           1.40   \n",
       "4          -73.939621         40.841522                1           0.89   \n",
       "5          -73.939034         40.842319                1           0.89   \n",
       "6          -73.971199         40.680859                1           1.51   \n",
       "7          -73.898750         40.873913                1           1.04   \n",
       "8          -73.922997         40.793377                1           3.27   \n",
       "9          -73.949638         40.802246                2           0.77   \n",
       "10         -73.808571         40.703892                6           0.30   \n",
       "11         -73.913010         40.765049                1           3.30   \n",
       "12         -73.865433         40.770844                1           6.70   \n",
       "13         -73.907654         40.816624                1           2.48   \n",
       "14         -73.973717         40.743927                1           5.88   \n",
       "\n",
       "    fare_amount  extra  mta_tax  tip_amount  tolls_amount  total_amount  \\\n",
       "0           3.0    0.0      0.5        0.00          0.00          3.50   \n",
       "1          13.0    0.5      0.5        0.00          0.00         14.00   \n",
       "2          16.0    0.5      0.5        3.20          0.00         20.20   \n",
       "3          10.0    0.0      0.5        2.10          0.00         12.60   \n",
       "4           6.5    0.0      0.5        0.00          0.00          7.00   \n",
       "5           6.5    1.0      0.5        1.00          0.00          9.00   \n",
       "6           6.5    0.5      0.5        0.00          0.00          7.50   \n",
       "7           7.5    0.0      0.5        0.00          0.00          8.00   \n",
       "8          13.0    0.5      0.5        5.65          5.33         24.98   \n",
       "9           5.0    0.0      0.5        1.50          0.00          7.00   \n",
       "10          3.5    0.5      0.5        0.00          0.00          4.50   \n",
       "11         14.5    0.0      0.5        0.00          0.00         15.00   \n",
       "12         22.5    0.0      0.5        3.00          5.33         31.33   \n",
       "13         10.5    1.0      0.5        0.00          0.00         12.00   \n",
       "14         18.5    0.5      0.5        2.00          0.00         21.50   \n",
       "\n",
       "    payment_type  trip_type  \n",
       "0              2        1.0  \n",
       "1              2        1.0  \n",
       "2              1        1.0  \n",
       "3              1        1.0  \n",
       "4              2        1.0  \n",
       "5              1        1.0  \n",
       "6              2        1.0  \n",
       "7              2        1.0  \n",
       "8              1        1.0  \n",
       "9              1        1.0  \n",
       "10             2        1.0  \n",
       "11             2        NaN  \n",
       "12             1        1.0  \n",
       "13             2        1.0  \n",
       "14             1        1.0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Looking at data types\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_1f5de_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"col_heading level0 col0\" >vendorid</th>\n",
       "      <th class=\"col_heading level0 col1\" >lpep_pickup_datetime</th>\n",
       "      <th class=\"col_heading level0 col2\" >lpep_dropoff_datetime</th>\n",
       "      <th class=\"col_heading level0 col3\" >store_and_fwd_flag</th>\n",
       "      <th class=\"col_heading level0 col4\" >ratecodeid</th>\n",
       "      <th class=\"col_heading level0 col5\" >pickup_longitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >pickup_latitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >dropoff_longitude</th>\n",
       "      <th class=\"col_heading level0 col8\" >dropoff_latitude</th>\n",
       "      <th class=\"col_heading level0 col9\" >passenger_count</th>\n",
       "      <th class=\"col_heading level0 col10\" >trip_distance</th>\n",
       "      <th class=\"col_heading level0 col11\" >fare_amount</th>\n",
       "      <th class=\"col_heading level0 col12\" >extra</th>\n",
       "      <th class=\"col_heading level0 col13\" >mta_tax</th>\n",
       "      <th class=\"col_heading level0 col14\" >tip_amount</th>\n",
       "      <th class=\"col_heading level0 col15\" >tolls_amount</th>\n",
       "      <th class=\"col_heading level0 col16\" >total_amount</th>\n",
       "      <th class=\"col_heading level0 col17\" >payment_type</th>\n",
       "      <th class=\"col_heading level0 col18\" >trip_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_1f5de_row0_col0\" class=\"data row0 col0\" >int64</td>\n",
       "      <td id=\"T_1f5de_row0_col1\" class=\"data row0 col1\" >object</td>\n",
       "      <td id=\"T_1f5de_row0_col2\" class=\"data row0 col2\" >object</td>\n",
       "      <td id=\"T_1f5de_row0_col3\" class=\"data row0 col3\" >object</td>\n",
       "      <td id=\"T_1f5de_row0_col4\" class=\"data row0 col4\" >int64</td>\n",
       "      <td id=\"T_1f5de_row0_col5\" class=\"data row0 col5\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col6\" class=\"data row0 col6\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col7\" class=\"data row0 col7\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col8\" class=\"data row0 col8\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col9\" class=\"data row0 col9\" >int64</td>\n",
       "      <td id=\"T_1f5de_row0_col10\" class=\"data row0 col10\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col11\" class=\"data row0 col11\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col12\" class=\"data row0 col12\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col13\" class=\"data row0 col13\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col14\" class=\"data row0 col14\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col15\" class=\"data row0 col15\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col16\" class=\"data row0 col16\" >float64</td>\n",
       "      <td id=\"T_1f5de_row0_col17\" class=\"data row0 col17\" >int64</td>\n",
       "      <td id=\"T_1f5de_row0_col18\" class=\"data row0 col18\" >float64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1b7b1f603c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Numeric Data Distribution for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ratecodeid</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>trip_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>250000.00000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>250000.000000</td>\n",
       "      <td>222131.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.09042</td>\n",
       "      <td>-73.810662</td>\n",
       "      <td>40.689003</td>\n",
       "      <td>-73.813095</td>\n",
       "      <td>40.687610</td>\n",
       "      <td>1.428756</td>\n",
       "      <td>2.981606</td>\n",
       "      <td>12.485714</td>\n",
       "      <td>0.358003</td>\n",
       "      <td>0.488924</td>\n",
       "      <td>1.080043</td>\n",
       "      <td>0.117523</td>\n",
       "      <td>14.530486</td>\n",
       "      <td>1.018165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.61273</td>\n",
       "      <td>3.006437</td>\n",
       "      <td>1.658172</td>\n",
       "      <td>2.977500</td>\n",
       "      <td>1.642033</td>\n",
       "      <td>1.132011</td>\n",
       "      <td>3.206786</td>\n",
       "      <td>9.926939</td>\n",
       "      <td>0.382783</td>\n",
       "      <td>0.075862</td>\n",
       "      <td>2.557057</td>\n",
       "      <td>2.228884</td>\n",
       "      <td>11.519333</td>\n",
       "      <td>0.133548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>-74.372162</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-83.052063</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-251.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-0.500000</td>\n",
       "      <td>-1.050000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-251.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>-73.943443</td>\n",
       "      <td>40.750332</td>\n",
       "      <td>-73.943649</td>\n",
       "      <td>40.753376</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>9.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>99.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>41.143089</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>42.403736</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>631.100000</td>\n",
       "      <td>600.020000</td>\n",
       "      <td>50.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>600.000000</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>766.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ratecodeid  pickup_longitude  pickup_latitude  dropoff_longitude  \\\n",
       "count  250000.00000     250000.000000    250000.000000      250000.000000   \n",
       "mean        1.09042        -73.810662        40.689003         -73.813095   \n",
       "std         0.61273          3.006437         1.658172           2.977500   \n",
       "min         1.00000        -74.372162         0.000000         -83.052063   \n",
       "50%         1.00000        -73.943443        40.750332         -73.943649   \n",
       "max        99.00000          0.000000        41.143089           0.000000   \n",
       "\n",
       "       dropoff_latitude  passenger_count  trip_distance    fare_amount  \\\n",
       "count     250000.000000    250000.000000  250000.000000  250000.000000   \n",
       "mean          40.687610         1.428756       2.981606      12.485714   \n",
       "std            1.642033         1.132011       3.206786       9.926939   \n",
       "min            0.000000         0.000000       0.000000    -251.000000   \n",
       "50%           40.753376         1.000000       2.000000       9.500000   \n",
       "max           42.403736         8.000000     631.100000     600.020000   \n",
       "\n",
       "               extra        mta_tax     tip_amount   tolls_amount  \\\n",
       "count  250000.000000  250000.000000  250000.000000  250000.000000   \n",
       "mean        0.358003       0.488924       1.080043       0.117523   \n",
       "std         0.382783       0.075862       2.557057       2.228884   \n",
       "min        -1.000000      -0.500000      -1.050000       0.000000   \n",
       "50%         0.500000       0.500000       0.000000       0.000000   \n",
       "max        50.500000       1.000000     600.000000     750.000000   \n",
       "\n",
       "        total_amount      trip_type  \n",
       "count  250000.000000  222131.000000  \n",
       "mean       14.530486       1.018165  \n",
       "std        11.519333       0.133548  \n",
       "min      -251.000000       1.000000  \n",
       "50%        11.000000       1.000000  \n",
       "max       766.000000       2.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance Percent in Numeric Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_0554f_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >ratecodeid</th>\n",
       "      <th class=\"col_heading level0 col1\" >pickup_longitude</th>\n",
       "      <th class=\"col_heading level0 col2\" >pickup_latitude</th>\n",
       "      <th class=\"col_heading level0 col3\" >dropoff_longitude</th>\n",
       "      <th class=\"col_heading level0 col4\" >dropoff_latitude</th>\n",
       "      <th class=\"col_heading level0 col5\" >passenger_count</th>\n",
       "      <th class=\"col_heading level0 col6\" >trip_distance</th>\n",
       "      <th class=\"col_heading level0 col7\" >fare_amount</th>\n",
       "      <th class=\"col_heading level0 col8\" >extra</th>\n",
       "      <th class=\"col_heading level0 col9\" >mta_tax</th>\n",
       "      <th class=\"col_heading level0 col10\" >tip_amount</th>\n",
       "      <th class=\"col_heading level0 col11\" >tolls_amount</th>\n",
       "      <th class=\"col_heading level0 col12\" >total_amount</th>\n",
       "      <th class=\"col_heading level0 col13\" >trip_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row0\" class=\"row_heading level0 row0\" >count</th>\n",
       "      <td id=\"T_0554f_row0_col0\" class=\"data row0 col0\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col1\" class=\"data row0 col1\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col2\" class=\"data row0 col2\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col3\" class=\"data row0 col3\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col4\" class=\"data row0 col4\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col5\" class=\"data row0 col5\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col6\" class=\"data row0 col6\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col7\" class=\"data row0 col7\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col8\" class=\"data row0 col8\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col9\" class=\"data row0 col9\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col10\" class=\"data row0 col10\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col11\" class=\"data row0 col11\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col12\" class=\"data row0 col12\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row0_col13\" class=\"data row0 col13\" >0.09%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row1\" class=\"row_heading level0 row1\" >mean</th>\n",
       "      <td id=\"T_0554f_row1_col0\" class=\"data row1 col0\" >0.04%</td>\n",
       "      <td id=\"T_0554f_row1_col1\" class=\"data row1 col1\" >0.01%</td>\n",
       "      <td id=\"T_0554f_row1_col2\" class=\"data row1 col2\" >0.01%</td>\n",
       "      <td id=\"T_0554f_row1_col3\" class=\"data row1 col3\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row1_col4\" class=\"data row1 col4\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row1_col5\" class=\"data row1 col5\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row1_col6\" class=\"data row1 col6\" >0.09%</td>\n",
       "      <td id=\"T_0554f_row1_col7\" class=\"data row1 col7\" >0.01%</td>\n",
       "      <td id=\"T_0554f_row1_col8\" class=\"data row1 col8\" >0.21%</td>\n",
       "      <td id=\"T_0554f_row1_col9\" class=\"data row1 col9\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row1_col10\" class=\"data row1 col10\" >1.36%</td>\n",
       "      <td id=\"T_0554f_row1_col11\" class=\"data row1 col11\" >1.72%</td>\n",
       "      <td id=\"T_0554f_row1_col12\" class=\"data row1 col12\" >0.08%</td>\n",
       "      <td id=\"T_0554f_row1_col13\" class=\"data row1 col13\" >0.01%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row2\" class=\"row_heading level0 row2\" >std</th>\n",
       "      <td id=\"T_0554f_row2_col0\" class=\"data row2 col0\" >0.11%</td>\n",
       "      <td id=\"T_0554f_row2_col1\" class=\"data row2 col1\" >3.81%</td>\n",
       "      <td id=\"T_0554f_row2_col2\" class=\"data row2 col2\" >3.81%</td>\n",
       "      <td id=\"T_0554f_row2_col3\" class=\"data row2 col3\" >1.36%</td>\n",
       "      <td id=\"T_0554f_row2_col4\" class=\"data row2 col4\" >1.36%</td>\n",
       "      <td id=\"T_0554f_row2_col5\" class=\"data row2 col5\" >0.05%</td>\n",
       "      <td id=\"T_0554f_row2_col6\" class=\"data row2 col6\" >7.48%</td>\n",
       "      <td id=\"T_0554f_row2_col7\" class=\"data row2 col7\" >1.52%</td>\n",
       "      <td id=\"T_0554f_row2_col8\" class=\"data row2 col8\" >3.00%</td>\n",
       "      <td id=\"T_0554f_row2_col9\" class=\"data row2 col9\" >0.24%</td>\n",
       "      <td id=\"T_0554f_row2_col10\" class=\"data row2 col10\" >12.12%</td>\n",
       "      <td id=\"T_0554f_row2_col11\" class=\"data row2 col11\" >37.42%</td>\n",
       "      <td id=\"T_0554f_row2_col12\" class=\"data row2 col12\" >1.40%</td>\n",
       "      <td id=\"T_0554f_row2_col13\" class=\"data row2 col13\" >0.30%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row3\" class=\"row_heading level0 row3\" >min</th>\n",
       "      <td id=\"T_0554f_row3_col0\" class=\"data row3 col0\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row3_col1\" class=\"data row3 col1\" >1.56%</td>\n",
       "      <td id=\"T_0554f_row3_col2\" class=\"data row3 col2\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row3_col3\" class=\"data row3 col3\" >9.62%</td>\n",
       "      <td id=\"T_0554f_row3_col4\" class=\"data row3 col4\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row3_col5\" class=\"data row3 col5\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row3_col6\" class=\"data row3 col6\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row3_col7\" class=\"data row3 col7\" >79.28%</td>\n",
       "      <td id=\"T_0554f_row3_col8\" class=\"data row3 col8\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row3_col9\" class=\"data row3 col9\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row3_col10\" class=\"data row3 col10\" >23.81%</td>\n",
       "      <td id=\"T_0554f_row3_col11\" class=\"data row3 col11\" >inf%</td>\n",
       "      <td id=\"T_0554f_row3_col12\" class=\"data row3 col12\" >79.28%</td>\n",
       "      <td id=\"T_0554f_row3_col13\" class=\"data row3 col13\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row4\" class=\"row_heading level0 row4\" >50%</th>\n",
       "      <td id=\"T_0554f_row4_col0\" class=\"data row4 col0\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col1\" class=\"data row4 col1\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col2\" class=\"data row4 col2\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col3\" class=\"data row4 col3\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col4\" class=\"data row4 col4\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col5\" class=\"data row4 col5\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col6\" class=\"data row4 col6\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col7\" class=\"data row4 col7\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col8\" class=\"data row4 col8\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col9\" class=\"data row4 col9\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row4_col10\" class=\"data row4 col10\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row4_col11\" class=\"data row4 col11\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row4_col12\" class=\"data row4 col12\" >1.09%</td>\n",
       "      <td id=\"T_0554f_row4_col13\" class=\"data row4 col13\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0554f_level0_row5\" class=\"row_heading level0 row5\" >max</th>\n",
       "      <td id=\"T_0554f_row5_col0\" class=\"data row5 col0\" >0.00%</td>\n",
       "      <td id=\"T_0554f_row5_col1\" class=\"data row5 col1\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row5_col2\" class=\"data row5 col2\" >0.44%</td>\n",
       "      <td id=\"T_0554f_row5_col3\" class=\"data row5 col3\" >Div_0</td>\n",
       "      <td id=\"T_0554f_row5_col4\" class=\"data row5 col4\" >2.55%</td>\n",
       "      <td id=\"T_0554f_row5_col5\" class=\"data row5 col5\" >12.50%</td>\n",
       "      <td id=\"T_0554f_row5_col6\" class=\"data row5 col6\" >78.76%</td>\n",
       "      <td id=\"T_0554f_row5_col7\" class=\"data row5 col7\" >16.67%</td>\n",
       "      <td id=\"T_0554f_row5_col8\" class=\"data row5 col8\" >51.49%</td>\n",
       "      <td id=\"T_0554f_row5_col9\" class=\"data row5 col9\" >50.00%</td>\n",
       "      <td id=\"T_0554f_row5_col10\" class=\"data row5 col10\" >16.67%</td>\n",
       "      <td id=\"T_0554f_row5_col11\" class=\"data row5 col11\" >27.56%</td>\n",
       "      <td id=\"T_0554f_row5_col12\" class=\"data row5 col12\" >22.22%</td>\n",
       "      <td id=\"T_0554f_row5_col13\" class=\"data row5 col13\" >0.00%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1b789827148>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Date Data Distributions for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lpep_pickup_datetime</th>\n",
       "      <th>lpep_dropoff_datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>250000</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2014-07-16 22:19:04.859136</td>\n",
       "      <td>2014-07-16 22:34:17.136755712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2014-01-01 00:08:15</td>\n",
       "      <td>2014-01-01 00:19:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2014-07-19 09:01:23</td>\n",
       "      <td>2014-07-19 09:19:34.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2014-12-31 23:58:27</td>\n",
       "      <td>2015-01-01 00:23:52</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             lpep_pickup_datetime          lpep_dropoff_datetime\n",
       "count                      250000                         250000\n",
       "mean   2014-07-16 22:19:04.859136  2014-07-16 22:34:17.136755712\n",
       "min           2014-01-01 00:08:15            2014-01-01 00:19:38\n",
       "50%           2014-07-19 09:01:23     2014-07-19 09:19:34.500000\n",
       "max           2014-12-31 23:58:27            2015-01-01 00:23:52"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance in Date Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lpep_pickup_datetime</th>\n",
       "      <th>lpep_dropoff_datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0 days 00:10:48.811144192</td>\n",
       "      <td>0 days 00:10:39.149603584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0 days 00:08:15</td>\n",
       "      <td>0 days 00:18:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0 days 08:37:23</td>\n",
       "      <td>0 days 08:41:21.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0 days 00:01:10</td>\n",
       "      <td>0 days 00:01:57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            lpep_pickup_datetime      lpep_dropoff_datetime\n",
       "count                          0                          0\n",
       "mean   0 days 00:10:48.811144192  0 days 00:10:39.149603584\n",
       "min              0 days 00:08:15            0 days 00:18:23\n",
       "50%              0 days 08:37:23     0 days 08:41:21.500000\n",
       "max              0 days 00:01:10            0 days 00:01:57"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Categorical Data Value Counts for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "N    247603\n",
       "Y      2397\n",
       "Name: store_and_fwd_flag, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2    194409\n",
       "1     55591\n",
       "Name: vendorid, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2    145572\n",
       "1    103236\n",
       "4       608\n",
       "3       578\n",
       "5         6\n",
       "Name: payment_type, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eda_csv([first_c,second_c], {'lpep_pickup_datetime', 'lpep_dropoff_datetime'}, {'vendorid', 'store_and_fwd_flag','payment_type'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CitiBike Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bike_jan =  pd.read_csv('D://Capstone Data//2014citibike//2014-01 - Citi Bike trip data.csv')\n",
    "bike_feb =  pd.read_csv('D://Capstone Data//2014citibike//2014-02 - Citi Bike trip data.csv')\n",
    "bike_jul =  pd.read_csv('D://Capstone Data//2014citibike//2014-07 - Citi Bike trip data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tripduration                 int64\n",
       "starttime                   object\n",
       "stoptime                    object\n",
       "start station id             int64\n",
       "start station name          object\n",
       "start station latitude     float64\n",
       "start station longitude    float64\n",
       "end station id               int64\n",
       "end station name            object\n",
       "end station latitude       float64\n",
       "end station longitude      float64\n",
       "bikeid                       int64\n",
       "usertype                    object\n",
       "birth year                  object\n",
       "gender                       int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bike_jan.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taking a look at the data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tripduration</th>\n",
       "      <th>starttime</th>\n",
       "      <th>stoptime</th>\n",
       "      <th>start station id</th>\n",
       "      <th>start station name</th>\n",
       "      <th>start station latitude</th>\n",
       "      <th>start station longitude</th>\n",
       "      <th>end station id</th>\n",
       "      <th>end station name</th>\n",
       "      <th>end station latitude</th>\n",
       "      <th>end station longitude</th>\n",
       "      <th>bikeid</th>\n",
       "      <th>usertype</th>\n",
       "      <th>birth year</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>471</td>\n",
       "      <td>2014-01-01 00:00:06</td>\n",
       "      <td>2014-01-01 00:07:57</td>\n",
       "      <td>2009</td>\n",
       "      <td>Catherine St &amp; Monroe St</td>\n",
       "      <td>40.711174</td>\n",
       "      <td>-73.996826</td>\n",
       "      <td>263</td>\n",
       "      <td>Elizabeth St &amp; Hester St</td>\n",
       "      <td>40.717290</td>\n",
       "      <td>-73.996375</td>\n",
       "      <td>16379</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1986</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1494</td>\n",
       "      <td>2014-01-01 00:00:38</td>\n",
       "      <td>2014-01-01 00:25:32</td>\n",
       "      <td>536</td>\n",
       "      <td>1 Ave &amp; E 30 St</td>\n",
       "      <td>40.741444</td>\n",
       "      <td>-73.975361</td>\n",
       "      <td>259</td>\n",
       "      <td>South St &amp; Whitehall St</td>\n",
       "      <td>40.701221</td>\n",
       "      <td>-74.012342</td>\n",
       "      <td>15611</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1963</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>464</td>\n",
       "      <td>2014-01-01 00:03:59</td>\n",
       "      <td>2014-01-01 00:11:43</td>\n",
       "      <td>228</td>\n",
       "      <td>E 48 St &amp; 3 Ave</td>\n",
       "      <td>40.754601</td>\n",
       "      <td>-73.971879</td>\n",
       "      <td>2022</td>\n",
       "      <td>E 59 St &amp; Sutton Pl</td>\n",
       "      <td>40.758491</td>\n",
       "      <td>-73.959206</td>\n",
       "      <td>16613</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1991</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>373</td>\n",
       "      <td>2014-01-01 00:05:15</td>\n",
       "      <td>2014-01-01 00:11:28</td>\n",
       "      <td>519</td>\n",
       "      <td>Pershing Square N</td>\n",
       "      <td>40.751884</td>\n",
       "      <td>-73.977702</td>\n",
       "      <td>526</td>\n",
       "      <td>E 33 St &amp; 5 Ave</td>\n",
       "      <td>40.747659</td>\n",
       "      <td>-73.984907</td>\n",
       "      <td>15938</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1989</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>660</td>\n",
       "      <td>2014-01-01 00:05:18</td>\n",
       "      <td>2014-01-01 00:16:18</td>\n",
       "      <td>83</td>\n",
       "      <td>Atlantic Ave &amp; Fort Greene Pl</td>\n",
       "      <td>40.683826</td>\n",
       "      <td>-73.976323</td>\n",
       "      <td>436</td>\n",
       "      <td>Hancock St &amp; Bedford Ave</td>\n",
       "      <td>40.682166</td>\n",
       "      <td>-73.953990</td>\n",
       "      <td>19830</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1990</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>330</td>\n",
       "      <td>2014-01-01 00:05:55</td>\n",
       "      <td>2014-01-01 00:11:25</td>\n",
       "      <td>422</td>\n",
       "      <td>W 59 St &amp; 10 Ave</td>\n",
       "      <td>40.770513</td>\n",
       "      <td>-73.988038</td>\n",
       "      <td>526</td>\n",
       "      <td>E 33 St &amp; 5 Ave</td>\n",
       "      <td>40.747659</td>\n",
       "      <td>-73.984907</td>\n",
       "      <td>17343</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1987</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>261</td>\n",
       "      <td>2014-01-01 00:06:04</td>\n",
       "      <td>2014-01-01 00:10:25</td>\n",
       "      <td>516</td>\n",
       "      <td>E 47 St &amp; 1 Ave</td>\n",
       "      <td>40.752069</td>\n",
       "      <td>-73.967844</td>\n",
       "      <td>167</td>\n",
       "      <td>E 39 St &amp; 3 Ave</td>\n",
       "      <td>40.748901</td>\n",
       "      <td>-73.976049</td>\n",
       "      <td>17880</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1983</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>337</td>\n",
       "      <td>2014-01-01 00:06:41</td>\n",
       "      <td>2014-01-01 00:12:18</td>\n",
       "      <td>380</td>\n",
       "      <td>W 4 St &amp; 7 Ave S</td>\n",
       "      <td>40.734011</td>\n",
       "      <td>-74.002939</td>\n",
       "      <td>435</td>\n",
       "      <td>W 21 St &amp; 6 Ave</td>\n",
       "      <td>40.741740</td>\n",
       "      <td>-73.994156</td>\n",
       "      <td>16275</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1963</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>429</td>\n",
       "      <td>2014-01-01 00:07:33</td>\n",
       "      <td>2014-01-01 00:14:42</td>\n",
       "      <td>296</td>\n",
       "      <td>Division St &amp; Bowery</td>\n",
       "      <td>40.714131</td>\n",
       "      <td>-73.997047</td>\n",
       "      <td>306</td>\n",
       "      <td>Cliff St &amp; Fulton St</td>\n",
       "      <td>40.708235</td>\n",
       "      <td>-74.005301</td>\n",
       "      <td>17318</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1972</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1025</td>\n",
       "      <td>2014-01-01 00:08:27</td>\n",
       "      <td>2014-01-01 00:25:32</td>\n",
       "      <td>540</td>\n",
       "      <td>Lexington Ave &amp; E 26 St</td>\n",
       "      <td>40.741473</td>\n",
       "      <td>-73.983209</td>\n",
       "      <td>447</td>\n",
       "      <td>8 Ave &amp; W 52 St</td>\n",
       "      <td>40.763707</td>\n",
       "      <td>-73.985162</td>\n",
       "      <td>15525</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1981</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>718</td>\n",
       "      <td>2014-01-01 00:09:32</td>\n",
       "      <td>2014-01-01 00:21:30</td>\n",
       "      <td>263</td>\n",
       "      <td>Elizabeth St &amp; Hester St</td>\n",
       "      <td>40.717290</td>\n",
       "      <td>-73.996375</td>\n",
       "      <td>251</td>\n",
       "      <td>Mott St &amp; Prince St</td>\n",
       "      <td>40.723180</td>\n",
       "      <td>-73.994800</td>\n",
       "      <td>15693</td>\n",
       "      <td>Customer</td>\n",
       "      <td>\\N</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>786</td>\n",
       "      <td>2014-01-01 00:10:59</td>\n",
       "      <td>2014-01-01 00:24:05</td>\n",
       "      <td>153</td>\n",
       "      <td>E 40 St &amp; 5 Ave</td>\n",
       "      <td>40.752062</td>\n",
       "      <td>-73.981632</td>\n",
       "      <td>290</td>\n",
       "      <td>2 Ave &amp; E 58 St</td>\n",
       "      <td>40.760203</td>\n",
       "      <td>-73.964785</td>\n",
       "      <td>15281</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1968</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>267</td>\n",
       "      <td>2014-01-01 00:11:17</td>\n",
       "      <td>2014-01-01 00:15:44</td>\n",
       "      <td>151</td>\n",
       "      <td>Cleveland Pl &amp; Spring St</td>\n",
       "      <td>40.721816</td>\n",
       "      <td>-73.997203</td>\n",
       "      <td>410</td>\n",
       "      <td>Suffolk St &amp; Stanton St</td>\n",
       "      <td>40.720664</td>\n",
       "      <td>-73.985180</td>\n",
       "      <td>15159</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1983</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>744</td>\n",
       "      <td>2014-01-01 00:12:23</td>\n",
       "      <td>2014-01-01 00:24:47</td>\n",
       "      <td>450</td>\n",
       "      <td>W 49 St &amp; 8 Ave</td>\n",
       "      <td>40.762272</td>\n",
       "      <td>-73.987882</td>\n",
       "      <td>505</td>\n",
       "      <td>6 Ave &amp; W 33 St</td>\n",
       "      <td>40.749013</td>\n",
       "      <td>-73.988484</td>\n",
       "      <td>15157</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1976</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>704</td>\n",
       "      <td>2014-01-01 00:12:25</td>\n",
       "      <td>2014-01-01 00:24:09</td>\n",
       "      <td>331</td>\n",
       "      <td>Pike St &amp; Monroe St</td>\n",
       "      <td>40.711731</td>\n",
       "      <td>-73.991930</td>\n",
       "      <td>195</td>\n",
       "      <td>Liberty St &amp; Broadway</td>\n",
       "      <td>40.709056</td>\n",
       "      <td>-74.010434</td>\n",
       "      <td>17080</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1980</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    tripduration            starttime             stoptime  start station id  \\\n",
       "0            471  2014-01-01 00:00:06  2014-01-01 00:07:57              2009   \n",
       "1           1494  2014-01-01 00:00:38  2014-01-01 00:25:32               536   \n",
       "2            464  2014-01-01 00:03:59  2014-01-01 00:11:43               228   \n",
       "3            373  2014-01-01 00:05:15  2014-01-01 00:11:28               519   \n",
       "4            660  2014-01-01 00:05:18  2014-01-01 00:16:18                83   \n",
       "5            330  2014-01-01 00:05:55  2014-01-01 00:11:25               422   \n",
       "6            261  2014-01-01 00:06:04  2014-01-01 00:10:25               516   \n",
       "7            337  2014-01-01 00:06:41  2014-01-01 00:12:18               380   \n",
       "8            429  2014-01-01 00:07:33  2014-01-01 00:14:42               296   \n",
       "9           1025  2014-01-01 00:08:27  2014-01-01 00:25:32               540   \n",
       "10           718  2014-01-01 00:09:32  2014-01-01 00:21:30               263   \n",
       "11           786  2014-01-01 00:10:59  2014-01-01 00:24:05               153   \n",
       "12           267  2014-01-01 00:11:17  2014-01-01 00:15:44               151   \n",
       "13           744  2014-01-01 00:12:23  2014-01-01 00:24:47               450   \n",
       "14           704  2014-01-01 00:12:25  2014-01-01 00:24:09               331   \n",
       "\n",
       "               start station name  start station latitude  \\\n",
       "0        Catherine St & Monroe St               40.711174   \n",
       "1                 1 Ave & E 30 St               40.741444   \n",
       "2                 E 48 St & 3 Ave               40.754601   \n",
       "3               Pershing Square N               40.751884   \n",
       "4   Atlantic Ave & Fort Greene Pl               40.683826   \n",
       "5                W 59 St & 10 Ave               40.770513   \n",
       "6                 E 47 St & 1 Ave               40.752069   \n",
       "7                W 4 St & 7 Ave S               40.734011   \n",
       "8            Division St & Bowery               40.714131   \n",
       "9         Lexington Ave & E 26 St               40.741473   \n",
       "10       Elizabeth St & Hester St               40.717290   \n",
       "11                E 40 St & 5 Ave               40.752062   \n",
       "12       Cleveland Pl & Spring St               40.721816   \n",
       "13                W 49 St & 8 Ave               40.762272   \n",
       "14            Pike St & Monroe St               40.711731   \n",
       "\n",
       "    start station longitude  end station id          end station name  \\\n",
       "0                -73.996826             263  Elizabeth St & Hester St   \n",
       "1                -73.975361             259   South St & Whitehall St   \n",
       "2                -73.971879            2022       E 59 St & Sutton Pl   \n",
       "3                -73.977702             526           E 33 St & 5 Ave   \n",
       "4                -73.976323             436  Hancock St & Bedford Ave   \n",
       "5                -73.988038             526           E 33 St & 5 Ave   \n",
       "6                -73.967844             167           E 39 St & 3 Ave   \n",
       "7                -74.002939             435           W 21 St & 6 Ave   \n",
       "8                -73.997047             306      Cliff St & Fulton St   \n",
       "9                -73.983209             447           8 Ave & W 52 St   \n",
       "10               -73.996375             251       Mott St & Prince St   \n",
       "11               -73.981632             290           2 Ave & E 58 St   \n",
       "12               -73.997203             410   Suffolk St & Stanton St   \n",
       "13               -73.987882             505           6 Ave & W 33 St   \n",
       "14               -73.991930             195     Liberty St & Broadway   \n",
       "\n",
       "    end station latitude  end station longitude  bikeid    usertype  \\\n",
       "0              40.717290             -73.996375   16379  Subscriber   \n",
       "1              40.701221             -74.012342   15611  Subscriber   \n",
       "2              40.758491             -73.959206   16613  Subscriber   \n",
       "3              40.747659             -73.984907   15938  Subscriber   \n",
       "4              40.682166             -73.953990   19830  Subscriber   \n",
       "5              40.747659             -73.984907   17343  Subscriber   \n",
       "6              40.748901             -73.976049   17880  Subscriber   \n",
       "7              40.741740             -73.994156   16275  Subscriber   \n",
       "8              40.708235             -74.005301   17318  Subscriber   \n",
       "9              40.763707             -73.985162   15525  Subscriber   \n",
       "10             40.723180             -73.994800   15693    Customer   \n",
       "11             40.760203             -73.964785   15281  Subscriber   \n",
       "12             40.720664             -73.985180   15159  Subscriber   \n",
       "13             40.749013             -73.988484   15157  Subscriber   \n",
       "14             40.709056             -74.010434   17080  Subscriber   \n",
       "\n",
       "   birth year  gender  \n",
       "0        1986       1  \n",
       "1        1963       1  \n",
       "2        1991       1  \n",
       "3        1989       1  \n",
       "4        1990       1  \n",
       "5        1987       1  \n",
       "6        1983       1  \n",
       "7        1963       1  \n",
       "8        1972       2  \n",
       "9        1981       1  \n",
       "10         \\N       0  \n",
       "11       1968       1  \n",
       "12       1983       1  \n",
       "13       1976       1  \n",
       "14       1980       2  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Looking at data types\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_efd35_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"col_heading level0 col0\" >tripduration</th>\n",
       "      <th class=\"col_heading level0 col1\" >starttime</th>\n",
       "      <th class=\"col_heading level0 col2\" >stoptime</th>\n",
       "      <th class=\"col_heading level0 col3\" >start station id</th>\n",
       "      <th class=\"col_heading level0 col4\" >start station name</th>\n",
       "      <th class=\"col_heading level0 col5\" >start station latitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >start station longitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >end station id</th>\n",
       "      <th class=\"col_heading level0 col8\" >end station name</th>\n",
       "      <th class=\"col_heading level0 col9\" >end station latitude</th>\n",
       "      <th class=\"col_heading level0 col10\" >end station longitude</th>\n",
       "      <th class=\"col_heading level0 col11\" >bikeid</th>\n",
       "      <th class=\"col_heading level0 col12\" >usertype</th>\n",
       "      <th class=\"col_heading level0 col13\" >birth year</th>\n",
       "      <th class=\"col_heading level0 col14\" >gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_efd35_row0_col0\" class=\"data row0 col0\" >int64</td>\n",
       "      <td id=\"T_efd35_row0_col1\" class=\"data row0 col1\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col2\" class=\"data row0 col2\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col3\" class=\"data row0 col3\" >int64</td>\n",
       "      <td id=\"T_efd35_row0_col4\" class=\"data row0 col4\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col5\" class=\"data row0 col5\" >float64</td>\n",
       "      <td id=\"T_efd35_row0_col6\" class=\"data row0 col6\" >float64</td>\n",
       "      <td id=\"T_efd35_row0_col7\" class=\"data row0 col7\" >int64</td>\n",
       "      <td id=\"T_efd35_row0_col8\" class=\"data row0 col8\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col9\" class=\"data row0 col9\" >float64</td>\n",
       "      <td id=\"T_efd35_row0_col10\" class=\"data row0 col10\" >float64</td>\n",
       "      <td id=\"T_efd35_row0_col11\" class=\"data row0 col11\" >int64</td>\n",
       "      <td id=\"T_efd35_row0_col12\" class=\"data row0 col12\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col13\" class=\"data row0 col13\" >object</td>\n",
       "      <td id=\"T_efd35_row0_col14\" class=\"data row0 col14\" >int64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x242e97ea488>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Numeric Data Distribution for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tripduration</th>\n",
       "      <th>start station id</th>\n",
       "      <th>start station latitude</th>\n",
       "      <th>start station longitude</th>\n",
       "      <th>end station id</th>\n",
       "      <th>end station latitude</th>\n",
       "      <th>end station longitude</th>\n",
       "      <th>bikeid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3.004000e+05</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "      <td>300400.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>7.339689e+02</td>\n",
       "      <td>436.936598</td>\n",
       "      <td>40.734996</td>\n",
       "      <td>-73.990135</td>\n",
       "      <td>437.742763</td>\n",
       "      <td>40.734814</td>\n",
       "      <td>-73.990328</td>\n",
       "      <td>17992.082044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.271464e+03</td>\n",
       "      <td>323.895643</td>\n",
       "      <td>0.019408</td>\n",
       "      <td>0.011779</td>\n",
       "      <td>328.931404</td>\n",
       "      <td>0.019383</td>\n",
       "      <td>0.011854</td>\n",
       "      <td>2000.832564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>6.000000e+01</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>40.680342</td>\n",
       "      <td>-74.017134</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>40.680342</td>\n",
       "      <td>-74.017134</td>\n",
       "      <td>14529.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.130000e+02</td>\n",
       "      <td>406.000000</td>\n",
       "      <td>40.737050</td>\n",
       "      <td>-73.990093</td>\n",
       "      <td>405.000000</td>\n",
       "      <td>40.736529</td>\n",
       "      <td>-73.990214</td>\n",
       "      <td>17971.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.990440e+06</td>\n",
       "      <td>3002.000000</td>\n",
       "      <td>40.770513</td>\n",
       "      <td>-73.950048</td>\n",
       "      <td>3002.000000</td>\n",
       "      <td>40.770513</td>\n",
       "      <td>-73.950048</td>\n",
       "      <td>21536.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       tripduration  start station id  start station latitude  \\\n",
       "count  3.004000e+05     300400.000000           300400.000000   \n",
       "mean   7.339689e+02        436.936598               40.734996   \n",
       "std    5.271464e+03        323.895643                0.019408   \n",
       "min    6.000000e+01         72.000000               40.680342   \n",
       "50%    5.130000e+02        406.000000               40.737050   \n",
       "max    1.990440e+06       3002.000000               40.770513   \n",
       "\n",
       "       start station longitude  end station id  end station latitude  \\\n",
       "count            300400.000000   300400.000000         300400.000000   \n",
       "mean                -73.990135      437.742763             40.734814   \n",
       "std                   0.011779      328.931404              0.019383   \n",
       "min                 -74.017134       72.000000             40.680342   \n",
       "50%                 -73.990093      405.000000             40.736529   \n",
       "max                 -73.950048     3002.000000             40.770513   \n",
       "\n",
       "       end station longitude         bikeid  \n",
       "count          300400.000000  300400.000000  \n",
       "mean              -73.990328   17992.082044  \n",
       "std                 0.011854    2000.832564  \n",
       "min               -74.017134   14529.000000  \n",
       "50%               -73.990214   17971.000000  \n",
       "max               -73.950048   21536.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance Percent in Numeric Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_91d22_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >tripduration</th>\n",
       "      <th class=\"col_heading level0 col1\" >start station id</th>\n",
       "      <th class=\"col_heading level0 col2\" >start station latitude</th>\n",
       "      <th class=\"col_heading level0 col3\" >start station longitude</th>\n",
       "      <th class=\"col_heading level0 col4\" >end station id</th>\n",
       "      <th class=\"col_heading level0 col5\" >end station latitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >end station longitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >bikeid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row0\" class=\"row_heading level0 row0\" >count</th>\n",
       "      <td id=\"T_91d22_row0_col0\" class=\"data row0 col0\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col1\" class=\"data row0 col1\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col2\" class=\"data row0 col2\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col3\" class=\"data row0 col3\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col4\" class=\"data row0 col4\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col5\" class=\"data row0 col5\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col6\" class=\"data row0 col6\" >222.52%</td>\n",
       "      <td id=\"T_91d22_row0_col7\" class=\"data row0 col7\" >222.52%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row1\" class=\"row_heading level0 row1\" >mean</th>\n",
       "      <td id=\"T_91d22_row1_col0\" class=\"data row1 col0\" >17.18%</td>\n",
       "      <td id=\"T_91d22_row1_col1\" class=\"data row1 col1\" >4.02%</td>\n",
       "      <td id=\"T_91d22_row1_col2\" class=\"data row1 col2\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row1_col3\" class=\"data row1 col3\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row1_col4\" class=\"data row1 col4\" >3.82%</td>\n",
       "      <td id=\"T_91d22_row1_col5\" class=\"data row1 col5\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row1_col6\" class=\"data row1 col6\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row1_col7\" class=\"data row1 col7\" >0.85%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row2\" class=\"row_heading level0 row2\" >std</th>\n",
       "      <td id=\"T_91d22_row2_col0\" class=\"data row2 col0\" >83.61%</td>\n",
       "      <td id=\"T_91d22_row2_col1\" class=\"data row2 col1\" >17.40%</td>\n",
       "      <td id=\"T_91d22_row2_col2\" class=\"data row2 col2\" >0.95%</td>\n",
       "      <td id=\"T_91d22_row2_col3\" class=\"data row2 col3\" >5.18%</td>\n",
       "      <td id=\"T_91d22_row2_col4\" class=\"data row2 col4\" >16.56%</td>\n",
       "      <td id=\"T_91d22_row2_col5\" class=\"data row2 col5\" >1.31%</td>\n",
       "      <td id=\"T_91d22_row2_col6\" class=\"data row2 col6\" >5.06%</td>\n",
       "      <td id=\"T_91d22_row2_col7\" class=\"data row2 col7\" >5.03%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row3\" class=\"row_heading level0 row3\" >min</th>\n",
       "      <td id=\"T_91d22_row3_col0\" class=\"data row3 col0\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col1\" class=\"data row3 col1\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col2\" class=\"data row3 col2\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col3\" class=\"data row3 col3\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col4\" class=\"data row3 col4\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col5\" class=\"data row3 col5\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col6\" class=\"data row3 col6\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row3_col7\" class=\"data row3 col7\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row4\" class=\"row_heading level0 row4\" >50%</th>\n",
       "      <td id=\"T_91d22_row4_col0\" class=\"data row4 col0\" >26.90%</td>\n",
       "      <td id=\"T_91d22_row4_col1\" class=\"data row4 col1\" >0.49%</td>\n",
       "      <td id=\"T_91d22_row4_col2\" class=\"data row4 col2\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row4_col3\" class=\"data row4 col3\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row4_col4\" class=\"data row4 col4\" >0.25%</td>\n",
       "      <td id=\"T_91d22_row4_col5\" class=\"data row4 col5\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row4_col6\" class=\"data row4 col6\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row4_col7\" class=\"data row4 col7\" >0.87%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_91d22_level0_row5\" class=\"row_heading level0 row5\" >max</th>\n",
       "      <td id=\"T_91d22_row5_col0\" class=\"data row5 col0\" >98.91%</td>\n",
       "      <td id=\"T_91d22_row5_col1\" class=\"data row5 col1\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col2\" class=\"data row5 col2\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col3\" class=\"data row5 col3\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col4\" class=\"data row5 col4\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col5\" class=\"data row5 col5\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col6\" class=\"data row5 col6\" >0.00%</td>\n",
       "      <td id=\"T_91d22_row5_col7\" class=\"data row5 col7\" >0.69%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x242f0e90808>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Date Data Distributions for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stoptime</th>\n",
       "      <th>starttime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>300400</td>\n",
       "      <td>300400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2014-01-17 11:14:15.004180224</td>\n",
       "      <td>2014-01-17 11:02:01.035226112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2014-01-01 00:07:57</td>\n",
       "      <td>2014-01-01 00:00:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2014-01-16 13:33:55.500000</td>\n",
       "      <td>2014-01-16 13:22:44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2014-02-20 18:32:08</td>\n",
       "      <td>2014-01-31 23:58:43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            stoptime                      starttime\n",
       "count                         300400                         300400\n",
       "mean   2014-01-17 11:14:15.004180224  2014-01-17 11:02:01.035226112\n",
       "min              2014-01-01 00:07:57            2014-01-01 00:00:06\n",
       "50%       2014-01-16 13:33:55.500000            2014-01-16 13:22:44\n",
       "max              2014-02-20 18:32:08            2014-01-31 23:58:43"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance in Date Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stoptime</th>\n",
       "      <th>starttime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>668442</td>\n",
       "      <td>668442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>180 days 21:40:02.880894720</td>\n",
       "      <td>180 days 21:37:56.765794048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>180 days 23:58:04</td>\n",
       "      <td>180 days 23:59:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>182 days 04:55:44.500000</td>\n",
       "      <td>182 days 04:52:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>161 days 07:11:16</td>\n",
       "      <td>181 days 00:01:15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          stoptime                    starttime\n",
       "count                       668442                       668442\n",
       "mean   180 days 21:40:02.880894720  180 days 21:37:56.765794048\n",
       "min              180 days 23:58:04            180 days 23:59:58\n",
       "50%       182 days 04:55:44.500000            182 days 04:52:42\n",
       "max              161 days 07:11:16            181 days 00:01:15"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Categorical Data Value Counts for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1    236106\n",
       "2     57025\n",
       "0      7269\n",
       "Name: gender, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Subscriber    293146\n",
       "Customer        7254\n",
       "Name: usertype, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eda_csv([bike_jan, bike_feb], {'starttime','stoptime'}, {'gender', 'usertype'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taking a look at the data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tripduration</th>\n",
       "      <th>starttime</th>\n",
       "      <th>stoptime</th>\n",
       "      <th>start station id</th>\n",
       "      <th>start station name</th>\n",
       "      <th>start station latitude</th>\n",
       "      <th>start station longitude</th>\n",
       "      <th>end station id</th>\n",
       "      <th>end station name</th>\n",
       "      <th>end station latitude</th>\n",
       "      <th>end station longitude</th>\n",
       "      <th>bikeid</th>\n",
       "      <th>usertype</th>\n",
       "      <th>birth year</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>404</td>\n",
       "      <td>2014-07-01 00:00:04</td>\n",
       "      <td>2014-07-01 00:06:48</td>\n",
       "      <td>545</td>\n",
       "      <td>E 23 St &amp; 1 Ave</td>\n",
       "      <td>40.736502</td>\n",
       "      <td>-73.978095</td>\n",
       "      <td>402</td>\n",
       "      <td>Broadway &amp; E 22 St</td>\n",
       "      <td>40.740343</td>\n",
       "      <td>-73.989551</td>\n",
       "      <td>19578</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1987</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>850</td>\n",
       "      <td>2014-07-01 00:00:06</td>\n",
       "      <td>2014-07-01 00:14:16</td>\n",
       "      <td>238</td>\n",
       "      <td>Bank St &amp; Washington St</td>\n",
       "      <td>40.736197</td>\n",
       "      <td>-74.008592</td>\n",
       "      <td>458</td>\n",
       "      <td>11 Ave &amp; W 27 St</td>\n",
       "      <td>40.751396</td>\n",
       "      <td>-74.005226</td>\n",
       "      <td>19224</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1987</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1550</td>\n",
       "      <td>2014-07-01 00:00:21</td>\n",
       "      <td>2014-07-01 00:26:11</td>\n",
       "      <td>223</td>\n",
       "      <td>W 13 St &amp; 7 Ave</td>\n",
       "      <td>40.737815</td>\n",
       "      <td>-73.999947</td>\n",
       "      <td>539</td>\n",
       "      <td>Metropolitan Ave &amp; Bedford Ave</td>\n",
       "      <td>40.715348</td>\n",
       "      <td>-73.960241</td>\n",
       "      <td>17627</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1973</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>397</td>\n",
       "      <td>2014-07-01 00:00:29</td>\n",
       "      <td>2014-07-01 00:07:06</td>\n",
       "      <td>224</td>\n",
       "      <td>Spruce St &amp; Nassau St</td>\n",
       "      <td>40.711464</td>\n",
       "      <td>-74.005524</td>\n",
       "      <td>2008</td>\n",
       "      <td>Little West St &amp; 1 Pl</td>\n",
       "      <td>40.705693</td>\n",
       "      <td>-74.016777</td>\n",
       "      <td>15304</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1982</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>609</td>\n",
       "      <td>2014-07-01 00:00:37</td>\n",
       "      <td>2014-07-01 00:10:46</td>\n",
       "      <td>346</td>\n",
       "      <td>Bank St &amp; Hudson St</td>\n",
       "      <td>40.736529</td>\n",
       "      <td>-74.006180</td>\n",
       "      <td>521</td>\n",
       "      <td>8 Ave &amp; W 31 St</td>\n",
       "      <td>40.750450</td>\n",
       "      <td>-73.994811</td>\n",
       "      <td>20062</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1972</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2245</td>\n",
       "      <td>2014-07-01 00:01:09</td>\n",
       "      <td>2014-07-01 00:38:34</td>\n",
       "      <td>416</td>\n",
       "      <td>Cumberland St &amp; Lafayette Ave</td>\n",
       "      <td>40.687534</td>\n",
       "      <td>-73.972652</td>\n",
       "      <td>473</td>\n",
       "      <td>Rivington St &amp; Chrystie St</td>\n",
       "      <td>40.721101</td>\n",
       "      <td>-73.991925</td>\n",
       "      <td>20653</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1976</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1323</td>\n",
       "      <td>2014-07-01 00:01:15</td>\n",
       "      <td>2014-07-01 00:23:18</td>\n",
       "      <td>501</td>\n",
       "      <td>FDR Drive &amp; E 35 St</td>\n",
       "      <td>40.744219</td>\n",
       "      <td>-73.971212</td>\n",
       "      <td>501</td>\n",
       "      <td>FDR Drive &amp; E 35 St</td>\n",
       "      <td>40.744219</td>\n",
       "      <td>-73.971212</td>\n",
       "      <td>21460</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1993</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>320</td>\n",
       "      <td>2014-07-01 00:01:16</td>\n",
       "      <td>2014-07-01 00:06:36</td>\n",
       "      <td>475</td>\n",
       "      <td>E 16 St &amp; Irving Pl</td>\n",
       "      <td>40.735243</td>\n",
       "      <td>-73.987586</td>\n",
       "      <td>116</td>\n",
       "      <td>W 17 St &amp; 8 Ave</td>\n",
       "      <td>40.741776</td>\n",
       "      <td>-74.001497</td>\n",
       "      <td>16746</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1985</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2430</td>\n",
       "      <td>2014-07-01 00:01:18</td>\n",
       "      <td>2014-07-01 00:41:48</td>\n",
       "      <td>469</td>\n",
       "      <td>Broadway &amp; W 53 St</td>\n",
       "      <td>40.763441</td>\n",
       "      <td>-73.982681</td>\n",
       "      <td>445</td>\n",
       "      <td>E 10 St &amp; Avenue A</td>\n",
       "      <td>40.727408</td>\n",
       "      <td>-73.981420</td>\n",
       "      <td>19441</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1983</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>700</td>\n",
       "      <td>2014-07-01 00:01:21</td>\n",
       "      <td>2014-07-01 00:13:01</td>\n",
       "      <td>320</td>\n",
       "      <td>Leonard St &amp; Church St</td>\n",
       "      <td>40.717571</td>\n",
       "      <td>-74.005549</td>\n",
       "      <td>393</td>\n",
       "      <td>E 5 St &amp; Avenue C</td>\n",
       "      <td>40.722992</td>\n",
       "      <td>-73.979955</td>\n",
       "      <td>17267</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1983</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>907</td>\n",
       "      <td>2014-07-01 00:01:22</td>\n",
       "      <td>2014-07-01 00:16:29</td>\n",
       "      <td>501</td>\n",
       "      <td>FDR Drive &amp; E 35 St</td>\n",
       "      <td>40.744219</td>\n",
       "      <td>-73.971212</td>\n",
       "      <td>501</td>\n",
       "      <td>FDR Drive &amp; E 35 St</td>\n",
       "      <td>40.744219</td>\n",
       "      <td>-73.971212</td>\n",
       "      <td>21355</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1990</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>647</td>\n",
       "      <td>2014-07-01 00:01:35</td>\n",
       "      <td>2014-07-01 00:12:22</td>\n",
       "      <td>500</td>\n",
       "      <td>Broadway &amp; W 51 St</td>\n",
       "      <td>40.762288</td>\n",
       "      <td>-73.983362</td>\n",
       "      <td>533</td>\n",
       "      <td>Broadway &amp; W 39 St</td>\n",
       "      <td>40.752996</td>\n",
       "      <td>-73.987216</td>\n",
       "      <td>15001</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1959</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>321</td>\n",
       "      <td>2014-07-01 00:01:46</td>\n",
       "      <td>2014-07-01 00:07:07</td>\n",
       "      <td>384</td>\n",
       "      <td>Fulton St &amp; Waverly Ave</td>\n",
       "      <td>40.683178</td>\n",
       "      <td>-73.965964</td>\n",
       "      <td>396</td>\n",
       "      <td>Lefferts Pl &amp; Franklin Ave</td>\n",
       "      <td>40.680342</td>\n",
       "      <td>-73.955769</td>\n",
       "      <td>19861</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1987</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1326</td>\n",
       "      <td>2014-07-01 00:01:50</td>\n",
       "      <td>2014-07-01 00:23:56</td>\n",
       "      <td>382</td>\n",
       "      <td>University Pl &amp; E 14 St</td>\n",
       "      <td>40.734927</td>\n",
       "      <td>-73.992005</td>\n",
       "      <td>525</td>\n",
       "      <td>W 34 St &amp; 11 Ave</td>\n",
       "      <td>40.755942</td>\n",
       "      <td>-74.002116</td>\n",
       "      <td>14604</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1986</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>454</td>\n",
       "      <td>2014-07-01 00:01:54</td>\n",
       "      <td>2014-07-01 00:09:28</td>\n",
       "      <td>438</td>\n",
       "      <td>St Marks Pl &amp; 1 Ave</td>\n",
       "      <td>40.727791</td>\n",
       "      <td>-73.985649</td>\n",
       "      <td>263</td>\n",
       "      <td>Elizabeth St &amp; Hester St</td>\n",
       "      <td>40.717290</td>\n",
       "      <td>-73.996375</td>\n",
       "      <td>21141</td>\n",
       "      <td>Subscriber</td>\n",
       "      <td>1984</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    tripduration            starttime             stoptime  start station id  \\\n",
       "0            404  2014-07-01 00:00:04  2014-07-01 00:06:48               545   \n",
       "1            850  2014-07-01 00:00:06  2014-07-01 00:14:16               238   \n",
       "2           1550  2014-07-01 00:00:21  2014-07-01 00:26:11               223   \n",
       "3            397  2014-07-01 00:00:29  2014-07-01 00:07:06               224   \n",
       "4            609  2014-07-01 00:00:37  2014-07-01 00:10:46               346   \n",
       "5           2245  2014-07-01 00:01:09  2014-07-01 00:38:34               416   \n",
       "6           1323  2014-07-01 00:01:15  2014-07-01 00:23:18               501   \n",
       "7            320  2014-07-01 00:01:16  2014-07-01 00:06:36               475   \n",
       "8           2430  2014-07-01 00:01:18  2014-07-01 00:41:48               469   \n",
       "9            700  2014-07-01 00:01:21  2014-07-01 00:13:01               320   \n",
       "10           907  2014-07-01 00:01:22  2014-07-01 00:16:29               501   \n",
       "11           647  2014-07-01 00:01:35  2014-07-01 00:12:22               500   \n",
       "12           321  2014-07-01 00:01:46  2014-07-01 00:07:07               384   \n",
       "13          1326  2014-07-01 00:01:50  2014-07-01 00:23:56               382   \n",
       "14           454  2014-07-01 00:01:54  2014-07-01 00:09:28               438   \n",
       "\n",
       "               start station name  start station latitude  \\\n",
       "0                 E 23 St & 1 Ave               40.736502   \n",
       "1         Bank St & Washington St               40.736197   \n",
       "2                 W 13 St & 7 Ave               40.737815   \n",
       "3           Spruce St & Nassau St               40.711464   \n",
       "4             Bank St & Hudson St               40.736529   \n",
       "5   Cumberland St & Lafayette Ave               40.687534   \n",
       "6             FDR Drive & E 35 St               40.744219   \n",
       "7             E 16 St & Irving Pl               40.735243   \n",
       "8              Broadway & W 53 St               40.763441   \n",
       "9          Leonard St & Church St               40.717571   \n",
       "10            FDR Drive & E 35 St               40.744219   \n",
       "11             Broadway & W 51 St               40.762288   \n",
       "12        Fulton St & Waverly Ave               40.683178   \n",
       "13        University Pl & E 14 St               40.734927   \n",
       "14            St Marks Pl & 1 Ave               40.727791   \n",
       "\n",
       "    start station longitude  end station id                end station name  \\\n",
       "0                -73.978095             402              Broadway & E 22 St   \n",
       "1                -74.008592             458                11 Ave & W 27 St   \n",
       "2                -73.999947             539  Metropolitan Ave & Bedford Ave   \n",
       "3                -74.005524            2008           Little West St & 1 Pl   \n",
       "4                -74.006180             521                 8 Ave & W 31 St   \n",
       "5                -73.972652             473      Rivington St & Chrystie St   \n",
       "6                -73.971212             501             FDR Drive & E 35 St   \n",
       "7                -73.987586             116                 W 17 St & 8 Ave   \n",
       "8                -73.982681             445              E 10 St & Avenue A   \n",
       "9                -74.005549             393               E 5 St & Avenue C   \n",
       "10               -73.971212             501             FDR Drive & E 35 St   \n",
       "11               -73.983362             533              Broadway & W 39 St   \n",
       "12               -73.965964             396      Lefferts Pl & Franklin Ave   \n",
       "13               -73.992005             525                W 34 St & 11 Ave   \n",
       "14               -73.985649             263        Elizabeth St & Hester St   \n",
       "\n",
       "    end station latitude  end station longitude  bikeid    usertype  \\\n",
       "0              40.740343             -73.989551   19578  Subscriber   \n",
       "1              40.751396             -74.005226   19224  Subscriber   \n",
       "2              40.715348             -73.960241   17627  Subscriber   \n",
       "3              40.705693             -74.016777   15304  Subscriber   \n",
       "4              40.750450             -73.994811   20062  Subscriber   \n",
       "5              40.721101             -73.991925   20653  Subscriber   \n",
       "6              40.744219             -73.971212   21460  Subscriber   \n",
       "7              40.741776             -74.001497   16746  Subscriber   \n",
       "8              40.727408             -73.981420   19441  Subscriber   \n",
       "9              40.722992             -73.979955   17267  Subscriber   \n",
       "10             40.744219             -73.971212   21355  Subscriber   \n",
       "11             40.752996             -73.987216   15001  Subscriber   \n",
       "12             40.680342             -73.955769   19861  Subscriber   \n",
       "13             40.755942             -74.002116   14604  Subscriber   \n",
       "14             40.717290             -73.996375   21141  Subscriber   \n",
       "\n",
       "   birth year  gender  \n",
       "0        1987       2  \n",
       "1        1987       1  \n",
       "2        1973       2  \n",
       "3        1982       1  \n",
       "4        1972       2  \n",
       "5        1976       1  \n",
       "6        1993       1  \n",
       "7        1985       1  \n",
       "8        1983       1  \n",
       "9        1983       1  \n",
       "10       1990       1  \n",
       "11       1959       2  \n",
       "12       1987       1  \n",
       "13       1986       1  \n",
       "14       1984       2  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Looking at data types\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_17c98_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"col_heading level0 col0\" >tripduration</th>\n",
       "      <th class=\"col_heading level0 col1\" >starttime</th>\n",
       "      <th class=\"col_heading level0 col2\" >stoptime</th>\n",
       "      <th class=\"col_heading level0 col3\" >start station id</th>\n",
       "      <th class=\"col_heading level0 col4\" >start station name</th>\n",
       "      <th class=\"col_heading level0 col5\" >start station latitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >start station longitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >end station id</th>\n",
       "      <th class=\"col_heading level0 col8\" >end station name</th>\n",
       "      <th class=\"col_heading level0 col9\" >end station latitude</th>\n",
       "      <th class=\"col_heading level0 col10\" >end station longitude</th>\n",
       "      <th class=\"col_heading level0 col11\" >bikeid</th>\n",
       "      <th class=\"col_heading level0 col12\" >usertype</th>\n",
       "      <th class=\"col_heading level0 col13\" >birth year</th>\n",
       "      <th class=\"col_heading level0 col14\" >gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_17c98_row0_col0\" class=\"data row0 col0\" >int64</td>\n",
       "      <td id=\"T_17c98_row0_col1\" class=\"data row0 col1\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col2\" class=\"data row0 col2\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col3\" class=\"data row0 col3\" >int64</td>\n",
       "      <td id=\"T_17c98_row0_col4\" class=\"data row0 col4\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col5\" class=\"data row0 col5\" >float64</td>\n",
       "      <td id=\"T_17c98_row0_col6\" class=\"data row0 col6\" >float64</td>\n",
       "      <td id=\"T_17c98_row0_col7\" class=\"data row0 col7\" >int64</td>\n",
       "      <td id=\"T_17c98_row0_col8\" class=\"data row0 col8\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col9\" class=\"data row0 col9\" >float64</td>\n",
       "      <td id=\"T_17c98_row0_col10\" class=\"data row0 col10\" >float64</td>\n",
       "      <td id=\"T_17c98_row0_col11\" class=\"data row0 col11\" >int64</td>\n",
       "      <td id=\"T_17c98_row0_col12\" class=\"data row0 col12\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col13\" class=\"data row0 col13\" >object</td>\n",
       "      <td id=\"T_17c98_row0_col14\" class=\"data row0 col14\" >int64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x24285b0f948>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Numeric Data Distribution for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tripduration</th>\n",
       "      <th>start station id</th>\n",
       "      <th>start station latitude</th>\n",
       "      <th>start station longitude</th>\n",
       "      <th>end station id</th>\n",
       "      <th>end station latitude</th>\n",
       "      <th>end station longitude</th>\n",
       "      <th>bikeid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "      <td>968842.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>860.084055</td>\n",
       "      <td>454.482602</td>\n",
       "      <td>40.734458</td>\n",
       "      <td>-73.991340</td>\n",
       "      <td>454.450605</td>\n",
       "      <td>40.734158</td>\n",
       "      <td>-73.991444</td>\n",
       "      <td>18144.415277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>863.906919</td>\n",
       "      <td>380.266297</td>\n",
       "      <td>0.019592</td>\n",
       "      <td>0.012389</td>\n",
       "      <td>383.405490</td>\n",
       "      <td>0.019638</td>\n",
       "      <td>0.012453</td>\n",
       "      <td>2101.399772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>40.680342</td>\n",
       "      <td>-74.017134</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>40.680342</td>\n",
       "      <td>-74.017134</td>\n",
       "      <td>14529.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>651.000000</td>\n",
       "      <td>404.000000</td>\n",
       "      <td>40.736245</td>\n",
       "      <td>-73.990985</td>\n",
       "      <td>404.000000</td>\n",
       "      <td>40.735877</td>\n",
       "      <td>-73.990985</td>\n",
       "      <td>18128.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>21597.000000</td>\n",
       "      <td>3002.000000</td>\n",
       "      <td>40.771522</td>\n",
       "      <td>-73.950048</td>\n",
       "      <td>3002.000000</td>\n",
       "      <td>40.771522</td>\n",
       "      <td>-73.950048</td>\n",
       "      <td>21684.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        tripduration  start station id  start station latitude  \\\n",
       "count  968842.000000     968842.000000           968842.000000   \n",
       "mean      860.084055        454.482602               40.734458   \n",
       "std       863.906919        380.266297                0.019592   \n",
       "min        60.000000         72.000000               40.680342   \n",
       "50%       651.000000        404.000000               40.736245   \n",
       "max     21597.000000       3002.000000               40.771522   \n",
       "\n",
       "       start station longitude  end station id  end station latitude  \\\n",
       "count            968842.000000   968842.000000         968842.000000   \n",
       "mean                -73.991340      454.450605             40.734158   \n",
       "std                   0.012389      383.405490              0.019638   \n",
       "min                 -74.017134       72.000000             40.680342   \n",
       "50%                 -73.990985      404.000000             40.735877   \n",
       "max                 -73.950048     3002.000000             40.771522   \n",
       "\n",
       "       end station longitude         bikeid  \n",
       "count          968842.000000  968842.000000  \n",
       "mean              -73.991444   18144.415277  \n",
       "std                 0.012453    2101.399772  \n",
       "min               -74.017134   14529.000000  \n",
       "50%               -73.990985   18128.000000  \n",
       "max               -73.950048   21684.000000  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance Percent in Numeric Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_8f6e7_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >tripduration</th>\n",
       "      <th class=\"col_heading level0 col1\" >start station id</th>\n",
       "      <th class=\"col_heading level0 col2\" >start station latitude</th>\n",
       "      <th class=\"col_heading level0 col3\" >start station longitude</th>\n",
       "      <th class=\"col_heading level0 col4\" >end station id</th>\n",
       "      <th class=\"col_heading level0 col5\" >end station latitude</th>\n",
       "      <th class=\"col_heading level0 col6\" >end station longitude</th>\n",
       "      <th class=\"col_heading level0 col7\" >bikeid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row0\" class=\"row_heading level0 row0\" >count</th>\n",
       "      <td id=\"T_8f6e7_row0_col0\" class=\"data row0 col0\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col1\" class=\"data row0 col1\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col2\" class=\"data row0 col2\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col3\" class=\"data row0 col3\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col4\" class=\"data row0 col4\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col5\" class=\"data row0 col5\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col6\" class=\"data row0 col6\" >76.80%</td>\n",
       "      <td id=\"T_8f6e7_row0_col7\" class=\"data row0 col7\" >76.80%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row1\" class=\"row_heading level0 row1\" >mean</th>\n",
       "      <td id=\"T_8f6e7_row1_col0\" class=\"data row1 col0\" >1.68%</td>\n",
       "      <td id=\"T_8f6e7_row1_col1\" class=\"data row1 col1\" >3.36%</td>\n",
       "      <td id=\"T_8f6e7_row1_col2\" class=\"data row1 col2\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row1_col3\" class=\"data row1 col3\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row1_col4\" class=\"data row1 col4\" >3.02%</td>\n",
       "      <td id=\"T_8f6e7_row1_col5\" class=\"data row1 col5\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row1_col6\" class=\"data row1 col6\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row1_col7\" class=\"data row1 col7\" >0.74%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row2\" class=\"row_heading level0 row2\" >std</th>\n",
       "      <td id=\"T_8f6e7_row2_col0\" class=\"data row2 col0\" >535.03%</td>\n",
       "      <td id=\"T_8f6e7_row2_col1\" class=\"data row2 col1\" >11.71%</td>\n",
       "      <td id=\"T_8f6e7_row2_col2\" class=\"data row2 col2\" >2.86%</td>\n",
       "      <td id=\"T_8f6e7_row2_col3\" class=\"data row2 col3\" >4.32%</td>\n",
       "      <td id=\"T_8f6e7_row2_col4\" class=\"data row2 col4\" >10.93%</td>\n",
       "      <td id=\"T_8f6e7_row2_col5\" class=\"data row2 col5\" >3.00%</td>\n",
       "      <td id=\"T_8f6e7_row2_col6\" class=\"data row2 col6\" >4.28%</td>\n",
       "      <td id=\"T_8f6e7_row2_col7\" class=\"data row2 col7\" >5.41%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row3\" class=\"row_heading level0 row3\" >min</th>\n",
       "      <td id=\"T_8f6e7_row3_col0\" class=\"data row3 col0\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col1\" class=\"data row3 col1\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col2\" class=\"data row3 col2\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col3\" class=\"data row3 col3\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col4\" class=\"data row3 col4\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col5\" class=\"data row3 col5\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col6\" class=\"data row3 col6\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row3_col7\" class=\"data row3 col7\" >0.00%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row4\" class=\"row_heading level0 row4\" >50%</th>\n",
       "      <td id=\"T_8f6e7_row4_col0\" class=\"data row4 col0\" >16.44%</td>\n",
       "      <td id=\"T_8f6e7_row4_col1\" class=\"data row4 col1\" >0.25%</td>\n",
       "      <td id=\"T_8f6e7_row4_col2\" class=\"data row4 col2\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row4_col3\" class=\"data row4 col3\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row4_col4\" class=\"data row4 col4\" >0.25%</td>\n",
       "      <td id=\"T_8f6e7_row4_col5\" class=\"data row4 col5\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row4_col6\" class=\"data row4 col6\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row4_col7\" class=\"data row4 col7\" >0.84%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8f6e7_level0_row5\" class=\"row_heading level0 row5\" >max</th>\n",
       "      <td id=\"T_8f6e7_row5_col0\" class=\"data row5 col0\" >3,447.29%</td>\n",
       "      <td id=\"T_8f6e7_row5_col1\" class=\"data row5 col1\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col2\" class=\"data row5 col2\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col3\" class=\"data row5 col3\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col4\" class=\"data row5 col4\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col5\" class=\"data row5 col5\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col6\" class=\"data row5 col6\" >0.00%</td>\n",
       "      <td id=\"T_8f6e7_row5_col7\" class=\"data row5 col7\" >0.65%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x242ffc562c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Date Data Distributions for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stoptime</th>\n",
       "      <th>starttime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>968842</td>\n",
       "      <td>968842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2014-07-17 08:54:17.885074944</td>\n",
       "      <td>2014-07-17 08:39:57.801020160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2014-07-01 00:06:01</td>\n",
       "      <td>2014-07-01 00:00:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2014-07-17 18:29:40</td>\n",
       "      <td>2014-07-17 18:15:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2014-08-01 01:43:24</td>\n",
       "      <td>2014-07-31 23:59:58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            stoptime                      starttime\n",
       "count                         968842                         968842\n",
       "mean   2014-07-17 08:54:17.885074944  2014-07-17 08:39:57.801020160\n",
       "min              2014-07-01 00:06:01            2014-07-01 00:00:04\n",
       "50%              2014-07-17 18:29:40            2014-07-17 18:15:26\n",
       "max              2014-08-01 01:43:24            2014-07-31 23:59:58"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Variance in Date Data Distributions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stoptime</th>\n",
       "      <th>starttime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>744106</td>\n",
       "      <td>744106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>150 days 21:42:06.602463488</td>\n",
       "      <td>150 days 21:42:21.038218240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>150 days 00:01:37</td>\n",
       "      <td>150 days 00:00:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>148 days 09:35:04</td>\n",
       "      <td>148 days 09:33:19.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>151 days 11:52:14</td>\n",
       "      <td>153 days 00:00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          stoptime                    starttime\n",
       "count                       744106                       744106\n",
       "mean   150 days 21:42:06.602463488  150 days 21:42:21.038218240\n",
       "min              150 days 00:01:37            150 days 00:00:04\n",
       "50%              148 days 09:35:04     148 days 09:33:19.500000\n",
       "max              151 days 11:52:14            153 days 00:00:11"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Categorical Data Value Counts for the First Frame\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1    647466\n",
       "2    202136\n",
       "0    119240\n",
       "Name: gender, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Subscriber    849778\n",
       "Customer      119064\n",
       "Name: usertype, dtype: int64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "eda_csv([bike_jul, bike_feb], {'starttime','stoptime'}, {'gender', 'usertype'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "zones = pd.read_csv('D://Capstone Data//taxizones.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>objectid</th>\n",
       "      <th>shape_leng</th>\n",
       "      <th>the_geom</th>\n",
       "      <th>shape_area</th>\n",
       "      <th>zone</th>\n",
       "      <th>location_id</th>\n",
       "      <th>borough</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.116357</td>\n",
       "      <td>{'type': 'MultiPolygon', 'coordinates': [[[[-7...</td>\n",
       "      <td>0.000782</td>\n",
       "      <td>Newark Airport</td>\n",
       "      <td>1</td>\n",
       "      <td>EWR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.433470</td>\n",
       "      <td>{'type': 'MultiPolygon', 'coordinates': [[[[-7...</td>\n",
       "      <td>0.004866</td>\n",
       "      <td>Jamaica Bay</td>\n",
       "      <td>2</td>\n",
       "      <td>Queens</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.084341</td>\n",
       "      <td>{'type': 'MultiPolygon', 'coordinates': [[[[-7...</td>\n",
       "      <td>0.000314</td>\n",
       "      <td>Allerton/Pelham Gardens</td>\n",
       "      <td>3</td>\n",
       "      <td>Bronx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.043567</td>\n",
       "      <td>{'type': 'MultiPolygon', 'coordinates': [[[[-7...</td>\n",
       "      <td>0.000112</td>\n",
       "      <td>Alphabet City</td>\n",
       "      <td>4</td>\n",
       "      <td>Manhattan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.092146</td>\n",
       "      <td>{'type': 'MultiPolygon', 'coordinates': [[[[-7...</td>\n",
       "      <td>0.000498</td>\n",
       "      <td>Arden Heights</td>\n",
       "      <td>5</td>\n",
       "      <td>Staten Island</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   objectid  shape_leng                                           the_geom  \\\n",
       "0         1    0.116357  {'type': 'MultiPolygon', 'coordinates': [[[[-7...   \n",
       "1         2    0.433470  {'type': 'MultiPolygon', 'coordinates': [[[[-7...   \n",
       "2         3    0.084341  {'type': 'MultiPolygon', 'coordinates': [[[[-7...   \n",
       "3         4    0.043567  {'type': 'MultiPolygon', 'coordinates': [[[[-7...   \n",
       "4         5    0.092146  {'type': 'MultiPolygon', 'coordinates': [[[[-7...   \n",
       "\n",
       "   shape_area                     zone  location_id        borough  \n",
       "0    0.000782           Newark Airport            1            EWR  \n",
       "1    0.004866              Jamaica Bay            2         Queens  \n",
       "2    0.000314  Allerton/Pelham Gardens            3          Bronx  \n",
       "3    0.000112            Alphabet City            4      Manhattan  \n",
       "4    0.000498            Arden Heights            5  Staten Island  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zones.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "geometry = eval(zones.the_geom[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "coordinates = geometry['coordinates']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "121"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(coordinates[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.00099853, 0.000978]    245\n",
       "(0.000978, 0.00195]         12\n",
       "(0.00195, 0.00292]           5\n",
       "(0.00292, 0.00389]           0\n",
       "(0.00389, 0.00487]           1\n",
       "Name: shape_area, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zones['shape_area'].value_counts(bins=5).sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>objectid</th>\n",
       "      <th>shape_leng</th>\n",
       "      <th>shape_area</th>\n",
       "      <th>location_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>263.000000</td>\n",
       "      <td>263.000000</td>\n",
       "      <td>263.000000</td>\n",
       "      <td>263.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>132.000000</td>\n",
       "      <td>0.094269</td>\n",
       "      <td>0.000402</td>\n",
       "      <td>131.984791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>76.065761</td>\n",
       "      <td>0.054594</td>\n",
       "      <td>0.000482</td>\n",
       "      <td>76.073787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.014306</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>66.500000</td>\n",
       "      <td>0.054722</td>\n",
       "      <td>0.000133</td>\n",
       "      <td>66.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>132.000000</td>\n",
       "      <td>0.084341</td>\n",
       "      <td>0.000268</td>\n",
       "      <td>132.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>197.500000</td>\n",
       "      <td>0.119488</td>\n",
       "      <td>0.000480</td>\n",
       "      <td>197.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>263.000000</td>\n",
       "      <td>0.433470</td>\n",
       "      <td>0.004866</td>\n",
       "      <td>263.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         objectid  shape_leng  shape_area  location_id\n",
       "count  263.000000  263.000000  263.000000   263.000000\n",
       "mean   132.000000    0.094269    0.000402   131.984791\n",
       "std     76.065761    0.054594    0.000482    76.073787\n",
       "min      1.000000    0.014306    0.000006     1.000000\n",
       "25%     66.500000    0.054722    0.000133    66.500000\n",
       "50%    132.000000    0.084341    0.000268   132.000000\n",
       "75%    197.500000    0.119488    0.000480   197.500000\n",
       "max    263.000000    0.433470    0.004866   263.000000"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zones.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grabbing some weather data from the open weather API using the first bike trip as an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import yaml\n",
    "import json\n",
    "\n",
    "with open('open_weather_token.yaml','r') as file:\n",
    "    config = yaml.safe_load(file)\n",
    "\n",
    "weather_api_key = config['api_key']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "trip = bike_jan.loc[0]\n",
    "\n",
    "start = (pd.to_datetime(trip.starttime) - pd.Timestamp(\"1970-01-01\")) // pd.Timedelta('1s')\n",
    "end = (pd.to_datetime(trip.stoptime) - pd.Timestamp(\"1970-01-01\")) // pd.Timedelta('1s')\n",
    "start += 18000\n",
    "end += 18000\n",
    "\n",
    "locs = [(start, trip['start station latitude'], trip['start station longitude']), (end, trip['end station latitude'], trip['end station longitude'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "responses = []\n",
    "for (time, lat, lon) in locs:\n",
    "    response = requests.get(f'https://api.openweathermap.org/data/3.0/onecall/timemachine?lat={lat}&lon={lon}&dt={time}&appid={weather_api_key}')\n",
    "    \n",
    "    responses.append(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OK'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = responses[0]\n",
    "r.reason"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "r1 = json.loads(responses[0].content)\n",
    "r2 = json.loads(responses[1].content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'lat': 40.7112,\n",
       " 'lon': -73.9968,\n",
       " 'timezone': 'America/New_York',\n",
       " 'timezone_offset': -18000,\n",
       " 'data': [{'dt': 1388552406,\n",
       "   'sunrise': 1388578799,\n",
       "   'sunset': 1388612335,\n",
       "   'temp': 270.29,\n",
       "   'feels_like': 263.96,\n",
       "   'pressure': 1026,\n",
       "   'humidity': 44,\n",
       "   'dew_point': 260.87,\n",
       "   'clouds': 0,\n",
       "   'visibility': 10000,\n",
       "   'wind_speed': 6.2,\n",
       "   'wind_deg': 300,\n",
       "   'wind_gust': 10.8,\n",
       "   'weather': [{'id': 800,\n",
       "     'main': 'Clear',\n",
       "     'description': 'clear sky',\n",
       "     'icon': '01n'}]}]}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'lat': 40.7173,\n",
       " 'lon': -73.9964,\n",
       " 'timezone': 'America/New_York',\n",
       " 'timezone_offset': -18000,\n",
       " 'data': [{'dt': 1388552877,\n",
       "   'sunrise': 1388578800,\n",
       "   'sunset': 1388612334,\n",
       "   'temp': 270.13,\n",
       "   'feels_like': 263.76,\n",
       "   'pressure': 1026,\n",
       "   'humidity': 43,\n",
       "   'dew_point': 260.47,\n",
       "   'clouds': 0,\n",
       "   'visibility': 10000,\n",
       "   'wind_speed': 6.2,\n",
       "   'wind_deg': 300,\n",
       "   'wind_gust': 10.8,\n",
       "   'weather': [{'id': 800,\n",
       "     'main': 'Clear',\n",
       "     'description': 'clear sky',\n",
       "     'icon': '01n'}]}]}"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Taxi data was unordered, so its date columns are distributed equally across the different chunks. The numerical and categorical data is relatively homogeneous across the chunks. Future small scale testing will be performed on one chunk of the data.\n",
    "\n",
    "The citibike data is separated by month. Citi Bikes were used differently throughout the year. Future small scale testing will be performed using 1 week of data from each month in order to keep the data volumes low, and use a representative sample of the complete data.\n",
    "\n",
    "Weather data will be optained using an API call in real time. This will only happen for the requested trip, and each api call will have the same available data. Temp, feels_like, clouds, and main weather description will be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do you anticipate this data will be used by data analysts and scientists downstream?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data will be used to make determinations about when it would be quicker to take a CitiBike instead of taking a cab. Both in terms of the trip origin and destination, and in terms of the time of the year and time of the day. Data can also be augmented using the open weather data to give an idea of the expected weather conditions for the bike trip."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Does your answer to the last question give you an indication of how you can store the data for optimal querying speed and storage file compression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The taxi data should be augmented with the taxi zone data to partition by the neighborhoods. This will allow the data to be stored in parquet files segregated by taxi trip origin and trip destination zones.\n",
    "\n",
    "The bike data should be segregated based on start and end station location. This location can also be augmented using the taxi zone data. Additional tables should be made to pull station locations into a separate table, so the closest bike station is easy to determine, and then the trip data can be queried based on the start and end stations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What cleaning steps do you need to perform to make your dataset ready for consumption?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Null values in the taxi data should be replaced with easier to understand values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What wrangling steps do you need to perform to enrich your dataset with additional information?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The green and yellow taxi data need to have a flag added and to be combined into a single dataset. The flag would differentiate between the yellow and green taxi. \n",
    "\n",
    "The taxi data also needs to have a taxi zone added in by determining which zone the pickup and dropoff lat, longs are within. The same will have to be done for the bike trip stations. This can be done by drawing a straight line between each point, and a point outside of all of the regions, boston for example, if the number of times this line crosses the region perimeter is even, the point is outside the region, if it is odd, then the point is inside the region. \n",
    "\n",
    "The datetime dimensions between the two datasets need to be conformed to be in the same format, and a date dimension needs to be created to allow for analysis and averaging across day of the week type, hour, holidays, etc. Datetimes will need to be converted to epoch timestamps for the weather data pull, and will also need to be converted to UTC timestamps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Data Model](Capstone_Data_Model.png \"Data Model\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "eda",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}