Parking Ocuppancy Predictor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parking Ocuppancy Predictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El objetivo es predecir el porcentaje de espacios libres para una cuadra determinada, dia y hora. En principio, se utiliza solo los datos historicos, posteriormente la idea es añadirle diversas fuentes de informacion. De modo, que estas mejoren la predicción."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib as matplotlib;\n",
    "import matplotlib.pyplot as plt;\n",
    "import pandas as pd;\n",
    "import numpy as np;\n",
    "import seaborn as sns;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Carga del dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = \"https://raw.githubusercontent.com/chirichignoa/Parking-Ocuppancy-Predictor/master/dataset/BD_parking_with_address.csv\"\n",
    "dataset = pd.read_csv(data_path);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocesamiento"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se eliminan los registros con el **id_cuadra** igual a 50 que corresponde a la oficina SUMO y solo se utiliza para cuestiones administrativas. En caso de querer **filtrar datos para una cuadra especifica** descomentar las siguientes lineas. La segunda es la que realiza el filtrado y la restante, visualiza los datos para comprobar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset[dataset['id_cuadra']!=50]\n",
    "# dataset = dataset[(dataset.id_cuadra == 11) | (dataset.id_cuadra == 59) | (dataset.id_cuadra == 39)]\n",
    "# dataset.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se ordena el dataset de acuerdo a la fecha y a la hora de las operaciones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset = dataset.sort_values(['fecha', 'hora'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se genera una columna nueva, con el merge de la **fecha** y de la **hora** y se la redondea hacia el valor menor cercano en un rango de 15 minutos. Por ejemplo, la hora 11:26 se traduce a 11:30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['time'] = pd.to_datetime(dataset.fecha + \" \" + dataset.hora,format=\"%Y-%m-%d %H:%M:%S.%f\").dt.round('15min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se desean eliminar los datos del dia domingo, ya que el sistema no se encuentra habilitado en dicho dia. Para esto, se debe convertir la fecha al formato **day of week**. Se genera una nueva columna que almacenara dicho dato. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset['fecha'] = pd.to_datetime(dataset.fecha ,format=\"%Y-%m-%d\")\n",
    "dataset['hora'] = pd.to_datetime(dataset.hora,format=\"%H:%M:%S.%f\")\n",
    "dataset['dow'] = dataset['fecha'].apply(lambda fecha: fecha.weekday())\n",
    "dataset = dataset[dataset['dow']!=6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luego se eliminan aquellas operaciones que no correspondan al horario funcional del sistema SUMO, el cual es desde las 10:00 AM hasta 20:00 PM. **Nota:** en caso de querer filtrar los datos por una cierta fecha se debe descomentar la linea que contiene la variable fecha, y utilizar la variable **_mask_** que contenga el filtro por la fecha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_start = pd.to_datetime('10:00:00.000000',format=\"%H:%M:%S.%f\");\n",
    "time_end = pd.to_datetime('20:00:00.000000',format=\"%H:%M:%S.%f\");\n",
    "# fecha = pd.to_datetime('2018-06-01' ,format=\"%Y-%m-%d\")\n",
    "\n",
    "mask = (time_start <= dataset['hora']) & (dataset['hora'] <= time_end);\n",
    "# mask = (dataset['fecha'] == fecha) & (time_start <= dataset['hora']) & (dataset['hora'] <= time_end);\n",
    "dataset = dataset[mask];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luego de haber filtrado los registros de acuerdo al horario funcional del sistema, se eliminan las columnas de fecha y hora, las cuales no son necesarias a partir de aqui."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.drop(['hora', 'fecha'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación se puede visualizar los datos, para ver el resultado de toda la transformación anterior:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_cuadra</th>\n",
       "      <th>operacion</th>\n",
       "      <th>tarjeta</th>\n",
       "      <th>direccion</th>\n",
       "      <th>time</th>\n",
       "      <th>dow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10059452</td>\n",
       "      <td>Mitre 348</td>\n",
       "      <td>2018-01-01 11:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351452</th>\n",
       "      <td>40</td>\n",
       "      <td>Salida</td>\n",
       "      <td>10059452</td>\n",
       "      <td>Mitre 348</td>\n",
       "      <td>2018-01-01 11:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>61</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10032179</td>\n",
       "      <td>Chacabuco 357</td>\n",
       "      <td>2018-01-01 13:15:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351453</th>\n",
       "      <td>61</td>\n",
       "      <td>Salida</td>\n",
       "      <td>10032179</td>\n",
       "      <td>Chacabuco 357</td>\n",
       "      <td>2018-01-01 13:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10026609</td>\n",
       "      <td>Sarmiento 849</td>\n",
       "      <td>2018-01-01 17:00:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>66</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10073847</td>\n",
       "      <td>Rodriguez 846</td>\n",
       "      <td>2018-01-01 17:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10012270</td>\n",
       "      <td>San Martin 852</td>\n",
       "      <td>2018-01-01 18:15:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>493</th>\n",
       "      <td>52</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10007005</td>\n",
       "      <td>Chacabuco 646</td>\n",
       "      <td>2018-01-01 18:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517</th>\n",
       "      <td>52</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10057514</td>\n",
       "      <td>Chacabuco 646</td>\n",
       "      <td>2018-01-01 18:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>564</th>\n",
       "      <td>52</td>\n",
       "      <td>Entrada</td>\n",
       "      <td>10096601</td>\n",
       "      <td>Chacabuco 646</td>\n",
       "      <td>2018-01-01 18:30:00</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id_cuadra operacion   tarjeta       direccion                time  dow\n",
       "1              40   Entrada  10059452       Mitre 348 2018-01-01 11:30:00    0\n",
       "351452         40    Salida  10059452       Mitre 348 2018-01-01 11:30:00    0\n",
       "2              61   Entrada  10032179   Chacabuco 357 2018-01-01 13:15:00    0\n",
       "351453         61    Salida  10032179   Chacabuco 357 2018-01-01 13:30:00    0\n",
       "3              34   Entrada  10026609   Sarmiento 849 2018-01-01 17:00:00    0\n",
       "4              66   Entrada  10073847   Rodriguez 846 2018-01-01 17:30:00    0\n",
       "5              16   Entrada  10012270  San Martin 852 2018-01-01 18:15:00    0\n",
       "493            52   Entrada  10007005   Chacabuco 646 2018-01-01 18:30:00    0\n",
       "517            52   Entrada  10057514   Chacabuco 646 2018-01-01 18:30:00    0\n",
       "564            52   Entrada  10096601   Chacabuco 646 2018-01-01 18:30:00    0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez obtenido los datos funcionales del sistema, se procede a separar los datos de acuerdo al tipo de operación. Se crean dos nuevos dataframes que contienen los datos asociados a los ingresos y egresos del sistema. A modo de comparación, finalmente se muestra el tamaño de cada uno, es decir la cantidad de operaciones de cada tipo. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingresos: 316370 - Egresos: 327930\n"
     ]
    }
   ],
   "source": [
    "dfi = dataset[dataset['operacion']=='Entrada']\n",
    "dfo = dataset[dataset['operacion']=='Salida']\n",
    "print(\"Ingresos: {} - Egresos: {}\".format(dfi.shape[0], dfo.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez separado los datos, es necesario llevar un conteo de todos los registros correspondientes a una cuadra y al mismo rango horario, discriminando si se corresponde a un ingreso o a un egreso. De este modo, se generan las columnas **count_in** y **count_out** que contienen la cantidad total de entradas y de salidas. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se eliminan las columnas **operacion**, **tarjeta** y **direccion** quedando solamente la informacion relacionada a la cuadra, el rango de tiempo y la cantidad de entradas o de salidas de cada dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_count(df, column_name):\n",
    "    dataframe = df;\n",
    "    dataframe = dataframe.set_index('time');\n",
    "    dataframe = dataframe.groupby(by=['id_cuadra', pd.Grouper(freq='15Min')]).count();\n",
    "    dataframe = dataframe[dataframe['operacion'] != 0];\n",
    "    dataframe[column_name] = dataframe['operacion'];\n",
    "    dataframe = dataframe.drop(['operacion', 'tarjeta', 'dow', 'direccion'], axis=1);\n",
    "    return dataframe;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfi = get_count(dfi, \"count_in\")\n",
    "dfo = get_count(dfo, \"count_out\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "De este modo, se reducen los datasets de acuerdo al rango horario y se cuenta la cantidad de registros que pertenecen a ese rango. A continuación se puede visualizar el tamaño de cada uno de los datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingresos: 184471 - Egresos: 192777\n"
     ]
    }
   ],
   "source": [
    "print(\"Ingresos: {} - Egresos: {}\".format(dfi.shape[0], dfo.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilizando los datos obtenidos, se genera un dataframe definitivo realizando un merge a izquierda, ya que el dataset asociado a los egresos es el que mas filas contiene. Si se mergeara a partir de los egresos se generaria informacion que no se condice con la realidad, de modo que existen egresos sin haberse hecho ingresos. Como resultado de esta operación de mergeo, pueden existir valores faltantes en el caso de que no existan filas que tengan el mismo rango horario. En este caso, se procede a completar dichos valores faltantes (expresados como _NaN_) con el valor de 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_count = pd.merge(dfi, dfo, left_index=True, right_index=True, how='left')\n",
    "df_count = df_count.fillna(0,axis=1)\n",
    "df_count['count_out'] = df_count.count_out.astype(int)\n",
    "df_count['count_in'] = df_count.count_in.astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "De este modo, se puede ver el tamaño del dataset resultante y una visualizacion de los datos. Cabe destacar, que estos datos se agruparon conun doble index compuesto de **'id_cuadra'** y **'time'**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tamaño del dataset definitivo: 184471\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count_in</th>\n",
       "      <th>count_out</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id_cuadra</th>\n",
       "      <th>time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2</th>\n",
       "      <th>2018-01-02 10:00:00</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02 10:15:00</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02 10:45:00</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02 11:00:00</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018-01-02 11:15:00</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               count_in  count_out\n",
       "id_cuadra time                                    \n",
       "2         2018-01-02 10:00:00         2          0\n",
       "          2018-01-02 10:15:00         2          0\n",
       "          2018-01-02 10:45:00         1          3\n",
       "          2018-01-02 11:00:00         1          1\n",
       "          2018-01-02 11:15:00         4          0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Tamaño del dataset definitivo: {}\".format(df_count.shape[0]))\n",
    "df_count.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generacion de lugares disponibles por cuadras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estos datos, ya se encuentran generados y almacenados en un documento csv, por lo que se procede a cargar dichos datos en un dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_cuadra</th>\n",
       "      <th>direccion</th>\n",
       "      <th>cant_espacios</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>General Pinto 545</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>San Marti­n 452</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>Rodriguez 552</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>9 de Julio 441</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>General Pinto 759</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_cuadra          direccion  cant_espacios\n",
       "0          2  General Pinto 545             29\n",
       "1          3    San Marti­n 452             14\n",
       "2          4      Rodriguez 552             15\n",
       "3          5     9 de Julio 441             16\n",
       "4          6  General Pinto 759              9"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = \"https://raw.githubusercontent.com/chirichignoa/Parking-Ocuppancy-Predictor/master/dataset/cant_espacios.csv\"\n",
    "cant_lugares = pd.read_csv(data_path);\n",
    "cant_lugares.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luego se seleccionan los datos relevantes de dicha información cargada, los cuales son el **'id_cuadra'** y **'cant_espacios'**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "cant_lugares = cant_lugares[['id_cuadra', 'cant_espacios']]\n",
    "cant_lugares = cant_lugares.set_index('id_cuadra')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La siguiente función permite obtener la cantidad total de lugares disponibles para cada una de las cuadras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_total_spaces(id_cuadra):\n",
    "    return cant_lugares.at[id_cuadra,'cant_espacios']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A partir de los datos obtenidos, mediante el proceso de preprocesamiento y de los datos de la cantidad total de lugares disponibles de estacionamiento, se procede a calcular la disponibilidad para cada uno de los rangos de tiempo. Mediante la siguiente funcion se alcanza dicho objetivo, realizando una iteracion por cada una de las filas de dataframe, y utilizando la función definida que retorna la cantidad de lugares disponibles, se realiza  la siguiente suma:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\\begin{align}\n",
    "{Espacios\\,Disponibles_{P,T_{i-1}} - Ingresos_{P,T_{i}} + Egresos_{P,T_{i}}}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Donde P es el identificador de cuadra, y T es la ventana de tiempo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_occupancy(df):\n",
    "    df_count = df.copy();\n",
    "    prev_date = \"1900-00-00\";\n",
    "    prev_cuadra = 0;\n",
    "    total_spaces = 0;\n",
    "    ret = [];\n",
    "    partial_sum = 0;\n",
    "    available = 0;\n",
    "    for row in df_count.iterrows():\n",
    "        # row[0][0] id_cuadra\n",
    "        # row[0][1] time\n",
    "        # row[1][0] count_in\n",
    "        # row[1][1] count_out\n",
    "        id_cuadra = row[0][0];\n",
    "        date = row[0][1];\n",
    "        splitted = str(row[0][1]).split();\n",
    "        if(prev_cuadra != id_cuadra):\n",
    "            total_spaces = get_total_spaces(id_cuadra);\n",
    "            prev_cuadra = id_cuadra;\n",
    "        if(prev_date != splitted[0]):\n",
    "            available = total_spaces;\n",
    "            prev_date = splitted[0];\n",
    "            partial_sum = available;\n",
    "\n",
    "        partial_sum = partial_sum - row[1][0] + row[1][1];\n",
    "        ret.append(partial_sum);\n",
    "    df_count['available'] = ret;\n",
    "    return df_count;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_count = get_occupancy(df_count);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luego se genera una columna con la cantidad de espacios totales para cada cuadra:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_count['total'] = df_count.apply(lambda row: get_total_spaces(row.name[0]), axis=1)\n",
    "df_count.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora en base a los datos generados, se obtiene el porcentaje de disponibilidad o **AVR**. El cual se define formalmente como:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align} AVR_{P,T_{i}}\t=\\cfrac{Espacios\\,Disponibles_{P,T_{i-1}}-Ingresos_{P,T_{i}}+Egresos_{P,T_{i}}}{Espacios\\,Totales_{P}}\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_avr(available, total):\n",
    "    avr = (available / total);\n",
    "    if(avr > 1):\n",
    "        return 1;\n",
    "    if(avr < 0):\n",
    "        return 0;\n",
    "    return avr;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cabe destacar que, al haber generado manuelmente la cantidad de espacios disponibles por cuadra, pueda que exista un error de estimación. Esto significa que puede haber cuadras donde la cantidad de espacios disponibles supere el total, o sea menor que 0. De este modo, para evitar cifras no deseadas, se decidió por acotar los valores que se exceden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_count['avr'] = df_count.apply(lambda row: get_avr(row.available, row.total), axis=1)\n",
    "df_count.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Una vez generado el AVR, se procede a eliminar las columnas con los lugares disponibles y totales, ya que no son necesarias para el calculo posterior. Por lo que se genera el dataframe que servira para la predicción. Ademas, se transforma la columna time en el dia de la semana asociada a esa fecha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_feature = df_count.copy()\n",
    "df_feature = df_feature.reset_index()\n",
    "df_feature = df_feature[['id_cuadra', 'count_in','count_out', 'time', 'avr']]\n",
    "df_feature['dow'] = df_feature['time'].dt.dayofweek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para lograr una mejor eficiencia en cuanto a predicción, se categorizaran las ventanas de tiempo sobre la columna **time** en base al arreglo generado a continuación, y seleccionando solo el horario. Por ejemplo para la hora 10:00 correspondera el indice o categoria 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "times = [\"10:00:00\", \"10:15:00\", \"10:30:00\", \"10:45:00\", \"11:00:00\", \"11:15:00\", \"11:30:00\",\n",
    "\"11:45:00\", \"12:00:00\", \"12:15:00\", \"12:30:00\", \"12:45:00\", \"13:00:00\", \"13:15:00\", \"13:30:00\", \"13:45:00\", \"14:00:00\",\n",
    "\"14:15:00\", \"14:30:00\", \"14:45:00\", \"15:00:00\", \"15:15:00\", \"15:30:00\", \"15:45:00\", \"16:00:00\", \"16:15:00\", \"16:30:00\",\n",
    "\"16:45:00\", \"17:00:00\", \"17:15:00\", \"17:30:00\", \"17:45:00\", \"18:00:00\", \"18:15:00\", \"18:30:00\", \"18:45:00\", \"19:00:00\",\n",
    "\"19:15:00\", \"19:30:00\", \"19:45:00\", \"20:00:00\"]\n",
    "def get_index(time):\n",
    "    return times.index(time);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_feature['time'] = df_feature['time'].apply(lambda time: get_index(str(time).split()[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicción"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Debido a que se pretende predecir un valor numerico como lo es el porcentaje de disponibilidad. Se utilizaran las siguientes tecnicas de _Machine Learning_:\n",
    " 1. **Linear Regression** (LNR)\n",
    " 2. **Regression Tree** (RT)\n",
    " 3. **Gradient Boosting Regression** (GBR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Se medira la performance de estas tecnicas utilizando distintas metricas, realizando la comparación frente a dos Dummy Regressors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Primero, separo los conjuntos de caracteristicas del conjunto de datos a predecir, donde X es el conjunto de caracteristicas; y el conjunto de porcentaje de estacionamiento disponible. De modo que dado X se buscara predecir y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = df_feature['avr']\n",
    "df_feature = df_feature.drop('avr', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para la validación de las predicciones se utilizara la tecnica _k-fold cross validation_ con un total de 5 folds. Para esto es necesario separar las caracteristicas en dos conjuntos: conjunto de entrenamiento y conjunto de test. Esto se hace tanto para las caracteristicas como para las datos a predecir."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_feature, labels, test_size=0.25,random_state =1 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dummy Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import dummy\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
    "\n",
    "model = dummy.DummyRegressor(strategy='mean') \n",
    "model.fit(X_train, y_train);\n",
    "y_predicted_dr = model.predict(X_test);\n",
    "# The mean squared error\n",
    "dummy_mse = mean_squared_error(y_test, y_predicted_dr);\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % dummy_mse);\n",
    "# The mean absolute error\n",
    "dummy_mae = mean_absolute_error(y_test, y_predicted_dr);\n",
    "print(\"Mean absolute error: %.3f\"\n",
    "      % dummy_mae);\n",
    "# Explained variance score: 1 is perfect prediction\n",
    "dummy_r2 = r2_score(y_test, y_predicted_dr);\n",
    "print('R2 score: %.8f' % dummy_r2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improved Dummy Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_dummy_avr(id_cuadra, time, dow, k):\n",
    "    ret = df_train[(df_train.id_cuadra == id_cuadra) & (df_train.time == time) & (df_train.dow == dow)]\n",
    "    if (ret.shape[0] >= k):\n",
    "        ret = ret.sample(k);\n",
    "    return np.nan_to_num(np.mean(ret.avr));\n",
    "\n",
    "df_train = X_train.copy();\n",
    "df_train['avr'] = y_train;\n",
    "y_predicted_drm = [];\n",
    "k = 2;\n",
    "X_test.apply(lambda row: y_predicted_drm.append(predict_dummy_avr(row.id_cuadra, row.time, row.dow, k)), axis=1);\n",
    "# The mean squared error\n",
    "a_dummy_mse = mean_squared_error(y_test, y_predicted_drm);\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % a_dummy_mse);\n",
    "# The mean absolute error\n",
    "a_dummy_mae = mean_absolute_error(y_test, y_predicted_drm);\n",
    "print(\"Mean absolute error: %.3f\"\n",
    "      % a_dummy_mae);\n",
    "# Explained variance score: 1 is perfect prediction\n",
    "a_dummy_r2 = r2_score(y_test, y_predicted_drm);\n",
    "print('R2 score: %.8f' % a_dummy_r2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
    "\n",
    "model = linear_model.LinearRegression() \n",
    "parameters = {'fit_intercept':[True,False], 'normalize':[True,False], 'copy_X':[True, False]} \n",
    "grid = GridSearchCV(model,parameters, cv=7, scoring='r2')\n",
    "grid.fit(X_train, y_train);\n",
    "print(\"Best parameters set found on development set:\")\n",
    "print()\n",
    "print(grid.best_params_)\n",
    "print()\n",
    "print(\"Grid scores on development set:\")\n",
    "print()\n",
    "means = grid.cv_results_['mean_test_score']\n",
    "stds = grid.cv_results_['std_test_score']\n",
    "for mean, std, params in zip(means, stds, grid.cv_results_['params']):\n",
    "    print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "          % (mean, std * 2, params))\n",
    "print()\n",
    "y_predicted_lr = grid.predict(X_test)\n",
    "# The mean squared error\n",
    "linear_mse = mean_squared_error(y_test, y_predicted_lr)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % linear_mse)\n",
    "# The mean absolute error\n",
    "linear_mae = mean_absolute_error(y_test, y_predicted_lr)\n",
    "print(\"Mean absolute error: %.3f\"\n",
    "      % linear_mae)\n",
    "# Explained variance score: 1 is perfect prediction\n",
    "linear_r2 = r2_score(y_test, y_predicted_lr)\n",
    "print('R2 score: %.3f' % linear_r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
    "\n",
    "model = DecisionTreeRegressor(max_depth=3)\n",
    "parameters = {'max_depth':[10]} \n",
    "grid = GridSearchCV(model,parameters, cv=7, scoring='r2')\n",
    "grid.fit(X_train, y_train); \n",
    "print(\"Best parameters set found on development set:\")\n",
    "print()\n",
    "print(grid.best_params_)\n",
    "print()\n",
    "print(\"Grid scores on development set:\")\n",
    "print()\n",
    "means = grid.cv_results_['mean_test_score']\n",
    "stds = grid.cv_results_['std_test_score']\n",
    "for mean, std, params in zip(means, stds, grid.cv_results_['params']):\n",
    "    print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "          % (mean, std * 2, params))\n",
    "print()\n",
    "y_predicted_rt = grid.predict(X_test)\n",
    "# The mean squared error\n",
    "regtree_mse = mean_squared_error(y_test, y_predicted_rt)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % regtree_mse)\n",
    "# The mean absolute error\n",
    "regtree_mae = mean_absolute_error(y_test, y_predicted_rt)\n",
    "print(\"Mean absolute error: %.3f\"\n",
    "      % regtree_mae)\n",
    "# Explained variance score: 1 is perfect prediction\n",
    "regtree_r2 = r2_score(y_test, y_predicted_rt)\n",
    "print('R2 score: %.3f' % regtree_r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boost Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import ensemble\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
    "\n",
    "model = ensemble.GradientBoostingRegressor()\n",
    "# parameters = {'n_estimators': [500], 'max_depth': [3,5,7], 'min_samples_split': [2,4,6,8],\n",
    "#               'min_samples_leaf': [4], 'max_features': [0.2], 'random_state': [0,1] }\n",
    "parameters = {'n_estimators': [500], 'max_depth': [5], 'min_samples_split': [4],\n",
    "              'min_samples_leaf': [4], 'max_features': [0.2], 'random_state': [1] }\n",
    "grid = GridSearchCV(model,parameters, cv=7, scoring='r2')\n",
    "grid.fit(X_train, y_train); \n",
    "print(\"Best parameters set found on development set:\")\n",
    "print()\n",
    "print(grid.best_params_)\n",
    "print()\n",
    "print(\"Grid scores on development set:\")\n",
    "print()\n",
    "means = grid.cv_results_['mean_test_score']\n",
    "stds = grid.cv_results_['std_test_score']\n",
    "for mean, std, params in zip(means, stds, grid.cv_results_['params']):\n",
    "    print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "          % (mean, std * 2, params))\n",
    "print()\n",
    "y_predicted_gbr = grid.predict(X_test)\n",
    "# The mean squared error\n",
    "gradientboost_mse = mean_squared_error(y_test, y_predicted_gbr)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % gradientboost_mse)\n",
    "# The mean absolute error\n",
    "gradientboost_mae = mean_absolute_error(y_test, y_predicted_gbr)\n",
    "print(\"Mean absolute error: %.3f\"\n",
    "      % gradientboost_mae)\n",
    "# Explained variance score: 1 is perfect prediction\n",
    "gradientboost_r2 = r2_score(y_test, y_predicted_gbr)\n",
    "print('R2 score: %.3f' % gradientboost_r2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resultados de las métricas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ahora se puede ver los resultados de las metricas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [['Dummy Regressor',dummy_mae, dummy_mse, dummy_r2],\n",
    "        ['Improved Dummy Regressor',a_dummy_mae, a_dummy_mse, a_dummy_r2],\n",
    "        ['Linear Regression',linear_mae, linear_mse, linear_r2],\n",
    "        ['Regression Tree',regtree_mae, regtree_mse, regtree_r2],\n",
    "        ['Gradient Boost Regression', gradientboost_mae, gradientboost_mse, gradientboost_r2]]\n",
    "df_measure = pd.DataFrame(data,columns=['Regressor','MAE', 'MSE', 'R2'])\n",
    "df_measure.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure() # Create matplotlib figure\n",
    "ax =df_measure[['Regressor', 'MAE', 'MSE', 'R2']].plot(x=\"Regressor\",\n",
    "                                                   kind=\"bar\",\n",
    "                                                   fontsize=\"12\",\n",
    "                                                   title=\"Comparación de técnicas\", \n",
    "                                                   figsize=(8,5), \n",
    "                                                   rot=25)\n",
    "ax.title.set_size(16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(nrows=1, ncols=5)\n",
    "fig.set_size_inches(22, 4)\n",
    "n = 500\n",
    "sns.regplot(x=y_predicted_dr[:n], y=y_test[:n], fit_reg=False, ax=ax[0])\n",
    "ax[0].set_xlabel('Predicción')\n",
    "ax[0].set_ylabel('Valor real')\n",
    "ax[0].set_title('a) DR', fontsize=22, alpha=0.7, ha='center')\n",
    "# ax[0].grid(True)\n",
    "sns.regplot(x=pd.Series(y_predicted_drm[:n]), y=y_test[:n], fit_reg=False, ax=ax[1])\n",
    "ax[1].set_xlabel('Predicción')\n",
    "ax[1].set_ylabel('Valor real')\n",
    "ax[1].set_title('b) IDR', fontsize=22, alpha=0.7, ha='center')\n",
    "# ax[1].grid(True)\n",
    "\n",
    "sns.regplot(x=y_predicted_lr[:n], y=y_test[:n], fit_reg=False, ax=ax[2])\n",
    "ax[2].set_xlabel('Predicción')\n",
    "ax[2].set_ylabel('Valor real')\n",
    "ax[2].set_title('c) LR', fontsize=22, alpha=0.7, ha='center')\n",
    "# ax[2].grid(True)\n",
    "\n",
    "sns.regplot(x=y_predicted_rt[:n], y=y_test[:n], fit_reg=False, ax=ax[3])\n",
    "ax[3].set_xlabel('Predicción')\n",
    "ax[3].set_ylabel('Valor real')\n",
    "ax[3].set_title('d) RT', fontsize=22, alpha=0.7, ha='center')\n",
    "# ax[3].grid(True)\n",
    "\n",
    "sns.regplot(x=y_predicted_gbr[:n], y=y_test[:n], fit_reg=False, ax=ax[4])\n",
    "ax[4].set_xlabel('Predicción')\n",
    "ax[4].set_ylabel('Valor real')\n",
    "ax[4].set_title('e) GBR', fontsize=22, alpha=0.7, ha='center')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}