0. Preliminary_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8431044e-7d7e-4d84-acf0-be2ec580984d",
   "metadata": {},
   "source": [
    "***\n",
    "# YELP DATASET\n",
    "***\n",
    "The initial data was gotten from [YELP](https://www.yelp.com/dataset/download), from which we based our project on.\n",
    "The dataset contained 5 JSON files from which 2 were picked to use on our academic recommendation systems project. The datasets chosen were on:-  \n",
    "-  Business Information\n",
    "-  User Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "663c019a-73ca-4b16-9447-e37709dce2f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing libraries\n",
    "import pandas as pd\n",
    "import tarfile\n",
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8a061847-a6cc-4dd7-a326-3fc6d4b43f46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the path to your tar file\n",
    "tar_path = 'yelp_dataset.tar'\n",
    "\n",
    "# Open the tar file and save to a hidden file\n",
    "with tarfile.open(tar_path, 'r') as tar:\n",
    "    # Extract all contents to a specific directory\n",
    "    tar.extractall(path='.extracted_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bedcecfb-d0f2-4a62-888b-e0bdb43bd49e",
   "metadata": {},
   "source": [
    "***\n",
    "## FIRST DATASET CONTAINING BUSINESS INFORMATION\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "207299ca-d5f3-4d14-a58a-3fc7404417fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>name</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>postal_code</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>stars</th>\n",
       "      <th>review_count</th>\n",
       "      <th>is_open</th>\n",
       "      <th>attributes</th>\n",
       "      <th>categories</th>\n",
       "      <th>hours</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pns2l4eNsfO8kk83dixA6A</td>\n",
       "      <td>Abby Rappoport, LAC, CMQ</td>\n",
       "      <td>1616 Chapala St, Ste 2</td>\n",
       "      <td>Santa Barbara</td>\n",
       "      <td>CA</td>\n",
       "      <td>93101</td>\n",
       "      <td>34.426679</td>\n",
       "      <td>-119.711197</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>{'ByAppointmentOnly': 'True'}</td>\n",
       "      <td>Doctors, Traditional Chinese Medicine, Naturop...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mpf3x-BjTdTEA3yCZrAYPw</td>\n",
       "      <td>The UPS Store</td>\n",
       "      <td>87 Grasso Plaza Shopping Center</td>\n",
       "      <td>Affton</td>\n",
       "      <td>MO</td>\n",
       "      <td>63123</td>\n",
       "      <td>38.551126</td>\n",
       "      <td>-90.335695</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>{'BusinessAcceptsCreditCards': 'True'}</td>\n",
       "      <td>Shipping Centers, Local Services, Notaries, Ma...</td>\n",
       "      <td>{'Monday': '0:0-0:0', 'Tuesday': '8:0-18:30', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>tUFrWirKiKi_TAnsVWINQQ</td>\n",
       "      <td>Target</td>\n",
       "      <td>5255 E Broadway Blvd</td>\n",
       "      <td>Tucson</td>\n",
       "      <td>AZ</td>\n",
       "      <td>85711</td>\n",
       "      <td>32.223236</td>\n",
       "      <td>-110.880452</td>\n",
       "      <td>3.5</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>{'BikeParking': 'True', 'BusinessAcceptsCredit...</td>\n",
       "      <td>Department Stores, Shopping, Fashion, Home &amp; G...</td>\n",
       "      <td>{'Monday': '8:0-22:0', 'Tuesday': '8:0-22:0', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MTSW4McQd7CbVtyjqoe9mw</td>\n",
       "      <td>St Honore Pastries</td>\n",
       "      <td>935 Race St</td>\n",
       "      <td>Philadelphia</td>\n",
       "      <td>PA</td>\n",
       "      <td>19107</td>\n",
       "      <td>39.955505</td>\n",
       "      <td>-75.155564</td>\n",
       "      <td>4.0</td>\n",
       "      <td>80</td>\n",
       "      <td>1</td>\n",
       "      <td>{'RestaurantsDelivery': 'False', 'OutdoorSeati...</td>\n",
       "      <td>Restaurants, Food, Bubble Tea, Coffee &amp; Tea, B...</td>\n",
       "      <td>{'Monday': '7:0-20:0', 'Tuesday': '7:0-20:0', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mWMc6_wTdE0EUBKIGXDVfA</td>\n",
       "      <td>Perkiomen Valley Brewery</td>\n",
       "      <td>101 Walnut St</td>\n",
       "      <td>Green Lane</td>\n",
       "      <td>PA</td>\n",
       "      <td>18054</td>\n",
       "      <td>40.338183</td>\n",
       "      <td>-75.471659</td>\n",
       "      <td>4.5</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "      <td>{'BusinessAcceptsCreditCards': 'True', 'Wheelc...</td>\n",
       "      <td>Brewpubs, Breweries, Food</td>\n",
       "      <td>{'Wednesday': '14:0-22:0', 'Thursday': '16:0-2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              business_id                      name  \\\n",
       "0  Pns2l4eNsfO8kk83dixA6A  Abby Rappoport, LAC, CMQ   \n",
       "1  mpf3x-BjTdTEA3yCZrAYPw             The UPS Store   \n",
       "2  tUFrWirKiKi_TAnsVWINQQ                    Target   \n",
       "3  MTSW4McQd7CbVtyjqoe9mw        St Honore Pastries   \n",
       "4  mWMc6_wTdE0EUBKIGXDVfA  Perkiomen Valley Brewery   \n",
       "\n",
       "                           address           city state postal_code  \\\n",
       "0           1616 Chapala St, Ste 2  Santa Barbara    CA       93101   \n",
       "1  87 Grasso Plaza Shopping Center         Affton    MO       63123   \n",
       "2             5255 E Broadway Blvd         Tucson    AZ       85711   \n",
       "3                      935 Race St   Philadelphia    PA       19107   \n",
       "4                    101 Walnut St     Green Lane    PA       18054   \n",
       "\n",
       "    latitude   longitude  stars  review_count  is_open  \\\n",
       "0  34.426679 -119.711197    5.0             7        0   \n",
       "1  38.551126  -90.335695    3.0            15        1   \n",
       "2  32.223236 -110.880452    3.5            22        0   \n",
       "3  39.955505  -75.155564    4.0            80        1   \n",
       "4  40.338183  -75.471659    4.5            13        1   \n",
       "\n",
       "                                          attributes  \\\n",
       "0                      {'ByAppointmentOnly': 'True'}   \n",
       "1             {'BusinessAcceptsCreditCards': 'True'}   \n",
       "2  {'BikeParking': 'True', 'BusinessAcceptsCredit...   \n",
       "3  {'RestaurantsDelivery': 'False', 'OutdoorSeati...   \n",
       "4  {'BusinessAcceptsCreditCards': 'True', 'Wheelc...   \n",
       "\n",
       "                                          categories  \\\n",
       "0  Doctors, Traditional Chinese Medicine, Naturop...   \n",
       "1  Shipping Centers, Local Services, Notaries, Ma...   \n",
       "2  Department Stores, Shopping, Fashion, Home & G...   \n",
       "3  Restaurants, Food, Bubble Tea, Coffee & Tea, B...   \n",
       "4                          Brewpubs, Breweries, Food   \n",
       "\n",
       "                                               hours  \n",
       "0                                               None  \n",
       "1  {'Monday': '0:0-0:0', 'Tuesday': '8:0-18:30', ...  \n",
       "2  {'Monday': '8:0-22:0', 'Tuesday': '8:0-22:0', ...  \n",
       "3  {'Monday': '7:0-20:0', 'Tuesday': '7:0-20:0', ...  \n",
       "4  {'Wednesday': '14:0-22:0', 'Thursday': '16:0-2...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# File Path to JSON file\n",
    "file_path = '.extracted_data/yelp_academic_dataset_business.json'\n",
    "\n",
    "# Load the JSON file into a DataFrame,\n",
    "business = pd.read_json(file_path, lines=True)\n",
    "\n",
    "# Displaying first 5 entries\n",
    "business.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8b7d6725-23e9-4d93-8cd9-5ae4bc8bfbfd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 150346 entries, 0 to 150345\n",
      "Data columns (total 14 columns):\n",
      " #   Column        Non-Null Count   Dtype  \n",
      "---  ------        --------------   -----  \n",
      " 0   business_id   150346 non-null  object \n",
      " 1   name          150346 non-null  object \n",
      " 2   address       150346 non-null  object \n",
      " 3   city          150346 non-null  object \n",
      " 4   state         150346 non-null  object \n",
      " 5   postal_code   150346 non-null  object \n",
      " 6   latitude      150346 non-null  float64\n",
      " 7   longitude     150346 non-null  float64\n",
      " 8   stars         150346 non-null  float64\n",
      " 9   review_count  150346 non-null  int64  \n",
      " 10  is_open       150346 non-null  int64  \n",
      " 11  attributes    136602 non-null  object \n",
      " 12  categories    150243 non-null  object \n",
      " 13  hours         127123 non-null  object \n",
      "dtypes: float64(3), int64(2), object(9)\n",
      "memory usage: 16.1+ MB\n"
     ]
    }
   ],
   "source": [
    "business.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e07b001-2e85-451b-9ec9-b81169abbe40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>business_id</th>\n",
       "      <th>name</th>\n",
       "      <th>address</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>postal_code</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>stars</th>\n",
       "      <th>review_count</th>\n",
       "      <th>is_open</th>\n",
       "      <th>attributes</th>\n",
       "      <th>categories</th>\n",
       "      <th>hours</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MTSW4McQd7CbVtyjqoe9mw</td>\n",
       "      <td>St Honore Pastries</td>\n",
       "      <td>935 Race St</td>\n",
       "      <td>Philadelphia</td>\n",
       "      <td>PA</td>\n",
       "      <td>19107</td>\n",
       "      <td>39.955505</td>\n",
       "      <td>-75.155564</td>\n",
       "      <td>4.0</td>\n",
       "      <td>80</td>\n",
       "      <td>1</td>\n",
       "      <td>{'RestaurantsDelivery': 'False', 'OutdoorSeati...</td>\n",
       "      <td>Restaurants, Food, Bubble Tea, Coffee &amp; Tea, B...</td>\n",
       "      <td>{'Monday': '7:0-20:0', 'Tuesday': '7:0-20:0', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CF33F8-E6oudUQ46HnavjQ</td>\n",
       "      <td>Sonic Drive-In</td>\n",
       "      <td>615 S Main St</td>\n",
       "      <td>Ashland City</td>\n",
       "      <td>TN</td>\n",
       "      <td>37015</td>\n",
       "      <td>36.269593</td>\n",
       "      <td>-87.058943</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>{'BusinessParking': 'None', 'BusinessAcceptsCr...</td>\n",
       "      <td>Burgers, Fast Food, Sandwiches, Food, Ice Crea...</td>\n",
       "      <td>{'Monday': '0:0-0:0', 'Tuesday': '6:0-22:0', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>k0hlBqXX-Bt0vf1op7Jr1w</td>\n",
       "      <td>Tsevi's Pub And Grill</td>\n",
       "      <td>8025 Mackenzie Rd</td>\n",
       "      <td>Affton</td>\n",
       "      <td>MO</td>\n",
       "      <td>63123</td>\n",
       "      <td>38.565165</td>\n",
       "      <td>-90.321087</td>\n",
       "      <td>3.0</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>{'Caters': 'True', 'Alcohol': 'u'full_bar'', '...</td>\n",
       "      <td>Pubs, Restaurants, Italian, Bars, American (Tr...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>bBDDEgkFA1Otx9Lfe7BZUQ</td>\n",
       "      <td>Sonic Drive-In</td>\n",
       "      <td>2312 Dickerson Pike</td>\n",
       "      <td>Nashville</td>\n",
       "      <td>TN</td>\n",
       "      <td>37207</td>\n",
       "      <td>36.208102</td>\n",
       "      <td>-86.768170</td>\n",
       "      <td>1.5</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'RestaurantsAttire': ''casual'', 'Restaurants...</td>\n",
       "      <td>Ice Cream &amp; Frozen Yogurt, Fast Food, Burgers,...</td>\n",
       "      <td>{'Monday': '0:0-0:0', 'Tuesday': '6:0-21:0', '...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>eEOYSgkmpB90uNA7lDOMRA</td>\n",
       "      <td>Vietnamese Food Truck</td>\n",
       "      <td></td>\n",
       "      <td>Tampa Bay</td>\n",
       "      <td>FL</td>\n",
       "      <td>33602</td>\n",
       "      <td>27.955269</td>\n",
       "      <td>-82.456320</td>\n",
       "      <td>4.0</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>{'Alcohol': ''none'', 'OutdoorSeating': 'None'...</td>\n",
       "      <td>Vietnamese, Food, Restaurants, Food Trucks</td>\n",
       "      <td>{'Monday': '11:0-14:0', 'Tuesday': '11:0-14:0'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               business_id                   name              address  \\\n",
       "3   MTSW4McQd7CbVtyjqoe9mw     St Honore Pastries          935 Race St   \n",
       "5   CF33F8-E6oudUQ46HnavjQ         Sonic Drive-In        615 S Main St   \n",
       "8   k0hlBqXX-Bt0vf1op7Jr1w  Tsevi's Pub And Grill    8025 Mackenzie Rd   \n",
       "9   bBDDEgkFA1Otx9Lfe7BZUQ         Sonic Drive-In  2312 Dickerson Pike   \n",
       "11  eEOYSgkmpB90uNA7lDOMRA  Vietnamese Food Truck                        \n",
       "\n",
       "            city state postal_code   latitude  longitude  stars  review_count  \\\n",
       "3   Philadelphia    PA       19107  39.955505 -75.155564    4.0            80   \n",
       "5   Ashland City    TN       37015  36.269593 -87.058943    2.0             6   \n",
       "8         Affton    MO       63123  38.565165 -90.321087    3.0            19   \n",
       "9      Nashville    TN       37207  36.208102 -86.768170    1.5            10   \n",
       "11     Tampa Bay    FL       33602  27.955269 -82.456320    4.0            10   \n",
       "\n",
       "    is_open                                         attributes  \\\n",
       "3         1  {'RestaurantsDelivery': 'False', 'OutdoorSeati...   \n",
       "5         1  {'BusinessParking': 'None', 'BusinessAcceptsCr...   \n",
       "8         0  {'Caters': 'True', 'Alcohol': 'u'full_bar'', '...   \n",
       "9         1  {'RestaurantsAttire': ''casual'', 'Restaurants...   \n",
       "11        1  {'Alcohol': ''none'', 'OutdoorSeating': 'None'...   \n",
       "\n",
       "                                           categories  \\\n",
       "3   Restaurants, Food, Bubble Tea, Coffee & Tea, B...   \n",
       "5   Burgers, Fast Food, Sandwiches, Food, Ice Crea...   \n",
       "8   Pubs, Restaurants, Italian, Bars, American (Tr...   \n",
       "9   Ice Cream & Frozen Yogurt, Fast Food, Burgers,...   \n",
       "11         Vietnamese, Food, Restaurants, Food Trucks   \n",
       "\n",
       "                                                hours  \n",
       "3   {'Monday': '7:0-20:0', 'Tuesday': '7:0-20:0', ...  \n",
       "5   {'Monday': '0:0-0:0', 'Tuesday': '6:0-22:0', '...  \n",
       "8                                                None  \n",
       "9   {'Monday': '0:0-0:0', 'Tuesday': '6:0-21:0', '...  \n",
       "11  {'Monday': '11:0-14:0', 'Tuesday': '11:0-14:0'...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter the DataFrame to keep only rows that contain the word \"restaurant\"\n",
    "restaurants_df = business[business['categories'].str.contains(\"restaurant\", case=False, na=False)]\n",
    "\n",
    "# Display the first few rows of the filtered DataFrame\n",
    "restaurants_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "105f4675-3098-4aed-bb70-d087565b32b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 52286 entries, 3 to 150340\n",
      "Data columns (total 14 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   business_id   52286 non-null  object \n",
      " 1   name          52286 non-null  object \n",
      " 2   address       52286 non-null  object \n",
      " 3   city          52286 non-null  object \n",
      " 4   state         52286 non-null  object \n",
      " 5   postal_code   52286 non-null  object \n",
      " 6   latitude      52286 non-null  float64\n",
      " 7   longitude     52286 non-null  float64\n",
      " 8   stars         52286 non-null  float64\n",
      " 9   review_count  52286 non-null  int64  \n",
      " 10  is_open       52286 non-null  int64  \n",
      " 11  attributes    51720 non-null  object \n",
      " 12  categories    52286 non-null  object \n",
      " 13  hours         45007 non-null  object \n",
      "dtypes: float64(3), int64(2), object(9)\n",
      "memory usage: 6.0+ MB\n"
     ]
    }
   ],
   "source": [
    "restaurants_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9bc6883c-790c-46b4-89b3-c649175bbc31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving restaurant data to csv \n",
    "restaurants_df.to_csv('data/restaurants.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83b4a9a6-db5e-4907-a16d-2b23535a0088",
   "metadata": {},
   "source": [
    "***\n",
    "## SECOND DATASET CONTAINING REVIEWS INFORMATION\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5444f411-595c-4059-a1df-3ae84ae8ff3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# File Path to  JSON file\n",
    "file_path = '.extracted_data/yelp_academic_dataset_review.json'\n",
    "\n",
    "# Define a chunk size\n",
    "chunk_size = 10000\n",
    "\n",
    "# Initialize an empty list to hold chunks\n",
    "chunks = []\n",
    "\n",
    "# Read the JSON file in chunks\n",
    "with pd.read_json(file_path, lines=True, chunksize=chunk_size) as reader:\n",
    "    for chunk in reader:\n",
    "        # Process each chunk (e.g., filter, aggregate)\n",
    "        chunks.append(chunk)\n",
    "\n",
    "# Concatenate all chunks into a single DataFrame\n",
    "reviews = pd.concat(chunks, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e37fd0be-801f-4433-969c-cb6d8e3fab71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 6990280 entries, 0 to 6990279\n",
      "Data columns (total 9 columns):\n",
      " #   Column       Dtype         \n",
      "---  ------       -----         \n",
      " 0   review_id    object        \n",
      " 1   user_id      object        \n",
      " 2   business_id  object        \n",
      " 3   stars        int64         \n",
      " 4   useful       int64         \n",
      " 5   funny        int64         \n",
      " 6   cool         int64         \n",
      " 7   text         object        \n",
      " 8   date         datetime64[ns]\n",
      "dtypes: datetime64[ns](1), int64(4), object(4)\n",
      "memory usage: 480.0+ MB\n"
     ]
    }
   ],
   "source": [
    "# information on the reviews dataset\n",
    "reviews.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "49c1780e-0235-4890-a707-7b4138ff934a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "52286"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting the unique values in the restaurant_df's business Id column\n",
    "restaurants_ids = restaurants_df['business_id'].unique()\n",
    "len(restaurants_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "47ee3b82-6054-47f4-a935-85fdc3904a83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filtering the reveiws dataframe to include only reviews related to restaurants\n",
    "rest_reviews = reviews[reviews['business_id'].isin(restaurants_ids)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1a4e8276-e192-4c71-9528-37b9fcbcca7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 4724684 entries, 0 to 6990279\n",
      "Data columns (total 9 columns):\n",
      " #   Column       Dtype         \n",
      "---  ------       -----         \n",
      " 0   review_id    object        \n",
      " 1   user_id      object        \n",
      " 2   business_id  object        \n",
      " 3   stars        int64         \n",
      " 4   useful       int64         \n",
      " 5   funny        int64         \n",
      " 6   cool         int64         \n",
      " 7   text         object        \n",
      " 8   date         datetime64[ns]\n",
      "dtypes: datetime64[ns](1), int64(4), object(4)\n",
      "memory usage: 360.5+ MB\n"
     ]
    }
   ],
   "source": [
    "# Getting resturant reviews infor\n",
    "rest_reviews.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4f142437-52fd-4f8b-9926-e289bc24ce16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>business_id</th>\n",
       "      <th>stars</th>\n",
       "      <th>useful</th>\n",
       "      <th>funny</th>\n",
       "      <th>cool</th>\n",
       "      <th>text</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KU_O5udG6zpxOg-VcAEodg</td>\n",
       "      <td>mh_-eMZ6K5RLWhZyISBhwA</td>\n",
       "      <td>XQfwVwDr-v0ZS3_CbbE5Xw</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>If you decide to eat here, just be aware it is...</td>\n",
       "      <td>2018-07-07 22:09:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>saUsX_uimxRlCVr67Z4Jig</td>\n",
       "      <td>8g_iMtfSiwikVnbP2etR0A</td>\n",
       "      <td>YjUWPpI6HXG530lwP-fb2A</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Family diner. Had the buffet. Eclectic assortm...</td>\n",
       "      <td>2014-02-05 20:30:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AqPFMleE6RsU23_auESxiA</td>\n",
       "      <td>_7bHUi9Uuf5__HHc_Q8guQ</td>\n",
       "      <td>kxX2SOes4o-D3ZQBkiMRfA</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Wow!  Yummy, different,  delicious.   Our favo...</td>\n",
       "      <td>2015-01-04 00:01:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sx8TMOWLNuJBWer-0pcmoA</td>\n",
       "      <td>bcjbaE6dDog4jkNY91ncLQ</td>\n",
       "      <td>e4Vwtrqf-wpJfwesgvdgxQ</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Cute interior and owner (?) gave us tour of up...</td>\n",
       "      <td>2017-01-14 20:54:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>JrIxlS1TzJ-iCu79ul40cQ</td>\n",
       "      <td>eUta8W_HdHMXPzLBBZhL1A</td>\n",
       "      <td>04UD14gamNjLY0IDYVhHJg</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>I am a long term frequent customer of this est...</td>\n",
       "      <td>2015-09-23 23:10:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                review_id                 user_id             business_id  \\\n",
       "0  KU_O5udG6zpxOg-VcAEodg  mh_-eMZ6K5RLWhZyISBhwA  XQfwVwDr-v0ZS3_CbbE5Xw   \n",
       "2  saUsX_uimxRlCVr67Z4Jig  8g_iMtfSiwikVnbP2etR0A  YjUWPpI6HXG530lwP-fb2A   \n",
       "3  AqPFMleE6RsU23_auESxiA  _7bHUi9Uuf5__HHc_Q8guQ  kxX2SOes4o-D3ZQBkiMRfA   \n",
       "4  Sx8TMOWLNuJBWer-0pcmoA  bcjbaE6dDog4jkNY91ncLQ  e4Vwtrqf-wpJfwesgvdgxQ   \n",
       "5  JrIxlS1TzJ-iCu79ul40cQ  eUta8W_HdHMXPzLBBZhL1A  04UD14gamNjLY0IDYVhHJg   \n",
       "\n",
       "   stars  useful  funny  cool  \\\n",
       "0      3       0      0     0   \n",
       "2      3       0      0     0   \n",
       "3      5       1      0     1   \n",
       "4      4       1      0     1   \n",
       "5      1       1      2     1   \n",
       "\n",
       "                                                text                date  \n",
       "0  If you decide to eat here, just be aware it is... 2018-07-07 22:09:11  \n",
       "2  Family diner. Had the buffet. Eclectic assortm... 2014-02-05 20:30:30  \n",
       "3  Wow!  Yummy, different,  delicious.   Our favo... 2015-01-04 00:01:03  \n",
       "4  Cute interior and owner (?) gave us tour of up... 2017-01-14 20:54:15  \n",
       "5  I am a long term frequent customer of this est... 2015-09-23 23:10:31  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " # first 5 entries in restaurant reviews\n",
    "rest_reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c44c9666-6d23-42ee-83c3-742e5803967a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2005-02-16 03:23:22'), Timestamp('2022-01-19 19:48:25'))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Timeframe of the dataset\n",
    "rest_reviews.date.min(), rest_reviews.date.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "efa4344a-fedd-4f1d-a7ec-743b2125bb6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>business_id</th>\n",
       "      <th>stars</th>\n",
       "      <th>useful</th>\n",
       "      <th>funny</th>\n",
       "      <th>cool</th>\n",
       "      <th>text</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>KU_O5udG6zpxOg-VcAEodg</td>\n",
       "      <td>mh_-eMZ6K5RLWhZyISBhwA</td>\n",
       "      <td>XQfwVwDr-v0ZS3_CbbE5Xw</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>If you decide to eat here, just be aware it is...</td>\n",
       "      <td>2018-07-07 22:09:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sx8TMOWLNuJBWer-0pcmoA</td>\n",
       "      <td>bcjbaE6dDog4jkNY91ncLQ</td>\n",
       "      <td>e4Vwtrqf-wpJfwesgvdgxQ</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Cute interior and owner (?) gave us tour of up...</td>\n",
       "      <td>2017-01-14 20:54:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>lUUhg8ltDsUZ9h0xnwY4Dg</td>\n",
       "      <td>RreNy--tOmXMl1en0wiBOg</td>\n",
       "      <td>cPepkJeRMtHapc_b2Oe_dw</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>I was really between 3 and 4 stars for this on...</td>\n",
       "      <td>2018-07-17 03:30:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>-P5E9BYUaK7s3PwBF5oAyg</td>\n",
       "      <td>Jha0USGDMefGFRLik_xFQg</td>\n",
       "      <td>bMratNjTG5ZFEA6hVyr-xQ</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>First time there and it was excellent!!! It fe...</td>\n",
       "      <td>2017-02-19 13:32:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>YbMyvlDA2W3Py5lTz8VK-A</td>\n",
       "      <td>4hBhtCSgoxkrFgHa4YAD-w</td>\n",
       "      <td>bbEXAEFr4RYHLlZ-HFssTA</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Great burgers,fries and salad!  Burgers have a...</td>\n",
       "      <td>2017-01-02 03:17:34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 review_id                 user_id             business_id  \\\n",
       "0   KU_O5udG6zpxOg-VcAEodg  mh_-eMZ6K5RLWhZyISBhwA  XQfwVwDr-v0ZS3_CbbE5Xw   \n",
       "4   Sx8TMOWLNuJBWer-0pcmoA  bcjbaE6dDog4jkNY91ncLQ  e4Vwtrqf-wpJfwesgvdgxQ   \n",
       "24  lUUhg8ltDsUZ9h0xnwY4Dg  RreNy--tOmXMl1en0wiBOg  cPepkJeRMtHapc_b2Oe_dw   \n",
       "30  -P5E9BYUaK7s3PwBF5oAyg  Jha0USGDMefGFRLik_xFQg  bMratNjTG5ZFEA6hVyr-xQ   \n",
       "31  YbMyvlDA2W3Py5lTz8VK-A  4hBhtCSgoxkrFgHa4YAD-w  bbEXAEFr4RYHLlZ-HFssTA   \n",
       "\n",
       "    stars  useful  funny  cool  \\\n",
       "0       3       0      0     0   \n",
       "4       4       1      0     1   \n",
       "24      4       1      0     1   \n",
       "30      5       0      0     0   \n",
       "31      5       0      0     0   \n",
       "\n",
       "                                                 text                date  \n",
       "0   If you decide to eat here, just be aware it is... 2018-07-07 22:09:11  \n",
       "4   Cute interior and owner (?) gave us tour of up... 2017-01-14 20:54:15  \n",
       "24  I was really between 3 and 4 stars for this on... 2018-07-17 03:30:07  \n",
       "30  First time there and it was excellent!!! It fe... 2017-02-19 13:32:05  \n",
       "31  Great burgers,fries and salad!  Burgers have a... 2017-01-02 03:17:34  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Starting date of the dataframe to get 5 year worth of data\n",
    "start_date = '2017-01-01 00:00:00'\n",
    "\n",
    "# Getting 5 years worth of reviews\n",
    "five_year_reviews = rest_reviews[(rest_reviews['date'] >= start_date)]\n",
    "\n",
    "# first five entries\n",
    "five_year_reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bb197814-9fae-4890-9dcb-324c212d1631",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2559586 entries, 0 to 6990279\n",
      "Data columns (total 9 columns):\n",
      " #   Column       Dtype         \n",
      "---  ------       -----         \n",
      " 0   review_id    object        \n",
      " 1   user_id      object        \n",
      " 2   business_id  object        \n",
      " 3   stars        int64         \n",
      " 4   useful       int64         \n",
      " 5   funny        int64         \n",
      " 6   cool         int64         \n",
      " 7   text         object        \n",
      " 8   date         datetime64[ns]\n",
      "dtypes: datetime64[ns](1), int64(4), object(4)\n",
      "memory usage: 195.3+ MB\n"
     ]
    }
   ],
   "source": [
    "# Getting information on the 5 year reviews dataset\n",
    "five_year_reviews.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f2852f91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2559586 entries, 0 to 6990279\n",
      "Data columns (total 4 columns):\n",
      " #   Column       Dtype         \n",
      "---  ------       -----         \n",
      " 0   user_id      object        \n",
      " 1   business_id  object        \n",
      " 2   stars        int64         \n",
      " 3   date         datetime64[ns]\n",
      "dtypes: datetime64[ns](1), int64(1), object(2)\n",
      "memory usage: 97.6+ MB\n"
     ]
    }
   ],
   "source": [
    "# Filtering to have only the relevant dataset\n",
    "five_year_reviews=five_year_reviews[['user_id','business_id','stars','date']]\n",
    "\n",
    "five_year_reviews.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1a807316",
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving the one_year_reviews to users.csv\n",
    "five_year_reviews.to_csv('data/users.csv', index= False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "285b6bf5-06e6-4c0c-832c-0b2b9b677028",
   "metadata": {},
   "source": [
    "***\n",
    "### CONCLUSION\n",
    "***\n",
    "The JSON datasets used were filtered for ease of use as follows:\n",
    "\n",
    "#### 1. Business Dataset\n",
    "- This dataset initially contained 150346 unique business entries but was filtered down to 52286 entries as we only considered restaurants\n",
    "- It was then saved as restauarant.csv in the data folder\n",
    "\n",
    "#### 2. Reviews Dataset\n",
    "- This dataset initially had approximately 6.99M reviews from all businesses. \n",
    "- It was filtered initially to only include reviews from restaurants bringing them down to 4.7M reviews\n",
    "- Then further filtered to only include reviews for 5 year i.e. 2017-2022 bringing the dataset to approximately 2.5M reviews\n",
    "- This was done to keep the dataset as current as possible and to save on computational capacity.\n",
    "- It was then saved as users.csv in the data folder"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python keras-env",
   "language": "python",
   "name": "keras-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}