teabook/api-demo-socrata-yelp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a46ecd25",
   "metadata": {},
   "source": [
    "# Intro to the Socrata API with the NYC Dog Licensing Dataset & Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0586f7df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sodapy import Socrata\n",
    "import requests\n",
    "import plotly.express as px\n",
    "from urllib.request import urlopen\n",
    "import json\n",
    "pd.options.display.max_rows = 100\n",
    "pd.options.display.max_columns = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "000c04cc",
   "metadata": {},
   "source": [
    "## What is an API?\n",
    "\n",
    "The Socrata API follows the REST (REpresentational State Transfer) design pattern.\n",
    "\n",
    "REST stands for REpresentational State Transfer. It originally had a more abstract meaning, but has come to be a shorthand name for web sites that act a bit like python functions, taking as inputs values for certain parameters and producing outputs in the form of a long text string.\n",
    "\n",
    "API stands for Application Programming Interface. An API specifies how an external program (an application program) can request that a program perform certain computations.\n",
    "\n",
    "Putting the two together, a REST API specifies how external programs can make HTTP requests to a web site in order to request that some computation be carried out and data returned as output. When a website is designed to accept requests generated by other computer programs, and produce outputs to be consumed by other programs, it is sometimes called a web service, as opposed to a web site which produces output meant for humans to consume in a web browser."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b6bd2f8",
   "metadata": {},
   "source": [
    "## Anatomy of a URL with the Socrata API\n",
    "\n",
    "For this demonstration we will be making requests to the Socrata API with the The [NYC Dog License Dataset](https://data.cityofnewyork.us/Health/NYC-Dog-Licensing-Dataset/nu7n-tubp).\n",
    "\n",
    "Components of the url:\n",
    "- headers (required sometimes, but not always)\n",
    "- endpoints\n",
    "- parameters\n",
    "- API key\n",
    "\n",
    "In a REST API, the client or application program makes an HTTP request that includes information about what kind of request it is making. Web sites are free to define whatever format they want for how the request should be formatted. \n",
    "\n",
    "In this format, the URL has a standard structure:\n",
    "\n",
    "- the base URL: https://data.cityofnewyork.us/resource/nu7n-tubp.json\n",
    "- a ? character\n",
    "- or more key-value pairs (parameters), formatted as key=value pairs and separated by the & character\n",
    "\n",
    "For example, consider the following url requests to the NYC Dog License Dataset with the Socrata API:\n",
    "- https://data.cityofnewyork.us/resource/nu7n-tubp.json?rownumber=40895\n",
    "- https://data.cityofnewyork.us/resource/nu7n-tubp.json?licenseissueddate=2014-09-12T00:00:00.000\n",
    "- https://data.cityofnewyork.us/resource/nu7n-tubp.json?animalname=PAIGE&zipcode=10035\n",
    "\n",
    "Try copying that URL into a browser, or just clicking on it. Depending on your browser, it may put the contents into a file attachment that you have to open up to see the contents, or it may just show the contents in a browser window."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3aed5c0",
   "metadata": {},
   "source": [
    "## API Documentation\n",
    "The [API documentation for the NYC Dog License Dataset](https://dev.socrata.com/foundry/data.cityofnewyork.us/nu7n-tubp) contains all of the parts of the url that we need. The fields in the documentation describe the parameters we can use to filter the data in the url request. It is important to read the API documentation for every dataset you use in NYC Open Data, as each dataset has unique features that need to be considered when making requests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e12d2ee",
   "metadata": {},
   "source": [
    "## Application Tokens and API Keys\n",
    "The Socrata Open Data API has two concepts around API access: authentication and application tokens. You only need to authenticate if you wish to add, delete, or modify data that is attached to your account, or if you wish to read data you own that you have marked as private. Read-only requests only require the application token. See the [SODA Producer API documentation](http://dev.socrata.com/publishers/soda-producer/soda-producer-basics.html) for more details on changing data. See the [API docs](https://dev.socrata.com/docs/app-tokens.html) for more information on app tokens and API keys. For this example we are requesting data from the API for analysis, so only the application token is needed.\n",
    "\n",
    "The Socrata Open Data API uses application tokens for two purposes:\n",
    "- Using an application token allows the API to throttle by application, rather than via IP address, which gives you a higher throttling limit\n",
    "- Authentication using OAuth (OAuth 2.0 is the preferred option for cases where you are building a web or mobile application that needs to perform actions on behalf of the user, like accessing data, and the interaction model allows you to present the user with a form to obtain their permission for the app to do so.)\n",
    "\n",
    "See [Generating App Tokens and API Keys](https://support.socrata.com/hc/en-us/articles/210138558-Generating-an-App-Token) for instructions on generating your own app token.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efc86ff5",
   "metadata": {},
   "source": [
    "## Using the Socrata client to make requests\n",
    "We are using the Python sodapy library to make our request. The client sends our request for the data, and the API sends the data to us in .json format. Then the pandas library is used to format the results into a dataframe. The get() method is used with parameters to filter the data in our request. Check out the [SoSQL examples](https://github.com/xmunoz/sodapy/blob/master/examples/soql_queries.ipynb) in the sodapy github for more info.  The filters use the [SoSQL statements](https://dev.socrata.com/docs/queries/), which are based on SQL and have similar syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd162e5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unauthenticated client only works with public data sets. Note 'None'\n",
    "# in place of application token, and no username or password:\n",
    "client = Socrata(\"data.cityofnewyork.us\", 'tpwmd1DMwYMlz8q2BgPqJ48Ky')\n",
    "\n",
    "# Example authenticated client (needed for non-public datasets):\n",
    "# client = Socrata('data.cityofnewyork.us',\n",
    "#                  'appTOKEN',\n",
    "#                   username='username',\n",
    "#                   password='password')\n",
    "\n",
    "# Results returned as JSON from API / converted to Python list of\n",
    "# dictionaries by sodapy.\n",
    "results = client.get(\"nu7n-tubp\", \n",
    "                     limit=5000,\n",
    "                     #where = \"extract_year = '2017' AND breedname = 'Boxer'\",\n",
    "                     where = \"breedname = 'Akita'\",\n",
    "                     #where = \"animalname = 'PAIGE'\",\n",
    "                     select = \"animalname, breedname, zipcode, extract_year\",\n",
    "                     order = \"zipcode\")\n",
    "\n",
    "# Convert to pandas DataFrame\n",
    "doggy_data = pd.DataFrame.from_records(results)\n",
    "doggy_data['count'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ea18e63",
   "metadata": {},
   "outputs": [],
   "source": [
    "doggy_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9638cfee",
   "metadata": {},
   "outputs": [],
   "source": [
    "doggy_data.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e316fc48",
   "metadata": {},
   "outputs": [],
   "source": [
    "doggy_data['breedname'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a828306f",
   "metadata": {},
   "outputs": [],
   "source": [
    "doggy_data['extract_year'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819be223",
   "metadata": {},
   "source": [
    "## Visualize the data with plotly express"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07ec64f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create grouped dataframe by count of dog registration by zipcode\n",
    "doggy_zips_grouped = doggy_data[['zipcode', 'count']].groupby(by = 'zipcode').sum().reset_index()\n",
    "\n",
    "# use GeoJSON file from NYC opendata which contains GIS data for zip code boundaries in NYC\n",
    "# web page with info on data set url here: \n",
    "# https://data.cityofnewyork.us/Health/Modified-Zip-Code-Tabulation-Areas-MODZCTA-/pri4-ifjk\n",
    "with urlopen('https://data.cityofnewyork.us/resource/pri4-ifjk.geojson') as response:\n",
    "    zip_codes = json.load(response)\n",
    "\n",
    "\n",
    "fig = px.choropleth_mapbox(doggy_zips_grouped, geojson=zip_codes, locations='zipcode', color='count',\n",
    "                           featureidkey='properties.modzcta',\n",
    "                           color_continuous_scale=\"Viridis\",\n",
    "                           range_color=(0, max(doggy_zips_grouped['count'])),\n",
    "                           mapbox_style=\"carto-positron\",\n",
    "                           zoom=9.25, center = {\"lat\": 40.743, \"lon\": -73.988},\n",
    "                           opacity=0.5,\n",
    "                           labels={},\n",
    "                           title=\"Number of Dog License Registrations in NYC by Zipcode\"\n",
    "                          ).update(layout=dict(title=dict(x=0.5)))\n",
    "fig.update_layout(margin={\"r\":0,\"t\":30,\"l\":0,\"b\":0})\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82fc0aa",
   "metadata": {},
   "source": [
    "# Extracting Data with the Yelp Fusion API\n",
    "The Yelp Fusion API allows you to get the best local content and user reviews from millions of businesses across 32 countries."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c334c3c",
   "metadata": {},
   "source": [
    "## Authentication & App Keys\n",
    "The [Yelp Fusion API](https://www.yelp.com/developers/documentation/v3/get_started) uses private key authentication to authenticate all endpoints. Your private API Key will be automatically generated after you create your app. For detailed instructions, refer to the [authentication guide](https://www.yelp.com/developers/documentation/v3/authentication). Follow the instructions in this guide to create an app and receive an app key. \n",
    "### To create an App and API Key:\n",
    "1. **Sign up for an account**\n",
    "![alt text](yelp_sign_up.png)\n",
    "2. **Fill out App form**\n",
    "![alt text](yelp_create_app.png)\n",
    "3. **The API Key will be used to query the Yelp Fusion API.** \n",
    "![alt text](yelp_api_key.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62555d58",
   "metadata": {},
   "source": [
    "## Url Components\n",
    "- **headers**: set the Authorization HTTP header value as Bearer API_KEY, i.e. 'Authorization': 'Bearer {API_KEY}'\n",
    "- **endpoints**: There are several endpoints under the base url https://api.yelp.com/v3. For our request we will be using the business search endpoint: /businesses/search. Our complete endpoint is https://api.yelp.com/v3/businesses/search. See the [Yelp Fusion API docs](https://www.yelp.com/developers/documentation/v3/get_started) for a complete list of endpoints.\n",
    "- **parameters**: for a complete list of parameters that can be used to query the data see the [/businesses/search](https://www.yelp.com/developers/documentation/v3/business_search) endpoint in the documentation. Some parameters used in our request are:\n",
    "    - term\n",
    "    - location\n",
    "    - categories\n",
    "    - offset\n",
    "    - limit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52aa7f68",
   "metadata": {},
   "source": [
    "## Python requests module\n",
    "In Python, there’s a module available called requests. You can use the get function in the requests module to fetch the contents of a page. The [requests.get()](https://docs.python-requests.org/en/latest/) method will be used to query the API.To authenticate API calls with the API Key, set the Authorization HTTP header value as Bearer API_KEY.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc4065c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# API only returns 1,000 results and 50 per request\n",
    "# we use the offset parameter to page through to the next 50\n",
    "# source: https://www.yelp.com/developers/faq\n",
    "\n",
    "# saving my api key as an environment variable\n",
    "# API_KEY = os.environ.get('YELP_API')\n",
    "# if API_KEY is None:\n",
    "#     config = AutoConfig(search_path = '.')\n",
    "#     API_KEY = config('YELP_API')\n",
    "\n",
    "API_KEY = 'HZzYHUHDbFqeNYQEiN8Y86945mE-1MKiUFgegyf-2AscR2J1owV1Smb8Vw4vbzkTCsGU1sC0cgwkIMYOB3_Io7OUnSySQ0dlrcW_PjX68HMsnH0K4VTtkVFFhYgBYnYx'\n",
    "\n",
    "# empty list to place our data for each page\n",
    "lst = []\n",
    "\n",
    "# our offset parameter - each page 50 rows\n",
    "offset = 0\n",
    "print('initial offset number: {}'.format(offset))\n",
    "\n",
    "# loop through the api 20 times (limit is 1000 rows with each page includes 50 rows)\n",
    "for i in range(20):\n",
    "    \n",
    "    try:\n",
    "        headers = {'Authorization': 'Bearer {}'.format(API_KEY)}\n",
    "        search_api_url = 'https://api.yelp.com/v3/businesses/search'\n",
    "        params = {\n",
    "                  'term': 'bubble tea', \n",
    "                  'categories': 'bubbletea',\n",
    "                  'location': 'New York City',\n",
    "                  'offset': offset,\n",
    "                  'limit': 50}\n",
    "\n",
    "        response = requests.get(search_api_url, \n",
    "                                headers=headers, \n",
    "                                params=params, \n",
    "                                timeout=10)\n",
    "\n",
    "        # return a dictionary\n",
    "        data_dict = response.json()\n",
    "        \n",
    "        # convert the business dictionary to a pandas dataframe\n",
    "        data = pd.DataFrame(data_dict['businesses'])\n",
    "        # append dataframe to list\n",
    "        lst.append(data)\n",
    "      \n",
    "        # add 50 to the offset to access a new page\n",
    "        offset += 50\n",
    "        print('current offset number: {}'.format(offset))\n",
    "        \n",
    "    except Exception as ex:\n",
    "        print('exception: {}\\nexit loop.'.format(ex))\n",
    "        break\n",
    "# concatenate all pages to one dataframe\n",
    "df = pd.concat(lst)\n",
    "df = df.reset_index(drop=True)\n",
    "\n",
    "rows, columns = df.shape\n",
    "print()\n",
    "print('query includes {:,} rows and {} columns.'.format(rows, columns))\n",
    "print('row id is unique: {}.'.format(df['id'].is_unique))\n",
    "\n",
    "if df['id'].is_unique == False:\n",
    "    duplicates = df.loc[df.duplicated(subset=['id'])]\n",
    "    vals = list(duplicates['name'].values) # this said duplicates.head()['name'].values, took out .head() to get all values\n",
    "    print('\\nduplicates found: {}.'.format(vals))\n",
    "    \n",
    "    df = df.drop_duplicates(subset=['id']).reset_index(drop=True)\n",
    "    print('dropping duplicates...')\n",
    "    \n",
    "    rows, columns = df.shape\n",
    "    print('\\nrow id is unique: {}.'.format(df['id'].is_unique))\n",
    "    print('query includes {:,} rows and {} columns.'.format(rows, columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14ee5adf",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ae1a2d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d35258f",
   "metadata": {},
   "source": [
    "## Visualize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e74471e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create grouped dataframe by count of dog registration by zipcode\n",
    "loc_df = pd.concat([df, df['location'].apply(pd.Series)], axis=1)\n",
    "loc_df['count'] = 1\n",
    "loc_df_grouped = loc_df[['name','zip_code','count']].groupby(by = 'zip_code').sum().reset_index()\n",
    "\n",
    "# use GeoJSON file from NYC opendata which contains GIS data for zip code boundaries in NYC\n",
    "# web page with info on data set url here: \n",
    "# https://data.cityofnewyork.us/Health/Modified-Zip-Code-Tabulation-Areas-MODZCTA-/pri4-ifjk\n",
    "with urlopen('https://data.cityofnewyork.us/resource/pri4-ifjk.geojson') as response:\n",
    "    zip_codes = json.load(response)\n",
    "\n",
    "\n",
    "fig = px.choropleth_mapbox(loc_df_grouped, geojson=zip_codes, locations='zip_code', color='count',\n",
    "                           featureidkey='properties.modzcta',\n",
    "                           color_continuous_scale=\"Viridis\",\n",
    "                           range_color=(0, max(loc_df_grouped['count'])),\n",
    "                           mapbox_style=\"carto-positron\",\n",
    "                           zoom=9.25, center = {\"lat\": 40.743, \"lon\": -73.988},\n",
    "                           opacity=0.5,\n",
    "                           labels={},\n",
    "                           title=\"Number of Boba Tea Shops in NYC by Zipcode\"\n",
    "                          ).update(layout=dict(title=dict(x=0.5)))\n",
    "fig.update_layout(margin={\"r\":0,\"t\":30,\"l\":0,\"b\":0})\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "884d721c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# scatter_plot with plotly express\n",
    "\n",
    "coordinates_df = pd.concat([df, df['coordinates'].apply(pd.Series)], axis=1)\n",
    "\n",
    "fig = px.scatter_mapbox(coordinates_df, \n",
    "                        lat='latitude', \n",
    "                        lon='longitude', \n",
    "                        hover_name=None,\n",
    "                        hover_data=['name'],\n",
    "                        color_discrete_sequence=['goldenrod'], \n",
    "                        zoom=9, \n",
    "                        title='Boba Tea Shops in NYC',\n",
    "                        size_max = 0.1).update(layout=dict(title=dict(x=0.5)))\n",
    "fig.update_layout(mapbox_style=\"carto-positron\")\n",
    "fig.update_layout(margin={\"r\":0,\"t\":30,\"l\":0,\"b\":0})\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e796b7ba",
   "metadata": {},
   "source": [
    "## Further data exploration\n",
    "- Where are the cheapest Boba Tea shops located in NYC? (hint: use the price parameter to filter the API query)\n",
    "- Which Boba shops in NYC have the highest rating?\n",
    "- Which Boba shops in NYC have the most ratings?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb2bd3b3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}