DV0101EN-3-5-1-Generating-Maps-in-Python-py-v2.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DV0101EN-SkillsNetwork/labs/Module%205/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n",
    "\n",
    "# Generating Maps with Python\n",
    "\n",
    "Estimated time needed: **30** minutes\n",
    "\n",
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    "*   Visualize geospatial data with Folium\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "In this lab, we will learn how to create maps for different objectives. To do that, we will part ways with Matplotlib and work with another Python visualization library, namely **Folium**. What is nice about **Folium** is that it was developed for the sole purpose of visualizing geospatial data. While other libraries are available to visualize geospatial data, such as **plotly**, they might have a cap on how many API calls you can make within a defined time frame. **Folium**, on the other hand, is completely free.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Table of Contents\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "\n",
    "1.  [Exploring Datasets with *pandas*](#0)<br>\n",
    "2.  [Downloading and Prepping Data](#2)<br>\n",
    "3.  [Introduction to Folium](#4) <br>\n",
    "4.  [Map with Markers](#6) <br>\n",
    "5.  [Choropleth Maps](#8) <br>\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Exploring Datasets with *pandas* and Matplotlib<a id=\"0\"></a>\n",
    "\n",
    "Toolkits: This lab heavily relies on [*pandas*](http://pandas.pydata.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDV0101ENSkillsNetwork20297740-2021-01-01) and [*Numpy*](http://www.numpy.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDV0101ENSkillsNetwork20297740-2021-01-01) for data wrangling, analysis, and visualization. The primary plotting library we will explore in this lab is [**Folium**](https://github.com/python-visualization/folium/).\n",
    "\n",
    "Datasets:\n",
    "\n",
    "1.  San Francisco Police Department Incidents for the year 2016 - [Police Department Incidents](https://data.sfgov.org/Public-Safety/Police-Department-Incidents-Previous-Year-2016-/ritf-b9ki?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDV0101ENSkillsNetwork20297740-2021-01-01) from San Francisco public data portal. Incidents derived from San Francisco Police Department (SFPD) Crime Incident Reporting system. Updated daily, showing data for the entire year of 2016. Address and location has been anonymized by moving to mid-block or to an intersection.\n",
    "\n",
    "2.  Immigration to Canada from 1980 to 2013 - [International migration flows to and from selected countries - The 2015 revision](http://www.un.org/en/development/desa/population/migration/data/empirical2/migrationflows.shtml?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDV0101ENSkillsNetwork20297740-2021-01-01) from United Nation's website. The dataset contains annual data on the flows of international migrants as recorded by the countries of destination. The data presents both inflows and outflows according to the place of birth, citizenship or place of previous / next residence both for foreigners and nationals. For this lesson, we will focus on the Canadian Immigration data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Downloading and Prepping Data <a id=\"2\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Import Primary Modules:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np  # useful for many scientific computing in Python\n",
    "import pandas as pd # primary data structure library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Introduction to Folium <a id=\"4\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Folium is a powerful Python library that helps you create several types of Leaflet maps. The fact that the Folium results are interactive makes this library very useful for dashboard building.\n",
    "\n",
    "From the official Folium documentation page:\n",
    "\n",
    "> Folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js library. Manipulate your data in Python, then visualize it in on a Leaflet map via Folium.\n",
    "\n",
    "> Folium makes it easy to visualize data that's been manipulated in Python on an interactive Leaflet map. It enables both the binding of data to a map for choropleth visualizations as well as passing Vincent/Vega visualizations as markers on the map.\n",
    "\n",
    "> The library has a number of built-in tilesets from OpenStreetMap, Mapbox, and Stamen, and supports custom tilesets with Mapbox or Cloudmade API keys. Folium supports both GeoJSON and TopoJSON overlays, as well as the binding of data to those overlays to create choropleth maps with color-brewer color schemes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let's install **Folium**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Folium** is not available by default. So, we first need to install it before we are able to import it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!conda install -c conda-forge folium=0.5.0 --yes\n",
    "import folium\n",
    "\n",
    "print('Folium installed and imported!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Generating the world map is straightforward in **Folium**. You simply create a **Folium** *Map* object, and then you display it. What is attractive about **Folium** maps is that they are interactive, so you can zoom into any region of interest despite the initial zoom level.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# define the world map\n",
    "world_map = folium.Map()\n",
    "\n",
    "# display world map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Go ahead. Try zooming in and out of the rendered map above.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "You can customize this default definition of the world map by specifying the centre of your map, and the initial zoom level.\n",
    "\n",
    "All locations on a map are defined by their respective *Latitude* and *Longitude* values. So you can create a map and pass in a center of *Latitude* and *Longitude* values of **\\[0, 0]**.\n",
    "\n",
    "For a defined center, you can also define the initial zoom level into that location when the map is rendered. **The higher the zoom level the more the map is zoomed into the center**.\n",
    "\n",
    "Let's create a map centered around Canada and play with the zoom level to see how it affects the rendered map.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# define the world map centered around Canada with a low zoom level\n",
    "world_map = folium.Map(location=[56.130, -106.35], zoom_start=4)\n",
    "\n",
    "# display world map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's create the map again with a higher zoom level.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# define the world map centered around Canada with a higher zoom level\n",
    "world_map = folium.Map(location=[56.130, -106.35], zoom_start=8)\n",
    "\n",
    "# display world map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As you can see, the higher the zoom level the more the map is zoomed into the given center.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Question**: Create a map of Mexico with a zoom level of 4.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<details><summary>Click here for a sample python solution</summary>\n",
    "\n",
    "```python\n",
    "    #The correct answer is:\n",
    "    \n",
    "    #define Mexico's geolocation coordinates\n",
    "    mexico_latitude = 23.6345 \n",
    "    mexico_longitude = -102.5528\n",
    "\n",
    "    # define the world map centered around Canada with a higher zoom level\n",
    "    mexico_map = folium.Map(location=[mexico_latitude, mexico_longitude], zoom_start=4)\n",
    "\n",
    "    # display world map\n",
    "    mexico_map\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Another cool feature of **Folium** is that you can generate different map styles.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### A. Stamen Toner Maps\n",
    "\n",
    "These are high-contrast B+W (black and white) maps. They are perfect for data mashups and exploring river meanders and coastal zones.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's create a Stamen Toner map of canada with a zoom level of 4.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# create a Stamen Toner map of the world centered around Canada\n",
    "world_map = folium.Map(location=[56.130, -106.35], zoom_start=4, tiles='Stamen Toner')\n",
    "\n",
    "# display map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Feel free to zoom in and out to see how this style compares to the default one.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### B. Stamen Terrain Maps\n",
    "\n",
    "These are maps that feature hill shading and natural vegetation colors. They showcase advanced labeling and linework generalization of dual-carriageway roads.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's create a Stamen Terrain map of Canada with zoom level 4.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# create a Stamen Toner map of the world centered around Canada\n",
    "world_map = folium.Map(location=[56.130, -106.35], zoom_start=4, tiles='Stamen Terrain')\n",
    "\n",
    "# display map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Feel free to zoom in and out to see how this style compares to Stamen Toner, and the default style.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Zoom in and notice how the borders start showing as you zoom in, and the displayed country names are in English.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Question**: Create a map of Mexico to visualize its hill shading and natural vegetation. Use a zoom level of 6.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "### type your answer here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<details><summary>Click here for a sample python solution</summary>\n",
    "\n",
    "```python\n",
    "    #The correct answer is:\n",
    "    \n",
    "    #define Mexico's geolocation coordinates\n",
    "    mexico_latitude = 23.6345 \n",
    "    mexico_longitude = -102.5528\n",
    "\n",
    "    # define the world map centered around Canada with a higher zoom level\n",
    "    mexico_map = folium.Map(location=[mexico_latitude, mexico_longitude], zoom_start=6, tiles='Stamen Terrain')\n",
    "\n",
    "    # display world map\n",
    "    mexico_map\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Maps with Markers <a id=\"6\"></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's download and import the data on police department incidents using *pandas* `read_csv()` method.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Download the dataset and read it into a *pandas* dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_incidents = pd.read_csv('https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DV0101EN-SkillsNetwork/Data%20Files/Police_Department_Incidents_-_Previous_Year__2016_.csv')\n",
    "\n",
    "print('Dataset downloaded and read into a pandas dataframe!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's take a look at the first five items in our dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_incidents.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "So each row consists of 13 features:\n",
    "\n",
    "> 1.  **IncidntNum**: Incident Number\n",
    "> 2.  **Category**: Category of crime or incident\n",
    "> 3.  **Descript**: Description of the crime or incident\n",
    "> 4.  **DayOfWeek**: The day of week on which the incident occurred\n",
    "> 5.  **Date**: The Date on which the incident occurred\n",
    "> 6.  **Time**: The time of day on which the incident occurred\n",
    "> 7.  **PdDistrict**: The police department district\n",
    "> 8.  **Resolution**: The resolution of the crime in terms whether the perpetrator was arrested or not\n",
    "> 9.  **Address**: The closest address to where the incident took place\n",
    "> 10. **X**: The longitude value of the crime location\n",
    "> 11. **Y**: The latitude value of the crime location\n",
    "> 12. **Location**: A tuple of the latitude and the longitude values\n",
    "> 13. **PdId**: The police department ID\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's find out how many entries there are in our dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_incidents.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "So the dataframe consists of 150,500 crimes, which took place in the year 2016. In order to reduce computational cost, let's just work with the first 100 incidents in this dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# get the first 100 crimes in the df_incidents dataframe\n",
    "limit = 100\n",
    "df_incidents = df_incidents.iloc[0:limit, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's confirm that our dataframe now consists only of 100 crimes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_incidents.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now that we reduced the data a little, let's visualize where these crimes took place in the city of San Francisco. We will use the default style, and we will initialize the zoom level to 12.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# San Francisco latitude and longitude values\n",
    "latitude = 37.77\n",
    "longitude = -122.42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# create map and display it\n",
    "sanfran_map = folium.Map(location=[latitude, longitude], zoom_start=12)\n",
    "\n",
    "# display the map of San Francisco\n",
    "sanfran_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's superimpose the locations of the crimes onto the map. The way to do that in **Folium** is to create a *feature group* with its own features and style and then add it to the `sanfran_map`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# instantiate a feature group for the incidents in the dataframe\n",
    "incidents = folium.map.FeatureGroup()\n",
    "\n",
    "# loop through the 100 crimes and add each to the incidents feature group\n",
    "for lat, lng, in zip(df_incidents.Y, df_incidents.X):\n",
    "    incidents.add_child(\n",
    "        folium.features.CircleMarker(\n",
    "            [lat, lng],\n",
    "            radius=5, # define how big you want the circle markers to be\n",
    "            color='yellow',\n",
    "            fill=True,\n",
    "            fill_color='blue',\n",
    "            fill_opacity=0.6\n",
    "        )\n",
    "    )\n",
    "\n",
    "# add incidents to map\n",
    "sanfran_map.add_child(incidents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "You can also add some pop-up text that would get displayed when you hover over a marker. Let's make each marker display the category of the crime when hovered over.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# instantiate a feature group for the incidents in the dataframe\n",
    "incidents = folium.map.FeatureGroup()\n",
    "\n",
    "# loop through the 100 crimes and add each to the incidents feature group\n",
    "for lat, lng, in zip(df_incidents.Y, df_incidents.X):\n",
    "    incidents.add_child(\n",
    "        folium.features.CircleMarker(\n",
    "            [lat, lng],\n",
    "            radius=5, # define how big you want the circle markers to be\n",
    "            color='yellow',\n",
    "            fill=True,\n",
    "            fill_color='blue',\n",
    "            fill_opacity=0.6\n",
    "        )\n",
    "    )\n",
    "\n",
    "# add pop-up text to each marker on the map\n",
    "latitudes = list(df_incidents.Y)\n",
    "longitudes = list(df_incidents.X)\n",
    "labels = list(df_incidents.Category)\n",
    "\n",
    "for lat, lng, label in zip(latitudes, longitudes, labels):\n",
    "    folium.Marker([lat, lng], popup=label).add_to(sanfran_map)    \n",
    "    \n",
    "# add incidents to map\n",
    "sanfran_map.add_child(incidents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Isn't this really cool? Now you are able to know what crime category occurred at each marker.\n",
    "\n",
    "If you find the map to be so congested will all these markers, there are two remedies to this problem. The simpler solution is to remove these location markers and just add the text to the circle markers themselves as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# create map and display it\n",
    "sanfran_map = folium.Map(location=[latitude, longitude], zoom_start=12)\n",
    "\n",
    "# loop through the 100 crimes and add each to the map\n",
    "for lat, lng, label in zip(df_incidents.Y, df_incidents.X, df_incidents.Category):\n",
    "    folium.features.CircleMarker(\n",
    "        [lat, lng],\n",
    "        radius=5, # define how big you want the circle markers to be\n",
    "        color='yellow',\n",
    "        fill=True,\n",
    "        popup=label,\n",
    "        fill_color='blue',\n",
    "        fill_opacity=0.6\n",
    "    ).add_to(sanfran_map)\n",
    "\n",
    "# show map\n",
    "sanfran_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The other proper remedy is to group the markers into different clusters. Each cluster is then represented by the number of crimes in each neighborhood. These clusters can be thought of as pockets of San Francisco which you can then analyze separately.\n",
    "\n",
    "To implement this, we start off by instantiating a *MarkerCluster* object and adding all the data points in the dataframe to this object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from folium import plugins\n",
    "\n",
    "# let's start again with a clean copy of the map of San Francisco\n",
    "sanfran_map = folium.Map(location = [latitude, longitude], zoom_start = 12)\n",
    "\n",
    "# instantiate a mark cluster object for the incidents in the dataframe\n",
    "incidents = plugins.MarkerCluster().add_to(sanfran_map)\n",
    "\n",
    "# loop through the dataframe and add each data point to the mark cluster\n",
    "for lat, lng, label, in zip(df_incidents.Y, df_incidents.X, df_incidents.Category):\n",
    "    folium.Marker(\n",
    "        location=[lat, lng],\n",
    "        icon=None,\n",
    "        popup=label,\n",
    "    ).add_to(incidents)\n",
    "\n",
    "# display map\n",
    "sanfran_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Notice how when you zoom out all the way, all markers are grouped into one cluster, *the global cluster*, of 100 markers or crimes, which is the total number of crimes in our dataframe. Once you start zooming in, the *global cluster* will start breaking up into smaller clusters. Zooming in all the way will result in individual markers.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Choropleth Maps <a id=\"8\"></a>\n",
    "\n",
    "A `Choropleth` map is a thematic map in which areas are shaded or patterned in proportion to the measurement of the statistical variable being displayed on the map, such as population density or per-capita income. The choropleth map provides an easy way to visualize how a measurement varies across a geographic area, or it shows the level of variability within a region. Below is a `Choropleth` map of the US depicting the population by square mile per state.\n",
    "\n",
    "<img src = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DV0101EN-SkillsNetwork/labs/Module%205/images/2000_census_population_density_map_by_state.png\" width = 600> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now, let's create our own `Choropleth` map of the world depicting immigration from various countries to Canada.\n",
    "\n",
    "Let's first download and import our primary Canadian immigration dataset using *pandas* `read_excel()` method. Normally, before we can do that, we would need to download a module which *pandas* requires reading in Excel files. This module was **openpyxl** (formerly **xlrd**). For your convenience, we have pre-installed this module, so you would not have to worry about that. Otherwise, you would need to run the following line of code to install the **openpyxl** module:\n",
    "\n",
    "```\n",
    "! pip3 install openpyxl\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Download the dataset and read it into a *pandas* dataframe:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_can = pd.read_excel(\n",
    "    'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DV0101EN-SkillsNetwork/Data%20Files/Canada.xlsx',\n",
    "    sheet_name='Canada by Citizenship',\n",
    "    skiprows=range(20),\n",
    "    skipfooter=2)\n",
    "\n",
    "print('Data downloaded and read into a dataframe!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's take a look at the first five items in our dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_can.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's find out how many entries there are in our dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# print the dimensions of the dataframe\n",
    "print(df_can.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Clean up data. We will make some modifications to the original dataset to make it easier to create our visualizations. Refer to *Introduction to Matplotlib and Line Plots* and *Area Plots, Histograms, and Bar Plots* notebooks for a detailed description of this preprocessing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# clean up the dataset to remove unnecessary columns (eg. REG) \n",
    "df_can.drop(['AREA','REG','DEV','Type','Coverage'], axis=1, inplace=True)\n",
    "\n",
    "# let's rename the columns so that they make sense\n",
    "df_can.rename(columns={'OdName':'Country', 'AreaName':'Continent','RegName':'Region'}, inplace=True)\n",
    "\n",
    "# for sake of consistency, let's also make all column labels of type string\n",
    "df_can.columns = list(map(str, df_can.columns))\n",
    "\n",
    "# add total column\n",
    "df_can['Total'] = df_can.sum(axis=1)\n",
    "\n",
    "# years that we will be using in this lesson - useful for plotting later on\n",
    "years = list(map(str, range(1980, 2014)))\n",
    "print ('data dimensions:', df_can.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's take a look at the first five items of our cleaned dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df_can.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In order to create a `Choropleth` map, we need a GeoJSON file that defines the areas/boundaries of the state, county, or country that we are interested in. In our case, since we are endeavoring to create a world map, we want a GeoJSON that defines the boundaries of all world countries. For your convenience, we will be providing you with this file, so let's go ahead and download it. Let's name it **world_countries.json**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# download countries geojson file\n",
    "! wget --quiet https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DV0101EN-SkillsNetwork/Data%20Files/world_countries.json\n",
    "    \n",
    "print('GeoJSON file downloaded!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now that we have the GeoJSON file, let's create a world map, centered around **\\[0, 0]** *latitude* and *longitude* values, with an initisal zoom level of 2.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "world_geo = r'world_countries.json' # geojson file\n",
    "\n",
    "# create a plain world map\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "And now to create a `Choropleth` map, we will use the *choropleth* method with the following main parameters:\n",
    "\n",
    "1.  `geo_data`, which is the GeoJSON file.\n",
    "2.  `data`, which is the dataframe containing the data.\n",
    "3.  `columns`, which represents the columns in the dataframe that will be used to create the `Choropleth` map.\n",
    "4.  `key_on`, which is the key or variable in the GeoJSON file that contains the name of the variable of interest. To determine that, you will need to open the GeoJSON file using any text editor and note the name of the key or variable that contains the name of the countries, since the countries are our variable of interest. In this case, **name** is the key in the GeoJSON file that contains the name of the countries. Note that this key is case_sensitive, so you need to pass exactly as it exists in the GeoJSON file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# generate choropleth map using the total immigration of each country to Canada from 1980 to 2013\n",
    "world_map.choropleth(\n",
    "    geo_data=world_geo,\n",
    "    data=df_can,\n",
    "    columns=['Country', 'Total'],\n",
    "    key_on='feature.properties.name',\n",
    "    fill_color='YlOrRd', \n",
    "    fill_opacity=0.7, \n",
    "    line_opacity=0.2,\n",
    "    legend_name='Immigration to Canada'\n",
    ")\n",
    "\n",
    "# display map\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As per our `Choropleth` map legend, the darker the color of a country and the closer the color to red, the higher the number of immigrants from that country. Accordingly, the highest immigration over the course of 33 years (from 1980 to 2013) was from China, India, and the Philippines, followed by Poland, Pakistan, and interestingly, the US.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Notice how the legend is displaying a negative boundary or threshold. Let's fix that by defining our own thresholds and starting with 0 instead of -6,918!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "world_geo = r'world_countries.json'\n",
    "\n",
    "# create a numpy array of length 6 and has linear spacing from the minimum total immigration to the maximum total immigration\n",
    "threshold_scale = np.linspace(df_can['Total'].min(),\n",
    "                              df_can['Total'].max(),\n",
    "                              6, dtype=int)\n",
    "threshold_scale = threshold_scale.tolist() # change the numpy array to a list\n",
    "threshold_scale[-1] = threshold_scale[-1] + 1 # make sure that the last value of the list is greater than the maximum immigration\n",
    "\n",
    "# let Folium determine the scale.\n",
    "world_map = folium.Map(location=[0, 0], zoom_start=2)\n",
    "world_map.choropleth(\n",
    "    geo_data=world_geo,\n",
    "    data=df_can,\n",
    "    columns=['Country', 'Total'],\n",
    "    key_on='feature.properties.name',\n",
    "    threshold_scale=threshold_scale,\n",
    "    fill_color='YlOrRd', \n",
    "    fill_opacity=0.7, \n",
    "    line_opacity=0.2,\n",
    "    legend_name='Immigration to Canada',\n",
    "    reset=True\n",
    ")\n",
    "world_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Much better now! Feel free to play around with the data and perhaps create `Choropleth` maps for individuals years, or perhaps decades, and see how they compare with the entire period from 1980 to 2013.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Thank you for completing this lab!\n",
    "\n",
    "## Author\n",
    "\n",
    "[Alex Aklson](https://www.linkedin.com/in/aklson/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkDV0101ENSkillsNetwork20297740-2021-01-01).\n",
    "\n",
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By    | Change Description                 |\n",
    "| ----------------- | ------- | ------------- | ---------------------------------- |\n",
    "| 2020-05-29        | 2.4     | Weiqing Wang  | Fixed typos and code smells.       |\n",
    "| 2020-01-20        | 2.3     | Lakshmi Holla | Updated TOC markdown               |\n",
    "| 2020-11-03        | 2.2     | Lakshmi Holla | Made changes in URL                |\n",
    "| 2020-10-06        | 2.1     | Lakshmi Holla | Removed Map Box Bright Style       |\n",
    "| 2020-08-27        | 2.0     | Lavanya       | Moved lab to course repo in GitLab |\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}