jupyter-labs-webscraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Space X  Falcon 9 First Stage Landing Prediction**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web scraping Falcon 9 and Falcon Heavy Launches Records from Wikipedia\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **40** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you will be performing web scraping to collect Falcon 9 historical launch records from a Wikipedia page titled `List of Falcon 9 and Falcon Heavy launches`\n",
    "\n",
    "https://en.wikipedia.org/wiki/List_of_Falcon_9_and_Falcon_Heavy_launches\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module_1_L2/images/Falcon9_rocket_family.svg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Falcon 9 first stage will land successfully\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/landing_1.gif)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several examples of an unsuccessful landing are shown here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DS0701EN-SkillsNetwork/api/Images/crash.gif)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More specifically, the launch records are stored in a HTML table shown below:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DS0321EN-SkillsNetwork/labs/module_1_L2/images/falcon9-launches-wiki.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  ## Objectives\n",
    "Web scrap Falcon 9 launch records with `BeautifulSoup`: \n",
    "- Extract a Falcon 9 launch records HTML table from Wikipedia\n",
    "- Parse the table and convert it into a Pandas data frame\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's import required packages for this lab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install beautifulsoup4\n",
    "!pip3 install requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "import unicodedata\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we will provide some helper functions for you to process web scraped HTML table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def date_time(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the data and time from the HTML  table cell\n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    return [data_time.strip() for data_time in list(table_cells.strings)][0:2]\n",
    "\n",
    "def booster_version(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the booster version from the HTML  table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    out=''.join([booster_version for i,booster_version in enumerate( table_cells.strings) if i%2==0][0:-1])\n",
    "    return out\n",
    "\n",
    "def landing_status(table_cells):\n",
    "    \"\"\"\n",
    "    This function returns the landing status from the HTML table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    out=[i for i in table_cells.strings][0]\n",
    "    return out\n",
    "\n",
    "\n",
    "def get_mass(table_cells):\n",
    "    mass=unicodedata.normalize(\"NFKD\", table_cells.text).strip()\n",
    "    if mass:\n",
    "        mass.find(\"kg\")\n",
    "        new_mass=mass[0:mass.find(\"kg\")+2]\n",
    "    else:\n",
    "        new_mass=0\n",
    "    return new_mass\n",
    "\n",
    "\n",
    "def extract_column_from_header(row):\n",
    "    \"\"\"\n",
    "    This function returns the landing status from the HTML table cell \n",
    "    Input: the  element of a table data cell extracts extra row\n",
    "    \"\"\"\n",
    "    if (row.br):\n",
    "        row.br.extract()\n",
    "    if row.a:\n",
    "        row.a.extract()\n",
    "    if row.sup:\n",
    "        row.sup.extract()\n",
    "        \n",
    "    colunm_name = ' '.join(row.contents)\n",
    "    \n",
    "    # Filter the digit and empty names\n",
    "    if not(colunm_name.strip().isdigit()):\n",
    "        colunm_name = colunm_name.strip()\n",
    "        return colunm_name    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep the lab tasks consistent, you will be asked to scrape the data from a snapshot of the  `List of Falcon 9 and Falcon Heavy launches` Wikipage updated on\n",
    "`9th June 2021`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "static_url = \"https://en.wikipedia.org/w/index.php?title=List_of_Falcon_9_and_Falcon_Heavy_launches&oldid=1027686922\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, request the HTML page from the above URL and get a `response` object\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 1: Request the Falcon9 Launch Wiki page from its URL\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's perform an HTTP GET method to request the Falcon9 Launch HTML page, as an HTTP response.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use requests.get() method with the provided static_url\n",
    "# assign the response to a object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `BeautifulSoup` object from the HTML `response`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use BeautifulSoup() to create a BeautifulSoup object from a response text content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the page title to verify if the `BeautifulSoup` object was created properly \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use soup.title attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TASK 2: Extract all column/variable names from the HTML table header\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we want to collect all relevant column names from the HTML table header\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to find all tables on the wiki page first. If you need to refresh your memory about `BeautifulSoup`, please check the external reference link towards the end of this lab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the find_all function in the BeautifulSoup object, with element type `table`\n",
    "# Assign the result to a list called `html_tables`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from the third table is our target table contains the actual launch records.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's print the third table and check its content\n",
    "first_launch_table = html_tables[2]\n",
    "print(first_launch_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should able to see the columns names embedded in the table header elements `<th>` as follows:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "<tr>\n",
    "<th scope=\"col\">Flight No.\n",
    "</th>\n",
    "<th scope=\"col\">Date and<br/>time (<a href=\"/wiki/Coordinated_Universal_Time\" title=\"Coordinated Universal Time\">UTC</a>)\n",
    "</th>\n",
    "<th scope=\"col\"><a href=\"/wiki/List_of_Falcon_9_first-stage_boosters\" title=\"List of Falcon 9 first-stage boosters\">Version,<br/>Booster</a> <sup class=\"reference\" id=\"cite_ref-booster_11-0\"><a href=\"#cite_note-booster-11\">[b]</a></sup>\n",
    "</th>\n",
    "<th scope=\"col\">Launch site\n",
    "</th>\n",
    "<th scope=\"col\">Payload<sup class=\"reference\" id=\"cite_ref-Dragon_12-0\"><a href=\"#cite_note-Dragon-12\">[c]</a></sup>\n",
    "</th>\n",
    "<th scope=\"col\">Payload mass\n",
    "</th>\n",
    "<th scope=\"col\">Orbit\n",
    "</th>\n",
    "<th scope=\"col\">Customer\n",
    "</th>\n",
    "<th scope=\"col\">Launch<br/>outcome\n",
    "</th>\n",
    "<th scope=\"col\"><a href=\"/wiki/Falcon_9_first-stage_landing_tests\" title=\"Falcon 9 first-stage landing tests\">Booster<br/>landing</a>\n",
    "</th></tr>\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we just need to iterate through the `<th>` elements and apply the provided `extract_column_from_header()` to extract column name one by one\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names = []\n",
    "\n",
    "# Apply find_all() function with `th` element on first_launch_table\n",
    "# Iterate each th element and apply the provided extract_column_from_header() to get a column name\n",
    "# Append the Non-empty column name (`if name is not None and len(name) > 0`) into a list called column_names\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the extracted column names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TASK 3: Create a data frame by parsing the launch HTML tables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create an empty dictionary with keys from the extracted column names in the previous task. Later, this dictionary will be converted into a Pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "launch_dict= dict.fromkeys(column_names)\n",
    "\n",
    "# Remove an irrelvant column\n",
    "del launch_dict['Date and time ( )']\n",
    "\n",
    "# Let's initial the launch_dict with each value to be an empty list\n",
    "launch_dict['Flight No.'] = []\n",
    "launch_dict['Launch site'] = []\n",
    "launch_dict['Payload'] = []\n",
    "launch_dict['Payload mass'] = []\n",
    "launch_dict['Orbit'] = []\n",
    "launch_dict['Customer'] = []\n",
    "launch_dict['Launch outcome'] = []\n",
    "# Added some new columns\n",
    "launch_dict['Version Booster']=[]\n",
    "launch_dict['Booster landing']=[]\n",
    "launch_dict['Date']=[]\n",
    "launch_dict['Time']=[]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we just need to fill up the `launch_dict` with launch records extracted from table rows.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, HTML tables in Wiki pages are likely to contain unexpected annotations and other types of noises, such as reference links `B0004.1[8]`, missing values `N/A [e]`, inconsistent formatting, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the parsing process, we have provided an incomplete code snippet below to help you to fill up the `launch_dict`. Please complete the following code snippet with TODOs or you can choose to write your own logic to parse all launch tables:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_row = 0\n",
    "#Extract each table \n",
    "for table_number,table in enumerate(soup.find_all('table',\"wikitable plainrowheaders collapsible\")):\n",
    "   # get table row \n",
    "    for rows in table.find_all(\"tr\"):\n",
    "        #check to see if first table heading is as number corresponding to launch a number \n",
    "        if rows.th:\n",
    "            if rows.th.string:\n",
    "                flight_number=rows.th.string.strip()\n",
    "                flag=flight_number.isdigit()\n",
    "        else:\n",
    "            flag=False\n",
    "        #get table element \n",
    "        row=rows.find_all('td')\n",
    "        #if it is number save cells in a dictonary \n",
    "        if flag:\n",
    "            extracted_row += 1\n",
    "            # Flight Number value\n",
    "            # TODO: Append the flight_number into launch_dict with key `Flight No.`\n",
    "            #print(flight_number)\n",
    "            datatimelist=date_time(row[0])\n",
    "            \n",
    "            # Date value\n",
    "            # TODO: Append the date into launch_dict with key `Date`\n",
    "            date = datatimelist[0].strip(',')\n",
    "            #print(date)\n",
    "            \n",
    "            # Time value\n",
    "            # TODO: Append the time into launch_dict with key `Time`\n",
    "            time = datatimelist[1]\n",
    "            #print(time)\n",
    "              \n",
    "            # Booster version\n",
    "            # TODO: Append the bv into launch_dict with key `Version Booster`\n",
    "            bv=booster_version(row[1])\n",
    "            if not(bv):\n",
    "                bv=row[1].a.string\n",
    "            print(bv)\n",
    "            \n",
    "            # Launch Site\n",
    "            # TODO: Append the bv into launch_dict with key `Launch Site`\n",
    "            launch_site = row[2].a.string\n",
    "            #print(launch_site)\n",
    "            \n",
    "            # Payload\n",
    "            # TODO: Append the payload into launch_dict with key `Payload`\n",
    "            payload = row[3].a.string\n",
    "            #print(payload)\n",
    "            \n",
    "            # Payload Mass\n",
    "            # TODO: Append the payload_mass into launch_dict with key `Payload mass`\n",
    "            payload_mass = get_mass(row[4])\n",
    "            #print(payload)\n",
    "            \n",
    "            # Orbit\n",
    "            # TODO: Append the orbit into launch_dict with key `Orbit`\n",
    "            orbit = row[5].a.string\n",
    "            #print(orbit)\n",
    "            \n",
    "            # Customer\n",
    "            # TODO: Append the customer into launch_dict with key `Customer`\n",
    "            customer = row[6].a.string\n",
    "            #print(customer)\n",
    "            \n",
    "            # Launch outcome\n",
    "            # TODO: Append the launch_outcome into launch_dict with key `Launch outcome`\n",
    "            launch_outcome = list(row[7].strings)[0]\n",
    "            #print(launch_outcome)\n",
    "            \n",
    "            # Booster landing\n",
    "            # TODO: Append the launch_outcome into launch_dict with key `Booster landing`\n",
    "            booster_landing = landing_status(row[8])\n",
    "            #print(booster_landing)\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you have fill in the parsed launch record values into `launch_dict`, you can create a dataframe from it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df= pd.DataFrame({ key:pd.Series(value) for key, value in launch_dict.items() })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now export it to a <b>CSV</b> for the next section, but to make the answers consistent and in case you have difficulties finishing this lab. \n",
    "\n",
    "Following labs will be using a provided dataset to make each lab independent. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>df.to_csv('spacex_web_scraped.csv', index=False)</code>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.linkedin.com/in/yan-luo-96288783/\">Yan Luo</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.linkedin.com/in/nayefaboutayoun/\">Nayef Abou Tayoun</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description      |\n",
    "| ----------------- | ------- | ---------- | ----------------------- |\n",
    "| 2021-06-09        | 1.0     | Yan Luo    | Tasks updates           |\n",
    "| 2020-11-10        | 1.0     | Nayef      | Created the initial version |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2021 IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}