diff --git a/test_notebooks/lab11.ipynb b/test_notebooks/lab11.ipynb new file mode 100644 index 0000000..ab767f2 --- /dev/null +++ b/test_notebooks/lab11.ipynb @@ -0,0 +1,1571 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "e6749889", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "# Initialize Otter\n", + "import otter\n", + "grader = otter.Notebook(\"lab11.ipynb\")" + ] + }, + { + "cell_type": "markdown", + "id": "76a5277e", + "metadata": { + "nbgrader": { + "grade": false, + "grade_id": "title", + "locked": true, + "schema_version": 2, + "solution": false + } + }, + "source": [ + "# Lab 11: SQL\n", + "\n", + "In this lab, we are going to practice viewing, sorting, grouping, and merging tables with SQL. We will explore two datasets:\n", + "1. A \"minified\" version of the [Internet Movie Database](https://www.imdb.com/interfaces/) (IMDb). This SQLite database (~10MB) is a tiny sample of the much larger database (more than a few GBs). As a result, disclaimer that we may get wildly different results than if we use the whole database!\n", + "\n", + "1. The money donated during the 2016 election using the [Federal Election Commission (FEC)'s public records](https://www.fec.gov/data/). You will be connecting to a SQLite database containing the data. The data we will be working with in this lab is quite small (~16MB); however, it is a sample taken from a much larger database (more than a few GBs).\n", + "\n", + "\n", + "The on-time deadline is **Tuesday, April 11th, 11:59 PM PT**. Please read the syllabus for the grace period policy. No late submissions beyond the grace period will be accepted." + ] + }, + { + "cell_type": "markdown", + "id": "0f67ac48", + "metadata": {}, + "source": [ + "### Lab Walk-Through\n", + "In addition to the lab notebook, we have also released a prerecorded walk-through video of the lab. We encourage you to reference this video as you work through the lab. Run the cell below to display the video.\n", + "\n", + "**Note:** the walkthrough video is from Spring 2022, where this lab was labeled Lab 10." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38d360a9", + "metadata": {}, + "outputs": [], + "source": [ + "from IPython.display import YouTubeVideo\n", + "YouTubeVideo(\"uQ3E4pejmD8\", list = 'PLQCcNQgUcDfpdBnhS-lPq8LPas48tkMgp', listType = 'playlist')" + ] + }, + { + "cell_type": "markdown", + "id": "7bf5a879", + "metadata": {}, + "source": [ + "### Collaboration Policy\n", + "Data science is a collaborative activity. While you may talk with others about this assignment, we ask that you **write your solutions individually**. If you discuss the assignment with others, please **include their names** in the cell below." + ] + }, + { + "cell_type": "markdown", + "id": "917bd1e2", + "metadata": {}, + "source": [ + "**Collaborators:** *list names here*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e0230e0f", + "metadata": { + "nbgrader": { + "grade": false, + "grade_id": "setup", + "locked": true, + "schema_version": 2, + "solution": false + }, + "tags": [] + }, + "outputs": [], + "source": [ + "# Run this cell and the next one to set up your notebook.\n", + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "#import plotly.express as px\n", + "import sqlalchemy\n", + "from pathlib import Path\n", + "from zipfile import ZipFile\n", + "\n", + "with ZipFile('data.zip', 'r') as zipObj:\n", + " # Extract all the contents of zip file in current directory\n", + " zipObj.extractall()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "54e196cc", + "metadata": {}, + "outputs": [], + "source": [ + "# Run this cell to set up SQL.\n", + "%load_ext sql" + ] + }, + { + "cell_type": "markdown", + "id": "ee94bde7", + "metadata": {}, + "source": [ + "## SQL Query Syntax\n", + "\n", + "Throughout this lab, you will become familiar with the following syntax for the `SELECT` query:\n", + "\n", + "```\n", + "SELECT [DISTINCT] \n", + " {* | expr [[AS] c_alias] \n", + " {,expr [[AS] c_alias] ...}}\n", + "FROM tableref {, tableref}\n", + "[[INNER | LEFT ] JOIN table_name\n", + " ON qualification_list]\n", + "[WHERE search_condition]\n", + "[GROUP BY colname {,colname...}]\n", + "[HAVING search condition]\n", + "[ORDER BY column_list]\n", + "[LIMIT number]\n", + "[OFFSET number of rows];\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "fa7135d1", + "metadata": {}, + "source": [ + "

\n", + "
\n", + "
\n", + "\n", + "# Part 0 [Tutorial]: Writing SQL in Jupyter Notebooks" + ] + }, + { + "cell_type": "markdown", + "id": "6ff423e7", + "metadata": {}, + "source": [ + "### 1. `%%sql` cell magic\n", + "\n", + "In lecture, we used the `sql` extension to call **`%%sql` cell magic**, which enables us to connect to SQL databases and issue SQL commands within Jupyter Notebooks.\n", + "\n", + "Run the below cell to connect to a mini IMDb database." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6d77eca", + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "import shutil\n", + "import pathlib\n", + "\n", + "dbfile = 'imdbmini.db'\n", + "\n", + "# Do not modify following lines.\n", + "# File structure on gradescope is slightly different from datahub, \n", + "# hence we have to include this if-else statement to make sure we can get the correct database\n", + "if Path('../../../../../../../gradescope').is_dir():\n", + " tmpdb = Path('.') / dbfile \n", + "else:\n", + " # Staff note: File path if running from dist folder: \"../../../../../../../tmp\"\n", + " # remove this in student version manually\n", + " tmpdb = Path('../../../../../../../tmp') / dbfile \n", + "\n", + "if not tmpdb.is_file():\n", + " print('Copying DB file to /tmp for faster access.')\n", + " shutil.copy2(Path('.') / dbfile, tmpdb)\n", + " \n", + "sqlite_conn = 'sqlite:///' + str(tmpdb)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0ff41c1c", + "metadata": {}, + "outputs": [], + "source": [ + "%sql {sqlite_conn}" + ] + }, + { + "cell_type": "markdown", + "id": "aa806c7d", + "metadata": {}, + "source": [ + "
\n", + "\n", + "Above, prefixing our single-line command with `%sql` means that the entire line will be treated as a SQL command (this is called \"line magic\"). In this class we will most often write multi-line SQL, meaning we need \"cell magic\", where the first line has `%%sql` (note the double `%` operator).\n", + "\n", + "The database `imdbmini.db` includes several tables, one of which is `Title`. Running the below cell will return first 5 lines of that table. Note that `%%sql` is on its own line.\n", + "\n", + "We've also included syntax for single-line comments, which are surrounded by `--`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7aedc46e", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/*\n", + " * This is a\n", + " * multi-line comment.\n", + " */\n", + "-- This is a single-line/inline comment. --\n", + "SELECT *\n", + "FROM Name\n", + "LIMIT 5;" + ] + }, + { + "cell_type": "markdown", + "id": "82b73975", + "metadata": {}, + "source": [ + "
\n", + "\n", + "### 2. The Pandas command `pd.read_sql`" + ] + }, + { + "cell_type": "markdown", + "id": "45c54b19", + "metadata": {}, + "source": [ + "As of 2023, the `%sql` magic for Jupyter Notebooks is still in development (check out its [GitHub](https://github.com/catherinedevlin/ipython-sql). It is still missing many features that would justify real-world use with Python. In particular, its returned tables are *not* Pandas dataframes (for example, the query result from the above cell is missing an index).\n", + "\n", + "The rest of this section describes how data scientists use SQL and Python in practice, using the Pandas command `pd.read_sql` ([documentation](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html)). **You will see both `%sql` magic and `pd.read_sql` in this course**. \n" + ] + }, + { + "cell_type": "markdown", + "id": "1cbc85db", + "metadata": {}, + "source": [ + "The below cell connects to the same database using the SQLAlchemy Python library, which can connect to several different database management systems, including sqlite3, MySQL, PostgreSQL, and Oracle. The library also supports an advanced feature for generating queries called an [object relational mapper](https://docs.sqlalchemy.org/en/latest/orm/tutorial.html) or ORM, which we won't discuss in this course but is quite useful for application development." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0ed8209f", + "metadata": {}, + "outputs": [], + "source": [ + "# We imported sqlalchemy at the top of the notebook.\n", + "# Here's a reminder about how to import it.\n", + "import sqlalchemy\n", + "\n", + "# Create a SQL Alchemy connection to the database.\n", + "engine = sqlalchemy.create_engine(sqlite_conn)\n", + "connection = engine.connect()" + ] + }, + { + "cell_type": "markdown", + "id": "75a1d48b", + "metadata": {}, + "source": [ + "With the SQLAlchemy object `connection`, we can then call `pd.read_sql` which takes in a `query` **string**. Note the `\"\"\"` to define our multi-line string, which allows us to have a query span multiple lines. The resulting `df` DataFrame stores the results of the same SQL query from the previous section." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2e06ea75", + "metadata": {}, + "outputs": [], + "source": [ + "# Run this cell to see the demo.\n", + "query = \"\"\"\n", + "SELECT *\n", + "FROM Title\n", + "LIMIT 5;\n", + "\"\"\"\n", + "\n", + "df = pd.read_sql(query, engine)\n", + "df" + ] + }, + { + "cell_type": "markdown", + "id": "8339c7ad", + "metadata": {}, + "source": [ + "
\n", + "\n", + "**Long error messages**: Given that the SQL query is now in the string, the errors become more unintelligible. Consider the below (incorrect) query, which has a semicolon in the wrong place.\n", + "\n", + "**Note**: You can uncomment/comment out multiple cells at the same time by selecting the lines and press ctrl+/ or command+/ " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a14fbe80", + "metadata": {}, + "outputs": [], + "source": [ + "# Uncomment the below code and check out the error.\n", + "# query = \"\"\"\n", + "# SELECT *\n", + "# FROM Title;\n", + "# LIMIT 5\n", + "# \"\"\"\n", + "# pd.read_sql(query, engine)" + ] + }, + { + "cell_type": "markdown", + "id": "22fc96a6", + "metadata": {}, + "source": [ + "
\n", + "\n", + "Now that's an unruly error message! Can you see what's wrong in the cell above and correct the query?" + ] + }, + { + "cell_type": "markdown", + "id": "5c3207c1", + "metadata": { + "tags": [] + }, + "source": [ + "
\n", + "\n", + "### 3. A suggested workflow for writing SQL in Jupyter Notebooks\n", + "\n", + "Which approach is better, `%sql` magic or `pd.read_sql`?\n", + "\n", + "The SQL database generally contains much more data than what you would analyze in detail. As a Python-fluent data scientist, you will often query SQL databases to perform initial exploratory data analysis, a subset of which you load into Python for further processing.\n", + "\n", + "In practice, you would likely use a combination of the two approaches. First, you'd try out some SQL queries with `%sql` magic to get an interesting subset of data. Then, you'd copy over the query into a `pd.read_sql` command for visualization, modeling, and export with Pandas, sklearn, and other Python libraries.\n", + "\n", + "For SQL assignments in this course, to minimize unruly error messages while maximizing Python compatibility, we suggest the following \"sandboxed\" workflow:\n", + "1. Create a `%%sql` magic cell using the code below. \n", + "\n", + " ```\n", + " %% sql\n", + " -- This is a comment. Put your code here... --\n", + " ```\n", + " \n", + "
\n", + "\n", + "2. Work on the SQL query in the `%%sql` cell; e.g., `SELECT ... ;`\n", + "3. Then, once you're satisfied with your SQL query, copy it into the multi-string query in the answer cell (the one that contains the `pd.read_sql` call).\n", + "\n", + "You don't have to follow the above workflow to get full credit on assignments, but we suggest it to reduce debugging headaches. We've created the scratchwork `%%sql` cells for you in this assignment—but **if you add any cells, please delete them before submitting**. Newly added cells will cause errors when we run the autograder, and it will sometimes cause a failure to generate the PDF file.\n" + ] + }, + { + "cell_type": "markdown", + "id": "ffb2828a", + "metadata": {}, + "source": [ + "

\n", + "
\n", + "\n", + "# Part 1: The IMDb (mini) Dataset\n", + "\n", + "Let's explore a miniature version of the [IMDb Dataset](https://www.imdb.com/interfaces/). This is the same dataset that we will use for the upcoming homework.\n", + "\n", + "\n", + "Let's load in the database in two ways (using both Python and cell magic) so that we can flexibly explore the database." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a13de639", + "metadata": {}, + "outputs": [], + "source": [ + "engine = sqlalchemy.create_engine(sqlite_conn)\n", + "connection = engine.connect()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12de591c", + "metadata": {}, + "outputs": [], + "source": [ + "%sql {sqlite_conn}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2a37bfd5", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "SELECT * FROM sqlite_master WHERE type='table';" + ] + }, + { + "cell_type": "markdown", + "id": "2e303ea2", + "metadata": {}, + "source": [ + "From running the above cell, we see the database has 4 tables: `Name`, `Role`, `Rating`, and `Title`.\n", + "\n", + "
\n", + " [Click to Expand] See descriptions of each table's schema.\n", + " \n", + "**`Name`** – Contains the following information for names of people.\n", + " \n", + "- nconst (text) - alphanumeric unique identifier of the name/person\n", + "- primaryName (text)– name by which the person is most often credited\n", + "- birthYear (integer) – in YYYY format\n", + "- deathYear (integer) – in YYYY format\n", + " \n", + " \n", + "**`Role`** – Contains the principal cast/crew for titles.\n", + " \n", + "- tconst (text) - alphanumeric unique identifier of the title\n", + "- ordering (integer) – a number to uniquely identify rows for a given tconst\n", + "- nconst (text) - alphanumeric unique identifier of the name/person\n", + "- category (text) - the category of job that person was in\n", + "- characters (text) - the name of the character played if applicable, else '\\\\N'\n", + " \n", + "**`Rating`** – Contains the IMDb rating and votes information for titles.\n", + " \n", + "- tconst (text) - alphanumeric unique identifier of the title\n", + "- averageRating (real) – weighted average of all the individual user ratings\n", + "- numVotes (integer) - number of votes (i.e., ratings) the title has received\n", + " \n", + "**`Title`** - Contains the following information for titles.\n", + " \n", + "- tconst (text) - alphanumeric unique identifier of the title\n", + "- titleType (text) - the type/format of the title\n", + "- primaryTitle (text) - the more popular title / the title used by the filmmakers on promotional materials at the point of release\n", + "- isAdult (text) - 0: non-adult title; 1: adult title\n", + "- year (YYYY) – represents the release year of a title.\n", + "- runtimeMinutes (integer) – primary runtime of the title, in minutes\n", + " \n", + "
\n", + "\n", + "

\n", + "\n", + "From the above descriptions, we can conclude the following:\n", + "* `Name.nconst` and `Title.tconst` are primary keys of the `Name` and `Title` tables, respectively.\n", + "* that `Role.nconst` and `Role.tconst` are **foreign keys** that point to `Name.nconst` and `Title.tconst`, respectively." + ] + }, + { + "cell_type": "markdown", + "id": "30453011", + "metadata": {}, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 1\n", + "\n", + "What are the different kinds of `titleType`s included in the `Title` table? Write a query to find out all the unique `titleType`s of films using the `DISTINCT` keyword. (**You may not use `GROUP BY`.**)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b7ae5069", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e9291e88", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q1 = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "res_q1 = pd.read_sql(query_q1, engine)\n", + "res_q1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b13332b0", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q1\")" + ] + }, + { + "cell_type": "markdown", + "id": "9249c061", + "metadata": { + "tags": [] + }, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 2\n", + "\n", + "Before we proceed we want to get a better picture of the kinds of jobs that exist. To do this examine the `Role` table by computing the number of records with each job `category`. Present the results in descending order by the total counts.\n", + "\n", + "The top of your table should look like this (however, you should have more rows):\n", + "\n", + "| |category|total|\n", + "|-----|-----|-----|\n", + "|**0**|actor|21665|\n", + "|**1**|writer|13830|\n", + "|**2**|...|...|" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "749712fb", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d5477233", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q2 = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "res_q2 = pd.read_sql(query_q2, engine)\n", + "res_q2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "768543b0", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q2\")" + ] + }, + { + "cell_type": "markdown", + "id": "8f68ea18", + "metadata": {}, + "source": [ + "
\n", + "If we computed the results correctly we should see a nice horizontal bar chart of the counts per category below:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f5ac1453", + "metadata": {}, + "outputs": [], + "source": [ + "# Run this cell to make a bar plot.\n", + "plt.barh(res_q2[\"category\"], res_q2[\"total\"])" + ] + }, + { + "cell_type": "markdown", + "id": "d6b9cbeb", + "metadata": {}, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 3\n", + "\n", + "Now that we have a better sense of the basics of our data, we can ask some more interesting questions.\n", + "\n", + "The `Rating` table has the `numVotes` and the `averageRating` for each title. Which 10 films have the most ratings?\n", + "\n", + "Write a SQL query that outputs three fields: the `title`, `numVotes`, and `averageRating` for the 10 films that have the highest number of ratings. Sort the result in descending order by the number of votes.\n", + "\n", + "**Hint**: The `numVotes` in the `Rating` table is not an integer! Use `CAST(Rating.numVotes AS int) AS numVotes` to convert the attribute to an integer. Sometimes SQL will cast your data automatically when performing operation, but you should cast them explicitly for readability and reproducibility." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c11c66fa", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1db1a3a7", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q3 = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "\n", + "res_q3 = pd.read_sql(query_q3, engine)\n", + "res_q3" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "35e60520", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q3\")" + ] + }, + { + "cell_type": "markdown", + "id": "ed4a6c82", + "metadata": {}, + "source": [ + "

\n", + "
\n", + "\n", + "# Part 2: Election Donations in New York City\n", + "\n", + "Finally, let's analyze the Federal Election Commission (FEC)'s public records. We connect to the database in two ways (using both Python and cell magic) so that we can flexibly explore the database." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "436f458b", + "metadata": {}, + "outputs": [], + "source": [ + "# Create a SQL Alchemy connection to the database.\n", + "dbfile = 'fec_nyc.db'\n", + "\n", + "# Do not modify following lines.\n", + "# File structure on gradescope is slightly different from datahub, \n", + "# hence we have to include this if-else statement to make sure we can get the correct database\n", + "if Path('../../../../../../../gradescope').is_dir():\n", + " tmpdb = Path('.') / dbfile \n", + "else:\n", + " # Staff note: File path if running from dist folder: \"../../../../../../../tmp\"\n", + " # remove this in student version manually\n", + " tmpdb = Path('../../../../../../../tmp') / dbfile \n", + "\n", + "if not tmpdb.is_file():\n", + " print('Copying DB file to /tmp for faster access.')\n", + " shutil.copy2(Path('.') / dbfile, tmpdb)\n", + " \n", + "sqlite_conn_2 = 'sqlite:///' + str(tmpdb)\n", + "\n", + "engine = sqlalchemy.create_engine(sqlite_conn_2)\n", + "connection = engine.connect()\n", + "\n", + "%sql {sqlite_conn_2}" + ] + }, + { + "cell_type": "markdown", + "id": "e3c4e587", + "metadata": { + "tags": [] + }, + "source": [ + "### Table Descriptions\n", + "\n", + "Run the below cell to explore the **schemas** of all tables saved in the database.\n", + "\n", + "If you'd like, you can consult the below linked FEC pages for the descriptions of the tables themselves.\n", + "\n", + "* `cand` ([link](https://www.fec.gov/campaign-finance-data/candidate-summary-file-description/)): Candidates table. Contains names and party affiliation.\n", + "* `comm` ([link](https://www.fec.gov/campaign-finance-data/committee-summary-file-description/)): Committees table. Contains committee names and types.\n", + "* `indiv_sample_nyc` ([link](https://www.fec.gov/campaign-finance-data/contributions-individuals-file-description)): All individual contributions from New York City ." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b494384f", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/* just run this cell */\n", + "SELECT sql FROM sqlite_master WHERE type='table';" + ] + }, + { + "cell_type": "markdown", + "id": "ec2f083d", + "metadata": {}, + "source": [ + "

\n", + "\n", + "Let's look at the `indiv_sample_nyc` table. The below cell displays individual donations made by residents of the state of New York. We use `LIMIT 5` to avoid loading and displaying a huge table." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "efdf697c", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/* just run this cell */\n", + "SELECT *\n", + "FROM indiv_sample_nyc\n", + "LIMIT 5;" + ] + }, + { + "cell_type": "markdown", + "id": "09b3b41e", + "metadata": {}, + "source": [ + "You can write a SQL query to return the id and name of the first five candidates from the Democratic party, as below:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0dbf30b2", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/* just run this cell */\n", + "SELECT cand_id, cand_name\n", + "FROM cand\n", + "WHERE cand_pty_affiliation = 'DEM'\n", + "LIMIT 5;" + ] + }, + { + "cell_type": "markdown", + "id": "edbaa7f1", + "metadata": {}, + "source": [ + "

\n", + "
\n", + "\n", + "## [Tutorial] Matching Text with `LIKE`" + ] + }, + { + "cell_type": "markdown", + "id": "0c8b44b9", + "metadata": {}, + "source": [ + "First, let's look at 2016 election contributions made by Donald Trump, who was a New York (NY) resident during that year. The following SQL query returns the `cmte_id`, `transaction_amt`, and `name` for every contribution made by any donor with \"DONALD\" and \"TRUMP\" in their name in the `indiv_sample_nyc` table.\n", + "\n", + "**Notes:**\n", + "* We use the `WHERE ... LIKE '...'` to match fields with text patterns. The `%` wildcard represents at least zero characters. Compare this to what you know from regex!\n", + "* We use `pd.read_sql` syntax here because we will do some EDA on the result `res`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d2c7be96", + "metadata": {}, + "outputs": [], + "source": [ + "# Run this cell to see an example of LIKE.\n", + "example_query = \"\"\"\n", + "SELECT \n", + " cmte_id,\n", + " transaction_amt,\n", + " name\n", + "FROM indiv_sample_nyc\n", + "WHERE name LIKE '%TRUMP%' AND name LIKE '%DONALD%';\n", + "\"\"\"\n", + "\n", + "example_res = pd.read_sql(example_query, engine)\n", + "example_res" + ] + }, + { + "cell_type": "markdown", + "id": "5284e3bc", + "metadata": {}, + "source": [ + "If we look at the list above, it appears that some donations were not by Donald Trump himself, but instead by an entity called \"DONALD J TRUMP FOR PRESIDENT INC\". Fortunately, we see that our query only seems to have picked up one such anomalous name." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0af244d4", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Run this cell to see the value counts for each candidate.\n", + "example_res['name'].value_counts()" + ] + }, + { + "cell_type": "markdown", + "id": "37253eda", + "metadata": {}, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 4\n", + "\n", + "\n", + "\n", + "In the cell below, revise the above query so that the 15 anomalous donations made by \"DONALD J TRUMP FOR PRESIDENT INC\" do not appear. Your resulting table should have 142 rows. \n", + "\n", + "**Hints:**\n", + "* Consider using the above query as a starting point, or checking out the SQL query skeleton at the top of this lab. \n", + "* The `NOT` keyword may also be useful here.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "898ca2a3", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "74d0b912", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q4 = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "res_q4 = pd.read_sql(query_q4, engine)\n", + "res_q4" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "719ac320", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q4\")" + ] + }, + { + "cell_type": "markdown", + "id": "dfdab7a3", + "metadata": {}, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 5: `JOIN`ing Tables\n", + "\n", + "Let's explore the other two tables in our database: `cand` and `comm`.\n", + "\n", + "The `cand` table contains summary financial information about each candidate registered with the FEC or appearing on an official state ballot for House, Senate or President." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5ffa2157", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/* just run this cell */\n", + "SELECT *\n", + "FROM cand\n", + "LIMIT 5;" + ] + }, + { + "cell_type": "markdown", + "id": "7f1f6225", + "metadata": {}, + "source": [ + "The `comm` table contains summary financial information about each committee registered with the FEC. Committees are organizations that spend money for political action or parties, or spend money for or against political candidates." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9b33a0e0", + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "/* just run this cell */\n", + "SELECT *\n", + "FROM comm\n", + "LIMIT 5;" + ] + }, + { + "cell_type": "markdown", + "id": "bccc37b1", + "metadata": { + "tags": [] + }, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "### Question 5a\n", + "\n", + "Notice that both the `cand` and `comm` tables have a `cand_id` column. Let's try joining these two tables on this column to print out committee information for candidates.\n", + "\n", + "List the first 5 candidate names (`cand_name`) in reverse lexicographic order by `cand_name`, along with their corresponding committee names. **Only select rows that have a matching `cand_id` in both tables.**\n", + "\n", + "Your output should look similar to the following:\n", + "\n", + "| |cand_name|cmte_nm|\n", + "|----|----|----|\n", + "|**0**|ZUTLER, DANIEL PAUL MR|CITIZENS TO ELECT DANIEL P ZUTLER FOR PRESIDENT|\n", + "|**1**|ZUMWALT, JAMES|ZUMWALT FOR CONGRESS|\n", + "|**...**|...|...|\n", + "\n", + "Consider starting from the following query skeleton, which uses the `AS` keyword to rename the `cand` and `comm` tables to `c1` and `c2`, respectively.\n", + "Which join is most appropriate?\n", + "\n", + " SELECT ...\n", + " FROM cand AS c1\n", + " [INNER | {LEFT |RIGHT | FULL } {OUTER}] JOIN comm AS c2\n", + " ON ...\n", + " ...\n", + " ...;\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a082af2b", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3ac50768", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q5a = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "res_q5a = pd.read_sql(query_q5a, engine)\n", + "res_q5a" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2737fc1", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q5a\")" + ] + }, + { + "cell_type": "markdown", + "id": "5f34df38", + "metadata": { + "tags": [] + }, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "### Question 5b\n", + "\n", + "Suppose we modify the query from the previous part to include *all* candidates, **including those that don't have a committee.**\n", + "\n", + "\n", + "List the first 5 candidate names (`cand_name`) in reverse lexicographic order by `cand_name`, along with their corresponding committee names. If the candidate has no committee in the `comm` table, then `cmte_nm` should be NULL (or None in the Python representation).\n", + "\n", + "Your output should look similar to the following:\n", + "\n", + "| |cand_name|cmte_nm|\n", + "|----|----|----|\n", + "|**0**|ZUTLER, DANIEL PAUL MR|CITIZENS TO ELECT DANIEL P ZUTLER FOR PRESIDENT|\n", + "|**...**|...|...|\n", + "|**4**|ZORNOW, TODD MR|None|\n", + "\n", + "**Hint**: Start from the same query skeleton as the previous part. \n", + "Which join is most appropriate?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5f7c7ca8", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9d541699", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q5b = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "res_q5b = pd.read_sql(query_q5b, engine)\n", + "res_q5b" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "50b70f02", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q5b\")" + ] + }, + { + "cell_type": "markdown", + "id": "6ad0ea36", + "metadata": {}, + "source": [ + "

\n", + "\n", + "---\n", + "\n", + "## Question 6: Subqueries and Grouping\n", + "\n", + "If we return to our results from Question 4, we see that many of the contributions were to the same committee:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ba1ec7e3", + "metadata": {}, + "outputs": [], + "source": [ + "# Your SQL query result from Question 4\n", + "# Reprinted for your convenience\n", + "res_q4['cmte_id'].value_counts()" + ] + }, + { + "cell_type": "markdown", + "id": "563db106", + "metadata": {}, + "source": [ + "

\n", + "\n", + "For this question, create a new SQL query that returns the total amount that Donald Trump contributed to each committee.\n", + "\n", + "Your table should have four columns: `cmte_id`, `total_amount` (total amount contributed to that committee), `num_donations` (total number of donations), and `cmte_nm` (name of the committee). Your table should be sorted in **decreasing order** of `total_amount`.\n", + "\n", + "Your output should look similar to the following:\n", + "\n", + "| |cmte_id|total_amount|num_donations|cmte_nm|\n", + "|----|----|----|----|----|\n", + "|**0**|C00580100|18633157|131|DONALD J. TRUMP FOR PRESIDENT, INC.|\n", + "|**1**|C00055582|10000|1|NY REPUBLICAN FEDERAL CAMPAIGN COMMITTEE\n", + "|**...**|...|...|...|\n", + "\n", + "**This is a hard question!** Don't be afraid to reference the lecture slides, or the overall SQL query skeleton at the top of this lab.\n", + "\n", + "**Hint**:\n", + "\n", + "* Note that committee names are not available in `indiv_sample_nyc`, so you will have to obtain information somehow from the `comm` table (perhaps a `JOIN` would be useful).\n", + "* Remember that you can compute summary statistics after grouping by using aggregates like `COUNT(*)`, `SUM()` as output fields.\n", + "* A **subquery** may be useful to break your question down into subparts. Consider the following query skeleton, which uses the `WITH` operator to store a subquery's results in a temporary table named `donations`.\n", + "\n", + " WITH donations AS (\n", + " SELECT ...\n", + " FROM ...\n", + " ... JOIN ...\n", + " ON ...\n", + " WHERE ...\n", + " )\n", + " SELECT ...\n", + " FROM donations\n", + " GROUP BY ...\n", + " ORDER BY ...;" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fb46ae7f", + "metadata": {}, + "outputs": [], + "source": [ + "# %%sql\n", + "# /*\n", + "# * Code in this scratchwork cell is __not graded.__\n", + "# * Copy over any SQL queries you write here into the below Python cell.\n", + "# * Do __not__ insert any new cells in between the SQL/Python cells!\n", + "# * Doing so may break the autograder.\n", + "# */\n", + "# -- Write below this comment. --" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fafd4bd0", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "query_q6 = \"\"\"\n", + "... # replace this with\n", + "...; # your multi-line solution\n", + "\"\"\"\n", + "\n", + "\n", + "res_q6 = pd.read_sql(query_q6, engine)\n", + "res_q6" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6dbbb2ec", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "grader.check(\"q6\")" + ] + }, + { + "cell_type": "markdown", + "id": "0027feb5", + "metadata": {}, + "source": [ + "

\n", + "
\n", + "
\n", + "\n", + "## Congratulations! You finished Lab 11!" + ] + }, + { + "cell_type": "markdown", + "id": "e5c2275a", + "metadata": { + "deletable": false, + "editable": false + }, + "source": [ + "## Submission\n", + "\n", + "Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. **Please save before exporting!**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4ae38458", + "metadata": { + "deletable": false, + "editable": false + }, + "outputs": [], + "source": [ + "# Save your notebook first, then run this cell to export your submission.\n", + "grader.export(pdf=False, run_tests=True)" + ] + }, + { + "cell_type": "markdown", + "id": "dbd621ce", + "metadata": {}, + "source": [ + " " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.0" + }, + "otter": { + "OK_FORMAT": true, + "tests": { + "q1": { + "name": "q1", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q1.shape == (9, 1)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> \"group by\" not in query_q1.lower()\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(list(res_q1.iloc[:,0])) == ['movie', 'short', 'tvEpisode', 'tvMiniSeries', 'tvMovie', 'tvSeries', 'tvSpecial', 'video', 'videoGame']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q2": { + "name": "q2", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q2.shape == (12, 2)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> list(res_q2['total']) == [21665, 13830, 12175, 11028, 6995, 4123, 2747, 1558, 623, 410, 66, 6]\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> list(res_q2['category']) == ['actor',\n... 'writer',\n... 'actress',\n... 'producer',\n... 'director',\n... 'composer',\n... 'cinematographer',\n... 'editor',\n... 'self',\n... 'production_designer',\n... 'archive_footage',\n... 'archive_sound']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q3": { + "name": "q3", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q3.shape == (10, 3)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> set(res_q3.columns) == set(['title', 'numVotes', 'averageRating'])\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> np.sum(res_q3['numVotes'].astype(int)) == 19870486\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> list(res_q3['title']) == ['The Shawshank Redemption',\n... 'The Dark Knight',\n... 'Inception',\n... 'Fight Club',\n... 'Pulp Fiction',\n... 'Forrest Gump',\n... 'Game of Thrones',\n... 'The Matrix',\n... 'The Lord of the Rings: The Fellowship of the Ring',\n... 'The Lord of the Rings: The Return of the King']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q4": { + "name": "q4", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q4.shape == (142, 3)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(res_q4['transaction_amt'].unique()) \\\n... == [224, 514, 1000, 1029, 1216, 1713, 2000, 2426, 2460, \\\n... 2529, 2531, 2536, 2544, 2574, 2587, 2600, 2620, 2679, \\\n... 2692, 2693, 2700, 2991, 3063, 3347, 3360, 3371, 3499, \\\n... 3548, 3706, 3954, 4049, 4127, 4164, 4591, 4931, 4958, \\\n... 5000, 5297, 5574, 6197, 6250, 6990, 7401, 7519, 7990, \\\n... 8735, 8896, 9000, 9013, 9106, 9651, 9752, 10000, 10418, \\\n... 11023, 15000, 23775, 2000000, 10000000]\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q5a": { + "name": "q5a", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q5a.shape == (5, 2)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> all(res_q5a == res_q5a.sort_values('cand_name', ascending=False))\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(res_q5a['cmte_nm'].unique()) == [\n... 'CITIZENS TO ELECT DANIEL P ZUTLER FOR PRESIDENT',\n... 'CONSTITUTIONAL COMMITTEE',\n... 'JOE ZUCCOLO FOR CONGRESS',\n... 'ZUKOWSKI FOR CONGRESS',\n... 'ZUMWALT FOR CONGRESS']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q5b": { + "name": "q5b", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q5b.shape == (5, 2)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> all(res_q5b == res_q5b.sort_values('cand_name', ascending=False))\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(map(str, res_q5b['cmte_nm'].unique())) == [\n... 'CITIZENS TO ELECT DANIEL P ZUTLER FOR PRESIDENT',\n... 'JOE ZUCCOLO FOR CONGRESS',\n... 'None',\n... 'ZUKOWSKI FOR CONGRESS',\n... 'ZUMWALT FOR CONGRESS']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + }, + "q6": { + "name": "q6", + "points": null, + "suites": [ + { + "cases": [ + { + "code": ">>> res_q6.shape == (10, 4)\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(res_q6['total_amount']) == [1000, 2000, 2600, 5000, 5000, 5200, 5400, 9000, 10000, 18633157]\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(res_q6['num_donations']) == [1, 1, 1, 1, 1, 1, 1, 2, 2, 131]\nTrue", + "hidden": false, + "locked": false + }, + { + "code": ">>> sorted(res_q6['cmte_nm']) == [\n... 'DONALD J. TRUMP FOR PRESIDENT, INC.',\n... 'DONOVAN FOR CONGRESS',\n... 'FRIENDS OF DAVE BRAT INC.',\n... 'GRASSLEY COMMITTEE INC',\n... 'HELLER FOR SENATE',\n... 'NEW HAMPSHIRE REPUBLICAN STATE COMMITTEE',\n... 'NY REPUBLICAN FEDERAL CAMPAIGN COMMITTEE',\n... 'REPUBLICAN PARTY OF IOWA',\n... 'SOUTH CAROLINA REPUBLICAN PARTY',\n... 'TEXANS FOR SENATOR JOHN CORNYN INC']\nTrue", + "hidden": false, + "locked": false + } + ], + "scored": true, + "setup": "", + "teardown": "", + "type": "doctest" + } + ] + } + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}