05-plotnine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis and Visualization in Python\n",
    "## Making Plots With plotnine\n",
    "Questions\n",
    "* How can I visualize data in Python?\n",
    "* What is ‘grammar of graphics’?\n",
    "\n",
    "Objectives\n",
    "* Create a `plotnine` object.\n",
    "* Set universal plot settings.\n",
    "* Change the aesthetics of a plot such as color.\n",
    "* Edit the axis labels.\n",
    "* Build complex plots using a step-by-step approach.\n",
    "* Create scatter plots, box plots, and time series plots.\n",
    "* Use the `facet_wrap` and `facet_grid` commands to create a collection of plots splitting the data by a \"factor\" variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load and clean the data\n",
    "surveys_complete = pd.read_csv('data/surveys.csv')\n",
    "surveys_complete = surveys_complete.###()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why `plotnine`? Why not `matplotlib`?\n",
    "Python has powerful built-in plotting capabilities such as `matplotlib`, but for this episode, we will be using the `plotnine` package, which facilitates the creation of highly-informative plots of structured data based on the R implementation of `ggplot2` and [The Grammar of Graphics](https://link.springer.com/book/10.1007%2F0-387-28695-0) by Leland Wilkinson. The `plotnine` package is built on top of Matplotlib and interacts well with Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ### as ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting with plotnine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`plotnine` graphics are built step by step by adding new elementsadding different elements on top of each other using the `+` operator. Putting the individual steps together in brackets `()` provides Python-compatible syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial empty plot\n",
    "(###(###=surveys_complete))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "* Define aesthetics (`aes`), by **selecting variables** used in the plot and `mapping` them to a presentation. The most important aes mappings are: `x`, `y`, `alpha`, `color`, `colour`, `fill`, `linetype`, `shape`, `size` and `stroke`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding x and y axis\n",
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=###(###='weight', ###='hindfoot_length')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To add a `geom_*` object to the plot use `+` operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length'))\n",
    "    ### p9.###()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* You can easily set up plot templates and conveniently explore different types of plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a template\n",
    "### = p9.ggplot(data=surveys_complete,\n",
    "                         mapping=p9.aes(x='weight', y='hindfoot_length'))\n",
    "\n",
    "# Draw the plot\n",
    "surveys_plot ### p9.geom_point()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* After creating your plot, you can save it to a file in your favourite format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scatterWL ### surveys_plot + p9.geom_point()\n",
    "\n",
    "scatterWL.###(\"scatterplot.png\", width=10, height=10, ###)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Exercises - bar chart\n",
    "Working on the `surveys_complete` data set, use the `plot_id` column to create a `bar` plot that counts the number of records for each plot. Hint: the count will be done implicitly by the `geom_bar()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(p9.ggplot(###=surveys_complete,\n",
    "           mapping=p9.###(x='###'))\n",
    "    + p9.###()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building your plots iteratively\n",
    "* Usually, `data`, `aes` and `geom-*` are the elementary elements of any graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(###=surveys_complete,\n",
    "           mapping=p9.###(x='weight', y='hindfoot_length'))\n",
    "    + p9.###()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Then, we start modifying this plot to extract more information from it. With transparency (alpha):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length'))\n",
    "    + p9.geom_point(###)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We can also get a different color for each species (by mapping the `species_id` column to the `color` aesthetic):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length', ###))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Changing labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length', color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + ###(\"Weight (g)\") ### ###(\"Hindfoot length (mm)\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Defining scale for colors, axes,... For example, a log-version of the x-axis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length', color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + p9.xlab(\"Weight (g)\") + p9.ylab(\"Hindfoot length (mm)\")\n",
    "    ###\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Changing the theme (`theme_*`) or some specific theming (`theme`) elements. We can set the background to white using the function `theme_bw()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight', y='hindfoot_length', color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + p9.xlab(\"Weight (g)\") + p9.ylab(\"Hindfoot length (mm)\")\n",
    "    + p9.scale_x_log10()\n",
    "    + p9.###\n",
    "    # + p9.###(###=p9.element_###(size=12))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Bar plot adaptations\n",
    "Adapt the bar plot of the previous exercise by mapping the `sex` variable to the aesthetic `fill`. Then, use the `scale_fill_manual()` function in order to specify both colors `blue` and `orange` (see [API reference](https://plotnine.readthedocs.io/en/stable/api.html#color-and-fill-scales) for other `scale*` functions and how to use them)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='plot_id',\n",
    "                          ###='sex'))\n",
    "    + p9.geom_bar()\n",
    "    + p9.###([\"blue\", \"orange\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting distributions\n",
    "* A boxplot can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='###',\n",
    "                          y='###'))\n",
    "    + p9.geom_###()\n",
    "    + p9.scale_###_log10()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Adding points behind the boxplot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='species_id',\n",
    "                          y='weight'))\n",
    "    + p9.###(###)\n",
    "    + p9.geom_boxplot(###)\n",
    "    + p9.scale_y_log10()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Distributions\n",
    "An alternative to the boxplot is the violin plot (sometimes known as a beanplot), where the shape of the density of points is drawn.\n",
    "\n",
    "* For the `x` axis, we want all `plot_id` values to be handled as category values. This can be specified in the `aes` object with `'factor(plot_id)'`.\n",
    "* Data points must have a color according to the `species_id` from which the sample was taken.\n",
    "* Generate a transparent **violin-plot** with a `grey` line on top of points. See [`geom_violin()`](https://plotnine.readthedocs.io/en/stable/generated/plotnine.geoms.geom_violin.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='###',\n",
    "                          y='weight',\n",
    "                          color='###'))\n",
    "    + p9.geom_jitter(alpha=0.3)\n",
    "    + p9.###(alpha=###, color=\"###\")\n",
    "    + p9.scale_y_log10()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting time series data\n",
    "* Let’s calculate number of records per year for each species.\n",
    "* Reset the index - `year` and `species_id` will become columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_counts = surveys_complete.###(['###', 'species_id'])['species_id'].###()\n",
    "# yearly_counts = yearly_counts.###(name='counts')\n",
    "yearly_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Timelapse data can be visualised as a line plot (i.e. `geom_line`) with years on `x` axis and counts on the `y` axis.\n",
    "* We need to tell `plotnine` to draw a line for each species by modifying the aesthetic function and map the `species_id` to the color:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=yearly_counts,\n",
    "           mapping=p9.aes(###='year',\n",
    "                          ###='counts',\n",
    "                          ###='species_id'))\n",
    "    + p9.###()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Faceting\n",
    "* `plotnine` has a special technique called *faceting* that allows to split one plot into multiple plots based on a factor variable included in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight',\n",
    "                          y='hindfoot_length',\n",
    "                          color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + p9.###(###)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We can apply the same concept on any of the available categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='weight',\n",
    "                          y='hindfoot_length',\n",
    "                          color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + p9.facet_wrap(###)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The `facet_grid` geometry allows you to explicitly specify how you want your plots to be arranged via formula notation `rows ~ columns`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only select years of interest\n",
    "survey_2000 = surveys_complete[surveys_complete[\"year\"].###([2000, 2001, 2002])]\n",
    "\n",
    "(p9.ggplot(data=survey_2000,\n",
    "           mapping=p9.aes(x='weight',\n",
    "                          y='hindfoot_length',\n",
    "                          color='species_id'))\n",
    "    + p9.geom_point(alpha=0.1)\n",
    "    + p9.###('sex ### year')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Faceting\n",
    "Visually compare how the average `weight` of females and males has changed through time by creating a separate plot for each `sex` and an individual color assigned to each `species_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yearly_weight = surveys_complete.groupby(['year',\n",
    "                                          'species_id',\n",
    "                                          '###'])['###'].###().reset_index()\n",
    "(p9.ggplot(data=yearly_weight,\n",
    "           mapping=p9.aes(x='###',\n",
    "                          y='###',\n",
    "                          ###='species_id'))\n",
    "    + p9.geom_line()\n",
    "    + p9.###(###)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further customization\n",
    "* The `theme` functionality provides a way to rotate the text of the x-axis labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_custom_theme = p9.###(axis_text_x = p9.element_###(###))\n",
    "\n",
    "(p9.ggplot(data=surveys_complete,\n",
    "           mapping=p9.aes(x='factor(year)'))\n",
    "    + p9.geom_bar()\n",
    "    + p9.xlab(\"Year\")\n",
    "    + p9.theme_bw()\n",
    "    + ###\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}