14-ScientificComputing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scientific Computing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "<b>Scientific Computing</b> is the application of computer programming to scientific applications: data analysis, simulation & modelling, plotting, etc. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scientific Python: Scipy Stack\n",
    "\n",
    "Scipy = Scientific Python\n",
    "\n",
    "- `scipy`\n",
    "- `numpy`\n",
    "- `pandas`\n",
    "- Data Analysis in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "<b><code>Scipy</code></b> is an <i>ecosystem</i>, including a collection of open-source packages for scientific computing in Python.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "A 'family' of packages that all work well together to do scientific computing.\n",
    "\n",
    "Not made by the same people who manage the standard library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Packages must be installed before you can use them.** Install once. Import as many times as you want.\n",
    "\n",
    "The packages we're using today have 1) already been installed for you on datahub and 2) are part of the Anaconda distribution.\n",
    "\n",
    "However, for your own work, you may want to install additional packages. To do so: `pip install --user package` (where you replace `package` with the package to be installed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Homogenous Data\n",
    "\n",
    "- for example: store data of the same type (i.e. all numerics)\n",
    "- recordings of values from experimental participants\n",
    "- heights or quantitative information from survey data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists are a start, and lists of lists are possible.\n",
    "\n",
    "But, they can get nightmareish.\n",
    "\n",
    "So we use arrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `numpy`\n",
    "\n",
    "**`numpy`** - stands for numerical python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**arrays** - work with arrays (matrices)\n",
    "\n",
    "Allow you to efficiently operate on arrays (linear algebra, matrix operations, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create some arrays of data\n",
    "arr0 = np.array([1, 2, 3])\n",
    "arr1 = np.array([[1, 2], [3, 4]])\n",
    "arr2 = np.array([[5, 6], [7, 8]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "arr1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# lists of lists don't store dimensionality well \n",
    "[[1, 2, 2, 2, 2, 2], [3, 4, 2, 2, 2, 2], [3, 4, 2, 2, 2, 2], [3, 4, 2, 2, 2, 2], [3, 4, 2, 2, 2, 2]] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Arrays: attributes, methods, & indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Check out an array of data\n",
    "np.array([[1, 2, 2, 2, 2], [3, 4, 2, 2, 2], [3, 2, 2, 2, 2], [3, 2, 2, 2, 2], [3, 2, 2, 2, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`numpy` arrays are an object type...so they have associated attributes (below) and methods (we'll get to these in a second)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Check the shape of the array\n",
    "print(arr1)\n",
    "arr1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Index into a numpy array\n",
    "# instead of using start:stop:step, it uses \"coordinates\", starting with 0\n",
    "# use : to represent an entire row/column\n",
    "arr1[1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Working with N-dimensional (multidimensional) arrays is easy within `numpy`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Notes on Arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# arrays need to have the same length in each list\n",
    "np.array([[1, 2, 3, 4], [2, 3, 4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# arrays are meant to store homogeneous data\n",
    "# and it will convert data types if needed\n",
    "np.array([[1, 2, 'cogs18'], [2, 3, 4]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Working with Arrays\n",
    "\n",
    "(Things you can't do with lists)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Add arrays together\n",
    "arr1 + arr2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Matrix mutliplication\n",
    "arr1 * arr2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### A brief reminder/aside: `zip()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`zip()` takes two iterables (things you can loop over) and loop over them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for a, b in zip([1,2,3], ['a','b','c']):\n",
    "    print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Class Question #1\n",
    "\n",
    "Given the following code, what will it print out?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data = np.array([[1, 2, 3, 4],\n",
    "                 [5, 6, 7, 8]])\n",
    " \n",
    "output = []\n",
    "for d1, d2 in zip(data[0, :], data[1, :]):\n",
    "    output.append(d1 + d2)\n",
    "\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- A) [1, 2, 3, 4]\n",
    "- B) [1, 2, 3, 4, 5, 6, 7, 8]\n",
    "- C) [6, 8, 10, 12]\n",
    "- D) [10, 26]\n",
    "- E) [36]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that if you find yourself looping over arrays...there is probably a better way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# sum method\n",
    "# by default sums all values in array  \n",
    "data.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# sum method\n",
    "# has an axis parameter\n",
    "# axis=0 sums across columns\n",
    "print(data)\n",
    "data.sum(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# typecasting to a different variable type\n",
    "out_list = data.sum(axis=0).tolist()\n",
    "print(out_list)\n",
    "type(out_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What if you wanted to find the max value in an array...there's a method for that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# find max value in array\n",
    "max_val = data.max()\n",
    "max_val  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But what if you wanted to know not what the max value was...but where in your original array that value was found. \n",
    "\n",
    "There are also *functions* in `numpy` that operate on arrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# see documentation for np.where()\n",
    "np.where?\n",
    "#data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# find position in array with max value\n",
    "out = np.where(data == max_val)\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# check to be sure\n",
    "data[1,3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Heterogenous Data\n",
    "\n",
    "- have continuous (numeric) and categorical (discrete) data\n",
    "- use when different data types need to be stored\n",
    "- uses a DataFrame object (think: spreadsheet)\n",
    "- allows for column and row labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Pandas is Python library for managing heterogenous data.\n",
    "\n",
    "At it's core, Pandas is built around the **DataFrame** object, which is:\n",
    "- a data structure for labeled rows and columns of data\n",
    "- associated methods and utilities for working with data.\n",
    "- each column contains a `pandas` **Series**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create some example heterogenous data\n",
    "d1 = {'Subj_ID': '001', 'score': 16, 'group' : 2, 'condition': 'cognition'}\n",
    "d2 = {'Subj_ID': '002', 'score': 22, 'group' : 1, 'condition': 'perception'}\n",
    "d3 = {'Subj_ID': '003', 'score': 18, 'group' : 1, 'condition': 'perception'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create a dataframe  \n",
    "df = pd.DataFrame([d1, d2, d3], [0, 1, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Check out the dataframe\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You can index in pandas\n",
    "# columns store information in series\n",
    "df['condition']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You can index in pandas\n",
    "# loc specifies row, column position\n",
    "df.loc[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# attribute of df object\n",
    "# row, columns\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# how many rows there are in a series/df\n",
    "df.shape[1] # len(df) would also work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Working with DataFrames\n",
    "\n",
    "There are *a lot* of functions and methods within `pandas`. The general syntax is `df.method()` where the `method()` operates directly on the dataframe `df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# calculate summary statistics\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Take the average of all columns\n",
    "df.mean(numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# edit values within a column and replace original values\n",
    "df['Subj_ID'] = df['Subj_ID'].replace('00', '000', regex=True)\n",
    "df['Subj_ID']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# specify the type of a variable in a column\n",
    "df['score'] = df['score'].astype(float)\n",
    "df['score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# breakdown of how many of each category there are\n",
    "val_counts = df['condition'].value_counts()\n",
    "val_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# which unique values are there in condition? \n",
    "df['condition'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# how many unique values are there\n",
    "df['condition'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# what's the category that shows up the most \n",
    "val_counts.idxmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# what's the count of the value that shows up the most\n",
    "val_counts.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Class Question #2\n",
    "\n",
    "Comparing them to standard library Python types, which is the best mapping for these new data types?\n",
    "\n",
    "- A) DataFrames are like lists, arrays are like tuples\n",
    "- B) DataFrames and arrays are like lists\n",
    "- C) DataFrames are like tuples, arrays are like lists\n",
    "- D) DataFrames and arrays are like dictionaries\n",
    "- E) Dataframes are like dictionaries, arrays are like lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# This allows plots to appear within the jupyter notebook\n",
    "%matplotlib inline  \n",
    "\n",
    "import matplotlib.pyplot as plt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Create some data\n",
    "dat = np.array([1, 2, 4, 8, 16, 32])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Plot the data\n",
    "plt.plot(dat);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- can change plot type\n",
    "- _lots_ of customizations possible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Analysis\n",
    "\n",
    "- `scipy` - statistical analysis\n",
    "- `scikit-learn` - machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import scipy as sp\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Simulate some data\n",
    "d1 = stats.norm.rvs(loc=0, size=1000)\n",
    "d2 = stats.norm.rvs(loc=0.5, size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Analysis - Plotting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Plot the data\n",
    "plt.hist(d1, 25, alpha=0.6);\n",
    "plt.hist(d2, 25, alpha=0.6);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Analysis - Statistical Comparisons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Statistically compare the two distributions\n",
    "stats.ttest_ind(d1, d2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## COGS 108: Data Science in Practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div class=\"alert alert-info\">\n",
    "If you are interested in data science and scientific computing in Python, consider taking <b>COGS 108</b> : <a>https://github.com/COGS108/</a>.\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  },
  "rise": {
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}