03_numpy_io_matlab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NumPy Seralisation and I/O\n",
    "\n",
    "In this notebook we will focus on NumPy built-in support for **Serialisation** and **I/O**. In other words, we will learn how to save and load NumPy `ndarray` objects in native (binary) format for easy sharing. Moreover we are going to discover how NumPy can load data from external files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Comma-separated values (CSV)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A very common file format for data files are the comma-separated values (CSV), or related format such as TSV (tab-separated values). \n",
    "\n",
    "To read data from such file into Numpy arrays we can use the `numpy.genfromtxt` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year Month Day T_6 T12 T18 Valid \r\n",
      "1800  1  1    -6.1    -6.1    -6.1 1\r\n",
      "1800  1  2   -15.4   -15.4   -15.4 1\r\n",
      "1800  1  3   -15.0   -15.0   -15.0 1\r\n",
      "1800  1  4   -19.3   -19.3   -19.3 1\r\n",
      "1800  1  5   -16.8   -16.8   -16.8 1\r\n",
      "1800  1  6   -11.4   -11.4   -11.4 1\r\n",
      "1800  1  7    -7.6    -7.6    -7.6 1\r\n",
      "1800  1  8    -7.1    -7.1    -7.1 1\r\n",
      "1800  1  9   -10.1   -10.1   -10.1 1\r\n"
     ]
    }
   ],
   "source": [
    "# In Jupyter, all commands starting with ! are mapped as SHELL commands\n",
    "!head stockholm_td_adj.dat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.genfromtxt?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "st_temperatures = np.genfromtxt('stockholm_td_adj.dat', \n",
    "                                skip_header=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(77431, 7)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "st_temperatures.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DYI\n",
    "\n",
    "Let's play a bit with the data loaded `st_temperatures` to combine **fancy indexing** (i.e. defining conditions to get subset of data) and very simple statistics.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.80e+03,  1.00e+00,  1.00e+00, -6.10e+00, -6.10e+00, -6.10e+00,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  2.00e+00, -1.54e+01, -1.54e+01, -1.54e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  3.00e+00, -1.50e+01, -1.50e+01, -1.50e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  4.00e+00, -1.93e+01, -1.93e+01, -1.93e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  5.00e+00, -1.68e+01, -1.68e+01, -1.68e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  6.00e+00, -1.14e+01, -1.14e+01, -1.14e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  7.00e+00, -7.60e+00, -7.60e+00, -7.60e+00,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  8.00e+00, -7.10e+00, -7.10e+00, -7.10e+00,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  9.00e+00, -1.01e+01, -1.01e+01, -1.01e+01,\n",
       "         1.00e+00],\n",
       "       [ 1.80e+03,  1.00e+00,  1.00e+01, -9.50e+00, -9.50e+00, -9.50e+00,\n",
       "         1.00e+00]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "st_temperatures[:10, ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "st_temperatures.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810,\n",
       "        1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821,\n",
       "        1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832,\n",
       "        1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843,\n",
       "        1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854,\n",
       "        1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865,\n",
       "        1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876,\n",
       "        1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887,\n",
       "        1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898,\n",
       "        1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909,\n",
       "        1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920,\n",
       "        1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931,\n",
       "        1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942,\n",
       "        1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953,\n",
       "        1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964,\n",
       "        1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975,\n",
       "        1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986,\n",
       "        1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997,\n",
       "        1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,\n",
       "        2009, 2010, 2011]), 212)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Calculate which and how many years we have in our data\n",
    "years = np.unique(st_temperatures[:, 0]).astype(np.int)\n",
    "years, len(years)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1800, 2011)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "years.min(), years.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year Month Day T_6 T12 T18 Valid \r\n",
      "1800  1  1    -6.1    -6.1    -6.1 1\r\n",
      "1800  1  2   -15.4   -15.4   -15.4 1\r\n",
      "1800  1  3   -15.0   -15.0   -15.0 1\r\n",
      "1800  1  4   -19.3   -19.3   -19.3 1\r\n",
      "1800  1  5   -16.8   -16.8   -16.8 1\r\n",
      "1800  1  6   -11.4   -11.4   -11.4 1\r\n",
      "1800  1  7    -7.6    -7.6    -7.6 1\r\n",
      "1800  1  8    -7.1    -7.1    -7.1 1\r\n",
      "1800  1  9   -10.1   -10.1   -10.1 1\r\n"
     ]
    }
   ],
   "source": [
    "!head stockholm_td_adj.dat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_year = st_temperatures[:, 0] == 1984"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_feb = st_temperatures[:, 1] == 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(77431,)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_feb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('bool')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask_year.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(mask_year)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Calculate the mean temperature of mid-days on February in 1984\n",
    "feb_noon_temps = st_temperatures[(mask_year & mask_feb), 4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(feb_noon_temps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feb_noon_temps.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.7344827586206901"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feb_noon_temps.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "## ...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Numpy's native file format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Useful when storing and reading back numpy array data. \n",
    "\n",
    "* Use the functions `np.save` and `np.load`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `np.save`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np.save(\"st_temperatures.npy\", st_temperatures)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**See also**:\n",
    "\n",
    "- `np.savez` : save several NumPy arrays into one single file\n",
    "- `np.savez_compressed`\n",
    "- `np.savetxt`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `np.load`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(77431, 7) float64\n"
     ]
    }
   ],
   "source": [
    "T = np.load(\"st_temperatures.npy\")\n",
    "print(T.shape, T.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## NumPy for Matlab Users (really?)\n",
    "\n",
    "\n",
    "If you are a MATLAB&reg; user I do recommend to read [Numpy for MATLAB Users](https://docs.scipy.org/doc/numpy-1.15.0/user/numpy-for-matlab-users.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy can load and save native MATLAB® files:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The `Matrix` Array Type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In addition to the `numpy.ndarray` type, NumPy also support a very specific data type called `Matrix`. \n",
    "\n",
    "This special type of object has been introduced to allow for API and programming compatibility with\n",
    "MATLAB®. \n",
    "\n",
    "**Note**: The most relevant feature of this new _array type_ is the behavior of the standard arithmetic operators `+, -, *` to use matrix algebra, which work as they would in MATLAB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from numpy import matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(0, 5)\n",
    "A = np.array([[n+m*10 for n in range(5)] for m in range(5)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3,  4],\n",
       "       [10, 11, 12, 13, 14],\n",
       "       [20, 21, 22, 23, 24],\n",
       "       [30, 31, 32, 33, 34],\n",
       "       [40, 41, 42, 43, 44]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "M = matrix(A)\n",
    "v = matrix(a).T # make it a column vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 300,  310,  320,  330,  340],\n",
       "        [1300, 1360, 1420, 1480, 1540],\n",
       "        [2300, 2410, 2520, 2630, 2740],\n",
       "        [3300, 3460, 3620, 3780, 3940],\n",
       "        [4300, 4510, 4720, 4930, 5140]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M * M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 300,  310,  320,  330,  340],\n",
       "       [1300, 1360, 1420, 1480, 1540],\n",
       "       [2300, 2410, 2520, 2630, 2740],\n",
       "       [3300, 3460, 3620, 3780, 3940],\n",
       "       [4300, 4510, 4720, 4930, 5140]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A @ A  # @ operator equivalent to np.dot(A, A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[   0,    1,    4,    9,   16],\n",
       "       [ 100,  121,  144,  169,  196],\n",
       "       [ 400,  441,  484,  529,  576],\n",
       "       [ 900,  961, 1024, 1089, 1156],\n",
       "       [1600, 1681, 1764, 1849, 1936]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Element wise multiplication in NumPy\n",
    "A * A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 30],\n",
       "        [130],\n",
       "        [230],\n",
       "        [330],\n",
       "        [430]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M * v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   1,   4,   9,  16],\n",
       "       [  0,  11,  24,  39,  56],\n",
       "       [  0,  21,  44,  69,  96],\n",
       "       [  0,  31,  64,  99, 136],\n",
       "       [  0,  41,  84, 129, 176]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A * a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[30]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inner product\n",
    "v.T * v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 30],\n",
       "        [131],\n",
       "        [232],\n",
       "        [333],\n",
       "        [434]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with matrix objects, standard matrix algebra applies\n",
    "v + M*v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If we try to add, subtract or multiply objects with incomplatible shapes we get an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "v_incompat = matrix(list(range(1, 7))).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((5, 5), (6, 1))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M.shape, v_incompat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "shapes (5,5) and (6,1) not aligned: 5 (dim 1) != 6 (dim 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-17-bd00a30033f6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mM\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mv_incompat\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/envs/numpy-euroscipy/lib/python3.7/site-packages/numpy/matrixlib/defmatrix.py\u001b[0m in \u001b[0;36m__mul__\u001b[0;34m(self, other)\u001b[0m\n\u001b[1;32m    218\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    219\u001b[0m             \u001b[0;31m# This promotes 1-D vectors to row vectors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 220\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0masmatrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    221\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misscalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'__rmul__'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    222\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mN\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mdot\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: shapes (5,5) and (6,1) not aligned: 5 (dim 1) != 6 (dim 0)"
     ]
    }
   ],
   "source": [
    "M * v_incompat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "See also the related functions: `inner`, `outer`, `cross`, `kron`, `tensordot`. \n",
    "\n",
    "Try for example `help(inner)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Saving `.mat` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a `numpy.ndarray` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "A = np.random.rand(10000, 300, 50)  # note: this may take a while"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.30788845, 0.60569692, 0.74159203, ..., 0.99513856,\n",
       "         0.86615676, 0.65581839],\n",
       "        [0.29972906, 0.1727805 , 0.73877596, ..., 0.57321798,\n",
       "         0.52657155, 0.15148499],\n",
       "        [0.91677054, 0.30289045, 0.47086303, ..., 0.91076997,\n",
       "         0.15659756, 0.74502433],\n",
       "        ...,\n",
       "        [0.16246413, 0.57601666, 0.64519549, ..., 0.04166688,\n",
       "         0.71115738, 0.75984878],\n",
       "        [0.99626814, 0.89529207, 0.89520696, ..., 0.927474  ,\n",
       "         0.46998733, 0.809978  ],\n",
       "        [0.52545775, 0.42922203, 0.40999633, ..., 0.7497839 ,\n",
       "         0.26582518, 0.68821719]],\n",
       "\n",
       "       [[0.93763072, 0.68660253, 0.03060252, ..., 0.08489496,\n",
       "         0.3368953 , 0.0040575 ],\n",
       "        [0.17680589, 0.44922269, 0.32552186, ..., 0.49081397,\n",
       "         0.7718607 , 0.91216332],\n",
       "        [0.48935017, 0.28293444, 0.57762148, ..., 0.64988995,\n",
       "         0.96036063, 0.62395338],\n",
       "        ...,\n",
       "        [0.77554755, 0.23174591, 0.80126054, ..., 0.34982511,\n",
       "         0.13648038, 0.63953428],\n",
       "        [0.4502637 , 0.74376194, 0.47531237, ..., 0.94077276,\n",
       "         0.64544446, 0.20241967],\n",
       "        [0.65158873, 0.93520847, 0.1153165 , ..., 0.92607143,\n",
       "         0.42194542, 0.49231582]],\n",
       "\n",
       "       [[0.60652634, 0.55707594, 0.7861307 , ..., 0.49618863,\n",
       "         0.26073645, 0.57230289],\n",
       "        [0.33445447, 0.51254754, 0.89760192, ..., 0.20161607,\n",
       "         0.54935607, 0.97355349],\n",
       "        [0.82742407, 0.13811956, 0.77549593, ..., 0.97417726,\n",
       "         0.75828111, 0.20726388],\n",
       "        ...,\n",
       "        [0.89885131, 0.95168761, 0.04908857, ..., 0.26560786,\n",
       "         0.19828306, 0.34056713],\n",
       "        [0.37462286, 0.00294645, 0.46417234, ..., 0.98287275,\n",
       "         0.63560479, 0.37498829],\n",
       "        [0.80824186, 0.77414402, 0.27137252, ..., 0.97397635,\n",
       "         0.73792667, 0.47235421]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0.79534194, 0.19495982, 0.69419483, ..., 0.98484659,\n",
       "         0.07524489, 0.35898295],\n",
       "        [0.75246125, 0.1448565 , 0.31596133, ..., 0.97989236,\n",
       "         0.66466035, 0.09253075],\n",
       "        [0.13218267, 0.24674062, 0.93687433, ..., 0.26530807,\n",
       "         0.64653497, 0.25848279],\n",
       "        ...,\n",
       "        [0.01839164, 0.4127106 , 0.36428583, ..., 0.97212349,\n",
       "         0.867556  , 0.58971199],\n",
       "        [0.49075206, 0.80264193, 0.82420669, ..., 0.13249282,\n",
       "         0.70465219, 0.97575252],\n",
       "        [0.2735621 , 0.37780973, 0.19581884, ..., 0.55415141,\n",
       "         0.33630774, 0.62376131]],\n",
       "\n",
       "       [[0.95740591, 0.6409855 , 0.29668168, ..., 0.85582114,\n",
       "         0.02653775, 0.07433918],\n",
       "        [0.97968508, 0.7192658 , 0.96627464, ..., 0.25708965,\n",
       "         0.60037787, 0.8001345 ],\n",
       "        [0.98598865, 0.7660025 , 0.05743886, ..., 0.84864957,\n",
       "         0.5717346 , 0.48107095],\n",
       "        ...,\n",
       "        [0.04048004, 0.24279597, 0.43556563, ..., 0.74962769,\n",
       "         0.71872639, 0.08429666],\n",
       "        [0.09697323, 0.51034331, 0.6199531 , ..., 0.95157892,\n",
       "         0.52082535, 0.36331146],\n",
       "        [0.91967882, 0.47842183, 0.55403126, ..., 0.99053768,\n",
       "         0.68606411, 0.4186365 ]],\n",
       "\n",
       "       [[0.83101977, 0.7800826 , 0.52552153, ..., 0.45411436,\n",
       "         0.96688267, 0.14787061],\n",
       "        [0.76365986, 0.97841123, 0.99583821, ..., 0.96043423,\n",
       "         0.72406206, 0.97100977],\n",
       "        [0.92772653, 0.01373546, 0.59448744, ..., 0.64587074,\n",
       "         0.13641851, 0.40625453],\n",
       "        ...,\n",
       "        [0.24169963, 0.22511255, 0.85599095, ..., 0.75448232,\n",
       "         0.42633244, 0.31373371],\n",
       "        [0.28480721, 0.83815003, 0.77828307, ..., 0.52597019,\n",
       "         0.88834579, 0.09847287],\n",
       "        [0.32613764, 0.67313394, 0.82862416, ..., 0.87137257,\n",
       "         0.13503096, 0.0888404 ]]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introducing SciPy (ecosystem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![scipy](images/scipy.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `scipy.io`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from scipy import io as spio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy $\\mapsto$ MATLAB :  `scipy.io.savemat`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "spio.savemat('numpy_to.mat', {'A': A}, oned_as='row')  # savemat expects a dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MATLAB $\\mapsto$ NumPy: `scipy.io.loadmat`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dictionary = spio.loadmat('numpy_to.mat')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__header__', '__version__', '__globals__', 'A']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(data_dictionary.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.30788845, 0.60569692, 0.74159203, ..., 0.99513856,\n",
       "         0.86615676, 0.65581839],\n",
       "        [0.29972906, 0.1727805 , 0.73877596, ..., 0.57321798,\n",
       "         0.52657155, 0.15148499],\n",
       "        [0.91677054, 0.30289045, 0.47086303, ..., 0.91076997,\n",
       "         0.15659756, 0.74502433],\n",
       "        ...,\n",
       "        [0.16246413, 0.57601666, 0.64519549, ..., 0.04166688,\n",
       "         0.71115738, 0.75984878],\n",
       "        [0.99626814, 0.89529207, 0.89520696, ..., 0.927474  ,\n",
       "         0.46998733, 0.809978  ],\n",
       "        [0.52545775, 0.42922203, 0.40999633, ..., 0.7497839 ,\n",
       "         0.26582518, 0.68821719]],\n",
       "\n",
       "       [[0.93763072, 0.68660253, 0.03060252, ..., 0.08489496,\n",
       "         0.3368953 , 0.0040575 ],\n",
       "        [0.17680589, 0.44922269, 0.32552186, ..., 0.49081397,\n",
       "         0.7718607 , 0.91216332],\n",
       "        [0.48935017, 0.28293444, 0.57762148, ..., 0.64988995,\n",
       "         0.96036063, 0.62395338],\n",
       "        ...,\n",
       "        [0.77554755, 0.23174591, 0.80126054, ..., 0.34982511,\n",
       "         0.13648038, 0.63953428],\n",
       "        [0.4502637 , 0.74376194, 0.47531237, ..., 0.94077276,\n",
       "         0.64544446, 0.20241967],\n",
       "        [0.65158873, 0.93520847, 0.1153165 , ..., 0.92607143,\n",
       "         0.42194542, 0.49231582]],\n",
       "\n",
       "       [[0.60652634, 0.55707594, 0.7861307 , ..., 0.49618863,\n",
       "         0.26073645, 0.57230289],\n",
       "        [0.33445447, 0.51254754, 0.89760192, ..., 0.20161607,\n",
       "         0.54935607, 0.97355349],\n",
       "        [0.82742407, 0.13811956, 0.77549593, ..., 0.97417726,\n",
       "         0.75828111, 0.20726388],\n",
       "        ...,\n",
       "        [0.89885131, 0.95168761, 0.04908857, ..., 0.26560786,\n",
       "         0.19828306, 0.34056713],\n",
       "        [0.37462286, 0.00294645, 0.46417234, ..., 0.98287275,\n",
       "         0.63560479, 0.37498829],\n",
       "        [0.80824186, 0.77414402, 0.27137252, ..., 0.97397635,\n",
       "         0.73792667, 0.47235421]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0.79534194, 0.19495982, 0.69419483, ..., 0.98484659,\n",
       "         0.07524489, 0.35898295],\n",
       "        [0.75246125, 0.1448565 , 0.31596133, ..., 0.97989236,\n",
       "         0.66466035, 0.09253075],\n",
       "        [0.13218267, 0.24674062, 0.93687433, ..., 0.26530807,\n",
       "         0.64653497, 0.25848279],\n",
       "        ...,\n",
       "        [0.01839164, 0.4127106 , 0.36428583, ..., 0.97212349,\n",
       "         0.867556  , 0.58971199],\n",
       "        [0.49075206, 0.80264193, 0.82420669, ..., 0.13249282,\n",
       "         0.70465219, 0.97575252],\n",
       "        [0.2735621 , 0.37780973, 0.19581884, ..., 0.55415141,\n",
       "         0.33630774, 0.62376131]],\n",
       "\n",
       "       [[0.95740591, 0.6409855 , 0.29668168, ..., 0.85582114,\n",
       "         0.02653775, 0.07433918],\n",
       "        [0.97968508, 0.7192658 , 0.96627464, ..., 0.25708965,\n",
       "         0.60037787, 0.8001345 ],\n",
       "        [0.98598865, 0.7660025 , 0.05743886, ..., 0.84864957,\n",
       "         0.5717346 , 0.48107095],\n",
       "        ...,\n",
       "        [0.04048004, 0.24279597, 0.43556563, ..., 0.74962769,\n",
       "         0.71872639, 0.08429666],\n",
       "        [0.09697323, 0.51034331, 0.6199531 , ..., 0.95157892,\n",
       "         0.52082535, 0.36331146],\n",
       "        [0.91967882, 0.47842183, 0.55403126, ..., 0.99053768,\n",
       "         0.68606411, 0.4186365 ]],\n",
       "\n",
       "       [[0.83101977, 0.7800826 , 0.52552153, ..., 0.45411436,\n",
       "         0.96688267, 0.14787061],\n",
       "        [0.76365986, 0.97841123, 0.99583821, ..., 0.96043423,\n",
       "         0.72406206, 0.97100977],\n",
       "        [0.92772653, 0.01373546, 0.59448744, ..., 0.64587074,\n",
       "         0.13641851, 0.40625453],\n",
       "        ...,\n",
       "        [0.24169963, 0.22511255, 0.85599095, ..., 0.75448232,\n",
       "         0.42633244, 0.31373371],\n",
       "        [0.28480721, 0.83815003, 0.77828307, ..., 0.52597019,\n",
       "         0.88834579, 0.09847287],\n",
       "        [0.32613764, 0.67313394, 0.82862416, ..., 0.87137257,\n",
       "         0.13503096, 0.0888404 ]]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_dictionary['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_load = data_dictionary['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.all(A == A_load)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(A_load)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 (NumPy EuroSciPy)",
   "language": "python",
   "name": "numpy-euroscipy"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}