notebooks/Gradient_Methods.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:fb7f596c6f4a52227329b159e225c3f348427f3a51067709de4a699b27596fe0"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Gradient Descent (GD) Algorithms for Deep learning"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Given a function $\\ell: \\mathbb{R}^K \\rightarrow \\mathbb{R} $, the standard `gradient descent` algorithm is given by:\n",
      "> $\\theta_{t+1} \\leftarrow \\theta_{t} - \\eta \\nabla_{\\theta_{t} } \\ell(\\theta_{t})  $.\n",
      "\n",
      "In the context of deep learning, $\\ell$ can be considered as a loss function, $K$ the number of parameters (millions!)\n",
      "and the main goal is to minimize it over a given dataset $\\mathcal{D} = \\{\\mathbf{x}_i, \\mathbf{y}_i\\}_{i=1}^{n}$. \n",
      "More formally,\n",
      ">  $ \\underset{\\theta}{\\text{ minimize }} \\Big[ \\ell(\\theta) = \\frac{1}{n} \\sum_{i=1}^n \\text{dist}(f(\\mathbf{x}_i; \\theta), \\mathbf{y}) \\Big] $,\n",
      "\n",
      "where $f(\\mathbf{x}_i; \\theta)$ is the network prediction, and `dist` is the loss function. \n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### This notebook implements and visualizes well-known neural networks training algorithms using some toy functions. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The following algorithms are implemented. \n",
      "\n",
      "** 1. Momemtum **\n",
      "> $\\nu_{t+1} \\leftarrow \\beta \\nu_{t} + (1- \\beta) \\nabla_{\\theta_{t} } \\ell(\\theta_{t})$\n",
      "\n",
      "> $\\theta_{t+1} \\leftarrow \\theta_{t} - \\eta \\nu_{t+1}  $"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** 2. RMSProp **\n",
      "> $\\gamma_{t+1} \\leftarrow \\beta \\gamma_{t} + (1 - \\beta) (\\nabla_{\\theta_{t} } \\ell(\\theta_{t}))^2$\n",
      "\n",
      "> $\\theta_{t+1} \\leftarrow \\theta_{t} - \\eta \\nabla_{\\theta_{t} } \\ell(\\theta_{t}) (\\gamma_{t+1} + \\epsilon)^{-1/2}  $\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "** 3. [AdaDelta](https://arxiv.org/abs/1212.5701) **\n",
      "> $\\gamma_{t+1} \\leftarrow \\beta \\gamma_{t} + (1 - \\beta) (\\nabla_{\\theta_{t} } \\ell(\\theta_{t}))^2$\n",
      "\n",
      "> $\\Delta_{t+1} \\leftarrow (\\mu_{t-1} + \\epsilon)^{1/2} (\\gamma_{t+1} + \\epsilon)^{-1/2} \\nabla_{\\theta_{t} } \\ell(\\theta_{t})$\n",
      "\n",
      "> $\\mu_{t+1} \\leftarrow \\alpha \\mu_{t} +  (1 - \\alpha) (\\Delta_t)^2$\n",
      "\n",
      "> $\\theta_{t+1} \\leftarrow \\theta_{t} - \\Delta_{t+1}  $"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** 4. [Adam](https://arxiv.org/pdf/1412.6980v8.pdf) **\n",
      "> $\\nu_{t+1} \\leftarrow \\beta \\nu_{t} + (1- \\beta)  \\nabla_{\\theta_{t} } \\ell(\\theta_{t})$\n",
      "\n",
      "> $\\nu_{t+1} \\leftarrow \\nu_{t+1} ( 1 - \\beta^{t})^{-1}  $\n",
      "\n",
      "> $\\gamma_{t+1} \\leftarrow \\alpha \\gamma_{t} + (1 - \\alpha) (\\nabla_{\\theta_{t} } \\ell(\\theta_{t}))^2$\n",
      "\n",
      "> $\\gamma_{t+1} \\leftarrow \\gamma_{t+1} ( 1 - \\alpha^{t})^{-1}  $\n",
      "\n",
      "> $\\theta_{t+1} \\leftarrow \\theta_{t} - \\eta \\nu_{t+1} (\\gamma_{t+1} + \\epsilon)^{-1/2}   $"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "from numpy import random\n",
      "import matplotlib.pyplot as plt\n",
      "from mpl_toolkits.mplot3d import Axes3D\n",
      "\n",
      "from matplotlib import cm\n",
      "#%pylab inline\n",
      "import matplotlib.animation as animation"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Defining some non-linear functions "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class Func1:\n",
      "    def __init__(self):\n",
      "        self.dim = 2\n",
      "        self.base = lambda x, y : np.exp(-x**2 - y**2)\n",
      "    \n",
      "    def f(self, x):\n",
      "        x =  x if x.ndim >1  else x[np.newaxis] \n",
      "        return 2 * (-.1*self.base(x[:,0], x[:,1]) -.5*self.base(x[:,0]+4, x[:,1]+4)) + .09*np.tanh(x[:,0])  \n",
      "    def grad_f(self, x):\n",
      "        x = x if x.ndim >1  else x[np.newaxis]\n",
      "        res = [-4*x[:,0]*.1*self.base(x[:,0], x[:,1]) + 4*(x[:,0]-1)*.5*self.base(x[:,0]+4, x[:,1]+4) + .09- .09*np.tanh(x[:,0])**2,\n",
      "               4*x[:,1]*.1*self.base(x[:,0], x[:,1]) + 4*(x[:,1]-1)*.5*self.base(x[:,0]+4, x[:,1]+4) ]\n",
      "        return np.array([res[0][0], res[1][0]])\n",
      "    \n",
      "    def x_random(self,low=0,high=1.):\n",
      "        return random.random(self.dim)*(high-low)+low\n",
      "    \n",
      "class Func2:\n",
      "    def __init__(self):\n",
      "        self.dim = 2 \n",
      "    \n",
      "    def f(self, x):\n",
      "        x =  x if x.ndim >1  else x[np.newaxis] \n",
      "        return (x[:,0]**2)*np.sin(x[:,0]) #+ x[:,1]**2  \n",
      "    def grad_f(self, x):\n",
      "        x = x if x.ndim >1  else x[np.newaxis]\n",
      "        res = [2*x[:,0]*np.sin(x[:,0]) + (x[:,0]**2)*np.cos(x[:,0]), [0]]#2*x[:,1]]\n",
      "        return np.array([res[0][0], res[1][0]])\n",
      "    \n",
      "    def x_random(self,low=-5,high=5.):\n",
      "        return random.random(self.dim)*(high-low)+low\n",
      "    \n",
      "\n",
      "class Func3:\n",
      "    def __init__(self):\n",
      "        self.dim = 2 \n",
      "    \n",
      "    def f(self, x):\n",
      "        x =  x if x.ndim >1  else x[np.newaxis] \n",
      "        return x[:,0]**2 -  x[:,1]**2  \n",
      "    def grad_f(self, x):\n",
      "        x = x if x.ndim >1  else x[np.newaxis]\n",
      "        res = [2*x[:,0], -2*x[:,1]]\n",
      "        return np.array([res[0][0], res[1][0]])\n",
      "    \n",
      "    def x_random(self,low=0,high=1.):\n",
      "        return random.random(self.dim)*(high-low)+low"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Algorithms "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class GradientDescent:\n",
      "    def __init__(self, optim_f, alg=1, lr=1e-4, max_iter=5000, delta=1e-6, beta=.5, beta2=.8):\n",
      "        self.alg = alg\n",
      "        self.lr = lr\n",
      "        self.optim_f = optim_f\n",
      "        self.max_iter = max_iter\n",
      "        self.delta = delta\n",
      "        self.beta = beta\n",
      "        self.beta2 = beta2\n",
      "        self.reset() \n",
      "    \n",
      "    def reset(self, init_x=None):\n",
      "        self.i = 0\n",
      "        self.x = init_x if init_x is not None else self.optim_f.x_random() \n",
      "        self.log_x = np.array(self.x)\n",
      "        self.log_f = np.array(self.optim_f.f(self.x))\n",
      "        self.log_grad = np.array(self.optim_f.grad_f(self.x))\n",
      "        self.momentum = 0\n",
      "        self.running_average = 0\n",
      "        self.running_update = 0\n",
      "    \n",
      "    \n",
      "    def optimize(self,reset=True, init_x=None):\n",
      "        if reset:\n",
      "            self.reset(init_x)\n",
      "        while not self.stop():\n",
      "            self.i = self.i + 1\n",
      "            self.x = self.x - self.get_update()\n",
      "            self.log_x = np.vstack((self.log_x, self.x))\n",
      "            self.log_f = np.vstack((self.log_f , self.optim_f.f(self.x)))\n",
      "            self.log_grad = np.vstack((self.log_grad, self.optim_f.grad_f(self.x)))\n",
      "        print \"Optimized in %d iterations\" %(self.i)\n",
      "    def stop(self):\n",
      "        return (self.i>2) and ((self.max_iter and (self.i>=self.max_iter)) or ((self.delta and np.abs(self.log_f[-1]-self.log_f[-2]))<self.delta))\n",
      "    \n",
      "    def get_update(self):\n",
      "        if self.alg == 0: # Standard GD\n",
      "            return self.lr*self.log_grad[-1]\n",
      "        \n",
      "        if self.alg == 1: # Momentum GD\n",
      "            self.momentum = self.beta*self.momentum + (1 - self.beta)*self.log_grad[-1]\n",
      "            return self.lr*self.momentum\n",
      "        \n",
      "        if self.alg == 2: # RMSProp\n",
      "            self.running_average = self.beta*self.running_average + (1. - self.beta)*(self.log_grad[-1]**2)\n",
      "            return self.lr * self.log_grad[-1]/(np.sqrt(self.running_average) + self.delta)\n",
      "        \n",
      "        if self.alg == 3: # AdaDelta\n",
      "            self.running_average = self.beta*self.running_average + (1. - self.beta)*(self.log_grad[-1]**2)\n",
      "            update  = ((np.sqrt(self.running_update) + self.delta) * self.log_grad[-1])/(np.sqrt(self.running_average) + self.delta)\n",
      "            self.running_update = self.beta2*self.running_update + (1 - self.beta2)*(update**2) \n",
      "            return self.lr*update\n",
      "        \n",
      "        if self.alg == 4: # Adam\n",
      "            self.momentum = self.beta*self.momentum + (1-self.beta)*self.log_grad[-1]\n",
      "            self.momentum = self.momentum / (1 - self.beta**self.i)\n",
      "            self.running_average = self.beta2*self.running_average + (1-self.beta2)*(self.log_grad[-1]**2)\n",
      "            self.running_average = self.running_average / (1 - self.beta2**self.i)\n",
      "            return self.lr * self.momentum / (np.sqrt(self.running_average) + self.delta)\n",
      "        "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Visualization"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m=6;nb_pts=50\n",
      "func = Func2() \n",
      "xx,yy=np.meshgrid(np.linspace(-m,m,nb_pts), np.linspace(-m, m,nb_pts))\n",
      "grid=np.c_[xx.ravel(),yy.ravel()]\n",
      "z=func.f(grid).reshape(xx.shape)\n",
      "init_x = func.x_random(low=-6, high=-3)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "grd = GradientDescent(func, alg=1, lr=1, max_iter=50, beta=.8, beta2=.5, delta=1e-6)\n",
      "grd.optimize(reset=True, init_x=init_x)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Optimized in 50 iterations\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig = plt.figure()\n",
      "ax = fig.gca()\n",
      "ct = ax.contour(xx, yy, z)\n",
      "ax.clabel(ct, inline=1, fontsize=10)\n",
      "for i in range(0,len(grd.log_x)):\n",
      "    ax.scatter(grd.log_x[:,0][i], grd.log_x[:,1][i],color='black', marker='.')\n",
      "    plt.pause(1e-1) \n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "TclError",
       "evalue": "can't invoke \"update\" command: application has been destroyed",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mTclError\u001b[0m                                  Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-14-98dc78fa255a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog_x\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0max\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog_x\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog_x\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcolor\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'black'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmarker\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'.'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpause\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1e-1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/matplotlib/pyplot.pyc\u001b[0m in \u001b[0;36mpause\u001b[0;34m(interval)\u001b[0m\n\u001b[1;32m    294\u001b[0m             \u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw_idle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    295\u001b[0m         \u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 296\u001b[0;31m         \u001b[0mcanvas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_event_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minterval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    297\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    298\u001b[0m         \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minterval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/matplotlib/backend_bases.pyc\u001b[0m in \u001b[0;36mstart_event_loop\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m   2458\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_looping\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2459\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_looping\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mcounter\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mtimestep\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2460\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush_events\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2461\u001b[0m             \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimestep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2462\u001b[0m             \u001b[0mcounter\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/usr/local/lib/python2.7/dist-packages/matplotlib/backends/_backend_tk.pyc\u001b[0m in \u001b[0;36mflush_events\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    450\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    451\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mflush_events\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 452\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_master\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    453\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    454\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;32m/usr/lib/python2.7/lib-tk/Tkinter.pyc\u001b[0m in \u001b[0;36mupdate\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1020\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1021\u001b[0m         \u001b[0;34m\"\"\"Enter event loop until all pending events have been processed by Tcl.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1022\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'update'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1023\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mupdate_idletasks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1024\u001b[0m         \"\"\"Enter event loop until all idle callbacks have been called. This\n",
        "\u001b[0;31mTclError\u001b[0m: can't invoke \"update\" command: application has been destroyed"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "fig = plt.figure()\n",
      "ax = fig.gca(projection='3d')\n",
      "ax.plot_surface(xx, yy, z)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7f9dc0cbb2d0>"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}