09b_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Music machine learning - Pytorch\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)\n",
    "\n",
    "In this course we will cover\n",
    "1. A [quick introduction](#intro) to Pytorch \n",
    "2. An implementation for [advanced models](#models) implementation\n",
    "3. A quick proposal for [attention](#attention) layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Pytorch\n",
    "\n",
    "`Pytorch` is a Python-based scientific computing package targeted at deep learning, which provides a very large flexibility and easeness of use for GPU calculation. `Pytorch` is constructed around the concept of `Tensor`, which is very similar to `numpy.ndarray`, but can be seamlessly run on GPU.\n",
    "\n",
    "Here are some examples of different `Tensor` creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "# Create a 5 x 3 Tensor of zeros\n",
    "x = torch.empty(5, 3)\n",
    "# Create a 64 x 3 x 32 x 32 random Tensor\n",
    "x = torch.rand(64, 3, 32, 32)\n",
    "# Create a Tensor of zeros with _long_ type\n",
    "x = torch.zeros(10, 10, dtype=torch.long)\n",
    "# Construct a Tensor from the data\n",
    "x = torch.tensor([5.5, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or create a tensor based on an existing tensor. These methods\n",
    "will reuse properties of the input tensor, e.g. dtype, unless\n",
    "new values are provided by user\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 2])\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "x = x.new_ones(8, 2, dtype=torch.double)      # new_* methods take in sizes\n",
    "x = torch.randn_like(x, dtype=torch.float)    # override dtype!\n",
    "print(x.size())\n",
    "print(x.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arithmetic operations\n",
    "\n",
    "Tensors provide access to a transparent library of arithmetic operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5749, 0.5255],\n",
      "        [0.9960, 0.9434],\n",
      "        [1.7060, 1.3024],\n",
      "        [1.4466, 0.8099],\n",
      "        [1.0718, 1.0909],\n",
      "        [0.6841, 0.9996],\n",
      "        [0.5667, 1.8098],\n",
      "        [1.3244, 1.0366]])\n",
      "tensor([[0.5749, 0.5255],\n",
      "        [0.9960, 0.9434],\n",
      "        [1.7060, 1.3024],\n",
      "        [1.4466, 0.8099],\n",
      "        [1.0718, 1.0909],\n",
      "        [0.6841, 0.9996],\n",
      "        [0.5667, 1.8098],\n",
      "        [1.3244, 1.0366]])\n",
      "tensor([[0.5749, 0.5255],\n",
      "        [0.9960, 0.9434],\n",
      "        [1.7060, 1.3024],\n",
      "        [1.4466, 0.8099],\n",
      "        [1.0718, 1.0909],\n",
      "        [0.6841, 0.9996],\n",
      "        [0.5667, 1.8098],\n",
      "        [1.3244, 1.0366]])\n",
      "tensor([[0.5919, 0.9298],\n",
      "        [1.1173, 1.4515],\n",
      "        [2.5675, 1.7883],\n",
      "        [2.0762, 0.9926],\n",
      "        [1.5930, 1.7934],\n",
      "        [1.0501, 1.9560],\n",
      "        [0.6003, 2.7626],\n",
      "        [2.2344, 1.6342]])\n",
      "tensor([[0.0097, 0.2125],\n",
      "        [0.1208, 0.4794],\n",
      "        [1.4697, 0.6328],\n",
      "        [0.9108, 0.1480],\n",
      "        [0.5586, 0.7664],\n",
      "        [0.2503, 0.9560],\n",
      "        [0.0190, 1.7244],\n",
      "        [1.2052, 0.6195]])\n",
      "tensor([[0.6307, 0.5176, 0.4123, 0.0590],\n",
      "        [1.1036, 0.9269, 0.7322, 0.1031],\n",
      "        [1.7854, 1.3017, 1.0830, 0.1676],\n",
      "        [1.4153, 0.8352, 0.7575, 0.1336],\n",
      "        [1.2130, 1.0665, 0.8294, 0.1131],\n",
      "        [0.8757, 0.9574, 0.6950, 0.0809],\n",
      "        [1.0539, 1.6884, 1.1119, 0.0952],\n",
      "        [1.3946, 1.0338, 0.8547, 0.1308]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(8, 2)\n",
    "y = torch.rand(8, 2)\n",
    "z = torch.rand(2, 4)\n",
    "# Equivalent additions\n",
    "print(x + y)\n",
    "print(torch.add(x, y))\n",
    "# Add in place\n",
    "x.add_(y)\n",
    "print(x)\n",
    "# Put in target Tensor\n",
    "result = torch.empty(5, 3)\n",
    "torch.add(x, y, out=result)\n",
    "print(result)\n",
    "# Element_wise multiplication\n",
    "print(x * y)\n",
    "# Matrix product\n",
    "print(x @ z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing and resizing\n",
    "\n",
    "You can slice tensors using the usual Python operators. For resizing and reshaping tensor, you can use ``torch.view`` or ``torch.reshape``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.4963, -0.3877,  0.1876, -0.5845,  1.5996,  0.9076,  2.9362, -1.5117])\n",
      "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])\n"
     ]
    }
   ],
   "source": [
    "print(x[:, 1])\n",
    "x = torch.randn(4, 4)\n",
    "y = x.view(16)\n",
    "z = x.view(-1, 8)  # the size -1 is inferred from other dimensions\n",
    "print(x.size(), y.size(), z.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a one element tensor, use ``.item()`` to get the value as a\n",
    "Python number\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "x = torch.randn(1)\n",
    "print(x)\n",
    "print(x.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors have more than 100 operations, including transposing, indexing, slicing, mathematical operations, linear algebra, random numbers, which are all described at [https://pytorch.org/docs/torch](https://pytorch.org/docs/torch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Numpy bridge\n",
    "\n",
    "Converting a Torch Tensor to a Numpy array and vice versa is extremely simple. Note that the Pytorch Tensor and Numpy array **will share their underlying memory locations** (if the Tensor is on CPU), and changing one will change the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 2., 2., 2., 2.])\n",
      "[2. 2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(5)\n",
    "b = a.numpy()\n",
    "a.add_(1)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Going GPU\n",
    "\n",
    "Tensors can be moved onto any device using the ``.to`` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "# We will use ``torch.device`` objects to move tensors in and out of GPU\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")          # a CUDA device object\n",
    "    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU\n",
    "    x = x.to(device)                       # or just use strings ``.to(\"cuda\")``\n",
    "    z = x + y\n",
    "    print(z)\n",
    "    print(z.to(\"cpu\", torch.double))       # ``.to`` can also change dtype together!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computation Graphs\n",
    "\n",
    "The concept of a computation graph is essential to efficient deep learning programming, because it allows you to not have to write the back propagation gradients yourself. A computation graph is simply a specification of how your data is combined to give you the output (the forward pass). Since the graph totally specifies what parameters were involved with which operations, it contains enough information to compute derivatives. \n",
    "\n",
    "The fundamental flag ``requires_grad`` allows to specify which variables are going to need differentiation in all these operations. If ``requires_grad=True``, the Tensor object keeps track of how it was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5., 7., 9.], grad_fn=<AddBackward0>)\n",
      "<AddBackward0 object at 0x12d601fd0>\n"
     ]
    }
   ],
   "source": [
    "# Tensor factory methods have a ``requires_grad`` flag\n",
    "x = torch.tensor([1., 2., 3], requires_grad=True)\n",
    "# With requires_grad=True, we can still do all the operations \n",
    "y = torch.tensor([4., 5., 6], requires_grad=True)\n",
    "z = x + y\n",
    "print(z)\n",
    "# But z now knows something extra.\n",
    "print(z.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, `z` knows that it is the direct result of an addition. Furthermore, if we keep following z.grad_fn, we can even find back both `x` and `y`. But how does that help us compute a gradient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(21., grad_fn=<SumBackward0>)\n",
      "<SumBackward0 object at 0x12d5f4d90>\n"
     ]
    }
   ],
   "source": [
    "# Lets sum up all the entries in z\n",
    "s = z.sum()\n",
    "print(s)\n",
    "print(s.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now, what is the derivative of this sum with respect to the first\n",
    "component of x? In math, we want\n",
    "\n",
    "\\begin{align}\\frac{\\partial s}{\\partial x_0}\\end{align}\n",
    "\n",
    "\n",
    "\n",
    "Well, s knows that it was created as a sum of the tensor z. z knows\n",
    "that it was the sum x + y. So\n",
    "\n",
    "\\begin{align}s = \\overbrace{x_0 + y_0}^\\text{$z_0$} + \\overbrace{x_1 + y_1}^\\text{$z_1$} + \\overbrace{x_2 + y_2}^\\text{$z_2$}\\end{align}\n",
    "\n",
    "And so s contains enough information to determine that the derivative we want is 1. We can have Pytorch compute the gradient, and see that we were right:\n",
    "\n",
    "**Note** : If you run this block multiple times, the gradient will increment. That is because Pytorch *accumulates* the gradient into the .grad property, since for many models this is very convenient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "# calling .backward() on any variable will run backprop, starting from it.\n",
    "s.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding what is going on in the block below is crucial for being a\n",
    "successful programmer in deep learning.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False False\n",
      "None\n",
      "<AddBackward0 object at 0x12d60ca10>\n",
      "True\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(2, 2)\n",
    "y = torch.randn(2, 2)\n",
    "# By default, user created Tensors have ``requires_grad=False``\n",
    "print(x.requires_grad, y.requires_grad)\n",
    "z = x + y\n",
    "# So you can't backprop through z\n",
    "print(z.grad_fn)\n",
    "# ``.requires_grad_( ... )`` changes an existing Tensor's ``requires_grad``\n",
    "x = x.requires_grad_()\n",
    "y = y.requires_grad_()\n",
    "# z contains enough information to compute gradients, as we saw above\n",
    "z = x + y\n",
    "print(z.grad_fn)\n",
    "# If any input to an operation has ``requires_grad=True``, so will the output\n",
    "print(z.requires_grad)\n",
    "# Now z has the computation history, which we can **detach**\n",
    "new_z = z.detach()\n",
    "# Which means that we have no gradient attached anymore\n",
    "print(new_z.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also stop autograd from tracking history on Tensors\n",
    "with ``.requires_grad=True`` by wrapping the code block in\n",
    "``with torch.no_grad():``\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "with torch.no_grad():\n",
    "\tprint((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining networks \n",
    "\n",
    "Here, we briefly recall that in `PyTorch`, the `nn` package provides higher-level abstractions over raw computational graphs that are useful for building neural networks. The `nn` package defines a set of `Modules`, which are roughly equivalent to neural network layers. A `Module` receives input `Tensors` and computes output `Tensors`, but may also hold internal state such as `Tensors` containing learnable parameters. In the following example, we use the `nn` package to show how easy it is to instantiate a three-layer network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "# Define the input dimensions\n",
    "in_size = 1000\n",
    "# Number of neurons in a layer\n",
    "hidden_size = 100\n",
    "# Output (target) dimension\n",
    "output_size = 10\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = torch.nn.Sequential(\n",
    "    nn.Linear(in_size, hidden_size),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(hidden_size, hidden_size),\n",
    "    nn.Tanh(),\n",
    "    nn.Linear(hidden_size, output_size),\n",
    "    nn.Softmax()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen in the slides, we can as easily mix between pre-defined modules and arithmetic operations. Here, we will define a *residual* block, and then combine them in a more complex network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResBlock(nn.Module):\n",
    "    def __init__(self, dim, dim_res=32):\n",
    "        super().__init__()\n",
    "        self.block = nn.Sequential(\n",
    "            nn.Conv2d(dim, dim_res, 3, 1, 1),\n",
    "            nn.ReLU(True),\n",
    "            nn.Conv2d(dim_res, dim, 1),\n",
    "            nn.ReLU(True)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return x + self.block(x)\n",
    "\n",
    "model = nn.Sequential(\n",
    "\tResBlock(64, 32),\n",
    "\tResBlock(64, 32),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining our own layers\n",
    "\n",
    "In the following, we re-implement the *attention* layer, which is the basis of the infamous `Transformer` models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionLayer(nn.Module):\n",
    "    def __init__(self, n_hidden):\n",
    "        super(ChordLevelAttention, self).__init__()\n",
    "        self.mlp = nn.Linear(n_hidden, n_hidden)\n",
    "        self.u_w = nn.Parameter(torch.rand(n_hidden))\n",
    "\n",
    "    def forward(self, X):\n",
    "        # get the hidden representation of the sequence\n",
    "        u_it = F.tanh(self.mlp(X))\n",
    "        # get attention weights for each timestep\n",
    "        alpha = F.softmax(torch.matmul(u_it, self.u_w), dim=1)\n",
    "        # get the weighted sum of the sequence\n",
    "        out = torch.sum(torch.matmul(alpha, X), dim=1)\n",
    "        return out, alpha\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}