pset3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Set 3\n",
    "Designed by Sarah Adel Bargal, Ben Usman, and Kun He, with help from Kate Saenko and Brian Kulis.\n",
    "\n",
    "This assignment will introduce you to:\n",
    "- building, training, and testing of neural networks in TensorFlow\n",
    "- using TensorFlow on the SCC cluster\n",
    "- extracting and visualizing neural network features\n",
    "\n",
    "Note: For programming, the use of any additional abstraction of TensorFlow (like `Keras` and `tf.contrib.learn`) is **not permitted**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 0: Tutorials\n",
    "The contents of these tutorials are helpful for solving this assignment and upcoming assignments. Please be sure to go through them before proceeding.\n",
    "1. [MNIST For ML Beginners](https://www.tensorflow.org/versions/r0.10/tutorials/mnist/beginners/)\n",
    "2. [Deep MNIST for Experts](https://www.tensorflow.org/versions/r0.10/tutorials/mnist/pros/)\n",
    "3. [TensorFlow Mechanics 101](https://www.tensorflow.org/versions/r0.10/tutorials/mnist/tf/)\n",
    "4. [Sharing Variables](https://www.tensorflow.org/programmers_guide/variable_scope)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: MNIST Softmax Classifier Demo in TensorFlow\n",
    "\n",
    "TensorFlow is already installed on the SCC. Please review the instructions on [connecting to SCC](https://docs.google.com/document/d/1C4XrrTZIRfmJ6-LHmuJ4levTnunezujVrxCz8jJ4rec), and [running jobs on SCC](http://www.bu.edu/tech/support/research/system-usage/running-jobs/).\n",
    "\n",
    "Make sure you are capable of running this demo (using `qsub`) on the SCC cluster: [`mnist_softmax_scope.py`](https://github.com/kunhe/cs591s2/blob/master/mnist_softmax_scope.py). There is no required deliverable, but this exercise is helpful for running jobs on the SCC in the future.\n",
    "\n",
    "The demo is another implementation of `mnist_softmax.py` presented in the MNIST for ML Beginners tutorial, but using scopes. For the purposes of this assignment, the contents of the demo is also replicated below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "def logistic_regression(x_):\n",
    "    # create the actual model\n",
    "    scope_args = {'initializer': tf.random_normal_initializer(stddev=1e-4)}\n",
    "    with tf.variable_scope(\"weights\", **scope_args):\n",
    "        W = tf.get_variable('W', shape=[784, 10])\n",
    "        b = tf.get_variable('b', shape=[10])\n",
    "        y_logits = tf.matmul(x_, W) + b\n",
    "    return y_logits\n",
    "\n",
    "def test_classification(model_function, learning_rate=0.1):\n",
    "    # import data\n",
    "    mnist = input_data.read_data_sets('./datasets/mnist/', one_hot=True)\n",
    "\n",
    "    with tf.Graph().as_default() as g:\n",
    "        # where are you going to allocate memory and perform computations\n",
    "        with tf.device(\"/cpu:0\"):\n",
    "            \n",
    "            # define model \"input placeholders\", i.e. variables that are\n",
    "            # going to be substituted with input data on train/test time\n",
    "            x_ = tf.placeholder(tf.float32, [None, 784])\n",
    "            y_ = tf.placeholder(tf.float32, [None, 10])\n",
    "            y_logits = model_function(x_)\n",
    "            \n",
    "            # naive implementation of loss:\n",
    "            # > losses = y_ * tf.log(tf.nn.softmax(y_logits))\n",
    "            # > tf.reduce_mean(-tf.reduce_sum(losses, 1))\n",
    "            # can be numerically unstable.\n",
    "            #\n",
    "            # so here we use tf.nn.softmax_cross_entropy_with_logits on the raw\n",
    "            # outputs of 'y', and then average across the batch.\n",
    "            \n",
    "            losses = tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_logits)\n",
    "            cross_entropy_loss = tf.reduce_mean(losses)\n",
    "            train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cross_entropy_loss)\n",
    "            \n",
    "            y_pred = tf.argmax(tf.nn.softmax(y_logits), dimension=1)\n",
    "            correct_prediction = tf.equal(y_pred, tf.argmax(y_, 1))\n",
    "            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "    with g.as_default(), tf.Session() as sess:\n",
    "        # that is how we \"execute\" statements \n",
    "        # (return None, e.g. init() or train_op())\n",
    "        # or compute parts of graph defined above (loss, output, etc.)\n",
    "        # given certain input (x_, y_)\n",
    "        sess.run(tf.initialize_all_variables())\n",
    "        \n",
    "        # train\n",
    "        for iter_i in range(20001):\n",
    "            batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "            sess.run(train_step, feed_dict={x_: batch_xs, y_: batch_ys})\n",
    "            \n",
    "            # test trained model\n",
    "            if iter_i % 2000 == 0:\n",
    "                tf_feed_dict = {x_: mnist.test.images, y_: mnist.test.labels}\n",
    "                acc_value = sess.run(accuracy, feed_dict=tf_feed_dict)\n",
    "                print('iteration %d\\t accuracy: %.3f'%(iter_i, acc_value))\n",
    "                \n",
    "test_classification(logistic_regression, learning_rate=0.1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Building Neural Networks in TensorFlow\n",
    "\n",
    "(45 points)\n",
    "\n",
    "### Q2.1 MLP in TensorFlow\n",
    "\n",
    "Task: \n",
    "\n",
    "- Implement a multi-layer perceptron function **`mlp(x, hidden_sizes, activation_fn)`** in TensorFlow that has the following input and output: \n",
    "\n",
    " **Inputs**\n",
    " - `x`, an input tensor of the images in the current batch `[batch_size, 28x28]`\n",
    " - `hidden_sizes`, a list of the number of hidden units per layer. For example: `[5,2]` means  5 hidden units in the first layer, and 2 hidden units in the second (output) layer.\n",
    " (Note: for MNIST, we need `hidden_sizes[-1]==10` since it has 10 classes.)\n",
    " - `activation_fn`, the activation function to be applied\n",
    "\n",
    " **Output**\n",
    " - a tensor of shape `[batch_size, hidden_sizes[-1]]`. \n",
    "\n",
    "**Note: ** \n",
    "- Make sure the activation function is not applied to the final (output) layer.\n",
    "- It is recommended to use scopes and `tf.get_variable()` (as opposed to `tf.Variable()` which you may see elsewhere), as demonstrated in the sample code, and explained in the \"Sharing Variables\" tutorial in Part 0. Also see [here](http://stackoverflow.com/questions/37098546/difference-between-variable-and-get-variable-in-tensorflow)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def mlp(x, hidden_sizes, activation_fn=tf.nn.relu):\n",
    "    if not isinstance(hidden_sizes, (list, tuple)):\n",
    "        raise ValueError(\"hidden_sizes must be a list or a tuple\")\n",
    "        \n",
    "    ###################################\n",
    "    ####     PUT YOUR CODE HERE    ####\n",
    "    ###################################\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code tests your  implementation of the **`mlp()`** function. It basically recreates the 2-layer MLP with ReLU activation that you implemented in Problem Set 2. It should give an accuracy of above 0.97."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_classification(lambda x: mlp(x, [64, 10], activation_fn=tf.nn.relu), learning_rate=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q2.2 Siamese Network\n",
    "A siamese network is a network having two identical subnetworks that share parameters. A siamese network gives a *similarity metric* between pairs of inputs, which assigns high values to similar pairs and low values to dissimilar pairs. In MNIST, we can construct similar pairs by taking images from the same digit class, and dissimilar pairs from images from different classes. \n",
    "\n",
    "\n",
    "A sample Siamese network is presented below [Koch et al., Siamese Neural Networks for One-shot Image Recognition, ICML 2015]. Note how the two subnetworks have identical parameters:\n",
    "\n",
    "<img src=\"siamese.png\" style=\"width:480px;\">\n",
    "\n",
    "\n",
    "Given a Siamese network, we can determine the similarity between inputs $x_1$ and $x_2$, by taking the sign of the cosine similarity $\\frac{\\langle r_1,r_2\\rangle}{\\|r_1\\|\\|r_2\\|}$, where $r_1$ and $r_2$ are the hidden representations produced by the network for $x_1$ and $x_2$, respectively. \n",
    "\n",
    "Tasks:\n",
    "\n",
    "1. Build a Siamese Network in TensorFlow that receives as input two MNIST images, and decides whether they belong to the same digit category. Each subnetwork is an MLP with 2 hidden layers with 64 units, followed by a \"distance layer\" with 32 units. They will be created using the **`mlp()`** function you implemented earlier, using input argument `hidden_sizes=[64,64,32]`. The network computes the cosine similarity, as defined above, in the final output layer.\n",
    "\n",
    "2. Train and test this model, and  report your test accuracy for this task.\n",
    "\n",
    "You will need to implement the following two functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#################################################################\n",
    "# Inputs: arr1 and arr2 have shape [batch_size, hidden_sizes[-1]]\n",
    "# Output: return tensor of shape [batch_size, ], the cosine \n",
    "#         similarity betwwen arr1 and arr2\n",
    "# \n",
    "# Hint: use tf.l2_normalize, tf.mul, tf.reduce_sum\n",
    "#################################################################\n",
    "def cosine_similarity(arr1, arr2):\n",
    "    ###################################\n",
    "    ####     PUT YOUR CODE HERE    ####\n",
    "    ###################################\n",
    "    \n",
    "    \n",
    "#################################################################\n",
    "# Inputs: mlp_args is a dictionary of arguments to the mlp() \n",
    "#         function. \n",
    "#         Example: mlp_args = {'hidden_sizes':[64, 64, 32]}\n",
    "#################################################################\n",
    "def build_model(mlp_args):\n",
    "    with tf.Graph().as_default() as g:\n",
    "        with tf.device(\"/cpu:0\"):\n",
    "            x1 = tf.placeholder(tf.float32, [None, 784])\n",
    "            x2 = tf.placeholder(tf.float32, [None, 784])\n",
    "            y = tf.placeholder(tf.float32, [None])\n",
    "\n",
    "            with tf.variable_scope(\"siamese\") as var_scope:\n",
    "                x_repr1 = mlp(x1, **mlp_args)  # hidden representation of x1\n",
    "                var_scope.reuse_variables()    # weight sharing! \n",
    "                x_repr2 = mlp(x2, **mlp_args)  # hidden representation of x2\n",
    "                logits = cosine_similarity(x_repr1, x_repr2)  # similarity\n",
    "                \n",
    "            ###################################\n",
    "            ####     PUT YOUR CODE HERE    ####\n",
    "            ###################################\n",
    "            \n",
    "            # define scalar: loss \n",
    "            # define vector: y_prob as sigmoid(cosine_similarity)\n",
    "            # define vector: y_pred as sign(cosine_similarity)\n",
    "            # define scalar: accuracy as the fraction of correct predictions\n",
    "            \n",
    "    return {'graph': g, 'inputs': [x1, x2, y], 'pred': y_pred, 'logits': logits,\n",
    "            'prob': y_prob, 'loss': loss, 'accuracy': accuracy}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code segment prepares minibatch training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# data preparation\n",
    "def mnist_siamese_dataset_iterator(batch_size, dataset_name):\n",
    "    assert dataset_name in ['train', 'test']\n",
    "    assert batch_size > 0 or batch_size == -1 # -1 for entire dataset\n",
    "    mnist = input_data.read_data_sets('./datasets/mnist/', one_hot=True)\n",
    "    dataset = getattr(mnist, dataset_name)\n",
    "    \n",
    "    while True:\n",
    "        if batch_size > 0:\n",
    "            X1, y1 = dataset.next_batch(batch_size)\n",
    "            X2, y2 = dataset.next_batch(batch_size)\n",
    "            y = np.argmax(y1, axis=1) == np.argmax(y2, axis=1)\n",
    "            yield X1, X2, y\n",
    "        else:\n",
    "            X1 = dataset.images\n",
    "            idx = np.arange(len(X1))\n",
    "            np.random.shuffle(idx)\n",
    "            X2 = X1[idx]\n",
    "            y1 = dataset.labels\n",
    "            y2 = y1[idx]\n",
    "            y = np.argmax(y1, axis=1) == np.argmax(y2, axis=1)\n",
    "            yield X1, X2, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the training.\n",
    "\n",
    "Note: if you have a good GPU, you could change `tf.device(\"/cpu:0\")` to `tf.device(\"/gpu:0\")` in your **`build_model()`** function to speed up training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    from itertools import izip as zip\n",
    "except ImportError:\n",
    "    print('This is Python 3')\n",
    "\n",
    "def run_training(model_dict, train_data_iterator, test_full_iter, \n",
    "                 train_full_iter, n_iter=1000, print_every=100):\n",
    "    with model_dict['graph'].as_default():\n",
    "        optimizer = tf.train.GradientDescentOptimizer(learning_rate = 0.1)\n",
    "        train_op = optimizer.minimize(model_dict['loss'])\n",
    "        with tf.Session() as sess:\n",
    "            sess.run(tf.global_variables_initializer())\n",
    "            for iter_i, data_batch in zip(range(n_iter), train_data_iterator):\n",
    "                batch_feed_dict = dict(zip(model_dict['inputs'], data_batch))\n",
    "                sess.run(train_op, feed_dict=batch_feed_dict)\n",
    "                if iter_i % print_every == 0:\n",
    "                    print_zip_iter = zip([test_full_iter, train_full_iter], ['test', 'train'])\n",
    "                    for data_iterator, data_name in print_zip_iter:\n",
    "                        test_batch = next(data_iterator)\n",
    "                        batch_feed_dict = dict(zip(model_dict['inputs'], test_batch))\n",
    "                        to_compute = [model_dict['accuracy'], model_dict['loss']]\n",
    "                        acc_value, loss_val = sess.run(to_compute, batch_feed_dict)\n",
    "                        fmt = (iter_i, acc_value, loss_val)\n",
    "                        print(data_name, 'iteration %d\\t accuracy: %.3f, loss: %.3f'%fmt)\n",
    "\n",
    "train_data_iterator = mnist_siamese_dataset_iterator(100, 'train')\n",
    "test_full_iter = mnist_siamese_dataset_iterator(-1, 'test')\n",
    "train_full_iter = mnist_siamese_dataset_iterator(-1, 'train')\n",
    "\n",
    "mlp_args = {'hidden_sizes':[64, 64, 32]}\n",
    "model = build_model(mlp_args)\n",
    "run_training(model, train_data_iterator, test_full_iter, train_full_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Model Variants\n",
    "\n",
    "(30 points)\n",
    "\n",
    "Tasks:\n",
    " \n",
    " 1. Create an improved version of the model in part 2. You are welcome to use techniques covered in class, including: increasing network depth, varying activation functions, learning rates, optimizers, cost functions, regularization etc. \n",
    " \n",
    " 2. Report the test accuracy.\n",
    "\n",
    "Note: you should **not** use convolutional layers for this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_build_model(mlp_args):\n",
    "    \n",
    "    ###################################\n",
    "    ####     PUT YOUR CODE HERE    ####\n",
    "    ###################################\n",
    "    \n",
    "    return {'graph': g, 'inputs': [x1, x2, y], 'pred': y_pred, 'logits': logits,\n",
    "            'prob': y_prob, 'loss': loss, 'accuracy': accuracy}\n",
    "\n",
    "my_mlp_args = {}  # you can define your own arguments\n",
    "model = my_build_model(my_mlp_args)\n",
    "run_training(model, train_data_iterator, test_full_iter, train_full_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Visualize learned features\n",
    "(25 points)\n",
    "\n",
    "Once a neural network is trained, we can think of the last layer as performing prediction, and the activations from layer before  become the input \"features\" to the predictor. If the neural network is performing good predictions, then this means that these activations encode useful features that are 1) representative of the input, and 2) discriminative for the prediction task. Activations at a selected layer could then be used as generic feature encodings of the input. \n",
    "\n",
    "We  expect a \"good\" feature encoding scheme to group similar inputs together in the feature encoding space. To check that, we can visualize the features in 2-dimensional space and check if similar examples (in this case, sharing the same labels) are close together. \n",
    "\n",
    "Tasks:\n",
    "\n",
    "1. In TensorFlow, extract 32-dimensional features from the **distance layer** of your trained Siamese network, for the MNIST **test set**.\n",
    "2. Reduce the dimensionality of your deep features to 2 using [t-SNE embedding](https://lvdmaaten.github.io/tsne/). You may use [this fast implementation](https://github.com/lvdmaaten/bhtsne/) with correct attribution. You could alternatively use [sklearn.manifold.TSNE](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html), but please take note that it might be slow.\n",
    "3. Visualize the 2-dimensional embeddings. If your extracted features are good, data points representing a specific digit should appear within a compact cluster. In the example below, each color corresponds to a digit class.\n",
    "\n",
    " <img src=\"tsne.png\" style=\"width:480px;\">\n",
    " \n",
    "For this part, the starter code is in minimalist fashion. Good luck!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def visualize(features, labels):\n",
    "    ###################################\n",
    "    ####     PUT YOUR CODE HERE    ####\n",
    "    ###################################"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}