DeepBugs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DeepBugs.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNQMRJezRIP46CCEx8ouxt1"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQ_cefu2VZMX",
        "colab_type": "text"
      },
      "source": [
        "# A Simple, DeepBugs-like Deep Learning-Based Bug Detector\n",
        "\n",
        "## Goal\n",
        "\n",
        "This interactive tutorial shows how to learn a simple, [DeepBugs](https://github.com/michaelpradel/DeepBugs)-like bug detector that finds [argument swapping bugs](http://software-lab.org/publications/oopsla2017.pdf). This kind of bug occurs when accidentally swapped the arguments passed to a function, e.g., writing `setPoint(y, x)` instead of `setPoint(x, y)`.\n",
        "\n",
        "\n",
        "## Loading the Data\n",
        "\n",
        "Get a small training data set for finding swapped function arguments:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybAlQ2RfV5R1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "1285bf2f-5512-4148-e68e-2332fb9bafaa"
      },
      "source": [
        "!gdown https://drive.google.com/uc?id=1w7KZ5ugVfcz4RGsY2feSalH9xpa_JFLh\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1w7KZ5ugVfcz4RGsY2feSalH9xpa_JFLh\n",
            "To: /content/DeepBugs_data.tar.gz\n",
            "20.1MB [00:00, 64.2MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwQ1KhLQa5tL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tar -xzf DeepBugs_data.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lPDiRt6cToAh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        },
        "outputId": "ced9e43f-5a85-4267-fe86-f3c0a928d925"
      },
      "source": [
        "!ls DeepBugs_data/calls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "calls_1523347143670.json  calls_1523347146998.json  calls_1523347147613.json\n",
            "calls_1523347145433.json  calls_1523347147049.json  calls_1523347147639.json\n",
            "calls_1523347145539.json  calls_1523347147069.json\n",
            "calls_1523347145593.json  calls_1523347147513.json\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XmrNZLM1Sih-",
        "colab_type": "text"
      },
      "source": [
        "The data are function calls extracted from open-source JavaScript code. Let's read the JSON data into our Python-based learning code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CiewuASiYCDa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import json\n",
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers.core import Dense, Dropout"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9tIjdWO1ZjCC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c9d8ac83-776a-4e14-9bc1-8b501ad12e24"
      },
      "source": [
        "calls = []\n",
        "for file in os.listdir(\"DeepBugs_data/calls\"):\n",
        "  with open(os.path.join(\"DeepBugs_data/calls\", file)) as fp:\n",
        "    calls.extend(json.load(fp))\n",
        "\n",
        "print(f\"Have read {len(calls)} function calls\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Have read 28005 function calls\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P4UtOgb6TZFo",
        "colab_type": "text"
      },
      "source": [
        "We'll also use a pre-trained embedding of code tokens. It's a Word2Vec model trained on tokenized JavaScript code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bHTH0jNwc59S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "85d0daa2-6ea0-40ff-93bc-1903e182306d"
      },
      "source": [
        "with open(\"DeepBugs_data/token_to_vector.json\") as fp:\n",
        "  token_to_vector = json.load(fp)\n",
        "\n",
        "print(f\"Have loaded {len(token_to_vector)} token embeddings.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Have loaded 9930 token embeddings.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q8_3_tLwcoXP",
        "colab_type": "text"
      },
      "source": [
        "## Preparing the Data: From Code Snippets to Vectors\n",
        "\n",
        "We will train a model that distinguishes correct from buggy function calls. To this end, we prepare positive (i.e., correct) and negative (i.e., buggy) code examples. The positive examples are function calls as found in the open-source code (assuming that most code is correct). We create negative examples by swapping function arguments, i.e., artificially creating buggy code.\n",
        "\n",
        "The following also represents each function call as a vector. The vector representation consists of the token embeddings of\n",
        " * the called function, i.e., the callee,\n",
        " * the first argument, and\n",
        " * the second argument."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b_oiHx4HcArX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "7520c3de-c31d-49c9-e785-15a084a233a5"
      },
      "source": [
        "xs = []   # Inputs given to the model: Each element is\n",
        "          #   the vector representation of a function call.\n",
        "ys = []   # Outputs expected from the model: For each\n",
        "          #   call, predict the probability that it's buggy.\n",
        "\n",
        "for call in calls:\n",
        "  if (call[\"callee\"] in token_to_vector and\n",
        "      call[\"arguments\"][0] in token_to_vector and\n",
        "      call[\"arguments\"][1] in token_to_vector):\n",
        "    callee_vec = token_to_vector[call[\"callee\"]]\n",
        "    arg1_vec = token_to_vector[call[\"arguments\"][0]]\n",
        "    arg2_vec = token_to_vector[call[\"arguments\"][1]]\n",
        "\n",
        "    # Positive, i.e., correct example\n",
        "    x_correct = callee_vec + arg1_vec + arg2_vec\n",
        "    # Negative, i.e., buggy example\n",
        "    x_buggy = callee_vec + arg2_vec + arg1_vec\n",
        "\n",
        "    xs.append(x_correct)\n",
        "    ys.append(0)  # Probability that buggy is 0\n",
        "    xs.append(x_buggy)\n",
        "    ys.append(1)  # Probability that buggy is 1\n",
        "\n",
        "# Split into training and validation data\n",
        "nb_training = int(0.9*len(xs))\n",
        "xs_training = np.array(xs[:nb_training])\n",
        "ys_training = np.array(ys[:nb_training])\n",
        "xs_validation = np.array(xs[nb_training:])\n",
        "ys_validation = np.array(ys[nb_training:])\n",
        "\n",
        "print(f\"{len(xs_training)} training examples\")\n",
        "print(f\"{len(xs_validation)} validation examples\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "21592 training examples\n",
            "2400 validation examples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T5aQ9C3rWyyF",
        "colab_type": "text"
      },
      "source": [
        "## Training the Model\n",
        "\n",
        "We train a simple feedforward model that takes the vector representation of a call and predicts the probability that the call is buggy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lFoC7_ZjeT3S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 219
        },
        "outputId": "852274a6-5927-4426-e14a-35036de9f483"
      },
      "source": [
        "x_length = len(xs[0])\n",
        "model = Sequential()\n",
        "model.add(Dropout(0.2, input_shape=(x_length,)))\n",
        "model.add(Dense(200, input_dim=x_length, activation=\"relu\", kernel_initializer='normal'))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(1, activation=\"sigmoid\", kernel_initializer='normal'))\n",
        "\n",
        "model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])\n",
        "model.fit(xs_training, ys_training, batch_size=100, epochs=5, verbose=1)        "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "21592/21592 [==============================] - 1s 49us/step - loss: 0.5058 - accuracy: 0.7411\n",
            "Epoch 2/5\n",
            "21592/21592 [==============================] - 1s 45us/step - loss: 0.3610 - accuracy: 0.8311\n",
            "Epoch 3/5\n",
            "21592/21592 [==============================] - 1s 44us/step - loss: 0.2969 - accuracy: 0.8650\n",
            "Epoch 4/5\n",
            "21592/21592 [==============================] - 1s 44us/step - loss: 0.2533 - accuracy: 0.8883\n",
            "Epoch 5/5\n",
            "21592/21592 [==============================] - 1s 44us/step - loss: 0.2282 - accuracy: 0.8991\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.callbacks.History at 0x7f83f3af9c88>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w_wARBIyive8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "6f2a81d3-2c07-41df-942f-f1131887b4a7"
      },
      "source": [
        "validation_stats = model.evaluate(xs_validation, ys_validation)\n",
        "print(f\"Validation accuracy: {validation_stats[1]}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2400/2400 [==============================] - 0s 36us/step\n",
            "Validation accuracy: 0.8383333086967468\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3FvUzJ15XEMR",
        "colab_type": "text"
      },
      "source": [
        "## Using the Learned Bug Detection Model\n",
        "\n",
        "Once trained, we can query the model with a given function call. In a full implementation, the model would reason about calls extracted from JavaScript code. Here, we simply give the callee and arguments as a string:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1KnyjO-ojXh0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4eab28b8-5d8f-4b35-ddf4-41325e6893db"
      },
      "source": [
        "# Function call: setTimeout(delay, fn)\n",
        "callee = \"ID:setTimeout\"  # Prefix \"ID:\" is to indicate that it's an identifier.\n",
        "arg1 = \"ID:delay\"\n",
        "arg2 = \"ID:fn\"\n",
        "\n",
        "x = token_to_vector[callee] + token_to_vector[arg1] + token_to_vector[arg2]\n",
        "xs = np.array([x])\n",
        "\n",
        "buggy_probabilities = model.predict(xs)\n",
        "print(f\"Call is buggy with probability {str(round(buggy_probabilities[0][0], 4))}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Call is buggy with probability 0.9904\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ZGaQQsdXg6P",
        "colab_type": "text"
      },
      "source": [
        "The above call is indeed buggy, because `setTimeout`, one of the built-in functions in JavaScript, expects a function as the first argument and the delay after which to call the function as the second argument."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jA5mAAwgXxn-",
        "colab_type": "text"
      },
      "source": [
        "## Exercises\n",
        "\n",
        "A few questions and tasks to check your understanding of this code. Modify the above code to answer them.\n",
        " * Inspect the provided JSON data. What other properties of function calls are there that may be useful for learning a bug detector?\n",
        " * How long is the vector that embeds a single token?\n",
        " * Does the model get any better when training for more epochs?\n",
        " * If you fix the call to `setTimeout`, i.e., pass the arguments in the correct order, what's the predicted probability that the code is buggy?"
      ]
    }
  ]
}