solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AGT Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task at hand is a multi-class classification problem, for which both a training and a test (validation) set are provided as csv files, 'train.csv' and 'test.csv' accordingly.\n",
    "What we ask is that you work on this classification task by building a classifier using only the training data, with the goal of achieving the best performance possible on the test data, classifying as correctly as possible the 'label' variable.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "The goal of this script is to predict the label of the unseen data, i.e., test data, by using machine learning methods. In the following section, I will go through the following steps:\n",
    "- Reading the dataset\n",
    "- Generating training, test, and validation sets\n",
    "- Cleaning the data\n",
    "    - converting string values to integer values\n",
    "    - replacing the missing features \n",
    "    - standardization of the features\n",
    "- Training an SVM and evaluating the classification accuracy with all the given features\n",
    "- Applying Principal Component Analysis (PCA) to reduce the dimensionality of the features\n",
    "- Trainng an SVM on the dimensionality reduced features and evaluating the classification accuracy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from csv import reader\n",
    "import numpy as np\n",
    "from sklearn import preprocessing, svm\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.preprocessing import Imputer\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.svm import SVC\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading dataset and cleaning data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This function is used for loading data from a CSV file.\n",
    "def load_csv(filename):\n",
    "    \"\"\"\n",
    "    Load data from csv file.\n",
    "    :param filename: csv file path\n",
    "    :return: data matrix\n",
    "    \"\"\"\n",
    "    dataset = list()\n",
    "    with open(filename, 'r') as file:\n",
    "        csv_reader = reader(file)\n",
    "        for row in csv_reader:\n",
    "            if not row:\n",
    "                continue\n",
    "            dataset.append(row)\n",
    "\n",
    "    return np.array(dataset)\n",
    "\n",
    "# This function splits data into two parts.\n",
    "# A number of percentage is used to indicate how much data is put into the first part.\n",
    "def split_data(X, y, per=0.1):\n",
    "    \"\"\"\n",
    "    Separated data into two parts with the specified percentage.\n",
    "    :param X: features\n",
    "    :param y: label\n",
    "    :param per: percentage\n",
    "    :return: the separated data\n",
    "    \"\"\"\n",
    "    length = int(len(X) * per)\n",
    "\n",
    "    return X[0:length, :], y[0:length], X[length:, :], y[length:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions are used to convert string values into integer values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This function tests if a given value is a number or not.\n",
    "def is_number(s):\n",
    "    return is_int(s) or is_float(s)\n",
    "\n",
    "def is_int(s):\n",
    "    try:\n",
    "        int(s)\n",
    "        return True\n",
    "    except ValueError:\n",
    "        return False\n",
    "\n",
    "\n",
    "def is_float(s):\n",
    "    try:\n",
    "        float(s)\n",
    "        return True\n",
    "    except ValueError:\n",
    "        return False\n",
    "    \n",
    "\n",
    "# This function extracts the indices of columns where the values in the columns are string.\n",
    "def get_str_column_indices(row):\n",
    "    \"\"\"\n",
    "    Get the indices of columns which contain only string values.\n",
    "    :param row: data row\n",
    "    :return: column indices\n",
    "    \"\"\"\n",
    "    column_indices = []\n",
    "    for i, value in enumerate(row):\n",
    "        if not is_number(value):\n",
    "            column_indices.append(i)\n",
    "    \n",
    "    return column_indices\n",
    "\n",
    "# This function converts string values into integer values.\n",
    "def str_column_to_int(dataset, column):\n",
    "    \"\"\"\n",
    "    For each column of a matrix (dataset), change string values to integer values.\n",
    "    :param dataset: data matrix\n",
    "    :param column: column index\n",
    "    :return: a dictionary\n",
    "    \"\"\"\n",
    "    values = [row[column] for row in dataset]\n",
    "    unique = set(values)\n",
    "    lookup = dict()\n",
    "    for i, value in enumerate(unique):\n",
    "        lookup[value] = i\n",
    "    for row in dataset:\n",
    "        row[column] = lookup[row[column]]\n",
    "    \n",
    "    return lookup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The next function first loads data from a CSV file. Then it cleans the data by applying the following steps:\n",
    "    - Removing the header of the content.\n",
    "    - Removing the first and second columns, since they only contain the numbers of samples. The number is not useful for classification.\n",
    "    - Dividing the data into X(features) and y(labels).\n",
    "    - Converting string values in X into integer values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_data(train_filename, shuffle=False):\n",
    "    \"\"\"\n",
    "    Load data from csv file. Then clean up the data.\n",
    "    :param filename: csv file path\n",
    "    :return: X (features), y (labels)\n",
    "    \"\"\"\n",
    "    \n",
    "    # read data from csv\n",
    "    dataset = load_csv(filename)\n",
    "\n",
    "    # remove header, the first, and second columns\n",
    "    dataset = dataset[1:, 2:]\n",
    "\n",
    "    # shuffle data\n",
    "    if shuffle:\n",
    "        np.random.shuffle(dataset)\n",
    "\n",
    "    # split data into X(features) and y(labels)\n",
    "    X = dataset[:, 0:len(dataset[0])-2]\n",
    "    y = dataset[:, len(dataset[0])-1]\n",
    "\n",
    "    # convert string to integer\n",
    "    str_column_indices = get_str_column_indices(X[0])\n",
    "    for str_column in str_column_indices:\n",
    "        str_column_to_int(X, str_column)\n",
    "\n",
    "    # convert string to float and replace missing value (blank) with NaN\n",
    "    i = 0\n",
    "    for i, row in enumerate(X):\n",
    "        for j, value in enumerate(row):\n",
    "            if(len(value) == 0):\n",
    "                X[i, j] = np.nan\n",
    "            else:\n",
    "                X[i, j] = float(value)\n",
    "\n",
    "    X = np.asarray(X, dtype=float)\n",
    "\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating cleaned training, test, and validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "loading data ...\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAFkCAYAAACq4KjhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFaFJREFUeJzt3X+sX3Wd5/HX28UWYbbFgaWMwxB1GEnN7LDbsiBxYJxg\nEDWL7rqZpZmGCH8YVjSkm01cEx0Yye5kNFKCSuIfxpEF2RBcIxqgOowy8kOaKTiOY8HowlQFChdc\nSmDLz8/+8T3Vy91Pod9v773f297HIzmpPed8T9/3G2KfPed8v6daawEAmOtV0x4AAFiaRAIA0CUS\nAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQNVYkVNVHq2prVe2qqp1V9dWqetOcfb5Y\nVS/OWW6cs8/KqvpcVc1U1ZNVdX1VHT0fPxAAMD/GPZNwWpLPJDklyduTvDrJN6vqNXP2uynJmiTH\nDMuGOdsvT/LuJO9LcnqS1yX5ypizAAALqPbnAU9VdVSSR5Kc3lq7bVj3xSSrW2v/fi+vWZXk0STn\ntNa+Oqw7Icn2JG9prW2deCAAYN7s7z0JRyRpSR6fs/5tw+WIe6vqyqr6zVnb1ic5JMkte1a01u5L\nsiPJqfs5DwAwTw6Z9IVVVRldNrittfajWZtuyujSwf1JfjfJXyS5sapObaPTFsckeba1tmvOIXcO\n23p/1pFJ3pHkgSS7J50ZAJahQ5O8PsmW1tpj47xw4khIcmWSNyd56+yVrbXrZv32H6vqH5L8NMnb\nknx7wj/rHUmumfC1AEDyp0m+PM4LJoqEqvpsknclOa219tDL7dtau7+qZpIcn1EkPJxkRVWtmnM2\nYc2wreeBJLn66quzdu3aSUZmAps2bcrmzZunPcay4j1ffN7zxec9X1zbt2/Pxo0bk+Hv0nGMHQlD\nILwnyR+11nbsw/7HJjkyyZ6Y2Jbk+SRnJJl94+JxSe7cy2F2J8natWuzbt26cUdmQqtXr/Z+LzLv\n+eLzni8+7/nUjH25fqxIqKorM/o449lJnqqqNcOmJ1pru6vq8CQXZ3RPwsMZnT34yyQ/TrIlSVpr\nu6rqC0kuq6pfJnkyyRVJbvfJBgBYOsY9k3BBRp9m+M6c9ecluSrJC0n+IMm5GX3y4cGM4uDPWmvP\nzdp/07Dv9UlWJrk5yYVjzgIALKCxIqG19rIfmWyt7U5y1j4c55kkHx4WAGAJ8uwG9mrDhrlflMlC\n854vPu/54vOeHzj26xsXF0tVrUuybdu2bW52AYAx3H333Vm/fn2SrG+t3T3Oa51JAAC6RAIA0CUS\nAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJ\nAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkAQJdIAAC6RAIA0CUSAIAukQAAdIkE\nAKDrkGkPAMDysmPHjszMzEx7jGVj+/btE79WJACwaHbs2JETTlib3bufnvYo7AORAMCimZmZGQLh\n6iRrpz3OMnFjko9P9EqRAMAUrE2ybtpDLBOTX25w4yIA0CUSAIAukQAAdIkEAKBLJAAAXSIBAOgS\nCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAECXSAAAusaKhKr6aFVtrapdVbWz\nqr5aVW/q7PeJqnqwqp6uqm9V1fFztq+sqs9V1UxVPVlV11fV0fv7wwAA82fcMwmnJflMklOSvD3J\nq5N8s6pes2eHqvpIkg8l+UCSk5M8lWRLVa2YdZzLk7w7yfuSnJ7kdUm+MuHPAAAsgEPG2bm19q7Z\nv6+q9yd5JMn6JLcNqy9Kcmlr7RvDPucm2ZnkvUmuq6pVSc5Pck5r7dZhn/OSbK+qk1trWyf/cQCA\n+bK/9yQckaQleTxJquoNSY5JcsueHVpru5LcleTUYdVJGcXJ7H3uS7Jj1j4AwJRNHAlVVRldNrit\ntfajYfUxGUXDzjm77xy2JcmaJM8O8bC3fQCAKRvrcsMcVyZ5c5K3ztMsr2jTpk1ZvXr1S9Zt2LAh\nGzZsWKwRAGAJu3ZYZvv5xEebKBKq6rNJ3pXktNbaQ7M2PZykMjpbMPtswpok98zaZ0VVrZpzNmHN\nsG2vNm/enHXr1k0yMgAsAxuGZbZrkmyc6GhjX24YAuE9Sf64tbZj9rbW2v0Z/UV/xqz9V2X0aYg7\nhlXbkjw/Z58TkhyX5M5x5wEAFsZYZxKq6sqMEuXsJE9V1Zph0xOttd3D/748yceq6idJHkhyaUbn\nOr6WjG5krKovJLmsqn6Z5MkkVyS53ScbAGDpGPdywwUZ3Zj4nTnrz0tyVZK01j5ZVYcl+XxGn374\nbpJ3ttaenbX/piQvJLk+ycokNye5cNzhAYCFM+73JOzT5YnW2iVJLnmZ7c8k+fCwAABLkGc3AABd\nIgEA6BIJAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkAQJdIAAC6RAIA0CUSAIAu\nkQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAECX\nSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkAQJdIAAC6RAIA0CUSAIAukQAAdIkEAKBL\nJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAEDX2JFQVadV1Q1V\n9YuqerGqzp6z/YvD+tnLjXP2WVlVn6uqmap6sqqur6qj9/eHAQDmzyRnEg5P8v0kH0zS9rLPTUnW\nJDlmWDbM2X55kncneV+S05O8LslXJpgFAFggh4z7gtbazUluTpKqqr3s9kxr7dHehqpaleT8JOe0\n1m4d1p2XZHtVndxa2zruTADA/FuoexLeVlU7q+reqrqyqn5z1rb1GcXJLXtWtNbuS7IjyakLNA8A\nMKaxzyTsg5syunRwf5LfTfIXSW6sqlNbay2jyw/PttZ2zXndzmEbALAEzHsktNaum/Xbf6yqf0jy\n0yRvS/Lt/Tn2pk2bsnr16pes27BhQzZsmHvLAwAsR9cOy2w/n/hoC3Em4SVaa/dX1UyS4zOKhIeT\nrKiqVXPOJqwZtu3V5s2bs27duoUbFgAOaBvy/39W4JokGyc62oJ/T0JVHZvkyCQPDau2JXk+yRmz\n9jkhyXFJ7lzoeQCAfTP2mYSqOjyjswJ7Ptnwxqo6Mcnjw3JxRvckPDzs95dJfpxkS5K01nZV1ReS\nXFZVv0zyZJIrktzukw0AsHRMcrnhpIwuG7Rh+fSw/ksZfXfCHyQ5N8kRSR7MKA7+rLX23KxjbEry\nQpLrk6zM6COVF04wCwCwQCb5noRb8/KXKc7ah2M8k+TDwwIALEGe3QAAdIkEAKBLJAAAXSIBAOha\n8C9Tmk+PPfZYdu7cOe0xlo3Xvva1WbFixbTHAGBKDqhIOPPMM6c9wrJy1ln/NjfddMO0xwBgSg6o\nSEg2Z/TMKBbe/8gPfuC7rQCWswMsEk5P4tkNi+OuJCIBYDlz4yIA0CUSAIAukQAAdIkEAKBLJAAA\nXQfYpxvg4LZjx47MzMxMe4xl5aijjspxxx037TFgSRIJsETs2LEjJ5ywNrt3Pz3tUZaVQw89LPfd\nt10oQIdIgCViZmZmCISrk6yd9jjLxPbs3r0xMzMzIgE6RAIsOWvjS8OApcCNiwBAl0gAALpEAgDQ\nJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAECXSAAAukQCANDl2Q3Asrd9+/Zpj7BseK8PLCIBWMYe\nSvKqbNy4cdqDwJIkEoBl7P8keTEez72Ybkzy8WkPwT4SCQAez72IXG44kLhxEQDociaBvXr++edz\n9913T3uMZcMNXcBSIxLYiyfyyCMPZ/369dMeBIApEQnsxdNJXogbuhaTG7qApUUk8Arc0LV4XG4A\nlhY3LgIAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAECXSAAAukQC\nANAlEgCArrEjoapOq6obquoXVfViVZ3d2ecTVfVgVT1dVd+qquPnbF9ZVZ+rqpmqerKqrq+qo/fn\nBwEA5tckZxIOT/L9JB9M0uZurKqPJPlQkg8kOTnJU0m2VNWKWbtdnuTdSd6X5PQkr0vylQlmAQAW\nyCHjvqC1dnOSm5Okqqqzy0VJLm2tfWPY59wkO5O8N8l1VbUqyflJzmmt3Trsc16S7VV1cmtt60Q/\nCQAwr+b1noSqekOSY5Lcsmdda21XkruSnDqsOimjOJm9z31JdszaBwCYsvm+cfGYjC5B7Jyzfuew\nLUnWJHl2iIe97QMATNnYlxuma1OS1XPWbRgWAFjurh2W2X4+8dHmOxIeTlIZnS2YfTZhTZJ7Zu2z\noqpWzTmbsGbY9jI2J1k3X7MCwEGm9w/na5JsnOho83q5obV2f0Z/0Z+xZ91wo+IpSe4YVm1L8vyc\nfU5IclySO+dzHgBgcmOfSaiqw5Mcn9EZgyR5Y1WdmOTx1trPMvp448eq6idJHkhyaUbnOr6WjG5k\nrKovJLmsqn6Z5MkkVyS53ScbAGDpmORyw0lJvp3RDYotyaeH9V9Kcn5r7ZNVdViSzyc5Isl3k7yz\ntfbsrGNsSvJCkuuTrMzoI5UXTvQTAAALYpLvSbg1r3CZorV2SZJLXmb7M0k+PCwAwBLk2Q0AQJdI\nAAC6RAIA0CUSAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEsk\nAABdIgEA6BIJAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkAQJdIAAC6RAIA0CUS\nAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJ\nAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkAQJdIAAC6RAIA0DXvkVBVF1fVi3OW\nH83Z5xNV9WBVPV1V36qq4+d7DgBg/yzUmYQfJlmT5Jhh+cM9G6rqI0k+lOQDSU5O8lSSLVW1YoFm\nAQAmcMgCHff51tqje9l2UZJLW2vfSJKqOjfJziTvTXLdAs0DAIxpoc4k/F5V/aKqflpVV1fV7yRJ\nVb0hozMLt+zZsbW2K8ldSU5doFkAgAksRCR8L8n7k7wjyQVJ3pDkb6vq8IwCoWV05mC2ncM2AGCJ\nmPfLDa21LbN++8Oq2prkn5L8SZJ79+/om5KsnrNuw7AAwHJ37bDM9vOJj7ZQ9yT8Smvtiar6cZLj\nk3wnSWV0U+PsswlrktzzykfbnGTdvM8IAAeH3j+cr0mycaKjLfj3JFTVb2QUCA+21u5P8nCSM2Zt\nX5XklCR3LPQsAMC+m/czCVX1qSRfz+gSw28n+fMkzyX5n8Mulyf5WFX9JMkDSS7N6FzI1+Z7FgBg\ncgtxueHYJF9OcmSSR5PcluQtrbXHkqS19smqOizJ55MckeS7Sd7ZWnt2AWYBACa0EDcuvuJdhK21\nS5JcMt9/NgAwfzy7AQDoEgkAQJdIAAC6RAIA0CUSAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gA\nALpEAgDQJRIAgC6RAAB0iQQAoEskAABdIgEA6BIJAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQA\nAF0iAQDoEgkAQJdIAAC6RAIA0CUSAIAukQAAdIkEAKBLJAAAXSIBAOgSCQBAl0gAALpEAgDQJRIA\ngC6RAAB0iQQAoEskAABdIgEA6BIJAECXSAAAukQCANAlEgCALpEAAHSJBACgSyQAAF0iAQDoEgkA\nQJdIAAC6phoJVXVhVd1fVf+3qr5XVf9mmvMAAL82tUioqv+Y5NNJLk7yr5P8fZItVXXUtGYCAH5t\nmmcSNiX5fGvtqtbavUkuSPJ0kvOnOBMAMJhKJFTVq5OsT3LLnnWttZbkr5OcOo2ZAICXOmRKf+5R\nSf5Zkp1z1u9MckJn/0NHv/yvJH+3kHPxK/cOv96YZPs0B1lGbh9+9Z4vHu/54vOeL7497/mev0v3\n3bQiYVyvH/3y36Y6xPL08WkPsAx5zxef93zxec+n4PVJ7hjnBdOKhJkkLyRZM2f9miQPd/bfkuRP\nkzyQZPeCTgYAB5dDMwqELeO+sEa3Aiy+qvpekrtaaxcNv68kO5Jc0Vr71FSGAgB+ZZqXGy5L8ldV\ntS3J1ow+7XBYkr+a4kwAwGBqkdBau274ToRPZHSZ4ftJ3tFae3RaMwEAvza1yw0AwNLm2Q0AQJdI\nAAC6DohI8CCoxVNVp1XVDVX1i6p6sarOnvZMB7uq+mhVba2qXVW1s6q+WlVvmvZcB7OquqCq/r6q\nnhiWO6rqrGnPtVxU1X8d/v/lsmnPcjCrqouH93n28qNxjrHkI8GDoBbd4RndRPrBJG5YWRynJflM\nklOSvD3Jq5N8s6peM9WpDm4/S/KRJOsy+or4v0nytapaO9WploHhH3kfyOj/y1l4P8zowwHHDMsf\njvPiJX/j4l6+T+FnGX2fwienOtxBrqpeTPLe1toN055lORkC+JEkp7fWbpv2PMtFVT2W5L+01r44\n7VkOVlX1G0m2JflPGX3l4j2ttf883akOXlV1cZL3tNbWTXqMJX0mwYOgWKaOyOgszuPTHmQ5qKpX\nVdU5GX1Py53Tnucg97kkX2+t/c20B1lGfm+4fPzTqrq6qn5nnBcv9Wc3jPsgKDigDWfKLk9yW2tt\nrGuHjKeqfj+jKDg0yZNJ/t3w2HoWwBBi/yrJSdOeZRn5XpL3J7kvyW8luSTJ31bV77fWntqXAyz1\nSIDl5sokb07y1mkPsgzcm+TEJKuT/IckV1XV6UJh/lXVsRnF79tba89Ne57lorU2+1kNP6yqrUn+\nKcmfJNmny2pLPRLGfRAUHLCq6rNJ3pXktNbaQ9Oe52DXWns+yf8efntPVZ2c5KKMrpczv9Yn+RdJ\n7h7OliWjs8SnV9WHkqxsS/0GuYNAa+2JqvpxkuP39TVL+p6EoTi3JTljz7rhP7AzMubjLmEpGwLh\nPUn+uLW2Y9rzLFOvSrJy2kMcpP46yb/M6HLDicPyd0muTnKiQFgcw42jxyfZ53+ELPUzCYkHQS2q\nqjo8o/+I9tT+G6vqxCSPt9Z+Nr3JDl5VdWWSDUnOTvJUVe05c/ZEa82j0RdAVf33JDdl9OTZf57R\no+j/KMmZ05zrYDVc/37JPTZV9VSSx1pr26cz1cGvqj6V5OsZXWL47SR/nuS5JNfu6zGWfCR4ENSi\nOynJtzO6u75l9B0VSfKlJOdPa6iD3AUZvdffmbP+vCRXLfo0y8PRGf03/VtJnkjygyRnuut+UTl7\nsPCOTfLlJEcmeTTJbUne0lp7bF8PsOS/JwEAmI4lfU8CADA9IgEA6BIJAECXSAAAukQCANAlEgCA\nLpEAAHSJBACgSyQAAF0iAQDoEgkAQNf/A8XMhf4G6++KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7efdfb0ecf60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('\\nloading data ...')\n",
    "\n",
    "# TODO: specific the directory of the csv files \n",
    "dir_path = '/media/ck/volume1/data/agt_challenge/'\n",
    "filename = dir_path + 'test.csv'\n",
    "X_test, y_test = load_data(filename)\n",
    "\n",
    "filename = dir_path + 'train.csv'\n",
    "X_train, y_train = load_data(filename, shuffle=True)\n",
    "\n",
    "# split data into training and validation sets.\n",
    "# 10% of the data is randomly chosen as validation set and 90% of the data is used for training.\n",
    "X_train, y_train, X_val, y_val = split_data(X_train, y_train, 0.1)\n",
    "\n",
    "# show number of samples per class\n",
    "y = y_train.astype(int)\n",
    "plt.hist(y.tolist(), range(min(y), max(y)+1))\n",
    "plt.show()\n",
    "\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Replacing missing features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "replacing missing features ...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('\\nreplacing missing features ...')\n",
    "\n",
    "imp = Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "imp.fit(X_train)\n",
    "\n",
    "X_train = imp.transform(X_train)\n",
    "X_val = imp.transform(X_val)\n",
    "X_test = imp.transform(X_test)\n",
    "\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data standardization\n",
    "The goal of standardization is to rescale the features, so that they will have the properties of a standard normal distribution with $\\mu = 0$ and $\\sigma = 1$. The scaled features are calculated as follows:\n",
    "$$z = \\frac{x- \\mu}{\\sigma}$$, where $x$ is the unscaled feature and $z$ is the scaled feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "standardization ...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('\\nstandardization ...')\n",
    "\n",
    "std_scale = preprocessing.StandardScaler().fit(X_train)\n",
    "\n",
    "X_train = std_scale.transform(X_train)\n",
    "X_val = std_scale.transform(X_val)\n",
    "X_test = std_scale.transform(X_test)\n",
    "\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reducing the dimensionality of features with PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pca(X_train, X_test, n_components = 200):\n",
    "    \"\"\"\n",
    "    Feature dimensionality reduction with PCA.\n",
    "    :param X_train: training features\n",
    "    :param X_test: test features\n",
    "    :param n_components: number or principal ccomponents\n",
    "    :return: dimensionality reduced training and test features\n",
    "    \"\"\"\n",
    "    pca = PCA(n_components=n_components).fit(X_train)\n",
    "    \n",
    "    X_train = pca.transform(X_train)\n",
    "    X_test = pca.transform(X_test)\n",
    "\n",
    "    return X_train, X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training an SVM and predicting the label of test data with the trained SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In this function, I first train an SVM on the training data, i.e., X_train (features) and y_train (labels).\n",
    "# Then the trained SVM is used to predict the labels for the test data.\n",
    "def classification(X_train, y_train, X_test, y_test, grid_search=False):\n",
    "    \"\"\"\n",
    "    Classification with SVM.\n",
    "    :param X_train: training features\n",
    "    :param y_train: training labels\n",
    "    :param X_test: test features\n",
    "    :param y_test: test labels\n",
    "    :return: predicted labels, classification accuracy\n",
    "    \"\"\"\n",
    "\n",
    "    print('training svm ...')\n",
    "    clf = svm.SVC()\n",
    "\n",
    "    if grid_search:\n",
    "        param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5], 'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1]}\n",
    "        clf = GridSearchCV(SVC(kernel='rbf', class_weight='balanced'), param_grid)\n",
    "\n",
    "    clf = clf.fit(X_train, y_train)\n",
    "\n",
    "    print('predicting labels with svm ...')\n",
    "    y_pred = clf.predict(X_test)\n",
    "\n",
    "    acc = np.sum(y_test == y_pred) / len(y_test)\n",
    "\n",
    "    print(classification_report(y_test, y_pred))\n",
    "    print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "    return y_pred, acc\n",
    "\n",
    "\n",
    "# This function first applies dimension reduction on the features with PCA.\n",
    "# Then it trains an SVM on the training data.\n",
    "# Finally it predict the labels of the test data with the trained SVM.\n",
    "def classification_pca(X_train, y_train, X_test, y_test, n_components, grid_search=False):\n",
    "    \"\"\"\n",
    "    Classification with SVM\n",
    "    :param X_train: training features\n",
    "    :param y_train: training labels\n",
    "    :param X_test: test features\n",
    "    :param y_test: test labels\n",
    "    :param n_components: number of components\n",
    "    :return: predicted labels, classification accuracy\n",
    "    \"\"\"\n",
    "    X_train_pca, X_test_pca = pca(X_train, X_test, n_components)\n",
    "    \n",
    "    y_pred, acc = classification(X_train_pca, y_train, X_test_pca, y_test, grid_search)\n",
    "\n",
    "    return y_pred, acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding optimal number of principla components for the classification\n",
    "\n",
    "The goal of this function is to get the optimal number of principal components in a given list.\n",
    "To find an optimal number of components, each number of components in a given list is used to reduce the dimensionality of the features. Then the dimension reduced features with the labels are used to train an SVM. \n",
    "The SVM is used to predict the labels of the validation data where the dimension of its features are also reduced with the PCA of the specified number of components. The number of components which achieves highest classification accuracy is considered as the optimal number of principal components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_n_components(X_train, y_train, X_val, y_val, n_components_list = [10, 50, 100, 200, 300]):\n",
    "    \"\"\"\n",
    "    Find optimal number of components of PCA.\n",
    "    :param X_train: training features\n",
    "    :param y_train: training labels\n",
    "    :param X_val: validation features\n",
    "    :param y_val: validation labels\n",
    "    :param n_components_list: a list contains the number of components of PCA\n",
    "    :return: the optimal number of components in the given list.\n",
    "    \"\"\"\n",
    "    best_acc = 0\n",
    "    best_n_components = 0\n",
    "    acc_list = []\n",
    "    for n_components in n_components_list:\n",
    "        y_pred, acc = classification_pca(X_train, y_train, X_val, y_val, n_components)\n",
    "        acc_list.append(acc)\n",
    "        print(\"number of components = %d accuracy = %f\" % (n_components, acc))\n",
    "        if acc > best_acc:\n",
    "            best_acc = acc\n",
    "            best_n_components = n_components\n",
    "\n",
    "    plt.plot(n_components_list, acc_list)\n",
    "    plt.show()\n",
    "\n",
    "    return best_n_components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting label of test data with all the given features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "evaluation ...\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.99      0.99       537\n",
      "          1       0.92      0.83      0.87       491\n",
      "          2       0.86      0.93      0.89       532\n",
      "          3       0.94      0.96      0.95       496\n",
      "          4       0.93      0.87      0.90       420\n",
      "          5       0.92      0.97      0.94       471\n",
      "\n",
      "avg / total       0.93      0.93      0.93      2947\n",
      "\n",
      "[[529   0   2   0   6   0]\n",
      " [  2 409  79   0   0   1]\n",
      " [  0  36 496   0   0   0]\n",
      " [  0   0   0 478  16   2]\n",
      " [  0   0   0  19 364  37]\n",
      " [  0   0   0  10   5 456]]\n",
      "accuracy on test set (all features): 0.927044 \n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('\\nevaluation ...')\n",
    "\n",
    "y_pred, acc = classification(X_train, y_train, X_test, y_test, grid_search=False)\n",
    "\n",
    "print(\"accuracy on test set (all features): %f \" % acc)\n",
    "\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting label of test data with dimensionality reduced features by applying PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "searching optimal number of principal components ...\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.96      0.87      0.91      1266\n",
      "          1       0.72      0.61      0.66      1168\n",
      "          2       0.77      0.72      0.74      1240\n",
      "          3       0.95      0.77      0.85      1104\n",
      "          4       0.49      0.95      0.64       877\n",
      "          5       0.93      0.72      0.81       962\n",
      "\n",
      "avg / total       0.81      0.77      0.78      6617\n",
      "\n",
      "[[1100   28    0    0  138    0]\n",
      " [  40  715  271    0  142    0]\n",
      " [   0  247  892    0  101    0]\n",
      " [   0    0    0  847  237   20]\n",
      " [   0    0    0   15  829   33]\n",
      " [   0    0    0   26  248  688]]\n",
      "number of components = 10 accuracy = 0.766359\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.88      0.93      1266\n",
      "          1       0.86      0.72      0.79      1168\n",
      "          2       0.84      0.80      0.82      1240\n",
      "          3       0.98      0.80      0.88      1104\n",
      "          4       0.50      0.97      0.66       877\n",
      "          5       0.95      0.73      0.83       962\n",
      "\n",
      "avg / total       0.87      0.81      0.83      6617\n",
      "\n",
      "[[1108    0    0    0  158    0]\n",
      " [   9  843  185    0  131    0]\n",
      " [   0  133  997    0  110    0]\n",
      " [   0    0    0  885  197   22]\n",
      " [   0    0    0   11  847   19]\n",
      " [   0    0    0    6  250  706]]\n",
      "number of components = 50 accuracy = 0.813964\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.93      0.96      1266\n",
      "          1       0.88      0.78      0.83      1168\n",
      "          2       0.87      0.86      0.87      1240\n",
      "          3       0.98      0.87      0.92      1104\n",
      "          4       0.63      0.96      0.76       877\n",
      "          5       0.93      0.83      0.87       962\n",
      "\n",
      "avg / total       0.89      0.87      0.87      6617\n",
      "\n",
      "[[1179    0    0    0   87    0]\n",
      " [  10  907  160    0   91    0]\n",
      " [   0  121 1067    0   52    0]\n",
      " [   0    0    0  960  106   38]\n",
      " [   0    0    0   10  845   22]\n",
      " [   0    0    0   10  158  794]]\n",
      "number of components = 100 accuracy = 0.869276\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.95      0.97      1266\n",
      "          1       0.90      0.84      0.87      1168\n",
      "          2       0.90      0.88      0.89      1240\n",
      "          3       0.98      0.90      0.94      1104\n",
      "          4       0.72      0.97      0.83       877\n",
      "          5       0.94      0.88      0.91       962\n",
      "\n",
      "avg / total       0.91      0.90      0.90      6617\n",
      "\n",
      "[[1201    0    0    0   65    0]\n",
      " [  12  985  127    0   44    0]\n",
      " [   1  115 1095    0   29    0]\n",
      " [   0    0    0  990   79   35]\n",
      " [   0    0    0    8  849   20]\n",
      " [   0    0    0   10  108  844]]\n",
      "number of components = 150 accuracy = 0.901315\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.96      0.97      1266\n",
      "          1       0.91      0.86      0.88      1168\n",
      "          2       0.90      0.91      0.90      1240\n",
      "          3       0.98      0.92      0.95      1104\n",
      "          4       0.79      0.98      0.87       877\n",
      "          5       0.95      0.90      0.92       962\n",
      "\n",
      "avg / total       0.93      0.92      0.92      6617\n",
      "\n",
      "[[1213    0    0    0   53    0]\n",
      " [   9 1006  128    0   24    1]\n",
      " [   1  100 1125    0   14    0]\n",
      " [   0    0    0 1015   57   32]\n",
      " [   0    0    0    4  858   15]\n",
      " [   0    0    0   12   81  869]]\n",
      "number of components = 200 accuracy = 0.919752\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.97      0.98      1266\n",
      "          1       0.91      0.86      0.89      1168\n",
      "          2       0.89      0.92      0.90      1240\n",
      "          3       0.98      0.93      0.96      1104\n",
      "          4       0.83      0.98      0.90       877\n",
      "          5       0.95      0.92      0.93       962\n",
      "\n",
      "avg / total       0.93      0.93      0.93      6617\n",
      "\n",
      "[[1225    0    0    0   41    0]\n",
      " [   8 1007  136    0   16    1]\n",
      " [   1   98 1137    0    4    0]\n",
      " [   0    0    0 1027   43   34]\n",
      " [   0    0    0    5  856   16]\n",
      " [   0    0    0   12   66  884]]\n",
      "number of components = 250 accuracy = 0.927308\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.97      0.98      1266\n",
      "          1       0.91      0.86      0.89      1168\n",
      "          2       0.89      0.92      0.90      1240\n",
      "          3       0.98      0.94      0.96      1104\n",
      "          4       0.86      0.98      0.92       877\n",
      "          5       0.95      0.93      0.94       962\n",
      "\n",
      "avg / total       0.93      0.93      0.93      6617\n",
      "\n",
      "[[1233    0    0    0   32    1]\n",
      " [   6 1009  142    0   10    1]\n",
      " [   1   95 1142    0    2    0]\n",
      " [   0    0    0 1034   36   34]\n",
      " [   0    0    0    5  858   14]\n",
      " [   0    0    0   11   60  891]]\n",
      "number of components = 300 accuracy = 0.931993\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.97      0.98      1266\n",
      "          1       0.91      0.86      0.88      1168\n",
      "          2       0.89      0.92      0.90      1240\n",
      "          3       0.98      0.94      0.96      1104\n",
      "          4       0.88      0.98      0.92       877\n",
      "          5       0.94      0.94      0.94       962\n",
      "\n",
      "avg / total       0.93      0.93      0.93      6617\n",
      "\n",
      "[[1234    0    0    0   31    1]\n",
      " [   8 1004  148    0    7    1]\n",
      " [   0   97 1141    0    2    0]\n",
      " [   0    0    0 1035   33   36]\n",
      " [   0    0    0    5  856   16]\n",
      " [   0    0    0   13   48  901]]\n",
      "number of components = 350 accuracy = 0.932598\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.98      0.98      1266\n",
      "          1       0.91      0.85      0.88      1168\n",
      "          2       0.88      0.92      0.90      1240\n",
      "          3       0.98      0.94      0.96      1104\n",
      "          4       0.89      0.97      0.93       877\n",
      "          5       0.94      0.94      0.94       962\n",
      "\n",
      "avg / total       0.93      0.93      0.93      6617\n",
      "\n",
      "[[1236    0    0    0   29    1]\n",
      " [  10  998  152    0    7    1]\n",
      " [   1   97 1141    0    1    0]\n",
      " [   0    0    0 1037   31   36]\n",
      " [   0    0    0    5  855   17]\n",
      " [   0    0    0   15   41  906]]\n",
      "number of components = 400 accuracy = 0.932900\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.98      0.98      1266\n",
      "          1       0.91      0.85      0.88      1168\n",
      "          2       0.88      0.92      0.90      1240\n",
      "          3       0.98      0.94      0.96      1104\n",
      "          4       0.90      0.97      0.93       877\n",
      "          5       0.94      0.95      0.95       962\n",
      "\n",
      "avg / total       0.94      0.93      0.93      6617\n",
      "\n",
      "[[1242    0    0    0   23    1]\n",
      " [  13  998  153    0    3    1]\n",
      " [   1   98 1140    0    1    0]\n",
      " [   0    0    0 1041   30   33]\n",
      " [   0    0    0    7  851   19]\n",
      " [   0    0    0   14   37  911]]\n",
      "number of components = 450 accuracy = 0.934411\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.98      0.99      1266\n",
      "          1       0.91      0.85      0.88      1168\n",
      "          2       0.88      0.92      0.90      1240\n",
      "          3       0.98      0.94      0.96      1104\n",
      "          4       0.91      0.97      0.94       877\n",
      "          5       0.94      0.95      0.95       962\n",
      "\n",
      "avg / total       0.94      0.93      0.93      6617\n",
      "\n",
      "[[1245    0    0    0   20    1]\n",
      " [  12  993  159    0    3    1]\n",
      " [   1   99 1139    0    1    0]\n",
      " [   0    0    0 1042   27   35]\n",
      " [   0    0    0    7  850   20]\n",
      " [   0    0    0   13   32  917]]\n",
      "number of components = 500 accuracy = 0.934865\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.99      0.98      0.99      1266\n",
      "          1       0.91      0.85      0.88      1168\n",
      "          2       0.88      0.92      0.90      1240\n",
      "          3       0.98      0.95      0.96      1104\n",
      "          4       0.91      0.97      0.94       877\n",
      "          5       0.94      0.95      0.95       962\n",
      "\n",
      "avg / total       0.94      0.93      0.93      6617\n",
      "\n",
      "[[1245    0    0    0   20    1]\n",
      " [  14  993  158    0    2    1]\n",
      " [   1  100 1138    0    1    0]\n",
      " [   0    0    0 1045   24   35]\n",
      " [   0    0    0    7  847   23]\n",
      " [   0    0    0   12   32  918]]\n",
      "number of components = 550 accuracy = 0.934865\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAFdCAYAAABfMCThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3Xuc1VW9//HX4i4aiKISpqUnLSwrQSn1mJmamppZ3hAV\n72J66pAnszylD3/dTp3w5C9lEG0GvEyipJFppBbHG6CCov28ZXhXyBuoAYIz6/fH2sR2mAH2sGfW\nvryej8d+DPv7/e4vn70YZr9nrfVd3xBjRJIkqRx65C5AkiTVDoOFJEkqG4OFJEkqG4OFJEkqG4OF\nJEkqG4OFJEkqG4OFJEkqm165C+isEMLmwAHAM8DyvNVIklRV+gEfAmbEGF8r54mrNliQQsU1uYuQ\nJKmKjQauLecJqzlYPANw9dVXM2zYsMylVI9x48Zx8cUX5y6j6thupbPNOsd2K51tVrrHHnuM4447\nDgqfpeVUzcFiOcCwYcMYPnx47lqqxsCBA22vTrDdSmebdY7tVjrbbIOUfSqBkzclSVLZGCwkSVLZ\nGCwkSVLZGCzqzKhRo3KXUJVst9LZZp1ju5XONqssIcaYu4ZOCSEMB+bOnTvXSTuSJJVg3rx5jBgx\nAmBEjHFeOc9dzVeFSJKqTIzQ0gIrV6bHihWr/1z86Gh7e/taW3O/q47tumt61BODhSTVsRhh2TJ4\n4430eP311X8u3vbWW6s/0Dc0DJRbz57lP2e5fP/7BgtJUhVavnztwWBt21asaP+cG28Mm20GgwbB\ngAHQpw/07p0eG20EAweuft72UXxsufa1t79nTwihe9taa2ewkKTMYkzd+StXwuLFpQeDN95IwaI9\nG22UgsGqgDBoEOyww5rbVj1Wbdt00/QhLpXKYCGpKrS0wDvvtP9YsaK0fe++mx4tLWt+bW9bOY5d\n22taWtb+3vv2XfODf/vt19zW3vO+fbvn30daxWAhqWxaW+HJJ+H+++H55zc8ABQ/NmSCXu/e6QN2\n1WNVF3rPntCrV/tfO9rXt2/nXreufb16pcfAgWsGhI02Kt+/kdTVDBaSOiVGeOGFFCLuuy99feAB\nePPNtH/w4Pd+mLf3WDWG39H+Pn3W/vr12d+nD/RwxR6p2xgsJK2X115L4aE4SCxalPZtvTXsthuc\nd176uuuuaYxeUv0xWEhaw9tvw7x5q4PE/ffDggVp36BBKTycemr6uttuMHRo3nolVQ6DhVTnVqyA\nRx55b0/Eo4+mOQ0bbQTDh8Nhh60OEf/yL17eJ6ljBgupjrS2whNPvLcn4qGH0uTIXr1g551hjz3g\nG9+AkSNhp53SdklaX/7IkGpUjOnKjOKeiLlzV0+u3HHHFB6OPTb1RHzqU159IGnDGSykGvHqq2tO\nrvz739O+D3xg9eTKkSNhxAgnV0rqGgYLqUo9+ijceuvqEPH002n7oEEpPJx++up5Ee9/f95aJdUP\ng4VURR5/HK6/HqZOhb/8Bfr3T5Mrv/zlFCZ22y2tyOjkSkm5GCykCvfkk6vDxMMPw/vel67S+NGP\n4AtfcMlmSZXFYCFVoKeeSkFi6lSYPx822QS+9CW46CI44ADo1y93hZLUPoOFVCH+9rfVPRMPPpiW\nuz70ULjgAjjwQK/YkFQdDBZSRk8/vTpMzJ2b5kwccgicfz4cdFB6LknVxGAhdbNnn10dJu6/P/VE\nHHwwfPvb8MUvpp4KSapWBgupGzz3HNxwQwoTc+akORJf/CKcc04KFZtskrtCSSoPg4XURV54YXWY\nmDUrXb1x0EFw7bVpuON978tdoSSVn8FCKqMXX4Rp01KYuOce6NMnTby8+uo0EXPAgNwVSlLXMlhI\nG+jll1eHibvvTjftOuAAmDIlXSI6cGDuCiWp+xgspE5YuDCFieuvhzvvhJ4902JVjY1p8SrvwyGp\nXhkspPW0aBH85jepZ+J//zeFif32gyuvTGFis81yVyhJ+RkspLVoaYGmpjThcubMdA+OffeFSZPS\n/Tk23zx3hZJUWQwWUgeWLIFRo+APf0hhoqEBDj8cBg/OXZkkVS6DhdSOJ59MEy8XLUrB4gtfyF2R\nJFWHHrkLkCrNjBnpFuQhwH33GSokqRQGC6kgRhg/Pq2IueeeMHs27LBD7qokqboYLCRg+XI46aS0\nxPZ//AdMn+76E5LUGc6xUN17+WX4ylfSrcqvugqOOy53RZJUvQwWqmsPPJAuG40xLXQ1cmTuiiSp\nujkUorrV3Ax77QVbb51uX26okKQNZ7BQ3Wlpge98B449Fo48Mq2iOXRo7qokqTY4FKK68uabMHo0\n3HIL/OxnabJmCLmrkqTaYbBQ3XjqqbTo1Ysvws03w0EH5a5IkmqPQyGqC7ffnuZQtLTAnDmGCknq\nKgYL1bQY4ZJL4MADU7CYPRs++tHcVUlS7TJYqGa98w6cdhp84xvpcfPNMGhQ7qokqbY5x0I1adEi\n+OpX02WkTU0wZkzuiiSpPhgsVHMefBAOOwxWrkyXkn7mM7krkqT64VCIasrUqekGYltumXorDBWS\n1L0MFqoJra3wve/B0UfD4YfDXXfBBz6QuypJqj8OhajqvfUWHH98uiPpT34C557roleSlIvBQlVt\nwYK06NVzz6VgccghuSuSpPrmUIiq1p//DLvtBsuXp/UpDBWSlJ/BQlUnRrjsMth/fxg+HO67D3ba\nKXdVkiQwWKjKrFgBZ54JZ50FZ58Nt94Km22WuypJ0irOsVDVeOWVtOjV7NlwxRVwyim5K5IktWWw\nUFWYPz8terVsWZpbseeeuSuSJLXHoRBVvGnTYI890pDH/fcbKiSpkhksVLFaW+HCC+GII9IVH3ff\nDdtum7sqSdLaOBSiivT22+nGYb/5DfzgB/Dd77rolSRVA4OFKs4zz6T5FAsWwE03pT9LkqpDp4ZC\nQghnhRCeDiEsCyHMDiHsth7HPxpCWBpCeCyEcHw7xxxZ2LcshDA/hHBQZ2pTdbvzzrTo1VtvwaxZ\nhgpJqjYlB4sQwtHAz4ELgF2A+cCMEMLgDo4/E/gh8H1gJ+BC4NIQwsFFx+wBXAtMAj4F/Ba4KYTg\nskd1ZOJE2Hdf2HnnNEnz4x/PXZEkqVSd6bEYB0yMMU6JMT4OjAWWAid3cPxxheNviDE+E2O8Drgc\n+HbRMV8Hbo0xjo8xPhFj/D4wDzi7E/WpCn3zmzB2LJxxBsyYAZtvnrsiSVJnlBQsQgi9gRHAHau2\nxRgjcDuwewcv6wssb7NtOTAyhNCz8Hz3wjmKzVjLOVVDrr0WLr4YfvEL+OUvoXfv3BVJkjqr1B6L\nwUBPYFGb7YuAIR28ZgZwaghhOEAIYVfgFKB34XwUXlvKOVUjnnsOvvY1GDUKvv713NVIkjZUd1wV\n8n+ArYBZIYQewEKgCTgXaN3Qk48bN46BAwe+Z9uoUaMYNWrUhp5aXaylBU44AQYMgEsvzV2NJNWm\n5uZmmpub37NtyZIlXfb3lRosXgVaSEGh2FakwLCGGONyUo/FGYXjXgbOAN6KMb5SOGxhKecsdvHF\nFzN8+PD1fgOqHOPHp6tA7rgDBg3KXY0k1ab2ftmeN28eI0aM6JK/r6ShkBjjSmAusO+qbSGEUHh+\n7zpe2xJjfKkwJ+MY4HdFu2cVn7Ng/8J21aD58+H88+Gcc2CffXJXI0kql84MhYwHmkIIc4H7SFeJ\n9CcNbxBC+DEwNMY4pvB8B2AkMAfYDPgm8DHghKJz/gKYGUL4JvB7YBRpkuhpnahPFW7ZMhg9GoYN\nS6tqSpJqR8nBIsY4tbBmxUWk4YqHgAOKhjWGANsUvaQncA6wI7AS+DOwR4zxuaJzzgohHEta7+KH\nwF+Bw2KMj5b+llTpvvMdeOopeOAB6Ns3dzWSpHLq1OTNGONlwGUd7DupzfPHgXVOgogxTgOmdaYe\nVY/bbkuXlf7P/7gAliTVIu9uqm7z2mtw4omw337wb/+WuxpJUlcwWKhbxJhW1Vy2DJqaoIffeZJU\nk7y7qbrFlCkwbRpcfz1svXXuaiRJXcXfG9XlFiyAs8+GMWPgiCNyVyNJ6koGC3WpVatrDh4Ml1yS\nuxpJUldzKERd6r/+C2bNgpkz09LdkqTaZo+FuswDD8AFF8B558Fee+WuRpLUHQwW6hL/+EdaXfOT\nn0zhQpJUHxwKUZf41rfg+edh3jzo0yd3NZKk7mKwUNndcgtMmJBuhf7Rj+auRpLUnRwKUVn9/e9w\n0klw0EFw5pm5q5EkdTeDhcomRjjtNGhthV/9CkLIXZEkqbs5FKKyueIKmD4dbroJhgzJXY0kKQd7\nLFQWf/0r/Pu/px6Lww7LXY0kKReDhTbYypVw3HEwdCiMH5+7GklSTg6FaIP98Icwdy7cfTdssknu\naiRJOdljoQ0yaxb84Afwve/BZz6TuxpJUm4GC3Xa22/D8cfDrrvC+efnrkaSVAkcClGnjRsHCxfC\nH/4AvfxOkiRhsFAn3XRTurx00iT48IdzVyNJqhQOhahkL78Mp56aLis95ZTc1UiSKonBQiWJMYWJ\nXr1Sb4Wra0qSijkUopJMmAC33ppuNLbFFrmrkSRVGnsstN4eewzOOQfOOivdZEySpLYMFlovK1ak\n1TU/9CH46U9zVyNJqlQOhWi9XHghPPwwzJ4N/fvnrkaSVKkMFlqnu+6Cn/wkLd09YkTuaiRJlcyh\nEK3VkiVpdc0994Rzz81djSSp0tljobX6+tfh9ddh5kzo2TN3NZKkSmewUIemToUpU2Dy5DRpU5Kk\ndXEoRO168UUYOxaOPDINhUiStD4MFlpDayuceCJstBE0NLi6piRp/TkUojVccgncfjvcdhtstlnu\naiRJ1cQeC73HI4/AeeelW6Lvt1/uaiRJ1cZgoX9avhxGj4YddoAf/Sh3NZKkauRQiP7pP/8TnngC\n7rsP+vXLXY0kqRoZLATAn/4E48fDz34Gn/xk7mokSdXKoRDxxhswZgx87nNpboUkSZ1lsKhzMcKZ\nZ8Lbb6eFsHr4HSFJ2gAOhdS5a6+F666D5mbYZpvc1UiSqp2/n9axZ5+Fs85KV4Icc0zuaiRJtcBg\nUadaWtK8ioED4Ze/zF2NJKlWOBRSp37+c7jzTvjzn2HTTXNXI0mqFfZY1KEHH0xrVnzrW7D33rmr\nkSTVEoNFnVm2LM2p+NjH4KKLclcjSao1DoXUmfPOg6efhrlzoW/f3NVIkmqNwaKO/PGP6c6ll1wC\nO+2UuxpJUi1yKKSOfPe7aU7FWWflrkSSVKvssagTDzyQhj9uvtnVNSVJXcePmDoxYQJsuy0ceGDu\nSiRJtcxgUQcWL05Ldp9+OvTsmbsaSVItM1jUgauugpUr4ZRTclciSap1BosaFyM0NMDhh8OQIbmr\nkSTVOoNFjbv7bnj0URg7NnclkqR6YLCocQ0NsOOOsM8+uSuRJNUDg0UNe+UVuOEGOOMMCCF3NZKk\nemCwqGFNTSlQjBmTuxJJUr0wWNSo1laYOBGOOgo23zx3NZKkeuHKmzXq9tvhb3+DKVNyVyJJqif2\nWNSohgbYeWfYfffclUiS6onBoga9+CJMnw5nnumkTUlS9zJY1KArr4R+/WD06NyVSJLqjcGixrz7\nLkyalELFgAG5q5Ek1RuDRY255RZ44QVX2pQk5WGwqDETJsDIkbDLLrkrkSTVIy83rSELFsCMGWmO\nhSRJOdhjUUMmTUrzKo4+OnclkqR6ZbCoEStWpJ6KMWOgf//c1UiS6pXBokbceGO66ZiTNiVJORks\nakRDA+y9NwwblrsSSVI9c/JmDXj8cZg5E5qbc1ciSap39ljUgIYG2GILOPzw3JVIkuqdwaLKLV0K\nkyfDySdD3765q5Ek1TuDRZWbOhUWL4bTT89diSRJBouq19AABxwA22+fuxJJkjoZLEIIZ4UQng4h\nLAshzA4h7LaO40eHEB4KIfwjhPBSCOHKEMJmRfvHhBBaQwgtha+tIYSlnamtnjz4IMyZk26PLklS\nJSg5WIQQjgZ+DlwA7ALMB2aEEAZ3cPyewGRgErATcAQwEri8zaFLgCFFjw+WWlu9mTgRtt4aDj44\ndyWSJCWd6bEYB0yMMU6JMT4OjAWWAid3cPxngKdjjJfGGJ+NMd4LTCSFi2IxxvhKjPHvhccrnait\nbrz1FlxzDZx2GvTyomFJUoUoKViEEHoDI4A7Vm2LMUbgdmD3Dl42C9gmhHBQ4RxbAUcCv29z3CYh\nhGdCCM+FEG4KIexUSm315pprYNkyOPXU3JVIkrRaqT0Wg4GewKI22xeRhi/WUOihOA64LoSwAngZ\neAM4u+iwJ0g9Hl8CRhfqujeEMLTE+upCjOn26IcemoZCJEmqFF3eiV7oefgFcCHwR+D9wH+ThkNO\nBYgxzgZmF71mFvAYcAZpLkeHxo0bx8CBA9+zbdSoUYwaNaps76HSzJ4NDz8MP/1p7kokSZWuubmZ\n5jZLMy9ZsqTL/r6QRjLW8+A0FLIU+GqMcXrR9iZgYIxxjbUfQwhTgH4xxqOKtu0J3AW8P8bYtvdj\n1TFTgZUxxtEd7B8OzJ07dy7Dhw9f7/dQC8aMgbvugqeegh5eMCxJKtG8efMYMWIEwIgY47xynruk\nj6UY40pgLrDvqm0hhFB4fm8HL+sPvNtmWysQgdDeC0IIPYCdScMmKvL663DddXDGGYYKSVLl6cxQ\nyHigKYQwF7iPdJVIf6AJIITwY2BojHFM4fjfAZeHEMYCM4ChwMXAnBjjwsJrvkcaCnkK2BQ4F9gW\nuKJzb6t2TZ4Mra1pCW9JkipNycEixji1sGbFRcBWwEPAAUWXhw4Btik6fnIIYRPgLNLcisWkq0rO\nKzrtINK6FkNIEzvnArsXLmdVQYxppc0jjkg3HZMkqdJ0avJmjPEy4LIO9p3UzrZLgUvXcr5vAt/s\nTC31ZOZMePJJmDQpdyWSJLXPUfoqMmECDBsGe+2VuxJJktpnsKgSCxfCjTfC2LEQ2p3yKklSfgaL\nKvGrX0Hv3nD88bkrkSSpYwaLKtDSApdfDsccA4MG5a5GkqSOGSyqwIwZ8OyzaRhEkqRKZrCoAg0N\nMHw47LZb7kokSVo7g0WFe+45+P3vnbQpSaoOBosKN2kSbLwx1PA91SRJNcRgUcFWroQrrkhXgmyy\nSe5qJElaN4NFBZs+Pa1fccYZuSuRJGn9GCwqWEMD7LEHfOITuSuRJGn9dOpeIep6f/0r3H47TJmS\nuxJJktafPRYV6vLLYbPN4Mgjc1ciSdL6M1hUoOXLobERTjoJ+vXLXY0kSevPYFGBpk2D116D00/P\nXYkkSaUxWFSgCRNg331hxx1zVyJJUmmcvFlhHnkE7rkHrr8+dyWSJJXOHosKM3EiDBkChx2WuxJJ\nkkpnsKggb7+dLi895RTo3Tt3NZIklc5gUUF+/esULk47LXclkiR1jsGigjQ0wMEHwwc/mLsSSZI6\nx8mbFeKBB2DuXLj55tyVSJLUefZYVIgJE2DbbeHAA3NXIklS5xksKsDixdDcnBbE6tkzdzWSJHWe\nwaICXHUVrFwJJ5+cuxJJkjaMwSKzGNOkzS9/Gd7//tzVSJK0YQwWmd19Nzz6KIwdm7sSSZI2nMEi\ns4YG2GEH2Gef3JVIkrThDBYZvfIK3HBD6q3o4b+EJKkG+HGWUWMjhABjxuSuRJKk8jBYZNLamm44\ndtRRsPnmuauRJKk8XHkzk9tvhwUL0qWmkiTVCnssMmlogJ13ht13z12JJEnlY7DI4MUXYfr0NGkz\nhNzVSJJUPgaLDK68Evr1g+OOy12JJEnlZbDoZu++C5MmwbHHwoABuauRJKm8DBbd7JZb4IUX4Mwz\nc1ciSVL5GSy62YQJMHIk7LJL7kokSSo/LzftRgsWwIwZaY6FJEm1yB6LbjRpUppXcfTRuSuRJKlr\nGCy6yYoVqadizBjo3z93NZIkdQ2DRTe58cZ007EzzshdiSRJXcdg0U0aGuCzn4WddspdiSRJXcfJ\nm93g8cdh5kxobs5diSRJXcsei27Q0ABbbAGHH567EkmSupbBoostXQqTJ8PJJ0PfvrmrkSSpaxks\nutjUqbB4MZx+eu5KJEnqegaLLtbQAAccANtvn7sSSZK6npM3u9CDD8KcOelSU0mS6oE9Fl1o4kQY\nOhQOOSR3JZIkdQ+DRRd56y245ho47TToZb+QJKlOGCy6yNVXw7JlKVhIklQvDBZdIMZ0e/RDD4Wt\nt85djSRJ3cdg0QVmz4ZHHoGxY3NXIklS9zJYdIGGBthuO9h//9yVSJLUvQwWZfb663Dddekupj1s\nXUlSnfGjr8yuugpaW+Gkk3JXIklS9zNYlFGM8KtfwZe+BFtumbsaSZK6n8GijB56CB5+GE48MXcl\nkiTlYbAoo8ZGGDIEDjwwdyWSJOVhsCiTd95JK20ef7wrbUqS6pfBokx+97t0RYiTNiVJ9cxgUSaN\njfDpT8OwYbkrkSQpH4NFGbz8MvzhD07alCTJYFEGV10FffrAMcfkrkSSpLwMFhsoxjQMcvjhsOmm\nuauRJCkvg8UGmjMHHn/cSZuSJIHBYoM1NsI228DnP5+7EkmS8jNYbIBly+DXv4YTToCePXNXI0lS\nfgaLDXDjjfDmm14NIknSKgaLDdDYCHvtBR/+cO5KJEmqDAaLTnruObjjDidtSpJUzGDRSZMnQ//+\ncOSRuSuRJKlyGCw6IUZoaoIjjoBNNsldjSRJlcNg0Ql33QULFjgMIklSW50KFiGEs0IIT4cQloUQ\nZocQdlvH8aNDCA+FEP4RQngphHBlCGGzNsccGUJ4rHDO+SGEgzpTW3dobITtt4fPfjZ3JZIkVZaS\ng0UI4Wjg58AFwC7AfGBGCGFwB8fvCUwGJgE7AUcAI4HLi47ZA7i2cMyngN8CN4UQdiq1vq729ttw\n/fXpEtMQclcjSVJl6UyPxThgYoxxSozxcWAssBQ4uYPjPwM8HWO8NMb4bIzxXmAiKVys8nXg1hjj\n+BjjEzHG7wPzgLM7UV+Xuv56WLoUxozJXYkkSZWnpGARQugNjADuWLUtxhiB24HdO3jZLGCbVUMb\nIYStgCOB3xcds3vhHMVmrOWc2TQ1peW7t902dyWSJFWeUnssBgM9gUVtti8ChrT3gkIPxXHAdSGE\nFcDLwBu8tzdiSCnnzOVvf4M773TSpiRJHenyq0IK8yR+AVwIDAcOALYjDYdUlaYmGDAg3SJdkiSt\nqVeJx78KtABbtdm+FbCwg9ecB9wTYxxfeP6XEMLXgLtCCOfHGBcVXlvKOf9p3LhxDBw48D3bRo0a\nxahRo9b10pK0tKRFsY45Ji2MJUlSNWhubqa5ufk925YsWdJlf19JwSLGuDKEMBfYF5gOEEIIheeX\ndPCy/sCKNttagQisuq5iVjvn2L+wfa0uvvhihg8fvr5vodP+9Cd4/nmHQSRJ1aW9X7bnzZvHiBEj\nuuTvK7XHAmA80FQIGPeRrhLpDzQBhBB+DAyNMa66buJ3wOUhhLGkCZlDgYuBOTHGVT0SvwBmhhC+\nSZrUOYo0SfS0zryprtDUBB/5CHz607krkSSpcpUcLGKMUwtrVlxEGq54CDggxvhK4ZAhwDZFx08O\nIWwCnAX8N7CYdFXJeUXHzAohHAv8sPD4K3BYjPHRTr2rMlu8GH7zG7jwQteukCRpbTrTY0GM8TLg\nsg72rTFYEGO8FLh0HeecBkzrTD1d7brrYMUKOP743JVIklTZvFfIemhshAMPhKFDc1ciSVJl61SP\nRT157DGYMyetuClJktbOHot1aGqCzTaDQw/NXYkkSZXPYLEW774LU6bAscdC3765q5EkqfIZLNZi\nxgxYuNC1KyRJWl8Gi7VobIRPfAJ22SV3JZIkVQeDRQdefRWmT0+9Fa5dIUnS+jFYdKC5GWKE0aNz\nVyJJUvUwWHSgsREOOQS22CJ3JZIkVQ+DRTvmz4cHH3TSpiRJpTJYtKOxEbbcEg46KHclkiRVF4NF\nGytWwDXXpPuC9O6duxpJkqqLwaKNm29OV4SceGLuSiRJqj4GizaammDXXeHjH89diSRJ1cdgUWTh\nQrjlFidtSpLUWQaLIldfDb16wahRuSuRJKk6GSwKYkxXg3z5yzBoUO5qJEmqTgaLgvvvh0cfddKm\nJEkbwmBR0NQEW28N+++fuxJJkqqXwQJYvjzdG+SEE6Bnz9zVSJJUvQwWwE03weLFDoNIkrShDBak\nSZt77gk77pi7EkmSqlvdB4vnn4fbbrO3QpKkcqj7YHHVVdCvHxx1VO5KJEmqfnUdLFatXXHEETBg\nQO5qJEmqfnUdLO65B556yiW8JUkql7oOFo2N8KEPwd57565EkqTaULfB4h//gKlTYcwY6FG3rSBJ\nUnnV7UfqtGnw9tspWEiSpPKo22DR2Aj77APbbZe7EkmSakev3AXksGABzJwJU6bkrkSSpNpSlz0W\nkyfD+94HX/lK7kokSaotdRcsWltTsDjqKNh449zVSJJUW+ouWMycCc8+69oVkiR1hboLFo2N6WZj\ne+yRuxJJkmpPXQWLJUvSZaYnnggh5K5GkqTaU1fBYupUeOcdOP743JVIklSb6ipYNDbC/vvDBz6Q\nuxJJkmpT3axj8cQTMGsW/PrXuSuRJKl21U2PRVMTbLopHHZY7kokSapddREsWlrSKpvHHgv9+uWu\nRpKk2lUXweKPf4SXXkpXg0iSpK5TF8GisRE+9jHYddfclUiSVNtqPli8/jr89rdppU3XrpAkqWvV\nfLBobk5zLI47LnclkiTVvpoPFo2NcPDBsNVWuSuRJKn21XSweOQRmDvXSZuSJHWXmg4WjY0weHDq\nsZAkSV2vZoPFypVw9dVpbkWfPrmrkSSpPtRssLjlFnjllXQ1iCRJ6h41GywaG2H4cPjEJ3JXIklS\n/ajJYPH3v8Pvf++kTUmSultNBovrroMePdK9QSRJUvepydumjx0Ln/40bL557kokSaovNdlj0bs3\njByZuwpJkupPTQYLSZKUh8FCkiSVjcGizjQ3N+cuoSrZbqWzzTrHdiudbVZZDBZ1xv+AnWO7lc42\n6xzbrXS2WWUxWEiSpLIxWEiSpLIxWEiSpLKp5gWy+gE89thjueuoKkuWLGHevHm5y6g6tlvpbLPO\nsd1KZ5sO6+W3AAAFoUlEQVSVruizs1+5zx1ijOU+Z7cIIRwLXJO7DkmSqtjoGOO15TxhNQeLzYED\ngGeA5XmrkSSpqvQDPgTMiDG+Vs4TV22wkCRJlcfJm5IkqWwMFpIkqWwMFpIkqWwMFpIkqWwMFpIk\nqWyqMliEEM4KITwdQlgWQpgdQtgtd025hBD2CiFMDyG8GEJoDSF8qZ1jLgohvBRCWBpCuC2E8OE2\n+/uGEC4NIbwaQngrhHBDCGHL7nsX3SuE8J0Qwn0hhDdDCItCCDeGEHZs5zjbrUgIYWwIYX4IYUnh\ncW8I4cA2x9hmaxFCOK/w/3R8m+22W5EQwgWFdip+PNrmGNusjRDC0BDCVYX3vLTw/3V4m2O6vN2q\nLliEEI4Gfg5cAOwCzAdmhBAGZy0sn42Bh4CvAWtcOxxC+DZwNnA6MBL4B6m9+hQd9j/AwcBXgc8C\nQ4FpXVt2VnsB/xf4NLAf0Bv4Ywhho1UH2G7teh74NjAcGAH8CfhtCGEY2GbrUvgF6HTSz6zi7bZb\n+/4CbAUMKTz+ddUO22xNIYRNgXuAd0hrPA0DzgHeKDqme9otxlhVD2A28Iui5wF4ATg3d225H0Ar\n8KU2214CxhU9HwAsA44qev4OcHjRMR8pnGtk7vfUTe02uPB+/9V2K7ntXgNOss3W2U6bAE8Anwf+\nDIz3e22t7XUBMG8t+22zNdvkJ8D/ruOYbmm3quqxCCH0Jv2mdMeqbTG989uB3XPVValCCNuRkn5x\ne70JzGF1e+1KumdM8TFPAM9RP226Kam353Ww3dZHCKFHCOEYoD9wr222TpcCv4sx/ql4o+22VjsU\nhnj/FkK4OoSwDdhma3Eo8EAIYWphiHdeCOHUVTu7s92qKliQfrPsCSxqs30RqcH0XkNIH5hra6+t\ngBWFb7COjqlZIYRA6vq7O8a4agzXdutACOHjIYS3SL/VXEb6zeYJbLMOFQLYp4DvtLPbdmvfbOBE\nUpf+WGA74M4QwsbYZh3ZHjiT1DP2BWACcEkI4fjC/m5rt2q+u6lUDpcBOwF75i6kSjwOfBIYCBwB\nTAkhfDZvSZUrhPABUnDdL8a4Mnc91SLGOKPo6V9CCPcBzwJHkb4HtaYewH0xxu8Vns8PIXycFMyu\n6u5CqsmrQAspVRXbCljY/eVUvIWkOShra6+FQJ8QwoC1HFOTQgi/BL4IfC7G+HLRLtutAzHGd2OM\nC2KMD8YYzydNRPwGtllHRgBbAPNCCCtDCCuBvYFvhBBWkH4TtN3WIca4BHgS+DB+r3XkZeCxNtse\nA7Yt/Lnb2q2qgkUh8c8F9l21rdCVvS9wb666KlWM8WnSN0Nxew0gXQ2xqr3mAu+2OeYjpG/GWd1W\nbDcrhIrDgH1ijM8V77PdStID6Gubdeh2YGfSUMgnC48HgKuBT8YYF2C7rVMIYRNSqHjJ77UO3UOa\naFnsI6Senu79uZZ7JmsnZr4eBSwFTgA+CkwkzUzfIndtmdpjY9IPq0+RZu7+e+H5NoX95xba51DS\nD7ibgL8CfYrOcRnwNPA50m9Y9wB35X5vXdhml5EuwdqLlMRXPfoVHWO7rdluPyq02QeBjwM/LvwQ\n+rxtVlI7tr0qxHZbs41+RrrU8YPAHsBtpN6dzW2zDttsV9Lcp+8A/wIcC7wFHNPd32vZG6OTDfg1\n4BnSZTKzgF1z15SxLfYmBYqWNo9fFR1zIekyo6XADODDbc7Rl7Suw6uFb8TrgS1zv7cubLP22qsF\nOKHNcbbbe9/vFcCCwv+7hcAfKYQK26ykdvwTRcHCdmu3jZpJywgsI12RcC2wnW22znb7IvBwoU3+\nH3ByO8d0ebuFwokkSZI2WFXNsZAkSZXNYCFJksrGYCFJksrGYCFJksrGYCFJksrGYCFJksrGYCFJ\nksrGYCFJksrGYCFJksrGYCFJksrGYCFJksrm/wOMb09SDX0zIgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7efde59b10b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimal number of components: 500 \n",
      "\n",
      "evaluation ...\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.99      0.99       537\n",
      "          1       0.91      0.83      0.87       491\n",
      "          2       0.86      0.93      0.89       532\n",
      "          3       0.94      0.96      0.95       496\n",
      "          4       0.93      0.87      0.90       420\n",
      "          5       0.92      0.97      0.94       471\n",
      "\n",
      "avg / total       0.93      0.93      0.93      2947\n",
      "\n",
      "[[529   0   2   0   6   0]\n",
      " [  2 409  78   0   1   1]\n",
      " [  0  38 494   0   0   0]\n",
      " [  0   0   0 477  17   2]\n",
      " [  0   0   0  18 365  37]\n",
      " [  0   0   0  10   5 456]]\n",
      "accuracy on test set (PCA): 0.926366 \n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('\\nsearching optimal number of principal components ...')\n",
    "\n",
    "n_components_list = [10, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550]\n",
    "n_components = get_n_components(X_train, y_train, X_val, y_val, n_components_list)\n",
    "\n",
    "print(\"optimal number of components: %d \" % n_components)\n",
    "\n",
    "print('\\nevaluation ...')\n",
    "\n",
    "y_pred, acc = classification_pca(X_train, y_train, X_test, y_test, n_components, grid_search=False)\n",
    "\n",
    "print(\"accuracy on test set (PCA): %f \" % acc)\n",
    "\n",
    "print('Done.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improving classification by relabeling ambiguous data (Failed)\n",
    "\n",
    "The main goal is to relabel the data in the classes which is difficult to be correctly predicted. It can be seen that the data with label 1 and label 2 is difficult to be separated. Therefore, in order to separate the two classes, I train an SVM only on the data of the two classes. Then an SVM is used to relabel the test data which has the predicted label of 1 or 2 from the previous step. However, the proposed method does not achieve expected results. I expect that in the previous section, the principla compoenents selected by the PCA are suitable to separate the 5 classes. However, these components (features) are not good enough to separate the data from classes 1 and 2. Therefore, I try to select the optimal components (features) to seperate data from classes 1 and 2 with the same method presented in the previous section. However, the features are not discriminative enough to separate the data from classes 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relabeling ...\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.53      0.98      0.69       125\n",
      "          2       0.88      0.11      0.20       123\n",
      "\n",
      "avg / total       0.70      0.55      0.45       248\n",
      "\n",
      "[[123   2]\n",
      " [109  14]]\n",
      "number of components = 10 accuracy = 0.552419\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 50 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 100 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 150 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 200 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 250 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 300 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 350 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 400 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 450 accuracy = 0.540323\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.52      1.00      0.69       125\n",
      "          2       1.00      0.07      0.14       123\n",
      "\n",
      "avg / total       0.76      0.54      0.41       248\n",
      "\n",
      "[[125   0]\n",
      " [114   9]]\n",
      "number of components = 500 accuracy = 0.540323\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAFkCAYAAACAUFlOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X+MXeV95/H3dxhgylI7lQ3+wQ87vhdYULM4npAISEKJ\nw7JK1URNqsJssnWNQmBLVHYibZqoTaioulR1gTRNXFGljUVopqLZJAW1WihkKZsS14onhi0xoQQ7\nhsXYQGCwgp1g+7t/nDtwfbkznnPn/hjPfb+ko+Ge85xznvtgeT5+fpwTmYkkSVK3DPS6ApIkqb8Y\nPiRJUlcZPiRJUlcZPiRJUlcZPiRJUlcZPiRJUlcZPiRJUlcZPiRJUlcZPiRJUlcZPiRJUle1FD4i\n4rqI2BER+yNic0RcME3ZSyLicMN2KCJOrSuzrm7/ZJlXprnmp2plbmml/pIkqXcGy54QEVcANwMf\nA7YAo8A9EXF2Zj4/xWkJnA3se21H5t6GMhO1MlF3TrP7X1C798Nl6y5JknqvlZ6PUeC2zLw9Mx8D\nrgVeAa46ynnPZebeya3J8czM+jLPNRaIiJOBO4CPAi+1UHdJktRjpcJHRBwPDAP3T+7L4rW49wEX\nTncqsC0inomIeyPioiZlTo6InRGxKyK+GRHnNSnzReDuzPxWmXpLkqS5o+ywy2LgOGBPw/49wDlT\nnLMbuAb4LnAicDXwQES8PTO31cr8gKLn5BFgIfDfgYci4rzMfAYgIq4EVgNvm0lFI2IRcDmwEzgw\nk3MkSRIAQ8BK4J7MfKHdFy8956OszHwceLxu1+aIqFAM36yrldkMbJ4sEBHfAbZThJYbIuIM4HPA\nezPz1Rne+nLgr2f/DSRJ6lsfBr7a7ouWDR/PA4eAJQ37lwDPlrjOFuDiqQ5m5sGI+B5Qre1aA5wC\njEfE5ITU44B3R8THgRNrwz/1dgLccccdnHvuuSWqptkYHR3l1ltv7XU1+opt3n22effZ5t21fft2\nPvKRj0Dtd2m7lQofmflqRGwF1gJ3AdTCwFrg8yUutZpiOKapiBgA3gL8fW3XfbXP9TZR9I78UZPg\nAbWhlnPPPZc1a9aUqJpmY+HChbZ3l9nm3Webd59t3jMdmbbQyrDLLcCmWgiZXGp7EkUYICJuApZn\n5rra5+uBHcCjFGNIVwOXApdNXjAiPkMx7PIE8Cbgk8CZwJcAMvMnwPfrKxERPwFeyMztLXwHSZLU\nI6XDR2beGRGLgRsphlu2AZfXLY1dCpxRd8oJFM8FWU6xJPcRYG1mPlhX5heAv6id+yKwFbiwtpR3\nyqqUrbskSeq9liacZuZGYOMUx9Y3fN4AbDjK9T4BfKJkHd5TprwkSZobfLeL2mpkZKTXVeg7tnn3\n2ebdZ5vPL9F8ruaxLyLWAFu3bt3qJCVJkkoYHx9neHgYYDgzx9t9fXs+JElSVxk+JElSVxk+JElS\nVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+\nJElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElS\nVxk+JElSVxk+JElSVxk+JElSVxk+JElSV/V1+Dh8GB56CJ56qtc1kSSpf/R1+ABYuxa+9rVe10KS\npP7R1+FjYAAqFfjhD3tdE0mS+kdfhw8owscTT/S6FpIk9Y++Dx/Vqj0fkiR1U0vhIyKui4gdEbE/\nIjZHxAXTlL0kIg43bIci4tS6Muvq9k+WeaXhOp+OiC0R8XJE7ImIb0TE2a3Uv161Cjt3wsGDs72S\nJEmaidLhIyKuAG4GbgDeCjwM3BMRi6c5LYGzgKW1bVlm7m0oM1F3fCmwouH4u4A/A94BvBc4Hrg3\nIn6u7HeoV6kUwWPXrtlcRZIkzdRgC+eMArdl5u0AEXEt8MvAVcAfT3Pec5n58jTHMzOfm+bg++o/\nR8RvAnuBYeDbM6v6G1Wrxc8nnoBVq1q9iiRJmqlSPR8RcTzFL/v7J/dlZgL3ARdOdyqwLSKeiYh7\nI+KiJmVOjoidEbErIr4ZEecdpTpvouhR+XGZ79DozDNhcNB5H5IkdUvZYZfFwHHAnob9eyiGSprZ\nDVwDfAj4IPAU8EBErK4r8wOKnpP3Ax+u1euhiFje7IIREcDngG9n5vdLfocjDA7CypWueJEkqVta\nGXYpJTMfBx6v27U5IioUwzframU2A5snC0TEd4DtFKHlhiaX3QicB1zcjjq63FaSpO4pGz6eBw4B\nSxr2LwGeLXGdLUwTHDLzYER8D6g2HouILwDvA96VmbuPdqPR0VEWLlx4xL6RkRFGRkZe+1ytwgMP\nzLjukiTNG2NjY4yNjR2xb2JioqP3LBU+MvPViNgKrAXugteGQNYCny9xqdUUwzFNRcQA8Bbg7xv2\nfwH4AHBJZs5ofcqtt97KmjVrpi1TqcBf/mXxrpeBvn/yiSSpnzT+gxxgfHyc4eHhjt2zlWGXW4BN\ntRCyhWL45CRgE0BE3AQsz8x1tc/XAzuAR4Eh4GrgUuCyyQtGxGcohl2eoJhI+kngTOBLdWU2AiMU\n80J+EhGTvS8TmXmghe/xmmoVDhyA3bvhtNNmcyVJknQ0pcNHZt5Ze6bHjRTDLduAy+uWyS4Fzqg7\n5QSK54IsB14BHgHWZuaDdWV+AfiL2rkvAluBCzPzsboy11KsbnmgoUrrgdvLfo969cttDR+SJHVW\nSxNOM3MjxaTPZsfWN3zeAGw4yvU+AXziKGU6NiDy5jdDRLHc9pJLOnUXSZIEvtsFgKEhOP10V7xI\nktQNho+aSsUHjUmS1A2Gj5pq1Z4PSZK6wfBRM/mgscxe10SSpPnN8FFTrcLLL8MLL/S6JpIkzW+G\nj5pKpfjp0IskSZ1l+KiZDB9OOpUkqbMMHzULFsCpp9rzIUlSpxk+6rjcVpKkzjN81HG5rSRJnWf4\nqGPPhyRJnWf4qFOtwt69sG9fr2siSdL8Zfio44oXSZI6z/BRp1otfjrvQ5KkzjF81Fm0qFhya/iQ\nJKlzDB91IoreD4ddJEnqHMNHA5fbSpLUWYaPBi63lSSpswwfDapVePppOHCg1zWRJGl+Mnw0qFQg\nE3bs6HVNJEmanwwfDVxuK0lSZxk+GixbBkNDzvuQJKlTDB8NBgaKoRd7PiRJ6gzDRxOGD0mSOsfw\n0YQPGpMkqXMMH01Uq7BzJxw82OuaSJI0/xg+mqhUiuCxa1evayJJ0vxj+GjC5baSJHWO4aOJM8+E\nwUHnfUiS1AmGjyYGB2HlSns+JEnqBMPHFHzBnCRJnWH4mEK1as+HJEmdYPiYwmTPx+HDva6JJEnz\ni+FjCtUqHDgAu3f3uiaSJM0vLYWPiLguInZExP6I2BwRF0xT9pKIONywHYqIU+vKrKvbP1nmldnc\nd7YqleKnQy+SJLVX6fAREVcANwM3AG8FHgbuiYjF05yWwFnA0tq2LDP3NpSZqDu+FFjRhvu2bNUq\niHDSqSRJ7dZKz8cocFtm3p6ZjwHXAq8AVx3lvOcyc+/k1uR4ZmZ9mefadN+WDA3B6afb8yFJUruV\nCh8RcTwwDNw/uS8zE7gPuHC6U4FtEfFMRNwbERc1KXNyROyMiF0R8c2IOK8N950Vl9tKktR+ZXs+\nFgPHAXsa9u+hGCppZjdwDfAh4IPAU8ADEbG6rswPKHow3g98uFavhyJi+SzuO2sut5Ukqf0GO32D\nzHwceLxu1+aIqFAMo6yrldkMbJ4sEBHfAbZThJYbZnP/0dFRFi5ceMS+kZERRkZGjnpupQJ/+7eQ\nWcz/kCRpvhkbG2NsbOyIfRMTEx29Z9nw8TxwCFjSsH8J8GyJ62wBLp7qYGYejIjvAdXZ3vfWW29l\nzZo1Jar2umoVJibghRdgcUemtUqS1FvN/kE+Pj7O8PBwx+5ZatglM18FtgJrJ/dFRNQ+P1TiUqsp\nhmOaiogB4C2TZdp431Iml9s670OSpPZpZdjlFmBTRGyl6MEYBU4CNgFExE3A8sxcV/t8PbADeBQY\nAq4GLgUum7xgRHyGYtjlCeBNwCeBM4EvzfS+nVD/rI93vKNTd5Ekqb+UDh+ZeWft2Ro3Ugx7bAMu\nr1sauxQ4o+6UEyiez7GcYmnsI8DazHywrswvAH9RO/dFil6OC2tLamd637ZbsABOOcVJp5IktVNL\nE04zcyOwcYpj6xs+bwA2HOV6nwA+MZv7dkq16rCLJEnt5LtdjsLltpIktZfh4yh80JgkSe1l+DiK\nahX27oV9+3pdE0mS5gfDx1G43FaSpPYyfBxFtfaYM+d9SJLUHoaPo1i0qFhya8+HJEntYfg4ighX\nvEiS1E6GjxmoVAwfkiS1i+FjBnzQmCRJ7WP4mIFqFZ5+Gg4c6HVNJEk69hk+ZqBSgUzYsaPXNZEk\n6dhn+JgBl9tKktQ+ho8ZWLYMhoac9yFJUjsYPmZgYMAVL5IktYvhY4Z8wZwkSe1h+JghHzQmSVJ7\nGD5mqFKBnTvh4MFe10SSpGOb4WOGqtUieOza1euaSJJ0bDN8zJDLbSVJag/DxwydeSYMDjrpVJKk\n2TJ8zNDgIKxcac+HJEmzZfgoweW2kiTNnuGjBJfbSpI0e4aPEioVePJJOHy41zWRJOnYZfgooVqF\n/fth9+5e10SSpGOX4aOESqX46dCLJEmtM3yUsGoVRDjpVJKk2TB8lDA0BKefbs+HJEmzYfgoyeW2\nkiTNjuGjJJfbSpI0O4aPkiZ7PjJ7XRNJko5Nho+SqlWYmIAXXuh1TSRJOjYZPkqaXG7rvA9JklrT\nUviIiOsiYkdE7I+IzRFxwTRlL4mIww3boYg4dYryV9bKfL1h/0BE/EFEPBkRr0TEExHxe63UfzZ8\n1ockSbMzWPaEiLgCuBn4GLAFGAXuiYizM/P5KU5L4Gxg32s7Mvc2ufZKYAPwYJNrfAq4BvgN4PvA\n24BNEfFSZn6h7Pdo1YIFcMop9nxIktSqVno+RoHbMvP2zHwMuBZ4BbjqKOc9l5l7J7fGgxExANwB\nfBbY0eT8C4G/y8z/lZm7MvPrwL3A21v4DrPiihdJklpXKnxExPHAMHD/5L7MTOA+inAw5anAtoh4\nJiLujYiLmpS5AdiTmV+e4hoPAWsj4qxaXc4HLgb+ocx3aAfDhyRJrSs77LIYOA7Y07B/D3DOFOfs\nphgu+S5wInA18EBEvD0ztwFExDuB9cD509z7j4AFwGMRcYgiOP1uZv5Nye8wa5UK3HNPt+8qSdL8\nUHrOR1mZ+TjweN2uzRFRoRi+WRcRJwO3A1dn5ovTXOoK4D8DV1LM+VgN/GlEPJOZX5nqpNHRURYu\nXHjEvpGREUZGRlr6PlD0fOzdC/v2wc//fMuXkSSp58bGxhgbGzti38TEREfvGVniaVm1YZdXgA9l\n5l11+zcBCzPzV2d4nT8GLs7Mi2vDJ+PAIYrhGXh9OOgQcE5m7oiIXcBNmfnnddf5XeDDmXlek3us\nAbZu3bqVNWvWzPg7zsTmzXDhhfC978Hq1W29tCRJPTc+Ps7w8DDAcGaOt/v6peZ8ZOarwFZg7eS+\niIja54dKXGo1xXAMwGPAW2r7zq9tdwHfqv33U7VyJ1GEkXqHy36HdqhWi5/O+5AkqbxWhl1uoVji\nupXXl9qeBGwCiIibgOWZua72+XqK1SuPAkMUcz4uBS4DyMyfUgyjvCYiXioO5fa63XcDvxcRT9eu\ntaZ27y+18B1mZdGiYsmty20lSSqvdPjIzDsjYjFwI7AE2AZcnpnP1YosBc6oO+UEiueCLKcYsnkE\nWJuZzZ7lMZ2PA38AfBE4FXgG+PPavq6KcMWLJEmtamnCaWZuBDZOcWx9w+cNFA8OK3P99U32/QT4\nRG3ruckXzEmSpHJ8t0uL7PmQJKk1ho8WVavw9NNw4ECvayJJ0rHF8NGiSgUyYUezB8FLkqQpGT5a\n5HJbSZJaY/ho0bJlMDTkpFNJksoyfLRoYKAYerHnQ5Kkcgwfs+ByW0mSyjN8zILLbSVJKs/wMQuV\nCuzcCQcP9romkiQdOwwfs1CtFsFj165e10SSpGOH4WMWXG4rSVJ5ho9ZOPNMGBx00qkkSWUYPmZh\ncBBWrrTnQ5KkMgwfs+RyW0mSyjF8zJLLbSVJKsfwMUuVCjz5JBw+3OuaSJJ0bDB8zFK1Cvv3w+7d\nva6JJEnHBsPHLFUqxU/nfUiSNDOGj1latQoinPchSdJMGT5maWgITj/dng9JkmbK8NEGlYo9H5Ik\nzZThow1cbitJ0swZPtpg8kFjmb2uiSRJc5/how2qVZiYgBde6HVNJEma+wwfbeByW0mSZs7w0QaT\n4cN5H5IkHZ3how0WLIBTTrHnQ5KkmTB8tIkrXiRJmhnDR5tUq/Z8SJI0E4aPNvFBY5IkzYzho02q\nVdi7F/bt63VNJEma2wwfbeJyW0mSZsbw0SbVavHToRdJkqbXUviIiOsiYkdE7I+IzRFxwTRlL4mI\nww3boYg4dYryV9bKfL3JseUR8ZWIeD4iXomIhyNiTSvfod0WLSqW3NrzIUnS9AbLnhARVwA3Ax8D\ntgCjwD0RcXZmPj/FaQmcDbw2IyIz9za59kpgA/Bgk2NvAv4ZuB+4HHgeOAt4sex36IQIl9tKkjQT\npcMHRdi4LTNvB4iIa4FfBq4C/nia857LzJenOhgRA8AdwGeBdwMLG4p8CtiVmR+t2/ej8tXvnMkX\nzEmSpKmVGnaJiOOBYYreBwAyM4H7gAunOxXYFhHPRMS9EXFRkzI3AHsy88tTXONXgO9GxJ0RsSci\nxiPio1OU7Ql7PiRJOrqycz4WA8cBexr27wGWTnHObuAa4EPAB4GngAciYvVkgYh4J7AemC5MrAL+\nK/AD4D8Cfw58PiL+S8nv0DGVCjz9NBw40OuaSJI0d7Uy7FJKZj4OPF63a3NEVCiGb9ZFxMnA7cDV\nmTnd/I0BYEtmfqb2+eGI+EXgWuArHah6adUqZMKOHXDuub2ujSRJc1PZ8PE8cAhY0rB/CfBsiets\nAS6u/XcFWAHcHRFR2zcAEBE/A87JzB0UPSjbG66znaI3ZUqjo6MsXHjk9JGRkRFGRkZKVHdm6pfb\nGj4kSceCsbExxsbGjtg3MTHR0XuWCh+Z+WpEbAXWAncB1ALDWuDzJS61miJMADwGvKXh+B8CJwO/\nTTFMA8VKl3Mayp3DUSad3nrrraxZ053VuMuWwdCQk04lSceOZv8gHx8fZ3h4uGP3bGXY5RZgUy2E\nTC61PQnYBBARNwHLM3Nd7fP1wA7gUWAIuBq4FLgMIDN/Cny//gYR8VJxKOt7Om4F/jkiPg3cCbyD\nYo7I1S18h44YGPAdL5IkHU3p8JGZd0bEYuBGiuGWbcDlmflcrchS4Iy6U06geC7IcuAV4BFgbWa+\n4VkeR7nvdyPiV4E/Aj5DEWiuz8y/KfsdOsnltpIkTa+lCaeZuRHYOMWx9Q2fN1A8OKzM9ddPsf8f\ngH8oc61uq1bhrrt6XQtJkuYu3+3SZpUK7NwJBw/2uiaSJM1Nho82q1aL4LFrV69rIknS3GT4aLNK\npfjpvA9JkpozfLTZihUwOOiKF0mSpmL4aLPBQVi50p4PSZKmYvjoAJ/1IUnS1AwfHeDbbSVJmprh\nowMqFXjySTh8uNc1kSRp7jF8dEC1Cvv3w+7dRy8rSVK/MXx0gMttJUmamuGjA1atggjnfUiS1Izh\nowOGhuC00+z5kCSpGcNHh7jiRZKk5gwfHVKt2vMhSVIzho8OmXzQWGavayJJ0txi+OiQahUmJuCF\nF3pdE0mS5hbDR4e43FaSpOYMHx0yGT6cdCpJ0pEMHx2yYAGccoo9H5IkNTJ8dJDLbSVJeiPDRwdV\nKvZ8SJLUyPDRQfZ8SJL0RoaPDqpWYe9e2Lev1zWRJGnuMHx0kMttJUl6I8NHB1WrxU+HXiRJep3h\no4MWLSqW3NrzIUnS6wwfHRThpFNJkhoZPjrM5baSJB3J8NFh9nxIknQkw0eHVSrw9NNw4ECvayJJ\n0txg+OiwahUyYceOXtdEkqS5wfDRYZPLbZ33IUlSwfDRYcuWwdCQ8z4kSZrUUviIiOsiYkdE7I+I\nzRFxwTRlL4mIww3boYg4dYryV9bKfH2aa36qVuaWVurfTQMDrniRJKle6fAREVcANwM3AG8FHgbu\niYjF05yWwFnA0tq2LDP3Nrn2SmAD8OA0978A+FjtvseESsWeD0mSJrXS8zEK3JaZt2fmY8C1wCvA\nVUc577nM3Du5NR6MiAHgDuCzQNPpmRFxcq3MR4GXWqh7T7jcVpKk15UKHxFxPDAM3D+5LzMTuA+4\ncLpTgW0R8UxE3BsRFzUpcwOwJzO/PM11vgjcnZnfKlPvXqtUYOdOOHiw1zWRJKn3BkuWXwwcB+xp\n2L8HOGeKc3YD1wDfBU4ErgYeiIi3Z+Y2gIh4J7AeOH+qG0fElcBq4G0l69xz1WoRPHbtglWrel0b\nSZJ6q2z4KC0zHwcer9u1OSIqFMM362pDKbcDV2fmi82uERGnA58D3puZr3a6zu1WqRQ/f/hDw4ck\nSWXDx/PAIWBJw/4lwLMlrrMFuLj23xVgBXB3RERt3wBARPyMokflPwCnAON1ZY4D3h0RHwdOrA3/\nvMHo6CgLFy48Yt/IyAgjIyMlqjs7K1bA4GAx7+Oyy7p2W0mSjmpsbIyxsbEj9k1MTHT0njHF7+yp\nT4jYDPxLZl5f+xzALuDzmblhhte4F3g5M38tIk6kCCD1/hA4Gfht4N8ohmtWNJTZBGwH/igztze5\nxxpg69atW1mzZs1Mv17HnHUWfOAD8Cd/0uuaSJI0vfHxcYaHhwGGM3O83ddvZdjlFmBTRGyl6MEY\nBU6iCANExE3A8sxcV/t8PcXqlUeBIYo5H5cClwFk5k+B79ffICJeKg69FioONinzE+CFZsFjLnK5\nrSRJhdLhIzPvrD3T40aK4ZZtwOWZ+VytyFLgjLpTTqB4LshyiiW5jwBrM3PKZ3nMtCqzPL+rqlX4\np3/qdS0kSeq9liacZuZGYOMUx9Y3fN5A8eCwMtdfP4My7ylzzV6rVOCv/goOHy6eeipJUr/y12CX\nVKuwfz/s3t3rmkiS1FuGjy6pX24rSVI/M3x0yapVEOGkU0mSDB9dMjQEp51mz4ckSYaPLvIFc5Ik\nGT66qlq150OSJMNHF00+aKzkQ2UlSZpXDB9dVK3CxAT8+Me9rokkSb1j+OiiyeW2zvuQJPUzw0cX\nGT4kSTJ8dNWCBXDKKU46lST1N8NHl7ncVpLU7wwfXVap2PMhSepvho8us+dDktTvDB9dVq3C3r2w\nb1+vayJJUm8YPrrMt9tKkvqd4aPLqtXip+FDktSvDB9dtmhRseTWeR+SpH5l+OiyCCedSpL6m+Gj\nB1xuK0nqZ4aPHrDnQ5LUzwwfPVCpwNNPw4EDva6JJEndZ/jogWoVMmHHjl7XRJKk7jN89IDLbSVJ\n/czw0QPLlsHQkPM+JEn9yfDRAwMDrniRJPUvw0ePVCr2fEiS+pPho0eqVXs+JEn9yfDRI5VKsdrl\n4MFe10SSpO4yfPRItVoEj127el0TSZK6y/DRI5VK8dOhF0lSvzF89MiKFTA46KRTSVL/MXz0yOBg\nEUDs+ZAk9RvDRw/5gjlJUj9qKXxExHURsSMi9kfE5oi4YJqyl0TE4YbtUEScOkX5K2tlvt6w/9MR\nsSUiXo6IPRHxjYg4u5X6zxUut5Uk9aPS4SMirgBuBm4A3go8DNwTEYunOS2Bs4CltW1ZZu5tcu2V\nwAbgwSbXeBfwZ8A7gPcCxwP3RsTPlf0Oc8XkU04ze10TSZK6p5Wej1Hgtsy8PTMfA64FXgGuOsp5\nz2Xm3smt8WBEDAB3AJ8F3vC+18x8X2Z+JTO3Z+b/BX4TOBMYbuE7zAnVKuzfD7t397omkiR1T6nw\nERHHU/yyv39yX2YmcB9w4XSnAtsi4pmIuDciLmpS5gZgT2Z+eYbVeRNFj8qPZ1h+zplcbuu8D0lS\nPynb87EYOA7Y07B/D8VwSjO7gWuADwEfBJ4CHoiI1ZMFIuKdwHrgozOpREQE8Dng25n5/TJfYC5Z\ntQoiDB+SpP4y2OkbZObjwON1uzZHRIVi+GZdRJwM3A5cnZkvzvCyG4HzgIuPVnB0dJSFCxcesW9k\nZISRkZEZ3qpzhobgtNOcdCpJ6p2xsTHGxsaO2DcxMdHRe5YNH88Dh4AlDfuXAM+WuM4WXg8OFWAF\ncHetRwNqPTIR8TPgnMx8bQ5IRHwBeB/wrsw86myJW2+9lTVr1pSoWne53FaS1EvN/kE+Pj7O8HDn\nplSWGnbJzFeBrcDayX21wLAWeKjEpVZTDMcAPAa8pbbv/Np2F/Ct2n8/VXevLwAfAC7NzHnxVpTJ\nFS+SJPWLVoZdbgE2RcRWih6MUeAkYBNARNwELM/MdbXP11OsXnkUGAKuBi4FLgPIzJ8CR8zbiIiX\nikO5vW7fRmAEeD/wk4iY7H2ZyMwDLXyPOaFaha99rVhu+1q/jyRJ81jp8JGZd9ae6XEjxXDLNuDy\nzHyuVmQpcEbdKSdQPBdkOcWS3EeAtZnZ7Fke07mWYnXLAw3711PMGTkmVaswMQE//jEsWtTr2kiS\n1HktTTjNzI0Ukz6bHVvf8HkDxYPDylx/fZN98/JR8PXLbQ0fkqR+MC9/oR9LJsOH8z4kSf3C8NFj\nCxbAKae44kWS1D8MH3OAy20lSf3E8DEHuNxWktRPDB9zgD0fkqR+YviYAyoV2LsX9u3rdU0kSeo8\nw8ccUK0WPx16kST1A8PHHGD4kCT1E8PHHLBoUbHk1nkfkqR+YPiYAyKK3g97PiRJ/cDwMUdUKvZ8\nSJL6g+FjjrDnQ5LULwwfc0SlAk89BQcO9LomkiR1luFjjqhWIRN27Oh1TSRJ6izDxxzh220lSf3C\n8DFHLF8OQ0NOOpUkzX+GjzliYMAXzEmS+oPhYw5xua0kqR8YPuYQl9tKkvqB4WMOqVSK1S4HD/a6\nJpIkdY7hYw6pVovg8dRTva6JJEmdY/iYQyaX2zrvQ5I0nxk+5pAVK2Bw0PAhSZrfDB9zyOBgEUCc\ndCpJms8cRBHMAAAHSUlEQVQMH3NMtWrPhyRpfjN8zDEut5UkzXeGjzlm8imnmb2uiSRJnWH4mGOq\nVdi/H3bv7nVNJEnqDMPHHONyW0nSfGf4mGNWrYII531IkuavwV5XQEcaGoLTToNvfMPHrEuSeuNH\nP+rs9Q0fc9All8Bf/zXcfXevayJJUvs57DIH3XFHsdrlWNy++tWxnteh3zbb3Dbvh8027+62dWtn\nf8+1FD4i4rqI2BER+yNic0RcME3ZSyLicMN2KCJOnaL8lbUyX5/NfdUbY2Njva5C37HNu8827z7b\nfH4pHT4i4grgZuAG4K3Aw8A9EbF4mtMSOAtYWtuWZebeJtdeCWwAHmzTfSVJ0hzTSs/HKHBbZt6e\nmY8B1wKvAFcd5bznMnPv5NZ4MCIGgDuAzwI72nhfSZI0h5QKHxFxPDAM3D+5LzMTuA+4cLpTgW0R\n8UxE3BsRFzUpcwOwJzO/3Mb7SpKkOabsapfFwHHAnob9e4BzpjhnN3AN8F3gROBq4IGIeHtmbgOI\niHcC64Hz23jfIYDt27dP9V3UARMTE4yPj/e6Gn3FNu8+27z7bPPuqvvdOdSJ63d8qW1mPg48Xrdr\nc0RUKIZR1kXEycDtwNWZ+WIbb70S4CMf+UgbL6mZGB4e7nUV+o5t3n22effZ5j2xEnio3RctGz6e\nBw4BSxr2LwGeLXGdLcDFtf+uACuAuyMiavsGACLiZxQ9G0+3cN97gA8DO4EDJeomSVK/G6IIHvd0\n4uKlwkdmvhoRW4G1wF0AtcCwFvh8iUutphiOAXgMeEvD8T8ETgZ+G3gqMw+WvW9mvgB8tUSdJEnS\n69re4zGplWGXW4BNtTCwhWL45CRgE0BE3AQsz8x1tc/XU6xeeZQiSV0NXApcBpCZPwW+X3+DiHip\nOJT1Ezamva8kSTo2lA4fmXln7dkaN1IMe2wDLs/M52pFlgJn1J1yAsXzOZZTLI19BFibmW94lscs\n7ytJko4BUaxYlSRJ6g7f7SJJkrrK8CFJkrpq3oYPX0LXPhHxroi4KyL+X+2lf+9vUubG2hNsX4mI\nf4yIasPxEyPiixHxfETsi4ivTfVywX4XEZ+OiC0R8XJE7ImIb0TE2U3K2eZtEhHXRsTDETFR2x6K\niP/UUMb27qCI+FTt75dbGvbb7m0SETc0edFr44KPrrT3vAwfvoSu7f4dxQTf36J4SeARIuJ3gI8D\nHwPeDvyEor1PqCv2OeCXgQ8B76aYgPw/O1vtY9a7gD8D3gG8FzgeuDcifm6ygG3edk8BvwOsoXiV\nw7eAv4uIc8H27rTaPw4/RvF3df1+2739/pVi0cbki17fOXmgq+2dmfNuAzYDf1r3OSgeVPbJXtft\nWN+Aw8D7G/Y9A4zWfV4A7Ad+ve7zT4FfrStzTu1ab+/1d5rrG8XrBQ4D77TNu9ruLwDrbe+Ot/PJ\nwA+A9wD/G7il7pjt3t62vgEYn+Z419p73vV8+BK67oqIN1Ok5/r2fhn4F15v77dRLOuuL/MDYBf+\nP5mJN1H0OP0YbPNOi4iBiLiS4jlCD9neHfdF4O7M/Fb9Ttu9Y86qDaH/MCLuiIgzoPvt3fF3u/RA\nKy+hU+uWUvxibNbeS2v/vQT4We0P8lRl1ETtSb6fA76dmZNjs7Z5B0TELwLfoXgY4j6Kf939ICIu\nxPbuiFrIW03xS62Rf87bbzPwmxQ9TcuA3wcerP3Z72p7z8fwIc0nG4HzeP1dSOqcxyjerL0Q+DXg\n9oh4d2+rNH9FxOkUwfq9mflqr+vTDzKz/j0t/xoRW4AfAb9O8ee/a+bdsAvte/mdZuZZijk107X3\ns8AJEbFgmjJqEBFfAN4H/FJm7q47ZJt3QGYezMwnM/N7mfm7FJMfr8f27pRh4BRgPCJejYhXgUuA\n66N4qegebPeOyswJirfOV+nyn/N5Fz5qCXryJXTAES+h69hLcvpVZu6g+ENX394LKFZqTLb3VuBg\nQ5lzgDMpurnVoBY8PgBcmpm76o/Z5l0zAJxoe3fMfRQvFV1N0eN0PvBd4A7g/Mx8Etu9oyLiZIrg\n8UzX/5z3evZth2b0/jrFe2R+A/j3wG0UM9dP6XXdjsWNYqnt+RR/SRwG/lvt8xm145+ste+vUPxl\n8k3g34AT6q6xkeIFg79E8S+efwb+T6+/21zcam31IsWS2yV121BdGdu8vW3+P2rtvQL4ReCm2l+y\n77G9u/r/oXG1i+3e3vbdQLE8dgVwEfCPFD1Mi7rd3j1vjA428m8BOymWCX0HeFuv63SsbhRdoYcp\nhrPqt7+qK/P7FMu0XgHuAaoN1ziR4tkVz1NM5vtb4NRef7e5uE3R1oeA32goZ5u3r82/BDxZ+/vi\nWeDeyeBhe3f1/8O36sOH7d729h2jeOzEfooVKl8F3tyL9vbFcpIkqavm3ZwPSZI0txk+JElSVxk+\nJElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSVxk+JElSV/1/OkUp\nJNEeTWUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f389b2affd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "optimal number of components: 10\n",
      "training svm ...\n",
      "predicting labels with svm ...\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          1       0.49      0.98      0.65       487\n",
      "          2       0.77      0.08      0.14       532\n",
      "\n",
      "avg / total       0.64      0.51      0.38      1019\n",
      "\n",
      "[[475  12]\n",
      " [492  40]]\n",
      "accuracy on sub test set (PCA): 0.505397 \n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      0.99      0.99       537\n",
      "          1       0.49      0.97      0.65       491\n",
      "          2       0.77      0.08      0.14       532\n",
      "          3       0.92      0.97      0.94       496\n",
      "          4       0.93      0.86      0.89       420\n",
      "          5       0.91      0.93      0.92       471\n",
      "\n",
      "avg / total       0.84      0.79      0.75      2947\n",
      "\n",
      "[[532   0   0   0   5   0]\n",
      " [  2 475  12   0   1   1]\n",
      " [  0 492  40   0   0   0]\n",
      " [  0   0   0 481  15   0]\n",
      " [  0   0   0  19 361  40]\n",
      " [  0   0   0  23   8 440]]\n",
      "accuracy on test set (relabeling): 0.790295 \n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('Relabeling ...')\n",
    "\n",
    "# This function selects the data whose ground truth label is in a given label list.\n",
    "def get_train_data(X, y, labels=[]):\n",
    "    \"\"\"\n",
    "    Select data has the label in a given label list.\n",
    "    :param X: features\n",
    "    :param y: labels\n",
    "    :param labels: a label list\n",
    "    :return: selected data\n",
    "    \"\"\"\n",
    "    X_sub = []\n",
    "    y_sub = []\n",
    "    for i in range(len(X)):\n",
    "        if int(y[i]) in labels:\n",
    "            X_sub.append(X[i])\n",
    "            y_sub.append(y[i])\n",
    "    return np.array(X_sub), np.array(y_sub)\n",
    "\n",
    "# This function select the data whose predicted label is in a given label list.\n",
    "def get_test_data(X_test, y_test, y_pred, labels=[]):\n",
    "    \"\"\"\n",
    "    Select data has the predicted label in a given label list.\n",
    "    :param X_test: features\n",
    "    :param y_test: ground truth labels\n",
    "    :param y_pred: predicted labels\n",
    "    :param labels: a label list\n",
    "    :return: selected data\n",
    "    \"\"\"\n",
    "    X_sub = []\n",
    "    y_sub = []\n",
    "    for i in range(len(X_test)):\n",
    "        if int(y_pred[i]) in labels:\n",
    "            X_sub.append(X_test[i])\n",
    "            y_sub.append(y_test[i])\n",
    "    return np.array(X_sub), np.array(y_sub)\n",
    "\n",
    "\n",
    "sub_labels = [1, 2]\n",
    "\n",
    "# get a sub class of data in order to train a classifier to separate the data in the sub class.\n",
    "X_train_sub, y_train_sub = get_train_data(X_train, y_train, sub_labels)\n",
    "\n",
    "# split the sub class data into training and validation sets.\n",
    "X_train_sub, y_train_sub, X_val_sub, y_val_sub = split_data(X_train_sub, y_train_sub, 0.1)\n",
    "\n",
    "# get a sub class of test data whose predicted labels from the previous step are in [1, 2]\n",
    "X_test_sub, y_test_sub = get_test_data(X_test, y_test, y_pred, sub_labels)\n",
    "\n",
    "# find optimal number of principla components\n",
    "n_components = get_n_components(X_train_sub, y_train_sub, X_val_sub, y_val_sub, n_components_list)\n",
    "print('optimal number of components: %d' % n_components)\n",
    "\n",
    "# relabel the test data of the sub class\n",
    "y_pred_sub, acc = classification_pca(X_train_sub, y_train_sub, X_test_sub, y_test_sub, n_components, grid_search=False)\n",
    "\n",
    "\n",
    "print(\"accuracy on sub test set (PCA): %f \" % acc)\n",
    "\n",
    "# evaluation the whole test set\n",
    "y_test_all = []\n",
    "y_pred_all = []\n",
    "\n",
    "for i in range(len(y_pred)):\n",
    "    if int(y_pred[i]) not in sub_labels:\n",
    "        y_test_all.append(y_test[i])\n",
    "        y_pred_all.append(y_pred[i])\n",
    "\n",
    "for i in range(len(y_pred_sub)):\n",
    "    y_test_all.append(y_test_sub[i])\n",
    "    y_pred_all.append(y_pred_sub[i])\n",
    "\n",
    "y_test_all = np.array(y_test_all)\n",
    "y_pred_all = np.array(y_pred_all)\n",
    "\n",
    "acc = np.sum(y_test_all == y_pred_all) / len(y_test_all)\n",
    "\n",
    "print(classification_report(y_test_all, y_pred_all))\n",
    "print(confusion_matrix(y_test_all, y_pred_all))\n",
    "print(\"accuracy on test set (relabeling): %f \" % acc)\n",
    "\n",
    "print('Done.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This script shows one solution of predicting the label of unseen data (test data) by using machine learning methods. The data is cleaned by converting string values to integer values, replacing missing value with the mean of the column, and standardization. With an SVM, trained on the given features, I achieve about 92% classification accuracy. In order to reduce the dimensionality of the features, PCA is applied. I show that by applying a optimal number of principal components searching method, with 500 features, I achieve comparable performance compared to using all the features. While the original feature size is 564. I also find that the data from class 1 and class 2 is similar. The given features are not discriminative enough to separate the two classes. Therefore, one future work is to discover more discriminative features in order to separated the two classes. Collecting more training data may also increase the performance of the classification. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}