notebooks/03-traffic-category.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77de2d6b",
   "metadata": {},
   "source": [
    "# <b>Task03 - Traffic Category Classification</b>\n",
    "___\n",
    "\n",
    "In the third task, we try to classify the traffic category the collected flows belong to. \n",
    "\n",
    "We use the `ISCXVPN2016` dataset, which is designed for the classification of encrypted flow traffic and includes five categories of non-tunnelled traffic: VOIP, streaming, chat, mail, and file transfer. The resulting dataset is small, with only 609 flows that are well-balanced among the five classes (coefficient 0.82). Each flow is characterized by multiple measurements including the server IP address, the first 32 bytes of the application payload, statistics from the TCP payload, and the sequence of the first 32 receiver window sizes, packet inter-arrival times, and lengths. This dataset presents a challenge for deep learning training due to its small size and is used to test the performance of multi-modal autoencoders in scenarios with limited data.\n",
    "\n",
    "**Note** In this notebook we report only the validation of the models and the experiments _without the training_. If you want to inspect our training approach or run again a model training see README (**Training the models** section).\n",
    "\n",
    "# Table of Content\n",
    "- Configuration\n",
    "- Load features\n",
    "- Validate the models\n",
    "- k-nearest-neighborhood class probability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cfe6dbb-5cb5-4d72-9631-ec3811acfb1f",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "Before we begin, we need to set up our environment and load the necessary libraries and modules. We also need to specify the paths to the data files and define some global variables that will be used throughout the notebook. \n",
    "\n",
    "The `DEMO` flag controls whether we are running the notebook in demonstration mode (`True`) or full mode (`False`). In demonstration mode some experiments will be run with less samples and the output will not be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "822a33e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-09 10:41:08.290541: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "# Make mltoolbox and utls reachable from this folder\n",
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "from utils import*\n",
    "\n",
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
    "\n",
    "# Features and embeddings paths\n",
    "FEATURES = '../data/task03/features'\n",
    "EMBEDDINGS = '../data/task03/embeddings'\n",
    "INTERIM = '../data/interim'\n",
    "\n",
    "# Demonstrative flag\n",
    "DEMO = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd149fc9",
   "metadata": {},
   "source": [
    "## Load features\n",
    "\n",
    "In this section, we will load the data files that contain the features for our machine learning models. We will use the `pandas` library to read in the CSV files and store the data in dataframes. \n",
    "- The `ipaddress` dataframe will contain the word2vec embeddings for the IP addresses\n",
    "- The `payload` dataframe will contain the payload bytes\n",
    "- The `statistics` dataframe will contain Tstat-style features\n",
    "- The `sequences` dataframe will contain statistical features in sequence referred to each byte. \n",
    "\n",
    "The data in these dataframes will be used as input to our models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2b25c42f-a76b-45ea-a28d-66670f344b68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load ip address word2vec embeddings - entity\n",
    "ipaddress=pd.read_csv(f'{FEATURES}/ipaddress.csv', index_col=[0])\n",
    "# Load payload bytes - quantity\n",
    "payload=pd.read_csv(f'{FEATURES}/payload.csv', index_col=[0])\n",
    "# Load statistics features - quantity\n",
    "statistics=pd.read_csv(f'{FEATURES}/statistics.csv', index_col=[0])\n",
    "# Load statistics sequences - quantity\n",
    "sequences=pd.read_csv(f'{FEATURES}/sequences.csv', index_col=[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01063a4d-e1d0-4b35-9692-cd41a559410d",
   "metadata": {},
   "source": [
    "Then we merge the dataframes containing our features into a single dataframe called concat. We start by resetting the index of the payload dataframe and dropping the 'label' column. Then, we perform an inner join on the 'index' column with the statistics dataframe, also dropping the 'label' column. We repeat this process for the sequences dataframe and the ipaddress dataframe. Finally, we set the 'index' column as the index of the resulting dataframe. This results in a single dataframe that contains all of the features for our models, with the 'index' column serving as the primary key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4f528cb9-d28e-4bd3-8ca2-2aeeb27c84f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the features as raw concatenation\n",
    "concat = payload.reset_index().drop(columns=['label'])\\\n",
    "                .merge(statistics.reset_index().drop(columns=['label']), \n",
    "                       on='index', how='inner')\\\n",
    "                .merge(sequences.reset_index().drop(columns=['label']), \n",
    "                       on='index', how='inner')\\\n",
    "                .merge(ipaddress.reset_index(), on='index', how='inner')\\\n",
    "                .set_index('index')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8a1317-6b74-4a82-a378-9960a33ef2d5",
   "metadata": {},
   "source": [
    "Finally, we collect the features sets in a dictionary, we load the stratified-k-folds order we provide and retrieve the number of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f89fb0f4-04ee-4f47-9553-09ee2d0a5e22",
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "\n",
    "# Collect the features in a dictionary\n",
    "features = {'payload':payload, 'statistics':statistics,\n",
    "            'sequences':sequences, 'ipaddress':ipaddress,\n",
    "            'rawcat':concat, 'mae':None}\n",
    "\n",
    "# Load stratified k folds\n",
    "kfolds = joblib.load(f'../data/task03/skfolds/folds.save')\n",
    "\n",
    "# Get the number of classes\n",
    "n_classes = ipaddress.value_counts('label').shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df3e42bb",
   "metadata": {},
   "source": [
    "## Validate the models\n",
    "\n",
    "After having trained the models through the training scripts, we need to validate them.\n",
    "\n",
    "The following function is responsible for evaluating the pre-trained classifiers using cross-validation.  The model predicts the labels of each one of the provided fold at a time. It then generates a summary of the model's performance on the validation set in the form of a classification report, which includes metrics such as precision, recall, and f1-score. The function can be called multiple times with different values of K in order to validate the model's performance on all of the folds of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f5f205b4-2f4c-4bda-83b0-a7e817227fb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.notebook import tqdm\n",
    "from mltoolbox.classification import DeepClassifier\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "def validate_single_run(feature, fname, K, pbar):\n",
    "    # Retrieve the training and validation samples from the k-folds order\n",
    "    X_train, X_val, y_train, y_val = get_datasets(kfolds, K, feature)\n",
    "    \n",
    "    # Load the classifier model from the specified file path\n",
    "    mpath = f'../data/task03/classifiers/{fname}_k{K}'\n",
    "    classifier = DeepClassifier(_load_model=True, model_path=mpath)\n",
    "    \n",
    "    # Use the classifier to predict labels for the validation set\n",
    "    y_pred = classifier.predict(X_val, scale_data=True)\n",
    "    report = classification_report(y_val, y_pred, labels=np.unique(y_val), \n",
    "                                   output_dict=True)\n",
    "    \n",
    "     # Extract the macro average f1-score from the report\n",
    "    f1 = round(report['macro avg']['f1-score'], 2)\n",
    "    \n",
    "    # Update the progress bar object and set the postfix message\n",
    "    pbar.update(1)\n",
    "    pbar.set_postfix({'current fold':K, 'macro avg. f1': f1})\n",
    "    \n",
    "    return report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fb86bff-2c6f-403b-a17a-8479d9c88f16",
   "metadata": {},
   "source": [
    "Now we can validate the models. Namely, we run a full stratified-k-folds cross validation over:\n",
    "- Raw features independently\n",
    "- Concatenation of the raw features\n",
    "- Multi-modal embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3670704d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4546fe12e0c4e3285ae66c8c8c07250",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34fd1a523b834db89a0e61bbd3860800",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac223964c58642c582ecc5fd4104219d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bcfc82a6dc2942dbb025000dacc5ea36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f38754c74f93435796b58393fd8c5523",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0952f54bcab140b38ab0d98574b7f468",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for fname, feature in features.items():\n",
    "    # Initialize a progress bar with a total of 5 iterations (skf)\n",
    "    pbar = tqdm(total=5)\n",
    "    pbar.set_description(f'Validating {fname}')\n",
    "    \n",
    "    # Iterate over the stratified folds\n",
    "    for K in range(5):\n",
    "        if fname == 'mae':\n",
    "            # Load the pre-trained multimodal embeddings\n",
    "            feature=pd.read_csv(f'{EMBEDDINGS}/mae_embeddings_k{K}.csv', \n",
    "                               index_col=[0])\n",
    "        # Validate the classifier getting the classification metrics\n",
    "        report = validate_single_run(feature, fname, K, pbar)\n",
    "        \n",
    "        # Save the report to a CSV file if not demonstrative\n",
    "        if not DEMO:\n",
    "            pd.DataFrame(report).T.to_csv(f'{INTERIM}/task03_{fname}_deep_k{K}.csv')\n",
    "            \n",
    "    # Close the progress bar       \n",
    "    pbar.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c1771e2",
   "metadata": {},
   "source": [
    "## k-nearest-neighborhood class probability\n",
    "\n",
    "We now evaluate the embeddings space through the k-nearest-neighborhood class probability. It consists on applying a k-nearest-neighbors classifier on the whole dataset. Then, for each sample whose label is different from `unknown` (if present) we compute the probability of having samples with the same label in their neighborhood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a5e79d1e-782a-4356-a07f-c00a3f15f556",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mltoolbox.classification import KnnClassifier\n",
    "from mltoolbox.metrics import k_class_proba_report\n",
    "\n",
    "def kpc_single_run(feature, fname, K, k, pbar):\n",
    "    # Retrieve the training and validation datasets for the current fold\n",
    "    X_train, X_val, y_train, y_val = get_datasets(kfolds, K, feature)\n",
    "    X, y = np.vstack([X_train, X_val]), np.hstack([y_train, y_val])\n",
    "    \n",
    "    # If demonstrative load less samples\n",
    "    if DEMO: X, y = X[:3000], y[:3000]\n",
    "    \n",
    "    # Train a KNN classifier with cosine similarity and the specified number \n",
    "    # of neighbors\n",
    "    knn = KnnClassifier(n_neighbors=k, metric='cosine')\n",
    "    knn.fit(X, y, scale_data=True)\n",
    "    \n",
    "    # Keep only the samples with labels other than 'unknown' and predict labels\n",
    "    to_keep = np.where(y!='unknown')[0].reshape(-1, 1)\n",
    "    pcs = knn.predict_proba(to_keep)\n",
    "    \n",
    "    # Generate a report with the k-class probabilities\n",
    "    y_true = y[np.ravel(to_keep)]# Extract the true labels\n",
    "    report = k_class_proba_report(y_true, pcs, output_dict=True)\n",
    "    \n",
    "    # Extract the macro average k-class probability from the report\n",
    "    kpc = report['macro avg']['kpc']\n",
    "    \n",
    "    # Update the progress bar and set the postfix message\n",
    "    pbar.update(1)\n",
    "    pbar.set_postfix({'current fold':K, f'{k}Pc': kpc})\n",
    "\n",
    "    return report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89edc1da-0123-409a-a73b-1f78ebf1ae7a",
   "metadata": {},
   "source": [
    "We evaluate the neighborhood for the Multi-modal embeddings and the concatenation of the raw features. We average the experiment on the 5 folds. \n",
    "\n",
    "Note that, since we want to evaluate the embeddings neighborhood, we do not need to distinguish between training/validation samples, thus we merge together the subsets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "daa16574",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53ad93501a424614af60875b9b77d7c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/20 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "314ddc2f88574b2b981069f61a0b8de1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/20 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ranges = range(1, 20) # Ranges of neighborhood radius\n",
    "# If demonstrative limit the runs\n",
    "if DEMO: ranges = range(1, 5)\n",
    "\n",
    "# Evaluate only the 'rawcat' and 'mae' features\n",
    "for fname, feature in features.items():\n",
    "    if fname in ['rawcat', 'mae']:\n",
    "        # Initialize a progress bar\n",
    "        pbar = tqdm(total=len(ranges)*5)\n",
    "        pbar.set_description(f'Evaluating {fname} neighborhood')\n",
    "        \n",
    "        # Iterate over the stratified folds\n",
    "        for K in range(5):\n",
    "            for k in ranges: # Try different neighborhood radious\n",
    "                if fname == 'mae':\n",
    "                    # Load the pre-trained multimodal embeddings\n",
    "                    feature=pd.read_csv(f'{EMBEDDINGS}/mae_embeddings_k{K}.csv', \n",
    "                                       index_col=[0])\n",
    "                # Compute the class probability\n",
    "                report = kpc_single_run(feature, fname, K, k, pbar)\n",
    "                \n",
    "                # Save the report to a CSV file if not demonstrative\n",
    "                if not DEMO:\n",
    "                    pd.DataFrame(report).T.to_csv(f'{INTERIM}/task03_{fname}_{k}pc_k{K}.csv')\n",
    "        \n",
    "        # Close the progress bar \n",
    "        pbar.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "81dc8a169688350846d75f5b64795c14310fd5665d0ae2bea435e43803ad8b3b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}