examples/Vulture/00-single_node_parallel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "254cd3b1-e36b-42b5-9297-9b9d174a1011",
   "metadata": {},
   "source": [
    "# Vulture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f74105-f696-41c1-ad8e-1414f1e874b4",
   "metadata": {},
   "source": [
    "## Introduction to Text Pre-Processing with Vulture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8d1ed393-f5c5-400e-a922-f146bbcf55a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VultureModuleBase()\n"
     ]
    }
   ],
   "source": [
    "from TELF.pre_processing.Vulture.modules import VultureModuleBase\n",
    "\n",
    "vmb = VultureModuleBase()\n",
    "print(vmb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0da57bb7-c21f-478d-a322-ae9de19a4f59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1c0a0ad2-f847-4cdb-a35e-7ad4f298505b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import pathlib\n",
    "import pandas as pd\n",
    "\n",
    "from TELF.pre_processing import Vulture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1dc990-7de8-40e6-baca-0ecef35c6477",
   "metadata": {},
   "source": [
    "## 0. Load Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b12074f-691b-4a4a-b070-ec94307f3fd4",
   "metadata": {},
   "source": [
    "### Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "521bce7b-4e83-4639-b784-a066c22478c6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_DIR = os.path.join('..', '..', 'data')\n",
    "DATA_DIR = pathlib.Path(DATA_DIR).resolve()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d292d91b-c0da-487b-bb34-bdcee4fc4075",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_FILE = 'documents.p'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6f73de82-bd13-4c8b-acf3-5c0d11073fb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = pickle.load(open(os.path.join(DATA_DIR, DATA_FILE), 'rb'))\n",
    "len(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387ef96c-492f-4eab-a72e-bcdb3417efa8",
   "metadata": {},
   "source": [
    "### Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "21aaaf2c-8c07-4126-a68e-aeebb4d01b59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "RESULTS_DIR = 'results'\n",
    "RESULTS_DIR = pathlib.Path(RESULTS_DIR).resolve()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6f278716-0ba2-4037-8146-59e1386cf0da",
   "metadata": {},
   "outputs": [],
   "source": [
    "RESULTS_FILE = 'clean_documents.p'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eeec913a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    os.mkdir(RESULTS_DIR)\n",
    "except FileExistsError:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "845d3357-ac44-464e-b9f8-cfd683697a38",
   "metadata": {},
   "source": [
    "### Examine Data Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d76a0f09-cdcf-4cf2-ac54-1362136e1736",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ad68055e-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680626-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680658-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680680-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806a8-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806d0-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad6806f8-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad680716-677f-11ee-95d4-4ab2673ea3f0',\n",
       " 'ad68073e-677f-11ee-95d4-4ab2673ea3f0']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# key serve as document unique ids\n",
    "list(documents.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b2f2f25a-c3ad-40b1-bba1-652833367173",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# values are the text that needs to be cleaned\n",
    "documents[next(iter(documents))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a708c17b-f4ad-480c-b31f-f221e034a981",
   "metadata": {},
   "source": [
    "## 1. Clean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da2deb7d-a3a4-4ac1-8232-00b3a116d3fd",
   "metadata": {},
   "source": [
    "The Vulture library is composed of multiple cleaning modules that work together to process the text. These modules are flexible and their order can be re-arranged depending on the user's preferences. By default Vulture implements a simple cleaning pipeline so that new users can quickly get started. In this section we will examine the Vulture default pipeline and apply the default cleaning to the sample text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b316563-4e41-40cc-a31b-f56451b2d102",
   "metadata": {},
   "source": [
    "The pipeline is a just list of Vulture modules that are to be updated sequentially. The default pipeline contains a single module - the ```SimpleCleaner```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1c59c011-9d3c-41a4-8785-53cf45cf96ab",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[SimpleCleaner(module_type='CLEANER', effective_stop_words=['characteristics', 'characteristic', 'acknowledgment', 'predominantly', 'investigation', 'corresponding', 'significantly', 'approximately', 'unfortunately', 'automatically', 'substantially', 'specifically', 'demonstrated', 'introduction', 'applications', 'nevertheless', 'successfully', 'respectively', 'sufficiently', 'particularly', 'demonstrates', 'consequently', 'representing', 'appropriate', 'significant', ... (+1359 more)], patterns={'standardize_hyphens': (re.compile('[\\\\u002D\\\\u2010\\\\u2011\\\\u2012\\\\u2013\\\\u2014\\\\u2015\\\\u2212\\\\u2E3A\\\\u2E3B]'), '-'), 'remove_copyright_statement': None, 'remove_stop_phrases': None, 'make_lower_case': None, 'normalize': None, 'remove_trailing_dash': ('(?<!\\\\w)-|-(?!\\\\w)', ''), 'make_hyphens_words': ('([a-z])\\\\-([a-z])', ''), 'remove_next_line': ('\\\\n+', ' '), 'remove_email': ('\\\\S*@\\\\S*\\\\s?', ''), 'remove_formulas': ('\\\\b\\\\w*[\\\\=\\\\≈\\\\/\\\\\\\\\\\\±]\\\\w*\\\\b', ''), 'remove_dash': ('-', ''), 'remove_between_[]': ('\\\\[.*?\\\\]', ' '), 'remove_between_()': ('\\\\(.*?\\\\)', ' '), 'remove_[]': ('[\\\\[\\\\]]', ' '), 'remove_()': ('[()]', ' '), 'remove_\\\\': ('\\\\\\\\', ' '), 'remove_numbers': ('\\\\d+', ''), 'remove_standalone_numbers': ('\\\\b\\\\d+\\\\b', ''), 'remove_nonASCII_boundary': ('\\\\b[^\\\\x00-\\\\x7F]+\\\\b', ''), 'remove_nonASCII': ('[^\\\\x00-\\\\x7F]+', ''), 'remove_tags': ('&lt;/?.*?&gt;', ''), 'remove_special_characters': ('[!|\"|#|$|%|&|\\\\|\\\\\\'|(|)|*|+|,|.|/|:|;|<|=|>|?|@|[|\\\\|]|^|_|`|{|\\\\||}|~]', ''), 'isolate_frozen': None, 'remove_extra_whitespace': ('\\\\s+', ' '), 'remove_stop_words': None, 'min_characters': None}, exclude_hyphenated_stopwords=False, sw_pattern=re.compile('\\\\b[\\\\w-]+\\\\b'))]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Vulture.DEFAULT_PIPELINE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61e35977-c2aa-47ed-ae03-e04c527e4932",
   "metadata": {},
   "source": [
    "The most important attribute of the ```SimpleCleaner``` is ```order```. This attribute specifies which cleaning operations will be applied as well as the order that they will be applied in. These cleaning operations are all pre-defined and the full list of supported operations can be examined with ```SimpleCleaner.DEFAULT_PATTERNS```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "98da040e-798e-4b7e-8803-e828a5443843",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['standardize_hyphens',\n",
       " 'isolate_frozen',\n",
       " 'remove_copyright_statement',\n",
       " 'remove_stop_phrases',\n",
       " 'make_lower_case',\n",
       " 'remove_formulas',\n",
       " 'normalize',\n",
       " 'remove_next_line',\n",
       " 'remove_email',\n",
       " 'remove_()',\n",
       " 'remove_[]',\n",
       " 'remove_special_characters',\n",
       " 'remove_nonASCII_boundary',\n",
       " 'remove_nonASCII',\n",
       " 'remove_tags',\n",
       " 'remove_stop_words',\n",
       " 'remove_standalone_numbers',\n",
       " 'remove_extra_whitespace',\n",
       " 'min_characters']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Vulture.DEFAULT_PIPELINE[0].order"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "302365f7-5aa8-4e8f-befa-a2e94eaf0f70",
   "metadata": {},
   "source": [
    "### Setup Vulture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41a53e96-23d6-4e10-a538-430e6d6bc43b",
   "metadata": {},
   "source": [
    "Create a single-node single-process Vulture object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ea8b8813-e8c7-42bc-9aa0-538187487841",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "vulture = Vulture(n_jobs  = 1, \n",
    "                  verbose = 10,  # Disable == 0, Verbose >= 1\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f936d14-4fd1-4f43-bac7-0fc95348c72e",
   "metadata": {},
   "source": [
    "### Apply Default Clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d409a060-2039-4e05-a2a0-a64851ea4b2f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/maksim/Desktop/Code/telf_internal/TELF/pre_processing/Vulture/vulture.py:465: UserWarning: The file \"/Users/maksim/Desktop/Code/telf_internal/examples/Vulture/results/clean_documents.p\" already exists and will be overwritten!\n",
      "  warnings.warn(f'The file \"{save_path}\" already exists and will be overwritten!')\n",
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|          | 0/1 [00:00<?, ?it/s][Vulture]: Running SimpleCleaner module\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 332.63it/s]\n",
      "100%|██████████| 1/1 [00:00<00:00, 23.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.7 ms, sys: 17.7 ms, total: 45.5 ms\n",
      "Wall time: 61.6 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%time vulture.clean(documents, \\\n",
    "                    save_path=os.path.join(RESULTS_DIR, RESULTS_FILE))                   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a96259b6-c967-4102-913a-9e866e4ee111",
   "metadata": {},
   "source": [
    "### Look at Cleaned Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9aad256c-2645-4f1e-b98a-f44132e61c15",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "clean_documents = pickle.load(open(os.path.join(RESULTS_DIR, RESULTS_FILE), 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "69859fe2-48e5-4a8e-8b8d-ccc64abd51b7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def to_df(documents, clean_documents):\n",
    "    data = {\n",
    "        'id': [],\n",
    "        'text': [],\n",
    "        'clean_text': []\n",
    "    }\n",
    "\n",
    "    for i, text in documents.items():\n",
    "        data['id'].append(i)\n",
    "        data['text'].append(text)\n",
    "        data['clean_text'].append(clean_documents.get(i))\n",
    "\n",
    "    return pd.DataFrame.from_dict(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "087aa962-1a60-40c3-8de2-be90df8a975c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ad68055e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Supervisory Control and Data Acquisition (SCAD...</td>\n",
       "      <td>supervisory control acquisition scada serve ne...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ad680626-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Highly specific datasets of scientific literat...</td>\n",
       "      <td>highly specific scientific literature educatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ad680658-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient distributed out-of-mem...</td>\n",
       "      <td>efficient distributed implementation matrix fa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ad680680-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Identification of the family to which a malwar...</td>\n",
       "      <td>identification family malware specimen belongs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ad6806a8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>malware dangerous costly cyber threats nationa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad6806d0-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>malware dangerous costly cyber threats organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ad6806f8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Topic modeling is one of the key analytic tech...</td>\n",
       "      <td>topic modeling key analytic techniques organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ad680716-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Non-negative matrix factorization (NMF) with m...</td>\n",
       "      <td>matrix factorization nmf completion collaborat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ad68073e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient, distributed, out-of-m...</td>\n",
       "      <td>efficient distributed implementation truncated...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  \\\n",
       "0  ad68055e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "1  ad680626-677f-11ee-95d4-4ab2673ea3f0   \n",
       "2  ad680658-677f-11ee-95d4-4ab2673ea3f0   \n",
       "3  ad680680-677f-11ee-95d4-4ab2673ea3f0   \n",
       "4  ad6806a8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "5  ad6806d0-677f-11ee-95d4-4ab2673ea3f0   \n",
       "6  ad6806f8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "7  ad680716-677f-11ee-95d4-4ab2673ea3f0   \n",
       "8  ad68073e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "\n",
       "                                                text  \\\n",
       "0  Supervisory Control and Data Acquisition (SCAD...   \n",
       "1  Highly specific datasets of scientific literat...   \n",
       "2  We propose an efficient distributed out-of-mem...   \n",
       "3  Identification of the family to which a malwar...   \n",
       "4  Malware is one of the most dangerous and costl...   \n",
       "5  Malware is one of the most dangerous and costl...   \n",
       "6  Topic modeling is one of the key analytic tech...   \n",
       "7  Non-negative matrix factorization (NMF) with m...   \n",
       "8  We propose an efficient, distributed, out-of-m...   \n",
       "\n",
       "                                          clean_text  \n",
       "0  supervisory control acquisition scada serve ne...  \n",
       "1  highly specific scientific literature educatio...  \n",
       "2  efficient distributed implementation matrix fa...  \n",
       "3  identification family malware specimen belongs...  \n",
       "4  malware dangerous costly cyber threats nationa...  \n",
       "5  malware dangerous costly cyber threats organiz...  \n",
       "6  topic modeling key analytic techniques organiz...  \n",
       "7  matrix factorization nmf completion collaborat...  \n",
       "8  efficient distributed implementation truncated...  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = to_df(documents, clean_documents)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b1a48a6-2780-47f9-b9e6-452f87cdf494",
   "metadata": {},
   "source": [
    "#### Look at First Document Full-Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c24b8a19-df51-42c2-92d4-d4a8be09c8f3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " '',\n",
       " 'supervisory control acquisition scada serve nervous substations power grids facilitate monitoring acquisition control equipment ensure smooth efficient operation substation connected devices dependence scada grows risk potential malicious intrusions outages permanent damage grid dimensionality approaches principal analysis pca accurate identification anomalies scada scada matrix factorization nmf strong detecting anomalies wireless sensor networks unsupervised approaches normal expected behavior detect unseen types attacks anomalies identifying events deviate expected behavior approaches complex multi-dimensional interactions naturally scada differently tensor decomposition powerful unsupervised machine learning complex multi-faceted activity details scada events novelly tensor decomposition canonical polyadic alternating poisson regression probabilistic framework anomaly cyber network identify anomalies scada showcase statistical behavior analysis scada communication tensor decomposition improves specificity accuracy identifying anomalies electrical grid experiments scada electrical grid operated alamos national laboratory lanl transmission distribution service partnership alamos county detect synthetically generated anomalies')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = 0\n",
    "df.iloc[index].text, '', df.iloc[index].clean_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef16606c-c553-46bd-a953-317e2369eacd",
   "metadata": {},
   "source": [
    "### Add Some Substitutions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4cfd75e1-5f33-4515-b272-374798a7ceb4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "substitutions = {\n",
    "    'los alamos national laboratory': 'LANL',\n",
    "    'lanl': 'LANL',\n",
    "    'los alamos': 'Los_Alamos',\n",
    "    'supervisory control and data acquisition': 'SCADA',\n",
    "    'scada': 'SCADA',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "65a4eede-8dfa-46de-ac62-f9552b4ecb51",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/maksim/miniconda3/envs/TELF/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "[Vulture]: Cleaning 9 documents\n",
      "  0%|          | 0/3 [00:00<?, ?it/s][Vulture]: Running SubstitutionCleaner module\n",
      "100%|██████████| 9/9 [00:00<00:00, 1059.08it/s]\n",
      "[Vulture]: Running SimpleCleaner module\n",
      "100%|██████████| 9/9 [00:00<00:00, 374.97it/s]\n",
      "[Vulture]: Running SubstitutionCleaner module\n",
      "100%|██████████| 9/9 [00:00<00:00, 14358.59it/s]\n",
      "100%|██████████| 3/3 [00:00<00:00, 79.68it/s]\n"
     ]
    }
   ],
   "source": [
    "clean_documents_sub = vulture.clean(documents, \n",
    "                                    save_path=None, # if None, dict is returned from function\n",
    "                                    substitutions=substitutions)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "561e069b-44ff-4947-809d-1e96cda9753d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ad68055e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Supervisory Control and Data Acquisition (SCAD...</td>\n",
       "      <td>SCADA SCADA serve nervous substations power gr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ad680626-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Highly specific datasets of scientific literat...</td>\n",
       "      <td>highly specific scientific literature educatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ad680658-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient distributed out-of-mem...</td>\n",
       "      <td>efficient distributed implementation matrix fa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ad680680-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Identification of the family to which a malwar...</td>\n",
       "      <td>identification family malware specimen belongs...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ad6806a8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>malware dangerous costly cyber threats nationa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ad6806d0-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Malware is one of the most dangerous and costl...</td>\n",
       "      <td>malware dangerous costly cyber threats organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ad6806f8-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Topic modeling is one of the key analytic tech...</td>\n",
       "      <td>topic modeling key analytic techniques organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ad680716-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>Non-negative matrix factorization (NMF) with m...</td>\n",
       "      <td>matrix factorization nmf completion collaborat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ad68073e-677f-11ee-95d4-4ab2673ea3f0</td>\n",
       "      <td>We propose an efficient, distributed, out-of-m...</td>\n",
       "      <td>efficient distributed implementation truncated...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  \\\n",
       "0  ad68055e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "1  ad680626-677f-11ee-95d4-4ab2673ea3f0   \n",
       "2  ad680658-677f-11ee-95d4-4ab2673ea3f0   \n",
       "3  ad680680-677f-11ee-95d4-4ab2673ea3f0   \n",
       "4  ad6806a8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "5  ad6806d0-677f-11ee-95d4-4ab2673ea3f0   \n",
       "6  ad6806f8-677f-11ee-95d4-4ab2673ea3f0   \n",
       "7  ad680716-677f-11ee-95d4-4ab2673ea3f0   \n",
       "8  ad68073e-677f-11ee-95d4-4ab2673ea3f0   \n",
       "\n",
       "                                                text  \\\n",
       "0  Supervisory Control and Data Acquisition (SCAD...   \n",
       "1  Highly specific datasets of scientific literat...   \n",
       "2  We propose an efficient distributed out-of-mem...   \n",
       "3  Identification of the family to which a malwar...   \n",
       "4  Malware is one of the most dangerous and costl...   \n",
       "5  Malware is one of the most dangerous and costl...   \n",
       "6  Topic modeling is one of the key analytic tech...   \n",
       "7  Non-negative matrix factorization (NMF) with m...   \n",
       "8  We propose an efficient, distributed, out-of-m...   \n",
       "\n",
       "                                          clean_text  \n",
       "0  SCADA SCADA serve nervous substations power gr...  \n",
       "1  highly specific scientific literature educatio...  \n",
       "2  efficient distributed implementation matrix fa...  \n",
       "3  identification family malware specimen belongs...  \n",
       "4  malware dangerous costly cyber threats nationa...  \n",
       "5  malware dangerous costly cyber threats organiz...  \n",
       "6  topic modeling key analytic techniques organiz...  \n",
       "7  matrix factorization nmf completion collaborat...  \n",
       "8  efficient distributed implementation truncated...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sub = to_df(documents, clean_documents_sub)\n",
    "df_sub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a1b6097c-2260-497d-98a4-cb973c8f228e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Supervisory Control and Data Acquisition (SCADA) systems often serve as the nervous system for substations within power grids. These systems facilitate real-time monitoring, data acquisition, control of equipment, and ensure smooth and efficient operation of the substation and its connected devices. As the dependence on these SCADA systems grows, so does the risk of potential malicious intrusions that could lead to significant outages or even permanent damage to the grid. Previous work has shown that dimensionality reduction-based approaches, such as Principal Component Analysis (PCA), can be used for accurate identification of anomalies in SCADA systems. While not specifically applied to SCADA, non-negative matrix factorization (NMF) has shown strong results at detecting anomalies in wireless sensor networks. These unsupervised approaches model the normal or expected behavior and detect the unseen types of attacks or anomalies by identifying the events that deviate from the expected behavior. These approaches; however, do not model the complex and multi-dimensional interactions that are naturally present in SCADA systems. Differently, non-negative tensor decomposition is a powerful unsupervised machine learning (ML) method that can model the complex and multi-faceted activity details of SCADA events. In this work, we novelly apply the tensor decomposition method Canonical Polyadic Alternating Poisson Regression (CP-APR) with a probabilistic framework, which has previously shown state-of-the-art anomaly detection results on cyber network data, to identify anomalies in SCADA systems. We showcase that the use of statistical behavior analysis of SCADA communication with tensor decomposition improves the specificity and accuracy of identifying anomalies in electrical grid systems. In our experiments, we model real-world SCADA system data collected from the electrical grid operated by Los Alamos National Laboratory (LANL) which provides transmission and distribution service through a partnership with Los Alamos County, and detect synthetically generated anomalies.',\n",
       " '',\n",
       " 'SCADA SCADA serve nervous substations power grids facilitate monitoring acquisition control equipment ensure smooth efficient operation substation connected devices dependence SCADA grows risk potential malicious intrusions outages permanent damage grid dimensionality approaches principal analysis pca accurate identification anomalies SCADA SCADA matrix factorization nmf strong detecting anomalies wireless sensor networks unsupervised approaches normal expected behavior detect unseen types attacks anomalies identifying events deviate expected behavior approaches complex multi-dimensional interactions naturally SCADA differently tensor decomposition powerful unsupervised machine learning complex multi-faceted activity details SCADA events novelly tensor decomposition canonical polyadic alternating poisson regression probabilistic framework anomaly cyber network identify anomalies SCADA showcase statistical behavior analysis SCADA communication tensor decomposition improves specificity accuracy identifying anomalies electrical grid experiments SCADA electrical grid operated LANL LANL transmission distribution service partnership Los_Alamos county detect synthetically generated anomalies')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sub.iloc[index].text, '', df_sub.iloc[index].clean_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee179741-2172-4a56-931f-14d1109a44a4",
   "metadata": {},
   "source": [
    "## 2. Look at Top Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5578f1a8-f0d3-44b0-ae9b-e63498f720a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from TELF.pre_processing.Vulture.tokens_analysis.top_words import get_top_words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87f2c54b-2e85-44e0-8010-0ad3432783e4",
   "metadata": {},
   "source": [
    "### Before Substitution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0a6c007b-32e2-460a-9410-eabc50d1d43e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 22442.77it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>tf</th>\n",
       "      <th>df</th>\n",
       "      <th>df_fraction</th>\n",
       "      <th>tf_fraction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>malware</td>\n",
       "      <td>41</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.077947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>matrix</td>\n",
       "      <td>21</td>\n",
       "      <td>6</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.039924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>scada</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.017110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>family</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.017110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>topics</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.015209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>events</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>framework</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>accuracy</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>electrical</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>alamos</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003802</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          word  tf  df  df_fraction  tf_fraction\n",
       "0      malware  41   3     0.333333     0.077947\n",
       "1       matrix  21   6     0.666667     0.039924\n",
       "2        scada   9   1     0.111111     0.017110\n",
       "3       family   9   3     0.333333     0.017110\n",
       "4       topics   8   2     0.222222     0.015209\n",
       "..         ...  ..  ..          ...          ...\n",
       "95      events   2   1     0.111111     0.003802\n",
       "96   framework   2   2     0.222222     0.003802\n",
       "97    accuracy   2   2     0.222222     0.003802\n",
       "98  electrical   2   1     0.111111     0.003802\n",
       "99      alamos   2   1     0.111111     0.003802\n",
       "\n",
       "[100 rows x 5 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_words_df = get_top_words(clean_documents, \n",
    "                             top_n=100, \n",
    "                             n_gram=1, \n",
    "                             verbose=True,\n",
    "                             filename=None)\n",
    "top_words_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "190f66fe-7253-4b4d-acce-2127e83b2a7f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 21029.94it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>tf</th>\n",
       "      <th>df</th>\n",
       "      <th>df_fraction</th>\n",
       "      <th>tf_fraction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>malware families</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.008083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>malware family</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.006928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>matrix factorization</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>0.555556</td>\n",
       "      <td>0.005774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>factorization nmf</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>0.004619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>machine learning</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.004619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>topic modeling</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.004619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dense matrix</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.004619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>tensor decomposition</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sparse matrix</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>memory complexity</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003464</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   word  tf  df  df_fraction  tf_fraction\n",
       "0      malware families   7   3     0.333333     0.008083\n",
       "1        malware family   6   3     0.333333     0.006928\n",
       "2  matrix factorization   5   5     0.555556     0.005774\n",
       "3     factorization nmf   4   4     0.444444     0.004619\n",
       "4      machine learning   4   3     0.333333     0.004619\n",
       "5        topic modeling   4   2     0.222222     0.004619\n",
       "6          dense matrix   4   2     0.222222     0.004619\n",
       "7  tensor decomposition   3   1     0.111111     0.003464\n",
       "8         sparse matrix   3   2     0.222222     0.003464\n",
       "9     memory complexity   3   2     0.222222     0.003464"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_2grams_df = get_top_words(clean_documents, \n",
    "                             top_n=10, \n",
    "                             n_gram=2, \n",
    "                             verbose=True,\n",
    "                             filename=None,)\n",
    "top_2grams_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d041f617-91ba-4fd7-bbbd-1256496326a9",
   "metadata": {},
   "source": [
    "### After Substitution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1cd45e03-e615-4ecd-b261-55f14398a714",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 7157.52it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>tf</th>\n",
       "      <th>df</th>\n",
       "      <th>df_fraction</th>\n",
       "      <th>tf_fraction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>malware</td>\n",
       "      <td>41</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.078244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>matrix</td>\n",
       "      <td>21</td>\n",
       "      <td>6</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.040076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>SCADA</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.019084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>family</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.017176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>topics</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.015267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>electrical</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>LANL</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>specific</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>scientific</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>constructively</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003817</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              word  tf  df  df_fraction  tf_fraction\n",
       "0          malware  41   3     0.333333     0.078244\n",
       "1           matrix  21   6     0.666667     0.040076\n",
       "2            SCADA  10   1     0.111111     0.019084\n",
       "3           family   9   3     0.333333     0.017176\n",
       "4           topics   8   2     0.222222     0.015267\n",
       "..             ...  ..  ..          ...          ...\n",
       "95      electrical   2   1     0.111111     0.003817\n",
       "96            LANL   2   1     0.111111     0.003817\n",
       "97        specific   2   1     0.111111     0.003817\n",
       "98      scientific   2   1     0.111111     0.003817\n",
       "99  constructively   2   1     0.111111     0.003817\n",
       "\n",
       "[100 rows x 5 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_words_df = get_top_words(clean_documents_sub, \n",
    "                             top_n=100, \n",
    "                             n_gram=1, \n",
    "                             verbose=True,\n",
    "                             filename=None)\n",
    "top_words_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1653a3e0-5bbc-4620-810e-52fbcbda7e1a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 9355.32it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>tf</th>\n",
       "      <th>df</th>\n",
       "      <th>df_fraction</th>\n",
       "      <th>tf_fraction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>malware families</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.008130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>malware family</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.006969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>matrix factorization</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>0.555556</td>\n",
       "      <td>0.005807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>factorization nmf</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.444444</td>\n",
       "      <td>0.004646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>machine learning</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.004646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>topic modeling</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.004646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>dense matrix</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.004646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>tensor decomposition</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>0.003484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sparse matrix</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>memory complexity</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.003484</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   word  tf  df  df_fraction  tf_fraction\n",
       "0      malware families   7   3     0.333333     0.008130\n",
       "1        malware family   6   3     0.333333     0.006969\n",
       "2  matrix factorization   5   5     0.555556     0.005807\n",
       "3     factorization nmf   4   4     0.444444     0.004646\n",
       "4      machine learning   4   3     0.333333     0.004646\n",
       "5        topic modeling   4   2     0.222222     0.004646\n",
       "6          dense matrix   4   2     0.222222     0.004646\n",
       "7  tensor decomposition   3   1     0.111111     0.003484\n",
       "8         sparse matrix   3   2     0.222222     0.003484\n",
       "9     memory complexity   3   2     0.222222     0.003484"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_2grams_df = get_top_words(clean_documents_sub, \n",
    "                             top_n=10, \n",
    "                             n_gram=2, \n",
    "                             verbose=True,\n",
    "                             filename=None,)\n",
    "top_2grams_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TELF",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}