Report_JacobDanovitch_CSI4107A1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='text-align: center;'>\n",
    "<h1>Joogle - CSI4107 Search Engine Project</h1>\n",
    "<h4>By: Jacob Danovitch</h4>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Foreword\n",
    "\n",
    "The following is my report for the search engine project. Please note that rather than running my code, it is much easier to go to its website online at https://joogle-csi4107.herokuapp.com. Instructions are in `README.md`.\n",
    "\n",
    "This report contains a system architecture diagram, followed by explanations and demonstrations of the modules implemented in my program, and concludes with the demonstration screenshots. The bulk of the length of this report is due to the explanations/demonstrations of the modules, so if these are unnecessary you may skip directly to [Demo](#Demo) for the example queries suggested by Professor Barriere.\n",
    "\n",
    "I recommend viewing the HTML version of this report, or viewing it on [NBViewer](https://nbviewer.jupyter.org/github/jacobdanovitch/Joogle/blob/master/Report_JacobDanovitch_CSI4107A1.ipynb). However, a PDF version is also included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## System Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following diagram was created using [draw.io](https://draw.io) and their Google Cloud Platform templates.\n",
    "\n",
    "<img src='report_img/Joogle_Architecture.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules - Mandatory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `uottawa_scrape.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module scrapes and parses the uOttawa course catalogue using `requests` and `beautifulsoup4`, saving the data to [`data/catalogue-uottawa-ca.json`](data/catalogue-uottawa-ca.json).\n",
    "\n",
    "#### Demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the HTML is acquired using the `requests` library, and a `BeautifulSoup` object is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CSI 1306 Computing Concepts for Business (3 units)\n",
      "\n",
      "Introduction to computer-based problem solving from the perspective of the business world. Design of algorithms for solving business problems. Basics of computer programming in a modern programming language. Solving business problems using application packages including spreadsheets and databases. Basics of web design. Collaborative tools. Using open source software.Course Component: Laboratory, LectureThe courses ITI 1120, GNG 1106, CSI 1306, CSI 1308, CSI 1390 cannot be combined for credits.\n"
     ]
    }
   ],
   "source": [
    "from uottawa_scrape import *\n",
    "\n",
    "html = scrape(to_file=False)\n",
    "print(html.find(\"div\", {\"class\": \"courseblock\"}).text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTML is then parsed using `beautifulsoup4`, such that it is stored in the following format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 0, 'title': 'CSI 1306 Computing Concepts for Business (3 units)', 'body': 'Introduction to computer-based problem solving from the perspective of the business world. Design of algorithms for solving business problems. Basics of computer programming in a modern programming language. Solving business problems using application packages including spreadsheets and databases. Basics of web design. Collaborative tools. Using open source software.'}\n"
     ]
    }
   ],
   "source": [
    "output = to_json(html, filename=None)\n",
    "print(output[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This format facilitates easily manipulating and rendering the data using other modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `app.py`\n",
    "* `templates/`\n",
    "    * `static/`\n",
    "        * `css/`\n",
    "        * `js/`\n",
    "        * `img/`\n",
    "    * `index.html`\n",
    "    * `layouts.html`\n",
    "    * `about.html`\n",
    "    * `404.html`\n",
    "\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module is implemented as a small Flask app. The UI is a parody of Google (see acknowledgements section below), allowing users to search the catalogue using either retrieval module.\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "See [Demo](#Demo) below for a full demonstration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionary building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `build_dictionary.py`\n",
    "* `construct_index.py`\n",
    "* `retrieval_model.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module contains utilities used to build the dictionary for the retrieval models. The actual construction of the dictionary takes place in `retrieval_model.py`.\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "Note that due to the inclusion of the optional [phrase query indexing](#Phrase-Query-Indexing) module, said module is also a part of the dictionary building process. This will be expanded upon further in its own section below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we reference the data created from the scraping model as seen above. We narrow the data to focus only on the descriptions (the titles were **not** made part of the dictionary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Introduction to computer-based problem solving from the perspective of the business world. Design of algorithms for solving business problems. Basics of computer programming in a modern programming language. Solving business problems using application packages including spreadsheets and databases. Basics of web design. Collaborative tools. Using open source software.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "data = json.load(open(\"data/catalogue-uottawa-ca.json\"))\n",
    "descriptions = [row['body'] for row in data]\n",
    "descriptions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These descriptions are then passed to the `clean` function from `build_dictionary.py`, which performs casing, normalization, lemmatization, stopword removal, and tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source', 'open', 'perspective', 'language', 'problem', 'programming', 'package', 'including', 'collaborative', 'computer', 'spreadsheet', 'based', 'software', 'business', 'web', 'tool', 'modern', 'using', 'algorithm', 'design', 'introduction', 'world', 'solving', 'application', 'database', 'basic'}\n"
     ]
    }
   ],
   "source": [
    "from build_dictionary import clean\n",
    "print(clean(descriptions[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverted Index Construction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `construct_index.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "#### Demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The descriptions above are passed to the `build_postings` function, which returns both the index and the term dictionary. The index is constructed first, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, ['source', 'open', 'spreadsheet', 'perspective', 'language', 'problem', 'package', 'programming', 'web', 'modern', 'database', 'problem', 'using', 'algorithm', 'including', 'collaborative', 'computer', 'design', 'introduction', 'world', 'solving', 'tool', 'application', 'based', 'software', 'business', 'basic'])\n"
     ]
    }
   ],
   "source": [
    "from construct_index import build_postings\n",
    "\n",
    "index, term_dict = build_postings(descriptions)\n",
    "print(index[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, the term dictionary is constructed from the index. As it is, of course, a dictionary, retrieval is completed in **constant** $\\mathcal{O}(1)$ time. \n",
    "\n",
    "Its construction is completed in $\\mathcal{O}(n \\cdot (m+1))$ time, where $m$ is the number of words in the longest document in the corpus. The $n$ documents contained in the `.json` data are iterated over once to construct the index (as seen above) containing $n$ rows. Then, each of the $n$ rows of the index is  to construct the term dictionary. At each row, $m$ words are iterated over, incrementally updating its associated count. Pseudocode is presented below.\n",
    "\n",
    "```python\n",
    "for (doc_id, words) in index:\n",
    "    for word in words:\n",
    "        term_dict[doc_id][word] += 1\n",
    "```\n",
    "\n",
    "The outer loop performs $n$ iterations of $m$ steps each for $O(n\\cdot m)$ time. Combined with the $n$ iterations of the index construction, we have $\\mathcal{O}(n+n\\cdot m) = \\mathcal{O}(n\\cdot(m+1))$ time.\n",
    "\n",
    "Finally, both are returned.\n",
    "\n",
    "**Note**: As an implementation decision, the weights are calculated later, in the `VSM` model itself. This simply made for a cleaner and more organized project structure. The weights could have been included here otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 9: 2, 17: 1, 21: 1, 31: 2, 34: 1, 37: 1, 39: 1, 49: 1, 51: 1, 52: 1, 56: 1, 67: 1, 72: 1, 74: 1, 75: 1, 76: 1, 77: 1, 79: 1, 85: 1, 88: 1, 89: 1, 94: 1}\n"
     ]
    }
   ],
   "source": [
    "print(term_dict['algorithm'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus Access"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* same as in [User Interface](#User-Interface)\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module is implemented within the user interface, seen within the search results.\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "See [Demo](#Demo) below for a full demonstration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean model of information retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `brm.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "The `BRM` (Boolean Retrieval Model) class operates as a model to query the corpus using boolean searches.\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "The `boolean.py` library is used for symbolic manipulation. The query is parsed as a logical expression, and then each literal word of the query is cleaned using the previous module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wildcards: []\n",
      "computer OR systems AND data\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "OR(Symbol('computer'), AND(Symbol('systems'), Symbol('data')))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from brm import *\n",
    "\n",
    "b = BRM()\n",
    "q = \"(computer OR systems) AND (data)\"\n",
    "\n",
    "b.preprocess_query(q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each document, the truth value of each symbol is evaluated as the presence of the word in the document to resolve the expression. Each of the $n$ documents are evaluated, for a query complexity of $\\mathcal{O}(n)$.\n",
    "\n",
    "As the `BRM` is unranked, the `top_n` argument will simply return the first `n` documents found and is only useful for demonstration purposes (as in this instance) or for the UI to paginate the results (not yet implemented)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wildcards: []\n",
      "computer OR systems AND data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CSI 1306 Computing Concepts for Business (3 un...</td>\n",
       "      <td>Introduction to computer-based problem solving...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CSI 1308 Introduction to Computing Concepts (3...</td>\n",
       "      <td>Introduction to computer based problem solving...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CSI 1390 Introduction to Computers (3 units)</td>\n",
       "      <td>Computing and computers. Problem solving and a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CSI 2101 Discrete Structures (3 units)</td>\n",
       "      <td>Discrete structures as they apply to computer ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>CSI 4109 Introduction to Distributed Computing...</td>\n",
       "      <td>Computational models. Communication complexity...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                title  \\\n",
       "0   CSI 1306 Computing Concepts for Business (3 un...   \n",
       "1   CSI 1308 Introduction to Computing Concepts (3...   \n",
       "2        CSI 1390 Introduction to Computers (3 units)   \n",
       "3              CSI 2101 Discrete Structures (3 units)   \n",
       "21  CSI 4109 Introduction to Distributed Computing...   \n",
       "\n",
       "                                                 body  \n",
       "0   Introduction to computer-based problem solving...  \n",
       "1   Introduction to computer based problem solving...  \n",
       "2   Computing and computers. Problem solving and a...  \n",
       "3   Discrete structures as they apply to computer ...  \n",
       "21  Computational models. Communication complexity...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.query(q, top_n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector Space Model of information retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `vsm.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "The `VSM` (Vector Space Model) class operates as a model to query the corpus using $\\mathtt{tf-idf}$ retrieval. \n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "The $\\mathtt{tf-idf}$ scores are computed for each $w_i, d_i$ pair upon construction of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>94</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "      <th>101</th>\n",
       "      <th>102</th>\n",
       "      <th>103</th>\n",
       "      <th>104</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>source</th>\n",
       "      <td>1.711807</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>open</th>\n",
       "      <td>1.711807</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spreadsheet</th>\n",
       "      <td>2.012837</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perspective</th>\n",
       "      <td>1.313867</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>language</th>\n",
       "      <td>0.808717</td>\n",
       "      <td>0.808717</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.808717</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.808717</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  0         1    2    3    4         5    6    7         8    \\\n",
       "source       1.711807  0.000000  0.0  0.0  0.0  0.000000  0.0  0.0  0.000000   \n",
       "open         1.711807  0.000000  0.0  0.0  0.0  0.000000  0.0  0.0  0.000000   \n",
       "spreadsheet  2.012837  0.000000  0.0  0.0  0.0  0.000000  0.0  0.0  0.000000   \n",
       "perspective  1.313867  0.000000  0.0  0.0  0.0  0.000000  0.0  0.0  0.000000   \n",
       "language     0.808717  0.808717  0.0  0.0  0.0  0.808717  0.0  0.0  0.808717   \n",
       "\n",
       "             9   ...   94   96   97   98   99   100  101  102  103  104  \n",
       "source       0.0 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "open         0.0 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "spreadsheet  0.0 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "perspective  0.0 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "language     0.0 ...   0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "\n",
       "[5 rows x 103 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from vsm import *\n",
    "\n",
    "v = VSM()\n",
    "v.d_w.T.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At runtime, the query is pre-processed in the same fashion as the documents were processed orginally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'operating', 'system'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = \"operating systems\"\n",
    "q = clean(q)\n",
    "q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only relevant documents are selected from the weight matrix. As the matrix is a $2$-d array, this access takes place in $\\mathcal{O}(w)$ time, where $w$ is the number of words in the processed query, which is effectively constant $\\mathcal{O}(1)$ time (assuming the user does not query every word in the vocabulary).\n",
    "\n",
    "This is mathematically equivalent to performing an inner product on the weight matrix with one-hot vectors for each word, but it is clearly more space-efficient to simply slice the rows in question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>94</th>\n",
       "      <th>96</th>\n",
       "      <th>97</th>\n",
       "      <th>98</th>\n",
       "      <th>99</th>\n",
       "      <th>100</th>\n",
       "      <th>101</th>\n",
       "      <th>102</th>\n",
       "      <th>103</th>\n",
       "      <th>104</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>system</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>operating</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0    1    2    3    4    5    6    7    8    9   ...   94   96   \\\n",
       "system     0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "operating  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 ...   0.0  0.0   \n",
       "\n",
       "           97   98   99   100  101  102  103  104  \n",
       "system     0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "operating  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  \n",
       "\n",
       "[2 rows x 103 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.d_w.loc[:, q].T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The matrix is then summed along each column (corresponding to a document in the corpus), and sorted in descending order. This is then returned to our original data set and used to return the relevant documents to the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30    2.573724\n",
       "12    2.573724\n",
       "90    2.573724\n",
       "83    1.992251\n",
       "91    1.162947\n",
       "dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.d_w.loc[:, q].T.apply(sum).sort_values(ascending=False)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"confidence\" score is obtained by using the $\\mathtt{tf-idf}$ score through the sigmoid function (the s-curve), squishing all values between $[0,1]$ with more extreme values on both ends. As of writing, this is not currently displayed to the user, but could be in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>body</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>CSI 4139 Design of Secure Computer Systems (3 ...</td>\n",
       "      <td>Security policies. Security mechanisms. Physic...</td>\n",
       "      <td>0.929151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CSI 3131 Operating Systems (3 units)</td>\n",
       "      <td>Principles of operating systems. Operating sys...</td>\n",
       "      <td>0.929151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>CSI 5312 Distributed Operating Systems Enginee...</td>\n",
       "      <td>Design issues of advanced multiprocessor distr...</td>\n",
       "      <td>0.929151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>CSI 5175 Mobile Commerce Technologies (3 units)</td>\n",
       "      <td>Wireless networks support for m-commerce; m-co...</td>\n",
       "      <td>0.879981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>CSI 5314 Object-Oriented Software Development ...</td>\n",
       "      <td>Issues in modeling and verifying quality and v...</td>\n",
       "      <td>0.761868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>CSI 5134 Fault Tolerance (3 units)</td>\n",
       "      <td>Hardware and software techniques for fault tol...</td>\n",
       "      <td>0.761868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>CSI 4141 Real Time Systems Design (3 units)</td>\n",
       "      <td>Definition of real-time systems;  examples.  C...</td>\n",
       "      <td>0.761868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>CSI 5311 Distributed Databases and Transaction...</td>\n",
       "      <td>Principles involved in the design and implemen...</td>\n",
       "      <td>0.761868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>CSI 4124 Foundation of Modelling and Simulatio...</td>\n",
       "      <td>The modelling and simulation process from a pr...</td>\n",
       "      <td>0.761868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>CSI 5100 Data Integration (3 units)</td>\n",
       "      <td>Materialized and virtual approaches to integra...</td>\n",
       "      <td>0.641406</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                title  \\\n",
       "id                                                      \n",
       "30  CSI 4139 Design of Secure Computer Systems (3 ...   \n",
       "12               CSI 3131 Operating Systems (3 units)   \n",
       "90  CSI 5312 Distributed Operating Systems Enginee...   \n",
       "83    CSI 5175 Mobile Commerce Technologies (3 units)   \n",
       "91  CSI 5314 Object-Oriented Software Development ...   \n",
       "56                 CSI 5134 Fault Tolerance (3 units)   \n",
       "32        CSI 4141 Real Time Systems Design (3 units)   \n",
       "89  CSI 5311 Distributed Databases and Transaction...   \n",
       "24  CSI 4124 Foundation of Modelling and Simulatio...   \n",
       "35                CSI 5100 Data Integration (3 units)   \n",
       "\n",
       "                                                 body  confidence  \n",
       "id                                                                 \n",
       "30  Security policies. Security mechanisms. Physic...    0.929151  \n",
       "12  Principles of operating systems. Operating sys...    0.929151  \n",
       "90  Design issues of advanced multiprocessor distr...    0.929151  \n",
       "83  Wireless networks support for m-commerce; m-co...    0.879981  \n",
       "91  Issues in modeling and verifying quality and v...    0.761868  \n",
       "56  Hardware and software techniques for fault tol...    0.761868  \n",
       "32  Definition of real-time systems;  examples.  C...    0.761868  \n",
       "89  Principles involved in the design and implemen...    0.761868  \n",
       "24  The modelling and simulation process from a pr...    0.761868  \n",
       "35  Materialized and virtual approaches to integra...    0.641406  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.query(\"operating systems\", top_n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules - Optional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase Query Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `phrase_indexing.py`\n",
    "\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module uses the Jaccard coefficient to identify phrases from a candidate set (identified as hyphenated words), as seen in class.\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "First, the candidate set is assembled as all hyphenated words in the corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('computer', 'based'),\n",
       " ('object', 'oriented'),\n",
       " ('binary', 'trees'),\n",
       " ('object', 'oriented'),\n",
       " ('constraint', 'based')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from phrase_indexing import *\n",
    "\n",
    "candidates = identify_candidates(descriptions)\n",
    "candidates[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, bigrams of the corpus text are constructed, for calculation of Jaccard coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'entity-relationship': 1.0, 'one-way': 1.0, 'trap-door': 1.0, 'grey-box': 1.0, 'well-separated': 1.0, 'fail-safe': 1.0, 'locality-sensitive': 1.0, 'large-scale': 1.0}\n"
     ]
    }
   ],
   "source": [
    "corpus = [remove_punc(t.lower(), rm_hyphens=True) for t in descriptions]\n",
    "bigrams = make_bigrams(corpus)\n",
    "\n",
    "print(find_phrases(candidates, bigrams, threshold=0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, the threshold can be tuned as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'object-oriented': 0.5294117647058824, 'entity-relationship': 1.0, 'one-way': 1.0, 'trap-door': 1.0, 'lambda-calculus': 0.6666666666666666, 'grey-box': 1.0, 'well-separated': 1.0, 'fail-safe': 1.0, 'locality-sensitive': 1.0, 'large-scale': 1.0, 'fault-tolerance': 0.5555555555555556}\n"
     ]
    }
   ],
   "source": [
    "print(find_phrases(candidates, bigrams, threshold=0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These phrases are then stored for later use in preprocessing the corpus and user queries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spelling Correction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### File(s) \n",
    "\n",
    "* `spelling.py`\n",
    "\n",
    "#### Description\n",
    "\n",
    "This module performs spelling corrections on user queries. However, as this is an extra module (working alone, I only have to implement one), I took some liberties with my implementation and elected to use a character-gram model instead of minimum edit distance (mostly out of curiosity).\n",
    "\n",
    "#### Demonstration\n",
    "\n",
    "First, as usual, the query is pre-processed (note: this happens slightly differently for the `BRM`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data', 'maagement'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from spelling import *\n",
    "\n",
    "q = \"data maagement\"\n",
    "q = clean(q)\n",
    "q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, for each word in the query, if the word is not in the vocabulary, the spell check is applied. This is performed by computing the ratio of shared character-bigrams between the words. If the word is in the vocabulary, it is simply ignored (returned with $100\\%$ confidence). The word is also ignored if it is shorter than the `minimum_word_len` parameter, the default value of which is `4`, or if it is in `nltk.corpus.stopwords.words()`.\n",
    "\n",
    "If the word ends in 's', and the word's singular form is in the vocabulary, then the word's singular form is returned. (_Note: This is a potential argument for having used stemming over lemmatization. Feedback on this is welcome._)\n",
    "\n",
    "Like minimum edit distance, then, it cannot correct words that are already spelled correctly; it is not context-aware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data => data (100%)\n",
      "maagement => management (50.0%)\n"
     ]
    }
   ],
   "source": [
    "for w in q:\n",
    "    top_candidate, conf = spell_check(w, b.build_vocab())[0]\n",
    "    print(f\"{w} => {top_candidate} ({conf*100}%)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well, the threshold for which candidates are returned can be adjusted from its default value of $0.25$. The low threshold reflects the choice of trigrams instead of bigrams as well as the fact that having at least one match is preferable to none at all (as the UI presently only takes the first match, though this will change in the future)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('computational', 0.6666666666666666), ('computation', 0.5), ('computing', 0.45454545454545453), ('computer', 0.36363636363636365)]\n"
     ]
    }
   ],
   "source": [
    "print(spell_check(\"computional\", b.build_vocab(), threshold=0.35))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo\n",
    "\n",
    "The following demonstrations are reflective of the [live website](https://joogle-csi4107.herokuapp.com) as of the date of submission."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BRM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (operating AND system)\n",
    "\n",
    "<img src='report_img/brm-operatingANDsystem.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (comput* AND graph*)\n",
    "\n",
    "<img src='report_img/brm-computANDgraph.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (crypto* OR security)\n",
    "\n",
    "<img src='report_img/brm-cryptoORsecurity.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VSM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### operating system\n",
    "\n",
    "<img src='report_img/vsm-operating_system.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### computers graphical\n",
    "\n",
    "<img src='report_img/vsm-computers_graphical.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### cryptographic security\n",
    "\n",
    "<img src='report_img/vsm-cryptographic_security.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### data maagement\n",
    "\n",
    "The first additional example showcases the implementation of the spell checking module (though it is also visible above). _Data maagement_ is correctly revised and suggested to the user as _data management_.\n",
    "\n",
    "<img src='report_img/additional-data_maagement.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### entity-relationship\n",
    "\n",
    "The second additional example showcases the implementation of phrase indexing. A search for `entity-relationship` would otherwise yield no results as _Entity-Relationship_ would normally match neither (it would normally be stored as two separate words). However, the phrase is correctly indexed and matched in the boolean retrieval.\n",
    "\n",
    "For some reason, the spell check fires here, despite the presence of the word in the vocabulary - this should not be the case and will be fixed in future versions.\n",
    "\n",
    "<img src='report_img/additional-entityANDrelationship.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First and foremost, most of the UI was inspired by/adapted from two sources. The index page was adapted from [nicknish](https://codepen.io/nicknish/pen/cLbAg) on codepen.io, while the search result page was adapted from [rmlynt](https://codepen.io/rmlynt/pen/AXkqJL). Rather than try to hide this by superficially changing things like variable names, I have left them primarily identical. However, all functionality (search, interactivity, infinite scrolling, etc.) is my own.\n",
    "\n",
    "The logo was created [festisite](https://www.festisite.com/logo/google/), and edited using [lunapic](https://www110.lunapic.com/editor/).\n",
    "\n",
    "As well, many code snippets from sources such as stack overflow have been referenced where appropriate throughout my code."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}