Semantic_Search.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IhBUjypAtYbi"
      },
      "source": [
        "# **Cyshield Task #3**: Semantic Search in English <br>by **Mohammed Aly**\n",
        "- [LinkedIn](https://www.linkedin.com/in/mohammed-aly-1854a020a/)  \n",
        "- [GitHub](https://github.com/MohammedAly22/)\n",
        "\n",
        "\n",
        "<html>\n",
        "    <img src=\"https://pbs.twimg.com/media/FqyH_WzaYAQl3w6.png:large\" width=\"100%\">\n",
        "</html>\n",
        "\n",
        "**Semantic search** in Natural Language Processing (NLP) is an advanced approach to **information retrieval** that goes beyond the traditional method of matching keywords. It involves a profound understanding of the **meanings behind words** and the contextual nuances in which they are used.\n",
        "\n",
        "By leveraging techniques from NLP, semantic search aims to comprehend the intricacies of human language. This includes recognizing entities, such as people, places, and organizations, and understanding the relationships between them.\n",
        "\n",
        "The ultimate goal is to provide **more precise and relevant search results** by considering **not just the words** in a query but also the **underlying semantics** and user intent, enhancing the overall search experience."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cya1fJ736943"
      },
      "source": [
        "# 1.0 Importing Required Packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "execution": {
          "iopub.execute_input": "2024-01-17T17:08:00.593301Z",
          "iopub.status.busy": "2024-01-17T17:08:00.592910Z",
          "iopub.status.idle": "2024-01-17T17:08:00.598866Z",
          "shell.execute_reply": "2024-01-17T17:08:00.597927Z",
          "shell.execute_reply.started": "2024-01-17T17:08:00.593270Z"
        },
        "id": "IBsYkrHKtYb_",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "import re\n",
        "from tqdm import tqdm\n",
        "\n",
        "from nltk.corpus import stopwords\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "from transformers import AutoTokenizer, TFAutoModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UNGFu0-g7Cus"
      },
      "source": [
        "# 2.0 Loading & Exploring the Dataset\n",
        "I've chosen the [AG-News-Classification-Dataset](https://www.kaggle.com/datasets/amananandrai/ag-news-classification-dataset) due to its substantial size which is large enough to train a quite robust semantic search algorithm. It consists of the following fields: [`Title`, `Description`, `Class Index`]. The `Class Index` column is an integer ranging from 1 to 4 with these corresponding classes:\n",
        "- 1 -> \"World\"\n",
        "- 2 ->  \"Sports\"\n",
        "- 3 -> \"Business\"\n",
        "- 4 -> \"Science/Technology\"\n",
        "\n",
        "In total, there are **120,000 training samples** and **7600 testing samples** split into two files.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:19.304900Z",
          "iopub.status.busy": "2024-01-17T15:20:19.304020Z",
          "iopub.status.idle": "2024-01-17T15:20:20.292622Z",
          "shell.execute_reply": "2024-01-17T15:20:20.291653Z",
          "shell.execute_reply.started": "2024-01-17T15:20:19.304852Z"
        },
        "id": "0B0lSu1ptYcH",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "train_df = pd.read_csv(\"/kaggle/input/ag-news-classification-dataset/train.csv\")\n",
        "test_df = pd.read_csv(\"/kaggle/input/ag-news-classification-dataset/test.csv\")\n",
        "\n",
        "df = pd.concat([train_df, test_df])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:22.836960Z",
          "iopub.status.busy": "2024-01-17T15:20:22.836600Z",
          "iopub.status.idle": "2024-01-17T15:20:22.858856Z",
          "shell.execute_reply": "2024-01-17T15:20:22.857737Z",
          "shell.execute_reply.started": "2024-01-17T15:20:22.836934Z"
        },
        "id": "t9azKNQ7tYcJ",
        "outputId": "557170ca-c6c5-44e3-8464-228f92e84649",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Class Index</th>\n",
              "      <th>Title</th>\n",
              "      <th>Description</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "      <td>Reuters - Short-sellers, Wall Street's dwindli...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>Reuters - Private investment firm Carlyle Grou...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "      <td>Reuters - Soaring crude prices plus worries\\ab...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>Reuters - Authorities have halted oil export\\f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>AFP - Tearaway world oil prices, toppling reco...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Class Index                                              Title  \\\n",
              "0            3  Wall St. Bears Claw Back Into the Black (Reuters)   \n",
              "1            3  Carlyle Looks Toward Commercial Aerospace (Reu...   \n",
              "2            3    Oil and Economy Cloud Stocks' Outlook (Reuters)   \n",
              "3            3  Iraq Halts Oil Exports from Main Southern Pipe...   \n",
              "4            3  Oil prices soar to all-time record, posing new...   \n",
              "\n",
              "                                         Description  \n",
              "0  Reuters - Short-sellers, Wall Street's dwindli...  \n",
              "1  Reuters - Private investment firm Carlyle Grou...  \n",
              "2  Reuters - Soaring crude prices plus worries\\ab...  \n",
              "3  Reuters - Authorities have halted oil export\\f...  \n",
              "4  AFP - Tearaway world oil prices, toppling reco...  "
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:26.036381Z",
          "iopub.status.busy": "2024-01-17T15:20:26.036032Z",
          "iopub.status.idle": "2024-01-17T15:20:26.094242Z",
          "shell.execute_reply": "2024-01-17T15:20:26.093150Z",
          "shell.execute_reply.started": "2024-01-17T15:20:26.036353Z"
        },
        "id": "If6iomiUtYcR",
        "outputId": "8439cf76-044d-4939-f574-76426dc0e848",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 127600 entries, 0 to 7599\n",
            "Data columns (total 3 columns):\n",
            " #   Column       Non-Null Count   Dtype \n",
            "---  ------       --------------   ----- \n",
            " 0   Class Index  127600 non-null  int64 \n",
            " 1   Title        127600 non-null  object\n",
            " 2   Description  127600 non-null  object\n",
            "dtypes: int64(1), object(2)\n",
            "memory usage: 3.9+ MB\n"
          ]
        }
      ],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7zFNc8G7MU2"
      },
      "source": [
        "# 3.0 Preparing the Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cGcp0yp58r9n"
      },
      "source": [
        "## 3.1 Normalize Column Names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:40.245652Z",
          "iopub.status.busy": "2024-01-17T15:20:40.245260Z",
          "iopub.status.idle": "2024-01-17T15:20:40.257184Z",
          "shell.execute_reply": "2024-01-17T15:20:40.256255Z",
          "shell.execute_reply.started": "2024-01-17T15:20:40.245622Z"
        },
        "id": "vW4dD8VPtYcU",
        "outputId": "fd2b815b-bdbd-48d3-afd5-054c5b2a5bf0",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>class index</th>\n",
              "      <th>title</th>\n",
              "      <th>description</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "      <td>Reuters - Short-sellers, Wall Street's dwindli...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>Reuters - Private investment firm Carlyle Grou...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "      <td>Reuters - Soaring crude prices plus worries\\ab...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>Reuters - Authorities have halted oil export\\f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>AFP - Tearaway world oil prices, toppling reco...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   class index                                              title  \\\n",
              "0            3  Wall St. Bears Claw Back Into the Black (Reuters)   \n",
              "1            3  Carlyle Looks Toward Commercial Aerospace (Reu...   \n",
              "2            3    Oil and Economy Cloud Stocks' Outlook (Reuters)   \n",
              "3            3  Iraq Halts Oil Exports from Main Southern Pipe...   \n",
              "4            3  Oil prices soar to all-time record, posing new...   \n",
              "\n",
              "                                         description  \n",
              "0  Reuters - Short-sellers, Wall Street's dwindli...  \n",
              "1  Reuters - Private investment firm Carlyle Grou...  \n",
              "2  Reuters - Soaring crude prices plus worries\\ab...  \n",
              "3  Reuters - Authorities have halted oil export\\f...  \n",
              "4  AFP - Tearaway world oil prices, toppling reco...  "
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.columns = [col.lower() for col in df.columns]\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EM_RRBxk8uVo"
      },
      "source": [
        "## 3.2 Create `text` Column by Compining `title` and `description`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:50.559299Z",
          "iopub.status.busy": "2024-01-17T15:20:50.558408Z",
          "iopub.status.idle": "2024-01-17T15:20:50.623638Z",
          "shell.execute_reply": "2024-01-17T15:20:50.622611Z",
          "shell.execute_reply.started": "2024-01-17T15:20:50.559263Z"
        },
        "id": "dlNsQVjutYcY",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "df[\"text\"] = df[\"title\"] + \" \" + df[\"description\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:20:53.789064Z",
          "iopub.status.busy": "2024-01-17T15:20:53.788274Z",
          "iopub.status.idle": "2024-01-17T15:20:53.800220Z",
          "shell.execute_reply": "2024-01-17T15:20:53.799255Z",
          "shell.execute_reply.started": "2024-01-17T15:20:53.789028Z"
        },
        "id": "2Oc7Yn10tYca",
        "outputId": "0aec7904-bb8c-4088-98b6-20ea9bcbc7a7",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>class index</th>\n",
              "      <th>title</th>\n",
              "      <th>description</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "      <td>Reuters - Short-sellers, Wall Street's dwindli...</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reute...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>Reuters - Private investment firm Carlyle Grou...</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "      <td>Reuters - Soaring crude prices plus worries\\ab...</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>Reuters - Authorities have halted oil export\\f...</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>AFP - Tearaway world oil prices, toppling reco...</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   class index                                              title  \\\n",
              "0            3  Wall St. Bears Claw Back Into the Black (Reuters)   \n",
              "1            3  Carlyle Looks Toward Commercial Aerospace (Reu...   \n",
              "2            3    Oil and Economy Cloud Stocks' Outlook (Reuters)   \n",
              "3            3  Iraq Halts Oil Exports from Main Southern Pipe...   \n",
              "4            3  Oil prices soar to all-time record, posing new...   \n",
              "\n",
              "                                         description  \\\n",
              "0  Reuters - Short-sellers, Wall Street's dwindli...   \n",
              "1  Reuters - Private investment firm Carlyle Grou...   \n",
              "2  Reuters - Soaring crude prices plus worries\\ab...   \n",
              "3  Reuters - Authorities have halted oil export\\f...   \n",
              "4  AFP - Tearaway world oil prices, toppling reco...   \n",
              "\n",
              "                                                text  \n",
              "0  Wall St. Bears Claw Back Into the Black (Reute...  \n",
              "1  Carlyle Looks Toward Commercial Aerospace (Reu...  \n",
              "2  Oil and Economy Cloud Stocks' Outlook (Reuters...  \n",
              "3  Iraq Halts Oil Exports from Main Southern Pipe...  \n",
              "4  Oil prices soar to all-time record, posing new...  "
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDeD0P3a80F1"
      },
      "source": [
        "## 3.3 Select the Relvenat Features Only"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:21:06.753608Z",
          "iopub.status.busy": "2024-01-17T15:21:06.752772Z",
          "iopub.status.idle": "2024-01-17T15:21:06.768157Z",
          "shell.execute_reply": "2024-01-17T15:21:06.767193Z",
          "shell.execute_reply.started": "2024-01-17T15:21:06.753574Z"
        },
        "id": "YIWG3hrltYcd",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "df = df[[\"text\", \"class index\"]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:21:09.304949Z",
          "iopub.status.busy": "2024-01-17T15:21:09.304319Z",
          "iopub.status.idle": "2024-01-17T15:21:09.314674Z",
          "shell.execute_reply": "2024-01-17T15:21:09.313550Z",
          "shell.execute_reply.started": "2024-01-17T15:21:09.304909Z"
        },
        "id": "tSfmrTqOtYcg",
        "outputId": "72c68841-70ad-43f2-89ec-cee88e437d38",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>class index</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reute...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  class index\n",
              "0  Wall St. Bears Claw Back Into the Black (Reute...            3\n",
              "1  Carlyle Looks Toward Commercial Aerospace (Reu...            3\n",
              "2  Oil and Economy Cloud Stocks' Outlook (Reuters...            3\n",
              "3  Iraq Halts Oil Exports from Main Southern Pipe...            3\n",
              "4  Oil prices soar to all-time record, posing new...            3"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8m69pif98260"
      },
      "source": [
        "## 3.4 Create `category` Column by Mapping the `class index` to a String"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:21:22.453908Z",
          "iopub.status.busy": "2024-01-17T15:21:22.453562Z",
          "iopub.status.idle": "2024-01-17T15:21:22.493911Z",
          "shell.execute_reply": "2024-01-17T15:21:22.492989Z",
          "shell.execute_reply.started": "2024-01-17T15:21:22.453882Z"
        },
        "id": "MKvHZ0CytYcj",
        "outputId": "298396c5-329c-41fd-acf6-5347e128b9b8",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_26/2839029422.py:13: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  df[\"category\"] = df[\"class index\"].apply(convert_id_to_class)\n"
          ]
        }
      ],
      "source": [
        "class_mapper = {\n",
        "    1: \"World\",\n",
        "    2: \"Sports\",\n",
        "    3: \"Business\",\n",
        "    4: \"Science/Technology\"\n",
        "}\n",
        "\n",
        "\n",
        "def convert_id_to_class(row):\n",
        "    \"\"\"\n",
        "    Convert `row` to its corresponding class from `class_mapper`.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - row : pd.Series\n",
        "        A pandas series of class index.\n",
        "    \n",
        "    Returns\n",
        "    -------\n",
        "    - str\n",
        "        The corresponding string value of the class index.\n",
        "    \"\"\"\n",
        "    \n",
        "    return class_mapper[row]\n",
        "\n",
        "\n",
        "df[\"category\"] = df[\"class index\"].apply(convert_id_to_class)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:21:31.556429Z",
          "iopub.status.busy": "2024-01-17T15:21:31.555831Z",
          "iopub.status.idle": "2024-01-17T15:21:31.567301Z",
          "shell.execute_reply": "2024-01-17T15:21:31.566416Z",
          "shell.execute_reply.started": "2024-01-17T15:21:31.556395Z"
        },
        "id": "IY8touF9tYcm",
        "outputId": "8bd1f9a4-fe62-45ee-b7c7-1194d20ced92",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>class index</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reute...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  class index  category\n",
              "0  Wall St. Bears Claw Back Into the Black (Reute...            3  Business\n",
              "1  Carlyle Looks Toward Commercial Aerospace (Reu...            3  Business\n",
              "2  Oil and Economy Cloud Stocks' Outlook (Reuters...            3  Business\n",
              "3  Iraq Halts Oil Exports from Main Southern Pipe...            3  Business\n",
              "4  Oil prices soar to all-time record, posing new...            3  Business"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x3nGJvgU9Hfp"
      },
      "source": [
        "# 4.0 Preprocessing the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:38:03.585994Z",
          "iopub.status.busy": "2024-01-17T15:38:03.585597Z",
          "iopub.status.idle": "2024-01-17T15:38:03.593788Z",
          "shell.execute_reply": "2024-01-17T15:38:03.592163Z",
          "shell.execute_reply.started": "2024-01-17T15:38:03.585963Z"
        },
        "id": "G2h79LgItYcn",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def _clean_text(text):\n",
        "    \"\"\"\n",
        "    Preproces and clean `text` including:\n",
        "    - lowercasing\n",
        "    - keep only alphanumeric characters\n",
        "    - remove stopwords\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - text: str\n",
        "        A string needed to be processed.\n",
        "    \n",
        "    Returns\n",
        "    -------\n",
        "    - str\n",
        "        The cleaned version of the passed `text`.\n",
        "    \"\"\"\n",
        "\n",
        "    text = text.lower()\n",
        "    text = re.sub(\"[^a-zA-Z0-9]\", \" \", text)\n",
        "    text = text.split()\n",
        "    text = [word for word in text if word not in stopwords.words(\"english\")]\n",
        "\n",
        "    return \" \".join(text)\n",
        "\n",
        "\n",
        "def clean_df_text(texts):\n",
        "    \"\"\"\n",
        "    Apply `_clean_text()` method on each text in `texts`.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - texts : pd.Series\n",
        "        A pandas series of texts.\n",
        "    \n",
        "    Returns\n",
        "    -------\n",
        "    - all_cleaned_texts : list[str]\n",
        "        A list contains the cleaned version for each text in `texts`.\n",
        "    \"\"\"\n",
        "\n",
        "    all_cleaned_texts = []\n",
        "\n",
        "    for text in tqdm(texts):\n",
        "        cleaned_text = _clean_text(text)\n",
        "        all_cleaned_texts.append(cleaned_text)\n",
        "\n",
        "    return all_cleaned_texts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:38:05.811535Z",
          "iopub.status.busy": "2024-01-17T15:38:05.810764Z",
          "iopub.status.idle": "2024-01-17T15:48:28.839542Z",
          "shell.execute_reply": "2024-01-17T15:48:28.838612Z",
          "shell.execute_reply.started": "2024-01-17T15:38:05.811500Z"
        },
        "id": "IimIZQGNtYcp",
        "outputId": "4ae448f1-07ec-44c4-c8c9-26f22fc58bd7",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 127600/127600 [10:23<00:00, 204.81it/s]\n"
          ]
        }
      ],
      "source": [
        "cleaned_texts = clean_df_text(df[\"text\"])\n",
        "df[\"clean_text\"] = cleaned_texts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:48:32.562092Z",
          "iopub.status.busy": "2024-01-17T15:48:32.561453Z",
          "iopub.status.idle": "2024-01-17T15:48:32.572508Z",
          "shell.execute_reply": "2024-01-17T15:48:32.571536Z",
          "shell.execute_reply.started": "2024-01-17T15:48:32.562061Z"
        },
        "id": "kpiusLdqtYcs",
        "outputId": "6b2dae01-998f-408c-f0b2-748b2e09de3a",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>class index</th>\n",
              "      <th>category</th>\n",
              "      <th>clean_text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reute...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "      <td>wall st bears claw back black reuters reuters ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "      <td>carlyle looks toward commercial aerospace reut...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "      <td>oil economy cloud stocks outlook reuters reute...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "      <td>iraq halts oil exports main southern pipeline ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>3</td>\n",
              "      <td>Business</td>\n",
              "      <td>oil prices soar time record posing new menace ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  class index  category  \\\n",
              "0  Wall St. Bears Claw Back Into the Black (Reute...            3  Business   \n",
              "1  Carlyle Looks Toward Commercial Aerospace (Reu...            3  Business   \n",
              "2  Oil and Economy Cloud Stocks' Outlook (Reuters...            3  Business   \n",
              "3  Iraq Halts Oil Exports from Main Southern Pipe...            3  Business   \n",
              "4  Oil prices soar to all-time record, posing new...            3  Business   \n",
              "\n",
              "                                          clean_text  \n",
              "0  wall st bears claw back black reuters reuters ...  \n",
              "1  carlyle looks toward commercial aerospace reut...  \n",
              "2  oil economy cloud stocks outlook reuters reute...  \n",
              "3  iraq halts oil exports main southern pipeline ...  \n",
              "4  oil prices soar time record posing new menace ...  "
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXAhX2VZ9KDD"
      },
      "source": [
        "# 5.0 Semantic Search\n",
        "Now that our dataset is all cleaned up and good to go, we can start training semantic search algorithms on it. I've specifically chosen three techniques for this, covering a range from simple and traditional ones like `TF-IDF`, to a slightly more advanced approach using word embeddings with `Doc2Vec`, and finally, a more sophisticated method based on the transformer architecture with attention mechanisms called `MiniLM`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wA3rHYBE9NIw"
      },
      "source": [
        "## 5.1 Semantic Search Using Term-Frequency Inverse-Document-Frequency (TF-IDF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6rk__gC9g1P"
      },
      "source": [
        "### 5.1.1 Training a `TfidfVectorizer`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T15:48:35.201668Z",
          "iopub.status.busy": "2024-01-17T15:48:35.201237Z",
          "iopub.status.idle": "2024-01-17T15:48:39.837679Z",
          "shell.execute_reply": "2024-01-17T15:48:39.836910Z",
          "shell.execute_reply.started": "2024-01-17T15:48:35.201638Z"
        },
        "id": "WAVnXjzwtYcv",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "tf_idf_vectorizer = TfidfVectorizer(stop_words=\"english\")\n",
        "tf_idf_matrix = tf_idf_vectorizer.fit_transform(cleaned_texts)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T16:23:24.984259Z",
          "iopub.status.busy": "2024-01-17T16:23:24.983913Z",
          "iopub.status.idle": "2024-01-17T16:23:24.990785Z",
          "shell.execute_reply": "2024-01-17T16:23:24.989911Z",
          "shell.execute_reply.started": "2024-01-17T16:23:24.984232Z"
        },
        "id": "8iHr2yLrtYcw",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def semantic_search_tf_idf(query, tf_idf_matrix, tf_idf_vectorizer, top_n=5):\n",
        "    \"\"\"\n",
        "    Perform semantic search for the `query` and print the `top_n`\n",
        "    similar results the the given `query` based on TF-IDF technique.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - query : str\n",
        "        A specific query that we need to return similar documents to it.\n",
        "\n",
        "    - tf_idf_matrix : np.sparse_array\n",
        "        A numpy array contains the tf_idf vectors for each sample in the\n",
        "        dataset.\n",
        "    \n",
        "    - tf_idf_vectorizer : keras.TfidfVectorizer\n",
        "        A TfidfVectroizer to be able to vectorize the given `query`.\n",
        "    \n",
        "    - top_n : int, default=5\n",
        "        An integer value indicated the number of returned similar documents.\n",
        "    \"\"\"\n",
        "\n",
        "    cleaned_query = _clean_text(query)\n",
        "    query_vector = tf_idf_vectorizer.transform([cleaned_query])\n",
        "\n",
        "    cosine_similarities = cosine_similarity(query_vector, tf_idf_matrix).flatten()\n",
        "    related_docs_indices = cosine_similarities.argsort()[:-top_n-1:-1]\n",
        "    realted_doc_scores = sorted(cosine_similarities, reverse=True)[:top_n]\n",
        "\n",
        "    # Display the top related documents\n",
        "    print(f\"Top {top_n} Results for Query: '{query}'\")\n",
        "    for i, idx in enumerate(related_docs_indices):\n",
        "        print(f\"{i + 1}. Category: {df.iloc[idx]['category']}\\n\")\n",
        "        print(f\"   Text: {df.iloc[idx]['clean_text']}\\n\")\n",
        "        print(f\"   Similarity: {realted_doc_scores[i]:.4f}\")\n",
        "        print(\"=\"*50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwgpnUhu9bNk"
      },
      "source": [
        "### 5.1.2 Testing on a Random Query from the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:32:00.028723Z",
          "iopub.status.busy": "2024-01-17T18:32:00.028013Z",
          "iopub.status.idle": "2024-01-17T18:32:00.034507Z",
          "shell.execute_reply": "2024-01-17T18:32:00.033606Z",
          "shell.execute_reply.started": "2024-01-17T18:32:00.028695Z"
        },
        "id": "-zlHnC0UtYcx",
        "outputId": "cdae2816-b265-4666-a907-ddbdf470e2c4",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Query of index: 100955: \n",
            "spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night\n",
            "\n",
            "Category of index: 100955: \n",
            "Sports\n"
          ]
        }
      ],
      "source": [
        "random_index = np.random.randint(0, len(df))\n",
        "query = df[\"clean_text\"].iloc[random_index]\n",
        "category = df[\"category\"].iloc[random_index]\n",
        "\n",
        "print(f\"Query of index: {random_index}: \")\n",
        "print(query)\n",
        "print(f\"\\nCategory of index: {random_index}: \")\n",
        "print(category)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:32:11.160837Z",
          "iopub.status.busy": "2024-01-17T18:32:11.160453Z",
          "iopub.status.idle": "2024-01-17T18:32:11.290090Z",
          "shell.execute_reply": "2024-01-17T18:32:11.289180Z",
          "shell.execute_reply.started": "2024-01-17T18:32:11.160810Z"
        },
        "id": "W3ERyoMotYc0",
        "outputId": "e8db4c7d-4822-4911-94ad-5a8986a39d03",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 5 Results for Query: 'spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night\n",
            "\n",
            "   Similarity: 1.0000\n",
            "==================================================\n",
            "2. Category: Sports\n",
            "\n",
            "   Text: spurs beat magic 94 91 ap ap tim duncan 24 points 14 rebounds lead san antonio spurs 94 91 victory wednesday night orlando magic\n",
            "\n",
            "   Similarity: 0.4976\n",
            "==================================================\n",
            "3. Category: Sports\n",
            "\n",
            "   Text: nba game summary san antonio dallas dallas tx sports network tim duncan 20 points 13 rebounds five blocks devin brown scored 14 16 points fourth quarter leading san antonio spurs 107 89 victory dallas mavericks american airlines center\n",
            "\n",
            "   Similarity: 0.4658\n",
            "==================================================\n",
            "4. Category: Sports\n",
            "\n",
            "   Text: streaking spurs roll past sixers 88 80 ap ap tim duncan scored season high 34 points grabbed 13 rebounds lead san antonio spurs fifth straight victory 88 80 philadelphia 76ers thursday night\n",
            "\n",
            "   Similarity: 0.4473\n",
            "==================================================\n",
            "5. Category: Sports\n",
            "\n",
            "   Text: spurs past sixers 88 80 san antonio spurs snatched fifth straight victory away game philadelphia 76ers tim duncan led spurs season high 34 points 13 rebounds china radio international reported friday\n",
            "\n",
            "   Similarity: 0.4399\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using TF-IDF\n",
        "semantic_search_tf_idf(query, tf_idf_matrix, tf_idf_vectorizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NaAQ_mep9mxj"
      },
      "source": [
        "### 5.1.3 Conclusion\n",
        "As observed, despite its **simplicity**, this technique performs quite well and delivers quick and effective results. With minimal effort, we can obtain the top similar results from our dataset for a query like: `spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night`.\n",
        "\n",
        "Additionally, we notice that the category of this query is **sport**, and our `TF-IDF-based semantic search algorithm` aims to retrieve similar **sports-related** results as much as possible."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6dq3aMm9pm1"
      },
      "source": [
        "## 5.2 Semantic Search Using Doc2Vec\n",
        "**Doc2Vec**, an abbreviation for **Document to Vector**, is a notable natural language processing (NLP) technique that extends the principles of **Word2Vec** to entire documents or sentences.\n",
        "\n",
        "In contrast to Word2Vec, which represents words as vectors in a continuous vector space, Doc2Vec focuses on encoding the semantic meaning of **entire documents**. The primary implementation of Doc2Vec is known as **the Paragraph Vector model**, where each document in a corpus is associated with a **unique vector**.\n",
        "\n",
        "This model employs two training approaches:\n",
        "- `PV-DM (Distributed Memory)`, akin to Word2Vec's Continuous Bag of Words (CBOW) model, considers both context words and the paragraph vector for word predictions.\n",
        "- `PV-DBOW (Distributed Bag of Words)` relies solely on the paragraph vector for predicting target words. The resulting vector representations encapsulate the semantic content of documents, facilitating tasks like document similarity, clustering, and classification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPhXKE3q9smL"
      },
      "source": [
        "### 5.2.1 Creating a `Doc2Vec` Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:15:26.276822Z",
          "iopub.status.busy": "2024-01-17T18:15:26.276451Z",
          "iopub.status.idle": "2024-01-17T18:15:31.568629Z",
          "shell.execute_reply": "2024-01-17T18:15:31.567790Z",
          "shell.execute_reply.started": "2024-01-17T18:15:26.276795Z"
        },
        "id": "fg-hMdMmtYc2",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
        "\n",
        "# tokenize that data to be in the form that is expected by `Doc2Vec` model.\n",
        "tokenized_data = [text.split() for text in cleaned_texts]\n",
        "# Tag each document with an index\n",
        "tagged_data = [TaggedDocument(words=words, tags=[str(i)]) for i, words in enumerate(tokenized_data)]\n",
        "# Initialize the Doc2Vec model\n",
        "doc2vec_model = Doc2Vec(vector_size=300, window=5, min_count=1, workers=4, epochs=20)\n",
        "# Build the vocabulary\n",
        "doc2vec_model.build_vocab(tagged_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tr6u62U490vp"
      },
      "source": [
        "### 5.2.2 Training the `Doc2Vec` Model on the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:15:41.883268Z",
          "iopub.status.busy": "2024-01-17T18:15:41.882602Z",
          "iopub.status.idle": "2024-01-17T18:23:12.662017Z",
          "shell.execute_reply": "2024-01-17T18:23:12.661196Z",
          "shell.execute_reply.started": "2024-01-17T18:15:41.883240Z"
        },
        "id": "6uSV3FuctYc4",
        "outputId": "10f53c35-6e40-4239-c81e-142d3cc6d2c0",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 127600/127600 [00:22<00:00, 5591.81it/s]\n"
          ]
        }
      ],
      "source": [
        "# Train the model\n",
        "doc2vec_model.train(tqdm(tagged_data), total_examples=doc2vec_model.corpus_count, epochs=doc2vec_model.epochs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kpIzh9Tc948X"
      },
      "source": [
        "### 5.2.3 Using the Trained `Doc2Vec` Model for Getting the Embeddings of each Sample in the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:25:06.543449Z",
          "iopub.status.busy": "2024-01-17T18:25:06.543038Z",
          "iopub.status.idle": "2024-01-17T18:30:38.060952Z",
          "shell.execute_reply": "2024-01-17T18:30:38.059978Z",
          "shell.execute_reply.started": "2024-01-17T18:25:06.543418Z"
        },
        "id": "Tjm34klvtYc6",
        "outputId": "61d95907-9b5a-418c-f720-89cfe37bf4b0",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 127600/127600 [05:31<00:00, 385.09it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "documents_doc2vec_embeddings shape: (127600, 300)\n"
          ]
        }
      ],
      "source": [
        "# Get the embeddings for each document in dataframe\n",
        "documents_doc2vec_embeddings = np.array([doc2vec_model.infer_vector(words) for words in tqdm(tokenized_data)])\n",
        "print(f\"documents_doc2vec_embeddings shape: {documents_doc2vec_embeddings.shape}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:30:42.865934Z",
          "iopub.status.busy": "2024-01-17T18:30:42.865221Z",
          "iopub.status.idle": "2024-01-17T18:30:42.873609Z",
          "shell.execute_reply": "2024-01-17T18:30:42.872528Z",
          "shell.execute_reply.started": "2024-01-17T18:30:42.865899Z"
        },
        "id": "8mGb9Zg4tYc8",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def semantic_search_doc2vec(query, documents_doc2vec_embeddings, model, top_n=5):\n",
        "    \"\"\"\n",
        "    Perform semantic search for the `query` and print the `top_n`\n",
        "    similar results the the given `query` based on Doc2Vec model.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - query : str\n",
        "        A specific query that we need to return similar documents to it.\n",
        "    \n",
        "    - documents_doc2vec_embeddings : np.array\n",
        "        A numpy array contains the doc2vec embedding vectors for each\n",
        "        sample in the dataset.\n",
        "    \n",
        "    - model : gensim.Doc2Vec\n",
        "        A Doc2Vec model to be able to vectorize the given `query`.\n",
        "    \n",
        "    - top_n : int, default=5\n",
        "        An integer value indicated the number of returned similar documents.\n",
        "    \"\"\"\n",
        "    \n",
        "    # Clean the query\n",
        "    cleaned_query = _clean_text(query)\n",
        "    # Put the query in format suitable for the `model`\n",
        "    query_words = cleaned_query.split()\n",
        "    # Get the embedding of the query\n",
        "    query_embedding = model.infer_vector(query_words)\n",
        "    # Calculate cosine similarities between the query and all vectors in the dataset\n",
        "    cosine_similarities = cosine_similarity([query_embedding], documents_doc2vec_embeddings)[0]\n",
        "    # Get the top_n similar documents to the query\n",
        "    related_docs_indices = np.argsort(cosine_similarities)[-top_n:][::-1]\n",
        "\n",
        "    # Display the top related documents\n",
        "    print(f\"Top {top_n} Results for Query: '{query}'\")\n",
        "    for i, idx in enumerate(related_docs_indices):\n",
        "        print(f\"{i + 1}. Category: {df.iloc[idx]['category']}\\n\")\n",
        "        print(f\"   Text: {df.iloc[idx]['clean_text']}\\n\")\n",
        "        print(f\"   Similarity: {cosine_similarities[idx]:.4f}\")\n",
        "        print(\"=\"*50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0YPhGo7v-BM-"
      },
      "source": [
        "### 5.2.4 Testing on a Random Query from the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:32:24.670140Z",
          "iopub.status.busy": "2024-01-17T18:32:24.669791Z",
          "iopub.status.idle": "2024-01-17T18:32:24.839666Z",
          "shell.execute_reply": "2024-01-17T18:32:24.838454Z",
          "shell.execute_reply.started": "2024-01-17T18:32:24.670105Z"
        },
        "id": "Wxc4kf1mtYc9",
        "outputId": "30c400f2-a2cf-4383-debc-d4c10ee1db8e",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 5 Results for Query: 'spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night\n",
            "\n",
            "   Similarity: 0.9542\n",
            "==================================================\n",
            "2. Category: World\n",
            "\n",
            "   Text: 3 bombings resort towns sinai three explosions shook three egyptian sinai resorts popular vacationing israelis killing least 30 people wounding 100\n",
            "\n",
            "   Similarity: 0.7559\n",
            "==================================================\n",
            "3. Category: Business\n",
            "\n",
            "   Text: sec sues 3 former kmart execs washington federal regulators filed civil fraud charges three former kmart executives five current former managers suppliers\n",
            "\n",
            "   Similarity: 0.7360\n",
            "==================================================\n",
            "4. Category: Science/Technology\n",
            "\n",
            "   Text: siebel moves toward self repairing software com october 11 2004 3 34 pm pt fourth priority 39 main focus enterprise directories organizations spawn projects around identity infrastructure\n",
            "\n",
            "   Similarity: 0.7080\n",
            "==================================================\n",
            "5. Category: Business\n",
            "\n",
            "   Text: cerberus buy lnr property 3 8 bn new york august 30 new ratings lnr property corporation lnr nys reportedly agreed acquired riley property holdings llc 3\n",
            "\n",
            "   Similarity: 0.7076\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using Doc2Vec\n",
        "semantic_search_doc2vec(query, documents_doc2vec_embeddings, doc2vec_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iW3ixk33-G5B"
      },
      "source": [
        "### 5.2.5 Conclusion\n",
        "As observed, the outcomes are somewhat subpar when compared to the performance of the `TF-IDF based semantic search algorithm`. Once more, despite the query falling under the **sports** category, the model yielded results from different categories such as **world** and **business**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dF3X-4Yt-IsX"
      },
      "source": [
        "## 5.3 Semantic Search Using Sentence Transformers\n",
        "**Sentence Transformer** is a state-of-the-art natural language processing (NLP) model designed for **transforming sentences or phrases into meaningful vector representations in a continuous vector space**. Unlike traditional embeddings that capture word meanings, Sentence Transformer focuses on **encoding the semantic content of entire sentences**.\n",
        "\n",
        "The model is based on **transformer architecture**, a powerful neural network architecture that has shown remarkable success in various NLP tasks. Sentence Transformer is trained on large corpora using unsupervised learning, where it learns to generate dense vectors for sentences. One of the key advantages of Sentence Transformer is its ability to produce **contextualized embeddings**, meaning the representation of a sentence can vary based on the context in which it appears."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3IVTs0X-NPu"
      },
      "source": [
        "### 5.3.1 Loading the Tokenizer and the Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "referenced_widgets": [
            "69a79620459c40fd8c84e2cc469e8f85",
            "745301ab3cce4fa1bbba49a11001822f",
            "968ca121a0e74e578d6f7a4d1df21820",
            "b57eb1ec47c64fc99c16b19d5e492b97",
            "cdbf21c4d9174d9fa5efaa95e0f6b238",
            "8af47e1835b8456d90897c841e843f2d"
          ]
        },
        "execution": {
          "iopub.execute_input": "2024-01-17T17:04:35.468179Z",
          "iopub.status.busy": "2024-01-17T17:04:35.467070Z",
          "iopub.status.idle": "2024-01-17T17:04:39.829064Z",
          "shell.execute_reply": "2024-01-17T17:04:39.828237Z",
          "shell.execute_reply.started": "2024-01-17T17:04:35.468138Z"
        },
        "id": "__ZcklUdtYc_",
        "outputId": "b7537320-75ce-4f0e-99fb-2bb31ad91e8c",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "69a79620459c40fd8c84e2cc469e8f85",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "745301ab3cce4fa1bbba49a11001822f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "968ca121a0e74e578d6f7a4d1df21820",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b57eb1ec47c64fc99c16b19d5e492b97",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "cdbf21c4d9174d9fa5efaa95e0f6b238",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8af47e1835b8456d90897c841e843f2d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tf_model.h5:   0%|          | 0.00/91.0M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "All model checkpoint layers were used when initializing TFBertModel.\n",
            "\n",
            "All the layers of TFBertModel were initialized from the model checkpoint at sentence-transformers/all-MiniLM-L6-v2.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.\n"
          ]
        }
      ],
      "source": [
        "model_ckpt = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_ckpt)\n",
        "model = TFAutoModel.from_pretrained(model_ckpt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Az72zeEb-QCQ"
      },
      "source": [
        "### 5.3.2 Getting the `input_ids` and `attention_mask` for each Sample in the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T17:42:06.470644Z",
          "iopub.status.busy": "2024-01-17T17:42:06.470281Z",
          "iopub.status.idle": "2024-01-17T17:42:27.543682Z",
          "shell.execute_reply": "2024-01-17T17:42:27.542819Z",
          "shell.execute_reply.started": "2024-01-17T17:42:06.470616Z"
        },
        "id": "mb5X4rWvtYdA",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "encoded_documents = tokenizer(cleaned_texts, padding=True, truncation=True, return_tensors=\"tf\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hh-pYOzb-X0A"
      },
      "source": [
        "### 5.3.3 Get the Embeddings for each Sample in the Dataset\n",
        "Because of the constraints on the **available GPU memory**, we are unable to input the entire set of `encoded_documents` into the model. Consequently, I divided the original `encoded_documents` into **4400** splits, with each split constituting a batch comprising **29** examples. This division ensures that the batches can be accommodated by the model without encountering any memory allocation errors. It's worth noting that the number **384** corresponds to the `embedding_dim` of the **MiniLM** model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:00:39.522282Z",
          "iopub.status.busy": "2024-01-17T18:00:39.521411Z",
          "iopub.status.idle": "2024-01-17T18:08:25.102800Z",
          "shell.execute_reply": "2024-01-17T18:08:25.101890Z",
          "shell.execute_reply.started": "2024-01-17T18:00:39.522253Z"
        },
        "id": "51bLwSNWtYdB",
        "outputId": "80b6fb69-2555-47d1-e53e-6e1ca92dc022",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "4400it [07:45,  9.45it/s]\n"
          ]
        }
      ],
      "source": [
        "documents_minilm_embeddings = np.zeros((len(df), 384))\n",
        "input_ids_batches = np.array_split(encoded_documents['input_ids'], 4400)\n",
        "attention_mask_batches = np.array_split(encoded_documents['attention_mask'], 4400)\n",
        "batch_size = len(df) // 4400  # 29\n",
        "\n",
        "for i, (input_ids_batch, attention_mask_batch) in tqdm(enumerate(zip(input_ids_batches, attention_mask_batches))):\n",
        "    batch_minilm_embeddings = model(input_ids=input_ids_batch, attention_mask=attention_mask_batch).pooler_output.numpy()\n",
        "    documents_minilm_embeddings[i*batch_size: (i+1)*batch_size] = batch_minilm_embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:08:35.200456Z",
          "iopub.status.busy": "2024-01-17T18:08:35.199620Z",
          "iopub.status.idle": "2024-01-17T18:08:35.206929Z",
          "shell.execute_reply": "2024-01-17T18:08:35.206047Z",
          "shell.execute_reply.started": "2024-01-17T18:08:35.200427Z"
        },
        "id": "b3ZDl4IltYdC",
        "outputId": "8f8861cd-8230-4a78-b73b-01896c3c81cb",
        "trusted": true
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[-0.02879272,  0.06663623,  0.04896924, ..., -0.04945499,\n",
              "        -0.09991661,  0.05025662],\n",
              "       [ 0.03139488,  0.01146773, -0.10593864, ..., -0.02002352,\n",
              "        -0.13299237,  0.014741  ],\n",
              "       [-0.00694024,  0.064034  , -0.02520756, ...,  0.09505007,\n",
              "        -0.05253974, -0.00017   ],\n",
              "       ...,\n",
              "       [-0.01191332, -0.03757676, -0.04574275, ..., -0.05708188,\n",
              "        -0.07425457, -0.02271388],\n",
              "       [-0.03884383,  0.02682269,  0.01142227, ..., -0.04864487,\n",
              "        -0.10057689, -0.1114378 ],\n",
              "       [-0.07792903,  0.06630514,  0.09481338, ...,  0.06931278,\n",
              "        -0.04942315, -0.0144629 ]])"
            ]
          },
          "execution_count": 269,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "documents_minilm_embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:08:58.981775Z",
          "iopub.status.busy": "2024-01-17T18:08:58.981405Z",
          "iopub.status.idle": "2024-01-17T18:08:58.988772Z",
          "shell.execute_reply": "2024-01-17T18:08:58.987884Z",
          "shell.execute_reply.started": "2024-01-17T18:08:58.981746Z"
        },
        "id": "80YL7PQGtYdD",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "def semantic_search_minilm(query, documents_minilm_embeddings, tokenizer, model, top_n=5):\n",
        "    \"\"\"\n",
        "    Perform semantic search for the `query` and print the `top_n`\n",
        "    similar results the the given `query` based on MiniLM model.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    - query : str\n",
        "        A specific query that we need to return similar documents to it.\n",
        "    \n",
        "    - documents_minilm_embeddings : np.array\n",
        "        A numpy array contains the MiniLM embedding vectors for each sample\n",
        "        in the dataset.\n",
        "    \n",
        "    - tokenizer : transformers.AutoTokenizer\n",
        "        A pretrained tokenizer associated with the `model` to tokenize the\n",
        "        `query`.\n",
        "    \n",
        "    - model : transformers.TFAutoModel\n",
        "        A pretrained transformer mode to be able to get the embedding of the\n",
        "        `query`.\n",
        "    \n",
        "    - top_n : int, default=5\n",
        "        An integer value indicated the number of returned similar documents.\n",
        "    \"\"\"\n",
        "    \n",
        "    # Clean the query\n",
        "    cleaned_query = _clean_text(query)\n",
        "    # Tokenize the `cleaned_query` to get `input_ids` and `attention_mask` \n",
        "    encoded_input = tokenizer(cleaned_query, truncation=True, return_tensors=\"tf\")\n",
        "    # Get the query embedding by passing the encoded input to the model\n",
        "    query_embedding = model(**encoded_input).pooler_output.numpy()\n",
        "    # Calculate cosine similarities between the query and all other documents in the dataset\n",
        "    cosine_similarities = cosine_similarity(query_embedding, documents_minilm_embeddings)[0]\n",
        "    # Get the top_n similar documents to the query\n",
        "    related_docs_indices = np.argsort(cosine_similarities)[-top_n:][::-1]\n",
        "\n",
        "    # Display the top related documents\n",
        "    print(f\"Top {top_n} Results for Query: '{query}'\")\n",
        "    for i, idx in enumerate(related_docs_indices):\n",
        "        print(f\"{i + 1}. Category: {df.iloc[idx]['category']}\\n\")\n",
        "        print(f\"   Text: {df.iloc[idx]['clean_text']}\\n\")\n",
        "        print(f\"   Similarity: {cosine_similarities[idx]:.4f}\")\n",
        "        print(\"=\"*50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1bSldc0M-o77"
      },
      "source": [
        "### 5.3.4 Testing on a Random Query from the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:32:35.972636Z",
          "iopub.status.busy": "2024-01-17T18:32:35.972259Z",
          "iopub.status.idle": "2024-01-17T18:32:36.435583Z",
          "shell.execute_reply": "2024-01-17T18:32:36.434175Z",
          "shell.execute_reply.started": "2024-01-17T18:32:35.972605Z"
        },
        "id": "fZrHM6yAtYdE",
        "outputId": "399df766-6afd-49e2-9508-fc5a768c2ccc",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 5 Results for Query: 'spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: spurs defeat mavericks 94 80 tim duncan scored 27 points san antonio spurs held dallas 3 17 shooting fourth quarter 94 80 victory mavericks wednesday night\n",
            "\n",
            "   Similarity: 1.0000\n",
            "==================================================\n",
            "2. Category: Sports\n",
            "\n",
            "   Text: spurs run mavericks 107 89 ap ap devin brown sparked fourth quarter spurt two three point plays two dunks helping san antonio spurs beat dallas mavericks 107 89 monday night spoil pseudo coaching debut avery johnson\n",
            "\n",
            "   Similarity: 0.9195\n",
            "==================================================\n",
            "3. Category: Sports\n",
            "\n",
            "   Text: spurs 107 mavericks 89 devin brown sparked fourth quarter spurt two three point plays two dunks helping san antonio spurs beat dallas mavericks 107 89 monday night spoil pseudo coaching debut avery johnson\n",
            "\n",
            "   Similarity: 0.9107\n",
            "==================================================\n",
            "4. Category: Sports\n",
            "\n",
            "   Text: duncan leads spurs past hornets 83 69 ap ap tim duncan 19 points 12 rebounds lead san antonio spurs third straight victory 83 69 new orleans hornets friday night\n",
            "\n",
            "   Similarity: 0.9098\n",
            "==================================================\n",
            "5. Category: Sports\n",
            "\n",
            "   Text: nba game summary dallas san antonio mavericks 4 3 road season spurs 18 straight regular season home games dating back last year dallas season low eight assists san antonio tx sports network tim\n",
            "\n",
            "   Similarity: 0.9064\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using MiniLM\n",
        "semantic_search_minilm(query, documents_minilm_embeddings, tokenizer, model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MmkUuKQD6gJ"
      },
      "source": [
        "### 5.3.5 Conclusion\n",
        "As evident from the results, the **attention mechanisms** play a crucial role in providing **contextualized embeddings** for each sample in the dataset. This feature enables us to obtain the most accurate matching results for our query, which specifically discusses a **basketball match between the Spurs and Mavericks**. The model successfully retrieves all documents related to the **Spurs and Mavericks**, showcasing a commendable similarity score."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a5duR6p3-tLc"
      },
      "source": [
        "## 5.4 Testing all Techniques on an External Query"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:33:49.996333Z",
          "iopub.status.busy": "2024-01-17T18:33:49.995632Z",
          "iopub.status.idle": "2024-01-17T18:33:50.000395Z",
          "shell.execute_reply": "2024-01-17T18:33:49.999420Z",
          "shell.execute_reply.started": "2024-01-17T18:33:49.996305Z"
        },
        "id": "Lkp_NvnHtYdF",
        "trusted": true
      },
      "outputs": [],
      "source": [
        "query = \"Real Madrid beat Barcelona 4-1 in the Spanish Super Cup in Saudi Arabia, as El Clasico delivered drama and brilliant goals once again.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:33:58.390973Z",
          "iopub.status.busy": "2024-01-17T18:33:58.390238Z",
          "iopub.status.idle": "2024-01-17T18:33:58.507521Z",
          "shell.execute_reply": "2024-01-17T18:33:58.506439Z",
          "shell.execute_reply.started": "2024-01-17T18:33:58.390942Z"
        },
        "id": "eGDqXMBetYdG",
        "outputId": "b43894c6-aa2e-4992-d1e1-0dd3ff418743",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TF-IDF Model\n",
            "\n",
            "Top 5 Results for Query: 'Real Madrid beat Barcelona 4-1 in the Spanish Super Cup in Saudi Arabia, as El Clasico delivered drama and brilliant goals once again.'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: barcelona beat real madrid barcelona moved seven points clear top spanish league saturday following three nil victory home second placed real madrid\n",
            "\n",
            "   Similarity: 0.3614\n",
            "==================================================\n",
            "2. Category: Sports\n",
            "\n",
            "   Text: barcelona shuts rival real madrid madrid spain barcelona moved ahead spanish league beating rival real madrid 3 0 saturday country 39 biggest match\n",
            "\n",
            "   Similarity: 0.3411\n",
            "==================================================\n",
            "3. Category: Sports\n",
            "\n",
            "   Text: barcelona real madrid post home wins barcelona spain sports network david beckham scored game winner real madrid 39 galacticos 39 barcelona week two spanish premier division\n",
            "\n",
            "   Similarity: 0.3307\n",
            "==================================================\n",
            "4. Category: Sports\n",
            "\n",
            "   Text: barcelona beats real madrid spanish league barcelona moved ahead spanish league beating rival real madrid 3 0 saturday country 39 biggest match samuel eto 39 giovanni van bronckhorst scored first half ronaldinho\n",
            "\n",
            "   Similarity: 0.2882\n",
            "==================================================\n",
            "5. Category: Sports\n",
            "\n",
            "   Text: real madrid stays touch leader barcelona spanish lt b gt lt b gt four goals 11 minutes allowed real madrid destroy bernd schuster 39 levante win frustrated fans secured second place spanish league standings\n",
            "\n",
            "   Similarity: 0.2796\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using TF-IDF\n",
        "print(\"TF-IDF Model\\n\")\n",
        "semantic_search_tf_idf(query, tf_idf_matrix, tf_idf_vectorizer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:34:02.213769Z",
          "iopub.status.busy": "2024-01-17T18:34:02.213398Z",
          "iopub.status.idle": "2024-01-17T18:34:02.374213Z",
          "shell.execute_reply": "2024-01-17T18:34:02.372961Z",
          "shell.execute_reply.started": "2024-01-17T18:34:02.213743Z"
        },
        "id": "170CpuvGtYdI",
        "outputId": "c5e026fe-bd1c-4c0d-f4c7-742f4daff1e3",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "DOC2VEC Model\n",
            "\n",
            "Top 5 Results for Query: 'Real Madrid beat Barcelona 4-1 in the Spanish Super Cup in Saudi Arabia, as El Clasico delivered drama and brilliant goals once again.'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: uefa cup champ takes super cup 2 1 victory porto uefa cup holders valencia beat european champion porto 2 1 win super cup monaco 39 stade louis ii friday midfielder vicente laid valencia goals ruben baraja heading\n",
            "\n",
            "   Similarity: 0.4815\n",
            "==================================================\n",
            "2. Category: Sports\n",
            "\n",
            "   Text: fa investigate chelsea west ham violence league cup match football association investigate crowd violence marred chelsea 39 1 0 win west ham league cup mateja kezman scored goal wednesday night stamford\n",
            "\n",
            "   Similarity: 0.3916\n",
            "==================================================\n",
            "3. Category: Sports\n",
            "\n",
            "   Text: marseille 39 european cup winning 39 sorcerer 39 dies belgian 1993 european cup french side marseille time side france captured european club football 39 premier trophy 1978 cup winners cup belgian giants anderlecht\n",
            "\n",
            "   Similarity: 0.3915\n",
            "==================================================\n",
            "4. Category: Sports\n",
            "\n",
            "   Text: update 1 inter make hard work cup win bologna inter milan came behind beat bologna 3 1 italian cup third round first leg match san siro stadium sunday\n",
            "\n",
            "   Similarity: 0.3821\n",
            "==================================================\n",
            "5. Category: World\n",
            "\n",
            "   Text: golf woods reveals cup ambition tiger woods wants playing vice captain next us ryder cup team\n",
            "\n",
            "   Similarity: 0.3502\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using Doc2Vec\n",
        "print(\"DOC2VEC Model\\n\")\n",
        "semantic_search_doc2vec(query, documents_doc2vec_embeddings, doc2vec_model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-01-17T18:34:05.628880Z",
          "iopub.status.busy": "2024-01-17T18:34:05.627744Z",
          "iopub.status.idle": "2024-01-17T18:34:06.078356Z",
          "shell.execute_reply": "2024-01-17T18:34:06.076978Z",
          "shell.execute_reply.started": "2024-01-17T18:34:05.628836Z"
        },
        "id": "khNNBjBItYdO",
        "outputId": "103bd07e-4f8f-4d34-f4de-982fa1b4ff39",
        "trusted": true
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "MINILM Model\n",
            "\n",
            "Top 5 Results for Query: 'Real Madrid beat Barcelona 4-1 in the Spanish Super Cup in Saudi Arabia, as El Clasico delivered drama and brilliant goals once again.'\n",
            "1. Category: Sports\n",
            "\n",
            "   Text: spain real madrid crush levante ronaldo scored twice real madrid ended two game winless slide 5 0 spanish league victory seventh placed levante santiago bernabeu sunday\n",
            "\n",
            "   Similarity: 0.8590\n",
            "==================================================\n",
            "2. Category: Sports\n",
            "\n",
            "   Text: barcelona beats real madrid spanish league barcelona moved ahead spanish league beating rival real madrid 3 0 saturday country 39 biggest match samuel eto 39 giovanni van bronckhorst scored first half ronaldinho\n",
            "\n",
            "   Similarity: 0.8589\n",
            "==================================================\n",
            "3. Category: Sports\n",
            "\n",
            "   Text: liga sunday wrap madrid answer critics real madrid ended talk crisis club thumped levante 5 0 bernabeu valencia moved back champions league places 2 0 win mallorca\n",
            "\n",
            "   Similarity: 0.8574\n",
            "==================================================\n",
            "4. Category: Sports\n",
            "\n",
            "   Text: barcelona 3 0 real madrid cameroon 39 samuel eto 39 fils helped barcelona trounce real madrid 3 0 move seven points clear great rivals spain 39 la liga\n",
            "\n",
            "   Similarity: 0.8547\n",
            "==================================================\n",
            "5. Category: Sports\n",
            "\n",
            "   Text: real madrid ponders biggest champions league loss four years real madrid began yesterday 39 match bayer leverkusen bookmakers 39 favorite win champions league record nine time european champion finished worst defeat competition four years\n",
            "\n",
            "   Similarity: 0.8535\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "# Perform semantic search using MiniLM\n",
        "print(\"MINILM Model\\n\")\n",
        "semantic_search_minilm(query, documents_minilm_embeddings, tokenizer, model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NmA6zytd-z23"
      },
      "source": [
        "# 6.0 Final Conclusion\n",
        "In conclusion, based on the outcomes discussed above, it is evident that fundamental techniques like `TF-IDF` continue to perform remarkably well even without the use of neural networks. The results obtained with `Doc2Vec` demonstrate decent performance relying on fixed embeddings. However, the most effective technique appears to be the `MiniLM transformer-based` model, primarily owing to its utilization of **attention mechanisms** that can harness **contextualized embeddings**."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kaggle": {
      "accelerator": "nvidiaTeslaT4",
      "dataSources": [
        {
          "datasetId": 612351,
          "sourceId": 1095715,
          "sourceType": "datasetVersion"
        },
        {
          "datasetId": 4314850,
          "sourceId": 7417144,
          "sourceType": "datasetVersion"
        }
      ],
      "dockerImageVersionId": 30636,
      "isGpuEnabled": true,
      "isInternetEnabled": true,
      "language": "python",
      "sourceType": "notebook"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}