outlier_mnli.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding Outliers in Text Data with Transformers, Cleanlab, and Topic Modeling\n",
    "\n",
    "<figure>\n",
    "<img \n",
    "src=\"https://cdn.pixabay.com/photo/2016/02/16/21/07/books-1204029_960_720.jpg\" width=\"900\"\n",
    "alt=\"books\">\n",
    "    <figcaption>Image taken from \n",
    "    <a href=\"https://pixabay.com/photos/books-bookstore-book-reading-1204029/\">Pixabay</a>.\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "\n",
    "Many text corpora contain heterogeneous documents, some of which may be anomalous and worth understanding more. In particular for deployed ML systems, we may want to automatically flag test documents that do not stem from the same distribution as their training data and understand emerging themes within these new documents that were absent from the training data.  This post demonstrates how to find anomalous texts in large NLP corpora using open-source Python libraries like Hugging Face, CleanLab, and PyTorch, as well as how to discover new topics within these texts using c-TF-IDF in order to better understand these anomalies.\n",
    "\n",
    "\n",
    "We will use the [**MultiNLI** dataset on the Hugging Face Hub](https://huggingface.co/datasets/multi_nli), a natural language inference dataset commonly used to train language understanding models.\n",
    "\n",
    "- The dataset contains multiple pairs of sentences (premise, hypothesis) that have been labelled whether the premise entails the hypothesis (`\"entailment\"`) or not (`\"contradiction\"`). A neutral label is also included (`\"neutral\"`).\n",
    "- The corpus is split into a single training set and two validation sets.\n",
    "    - The training set is sourced from 5 different genres: `[fiction, government, slate, telephone, travel]`.\n",
    "    - The *matched validation set* is sourced from genres that *match* those in the the training set\n",
    "    - The other validation set, also referred to as the *mismatched validation set*, is sourced from *other* genres not present in the training data: `[nineeleven, facetoface, letters, oup, verbatim]`. \n",
    "- More information about the corpus can be found [here](https://cims.nyu.edu/~sbowman/multinli/).\n",
    "\n",
    "\n",
    "\n",
    "The steps in this post can be applied with your own word/sentence embedding models and any dataset containing multiple sources of text.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#### Too Long; Didn't Run (the code)\n",
    "Here's our general workflow for detecting outliers from multiple text sources and finding new topics within them:\n",
    "\n",
    "\n",
    "\n",
    "- Load and preprocess text datasets from the Hugging Face Hub to create PyTorch datasets.\n",
    "- Apply pretrained sentence embedding model to create vector embeddings from the text. \n",
    " - Here we utilize a bi-encoder based on a siamese neural network from the [SentenceTransformers](https://huggingface.co/sentence-transformers) library.\n",
    "- Use the [cleanlab](https://github.com/cleanlab/cleanlab) library to find outlier texts in the training data.\n",
    "- Find outlier examples in the validation data that don't come from the data distribution in the training set.\n",
    " - This would be analogous to looking for anomalies in new data sources/feeds.\n",
    "- Select a threshold for deciding whether to consider an example an outlier or not.\n",
    "- Cluster the selected outlier examples to find anomalous genres/sources of text.\n",
    "- Identify topics within the anomalous genres/sources.\n",
    "\n",
    "\n",
    "\n",
    "Our main goal is to find out-of-distribution examples in a dataset, paying more attention to new genres/domains/sources.  In the case for the MultiNLI dataset, only 1 out of the following 4 examples are considered anomalous with these methods. (Can you guess which?)\n",
    "\n",
    "| **Premise**                                                     | **Hypothesis**                                                     | |  **Genre**  |\n",
    "|-----------------------------------------------------------------|--------------------------------------------------------------------|-|:-----------:|\n",
    "| said San'doro.                                                  | San'doro spoke.                                                    | |   fiction   |\n",
    "| Answer? said Julius.                                            | Julius needed an answer right then.                                | |   fiction   |\n",
    "| The space age began with the launch of Sputnik in October 1957. | In October 1957 the space age started after the launch of Sputnik. | | **letters** |\n",
    "| Then he turned to Tommy.                                        | He turned to Tommy next.                                           | |   fiction   |\n",
    "\n",
    "It will turn out that the most likely outliers identified by our method come from the genres in the mismatched validation set, as is to be expected.\n",
    "\n",
    "Many of these outlier examples form clusters based on their respective genres, which can be used to find out-of-distribution topics in the data.\n",
    "\n",
    "<figure>\n",
    "<img \n",
    "src=\"images/outlier_umap.png\" width=\"600\"\n",
    "alt=\"Outlier UMAP\" \\>\n",
    "    <figcaption>Image by author</figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's get coding!\n",
    "\n",
    "The remainder of this article will demonstrate how we implement our strategy, with fully runnable code! Here's a link to a notebook where you can run the same code: [link](https://github.com/elisno/outlier_mnli/blob/main/outlier_mnli.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install dependencies\n",
    "\n",
    "You can install all the required packages by running:\n",
    "\n",
    "```ipython\n",
    "!pip install cleanlab datasets hdbscan matplotlib nltk sklearn torch tqdm transformers umap-learn\n",
    "```\n",
    "\n",
    "Next we'll import the necessary packages, set logging level to 'ERROR' and set some RNG seeds for reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cleanlab\n",
    "import datasets\n",
    "import hdbscan\n",
    "import nltk\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import re\n",
    "import torch\n",
    "\n",
    "from cleanlab.outlier import OutOfDistribution\n",
    "from datasets import load_dataset, concatenate_datasets\n",
    "from IPython.display import display\n",
    "from sklearn.metrics import precision_recall_curve\n",
    "from torch.utils.data import DataLoader\n",
    "from tqdm.auto import tqdm\n",
    "from transformers import AutoTokenizer, AutoModel\n",
    "from umap import UMAP\n",
    "\n",
    "\n",
    "try:\n",
    "    nltk.corpus.stopwords.words\n",
    "except LookupError:\n",
    "    nltk.download('stopwords')\n",
    "\n",
    "datasets.logging.set_verbosity_error()\n",
    "\n",
    "SEED = 42\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False\n",
    "torch.cuda.manual_seed_all(SEED)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess datasets\n",
    "\n",
    "The MultiNLI dataset can be fetched from the Hugging Face Hub via its `datasets` api. The only preprocessing we perform is removing unused columns/features from the datasets. Note that for this post we're *not* looking at the entailment labels (`label`) in the dataset. Rather we are simply trying to automatically identify out of distribution examples based only on their text.\n",
    "\n",
    "\n",
    "For evaluating our outlier detection algorithm, we consider all examples from the mismatched validation set to be out-of-distribution examples. \n",
    "We'll still use the matched validation set to find naturally occurring outlier examples. Our algorithms also do not require the genre information, this is only used for evaluation purposes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_datasets(\n",
    "    *datasets,\n",
    "    sample_sizes = [45000, 9000, 9000],\n",
    "    columns_to_remove = ['premise_binary_parse', 'premise_parse', 'hypothesis_binary_parse', 'hypothesis_parse', 'promptID', 'pairID', 'label'],\n",
    "):\n",
    "    # Remove -1 labels (no gold label)\n",
    "    f = lambda ex: ex[\"label\"] != -1\n",
    "    datasets = [dataset.filter(f) for dataset in datasets]\n",
    "\n",
    "    # Sample a subset of the data\n",
    "    assert len(sample_sizes) == len(datasets), \"Number of datasets and sample sizes must match\"\n",
    "    datasets = [\n",
    "        dataset.shuffle(seed=SEED).select([idx for idx in range(sample_size)])\n",
    "        for dataset, sample_size in zip(datasets, sample_sizes)\n",
    "    ]\n",
    "    \n",
    "    # Remove columns\n",
    "    datasets = [data.remove_columns(columns_to_remove) for data in datasets]\n",
    "\n",
    "    return datasets\n",
    "\n",
    "train_data = load_dataset(\"multi_nli\", split=\"train\")\n",
    "val_matched_data = load_dataset(\"multi_nli\", split=\"validation_matched\")\n",
    "val_mismatched_data = load_dataset(\"multi_nli\", split=\"validation_mismatched\")\n",
    "\n",
    "train_data, val_matched_data, val_mismatched_data = preprocess_datasets(\n",
    "    train_data, val_matched_data, val_mismatched_data\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get some idea of the data format, we'll take a look at a few examples from each dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Training data\")\n",
    "print(f\"Genres: {np.unique(train_data['genre'])}\")\n",
    "display(train_data.to_pandas().head())\n",
    "\n",
    "print(\"Validation matched data\")\n",
    "print(f\"Genres: {np.unique(val_matched_data['genre'])}\")\n",
    "display(val_matched_data.to_pandas().head())\n",
    "\n",
    "print(\"Validation mismatched data\")\n",
    "print(f\"Genres: {np.unique(val_mismatched_data['genre'])}\")\n",
    "display(val_mismatched_data.to_pandas().head())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transform NLI data into vector embeddings\n",
    "\n",
    "We'll use pretrained SentenceTransformer models to embed the sentence pairs in the MultiNLI dataset.\n",
    "\n",
    "One way to train sentence encoders from NLI data (including MultiNLI) is to add a 3-way softmax classifier on top of a Siamese BERT-Network like the one shown below.\n",
    "\n",
    "<figure>\n",
    "<img \n",
    "src=\"https://raw.githubusercontent.com/UKPLab/sentence-transformers/master/docs/img/SBERT_SoftmaxLoss.png\" width=\"400\"\n",
    "alt=\"Softmax loss with siamese network\">\n",
    "    <figcaption>Siamese network with softmax classifier. Image taken from \n",
    "    <a href=\"https://www.sbert.net/examples/training/nli/README.html\">SBERT docs</a>.\n",
    "    </figcaption>\n",
    "</figure>\n",
    "\n",
    "We will use outputs of the (u, v, | u - v |)-layer from such a network as a single vector embedding for each  sentence pair.  This is preferable to concatenating the sentence pairs into single strings as it would increase the risk of truncating the model inputs and losing information (particularly from the hypothesis).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Mean Pooling - Take attention mask into account for correct averaging\n",
    "def mean_pooling(model_output, attention_mask):\n",
    "    token_embeddings = model_output[0] #First element of model_output contains all token embeddings\n",
    "    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()\n",
    "    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)\n",
    "\n",
    "def embed_sentence_pairs(dataloader, tokenizer, model, disable_tqdm=False):\n",
    "    premise_embeddings  = []\n",
    "    hypothesis_embeddings = []\n",
    "    feature_embeddings = []\n",
    "\n",
    "    device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
    "    model.to(device)\n",
    "    model.eval()\n",
    "\n",
    "    loop = tqdm(dataloader, desc=f\"Embedding sentences...\", disable=disable_tqdm)\n",
    "    for data in loop:\n",
    "\n",
    "        premise, hypothesis = data['premise'], data['hypothesis']\n",
    "        encoded_premise, encoded_hypothesis = (\n",
    "            tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')\n",
    "            for sentences in (premise, hypothesis)\n",
    "        )\n",
    "\n",
    "        # Compute token embeddings\n",
    "        with torch.no_grad():\n",
    "            encoded_premise = encoded_premise.to(device)\n",
    "            encoded_hypothesis = encoded_hypothesis.to(device)\n",
    "            model_premise_output = model(**encoded_premise)\n",
    "            model_hypothesis_output = model(**encoded_hypothesis)\n",
    "\n",
    "        # Perform pooling\n",
    "        pooled_premise = mean_pooling(model_premise_output, encoded_premise['attention_mask']).cpu().numpy()\n",
    "        pooled_hypothesis = mean_pooling(model_hypothesis_output, encoded_hypothesis['attention_mask']).cpu().numpy()\n",
    "    \n",
    "        premise_embeddings.extend(pooled_premise)\n",
    "        hypothesis_embeddings.extend(pooled_hypothesis)\n",
    "\n",
    "        \n",
    "    # Concatenate premise and hypothesis embeddings, as well as their absolute difference\n",
    "    feature_embeddings = np.concatenate(\n",
    "        [\n",
    "            np.array(premise_embeddings),\n",
    "            np.array(hypothesis_embeddings),\n",
    "            np.abs(np.array(premise_embeddings) - np.array(hypothesis_embeddings))\n",
    "        ],\n",
    "        axis=1\n",
    "    )\n",
    "    # feature_embeddings = normalize(feature_embeddings, norm='l2', axis=1)\n",
    "    return feature_embeddings\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the next step, you have to choose a pretrained tokenizer+model from the Hugging Face Hub that will provide the token embeddings to the pooling layer of the network.\n",
    "\n",
    "This is done by providing the name of the model on the Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model from Hugging Face Hub\n",
    "\n",
    "# Pretrained SentenceTransformers handle this task better than regular Transformers\n",
    "model_name = 'sentence-transformers/all-MiniLM-L6-v2'\n",
    "\n",
    "# Uncomment the following line to try a regular Transformers model trained on MultiNLI\n",
    "# model_name = 'sileod/roberta-base-mnli'\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = AutoModel.from_pretrained(model_name)\n",
    "\n",
    "batch_size = 128\n",
    "\n",
    "# Place Hugging Face datasets in a PyTorch DataLoader\n",
    "trainloader = DataLoader(train_data, batch_size=batch_size, shuffle=False)\n",
    "valmatchedloader = DataLoader(val_matched_data, batch_size=batch_size, shuffle=False)\n",
    "valmismatchedloader = DataLoader(val_mismatched_data, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "# Get embeddings\n",
    "train_embeddings = embed_sentence_pairs(trainloader, tokenizer, model, disable_tqdm=True)\n",
    "val_matched_embeddings = embed_sentence_pairs(valmatchedloader, tokenizer, model, disable_tqdm=True)\n",
    "val_mismatched_embeddings = embed_sentence_pairs(valmismatchedloader, tokenizer, model, disable_tqdm=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find outliers in the datasets with cleanlab\n",
    "\n",
    "We can find outliers in the training data with cleanlab's `OutOfDistribution` class. This fits a nearest neighbor estimator to the training data (in feature space) and returns an outlier score for each example based on its average distance from its *K* nearest neighbors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get outlier scores for each of the training data feature embeddings\n",
    "ood = OutOfDistribution()\n",
    "train_outlier_scores = ood.fit_score(features=train_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the top outliers in the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# View images with top 15 outlier scores (outliers have low similarity scores)\n",
    "top_train_outlier_idxs = (train_outlier_scores).argsort()[:15]\n",
    "top_train_outlier_subset = train_data.select(top_train_outlier_idxs)\n",
    "top_train_outlier_subset.to_pandas().head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use the fitted nearest neighbor estimator to get outlier scores for the validation data, both the matched and mismatched validation sets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get outlier scores for each of the feature embeddings in the *combined* validation set\n",
    "test_feature_embeddings = np.concatenate([val_matched_embeddings, val_mismatched_embeddings], axis=0)\n",
    "test_outlier_scores = ood.score(features=test_feature_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we look at the top outliers in the validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize 15 most severe outliers in test data\n",
    "test_data = concatenate_datasets([val_matched_data, val_mismatched_data])\n",
    "\n",
    "top_outlier_idxs = (test_outlier_scores).argsort()[:20]\n",
    "top_outlier_subset = test_data.select(top_outlier_idxs)\n",
    "top_outlier_subset.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the combined validation set is balanced with respect to matched and mismatched genres, most of the examples with high outlier scores are from the mismatched validation set (`[nineeleven, facetoface, letters, oup, verbatim]`).\n",
    "\n",
    "\n",
    "Compare this with examples at the other end of the spectrum that are considered unlikely to be outliers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize 15 least severe outliers in test data\n",
    "bottom_outlier_idxs = (-test_outlier_scores).argsort()[:20]\n",
    "bottom_outlier_subset = test_data.select(bottom_outlier_idxs)\n",
    "bottom_outlier_subset.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These examples are only from 4 of the 5 genres in the matched validation set (`[fiction, government, telephone, travel]`). \n",
    "The only exception is the `slate` genre, but the first example appears much further down this list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate outlier scores\n",
    "\n",
    "Realistically, if we already knew that the mismatched dataset contained different genres from those in the training set, we could do the outlier detection on each genre separately.\n",
    "\n",
    "  - I.e. detect outlier sentence pairs from `nineeleven`, then outliers from `facetoface`, etc.\n",
    "\n",
    "To keep things brief for now, let's consider outlier examples from the combined validation set.\n",
    "\n",
    "We can set a threshold to decide what examples in the combined validation set are outliers.\n",
    "We'll be conservative and use the 2.5-th percentile of the outlier score distribution in the training data as the threshold.\n",
    "This threshold is used to select examples from the combined validation set as outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take the 2.5th percentile of the outlier scores in the training data as the threshold\n",
    "threshold = np.percentile(test_outlier_scores, 2.5)\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 5))\n",
    "plt_range = [min(train_outlier_scores.min(),test_outlier_scores.min()), \\\n",
    "             max(train_outlier_scores.max(),test_outlier_scores.max())]\n",
    "\n",
    "axes[0].hist(train_outlier_scores, range=plt_range, bins=50)\n",
    "axes[0].set(title='train_outlier_scores distribution', ylabel='Frequency')\n",
    "axes[0].axvline(x=threshold, color='red', linewidth=2)\n",
    "axes[1].hist(test_outlier_scores, range=plt_range, bins=50)\n",
    "axes[1].set(title='test_outlier_scores distribution', ylabel='Frequency')\n",
    "axes[1].axvline(x=threshold, color='red', linewidth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will result in a few false positives, as can be seen below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get embeddings of test examples whose outlier scores are below the threshold\n",
    "\n",
    "sorted_ids = test_outlier_scores.argsort()\n",
    "outlier_scores = test_outlier_scores[sorted_ids]\n",
    "outlier_ids = sorted_ids[outlier_scores < threshold]\n",
    "\n",
    "selected_outlier_subset = test_data.select(outlier_ids)\n",
    "selected_outlier_subset.to_pandas().tail(15)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cluster outliers\n",
    "\n",
    "Let's assume that we don't know the content of the genres from the mismatched dataset.\n",
    "We can try clustering the outliers from the validation set to see if we can get a better idea about the mismatched genres.\n",
    "\n",
    "With this assumption, it would make sense to use a density based clustering algorithm like HDBSCAN which can handle noise in the selected outlier examples. Unfortunately, it doesn't perform well on high dimensional data. We'll use UMAP to reduce the dimensionality of the data. For visualization purposes, we'll reduce the dimensionality to 2 dimensions, but you may benefit from a slightly higher dimensionality if you expect some overlapping clusters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get embeddings of selected outliers\n",
    "selected_outlier_subset_embeddings = test_feature_embeddings[outlier_ids]\n",
    "\n",
    "# Reduce dimensionality with UMAP\n",
    "umap_fit = UMAP(n_components=2, n_neighbors=8, random_state=SEED)\n",
    "selected_outlier_subset_embeddings_umap = umap_fit.fit_transform(selected_outlier_subset_embeddings)\n",
    "\n",
    "# Set plot labels\n",
    "mismatched_labels = {\"nineeleven\": 0, \"facetoface\": 1, \"letters\": 2, \"oup\": 3, \"verbatim\": 4}\n",
    "matched_labels = {\"fiction\": 5, \"government\": 6, \"slate\": 7, \"telephone\": 8, \"travel\": 9}\n",
    "labels_dict = {**mismatched_labels, **matched_labels}\n",
    "genre_labels = np.array([labels_dict.get(x, 0) for x in selected_outlier_subset[\"genre\"]])\n",
    "\n",
    "# Plot reduced embeddings\n",
    "plt.figure(figsize=(10, 10))\n",
    "x_plot, y_plot = selected_outlier_subset_embeddings_umap[:, 0], selected_outlier_subset_embeddings_umap[:, 1]\n",
    "\n",
    "for i, genre in enumerate(labels_dict.keys()):\n",
    "    x, y = x_plot[genre_labels == i], y_plot[genre_labels == i]\n",
    "    if genre in mismatched_labels:\n",
    "        # Mismatched genres are filled circles\n",
    "        plt.scatter(x, y, label=genre)\n",
    "    else:\n",
    "        # Matched genres are transparent triangles\n",
    "        plt.scatter(x, y, label=genre, alpha=0.5, marker=\"^\")\n",
    "plt.legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At a quick glance, we see that the mismatched genres tend to cluster together. Only `facetoface` overlaps with `verbatim` and the majority of the matched genres. Our best bet would be to look for small local clusters to see how a single genre contains multiple topics. We'll have to set a relatively small minimum cluster size and allow more localized clusters.\n",
    "This is done by lowering the `min_cluster_size` and `min_samples` parameters in the HDBSCAN algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clusterer = hdbscan.HDBSCAN(min_cluster_size=6, min_samples=4)\n",
    "clusterer.fit(selected_outlier_subset_embeddings_umap)\n",
    "cluster_labels = clusterer.labels_\n",
    "\n",
    "clusterer.condensed_tree_.plot(select_clusters=True)\n",
    "\n",
    "# plot each set of points in a different color\n",
    "plt.figure(figsize=(10, 10))\n",
    "for i in np.unique(cluster_labels):\n",
    "    if i != -1:\n",
    "        x, y = x_plot[cluster_labels == i], y_plot[cluster_labels == i]\n",
    "        plt.scatter(x, y, label=f\"cluster {i}\")\n",
    "\n",
    "# Plot outliers in gray\n",
    "x, y = x_plot[cluster_labels == -1], y_plot[cluster_labels == -1]\n",
    "plt.scatter(x, y, label=\"outliers\", color=\"gray\", alpha=0.15)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The clusters on the edges are relatively pure based on visual inspection, i.e. the majority of the points in each cluster are from the same genre.\n",
    "\n",
    "The main exceptions are the:\n",
    "\n",
    "- Violet cluster consisting of 3 genres.\n",
    "- Yellow-green cluster in the center with multiple overlapping genres.\n",
    "  - This suggests that `verbatim` is an \"in-distribution\" topic. This is not useful for testing NLI models.\n",
    "  - This can be removed in some cases.\n",
    "\n",
    "Most of the \"pure\" `verbatim` clusters might be too small to be insightful, but the larger `nineeleven` and `oup` clusters are promising.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Topics with c-TF-IDF\n",
    "\n",
    "A useful way of extracting topics from clusters of dense sentence/document embeddings is with [c-TF-IDF](https://maartengr.github.io/BERTopic/api/ctfidf.html). An existing algorithm, [BERTopic](https://maartengr.github.io/BERTopic/index.html), leverages Transformers (specifically the BERT model) to do exactly this in an easy manner. BERTopic essentially does the following:\n",
    "\n",
    "\n",
    "- Reduce the dimensionality of the embeddings with UMAP\n",
    "- Cluster the reduced embeddings with HDBSCAN.\n",
    "- Compute the TF-IDF scores of the words in each sentence/document cluster with c-TF-IDF.\n",
    "  - TF-IDF scores represent the importance of the words in the cluster.\n",
    "  - Extracting the main topic from a cluster can be done by collecting the words with the highest scores. \n",
    "\n",
    "We've already performed the dimensionality reduction and clustering with the aforementioned methods, so it's redundant to do it again with BERTopic.  A [nice article by James Briggs](https://www.pinecone.io/learn/bertopic/) goes through the same steps in detail and provides a clear implementation of c-TF-IDF for pre-computed embeddings and clusters.  We reuse parts of that implementation below. To keep things simple, we use unigrams to extract topics.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "###### Create documents from sentence pairs\n",
    "\n",
    "# Get combined text from the selected outliers\n",
    "# Joining the premise and hypothesis together\n",
    "def join_sentence_pair(example):\n",
    "    docs = []\n",
    "    for premise, hypothesis in zip(example[\"premise\"], example[\"hypothesis\"]):\n",
    "        docs.append(premise + \" \" + hypothesis)\n",
    "    example[\"docs\"] = docs\n",
    "    return example\n",
    "\n",
    "selected_outlier_subset = selected_outlier_subset.map(join_sentence_pair, batched=True)\n",
    "\n",
    "###### Build vocabularies for classes\n",
    "\n",
    "classes = {}\n",
    "for label in set(clusterer.labels_):\n",
    "    classes[label] = {\n",
    "        'vocab': set(),\n",
    "        'tokens': [],\n",
    "        'tfidf_array': None\n",
    "    }\n",
    "selected_outlier_subset = selected_outlier_subset.add_column('class', clusterer.labels_)\n",
    "\n",
    "\n",
    "# Lowercase and remove punctuation\n",
    "alpha = re.compile(r'[^a-zA-Z ]+')\n",
    "selected_outlier_subset = selected_outlier_subset.map(lambda x: {\n",
    "    'tokens': alpha.sub('', x['docs']).lower()\n",
    "})\n",
    "\n",
    "# Tokenize\n",
    "selected_outlier_subset = selected_outlier_subset.map(lambda x: {\n",
    "    'tokens': nltk.tokenize.wordpunct_tokenize(x['tokens'])\n",
    "})\n",
    "\n",
    "# Collect tokens from all examples for their respective classes\n",
    "for example in selected_outlier_subset:\n",
    "    classes[example['class']]['tokens'].extend(example['tokens'])\n",
    "\n",
    "# Remove stopwords\n",
    "for c in classes.keys():\n",
    "    stopwords = set(nltk.corpus.stopwords.words('english'))\n",
    "    classes[c]['tokens'] = [\n",
    "        word for word in classes[c]['tokens'] if word not in stopwords\n",
    "    ]\n",
    "\n",
    "# Build class vocabulary\n",
    "vocab = set()\n",
    "for c in classes.keys():\n",
    "    vocab = vocab.union(set(classes[c]['tokens']))\n",
    "    classes[c]['vocab'] = set(classes[c]['tokens'])\n",
    "\n",
    "\n",
    "###### c-TF-IDF scores\n",
    "\n",
    "tf = np.zeros((len(classes.keys()), len(vocab)))\n",
    "\n",
    "for c, _class in enumerate(classes.keys()):\n",
    "    for t, term in enumerate(tqdm(vocab, disable=True)):\n",
    "        tf[c, t] = classes[_class]['tokens'].count(term)\n",
    "\n",
    "idf = np.zeros((1, len(vocab)))\n",
    "\n",
    "# Calculate average number of words per class\n",
    "A = tf.sum() / tf.shape[0]\n",
    "\n",
    "for t, term in enumerate(tqdm(vocab, disable=True)):\n",
    "    # Frequency of term t across all classes\n",
    "    f_t = tf[:,t].sum()\n",
    "    # Calculate IDF\n",
    "    idf_score = np.log(1 + (A / f_t))\n",
    "    idf[0, t] = idf_score\n",
    "\n",
    "tf_idf = tf*idf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the words with the top c-TF-IDF scores in each cluster should give us some idea of this cluster's main topic. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 7\n",
    "\n",
    "top_idx = np.argpartition(tf_idf, -n)[:, -n:]\n",
    "vlist = list(vocab)\n",
    "for c, _class in enumerate(classes.keys()):\n",
    "    topn_idx = top_idx[c, :]\n",
    "    topn_terms = [vlist[idx] for idx in topn_idx]\n",
    "    if _class != -1:\n",
    "        print(f\"Topic class {_class}: {topn_terms}\")\n",
    "    else:\n",
    "        print(f\"Outliers: {topn_terms}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the clustered embeddings with their associated topics (left). For comparison, we'll visualize the same embeddings with their original genres as labels (right). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot two figures\n",
    "plt.subplots(nrows=1, ncols=2, figsize=(20, 10))\n",
    "\n",
    "\n",
    "# LEFT PLOT\n",
    "# Plot scatter plot of umap embeddings with clusterer labels as colors\n",
    "\n",
    "x_plot, y_plot = selected_outlier_subset_embeddings_umap[:, 0], selected_outlier_subset_embeddings_umap[:, 1]\n",
    "plt.subplot(1, 2, 1)\n",
    "for i, topic in enumerate(np.unique(cluster_labels)):\n",
    "    if topic != -1:\n",
    "        if i > 10:\n",
    "            marker = \"x\"\n",
    "        else:\n",
    "            marker = \"o\"\n",
    "        x, y = x_plot[cluster_labels == topic], y_plot[cluster_labels == topic]\n",
    "        label = \"_\".join([vlist[idx] for idx in top_idx[topic, :]])\n",
    "        # Truncate label to fit in legend\n",
    "        label = label[:10]\n",
    "        plt.scatter(x, y, label=f\"{topic}: {label}\", marker=marker)\n",
    "\n",
    "# Plot outliers in gray with lower alpha\n",
    "plt.scatter(x_plot[cluster_labels == -1], y_plot[cluster_labels == -1], label=\"outliers\", color=\"gray\", alpha=0.25)\n",
    "plt.title(\"Clustered by HDBSCAN\")\n",
    "plt.xlabel(\"UMAP 1\")\n",
    "plt.ylabel(\"UMAP 2\")\n",
    "plt.legend()\n",
    "\n",
    "\n",
    "# RIGHT PLOT\n",
    "# Plot scatter plot of umap embeddings with genre labels as colors\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "genre_labels = np.array([labels_dict.get(x, 0) for x in selected_outlier_subset[\"genre\"]])\n",
    "for i, genre in enumerate(labels_dict.keys()):\n",
    "    x, y = x_plot[genre_labels == i], y_plot[genre_labels == i]\n",
    "    if genre in mismatched_labels:\n",
    "        plt.scatter(x, y, label=genre)\n",
    "    else:\n",
    "        plt.scatter(x, y, label=genre, alpha=0.5, marker=\"^\")\n",
    "plt.title(\"Labelled by genre\")\n",
    "plt.xlabel(\"UMAP 1\")\n",
    "plt.ylabel(\"UMAP 2\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `nineeleven` genre, several topics stick out, some on US airline flights and others on middle eastern leaders. One topic is discovered in the `oup` genre, which appears to  be about textiles.  The remaining genres overlap too much to obtain meaningful topics. One way to handle the overlapping clusters is to redo the previous clustering exclusively on those points, e.g. by removing `nineeleven` from the analysis and recursively repeat this process as needed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This analysis demonstrated how to identify and understand outliers in text data. The required methods are all available and easy to use in open-source Python libraries, and you should be able to apply the same code demonstrated here to your own text datasets. I hope identifying and understanding outliers helps you ensure better quality data and ML performance in your own applications. You might choose to either omit such examples from your dataset, or to expand your data collection to obtain better coverage of such cases (if they seem relevant).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}