tutorials/How_to_add_a_custom_text_descriptor.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mAGKZvMaBmWe",
        "outputId": "96aacb80-bb4c-4d0e-fce1-2862c223ce15"
      },
      "source": [
        "# How to evaluate text data using Descriptors"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This tutorial explains:\n",
        "* how to evaluate text data using Descriptors\n",
        "* how to use external models to generate additional features for the text data"
      ],
      "metadata": {
        "id": "_RGsbMtFCpSQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Installation"
      ],
      "metadata": {
        "id": "dMxxAqXQ8epu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install Evidently following the instructions for your environment: https://docs.evidentlyai.com/user-guide/install-evidently"
      ],
      "metadata": {
        "id": "bB-tYHRK8ASY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    import evidently\n",
        "except:\n",
        "    !pip install evidently==0.3.3"
      ],
      "metadata": {
        "id": "7su7ZfN88CfX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install transformers to be able to use the external model. Instructions: https://huggingface.co/docs/transformers/installation"
      ],
      "metadata": {
        "id": "UMnoRp973STi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "id": "W6NmVZ21dw5U"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tmBHvzszAEN-"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "from sklearn import datasets, ensemble, model_selection"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the required Evidently components that you will use in the tutorial."
      ],
      "metadata": {
        "id": "NeZBb7Wz8qb5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from evidently import ColumnMapping\n",
        "from evidently.report import Report\n",
        "from evidently.test_suite import TestSuite\n",
        "\n",
        "from evidently.metrics import DataDriftTable, TextDescriptorsDriftMetric, ColumnDriftMetric\n",
        "from evidently.metric_preset import TextOverviewPreset\n",
        "from evidently.descriptors import TextLength, TriggerWordsPresence, OOV, NonLetterCharacterPercentage, SentenceCount, WordCount, Sentiment\n",
        "from evidently.tests import *"
      ],
      "metadata": {
        "id": "L6wJ_jARrRtT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the components from NLTK required to compute some of the metrics."
      ],
      "metadata": {
        "id": "uyuFS3Ys9SCS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('words')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')\n",
        "nltk.download('vader_lexicon')"
      ],
      "metadata": {
        "id": "aAvcs9tYsRdo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Import the components from Transformers to be able to use the external model."
      ],
      "metadata": {
        "id": "60Z9Tmk23eeI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import pipeline"
      ],
      "metadata": {
        "id": "C2VQnDAA3sm-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Prepare the data"
      ],
      "metadata": {
        "id": "PG0lch6x8Ua3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load the e-commerce review demo dataset."
      ],
      "metadata": {
        "id": "jR3FNxb1q5tR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_data = datasets.fetch_openml(name='Womens-E-Commerce-Clothing-Reviews', version=2, as_frame='auto')\n",
        "reviews = reviews_data.frame"
      ],
      "metadata": {
        "id": "SIGfndefq7r5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Split into two datasets: reference and current. Let's imagine that \"reference\" data is the data for some representative past period (e.g., last month) and \"current\" is the current production data (e.g., this month)."
      ],
      "metadata": {
        "id": "g0lER7rr8i9q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_ref = reviews[reviews.Rating > 3].sample(n=5000, replace=True, ignore_index=True, random_state=42)\n",
        "reviews_cur = reviews[reviews.Rating < 3].sample(n=5000, replace=True, ignore_index=True, random_state=42)"
      ],
      "metadata": {
        "id": "vAQV2Nxyq75H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Add column mapping to help Evidently parse the input data correctly and specify the columns with text."
      ],
      "metadata": {
        "id": "Sv_Mal1u9GmW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "column_mapping = ColumnMapping(\n",
        "    numerical_features=['Age', 'Positive_Feedback_Count'],\n",
        "    categorical_features=['Division_Name', 'Department_Name', 'Class_Name'],\n",
        "    text_features=['Review_Text', 'Title']\n",
        ")"
      ],
      "metadata": {
        "id": "y2KVgRqdrb8X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's look at the data stucture!"
      ],
      "metadata": {
        "id": "RR2xB5QCCUa9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_ref.head()"
      ],
      "metadata": {
        "id": "35t90hHnCctz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Get the text overview report"
      ],
      "metadata": {
        "id": "sdZXapkA9ZXN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Evidently generates a lot of metrics out of the box. For example, you can generate a comparative Report to visualize the characteristics of the review texts in the two datasets. There is a pre-built **Text Overview Preset** that combines different descriptive checks and evaluates data drift.\n",
        "\n",
        "We also use **Text Descriptors** - standard auto-generated features that describe the text dataset (e.g. text length, % of words out of vocabulary, etc.)\n",
        "We defined two additional Descriptors using Trigger Words. Each checks whether any word from the list appears in the dataset.\n",
        "\n",
        "Check out the Evidently docs on Descriptors for details: https://docs.evidentlyai.com/user-guide/customization/text-descriptors-parameters"
      ],
      "metadata": {
        "id": "yshlGpI1rMdZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_overview_report = Report(metrics=[\n",
        "    TextOverviewPreset(column_name=\"Review_Text\", descriptors={\n",
        "        \"Review texts - OOV %\" : OOV(),\n",
        "        \"Review texts - Non Letter %\" : NonLetterCharacterPercentage(),\n",
        "        \"Review texts - Symbol Length\" : TextLength(),\n",
        "        \"Review texts - Sentence Count\" : SentenceCount(),\n",
        "        \"Review texts - Word Count\" : WordCount(),\n",
        "        \"Review texts - Sentiment\" : Sentiment(),\n",
        "        \"Reviews about Dress\" : TriggerWordsPresence(words_list=['dress', 'gown']),\n",
        "        \"Reviews about Blouses\" : TriggerWordsPresence(words_list=['blouse', 'shirt']),\n",
        "    })\n",
        "])\n",
        "\n",
        "text_overview_report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "text_overview_report"
      ],
      "metadata": {
        "id": "oPmi3a18rfpj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Customize the report - data drift"
      ],
      "metadata": {
        "id": "YMKLTnSm92nC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's say you are only interested in tracking a few relevant text properties and detecting when there is a change.\n",
        "* Whether the review is about Dresses\n",
        "* The word length of the review\n",
        "* The review sentiment\n",
        "\n",
        "Let's create a simple custom report to track drift in these properties."
      ],
      "metadata": {
        "id": "oC_Lc6c4qa4f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "descriptors_report = Report(metrics=[\n",
        "    ColumnDriftMetric(WordCount().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(Sentiment().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(TriggerWordsPresence(words_list=['dress', 'gown']).for_column(\"Review_Text\")),\n",
        "])\n",
        "\n",
        "descriptors_report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "descriptors_report"
      ],
      "metadata": {
        "id": "1FzzhfRtSnOK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also include raw data in the reports: the visuals might be more informative (but will take longer to compute and load: use with caution)."
      ],
      "metadata": {
        "id": "SnHgobDIzgtZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "report = Report(\n",
        "    metrics=[\n",
        "      ColumnDriftMetric(Sentiment().for_column(\"Review_Text\")),\n",
        "    ],\n",
        "    options={\"render\": {\"raw_data\": True}}\n",
        "  )\n",
        "report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "report"
      ],
      "metadata": {
        "id": "5q_R2I5_0WDO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# How to add a new text property?"
      ],
      "metadata": {
        "id": "BWlE6nkI9yhf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To add a new text property, you can use an external open-source model to score your dataset. Then you will work with this property as an additional column.\n",
        "\n",
        "As an example, we will take Distilibert model that classifies the text by 5 emotions. Source: https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion\n",
        "\n",
        "You can consider any other model, for example, for named entity recognition, language detection, toxicity detection, etc."
      ],
      "metadata": {
        "id": "ec--rjq-sflk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "classifier = pipeline(\"text-classification\", model='bhadresh-savani/distilbert-base-uncased-emotion', top_k=1)\n",
        "prediction = classifier(\"I love using evidently! It's easy to use\", )\n",
        "print(prediction)"
      ],
      "metadata": {
        "id": "7q-m20qZpNWQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Score the reviews by emotion"
      ],
      "metadata": {
        "id": "Udv4Voa0csWy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Note**: this step will score the dataset using external model. It will take some time to execute. If you want to understand the principle without waiting, scroll down to the \"Simple example\" section below."
      ],
      "metadata": {
        "id": "KL9-xPH9PeWt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_ref['emotion'] = [x[0]['label'] for x in classifier(list(reviews_ref.Review_Text.fillna('')))]\n",
        "reviews_cur['emotion'] = [x[0]['label'] for x in classifier(list(reviews_cur.Review_Text.fillna('')))]"
      ],
      "metadata": {
        "id": "uf5k9vyxsne8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Update column mapping"
      ],
      "metadata": {
        "id": "XGI3bBdLPaOY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's take a look at the new dataset that now contains \"emotion\" column."
      ],
      "metadata": {
        "id": "G_6exK4sP3Zy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "reviews_cur.head()"
      ],
      "metadata": {
        "id": "n46NcWX0P8uQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You should reflect this in Column Mapping."
      ],
      "metadata": {
        "id": "975z0njFQGI0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "column_mapping = ColumnMapping(\n",
        "    numerical_features=['Age', 'Positive_Feedback_Count'],\n",
        "    categorical_features=['Division_Name', 'Department_Name', 'Class_Name', 'emotion'],\n",
        "    text_features=['Review_Text', 'Title']\n",
        ")"
      ],
      "metadata": {
        "id": "45qAfayRtnLJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Add \"emotion drift\" checks"
      ],
      "metadata": {
        "id": "wgk7RVpAQJ_6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can now add the drift check for the \"emotion\" column to the Report."
      ],
      "metadata": {
        "id": "_ic-c0AkQxAF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "descriptors_report = Report(metrics=[\n",
        "    ColumnDriftMetric(WordCount().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(Sentiment().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(TriggerWordsPresence(words_list=['dress', 'gown']).for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric('emotion'),\n",
        "])\n",
        "\n",
        "descriptors_report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "descriptors_report"
      ],
      "metadata": {
        "id": "CyxcdELpQopy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run pipeline tests"
      ],
      "metadata": {
        "id": "-depRab9Q4gd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To execute regular checks, you can use Evidently Test Suites."
      ],
      "metadata": {
        "id": "zzwJpER3uxi_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "descriptors_test_suite = TestSuite(tests=[\n",
        "    TestColumnDrift(column_name = 'emotion'),\n",
        "    TestColumnDrift(column_name = WordCount().for_column(\"Review_Text\")),\n",
        "    TestColumnDrift(column_name = Sentiment().for_column(\"Review_Text\")),\n",
        "    TestColumnDrift(column_name = TriggerWordsPresence(words_list=['dress', 'gown']).for_column(\"Review_Text\")),\n",
        "])\n",
        "\n",
        "descriptors_test_suite.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "descriptors_test_suite"
      ],
      "metadata": {
        "id": "32-aYlGlvb0_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Beyond drift"
      ],
      "metadata": {
        "id": "xddmnKlJeupc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Detecting statistical distribution drift is one of the ways to monitor changes in the property. However, sometimes it is convenient to use other checks: for example, rule-based expectations on min-max values."
      ],
      "metadata": {
        "id": "_tLbLYx-eyTx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's say that we want to check that:\n",
        "* The reviews are longer than 2 words. We want the test to fail if at least 1 review is < 2 words, and see the number of short texts."
      ],
      "metadata": {
        "id": "08hSPrame_cX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "descriptors_test_suite = TestSuite(tests=[\n",
        "    TestNumberOfOutRangeValues(column_name = WordCount().for_column(\"Review_Text\"), left=2, eq=0),\n",
        "])\n",
        "\n",
        "descriptors_test_suite.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "descriptors_test_suite"
      ],
      "metadata": {
        "id": "blSroWCuet8I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Simple example"
      ],
      "metadata": {
        "id": "362F4JSmRbpX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To avoid waiting until the model scores the dataset, let's assume that the existing column \"Class_Name\" is the new descriptor."
      ],
      "metadata": {
        "id": "22EwTh-JRdlT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "simple_descriptors_report = Report(metrics=[\n",
        "    ColumnDriftMetric(WordCount().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(Sentiment().for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric(TriggerWordsPresence(words_list=['dress', 'gown']).for_column(\"Review_Text\")),\n",
        "    ColumnDriftMetric('Class_Name'),\n",
        "])\n",
        "\n",
        "simple_descriptors_report.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "simple_descriptors_report"
      ],
      "metadata": {
        "id": "MqDFswzaSGwM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "descriptors_test_suite = TestSuite(tests=[\n",
        "    TestColumnDrift(column_name = 'Class_Name'),\n",
        "    TestColumnDrift(column_name = WordCount().for_column(\"Review_Text\")),\n",
        "    TestColumnDrift(column_name = Sentiment().for_column(\"Review_Text\")),\n",
        "    TestColumnDrift(column_name = TriggerWordsPresence(words_list=['dress', 'gown']).for_column(\"Review_Text\")),\n",
        "])\n",
        "\n",
        "descriptors_test_suite.run(reference_data=reviews_ref, current_data=reviews_cur, column_mapping=column_mapping)\n",
        "descriptors_test_suite"
      ],
      "metadata": {
        "id": "0HP13HEnSRX-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Support Evidently\n",
        "\n",
        "Enjoyed the tutorial? Star Evidently on GitHub to contribute back! This helps us continue creating free open-source tools for the community. https://github.com/evidentlyai/evidently\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "-sGufnSZBX-l"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}