tutorials/LLM_as_a_judge_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Installation and imports"
      ],
      "metadata": {
        "id": "Tdvm881pyC8w"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "thDJOro8nijJ"
      },
      "outputs": [],
      "source": [
        "!pip install evidently[llm]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIWiUlf6xuxD"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dIVmb7pBxyHp"
      },
      "outputs": [],
      "source": [
        "from evidently import ColumnMapping\n",
        "from evidently.report import Report\n",
        "from evidently.metric_preset import TextEvals\n",
        "from evidently.descriptors import *\n",
        "from evidently.metrics import *\n",
        "from evidently.features.llm_judge import BinaryClassificationPromptTemplate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "guwEkvg84o-B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_KEY\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iNFjnBiN382e"
      },
      "source": [
        "# Prepare the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "noZ_CNMd4CCY"
      },
      "source": [
        "Labeled dataset to evaluate the LLM judge on:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1tj6YjKJypVw"
      },
      "outputs": [],
      "source": [
        "data = [\n",
        "    [\"Hi there, how do I reset my password?\",\n",
        "     \"To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.\",\n",
        "     \"To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address. If you don't receive the email, check your spam folder or contact support for assistance.\",\n",
        "     \"incorrect\", \"adds new infromation (contact support)\"],\n",
        "\n",
        "    [\"Where can I find my transaction history?\",\n",
        "     \"You can view your transaction history by logging into your account and navigating to the 'Transaction History' section. Here, you can see all your past transactions. You can also filter the transactions by date or type for easier viewing.\",\n",
        "     \"Log into your account and go to 'Transaction History' to see all your past transactions. In this section, you can view and filter your transactions by date or type. This allows you to find specific transactions quickly and easily.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"How do I add another user to my account?\",\n",
        "     \"I am afraid it is not currently possible to add multiple users to the account. Our system supports only one user per account for security reasons. We recommend creating separate accounts for different users.\",\n",
        "     \"To add a secondary user, go to 'Account Settings', select 'Manage Users', and enter the details of the person you want to add. You can set permissions for their access, deciding what they can and cannot do within the account.\",\n",
        "     \"incorrect\", \"contradiction (incorrect answer)\"],\n",
        "\n",
        "    [\"Is it possible to link multiple bank accounts?\",\n",
        "     \"Yes, you can link multiple bank accounts by going to 'Account Settings' in the menu and selecting 'Add Bank Account'. Follow the prompts to add your bank account details. Make sure to verify each bank account by following the verification process.\",\n",
        "     \"You can add multiple bank accounts by visiting 'Accounts' in the menu and choosing 'Add Bank Account'. Enter your bank details as prompted and complete the verification process for each account to link them successfully.\",\n",
        "     \"incorrect\", \"contradiction (incorrect menu item)\"],\n",
        "\n",
        "    [\"Can I use your service for cryptocurrency transactions?\",\n",
        "     \"Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.\",\n",
        "     \"Currently, our service does not support cryptocurrency transactions. Please check our website for updates regarding this feature. We are continuously working to expand our services and may include this in the future.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"Hi, can I get a detailed report of my monthly transactions?\",\n",
        "     \"Yes, you can generate a detailed monthly report of your transactions by logging into your account, going to 'Transaction History', and selecting 'Generate Report'. You can customize the report by selecting specific dates or transaction types.\",\n",
        "     \"You can get a detailed monthly report by logging into your account, navigating to 'Transaction History', and clicking on 'Generate Report'. Customize your report by choosing the date range and types of transactions you want to include.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"I am traveling to the US. Can I use the app there?\",\n",
        "     \"Yes, you can use the app in the US just like you do at home. Ensure you have an internet connection. You may also want to update your app to the latest version before traveling for optimal performance.\",\n",
        "     \"The app will work in the US without any issues. Just make sure you have access to the internet. For the best experience, update your app to the latest version before you travel.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"How do I link my payment account to a new mobile number?\",\n",
        "     \"To link a new mobile number, log in to your account, go to 'Account Settings', select 'Mobile Number', and follow the instructions to verify your new number. You will need to enter the new number and verify it via a code sent to your phone.\",\n",
        "     \"To add a new number, navigate to the 'Account Settings' section, select 'Mobile Number' and proceed with the steps to add and confirm the new number. Enter the new mobile number and verify it using the code sent to your phone.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"Can I receive notifications for transactions in real-time?\",\n",
        "     \"Yes, you can enable real-time notifications for transactions by going to 'Account Settings', then 'Notifications', and turning on 'Transaction Alerts'. You can choose to receive alerts via SMS, email, or push notifications on your mobile device.\",\n",
        "     \"To receive real-time notifications for transactions, log into your account, go to 'Account Settings', select 'Notifications', and enable 'Transaction Alerts'. Choose your preferred notification method between email or push notifications.\",\n",
        "     \"incorrect\", \"omits information (sms notification)\"],\n",
        "\n",
        "    [\"Hey, can I set up automatic transfers to my savings account?\",\n",
        "     \"Yes, you can set up automatic transfers by going to 'Account Settings', selecting 'Automatic Transfers', and specifying the amount and frequency. You can choose to transfer weekly, bi-weekly, or monthly. Make sure to save the settings to activate the transfers.\",\n",
        "     \"You can arrange automatic transfers by going to 'Account Settings', choosing 'Automatic Transfers', and setting the desired amount and frequency. Don't forget to save the changes to enable the automatic transfers.\",\n",
        "     \"incorrect\", \"omits information (limited frequency of transfers available)\"],\n",
        "\n",
        "    [\"Hi there, how do I reset my password?\",\n",
        "     \"To reset your password, click on 'Forgot Password' on the login page and follow the instructions sent to your registered email.\",\n",
        "     \"To change your password, select 'Forgot Password' on the login screen and follow the steps sent to your registered email address. If you don't receive the email, check your spam folder.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"How can I update my billing address?\",\n",
        "     \"To update your billing address, log into your account, go to 'Account Settings', select 'Billing Information', and enter your new address. Make sure to save the changes once you are done.\",\n",
        "     \"To update your billing address, log into your account, navigate to 'Account Settings', and select 'Billing Information'. Enter your new address and ensure all fields are filled out correctly. Save the changes, and you will receive a confirmation email with the updated address details.\",\n",
        "     \"incorrect\", \"adds new information (confirmation email)\"],\n",
        "\n",
        "    [\"How do I contact customer support?\",\n",
        "     \"You can contact customer support by logging into your account, going to the 'Help' section, and selecting 'Contact Us'. You can choose to reach us via email, phone, or live chat for immediate assistance.\",\n",
        "     \"To contact customer support, log into your account and go to the 'Help' section. Select 'Contact Us' and choose your preferred method: email, phone, or live chat. Our support team is available 24/7 to assist you with any issues. Additionally, you can find a FAQ section that may answer your questions without needing to contact support.\",\n",
        "     \"incorrect\", \"adds new information (24/7 availability, FAQ section)\"],\n",
        "\n",
        "    [\"What should I do if my card is lost or stolen?\",\n",
        "     \"If your card is lost or stolen, immediately log into your account, go to 'Card Management', and select 'Report Lost/Stolen'. Follow the instructions to block your card and request a replacement. You can also contact our support team for assistance.\",\n",
        "     \"If your card is lost or stolen, navigate to 'Card Management' in your account, and select 'Report Lost/Stolen'. Follow the prompts to block your card and request a replacement. Additionally, you can contact our support team for help.\",\n",
        "     \"correct\", \"\"],\n",
        "\n",
        "    [\"How do I enable two-factor authentication (2FA)?\",\n",
        "     \"To enable two-factor authentication, log into your account, go to 'Security Settings', and select 'Enable 2FA'. Follow the instructions to link your account with a 2FA app like Google Authenticator. Once set up, you will need to enter a code from the app each time you log in.\",\n",
        "     \"To enable two-factor authentication, log into your account, navigate to 'Security Settings', and choose 'Enable 2FA'. Follow the on-screen instructions to link your account with a 2FA app such as Google Authenticator. After setup, each login will require a code from the app. Additionally, you can set up backup codes in case you lose access to the 2FA app.\",\n",
        "     \"incorrect\", \"adds new information (backup codes)\"]\n",
        "]\n",
        "\n",
        "columns = [\"question\", \"target_response\", \"new_response\", \"label\", \"comment\"]\n",
        "\n",
        "golden_dataset = pd.DataFrame(data, columns=columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Preview the dataset"
      ],
      "metadata": {
        "id": "fAeh4efkZSxe"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4NnMJHK6y0r2"
      },
      "outputs": [],
      "source": [
        "pd.set_option('display.max_colwidth', None)\n",
        "golden_dataset.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Preview the distribution of classes:"
      ],
      "metadata": {
        "id": "jYTDRwu9I3PG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "report = Report(metrics=[\n",
        "    ColumnDistributionMetric(column_name=\"label\")\n",
        "])\n",
        "\n",
        "report.run(reference_data= None,\n",
        "           current_data= golden_dataset)\n",
        "\n",
        "report"
      ],
      "metadata": {
        "id": "qbZvM0EFgLpD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uifPvCmI34sj"
      },
      "source": [
        "# Design the LLM judge"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Judge to compare correctness of the response to reference."
      ],
      "metadata": {
        "id": "j3zPanZttrRX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "correctness_eval= LLMEval(\n",
        "    subcolumn=\"category\",\n",
        "    additional_columns={\"target_response\": \"target_response\"},\n",
        "    template = BinaryClassificationPromptTemplate(\n",
        "        criteria = \"\"\"\n",
        "An ANSWER is correct when it is the same as the REFERENCE in all facts and details, even if worded differently.\n",
        "The ANSWER is incorrect if it contradicts the REFERENCE, adds additional claims, omits or changes details.\n",
        "\n",
        "REFERENCE:\n",
        "\n",
        "=====\n",
        "{target_response}\n",
        "=====\n",
        "        \"\"\",\n",
        "        target_category=\"incorrect\",\n",
        "        non_target_category=\"correct\",\n",
        "        uncertainty=\"unknown\",\n",
        "        include_reasoning=True,\n",
        "        pre_messages=[(\"system\", \"You are an expert evaluator. will be given an ANSWER and REFERENCE.\")],\n",
        "        ),\n",
        "    provider = \"openai\",\n",
        "    model = \"gpt-4o-mini\",\n",
        "    display_name = \"Correctness\",\n",
        ")"
      ],
      "metadata": {
        "id": "Cq4BWh9PxomF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "correctness_report = Report(metrics=[\n",
        "    TextEvals(column_name=\"new_response\", descriptors=[\n",
        "        correctness_eval\n",
        "    ])\n",
        "])\n",
        "\n",
        "correctness_report.run(reference_data=None,\n",
        "           current_data=golden_dataset)\n",
        "correctness_report"
      ],
      "metadata": {
        "id": "lgKFVsesygYc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "correctness_report.as_dict()"
      ],
      "metadata": {
        "id": "XTkFj_Fy5oHA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "See the raw scores with explanation:"
      ],
      "metadata": {
        "id": "JKPozp0wD4Y1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "correctness_report.datasets().current"
      ],
      "metadata": {
        "id": "bwhXL_LS0F9I"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Evaluate the LLM judge quality"
      ],
      "metadata": {
        "id": "KcbRnaFcdKXa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the dataframe\n",
        "df = pd.DataFrame(correctness_report.datasets().current)\n",
        "\n",
        "# Map data schema\n",
        "column_mapping = ColumnMapping()\n",
        "column_mapping.target = 'label'\n",
        "column_mapping.prediction = 'Correctness category'\n",
        "column_mapping.pos_label = 'incorrect'\n",
        "\n",
        "# Create a classification Report\n",
        "classification_report = Report(metrics=[\n",
        "    ClassificationQualityMetric(),\n",
        "    ClassificationClassBalance(),\n",
        "    ClassificationConfusionMatrix(),\n",
        "])\n",
        "\n",
        "classification_report.run(reference_data=None, current_data=df, column_mapping=column_mapping)\n",
        "\n",
        "classification_report"
      ],
      "metadata": {
        "id": "2KAQiup4dM4V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "classification_report.as_dict()"
      ],
      "metadata": {
        "id": "p-rMasFP5r7r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Verbosity judge"
      ],
      "metadata": {
        "id": "ay7DLXq4rqVL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Single-column judge."
      ],
      "metadata": {
        "id": "Q5axM6uEuD44"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "verbosity_eval = LLMEval(\n",
        "    subcolumn=\"category\",\n",
        "    template = BinaryClassificationPromptTemplate(\n",
        "        criteria = \"\"\"Conciseness refers to the quality of being brief and to the point, while still providing all necessary information.\n",
        "            A concise response should:\n",
        "            - Provide the necessary information without unnecessary details or repetition.\n",
        "            - Be brief yet comprehensive enough to address the query.\n",
        "            - Use simple and direct language to convey the message effectively.\n",
        "        \"\"\",\n",
        "        target_category=\"concise\",\n",
        "        non_target_category=\"verbose\",\n",
        "        uncertainty=\"unknown\",\n",
        "        include_reasoning=True,\n",
        "        pre_messages=[(\"system\", \"You are a judge which evaluates text.\")],\n",
        "        ),\n",
        "    provider = \"openai\",\n",
        "    model = \"gpt-4o-mini\",\n",
        "    display_name = \"verbosity\",\n",
        ")"
      ],
      "metadata": {
        "id": "ihQ3LY0arp5V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "verbosity_report = Report(metrics=[\n",
        "    TextEvals(column_name=\"new_response\", descriptors=[\n",
        "        verbosity_eval\n",
        "    ])\n",
        "])\n",
        "\n",
        "verbosity_report.run(reference_data=None,\n",
        "           current_data=golden_dataset)\n",
        "verbosity_report"
      ],
      "metadata": {
        "id": "ODKS5BZ9sS14"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "verbosity_report.as_dict()"
      ],
      "metadata": {
        "id": "srg5Rox55xht"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "verbosity_report.datasets().current"
      ],
      "metadata": {
        "id": "AehifkvWspCs"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}