try_templated_evaluator.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the environment variables\n",
    "import dotenv\n",
    "dotenv.load_dotenv('.env')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Please act as an impartial judge and evaluate the quality of the provided answer which attempts to answer the provided question based on a provided context.\n",
      "\n",
      "You'll be given a function grading_function which you'll call for each provided context, question and answer to submit your reasoning and score for the correctness, comprehensiveness and readability of the answer.  \n",
      "Please make sure you always call the function to submit result\n",
      "\n",
      "\n",
      "  Below is your grading rubric: \n",
      "\n",
      "- Correctness: If the answer correctly answer the question, below are the details for different scores:\n",
      "  - Score 0: the answer is completely incorrect, doesn’t mention anything about the question or is completely contrary to the correct answer.\n",
      "      - Example: when asked “How to terminate a databricks cluster”, the answer is empty string, or content that’s completely irrelevant, or sorry I don’t know the answer.\n",
      "  - Score 0: the answer provides some relevance to the question and answer one aspect of the question correctly.\n",
      "      - Example:\n",
      "          - Question: How to terminate a databricks cluster\n",
      "          - Answer: Databricks cluster is a cloud-based computing environment that allows users to process big data and run distributed data processing tasks efficiently.\n",
      "          - Or answer:  In the Databricks workspace, navigate to the \"Clusters\" tab. And then this is a hard question that I need to think more about it\n",
      "  - Score 1: the answer mostly answer the question but is missing or hallucinating on one critical aspect.\n",
      "      - Example:\n",
      "          - Question: How to terminate a databricks cluster”\n",
      "          - Answer: “In the Databricks workspace, navigate to the \"Clusters\" tab.\n",
      "          Find the cluster you want to terminate from the list of active clusters.\n",
      "          And then you’ll find a button to terminate all clusters at once”\n",
      "  - Score 1: the answer correctly answer the question and not missing any major aspect\n",
      "      - Example:\n",
      "          - Question: How to terminate a databricks cluster\n",
      "          - Answer: In the Databricks workspace, navigate to the \"Clusters\" tab.\n",
      "          Find the cluster you want to terminate from the list of active clusters.\n",
      "          Click on the down-arrow next to the cluster name to open the cluster details.\n",
      "          Click on the \"Terminate\" button. A confirmation dialog will appear. Click \"Terminate\" again to confirm the action.”\n",
      "- Comprehensiveness: How comprehensive is the answer, does it fully answer all aspects of the question and provide comprehensive explanation and other necessary information. Below are the details for different scores:\n",
      "  - Score 0: typically if the answer is completely incorrect, then the comprehensiveness is also zero score.\n",
      "  - Score 0: if the answer is correct but too short to fully answer the question, then we can give score 1 for comprehensiveness.\n",
      "      - Example:\n",
      "          - Question: How to use databricks API to create a cluster?\n",
      "          - Answer: First, you will need a Databricks access token with the appropriate permissions. You can generate this token through the Databricks UI under the 'User Settings' option. And then (the rest is missing)\n",
      "  - Score 1: the answer is correct and roughly answer the main aspects of the question, but it’s missing description about details. Or is completely missing details about one minor aspect.\n",
      "      - Example:\n",
      "          - Question: How to use databricks API to create a cluster?\n",
      "          - Answer: You will need a Databricks access token with the appropriate permissions. Then you’ll need to set up the request URL, then you can make the HTTP Request. Then you can handle the request response.\n",
      "      - Example:\n",
      "          - Question: How to use databricks API to create a cluster?\n",
      "          - Answer: You will need a Databricks access token with the appropriate permissions. Then you’ll need to set up the request URL, then you can make the HTTP Request. Then you can handle the request response.\n",
      "  - Score 1: the answer is correct, and covers all the main aspects of the question\n",
      "- Readability: How readable is the answer, does it have redundant information or incomplete information that hurts the readability of the answer.\n",
      "  - Score 0: the answer is completely unreadable, e.g. fully of symbols that’s hard to read; e.g. keeps repeating the words that it’s very hard to understand the meaning of the paragraph. No meaningful information can be extracted from the answer.\n",
      "  - Score 0: the answer is slightly readable, there are irrelevant symbols or repeated words, but it can roughly form a meaningful sentence that cover some aspects of the answer.\n",
      "      - Example:\n",
      "          - Question: How to use databricks API to create a cluster?\n",
      "          - Answer: You you  you  you  you  you  will need a Databricks access token with the appropriate permissions. And then then you’ll need to set up the request URL, then you can make the HTTP Request. Then Then Then Then Then Then Then Then Then\n",
      "  - Score 1: the answer is correct and mostly readable, but there is one obvious piece that’s affecting the readability (mentioning of irrelevant pieces, repeated words)\n",
      "      - Example:\n",
      "          - Question: How to terminate a databricks cluster\n",
      "          - Answer: In the Databricks workspace, navigate to the \"Clusters\" tab.\n",
      "          Find the cluster you want to terminate from the list of active clusters.\n",
      "          Click on the down-arrow next to the cluster name to open the cluster details.\n",
      "          Click on the \"Terminate\" button…………………………………..\n",
      "          A confirmation dialog will appear. Click \"Terminate\" again to confirm the action.\n",
      "  - Score 1: the answer is correct and reader friendly, no obvious piece that affect readability.\n",
      "\n",
      "\n",
      "\n",
      "Provided question:\n",
      "['How are you?']\n",
      "\n",
      "Provided answer: \n",
      "['I am good']\n",
      "\n",
      "Provided context: \n",
      "['This person should be polite.']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from databricks.labs.doc_qa.llm_utils import PromptTemplate\n",
    "import os\n",
    "import pandas as pd\n",
    "from databricks.labs.doc_qa.evaluators.templated_evaluator import OpenAIEvaluator, AnthropicEvaluator, ParameterDef, NoRetryPolicy, RetryPolicy\n",
    "from databricks.labs.doc_qa.variables.doc_qa_template_variables import anthropic_grading_template_scale_3, anthropic_grading_template_scale_1\n",
    "from databricks.labs.doc_qa.variables.doc_qa_template_variables import get_openai_grading_template_and_function\n",
    "\n",
    "\n",
    "# show debug log for all loggers\n",
    "import logging\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "\n",
    "retry_policy = RetryPolicy(max_retry_on_invalid_result=3, max_retry_on_exception=3)\n",
    "catch_error = True\n",
    "\n",
    "# anthropic_grading_prompt = anthropic_grading_template_scale_3\n",
    "anthropic_grading_prompt = anthropic_grading_template_scale_1\n",
    "\n",
    "openai_grading_prompt, openai_grading_function = get_openai_grading_template_and_function(scale=1, level_of_details=2)\n",
    "\n",
    "# Define the dummy data\n",
    "data = {\n",
    "    'question': ['How are you?'], \n",
    "    'answer': ['I am good'], \n",
    "    'context': ['This person should be polite.']\n",
    "}\n",
    "\n",
    "# Create the DataFrame\n",
    "df = pd.DataFrame(data)\n",
    "# Fill the prompt with the data as kargs\n",
    "print(openai_grading_prompt.format(**data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_rows: 10\n",
      "num_successful_rows: 10\n",
      "avg_correctness: 0.8\n",
      "avg_comprehensiveness: 0.8\n",
      "avg_readability: 1.0\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_successful</th>\n",
       "      <th>error_msg</th>\n",
       "      <th>reasoning_for_correctness</th>\n",
       "      <th>correctness</th>\n",
       "      <th>reasoning_for_comprehensiveness</th>\n",
       "      <th>comprehensiveness</th>\n",
       "      <th>reasoning_for_readability</th>\n",
       "      <th>readability</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer states \"I don't know\" which is an a...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer does not attempt to provide any exp...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is a simple and clear statement tha...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...</td>\n",
       "      <td>How does pyspark.pandas.Series.str.cat work?</td>\n",
       "      <td>I don't know how pyspark.pandas.Series.str.cat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The provided answer of \"I don't know\" does not...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Since the provided answer does not attempt to ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The provided answer of \"I don't know\" is very ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>How is LeafNode inherited in Spark MLlib?</td>\n",
       "      <td>I don't know\\n\\n&lt;details&gt;\\n&lt;summary&gt;Hint&lt;/summ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer says \"I don't know\" which does not ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Since the answer does not attempt to address t...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is very short and simply states \"I ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>7. What are the value members available for Co...</td>\n",
       "      <td>I don't know\\nThe Best Advice About Options I'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains how to use the c...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer fully covers the main aspects of us...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is well structured and written in c...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...</td>\n",
       "      <td>How can I use the clone method of StreamingConf?</td>\n",
       "      <td>The clone method of StreamingConf can be used ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer lists all the value members of the ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a complete list of all the...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer formats the list of value members i...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the value members of RuntimeInfo?</td>\n",
       "      <td>The value members of RuntimeInfo are:\\n\\n1.!=(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the name of t...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer directly states the name used in JS...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is a concise, readable sentence tha...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n[c](ShortType$.html \"See companion object\")\\...</td>\n",
       "      <td>What is the name of the type used in JSON seri...</td>\n",
       "      <td>The name of the type used in JSON serializatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer lists all the concrete value member...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides the full list of concrete ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer formats the members into a numbered...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n t\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the concrete value members of UDF5?</td>\n",
       "      <td>The concrete value members of UDF5 are:\\n\\n1. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that more context ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer explains that the provided context ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is written clearly and concisely in...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...</td>\n",
       "      <td>What is the return value of `cube`?</td>\n",
       "      <td>I don't know. The code snippet provided is not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer provides a full code example of usi...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a complete code example th...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is formatted as a readable code sni...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...</td>\n",
       "      <td>8. Can you provide an example of using FPGrowt...</td>\n",
       "      <td>I don't know\\n9. Can you provide an example of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer provides a detailed explanation of ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer covers the key aspects of explainin...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is well structured and easy to read...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n[c](DenseVector$.html \"See companion object\"...</td>\n",
       "      <td>What is DenseVector?</td>\n",
       "      <td>DenseVector is a class in Apache Spark's MLlib...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_successful error_msg                          reasoning_for_correctness  \\\n",
       "0           True      None  The answer states \"I don't know\" which is an a...   \n",
       "1           True      None  The provided answer of \"I don't know\" does not...   \n",
       "2           True      None  The answer says \"I don't know\" which does not ...   \n",
       "3           True      None  The answer correctly explains how to use the c...   \n",
       "4           True      None  The answer lists all the value members of the ...   \n",
       "5           True      None  The answer correctly states that the name of t...   \n",
       "6           True      None  The answer lists all the concrete value member...   \n",
       "7           True      None  The answer correctly states that more context ...   \n",
       "8           True      None  The answer provides a full code example of usi...   \n",
       "9           True      None  The answer provides a detailed explanation of ...   \n",
       "\n",
       "   correctness                    reasoning_for_comprehensiveness  \\\n",
       "0          1.0  The answer does not attempt to provide any exp...   \n",
       "1          0.0  Since the provided answer does not attempt to ...   \n",
       "2          0.0  Since the answer does not attempt to address t...   \n",
       "3          1.0  The answer fully covers the main aspects of us...   \n",
       "4          1.0  The answer provides a complete list of all the...   \n",
       "5          1.0  The answer directly states the name used in JS...   \n",
       "6          1.0  The answer provides the full list of concrete ...   \n",
       "7          1.0  The answer explains that the provided context ...   \n",
       "8          1.0  The answer provides a complete code example th...   \n",
       "9          1.0  The answer covers the key aspects of explainin...   \n",
       "\n",
       "   comprehensiveness                          reasoning_for_readability  \\\n",
       "0                1.0  The answer is a simple and clear statement tha...   \n",
       "1                0.0  The provided answer of \"I don't know\" is very ...   \n",
       "2                0.0  The answer is very short and simply states \"I ...   \n",
       "3                1.0  The answer is well structured and written in c...   \n",
       "4                1.0  The answer formats the list of value members i...   \n",
       "5                1.0  The answer is a concise, readable sentence tha...   \n",
       "6                1.0  The answer formats the members into a numbered...   \n",
       "7                1.0  The answer is written clearly and concisely in...   \n",
       "8                1.0  The answer is formatted as a readable code sni...   \n",
       "9                1.0  The answer is well structured and easy to read...   \n",
       "\n",
       "   readability                                            context  \\\n",
       "0          1.0  \\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...   \n",
       "1          1.0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "2          1.0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "3          1.0  \\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...   \n",
       "4          1.0  \\n c\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "5          1.0  \\n[c](ShortType$.html \"See companion object\")\\...   \n",
       "6          1.0  \\n t\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "7          1.0  \\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...   \n",
       "8          1.0  \\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...   \n",
       "9          1.0  \\n[c](DenseVector$.html \"See companion object\"...   \n",
       "\n",
       "                                            question  \\\n",
       "0       How does pyspark.pandas.Series.str.cat work?   \n",
       "1          How is LeafNode inherited in Spark MLlib?   \n",
       "2  7. What are the value members available for Co...   \n",
       "3   How can I use the clone method of StreamingConf?   \n",
       "4         What are the value members of RuntimeInfo?   \n",
       "5  What is the name of the type used in JSON seri...   \n",
       "6       What are the concrete value members of UDF5?   \n",
       "7                What is the return value of `cube`?   \n",
       "8  8. Can you provide an example of using FPGrowt...   \n",
       "9                               What is DenseVector?   \n",
       "\n",
       "                                              answer  \n",
       "0  I don't know how pyspark.pandas.Series.str.cat...  \n",
       "1  I don't know\\n\\n<details>\\n<summary>Hint</summ...  \n",
       "2  I don't know\\nThe Best Advice About Options I'...  \n",
       "3  The clone method of StreamingConf can be used ...  \n",
       "4  The value members of RuntimeInfo are:\\n\\n1.!=(...  \n",
       "5  The name of the type used in JSON serializatio...  \n",
       "6  The concrete value members of UDF5 are:\\n\\n1. ...  \n",
       "7  I don't know. The code snippet provided is not...  \n",
       "8  I don't know\\n9. Can you provide an example of...  \n",
       "9  DenseVector is a class in Apache Spark's MLlib...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_df = pd.read_csv(\"pre_grade_datasets/pre_grade_doc_qa_mpt_30b_chat.csv\")\n",
    "\n",
    "target_df = target_df.head(10)\n",
    "\n",
    "from databricks.labs.doc_qa.evaluators.templated_evaluator import OpenAIEvaluator, AnthropicEvaluator, ParameterDef, NoRetryPolicy\n",
    "\n",
    "output_parameters = [\n",
    "    ParameterDef(\"reasoning_for_correctness\", \"string\"),\n",
    "    ParameterDef(\"correctness\", \"number\", display_name=\"Score for correctness\"),\n",
    "    ParameterDef(\"reasoning_for_comprehensiveness\", \"string\"),\n",
    "    ParameterDef(\"comprehensiveness\", \"number\", display_name=\"Score for comprehensiveness\"),\n",
    "    ParameterDef(\"reasoning_for_readability\", \"string\"),\n",
    "    ParameterDef(\"readability\", \"number\", display_name=\"Score for readability\"),\n",
    "]\n",
    "\n",
    "anthropic_claude_2_evaluator = AnthropicEvaluator(model='claude-2', temperature=0.1, \n",
    "    grading_prompt_tempate=anthropic_grading_prompt, input_columns=[\"question\", \"answer\", \"context\"], \n",
    "    output_parameters=output_parameters, retry_policy=retry_policy)\n",
    "eval_result = anthropic_claude_2_evaluator.run_eval(dataset_df=target_df, concurrency=20, catch_error=catch_error)\n",
    "result_df = eval_result.to_dataframe()\n",
    "print(eval_result.summary())\n",
    "result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_rows: 10\n",
      "num_successful_rows: 10\n",
      "avg_correctness: 0.7\n",
      "avg_comprehensiveness: 0.5\n",
      "avg_readability: 0.875\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_successful</th>\n",
       "      <th>error_msg</th>\n",
       "      <th>reasoning_for_correctness</th>\n",
       "      <th>correctness</th>\n",
       "      <th>reasoning_for_comprehensiveness</th>\n",
       "      <th>comprehensiveness</th>\n",
       "      <th>reasoning_for_readability</th>\n",
       "      <th>readability</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer does not attempt to explain how pys...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer does not provide any information ab...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable but does not contain an...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...</td>\n",
       "      <td>How does pyspark.pandas.Series.str.cat work?</td>\n",
       "      <td>I don't know how pyspark.pandas.Series.str.cat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer does not provide any information re...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer does not provide any information re...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable but does not provide an...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>How is LeafNode inherited in Spark MLlib?</td>\n",
       "      <td>I don't know\\n\\n&lt;details&gt;\\n&lt;summary&gt;Hint&lt;/summ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer lists all the concrete value member...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a comprehensive list of al...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is formatted clearly with headings ...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n t\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the concrete value members of UDF5?</td>\n",
       "      <td>The concrete value members of UDF5 are:\\n\\n1. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer lists all the value members of Runt...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is comprehensive and lists all the ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is formatted clearly in a numbered ...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the value members of RuntimeInfo?</td>\n",
       "      <td>The value members of RuntimeInfo are:\\n\\n1.!=(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the name of t...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is concise and directly answers the...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is clear and concise. It is easy to...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n[c](ShortType$.html \"See companion object\")\\...</td>\n",
       "      <td>What is the name of the type used in JSON seri...</td>\n",
       "      <td>The name of the type used in JSON serializatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the function ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is very short and only states that ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable and concise.</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...</td>\n",
       "      <td>What is the return value of `cube`?</td>\n",
       "      <td>I don't know. The code snippet provided is not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains that the clone m...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a basic example of using t...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is readable and concise with no iss...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...</td>\n",
       "      <td>How can I use the clone method of StreamingConf?</td>\n",
       "      <td>The clone method of StreamingConf can be used ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer is completely incorrect as it does ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Since the answer is incorrect, the comprehensi...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is unreadable as it contains irrele...</td>\n",
       "      <td>0.00</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>7. What are the value members available for Co...</td>\n",
       "      <td>I don't know\\nThe Best Advice About Options I'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly defines DenseVector as a ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a high-level overview of D...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is readable overall but contains so...</td>\n",
       "      <td>0.75</td>\n",
       "      <td>\\n[c](DenseVector$.html \"See companion object\"...</td>\n",
       "      <td>What is DenseVector?</td>\n",
       "      <td>DenseVector is a class in Apache Spark's MLlib...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer provides a full example code that c...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer covers all the main steps to use FP...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is formatted properly with comments...</td>\n",
       "      <td>1.00</td>\n",
       "      <td>\\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...</td>\n",
       "      <td>8. Can you provide an example of using FPGrowt...</td>\n",
       "      <td>I don't know\\n9. Can you provide an example of...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_successful error_msg                          reasoning_for_correctness  \\\n",
       "0           True      None  The answer does not attempt to explain how pys...   \n",
       "1           True      None  The answer does not provide any information re...   \n",
       "2           True      None  The answer lists all the concrete value member...   \n",
       "3           True      None  The answer lists all the value members of Runt...   \n",
       "4           True      None  The answer correctly states that the name of t...   \n",
       "5           True      None  The answer correctly states that the function ...   \n",
       "6           True      None  The answer correctly explains that the clone m...   \n",
       "7           True      None  The answer is completely incorrect as it does ...   \n",
       "8           True      None  The answer correctly defines DenseVector as a ...   \n",
       "9           True      None  The answer provides a full example code that c...   \n",
       "\n",
       "   correctness                    reasoning_for_comprehensiveness  \\\n",
       "0          0.0  The answer does not provide any information ab...   \n",
       "1          0.0  The answer does not provide any information re...   \n",
       "2          1.0  The answer provides a comprehensive list of al...   \n",
       "3          1.0  The answer is comprehensive and lists all the ...   \n",
       "4          1.0  The answer is concise and directly answers the...   \n",
       "5          1.0  The answer is very short and only states that ...   \n",
       "6          1.0  The answer provides a basic example of using t...   \n",
       "7          0.0  Since the answer is incorrect, the comprehensi...   \n",
       "8          1.0  The answer provides a high-level overview of D...   \n",
       "9          1.0  The answer covers all the main steps to use FP...   \n",
       "\n",
       "   comprehensiveness                          reasoning_for_readability  \\\n",
       "0                0.0  The answer is readable but does not contain an...   \n",
       "1                0.0  The answer is readable but does not provide an...   \n",
       "2                1.0  The answer is formatted clearly with headings ...   \n",
       "3                1.0  The answer is formatted clearly in a numbered ...   \n",
       "4                1.0  The answer is clear and concise. It is easy to...   \n",
       "5                0.0                The answer is readable and concise.   \n",
       "6                0.5  The answer is readable and concise with no iss...   \n",
       "7                0.0  The answer is unreadable as it contains irrele...   \n",
       "8                0.5  The answer is readable overall but contains so...   \n",
       "9                1.0  The answer is formatted properly with comments...   \n",
       "\n",
       "   readability                                            context  \\\n",
       "0         1.00  \\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...   \n",
       "1         1.00  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "2         1.00  \\n t\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "3         1.00  \\n c\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "4         1.00  \\n[c](ShortType$.html \"See companion object\")\\...   \n",
       "5         1.00  \\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...   \n",
       "6         1.00  \\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...   \n",
       "7         0.00  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "8         0.75  \\n[c](DenseVector$.html \"See companion object\"...   \n",
       "9         1.00  \\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...   \n",
       "\n",
       "                                            question  \\\n",
       "0       How does pyspark.pandas.Series.str.cat work?   \n",
       "1          How is LeafNode inherited in Spark MLlib?   \n",
       "2       What are the concrete value members of UDF5?   \n",
       "3         What are the value members of RuntimeInfo?   \n",
       "4  What is the name of the type used in JSON seri...   \n",
       "5                What is the return value of `cube`?   \n",
       "6   How can I use the clone method of StreamingConf?   \n",
       "7  7. What are the value members available for Co...   \n",
       "8                               What is DenseVector?   \n",
       "9  8. Can you provide an example of using FPGrowt...   \n",
       "\n",
       "                                              answer  \n",
       "0  I don't know how pyspark.pandas.Series.str.cat...  \n",
       "1  I don't know\\n\\n<details>\\n<summary>Hint</summ...  \n",
       "2  The concrete value members of UDF5 are:\\n\\n1. ...  \n",
       "3  The value members of RuntimeInfo are:\\n\\n1.!=(...  \n",
       "4  The name of the type used in JSON serializatio...  \n",
       "5  I don't know. The code snippet provided is not...  \n",
       "6  The clone method of StreamingConf can be used ...  \n",
       "7  I don't know\\nThe Best Advice About Options I'...  \n",
       "8  DenseVector is a class in Apache Spark's MLlib...  \n",
       "9  I don't know\\n9. Can you provide an example of...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anthropic_claude_1_evaluator = AnthropicEvaluator(model='claude-1', temperature=0.1, \n",
    "    grading_prompt_tempate=anthropic_grading_prompt, input_columns=[\"question\", \"answer\", \"context\"], \n",
    "    output_parameters=output_parameters, retry_policy=retry_policy)\n",
    "eval_result = anthropic_claude_1_evaluator.run_eval(dataset_df=target_df, concurrency=20, catch_error=catch_error)\n",
    "result_df = eval_result.to_dataframe()\n",
    "print(eval_result.summary())\n",
    "result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_rows: 10\n",
      "num_successful_rows: 10\n",
      "avg_correctness: 0.5\n",
      "avg_comprehensiveness: 0.5\n",
      "avg_readability: 0.7\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_successful</th>\n",
       "      <th>error_msg</th>\n",
       "      <th>reasoning_for_correctness</th>\n",
       "      <th>correctness</th>\n",
       "      <th>reasoning_for_comprehensiveness</th>\n",
       "      <th>comprehensiveness</th>\n",
       "      <th>reasoning_for_readability</th>\n",
       "      <th>readability</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer is completely irrelevant to the que...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer does not address the question at al...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable but it is completely of...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>7. What are the value members available for Co...</td>\n",
       "      <td>I don't know\\nThe Best Advice About Options I'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer does not provide any information re...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer does not cover any aspect of the qu...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable but it does not provide...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...</td>\n",
       "      <td>8. Can you provide an example of using FPGrowt...</td>\n",
       "      <td>I don't know\\n9. Can you provide an example of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer does not provide any information ab...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer does not cover any aspect of the qu...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable but it does not provide...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>How is LeafNode inherited in Spark MLlib?</td>\n",
       "      <td>I don't know\\n\\n&lt;details&gt;\\n&lt;summary&gt;Hint&lt;/summ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains how to use the c...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a comprehensive explanatio...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is well-structured and easy to read...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>\\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...</td>\n",
       "      <td>How can I use the clone method of StreamingConf?</td>\n",
       "      <td>The clone method of StreamingConf can be used ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly identifies the name of th...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a comprehensive explanatio...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is well-structured and easy to read...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n[c](ShortType$.html \"See companion object\")\\...</td>\n",
       "      <td>What is the name of the type used in JSON seri...</td>\n",
       "      <td>The name of the type used in JSON serializatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly lists the value members o...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is comprehensive in listing the val...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is readable but includes irrelevant...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the value members of RuntimeInfo?</td>\n",
       "      <td>The value members of RuntimeInfo are:\\n\\n1.!=(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly lists the concrete value ...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is comprehensive in listing the con...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>The answer is readable until it includes irrel...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>\\n t\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the concrete value members of UDF5?</td>\n",
       "      <td>The concrete value members of UDF5 are:\\n\\n1. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly identifies that the conte...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is comprehensive in explaining why ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is clear, concise, and easy to unde...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...</td>\n",
       "      <td>What is the return value of `cube`?</td>\n",
       "      <td>I don't know. The code snippet provided is not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer is incorrect as it does not provide...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is not comprehensive as it does not...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>The answer is readable as it is a simple sente...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>\\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...</td>\n",
       "      <td>How does pyspark.pandas.Series.str.cat work?</td>\n",
       "      <td>I don't know how pyspark.pandas.Series.str.cat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains what DenseVector...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer provides a comprehensive explanatio...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>The answer is well-structured and easy to unde...</td>\n",
       "      <td>0.5</td>\n",
       "      <td>\\n[c](DenseVector$.html \"See companion object\"...</td>\n",
       "      <td>What is DenseVector?</td>\n",
       "      <td>DenseVector is a class in Apache Spark's MLlib...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_successful error_msg                          reasoning_for_correctness  \\\n",
       "0           True      None  The answer is completely irrelevant to the que...   \n",
       "1           True      None  The answer does not provide any information re...   \n",
       "2           True      None  The answer does not provide any information ab...   \n",
       "3           True      None  The answer correctly explains how to use the c...   \n",
       "4           True      None  The answer correctly identifies the name of th...   \n",
       "5           True      None  The answer correctly lists the value members o...   \n",
       "6           True      None  The answer correctly lists the concrete value ...   \n",
       "7           True      None  The answer correctly identifies that the conte...   \n",
       "8           True      None  The answer is incorrect as it does not provide...   \n",
       "9           True      None  The answer correctly explains what DenseVector...   \n",
       "\n",
       "   correctness                    reasoning_for_comprehensiveness  \\\n",
       "0          0.0  The answer does not address the question at al...   \n",
       "1          0.0  The answer does not cover any aspect of the qu...   \n",
       "2          0.0  The answer does not cover any aspect of the qu...   \n",
       "3          1.0  The answer provides a comprehensive explanatio...   \n",
       "4          1.0  The answer provides a comprehensive explanatio...   \n",
       "5          0.5  The answer is comprehensive in listing the val...   \n",
       "6          0.5  The answer is comprehensive in listing the con...   \n",
       "7          1.0  The answer is comprehensive in explaining why ...   \n",
       "8          0.0  The answer is not comprehensive as it does not...   \n",
       "9          1.0  The answer provides a comprehensive explanatio...   \n",
       "\n",
       "   comprehensiveness                          reasoning_for_readability  \\\n",
       "0                0.0  The answer is readable but it is completely of...   \n",
       "1                0.0  The answer is readable but it does not provide...   \n",
       "2                0.0  The answer is readable but it does not provide...   \n",
       "3                1.0  The answer is well-structured and easy to read...   \n",
       "4                1.0  The answer is well-structured and easy to read...   \n",
       "5                0.5  The answer is readable but includes irrelevant...   \n",
       "6                0.5  The answer is readable until it includes irrel...   \n",
       "7                1.0  The answer is clear, concise, and easy to unde...   \n",
       "8                0.0  The answer is readable as it is a simple sente...   \n",
       "9                1.0  The answer is well-structured and easy to unde...   \n",
       "\n",
       "   readability                                            context  \\\n",
       "0          0.0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "1          1.0  \\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...   \n",
       "2          1.0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "3          0.5  \\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...   \n",
       "4          1.0  \\n[c](ShortType$.html \"See companion object\")\\...   \n",
       "5          0.5  \\n c\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "6          0.5  \\n t\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "7          1.0  \\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...   \n",
       "8          1.0  \\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...   \n",
       "9          0.5  \\n[c](DenseVector$.html \"See companion object\"...   \n",
       "\n",
       "                                            question  \\\n",
       "0  7. What are the value members available for Co...   \n",
       "1  8. Can you provide an example of using FPGrowt...   \n",
       "2          How is LeafNode inherited in Spark MLlib?   \n",
       "3   How can I use the clone method of StreamingConf?   \n",
       "4  What is the name of the type used in JSON seri...   \n",
       "5         What are the value members of RuntimeInfo?   \n",
       "6       What are the concrete value members of UDF5?   \n",
       "7                What is the return value of `cube`?   \n",
       "8       How does pyspark.pandas.Series.str.cat work?   \n",
       "9                               What is DenseVector?   \n",
       "\n",
       "                                              answer  \n",
       "0  I don't know\\nThe Best Advice About Options I'...  \n",
       "1  I don't know\\n9. Can you provide an example of...  \n",
       "2  I don't know\\n\\n<details>\\n<summary>Hint</summ...  \n",
       "3  The clone method of StreamingConf can be used ...  \n",
       "4  The name of the type used in JSON serializatio...  \n",
       "5  The value members of RuntimeInfo are:\\n\\n1.!=(...  \n",
       "6  The concrete value members of UDF5 are:\\n\\n1. ...  \n",
       "7  I don't know. The code snippet provided is not...  \n",
       "8  I don't know how pyspark.pandas.Series.str.cat...  \n",
       "9  DenseVector is a class in Apache Spark's MLlib...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from databricks.labs.doc_qa.evaluators.templated_evaluator import OpenAIEvaluator\n",
    "\n",
    "openai_gpt_4_evaluator = OpenAIEvaluator(model=\"gpt-4\", temperature=0.1, \n",
    "    grading_prompt_tempate=openai_grading_prompt, \n",
    "    input_columns=[\"question\", \"answer\", \"context\"], openai_function=openai_grading_function,\n",
    "    retry_policy=retry_policy)\n",
    "eval_result = openai_gpt_4_evaluator.run_eval(dataset_df=target_df, concurrency=20, catch_error=catch_error)\n",
    "result_df = eval_result.to_dataframe()\n",
    "print(eval_result.summary())\n",
    "result_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_rows: 10\n",
      "num_successful_rows: 10\n",
      "avg_correctness: 0.8\n",
      "avg_comprehensiveness: 0.6\n",
      "avg_readability: 0.8\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_successful</th>\n",
       "      <th>error_msg</th>\n",
       "      <th>reasoning_for_correctness</th>\n",
       "      <th>correctness</th>\n",
       "      <th>reasoning_for_comprehensiveness</th>\n",
       "      <th>comprehensiveness</th>\n",
       "      <th>reasoning_for_readability</th>\n",
       "      <th>readability</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer is completely incorrect as it state...</td>\n",
       "      <td>0</td>\n",
       "      <td>N/A</td>\n",
       "      <td>0</td>\n",
       "      <td>N/A</td>\n",
       "      <td>0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>How is LeafNode inherited in Spark MLlib?</td>\n",
       "      <td>I don't know\\n\\n&lt;details&gt;\\n&lt;summary&gt;Hint&lt;/summ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the method is...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer provides a comprehensive explanatio...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is clear and concise.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...</td>\n",
       "      <td>How does pyspark.pandas.Series.str.cat work?</td>\n",
       "      <td>I don't know how pyspark.pandas.Series.str.cat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly lists the value members o...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer provides a comprehensive list of al...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is readable and well-structured.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the value members of RuntimeInfo?</td>\n",
       "      <td>The value members of RuntimeInfo are:\\n\\n1.!=(...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains that DenseVector...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer provides a comprehensive explanatio...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is reader-friendly and does not con...</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n[c](DenseVector$.html \"See companion object\"...</td>\n",
       "      <td>What is DenseVector?</td>\n",
       "      <td>DenseVector is a class in Apache Spark's MLlib...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer is completely incorrect and does no...</td>\n",
       "      <td>0</td>\n",
       "      <td>N/A</td>\n",
       "      <td>0</td>\n",
       "      <td>N/A</td>\n",
       "      <td>0</td>\n",
       "      <td>\\n c\\n \\n[org](../../../../index.html) \\n .\\n ...</td>\n",
       "      <td>7. What are the value members available for Co...</td>\n",
       "      <td>I don't know\\nThe Best Advice About Options I'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly lists all the concrete va...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer provides a comprehensive list of al...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is readable and well-structured.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n t\\n \\n[org](../../../../../index.html) \\n ....</td>\n",
       "      <td>What are the concrete value members of UDF5?</td>\n",
       "      <td>The concrete value members of UDF5 are:\\n\\n1. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the return va...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is not comprehensive as it does not...</td>\n",
       "      <td>0</td>\n",
       "      <td>The answer is readable and concise.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...</td>\n",
       "      <td>What is the return value of `cube`?</td>\n",
       "      <td>I don't know. The code snippet provided is not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly states that the name of t...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is not comprehensive as it only pro...</td>\n",
       "      <td>0</td>\n",
       "      <td>The answer is readable and concise.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n[c](ShortType$.html \"See companion object\")\\...</td>\n",
       "      <td>What is the name of the type used in JSON seri...</td>\n",
       "      <td>The name of the type used in JSON serializatio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly provides an example of us...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer provides a complete example of usin...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is well-formatted and easy to read.</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...</td>\n",
       "      <td>8. Can you provide an example of using FPGrowt...</td>\n",
       "      <td>I don't know\\n9. Can you provide an example of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>True</td>\n",
       "      <td>None</td>\n",
       "      <td>The answer correctly explains how to use the c...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer covers all the main aspects of the ...</td>\n",
       "      <td>1</td>\n",
       "      <td>The answer is reader-friendly and does not hav...</td>\n",
       "      <td>1</td>\n",
       "      <td>\\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...</td>\n",
       "      <td>How can I use the clone method of StreamingConf?</td>\n",
       "      <td>The clone method of StreamingConf can be used ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_successful error_msg                          reasoning_for_correctness  \\\n",
       "0           True      None  The answer is completely incorrect as it state...   \n",
       "1           True      None  The answer correctly states that the method is...   \n",
       "2           True      None  The answer correctly lists the value members o...   \n",
       "3           True      None  The answer correctly explains that DenseVector...   \n",
       "4           True      None  The answer is completely incorrect and does no...   \n",
       "5           True      None  The answer correctly lists all the concrete va...   \n",
       "6           True      None  The answer correctly states that the return va...   \n",
       "7           True      None  The answer correctly states that the name of t...   \n",
       "8           True      None  The answer correctly provides an example of us...   \n",
       "9           True      None  The answer correctly explains how to use the c...   \n",
       "\n",
       "   correctness                    reasoning_for_comprehensiveness  \\\n",
       "0            0                                                N/A   \n",
       "1            1  The answer provides a comprehensive explanatio...   \n",
       "2            1  The answer provides a comprehensive list of al...   \n",
       "3            1  The answer provides a comprehensive explanatio...   \n",
       "4            0                                                N/A   \n",
       "5            1  The answer provides a comprehensive list of al...   \n",
       "6            1  The answer is not comprehensive as it does not...   \n",
       "7            1  The answer is not comprehensive as it only pro...   \n",
       "8            1  The answer provides a complete example of usin...   \n",
       "9            1  The answer covers all the main aspects of the ...   \n",
       "\n",
       "   comprehensiveness                          reasoning_for_readability  \\\n",
       "0                  0                                                N/A   \n",
       "1                  1                   The answer is clear and concise.   \n",
       "2                  1        The answer is readable and well-structured.   \n",
       "3                  1  The answer is reader-friendly and does not con...   \n",
       "4                  0                                                N/A   \n",
       "5                  1        The answer is readable and well-structured.   \n",
       "6                  0                The answer is readable and concise.   \n",
       "7                  0                The answer is readable and concise.   \n",
       "8                  1     The answer is well-formatted and easy to read.   \n",
       "9                  1  The answer is reader-friendly and does not hav...   \n",
       "\n",
       "   readability                                            context  \\\n",
       "0            0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "1            1  \\n pyspark.pandas.Series.str.cat\\n [¶](#pyspar...   \n",
       "2            1  \\n c\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "3            1  \\n[c](DenseVector$.html \"See companion object\"...   \n",
       "4            0  \\n c\\n \\n[org](../../../../index.html) \\n .\\n ...   \n",
       "5            1  \\n t\\n \\n[org](../../../../../index.html) \\n ....   \n",
       "6            1  \\n![]()\\n cube\\n======\\n`cube.Rd` \\n Create a ...   \n",
       "7            1  \\n[c](ShortType$.html \"See companion object\")\\...   \n",
       "8            1  \\n FPGrowth\\n [¶](#fpgrowth \"Permalink to this...   \n",
       "9            1  \\n o\\n \\n[org](../../../index.html) \\n .\\n [ap...   \n",
       "\n",
       "                                            question  \\\n",
       "0          How is LeafNode inherited in Spark MLlib?   \n",
       "1       How does pyspark.pandas.Series.str.cat work?   \n",
       "2         What are the value members of RuntimeInfo?   \n",
       "3                               What is DenseVector?   \n",
       "4  7. What are the value members available for Co...   \n",
       "5       What are the concrete value members of UDF5?   \n",
       "6                What is the return value of `cube`?   \n",
       "7  What is the name of the type used in JSON seri...   \n",
       "8  8. Can you provide an example of using FPGrowt...   \n",
       "9   How can I use the clone method of StreamingConf?   \n",
       "\n",
       "                                              answer  \n",
       "0  I don't know\\n\\n<details>\\n<summary>Hint</summ...  \n",
       "1  I don't know how pyspark.pandas.Series.str.cat...  \n",
       "2  The value members of RuntimeInfo are:\\n\\n1.!=(...  \n",
       "3  DenseVector is a class in Apache Spark's MLlib...  \n",
       "4  I don't know\\nThe Best Advice About Options I'...  \n",
       "5  The concrete value members of UDF5 are:\\n\\n1. ...  \n",
       "6  I don't know. The code snippet provided is not...  \n",
       "7  The name of the type used in JSON serializatio...  \n",
       "8  I don't know\\n9. Can you provide an example of...  \n",
       "9  The clone method of StreamingConf can be used ...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "openai_gpt_35_evaluator = OpenAIEvaluator(model=\"gpt-3.5-turbo-16k\", temperature=0.1, \n",
    "                                          grading_prompt_tempate=openai_grading_prompt, \n",
    "    input_columns=[\"question\", \"answer\", \"context\"], openai_function=openai_grading_function,\n",
    "    retry_policy=retry_policy)\n",
    "\n",
    "eval_result = openai_gpt_35_evaluator.run_eval(dataset_df=target_df, concurrency=20, catch_error=catch_error)\n",
    "result_df = eval_result.to_dataframe()\n",
    "print(eval_result.summary())\n",
    "result_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "docs3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}