sdk/python/foundation-models/system/evaluation/text-generation/text-generation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Generation Evaluation\n",
    "\n",
    "This sample shows how use the evaluate a group of models against a given set of metrics for the `text-generation` task.\n",
    "\n",
    "### Evaluation dataset\n",
    "The CNN / DailyMail Dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. The current version supports both extractive and abstractive summarization, though the original version was created for machine reading and comprehension and abstractive question answering. Reference [cnn_dailymail](https://huggingface.co/datasets/cnn_dailymail).\n",
    "\n",
    "### Model\n",
    "The goal of evaluating models is to compare their performance on a variety of metrics. `text-generation` is generic task type that can be used for scenarios to generate text based on context provided. As such, the models you pick to compare must be finetuned for same scenario. Given that we have the cnn_dailymail dataset, we would like to look for models finetuned for this specific scenario. We will compare `gpt2`, `gpt2-medium` and `distilgpt2` in this sample, which are available in the `azureml` system registry.\n",
    "\n",
    "If you'd like to evaluate models that are not in the system registry, you can import those models to your workspace or organization registry and then evaluate them using the approach outlined in this sample.\n",
    "\n",
    "### Outline\n",
    "* Setup pre-requisites such as compute.\n",
    "* Pick the models to evaluate.\n",
    "* Pick and explore evaluate data.\n",
    "* Configure the evaluation jobs.\n",
    "* Run the evaluation jobs.\n",
    "* Review the evaluation metrics."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Setup pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `azureml` system registry\n",
    "* Set an optional experiment name\n",
    "* Check or create compute. A single GPU node can have multiple GPU cards. For example, in one node of `Standard_NC24rs_v3` there are 4 NVIDIA V100 GPUs while in `Standard_NC12s_v3`, there are 2 NVIDIA V100 GPUs. Refer to the [docs](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) for this information. The number of GPU cards per node is set in the param `gpus_per_node` below. Setting this value correctly will ensure utilization of all GPUs in the node. The recommended GPU compute SKUs can be found [here](https://learn.microsoft.com/en-us/azure/virtual-machines/ncv3-series) and [here](https://learn.microsoft.com/en-us/azure/virtual-machines/ndv2-series)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install dependencies by running below cell. This is not an optional step if running in a new environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "%pip install --upgrade azure-ai-ml\n",
    "%pip install --upgrade azure-identity\n",
    "%pip install --upgrade datasets==2.9.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1679319346668
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "workspace_ml_client = None\n",
    "try:\n",
    "    workspace_ml_client = MLClient.from_config(credential)\n",
    "    subscription_id = workspace_ml_client.subscription_id\n",
    "    workspace = workspace_ml_client.workspace_name\n",
    "    resource_group = workspace_ml_client.resource_group_name\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # Enter details of your AML workspace\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "    workspace_ml_client = MLClient(\n",
    "        credential, subscription_id, resource_group, workspace\n",
    "    )\n",
    "\n",
    "# the models, fine tuning pipelines and environments are available in the AzureML system registry, \"azureml-preview\"\n",
    "registry = \"azureml\"\n",
    "\n",
    "registry_ml_client = MLClient(\n",
    "    credential, subscription_id, resource_group, registry_name=registry\n",
    ")\n",
    "registry_ml_client"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Compute\n",
    "\n",
    "There are two ways to submit a job - through a compute or a serverless job.\n",
    "\n",
    "##### Serverless Job:\n",
    "\n",
    "In a serverless job, there is no need to create a compute explicitly.\n",
    "Simply pass the desired instance type value to the `instance_type` parameter while creating a pipeline job.\n",
    "This allows for quick and convenient job submission without the need for managing a compute cluster.\n",
    "\n",
    "##### Compute Job:\n",
    "\n",
    "To submit a job through a compute, you need to create a compute cluster beforehand.\n",
    "The following code below demonstrates how to create a gpu compute cluster.\n",
    "After creating the compute cluster, pass the name of the compute cluster to the `compute_name` parameter while submitting the pipeline job. This ensures that the job runs on the specified compute cluster, allowing for more control and customization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you already have a gpu cluster, mention it here. Else will create a new one with the name 'gpu-cluster-big'\n",
    "compute_cluster = \"gpu-cluster-big\"\n",
    "try:\n",
    "    compute = workspace_ml_client.compute.get(compute_cluster)\n",
    "    print(f\"GPU compute '{compute_cluster}' found.\")\n",
    "except Exception as ex:\n",
    "    print(f\"GPU compute '{compute_cluster}' not found. Creating new one.\")\n",
    "    compute = AmlCompute(\n",
    "        name=compute_cluster,\n",
    "        size=\"Standard_NC24rs_v3\",\n",
    "        max_instances=2,  # For multi node training set this to an integer value more than 1\n",
    "    )\n",
    "    workspace_ml_client.compute.begin_create_or_update(compute).wait()\n",
    "\n",
    "# generating a unique timestamp that can be used for names and versions that need to be unique\n",
    "timestamp = str(int(time.time()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below snippet will allow us to query number of GPU's present on the compute. We can use it to set `gpu_per_node` to ensure utilization of all GPUs in the node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the number of GPUs in a single node of the selected 'vm_size' compute.\n",
    "# Setting this to less than the number of GPUs will result in underutilized GPUs, taking longer to train.\n",
    "# Setting this to more than the number of GPUs will result in an error.\n",
    "gpus_per_node = 1  # default value\n",
    "gpu_count_found = False\n",
    "ws_computes = workspace_ml_client.compute.list_sizes()\n",
    "for ws_compute in ws_computes:\n",
    "    if ws_compute.name.lower() == compute.size.lower():\n",
    "        gpus_per_node = ws_compute.gpus\n",
    "        print(f\"Number of GPUs in compute {ws_compute.name} are {ws_compute.gpus}\")\n",
    "# if gpu_count_found not found, then print an error\n",
    "if gpus_per_node > 0:\n",
    "    gpu_count_found = True\n",
    "else:\n",
    "    gpu_count_found = False\n",
    "    print(f\"No GPUs found in compute. Number of GPUs in compute {compute.size} 0.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Pick the models to evaluate\n",
    "\n",
    "Verify that the models selected for evaluation are available in system registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1679319354708
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# need to specify model versions until the bug to support fetching the latest version using latest label is fixed\n",
    "model_details = [\n",
    "    {\"name\": \"distilgpt2\"},\n",
    "    {\"name\": \"gpt2\"},\n",
    "    {\"name\": \"gpt2-large\"},\n",
    "    {\"name\": \"gpt2-medium\"},\n",
    "    {\"name\": \"databricks-dolly-v2-12b\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "models = []\n",
    "\n",
    "for model in model_details:\n",
    "    reg_model = list(registry_ml_client.models.list(name=model[\"name\"]))[0]\n",
    "    print(reg_model.id)\n",
    "    models.append({**model, \"version\": reg_model.version})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "models"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Pick the test dataset for evaluation\n",
    "The next few cells show basic data preparation:\n",
    "* Visualize some data rows\n",
    "* We want this sample to run quickly, so we use a smaller dataset containing 10% of the original.\n",
    "* To use the entire dataset, uncomment the cells below and run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "hf_test_data = load_dataset(\"cnn_dailymail\", \"3.0.0\", split=\"test\", streaming=True)\n",
    "\n",
    "test_data_df = pd.DataFrame(hf_test_data.take(1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_data_df[\"input_string\"] = test_data_df[\"article\"].apply(lambda x: x[:100])\n",
    "test_data_df[\"ground_truth\"] = test_data_df[\"article\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_data_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = \"./small-test.jsonl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_data_df.to_json(test_data, lines=True, orient=\"records\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.read_json(test_data, lines=True).head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Submit the evaluation jobs using the model and data as inputs\n",
    "\n",
    "Create the job that uses the `model_evaluation_pipeline` component. We will submit one job per model.\n",
    "\n",
    "Note that the metrics that the evaluation jobs need to calculate are specified in the [eval_config.json](./eval_config.json) file. We calculate `rouge1`, `rouge2`, `bleu_3` and `bleu_4` in this sample.\n",
    "\n",
    "All supported evaluation configurations for `text-generation` can be found in [README](./README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.dsl import pipeline\n",
    "from azure.ai.ml import Input\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "# fetch the pipeline component\n",
    "pipeline_component_func = registry_ml_client.components.get(\n",
    "    name=\"model_evaluation_pipeline\", label=\"latest\"\n",
    ")\n",
    "\n",
    "\n",
    "# define the pipeline job\n",
    "@pipeline()\n",
    "def evaluation_pipeline(mlflow_model):\n",
    "    evaluation_job = pipeline_component_func(\n",
    "        # specify the foundation model available in the azureml system registry or a model from the workspace\n",
    "        # mlflow_model = Input(type=AssetTypes.MLFLOW_MODEL, path=f\"{mlflow_model_path}\"),\n",
    "        mlflow_model=mlflow_model,\n",
    "        # test data\n",
    "        test_data=Input(type=AssetTypes.URI_FILE, path=test_data),\n",
    "        # The following parameters map to the dataset fields\n",
    "        input_column_names=\"input_string\",\n",
    "        label_column_name=\"ground_truth\",\n",
    "        # compute settings\n",
    "        compute_name=compute_cluster,\n",
    "        # specify the instance type for serverless job\n",
    "        # instance_type= \"STANDARD_NC24\",\n",
    "        # Evaluation settings\n",
    "        task=\"text-generation\",\n",
    "        # config file containing the details of evaluation metrics to calculate\n",
    "        evaluation_config=Input(type=AssetTypes.URI_FILE, path=\"./eval-config.json\"),\n",
    "        # config cluster/device job is running on\n",
    "        # set device to GPU/CPU on basis if GPU count was found\n",
    "        device=\"auto\",\n",
    "    )\n",
    "    return {\"evaluation_result\": evaluation_job.outputs.evaluation_result}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the jobs, passing the model as a parameter to the pipeline created in the above step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# submit the pipeline job for each model that we want to evaluate\n",
    "# you could consider submitting the pipeline jobs in parallel, provided your cluster has multiple nodes\n",
    "pipeline_jobs = []\n",
    "\n",
    "experiment_name = \"text-generation-evaluation\"\n",
    "\n",
    "for model in models:\n",
    "    model_object = registry_ml_client.models.get(\n",
    "        model[\"name\"], version=model[\"version\"]\n",
    "    )\n",
    "    pipeline_object = evaluation_pipeline(\n",
    "        mlflow_model=Input(type=AssetTypes.MLFLOW_MODEL, path=f\"{model_object.id}\"),\n",
    "    )\n",
    "    # don't reuse cached results from previous jobs\n",
    "    pipeline_object.settings.force_rerun = True\n",
    "    pipeline_object.settings.default_compute = compute_cluster\n",
    "    pipeline_job = workspace_ml_client.jobs.create_or_update(\n",
    "        pipeline_object, experiment_name=experiment_name\n",
    "    )\n",
    "    # add model['name'] and pipeline_job.name as key value pairs to a dictionary\n",
    "    pipeline_jobs.append({\"model_name\": model[\"name\"], \"job_name\": pipeline_job.name})\n",
    "    # wait for the pipeline job to complete\n",
    "    workspace_ml_client.jobs.stream(pipeline_job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Review evaluation metrics\n",
    "Viewing the job in AzureML studio is the best way to analyze logs, metrics and outputs of jobs. You can create custom charts and compare metics across different jobs. See https://learn.microsoft.com/en-us/azure/machine-learning/how-to-log-view-metrics?tabs=interactive#view-jobsruns-information-in-the-studio to learn more.\n",
    "\n",
    "![Model evaluation dashboard in AzureML studio](./text-generation-eval-dashboard.png)\n",
    "\n",
    "However, we may need to access and review metrics programmatically for which we will use MLflow, which is the recommended client for logging and querying metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow, json\n",
    "\n",
    "mlflow_tracking_uri = workspace_ml_client.workspaces.get(\n",
    "    workspace_ml_client.workspace_name\n",
    ").mlflow_tracking_uri\n",
    "mlflow.set_tracking_uri(mlflow_tracking_uri)\n",
    "\n",
    "metrics_df = pd.DataFrame()\n",
    "for job in pipeline_jobs:\n",
    "    # concat 'tags.mlflow.rootRunId=' and pipeline_job.name in single quotes as filter variable\n",
    "    filter = \"tags.mlflow.rootRunId='\" + job[\"job_name\"] + \"'\"\n",
    "    runs = mlflow.search_runs(\n",
    "        experiment_names=[experiment_name], filter_string=filter, output_format=\"list\"\n",
    "    )\n",
    "    # get the compute_metrics runs.\n",
    "    # using a hacky way till 'Bug 2320997: not able to show eval metrics in FT notebooks - mlflow client now showing display names' is fixed\n",
    "    for run in runs:\n",
    "        # else, check if run.data.metrics.accuracy exists\n",
    "        if \"exact_match\" in run.data.metrics:\n",
    "            # get the metrics from the mlflow run\n",
    "            run_metric = run.data.metrics\n",
    "            # add the model name to the run_metric dictionary\n",
    "            run_metric[\"model_name\"] = job[\"model_name\"]\n",
    "            # convert the run_metric dictionary to a pandas dataframe\n",
    "            temp_df = pd.DataFrame(run_metric, index=[0])\n",
    "            # concat the temp_df to the metrics_df\n",
    "            metrics_df = pd.concat([metrics_df, temp_df], ignore_index=True)\n",
    "\n",
    "# move the model_name columns to the first column\n",
    "cols = metrics_df.columns.tolist()\n",
    "cols = cols[-1:] + cols[:-1]\n",
    "metrics_df = metrics_df[cols]\n",
    "metrics_df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}