08B - Using Automated Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Automated Machine Learning\n",
    "\n",
    "There are many kinds of machine learning algorithm that you can use to train a model, and sometimes it's not easy to determine the most effective algorithm for your particular data and prediction requirements. Additionally, you can significantly affect the predictive performance of a model by preprocessing the training data, using techniques such as normalization, missing feature imputation, and others. In your quest to find the *best* model for your requirements, you may need to try many combinations of algorithms and preprocessing transformations; which takes a lot of time and compute resources.\n",
    "\n",
    "Azure Machine Learning enables you to automate the comparison of models trained using different algorithms and preprocessing options. You can use the visual interface in [Azure Machine Learning studio](https://ml/azure.com) or the SDK to leverage this capability. he SDK gives you greater control over the settings for the automated machine learning experiment, but the visual interface is easier to use. In this lab, you'll explore automated machine learning using the SDK.\n",
    "\n",
    "## Connect to Your Workspace\n",
    "\n",
    "The first thing you need to do is to connect to your workspace using the Azure ML SDK.\n",
    "\n",
    "> **Note**: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ready to use Azure ML 1.10.0 to work with Lab01A\n"
     ]
    }
   ],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data for Automated Machine Learning\n",
    "\n",
    "You don't need to create a training script for automated machine learning, but you do need to load the training data. In this case, you'll create a dataset containing details of diabetes patients (just as you did in previous labs), and then split this into two datasets: one for training, and another for model validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset already registered.\n",
      "Data ready!\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "if 'diabetes dataset' not in ws.datasets:\n",
    "    default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                        target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                        overwrite=True, # Replace existing files of the same name\n",
    "                        show_progress=True)\n",
    "\n",
    "    #Create a tabular dataset from the path on the datastore (this may take a short while)\n",
    "    tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "    # Register the tabular dataset\n",
    "    try:\n",
    "        tab_data_set = tab_data_set.register(workspace=ws, \n",
    "                                name='diabetes dataset',\n",
    "                                description='diabetes data',\n",
    "                                tags = {'format':'CSV'},\n",
    "                                create_new_version=True)\n",
    "        print('Dataset registered.')\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "else:\n",
    "    print('Dataset already registered.')\n",
    "\n",
    "\n",
    "# Split the dataset into training and validation subsets\n",
    "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
    "train_ds, test_ds = diabetes_ds.random_split(percentage=0.7, seed=123)\n",
    "print(\"Data ready!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Automated Machine Learning\n",
    "\n",
    "Now you're ready to configure the automated machine learning experiment. To do this, you'll need a run configuration that includes the required packages for the experiment environment, and a set of configuration settings that specifies how many combinations to try, which metric to use when evaluating models, and so on.\n",
    "\n",
    "> **Note**: In this example, you'll run the automated machine learning experiment on local compute to avoid waiting for a cluster to start. This will cause each iteration (child-run) to run serially rather than in parallel. For this reason, we're restricting the experiment to 6 iterations to reduce the amount of time taken. In reality, you'd likely try many more iterations on a compute cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ready for Auto ML run.\n"
     ]
    }
   ],
   "source": [
    "from azureml.train.automl import AutoMLConfig\n",
    "\n",
    "automl_config = AutoMLConfig(name='Automated ML Experiment',\n",
    "                             task='classification',\n",
    "                             compute_target='local',\n",
    "                             enable_local_managed=True,\n",
    "                             training_data = train_ds,\n",
    "                             validation_data = test_ds,\n",
    "                             label_column_name='Diabetic',\n",
    "                             iterations=6,\n",
    "                             primary_metric = 'AUC_weighted',\n",
    "                             max_concurrent_iterations=4,\n",
    "                             featurization='auto'\n",
    "                             )\n",
    "\n",
    "print(\"Ready for Auto ML run.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run an Automated Machine Learning Experiment\n",
    "\n",
    "OK, you're ready to go. Let's run the automated machine learning experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitting Auto ML experiment...\n",
      "Running on local conda or docker.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bd81bce740e6438fa6bc25a16c671d9f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_AutoMLWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/diabetes_automl/runs/AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e?wsid=/subscriptions/35241d74-3b9e-4778-bb92-4bb15e7b0410/resourcegroups/DP-100/workspaces/Lab01A\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"run_properties\": {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"created_utc\": \"2020-08-09T05:27:12.802783Z\", \"properties\": {\"num_iterations\": \"6\", \"training_type\": \"TrainFull\", \"acquisition_function\": \"EI\", \"primary_metric\": \"AUC_weighted\", \"train_split\": \"0\", \"acquisition_parameter\": \"0\", \"num_cross_validation\": null, \"target\": \"local\", \"RawAMLSettingsString\": null, \"AMLSettingsJsonString\": \"{\\\"path\\\":null,\\\"name\\\":\\\"Automated ML Experiment\\\",\\\"subscription_id\\\":null,\\\"resource_group\\\":null,\\\"workspace_name\\\":null,\\\"region\\\":null,\\\"compute_target\\\":null,\\\"spark_service\\\":null,\\\"azure_service\\\":null,\\\"_local_managed_run_id\\\":null,\\\"iterations\\\":6,\\\"primary_metric\\\":\\\"AUC_weighted\\\",\\\"task_type\\\":\\\"classification\\\",\\\"data_script\\\":null,\\\"validation_size\\\":0.0,\\\"n_cross_validations\\\":null,\\\"y_min\\\":null,\\\"y_max\\\":null,\\\"num_classes\\\":null,\\\"featurization\\\":\\\"auto\\\",\\\"_ignore_package_version_incompatibilities\\\":false,\\\"is_timeseries\\\":false,\\\"max_cores_per_iteration\\\":1,\\\"max_concurrent_iterations\\\":4,\\\"iteration_timeout_minutes\\\":null,\\\"mem_in_mb\\\":null,\\\"enforce_time_on_windows\\\":false,\\\"experiment_timeout_minutes\\\":8640,\\\"experiment_exit_score\\\":null,\\\"whitelist_models\\\":null,\\\"blacklist_algos\\\":[\\\"TensorFlowLinearClassifier\\\"],\\\"supported_models\\\":[\\\"LogisticRegression\\\",\\\"SVM\\\",\\\"ExtremeRandomTrees\\\",\\\"KNN\\\",\\\"TensorFlowDNN\\\",\\\"DecisionTree\\\",\\\"TensorFlowLinearClassifier\\\",\\\"LinearSVM\\\",\\\"XGBoostClassifier\\\",\\\"SGD\\\",\\\"MultinomialNaiveBayes\\\",\\\"AveragedPerceptronClassifier\\\",\\\"BernoulliNaiveBayes\\\",\\\"GradientBoosting\\\",\\\"RandomForest\\\",\\\"LightGBM\\\"],\\\"auto_blacklist\\\":true,\\\"blacklist_samples_reached\\\":false,\\\"exclude_nan_labels\\\":true,\\\"verbosity\\\":20,\\\"_debug_log\\\":\\\"automl.log\\\",\\\"show_warnings\\\":false,\\\"model_explainability\\\":true,\\\"service_url\\\":null,\\\"sdk_url\\\":null,\\\"sdk_packages\\\":null,\\\"enable_onnx_compatible_models\\\":false,\\\"enable_split_onnx_featurizer_estimator_models\\\":false,\\\"vm_type\\\":null,\\\"telemetry_verbosity\\\":20,\\\"send_telemetry\\\":true,\\\"enable_dnn\\\":false,\\\"force_text_dnn\\\":false,\\\"enable_feature_sweeping\\\":true,\\\"enable_early_stopping\\\":false,\\\"early_stopping_n_iters\\\":10,\\\"metrics\\\":null,\\\"enable_ensembling\\\":true,\\\"enable_stack_ensembling\\\":true,\\\"ensemble_iterations\\\":6,\\\"enable_tf\\\":false,\\\"enable_subsampling\\\":false,\\\"subsample_seed\\\":null,\\\"enable_nimbusml\\\":false,\\\"enable_streaming\\\":false,\\\"force_streaming\\\":false,\\\"track_child_runs\\\":true,\\\"label_column_name\\\":\\\"Diabetic\\\",\\\"weight_column_name\\\":null,\\\"cv_split_column_names\\\":null,\\\"enable_local_managed\\\":true,\\\"cost_mode\\\":1,\\\"lag_length\\\":0,\\\"metric_operation\\\":\\\"maximize\\\",\\\"preprocess\\\":true,\\\"scenario\\\":\\\"SDK-Compatible\\\"}\", \"DataPrepJsonString\": null, \"EnableSubsampling\": \"False\", \"runTemplate\": \"AutoML\", \"azureml.runsource\": \"automl\", \"display_task_type\": \"classification\", \"dependencies_versions\": \"{\\\"azureml-widgets\\\": \\\"1.10.0\\\", \\\"azureml-train\\\": \\\"1.10.0\\\", \\\"azureml-train-restclients-hyperdrive\\\": \\\"1.10.0\\\", \\\"azureml-train-core\\\": \\\"1.10.0\\\", \\\"azureml-train-automl\\\": \\\"1.10.0\\\", \\\"azureml-train-automl-runtime\\\": \\\"1.10.0\\\", \\\"azureml-train-automl-client\\\": \\\"1.10.0\\\", \\\"azureml-tensorboard\\\": \\\"1.10.0\\\", \\\"azureml-telemetry\\\": \\\"1.10.0\\\", \\\"azureml-sdk\\\": \\\"1.10.0\\\", \\\"azureml-samples\\\": \\\"0+unknown\\\", \\\"azureml-pipeline\\\": \\\"1.10.0\\\", \\\"azureml-pipeline-steps\\\": \\\"1.10.0\\\", \\\"azureml-pipeline-core\\\": \\\"1.10.0\\\", \\\"azureml-opendatasets\\\": \\\"1.10.0\\\", \\\"azureml-monitoring\\\": \\\"0.1.0a21\\\", \\\"azureml-model-management-sdk\\\": \\\"1.0.1b6.post1\\\", \\\"azureml-mlflow\\\": \\\"1.10.0\\\", \\\"azureml-interpret\\\": \\\"1.10.0\\\", \\\"azureml-explain-model\\\": \\\"1.10.0\\\", \\\"azureml-defaults\\\": \\\"1.10.0\\\", \\\"azureml-dataset-runtime\\\": \\\"1.10.0\\\", \\\"azureml-dataprep\\\": \\\"1.9.3\\\", \\\"azureml-dataprep-native\\\": \\\"14.2.1\\\", \\\"azureml-datadrift\\\": \\\"1.10.0\\\", \\\"azureml-core\\\": \\\"1.10.0\\\", \\\"azureml-contrib-services\\\": \\\"1.10.0\\\", \\\"azureml-contrib-server\\\": \\\"1.10.0\\\", \\\"azureml-contrib-reinforcementlearning\\\": \\\"1.10.0\\\", \\\"azureml-contrib-pipeline-steps\\\": \\\"1.10.0\\\", \\\"azureml-contrib-notebook\\\": \\\"1.10.0\\\", \\\"azureml-contrib-interpret\\\": \\\"1.10.0\\\", \\\"azureml-contrib-gbdt\\\": \\\"1.10.0\\\", \\\"azureml-contrib-fairness\\\": \\\"1.10.0\\\", \\\"azureml-contrib-dataset\\\": \\\"1.10.0\\\", \\\"azureml-cli-common\\\": \\\"1.10.0\\\", \\\"azureml-automl-runtime\\\": \\\"1.10.0\\\", \\\"azureml-automl-core\\\": \\\"1.10.0\\\", \\\"azureml-accel-models\\\": \\\"1.10.0\\\"}\", \"ClientSdkVersion\": \"1.10.0\", \"ClientType\": \"SDK\", \"environment_cpu_name\": \"AzureML-AutoML\", \"environment_cpu_version\": \"29\", \"environment_gpu_name\": \"AzureML-AutoML-GPU\", \"environment_gpu_version\": \"18\", \"root_attribution\": \"automl\", \"attribution\": \"AutoML\", \"CancelUri\": \"https://eastus.experiments.azureml.net/jasmine/v1.0/subscriptions/35241d74-3b9e-4778-bb92-4bb15e7b0410/resourceGroups/DP-100/providers/Microsoft.MachineLearningServices/workspaces/Lab01A/experiment/diabetes_automl/cancel/AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"Orchestrator\": \"AutoML\", \"_azureml.ComputeTargetType\": \"local\", \"_wrapper_run_id\": \"diabetes_automl_1596950833_eb6b0cff\", \"azureml.git.repository_uri\": \"https://github.com/MicrosoftLearning/DP100\", \"mlflow.source.git.repoURL\": \"https://github.com/MicrosoftLearning/DP100\", \"azureml.git.branch\": \"master\", \"mlflow.source.git.branch\": \"master\", \"azureml.git.commit\": \"17ac2a2b5518c3cfae2d3782d6184374c6c20ed1\", \"mlflow.source.git.commit\": \"17ac2a2b5518c3cfae2d3782d6184374c6c20ed1\", \"azureml.git.dirty\": \"True\", \"ProblemInfoJsonString\": \"{\\\"dataset_num_categorical\\\": 0, \\\"is_sparse\\\": true, \\\"subsampling\\\": false, \\\"dataset_classes\\\": 2, \\\"dataset_features\\\": 23, \\\"dataset_samples\\\": 10555, \\\"single_frequency_class_detected\\\": false}\", \"feature_skus\": \"automatedml_sdk_guardrails\"}, \"tags\": {\"model_explain_run\": \"best_run\", \"azureml.automlComponent\": \"AutoML\", \"model_explain_best_run_child_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0\", \"best_score\": \"0.9904812577250306\", \"best_pipeline\": \"LightGBM\", \"automl_best_child_run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0\"}, \"end_time_utc\": \"2020-08-09T05:34:08.841906Z\", \"status\": \"Completed\", \"log_files\": {}, \"log_groups\": [], \"run_duration\": \"0:06:56\"}, \"child_runs\": [{\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0\", \"run_number\": 3, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:32:31.562138Z\", \"end_time\": \"2020-08-09T05:32:43.765476Z\", \"created_time\": \"2020-08-09T05:32:30.506745Z\", \"created_time_dt\": \"2020-08-09T05:32:30.506745Z\", \"duration\": \"0:00:13\", \"iteration\": \"0\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.99048126, \"best_metric\": 0.99048126}, {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_1\", \"run_number\": 4, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:32:46.397131Z\", \"end_time\": \"2020-08-09T05:32:59.244024Z\", \"created_time\": \"2020-08-09T05:32:45.464039Z\", \"created_time_dt\": \"2020-08-09T05:32:45.464039Z\", \"duration\": \"0:00:13\", \"iteration\": \"1\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"MaxAbsScaler, XGBoostClassifier\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.98822706, \"best_metric\": 0.99048126}, {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_2\", \"run_number\": 5, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:33:00.774716Z\", \"end_time\": \"2020-08-09T05:33:13.000126Z\", \"created_time\": \"2020-08-09T05:33:00.01384Z\", \"created_time_dt\": \"2020-08-09T05:33:00.01384Z\", \"duration\": \"0:00:12\", \"iteration\": \"2\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.96464438, \"best_metric\": 0.99048126}, {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_3\", \"run_number\": 6, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:33:15.043101Z\", \"end_time\": \"2020-08-09T05:33:26.934344Z\", \"created_time\": \"2020-08-09T05:33:14.664131Z\", \"created_time_dt\": \"2020-08-09T05:33:14.664131Z\", \"duration\": \"0:00:12\", \"iteration\": \"3\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.93542433, \"best_metric\": 0.99048126}, {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_4\", \"run_number\": 7, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:33:27.958848Z\", \"end_time\": \"2020-08-09T05:33:40.561402Z\", \"created_time\": \"2020-08-09T05:33:27.638584Z\", \"created_time_dt\": \"2020-08-09T05:33:27.638584Z\", \"duration\": \"0:00:12\", \"iteration\": \"4\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"VotingEnsemble\", \"run_properties\": \"classification_labels=None,\\n               estimators=[('0', Pipeline(memory=None,\\n     steps=[('maxabsscaler', MaxAbsScaler(copy=True\", \"primary_metric\": 0.98912033, \"best_metric\": 0.99048126}, {\"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_5\", \"run_number\": 8, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-08-09T05:33:41.211753Z\", \"end_time\": \"2020-08-09T05:33:55.756211Z\", \"created_time\": \"2020-08-09T05:33:41.055471Z\", \"created_time_dt\": \"2020-08-09T05:33:41.055471Z\", \"duration\": \"0:00:14\", \"iteration\": \"5\", \"goal\": \"AUC_weighted_max\", \"run_name\": \"StackEnsemble\", \"run_properties\": \"base_learners=[('0', Pipeline(memory=None,\\n     steps=[('maxabsscaler', MaxAbsScaler(copy=True\", \"primary_metric\": 0.99011173, \"best_metric\": 0.99048126}], \"children_metrics\": {\"categories\": [0], \"series\": {\"balanced_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"balanced_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9451102807249344, 0.9353118302218928, 0.879876258174114, 0.8589217350390292, 0.9432495485231087, 0.9410489263717672]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"balanced_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344]}], \"AUC_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"AUC_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9904812577250306, 0.9882270647231618, 0.9646443784573144, 0.9354243290080001, 0.9891203301605712, 0.9901117328369252]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"AUC_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306]}], \"log_loss\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"log_loss\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.1186120500342212, 0.14557718392083688, 0.2792120641292046, 0.3291011616640987, 0.14047606526049503, 0.13282828606264097]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"log_loss_min\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.1186120500342212, 0.1186120500342212, 0.1186120500342212, 0.1186120500342212, 0.1186120500342212, 0.1186120500342212]}], \"matthews_correlation\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"matthews_correlation\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8918972733002116, 0.8739368529349385, 0.7839990705804014, 0.6873432814175866, 0.8897163768716294, 0.887982630247864]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"matthews_correlation_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8918972733002116, 0.8918972733002116, 0.8918972733002116, 0.8918972733002116, 0.8918972733002116, 0.8918972733002116]}], \"f1_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"f1_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9459413118209691, 0.9369393323381163, 0.8904325791498287, 0.8377531294448342, 0.9448314372240936, 0.9439033064867718]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"f1_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9459413118209691, 0.9459413118209691, 0.9459413118209691, 0.9459413118209691, 0.9459413118209691, 0.9459413118209691]}], \"recall_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"recall_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.944206974128234, 0.905511811023622, 0.8483689538807649, 0.9511811023622048, 0.9505061867266592]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"recall_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654]}], \"AUC_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"AUC_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9916008305485999, 0.9896202682201284, 0.9680211964505562, 0.9335004384294483, 0.9904181212444058, 0.9901889991535083]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"AUC_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9916008305485999, 0.9916008305485999, 0.9916008305485999, 0.9916008305485999, 0.9916008305485999, 0.9916008305485999]}], \"recall_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"recall_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9451102807249344, 0.9353118302218928, 0.879876258174114, 0.8589217350390292, 0.9432495485231087, 0.9410489263717672]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"recall_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344, 0.9451102807249344]}], \"accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.944206974128234, 0.905511811023622, 0.8483689538807649, 0.9511811023622048, 0.9505061867266592]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654]}], \"weighted_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"weighted_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9576485145517856, 0.951311550164289, 0.9259869979754377, 0.8399404183876749, 0.957516056346901, 0.9580597267246126]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"weighted_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9576485145517856, 0.9576485145517856, 0.9576485145517856, 0.9576485145517856, 0.9576485145517856, 0.9580597267246126]}], \"recall_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"recall_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.944206974128234, 0.905511811023622, 0.8483689538807649, 0.9511811023622048, 0.9505061867266592]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"recall_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654]}], \"f1_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"f1_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.944206974128234, 0.905511811023622, 0.8483689538807649, 0.9511811023622048, 0.9505061867266592]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"f1_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654]}], \"precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.944206974128234, 0.905511811023622, 0.8483689538807649, 0.9511811023622048, 0.9505061867266592]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654, 0.9520809898762654]}], \"precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9467885716011675, 0.9386313269550405, 0.904509711679066, 0.8290695020590788, 0.9464726664195969, 0.946953333561266]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9467885716011675, 0.9467885716011675, 0.9467885716011675, 0.9467885716011675, 0.9467885716011675, 0.946953333561266]}], \"average_precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"average_precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9918359683181054, 0.9898548572689291, 0.9684854159721146, 0.9355429451983861, 0.9904731978164714, 0.9900902573834296]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"average_precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9918359683181054, 0.9918359683181054, 0.9918359683181054, 0.9918359683181054, 0.9918359683181054, 0.9918359683181054]}], \"precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9520038306966419, 0.9440467334517999, 0.905332528755463, 0.8655349211935734, 0.9510517666463189, 0.9503334312304531]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9520038306966419, 0.9520038306966419, 0.9520038306966419, 0.9520038306966419, 0.9520038306966419, 0.9520038306966419]}], \"f1_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"f1_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.952035905457969, 0.9441010177866949, 0.9040304068007265, 0.8516367978144287, 0.9510926786434379, 0.9503416559403539]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"f1_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.952035905457969, 0.952035905457969, 0.952035905457969, 0.952035905457969, 0.952035905457969, 0.952035905457969]}], \"norm_macro_recall\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"norm_macro_recall\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8902205614498688, 0.8706236604437856, 0.7597525163482279, 0.7178434700780585, 0.8864990970462174, 0.8820978527435344]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"norm_macro_recall_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8902205614498688, 0.8902205614498688, 0.8902205614498688, 0.8902205614498688, 0.8902205614498688, 0.8902205614498688]}], \"AUC_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"AUC_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9904812577250306, 0.9882270647231617, 0.9646443784573144, 0.935424329008, 0.9891203301605712, 0.9901117328369253]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"AUC_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306, 0.9904812577250306]}], \"average_precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"average_precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9910085472336622, 0.9889316609728803, 0.9667965312520298, 0.9383587959638999, 0.9897982268546099, 0.9904757667767728]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"average_precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9910085472336622, 0.9910085472336622, 0.9910085472336622, 0.9910085472336622, 0.9910085472336622, 0.9910085472336622]}], \"average_precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"markers\", \"name\": \"average_precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9888863976623419, 0.9863803228939675, 0.9594950407384583, 0.9240998400520166, 0.9875675359339622, 0.9883757670690676]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\"], \"mode\": \"lines\", \"name\": \"average_precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9888863976623419, 0.9888863976623419, 0.9888863976623419, 0.9888863976623419, 0.9888863976623419, 0.9888863976623419]}]}, \"metricName\": null, \"primaryMetricName\": \"AUC_weighted\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"experiment_status\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], \"series\": [{\"data\": [\"DatasetEvaluation\", \"FeaturesGeneration\", \"DatasetFeaturization\", \"DatasetFeaturizationCompleted\", \"ModelSelection\", \"BestRunExplainModel\", \"ModelExplanationDataSetSetup\", \"PickSurrogateModel\", \"EngineeredFeatureExplanations\", \"EngineeredFeatureExplanations\", \"BestRunExplainModel\"]}]}, {\"name\": \"experiment_status_description\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], \"series\": [{\"data\": [\"Gathering dataset statistics.\", \"Generating features for the dataset.\", \"Beginning to fit featurizers and featurize the dataset.\", \"Completed fit featurizers and featurizing the dataset.\", \"Beginning model selection.\", \"Best run model explanations started\", \"Model explanations data setup completed\", \"Choosing LightGBM as the surrogate model for explanations\", \"Computation of engineered features started\", \"Computation of engineered features completed\", \"Best run model explanations completed\"]}]}, {\"name\": \"balanced_accuracy\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9451102807249344]}]}, {\"name\": \"log_loss\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.1186120500342212]}]}, {\"name\": \"f1_score_macro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9459413118209691]}]}, {\"name\": \"recall_score_macro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9451102807249344]}]}, {\"name\": \"AUC_micro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9916008305485999]}]}, {\"name\": \"weighted_accuracy\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9576485145517856]}]}, {\"name\": \"AUC_weighted\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9904812577250306]}]}, {\"name\": \"matthews_correlation\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.8918972733002116]}]}, {\"name\": \"precision_score_micro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520809898762654]}]}, {\"name\": \"f1_score_micro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520809898762654]}]}, {\"name\": \"accuracy\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520809898762654]}]}, {\"name\": \"norm_macro_recall\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.8902205614498688]}]}, {\"name\": \"recall_score_micro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520809898762654]}]}, {\"name\": \"average_precision_score_micro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9918359683181054]}]}, {\"name\": \"recall_score_weighted\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520809898762654]}]}, {\"name\": \"precision_score_macro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9467885716011675]}]}, {\"name\": \"precision_score_weighted\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9520038306966419]}]}, {\"name\": \"f1_score_weighted\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.952035905457969]}]}, {\"name\": \"average_precision_score_weighted\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9910085472336622]}]}, {\"name\": \"average_precision_score_macro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9888863976623419]}]}, {\"name\": \"AUC_macro\", \"run_id\": \"AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e\", \"categories\": [0], \"series\": [{\"data\": [0.9904812577250306]}]}], \"run_logs\": \"\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.10.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Current status: DatasetFeaturizationCompleted. Completed fit featurizers and featurizing the dataset.\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Class balancing detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  Your inputs were analyzed, and all classes are balanced in your training data.\n",
      "              Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  No feature missing values were detected in the training data.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         High cardinality feature detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  Your inputs were analyzed, and no high cardinality features were detected.\n",
      "              Learn more about high cardinality feature handling: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   MaxAbsScaler LightGBM                          0:00:12       0.9905    0.9905\n",
      "         1   MaxAbsScaler XGBoostClassifier                 0:00:13       0.9882    0.9905\n",
      "         2   MaxAbsScaler RandomForest                      0:00:12       0.9646    0.9905\n",
      "         3   MaxAbsScaler SGD                               0:00:12       0.9354    0.9905\n",
      "         4    VotingEnsemble                                0:00:13       0.9891    0.9905\n",
      "         5    StackEnsemble                                 0:00:15       0.9901    0.9905\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'runId': 'AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e',\n",
       " 'target': 'local',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2020-08-09T05:32:18.016977Z',\n",
       " 'endTimeUtc': '2020-08-09T05:34:08.841906Z',\n",
       " 'properties': {'num_iterations': '6',\n",
       "  'training_type': 'TrainFull',\n",
       "  'acquisition_function': 'EI',\n",
       "  'primary_metric': 'AUC_weighted',\n",
       "  'train_split': '0',\n",
       "  'acquisition_parameter': '0',\n",
       "  'num_cross_validation': None,\n",
       "  'target': 'local',\n",
       "  'RawAMLSettingsString': None,\n",
       "  'AMLSettingsJsonString': '{\"path\":null,\"name\":\"Automated ML Experiment\",\"subscription_id\":null,\"resource_group\":null,\"workspace_name\":null,\"region\":null,\"compute_target\":null,\"spark_service\":null,\"azure_service\":null,\"_local_managed_run_id\":null,\"iterations\":6,\"primary_metric\":\"AUC_weighted\",\"task_type\":\"classification\",\"data_script\":null,\"validation_size\":0.0,\"n_cross_validations\":null,\"y_min\":null,\"y_max\":null,\"num_classes\":null,\"featurization\":\"auto\",\"_ignore_package_version_incompatibilities\":false,\"is_timeseries\":false,\"max_cores_per_iteration\":1,\"max_concurrent_iterations\":4,\"iteration_timeout_minutes\":null,\"mem_in_mb\":null,\"enforce_time_on_windows\":false,\"experiment_timeout_minutes\":8640,\"experiment_exit_score\":null,\"whitelist_models\":null,\"blacklist_algos\":[\"TensorFlowLinearClassifier\"],\"supported_models\":[\"LogisticRegression\",\"SVM\",\"ExtremeRandomTrees\",\"KNN\",\"TensorFlowDNN\",\"DecisionTree\",\"TensorFlowLinearClassifier\",\"LinearSVM\",\"XGBoostClassifier\",\"SGD\",\"MultinomialNaiveBayes\",\"AveragedPerceptronClassifier\",\"BernoulliNaiveBayes\",\"GradientBoosting\",\"RandomForest\",\"LightGBM\"],\"auto_blacklist\":true,\"blacklist_samples_reached\":false,\"exclude_nan_labels\":true,\"verbosity\":20,\"_debug_log\":\"automl.log\",\"show_warnings\":false,\"model_explainability\":true,\"service_url\":null,\"sdk_url\":null,\"sdk_packages\":null,\"enable_onnx_compatible_models\":false,\"enable_split_onnx_featurizer_estimator_models\":false,\"vm_type\":null,\"telemetry_verbosity\":20,\"send_telemetry\":true,\"enable_dnn\":false,\"force_text_dnn\":false,\"enable_feature_sweeping\":true,\"enable_early_stopping\":false,\"early_stopping_n_iters\":10,\"metrics\":null,\"enable_ensembling\":true,\"enable_stack_ensembling\":true,\"ensemble_iterations\":6,\"enable_tf\":false,\"enable_subsampling\":false,\"subsample_seed\":null,\"enable_nimbusml\":false,\"enable_streaming\":false,\"force_streaming\":false,\"track_child_runs\":true,\"label_column_name\":\"Diabetic\",\"weight_column_name\":null,\"cv_split_column_names\":null,\"enable_local_managed\":true,\"cost_mode\":1,\"lag_length\":0,\"metric_operation\":\"maximize\",\"preprocess\":true,\"scenario\":\"SDK-Compatible\"}',\n",
       "  'DataPrepJsonString': None,\n",
       "  'EnableSubsampling': 'False',\n",
       "  'runTemplate': 'AutoML',\n",
       "  'azureml.runsource': 'automl',\n",
       "  'display_task_type': 'classification',\n",
       "  'dependencies_versions': '{\"azureml-widgets\": \"1.10.0\", \"azureml-train\": \"1.10.0\", \"azureml-train-restclients-hyperdrive\": \"1.10.0\", \"azureml-train-core\": \"1.10.0\", \"azureml-train-automl\": \"1.10.0\", \"azureml-train-automl-runtime\": \"1.10.0\", \"azureml-train-automl-client\": \"1.10.0\", \"azureml-tensorboard\": \"1.10.0\", \"azureml-telemetry\": \"1.10.0\", \"azureml-sdk\": \"1.10.0\", \"azureml-samples\": \"0+unknown\", \"azureml-pipeline\": \"1.10.0\", \"azureml-pipeline-steps\": \"1.10.0\", \"azureml-pipeline-core\": \"1.10.0\", \"azureml-opendatasets\": \"1.10.0\", \"azureml-monitoring\": \"0.1.0a21\", \"azureml-model-management-sdk\": \"1.0.1b6.post1\", \"azureml-mlflow\": \"1.10.0\", \"azureml-interpret\": \"1.10.0\", \"azureml-explain-model\": \"1.10.0\", \"azureml-defaults\": \"1.10.0\", \"azureml-dataset-runtime\": \"1.10.0\", \"azureml-dataprep\": \"1.9.3\", \"azureml-dataprep-native\": \"14.2.1\", \"azureml-datadrift\": \"1.10.0\", \"azureml-core\": \"1.10.0\", \"azureml-contrib-services\": \"1.10.0\", \"azureml-contrib-server\": \"1.10.0\", \"azureml-contrib-reinforcementlearning\": \"1.10.0\", \"azureml-contrib-pipeline-steps\": \"1.10.0\", \"azureml-contrib-notebook\": \"1.10.0\", \"azureml-contrib-interpret\": \"1.10.0\", \"azureml-contrib-gbdt\": \"1.10.0\", \"azureml-contrib-fairness\": \"1.10.0\", \"azureml-contrib-dataset\": \"1.10.0\", \"azureml-cli-common\": \"1.10.0\", \"azureml-automl-runtime\": \"1.10.0\", \"azureml-automl-core\": \"1.10.0\", \"azureml-accel-models\": \"1.10.0\"}',\n",
       "  'ClientSdkVersion': '1.10.0',\n",
       "  'ClientType': 'SDK',\n",
       "  'environment_cpu_name': 'AzureML-AutoML',\n",
       "  'environment_cpu_version': '29',\n",
       "  'environment_gpu_name': 'AzureML-AutoML-GPU',\n",
       "  'environment_gpu_version': '18',\n",
       "  'root_attribution': 'automl',\n",
       "  'attribution': 'AutoML',\n",
       "  'CancelUri': 'https://eastus.experiments.azureml.net/jasmine/v1.0/subscriptions/35241d74-3b9e-4778-bb92-4bb15e7b0410/resourceGroups/DP-100/providers/Microsoft.MachineLearningServices/workspaces/Lab01A/experiment/diabetes_automl/cancel/AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e',\n",
       "  'Orchestrator': 'AutoML',\n",
       "  '_azureml.ComputeTargetType': 'local',\n",
       "  '_wrapper_run_id': 'diabetes_automl_1596950833_eb6b0cff',\n",
       "  'azureml.git.repository_uri': 'https://github.com/MicrosoftLearning/DP100',\n",
       "  'mlflow.source.git.repoURL': 'https://github.com/MicrosoftLearning/DP100',\n",
       "  'azureml.git.branch': 'master',\n",
       "  'mlflow.source.git.branch': 'master',\n",
       "  'azureml.git.commit': '17ac2a2b5518c3cfae2d3782d6184374c6c20ed1',\n",
       "  'mlflow.source.git.commit': '17ac2a2b5518c3cfae2d3782d6184374c6c20ed1',\n",
       "  'azureml.git.dirty': 'True',\n",
       "  'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": true, \"subsampling\": false, \"dataset_classes\": 2, \"dataset_features\": 23, \"dataset_samples\": 10555, \"single_frequency_class_detected\": false}',\n",
       "  'feature_skus': 'automatedml_sdk_guardrails'},\n",
       " 'inputDatasets': [],\n",
       " 'logFiles': {}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "print('Submitting Auto ML experiment...')\n",
    "automl_experiment = Experiment(ws, 'diabetes_automl')\n",
    "automl_run = automl_experiment.submit(automl_config)\n",
    "RunDetails(automl_run).show()\n",
    "automl_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine the Best Performing Model\n",
    "\n",
    "When the experiment has completed, view the output in the widget, and click the run that produced the best result to see its details.\n",
    "Then click the link to view the experiment details in the Azure portal and view the overall experiment details before viewing the details for the individual run that produced the best result. There's lots of information here about the performance of the model generated.\n",
    "\n",
    "Let's get the best run and the model that it produced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run(Experiment: diabetes_automl,\n",
      "Id: AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0,\n",
      "Type: None,\n",
      "Status: Completed)\n",
      "Pipeline(memory=None,\n",
      "     steps=[('datatransformer', DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n",
      "        feature_sweeping_config=None, feature_sweeping_timeout=None,\n",
      "        featurization_config=None, force_text_dnn=None,\n",
      "        is_cross_validation=None, is_onnx_compatible=None, logger=None,\n",
      "        obser...    silent=True, subsample=1.0, subsample_for_bin=200000,\n",
      "          subsample_freq=0, verbose=-10))])\n",
      "balanced_accuracy 0.9451102807249344\n",
      "AUC_weighted 0.9904812577250306\n",
      "log_loss 0.1186120500342212\n",
      "matthews_correlation 0.8918972733002116\n",
      "f1_score_macro 0.9459413118209691\n",
      "recall_score_weighted 0.9520809898762654\n",
      "AUC_micro 0.9916008305485999\n",
      "recall_score_macro 0.9451102807249344\n",
      "accuracy 0.9520809898762654\n",
      "weighted_accuracy 0.9576485145517856\n",
      "recall_score_micro 0.9520809898762654\n",
      "f1_score_micro 0.9520809898762654\n",
      "precision_score_micro 0.9520809898762654\n",
      "precision_score_macro 0.9467885716011675\n",
      "average_precision_score_micro 0.9918359683181054\n",
      "precision_score_weighted 0.9520038306966419\n",
      "f1_score_weighted 0.952035905457969\n",
      "norm_macro_recall 0.8902205614498688\n",
      "AUC_macro 0.9904812577250306\n",
      "average_precision_score_weighted 0.9910085472336622\n",
      "average_precision_score_macro 0.9888863976623419\n",
      "confusion_matrix aml://artifactId/ExperimentRun/dcid.AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0/confusion_matrix\n",
      "accuracy_table aml://artifactId/ExperimentRun/dcid.AutoML_a6c63c69-227f-42b6-8f56-bf1ec8d3375e_0/accuracy_table\n"
     ]
    }
   ],
   "source": [
    "best_run, fitted_model = automl_run.get_output()\n",
    "print(best_run)\n",
    "print(fitted_model)\n",
    "best_run_metrics = best_run.get_metrics()\n",
    "for metric_name in best_run_metrics:\n",
    "    metric = best_run_metrics[metric_name]\n",
    "    print(metric_name, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automated machine learning includes the option to try preprocessing the data, which is accomplished through the use of [Scikit-Learn transformation pipelines](https://scikit-learn.org/stable/modules/compose.html#combining-estimators) (not to be confused with Azure Machine Learning pipelines!). These produce models that include steps to transform the data before inferencing. You can view the steps in a model like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "datatransformer\n",
      "MaxAbsScaler\n",
      "LightGBMClassifier\n"
     ]
    }
   ],
   "source": [
    "for step in fitted_model.named_steps:\n",
    "    print(step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, having found the best performing model, you can register it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Model\n",
    "\n",
    "# Register model\n",
    "best_run.register_model(model_path='outputs/model.pkl', model_name='diabetes_model_automl',\n",
    "                        tags={'Training context':'Auto ML'},\n",
    "                        properties={'AUC': best_run_metrics['AUC_weighted'], 'Accuracy': best_run_metrics['accuracy']})\n",
    "\n",
    "# List registered models\n",
    "for model in Model.list(ws):\n",
    "    print(model.name, 'version:', model.version)\n",
    "    for tag_name in model.tags:\n",
    "        tag = model.tags[tag_name]\n",
    "        print ('\\t',tag_name, ':', tag)\n",
    "    for prop_name in model.properties:\n",
    "        prop = model.properties[prop_name]\n",
    "        print ('\\t',prop_name, ':', prop)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **More Information**: For more information Automated Machine Learning, see the [Azure ML documentation](https://docs.microsoft.com/azure/machine-learning/how-to-configure-auto-train)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}