ml-insights/sample_notebooks/6_Data_Quality_and_Data_Integrity_Metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "73d25380",
   "metadata": {},
   "source": [
    "# ML Insights : Data Quality & Data Integrity Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a675a436",
   "metadata": {},
   "source": [
    "# Use Case\n",
    "\n",
    "This Notebook shows how to configure following Data Quality & Data Integrity Metrics using ML Insights .\n",
    "\n",
    "    Sum = Compute Sum metrics of input .\n",
    "    TypeMetric = Computes TypeMetric of input.\n",
    "    Mean = Computes Mean Metric of input.\n",
    "    Min = Computes Minimum of input .\n",
    "    StandardDeviation = Computes Standard Deviation Metric of input.\n",
    "    Variance = Computes Variance Metric fof input.\n",
    "    Max = Computes Maximum of input .\n",
    "    Range = Computes Range of input .\n",
    "    Count = Compute total count , nan/missing count & missing count percentage of input.\n",
    "    Skewness = Compute Skewness metrics of input .\n",
    "    Kurtosis = Compute Kurtosis metrics of input .\n",
    "    Quartiles = Compute Quartiles metrics of input .\n",
    "    IQR = Compute IQR metrics of input .\n",
    "    TopKFrequentElements = Compute Top K Frequent Elements Metric of input .\n",
    "    FrequencyDistribution = Compute Frequency Distribution of input .\n",
    "    DistinctCount = Compute Distinct Count of input .\n",
    "    DuplicateCount = Compute the number of items that are  duplicate of another item in the data and percentage of duplicate count out of the total count.\n",
    "    Mode = Computes Mode Metric of input.\n",
    "    IsConstantFeature = Computes IsConstantFeature Univariate Metric of input.\n",
    "    IsQuasiConstantFeature =  Computes IsQuasiConstantFeature Univariate Metric of input.\n",
    "    ProbabilityDistribution = Computes PDF Metric for the dataset for configured number of bins or default number of bins.\n",
    "\n",
    "## Note\n",
    "    \n",
    "- All Data Quality & Data Integrity Metrics requires to define VariableType and DataType of feature in feature schema. This includes:\n",
    "    - variable_type=VariableType.CONTINUOUS for Numerical features\n",
    "    - variable_type=VariableType.NOMINAL for Categorical features\n",
    "    - Supported datatype : data_type=DataType.FLOAT , data_type=DataType.INTEGER ,data_type=DataType.STRING , data_type=DataType.TEXT ,data_type=DataType.BOOLEAN\n",
    "- All Conflict metrics can be view using following Profile API:\n",
    "    - to_json\n",
    "    - to_pandas\n",
    "    \n",
    "### About Dataset \n",
    "\n",
    "The Iris flower data set or Fisher's Iris data set is a multivariate data set . The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimeters.\n",
    "\n",
    "Dataset source : https://archive.ics.uci.edu/dataset/53/iris"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae05b17d",
   "metadata": {},
   "source": [
    "# Install ML Observability Insights Library SDK\n",
    "\n",
    "- Prerequisites\n",
    "    - Linux/Mac (Intel CPU)\n",
    "    - Python 3.8 and 3.9 only\n",
    "\n",
    "\n",
    "- Installation\n",
    "    - ML Insights is made available as a Python package (via Artifactory) which can be installed using pip install as shown below. Depending on the execution engine on which to do the run, one can use scoped package. For eg: if we want to run on dask, use oracle-ml-insights[dask], for spark use oracle-ml-insights[spark], for native use oracle-ml-insights. One can install all the dependencies as use oracle-ml-insights[all]\n",
    "\n",
    "      !pip install oracle-ml-insights\n",
    "\n",
    "Refer : [Installation and Setup](https://docs.oracle.com/en-us/iaas/tools/ml-insights-docs/latest/ml-insights-documentation/html/user_guide/tutorials/install.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f0b943e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!python3 -m pip install oracle-ml-insights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df0e3aab",
   "metadata": {},
   "source": [
    "# 1 ML Insights Imports "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a6504a49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os\n",
    "from typing import Any\n",
    "import pyarrow as pa\n",
    "import pandas as pd\n",
    "import json\n",
    "\n",
    "\n",
    "# Import metrics\n",
    "from mlm_insights.core.features.feature import FeatureMetadata\n",
    "from mlm_insights.core.metrics.count import Count\n",
    "from mlm_insights.core.metrics.distinct_count import DistinctCount\n",
    "from mlm_insights.core.metrics.duplicate_count import DuplicateCount\n",
    "from mlm_insights.core.metrics.iqr import IQR\n",
    "from mlm_insights.core.metrics.is_constant_feature import IsConstantFeature\n",
    "from mlm_insights.core.metrics.is_quasi_constant_feature import IsQuasiConstantFeature\n",
    "from mlm_insights.core.metrics.kurtosis import Kurtosis\n",
    "from mlm_insights.core.metrics.max import Max\n",
    "from mlm_insights.core.metrics.mean import Mean\n",
    "from mlm_insights.core.metrics.min import Min\n",
    "from mlm_insights.core.metrics.mode import Mode\n",
    "from mlm_insights.core.metrics.range import Range\n",
    "from mlm_insights.core.metrics.rows_count import RowCount\n",
    "from mlm_insights.core.metrics.skewness import Skewness\n",
    "from mlm_insights.core.metrics.standard_deviation import StandardDeviation\n",
    "from mlm_insights.core.metrics.top_k_frequent_elements import TopKFrequentElements\n",
    "from mlm_insights.core.metrics.type_metric import TypeMetric\n",
    "from mlm_insights.core.metrics.variance import Variance\n",
    "from mlm_insights.core.metrics.quartiles import Quartiles\n",
    "from mlm_insights.core.metrics.sum import Sum\n",
    "\n",
    "from mlm_insights.builder.builder_component import MetricDetail, EngineDetail\n",
    "from mlm_insights.constants.types import FeatureType, DataType, VariableType\n",
    "from mlm_insights.core.metrics.metric_metadata import MetricMetadata\n",
    "from mlm_insights.core.data_sources import LocalDatePrefixDataSource\n",
    "from mlm_insights.mlm_native.readers import CSVNativeDataReader\n",
    "from mlm_insights.core.post_processors.local_writer_post_processor import LocalWriterPostProcessor\n",
    "from mlm_insights.builder.insights_builder import InsightsBuilder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d58f9581",
   "metadata": {},
   "source": [
    "# 2 Configure Feature schema\n",
    "\n",
    "Feature Schema defines the structure and metadata of the input data, which includes data type, column type, column mapping . The framework, uses this information as the ground truth and any deviation in the actual data is taken as an anomaly and the framework usually will ignore such all such anomaly in data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6e83250b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_input_schema():\n",
    "    return {\n",
    "        \"sepal length (cm)\": FeatureType(data_type=DataType.FLOAT, variable_type=VariableType.CONTINUOUS),\n",
    "        \"sepal width (cm)\": FeatureType(data_type=DataType.FLOAT, variable_type=VariableType.CONTINUOUS),\n",
    "        \"petal length (cm)\": FeatureType(data_type=DataType.FLOAT, variable_type=VariableType.CONTINUOUS),\n",
    "        \"petal width (cm)\": FeatureType(data_type=DataType.FLOAT, variable_type=VariableType.CONTINUOUS)\n",
    "    }\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e9a74a",
   "metadata": {},
   "source": [
    "# 3 Configure Metrics\n",
    "\n",
    "Metrics are the core construct for the framework. This component is responsible for calculating all statistical metrics and algorithms. Metric components work based on the type of features (eg. input feature, output feature etc.) available, their data type (eg. int, float, string etc.) as well as additional context (e.g. if any previous computation is available to compare against). ML Insights provides commonly used metrics out of the box for different ML observability use cases.\n",
    "\n",
    "Refer : [Metrics Component Documentation](https://docs.oracle.com/en-us/iaas/tools/ml-insights-docs/latest/ml-insights-documentation/html/user_guide/getting_started/metrics_component.html)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "60ee9e80",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics():\n",
    "    metrics = [\n",
    "               MetricMetadata(klass=Sum),\n",
    "               MetricMetadata(klass=Quartiles),\n",
    "               MetricMetadata(klass=Max),\n",
    "               MetricMetadata(klass=Min),\n",
    "               MetricMetadata(klass=Count),\n",
    "               MetricMetadata(klass=Mean),\n",
    "               MetricMetadata(klass=Skewness),\n",
    "               MetricMetadata(klass=TypeMetric),\n",
    "               MetricMetadata(klass=StandardDeviation),\n",
    "               MetricMetadata(klass=Variance),\n",
    "               MetricMetadata(klass=IsConstantFeature),\n",
    "               MetricMetadata(klass=IsQuasiConstantFeature),\n",
    "               MetricMetadata(klass=Kurtosis),\n",
    "               MetricMetadata(klass=DistinctCount),\n",
    "               MetricMetadata(klass=DuplicateCount),\n",
    "               MetricMetadata(klass=IQR),\n",
    "               MetricMetadata(klass=Mode),\n",
    "               MetricMetadata(klass=Range)\n",
    "              ]\n",
    "    uni_variate_metrics = {\n",
    "        \"sepal length (cm)\": metrics,\n",
    "        \"sepal width (cm)\": metrics\n",
    "    }\n",
    "    metric_details = MetricDetail(univariate_metric=uni_variate_metrics,\n",
    "                                  dataset_metrics=[])\n",
    "    return metric_details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71cf8492",
   "metadata": {},
   "source": [
    "# 4 Configure Data Reader\n",
    "\n",
    "Data Reader allows for ingestion of raw data into the framework. This component is primarily responsible for understanding different formats of data (e.g. jsonl, csv) etc. and how to properly read them. At its essence, the primary responsibility of this component is that given a set of valid file locations which represents file of a specific type, reader can properly decode the content and load them in memory.\n",
    "\n",
    "Additionally, Data Source component is an optional subcomponent, which is usually used along side the Reader. The primary responsibility of the data source component is to embed logic on filtering and partitioning of files to be read by the framework.\n",
    "\n",
    "Refer : [Data Reader Documentation](https://docs.oracle.com/en-us/iaas/tools/ml-insights-docs/latest/ml-insights-documentation/html/user_guide/getting_started/data_reader_component.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d65a89e3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# def get_data_frame():\n",
    "#     iris_dataset = load_iris(as_frame=True)\n",
    "#     iris_data_frame = pd.DataFrame(data=iris_dataset.data, columns=iris_dataset.feature_names)\n",
    "#     return iris_data_frame\n",
    "\n",
    "# def get_reader():\n",
    "#     data = {\n",
    "#         \"file_type\": \"csv\",\n",
    "#         \"date_range\": {\"start\": \"2023-06-24\", \"end\": \"2023-06-27\"}\n",
    "#     }\n",
    "#     base_location =\"input_data/iris_dataset\"\n",
    "#     ds = LocalDatePrefixDataSource(base_location, **data)\n",
    "#     print(ds.get_data_location())\n",
    "#     csv_reader = CSVNativeDataReader(data_source=ds)\n",
    "#     return csv_reader\n",
    "\n",
    "def get_data_reader(start_date, end_date):\n",
    "    \n",
    "    # Define Data Format\n",
    "    data = {\n",
    "        \"file_type\": \"csv\",\n",
    "        \"date_range\": {\"start\": start_date, \"end\": end_date}\n",
    "    }\n",
    "    \n",
    "    # Define Data Location\n",
    "    base_location =\"input_data/iris_dataset\"\n",
    "    \n",
    "    # Create new Dataset\n",
    "    ds = LocalDatePrefixDataSource(base_location, **data)\n",
    "    \n",
    "    # Load Dataset\n",
    "    csv_reader = CSVNativeDataReader(data_source=ds)\n",
    "    \n",
    "    \n",
    "    return csv_reader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8118995",
   "metadata": {},
   "source": [
    "# 5 Compute the Profile \n",
    "\n",
    "Create the builder object which provides core set of api, using which user can set the behavior of their monitoring. By selecting what components and variants to run all aspects of the monitoring task can be customised and configured. \n",
    "\n",
    "The run() method is responsible to run the internal workflow. It also handles the life cycle of each component passed, which includes creation (if required), invoking interface functions, destroying etc . Additionally, runner also handles some more advanced operations like thread pooling, compute engine abstraction etc.\n",
    "\n",
    "Refer : [Builder Object Documentation](https://docs.oracle.com/en-us/iaas/tools/ml-insights-docs/latest/ml-insights-documentation/html/user_guide/getting_started/builder_object.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8b1a870c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create the ML Monitoring Metrics\n",
    "def run_evaluation(start_date, end_date, output_location, output_file):    \n",
    "    \n",
    "    # Set up the insights builder by passing: input schema, metric, reader and engine details\n",
    "    runner = InsightsBuilder(). \\\n",
    "        with_input_schema(get_input_schema()). \\\n",
    "        with_metrics(metrics=get_metrics()). \\\n",
    "        with_reader(reader=get_data_reader(start_date, end_date)). \\\n",
    "        with_post_processors(post_processors=[LocalWriterPostProcessor(file_location=output_location, file_name=output_file)]). \\\n",
    "        build()\n",
    "\n",
    "    # Run the Evaluation of Metrics\n",
    "    run_result = runner.run()\n",
    "    \n",
    "    return run_result.profile\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e242b327-98e6-43a8-bf9a-a670d8200073",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define Baseline Dates\n",
    "base_start_date = '2023-06-26'\n",
    "base_end_date = '2023-06-29'\n",
    "\n",
    "# Execute Profile - Pass in Data Start, Data End, Output Location, Output File\n",
    "profile = run_evaluation(base_start_date, base_end_date, 'output_data/profiles', 'classification_metrics_profile.bin')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec3b0666",
   "metadata": {},
   "source": [
    "## 6 Profile Result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb19b190",
   "metadata": {},
   "source": [
    "## 6.1 Visualize the Profile in tabular format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d202ad4e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Skewness</th>\n",
       "      <th>StandardDeviation</th>\n",
       "      <th>Min</th>\n",
       "      <th>IsConstantFeature</th>\n",
       "      <th>IQR</th>\n",
       "      <th>Mode</th>\n",
       "      <th>Range</th>\n",
       "      <th>TypeMetric.string_type_count</th>\n",
       "      <th>TypeMetric.integral_type_count</th>\n",
       "      <th>TypeMetric.fractional_type_count</th>\n",
       "      <th>...</th>\n",
       "      <th>Count.missing_count_percentage</th>\n",
       "      <th>Max</th>\n",
       "      <th>DistinctCount</th>\n",
       "      <th>Sum</th>\n",
       "      <th>Kurtosis</th>\n",
       "      <th>Quartiles.q1</th>\n",
       "      <th>Quartiles.q2</th>\n",
       "      <th>Quartiles.q3</th>\n",
       "      <th>Mean</th>\n",
       "      <th>IsQuasiConstantFeature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <td>0.311753</td>\n",
       "      <td>0.825301</td>\n",
       "      <td>4.3</td>\n",
       "      <td>False</td>\n",
       "      <td>1.3</td>\n",
       "      <td>[5.0]</td>\n",
       "      <td>3.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>600</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.9</td>\n",
       "      <td>35</td>\n",
       "      <td>3506.0</td>\n",
       "      <td>-0.573568</td>\n",
       "      <td>5.1</td>\n",
       "      <td>5.8</td>\n",
       "      <td>6.4</td>\n",
       "      <td>5.843333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <td>0.315767</td>\n",
       "      <td>0.434411</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "      <td>0.5</td>\n",
       "      <td>[3.0]</td>\n",
       "      <td>2.4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>600</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.4</td>\n",
       "      <td>23</td>\n",
       "      <td>1834.4</td>\n",
       "      <td>0.180976</td>\n",
       "      <td>2.8</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.3</td>\n",
       "      <td>3.057333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Skewness  StandardDeviation  Min  IsConstantFeature  IQR  \\\n",
       "sepal length (cm)  0.311753           0.825301  4.3              False  1.3   \n",
       "sepal width (cm)   0.315767           0.434411  2.0              False  0.5   \n",
       "\n",
       "                    Mode  Range  TypeMetric.string_type_count  \\\n",
       "sepal length (cm)  [5.0]    3.6                             0   \n",
       "sepal width (cm)   [3.0]    2.4                             0   \n",
       "\n",
       "                   TypeMetric.integral_type_count  \\\n",
       "sepal length (cm)                               0   \n",
       "sepal width (cm)                                0   \n",
       "\n",
       "                   TypeMetric.fractional_type_count  ...  \\\n",
       "sepal length (cm)                               600  ...   \n",
       "sepal width (cm)                                600  ...   \n",
       "\n",
       "                   Count.missing_count_percentage  Max  DistinctCount     Sum  \\\n",
       "sepal length (cm)                             0.0  7.9             35  3506.0   \n",
       "sepal width (cm)                              0.0  4.4             23  1834.4   \n",
       "\n",
       "                   Kurtosis  Quartiles.q1  Quartiles.q2  Quartiles.q3  \\\n",
       "sepal length (cm) -0.573568           5.1           5.8           6.4   \n",
       "sepal width (cm)   0.180976           2.8           3.0           3.3   \n",
       "\n",
       "                       Mean  IsQuasiConstantFeature  \n",
       "sepal length (cm)  5.843333                   False  \n",
       "sepal width (cm)   3.057333                   False  \n",
       "\n",
       "[2 rows x 26 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1899d8",
   "metadata": {},
   "source": [
    "## 6.2 Visualize the Profile in JSON format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "265f1c3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"dataset_metrics\": {},\n",
      "    \"feature_metrics\": {\n",
      "        \"sepal length (cm)\": {\n",
      "            \"Count\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    600.0,\n",
      "                    0.0,\n",
      "                    0.0\n",
      "                ],\n",
      "                \"metric_description\": \"Feature metric that returns total count, missing count and missing count percentage\",\n",
      "                \"metric_name\": \"Count\",\n",
      "                \"variable_count\": 3,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"total_count\",\n",
      "                    \"missing_count\",\n",
      "                    \"missing_count_percentage\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"DistinctCount\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    35\n",
      "                ],\n",
      "                \"metric_description\": \"Approximate Distinct Count\",\n",
      "                \"metric_name\": \"DistinctCount\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"distinct_count\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"DuplicateCount\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    565,\n",
      "                    94.16666666666667\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute duplicate count and duplicate count percentage\",\n",
      "                \"metric_name\": \"DuplicateCount\",\n",
      "                \"variable_count\": 2,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"count\",\n",
      "                    \"percentage\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"IQR\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    1.3000000000000007\n",
      "                ],\n",
      "                \"metric_description\": \"Compute the interquartile range of the data. Interquartile range (IQR) is the difference between the 75th and 25th percentile of the data. It is a measure of the dispersion similar to standard deviation or variance, but is much more robust against outliers\",\n",
      "                \"metric_name\": \"IQR\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"iqr\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"IsConstantFeature\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    false\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute if all values are same\",\n",
      "                \"metric_name\": \"IsConstantFeature\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"BOOLEAN\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"is_constant\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"BINARY\"\n",
      "                ]\n",
      "            },\n",
      "            \"IsQuasiConstantFeature\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    false\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute if all values are almost same\",\n",
      "                \"metric_name\": \"IsQuasiConstantFeature\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"BOOLEAN\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"is_quasi_constant\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"BINARY\"\n",
      "                ]\n",
      "            },\n",
      "            \"Kurtosis\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    -0.5735679489249765\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Kurtosis\",\n",
      "                \"metric_name\": \"Kurtosis\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"kurtosis\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Max\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    7.9\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute maximum value\",\n",
      "                \"metric_name\": \"Max\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"max\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Mean\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    5.843333333333334\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute mean\",\n",
      "                \"metric_name\": \"Mean\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"mean\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Min\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    4.3\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute minimum value\",\n",
      "                \"metric_name\": \"Min\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"min\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Mode\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    [\n",
      "                        \"5.0\"\n",
      "                    ]\n",
      "                ],\n",
      "                \"metric_description\": \"Mode\",\n",
      "                \"metric_name\": \"Mode\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    1\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"STRING\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"mode\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"NOMINAL\"\n",
      "                ]\n",
      "            },\n",
      "            \"Quartiles\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    5.1,\n",
      "                    5.8,\n",
      "                    6.4\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Quartiles (Q1, Q2, Q3)\",\n",
      "                \"metric_name\": \"Quartiles\",\n",
      "                \"variable_count\": 3,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\",\n",
      "                    \"FLOAT\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"q1\",\n",
      "                    \"q2\",\n",
      "                    \"q3\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Range\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    3.6000000000000005\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute range value\",\n",
      "                \"metric_name\": \"Range\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"range\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Skewness\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.311753058502296\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Skewness\",\n",
      "                \"metric_name\": \"Skewness\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"skewness\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"StandardDeviation\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.8253012917851409\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Standard Deviation\",\n",
      "                \"metric_name\": \"StandardDeviation\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"standard_deviation\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Sum\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    3506.0\n",
      "                ],\n",
      "                \"metric_description\": \"\",\n",
      "                \"metric_name\": \"Sum\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"sum\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"TypeMetric\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    600,\n",
      "                    0\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute count of data types for feature values\",\n",
      "                \"metric_name\": \"TypeMetric\",\n",
      "                \"variable_count\": 4,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"string_type_count\",\n",
      "                    \"integral_type_count\",\n",
      "                    \"fractional_type_count\",\n",
      "                    \"boolean_type_count\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\"\n",
      "                ]\n",
      "            },\n",
      "            \"Variance\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.6811222222222223\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Variance\",\n",
      "                \"metric_name\": \"Variance\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"variance\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            }\n",
      "        },\n",
      "        \"sepal width (cm)\": {\n",
      "            \"Count\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    600.0,\n",
      "                    0.0,\n",
      "                    0.0\n",
      "                ],\n",
      "                \"metric_description\": \"Feature metric that returns total count, missing count and missing count percentage\",\n",
      "                \"metric_name\": \"Count\",\n",
      "                \"variable_count\": 3,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"total_count\",\n",
      "                    \"missing_count\",\n",
      "                    \"missing_count_percentage\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"DistinctCount\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    23\n",
      "                ],\n",
      "                \"metric_description\": \"Approximate Distinct Count\",\n",
      "                \"metric_name\": \"DistinctCount\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"distinct_count\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"DuplicateCount\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    577,\n",
      "                    96.16666666666667\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute duplicate count and duplicate count percentage\",\n",
      "                \"metric_name\": \"DuplicateCount\",\n",
      "                \"variable_count\": 2,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"count\",\n",
      "                    \"percentage\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"IQR\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.5\n",
      "                ],\n",
      "                \"metric_description\": \"Compute the interquartile range of the data. Interquartile range (IQR) is the difference between the 75th and 25th percentile of the data. It is a measure of the dispersion similar to standard deviation or variance, but is much more robust against outliers\",\n",
      "                \"metric_name\": \"IQR\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"iqr\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"IsConstantFeature\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    false\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute if all values are same\",\n",
      "                \"metric_name\": \"IsConstantFeature\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"BOOLEAN\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"is_constant\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"BINARY\"\n",
      "                ]\n",
      "            },\n",
      "            \"IsQuasiConstantFeature\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    false\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute if all values are almost same\",\n",
      "                \"metric_name\": \"IsQuasiConstantFeature\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"BOOLEAN\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"is_quasi_constant\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"BINARY\"\n",
      "                ]\n",
      "            },\n",
      "            \"Kurtosis\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.18097631752246723\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Kurtosis\",\n",
      "                \"metric_name\": \"Kurtosis\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"kurtosis\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Max\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    4.4\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute maximum value\",\n",
      "                \"metric_name\": \"Max\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"max\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Mean\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    3.0573333333333337\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute mean\",\n",
      "                \"metric_name\": \"Mean\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"mean\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Min\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    2.0\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute minimum value\",\n",
      "                \"metric_name\": \"Min\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"min\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Mode\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    [\n",
      "                        \"3.0\"\n",
      "                    ]\n",
      "                ],\n",
      "                \"metric_description\": \"Mode\",\n",
      "                \"metric_name\": \"Mode\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    1\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"STRING\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"mode\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"NOMINAL\"\n",
      "                ]\n",
      "            },\n",
      "            \"Quartiles\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    2.8,\n",
      "                    3.0,\n",
      "                    3.3\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Quartiles (Q1, Q2, Q3)\",\n",
      "                \"metric_name\": \"Quartiles\",\n",
      "                \"variable_count\": 3,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\",\n",
      "                    \"FLOAT\",\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"q1\",\n",
      "                    \"q2\",\n",
      "                    \"q3\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\",\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Range\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    2.4000000000000004\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute range value\",\n",
      "                \"metric_name\": \"Range\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"range\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Skewness\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.31576710633893473\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Skewness\",\n",
      "                \"metric_name\": \"Skewness\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"skewness\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"StandardDeviation\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.4344109677354946\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Standard Deviation\",\n",
      "                \"metric_name\": \"StandardDeviation\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"standard_deviation\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"Sum\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    1834.4\n",
      "                ],\n",
      "                \"metric_description\": \"\",\n",
      "                \"metric_name\": \"Sum\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"sum\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            },\n",
      "            \"TypeMetric\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    600,\n",
      "                    0\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute count of data types for feature values\",\n",
      "                \"metric_name\": \"TypeMetric\",\n",
      "                \"variable_count\": 4,\n",
      "                \"variable_dimensions\": [\n",
      "                    0,\n",
      "                    0,\n",
      "                    0,\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\",\n",
      "                    \"INTEGER\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"string_type_count\",\n",
      "                    \"integral_type_count\",\n",
      "                    \"fractional_type_count\",\n",
      "                    \"boolean_type_count\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\",\n",
      "                    \"DISCRETE\"\n",
      "                ]\n",
      "            },\n",
      "            \"Variance\": {\n",
      "                \"metadata\": {},\n",
      "                \"metric_data\": [\n",
      "                    0.1887128888888889\n",
      "                ],\n",
      "                \"metric_description\": \"Feature Metric to compute Variance\",\n",
      "                \"metric_name\": \"Variance\",\n",
      "                \"variable_count\": 1,\n",
      "                \"variable_dimensions\": [\n",
      "                    0\n",
      "                ],\n",
      "                \"variable_dtypes\": [\n",
      "                    \"FLOAT\"\n",
      "                ],\n",
      "                \"variable_names\": [\n",
      "                    \"variance\"\n",
      "                ],\n",
      "                \"variable_types\": [\n",
      "                    \"CONTINUOUS\"\n",
      "                ]\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "profile_json = profile.to_json()\n",
    "dataset_metrics = profile_json\n",
    "print(json.dumps(dataset_metrics,sort_keys=True, indent=4))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9c4985dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal length (cm).Skewness.metric_name</th>\n",
       "      <td>Skewness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal length (cm).Skewness.metric_description</th>\n",
       "      <td>Feature Metric to compute Skewness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal length (cm).Skewness.variable_count</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal length (cm).Skewness.variable_names</th>\n",
       "      <td>[skewness]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal length (cm).Skewness.variable_types</th>\n",
       "      <td>[CONTINUOUS]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal width (cm).IsQuasiConstantFeature.variable_names</th>\n",
       "      <td>[is_quasi_constant]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal width (cm).IsQuasiConstantFeature.variable_types</th>\n",
       "      <td>[BINARY]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal width (cm).IsQuasiConstantFeature.variable_dtypes</th>\n",
       "      <td>[BOOLEAN]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal width (cm).IsQuasiConstantFeature.variable_dimensions</th>\n",
       "      <td>[0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>feature_metrics.sepal width (cm).IsQuasiConstantFeature.metric_data</th>\n",
       "      <td>[False]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>288 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                     0\n",
       "feature_metrics.sepal length (cm).Skewness.metr...                            Skewness\n",
       "feature_metrics.sepal length (cm).Skewness.metr...  Feature Metric to compute Skewness\n",
       "feature_metrics.sepal length (cm).Skewness.vari...                                   1\n",
       "feature_metrics.sepal length (cm).Skewness.vari...                          [skewness]\n",
       "feature_metrics.sepal length (cm).Skewness.vari...                        [CONTINUOUS]\n",
       "...                                                                                ...\n",
       "feature_metrics.sepal width (cm).IsQuasiConstan...                 [is_quasi_constant]\n",
       "feature_metrics.sepal width (cm).IsQuasiConstan...                            [BINARY]\n",
       "feature_metrics.sepal width (cm).IsQuasiConstan...                           [BOOLEAN]\n",
       "feature_metrics.sepal width (cm).IsQuasiConstan...                                 [0]\n",
       "feature_metrics.sepal width (cm).IsQuasiConstan...                             [False]\n",
       "\n",
       "[288 rows x 1 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.json_normalize(dataset_metrics).T.dropna()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}