docs/03-tasks-and-benchmarks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook covers the following topics:\n",
    "1. Defining a time series forecasting `Task`\n",
    "2. Backtesting / evaluation using multiple cutoff dates\n",
    "3. Evaluation on a `Benchmark` consisting of multiple tasks\n",
    "4. Aggregating benchmark results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import datasets\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "import fev\n",
    "\n",
    "warnings.simplefilter(\"ignore\")\n",
    "datasets.disable_progress_bars()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation on a single Task\n",
    "A `fev.Task` object contains all information that uniquely identifies a time series forecasting task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data sources\n",
    "Dataset stored on Hugging Face Hub: https://huggingface.co/datasets/autogluon/chronos_datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets\",\n",
    "    dataset_config=\"monash_cif_2016\",\n",
    "    horizon=12,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset stored on S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset consisting of a single parquet / arrow file\n",
    "task = fev.Task(\n",
    "    dataset_path=\"s3://autogluon/datasets/timeseries/m1_monthly/data.parquet\",\n",
    "    horizon=12,\n",
    ")\n",
    "# Dataset consisting of multiple parquet / arrow files\n",
    "task = fev.Task(\n",
    "    dataset_path=\"s3://autogluon/datasets/timeseries/m1_monthly/*.parquet\",\n",
    "    horizon=12,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset stored locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download dataset from HF Hub and save it locally\n",
    "ds = datasets.load_dataset(\"autogluon/chronos_datasets\", name=\"m4_hourly\", split=\"train\")\n",
    "local_path = \"/tmp/m4_hourly/data.parquet\"\n",
    "ds.to_parquet(local_path)\n",
    "\n",
    "task = fev.Task(\n",
    "    dataset_path=local_path,\n",
    "    horizon=48,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Covariates\n",
    "By default, all columns of type `Sequence` are interpreted as known covariates, and all remaining columns are interpreted as static covariates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset({\n",
      "    features: ['id', 'timestamp', 'HUFL', 'HULL', 'MUFL', 'MULL', 'LUFL', 'LULL', 'OT'],\n",
      "    num_rows: 2\n",
      "})\n",
      "Dataset({\n",
      "    features: ['id', 'timestamp', 'HUFL', 'HULL', 'MUFL', 'MULL', 'LUFL', 'LULL'],\n",
      "    num_rows: 2\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "    dataset_config=\"ETTh\",\n",
    "    horizon=24,\n",
    "    target_column=\"OT\",\n",
    ")\n",
    "past_data, future_data = task.get_input_data()\n",
    "print(past_data)\n",
    "print(future_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can configure how the covariates are used as part of the task definition.\n",
    "\n",
    "For example, here we say that \n",
    "- columns `HUFL` and `HULL` are known only in the past\n",
    "- columns `MUFL` and `MULL` are excluded from the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset({\n",
      "    features: ['id', 'timestamp', 'HUFL', 'HULL', 'LUFL', 'LULL', 'OT'],\n",
      "    num_rows: 2\n",
      "})\n",
      "Dataset({\n",
      "    features: ['id', 'timestamp', 'LUFL', 'LULL'],\n",
      "    num_rows: 2\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "    dataset_config=\"ETTh\",\n",
    "    horizon=24,\n",
    "    target_column=\"OT\",\n",
    "    past_dynamic_columns=[\"HUFL\", \"HULL\"],\n",
    "    excluded_columns=[\"MUFL\", \"MULL\"],\n",
    ")\n",
    "\n",
    "past_data, future_data = task.get_input_data()\n",
    "print(past_data)\n",
    "print(future_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions format\n",
    "Each task expects predictions to follow a certain format that is specified by `task.predictions_schema`.\n",
    "\n",
    "For point forecasting tasks (i.e., if `quantile_levels=None`), predictions must contain a single array of length `horizon` for each time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets\",\n",
    "    dataset_config=\"m4_hourly\",\n",
    "    horizon=48,\n",
    "    eval_metric=\"MASE\",\n",
    "    seasonality=24,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=48, id=None)}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.predictions_schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For probabilistic forecasting tasks (i.e., if `quantile_levels` is provided), predictions must additionally contain a prediction for each quantile level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets\",\n",
    "    dataset_config=\"m4_hourly\",\n",
    "    horizon=48,\n",
    "    seasonality=24,\n",
    "    quantile_levels=[0.1, 0.5, 0.9],\n",
    "    eval_metric=\"WQL\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictions': Sequence(feature=Value(dtype='float64', id=None), length=48, id=None),\n",
       " '0.1': Sequence(feature=Value(dtype='float64', id=None), length=48, id=None),\n",
       " '0.5': Sequence(feature=Value(dtype='float64', id=None), length=48, id=None),\n",
       " '0.9': Sequence(feature=Value(dtype='float64', id=None), length=48, id=None)}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task.predictions_schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backtesting & custom cutoffs\n",
    "By default, the train/test split is generated as follows:\n",
    "- test set contains the last `horizon` time steps of each time series\n",
    "- train set contains everything up to the last `horizon` time steps of each time series\n",
    "\n",
    "We can create the train/test splits at custom points in the time series using the `cutoff` argument.\n",
    "\n",
    "The default behavior corresponds to setting `cutoff = -horizon`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "    dataset_config=\"ETTh\",\n",
    "    horizon=24,\n",
    "    target_column=\"OT\",\n",
    "    cutoff=-24,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can set cutoff to a positive or negative integer. In this case, the training data will correspond to `y[:cutoff]` and the test set will be `y[cutoff : cutoff + horizon]`.\n",
    "\n",
    "We can also set `cutoff` to a datetime-like string. In this case, `cutoff` will be the last timestamp in the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last train timestamp: 2017-01-01T00:00:00.000000000\n",
      "First test timestamp: 2017-01-01T01:00:00.000000000\n"
     ]
    }
   ],
   "source": [
    "task = fev.Task(\n",
    "    dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "    dataset_config=\"ETTh\",\n",
    "    horizon=24,\n",
    "    target_column=\"OT\",\n",
    "    cutoff=\"2017-01-01\",\n",
    ")\n",
    "past_data, future_data = task.get_input_data()\n",
    "print(f\"Last train timestamp: {past_data[0]['timestamp'][-1]}\")\n",
    "print(f\"First test timestamp: {future_data[0]['timestamp'][0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create tasks corresponding to multiple backtests by providing different values for the `cutoff`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tasks = [\n",
    "    fev.Task(\n",
    "        dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "        dataset_config=\"ETTh\",\n",
    "        horizon=24,\n",
    "        target_column=\"OT\",\n",
    "        cutoff=\"2017-01-01\",\n",
    "    ),\n",
    "    fev.Task(\n",
    "        dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "        dataset_config=\"ETTh\",\n",
    "        horizon=24,\n",
    "        target_column=\"OT\",\n",
    "        cutoff=\"2017-02-07\",\n",
    "    ),\n",
    "    fev.Task(\n",
    "        dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "        dataset_config=\"ETTh\",\n",
    "        horizon=24,\n",
    "        target_column=\"OT\",\n",
    "        cutoff=\"2017-06-03\",\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fev.TaskGenerator` class provides a more concise way to create multiple related configurations, e.g., for backtesting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task 0\n",
      "\tLast train timestamp: 2017-01-01T00:00:00.000000000\n",
      "\tFirst test timestamp: 2017-01-01T01:00:00.000000000\n",
      "Task 1\n",
      "\tLast train timestamp: 2017-02-07T00:00:00.000000000\n",
      "\tFirst test timestamp: 2017-02-07T01:00:00.000000000\n",
      "Task 2\n",
      "\tLast train timestamp: 2017-06-03T00:00:00.000000000\n",
      "\tFirst test timestamp: 2017-06-03T01:00:00.000000000\n"
     ]
    }
   ],
   "source": [
    "task_generator = fev.TaskGenerator(\n",
    "    dataset_path=\"autogluon/chronos_datasets_extra\",\n",
    "    dataset_config=\"ETTh\",\n",
    "    horizon=24,\n",
    "    target_column=\"OT\",\n",
    "    variants=[\n",
    "        {\"cutoff\": \"2017-01-01\"},\n",
    "        {\"cutoff\": \"2017-02-07\"},\n",
    "        {\"cutoff\": \"2017-06-03\"},\n",
    "    ]\n",
    ")\n",
    "tasks = task_generator.generate_tasks()\n",
    "for i, task in enumerate(tasks):\n",
    "    print(f\"Task {i}\")\n",
    "    past_data, future_data = task.get_input_data()\n",
    "    print(f\"\\tLast train timestamp: {past_data[0]['timestamp'][-1]}\")\n",
    "    print(f\"\\tFirst test timestamp: {future_data[0]['timestamp'][0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't specify `variants`, then `TaskGenerator.generate_tasks()` will produce a single `Task`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Task(dataset_path='my_dataset', dataset_config='my_config', horizon=12, cutoff=-12, lead_time=1, min_ts_length=13, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[])]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_generator = fev.TaskGenerator(\n",
    "    dataset_path=\"my_dataset\",\n",
    "    dataset_config=\"my_config\",\n",
    "    horizon=12,\n",
    ")\n",
    "task_generator.generate_tasks()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we do specify `variants`, then `TaskGenerator.generate_tasks()` will produce a single `Task` for each variant in `variants`.\n",
    "\n",
    "In each of the variants, the dict provided in variants will override the default parameters for the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Task(dataset_path='my_dataset', dataset_config='my_config', horizon=12, cutoff=-12, lead_time=1, min_ts_length=13, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[]),\n",
       " Task(dataset_path='my_dataset', dataset_config='my_config', horizon=24, cutoff=-24, lead_time=1, min_ts_length=25, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[])]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_generator = fev.TaskGenerator(\n",
    "    dataset_path=\"my_dataset\",\n",
    "    dataset_config=\"my_config\",\n",
    "    variants=[\n",
    "        {\"horizon\": 12},\n",
    "        {\"horizon\": 24},\n",
    "    ]\n",
    ")\n",
    "task_generator.generate_tasks()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation on a Benchmark consisting of multiple tasks\n",
    "A `fev.Benchmark` object is essentially a collection of `Task`s.\n",
    "\n",
    "We can create a benchmark from a list of dictionaries. Each dictionary is interpreted as a `fev.TaskGenerator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "task_generators = [\n",
    "    {\n",
    "        \"dataset_path\": \"autogluon/chronos_datasets\",\n",
    "        \"dataset_config\": \"monash_m3_monthly\",\n",
    "        \"horizon\": 18,\n",
    "        \"seasonality\": 12,\n",
    "        \"eval_metric\": \"MASE\",\n",
    "    },\n",
    "    {\n",
    "        \"dataset_path\": \"autogluon/chronos_datasets\",\n",
    "        \"dataset_config\": \"monash_electricity_weekly\",\n",
    "        \"horizon\": 8,\n",
    "        \"quantile_levels\": [0.1, 0.5, 0.9],\n",
    "        \"eval_metric\": \"WQL\",\n",
    "        \"variants\": [\n",
    "            {\"cutoff\": \"2013-01-01\"},\n",
    "            {\"cutoff\": \"2014-01-01\"},\n",
    "        ]\n",
    "    },\n",
    "]\n",
    "benchmark = fev.Benchmark.from_list(task_generators)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or from a YAML file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tasks:\n",
      "- dataset_path: autogluon/chronos_datasets\n",
      "  dataset_config: monash_m1_yearly\n",
      "  horizon: 8\n",
      "- dataset_path: autogluon/chronos_datasets\n",
      "  dataset_config: monash_electricity_weekly\n",
      "  horizon: 8\n",
      "  seasonality: 1\n",
      "  variants:\n",
      "  - cutoff: \"2013-01-01\"\n",
      "  - cutoff: \"2014-01-01\"\n"
     ]
    }
   ],
   "source": [
    "benchmark_path = Path(fev.__file__).parents[2] / \"benchmarks\" / \"example\" / \"tasks.yaml\"\n",
    "# Show contents of the benchmark YAML file\n",
    "!cat {benchmark_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "benchmark = fev.Benchmark.from_yaml(benchmark_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Task(dataset_path='autogluon/chronos_datasets', dataset_config='monash_m1_yearly', horizon=8, cutoff=-8, lead_time=1, min_ts_length=9, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[]),\n",
       " Task(dataset_path='autogluon/chronos_datasets', dataset_config='monash_electricity_weekly', horizon=8, cutoff='2013-01-01T00:00:00', lead_time=1, min_ts_length=9, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[]),\n",
       " Task(dataset_path='autogluon/chronos_datasets', dataset_config='monash_electricity_weekly', horizon=8, cutoff='2014-01-01T00:00:00', lead_time=1, min_ts_length=9, max_context_length=None, seasonality=1, eval_metric='MASE', extra_metrics=[], quantile_levels=None, id_column='id', timestamp_column='timestamp', target_column='target', multiple_target_columns=None, past_dynamic_columns=[], excluded_columns=[])]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "benchmark.tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's evaluate some simple forecasting models on this toy benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zsh:1: command not found: pip\n"
     ]
    }
   ],
   "source": [
    "!pip install statsforecast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsforecast.models import ARIMA, SeasonalNaive, Theta\n",
    "\n",
    "\n",
    "def predict_with_model(task: fev.Task, model_name: str = \"naive\") -> list[dict]:\n",
    "    past_data, future_data = task.get_input_data()\n",
    "    if model_name == \"seasonal_naive\":\n",
    "        model = SeasonalNaive(season_length=task.seasonality)\n",
    "    elif model_name == \"theta\":\n",
    "        model = Theta(season_length=task.seasonality)\n",
    "    elif model_name == \"arima\":\n",
    "        model = ARIMA(season_length=task.seasonality)\n",
    "    else:\n",
    "        raise ValueError(f\"Unknown model_name: {model_name}\")\n",
    "\n",
    "    predictions = []\n",
    "    for ts in past_data:\n",
    "        predictions.append(\n",
    "            {\"predictions\": model.forecast(y=ts[task.target_column], h=task.horizon)[\"mean\"]}\n",
    "        )\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa89e5d2ec8d4c8f8c83aa348d47218d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Tasks completed:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ef224484a4c4aab82cf3138f8048471",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "train-00000-of-00001.parquet:   0%|          | 0.00/334k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "summaries = []\n",
    "for task in tqdm(benchmark.tasks, desc=\"Tasks completed\"):\n",
    "    for model_name in [\"seasonal_naive\", \"arima\", \"theta\"]:\n",
    "        start_time = time.time()\n",
    "        predictions = predict_with_model(task, model_name=model_name)\n",
    "        infer_time_s = time.time() - start_time\n",
    "        eval_summary = task.evaluation_summary(predictions, model_name=model_name, inference_time_s=infer_time_s, training_time_s=0.0)\n",
    "\n",
    "        summaries.append(eval_summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gmean_relative_error</th>\n",
       "      <th>avg_rank</th>\n",
       "      <th>avg_inference_time_s</th>\n",
       "      <th>median_inference_time_s</th>\n",
       "      <th>avg_training_time_s</th>\n",
       "      <th>median_training_time_s</th>\n",
       "      <th>training_corpus_overlap</th>\n",
       "      <th>num_failures</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>theta</th>\n",
       "      <td>0.914107</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.312378</td>\n",
       "      <td>2.221101</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seasonal_naive</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.534666</td>\n",
       "      <td>4.066241</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>arima</th>\n",
       "      <td>1.870027</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.637881</td>\n",
       "      <td>0.692875</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                gmean_relative_error  avg_rank  avg_inference_time_s  \\\n",
       "model_name                                                             \n",
       "theta                       0.914107       1.0              9.312378   \n",
       "seasonal_naive              1.000000       2.0              3.534666   \n",
       "arima                       1.870027       3.0              0.637881   \n",
       "\n",
       "                median_inference_time_s  avg_training_time_s  \\\n",
       "model_name                                                     \n",
       "theta                          2.221101                  0.0   \n",
       "seasonal_naive                 4.066241                  0.0   \n",
       "arima                          0.692875                  0.0   \n",
       "\n",
       "                median_training_time_s  training_corpus_overlap  num_failures  \n",
       "model_name                                                                     \n",
       "theta                              0.0                      0.0             0  \n",
       "seasonal_naive                     0.0                      0.0             0  \n",
       "arima                              0.0                      0.0             0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fev.leaderboard(summaries, baseline_model=\"seasonal_naive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `leaderboard` method aggregates the performance into a single number.\n",
    "\n",
    "We can investigate the performance for individual tasks using the `pivot_table` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>model_name</th>\n",
       "      <th>arima</th>\n",
       "      <th>seasonal_naive</th>\n",
       "      <th>theta</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dataset_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>chronos_datasets_monash_electricity_weekly</th>\n",
       "      <td>3.056930</td>\n",
       "      <td>1.573758</td>\n",
       "      <td>1.497915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chronos_datasets_monash_m1_yearly</th>\n",
       "      <td>10.236634</td>\n",
       "      <td>5.898890</td>\n",
       "      <td>4.988582</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "model_name                                      arima  seasonal_naive  \\\n",
       "dataset_name                                                            \n",
       "chronos_datasets_monash_electricity_weekly   3.056930        1.573758   \n",
       "chronos_datasets_monash_m1_yearly           10.236634        5.898890   \n",
       "\n",
       "model_name                                     theta  \n",
       "dataset_name                                          \n",
       "chronos_datasets_monash_electricity_weekly  1.497915  \n",
       "chronos_datasets_monash_m1_yearly           4.988582  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fev.pivot_table(summaries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that our benchmark definition contains two tasks for `monash_electricity_weekly` with different cutoff dates. The above cell averaged the results across both cutoff dates.\n",
    "\n",
    "We can have a look at the results for individual cutoffs as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_name</th>\n",
       "      <th>arima</th>\n",
       "      <th>seasonal_naive</th>\n",
       "      <th>theta</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dataset_name</th>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">chronos_datasets_monash_electricity_weekly</th>\n",
       "      <th>2013-01-01T00:00:00</th>\n",
       "      <td>2.907207</td>\n",
       "      <td>1.520114</td>\n",
       "      <td>1.401086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-01T00:00:00</th>\n",
       "      <td>3.206652</td>\n",
       "      <td>1.627403</td>\n",
       "      <td>1.594743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chronos_datasets_monash_m1_yearly</th>\n",
       "      <th>-8</th>\n",
       "      <td>10.236634</td>\n",
       "      <td>5.898890</td>\n",
       "      <td>4.988582</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "model_name                                                          arima  \\\n",
       "dataset_name                               cutoff                           \n",
       "chronos_datasets_monash_electricity_weekly 2013-01-01T00:00:00   2.907207   \n",
       "                                           2014-01-01T00:00:00   3.206652   \n",
       "chronos_datasets_monash_m1_yearly          -8                   10.236634   \n",
       "\n",
       "model_name                                                      seasonal_naive  \\\n",
       "dataset_name                               cutoff                                \n",
       "chronos_datasets_monash_electricity_weekly 2013-01-01T00:00:00        1.520114   \n",
       "                                           2014-01-01T00:00:00        1.627403   \n",
       "chronos_datasets_monash_m1_yearly          -8                         5.898890   \n",
       "\n",
       "model_name                                                         theta  \n",
       "dataset_name                               cutoff                         \n",
       "chronos_datasets_monash_electricity_weekly 2013-01-01T00:00:00  1.401086  \n",
       "                                           2014-01-01T00:00:00  1.594743  \n",
       "chronos_datasets_monash_m1_yearly          -8                   4.988582  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fev.pivot_table(summaries, task_columns=[\"dataset_name\", \"cutoff\"])  # you can filter any task properties such as `eval_metric`, `horizon`, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both `leaderboard()` and `pivot_table()` methods can handle single or multiple evaluation summaries in different formats:\n",
    "- `pandas.DataFrame`\n",
    "- list of dictionaries\n",
    "- paths to JSONL (orient=\"record\") or CSV files\n",
    "\n",
    "Here is an example of how we can work with URLs of CSV files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gmean_relative_error</th>\n",
       "      <th>avg_rank</th>\n",
       "      <th>avg_inference_time_s</th>\n",
       "      <th>median_inference_time_s</th>\n",
       "      <th>avg_training_time_s</th>\n",
       "      <th>median_training_time_s</th>\n",
       "      <th>training_corpus_overlap</th>\n",
       "      <th>num_failures</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>auto_theta</th>\n",
       "      <td>0.863439</td>\n",
       "      <td>1.750000</td>\n",
       "      <td>276.781533</td>\n",
       "      <td>23.853687</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>auto_arima</th>\n",
       "      <td>0.873645</td>\n",
       "      <td>1.678571</td>\n",
       "      <td>1615.091113</td>\n",
       "      <td>65.344211</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>seasonal_naive</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.571429</td>\n",
       "      <td>2.327086</td>\n",
       "      <td>0.091613</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                gmean_relative_error  avg_rank  avg_inference_time_s  \\\n",
       "model_name                                                             \n",
       "auto_theta                  0.863439  1.750000            276.781533   \n",
       "auto_arima                  0.873645  1.678571           1615.091113   \n",
       "seasonal_naive              1.000000  2.571429              2.327086   \n",
       "\n",
       "                median_inference_time_s  avg_training_time_s  \\\n",
       "model_name                                                     \n",
       "auto_theta                    23.853687                  NaN   \n",
       "auto_arima                    65.344211                  NaN   \n",
       "seasonal_naive                 0.091613                  NaN   \n",
       "\n",
       "                median_training_time_s  training_corpus_overlap  num_failures  \n",
       "model_name                                                                     \n",
       "auto_theta                         NaN                      0.0             0  \n",
       "auto_arima                         NaN                      0.0             0  \n",
       "seasonal_naive                     NaN                      0.0             0  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summaries = [\n",
    "    \"https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/auto_arima.csv\",\n",
    "    \"https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/auto_theta.csv\",\n",
    "    \"https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/seasonal_naive.csv\",\n",
    "]\n",
    "fev.leaderboard(summaries, metric_column=\"MASE\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>model_name</th>\n",
       "      <th>auto_arima</th>\n",
       "      <th>auto_theta</th>\n",
       "      <th>seasonal_naive</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dataset_config</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ETTh</th>\n",
       "      <td>0.089012</td>\n",
       "      <td>0.132979</td>\n",
       "      <td>0.122090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ETTm</th>\n",
       "      <td>0.104990</td>\n",
       "      <td>0.078587</td>\n",
       "      <td>0.141348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dominick</th>\n",
       "      <td>0.484773</td>\n",
       "      <td>0.485493</td>\n",
       "      <td>0.452916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ercot</th>\n",
       "      <td>0.041214</td>\n",
       "      <td>0.041004</td>\n",
       "      <td>0.036604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>exchange_rate</th>\n",
       "      <td>0.010667</td>\n",
       "      <td>0.009714</td>\n",
       "      <td>0.012984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m4_quarterly</th>\n",
       "      <td>0.079384</td>\n",
       "      <td>0.079077</td>\n",
       "      <td>0.118648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m4_yearly</th>\n",
       "      <td>0.125041</td>\n",
       "      <td>0.114640</td>\n",
       "      <td>0.161439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m5</th>\n",
       "      <td>0.616520</td>\n",
       "      <td>0.636228</td>\n",
       "      <td>1.024088</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_australian_electricity</th>\n",
       "      <td>0.066902</td>\n",
       "      <td>0.054564</td>\n",
       "      <td>0.083695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_car_parts</th>\n",
       "      <td>1.333026</td>\n",
       "      <td>1.336601</td>\n",
       "      <td>1.599952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_cif_2016</th>\n",
       "      <td>0.033184</td>\n",
       "      <td>0.027333</td>\n",
       "      <td>0.015083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_covid_deaths</th>\n",
       "      <td>0.028973</td>\n",
       "      <td>0.094479</td>\n",
       "      <td>0.133085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_fred_md</th>\n",
       "      <td>0.035140</td>\n",
       "      <td>0.057386</td>\n",
       "      <td>0.122224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_hospital</th>\n",
       "      <td>0.058569</td>\n",
       "      <td>0.055279</td>\n",
       "      <td>0.072626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m1_monthly</th>\n",
       "      <td>0.154444</td>\n",
       "      <td>0.159026</td>\n",
       "      <td>0.191463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m1_quarterly</th>\n",
       "      <td>0.088292</td>\n",
       "      <td>0.082034</td>\n",
       "      <td>0.149502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m1_yearly</th>\n",
       "      <td>0.132822</td>\n",
       "      <td>0.137302</td>\n",
       "      <td>0.209296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m3_monthly</th>\n",
       "      <td>0.098059</td>\n",
       "      <td>0.094722</td>\n",
       "      <td>0.148545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m3_quarterly</th>\n",
       "      <td>0.076532</td>\n",
       "      <td>0.070173</td>\n",
       "      <td>0.101252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_m3_yearly</th>\n",
       "      <td>0.155759</td>\n",
       "      <td>0.127728</td>\n",
       "      <td>0.166533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_nn5_weekly</th>\n",
       "      <td>0.084427</td>\n",
       "      <td>0.089608</td>\n",
       "      <td>0.122691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_saugeenday</th>\n",
       "      <td>0.413626</td>\n",
       "      <td>0.479818</td>\n",
       "      <td>0.422968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_tourism_monthly</th>\n",
       "      <td>0.090934</td>\n",
       "      <td>0.090997</td>\n",
       "      <td>0.104182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_tourism_quarterly</th>\n",
       "      <td>0.099659</td>\n",
       "      <td>0.061241</td>\n",
       "      <td>0.119375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_tourism_yearly</th>\n",
       "      <td>0.128889</td>\n",
       "      <td>0.176017</td>\n",
       "      <td>0.209183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_traffic</th>\n",
       "      <td>0.353603</td>\n",
       "      <td>0.905307</td>\n",
       "      <td>0.361853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>monash_weather</th>\n",
       "      <td>0.214781</td>\n",
       "      <td>0.216555</td>\n",
       "      <td>0.216595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nn5</th>\n",
       "      <td>0.247710</td>\n",
       "      <td>0.293651</td>\n",
       "      <td>0.424621</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "model_name                     auto_arima  auto_theta  seasonal_naive\n",
       "dataset_config                                                       \n",
       "ETTh                             0.089012    0.132979        0.122090\n",
       "ETTm                             0.104990    0.078587        0.141348\n",
       "dominick                         0.484773    0.485493        0.452916\n",
       "ercot                            0.041214    0.041004        0.036604\n",
       "exchange_rate                    0.010667    0.009714        0.012984\n",
       "m4_quarterly                     0.079384    0.079077        0.118648\n",
       "m4_yearly                        0.125041    0.114640        0.161439\n",
       "m5                               0.616520    0.636228        1.024088\n",
       "monash_australian_electricity    0.066902    0.054564        0.083695\n",
       "monash_car_parts                 1.333026    1.336601        1.599952\n",
       "monash_cif_2016                  0.033184    0.027333        0.015083\n",
       "monash_covid_deaths              0.028973    0.094479        0.133085\n",
       "monash_fred_md                   0.035140    0.057386        0.122224\n",
       "monash_hospital                  0.058569    0.055279        0.072626\n",
       "monash_m1_monthly                0.154444    0.159026        0.191463\n",
       "monash_m1_quarterly              0.088292    0.082034        0.149502\n",
       "monash_m1_yearly                 0.132822    0.137302        0.209296\n",
       "monash_m3_monthly                0.098059    0.094722        0.148545\n",
       "monash_m3_quarterly              0.076532    0.070173        0.101252\n",
       "monash_m3_yearly                 0.155759    0.127728        0.166533\n",
       "monash_nn5_weekly                0.084427    0.089608        0.122691\n",
       "monash_saugeenday                0.413626    0.479818        0.422968\n",
       "monash_tourism_monthly           0.090934    0.090997        0.104182\n",
       "monash_tourism_quarterly         0.099659    0.061241        0.119375\n",
       "monash_tourism_yearly            0.128889    0.176017        0.209183\n",
       "monash_traffic                   0.353603    0.905307        0.361853\n",
       "monash_weather                   0.214781    0.216555        0.216595\n",
       "nn5                              0.247710    0.293651        0.424621"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fev.pivot_table(summaries, task_columns=\"dataset_config\", metric_column=\"WQL\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}