ML_quarterly.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ML_quarterly.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GgMrPZLBHnOo"
      },
      "source": [
        "# this notebook represents short-term forecasting(quarterly)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IKIY1WhO8z57"
      },
      "source": [
        "refs:\n",
        "\n",
        "\n",
        "1.   https://towardsdatascience.com/how-not-to-use-machine-learning-for-time-series-forecasting-avoiding-the-pitfalls-19f9d7adf424\n",
        "2.   (sklearn)https://towardsdatascience.com/time-series-modeling-using-scikit-pandas-and-numpy-682e3b8db8d1\n",
        "3.   https://towardsdatascience.com/sktime-a-unified-python-library-for-time-series-machine-learning-3c103c139a55\n",
        "4.   https://stackoverflow.com/questions/63517126/any-way-to-predict-monthly-time-series-with-scikit-learn-in-python\n",
        "5.   https://paperswithcode.com/paper/forecasting-foreign-exchange-rate-a\n",
        "6. https://github.com/alan-turing-institute/sktime/blob/main/examples/01_forecasting.ipynb\n",
        "7. https://arxiv.org/pdf/2005.08067.pdf (sktime paper-metrics p9)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iMMCpIyL46i"
      },
      "source": [
        "fine tuning RF : \n",
        "\n",
        "*   https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74\n",
        "*   xgboost: https://machinelearningmastery.com/xgboost-for-time-series-forecasting/\n",
        "* to read: https://towardsdatascience.com/random-forest-hyperparameters-and-how-to-fine-tune-them-17aee785ee0d \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V3mfVfCumG9k"
      },
      "source": [
        "Regarding a machine learning paper, consider starting by reviewing papers related to your work (Google Scholar may help) and write a section about it (keep in mind how your work relates to them and what innovation it brings to the field, and write about it). Then write sections for describing your dataset(s), for describing your algorithms and methodology, and why you have selected them. Also, don’t forget to write about your results (you will see how a machine learning paper is organized by reviewing other papers anyway).\n",
        "\n",
        "At the end, get ready to answer the reviews you will receive some time after submitting.\n",
        "\n",
        "It would be easier if you already have academic experience on researching or if you have a professor or experienced researcher to advise you with that. Also, at least talk to someone with academic experience to advise you WHERE you should submit your paper, given your work subject.\n",
        "\n",
        "In summary: yes, you can! But it will be a lot easier if you have an advisor, at least…\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYirXA1MqJYx"
      },
      "source": [
        "One also needs to choose the journal which is most suited to the subject field of the article. Is the article a review article or a piece of original investigation? Does the study provide incremental gains in knowledge or is there something exceptionally novel to report? Journals and fields will vary in what they consider appropriate and which formats they publish.\n",
        "\n",
        "Depending on the journal, choosing an appropriate editor(s) is the final step. There may be a list of editors and reviewers to choose from and it makes sense to choose someone who works in a closely related field."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jafUzMrprfnK"
      },
      "source": [
        "The stages of publishing research may differ by field and publisher, but broadly you'll follow these steps:\n",
        "\n",
        "Do the research\n",
        "Decide what research results you want to publish together as an account of your research findings. This should be decided independently of the direction of the results, i.e. whether or not you got the results you wanted.\n",
        "Discuss authorship with those who contributed to the research\n",
        "Draft the text, tables and figures and work with all the authors on revising and editing the paper. Anyone involved in this process who is not an author should be acknowledged.\n",
        "Optionally, get feedback from colleagues about the manuscript\n",
        "Optionally, get the paper edited by a language editing service\n",
        "Get approval from all the authors of the version to be submitted\n",
        "Agree among the authors where you will submit and how any page charges or article processing charges will be paid\n",
        "Have the underlying data and any code ready to share on request or ideally deposit this publicly. Also have documentation such as of ethics approval ready.\n",
        "Prepare a cover letter and the manuscript files\n",
        "Read the instructions for authors of the target journal, check you comply with their policies\n",
        "Submit the article via the online submission\n",
        "The article will undergo different initials checks / triage depending on the journal, such as scope or issues like reporting.\n",
        "If the paper is not rejected, it will be assigned to an editor, either a staff editor or an academic editor. In some journals, the Editor-in-Chief decides this allocation.\n",
        "The editor will decide if the paper is suitable for peer review. If not, they may reject the paper (often called a “desk reject”) or ask for revisions. Some journals discuss decisions or require approval by the EiC or a section editor, others give individual editors autonomy in their decisions.\n",
        "Most journals require peer review before a paper can be accepted, though some rely on editorial review. Peer reviewers are experts in the topic and/or methods of the research. They may be selected from a database of reviewers, researchers the editors knows of themselves, or chosen based on their publication record.\n",
        "Potential reviewers are asked to review, usually by email but occasionally by phone. If they agree, they will be sent the paper and given anything from a few days to several weeks to submit their review. The number of reviewers is usually two or three, but can vary. If reviewers decline, they may suggest alternatives. Publishing staff often support editors in this process.\n",
        "Reviewers will usually send their comments via the online system or email. They might be asked to answer specific questions and are often asked about their recommendation (accept/revise/reject).\n",
        "Once the editor has enough reviews, they will make their decision based on the reviews, the journal guidelines, and their own assessment of the paper and the reviewers' expertise. The editor may be able to overrule the reviewers. Their decision might need to be approved or discussed. They will send their decision, usually by email, to the authors.\n",
        "If the article needs revisions, the authors will be asked to resubmit by a certain date. That deadline can often be extended. Authors might need to do more research, a reanalysis, or revise the language.\n",
        "If the article is rejected, the authors might appeal the decision or be allowed to resubmit to the same journal if they address all the editor and reviewer concerns. Otherwise, go back to step 10 at the next journal of the authors' choice.\n",
        "If revised, editor might make the next decision themselves or send the paper for re-review\n",
        "Once accepted, the paper might be posted publicly as an “in press” version or this might wait until production is complete\n",
        "During production, typesetting, formatting, copyediting and other checks will be performed by the publisher\n",
        "The final article is then formally published, usually as HTML and/or PDF, and sent to indexing services"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "scyRsTjNg2Jt"
      },
      "source": [
        "#ref 1\n",
        "#make zoom on pred vs test data\n",
        "#make cross-correlation between pred-test data\n",
        "#dont be overconfident with traditional accuracy metrics : r2, mae,...\n",
        "#ref2\n",
        "#A better alternative for cross validation on time series data (than K-fold CV) is \n",
        "#Forward Chaining strategy.\n",
        "\n",
        "#sktime tuto github [6]\n",
        "#1.2.1, 1.2.2, 1.3, 1.4.1, 3.3(tuning), 3.4 (ensembling)\n",
        "#LSTM can overcome the problem of back-propagated error decay through memory blocks of LSTM. In \n",
        "#contrast, MLP) models very often face the problem ofoverfitting, backpropagated error decay, \n",
        "#and it cannot automatically determine the optimal time lags while fitting time-series data[5]\n",
        "\n",
        "#ML mastery: xgboost, Random Forest"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1HRWeLc2Gc10"
      },
      "source": [
        "\n",
        "from numpy import argsort\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "from sklearn.ensemble import VotingRegressor\n",
        "import pandas as pd\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from numpy import concatenate\n",
        "from sklearn.model_selection import TimeSeriesSplit\n",
        "from xgboost import XGBRegressor\n",
        "from sklearn.model_selection import cross_val_score\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.neural_network import MLPRegressor\n",
        "from sklearn.neighbors import KNeighborsRegressor\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.gaussian_process import GaussianProcessRegressor\n",
        "from sklearn.linear_model import Ridge\n",
        "from sklearn import linear_model\n",
        "from sklearn.svm import SVR\n",
        "from sklearn.linear_model import ElasticNet\n",
        "import sklearn\n",
        "from sklearn import metrics\n",
        "import numpy as np\n",
        "import matplotlib.dates as mdates"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pAw-KeR0XqqF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc908999-74df-4593-c717-0818b0ada174"
      },
      "source": [
        "#read multivariate data\n",
        "df=pd.read_csv('/content/clean_quarterly_df_human_supervised.csv')\n",
        "df.time=pd.to_datetime(df.time,dayfirst=True)\n",
        "df.columns"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['time', 'MIN_TEMPERATURE_C_t_1', 'max_clients_quarter_t_1',\n",
              "       'total_Holiday_Food_Related_t_1', 'total_consommation_t_1',\n",
              "       'pdl_max_total_daily_load_t_1', 'pdl_max_daily_load_t_1',\n",
              "       'MIN_TEMPERATURE_C_t', 'max_clients_quarter_t',\n",
              "       'total_Holiday_Food_Related_t', 'total_consommation_t',\n",
              "       'pdl_max_total_daily_load_t', 'pdl_max_daily_load_t'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wgKOf0xp8ZFC"
      },
      "source": [
        "data=pd.read_csv('/content/clean_quarterly_df_human_supervised.csv',\n",
        "                 index_col='time')[['MIN_TEMPERATURE_C_t_1', 'max_clients_quarter_t_1',\n",
        "       'total_Holiday_Food_Related_t_1', 'total_consommation_t_1',\n",
        "       'pdl_max_total_daily_load_t_1', 'pdl_max_daily_load_t_1',\n",
        "\n",
        "       'MIN_TEMPERATURE_C_t', 'max_clients_quarter_t',\n",
        "       'total_Holiday_Food_Related_t','pdl_max_daily_load_t']].dropna()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e9QfJ8FPyGYL",
        "outputId": "4880317b-7ef5-44cc-96a7-d70276292b20"
      },
      "source": [
        "data.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(21, 10)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dRaIduTRqmPq"
      },
      "source": [
        "data.index=pd.to_datetime(data.index,dayfirst=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "id": "XOKVBlf2-uTH",
        "outputId": "34a4145f-e661-4868-e43d-ab56d9628f81"
      },
      "source": [
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MIN_TEMPERATURE_C_t_1</th>\n",
              "      <th>max_clients_quarter_t_1</th>\n",
              "      <th>total_Holiday_Food_Related_t_1</th>\n",
              "      <th>total_consommation_t_1</th>\n",
              "      <th>pdl_max_total_daily_load_t_1</th>\n",
              "      <th>pdl_max_daily_load_t_1</th>\n",
              "      <th>MIN_TEMPERATURE_C_t</th>\n",
              "      <th>max_clients_quarter_t</th>\n",
              "      <th>total_Holiday_Food_Related_t</th>\n",
              "      <th>pdl_max_daily_load_t</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>time</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2015-06-30</th>\n",
              "      <td>8</td>\n",
              "      <td>4060</td>\n",
              "      <td>2</td>\n",
              "      <td>757059.40</td>\n",
              "      <td>22257</td>\n",
              "      <td>1551</td>\n",
              "      <td>14</td>\n",
              "      <td>4137</td>\n",
              "      <td>13</td>\n",
              "      <td>1253</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-09-30</th>\n",
              "      <td>14</td>\n",
              "      <td>4137</td>\n",
              "      <td>13</td>\n",
              "      <td>690218.16</td>\n",
              "      <td>16752</td>\n",
              "      <td>1253</td>\n",
              "      <td>23</td>\n",
              "      <td>4194</td>\n",
              "      <td>23</td>\n",
              "      <td>1051</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-12-31</th>\n",
              "      <td>23</td>\n",
              "      <td>4194</td>\n",
              "      <td>23</td>\n",
              "      <td>669288.36</td>\n",
              "      <td>9186</td>\n",
              "      <td>1051</td>\n",
              "      <td>13</td>\n",
              "      <td>4242</td>\n",
              "      <td>3</td>\n",
              "      <td>1327</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2016-03-31</th>\n",
              "      <td>13</td>\n",
              "      <td>4242</td>\n",
              "      <td>3</td>\n",
              "      <td>702182.67</td>\n",
              "      <td>18801</td>\n",
              "      <td>1327</td>\n",
              "      <td>11</td>\n",
              "      <td>4346</td>\n",
              "      <td>1</td>\n",
              "      <td>1391</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2016-06-30</th>\n",
              "      <td>11</td>\n",
              "      <td>4346</td>\n",
              "      <td>1</td>\n",
              "      <td>735924.27</td>\n",
              "      <td>20212</td>\n",
              "      <td>1391</td>\n",
              "      <td>16</td>\n",
              "      <td>4385</td>\n",
              "      <td>25</td>\n",
              "      <td>1269</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            MIN_TEMPERATURE_C_t_1  ...  pdl_max_daily_load_t\n",
              "time                               ...                      \n",
              "2015-06-30                      8  ...                  1253\n",
              "2015-09-30                     14  ...                  1051\n",
              "2015-12-31                     23  ...                  1327\n",
              "2016-03-31                     13  ...                  1391\n",
              "2016-06-30                     11  ...                  1269\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "id": "xktaLYwMkSPU",
        "outputId": "28c12ec5-cae1-454d-a5a6-2ca3a944626b"
      },
      "source": [
        "data.tail()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>MIN_TEMPERATURE_C_t_1</th>\n",
              "      <th>max_clients_quarter_t_1</th>\n",
              "      <th>total_Holiday_Food_Related_t_1</th>\n",
              "      <th>total_consommation_t_1</th>\n",
              "      <th>pdl_max_total_daily_load_t_1</th>\n",
              "      <th>pdl_max_daily_load_t_1</th>\n",
              "      <th>MIN_TEMPERATURE_C_t</th>\n",
              "      <th>max_clients_quarter_t</th>\n",
              "      <th>total_Holiday_Food_Related_t</th>\n",
              "      <th>pdl_max_daily_load_t</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>time</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019-06-30</th>\n",
              "      <td>9</td>\n",
              "      <td>5217</td>\n",
              "      <td>1</td>\n",
              "      <td>728026.49</td>\n",
              "      <td>26742</td>\n",
              "      <td>1600</td>\n",
              "      <td>13</td>\n",
              "      <td>5248</td>\n",
              "      <td>33</td>\n",
              "      <td>1556</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-09-30</th>\n",
              "      <td>13</td>\n",
              "      <td>5248</td>\n",
              "      <td>33</td>\n",
              "      <td>489350.13</td>\n",
              "      <td>18087</td>\n",
              "      <td>1556</td>\n",
              "      <td>22</td>\n",
              "      <td>5308</td>\n",
              "      <td>4</td>\n",
              "      <td>1077</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019-12-31</th>\n",
              "      <td>22</td>\n",
              "      <td>5308</td>\n",
              "      <td>4</td>\n",
              "      <td>699426.90</td>\n",
              "      <td>12408</td>\n",
              "      <td>1077</td>\n",
              "      <td>12</td>\n",
              "      <td>5354</td>\n",
              "      <td>1</td>\n",
              "      <td>1469</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2020-03-31</th>\n",
              "      <td>12</td>\n",
              "      <td>5354</td>\n",
              "      <td>1</td>\n",
              "      <td>878253.16</td>\n",
              "      <td>27936</td>\n",
              "      <td>1469</td>\n",
              "      <td>8</td>\n",
              "      <td>5348</td>\n",
              "      <td>1</td>\n",
              "      <td>1560</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2020-06-30</th>\n",
              "      <td>8</td>\n",
              "      <td>5348</td>\n",
              "      <td>1</td>\n",
              "      <td>691127.62</td>\n",
              "      <td>26139</td>\n",
              "      <td>1560</td>\n",
              "      <td>12</td>\n",
              "      <td>5326</td>\n",
              "      <td>33</td>\n",
              "      <td>1599</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            MIN_TEMPERATURE_C_t_1  ...  pdl_max_daily_load_t\n",
              "time                               ...                      \n",
              "2019-06-30                      9  ...                  1556\n",
              "2019-09-30                     13  ...                  1077\n",
              "2019-12-31                     22  ...                  1469\n",
              "2020-03-31                     12  ...                  1560\n",
              "2020-06-30                      8  ...                  1599\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmwYalon6Y4C"
      },
      "source": [
        "# data scaling made difference and improved the performance of some ML models (KNN, SVR, Ridge), but scaling is not required for decision tree based models (decision tree, random forest, xgboost,..) https://towardsdatascience.com/do-decision-trees-need-feature-scaling-97809eaa60c6"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gdnkSpWaENYQ"
      },
      "source": [
        "scaler = MinMaxScaler(feature_range=(0, 1))\n",
        "data_scaled = scaler.fit_transform(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IEZuMY99zfLU"
      },
      "source": [
        "features=data_scaled[:,:-1]\n",
        "target=data_scaled[:,-1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nDaUCPH6njFx"
      },
      "source": [
        "# split into train and test sets\n",
        "\n",
        "n_train_days = -2\n",
        "\n",
        "X_train,y_train = features[:n_train_days, :],target[:n_train_days]\n",
        "X_test, y_test = features[n_train_days:, :],target[n_train_days:]\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KmrKiF8_LVs6"
      },
      "source": [
        "def timeseries_evaluation_metrics_func(y_true, y_pred):\n",
        "  def mean_absolute_percentage_error(y_true, y_pred):\n",
        "    y_true, y_pred = np.array(y_true), np.array(y_pred)\n",
        "    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100\n",
        "  def smape(y_true,y_pred):\n",
        "    y_true, y_pred = np.array(y_true), np.array(y_pred)\n",
        "    return 2/y_true.shape[0]*np.sum(np.abs(y_true-y_pred))/np.sum((np.abs(y_true)+np.abs(y_pred)))*100\n",
        "  def mase(y_true,y_pred):\n",
        "    y_true, y_pred = np.array(y_true), np.array(y_pred)\n",
        "    naive_pred=y_true[1:]\n",
        "    return 1/y_true.shape[0]*np.sum(np.abs(y_true-y_pred))/np.sum(np.abs(y_true[:len(y_true)-1]-naive_pred))*(y_true.shape[0]-1)\n",
        "  #def owe(y_true,y_pred):\n",
        "   # return (mase(y_true,y_pred)+smape(y_true,y_pred))/2\n",
        "\n",
        "  print('Evaluation metric results:-')\n",
        "  print(f'MSE is : {metrics.mean_squared_error(y_true, y_pred)}')\n",
        "  print(f'MAE is : {metrics.mean_absolute_error(y_true, y_pred)}')\n",
        "  rmse=np.sqrt(metrics.mean_squared_error(y_true, y_pred))\n",
        "  print(f'RMSE is : {rmse}')\n",
        "  mape=mean_absolute_percentage_error(y_true, y_pred)\n",
        "  print(f'MAPE is : {mape} %')\n",
        "  r2=metrics.r2_score(y_true, y_pred)\n",
        "  print(f'R2 is : {r2}',end='\\n\\n')\n",
        "  print('SMAPE is : '+str(smape(y_true,y_pred))+'\\n')\n",
        "  print('MASE is : '+str(mase(y_true,y_pred)))\n",
        "  #model_scores[model_name] = [rmse, mape, r2]\n",
        "  return"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o0gtYxCKB8cy",
        "outputId": "c90d106a-446b-42db-9050-cc2395048299"
      },
      "source": [
        "!pip install xgboost"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: xgboost in /usr/local/lib/python3.7/dist-packages (0.90)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.19.5)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 436
        },
        "id": "5z6WpGP7BWs8",
        "outputId": "743176a2-29d9-4e00-96cc-75318799e173"
      },
      "source": [
        "# ML models to test\n",
        "models = []\n",
        "models.append(('LR', LinearRegression()))\n",
        "models.append(('Ridge R',Ridge()))\n",
        "models.append(('lasso',linear_model.Lasso()))\n",
        "models.append(('elastic net',ElasticNet()))\n",
        "#models.append(('GP',GaussianProcessRegressor())) (very poor)\n",
        "models.append(('NN', MLPRegressor(solver = 'lbfgs')))  #neural network\n",
        "models.append(('KNN', KNeighborsRegressor(n_neighbors=2))) \n",
        "models.append(('RF', RandomForestRegressor(n_estimators = 1000))) # Ensemble method - collection of many decision trees\n",
        "models.append(('SVR', SVR(gamma='auto'))) # kernel = linear\n",
        "models.append(('Xgboost',XGBRegressor(objective='reg:squarederror', n_estimators=1000)))\n",
        "# Evaluate each model in turn\n",
        "results = []\n",
        "names = []\n",
        "for name, model in models:\n",
        "    # TimeSeries Cross validation\n",
        "  tscv = TimeSeriesSplit(n_splits=4)\n",
        "    #n_splits=3 : because in train data there are 3 steps of prediction\n",
        "    #1-2016 features -->predict 2017; 2-16,17-->predict 18; 3-16,17,18-->predict 19\n",
        "  cv_results = cross_val_score(model, X_train, y_train, cv=tscv, scoring='r2')\n",
        "  results.append(cv_results)\n",
        "  names.append(name)\n",
        "  print('%s: %f (%f)' % (name, cv_results.mean(), cv_results.std()))\n",
        "    \n",
        "# Compare Algorithms\n",
        "plt.boxplot(results, labels=names)\n",
        "plt.title('Algorithm Comparison')\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "LR: -4547.064489 (7877.141700)\n",
            "Ridge R: -13.479984 (24.491960)\n",
            "lasso: -37.318982 (64.584685)\n",
            "elastic net: -37.318982 (64.584685)\n",
            "NN: -25.704828 (45.936461)\n",
            "KNN: -17.810301 (32.102403)\n",
            "RF: -11.842626 (21.804875)\n",
            "SVR: -19.963895 (35.727702)\n",
            "Xgboost: -21.033889 (37.715766)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEICAYAAACj2qi6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7xVdZ3/8debi2iZikreQKERGzhYTOxf6aSWSRM2/iLNFMdJbCjyl9nFepQOTZGTjXYZG6+FWWIKiJZKaV4Q1KhMD4kKkgmoCaIexbyMys3P74/1PbTYnX3O91z2OQd4Px+P9Thrfb/ftdZn7bPX/qz1/a5ztiICMzOzHH16OgAzM9tyOGmYmVk2Jw0zM8vmpGFmZtmcNMzMLJuThpmZZXPSsF5D0uWSvlmnbZ8o6dZW6t8raWU99r2lk/Tvkn7U03FY7+CkYd1O0h2Snpc0oLv2GRFXRcQ/lWIISft31/5V+KykxZL+V9JKSddIOrC7YuioiPhWRHyip+Ow3sFJw7qVpKHAoUAAH+qmffbrjv204X+AzwGfBXYFDgCuB/65J4NqSy957awXcdKw7nYScDdwOTCxtYaSvixptaQnJX2ifHcgaWdJV0hqkvS4pK9K6pPqTpb0G0nnSXoOmJrKFqT6u9Iu7pf0sqTjS/v8oqRn0n4/Xiq/XNLFkn6V1vmNpD0lfT/dNf1R0j/UOI7hwKnACRExLyLWRsQr6e7nnHYez18krZD0j6n8iRTvxKpYfyDpNkkvSbpT0n6l+v9J670oaaGkQ0t1UyVdK+lKSS8CJ6eyK1P99qnuuRTLvZL2SHV7S5ojaY2kZZI+WbXd2ekYX5K0RFKltd+/9U5OGtbdTgKuStMHmj9wqkkaB5wOjAX2B95b1eQCYGfgLcB70nY/Xqp/F7AC2AM4u7xiRByWZt8eETtGxNVpec+0zX2AScBFkgaWVj0O+CqwO7AW+B3wh7R8LfDfNY75CGBlRNxToz73eB4AdgNmALOA/0Px2vwrcKGkHUvtTwT+M8W2iOL1bnYvMJrijmcGcI2k7Uv149Px7FK1HhSJfmdgSIrlFODVVDcLWAnsDRwLfEvS+0rrfii12QWYA1zYyuthvZSThnUbSYcA+wGzI2IhsBz4lxrNjwN+EhFLIuIVYGppO32BCcCZEfFSRDwGfA/4WGn9JyPigojYEBGvkmc9cFZErI+Im4CXgbeW6q+LiIUR8RpwHfBaRFwRERuBq4EW7zQoPlxX19pp5vE8GhE/Ke1rSIp1bUTcCqyjSCDNboyIuyJiLTAFOFjSEICIuDIinkuvzfeAAVXH+buIuD4iXm/htVufjmf/iNiYXo8X07bfDXwlIl6LiEXAjyiSX7MFEXFTOoafAm+v9ZpY7+WkYd1pInBrRDyblmdQu4tqb+CJ0nJ5fnegP/B4qexxijuEltrnei4iNpSWXwHKV+9Pl+ZfbWG53Haz7QJ7tbLfnOOp3hcR0dr+Nx1/RLwMrKF4TZH0JUlLJb0g6S8Udw67t7RuC34K3ALMSt2G35bUP217TUS81MoxPFWafwXY3mMmWx4nDesWknaguHt4j6SnJD0FfAF4u6SWrjhXA4NLy0NK889SXPHuVyrbF1hVWu5N/775dmBwK334OcfTXpter9RttSvwZBq/+DLF72JgROwCvACotG7N1y7dhX0jIkYC/wgcRXE38SSwq6Q3deExWC/kpGHd5cPARmAkRX/6aGAE8Gs278JoNhv4uKQRkt4A/EdzReremA2cLelNaZD3dODKdsTzNMX4Qd1FxCPAxcBMFX8Psl0aUJ4g6YwuOp5qH5R0iKTtKMY27o6IJ4A3ARuAJqCfpK8BO+VuVNLhkg5MXWovUiS719O2fwv8Vzq2t1GMC3XmGKwXctKw7jKRYozizxHxVPNEMRh6YnU3RUT8CjgfmA8so3jiCooBaIDTgP+lGOxeQNHV9eN2xDMVmJ6eADqug8fUHp+lONaLgL9QjOccDfwi1Xf2eKrNAL5O0S01hmKwHIqupZuBP1F0H71G+7ry9qQYJH8RWArcSdFlBXACMJTiruM64OsRMbcTx2C9kPwlTLYlkDQCWAwMqBp3sCqSLqd4WuurPR2LbX18p2G9lqSjJQ1Ij72eC/zCCcOsZzlpWG/2KeAZiq6cjcD/69lwzMzdU2Zmls13GmZmlm2r/cOa3XffPYYOHdrTYZiZbVEWLlz4bEQMqlW/1SaNoUOH0tjY2NNhmJltUSQ93lq9u6fMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsm1RSUPSOEkPp6+SPKOn4zEz29ZsMUkj/Svmi4AjKf699gmSRvZsVN1v5syZjBo1ir59+zJq1ChmzpzZ0yFZO/TG359jckztEhFbxAQcDNxSWj6T4usxW2w/ZsyY2NrMmDEjhg0bFvPmzYt169bFvHnzYtiwYTFjxowej6uhoSH69OkTDQ0NPR5Pb46pt/3+HJNjqgY0RiufxVvM/56SdCwwLiI+kZY/BrwrIj5TajMZmAyw7777jnn88Vb/RqUwdeeuC3LqC120na08JuiauBxTO7azlb+nemNMsEW+pyQtjIha3zK5dSWNskqlEjl/ES6JrngNumo7renbty+vvfYa/fv331S2fv16tt9+ezZu3FjXfUtqu1GmgQMHsmbNmk5vxzF1Xk++pxxT74ypraSxxYxpUHzXcPl7ogezjX3/8IgRI1iwYMFmZQsWLGDEiBF133etW9U+ffqwbt26zcrWrVtHnz59aq7TVR+EW1JMEUFDQwPz5s3brGzevHk0NDTUNabW9OR7qhbHlKfHYmrtTd6bJor/k7UCGAZsB9wPNNRqnzumUbwEnddV22lNb+xXbWhoiHnz5m1WNm/evGhoaOihiHpnTBG98/fnmBxTNdoY0+jxZNCeCfggxXcbLwemtNZ2a0waEb1vgHdbOpm6Krbe9PtzTI6p2laVNNozba1JozfaVk4ms21BW0ljixkIb6+tcSDczKzetqaBcDMz62FOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2x1SxqSviPpj5IekHSdpF1S+VBJr0palKYflNYZI+lBScsknS9JqXxXSbdJeiT9HFivuM3MrLZ63mncBoyKiLcBfwLOLNUtj4jRaTqlVH4J8ElgeJrGpfIzgNsjYjhwe1o2M7NuVrekERG3RsSGtHg3MLi19pL2AnaKiLvTl5tfAXw4VY8Hpqf56aVyMzPrRt01pvFvwK9Ky8Mk3SfpTkmHprJ9gJWlNitTGcAeEbE6zT8F7NHSTiRNltQoqbGpqakLwzczM4B+nVlZ0lxgzxaqpkTEDanNFGADcFWqWw3sGxHPSRoDXC+pIXefERGSokbdNGAaQKVSabGNmZl1XKeSRkSMba1e0snAUcARqcuJiFgLrE3zCyUtBw4AVrF5F9bgVAbwtKS9ImJ16sZ6pjNxm5lZx9Tz6alxwJeBD0XEK6XyQZL6pvm3UAx4r0jdTy9KOig9NXUScENabQ4wMc1PLJWbmVk36tSdRhsuBAYAt6UnZ+9OT0odBpwlaT3wOnBKRKxJ63wauBzYgWIMpHkc5BxgtqRJwOPAcXWM28zMaqhb0oiI/WuU/wz4WY26RmBUC+XPAUd0aYBmZtZu/otwMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPLVs+ve91ipK+j7ZSBAwd2QSRmZr1b3e40JE2VtErSojR9sFR3pqRlkh6W9IFS+bhUtkzSGaXyYZJ+n8qvlrRdV8UZEV0yrVmzpu2dmZlt4erdPXVeRIxO000AkkYCE4AGYBxwsaS+kvoCFwFHAiOBE1JbgHPTtvYHngcm1TluMzNrQU+MaYwHZkXE2oh4FFgGvDNNyyJiRUSsA2YB41X0Hb0PuDatPx34cA/EbWa2zat30viMpAck/VhSc6f/PsATpTYrU1mt8t2Av0TEhqpyMzPrZp1KGpLmSlrcwjQeuAT4O2A0sBr4XhfE21Y8kyU1Smpsamqq9+7MzLY5nXp6KiLG5rSTdCnwy7S4ChhSqh6cyqhR/hywi6R+6W6j3L46nmnANIBKpRKZh2FmZpnq+fTUXqXFo4HFaX4OMEHSAEnDgOHAPcC9wPD0pNR2FIPlcyIigPnAsWn9icAN9YrbzMxqq+ffaXxb0mgggMeATwFExBJJs4GHgA3AqRGxEUDSZ4BbgL7AjyNiSdrWV4BZkr4J3AdcVse4zcysBhUX8lufSqUSjY2NPR2GmdkWRdLCiKjUqve/ETEzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMstUtaUi6WtKiND0maVEqHyrp1VLdD0rrjJH0oKRlks6XpFS+q6TbJD2Sfg6sV9xmZlZb3ZJGRBwfEaMjYjTwM+DnperlzXURcUqp/BLgk8DwNI1L5WcAt0fEcOD2tGxmZt2s7t1T6W7hOGBmG+32AnaKiLsjIoArgA+n6vHA9DQ/vVRuZmbdqDvGNA4Fno6IR0plwyTdJ+lOSYemsn2AlaU2K1MZwB4RsTrNPwXs0dKOJE2W1CipsampqQsPwczMAPp1ZmVJc4E9W6iaEhE3pPkT2PwuYzWwb0Q8J2kMcL2khtx9RkRIihp104BpAJVKpcU2ZmbWcZ1KGhExtrV6Sf2AY4AxpXXWAmvT/EJJy4EDgFXA4NLqg1MZwNOS9oqI1akb65nOxG1mZh1T7+6pscAfI2JTt5OkQZL6pvm3UAx4r0jdTy9KOiiNg5wENN+tzAEmpvmJpXIzM+tGnbrTyDCBvx0APww4S9J64HXglIhYk+o+DVwO7AD8Kk0A5wCzJU0CHqcYWDczs26m4kGlrU+lUonGxsaeDsPMbIsiaWFEVGrV+y/Czcwsm5OGmZllc9IwM7NsThpmZpbNScPMzLI5aZiZWTYnDTMzy+akYWZm2Zw0zMwsm5OGmZllc9IwM7NsThpmZpbNScPMzLI5aZiZWTYnDTMzy+akYWZm2Zw0zMwsm5OGmZll63TSkPRRSUskvS6pUlV3pqRlkh6W9IFS+bhUtkzSGaXyYZJ+n8qvlrRdKh+Qlpel+qGdjdvMzNqvK+40FgPHAHeVCyWNBCYADcA44GJJfSX1BS4CjgRGAiektgDnAudFxP7A88CkVD4JeD6Vn5famZlZN+t00oiIpRHxcAtV44FZEbE2Ih4FlgHvTNOyiFgREeuAWcB4SQLeB1yb1p8OfLi0relp/lrgiNTezMy6UT3HNPYBnigtr0xltcp3A/4SERuqyjfbVqp/IbXfjKTJkholNTY1NXXhoZiZGUC/nEaS5gJ7tlA1JSJu6NqQOi4ipgHTACqVSvRwOGZmW52spBERYzuw7VXAkNLy4FRGjfLngF0k9Ut3E+X2zdtaKakfsHNqb2Zm3aie3VNzgAnpyadhwHDgHuBeYHh6Umo7isHyORERwHzg2LT+ROCG0rYmpvljgXmpvZmZdaOueOT2aEkrgYOBGyXdAhARS4DZwEPAzcCpEbEx3UV8BrgFWArMTm0BvgKcLmkZxZjFZan8MmC3VH46sOkxXTMz6z7aWi/YK5VKNDY29nQYZmZbFEkLI6JSq95/EW5mZtmcNMzMLJuThpmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtmcNMzMLJuThpmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtmcNMzMLFunkoakj0paIul1SZVS+fslLZT0YPr5vlLdHZIelrQoTW9O5QMkXS1pmaTfSxpaWufMVP6wpA90JmYzM+u4fp1cfzFwDPDDqvJngf8bEU9KGgXcAuxTqj8xIqq/wHsS8HxE7C9pAnAucLykkcAEoAHYG5gr6YCI2NjJ2M3MrJ06dacREUsj4uEWyu+LiCfT4hJgB0kD2tjceGB6mr8WOEKSUvmsiFgbEY8Cy4B3diZuMzPrmO4Y0/gI8IeIWFsq+0nqmvqPlBiguBN5AiAiNgAvALuVy5OVbH7XsomkyZIaJTU2NTV19XGYmW3z2kwakuZKWtzCND5j3QaKbqZPlYpPjIgDgUPT9LGOBl8tIqZFRCUiKoMGDeqqzZqZWdLmmEZEjO3IhiUNBq4DToqI5aXtrUo/X5I0g6Kr6QpgFTAEWCmpH7Az8FypvNngVGZmZt2sLt1TknYBbgTOiIjflMr7Sdo9zfcHjqIYTAeYA0xM88cC8yIiUvmE9HTVMGA4cE894jYzs9Z19pHboyWtBA4GbpR0S6r6DLA/8LWqR2sHALdIegBYRHHHcGla5zJgN0nLgNOBMwAiYgkwG3gIuBk41U9OmZn1DBUX81ufSqUSjY3VT/WamVlrJC2MiEqtev9FuJmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtmcNMzMLJuThpmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtmcNMzMLJuThpmZZXPSMDOzbJ39jvCPSloi6XVJlVL5UEmvlr4f/AelujGSHpS0TNL5kpTKd5V0m6RH0s+BqVyp3TJJD0h6R2diNjOzjuvsncZi4BjgrhbqlkfE6DSdUiq/BPgkMDxN41L5GcDtETEcuD0tAxxZajs5rW9mZj2gU0kjIpZGxMO57SXtBewUEXdHRABXAB9O1eOB6Wl+elX5FVG4G9glbcfMzLpZPcc0hkm6T9Kdkg5NZfsAK0ttVqYygD0iYnWafwrYo7TOEzXW2YykyZIaJTU2NTV1yUGYmdlf9WurgaS5wJ4tVE2JiBtqrLYa2DcinpM0BrheUkNuUBERkiK3fWm9acA0gEql0u71zcysdW0mjYgY296NRsRaYG2aXyhpOXAAsAoYXGo6OJUBPC1pr4hYnbqfnknlq4AhNdYxM7NuVJfuKUmDJPVN82+hGMRekbqfXpR0UHpq6iSg+W5lDjAxzU+sKj8pPUV1EPBCqRvLzMy6UWcfuT1a0krgYOBGSbekqsOAByQtAq4FTomINanu08CPgGXAcuBXqfwc4P2SHgHGpmWAm4AVqf2laX0zM+sBKh5i2vpUKpVobGzs6TDMzLYokhZGRKVWvf8i3MzMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZts5+R/hHJS2R9LqkSqn8REmLStPrkkanujskPVyqe3MqHyDpaknLJP1e0tDS9s5M5Q9L+kBnYjYzs47r18n1FwPHAD8sF0bEVcBVAJIOBK6PiEWlJidGRPUXeE8Cno+I/SVNAM4Fjpc0EpgANAB7A3MlHRARGzsZu5mZtVOn7jQiYmlEPNxGsxOAWRmbGw9MT/PXAkdIUiqfFRFrI+JRYBnwzo7GbGZmHdcdYxrHAzOryn6Suqb+IyUGgH2AJwAiYgPwArBbuTxZmcr+hqTJkholNTY1NXXlMZiZGRlJQ9JcSYtbmMZnrPsu4JWIWFwqPjEiDgQOTdPHOhx9lYiYFhGViKgMGjSoqzZrZmZJm2MaETG2E9ufQNVdRkSsSj9fkjSDoqvpCmAVMARYKakfsDPwXKm82eBUZmZm3axu3VOS+gDHURrPkNRP0u5pvj9wFMVgOsAcYGKaPxaYFxGRyiekp6uGAcOBe+oVt5mZ1dapp6ckHQ1cAAwCbpS0KCKaH4k9DHgiIlaUVhkA3JISRl9gLnBpqrsM+KmkZcAairsUImKJpNnAQ8AG4FQ/OWVm1jNUXMxvfSqVSjQ2Vj/V2z4zZ87k7LPPZunSpYwYMYIpU6ZwwgkndFGEZma9j6SFEVGpVd/Zv9PYas2cOZMpU6Zw2WWXccghh7BgwQImTZoE4MRhZtss32nUMGrUKC644AIOP/zwTWXz58/ntNNOY/Hixa2saWa25WrrTsNJo4a+ffvy2muv0b9//01l69evZ/vtt2fjRg+pmNnWqa2k4X9YWMOIESNYsGDBZmULFixgxIgRPRSRmVnPc9KoYcqUKUyaNIn58+ezfv165s+fz6RJk5gyZUpPh2Zm1mM8EF5D82D3aaedtunpqbPPPtuD4Ga2TfOYhpmZbeIxDTMz6zJOGmZmls1Jw8zMsjlpmJlZNicNMzPLttU+PSWpCXi8iza3O/BsF22rqzimPL0xJuidcTmmPFt7TPtFRM1vsdtqk0ZXktTY2iNoPcEx5emNMUHvjMsx5dnWY3L3lJmZZXPSMDOzbE4aeab1dAAtcEx5emNM0Dvjckx5tumYPKZhZmbZfKdhZmbZnDTMzCybk0YVSS+3UDZV0ipJiyQ9JKnD/x9d0sa0ncWSfiFpl1S+t6Rra6xzh6ROP04n6b2SXkj7/6Ok77bR/m9ei+4g6TFJu3dgvZMl7V1a/pGkkV0b3Wb7+7ykN7RSH5K+V1r+kqSpaX6qpFckvblUX5fXu7xdSR+U9CdJ+7UVQ2vx1ynOWufGUEmvprrmabs6xjFF0hJJD6R9fV3Sf1W1GS1paZp/TNKDqf2dkvbL2McQSY9K2jUtD0zLQ1tZp0PnRY50PB/Maeukke+8iBgNjAd+KKl/WyvU8GpEjI6IUcAa4FSAiHgyIo7tolhb8+t0HP8AHCXp3d2wz+5yMrApaUTEJyLioTru7/NAzaQBrAWOaeVEfxb4YpdHVYOkI4DzgSMjovkPX1uLoa34u1qL50ayPNU1T+vqEYCkg4GjgHdExNuAscB84PiqphOAmaXlw1P7O4CvtrWfiHgCuAQ4JxWdA0yLiMc6E38njAacNOohIh4BXgEGdsHmfgfsA5uuphan+R0kzZK0VNJ1wA7NK0ialK4U75F0qaQLU/kgST+TdG+aWk0GEfEqsKh5/62RtKOk2yX9IV1RjU/lb5R0o6T709Xh8an8nHRH9kDz3Uw6vnmp7HZJ+0r613QciyT9UFLfFvZ9vaSF6cpvcirrK+nytM8HJX1B0rFABbgqbW8Hle7QJI1L8d8v6fYW9nOypJ9LulnSI5K+Xar7J0m/S+tfk16Pz1IkqPmS5td46TZQPNXyhRr1PwaOb77arCdJhwGXAkdFxPLMGNqKv542nRvdbC/g2YhYCxARz0bEXcDzkt5VanccmyeNZu2J+zzgIEmfBw4Bviupj6SLVfQE3CbppvTebvbl9J6/R9L+0PK51Ub5R9O5c7+ku1TctZ1F8T5Y1Hwe1xQRnkoT8HILZVOBL6X5d1BcrXdq+0Bf4BpgXFoeCixO86cDP07zb6M4eSsUH1KPAbsC/YFfAxemdjOAQ9L8vsDSFvb9XuCXaX4gsBDYMyPWfsBOaX53YBkg4CPApaX2OwO7AQ/z1yfzdkk/fwFMTPP/BsxNZf1T2cXASWn+MWD3NL9r+rkDsDhtfwxwW2m/zfu4A6iUyu9Ir9sg4AlgWHmbVcd6MrAiHcP2FP+CZkg63ruAN6Z2XwG+Vh1nrdcP2Cm12xn4EjC1/J4CvgZ8o9Z7r4ve0+sprtzf1tL7ulYMrcVfz3OPls+N5oucRcBFdYxhx7SPP6X35HtS+ZcoehsADgIaS+uU36/fBya3Y38fAAJ4f1o+FriJ4oJ+T+B54NjSfqak+ZP467lcfW5d30b5g8A+VefOyaTPkrYm32nk+4KkJcDvgbM7sZ0dJC0CngL2AG5roc1hwJUAEfEA8EAqfydwZ0SsiYj1FCdWs7HAhWnbc4CdJO3YwrYPlXQ/sAq4JSKeyohZwLckPUDxYb9Piv1B4P2SzpV0aES8ALwAvAZcJukYirsygIMpEhvATylOvDHAvSnmI4C3tLDvz6Z476b4EB9O8eH+FkkXSBoHvNhG/AcBd0XEowARsaZGu9sj4oWIeA14CNgvrTsS+E2Kc2IqzxIRLwJXAJ+t0eR8YKKkN+VuswPWA78FJrU3hoz4u1Jr50a5e+rUllfvvIh4meJ9ORloAq6WdDJwNXCspD78bdcUFHecq4AjW6hrzZHAamBUWj4EuCYiXk/nZvVd7MzSz4PTfPW5dUgb5b8BLpf0SYoE3S5OGvnOi4gGiqvryyRt38HtvBrFmMJ+FB/GXXUC9AEOKp1Y+6QToNqvI+LtQAMwSdLojG2fSHG1PibF/jSwfUT8ieLO60Hgm5K+FhEbKJLbtRR9wze3st3ppXjfGhFTy5WS3kuRDA9OMd+X9vs88HaKO4lTgB9lHEOOtaX5jRR3WKK4q2mOc2RE1PrwreX7FB/Yb6yuiIi/UJzYdfsgBF6n6E55p6R/70AMNePvYvU6N9olIjZGxB0R8XXgM8BHohiDeBR4D8VnwNVVqx1OEfci4Bs5+0nn3vspLky+IGmvnPBqzGeLiFMoxl2GAAsl7dae9Z002iki5gCNFFecndnOKxRXb1+U1K+q+i7gXwAkjaLoogK4F3iPiict+lG8eZvdCpzWvNBWMkhX3edQdLe0ZWfgmYhYL6n55EDFk0qvRMSVwHeAd6S7m50j4iaKvvC3p238luIKDYokdDfFldub07Z21d8+dbIz8HxEvCLp7ylOLlQMzPaJiJ9RvPnfkdq/BLR0xX43cJikYc37yjjm8rrvLvUfv1HSAW3sbzPpzmY2ta/0/xv4FEWSqov0fvtn4ERJLcVRM4aM+LtUG+dGXUl6q6ThpaLR/PW/Zc+kGIdYERErq9dNF0yfB05q6z0mSRQD4Z+PiD9TnD/fpbgL+Ega29iDoku57PjSz9+l+epz69etlUv6u4j4fUR8jeJuagiZ72Vw0mjJGyStLE2nt9DmLOD0dKvaYRFxH0XXU/UjvJcAO6p4pAz+myAAAAGASURBVO8sirEHImIV8C3gHoo312MU3UFQnGSVNOj1EMUVeFt+QPFhOrSNdlelbT9I0Zf6x1R+IHBP6lL4OvBNijfeL1NX1gKK8RkoEtrHU/nHKPpYvwrcmspuoxiELLsZ6Jdeh3MoPsCh6B67I+33SuDMVH458IPmgfDmjUREE0V3w89TV1f1VWJNad2TgZkpzt8Bf5+qpwE3tzIQXvY9ivGRlvbxLHAdMCA3ro5IH/7jgK9K+lA7Y6gZfz20cm7U247AdKUHOSi6Jqemumso7tBrdj9FxOpU39Zd0ieBP0dEcxfcxcAI4BlgJUX36JXAH/jrOQ4wMMX1Of76gEL1ufW5Nsq/kwbTF1MklvspusFG5gyE+9+IbGEk7RgRL6crsOsoBsyv6+m4zKxrlM7x3SguEN+dOfbYLbr11s+6xFRJYyme8LkVuL6H4zGzrvVLFX/YuB3wn70pYYDvNMzMrB08pmFmZtmcNMzMLJuThpmZZXPSMDOzbE4aZmaW7f8D4ONjKaAp8RIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQYbEMC7a3V6"
      },
      "source": [
        "# **poor performance of the models**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-CbaLSOa6UJ"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}